diff --git a/AUTHORS.md b/AUTHORS.md
new file mode 100644
index 0000000000000000000000000000000000000000..97ebde0089210d4d45305bc44bdd7a0435cd34ca
--- /dev/null
+++ b/AUTHORS.md
@@ -0,0 +1,25 @@
+# Developers and attribution
+
+## Core project
+
+- **Yu Haoxin** ([@reshuibuduo](https://github.com/reshuibuduo)) — project creator, lead developer, and TMCRA algorithm engineering.
+
+## Development assistance
+
+- **OpenAI Codex** — development and reproducibility engineering assistant, including code organization, benchmark packaging, documentation, and release validation support.
+
+OpenAI Codex is credited here as an AI development assistant; project ownership and release responsibility remain with Yu Haoxin.
+
+---
+
+# 开发者与署名
+
+## 核心项目
+
+- **Yu Haoxin**（[@reshuibuduo](https://github.com/reshuibuduo)）— 项目创建者、主要开发者与 TMCRA 算法工程。
+
+## 开发协助
+
+- **OpenAI Codex** — 开发与可复现性工程助手，协助完成代码整理、benchmark 封装、文档编写和发布校验。
+
+OpenAI Codex 在此作为 AI 开发助手署名；项目所有权与发布责任归 Yu Haoxin 所有。
diff --git a/CHANGELOG.md b/CHANGELOG.md
new file mode 100644
index 0000000000000000000000000000000000000000..e9a63e92731975b4563152adc5bee4b17967b91a
--- /dev/null
+++ b/CHANGELOG.md
@@ -0,0 +1,16 @@
+# Changelog
+
+## 2026.07.18
+
+- Repositioned TMCRA as an Agent Memory Engine with a clear Source / Fast /
+ Slow layered architecture and scope isolation.
+- Added the maintained LongMemEval-S500 reproduction package.
+- Published the 82.2% scorecard with six task-level results.
+- Added the latest node scorer, path scorer, and retrieval reranker weights.
+- Hardened checkpoint loading with `weights_only=True`, published an
+ inference-only reranker checkpoint without training-machine paths, pinned
+ third-party model revisions, and added offline CI and gold-isolation checks.
+- Versioned the maintained answer contract as `evidence_operation_bound_v6`
+ and made compatible interrupted benchmark stages resumable.
+- Switched the project license to Apache-2.0.
+- Credited Yu Haoxin and OpenAI Codex for their respective development roles.
diff --git a/CITATION.cff b/CITATION.cff
new file mode 100644
index 0000000000000000000000000000000000000000..661dcfa45d85f7848dea85482968a9c9f6161fa1
--- /dev/null
+++ b/CITATION.cff
@@ -0,0 +1,27 @@
+cff-version: 1.2.0
+message: "If you use TMCRA, please cite this software. OpenAI Codex assisted with development and reproducibility engineering."
+title: "TMCRA — Agent Memory Engine"
+type: software
+version: "2026.07.18"
+date-released: 2026-07-18
+license: Apache-2.0
+authors:
+ - family-names: Yu
+ given-names: Haoxin
+ alias: reshuibuduo
+ website: "https://github.com/reshuibuduo"
+ - name: "OpenAI Codex"
+ website: "https://openai.com/codex/"
+repository-code: "https://github.com/reshuibuduo/TMCRA-Agent-Memory"
+url: "https://github.com/reshuibuduo/TMCRA-Agent-Memory"
+abstract: >-
+ TMCRA is an agent memory engine that builds scope-isolated, source-traceable
+ memory from long-running conversations and compiles relevant evidence for
+ downstream agents. This release includes the algorithm snapshot, learned
+ graph-scoring artifacts, and a reproducible LongMemEval pipeline.
+keywords:
+ - agent memory
+ - long-term memory
+ - memory graph
+ - retrieval
+ - LongMemEval
diff --git a/LICENSE b/LICENSE
index 459a254c0f1391b8608bb68731cb579a6918e544..effccedafe708c5f408bb892d80ade63f3539a51 100644
--- a/LICENSE
+++ b/LICENSE
@@ -1,21 +1,201 @@
-MIT License
-
-Copyright (c) 2026 TMCRA contributors
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright 2026 Yu Haoxin and TMCRA contributors
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
diff --git a/NOTICE b/NOTICE
new file mode 100644
index 0000000000000000000000000000000000000000..dac6370bfe19e628fd1d879541d4de460636c917
--- /dev/null
+++ b/NOTICE
@@ -0,0 +1,9 @@
+TMCRA — Agent Memory Engine
+Copyright 2026 Yu Haoxin and TMCRA contributors
+
+Development and reproducibility engineering assistance was provided by
+OpenAI Codex. See AUTHORS.md for contributor roles.
+
+This product includes or adapts third-party software and model interfaces.
+The corresponding copyright notices and license terms are documented in
+benchmarks/longmemeval/THIRD_PARTY_NOTICES.md.
diff --git a/README.md b/README.md
index 028a7e8da6d74aa4827f6b70c760fc91b32967d4..7590184ef0ad420a0c478edd48b862fa19970db9 100644
--- a/README.md
+++ b/README.md
@@ -1,207 +1,115 @@
-# TMCRA Long-Memory Runtime
+---
+license: apache-2.0
+library_name: pytorch
+language:
+ - en
+ - zh
+tags:
+ - agent-memory
+ - long-term-memory
+ - memory-graph
+ - information-retrieval
+ - longmemeval
+ - pytorch
+---
+
+# TMCRA — Agent Memory Engine
-[中文版本](README.zh-CN.md)
-
-TMCRA is a runtime layer that gives agent systems long-term memory. It helps an LLM retrieve, connect, and reason over long dialogue histories without exposing the full conversation context on every turn.
-
-This repository contains a frozen TMCRA baseline package with model weights, runtime code snapshots, training metadata, and LongMemEval S500 benchmark results.
-
-License: MIT.
-
-## What TMCRA Does
-
-TMCRA adds a dedicated memory runtime between an agent application and its answer model.
-
-At write time, TMCRA turns dialogue into memory nodes, event units, profile signals, and graph paths. This lets the system preserve not only isolated facts, but also relationships between facts across turns and sessions.
+
+ 简体中文 ·
+ GitHub ·
+ Release v2026.07.18
+
-At retrieval time, TMCRA scores graph nodes and paths, selects compact evidence, and injects only the most relevant memory context into the answer model. The answer model still performs natural-language reasoning, while TMCRA handles long-memory organization, recall, and evidence surfacing.
+TMCRA is a memory engine for long-running agents. It converts multi-turn and multi-session conversations into scope-isolated, source-traceable memory, then recalls compact evidence after the user's next question arrives.
-The current runtime focuses on:
+This Hugging Face repository contains the Apache-2.0 release of the TMCRA algorithm, its LongMemEval reproduction pipeline, and the inference-only graph scoring weights used by the released benchmark chain. Hosted APIs, accounts, billing, and production control-plane services are not included.
-- user fact memory
-- assistant-response memory
-- profile and preference memory
-- temporal memory
-- cross-session graph tunneling
-- learned node/path scoring
-- compact evidence selection for downstream LLMs
+## LongMemEval S500
-## Algorithm Structure
+TMCRA achieved **411 / 500 = 82.2%**.
-```mermaid
-flowchart LR
- A["Dialogue Turns"] --> B["Writer Layer"]
- B --> C["Memory Nodes"]
- B --> D["Event Units"]
- B --> E["Profile Signals"]
- B --> F["Temporal Signals"]
+| Task category | Correct / Total | Accuracy |
+| --- | ---: | ---: |
+| Knowledge Update | 71 / 78 | 91.0% |
+| Multi-session | 90 / 133 | 67.7% |
+| Single-session Assistant | 55 / 56 | 98.2% |
+| Single-session Preference | 27 / 30 | 90.0% |
+| Single-session User | 67 / 70 | 95.7% |
+| Temporal Reasoning | 101 / 133 | 75.9% |
+| **Overall** | **411 / 500** | **82.2%** |
- C --> G["Graph Memory Store"]
- D --> G
- E --> G
- F --> G
+The machine-readable scorecard is [`results/latest_benchmark.json`](results/latest_benchmark.json). The maintained reproduction entrypoint is [`benchmarks/longmemeval/`](benchmarks/longmemeval/README.md). The benchmark answer layer is fixed to GPT-5.4; fresh runs that call external model services can vary slightly.
- G --> H["Learned Node Scorer"]
- G --> I["Learned Path Scorer"]
- H --> J["Evidence Selection"]
- I --> J
+## Architecture
- J --> K["Compact Memory Context"]
- K --> L["Answer Model"]
- L --> M["Agent Response"]
+TMCRA separates memory construction from downstream answer generation.
- N["User Query"] --> H
- N --> I
- N --> J
- N --> L
+```text
+Conversation history
+ │
+ ▼
+ Writer ─────────────── subject / entity attribution
+ │
+ ├── Source layer ─── immutable, traceable evidence
+ ├── Fast layer ───── atomic current-state memory
+ └── Slow layer ───── durable semantic graph
+ │
+ └────────── scope-bound indexes
+
+New question
+ │
+ ▼
+Recall Planner → layered retrieval → node/path scoring → reranker
+ │
+ ▼
+Source-bound evidence packet → downstream agent
```
-The writer layer produces memory units from dialogue. The graph memory store preserves facts, events, profile signals, temporal signals, and cross-session links. The learned node/path scorers select relevant evidence for the current query, and the answer model uses that compact evidence to produce the final response.
+The TMCRA core ends at the evidence packet. The answer model and benchmark judge are harness components, not part of the online memory engine. User statements and assistant progress remain separately attributed, while both can be recalled when relevant.
-## Why TMCRA
+## Released weights
-Long-running agents need more than simple vector recall. They need to preserve user facts, preferences, timeline changes, cross-session events, and multi-step evidence chains.
+All three checkpoints contain only their public inference contracts and model state. Training state, machine paths, and private run metadata are excluded. Load them with `torch.load(..., weights_only=True)` through the repository runtime.
-TMCRA organizes memory into graph nodes and learned retrieval paths, then surfaces compact evidence to the answer model. The goal is to let external agents use long-term memory through a runtime/API layer while keeping the memory algorithm and model weights independently deployable.
+| File | Purpose | Size | SHA-256 |
+| --- | --- | ---: | --- |
+| `node_scorer.pt` | Graph node scoring | 207,356,243 bytes | `d2318aafb07f9a15a1d95e6eac1b3e09afa07c570dd9d15d3f75544c9950f201` |
+| `path_scorer.pt` | Graph path scoring | 207,356,243 bytes | `ecdc2bca51b7646a8a56c7db9cea4aeec581e64c3b61e6e06f7e9fa2a93b3bf5` |
+| `tmcra_v3_reranker.pt` | Learned retrieval reranking | 4,232,213 bytes | `09a285b484ca857b24b53ad19d5998302f16d224ca9e4fd73a4eb5a52f022942` |
-## How to Use
+The files and their machine-readable contract are under [`models/tmcra_v4_longmemeval_s500_20260715/`](models/tmcra_v4_longmemeval_s500_20260715/README.md).
-For inference/runtime use, load the graph scorer weights from:
-
-```text
-models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/
-```
+## Download
-The main runtime files are:
-
-```text
-node_scorer.pt
-path_scorer.pt
-export_manifest.json
-```
-
-A typical runtime configuration points TMCRA to these weights:
+Install the current Hugging Face CLI and download only the released inference package:
```bash
-export TMCRA_NODE_MODEL_PATH="models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer.pt"
-export TMCRA_PATH_MODEL_PATH="models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer.pt"
-export TMCRA_RETRIEVAL_MODE="hybrid_node_scored"
-export TMCRA_REQUIRE_LEARNED_SCORER="1"
-```
-
-The evaluation entrypoint snapshot is:
+python -m pip install -U huggingface_hub
-```text
-code/run_lme_s10_native_tmcra.py
-```
-
-The core adapter snapshot is:
-
-```text
-code/memory_adapters.py
+hf download 2009YU/TMCRA-Agent-Memory \
+ --include "models/tmcra_v4_longmemeval_s500_20260715/*" \
+ --local-dir ./TMCRA-Agent-Memory
```
-For a deployment build, load the two scorer files into the TMCRA adapter and point the agent's memory middleware to the TMCRA retrieval API. The answer model can be any OpenAI-compatible or local LLM endpoint; TMCRA supplies the selected memory evidence, and the answer model produces the final response.
-
-## Dependency Environment
+For source, benchmark setup, environment examples, asset checksums, and stage validation, use the [GitHub release](https://github.com/reshuibuduo/TMCRA-Agent-Memory/releases/tag/v2026.07.18) or [`benchmarks/longmemeval/README.md`](benchmarks/longmemeval/README.md).
-The included code snapshot is Python-based. A practical runtime environment should include:
+## Version history
-- Python 3.10 or newer
-- PyTorch, with CUDA recommended for model inference
-- NumPy and standard Python data-processing libraries
-- an OpenAI-compatible or local LLM endpoint for the answer layer and writer layer
-- optional Git LFS support when pulling the full model package from GitHub
+- Current release: [`v2026.07.18`](https://huggingface.co/2009YU/TMCRA-Agent-Memory/tree/v2026.07.18)
+- Preserved pre-release snapshot: [`legacy-2026.05.25`](https://huggingface.co/2009YU/TMCRA-Agent-Memory/tree/legacy-2026.05.25)
-The benchmark scripts expect LongMemEval-format input data and write JSONL predictions and judge outputs. Runtime deployments can use the same model files without running the benchmark harness.
+## Developers
-## Optional Modules
+- **Yu Haoxin** — creator, lead developer, and TMCRA algorithm engineering.
+- **OpenAI Codex** — development and reproducibility engineering assistant.
-TMCRA also keeps optional extension points for retrieval and planning experiments. These modules can be enabled in deployment or evaluation builds when the target use case needs them.
+See [`AUTHORS.md`](AUTHORS.md) and [`CITATION.cff`](CITATION.cff) for attribution and citation details.
-- **Embedder interface**: an optional semantic embedding channel that can run alongside the graph-memory scorer. It is intended to provide additional dense semantic recall before or during graph evidence selection, without replacing the learned graph node/path scorers.
-- **LLM planner interface**: an optional planner hook that can use an external LLM to organize evidence, expand query intent, or create an answer plan before the final answer call. This is useful for experiments and higher-cost deployments, while the default baseline keeps the core graph scorer path independently measurable.
+## License
-These interfaces are integration points, not required dependencies for the frozen S500 baseline. They are designed so downstream deployments can decide whether to run a lighter scorer-only path or a heavier path with embedder/planner assistance.
-
-## Included Artifacts
-
-- `code/`: runtime and evaluation code snapshot for this baseline.
-- `models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/`: full trained graph-model output directory.
-- `results/`: predictions, judge output, summary metrics, and compressed run artifacts.
-- `docs/`: baseline record and result notes.
-
-Additional documentation:
-
-- `docs/BASELINE_S500_20260525.md`: benchmark record and subtask metrics.
-- `docs/TRAINING.md`: model training direction and released training artifacts.
-- `docs/OPTIONAL_MODULES_AND_PARALLEL.md`: optional embedder/planner integration points and parallel evaluation plan.
-- `docs/OPTIONAL_MODULES_AND_PARALLEL.zh-CN.md`: Chinese version of the optional module and parallel evaluation guide.
-
-## Model Package
-
-The included model package preserves the full training output for the graph scorer stack:
-
-- `node_scorer.pt` and `path_scorer.pt`: runtime graph scoring weights.
-- `node_scorer_best.pt` and `path_scorer_best.pt`: best checkpoint aliases.
-- `node_scorer_last.pt` and `path_scorer_last.pt`: final training aliases.
-- `checkpoints/`: epoch and step checkpoints.
-- `export_manifest.json`, `train_summary.json`, and `train.log`: model metadata and training trace.
-
-## Current Strengths
-
-- Strong direct user-fact recall in single-session settings.
-- Strong assistant-detail recall.
-- Competitive knowledge-update behavior for changing facts.
-- Working temporal and preference retrieval layers with clear room for further specialization.
-
-## Active Improvement Areas
-
-- Multi-session aggregation and unit coverage.
-- Deeper time-graph reasoning.
-- Preference-profile abstraction and cross-session tunneling.
-- Query-graph to memory-graph matching for complex questions.
-
-## Intended Use
-
-This repository is a public-facing evidence package for TMCRA's long-memory runtime work. It is suitable for:
-
-- Benchmark review.
-- Model and result inspection.
-- Reproducing the frozen baseline.
-- Demonstrating how TMCRA can be packaged as an external memory runtime for agents.
-
-## Benchmark Result
-
-This package includes a full LongMemEval S500 run.
-
-- Benchmark: LongMemEval S set, 500 samples
-- Evaluation: official-compatible LongMemEval judge prompt
-- Judge model: `gpt-4o`, resolved as `gpt-4o-2024-08-06`
-- Writer layer used in this run: DeepSeek v4 Flash
-- Answer layer used in this run: GPT5.4-compatible API
-- Overall accuracy: `310 / 500 = 62.00%`
-
-## Results by Task Type
-
-| task type | accuracy | count |
-| --- | ---: | ---: |
-| single-session-user | 81.43% | 70 |
-| single-session-assistant | 78.57% | 56 |
-| knowledge-update | 70.51% | 78 |
-| temporal-reasoning | 63.16% | 133 |
-| single-session-preference | 56.67% | 30 |
-| multi-session | 39.85% | 133 |
-
-The benchmark outputs are available in:
-
-```text
-results/predictions.jsonl
-results/judge_gpt4o_alias_vectorengine.jsonl
-results/judge_gpt4o_alias_vectorengine.jsonl.summary.json
-results/lme_s500_frozen_baseline38_full10_20260525_results.tar.gz
-```
+TMCRA is released under the [Apache License 2.0](LICENSE). Third-party datasets, models, and components retain their own licenses.
diff --git a/README.zh-CN.md b/README.zh-CN.md
index a779dd5f982504f4017f5d406d7d48e60f78baad..d2d792a6da24a76e0992917ad7ffadfda9a93e6f 100644
--- a/README.zh-CN.md
+++ b/README.zh-CN.md
@@ -1,207 +1,100 @@
-# TMCRA 长记忆运行时
+# TMCRA — 面向 Agent 的记忆引擎
-[English version](README.md)
-
-TMCRA 是给 Agent 系统提供长期记忆能力的运行时层。它帮助大语言模型在长对话历史中检索、连接和推理相关记忆，而不需要在每一轮都暴露完整上下文。
-
-本仓库包含一个冻结的 TMCRA 基线包，包括模型权重、运行时代码快照、训练元数据和 LongMemEval S500 测评结果。
-
-许可证：MIT。
-
-## TMCRA 的作用
-
-TMCRA 在 Agent 应用和回答模型之间增加一层专门的记忆运行时。
-
-在写入阶段，TMCRA 会把对话转成记忆节点、事件单元、profile 信号和图路径。这让系统不仅能保存孤立事实，也能保存不同轮次、不同会话之间的事实关系。
+
+ English ·
+ GitHub ·
+ v2026.07.18 发布页
+
-在召回阶段，TMCRA 会对图节点和路径进行打分，选择紧凑的证据，并只把最相关的记忆上下文注入给回答模型。回答模型仍然负责自然语言推理和最终表达，TMCRA 负责长记忆组织、召回和证据呈现。
+TMCRA 是面向长期运行 Agent 的记忆引擎。它把多轮、跨会话的对话整理成作用域隔离、来源可追溯的记忆，并在用户提出下一轮问题后，召回与当前问题相关的紧凑证据。
-当前运行时重点覆盖：
+这个 Hugging Face 仓库包含采用 Apache-2.0 许可发布的 TMCRA 算法、LongMemEval 复现链路，以及本次 benchmark 使用的推理权重。线上 API、账号、计费和生产控制台不在本仓库的开源范围内。
-- 用户事实记忆
-- 助手回答细节记忆
-- profile 与偏好记忆
-- 时间记忆
-- 跨会话图隧穿
-- 学习式节点/路径打分
-- 面向下游 LLM 的紧凑证据选择
+## LongMemEval S500 成绩
-## 算法结构图
+TMCRA 的成绩为 **411 / 500 = 82.2%**。
-```mermaid
-flowchart LR
- A["对话轮次"] --> B["写入层"]
- B --> C["记忆节点"]
- B --> D["事件单元"]
- B --> E["Profile 信号"]
- B --> F["时间信号"]
+| 任务类型 | 正确数 / 总数 | 准确率 |
+| --- | ---: | ---: |
+| 信息更新（Knowledge Update） | 71 / 78 | 91.0% |
+| 跨会话整合（Multi-session） | 90 / 133 | 67.7% |
+| 单会话助手信息（Single-session Assistant） | 55 / 56 | 98.2% |
+| 单会话偏好（Single-session Preference） | 27 / 30 | 90.0% |
+| 单会话用户信息（Single-session User） | 67 / 70 | 95.7% |
+| 时间推理（Temporal Reasoning） | 101 / 133 | 75.9% |
+| **总成绩** | **411 / 500** | **82.2%** |
- C --> G["图记忆库"]
- D --> G
- E --> G
- F --> G
+机器可读的成绩文件位于 [`results/latest_benchmark.json`](results/latest_benchmark.json)，完整复现入口位于 [`benchmarks/longmemeval/`](benchmarks/longmemeval/README.zh-CN.md)。复现链路固定使用 GPT-5.4 作为回答层；重新运行时如果调用外部模型服务，结果可能有小幅波动。
- G --> H["学习式节点 Scorer"]
- G --> I["学习式路径 Scorer"]
- H --> J["证据选择"]
- I --> J
+## 架构
- J --> K["紧凑记忆上下文"]
- K --> L["回答模型"]
- L --> M["Agent 回复"]
+TMCRA 把记忆构建与最终回答生成明确分开。
- N["用户问题"] --> H
- N --> I
- N --> J
- N --> L
+```text
+历史对话
+ │
+ ▼
+ Writer ─────────────── 说话者与实体归因
+ │
+ ├── Source 层 ────── 不可变、可追溯的原始证据
+ ├── Fast 层 ──────── 反映当前状态的原子记忆
+ └── Slow 层 ──────── 可跨会话保留的语义图
+ │
+ └──────── 作用域内索引
+
+用户的新问题
+ │
+ ▼
+召回规划 → 分层检索 → 节点/路径打分 → 重排
+ │
+ ▼
+带来源绑定的证据包 → 下游 Agent
```
-写入层从对话中生成记忆单元。图记忆库保存事实、事件、profile 信号、时间信号和跨会话连接。学习式节点/路径 scorer 会根据当前问题选择相关证据，回答模型使用这些紧凑证据生成最终回复。
+TMCRA 的核心边界止于证据包；回答模型和 benchmark Judge 属于评测链路，不是在线记忆引擎本身。用户陈述和 Agent 完成的工作进度会分别保留各自的主体归属，同时都能在相关时被召回。
-## 为什么需要 TMCRA
+## 本次发布的权重
-长期运行的 Agent 不能只依赖简单向量召回。它需要保存用户事实、偏好、时间线变化、跨会话事件，以及多步证据链。
+三份权重只保留公开推理所需的结构定义和模型状态，不包含训练状态、训练机器路径或私有运行信息。请通过仓库运行时使用 `torch.load(..., weights_only=True)` 安全加载。
-TMCRA 将记忆组织成图节点和学习得到的召回路径，再把压缩后的证据提供给回答模型。目标是让外部 Agent 可以通过运行时/API 层使用长期记忆，同时保持记忆算法和模型权重可以独立部署。
+| 文件 | 用途 | 大小 | SHA-256 |
+| --- | --- | ---: | --- |
+| `node_scorer.pt` | 图节点打分 | 207,356,243 bytes | `d2318aafb07f9a15a1d95e6eac1b3e09afa07c570dd9d15d3f75544c9950f201` |
+| `path_scorer.pt` | 图路径打分 | 207,356,243 bytes | `ecdc2bca51b7646a8a56c7db9cea4aeec581e64c3b61e6e06f7e9fa2a93b3bf5` |
+| `tmcra_v3_reranker.pt` | 召回重排 | 4,232,213 bytes | `09a285b484ca857b24b53ad19d5998302f16d224ca9e4fd73a4eb5a52f022942` |
-## 如何使用
+权重和机器可读的发布清单位于 [`models/tmcra_v4_longmemeval_s500_20260715/`](models/tmcra_v4_longmemeval_s500_20260715/README.md)。
-推理或运行时使用时，加载下面目录中的图 scorer 权重：
-
-```text
-models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/
-```
+## 下载
-主要运行时文件是：
-
-```text
-node_scorer.pt
-path_scorer.pt
-export_manifest.json
-```
-
-典型运行配置如下：
+安装最新版 Hugging Face CLI，然后只下载本次发布的推理包：
```bash
-export TMCRA_NODE_MODEL_PATH="models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer.pt"
-export TMCRA_PATH_MODEL_PATH="models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer.pt"
-export TMCRA_RETRIEVAL_MODE="hybrid_node_scored"
-export TMCRA_REQUIRE_LEARNED_SCORER="1"
-```
-
-测评入口代码快照：
+python -m pip install -U huggingface_hub
-```text
-code/run_lme_s10_native_tmcra.py
-```
-
-核心适配器代码快照：
-
-```text
-code/memory_adapters.py
+hf download 2009YU/TMCRA-Agent-Memory \
+ --include "models/tmcra_v4_longmemeval_s500_20260715/*" \
+ --local-dir ./TMCRA-Agent-Memory
```
-部署时，将两个 scorer 文件加载到 TMCRA adapter 中，并让 Agent 的记忆中间件调用 TMCRA 召回 API。回答模型可以是任意 OpenAI 兼容接口或本地 LLM；TMCRA 提供选中的记忆证据，回答模型生成最终回复。
-
-## 依赖环境
+源代码、benchmark 配置、环境示例、资源校验和分阶段检查说明可查看 [GitHub 发布页](https://github.com/reshuibuduo/TMCRA-Agent-Memory/releases/tag/v2026.07.18) 或 [`benchmarks/longmemeval/README.zh-CN.md`](benchmarks/longmemeval/README.zh-CN.md)。
-当前代码快照基于 Python。建议运行环境包括：
+## 版本记录
-- Python 3.10 或更高版本
-- PyTorch，推荐使用 CUDA 做模型推理
-- NumPy 及常见 Python 数据处理库
-- 用于回答层和写入层的 OpenAI 兼容接口或本地 LLM endpoint
-- 从 GitHub 拉取完整模型包时建议支持 Git LFS
+- 当前版本：[`v2026.07.18`](https://huggingface.co/2009YU/TMCRA-Agent-Memory/tree/v2026.07.18)
+- 更新前保留版本：[`legacy-2026.05.25`](https://huggingface.co/2009YU/TMCRA-Agent-Memory/tree/legacy-2026.05.25)
-benchmark 脚本使用 LongMemEval 格式输入数据，并输出 JSONL 格式的预测和 judge 结果。实际运行时部署可以直接使用同一套模型文件，不需要运行 benchmark harness。
+## 开发者
-## 可开启模块
+- **Yu Haoxin**：项目创建者、主要开发者与 TMCRA 算法工程。
+- **OpenAI Codex**：开发与可复现性工程协作。
-TMCRA 当前也保留了用于召回和规划实验的可选接入口。部署或评估时，如果目标场景需要，可以按需开启这些模块。
+署名与引用信息见 [`AUTHORS.md`](AUTHORS.md) 和 [`CITATION.cff`](CITATION.cff)。
-- **Embedder 接入口**：可选的语义 embedding 通道，可以和图记忆 scorer 并行工作。它用于在图证据选择前或选择过程中提供额外的稠密语义召回，但不替代学习式图节点/路径 scorer。
-- **LLM planner 接入口**：可选的规划 hook，可以调用外部 LLM 来整理证据、扩展 query intent，或在最终回答前生成 answer plan。它适合实验和更高成本的部署场景；默认基线仍保持核心图 scorer 链路可以独立测量。
+## 许可证
-这些接口是可集成的扩展点，不是冻结 S500 基线的必需依赖。下游部署可以根据成本和效果选择轻量的 scorer-only 路径，或开启 embedder/planner 辅助的增强路径。
-
-## 包含内容
-
-- `code/`：该基线使用的运行时和测评代码快照。
-- `models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/`：完整训练后的图模型输出目录。
-- `results/`：预测结果、judge 输出、汇总指标和压缩后的运行结果。
-- `docs/`：基线记录和结果说明。
-
-补充文档：
-
-- `docs/BASELINE_S500_20260525.md`：benchmark 记录和子任务指标。
-- `docs/TRAINING.md`：模型训练方向和随包发布的训练产物说明。
-- `docs/OPTIONAL_MODULES_AND_PARALLEL.zh-CN.md`：embedder/planner 可选接入口和并行评估推进方案。
-- `docs/OPTIONAL_MODULES_AND_PARALLEL.md`：上述文档的英文版本。
-
-## 模型包
-
-随包发布的模型目录保留了图 scorer 栈的完整训练输出：
-
-- `node_scorer.pt` 和 `path_scorer.pt`：运行时图打分权重。
-- `node_scorer_best.pt` 和 `path_scorer_best.pt`：验证表现最好的 checkpoint 别名。
-- `node_scorer_last.pt` 和 `path_scorer_last.pt`：训练结束时的 checkpoint 别名。
-- `checkpoints/`：按 epoch 和 step 保存的中间 checkpoint。
-- `export_manifest.json`、`train_summary.json` 和 `train.log`：模型元数据和训练轨迹。
-
-## 当前优势
-
-- 单会话直接用户事实召回较强。
-- 助手细节召回较强。
-- 对变化事实的 knowledge-update 任务有较稳定表现。
-- 已具备可工作的时间记忆和偏好召回层，后续仍有专项优化空间。
-
-## 当前改进方向
-
-- multi-session 聚合和 unit coverage。
-- 更深的时间图推理。
-- 偏好 profile 抽象和跨会话隧穿。
-- query graph 与 memory graph 的复杂问题匹配。
-
-## 适用场景
-
-本仓库是 TMCRA 长记忆运行时工作的对外展示包，适合用于：
-
-- benchmark 审阅
-- 模型和结果检查
-- 复现冻结基线
-- 展示 TMCRA 如何作为外部 Agent 的记忆运行时进行封装
-
-## Benchmark 结果
-
-本包包含一次完整 LongMemEval S500 测试。
-
-- Benchmark：LongMemEval S set，500 条样本
-- Evaluation：official-compatible LongMemEval judge prompt
-- Judge 模型：`gpt-4o`，解析为 `gpt-4o-2024-08-06`
-- 本轮使用的写入层：DeepSeek v4 Flash
-- 本轮使用的回答层：GPT5.4-compatible API
-- 总准确率：`310 / 500 = 62.00%`
-
-## 各任务类型结果
-
-| 任务类型 | 准确率 | 数量 |
-| --- | ---: | ---: |
-| single-session-user | 81.43% | 70 |
-| single-session-assistant | 78.57% | 56 |
-| knowledge-update | 70.51% | 78 |
-| temporal-reasoning | 63.16% | 133 |
-| single-session-preference | 56.67% | 30 |
-| multi-session | 39.85% | 133 |
-
-benchmark 输出文件位于：
-
-```text
-results/predictions.jsonl
-results/judge_gpt4o_alias_vectorengine.jsonl
-results/judge_gpt4o_alias_vectorengine.jsonl.summary.json
-results/lme_s500_frozen_baseline38_full10_20260525_results.tar.gz
-```
+TMCRA 采用 [Apache License 2.0](LICENSE) 开源；第三方数据集、模型和组件仍遵循各自的许可证。
diff --git a/SECURITY.md b/SECURITY.md
new file mode 100644
index 0000000000000000000000000000000000000000..369827783bf24c1c9377998f707d433add14819e
--- /dev/null
+++ b/SECURITY.md
@@ -0,0 +1,12 @@
+# Security policy
+
+Do not open a public issue containing credentials, private memory data, or a
+working exploit. Use GitHub's private vulnerability reporting for this
+repository when available.
+
+TMCRA checkpoints must be loaded with `torch.load(..., weights_only=True)` and
+verified against the published SHA-256 manifest. The maintainers do not
+support disabling these checks for untrusted checkpoints.
+
+Provider API keys belong only in local `*.env` files. The example files contain
+placeholders and real environment files are excluded by `.gitignore`.
diff --git a/benchmarks/longmemeval/README.md b/benchmarks/longmemeval/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..cd1354ea7f0468c4664b0a56191bc6075397a947
--- /dev/null
+++ b/benchmarks/longmemeval/README.md
@@ -0,0 +1,216 @@
+# TMCRA Agent Memory System — LongMemEval Benchmark
+
+[简体中文](README.zh-CN.md)
+
+TMCRA is a scope-isolated, layered memory engine for agents. It keeps durable evidence outside the answer model and returns a compact, traceable memory context for each query. This directory contains the LongMemEval pipeline used to build memory, retrieve across multiple memory layers, compile evidence, generate answers, and evaluate the predictions.
+
+## Benchmark result
+
+TMCRA achieves **82.2%** on LongMemEval.
+
+| Task category | Correct / Total | Accuracy |
+| --- | ---: | ---: |
+| Knowledge Update | 71 / 78 | 91.0% |
+| Multi-session | 90 / 133 | 67.7% |
+| Single-session Assistant | 55 / 56 | 98.2% |
+| Single-session Preference | 27 / 30 | 90.0% |
+| Single-session User | 67 / 70 | 95.7% |
+| Temporal Reasoning | 101 / 133 | 75.9% |
+| **Overall** | **411 / 500** | **82.2%** |
+
+The machine-readable aggregate result is available in [`results/benchmark.json`](results/benchmark.json).
+
+## Architecture
+
+TMCRA separates immutable source evidence from two derived memory layers. The Writer stores conversation messages in Source and extracts normalized, current assertions into Fast. Subject Attribution prevents quoted or third-party facts from being promoted under the wrong identity. Eligible records are reorganized into the durable Slow semantic graph, while every derived memory remains traceable to Source.
+
+At query time, the Recall Planner resolves the question and assigns layer roles. Source, Fast, and Slow candidates are retrieved, graph-ranked, reranked, and packed into a bounded evidence set. The Evidence Compiler converts this set into a structured, provenance-bound packet. TMCRA's online memory boundary ends at prompt-ready evidence; answer generation and the official Judge belong to the benchmark harness.
+
+```mermaid
+flowchart LR
+ subgraph BENCH["Benchmark Adapter"]
+ D["LongMemEval Dataset"] --> H["Data Adapter / Harness"]
+ Q["Query"]
+ G["Gold Answer"]
+ D --> G
+ end
+
+ subgraph CORE["TMCRA Memory and Retrieval Core"]
+ W["Writer"]
+ S["Source Layer
immutable records"]
+ W --> F["Fast Layer
atomic current memory"]
+ W --> S
+ F --> A["Subject Attribution
promotion gate"]
+ A --> L["Slow Layer
durable semantic graph"]
+ S --> I["Index
searchable snapshot"]
+ F --> I
+ L --> I
+ P["Recall Planner
query and layer roles"]
+ P --> R["Layered Retrieval
Source + Fast + Slow"]
+ I --> R
+ R --> K["Graph / Reranker
Top-K evidence packing"]
+ K --> C["Evidence Compiler
traceable evidence packet"]
+ end
+
+ subgraph EVAL["Benchmark Evaluation Only"]
+ C --> M["Answer Model"]
+ M --> J["Official Judge"]
+ J --> O["Overall and task-level scores"]
+ end
+
+ H -->|"conversation history"| W
+ H -->|"question only"| Q
+ Q --> P
+ G -->|"reference used only here"| J
+```
+
+### Component responsibilities
+
+- **Source** retains the immutable conversation record and is the final evidence authority.
+- **Fast** stores source-bound atomic facts, relations, updates, and current state.
+- **Subject Attribution** stops quoted or third-party information from being assigned to the wrong subject.
+- **Slow** organizes eligible cross-session information into a durable semantic graph; it supplements rather than replaces Source.
+- **Recall Planner** interprets the question and memory-layer roles without reading the gold answer.
+- **Layered Retrieval** searches Source, Fast, and Slow, then performs graph ranking, reranking, deduplication, temporal handling, and fixed Top-K packing.
+- **Evidence Compiler** produces a structured evidence packet whose claims remain bound to source IDs.
+- **Answer Model and Judge** are benchmark-only components and are not part of the TMCRA online recall service.
+
+Gold answers are isolated from Writer, Planner, Retrieval, Evidence Compiler, and Answer Model. They are only read by the final Judge.
+
+## Viewing and reproducing the benchmark
+
+- **View the published result:** [`results/benchmark.json`](results/benchmark.json) contains the public overall and task-level summary shown above.
+- **Rerun the pipeline:** the commands below execute memory construction, layered retrieval, evidence compilation, answer generation, and evaluation. The published 82.2% scorecard is a frozen v5 result; this release uses the security-hardened v6 answer protocol, and fresh external model calls can produce a different score.
+
+### Tested environment
+
+- Linux
+- Python 3.12.3
+- NVIDIA GPU with 24 GiB VRAM or more
+- CUDA 12.8
+- PyTorch 2.11.0 + cu128
+- Transformers 5.10.2
+
+The full S500 run also needs sufficient system memory and disk space for Writer journals, graphs, indexes, checkpoints, and intermediate evidence artifacts.
+
+### 1. Clone and install
+
+```bash
+git lfs install
+GIT_LFS_SKIP_SMUDGE=1 git clone https://github.com/reshuibuduo/TMCRA-Agent-Memory.git
+cd TMCRA-Agent-Memory/benchmarks/longmemeval
+git lfs pull --include="models/tmcra_v4_longmemeval_s500_20260715/*.pt"
+
+python3.12 -m venv .venv
+source .venv/bin/activate
+python -m pip install --upgrade pip
+python -m pip install -r requirements.txt
+python -m pip install -e .
+```
+
+`GIT_LFS_SKIP_SMUDGE=1` prevents the clone from downloading unrelated historical LFS objects; the next command pulls only the three checkpoints required by this benchmark.
+
+### 2. Verify checkpoints and download external assets
+
+The three TMCRA checkpoints are tracked in the repository with Git LFS under `models/tmcra_v4_longmemeval_s500_20260715/`. The asset script verifies their byte sizes and SHA-256 hashes in place; it does not create a second copy. It then downloads the official cleaned LongMemEval-S dataset and the revision-pinned BGE models:
+
+```bash
+python scripts/fetch_assets.py --manifest configs/assets.lock.json --kind all
+```
+
+The dataset comes from the [official LongMemEval release](https://github.com/xiaowu0162/LongMemEval) and [cleaned dataset repository](https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned). BGE-M3 and the reranker are downloaded from the official [BAAI/bge-m3](https://huggingface.co/BAAI/bge-m3) and [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) model pages.
+
+File assets are verified against `configs/assets.lock.json`. If the checkout contains only LFS pointer files, the script stops with the exact `git lfs pull` command required to fetch the weights.
+
+### 3. Configure model providers
+
+```bash
+cp configs/writer.env.example configs/writer.env
+cp configs/answer.env.example configs/answer.env
+cp configs/judge.env.example configs/judge.env
+cp configs/run.env.example configs/run.env
+```
+
+Edit the four generated files:
+
+- `writer.env` configures one OpenAI-compatible DeepSeek endpoint and a comma-separated key pool used by the Writer, subject-attribution pass, and evidence planner. The endpoint must accept `deepseek-v4-flash` and `deepseek-v4-pro`, or map those names to compatible deployments.
+- `answer.env` must expose the exact model identifier `gpt-5.4` through an OpenAI-compatible endpoint.
+- `judge.env` must expose `gpt-4o-2024-08-06`. The public command passes the `gpt-4o` alias, and the bundled evaluator resolves that alias to the fixed version.
+- `run.env` selects local paths, concurrency, and PyTorch devices.
+
+Keep API keys local; the repository `.gitignore` excludes the real configuration files.
+
+The default Judge wire format is OpenAI Chat Completions. Set `OPENAI_WIRE_API=responses` in `judge.env` only if the configured endpoint implements the Responses API.
+
+### 4. Run the offline smoke test
+
+This test uses a synthetic fixture and does not call external model providers:
+
+```bash
+bash scripts/reproduce_smoke.sh
+```
+
+### 5. Run LongMemEval-S500
+
+```bash
+bash scripts/reproduce_s500.sh
+```
+
+The script executes and validates the following stages:
+
+```text
+prepare data and ordered QIDs
+ -> build Writer journals, Source/Fast/Slow memory, and indexes
+ -> retrieve Source + Fast + Slow evidence
+ -> compile operation-bound evidence packets
+ -> generate answers with GPT-5.4
+ -> evaluate with the official GPT-4o Judge
+ -> export overall and task-level scores
+```
+
+Important outputs are written under `runs/longmemeval_s500/`:
+
+```text
+BUILD_COMPLETE
+retrieval_s500/evidence_windows.jsonl
+retrieval_s500/retrieval_debug.jsonl
+retrieval_s500_compiled/COMPILE_COMPLETE
+retrieval_s500_compiled/evidence_windows.jsonl
+answers_s500/answers.jsonl
+official_judge_gpt4o_20240806.jsonl
+scorecard.json
+```
+
+Before the next stage starts, the runner requires completion markers or reports, exact row counts, and the same ordered QID set. Judge rows are also checked for boolean labels from `gpt-4o-2024-08-06`. Compatible incomplete retrieval, compilation, answer, and judge artifacts are resumed. Build recovery is fail-closed and may require the explicit review flags printed by the build runner; an incompatible answer protocol requires a new answer output directory.
+
+## Repository layout
+
+```text
+configs/ provider examples and hash-pinned asset manifest
+fixtures/ synthetic offline test data
+results/ published aggregate score
+scripts/ asset download, validation, smoke, and S500 runners
+src/ TMCRA V4 benchmark core, LongMemEval harness, and minimal adapter
+tests/ offline schema, gold-isolation, and score aggregation tests
+../../models/tmcra_v4_longmemeval_s500_20260715/
+ Git LFS checkpoints used by this benchmark
+```
+
+Production HTTP APIs, account systems, billing, tenant authorization, deployment files, credentials, runtime databases, raw provider logs, and control-plane components are intentionally excluded.
+
+## Validation
+
+```bash
+python -m compileall -q src scripts tests
+python -m pytest
+```
+
+Before publishing a release, also run a secret scan, dependency vulnerability scan, SBOM generation, and a manual review of staged files and release assets.
+
+## Citation
+
+When reporting these results, identify the repository version and cite the [LongMemEval project](https://github.com/xiaowu0162/LongMemEval) and its paper.
+
+## License
+
+TMCRA benchmark code is released under the [Apache License 2.0](../../LICENSE). LongMemEval and the downloaded BGE models retain their own licenses; see [`THIRD_PARTY_NOTICES.md`](THIRD_PARTY_NOTICES.md).
diff --git a/benchmarks/longmemeval/README.zh-CN.md b/benchmarks/longmemeval/README.zh-CN.md
new file mode 100644
index 0000000000000000000000000000000000000000..ce803ba324f51bd6f6a2cad6835a5f6df69f04f4
--- /dev/null
+++ b/benchmarks/longmemeval/README.zh-CN.md
@@ -0,0 +1,216 @@
+# TMCRA 智能体记忆系统 — LongMemEval Benchmark
+
+[English](README.md)
+
+TMCRA 是面向智能体的作用域隔离、分层记忆引擎。它将可追溯的长期证据保存在回答模型之外，并在每次查询时返回紧凑、可核验的记忆上下文。本目录包含 TMCRA 的 LongMemEval benchmark 链路，覆盖记忆构建、多层召回、证据整理、回答生成与结果评测。
+
+## Benchmark 成绩
+
+TMCRA 在 LongMemEval benchmark 上取得 **82.2%**。
+
+| 任务类型 | 正确数 / 样本数 | 准确率 |
+| --- | ---: | ---: |
+| 信息更新（Knowledge Update） | 71 / 78 | 91.0% |
+| 跨会话信息整合（Multi-session） | 90 / 133 | 67.7% |
+| 单会话助手信息（Single-session Assistant） | 55 / 56 | 98.2% |
+| 单会话偏好理解（Single-session Preference） | 27 / 30 | 90.0% |
+| 单会话用户信息（Single-session User） | 67 / 70 | 95.7% |
+| 时间推理（Temporal Reasoning） | 101 / 133 | 75.9% |
+| **总成绩** | **411 / 500** | **82.2%** |
+
+机器可读的汇总成绩位于 [`results/benchmark.json`](results/benchmark.json)。
+
+## 架构
+
+TMCRA 将不可变证据与两类派生记忆分离。Writer 把对话消息保存到 Source，同时将可复用、反映当前状态的事实提取并规范化到 Fast。Subject Attribution 用于阻止引用内容或第三方事实被错误归属；通过门控的记录会进一步组织为稳定的 Slow 语义图，而所有派生记忆始终能够追溯到 Source。
+
+查询时，Recall Planner 先解析问题并为各记忆层分配职责。系统随后从 Source、Fast、Slow 生成候选，经过图排序、重排、去重、时间处理和有界 Top-K 打包，再由 Evidence Compiler 生成带来源绑定的结构化证据包。TMCRA 线上记忆服务的边界止于可注入提示词的证据；回答生成和 Official Judge 属于 benchmark harness。
+
+```mermaid
+flowchart LR
+ subgraph BENCH["Benchmark 适配层"]
+ D["LongMemEval 数据集"] --> H["数据适配器 / 测试框架"]
+ Q["查询问题"]
+ G["标准答案"]
+ D --> G
+ end
+
+ subgraph CORE["TMCRA 记忆与召回核心"]
+ W["记忆写入器（Writer）"]
+ S["原始证据层（Source）
不可变记录"]
+ W --> F["快速记忆层（Fast）
当前原子记忆"]
+ W --> S
+ F --> A["主体归属判断
晋升门控"]
+ A --> L["慢速记忆层（Slow）
持久语义图"]
+ S --> I["检索索引
可搜索快照"]
+ F --> I
+ L --> I
+ P["召回规划器
解析查询与各层职责"]
+ P --> R["分层召回
Source + Fast + Slow"]
+ I --> R
+ R --> K["图排序 / 重排
Top-K 证据打包"]
+ K --> C["证据编译器
可追溯证据包"]
+ end
+
+ subgraph EVAL["仅用于 Benchmark 评测"]
+ C --> M["回答模型"]
+ M --> J["官方评测器（Judge）"]
+ J --> O["总成绩与分项成绩"]
+ end
+
+ H -->|"对话历史"| W
+ H -->|"仅问题"| Q
+ Q --> P
+ G -->|"只在此处作为评测参考"| J
+```
+
+### 核心组件职责
+
+- **Source** 保存不可变的对话原文，是最终证据来源。
+- **Fast** 保存与 Source 绑定的原子事实、关系、更新和当前状态。
+- **Subject Attribution** 防止引用内容或第三方信息被归属到错误主体。
+- **Slow** 把符合条件的跨会话信息组织为稳定的语义图；它补充 Source，但不能替代 Source。
+- **Recall Planner** 解析问题和各层职责，但不读取 gold answer。
+- **Layered Retrieval** 同时搜索 Source、Fast、Slow，并执行图排序、重排、去重、时间处理与固定 Top-K 打包。
+- **Evidence Compiler** 输出带 Source ID 绑定的结构化证据包。
+- **Answer Model 与 Judge** 只属于 benchmark，不属于 TMCRA 线上召回服务。
+
+Gold answer 与 Writer、Planner、Retrieval、Evidence Compiler、Answer Model 完全隔离，只允许最终 Judge 读取。
+
+## 查看与复现 benchmark
+
+- **查看公开成绩：** [`results/benchmark.json`](results/benchmark.json) 包含上表所示的总成绩与六类分项汇总。
+- **重跑完整链路：** 下面的命令会重新执行记忆构建、分层召回、证据整理、回答生成和评测。公开的 82.2% 是冻结的 v5 成绩；本次发布使用经过安全加固的 v6 回答协议，重新调用外部模型时成绩可能发生波动。
+
+### 已验证环境
+
+- Linux
+- Python 3.12.3
+- 24 GiB 或更高显存的 NVIDIA GPU
+- CUDA 12.8
+- PyTorch 2.11.0 + cu128
+- Transformers 5.10.2
+
+完整 S500 运行还需要足够的内存与磁盘空间，用于 Writer 日志、图、索引、checkpoint 和中间证据产物。
+
+### 1. 克隆并安装
+
+```bash
+git lfs install
+GIT_LFS_SKIP_SMUDGE=1 git clone https://github.com/reshuibuduo/TMCRA-Agent-Memory.git
+cd TMCRA-Agent-Memory/benchmarks/longmemeval
+git lfs pull --include="models/tmcra_v4_longmemeval_s500_20260715/*.pt"
+
+python3.12 -m venv .venv
+source .venv/bin/activate
+python -m pip install --upgrade pip
+python -m pip install -r requirements.txt
+python -m pip install -e .
+```
+
+`GIT_LFS_SKIP_SMUDGE=1` 会阻止 clone 阶段下载无关的历史 LFS 对象；后续命令只拉取本 benchmark 所需的三个 checkpoint。
+
+### 2. 校验 checkpoint 并下载外部资产
+
+三个 TMCRA checkpoint 已通过 Git LFS 保存在仓库的 `models/tmcra_v4_longmemeval_s500_20260715/` 目录。资产脚本会直接校验这些文件的字节数与 SHA-256，不会重复复制；随后下载官方清洗版 LongMemEval-S 和固定 revision 的 BGE 模型：
+
+```bash
+python scripts/fetch_assets.py --manifest configs/assets.lock.json --kind all
+```
+
+数据来自 [LongMemEval 官方仓库](https://github.com/xiaowu0162/LongMemEval)和[清洗版数据集](https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned)。BGE-M3 与 reranker 来自官方 [BAAI/bge-m3](https://huggingface.co/BAAI/bge-m3) 和 [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) 页面。
+
+文件类资产在启用前都会按照 `configs/assets.lock.json` 校验。如果当前检出只有 LFS 指针文件，脚本会停止并给出获取权重所需的完整 `git lfs pull` 命令。
+
+### 3. 配置模型服务
+
+```bash
+cp configs/writer.env.example configs/writer.env
+cp configs/answer.env.example configs/answer.env
+cp configs/judge.env.example configs/judge.env
+cp configs/run.env.example configs/run.env
+```
+
+编辑生成的四个文件：
+
+- `writer.env` 配置一个兼容 OpenAI API 格式的 DeepSeek 服务和逗号分隔的 key 池，供 Writer、主体归属判断和证据规划器共同使用。该服务需要接受 `deepseek-v4-flash` 与 `deepseek-v4-pro`，或者在网关中把这两个名称映射到兼容部署。
+- `answer.env` 的兼容 OpenAI API 服务必须以 `gpt-5.4` 这一精确名称提供回答模型。
+- `judge.env` 的服务必须提供 `gpt-4o-2024-08-06`。公开命令传入 `gpt-4o` 别名，内置评测器会将其固定解析为该版本。
+- `run.env` 配置本地路径、并发数与 PyTorch 设备。
+
+API key 只保留在本地，仓库的 `.gitignore` 已排除真实配置文件。
+
+Judge 默认使用 OpenAI Chat Completions 请求格式。只有当服务端实现了 Responses API 时，才在 `judge.env` 中设置 `OPENAI_WIRE_API=responses`。
+
+### 4. 运行离线 smoke test
+
+该测试只使用合成 fixture，不会调用外部模型服务：
+
+```bash
+bash scripts/reproduce_smoke.sh
+```
+
+### 5. 运行 LongMemEval-S500
+
+```bash
+bash scripts/reproduce_s500.sh
+```
+
+脚本会顺序执行并校验：
+
+```text
+准备数据与有序 QID
+ -> 构建 Writer 日志、Source/Fast/Slow 记忆和索引
+ -> 召回 Source + Fast + Slow 证据
+ -> 整理 operation-bound evidence packet
+ -> 使用 GPT-5.4 生成回答
+ -> 使用官方 GPT-4o Judge 评测
+ -> 导出总成绩与分项成绩
+```
+
+关键产物位于 `runs/longmemeval_s500/`：
+
+```text
+BUILD_COMPLETE
+retrieval_s500/evidence_windows.jsonl
+retrieval_s500/retrieval_debug.jsonl
+retrieval_s500_compiled/COMPILE_COMPLETE
+retrieval_s500_compiled/evidence_windows.jsonl
+answers_s500/answers.jsonl
+official_judge_gpt4o_20240806.jsonl
+scorecard.json
+```
+
+进入下一阶段前，脚本会检查完成标记或报告、准确的产物行数，以及与输入一致的有序 QID。Judge 产物还必须包含由 `gpt-4o-2024-08-06` 生成的布尔标签。兼容的召回、证据整理、回答和 Judge 产物会从断点继续；构建恢复采用故障关闭策略，必要时会要求使用构建脚本打印出的显式审查参数。回答协议不兼容时必须使用新的回答输出目录。
+
+## 仓库结构
+
+```text
+configs/ 配置示例与 hash 固定的资产清单
+fixtures/ 合成离线测试数据
+results/ 公开的汇总成绩
+scripts/ 资产下载、校验、smoke 与 S500 运行脚本
+src/ TMCRA V4 benchmark 核心、LongMemEval harness 与最小 adapter
+tests/ 离线 schema、gold 隔离和成绩汇总测试
+../../models/tmcra_v4_longmemeval_s500_20260715/
+ 本 benchmark 使用的 Git LFS checkpoint
+```
+
+线上 HTTP API、账户系统、计费、租户授权、部署文件、凭据、运行数据库、供应商原始日志和控制面均不在本仓库范围内。
+
+## 校验
+
+```bash
+python -m compileall -q src scripts tests
+python -m pytest
+```
+
+正式发布前还需要执行 secret scan、依赖漏洞扫描、SBOM，并人工审查暂存文件和 Release 资产。
+
+## 引用
+
+报告本结果时，请注明所使用的仓库版本，并引用 [LongMemEval 项目](https://github.com/xiaowu0162/LongMemEval)及其论文。
+
+## 许可证
+
+TMCRA benchmark 代码采用 [Apache License 2.0](../../LICENSE)。LongMemEval 与下载的 BGE 模型保留各自许可证，详见 [`THIRD_PARTY_NOTICES.md`](THIRD_PARTY_NOTICES.md)。
diff --git a/benchmarks/longmemeval/THIRD_PARTY_NOTICES.md b/benchmarks/longmemeval/THIRD_PARTY_NOTICES.md
new file mode 100644
index 0000000000000000000000000000000000000000..bd9f06953c6311cf01869efbc107ad1f808c8b87
--- /dev/null
+++ b/benchmarks/longmemeval/THIRD_PARTY_NOTICES.md
@@ -0,0 +1,52 @@
+# Third-Party Notices
+
+This benchmark runner downloads or adapts third-party material. Those components remain subject to their original licenses; the repository's Apache-2.0 license does not replace them.
+
+## LongMemEval
+
+- Project:
+- Cleaned dataset:
+- License source:
+- License: MIT
+
+The bundled evaluation wrapper follows the public LongMemEval evaluation prompt and rule. The dataset is downloaded separately by `scripts/fetch_assets.py`.
+
+The complete LongMemEval MIT license text follows:
+
+> MIT License
+>
+> Copyright (c) 2024 Di Wu
+>
+> Permission is hereby granted, free of charge, to any person obtaining a copy
+> of this software and associated documentation files (the "Software"), to deal
+> in the Software without restriction, including without limitation the rights
+> to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+> copies of the Software, and to permit persons to whom the Software is
+> furnished to do so, subject to the following conditions:
+>
+> The above copyright notice and this permission notice shall be included in all
+> copies or substantial portions of the Software.
+>
+> THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+> IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+> FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+> AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+> LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+> OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+> SOFTWARE.
+
+## BAAI/bge-m3
+
+- Model page:
+- Locked revision: `5617a9f61b028005a4858fdac845db406aefb181`
+- License declared by the model repository: MIT
+
+The model files are downloaded from Hugging Face and are not relicensed by TMCRA. Review the model repository and its license before redistribution.
+
+## BAAI/bge-reranker-v2-m3
+
+- Model page:
+- Locked revision: `953dc6f6f85a1b2dbfca4c34a2796e7dde08d41e`
+- License declared by the model repository: Apache License 2.0
+
+The model files are downloaded from Hugging Face and are not relicensed by TMCRA. Review the model repository and its license before redistribution.
diff --git a/benchmarks/longmemeval/configs/answer.env.example b/benchmarks/longmemeval/configs/answer.env.example
new file mode 100644
index 0000000000000000000000000000000000000000..48b9745b9d6e7d117d2d990d2fd2ae7cc4a09d91
--- /dev/null
+++ b/benchmarks/longmemeval/configs/answer.env.example
@@ -0,0 +1,6 @@
+# The answer runner requires this exact model identifier. The endpoint may be
+# an OpenAI-compatible gateway, but it must expose gpt-5.4 under this name.
+TMCRA_ANSWER_MODEL=gpt-5.4
+TMCRA_ANSWER_BASE_URL=https://your-answer-endpoint.example/v1
+TMCRA_ANSWER_API_KEY=replace_me
+TMCRA_ANSWER_MAX_TOKENS=1536
diff --git a/benchmarks/longmemeval/configs/assets.lock.json b/benchmarks/longmemeval/configs/assets.lock.json
new file mode 100644
index 0000000000000000000000000000000000000000..810f124aa1a77a64f4f22c42742823e9bbff535d
--- /dev/null
+++ b/benchmarks/longmemeval/configs/assets.lock.json
@@ -0,0 +1,51 @@
+{
+ "schema_version": "tmcra.benchmark-assets.1",
+ "files": [
+ {
+ "id": "longmemeval_s_cleaned",
+ "kind": "dataset",
+ "destination": "data/longmemeval_s_cleaned.json",
+ "url": "https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_s_cleaned.json",
+ "bytes": 277383467,
+ "sha256": "d6f21ea9d60a0d56f34a05b609c79c88a451d2ae03597821ea3d5a9678c3a442"
+ },
+ {
+ "id": "tmcra_node_scorer",
+ "kind": "checkpoint",
+ "storage": "git_lfs",
+ "destination": "../../models/tmcra_v4_longmemeval_s500_20260715/node_scorer.pt",
+ "bytes": 207356243,
+ "sha256": "d2318aafb07f9a15a1d95e6eac1b3e09afa07c570dd9d15d3f75544c9950f201"
+ },
+ {
+ "id": "tmcra_path_scorer",
+ "kind": "checkpoint",
+ "storage": "git_lfs",
+ "destination": "../../models/tmcra_v4_longmemeval_s500_20260715/path_scorer.pt",
+ "bytes": 207356243,
+ "sha256": "ecdc2bca51b7646a8a56c7db9cea4aeec581e64c3b61e6e06f7e9fa2a93b3bf5"
+ },
+ {
+ "id": "tmcra_v3_reranker",
+ "kind": "checkpoint",
+ "storage": "git_lfs",
+ "destination": "../../models/tmcra_v4_longmemeval_s500_20260715/tmcra_v3_reranker.pt",
+ "bytes": 4232213,
+ "sha256": "09a285b484ca857b24b53ad19d5998302f16d224ca9e4fd73a4eb5a52f022942"
+ }
+ ],
+ "huggingface_models": [
+ {
+ "id": "bge_m3",
+ "repo_id": "BAAI/bge-m3",
+ "revision": "5617a9f61b028005a4858fdac845db406aefb181",
+ "destination": "models/BAAI/bge-m3"
+ },
+ {
+ "id": "bge_reranker_v2_m3",
+ "repo_id": "BAAI/bge-reranker-v2-m3",
+ "revision": "953dc6f6f85a1b2dbfca4c34a2796e7dde08d41e",
+ "destination": "models/BAAI/bge-reranker-v2-m3"
+ }
+ ]
+}
diff --git a/benchmarks/longmemeval/configs/judge.env.example b/benchmarks/longmemeval/configs/judge.env.example
new file mode 100644
index 0000000000000000000000000000000000000000..cb30964362d9dd9af6abca83f30c1d8542f91abe
--- /dev/null
+++ b/benchmarks/longmemeval/configs/judge.env.example
@@ -0,0 +1,9 @@
+# OpenAI-compatible Judge endpoint. The public runner selects the gpt-4o alias;
+# the evaluator resolves it to the fixed gpt-4o-2024-08-06 model identifier.
+TMCRA_JUDGE_BASE_URL=https://your-judge-endpoint.example/v1
+TMCRA_JUDGE_API_KEY=replace_me
+TMCRA_OFFICIAL_JUDGE_WORKERS=8
+
+# The default wire format is POST /chat/completions. Uncomment this only when
+# the configured endpoint implements POST /responses instead.
+# OPENAI_WIRE_API=responses
diff --git a/benchmarks/longmemeval/configs/run.env.example b/benchmarks/longmemeval/configs/run.env.example
new file mode 100644
index 0000000000000000000000000000000000000000..de39319199e9e832136d8645d2720d15e5fc2c3b
--- /dev/null
+++ b/benchmarks/longmemeval/configs/run.env.example
@@ -0,0 +1,18 @@
+# reproduce_s500.sh changes into benchmarks/longmemeval before loading this
+# file, so $PWD resolves to that directory.
+DATA="$PWD/data/longmemeval_s_cleaned.json"
+QIDS="$PWD/data/longmemeval_s500.qids.txt"
+RUN="$PWD/runs/longmemeval_s500"
+# TMCRA scorer weights are tracked by Git LFS in the repository-level model
+# package. BGE snapshots are downloaded locally under this benchmark directory.
+MODELS="$PWD/../../models/tmcra_v4_longmemeval_s500_20260715"
+BGE_MODELS="$PWD/models/BAAI"
+
+# Tune concurrency to provider rate limits and local memory. Device values are
+# passed directly to PyTorch and therefore require a matching local runtime.
+WRITER_CONCURRENCY=20
+SLOW_CONCURRENCY=20
+RETRIEVAL_DEVICE=cuda
+GRAPH_DEVICE=cuda
+ANSWER_WORKERS=12
+JUDGE_WORKERS=8
diff --git a/benchmarks/longmemeval/configs/writer.env.example b/benchmarks/longmemeval/configs/writer.env.example
new file mode 100644
index 0000000000000000000000000000000000000000..69a764b1d60bd7a40391bfd67ea354dd6db8c9d9
--- /dev/null
+++ b/benchmarks/longmemeval/configs/writer.env.example
@@ -0,0 +1,12 @@
+# One OpenAI-compatible DeepSeek endpoint is used by the benchmark Writer,
+# subject-attribution pass, and evidence planner. The gateway must accept the
+# model names deepseek-v4-flash and deepseek-v4-pro, or alias them accordingly.
+TMCRA_WRITER_BASE_URL=https://your-writer-endpoint.example/v1
+TMCRA_WRITER_TIMEOUT_SECONDS=180
+TMCRA_WRITER_MAX_TOKENS=16384
+
+# Comma-separated, non-empty, unique API keys. For a single key, keep the
+# count at 1. If you add keys, update the count to the exact number of keys.
+TMCRA_DEEPSEEK_WRITER_BASE_URL=https://your-planner-endpoint.example/v1
+TMCRA_DEEPSEEK_WRITER_KEY_POOL=replace_me
+TMCRA_DEEPSEEK_WRITER_KEY_POOL_COUNT=1
diff --git a/benchmarks/longmemeval/fixtures/tiny_longmemeval.json b/benchmarks/longmemeval/fixtures/tiny_longmemeval.json
new file mode 100644
index 0000000000000000000000000000000000000000..7dbfae00cc70805c76d072a54088f0df9ef334bc
--- /dev/null
+++ b/benchmarks/longmemeval/fixtures/tiny_longmemeval.json
@@ -0,0 +1,22 @@
+[
+ {
+ "question_id": "tmcra_fixture_user",
+ "question_type": "single-session-user",
+ "question": "Which drink did the user choose?",
+ "answer": "Tea.",
+ "question_date": "2026-01-02",
+ "haystack_session_ids": ["fixture-session-1"],
+ "haystack_dates": ["2026-01-01"],
+ "haystack_sessions": [[{"role": "user", "content": "I chose tea."}]]
+ },
+ {
+ "question_id": "tmcra_fixture_update",
+ "question_type": "knowledge-update",
+ "question": "Where does the user live now?",
+ "answer": "Shanghai.",
+ "question_date": "2026-01-03",
+ "haystack_session_ids": ["fixture-session-2"],
+ "haystack_dates": ["2026-01-02"],
+ "haystack_sessions": [[{"role": "user", "content": "I moved from Suzhou to Shanghai."}]]
+ }
+]
diff --git a/benchmarks/longmemeval/pyproject.toml b/benchmarks/longmemeval/pyproject.toml
new file mode 100644
index 0000000000000000000000000000000000000000..d63f75c96263255e9895bfd6951034881c4889e7
--- /dev/null
+++ b/benchmarks/longmemeval/pyproject.toml
@@ -0,0 +1,30 @@
+[build-system]
+requires = ["setuptools>=80", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "tmcra-agent-memory-longmemeval"
+version = "0.1.0"
+description = "TMCRA LongMemEval benchmark reproduction pipeline"
+readme = "README.md"
+requires-python = ">=3.12,<3.13"
+license = "Apache-2.0"
+authors = [{ name = "Yu Haoxin" }]
+dependencies = [
+ "huggingface-hub==1.23.0",
+ "numpy==2.4.3",
+ "safetensors==0.8.0",
+ "tokenizers==0.22.2",
+ "transformers==5.10.2",
+]
+
+[project.optional-dependencies]
+test = ["pytest==9.0.2"]
+
+[tool.setuptools.packages.find]
+where = ["src"]
+namespaces = true
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = "-q"
diff --git a/benchmarks/longmemeval/requirements.txt b/benchmarks/longmemeval/requirements.txt
new file mode 100644
index 0000000000000000000000000000000000000000..9359c1ae1563f9ce66918ab960140b3d6a07092c
--- /dev/null
+++ b/benchmarks/longmemeval/requirements.txt
@@ -0,0 +1,8 @@
+--extra-index-url https://download.pytorch.org/whl/cu128
+torch==2.11.0+cu128
+huggingface-hub==1.23.0
+numpy==2.4.3
+safetensors==0.8.0
+tokenizers==0.22.2
+transformers==5.10.2
+pytest==9.0.2
diff --git a/benchmarks/longmemeval/results/benchmark.json b/benchmarks/longmemeval/results/benchmark.json
new file mode 100644
index 0000000000000000000000000000000000000000..5c9009d5be7f7d0058dccbbec48cabe34ea28eca
--- /dev/null
+++ b/benchmarks/longmemeval/results/benchmark.json
@@ -0,0 +1,16 @@
+{
+ "schema_version": "tmcra.benchmark-result.1",
+ "benchmark": "LongMemEval",
+ "metric": "accuracy",
+ "score_percent": 82.2,
+ "correct": 411,
+ "total": 500,
+ "by_question_type": {
+ "knowledge-update": {"correct": 71, "total": 78, "score_percent": 91.0},
+ "multi-session": {"correct": 90, "total": 133, "score_percent": 67.7},
+ "single-session-assistant": {"correct": 55, "total": 56, "score_percent": 98.2},
+ "single-session-preference": {"correct": 27, "total": 30, "score_percent": 90.0},
+ "single-session-user": {"correct": 67, "total": 70, "score_percent": 95.7},
+ "temporal-reasoning": {"correct": 101, "total": 133, "score_percent": 75.9}
+ }
+}
diff --git a/benchmarks/longmemeval/scripts/fetch_assets.py b/benchmarks/longmemeval/scripts/fetch_assets.py
new file mode 100644
index 0000000000000000000000000000000000000000..b6a2d969b89e9966baa00f2c183ecbd2c86daebf
--- /dev/null
+++ b/benchmarks/longmemeval/scripts/fetch_assets.py
@@ -0,0 +1,95 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import os
+import urllib.request
+from pathlib import Path
+
+LFS_INCLUDE = "models/tmcra_v4_longmemeval_s500_20260715/*.pt"
+
+
+def digest(path: Path) -> str:
+ hasher = hashlib.sha256()
+ with path.open("rb") as handle:
+ for chunk in iter(lambda: handle.read(1024 * 1024), b""):
+ hasher.update(chunk)
+ return hasher.hexdigest()
+
+
+def fetch_file(root: Path, row: dict[str, object]) -> None:
+ destination = (root / str(row["destination"])).resolve()
+ expected = str(row["sha256"])
+ expected_bytes = int(row["bytes"])
+ if destination.is_file() and destination.stat().st_size == expected_bytes and digest(destination) == expected:
+ print(f"verified {row['id']}: {destination}")
+ return
+ if row.get("storage") == "git_lfs":
+ detail = "file is missing"
+ if destination.is_file():
+ with destination.open("rb") as handle:
+ prefix = handle.read(128)
+ if prefix.startswith(b"version https://git-lfs.github.com/spec/v1"):
+ detail = "only a Git LFS pointer is present"
+ else:
+ detail = f"size or SHA-256 mismatch ({destination.stat().st_size} bytes)"
+ raise RuntimeError(
+ f"Git LFS checkpoint {row['id']} is unavailable: {detail}: {destination}\n"
+ f"run from the repository root: git lfs pull --include=\"{LFS_INCLUDE}\""
+ )
+ if not row.get("url"):
+ raise RuntimeError(f"asset {row['id']} has neither a valid local Git LFS file nor a download URL")
+ destination.parent.mkdir(parents=True, exist_ok=True)
+ temporary = destination.with_suffix(destination.suffix + ".part")
+ temporary.unlink(missing_ok=True)
+ print(f"downloading {row['id']} -> {destination}")
+ urllib.request.urlretrieve(str(row["url"]), temporary)
+ if temporary.stat().st_size != expected_bytes:
+ raise RuntimeError(f"size mismatch for {row['id']}")
+ actual = digest(temporary)
+ if actual != expected:
+ raise RuntimeError(f"SHA-256 mismatch for {row['id']}: {actual}")
+ os.replace(temporary, destination)
+
+
+def fetch_model(root: Path, row: dict[str, object]) -> None:
+ from huggingface_hub import snapshot_download
+
+ destination = root / str(row["destination"])
+ destination.parent.mkdir(parents=True, exist_ok=True)
+ print(f"downloading {row['repo_id']}@{row['revision']} -> {destination}")
+ snapshot_download(
+ repo_id=str(row["repo_id"]),
+ revision=str(row["revision"]),
+ local_dir=destination,
+ )
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--manifest", type=Path, default=Path("configs/assets.lock.json"))
+ parser.add_argument("--root", type=Path, default=Path.cwd())
+ parser.add_argument("--kind", choices=("all", "dataset", "checkpoint", "model"), default="all")
+ args = parser.parse_args()
+ payload = json.loads(args.manifest.read_text(encoding="utf-8"))
+ if args.kind in {"all", "dataset", "checkpoint"}:
+ selected = [
+ row
+ for row in payload["files"]
+ if args.kind == "all" or row["kind"] == args.kind
+ ]
+ # Validate repository-owned LFS files before starting any large network
+ # download, so an incomplete clone fails quickly and predictably.
+ selected.sort(key=lambda row: row.get("storage") != "git_lfs")
+ for row in selected:
+ fetch_file(args.root, row)
+ if args.kind in {"all", "model"}:
+ for row in payload["huggingface_models"]:
+ fetch_model(args.root, row)
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/longmemeval/scripts/prepare_longmemeval.py b/benchmarks/longmemeval/scripts/prepare_longmemeval.py
new file mode 100644
index 0000000000000000000000000000000000000000..b3ea2aa6cc9b6eec8b8e741249d48469b26a1830
--- /dev/null
+++ b/benchmarks/longmemeval/scripts/prepare_longmemeval.py
@@ -0,0 +1,70 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+from collections import Counter
+from pathlib import Path
+
+
+QUESTION_TYPES = {
+ "knowledge-update",
+ "multi-session",
+ "single-session-assistant",
+ "single-session-preference",
+ "single-session-user",
+ "temporal-reasoning",
+}
+
+
+def load_and_validate(path: Path) -> list[dict[str, object]]:
+ rows = json.loads(path.read_text(encoding="utf-8"))
+ if not isinstance(rows, list) or not rows:
+ raise ValueError("LongMemEval input must be a non-empty JSON array")
+ seen: set[str] = set()
+ for index, row in enumerate(rows):
+ if not isinstance(row, dict):
+ raise ValueError(f"row {index} is not an object")
+ qid = str(row.get("question_id") or "").strip()
+ if not qid or qid in seen:
+ raise ValueError(f"invalid or duplicate question_id at row {index}: {qid!r}")
+ seen.add(qid)
+ question_type = str(row.get("question_type") or "").strip()
+ if question_type not in QUESTION_TYPES:
+ raise ValueError(f"{qid}: unexpected question_type {question_type!r}")
+ for field in ("question", "answer", "haystack_sessions", "haystack_dates"):
+ if field not in row:
+ raise ValueError(f"{qid}: missing {field}")
+ return rows
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--data", required=True, type=Path)
+ parser.add_argument("--qid-output", required=True, type=Path)
+ parser.add_argument("--summary-output", type=Path)
+ parser.add_argument("--limit", type=int, default=0)
+ args = parser.parse_args()
+ rows = load_and_validate(args.data)
+ selected = rows[: args.limit] if args.limit else rows
+ qids = [str(row["question_id"]) for row in selected]
+ args.qid_output.parent.mkdir(parents=True, exist_ok=True)
+ args.qid_output.write_text("\n".join(qids) + "\n", encoding="utf-8")
+ summary = {
+ "schema_version": "tmcra.longmemeval-input.1",
+ "dataset_sha256": hashlib.sha256(args.data.read_bytes()).hexdigest(),
+ "source_count": len(rows),
+ "selected_count": len(selected),
+ "question_types": dict(sorted(Counter(str(row["question_type"]) for row in selected).items())),
+ "qid_output": str(args.qid_output),
+ }
+ if args.summary_output:
+ args.summary_output.parent.mkdir(parents=True, exist_ok=True)
+ args.summary_output.write_text(json.dumps(summary, indent=2, sort_keys=True) + "\n", encoding="utf-8")
+ print(json.dumps(summary, sort_keys=True))
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/longmemeval/scripts/reproduce_s500.sh b/benchmarks/longmemeval/scripts/reproduce_s500.sh
new file mode 100644
index 0000000000000000000000000000000000000000..a843468c88d7673bbbae07bf555fa0eb631cc647
--- /dev/null
+++ b/benchmarks/longmemeval/scripts/reproduce_s500.sh
@@ -0,0 +1,152 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+cd "$ROOT"
+
+for file in configs/writer.env configs/answer.env configs/judge.env configs/run.env; do
+ if [[ ! -f "$file" ]]; then
+ echo "missing $file; copy the matching .example file and fill in your values" >&2
+ exit 2
+ fi
+done
+
+set -a
+source configs/run.env
+set +a
+
+LEGACY="$ROOT/src/tmcra_benchmark/legacy"
+VENDOR="$ROOT/src/tmcra_benchmark/vendor/tmcra_integrated"
+HARNESS="$ROOT/src/tmcra_benchmark/harness/run_lme_s10_native_tmcra.py"
+JUDGE="$ROOT/src/tmcra_benchmark/harness/evaluate_qa_official_judge.py"
+export PYTHONPATH="$LEGACY:$VENDOR:$ROOT/src${PYTHONPATH:+:$PYTHONPATH}"
+
+: "${DATA:?DATA is required in configs/run.env}"
+: "${QIDS:?QIDS is required in configs/run.env}"
+: "${RUN:?RUN is required in configs/run.env}"
+: "${MODELS:?MODELS is required in configs/run.env}"
+: "${BGE_MODELS:?BGE_MODELS is required in configs/run.env}"
+
+python scripts/prepare_longmemeval.py \
+ --data "$DATA" \
+ --qid-output "$QIDS" \
+ --summary-output "$RUN.input_summary.json"
+
+if [[ ! -f "$RUN/BUILD_COMPLETE" ]]; then
+ build_resume_args=()
+ if [[ -d "$RUN" ]]; then
+ build_resume_args+=(--resume)
+ fi
+ python "$LEGACY/run_tmcra_v4_build.py" \
+ "${build_resume_args[@]}" \
+ --out-dir "$RUN" \
+ --qid-list "$QIDS" \
+ --data "$DATA" \
+ --repo "$VENDOR" \
+ --writer-env configs/writer.env \
+ --embedding-model "$BGE_MODELS/bge-m3" \
+ --writer-concurrency "${WRITER_CONCURRENCY:-20}" \
+ --slow-concurrency "${SLOW_CONCURRENCY:-20}" \
+ --index-batch-size 16 \
+ --device "${RETRIEVAL_DEVICE:-cuda}"
+fi
+python scripts/validate_stage.py build --run-dir "$RUN" --expected-qids "$QIDS"
+
+export TMCRA_NODE_MODEL_PATH="$MODELS/node_scorer.pt"
+export TMCRA_PATH_MODEL_PATH="$MODELS/path_scorer.pt"
+export TMCRA_BGE_M3_PATH="$BGE_MODELS/bge-m3"
+export TMCRA_BGE_RERANKER_PATH="$BGE_MODELS/bge-reranker-v2-m3"
+
+if [[ ! -f "$RUN/retrieval_s500/RETRIEVAL_COMPLETE" ]]; then
+ retrieval_resume_args=()
+ if [[ -d "$RUN/retrieval_s500" ]]; then
+ retrieval_resume_args+=(--resume)
+ fi
+ python "$LEGACY/run_tmcra_v4_retrieve.py" \
+ "${retrieval_resume_args[@]}" \
+ --run-dir "$RUN" \
+ --tag retrieval_s500 \
+ --qid-list "$QIDS" \
+ --writer-env configs/writer.env \
+ --repo "$VENDOR" \
+ --harness "$HARNESS" \
+ --node-model "$MODELS/node_scorer.pt" \
+ --path-model "$MODELS/path_scorer.pt" \
+ --checkpoint "$MODELS/tmcra_v3_reranker.pt" \
+ --embedding-model "$BGE_MODELS/bge-m3" \
+ --cross-model "$BGE_MODELS/bge-reranker-v2-m3" \
+ --device "${RETRIEVAL_DEVICE:-cuda}" \
+ --graph-device "${GRAPH_DEVICE:-cuda}" \
+ --dense-k 32 \
+ --slow-dense-k 24 \
+ --graph-k 24 \
+ --composition-mode layered \
+ --packing-budget-mode fixed \
+ --top-k 8 \
+ --cross-batch-size 24
+fi
+python scripts/validate_stage.py retrieve --run-dir "$RUN" --tag retrieval_s500 --expected-qids "$QIDS"
+
+if [[ ! -f "$RUN/retrieval_s500_compiled/COMPILE_COMPLETE" ]]; then
+ python "$LEGACY/run_tmcra_v4_compile_evidence.py" \
+ --evidence "$RUN/retrieval_s500/evidence_windows.jsonl" \
+ --retrieval-debug "$RUN/retrieval_s500/retrieval_debug.jsonl" \
+ --require-retrieval-debug \
+ --out-dir "$RUN/retrieval_s500_compiled" \
+ --qid-list "$QIDS" \
+ --writer-env configs/writer.env \
+ --planner-provider deepseek \
+ --planner-model deepseek-v4-pro \
+ --workers 4 \
+ --timeout 180
+fi
+python scripts/validate_stage.py compile --run-dir "$RUN/retrieval_s500_compiled" --expected-qids "$QIDS"
+
+set -a
+source configs/answer.env
+set +a
+# The answer runner resumes from its validated answers.jsonl. Running it again
+# also repairs an incomplete report without repeating completed provider calls.
+python "$LEGACY/run_tmcra_v4_gpt54_answers.py" \
+ --evidence "$RUN/retrieval_s500_compiled/evidence_windows.jsonl" \
+ --harness "$HARNESS" \
+ --out-dir "$RUN/answers_s500" \
+ --qid-list "$QIDS" \
+ --workers "${ANSWER_WORKERS:-12}" \
+ --attempts 3 \
+ --answer-window-limit 8 \
+ --lane production
+python scripts/validate_stage.py answer --run-dir "$RUN/answers_s500" --expected-qids "$QIDS"
+
+set -a
+source configs/judge.env
+set +a
+: "${TMCRA_JUDGE_BASE_URL:?TMCRA_JUDGE_BASE_URL is required}"
+: "${TMCRA_JUDGE_API_KEY:?TMCRA_JUDGE_API_KEY is required}"
+
+python "$LEGACY/run_tmcra_v4_parallel_official_judge.py" \
+ --judge "$JUDGE" \
+ --metric-model gpt-4o \
+ --hyp-file "$RUN/answers_s500/answers.jsonl" \
+ --ref-file "$DATA" \
+ --result-file "$RUN/official_judge_gpt4o_20240806.jsonl" \
+ --work-dir "$RUN/official_judge_work" \
+ --base-url "$TMCRA_JUDGE_BASE_URL" \
+ --api-key-env TMCRA_JUDGE_API_KEY \
+ --workers "${JUDGE_WORKERS:-8}" \
+ --timeout 180 \
+ --process-timeout 240 \
+ --max-retries 0 \
+ --resume
+
+python scripts/validate_stage.py judge \
+ --run-dir "$RUN" \
+ --expected-qids "$QIDS" \
+ --judge-results "$RUN/official_judge_gpt4o_20240806.jsonl"
+
+python scripts/summarize_official_judge.py \
+ --judge-results "$RUN/official_judge_gpt4o_20240806.jsonl" \
+ --dataset "$DATA" \
+ --output "$RUN/scorecard.json"
+
+echo "TMCRA LongMemEval run complete: $RUN/scorecard.json"
diff --git a/benchmarks/longmemeval/scripts/reproduce_smoke.sh b/benchmarks/longmemeval/scripts/reproduce_smoke.sh
new file mode 100644
index 0000000000000000000000000000000000000000..ef9dffc4a6b05b139d829f3759f3b8ad2d526907
--- /dev/null
+++ b/benchmarks/longmemeval/scripts/reproduce_smoke.sh
@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+cd "$ROOT"
+
+DATA="${DATA:-$ROOT/fixtures/tiny_longmemeval.json}"
+OUT="${OUT:-$ROOT/runs/offline_fixture_check}"
+mkdir -p "$OUT"
+
+python scripts/prepare_longmemeval.py \
+ --data "$DATA" \
+ --qid-output "$OUT/qids.txt" \
+ --summary-output "$OUT/input_summary.json"
+python -m pytest
+
+echo "Offline fixture and unit tests passed."
+echo "This smoke test does not call Writer, Answer, or Judge providers."
diff --git a/benchmarks/longmemeval/scripts/summarize_official_judge.py b/benchmarks/longmemeval/scripts/summarize_official_judge.py
new file mode 100644
index 0000000000000000000000000000000000000000..0c32bb4351474c8c7b847fabf567a6139e4f14fe
--- /dev/null
+++ b/benchmarks/longmemeval/scripts/summarize_official_judge.py
@@ -0,0 +1,82 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import json
+from collections import defaultdict
+from pathlib import Path
+from typing import Any
+
+
+def read_jsonl(path: Path) -> list[dict[str, Any]]:
+ return [json.loads(line) for line in path.read_text(encoding="utf-8").splitlines() if line.strip()]
+
+
+def label(row: dict[str, Any]) -> bool:
+ value = row.get("autoeval_label")
+ if isinstance(value, dict):
+ value = value.get("label")
+ if isinstance(value, bool):
+ return value
+ if isinstance(value, str):
+ normalized = value.strip().lower()
+ if normalized in {"true", "1", "yes"}:
+ return True
+ if normalized in {"false", "0", "no"}:
+ return False
+ raise ValueError(f"invalid autoeval label: {value!r}")
+
+
+def summarize(dataset: list[dict[str, Any]], judged: list[dict[str, Any]]) -> dict[str, Any]:
+ type_by_qid = {str(row["question_id"]): str(row["question_type"]) for row in dataset}
+ seen: set[str] = set()
+ buckets: dict[str, list[bool]] = defaultdict(list)
+ all_labels: list[bool] = []
+ for row in judged:
+ qid = str(row.get("question_id") or "")
+ if qid not in type_by_qid or qid in seen:
+ raise ValueError(f"unknown or duplicate question_id: {qid!r}")
+ seen.add(qid)
+ result = label(row)
+ buckets[type_by_qid[qid]].append(result)
+ all_labels.append(result)
+ if len(seen) != len(dataset):
+ missing = sorted(set(type_by_qid) - seen)
+ raise ValueError(f"judge results are incomplete: {len(seen)}/{len(dataset)}; first missing={missing[:5]}")
+ by_type = {}
+ for name, values in sorted(buckets.items()):
+ correct = sum(values)
+ by_type[name] = {
+ "correct": correct,
+ "total": len(values),
+ "accuracy": round(correct / len(values), 6),
+ "accuracy_percent": round(correct * 100 / len(values), 1),
+ }
+ correct = sum(all_labels)
+ return {
+ "schema_version": "tmcra.longmemeval-scorecard.1",
+ "benchmark": "LongMemEval",
+ "correct": correct,
+ "total": len(all_labels),
+ "accuracy": round(correct / len(all_labels), 6),
+ "accuracy_percent": round(correct * 100 / len(all_labels), 1),
+ "by_question_type": by_type,
+ }
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--judge-results", required=True, type=Path)
+ parser.add_argument("--dataset", required=True, type=Path)
+ parser.add_argument("--output", required=True, type=Path)
+ args = parser.parse_args()
+ dataset = json.loads(args.dataset.read_text(encoding="utf-8"))
+ scorecard = summarize(dataset, read_jsonl(args.judge_results))
+ args.output.parent.mkdir(parents=True, exist_ok=True)
+ args.output.write_text(json.dumps(scorecard, indent=2, sort_keys=True) + "\n", encoding="utf-8")
+ print(json.dumps(scorecard, sort_keys=True))
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/longmemeval/scripts/validate_stage.py b/benchmarks/longmemeval/scripts/validate_stage.py
new file mode 100644
index 0000000000000000000000000000000000000000..3c1378d20071e16d6b07763fa9c7e6c011f7eb20
--- /dev/null
+++ b/benchmarks/longmemeval/scripts/validate_stage.py
@@ -0,0 +1,237 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import json
+from collections.abc import Mapping
+from pathlib import Path
+from typing import Any
+
+
+PRODUCTION_ANSWER_PROTOCOL = "evidence_operation_bound_v6"
+
+
+def fail(message: str) -> None:
+ raise SystemExit(message)
+
+
+def require(path: Path) -> Path:
+ if not path.is_file():
+ fail(f"missing required artifact: {path}")
+ return path
+
+
+def read_json(path: Path) -> dict[str, Any]:
+ require(path)
+ try:
+ value = json.loads(path.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError) as exc:
+ fail(f"invalid JSON artifact {path}: {exc}")
+ if not isinstance(value, dict):
+ fail(f"JSON artifact must be an object: {path}")
+ return value
+
+
+def read_jsonl(path: Path) -> list[dict[str, Any]]:
+ require(path)
+ rows: list[dict[str, Any]] = []
+ for line_number, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1):
+ if not line.strip():
+ continue
+ try:
+ value = json.loads(line)
+ except json.JSONDecodeError as exc:
+ fail(f"invalid JSONL row {path}:{line_number}: {exc}")
+ if not isinstance(value, dict):
+ fail(f"JSONL row must be an object: {path}:{line_number}")
+ rows.append(value)
+ return rows
+
+
+def qids(path: Path) -> list[str]:
+ require(path)
+ values = [line.strip() for line in path.read_text(encoding="utf-8").splitlines() if line.strip()]
+ if not values:
+ fail(f"QID list is empty: {path}")
+ if len(values) != len(set(values)):
+ fail(f"QID list contains duplicates: {path}")
+ return values
+
+
+def row_qids(path: Path) -> list[str]:
+ values = [str(row.get("question_id") or row.get("qid") or "").strip() for row in read_jsonl(path)]
+ if any(not value for value in values):
+ fail(f"artifact contains an empty QID: {path}")
+ if len(values) != len(set(values)):
+ fail(f"artifact contains duplicate QIDs: {path}")
+ return values
+
+
+def assert_qid_order(path: Path, expected: list[str]) -> None:
+ actual = row_qids(path)
+ if actual != expected:
+ fail(f"QID mismatch in {path}: got {len(actual)} ordered rows, expected {len(expected)}")
+
+
+def assert_complete_report(
+ path: Path,
+ *,
+ expected_count: int,
+ count_field: str,
+) -> dict[str, Any]:
+ report = read_json(path)
+ if report.get("status") != "complete":
+ fail(f"report is not complete: {path}")
+ try:
+ actual_count = int(report.get(count_field, -1))
+ except (TypeError, ValueError):
+ fail(f"report has an invalid {count_field}: {path}")
+ if actual_count != expected_count:
+ fail(f"report {count_field} mismatch in {path}: {actual_count}/{expected_count}")
+ return report
+
+
+def validate_build(run_dir: Path, expected: list[str]) -> None:
+ require(run_dir / "BUILD_COMPLETE")
+ require(run_dir / "qids.txt")
+ if qids(run_dir / "qids.txt") != expected:
+ fail("build qids.txt does not match the requested QID order")
+
+ manifest = read_json(run_dir / "input_manifest.json")
+ if manifest.get("status") != "prepared":
+ fail("build input manifest is not in prepared state")
+ if list(manifest.get("qids") or []) != expected:
+ fail("build input manifest QIDs do not match the requested QID order")
+ if int(manifest.get("row_count", -1)) != len(expected):
+ fail("build input manifest row_count is inconsistent")
+ workers = manifest.get("workers")
+ if not isinstance(workers, list):
+ fail("build input manifest workers must be a list")
+ worker_qids = [str(row.get("question_id") or "").strip() for row in workers if isinstance(row, Mapping)]
+ if len(worker_qids) != len(workers) or worker_qids != expected:
+ fail("build worker QIDs do not match the requested QID order")
+
+ assert_qid_order(run_dir / "scope_manifest.jsonl", expected)
+ assert_qid_order(run_dir / "query_manifest.jsonl", expected)
+ assert_complete_report(
+ run_dir / "build_report.json",
+ expected_count=len(expected),
+ count_field="row_count",
+ )
+ assert_complete_report(
+ run_dir / "index_report.json",
+ expected_count=len(expected),
+ count_field="row_count",
+ )
+
+
+def validate_retrieve(run_dir: Path, tag: str, expected: list[str]) -> None:
+ directory = run_dir / tag
+ require(directory / "RETRIEVAL_COMPLETE")
+ assert_complete_report(
+ directory / "report.json",
+ expected_count=len(expected),
+ count_field="query_count",
+ )
+ run_report = assert_complete_report(
+ directory / "retrieval_run_report.json",
+ expected_count=len(expected),
+ count_field="query_count",
+ )
+ if str(run_report.get("tag") or "") != tag:
+ fail(f"retrieval tag mismatch in {directory / 'retrieval_run_report.json'}")
+ assert_qid_order(directory / "evidence_windows.jsonl", expected)
+ assert_qid_order(directory / "retrieval_debug.jsonl", expected)
+
+
+def validate_compile(run_dir: Path, expected: list[str]) -> None:
+ require(run_dir / "COMPILE_COMPLETE")
+ assert_complete_report(
+ run_dir / "report.json",
+ expected_count=len(expected),
+ count_field="row_count",
+ )
+ assert_qid_order(run_dir / "evidence_windows.jsonl", expected)
+ assert_qid_order(run_dir / "retrieval_debug.jsonl", expected)
+
+
+def validate_answer(run_dir: Path, expected: list[str]) -> None:
+ report = assert_complete_report(
+ run_dir / "report.json",
+ expected_count=len(expected),
+ count_field="requested",
+ )
+ if int(report.get("completed", -1)) != len(expected):
+ fail(f"answer report completed count mismatch: {report.get('completed')}/{len(expected)}")
+ if int(report.get("failure_count", -1)) != 0:
+ fail(f"answer report contains failures: {report.get('failure_count')}")
+ if report.get("answer_model") != "gpt-5.4":
+ fail(f"answer report used an unexpected model: {report.get('answer_model')!r}")
+ if report.get("answer_protocol") != PRODUCTION_ANSWER_PROTOCOL:
+ fail(f"answer report used an unexpected protocol: {report.get('answer_protocol')!r}")
+ if report.get("lane") != "production" or report.get("promotion_eligible") is not True:
+ fail("answer report is not a promotion-eligible production run")
+ if report.get("compiled_evidence_packet_required") is not True:
+ fail("answer report did not require compiled evidence packets")
+ answers_path = run_dir / "answers.jsonl"
+ assert_qid_order(answers_path, expected)
+ for row in read_jsonl(answers_path):
+ if row.get("answer_model") != "gpt-5.4":
+ fail(f"answer row used an unexpected model: {row.get('question_id')}")
+ if row.get("answer_protocol") != PRODUCTION_ANSWER_PROTOCOL:
+ fail(f"answer row used an unexpected protocol: {row.get('question_id')}")
+
+
+def validate_judge(results: Path, expected: list[str]) -> None:
+ assert_qid_order(results, expected)
+ rows = read_jsonl(results)
+ for row in rows:
+ label = row.get("autoeval_label")
+ if not isinstance(label, Mapping) or type(label.get("label")) is not bool:
+ fail(f"judge row has no valid boolean label: {row.get('question_id')}")
+ if label.get("model") != "gpt-4o-2024-08-06":
+ fail(f"judge row used an unexpected model: {label.get('model')!r}")
+
+ report_path = results.with_name(f"{results.name}.isolated_report.json")
+ report = read_json(report_path)
+ if report.get("status") != "complete":
+ fail(f"judge report is not complete: {report_path}")
+ if int(report.get("row_count", -1)) != len(expected):
+ fail(f"judge row_count mismatch in {report_path}")
+ if int(report.get("committed_count", -1)) != len(expected):
+ fail(f"judge committed_count mismatch in {report_path}")
+ if int(report.get("failure_count", -1)) != 0:
+ fail(f"judge report contains failures: {report_path}")
+ if report.get("metric_model") not in {"gpt-4o", "gpt-4o-2024-08-06"}:
+ fail(f"judge report used an unexpected model alias: {report.get('metric_model')!r}")
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser()
+ parser.add_argument("stage", choices=("build", "retrieve", "compile", "answer", "judge"))
+ parser.add_argument("--run-dir", required=True, type=Path)
+ parser.add_argument("--expected-qids", required=True, type=Path)
+ parser.add_argument("--tag", default="retrieval_s500")
+ parser.add_argument("--judge-results", type=Path)
+ args = parser.parse_args()
+ expected = qids(args.expected_qids)
+
+ if args.stage == "build":
+ validate_build(args.run_dir, expected)
+ elif args.stage == "retrieve":
+ validate_retrieve(args.run_dir, args.tag, expected)
+ elif args.stage == "compile":
+ validate_compile(args.run_dir, expected)
+ elif args.stage == "answer":
+ validate_answer(args.run_dir, expected)
+ else:
+ if args.judge_results is None:
+ fail("--judge-results is required for the judge stage")
+ validate_judge(args.judge_results, expected)
+
+ print(f"{args.stage} validation: OK ({len(expected)} QIDs)")
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/__init__.py b/benchmarks/longmemeval/src/tmcra_benchmark/__init__.py
new file mode 100644
index 0000000000000000000000000000000000000000..60f09a2f7975b1d259b5a9b66c4f28fc8ff0619c
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/__init__.py
@@ -0,0 +1,3 @@
+"""TMCRA LongMemEval benchmark reproduction package."""
+
+__version__ = "0.1.0"
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/harness/__init__.py b/benchmarks/longmemeval/src/tmcra_benchmark/harness/__init__.py
new file mode 100644
index 0000000000000000000000000000000000000000..b8fe9efdb2418d5809c5ee5f5b1f30974ecb0278
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/harness/__init__.py
@@ -0,0 +1 @@
+"""LongMemEval adapter and official judge integration."""
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/harness/evaluate_qa_official_judge.py b/benchmarks/longmemeval/src/tmcra_benchmark/harness/evaluate_qa_official_judge.py
new file mode 100644
index 0000000000000000000000000000000000000000..d2c6153501a7f154096f3095799a9487b3a5f199
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/harness/evaluate_qa_official_judge.py
@@ -0,0 +1,295 @@
+#!/usr/bin/env python3
+"""LongMemEval official-prompt QA judge.
+
+This is an implementation-compatible wrapper around the public LongMemEval
+`src/evaluation/evaluate_qa.py` logic. The scoring prompt and yes/no rule are
+kept equivalent, while the runner adds resume support and avoids extra Python
+package dependencies by using urllib.
+
+For official reporting, use:
+ --metric-model gpt-4o
+with OPENAI_API_KEY set. The resolved model is gpt-4o-2024-08-06, matching the
+official evaluator's model alias.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+import time
+import urllib.error
+import urllib.request
+from pathlib import Path
+from statistics import mean
+from typing import Any
+
+
+MODEL_ZOO = {
+ "gpt5.4": ("gpt-5.4", "openai"),
+ "gpt-5.4": ("gpt-5.4", "openai"),
+ "gpt-4o-mini": ("gpt-4o-mini-2024-07-18", "openai"),
+ "gpt-4o": ("gpt-4o-2024-08-06", "openai"),
+ "gpt-4o-2024-08-06": ("gpt-4o-2024-08-06", "openai"),
+ "llama-3.1-70b-instruct": ("meta-llama/Meta-Llama-3.1-70B-Instruct", "local"),
+}
+
+
+def get_anscheck_prompt(task: str, question: str, answer: str, response: str, abstention: bool = False) -> str:
+ if not abstention:
+ if task in ["single-session-user", "single-session-assistant", "multi-session"]:
+ template = (
+ "I will give you a question, a correct answer, and a response from a model. "
+ "Please answer yes if the response contains the correct answer. Otherwise, answer no. "
+ "If the response is equivalent to the correct answer or contains all the intermediate steps to get the correct answer, you should also answer yes. "
+ "If the response only contains a subset of the information required by the answer, answer no. \n\n"
+ "Question: {}\n\nCorrect Answer: {}\n\nModel Response: {}\n\n"
+ "Is the model response correct? Answer yes or no only."
+ )
+ return template.format(question, answer, response)
+ if task == "temporal-reasoning":
+ template = (
+ "I will give you a question, a correct answer, and a response from a model. "
+ "Please answer yes if the response contains the correct answer. Otherwise, answer no. "
+ "If the response is equivalent to the correct answer or contains all the intermediate steps to get the correct answer, you should also answer yes. "
+ "If the response only contains a subset of the information required by the answer, answer no. "
+ "In addition, do not penalize off-by-one errors for the number of days. "
+ "If the question asks for the number of days/weeks/months, etc., and the model makes off-by-one errors (e.g., predicting 19 days when the answer is 18), the model's response is still correct. \n\n"
+ "Question: {}\n\nCorrect Answer: {}\n\nModel Response: {}\n\n"
+ "Is the model response correct? Answer yes or no only."
+ )
+ return template.format(question, answer, response)
+ if task == "knowledge-update":
+ template = (
+ "I will give you a question, a correct answer, and a response from a model. "
+ "Please answer yes if the response contains the correct answer. Otherwise, answer no. "
+ "If the response contains some previous information along with an updated answer, the response should be considered as correct as long as the updated answer is the required answer.\n\n"
+ "Question: {}\n\nCorrect Answer: {}\n\nModel Response: {}\n\n"
+ "Is the model response correct? Answer yes or no only."
+ )
+ return template.format(question, answer, response)
+ if task == "single-session-preference":
+ template = (
+ "I will give you a question, a rubric for desired personalized response, and a response from a model. "
+ "Please answer yes if the response satisfies the desired response. Otherwise, answer no. "
+ "The model does not need to reflect all the points in the rubric. "
+ "The response is correct as long as it recalls and utilizes the user's personal information correctly.\n\n"
+ "Question: {}\n\nRubric: {}\n\nModel Response: {}\n\n"
+ "Is the model response correct? Answer yes or no only."
+ )
+ return template.format(question, answer, response)
+ raise NotImplementedError(f"Unsupported question_type: {task}")
+
+ template = (
+ "I will give you an unanswerable question, an explanation, and a response from a model. "
+ "Please answer yes if the model correctly identifies the question as unanswerable. "
+ "The model could say that the information is incomplete, or some other information is given but the asked information is not.\n\n"
+ "Question: {}\n\nExplanation: {}\n\nModel Response: {}\n\n"
+ "Does the model correctly identify the question as unanswerable? Answer yes or no only."
+ )
+ return template.format(question, answer, response)
+
+
+def load_json_or_jsonl(path: Path) -> list[dict[str, Any]]:
+ text = path.read_text(encoding="utf-8")
+ try:
+ data = json.loads(text)
+ if isinstance(data, list):
+ return data
+ if isinstance(data, dict):
+ return [data]
+ raise ValueError(f"{path} JSON root is not a list")
+ except json.JSONDecodeError:
+ return [json.loads(line) for line in text.splitlines() if line.strip()]
+
+
+def call_chat_completion(
+ *,
+ base_url: str,
+ api_key: str,
+ model: str,
+ prompt: str,
+ timeout: int,
+ max_retries: int,
+) -> str:
+ wire_api = os.environ.get("OPENAI_WIRE_API", "").strip().lower()
+ if wire_api == "responses":
+ url = base_url.rstrip("/") + "/responses"
+ payload = {
+ "model": model,
+ "input": [{"role": "user", "content": prompt}],
+ "temperature": 0,
+ "max_output_tokens": 10,
+ }
+ else:
+ url = base_url.rstrip("/") + "/chat/completions"
+ payload = {
+ "model": model,
+ "messages": [{"role": "user", "content": prompt}],
+ "n": 1,
+ "temperature": 0,
+ "max_tokens": 10,
+ }
+ body = json.dumps(payload).encode("utf-8")
+ headers = {
+ "Content-Type": "application/json",
+ "Authorization": f"Bearer {api_key}",
+ }
+ last_error: Exception | None = None
+ for attempt in range(max_retries + 1):
+ req = urllib.request.Request(url, data=body, headers=headers, method="POST")
+ try:
+ with urllib.request.urlopen(req, timeout=timeout) as resp:
+ data = json.loads(resp.read().decode("utf-8"))
+ if wire_api == "responses":
+ output_text = str(data.get("output_text", "")).strip()
+ if output_text:
+ return output_text
+ chunks: list[str] = []
+ for item in data.get("output", []) or []:
+ for content in item.get("content", []) or []:
+ text = content.get("text") or content.get("content") or ""
+ if text:
+ chunks.append(str(text))
+ return "\n".join(chunks).strip()
+ return data["choices"][0]["message"]["content"].strip()
+ except (urllib.error.HTTPError, urllib.error.URLError, TimeoutError) as exc:
+ last_error = exc
+ if attempt >= max_retries:
+ break
+ time.sleep(min(60, 2**attempt))
+ raise RuntimeError(f"chat completion failed after {max_retries + 1} attempts: {last_error}")
+
+
+def read_existing_labels(result_file: Path) -> dict[str, dict[str, Any]]:
+ done: dict[str, dict[str, Any]] = {}
+ if not result_file.exists():
+ return done
+ with result_file.open("r", encoding="utf-8") as f:
+ for line in f:
+ if not line.strip():
+ continue
+ item = json.loads(line)
+ qid = str(item.get("question_id", ""))
+ if qid:
+ done[qid] = item
+ return done
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--metric-model", default="gpt-4o", choices=sorted(MODEL_ZOO))
+ parser.add_argument("--hyp-file", required=True, type=Path)
+ parser.add_argument("--ref-file", required=True, type=Path)
+ parser.add_argument("--result-file", type=Path)
+ parser.add_argument("--base-url", default=None)
+ parser.add_argument("--api-key-env", default="OPENAI_API_KEY")
+ parser.add_argument("--limit", type=int, default=0)
+ parser.add_argument("--timeout", type=int, default=120)
+ parser.add_argument("--max-retries", type=int, default=6)
+ parser.add_argument("--resume", action="store_true")
+ parser.add_argument("--quiet", action="store_true")
+ args = parser.parse_args()
+
+ model, source = MODEL_ZOO[args.metric_model]
+ if source == "openai":
+ base_url = args.base_url or "https://api.openai.com/v1"
+ else:
+ base_url = args.base_url or "http://localhost:8001/v1"
+
+ api_key = os.getenv(args.api_key_env)
+ if not api_key:
+ print(f"Missing API key env: {args.api_key_env}", file=sys.stderr)
+ return 2
+
+ hypotheses = load_json_or_jsonl(args.hyp_file)
+ references = load_json_or_jsonl(args.ref_file)
+ qid2qdata = {str(entry["question_id"]): entry for entry in references}
+ qid2qtype = {str(entry["question_id"]): entry["question_type"] for entry in references}
+
+ if args.limit and args.limit > 0:
+ hypotheses = hypotheses[: args.limit]
+
+ result_file = args.result_file or Path(str(args.hyp_file) + f".eval-results-{args.metric_model}")
+ result_file.parent.mkdir(parents=True, exist_ok=True)
+ existing = read_existing_labels(result_file) if args.resume else {}
+
+ logs: list[dict[str, Any]] = list(existing.values())
+ qtype2acc: dict[str, list[int]] = {}
+ for item in logs:
+ qid = str(item.get("question_id"))
+ if qid in qid2qtype and item.get("autoeval_label"):
+ qtype2acc.setdefault(qid2qtype[qid], []).append(1 if item["autoeval_label"].get("label") else 0)
+
+ mode = "a" if args.resume and result_file.exists() else "w"
+ with result_file.open(mode, encoding="utf-8") as out_f:
+ for idx, entry in enumerate(hypotheses, 1):
+ qid = str(entry.get("question_id", ""))
+ if not qid or qid not in qid2qdata:
+ print(f"Warning: skipping unknown question_id={qid}", file=sys.stderr)
+ continue
+ if qid in existing:
+ continue
+ qtype = qid2qtype[qid]
+ qdata = qid2qdata[qid]
+ prompt = get_anscheck_prompt(
+ qtype,
+ str(qdata["question"]),
+ str(qdata["answer"]),
+ str(entry.get("hypothesis", "")),
+ abstention="_abs" in qid,
+ )
+ eval_response = call_chat_completion(
+ base_url=base_url,
+ api_key=api_key,
+ model=model,
+ prompt=prompt,
+ timeout=args.timeout,
+ max_retries=args.max_retries,
+ )
+ label = "yes" in eval_response.lower()
+ output = dict(entry)
+ output["autoeval_label"] = {"model": model, "label": label, "raw_response": eval_response}
+ print(json.dumps(output, ensure_ascii=False), file=out_f, flush=True)
+ logs.append(output)
+ qtype2acc.setdefault(qtype, []).append(1 if label else 0)
+ if not args.quiet:
+ print(
+ json.dumps(
+ {
+ "index": idx,
+ "question_id": qid,
+ "label": label,
+ "raw_response": eval_response,
+ "question": qdata["question"],
+ "answer": qdata["answer"],
+ "hypothesis": entry.get("hypothesis", ""),
+ },
+ ensure_ascii=False,
+ ),
+ flush=True,
+ )
+
+ labels = [1 if x.get("autoeval_label", {}).get("label") else 0 for x in logs if x.get("autoeval_label")]
+ summary = {
+ "metric_model_alias": args.metric_model,
+ "metric_model": model,
+ "base_url": base_url,
+ "hyp_file": str(args.hyp_file),
+ "ref_file": str(args.ref_file),
+ "result_file": str(result_file),
+ "evaluated_count": len(labels),
+ "accuracy": round(mean(labels), 4) if labels else None,
+ "by_question_type": {
+ k: {"accuracy": round(mean(v), 4), "count": len(v)} for k, v in sorted(qtype2acc.items()) if v
+ },
+ }
+ summary_file = Path(str(result_file) + ".summary.json")
+ summary_file.write_text(json.dumps(summary, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
+ print(json.dumps(summary, ensure_ascii=False, indent=2))
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/harness/run_lme_s10_native_tmcra.py b/benchmarks/longmemeval/src/tmcra_benchmark/harness/run_lme_s10_native_tmcra.py
new file mode 100644
index 0000000000000000000000000000000000000000..a25cb22ab79e2c947cd503b35a029e871d5bd2ce
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/harness/run_lme_s10_native_tmcra.py
@@ -0,0 +1,3797 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import os
+import re
+import sys
+import time
+import urllib.error
+import urllib.request
+from datetime import datetime, timezone
+from pathlib import Path
+from types import SimpleNamespace
+from typing import Any, Iterable, Mapping
+
+
+PACKAGE_ROOT = Path(__file__).resolve().parent.parent
+PROJECT_ROOT = PACKAGE_ROOT.parents[1]
+DEFAULT_SERVICE_ROOT = PROJECT_ROOT
+DEFAULT_REPO = PACKAGE_ROOT / "vendor" / "tmcra_integrated"
+DEFAULT_DATA = PROJECT_ROOT / "data" / "longmemeval_s_cleaned.json"
+DEFAULT_OUT_ROOT = PROJECT_ROOT / "runs"
+
+
+def log(event: str, **payload: Any) -> None:
+ stamp = datetime.now().isoformat(timespec="seconds")
+ details = " ".join(f"{key}={json.dumps(value, ensure_ascii=False)}" for key, value in sorted(payload.items()))
+ print(f"[lme_native] {stamp} {event}" + (f" {details}" if details else ""), flush=True)
+
+
+def read_env_file(path: Path) -> dict[str, str]:
+ env: dict[str, str] = {}
+ if not path.exists():
+ return env
+ for raw in path.read_text(encoding="utf-8", errors="replace").splitlines():
+ line = raw.strip()
+ if not line or line.startswith("#") or "=" not in line:
+ continue
+ if line.startswith("export "):
+ line = line[len("export ") :].strip()
+ key, value = line.split("=", 1)
+ env[key.strip()] = value.strip().strip('"').strip("'")
+ return env
+
+
+def apply_env_defaults(service_root: Path) -> None:
+ for env_path in [service_root / "env" / "tmcra-api.env", service_root / "env" / "tmcra-gemma.env"]:
+ for key, value in read_env_file(env_path).items():
+ if value == "":
+ continue
+ os.environ.setdefault(key, value)
+
+
+def iter_json_array(path: Path, *, limit: int = 0) -> Iterable[dict[str, Any]]:
+ import codecs
+
+ decoder = codecs.getincrementaldecoder("utf-8")(errors="replace")
+ yielded = 0
+ array_started = False
+ object_started = False
+ object_depth = 0
+ in_string = False
+ escape_next = False
+ buffer: list[str] = []
+ with path.open("rb") as handle:
+ while True:
+ chunk = handle.read(1 << 20)
+ if not chunk:
+ text = decoder.decode(b"", final=True)
+ else:
+ text = decoder.decode(chunk)
+ if not text and not chunk:
+ break
+ for char in text:
+ if not array_started:
+ if char.isspace():
+ continue
+ if char != "[":
+ raise RuntimeError(f"expected top-level JSON array in {path}")
+ array_started = True
+ continue
+ if not object_started:
+ if char.isspace() or char == ",":
+ continue
+ if char == "]":
+ return
+ if char != "{":
+ raise RuntimeError(f"expected object item in {path}, got {char!r}")
+ object_started = True
+ object_depth = 1
+ in_string = False
+ escape_next = False
+ buffer = ["{"]
+ continue
+ buffer.append(char)
+ if in_string:
+ if escape_next:
+ escape_next = False
+ elif char == "\\":
+ escape_next = True
+ elif char == '"':
+ in_string = False
+ continue
+ if char == '"':
+ in_string = True
+ continue
+ if char == "{":
+ object_depth += 1
+ elif char == "}":
+ object_depth -= 1
+ if object_depth == 0:
+ row = json.loads("".join(buffer))
+ if isinstance(row, dict):
+ yield row
+ yielded += 1
+ if limit > 0 and yielded >= limit:
+ return
+ object_started = False
+ buffer = []
+ if not chunk:
+ break
+ if object_started:
+ raise RuntimeError(f"unterminated JSON object in {path}")
+
+
+def http_json(
+ method: str,
+ url: str,
+ payload: dict[str, Any] | None = None,
+ *,
+ timeout: int = 300,
+ api_key: str = "",
+) -> dict[str, Any]:
+ data = json.dumps(payload, ensure_ascii=False).encode("utf-8") if payload is not None else None
+ headers = {"Content-Type": "application/json; charset=utf-8"}
+ if api_key:
+ headers["Authorization"] = f"Bearer {api_key}"
+ max_attempts = max(1, int(os.getenv("TMCRA_HTTP_JSON_MAX_ATTEMPTS", "8") or 8))
+ retry_codes = {429, 500, 502, 503, 504}
+ last_error: Exception | None = None
+ for attempt in range(1, max_attempts + 1):
+ req = urllib.request.Request(url, data=data, method=method, headers=headers)
+ try:
+ with urllib.request.urlopen(req, timeout=timeout) as response:
+ body = response.read().decode("utf-8", "replace")
+ return json.loads(body) if body else {}
+ except urllib.error.HTTPError as exc:
+ body = exc.read().decode("utf-8", "replace")
+ last_error = RuntimeError(f"HTTP {exc.code} {url}: {body[:500]}")
+ if exc.code not in retry_codes or attempt >= max_attempts:
+ raise last_error from exc
+ except (urllib.error.URLError, TimeoutError, json.JSONDecodeError) as exc:
+ last_error = exc
+ if attempt >= max_attempts:
+ raise RuntimeError(f"HTTP request failed {url}: {exc}") from exc
+ time.sleep(min(2 ** (attempt - 1), 8))
+ raise RuntimeError(f"HTTP request failed {url}: {last_error}")
+
+
+def chat_completion(
+ base_url: str,
+ model: str,
+ messages: list[dict[str, str]],
+ *,
+ max_tokens: int = 180,
+ temperature: float = 0.0,
+ api_key: str = "",
+) -> str:
+ wire_api = clean_text(os.getenv("TMCRA_ANSWER_WIRE_API", os.getenv("OPENAI_WIRE_API", ""))).lower()
+ reasoning_effort = clean_text(os.getenv("TMCRA_CHAT_COMPLETION_REASONING_EFFORT", ""))
+ reasoning_models = {
+ item.strip().lower()
+ for item in os.getenv("TMCRA_CHAT_COMPLETION_REASONING_EFFORT_MODELS", "").split(",")
+ if item.strip()
+ }
+ model_l = clean_text(model).lower()
+ reasoning_model_allowed = (
+ model_l in reasoning_models
+ if reasoning_models
+ else (
+ "vectorengine.ai" in base_url.lower()
+ or "api.openai.com" in base_url.lower()
+ or model_l.startswith(("gpt-", "gpt5", "gpt4"))
+ )
+ )
+ reasoning_effort_allowed = bool(reasoning_effort and reasoning_model_allowed)
+ cache_mode = os.getenv("TMCRA_LLM_CACHE_MODE", "").strip().lower()
+ cache_dir_raw = os.getenv("TMCRA_LLM_CACHE_DIR", "").strip()
+ cache_path: Path | None = None
+ if cache_mode in {"record", "replay"} and cache_dir_raw:
+ cache_key_payload = {
+ "schema_version": 1,
+ "wire_api": wire_api or "chat_completions",
+ "base_url": base_url.rstrip("/"),
+ "model": model,
+ "messages": messages,
+ "max_tokens": max_tokens,
+ "temperature": temperature,
+ "reasoning_effort": reasoning_effort if reasoning_effort_allowed else "",
+ }
+ cache_key = hashlib.sha256(
+ json.dumps(cache_key_payload, ensure_ascii=False, sort_keys=True).encode("utf-8")
+ ).hexdigest()
+ cache_path = Path(cache_dir_raw) / f"{cache_key}.json"
+ if cache_mode == "replay":
+ if cache_path is None:
+ raise RuntimeError("TMCRA_LLM_CACHE_MODE=replay requires TMCRA_LLM_CACHE_DIR")
+ if not cache_path.exists():
+ raise RuntimeError(f"missing LLM cache entry: {cache_path}")
+ cached = json.loads(cache_path.read_text(encoding="utf-8"))
+ return str(cached.get("text", ""))
+
+ if wire_api == "responses":
+ body = http_json(
+ "POST",
+ base_url.rstrip("/") + "/responses",
+ {
+ "model": model,
+ "input": messages,
+ "max_output_tokens": max_tokens,
+ "temperature": temperature,
+ },
+ timeout=360,
+ api_key=api_key,
+ )
+ output_text = clean_text(body.get("output_text", ""))
+ if output_text:
+ if cache_mode == "record" and cache_path is not None:
+ cache_path.parent.mkdir(parents=True, exist_ok=True)
+ tmp_path = cache_path.with_suffix(cache_path.suffix + ".tmp")
+ tmp_path.write_text(
+ json.dumps({"schema_version": 1, "text": output_text}, ensure_ascii=False, sort_keys=True),
+ encoding="utf-8",
+ )
+ tmp_path.replace(cache_path)
+ return output_text
+ chunks: list[str] = []
+ for item in body.get("output", []) or []:
+ for content in item.get("content", []) or []:
+ text = content.get("text") or content.get("content") or ""
+ if text:
+ chunks.append(str(text))
+ response_text = "\n".join(chunks).strip()
+ if cache_mode == "record" and cache_path is not None:
+ cache_path.parent.mkdir(parents=True, exist_ok=True)
+ tmp_path = cache_path.with_suffix(cache_path.suffix + ".tmp")
+ tmp_path.write_text(
+ json.dumps({"schema_version": 1, "text": response_text}, ensure_ascii=False, sort_keys=True),
+ encoding="utf-8",
+ )
+ tmp_path.replace(cache_path)
+ return response_text
+ body = http_json(
+ "POST",
+ base_url.rstrip("/") + "/chat/completions",
+ {
+ "model": model,
+ "messages": messages,
+ "max_tokens": max_tokens,
+ "temperature": temperature,
+ "stream": False,
+ **({"reasoning_effort": reasoning_effort} if reasoning_effort_allowed else {}),
+ },
+ timeout=360,
+ api_key=api_key,
+ )
+ response_text = (((body.get("choices") or [{}])[0].get("message") or {}).get("content") or "").strip()
+ if cache_mode == "record" and cache_path is not None:
+ cache_path.parent.mkdir(parents=True, exist_ok=True)
+ tmp_path = cache_path.with_suffix(cache_path.suffix + ".tmp")
+ tmp_path.write_text(
+ json.dumps({"schema_version": 1, "text": response_text}, ensure_ascii=False, sort_keys=True),
+ encoding="utf-8",
+ )
+ tmp_path.replace(cache_path)
+ return response_text
+
+
+def answer_llm_config() -> tuple[str, str, str]:
+ answer_base_url = clean_text(os.getenv("TMCRA_ANSWER_BASE_URL", ""))
+ answer_model = clean_text(os.getenv("TMCRA_ANSWER_MODEL", ""))
+ answer_api_key = clean_text(os.getenv("TMCRA_ANSWER_API_KEY", ""))
+ if answer_base_url or answer_model or answer_api_key:
+ return (
+ answer_base_url or os.getenv("GEMMA_BASE_URL", "http://127.0.0.1:18002/v1"),
+ answer_model or os.getenv("GEMMA_MODEL", os.getenv("TMCRA_GEMMA_MODEL_NAME", "gemma-4-e4b-it")),
+ answer_api_key or clean_text(os.getenv("OPENAI_API_KEY", "")),
+ )
+ return (
+ os.getenv("GEMMA_BASE_URL", "http://127.0.0.1:18002/v1"),
+ os.getenv("GEMMA_MODEL", os.getenv("TMCRA_GEMMA_MODEL_NAME", "gemma-4-e4b-it")),
+ clean_text(os.getenv("GEMMA_API_KEY", "")),
+ )
+
+
+def clean_text(value: Any) -> str:
+ return re.sub(r"\s+", " ", str(value or "").strip())
+
+
+def truncate(value: Any, n: int = 500) -> str:
+ text = clean_text(value)
+ return text[:n]
+
+
+def session_text_chunks(
+ *,
+ session_id: str,
+ date: str,
+ turns: list[Mapping[str, Any]],
+ max_chars: int,
+ max_chunks: int,
+) -> list[str]:
+ chunks: list[str] = []
+ current: list[str] = [f"LongMemEval session_id={session_id} date={date}"]
+ current_len = len(current[0])
+ for index, turn in enumerate(turns, start=1):
+ role = clean_text(turn.get("role", "unknown"))
+ content = clean_text(turn.get("content", ""))
+ if not content:
+ continue
+ part = f"[{session_id} turn={index} role={role}] {content}"
+ if current_len + len(part) + 2 > max_chars and len(current) > 1:
+ chunks.append("\n".join(current))
+ if max_chunks > 0 and len(chunks) >= max_chunks:
+ return chunks
+ current = [f"LongMemEval session_id={session_id} date={date} continued=true"]
+ current_len = len(current[0])
+ current.append(part)
+ current_len += len(part) + 1
+ if len(current) > 1 and (max_chunks <= 0 or len(chunks) < max_chunks):
+ chunks.append("\n".join(current))
+ return chunks
+
+
+def select_session_indices(row: Mapping[str, Any], *, max_distractors: int) -> list[int]:
+ session_ids = [clean_text(item) for item in list(row.get("haystack_session_ids") or [])]
+ answer_ids = {clean_text(item) for item in list(row.get("answer_session_ids") or [])}
+ answer_indices = [index for index, sid in enumerate(session_ids) if sid in answer_ids]
+ distractors: list[int] = []
+ latest_candidates = list(range(max(0, len(session_ids) - 4), len(session_ids)))
+ first_candidates = list(range(min(3, len(session_ids))))
+ mid = len(session_ids) // 2
+ middle_candidates = [max(0, mid - 1), mid, min(len(session_ids) - 1, mid + 1)] if session_ids else []
+ for index in [*first_candidates, *middle_candidates, *latest_candidates]:
+ if index not in answer_indices and index not in distractors and 0 <= index < len(session_ids):
+ distractors.append(index)
+ if len(distractors) >= max_distractors:
+ break
+ return sorted(set(answer_indices + distractors))
+
+
+def select_official_full_history_indices(row: Mapping[str, Any]) -> list[int]:
+ """Official LongMemEval-style input: use every haystack session in file order."""
+ return list(range(len(list(row.get("haystack_sessions") or []))))
+
+
+def build_writer() -> Any:
+ from experiments.replacement.adapters.base import LLMProfile
+ from experiments.replacement.multi_layer_tmcra_writer import TMCRASuspectAnchoredTransformerMemoryWriter
+ from experiments.replacement.semantic_memory_writer import DeterministicMemoryWriteGate, OpenAICompatSemanticMemoryWriter
+
+ profile = LLMProfile(
+ name="longmemeval_native_writer",
+ model=os.getenv("TMCRA_WRITER_MODEL", "gemma-4-e4b-it"),
+ base_url=os.getenv("TMCRA_WRITER_BASE_URL", "http://127.0.0.1:18002/v1"),
+ api_key=os.getenv("TMCRA_WRITER_API_KEY", ""),
+ timeout_seconds=float(os.getenv("TMCRA_WRITER_TIMEOUT_SECONDS", "180")),
+ temperature=float(os.getenv("TMCRA_WRITER_TEMPERATURE", "0.0")),
+ max_tokens=int(os.getenv("TMCRA_WRITER_MAX_TOKENS", "512")),
+ )
+ base_writer = OpenAICompatSemanticMemoryWriter(
+ profile,
+ gate=DeterministicMemoryWriteGate(min_grounding_score=float(os.getenv("TMCRA_WRITER_MIN_GROUNDING_SCORE", "0.35"))),
+ max_proposals=int(os.getenv("TMCRA_WRITER_MAX_PROPOSALS", "2")),
+ )
+ return TMCRASuspectAnchoredTransformerMemoryWriter(
+ base_writer,
+ max_cells=int(os.getenv("TMCRA_WRITER_MAX_PROPOSALS", "2")),
+ value_char_budget=int(os.getenv("TMCRA_WRITER_VALUE_CHAR_BUDGET", "120")),
+ source_span_char_budget=int(os.getenv("TMCRA_WRITER_SOURCE_SPAN_CHAR_BUDGET", "180")),
+ transformer_layers=int(os.getenv("TMCRA_WRITER_TRANSFORMER_LAYERS", "2")),
+ state_attention_k=int(os.getenv("TMCRA_WRITER_STATE_ATTENTION_K", "16")),
+ suspect_threshold=float(os.getenv("TMCRA_WRITER_SUSPECT_THRESHOLD", "0.48")),
+ suspect_promote_count=int(os.getenv("TMCRA_WRITER_SUSPECT_PROMOTE_COUNT", "2")),
+ )
+
+
+def build_adapter(scope_id: str, storage_path: Path) -> Any:
+ from experiments.replacement.adapters.memory_adapters import GraphSessionMemoryAdapter
+
+ return GraphSessionMemoryAdapter(
+ auto_extract=False,
+ storage_backend="sqlite",
+ storage_path=str(storage_path),
+ scope_id=scope_id,
+ retrieval_mode=os.getenv("TMCRA_RETRIEVAL_MODE", "hybrid_node_scored"),
+ node_model_path=os.getenv("TMCRA_NODE_MODEL_PATH", ""),
+ path_model_path=os.getenv("TMCRA_PATH_MODEL_PATH", ""),
+ node_model_device=os.getenv("TMCRA_NODE_MODEL_DEVICE", "cpu"),
+ candidate_event_k=int(os.getenv("TMCRA_CANDIDATE_EVENT_K", "24")),
+ support_path_k=int(os.getenv("TMCRA_SUPPORT_PATH_K", "3")),
+ path_tunnel_rescue_k=int(os.getenv("TMCRA_PATH_TUNNEL_RESCUE_K", "2")),
+ path_tunnel_rescue_score_floor=float(os.getenv("TMCRA_PATH_TUNNEL_RESCUE_SCORE_FLOOR", "0.0")),
+ path_tunnel_rescue_min_age=int(os.getenv("TMCRA_PATH_TUNNEL_RESCUE_MIN_AGE", "0")),
+ path_tunnel_rescue_min_score_margin=float(os.getenv("TMCRA_PATH_TUNNEL_RESCUE_MIN_SCORE_MARGIN", "0.0")),
+ )
+
+
+def disable_topic_bucket_runtime() -> None:
+ import experiments.replacement.adapters.memory_adapters as memory_adapters
+
+ def no_topic_bucket(*args: Any, **kwargs: Any) -> dict[str, Any]:
+ return {}
+
+ def no_apply_topic_bucket(records: list[Any], topic_bucket: Mapping[str, Any]) -> None:
+ return None
+
+ def no_topic_edges(*args: Any, **kwargs: Any) -> dict[str, Any]:
+ return {
+ "topic_bridge_disabled": True,
+ "dialogue_tunnel_disabled": True,
+ "disabled_reason": "longmemeval_native_no_topic_bucket",
+ }
+
+ def no_topic_rerank(graph: Any, query: str, hits: list[Any], *, top_k: int) -> dict[str, Any]:
+ limit = max(1, int(top_k or 1))
+ return {
+ "hits": list(hits)[:limit],
+ "metadata": {
+ "topic_bucket_rerank_enabled": False,
+ "topic_bucket_disabled": True,
+ "topic_bucket_disable_reason": "longmemeval_native_no_topic_bucket",
+ "topic_bucket_candidate_count": len(list(hits)),
+ "topic_bucket_final_count": len(list(hits)[:limit]),
+ },
+ }
+
+ memory_adapters._assign_topic_bucket_for_text = no_topic_bucket
+ memory_adapters._apply_topic_bucket_to_records = no_apply_topic_bucket
+ memory_adapters._last_topic_turn = no_topic_bucket
+ memory_adapters._add_topic_bridge_edges = no_topic_edges
+ memory_adapters._add_dialogue_tunnel_edges = no_topic_edges
+ memory_adapters._topic_bucket_rerank_hits = no_topic_rerank
+
+
+def writer_ingest(adapter: Any, writer: Any, text: str, *, qid: str, chunk_id: str) -> dict[str, Any]:
+ from experiments.replacement.semantic_memory_writer import build_modelized_facet_unit_records
+
+ cache_mode = os.getenv("TMCRA_WRITER_CACHE_MODE", "").strip().lower()
+ cache_dir_raw = os.getenv("TMCRA_WRITER_CACHE_DIR", "").strip()
+ cache_path: Path | None = None
+ if cache_mode in {"record", "replay"} and cache_dir_raw:
+ safe_qid = re.sub(r"[^A-Za-z0-9_.-]+", "_", qid)[:120] or "sample"
+ safe_chunk_id = re.sub(r"[^A-Za-z0-9_.-]+", "_", chunk_id)[:80] or "chunk"
+ cache_path = Path(cache_dir_raw) / f"{safe_qid}__{safe_chunk_id}.json"
+ if cache_mode == "replay":
+ if cache_path is None:
+ raise RuntimeError("TMCRA_WRITER_CACHE_MODE=replay requires TMCRA_WRITER_CACHE_DIR")
+ if not cache_path.exists():
+ raise RuntimeError(f"missing writer cache entry: {cache_path}")
+ cached = json.loads(cache_path.read_text(encoding="utf-8"))
+ cached_text = str(cached.get("text", ""))
+ if cached_text != text:
+ raise RuntimeError(f"writer cache text mismatch for {qid}/{chunk_id}")
+ timestamp = str(cached.get("timestamp", ""))
+ payload = dict(cached.get("payload") or {})
+ if not timestamp or not payload:
+ raise RuntimeError(f"invalid writer cache entry: {cache_path}")
+ adapter.ingest_turn(
+ f"[{timestamp}] user: {text}",
+ assistant_text="",
+ answer_payload=payload,
+ extraction_result={},
+ )
+ result = dict(cached.get("result") or {})
+ result["seconds"] = 0.0
+ result["writer_cache_mode"] = "replay"
+ result["writer_cache_path"] = str(cache_path)
+ return result
+
+ timestamp = datetime.now(timezone.utc).isoformat(timespec="seconds")
+ sidecar_hints = {
+ "metadata": {
+ "source": "longmemeval_native_flow",
+ "question_id": qid,
+ "chunk_id": chunk_id,
+ }
+ }
+ t0 = time.perf_counter()
+ proposals, writer_metadata = writer.propose_public_turn(
+ current_turn=text,
+ previous_turn="",
+ next_turn="",
+ speaker="user",
+ session_timestamp=timestamp,
+ sidecar_hints=sidecar_hints,
+ auxiliary_evidence_texts=[],
+ input_mode=os.getenv("TMCRA_WRITER_INPUT_MODE", "delta"),
+ )
+ graph = getattr(adapter, "graph", None)
+ turn_index = int(getattr(graph, "turn_index", 0) or 0) + 1
+ gate_result = writer.gate.build_payload(
+ proposals=proposals,
+ text=f"[{timestamp}] user: {text}",
+ raw_text=text,
+ speaker="user",
+ session_key=qid,
+ turn_index=turn_index,
+ timestamp=timestamp,
+ dia_id=f"longmemeval:{qid}:{chunk_id}",
+ sidecar_hints=sidecar_hints,
+ writer_metadata=writer_metadata,
+ )
+ payload = dict(gate_result.payload or {})
+ replacement_records = list(payload.get("replacement_memory_records", []) or [])
+ base_writer = getattr(writer, "base_writer", writer)
+ unit_records, unit_metadata = build_modelized_facet_unit_records(
+ base_writer,
+ current_turn_text=text,
+ parent_records=replacement_records,
+ speaker="user",
+ timestamp=timestamp,
+ turn_index=turn_index,
+ max_units=int(os.getenv("TMCRA_MODELIZED_UNIT_WRITER_MAX_UNITS", "14") or 14),
+ )
+ if unit_records:
+ replacement_records.extend(unit_records)
+ payload["replacement_memory_records"] = replacement_records
+ payload["modelized_facet_unit_writer"] = dict(unit_metadata or {})
+ ingest_text = f"[{timestamp}] user: {text}"
+ adapter.ingest_turn(
+ ingest_text,
+ assistant_text="",
+ answer_payload=payload,
+ extraction_result={},
+ )
+ result = {
+ "seconds": round(time.perf_counter() - t0, 3),
+ "proposal_count": len(proposals or []),
+ "accepted_count": int(getattr(gate_result, "accepted_count", 0) or 0),
+ "suspected_count": int(getattr(gate_result, "suspected_count", 0) or 0),
+ "record_count": len(payload.get("replacement_memory_records", []) or []),
+ "unit_record_count": len(unit_records),
+ "unit_writer_enabled": bool(unit_metadata.get("enabled", False)),
+ "unit_writer_metadata": dict(unit_metadata or {}),
+ }
+ if cache_mode == "record":
+ if cache_path is None:
+ raise RuntimeError("TMCRA_WRITER_CACHE_MODE=record requires TMCRA_WRITER_CACHE_DIR")
+ cache_path.parent.mkdir(parents=True, exist_ok=True)
+ tmp_path = cache_path.with_suffix(cache_path.suffix + ".tmp")
+ tmp_path.write_text(
+ json.dumps(
+ {
+ "schema_version": 1,
+ "qid": qid,
+ "chunk_id": chunk_id,
+ "timestamp": timestamp,
+ "text": text,
+ "ingest_text": ingest_text,
+ "payload": payload,
+ "result": result,
+ },
+ ensure_ascii=False,
+ sort_keys=True,
+ ),
+ encoding="utf-8",
+ )
+ tmp_path.replace(cache_path)
+ result["writer_cache_mode"] = "record"
+ result["writer_cache_path"] = str(cache_path)
+ return result
+
+
+def retrieval_debug(retrieval: Any) -> dict[str, Any]:
+ metadata = dict(getattr(retrieval, "metadata", {}) or {})
+ hits = list(getattr(retrieval, "hits", []) or [])
+ return {
+ "hit_count": len(hits),
+ "retrieval_seconds": round(float(getattr(retrieval, "retrieval_seconds", 0.0) or 0.0), 4),
+ "retrieval_mode": metadata.get("retrieval_mode"),
+ "hybrid_enabled": metadata.get("hybrid_enabled"),
+ "decision_score_source": metadata.get("decision_score_source"),
+ "selected_event_count": len(metadata.get("selected_event_ids", []) or []),
+ "selected_path_count": len(metadata.get("selected_path_ids", []) or []),
+ "path_tunnel_enabled": metadata.get("path_tunnel_enabled"),
+ "path_tunnel_rescue_candidate_count": metadata.get("path_tunnel_rescue_candidate_count"),
+ "path_tunnel_rescue_path_count": len(metadata.get("path_tunnel_rescue_path_ids", []) or []),
+ "profile_focused_pack_enabled": metadata.get("profile_focused_pack_enabled"),
+ "profile_focused_pack_reason": metadata.get("profile_focused_pack_reason"),
+ "profile_focused_pack_hit_count": metadata.get("profile_focused_pack_hit_count"),
+ "profile_focused_pack_event_ids": list(metadata.get("profile_focused_pack_event_ids", []) or []),
+ "profile_focused_pack_memory_ids": list(metadata.get("profile_focused_pack_memory_ids", []) or []),
+ "profile_first_hybrid_enabled": metadata.get("profile_first_hybrid_enabled"),
+ "profile_first_event_ids": list(metadata.get("profile_first_event_ids", []) or []),
+ "profile_first_memory_ids": list(metadata.get("profile_first_memory_ids", []) or []),
+ "facet_query_pack_enabled": metadata.get("facet_query_pack_enabled"),
+ "facet_query_pack_inserted_hit_count": metadata.get("facet_query_pack_inserted_hit_count"),
+ "unit_coverage_pack_enabled": metadata.get("unit_coverage_pack_enabled"),
+ "unit_coverage_candidate_count": metadata.get("unit_coverage_candidate_count"),
+ "unit_coverage_selected_unit_count": metadata.get("unit_coverage_selected_unit_count"),
+ "unit_coverage_inserted_hit_count": metadata.get("unit_coverage_inserted_hit_count"),
+ "multi_unit_chain_slot_enabled": metadata.get("multi_unit_chain_slot_enabled"),
+ "multi_unit_chain_slot_formed": metadata.get("multi_unit_chain_slot_formed"),
+ "multi_unit_chain_slot_reason": metadata.get("multi_unit_chain_slot_reason"),
+ "multi_unit_chain_candidate_count": metadata.get("multi_unit_chain_candidate_count"),
+ "multi_unit_chain_selected_unit_count": metadata.get("multi_unit_chain_selected_unit_count"),
+ "multi_unit_chain_parent_count": metadata.get("multi_unit_chain_parent_count"),
+ "multi_unit_chain_inserted_hit_count": metadata.get("multi_unit_chain_inserted_hit_count"),
+ "multi_unit_chain_memory_ids": list(metadata.get("multi_unit_chain_memory_ids", []) or []),
+ "multi_unit_chain_focus_tokens": list(metadata.get("multi_unit_chain_focus_tokens", []) or []),
+ "top_values": [truncate(getattr(hit, "value", ""), 220) for hit in hits[:5]],
+ }
+
+
+STOPWORDS = {
+ "a",
+ "an",
+ "and",
+ "are",
+ "as",
+ "at",
+ "be",
+ "by",
+ "can",
+ "did",
+ "do",
+ "does",
+ "for",
+ "from",
+ "had",
+ "has",
+ "have",
+ "how",
+ "i",
+ "in",
+ "is",
+ "it",
+ "me",
+ "my",
+ "of",
+ "on",
+ "or",
+ "that",
+ "the",
+ "to",
+ "was",
+ "were",
+ "what",
+ "when",
+ "where",
+ "which",
+ "who",
+ "with",
+}
+
+_ANSWER_WINDOW_PLANNER_CACHE: dict[str, Any] = {}
+_EVIDENCE_UNIT_SELECTOR_CACHE: dict[str, Any] = {}
+_UNIFIED_OPERATION_PLANNER_CACHE: dict[str, Any] = {}
+
+
+def _vector_dot(left: list[float], right: list[float]) -> float:
+ if not left or not right:
+ return 0.0
+ total = 0.0
+ for a, b in zip(left, right):
+ try:
+ total += float(a) * float(b)
+ except (TypeError, ValueError):
+ continue
+ if total != total:
+ return 0.0
+ return max(0.0, min(1.0, float(total)))
+
+
+def apply_answer_window_semantic_features(question: str, evidence_windows: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ mode = os.getenv("TMCRA_ANSWER_WINDOW_PLANNER_SEMANTIC_MODE", "").strip().lower()
+ if mode in {"", "off", "false", "0", "disabled", "none"} or not evidence_windows:
+ return evidence_windows
+ if mode in {"auto", "on", "true", "1"}:
+ mode = os.getenv("TMCRA_EMBEDDER_INDEX_RECALL_MODE", "").strip().lower() or "bge_m3"
+ try:
+ from experiments.replacement.adapters import memory_adapters
+ except Exception:
+ return evidence_windows
+ texts = [clean_text(question), *[clean_text(item.get("text", "")) for item in evidence_windows]]
+ try:
+ vectors, metadata = memory_adapters._embedder_dense_vectors_for_texts(texts, mode=mode)
+ except Exception:
+ return evidence_windows
+ if not vectors or not vectors[0]:
+ return evidence_windows
+ query_vector = vectors[0]
+ enriched: list[dict[str, Any]] = []
+ dense_enabled = bool(metadata.get("write_embedder_dense_enabled"))
+ for item, vector in zip(evidence_windows, vectors[1:]):
+ score = _vector_dot(query_vector, list(vector or []))
+ next_item = dict(item)
+ next_item["answer_window_semantic_similarity"] = round(score, 6)
+ next_item["answer_window_semantic_mode"] = mode
+ next_item["answer_window_semantic_enabled"] = bool(dense_enabled and score > 0.0)
+ enriched.append(next_item)
+ return enriched
+
+
+def evidence_unit_planner_config() -> tuple[str, str, str]:
+ # TMCRA_LAYER_TAG: answer-side LLM planner config; not graph-memory model core.
+ answer_base_url, answer_model, answer_api_key = answer_llm_config()
+ return (
+ clean_text(os.getenv("TMCRA_EVIDENCE_UNIT_PLANNER_BASE_URL", "")) or answer_base_url,
+ clean_text(os.getenv("TMCRA_EVIDENCE_UNIT_PLANNER_MODEL", "")) or answer_model,
+ clean_text(os.getenv("TMCRA_EVIDENCE_UNIT_PLANNER_API_KEY", "")) or answer_api_key,
+ )
+
+
+def llm_channel_planner_config() -> tuple[str, str, str]:
+ # TMCRA_LAYER_TAG: optional answer-side LLM channel planner config; not graph-memory model core.
+ answer_base_url, answer_model, answer_api_key = answer_llm_config()
+ return (
+ clean_text(os.getenv("TMCRA_LLM_CHANNEL_PLANNER_BASE_URL", "")) or answer_base_url,
+ clean_text(os.getenv("TMCRA_LLM_CHANNEL_PLANNER_MODEL", "")) or answer_model,
+ clean_text(os.getenv("TMCRA_LLM_CHANNEL_PLANNER_API_KEY", "")) or answer_api_key,
+ )
+
+
+def token_stem(token: str) -> str:
+ token = token.lower()
+ for suffix in ("ing", "ed", "es", "s"):
+ if len(token) > len(suffix) + 3 and token.endswith(suffix):
+ return token[: -len(suffix)]
+ return token
+
+
+def content_tokens(text: str) -> list[str]:
+ return [
+ token_stem(item)
+ for item in re.findall(r"[a-zA-Z0-9$]+", text.lower())
+ if item and item not in STOPWORDS
+ ]
+
+
+def quoted_query_phrases(question: str) -> list[str]:
+ phrases: list[str] = []
+ seen: set[str] = set()
+ for match in re.finditer(r"""['"]([^'"]{2,120})['"]""", question):
+ phrase = clean_text(match.group(1))
+ key = phrase.lower()
+ if phrase and key not in seen:
+ phrases.append(phrase)
+ seen.add(key)
+ return phrases
+
+
+def strict_quoted_phrase_hits(text: str, phrases: list[str]) -> int:
+ hits = 0
+ for phrase in phrases:
+ pattern = re.compile(
+ r"(? bool:
+ text = question.lower()
+ return bool(re.search(r"\b(day|days|week|weeks|month|months|year|years|hour|hours|duration|how long|spent)\b", text))
+
+
+def temporal_event_anchor_text(text: str) -> bool:
+ return bool(
+ re.search(
+ r"\b(start|started|starting|begin|began|finished|finish|completed|complete|ended|end|stopped|since|until)\b",
+ text.lower(),
+ )
+ )
+
+
+def parent_dialogue_header(text: str) -> str:
+ value = clean_text(text)
+ if not value:
+ return ""
+ match = re.search(
+ r"\bLongMemEval\s+session_id=[A-Za-z0-9_.:-]+\s+date=\d{4}/\d{2}/\d{2}\s+\([^)]+\)\s+\d{2}:\d{2}",
+ value,
+ )
+ if match:
+ return clean_text(match.group(0))
+ match = re.search(
+ r"\[[0-9T:+-]{19,25}\]\s+user:\s+LongMemEval\s+session_id=[A-Za-z0-9_.:-]+\s+date=\d{4}/\d{2}/\d{2}\s+\([^)]+\)\s+\d{2}:\d{2}",
+ value,
+ )
+ return clean_text(match.group(0)) if match else ""
+
+
+def attach_parent_temporal_context(unit: str, parent_text: str) -> str:
+ unit_text = clean_text(unit)
+ if not unit_text:
+ return ""
+ if re.search(r"\bdate=\d{4}/\d{2}/\d{2}\b", unit_text) or re.search(r"\bLongMemEval\s+session_id=", unit_text):
+ return unit_text
+ header = parent_dialogue_header(parent_text)
+ if not header:
+ return unit_text
+ return clean_text(f"{header} {unit_text}")
+
+
+# TMCRA_LAYER_TAG: rule-based evidence query expansion; candidate for modelized retrieval/intent replacement.
+EVIDENCE_QUERY_GENERIC_TERMS = {
+ "complement",
+ "current",
+ "recommend",
+ "recommendation",
+ "recommendations",
+ "suggest",
+ "suggestion",
+ "suggestions",
+}
+
+
+# TMCRA_LAYER_TAG: manual semantic alias buckets; not part of trained graph-memory model core.
+EVIDENCE_QUERY_ALIAS_GROUPS = (
+ {
+ "accessory",
+ "accessories",
+ "bag",
+ "camera",
+ "cameras",
+ "equipment",
+ "flash",
+ "gear",
+ "lens",
+ "lenses",
+ "photo",
+ "photography",
+ "sony",
+ "tripod",
+ },
+ {
+ "app",
+ "apps",
+ "dashboard",
+ "interface",
+ "layout",
+ "panel",
+ "software",
+ "tool",
+ "tools",
+ "ui",
+ "workflow",
+ },
+ {
+ "diet",
+ "drink",
+ "food",
+ "meal",
+ "restaurant",
+ "snack",
+ "taste",
+ },
+)
+
+
+def evidence_query_terms(question: str) -> set[str]:
+ terms = {term for term in content_tokens(question) if term not in EVIDENCE_QUERY_GENERIC_TERMS}
+ expanded = set(terms)
+ for group in EVIDENCE_QUERY_ALIAS_GROUPS:
+ if terms & group:
+ expanded.update(group)
+ return expanded
+
+
+def turn_units(text: str) -> list[str]:
+ text = clean_text(text)
+ if not text:
+ return []
+ marker = re.compile(r"(\[[^\]]+\bturn=\d+\s+role=(?:user|assistant)\])", re.IGNORECASE)
+ parts = marker.split(text)
+ units: list[str] = []
+ prefix = ""
+ index = 0
+ sentence_mode = os.getenv("TMCRA_DIALOGUE_TURN_SENTENCE_UNITS", "").strip().lower() not in {
+ "",
+ "0",
+ "false",
+ "off",
+ "no",
+ "disabled",
+ "none",
+ }
+ if parts and not marker.match(parts[0]):
+ prefix = parts[0].strip()
+ index = 1
+ while index < len(parts):
+ head = parts[index].strip()
+ body = parts[index + 1].strip() if index + 1 < len(parts) else ""
+ body_units = [body]
+ if sentence_mode:
+ split_units = [clean_text(item) for item in re.split(r"(?<=[.!?])\s+", body) if clean_text(item)]
+ if split_units:
+ body_units = split_units
+ for body_unit in body_units:
+ unit = clean_text(f"{head} {body_unit}")
+ if unit:
+ units.append(unit)
+ index += 2
+ if units:
+ if prefix:
+ units[0] = clean_text(f"{prefix} {units[0]}")
+ return units
+ sentence_units = re.split(r"(?<=[.!?])\s+", text)
+ return [clean_text(item) for item in sentence_units if clean_text(item)]
+
+
+def score_evidence_unit(question_terms: set[str], question: str, unit: str) -> float:
+ unit_norm = unit.lower()
+ unit_terms = set(content_tokens(unit))
+ overlap = question_terms & unit_terms
+ score = float(len(overlap))
+ for term in overlap:
+ if len(term) >= 5:
+ score += 0.35
+ raw_terms = [
+ item
+ for item in re.findall(r"[a-zA-Z0-9$]+", question.lower())
+ if item not in STOPWORDS and item not in EVIDENCE_QUERY_GENERIC_TERMS
+ ]
+ for size in (3, 2):
+ for index in range(0, max(0, len(raw_terms) - size + 1)):
+ phrase = " ".join(raw_terms[index : index + size])
+ if phrase and phrase in unit_norm:
+ score += 1.5 if size == 3 else 1.0
+ if re.search(r"\b\d+\b|\$\d+", unit_norm):
+ score += 0.3
+ if "role=user" in unit_norm:
+ score += 0.25
+ return score
+
+
+def centered_excerpt(unit: str, question_terms: set[str], *, max_chars: int) -> str:
+ unit = clean_text(unit)
+ if len(unit) <= max_chars:
+ return unit
+ unit_lower = unit.lower()
+ positions = [
+ unit_lower.find(term)
+ for term in sorted(question_terms, key=len, reverse=True)
+ if len(term) >= 4 and unit_lower.find(term) >= 0
+ ]
+ center = min(positions) if positions else 0
+ start = max(0, center - max_chars // 3)
+ end = min(len(unit), start + max_chars)
+ start = max(0, end - max_chars)
+ prefix = "..." if start > 0 else ""
+ suffix = "..." if end < len(unit) else ""
+ return prefix + unit[start:end].strip() + suffix
+
+
+def make_centered_window(
+ units: list[str],
+ seed_index: int,
+ score: float,
+ question_terms: set[str],
+ *,
+ max_chars: int,
+) -> dict[str, Any]:
+ seed_unit = units[seed_index].lower() if 0 <= seed_index < len(units) else ""
+ is_dialogue_question_seed = "role=user" in seed_unit and ("?" in seed_unit or "how many" in seed_unit)
+ if is_dialogue_question_seed:
+ max_chars = max(max_chars, 2200)
+ candidate_order = [seed_index, seed_index + 1, seed_index + 2, seed_index - 1, seed_index - 2]
+ order = [index for index in candidate_order if 0 <= index < len(units)]
+ else:
+ order = [seed_index]
+ for radius in (1, 2):
+ left = seed_index - radius
+ right = seed_index + radius
+ if left >= 0:
+ order.append(left)
+ if right < len(units):
+ order.append(right)
+
+ selected: list[tuple[int, str]] = []
+ used = 0
+ for index in order:
+ unit_l = units[index].lower() if 0 <= index < len(units) else ""
+ if "role=assistant" in unit_l:
+ budget = 1800 if index == seed_index else 900
+ else:
+ budget = 950 if index == seed_index else 350 if is_dialogue_question_seed else 520
+ piece = centered_excerpt(units[index], question_terms, max_chars=budget)
+ extra = len(piece) + 2
+ if selected and used + extra > max_chars:
+ continue
+ selected.append((index, piece))
+ used += extra
+ selected.sort(key=lambda item: item[0])
+ return {
+ "score": round(float(score), 3),
+ "unit_indexes": [index for index, _ in selected],
+ "text": "\n".join(piece for _, piece in selected),
+ }
+
+
+def assistant_answer_candidate_windows(
+ value: str,
+ question: str,
+ question_terms: set[str],
+ *,
+ max_windows: int = 4,
+ max_chars: int = 900,
+) -> list[dict[str, Any]]:
+ if not assistant_memory_query(question) or not assistant_origin_evidence(value):
+ return []
+ marker = re.compile(r"(\[[^\]]+\bturn=\d+\s+role=assistant\])", re.IGNORECASE)
+ parts = marker.split(clean_text(value))
+ candidates: list[tuple[float, int, str]] = []
+ raw_question_terms = [
+ item
+ for item in re.findall(r"[a-zA-Z0-9$]+", clean_text(question).lower())
+ if item not in STOPWORDS and item not in EVIDENCE_QUERY_GENERIC_TERMS
+ ]
+ focus_phrases: set[str] = set()
+ for size in (4, 3, 2):
+ for raw_index in range(0, max(0, len(raw_question_terms) - size + 1)):
+ phrase = " ".join(raw_question_terms[raw_index : raw_index + size])
+ if phrase:
+ focus_phrases.add(phrase)
+ for index in range(1, len(parts), 2):
+ head = parts[index].strip()
+ body = parts[index + 1].strip() if index + 1 < len(parts) else ""
+ if not body:
+ continue
+ pieces = [
+ clean_text(item)
+ for item in re.split(r"(?=(?:\d+\.|[-*]\s+|[A-Z][A-Za-z '’-]{2,40}:))|(?<=[.!?])\s+", body)
+ if clean_text(item)
+ ]
+ for local_index, piece in enumerate(pieces):
+ if len(piece) < 24:
+ continue
+ full_piece = clean_text(f"{head} {piece}")
+ score = score_evidence_unit(question_terms, question, full_piece)
+ piece_terms = set(content_tokens(piece))
+ overlap = piece_terms & question_terms
+ named_entity_like = bool(
+ re.search(r"\b(?:The\s+)?[A-Z][A-Za-z'’-]+(?:\s+(?:of|at|and|the|de|del|la|el|[A-Z][A-Za-z'’-]+)){1,7}\b", piece)
+ or re.search(r"['\"][^'\"]{3,80}['\"]", piece)
+ )
+ if named_entity_like:
+ score += 1.25
+ if overlap and named_entity_like:
+ score += 2.0
+ if len(overlap) >= 2:
+ score += 1.0
+ piece_l = clean_text(piece).lower()
+ phrase_hits = sum(1 for phrase in focus_phrases if len(phrase) >= 8 and phrase in piece_l)
+ if phrase_hits:
+ score += min(4.0, 1.5 * phrase_hits)
+ if score < 2.0:
+ continue
+ text = centered_excerpt(full_piece, question_terms, max_chars=max_chars)
+ candidates.append((score, local_index, text))
+ candidates.sort(key=lambda item: (item[0], -item[1]), reverse=True)
+ windows: list[dict[str, Any]] = []
+ seen: set[str] = set()
+ for score, _, text in candidates:
+ key = text[:220].lower()
+ if key in seen:
+ continue
+ seen.add(key)
+ windows.append(
+ {
+ "score": round(float(score) + 2.0, 3),
+ "unit_indexes": [],
+ "text": text,
+ "assistant_answer_candidate": True,
+ "assistant_memory_query": True,
+ "assistant_origin_evidence": True,
+ "assistant_candidate_side_channel": True,
+ "evidence_role": "assistant_answer_candidate",
+ "planner_selected": True,
+ "planner_score": round(float(score), 3),
+ }
+ )
+ if len(windows) >= max_windows:
+ break
+ return windows
+
+
+def _load_evidence_unit_selector() -> tuple[Any, Any, Any, dict[str, float]] | None:
+ mode = os.getenv("TMCRA_EVIDENCE_UNIT_SELECTOR_MODE", "").strip().lower()
+ if mode in {"", "off", "false", "0", "disabled", "none"}:
+ return None
+ model_path = os.getenv("TMCRA_EVIDENCE_UNIT_SELECTOR_MODEL_PATH", "").strip()
+ if not model_path:
+ return None
+ cache_key = "|".join([model_path, os.getenv("TMCRA_EVIDENCE_UNIT_SELECTOR_DEVICE", "cpu")])
+ cached = _EVIDENCE_UNIT_SELECTOR_CACHE.get(cache_key)
+ if cached is not None:
+ return cached
+ try:
+ from experiments.replacement import injection_planner as planner
+ except Exception:
+ return None
+ torch_module = getattr(planner, "torch", None)
+ if torch_module is None:
+ return None
+ try:
+ device = torch_module.device(os.getenv("TMCRA_EVIDENCE_UNIT_SELECTOR_DEVICE", "cpu") or "cpu")
+ payload = torch_module.load(Path(model_path), map_location=device, weights_only=True)
+ config = planner.InjectionPlannerConfig.from_dict(dict(payload.get("config", {}) or {}))
+ model = planner.InjectionPlannerModel(config).to(device)
+ model.load_state_dict(dict(payload.get("state_dict", {}) or {}), strict=False)
+ model.eval()
+ thresholds = {
+ "selection_threshold": float(os.getenv("TMCRA_EVIDENCE_UNIT_SELECTOR_SELECTION_THRESHOLD", "0.45")),
+ "row_threshold": float(os.getenv("TMCRA_EVIDENCE_UNIT_SELECTOR_ROW_THRESHOLD", "0.35")),
+ "logic_threshold": float(os.getenv("TMCRA_EVIDENCE_UNIT_SELECTOR_LOGIC_THRESHOLD", "0.45")),
+ }
+ except Exception:
+ return None
+ loaded = (planner, torch_module, model, thresholds)
+ _EVIDENCE_UNIT_SELECTOR_CACHE[cache_key] = loaded
+ return loaded
+
+
+def model_selected_evidence_unit_indexes(
+ question: str,
+ units: list[str],
+ hit_metadata: Mapping[str, Any],
+ scored_units: list[tuple[float, int]],
+) -> list[tuple[float, int]]:
+ loaded = _load_evidence_unit_selector()
+ if loaded is None or not units:
+ return []
+ planner, torch_module, model, thresholds = loaded
+ max_units = max(4, int(os.getenv("TMCRA_EVIDENCE_UNIT_SELECTOR_MAX_UNITS", "14") or 14))
+ top_scored = sorted(scored_units, key=lambda item: item[0], reverse=True)[:max_units]
+ candidate_indexes = sorted({index for _, index in top_scored if 0 <= index < len(units)})
+ if not candidate_indexes:
+ return []
+ score_by_index = {index: score for score, index in scored_units}
+ max_unit_score = max([abs(float(score)) for score, _ in top_scored] + [1.0])
+ retrieval_score = max(
+ 0.0,
+ min(
+ 1.0,
+ max(
+ metadata_float_from_mapping(hit_metadata, "hybrid_score"),
+ metadata_float_from_mapping(hit_metadata, "recall_score"),
+ metadata_float_from_mapping(hit_metadata, "event_score"),
+ metadata_float_from_mapping(hit_metadata, "path_score"),
+ ),
+ ),
+ )
+ graph_score = max(
+ 0.0,
+ min(
+ 1.0,
+ max(
+ metadata_float_from_mapping(hit_metadata, "answer_plan_score"),
+ metadata_float_from_mapping(hit_metadata, "answer_plan_selected_score"),
+ metadata_float_from_mapping(hit_metadata, "answer_plan_adjusted_score"),
+ ),
+ ),
+ )
+ temporal_state = clean_text(hit_metadata.get("injection_planner_temporal_state", "")) or "irrelevant"
+ evidence_role = clean_text(hit_metadata.get("injection_planner_evidence_role", "")) or clean_text(
+ hit_metadata.get("evidence_role", "")
+ )
+ candidates = []
+ for rank, index in enumerate(candidate_indexes, start=1):
+ unit_score = float(score_by_index.get(index, 0.0) or 0.0)
+ candidates.append(
+ {
+ "id": f"unit_{index}",
+ "text": clean_text(units[index]),
+ "layer": "event",
+ "temporal_state": temporal_state if temporal_state in getattr(planner, "TEMPORAL_STATES", ()) else "irrelevant",
+ "logic_roles": ["evidence"],
+ "evidence_role": evidence_role if evidence_role in getattr(planner, "EVIDENCE_ROLES", ()) else "",
+ "retrieval_score": retrieval_score,
+ "graph_score": graph_score,
+ "tunnel_score": max(0.0, min(1.0, metadata_float_from_mapping(hit_metadata, "path_score"))),
+ "topic_similarity": max(0.0, min(1.0, unit_score / max_unit_score)),
+ "confidence": retrieval_score,
+ "rank_score": round(1.0 / float(rank), 6),
+ "branch_depth": 1,
+ }
+ )
+ if len(candidates) < 3:
+ return []
+ try:
+ row = {"id": "evidence_unit_selector", "query": question, "candidates": candidates, "gold": {}}
+ dataset = planner.InjectionPlannerDataset([row], model.config)
+ batch = planner.collate_injection_batch([dataset[0]])
+ device = next(model.parameters()).device
+ model_batch = {key: value.to(device) if hasattr(value, "to") else value for key, value in dict(batch).items()}
+ with torch_module.no_grad():
+ outputs = model(model_batch["features"], model_batch["valid_mask"])
+ row_score = float(torch_module.sigmoid(outputs["should_inject_logits"])[0].detach().cpu().item())
+ selection_scores = torch_module.sigmoid(outputs["selection_logits"])[0].detach().cpu().tolist()
+ role_indices = torch_module.argmax(outputs["evidence_role_logits"], dim=-1)[0].detach().cpu().tolist()
+ except Exception:
+ return []
+ if row_score < thresholds["row_threshold"]:
+ return []
+ selected: list[tuple[float, int]] = []
+ for local_index, unit_index in enumerate(candidate_indexes):
+ role = planner.EVIDENCE_ROLES[int(role_indices[local_index])]
+ if role in {"noise", "negative_evidence"}:
+ continue
+ score = float(selection_scores[local_index])
+ if score >= thresholds["selection_threshold"]:
+ selected.append((score, unit_index))
+ selected.sort(key=lambda item: item[0], reverse=True)
+ return selected[: max(1, int(os.getenv("TMCRA_EVIDENCE_UNIT_SELECTOR_TOP_K", "4") or 4))]
+
+
+def metadata_float_from_mapping(metadata: Mapping[str, Any], key: str, default: float = 0.0) -> float:
+ try:
+ return float(metadata.get(key, default) or default)
+ except (TypeError, ValueError):
+ return default
+
+
+def evidence_windows_for_hit(question: str, hit: Any, *, max_chars: int = 1800) -> list[dict[str, Any]]:
+ value = clean_text(getattr(hit, "value", ""))
+ if not value:
+ return []
+ hit_metadata = dict(getattr(hit, "metadata", {}) or {})
+ evidence_snippet_role = clean_text(hit_metadata.get("evidence_snippet_role", ""))
+ unit_coverage_parent_event = bool(hit_metadata.get("unit_coverage_pack")) and evidence_snippet_role == "unit_coverage_parent_event"
+ multi_chain_parent_event = bool(hit_metadata.get("multi_unit_chain_slot")) and evidence_snippet_role == "multi_unit_chain_parent_event"
+ if unit_coverage_parent_event or multi_chain_parent_event:
+ try:
+ max_chars = max(max_chars, int(os.getenv("TMCRA_UNIT_COVERAGE_PARENT_WINDOW_CHARS", "3800") or 3800))
+ except (TypeError, ValueError):
+ max_chars = max(max_chars, 3800)
+ if assistant_memory_query(question) and assistant_origin_evidence(value):
+ try:
+ max_chars = max(max_chars, int(os.getenv("TMCRA_ASSISTANT_MEMORY_WINDOW_CHARS", "4200") or 4200))
+ except (TypeError, ValueError):
+ max_chars = max(max_chars, 4200)
+ def metadata_float(key: str, default: float = 0.0) -> float:
+ try:
+ return float(hit_metadata.get(key, default) or default)
+ except (TypeError, ValueError):
+ return default
+
+ answer_plan_score = max(
+ metadata_float("answer_plan_score"),
+ metadata_float("answer_plan_selected_score"),
+ metadata_float("answer_plan_current_score"),
+ metadata_float("answer_plan_adjusted_score"),
+ )
+ answer_plan_selected = bool(hit_metadata.get("answer_plan_selected", False)) and answer_plan_score > 0.0
+ try:
+ answer_plan_rank = int(hit_metadata.get("answer_plan_rank", 0) or 0)
+ except (TypeError, ValueError):
+ answer_plan_rank = 0
+ parent_profile_summary = clean_text(
+ hit_metadata.get("profile_first_parent_summary", "")
+ or hit_metadata.get("profile_summary", "")
+ or hit_metadata.get("profile_value", "")
+ )
+ if hit_metadata.get("profile_first_source_support") and parent_profile_summary:
+ value = clean_text(f"[profile summary] {parent_profile_summary} {value}")
+ if unit_coverage_parent_event or multi_chain_parent_event:
+ child_value = clean_text(hit_metadata.get("unit_coverage_child_value", "") or hit_metadata.get("multi_unit_chain_child_value", ""))
+ child_source_span = clean_text(
+ hit_metadata.get("unit_coverage_child_source_span", "")
+ or hit_metadata.get("multi_unit_chain_child_source_span", "")
+ )
+ child_anchor = child_source_span or child_value
+ if child_anchor:
+ parent_l = value.lower()
+ child_l = child_anchor.lower()
+ pos = parent_l.find(child_l[: min(len(child_l), 120)])
+ if pos >= 0:
+ half = max_chars // 2
+ start = max(0, pos - half)
+ end = min(len(value), pos + len(child_anchor) + half)
+ value = clean_text(value[start:end])
+ elif child_value or child_source_span:
+ value = clean_text(
+ f"[unit evidence] {child_source_span or child_value}\n[parent evidence] {value}"
+ )
+ units = turn_units(value)
+ if not units:
+ return []
+ question_terms = evidence_query_terms(question)
+ profile_bonus = 0.0
+ if hit_metadata.get("profile_first_hybrid_rescue"):
+ profile_bonus += 8.0
+ elif hit_metadata.get("profile_layer") or str(getattr(hit, "source_kind", "")).startswith("public_dialog_profile"):
+ profile_bonus += 4.0
+ scored = [
+ (score_evidence_unit(question_terms, question, unit) + profile_bonus, index)
+ for index, unit in enumerate(units)
+ ]
+ scored.sort(key=lambda item: item[0], reverse=True)
+ seed_indexes: list[tuple[float, int]] = []
+ model_unit_indexes = model_selected_evidence_unit_indexes(question, units, hit_metadata, scored)
+ for model_score, index in model_unit_indexes:
+ if any(existing == index for _, existing in seed_indexes):
+ continue
+ seed_indexes.append((max(float(model_score) * 10.0, float(scored[0][0]) if scored else 0.0), index))
+ seed_limit = 5 if assistant_memory_query(question) and assistant_origin_evidence(value) else 3
+ if len(seed_indexes) >= seed_limit:
+ break
+ for score, index in scored:
+ if score <= 0 and seed_indexes:
+ break
+ if score <= 0 and not seed_indexes:
+ seed_indexes.append((score, index))
+ break
+ if any(abs(index - existing) <= 1 for _, existing in seed_indexes):
+ continue
+ seed_indexes.append((score, index))
+ seed_limit = 5 if assistant_memory_query(question) and assistant_origin_evidence(value) else 3
+ if len(seed_indexes) >= seed_limit:
+ break
+ windows: list[dict[str, Any]] = []
+ for score, index in seed_indexes:
+ window = make_centered_window(units, index, score, question_terms, max_chars=max_chars)
+ window["memory_id"] = getattr(hit, "memory_id", "")
+ window["original_chars"] = len(value)
+ window["planner_selected"] = bool(hit_metadata.get("injection_planner_selected", False)) or answer_plan_selected
+ evidence_role = clean_text(hit_metadata.get("injection_planner_evidence_role", ""))
+ if not evidence_role and answer_plan_selected:
+ evidence_role = "direct_answer"
+ memory_id = clean_text(getattr(hit, "memory_id", ""))
+ source_kind = clean_text(getattr(hit, "source_kind", ""))
+ if (
+ not evidence_role
+ and (
+ bool(hit_metadata.get("profile_first_hybrid_rescue"))
+ or bool(hit_metadata.get("profile_protected_slot"))
+ or bool(hit_metadata.get("profile_layer"))
+ or ".subject." in memory_id
+ or source_kind.startswith("public_dialog_profile")
+ )
+ ):
+ evidence_role = "profile_fact"
+ window["evidence_role"] = evidence_role
+ window["temporal_state"] = clean_text(hit_metadata.get("injection_planner_temporal_state", ""))
+ promoted_answer_plan_score = answer_plan_score if answer_plan_selected else 0.0
+ window["planner_score"] = max(metadata_float("injection_planner_score"), promoted_answer_plan_score)
+ window["answer_plan_score"] = answer_plan_score
+ window["answer_plan_selected"] = answer_plan_selected
+ window["answer_plan_rank"] = answer_plan_rank
+ if hit_metadata.get("semantic_coverage_expansion"):
+ window["semantic_coverage_expansion"] = True
+ window["semantic_coverage_score"] = metadata_float("semantic_coverage_score")
+ window["semantic_coverage_source_memory_id"] = clean_text(hit_metadata.get("semantic_coverage_source_memory_id", ""))
+ if hit_metadata.get("unit_coverage_pack"):
+ window["unit_coverage_pack"] = True
+ window["unit_kind"] = clean_text(hit_metadata.get("unit_kind", ""))
+ window["facet_type"] = clean_text(hit_metadata.get("facet_type", ""))
+ window["unit_coverage_semantic_event_unit"] = bool(hit_metadata.get("unit_coverage_semantic_event_unit", False))
+ window["unit_coverage_semantic_event_priority"] = bool(hit_metadata.get("unit_coverage_semantic_event_priority", False))
+ window["unit_coverage_profile_shadow_unit"] = bool(hit_metadata.get("unit_coverage_profile_shadow_unit", False))
+ window["evidence_role"] = window["evidence_role"] or clean_text(hit_metadata.get("unit_kind", ""))
+ if hit_metadata.get("multi_unit_chain_slot"):
+ window["multi_unit_chain_slot"] = True
+ window["multi_unit_chain_bundle"] = bool(hit_metadata.get("multi_unit_chain_bundle", False))
+ window["multi_unit_chain_temporal_comparison"] = bool(hit_metadata.get("multi_unit_chain_temporal_comparison", False))
+ window["multi_unit_chain_semantic_event_unit"] = bool(hit_metadata.get("multi_unit_chain_semantic_event_unit", False))
+ window["multi_unit_chain_semantic_event_priority"] = bool(hit_metadata.get("multi_unit_chain_semantic_event_priority", False))
+ window["multi_unit_chain_profile_shadow_unit"] = bool(hit_metadata.get("multi_unit_chain_profile_shadow_unit", False))
+ window["unit_kind"] = window.get("unit_kind", "") or clean_text(hit_metadata.get("unit_kind", ""))
+ window["facet_type"] = window.get("facet_type", "") or clean_text(hit_metadata.get("facet_type", ""))
+ window["evidence_role"] = window["evidence_role"] or "multi_unit_chain"
+ window["multi_unit_chain_score"] = float(hit_metadata.get("multi_unit_chain_score", 0.0) or 0.0)
+ if hit_metadata.get("facet_query_pack"):
+ window["facet_query_pack"] = True
+ window["facet_type"] = window.get("facet_type", "") or clean_text(hit_metadata.get("facet_type", ""))
+ windows.append(window)
+ try:
+ assistant_candidate_limit = max(0, int(os.getenv("TMCRA_ASSISTANT_ANSWER_CANDIDATE_LIMIT", "0") or 0))
+ except (TypeError, ValueError):
+ assistant_candidate_limit = 4
+ if assistant_candidate_limit:
+ for candidate in assistant_answer_candidate_windows(
+ value,
+ question,
+ question_terms,
+ max_windows=assistant_candidate_limit,
+ max_chars=max(700, min(max_chars, 1200)),
+ ):
+ candidate["memory_id"] = getattr(hit, "memory_id", "")
+ candidate["original_chars"] = len(value)
+ candidate["temporal_state"] = clean_text(hit_metadata.get("injection_planner_temporal_state", ""))
+ candidate["answer_plan_score"] = answer_plan_score
+ candidate["answer_plan_selected"] = answer_plan_selected
+ candidate["answer_plan_rank"] = answer_plan_rank
+ windows.append(candidate)
+ return windows
+
+
+def build_answer_evidence(question: str, memory_hits: list[Any]) -> list[dict[str, Any]]:
+ windows: list[dict[str, Any]] = []
+ try:
+ hit_scan_limit = max(8, int(os.getenv("TMCRA_ANSWER_EVIDENCE_HIT_SCAN_LIMIT", "16") or 16))
+ build_limit = max(8, int(os.getenv("TMCRA_ANSWER_EVIDENCE_BUILD_LIMIT", "16") or 16))
+ except (TypeError, ValueError):
+ hit_scan_limit, build_limit = 16, 16
+ for hit in memory_hits[:hit_scan_limit]:
+ windows.extend(evidence_windows_for_hit(question, hit))
+ assistant_query = assistant_memory_query(question)
+ for index, item in enumerate(windows):
+ text = clean_text(item.get("text", ""))
+ if assistant_query and assistant_origin_evidence(text):
+ next_item = dict(item)
+ next_item["assistant_memory_query"] = True
+ next_item["assistant_origin_evidence"] = True
+ if not clean_text(next_item.get("evidence_role", "")):
+ next_item["evidence_role"] = "assistant_origin_detail"
+ windows[index] = next_item
+
+ def planner_window_key(item: dict[str, Any]) -> tuple[bool, bool, float, float, int]:
+ evidence_role = clean_text(item.get("evidence_role", ""))
+ role_allows_evidence = evidence_role not in {"noise", "negative_evidence"}
+ planner_score = float(item.get("planner_score", 0.0) or 0.0) if role_allows_evidence else 0.0
+ return (
+ bool(item.get("planner_selected", False)),
+ role_allows_evidence,
+ planner_score,
+ float(item.get("score", 0.0) or 0.0),
+ len(item.get("text", "")),
+ )
+
+ assistant_candidates = [item for item in windows if bool(item.get("assistant_candidate_side_channel", False))]
+ windows = [item for item in windows if not bool(item.get("assistant_candidate_side_channel", False))]
+
+ windows.sort(
+ key=planner_window_key,
+ reverse=True,
+ )
+ windows = compose_layered_evidence_windows(windows, build_limit=build_limit)
+ windows = diversify_evidence_windows(windows)
+ assistant_candidates.sort(key=planner_window_key, reverse=True)
+ try:
+ side_limit = max(0, int(os.getenv("TMCRA_ASSISTANT_ANSWER_CANDIDATE_SIDE_LIMIT", "12") or 12))
+ except (TypeError, ValueError):
+ side_limit = 12
+ return windows[:build_limit] + assistant_candidates[:side_limit]
+
+
+def assistant_memory_query(question: str) -> bool:
+ question_l = clean_text(question).lower()
+ return bool(
+ re.search(
+ r"\b(?:you said|you suggested|you recommended|you told|you wrote|you gave|you mentioned|"
+ r"previous chat|previous conversation|last time|remind me|we talked|we discussed|"
+ r"script you wrote|list you provided|what was the .* you said|what did you say)\b",
+ question_l,
+ )
+ )
+
+
+def assistant_origin_evidence(text: str) -> bool:
+ text_l = clean_text(text).lower()
+ if not text_l:
+ return False
+ if re.search(r"\[[^\]]+\brole=assistant\]", text_l):
+ return True
+ return bool(re.search(r"\b(?:assistant\]|role=assistant|assistant:)\b", text_l))
+
+
+def evidence_window_layer(item: dict[str, Any]) -> str:
+ memory_id = clean_text(item.get("memory_id", ""))
+ evidence_role = clean_text(item.get("evidence_role", "")).lower()
+ text = clean_text(item.get("text", "")).lower()
+ if bool(item.get("assistant_origin_evidence", False)) or evidence_role == "assistant_origin_detail":
+ return "assistant"
+ if bool(item.get("multi_unit_chain_slot", False)):
+ return "chain"
+ if bool(item.get("unit_coverage_pack", False)):
+ return "unit"
+ if bool(item.get("facet_query_pack", False)):
+ return "facet"
+ if (
+ "public_dialog_profile" in memory_id
+ or ".subject." in memory_id
+ or "profile" in evidence_role
+ or text.startswith("[profile summary]")
+ ):
+ return "profile"
+ if bool(item.get("answer_plan_selected", False)) or evidence_role in {"direct_answer", "positive", "supporting_fact"}:
+ return "direct"
+ return "base"
+
+
+def compose_layered_evidence_windows(evidence_windows: list[dict[str, Any]], *, build_limit: int) -> list[dict[str, Any]]:
+ mode = os.getenv("TMCRA_LAYERED_EVIDENCE_COMPOSE_MODE", "on").strip().lower()
+ if mode in {"", "0", "false", "off", "no", "disabled", "none"}:
+ return evidence_windows
+ try:
+ profile_limit = max(0, int(os.getenv("TMCRA_LAYERED_EVIDENCE_PROFILE_LIMIT", "3") or 3))
+ assistant_limit = max(0, int(os.getenv("TMCRA_LAYERED_EVIDENCE_ASSISTANT_LIMIT", "4") or 4))
+ direct_limit = max(0, int(os.getenv("TMCRA_LAYERED_EVIDENCE_DIRECT_LIMIT", "4") or 4))
+ base_limit = max(0, int(os.getenv("TMCRA_LAYERED_EVIDENCE_BASE_LIMIT", "5") or 5))
+ unit_limit = max(0, int(os.getenv("TMCRA_LAYERED_EVIDENCE_UNIT_LIMIT", "4") or 4))
+ chain_limit = max(0, int(os.getenv("TMCRA_LAYERED_EVIDENCE_CHAIN_LIMIT", "4") or 4))
+ facet_limit = max(0, int(os.getenv("TMCRA_LAYERED_EVIDENCE_FACET_LIMIT", "2") or 2))
+ except (TypeError, ValueError):
+ profile_limit, assistant_limit, direct_limit, base_limit, unit_limit, chain_limit, facet_limit = 3, 4, 4, 5, 4, 4, 2
+ buckets: dict[str, list[dict[str, Any]]] = {
+ "profile": [],
+ "assistant": [],
+ "direct": [],
+ "base": [],
+ "unit": [],
+ "chain": [],
+ "facet": [],
+ }
+ for item in evidence_windows:
+ buckets.setdefault(evidence_window_layer(item), buckets["base"]).append(item)
+
+ def coverage_bucket_key(item: dict[str, Any]) -> tuple[int, int, int, int, float, float, int]:
+ unit_kind = clean_text(item.get("unit_kind", "")).lower()
+ evidence_role = clean_text(item.get("evidence_role", "")).lower()
+ memory_id = clean_text(item.get("memory_id", "")).lower()
+ semantic_event = int(
+ bool(item.get("multi_unit_chain_semantic_event_priority", False))
+ or bool(item.get("unit_coverage_semantic_event_priority", False))
+ )
+ profile_shadow = int(
+ unit_kind == "profile_shadow_unit"
+ or bool(item.get("multi_unit_chain_profile_shadow_unit", False))
+ or bool(item.get("unit_coverage_profile_shadow_unit", False))
+ or "profile_shadow" in memory_id
+ )
+ selected = int(bool(item.get("planner_selected", False)) or bool(item.get("evidence_unit_planner_selected", False)))
+ parent_event = int("parent_event" in evidence_role or "#multi_parent:" in memory_id or "#unit_parent:" in memory_id)
+ return (
+ semantic_event,
+ selected,
+ profile_shadow,
+ parent_event,
+ float(item.get("multi_unit_chain_score", 0.0) or 0.0),
+ float(item.get("score", 0.0) or 0.0),
+ len(clean_text(item.get("text", ""))),
+ )
+
+ buckets["chain"].sort(key=coverage_bucket_key, reverse=True)
+ buckets["unit"].sort(key=coverage_bucket_key, reverse=True)
+
+ ordered: list[dict[str, Any]] = []
+ seen_keys: set[str] = set()
+
+ def add_from(items: list[dict[str, Any]], limit: int) -> None:
+ if limit <= 0:
+ return
+ added = 0
+ for item in items:
+ memory_id = clean_text(item.get("memory_id", ""))
+ text_key = clean_text(item.get("text", ""))[:240].lower()
+ key = memory_id or text_key
+ if key and key in seen_keys:
+ continue
+ if key:
+ seen_keys.add(key)
+ ordered.append(item)
+ added += 1
+ if len(ordered) >= build_limit or added >= limit:
+ break
+
+ # Stable evidence composition: profile/direct/base keep the normal answer surface;
+ # unit and chain provide bounded coverage slots instead of competing for all top ranks.
+ temporal_chain_first = any(bool(item.get("multi_unit_chain_temporal_comparison", False)) for item in buckets["chain"])
+ add_from(buckets["profile"], profile_limit)
+ add_from(buckets["assistant"], assistant_limit)
+ add_from(buckets["direct"], direct_limit)
+ if temporal_chain_first:
+ add_from(buckets["chain"], chain_limit)
+ add_from(buckets["base"], base_limit)
+ add_from(buckets["unit"], unit_limit)
+ else:
+ add_from(buckets["base"], base_limit)
+ add_from(buckets["unit"], unit_limit)
+ add_from(buckets["chain"], chain_limit)
+ add_from(buckets["facet"], facet_limit)
+ for item in evidence_windows:
+ if len(ordered) >= build_limit:
+ break
+ memory_id = clean_text(item.get("memory_id", ""))
+ text_key = clean_text(item.get("text", ""))[:240].lower()
+ key = memory_id or text_key
+ if key and key in seen_keys:
+ continue
+ if key:
+ seen_keys.add(key)
+ ordered.append(item)
+ return ordered
+
+
+def expand_dialogue_chain_hits(question: str, memory_hits: list[Any], graph: Any | None) -> list[Any]:
+ if graph is None or not memory_hits:
+ return list(memory_hits)
+ mode = os.getenv("TMCRA_DIALOGUE_CHAIN_EXPANSION_MODE", "adjacent").strip().lower()
+ if mode in {"", "off", "false", "0", "disabled", "none"}:
+ return list(memory_hits)
+ try:
+ radius = max(0, int(os.getenv("TMCRA_DIALOGUE_CHAIN_EXPANSION_RADIUS", "3") or 3))
+ max_total = max(0, int(os.getenv("TMCRA_DIALOGUE_CHAIN_EXPANSION_MAX_TOTAL", "5") or 5))
+ max_per_seed = max(1, int(os.getenv("TMCRA_DIALOGUE_CHAIN_EXPANSION_MAX_PER_SEED", "2") or 2))
+ except (TypeError, ValueError):
+ radius, max_total, max_per_seed = 3, 5, 2
+ if radius <= 0 or max_total <= 0:
+ return list(memory_hits)
+ question_terms = evidence_query_terms(question)
+ if not question_terms:
+ return list(memory_hits)
+ records_by_id = getattr(graph, "records_by_id", {}) or {}
+ if not records_by_id:
+ return list(memory_hits)
+ existing_ids = {clean_text(getattr(hit, "memory_id", "")) for hit in memory_hits if clean_text(getattr(hit, "memory_id", ""))}
+ seed_infos = []
+ for seed_index, hit in enumerate(memory_hits[:12]):
+ session_id = dialogue_session_id_from_text(hit_text_for_chain(hit))
+ if not session_id:
+ continue
+ seed_infos.append((seed_index, session_id, int(getattr(hit, "turn_index", 0) or 0), clean_text(getattr(hit, "memory_id", ""))))
+ if not seed_infos:
+ return list(memory_hits)
+
+ candidates_by_seed: dict[str, list[Any]] = {}
+ for record in records_by_id.values():
+ memory_id = clean_text(getattr(record, "memory_id", ""))
+ if not memory_id or memory_id in existing_ids:
+ continue
+ state = clean_text(getattr(record, "state", "active")).lower()
+ if state not in {"active", "parallel_active", "evidence"}:
+ continue
+ value = clean_text(getattr(record, "value", ""))
+ metadata = dict(getattr(record, "metadata", {}) or {})
+ session_id = dialogue_session_id_from_text(" ".join([value, clean_text(metadata.get("raw_text", "")), clean_text(metadata.get("source_turn_text", ""))]))
+ if not session_id:
+ continue
+ record_turn = int(getattr(record, "turn_index", 0) or 0)
+ for seed_index, seed_session, seed_turn, seed_id in seed_infos:
+ if session_id != seed_session:
+ continue
+ distance = abs(record_turn - seed_turn) if seed_turn and record_turn else 0
+ if distance > radius:
+ continue
+ score_metadata = {**metadata, "source_kind": clean_text(getattr(record, "source_kind", ""))}
+ score = dialogue_chain_candidate_score(question, question_terms, value, score_metadata, distance)
+ if score <= 0:
+ continue
+ hit = record_to_chain_hit(record, score=score, seed_id=seed_id, distance=distance, session_id=session_id)
+ candidates_by_seed.setdefault(seed_id, []).append(hit)
+
+ expanded: list[Any] = []
+ added_ids: set[str] = set()
+ total_added = 0
+ for hit_index, hit in enumerate(memory_hits):
+ expanded.append(hit)
+ seed_id = clean_text(getattr(hit, "memory_id", ""))
+ candidates = candidates_by_seed.get(seed_id, [])
+ candidates.sort(
+ key=lambda item: (
+ -float(getattr(item, "score", 0.0) or 0.0),
+ int((getattr(item, "metadata", {}) or {}).get("dialogue_chain_distance", 99) or 99),
+ int(getattr(item, "turn_index", 0) or 0),
+ clean_text(getattr(item, "memory_id", "")),
+ )
+ )
+ per_seed_added = 0
+ for candidate in candidates:
+ candidate_id = clean_text(getattr(candidate, "memory_id", ""))
+ if not candidate_id or candidate_id in existing_ids or candidate_id in added_ids:
+ continue
+ expanded.append(candidate)
+ added_ids.add(candidate_id)
+ total_added += 1
+ per_seed_added += 1
+ if per_seed_added >= max_per_seed or total_added >= max_total:
+ break
+ if total_added >= max_total:
+ expanded.extend(memory_hits[hit_index + 1 :])
+ break
+ if not added_ids:
+ return list(memory_hits)
+ return dedupe_hits_by_memory_id(expanded)
+
+
+def expand_semantic_coverage_hits(question: str, memory_hits: list[Any], graph: Any | None) -> list[Any]:
+ mode = os.getenv("TMCRA_SEMANTIC_COVERAGE_EXPANSION_MODE", "").strip().lower()
+ if mode in {"", "off", "false", "0", "disabled", "none"}:
+ return list(memory_hits)
+ if graph is None:
+ return list(memory_hits)
+ records_by_id = getattr(graph, "records_by_id", {}) or {}
+ if not records_by_id:
+ return list(memory_hits)
+ embedder_mode = os.getenv("TMCRA_SEMANTIC_COVERAGE_EMBEDDER_MODE", "").strip() or os.getenv(
+ "TMCRA_EMBEDDER_INDEX_RECALL_MODE", ""
+ ).strip() or "bge_m3"
+ try:
+ max_records = max(16, int(os.getenv("TMCRA_SEMANTIC_COVERAGE_MAX_RECORDS", "160") or 160))
+ max_units = max(24, int(os.getenv("TMCRA_SEMANTIC_COVERAGE_MAX_UNITS", "260") or 260))
+ max_add = max(1, int(os.getenv("TMCRA_SEMANTIC_COVERAGE_MAX_ADD", "16") or 16))
+ score_floor = float(os.getenv("TMCRA_SEMANTIC_COVERAGE_SCORE_FLOOR", "0.74") or 0.74)
+ except (TypeError, ValueError):
+ max_records, max_units, max_add, score_floor = 160, 260, 16, 0.74
+ try:
+ from experiments.replacement.adapters import memory_adapters
+ except Exception:
+ return list(memory_hits)
+ existing_unit_texts = {clean_text(getattr(hit, "value", "")) for hit in memory_hits if clean_text(getattr(hit, "value", ""))}
+ candidates: list[tuple[Any, int, str]] = []
+ for record in list(records_by_id.values())[:max_records]:
+ state = clean_text(getattr(record, "state", "active")).lower()
+ if state not in {"active", "parallel_active", "evidence"}:
+ continue
+ value = clean_text(getattr(record, "value", ""))
+ if not value:
+ continue
+ for unit_index, unit in enumerate((turn_units(value) or [value])[:8]):
+ text = clean_text(unit)
+ if not text or text in existing_unit_texts:
+ continue
+ candidates.append((record, unit_index, text))
+ if len(candidates) >= max_units:
+ break
+ if len(candidates) >= max_units:
+ break
+ if not candidates:
+ return list(memory_hits)
+ texts = [clean_text(question), *[text for _, _, text in candidates]]
+ try:
+ vectors, metadata = memory_adapters._embedder_dense_vectors_for_texts(texts, mode=embedder_mode)
+ except Exception:
+ return list(memory_hits)
+ if not vectors or not vectors[0] or not bool(metadata.get("write_embedder_dense_enabled", False)):
+ return list(memory_hits)
+ query_vector = list(vectors[0] or [])
+ query_phrases = quoted_query_phrases(question)
+ query_needs_temporal_anchors = bool(query_phrases and temporal_duration_query(question))
+ scored: list[tuple[float, str, Any, int, str]] = []
+ for (record, unit_index, text), vector in zip(candidates, vectors[1:]):
+ if query_needs_temporal_anchors:
+ text = attach_parent_temporal_context(text, clean_text(getattr(record, "value", "")))
+ score = _vector_dot(query_vector, list(vector or []))
+ phrase_hits = strict_quoted_phrase_hits(text, query_phrases)
+ if query_phrases and phrase_hits <= 0:
+ continue
+ if query_needs_temporal_anchors and not temporal_event_anchor_text(text):
+ continue
+ if phrase_hits:
+ score = min(1.0, score + (0.025 * min(3, phrase_hits)))
+ if score < score_floor:
+ continue
+ memory_id = clean_text(getattr(record, "memory_id", "")) or clean_text(getattr(record, "slot_key", ""))
+ scored.append((score, memory_id, record, unit_index, text))
+ if not scored:
+ return list(memory_hits)
+ scored.sort(key=lambda item: (-item[0], item[1], item[3]))
+ expanded = list(memory_hits)
+ added = 0
+ seen_semantic_ids: set[str] = set()
+ for score, memory_id, record, unit_index, text in scored:
+ semantic_id = f"{memory_id}#semantic_unit_{unit_index}"
+ if semantic_id in seen_semantic_ids:
+ continue
+ metadata = dict(getattr(record, "metadata", {}) or {})
+ metadata.update(
+ {
+ "semantic_coverage_expansion": True,
+ "semantic_coverage_score": round(float(score), 6),
+ "semantic_coverage_unit_index": int(unit_index),
+ "semantic_coverage_source_memory_id": memory_id,
+ "injection_planner_selected": True,
+ "injection_planner_score": round(float(score), 6),
+ "injection_planner_evidence_role": "positive",
+ "hybrid_score": max(float(score), metadata_float_from_mapping(metadata, "hybrid_score")),
+ "recall_score": max(float(score), metadata_float_from_mapping(metadata, "recall_score")),
+ }
+ )
+ expanded.append(
+ SimpleNamespace(
+ memory_id=semantic_id,
+ category=clean_text(getattr(record, "category", "")),
+ value=text,
+ relation=clean_text(getattr(record, "relation", "related_to")) or "related_to",
+ anchors=list(getattr(record, "anchor_concepts", []) or []),
+ score=float(score),
+ source_kind="semantic_coverage_unit",
+ slot_key=clean_text(getattr(record, "slot_key", "")),
+ state=clean_text(getattr(record, "state", "active")) or "active",
+ turn_index=int(getattr(record, "turn_index", 0) or 0),
+ metadata=metadata,
+ )
+ )
+ seen_semantic_ids.add(semantic_id)
+ added += 1
+ if added >= max_add:
+ break
+ return dedupe_hits_by_memory_id(expanded)
+
+
+def hit_text_for_chain(hit: Any) -> str:
+ metadata = dict(getattr(hit, "metadata", {}) or {})
+ return " ".join(
+ [
+ clean_text(getattr(hit, "value", "")),
+ clean_text(metadata.get("raw_text", "")),
+ clean_text(metadata.get("source_turn_text", "")),
+ clean_text(metadata.get("source_span", "")),
+ ]
+ )
+
+
+def dialogue_session_id_from_text(text: str) -> str:
+ value = clean_text(text)
+ if not value:
+ return ""
+ match = re.search(r"\bLongMemEval\s+session_id=([A-Za-z0-9_.:-]+)", value)
+ if match:
+ return match.group(1)
+ match = re.search(r"\[([A-Za-z0-9_.:-]+)\s+turn=\d+\s+role=", value)
+ return match.group(1) if match else ""
+
+
+def dialogue_chain_candidate_score(question: str, question_terms: set[str], value: str, metadata: Mapping[str, Any], distance: int) -> float:
+ text = " ".join([value, clean_text(metadata.get("source_span", "")), clean_text(metadata.get("event_text", ""))])
+ units = turn_units(text) or [text]
+ base = max(score_evidence_unit(question_terms, question, unit) for unit in units)
+ if base <= 0:
+ return 0.0
+ source_kind = clean_text(metadata.get("source_kind", ""))
+ if source_kind.startswith("public_dialog_") and source_kind != "public_dialog_turn":
+ base += 0.35
+ if clean_text(metadata.get("event_id", "")):
+ base += 0.2
+ return round(max(0.0, base - (0.15 * max(0, distance - 1))), 6)
+
+
+def record_to_chain_hit(record: Any, *, score: float, seed_id: str, distance: int, session_id: str) -> Any:
+ metadata = dict(getattr(record, "metadata", {}) or {})
+ metadata.update(
+ {
+ "dialogue_chain_expansion": True,
+ "dialogue_chain_seed_memory_id": seed_id,
+ "dialogue_chain_distance": int(distance),
+ "dialogue_chain_session_id": session_id,
+ }
+ )
+ return SimpleNamespace(
+ memory_id=clean_text(getattr(record, "memory_id", "")),
+ category=clean_text(getattr(record, "category", "")),
+ value=clean_text(getattr(record, "value", "")),
+ relation=clean_text(getattr(record, "relation", "related_to")) or "related_to",
+ anchors=list(getattr(record, "anchor_concepts", []) or []),
+ score=float(score),
+ source_kind=clean_text(getattr(record, "source_kind", "memory")) or "memory",
+ slot_key=clean_text(getattr(record, "slot_key", "")),
+ state=clean_text(getattr(record, "state", "active")) or "active",
+ turn_index=int(getattr(record, "turn_index", 0) or 0),
+ metadata=metadata,
+ )
+
+
+def dedupe_hits_by_memory_id(hits: list[Any]) -> list[Any]:
+ deduped: list[Any] = []
+ seen: set[str] = set()
+ for hit in hits:
+ memory_id = clean_text(getattr(hit, "memory_id", ""))
+ if memory_id and memory_id in seen:
+ continue
+ if memory_id:
+ seen.add(memory_id)
+ deduped.append(hit)
+ return deduped
+
+
+def _load_answer_window_planner() -> tuple[Any, Any, Any, dict[str, float]] | None:
+ mode = os.getenv("TMCRA_ANSWER_WINDOW_PLANNER_MODE", "").strip().lower()
+ if mode in {"", "off", "false", "0", "disabled", "none"}:
+ return None
+ model_path = (
+ os.getenv("TMCRA_ANSWER_WINDOW_PLANNER_MODEL_PATH", "").strip()
+ or os.getenv("TMCRA_INJECTION_PLANNER_MODEL_PATH", "").strip()
+ )
+ if not model_path:
+ return None
+ cache_key = "|".join(
+ [
+ model_path,
+ os.getenv("TMCRA_ANSWER_WINDOW_PLANNER_DEVICE", os.getenv("TMCRA_INJECTION_PLANNER_DEVICE", "cpu")),
+ ]
+ )
+ cached = _ANSWER_WINDOW_PLANNER_CACHE.get(cache_key)
+ if cached is not None:
+ return cached
+ try:
+ from experiments.replacement import injection_planner as planner
+ except Exception:
+ return None
+ torch_module = getattr(planner, "torch", None)
+ if torch_module is None:
+ return None
+ try:
+ device = torch_module.device(os.getenv("TMCRA_ANSWER_WINDOW_PLANNER_DEVICE", os.getenv("TMCRA_INJECTION_PLANNER_DEVICE", "cpu")) or "cpu")
+ payload = torch_module.load(Path(model_path), map_location=device, weights_only=True)
+ config = planner.InjectionPlannerConfig.from_dict(dict(payload.get("config", {}) or {}))
+ model = planner.InjectionPlannerModel(config).to(device)
+ model.load_state_dict(dict(payload.get("state_dict", {}) or {}), strict=False)
+ model.eval()
+ thresholds = {
+ "selection_threshold": float(os.getenv("TMCRA_ANSWER_WINDOW_PLANNER_SELECTION_THRESHOLD", os.getenv("TMCRA_INJECTION_PLANNER_SELECTION_THRESHOLD", "0.3"))),
+ "row_threshold": float(os.getenv("TMCRA_ANSWER_WINDOW_PLANNER_ROW_THRESHOLD", os.getenv("TMCRA_INJECTION_PLANNER_ROW_THRESHOLD", "0.3"))),
+ "logic_threshold": float(os.getenv("TMCRA_ANSWER_WINDOW_PLANNER_LOGIC_THRESHOLD", os.getenv("TMCRA_INJECTION_PLANNER_LOGIC_THRESHOLD", "0.45"))),
+ }
+ except Exception:
+ return None
+ loaded = (planner, torch_module, model, thresholds)
+ _ANSWER_WINDOW_PLANNER_CACHE[cache_key] = loaded
+ return loaded
+
+
+def apply_answer_window_planner(question: str, evidence_windows: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ loaded = _load_answer_window_planner()
+ if loaded is None or not evidence_windows:
+ return evidence_windows
+ planner, torch_module, model, thresholds = loaded
+ evidence_windows = apply_answer_window_semantic_features(question, evidence_windows)
+ max_score = max([float(item.get("score", 0.0) or 0.0) for item in evidence_windows] + [1.0])
+ candidates = []
+ for index, item in enumerate(evidence_windows):
+ original_role = clean_text(item.get("evidence_role", ""))
+ role_is_noise = original_role in {"noise", "negative_evidence"}
+ semantic_similarity = max(
+ 0.0,
+ min(
+ 1.0,
+ float(item.get("answer_window_semantic_similarity", item.get("semantic_similarity", 0.0)) or 0.0),
+ ),
+ )
+ candidates.append(
+ {
+ "id": f"window_{index}",
+ "text": clean_text(item.get("text", "")),
+ "layer": "event",
+ "temporal_state": clean_text(item.get("temporal_state", "")) or "irrelevant",
+ "logic_roles": ["noise"] if role_is_noise else ["evidence"],
+ "retrieval_score": min(1.0, max(0.0, float(item.get("score", 0.0) or 0.0) / max_score)),
+ "graph_score": min(1.0, max(0.0, float(item.get("planner_score", 0.0) or 0.0))),
+ "tunnel_score": 1.0 if original_role == "bridge_context" else 0.0,
+ "topic_similarity": 0.0,
+ "semantic_similarity": semantic_similarity,
+ "confidence": min(1.0, max(0.0, float(item.get("score", 0.0) or 0.0) / max_score)),
+ "rank_score": round(1.0 / float(index + 1), 6),
+ "branch_depth": 1,
+ "evidence_role": original_role,
+ }
+ )
+ try:
+ row = {"id": "answer_window_plan", "query": question, "candidates": candidates, "gold": {}}
+ dataset = planner.InjectionPlannerDataset([row], model.config)
+ batch = planner.collate_injection_batch([dataset[0]])
+ device = next(model.parameters()).device
+ model_batch = {key: value.to(device) if hasattr(value, "to") else value for key, value in dict(batch).items()}
+ with torch_module.no_grad():
+ outputs = model(model_batch["features"], model_batch["valid_mask"])
+ should_inject_score = float(torch_module.sigmoid(outputs["should_inject_logits"])[0].detach().cpu().item())
+ selection_scores = torch_module.sigmoid(outputs["selection_logits"])[0].detach().cpu().tolist()
+ temporal_indices = torch_module.argmax(outputs["temporal_logits"], dim=-1)[0].detach().cpu().tolist()
+ role_indices = torch_module.argmax(outputs["evidence_role_logits"], dim=-1)[0].detach().cpu().tolist()
+ logic_scores = torch_module.sigmoid(outputs["logic_logits"])[0].detach().cpu().tolist()
+ except Exception:
+ return evidence_windows
+ row_allows = should_inject_score >= thresholds["row_threshold"]
+ planned: list[dict[str, Any]] = []
+ for index, item in enumerate(evidence_windows):
+ next_item = dict(item)
+ item_semantic_similarity = max(
+ 0.0,
+ min(
+ 1.0,
+ float(item.get("answer_window_semantic_similarity", item.get("semantic_similarity", 0.0)) or 0.0),
+ ),
+ )
+ role = planner.EVIDENCE_ROLES[int(role_indices[index])]
+ role_allows = role not in {"noise", "negative_evidence"}
+ selected = bool(row_allows and role_allows and float(selection_scores[index]) >= thresholds["selection_threshold"])
+ logic_roles = [
+ logic_role
+ for logic_role, score in zip(planner.LOGIC_ROLES, logic_scores[index])
+ if float(score) >= thresholds["logic_threshold"]
+ ]
+ next_item.update(
+ {
+ "hit_evidence_role": clean_text(item.get("evidence_role", "")),
+ "hit_temporal_state": clean_text(item.get("temporal_state", "")),
+ "planner_selected": selected,
+ "planner_score": round(float(selection_scores[index]), 6),
+ "semantic_similarity": round(float(item_semantic_similarity), 6),
+ "evidence_role": role,
+ "temporal_state": planner.TEMPORAL_STATES[int(temporal_indices[index])],
+ "window_planner_enabled": True,
+ "window_planner_should_inject_score": round(float(should_inject_score), 6),
+ "window_planner_logic_roles": logic_roles or ["evidence"],
+ }
+ )
+ planned.append(next_item)
+
+ def window_key(item: dict[str, Any]) -> tuple[bool, bool, float, float, int]:
+ evidence_role = clean_text(item.get("evidence_role", ""))
+ role_allows_evidence = evidence_role not in {"noise", "negative_evidence"}
+ planner_score = float(item.get("planner_score", 0.0) or 0.0) if role_allows_evidence else 0.0
+ return (
+ bool(item.get("planner_selected", False)),
+ role_allows_evidence,
+ planner_score,
+ float(item.get("score", 0.0) or 0.0),
+ len(item.get("text", "")),
+ )
+
+ planned.sort(key=window_key, reverse=True)
+ return planned
+
+
+def apply_llm_evidence_selector(question: str, evidence_windows: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ # TMCRA_LAYER_TAG: answer-side LLM evidence selector; post-retrieval rerank only, not recall.
+ mode = os.getenv("TMCRA_LLM_EVIDENCE_SELECTOR_MODE", "on").strip().lower()
+ if mode in {"", "off", "false", "0", "disabled", "none"} or not evidence_windows:
+ return evidence_windows
+ try:
+ max_candidates = max(1, int(os.getenv("TMCRA_LLM_EVIDENCE_SELECTOR_MAX_CANDIDATES", "12") or 12))
+ max_chars = max(200, int(os.getenv("TMCRA_LLM_EVIDENCE_SELECTOR_CHARS", "900") or 900))
+ except (TypeError, ValueError):
+ max_candidates, max_chars = 12, 900
+ candidates = list(evidence_windows[:max_candidates])
+ candidate_lines = []
+ for index, item in enumerate(candidates, start=1):
+ memory_id = clean_text(item.get("memory_id", ""))
+ text = truncate(item.get("text", ""), max_chars)
+ if not text:
+ continue
+ header = f"[{index}]"
+ if memory_id:
+ header += f" memory_id={memory_id}"
+ candidate_lines.append(f"{header}\n{text}")
+ if not candidate_lines:
+ return evidence_windows
+ messages = [
+ {
+ "role": "system",
+ "content": (
+ "You are a semantic evidence selector for a long-memory runtime. "
+ "Your job is only to select evidence windows that could help answer the current question. "
+ "Use semantic relevance and discourse continuity, not keyword lists or domain-specific rules. "
+ "Select every distinct useful clue, including indirect clues, continuation turns, and separate pieces that must be combined. "
+ "If two windows repeat the same clue, keep the clearest one. "
+ "If two windows contain different clues, keep both even if one looks weaker or appears later in the list. "
+ "Do not answer the question. "
+ "Return strict JSON only with keys: selected_indices, rejected_indices. "
+ "selected_indices must be 1-based integers from the candidate list."
+ ),
+ },
+ {
+ "role": "user",
+ "content": f"Question:\n{question}\n\nCandidate evidence windows:\n" + "\n\n".join(candidate_lines),
+ },
+ ]
+ try:
+ raw = chat_completion(
+ os.getenv("TMCRA_EVIDENCE_SELECTOR_BASE_URL", os.getenv("GEMMA_BASE_URL", "http://127.0.0.1:18002/v1")),
+ os.getenv("TMCRA_EVIDENCE_SELECTOR_MODEL", os.getenv("GEMMA_MODEL", os.getenv("TMCRA_GEMMA_MODEL_NAME", "gemma-4-e4b-it"))),
+ messages,
+ max_tokens=160,
+ temperature=0.0,
+ )
+ except Exception:
+ return evidence_windows
+ payload = parse_json_object(raw)
+ raw_indices = payload.get("selected_indices", []) if payload else []
+ if not isinstance(raw_indices, list):
+ return evidence_windows
+ selected_order: list[int] = []
+ for value in iter_selector_indices(raw_indices):
+ try:
+ index = int(value) - 1
+ except (TypeError, ValueError):
+ continue
+ if 0 <= index < len(candidates) and index not in selected_order:
+ selected_order.append(index)
+ if not selected_order:
+ return evidence_windows
+ keep_model_selected = os.getenv("TMCRA_LLM_EVIDENCE_SELECTOR_KEEP_MODEL_SELECTED", "1").strip().lower() not in {
+ "0",
+ "false",
+ "off",
+ "no",
+ }
+ model_selected_order: list[int] = []
+ if keep_model_selected:
+ for index, item in enumerate(candidates):
+ if bool(item.get("planner_selected", False)) or bool(item.get("answer_plan_selected", False)):
+ model_selected_order.append(index)
+ selected_order = list(dict.fromkeys([*selected_order, *model_selected_order]))
+ selected_set = set(selected_order)
+ selected_rank = {index: rank for rank, index in enumerate(selected_order)}
+ planned: list[tuple[int, int, dict[str, Any]]] = []
+ for index, item in enumerate(evidence_windows):
+ next_item = dict(item)
+ llm_selected = index in selected_set
+ next_item["llm_evidence_selected"] = llm_selected
+ next_item["llm_evidence_rank"] = selected_rank.get(index, 9999)
+ next_item["llm_evidence_selector_enabled"] = True
+ next_item["llm_evidence_model_kept"] = bool(index in model_selected_order)
+ group = 0 if llm_selected else 1
+ rank = selected_rank.get(index, index)
+ planned.append((group, rank, next_item))
+ planned.sort(key=lambda row: (row[0], row[1]))
+ return diversify_evidence_windows([item for _, _, item in planned])
+
+
+def apply_evidence_unit_planner(question: str, evidence_windows: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ # TMCRA_LAYER_TAG: answer-side LLM evidence-unit planner; normalizes retrieved windows after recall.
+ mode = os.getenv("TMCRA_EVIDENCE_UNIT_PLANNER_MODE", "on").strip().lower()
+ if mode in {"", "off", "false", "0", "disabled", "none"} or not evidence_windows:
+ return evidence_windows
+ if evidence_unit_plan_from_windows(evidence_windows):
+ return evidence_windows
+ plan = build_evidence_unit_plan(question, evidence_windows)
+ if not plan:
+ return evidence_windows
+ selected_ranks = evidence_unit_selected_source_ranks(plan)
+ reorder_mode = os.getenv("TMCRA_EVIDENCE_UNIT_PLANNER_REORDER", "0").strip().lower()
+ reorder_enabled = reorder_mode not in {"", "0", "false", "off", "no", "disabled", "none"}
+ planned: list[tuple[int, int, int, dict[str, Any]]] = []
+ for index, item in enumerate(evidence_windows):
+ next_item = dict(item)
+ if index == 0:
+ next_item["evidence_unit_plan"] = plan
+ source_index = index + 1
+ unit_rank = selected_ranks.get(source_index)
+ next_item["evidence_unit_planner_enabled"] = True
+ next_item["evidence_unit_planner_selected"] = unit_rank is not None
+ next_item["evidence_unit_planner_rank"] = unit_rank if unit_rank is not None else 9999
+ group = 0 if unit_rank is not None else 1
+ rank = unit_rank if unit_rank is not None else index
+ planned.append((group, rank, index, next_item))
+ if not reorder_enabled:
+ return [item for _, _, _, item in sorted(planned, key=lambda row: row[2])]
+ planned.sort(key=lambda row: (row[0], row[1], row[2]))
+ return diversify_evidence_windows([item for _, _, _, item in planned])
+
+
+def build_evidence_unit_plan(question: str, evidence_windows: list[dict[str, Any]]) -> dict[str, Any]:
+ try:
+ max_candidates = max(1, int(os.getenv("TMCRA_EVIDENCE_UNIT_PLANNER_MAX_CANDIDATES", "10") or 10))
+ max_chars = max(240, int(os.getenv("TMCRA_EVIDENCE_UNIT_PLANNER_CHARS", "1100") or 1100))
+ max_tokens = max(280, int(os.getenv("TMCRA_EVIDENCE_UNIT_PLANNER_MAX_TOKENS", "760") or 760))
+ except (TypeError, ValueError):
+ max_candidates, max_chars, max_tokens = 10, 1100, 760
+ candidate_lines = []
+ for index, item in enumerate(evidence_windows[:max_candidates], start=1):
+ text = truncate(item.get("text", ""), max_chars)
+ if not text:
+ continue
+ memory_id = clean_text(item.get("memory_id", ""))
+ role = clean_text(item.get("evidence_role", ""))
+ temporal_state = clean_text(item.get("temporal_state", ""))
+ label_bits = []
+ if memory_id:
+ label_bits.append(f"memory_id={memory_id}")
+ if role:
+ label_bits.append(f"role={role}")
+ if temporal_state:
+ label_bits.append(f"time={temporal_state}")
+ label = " ".join(label_bits)
+ candidate_lines.append(f"[{index}] {label}\n{text}".strip())
+ if not candidate_lines:
+ return {}
+ messages = [
+ {
+ "role": "system",
+ "content": (
+ "You are a model-based evidence-unit planner for a long-memory runtime. "
+ "Do not answer from prior knowledge. Do not use domain-specific keyword rules. "
+ "Read the current question and the retrieved evidence windows, then normalize the evidence into answer units that a final answer model can reason over. "
+ "Your job is to expose the operation needed by the question: direct fact lookup, distinct-item counting, temporal difference, temporal disambiguation, current/latest value selection, negative/absence judgment, preference synthesis, or multi-evidence synthesis. "
+ "For count questions, infer the answer unit from the question and create one unit per distinct evidence-backed candidate. "
+ "When the question requests multiple actions joined by and/or, keep separate units for separate action-target instances, especially original-versus-replacement or return-versus-pick-up instances. "
+ "Do not collapse a stated pending need, intention, or obligation into mere context just because the same sentence later mentions an exchange, replacement, reschedule, or follow-up action. "
+ "Treat the original action and the replacement/follow-up action as separate units unless the evidence explicitly says the original action was completed, cancelled, or no longer needed. "
+ "Do not mark a clue as a distractor solely because it appears in an assistant turn; assistant turns can preserve user-specific obligations, summaries, answers, and confirmations when they refer to the surrounding dialogue. "
+ "If an assistant turn says the user needs to do an action, or summarizes a user obligation, keep it as a candidate evidence unit unless contradicted by user evidence. "
+ "For temporal questions, identify the event date or relative time anchor, the query-time anchor if available, and which event should be compared or excluded. "
+ "Use session dates and relative-time phrases from the evidence window text when present; connect nearby event descriptions to the session date when the dialogue makes that local anchor clear. "
+ "For month-level questions, if a past-tense personal event appears inside a dated session and no older or future date is stated, treat it as compatible with that session month; mark the time_anchor as inferred from session date rather than excluding it. "
+ "For negative or absence questions, first identify the relevant event/time bucket, then state whether the requested attribute is present, absent, or contradicted in that event. "
+ "Keep weak but potentially useful clues; mark distractors separately instead of deleting them. "
+ "Return strict JSON only with these keys: question_task, answer_unit, required_operation, evidence_units, plan_steps, candidate_answer, uncertainty. "
+ "evidence_units must be a list of objects with keys: source_index, role, unit_key, value, action, target, instance, time_anchor, polarity, relation_to_question. "
+ "Allowed role values: answer_unit, positive_evidence, temporal_anchor, current_value, old_value, constraint, negative_evidence, distractor, context. "
+ "source_index must be the 1-based evidence window index. candidate_answer may be empty when computation should be left to the answer model."
+ ),
+ },
+ {
+ "role": "user",
+ "content": f"Question:\n{question}\n\nRetrieved evidence windows:\n" + "\n\n".join(candidate_lines),
+ },
+ ]
+ base_url, model, api_key = evidence_unit_planner_config()
+ try:
+ raw = chat_completion(
+ base_url,
+ model,
+ messages,
+ max_tokens=max_tokens,
+ temperature=0.0,
+ api_key=api_key,
+ )
+ except Exception as exc:
+ return {
+ "enabled": False,
+ "error": f"{exc.__class__.__name__}: {str(exc)[:240]}",
+ }
+ payload = parse_json_object(raw)
+ if not payload:
+ return {
+ "enabled": False,
+ "error": "planner_returned_non_json",
+ "raw": truncate(raw, 500),
+ }
+ return sanitize_evidence_unit_plan(payload, max_source_index=len(candidate_lines))
+
+
+def sanitize_evidence_unit_plan(payload: Mapping[str, Any], *, max_source_index: int) -> dict[str, Any]:
+ allowed_roles = {
+ "answer_unit",
+ "positive_evidence",
+ "temporal_anchor",
+ "current_value",
+ "old_value",
+ "constraint",
+ "negative_evidence",
+ "distractor",
+ "context",
+ }
+ plan: dict[str, Any] = {
+ "enabled": True,
+ "question_task": truncate(payload.get("question_task", ""), 80),
+ "answer_unit": truncate(payload.get("answer_unit", ""), 120),
+ "required_operation": truncate(payload.get("required_operation", ""), 180),
+ "evidence_units": [],
+ "plan_steps": [],
+ "candidate_answer": truncate(payload.get("candidate_answer", ""), 160),
+ "uncertainty": truncate(payload.get("uncertainty", ""), 180),
+ }
+ raw_units = payload.get("evidence_units", [])
+ if isinstance(raw_units, list):
+ for raw_unit in raw_units[:24]:
+ if not isinstance(raw_unit, Mapping):
+ continue
+ try:
+ source_index = int(raw_unit.get("source_index", 0) or 0)
+ except (TypeError, ValueError):
+ source_index = 0
+ if source_index < 1 or source_index > max_source_index:
+ continue
+ role = clean_text(raw_unit.get("role", "context")).lower()
+ if role not in allowed_roles:
+ role = "context"
+ plan["evidence_units"].append(
+ {
+ "source_index": source_index,
+ "role": role,
+ "unit_key": truncate(raw_unit.get("unit_key", ""), 100),
+ "value": truncate(raw_unit.get("value", ""), 220),
+ "action": truncate(raw_unit.get("action", ""), 80),
+ "target": truncate(raw_unit.get("target", ""), 100),
+ "instance": truncate(raw_unit.get("instance", ""), 100),
+ "time_anchor": truncate(raw_unit.get("time_anchor", ""), 100),
+ "polarity": truncate(raw_unit.get("polarity", ""), 60),
+ "relation_to_question": truncate(raw_unit.get("relation_to_question", ""), 180),
+ }
+ )
+ raw_steps = payload.get("plan_steps", [])
+ if isinstance(raw_steps, list):
+ plan["plan_steps"] = [truncate(item, 180) for item in raw_steps[:8] if clean_text(item)]
+ return plan
+
+
+def evidence_unit_selected_source_ranks(plan: Mapping[str, Any]) -> dict[int, int]:
+ selected: dict[int, int] = {}
+ ignored_roles = {"distractor"}
+ for unit in list(plan.get("evidence_units") or []):
+ if not isinstance(unit, Mapping):
+ continue
+ role = clean_text(unit.get("role", "")).lower()
+ if role in ignored_roles:
+ continue
+ try:
+ source_index = int(unit.get("source_index", 0) or 0)
+ except (TypeError, ValueError):
+ continue
+ if source_index <= 0 or source_index in selected:
+ continue
+ selected[source_index] = len(selected)
+ return selected
+
+
+def evidence_unit_plan_from_windows(evidence_windows: list[dict[str, Any]]) -> dict[str, Any]:
+ for item in evidence_windows:
+ plan = item.get("evidence_unit_plan")
+ if isinstance(plan, dict) and plan:
+ return plan
+ return {}
+
+
+def _load_unified_operation_planner() -> tuple[Any, Any, Any] | None:
+ mode = os.getenv("TMCRA_UNIFIED_OPERATION_PLANNER_MODE", "").strip().lower()
+ if mode in {"", "off", "false", "0", "disabled", "none"}:
+ return None
+ model_path = os.getenv("TMCRA_UNIFIED_OPERATION_PLANNER_MODEL_PATH", "").strip()
+ if not model_path:
+ return None
+ cache_key = "|".join([model_path, os.getenv("TMCRA_UNIFIED_OPERATION_PLANNER_DEVICE", "cpu")])
+ cached = _UNIFIED_OPERATION_PLANNER_CACHE.get(cache_key)
+ if cached is not None:
+ return cached
+ try:
+ from experiments.replacement import unified_operation_planner as planner
+ except Exception:
+ return None
+ torch_module = getattr(planner, "torch", None)
+ if torch_module is None:
+ return None
+ try:
+ device = torch_module.device(os.getenv("TMCRA_UNIFIED_OPERATION_PLANNER_DEVICE", "cpu") or "cpu")
+ payload = torch_module.load(Path(model_path), map_location=device, weights_only=True)
+ config = planner.UnifiedPlannerConfig.from_dict(dict(payload.get("config", {}) or {}))
+ model = planner.UnifiedOperationPlannerModel(config).to(device)
+ state = dict(payload.get("model_state") or payload.get("state_dict") or {})
+ model.load_state_dict(state, strict=False)
+ model.eval()
+ except Exception:
+ return None
+ loaded = (planner, torch_module, model)
+ _UNIFIED_OPERATION_PLANNER_CACHE[cache_key] = loaded
+ return loaded
+
+
+def apply_unified_operation_planner(question: str, evidence_windows: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ loaded = _load_unified_operation_planner()
+ if loaded is None or not evidence_windows:
+ return evidence_windows
+ planner, torch_module, model = loaded
+ try:
+ max_windows = max(3, int(os.getenv("TMCRA_UNIFIED_OPERATION_PLANNER_MAX_WINDOWS", "16") or 16))
+ answer_threshold = float(os.getenv("TMCRA_UNIFIED_OPERATION_PLANNER_ANSWER_THRESHOLD", "0.52"))
+ except (TypeError, ValueError):
+ max_windows, answer_threshold = 16, 0.52
+ scoped = list(evidence_windows[:max_windows])
+ all_text = " ".join(clean_text(item.get("text", "")) for item in scoped).lower()
+ units = []
+ for index, item in enumerate(scoped, start=1):
+ text = clean_text(item.get("text", ""))
+ memory_id = clean_text(item.get("memory_id", "")) or f"window_{index}"
+ units.append(
+ {
+ "unit_id": f"u{index}",
+ "record_id": memory_id,
+ "session_id": memory_id.split(".turn_", 1)[0],
+ "turn_index": index,
+ "speaker": "unknown",
+ "text": text,
+ "timestamp": "",
+ "topic_bucket": clean_text(item.get("topic_bucket", "")),
+ "node_features": {
+ "retrieval_score": float(item.get("score", 0.0) or 0.0) / 10.0,
+ "graph_score": float(item.get("planner_score", 0.0) or item.get("answer_plan_score", 0.0) or 0.0),
+ "tunnel_score": float(item.get("path_score", 0.0) or 0.0),
+ },
+ "graph_neighbors": [],
+ }
+ )
+ row = {
+ "id": "runtime_unified_operation_planner",
+ "query": question,
+ "retrieval_metadata": {
+ "candidate_count": len(scoped),
+ "session_span": len({clean_text(item.get("memory_id", "")).split(".turn_", 1)[0] for item in scoped}),
+ "topic_span": len({clean_text(item.get("topic_bucket", "")) for item in scoped if clean_text(item.get("topic_bucket", ""))}),
+ "has_temporal_anchor": bool(re.search(r"\b(before|after|latest|current|previous|ago|week|month|year|date)\b", all_text)),
+ "has_numeric_units": bool(re.search(r"\b\d[\d,.]*\b|\$\s?\d|%", all_text)),
+ "has_profile_units": bool(re.search(r"\b(prefer|like|avoid|favorite|usually|always|never)\b", all_text)),
+ },
+ "memory_units": units,
+ "gold": {},
+ }
+ try:
+ dataset = planner.UnifiedOperationPlannerDataset([row], model.config)
+ batch = planner.collate_unified_planner_batch([dataset[0]])
+ device = next(model.parameters()).device
+ model_batch = {key: value.to(device) if hasattr(value, "to") else value for key, value in dict(batch).items()}
+ with torch_module.no_grad():
+ outputs = model(model_batch["features"], model_batch["valid_mask"])
+ relevance_scores = torch_module.sigmoid(outputs["relevance_logits"])[0].detach().cpu().tolist()
+ answer_scores = torch_module.sigmoid(outputs["answer_logits"])[0].detach().cpu().tolist()
+ temporal_scores = torch_module.sigmoid(outputs["temporal_logits"])[0].detach().cpu().tolist()
+ aggregation_scores = torch_module.sigmoid(outputs["aggregation_logits"])[0].detach().cpu().tolist()
+ current_value_scores = torch_module.sigmoid(outputs["current_value_logits"])[0].detach().cpu().tolist()
+ operation_scores = torch_module.sigmoid(outputs["operation_required_logits"])[0].detach().cpu().tolist()
+ operation_family_index = int(torch_module.argmax(outputs["operation_family_logits"], dim=-1)[0].detach().cpu().item())
+ except Exception:
+ return evidence_windows
+ operation_family = planner.OPERATION_FAMILIES[operation_family_index]
+ ranked: list[tuple[int, float, int, dict[str, Any]]] = []
+ for index, item in enumerate(evidence_windows):
+ next_item = dict(item)
+ if index < len(scoped):
+ relevance_score = float(relevance_scores[index])
+ answer_score = float(answer_scores[index])
+ temporal_score = float(temporal_scores[index])
+ aggregation_score = float(aggregation_scores[index])
+ current_value_score = float(current_value_scores[index])
+ unified_score = max(answer_score, temporal_score, aggregation_score, current_value_score, relevance_score * 0.8)
+ next_item["unified_operation_planner"] = True
+ next_item["unified_operation_family"] = operation_family
+ next_item["unified_operation_required_scores"] = {
+ "temporal": round(float(operation_scores[0]), 4),
+ "aggregation": round(float(operation_scores[1]), 4),
+ "profile": round(float(operation_scores[2]), 4),
+ "current_value": round(float(operation_scores[3]), 4),
+ "multi_hop": round(float(operation_scores[4]), 4),
+ }
+ next_item["unified_scores"] = {
+ "relevance": round(relevance_score, 4),
+ "answer": round(answer_score, 4),
+ "temporal": round(temporal_score, 4),
+ "aggregation": round(aggregation_score, 4),
+ "current_value": round(current_value_score, 4),
+ "combined": round(unified_score, 4),
+ }
+ next_item["unified_selected"] = bool(answer_score >= answer_threshold or unified_score >= max(answer_threshold, 0.58))
+ old_planner_score = float(next_item.get("planner_score", 0.0) or next_item.get("answer_plan_score", 0.0) or 0.0)
+ next_item["planner_score"] = max(old_planner_score, unified_score)
+ group = 0 if next_item["unified_selected"] else 1
+ ranked.append((group, -unified_score, index, next_item))
+ else:
+ ranked.append((2, 0.0, index, next_item))
+ ranked.sort(key=lambda row: (row[0], row[1], row[2]))
+ return [item for _, _, _, item in ranked]
+
+
+def format_evidence_unit_plan_for_prompt(plan: Mapping[str, Any]) -> str:
+ if not plan or not bool(plan.get("enabled", False)):
+ return ""
+ compact = {
+ "question_task": plan.get("question_task", ""),
+ "answer_unit": plan.get("answer_unit", ""),
+ "required_operation": plan.get("required_operation", ""),
+ "evidence_units": list(plan.get("evidence_units") or [])[:16],
+ "plan_steps": list(plan.get("plan_steps") or [])[:6],
+ "candidate_answer": plan.get("candidate_answer", ""),
+ "uncertainty": plan.get("uncertainty", ""),
+ }
+ return json.dumps(compact, ensure_ascii=False, separators=(",", ":"))
+
+
+def focused_evidence_unit_lines(
+ question: str,
+ evidence_windows: list[dict[str, Any]],
+ *,
+ max_items: int | None = None,
+ max_chars: int | None = None,
+) -> list[str]:
+ mode = os.getenv("TMCRA_FOCUSED_EVIDENCE_UNIT_MODE", "").strip().lower()
+ if mode in {"", "off", "0", "false", "disabled", "none"}:
+ return []
+ try:
+ item_limit = max(1, int(os.getenv("TMCRA_FOCUSED_EVIDENCE_UNIT_LIMIT", "18") if max_items is None else max_items))
+ char_limit = max(120, int(os.getenv("TMCRA_FOCUSED_EVIDENCE_UNIT_CHARS", "420") if max_chars is None else max_chars))
+ per_window = max(1, int(os.getenv("TMCRA_FOCUSED_EVIDENCE_UNITS_PER_WINDOW", "2") or 2))
+ except (TypeError, ValueError):
+ item_limit, char_limit, per_window = 18, 420, 2
+ question_terms = evidence_query_terms(question)
+ lines: list[str] = []
+ seen: set[str] = set()
+ for window_index, item in enumerate(evidence_windows, start=1):
+ text = clean_text(item.get("text", ""))
+ if not text:
+ continue
+ units = turn_units(text) or [text]
+ scored_units: list[tuple[float, int, str]] = []
+ for unit_index, unit in enumerate(units):
+ unit_text = clean_text(unit)
+ if not unit_text:
+ continue
+ score = score_evidence_unit(question_terms, question, unit_text)
+ if bool(item.get("planner_selected", False)) or bool(item.get("evidence_unit_planner_selected", False)):
+ score += 0.4
+ if bool(item.get("semantic_coverage_expansion", False)):
+ score += 0.25
+ if score <= 0 and len(units) > 1:
+ continue
+ scored_units.append((float(score), unit_index, unit_text))
+ if not scored_units:
+ scored_units = [(0.0, 0, text)]
+ scored_units.sort(key=lambda row: (-row[0], row[1]))
+ added_for_window = 0
+ for _, unit_index, unit_text in scored_units:
+ snippet = centered_excerpt(unit_text, question_terms, max_chars=char_limit)
+ key = re.sub(r"\W+", " ", snippet.lower()).strip()
+ if not key or key in seen:
+ continue
+ seen.add(key)
+ memory_id = clean_text(item.get("memory_id", ""))
+ source = f"source={window_index}"
+ if memory_id:
+ source += f" memory_id={memory_id}"
+ lines.append(f"{len(lines) + 1}. [{source} unit={unit_index}] {snippet}")
+ added_for_window += 1
+ if len(lines) >= item_limit or added_for_window >= per_window:
+ break
+ if len(lines) >= item_limit:
+ break
+ return lines
+
+
+def diversify_evidence_windows(evidence_windows: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ mode = os.getenv("TMCRA_ANSWER_EVIDENCE_DIVERSIFY_MEMORY_IDS", "1").strip().lower()
+ if mode in {"", "0", "false", "off", "no"} or len(evidence_windows) <= 1:
+ return evidence_windows
+ first_pass: list[dict[str, Any]] = []
+ overflow: list[dict[str, Any]] = []
+ seen: set[str] = set()
+ for item in evidence_windows:
+ memory_id = clean_text(item.get("memory_id", ""))
+ if not memory_id:
+ first_pass.append(item)
+ continue
+ if memory_id in seen:
+ overflow.append(item)
+ continue
+ seen.add(memory_id)
+ first_pass.append(item)
+ return [*first_pass, *overflow]
+
+
+def final_answer_channel_intent(question: str) -> dict[str, bool]:
+ terms = evidence_query_terms(question)
+ question_l = question.lower()
+ aggregation = bool(
+ terms
+ & {
+ "amount",
+ "count",
+ "different",
+ "how",
+ "many",
+ "minimum",
+ "number",
+ "percent",
+ "percentage",
+ "sum",
+ "total",
+ }
+ ) or bool(re.search(r"\b(?:how many|how much|total|sum|minimum|maximum|combined|different)\b", question_l))
+ temporal = bool(
+ re.search(
+ r"\b(?:before|after|first|last|earlier|later|weeks?|days?|months?|years?|when|date|how long)\b",
+ question_l,
+ )
+ )
+ assistant_detail = assistant_memory_query(question)
+ return {
+ "aggregation": aggregation,
+ "temporal": temporal,
+ "assistant_detail": assistant_detail,
+ }
+
+
+def final_answer_surface_windows(question: str, evidence_windows: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ mode = os.getenv("TMCRA_FINAL_ANSWER_SURFACE_CHANNEL_MODE", "on").strip().lower()
+ if mode in {"", "0", "false", "off", "no", "disabled", "none"} or not evidence_windows:
+ return evidence_windows
+ try:
+ final_limit = max(4, int(os.getenv("TMCRA_ANSWER_EVIDENCE_WINDOW_LIMIT", "8") or 8))
+ except (TypeError, ValueError):
+ final_limit = 8
+ intent = final_answer_channel_intent(question)
+ if intent["aggregation"]:
+ main_hard_limit = max(3, int(os.getenv("TMCRA_FINAL_MAIN_HARD_LIMIT_AGG", "5") or 5))
+ coverage_hard_limit = max(1, int(os.getenv("TMCRA_FINAL_COVERAGE_HARD_LIMIT_AGG", "3") or 3))
+ elif intent["temporal"] or intent["assistant_detail"]:
+ main_hard_limit = max(4, int(os.getenv("TMCRA_FINAL_MAIN_HARD_LIMIT_DETAIL", "7") or 7))
+ coverage_hard_limit = max(0, int(os.getenv("TMCRA_FINAL_COVERAGE_HARD_LIMIT_DETAIL", "1") or 1))
+ else:
+ main_hard_limit = max(4, int(os.getenv("TMCRA_FINAL_MAIN_HARD_LIMIT_DEFAULT", "6") or 6))
+ coverage_hard_limit = max(0, int(os.getenv("TMCRA_FINAL_COVERAGE_HARD_LIMIT_DEFAULT", "2") or 2))
+ coverage_hard_limit = min(coverage_hard_limit, max(0, final_limit - 1))
+ main_hard_limit = min(main_hard_limit, final_limit - coverage_hard_limit)
+
+ def is_coverage(item: dict[str, Any]) -> bool:
+ return bool(item.get("multi_unit_chain_slot", False)) or bool(item.get("unit_coverage_pack", False))
+
+ def is_assistant_candidate(item: dict[str, Any]) -> bool:
+ return bool(item.get("assistant_candidate_side_channel", False)) or clean_text(item.get("evidence_role", "")).lower() == "assistant_answer_candidate"
+
+ def is_assistant(item: dict[str, Any]) -> bool:
+ if is_assistant_candidate(item):
+ return False
+ return bool(item.get("assistant_origin_evidence", False)) or clean_text(item.get("evidence_role", "")).lower() == "assistant_origin_detail"
+
+ def key(item: dict[str, Any]) -> tuple[int, int, float, float, int]:
+ role = clean_text(item.get("evidence_role", "")).lower()
+ role_allows = int(role not in {"noise", "negative_evidence"})
+ selected = int(bool(item.get("planner_selected", False)) or bool(item.get("evidence_unit_planner_selected", False)))
+ planner_score = max(float(item.get("planner_score", 0.0) or 0.0), 0.0)
+ return (
+ selected,
+ role_allows,
+ planner_score,
+ float(item.get("score", 0.0) or 0.0),
+ len(clean_text(item.get("text", ""))),
+ )
+
+ assistant_candidates = [item for item in evidence_windows if is_assistant_candidate(item)]
+ assistant = [item for item in evidence_windows if is_assistant(item)]
+ main = [item for item in evidence_windows if not is_coverage(item) and not is_assistant(item) and not is_assistant_candidate(item)]
+ coverage = [item for item in evidence_windows if is_coverage(item)]
+ assistant_candidates.sort(key=key, reverse=True)
+ assistant.sort(key=key, reverse=True)
+ main.sort(key=key, reverse=True)
+ coverage.sort(key=key, reverse=True)
+ top_assistant_memory_ids = {
+ clean_text(item.get("memory_id", ""))
+ for item in assistant[: max(1, int(os.getenv("TMCRA_FINAL_ASSISTANT_HARD_LIMIT", "4") or 4))]
+ if clean_text(item.get("memory_id", ""))
+ }
+ if top_assistant_memory_ids:
+ assistant_candidates.sort(
+ key=lambda item: (
+ int(clean_text(item.get("memory_id", "")) in top_assistant_memory_ids),
+ *key(item),
+ ),
+ reverse=True,
+ )
+
+ ordered: list[dict[str, Any]] = []
+ seen: set[str] = set()
+
+ def add(items: list[dict[str, Any]], limit: int) -> None:
+ added = 0
+ for item in items:
+ if len(ordered) >= final_limit or added >= limit:
+ break
+ memory_id = clean_text(item.get("memory_id", ""))
+ text_key = clean_text(item.get("text", ""))[:240].lower()
+ dedupe_key = text_key if is_assistant_candidate(item) else memory_id or text_key
+ if dedupe_key and dedupe_key in seen:
+ continue
+ if dedupe_key:
+ seen.add(dedupe_key)
+ next_item = dict(item)
+ if is_assistant_candidate(item):
+ channel = "assistant_side"
+ elif is_assistant(item):
+ channel = "assistant"
+ elif is_coverage(item):
+ channel = "coverage"
+ else:
+ channel = "main"
+ next_item["final_answer_channel"] = channel
+ next_item["final_answer_channel_intent"] = dict(intent)
+ ordered.append(next_item)
+ added += 1
+
+ if intent["assistant_detail"]:
+ assistant_limit = max(1, int(os.getenv("TMCRA_FINAL_ASSISTANT_HARD_LIMIT", "4") or 4))
+ add(assistant, min(assistant_limit, final_limit))
+ try:
+ assistant_side_limit = max(0, int(os.getenv("TMCRA_FINAL_ASSISTANT_SIDE_LIMIT", "1") or 1))
+ except (TypeError, ValueError):
+ assistant_side_limit = 1
+ if assistant_side_limit:
+ add(assistant_candidates, assistant_side_limit)
+ add(main, main_hard_limit)
+ add(coverage, coverage_hard_limit)
+ add(assistant, final_limit)
+ add(assistant_candidates, final_limit)
+ add(main, final_limit)
+ add(coverage, final_limit)
+ return ordered[:final_limit]
+
+
+def _llm_channel_planner_enabled() -> bool:
+ mode = os.getenv("TMCRA_LLM_CHANNEL_PLANNER_MODE", "").strip().lower()
+ return mode not in {"", "0", "false", "off", "no", "disabled", "none"}
+
+
+def _int_list_from_payload(value: Any, *, upper_bound: int) -> list[int]:
+ values = iter_selector_indices(value)
+ result: list[int] = []
+ seen: set[int] = set()
+ for raw in values:
+ try:
+ index = int(str(raw).strip())
+ except (TypeError, ValueError):
+ continue
+ if index < 1 or index > upper_bound or index in seen:
+ continue
+ seen.add(index)
+ result.append(index)
+ return result
+
+
+def apply_llm_channel_planner(question: str, evidence_windows: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ if not _llm_channel_planner_enabled() or not evidence_windows:
+ return evidence_windows
+ if any(bool(item.get("llm_channel_planner_enabled", False)) for item in evidence_windows):
+ return evidence_windows
+ try:
+ max_windows = max(4, int(os.getenv("TMCRA_LLM_CHANNEL_PLANNER_MAX_WINDOWS", "16") or 16))
+ max_chars = max(160, int(os.getenv("TMCRA_LLM_CHANNEL_PLANNER_WINDOW_CHARS", "520") or 520))
+ max_tokens = max(160, int(os.getenv("TMCRA_LLM_CHANNEL_PLANNER_MAX_TOKENS", "700") or 700))
+ except (TypeError, ValueError):
+ max_windows, max_chars, max_tokens = 16, 520, 700
+ scoped = list(evidence_windows[:max_windows])
+ if not scoped:
+ return evidence_windows
+
+ lines: list[str] = []
+ for index, item in enumerate(scoped, start=1):
+ text = truncate(item.get("text", ""), max_chars)
+ metadata = {
+ "memory_id": clean_text(item.get("memory_id", "")),
+ "score": round(float(item.get("score", 0.0) or 0.0), 4),
+ "planner_score": round(float(item.get("planner_score", 0.0) or 0.0), 4),
+ "evidence_role": clean_text(item.get("evidence_role", "")),
+ "temporal_state": clean_text(item.get("temporal_state", "")),
+ "unit_coverage": bool(item.get("unit_coverage_pack", False)),
+ "multi_chain": bool(item.get("multi_unit_chain_slot", False)),
+ "profile": bool(item.get("profile_shadow_unit", False) or item.get("profile_layer", False)),
+ "unified_family": clean_text(item.get("unified_operation_family", "")),
+ }
+ lines.append(f"{index}. metadata={json.dumps(metadata, ensure_ascii=False, separators=(',', ':'))}\n{text}")
+
+ system_prompt = (
+ "You are an evidence-channel planner for a long-memory QA system. "
+ "Do not answer the user question. Select evidence windows only. "
+ "Return one strict JSON object and no markdown."
+ )
+ user_prompt = (
+ "Plan the final evidence channels for the answer model.\n"
+ "Rules:\n"
+ "- main_indices must keep direct facts, stable base evidence, temporal anchors, assistant details, and profile facts.\n"
+ "- coverage_indices are for count, sum, ratio, duration, repeated actions, multi-unit coverage, or unit-to-unit chains.\n"
+ "- coverage_indices must be evidence-backed units about the user's actual stated actions, quantities, states, or commitments.\n"
+ "- Generic assistant advice, examples, defaults, or hypothetical recommendations must not be coverage unless the user explicitly adopted or confirmed them.\n"
+ "- coverage evidence supplements main evidence; do not let coverage replace main facts.\n"
+ "- support_indices are useful context only.\n"
+ "- suppress_indices are duplicates, stale/conflicting values, or noise.\n"
+ "- Use 1-based indices from the candidate list.\n"
+ "- If unsure, keep a window in main or support instead of suppressing it.\n\n"
+ "Return JSON schema:\n"
+ "{\"operation_family\":\"direct|temporal|count|sum|ratio|profile|current_value|synthesis|unknown\","
+ "\"main_indices\":[1],\"coverage_indices\":[],\"support_indices\":[],\"suppress_indices\":[],"
+ "\"coverage_complete\":true,\"missing_evidence\":\"\",\"reasoning_policy\":\"short label\"}\n\n"
+ f"Question:\n{question}\n\n"
+ "Candidate evidence windows:\n"
+ + "\n\n".join(lines)
+ )
+ try:
+ base_url, model, api_key = llm_channel_planner_config()
+ raw = chat_completion(
+ base_url,
+ model,
+ [{"role": "system", "content": system_prompt}, {"role": "user", "content": user_prompt}],
+ max_tokens=max_tokens,
+ temperature=0.0,
+ api_key=api_key,
+ )
+ payload = parse_json_object(raw)
+ except Exception as exc:
+ annotated = [dict(item) for item in evidence_windows]
+ if annotated:
+ annotated[0]["llm_channel_planner_error"] = f"{exc.__class__.__name__}:{str(exc)[:160]}"
+ return annotated
+ if not payload:
+ return evidence_windows
+
+ main_indices = _int_list_from_payload(payload.get("main_indices", []), upper_bound=len(scoped))
+ coverage_indices = _int_list_from_payload(payload.get("coverage_indices", []), upper_bound=len(scoped))
+ support_indices = _int_list_from_payload(payload.get("support_indices", []), upper_bound=len(scoped))
+ suppress_indices = _int_list_from_payload(payload.get("suppress_indices", []), upper_bound=len(scoped))
+ channels: dict[int, str] = {}
+ for channel, indices in (
+ ("main", main_indices),
+ ("coverage", coverage_indices),
+ ("support", support_indices),
+ ("suppress", suppress_indices),
+ ):
+ for index in indices:
+ channels.setdefault(index - 1, channel)
+
+ if not any(channel in {"main", "coverage", "support"} for channel in channels.values()):
+ return evidence_windows
+
+ channel_rank = {"main": 0, "coverage": 1, "support": 2, "unassigned": 3, "suppress": 4}
+ planned: list[tuple[int, int, dict[str, Any]]] = []
+ for index, item in enumerate(evidence_windows):
+ next_item = dict(item)
+ channel = channels.get(index, "unassigned") if index < len(scoped) else "unassigned"
+ next_item["llm_channel_planner_enabled"] = True
+ next_item["llm_channel"] = channel
+ next_item["llm_channel_operation_family"] = clean_text(payload.get("operation_family", ""))
+ next_item["llm_channel_coverage_complete"] = bool(payload.get("coverage_complete", False))
+ missing_evidence = clean_text(payload.get("missing_evidence", ""))
+ reasoning_policy = clean_text(payload.get("reasoning_policy", ""))
+ if missing_evidence:
+ next_item["llm_channel_missing_evidence"] = missing_evidence
+ if reasoning_policy:
+ next_item["llm_channel_reasoning_policy"] = reasoning_policy
+ planned.append((channel_rank.get(channel, 3), index, next_item))
+ planned.sort(key=lambda row: (row[0], row[1]))
+ return [item for _, _, item in planned]
+
+
+def iter_selector_indices(value: Any) -> list[Any]:
+ if isinstance(value, list):
+ flattened: list[Any] = []
+ for item in value:
+ flattened.extend(iter_selector_indices(item))
+ return flattened
+ return [value]
+
+
+def profile_facts_for_answer(memory_hits: list[Any], *, max_items: int = 8) -> list[str]:
+ facts: list[str] = []
+ seen: set[str] = set()
+ for hit in memory_hits[:12]:
+ metadata = dict(getattr(hit, "metadata", {}) or {})
+ source_kind = str(getattr(hit, "source_kind", "") or "")
+ is_profile_hit = (
+ bool(metadata.get("profile_first_hybrid_rescue"))
+ or bool(metadata.get("profile_protected_slot"))
+ or bool(metadata.get("profile_layer"))
+ or source_kind.startswith("public_dialog_profile")
+ or ".subject." in str(getattr(hit, "memory_id", "") or "")
+ or source_kind in {"public_dialog_preference", "public_dialog_goal", "public_dialog_constraint", "public_dialog_status"}
+ or str(getattr(hit, "category", "") or "").lower() in {"profile", "preference", "goal", "constraint", "status"}
+ )
+ if not is_profile_hit:
+ continue
+ profile_type = clean_text(metadata.get("profile_type", "")) or clean_text(getattr(hit, "category", ""))
+ domain = clean_text(metadata.get("profile_domain_label", "") or metadata.get("profile_domain", ""))
+ summary = clean_text(metadata.get("profile_summary", "") or metadata.get("profile_value", "") or getattr(hit, "value", ""))
+ if not summary:
+ continue
+ prefix = "profile"
+ if profile_type:
+ prefix = profile_type.replace("_", " ")
+ if domain:
+ prefix = f"{prefix} / {domain}"
+ fact = f"{prefix}: {summary}"
+ key = fact.lower()
+ if key in seen:
+ continue
+ seen.add(key)
+ facts.append(fact)
+ if len(facts) >= max_items:
+ break
+ return facts
+
+
+def complete_profile_answer_from_evidence(
+ question: str,
+ answer: str,
+ evidence_windows: list[dict[str, Any]],
+) -> str:
+ # TMCRA_LAYER_TAG: answer-side profile completion; preserves bound qualifiers from retrieved profile facts.
+ answer_clean = clean_text(answer)
+ if not answer_clean:
+ return answer
+ answer_l = answer_clean.lower()
+ question_l = clean_text(question).lower()
+ role_attribute_query = bool(re.search(r"\b(?:occupation|job|role|worked\s+as|work\s+as|career|position)\b", question_l))
+ if len(answer_l) < 4:
+ return answer
+ for item in evidence_windows[:10]:
+ if clean_text(item.get("evidence_role", "")).lower() != "profile_fact":
+ continue
+ fact = clean_text(item.get("text", ""))
+ if not fact or len(fact) > 320:
+ continue
+ fact_l = fact.lower()
+ if answer_l not in fact_l:
+ continue
+ candidates: list[str] = []
+ for pattern in (
+ r"\bas\s+(?:a|an|the)?\s*([^.;,\n]+)",
+ r"\bworked\s+as\s+(?:a|an|the)?\s*([^.;,\n]+)",
+ r"\bpreviously\s+(?:was|worked\s+as)\s+(?:a|an|the)?\s*([^.;,\n]+)",
+ r"\brole\s+as\s+(?:a|an|the)?\s*([^.;,\n]+)",
+ ):
+ match = re.search(pattern, fact, flags=re.IGNORECASE)
+ if match:
+ candidates.append(clean_text(match.group(1)))
+ if role_attribute_query and candidates:
+ return candidates[0].strip(" .")
+ for candidate in candidates:
+ candidate_l = candidate.lower()
+ if answer_l in candidate_l and len(candidate) > len(answer_clean) and len(candidate) <= 180:
+ return candidate.strip(" .")
+ return answer
+
+
+def is_chain_evidence_slot(item: dict[str, Any]) -> bool:
+ return bool(item.get("chain_evidence_organizer_slot", False)) or clean_text(item.get("evidence_channel", "")) == "chain_summary"
+
+
+def is_strong_chain_evidence_slot(item: dict[str, Any]) -> bool:
+ if not is_chain_evidence_slot(item):
+ return False
+ sufficiency = clean_text(item.get("chain_evidence_sufficiency", "")).lower()
+ candidate_answer = clean_text(item.get("chain_evidence_candidate_answer", ""))
+ if not sufficiency or not candidate_answer:
+ text = clean_text(item.get("text", ""))
+ match = re.search(r"\{.*\}", text, flags=re.DOTALL)
+ if match:
+ try:
+ payload = json.loads(match.group(0))
+ sufficiency = sufficiency or clean_text(payload.get("evidence_sufficiency", "")).lower()
+ candidate_answer = candidate_answer or clean_text(payload.get("candidate_answer", ""))
+ except Exception:
+ pass
+ return sufficiency == "sufficient" and bool(candidate_answer)
+
+
+def answer_question(question: str, memory_hits: list[Any], evidence_windows: list[dict[str, Any]] | None = None) -> str:
+ if evidence_windows is None:
+ evidence_windows = build_answer_evidence(question, memory_hits)
+ if not any(bool(item.get("window_planner_enabled", False)) for item in evidence_windows):
+ evidence_windows = apply_answer_window_planner(question, list(evidence_windows))
+ if not any(bool(item.get("llm_evidence_selector_enabled", False)) for item in evidence_windows):
+ evidence_windows = apply_llm_evidence_selector(question, list(evidence_windows))
+ if not evidence_unit_plan_from_windows(evidence_windows):
+ evidence_windows = apply_evidence_unit_planner(question, list(evidence_windows))
+ evidence_windows = apply_unified_operation_planner(question, list(evidence_windows))
+ evidence_windows = apply_llm_channel_planner(question, list(evidence_windows))
+ evidence_windows = final_answer_surface_windows(question, list(evidence_windows))
+ question_l = question.lower()
+ # TMCRA_LAYER_TAG: keyword-based personalized answer gate; replace with turn-intent/profile model.
+ personalized_request = any(
+ marker in question_l
+ for marker in (
+ "recommend",
+ "recommendation",
+ "suggest",
+ "accessories",
+ "my current",
+ "my setup",
+ "for me",
+ "preference",
+ "prefer",
+ "what should",
+ "should i",
+ "any tips",
+ "trouble with",
+ "serve",
+ "dinner",
+ "battery life",
+ )
+ )
+ force_answer_for_eval = os.getenv("TMCRA_LME_FORCE_ANSWER", "1").strip().lower() not in {"0", "false", "no"}
+ profile_facts = profile_facts_for_answer(memory_hits)
+ compact_hit_lines = compact_memory_hit_lines(memory_hits)
+ memory_lines = []
+ evidence_window_limit = int(os.getenv("TMCRA_ANSWER_EVIDENCE_WINDOW_LIMIT", "8"))
+ ranked_evidence_windows = list(evidence_windows[: max(1, evidence_window_limit)])
+ for index, item in enumerate(ranked_evidence_windows, start=1):
+ if is_chain_evidence_slot(item):
+ continue
+ value = clean_text(item.get("text", ""))
+ if value:
+ memory_id = clean_text(item.get("memory_id", ""))
+ prefix = f"{index}. "
+ if memory_id:
+ prefix += f"[{memory_id}] "
+ role_bits = []
+ evidence_role = clean_text(item.get("evidence_role", ""))
+ temporal_state = clean_text(item.get("temporal_state", ""))
+ if evidence_role:
+ role_bits.append(f"role={evidence_role}")
+ if temporal_state:
+ role_bits.append(f"time={temporal_state}")
+ if role_bits:
+ prefix += "[" + " ".join(role_bits) + "] "
+ memory_lines.append(prefix + value)
+ model_plan_lines = []
+ model_plan_has_temporal_units = False
+ model_plan_has_computed_temporal_answer = False
+ model_plan_has_computed_coverage_answer = False
+ model_plan_has_chain_evidence = False
+ seen_model_plan_ids: set[str] = set()
+ for item in evidence_windows:
+ if not (
+ bool(item.get("temporal_comparison_model_planner", False))
+ or bool(item.get("aggregation_unit_model_planner", False))
+ or bool(item.get("operation_planner", False))
+ or bool(item.get("unified_operation_planner", False))
+ or bool(item.get("temporal_operation_layer", False))
+ or is_strong_chain_evidence_slot(item)
+ ):
+ continue
+ value = clean_text(item.get("text", ""))
+ if not value:
+ continue
+ memory_id = clean_text(item.get("memory_id", "")) or f"model_plan_{len(model_plan_lines) + 1}"
+ if memory_id in seen_model_plan_ids:
+ continue
+ seen_model_plan_ids.add(memory_id)
+ if '"temporal_units":[{' in value:
+ model_plan_has_temporal_units = True
+ if re.search(r'"computed_temporal_answer":"[^"]+', value):
+ model_plan_has_computed_temporal_answer = True
+ if re.search(r'"computed_coverage_answer":\{"answer":"[^"]+', value):
+ model_plan_has_computed_coverage_answer = True
+ if is_strong_chain_evidence_slot(item):
+ model_plan_has_chain_evidence = True
+ model_plan_lines.append(f"- [{memory_id}] {value}")
+ if len(model_plan_lines) >= 3:
+ break
+ profile_block = "\n".join(f"- {fact}" for fact in profile_facts)
+ memory_block = "\n".join(memory_lines) if memory_lines else "(no relevant long-term memory retrieved)"
+ model_plan_section = (
+ "Model planner outputs:\n"
+ + "\n".join(model_plan_lines)
+ + "\n\n"
+ if model_plan_lines
+ else ""
+ )
+ evidence_unit_plan = evidence_unit_plan_from_windows(evidence_windows)
+ evidence_unit_plan_json = format_evidence_unit_plan_for_prompt(evidence_unit_plan)
+ evidence_unit_plan_section = (
+ f"Model evidence-unit plan:\n{evidence_unit_plan_json}\n\n" if evidence_unit_plan_json else ""
+ )
+ focused_unit_lines = focused_evidence_unit_lines(question, ranked_evidence_windows)
+ focused_unit_section = (
+ "Focused evidence units selected from the retrieved windows:\n"
+ + "\n".join(focused_unit_lines)
+ + "\n\n"
+ if focused_unit_lines
+ else ""
+ )
+ # TMCRA_LAYER_TAG: answer prompt protocol; affects answer utilization, not memory retrieval.
+ reasoning_protocol = (
+ "Treat TMCRA memory as evidence clues, not as a pre-written answer. "
+ "If model temporal, aggregation, or operation plans are provided, read them before the raw evidence because they contain model-selected answer events, unit coverage, time anchors, and comparison direction. "
+ "Use those plans as normalized evidence maps, but verify the answer against the raw memory evidence. "
+ "If a model evidence-unit plan is provided, use it as the primary normalized view of answer units, time anchors, and distinct clues, while verifying it against the raw evidence lines. "
+ "If focused evidence units are provided, scan them before the raw windows; they are compressed excerpts from retrieved memory, not external facts. "
+ "If the evidence-unit plan lists answer_unit, positive_evidence, current_value, old_value, temporal_anchor, or negative_evidence units, reason over those units explicitly before deciding the answer. "
+ "If an aggregation/unit coverage plan contains computed_coverage_answer.answer and its listed units_used or terms match the question, use that computed value unless raw evidence directly contradicts it. "
+ "If a TMCRA chain evidence slot is provided, read it as a chain-specific model plan before raw evidence. When evidence_sufficiency is sufficient and candidate_answer is present, use that candidate_answer unless the listed evidence units are contradicted by raw evidence. When evidence_sufficiency is partial or insufficient, use the listed units and missing_roles to decide whether direct evidence can still support an answer; do not treat a partial chain slot as a complete total. "
+ "If candidate_answer is present in the plan and is supported by the listed units, prefer it over re-summarizing the raw window text. "
+ "Before writing the JSON answer, internally infer the question task: direct fact lookup, arithmetic, distinct-item counting, temporal difference, current-value selection, preference/recommendation synthesis, or multi-evidence synthesis. "
+ "Then use the retrieved evidence as clues for that task. "
+ "For arithmetic questions, combine quantities found in different evidence lines and compute the requested value. "
+ "For count questions, infer the answer unit from the current question and count distinct evidence-backed answer units across relevant evidence, not just one sentence. "
+ "For offline benchmark count questions, include plausible evidence-backed units when the only uncertainty is an implicit date or an assistant-confirmed action and there is no contradictory evidence. "
+ "For month-level count questions, a past-tense user event inside a dated session is a usable candidate for that session month when no older or future date contradicts it. "
+ "When the question contains multiple actions joined by and/or, count each distinct target instance attached to any requested action. "
+ "Do not merge two answer units merely because they share the same broad noun; keep them separate when the evidence distinguishes their action, status, time, location, or original-versus-replacement instance. "
+ "For current/now/since/update questions, compare old and later evidence and choose the current or latest value when the question asks for it. "
+ "For advice, preference, dinner, shopping, setup, or tips questions, synthesize a useful answer from remembered personal setup, owned items, ingredients, goals, habits, and constraints; exact prior wording is not required. "
+ "Ignore irrelevant retrieved windows. "
+ "Only answer that information is unavailable when none of the evidence provides a usable clue for the requested task. "
+ )
+ temporal_plan_protocol = ""
+ if model_plan_has_temporal_units or model_plan_has_computed_temporal_answer:
+ temporal_plan_protocol = (
+ "When a model temporal plan explicitly contains non-empty temporal_units or computed_temporal_answer, use those fields as the time math layer. "
+ "For how-many-ago questions, use computed_temporal_answer or delta_to_query fields instead of copying local words such as today. "
+ "For first/order questions with explicit temporal_units, compare normalized_event_date values for the compared events. "
+ "Ignore this temporal-plan instruction when the temporal plan has no temporal_units and no computed_temporal_answer. "
+ )
+ coverage_plan_protocol = ""
+ if model_plan_has_computed_coverage_answer:
+ coverage_plan_protocol = (
+ "When a model aggregation/unit plan explicitly contains computed_coverage_answer, treat it as the unit coverage layer. "
+ "For count, total, sum, and arithmetic questions, prefer the computed answer when the listed units_used or terms are relevant evidence for the requested unit. "
+ "Do not drop a listed unit merely because its raw evidence window also contains unrelated topic text. "
+ )
+ chain_plan_protocol = ""
+ if model_plan_has_chain_evidence:
+ chain_plan_protocol = (
+ "When a TMCRA chain evidence slot is present in Model planner outputs, it is the normalized chain-evidence view for this question. "
+ "Use it to bind distinct units, temporal roles, and calculation direction before scanning raw evidence. "
+ "If the chain slot says evidence_sufficiency=sufficient and gives a candidate_answer, answer with that candidate unless raw evidence directly contradicts the listed units. "
+ "If a chain slot is partial or insufficient, do not invent missing units; combine only the listed units that raw evidence supports. "
+ )
+ personalized_instruction = ""
+ if personalized_request:
+ personalized_instruction = (
+ "For personalized recommendation or preference questions, synthesize a concise recommendation using the user's retrieved setup, preferences, goals, and constraints. "
+ "If a Structured personalized facts block is present, use it first; otherwise use the long-term memory evidence directly. "
+ "Do not abstain when the memory contains relevant personal setup or preference evidence. "
+ "Do not say the memory lacks the user's setup when the evidence mentions a current brand, device, activity, goal, style, constraint, or previous preference. "
+ "Exact prior accessory lists are not required; infer a compatible recommendation from the remembered setup or preference. "
+ + (
+ "For this offline benchmark, do not ask follow-up questions; a partial remembered setup is enough to produce the best available personalized answer. "
+ if force_answer_for_eval
+ else "In realtime product mode, you may ask a brief follow-up question after giving the best available memory-backed suggestion. "
+ )
+ + "The answer should mention the personalized basis when useful, such as compatibility, brand, quality, durability, or current gear. "
+ )
+ profile_section = f"Structured personalized facts:\n{profile_block}\n\n" if profile_block else ""
+ system_prompt = (
+ "You answer LongMemEval questions using only the current question and the provided long-term memory. "
+ "First identify the requested attribute, then answer only that attribute. "
+ + reasoning_protocol
+ + temporal_plan_protocol
+ + coverage_plan_protocol
+ + chain_plan_protocol
+ + "If the question has multiple parts, compares earlier versus now, or asks for both before/current values, answer every requested part explicitly. "
+ "If evidence lines are tagged with role=initial_value and role=current_value, use initial_value for the earlier/starting state and current_value for the now/current state; do not collapse them into one value. "
+ "For current or now values, prefer later-dated evidence over earlier evidence when both are present. "
+ "The answer value must not be an object, list, or multiple JSON keys. "
+ "For where questions, answer with the place, store, venue, retailer, or location. "
+ "Never answer a where question with a time or date such as last Sunday. "
+ "For retail or shopping questions, a nearby store, retailer, or app/store name is the location. "
+ "For how long questions, answer with the duration. "
+ "For name questions, answer with the name. "
+ "If the answer is implied by nearby turns in the same evidence window, use that local context. "
+ "For profile, identity, occupation, role, ownership, and preference facts, preserve attached qualifiers that are part of the remembered fact, such as company type, setting, object owner, condition, strength, or scope. "
+ "If a role=profile_fact evidence line or Structured personalized fact directly answers the requested attribute, preserve the complete specific fact rather than only the head noun or job title. "
+ "Do not shorten a complete remembered fact into only its broad category when the evidence gives a more specific attribute requested by the question. "
+ + personalized_instruction
+ + "Return strict JSON only, with exactly one key named answer. "
+ + "The answer value must be the shortest complete answer phrase or sentence. Do not explain. Do not cite evidence. "
+ + "Example: {\"answer\":\"45 minutes each way\"}\n\n"
+ + profile_section
+ + model_plan_section
+ + evidence_unit_plan_section
+ + focused_unit_section
+ + f"Long-term memory evidence:\n{memory_block}"
+ )
+ messages = [
+ {
+ "role": "system",
+ "content": system_prompt,
+ },
+ {"role": "user", "content": f"Question: {question}\nReturn only JSON."},
+ ]
+ answer_base_url, answer_model, answer_api_key = answer_llm_config()
+ answer = clean_answer_output(chat_completion(
+ answer_base_url,
+ answer_model,
+ messages,
+ max_tokens=int(os.getenv("TMCRA_ANSWER_MAX_TOKENS", "512")),
+ temperature=0.0,
+ api_key=answer_api_key,
+ ))
+ answer = complete_profile_answer_from_evidence(question, answer, evidence_windows)
+ organizer_uses_model_plan = bool(
+ re.search(
+ r"\bhow many\s+(?:days?|weeks?|months?|years?)\b.*\bago\b|"
+ r"\b(?:\d+|one|two|three|four|five|six|seven|eight|nine|ten|eleven|twelve)\s+"
+ r"(?:days?|weeks?|months?|years?)\s+ago\b",
+ question_l,
+ )
+ )
+ organizer_context_section = (
+ profile_section
+ + (model_plan_section if organizer_uses_model_plan else "")
+ + evidence_unit_plan_section
+ )
+ if memory_lines and should_run_answer_reasoning_organizer(
+ question,
+ answer,
+ personalized_request=personalized_request,
+ force_answer_for_eval=force_answer_for_eval,
+ ):
+ organized_answer = organize_answer_from_evidence(
+ question,
+ memory_lines,
+ profile_section=organizer_context_section,
+ force_answer_for_eval=force_answer_for_eval,
+ prior_answer=answer,
+ )
+ if (not organized_answer or looks_like_no_memory_answer(organized_answer)) and compact_hit_lines:
+ organized_answer = organize_answer_from_evidence(
+ question,
+ compact_hit_lines,
+ profile_section=organizer_context_section,
+ force_answer_for_eval=force_answer_for_eval,
+ prior_answer=answer,
+ )
+ if organized_answer and not looks_like_no_memory_answer(organized_answer):
+ return complete_profile_answer_from_evidence(question, organized_answer, evidence_windows)
+ focused_memory_block = "\n".join(memory_lines[:8])
+ # TMCRA_LAYER_TAG: answer-side fallback organizer for conservative/no-memory answers.
+ retry_messages = [
+ {
+ "role": "system",
+ "content": (
+ "You are correcting an over-conservative answer. "
+ "Your previous answer treated memory as a complete answer database; that is wrong. "
+ "The focused evidence below is non-empty and may contain indirect clues rather than an exact answer sentence. "
+ "You must first infer the task required by the question, then combine the evidence clues to answer it. "
+ "Use arithmetic for total/each/save questions, distinct counting for how-many questions, date comparison for temporal questions, current/latest evidence for now/current questions, and synthesis from remembered setup or preferences for advice/tips/recommendation questions. "
+ "For advice, tips, what-should-I, or recommendation questions, a remembered owned item, ingredient, activity, goal, preference, or constraint is enough to produce a useful answer; the memory does not need to contain the exact advice sentence. "
+ "For count questions, infer the answer unit from the current question and combine distinct evidence-backed units across the focused evidence instead of requiring a precomputed total. "
+ "If a model evidence-unit plan is provided, use its answer_unit and positive_evidence units as normalized candidates and only exclude units that are contradicted or outside the requested time/action scope. "
+ "For month-level count questions, do not discard a past-tense user event merely because only the session date provides the month anchor. "
+ "Do not merge units only because they share the same broad noun; keep distinct action targets or original/replacement instances separate when the evidence separates them. "
+ "Do not claim no memory if any focused evidence gives a usable clue for the requested task. "
+ "Do not answer with Information unavailable, not available, no information, or similar wording unless the focused evidence block is empty or entirely unrelated. "
+ "Ignore evidence that is unrelated to the requested task. "
+ + (
+ "For this offline benchmark, do not ask follow-up questions; give the best direct evidence-backed answer. "
+ if force_answer_for_eval
+ else "In realtime product mode, you may ask a brief follow-up question after giving the best memory-backed answer. "
+ )
+ +
+ "Return strict JSON only, with exactly one key named answer.\n\n"
+ + organizer_context_section
+ + f"Focused long-term memory evidence:\n{focused_memory_block}"
+ ),
+ },
+ {"role": "user", "content": f"Question: {question}\nReturn only JSON."},
+ ]
+ retry_answer = clean_answer_output(chat_completion(
+ answer_base_url,
+ answer_model,
+ retry_messages,
+ max_tokens=128,
+ temperature=0.0,
+ api_key=answer_api_key,
+ ))
+ if retry_answer and not looks_like_no_memory_answer(retry_answer):
+ return complete_profile_answer_from_evidence(question, retry_answer, evidence_windows)
+ return complete_profile_answer_from_evidence(question, answer, evidence_windows)
+
+
+def should_run_answer_reasoning_organizer(
+ question: str,
+ answer: str,
+ *,
+ personalized_request: bool,
+ force_answer_for_eval: bool,
+) -> bool:
+ # TMCRA_LAYER_TAG: rule gate for answer-side retry/organizer; not a trained routing model.
+ mode = os.getenv("TMCRA_ANSWER_REASONING_ORGANIZER", "1").strip().lower()
+ if mode in {"", "0", "false", "off", "disabled", "none"}:
+ return False
+ if looks_like_no_memory_answer(answer):
+ return True
+ if personalized_request and force_answer_for_eval and looks_like_followup_answer(answer):
+ return True
+ question_l = clean_text(question).lower()
+ count_markers = (
+ "how many",
+ "number of",
+ "count of",
+ "count the",
+ "count my",
+ "total number",
+ )
+ return any(marker in question_l for marker in count_markers)
+
+
+def organize_answer_from_evidence(
+ question: str,
+ memory_lines: list[str],
+ *,
+ profile_section: str = "",
+ force_answer_for_eval: bool = True,
+ prior_answer: str = "",
+) -> str:
+ if not memory_lines:
+ return ""
+ evidence_block = "\n".join(memory_lines)
+ prior_answer = clean_text(prior_answer)
+ prior_section = f"Previous candidate answer, possibly incomplete: {prior_answer}\n\n" if prior_answer else ""
+ messages = [
+ {
+ "role": "system",
+ "content": (
+ "You are the evidence organizer for a long-memory runtime. "
+ "The final answer model may have been too conservative or may have under-used the retrieved clues, so your job is to read the retrieved memory lines as clues and produce the best evidence-backed answer. "
+ "Do not treat memory as a database that must contain the exact answer sentence. "
+ "Ignore unrelated lines, but if any line contains a usable clue for the question, mark relevant=true. "
+ "For advice, tips, recommendations, dinner, shopping, setup, or preference questions, remembered owned items, tools, accessories, ingredients, goals, habits, constraints, or preferences are enough to synthesize an answer. "
+ "Infer the affordance of remembered items and ingredients: a relevant tool, accessory, setup, ingredient, preference, or constraint can answer the user's underlying need even when the old conversation asked a different surface question. "
+ "Do not answer the old conversation's surface task; adapt the remembered clue to the current question's need. "
+ "When advice uses a remembered owned item, phrase the answer as using or preparing that already-mentioned item and include one practical use step; do not turn it into a generic purchase suggestion unless the question asks what to buy. "
+ "If a model evidence-unit plan is provided, enumerate the plan's answer_unit and positive_evidence units first, then reconcile duplicates or exclusions against the raw evidence. "
+ "For count questions, infer the count unit from the current question itself, then enumerate all relevant candidate clues, merge duplicate mentions of the same clue, and count only evidence-backed units matching that inferred unit. "
+ "For month-level count questions, include past-tense user events from a dated session as candidates for that session month unless another date excludes them. "
+ "When the current question contains multiple actions joined by and/or, the count unit is each distinct target instance attached to any requested action, unless the question explicitly asks for object categories. "
+ "Do not merge two candidate units merely because they share the same broad noun; keep them separate when the evidence distinguishes their action, status, time, location, or original-versus-replacement instance. "
+ "If one memory line contains multiple distinct obligations or requested action targets, count them separately only when the inferred count unit requires action-level counting. "
+ "Do not count broad object categories, generic advice, examples, or unrelated context as answer units. "
+ "Do not stop after the first relevant clue; scan every retrieved line before producing the count. "
+ "For current/now questions, prefer later or current evidence over old evidence when both appear. "
+ "For temporal questions, compare dates or durations when evidence provides them. "
+ "Return strict JSON only with keys: relevant, task, clues, answer. "
+ "answer must be the final shortest useful answer, not an explanation. "
+ "If evidence is genuinely empty or unrelated, return relevant=false and answer=\"\". "
+ + (
+ "For this offline benchmark, do not ask follow-up questions; use the best available evidence-backed inference. "
+ if force_answer_for_eval
+ else "In realtime product mode, prefer a direct memory-backed answer; only ask follow-up if evidence is too thin. "
+ )
+ + "\n\n"
+ + profile_section
+ + prior_section
+ + f"Retrieved memory lines:\n{evidence_block}"
+ ),
+ },
+ {"role": "user", "content": f"Question: {question}\nReturn only JSON."},
+ ]
+ answer_base_url, answer_model, answer_api_key = answer_llm_config()
+ raw = chat_completion(
+ answer_base_url,
+ answer_model,
+ messages,
+ max_tokens=220,
+ temperature=0.0,
+ api_key=answer_api_key,
+ )
+ payload = parse_json_object(raw)
+ if not payload:
+ fallback = clean_answer_output(raw)
+ return "" if fallback.lstrip().startswith("{") else fallback
+ relevant = payload.get("relevant", False)
+ if isinstance(relevant, str):
+ relevant = relevant.strip().lower() in {"1", "true", "yes", "y"}
+ answer = clean_text(payload.get("answer", ""))
+ if not relevant or not answer:
+ return ""
+ return answer.strip(" \t\n\r\"'")
+
+
+def compact_memory_hit_lines(memory_hits: list[Any], *, max_items: int | None = None, max_chars: int | None = None) -> list[str]:
+ if max_items is None:
+ max_items = int(os.getenv("TMCRA_COMPACT_MEMORY_HIT_LIMIT", "8"))
+ if max_chars is None:
+ max_chars = int(os.getenv("TMCRA_COMPACT_MEMORY_HIT_CHARS", "260"))
+ lines: list[str] = []
+ seen: set[str] = set()
+ for index, hit in enumerate(memory_hits[: max(1, max_items)], start=1):
+ value = truncate(getattr(hit, "value", ""), max(80, max_chars))
+ if not value:
+ continue
+ memory_id = clean_text(getattr(hit, "memory_id", ""))
+ category = clean_text(getattr(hit, "category", ""))
+ source_kind = clean_text(getattr(hit, "source_kind", ""))
+ label_bits = []
+ if category:
+ label_bits.append(category)
+ if source_kind:
+ label_bits.append(source_kind)
+ label = " / ".join(label_bits)
+ prefix = f"{index}. "
+ if memory_id:
+ prefix += f"[{memory_id}] "
+ if label:
+ prefix += f"({label}) "
+ line = prefix + value
+ key = re.sub(r"\s+", " ", line.lower())
+ if key in seen:
+ continue
+ seen.add(key)
+ lines.append(line)
+ return lines
+
+
+def parse_json_object(text: str) -> dict[str, Any]:
+ text = clean_text(text)
+ text = re.sub(r"^```(?:\w+)?\s*|\s*```$", "", text).strip()
+ decoder = json.JSONDecoder()
+ for index, char in enumerate(text):
+ if char != "{":
+ continue
+ try:
+ payload, _ = decoder.raw_decode(text[index:])
+ except Exception:
+ continue
+ return payload if isinstance(payload, dict) else {}
+ return {}
+
+
+def looks_like_no_memory_answer(answer: str) -> bool:
+ # TMCRA_LAYER_TAG: string-marker abstention detector used to trigger answer-side retry.
+ value = clean_text(answer).lower()
+ if not value:
+ return True
+ no_memory_markers = (
+ "no information",
+ "no relevant",
+ "not contain",
+ "does not contain",
+ "doesn't contain",
+ "do not have",
+ "don't have",
+ "cannot determine",
+ "can't determine",
+ "not enough information",
+ "insufficient information",
+ "information unavailable",
+ "not available",
+ "unavailable",
+ "not mention",
+ "not mentioned",
+ "unknown",
+ )
+ return any(marker in value for marker in no_memory_markers)
+
+
+def looks_like_followup_answer(answer: str) -> bool:
+ # TMCRA_LAYER_TAG: string-marker follow-up detector used by offline benchmark force-answer mode.
+ value = clean_text(answer).lower()
+ if not value:
+ return False
+ followup_markers = (
+ "could you tell me",
+ "can you tell me",
+ "i need more information",
+ "need more information",
+ "tell me more",
+ "what is your budget",
+ "what's your budget",
+ "what kind of",
+ "would you like",
+ "are you looking",
+ )
+ return any(marker in value for marker in followup_markers) or value.endswith("?")
+
+
+def clean_answer_output(text: str) -> str:
+ text = clean_text(text)
+ text = re.sub(r"^```(?:\w+)?\s*|\s*```$", "", text).strip()
+ json_match = re.search(r"\{.*?\}", text, flags=re.DOTALL)
+ if json_match:
+ try:
+ payload = json.loads(json_match.group(0))
+ answer = clean_text(payload.get("answer", ""))
+ if answer:
+ return answer.strip(" \t\n\r\"'")
+ except Exception:
+ pass
+ tail_markers = [
+ "Referencing the",
+ "Based on the",
+ "While the",
+ "While ",
+ "Although ",
+ "However,",
+ "There is no mention",
+ "There is no information",
+ "Give a direct",
+ "Do not mention",
+ "The provided context",
+ "In the provided",
+ ]
+ for marker in tail_markers:
+ index = text.find(marker)
+ if index > 0:
+ text = text[:index].strip()
+ return text.strip(" \t\n\r\"'")
+
+
+def run(args: argparse.Namespace) -> None:
+ service_root = Path(args.service_root).resolve()
+ repo = Path(args.repo).resolve()
+ apply_env_defaults(service_root)
+ os.environ.setdefault("GEMMA_BASE_URL", f"http://{os.getenv('TMCRA_GEMMA_HOST', '127.0.0.1')}:{os.getenv('TMCRA_GEMMA_PORT', '18002')}/v1")
+ os.environ.setdefault("GEMMA_MODEL", os.getenv("TMCRA_GEMMA_MODEL_NAME", "gemma-4-e4b-it"))
+ if str(repo) not in sys.path:
+ sys.path.insert(0, str(repo))
+ if not args.enable_topic_buckets:
+ disable_topic_bucket_runtime()
+
+ started = time.time()
+ out_dir = Path(args.out) if args.out else DEFAULT_OUT_ROOT / f"lme_s10_native_{datetime.now().strftime('%Y%m%d_%H%M%S')}"
+ out_dir.mkdir(parents=True, exist_ok=True)
+ predictions_path = out_dir / "predictions.jsonl"
+ debug_path = out_dir / "samples_debug.jsonl"
+ summary_path = out_dir / "summary.json"
+ storage_path = out_dir / "native_memory.sqlite3"
+
+ writer = build_writer()
+ records: list[dict[str, Any]] = []
+ log("run_start", data=str(args.data), limit=args.limit, out=str(out_dir), repo=str(repo))
+ for sample_index, row in enumerate(iter_json_array(Path(args.data), limit=args.limit), start=1):
+ qid = clean_text(row.get("question_id")) or f"sample_{sample_index:04d}"
+ question = clean_text(row.get("question"))
+ gold = clean_text(row.get("answer"))
+ sessions = list(row.get("haystack_sessions") or [])
+ session_ids = [clean_text(item) for item in list(row.get("haystack_session_ids") or [])]
+ dates = [clean_text(item) for item in list(row.get("haystack_dates") or [])]
+ selected_indices = (
+ select_official_full_history_indices(row)
+ if args.official_full_history
+ else select_session_indices(row, max_distractors=args.max_distractor_sessions)
+ )
+ adapter = build_adapter(f"longmemeval_native:{qid}:{sample_index}", storage_path)
+ sample_start = time.perf_counter()
+ writer_calls = 0
+ writer_accepted = 0
+ writer_suspected = 0
+ writer_records = 0
+ writer_seconds = 0.0
+ writer_unit_records = 0
+ writer_unit_calls = 0
+ chunk_errors: list[str] = []
+ selected_session_ids = []
+
+ for session_index in selected_indices:
+ if session_index >= len(sessions):
+ continue
+ sid = session_ids[session_index] if session_index < len(session_ids) else f"session_{session_index}"
+ selected_session_ids.append(sid)
+ date = dates[session_index] if session_index < len(dates) else ""
+ is_answer_session = sid in {clean_text(item) for item in list(row.get("answer_session_ids") or [])}
+ max_chunks = (
+ args.max_session_chunks
+ if args.official_full_history
+ else args.max_answer_chunks if is_answer_session else args.max_distractor_chunks
+ )
+ chunks = session_text_chunks(
+ session_id=sid,
+ date=date,
+ turns=list(sessions[session_index] or []),
+ max_chars=args.chunk_chars,
+ max_chunks=max_chunks,
+ )
+ for chunk_index, chunk in enumerate(chunks, start=1):
+ chunk_id = f"s{session_index:03d}_c{chunk_index:02d}"
+ try:
+ result = writer_ingest(adapter, writer, chunk, qid=qid, chunk_id=chunk_id)
+ writer_calls += 1
+ writer_accepted += result["accepted_count"]
+ writer_suspected += result["suspected_count"]
+ writer_records += result["record_count"]
+ writer_seconds += float(result["seconds"])
+ writer_unit_records += int(result.get("unit_record_count", 0) or 0)
+ if bool(result.get("unit_writer_enabled", False)):
+ writer_unit_calls += 1
+ except Exception as exc:
+ chunk_errors.append(f"{chunk_id}:{exc.__class__.__name__}:{str(exc)[:200]}")
+ log("chunk_error", sample=sample_index, qid=qid, chunk=chunk_id, error=str(exc)[:240])
+
+ if getattr(args, "write_only", False):
+ graph = getattr(adapter, "graph", None)
+ record = {
+ "sample_index": sample_index,
+ "question_id": qid,
+ "question_type": row.get("question_type"),
+ "question": question,
+ "question_date": clean_text(row.get("question_date", "")),
+ "gold_answer": gold,
+ "hypothesis": "",
+ "answer_session_ids": list(row.get("answer_session_ids") or []),
+ "selected_session_ids": selected_session_ids,
+ "total_sessions": len(sessions),
+ "total_turns": sum(len(s or []) for s in sessions),
+ "selected_sessions": len(selected_session_ids),
+ "history_mode": "official_full_history" if args.official_full_history else "controlled_answer_plus_distractors",
+ "write_only": True,
+ "writer_calls": writer_calls,
+ "writer_accepted": writer_accepted,
+ "writer_suspected": writer_suspected,
+ "writer_records": writer_records,
+ "writer_unit_records": writer_unit_records,
+ "writer_unit_calls": writer_unit_calls,
+ "writer_seconds": round(writer_seconds, 3),
+ "retrieve_seconds_wall": 0.0,
+ "answer_seconds": 0.0,
+ "sample_seconds": round(time.perf_counter() - sample_start, 3),
+ "graph_records": len(getattr(graph, "records_by_id", {}) or {}),
+ "graph_edges": len(getattr(graph, "edges", []) or []),
+ "chunk_errors": chunk_errors,
+ "retrieval": {"hit_count": 0, "selected_event_count": 0, "selected_path_count": 0, "hits": []},
+ "evidence_unit_plan": {},
+ "answer_evidence_windows": [],
+ }
+ records.append(record)
+ with predictions_path.open("a", encoding="utf-8") as handle:
+ handle.write(json.dumps({"question_id": qid, "hypothesis": ""}, ensure_ascii=False) + "\n")
+ with debug_path.open("a", encoding="utf-8") as handle:
+ handle.write(json.dumps(record, ensure_ascii=False) + "\n")
+ log(
+ "sample_write_only_done",
+ sample=sample_index,
+ qid=qid,
+ writer_calls=writer_calls,
+ graph_records=record["graph_records"],
+ graph_edges=record["graph_edges"],
+ seconds=record["sample_seconds"],
+ )
+ continue
+
+ question_date = clean_text(row.get("question_date", ""))
+ runtime_question = question
+ if question_date:
+ runtime_question = f"{question}\nQuestion date: {question_date}"
+ retrieve_start = time.perf_counter()
+ retrieval = adapter.retrieve(runtime_question, top_k=args.top_k)
+ retrieve_seconds = time.perf_counter() - retrieve_start
+ hits = list(getattr(retrieval, "hits", []) or [])
+ hits = expand_dialogue_chain_hits(runtime_question, hits, getattr(adapter, "graph", None))
+ hits = expand_semantic_coverage_hits(runtime_question, hits, getattr(adapter, "graph", None))
+ answer_evidence_windows = build_answer_evidence(runtime_question, hits)
+ answer_evidence_windows = apply_answer_window_planner(runtime_question, answer_evidence_windows)
+ answer_evidence_windows = apply_llm_evidence_selector(runtime_question, answer_evidence_windows)
+ answer_evidence_windows = apply_evidence_unit_planner(runtime_question, answer_evidence_windows)
+ answer_evidence_windows = apply_unified_operation_planner(runtime_question, answer_evidence_windows)
+ answer_evidence_windows = diversify_evidence_windows(answer_evidence_windows)
+ answer_evidence_windows = apply_llm_channel_planner(runtime_question, answer_evidence_windows)
+ answer_evidence_windows = final_answer_surface_windows(runtime_question, answer_evidence_windows)
+ answer_start = time.perf_counter()
+ hypothesis = answer_question(runtime_question, hits, answer_evidence_windows)
+ answer_seconds = time.perf_counter() - answer_start
+ graph = getattr(adapter, "graph", None)
+ record = {
+ "sample_index": sample_index,
+ "question_id": qid,
+ "question_type": row.get("question_type"),
+ "question": question,
+ "question_date": question_date,
+ "gold_answer": gold,
+ "hypothesis": hypothesis,
+ "answer_session_ids": list(row.get("answer_session_ids") or []),
+ "selected_session_ids": selected_session_ids,
+ "total_sessions": len(sessions),
+ "total_turns": sum(len(s or []) for s in sessions),
+ "selected_sessions": len(selected_session_ids),
+ "history_mode": "official_full_history" if args.official_full_history else "controlled_answer_plus_distractors",
+ "writer_calls": writer_calls,
+ "writer_accepted": writer_accepted,
+ "writer_suspected": writer_suspected,
+ "writer_records": writer_records,
+ "writer_unit_records": writer_unit_records,
+ "writer_unit_calls": writer_unit_calls,
+ "writer_seconds": round(writer_seconds, 3),
+ "retrieve_seconds_wall": round(retrieve_seconds, 3),
+ "answer_seconds": round(answer_seconds, 3),
+ "sample_seconds": round(time.perf_counter() - sample_start, 3),
+ "graph_records": len(getattr(graph, "records_by_id", {}) or {}),
+ "graph_edges": len(getattr(graph, "edges", []) or []),
+ "chunk_errors": chunk_errors,
+ "retrieval": retrieval_debug(retrieval),
+ "evidence_unit_plan": evidence_unit_plan_from_windows(answer_evidence_windows),
+ "answer_evidence_windows": [
+ {
+ "memory_id": item.get("memory_id", ""),
+ "score": item.get("score", 0.0),
+ "planner_selected": bool(item.get("planner_selected", False)),
+ "planner_score": item.get("planner_score", 0.0),
+ "semantic_similarity": item.get("semantic_similarity", item.get("answer_window_semantic_similarity", 0.0)),
+ "answer_window_semantic_similarity": item.get("answer_window_semantic_similarity", 0.0),
+ "answer_window_semantic_enabled": bool(item.get("answer_window_semantic_enabled", False)),
+ "answer_window_semantic_mode": item.get("answer_window_semantic_mode", ""),
+ "window_planner_should_inject_score": item.get("window_planner_should_inject_score", 0.0),
+ "answer_plan_selected": bool(item.get("answer_plan_selected", False)),
+ "answer_plan_score": item.get("answer_plan_score", 0.0),
+ "answer_plan_rank": item.get("answer_plan_rank", 0),
+ "evidence_role": item.get("evidence_role", ""),
+ "temporal_state": item.get("temporal_state", ""),
+ "assistant_memory_query": bool(item.get("assistant_memory_query", False)),
+ "assistant_origin_evidence": bool(item.get("assistant_origin_evidence", False)),
+ "hit_evidence_role": item.get("hit_evidence_role", ""),
+ "hit_temporal_state": item.get("hit_temporal_state", ""),
+ "window_planner_enabled": bool(item.get("window_planner_enabled", False)),
+ "llm_evidence_selector_enabled": bool(item.get("llm_evidence_selector_enabled", False)),
+ "llm_evidence_selected": bool(item.get("llm_evidence_selected", False)),
+ "llm_evidence_rank": item.get("llm_evidence_rank", None),
+ "llm_evidence_model_kept": bool(item.get("llm_evidence_model_kept", False)),
+ "evidence_unit_planner_enabled": bool(item.get("evidence_unit_planner_enabled", False)),
+ "evidence_unit_planner_selected": bool(item.get("evidence_unit_planner_selected", False)),
+ "evidence_unit_planner_rank": item.get("evidence_unit_planner_rank", None),
+ "semantic_coverage_expansion": bool(item.get("semantic_coverage_expansion", False)),
+ "semantic_coverage_score": item.get("semantic_coverage_score", 0.0),
+ "unit_coverage_pack": bool(item.get("unit_coverage_pack", False)),
+ "multi_unit_chain_slot": bool(item.get("multi_unit_chain_slot", False)),
+ "multi_unit_chain_bundle": bool(item.get("multi_unit_chain_bundle", False)),
+ "multi_unit_chain_score": float(item.get("multi_unit_chain_score", 0.0) or 0.0),
+ "unit_kind": item.get("unit_kind", ""),
+ "facet_type": item.get("facet_type", ""),
+ "unified_operation_planner": bool(item.get("unified_operation_planner", False)),
+ "unified_operation_family": item.get("unified_operation_family", ""),
+ "unified_selected": bool(item.get("unified_selected", False)),
+ "unified_scores": item.get("unified_scores", {}),
+ "llm_channel_planner_enabled": bool(item.get("llm_channel_planner_enabled", False)),
+ "llm_channel": item.get("llm_channel", ""),
+ "llm_channel_operation_family": item.get("llm_channel_operation_family", ""),
+ "llm_channel_coverage_complete": bool(item.get("llm_channel_coverage_complete", False)),
+ "llm_channel_missing_evidence": item.get("llm_channel_missing_evidence", ""),
+ "llm_channel_reasoning_policy": item.get("llm_channel_reasoning_policy", ""),
+ "llm_channel_planner_error": item.get("llm_channel_planner_error", ""),
+ "unit_indexes": item.get("unit_indexes", []),
+ "original_chars": item.get("original_chars", 0),
+ "text": truncate(item.get("text", ""), 700),
+ }
+ for item in answer_evidence_windows[:8]
+ ],
+ }
+ records.append(record)
+ with predictions_path.open("a", encoding="utf-8") as handle:
+ handle.write(json.dumps({"question_id": qid, "hypothesis": hypothesis}, ensure_ascii=False) + "\n")
+ with debug_path.open("a", encoding="utf-8") as handle:
+ handle.write(json.dumps(record, ensure_ascii=False) + "\n")
+ log(
+ "sample_done",
+ sample=sample_index,
+ qid=qid,
+ hits=record["retrieval"]["hit_count"],
+ selected_events=record["retrieval"]["selected_event_count"],
+ paths=record["retrieval"]["selected_path_count"],
+ writer_calls=writer_calls,
+ seconds=record["sample_seconds"],
+ hypothesis=truncate(hypothesis, 120),
+ )
+
+ summary = {
+ "status": "completed",
+ "mode": "native_tmcra_flow_smoke",
+ "history_mode": "official_full_history" if args.official_full_history else "controlled_answer_plus_distractors",
+ "limit": args.limit,
+ "sample_count": len(records),
+ "out_dir": str(out_dir),
+ "data": str(args.data),
+ "started_at": datetime.fromtimestamp(started).isoformat(timespec="seconds"),
+ "completed_at": datetime.now().isoformat(timespec="seconds"),
+ "elapsed_seconds": round(time.time() - started, 3),
+ "total_writer_calls": sum(int(item["writer_calls"]) for item in records),
+ "total_writer_accepted": sum(int(item["writer_accepted"]) for item in records),
+ "total_writer_suspected": sum(int(item["writer_suspected"]) for item in records),
+ "total_writer_unit_calls": sum(int(item.get("writer_unit_calls", 0)) for item in records),
+ "total_writer_unit_records": sum(int(item.get("writer_unit_records", 0)) for item in records),
+ "total_graph_records": sum(int(item["graph_records"]) for item in records),
+ "total_graph_edges": sum(int(item["graph_edges"]) for item in records),
+ "avg_retrieval_hits": round(sum(float(item["retrieval"]["hit_count"]) for item in records) / max(1, len(records)), 4),
+ "avg_selected_events": round(sum(float(item["retrieval"]["selected_event_count"]) for item in records) / max(1, len(records)), 4),
+ "avg_selected_paths": round(sum(float(item["retrieval"]["selected_path_count"]) for item in records) / max(1, len(records)), 4),
+ "samples": [
+ {
+ "question_id": item["question_id"],
+ "question": item["question"],
+ "gold_answer": item["gold_answer"],
+ "hypothesis": item["hypothesis"],
+ "retrieval_hits": item["retrieval"]["hit_count"],
+ "selected_event_count": item["retrieval"]["selected_event_count"],
+ "selected_path_count": item["retrieval"]["selected_path_count"],
+ "writer_calls": item["writer_calls"],
+ "sample_seconds": item["sample_seconds"],
+ "chunk_errors": item["chunk_errors"],
+ }
+ for item in records
+ ],
+ }
+ summary_path.write_text(json.dumps(summary, ensure_ascii=False, indent=2), encoding="utf-8")
+ log("run_done", out=str(out_dir), samples=len(records), elapsed=summary["elapsed_seconds"])
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--data", default=str(DEFAULT_DATA))
+ parser.add_argument("--repo", default=str(DEFAULT_REPO))
+ parser.add_argument("--service-root", default=str(DEFAULT_SERVICE_ROOT))
+ parser.add_argument("--out", default="")
+ parser.add_argument("--limit", type=int, default=10)
+ parser.add_argument("--top-k", type=int, default=8)
+ parser.add_argument("--max-distractor-sessions", type=int, default=5)
+ parser.add_argument("--max-distractor-chunks", type=int, default=1)
+ parser.add_argument("--max-answer-chunks", type=int, default=4)
+ parser.add_argument("--official-full-history", action="store_true")
+ parser.add_argument("--max-session-chunks", type=int, default=0)
+ parser.add_argument("--chunk-chars", type=int, default=7000)
+ parser.add_argument("--enable-topic-buckets", action="store_true")
+ parser.add_argument("--write-only", action="store_true", help="Ingest/write graph only; skip retrieval, answer generation, and answer-side LLM calls.")
+ args = parser.parse_args()
+ run(args)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/legacy/__init__.py b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/__init__.py
new file mode 100644
index 0000000000000000000000000000000000000000..858d22ea1afa79a70d9d7875c1c4140ddf85f9fe
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/__init__.py
@@ -0,0 +1 @@
+"""Hash-pinned TMCRA V4 benchmark implementation."""
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/legacy/audit_tmcra_v4_chain.py b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/audit_tmcra_v4_chain.py
new file mode 100644
index 0000000000000000000000000000000000000000..abc42bd0af9fa8ff51c4c5815d0b7999a40cf09b
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/audit_tmcra_v4_chain.py
@@ -0,0 +1,4268 @@
+"""Strict post-run audit for the TMCRA V4 writer/slow/retrieval chain.
+
+The audit deliberately reads SQLite and JSON artifacts instead of importing the
+controllers. This keeps it useful after a process has failed and lets it fail
+closed when a future V4 schema is not recognizable.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import re
+import sqlite3
+import sys
+from collections import defaultdict
+from pathlib import Path
+from typing import Any, Iterable, Mapping, Sequence
+
+
+KNOWN_DBS = {"native_memory.sqlite3", "memory.sqlite3", "writer.sqlite3", "graph.sqlite3"}
+SOURCE_TABLES = ("records",)
+REQUIRED_GRAPH_TABLES = {"records", "memory_edges", "slot_heads"}
+FORBIDDEN_SHADOW_TABLES = {"v4_source_records", "v4_fast_assertion_leaves"}
+FORBIDDEN_INPUT_KEYS = {
+ "answer", "gold_answer", "expected_answer", "answer_session_ids", "labels",
+ "label", "supervision", "judge", "judge_output", "benchmark", "gold",
+}
+FORBIDDEN_ARTIFACT_KEY_PARTS = ("fallback", "retry", "alternate_model", "silent_repair")
+OPERATIONAL_MARKER_VALUE_PATH_PARTS = (
+ "route",
+ "reason",
+ "status",
+ "error",
+ "policy",
+ "mode",
+ "strategy",
+ "recovery",
+ "handler",
+ "model",
+)
+SEMANTIC_IDENTIFIER_PATH_PARTS = (
+ "evidence_id",
+ "memory_id",
+ "source_record_id",
+ "capsule_id",
+ "claim_id",
+ "interaction_id",
+ "region_key",
+)
+FORBIDDEN_RETRIEVAL_KEY_PARTS = ("hard_negative", "hard_mode", "gold", "label", "supervision", "judge")
+MARKER_RE = re.compile(r"\b(?:fallback|retry|retryable|alternate\s+model|silent\s+repair)\b", re.I)
+CONFLICT_RE = re.compile(r"same[_ -]?slot|counterevidence|unresolved[_ -]?challenge|conflict", re.I)
+SLOW_OPERATIONAL_SUMMARY_RE = re.compile(
+ r"^\s*deterministic\s+(?:create|revise|retire|cleanup|migration)\b|"
+ r"^\s*(?:initial\s+)?consolidat(?:e|ed|es|ing|ion)\b[^.]{0,120}\b(?:evidence|claims?|facts?|preferences?|routines?|memory)\b|"
+ r"^\s*(?:add|added|adding|challenge|challenged|challenging)\b[^.]{0,80}\bevidence\b|"
+ r"^\s*add\b|"
+ r"\b(?:supplied|required|current\s+fast)\s+evidence\b|"
+ r"^\s*create\s+(?:an?\s+|the\s+)?(?:initial\s+)?[^.]{0,80}\b(?:memory|claims?|record)\b|"
+ r"^\s*(?:memory|record)\s+(?:revision|cleanup|migration)\b|"
+ r"^\s*(?:创建|新增|更新|合并|整理).{0,24}(?:记忆|胶囊|证据|声明|记录)|"
+ r"^\s*(?:控制器|确定性创建|确定性修订).{0,24}(?:记忆|证据|声明|记录|胶囊)|"
+ r"(?:快速图证据)",
+ re.I,
+)
+SLOW_GENERIC_SUMMARY_RE = re.compile(
+ r"^user(?:'s)?\s+(?:fitness\s+)?(?:goals?\s+and\s+facts?|commute\s+details?|"
+ r"schedule\s+routine|preferences?\s+and\s+facts?)(?:\.)?$|"
+ r"^user\s+(?:has\s+)?social\s+media\s+goals?\s+and\s+routines?(?:\.)?$|"
+ r"^user\s+opinions?\s+(?:on|about)\s+[^.]+(?:\.)?$|"
+ r"^user\s+preferences?\s+and\s+facts?\s+about\s+[^.]+(?:\.)?$|"
+ r"^user(?:'s)?\s+[^.]+\s+preferences?\s+and\s+wearing\s+frequency(?:\.)?$",
+ re.I,
+)
+CURRENT_FAST_INDEX_POLICY = (
+ "current-fast-states-v1:active,challenged,parallel_active,promoted"
+)
+SOURCE_COVERAGE_TRACE_K = 24
+RETRIEVAL_SCHEMA_VERSION = "tmcra.v4.online-retrieval.6"
+RETRIEVAL_REPORT_SCHEMA_VERSION = "tmcra.v4.online-retrieval-report.6"
+RETRIEVAL_CONTRACT_SCHEMA_VERSION = "tmcra.v4.layered-retrieval-contract.3"
+ONLINE_INDEX_SCHEMA_VERSION = "tmcra.v4.online-index.3"
+SLOW_INVENTORY_SCHEMA_VERSION = "tmcra.v4.slow-inventory.1"
+SLOW_SUMMARY_CONTRACT_VERSION = "tmcra.v4.slow-lossless-summary.2"
+CURRENT_SLOW_PROMPT_VERSION = "tmcra-v4-slow-graph-2026-07-14.16"
+CURRENT_SLOW_PARTITION_CONTRACT_VERSION = "tmcra.v4.slow-semantic-partition.2"
+SESSION_ORDERING_POLICY = "session_rrf_then_chronological_v1"
+KEEP_PARALLEL_MIGRATION_SCHEMA_VERSION = "tmcra.v4.keep-parallel-migration.1"
+KEEP_PARALLEL_MIGRATION_VERSION = "keep-parallel-authority-2026-07-12.1"
+CHALLENGE_MIGRATION_SCHEMA_VERSION = "tmcra.v4.challenge-lifecycle-migration.1"
+CHALLENGE_MIGRATION_VERSION = "challenge-lifecycle-authority-2026-07-15.1"
+ACTIVE_RECORD_STATES = {"active", "parallel_active", "promoted"}
+CURRENT_FAST_RECORD_STATES = {
+ "active",
+ "parallel_active",
+ "promoted",
+ "challenged",
+}
+PRODUCTION_FAST_NODE_KIND = "atomic_user_assertion"
+PRODUCTION_FAST_VARIANT = "product_semantic_memory"
+SUBJECT_ATTRIBUTION_PROMPT_VERSION = "tmcra-v4-subject-attribution-2026-07-14.3"
+FAST_SOURCE_PARENT_KEYS = {
+ "session_index",
+ "parent_chunk_index",
+ "source_record_id",
+ "evidence_char_start",
+ "evidence_char_end",
+}
+SLOW_SOURCE_PARENT_KEYS = {
+ *FAST_SOURCE_PARENT_KEYS,
+ "message_index",
+ "event_id",
+}
+
+Identity = tuple[str, str, str]
+SourceMessageKey = tuple[str, str, str]
+SourceScopeMessageKey = tuple[str, str]
+VALID_SUPERSESSION_LIFECYCLE_REASONS = {
+ "same_state_revision",
+ "slot_disallows_parallel",
+ "v4_reconciliation_replace_current",
+}
+
+
+class AuditError(RuntimeError):
+ pass
+
+
+def _json(value: Any) -> Any:
+ if isinstance(value, (dict, list)):
+ return value
+ if value is None or value == "":
+ return {}
+ try:
+ return json.loads(value)
+ except (TypeError, json.JSONDecodeError) as exc:
+ raise AuditError(f"invalid JSON artifact value: {exc}") from exc
+
+
+def _text(value: Any) -> str:
+ return str(value).strip() if value is not None else ""
+
+
+def _normal_text(value: Any) -> str:
+ return " ".join(_text(value).casefold().split())
+
+
+def _leaf_metadata(leaf: Mapping[str, Any]) -> Mapping[str, Any]:
+ metadata = leaf.get("metadata")
+ return metadata if isinstance(metadata, Mapping) else {}
+
+
+def _leaf_slot(leaf: Mapping[str, Any]) -> str:
+ metadata = _leaf_metadata(leaf)
+ return _text(
+ metadata.get("canonical_slot_key") or metadata.get("canonical_slot")
+ )
+
+
+def _leaf_text(leaf: Mapping[str, Any]) -> str:
+ metadata = _leaf_metadata(leaf)
+ return _text(
+ leaf.get("value") or metadata.get("source_span") or metadata.get("raw_content")
+ )
+
+
+def _leaf_state(leaf: Mapping[str, Any]) -> str:
+ metadata = _leaf_metadata(leaf)
+ return _text(
+ leaf.get("record_state")
+ or leaf.get("state")
+ or metadata.get("record_state")
+ ).casefold()
+
+
+def _is_current_durable_leaf(leaf: Mapping[str, Any]) -> bool:
+ metadata = _leaf_metadata(leaf)
+ durability = _text(
+ metadata.get("durability") or metadata.get("durability_class")
+ ).casefold()
+ return durability in {
+ "durable",
+ "long_term",
+ "long-term",
+ "hard",
+ "persistent",
+ } and _leaf_state(leaf) in {"active", "parallel_active", "promoted"}
+
+
+def _is_counterevidence_leaf(leaf: Mapping[str, Any]) -> bool:
+ metadata = _leaf_metadata(leaf)
+ return bool(metadata.get("counterevidence")) or bool(
+ metadata.get("is_counterevidence")
+ )
+
+
+def _controlled_complementary_support_bundle(
+ claim_slot: str,
+ evidence_ids: Iterable[Identity],
+ evidence_by_id: Mapping[Identity, Mapping[str, Any]],
+) -> bool:
+ """Recognize one durable parent fact plus compatible subslot refinements."""
+ unique_ids = sorted(set(evidence_ids))
+ if len(unique_ids) < 2:
+ return False
+ leaves = [evidence_by_id.get(evidence_id) for evidence_id in unique_ids]
+ if any(
+ leaf is None
+ or not _is_current_durable_leaf(leaf)
+ or _is_counterevidence_leaf(leaf)
+ for leaf in leaves
+ ):
+ return False
+ typed_leaves = [leaf for leaf in leaves if leaf is not None]
+ slots = [_leaf_slot(leaf) for leaf in typed_leaves]
+ if claim_slot not in slots or len(set(slots)) < 2:
+ return False
+ if any(
+ slot != claim_slot and not slot.startswith(claim_slot + ".")
+ for slot in slots
+ ):
+ return False
+
+ texts_by_slot: dict[str, set[str]] = {}
+ for leaf in typed_leaves:
+ texts_by_slot.setdefault(_leaf_slot(leaf), set()).add(
+ _normal_text(_leaf_text(leaf))
+ )
+ if any(len(texts) != 1 or "" in texts for texts in texts_by_slot.values()):
+ return False
+
+ def one_shared_metadata_value(key: str) -> bool:
+ values = {
+ _text(_leaf_metadata(leaf).get(key)).casefold()
+ for leaf in typed_leaves
+ }
+ return len(values) == 1 and "" not in values
+
+ if not all(
+ one_shared_metadata_value(key)
+ for key in ("subject_signature", "graph_entity_key", "memory_family")
+ ):
+ return False
+ relations = {
+ _text(
+ leaf.get("relation") or _leaf_metadata(leaf).get("semantic_slot")
+ ).casefold()
+ for leaf in typed_leaves
+ }
+ polarities = {
+ _text(_leaf_metadata(leaf).get("polarity")).casefold()
+ for leaf in typed_leaves
+ }
+ return (
+ len(relations) == 1
+ and "" not in relations
+ and len(polarities) == 1
+ and "" not in polarities
+ )
+
+
+def _lossless_summary_projection(claims: Any) -> str:
+ if not isinstance(claims, list) or not claims:
+ raise AuditError("lossless Slow summary projection requires claims")
+ texts: list[str] = []
+ for index, claim in enumerate(claims):
+ if not isinstance(claim, Mapping):
+ raise AuditError(f"lossless Slow summary claim {index} is not an object")
+ text = " ".join(_text(claim.get("text")).split())
+ if not text:
+ raise AuditError(f"lossless Slow summary claim {index} lacks text")
+ texts.append(text)
+ return " ".join(texts)
+
+
+def _current_summary_contract(
+ record_metadata: Mapping[str, Any],
+ patch_metadata: Mapping[str, Any] | None = None,
+) -> bool:
+ for value in (record_metadata, patch_metadata or {}):
+ if not isinstance(value, Mapping):
+ continue
+ if (
+ value.get("summary_contract_version") == SLOW_SUMMARY_CONTRACT_VERSION
+ or value.get("prompt_version") == CURRENT_SLOW_PROMPT_VERSION
+ or value.get("partition_contract_version")
+ == "tmcra.v4.slow-semantic-partition.1"
+ ):
+ return True
+ provenance = value.get("provenance")
+ if isinstance(provenance, Mapping) and _current_summary_contract(provenance):
+ return True
+ return False
+
+
+def _current_v47_artifact(value: Any) -> bool:
+ """Return whether persisted metadata explicitly identifies current V4.7."""
+ if isinstance(value, Mapping):
+ if (
+ value.get("prompt_version") == CURRENT_SLOW_PROMPT_VERSION
+ or value.get("summary_contract_version") == SLOW_SUMMARY_CONTRACT_VERSION
+ or value.get("partition_contract_version")
+ == CURRENT_SLOW_PARTITION_CONTRACT_VERSION
+ ):
+ return True
+ return any(_current_v47_artifact(item) for item in value.values())
+ if isinstance(value, list):
+ return any(_current_v47_artifact(item) for item in value)
+ return False
+
+
+def _current_v47_capsule_id(
+ scope_id: Any, region_key: Any, capsule_key: Any
+) -> str:
+ """Mirror the V4.7 deterministic create-target mapping without importing runtime code."""
+ normalized_key = _text(capsule_key).casefold()
+ payload = {
+ "scope_id": _text(scope_id),
+ "region_key": _text(region_key),
+ "capsule_key": normalized_key,
+ "partition_contract_version": CURRENT_SLOW_PARTITION_CONTRACT_VERSION,
+ }
+ encoded = json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
+ return "cap_" + _sha(encoded)[:24]
+
+
+def _audit_current_v47_patch_reconciliation(
+ *,
+ jobs: Sequence[tuple[Path, Mapping[str, Any]]],
+ patches: Sequence[tuple[Path, Mapping[str, Any]]],
+ operation_rows: Sequence[tuple[Path, Mapping[str, Any]]],
+ slow_records: Sequence[Mapping[str, Any]],
+ issues: list[str],
+) -> None:
+ """Reconcile current V4.7 patch envelopes, operation rows, and result records."""
+ job_by_key = {
+ (_db_path(path), _text(row.get("job_id"))): row
+ for path, row in jobs
+ if _text(row.get("job_id"))
+ }
+ patch_by_key = {
+ (_db_path(path), _text(row.get("patch_id"))): row
+ for path, row in patches
+ if _text(row.get("patch_id"))
+ }
+ patches_by_job: dict[tuple[str, str], list[Mapping[str, Any]]] = defaultdict(list)
+ for path, row in patches:
+ patch_id = _text(row.get("patch_id"))
+ job_id = _text(row.get("job_id"))
+ if patch_id and job_id:
+ patches_by_job[(_db_path(path), job_id)].append(row)
+
+ operations_by_patch: dict[tuple[str, str], list[Mapping[str, Any]]] = defaultdict(list)
+ for path, row in operation_rows:
+ patch_id = _text(row.get("patch_id"))
+ patch_key = (_db_path(path), patch_id)
+ if patch_key not in patch_by_key:
+ issues.append(
+ f"{path}: orphan slow graph patch operation {_text(row.get('operation_id'))}"
+ )
+ else:
+ operations_by_patch[patch_key].append(row)
+
+ result_records: dict[tuple[str, str, str, int], list[Mapping[str, Any]]] = defaultdict(list)
+ current_record_patch_ids: set[tuple[str, str]] = set()
+ for record in slow_records:
+ metadata = record.get("metadata")
+ if not isinstance(metadata, Mapping):
+ continue
+ patch_id = _text(metadata.get("patch_id"))
+ capsule_id = _text(metadata.get("capsule_id"))
+ if not patch_id or not capsule_id:
+ continue
+ revision_value = metadata.get("revision")
+ if isinstance(revision_value, bool) or not isinstance(revision_value, int):
+ continue
+ revision = revision_value
+ key = (_db_path(record.get("db")), patch_id)
+ if _current_v47_artifact(metadata):
+ current_record_patch_ids.add(key)
+ result_records[(*key, capsule_id, revision)].append(record)
+
+ current_jobs: set[tuple[str, str]] = set()
+ for path, row in jobs:
+ job_key = (_db_path(path), _text(row.get("job_id")))
+ if job_key[1] and _current_v47_artifact(_json(row.get("metadata_json"))):
+ current_jobs.add(job_key)
+
+ current_patches: set[tuple[str, str]] = set(current_record_patch_ids)
+ for path, row in patches:
+ patch_key = (_db_path(path), _text(row.get("patch_id")))
+ if not patch_key[1]:
+ continue
+ if _current_v47_artifact(_json(row.get("call_metadata_json"))):
+ current_patches.add(patch_key)
+ job_key = (_db_path(path), _text(row.get("job_id")))
+ if job_key in current_jobs:
+ current_patches.add(patch_key)
+
+ for job_key in sorted(current_jobs):
+ matching = patches_by_job.get(job_key, [])
+ if len(matching) != 1:
+ issues.append(
+ f"{job_key[0]}: current V4.7 job {job_key[1]} has {len(matching)} patches; expected exactly one"
+ )
+
+ for patch_key in sorted(current_patches):
+ patch = patch_by_key.get(patch_key)
+ if patch is None:
+ issues.append(f"{patch_key[0]}: current V4.7 result references missing patch {patch_key[1]}")
+ continue
+ job_id = _text(patch.get("job_id"))
+ job_key = (patch_key[0], job_id)
+ job = job_by_key.get(job_key)
+ matching = patches_by_job.get(job_key, [])
+ if job is None:
+ issues.append(
+ f"{patch_key[0]}: current V4.7 patch {patch_key[1]} references missing job {job_id}"
+ )
+ if len(matching) != 1:
+ issues.append(
+ f"{patch_key[0]}: current V4.7 patch {patch_key[1]} does not have exactly one patch for job {job_id}"
+ )
+ if job is not None and _text(patch.get("scope_id")) != _text(job.get("scope_id")):
+ issues.append(f"{patch_key[0]}: current V4.7 patch {patch_key[1]} scope differs from its job")
+ if job is not None and _text(patch.get("region_key")) != _text(job.get("region_key")):
+ issues.append(f"{patch_key[0]}: current V4.7 patch {patch_key[1]} region differs from its job")
+
+ try:
+ patch_json = _json(patch.get("patch_json"))
+ except AuditError as exc:
+ issues.append(f"{patch_key[0]}: current V4.7 patch {patch_key[1]} has invalid patch_json: {exc}")
+ continue
+ if not isinstance(patch_json, Mapping) or not isinstance(patch_json.get("operations"), list):
+ issues.append(f"{patch_key[0]}: current V4.7 patch {patch_key[1]} operations are missing")
+ continue
+ operations = patch_json["operations"]
+ rows = list(operations_by_patch.get(patch_key, ()))
+ ordinals: list[int | None] = []
+ for row in rows:
+ try:
+ ordinals.append(int(row.get("ordinal")))
+ except (TypeError, ValueError):
+ ordinals.append(None)
+ duplicate_ordinals = sorted(
+ ordinal for ordinal in set(item for item in ordinals if item is not None)
+ if ordinals.count(ordinal) > 1
+ )
+ if duplicate_ordinals:
+ issues.append(
+ f"{patch_key[0]}: current V4.7 patch {patch_key[1]} has duplicate operation ordinals {duplicate_ordinals}"
+ )
+ operation_ids = [_text(row.get("operation_id")) for row in rows]
+ duplicate_operation_ids = sorted(
+ operation_id for operation_id in set(operation_ids) if operation_ids.count(operation_id) > 1
+ )
+ if duplicate_operation_ids:
+ issues.append(
+ f"{patch_key[0]}: current V4.7 patch {patch_key[1]} has duplicate operation rows {duplicate_operation_ids}"
+ )
+ def operation_row_sort_key(row: Mapping[str, Any]) -> tuple[int, int, str]:
+ try:
+ return (0, int(row.get("ordinal")), _text(row.get("operation_id")))
+ except (TypeError, ValueError):
+ return (1, 0, _text(row.get("operation_id")))
+
+ rows.sort(key=operation_row_sort_key)
+ if len(rows) != len(operations):
+ issues.append(
+ f"{patch_key[0]}: current V4.7 patch {patch_key[1]} operation row count {len(rows)} differs from patch_json count {len(operations)}"
+ )
+ sorted_ordinals: list[int | None] = []
+ for row in rows:
+ try:
+ sorted_ordinals.append(int(row.get("ordinal")))
+ except (TypeError, ValueError):
+ sorted_ordinals.append(None)
+ if (
+ any(ordinal is None for ordinal in sorted_ordinals)
+ or len(rows) != len(operations)
+ or sorted_ordinals != list(range(len(operations)))
+ ):
+ issues.append(f"{patch_key[0]}: current V4.7 patch {patch_key[1]} operation ordinals are not exact")
+
+ represented_results: set[tuple[str, int]] = set()
+ for ordinal, operation in enumerate(operations):
+ if not isinstance(operation, Mapping):
+ issues.append(f"{patch_key[0]}: current V4.7 patch {patch_key[1]} operation {ordinal} is not an object")
+ continue
+ if ordinal >= len(rows):
+ continue
+ row = rows[ordinal]
+ try:
+ stored_operation = _json(row.get("operation_json"))
+ except AuditError as exc:
+ issues.append(f"{patch_key[0]}: current V4.7 patch {patch_key[1]} operation {ordinal} has invalid operation_json: {exc}")
+ stored_operation = None
+ if stored_operation != operation:
+ issues.append(f"{patch_key[0]}: current V4.7 patch {patch_key[1]} operation {ordinal} differs from patch_json")
+ action = _text(operation.get("action"))
+ if _text(row.get("action")) != action:
+ issues.append(f"{patch_key[0]}: current V4.7 patch {patch_key[1]} operation {ordinal} action differs")
+ row_capsule_id = _text(row.get("capsule_id"))
+ region_key = _text(patch.get("region_key"))
+ if action == "create":
+ if not _text(operation.get("capsule_key")):
+ issues.append(f"{patch_key[0]}: current V4.7 patch {patch_key[1]} operation {ordinal} create capsule key is missing")
+ expected_capsule_id = _current_v47_capsule_id(
+ patch.get("scope_id"), region_key, operation.get("capsule_key")
+ )
+ if operation.get("capsule_id") is not None or row_capsule_id != expected_capsule_id:
+ issues.append(f"{patch_key[0]}: current V4.7 patch {patch_key[1]} operation {ordinal} create target capsule differs")
+ expected_base, expected_result = None, 1
+ elif action == "noop":
+ expected_target = _text(operation.get("capsule_id")) or "region:" + region_key
+ if row_capsule_id != expected_target:
+ issues.append(f"{patch_key[0]}: current V4.7 patch {patch_key[1]} operation {ordinal} target capsule differs")
+ expected_base = row.get("base_revision")
+ expected_result = row.get("result_revision")
+ if expected_base != expected_result:
+ issues.append(f"{patch_key[0]}: current V4.7 patch {patch_key[1]} operation {ordinal} noop revisions differ")
+ else:
+ expected_target = _text(operation.get("capsule_id"))
+ if not expected_target or row_capsule_id != expected_target:
+ issues.append(f"{patch_key[0]}: current V4.7 patch {patch_key[1]} operation {ordinal} target capsule differs")
+ expected_base = operation.get("base_revision")
+ expected_result = expected_base + 1 if isinstance(expected_base, int) and not isinstance(expected_base, bool) else None
+ if row.get("base_revision") != expected_base or row.get("result_revision") != expected_result:
+ issues.append(f"{patch_key[0]}: current V4.7 patch {patch_key[1]} operation {ordinal} revisions differ")
+ if action == "create" and (
+ row.get("base_revision") != expected_base or row.get("result_revision") != expected_result
+ ):
+ issues.append(f"{patch_key[0]}: current V4.7 patch {patch_key[1]} operation {ordinal} revisions differ")
+ if action != "noop" and isinstance(expected_result, int) and row_capsule_id:
+ result_key = (row_capsule_id, expected_result)
+ represented_results.add(result_key)
+ matches = result_records.get((*patch_key, *result_key), [])
+ if len(matches) != 1:
+ issues.append(
+ f"{patch_key[0]}: current V4.7 patch {patch_key[1]} operation {ordinal} lacks exactly one resulting record metadata match"
+ )
+ elif _text(matches[0]["metadata"].get("action")) not in {"", action}:
+ issues.append(f"{patch_key[0]}: current V4.7 patch {patch_key[1]} operation {ordinal} result action differs")
+
+ for result_key, matches in result_records.items():
+ if result_key[:2] != patch_key or result_key[2:] in represented_results:
+ continue
+ issues.append(
+ f"{patch_key[0]}: current V4.7 patch {patch_key[1]} has an unrepresented resulting record metadata row"
+ )
+
+
+def _active_slow_partition_issues(
+ slow_records: Sequence[Mapping[str, Any]],
+ patch_metadata: Mapping[tuple[str, str], Mapping[str, Any]] | None = None,
+) -> list[dict[str, Any]]:
+ """Audit V4.7 partitioning without treating several heads as an error."""
+ patch_metadata = patch_metadata or {}
+ by_region: dict[tuple[str, str, str], dict[str, Mapping[str, Any]]] = {}
+ issues: list[dict[str, Any]] = []
+ for record in slow_records:
+ metadata = record.get("metadata")
+ if not isinstance(metadata, Mapping) or not _current_summary_contract(
+ metadata,
+ patch_metadata.get(
+ (_db_path(record.get("db")), _text(metadata.get("patch_id"))),
+ ),
+ ):
+ continue
+ region_key = _text(metadata.get("region_key"))
+ capsule_id = _text(metadata.get("capsule_id"))
+ if (
+ metadata.get("partition_contract_version")
+ != CURRENT_SLOW_PARTITION_CONTRACT_VERSION
+ ):
+ issues.append(
+ {
+ "code": "semantic_partition_migration_required",
+ "db": _db_path(record.get("db")),
+ "scope_id": _text(record.get("scope_id")),
+ "capsule_id": capsule_id,
+ "revision": record.get("slow_revision"),
+ "region_key": region_key,
+ "stored_contract_version": metadata.get(
+ "partition_contract_version"
+ ),
+ }
+ )
+ if not region_key:
+ issues.append(
+ {
+ "code": "current_v4_7_missing_region_key",
+ "db": _db_path(record.get("db")),
+ "scope_id": _text(record.get("scope_id")),
+ "capsule_id": capsule_id,
+ "revision": record.get("slow_revision"),
+ }
+ )
+ continue
+ by_region.setdefault(
+ (_db_path(record.get("db")), _text(record.get("scope_id")), region_key),
+ {},
+ )[capsule_id] = record
+
+ for (db_path, scope_id, region_key), capsules in by_region.items():
+ evidence_locations: dict[
+ str, list[tuple[str, int, str, str, str]]
+ ] = defaultdict(list)
+ claim_identity_capsules: dict[tuple[str, str], set[str]] = defaultdict(set)
+ for capsule_id, record in capsules.items():
+ metadata = record["metadata"]
+ for claim_index, claim in enumerate(list(metadata.get("claims") or [])):
+ if not isinstance(claim, Mapping):
+ continue
+ slot = _text(claim.get("canonical_slot"))
+ claim_text = _normal_text(claim.get("text"))
+ if slot and claim_text:
+ claim_identity_capsules[(slot, claim_text)].add(capsule_id)
+ for role in ("support", "counterevidence"):
+ for evidence_id in list(claim.get(role) or []):
+ evidence_id = _text(evidence_id)
+ if evidence_id:
+ evidence_locations[evidence_id].append(
+ (capsule_id, claim_index, role, slot, claim_text)
+ )
+ invalid_repeated_evidence: dict[str, list[str]] = {}
+ for evidence_id, locations in evidence_locations.items():
+ if len(locations) <= 1:
+ continue
+ roles = {role for _, _, role, _, _ in locations}
+ claim_identities = [
+ (slot, text) for _, _, _, slot, text in locations
+ ]
+ if roles == {"support"} and len(set(claim_identities)) == len(
+ claim_identities
+ ):
+ continue
+ invalid_repeated_evidence[evidence_id] = [
+ f"{capsule}:{claim_index}:{role}"
+ for capsule, claim_index, role, _, _ in locations
+ ]
+ if invalid_repeated_evidence:
+ issues.append(
+ {
+ "code": "duplicate_evidence_across_active_capsules",
+ "db": db_path,
+ "scope_id": scope_id,
+ "region_key": region_key,
+ "capsule_ids": sorted(capsules),
+ "evidence": invalid_repeated_evidence,
+ }
+ )
+ duplicated_claim_identities = {
+ f"{slot}\u241f{text}": sorted(capsule_ids)
+ for (slot, text), capsule_ids in claim_identity_capsules.items()
+ if len(capsule_ids) > 1
+ }
+ if duplicated_claim_identities:
+ issues.append(
+ {
+ "code": "semantic_claim_in_multiple_active_capsules",
+ "db": db_path,
+ "scope_id": scope_id,
+ "region_key": region_key,
+ "capsule_ids": sorted(capsules),
+ "claim_identities": duplicated_claim_identities,
+ }
+ )
+ return issues
+
+
+def _sha(value: str) -> str:
+ return hashlib.sha256(value.encode("utf-8")).hexdigest()
+
+
+def _db_path(value: Any) -> str:
+ """Normalize a persisted DB identity before using it as a graph key."""
+ return str(Path(str(value)).resolve())
+
+
+def _identity(db_path: Any, scope_id: Any, memory_id: Any) -> Identity:
+ return (_db_path(db_path), _text(scope_id), _text(memory_id))
+
+
+def _fast_quote(metadata: Mapping[str, Any]) -> str:
+ return _text(
+ metadata.get("evidence_quote")
+ or metadata.get("raw_content")
+ or metadata.get("source_span")
+ )
+
+
+def _fast_source_parent(metadata: Mapping[str, Any]) -> dict[str, Any]:
+ return {
+ "session_index": metadata.get("session_index"),
+ "parent_chunk_index": metadata.get(
+ "parent_chunk_index", metadata.get("message_index")
+ ),
+ "source_record_id": metadata.get("source_record_id"),
+ "evidence_char_start": metadata.get("evidence_char_start"),
+ "evidence_char_end": metadata.get("evidence_char_end"),
+ }
+
+
+def _slow_source_parent(metadata: Mapping[str, Any]) -> dict[str, Any]:
+ return {
+ **_fast_source_parent(metadata),
+ "message_index": metadata.get("message_index"),
+ "event_id": metadata.get("event_id"),
+ }
+
+
+def _expected_slow_context_provenance(
+ record_metadata: Mapping[str, Any],
+ claim: Mapping[str, Any],
+ source_parents: list[Any],
+ summary_candidate_id: str,
+ *,
+ patch_metadata: Mapping[str, Any] | None = None,
+) -> dict[str, Any]:
+ provenance = {
+ "memory_layer": "slow",
+ "content_variant": "slow_memory_capsule",
+ "capsule_id": _text(record_metadata.get("capsule_id")),
+ "revision": int(record_metadata.get("revision")),
+ "claim_id": _text(claim.get("claim_id")),
+ "canonical_slot": _text(claim.get("canonical_slot")),
+ "patch_id": _text(record_metadata.get("patch_id")),
+ "source_parents": source_parents,
+ "candidate_kind": "capsule_claim",
+ "capsule_summary_candidate_id": summary_candidate_id,
+ }
+ if _current_summary_contract(record_metadata, patch_metadata):
+ provenance["summary_contract_version"] = SLOW_SUMMARY_CONTRACT_VERSION
+ return provenance
+
+
+def _production_fast_predicate(row: Mapping[str, Any], metadata: Mapping[str, Any]) -> bool:
+ """Mirror the production semantic-leaf predicate used by Fast retrieval."""
+ return (
+ _text(metadata.get("memory_layer")) == "fast"
+ and _text(metadata.get("content_variant")) == PRODUCTION_FAST_VARIANT
+ and _text(metadata.get("node_kind")) == PRODUCTION_FAST_NODE_KIND
+ and metadata.get("atomic_evidence_leaf") is True
+ and _text(metadata.get("authority")) == "user_assertion"
+ and bool(_text(row.get("value")))
+ )
+
+
+def _reference_identity(
+ value: Any,
+ *,
+ db_path: Any,
+ scope_id: Any,
+ field: str = "memory_id",
+) -> Identity | None:
+ """Resolve a local reference without ever consulting a global bare-ID index."""
+ if isinstance(value, Mapping):
+ memory_id = _text(
+ value.get(field)
+ or value.get("memory_id")
+ or value.get("leaf_id")
+ or value.get("evidence_memory_id")
+ )
+ declared_scope = _text(value.get("scope_id"))
+ declared_db = value.get("db_path")
+ if declared_scope and declared_scope != _text(scope_id):
+ return None
+ if declared_db and _db_path(declared_db) != _db_path(db_path):
+ return None
+ reference_scope = _text(scope_id)
+ reference_db = db_path
+ else:
+ memory_id = _text(value)
+ reference_scope = _text(scope_id)
+ reference_db = db_path
+ if not memory_id or not reference_scope:
+ return None
+ return _identity(reference_db, reference_scope, memory_id)
+
+
+def _source_for_reference(
+ sources: Mapping[tuple[str, str, str], Mapping[str, Any]],
+ *,
+ db_path: Any,
+ scope_id: Any,
+ source_record_id: Any,
+) -> Mapping[str, Any] | None:
+ return sources.get(_identity(db_path, scope_id, source_record_id))
+
+
+def _source_message_indices(
+ sources: Mapping[Identity, Mapping[str, Any]],
+) -> tuple[
+ dict[SourceMessageKey, list[Mapping[str, Any]]],
+ dict[SourceScopeMessageKey, list[Mapping[str, Any]]],
+]:
+ by_database: dict[SourceMessageKey, list[Mapping[str, Any]]] = defaultdict(list)
+ by_scope: dict[SourceScopeMessageKey, list[Mapping[str, Any]]] = defaultdict(list)
+ for identity, source in sources.items():
+ message_id = _text(source.get("message_id"))
+ by_database[(identity[0], identity[1], message_id)].append(source)
+ by_scope[(identity[1], message_id)].append(source)
+ return dict(by_database), dict(by_scope)
+
+
+def _source_for_message(
+ source_messages: Mapping[SourceMessageKey, Sequence[Mapping[str, Any]]],
+ *,
+ db_path: Any,
+ scope_id: Any,
+ message_id: Any,
+) -> Mapping[str, Any] | None:
+ key = (_db_path(db_path), _text(scope_id), _text(message_id))
+ matches = list(source_messages.get(key, ()))
+ return matches[0] if len(matches) == 1 else None
+
+
+def _source_span_quote(
+ source: Mapping[str, Any],
+ start: Any,
+ end: Any,
+ quote: Any,
+ *,
+ label: str,
+ issues: list[str],
+) -> bool:
+ if not isinstance(quote, str) or not quote:
+ issues.append(f"{label}: quote is missing")
+ return False
+ if isinstance(start, bool) or isinstance(end, bool):
+ issues.append(f"{label}: source character span is invalid")
+ return False
+ try:
+ start_value, end_value = int(start), int(end)
+ except (TypeError, ValueError):
+ issues.append(f"{label}: source character span is invalid")
+ return False
+ content = str(source.get("content") or "")
+ if start_value < 0 or end_value <= start_value or end_value > len(content):
+ issues.append(f"{label}: source character span is out of bounds")
+ return False
+ actual = content[start_value:end_value]
+ if actual != quote:
+ issues.append(f"{label}: source slice does not equal quote exactly")
+ return False
+ return True
+
+
+def _unique_source_quote_span(
+ source: Mapping[str, Any],
+ quote: Any,
+ *,
+ label: str,
+ issues: list[str],
+) -> tuple[int, int] | None:
+ if not isinstance(quote, str) or not quote:
+ issues.append(f"{label}: quote is missing")
+ return None
+ content = str(source.get("content") or "")
+ start = content.find(quote)
+ if start < 0:
+ issues.append(f"{label}: quote is not an exact Source substring")
+ return None
+ if content.find(quote, start + 1) >= 0:
+ issues.append(f"{label}: quote is not unique within Source")
+ return None
+ return start, start + len(quote)
+
+
+def _validate_source_parent(
+ parent: Any,
+ *,
+ db_path: Any,
+ scope_id: Any,
+ sources: Mapping[tuple[str, str, str], Mapping[str, Any]],
+ issues: list[str],
+ label: str,
+ quote: Any = None,
+) -> Mapping[str, Any] | None:
+ parent_keys = frozenset(parent) if isinstance(parent, Mapping) else frozenset()
+ if not isinstance(parent, Mapping) or parent_keys not in {
+ frozenset(FAST_SOURCE_PARENT_KEYS),
+ frozenset(SLOW_SOURCE_PARENT_KEYS),
+ }:
+ issues.append(f"{label}: source parent schema is invalid")
+ return None
+ source_id = _text(parent.get("source_record_id"))
+ source = _source_for_reference(
+ sources,
+ db_path=db_path,
+ scope_id=scope_id,
+ source_record_id=source_id,
+ )
+ if source is None:
+ issues.append(f"{label}: source parent is not the current immutable Source")
+ return None
+ source_meta = source.get("metadata") if isinstance(source.get("metadata"), Mapping) else {}
+ if parent.get("session_index") != source_meta.get("session_index"):
+ issues.append(f"{label}: source parent session_index is not bound to immutable Source")
+ if parent.get("parent_chunk_index") != source_meta.get("message_index"):
+ issues.append(f"{label}: source parent chunk coordinate is not bound to immutable Source")
+ if "message_index" in parent and parent.get("message_index") != source_meta.get(
+ "message_index"
+ ):
+ issues.append(f"{label}: source parent message_index is not bound to immutable Source")
+ if "event_id" in parent and parent.get("event_id") != source_meta.get("event_id"):
+ issues.append(f"{label}: source parent event_id is not bound to immutable Source")
+ if quote is None:
+ content = str(source.get("content") or "")
+ try:
+ start_value, end_value = int(parent.get("evidence_char_start")), int(parent.get("evidence_char_end"))
+ quote = content[start_value:end_value]
+ except (TypeError, ValueError):
+ quote = ""
+ _source_span_quote(
+ source,
+ parent.get("evidence_char_start"),
+ parent.get("evidence_char_end"),
+ quote,
+ label=label,
+ issues=issues,
+ )
+ return source
+
+
+def _tables(connection: sqlite3.Connection) -> set[str]:
+ return {str(row[0]) for row in connection.execute("SELECT name FROM sqlite_master WHERE type='table'")}
+
+
+def _rows(connection: sqlite3.Connection, table: str) -> list[dict[str, Any]]:
+ return [dict(row) for row in connection.execute(f'SELECT * FROM "{table}"')]
+
+
+def _metadata(row: Mapping[str, Any]) -> dict[str, Any]:
+ raw = row.get("metadata_json", row.get("metadata", "{}"))
+ value = _json(raw)
+ if not isinstance(value, dict):
+ raise AuditError("metadata_json must contain an object")
+ return value
+
+
+def _valid_active_to_superseded_lifecycle(
+ connection: sqlite3.Connection,
+ scope_id: str,
+ memory_id: str,
+) -> bool:
+ seen: set[str] = set()
+ expected_slot = ""
+ prior_turn = -1
+ while memory_id and memory_id not in seen:
+ seen.add(memory_id)
+ row = connection.execute(
+ "SELECT slot_key,turn_index,state,metadata_json FROM records "
+ "WHERE scope_id=? AND memory_id=?",
+ (scope_id, memory_id),
+ ).fetchone()
+ if row is None:
+ return False
+ slot_key = _text(row[0])
+ turn_index = int(row[1])
+ state = _text(row[2])
+ metadata = _json(row[3])
+ if not isinstance(metadata, Mapping):
+ return False
+ if not expected_slot:
+ expected_slot = slot_key
+ if slot_key != expected_slot or turn_index < prior_turn:
+ return False
+ if state in ACTIVE_RECORD_STATES:
+ return True
+ if (
+ state != "superseded"
+ or _text(metadata.get("superseded_reason"))
+ not in VALID_SUPERSESSION_LIFECYCLE_REASONS
+ ):
+ return False
+ memory_id = _text(metadata.get("superseded_by"))
+ prior_turn = turn_index
+ return False
+
+
+def _walk(value: Any, path: str = "") -> Iterable[tuple[str, Any]]:
+ yield path, value
+ if isinstance(value, Mapping):
+ for key, child in value.items():
+ yield from _walk(child, f"{path}.{key}" if path else str(key))
+ elif isinstance(value, list):
+ for index, child in enumerate(value):
+ yield from _walk(child, f"{path}[{index}]")
+
+
+def _has_forbidden_input(value: Any) -> str | None:
+ for path, item in _walk(value):
+ if isinstance(item, Mapping):
+ for key in item:
+ lowered = str(key).casefold()
+ if lowered in FORBIDDEN_INPUT_KEYS or any(token in lowered for token in ("gold_", "benchmark_", "judge_")):
+ return f"{path}.{key}"
+ return None
+
+
+def _artifact_markers(value: Any, path: str = "") -> list[str]:
+ found: list[str] = []
+ for current, item in _walk(value, path):
+ if isinstance(item, Mapping):
+ for key in item:
+ if any(part in str(key).casefold() for part in FORBIDDEN_ARTIFACT_KEY_PARTS):
+ found.append(f"{current}.{key}")
+ elif (
+ isinstance(item, str)
+ and not any(part in current.casefold() for part in SEMANTIC_IDENTIFIER_PATH_PARTS)
+ and ".source_spans[" not in current.casefold()
+ and (
+ not current
+ or any(
+ part in current.casefold()
+ for part in OPERATIONAL_MARKER_VALUE_PATH_PARTS
+ )
+ )
+ and MARKER_RE.search(item)
+ ):
+ found.append(current)
+ return found
+
+
+def _slow_pro_route_is_justified(metadata: Mapping[str, Any]) -> bool:
+ route = _text(metadata.get("route"))
+ if route not in {"pro", "flash_to_pro"}:
+ return True
+ reason = _text(metadata.get("route_reason"))
+ if CONFLICT_RE.search(reason):
+ return True
+ reason_codes = {item for item in reason.split("+") if item}
+ partition_ids = metadata.get("semantic_partition_capsule_ids")
+ migration = (
+ route == "pro"
+ and "semantic_partition_migration" in reason_codes
+ and metadata.get("semantic_partition_contract_version")
+ == CURRENT_SLOW_PARTITION_CONTRACT_VERSION
+ and metadata.get("semantic_partition_mode") == "migrate"
+ and isinstance(partition_ids, list)
+ and bool(partition_ids)
+ and all(isinstance(item, str) and bool(item.strip()) for item in partition_ids)
+ and len(partition_ids) == len(set(partition_ids))
+ )
+ required_ids = metadata.get("required_operation_evidence_ids")
+ semantic_management = (
+ route == "pro"
+ and bool(
+ reason_codes
+ & {
+ "generic_region_semantic_management",
+ "initial_multi_slot_semantic_partition",
+ }
+ )
+ and metadata.get("semantic_partition_contract_version")
+ == CURRENT_SLOW_PARTITION_CONTRACT_VERSION
+ and metadata.get("semantic_partition_mode") == "manage"
+ and not partition_ids
+ and isinstance(required_ids, list)
+ and bool(required_ids)
+ and all(isinstance(item, str) and bool(item.strip()) for item in required_ids)
+ and len(required_ids) == len(set(required_ids))
+ )
+ return migration or semantic_management
+
+
+def _retrieval_forbidden(value: Any, path: str = "") -> list[str]:
+ found: list[str] = []
+ for current, item in _walk(value, path):
+ if isinstance(item, Mapping):
+ for key, child in item.items():
+ lowered = str(key).casefold()
+ if lowered == "runtime_input_has_gold" and child is False:
+ continue
+ if any(part in lowered for part in FORBIDDEN_RETRIEVAL_KEY_PARTS):
+ found.append(f"{current}.{key}")
+ return found
+
+
+def discover_worker_databases(run_dir: Path) -> dict[str, list[Path]]:
+ """Return the frozen manifest DBs, or recursively discover legacy runs."""
+ run_dir = Path(run_dir).resolve()
+ manifest = run_dir / "scope_manifest.jsonl"
+ if manifest.is_file():
+ result: dict[str, list[Path]] = defaultdict(list)
+ seen_paths: set[Path] = set()
+ try:
+ lines = manifest.read_text(encoding="utf-8").splitlines()
+ except OSError as exc:
+ raise AuditError(f"cannot read frozen scope manifest: {manifest}") from exc
+ if not lines:
+ raise AuditError(f"frozen scope manifest is empty: {manifest}")
+ for line_number, raw in enumerate(lines, start=1):
+ try:
+ row = json.loads(raw)
+ except json.JSONDecodeError as exc:
+ raise AuditError(
+ f"frozen scope manifest line {line_number} is invalid JSON"
+ ) from exc
+ if not isinstance(row, Mapping):
+ raise AuditError(
+ f"frozen scope manifest line {line_number} is not an object"
+ )
+ raw_path = _text(row.get("db_path"))
+ path = Path(raw_path)
+ if not path.is_absolute():
+ path = run_dir / path
+ path = path.resolve()
+ try:
+ relative = path.relative_to(run_dir)
+ except ValueError as exc:
+ raise AuditError(
+ f"frozen worker database is outside the run directory: {path}"
+ ) from exc
+ if not path.is_file():
+ raise AuditError(f"frozen worker database does not exist: {path}")
+ if path in seen_paths:
+ raise AuditError(f"frozen worker database is duplicated: {path}")
+ worker = next(
+ (part for part in relative.parts[:-1] if part.startswith("worker")),
+ "",
+ )
+ if not worker:
+ raise AuditError(f"frozen database has no worker directory: {path}")
+ if result[worker]:
+ raise AuditError(f"frozen manifest repeats worker {worker}")
+ seen_paths.add(path)
+ result[worker].append(path)
+ return {key: values for key, values in sorted(result.items())}
+
+ result: dict[str, list[Path]] = defaultdict(list)
+ for path in sorted(run_dir.rglob("*")):
+ if not path.is_file() or path.suffix.casefold() not in {".sqlite3", ".sqlite", ".db"}:
+ continue
+ if path.name.startswith(".") or path.name.startswith("tmp_"):
+ continue
+ relative = path.relative_to(run_dir)
+ worker = next((part for part in relative.parts[:-1] if part.startswith("worker")), "root")
+ result[worker].append(path)
+ return {key: sorted(values) for key, values in sorted(result.items())}
+
+
+def _parse_worker_db_specs(run_dir: Path, specs: Iterable[str]) -> dict[str, list[Path]]:
+ result: dict[str, list[Path]] = defaultdict(list)
+ for spec in specs:
+ if "=" not in spec:
+ raise AuditError(f"--worker-db must be WORKER=PATH: {spec!r}")
+ worker, raw_path = spec.split("=", 1)
+ worker = _text(worker)
+ path = Path(raw_path)
+ if not path.is_absolute():
+ path = run_dir / path
+ path = path.resolve()
+ if not worker or not path.is_file():
+ raise AuditError(f"worker DB does not exist: {spec!r}")
+ result[worker].append(path)
+ return {key: sorted(set(values)) for key, values in sorted(result.items())}
+
+
+def _load_input(run_dir: Path) -> tuple[Path, list[Mapping[str, Any]]]:
+ candidates = [
+ run_dir / name for name in ("writer_input.json", "product_writer_input.json", "input.json")
+ ]
+ candidates += sorted(run_dir.rglob("*writer*input*.json"))
+ candidates = list(dict.fromkeys(path for path in candidates if path.is_file()))
+ if not candidates:
+ raise AuditError("writer input artifact is missing")
+ path = candidates[0]
+ try:
+ raw = json.loads(path.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError) as exc:
+ raise AuditError(f"cannot read writer input {path}: {exc}") from exc
+ rows = raw.get("rows") if isinstance(raw, Mapping) and isinstance(raw.get("rows"), list) else raw
+ if not isinstance(rows, list) or not rows or not all(isinstance(row, Mapping) for row in rows):
+ raise AuditError("writer input must be a non-empty JSON array of objects")
+ forbidden = _has_forbidden_input(raw)
+ if forbidden:
+ raise AuditError(f"writer input contains non-history field: {forbidden}")
+ return path, list(rows)
+
+
+def _input_message_inventory(
+ rows: Iterable[Mapping[str, Any]],
+) -> tuple[
+ dict[tuple[str, str], dict[str, Any]],
+ dict[tuple[str, str], dict[str, Any]],
+]:
+ expected: dict[tuple[str, str], dict[str, Any]] = {}
+ excluded: dict[tuple[str, str], dict[str, Any]] = {}
+ for row_index, row in enumerate(rows):
+ qid = _text(row.get("question_id")) or f"row{row_index:04d}"
+ scope = _text(row.get("scope_id")) or f"tmcra_v4:{qid}"
+ if "haystack_sessions" in row:
+ sessions = list(row.get("haystack_sessions") or [])
+ session_ids = list(row.get("haystack_session_ids") or [])
+ elif "sessions" in row:
+ sessions = list(row.get("sessions") or [])
+ session_ids = []
+ else:
+ sessions = [list(row.get("messages") or [])]
+ session_ids = []
+ for session_index, messages in enumerate(sessions):
+ if not isinstance(messages, list):
+ raise AuditError(f"input row {row_index} session {session_index} is not a message list")
+ session_id = (_text(session_ids[session_index]) if session_index < len(session_ids) else "") or _text(row.get("session_id")) or f"session-{session_index:03d}"
+ for message_index, message in enumerate(messages):
+ if not isinstance(message, Mapping):
+ raise AuditError(f"input message {row_index}/{session_index}/{message_index} is not an object")
+ content = str(message.get("content") or "")
+ role = _text(message.get("role")).casefold()
+ message_id = _text(message.get("message_id")) or f"s{session_index:03d}_m{message_index:03d}"
+ if role not in {"user", "assistant", "system", "tool"}:
+ raise AuditError(f"invalid input message {scope}/{message_id}")
+ key = (scope, message_id)
+ if not content.strip():
+ value = {
+ "scope_id": scope,
+ "session_id": session_id,
+ "session_index": session_index,
+ "message_index": message_index,
+ "message_id": message_id,
+ "message_role": role,
+ "reason": "empty_content",
+ "content_sha256": _sha(content),
+ }
+ if key in excluded and excluded[key] != value:
+ raise AuditError(
+ f"duplicate excluded input message with conflicting content: {key}"
+ )
+ excluded[key] = value
+ continue
+ value = {"scope_id": scope, "session_id": session_id, "message_id": message_id, "content": content, "role": role, "session_index": session_index, "message_index": message_index}
+ if key in expected and expected[key] != value:
+ raise AuditError(f"duplicate input message with conflicting content: {key}")
+ expected[key] = value
+ return expected, excluded
+
+
+def _expected_messages(rows: Iterable[Mapping[str, Any]]) -> dict[tuple[str, str], dict[str, Any]]:
+ return _input_message_inventory(rows)[0]
+
+
+def _audit_source_exclusions(
+ run_dir: Path,
+ expected: Mapping[tuple[str, str], Mapping[str, Any]],
+ issues: list[str],
+) -> None:
+ observed: dict[tuple[str, str], dict[str, Any]] = {}
+ paths = sorted(run_dir.rglob("source_exclusions.json"))
+ for path in paths:
+ try:
+ payload = json.loads(path.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError) as exc:
+ issues.append(f"{path}: invalid source exclusion artifact: {exc}")
+ continue
+ if (
+ not isinstance(payload, Mapping)
+ or payload.get("schema_version") != "tmcra.v4.source-exclusions.1"
+ or not isinstance(payload.get("messages"), list)
+ or payload.get("count") != len(payload["messages"])
+ ):
+ issues.append(f"{path}: invalid source exclusion schema")
+ continue
+ for item in payload["messages"]:
+ if not isinstance(item, Mapping):
+ issues.append(f"{path}: source exclusion entry is not an object")
+ continue
+ key = (_text(item.get("scope_id")), _text(item.get("message_id")))
+ normalized = dict(item)
+ old = observed.get(key)
+ if old is not None and old != normalized:
+ issues.append(f"{path}: conflicting source exclusion entry {key}")
+ else:
+ observed[key] = normalized
+ if set(observed) != set(expected):
+ missing = sorted(set(expected) - set(observed))
+ unexpected = sorted(set(observed) - set(expected))
+ if missing:
+ issues.append(f"source exclusion artifact lacks empty input messages: {missing}")
+ if unexpected:
+ issues.append(f"source exclusion artifact contains unexpected messages: {unexpected}")
+ for key in sorted(set(observed) & set(expected)):
+ if observed[key] != dict(expected[key]):
+ issues.append(f"source exclusion metadata differs for {key}")
+
+
+def _source_parent(meta: Mapping[str, Any], row: Mapping[str, Any], source_id: str) -> dict[str, Any]:
+ raw = meta.get("source_parent", meta.get("parent"))
+ if isinstance(raw, Mapping):
+ parent = dict(raw)
+ parent.setdefault("source_record_id", source_id)
+ return parent
+ parent = {"source_record_id": _text(meta.get("source_record_id")) or source_id}
+ for key in ("session_index", "parent_chunk_index", "message_index", "event_id", "evidence_char_start", "evidence_char_end"):
+ if key in meta:
+ parent[key] = meta[key]
+ return parent
+
+
+def _source_rows(db_paths: Iterable[Path], expected_scopes: set[str]) -> tuple[dict[Identity, dict[str, Any]], list[str], set[str]]:
+ sources: dict[Identity, dict[str, Any]] = {}
+ errors: list[str] = []
+ seen_tables: set[str] = set()
+ for path in db_paths:
+ with sqlite3.connect(path) as con:
+ con.row_factory = sqlite3.Row
+ tables = _tables(con)
+ for table in SOURCE_TABLES:
+ if table not in tables:
+ continue
+ seen_tables.add(table)
+ for row in _rows(con, table):
+ scope = _text(row.get("scope_id"))
+ meta = _metadata(row)
+ if scope not in expected_scopes:
+ continue
+ if meta.get("content_variant") != "source_message" or meta.get("node_kind") != "immutable_source_message":
+ continue
+ message_id = _text(meta.get("message_id"))
+ raw_content = meta.get("raw_content")
+ if not isinstance(raw_content, str) or not raw_content:
+ errors.append(f"{path}:{table}: actual source lacks exact metadata.raw_content")
+ continue
+ content = raw_content
+ for field in ("source_span", "source_turn_text"):
+ if meta.get(field) != content:
+ errors.append(
+ f"{path}:{table}: actual source {scope}/{message_id} {field} differs from raw_content"
+ )
+ stored_value = str(row.get("value") or "")
+ if " ".join(stored_value.split()) != " ".join(content.split()):
+ errors.append(
+ f"{path}:{table}: actual source {scope}/{message_id} graph value differs from raw_content"
+ )
+ if not scope or not message_id or not content:
+ errors.append(f"{path}:{table}: actual source record lacks message/content")
+ continue
+ source_id = _text(meta.get("source_record_id")) or _text(row.get("memory_id"))
+ status = _text(meta.get("enrichment_status"))
+ if status not in {"pending", "failed", "enriched"}:
+ errors.append(f"{path}: actual source {scope}/{message_id} has unknown enrichment status")
+ item = {"source_record_id": source_id, "scope_id": scope, "message_id": message_id, "content": content, "role": _text(meta.get("role") or meta.get("speaker")).casefold(), "status": status, "turn_index": row.get("turn_index"), "metadata": meta, "db_path": _db_path(path)}
+ key = (_db_path(path), scope, source_id)
+ old = sources.get(key)
+ if not source_id:
+ errors.append(f"{path}: actual source {key} lacks source_record_id")
+ if old and (old["content"] != content or old["source_record_id"] != item["source_record_id"]):
+ errors.append(f"{path}:{table}: conflicting immutable source {key}")
+ else:
+ sources[key] = item
+ return sources, errors, seen_tables
+
+
+def _collect_jsonl(run_dir: Path, names: tuple[str, ...]) -> list[tuple[Path, dict[str, Any]]]:
+ result: list[tuple[Path, dict[str, Any]]] = []
+ for path in sorted(run_dir.rglob("*.jsonl")):
+ if path.name not in names:
+ continue
+ relative = path.relative_to(run_dir)
+ if len(relative.parts) > 1:
+ canonical_worker = (
+ re.fullmatch(r"worker_\d+", relative.parts[0]) is not None
+ or (
+ len(relative.parts) >= 3
+ and relative.parts[0] == "writer"
+ and re.fullmatch(r"worker_\d+", relative.parts[1]) is not None
+ )
+ )
+ if not canonical_worker:
+ continue
+ for index, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1):
+ try:
+ value = json.loads(line)
+ except json.JSONDecodeError as exc:
+ raise AuditError(f"invalid JSONL {path}:{index}: {exc}") from exc
+ if not isinstance(value, dict):
+ raise AuditError(f"JSONL object required at {path}:{index}")
+ result.append((path, value))
+ return result
+
+
+def _audit_writer_input(run_dir: Path, expected: Mapping[tuple[str, str], Mapping[str, Any]], db_paths: list[Path], issues: list[str]) -> None:
+ calls = _collect_jsonl(run_dir, ("product_writer_calls.jsonl", "writer_calls.jsonl"))
+ raw_calls = _collect_jsonl(run_dir, ("product_writer_raw_responses.jsonl",))
+ revalidations = _collect_jsonl(run_dir, ("product_writer_revalidations.jsonl",))
+ reconciliation_revalidations = _collect_jsonl(
+ run_dir, ("product_writer_reconciliation_revalidations.jsonl",)
+ )
+ validated_batch_recoveries = _collect_jsonl(
+ run_dir, ("product_writer_validated_batch_recoveries.jsonl",)
+ )
+ historical_binding_recoveries = _collect_jsonl(
+ run_dir, ("product_writer_historical_binding_recoveries.jsonl",)
+ )
+ keep_parallel_migrations = _collect_jsonl(
+ run_dir, ("product_writer_keep_parallel_migrations.jsonl",)
+ )
+ challenge_migrations = _collect_jsonl(
+ run_dir, ("product_writer_challenge_lifecycle_migrations.jsonl",)
+ )
+ interrupted_calls = _collect_jsonl(
+ run_dir, ("product_writer_interrupted_calls.jsonl",)
+ )
+ revalidation_by_batch: dict[str, tuple[Path, dict[str, Any]]] = {}
+ for path, record in revalidations:
+ batch_id = _text(record.get("batch_id"))
+ if not batch_id or batch_id in revalidation_by_batch:
+ issues.append(f"writer revalidation has missing or duplicate batch ID: {path}")
+ continue
+ if int(record.get("physical_api_calls", -1)) != 0:
+ issues.append(f"writer revalidation claims a physical API call: {path}:{batch_id}")
+ if not _text(record.get("raw_response_sha256")) or not _text(
+ record.get("validated_response_sha256")
+ ):
+ issues.append(f"writer revalidation hashes are incomplete: {path}:{batch_id}")
+ revalidation_by_batch[batch_id] = (path, record)
+ reconciliation_revalidation_by_job: dict[
+ str, tuple[Path, dict[str, Any]]
+ ] = {}
+ for path, record in reconciliation_revalidations:
+ job_id = _text(record.get("job_id"))
+ if not job_id or job_id in reconciliation_revalidation_by_job:
+ issues.append(
+ f"reconciliation revalidation has missing or duplicate job ID: {path}"
+ )
+ continue
+ if int(record.get("physical_api_calls", -1)) != 0:
+ issues.append(
+ f"reconciliation revalidation claims a physical API call: {path}:{job_id}"
+ )
+ if not _text(record.get("raw_response_sha256")) or not _text(
+ record.get("normalized_adjudication_sha256")
+ ):
+ issues.append(
+ f"reconciliation revalidation hashes are incomplete: {path}:{job_id}"
+ )
+ reconciliation_revalidation_by_job[job_id] = (path, record)
+ validated_recovery_by_batch: dict[str, tuple[Path, dict[str, Any]]] = {}
+ for path, record in validated_batch_recoveries:
+ batch_id = _text(record.get("batch_id"))
+ if not batch_id or batch_id in validated_recovery_by_batch:
+ issues.append(
+ f"validated batch recovery has missing or duplicate batch ID: {path}"
+ )
+ continue
+ if int(record.get("physical_api_calls", -1)) != 0:
+ issues.append(
+ f"validated batch recovery claims a physical API call: {path}:{batch_id}"
+ )
+ if not _text(record.get("prior_error_sha256")) or not _text(
+ record.get("response_sha256")
+ ):
+ issues.append(
+ f"validated batch recovery hashes are incomplete: {path}:{batch_id}"
+ )
+ validated_recovery_by_batch[batch_id] = (path, record)
+ historical_recovery_by_job: dict[str, tuple[Path, dict[str, Any]]] = {}
+ for path, record in historical_binding_recoveries:
+ job_id = _text(record.get("job_id"))
+ if not job_id or job_id in historical_recovery_by_job:
+ issues.append(
+ f"historical binding recovery has missing or duplicate job ID: {path}"
+ )
+ continue
+ binding_mode = _text(record.get("binding_mode")) or (
+ "verified_historical_selected"
+ if _text(record.get("resolved_memory_id"))
+ in {"", _text(record.get("selected_memory_id"))}
+ else ""
+ )
+ mode_invalid = False
+ if binding_mode == "verified_historical_selected":
+ mode_invalid = _text(record.get("resolved_memory_id")) not in {
+ "",
+ _text(record.get("selected_memory_id")),
+ }
+ elif binding_mode == "unique_active_semantic_equivalent":
+ mode_invalid = (
+ not _text(record.get("resolved_memory_id"))
+ or _text(record.get("resolved_memory_id"))
+ == _text(record.get("selected_memory_id"))
+ or not _text(record.get("frozen_semantic_identity_sha256"))
+ or _text(record.get("frozen_semantic_identity_sha256"))
+ != _text(record.get("resolved_semantic_identity_sha256"))
+ )
+ else:
+ mode_invalid = True
+ if (
+ int(record.get("physical_api_calls", -1)) != 0
+ or _text(record.get("decision"))
+ not in {"replace_current", "keep_parallel", "challenge", "quarantine"}
+ or _text(record.get("superseded_reason"))
+ != "v4_reconciliation_replace_current"
+ or not _text(record.get("selected_memory_id"))
+ or not _text(record.get("frozen_binding_identity_sha256"))
+ or _text(record.get("frozen_binding_identity_sha256"))
+ != _text(record.get("historical_binding_identity_sha256"))
+ or mode_invalid
+ ):
+ issues.append(
+ f"historical binding recovery contract is malformed: {path}:{job_id}"
+ )
+ historical_recovery_by_job[job_id] = (path, record)
+ keep_parallel_migration_by_job: dict[
+ str, tuple[Path, dict[str, Any]]
+ ] = {}
+ for path, record in keep_parallel_migrations:
+ job_id = _text(record.get("job_id"))
+ if not job_id or job_id in keep_parallel_migration_by_job:
+ issues.append(
+ f"keep_parallel migration has missing or duplicate job ID: {path}"
+ )
+ continue
+ if (
+ _text(record.get("schema_version"))
+ != KEEP_PARALLEL_MIGRATION_SCHEMA_VERSION
+ or _text(record.get("migration_version"))
+ != KEEP_PARALLEL_MIGRATION_VERSION
+ or _text(record.get("status")) != "completed"
+ or int(record.get("physical_api_calls", -1)) != 0
+ or not _text(record.get("migration_id"))
+ or not _text(record.get("selected_memory_id"))
+ or not _text(record.get("resolved_memory_id"))
+ or not _text(record.get("incoming_memory_id"))
+ or _text(record.get("assertion_binding_mode"))
+ not in {
+ "original_assertion_index",
+ "exact_evidence_identity_after_commit_reindex",
+ }
+ or not re.fullmatch(r"[0-9a-f]{64}", _text(record.get("before_sha256")))
+ or not re.fullmatch(r"[0-9a-f]{64}", _text(record.get("after_sha256")))
+ ):
+ issues.append(
+ f"keep_parallel migration contract is malformed: {path}:{job_id}"
+ )
+ keep_parallel_migration_by_job[job_id] = (path, record)
+ challenge_migration_by_job: dict[
+ str, tuple[Path, dict[str, Any]]
+ ] = {}
+ for path, record in challenge_migrations:
+ job_id = _text(record.get("job_id"))
+ if not job_id or job_id in challenge_migration_by_job:
+ issues.append(
+ f"challenge migration has missing or duplicate job ID: {path}"
+ )
+ continue
+ if (
+ _text(record.get("schema_version"))
+ != CHALLENGE_MIGRATION_SCHEMA_VERSION
+ or _text(record.get("migration_version"))
+ != CHALLENGE_MIGRATION_VERSION
+ or _text(record.get("status")) != "completed"
+ or int(record.get("physical_api_calls", -1)) != 0
+ or not _text(record.get("migration_id"))
+ or not _text(record.get("selected_memory_id"))
+ or not _text(record.get("resolved_memory_id"))
+ or not _text(record.get("incoming_memory_id"))
+ or _text(record.get("assertion_binding_mode"))
+ not in {
+ "original_assertion_index",
+ "exact_evidence_identity_after_commit_reindex",
+ }
+ or _text(record.get("restored_slot_head"))
+ != _text(record.get("resolved_memory_id"))
+ or not re.fullmatch(r"[0-9a-f]{64}", _text(record.get("before_sha256")))
+ or not re.fullmatch(r"[0-9a-f]{64}", _text(record.get("after_sha256")))
+ ):
+ issues.append(
+ f"challenge migration contract is malformed: {path}:{job_id}"
+ )
+ challenge_migration_by_job[job_id] = (path, record)
+ interrupted_by_key: dict[str, tuple[Path, dict[str, Any]]] = {}
+ for path, record in interrupted_calls:
+ call_key = _text(record.get("call_key"))
+ if not call_key or call_key in interrupted_by_key:
+ issues.append(
+ f"interrupted writer call has missing or duplicate call key: {path}"
+ )
+ continue
+ for marker in _artifact_markers(record):
+ issues.append(f"interrupted writer artifact marker {path}:{marker}")
+ if (
+ record.get("physical_api_call") is not True
+ or int(record.get("physical_api_calls", 0) or 0) != 1
+ or _text(record.get("status"))
+ != "outcome_unknown_after_confirmed_process_loss"
+ or record.get("same_model_replacement") is not True
+ or _text(record.get("replacement_model")) != _text(record.get("model"))
+ or not _text(record.get("physical_call_id"))
+ ):
+ issues.append(
+ f"interrupted writer call recovery contract is malformed: {path}:{call_key}"
+ )
+ interrupted_by_key[call_key] = (path, record)
+ raw_by_key: dict[str, tuple[Path, dict[str, Any]]] = {}
+ for path, raw_call in raw_calls:
+ call_key = _text(raw_call.get("call_key"))
+ if not call_key or call_key in raw_by_key:
+ issues.append(f"raw writer response has missing or duplicate call key: {path}")
+ continue
+ raw_response = raw_call.get("raw_response")
+ if not isinstance(raw_response, str) or not raw_response:
+ issues.append(f"raw writer response is empty: {path}:{call_key}")
+ continue
+ if _sha(raw_response) != _text(raw_call.get("raw_response_sha256")):
+ issues.append(f"raw writer response hash differs: {path}:{call_key}")
+ metadata_hash = _text(raw_call.get("metadata_response_sha256"))
+ if metadata_hash and metadata_hash != _sha(raw_response):
+ issues.append(f"raw writer response differs from API metadata hash: {path}:{call_key}")
+ raw_by_key[call_key] = (path, raw_call)
+ calls_by_key: dict[str, list[tuple[Path, dict[str, Any]]]] = defaultdict(list)
+ for path, call in calls:
+ for marker in _artifact_markers(call):
+ issues.append(f"writer artifact marker {path}:{marker}")
+ model = _text(call.get("model"))
+ metadata = call.get("metadata") if isinstance(call.get("metadata"), Mapping) else {}
+ call_key = _text(call.get("call_key"))
+ calls_by_key[call_key].append((path, call))
+ if _text(metadata.get("status")) == "completed" and call_key not in raw_by_key:
+ issues.append(f"completed writer call lacks raw response: {path}:{call_key}")
+ if "pro" in model.casefold():
+ route = _text(call.get("route") or call.get("routing_reason") or call.get("stage") or metadata.get("route") or metadata.get("stage") or metadata.get("routing_reason"))
+ if not re.search(r"reconcil|conflict|same[_ -]?slot", route, re.I):
+ issues.append(f"Pro writer call is not a reconciliation route: {path}")
+ for call_key, (path, interrupted) in interrupted_by_key.items():
+ replacements = calls_by_key.get(call_key, [])
+ if len(replacements) != 1:
+ issues.append(
+ f"interrupted writer call lacks exactly one replacement: {path}:{call_key}"
+ )
+ continue
+ replacement_model = _text(replacements[0][1].get("model"))
+ if replacement_model != _text(interrupted.get("model")):
+ issues.append(
+ f"interrupted writer call changed model: {path}:{call_key}"
+ )
+ if call_key not in raw_by_key:
+ issues.append(
+ f"interrupted writer replacement lacks raw response: {path}:{call_key}"
+ )
+ seen_historical_recovery_jobs: set[str] = set()
+ seen_keep_parallel_migration_jobs: set[str] = set()
+ seen_keep_parallel_journal_jobs: set[str] = set()
+ seen_challenge_migration_jobs: set[str] = set()
+ seen_challenge_journal_jobs: set[str] = set()
+ for path in db_paths:
+ with sqlite3.connect(path) as con:
+ con.row_factory = sqlite3.Row
+ tables = _tables(con)
+ if "v4_batch_journal" not in tables:
+ continue
+ migration_journal_by_job: dict[str, dict[str, Any]] = {}
+ challenge_journal_by_job: dict[str, dict[str, Any]] = {}
+ if "v4_keep_parallel_migrations" in tables:
+ for journal in _rows(con, "v4_keep_parallel_migrations"):
+ journal_job_id = _text(journal.get("job_id"))
+ if (
+ not journal_job_id
+ or journal_job_id in seen_keep_parallel_journal_jobs
+ ):
+ issues.append(
+ f"{path}: keep_parallel migration journal has missing or duplicate job ID"
+ )
+ continue
+ seen_keep_parallel_journal_jobs.add(journal_job_id)
+ migration_journal_by_job[journal_job_id] = journal
+ artifact = keep_parallel_migration_by_job.get(journal_job_id)
+ journal_artifact = _json(journal.get("artifact_json"))
+ before_json = _text(journal.get("before_json"))
+ after_json = _text(journal.get("after_json"))
+ if (
+ artifact is None
+ or _text(journal.get("status")) != "completed"
+ or not isinstance(journal_artifact, Mapping)
+ or dict(journal_artifact) != artifact[1]
+ or _sha(before_json)
+ != _text(journal_artifact.get("before_sha256"))
+ or _sha(after_json)
+ != _text(journal_artifact.get("after_sha256"))
+ ):
+ issues.append(
+ f"{path}: keep_parallel migration journal/artifact differs: {journal_job_id}"
+ )
+ if "v4_challenge_lifecycle_migrations" in tables:
+ for journal in _rows(con, "v4_challenge_lifecycle_migrations"):
+ journal_job_id = _text(journal.get("job_id"))
+ if (
+ not journal_job_id
+ or journal_job_id in seen_challenge_journal_jobs
+ ):
+ issues.append(
+ f"{path}: challenge migration journal has missing or duplicate job ID"
+ )
+ continue
+ seen_challenge_journal_jobs.add(journal_job_id)
+ challenge_journal_by_job[journal_job_id] = journal
+ artifact = challenge_migration_by_job.get(journal_job_id)
+ journal_artifact = _json(journal.get("artifact_json"))
+ before_json = _text(journal.get("before_json"))
+ after_json = _text(journal.get("after_json"))
+ if (
+ artifact is None
+ or _text(journal.get("status")) != "completed"
+ or not isinstance(journal_artifact, Mapping)
+ or dict(journal_artifact) != artifact[1]
+ or _sha(before_json)
+ != _text(journal_artifact.get("before_sha256"))
+ or _sha(after_json)
+ != _text(journal_artifact.get("after_sha256"))
+ ):
+ issues.append(
+ f"{path}: challenge migration journal/artifact differs: {journal_job_id}"
+ )
+ if "v4_reconciliation_jobs" in tables:
+ for job in _rows(con, "v4_reconciliation_jobs"):
+ job_id = _text(job.get("job_id"))
+ response = _json(job.get("response_json"))
+ migration_artifact = keep_parallel_migration_by_job.get(job_id)
+ if migration_artifact is not None:
+ seen_keep_parallel_migration_jobs.add(job_id)
+ if (
+ _text(job.get("status")) == "completed"
+ and _text(job.get("decision")) == "keep_parallel"
+ and isinstance(response, Mapping)
+ ):
+ selected_memory_id = _text(
+ response.get("selected_memory_id")
+ )
+ historical_artifact = historical_recovery_by_job.get(job_id)
+ resolved_memory_id = (
+ _text(
+ historical_artifact[1].get("resolved_memory_id")
+ )
+ if historical_artifact is not None
+ else ""
+ ) or selected_memory_id
+ scope_id = _text(job.get("scope_id"))
+ selected = con.execute(
+ "SELECT state,slot_key,turn_index,supersedes_json,metadata_json "
+ "FROM records WHERE scope_id=? AND memory_id=?",
+ (scope_id, resolved_memory_id),
+ ).fetchone() if "records" in tables else None
+ immediate_overwrite = False
+ selected_metadata: Mapping[str, Any] = {}
+ incoming = None
+ incoming_metadata: Mapping[str, Any] = {}
+ if selected is not None:
+ selected_metadata_value = _json(selected[4])
+ if isinstance(selected_metadata_value, Mapping):
+ selected_metadata = selected_metadata_value
+ incoming_id = _text(
+ selected_metadata.get("superseded_by")
+ )
+ if incoming_id:
+ incoming = con.execute(
+ "SELECT state,slot_key,turn_index,supersedes_json,metadata_json "
+ "FROM records WHERE scope_id=? AND memory_id=?",
+ (scope_id, incoming_id),
+ ).fetchone()
+ if incoming is not None:
+ incoming_metadata_value = _json(incoming[4])
+ if isinstance(incoming_metadata_value, Mapping):
+ incoming_metadata = incoming_metadata_value
+ immediate_overwrite = (
+ _text(selected[0]) == "superseded"
+ and _text(
+ incoming_metadata.get("message_id")
+ )
+ == _text(job.get("message_id"))
+ and _text(
+ incoming_metadata.get(
+ "reconciliation_decision"
+ )
+ )
+ == "keep_parallel"
+ )
+ if immediate_overwrite:
+ issues.append(
+ f"{path}: keep_parallel decision was overwritten by graph policy: {job_id}"
+ )
+ if migration_artifact is not None:
+ artifact = migration_artifact[1]
+ journal = migration_journal_by_job.get(job_id)
+ incoming_id = _text(artifact.get("incoming_memory_id"))
+ migrated_incoming = con.execute(
+ "SELECT slot_key,supersedes_json,metadata_json FROM records "
+ "WHERE scope_id=? AND memory_id=?",
+ (scope_id, incoming_id),
+ ).fetchone() if "records" in tables else None
+ migrated_incoming_metadata = (
+ _json(migrated_incoming[2])
+ if migrated_incoming is not None
+ else {}
+ )
+ migrated_supersedes = (
+ _json(migrated_incoming[1])
+ if migrated_incoming is not None
+ else []
+ )
+ request = _json(job.get("request_json"))
+ cited = (
+ request.get("new_cited_assertion")
+ if isinstance(request, Mapping)
+ else None
+ )
+ binding_mode = _text(
+ artifact.get("assertion_binding_mode")
+ )
+ if binding_mode == "original_assertion_index":
+ binding_mode_valid = (
+ isinstance(
+ migrated_incoming_metadata, Mapping
+ )
+ and int(
+ migrated_incoming_metadata.get(
+ "llm_write_proposal_index", -1
+ )
+ )
+ == int(job.get("assertion_index") or 0)
+ )
+ else:
+ binding_mode_valid = (
+ migrated_incoming is not None
+ and isinstance(request, Mapping)
+ and isinstance(cited, Mapping)
+ and _text(migrated_incoming[0])
+ == _text(request.get("canonical_slot_key"))
+ and _text(
+ migrated_incoming_metadata.get(
+ "source_span"
+ )
+ )
+ == _text(cited.get("evidence_quote"))
+ and _text(
+ migrated_incoming_metadata.get(
+ "memory_type"
+ )
+ )
+ == _text(cited.get("memory_type"))
+ and _text(
+ migrated_incoming_metadata.get(
+ "entity_key"
+ )
+ )
+ == _text(cited.get("entity_key"))
+ and _text(
+ migrated_incoming_metadata.get(
+ "attribute_key"
+ )
+ )
+ == _text(cited.get("attribute_key"))
+ )
+ if (
+ journal is None
+ or _text(artifact.get("selected_memory_id"))
+ != selected_memory_id
+ or _text(artifact.get("resolved_memory_id"))
+ != resolved_memory_id
+ or _text(artifact.get("scope_id")) != scope_id
+ or _text(artifact.get("message_id"))
+ != _text(job.get("message_id"))
+ or int(artifact.get("assertion_index", -1))
+ != int(job.get("assertion_index") or 0)
+ or selected is None
+ or not binding_mode_valid
+ or not _valid_active_to_superseded_lifecycle(
+ con, scope_id, resolved_memory_id
+ )
+ or migrated_incoming is None
+ or not isinstance(migrated_supersedes, list)
+ or resolved_memory_id
+ in {_text(item) for item in migrated_supersedes}
+ or not isinstance(
+ migrated_incoming_metadata, Mapping
+ )
+ or _text(
+ migrated_incoming_metadata.get(
+ "conflict_action"
+ )
+ )
+ != "keep_parallel"
+ or _text(
+ migrated_incoming_metadata.get(
+ "conflict_reason"
+ )
+ )
+ != "v4_reconciliation_keep_parallel"
+ ):
+ issues.append(
+ f"{path}: keep_parallel migration does not match graph/job state: {job_id}"
+ )
+ challenge_artifact = challenge_migration_by_job.get(job_id)
+ if challenge_artifact is not None:
+ seen_challenge_migration_jobs.add(job_id)
+ if not (
+ _text(job.get("status")) == "completed"
+ and _text(job.get("decision")) == "challenge"
+ ):
+ issues.append(
+ f"{path}: challenge migration does not reference a completed challenge job: {job_id}"
+ )
+ if (
+ _text(job.get("status")) == "completed"
+ and _text(job.get("decision")) == "challenge"
+ and isinstance(response, Mapping)
+ ):
+ selected_memory_id = _text(
+ response.get("selected_memory_id")
+ )
+ historical_artifact = historical_recovery_by_job.get(job_id)
+ resolved_memory_id = (
+ _text(
+ historical_artifact[1].get("resolved_memory_id")
+ )
+ if historical_artifact is not None
+ else ""
+ ) or selected_memory_id
+ scope_id = _text(job.get("scope_id"))
+ selected = con.execute(
+ "SELECT state,slot_key,turn_index,supersedes_json,metadata_json "
+ "FROM records WHERE scope_id=? AND memory_id=?",
+ (scope_id, resolved_memory_id),
+ ).fetchone() if "records" in tables else None
+ immediate_overwrite = False
+ if selected is not None:
+ selected_metadata = _json(selected[4])
+ selected_metadata = (
+ selected_metadata
+ if isinstance(selected_metadata, Mapping)
+ else {}
+ )
+ immediate_id = _text(
+ selected_metadata.get("superseded_by")
+ )
+ immediate = con.execute(
+ "SELECT state,metadata_json FROM records "
+ "WHERE scope_id=? AND memory_id=?",
+ (scope_id, immediate_id),
+ ).fetchone() if immediate_id else None
+ immediate_metadata = (
+ _json(immediate[1]) if immediate is not None else {}
+ )
+ immediate_overwrite = (
+ _text(selected[0]) == "superseded"
+ and immediate is not None
+ and _text(immediate[0]) == "challenged"
+ and isinstance(immediate_metadata, Mapping)
+ and _text(immediate_metadata.get("message_id"))
+ == _text(job.get("message_id"))
+ and _text(
+ immediate_metadata.get(
+ "reconciliation_decision"
+ )
+ )
+ == "challenge"
+ )
+ if immediate_overwrite:
+ issues.append(
+ f"{path}: challenge decision was overwritten by graph policy: {job_id}"
+ )
+ if challenge_artifact is not None:
+ artifact = challenge_artifact[1]
+ journal = challenge_journal_by_job.get(job_id)
+ incoming_id = _text(
+ artifact.get("incoming_memory_id")
+ )
+ migrated_incoming = con.execute(
+ "SELECT state,slot_key,supersedes_json,metadata_json "
+ "FROM records WHERE scope_id=? AND memory_id=?",
+ (scope_id, incoming_id),
+ ).fetchone() if "records" in tables else None
+ migrated_incoming_metadata = (
+ _json(migrated_incoming[3])
+ if migrated_incoming is not None
+ else {}
+ )
+ migrated_supersedes = (
+ _json(migrated_incoming[2])
+ if migrated_incoming is not None
+ else []
+ )
+ request = _json(job.get("request_json"))
+ cited = (
+ request.get("new_cited_assertion")
+ if isinstance(request, Mapping)
+ else None
+ )
+ binding_mode = _text(
+ artifact.get("assertion_binding_mode")
+ )
+ if binding_mode == "original_assertion_index":
+ binding_mode_valid = (
+ isinstance(
+ migrated_incoming_metadata, Mapping
+ )
+ and int(
+ migrated_incoming_metadata.get(
+ "llm_write_proposal_index", -1
+ )
+ )
+ == int(job.get("assertion_index") or 0)
+ )
+ else:
+ binding_mode_valid = (
+ migrated_incoming is not None
+ and isinstance(request, Mapping)
+ and isinstance(cited, Mapping)
+ and _text(migrated_incoming[1])
+ == _text(request.get("canonical_slot_key"))
+ and _text(
+ migrated_incoming_metadata.get(
+ "source_span"
+ )
+ )
+ == _text(cited.get("evidence_quote"))
+ and _text(
+ migrated_incoming_metadata.get(
+ "memory_type"
+ )
+ )
+ == _text(cited.get("memory_type"))
+ and _text(
+ migrated_incoming_metadata.get(
+ "entity_key"
+ )
+ )
+ == _text(cited.get("entity_key"))
+ and _text(
+ migrated_incoming_metadata.get(
+ "attribute_key"
+ )
+ )
+ == _text(cited.get("attribute_key"))
+ )
+ if (
+ journal is None
+ or _text(response.get("slot_decision"))
+ != "bind_existing"
+ or _text(response.get("decision")) != "challenge"
+ or _text(artifact.get("selected_memory_id"))
+ != selected_memory_id
+ or _text(artifact.get("resolved_memory_id"))
+ != resolved_memory_id
+ or _text(artifact.get("scope_id")) != scope_id
+ or _text(artifact.get("message_id"))
+ != _text(job.get("message_id"))
+ or int(artifact.get("assertion_index", -1))
+ != int(job.get("assertion_index") or 0)
+ or selected is None
+ or not binding_mode_valid
+ or not _valid_active_to_superseded_lifecycle(
+ con, scope_id, resolved_memory_id
+ )
+ or migrated_incoming is None
+ or _text(migrated_incoming[0]) != "challenged"
+ or not isinstance(migrated_supersedes, list)
+ or resolved_memory_id
+ in {_text(item) for item in migrated_supersedes}
+ or not isinstance(
+ migrated_incoming_metadata, Mapping
+ )
+ or _text(
+ migrated_incoming_metadata.get(
+ "conflict_action"
+ )
+ )
+ != "challenge"
+ or _text(
+ migrated_incoming_metadata.get(
+ "conflict_reason"
+ )
+ )
+ != "v4_reconciliation_challenge"
+ ):
+ issues.append(
+ f"{path}: challenge migration does not match graph/job state: {job_id}"
+ )
+ historical_artifact = historical_recovery_by_job.get(job_id)
+ if historical_artifact is not None:
+ seen_historical_recovery_jobs.add(job_id)
+ if (
+ not isinstance(response, Mapping)
+ or _text(job.get("batch_id"))
+ != _text(historical_artifact[1].get("batch_id"))
+ or _text(response.get("selected_memory_id"))
+ != _text(
+ historical_artifact[1].get("selected_memory_id")
+ )
+ or _text(response.get("decision"))
+ != _text(historical_artifact[1].get("decision"))
+ ):
+ issues.append(
+ f"{path}: historical binding recovery differs from frozen job: {job_id}"
+ )
+ if (
+ _text(historical_artifact[1].get("binding_mode"))
+ == "unique_active_semantic_equivalent"
+ and "records" in tables
+ ):
+ if not _valid_active_to_superseded_lifecycle(
+ con,
+ _text(job.get("scope_id")),
+ _text(
+ historical_artifact[1].get(
+ "resolved_memory_id"
+ )
+ ),
+ ):
+ issues.append(
+ f"{path}: historical binding recovery resolved leaf has no valid active-to-superseded lifecycle: {job_id}"
+ )
+ metadata = _json(job.get("response_metadata_json"))
+ if not (
+ isinstance(metadata, Mapping)
+ and metadata.get("raw_response_revalidated") is True
+ ):
+ continue
+ artifact = reconciliation_revalidation_by_job.get(job_id)
+ if artifact is None:
+ issues.append(
+ f"{path}: revalidated reconciliation lacks recovery artifact: {job_id}"
+ )
+ continue
+ response_json = _text(job.get("response_json"))
+ if (
+ not response_json
+ or _text(
+ artifact[1].get("normalized_adjudication_sha256")
+ )
+ != _sha(response_json)
+ ):
+ issues.append(
+ f"{path}: revalidated reconciliation response hash differs: {job_id}"
+ )
+ for row in _rows(con, "v4_batch_journal"):
+ request = _json(row.get("request_json"))
+ response_metadata = _json(row.get("response_metadata_json"))
+ for marker in _artifact_markers(request) + _artifact_markers(response_metadata):
+ issues.append(f"{path}: writer sidecar contains fallback/retry marker at {marker}")
+ if isinstance(response_metadata, Mapping) and response_metadata.get("raw_response_revalidated") is True:
+ batch_id = _text(row.get("batch_id"))
+ artifact = revalidation_by_batch.get(batch_id)
+ if artifact is None:
+ issues.append(f"{path}: revalidated batch lacks recovery artifact: {batch_id}")
+ elif _text(artifact[1].get("validated_response_sha256")) != _text(row.get("response_sha256")):
+ issues.append(f"{path}: revalidated response hash differs: {batch_id}")
+ if (
+ isinstance(response_metadata, Mapping)
+ and response_metadata.get("validated_batch_commit_recovered") is True
+ ):
+ batch_id = _text(row.get("batch_id"))
+ artifact = validated_recovery_by_batch.get(batch_id)
+ if artifact is None:
+ issues.append(
+ f"{path}: validated batch recovery lacks a recovery artifact: {batch_id}"
+ )
+ elif _text(artifact[1].get("response_sha256")) != _text(
+ row.get("response_sha256")
+ ):
+ issues.append(
+ f"{path}: validated batch recovery response hash differs: {batch_id}"
+ )
+ if not isinstance(request, Mapping):
+ issues.append(f"{path}: batch request is not an object")
+ continue
+ forbidden = _has_forbidden_input(request)
+ if forbidden:
+ issues.append(f"{path}: batch request contains benchmark field {forbidden}")
+ messages = request.get("messages")
+ if not isinstance(messages, list):
+ issues.append(f"{path}: batch request messages missing")
+ continue
+ for item in messages:
+ if not isinstance(item, Mapping):
+ issues.append(f"{path}: batch request message is not an object")
+ continue
+ scope = _text(row.get("scope_id")); key = (scope, _text(item.get("message_id")))
+ source = expected.get(key)
+ if source is None:
+ issues.append(f"{path}: batch request message is not input history: {key}")
+ continue
+ spans = item.get("source_spans")
+ if (
+ not isinstance(spans, list)
+ or not spans
+ or any(
+ not isinstance(span, Mapping)
+ or set(span) != {"span_id", "text"}
+ or not _text(span.get("span_id"))
+ for span in spans
+ )
+ ):
+ issues.append(f"{path}: {key} lacks lossless source spans")
+ continue
+ if "content" in item or "source_tokens" in item or "token_strings" in item:
+ issues.append(f"{path}: {key} repeats full content/token strings")
+ text = "".join(str(span["text"]) for span in spans)
+ if text != source["content"]:
+ issues.append(f"{path}: source spans do not reproduce {key}")
+ span_ids = [_text(span.get("span_id")) for span in spans]
+ if len(set(span_ids)) != len(span_ids):
+ issues.append(f"{path}: source span IDs are not unique for {key}")
+ for job_id, (path, _) in historical_recovery_by_job.items():
+ if job_id not in seen_historical_recovery_jobs:
+ issues.append(
+ f"historical binding recovery does not match a persisted job: {path}:{job_id}"
+ )
+ for job_id, (path, _) in keep_parallel_migration_by_job.items():
+ if job_id not in seen_keep_parallel_migration_jobs:
+ issues.append(
+ f"keep_parallel migration does not match a persisted job: {path}:{job_id}"
+ )
+ if job_id not in seen_keep_parallel_journal_jobs:
+ issues.append(
+ f"keep_parallel migration lacks a committed SQLite journal: {path}:{job_id}"
+ )
+ for job_id, (path, _) in challenge_migration_by_job.items():
+ if job_id not in seen_challenge_migration_jobs:
+ issues.append(
+ f"challenge migration does not match a persisted job: {path}:{job_id}"
+ )
+ if job_id not in seen_challenge_journal_jobs:
+ issues.append(
+ f"challenge migration lacks a committed SQLite journal: {path}:{job_id}"
+ )
+
+
+def _audit_source_before_api(
+ db_paths: list[Path],
+ expected: Mapping[tuple[str, str], Mapping[str, Any]],
+ source_messages: Mapping[SourceMessageKey, Sequence[Mapping[str, Any]]],
+ issues: list[str],
+) -> None:
+ """Use actual graph turn events to establish source persistence before enrichment."""
+ for path in db_paths:
+ with sqlite3.connect(path) as con:
+ con.row_factory = sqlite3.Row
+ tables = _tables(con)
+ if "v4_batch_journal" not in tables:
+ continue
+ if "audit_turn_log" not in tables:
+ issues.append(f"{path}: audit_turn_log is missing; source-before-API is unproven")
+ continue
+ source_turn_events: set[tuple[str, str, str]] = set()
+ for event in _rows(con, "audit_turn_log"):
+ payload = _json(event.get("payload_json"))
+ if isinstance(payload, Mapping):
+ metadata = (
+ payload.get("metadata")
+ if isinstance(payload.get("metadata"), Mapping)
+ else {}
+ )
+ source_record_id = _text(metadata.get("source_record_id"))
+ record_ids = {
+ _text(value)
+ for value in (
+ payload.get("record_ids")
+ if isinstance(payload.get("record_ids"), list)
+ else []
+ )
+ }
+ if source_record_id and source_record_id in record_ids:
+ source_turn_events.add(
+ (
+ _text(event.get("scope_id")),
+ _text(metadata.get("message_id")),
+ source_record_id,
+ )
+ )
+ source_journal = {
+ (_text(row.get("scope_id")), _text(row.get("message_id"))): row
+ for row in _rows(con, "v4_source_journal")
+ } if "v4_source_journal" in tables else {}
+ for batch in _rows(con, "v4_batch_journal"):
+ request = _json(batch.get("request_json"))
+ if not isinstance(request, Mapping) or not batch.get("api_started_at") and _text(batch.get("status")) == "prepared":
+ continue
+ scope = _text(batch.get("scope_id"))
+ for item in list(request.get("messages") or []):
+ message_id = _text(item.get("message_id")) if isinstance(item, Mapping) else ""
+ key = (scope, message_id)
+ source = _source_for_message(
+ source_messages,
+ db_path=path,
+ scope_id=scope,
+ message_id=key[1],
+ )
+ if source is None:
+ issues.append(f"{path}: API batch lacks actual source record {key}")
+ continue
+ journal = source_journal.get(key)
+ if journal is None:
+ issues.append(f"{path}: source journal is missing for API batch message {key}")
+ elif _text(batch.get("api_started_at")):
+ persisted_at = _text(journal.get("source_persisted_at"))
+ if not persisted_at or persisted_at > _text(batch.get("api_started_at")):
+ issues.append(
+ f"{path}: source persistence was not recorded before API start for {key}"
+ )
+ if (
+ scope,
+ message_id,
+ _text(source.get("source_record_id")),
+ ) not in source_turn_events:
+ issues.append(f"{path}: actual source turn event is missing for API batch message {key}")
+
+
+def audit_run(
+ run_dir: Path,
+ *,
+ output: Path | None = None,
+ worker_db_specs: Iterable[str] = (),
+ retrieval_dir: Path | None = None,
+ build_only: bool = False,
+) -> dict[str, Any]:
+ run_dir = Path(run_dir).resolve()
+ issues: list[str] = []
+ try:
+ input_path, input_rows = _load_input(run_dir)
+ expected, excluded_empty = _input_message_inventory(input_rows)
+ except AuditError as exc:
+ report = {"schema_version": "tmcra.v4.chain-audit.1", "status": "failed", "passed": False, "issues": [str(exc)]}
+ if output:
+ Path(output).parent.mkdir(parents=True, exist_ok=True)
+ Path(output).write_text(json.dumps(report, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
+ return report
+ worker_db_specs = list(worker_db_specs)
+ _audit_source_exclusions(run_dir, excluded_empty, issues)
+ db_groups = _parse_worker_db_specs(run_dir, worker_db_specs) if worker_db_specs else discover_worker_databases(run_dir)
+ db_paths = sorted({path for paths in db_groups.values() for path in paths})
+ if not db_paths:
+ issues.append("no per-worker SQLite databases discovered")
+ expected_scopes = {key[0] for key in expected}
+ sources, source_errors, source_tables = _source_rows(db_paths, expected_scopes)
+ source_messages, scoped_source_messages = _source_message_indices(sources)
+ issues.extend(source_errors)
+ graph_tables: set[str] = set()
+ for path in db_paths:
+ with sqlite3.connect(path) as con:
+ con.row_factory = sqlite3.Row
+ tables = _tables(con)
+ graph_tables.update(tables)
+ shadow_tables = sorted(FORBIDDEN_SHADOW_TABLES.intersection(tables))
+ if shadow_tables:
+ issues.append(
+ f"{path}: forbidden graph shadow tables exist: {','.join(shadow_tables)}"
+ )
+ if "v4_batch_journal" in tables:
+ missing = sorted(REQUIRED_GRAPH_TABLES - tables)
+ if missing:
+ issues.append(
+ f"{path}: writer database lacks actual graph tables: {','.join(missing)}"
+ )
+ for batch_row in _rows(con, "v4_batch_journal"):
+ if _text(batch_row.get("status")) != "committed":
+ issues.append(
+ f"{path}: writer batch {_text(batch_row.get('batch_id'))} is not committed"
+ )
+ if "v4_source_journal" not in tables:
+ issues.append(f"{path}: v4_source_journal is missing")
+ else:
+ for journal in _rows(con, "v4_source_journal"):
+ key = (_text(journal.get("scope_id")), _text(journal.get("message_id")))
+ source = _source_for_message(
+ source_messages,
+ db_path=path,
+ scope_id=key[0],
+ message_id=key[1],
+ )
+ if _text(journal.get("status")) != "enriched":
+ issues.append(f"{path}: source journal is not enriched for {key}")
+ if source is None or _text(journal.get("source_record_id")) != _text(source.get("source_record_id")):
+ issues.append(f"{path}: source journal does not target actual graph source for {key}")
+ if not _text(journal.get("source_persisted_at")):
+ issues.append(f"{path}: source persistence timestamp is missing for {key}")
+ missing_graph_tables = sorted(REQUIRED_GRAPH_TABLES - graph_tables)
+ if missing_graph_tables:
+ issues.append("actual TMCRA graph tables are missing: " + ",".join(missing_graph_tables))
+ if not source_tables:
+ issues.append("actual TMCRA records contain no immutable source records")
+ if len(sources) != len(expected):
+ issues.append(f"source record count {len(sources)} does not equal input message count {len(expected)}")
+ for key, message in expected.items():
+ source_matches = list(scoped_source_messages.get(key, ()))
+ if len(source_matches) != 1:
+ source = None
+ if len(source_matches) > 1:
+ issues.append(f"ambiguous immutable Source composite identity for {key}")
+ issues.append(f"missing immutable source record {key}")
+ else:
+ source = source_matches[0]
+ if source is not None and source["content"] != message["content"]:
+ issues.append(f"immutable source content differs for {key}")
+ elif source is not None and source["role"] and source["role"] != message["role"]:
+ issues.append(f"immutable source role differs for {key}")
+ elif source is not None and source["status"] != "enriched":
+ issues.append(f"immutable source is not enriched for {key}")
+ _audit_writer_input(run_dir, expected, db_paths, issues)
+ _audit_source_before_api(db_paths, expected, source_messages, issues)
+
+ fast: dict[Identity, dict[str, Any]] = {}
+ all_fast: dict[Identity, dict[str, Any]] = {}
+ records: dict[Identity, dict[str, Any]] = {}
+ interactions: dict[Identity, dict[str, Any]] = {}
+ jobs: list[dict[str, Any]] = []
+ edges: list[tuple[Path, dict[str, Any]]] = []
+ slow_candidates: list[dict[str, Any]] = []
+ slow_provenance: list[tuple[Path, dict[str, Any]]] = []
+ slow_patch_routes: dict[tuple[str, str], str] = {}
+ slow_patch_metadata: dict[tuple[str, str], dict[str, Any]] = {}
+ slow_graph_jobs: list[tuple[Path, dict[str, Any]]] = []
+ slow_graph_patches: list[tuple[Path, dict[str, Any]]] = []
+ slow_graph_patch_operations: list[tuple[Path, dict[str, Any]]] = []
+ subject_attribution_audits: list[tuple[Path, dict[str, Any]]] = []
+ for path in db_paths:
+ with sqlite3.connect(path) as con:
+ con.row_factory = sqlite3.Row
+ tables = _tables(con)
+ if "v4_reconciliation_jobs" in tables:
+ jobs.extend({**row, "db": _db_path(path)} for row in _rows(con, "v4_reconciliation_jobs"))
+ if "slow_graph_jobs" in tables:
+ slow_graph_jobs.extend((path, row) for row in _rows(con, "slow_graph_jobs"))
+ if "v4_subject_attribution_audits" in tables:
+ subject_attribution_audits.extend(
+ (path, row)
+ for row in _rows(con, "v4_subject_attribution_audits")
+ )
+ if "records" in tables:
+ for row in _rows(con, "records"):
+ scope = _text(row.get("scope_id"))
+ memory_id = _text(row.get("memory_id"))
+ record_identity = _identity(path, scope, memory_id)
+ record = dict(row)
+ record["metadata"] = _metadata(row)
+ record["db"] = _db_path(path)
+ records[record_identity] = record
+ meta = record["metadata"]
+ variant = _text(meta.get("content_variant"))
+ fast_like = (
+ variant == PRODUCTION_FAST_VARIANT
+ or _text(meta.get("node_kind")) in {PRODUCTION_FAST_NODE_KIND, "fast_assertion"}
+ )
+ if fast_like:
+ if not _production_fast_predicate(row, meta):
+ issues.append(
+ f"{path}: Fast record {scope}/{memory_id} fails the production strict predicate"
+ )
+ else:
+ all_fast[record_identity] = record
+ if _text(row.get("state")) in CURRENT_FAST_RECORD_STATES:
+ fast[record_identity] = record
+ if variant == "product_interaction" or meta.get("node_kind") in {"atomic_interaction", "product_interaction"}:
+ interactions[record_identity] = record
+ if variant == "slow_memory_capsule" or meta.get("memory_layer") == "slow":
+ slow_candidates.append(record)
+ if "memory_edges" in tables:
+ edges.extend((path, row) for row in _rows(con, "memory_edges"))
+ if "slow_graph_provenance" in tables:
+ slow_provenance.extend((path, row) for row in _rows(con, "slow_graph_provenance"))
+ if "slow_graph_patches" in tables:
+ for row in _rows(con, "slow_graph_patches"):
+ slow_graph_patches.append((path, row))
+ patch_metadata = _json(row.get("call_metadata_json"))
+ patch_key = (_db_path(path), _text(row.get("patch_id")))
+ slow_patch_routes[patch_key] = _text(patch_metadata.get("route"))
+ if isinstance(patch_metadata, Mapping):
+ slow_patch_metadata[patch_key] = dict(patch_metadata)
+ if "slow_graph_patch_operations" in tables:
+ slow_graph_patch_operations.extend(
+ (path, row) for row in _rows(con, "slow_graph_patch_operations")
+ )
+ if "slow_graph_attempts" in tables:
+ for row in _rows(con, "slow_graph_attempts"):
+ metadata = _json(row.get("call_metadata_json"))
+ if _artifact_markers(metadata):
+ issues.append(f"{path}: slow call contains fallback/retry marker")
+ route = _text(metadata.get("route"))
+ if not _slow_pro_route_is_justified(metadata):
+ issues.append(f"{path}: slow Pro call lacks a conflict route")
+ if "slow_graph_jobs" in tables:
+ for job in _rows(con, "slow_graph_jobs"):
+ if _text(job.get("status")) == "retryable" or _artifact_markers(job.get("last_error", "")):
+ issues.append(f"{path}: slow graph job contains a fallback/retry marker")
+
+ if "slot_heads" in tables:
+ for head in _rows(con, "slot_heads"):
+ scope = _text(head.get("scope_id")); memory_id = _text(head.get("memory_id"))
+ if not scope or not memory_id:
+ issues.append(f"{path}: slot_heads contains an incomplete head")
+ if "records" in tables:
+ target = con.execute(
+ "SELECT state FROM records WHERE scope_id=? AND memory_id=?",
+ (scope, memory_id),
+ ).fetchone()
+ if target is None:
+ issues.append(f"{path}: slot_heads targets missing actual record {memory_id}")
+ elif _text(target[0]) not in {
+ "active",
+ "parallel_active",
+ "promoted",
+ }:
+ issues.append(
+ f"{path}: slot_heads targets non-active record {memory_id}"
+ )
+
+ _audit_current_v47_patch_reconciliation(
+ jobs=slow_graph_jobs,
+ patches=slow_graph_patches,
+ operation_rows=slow_graph_patch_operations,
+ slow_records=slow_candidates,
+ issues=issues,
+ )
+
+ subject_attribution_decision_count = 0
+ subject_attribution_quarantine_count = 0
+ subject_attribution_superseded_count = 0
+ subject_audit_index = {
+ (_db_path(path), _text(row.get("audit_id"))): row
+ for path, row in subject_attribution_audits
+ if _text(row.get("audit_id"))
+ }
+ active_subject_messages: set[tuple[str, str, str]] = set()
+ for path, row in subject_attribution_audits:
+ scope_id = _text(row.get("scope_id"))
+ audit_id = _text(row.get("audit_id"))
+ status = _text(row.get("status"))
+ if status == "superseded":
+ prefix = "superseded_by:"
+ error = _text(row.get("error"))
+ target_id = error[len(prefix):] if error.startswith(prefix) else ""
+ target = subject_audit_index.get((_db_path(path), target_id))
+ if (
+ not target_id
+ or target is None
+ or _text(target.get("status")) != "completed"
+ or _text(target.get("scope_id")) != scope_id
+ or _text(target.get("message_id")) != _text(row.get("message_id"))
+ ):
+ issues.append(
+ f"{path}: superseded subject-attribution audit has no active successor"
+ )
+ else:
+ subject_attribution_superseded_count += 1
+ continue
+ if (
+ status != "completed"
+ or _text(row.get("prompt_version"))
+ != SUBJECT_ATTRIBUTION_PROMPT_VERSION
+ or _text(row.get("model")) != "deepseek-v4-pro"
+ or not audit_id
+ or not _text(row.get("request_sha256"))
+ or not _text(row.get("response_sha256"))
+ ):
+ issues.append(f"{path}: subject-attribution journal is incomplete or drifted")
+ continue
+ message_identity = (
+ _db_path(path),
+ scope_id,
+ _text(row.get("message_id")),
+ )
+ if message_identity in active_subject_messages:
+ issues.append(f"{path}: multiple active subject-attribution audits for one message")
+ continue
+ active_subject_messages.add(message_identity)
+ decisions = _json(row.get("decisions_json"))
+ if not isinstance(decisions, list) or not decisions:
+ issues.append(f"{path}: subject-attribution decisions are missing")
+ continue
+ for decision in decisions:
+ if not isinstance(decision, Mapping):
+ issues.append(f"{path}: subject-attribution decision is malformed")
+ continue
+ memory_id = _text(decision.get("memory_id"))
+ value = _text(decision.get("decision"))
+ bridge_quote = _text(decision.get("chat_user_bridge_quote"))
+ record = records.get(_identity(path, scope_id, memory_id))
+ if value not in {
+ "keep_user",
+ "quarantine_third_party",
+ "quarantine_ambiguous",
+ } or record is None:
+ issues.append(f"{path}: subject-attribution decision identity is invalid")
+ continue
+ subject_attribution_decision_count += 1
+ metadata = record["metadata"]
+ if value == "keep_user":
+ source_id = _text(metadata.get("source_record_id"))
+ source = sources.get(_identity(path, scope_id, source_id))
+ if (
+ not bridge_quote
+ or bridge_quote == _fast_quote(metadata)
+ or source is None
+ or bridge_quote not in _text(source.get("content"))
+ ):
+ issues.append(
+ f"{path}: keep_user subject attribution lacks an exact Source bridge"
+ )
+ elif bridge_quote:
+ issues.append(
+ f"{path}: quarantined subject attribution carries a chat-user bridge"
+ )
+ if (
+ _text(metadata.get("subject_attribution_audit_id")) != audit_id
+ or _text(metadata.get("subject_attribution_decision")) != value
+ or _text(
+ metadata.get("subject_attribution_chat_user_bridge_quote")
+ )
+ != bridge_quote
+ or _text(metadata.get("subject_attribution_prompt_version"))
+ != SUBJECT_ATTRIBUTION_PROMPT_VERSION
+ ):
+ issues.append(f"{path}: subject-attribution record binding changed")
+ if value != "keep_user":
+ subject_attribution_quarantine_count += 1
+ if (
+ _text(record.get("state")) != "quarantined"
+ or metadata.get("excluded_from_retrieval") is not True
+ ):
+ issues.append(
+ f"{path}: quarantined subject-attribution record is still current"
+ )
+
+ edge_keys: set[tuple[str, str, str, str, str]] = set()
+ for path, edge in edges:
+ scope = _text(edge.get("scope_id"))
+ edge_db = _text(edge.get("db_path"))
+ if edge_db and _db_path(edge_db) != _db_path(path):
+ issues.append(f"{path}: edge has a cross-database identity")
+ source_identity = _identity(path, scope, edge.get("source_memory_id"))
+ target_identity = _identity(path, scope, edge.get("target_memory_id"))
+ edge_keys.add(
+ (*source_identity, target_identity[2], _text(edge.get("edge_type")))
+ )
+ if source_identity not in records or target_identity not in records:
+ issues.append(
+ f"{path}: memory edge targets a missing composite record identity: {source_identity}->{target_identity}"
+ )
+
+ for leaf_identity, leaf in fast.items():
+ db_path, scope_id, leaf_id = leaf_identity
+ meta = leaf["metadata"]
+ durability = _text(leaf.get("durability")) or _text(meta.get("durability"))
+ if durability.casefold() not in {
+ "durable",
+ "long_term",
+ "long-term",
+ "hard",
+ "persistent",
+ "episodic",
+ "uncertain",
+ }:
+ issues.append(f"fast leaf {leaf_id}: invalid or missing durability")
+ source_id = _text(meta.get("source_record_id"))
+ source = _source_for_reference(
+ sources,
+ db_path=db_path,
+ scope_id=scope_id,
+ source_record_id=source_id,
+ )
+ if not source or not source_id:
+ issues.append(f"fast leaf {leaf_id}: source parent is not exact")
+ elif _text(meta.get("message_id")) != _text(source.get("message_id")):
+ issues.append(f"fast leaf {leaf_id}: message identity is not bound to immutable Source")
+ quote = _fast_quote(meta)
+ if not source or not _source_span_quote(
+ source,
+ meta.get("evidence_char_start"),
+ meta.get("evidence_char_end"),
+ quote,
+ label=f"fast leaf {leaf_id}",
+ issues=issues,
+ ):
+ issues.append(f"fast leaf {leaf_id}: evidence is not grounded in immutable source")
+ parent = meta.get("source_parent") or _slow_source_parent(meta)
+ validated_parent = _validate_source_parent(
+ parent,
+ db_path=db_path,
+ scope_id=scope_id,
+ sources=sources,
+ issues=issues,
+ label=f"fast leaf {leaf_id}",
+ quote=quote,
+ )
+ expected_parent = (
+ _slow_source_parent(meta)
+ if isinstance(parent, Mapping)
+ and set(parent) == SLOW_SOURCE_PARENT_KEYS
+ else _fast_source_parent(meta)
+ )
+ if validated_parent is not None and dict(parent) != expected_parent:
+ issues.append(f"fast leaf {leaf_id}: source_parent disagrees with record span or source identity")
+ if (*leaf_identity, source_id, "grounded_in") not in edge_keys:
+ issues.append(f"fast leaf {leaf_id}: missing grounded_in edge to exact source parent")
+ raw_provenance = meta.get("provenance") or []
+ if isinstance(raw_provenance, Mapping):
+ provenance_entries = [raw_provenance]
+ elif isinstance(raw_provenance, list):
+ provenance_entries = raw_provenance
+ else:
+ provenance_entries = []
+ issues.append(f"fast leaf {leaf_id}: provenance is not an object or array")
+ for provenance in provenance_entries:
+ if not isinstance(provenance, Mapping):
+ issues.append(f"fast leaf {leaf_id}: malformed provenance")
+ continue
+ p_source = _text(provenance.get("source_record_id"))
+ declared_db = provenance.get("db_path")
+ declared_scope = _text(provenance.get("scope_id"))
+ if declared_db and _db_path(declared_db) != db_path:
+ issues.append(f"fast leaf {leaf_id}: provenance crosses database identity")
+ continue
+ if declared_scope and declared_scope != scope_id:
+ issues.append(f"fast leaf {leaf_id}: provenance crosses scope identity")
+ continue
+ p_source_row = _source_for_reference(
+ sources,
+ db_path=db_path,
+ scope_id=scope_id,
+ source_record_id=p_source,
+ )
+ if not p_source_row:
+ issues.append(f"fast leaf {leaf_id}: provenance target is invalid")
+ continue
+ if _text(provenance.get("message_id")) != _text(p_source_row.get("message_id")):
+ issues.append(f"fast leaf {leaf_id}: provenance message identity is invalid")
+ source_turn_index = provenance.get("source_turn_index")
+ if source_turn_index is not None:
+ try:
+ turn_matches = int(source_turn_index) == int(
+ p_source_row.get("turn_index")
+ )
+ except (TypeError, ValueError):
+ turn_matches = False
+ if not turn_matches:
+ issues.append(f"fast leaf {leaf_id}: provenance turn identity is invalid")
+ start = provenance.get("source_char_start")
+ end = provenance.get("source_char_end")
+ if start is None and end is None:
+ issues.append(
+ f"fast leaf {leaf_id}: provenance exact source offsets are missing"
+ )
+ elif start is None or end is None:
+ issues.append(
+ f"fast leaf {leaf_id}: provenance exact source offsets are incomplete"
+ )
+ else:
+ _source_span_quote(
+ p_source_row,
+ start,
+ end,
+ provenance.get("evidence_quote"),
+ label=f"fast leaf {leaf_id} provenance",
+ issues=issues,
+ )
+ if (*leaf_identity, p_source, "grounded_in") not in edge_keys:
+ issues.append(f"fast leaf {leaf_id}: provenance grounded_in edge is missing")
+
+ for interaction_identity, row in interactions.items():
+ _, interaction_scope, interaction_id = interaction_identity
+ meta = row.get("metadata") if isinstance(row.get("metadata"), Mapping) else _metadata(row)
+ source = _source_for_message(
+ source_messages,
+ db_path=interaction_identity[0],
+ scope_id=interaction_scope,
+ message_id=meta.get("message_id"),
+ )
+ if not interaction_id or not source:
+ issues.append(f"interaction target is invalid: {interaction_id}")
+ interaction = meta
+ if isinstance(interaction, Mapping) and _identity(interaction_identity[0], interaction_scope, interaction.get("source_record_id")) not in {None, _identity(interaction_identity[0], interaction_scope, source.get("source_record_id") if source else "") }:
+ issues.append(f"interaction {interaction_id}: source target is invalid")
+ history = meta.get("resolution_history", [])
+ if not isinstance(history, list):
+ issues.append(f"interaction {interaction_id}: resolution history is not a list")
+ else:
+ for item in history:
+ if not isinstance(item, Mapping):
+ issues.append(f"interaction {interaction_id}: resolution target is invalid")
+ elif not _source_for_message(source_messages, db_path=interaction_identity[0], scope_id=interaction_scope, message_id=item.get("message_id")):
+ issues.append(f"interaction {interaction_id}: resolution message target is invalid")
+ elif _source_for_reference(sources, db_path=interaction_identity[0], scope_id=interaction_scope, source_record_id=item.get("source_record_id")) is None:
+ issues.append(f"interaction {interaction_id}: resolution source target is invalid")
+ for job in jobs:
+ request = _json(job.get("request_json"))
+ if _artifact_markers(request):
+ issues.append("writer reconciliation request contains fallback/retry marker")
+ job_db = _db_path(job.get("db"))
+ job_scope = _text(job.get("scope_id"))
+ candidate_ids: set[Identity] = set()
+ for current in list(request.get("candidate_cited_leaves") or []) if isinstance(request, Mapping) else []:
+ if isinstance(current, Mapping):
+ reference = _reference_identity(
+ current,
+ db_path=job_db,
+ scope_id=job_scope,
+ )
+ if reference is not None:
+ candidate_ids.add(reference)
+ else:
+ reference = None
+ if reference is None or reference not in records:
+ issues.append(
+ f"reconciliation job {_text(job.get('job_id'))}: historical candidate record is missing"
+ )
+ if _text(job.get("status")) == "completed" and not _text(job.get("decision")):
+ issues.append(f"reconciliation job {_text(job.get('job_id'))}: completed without decision")
+ elif _text(job.get("status")) != "completed":
+ issues.append(
+ f"reconciliation job {_text(job.get('job_id'))}: status is not completed"
+ )
+ if _text(job.get("status")) == "completed":
+ response = _json(job.get("response_json"))
+ if not isinstance(response, Mapping):
+ issues.append(f"reconciliation job {_text(job.get('job_id'))}: response is not an object")
+ elif (
+ _text(response.get("slot_decision")) == "bind_existing"
+ and _reference_identity(
+ response.get("selected_memory_id"),
+ db_path=job_db,
+ scope_id=job_scope,
+ ) not in candidate_ids
+ ):
+ issues.append(f"reconciliation job {_text(job.get('job_id'))}: selected slot was not a supplied candidate")
+
+ # Capsules are append-only. Only one row may represent the latest revision;
+ # older revisions are historical data and must not satisfy current retrieval.
+ slow_by_capsule: dict[tuple[str, str, str], list[dict[str, Any]]] = defaultdict(list)
+ for record in slow_candidates:
+ db_path = _db_path(record["db"])
+ scope_id = _text(record.get("scope_id"))
+ metadata = record["metadata"]
+ capsule_id = _text(metadata.get("capsule_id"))
+ try:
+ revision = int(metadata.get("revision"))
+ except (TypeError, ValueError):
+ revision = -1
+ if not capsule_id or revision < 1:
+ issues.append(f"slow record {_text(record.get('memory_id'))}: capsule identity/revision is invalid")
+ continue
+ record["slow_group"] = (db_path, scope_id, capsule_id)
+ record["slow_revision"] = revision
+ slow_by_capsule[record["slow_group"]].append(record)
+
+ slow_records: list[dict[str, Any]] = []
+ for group, revisions in slow_by_capsule.items():
+ latest_revision = max(record["slow_revision"] for record in revisions)
+ latest = [record for record in revisions if record["slow_revision"] == latest_revision]
+ if len(latest) != 1:
+ issues.append(f"slow capsule {group[2]} does not have a unique latest revision")
+ continue
+ record = latest[0]
+ if (
+ _text(record.get("state")) != "active"
+ or _text(record["metadata"].get("status"))
+ not in {"active", "challenged"}
+ ):
+ continue
+ slow_records.append(record)
+
+ slow_current_groups = {record["slow_group"] for record in slow_records}
+ slow_semantic_integrity_issues: list[dict[str, Any]] = []
+ slow_semantic_integrity_issues.extend(
+ _active_slow_partition_issues(slow_records, slow_patch_metadata)
+ )
+ slow_support_durability = {
+ "durable",
+ "long_term",
+ "long-term",
+ "hard",
+ "persistent",
+ }
+ slow_support_state = {
+ "active",
+ "parallel_active",
+ "promoted",
+ "challenged",
+ }
+ for record in slow_records:
+ db_path = _db_path(record["db"])
+ scope_id = _text(record.get("scope_id"))
+ meta = record["metadata"]
+ summary = " ".join(_text(record.get("value")).split())
+ structured_summary = False
+ if summary[:1] in "[{":
+ try:
+ structured_summary = isinstance(
+ json.loads(summary), (dict, list)
+ )
+ except json.JSONDecodeError:
+ structured_summary = False
+ if (
+ not summary
+ or len(summary) > 4096
+ or structured_summary
+ or SLOW_OPERATIONAL_SUMMARY_RE.search(summary)
+ or SLOW_GENERIC_SUMMARY_RE.fullmatch(summary)
+ ):
+ slow_semantic_integrity_issues.append(
+ {
+ "code": "invalid_semantic_summary",
+ "db": db_path,
+ "scope_id": scope_id,
+ "capsule_id": _text(meta.get("capsule_id")),
+ "revision": meta.get("revision"),
+ }
+ )
+ route = slow_patch_routes.get(
+ (db_path, _text(meta.get("patch_id"))), ""
+ )
+ prior_counterevidence: set[Identity] = set()
+ prior_revision = int(record["slow_revision"]) - 1
+ if prior_revision >= 1:
+ prior = [
+ candidate
+ for candidate in slow_by_capsule[record["slow_group"]]
+ if int(candidate["slow_revision"]) == prior_revision
+ ]
+ if len(prior) == 1:
+ for prior_claim in list(
+ prior[0]["metadata"].get("claims") or []
+ ):
+ if not isinstance(prior_claim, Mapping):
+ continue
+ for raw_reference in list(
+ prior_claim.get("counterevidence") or []
+ ):
+ reference = _reference_identity(
+ raw_reference,
+ db_path=db_path,
+ scope_id=scope_id,
+ )
+ if reference is not None:
+ prior_counterevidence.add(reference)
+ claims = meta.get("claims")
+ if not isinstance(claims, list) or not claims:
+ issues.append(f"slow record {_text(record.get('memory_id'))}: claims are missing")
+ continue
+ patch_metadata = slow_patch_metadata.get(
+ (db_path, _text(meta.get("patch_id"))), {}
+ )
+ if _current_summary_contract(meta, patch_metadata):
+ try:
+ expected_summary = _lossless_summary_projection(claims)
+ except AuditError as exc:
+ slow_semantic_integrity_issues.append(
+ {
+ "code": "invalid_lossless_summary_claims",
+ "db": db_path,
+ "scope_id": scope_id,
+ "capsule_id": _text(meta.get("capsule_id")),
+ "revision": record["slow_revision"],
+ "error": str(exc),
+ }
+ )
+ else:
+ if _text(record.get("value")) != expected_summary:
+ slow_semantic_integrity_issues.append(
+ {
+ "code": "stored_summary_not_lossless_projection",
+ "db": db_path,
+ "scope_id": scope_id,
+ "capsule_id": _text(meta.get("capsule_id")),
+ "revision": record["slow_revision"],
+ "expected_summary": expected_summary,
+ "stored_summary": _text(record.get("value")),
+ }
+ )
+ claim_roles: list[tuple[str, set[Identity], set[Identity]]] = []
+ for claim in claims:
+ if not isinstance(claim, Mapping):
+ issues.append(f"slow record {_text(record.get('memory_id'))}: malformed claim")
+ continue
+ claim_id = _text(claim.get("claim_id"))
+ claim_slot = _text(claim.get("canonical_slot"))
+ claim_text = _normal_text(claim.get("text"))
+ support_references = {
+ reference
+ for raw_reference in list(claim.get("support") or [])
+ if (
+ reference := _reference_identity(
+ raw_reference, db_path=db_path, scope_id=scope_id
+ )
+ )
+ is not None
+ }
+ counter_references = {
+ reference
+ for raw_reference in list(claim.get("counterevidence") or [])
+ if (
+ reference := _reference_identity(
+ raw_reference, db_path=db_path, scope_id=scope_id
+ )
+ )
+ is not None
+ }
+ complementary_support_bundle = _controlled_complementary_support_bundle(
+ claim_slot,
+ support_references,
+ all_fast,
+ )
+ shared_roles = support_references & counter_references
+ if shared_roles:
+ slow_semantic_integrity_issues.append(
+ {
+ "code": "same_evidence_support_and_counterevidence",
+ "db": db_path,
+ "scope_id": scope_id,
+ "capsule_id": _text(meta.get("capsule_id")),
+ "revision": record["slow_revision"],
+ "claim_id": claim_id,
+ "evidence_ids": sorted(item[2] for item in shared_roles),
+ }
+ )
+ if meta.get("action") == "create" and counter_references:
+ slow_semantic_integrity_issues.append(
+ {
+ "code": "active_create_contains_counterevidence",
+ "db": db_path,
+ "scope_id": scope_id,
+ "capsule_id": _text(meta.get("capsule_id")),
+ "revision": record["slow_revision"],
+ "claim_id": claim_id,
+ "evidence_ids": sorted(
+ item[2] for item in counter_references
+ ),
+ }
+ )
+ support_text_groups: dict[str, list[str]] = {}
+ for reference in sorted(support_references):
+ leaf = all_fast.get(reference)
+ if leaf is None:
+ continue
+ leaf_meta = leaf["metadata"]
+ evidence_text = _normal_text(
+ leaf.get("value")
+ or leaf_meta.get("source_span")
+ or leaf_meta.get("raw_content")
+ )
+ if evidence_text:
+ support_text_groups.setdefault(evidence_text, []).append(
+ reference[2]
+ )
+ if len(support_text_groups) > 1 and not complementary_support_bundle:
+ slow_semantic_integrity_issues.append(
+ {
+ "code": "support_distinct_fast_values_merged",
+ "db": db_path,
+ "scope_id": scope_id,
+ "capsule_id": _text(meta.get("capsule_id")),
+ "revision": record["slow_revision"],
+ "claim_id": claim_id,
+ "evidence_groups": [
+ {
+ "normalized_text": text,
+ "evidence_ids": evidence_ids,
+ }
+ for text, evidence_ids in sorted(
+ support_text_groups.items()
+ )
+ ],
+ }
+ )
+ claim_roles.append(
+ (claim_id, support_references, counter_references)
+ )
+ cited: list[Identity] = []
+ for raw_reference in [*(claim.get("support") or []), *(claim.get("counterevidence") or [])]:
+ reference = _reference_identity(
+ raw_reference,
+ db_path=db_path,
+ scope_id=scope_id,
+ )
+ if reference is None or reference not in all_fast:
+ issues.append(
+ f"slow claim {_text(claim.get('claim_id'))}: cited fast leaf does not exist under composite identity: {raw_reference}"
+ )
+ else:
+ cited.append(reference)
+ for raw_reference in list(claim.get("support") or []):
+ reference = _reference_identity(
+ raw_reference,
+ db_path=db_path,
+ scope_id=scope_id,
+ )
+ leaf = all_fast.get(reference) if reference is not None else None
+ if leaf is None:
+ continue
+ evidence_slot = _text(
+ leaf["metadata"].get("canonical_slot_key")
+ or leaf["metadata"].get("canonical_slot")
+ )
+ if evidence_slot != claim_slot and not complementary_support_bundle:
+ slow_semantic_integrity_issues.append(
+ {
+ "code": "support_canonical_slot_mismatch",
+ "db": db_path,
+ "scope_id": scope_id,
+ "capsule_id": _text(meta.get("capsule_id")),
+ "revision": record["slow_revision"],
+ "claim_id": claim_id,
+ "claim_slot": claim_slot,
+ "evidence_id": reference[2],
+ "evidence_slot": evidence_slot,
+ }
+ )
+ leaf_meta = leaf["metadata"]
+ if not (
+ _text(
+ leaf_meta.get("durability")
+ or leaf_meta.get("durability_class")
+ ).casefold()
+ in slow_support_durability
+ and _text(leaf.get("state")).casefold()
+ in slow_support_state
+ ):
+ slow_semantic_integrity_issues.append(
+ {
+ "code": "support_noncurrent_fast_leaf",
+ "db": db_path,
+ "scope_id": scope_id,
+ "capsule_id": _text(meta.get("capsule_id")),
+ "revision": record["slow_revision"],
+ "claim_id": claim_id,
+ "evidence_id": reference[2],
+ "evidence_state": _text(leaf.get("state")),
+ }
+ )
+ for raw_reference in list(claim.get("counterevidence") or []):
+ reference = _reference_identity(
+ raw_reference,
+ db_path=db_path,
+ scope_id=scope_id,
+ )
+ leaf = all_fast.get(reference) if reference is not None else None
+ if leaf is None:
+ continue
+ if _normal_text(leaf.get("value")) == claim_text:
+ slow_semantic_integrity_issues.append(
+ {
+ "code": "counterevidence_identical_to_claim",
+ "db": db_path,
+ "scope_id": scope_id,
+ "capsule_id": _text(meta.get("capsule_id")),
+ "revision": record["slow_revision"],
+ "claim_id": claim_id,
+ "evidence_id": reference[2],
+ }
+ )
+ leaf_meta = leaf["metadata"]
+ if (
+ route == "flash"
+ and reference not in prior_counterevidence
+ and not bool(leaf_meta.get("counterevidence"))
+ and not bool(leaf_meta.get("is_counterevidence"))
+ ):
+ slow_semantic_integrity_issues.append(
+ {
+ "code": "flash_invented_counterevidence",
+ "db": db_path,
+ "scope_id": scope_id,
+ "capsule_id": _text(meta.get("capsule_id")),
+ "revision": record["slow_revision"],
+ "claim_id": claim_id,
+ "evidence_id": reference[2],
+ "route": route,
+ }
+ )
+ parents = claim.get("source_parents", meta.get("source_parents"))
+ if not isinstance(parents, list) or not parents:
+ issues.append(f"slow claim {_text(claim.get('claim_id'))}: source parents missing")
+ continue
+ for parent in parents:
+ _validate_source_parent(
+ parent,
+ db_path=db_path,
+ scope_id=scope_id,
+ sources=sources,
+ issues=issues,
+ label=f"slow claim {_text(claim.get('claim_id'))}",
+ )
+ for leaf_identity in cited:
+ leaf = all_fast[leaf_identity]
+ leaf_meta = leaf["metadata"]
+ leaf_parent = _slow_source_parent(leaf_meta)
+ matching_parents = [
+ parent for parent in parents
+ if isinstance(parent, Mapping)
+ and dict(parent) == dict(leaf_parent)
+ ]
+ if not matching_parents:
+ issues.append(
+ f"slow claim {_text(claim.get('claim_id'))}: cited Fast source parent coordinates changed"
+ )
+ else:
+ _validate_source_parent(
+ matching_parents[0],
+ db_path=db_path,
+ scope_id=scope_id,
+ sources=sources,
+ issues=issues,
+ label=f"slow claim {_text(claim.get('claim_id'))} citation",
+ quote=_fast_quote(leaf_meta),
+ )
+ for left_index, (
+ left_claim_id,
+ left_support,
+ left_counter,
+ ) in enumerate(claim_roles):
+ for (
+ right_claim_id,
+ right_support,
+ right_counter,
+ ) in claim_roles[left_index + 1 :]:
+ if left_support & right_counter and right_support & left_counter:
+ slow_semantic_integrity_issues.append(
+ {
+ "code": "reciprocal_counterevidence_cycle",
+ "db": db_path,
+ "scope_id": scope_id,
+ "capsule_id": _text(meta.get("capsule_id")),
+ "revision": record["slow_revision"],
+ "claim_ids": sorted(
+ [left_claim_id, right_claim_id]
+ ),
+ }
+ )
+ for path, row in slow_provenance:
+ db_path = _db_path(path)
+ scope_id = _text(row.get("scope_id"))
+ group = (db_path, scope_id, _text(row.get("capsule_id")))
+ try:
+ revision = int(row.get("revision"))
+ except (TypeError, ValueError):
+ revision = -1
+ if group not in slow_current_groups:
+ continue
+ current = next(record for record in slow_records if record["slow_group"] == group)
+ if revision != current["slow_revision"]:
+ continue
+ reference = _reference_identity(
+ row.get("evidence_memory_id"),
+ db_path=db_path,
+ scope_id=scope_id,
+ )
+ leaf = all_fast.get(reference) if reference is not None else None
+ if leaf is None:
+ issues.append(f"slow provenance cites missing Fast leaf under composite identity: {row.get('evidence_memory_id')}")
+ continue
+ parent = _json(row.get("source_parent_json"))
+ _validate_source_parent(
+ parent,
+ db_path=db_path,
+ scope_id=scope_id,
+ sources=sources,
+ issues=issues,
+ label="slow provenance",
+ quote=_fast_quote(leaf["metadata"]),
+ )
+ expected_parent = _slow_source_parent(leaf["metadata"])
+ if isinstance(parent, Mapping) and dict(parent) != dict(expected_parent):
+ issues.append("slow provenance source parent does not match cited Fast leaf")
+
+ eligible_durability = {"durable", "long_term", "long-term", "hard", "persistent"}
+ eligible_state = {"active", "parallel_active", "promoted"}
+ eligible_fast_ids = {
+ leaf_identity
+ for leaf_identity, leaf in fast.items()
+ if _text(
+ leaf["metadata"].get("durability")
+ or leaf["metadata"].get("durability_class")
+ ).casefold()
+ in eligible_durability
+ and _text(leaf.get("state")).casefold() in eligible_state
+ }
+ cited_by_current_slow: set[Identity] = set()
+ for record in slow_records:
+ metadata = record["metadata"]
+ if (
+ _text(record.get("state")) != "active"
+ or _text(metadata.get("status")).casefold() not in {"active", "challenged"}
+ ):
+ continue
+ for claim in list(metadata.get("claims") or []):
+ if isinstance(claim, Mapping):
+ for raw_reference in [*(claim.get("support") or []), *(claim.get("counterevidence") or [])]:
+ reference = _reference_identity(
+ raw_reference,
+ db_path=record["db"],
+ scope_id=record.get("scope_id"),
+ )
+ if reference is not None:
+ cited_by_current_slow.add(reference)
+ cited_eligible = eligible_fast_ids & cited_by_current_slow
+ uncited_eligible = sorted(eligible_fast_ids - cited_by_current_slow)
+ slow_semantic_integrity_issues.sort(
+ key=lambda item: json.dumps(item, ensure_ascii=False, sort_keys=True)
+ )
+ for semantic_issue in slow_semantic_integrity_issues:
+ issues.append(
+ "active Slow claim semantic integrity failed: "
+ + json.dumps(semantic_issue, ensure_ascii=False, sort_keys=True)
+ )
+ enforce_slow_promotion = build_only or retrieval_dir is not None
+ slow_promotion_coverage = {
+ "schema_version": "tmcra.v4.slow-promotion-coverage.3",
+ "enforced": enforce_slow_promotion,
+ "complete": not uncited_eligible and not slow_semantic_integrity_issues,
+ "eligible_current_durable_count": len(eligible_fast_ids),
+ "cited_current_durable_count": len(cited_eligible),
+ "coverage_ratio": (
+ round(len(cited_eligible) / len(eligible_fast_ids), 6)
+ if eligible_fast_ids
+ else 1.0
+ ),
+ "uncited_current_durable_ids": uncited_eligible,
+ "semantic_integrity_issue_count": len(slow_semantic_integrity_issues),
+ "semantic_integrity_issues": slow_semantic_integrity_issues,
+ }
+ if enforce_slow_promotion and uncited_eligible:
+ issues.append(
+ "current durable Fast evidence is missing from active Slow claims: "
+ + ",".join(identity[2] for identity in uncited_eligible[:20])
+ )
+
+ if build_only:
+ retrieval = {
+ "present": any(run_dir.rglob("retrieval_debug.jsonl")),
+ "passed": None,
+ "skipped": True,
+ "reason": "build_only_audit",
+ }
+ else:
+ retrieval = _audit_retrieval(
+ run_dir,
+ retrieval_dir,
+ sources,
+ issues,
+ fast=fast,
+ slow_records=slow_records,
+ slow_patch_metadata=slow_patch_metadata,
+ records=records,
+ )
+ report = {"schema_version": "tmcra.v4.chain-audit.1", "status": "passed" if not issues else "failed", "passed": not issues, "run_dir": str(run_dir), "input": str(input_path), "workers": {key: [str(path) for path in values] for key, values in db_groups.items()}, "counts": {"input_messages": len(expected) + len(excluded_empty), "nonempty_input_messages": len(expected), "excluded_empty_input_messages": len(excluded_empty), "source_records": len(sources), "fast_leaves": len(fast), "slow_records": len(slow_records), "interactions": len(interactions), "edges": len(edges)}, "subject_attribution": {"journal_count": len(subject_attribution_audits), "superseded_count": subject_attribution_superseded_count, "decision_count": subject_attribution_decision_count, "quarantine_count": subject_attribution_quarantine_count, "prompt_version": SUBJECT_ATTRIBUTION_PROMPT_VERSION, "model": "deepseek-v4-pro"}, "slow_promotion_coverage": slow_promotion_coverage, "source_tables": sorted(source_tables), "retrieval": retrieval, "issues": sorted(set(issues))}
+ if output:
+ Path(output).parent.mkdir(parents=True, exist_ok=True)
+ Path(output).write_text(json.dumps(report, ensure_ascii=False, indent=2, sort_keys=True) + "\n", encoding="utf-8")
+ return report
+
+
+def _text_list(value: Any) -> list[str]:
+ return [_text(item) for item in value] if isinstance(value, list) else []
+
+
+def _audit_session_ordering(
+ windows: list[Any], row_index: int, issues: list[str]
+) -> None:
+ pre_ranks: list[int] = []
+ session_sequence: list[int] = []
+ session_entries: dict[int, list[tuple[int, int, Mapping[str, Any]]]] = {}
+ closed_sessions: set[int] = set()
+ current_session: int | None = None
+ malformed = False
+ for window in windows:
+ if not isinstance(window, Mapping):
+ malformed = True
+ continue
+ metadata = window.get("retrieval_metadata")
+ try:
+ session_index = int(window["session_index"])
+ parent_index = int(window["parent_chunk_index"])
+ subchunk_index = int(window["subchunk_index"])
+ except (KeyError, TypeError, ValueError):
+ malformed = True
+ continue
+ if not isinstance(metadata, Mapping):
+ malformed = True
+ continue
+ pre_rank = metadata.get("pre_session_order_rank")
+ if type(pre_rank) is not int or pre_rank <= 0:
+ malformed = True
+ continue
+ pre_ranks.append(pre_rank)
+ if session_index != current_session:
+ if current_session is not None:
+ closed_sessions.add(current_session)
+ if session_index in closed_sessions:
+ issues.append(
+ f"retrieval row {row_index}: a session is interleaved after grouping"
+ )
+ current_session = session_index
+ session_sequence.append(session_index)
+ session_entries.setdefault(session_index, []).append(
+ (parent_index, subchunk_index, metadata)
+ )
+ if malformed:
+ issues.append(
+ f"retrieval row {row_index}: session ordering coordinates are malformed"
+ )
+ return
+ if sorted(pre_ranks) != list(range(1, len(windows) + 1)):
+ issues.append(
+ f"retrieval row {row_index}: pre-session ranks are not a permutation"
+ )
+ for expected_session_rank, session_index in enumerate(session_sequence, start=1):
+ entries = session_entries[session_index]
+ coordinates = [(parent, subchunk) for parent, subchunk, _ in entries]
+ if coordinates != sorted(coordinates):
+ issues.append(
+ f"retrieval row {row_index}: a grouped session is not chronological"
+ )
+ expected_count = len(entries)
+ expected_rrf = round(
+ sum(1.0 / float(metadata["pre_session_order_rank"]) for _, _, metadata in entries),
+ 8,
+ )
+ for _, _, metadata in entries:
+ observed_rrf = metadata.get("session_support_rrf")
+ if (
+ metadata.get("session_order_rank") != expected_session_rank
+ or metadata.get("session_selected_window_count") != expected_count
+ or isinstance(observed_rrf, bool)
+ or not isinstance(observed_rrf, (int, float))
+ or abs(float(observed_rrf) - expected_rrf) > 1e-8
+ ):
+ issues.append(
+ f"retrieval row {row_index}: session ordering aggregate is inconsistent"
+ )
+ break
+
+
+def _audit_retrieval_contract(
+ evidence_row: Mapping[str, Any],
+ debug_row: Mapping[str, Any],
+ windows: list[Any],
+ row_index: int,
+ issues: list[str],
+) -> None:
+ contract = evidence_row.get("retrieval_contract")
+ if not isinstance(contract, Mapping):
+ issues.append(f"retrieval row {row_index}: retrieval contract is missing")
+ return
+ if contract.get("schema_version") != RETRIEVAL_CONTRACT_SCHEMA_VERSION:
+ issues.append(f"retrieval row {row_index}: retrieval contract schema is stale")
+ lane = _text(contract.get("execution_lane"))
+ mode = _text(contract.get("composition_mode"))
+ if lane not in {"production", "diagnostic"}:
+ issues.append(f"retrieval row {row_index}: execution lane is invalid")
+ if lane == "production" and mode != "layered":
+ issues.append(
+ f"retrieval row {row_index}: production retrieval is not layered"
+ )
+ packing_mode = _text(contract.get("packing_budget_mode"))
+ packing_budget = contract.get("packing_budget")
+ if (
+ isinstance(packing_budget, bool)
+ or not isinstance(packing_budget, int)
+ or packing_budget <= 0
+ ):
+ issues.append(f"retrieval row {row_index}: packing budget is invalid")
+ if lane == "production" and (
+ packing_mode != "fixed" or packing_budget != 8
+ ):
+ issues.append(
+ f"retrieval row {row_index}: production packing is not fixed Top8"
+ )
+ if contract.get("source_coverage_trace_k") != SOURCE_COVERAGE_TRACE_K:
+ issues.append(
+ f"retrieval row {row_index}: retrieval contract is not Source Top24"
+ )
+ if contract.get("final_window_count") != len(windows):
+ issues.append(
+ f"retrieval row {row_index}: final window count disagrees with evidence"
+ )
+ if lane == "production" and len(windows) > 8:
+ issues.append(f"retrieval row {row_index}: final evidence exceeds Top8")
+ budget_decision = debug_row.get("packing_budget_decision")
+ if (
+ not isinstance(budget_decision, Mapping)
+ or _text(budget_decision.get("mode")) != packing_mode
+ or budget_decision.get("budget") != packing_budget
+ or debug_row.get("packing_budget_top_k") != packing_budget
+ ):
+ issues.append(
+ f"retrieval row {row_index}: packing contract disagrees with runtime debug"
+ )
+
+ actual = {"source": 0, "fast": 0, "slow": 0}
+ for window in windows:
+ metadata = window.get("retrieval_metadata") if isinstance(window, Mapping) else None
+ contributions = (
+ metadata.get("layer_contributions") if isinstance(metadata, Mapping) else []
+ )
+ layers = {
+ _text(item.get("layer"))
+ for item in contributions
+ if isinstance(item, Mapping)
+ }
+ for layer in layers:
+ if layer in actual:
+ actual[layer] += 1
+ inventory = contract.get("inventory_counts")
+ inventory_counts: dict[str, int] = {}
+ expected_inventory_keys = {
+ "source",
+ "fast",
+ "fast_semantic",
+ "slow",
+ "slow_capsule_heads",
+ "slow_summaries",
+ "slow_claims",
+ "slow_ranked_claims",
+ }
+ if (
+ not isinstance(inventory, Mapping)
+ or set(inventory) != expected_inventory_keys
+ or any(
+ type(inventory.get(layer)) is not int or inventory.get(layer) < 0
+ for layer in expected_inventory_keys
+ )
+ ):
+ issues.append(f"retrieval row {row_index}: inventory contract is invalid")
+ else:
+ inventory_counts = {layer: int(inventory[layer]) for layer in inventory}
+ if inventory_counts["fast_semantic"] > inventory_counts["fast"]:
+ issues.append(
+ f"retrieval row {row_index}: semantic Fast inventory exceeds Fast"
+ )
+ if (
+ inventory_counts["slow_capsule_heads"]
+ != inventory_counts["slow_summaries"]
+ or inventory_counts["slow_ranked_claims"] != inventory_counts["slow"]
+ or inventory_counts["slow_claims"] < inventory_counts["slow_capsule_heads"]
+ ):
+ issues.append(
+ f"retrieval row {row_index}: Slow summary/claim inventory counts are inconsistent"
+ )
+ selected = contract.get("selected_layer_window_counts")
+ if not isinstance(selected, Mapping) or set(selected) != set(actual) or any(
+ type(selected.get(layer)) is not int or selected.get(layer) != count
+ for layer, count in actual.items()
+ ):
+ issues.append(
+ f"retrieval row {row_index}: selected layer counts do not match final evidence"
+ )
+ required = contract.get("required_selected_layers")
+ if (
+ not isinstance(required, list)
+ or not required
+ or len(required) != len(set(map(_text, required)))
+ or any(_text(layer) not in actual for layer in required)
+ ):
+ issues.append(f"retrieval row {row_index}: required layer contract is invalid")
+ elif lane == "production":
+ expected_required = {
+ layer
+ for layer in ("source", "fast", "slow")
+ if inventory_counts.get(layer, 0) > 0
+ }
+ if set(map(_text, required)) != expected_required:
+ issues.append(
+ f"retrieval row {row_index}: required layers disagree with inventories"
+ )
+ missing = sorted(_text(layer) for layer in required if actual[_text(layer)] <= 0)
+ if not required or missing:
+ issues.append(
+ f"retrieval row {row_index}: production layers are missing: {missing}"
+ )
+ paths = contract.get("candidate_paths_executed")
+ if (
+ not isinstance(paths, Mapping)
+ or set(paths) != set(actual)
+ or any(type(paths.get(layer)) is not bool for layer in actual)
+ or paths != debug_row.get("candidate_paths_executed")
+ ):
+ issues.append(
+ f"retrieval row {row_index}: path execution contract disagrees with debug"
+ )
+
+
+def _audit_retrieval(
+ run_dir: Path,
+ retrieval_dir: Path | None,
+ sources: Mapping[Identity, Mapping[str, Any]],
+ issues: list[str],
+ *,
+ fast: Mapping[Identity, Mapping[str, Any]] | None = None,
+ slow_records: Iterable[Mapping[str, Any]] = (),
+ slow_patch_metadata: Mapping[tuple[str, str], Mapping[str, Any]] | None = None,
+ records: Mapping[Identity, Mapping[str, Any]] | None = None,
+) -> dict[str, Any] | None:
+ candidates = [Path(retrieval_dir)] if retrieval_dir else []
+ if not candidates:
+ candidates = sorted({path.parent for path in run_dir.rglob("retrieval_debug.jsonl")})
+ if not candidates:
+ return None
+ directory = candidates[0]
+ debug_path, evidence_path, report_path = directory / "retrieval_debug.jsonl", directory / "evidence_windows.jsonl", directory / "report.json"
+ if not all(path.is_file() for path in (debug_path, evidence_path, report_path)):
+ issues.append(f"retrieval output is incomplete: {directory}")
+ return {"present": True, "passed": False}
+ debug = [json.loads(line) for line in debug_path.read_text(encoding="utf-8").splitlines() if line.strip()]
+ evidence = [json.loads(line) for line in evidence_path.read_text(encoding="utf-8").splitlines() if line.strip()]
+ report = json.loads(report_path.read_text(encoding="utf-8"))
+ for item in [*debug, *evidence, report]:
+ forbidden = _retrieval_forbidden(item)
+ if forbidden:
+ issues.extend(f"retrieval contains forbidden hard/evaluation field: {path}" for path in forbidden)
+ if report.get("schema_version") != RETRIEVAL_REPORT_SCHEMA_VERSION:
+ issues.append("retrieval report schema is missing or stale")
+ if (
+ report.get("runtime_schema_version") != RETRIEVAL_SCHEMA_VERSION
+ or report.get("online_index_schema_version")
+ != ONLINE_INDEX_SCHEMA_VERSION
+ or report.get("slow_inventory_schema_version")
+ != SLOW_INVENTORY_SCHEMA_VERSION
+ ):
+ issues.append("retrieval report index/inventory contract is missing or stale")
+ if report.get("source_coverage_trace_k") != SOURCE_COVERAGE_TRACE_K:
+ issues.append("retrieval report source coverage trace policy is missing or stale")
+ if (
+ report.get("answer_attachment_contract") != "object-list-v1"
+ or report.get("ranking_metadata_field") != "retrieval_metadata"
+ ):
+ issues.append("retrieval report answer attachment contract is missing or stale")
+ if (
+ report.get("session_coherent_ordering") is not True
+ or report.get("session_ordering_policy") != SESSION_ORDERING_POLICY
+ ):
+ issues.append("retrieval report session ordering contract is missing or stale")
+ if report.get("status") != "complete" or len(debug) != len(evidence):
+ issues.append("retrieval report/debug/evidence counts are not complete")
+ fast = fast or {}
+ slow_patch_metadata = slow_patch_metadata or {}
+ slow_lookup: dict[tuple[str, str, str, int], Mapping[str, Any]] = {}
+ for record in slow_records:
+ metadata = record.get("metadata")
+ if not isinstance(metadata, Mapping):
+ continue
+ if (
+ _text(record.get("state")) != "active"
+ or _text(metadata.get("status")) not in {"active", "challenged"}
+ ):
+ continue
+ try:
+ revision = int(metadata.get("revision", -1))
+ except (TypeError, ValueError):
+ continue
+ slow_lookup[
+ (
+ _db_path(record.get("db")),
+ _text(record.get("scope_id")),
+ _text(metadata.get("capsule_id")),
+ revision,
+ )
+ ] = record
+ executed = 0
+ source_candidate_id_collisions: set[tuple[int, str]] = set()
+ for index, (debug_row, evidence_row) in enumerate(zip(debug, evidence)):
+ if evidence_row.get("runtime_schema_version") != RETRIEVAL_SCHEMA_VERSION:
+ issues.append(f"retrieval row {index}: runtime schema is missing or stale")
+ if (
+ debug_row.get("session_coherent_ordering") is not True
+ or debug_row.get("session_ordering_policy") != SESSION_ORDERING_POLICY
+ ):
+ issues.append(
+ f"retrieval row {index}: debug session ordering contract is missing or stale"
+ )
+ if debug_row.get("fast_semantic_state_policy") != CURRENT_FAST_INDEX_POLICY:
+ issues.append(
+ f"retrieval row {index}: fast semantic state policy is missing or stale"
+ )
+ inventories = {"source": int(debug_row.get("source_inventory_count", 0) or 0), "fast": int(debug_row.get("fast_inventory_count", 0) or 0), "slow": int(debug_row.get("slow_capsule_count", 0) or 0)}
+ slow_count_fields = {
+ "heads": debug_row.get("slow_capsule_head_count"),
+ "summaries": debug_row.get("slow_summary_candidate_count"),
+ "claims": debug_row.get("slow_claim_candidate_count"),
+ "shortlist": debug_row.get("slow_shortlist_count"),
+ "summary_hits": debug_row.get("slow_summary_hit_count"),
+ "direct_hits": debug_row.get("slow_direct_claim_hit_count"),
+ }
+ if any(
+ type(value) is not int or value < 0
+ for value in slow_count_fields.values()
+ ) or (
+ slow_count_fields["heads"] != slow_count_fields["summaries"]
+ or inventories["slow"] != slow_count_fields["heads"]
+ or slow_count_fields["claims"] < slow_count_fields["heads"]
+ or slow_count_fields["summary_hits"] > slow_count_fields["shortlist"]
+ or slow_count_fields["direct_hits"] > slow_count_fields["shortlist"]
+ ):
+ issues.append(
+ f"retrieval row {index}: Slow summary/claim debug counts are invalid"
+ )
+ slow_trace = debug_row.get("slow_retrieval_trace")
+ trace_keys = {
+ "inventory_schema_version",
+ "direct_claim_hit",
+ "claim_candidate_id",
+ "claim_dense_rank",
+ "claim_dense_score",
+ "summary_expansion_hit",
+ "summary_hit",
+ "final_claim_cross_score",
+ "source_parents",
+ }
+ if (
+ not isinstance(slow_trace, list)
+ or len(slow_trace) != slow_count_fields["shortlist"]
+ ):
+ issues.append(f"retrieval row {index}: Slow retrieval trace is incomplete")
+ else:
+ for trace_index, item in enumerate(slow_trace):
+ if (
+ not isinstance(item, Mapping)
+ or set(item) != trace_keys
+ or item.get("inventory_schema_version")
+ != SLOW_INVENTORY_SCHEMA_VERSION
+ or type(item.get("direct_claim_hit")) is not bool
+ or type(item.get("summary_expansion_hit")) is not bool
+ or not (
+ item.get("direct_claim_hit")
+ or item.get("summary_expansion_hit")
+ )
+ or not _text(item.get("claim_candidate_id"))
+ or type(item.get("claim_dense_rank")) is not int
+ or item.get("claim_dense_rank") < 0
+ or not isinstance(item.get("claim_dense_score"), (int, float))
+ or not isinstance(
+ item.get("final_claim_cross_score"), (int, float)
+ )
+ or not isinstance(item.get("source_parents"), list)
+ or not item.get("source_parents")
+ ):
+ issues.append(
+ f"retrieval row {index}: Slow trace {trace_index} is malformed"
+ )
+ continue
+ summary_hit = item.get("summary_hit")
+ if item.get("summary_expansion_hit"):
+ if (
+ not isinstance(summary_hit, Mapping)
+ or not _text(summary_hit.get("summary_candidate_id"))
+ or not _text(summary_hit.get("summary_text"))
+ or type(summary_hit.get("summary_dense_rank")) is not int
+ or summary_hit.get("summary_dense_rank") < 0
+ or not isinstance(
+ summary_hit.get("summary_dense_score"), (int, float)
+ )
+ ):
+ issues.append(
+ f"retrieval row {index}: Slow trace {trace_index} summary hit is malformed"
+ )
+ elif summary_hit is not None:
+ issues.append(
+ f"retrieval row {index}: Slow trace {trace_index} has an unexecuted summary hit"
+ )
+ paths = debug_row.get("candidate_paths_executed")
+ if not isinstance(paths, Mapping):
+ issues.append(f"retrieval row {index}: candidate path execution audit is missing")
+ else:
+ for layer, count in inventories.items():
+ if count > 0 and paths.get(layer) is not True:
+ issues.append(f"retrieval row {index}: {layer} inventory exists but path did not execute")
+ executed += int(all(paths.get(layer) is True for layer, count in inventories.items() if count > 0))
+ source_count_value = debug_row.get("source_candidate_count")
+ source_count = (
+ source_count_value
+ if type(source_count_value) is int and source_count_value > 0
+ else 0
+ )
+ if source_count == 0:
+ issues.append(f"retrieval row {index}: source candidate count is invalid")
+ trace = debug_row.get("source_top24_candidates")
+ pool = debug_row.get("source_candidate_pool_trace")
+ if debug_row.get("source_coverage_trace_k") != SOURCE_COVERAGE_TRACE_K:
+ issues.append(
+ f"retrieval row {index}: source coverage trace policy is missing or stale"
+ )
+ if not isinstance(trace, list) or len(trace) != min(
+ SOURCE_COVERAGE_TRACE_K, source_count
+ ) or not isinstance(pool, list) or len(pool) != source_count:
+ issues.append(f"retrieval row {index}: source Top24 trace is incomplete")
+ else:
+ required = {
+ "rank",
+ "candidate_id",
+ "session_id",
+ "session_index",
+ "parent_chunk_index",
+ "subchunk_index",
+ }
+ def trace_identities(
+ rows: list[Any], label: str
+ ) -> list[tuple[str, str, int, int, int]]:
+ identities: list[tuple[str, str, int, int, int]] = []
+ candidate_ids: set[str] = set()
+ locations: set[tuple[str, int, int, int]] = set()
+ for rank, item in enumerate(rows, start=1):
+ if not isinstance(item, Mapping) or set(item) != required:
+ issues.append(
+ f"retrieval row {index}: {label} entry is malformed"
+ )
+ continue
+ coordinates = (
+ item.get("session_index"),
+ item.get("parent_chunk_index"),
+ item.get("subchunk_index"),
+ )
+ candidate_id = _text(item.get("candidate_id"))
+ session_id = _text(item.get("session_id"))
+ if (
+ item.get("rank") != rank
+ or not candidate_id
+ or not session_id
+ or any(type(value) is not int or value < 0 for value in coordinates)
+ ):
+ issues.append(
+ f"retrieval row {index}: {label} rank or identity is invalid"
+ )
+ continue
+ location = (session_id, *coordinates)
+ if candidate_id in candidate_ids:
+ source_candidate_id_collisions.add((index, candidate_id))
+ if location in locations:
+ issues.append(
+ f"retrieval row {index}: {label} contains duplicate evidence"
+ )
+ candidate_ids.add(candidate_id)
+ locations.add(location)
+ identities.append((candidate_id, session_id, *coordinates))
+ return identities
+
+ trace_id = trace_identities(trace, "source Top24 trace")
+ pool_id = trace_identities(pool, "source candidate pool trace")
+ if trace_id != pool_id[: len(trace)]:
+ issues.append(
+ f"retrieval row {index}: source Top24 trace is not the exact pool prefix"
+ )
+ windows = evidence_row.get("evidence_windows")
+ if not isinstance(windows, list):
+ issues.append(f"retrieval row {index}: evidence_windows missing")
+ continue
+ _audit_retrieval_contract(evidence_row, debug_row, windows, index, issues)
+ _audit_session_ordering(windows, index, issues)
+ for rank, window in enumerate(windows, start=1):
+ if not isinstance(window, Mapping) or not _text(window.get("text")):
+ issues.append(f"retrieval row {index}: final evidence text missing")
+ continue
+ if window.get("rank") != rank:
+ issues.append(f"retrieval row {index}: final evidence ranks are not contiguous")
+ attachments = window.get("attachments")
+ if not isinstance(attachments, list) or any(
+ not isinstance(item, Mapping) for item in attachments
+ ):
+ issues.append(
+ f"retrieval row {index}: attachments must be an object list"
+ )
+ contexts = window.get("memory_contexts")
+ if not isinstance(contexts, list) or any(
+ not isinstance(item, Mapping) for item in contexts
+ ):
+ issues.append(
+ f"retrieval row {index}: memory_contexts must be an object list"
+ )
+ provenance = window.get("provenance")
+ if not isinstance(provenance, list) or any(
+ not isinstance(item, Mapping) for item in provenance
+ ):
+ issues.append(
+ f"retrieval row {index}: provenance must be an object list"
+ )
+ retrieval_metadata = window.get("retrieval_metadata")
+ contributions = (
+ retrieval_metadata.get("layer_contributions")
+ if isinstance(retrieval_metadata, Mapping)
+ else None
+ )
+ if (
+ not isinstance(retrieval_metadata, Mapping)
+ or not isinstance(contributions, list)
+ or not contributions
+ or any(not isinstance(item, Mapping) for item in contributions)
+ ):
+ issues.append(
+ f"retrieval row {index}: retrieval_metadata is missing or malformed"
+ )
+ elif (
+ retrieval_metadata.get("session_ordering_policy")
+ != SESSION_ORDERING_POLICY
+ or not isinstance(
+ retrieval_metadata.get("pre_session_order_rank"), int
+ )
+ or not isinstance(retrieval_metadata.get("session_order_rank"), int)
+ or not isinstance(
+ retrieval_metadata.get("session_selected_window_count"), int
+ )
+ or not isinstance(
+ retrieval_metadata.get("session_support_rrf"), (int, float)
+ )
+ ):
+ issues.append(
+ f"retrieval row {index}: session ordering audit is missing or malformed"
+ )
+ text = str(window["text"])
+ memory_id = _text(window.get("memory_id"))
+ source_id = _text(window.get("source_record_id")) or memory_id
+ declared_scope = _text(window.get("scope_id"))
+ declared_db = window.get("db_path")
+ if not source_id or not declared_scope or not _text(declared_db):
+ issues.append(
+ f"retrieval row {index}: final evidence lacks composite Source identity"
+ )
+ continue
+ source_identity = _identity(declared_db, declared_scope, source_id)
+ source = sources.get(source_identity)
+ if source is None:
+ issues.append(
+ f"retrieval row {index}: final evidence source identity is missing or ambiguous: {source_id}"
+ )
+ continue
+ owner_db, owner_scope = source_identity[0], source_identity[1]
+
+ def retrieval_fast_reference(value: Any, label: str) -> Identity | None:
+ reference = _reference_identity(
+ value,
+ db_path=owner_db,
+ scope_id=owner_scope,
+ )
+ if reference is None or reference not in fast:
+ issues.append(
+ f"retrieval row {index}: {label} does not map to current Fast DB identity: {value}"
+ )
+ return None
+ if records and reference not in records:
+ issues.append(
+ f"retrieval row {index}: {label} does not map to current DB record identity: {reference}"
+ )
+ return None
+ return reference
+
+ for attachment_index, attachment in enumerate(attachments if isinstance(attachments, list) else []):
+ if not isinstance(attachment, Mapping):
+ continue
+ attachment_identity = retrieval_fast_reference(
+ attachment,
+ f"Fast attachment {attachment_index}",
+ )
+ if attachment_identity is None:
+ continue
+ leaf = fast[attachment_identity]
+ leaf_meta = leaf["metadata"]
+ if _text(attachment.get("text")) != _text(leaf.get("value")):
+ issues.append(
+ f"retrieval row {index}: Fast attachment {attachment_index} text differs from current DB record"
+ )
+ if _text(attachment.get("record_state")) != _text(leaf.get("state")):
+ issues.append(
+ f"retrieval row {index}: Fast attachment {attachment_index} record state differs from current DB record"
+ )
+ expected_slot = _text(
+ leaf_meta.get("canonical_slot")
+ or leaf_meta.get("canonical_slot_key")
+ )
+ if _text(attachment.get("canonical_slot")) != expected_slot:
+ issues.append(
+ f"retrieval row {index}: Fast attachment {attachment_index} slot differs from current DB record"
+ )
+ expected_fast_fields = {
+ "memory_type": _text(leaf_meta.get("memory_type")),
+ "durability": _text(
+ leaf_meta.get("durability")
+ or leaf_meta.get("durability_class")
+ ),
+ "temporal_status": _text(
+ leaf_meta.get("temporal_status")
+ or leaf_meta.get("target_status")
+ ),
+ }
+ if any(
+ _text(attachment.get(field)) != expected
+ for field, expected in expected_fast_fields.items()
+ ):
+ issues.append(
+ f"retrieval row {index}: Fast attachment {attachment_index} typed state differs from current DB record"
+ )
+ parent = attachment.get("source_parent")
+ expected_parent = _fast_source_parent(leaf_meta)
+ if not isinstance(parent, Mapping) or dict(parent) != expected_parent:
+ issues.append(
+ f"retrieval row {index}: Fast attachment {attachment_index} source parent is not current"
+ )
+ _validate_source_parent(
+ parent,
+ db_path=owner_db,
+ scope_id=owner_scope,
+ sources=sources,
+ issues=issues,
+ label=f"retrieval row {index} Fast attachment {attachment_index}",
+ quote=_fast_quote(leaf_meta),
+ )
+ attachment_provenance = attachment.get("provenance")
+ expected_provenance = {
+ "memory_layer": "fast",
+ "content_variant": PRODUCTION_FAST_VARIANT,
+ "source_record_id": _text(leaf_meta.get("source_record_id")),
+ "semantic_memory_id": attachment_identity[2],
+ }
+ if (
+ not isinstance(attachment_provenance, Mapping)
+ or dict(attachment_provenance) != expected_provenance
+ ):
+ issues.append(
+ f"retrieval row {index}: Fast attachment {attachment_index} provenance is invalid"
+ )
+
+ for context_index, context in enumerate(contexts if isinstance(contexts, list) else []):
+ if not isinstance(context, Mapping):
+ continue
+ try:
+ context_revision = int(context.get("revision"))
+ except (TypeError, ValueError):
+ context_revision = -1
+ context_key = (
+ owner_db,
+ owner_scope,
+ _text(context.get("capsule_id")),
+ context_revision,
+ )
+ slow_record = slow_lookup.get(context_key)
+ if slow_record is None:
+ issues.append(
+ f"retrieval row {index}: Slow context {context_index} is not the current DB revision"
+ )
+ continue
+ if _text(context.get("status")) != _text(slow_record["metadata"].get("status")):
+ issues.append(
+ f"retrieval row {index}: Slow context {context_index} status differs from current DB state"
+ )
+ current_claims = list(slow_record["metadata"].get("claims") or [])
+ current_claim_index = next(
+ (
+ claim_index
+ for claim_index, claim in enumerate(current_claims)
+ if isinstance(claim, Mapping)
+ and _text(claim.get("claim_id"))
+ == _text(context.get("claim_id"))
+ ),
+ -1,
+ )
+ current_claim = next(
+ (
+ claim for claim in current_claims
+ if isinstance(claim, Mapping)
+ and _text(claim.get("claim_id")) == _text(context.get("claim_id"))
+ ),
+ None,
+ )
+ if current_claim is None:
+ issues.append(
+ f"retrieval row {index}: Slow context {context_index} claim is not current DB state"
+ )
+ elif (
+ _text(context.get("canonical_slot")) != _text(current_claim.get("canonical_slot"))
+ or _text(context.get("claim_text")) != _text(current_claim.get("text"))
+ ):
+ issues.append(
+ f"retrieval row {index}: Slow context {context_index} claim text/slot differs from current DB state"
+ )
+ expected_summary_id = (
+ f"capsule-summary::{_text(slow_record['metadata'].get('capsule_id'))}:"
+ f"r{int(slow_record['metadata'].get('revision'))}"
+ )
+ expected_claim_candidate_id = (
+ f"capsule::{_text(slow_record['metadata'].get('capsule_id'))}:"
+ f"r{int(slow_record['metadata'].get('revision'))}:c{current_claim_index}"
+ )
+ if (
+ _text(context.get("capsule_summary"))
+ != _text(slow_record.get("value"))
+ or _text(context.get("capsule_summary_candidate_id"))
+ != expected_summary_id
+ ):
+ issues.append(
+ f"retrieval row {index}: Slow context {context_index} capsule summary differs from current DB state"
+ )
+ if current_claim is not None and (
+ list(context.get("support") or [])
+ != list(current_claim.get("support") or [])
+ or list(context.get("counterevidence") or [])
+ != list(current_claim.get("counterevidence") or [])
+ ):
+ issues.append(
+ f"retrieval row {index}: Slow context {context_index} citations differ from current DB claim"
+ )
+ context_refs: list[Identity] = []
+ for raw_reference in [*(context.get("support") or []), *(context.get("counterevidence") or [])]:
+ reference = retrieval_fast_reference(
+ raw_reference,
+ f"Slow context {context_index} citation",
+ )
+ if reference is not None:
+ context_refs.append(reference)
+ context_parents = context.get("source_parents")
+ expected_context_parents = (
+ list(
+ current_claim.get(
+ "source_parents",
+ slow_record["metadata"].get("source_parents"),
+ )
+ or []
+ )
+ if current_claim is not None
+ else []
+ )
+ if not isinstance(context_parents, list) or not context_parents:
+ issues.append(
+ f"retrieval row {index}: Slow context {context_index} source parents are missing"
+ )
+ else:
+ if context_parents != expected_context_parents:
+ issues.append(
+ f"retrieval row {index}: Slow context {context_index} source parents differ from current DB claim"
+ )
+ for parent in context_parents:
+ _validate_source_parent(
+ parent,
+ db_path=owner_db,
+ scope_id=owner_scope,
+ sources=sources,
+ issues=issues,
+ label=f"retrieval row {index} Slow context {context_index}",
+ )
+ for reference in context_refs:
+ leaf_meta = fast[reference]["metadata"]
+ expected_parent = _slow_source_parent(leaf_meta)
+ if not any(isinstance(parent, Mapping) and dict(parent) == dict(expected_parent) for parent in context_parents):
+ issues.append(
+ f"retrieval row {index}: Slow context {context_index} citation parent is not current"
+ )
+ context_provenance = context.get("provenance")
+ expected_context_provenance = (
+ _expected_slow_context_provenance(
+ slow_record["metadata"],
+ current_claim,
+ expected_context_parents,
+ expected_summary_id,
+ patch_metadata=slow_patch_metadata.get(
+ (
+ owner_db,
+ _text(slow_record["metadata"].get("patch_id")),
+ )
+ ),
+ )
+ if current_claim is not None
+ else None
+ )
+ if (
+ not isinstance(context_provenance, Mapping)
+ or expected_context_provenance is None
+ or dict(context_provenance) != expected_context_provenance
+ ):
+ issues.append(
+ f"retrieval row {index}: Slow context {context_index} provenance is invalid"
+ )
+ context_trace = context.get("retrieval_trace")
+ if (
+ not isinstance(context_trace, Mapping)
+ or context_trace.get("inventory_schema_version")
+ != SLOW_INVENTORY_SCHEMA_VERSION
+ or _text(context_trace.get("claim_candidate_id"))
+ != expected_claim_candidate_id
+ or type(context_trace.get("direct_claim_hit")) is not bool
+ or type(context_trace.get("summary_expansion_hit")) is not bool
+ or not (
+ context_trace.get("direct_claim_hit")
+ or context_trace.get("summary_expansion_hit")
+ )
+ or list(context_trace.get("source_parents") or [])
+ != expected_context_parents
+ ):
+ issues.append(
+ f"retrieval row {index}: Slow context {context_index} summary-to-claim-to-Source trace is invalid"
+ )
+ elif context_trace.get("summary_expansion_hit"):
+ summary_hit = context_trace.get("summary_hit")
+ if (
+ not isinstance(summary_hit, Mapping)
+ or _text(summary_hit.get("summary_candidate_id"))
+ != expected_summary_id
+ or _text(summary_hit.get("summary_text"))
+ != _text(slow_record.get("value"))
+ ):
+ issues.append(
+ f"retrieval row {index}: Slow context {context_index} summary expansion is not current DB state"
+ )
+
+ for citation_index, citation in enumerate(window.get("citations") or []):
+ retrieval_fast_reference(citation, f"final citation {citation_index}")
+ for provenance_index, item in enumerate(provenance if isinstance(provenance, list) else []):
+ if not isinstance(item, Mapping):
+ continue
+ raw_reference = item.get("evidence_memory_id") or item.get("memory_id") or item.get("leaf_id")
+ if raw_reference:
+ retrieval_fast_reference(raw_reference, f"final provenance {provenance_index}")
+
+ if window.get("source_char_start") is None or window.get("source_char_end") is None:
+ issues.append(
+ f"retrieval row {index}: final evidence lacks auditable source offsets: {source_id}"
+ )
+ continue
+ if not _source_span_quote(
+ source,
+ window.get("source_char_start"),
+ window.get("source_char_end"),
+ text,
+ label=f"retrieval row {index} final evidence",
+ issues=issues,
+ ):
+ issues.append(
+ f"retrieval row {index}: final evidence text/offsets do not match current immutable Source"
+ )
+ return {
+ "present": True,
+ "passed": not any(item.startswith("retrieval") for item in issues),
+ "rows": len(debug),
+ "all_layer_path_rows": executed,
+ "source_candidate_id_collision_count": len(source_candidate_id_collisions),
+ "source_candidate_id_collision_rows": sorted(
+ {row_index for row_index, _ in source_candidate_id_collisions}
+ ),
+ }
+
+
+def main(argv: list[str] | None = None) -> int:
+ parser = argparse.ArgumentParser(description="Strict TMCRA V4 post-run chain audit")
+ parser.add_argument("--run-dir", type=Path, required=True)
+ parser.add_argument("--output", type=Path, required=True)
+ parser.add_argument("--worker-db", action="append", default=[], metavar="WORKER=PATH", help="explicit per-worker SQLite path; repeatable")
+ parser.add_argument("--retrieval-dir", type=Path)
+ parser.add_argument(
+ "--build-only",
+ action="store_true",
+ help="audit source/writer/fast/slow build state without selecting historical retrieval output",
+ )
+ args = parser.parse_args(argv)
+ try:
+ if args.build_only and args.retrieval_dir is not None:
+ raise AuditError("--build-only and --retrieval-dir are mutually exclusive")
+ report = audit_run(
+ args.run_dir,
+ output=args.output,
+ worker_db_specs=args.worker_db,
+ retrieval_dir=args.retrieval_dir,
+ build_only=args.build_only,
+ )
+ except (AuditError, OSError, sqlite3.Error) as exc:
+ report = {"schema_version": "tmcra.v4.chain-audit.1", "status": "failed", "passed": False, "issues": [str(exc)]}
+ args.output.parent.mkdir(parents=True, exist_ok=True)
+ args.output.write_text(json.dumps(report, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
+ print(json.dumps(report, ensure_ascii=False, sort_keys=True))
+ return 0 if report.get("passed") else 1
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/legacy/build_v3_runtime_dataset.py b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/build_v3_runtime_dataset.py
new file mode 100644
index 0000000000000000000000000000000000000000..a429a1cc350010a5da69526ba8740559ec80cfc2
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/build_v3_runtime_dataset.py
@@ -0,0 +1,791 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import re
+import time
+from collections import Counter
+from pathlib import Path
+from typing import Any, Iterable, Mapping, Sequence
+
+import torch
+
+from tmcra_v2_lme_pipeline import EmbeddingCacheVectorizer, session_text_chunks
+from tmcra_v3_schema import (
+ CHANNEL_NAMES,
+ SCHEMA_VERSION,
+ clean_text,
+ read_jsonl,
+ validate_sample,
+ validate_split_isolation,
+ write_jsonl,
+)
+
+
+EVENT_INDEX_RE = re.compile(
+ r"^event::longmemeval:(?P[^:]+):s(?P\d+)_c(?P\d+)$"
+)
+TEACHER_EVENT_INDEX_RE = re.compile(r":s(?P\d+)_c(?P\d+)$")
+RAW_TEACHER_EVENT_RE = re.compile(
+ r"^lme:[^:]+:session:(?P[^:]+):turn:(?P\d+)$"
+)
+TURN_BOUNDARY_RE = re.compile(r"\n\[[^\]]+ turn=\d+ role=")
+
+
+def stable_int(value: str) -> int:
+ return int.from_bytes(hashlib.blake2b(value.encode("utf-8"), digest_size=8).digest(), "big")
+
+
+def normalized_question(value: Any) -> str:
+ return clean_text(value).casefold()
+
+
+def load_teacher_supervision(path: Path) -> dict[str, dict[str, Any]]:
+ output: dict[str, dict[str, Any]] = {}
+ for row in read_jsonl(path):
+ question_key = normalized_question(row.get("question"))
+ if not question_key:
+ raise RuntimeError("teacher row is missing question text")
+ if question_key in output:
+ raise RuntimeError(f"duplicate teacher question after exact normalization: {row.get('question')!r}")
+ if not bool(row.get("usable_for_training", False)):
+ continue
+ if not list(row.get("positive_event_ids") or []):
+ raise RuntimeError(f"teacher row has no positive events: {row.get('question_id')}")
+ output[question_key] = row
+ if not output:
+ raise RuntimeError(f"no usable teacher supervision rows: {path}")
+ return output
+
+
+def teacher_parent_locations(event_ids: Iterable[Any], valid_locations: set[tuple[int, int]], qid: str) -> list[tuple[int, int]]:
+ output: list[tuple[int, int]] = []
+ seen: set[tuple[int, int]] = set()
+ for raw in event_ids:
+ event_id = clean_text(raw)
+ match = TEACHER_EVENT_INDEX_RE.search(event_id)
+ if match is None:
+ raise RuntimeError(f"{qid}: teacher event is not a runtime parent event: {event_id}")
+ location = (int(match.group("session")), int(match.group("parent")))
+ if location not in valid_locations:
+ raise RuntimeError(f"{qid}: teacher event has no parent in the current inventory: {event_id}")
+ if location not in seen:
+ seen.add(location)
+ output.append(location)
+ return output
+
+
+def reconstruct_parent_text(candidates: Sequence[Mapping[str, Any]], indexes: Sequence[int], qid: str) -> str:
+ if not indexes:
+ raise RuntimeError(f"{qid}: cannot reconstruct an empty parent")
+ total = max(int(candidates[index]["source_char_end"]) for index in indexes)
+ characters: list[str | None] = [None] * total
+ for index in indexes:
+ candidate = candidates[index]
+ text = str(candidate["text"])
+ if "\n" not in text:
+ raise RuntimeError(f"{qid}: inventory candidate is missing its payload separator")
+ payload = text.split("\n", 1)[1]
+ start = int(candidate["source_char_start"])
+ end = int(candidate["source_char_end"])
+ if len(payload) != end - start:
+ raise RuntimeError(f"{qid}: inventory candidate payload length does not match its source span")
+ for offset, character in enumerate(payload, start=start):
+ existing = characters[offset]
+ if existing is not None and existing != character:
+ raise RuntimeError(f"{qid}: overlapping inventory subchunks disagree")
+ characters[offset] = character
+ if any(character is None for character in characters):
+ raise RuntimeError(f"{qid}: reconstructed parent contains a character coverage gap")
+ return "".join(character for character in characters if character is not None)
+
+
+def teacher_turn_candidate_indexes(
+ event_ids: Iterable[Any],
+ *,
+ candidates: Sequence[Mapping[str, Any]],
+ parent_candidates: Mapping[tuple[int, int], Sequence[int]],
+ qid: str,
+) -> set[int]:
+ output: set[int] = set()
+ session_locations: dict[str, list[tuple[int, int]]] = {}
+ for location, indexes in parent_candidates.items():
+ session_id = clean_text(candidates[indexes[0]].get("session_id"))
+ session_locations.setdefault(session_id, []).append(location)
+ parent_text_cache: dict[tuple[int, int], str] = {}
+ for raw in event_ids:
+ event_id = clean_text(raw)
+ match = RAW_TEACHER_EVENT_RE.match(event_id)
+ if match is None:
+ raise RuntimeError(f"{qid}: teacher raw event is malformed: {event_id}")
+ session_id = clean_text(match.group("session"))
+ turn = int(match.group("turn"))
+ marker = f"[{session_id} turn={turn} role="
+ matches: list[tuple[tuple[int, int], int, int]] = []
+ for location in session_locations.get(session_id, []):
+ if location not in parent_text_cache:
+ parent_text_cache[location] = reconstruct_parent_text(
+ candidates, parent_candidates[location], qid
+ )
+ parent_text = parent_text_cache[location]
+ turn_start = parent_text.find(marker)
+ if turn_start < 0:
+ continue
+ next_turn = TURN_BOUNDARY_RE.search(parent_text, turn_start + len(marker))
+ turn_end = next_turn.start() if next_turn else len(parent_text)
+ matches.append((location, turn_start, turn_end))
+ if len(matches) != 1:
+ raise RuntimeError(
+ f"{qid}: teacher event must map to exactly one current parent turn, got {len(matches)}: {event_id}"
+ )
+ location, turn_start, turn_end = matches[0]
+ matched_subchunks = {
+ index
+ for index in parent_candidates[location]
+ if int(candidates[index]["source_char_start"]) < turn_end
+ and int(candidates[index]["source_char_end"]) > turn_start
+ }
+ if not matched_subchunks:
+ raise RuntimeError(f"{qid}: teacher turn has no overlapping subchunk: {event_id}")
+ output.update(matched_subchunks)
+ return output
+
+
+def covered_windows(text: str, max_chars: int, overlap: int) -> list[tuple[int, int]]:
+ if max_chars <= 0:
+ raise RuntimeError("subchunk payload size must be positive")
+ if overlap < 0 or overlap >= max_chars:
+ raise RuntimeError("subchunk overlap must be non-negative and smaller than the payload")
+ if not text:
+ return []
+ spans: list[tuple[int, int]] = []
+ start = 0
+ while start < len(text):
+ hard_end = min(len(text), start + max_chars)
+ end = hard_end
+ if hard_end < len(text):
+ floor = max(start + max_chars // 2, hard_end - 160)
+ split_at = text.rfind(" ", floor, hard_end)
+ if split_at > start:
+ end = split_at
+ if end <= start:
+ end = hard_end
+ spans.append((start, end))
+ if end >= len(text):
+ break
+ start = max(start + 1, end - overlap)
+ if spans[0][0] != 0 or spans[-1][1] != len(text):
+ raise AssertionError("subchunk coverage does not reach both text boundaries")
+ if any(right_start > left_end for (_, left_end), (right_start, _) in zip(spans, spans[1:])):
+ raise AssertionError("subchunk coverage contains a gap")
+ return spans
+
+
+def load_lme(path: Path) -> list[dict[str, Any]]:
+ value = json.loads(path.read_text(encoding="utf-8"))
+ if not isinstance(value, list) or not value:
+ raise RuntimeError(f"LongMemEval data must be a non-empty list: {path}")
+ rows = [row for row in value if isinstance(row, dict)]
+ qids = [clean_text(row.get("question_id")) for row in rows]
+ if len(qids) != len(set(qids)):
+ raise RuntimeError("LongMemEval question_id values are not unique")
+ return rows
+
+
+def choose_holdout(rows: Sequence[Mapping[str, Any]], count: int) -> list[str]:
+ if count <= 0 or count >= len(rows):
+ raise RuntimeError(f"holdout count must be between 1 and {len(rows) - 1}")
+ by_type: dict[str, list[str]] = {}
+ for row in rows:
+ qid = clean_text(row.get("question_id"))
+ qtype = clean_text(row.get("question_type")) or "unknown"
+ by_type.setdefault(qtype, []).append(qid)
+ selected: list[str] = []
+ for qtype, qids in sorted(by_type.items()):
+ qids.sort(key=lambda qid: (stable_int(f"tmcra-v3-holdout:{qid}"), qid))
+ quota = int(len(qids) * count / len(rows))
+ selected.extend(qids[:quota])
+ selected_set = set(selected)
+ remaining = [clean_text(row.get("question_id")) for row in rows if clean_text(row.get("question_id")) not in selected_set]
+ remaining.sort(key=lambda qid: (stable_int(f"tmcra-v3-holdout-fill:{qid}"), qid))
+ selected.extend(remaining[: max(0, count - len(selected))])
+ selected = sorted(selected[:count])
+ if len(selected) != count or len(set(selected)) != count:
+ raise RuntimeError("failed to construct an exact unique holdout")
+ return selected
+
+
+def make_inventory_row(
+ row: Mapping[str, Any],
+ split: str,
+ parent_chars: int,
+ subchunk_chars: int,
+ subchunk_overlap: int,
+) -> dict[str, Any]:
+ qid = clean_text(row.get("question_id"))
+ question = clean_text(row.get("question"))
+ question_date = clean_text(row.get("question_date"))
+ query_text = f"{question}\nQuestion date: {question_date}" if question_date else question
+ sessions = list(row.get("haystack_sessions") or [])
+ session_ids = [clean_text(value) for value in list(row.get("haystack_session_ids") or [])]
+ dates = [clean_text(value) for value in list(row.get("haystack_dates") or [])]
+ answer_session_ids = {clean_text(value) for value in list(row.get("answer_session_ids") or []) if clean_text(value)}
+ if not sessions or len(sessions) != len(session_ids):
+ raise RuntimeError(f"{qid}: haystack session/id mismatch")
+ candidates: list[dict[str, Any]] = []
+ found_gold: set[str] = set()
+ parent_chunk_count = 0
+ source_parent_chars = 0
+ for index, session in enumerate(sessions):
+ session_id = session_ids[index]
+ date = dates[index] if index < len(dates) else ""
+ if not isinstance(session, Sequence) or isinstance(session, (str, bytes, bytearray)):
+ raise RuntimeError(f"{qid}: session {index} is not a turn list")
+ turns = list(session)
+ if any(not isinstance(turn, Mapping) for turn in turns):
+ raise RuntimeError(f"{qid}: session {index} contains a non-object turn")
+ parent_chunks = session_text_chunks(
+ session_id=session_id,
+ date=date,
+ turns=turns,
+ max_chars=parent_chars,
+ max_chunks=0,
+ )
+ if not parent_chunks:
+ parent_chunks = [f"LongMemEval session_id={session_id} date={date}"]
+ relevant = session_id in answer_session_ids
+ if relevant:
+ found_gold.add(session_id)
+ for parent_index, parent_text in enumerate(parent_chunks, start=1):
+ parent_chunk_count += 1
+ source_parent_chars += len(parent_text)
+ prefix = (
+ f"LongMemEval session_id={session_id} date={date} "
+ f"parent_chunk={parent_index:02d}"
+ )
+ payload_chars = subchunk_chars - len(prefix) - 1
+ spans = covered_windows(parent_text, payload_chars, subchunk_overlap)
+ for subchunk_index, (char_start, char_end) in enumerate(spans, start=1):
+ text = f"{prefix}\n{parent_text[char_start:char_end]}"
+ if len(text) > subchunk_chars:
+ raise AssertionError(f"{qid}: generated subchunk exceeds configured character cap")
+ candidates.append(
+ {
+ "candidate_id": (
+ f"chunk::longmemeval:{qid}:s{index:03d}_c{parent_index:02d}_p{subchunk_index:02d}"
+ ),
+ "text": text,
+ "session_id": session_id,
+ "session_index": index,
+ "parent_chunk_index": parent_index,
+ "subchunk_index": subchunk_index,
+ "source_char_start": char_start,
+ "source_char_end": char_end,
+ "labels": {
+ "relevance": relevant,
+ "hard_negative": False,
+ "evidence_role": "positive" if relevant else "negative",
+ "target_scope": "answer_session_bag",
+ },
+ }
+ )
+ if found_gold != answer_session_ids:
+ raise RuntimeError(f"{qid}: answer sessions missing from haystack: {sorted(answer_session_ids - found_gold)}")
+ return {
+ "schema_version": SCHEMA_VERSION,
+ "question_id": qid,
+ "question": question,
+ "question_date": question_date,
+ "query_text": query_text,
+ "question_type": clean_text(row.get("question_type")),
+ "split": split,
+ "gold_answer": clean_text(row.get("answer")),
+ "answer_session_ids": sorted(answer_session_ids),
+ "supervision": {
+ "target_type": "multi_instance_answer_session_bag",
+ "loss": "negative_log_positive_probability_mass",
+ },
+ "inventory_metadata": {
+ "session_count": len(sessions),
+ "parent_chunk_count": parent_chunk_count,
+ "candidate_subchunk_count": len(candidates),
+ "source_parent_chars": source_parent_chars,
+ "parent_chars": parent_chars,
+ "subchunk_chars": subchunk_chars,
+ "subchunk_overlap": subchunk_overlap,
+ "truncated": False,
+ },
+ "candidates": candidates,
+ }
+
+
+def command_inventory(args: argparse.Namespace) -> None:
+ out_dir = Path(args.out_dir)
+ out_dir.mkdir(parents=True, exist_ok=False)
+ rows = load_lme(Path(args.data))
+ holdout = set(choose_holdout(rows, args.holdout_count))
+ inventory_rows = [
+ make_inventory_row(
+ row,
+ "holdout" if clean_text(row.get("question_id")) in holdout else "train",
+ args.parent_chars,
+ args.subchunk_chars,
+ args.subchunk_overlap,
+ )
+ for row in rows
+ ]
+ holdout_rows = [row for row in inventory_rows if row["split"] == "holdout"]
+ train_rows = [row for row in inventory_rows if row["split"] == "train"]
+ isolation = validate_split_isolation(train_rows, holdout_rows)
+ write_jsonl(out_dir / "inventory.jsonl", inventory_rows)
+ (out_dir / "holdout_qids.txt").write_text("\n".join(sorted(holdout)) + "\n", encoding="utf-8")
+ report = {
+ "schema_version": SCHEMA_VERSION,
+ "created_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
+ "data": str(Path(args.data).resolve()),
+ "row_count": len(inventory_rows),
+ "train_count": len(train_rows),
+ "holdout_count": len(holdout_rows),
+ "candidate_count": sum(len(row["candidates"]) for row in inventory_rows),
+ "candidate_count_avg": round(sum(len(row["candidates"]) for row in inventory_rows) / len(inventory_rows), 4),
+ "parent_chunk_count": sum(row["inventory_metadata"]["parent_chunk_count"] for row in inventory_rows),
+ "session_count": sum(row["inventory_metadata"]["session_count"] for row in inventory_rows),
+ "positive_bag_candidate_count": sum(
+ int(candidate["labels"]["relevance"])
+ for row in inventory_rows
+ for candidate in row["candidates"]
+ ),
+ "max_candidate_chars": max(len(candidate["text"]) for row in inventory_rows for candidate in row["candidates"]),
+ "truncated_candidate_count": 0,
+ "parent_chars": args.parent_chars,
+ "subchunk_chars": args.subchunk_chars,
+ "subchunk_overlap": args.subchunk_overlap,
+ "question_types": dict(Counter(row["question_type"] for row in inventory_rows)),
+ "holdout_question_types": dict(Counter(row["question_type"] for row in holdout_rows)),
+ "isolation": isolation,
+ }
+ (out_dir / "inventory_report.json").write_text(json.dumps(report, indent=2, sort_keys=True), encoding="utf-8")
+ print(json.dumps(report, indent=2, sort_keys=True))
+
+
+def load_graph_debug(run_dir: Path) -> dict[str, dict[str, Any]]:
+ output: dict[str, dict[str, Any]] = {}
+ paths = sorted(run_dir.glob("shard_*_out/samples_debug.jsonl"))
+ if not paths:
+ raise FileNotFoundError(f"no graph debug shards under {run_dir}")
+ for path in paths:
+ for row in read_jsonl(path):
+ qid = clean_text(row.get("question_id"))
+ if qid:
+ output[qid] = row
+ return output
+
+
+def ordered_parent_locations(
+ event_ids: Iterable[Any],
+ *,
+ qid: str,
+ valid_locations: set[tuple[int, int]],
+) -> list[tuple[int, int]]:
+ output: list[tuple[int, int]] = []
+ seen: set[tuple[int, int]] = set()
+ for raw in event_ids:
+ event_id = clean_text(raw)
+ match = EVENT_INDEX_RE.match(event_id)
+ if match is None:
+ continue
+ if clean_text(match.group("qid")) != qid:
+ raise RuntimeError(f"{qid}: graph debug contains an event for another question: {event_id}")
+ location = (int(match.group("session")), int(match.group("parent")))
+ if location not in valid_locations:
+ raise RuntimeError(f"{qid}: graph event has no exact parent chunk mapping: {event_id}")
+ if location in seen:
+ continue
+ seen.add(location)
+ output.append(location)
+ return output
+
+
+def expand_parent_locations(
+ locations: Iterable[tuple[int, int]],
+ parent_candidates: Mapping[tuple[int, int], Sequence[int]],
+) -> list[int]:
+ output: list[int] = []
+ seen: set[int] = set()
+ for location in locations:
+ for index in parent_candidates[location]:
+ if index not in seen:
+ seen.add(index)
+ output.append(index)
+ return output
+
+
+def rrank(rank: int | None) -> float:
+ return 0.0 if rank is None else 1.0 / float(rank + 1)
+
+
+def canonical_candidate(
+ raw: Mapping[str, Any],
+ *,
+ relevant: bool,
+ session_relevant: bool,
+ teacher_relevant: bool | None,
+ dense_score: float,
+ dense_rank: int,
+ graph_rank: int | None,
+ graph_selected: bool,
+ graph_final: bool,
+ session_count: int,
+ forced_positive: bool,
+ hard_negative: bool,
+) -> dict[str, Any]:
+ relevant = bool(relevant)
+ hard_negative = bool(hard_negative and not relevant)
+ channels = {
+ "dense_score": float(dense_score),
+ "dense_rank_rr": rrank(dense_rank),
+ "graph_rank_rr": rrank(graph_rank),
+ "graph_selected": float(bool(graph_selected)),
+ "graph_final": float(bool(graph_final)),
+ "recency_norm": float(raw["session_index"]) / float(max(1, session_count - 1)),
+ }
+ if tuple(channels.keys()) != CHANNEL_NAMES:
+ raise AssertionError("channel construction order changed")
+ return {
+ "candidate_id": raw["candidate_id"],
+ "text": raw["text"],
+ "session_id": raw["session_id"],
+ "session_index": int(raw["session_index"]),
+ "parent_chunk_index": int(raw["parent_chunk_index"]),
+ "subchunk_index": int(raw["subchunk_index"]),
+ "source_char_start": int(raw["source_char_start"]),
+ "source_char_end": int(raw["source_char_end"]),
+ "channels": channels,
+ "labels": {
+ "relevance": relevant,
+ "session_relevance": bool(session_relevant),
+ "teacher_evidence_relevance": teacher_relevant,
+ "hard_negative": hard_negative,
+ "evidence_role": "positive" if relevant else ("hard_negative" if hard_negative else "negative"),
+ "target_scope": "answer_session_bag",
+ },
+ "provenance": {
+ "dense_retrieved": dense_rank >= 0,
+ "graph_retrieved": graph_rank is not None,
+ "graph_selected": bool(graph_selected),
+ "graph_final": bool(graph_final),
+ "forced_positive_for_training": bool(forced_positive),
+ },
+ }
+
+
+def metric_hit(order: Sequence[int], positives: set[int], k: int) -> int:
+ return int(any(index in positives for index in order[:k]))
+
+
+def command_finalize(args: argparse.Namespace) -> None:
+ inventory_rows = read_jsonl(Path(args.inventory))
+ graph_debug = load_graph_debug(Path(args.graph_debug_run))
+ teacher_by_question = load_teacher_supervision(Path(args.teacher_labels))
+ vectorizer = EmbeddingCacheVectorizer(
+ cache_dir=args.embedding_cache,
+ dim=args.text_dim,
+ expected_backend="hf",
+ expected_model=args.embedding_model,
+ )
+ out_dir = Path(args.out_dir)
+ out_dir.mkdir(parents=True, exist_ok=False)
+ train_rows: list[dict[str, Any]] = []
+ holdout_rows: list[dict[str, Any]] = []
+ full_rows: list[dict[str, Any]] = []
+ full_inventory_rows: list[dict[str, Any]] = []
+ provider = Counter()
+ teacher_provider = Counter()
+ supervision_counts = Counter()
+ for source in inventory_rows:
+ qid = clean_text(source["question_id"])
+ candidates = list(source["candidates"])
+ session_count = int(source["inventory_metadata"]["session_count"])
+ candidate_count = len(candidates)
+ parent_candidates: dict[tuple[int, int], list[int]] = {}
+ for index, candidate in enumerate(candidates):
+ location = (int(candidate["session_index"]), int(candidate["parent_chunk_index"]))
+ parent_candidates.setdefault(location, []).append(index)
+ valid_parent_locations = set(parent_candidates)
+ qvec = vectorizer.encode_one(source["query_text"])
+ evec = torch.stack([vectorizer.encode_one(candidate["text"]) for candidate in candidates])
+ dense_scores = (evec @ qvec).tolist()
+ dense_order = sorted(range(candidate_count), key=lambda index: (-dense_scores[index], index))
+ dense_rank = {index: rank for rank, index in enumerate(dense_order)}
+ debug = graph_debug.get(qid, {})
+ retrieval = dict(debug.get("retrieval") or {})
+ selected_parents = ordered_parent_locations(
+ retrieval.get("selected_event_ids") or [], qid=qid, valid_locations=valid_parent_locations
+ )
+ final_parents = ordered_parent_locations(
+ retrieval.get("final_hit_event_ids") or [], qid=qid, valid_locations=valid_parent_locations
+ )
+ recall_parents = ordered_parent_locations(
+ retrieval.get("recall_event_ids") or [], qid=qid, valid_locations=valid_parent_locations
+ )
+ graph_parents: list[tuple[int, int]] = []
+ graph_parent_seen: set[tuple[int, int]] = set()
+ for location in [*selected_parents, *recall_parents]:
+ if location not in graph_parent_seen:
+ graph_parent_seen.add(location)
+ graph_parents.append(location)
+ graph_parent_rank = {location: rank for rank, location in enumerate(graph_parents)}
+ graph_order = expand_parent_locations(graph_parents, parent_candidates)
+ graph_rank = {
+ index: graph_parent_rank[(int(candidates[index]["session_index"]), int(candidates[index]["parent_chunk_index"]))]
+ for index in graph_order
+ }
+ selected_indexes = set(expand_parent_locations(selected_parents, parent_candidates))
+ final_indexes = set(expand_parent_locations(final_parents, parent_candidates))
+ session_positives = {index for index, candidate in enumerate(candidates) if candidate["labels"]["relevance"]}
+ teacher_row = teacher_by_question.get(normalized_question(source["question"]))
+ teacher_positives: set[int] | None = None
+ teacher_hard: set[int] = set()
+ if teacher_row is not None:
+ teacher_metadata = dict(teacher_row.get("metadata") or {})
+ raw_teacher_positive = list(teacher_metadata.get("raw_teacher_positive_event_ids") or [])
+ raw_teacher_hard = list(teacher_metadata.get("raw_teacher_hard_negative_event_ids") or [])
+ if not raw_teacher_positive:
+ raise RuntimeError(f"{qid}: aligned teacher row is missing raw positive turn ids")
+ teacher_positives = teacher_turn_candidate_indexes(
+ raw_teacher_positive,
+ candidates=candidates,
+ parent_candidates=parent_candidates,
+ qid=qid,
+ )
+ teacher_hard = teacher_turn_candidate_indexes(
+ raw_teacher_hard,
+ candidates=candidates,
+ parent_candidates=parent_candidates,
+ qid=qid,
+ )
+ if not teacher_positives:
+ raise RuntimeError(f"{qid}: teacher supervision produced no positive subchunks")
+ mixed_teacher_subchunks = teacher_positives & teacher_hard
+ if mixed_teacher_subchunks:
+ supervision_counts["teacher_mixed_positive_hard_subchunks"] += len(mixed_teacher_subchunks)
+ teacher_hard -= teacher_positives
+ positives = teacher_positives
+ supervision = {
+ "target_type": "teacher_aligned_turn_bag",
+ "loss": "negative_log_positive_probability_mass",
+ "training_weight": float(teacher_row.get("training_weight_hint", 1.0) or 1.0),
+ "teacher_label_model": clean_text((teacher_row.get("metadata") or {}).get("teacher_label_model")),
+ "teacher_verify_model": clean_text((teacher_row.get("metadata") or {}).get("teacher_verify_model")),
+ "teacher_confidence": clean_text(teacher_row.get("label_confidence")),
+ }
+ else:
+ positives = session_positives
+ supervision = {
+ "target_type": "multi_instance_answer_session_bag",
+ "loss": "negative_log_positive_probability_mass",
+ "training_weight": float(args.weak_supervision_weight),
+ }
+ supervision_counts[supervision["target_type"]] += 1
+ provider["n"] += 1
+ for k in (1, 5, 10, 20, 32):
+ provider[f"primary_dense_r@{k}"] += metric_hit(dense_order, positives, k)
+ provider[f"primary_graph_r@{k}"] += metric_hit(graph_order, positives, k)
+ provider[f"session_dense_r@{k}"] += metric_hit(dense_order, session_positives, k)
+ provider[f"session_graph_r@{k}"] += metric_hit(graph_order, session_positives, k)
+ if teacher_positives is not None:
+ teacher_provider[f"dense_r@{k}"] += metric_hit(dense_order, teacher_positives, k)
+ teacher_provider[f"graph_r@{k}"] += metric_hit(graph_order, teacher_positives, k)
+ if teacher_positives is not None:
+ teacher_provider["n"] += 1
+ runtime_indexes: list[int] = []
+ runtime_seen: set[int] = set()
+ graph_runtime_order = expand_parent_locations(graph_parents[: args.graph_k], parent_candidates)
+ for index in [*dense_order[: args.dense_k], *graph_runtime_order]:
+ if index not in runtime_seen:
+ runtime_seen.add(index)
+ runtime_indexes.append(index)
+ provider["union_gold_hit"] += int(bool(positives & runtime_seen))
+ provider["union_session_hit"] += int(bool(session_positives & runtime_seen))
+ if teacher_positives is not None:
+ teacher_provider["union_hit"] += int(bool(teacher_positives & runtime_seen))
+ pre_force_hit = bool(positives & runtime_seen)
+ training_indexes = list(runtime_indexes)
+ forced_indexes: set[int] = set()
+ if source["split"] == "train" and not pre_force_hit:
+ best_positive = next(index for index in dense_order if index in positives)
+ training_indexes.append(best_positive)
+ runtime_seen.add(best_positive)
+ forced_indexes.add(best_positive)
+ def build_row(indexes: Sequence[int], split: str, allow_forced: bool) -> dict[str, Any]:
+ output_candidates = []
+ for index in indexes:
+ hard = index not in positives and (
+ index in teacher_hard
+ or dense_rank[index] < args.hard_negative_k
+ or graph_rank.get(index, 10**9) < args.hard_negative_k
+ )
+ output_candidates.append(
+ canonical_candidate(
+ candidates[index],
+ relevant=index in positives,
+ session_relevant=index in session_positives,
+ teacher_relevant=None if teacher_positives is None else index in teacher_positives,
+ dense_score=dense_scores[index],
+ dense_rank=dense_rank[index],
+ graph_rank=graph_rank.get(index),
+ graph_selected=index in selected_indexes,
+ graph_final=index in final_indexes,
+ session_count=session_count,
+ forced_positive=allow_forced and index in forced_indexes,
+ hard_negative=hard,
+ )
+ )
+ return {
+ "schema_version": SCHEMA_VERSION,
+ "question_id": qid,
+ "question": source["question"],
+ "question_date": source["question_date"],
+ "query_text": source["query_text"],
+ "question_type": source["question_type"],
+ "split": split,
+ "gold_answer": source["gold_answer"],
+ "answer_session_ids": source["answer_session_ids"],
+ "supervision": supervision,
+ "candidates": output_candidates,
+ "pool_metadata": {
+ "session_count": session_count,
+ "inventory_count": candidate_count,
+ "graph_parent_event_count": len(graph_parents),
+ "graph_runtime_candidate_count": len(graph_runtime_order),
+ "runtime_pool_count": len(runtime_indexes),
+ "pre_force_gold_hit": pre_force_hit,
+ "pre_force_session_hit": bool(session_positives & set(runtime_indexes)),
+ "pre_force_teacher_hit": None if teacher_positives is None else bool(teacher_positives & set(runtime_indexes)),
+ "forced_positive_count": len(forced_indexes) if allow_forced else 0,
+ "dense_k": args.dense_k,
+ "graph_k": args.graph_k,
+ "graph_debug_available": bool(debug),
+ "graph_retrieval_mode": retrieval.get("retrieval_mode"),
+ },
+ }
+ runtime_row = build_row(runtime_indexes, "full_eval", False)
+ validate_sample(runtime_row, require_positive=False)
+ full_rows.append(runtime_row)
+ all_row = build_row(list(range(candidate_count)), "full_eval", False)
+ validate_sample(all_row, require_positive=True)
+ full_inventory_rows.append(all_row)
+ if source["split"] == "train":
+ train_row = build_row(training_indexes, "train", True)
+ validate_sample(train_row, require_positive=True)
+ train_rows.append(train_row)
+ else:
+ holdout_row = build_row(runtime_indexes, "holdout", False)
+ validate_sample(holdout_row, require_positive=False)
+ holdout_rows.append(holdout_row)
+ isolation = validate_split_isolation(train_rows, holdout_rows)
+ if len(train_rows) + len(holdout_rows) != len(inventory_rows):
+ raise RuntimeError("finalized row count mismatch")
+ write_jsonl(out_dir / "train.jsonl", train_rows)
+ write_jsonl(out_dir / "holdout.jsonl", holdout_rows)
+ write_jsonl(out_dir / "full_eval.jsonl", full_rows)
+ write_jsonl(out_dir / "full_inventory_eval.jsonl", full_inventory_rows)
+ if args.dev_count <= 0 or args.dev_count >= len(train_rows):
+ raise RuntimeError("dev_count must leave non-empty optimization train and dev sets")
+ ordered_train_qids = sorted(
+ (row["question_id"] for row in train_rows),
+ key=lambda qid: (stable_int(f"tmcra-v3-dev:{qid}"), qid),
+ )
+ dev_qids = set(ordered_train_qids[: args.dev_count])
+ model_train_rows = [row for row in train_rows if row["question_id"] not in dev_qids]
+ full_by_qid = {row["question_id"]: row for row in full_rows}
+ model_dev_rows = [full_by_qid[qid] for qid in ordered_train_qids[: args.dev_count]]
+ if any(row["pool_metadata"]["forced_positive_count"] for row in model_dev_rows):
+ raise RuntimeError("runtime dev rows must never contain forced positives")
+ write_jsonl(out_dir / "model_train.jsonl", model_train_rows)
+ write_jsonl(out_dir / "model_dev_runtime.jsonl", model_dev_rows)
+ (out_dir / "model_dev_qids.txt").write_text("\n".join(ordered_train_qids[: args.dev_count]) + "\n", encoding="utf-8")
+ cross_cache_rows = [*train_rows, *holdout_rows]
+ if len({row["question_id"] for row in cross_cache_rows}) != len(inventory_rows):
+ raise RuntimeError("cross-cache input must contain exactly one row per question")
+ write_jsonl(out_dir / "cross_cache_samples.jsonl", cross_cache_rows)
+ n = max(1, int(provider["n"]))
+ report = {
+ "schema_version": SCHEMA_VERSION,
+ "created_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
+ "train_count": len(train_rows),
+ "holdout_count": len(holdout_rows),
+ "full_eval_count": len(full_rows),
+ "full_inventory_eval_count": len(full_inventory_rows),
+ "cross_cache_sample_count": len(cross_cache_rows),
+ "cross_cache_pair_count": sum(len(row["candidates"]) for row in cross_cache_rows),
+ "model_train_count": len(model_train_rows),
+ "model_dev_runtime_count": len(model_dev_rows),
+ "model_dev_runtime_candidate_miss_count": sum(
+ int(not any(candidate["labels"]["relevance"] for candidate in row["candidates"]))
+ for row in model_dev_rows
+ ),
+ "isolation": isolation,
+ "candidate_pool": {
+ "dense_k": args.dense_k,
+ "graph_k": args.graph_k,
+ "hard_negative_k": args.hard_negative_k,
+ "train_candidate_avg": round(sum(len(row["candidates"]) for row in train_rows) / max(1, len(train_rows)), 4),
+ "holdout_candidate_avg": round(sum(len(row["candidates"]) for row in holdout_rows) / max(1, len(holdout_rows)), 4),
+ "full_inventory_candidate_avg": round(sum(len(row["candidates"]) for row in full_inventory_rows) / max(1, len(full_inventory_rows)), 4),
+ "train_forced_positive_rows": sum(int(row["pool_metadata"]["forced_positive_count"] > 0) for row in train_rows),
+ },
+ "provider_metrics": {
+ key: round(value / n, 6) for key, value in sorted(provider.items()) if key != "n"
+ },
+ "teacher_provider_metrics": {
+ key: round(value / max(1, int(teacher_provider["n"])), 6)
+ for key, value in sorted(teacher_provider.items())
+ if key != "n"
+ },
+ "teacher_supervision_count": int(teacher_provider["n"]),
+ "supervision_counts": dict(supervision_counts),
+ "teacher_labels": str(Path(args.teacher_labels).resolve()),
+ "weak_supervision_weight": args.weak_supervision_weight,
+ "embedding_cache": str(Path(args.embedding_cache).resolve()),
+ "graph_debug_run": str(Path(args.graph_debug_run).resolve()),
+ }
+ (out_dir / "report.json").write_text(json.dumps(report, indent=2, sort_keys=True), encoding="utf-8")
+ print(json.dumps(report, indent=2, sort_keys=True))
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser()
+ sub = parser.add_subparsers(dest="command", required=True)
+ inventory = sub.add_parser("inventory")
+ inventory.add_argument("--data", required=True)
+ inventory.add_argument("--out-dir", required=True)
+ inventory.add_argument("--holdout-count", type=int, default=100)
+ inventory.add_argument("--parent-chars", type=int, default=7000)
+ inventory.add_argument("--subchunk-chars", type=int, default=1800)
+ inventory.add_argument("--subchunk-overlap", type=int, default=200)
+ finalize = sub.add_parser("finalize")
+ finalize.add_argument("--inventory", required=True)
+ finalize.add_argument("--embedding-cache", required=True)
+ finalize.add_argument("--embedding-model", required=True)
+ finalize.add_argument("--graph-debug-run", required=True)
+ finalize.add_argument("--teacher-labels", required=True)
+ finalize.add_argument("--out-dir", required=True)
+ finalize.add_argument("--text-dim", type=int, default=1024)
+ finalize.add_argument("--dense-k", type=int, default=32)
+ finalize.add_argument("--graph-k", type=int, default=24)
+ finalize.add_argument("--hard-negative-k", type=int, default=12)
+ finalize.add_argument("--dev-count", type=int, default=40)
+ finalize.add_argument("--weak-supervision-weight", type=float, default=0.35)
+ args = parser.parse_args()
+ if args.command == "inventory":
+ command_inventory(args)
+ else:
+ command_finalize(args)
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/legacy/ops/audit_tmcra_v4_subject_attribution.py b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/ops/audit_tmcra_v4_subject_attribution.py
new file mode 100644
index 0000000000000000000000000000000000000000..3c2ba7da740d9f7ce3ad20e11ef630c5e843f2dd
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/ops/audit_tmcra_v4_subject_attribution.py
@@ -0,0 +1,877 @@
+#!/usr/bin/env python3
+"""Audit user-memory ownership inside pasted or forwarded documents.
+
+The deterministic router only selects document-shaped source messages. DeepSeek
+Pro makes every semantic keep/quarantine decision from exact Source excerpts.
+Original Source records remain immutable; quarantined Fast records stay in the
+append-only graph but are excluded from current retrieval.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import os
+import re
+import sqlite3
+import sys
+from contextlib import closing
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any, Mapping, Protocol, Sequence
+
+REPO_ROOT = Path(__file__).resolve().parents[1]
+if str(REPO_ROOT) not in sys.path:
+ sys.path.insert(0, str(REPO_ROOT))
+
+from tmcra_v4_batch_writer import DeepSeekBatchClient
+
+
+PROMPT_VERSION = "tmcra-v4-subject-attribution-2026-07-14.3"
+MODEL = "deepseek-v4-pro"
+CURRENT_STATES = {"active", "parallel_active", "promoted", "challenged"}
+DECISIONS = {"keep_user", "quarantine_third_party", "quarantine_ambiguous"}
+
+SYSTEM_PROMPT = """You audit subject ownership for a production personal-memory system.
+Return exactly one JSON object and no prose:
+{"decisions":[{"memory_id":"exact supplied ID","decision":"keep_user|quarantine_third_party|quarantine_ambiguous","actual_subject":"chat_user for keep_user; named local subject for quarantine_third_party; otherwise empty","chat_user_bridge_quote":"exact outside-artifact identity bridge for keep_user; otherwise empty","reason":"concise source-grounded reason"}]}.
+Return every supplied candidate exactly once and never invent an ID.
+
+The chat user is the person who submitted the outer message to this memory system. An author, sender,
+signatory, quoted speaker, recipient, company, or document subject inside an embedded artifact is a separate
+identity and must never be equated with the chat user merely because the artifact appears in a user-role
+message. Text inside a pasted email thread, article, resume, log, transcript, or document is not the chat
+user's conversational voice. A sender or signatory inside the artifact is third party by default.
+
+keep_user is allowed only when text outside the embedded artifact explicitly bridges the chat user to the
+local author or subject, for example "I wrote the email below" or "this is my resume". A sender name,
+signature, first-person wording inside the artifact, or mailbox label is not an identity bridge. If the
+source starts directly with mailbox UI or document text and contains no outside bridge, do not use
+keep_user. A mailbox line such as "to me" identifies the mailbox owner as a recipient, not as the sender.
+For keep_user, actual_subject must be exactly "chat_user". Use quarantine_third_party when the source
+locally attributes the fact to a named author, sender, signatory, quoted speaker, company, or document
+subject. Use quarantine_ambiguous when no actual subject can be established. Useful document facts remain
+in immutable Source; do not force them into the chat user's Fast profile.
+Every keep_user decision must cite chat_user_bridge_quote as an exact Source substring outside the artifact.
+The candidate evidence quote itself cannot be used as that bridge. For both quarantine decisions,
+chat_user_bridge_quote must be empty. Candidate objects intentionally contain no Writer claim or slot;
+judge ownership from Source only.
+Do not use outside knowledge, model confidence, benchmark labels, or the assertion wording as authority.
+Judge against the exact source excerpts and offsets."""
+
+HEADER_RE = re.compile(
+ r"(?im)^(?:from|to|subject|sent|de|para|asunto|enviado):\s*\S"
+)
+MAILBOX_RE = re.compile(r"(?im)^\s*to\s+(?:me|[A-Z][^\n,]{0,40})(?:,|\s*$)")
+SIGNATURE_RE = re.compile(r"(?im)^\s*(?:regards|best regards|sincerely),?\s*$")
+LEGAL_RE = re.compile(r"(?i)confidentiality notice|electronic communications privacy act")
+THREAD_RE = re.compile(r"(?i)view entire message|scanned by gmail|\bon .{0,120} wrote:\s*$", re.M)
+
+
+class AttributionError(RuntimeError):
+ pass
+
+
+class AttributionClient(Protocol):
+ def complete(self, payload: Mapping[str, Any]) -> tuple[str, Mapping[str, Any]]:
+ ...
+
+
+class DeepSeekProAttributionClient:
+ def __init__(self) -> None:
+ base_url = (
+ os.getenv("TMCRA_DEEPSEEK_PRO_BASE_URL")
+ or os.getenv("TMCRA_WRITER_BASE_URL")
+ or "https://api.deepseek.com/v1"
+ )
+ raw_keys = (
+ os.getenv("TMCRA_DEEPSEEK_PRO_KEY_POOL")
+ or os.getenv("TMCRA_WRITER_API_KEY_POOL")
+ or os.getenv("TMCRA_DEEPSEEK_WRITER_KEY_POOL")
+ or ""
+ )
+ keys = [item.strip() for item in raw_keys.split(",") if item.strip()]
+ if not keys:
+ raise AttributionError("DeepSeek Pro key pool is empty; no fallback is allowed")
+ max_tokens = int(os.getenv("TMCRA_DEEPSEEK_PRO_MAX_TOKENS", "16384"))
+ self.client = DeepSeekBatchClient(
+ base_url=base_url,
+ model=MODEL,
+ api_keys=keys,
+ timeout=float(os.getenv("TMCRA_DEEPSEEK_PRO_TIMEOUT", "180")),
+ max_tokens=max_tokens,
+ )
+
+ def complete(self, payload: Mapping[str, Any]) -> tuple[str, Mapping[str, Any]]:
+ return self.client._complete(
+ model=MODEL,
+ system_prompt=SYSTEM_PROMPT,
+ payload=payload,
+ stage="subject_attribution_pro",
+ )
+
+
+def _now() -> str:
+ return datetime.now(timezone.utc).isoformat(timespec="milliseconds")
+
+
+def _json(value: Any) -> str:
+ return json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
+
+
+def _digest(value: Any) -> str:
+ return hashlib.sha256(_json(value).encode("utf-8")).hexdigest()
+
+
+def _text(value: Any) -> str:
+ return str(value).strip() if value is not None else ""
+
+
+def document_route_reasons(content: str) -> list[str]:
+ """Route only; these signals never decide whether an assertion is valid."""
+ reasons: list[str] = []
+ if HEADER_RE.search(content):
+ reasons.append("mail_header")
+ if MAILBOX_RE.search(content):
+ reasons.append("mailbox_recipient_line")
+ if SIGNATURE_RE.search(content):
+ reasons.append("signature_block")
+ if LEGAL_RE.search(content):
+ reasons.append("mail_legal_notice")
+ if THREAD_RE.search(content):
+ reasons.append("quoted_thread")
+ return reasons if len(reasons) >= 2 else []
+
+
+def _metadata(row: Mapping[str, Any]) -> dict[str, Any]:
+ try:
+ value = json.loads(str(row["metadata_json"]))
+ except (KeyError, TypeError, json.JSONDecodeError) as exc:
+ raise AttributionError("record metadata is invalid JSON") from exc
+ if not isinstance(value, dict):
+ raise AttributionError("record metadata must be an object")
+ return value
+
+
+def _initialize(connection: sqlite3.Connection) -> None:
+ connection.execute(
+ """
+ CREATE TABLE IF NOT EXISTS v4_subject_attribution_audits (
+ audit_id TEXT PRIMARY KEY,
+ scope_id TEXT NOT NULL,
+ message_id TEXT NOT NULL,
+ prompt_version TEXT NOT NULL,
+ model TEXT NOT NULL,
+ request_json TEXT NOT NULL,
+ request_sha256 TEXT NOT NULL,
+ status TEXT NOT NULL,
+ response_json TEXT NOT NULL DEFAULT '',
+ response_sha256 TEXT NOT NULL DEFAULT '',
+ call_metadata_json TEXT NOT NULL DEFAULT '{}',
+ decisions_json TEXT NOT NULL DEFAULT '[]',
+ error TEXT NOT NULL DEFAULT '',
+ created_at TEXT NOT NULL,
+ updated_at TEXT NOT NULL,
+ UNIQUE(scope_id,message_id,request_sha256)
+ )
+ """
+ )
+
+
+def _source_rows(connection: sqlite3.Connection, scope_id: str) -> list[sqlite3.Row]:
+ return connection.execute(
+ "SELECT scope_id,message_id,session_index,message_index,message_role,"
+ "source_turn_index,content,content_sha256 FROM v4_source_journal "
+ "WHERE scope_id=? AND message_role='user' AND status='enriched' "
+ "ORDER BY session_index,message_index",
+ (scope_id,),
+ ).fetchall()
+
+
+def _candidate_records(
+ connection: sqlite3.Connection, scope_id: str, source: Mapping[str, Any]
+) -> list[dict[str, Any]]:
+ output: list[dict[str, Any]] = []
+ rows = connection.execute(
+ "SELECT memory_id,value,slot_key,turn_index,state,metadata_json FROM records "
+ "WHERE scope_id=? AND turn_index=? ORDER BY memory_id",
+ (scope_id, int(source["source_turn_index"])),
+ ).fetchall()
+ for row in rows:
+ metadata = _metadata(row)
+ if (
+ metadata.get("content_variant") != "product_semantic_memory"
+ or metadata.get("memory_layer") != "fast"
+ or metadata.get("node_kind") != "atomic_user_assertion"
+ or _text(row["state"]) not in CURRENT_STATES
+ or _text(metadata.get("message_id")) != _text(source["message_id"])
+ ):
+ continue
+ start = metadata.get("evidence_char_start")
+ end = metadata.get("evidence_char_end")
+ if (
+ isinstance(start, bool)
+ or isinstance(end, bool)
+ or not isinstance(start, int)
+ or not isinstance(end, int)
+ or not 0 <= start < end <= len(str(source["content"]))
+ ):
+ raise AttributionError(f"{row['memory_id']}: evidence offsets are invalid")
+ quote = str(source["content"])[start:end]
+ expected_quote = _text(metadata.get("source_span") or metadata.get("raw_content"))
+ if quote != expected_quote:
+ raise AttributionError(f"{row['memory_id']}: Source quote drift")
+ output.append(
+ {
+ "memory_id": str(row["memory_id"]),
+ "claim_text": str(row["value"]),
+ "canonical_slot": _text(
+ metadata.get("canonical_slot_key") or row["slot_key"]
+ ),
+ "evidence_quote": quote,
+ "evidence_char_start": start,
+ "evidence_char_end": end,
+ }
+ )
+ return output
+
+
+def _source_segments(content: str, candidates: Sequence[Mapping[str, Any]]) -> list[dict[str, Any]]:
+ ranges = [(0, min(len(content), 700))]
+ for candidate in candidates:
+ start = max(0, int(candidate["evidence_char_start"]) - 420)
+ end = min(len(content), int(candidate["evidence_char_end"]) + 420)
+ ranges.append((start, end))
+ ranges.sort()
+ merged: list[list[int]] = []
+ for start, end in ranges:
+ if merged and start <= merged[-1][1] + 80:
+ merged[-1][1] = max(merged[-1][1], end)
+ else:
+ merged.append([start, end])
+ return [
+ {"char_start": start, "char_end": end, "text": content[start:end]}
+ for start, end in merged
+ ]
+
+
+def scan_database(database: Path, scope_id: str) -> list[dict[str, Any]]:
+ with closing(sqlite3.connect(database)) as connection:
+ connection.row_factory = sqlite3.Row
+ jobs: list[dict[str, Any]] = []
+ for source in _source_rows(connection, scope_id):
+ reasons = document_route_reasons(str(source["content"]))
+ if not reasons:
+ continue
+ review_candidates = _candidate_records(connection, scope_id, source)
+ if not review_candidates:
+ continue
+ candidates = [
+ {
+ "memory_id": item["memory_id"],
+ "evidence_quote": item["evidence_quote"],
+ "evidence_char_start": item["evidence_char_start"],
+ "evidence_char_end": item["evidence_char_end"],
+ }
+ for item in review_candidates
+ ]
+ payload = {
+ "scope_id": scope_id,
+ "message_id": str(source["message_id"]),
+ "message_role": "user",
+ "route_reasons": reasons,
+ "source_segments": _source_segments(
+ str(source["content"]), review_candidates
+ ),
+ "candidates": candidates,
+ }
+ jobs.append(
+ {
+ "database": str(database),
+ "scope_id": scope_id,
+ "message_id": str(source["message_id"]),
+ "session_index": int(source["session_index"]),
+ "message_index": int(source["message_index"]),
+ "source_turn_index": int(source["source_turn_index"]),
+ "route_reasons": reasons,
+ "payload": payload,
+ "review_candidates": review_candidates,
+ "request_sha256": _digest(
+ {
+ "prompt_version": PROMPT_VERSION,
+ "model": MODEL,
+ "payload": payload,
+ }
+ ),
+ }
+ )
+ return jobs
+
+
+def validate_decisions(
+ raw: Any, payload: Mapping[str, Any]
+) -> list[dict[str, str]]:
+ candidates = payload.get("candidates")
+ source_segments = payload.get("source_segments")
+ if not isinstance(candidates, list) or not isinstance(source_segments, list):
+ raise AttributionError("attribution request payload is malformed")
+ candidate_ids = [
+ _text(item.get("memory_id"))
+ for item in candidates
+ if isinstance(item, Mapping)
+ ]
+ if len(candidate_ids) != len(candidates) or not all(candidate_ids):
+ raise AttributionError("attribution request candidate identity is malformed")
+ candidate_by_id = {
+ _text(item["memory_id"]): item
+ for item in candidates
+ if isinstance(item, Mapping)
+ }
+ source_texts = [
+ _text(item.get("text"))
+ for item in source_segments
+ if isinstance(item, Mapping) and _text(item.get("text"))
+ ]
+ if not isinstance(raw, Mapping) or set(raw) != {"decisions"}:
+ raise AttributionError("attribution response must contain exactly decisions")
+ decisions = raw.get("decisions")
+ if not isinstance(decisions, list) or len(decisions) != len(candidate_ids):
+ raise AttributionError("attribution response decision count changed")
+ normalized: list[dict[str, str]] = []
+ seen: set[str] = set()
+ for index, item in enumerate(decisions):
+ if not isinstance(item, Mapping) or set(item) != {
+ "memory_id",
+ "decision",
+ "actual_subject",
+ "chat_user_bridge_quote",
+ "reason",
+ }:
+ raise AttributionError(f"decisions[{index}] has an invalid shape")
+ memory_id = _text(item.get("memory_id"))
+ decision = _text(item.get("decision"))
+ actual_subject = _text(item.get("actual_subject"))
+ bridge_quote = _text(item.get("chat_user_bridge_quote"))
+ reason = _text(item.get("reason"))
+ if memory_id not in candidate_ids or memory_id in seen:
+ raise AttributionError(f"decisions[{index}] changed candidate identity")
+ if decision not in DECISIONS or not reason:
+ raise AttributionError(f"decisions[{index}] has an invalid decision")
+ if decision == "keep_user" and actual_subject != "chat_user":
+ raise AttributionError(
+ f"decisions[{index}] keep_user must bind actual_subject to chat_user"
+ )
+ if decision == "keep_user":
+ evidence_quote = _text(candidate_by_id[memory_id].get("evidence_quote"))
+ if (
+ not bridge_quote
+ or bridge_quote == evidence_quote
+ or not any(bridge_quote in text for text in source_texts)
+ ):
+ raise AttributionError(
+ f"decisions[{index}] keep_user lacks an exact outside-artifact bridge"
+ )
+ elif bridge_quote:
+ raise AttributionError(
+ f"decisions[{index}] quarantine decision cannot cite a chat-user bridge"
+ )
+ if decision == "quarantine_third_party" and not actual_subject:
+ raise AttributionError(
+ f"decisions[{index}] must identify the third-party subject"
+ )
+ if decision == "quarantine_ambiguous" and actual_subject:
+ raise AttributionError(
+ f"decisions[{index}] ambiguous subject must remain empty"
+ )
+ seen.add(memory_id)
+ normalized.append(
+ {
+ "memory_id": memory_id,
+ "decision": decision,
+ "actual_subject": actual_subject,
+ "chat_user_bridge_quote": bridge_quote,
+ "reason": reason,
+ }
+ )
+ if seen != set(candidate_ids):
+ raise AttributionError("attribution response omitted candidate identities")
+ normalized.sort(key=lambda item: candidate_ids.index(item["memory_id"]))
+ return normalized
+
+
+def _replacement_head(
+ connection: sqlite3.Connection,
+ scope_id: str,
+ slot_key: str,
+ excluded_ids: set[str],
+) -> str:
+ rows = connection.execute(
+ "SELECT h.memory_id,r.state FROM slot_history h JOIN records r "
+ "ON r.scope_id=h.scope_id AND r.memory_id=h.memory_id "
+ "WHERE h.scope_id=? AND h.slot_key=? ORDER BY h.ordinal DESC",
+ (scope_id, slot_key),
+ ).fetchall()
+ return next(
+ (
+ str(row["memory_id"])
+ for row in rows
+ if str(row["memory_id"]) not in excluded_ids
+ and _text(row["state"]) in CURRENT_STATES
+ ),
+ "",
+ )
+
+
+def _repair_slot_head(
+ connection: sqlite3.Connection,
+ scope_id: str,
+ slot_key: str,
+ memory_id: str,
+ excluded_ids: set[str],
+) -> None:
+ head = connection.execute(
+ "SELECT memory_id FROM slot_heads WHERE scope_id=? AND slot_key=?",
+ (scope_id, slot_key),
+ ).fetchone()
+ if head is None or str(head["memory_id"]) != memory_id:
+ return
+ replacement = _replacement_head(connection, scope_id, slot_key, excluded_ids)
+ if replacement:
+ connection.execute(
+ "UPDATE slot_heads SET memory_id=? WHERE scope_id=? AND slot_key=?",
+ (replacement, scope_id, slot_key),
+ )
+ else:
+ connection.execute(
+ "DELETE FROM slot_heads WHERE scope_id=? AND slot_key=?",
+ (scope_id, slot_key),
+ )
+
+
+def _apply_decisions(
+ connection: sqlite3.Connection,
+ job: Mapping[str, Any],
+ audit_id: str,
+ decisions: Sequence[Mapping[str, str]],
+) -> dict[str, Any]:
+ candidate_ids = [item["memory_id"] for item in job["payload"]["candidates"]]
+ rows = connection.execute(
+ "SELECT memory_id,slot_key,state,metadata_json FROM records WHERE scope_id=? "
+ f"AND memory_id IN ({','.join('?' for _ in candidate_ids)})",
+ (job["scope_id"], *candidate_ids),
+ ).fetchall()
+ if {str(row["memory_id"]) for row in rows} != set(candidate_ids):
+ raise AttributionError("candidate records changed before attribution commit")
+ decision_by_id = {item["memory_id"]: item for item in decisions}
+ quarantine_ids = {
+ item["memory_id"]
+ for item in decisions
+ if item["decision"] != "keep_user"
+ }
+ parent_signature_map: dict[str, tuple[str, Mapping[str, str]]] = {}
+ for row in rows:
+ memory_id = str(row["memory_id"])
+ if memory_id not in quarantine_ids:
+ continue
+ signature = _text(_metadata(row).get("event_signature"))
+ if not signature:
+ continue
+ if signature in parent_signature_map:
+ raise AttributionError("quarantined parent event signature is not unique")
+ parent_signature_map[signature] = (memory_id, decision_by_id[memory_id])
+
+ dependent_rows: list[sqlite3.Row] = []
+ if parent_signature_map:
+ for row in connection.execute(
+ "SELECT memory_id,slot_key,state,metadata_json FROM records WHERE scope_id=?",
+ (job["scope_id"],),
+ ).fetchall():
+ if str(row["memory_id"]) in candidate_ids:
+ continue
+ parent_signature = _text(
+ _metadata(row).get("facet_parent_event_signature")
+ )
+ if parent_signature in parent_signature_map:
+ dependent_rows.append(row)
+
+ dependent_ids = {str(row["memory_id"]) for row in dependent_rows}
+ all_quarantine_ids = quarantine_ids | dependent_ids
+ changed_ids: list[str] = []
+ for row in rows:
+ memory_id = str(row["memory_id"])
+ decision = decision_by_id[memory_id]
+ metadata = _metadata(row)
+ metadata["subject_attribution_audit_id"] = audit_id
+ metadata["subject_attribution_prompt_version"] = PROMPT_VERSION
+ metadata["subject_attribution_model"] = MODEL
+ metadata["subject_attribution_decision"] = decision["decision"]
+ metadata["subject_attribution_actual_subject"] = decision["actual_subject"]
+ metadata["subject_attribution_chat_user_bridge_quote"] = decision[
+ "chat_user_bridge_quote"
+ ]
+ metadata["subject_attribution_reason"] = decision["reason"]
+ state = str(row["state"])
+ if memory_id in quarantine_ids:
+ state = "quarantined"
+ metadata["excluded_from_retrieval"] = True
+ metadata["conflict_action"] = "subject_attribution_quarantine"
+ connection.execute(
+ "UPDATE records SET state=?,metadata_json=? WHERE scope_id=? AND memory_id=?",
+ (state, _json(metadata), job["scope_id"], memory_id),
+ )
+ changed_ids.append(memory_id)
+ if memory_id not in quarantine_ids:
+ continue
+ slot_key = str(row["slot_key"])
+ _repair_slot_head(
+ connection,
+ job["scope_id"],
+ slot_key,
+ memory_id,
+ all_quarantine_ids,
+ )
+
+ for row in dependent_rows:
+ memory_id = str(row["memory_id"])
+ metadata = _metadata(row)
+ parent_signature = _text(metadata.get("facet_parent_event_signature"))
+ parent_memory_id, parent_decision = parent_signature_map[parent_signature]
+ metadata["subject_attribution_audit_id"] = audit_id
+ metadata["subject_attribution_prompt_version"] = PROMPT_VERSION
+ metadata["subject_attribution_model"] = MODEL
+ metadata["subject_attribution_decision"] = "quarantine_with_parent"
+ metadata["subject_attribution_parent_memory_id"] = parent_memory_id
+ metadata["subject_attribution_parent_decision"] = parent_decision["decision"]
+ metadata["subject_attribution_actual_subject"] = parent_decision[
+ "actual_subject"
+ ]
+ metadata["subject_attribution_chat_user_bridge_quote"] = ""
+ metadata["subject_attribution_reason"] = (
+ "structural facet of quarantined parent assertion"
+ )
+ metadata["excluded_from_retrieval"] = True
+ metadata["conflict_action"] = "subject_attribution_parent_quarantine"
+ connection.execute(
+ "UPDATE records SET state='quarantined',metadata_json=? "
+ "WHERE scope_id=? AND memory_id=?",
+ (_json(metadata), job["scope_id"], memory_id),
+ )
+ changed_ids.append(memory_id)
+ _repair_slot_head(
+ connection,
+ job["scope_id"],
+ str(row["slot_key"]),
+ memory_id,
+ all_quarantine_ids,
+ )
+ return {
+ "changed_memory_ids": sorted(changed_ids),
+ "quarantined_memory_ids": sorted(quarantine_ids),
+ "cascaded_quarantined_memory_ids": sorted(dependent_ids),
+ "kept_memory_ids": sorted(set(candidate_ids) - quarantine_ids),
+ }
+
+
+def _reused_application_state(
+ connection: sqlite3.Connection,
+ job: Mapping[str, Any],
+ audit_id: str,
+ decisions: Sequence[Mapping[str, str]],
+) -> dict[str, Any]:
+ candidate_ids = {item["memory_id"] for item in job["payload"]["candidates"]}
+ quarantine_ids = {
+ item["memory_id"] for item in decisions if item["decision"] != "keep_user"
+ }
+ dependent_ids: set[str] = set()
+ for row in connection.execute(
+ "SELECT memory_id,metadata_json FROM records WHERE scope_id=?",
+ (job["scope_id"],),
+ ).fetchall():
+ metadata = _metadata(row)
+ if (
+ _text(metadata.get("subject_attribution_audit_id")) == audit_id
+ and _text(metadata.get("subject_attribution_decision"))
+ == "quarantine_with_parent"
+ and _text(metadata.get("subject_attribution_parent_memory_id"))
+ in quarantine_ids
+ ):
+ dependent_ids.add(str(row["memory_id"]))
+ return {
+ "changed_memory_ids": sorted(candidate_ids | dependent_ids),
+ "quarantined_memory_ids": sorted(quarantine_ids),
+ "cascaded_quarantined_memory_ids": sorted(dependent_ids),
+ "kept_memory_ids": sorted(candidate_ids - quarantine_ids),
+ }
+
+
+def _cost(metadata: Mapping[str, Any]) -> float:
+ usage = metadata.get("usage")
+ usage = usage if isinstance(usage, Mapping) else metadata
+ prompt = int(usage.get("prompt_tokens", 0) or 0)
+ completion = int(usage.get("completion_tokens", 0) or 0)
+ hit = int(usage.get("prompt_cache_hit_tokens", 0) or 0)
+ miss = int(usage.get("prompt_cache_miss_tokens", prompt - hit) or 0)
+ prompt_rate = float(os.getenv("TMCRA_DEEPSEEK_PRO_PROMPT_COST_PER_MILLION", "3"))
+ completion_rate = float(os.getenv("TMCRA_DEEPSEEK_PRO_COMPLETION_COST_PER_MILLION", "6"))
+ cache_rate = float(os.getenv("TMCRA_DEEPSEEK_PRO_CACHE_COST_PER_MILLION", "0.025"))
+ return (miss * prompt_rate + hit * cache_rate + completion * completion_rate) / 1_000_000
+
+
+def execute_job(
+ database: Path,
+ job: Mapping[str, Any],
+ client: AttributionClient,
+) -> dict[str, Any]:
+ audit_id = "saa_" + job["request_sha256"][:24]
+ request_json = _json(job["payload"])
+ with closing(sqlite3.connect(database, timeout=30)) as connection:
+ connection.row_factory = sqlite3.Row
+ _initialize(connection)
+ existing = connection.execute(
+ "SELECT status,decisions_json,call_metadata_json FROM "
+ "v4_subject_attribution_audits WHERE audit_id=?",
+ (audit_id,),
+ ).fetchone()
+ if existing is not None:
+ if str(existing["status"]) != "completed":
+ raise AttributionError(
+ f"{audit_id}: prior attribution call is not safely reusable"
+ )
+ decisions = json.loads(str(existing["decisions_json"]))
+ applied = _reused_application_state(
+ connection, job, audit_id, decisions
+ )
+ return {
+ "audit_id": audit_id,
+ "status": "reused",
+ "physical_api_calls": 0,
+ "decisions": decisions,
+ "call_metadata": json.loads(str(existing["call_metadata_json"])),
+ **applied,
+ }
+ now = _now()
+ connection.execute(
+ "INSERT INTO v4_subject_attribution_audits VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
+ (
+ audit_id,
+ job["scope_id"],
+ job["message_id"],
+ PROMPT_VERSION,
+ MODEL,
+ request_json,
+ job["request_sha256"],
+ "api_started",
+ "",
+ "",
+ "{}",
+ "[]",
+ "",
+ now,
+ now,
+ ),
+ )
+ connection.commit()
+
+ try:
+ content, call_metadata = client.complete(job["payload"])
+ except Exception as exc:
+ with closing(sqlite3.connect(database, timeout=30)) as connection:
+ connection.execute(
+ "UPDATE v4_subject_attribution_audits SET status='failed',error=?,"
+ "call_metadata_json=?,updated_at=? "
+ "WHERE audit_id=? AND status='api_started'",
+ (
+ str(exc),
+ _json(dict(getattr(exc, "metadata", {}) or {})),
+ _now(),
+ audit_id,
+ ),
+ )
+ connection.commit()
+ raise
+
+ with closing(sqlite3.connect(database, timeout=30)) as connection:
+ received = connection.execute(
+ "UPDATE v4_subject_attribution_audits SET status='response_received',"
+ "response_json=?,response_sha256=?,call_metadata_json=?,updated_at=? "
+ "WHERE audit_id=? AND status='api_started'",
+ (
+ content,
+ hashlib.sha256(content.encode("utf-8")).hexdigest(),
+ _json(dict(call_metadata)),
+ _now(),
+ audit_id,
+ ),
+ )
+ if received.rowcount != 1:
+ raise AttributionError("attribution journal changed before response save")
+ connection.commit()
+ try:
+ raw = json.loads(content)
+ decisions = validate_decisions(raw, job["payload"])
+ except (json.JSONDecodeError, AttributionError) as exc:
+ error = (
+ "attribution response content is not JSON"
+ if isinstance(exc, json.JSONDecodeError)
+ else str(exc)
+ )
+ with closing(sqlite3.connect(database, timeout=30)) as connection:
+ connection.execute(
+ "UPDATE v4_subject_attribution_audits SET status='failed',error=?,"
+ "updated_at=? WHERE audit_id=? AND status='response_received'",
+ (error, _now(), audit_id),
+ )
+ connection.commit()
+ raise AttributionError(error) from exc
+ with closing(sqlite3.connect(database, timeout=30)) as connection:
+ connection.row_factory = sqlite3.Row
+ connection.execute("BEGIN IMMEDIATE")
+ applied = _apply_decisions(connection, job, audit_id, decisions)
+ superseded = connection.execute(
+ "UPDATE v4_subject_attribution_audits SET status='superseded',error=?,"
+ "updated_at=? WHERE scope_id=? AND message_id=? "
+ "AND status IN ('completed','superseded') "
+ "AND audit_id<>?",
+ (
+ f"superseded_by:{audit_id}",
+ _now(),
+ job["scope_id"],
+ job["message_id"],
+ audit_id,
+ ),
+ ).rowcount
+ updated = connection.execute(
+ "UPDATE v4_subject_attribution_audits SET status='completed',response_json=?,"
+ "response_sha256=?,call_metadata_json=?,decisions_json=?,updated_at=? "
+ "WHERE audit_id=? AND status='response_received'",
+ (
+ content,
+ hashlib.sha256(content.encode("utf-8")).hexdigest(),
+ _json(dict(call_metadata)),
+ _json(decisions),
+ _now(),
+ audit_id,
+ ),
+ )
+ if updated.rowcount != 1:
+ raise AttributionError("attribution journal changed before commit")
+ connection.commit()
+ return {
+ "audit_id": audit_id,
+ "status": "completed",
+ "physical_api_calls": 1,
+ "estimated_cost_cny": _cost(call_metadata),
+ "decisions": decisions,
+ "call_metadata": dict(call_metadata),
+ "superseded_audit_count": int(superseded),
+ **applied,
+ }
+
+
+def _worker_databases(run_dir: Path, workers: Sequence[str]) -> list[tuple[str, Path, str]]:
+ selected = set(workers)
+ output: list[tuple[str, Path, str]] = []
+ for database in sorted((run_dir / "writer").glob("worker_*/native_memory.sqlite3")):
+ worker = database.parent.name
+ if selected and worker not in selected:
+ continue
+ with closing(sqlite3.connect(database)) as connection:
+ row = connection.execute(
+ "SELECT scope_id FROM v4_source_journal LIMIT 1"
+ ).fetchone()
+ if row is None or not _text(row[0]):
+ raise AttributionError(f"{worker}: scope is missing")
+ output.append((worker, database, _text(row[0])))
+ if selected - {item[0] for item in output}:
+ raise AttributionError(
+ "unknown workers: " + ",".join(sorted(selected - {item[0] for item in output}))
+ )
+ return output
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--run-dir", type=Path, required=True)
+ parser.add_argument("--workers", default="")
+ parser.add_argument("--apply", action="store_true")
+ parser.add_argument("--output", type=Path, required=True)
+ args = parser.parse_args()
+ run_dir = args.run_dir.resolve()
+ workers = [item.strip() for item in args.workers.split(",") if item.strip()]
+ scanned: list[dict[str, Any]] = []
+ worker_by_database: dict[str, str] = {}
+ for worker, database, scope_id in _worker_databases(run_dir, workers):
+ worker_by_database[str(database)] = worker
+ for job in scan_database(database, scope_id):
+ scanned.append({"worker": worker, **job})
+ results: list[dict[str, Any]] = []
+ if args.apply and scanned:
+ client = DeepSeekProAttributionClient()
+ for job in scanned:
+ results.append(
+ {
+ "worker": job["worker"],
+ "message_id": job["message_id"],
+ **execute_job(Path(job["database"]), job, client),
+ }
+ )
+ report = {
+ "schema_version": "tmcra.v4.subject-attribution-report.1",
+ "status": "complete",
+ "mode": "apply" if args.apply else "scan_only",
+ "prompt_version": PROMPT_VERSION,
+ "model": MODEL,
+ "run_dir": str(run_dir),
+ "scanned_worker_count": len(_worker_databases(run_dir, workers)),
+ "routed_message_count": len(scanned),
+ "routed_candidate_count": sum(
+ len(job["payload"]["candidates"]) for job in scanned
+ ),
+ "physical_api_calls": sum(
+ int(item.get("physical_api_calls", 0) or 0) for item in results
+ ),
+ "estimated_cost_cny": round(
+ sum(float(item.get("estimated_cost_cny", 0.0) or 0.0) for item in results),
+ 8,
+ ),
+ "decision_quarantined_count": sum(
+ len(item.get("quarantined_memory_ids", [])) for item in results
+ ),
+ "cascaded_quarantined_count": sum(
+ len(item.get("cascaded_quarantined_memory_ids", []))
+ for item in results
+ ),
+ "quarantined_count": sum(
+ len(item.get("quarantined_memory_ids", []))
+ + len(item.get("cascaded_quarantined_memory_ids", []))
+ for item in results
+ ),
+ "routed": [
+ {
+ "worker": job["worker"],
+ "database": job["database"],
+ "scope_id": job["scope_id"],
+ "message_id": job["message_id"],
+ "session_index": job["session_index"],
+ "message_index": job["message_index"],
+ "source_turn_index": job["source_turn_index"],
+ "route_reasons": job["route_reasons"],
+ "request_sha256": job["request_sha256"],
+ "candidate_count": len(job["payload"]["candidates"]),
+ "candidates": job["review_candidates"],
+ }
+ for job in scanned
+ ],
+ "results": results,
+ }
+ args.output.parent.mkdir(parents=True, exist_ok=True)
+ args.output.write_text(
+ json.dumps(report, ensure_ascii=False, indent=2, sort_keys=True) + "\n",
+ encoding="utf-8",
+ )
+ print(json.dumps(report, ensure_ascii=False, sort_keys=True))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/legacy/prepare_tmcra_v4_e2e_data.py b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/prepare_tmcra_v4_e2e_data.py
new file mode 100644
index 0000000000000000000000000000000000000000..693287166c6adcd776734b299b60427178c929cb
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/prepare_tmcra_v4_e2e_data.py
@@ -0,0 +1,261 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+from collections import Counter
+from pathlib import Path
+from typing import Any, Mapping, Sequence
+
+
+WRITER_FIELDS = (
+ "question_id",
+ "haystack_sessions",
+ "haystack_session_ids",
+ "haystack_dates",
+)
+FORBIDDEN_WRITER_FIELDS = frozenset(
+ {
+ "question",
+ "question_date",
+ "question_type",
+ "answer",
+ "gold_answer",
+ "answer_session_ids",
+ "labels",
+ "supervision",
+ }
+)
+
+
+class PreparationError(RuntimeError):
+ pass
+
+
+def clean_text(value: Any) -> str:
+ return " ".join(str(value or "").split())
+
+
+def sha256_file(path: Path) -> str:
+ digest = hashlib.sha256()
+ with path.open("rb") as handle:
+ for chunk in iter(lambda: handle.read(1024 * 1024), b""):
+ digest.update(chunk)
+ return digest.hexdigest()
+
+
+def write_jsonl(path: Path, rows: Sequence[Mapping[str, Any]]) -> None:
+ with path.open("x", encoding="utf-8") as handle:
+ for row in rows:
+ handle.write(
+ json.dumps(dict(row), ensure_ascii=False, sort_keys=True) + "\n"
+ )
+
+
+def validate_source_row(row: Mapping[str, Any]) -> None:
+ qid = clean_text(row.get("question_id"))
+ sessions = row.get("haystack_sessions")
+ session_ids = row.get("haystack_session_ids")
+ dates = row.get("haystack_dates")
+ if not qid or not isinstance(sessions, list) or not sessions:
+ raise PreparationError(f"{qid or ''}: missing history sessions")
+ if not isinstance(session_ids, list) or not isinstance(dates, list):
+ raise PreparationError(f"{qid}: session IDs and dates must be arrays")
+ if len(sessions) != len(session_ids) or len(sessions) != len(dates):
+ raise PreparationError(f"{qid}: history/session/date lengths differ")
+ for session_index, session in enumerate(sessions):
+ if not isinstance(session, list) or not session:
+ raise PreparationError(f"{qid}: session {session_index} is empty")
+ for message_index, message in enumerate(session):
+ if not isinstance(message, Mapping):
+ raise PreparationError(
+ f"{qid}: session {session_index} message {message_index} is not an object"
+ )
+ role = clean_text(message.get("role")).lower()
+ if role not in {"user", "assistant", "system", "tool"}:
+ raise PreparationError(
+ f"{qid}: session {session_index} message {message_index} is invalid"
+ )
+
+
+def prepare(
+ *, data_path: Path, qid_path: Path, out_dir: Path
+) -> dict[str, Any]:
+ qids = [
+ clean_text(value)
+ for value in qid_path.read_text(encoding="utf-8").splitlines()
+ if clean_text(value)
+ ]
+ if not qids or len(qids) != len(set(qids)):
+ raise PreparationError("qid list must be non-empty and unique")
+ source = json.loads(data_path.read_text(encoding="utf-8"))
+ if not isinstance(source, list):
+ raise PreparationError("source dataset must be a JSON array")
+ source_rows: dict[str, Mapping[str, Any]] = {}
+ for raw_row in source:
+ if not isinstance(raw_row, Mapping):
+ raise PreparationError("source dataset contains a non-object row")
+ validate_source_row(raw_row)
+ source_qid = clean_text(raw_row.get("question_id"))
+ if source_qid in source_rows:
+ raise PreparationError(f"duplicate source question_id: {source_qid}")
+ source_rows[source_qid] = raw_row
+ missing = [qid for qid in qids if qid not in source_rows]
+ if missing:
+ raise PreparationError(f"missing qids in source data: {missing}")
+
+ out_dir.mkdir(parents=True, exist_ok=False)
+ scope_manifest: list[dict[str, Any]] = []
+ query_manifest: list[dict[str, Any]] = []
+ evaluation_refs: list[dict[str, Any]] = []
+ workers: list[dict[str, Any]] = []
+ writer_rows: list[dict[str, Any]] = []
+ total_input_messages = 0
+ total_nonempty_messages = 0
+ total_empty_messages = 0
+ total_duplicate_session_id_occurrences = 0
+ duplicate_session_id_qids: list[str] = []
+ for worker_index, qid in enumerate(qids):
+ source_row = source_rows[qid]
+ session_id_counts = Counter(
+ clean_text(value)
+ for value in source_row.get("haystack_session_ids") or []
+ if clean_text(value)
+ )
+ duplicate_session_ids = {
+ session_id: count
+ for session_id, count in sorted(session_id_counts.items())
+ if count > 1
+ }
+ duplicate_occurrences = sum(
+ count - 1 for count in duplicate_session_ids.values()
+ )
+ if duplicate_occurrences:
+ duplicate_session_id_qids.append(qid)
+ total_duplicate_session_id_occurrences += duplicate_occurrences
+ writer_row = {field: source_row.get(field) for field in WRITER_FIELDS}
+ if FORBIDDEN_WRITER_FIELDS.intersection(writer_row):
+ raise AssertionError("writer sanitizer retained a forbidden field")
+ worker_dir = out_dir / "writer" / f"worker_{worker_index:03d}"
+ worker_dir.mkdir(parents=True, exist_ok=False)
+ writer_input = worker_dir / "input.json"
+ writer_input.write_text(
+ json.dumps([writer_row], ensure_ascii=False) + "\n", encoding="utf-8"
+ )
+ writer_rows.append(writer_row)
+ database = worker_dir / "native_memory.sqlite3"
+ scope_id = f"tmcra_v4:{qid}"
+ index_path = out_dir / "indexes" / f"{qid}.pt"
+ scope_manifest.append(
+ {
+ "question_id": qid,
+ "db_path": str(database),
+ "scope_id": scope_id,
+ "index_path": str(index_path),
+ }
+ )
+ query_manifest.append(
+ {
+ "question_id": qid,
+ "question": clean_text(source_row.get("question")),
+ "question_date": clean_text(source_row.get("question_date")),
+ "question_type": clean_text(source_row.get("question_type")),
+ "db_path": str(database),
+ "scope_id": scope_id,
+ "index_path": str(index_path),
+ }
+ )
+ evaluation_refs.append(
+ {
+ "question_id": qid,
+ "answer": source_row.get("answer"),
+ "answer_session_ids": source_row.get("answer_session_ids"),
+ }
+ )
+ workers.append(
+ {
+ "worker_index": worker_index,
+ "question_id": qid,
+ "worker_dir": str(worker_dir),
+ "input": str(writer_input),
+ "input_sha256": sha256_file(writer_input),
+ "scope_id": scope_id,
+ "session_count": len(source_row.get("haystack_sessions") or []),
+ "message_count": sum(
+ len(session)
+ for session in source_row.get("haystack_sessions") or []
+ ),
+ "nonempty_message_count": sum(
+ 1
+ for session in source_row.get("haystack_sessions") or []
+ for message in session
+ if str(message.get("content") or "").strip()
+ ),
+ "empty_message_count": sum(
+ 1
+ for session in source_row.get("haystack_sessions") or []
+ for message in session
+ if not str(message.get("content") or "").strip()
+ ),
+ "duplicate_session_ids": duplicate_session_ids,
+ "duplicate_session_id_occurrence_count": duplicate_occurrences,
+ }
+ )
+ total_input_messages += workers[-1]["message_count"]
+ total_nonempty_messages += workers[-1]["nonempty_message_count"]
+ total_empty_messages += workers[-1]["empty_message_count"]
+ (out_dir / "indexes").mkdir(parents=True, exist_ok=False)
+ (out_dir / "writer_input.json").write_text(
+ json.dumps(writer_rows, ensure_ascii=False) + "\n", encoding="utf-8"
+ )
+ write_jsonl(out_dir / "scope_manifest.jsonl", scope_manifest)
+ write_jsonl(out_dir / "query_manifest.jsonl", query_manifest)
+ evaluation_dir = out_dir / "evaluation_only"
+ evaluation_dir.mkdir(parents=True, exist_ok=False)
+ write_jsonl(evaluation_dir / "references.jsonl", evaluation_refs)
+ (out_dir / "qids.txt").write_text("\n".join(qids) + "\n", encoding="utf-8")
+ manifest = {
+ "schema_version": "tmcra.v4.e2e-input.1",
+ "status": "prepared",
+ "source_data": str(data_path.resolve()),
+ "source_data_sha256": sha256_file(data_path),
+ "row_count": len(qids),
+ "input_message_count": total_input_messages,
+ "nonempty_message_count": total_nonempty_messages,
+ "empty_message_count": total_empty_messages,
+ "duplicate_session_id_occurrence_count": total_duplicate_session_id_occurrences,
+ "duplicate_session_id_qids": duplicate_session_id_qids,
+ "qids": qids,
+ "writer_fields": list(WRITER_FIELDS),
+ "forbidden_writer_fields": sorted(FORBIDDEN_WRITER_FIELDS),
+ "writer_inputs_have_query_or_evaluation_fields": False,
+ "gold_isolation_dir": str(evaluation_dir),
+ "workers": workers,
+ "combined_writer_input": str(out_dir / "writer_input.json"),
+ "scope_manifest": str(out_dir / "scope_manifest.jsonl"),
+ "query_manifest": str(out_dir / "query_manifest.jsonl"),
+ }
+ (out_dir / "input_manifest.json").write_text(
+ json.dumps(manifest, indent=2, sort_keys=True) + "\n", encoding="utf-8"
+ )
+ return manifest
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description="Prepare gold-isolated TMCRA V4 E2E inputs")
+ parser.add_argument("--data", required=True, type=Path)
+ parser.add_argument("--qid-list", required=True, type=Path)
+ parser.add_argument("--out-dir", required=True, type=Path)
+ args = parser.parse_args()
+ report = prepare(
+ data_path=args.data.resolve(),
+ qid_path=args.qid_list.resolve(),
+ out_dir=args.out_dir.resolve(),
+ )
+ print(json.dumps(report, indent=2, sort_keys=True))
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/legacy/run_tmcra_v4_build.py b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/run_tmcra_v4_build.py
new file mode 100644
index 0000000000000000000000000000000000000000..42df0b441c9f0316b8637352d1777f8f0563b378
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/run_tmcra_v4_build.py
@@ -0,0 +1,984 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sqlite3
+import subprocess
+import sys
+import time
+from contextlib import closing
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any, Mapping, Sequence
+
+from prepare_tmcra_v4_e2e_data import prepare
+from tmcra_v4_cost_report import build_report, collect_calls
+from tmcra_v4_slow_graph import (
+ PROCESS_LOSS_INTERRUPTION_ERROR,
+ SlowGraphStore,
+ load_graph_schema,
+)
+
+
+BASE = Path(__file__).resolve().parent
+PACKAGE_ROOT = BASE.parent
+PROJECT_ROOT = BASE.parents[2]
+DEFAULT_DATA = PROJECT_ROOT / "data" / "longmemeval_s_cleaned.json"
+DEFAULT_REPO = PACKAGE_ROOT / "vendor" / "tmcra_integrated"
+DEFAULT_WRITER_ENV = PROJECT_ROOT / "configs" / "writer.env"
+DEFAULT_EMBEDDING = PROJECT_ROOT / "models" / "BAAI" / "bge-m3"
+SUBJECT_ATTRIBUTION_PROMPT_VERSION = "tmcra-v4-subject-attribution-2026-07-14.3"
+
+
+class BuildError(RuntimeError):
+ pass
+
+
+def _now() -> str:
+ return datetime.now(timezone.utc).isoformat(timespec="seconds")
+
+
+def _stage(root: Path, name: str, **values: Any) -> None:
+ record = {"at": _now(), "stage": name, **values}
+ with (root / "build.log.jsonl").open("a", encoding="utf-8") as handle:
+ handle.write(json.dumps(record, sort_keys=True) + "\n")
+ print(json.dumps(record, sort_keys=True), flush=True)
+
+
+def _load_shell_environment(path: Path) -> dict[str, str]:
+ if not path.is_file():
+ raise BuildError(f"writer environment file is missing: {path}")
+ command = 'set -a; source "$1"; env -0'
+ result = subprocess.run(
+ ["bash", "-c", command, "tmcra-v4-env", str(path)],
+ check=True,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE,
+ )
+ loaded: dict[str, str] = {}
+ for entry in result.stdout.decode("utf-8").split("\0"):
+ if "=" in entry:
+ key, value = entry.split("=", 1)
+ loaded[key] = value
+ return loaded
+
+
+def _key_pool(environment: Mapping[str, str]) -> list[str]:
+ raw = environment.get("TMCRA_DEEPSEEK_WRITER_KEY_POOL", "")
+ keys = [item.strip() for item in raw.split(",") if item.strip()]
+ if not keys or len(keys) != len(set(keys)):
+ raise BuildError("DeepSeek writer key pool must be non-empty and unique")
+ configured_count = environment.get("TMCRA_DEEPSEEK_WRITER_KEY_POOL_COUNT", "")
+ if configured_count and int(configured_count) != len(keys):
+ raise BuildError("DeepSeek writer key pool count does not match actual keys")
+ return keys
+
+
+def _rotated(keys: Sequence[str], worker_index: int) -> str:
+ offset = worker_index % len(keys)
+ return ",".join([*keys[offset:], *keys[:offset]])
+
+
+def _worker_environment(base: Mapping[str, str], keys: Sequence[str], worker_index: int) -> dict[str, str]:
+ environment = dict(base)
+ pool = _rotated(keys, worker_index)
+ base_url = environment.get("TMCRA_DEEPSEEK_WRITER_BASE_URL") or environment.get("TMCRA_WRITER_BASE_URL") or "https://api.deepseek.com/v1"
+ max_tokens = environment.get("TMCRA_WRITER_MAX_TOKENS", "16384")
+ environment.update(
+ {
+ "TMCRA_WRITER_MAX_TOKENS": max_tokens,
+ "TMCRA_WRITER_BASE_URL": base_url,
+ "TMCRA_WRITER_MODEL": "deepseek-v4-flash",
+ "TMCRA_WRITER_API_KEY_POOL": pool,
+ "TMCRA_DEEPSEEK_FLASH_BASE_URL": base_url,
+ "TMCRA_DEEPSEEK_FLASH_KEY_POOL": pool,
+ "TMCRA_DEEPSEEK_FLASH_MAX_TOKENS": max_tokens,
+ "TMCRA_DEEPSEEK_FLASH_PROMPT_COST_PER_MILLION": "1",
+ "TMCRA_DEEPSEEK_FLASH_COMPLETION_COST_PER_MILLION": "2",
+ "TMCRA_DEEPSEEK_FLASH_CACHE_COST_PER_MILLION": "0.02",
+ "TMCRA_DEEPSEEK_PRO_BASE_URL": base_url,
+ "TMCRA_DEEPSEEK_PRO_KEY_POOL": pool,
+ "TMCRA_DEEPSEEK_PRO_MAX_TOKENS": max_tokens,
+ "TMCRA_DEEPSEEK_PRO_PROMPT_COST_PER_MILLION": "3",
+ "TMCRA_DEEPSEEK_PRO_COMPLETION_COST_PER_MILLION": "6",
+ "TMCRA_DEEPSEEK_PRO_CACHE_COST_PER_MILLION": "0.025",
+ }
+ )
+ return environment
+
+
+def _run(command: Sequence[str], log_path: Path, environment: Mapping[str, str]) -> None:
+ log_path.parent.mkdir(parents=True, exist_ok=True)
+ with log_path.open("x", encoding="utf-8") as log:
+ log.write(json.dumps({"command": list(command)}, sort_keys=True) + "\n")
+ log.flush()
+ subprocess.run(
+ list(command),
+ check=True,
+ stdout=log,
+ stderr=subprocess.STDOUT,
+ env=dict(environment),
+ )
+
+
+def _resume_log(directory: Path, stem: str) -> Path:
+ return directory / f"{stem}.resume.{time.time_ns()}.log"
+
+
+def _load_resume_manifest(out_dir: Path) -> dict[str, Any]:
+ manifest_path = out_dir / "input_manifest.json"
+ if not manifest_path.is_file():
+ raise BuildError("resume run has no input_manifest.json")
+ try:
+ manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError) as exc:
+ raise BuildError("resume input manifest is unreadable") from exc
+ if not isinstance(manifest, Mapping):
+ raise BuildError("resume input manifest must be an object")
+ workers = manifest.get("workers")
+ if (
+ manifest.get("status") != "prepared"
+ or not isinstance(workers, list)
+ or not workers
+ or int(manifest.get("row_count", 0) or 0) != len(workers)
+ ):
+ raise BuildError("resume input manifest is incomplete or stale")
+ return dict(manifest)
+
+
+def _verify_resume_writer(worker: Mapping[str, Any]) -> None:
+ worker_dir = Path(str(worker["worker_dir"])).resolve()
+ database = worker_dir / "native_memory.sqlite3"
+ report_path = worker_dir / "product_writer_report.json"
+ audit_path = worker_dir / "writer_chain_audit.json"
+ if not all(path.is_file() for path in (database, report_path, audit_path)):
+ raise BuildError(f"resume writer artifacts are incomplete: {worker_dir}")
+ report = json.loads(report_path.read_text(encoding="utf-8"))
+ audit = json.loads(audit_path.read_text(encoding="utf-8"))
+ if report.get("completed") is not True or audit.get("passed") is not True:
+ raise BuildError(f"resume writer audit is not complete: {worker_dir}")
+ with closing(sqlite3.connect(database)) as con:
+ statuses = dict(
+ con.execute("SELECT status,count(*) FROM v4_batch_journal GROUP BY status")
+ )
+ if statuses != {"committed": int(report.get("batches", -1))}:
+ raise BuildError(
+ f"resume writer journal is not fully committed: {worker_dir}: {statuses}"
+ )
+
+
+def _interrupted_writer_calls(worker: Mapping[str, Any]) -> list[dict[str, str]]:
+ worker_dir = Path(str(worker["worker_dir"])).resolve()
+ database = worker_dir / "native_memory.sqlite3"
+ if not database.is_file():
+ return []
+ output: list[dict[str, str]] = []
+ with closing(sqlite3.connect(database)) as con:
+ con.row_factory = sqlite3.Row
+ for row in con.execute(
+ "SELECT batch_id FROM v4_batch_journal WHERE status='api_started' ORDER BY batch_index"
+ ):
+ output.append(
+ {
+ "worker": str(worker["question_id"]),
+ "stage": "batch_flash",
+ "call_key": f"flash:{row['batch_id']}",
+ }
+ )
+ if con.execute(
+ "SELECT 1 FROM sqlite_master WHERE type='table' AND name='v4_reconciliation_jobs'"
+ ).fetchone():
+ for row in con.execute(
+ "SELECT job_id FROM v4_reconciliation_jobs WHERE status='pro_started' ORDER BY created_at,job_id"
+ ):
+ output.append(
+ {
+ "worker": str(worker["question_id"]),
+ "stage": "reconciliation_pro",
+ "call_key": f"pro:{row['job_id']}",
+ }
+ )
+ return output
+
+
+def _interrupted_slow_calls(worker: Mapping[str, Any]) -> list[dict[str, Any]]:
+ worker_dir = Path(str(worker["worker_dir"])).resolve()
+ database = worker_dir / "native_memory.sqlite3"
+ if not database.is_file():
+ return []
+ output: list[dict[str, Any]] = []
+ with closing(sqlite3.connect(database)) as con:
+ con.row_factory = sqlite3.Row
+ tables = {
+ str(row[0])
+ for row in con.execute(
+ "SELECT name FROM sqlite_master WHERE type='table'"
+ )
+ }
+ if not {"slow_graph_jobs", "slow_graph_attempts"} <= tables:
+ return []
+ recovered_attempts: set[str] = set()
+ if "slow_graph_process_loss_recoveries" in tables:
+ recovered_attempts = {
+ str(row[0])
+ for row in con.execute(
+ "SELECT attempt_id FROM slow_graph_process_loss_recoveries"
+ )
+ }
+ rows = con.execute(
+ "SELECT j.job_id,j.scope_id,j.region_key,j.status,j.claim_owner,"
+ "j.lease_expires_at,a.attempt_id,a.status AS attempt_status,"
+ "a.created_at FROM slow_graph_jobs j JOIN slow_graph_attempts a "
+ "ON a.job_id=j.job_id WHERE ("
+ "(j.status='pending' AND j.claim_token IS NOT NULL "
+ "AND a.status='started' AND a.claim_token=j.claim_token "
+ "AND a.claim_owner=j.claim_owner) OR "
+ "(j.status='failed' AND j.claim_token IS NULL AND j.last_error=? "
+ "AND a.status='expired' AND a.error=?)) "
+ "ORDER BY a.created_at,a.attempt_id",
+ (PROCESS_LOSS_INTERRUPTION_ERROR, PROCESS_LOSS_INTERRUPTION_ERROR),
+ ).fetchall()
+ for row in rows:
+ if str(row["attempt_id"]) in recovered_attempts:
+ continue
+ output.append(
+ {
+ "worker": str(worker.get("question_id") or worker_dir.name),
+ **dict(row),
+ }
+ )
+ return output
+
+
+def _recover_interrupted_slow_calls(
+ worker: Mapping[str, Any], *, repo: Path
+) -> list[dict[str, Any]]:
+ worker_dir = Path(str(worker["worker_dir"])).resolve()
+ database = worker_dir / "native_memory.sqlite3"
+ store = SlowGraphStore(database, schema=load_graph_schema(repo))
+ reviewed = store.interrupted_process_loss_attempts()
+ reports = [
+ store.recover_interrupted_process_loss(
+ str(item["job_id"]),
+ expected_attempt_id=str(item["attempt_id"]),
+ )
+ for item in reviewed
+ ]
+ if reports:
+ report_path = _resume_log(worker_dir, "slow_process_loss_recovery").with_suffix(
+ ".json"
+ )
+ report_path.write_text(
+ json.dumps(
+ {
+ "schema_version": "tmcra.v4.slow-process-loss-worker-report.1",
+ "worker": str(worker.get("question_id") or worker_dir.name),
+ "physical_api_calls_during_recovery": 0,
+ "recoveries": reports,
+ },
+ indent=2,
+ sort_keys=True,
+ )
+ + "\n",
+ encoding="utf-8",
+ )
+ return reports
+
+
+def _failed_slow_jobs(database: Path) -> list[dict[str, str]]:
+ with closing(sqlite3.connect(database)) as con:
+ con.row_factory = sqlite3.Row
+ if con.execute(
+ "SELECT 1 FROM sqlite_master WHERE type='table' AND name='slow_graph_jobs'"
+ ).fetchone() is None:
+ return []
+ return [
+ {
+ "job_id": str(row["job_id"]),
+ "region_key": str(row["region_key"]),
+ "last_error": str(row["last_error"]),
+ }
+ for row in con.execute(
+ "SELECT job_id,region_key,last_error FROM slow_graph_jobs "
+ "WHERE status IN ('failed','retryable') ORDER BY created_at,job_id"
+ )
+ ]
+
+
+def _slow_worker_resume(
+ worker: Mapping[str, Any],
+ *,
+ repo: Path,
+ environment: Mapping[str, str],
+ enqueue: bool = True,
+ recover_interrupted_slow_calls: bool = False,
+) -> None:
+ worker_dir = Path(str(worker["worker_dir"])).resolve()
+ database = worker_dir / "native_memory.sqlite3"
+ scope_id = str(worker["scope_id"])
+ prefix = [
+ sys.executable,
+ str(BASE / "tmcra_v4_slow_graph.py"),
+ str(database),
+ "--repo",
+ str(repo),
+ ]
+ interrupted = _interrupted_slow_calls(worker)
+ if interrupted and not recover_interrupted_slow_calls:
+ raise BuildError(
+ "resume has Slow calls without durable responses; explicit process-loss "
+ "recovery flag is required: " + json.dumps(interrupted, sort_keys=True)
+ )
+ if interrupted:
+ recovered = _recover_interrupted_slow_calls(worker, repo=repo)
+ if len(recovered) != len(interrupted):
+ raise BuildError(
+ "Slow process-loss recovery count drifted after preflight"
+ )
+ if enqueue:
+ _run(
+ [*prefix, "enqueue", scope_id],
+ _resume_log(worker_dir, "slow_enqueue"),
+ environment,
+ )
+ failed = _failed_slow_jobs(database)
+ if failed:
+ raise BuildError(
+ "resume requires explicit revalidation of failed slow jobs: "
+ + json.dumps(failed, sort_keys=True)
+ )
+ _run([*prefix, "drain"], _resume_log(worker_dir, "slow_drain"), environment)
+ _run(
+ [*prefix, "audit", scope_id, "--require-promotion-coverage"],
+ _resume_log(worker_dir, "slow_audit"),
+ environment,
+ )
+
+
+def _subject_attribution_stage(
+ out_dir: Path, environment: Mapping[str, str]
+) -> dict[str, Any]:
+ report_path = out_dir / "subject_attribution_report.json"
+ if not report_path.exists():
+ _run(
+ [
+ sys.executable,
+ str(BASE / "ops" / "audit_tmcra_v4_subject_attribution.py"),
+ "--run-dir",
+ str(out_dir),
+ "--apply",
+ "--output",
+ str(report_path),
+ ],
+ out_dir / "subject_attribution.log",
+ environment,
+ )
+ try:
+ report = json.loads(report_path.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError) as exc:
+ raise BuildError("subject-attribution report is unreadable") from exc
+ if (
+ not isinstance(report, Mapping)
+ or report.get("status") != "complete"
+ or report.get("mode") != "apply"
+ or report.get("prompt_version") != SUBJECT_ATTRIBUTION_PROMPT_VERSION
+ or report.get("model") != "deepseek-v4-pro"
+ ):
+ raise BuildError("subject-attribution stage is incomplete or drifted")
+ _stage(
+ out_dir,
+ "subject_attribution_complete",
+ routed_messages=int(report.get("routed_message_count", 0) or 0),
+ quarantined=int(report.get("quarantined_count", 0) or 0),
+ physical_api_calls=int(report.get("physical_api_calls", 0) or 0),
+ )
+ return dict(report)
+
+
+def _slow_process_loss_cost_uncertainty(
+ databases: Sequence[Path],
+) -> dict[str, int]:
+ recoveries = 0
+ potential_min = 0
+ potential_max = 0
+ for database in databases:
+ with closing(sqlite3.connect(database)) as con:
+ if con.execute(
+ "SELECT 1 FROM sqlite_master WHERE type='table' "
+ "AND name='slow_graph_process_loss_recoveries'"
+ ).fetchone() is None:
+ continue
+ row = con.execute(
+ "SELECT count(*),"
+ "coalesce(sum(potential_duplicate_physical_calls_min),0),"
+ "coalesce(sum(potential_duplicate_physical_calls_max),0) "
+ "FROM slow_graph_process_loss_recoveries"
+ ).fetchone()
+ recoveries += int(row[0])
+ potential_min += int(row[1])
+ potential_max += int(row[2])
+ return {
+ "unknown_external_call_outcomes": recoveries,
+ "potential_duplicate_physical_calls_min": potential_min,
+ "potential_duplicate_physical_calls_max": potential_max,
+ }
+
+
+def _finalize_build(
+ *,
+ out_dir: Path,
+ workers: Sequence[Mapping[str, Any]],
+ writer_concurrency: int,
+ slow_concurrency: int,
+ recovered: bool,
+) -> dict[str, Any]:
+ databases = [
+ Path(str(worker["worker_dir"])) / "native_memory.sqlite3"
+ for worker in workers
+ ]
+ interrupted_call_logs = [
+ path
+ for worker in workers
+ if (
+ path := Path(str(worker["worker_dir"]))
+ / "product_writer_interrupted_calls.jsonl"
+ ).is_file()
+ ]
+ cost = build_report(collect_calls(interrupted_call_logs, databases))
+ process_loss = _slow_process_loss_cost_uncertainty(databases)
+ cost["slow_process_loss_uncertainty"] = process_loss
+ cost["cost_is_fully_observed"] = (
+ process_loss["unknown_external_call_outcomes"] == 0
+ )
+ (out_dir / "build_cost_report.json").write_text(
+ json.dumps(cost, indent=2, sort_keys=True) + "\n", encoding="utf-8"
+ )
+ quality = _writer_quality_report(workers)
+ (out_dir / "writer_quality_report.json").write_text(
+ json.dumps(quality, indent=2, sort_keys=True) + "\n", encoding="utf-8"
+ )
+ report = {
+ "schema_version": "tmcra.v4.build.1",
+ "status": "complete",
+ "row_count": len(workers),
+ "writer_concurrency": writer_concurrency,
+ "slow_concurrency": slow_concurrency,
+ "physical_api_call_count": cost["physical_call_count"],
+ "exact_cost_cny": cost["exact_cost_cny"],
+ "min_cost_cny": cost["min_cost_cny"],
+ "max_cost_cny": cost["max_cost_cny"],
+ "cost_is_fully_observed": cost["cost_is_fully_observed"],
+ "slow_process_loss_unknown_external_outcomes": process_loss[
+ "unknown_external_call_outcomes"
+ ],
+ "slow_process_loss_potential_duplicate_physical_calls_min": (
+ process_loss["potential_duplicate_physical_calls_min"]
+ ),
+ "slow_process_loss_potential_duplicate_physical_calls_max": (
+ process_loss["potential_duplicate_physical_calls_max"]
+ ),
+ "interrupted_calls_without_usage": sum(
+ int(item.get("calls_without_usage") or 0)
+ for item in cost["by_stage_model"]
+ if str(item.get("stage") or "").endswith("_interrupted")
+ ),
+ "resumed": recovered,
+ "writer_quality_requires_review": quality["requires_review"],
+ "writer_quality_warning_count": quality["warning_count"],
+ "writer_quality_dropped_count": quality["dropped_count"],
+ }
+ (out_dir / "build_report.json").write_text(
+ json.dumps(report, indent=2, sort_keys=True) + "\n", encoding="utf-8"
+ )
+ failed = out_dir / "FAILED"
+ if failed.is_file():
+ history = {
+ "resolved_at": _now(),
+ "resolved_by_resume": recovered,
+ "failure": json.loads(failed.read_text(encoding="utf-8")),
+ }
+ with (out_dir / "build_failure_history.jsonl").open(
+ "a", encoding="utf-8"
+ ) as handle:
+ handle.write(json.dumps(history, sort_keys=True) + "\n")
+ failed.unlink()
+ (out_dir / "BUILD_COMPLETE").write_text(_now() + "\n", encoding="utf-8")
+ _stage(out_dir, "complete", resumed=recovered)
+ return report
+
+
+def _writer_quality_report(
+ workers: Sequence[Mapping[str, Any]],
+) -> dict[str, Any]:
+ warning_counts: dict[str, int] = {}
+ dropped_counts: dict[str, int] = {}
+ message_count = 0
+ messages_with_warnings = 0
+ semantic_proposals = 0
+ semantic_committed = 0
+ for worker in workers:
+ path = Path(str(worker["worker_dir"])) / "product_write_messages.jsonl"
+ if not path.is_file():
+ continue
+ for raw_line in path.read_text(encoding="utf-8").splitlines():
+ if not raw_line.strip():
+ continue
+ row = json.loads(raw_line)
+ message_count += 1
+ semantic_proposals += int(row.get("semantic_proposals") or 0)
+ semantic_committed += int(row.get("semantic_committed") or 0)
+ warnings = list(row.get("validation_warnings") or [])
+ if warnings:
+ messages_with_warnings += 1
+ for warning in warnings:
+ code = str(warning.get("code") or "unknown_warning")
+ warning_counts[code] = warning_counts.get(code, 0) + 1
+ dropped = int(warning.get("dropped_count") or 0)
+ if dropped:
+ dropped_counts[code] = dropped_counts.get(code, 0) + dropped
+ review_codes = {
+ "assistant_assertions_dropped",
+ "invalid_assertion_quarantined",
+ "invalid_interaction_quarantined",
+ "invalid_item_collection_defaulted_empty",
+ "durability_defaulted_uncertain",
+ "temporal_durability_defaulted_uncertain",
+ "conflicting_durability_defaulted_uncertain",
+ }
+ review_events = sum(warning_counts.get(code, 0) for code in review_codes)
+ return {
+ "schema_version": "tmcra.v4.writer-quality.1",
+ "message_count": message_count,
+ "messages_with_warnings": messages_with_warnings,
+ "warning_count": sum(warning_counts.values()),
+ "warnings_by_code": dict(sorted(warning_counts.items())),
+ "dropped_count": sum(dropped_counts.values()),
+ "dropped_by_code": dict(sorted(dropped_counts.items())),
+ "review_event_count": review_events,
+ "review_codes": sorted(review_codes),
+ "requires_review": review_events > 0,
+ "semantic_proposals": semantic_proposals,
+ "semantic_committed": semantic_committed,
+ }
+
+
+def _record_failure(out_dir: Path, exc: Exception) -> None:
+ failed = out_dir / "FAILED"
+ if failed.is_file():
+ with (out_dir / "build_failure_history.jsonl").open(
+ "a", encoding="utf-8"
+ ) as handle:
+ handle.write(
+ json.dumps(
+ {
+ "superseded_at": _now(),
+ "resolved": False,
+ "failure": json.loads(failed.read_text(encoding="utf-8")),
+ },
+ sort_keys=True,
+ )
+ + "\n"
+ )
+ failed.write_text(
+ json.dumps({"at": _now(), "error": f"{exc.__class__.__name__}: {exc}"})
+ + "\n",
+ encoding="utf-8",
+ )
+
+
+def _writer_worker(worker: Mapping[str, Any], *, repo: Path, environment: Mapping[str, str]) -> None:
+ worker_dir = Path(str(worker["worker_dir"]))
+ _run(
+ [
+ sys.executable,
+ str(BASE / "tmcra_v4_batch_writer.py"),
+ "--input",
+ str(worker["input"]),
+ "--out-dir",
+ str(worker_dir),
+ "--repo",
+ str(repo),
+ ],
+ worker_dir / "writer.log",
+ environment,
+ )
+ _run(
+ [
+ sys.executable,
+ str(BASE / "audit_tmcra_v4_chain.py"),
+ "--run-dir",
+ str(worker_dir),
+ "--output",
+ str(worker_dir / "writer_chain_audit.json"),
+ "--worker-db",
+ f"worker={worker_dir / 'native_memory.sqlite3'}",
+ ],
+ worker_dir / "writer_audit.log",
+ environment,
+ )
+
+
+def _writer_worker_resume(
+ worker: Mapping[str, Any],
+ *,
+ repo: Path,
+ environment: Mapping[str, str],
+ recover_interrupted_api_calls: bool,
+) -> None:
+ worker_dir = Path(str(worker["worker_dir"])).resolve()
+ command = [
+ sys.executable,
+ str(BASE / "tmcra_v4_batch_writer.py"),
+ "--input",
+ str(worker["input"]),
+ "--out-dir",
+ str(worker_dir),
+ "--repo",
+ str(repo),
+ "--revalidate-failed-raw-response",
+ ]
+ if recover_interrupted_api_calls:
+ command.append("--recover-interrupted-api-calls")
+ _run(
+ command,
+ _resume_log(worker_dir, "writer"),
+ environment,
+ )
+ _run(
+ [
+ sys.executable,
+ str(BASE / "audit_tmcra_v4_chain.py"),
+ "--run-dir",
+ str(worker_dir),
+ "--output",
+ str(worker_dir / "writer_chain_audit.json"),
+ "--worker-db",
+ f"worker={worker_dir / 'native_memory.sqlite3'}",
+ ],
+ _resume_log(worker_dir, "writer_audit"),
+ environment,
+ )
+
+
+def _slow_worker(worker: Mapping[str, Any], *, repo: Path, environment: Mapping[str, str]) -> None:
+ worker_dir = Path(str(worker["worker_dir"]))
+ database = worker_dir / "native_memory.sqlite3"
+ scope_id = str(worker["scope_id"])
+ prefix = [sys.executable, str(BASE / "tmcra_v4_slow_graph.py"), str(database), "--repo", str(repo)]
+ _run([*prefix, "enqueue", scope_id], worker_dir / "slow_enqueue.log", environment)
+ _run([*prefix, "drain"], worker_dir / "slow_drain.log", environment)
+ _run(
+ [*prefix, "audit", scope_id, "--require-promotion-coverage"],
+ worker_dir / "slow_audit.log",
+ environment,
+ )
+
+
+def _parallel(
+ workers: Sequence[Mapping[str, Any]],
+ concurrency: int,
+ task: Any,
+ environments: Sequence[Mapping[str, str]],
+) -> None:
+ if concurrency <= 0:
+ raise BuildError("concurrency must be positive")
+ with ThreadPoolExecutor(max_workers=min(concurrency, len(workers))) as executor:
+ futures = {
+ executor.submit(task, worker, environment=environments[index]): str(worker["question_id"])
+ for index, worker in enumerate(workers)
+ }
+ for future in as_completed(futures):
+ future.result()
+
+
+def resume_build(args: argparse.Namespace) -> dict[str, Any]:
+ out_dir = args.out_dir.resolve()
+ if not out_dir.is_dir():
+ raise BuildError(f"resume output directory does not exist: {out_dir}")
+ if (out_dir / "BUILD_COMPLETE").exists():
+ raise BuildError("resume output is already complete")
+ manifest = _load_resume_manifest(out_dir)
+ expected_qids = [
+ line.strip()
+ for line in args.qid_list.resolve().read_text(encoding="utf-8").splitlines()
+ if line.strip()
+ ]
+ if expected_qids != list(manifest.get("qids") or []):
+ raise BuildError("resume qid list does not match the frozen input manifest")
+ shell_environment = _load_shell_environment(args.writer_env.resolve())
+ base_environment = {**os.environ, **shell_environment}
+ keys = _key_pool(base_environment)
+ workers = list(manifest["workers"])
+ environments = [
+ _worker_environment(base_environment, keys, index)
+ for index in range(len(workers))
+ ]
+ try:
+ incomplete_workers: list[Mapping[str, Any]] = []
+ for worker in workers:
+ try:
+ _verify_resume_writer(worker)
+ except BuildError:
+ incomplete_workers.append(worker)
+ if incomplete_workers:
+ if not getattr(args, "revalidate_failed_writer_raw_response", False):
+ raise BuildError(
+ "resume has incomplete writer workers; explicit raw-response "
+ "revalidation flag is required: "
+ + json.dumps(
+ [str(worker["question_id"]) for worker in incomplete_workers]
+ )
+ )
+ interrupted = [
+ call
+ for worker in incomplete_workers
+ for call in _interrupted_writer_calls(worker)
+ ]
+ if interrupted and not getattr(
+ args, "recover_interrupted_api_calls", False
+ ):
+ raise BuildError(
+ "resume has started calls without durable responses; explicit "
+ "process-loss recovery flag is required: "
+ + json.dumps(interrupted, sort_keys=True)
+ )
+ _stage(
+ out_dir,
+ "writer_resume_started",
+ workers=len(incomplete_workers),
+ interrupted_calls=len(interrupted),
+ )
+ selected_environments = [
+ environments[workers.index(worker)] for worker in incomplete_workers
+ ]
+ _parallel(
+ incomplete_workers,
+ args.writer_concurrency,
+ lambda worker, environment: _writer_worker_resume(
+ worker,
+ repo=args.repo.resolve(),
+ environment=environment,
+ recover_interrupted_api_calls=bool(interrupted),
+ ),
+ selected_environments,
+ )
+ for worker in workers:
+ _verify_resume_writer(worker)
+ _stage(out_dir, "writer_resume_complete")
+ _stage(out_dir, "subject_attribution_started", resumed=True)
+ _subject_attribution_stage(out_dir, base_environment)
+ interrupted_slow = [
+ call for worker in workers for call in _interrupted_slow_calls(worker)
+ ]
+ if interrupted_slow and not getattr(
+ args, "recover_interrupted_slow_calls", False
+ ):
+ raise BuildError(
+ "resume has Slow calls without durable responses; explicit "
+ "--recover-interrupted-slow-calls is required: "
+ + json.dumps(interrupted_slow, sort_keys=True)
+ )
+ _stage(
+ out_dir,
+ "resume_started",
+ row_count=len(workers),
+ interrupted_slow_calls=len(interrupted_slow),
+ )
+ _parallel(
+ workers,
+ args.slow_concurrency,
+ lambda worker, environment: _slow_worker_resume(
+ worker,
+ repo=args.repo.resolve(),
+ environment=environment,
+ recover_interrupted_slow_calls=bool(interrupted_slow),
+ ),
+ environments,
+ )
+ _stage(out_dir, "slow_graph_complete", resumed=True)
+ runtime_environment = dict(base_environment)
+ runtime_environment["TMCRA_NODE_MODEL_DEVICE"] = args.device
+ _run(
+ [
+ sys.executable,
+ str(BASE / "tmcra_v4_online_runtime.py"),
+ "build-index",
+ "--scope-manifest",
+ str(out_dir / "scope_manifest.jsonl"),
+ "--out-report",
+ str(out_dir / "index_report.json"),
+ "--embedding-model",
+ str(args.embedding_model.resolve()),
+ "--device",
+ args.device,
+ "--batch-size",
+ str(args.index_batch_size),
+ ],
+ _resume_log(out_dir, "index"),
+ runtime_environment,
+ )
+ _stage(out_dir, "index_complete", resumed=True)
+ _run(
+ [
+ sys.executable,
+ str(BASE / "audit_tmcra_v4_chain.py"),
+ "--run-dir",
+ str(out_dir),
+ "--output",
+ str(out_dir / "build_chain_audit.json"),
+ "--build-only",
+ ],
+ _resume_log(out_dir, "build_chain_audit"),
+ runtime_environment,
+ )
+ _stage(out_dir, "build_audit_complete", resumed=True)
+ return _finalize_build(
+ out_dir=out_dir,
+ workers=workers,
+ writer_concurrency=args.writer_concurrency,
+ slow_concurrency=args.slow_concurrency,
+ recovered=True,
+ )
+ except Exception as exc:
+ _record_failure(out_dir, exc)
+ raise
+
+
+def build(args: argparse.Namespace) -> dict[str, Any]:
+ if getattr(args, "resume", False):
+ return resume_build(args)
+ out_dir = args.out_dir.resolve()
+ if out_dir.exists():
+ raise BuildError(f"output directory already exists: {out_dir}")
+ manifest = prepare(
+ data_path=args.data.resolve(),
+ qid_path=args.qid_list.resolve(),
+ out_dir=out_dir,
+ )
+ _stage(out_dir, "prepared", rows=manifest["row_count"])
+ shell_environment = _load_shell_environment(args.writer_env.resolve())
+ base_environment = {**os.environ, **shell_environment}
+ keys = _key_pool(base_environment)
+ workers = list(manifest["workers"])
+ environments = [
+ _worker_environment(base_environment, keys, index) for index in range(len(workers))
+ ]
+ try:
+ _stage(out_dir, "writer_started", concurrency=args.writer_concurrency)
+ _parallel(
+ workers,
+ args.writer_concurrency,
+ lambda worker, environment: _writer_worker(
+ worker, repo=args.repo.resolve(), environment=environment
+ ),
+ environments,
+ )
+ _stage(out_dir, "writer_complete")
+ _stage(out_dir, "subject_attribution_started")
+ _subject_attribution_stage(out_dir, base_environment)
+ _stage(out_dir, "slow_graph_started", concurrency=args.slow_concurrency)
+ _parallel(
+ workers,
+ args.slow_concurrency,
+ lambda worker, environment: _slow_worker(
+ worker, repo=args.repo.resolve(), environment=environment
+ ),
+ environments,
+ )
+ _stage(out_dir, "slow_graph_complete")
+ runtime_environment = dict(base_environment)
+ runtime_environment["TMCRA_NODE_MODEL_DEVICE"] = args.device
+ _run(
+ [
+ sys.executable,
+ str(BASE / "tmcra_v4_online_runtime.py"),
+ "build-index",
+ "--scope-manifest",
+ str(out_dir / "scope_manifest.jsonl"),
+ "--out-report",
+ str(out_dir / "index_report.json"),
+ "--embedding-model",
+ str(args.embedding_model.resolve()),
+ "--device",
+ args.device,
+ "--batch-size",
+ str(args.index_batch_size),
+ ],
+ out_dir / "index.log",
+ runtime_environment,
+ )
+ _stage(out_dir, "index_complete")
+ _run(
+ [
+ sys.executable,
+ str(BASE / "audit_tmcra_v4_chain.py"),
+ "--run-dir",
+ str(out_dir),
+ "--output",
+ str(out_dir / "build_chain_audit.json"),
+ "--build-only",
+ ],
+ out_dir / "build_chain_audit.log",
+ runtime_environment,
+ )
+ return _finalize_build(
+ out_dir=out_dir,
+ workers=workers,
+ writer_concurrency=args.writer_concurrency,
+ slow_concurrency=args.slow_concurrency,
+ recovered=False,
+ )
+ except Exception as exc:
+ _record_failure(out_dir, exc)
+ raise
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description="Build a frozen TMCRA V4 memory corpus")
+ parser.add_argument("--out-dir", type=Path, required=True)
+ parser.add_argument("--qid-list", type=Path, required=True)
+ parser.add_argument("--data", type=Path, default=DEFAULT_DATA)
+ parser.add_argument("--repo", type=Path, default=DEFAULT_REPO)
+ parser.add_argument("--writer-env", type=Path, default=DEFAULT_WRITER_ENV)
+ parser.add_argument("--embedding-model", type=Path, default=DEFAULT_EMBEDDING)
+ parser.add_argument("--writer-concurrency", type=int, default=1)
+ parser.add_argument("--slow-concurrency", type=int, default=1)
+ parser.add_argument("--index-batch-size", type=int, default=16)
+ parser.add_argument("--device", default="cuda")
+ parser.add_argument(
+ "--resume",
+ action="store_true",
+ help="resume a frozen incomplete build after explicit failed-job review",
+ )
+ parser.add_argument(
+ "--revalidate-failed-writer-raw-response",
+ action="store_true",
+ help="explicitly revalidate one saved clean writer response per failed worker",
+ )
+ parser.add_argument(
+ "--recover-interrupted-api-calls",
+ action="store_true",
+ help=(
+ "after explicit process-loss review, replace started writer calls "
+ "that have no durable response using the same model"
+ ),
+ )
+ parser.add_argument(
+ "--recover-interrupted-slow-calls",
+ action="store_true",
+ help=(
+ "after explicit review, journal expired Slow attempts with unknown "
+ "external outcomes and reopen only those jobs"
+ ),
+ )
+ args = parser.parse_args()
+ report = build(args)
+ print(json.dumps(report, sort_keys=True))
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/legacy/run_tmcra_v4_compile_evidence.py b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/run_tmcra_v4_compile_evidence.py
new file mode 100644
index 0000000000000000000000000000000000000000..affeef9f6b0a94d31bd388d5fb09d62994c10f20
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/run_tmcra_v4_compile_evidence.py
@@ -0,0 +1,1015 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import os
+import re
+import threading
+import time
+from collections.abc import Mapping, Sequence
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from pathlib import Path
+from typing import Any
+
+from run_tmcra_v4_build import DEFAULT_WRITER_ENV, _key_pool, _load_shell_environment
+from tmcra_v4_evidence_operations import (
+ PACKET_COMPILER_VERSION,
+ PACKET_SCHEMA,
+ PLAN_SCHEMA,
+ build_evidence_catalog,
+ compile_evidence_packet,
+ operation_plan_structural_risks,
+ unbound_memory_requirement_ids,
+ validate_operation_plan,
+)
+from tmcra_v4_evidence_planner import (
+ DeepSeekEvidenceOperationPlanner,
+ EvidencePlannerError,
+ PROMPT_VERSION as EVIDENCE_PLANNER_PROMPT_VERSION,
+ normalize_planner_output,
+)
+from tmcra_v4_route_policy import (
+ RoutePolicyError,
+ validate_diagnostic_retrieval_rows,
+ validate_production_retrieval_rows,
+)
+
+
+class EvidenceCompileError(RuntimeError):
+ pass
+
+
+ARTIFACT_BINDING_SCHEMA = "tmcra.v4.evidence-compiler-artifact-binding.1"
+REVIEW_POLICY_VERSION = "tmcra.v4.evidence-review-policy.4"
+
+
+def _text(value: Any) -> str:
+ return value.strip() if isinstance(value, str) else ""
+
+
+def _read_jsonl(path: Path) -> list[dict[str, Any]]:
+ return [json.loads(line) for line in path.read_text(encoding="utf-8").splitlines() if line.strip()]
+
+
+def _retrieval_debug_report(
+ path: Path,
+ expected_qids: Sequence[str],
+) -> dict[str, Any]:
+ if not path.is_file():
+ raise EvidenceCompileError(f"retrieval debug is missing: {path}")
+ raw = path.read_bytes()
+ try:
+ rows = [
+ json.loads(line)
+ for line in raw.decode("utf-8").splitlines()
+ if line.strip()
+ ]
+ except (UnicodeDecodeError, json.JSONDecodeError) as exc:
+ raise EvidenceCompileError(f"retrieval debug is invalid JSONL: {path}") from exc
+ actual_qids = [
+ _text(row.get("question_id") or row.get("qid"))
+ if isinstance(row, Mapping)
+ else ""
+ for row in rows
+ ]
+ if actual_qids != list(expected_qids):
+ raise EvidenceCompileError(
+ "retrieval debug rows do not match frozen evidence qid order"
+ )
+ return {
+ "schema_version": "tmcra.v4.compiled-retrieval-debug-binding.1",
+ "source_path": str(path),
+ "row_count": len(rows),
+ "source_sha256": hashlib.sha256(raw).hexdigest(),
+ }
+
+
+def _stage_retrieval_debug(
+ source: Path,
+ out_dir: Path,
+ expected_qids: Sequence[str],
+ *,
+ expected_sha256: str,
+) -> dict[str, Any]:
+ report = _retrieval_debug_report(source, expected_qids)
+ if report["source_sha256"] != expected_sha256:
+ raise EvidenceCompileError("retrieval debug changed during evidence compilation")
+ raw = source.read_bytes()
+ destination = out_dir / "retrieval_debug.jsonl"
+ temporary = destination.with_name(
+ f".{destination.name}.tmp.{os.getpid()}.{time.time_ns()}"
+ )
+ temporary.write_bytes(raw)
+ os.replace(temporary, destination)
+ artifact_sha256 = hashlib.sha256(destination.read_bytes()).hexdigest()
+ if artifact_sha256 != expected_sha256:
+ raise EvidenceCompileError("staged retrieval debug failed its integrity check")
+ return {
+ **report,
+ "artifact_path": destination.name,
+ "artifact_sha256": artifact_sha256,
+ "status": "staged",
+ }
+
+
+def _atomic_json(path: Path, value: Mapping[str, Any]) -> None:
+ temporary = path.with_name(f".{path.name}.tmp.{os.getpid()}.{time.time_ns()}")
+ temporary.write_text(json.dumps(dict(value), ensure_ascii=False, indent=2, sort_keys=True) + "\n", encoding="utf-8")
+ os.replace(temporary, path)
+
+
+def _atomic_jsonl(path: Path, rows: Sequence[Mapping[str, Any]]) -> None:
+ temporary = path.with_name(f".{path.name}.tmp.{os.getpid()}.{time.time_ns()}")
+ temporary.write_text("".join(json.dumps(dict(row), ensure_ascii=False, sort_keys=True) + "\n" for row in rows), encoding="utf-8")
+ os.replace(temporary, path)
+
+
+def _append_jsonl_atomic(path: Path, row: Mapping[str, Any]) -> None:
+ rows = _read_jsonl(path) if path.is_file() else []
+ rows.append(dict(row))
+ _atomic_jsonl(path, rows)
+
+
+def _identity(row: Mapping[str, Any]) -> str:
+ payload = {
+ "question_id": row.get("question_id"),
+ "question": row.get("question"),
+ "question_date": row.get("question_date"),
+ "evidence_windows": row.get("evidence_windows"),
+ # Retrieval is part of the compiler input. In particular, a diagnostic
+ # row must never collide with a production row for the same question.
+ "retrieval_contract": row.get("retrieval_contract"),
+ }
+ return hashlib.sha256(json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":")).encode("utf-8")).hexdigest()
+
+
+def _planner_contract(
+ *,
+ provider: str,
+ model: str,
+ review_policy_version: str = REVIEW_POLICY_VERSION,
+) -> dict[str, Any]:
+ return {
+ "provider": provider,
+ "model": model,
+ "prompt_version": EVIDENCE_PLANNER_PROMPT_VERSION,
+ "review_policy_version": review_policy_version,
+ "plan_schema": PLAN_SCHEMA,
+ }
+
+
+def _packet_compiler_contract(*, compiler_version: str = PACKET_COMPILER_VERSION) -> dict[str, Any]:
+ return {
+ "packet_schema": PACKET_SCHEMA,
+ "compiler_version": compiler_version,
+ }
+
+
+def _artifact_binding(
+ row: Mapping[str, Any],
+ *,
+ provider: str,
+ model: str,
+ compiler_version: str = PACKET_COMPILER_VERSION,
+ review_policy_version: str = REVIEW_POLICY_VERSION,
+) -> dict[str, Any]:
+ return {
+ "schema_version": ARTIFACT_BINDING_SCHEMA,
+ "input_sha256": _identity(row),
+ "input": {
+ "question_id": row.get("question_id"),
+ "question": row.get("question"),
+ "question_date": row.get("question_date"),
+ "evidence_windows": row.get("evidence_windows"),
+ "retrieval_route_contract": row.get("retrieval_contract"),
+ },
+ "planner": _planner_contract(
+ provider=provider,
+ model=model,
+ review_policy_version=review_policy_version,
+ ),
+ "packet_compiler": _packet_compiler_contract(compiler_version=compiler_version),
+ }
+
+
+def _binding_sha256(binding: Mapping[str, Any]) -> str:
+ return hashlib.sha256(
+ json.dumps(dict(binding), ensure_ascii=False, sort_keys=True, separators=(",", ":")).encode("utf-8")
+ ).hexdigest()
+
+
+def _saved_binding(
+ saved: Mapping[str, Any],
+ saved_row: Mapping[str, Any],
+) -> dict[str, Any]:
+ binding = saved.get("artifact_binding")
+ if isinstance(binding, Mapping):
+ return dict(binding)
+ planner = saved.get("planner")
+ if not isinstance(planner, Mapping):
+ raise EvidenceCompileError("completed artifact lacks its planner contract")
+ provider = _text(planner.get("provider"))
+ model = _text(planner.get("model"))
+ if not provider or not model:
+ raise EvidenceCompileError("completed artifact lacks its planner provider/model binding")
+ packet = saved_row.get("compiled_evidence_packet")
+ persisted_plan_schema = (
+ packet.get("operation_plan", {}).get("schema_version")
+ if isinstance(packet, Mapping)
+ and isinstance(packet.get("operation_plan"), Mapping)
+ else None
+ )
+ if persisted_plan_schema != PLAN_SCHEMA:
+ raise EvidenceCompileError("completed artifact planner contract schema is stale")
+ # Legacy artifacts did not persist this envelope. Reconstructing it is
+ # safe only from the persisted row and planner metadata; both are checked
+ # against the current invocation before reuse.
+ packet_compiler_version = (
+ _text(packet.get("packet_compiler_version"))
+ if isinstance(packet, Mapping)
+ else ""
+ )
+ return _artifact_binding(
+ saved_row,
+ provider=provider,
+ model=model,
+ compiler_version=packet_compiler_version or "legacy-unknown",
+ review_policy_version=(
+ _text(planner.get("review_policy_version")) or "legacy-unknown"
+ ),
+ )
+
+
+def _assert_artifact_binding(
+ saved: Mapping[str, Any],
+ row: Mapping[str, Any],
+ *,
+ input_sha256: str,
+ expected_binding: Mapping[str, Any],
+) -> None:
+ saved_row = saved.get("row")
+ if not isinstance(saved_row, Mapping):
+ raise EvidenceCompileError("completed artifact lacks its persisted input row")
+ if _identity(saved_row) != input_sha256:
+ raise EvidenceCompileError("persisted compiler identity mismatch")
+ current_binding = dict(expected_binding)
+ persisted_binding = _saved_binding(saved, saved_row)
+ if persisted_binding.get("schema_version") not in {None, ARTIFACT_BINDING_SCHEMA}:
+ raise EvidenceCompileError("completed artifact binding schema is stale")
+ if persisted_binding.get("input_sha256") != input_sha256:
+ raise EvidenceCompileError("persisted compiler identity mismatch")
+ if persisted_binding.get("input") != current_binding.get("input"):
+ raise EvidenceCompileError("completed artifact input contract mismatch")
+ if persisted_binding.get("planner") != current_binding.get("planner"):
+ raise EvidenceCompileError("completed artifact planner contract/provider/model mismatch")
+ persisted_packet_contract = persisted_binding.get("packet_compiler")
+ current_packet_contract = current_binding.get("packet_compiler")
+ if not isinstance(persisted_packet_contract, Mapping) or not isinstance(current_packet_contract, Mapping):
+ raise EvidenceCompileError("completed artifact packet compiler contract is missing")
+ if persisted_packet_contract.get("packet_schema") != current_packet_contract.get("packet_schema"):
+ raise EvidenceCompileError("completed artifact packet schema contract mismatch")
+ if row.get("retrieval_contract") != saved_row.get("retrieval_contract"):
+ raise EvidenceCompileError("completed artifact retrieval route contract mismatch")
+
+ persisted_packet = saved_row.get("compiled_evidence_packet")
+ if isinstance(persisted_packet, Mapping) and persisted_packet.get("schema_version") != PACKET_SCHEMA:
+ raise EvidenceCompileError("completed artifact packet schema is stale")
+
+
+def _cost_cny(calls: Sequence[Mapping[str, Any]]) -> float | None:
+ if any(call.get("provider") != "deepseek" for call in calls):
+ return None
+ total = 0.0
+ for call in calls:
+ usage = dict(call.get("usage") or {})
+ total += (
+ int(usage.get("prompt_cache_miss_tokens", 0)) * 3.0
+ + int(usage.get("completion_tokens", 0)) * 6.0
+ + int(usage.get("prompt_cache_hit_tokens", 0)) * 0.025
+ ) / 1_000_000.0
+ return round(total, 8)
+
+
+def _physical_calls(values: Sequence[Mapping[str, Any]]) -> list[dict[str, Any]]:
+ calls: dict[str, dict[str, Any]] = {}
+
+ def visit(value: Any) -> None:
+ if not isinstance(value, Mapping):
+ return
+ for key in ("calls", "prior_calls"):
+ nested = value.get(key)
+ if isinstance(nested, Sequence) and not isinstance(nested, (str, bytes)):
+ for item in nested:
+ visit(item)
+ call_id = _text(value.get("physical_call_id"))
+ if value.get("physical_api_call") is True and call_id:
+ calls.setdefault(call_id, dict(value))
+
+ for value in values:
+ visit(value)
+ return list(calls.values())
+
+
+def _aggregate_planner_calls(calls: Sequence[Mapping[str, Any]]) -> dict[str, Any]:
+ if not calls:
+ raise EvidenceCompileError("planner call list is empty")
+ usage_keys = ("prompt_tokens", "completion_tokens", "prompt_cache_hit_tokens", "prompt_cache_miss_tokens", "total_tokens")
+ usage = {key: sum(int((call.get("usage") or {}).get(key, 0)) for call in calls) for key in usage_keys}
+ return {
+ **dict(calls[-1]),
+ "physical_api_calls": len(calls),
+ "usage": usage,
+ "review_applied": len(calls) > 1,
+ "calls": [dict(call) for call in calls],
+ }
+
+
+RELATIVE_EVENT_REFERENCE_PATTERN = re.compile(
+ r"\b(?:when|before|after)\b", re.IGNORECASE
+)
+RELATIVE_DISTANCE_PATTERN = re.compile(
+ r"\b(?:ago|before|after)\b", re.IGNORECASE
+)
+
+
+def _relative_event_anchor_review_needed(
+ plan: Mapping[str, Any], catalog: Mapping[str, Any] | None
+) -> bool:
+ if not isinstance(catalog, Mapping):
+ return False
+ question = _text(catalog.get("question"))
+ if not (
+ RELATIVE_EVENT_REFERENCE_PATTERN.search(question)
+ and RELATIVE_DISTANCE_PATTERN.search(question)
+ ):
+ return False
+ evidence_by_id = {
+ _text(item.get("evidence_id")): item
+ for item in catalog.get("evidence") or []
+ if isinstance(item, Mapping)
+ }
+ anchor_ids = [
+ _text(value)
+ for value in catalog.get("lexical_anchor_ids") or []
+ if _text(value) in evidence_by_id
+ ]
+ anchor_sessions = {
+ _text(evidence_by_id[evidence_id].get("session_id"))
+ for evidence_id in anchor_ids
+ }
+ if len(anchor_sessions - {""}) < 2:
+ return False
+ question_atom_ids = {
+ _text(item.get("atom_id"))
+ for item in catalog.get("atoms") or []
+ if isinstance(item, Mapping) and item.get("evidence_id") == "QUESTION"
+ }
+ for operation in plan.get("operations") or []:
+ if (
+ not isinstance(operation, Mapping)
+ or operation.get("operation_type") != "date_difference"
+ or not question_atom_ids.intersection(operation.get("input_atom_ids") or [])
+ ):
+ continue
+ bound_ids = {
+ _text(value) for value in operation.get("input_evidence_ids") or []
+ }
+ bound_sessions = {
+ _text(evidence_by_id[value].get("session_id"))
+ for value in bound_ids
+ if value in evidence_by_id
+ }
+ if any(
+ evidence_id not in bound_ids
+ and _text(evidence_by_id[evidence_id].get("session_id"))
+ not in bound_sessions
+ for evidence_id in anchor_ids
+ ):
+ return True
+ return False
+
+
+def _required_memory_premise_binding(
+ plan: Mapping[str, Any], catalog: Mapping[str, Any] | None
+) -> dict[str, Any] | None:
+ if not isinstance(catalog, Mapping):
+ return None
+ anchor_ids = {
+ _text(value)
+ for value in catalog.get("lexical_anchor_ids") or []
+ if _text(value)
+ }
+ if not anchor_ids:
+ return None
+ required_memory_ids = {
+ _text(premise.get("premise_id"))
+ for premise in (plan.get("task_contract") or {}).get("premises") or []
+ if isinstance(premise, Mapping)
+ and premise.get("source") == "memory"
+ and premise.get("necessity") == "required"
+ and _text(premise.get("premise_id"))
+ }
+ if not required_memory_ids:
+ return None
+ bound_ids = {
+ _text(value)
+ for requirement in plan.get("requirements") or []
+ if isinstance(requirement, Mapping)
+ and _text(requirement.get("requirement_id")) in required_memory_ids
+ for value in requirement.get("evidence_ids") or []
+ if _text(value)
+ }
+ if not bound_ids or bound_ids.intersection(anchor_ids):
+ return None
+ return {
+ "required_memory_requirement_ids": sorted(required_memory_ids),
+ "bound_evidence_ids": sorted(bound_ids),
+ "lexical_anchor_ids": sorted(anchor_ids),
+ }
+
+
+def _review_context_for_plan(
+ plan: Mapping[str, Any], catalog: Mapping[str, Any] | None = None
+) -> dict[str, Any] | None:
+ missing_requirement_ids = unbound_memory_requirement_ids(plan)
+ structural_risks = operation_plan_structural_risks(plan)
+ relative_event_anchor_risk = _relative_event_anchor_review_needed(plan, catalog)
+ memory_relevance_signal = _required_memory_premise_binding(plan, catalog)
+ review_reasons = [
+ *structural_risks,
+ *(["unbound_required_memory_premises"] if missing_requirement_ids else []),
+ *(["relative_event_anchor_not_bound"] if relative_event_anchor_risk else []),
+ *(
+ ["required_memory_premise_outside_query_anchors"]
+ if memory_relevance_signal
+ else []
+ ),
+ ]
+ if not review_reasons:
+ return None
+ relative_anchor_instruction = (
+ " This review has already established that another remembered event "
+ "is the relative-time endpoint. QUESTION is forbidden as a "
+ "date_difference input; bind at least two distinct Source event dates."
+ if relative_event_anchor_risk
+ else ""
+ )
+ memory_relevance_instruction = (
+ " The required memory premise is currently bound only to evidence that "
+ "falls outside the query's lexical anchors. Do not satisfy the memory "
+ "contract with an unrelated remembered detail. Reinspect the candidate "
+ "evidence, including user-authored Source text and source-group context, "
+ "and bind memory that materially constrains the answer target. Lexical "
+ "anchors are review candidates rather than a hard allowlist, so an "
+ "implicitly related memory may remain only when the rebuilt plan changes "
+ "the evidence binding or adds relevant corroborating evidence."
+ if memory_relevance_signal
+ else ""
+ )
+ hard_constraints: dict[str, Any] = {}
+ if relative_event_anchor_risk:
+ hard_constraints.update(
+ {
+ "forbidden_date_difference_evidence_ids": ["QUESTION"],
+ "minimum_distinct_source_event_anchors": 2,
+ }
+ )
+ if memory_relevance_signal:
+ hard_constraints["forbidden_required_memory_evidence_ids"] = list(
+ memory_relevance_signal["bound_evidence_ids"]
+ )
+ return {
+ "initial_plan": dict(plan),
+ "review_reasons": review_reasons,
+ "missing_requirement_ids": missing_requirement_ids,
+ "memory_premise_relevance_signal": memory_relevance_signal or {},
+ "hard_constraints": hard_constraints,
+ "instruction": (
+ "Rebuild the complete plan. Treat a missing state as a hypothesis, "
+ "not proof of absence; for memory-conditioned generation, bind the "
+ "remembered constraints instead of searching for a ready-made answer. "
+ "When a relative-time question names another remembered event with "
+ "when, before, or after, compare the distinct event anchors and do not "
+ "default to the question date unless the wording explicitly makes it "
+ "the comparison endpoint."
+ + relative_anchor_instruction
+ + memory_relevance_instruction
+ ),
+ }
+
+
+def _review_resolution_failure(
+ plan: Mapping[str, Any],
+ catalog: Mapping[str, Any] | None,
+ review_context: Mapping[str, Any] | None,
+) -> str:
+ reasons = set((review_context or {}).get("review_reasons") or [])
+ if (
+ "relative_event_anchor_not_bound" in reasons
+ and _relative_event_anchor_review_needed(plan, catalog)
+ ):
+ return (
+ "relative event-anchor review still binds QUESTION instead of "
+ "two distinct Source event anchors"
+ )
+ if "required_memory_premise_outside_query_anchors" in reasons:
+ current_signal = _required_memory_premise_binding(plan, catalog)
+ initial_signal = (review_context or {}).get(
+ "memory_premise_relevance_signal"
+ ) or {}
+ if current_signal and set(current_signal.get("bound_evidence_ids") or []) == set(
+ initial_signal.get("bound_evidence_ids") or []
+ ):
+ return (
+ "required memory-premise relevance review kept the same "
+ "out-of-anchor evidence binding"
+ )
+ return ""
+
+
+def _recover_plan_from_failure(
+ failure_path: Path,
+ *,
+ question_id: str,
+ input_sha256: str,
+ catalog: Mapping[str, Any],
+ expected_binding: Mapping[str, Any] | None = None,
+) -> tuple[dict[str, Any], dict[str, Any]] | None:
+ if not failure_path.is_file():
+ return None
+ persisted = json.loads(failure_path.read_text(encoding="utf-8"))
+ if (
+ persisted.get("question_id") != question_id
+ or persisted.get("input_sha256") != input_sha256
+ ):
+ raise EvidenceCompileError(
+ f"{question_id}: persisted compiler failure identity mismatch"
+ )
+ metadata = dict(persisted.get("planner") or {})
+ if _text(metadata.get("prompt_version")) != EVIDENCE_PLANNER_PROMPT_VERSION:
+ return None
+ if _text(metadata.get("review_policy_version")) != REVIEW_POLICY_VERSION:
+ return None
+ if expected_binding is not None:
+ expected_planner = expected_binding.get("planner")
+ if not isinstance(expected_planner, Mapping) or any(
+ metadata.get(field) != expected_planner.get(field)
+ for field in ("provider", "model", "prompt_version")
+ ):
+ return None
+ raw_plan = metadata.get("raw_plan")
+ if not isinstance(raw_plan, Mapping) and metadata.get(
+ "review_resolution_failed"
+ ):
+ review_context = metadata.get("review_context")
+ if isinstance(review_context, Mapping):
+ raw_plan = review_context.get("initial_plan")
+ if not isinstance(raw_plan, Mapping):
+ return None
+ try:
+ plan, warnings = normalize_planner_output(raw_plan, catalog)
+ except Exception:
+ # The journal is an optimization, not a permanent failure latch. A raw
+ # response that cannot satisfy the current contract must be replanned.
+ return None
+ return plan, {
+ **metadata,
+ "raw_plan": dict(raw_plan),
+ "recovered_from_persisted_raw_plan": True,
+ "normalization_warnings": warnings,
+ }
+
+
+def _replan_context_from_failure(
+ failure_path: Path,
+ *,
+ question_id: str,
+ input_sha256: str,
+ expected_binding: Mapping[str, Any] | None = None,
+ catalog: Mapping[str, Any] | None = None,
+) -> dict[str, Any] | None:
+ if not failure_path.is_file():
+ return None
+ persisted = json.loads(failure_path.read_text(encoding="utf-8"))
+ if (
+ persisted.get("question_id") != question_id
+ or persisted.get("input_sha256") != input_sha256
+ ):
+ raise EvidenceCompileError(
+ f"{question_id}: persisted compiler failure identity mismatch"
+ )
+ metadata = dict(persisted.get("planner") or {})
+ if _text(metadata.get("prompt_version")) != EVIDENCE_PLANNER_PROMPT_VERSION:
+ return None
+ if expected_binding is not None:
+ expected_planner = expected_binding.get("planner")
+ if not isinstance(expected_planner, Mapping) or any(
+ metadata.get(field) != expected_planner.get(field)
+ for field in ("provider", "model", "prompt_version")
+ ):
+ return None
+ raw_plan = metadata.get("raw_plan")
+ error = _text(persisted.get("error"))
+ if not isinstance(raw_plan, Mapping) or not error:
+ return None
+ candidate_memory_evidence_ids = [
+ _text(value)
+ for value in (catalog or {}).get("lexical_anchor_ids") or []
+ if _text(value)
+ ]
+ return {
+ "initial_plan": dict(raw_plan),
+ "review_reasons": ["persisted_plan_validation_failure"],
+ "prior_validation_error": error,
+ "candidate_memory_evidence_ids": candidate_memory_evidence_ids,
+ "instruction": (
+ "Rebuild the complete plan and correct the prior runtime contract "
+ f"validation failure: {error}. Inspect the supplied evidence again. "
+ "For memory-conditioned generation, bind at least one relevant "
+ "required memory preference, constraint, owned tools, or resources. "
+ "For recommendation or advice, a previously mentioned owned tool or "
+ "accessory that can directly address the target problem is a valid "
+ "memory premise. Do not choose an unrelated preference merely because "
+ "it is more explicit; the premise must share the target domain or "
+ "intended action with the current question. Reinspect the listed "
+ "candidate_memory_evidence_ids plus their user-authored Source and "
+ "source-group context, and bind the relevant parent Evidence ID. "
+ "The candidate list is a review hint, not a hard allowlist. If no "
+ "relevant memory is supportable, "
+ "preserve an unbound memory requirement instead of relabeling query "
+ "context as remembered evidence."
+ ),
+ }
+
+
+def _refresh_completed_artifact(
+ saved: Mapping[str, Any],
+ row: Mapping[str, Any],
+ *,
+ question_id: str,
+ input_sha256: str,
+ expected_binding: Mapping[str, Any] | None = None,
+) -> tuple[dict[str, Any], bool]:
+ if (
+ saved.get("question_id") != question_id
+ ):
+ raise EvidenceCompileError(
+ f"{question_id}: persisted compiler identity mismatch"
+ )
+ saved_row = saved.get("row")
+ if saved.get("input_sha256") != input_sha256 and (
+ not isinstance(saved_row, Mapping) or _identity(saved_row) != input_sha256
+ ):
+ raise EvidenceCompileError(
+ f"{question_id}: persisted compiler identity mismatch"
+ )
+ output = dict(saved)
+ if expected_binding is None:
+ persisted_binding = saved.get("artifact_binding")
+ planner = (
+ persisted_binding.get("planner")
+ if isinstance(persisted_binding, Mapping)
+ else saved.get("planner")
+ )
+ if not isinstance(planner, Mapping):
+ raise EvidenceCompileError(
+ f"{question_id}: completed artifact lacks its planner contract"
+ )
+ expected_binding = _artifact_binding(
+ row,
+ provider=_text(planner.get("provider")),
+ model=_text(planner.get("model")),
+ )
+ _assert_artifact_binding(
+ saved,
+ row,
+ input_sha256=input_sha256,
+ expected_binding=expected_binding,
+ )
+ planner_metadata = output.get("planner")
+ if (
+ not isinstance(planner_metadata, Mapping)
+ or _text(planner_metadata.get("prompt_version"))
+ != EVIDENCE_PLANNER_PROMPT_VERSION
+ ):
+ raise EvidenceCompileError(
+ f"{question_id}: completed artifact uses a stale evidence planner contract"
+ )
+ saved_row = output.get("row")
+ packet = saved_row.get("compiled_evidence_packet") if isinstance(saved_row, Mapping) else None
+ if not isinstance(packet, Mapping):
+ raise EvidenceCompileError(
+ f"{question_id}: completed artifact lacks a compiled evidence packet"
+ )
+ binding_changed = output.get("artifact_binding") != dict(expected_binding)
+ if packet.get("packet_compiler_version") == PACKET_COMPILER_VERSION:
+ if binding_changed:
+ output["artifact_binding"] = dict(expected_binding)
+ output["compiler_binding_sha256"] = _binding_sha256(expected_binding)
+ return output, binding_changed
+ plan = packet.get("operation_plan")
+ if not isinstance(plan, Mapping):
+ raise EvidenceCompileError(
+ f"{question_id}: completed artifact lacks its persisted operation plan"
+ )
+ output_row = dict(row)
+ output_row["compiled_evidence_packet"] = compile_evidence_packet(row, plan)
+ output["row"] = output_row
+ output["packet_compiler_version"] = PACKET_COMPILER_VERSION
+ output["artifact_binding"] = dict(expected_binding)
+ output["compiler_binding_sha256"] = _binding_sha256(expected_binding)
+ output["local_packet_recompile_count"] = int(
+ output.get("local_packet_recompile_count", 0)
+ ) + 1
+ return output, True
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description="Compile immutable TMCRA evidence into operation-bound answer packets")
+ parser.add_argument("--evidence", type=Path, required=True)
+ parser.add_argument("--out-dir", type=Path, required=True)
+ parser.add_argument("--qid-list", type=Path)
+ parser.add_argument(
+ "--retrieval-debug",
+ type=Path,
+ help="retrieval_debug.jsonl to bind into the compiled evaluation artifact",
+ )
+ parser.add_argument(
+ "--require-retrieval-debug",
+ action="store_true",
+ help="fail before planner calls unless retrieval debug is present and qid-aligned",
+ )
+ parser.add_argument("--writer-env", type=Path, default=DEFAULT_WRITER_ENV)
+ parser.add_argument("--workers", type=int, default=4)
+ parser.add_argument("--timeout", type=float, default=180.0)
+ parser.add_argument("--planner-provider", choices=("deepseek", "xiaomi_mimo"), default="deepseek")
+ parser.add_argument("--planner-model")
+ parser.add_argument("--planner-base-url")
+ parser.add_argument("--planner-key-file", type=Path)
+ parser.add_argument(
+ "--diagnostic",
+ action="store_true",
+ help="explicitly compile a diagnostic retrieval lane that cannot enter production evaluation",
+ )
+ args = parser.parse_args()
+ if args.workers <= 0:
+ raise EvidenceCompileError("workers must be positive")
+ rows = _read_jsonl(args.evidence.resolve())
+ if args.qid_list:
+ qids = [line.strip() for line in args.qid_list.read_text(encoding="utf-8").splitlines() if line.strip()]
+ by_qid = {_text(row.get("question_id")): row for row in rows}
+ if not qids or len(qids) != len(set(qids)) or any(qid not in by_qid for qid in qids):
+ raise EvidenceCompileError("qid list is empty, duplicated, or absent from evidence")
+ rows = [by_qid[qid] for qid in qids]
+ try:
+ route_report = (
+ validate_diagnostic_retrieval_rows(rows)
+ if args.diagnostic
+ else validate_production_retrieval_rows(rows)
+ )
+ except RoutePolicyError as exc:
+ raise EvidenceCompileError(
+ f"retrieval route failed before evidence planner calls: {exc}"
+ ) from exc
+ expected_qids = [_text(row.get("question_id")) for row in rows]
+ retrieval_debug_source = (
+ args.retrieval_debug.resolve()
+ if args.retrieval_debug
+ else args.evidence.resolve().parent / "retrieval_debug.jsonl"
+ )
+ retrieval_debug_preflight: dict[str, Any] | None = None
+ if retrieval_debug_source.is_file():
+ retrieval_debug_preflight = _retrieval_debug_report(
+ retrieval_debug_source,
+ expected_qids,
+ )
+ elif args.require_retrieval_debug:
+ raise EvidenceCompileError(
+ f"retrieval debug is required before planner calls: {retrieval_debug_source}"
+ )
+ if args.planner_provider == "deepseek":
+ environment = _load_shell_environment(args.writer_env.resolve())
+ keys = _key_pool(environment)
+ base_url = args.planner_base_url or environment.get("TMCRA_DEEPSEEK_WRITER_BASE_URL") or environment.get("TMCRA_WRITER_BASE_URL") or "https://api.deepseek.com/v1"
+ planner_model = args.planner_model or "deepseek-v4-pro"
+ else:
+ if not args.planner_key_file or not args.planner_key_file.is_file():
+ raise EvidenceCompileError("xiaomi_mimo requires --planner-key-file")
+ key = args.planner_key_file.read_text(encoding="utf-8").strip()
+ if not key:
+ raise EvidenceCompileError("planner key file is empty")
+ keys = [key]
+ base_url = args.planner_base_url or "https://api.xiaomimimo.com/v1"
+ planner_model = args.planner_model or "mimo-v2.5"
+ out_dir = args.out_dir.resolve()
+ journal_dir = out_dir / "rows"
+ journal_dir.mkdir(parents=True, exist_ok=True)
+ failure_history_path = out_dir / "planner_failure_history.jsonl"
+ lock = threading.Lock()
+
+ def compile_one(index: int, row: Mapping[str, Any]) -> dict[str, Any]:
+ qid = _text(row.get("question_id"))
+ identity = _identity(row)
+ binding = _artifact_binding(
+ row,
+ provider=args.planner_provider,
+ model=planner_model,
+ )
+ artifact = journal_dir / f"{index:06d}_{qid}.json"
+ failure_path = journal_dir / f"{index:06d}_{qid}.failure.json"
+ if artifact.is_file():
+ saved = json.loads(artifact.read_text(encoding="utf-8"))
+ refreshed, changed = _refresh_completed_artifact(
+ saved,
+ row,
+ question_id=qid,
+ input_sha256=identity,
+ expected_binding=binding,
+ )
+ if changed:
+ with lock:
+ _atomic_json(artifact, refreshed)
+ return refreshed
+ catalog = build_evidence_catalog(row)
+ planner = DeepSeekEvidenceOperationPlanner(
+ base_url=base_url,
+ api_keys=[keys[index % len(keys)]],
+ timeout=args.timeout,
+ model=planner_model,
+ provider=args.planner_provider,
+ )
+ planner_calls: list[dict[str, Any]] = []
+ recovered_from_persisted_raw_plan = False
+ try:
+ persisted_replan_context = _replan_context_from_failure(
+ failure_path,
+ question_id=qid,
+ input_sha256=identity,
+ expected_binding=binding,
+ catalog=catalog,
+ )
+ recovered = _recover_plan_from_failure(
+ failure_path,
+ question_id=qid,
+ input_sha256=identity,
+ catalog=catalog,
+ expected_binding=binding,
+ )
+ if recovered is not None:
+ plan, recovered_metadata = recovered
+ recovered_from_persisted_raw_plan = True
+ planner_calls.append(recovered_metadata)
+ if not recovered_from_persisted_raw_plan:
+ plan, metadata = planner.plan(
+ catalog,
+ review_context=persisted_replan_context,
+ )
+ planner_calls.append(metadata)
+ review_context = _review_context_for_plan(plan, catalog)
+ if review_context is not None:
+ plan, review_metadata = planner.plan(
+ catalog,
+ review_context=review_context,
+ )
+ planner_calls.append(review_metadata)
+ review_failure = _review_resolution_failure(
+ plan,
+ catalog,
+ review_context,
+ )
+ if review_failure:
+ raise EvidencePlannerError(
+ review_failure,
+ metadata={
+ **review_metadata,
+ "review_resolution_failed": True,
+ "review_context": review_context,
+ },
+ )
+ metadata = _aggregate_planner_calls(planner_calls)
+ metadata["review_policy_version"] = REVIEW_POLICY_VERSION
+ metadata["recovered_from_persisted_raw_plan"] = (
+ recovered_from_persisted_raw_plan
+ )
+ metadata["new_physical_api_calls_this_resume"] = len(planner_calls) - int(
+ recovered_from_persisted_raw_plan
+ )
+ except EvidencePlannerError as exc:
+ failure = {
+ "failure_id": f"ecf_{time.time_ns()}",
+ "observed_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
+ "question_id": qid,
+ "input_sha256": identity,
+ "artifact_binding": binding,
+ "compiler_binding_sha256": _binding_sha256(binding),
+ "status": "failed",
+ "error": str(exc),
+ "planner": {
+ **exc.metadata,
+ "review_policy_version": REVIEW_POLICY_VERSION,
+ **({"prior_calls": planner_calls} if planner_calls else {}),
+ },
+ }
+ with lock:
+ _atomic_json(failure_path, failure)
+ _append_jsonl_atomic(failure_history_path, failure)
+ raise
+ plan = validate_operation_plan(plan, catalog)
+ packet = compile_evidence_packet(row, plan)
+ output_row = dict(row)
+ output_row["compiled_evidence_packet"] = packet
+ result = {
+ "question_id": qid,
+ "input_sha256": identity,
+ "artifact_binding": binding,
+ "compiler_binding_sha256": _binding_sha256(binding),
+ "status": "completed",
+ "planner": metadata,
+ "row": output_row,
+ }
+ with lock:
+ _atomic_json(artifact, result)
+ failure_path.unlink(missing_ok=True)
+ return result
+
+ results: dict[str, dict[str, Any]] = {}
+ failures: list[dict[str, str]] = []
+ with ThreadPoolExecutor(max_workers=args.workers) as executor:
+ futures = {executor.submit(compile_one, index, row): _text(row.get("question_id")) for index, row in enumerate(rows)}
+ for future in as_completed(futures):
+ qid = futures[future]
+ try:
+ result = future.result()
+ except Exception as exc:
+ failures.append({"question_id": qid, "error": f"{exc.__class__.__name__}: {exc}"})
+ else:
+ results[qid] = result
+ if failures:
+ _atomic_json(out_dir / "failures.json", {"failures": failures})
+ raise EvidenceCompileError(f"evidence compilation failed for {len(failures)} rows")
+ ordered_results = [results[_text(row.get("question_id"))] for row in rows]
+ (out_dir / "failures.json").unlink(missing_ok=True)
+ ordered_rows = [item["row"] for item in ordered_results]
+ calls = [item["planner"] for item in ordered_results]
+ failure_history = _read_jsonl(failure_history_path) if failure_history_path.is_file() else []
+ completed_physical_calls = _physical_calls(calls)
+ all_physical_calls = _physical_calls(
+ [*calls, *(dict(item.get("planner") or {}) for item in failure_history)]
+ )
+ _atomic_jsonl(out_dir / "evidence_windows.jsonl", ordered_rows)
+ retrieval_debug_report = (
+ _stage_retrieval_debug(
+ retrieval_debug_source,
+ out_dir,
+ expected_qids,
+ expected_sha256=retrieval_debug_preflight["source_sha256"],
+ )
+ if retrieval_debug_preflight is not None
+ else {
+ "schema_version": "tmcra.v4.compiled-retrieval-debug-binding.1",
+ "status": "not_provided",
+ }
+ )
+ _atomic_json(
+ out_dir / "report.json",
+ {
+ "schema_version": "tmcra.v4.evidence-compiler-run.1",
+ "status": "complete",
+ "row_count": len(ordered_rows),
+ "physical_call_count": len(all_physical_calls),
+ "completed_artifact_physical_call_count": len(completed_physical_calls),
+ "failure_history_record_count": len(failure_history),
+ "locally_recompiled_row_count": sum(
+ int(item.get("local_packet_recompile_count", 0) > 0)
+ for item in ordered_results
+ ),
+ "packet_compiler_version": PACKET_COMPILER_VERSION,
+ "planner_provider": args.planner_provider,
+ "planner_model": planner_model,
+ "retrieval_route_report": route_report,
+ "retrieval_debug": retrieval_debug_report,
+ "operation_count": sum(len(row["compiled_evidence_packet"]["operation_results"]) for row in ordered_rows),
+ "requirement_count": sum(len(row["compiled_evidence_packet"]["requirement_coverage"]) for row in ordered_rows),
+ "structural_risk_count": sum(
+ len(row["compiled_evidence_packet"].get("structural_risk_signals") or [])
+ for row in ordered_rows
+ ),
+ "typed_semantics_accepted_count": sum(
+ len((row["compiled_evidence_packet"].get("typed_semantics_report") or {}).get("accepted") or [])
+ for row in ordered_rows
+ ),
+ "typed_semantics_rejected_count": sum(
+ len((row["compiled_evidence_packet"].get("typed_semantics_report") or {}).get("rejected") or [])
+ for row in ordered_rows
+ ),
+ "exact_cost_cny": _cost_cny(all_physical_calls),
+ },
+ )
+ (out_dir / "COMPILE_COMPLETE").write_text(time.strftime("%Y-%m-%dT%H:%M:%S%z") + "\n", encoding="utf-8")
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/legacy/run_tmcra_v4_evaluate.py b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/run_tmcra_v4_evaluate.py
new file mode 100644
index 0000000000000000000000000000000000000000..b740bd451479b06dff9ba4bcbaba24826d170d05
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/run_tmcra_v4_evaluate.py
@@ -0,0 +1,725 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import subprocess
+import sys
+from collections.abc import Mapping
+from pathlib import Path
+from typing import Any
+
+from run_tmcra_v4_build import BASE, DEFAULT_DATA, _now, _resume_log, _run
+from run_tmcra_v4_retrieve import DEFAULT_HARNESS
+from tmcra_v4_route_policy import (
+ POLICY_SCHEMA,
+ PRODUCTION_ANSWER_MODEL,
+ PRODUCTION_ANSWER_PROTOCOL,
+ PRODUCTION_ANSWER_RUNNER,
+ PRODUCTION_LANE,
+ SOURCE_COVERAGE_TRACE_K,
+ RoutePolicyError,
+ assert_production_answer_runner,
+ validate_production_answers,
+ validate_production_evidence,
+)
+from tmcra_v4_regression_gate import (
+ DEFAULT_CASES as DEFAULT_REGRESSION_CASES,
+ evaluate_gate as evaluate_regression_gate,
+)
+
+
+PROJECT_ROOT = BASE.parents[2]
+PACKAGE_ROOT = BASE.parent
+DEFAULT_ANSWER_ENV = PROJECT_ROOT / "configs" / "answer.env"
+DEFAULT_JUDGE = PACKAGE_ROOT / "harness" / "evaluate_qa_official_judge.py"
+DEFAULT_ANSWER_RUNNER = BASE / PRODUCTION_ANSWER_RUNNER
+
+
+class EvaluationError(RuntimeError):
+ pass
+
+
+def _load_environment(path: Path) -> dict[str, str]:
+ if not path.is_file():
+ raise EvaluationError(f"answer environment file is missing: {path}")
+ result = subprocess.run(
+ ["bash", "-c", 'set -a; source "$1"; env -0', "tmcra-v4-answer-env", str(path)],
+ check=True,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE,
+ )
+ output: dict[str, str] = {}
+ for entry in result.stdout.decode("utf-8").split("\0"):
+ if "=" in entry:
+ key, value = entry.split("=", 1)
+ output[key] = value
+ return output
+
+
+def _read_jsonl(path: Path) -> list[dict[str, Any]]:
+ return [
+ json.loads(line)
+ for line in path.read_text(encoding="utf-8").splitlines()
+ if line.strip()
+ ]
+
+
+def _source_top24_session_ids(
+ debug_row: Mapping[str, Any], qid: str
+) -> list[str]:
+ trace = debug_row.get("source_top24_candidates")
+ pool = debug_row.get("source_candidate_pool_trace")
+ candidate_count = debug_row.get("source_candidate_count")
+ trace_k = debug_row.get("source_coverage_trace_k")
+ if type(candidate_count) is not int or candidate_count <= 0:
+ raise EvaluationError(f"{qid}: Source candidate count is invalid")
+ if trace_k != SOURCE_COVERAGE_TRACE_K:
+ raise EvaluationError(
+ f"{qid}: Source coverage trace contract is not Top{SOURCE_COVERAGE_TRACE_K}"
+ )
+ if not isinstance(trace, list) or not trace:
+ raise EvaluationError(f"{qid}: Source Top24 coverage trace is missing")
+ if not isinstance(pool, list):
+ raise EvaluationError(f"{qid}: Source candidate pool trace is missing")
+ expected_trace_count = min(SOURCE_COVERAGE_TRACE_K, candidate_count)
+ if len(trace) != expected_trace_count:
+ raise EvaluationError(
+ f"{qid}: Source Top24 coverage trace has {len(trace)} rows; "
+ f"expected {expected_trace_count}"
+ )
+ if len(pool) != candidate_count:
+ raise EvaluationError(
+ f"{qid}: Source candidate pool trace has {len(pool)} rows; "
+ f"expected {candidate_count}"
+ )
+
+ def validate_trace_rows(
+ rows: list[Any], *, label: str
+ ) -> list[tuple[str, str, int, int, int]]:
+ identities: list[tuple[str, str, int, int, int]] = []
+ locations: set[tuple[str, int, int, int]] = set()
+ for expected_rank, item in enumerate(rows, 1):
+ if not isinstance(item, Mapping):
+ raise EvaluationError(f"{qid}: {label} row must be an object")
+ rank = item.get("rank")
+ if type(rank) is not int or rank != expected_rank:
+ raise EvaluationError(f"{qid}: {label} is malformed")
+ candidate_id = str(item.get("candidate_id") or "").strip()
+ session_id = str(item.get("session_id") or "").strip()
+ coordinates = (
+ item.get("session_index"),
+ item.get("parent_chunk_index"),
+ item.get("subchunk_index"),
+ )
+ if (
+ not candidate_id
+ or not session_id
+ or any(type(value) is not int or value < 0 for value in coordinates)
+ ):
+ raise EvaluationError(f"{qid}: {label} has an invalid identity")
+ location = (session_id, *coordinates)
+ # Candidate IDs are content-derived and can collide when a dataset
+ # repeats the same session/chunk at a different session_index. The
+ # physical occurrence coordinates remain the authoritative identity.
+ if location in locations:
+ raise EvaluationError(f"{qid}: {label} contains duplicate evidence")
+ locations.add(location)
+ identities.append((candidate_id, session_id, *coordinates))
+ return identities
+
+ top_identities = validate_trace_rows(trace, label="Source Top24 coverage trace")
+ pool_identities = validate_trace_rows(pool, label="Source candidate pool trace")
+ if top_identities != pool_identities[:expected_trace_count]:
+ raise EvaluationError(
+ f"{qid}: Source Top24 trace is not the exact candidate-pool prefix"
+ )
+ session_ids: list[str] = []
+ for item in trace:
+ session_ids.append(str(item["session_id"]).strip())
+ return list(dict.fromkeys(session_ids))
+
+
+def _validate_retrieval_debug_rows(
+ expected_qids: list[str], retrieval_debug: list[dict[str, Any]]
+) -> None:
+ actual_qids = [
+ str(row.get("question_id") or row.get("qid") or "").strip()
+ if isinstance(row, Mapping)
+ else ""
+ for row in retrieval_debug
+ ]
+ if actual_qids != expected_qids:
+ raise EvaluationError("retrieval debug rows do not match frozen qid order")
+ for qid, row in zip(expected_qids, retrieval_debug, strict=True):
+ _source_top24_session_ids(row, qid)
+
+
+def _reference_rows(value: Any, source: str = "reference") -> list[dict[str, Any]]:
+ if isinstance(value, list):
+ raw_rows = value
+ elif isinstance(value, Mapping):
+ raw_rows: Any = None
+ for key in ("questions", "data", "rows", "references"):
+ if key in value:
+ raw_rows = value[key]
+ break
+ if raw_rows is None and value and all(isinstance(item, Mapping) for item in value.values()):
+ raw_rows = [dict(item, qid=key) for key, item in value.items()]
+ if raw_rows is None and ("qid" in value or "question_id" in value):
+ raw_rows = [value]
+ if not isinstance(raw_rows, list):
+ raise EvaluationError(f"{source} must contain a list of rows")
+ else:
+ raise EvaluationError(f"{source} must be a JSON object or list")
+
+ rows: list[dict[str, Any]] = []
+ seen: set[str] = set()
+ for index, raw_row in enumerate(raw_rows, 1):
+ if not isinstance(raw_row, Mapping):
+ raise EvaluationError(f"{source} row {index} must be an object")
+ qid_value = raw_row.get("qid")
+ question_id = raw_row.get("question_id")
+ if qid_value is not None and question_id is not None and str(qid_value).strip() != str(question_id).strip():
+ raise EvaluationError(f"{source} row {index} has disagreeing qid and question_id")
+ qid = str(qid_value or question_id or "").strip()
+ if not qid:
+ raise EvaluationError(f"{source} row {index} is missing qid")
+ if qid in seen:
+ raise EvaluationError(f"{source} contains duplicate qid {qid}")
+ session_ids = raw_row.get("answer_session_ids")
+ if not isinstance(session_ids, list) or not session_ids:
+ raise EvaluationError(f"{source} row {index} ({qid}) lacks answer_session_ids")
+ normalized_sessions = [str(item).strip() for item in session_ids]
+ if any(not item for item in normalized_sessions) or len(set(normalized_sessions)) != len(normalized_sessions):
+ raise EvaluationError(
+ f"{source} row {index} ({qid}) has blank or duplicate answer_session_ids"
+ )
+ gold_key = next(
+ (key for key in ("gold_answer", "answer", "expected_answer") if key in raw_row),
+ None,
+ )
+ if gold_key is None:
+ raise EvaluationError(f"{source} row {index} ({qid}) lacks a gold answer")
+ rows.append(
+ {
+ "qid": qid,
+ "gold_answer": raw_row[gold_key],
+ "answer_session_ids": normalized_sessions,
+ "question_type": str(
+ raw_row.get("question_type") or raw_row.get("type") or "unknown"
+ ).strip()
+ or "unknown",
+ }
+ )
+ seen.add(qid)
+ if not rows:
+ raise EvaluationError(f"{source} is empty")
+ return rows
+
+
+def _read_reference(path: Path) -> list[dict[str, Any]]:
+ try:
+ value = json.loads(path.read_text(encoding="utf-8"))
+ except OSError as exc:
+ raise EvaluationError(f"cannot read reference data: {exc}") from exc
+ except json.JSONDecodeError as exc:
+ raise EvaluationError("reference data is not valid JSON") from exc
+ return _reference_rows(value)
+
+
+def _build_evaluation_audit(
+ expected_qids: list[str],
+ reference: list[dict[str, Any]],
+ evidence: list[dict[str, Any]],
+ retrieval_debug: list[dict[str, Any]],
+ judged: list[dict[str, Any]],
+) -> list[dict[str, Any]]:
+ _validate_retrieval_debug_rows(expected_qids, retrieval_debug)
+ reference_by_qid = {row["qid"]: row for row in reference}
+ evidence_by_qid = {str(row.get("question_id") or row.get("qid") or "").strip(): row for row in evidence}
+ judged_by_qid = {str(row.get("question_id") or row.get("qid") or "").strip(): row for row in judged}
+ debug_by_qid = {
+ str(row.get("question_id") or row.get("qid") or "").strip(): row
+ for row in retrieval_debug
+ }
+ missing_reference = [qid for qid in expected_qids if qid not in reference_by_qid]
+ missing_evidence = [qid for qid in expected_qids if qid not in evidence_by_qid]
+ missing_judged = [qid for qid in expected_qids if qid not in judged_by_qid]
+ missing_debug = [qid for qid in expected_qids if qid not in debug_by_qid]
+ if missing_reference:
+ raise EvaluationError(f"reference is missing evaluated qids: {missing_reference[:10]}")
+ if missing_evidence:
+ raise EvaluationError(f"evidence is missing evaluated qids: {missing_evidence[:10]}")
+ if missing_judged:
+ raise EvaluationError(f"judge is missing evaluated qids: {missing_judged[:10]}")
+ if missing_debug:
+ raise EvaluationError(
+ f"retrieval debug is missing evaluated qids: {missing_debug[:10]}"
+ )
+
+ audit: list[dict[str, Any]] = []
+ for qid in expected_qids:
+ reference_row = reference_by_qid[qid]
+ selected = evidence_by_qid[qid].get("selected_session_ids")
+ if not isinstance(selected, list):
+ raise EvaluationError(f"{qid}: selected_session_ids must be a list")
+ selected_ids = [str(item).strip() for item in selected]
+ if any(not item for item in selected_ids) or len(set(selected_ids)) != len(selected_ids):
+ raise EvaluationError(f"{qid}: selected_session_ids has blank or duplicate values")
+ answer_ids = list(reference_row["answer_session_ids"])
+ source_top24_session_ids = _source_top24_session_ids(debug_by_qid[qid], qid)
+
+ def coverage(candidate_session_ids: list[str]) -> dict[str, Any]:
+ missing_ids = [
+ session_id
+ for session_id in answer_ids
+ if session_id not in set(candidate_session_ids)
+ ]
+ return {
+ "complete": not missing_ids,
+ "expected_session_count": len(answer_ids),
+ "selected_session_count": len(candidate_session_ids),
+ "covered_session_count": len(answer_ids) - len(missing_ids),
+ "hit_rate": round(
+ (len(answer_ids) - len(missing_ids)) / len(answer_ids), 6
+ ),
+ "missing_session_ids": missing_ids,
+ }
+
+ source_top24_coverage = coverage(source_top24_session_ids)
+ final_evidence_coverage = coverage(selected_ids)
+ official_label = official_judge_correct(judged_by_qid[qid])
+ audit.append(
+ {
+ "qid": qid,
+ "question_id": qid,
+ "question_type": reference_row["question_type"],
+ "gold_answer": reference_row["gold_answer"],
+ "gold_answer_session_ids": answer_ids,
+ "answer_session_ids": answer_ids,
+ "source_top24_session_ids": source_top24_session_ids,
+ "selected_session_ids": selected_ids,
+ "source_top24_coverage": source_top24_coverage,
+ "final_evidence_coverage": final_evidence_coverage,
+ "session_coverage": final_evidence_coverage,
+ "source_coverage_complete": source_top24_coverage["complete"],
+ "source_top24_coverage_complete": source_top24_coverage["complete"],
+ "final_evidence_coverage_complete": final_evidence_coverage["complete"],
+ "missing_source_top24_session_ids": source_top24_coverage[
+ "missing_session_ids"
+ ],
+ "missing_session_ids": final_evidence_coverage["missing_session_ids"],
+ "missing_answer_session_ids": final_evidence_coverage[
+ "missing_session_ids"
+ ],
+ "official_label": official_label,
+ "official_judge_label": official_label,
+ "official_correct": official_label,
+ }
+ )
+ return audit
+
+
+def _write_jsonl(path: Path, rows: list[Mapping[str, Any]]) -> None:
+ temporary = path.with_name(f".{path.name}.tmp.{os.getpid()}")
+ temporary.write_text(
+ "".join(json.dumps(dict(row), ensure_ascii=False, sort_keys=True) + "\n" for row in rows),
+ encoding="utf-8",
+ )
+ os.replace(temporary, path)
+
+
+def _validate_answer_facing_evidence(rows: list[dict[str, Any]]) -> None:
+ forbidden_fields = {
+ "answer",
+ "gold_answer",
+ "expected_answer",
+ "answer_session_ids",
+ "labels",
+ "supervision",
+ }
+ for row in rows:
+ qid = str(row.get("question_id") or "").strip() or ""
+ leaked = sorted(forbidden_fields.intersection(row))
+ if leaked:
+ raise EvaluationError(
+ f"{qid}: answer-facing evidence contains benchmark fields: {', '.join(leaked)}"
+ )
+
+
+def official_judge_correct(row: Mapping[str, Any]) -> bool:
+ if str(row.get("error") or "").strip():
+ raise EvaluationError("official judge row contains an error")
+ autoeval = row.get("autoeval_label")
+ if not isinstance(autoeval, Mapping) or type(autoeval.get("label")) is not bool:
+ raise EvaluationError(
+ "official judge row lacks boolean autoeval_label.label"
+ )
+ return bool(autoeval["label"])
+
+
+def _validate_judge_resume_rows(
+ answers: list[dict[str, Any]], judged: list[dict[str, Any]]
+) -> None:
+ answer_qids = [str(row.get("question_id") or "").strip() for row in answers]
+ judged_qids = [str(row.get("question_id") or "").strip() for row in judged]
+ if judged_qids != answer_qids[: len(judged_qids)]:
+ raise EvaluationError("existing judge rows are not a frozen answer prefix")
+ answer_by_qid = {str(row["question_id"]): row for row in answers}
+ for row in judged:
+ qid = str(row.get("question_id") or "").strip()
+ answer = answer_by_qid[qid]
+ for key in ("question", "hypothesis", "evidence_sha256", "answer_model"):
+ if row.get(key) != answer.get(key):
+ raise EvaluationError(
+ f"{qid}: existing judge row is bound to different answer evidence"
+ )
+
+
+def _archive_failure_marker(run_dir: Path, retrieval_tag: str) -> None:
+ marker = run_dir / f"FAILED.answer_{retrieval_tag}"
+ if not marker.is_file():
+ return
+ history = run_dir / f"answer_{retrieval_tag}.failure_history.jsonl"
+ with history.open("a", encoding="utf-8") as handle:
+ for line in marker.read_text(encoding="utf-8").splitlines():
+ if line.strip():
+ handle.write(line.rstrip() + "\n")
+ marker.unlink()
+
+
+def _record_failure(run_dir: Path, retrieval_tag: str, exc: Exception) -> None:
+ _archive_failure_marker(run_dir, retrieval_tag)
+ marker = run_dir / f"FAILED.answer_{retrieval_tag}"
+ temporary = marker.with_name(f".{marker.name}.tmp.{os.getpid()}")
+ temporary.write_text(
+ json.dumps({"at": _now(), "error": f"{exc.__class__.__name__}: {exc}"})
+ + "\n",
+ encoding="utf-8",
+ )
+ os.replace(temporary, marker)
+
+
+def evaluate(args: argparse.Namespace) -> dict[str, Any]:
+ run_dir = args.run_dir.resolve()
+ retrieval_dir = run_dir / args.retrieval_tag
+ if not any(
+ (retrieval_dir / marker).is_file()
+ for marker in ("RETRIEVAL_COMPLETE", "COMPILE_COMPLETE")
+ ):
+ raise EvaluationError("retrieval or evidence compilation run is incomplete")
+ answer_dir = run_dir / f"answer_{args.retrieval_tag}"
+ resume = bool(getattr(args, "resume", False))
+ failure_marker = run_dir / f"FAILED.answer_{args.retrieval_tag}"
+ judge_path = run_dir / f"{args.retrieval_tag}.official_judge.jsonl"
+ if answer_dir.exists() and not resume:
+ raise EvaluationError(f"answer output already exists: {answer_dir}")
+ if failure_marker.exists() and not resume:
+ raise EvaluationError("prior answer failure exists; use explicit --resume")
+ if resume and not (
+ answer_dir.exists() or failure_marker.exists() or judge_path.exists()
+ ):
+ raise EvaluationError("resume requested but no prior answer state exists")
+ try:
+ qid_list = Path(
+ getattr(args, "qid_list", None) or run_dir / "qids.txt"
+ ).resolve()
+ if not qid_list.is_file():
+ raise EvaluationError(f"answer qid list is missing: {qid_list}")
+ expected_qids = [
+ line.strip()
+ for line in qid_list.read_text(encoding="utf-8").splitlines()
+ if line.strip()
+ ]
+ if not expected_qids or len(expected_qids) != len(set(expected_qids)):
+ raise EvaluationError("answer qid list is empty or contains duplicates")
+ baseline_judge = getattr(args, "baseline_judge", None)
+ if baseline_judge is not None:
+ baseline_judge = Path(baseline_judge).resolve()
+ if not baseline_judge.is_file():
+ raise EvaluationError(
+ f"frozen baseline judge is missing: {baseline_judge}"
+ )
+
+ try:
+ assert_production_answer_runner(args.answer_runner.resolve())
+ answer_facing_evidence = _read_jsonl(retrieval_dir / "evidence_windows.jsonl")
+ _validate_answer_facing_evidence(answer_facing_evidence)
+ route_report = validate_production_evidence(answer_facing_evidence)
+ except RoutePolicyError as exc:
+ raise EvaluationError(f"production route policy rejected the run: {exc}") from exc
+ if [item.get("question_id") for item in answer_facing_evidence] != expected_qids:
+ raise EvaluationError("evidence rows do not match frozen qid order")
+ retrieval_debug = _read_jsonl(retrieval_dir / "retrieval_debug.jsonl")
+ _validate_retrieval_debug_rows(expected_qids, retrieval_debug)
+
+ loaded = _load_environment(args.answer_env.resolve())
+ environment = {**os.environ, **loaded}
+ if environment.get("TMCRA_ANSWER_MODEL") != PRODUCTION_ANSWER_MODEL:
+ raise EvaluationError(
+ f"answer layer must remain fixed to {PRODUCTION_ANSWER_MODEL}"
+ )
+ _run(
+ [
+ sys.executable,
+ str(args.answer_runner.resolve()),
+ "--evidence",
+ str(retrieval_dir / "evidence_windows.jsonl"),
+ "--harness",
+ str(args.harness.resolve()),
+ "--out-dir",
+ str(answer_dir),
+ "--workers",
+ str(args.answer_workers),
+ "--attempts",
+ str(args.answer_attempts),
+ "--answer-window-limit",
+ str(args.answer_window_limit),
+ "--qid-list",
+ str(qid_list),
+ "--lane",
+ "production",
+ ],
+ (
+ _resume_log(run_dir, f"answer_{args.retrieval_tag}")
+ if resume
+ else run_dir / f"answer_{args.retrieval_tag}.log"
+ ),
+ environment,
+ )
+ answers = _read_jsonl(answer_dir / "answers.jsonl")
+ try:
+ validate_production_answers(answers)
+ except RoutePolicyError as exc:
+ raise EvaluationError(f"production route policy rejected answers: {exc}") from exc
+ if [item.get("question_id") for item in answers] != expected_qids:
+ raise EvaluationError("answer rows do not match frozen qid order")
+ judge_resume_options: list[str] = []
+ if judge_path.is_file():
+ if not resume:
+ raise EvaluationError("judge output already exists without --resume")
+ existing_judged = _read_jsonl(judge_path)
+ _validate_judge_resume_rows(answers, existing_judged)
+ judge_resume_options = ["--resume"]
+ _run(
+ [
+ sys.executable,
+ str(args.judge.resolve()),
+ "--metric-model",
+ "gpt-5.4",
+ "--hyp-file",
+ str(answer_dir / "answers.jsonl"),
+ "--ref-file",
+ str(args.data.resolve()),
+ "--result-file",
+ str(judge_path),
+ "--base-url",
+ environment["TMCRA_ANSWER_BASE_URL"],
+ "--api-key-env",
+ "TMCRA_ANSWER_API_KEY",
+ "--timeout",
+ "180",
+ "--max-retries",
+ "0",
+ *judge_resume_options,
+ "--quiet",
+ ],
+ (
+ _resume_log(run_dir, f"{args.retrieval_tag}.judge")
+ if resume
+ else run_dir / f"{args.retrieval_tag}.judge.log"
+ ),
+ environment,
+ )
+ judged = _read_jsonl(judge_path)
+ if [item.get("question_id") for item in answers] != expected_qids:
+ raise EvaluationError("answer rows do not match frozen qid order")
+ if [item.get("question_id") for item in judged] != expected_qids:
+ raise EvaluationError("judge rows do not match frozen qid order")
+ if any(item.get("answer_model") != PRODUCTION_ANSWER_MODEL for item in answers):
+ raise EvaluationError(f"answer output did not use {PRODUCTION_ANSWER_MODEL}")
+ labels = [official_judge_correct(item) for item in judged]
+ correct_count = sum(labels)
+ reference = _read_reference(args.data.resolve())
+ evaluation_audit = _build_evaluation_audit(
+ expected_qids,
+ reference,
+ answer_facing_evidence,
+ retrieval_debug,
+ judged,
+ )
+ evaluation_audit_path = answer_dir / "evaluation_audit.jsonl"
+ _write_jsonl(evaluation_audit_path, evaluation_audit)
+ source_complete_count = sum(
+ bool(row["source_top24_coverage_complete"]) for row in evaluation_audit
+ )
+ final_complete_count = sum(
+ bool(row["final_evidence_coverage_complete"]) for row in evaluation_audit
+ )
+ expected_gold_sessions = sum(
+ int(row["source_top24_coverage"]["expected_session_count"])
+ for row in evaluation_audit
+ )
+ source_covered_sessions = sum(
+ int(row["source_top24_coverage"]["covered_session_count"])
+ for row in evaluation_audit
+ )
+ final_covered_sessions = sum(
+ int(row["final_evidence_coverage"]["covered_session_count"])
+ for row in evaluation_audit
+ )
+ source_top24_hit_rate = (
+ source_covered_sessions / expected_gold_sessions
+ if expected_gold_sessions
+ else 0.0
+ )
+ final_evidence_hit_rate = (
+ final_covered_sessions / expected_gold_sessions
+ if expected_gold_sessions
+ else 0.0
+ )
+ accuracy = correct_count / len(judged) if judged else 0.0
+ evaluation_quality_gate_passed = (
+ source_top24_hit_rate == 1.0
+ and final_evidence_hit_rate >= 0.9
+ and accuracy >= 0.9
+ )
+ regression_gate_report: dict[str, Any] = {
+ "schema_version": "tmcra.v4.regression-gate-reference.1",
+ "status": "not_run",
+ "promotion_eligible": False,
+ "reason": "frozen baseline judge was not supplied",
+ }
+ if baseline_judge is not None:
+ regression_evidence_path = answer_dir / "source_top24_regression_evidence.jsonl"
+ _write_jsonl(
+ regression_evidence_path,
+ [
+ {
+ "qid": row["qid"],
+ "question_id": row["question_id"],
+ "selected_session_ids": row["source_top24_session_ids"],
+ }
+ for row in evaluation_audit
+ ],
+ )
+ regression_gate_report = evaluate_regression_gate(
+ baseline_judge,
+ judge_path,
+ regression_evidence_path,
+ args.data.resolve(),
+ cases_path=Path(
+ getattr(args, "regression_cases", DEFAULT_REGRESSION_CASES)
+ ).resolve(),
+ min_candidate_accuracy=None,
+ max_regressions=0,
+ min_improvements=0,
+ )
+ (answer_dir / "v4_regression_gate_report.json").write_text(
+ json.dumps(regression_gate_report, indent=2, sort_keys=True) + "\n",
+ encoding="utf-8",
+ )
+ regression_gate_passed = bool(
+ regression_gate_report.get("promotion_eligible")
+ )
+ promotion_gate_passed = (
+ evaluation_quality_gate_passed and regression_gate_passed
+ )
+ summary = {
+ "schema_version": "tmcra.v4.evaluation.3",
+ "status": "complete",
+ "route_policy_schema": POLICY_SCHEMA,
+ "lane": PRODUCTION_LANE,
+ "answer_protocol": PRODUCTION_ANSWER_PROTOCOL,
+ "promotion_eligible": promotion_gate_passed,
+ "route_report": route_report,
+ "retrieval_tag": args.retrieval_tag,
+ "evaluated_count": len(judged),
+ "answer_model": PRODUCTION_ANSWER_MODEL,
+ "answer_attempt_limit": int(args.answer_attempts),
+ "judge_retry_limit": 0,
+ "judge_label_field": "autoeval_label.label",
+ "correct_count": correct_count,
+ "accuracy": round(accuracy, 6),
+ "evaluation_quality_gate_passed": evaluation_quality_gate_passed,
+ "promotion_gate": {
+ "passed": promotion_gate_passed,
+ "evaluation_quality_gate_passed": evaluation_quality_gate_passed,
+ "zero_api_regression_gate_passed": regression_gate_passed,
+ "source_top24_gold_session_hit_rate_min": 1.0,
+ "final_evidence_gold_session_hit_rate_min": 0.9,
+ "official_accuracy_min": 0.9,
+ "max_regressions": 0,
+ },
+ "regression_gate": regression_gate_report,
+ "evaluation_audit": {
+ "path": str(evaluation_audit_path),
+ "row_count": len(evaluation_audit),
+ "reference_row_count": len(reference),
+ "reference_is_superset": len(reference) >= len(expected_qids)
+ and set(expected_qids).issubset({row["qid"] for row in reference}),
+ "source_top24_coverage_complete_count": source_complete_count,
+ "source_top24_coverage_complete_rate": round(
+ source_complete_count / len(evaluation_audit), 6
+ )
+ if evaluation_audit
+ else 0.0,
+ "source_top24_gold_session_hit_rate": round(
+ source_top24_hit_rate, 6
+ ),
+ "final_evidence_coverage_complete_count": final_complete_count,
+ "final_evidence_coverage_complete_rate": round(
+ final_complete_count / len(evaluation_audit), 6
+ )
+ if evaluation_audit
+ else 0.0,
+ "final_evidence_gold_session_hit_rate": round(
+ final_evidence_hit_rate, 6
+ ),
+ "session_coverage_complete_count": final_complete_count,
+ "session_coverage_complete_rate": round(
+ final_complete_count / len(evaluation_audit), 6
+ )
+ if evaluation_audit
+ else 0.0,
+ },
+ }
+ (answer_dir / "v4_evaluation_report.json").write_text(
+ json.dumps(summary, indent=2, sort_keys=True) + "\n", encoding="utf-8"
+ )
+ (answer_dir / "ANSWER_COMPLETE").write_text(_now() + "\n", encoding="utf-8")
+ _archive_failure_marker(run_dir, args.retrieval_tag)
+ return summary
+ except Exception as exc:
+ _record_failure(run_dir, args.retrieval_tag, exc)
+ raise
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description="Evaluate a completed TMCRA V4 retrieval run")
+ parser.add_argument("--run-dir", type=Path, required=True)
+ parser.add_argument("--retrieval-tag", default="retrieval_1")
+ parser.add_argument("--resume", action="store_true")
+ parser.add_argument("--answer-env", type=Path, default=DEFAULT_ANSWER_ENV)
+ parser.add_argument("--answer-runner", type=Path, default=DEFAULT_ANSWER_RUNNER)
+ parser.add_argument("--harness", type=Path, default=DEFAULT_HARNESS)
+ parser.add_argument("--judge", type=Path, default=DEFAULT_JUDGE)
+ parser.add_argument("--data", type=Path, default=DEFAULT_DATA)
+ parser.add_argument("--qid-list", type=Path)
+ parser.add_argument("--answer-workers", type=int, default=1)
+ parser.add_argument("--answer-attempts", type=int, choices=(1, 2, 3), default=1)
+ parser.add_argument("--answer-window-limit", type=int, default=8)
+ parser.add_argument("--baseline-judge", type=Path)
+ parser.add_argument(
+ "--regression-cases", type=Path, default=DEFAULT_REGRESSION_CASES
+ )
+ args = parser.parse_args()
+ summary = evaluate(args)
+ print(json.dumps(summary, sort_keys=True))
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/legacy/run_tmcra_v4_gpt54_answers.py b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/run_tmcra_v4_gpt54_answers.py
new file mode 100644
index 0000000000000000000000000000000000000000..d4ab1a92f62478cfbdc49ba214b7695118cee2a2
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/run_tmcra_v4_gpt54_answers.py
@@ -0,0 +1,847 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import hashlib
+import importlib.util
+import json
+import os
+import sys
+import threading
+import time
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from pathlib import Path
+from typing import Any, Mapping
+
+from tmcra_v4_evidence_operations import (
+ ANSWER_SCHEMA,
+ PACKET_SCHEMA,
+ EvidenceOperationError,
+ validate_evidence_bound_answer,
+)
+from tmcra_v4_semantic_evidence import (
+ SEMANTIC_ANSWER_SCHEMA,
+ SEMANTIC_PACKET_SCHEMA,
+ SemanticEvidenceError,
+ normalize_semantic_answer_output,
+ semantic_packet_to_advisory_packet,
+ validate_semantic_answer,
+)
+from tmcra_v4_route_policy import (
+ PRODUCTION_ANSWER_PROTOCOL,
+ RoutePolicyError,
+ SHADOW_ANSWER_PROTOCOL,
+ validate_production_answers,
+ validate_production_evidence,
+)
+
+
+FORBIDDEN_EVALUATION_FIELDS = {
+ "answer",
+ "gold_answer",
+ "answer_session_ids",
+ "labels",
+ "supervision",
+}
+USER_MEMORY_CUE_LIMIT = 32
+USER_MEMORY_CUE_TEXT_LIMIT = 640
+
+
+def clean_text(value: Any) -> str:
+ return " ".join(str(value or "").split())
+
+
+def fixed_answer_environment(answer_window_limit: int) -> dict[str, str]:
+ return {
+ "TMCRA_ANSWER_WINDOW_PLANNER_MODE": "off",
+ "TMCRA_ANSWER_WINDOW_PLANNER_SEMANTIC_MODE": "off",
+ "TMCRA_LLM_EVIDENCE_SELECTOR_MODE": "off",
+ "TMCRA_EVIDENCE_UNIT_PLANNER_MODE": "off",
+ "TMCRA_UNIFIED_OPERATION_PLANNER_MODE": "off",
+ "TMCRA_LLM_CHANNEL_PLANNER_MODE": "off",
+ "TMCRA_FINAL_ANSWER_SURFACE_CHANNEL_MODE": "off",
+ "TMCRA_ANSWER_REASONING_ORGANIZER": "off",
+ "TMCRA_LLM_CACHE_MODE": "off",
+ "TMCRA_LME_FORCE_ANSWER": "1",
+ "TMCRA_HTTP_JSON_MAX_ATTEMPTS": "1",
+ "TMCRA_ANSWER_EVIDENCE_WINDOW_LIMIT": str(max(1, answer_window_limit)),
+ }
+
+
+def read_jsonl(path: Path) -> list[dict[str, Any]]:
+ rows: list[dict[str, Any]] = []
+ with path.open("r", encoding="utf-8", errors="strict") as handle:
+ for line_no, line in enumerate(handle, start=1):
+ if not line.strip():
+ continue
+ value = json.loads(line)
+ if not isinstance(value, dict):
+ raise RuntimeError(f"row is not an object at {path}:{line_no}")
+ rows.append(value)
+ if not rows:
+ raise RuntimeError(f"no rows: {path}")
+ return rows
+
+
+def evidence_sha256(row: Mapping[str, Any]) -> str:
+ return hashlib.sha256(
+ json.dumps(
+ dict(row), ensure_ascii=False, sort_keys=True, separators=(",", ":")
+ ).encode("utf-8")
+ ).hexdigest()
+
+
+def atomic_write_jsonl(path: Path, rows: list[Mapping[str, Any]]) -> None:
+ temporary = path.with_name(f".{path.name}.tmp.{os.getpid()}.{time.time_ns()}")
+ temporary.write_text(
+ "".join(json.dumps(dict(row), ensure_ascii=False, sort_keys=True) + "\n" for row in rows),
+ encoding="utf-8",
+ )
+ os.replace(temporary, path)
+
+
+def bind_existing_answers_to_evidence(
+ existing: dict[str, dict[str, Any]],
+ rows: list[dict[str, Any]],
+) -> tuple[dict[str, str], int]:
+ evidence_by_qid = {clean_text(row.get("question_id")): row for row in rows}
+ if not all(evidence_by_qid) or len(evidence_by_qid) != len(rows):
+ raise RuntimeError("answer evidence contains missing or duplicate question_id")
+ hashes = {qid: evidence_sha256(row) for qid, row in evidence_by_qid.items()}
+ backfills = 0
+ for qid, row in existing.items():
+ if qid not in evidence_by_qid:
+ continue
+ expected = evidence_by_qid[qid]
+ expected_hash = hashes[qid]
+ observed_hash = clean_text(row.get("evidence_sha256"))
+ if observed_hash and observed_hash != expected_hash:
+ raise RuntimeError(f"{qid}: existing answer is bound to different retrieval evidence")
+ if not observed_hash:
+ if (
+ clean_text(row.get("question")) != clean_text(expected.get("question"))
+ or list(row.get("selected_session_ids") or [])
+ != list(expected.get("selected_session_ids") or [])
+ or clean_text(row.get("answer_model")) != "gpt-5.4"
+ ):
+ raise RuntimeError(f"{qid}: legacy answer cannot be bound to current evidence")
+ row["evidence_sha256"] = expected_hash
+ backfills += 1
+ return hashes, backfills
+
+
+def ordered_completed_answers(
+ existing: Mapping[str, Mapping[str, Any]],
+ rows: list[Mapping[str, Any]],
+) -> list[dict[str, Any]]:
+ qids = [clean_text(row.get("question_id")) for row in rows]
+ if not all(qids) or len(qids) != len(set(qids)):
+ raise RuntimeError("answer evidence contains missing or duplicate question_id")
+ unexpected = sorted(set(existing) - set(qids))
+ if unexpected:
+ raise RuntimeError(
+ f"existing answer output contains qids outside the frozen request: {unexpected[:10]}"
+ )
+ return [dict(existing[qid]) for qid in qids if qid in existing]
+
+
+def load_harness(path: Path):
+ module_name = "tmcra_v3_answer_harness"
+ spec = importlib.util.spec_from_file_location(module_name, path)
+ if spec is None or spec.loader is None:
+ raise RuntimeError(f"cannot import answer harness: {path}")
+ module = importlib.util.module_from_spec(spec)
+ sys.modules[module_name] = module
+ spec.loader.exec_module(module)
+ return module
+
+
+def reject_evaluation_fields(rows: list[dict[str, Any]]) -> None:
+ for row in rows:
+ present = sorted(FORBIDDEN_EVALUATION_FIELDS.intersection(row))
+ if present:
+ qid = clean_text(row.get("question_id")) or ""
+ raise RuntimeError(
+ f"{qid}: answer input contains evaluation-only fields: {present}"
+ )
+
+
+def render_hierarchical_evidence(window: Mapping[str, Any]) -> str:
+ source = clean_text(window.get("text"))
+ if not source:
+ raise RuntimeError("evidence window has empty source text")
+ blocks: list[str] = []
+ for context in list(window.get("memory_contexts") or []):
+ if not isinstance(context, Mapping):
+ raise RuntimeError("memory_context must be an object")
+ claim = clean_text(context.get("claim_text"))
+ slot = clean_text(context.get("canonical_slot"))
+ role = clean_text(context.get("role"))
+ if not claim or not slot or not role:
+ raise RuntimeError("memory_context lacks role, canonical_slot, or claim_text")
+ blocks.append(f"[TMCRA slow memory role={role} slot={slot}]\n{claim}")
+ for attachment in list(window.get("attachments") or []):
+ if not isinstance(attachment, Mapping):
+ raise RuntimeError("memory attachment must be an object")
+ role = clean_text(attachment.get("role"))
+ slot = clean_text(attachment.get("canonical_slot"))
+ if role == "context_only":
+ summary = clean_text(attachment.get("summary"))
+ if not summary or not slot:
+ raise RuntimeError("slow context attachment lacks slot or summary")
+ blocks.append(
+ f"[TMCRA slow memory role=context_only slot={slot}]\n{summary}"
+ )
+ elif role in {"fast_context", "override"}:
+ override_text = clean_text(attachment.get("text"))
+ if not override_text or not slot:
+ raise RuntimeError("fast memory attachment lacks slot or text")
+ if role == "override":
+ blocks.append(
+ "[TMCRA fast memory role=override "
+ f"precedence=newer_fast_evidence slot={slot}]\n"
+ "This newer fast-layer evidence overrides conflicting slow-memory "
+ f"claims.\n{override_text}"
+ )
+ else:
+ blocks.append(
+ f"[TMCRA fast memory role=context slot={slot}]\n{override_text}"
+ )
+ else:
+ raise RuntimeError(f"unsupported memory attachment role: {role!r}")
+ blocks.append("[TMCRA immutable source evidence]\n" + source)
+ return "\n\n".join(blocks)
+
+
+def build_user_memory_cues(
+ evidence: list[Mapping[str, Any]],
+) -> list[dict[str, Any]]:
+ """Build a compact index of durable claims and user-authored evidence.
+
+ The index only points into the immutable Source reservoir. Round-robin
+ emission prevents an early, verbose source from hiding cues in later
+ retrieval windows.
+ """
+ cues_by_source: list[list[dict[str, Any]]] = []
+ for item in evidence:
+ evidence_id = clean_text(item.get("evidence_id"))
+ source_record_id = clean_text(item.get("source_record_id"))
+ source_cues: list[dict[str, Any]] = []
+
+ def add_cue(
+ kind: str,
+ text: Any,
+ *,
+ record_id: str = source_record_id,
+ historical_date: Any = None,
+ timestamp: Any = None,
+ ) -> None:
+ normalized = clean_text(text)
+ if not normalized:
+ return
+ source_cues.append(
+ {
+ "evidence_id": evidence_id,
+ "kind": kind,
+ "source_record_id": record_id,
+ "historical_date": clean_text(
+ historical_date
+ if historical_date is not None
+ else item.get("historical_date")
+ ),
+ "timestamp": clean_text(
+ timestamp if timestamp is not None else item.get("timestamp")
+ ),
+ "text": normalized[:USER_MEMORY_CUE_TEXT_LIMIT],
+ "text_truncated": len(normalized) > USER_MEMORY_CUE_TEXT_LIMIT,
+ }
+ )
+
+ for context in list(item.get("memory_contexts") or []):
+ if isinstance(context, Mapping):
+ add_cue(
+ "slow_claim",
+ context.get("claim_text") or context.get("capsule_summary"),
+ )
+ for attachment in list(item.get("attachments") or []):
+ if isinstance(attachment, Mapping):
+ add_cue(
+ "fast_memory",
+ attachment.get("text") or attachment.get("summary"),
+ )
+ if clean_text(item.get("message_role")).lower() == "user":
+ add_cue("user_source", item.get("text"))
+ for context in list(item.get("source_group_context") or []):
+ if not isinstance(context, Mapping):
+ continue
+ role = clean_text(
+ context.get("message_role") or context.get("role")
+ ).lower()
+ if role == "user":
+ add_cue(
+ "user_context",
+ context.get("text"),
+ record_id=clean_text(context.get("source_record_id")),
+ historical_date=context.get("historical_date"),
+ timestamp=context.get("timestamp"),
+ )
+ cues_by_source.append(source_cues)
+
+ output: list[dict[str, Any]] = []
+ seen_text: set[str] = set()
+ max_source_cues = max((len(items) for items in cues_by_source), default=0)
+ for cue_index in range(max_source_cues):
+ for source_cues in cues_by_source:
+ if cue_index >= len(source_cues):
+ continue
+ cue = source_cues[cue_index]
+ fingerprint = clean_text(cue["text"]).casefold()
+ if fingerprint in seen_text:
+ continue
+ seen_text.add(fingerprint)
+ output.append({"cue_order": len(output) + 1, **cue})
+ if len(output) >= USER_MEMORY_CUE_LIMIT:
+ return output
+ return output
+
+
+EVIDENCE_OPERATION_ANSWER_PROMPT = f"""You answer one memory question from a compiled, immutable evidence packet.
+Return exactly one JSON object and no prose using this schema:
+{{"schema_version":"{ANSWER_SCHEMA}","answerability":"sufficient|insufficient","claims":[{{"claim_id":"C01","text":"one factual claim","origin":"memory_fact|memory_inference|memory_derived|query_context|model_knowledge","support_ids":["E01"],"computation_ids":["O01"]}}],"missing_requirements":["R01"],"answer":"short complete answer"}}
+
+Read task_intent.output_origin before answering. task_intent and planner_source_groups are routing hints, not truth or answerability decisions. requirement_catalog lists the response requirement IDs but deliberately carries no planner missing/satisfied judgment. Determine support or absence yourself from the complete Source reservoir. user_memory_cues is a deterministic navigation index of slow-memory claims and user-authored Source or source-group context; it is not separate proof, so inspect the referenced Source before binding a remembered claim. For memory_conditioned_generation, first recover the user's grounded preferences or constraints and then generate the requested recommendation or advice; a ready-made historical answer is not required. Carry forward relevant prior choices, owned tools, resources, genres, constraints, and already-adopted solutions instead of defaulting to generic advice. When several relevant cues differ in specificity, preserve the narrower grounded detail rather than replacing it with a broad category. The final answer itself must explicitly show how at least one specific remembered constraint shaped the response; do not leave personalization only inside the claims array. Do not declare the task insufficient merely because the exact requested recommendation was not historical when grounded memory can personalize a new answer and model knowledge can fulfill the current target. Query context never requires Source evidence IDs. Model knowledge does not require Source evidence IDs when it is allowed for memory-conditioned generation. External-tool premises remain unresolved until a tool result exists.
+Source evidence is ordered by frozen retrieval priority and then by source order within each selected session; source_order is not an event date. Each Source item carries its immutable historical_date and timestamp when available. Resolve words such as today and yesterday against that Source item's timestamp, never against the question date. Inspect source_group_context as neighboring source context and use each neighbor's own timestamp. Before answering, internally build an exact evidence ledger for the requested entity: list every matching event or item, its event date or stated relative time, state (actual, planned, hypothetical, or mentioned), and Source ID. Deduplicate repeated mentions of the same event or item before counting. For latest/earliest/order/duration questions, compare event times grounded in Source text plus its timestamp, not source_order or database session_index. If a when, before, or after clause names another remembered event, treat that event as a candidate temporal endpoint and compare both Source dates before using the question date. Use the question date only when the wording actually asks for distance from the query time.
+Each Source item may include memory_contexts and attachments. A slow_context is the slow graph's durable interpretation and must retain its capsule, claim, and provenance identity. A fast_context is an atomic current-memory interpretation. An attachment marked override with precedence=newer_fast_evidence has controller-verified precedence over a conflicting slow claim in the same canonical slot. These graph views organize meaning but do not replace Source proof: support final remembered claims with the Source evidence ID carrying the context, and inspect its immutable text before relying on it.
+Use only supplied Source evidence IDs and verified computation IDs. memory_fact claims require direct support_ids. memory_inference claims require support_ids for every Source premise and represent source-grounded comparison or synthesis that has no verified operation result. memory_derived claims require computation_ids. query_context claims may have empty binding lists in every task when they state only information supplied by the current question, such as the question date or arithmetic derived solely from it. Only memory_conditioned_generation may use model_knowledge claims with empty binding lists; label their origin explicitly, and include at least one memory_fact, memory_inference, or memory_derived claim showing how remembered constraints shaped the answer. Never label remembered information as query_context or model_knowledge. planner_operation_candidates expose no result because their operand selection has not been semantically certified; independently inspect their Source supports. lexical_anchor_ids are deterministic search hints, not proof. Never transfer values between different entities, people, places, events, or activities. Preserve requested temporal order, distinct values, totals, and update precedence.
+For preferences, use the specific historical experiences in Source evidence. Treat named entities and activities exactly: a shorter, broader, or related term is not the requested multiword entity unless the source explicitly says they are equivalent. For false premises or entity mismatch, explicitly state that the requested target is not established, then optionally state related counterevidence. For count/list questions, enumerate distinct matching items before producing the total. For changing facts, apply latest update precedence using event time and source-group context. A sufficient answer must cover every requirement and have no missing_requirements. An insufficient answer must name supplied requirement IDs that remain unresolved after inspecting all Source evidence. Never use hidden supervision or outside facts for remembered claims."""
+
+
+SEMANTIC_EVIDENCE_ANSWER_PROMPT = f"""Execute one verified memory task contract.
+Return exactly one JSON object and no prose using this schema:
+{{"schema_version":"{SEMANTIC_ANSWER_SCHEMA}","response_status":"answered|partial|answered_with_context|requires_tool|clarification|not_answerable","claims":[{{"claim_id":"C01","text":"one final claim","origin":"memory_fact|memory_derived|model_knowledge|tool_result","premise_ids":["P01"],"source_claim_ids":["M01"],"computation_ids":["O01"]}}],"unresolved_premise_ids":[],"answer":"concise complete response"}}
+
+The task contract defines the requested target and output shape. The answerability certificate is authoritative; do not redo retrieval, invent missing memory, or change its answer policy.
+For memory_fact claims, use only Source claim IDs bound to the cited premises. For memory_derived claims, use only completed computation IDs bound to the cited premises. Preserve entity, time, polarity, modality, cardinality, ordering, and update state.
+When the policy is answer_with_memory_context, treat remembered claims as constraints or background. You may produce model_knowledge claims needed to fulfill the current request, but never describe them as remembered facts and never attach Source claim IDs to them.
+When the policy is answer_partial, answer only the supported portion, explicitly state the unresolved limitation, use response_status partial, and preserve the certificate's unresolved premise IDs. For a memory-conditioned task, the partial answer must explicitly use at least one bound memory claim rather than falling back to a generic response. Never turn a partial count or other incomplete derived aggregate into a final total.
+When the policy is call_tool, clarify, or abstain, use the matching response_status and explain the unresolved need without fabricating an answer. unresolved_premise_ids must exactly match the certificate.
+Do not cite raw evidence IDs directly. Do not use benchmark labels or hidden supervision."""
+
+
+def render_compiled_packet(packet: Mapping[str, Any]) -> dict[str, Any]:
+ if packet.get("schema_version") != PACKET_SCHEMA:
+ raise RuntimeError("compiled evidence packet schema is invalid")
+ evidence = list(packet.get("raw_evidence_reservoir") or [])
+ evidence_ids = {clean_text(item.get("evidence_id")) for item in evidence}
+ if not evidence or not all(evidence_ids):
+ raise RuntimeError("compiled evidence packet has no source reservoir")
+ bundles = []
+ for bundle in list(packet.get("evidence_bundles") or []):
+ ids = list(bundle.get("evidence_ids") or [])
+ if any(item not in evidence_ids for item in ids):
+ raise RuntimeError("compiled evidence bundle cites unknown source")
+ bundles.append(
+ {
+ "bundle_id": bundle.get("bundle_id"),
+ "role": bundle.get("role"),
+ "evidence_ids": ids,
+ }
+ )
+ question_contract = dict(packet.get("question_contract") or {})
+ task_contract = packet.get("task_contract")
+ if not isinstance(task_contract, Mapping):
+ task_contract = question_contract.get("task_contract")
+ question_contract.pop("task_contract", None)
+ requirements = list(packet.get("requirement_coverage") or [])
+ premise_by_id = {
+ clean_text(item.get("premise_id")): item
+ for item in (
+ task_contract.get("premises") or []
+ if isinstance(task_contract, Mapping)
+ else []
+ )
+ if isinstance(item, Mapping)
+ }
+ output_origin = (
+ clean_text(task_contract.get("output_origin"))
+ if isinstance(task_contract, Mapping)
+ else clean_text(question_contract.get("output_origin"))
+ )
+ requirement_catalog = []
+ bound_constraints = []
+ for requirement in requirements:
+ requirement_id = clean_text(requirement.get("requirement_id"))
+ evidence_ids_for_requirement = [
+ item
+ for item in list(requirement.get("evidence_ids") or [])
+ if item in evidence_ids
+ ]
+ premise = premise_by_id.get(requirement_id) or {}
+ description = clean_text(requirement.get("description"))
+ source_kind = clean_text(premise.get("source")) or "memory"
+ catalog_item = {
+ "requirement_id": requirement_id,
+ "description": (
+ "remembered personalization constraints; inspect all Source evidence"
+ if output_origin == "memory_conditioned_generation"
+ and source_kind == "memory"
+ and not evidence_ids_for_requirement
+ else description
+ ),
+ "source_kind": source_kind,
+ }
+ if evidence_ids_for_requirement:
+ catalog_item["candidate_source_ids"] = evidence_ids_for_requirement
+ requirement_catalog.append(catalog_item)
+ if evidence_ids_for_requirement and isinstance(premise, Mapping):
+ bound_constraints.append(
+ {
+ "premise_id": requirement_id,
+ "description": clean_text(premise.get("description")) or description,
+ "source_kind": source_kind,
+ "grounded_constraints": list(premise.get("grounded_constraints") or []),
+ "candidate_source_ids": evidence_ids_for_requirement,
+ }
+ )
+ operation_results = list(packet.get("operation_results") or [])
+ verified_operations = [
+ dict(item)
+ for item in operation_results
+ if item.get("status") == "completed"
+ and item.get("answer_authoritative") is True
+ ]
+ planner_operation_candidates = [
+ {
+ "operation_id": item.get("operation_id"),
+ "operation_type": item.get("operation_type"),
+ "support_ids": list(item.get("support_ids") or []),
+ "semantic_status": "uncertified_operand_selection",
+ }
+ for item in operation_results
+ if item.get("status") == "completed"
+ and item.get("answer_authoritative") is not True
+ ]
+ task_intent: dict[str, Any] = {"output_origin": output_origin}
+ if isinstance(task_contract, Mapping):
+ task_intent["target"] = dict(task_contract.get("target") or {})
+ task_intent["output"] = dict(task_contract.get("output") or {})
+ task_intent["bound_constraints"] = bound_constraints
+ rendered = {
+ "answer_view_schema": "tmcra.v4.answer-evidence-view.5",
+ "question_contract": question_contract,
+ "task_intent": task_intent,
+ "user_memory_cues": build_user_memory_cues(evidence),
+ "requirement_catalog": requirement_catalog,
+ "verified_operation_results": verified_operations,
+ "planner_operation_candidates": planner_operation_candidates,
+ "planner_source_groups": bundles,
+ "lexical_anchor_ids": list(packet.get("lexical_anchor_ids") or []),
+ "source_evidence": [
+ {
+ "source_order": index,
+ "evidence_id": item.get("evidence_id"),
+ "session_id": item.get("session_id"),
+ "session_index": item.get("session_index"),
+ "parent_chunk_index": item.get("parent_chunk_index"),
+ "historical_date": item.get("historical_date"),
+ "timestamp": item.get("timestamp"),
+ "message_role": item.get("message_role"),
+ "question_overlap_terms": item.get("question_overlap_terms") or [],
+ "question_overlap_score": int(item.get("question_overlap_score", 0)),
+ "text": item.get("text"),
+ "source_group_id": item.get("source_group_id"),
+ "source_group_context": list(item.get("source_group_context") or []),
+ "memory_contexts": list(item.get("memory_contexts") or []),
+ "attachments": list(item.get("attachments") or []),
+ "provenance": list(item.get("provenance") or []),
+ }
+ for index, item in enumerate(evidence, start=1)
+ ],
+ }
+ if isinstance(packet.get("semantic_advisory"), Mapping):
+ rendered["semantic_advisory"] = dict(packet["semantic_advisory"])
+ return rendered
+
+
+def parse_operation_bound_answer(raw: str, packet: Mapping[str, Any]) -> dict[str, Any]:
+ try:
+ value = json.loads(raw)
+ except json.JSONDecodeError as exc:
+ raise RuntimeError(f"operation-bound answer is not strict JSON: {exc}") from exc
+ try:
+ return validate_evidence_bound_answer(value, packet)
+ except EvidenceOperationError as exc:
+ raise RuntimeError(f"operation-bound answer failed validation: {exc}") from exc
+
+
+def render_semantic_packet(packet: Mapping[str, Any]) -> dict[str, Any]:
+ if packet.get("schema_version") != SEMANTIC_PACKET_SCHEMA:
+ raise RuntimeError("semantic evidence packet schema is invalid")
+ contract = packet.get("task_contract")
+ certificate = packet.get("answerability_certificate")
+ if not isinstance(contract, Mapping) or not isinstance(certificate, Mapping):
+ raise RuntimeError("semantic evidence packet lacks contract or certificate")
+ premise_by_id = {
+ clean_text(item.get("premise_id")): item
+ for item in contract.get("premises") or []
+ if isinstance(item, Mapping)
+ }
+ claims_by_id = {
+ clean_text(item.get("claim_id")): item
+ for item in packet.get("claim_ledger") or []
+ if isinstance(item, Mapping)
+ }
+ evidence_by_id = {
+ clean_text(item.get("evidence_id")): item
+ for item in packet.get("raw_evidence_reservoir") or []
+ if isinstance(item, Mapping)
+ }
+ operation_by_id = {
+ clean_text(item.get("operation_id")): item
+ for item in (packet.get("resolution_program") or {}).get("operation_results") or []
+ if isinstance(item, Mapping)
+ }
+ resolved_premises = []
+ for binding in packet.get("premise_bindings") or []:
+ if not isinstance(binding, Mapping):
+ raise RuntimeError("semantic premise binding must be an object")
+ premise_id = clean_text(binding.get("premise_id"))
+ if premise_id not in premise_by_id:
+ raise RuntimeError("semantic premise binding cites an unknown premise")
+ claim_ids = list(binding.get("claim_ids") or [])
+ evidence_ids = list(binding.get("evidence_ids") or [])
+ operation_ids = list(binding.get("operation_ids") or [])
+ if any(item not in claims_by_id for item in claim_ids):
+ raise RuntimeError("semantic premise binding cites an unknown Source claim")
+ if any(item not in evidence_by_id for item in evidence_ids):
+ raise RuntimeError("semantic premise binding cites unknown evidence")
+ if any(item not in operation_by_id for item in operation_ids):
+ raise RuntimeError("semantic premise binding cites an unknown operation")
+ resolved_premises.append(
+ {
+ "premise": premise_by_id[premise_id],
+ "coverage": binding.get("coverage"),
+ "relation": binding.get("relation"),
+ "source_claims": [claims_by_id[item] for item in claim_ids],
+ "operation_results": [operation_by_id[item] for item in operation_ids],
+ "source_evidence": [
+ {
+ "evidence_id": item,
+ "session_id": evidence_by_id[item].get("session_id"),
+ "session_index": evidence_by_id[item].get("session_index"),
+ "text": evidence_by_id[item].get("text"),
+ }
+ for item in evidence_ids
+ ],
+ }
+ )
+ return {
+ "query_context": dict(packet.get("query_context") or {}),
+ "task_contract": dict(contract),
+ "answerability_certificate": dict(certificate),
+ "resolved_premises": resolved_premises,
+ "non_memory_premises": [
+ premise
+ for premise in contract.get("premises") or []
+ if isinstance(premise, Mapping) and premise.get("source") != "memory"
+ ],
+ "target_operation_results": [
+ operation_by_id[item]
+ for item in certificate.get("target_operation_ids") or []
+ if item in operation_by_id
+ ],
+ }
+
+
+def parse_semantic_bound_answer(raw: str, packet: Mapping[str, Any]) -> dict[str, Any]:
+ try:
+ value = json.loads(raw)
+ except json.JSONDecodeError as exc:
+ raise RuntimeError(f"semantic-bound answer is not strict JSON: {exc}") from exc
+ try:
+ return validate_semantic_answer(normalize_semantic_answer_output(value, packet), packet)
+ except SemanticEvidenceError as exc:
+ raise RuntimeError(f"semantic-bound answer failed validation: {exc}") from exc
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--evidence", required=True)
+ parser.add_argument("--harness", required=True)
+ parser.add_argument("--out-dir", required=True)
+ parser.add_argument("--workers", type=int, default=12)
+ parser.add_argument("--attempts", type=int, default=3)
+ parser.add_argument("--answer-window-limit", type=int, default=8)
+ parser.add_argument("--limit", type=int, default=0)
+ parser.add_argument("--qid-list", default="")
+ parser.add_argument("--lane", choices=("production", "shadow"), default="production")
+ args = parser.parse_args()
+ answer_model = clean_text(os.getenv("TMCRA_ANSWER_MODEL"))
+ answer_base_url = clean_text(os.getenv("TMCRA_ANSWER_BASE_URL"))
+ answer_api_key = clean_text(os.getenv("TMCRA_ANSWER_API_KEY"))
+ if answer_model != "gpt-5.4":
+ raise RuntimeError(f"fixed answer model must be exactly gpt-5.4, got {answer_model!r}")
+ if not answer_base_url or not answer_api_key:
+ raise RuntimeError("TMCRA_ANSWER_BASE_URL and TMCRA_ANSWER_API_KEY are required")
+ fixed_answer_layer = fixed_answer_environment(args.answer_window_limit)
+ os.environ.update(fixed_answer_layer)
+ rows = read_jsonl(Path(args.evidence))
+ if args.qid_list:
+ requested = [
+ clean_text(value)
+ for value in Path(args.qid_list).read_text(encoding="utf-8").splitlines()
+ if clean_text(value)
+ ]
+ by_qid = {clean_text(row.get("question_id")): row for row in rows}
+ missing = [qid for qid in requested if qid not in by_qid]
+ if missing:
+ raise RuntimeError(f"requested qids are missing from evidence: {missing[:10]}")
+ rows = [by_qid[qid] for qid in requested]
+ if args.limit > 0:
+ rows = rows[: args.limit]
+ if not rows:
+ raise RuntimeError("no answer rows selected")
+ reject_evaluation_fields(rows)
+ if args.lane == "production":
+ try:
+ validate_production_evidence(rows)
+ except RoutePolicyError as exc:
+ raise RuntimeError(
+ f"production route failed before GPT-5.4 calls: {exc}"
+ ) from exc
+ harness = load_harness(Path(args.harness))
+ configured_base_url, configured_model, configured_api_key = harness.answer_llm_config()
+ if clean_text(configured_model) != "gpt-5.4":
+ raise RuntimeError(f"harness resolved unexpected model: {configured_model!r}")
+ if not clean_text(configured_base_url) or not clean_text(configured_api_key):
+ raise RuntimeError("harness did not resolve the required GPT5.4 endpoint")
+
+ # The legacy harness contains optional answer-side repair hooks. Keep its
+ # primary prompt, but make one logical GPT call the enforced contract.
+ harness.complete_profile_answer_from_evidence = lambda _question, answer, _windows: answer
+ call_state = threading.local()
+ original_chat_completion = harness.chat_completion
+
+ def counted_chat_completion(*call_args: Any, **call_kwargs: Any) -> str:
+ call_state.count = int(getattr(call_state, "count", 0)) + 1
+ return original_chat_completion(*call_args, **call_kwargs)
+
+ harness.chat_completion = counted_chat_completion
+ out_dir = Path(args.out_dir)
+ out_dir.mkdir(parents=True, exist_ok=True)
+ raw_attempt_dir = out_dir / "raw_attempts"
+ raw_attempt_dir.mkdir(exist_ok=True)
+ answers_path = out_dir / "answers.jsonl"
+ existing: dict[str, dict[str, Any]] = {}
+ if answers_path.exists():
+ for row in read_jsonl(answers_path):
+ qid = clean_text(row.get("question_id"))
+ if qid:
+ if qid in existing:
+ raise RuntimeError(f"duplicate existing answer row: {qid}")
+ existing[qid] = row
+ if existing and args.lane == "production":
+ try:
+ validate_production_answers(list(existing.values()))
+ except RoutePolicyError as exc:
+ raise RuntimeError(
+ "existing answers use an incompatible production protocol; "
+ "start a new answer output directory"
+ ) from exc
+ evidence_hashes, legacy_evidence_hash_backfills = bind_existing_answers_to_evidence(
+ existing, rows
+ )
+ lock = threading.Lock()
+
+ def answer_one(row: Mapping[str, Any]) -> dict[str, Any]:
+ qid = clean_text(row.get("question_id"))
+ question = clean_text(row.get("question"))
+ question_date = clean_text(row.get("question_date"))
+ runtime_question = f"{question}\nQuestion date: {question_date}" if question_date else question
+ semantic_packet = row.get("semantic_evidence_packet")
+ operation_packet = row.get("compiled_evidence_packet")
+ if args.lane == "production":
+ if isinstance(semantic_packet, Mapping):
+ raise RuntimeError(f"{qid}: semantic packets are forbidden in the production lane")
+ if not isinstance(operation_packet, Mapping):
+ raise RuntimeError(f"{qid}: production lane requires a compiled evidence packet")
+ packet = operation_packet
+ rendered_packet = render_compiled_packet(packet)
+ answer_prompt = EVIDENCE_OPERATION_ANSWER_PROMPT
+ answer_parser = parse_operation_bound_answer
+ answer_protocol = PRODUCTION_ANSWER_PROTOCOL
+ else:
+ if not isinstance(semantic_packet, Mapping) or isinstance(operation_packet, Mapping):
+ raise RuntimeError(f"{qid}: shadow lane requires only a semantic evidence packet")
+ packet = semantic_packet_to_advisory_packet(semantic_packet)
+ rendered_packet = render_compiled_packet(packet)
+ answer_prompt = EVIDENCE_OPERATION_ANSWER_PROMPT
+ answer_parser = parse_operation_bound_answer
+ answer_protocol = SHADOW_ANSWER_PROTOCOL
+ messages = [
+ {"role": "system", "content": answer_prompt},
+ {
+ "role": "user",
+ "content": json.dumps(rendered_packet, ensure_ascii=False, separators=(",", ":")),
+ },
+ ]
+ started = time.time()
+ last_error: Exception | None = None
+ answer = ""
+ evidence_binding: dict[str, Any] = {}
+ attempts_used = 0
+ logical_answer_calls = 0
+ validation_error = ""
+ for attempt in range(1, max(1, args.attempts) + 1):
+ attempts_used = attempt
+ call_state.count = 0
+ try:
+ attempt_path = raw_attempt_dir / f"{qid}.attempt_{attempt}.json"
+ if attempt_path.is_file():
+ persisted = json.loads(attempt_path.read_text(encoding="utf-8"))
+ if persisted.get("question_id") != qid or persisted.get("evidence_sha256") != evidence_hashes[qid]:
+ raise RuntimeError(f"{qid}: persisted raw answer identity mismatch")
+ raw_answer = str(persisted["raw_answer"])
+ attempt_call_count = 0
+ else:
+ attempt_messages = list(messages)
+ if validation_error:
+ attempt_messages.append(
+ {
+ "role": "user",
+ "content": (
+ "The previous JSON failed the deterministic validator: "
+ f"{validation_error}. Correct only the JSON structure or evidence bindings. "
+ "Use the same compiled packet and return the required schema."
+ ),
+ }
+ )
+ raw_answer = harness.chat_completion(
+ configured_base_url,
+ configured_model,
+ attempt_messages,
+ max_tokens=int(os.getenv("TMCRA_ANSWER_MAX_TOKENS", "1536")),
+ temperature=0.0,
+ api_key=configured_api_key,
+ )
+ attempt_call_count = int(getattr(call_state, "count", 0))
+ if attempt_call_count != 1:
+ raise RuntimeError(
+ f"{qid}: answer attempt made {attempt_call_count} logical GPT calls; expected exactly 1"
+ )
+ temporary = attempt_path.with_name(f".{attempt_path.name}.tmp.{os.getpid()}.{time.time_ns()}")
+ temporary.write_text(
+ json.dumps(
+ {
+ "question_id": qid,
+ "evidence_sha256": evidence_hashes[qid],
+ "attempt": attempt,
+ "raw_answer": raw_answer,
+ },
+ ensure_ascii=False,
+ sort_keys=True,
+ )
+ + "\n",
+ encoding="utf-8",
+ )
+ os.replace(temporary, attempt_path)
+ logical_answer_calls += attempt_call_count
+ if attempt_call_count not in {0, 1}:
+ raise RuntimeError(
+ f"{qid}: invalid physical call count for answer attempt"
+ )
+ evidence_binding = answer_parser(raw_answer, packet)
+ answer = evidence_binding["answer"]
+ break
+ except Exception as exc:
+ last_error = exc
+ validation_error = f"{exc.__class__.__name__}: {exc}"
+ if attempt >= max(1, args.attempts):
+ break
+ time.sleep(min(8.0, 1.5 * (2 ** (attempt - 1))))
+ if not answer:
+ assert last_error is not None
+ raise RuntimeError(
+ f"{qid}: GPT5.4 failed after {max(1, args.attempts)} same-model attempts: "
+ f"{last_error.__class__.__name__}:{last_error}"
+ )
+ return {
+ "question_id": qid,
+ "question_type": clean_text(row.get("question_type")),
+ "question": question,
+ "evidence_sha256": evidence_hashes[qid],
+ "hypothesis": answer,
+ "evidence_binding": evidence_binding,
+ "answer_protocol": answer_protocol,
+ "answer_model": configured_model,
+ "answer_attempt_limit": max(1, args.attempts),
+ "answer_attempts_used": attempts_used,
+ "logical_answer_calls": logical_answer_calls,
+ "selected_session_ids": list(row.get("selected_session_ids") or []),
+ "latency_sec": round(time.time() - started, 3),
+ "error": "",
+ }
+
+ pending = [row for row in rows if clean_text(row.get("question_id")) not in existing]
+ failures: list[dict[str, str]] = []
+ completed = 0
+ started = time.time()
+ with ThreadPoolExecutor(max_workers=max(1, args.workers)) as executor:
+ futures = {executor.submit(answer_one, row): clean_text(row.get("question_id")) for row in pending}
+ for future in as_completed(futures):
+ qid = futures[future]
+ try:
+ result = future.result()
+ except Exception as exc:
+ failures.append({"question_id": qid, "error": f"{exc.__class__.__name__}:{str(exc)[:500]}"})
+ continue
+ with lock:
+ with answers_path.open("a", encoding="utf-8") as handle:
+ handle.write(json.dumps(result, ensure_ascii=False, sort_keys=True) + "\n")
+ existing[qid] = result
+ completed += 1
+ if completed % 10 == 0 or completed == len(pending):
+ print(
+ json.dumps(
+ {
+ "completed_this_run": completed,
+ "pending_this_run": len(pending),
+ "total_saved": len(existing),
+ "failures": len(failures),
+ "elapsed_sec": round(time.time() - started, 3),
+ }
+ ),
+ flush=True,
+ )
+ final_rows = ordered_completed_answers(existing, rows)
+ atomic_write_jsonl(answers_path, final_rows)
+ protocols = sorted({str(row.get("answer_protocol") or "") for row in final_rows})
+ report = {
+ "status": "complete" if len(final_rows) == len(rows) and not failures else "incomplete",
+ "created_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
+ "answer_model": configured_model,
+ "lane": args.lane,
+ "promotion_eligible": args.lane == "production",
+ "fixed_answer_layer": fixed_answer_layer,
+ "disabled_harness_hooks": ["complete_profile_answer_from_evidence"],
+ "max_logical_answer_calls_per_success": max(1, args.attempts),
+ "hierarchical_memory_context_rendering": True,
+ "compiled_evidence_packet_required": True,
+ "answer_protocol": protocols[0] if len(protocols) == 1 else "mixed",
+ "evidence_binding_persisted": True,
+ "answers_bound_to_evidence_sha256": True,
+ "legacy_evidence_hash_backfills": legacy_evidence_hash_backfills,
+ "requested": len(rows),
+ "completed": len(final_rows),
+ "failure_count": len(failures),
+ "avg_latency_sec": round(sum(float(row["latency_sec"]) for row in final_rows) / max(1, len(final_rows)), 3),
+ "answers": str(answers_path),
+ }
+ (out_dir / "failures.json").write_text(json.dumps(failures, indent=2, sort_keys=True), encoding="utf-8")
+ (out_dir / "report.json").write_text(json.dumps(report, indent=2, sort_keys=True), encoding="utf-8")
+ print(json.dumps(report, indent=2, sort_keys=True))
+ return 0 if report["status"] == "complete" else 2
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/legacy/run_tmcra_v4_parallel_official_judge.py b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/run_tmcra_v4_parallel_official_judge.py
new file mode 100644
index 0000000000000000000000000000000000000000..bd16018ca699792f186cb050d6c89483a6c66fed
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/run_tmcra_v4_parallel_official_judge.py
@@ -0,0 +1,273 @@
+#!/usr/bin/env python3
+"""Run one isolated official LongMemEval judge process per question."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import os
+import subprocess
+import sys
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from pathlib import Path
+from typing import Any, Mapping
+
+
+DEFAULT_OFFICIAL_JUDGE = (
+ Path(__file__).resolve().parent.parent / "harness" / "evaluate_qa_official_judge.py"
+)
+METRIC_MODEL_ALIASES = {"gpt-4o-2024-08-06": "gpt-4o"}
+BOUND_FIELDS = ("question", "hypothesis", "evidence_sha256", "answer_model")
+
+
+class ParallelJudgeError(RuntimeError):
+ pass
+
+
+def _read_jsonl(path: Path) -> list[dict[str, Any]]:
+ if not path.is_file():
+ return []
+ return [
+ json.loads(line)
+ for line in path.read_text(encoding="utf-8").splitlines()
+ if line.strip()
+ ]
+
+
+def _write_jsonl_atomic(path: Path, rows: list[Mapping[str, Any]]) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ temporary = path.with_name(f".{path.name}.tmp.{os.getpid()}")
+ with temporary.open("w", encoding="utf-8") as handle:
+ for row in rows:
+ handle.write(json.dumps(row, ensure_ascii=False) + "\n")
+ handle.flush()
+ os.fsync(handle.fileno())
+ os.replace(temporary, path)
+
+
+def _validate_result(row: Mapping[str, Any], answer: Mapping[str, Any]) -> None:
+ qid = str(answer.get("question_id") or "").strip()
+ if str(row.get("question_id") or "").strip() != qid:
+ raise ParallelJudgeError(f"{qid}: judge result has a different question ID")
+ for key in BOUND_FIELDS:
+ if row.get(key) != answer.get(key):
+ raise ParallelJudgeError(f"{qid}: judge result is bound to different {key}")
+ label = row.get("autoeval_label")
+ if (
+ not isinstance(label, Mapping)
+ or type(label.get("label")) is not bool
+ or not str(label.get("model") or "").strip()
+ or not str(label.get("raw_response") or "").strip()
+ ):
+ raise ParallelJudgeError(f"{qid}: judge result has no valid official label")
+
+
+def _question_dir(work_dir: Path, index: int, qid: str) -> Path:
+ digest = hashlib.sha256(qid.encode("utf-8")).hexdigest()[:16]
+ return work_dir / f"{index:04d}_{digest}"
+
+
+def _judge_one(
+ *,
+ index: int,
+ answer: Mapping[str, Any],
+ args: argparse.Namespace,
+ environment: Mapping[str, str],
+ work_dir: Path,
+) -> tuple[int, dict[str, Any], bool]:
+ qid = str(answer["question_id"])
+ question_dir = _question_dir(work_dir, index, qid)
+ question_dir.mkdir(parents=True, exist_ok=True)
+ hypothesis_path = question_dir / "hypothesis.jsonl"
+ result_path = question_dir / "official_judge.jsonl"
+ log_path = question_dir / "official_judge.log"
+ _write_jsonl_atomic(hypothesis_path, [answer])
+
+ cached = _read_jsonl(result_path)
+ if cached:
+ if len(cached) != 1:
+ raise ParallelJudgeError(f"{qid}: cached judge result is not one row")
+ _validate_result(cached[0], answer)
+ return index, cached[0], True
+
+ command = [
+ sys.executable,
+ str(args.judge.resolve()),
+ "--metric-model",
+ METRIC_MODEL_ALIASES.get(args.metric_model, args.metric_model),
+ "--hyp-file",
+ str(hypothesis_path),
+ "--ref-file",
+ str(args.ref_file.resolve()),
+ "--result-file",
+ str(result_path),
+ "--base-url",
+ args.base_url,
+ "--api-key-env",
+ args.api_key_env,
+ "--timeout",
+ str(args.timeout),
+ "--max-retries",
+ str(args.max_retries),
+ "--quiet",
+ ]
+ completed = subprocess.run(
+ command,
+ env=dict(environment),
+ stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT,
+ text=True,
+ timeout=args.process_timeout,
+ check=False,
+ )
+ log_path.write_text(completed.stdout or "", encoding="utf-8")
+ if completed.returncode != 0:
+ raise ParallelJudgeError(
+ f"{qid}: official judge process exited {completed.returncode}; log={log_path}"
+ )
+ rows = _read_jsonl(result_path)
+ if len(rows) != 1:
+ raise ParallelJudgeError(f"{qid}: official judge process did not write one row")
+ _validate_result(rows[0], answer)
+ return index, rows[0], False
+
+
+def run_parallel_judge(
+ args: argparse.Namespace, *, environment: Mapping[str, str] | None = None
+) -> dict[str, Any]:
+ environment = dict(environment or os.environ)
+ if not environment.get(args.api_key_env):
+ raise ParallelJudgeError(f"missing API key env: {args.api_key_env}")
+ if not args.judge.is_file():
+ raise ParallelJudgeError(f"official judge is missing: {args.judge}")
+
+ hypotheses = _read_jsonl(args.hyp_file)
+ if args.limit and args.limit > 0:
+ hypotheses = hypotheses[: args.limit]
+ expected_qids = [str(item.get("question_id") or "").strip() for item in hypotheses]
+ if not expected_qids or any(not qid for qid in expected_qids):
+ raise ParallelJudgeError("hypotheses are empty or contain empty question IDs")
+ if len(expected_qids) != len(set(expected_qids)):
+ raise ParallelJudgeError("hypotheses contain duplicate question IDs")
+
+ result_file = args.result_file or Path(
+ str(args.hyp_file) + f".eval-results-{args.metric_model}"
+ )
+ if result_file.exists() and not args.resume:
+ raise ParallelJudgeError("judge result exists; use --resume")
+ existing = _read_jsonl(result_file)
+ if len(existing) > len(hypotheses):
+ raise ParallelJudgeError("existing judge result is longer than hypotheses")
+ for index, row in enumerate(existing):
+ _validate_result(row, hypotheses[index])
+
+ committed: dict[int, dict[str, Any]] = {
+ index: row for index, row in enumerate(existing)
+ }
+ work_dir = args.work_dir or result_file.with_name(f"{result_file.name}.work")
+ work_dir.mkdir(parents=True, exist_ok=True)
+ pending_indices = list(range(len(existing), len(hypotheses)))
+ failures: list[str] = []
+ cached_count = 0
+ called_count = 0
+
+ def commit_contiguous_prefix() -> int:
+ prefix: list[dict[str, Any]] = []
+ for index in range(len(hypotheses)):
+ if index not in committed:
+ break
+ prefix.append(committed[index])
+ _write_jsonl_atomic(result_file, prefix)
+ return len(prefix)
+
+ with ThreadPoolExecutor(max_workers=args.workers) as executor:
+ futures = {
+ executor.submit(
+ _judge_one,
+ index=index,
+ answer=hypotheses[index],
+ args=args,
+ environment=environment,
+ work_dir=work_dir,
+ ): index
+ for index in pending_indices
+ }
+ for future in as_completed(futures):
+ index = futures[future]
+ try:
+ completed_index, row, cached = future.result()
+ committed[completed_index] = row
+ cached_count += int(cached)
+ called_count += int(not cached)
+ commit_contiguous_prefix()
+ except Exception as exc:
+ failures.append(f"{expected_qids[index]}: {exc}")
+
+ committed_count = commit_contiguous_prefix()
+ report = {
+ "status": "complete" if committed_count == len(hypotheses) else "incomplete",
+ "metric_model": args.metric_model,
+ "row_count": len(hypotheses),
+ "committed_count": committed_count,
+ "existing_count": len(existing),
+ "cached_count": cached_count,
+ "physical_call_count": called_count,
+ "failure_count": len(failures),
+ "failures": failures,
+ "workers": args.workers,
+ "work_dir": str(work_dir),
+ }
+ if failures or committed_count != len(hypotheses):
+ report_path = result_file.with_name(f"{result_file.name}.isolated_report.json")
+ temporary = report_path.with_name(f".{report_path.name}.tmp.{os.getpid()}")
+ temporary.write_text(json.dumps(report, indent=2, sort_keys=True) + "\n", encoding="utf-8")
+ os.replace(temporary, report_path)
+ raise ParallelJudgeError(
+ f"official judge incomplete: committed={committed_count}/{len(hypotheses)}, "
+ f"failures={len(failures)}; report={report_path}"
+ )
+ labels = [bool((committed[index]["autoeval_label"])["label"]) for index in range(len(hypotheses))]
+ report["correct_count"] = sum(labels)
+ report["accuracy"] = round(sum(labels) / len(labels), 6)
+ report_path = result_file.with_name(f"{result_file.name}.isolated_report.json")
+ temporary = report_path.with_name(f".{report_path.name}.tmp.{os.getpid()}")
+ temporary.write_text(json.dumps(report, indent=2, sort_keys=True) + "\n", encoding="utf-8")
+ os.replace(temporary, report_path)
+ return report
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--judge", type=Path, default=DEFAULT_OFFICIAL_JUDGE)
+ parser.add_argument("--metric-model", default="gpt-4o")
+ parser.add_argument("--hyp-file", required=True, type=Path)
+ parser.add_argument("--ref-file", required=True, type=Path)
+ parser.add_argument("--result-file", type=Path)
+ parser.add_argument("--work-dir", type=Path)
+ parser.add_argument("--base-url", required=True)
+ parser.add_argument("--api-key-env", default="OPENAI_API_KEY")
+ parser.add_argument("--limit", type=int, default=0)
+ parser.add_argument("--timeout", type=int, default=180)
+ parser.add_argument("--process-timeout", type=int, default=240)
+ parser.add_argument("--max-retries", type=int, default=0)
+ parser.add_argument("--resume", action="store_true")
+ parser.add_argument("--quiet", action="store_true")
+ parser.add_argument(
+ "--workers",
+ type=int,
+ default=int(os.environ.get("TMCRA_OFFICIAL_JUDGE_WORKERS", "8")),
+ )
+ args = parser.parse_args()
+ if args.workers <= 0:
+ raise ParallelJudgeError("workers must be positive")
+ if args.process_timeout <= args.timeout:
+ raise ParallelJudgeError("process timeout must exceed request timeout")
+ summary = run_parallel_judge(args)
+ if not args.quiet:
+ print(json.dumps(summary, sort_keys=True))
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/legacy/run_tmcra_v4_retrieve.py b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/run_tmcra_v4_retrieve.py
new file mode 100644
index 0000000000000000000000000000000000000000..80890d48fda47f5623dae0aa61ff10e95199299b
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/run_tmcra_v4_retrieve.py
@@ -0,0 +1,524 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import os
+import re
+import sys
+from pathlib import Path
+from typing import Any
+
+import tmcra_v4_online_runtime as online
+from run_tmcra_v4_build import (
+ BASE,
+ DEFAULT_REPO,
+ DEFAULT_WRITER_ENV,
+ _key_pool,
+ _load_shell_environment,
+ _now,
+ _resume_log,
+ _run,
+)
+from tmcra_v4_cost_report import build_report, collect_calls
+from tmcra_v4_route_policy import (
+ RoutePolicyError,
+ validate_diagnostic_retrieval_rows,
+ validate_production_packing_budget,
+ validate_production_retrieval_mode,
+ validate_production_retrieval_rows,
+)
+
+
+PROJECT_ROOT = BASE.parents[2]
+PACKAGE_ROOT = BASE.parent
+DEFAULT_HARNESS = PACKAGE_ROOT / "harness" / "run_lme_s10_native_tmcra.py"
+DEFAULT_NODE = PROJECT_ROOT / "models" / "tmcra" / "node_scorer.pt"
+DEFAULT_PATH_MODEL = PROJECT_ROOT / "models" / "tmcra" / "path_scorer.pt"
+DEFAULT_CHECKPOINT = PROJECT_ROOT / "models" / "tmcra" / "tmcra_v3_reranker.pt"
+DEFAULT_EMBEDDING = PROJECT_ROOT / "models" / "BAAI" / "bge-m3"
+DEFAULT_CROSS = PROJECT_ROOT / "models" / "BAAI" / "bge-reranker-v2-m3"
+
+
+class RetrievalError(RuntimeError):
+ pass
+
+
+def _jsonl(path: Path) -> list[dict[str, Any]]:
+ return [
+ json.loads(line)
+ for line in path.read_text(encoding="utf-8").splitlines()
+ if line.strip()
+ ]
+
+
+def _fingerprints(rows: list[dict[str, Any]]) -> dict[str, str]:
+ return {
+ str(row["question_id"]): online.scope_fingerprint(
+ Path(str(row["db_path"])).resolve(), str(row["scope_id"])
+ )
+ for row in rows
+ }
+
+
+def _file_sha256(path: Path) -> str:
+ digest = hashlib.sha256()
+ with path.open("rb") as handle:
+ for chunk in iter(lambda: handle.read(1024 * 1024), b""):
+ digest.update(chunk)
+ return digest.hexdigest()
+
+
+def _atomic_write_text(path: Path, text: str) -> None:
+ temporary = path.with_name(f".{path.name}.tmp.{os.getpid()}")
+ temporary.write_text(text, encoding="utf-8")
+ os.replace(temporary, path)
+
+
+def _prepare_graph_boundary(
+ *,
+ run_dir: Path,
+ tag: str,
+ fingerprints: dict[str, str],
+ resume: bool,
+) -> None:
+ before_path = run_dir / f"{tag}.graph_before.json"
+ failed_path = run_dir / f"FAILED.{tag}"
+ staging_path = run_dir / f".{tag}.staging"
+ if resume:
+ if not failed_path.is_file():
+ raise RetrievalError("resume requires the existing FAILED marker")
+ if not before_path.is_file():
+ raise RetrievalError("resume requires the original graph_before boundary")
+ try:
+ original = json.loads(before_path.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError) as exc:
+ raise RetrievalError("original graph_before boundary is unreadable") from exc
+ if original != fingerprints:
+ raise RetrievalError("graph changed since the failed retrieval began")
+ return
+ if failed_path.exists() or before_path.exists() or staging_path.exists():
+ raise RetrievalError(
+ "prior retrieval state exists; inspect it and use explicit --resume"
+ )
+ _atomic_write_text(
+ before_path,
+ json.dumps(fingerprints, indent=2, sort_keys=True) + "\n",
+ )
+
+
+def _archive_failed_marker(run_dir: Path, tag: str) -> None:
+ failed_path = run_dir / f"FAILED.{tag}"
+ if not failed_path.is_file():
+ return
+ history_path = run_dir / f"{tag}.failure_history.jsonl"
+ content = failed_path.read_text(encoding="utf-8")
+ with history_path.open("a", encoding="utf-8") as handle:
+ for line in content.splitlines():
+ if line.strip():
+ handle.write(line.rstrip() + "\n")
+ failed_path.unlink()
+
+
+def _record_failure(run_dir: Path, tag: str, exc: Exception) -> None:
+ _archive_failed_marker(run_dir, tag)
+ _atomic_write_text(
+ run_dir / f"FAILED.{tag}",
+ json.dumps(
+ {"at": _now(), "error": f"{exc.__class__.__name__}: {exc}"}
+ )
+ + "\n",
+ )
+
+
+def _validate_manifest_pair(
+ scope_rows: list[dict[str, Any]], query_rows: list[dict[str, Any]]
+) -> None:
+ def inventory(rows: list[dict[str, Any]], label: str) -> dict[str, tuple[str, str, str]]:
+ result: dict[str, tuple[str, str, str]] = {}
+ for row in rows:
+ qid = str(row.get("question_id") or "").strip()
+ identity = tuple(
+ str(Path(str(row[key])).resolve()) if key != "scope_id" else str(row[key]).strip()
+ for key in ("db_path", "scope_id", "index_path")
+ )
+ if not qid or qid in result or not all(identity):
+ raise RetrievalError(f"{label} has a missing or duplicate retrieval identity")
+ result[qid] = identity
+ return result
+
+ scope_inventory = inventory(scope_rows, "scope manifest")
+ query_inventory = inventory(query_rows, "query manifest")
+ if scope_inventory != query_inventory:
+ raise RetrievalError("scope/query manifests do not reference the same frozen indexes")
+
+
+def _select_manifest_rows(
+ scope_rows: list[dict[str, Any]],
+ query_rows: list[dict[str, Any]],
+ qid_path: Path,
+) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
+ qids = [
+ line.strip()
+ for line in qid_path.read_text(encoding="utf-8").splitlines()
+ if line.strip()
+ ]
+ if not qids or len(qids) != len(set(qids)):
+ raise RetrievalError("qid list is empty or duplicated")
+ scope_by_qid = {str(row.get("question_id") or "").strip(): row for row in scope_rows}
+ query_by_qid = {str(row.get("question_id") or "").strip(): row for row in query_rows}
+ missing = [
+ qid
+ for qid in qids
+ if qid not in scope_by_qid or qid not in query_by_qid
+ ]
+ if missing:
+ raise RetrievalError(f"qid list contains unknown questions: {missing[:10]}")
+ return (
+ [scope_by_qid[qid] for qid in qids],
+ [query_by_qid[qid] for qid in qids],
+ )
+
+
+def _write_jsonl(path: Path, rows: list[dict[str, Any]]) -> None:
+ _atomic_write_text(
+ path,
+ "".join(json.dumps(row, ensure_ascii=False, sort_keys=True) + "\n" for row in rows),
+ )
+
+
+def _run_production_build_preflight(
+ run_dir: Path, tag: str, *, resume: bool
+) -> dict[str, Any]:
+ output = run_dir / f"{tag}.production_build_preflight.json"
+ _run(
+ [
+ sys.executable,
+ str(BASE / "audit_tmcra_v4_chain.py"),
+ "--run-dir",
+ str(run_dir),
+ "--output",
+ str(output),
+ "--build-only",
+ ],
+ (
+ _resume_log(run_dir, f"{tag}.production_build_preflight")
+ if resume
+ else run_dir / f"{tag}.production_build_preflight.log"
+ ),
+ dict(os.environ),
+ )
+ if not output.is_file():
+ raise RetrievalError("production build preflight did not emit an audit report")
+ try:
+ report = json.loads(output.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError) as exc:
+ raise RetrievalError("production build preflight report is unreadable") from exc
+ if report.get("passed") is not True:
+ raise RetrievalError("production build preflight did not pass")
+ return report
+
+
+def _validate_committed_retrieval_output(
+ out_dir: Path,
+ *,
+ expected_qids: list[str],
+ execution_lane: str,
+) -> dict[str, Any]:
+ evidence_rows = _jsonl(out_dir / "evidence_windows.jsonl")
+ debug_rows = _jsonl(out_dir / "retrieval_debug.jsonl")
+ try:
+ route_report = (
+ validate_production_retrieval_rows(evidence_rows)
+ if execution_lane == "production"
+ else validate_diagnostic_retrieval_rows(evidence_rows)
+ )
+ except RoutePolicyError as exc:
+ raise RetrievalError(
+ f"committed retrieval output does not match the requested lane: {exc}"
+ ) from exc
+ evidence_qids = [str(row.get("question_id") or "").strip() for row in evidence_rows]
+ debug_qids = [
+ str(row.get("question_id") or row.get("qid") or "").strip()
+ for row in debug_rows
+ ]
+ if evidence_qids != expected_qids or debug_qids != expected_qids:
+ raise RetrievalError(
+ "committed retrieval output does not match the selected query manifest"
+ )
+ try:
+ report = json.loads((out_dir / "report.json").read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError) as exc:
+ raise RetrievalError("committed retrieval report is unreadable") from exc
+ if report.get("query_count") != len(expected_qids):
+ raise RetrievalError("committed retrieval report query count is inconsistent")
+ return route_report
+
+
+def retrieve(args: argparse.Namespace) -> dict[str, Any]:
+ # The logical run path is part of durable retrieval identity. Following a
+ # storage symlink here would invalidate resumable checkpoints after a move.
+ run_dir = args.run_dir.absolute()
+ execution_lane = "diagnostic" if bool(getattr(args, "diagnostic", False)) else "production"
+ try:
+ validate_production_retrieval_mode(
+ getattr(args, "composition_mode", None), execution_lane=execution_lane
+ )
+ validate_production_packing_budget(
+ getattr(args, "packing_budget_mode", "fixed"),
+ getattr(args, "top_k", 8),
+ execution_lane=execution_lane,
+ )
+ except RoutePolicyError as exc:
+ raise RetrievalError(f"retrieval route policy rejected the run: {exc}") from exc
+ if not (run_dir / "BUILD_COMPLETE").is_file():
+ raise RetrievalError("frozen V4 build is incomplete")
+ if not re.fullmatch(r"[A-Za-z0-9_.-]+", args.tag):
+ raise RetrievalError("tag contains unsupported characters")
+ out_dir = run_dir / args.tag
+ resume = bool(getattr(args, "resume", False))
+ reuse_committed_output = False
+ if out_dir.exists():
+ required = (
+ out_dir / "evidence_windows.jsonl",
+ out_dir / "retrieval_debug.jsonl",
+ out_dir / "report.json",
+ )
+ if not resume:
+ raise RetrievalError(f"retrieval output already exists: {out_dir}")
+ if not out_dir.is_dir() or not all(path.is_file() for path in required):
+ raise RetrievalError(
+ f"resume found an incomplete committed retrieval output: {out_dir}"
+ )
+ reuse_committed_output = True
+ query_manifest = Path(
+ getattr(args, "query_manifest", None) or run_dir / "query_manifest.jsonl"
+ ).resolve()
+ scope_manifest = Path(
+ getattr(args, "scope_manifest", None) or run_dir / "scope_manifest.jsonl"
+ ).resolve()
+ if not query_manifest.is_file() or not scope_manifest.is_file():
+ raise RetrievalError("scope/query manifest is missing")
+ rows = _jsonl(scope_manifest)
+ query_rows = _jsonl(query_manifest)
+ if not rows or not query_rows:
+ raise RetrievalError("scope/query manifest is empty")
+ _validate_manifest_pair(rows, query_rows)
+ qid_list = getattr(args, "qid_list", None)
+ if qid_list is not None:
+ if not Path(qid_list).resolve().is_file():
+ raise RetrievalError("qid list is missing")
+ rows, query_rows = _select_manifest_rows(
+ rows, query_rows, Path(qid_list).resolve()
+ )
+ scope_manifest = run_dir / f"{args.tag}.scope_manifest.selected.jsonl"
+ query_manifest = run_dir / f"{args.tag}.query_manifest.selected.jsonl"
+ _write_jsonl(scope_manifest, rows)
+ _write_jsonl(query_manifest, query_rows)
+ _validate_manifest_pair(rows, query_rows)
+
+ if reuse_committed_output:
+ _validate_committed_retrieval_output(
+ out_dir,
+ expected_qids=[str(row["question_id"]).strip() for row in query_rows],
+ execution_lane=execution_lane,
+ )
+
+ if execution_lane == "production":
+ _run_production_build_preflight(run_dir, args.tag, resume=resume)
+
+ shell_environment = _load_shell_environment(args.writer_env.resolve())
+ environment = {**os.environ, **shell_environment}
+ keys = _key_pool(environment)
+ base_url = environment.get("TMCRA_DEEPSEEK_WRITER_BASE_URL") or environment.get("TMCRA_WRITER_BASE_URL") or "https://api.deepseek.com/v1"
+ environment.update(
+ {
+ "TMCRA_RECALL_PLANNER_BASE_URL": base_url,
+ "TMCRA_RECALL_PLANNER_MODEL": "deepseek-v4-flash",
+ "TMCRA_RECALL_PLANNER_API_KEY_POOL": ",".join(keys),
+ "TMCRA_NODE_MODEL_PATH": str(args.node_model.resolve()),
+ "TMCRA_PATH_MODEL_PATH": str(args.path_model.resolve()),
+ "TMCRA_NODE_MODEL_DEVICE": args.graph_device,
+ }
+ )
+ before = _fingerprints(rows)
+ _prepare_graph_boundary(
+ run_dir=run_dir,
+ tag=args.tag,
+ fingerprints=before,
+ resume=resume,
+ )
+ try:
+ if not reuse_committed_output:
+ runtime_options = ["--resume"] if resume else []
+ if args.planner_replay_dir is not None:
+ runtime_options.extend(
+ ["--planner-replay-dir", str(args.planner_replay_dir.resolve())]
+ )
+ _run(
+ [
+ sys.executable,
+ str(BASE / "tmcra_v4_online_runtime.py"),
+ "retrieve",
+ *runtime_options,
+ "--query-manifest",
+ str(query_manifest),
+ "--out-dir",
+ str(out_dir),
+ "--checkpoint",
+ str(args.checkpoint.resolve()),
+ "--cross-model",
+ str(args.cross_model.resolve()),
+ "--repo",
+ str(args.repo.resolve()),
+ "--harness",
+ str(args.harness.resolve()),
+ "--node-model",
+ str(args.node_model.resolve()),
+ "--path-model",
+ str(args.path_model.resolve()),
+ "--graph-device",
+ args.graph_device,
+ "--embedding-model",
+ str(args.embedding_model.resolve()),
+ "--slow-dense-k",
+ str(args.slow_dense_k),
+ "--dense-k",
+ str(args.dense_k),
+ "--graph-k",
+ str(args.graph_k),
+ "--composition-mode",
+ str(args.composition_mode),
+ "--execution-lane",
+ execution_lane,
+ "--packing-budget-mode",
+ str(args.packing_budget_mode),
+ "--top-k",
+ str(args.top_k),
+ "--adaptive-simple-k",
+ str(args.adaptive_simple_k),
+ "--adaptive-standard-k",
+ str(args.adaptive_standard_k),
+ "--adaptive-complex-k",
+ str(args.adaptive_complex_k),
+ "--cross-batch-size",
+ str(args.cross_batch_size),
+ "--device",
+ args.device,
+ ],
+ (
+ _resume_log(run_dir, args.tag)
+ if resume
+ else run_dir / f"{args.tag}.log"
+ ),
+ environment,
+ )
+ after = _fingerprints(rows)
+ _atomic_write_text(
+ run_dir / f"{args.tag}.graph_after.json",
+ json.dumps(after, indent=2, sort_keys=True) + "\n",
+ )
+ if after != before:
+ raise RetrievalError("retrieval changed graph structure or graph records")
+ _run(
+ [
+ sys.executable,
+ str(BASE / "audit_tmcra_v4_chain.py"),
+ "--run-dir",
+ str(run_dir),
+ "--retrieval-dir",
+ str(out_dir),
+ "--output",
+ str(run_dir / f"{args.tag}.chain_audit.json"),
+ ],
+ (
+ _resume_log(run_dir, f"{args.tag}.chain_audit")
+ if resume
+ else run_dir / f"{args.tag}.chain_audit.log"
+ ),
+ environment,
+ )
+ # Retrieval cost is incremental. Frozen graph databases contain the
+ # historical writer ledger and must not be charged to every query run.
+ calls = collect_calls([out_dir / "retrieval_debug.jsonl"], [])
+ cost = build_report(calls)
+ (run_dir / f"{args.tag}.cost_report.json").write_text(
+ json.dumps(cost, indent=2, sort_keys=True) + "\n", encoding="utf-8"
+ )
+ report = json.loads((out_dir / "report.json").read_text(encoding="utf-8"))
+ summary = {
+ "schema_version": "tmcra.v4.retrieval-run.1",
+ "status": "complete",
+ "tag": args.tag,
+ "execution_lane": execution_lane,
+ "composition_mode": str(args.composition_mode),
+ "query_count": report["query_count"],
+ "selected_output": str(out_dir / "evidence_windows.jsonl"),
+ "scope_manifest": str(scope_manifest),
+ "scope_manifest_sha256": _file_sha256(scope_manifest),
+ "query_manifest": str(query_manifest),
+ "query_manifest_sha256": _file_sha256(query_manifest),
+ "qid_list": str(Path(qid_list).resolve()) if qid_list is not None else None,
+ "qid_list_sha256": _file_sha256(Path(qid_list).resolve()) if qid_list is not None else None,
+ "graph_structure_unchanged": True,
+ "resumed": resume,
+ "reused_committed_retrieval_output": reuse_committed_output,
+ "physical_api_call_count_total": cost["physical_call_count"],
+ "exact_cost_cny_total": cost["exact_cost_cny"],
+ "min_cost_cny_total": cost["min_cost_cny"],
+ "max_cost_cny_total": cost["max_cost_cny"],
+ }
+ (out_dir / "retrieval_run_report.json").write_text(
+ json.dumps(summary, indent=2, sort_keys=True) + "\n", encoding="utf-8"
+ )
+ (out_dir / "RETRIEVAL_COMPLETE").write_text(_now() + "\n", encoding="utf-8")
+ _archive_failed_marker(run_dir, args.tag)
+ return summary
+ except Exception as exc:
+ _record_failure(run_dir, args.tag, exc)
+ raise
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description="Run one TMCRA V4 retrieval pass over a frozen build")
+ parser.add_argument("--run-dir", type=Path, required=True)
+ parser.add_argument("--tag", default="retrieval_1")
+ parser.add_argument("--resume", action="store_true")
+ parser.add_argument("--scope-manifest", type=Path)
+ parser.add_argument("--query-manifest", type=Path)
+ parser.add_argument("--qid-list", type=Path)
+ parser.add_argument("--planner-replay-dir", type=Path)
+ parser.add_argument("--writer-env", type=Path, default=DEFAULT_WRITER_ENV)
+ parser.add_argument("--repo", type=Path, default=DEFAULT_REPO)
+ parser.add_argument("--harness", type=Path, default=DEFAULT_HARNESS)
+ parser.add_argument("--node-model", type=Path, default=DEFAULT_NODE)
+ parser.add_argument("--path-model", type=Path, default=DEFAULT_PATH_MODEL)
+ parser.add_argument("--checkpoint", type=Path, default=DEFAULT_CHECKPOINT)
+ parser.add_argument("--embedding-model", type=Path, default=DEFAULT_EMBEDDING)
+ parser.add_argument("--cross-model", type=Path, default=DEFAULT_CROSS)
+ parser.add_argument("--device", default="cuda")
+ parser.add_argument("--graph-device", default="cuda")
+ parser.add_argument("--dense-k", type=int, default=32)
+ parser.add_argument("--slow-dense-k", type=int, default=24)
+ parser.add_argument("--graph-k", type=int, default=24)
+ parser.add_argument("--composition-mode", choices=("layered", "source-only-diagnostic"), default="layered")
+ parser.add_argument(
+ "--diagnostic",
+ action="store_true",
+ help="explicitly allow a non-production retrieval composition",
+ )
+ parser.add_argument("--packing-budget-mode", choices=("fixed", "adaptive"), default="fixed")
+ parser.add_argument("--top-k", type=int, default=8)
+ parser.add_argument("--adaptive-simple-k", type=int, default=8)
+ parser.add_argument("--adaptive-standard-k", type=int, default=12)
+ parser.add_argument("--adaptive-complex-k", type=int, default=16)
+ parser.add_argument("--cross-batch-size", type=int, default=24)
+ args = parser.parse_args()
+ summary = retrieve(args)
+ print(json.dumps(summary, sort_keys=True))
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_model_assets.py b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_model_assets.py
new file mode 100644
index 0000000000000000000000000000000000000000..2d8a670369cc038e4427cd47e834f84da084f151
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_model_assets.py
@@ -0,0 +1,29 @@
+from __future__ import annotations
+
+import json
+from collections.abc import Mapping
+from pathlib import Path
+from typing import Any
+
+
+def load_optional_hf_model_manifest(model_dir: str | Path) -> dict[str, Any]:
+ """Read optional provenance metadata for a locally cached Hugging Face model.
+
+ ``TMCRA_MODEL_MANIFEST.json`` is a TMCRA sidecar, not part of the Hugging
+ Face model layout. Official BGE snapshots therefore remain valid without
+ it. When a sidecar is supplied, validate it before using its provenance
+ fields in reports.
+ """
+
+ path = Path(model_dir) / "TMCRA_MODEL_MANIFEST.json"
+ if not path.exists():
+ return {}
+ if not path.is_file():
+ raise ValueError(f"model manifest must be a file: {path}")
+ try:
+ payload = json.loads(path.read_text(encoding="utf-8"))
+ except (OSError, UnicodeError, json.JSONDecodeError) as exc:
+ raise ValueError(f"invalid model manifest: {path}") from exc
+ if not isinstance(payload, Mapping):
+ raise ValueError(f"model manifest must contain a JSON object: {path}")
+ return dict(payload)
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v2_lme_pipeline.py b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v2_lme_pipeline.py
new file mode 100644
index 0000000000000000000000000000000000000000..d509d3f9fbf023957e2ee1eb07f4209b64fb949c
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v2_lme_pipeline.py
@@ -0,0 +1,855 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import math
+import os
+import random
+import re
+import time
+import urllib.request
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Mapping, Sequence
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+TOKEN_RE = re.compile(r"[a-zA-Z0-9]+")
+EVENT_RE = re.compile(r"^event::longmemeval:(?P[^:]+):s(?P\d+)_c(?P\d+)$")
+
+
+def clean_text(value: Any) -> str:
+ return " ".join(str(value or "").split())
+
+
+def norm_tokens(text: Any) -> list[str]:
+ return [m.group(0).lower() for m in TOKEN_RE.finditer(clean_text(text))]
+
+
+def stable_hash(value: str) -> int:
+ return int.from_bytes(hashlib.blake2b(value.encode("utf-8"), digest_size=8).digest(), "big")
+
+
+def embedding_text_key(text: Any) -> str:
+ cleaned = clean_text(text)
+ return hashlib.sha256(cleaned.encode("utf-8")).hexdigest()
+
+
+def iter_sample_texts(samples: Sequence[Mapping[str, Any]]) -> list[dict[str, Any]]:
+ items: list[dict[str, Any]] = []
+ seen: set[str] = set()
+ for sample in samples:
+ qid = clean_text(sample.get("question_id"))
+ question = clean_text(sample.get("question", ""))
+ if question:
+ key = embedding_text_key(question)
+ if key not in seen:
+ seen.add(key)
+ items.append({"key": key, "kind": "question", "question_id": qid, "text": question})
+ for cand in list(sample.get("candidates") or []):
+ text = clean_text(cand.get("text", ""))
+ if not text:
+ continue
+ key = embedding_text_key(text)
+ if key not in seen:
+ seen.add(key)
+ items.append({
+ "key": key,
+ "kind": "candidate",
+ "question_id": qid,
+ "event_id": clean_text(cand.get("event_id", "")),
+ "text": text,
+ })
+ if not items:
+ raise RuntimeError("no texts found for embedding precompute")
+ return items
+
+
+def split_name(qid: str, val_ratio: float = 0.2) -> str:
+ bucket = stable_hash(qid) % 10000
+ return "val" if bucket < int(val_ratio * 10000) else "train"
+
+
+def read_jsonl(path: Path) -> list[dict[str, Any]]:
+ if not path.exists():
+ raise FileNotFoundError(f"required jsonl file not found: {path}")
+ rows=[]
+ with path.open("r", encoding="utf-8", errors="replace") as f:
+ for lineno, line in enumerate(f, start=1):
+ if line.strip():
+ try:
+ rows.append(json.loads(line))
+ except json.JSONDecodeError as exc:
+ raise RuntimeError(f"invalid jsonl at {path}:{lineno}: {exc}") from exc
+ if not rows:
+ raise RuntimeError(f"required jsonl file is empty: {path}")
+ return rows
+
+
+def write_jsonl(path: Path, rows: Sequence[Mapping[str, Any]]) -> None:
+ with path.open("w", encoding="utf-8") as f:
+ for row in rows:
+ f.write(json.dumps(row, ensure_ascii=False, sort_keys=True) + "\n")
+
+
+def load_shard_rows(shard_glob: str) -> dict[str, list[dict[str, Any]]]:
+ base = Path("/") if shard_glob.startswith("/") else Path()
+ pattern = shard_glob[1:] if shard_glob.startswith("/") else shard_glob
+ shard_paths = sorted(base.glob(pattern))
+ if not shard_paths:
+ raise FileNotFoundError(f"no shard files matched --shard-glob: {shard_glob}")
+ rows_by_qid: dict[str, list[dict[str, Any]]] = {}
+ for shard in shard_paths:
+ data = json.loads(shard.read_text(encoding="utf-8", errors="replace"))
+ if not isinstance(data, list):
+ raise RuntimeError(f"shard is not a JSON list: {shard}")
+ for idx, row in enumerate(data):
+ if not isinstance(row, dict):
+ raise RuntimeError(f"shard row is not an object: {shard}[{idx}]")
+ qid = clean_text(row.get("question_id"))
+ if qid.endswith("_abs"):
+ qid = qid[:-4]
+ if not qid:
+ raise RuntimeError(f"shard row missing question_id: {shard}[{idx}]")
+ rows_by_qid.setdefault(qid, []).append(row)
+ if not rows_by_qid:
+ raise RuntimeError(f"no usable rows loaded from --shard-glob: {shard_glob}")
+ return rows_by_qid
+
+
+def session_text_chunks(*, session_id: str, date: str, turns: list[Mapping[str, Any]], max_chars: int = 7000, max_chunks: int = 0) -> list[str]:
+ chunks: list[str] = []
+ current: list[str] = [f"LongMemEval session_id={session_id} date={date}"]
+ current_len = len(current[0])
+ for index, turn in enumerate(turns, start=1):
+ role = clean_text(turn.get("role", "unknown"))
+ content = clean_text(turn.get("content", ""))
+ if not content:
+ continue
+ part = f"[{session_id} turn={index} role={role}] {content}"
+ if current_len + len(part) + 2 > max_chars and len(current) > 1:
+ chunks.append("\n".join(current))
+ if max_chunks > 0 and len(chunks) >= max_chunks:
+ return chunks
+ current = [f"LongMemEval session_id={session_id} date={date} continued=true"]
+ current_len = len(current[0])
+ current.append(part)
+ current_len += len(part) + 1
+ if len(current) > 1 and (max_chunks <= 0 or len(chunks) < max_chunks):
+ chunks.append("\n".join(current))
+ return chunks
+
+
+def event_text_map_for_row(row: Mapping[str, Any], qid: str) -> dict[str, dict[str, Any]]:
+ sessions = list(row.get("haystack_sessions") or [])
+ session_ids = [clean_text(x) for x in list(row.get("haystack_session_ids") or [])]
+ dates = [clean_text(x) for x in list(row.get("haystack_dates") or [])]
+ out: dict[str, dict[str, Any]] = {}
+ for sidx, sid in enumerate(session_ids):
+ if sidx >= len(sessions):
+ continue
+ chunks = session_text_chunks(
+ session_id=sid,
+ date=dates[sidx] if sidx < len(dates) else "",
+ turns=list(sessions[sidx] or []),
+ max_chars=7000,
+ max_chunks=0,
+ )
+ for cidx, text in enumerate(chunks, start=1):
+ eid = f"event::longmemeval:{qid}:s{sidx:03d}_c{cidx:02d}"
+ out[eid] = {
+ "event_id": eid,
+ "text": text,
+ "session_id": sid,
+ "session_index": sidx,
+ "chunk_index": cidx,
+ "date": dates[sidx] if sidx < len(dates) else "",
+ }
+ return out
+
+
+def choose_row_for_query(rows: list[dict[str, Any]], candidate_ids: Sequence[str], positive_ids: Sequence[str]) -> dict[str, Any] | None:
+ if not rows:
+ return None
+ wanted = set(candidate_ids or []) | set(positive_ids or [])
+ if not wanted:
+ return None
+ qid = clean_text(rows[0].get("question_id"))
+ if qid.endswith("_abs"):
+ qid = qid[:-4]
+ best = None
+ best_hits = -1
+ for row in rows:
+ rqid = clean_text(row.get("question_id"))
+ if rqid.endswith("_abs"):
+ rqid = rqid[:-4]
+ events = set(event_text_map_for_row(row, rqid))
+ hits = len(wanted & events)
+ if hits > best_hits:
+ best = row
+ best_hits = hits
+ return best
+
+
+def token_overlap(a: str, b: str) -> float:
+ at, bt = set(norm_tokens(a)), set(norm_tokens(b))
+ if not at or not bt:
+ return 0.0
+ return len(at & bt) / max(1, len(at | bt))
+
+
+def contains_number_overlap(a: str, b: str) -> float:
+ nums_a = {t for t in norm_tokens(a) if any(ch.isdigit() for ch in t)}
+ nums_b = {t for t in norm_tokens(b) if any(ch.isdigit() for ch in t)}
+ if not nums_a or not nums_b:
+ return 0.0
+ return len(nums_a & nums_b) / max(1, len(nums_a | nums_b))
+
+
+def build_dataset(args: argparse.Namespace) -> None:
+ label_path = Path(args.aligned_queries)
+ out_dir = Path(args.out_dir)
+ out_dir.mkdir(parents=True, exist_ok=True)
+ labels = read_jsonl(label_path)
+ rows_by_qid = load_shard_rows(args.shard_glob)
+ samples=[]; rejected=[]
+ for row in labels:
+ qid = clean_text(row.get("question_id"))
+ if qid.endswith("_abs"):
+ qid = qid[:-4]
+ question = clean_text(row.get("question"))
+ positives = list(row.get("positive_event_ids") or [])
+ hard_negs = set(row.get("hard_negative_event_ids") or [])
+ candidate_ids = list(row.get("candidate_event_ids") or [])
+ source_rows = rows_by_qid.get(qid, [])
+ source_row = choose_row_for_query(source_rows, candidate_ids, positives)
+ if not source_row:
+ rejected.append({"question_id": qid, "reason": "missing_source_row"})
+ continue
+ event_map = event_text_map_for_row(source_row, qid)
+ if not candidate_ids:
+ rejected.append({"question_id": qid, "reason": "missing_candidate_event_ids"})
+ continue
+ before_filter_count = len(candidate_ids)
+ candidate_ids = [eid for eid in candidate_ids if eid in event_map]
+ if len(candidate_ids) != before_filter_count:
+ rejected.append({"question_id": qid, "reason": "candidate_ids_not_in_event_map", "candidate_count": before_filter_count, "mapped_count": len(candidate_ids)})
+ continue
+ if not candidate_ids:
+ rejected.append({"question_id": qid, "reason": "no_candidate_events"})
+ continue
+ if not any(eid in event_map for eid in positives):
+ rejected.append({"question_id": qid, "reason": "positive_not_in_event_map", "positive_event_ids": positives[:10]})
+ continue
+ q_tokens = set(norm_tokens(question))
+ candidates=[]
+ max_session = max(1, len(source_row.get("haystack_session_ids") or []))
+ for eid in candidate_ids:
+ info = event_map[eid]
+ text = clean_text(info["text"])
+ pos = eid in positives
+ hard = eid in hard_negs
+ candidates.append({
+ "event_id": eid,
+ "text": text,
+ "session_id": info.get("session_id", ""),
+ "session_index": int(info.get("session_index", 0)),
+ "chunk_index": int(info.get("chunk_index", 0)),
+ "features": [
+ token_overlap(question, text),
+ contains_number_overlap(question, text),
+ min(1.0, len(set(norm_tokens(text)) & q_tokens) / max(1, len(q_tokens))),
+ float(info.get("session_index", 0)) / float(max_session),
+ float(info.get("chunk_index", 0)) / 10.0,
+ 1.0 if hard else 0.0,
+ ],
+ "label": 1 if pos else 0,
+ "role_label": 1 if pos else (2 if hard else 0),
+ })
+ if not any(c["label"] for c in candidates):
+ rejected.append({"question_id": qid, "reason": "positive_filtered_out", "positive_event_ids": positives[:10]})
+ continue
+ metadata = dict(row.get("metadata") or {})
+ metadata.setdefault("question_type", clean_text(source_row.get("question_type", "")))
+ metadata.setdefault("gold_answer", clean_text(source_row.get("answer", "")))
+ metadata.setdefault("answer_session_ids", list(source_row.get("answer_session_ids") or []))
+ samples.append({
+ "question_id": qid,
+ "question": question,
+ "question_type": clean_text(source_row.get("question_type", "")),
+ "split": split_name(qid, args.val_ratio),
+ "positive_event_ids": positives,
+ "positive_path_ids": list(row.get("positive_path_ids") or []),
+ "metadata": metadata,
+ "candidates": candidates,
+ })
+ write_jsonl(out_dir / "samples.jsonl", samples)
+ write_jsonl(out_dir / "rejected.jsonl", rejected)
+ report = {
+ "created_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
+ "aligned_queries": str(label_path),
+ "sample_count": len(samples),
+ "rejected_count": len(rejected),
+ "train_count": sum(1 for s in samples if s["split"] == "train"),
+ "val_count": sum(1 for s in samples if s["split"] == "val"),
+ "candidate_count_avg": round(sum(len(s["candidates"]) for s in samples) / max(1, len(samples)), 3),
+ "positive_count_avg": round(sum(sum(c["label"] for c in s["candidates"]) for s in samples) / max(1, len(samples)), 3),
+ "outputs": {"samples": str(out_dir / "samples.jsonl"), "rejected": str(out_dir / "rejected.jsonl")},
+ }
+ (out_dir / "dataset_report.json").write_text(json.dumps(report, ensure_ascii=False, indent=2, sort_keys=True), encoding="utf-8")
+ print(json.dumps(report, ensure_ascii=False, indent=2, sort_keys=True))
+
+
+class OpenAIEmbeddingVectorizer:
+ def __init__(self, *, dim: int, base_url: str, model: str, api_key: str = ""):
+ self.dim = int(dim)
+ self.base_url = base_url.rstrip("/")
+ self.model = model
+ self.api_key = api_key
+ self.cache: dict[str, torch.Tensor] = {}
+
+ def encode_one(self, text: str) -> torch.Tensor:
+ text = clean_text(text)
+ cached = self.cache.get(text)
+ if cached is not None:
+ return cached
+ payload = json.dumps({"model": self.model, "input": text}, ensure_ascii=False).encode("utf-8")
+ headers = {"Content-Type": "application/json"}
+ if self.api_key:
+ headers["Authorization"] = f"Bearer {self.api_key}"
+ req = urllib.request.Request(self.base_url + "/embeddings", data=payload, method="POST", headers=headers)
+ with urllib.request.urlopen(req, timeout=120) as resp:
+ body = json.loads(resp.read().decode("utf-8", "replace"))
+ data = body.get("data") or []
+ if not data or "embedding" not in data[0]:
+ raise RuntimeError(f"embedding response missing data[0].embedding from {self.base_url}")
+ vec = torch.tensor(list(data[0]["embedding"]), dtype=torch.float32)
+ if vec.numel() != self.dim:
+ raise RuntimeError(f"embedding dim mismatch: got {vec.numel()} expected {self.dim}")
+ vec = vec / vec.norm().clamp_min(1e-6)
+ self.cache[text] = vec
+ return vec
+
+
+class BgeM3DenseVectorizer:
+ def __init__(
+ self,
+ *,
+ dim: int,
+ model_path: str,
+ device: str = "cpu",
+ max_length: int = 8192,
+ strict_max_length: bool = False,
+ ):
+ from transformers import AutoModel, AutoTokenizer
+ self.dim = int(dim)
+ self.cache: dict[str, torch.Tensor] = {}
+ self.device = torch.device(device)
+ self.max_length = int(max_length)
+ self.strict_max_length = bool(strict_max_length)
+ if self.max_length <= 0:
+ raise RuntimeError("--embedding-max-length must be positive")
+ resolved = Path(model_path).resolve()
+ if "bge-m3" not in resolved.name.lower() and "bge-m3" not in str(resolved).lower():
+ raise RuntimeError(f"hf backend is currently wired only for local BAAI/bge-m3, got: {model_path}")
+ self.model_path = str(resolved)
+ self.tokenizer = AutoTokenizer.from_pretrained(
+ str(resolved),
+ local_files_only=True,
+ trust_remote_code=False,
+ )
+ self.model = AutoModel.from_pretrained(
+ str(resolved),
+ local_files_only=True,
+ trust_remote_code=False,
+ ).to(self.device)
+ self.model.eval()
+ hidden_size = int(getattr(self.model.config, "hidden_size", 0) or 0)
+ if hidden_size != self.dim:
+ raise RuntimeError(f"bge-m3 hidden_size mismatch: model has {hidden_size}, --text-dim is {self.dim}")
+
+ def encode_batch(self, texts: Sequence[str], batch_size: int = 4) -> torch.Tensor:
+ cleaned = [clean_text(t) for t in texts]
+ if not cleaned:
+ raise RuntimeError("encode_batch got no texts")
+ out_chunks: list[torch.Tensor] = []
+ with torch.no_grad():
+ for start in range(0, len(cleaned), max(1, int(batch_size))):
+ batch_texts = cleaned[start:start + max(1, int(batch_size))]
+ encoded = self.tokenizer(
+ batch_texts,
+ return_tensors="pt",
+ padding=True,
+ truncation=not self.strict_max_length,
+ max_length=self.max_length if not self.strict_max_length else None,
+ )
+ if self.strict_max_length:
+ token_lengths = encoded["attention_mask"].sum(dim=1)
+ longest = int(token_lengths.max())
+ if longest > self.max_length:
+ offending = int(torch.argmax(token_lengths))
+ raise RuntimeError(
+ f"BGE input has {longest} tokens, exceeding strict max_length={self.max_length}; "
+ f"batch_index={offending} chars={len(batch_texts[offending])}"
+ )
+ encoded = {k: v.to(self.device) for k, v in encoded.items()}
+ with torch.autocast(
+ device_type=self.device.type,
+ dtype=torch.float16,
+ enabled=self.device.type == "cuda",
+ ):
+ model_out = self.model(**encoded)
+ if not hasattr(model_out, "last_hidden_state"):
+ raise RuntimeError("bge-m3 model output missing last_hidden_state")
+ vecs = model_out.last_hidden_state[:, 0].detach().cpu().float()
+ if vecs.shape[1] != self.dim:
+ raise RuntimeError(f"bge-m3 embedding dim mismatch: got {vecs.shape[1]} expected {self.dim}")
+ vecs = vecs / vecs.norm(dim=1, keepdim=True).clamp_min(1e-6)
+ out_chunks.append(vecs)
+ return torch.cat(out_chunks, dim=0)
+
+ def encode_one(self, text: str) -> torch.Tensor:
+ text = clean_text(text)
+ cached = self.cache.get(text)
+ if cached is not None:
+ return cached
+ vec = self.encode_batch([text], batch_size=1)[0]
+ self.cache[text] = vec
+ return vec
+
+
+class EmbeddingCacheVectorizer:
+ def __init__(self, *, cache_dir: str, dim: int, expected_backend: str = "", expected_model: str = ""):
+ root = Path(cache_dir)
+ manifest_path = root / "manifest.json"
+ tensor_path = root / "embeddings.pt"
+ if not manifest_path.exists():
+ raise FileNotFoundError(f"embedding cache manifest not found: {manifest_path}")
+ if not tensor_path.exists():
+ raise FileNotFoundError(f"embedding cache tensor not found: {tensor_path}")
+ self.manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
+ self.dim = int(dim)
+ if int(self.manifest.get("text_dim", 0) or 0) != self.dim:
+ raise RuntimeError(f"embedding cache dim mismatch: cache has {self.manifest.get('text_dim')} expected {self.dim}")
+ backend = clean_text(self.manifest.get("embedding_backend", ""))
+ model = clean_text(self.manifest.get("embedding_model", ""))
+ if expected_backend and backend != expected_backend:
+ raise RuntimeError(f"embedding cache backend mismatch: cache has {backend} expected {expected_backend}")
+ if expected_model and model != expected_model:
+ raise RuntimeError(f"embedding cache model mismatch: cache has {model} expected {expected_model}")
+ payload = torch.load(tensor_path, map_location="cpu", weights_only=True)
+ keys = list(payload.get("keys") or [])
+ vectors = payload.get("vectors")
+ if not keys or vectors is None:
+ raise RuntimeError(f"invalid embedding cache payload: {tensor_path}")
+ if len(keys) != int(vectors.shape[0]):
+ raise RuntimeError(f"embedding cache row mismatch: {len(keys)} keys vs {vectors.shape[0]} vectors")
+ if int(vectors.shape[1]) != self.dim:
+ raise RuntimeError(f"embedding cache tensor dim mismatch: {vectors.shape[1]} expected {self.dim}")
+ self.keys = keys
+ self.index = {k: i for i, k in enumerate(keys)}
+ if len(self.index) != len(keys):
+ raise RuntimeError("embedding cache contains duplicate keys")
+ self.vectors = vectors.float().contiguous()
+
+ def encode_one(self, text: str) -> torch.Tensor:
+ key = embedding_text_key(text)
+ idx = self.index.get(key)
+ if idx is None:
+ preview = clean_text(text)[:120]
+ raise RuntimeError(f"embedding cache miss: key={key} preview={preview!r}")
+ return self.vectors[idx]
+
+
+def build_vectorizer(args: argparse.Namespace, device: torch.device):
+ backend = clean_text(getattr(args, "embedding_backend", ""))
+ cache_dir = clean_text(getattr(args, "embedding_cache", ""))
+ if not backend:
+ raise RuntimeError("embedding backend is required; pass --embedding-backend openai or --embedding-backend hf")
+ if cache_dir:
+ return EmbeddingCacheVectorizer(
+ cache_dir=cache_dir,
+ dim=args.text_dim,
+ expected_backend=backend,
+ expected_model=clean_text(getattr(args, "embedding_model", "")),
+ )
+ if backend == "openai":
+ base_url = clean_text(getattr(args, "embedding_base_url", ""))
+ model = clean_text(getattr(args, "embedding_model", ""))
+ api_key = clean_text(getattr(args, "embedding_api_key", ""))
+ if not base_url or not model:
+ raise RuntimeError("openai embedding backend requires explicit --embedding-base-url and --embedding-model")
+ return OpenAIEmbeddingVectorizer(dim=args.text_dim, base_url=base_url, model=model, api_key=api_key)
+ if backend == "hf":
+ model_path = clean_text(getattr(args, "embedding_model", ""))
+ if not model_path:
+ raise RuntimeError("hf embedding backend requires explicit --embedding-model local path")
+ if not os.path.exists(model_path):
+ raise RuntimeError(f"hf embedding backend requires a local model path; not found: {model_path}")
+ return BgeM3DenseVectorizer(dim=args.text_dim, model_path=model_path, device=str(device), max_length=args.embedding_max_length)
+ raise RuntimeError(f"unsupported embedding backend: {backend}")
+
+
+class V2EvidenceScorer(nn.Module):
+ def __init__(self, text_dim: int, feature_dim: int, hidden_dim: int = 256, layers: int = 2, roles: int = 3):
+ super().__init__()
+ self.query_proj = nn.Sequential(nn.LayerNorm(text_dim), nn.Linear(text_dim, hidden_dim), nn.SiLU())
+ self.event_proj = nn.Sequential(nn.LayerNorm(text_dim), nn.Linear(text_dim, hidden_dim), nn.SiLU())
+ self.feature_proj = nn.Sequential(nn.LayerNorm(feature_dim), nn.Linear(feature_dim, hidden_dim // 2), nn.SiLU())
+ self.session_embedding = nn.Embedding(256, hidden_dim)
+ self.chunk_embedding = nn.Embedding(32, hidden_dim)
+ encoder_layer = nn.TransformerEncoderLayer(
+ d_model=hidden_dim,
+ nhead=8,
+ dim_feedforward=hidden_dim * 4,
+ dropout=0.1,
+ activation="gelu",
+ batch_first=True,
+ norm_first=True,
+ )
+ self.graph_encoder = nn.TransformerEncoder(encoder_layer, num_layers=max(1, int(layers)))
+ pair_dim = hidden_dim * 4 + hidden_dim // 2
+ self.event_head = nn.Sequential(nn.LayerNorm(pair_dim), nn.Linear(pair_dim, hidden_dim), nn.SiLU(), nn.Linear(hidden_dim, 1))
+ self.pack_head = nn.Sequential(nn.LayerNorm(pair_dim), nn.Linear(pair_dim, hidden_dim), nn.SiLU(), nn.Linear(hidden_dim, 1))
+ self.role_head = nn.Sequential(nn.LayerNorm(pair_dim), nn.Linear(pair_dim, hidden_dim), nn.SiLU(), nn.Linear(hidden_dim, roles))
+
+ def forward(self, q_vec: torch.Tensor, e_vec: torch.Tensor, features: torch.Tensor, session_idx: torch.Tensor, chunk_idx: torch.Tensor, mask: torch.Tensor) -> dict[str, torch.Tensor]:
+ q = self.query_proj(q_vec)
+ e = self.event_proj(e_vec)
+ f = self.feature_proj(features)
+ session_idx = session_idx.clamp_min(0).clamp_max(255)
+ chunk_idx = chunk_idx.clamp_min(0).clamp_max(31)
+ h = e + self.session_embedding(session_idx) + self.chunk_embedding(chunk_idx)
+ h = self.graph_encoder(h, src_key_padding_mask=~mask)
+ qx = q.unsqueeze(1).expand_as(h)
+ pair = torch.cat([qx, h, qx * h, torch.abs(qx - h), f], dim=-1)
+ event_logits = self.event_head(pair).squeeze(-1).masked_fill(~mask, -1e4)
+ pack_logits = self.pack_head(pair).squeeze(-1).masked_fill(~mask, -1e4)
+ role_logits = self.role_head(pair).masked_fill(~mask.unsqueeze(-1), 0.0)
+ return {"event_logits": event_logits, "pack_logits": pack_logits, "role_logits": role_logits}
+
+
+@dataclass
+class Batch:
+ q_vec: torch.Tensor
+ e_vec: torch.Tensor
+ features: torch.Tensor
+ session_idx: torch.Tensor
+ chunk_idx: torch.Tensor
+ mask: torch.Tensor
+ labels: torch.Tensor
+ role_labels: torch.Tensor
+ qids: list[str]
+ event_ids: list[list[str]]
+
+
+def make_batch(samples: Sequence[Mapping[str, Any]], vectorizer: Any, text_dim: int, feature_dim: int, device: torch.device) -> Batch:
+ max_c = max(len(s["candidates"]) for s in samples)
+ b = len(samples)
+ q_vec = torch.zeros((b, text_dim), dtype=torch.float32)
+ e_vec = torch.zeros((b, max_c, text_dim), dtype=torch.float32)
+ features = torch.zeros((b, max_c, feature_dim), dtype=torch.float32)
+ session_idx = torch.zeros((b, max_c), dtype=torch.long)
+ chunk_idx = torch.zeros((b, max_c), dtype=torch.long)
+ mask = torch.zeros((b, max_c), dtype=torch.bool)
+ labels = torch.zeros((b, max_c), dtype=torch.float32)
+ role_labels = torch.zeros((b, max_c), dtype=torch.long)
+ qids=[]; event_ids=[]
+ for i, sample in enumerate(samples):
+ qids.append(str(sample["question_id"])); event_ids.append([])
+ q_vec[i] = vectorizer.encode_one(str(sample.get("question", "")))
+ for j, cand in enumerate(sample["candidates"]):
+ e_vec[i, j] = vectorizer.encode_one(str(cand.get("text", "")))
+ feat = list(cand.get("features") or [])
+ features[i, j, : min(feature_dim, len(feat))] = torch.tensor(feat[:feature_dim], dtype=torch.float32)
+ session_idx[i, j] = int(cand.get("session_index", 0) or 0)
+ chunk_idx[i, j] = int(cand.get("chunk_index", 0) or 0)
+ mask[i, j] = True
+ labels[i, j] = float(cand.get("label", 0) or 0)
+ role_labels[i, j] = int(cand.get("role_label", 0) or 0)
+ event_ids[-1].append(str(cand.get("event_id", "")))
+ return Batch(q_vec.to(device), e_vec.to(device), features.to(device), session_idx.to(device), chunk_idx.to(device), mask.to(device), labels.to(device), role_labels.to(device), qids, event_ids)
+
+
+def ranking_loss(event_logits: torch.Tensor, labels: torch.Tensor, mask: torch.Tensor) -> torch.Tensor:
+ losses=[]
+ for logits, y, m in zip(event_logits, labels, mask):
+ valid_logits = logits[m]
+ valid_y = y[m]
+ pos = valid_y > 0.5
+ if not bool(pos.any()):
+ continue
+ log_probs = F.log_softmax(valid_logits, dim=0)
+ target = valid_y / valid_y.sum().clamp_min(1.0)
+ losses.append(-(target * log_probs).sum())
+ if not losses:
+ return event_logits.sum() * 0.0
+ return torch.stack(losses).mean()
+
+
+def eval_model(model: V2EvidenceScorer, samples: Sequence[Mapping[str, Any]], vectorizer: Any, args: argparse.Namespace, device: torch.device) -> dict[str, Any]:
+ model.eval()
+ totals={"n":0,"event_recall_at_1":0,"event_recall_at_5":0,"event_recall_at_10":0,"pack_recall_at_10":0,"mrr":0.0}
+ with torch.no_grad():
+ for start in range(0, len(samples), args.batch_size):
+ batch_samples=samples[start:start+args.batch_size]
+ batch=make_batch(batch_samples, vectorizer, args.text_dim, args.feature_dim, device)
+ out=model(batch.q_vec,batch.e_vec,batch.features,batch.session_idx,batch.chunk_idx,batch.mask)
+ event_scores=out["event_logits"].detach().cpu()
+ pack_scores=out["pack_logits"].detach().cpu()
+ labels=batch.labels.detach().cpu()
+ mask=batch.mask.detach().cpu()
+ for i in range(len(batch_samples)):
+ valid=torch.nonzero(mask[i], as_tuple=False).squeeze(-1)
+ if valid.numel()==0: continue
+ pos=set(torch.nonzero(labels[i][valid]>0.5, as_tuple=False).squeeze(-1).tolist())
+ if not pos: continue
+ order=torch.argsort(event_scores[i][valid], descending=True).tolist()
+ pack_order=torch.argsort(pack_scores[i][valid], descending=True).tolist()
+ totals["n"]+=1
+ totals["event_recall_at_1"] += int(any(idx in pos for idx in order[:1]))
+ totals["event_recall_at_5"] += int(any(idx in pos for idx in order[:5]))
+ totals["event_recall_at_10"] += int(any(idx in pos for idx in order[:10]))
+ totals["pack_recall_at_10"] += int(any(idx in pos for idx in pack_order[:10]))
+ first_rank=next((r+1 for r,idx in enumerate(order) if idx in pos), None)
+ totals["mrr"] += 0.0 if first_rank is None else 1.0/first_rank
+ n=max(1, totals["n"])
+ return {k:(round(v/n,6) if k!="n" else v) for k,v in totals.items()}
+
+
+def precompute_embeddings(args: argparse.Namespace) -> None:
+ out_dir = Path(args.out_dir)
+ out_dir.mkdir(parents=True, exist_ok=True)
+ tensor_path = out_dir / "embeddings.pt"
+ manifest_path = out_dir / "manifest.json"
+ items_path = out_dir / "items.jsonl"
+ if (tensor_path.exists() or manifest_path.exists() or items_path.exists()) and not args.overwrite:
+ raise RuntimeError(f"embedding cache output already exists; pass --overwrite explicitly: {out_dir}")
+ samples = read_jsonl(Path(args.samples))
+ if args.limit > 0:
+ samples = samples[: args.limit]
+ if not samples:
+ raise RuntimeError("no samples for embedding precompute")
+ if not args.cpu and not torch.cuda.is_available():
+ raise RuntimeError("CUDA is unavailable; pass --cpu explicitly for CPU precompute")
+ if args.embedding_backend != "hf":
+ raise RuntimeError("precompute currently supports only --embedding-backend hf for local bge-m3")
+ device = torch.device("cpu" if args.cpu else "cuda")
+ vectorizer = BgeM3DenseVectorizer(
+ dim=args.text_dim,
+ model_path=args.embedding_model,
+ device=str(device),
+ max_length=args.embedding_max_length,
+ )
+ items = iter_sample_texts(samples)
+ keys = [item["key"] for item in items]
+ texts = [item["text"] for item in items]
+ vectors: list[torch.Tensor] = []
+ total = len(texts)
+ started = time.time()
+ for start in range(0, total, args.batch_size):
+ batch_texts = texts[start:start + args.batch_size]
+ vecs = vectorizer.encode_batch(batch_texts, batch_size=args.batch_size)
+ vectors.append(vecs.to(dtype=torch.float16 if args.dtype == "float16" else torch.float32))
+ done = min(total, start + len(batch_texts))
+ if done == total or done % max(1, args.log_every) == 0:
+ elapsed = time.time() - started
+ print(json.dumps({"done": done, "total": total, "elapsed_sec": round(elapsed, 3)}, ensure_ascii=False), flush=True)
+ matrix = torch.cat(vectors, dim=0).contiguous()
+ if matrix.shape != (len(keys), args.text_dim):
+ raise RuntimeError(f"embedding matrix shape mismatch: got {tuple(matrix.shape)} expected {(len(keys), args.text_dim)}")
+ torch.save({"keys": keys, "vectors": matrix}, tensor_path)
+ item_rows=[]
+ for item in items:
+ item_rows.append({
+ "key": item["key"],
+ "kind": item.get("kind", ""),
+ "question_id": item.get("question_id", ""),
+ "event_id": item.get("event_id", ""),
+ "text_len": len(item.get("text", "")),
+ "preview": clean_text(item.get("text", ""))[:200],
+ })
+ write_jsonl(items_path, item_rows)
+ manifest = {
+ "created_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
+ "samples": str(Path(args.samples).resolve()),
+ "sample_count": len(samples),
+ "unique_text_count": len(keys),
+ "text_dim": int(args.text_dim),
+ "dtype": args.dtype,
+ "embedding_backend": args.embedding_backend,
+ "embedding_model": clean_text(args.embedding_model),
+ "embedding_max_length": int(args.embedding_max_length),
+ "device": str(device),
+ "outputs": {"tensor": str(tensor_path), "items": str(items_path), "manifest": str(manifest_path)},
+ }
+ manifest_path.write_text(json.dumps(manifest, ensure_ascii=False, indent=2, sort_keys=True), encoding="utf-8")
+ print(json.dumps(manifest, ensure_ascii=False, indent=2, sort_keys=True))
+
+
+def smoke(args: argparse.Namespace) -> None:
+ out_dir = Path(args.out_dir)
+ out_dir.mkdir(parents=True, exist_ok=True)
+ samples = read_jsonl(Path(args.samples))
+ if args.limit > 0:
+ samples = samples[: args.limit]
+ if not samples:
+ raise RuntimeError("no samples for smoke")
+ if not args.cpu and not torch.cuda.is_available():
+ raise RuntimeError("CUDA is unavailable; pass --cpu explicitly for CPU smoke")
+ device = torch.device("cpu" if args.cpu else "cuda")
+ vectorizer = build_vectorizer(args, device)
+ model = V2EvidenceScorer(args.text_dim, args.feature_dim, args.hidden_dim, args.layers).to(device)
+ model.eval()
+ batch = make_batch(samples, vectorizer, args.text_dim, args.feature_dim, device)
+ with torch.no_grad():
+ out = model(batch.q_vec, batch.e_vec, batch.features, batch.session_idx, batch.chunk_idx, batch.mask)
+ loss_rank = ranking_loss(out["event_logits"], batch.labels, batch.mask)
+ pack_loss = F.binary_cross_entropy_with_logits(out["pack_logits"][batch.mask], batch.labels[batch.mask])
+ role_loss = F.cross_entropy(out["role_logits"][batch.mask], batch.role_labels[batch.mask])
+ metrics = eval_model(model, samples, vectorizer, args, device)
+ report = {
+ "status": "ok",
+ "mode": "forward_only_no_optimizer_step",
+ "device": str(device),
+ "sample_count": len(samples),
+ "candidate_count_max": max(len(s.get("candidates", [])) for s in samples),
+ "candidate_count_avg": round(sum(len(s.get("candidates", [])) for s in samples) / max(1, len(samples)), 3),
+ "losses": {
+ "ranking_loss": round(float(loss_rank.detach().cpu()), 6),
+ "pack_loss": round(float(pack_loss.detach().cpu()), 6),
+ "role_loss": round(float(role_loss.detach().cpu()), 6),
+ },
+ "metrics": metrics,
+ "embedding_backend": clean_text(getattr(args, "embedding_backend", "")),
+ "embedding_model": clean_text(getattr(args, "embedding_model", "")),
+ "embedding_cache": clean_text(getattr(args, "embedding_cache", "")),
+ "embedding_max_length": int(getattr(args, "embedding_max_length", 0) or 0),
+ "text_dim": int(args.text_dim),
+ "hidden_dim": int(args.hidden_dim),
+ "layers": int(args.layers),
+ }
+ (out_dir / "smoke_report.json").write_text(json.dumps(report, ensure_ascii=False, indent=2, sort_keys=True), encoding="utf-8")
+ print(json.dumps(report, ensure_ascii=False, indent=2, sort_keys=True))
+
+
+def train(args: argparse.Namespace) -> None:
+ out_dir=Path(args.out_dir); out_dir.mkdir(parents=True, exist_ok=True)
+ samples=read_jsonl(Path(args.samples))
+ if args.limit>0:
+ samples=samples[:args.limit]
+ train_samples=[s for s in samples if s.get("split") == "train"]
+ val_samples=[s for s in samples if s.get("split") == "val"]
+ if not train_samples:
+ raise RuntimeError("no train samples")
+ if not val_samples:
+ raise RuntimeError("no val samples; rebuild dataset with a non-empty validation split")
+ if not args.cpu and not torch.cuda.is_available():
+ raise RuntimeError("CUDA is unavailable; pass --cpu explicitly for CPU train")
+ device=torch.device("cpu" if args.cpu else "cuda")
+ vectorizer=build_vectorizer(args, device)
+ model=V2EvidenceScorer(args.text_dim,args.feature_dim,args.hidden_dim,args.layers).to(device)
+ opt=torch.optim.AdamW(model.parameters(), lr=args.lr, weight_decay=args.weight_decay)
+ best=None
+ history=[]
+ rng=random.Random(args.seed)
+ for epoch in range(1,args.epochs+1):
+ model.train(); rng.shuffle(train_samples)
+ losses=[]
+ for start in range(0,len(train_samples),args.batch_size):
+ batch_samples=train_samples[start:start+args.batch_size]
+ batch=make_batch(batch_samples, vectorizer, args.text_dim, args.feature_dim, device)
+ out=model(batch.q_vec,batch.e_vec,batch.features,batch.session_idx,batch.chunk_idx,batch.mask)
+ loss_rank=ranking_loss(out["event_logits"],batch.labels,batch.mask)
+ bce=F.binary_cross_entropy_with_logits(out["pack_logits"][batch.mask], batch.labels[batch.mask])
+ role=F.cross_entropy(out["role_logits"][batch.mask], batch.role_labels[batch.mask])
+ loss=loss_rank + args.pack_loss_weight*bce + args.role_loss_weight*role
+ opt.zero_grad(set_to_none=True); loss.backward(); nn.utils.clip_grad_norm_(model.parameters(), 1.0); opt.step()
+ losses.append(float(loss.detach().cpu()))
+ val=eval_model(model,val_samples,vectorizer,args,device)
+ train_eval=eval_model(model,train_samples[:min(len(train_samples),100)],vectorizer,args,device)
+ summary={"epoch":epoch,"loss":round(sum(losses)/max(1,len(losses)),6),"train":train_eval,"val":val}
+ history.append(summary)
+ print(json.dumps(summary,ensure_ascii=False), flush=True)
+ score=val.get("event_recall_at_5",0.0)+0.1*val.get("mrr",0.0)
+ if best is None or score > best[0]:
+ best=(score,epoch)
+ torch.save({"model_state":model.state_dict(),"args":vars(args),"epoch":epoch,"val":val}, out_dir/"tmcra_v2_scorer.pt")
+ (out_dir/"train_history.json").write_text(json.dumps(history,ensure_ascii=False,indent=2),encoding="utf-8")
+ report={"best": {"score": best[0] if best else None, "epoch": best[1] if best else None}, "final": history[-1] if history else {}, "checkpoint": str(out_dir/"tmcra_v2_scorer.pt")}
+ (out_dir/"train_report.json").write_text(json.dumps(report,ensure_ascii=False,indent=2,sort_keys=True),encoding="utf-8")
+ print(json.dumps(report,ensure_ascii=False,indent=2,sort_keys=True))
+
+
+def main() -> int:
+ p=argparse.ArgumentParser(description="TMCRA v2 LongMemEval semantic-graph scorer pipeline")
+ sub=p.add_subparsers(dest="cmd", required=True)
+ b=sub.add_parser("build")
+ b.add_argument("--aligned-queries", required=True)
+ b.add_argument("--shard-glob", default=os.getenv("TMCRA_SHARD_GLOB", "runs/shard_*.json"))
+ b.add_argument("--out-dir", required=True)
+ b.add_argument("--val-ratio", type=float, default=0.2)
+ pc=sub.add_parser("precompute")
+ pc.add_argument("--samples", required=True)
+ pc.add_argument("--out-dir", required=True)
+ pc.add_argument("--text-dim", type=int, default=1024)
+ pc.add_argument("--embedding-backend", choices=["hf"], required=True)
+ pc.add_argument("--embedding-model", required=True)
+ pc.add_argument("--embedding-max-length", type=int, default=8192)
+ pc.add_argument("--batch-size", type=int, default=2)
+ pc.add_argument("--log-every", type=int, default=100)
+ pc.add_argument("--dtype", choices=["float16", "float32"], default="float16")
+ pc.add_argument("--limit", type=int, default=0)
+ pc.add_argument("--overwrite", action="store_true")
+ pc.add_argument("--cpu", action="store_true")
+ t=sub.add_parser("train")
+ t.add_argument("--samples", required=True)
+ t.add_argument("--out-dir", required=True)
+ t.add_argument("--text-dim", type=int, default=1024)
+ t.add_argument("--embedding-backend", choices=["openai", "hf"], required=True)
+ t.add_argument("--embedding-base-url", default="")
+ t.add_argument("--embedding-model", default="")
+ t.add_argument("--embedding-cache", default="")
+ t.add_argument("--embedding-api-key", default="")
+ t.add_argument("--embedding-max-length", type=int, default=8192)
+ t.add_argument("--feature-dim", type=int, default=6)
+ t.add_argument("--hidden-dim", type=int, default=256)
+ t.add_argument("--layers", type=int, default=2)
+ t.add_argument("--batch-size", type=int, default=8)
+ t.add_argument("--epochs", type=int, default=8)
+ t.add_argument("--lr", type=float, default=2e-4)
+ t.add_argument("--weight-decay", type=float, default=0.01)
+ t.add_argument("--pack-loss-weight", type=float, default=0.3)
+ t.add_argument("--role-loss-weight", type=float, default=0.15)
+ t.add_argument("--seed", type=int, default=13)
+ t.add_argument("--limit", type=int, default=0)
+ t.add_argument("--cpu", action="store_true")
+ sm=sub.add_parser("smoke")
+ sm.add_argument("--samples", required=True)
+ sm.add_argument("--out-dir", required=True)
+ sm.add_argument("--text-dim", type=int, default=1024)
+ sm.add_argument("--embedding-backend", choices=["openai", "hf"], required=True)
+ sm.add_argument("--embedding-base-url", default="")
+ sm.add_argument("--embedding-model", default="")
+ sm.add_argument("--embedding-cache", default="")
+ sm.add_argument("--embedding-api-key", default="")
+ sm.add_argument("--embedding-max-length", type=int, default=8192)
+ sm.add_argument("--feature-dim", type=int, default=6)
+ sm.add_argument("--hidden-dim", type=int, default=96)
+ sm.add_argument("--layers", type=int, default=1)
+ sm.add_argument("--batch-size", type=int, default=8)
+ sm.add_argument("--limit", type=int, default=4)
+ sm.add_argument("--cpu", action="store_true")
+ args=p.parse_args()
+ if args.cmd == "build": build_dataset(args)
+ elif args.cmd == "precompute": precompute_embeddings(args)
+ elif args.cmd == "train": train(args)
+ elif args.cmd == "smoke": smoke(args)
+ return 0
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v3_online_runtime.py b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v3_online_runtime.py
new file mode 100644
index 0000000000000000000000000000000000000000..de68fef25daa0f4f51dc8998f74f1b2bc3479341
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v3_online_runtime.py
@@ -0,0 +1,1867 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import hashlib
+import importlib.util
+import json
+import os
+import re
+import sqlite3
+import sys
+import time
+from collections import Counter
+from pathlib import Path
+from typing import Any, Iterable, Mapping, Sequence
+
+import torch
+
+from build_v3_runtime_dataset import covered_windows, rrank
+from tmcra_v2_lme_pipeline import BgeM3DenseVectorizer
+from tmcra_v3_reranker import ChannelAwareMemoryReranker
+from tmcra_model_assets import load_optional_hf_model_manifest
+from tmcra_v3_schema import CHANNEL_NAMES, SCHEMA_VERSION, clean_text, write_jsonl
+from tmcra_v3_recall_planner import (
+ DEEPSEEK_FLASH_MODEL,
+ DeepSeekFlashRecallPlanner,
+ RecallPlannerError,
+ apply_recall_plan,
+)
+
+
+LEGACY_CHUNK_ID_RE = re.compile(r"^s(?P\d+)_c(?P\d+)$")
+MESSAGE_ID_RE = re.compile(r"^s(?P\d+)_m(?P\d+)$")
+EVENT_PARENT_RE = re.compile(r":s(?P\d+)_[cm](?P\d+)$")
+CURRENT_FAST_STATES = frozenset(
+ {"active", "parallel_active", "promoted", "challenged"}
+)
+ONLINE_INDEX_SCHEMA_VERSION = "tmcra.v3.online-index.3"
+FAST_SEMANTIC_STATE_POLICY = (
+ "current-fast-states-v1:" + ",".join(sorted(CURRENT_FAST_STATES))
+)
+INGEST_PREFIX_RE = re.compile(r"^\[[^\]\n]+\]\s+user:\s*", flags=re.IGNORECASE)
+SESSION_HEADER_RE = re.compile(
+ r"^LongMemEval session_id=(?P\S+) date=(?P.+?)(?: continued=true)? \[",
+ flags=re.DOTALL,
+)
+
+
+def read_jsonl(path: Path) -> list[dict[str, Any]]:
+ rows: list[dict[str, Any]] = []
+ with path.open("r", encoding="utf-8", errors="strict") as handle:
+ for line_no, line in enumerate(handle, start=1):
+ if not line.strip():
+ continue
+ value = json.loads(line)
+ if not isinstance(value, dict):
+ raise RuntimeError(f"row is not an object at {path}:{line_no}")
+ rows.append(value)
+ if not rows:
+ raise RuntimeError(f"no rows: {path}")
+ return rows
+
+
+def atomic_torch_save(payload: Mapping[str, Any], target: Path) -> None:
+ target.parent.mkdir(parents=True, exist_ok=True)
+ temporary = target.with_suffix(target.suffix + ".tmp")
+ torch.save(dict(payload), temporary)
+ os.replace(temporary, target)
+
+
+def sha256_file(path: Path) -> str:
+ digest = hashlib.sha256()
+ with path.open("rb") as handle:
+ for chunk in iter(lambda: handle.read(1024 * 1024), b""):
+ digest.update(chunk)
+ return digest.hexdigest()
+
+
+def load_native_harness(path: Path, repo: Path):
+ resolved_repo = str(repo.resolve())
+ if resolved_repo not in sys.path:
+ sys.path.insert(0, resolved_repo)
+ module_name = "tmcra_v3_native_runtime_harness"
+ spec = importlib.util.spec_from_file_location(module_name, path)
+ if spec is None or spec.loader is None:
+ raise RuntimeError(f"cannot import native TMCRA harness: {path}")
+ module = importlib.util.module_from_spec(spec)
+ sys.modules[module_name] = module
+ spec.loader.exec_module(module)
+ return module
+
+
+def graph_runtime_env(args: argparse.Namespace) -> dict[str, str]:
+ values = {
+ "TMCRA_RETRIEVAL_MODE": "hybrid_node_scored",
+ "TMCRA_NODE_MODEL_PATH": str(Path(args.node_model).resolve()),
+ "TMCRA_PATH_MODEL_PATH": str(Path(args.path_model).resolve()),
+ "TMCRA_NODE_MODEL_DEVICE": args.graph_device,
+ "TMCRA_SUPPORT_PATH_K": str(args.support_path_k),
+ "TMCRA_PATH_TUNNEL_RESCUE_K": str(args.path_tunnel_rescue_k),
+ "TMCRA_CANDIDATE_EVENT_K": str(args.candidate_event_k),
+ "TMCRA_DEEPSEEK_GRAPH_MODEL_MODE": "off",
+ "TMCRA_TOPIC_BUCKET_MODE": "off",
+ "TMCRA_LLM_CHANNEL_PLANNER_MODE": "off",
+ "TMCRA_LLM_EVIDENCE_SELECTOR_MODE": "off",
+ "TMCRA_EVIDENCE_UNIT_PLANNER_MODE": "off",
+ "TMCRA_UNIFIED_OPERATION_PLANNER_MODE": "off",
+ }
+ for name, value in values.items():
+ os.environ[name] = value
+ return values
+
+
+def scope_counts(db_path: Path, scope_id: str) -> dict[str, int]:
+ with sqlite3.connect(db_path) as connection:
+ output = {}
+ for table in ("records", "memory_edges", "audit_turn_log", "audit_retrieval_log"):
+ output[table] = int(
+ connection.execute(f'SELECT COUNT(*) FROM "{table}" WHERE scope_id=?', (scope_id,)).fetchone()[0]
+ )
+ return output
+
+
+def scope_fingerprint(db_path: Path, scope_id: str) -> str:
+ relevant_tables = ("records", "memory_edges", "slot_heads", "slot_history")
+ snapshot: dict[str, Any] = {}
+ with sqlite3.connect(db_path) as connection:
+ connection.row_factory = sqlite3.Row
+ for table in relevant_tables:
+ exists = connection.execute(
+ "SELECT 1 FROM sqlite_master WHERE type='table' AND name=?", (table,)
+ ).fetchone()
+ if exists is None:
+ snapshot[table] = None
+ continue
+ rows = []
+ for row in connection.execute(
+ f'SELECT * FROM "{table}" WHERE scope_id=?', (scope_id,)
+ ):
+ normalized: dict[str, Any] = {}
+ for key in row.keys():
+ value = row[key]
+ if key.endswith("_json") and isinstance(value, str):
+ try:
+ value = json.loads(value)
+ except json.JSONDecodeError as exc:
+ raise RuntimeError(
+ f"invalid JSON in {table}.{key} while fingerprinting scope"
+ ) from exc
+ normalized[key] = value
+ rows.append(normalized)
+ snapshot[table] = sorted(
+ rows,
+ key=lambda item: json.dumps(
+ item, ensure_ascii=False, sort_keys=True, separators=(",", ":")
+ ),
+ )
+ return hashlib.sha256(
+ json.dumps(
+ snapshot, ensure_ascii=False, sort_keys=True, separators=(",", ":")
+ ).encode("utf-8")
+ ).hexdigest()
+
+
+def load_recent_dialogue_context(
+ db_path: Path,
+ scope_id: str,
+ *,
+ current_query: str,
+ limit: int = 8,
+) -> list[dict[str, Any]]:
+ """Load a small, metadata-free dialogue tail for planner reference resolution."""
+ if limit <= 0 or limit > 8:
+ raise RuntimeError("recent dialogue limit must be between 1 and 8")
+ with sqlite3.connect(db_path) as connection:
+ columns = {
+ str(row[1]) for row in connection.execute('PRAGMA table_info("audit_turn_log")')
+ }
+ required = {"scope_id", "event_index", "payload_json"}
+ if not required.issubset(columns):
+ raise RuntimeError(
+ "audit_turn_log lacks required production dialogue columns: "
+ + ",".join(sorted(required - columns))
+ )
+ rows = connection.execute(
+ 'SELECT event_index,payload_json FROM "audit_turn_log" '
+ "WHERE scope_id=? ORDER BY event_index DESC LIMIT ?",
+ (scope_id, limit + 1),
+ ).fetchall()
+ dialogue: list[dict[str, Any]] = []
+ for event_index, raw_payload in reversed(rows):
+ try:
+ payload = json.loads(raw_payload)
+ except (TypeError, json.JSONDecodeError) as exc:
+ raise RuntimeError(f"invalid audit_turn_log payload at event_index={event_index}") from exc
+ if not isinstance(payload, Mapping):
+ raise RuntimeError(f"audit_turn_log payload is not an object at event_index={event_index}")
+ speaker = clean_text(payload.get("speaker")).lower()
+ text = clean_text(payload.get("text"))
+ if speaker not in {"user", "assistant"} or not text:
+ raise RuntimeError(f"audit_turn_log turn is not planner-safe at event_index={event_index}")
+ dialogue.append(
+ {"turn_index": int(event_index), "speaker": speaker, "text": text}
+ )
+ current = clean_text(current_query)
+ if dialogue and dialogue[-1]["speaker"] == "user" and dialogue[-1]["text"] == current:
+ dialogue.pop()
+ return dialogue[-limit:]
+
+
+def append_layered_retrieval_audit(
+ *,
+ repo: Path,
+ db_path: Path,
+ scope_id: str,
+ operation_id: str,
+ evidence: Mapping[str, Any],
+ debug: Mapping[str, Any],
+) -> dict[str, Any]:
+ """Persist the final layered retrieval decision without rewriting graph state."""
+ resolved_repo = str(repo.resolve())
+ if resolved_repo not in sys.path:
+ sys.path.insert(0, resolved_repo)
+ from experiments.replacement.memory_graph import SQLiteSessionMemoryStore
+
+ with sqlite3.connect(db_path) as connection:
+ row = connection.execute(
+ "SELECT value_json FROM meta WHERE scope_id=? AND key='audit_retention'",
+ (scope_id,),
+ ).fetchone()
+ if row is None:
+ raise RuntimeError(f"{scope_id}: missing audit_retention metadata")
+ audit_retention = int(json.loads(row[0]) or 0)
+ if audit_retention <= 0:
+ raise RuntimeError(f"{scope_id}: invalid audit_retention metadata: {audit_retention}")
+
+ windows = list(evidence.get("evidence_windows") or [])
+ memory_contexts = [
+ context
+ for window in windows
+ for context in list(window.get("memory_contexts") or [])
+ ]
+ graph = dict(debug.get("graph") or {})
+ payload = {
+ "event_kind": "tmcra.v3.layered_retrieval",
+ "schema_version": "tmcra.v3.layered-retrieval-audit.1",
+ "operation_id": clean_text(operation_id),
+ "question_id": clean_text(evidence.get("question_id")),
+ "query": clean_text(evidence.get("question")),
+ "question_date": clean_text(evidence.get("question_date")),
+ "recall_plan": dict(evidence.get("recall_plan") or {}),
+ "selected_memory_ids": [
+ clean_text(window.get("memory_id"))
+ for window in windows
+ if clean_text(window.get("memory_id"))
+ ],
+ "selected_session_ids": list(evidence.get("selected_session_ids") or []),
+ "selected_capsule_ids": sorted(
+ {
+ clean_text(context.get("capsule_id"))
+ for context in memory_contexts
+ if clean_text(context.get("capsule_id"))
+ }
+ ),
+ "selected_claim_ids": sorted(
+ {
+ clean_text(dict(context.get("provenance") or {}).get("claim_id"))
+ for context in memory_contexts
+ if clean_text(dict(context.get("provenance") or {}).get("claim_id"))
+ }
+ ),
+ "selected_graph_event_ids": list(graph.get("selected_event_ids") or []),
+ "runtime_input_has_gold": bool(debug.get("runtime_input_has_gold")),
+ "graph_fingerprint": clean_text(debug.get("graph_fingerprint")),
+ "evidence_sha256": hashlib.sha256(
+ json.dumps(
+ dict(evidence),
+ ensure_ascii=False,
+ sort_keys=True,
+ separators=(",", ":"),
+ ).encode("utf-8")
+ ).hexdigest(),
+ }
+ if payload["runtime_input_has_gold"]:
+ raise RuntimeError(f"{scope_id}: refusing to audit retrieval with evaluation labels")
+ if not payload["operation_id"]:
+ raise RuntimeError(f"{scope_id}: layered retrieval audit lacks operation_id")
+ store = SQLiteSessionMemoryStore(str(db_path), audit_retention=audit_retention)
+ return dict(
+ store.append_audit_event(
+ scope_id,
+ "retrieval_log",
+ payload,
+ idempotency_key=payload["operation_id"],
+ )
+ )
+
+
+def load_persisted_parent_chunks(db_path: Path, scope_id: str) -> list[dict[str, Any]]:
+ if not db_path.exists():
+ raise FileNotFoundError(db_path)
+ with sqlite3.connect(db_path) as connection:
+ record_rows = connection.execute(
+ "SELECT memory_id, turn_index, metadata_json FROM records WHERE scope_id=?",
+ (scope_id,),
+ ).fetchall()
+ if not record_rows:
+ raise RuntimeError(f"scope has no graph records: {scope_id}")
+ product_parents: list[dict[str, Any]] = []
+ product_message_ids: set[str] = set()
+ for memory_id, turn_index, raw_metadata in record_rows:
+ metadata = json.loads(raw_metadata)
+ if clean_text(metadata.get("content_variant")) != "source_message":
+ continue
+ message_id = clean_text(metadata.get("message_id"))
+ match = MESSAGE_ID_RE.fullmatch(message_id)
+ if match is None:
+ raise RuntimeError(f"{scope_id}: malformed persisted product message id: {message_id!r}")
+ session_index = int(metadata.get("session_index", -1))
+ message_index = int(metadata.get("message_index", -1))
+ if session_index != int(match.group("session")) or message_index != int(match.group("parent")):
+ raise RuntimeError(f"{scope_id}: product message location metadata disagrees with {message_id}")
+ raw_content = metadata.get("raw_content")
+ if not isinstance(raw_content, str) or not raw_content:
+ raise RuntimeError(f"{scope_id}: product source {message_id} has no persisted raw content")
+ if message_id in product_message_ids:
+ raise RuntimeError(f"{scope_id}: duplicate immutable product source message: {message_id}")
+ product_message_ids.add(message_id)
+ role = clean_text(metadata.get("speaker") or dict(metadata.get("sidecar_hint_metadata") or {}).get("role"))
+ session_id = clean_text(metadata.get("session_id") or dict(metadata.get("sidecar_hint_metadata") or {}).get("session_id"))
+ date = clean_text(metadata.get("historical_date") or dict(metadata.get("sidecar_hint_metadata") or {}).get("historical_date"))
+ timestamp = clean_text(metadata.get("timestamp"))
+ if role not in {"user", "assistant", "system", "tool"} or not session_id or not date or not timestamp:
+ raise RuntimeError(f"{scope_id}: incomplete product source metadata for {message_id}")
+ product_parents.append(
+ {
+ "chunk_id": message_id,
+ "parent_kind": "message",
+ "session_index": session_index,
+ "parent_chunk_index": message_index,
+ "message_index": message_index,
+ "session_id": session_id,
+ "date": date,
+ "timestamp": timestamp,
+ "role": role,
+ "text": raw_content,
+ "turn_index": int(turn_index),
+ "source_record_id": clean_text(memory_id),
+ }
+ )
+
+ audit_rows = connection.execute(
+ "SELECT event_index, payload_json FROM audit_turn_log WHERE scope_id=? ORDER BY event_index",
+ (scope_id,),
+ ).fetchall()
+ if product_parents:
+ audit_message_ids = {
+ clean_text(dict(json.loads(raw_payload).get("metadata") or {}).get("message_id"))
+ for _, raw_payload in audit_rows
+ if clean_text(dict(json.loads(raw_payload).get("metadata") or {}).get("message_id"))
+ }
+ if audit_message_ids != product_message_ids:
+ raise RuntimeError(
+ f"{scope_id}: immutable source/audit message sets differ: "
+ f"source={len(product_message_ids)} audit={len(audit_message_ids)}"
+ )
+ product_parents.sort(key=lambda row: (row["session_index"], row["message_index"]))
+ return product_parents
+
+ chunk_by_record: dict[str, str] = {}
+ chunks_by_turn: dict[int, set[str]] = {}
+ for memory_id, turn_index, raw_metadata in record_rows:
+ metadata = json.loads(raw_metadata)
+ sidecar = dict(metadata.get("sidecar_hint_metadata") or {})
+ chunk_id = clean_text(sidecar.get("chunk_id"))
+ if not chunk_id:
+ dia_id = clean_text(metadata.get("dia_id"))
+ chunk_id = dia_id.rsplit(":", 1)[-1] if dia_id else ""
+ if not LEGACY_CHUNK_ID_RE.match(chunk_id):
+ continue
+ chunk_by_record[clean_text(memory_id)] = chunk_id
+ chunks_by_turn.setdefault(int(turn_index), set()).add(chunk_id)
+
+ if not audit_rows:
+ raise RuntimeError(f"scope has no persisted write audit: {scope_id}")
+
+ parents: list[dict[str, Any]] = []
+ seen_locations: set[tuple[int, int]] = set()
+ for event_index, raw_payload in audit_rows:
+ payload = json.loads(raw_payload)
+ if clean_text(payload.get("kind")) != "memory_write":
+ continue
+ turn_index = int(payload.get("turn_index", 0) or 0)
+ record_ids = [clean_text(value) for value in list(payload.get("record_ids") or [])]
+ candidate_chunks = {chunk_by_record[value] for value in record_ids if value in chunk_by_record}
+ candidate_chunks.update(chunks_by_turn.get(turn_index, set()))
+ if len(candidate_chunks) != 1:
+ raise RuntimeError(
+ f"{scope_id}: persisted turn {turn_index} maps to {len(candidate_chunks)} chunk ids: {sorted(candidate_chunks)}"
+ )
+ chunk_id = next(iter(candidate_chunks))
+ match = LEGACY_CHUNK_ID_RE.match(chunk_id)
+ assert match is not None
+ session_index = int(match.group("session"))
+ parent_index = int(match.group("parent"))
+ location = (session_index, parent_index)
+ if location in seen_locations:
+ raise RuntimeError(f"{scope_id}: duplicate persisted parent location: {location}")
+ seen_locations.add(location)
+ persisted_text = str(payload.get("text", ""))
+ parent_text = INGEST_PREFIX_RE.sub("", persisted_text, count=1)
+ if parent_text == persisted_text or not parent_text:
+ raise RuntimeError(f"{scope_id}: cannot recover raw parent text from audit turn {turn_index}")
+ header = SESSION_HEADER_RE.match(parent_text)
+ if header is None:
+ raise RuntimeError(f"{scope_id}: malformed persisted LongMemEval parent header at {chunk_id}")
+ parents.append(
+ {
+ "chunk_id": chunk_id,
+ "parent_kind": "legacy_chunk",
+ "session_index": session_index,
+ "parent_chunk_index": parent_index,
+ "session_id": clean_text(header.group("session_id")),
+ "date": clean_text(header.group("date")),
+ "text": parent_text,
+ "turn_index": turn_index,
+ "audit_event_index": int(event_index),
+ }
+ )
+ parents.sort(key=lambda row: (row["session_index"], row["parent_chunk_index"]))
+ if len(parents) != len(audit_rows):
+ raise RuntimeError(
+ f"{scope_id}: not every persisted audit turn became a parent chunk: parents={len(parents)} audit={len(audit_rows)}"
+ )
+ return parents
+
+
+def parent_subchunks(
+ parents: Sequence[Mapping[str, Any]],
+ *,
+ scope_id: str,
+ subchunk_chars: int,
+ subchunk_overlap: int,
+) -> list[dict[str, Any]]:
+ candidates: list[dict[str, Any]] = []
+ for parent in parents:
+ if clean_text(parent.get("parent_kind")) == "message":
+ prefix = (
+ f"TMCRA conversation_id={parent['session_id']} timestamp={parent['timestamp']} "
+ f"message={int(parent['message_index']):03d} role={parent['role']}"
+ )
+ else:
+ prefix = (
+ f"LongMemEval session_id={parent['session_id']} date={parent['date']} "
+ f"parent_chunk={int(parent['parent_chunk_index']):02d}"
+ )
+ payload_chars = int(subchunk_chars) - len(prefix) - 1
+ spans = covered_windows(str(parent["text"]), payload_chars, int(subchunk_overlap))
+ for subchunk_index, (char_start, char_end) in enumerate(spans, start=1):
+ text = f"{prefix}\n{str(parent['text'])[char_start:char_end]}"
+ if len(text) > subchunk_chars:
+ raise RuntimeError("generated online subchunk exceeds the strict character limit")
+ candidates.append(
+ {
+ "candidate_id": f"chunk::{scope_id}:{parent['chunk_id']}_p{subchunk_index:02d}",
+ "text": text,
+ "session_id": parent["session_id"],
+ "session_index": int(parent["session_index"]),
+ "parent_chunk_index": int(parent["parent_chunk_index"]),
+ "parent_kind": clean_text(parent.get("parent_kind")) or "legacy_chunk",
+ "role": clean_text(parent.get("role")),
+ "message_role": clean_text(parent.get("role")),
+ "historical_date": clean_text(parent.get("date")),
+ "timestamp": clean_text(parent.get("timestamp")),
+ "subchunk_index": subchunk_index,
+ "source_char_start": char_start,
+ "source_char_end": char_end,
+ "source_record_id": clean_text(parent.get("source_record_id")),
+ }
+ )
+ if not candidates:
+ raise RuntimeError(f"no online candidates from persisted scope: {scope_id}")
+ return candidates
+
+
+def _metadata_json(value: Any, *, label: str) -> dict[str, Any]:
+ try:
+ parsed = json.loads(value)
+ except (TypeError, json.JSONDecodeError) as exc:
+ raise RuntimeError(f"invalid {label} metadata JSON") from exc
+ if not isinstance(parsed, dict):
+ raise RuntimeError(f"invalid {label} metadata object")
+ return parsed
+
+
+def _canonical_slot(value: Any, *, label: str) -> str:
+ slot = clean_text(value)
+ if not slot:
+ raise RuntimeError(f"{label} lacks canonical_slot")
+ return slot
+
+
+def _normalize_source_parents(value: Any, *, valid_locations: set[tuple[int, int]], label: str) -> list[dict[str, Any]]:
+ if not isinstance(value, Sequence) or isinstance(value, (str, bytes)) or not value:
+ raise RuntimeError(f"{label} lacks structured source_parents")
+ output: list[dict[str, Any]] = []
+ seen: set[tuple[int, int, int, int, str]] = set()
+ for raw in value:
+ if not isinstance(raw, Mapping):
+ raise RuntimeError(f"{label} source_parent is not an object")
+ try:
+ session = int(raw["session_index"])
+ parent = int(raw.get("parent_chunk_index", raw.get("message_index")))
+ except (KeyError, TypeError, ValueError) as exc:
+ raise RuntimeError(f"{label} source_parent lacks integer session/parent coordinates") from exc
+ message_index: int | None = None
+ if "message_index" in raw:
+ try:
+ message_index = int(raw["message_index"])
+ except (TypeError, ValueError) as exc:
+ raise RuntimeError(
+ f"{label} source_parent has a non-integer message_index"
+ ) from exc
+ if message_index != parent:
+ raise RuntimeError(
+ f"{label} source_parent message_index differs from parent_chunk_index"
+ )
+ location = (session, parent)
+ if location not in valid_locations:
+ raise RuntimeError(f"{label} source_parent cannot map to a persisted parent: {location}")
+ try:
+ char_start = int(raw["evidence_char_start"])
+ char_end = int(raw["evidence_char_end"])
+ except (KeyError, TypeError, ValueError) as exc:
+ raise RuntimeError(
+ f"{label} source_parent lacks integer evidence character span"
+ ) from exc
+ source_record_id = clean_text(raw.get("source_record_id"))
+ if char_start < 0 or char_end <= char_start or not source_record_id:
+ raise RuntimeError(f"{label} source_parent has invalid evidence provenance")
+ identity = (session, parent, char_start, char_end, source_record_id)
+ if identity not in seen:
+ seen.add(identity)
+ item = dict(raw)
+ item["session_index"] = session
+ item["parent_chunk_index"] = parent
+ item["evidence_char_start"] = char_start
+ item["evidence_char_end"] = char_end
+ item["source_record_id"] = source_record_id
+ if message_index is not None:
+ item["message_index"] = message_index
+ output.append(item)
+ if not output:
+ raise RuntimeError(f"{label} has no mapped source_parents")
+ return output
+
+
+def load_layered_inventory(
+ db_path: Path, scope_id: str, parents: Sequence[Mapping[str, Any]]
+) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
+ """Load current slow heads and map immutable fast semantic leaves to parents."""
+ valid_locations = {(int(item["session_index"]), int(item["parent_chunk_index"])) for item in parents}
+ slow_rows: dict[str, list[tuple[str, str, dict[str, Any]]]] = {}
+ semantic: list[dict[str, Any]] = []
+ with sqlite3.connect(db_path) as con:
+ rows = con.execute(
+ "SELECT memory_id, value, state, metadata_json FROM records WHERE scope_id=?", (scope_id,)
+ ).fetchall()
+ for memory_id, value, state, raw_metadata in rows:
+ metadata = _metadata_json(raw_metadata, label=str(memory_id))
+ layer = clean_text(metadata.get("memory_layer"))
+ variant = clean_text(metadata.get("content_variant"))
+ if layer == "slow" or variant == "slow_memory_capsule":
+ if layer != "slow" or variant != "slow_memory_capsule":
+ raise RuntimeError(f"{memory_id}: malformed slow capsule layer/variant")
+ capsule_id = clean_text(metadata.get("capsule_id"))
+ revision = metadata.get("revision")
+ if not capsule_id or not isinstance(revision, int) or revision < 1:
+ raise RuntimeError(f"{memory_id}: invalid capsule_id or revision")
+ slow_rows.setdefault(capsule_id, []).append((str(memory_id), str(state), {**metadata, "value": str(value)}))
+ continue
+ if layer != "fast" or variant != "product_semantic_memory":
+ continue
+ if clean_text(state) not in CURRENT_FAST_STATES:
+ continue
+ if (
+ clean_text(metadata.get("node_kind")) != "atomic_user_assertion"
+ or metadata.get("atomic_evidence_leaf") is not True
+ or clean_text(metadata.get("authority")) != "user_assertion"
+ ):
+ raise RuntimeError(f"{memory_id}: malformed fast semantic evidence leaf")
+ slot = clean_text(metadata.get("canonical_slot") or metadata.get("canonical_slot_key"))
+ if not slot:
+ # Non-slot fast leaves still remain retrievable through their source parent.
+ continue
+ try:
+ location = (int(metadata["session_index"]), int(metadata.get("parent_chunk_index", metadata.get("message_index"))))
+ except (KeyError, TypeError, ValueError) as exc:
+ raise RuntimeError(f"{memory_id}: fast semantic record lacks source parent location") from exc
+ if location not in valid_locations:
+ raise RuntimeError(f"{memory_id}: fast semantic record cannot map to persisted parent {location}")
+ source_record_id = clean_text(metadata.get("source_record_id"))
+ if not source_record_id:
+ raise RuntimeError(f"{memory_id}: fast semantic leaf lacks metadata.source_record_id")
+ try:
+ evidence_char_start = int(metadata["evidence_char_start"])
+ evidence_char_end = int(metadata["evidence_char_end"])
+ except (KeyError, TypeError, ValueError) as exc:
+ raise RuntimeError(
+ f"{memory_id}: fast semantic leaf lacks evidence character span"
+ ) from exc
+ if evidence_char_start < 0 or evidence_char_end <= evidence_char_start:
+ raise RuntimeError(f"{memory_id}: fast semantic leaf has invalid evidence character span")
+ semantic.append({
+ "memory_id": str(memory_id), "record_state": clean_text(state), "canonical_slot": slot, "source_parent": {
+ "session_index": location[0], "parent_chunk_index": location[1],
+ "source_record_id": source_record_id,
+ "evidence_char_start": evidence_char_start,
+ "evidence_char_end": evidence_char_end,
+ }, "provenance": {"memory_layer": "fast", "content_variant": variant, "source_record_id": source_record_id, "semantic_memory_id": str(memory_id)},
+ })
+ capsules: list[dict[str, Any]] = []
+ for capsule_id, revisions in slow_rows.items():
+ max_revision = max(int(meta["revision"]) for _, _, meta in revisions)
+ current = [(memory_id, state, meta) for memory_id, state, meta in revisions if int(meta["revision"]) == max_revision]
+ if len(current) != 1:
+ raise RuntimeError(f"{scope_id}: capsule {capsule_id} lacks a unique latest active/challenged revision")
+ memory_id, state, meta = current[0]
+ if state != "active" or clean_text(meta.get("status")) not in {"active", "challenged"}:
+ continue
+ claims = meta.get("claims")
+ if not isinstance(claims, Sequence) or isinstance(claims, (str, bytes)) or not claims:
+ raise RuntimeError(f"{memory_id}: capsule lacks claims")
+ capsule_parents = meta.get("source_parents")
+ for claim_index, claim in enumerate(claims):
+ if not isinstance(claim, Mapping):
+ raise RuntimeError(f"{memory_id}: claim {claim_index} is not an object")
+ slot = _canonical_slot(claim.get("canonical_slot"), label=f"{memory_id}: claim {claim_index}")
+ source_parents = _normalize_source_parents(
+ claim.get("source_parents", capsule_parents), valid_locations=valid_locations,
+ label=f"{memory_id}: claim {claim_index}",
+ )
+ claim_text = clean_text(claim.get("text"))
+ claim_id = clean_text(claim.get("claim_id"))
+ if not claim_text or not claim_id:
+ raise RuntimeError(f"{memory_id}: claim {claim_index} lacks claim_id or text")
+ capsules.append({
+ "candidate_id": f"capsule::{capsule_id}:r{meta['revision']}:c{claim_index}",
+ "memory_id": memory_id, "capsule_id": capsule_id, "revision": int(meta["revision"]),
+ "status": clean_text(meta.get("status")), "canonical_slot": slot, "claims": [dict(claim)],
+ "source_parents": source_parents, "text": claim_text,
+ "provenance": {"memory_layer": "slow", "content_variant": "slow_memory_capsule", "capsule_id": capsule_id, "revision": int(meta["revision"]), "claim_id": claim_id, "canonical_slot": slot, "patch_id": clean_text(meta.get("patch_id")), "source_parents": source_parents},
+ })
+ return capsules, semantic
+
+
+def command_build_index(args: argparse.Namespace) -> None:
+ rows = read_jsonl(Path(args.scope_manifest))
+ device = torch.device(args.device)
+ if device.type == "cuda" and not torch.cuda.is_available():
+ raise RuntimeError("CUDA is unavailable")
+ vectorizer: BgeM3DenseVectorizer | None = None
+ started = time.time()
+ report_rows: list[dict[str, Any]] = []
+ reused_index_count = 0
+ for row_index, row in enumerate(rows, start=1):
+ db_path = Path(row["db_path"]).resolve()
+ scope_id = clean_text(row.get("scope_id"))
+ index_path = Path(row["index_path"]).resolve()
+ if index_path.exists():
+ payload = torch.load(index_path, map_location="cpu", weights_only=True)
+ if not isinstance(payload, Mapping):
+ raise RuntimeError(f"existing online index is not an object: {index_path}")
+ expected = {
+ "schema_version": ONLINE_INDEX_SCHEMA_VERSION,
+ "fast_semantic_state_policy": FAST_SEMANTIC_STATE_POLICY,
+ "scope_id": scope_id,
+ "db_path": str(db_path),
+ "subchunk_chars": int(args.subchunk_chars),
+ "subchunk_overlap": int(args.subchunk_overlap),
+ "embedding_model": str(Path(args.embedding_model).resolve()),
+ "embedding_max_length": int(args.embedding_max_length),
+ "strict_no_truncation": True,
+ }
+ mismatches = {
+ key: {"expected": value, "actual": payload.get(key)}
+ for key, value in expected.items()
+ if payload.get(key) != value
+ }
+ current_fingerprint = scope_fingerprint(db_path, scope_id)
+ if clean_text(payload.get("graph_fingerprint")) != current_fingerprint:
+ mismatches["graph_fingerprint"] = {
+ "expected": current_fingerprint,
+ "actual": payload.get("graph_fingerprint"),
+ }
+ if mismatches:
+ raise RuntimeError(
+ f"existing online index is incompatible with current scope: "
+ f"{index_path} mismatches={json.dumps(mismatches, sort_keys=True)}"
+ )
+ report_row = {
+ "question_id": clean_text(row.get("question_id")),
+ "scope_id": scope_id,
+ "db_path": str(db_path),
+ "index_path": str(index_path),
+ "parent_count": int(payload["parent_count"]),
+ "candidate_count": int(payload["candidate_count"]),
+ "slow_capsule_count": int(payload["slow_capsule_count"]),
+ "fast_semantic_record_count": int(payload["fast_semantic_record_count"]),
+ "fast_semantic_state_policy": payload["fast_semantic_state_policy"],
+ "graph_counts": dict(payload["graph_counts_at_index"]),
+ "graph_fingerprint": current_fingerprint,
+ "reused_existing_index": True,
+ }
+ reused_index_count += 1
+ report_rows.append(report_row)
+ print(
+ json.dumps(
+ {"status": "index_reused", "row": row_index, "total": len(rows), **report_row}
+ ),
+ flush=True,
+ )
+ continue
+ if vectorizer is None:
+ vectorizer = BgeM3DenseVectorizer(
+ dim=args.text_dim,
+ model_path=args.embedding_model,
+ device=str(device),
+ max_length=args.embedding_max_length,
+ strict_max_length=True,
+ )
+ graph_fingerprint = scope_fingerprint(db_path, scope_id)
+ parents = load_persisted_parent_chunks(db_path, scope_id)
+ candidates = parent_subchunks(
+ parents,
+ scope_id=scope_id,
+ subchunk_chars=args.subchunk_chars,
+ subchunk_overlap=args.subchunk_overlap,
+ )
+ slow_capsules, fast_semantic_records = load_layered_inventory(db_path, scope_id, parents)
+ fast_vectors = vectorizer.encode_batch([candidate["text"] for candidate in candidates], batch_size=args.batch_size)
+ slow_vectors = (
+ vectorizer.encode_batch([capsule["text"] for capsule in slow_capsules], batch_size=args.batch_size)
+ if slow_capsules else torch.empty((0, args.text_dim), dtype=torch.float32)
+ )
+ fast_vectors = fast_vectors.to(torch.float16).contiguous()
+ slow_vectors = slow_vectors.to(torch.float16).contiguous()
+ counts = scope_counts(db_path, scope_id)
+ if scope_fingerprint(db_path, scope_id) != graph_fingerprint:
+ raise RuntimeError(
+ f"{scope_id}: graph changed while the online index was being built"
+ )
+ payload = {
+ "schema_version": ONLINE_INDEX_SCHEMA_VERSION,
+ "fast_semantic_state_policy": FAST_SEMANTIC_STATE_POLICY,
+ "created_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
+ "scope_id": scope_id,
+ "db_path": str(db_path),
+ "graph_counts_at_index": counts,
+ "graph_fingerprint": graph_fingerprint,
+ "parent_count": len(parents),
+ "candidate_count": len(candidates),
+ "slow_capsule_count": len(slow_capsules),
+ "fast_semantic_record_count": len(fast_semantic_records),
+ "subchunk_chars": args.subchunk_chars,
+ "subchunk_overlap": args.subchunk_overlap,
+ "embedding_model": str(Path(args.embedding_model).resolve()),
+ "embedding_max_length": args.embedding_max_length,
+ "strict_no_truncation": True,
+ "fast_candidates": candidates,
+ "fast_vectors": fast_vectors,
+ "slow_capsules": slow_capsules,
+ "slow_vectors": slow_vectors,
+ "fast_semantic_records": fast_semantic_records,
+ }
+ atomic_torch_save(payload, index_path)
+ report_row = {
+ "question_id": clean_text(row.get("question_id")),
+ "scope_id": scope_id,
+ "db_path": str(db_path),
+ "index_path": str(index_path),
+ "parent_count": len(parents),
+ "candidate_count": len(candidates),
+ "slow_capsule_count": len(slow_capsules),
+ "fast_semantic_record_count": len(fast_semantic_records),
+ "fast_semantic_state_policy": FAST_SEMANTIC_STATE_POLICY,
+ "graph_counts": counts,
+ "graph_fingerprint": graph_fingerprint,
+ "reused_existing_index": False,
+ }
+ report_rows.append(report_row)
+ print(json.dumps({"status": "indexed", "row": row_index, "total": len(rows), **report_row}), flush=True)
+ report = {
+ "status": "complete",
+ "schema_version": "tmcra.v3.online-index-report.2",
+ "row_count": len(report_rows),
+ "parent_count": sum(row["parent_count"] for row in report_rows),
+ "candidate_count": sum(row["candidate_count"] for row in report_rows),
+ "slow_capsule_count": sum(row["slow_capsule_count"] for row in report_rows),
+ "reused_index_count": reused_index_count,
+ "elapsed_sec": round(time.time() - started, 3),
+ "rows": report_rows,
+ }
+ out_report = Path(args.out_report)
+ out_report.parent.mkdir(parents=True, exist_ok=True)
+ _atomic_write_text(out_report, json.dumps(report, indent=2, sort_keys=True) + "\n")
+
+
+def ordered_graph_parents(
+ event_ids: Iterable[Any],
+ *,
+ valid_locations: set[tuple[int, int]],
+ strict_prefix: bool = False,
+) -> tuple[list[tuple[int, int]], list[str]]:
+ output: list[tuple[int, int]] = []
+ seen: set[tuple[int, int]] = set()
+ unmapped: list[str] = []
+ for raw in event_ids:
+ event_id = clean_text(raw)
+ match = EVENT_PARENT_RE.search(event_id)
+ if match is None:
+ if strict_prefix and event_id.startswith(("event::longmemeval:", "event::tmcra:")):
+ unmapped.append(event_id)
+ continue
+ location = (int(match.group("session")), int(match.group("parent")))
+ if location not in valid_locations:
+ unmapped.append(event_id)
+ continue
+ if location not in seen:
+ seen.add(location)
+ output.append(location)
+ return output, unmapped
+
+
+def expand_parent_locations(
+ locations: Iterable[tuple[int, int]],
+ parent_candidates: Mapping[tuple[int, int], Sequence[int]],
+) -> list[int]:
+ output: list[int] = []
+ seen: set[int] = set()
+ for location in locations:
+ for candidate_index in parent_candidates[location]:
+ if candidate_index not in seen:
+ seen.add(candidate_index)
+ output.append(candidate_index)
+ return output
+
+
+class OnlineModels:
+ def __init__(self, args: argparse.Namespace):
+ from transformers import AutoModelForSequenceClassification, AutoTokenizer
+
+ self.device = torch.device(args.device)
+ if self.device.type == "cuda" and not torch.cuda.is_available():
+ raise RuntimeError("CUDA is unavailable")
+ self.dense = BgeM3DenseVectorizer(
+ dim=args.text_dim,
+ model_path=args.embedding_model,
+ device=str(self.device),
+ max_length=args.embedding_max_length,
+ strict_max_length=True,
+ )
+ cross_path = Path(args.cross_model).resolve()
+ self.cross_manifest = load_optional_hf_model_manifest(cross_path)
+ self.cross_tokenizer = AutoTokenizer.from_pretrained(
+ str(cross_path),
+ local_files_only=True,
+ trust_remote_code=False,
+ )
+ self.cross_model = AutoModelForSequenceClassification.from_pretrained(
+ str(cross_path),
+ local_files_only=True,
+ trust_remote_code=False,
+ torch_dtype=torch.float16 if self.device.type == "cuda" else torch.float32,
+ ).to(self.device)
+ self.cross_model.eval()
+ hidden_size = int(getattr(self.cross_model.config, "hidden_size", 0) or 0)
+ if hidden_size <= 0:
+ raise RuntimeError("cross encoder hidden size is unavailable")
+ checkpoint_path = Path(args.checkpoint).resolve()
+ checkpoint = torch.load(checkpoint_path, map_location=self.device, weights_only=True)
+ if checkpoint.get("schema_version") != SCHEMA_VERSION:
+ raise RuntimeError("online checkpoint schema mismatch")
+ if tuple(checkpoint.get("channel_names") or ()) != CHANNEL_NAMES:
+ raise RuntimeError("online checkpoint channel mismatch")
+ config = dict(checkpoint.get("model_config") or {})
+ self.fusion = ChannelAwareMemoryReranker(
+ representation_dim=hidden_size,
+ channel_dim=len(CHANNEL_NAMES),
+ hidden_dim=int(config["hidden_dim"]),
+ layers=int(config["layers"]),
+ ).to(self.device)
+ self.fusion.load_state_dict(checkpoint["model_state"])
+ self.fusion.eval()
+ self.checkpoint = checkpoint
+ self.checkpoint_path = checkpoint_path
+ self.checkpoint_sha256 = sha256_file(checkpoint_path)
+ self.cross_max_length = int(args.cross_max_length)
+ self.cross_batch_size = int(args.cross_batch_size)
+
+ def encode_cross(self, query: str, texts: Sequence[str]) -> tuple[torch.Tensor, torch.Tensor]:
+ representations: list[torch.Tensor] = []
+ logits: list[torch.Tensor] = []
+ with torch.inference_mode():
+ for start in range(0, len(texts), self.cross_batch_size):
+ batch_texts = list(texts[start : start + self.cross_batch_size])
+ encoded = self.cross_tokenizer(
+ [query] * len(batch_texts),
+ batch_texts,
+ padding=True,
+ truncation=False,
+ return_tensors="pt",
+ )
+ token_lengths = encoded["attention_mask"].sum(dim=1)
+ longest = int(token_lengths.max())
+ if longest > self.cross_max_length:
+ offending = int(torch.argmax(token_lengths))
+ raise RuntimeError(
+ f"online cross pair has {longest} tokens, exceeding strict max={self.cross_max_length}; "
+ f"candidate_batch_index={offending}"
+ )
+ encoded = {key: value.to(self.device) for key, value in encoded.items()}
+ output = self.cross_model(**encoded, output_hidden_states=True, return_dict=True)
+ if output.hidden_states is None:
+ raise RuntimeError("online cross encoder returned no hidden states")
+ representations.append(output.hidden_states[-1][:, 0].float())
+ semantic = output.logits.float()
+ if semantic.ndim == 2 and semantic.shape[1] == 1:
+ semantic = semantic[:, 0]
+ elif semantic.ndim != 1:
+ raise RuntimeError(f"online cross logits have invalid shape: {tuple(semantic.shape)}")
+ logits.append(semantic)
+ return torch.cat(representations, dim=0), torch.cat(logits, dim=0)
+
+
+def load_online_index(path: Path, expected_db: Path, expected_scope: str) -> tuple[list[dict[str, Any]], torch.Tensor, list[dict[str, Any]], torch.Tensor, list[dict[str, Any]], dict[str, Any]]:
+ payload = torch.load(path, map_location="cpu", weights_only=True)
+ if payload.get("schema_version") != ONLINE_INDEX_SCHEMA_VERSION:
+ raise RuntimeError(f"online index schema mismatch: {path}")
+ if payload.get("fast_semantic_state_policy") != FAST_SEMANTIC_STATE_POLICY:
+ raise RuntimeError(f"online index fast semantic state policy mismatch: {path}")
+ if clean_text(payload.get("scope_id")) != expected_scope:
+ raise RuntimeError(f"online index scope mismatch: {path}")
+ if Path(payload.get("db_path", "")).resolve() != expected_db.resolve():
+ raise RuntimeError(f"online index database mismatch: {path}")
+ candidates = list(payload.get("fast_candidates") or [])
+ vectors = payload.get("fast_vectors")
+ slow_capsules = list(payload.get("slow_capsules") or [])
+ slow_vectors = payload.get("slow_vectors")
+ semantic_records = list(payload.get("fast_semantic_records") or [])
+ if not candidates or vectors is None or tuple(vectors.shape) != (len(candidates), 1024):
+ raise RuntimeError(f"online index payload is incomplete: {path}")
+ if slow_vectors is None or tuple(slow_vectors.shape) != (len(slow_capsules), 1024):
+ raise RuntimeError(f"online slow index payload is incomplete: {path}")
+ if any("labels" in candidate for candidate in candidates):
+ raise RuntimeError("runtime index must not contain benchmark labels")
+ return candidates, vectors.float().contiguous(), slow_capsules, slow_vectors.float().contiguous(), semantic_records, payload
+
+
+def choose_diverse(
+ candidates: Sequence[Mapping[str, Any]],
+ scores: torch.Tensor,
+ *,
+ top_k: int,
+ max_per_parent: int,
+ max_per_session: int,
+) -> list[int]:
+ order = torch.argsort(scores, descending=True).tolist()
+ selected: list[int] = []
+ parent_counts: Counter[tuple[int, int]] = Counter()
+ session_counts: Counter[str] = Counter()
+ for index in order:
+ candidate = candidates[index]
+ parent = (int(candidate["session_index"]), int(candidate["parent_chunk_index"]))
+ session_id = clean_text(candidate.get("session_id"))
+ if max_per_parent > 0 and parent_counts[parent] >= max_per_parent:
+ continue
+ if max_per_session > 0 and session_counts[session_id] >= max_per_session:
+ continue
+ selected.append(index)
+ parent_counts[parent] += 1
+ session_counts[session_id] += 1
+ if len(selected) >= top_k:
+ break
+ if len(selected) != top_k:
+ raise RuntimeError(f"online diversity policy produced only {len(selected)} of {top_k} required windows")
+ return selected
+
+
+def _deprecated_legacy_retrieve_one_v1(
+ row: Mapping[str, Any],
+ *,
+ args: argparse.Namespace,
+ harness: Any,
+ models: OnlineModels,
+) -> tuple[dict[str, Any], dict[str, Any]]:
+ """Retained only for forensic comparison; the v2 command never calls this schema .1 path."""
+ started = time.time()
+ qid = clean_text(row.get("question_id"))
+ question = clean_text(row.get("question"))
+ question_date = clean_text(row.get("question_date"))
+ runtime_question = f"{question}\nQuestion date: {question_date}" if question_date else question
+ db_path = Path(row["db_path"]).resolve()
+ scope_id = clean_text(row.get("scope_id"))
+ index_path = Path(row["index_path"]).resolve()
+ if not qid or not question or not scope_id:
+ raise RuntimeError("online retrieval manifest row lacks qid, question, or scope")
+ candidates, dense_vectors, index_payload = load_online_index(index_path, db_path, scope_id)
+ counts_before = scope_counts(db_path, scope_id)
+ if counts_before["records"] != int(index_payload["graph_counts_at_index"]["records"]):
+ raise RuntimeError(f"{qid}: graph records changed after online index creation")
+
+ adapter = harness.build_adapter(scope_id, db_path)
+ graph_started = time.time()
+ retrieval = adapter.retrieve(runtime_question, top_k=args.graph_top_k)
+ graph_elapsed = time.time() - graph_started
+ metadata = dict(getattr(retrieval, "metadata", {}) or {})
+ if clean_text(metadata.get("retrieval_mode")) != "hybrid_node_scored":
+ raise RuntimeError(f"{qid}: graph retrieval did not use hybrid_node_scored")
+ if not bool(metadata.get("hybrid_enabled")):
+ raise RuntimeError(f"{qid}: graph hybrid model path is not active")
+ selected_event_ids = list(metadata.get("selected_event_ids") or [])
+ recall_event_ids = list(metadata.get("recall_event_ids") or [])
+ final_event_ids = list(metadata.get("final_hit_event_ids") or [])
+ if not selected_event_ids:
+ raise RuntimeError(f"{qid}: graph model selected no events")
+
+ parent_candidates: dict[tuple[int, int], list[int]] = {}
+ for index, candidate in enumerate(candidates):
+ location = (int(candidate["session_index"]), int(candidate["parent_chunk_index"]))
+ parent_candidates.setdefault(location, []).append(index)
+ valid_locations = set(parent_candidates)
+ selected_parents, selected_unmapped = ordered_graph_parents(
+ selected_event_ids, valid_locations=valid_locations, strict_prefix=True
+ )
+ recall_parents, recall_unmapped = ordered_graph_parents(recall_event_ids, valid_locations=valid_locations)
+ final_parents, final_unmapped = ordered_graph_parents(final_event_ids, valid_locations=valid_locations)
+ critical_unmapped = [*selected_unmapped, *final_unmapped]
+ if critical_unmapped:
+ raise RuntimeError(f"{qid}: selected/final graph events cannot map to persisted chunks: {critical_unmapped[:8]}")
+ graph_parents: list[tuple[int, int]] = []
+ graph_parent_seen: set[tuple[int, int]] = set()
+ for location in [*selected_parents, *recall_parents]:
+ if location not in graph_parent_seen:
+ graph_parent_seen.add(location)
+ graph_parents.append(location)
+ graph_parents = graph_parents[: args.graph_k]
+ graph_parent_rank = {location: rank for rank, location in enumerate(graph_parents)}
+ graph_runtime_order = expand_parent_locations(graph_parents, parent_candidates)
+ graph_rank = {
+ index: graph_parent_rank[(int(candidates[index]["session_index"]), int(candidates[index]["parent_chunk_index"]))]
+ for index in graph_runtime_order
+ }
+ selected_indexes = set(expand_parent_locations(selected_parents, parent_candidates))
+ final_indexes = set(expand_parent_locations(final_parents, parent_candidates))
+
+ dense_started = time.time()
+ query_vector = models.dense.encode_one(runtime_question)
+ dense_scores = dense_vectors @ query_vector
+ dense_order = sorted(range(len(candidates)), key=lambda index: (-float(dense_scores[index]), index))
+ dense_rank = {index: rank for rank, index in enumerate(dense_order)}
+ dense_elapsed = time.time() - dense_started
+ runtime_indexes: list[int] = []
+ runtime_seen: set[int] = set()
+ for index in [*dense_order[: args.dense_k], *graph_runtime_order]:
+ if index not in runtime_seen:
+ runtime_seen.add(index)
+ runtime_indexes.append(index)
+ if not runtime_indexes:
+ raise RuntimeError(f"{qid}: graph+dense union is empty")
+
+ session_count = max(int(candidate["session_index"]) for candidate in candidates) + 1
+ runtime_candidates: list[dict[str, Any]] = []
+ for index in runtime_indexes:
+ source = dict(candidates[index])
+ channels = {
+ "dense_score": float(dense_scores[index]),
+ "dense_rank_rr": rrank(dense_rank[index]),
+ "graph_rank_rr": rrank(graph_rank.get(index)),
+ "graph_selected": float(index in selected_indexes),
+ "graph_final": float(index in final_indexes),
+ "recency_norm": float(source["session_index"]) / float(max(1, session_count - 1)),
+ }
+ if tuple(channels) != CHANNEL_NAMES:
+ raise AssertionError("online channel order changed")
+ source["channels"] = channels
+ runtime_candidates.append(source)
+
+ cross_started = time.time()
+ representations, semantic_logits = models.encode_cross(
+ runtime_question, [candidate["text"] for candidate in runtime_candidates]
+ )
+ channel_tensor = torch.tensor(
+ [[candidate["channels"][name] for name in CHANNEL_NAMES] for candidate in runtime_candidates],
+ dtype=torch.float32,
+ device=models.device,
+ )
+ mask = torch.ones((1, len(runtime_candidates)), dtype=torch.bool, device=models.device)
+ with torch.inference_mode():
+ fusion_scores = models.fusion(
+ representations.unsqueeze(0),
+ semantic_logits.unsqueeze(0),
+ channel_tensor.unsqueeze(0),
+ mask,
+ ablation="full",
+ )[0].detach().cpu()
+ semantic_cpu = semantic_logits.detach().cpu()
+ cross_elapsed = time.time() - cross_started
+ selected = choose_diverse(
+ runtime_candidates,
+ fusion_scores,
+ top_k=args.top_k,
+ max_per_parent=args.max_per_parent,
+ max_per_session=args.max_per_session,
+ )
+ evidence_windows = []
+ for rank, index in enumerate(selected, start=1):
+ candidate = runtime_candidates[index]
+ evidence_windows.append(
+ {
+ "memory_id": candidate["candidate_id"],
+ "session_id": candidate["session_id"],
+ "session_index": candidate["session_index"],
+ "parent_chunk_index": candidate["parent_chunk_index"],
+ "subchunk_index": candidate["subchunk_index"],
+ "rank": rank,
+ "score": round(float(fusion_scores[index]), 6),
+ "semantic_logit": round(float(semantic_cpu[index]), 6),
+ "channels": candidate["channels"],
+ "text": candidate["text"],
+ }
+ )
+ evidence_row = {
+ "schema_version": SCHEMA_VERSION,
+ "runtime_schema_version": "tmcra.v3.online-retrieval.1",
+ "question_id": qid,
+ "question": question,
+ "question_date": question_date,
+ "question_type": clean_text(row.get("question_type")),
+ "selected_session_ids": [window["session_id"] for window in evidence_windows],
+ "evidence_windows": evidence_windows,
+ }
+ debug_row = {
+ "question_id": qid,
+ "scope_id": scope_id,
+ "db_path": str(db_path),
+ "index_path": str(index_path),
+ "runtime_input_has_gold": any(
+ key in row for key in ("answer", "gold_answer", "answer_session_ids", "labels", "supervision")
+ ),
+ "graph": {
+ "retrieval_mode": metadata.get("retrieval_mode"),
+ "hybrid_enabled": metadata.get("hybrid_enabled"),
+ "decision_score_source": metadata.get("decision_score_source"),
+ "selected_event_count": len(selected_event_ids),
+ "recall_event_count": len(recall_event_ids),
+ "final_event_count": len(final_event_ids),
+ "selected_parent_count": len(selected_parents),
+ "recall_parent_count": len(recall_parents),
+ "final_parent_count": len(final_parents),
+ "unmapped_recall_event_ids": recall_unmapped,
+ "selected_event_ids": selected_event_ids,
+ "recall_event_ids": recall_event_ids,
+ "final_hit_event_ids": final_event_ids,
+ },
+ "inventory_count": len(candidates),
+ "dense_k": args.dense_k,
+ "graph_k": args.graph_k,
+ "union_count": len(runtime_candidates),
+ "selected_count": len(evidence_windows),
+ "checkpoint": str(models.checkpoint_path),
+ "checkpoint_sha256": models.checkpoint_sha256,
+ "cross_model_revision": models.cross_manifest.get("revision"),
+ "strict_no_truncation": True,
+ "restart_boundary_verified": True,
+ "graph_counts_before_query": counts_before,
+ "latency_sec": {
+ "graph": round(graph_elapsed, 4),
+ "dense": round(dense_elapsed, 4),
+ "cross_and_fusion": round(cross_elapsed, 4),
+ "total": round(time.time() - started, 4),
+ },
+ "ranked_union": [
+ {
+ "candidate_id": runtime_candidates[index]["candidate_id"],
+ "session_id": runtime_candidates[index]["session_id"],
+ "parent_chunk_index": runtime_candidates[index]["parent_chunk_index"],
+ "subchunk_index": runtime_candidates[index]["subchunk_index"],
+ "fusion_score": round(float(fusion_scores[index]), 6),
+ "semantic_logit": round(float(semantic_cpu[index]), 6),
+ "channels": runtime_candidates[index]["channels"],
+ }
+ for index in torch.argsort(fusion_scores, descending=True).tolist()
+ ],
+ }
+ if debug_row["runtime_input_has_gold"]:
+ raise RuntimeError(f"{qid}: runtime retrieval manifest contains forbidden evaluation labels")
+ return evidence_row, debug_row
+
+
+def planner_from_env() -> DeepSeekFlashRecallPlanner:
+ pool = [part.strip() for part in os.environ.get("TMCRA_RECALL_PLANNER_API_KEY_POOL", "").split(",") if part.strip()]
+ return DeepSeekFlashRecallPlanner(
+ base_url=os.environ.get("TMCRA_RECALL_PLANNER_BASE_URL", ""),
+ model=os.environ.get("TMCRA_RECALL_PLANNER_MODEL", DEEPSEEK_FLASH_MODEL), api_keys=pool,
+ )
+
+
+def _fast_candidates_with_slots(candidates: Sequence[Mapping[str, Any]], semantic_records: Sequence[Mapping[str, Any]]) -> list[dict[str, Any]]:
+ by_parent: dict[tuple[int, int], list[Mapping[str, Any]]] = {}
+ for record in semantic_records:
+ parent = dict(record["source_parent"])
+ by_parent.setdefault((int(parent["session_index"]), int(parent["parent_chunk_index"])), []).append(record)
+ output: list[dict[str, Any]] = []
+ for candidate in candidates:
+ location = (int(candidate["session_index"]), int(candidate["parent_chunk_index"]))
+ records = [
+ record
+ for record in by_parent.get(location, [])
+ if int(candidate["source_char_end"])
+ > int(record["source_parent"]["evidence_char_start"])
+ and int(candidate["source_char_start"])
+ < int(record["source_parent"]["evidence_char_end"])
+ ]
+ by_slot: dict[str, list[Mapping[str, Any]]] = {}
+ for record in records:
+ by_slot.setdefault(_canonical_slot(record.get("canonical_slot"), label="fast semantic record"), []).append(record)
+ if not by_slot:
+ by_slot[f"fast.parent.s{location[0]}.p{location[1]}"] = []
+ for slot, slot_records in by_slot.items():
+ item = dict(candidate)
+ item["canonical_slot"] = slot
+ item["semantic_record_ids"] = [str(record["memory_id"]) for record in slot_records]
+ item["provenance"] = {
+ "memory_layer": "fast",
+ "content_variant": "source_subchunk",
+ "source_parent": {
+ "session_index": location[0],
+ "parent_chunk_index": location[1],
+ "source_record_ids": sorted(
+ {
+ clean_text(record["source_parent"].get("source_record_id"))
+ for record in slot_records
+ if clean_text(record["source_parent"].get("source_record_id"))
+ }
+ )
+ or ([clean_text(candidate.get("source_record_id"))] if clean_text(candidate.get("source_record_id")) else []),
+ },
+ }
+ output.append(item)
+ return output
+
+
+def _unit_windows(unit: Mapping[str, Any], fast_candidates: Sequence[Mapping[str, Any]], *, qid: str) -> list[dict[str, Any]]:
+ """Fully descend one composed unit, merging duplicate physical source windows."""
+ by_parent: dict[tuple[int, int], list[Mapping[str, Any]]] = {}
+ for candidate in fast_candidates:
+ by_parent.setdefault((int(candidate["session_index"]), int(candidate["parent_chunk_index"])), []).append(candidate)
+ output: dict[tuple[int, int, int], dict[str, Any]] = {}
+
+ def add(candidate: Mapping[str, Any], *, role: str, capsule: Mapping[str, Any] | None = None) -> None:
+ key = (int(candidate["session_index"]), int(candidate["parent_chunk_index"]), int(candidate["subchunk_index"]))
+ item = output.setdefault(key, {**dict(candidate), "roles": [], "capsules": []})
+ if role not in item["roles"]:
+ item["roles"].append(role)
+ if capsule is not None and capsule not in item["capsules"]:
+ item["capsules"].append(dict(capsule))
+
+ def descend(capsule: Mapping[str, Any], role: str) -> None:
+ for parent in capsule["source_parents"]:
+ location = (int(parent["session_index"]), int(parent["parent_chunk_index"]))
+ char_start = int(parent["evidence_char_start"])
+ char_end = int(parent["evidence_char_end"])
+ matches = [
+ candidate
+ for candidate in by_parent.get(location, [])
+ if int(candidate["source_char_end"]) > char_start
+ and int(candidate["source_char_start"]) < char_end
+ ]
+ if not matches:
+ raise RuntimeError(f"{qid}: slow source_parent is unmapped during descent: {location}")
+ for candidate in matches:
+ add(candidate, role=role, capsule=capsule)
+
+ kind = str(unit["unit_type"])
+ if kind.startswith("fast_primary"):
+ add(unit["fast_candidate"], role="primary")
+ elif kind == "slow_primary_with_fast_override":
+ descend(unit["slow_capsule"], "primary")
+ for candidate in unit.get("fast_overrides", []):
+ add(candidate, role="override")
+ elif kind.startswith("slow_primary"):
+ descend(unit["slow_capsule"], "primary")
+ elif kind == "conflict_group":
+ for capsule in unit["slow_capsules"]:
+ descend(capsule, "slow_conflict_candidate")
+ for candidate in unit["fast_candidates"]:
+ add(candidate, role="fast_conflict_candidate")
+ else:
+ raise RuntimeError(f"{qid}: unsupported planned unit type {kind}")
+ return list(output.values())
+
+
+def _unit_attachments(unit: Mapping[str, Any]) -> list[dict[str, Any]]:
+ if unit.get("slow_context"):
+ return [{"role": "context_only", "capsule_id": item["capsule_id"], "canonical_slot": item["canonical_slot"], "summary": item["text"], "source_parents": item["source_parents"], "provenance": item["provenance"]} for item in unit["slow_context"]]
+ if unit.get("fast_overrides"):
+ return [{"role": "override", "memory_id": item["candidate_id"], "canonical_slot": item["canonical_slot"], "text": item["text"], "provenance": item["provenance"]} for item in unit["fast_overrides"]]
+ return []
+
+
+def pack_recall_units(units: Sequence[Mapping[str, Any]], fast_candidates: Sequence[Mapping[str, Any]], *, top_k: int, qid: str) -> list[tuple[Mapping[str, Any], list[dict[str, Any]]]]:
+ if top_k <= 0:
+ raise RuntimeError("top_k must be positive")
+ packed: list[tuple[Mapping[str, Any], list[dict[str, Any]]]] = []
+ used: set[tuple[int, int, int]] = set()
+ for unit in units:
+ windows = _unit_windows(unit, fast_candidates, qid=qid)
+ unique = [item for item in windows if (int(item["session_index"]), int(item["parent_chunk_index"]), int(item["subchunk_index"])) not in used]
+ if not unique:
+ if windows:
+ packed.append((unit, windows))
+ continue
+ if len(unique) > top_k and not packed:
+ raise RuntimeError(f"{qid}: first atomic recall unit requires {len(unique)} windows, exceeding strict packing budget {top_k}")
+ if len(used) + len(unique) > top_k:
+ break
+ packed.append((unit, windows))
+ used.update((int(item["session_index"]), int(item["parent_chunk_index"]), int(item["subchunk_index"])) for item in unique)
+ if not packed:
+ raise RuntimeError(f"{qid}: recall plan produced no packable evidence units")
+ return packed
+
+
+def retrieve_one(
+ row: Mapping[str, Any], *, args: argparse.Namespace, harness: Any, models: OnlineModels,
+ planner: DeepSeekFlashRecallPlanner,
+ graph_adapter_cache: dict[tuple[str, str], Any] | None = None,
+) -> tuple[dict[str, Any], dict[str, Any]]:
+ started = time.time()
+ qid, question, question_date = clean_text(row.get("question_id")), clean_text(row.get("question")), clean_text(row.get("question_date"))
+ if not qid or not question:
+ raise RuntimeError("online retrieval manifest row lacks qid or question")
+ if int(args.slow_dense_k) <= 0:
+ raise RuntimeError("slow_dense_k must be positive")
+ db_path, scope_id, index_path = Path(row["db_path"]).resolve(), clean_text(row.get("scope_id")), Path(row["index_path"]).resolve()
+ fast, fast_vectors, slow, slow_vectors, semantic_records, payload = load_online_index(index_path, db_path, scope_id)
+ counts_before = scope_counts(db_path, scope_id)
+ if counts_before["records"] != int(payload["graph_counts_at_index"]["records"]):
+ raise RuntimeError(f"{qid}: graph records changed after online index creation")
+ graph_fingerprint = scope_fingerprint(db_path, scope_id)
+ if graph_fingerprint != clean_text(payload.get("graph_fingerprint")):
+ raise RuntimeError(f"{qid}: graph fingerprint changed after online index creation")
+ recent_dialogue = load_recent_dialogue_context(
+ db_path, scope_id, current_query=question, limit=8
+ )
+ raw_plan, planner_metadata = planner.plan(
+ query=question, question_date=question_date or "unknown",
+ recent_dialogue=recent_dialogue,
+ available_layers={"fast": {"available": bool(fast), "candidate_count": len(fast)}, "slow": {"available": bool(slow), "capsule_count": len(slow)}},
+ )
+ plan = dict(raw_plan)
+ planner_decision_reason = clean_text(plan.get("decision_reason"))
+ if not planner_decision_reason:
+ raise RuntimeError(f"{qid}: Flash recall plan lacks a decision reason")
+ resolved_query = clean_text(plan.get("resolved_query"))
+ if not resolved_query:
+ raise RuntimeError(f"{qid}: Flash recall plan lacks a resolved query")
+ runtime_question = (
+ f"{resolved_query}\nQuestion date: {question_date}"
+ if question_date
+ else resolved_query
+ )
+ authoritative_plan = dict(plan)
+ authoritative_plan.pop("decision_reason", None)
+ if not slow and plan["mode"] != "FAST_ONLY":
+ raise RuntimeError(f"{qid}: Flash must select FAST_ONLY when no slow capsules are available")
+
+ graph_metadata: dict[str, Any] = {"skipped": plan["mode"] == "SLOW_ONLY"}
+ graph_elapsed = 0.0
+ graph_runtime_order: list[int] = []
+ selected_indexes: set[int] = set()
+ final_indexes: set[int] = set()
+ graph_rank: dict[int, int] = {}
+ parent_candidates: dict[tuple[int, int], list[int]] = {}
+ graph_adapter_reused = False
+ for index, candidate in enumerate(fast):
+ parent_candidates.setdefault((int(candidate["session_index"]), int(candidate["parent_chunk_index"])), []).append(index)
+ if plan["mode"] != "SLOW_ONLY":
+ adapter_key = (scope_id, str(db_path))
+ adapter = graph_adapter_cache.get(adapter_key) if graph_adapter_cache is not None else None
+ if adapter is None:
+ adapter = harness.build_adapter(scope_id, db_path)
+ if graph_adapter_cache is not None:
+ graph_adapter_cache[adapter_key] = adapter
+ else:
+ graph_adapter_reused = True
+ graph_started = time.time()
+ retrieval = adapter.retrieve(runtime_question, top_k=args.graph_top_k)
+ graph_elapsed = time.time() - graph_started
+ metadata = dict(getattr(retrieval, "metadata", {}) or {})
+ if clean_text(metadata.get("retrieval_mode")) != "hybrid_node_scored" or not bool(metadata.get("hybrid_enabled")):
+ raise RuntimeError(f"{qid}: graph hybrid_node_scored path is not active")
+ selected_events, recall_events, final_events = list(metadata.get("selected_event_ids") or []), list(metadata.get("recall_event_ids") or []), list(metadata.get("final_hit_event_ids") or [])
+ slow_graph_events = [str(event_id) for event_id in [*selected_events, *recall_events, *final_events] if str(event_id).startswith("slow.")]
+ if slow_graph_events:
+ raise RuntimeError(f"{qid}: fast graph execution crossed the slow-layer boundary: " + ",".join(dict.fromkeys(slow_graph_events)))
+ if not selected_events:
+ raise RuntimeError(f"{qid}: graph model selected no events")
+ valid_locations = set(parent_candidates)
+ selected_parents, selected_unmapped = ordered_graph_parents(selected_events, valid_locations=valid_locations, strict_prefix=True)
+ recall_parents, recall_unmapped = ordered_graph_parents(recall_events, valid_locations=valid_locations)
+ final_parents, final_unmapped = ordered_graph_parents(final_events, valid_locations=valid_locations)
+ if [*selected_unmapped, *final_unmapped]:
+ raise RuntimeError(f"{qid}: graph events cannot map to persisted chunks")
+ graph_parents = list(dict.fromkeys([*selected_parents, *recall_parents]))[:args.graph_k]
+ graph_runtime_order = expand_parent_locations(graph_parents, parent_candidates)
+ graph_rank = {index: rank for rank, location in enumerate(graph_parents) for index in parent_candidates[location]}
+ selected_indexes, final_indexes = set(expand_parent_locations(selected_parents, parent_candidates)), set(expand_parent_locations(final_parents, parent_candidates))
+ graph_metadata = {"skipped": False, "adapter_reused": graph_adapter_reused, "selected_event_ids": selected_events, "recall_event_ids": recall_events, "final_hit_event_ids": final_events, "unmapped_recall_event_ids": recall_unmapped, "retrieval_mode": metadata.get("retrieval_mode"), "runtime_graph_cache_hit": bool(metadata.get("runtime_graph_cache_hit", False)), "hybrid_candidate_union_rescored": bool(metadata.get("hybrid_candidate_union_rescored", False)), "node_runtime_profile": dict(metadata.get("node_runtime_profile", {}) or {}), "hybrid_runtime_profile": dict(metadata.get("hybrid_runtime_profile", {}) or {}), "adapter_runtime_profile": dict(metadata.get("adapter_runtime_profile", {}) or {})}
+
+ fast_ranked: list[dict[str, Any]] = []
+ fast_scores: dict[str, float] = {}
+ dense_elapsed = 0.0
+ cross_elapsed = 0.0
+ if plan["mode"] != "SLOW_ONLY":
+ dense_started = time.time()
+ dense_scores = fast_vectors @ models.dense.encode_one(runtime_question)
+ dense_order = sorted(range(len(fast)), key=lambda i: (-float(dense_scores[i]), i))
+ dense_rank = {index: rank for rank, index in enumerate(dense_order)}
+ dense_elapsed = time.time() - dense_started
+ indexes = list(dict.fromkeys([*dense_order[:args.dense_k], *graph_runtime_order]))
+ if not indexes:
+ raise RuntimeError(f"{qid}: graph+dense union is empty")
+ session_count = max(int(candidate["session_index"]) for candidate in fast) + 1
+ runtime = []
+ for index in indexes:
+ item = dict(fast[index])
+ item["channels"] = {"dense_score": float(dense_scores[index]), "dense_rank_rr": rrank(dense_rank[index]), "graph_rank_rr": rrank(graph_rank.get(index)), "graph_selected": float(index in selected_indexes), "graph_final": float(index in final_indexes), "recency_norm": float(item["session_index"]) / max(1, session_count - 1)}
+ runtime.append(item)
+ cross_started = time.time()
+ reps, logits = models.encode_cross(runtime_question, [item["text"] for item in runtime])
+ channel_tensor = torch.tensor([[item["channels"][name] for name in CHANNEL_NAMES] for item in runtime], dtype=torch.float32, device=models.device)
+ with torch.inference_mode():
+ scores = models.fusion(reps.unsqueeze(0), logits.unsqueeze(0), channel_tensor.unsqueeze(0), torch.ones((1, len(runtime)), dtype=torch.bool, device=models.device), ablation="full")[0].detach().cpu()
+ cross_elapsed += time.time() - cross_started
+ for item, score, semantic in zip(runtime, scores.tolist(), logits.detach().cpu().tolist()):
+ item["score"], item["semantic_logit"] = float(score), float(semantic)
+ fast_ranked = _fast_candidates_with_slots(sorted(runtime, key=lambda item: -float(item["score"])), semantic_records)
+ fast_scores = {str(item["candidate_id"]): float(item["score"]) for item in fast_ranked}
+
+ slow_ranked: list[dict[str, Any]] = []
+ slow_dense_elapsed = 0.0
+ if slow and plan["mode"] != "FAST_ONLY":
+ slow_dense_started = time.time()
+ slow_dense_scores = slow_vectors @ models.dense.encode_one(runtime_question)
+ slow_dense_order = sorted(range(len(slow)), key=lambda i: (-float(slow_dense_scores[i]), i))
+ slow_dense_elapsed = time.time() - slow_dense_started
+ slow_indexes = slow_dense_order[: min(len(slow_dense_order), int(args.slow_dense_k))]
+ if not slow_indexes:
+ raise RuntimeError(f"{qid}: slow inventory is nonempty but dense shortlist is empty")
+ slow_ranked = [dict(slow[index]) for index in slow_indexes]
+ for index, item in zip(slow_indexes, slow_ranked):
+ item["slow_dense_score"] = float(slow_dense_scores[index])
+ item["slow_dense_rank"] = int(slow_dense_order.index(index))
+ cross_started = time.time()
+ _, logits = models.encode_cross(runtime_question, [item["text"] for item in slow_ranked])
+ cross_elapsed += time.time() - cross_started
+ for item, score in zip(slow_ranked, logits.detach().cpu().tolist()):
+ item["semantic_logit"] = float(score)
+ slow_ranked.sort(key=lambda item: -float(item["semantic_logit"]))
+ try:
+ units = apply_recall_plan(plan, fast_ranked, slow_ranked)
+ except RecallPlannerError as exc:
+ raise RuntimeError(f"{qid}: invalid planned evidence composition: {exc}") from exc
+ if plan["mode"] == "CONFLICT_COMPARE":
+ units.sort(
+ key=lambda item: (
+ -max(
+ (
+ fast_scores.get(str(candidate.get("candidate_id", "")), float("-inf"))
+ for candidate in item.get("fast_candidates", [])
+ ),
+ default=float("-inf"),
+ ),
+ str(item["canonical_slot"]),
+ )
+ )
+ elif plan["primary_layer"] == "fast":
+ units.sort(key=lambda item: -fast_scores.get(str(item.get("fast_candidate", {}).get("candidate_id", "")), float("-inf")))
+ else:
+ units.sort(key=lambda item: -float(item.get("slow_capsule", {}).get("semantic_logit", float("-inf"))))
+
+ packed_units = pack_recall_units(units, fast, top_k=args.top_k, qid=qid)
+ evidence_by_location: dict[tuple[int, int, int], dict[str, Any]] = {}
+ for unit_rank, (unit, entries) in enumerate(packed_units, start=1):
+ attachments = _unit_attachments(unit)
+ for entry in entries:
+ capsules = list(entry.get("capsules") or [])
+ memory_contexts = [
+ {
+ "role": next(
+ (
+ role
+ for role in entry["roles"]
+ if role in {"primary", "slow_conflict_candidate"}
+ ),
+ "slow_context",
+ ),
+ "capsule_id": item["capsule_id"],
+ "canonical_slot": item["canonical_slot"],
+ "claim_text": item["text"],
+ "provenance": item["provenance"],
+ }
+ for item in capsules
+ ]
+ location = (
+ int(entry["session_index"]),
+ int(entry["parent_chunk_index"]),
+ int(entry["subchunk_index"]),
+ )
+ candidate = {
+ "memory_id": entry.get("candidate_id"),
+ "session_id": entry["session_id"],
+ "session_index": entry["session_index"],
+ "parent_chunk_index": entry["parent_chunk_index"],
+ "subchunk_index": entry["subchunk_index"],
+ "score": entry.get("score", entry.get("semantic_logit")),
+ "semantic_logit": entry.get("semantic_logit"),
+ "channels": entry.get("channels"),
+ "text": entry["text"],
+ "unit_type": unit["unit_type"],
+ "unit_types": [unit["unit_type"]],
+ "canonical_slot": unit["canonical_slot"],
+ "canonical_slots": [unit["canonical_slot"]],
+ "capsule_id": [item["capsule_id"] for item in capsules],
+ "role": list(entry["roles"]),
+ "provenance": [item["provenance"] for item in capsules]
+ or [entry.get("provenance")],
+ "memory_contexts": memory_contexts,
+ "attachments": attachments,
+ }
+ existing = evidence_by_location.get(location)
+ if existing is None:
+ evidence_by_location[location] = candidate
+ continue
+ for field in (
+ "unit_types",
+ "canonical_slots",
+ "capsule_id",
+ "role",
+ "provenance",
+ "memory_contexts",
+ "attachments",
+ ):
+ for item in candidate[field]:
+ if item not in existing[field]:
+ existing[field].append(item)
+ evidence_windows = list(evidence_by_location.values())
+ for rank, item in enumerate(evidence_windows, start=1):
+ item["rank"] = rank
+ selected_session_ids = list(
+ dict.fromkeys(str(item["session_id"]) for item in evidence_windows)
+ )
+ evidence = {"schema_version": SCHEMA_VERSION, "runtime_schema_version": "tmcra.v3.online-retrieval.3", "question_id": qid, "question": question, "question_date": question_date, "question_type": clean_text(row.get("question_type")), "selected_session_ids": selected_session_ids, "recall_plan": authoritative_plan, "evidence_windows": evidence_windows}
+ runtime_input_has_gold = any(key in row for key in ("answer", "gold_answer", "answer_session_ids", "labels", "supervision"))
+ debug = {"question_id": qid, "scope_id": scope_id, "db_path": str(db_path), "index_path": str(index_path), "recall_plan": authoritative_plan, "planner_decision_reason": planner_decision_reason, "planner_resolved_query": resolved_query, "recent_dialogue_count": len(recent_dialogue), "planner": planner_metadata, "cross_layer_weighted_fusion": False, "graph": graph_metadata, "runtime_input_has_gold": runtime_input_has_gold, "inventory_count": len(fast) + len(slow), "union_count": len(fast_ranked) + len(slow_ranked), "fast_inventory_count": len(fast), "fast_shortlist_count": len(fast_ranked), "slow_capsule_count": len(slow), "slow_shortlist_count": len(slow_ranked), "slow_dense_k": int(args.slow_dense_k), "fast_semantic_record_count": len(semantic_records), "planned_unit_count": len(units), "packed_unit_count": len(packed_units), "budget_excluded_unit_count": max(0, len(units) - len(packed_units)), "selected_count": len(evidence_windows), "atomic_unit_packing": True, "strict_no_truncation": True, "restart_boundary_verified": True, "graph_counts_before_query": counts_before, "graph_fingerprint": graph_fingerprint, "latency_sec": {"graph": round(graph_elapsed, 4), "dense": round(dense_elapsed, 4), "slow_dense": round(slow_dense_elapsed, 4), "cross": round(cross_elapsed, 4), "total": round(time.time() - started, 4)}}
+ if any(key in row for key in ("answer", "gold_answer", "answer_session_ids", "labels", "supervision")):
+ raise RuntimeError(f"{qid}: runtime retrieval manifest contains forbidden evaluation labels")
+ return evidence, debug
+
+
+def layered_retrieval_operation_id(out_dir: Path, scope_id: str, question_id: str) -> str:
+ identity = "\n".join(
+ (
+ "tmcra.v3.layered-retrieval",
+ str(out_dir.absolute()),
+ clean_text(scope_id),
+ clean_text(question_id),
+ )
+ )
+ return hashlib.sha256(identity.encode("utf-8")).hexdigest()
+
+
+def _atomic_write_text(path: Path, text: str) -> None:
+ temporary = path.with_name(f".{path.name}.tmp.{os.getpid()}.{time.time_ns()}")
+ temporary.write_text(text, encoding="utf-8")
+ os.replace(temporary, path)
+
+
+def _reconcile_completed_retrieval_output(
+ *,
+ out_dir: Path,
+ rows: Sequence[Mapping[str, Any]],
+ repo: Path,
+) -> dict[str, Any] | None:
+ """Reuse a committed retrieval directory and repair only missing idempotent audit state."""
+ if not out_dir.exists():
+ return None
+ required = {
+ "evidence": out_dir / "evidence_windows.jsonl",
+ "debug": out_dir / "retrieval_debug.jsonl",
+ "report": out_dir / "report.json",
+ }
+ missing = [name for name, path in required.items() if not path.is_file()]
+ if missing:
+ raise RuntimeError(
+ f"retrieval output directory is incomplete and will not be overwritten: "
+ f"{out_dir} missing={','.join(missing)}"
+ )
+ evidence_rows = read_jsonl(required["evidence"])
+ debug_rows = read_jsonl(required["debug"])
+ report = json.loads(required["report"].read_text(encoding="utf-8"))
+ expected_qids = [clean_text(row.get("question_id")) for row in rows]
+ evidence_qids = [clean_text(row.get("question_id")) for row in evidence_rows]
+ debug_qids = [clean_text(row.get("question_id")) for row in debug_rows]
+ if (
+ report.get("status") != "complete"
+ or evidence_qids != expected_qids
+ or debug_qids != expected_qids
+ ):
+ raise RuntimeError(f"existing retrieval output does not match its query manifest: {out_dir}")
+
+ newly_appended = 0
+ query_ids: list[str] = []
+ for evidence, debug in zip(evidence_rows, debug_rows):
+ plan = dict(evidence.get("recall_plan") or {})
+ legacy_reason = clean_text(plan.pop("decision_reason", ""))
+ if legacy_reason:
+ observed_reason = clean_text(debug.get("planner_decision_reason"))
+ if observed_reason and observed_reason != legacy_reason:
+ raise RuntimeError(
+ f"{evidence['question_id']}: persisted planner reason disagrees with debug output"
+ )
+ debug["planner_decision_reason"] = legacy_reason
+ evidence["recall_plan"] = plan
+ operation_id = layered_retrieval_operation_id(
+ out_dir,
+ str(debug["scope_id"]),
+ str(evidence["question_id"]),
+ )
+ persisted = append_layered_retrieval_audit(
+ repo=repo,
+ db_path=Path(debug["db_path"]),
+ scope_id=str(debug["scope_id"]),
+ operation_id=operation_id,
+ evidence=evidence,
+ debug=debug,
+ )
+ audit_payload = dict(persisted.get("payload") or {})
+ query_id = clean_text(audit_payload.get("query_id"))
+ if not query_id:
+ raise RuntimeError(f"{evidence['question_id']}: persisted retrieval audit lacks query_id")
+ newly_appended += int(bool(persisted.get("appended")))
+ query_ids.append(query_id)
+ debug["layered_retrieval_audit"] = {
+ "operation_id": operation_id,
+ "query_id": query_id,
+ "event_total": int(persisted.get("event_total") or 0),
+ "trimmed_total": int(persisted.get("trimmed_total") or 0),
+ "newly_appended": bool(persisted.get("appended")),
+ }
+
+ report["layered_retrieval_audit_confirmed_count"] = len(query_ids)
+ report["layered_retrieval_audit_newly_appended_count"] = newly_appended
+ report["layered_retrieval_audit_query_ids"] = query_ids
+ report["reused_completed_output"] = True
+ _atomic_write_text(
+ required["evidence"],
+ "".join(json.dumps(row, ensure_ascii=False, sort_keys=True) + "\n" for row in evidence_rows),
+ )
+ _atomic_write_text(
+ required["debug"],
+ "".join(json.dumps(row, ensure_ascii=False, sort_keys=True) + "\n" for row in debug_rows),
+ )
+ _atomic_write_text(
+ required["report"], json.dumps(report, indent=2, sort_keys=True) + "\n"
+ )
+ return report
+
+
+def command_retrieve(args: argparse.Namespace) -> None:
+ query_manifest = Path(args.query_manifest)
+ try:
+ rows = read_jsonl(query_manifest)
+ except RuntimeError as exc:
+ if str(exc) == f"no rows: {query_manifest}":
+ raise RuntimeError(f"query manifest is empty: {query_manifest}") from exc
+ raise
+ out_dir = Path(args.out_dir).absolute()
+ completed_report = _reconcile_completed_retrieval_output(
+ out_dir=out_dir,
+ rows=rows,
+ repo=Path(args.repo),
+ )
+ if completed_report is not None:
+ print(json.dumps(completed_report, indent=2, sort_keys=True))
+ return
+ runtime_env = graph_runtime_env(args)
+ for path in (Path(args.node_model), Path(args.path_model), Path(args.checkpoint)):
+ if not path.exists():
+ raise FileNotFoundError(path)
+ planner = planner_from_env()
+ harness = load_native_harness(Path(args.harness), Path(args.repo))
+ harness.disable_topic_bucket_runtime()
+ models = OnlineModels(args)
+ started = time.time()
+ evidence_rows: list[dict[str, Any]] = []
+ debug_rows: list[dict[str, Any]] = []
+ graph_adapter_cache: dict[tuple[str, str], Any] = {}
+ newly_appended_audits = 0
+ for row_index, row in enumerate(rows, start=1):
+ evidence, debug = retrieve_one(
+ row,
+ args=args,
+ harness=harness,
+ models=models,
+ planner=planner,
+ graph_adapter_cache=graph_adapter_cache,
+ )
+ operation_id = layered_retrieval_operation_id(
+ out_dir,
+ str(debug["scope_id"]),
+ str(evidence["question_id"]),
+ )
+ persisted_audit = append_layered_retrieval_audit(
+ repo=Path(args.repo),
+ db_path=Path(debug["db_path"]),
+ scope_id=str(debug["scope_id"]),
+ operation_id=operation_id,
+ evidence=evidence,
+ debug=debug,
+ )
+ newly_appended_audits += int(bool(persisted_audit.get("appended")))
+ debug["layered_retrieval_audit"] = {
+ "operation_id": operation_id,
+ "query_id": clean_text(dict(persisted_audit.get("payload") or {}).get("query_id")),
+ "event_total": int(persisted_audit.get("event_total") or 0),
+ "trimmed_total": int(persisted_audit.get("trimmed_total") or 0),
+ "newly_appended": bool(persisted_audit.get("appended")),
+ }
+ evidence_rows.append(evidence)
+ debug_rows.append(debug)
+ print(
+ json.dumps(
+ {
+ "status": "retrieved",
+ "row": row_index,
+ "total": len(rows),
+ "question_id": evidence["question_id"],
+ "inventory": debug["inventory_count"],
+ "union": debug["union_count"],
+ "selected": debug["selected_count"],
+ "latency_sec": debug["latency_sec"]["total"],
+ }
+ ),
+ flush=True,
+ )
+ latency = [float(row["latency_sec"]["total"]) for row in debug_rows]
+ report = {
+ "status": "complete",
+ "schema_version": "tmcra.v3.online-retrieval-report.2",
+ "created_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
+ "query_count": len(rows),
+ "runtime_input_with_gold_count": sum(int(row["runtime_input_has_gold"]) for row in debug_rows),
+ "restart_boundary_verified_count": sum(int(row["restart_boundary_verified"]) for row in debug_rows),
+ "layered_retrieval_audit_confirmed_count": sum(
+ int(bool(row.get("layered_retrieval_audit"))) for row in debug_rows
+ ),
+ "layered_retrieval_audit_newly_appended_count": newly_appended_audits,
+ "layered_retrieval_audit_query_ids": [
+ row["layered_retrieval_audit"]["query_id"] for row in debug_rows
+ ],
+ "checkpoint": str(models.checkpoint_path),
+ "checkpoint_sha256": models.checkpoint_sha256,
+ "cross_model_revision": models.cross_manifest.get("revision"),
+ "graph_runtime_env": runtime_env,
+ "strict_no_truncation": True,
+ "cross_layer_weighted_fusion": False,
+ "graph_adapter_cache_size": len(graph_adapter_cache),
+ "planner": {
+ "base_url_configured": bool(os.environ.get("TMCRA_RECALL_PLANNER_BASE_URL")),
+ "model": os.environ.get("TMCRA_RECALL_PLANNER_MODEL", DEEPSEEK_FLASH_MODEL),
+ "api_key_pool_size": len(planner.api_keys),
+ },
+ "top_k": args.top_k,
+ "dense_k": args.dense_k,
+ "graph_k": args.graph_k,
+ "slow_dense_k": args.slow_dense_k,
+ "atomic_unit_packing": True,
+ "avg_planned_unit_count": round(sum(row["planned_unit_count"] for row in debug_rows) / len(debug_rows), 4),
+ "avg_packed_unit_count": round(sum(row["packed_unit_count"] for row in debug_rows) / len(debug_rows), 4),
+ "avg_budget_excluded_unit_count": round(sum(row["budget_excluded_unit_count"] for row in debug_rows) / len(debug_rows), 4),
+ "avg_inventory_count": round(sum(row["inventory_count"] for row in debug_rows) / len(debug_rows), 4),
+ "avg_union_count": round(sum(row["union_count"] for row in debug_rows) / len(debug_rows), 4),
+ "avg_latency_sec": round(sum(latency) / len(latency), 4),
+ "max_latency_sec": round(max(latency), 4),
+ "elapsed_sec": round(time.time() - started, 3),
+ "evidence": str(out_dir / "evidence_windows.jsonl"),
+ "debug": str(out_dir / "retrieval_debug.jsonl"),
+ "reused_completed_output": False,
+ }
+ out_dir.parent.mkdir(parents=True, exist_ok=True)
+ staging = out_dir.with_name(
+ f".{out_dir.name}.staging.{os.getpid()}.{time.time_ns()}"
+ )
+ staging.mkdir(parents=False, exist_ok=False)
+ write_jsonl(staging / "evidence_windows.jsonl", evidence_rows)
+ write_jsonl(staging / "retrieval_debug.jsonl", debug_rows)
+ (staging / "report.json").write_text(
+ json.dumps(report, indent=2, sort_keys=True) + "\n", encoding="utf-8"
+ )
+ os.replace(staging, out_dir)
+ print(json.dumps(report, indent=2, sort_keys=True))
+
+
+def add_common_model_args(parser: argparse.ArgumentParser) -> None:
+ parser.add_argument("--embedding-model", default=os.getenv("TMCRA_BGE_M3_PATH", "BAAI/bge-m3"))
+ parser.add_argument("--text-dim", type=int, default=1024)
+ parser.add_argument("--embedding-max-length", type=int, default=8192)
+ parser.add_argument("--device", default="cuda")
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description="TMCRA V3 production online index and retrieval runtime")
+ sub = parser.add_subparsers(dest="command", required=True)
+ index = sub.add_parser("build-index")
+ index.add_argument("--scope-manifest", required=True)
+ index.add_argument("--out-report", required=True)
+ index.add_argument("--subchunk-chars", type=int, default=1800)
+ index.add_argument("--subchunk-overlap", type=int, default=200)
+ index.add_argument("--batch-size", type=int, default=16)
+ add_common_model_args(index)
+
+ retrieve = sub.add_parser("retrieve")
+ retrieve.add_argument("--query-manifest", required=True)
+ retrieve.add_argument("--out-dir", required=True)
+ retrieve.add_argument("--checkpoint", required=True)
+ retrieve.add_argument("--cross-model", default=os.getenv("TMCRA_BGE_RERANKER_PATH", "BAAI/bge-reranker-v2-m3"))
+ retrieve.add_argument("--cross-max-length", type=int, default=1280)
+ retrieve.add_argument("--cross-batch-size", type=int, default=24)
+ retrieve.add_argument("--repo", required=True)
+ retrieve.add_argument("--harness", required=True)
+ retrieve.add_argument("--node-model", required=True)
+ retrieve.add_argument("--path-model", required=True)
+ retrieve.add_argument("--graph-device", default="cuda")
+ retrieve.add_argument("--candidate-event-k", type=int, default=24)
+ retrieve.add_argument("--support-path-k", type=int, default=3)
+ retrieve.add_argument("--path-tunnel-rescue-k", type=int, default=2)
+ retrieve.add_argument("--graph-top-k", type=int, default=12)
+ retrieve.add_argument("--dense-k", type=int, default=32)
+ retrieve.add_argument("--slow-dense-k", type=int, default=24)
+ retrieve.add_argument("--graph-k", type=int, default=24)
+ retrieve.add_argument("--top-k", type=int, default=8)
+ retrieve.add_argument("--max-per-parent", type=int, default=2)
+ retrieve.add_argument("--max-per-session", type=int, default=4)
+ add_common_model_args(retrieve)
+ args = parser.parse_args()
+ if args.command == "build-index":
+ command_build_index(args)
+ else:
+ command_retrieve(args)
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v3_product_writer.py b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v3_product_writer.py
new file mode 100644
index 0000000000000000000000000000000000000000..f64f9b81483df4a031cdd38c053f5885ba32f8d3
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v3_product_writer.py
@@ -0,0 +1,3252 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import os
+import re
+import sqlite3
+import sys
+import threading
+import time
+import urllib.error
+import urllib.request
+import uuid
+from contextlib import closing
+from datetime import datetime, timedelta, timezone
+from pathlib import Path
+from typing import Any, Callable, Mapping, Sequence, TypeVar
+
+
+WRITE_SCHEMA_VERSION = "tmcra.memory-write.v3.4"
+PROMPT_VERSION = "tmcra-product-writer-2026-07-10.21"
+SOURCE_JOURNAL_CLAIM_LEASE_SECONDS = 900
+T = TypeVar("T")
+
+FORBIDDEN_WRITER_FIELDS = {
+ "question",
+ "question_date",
+ "question_type",
+ "answer",
+ "gold_answer",
+ "answer_session_ids",
+ "labels",
+ "supervision",
+}
+MEMORY_TYPES = {
+ "fact",
+ "event",
+ "state",
+ "preference",
+ "goal",
+ "constraint",
+ "plan",
+ "identity",
+ "relationship",
+ "possession",
+ "routine",
+}
+TEMPORAL_STATUSES = {"past", "current", "planned", "future", "timeless", "uncertain"}
+POLARITIES = {"positive", "negative"}
+OPERATIONS = {"append", "replace"}
+FACET_TYPES = {"entity", "time", "quantity", "state", "location", "role"}
+INTERACTION_TYPES = {"question", "request", "reminder", "task", "clarification", "feedback"}
+INTERACTION_STATUSES = {"open", "informational"}
+RESOLUTION_STATES = {"resolved", "partial", "unresolved"}
+KEY_RE = re.compile(r"^[a-z0-9][a-z0-9_.-]{1,159}$")
+ROLE_RE = re.compile(r"^[a-z][a-z0-9_]{1,63}$")
+SOURCE_TOKEN_RE = re.compile(r"[\u3400-\u9fff]|[^\s\u3400-\u9fff]+")
+SENTENCE_BREAK_RE = re.compile(r"(?<=[.!?])\s+|\n+|(?<=[。！？])")
+
+
+SYSTEM_PROMPT = """You are the semantic write stage of a production personal-memory system.
+Return one strict JSON object and nothing else. current_message is the only new source. previous_message,
+existing_memory_slots, and pending_interactions are read-only context for reference, slot, and resolution.
+When capacity_segment is present, current_message is an exact slice of full_current_message_context. Extract
+every item whose evidence is inside that slice, using the full message only to interpret references and context.
+Do not emit an item whose only evidence is outside the slice. Capacity segmentation never changes the semantic
+contract and never permits omission, ranking, or summarization.
+
+The output has three independent layers:
+1. assertions: facts explicitly asserted by the user, including events, updates, preferences, goals,
+ constraints, plans, identity, relationships, possessions, and routines. Questions are never assertions.
+ A question's presupposition is not a fact. Assistant statements are not user assertions.
+2. interactions: every explicit question, request, reminder, task, clarification, or meaningful feedback in
+ current_message. A question-only message must produce an interaction even when assertions is empty.
+ A mixed message can and usually should produce both assertions and interactions.
+3. resolutions: whether current_message resolves a supplied pending_interaction. Use resolved only when the
+ current message actually answers or completes it, partial when it advances it without completing it, and
+ unresolved when it responds without supplying the requested result. Use exact supplied interaction_id.
+ pending_interactions are candidates, not a checklist. Omit an unrelated candidate instead of emitting a
+ resolution. Every emitted resolution must reference non-empty evidence from current_message.
+ Use unresolved only when current_message explicitly responds without a result, such as a refusal or stated
+ inability; the mere absence of an answer is not resolution evidence.
+
+For assistant messages, assertions must be empty because assistant claims have different authority. Assistant
+messages may create interactions only when the assistant explicitly asks the user a question, requests an
+action, sets a task or reminder, or asks for clarification. An answer, recommendation, apology, correction
+acknowledgement, confirmation, or explanatory statement is not a new interaction; it may resolve an existing
+interaction and otherwise remains only immutable source. For user messages, extract user assertions, create
+interactions, and resolve assistant interactions when applicable. Never infer an unstated fact. Never store
+passwords, authentication secrets, private keys, or account IDs as assertions.
+
+Each assertion and interaction must be atomic. Never copy or rewrite source text in the output. Instead, select
+one exact evidence_span_id from evidence_spans. Prefer the shortest catalog span that fully supports the item;
+e0 always means the full current_message and is represented compactly without repeating its text. Resolutions
+also select one evidence_span_id. source_tokens is a compact string array whose zero-based array position is the
+token id. Every facet/about item selects an inclusive token_start and token_end from source_tokens.
+The token range must be the shortest exact source range that names the entity, time, quantity, state, location,
+or role. Preserve negation, uncertainty, and correction semantics. Assertion authority is always the user; do
+not emit redundant subject or object fields.
+
+For assertions, entity_key identifies the stable real-world subject or domain and attribute_key identifies
+the specific property, goal, state, or event kind. Use lowercase dot-separated tokens. Prefer the concrete
+topic domain (for example starbucks.rewards or laptop.purchase) over the generic entity_key user. Never put
+the changing value in either key. Reuse an existing entity_key plus attribute_key only when both the entity
+and the predicate are the same. Topic similarity is insufficient: a goal to reach a membership level and the
+number of points required for that level are different attributes. The runtime also isolates different memory
+families, so a fact cannot replace a goal even when keys are imperfect. Use replace for mutable attributes and
+append for repeatable events. memory_type goal is only the desired outcome itself. A threshold, requirement,
+or current value is fact or state even when phrased as "I need N units to reach ...". relation and all
+role/intent fields are lowercase snake_case.
+
+The output message_role must exactly copy current_message.role. It is source
+metadata, not a semantic classification task.
+
+Return exactly:
+{"schema_version":"tmcra.memory-write.v3.4","message_role":"user|assistant|system|tool",
+ "assertions":[
+ {"memory_type":"fact|event|state|preference|goal|constraint|plan|identity|relationship|possession|routine",
+ "entity_key":"stable.entity.or.domain","attribute_key":"specific_attribute_or_event","operation":"append|replace",
+ "evidence_span_id":"eN","relation":"snake_case",
+ "temporal_status":"past|current|planned|future|timeless|uncertain",
+ "polarity":"positive|negative","facets":[{"type":"entity|time|quantity|state|location|role",
+ "role":"snake_case","token_start":0,"token_end":0}]}
+ ],
+ "interactions":[
+ {"interaction_type":"question|request|reminder|task|clarification|feedback",
+ "status":"open|informational","evidence_span_id":"eN","intent":"snake_case",
+ "about":[{"type":"entity|time|quantity|state|location|role","role":"snake_case",
+ "token_start":0,"token_end":0}]}
+ ],
+ "resolutions":[
+ {"interaction_id":"exact supplied id","resolution":"resolved|partial|unresolved",
+ "evidence_span_id":"eN"}
+ ]}
+
+Use empty arrays when a layer has no output. Return every distinct assertion, interaction, resolution, and
+semantically relevant facet/about item. Never omit an item to meet an arbitrary count limit.
+Do not rank, summarize, repair, or select only the most interesting items."""
+
+PRO_PROMPT_VERSION = "tmcra-product-writer-pro-2026-07-10.10"
+PRO_SYSTEM_PROMPT = """You are the high-accuracy primary writer for a production personal-memory system.
+Return a complete output under the exact contract below. Recover every explicit assertion or interaction,
+remove inferred content, choose precise memory_type/entity_key/attribute_key/operation values, and verify
+resolutions against pending_interactions. Return one strict JSON object and nothing else.
+
+Before returning, independently inspect every clause in current_message rather than reviewing only the Flash
+items. Every explicit user self-report about an intention, plan, goal, consideration, current or past state,
+preference, possession, relationship, identity, routine, event, or constraint must appear as an assertion.
+This remains true when another clause in the same message asks a question or requests advice. Do not let an
+interaction suppress a distinct assertion, and do not turn the question itself into an assertion.
+
+""" + SYSTEM_PROMPT
+
+
+class ProductWriterError(RuntimeError):
+ pass
+
+
+class ProductWriterResponseError(ProductWriterError):
+ def __init__(
+ self,
+ message: str,
+ *,
+ response_content: str = "",
+ request_metadata: Mapping[str, Any] | None = None,
+ ) -> None:
+ super().__init__(message)
+ self.response_content = response_content
+ self.request_metadata = dict(request_metadata or {})
+
+
+def clean_text(value: Any) -> str:
+ return " ".join(str(value or "").split())
+
+
+def sha256_text(value: str) -> str:
+ return hashlib.sha256(value.encode("utf-8")).hexdigest()
+
+
+def physical_requests_from_metadata(metadata: Mapping[str, Any]) -> list[dict[str, Any]]:
+ requests = metadata.get("requests")
+ if isinstance(requests, list):
+ return [dict(value) for value in requests if isinstance(value, Mapping)]
+ return [dict(metadata)] if metadata.get("model") else []
+
+
+def load_jsonl_for_resume(path: Path) -> tuple[list[dict[str, Any]], bool]:
+ if not path.exists():
+ return [], False
+ data = path.read_bytes()
+ rows: list[dict[str, Any]] = []
+ offset = 0
+ lines = data.splitlines(keepends=True)
+ for index, raw_line in enumerate(lines):
+ next_offset = offset + len(raw_line)
+ if not raw_line.strip():
+ offset = next_offset
+ continue
+ try:
+ decoded = raw_line.decode("utf-8")
+ value = json.loads(decoded)
+ except (UnicodeDecodeError, json.JSONDecodeError) as exc:
+ later_content = any(line.strip() for line in lines[index + 1 :])
+ has_complete_ending = raw_line.endswith((b"\n", b"\r"))
+ if later_content or has_complete_ending:
+ raise ProductWriterError(f"{path.name}: malformed non-tail JSONL row {index + 1}: {exc}") from exc
+ with path.open("r+b") as handle:
+ handle.truncate(offset)
+ return rows, True
+ if not isinstance(value, dict):
+ raise ProductWriterError(f"{path.name}: JSONL row {index + 1} must be an object")
+ rows.append(value)
+ offset = next_offset
+ return rows, False
+
+
+def exact_evidence_spans(value: str) -> list[dict[str, Any]]:
+ spans: list[dict[str, Any]] = [
+ {"span_id": "e0", "text": value, "char_start": 0, "char_end": len(value)}
+ ]
+ cursor = 0
+ for match in [*SENTENCE_BREAK_RE.finditer(value), None]:
+ boundary = match.start() if match is not None else len(value)
+ start = cursor
+ end = boundary
+ while start < end and value[start].isspace():
+ start += 1
+ while end > start and value[end - 1].isspace():
+ end -= 1
+ if start < end:
+ spans.append(
+ {
+ "span_id": f"e{len(spans)}",
+ "text": value[start:end],
+ "char_start": start,
+ "char_end": end,
+ }
+ )
+ cursor = match.end() if match is not None else len(value)
+ return spans
+
+
+def compact_writer_evidence_spans(value: str) -> list[dict[str, Any]]:
+ spans = exact_evidence_spans(value)
+ return [
+ {
+ "span_id": "e0",
+ "source": "full_current_message",
+ "char_start": 0,
+ "char_end": len(value),
+ },
+ *[span for span in spans[1:]],
+ ]
+
+
+def exact_source_tokens(value: str) -> list[dict[str, Any]]:
+ return [
+ {"token_id": index, "text": match.group(0)}
+ for index, match in enumerate(source_token_matches(value))
+ ]
+
+
+def source_token_matches(value: str) -> list[re.Match[str]]:
+ return list(SOURCE_TOKEN_RE.finditer(value))
+
+
+def split_capacity_range(value: str, start: int, end: int) -> tuple[tuple[int, int], tuple[int, int]] | None:
+ while start < end and value[start].isspace():
+ start += 1
+ while end > start and value[end - 1].isspace():
+ end -= 1
+ if end - start < 2:
+ return None
+
+ segment = value[start:end]
+ midpoint = start + (end - start) / 2
+ boundaries: list[tuple[int, int]] = []
+ for match in SENTENCE_BREAK_RE.finditer(segment):
+ boundaries.append((start + match.start(), start + match.end()))
+ if len(boundaries) < 3:
+ return None
+
+ for boundary_index in sorted(
+ range(1, len(boundaries) - 1),
+ key=lambda index: abs(((boundaries[index][0] + boundaries[index][1]) / 2) - midpoint),
+ ):
+ # Keep one complete sentence on both sides of the split. A source that cannot
+ # be divided at sentence boundaries fails visibly instead of cutting a fact.
+ left_end = boundaries[boundary_index + 1][0]
+ right_start = boundaries[boundary_index - 1][1]
+ if start < left_end and right_start < end:
+ return (start, left_end), (right_start, end)
+ return None
+
+
+def merge_segment_outputs(
+ outputs: Sequence[Mapping[str, Any]],
+ *,
+ message_role: str,
+) -> tuple[dict[str, Any], int]:
+ merged: dict[str, Any] = {
+ "schema_version": WRITE_SCHEMA_VERSION,
+ "message_role": message_role,
+ "assertions": [],
+ "interactions": [],
+ "resolutions": [],
+ "validation_warnings": [],
+ "quarantined_item_count": 0,
+ }
+ assertion_by_key: dict[tuple[str, ...], dict[str, Any]] = {}
+ interaction_by_key: dict[tuple[str, ...], dict[str, Any]] = {}
+ resolution_by_id: dict[str, dict[str, Any]] = {}
+ duplicate_count = 0
+
+ def merge_facets(target: list[dict[str, Any]], incoming: Sequence[Mapping[str, Any]]) -> None:
+ nonlocal duplicate_count
+ seen = {
+ (str(item.get("type")), str(item.get("role")), str(item.get("quote")))
+ for item in target
+ }
+ for item in incoming:
+ facet = dict(item)
+ key = (str(facet.get("type")), str(facet.get("role")), str(facet.get("quote")))
+ if key in seen:
+ duplicate_count += 1
+ continue
+ seen.add(key)
+ target.append(facet)
+
+ for segment_index, raw_output in enumerate(outputs):
+ output = dict(raw_output)
+ merged["quarantined_item_count"] += int(output.get("quarantined_item_count", 0) or 0)
+ for raw_warning in output.get("validation_warnings") or []:
+ warning = dict(raw_warning)
+ warning["path"] = f"capacity_segments[{segment_index}].{warning.get('path', 'root')}"
+ merged["validation_warnings"].append(warning)
+
+ for raw_assertion in output.get("assertions") or []:
+ assertion = dict(raw_assertion)
+ append_member_identity = ()
+ if str(assertion.get("operation")) == "append":
+ append_member_identity = tuple(
+ sorted(
+ (
+ str(facet.get("type")),
+ str(facet.get("role")),
+ int(facet.get("token_start", -1)),
+ int(facet.get("token_end", -1)),
+ )
+ for facet in assertion.get("facets") or []
+ )
+ )
+ key = (
+ str(assertion.get("canonical_key")),
+ str(assertion.get("evidence_quote")),
+ str(assertion.get("operation")),
+ str(assertion.get("temporal_status")),
+ str(assertion.get("polarity")),
+ str(assertion.get("memory_type")),
+ str(assertion.get("relation")),
+ str(assertion.get("evidence_char_start")),
+ str(assertion.get("evidence_char_end")),
+ append_member_identity,
+ )
+ existing = assertion_by_key.get(key)
+ if existing is not None:
+ duplicate_count += 1
+ merge_facets(existing["facets"], assertion.get("facets") or [])
+ continue
+ assertion_by_key[key] = assertion
+ merged["assertions"].append(assertion)
+
+ for raw_interaction in output.get("interactions") or []:
+ interaction = dict(raw_interaction)
+ key = (
+ str(interaction.get("interaction_type")),
+ str(interaction.get("intent")),
+ str(interaction.get("evidence_quote")),
+ str(interaction.get("status")),
+ str(interaction.get("evidence_char_start")),
+ str(interaction.get("evidence_char_end")),
+ )
+ existing = interaction_by_key.get(key)
+ if existing is not None:
+ duplicate_count += 1
+ merge_facets(existing["about"], interaction.get("about") or [])
+ continue
+ interaction_by_key[key] = interaction
+ merged["interactions"].append(interaction)
+
+ for raw_resolution in output.get("resolutions") or []:
+ resolution = dict(raw_resolution)
+ interaction_id = str(resolution.get("interaction_id"))
+ existing = resolution_by_id.get(interaction_id)
+ if existing is None:
+ resolution_by_id[interaction_id] = resolution
+ merged["resolutions"].append(resolution)
+ continue
+ duplicate_count += 1
+ if existing.get("resolution") != resolution.get("resolution"):
+ raise ProductWriterError(
+ f"capacity segments produced conflicting resolutions for {interaction_id!r}: "
+ f"{existing.get('resolution')!r} != {resolution.get('resolution')!r}"
+ )
+
+ return merged, duplicate_count
+
+
+def slug(value: Any, *, limit: int = 80) -> str:
+ result = re.sub(r"[^a-z0-9]+", "_", clean_text(value).lower()).strip("_")
+ return (result or "unknown")[:limit]
+
+
+def normalize_canonical_key(value: str) -> str:
+ return ".".join(re.findall(r"[a-z0-9]+", value.lower()))
+
+
+def memory_family(memory_type: str) -> str:
+ return {
+ "fact": "fact",
+ "state": "fact",
+ "event": "event",
+ "preference": "preference",
+ "goal": "goal",
+ "constraint": "constraint",
+ "plan": "plan",
+ "identity": "identity",
+ "relationship": "relationship",
+ "possession": "possession",
+ "routine": "routine",
+ }[memory_type]
+
+
+def graph_entity_key(entity_key: str, attribute_key: str) -> str:
+ normalized_entity = normalize_canonical_key(entity_key)
+ if normalized_entity not in {"user", "self", "person.user"}:
+ return normalized_entity
+ attribute_parts = normalize_canonical_key(attribute_key).split(".")
+ domain = ".".join(attribute_parts[:2]) if len(attribute_parts) >= 2 else attribute_parts[0]
+ return f"user.{domain}"
+
+
+def exact_json_object(
+ content: str,
+ *,
+ warnings: list[dict[str, str]] | None = None,
+) -> dict[str, Any]:
+ if not content or not content.strip():
+ raise ProductWriterError("writer response must be a non-empty JSON object")
+ normalized = content.strip()
+ fenced = re.fullmatch(r"```(?:json)?\s*\n([\s\S]*?)\n```", normalized, flags=re.IGNORECASE)
+ if fenced:
+ normalized = fenced.group(1).strip()
+ if warnings is not None:
+ warnings.append(
+ {
+ "path": "root",
+ "code": "json_fence_removed",
+ "error": "removed one outer Markdown JSON fence",
+ }
+ )
+ try:
+ parsed = json.loads(normalized)
+ except json.JSONDecodeError as exc:
+ raise ProductWriterError(f"writer response is not strict JSON: {exc}") from exc
+ if not isinstance(parsed, dict):
+ raise ProductWriterError("writer response root must be an object")
+ return parsed
+
+
+def _require_exact_keys(value: Mapping[str, Any], expected: set[str], *, path: str) -> None:
+ actual = set(value)
+ if actual != expected:
+ raise ProductWriterError(
+ f"{path} keys differ from schema; missing={sorted(expected - actual)}, extra={sorted(actual - expected)}"
+ )
+
+
+def _require_string(value: Any, *, path: str, allow_empty: bool = False) -> str:
+ if not isinstance(value, str):
+ raise ProductWriterError(f"{path} must be a string")
+ if not allow_empty and not value:
+ raise ProductWriterError(f"{path} must not be empty")
+ if value != value.strip():
+ raise ProductWriterError(f"{path} must not have surrounding whitespace")
+ return value
+
+
+def _require_int(value: Any, *, path: str) -> int:
+ if isinstance(value, bool) or not isinstance(value, int):
+ raise ProductWriterError(f"{path} must be an integer")
+ return value
+
+
+def validate_writer_output(
+ payload: Mapping[str, Any],
+ current_message: str,
+ *,
+ message_role: str,
+ pending_interaction_ids: Sequence[str] = (),
+) -> dict[str, Any]:
+ root_keys = {"schema_version", "message_role", "assertions", "interactions", "resolutions"}
+ _require_exact_keys(dict(payload), root_keys, path="root")
+ if payload.get("schema_version") != WRITE_SCHEMA_VERSION:
+ raise ProductWriterError(f"unexpected writer schema: {payload.get('schema_version')!r}")
+ output_role = _require_string(payload.get("message_role"), path="root.message_role")
+ role_warning: dict[str, Any] | None = None
+ if output_role != message_role:
+ role_warning = {
+ "path": "root.message_role",
+ "code": "model_message_role_overridden",
+ "error": (
+ "source message role is controller-owned; "
+ f"overrode model value {output_role!r} with {message_role!r}"
+ ),
+ "dropped_count": 0,
+ }
+ output_role = message_role
+
+ raw_layers = {
+ "assertions": payload.get("assertions"),
+ "interactions": payload.get("interactions"),
+ "resolutions": payload.get("resolutions"),
+ }
+ warnings: list[dict[str, Any]] = [role_warning] if role_warning else []
+
+ def warn(path: str, code: str, error: str, *, dropped_count: int = 0) -> None:
+ warnings.append(
+ {
+ "path": path,
+ "code": code,
+ "error": error,
+ "dropped_count": max(0, int(dropped_count)),
+ }
+ )
+
+ def enum_value(value: Any, allowed: set[str], *, path: str) -> str:
+ original = _require_string(value, path=path)
+ normalized = original.lower()
+ if normalized != original:
+ warn(path, "identifier_case_normalized", f"normalized {original!r} to {normalized!r}")
+ if normalized not in allowed:
+ raise ProductWriterError(f"{path} is unsupported: {original!r}")
+ return normalized
+
+ def role_identifier(value: Any, *, path: str) -> str:
+ original = _require_string(value, path=path)
+ normalized = original.lower()
+ if normalized != original:
+ warn(path, "identifier_case_normalized", f"normalized {original!r} to {normalized!r}")
+ if not ROLE_RE.fullmatch(normalized):
+ raise ProductWriterError(f"{path} is not snake_case: {original!r}")
+ return normalized
+
+ def canonical_identifier(value: Any, *, path: str) -> str:
+ original = _require_string(value, path=path)
+ lowered = original.lower()
+ if lowered != original:
+ warn(path, "identifier_case_normalized", f"normalized {original!r} to {lowered!r}")
+ if not KEY_RE.fullmatch(lowered):
+ raise ProductWriterError(f"{path} is not a canonical identifier: {original!r}")
+ normalized = normalize_canonical_key(lowered)
+ if normalized != lowered:
+ warn(path, "identifier_separator_normalized", f"normalized {lowered!r} to {normalized!r}")
+ return normalized
+
+ for name, values in raw_layers.items():
+ if not isinstance(values, list):
+ raise ProductWriterError(f"root.{name} must be an array")
+ assertions = list(raw_layers["assertions"])
+ interactions = list(raw_layers["interactions"])
+ resolutions = list(raw_layers["resolutions"])
+ if message_role != "user" and assertions:
+ raise ProductWriterError(f"{message_role} messages cannot emit user assertions")
+
+ evidence_catalog = {span["span_id"]: span for span in exact_evidence_spans(current_message)}
+ token_matches = source_token_matches(current_message)
+
+ def resolve_evidence_span(value: Any, *, path: str) -> tuple[str, str, int, int]:
+ span_id = _require_string(value, path=path)
+ evidence_span = evidence_catalog.get(span_id)
+ if evidence_span is None:
+ raise ProductWriterError(f"{path} is not in the current-message evidence catalog: {span_id!r}")
+ return (
+ span_id,
+ str(evidence_span["text"]),
+ int(evidence_span["char_start"]),
+ int(evidence_span["char_end"]),
+ )
+
+ def validate_facets(
+ raw_facets: Any,
+ *,
+ path: str,
+ ) -> list[dict[str, Any]]:
+ if not isinstance(raw_facets, list):
+ warn(path, "invalid_facet_array_dropped", f"{path} must be an array")
+ return []
+ output: list[dict[str, Any]] = []
+ seen_facets: set[tuple[str, str, int, int]] = set()
+ for facet_index, raw_facet in enumerate(raw_facets):
+ facet_path = f"{path}[{facet_index}]"
+ try:
+ if not isinstance(raw_facet, dict):
+ raise ProductWriterError(f"{facet_path} must be an object")
+ required_facet_keys = {"type", "role", "token_start", "token_end"}
+ actual_facet_keys = frozenset(raw_facet)
+ if actual_facet_keys not in {frozenset(required_facet_keys), frozenset({*required_facet_keys, "quote"})}:
+ raise ProductWriterError(
+ f"{facet_path} keys differ from schema; "
+ f"missing={sorted(required_facet_keys - actual_facet_keys)}, "
+ f"extra={sorted(actual_facet_keys - required_facet_keys)}"
+ )
+ facet_type = enum_value(raw_facet.get("type"), FACET_TYPES, path=f"{facet_path}.type")
+ facet_role = role_identifier(raw_facet.get("role"), path=f"{facet_path}.role")
+ token_start = _require_int(raw_facet.get("token_start"), path=f"{facet_path}.token_start")
+ token_end = _require_int(raw_facet.get("token_end"), path=f"{facet_path}.token_end")
+ if token_start < 0 or token_end < token_start or token_end >= len(token_matches):
+ raise ProductWriterError(f"{facet_path} has an out-of-range source token interval")
+ facet_quote = current_message[token_matches[token_start].start() : token_matches[token_end].end()]
+ if "quote" in raw_facet:
+ supplied_quote = raw_facet.get("quote")
+ warn(
+ facet_path,
+ (
+ "redundant_facet_quote_ignored"
+ if isinstance(supplied_quote, str) and supplied_quote == facet_quote
+ else "mismatched_redundant_facet_quote_ignored"
+ ),
+ "token coordinates remain the authoritative exact-source facet evidence",
+ )
+ key = (facet_type, facet_role, token_start, token_end)
+ if key in seen_facets:
+ warn(facet_path, "duplicate_facet_dropped", "duplicates an earlier facet", dropped_count=1)
+ continue
+ seen_facets.add(key)
+ output.append(
+ {
+ "type": facet_type,
+ "role": facet_role,
+ "token_start": token_start,
+ "token_end": token_end,
+ "quote": facet_quote,
+ }
+ )
+ except ProductWriterError as exc:
+ warn(facet_path, "invalid_facet_quarantined", str(exc), dropped_count=1)
+ return output
+
+ normalized_assertions: list[dict[str, Any]] = []
+ assertion_seen: dict[tuple[Any, ...], dict[str, Any]] = {}
+ assertion_keys = {
+ "memory_type", "entity_key", "attribute_key", "operation", "evidence_span_id", "relation",
+ "temporal_status", "polarity", "facets",
+ }
+ for assertion_index, raw_assertion in enumerate(assertions):
+ path = f"root.assertions[{assertion_index}]"
+ try:
+ if not isinstance(raw_assertion, dict):
+ raise ProductWriterError(f"{path} must be an object")
+ _require_exact_keys(raw_assertion, assertion_keys, path=path)
+ memory_type = enum_value(raw_assertion.get("memory_type"), MEMORY_TYPES, path=f"{path}.memory_type")
+ entity_key = canonical_identifier(raw_assertion.get("entity_key"), path=f"{path}.entity_key")
+ attribute_key = canonical_identifier(raw_assertion.get("attribute_key"), path=f"{path}.attribute_key")
+ operation = enum_value(raw_assertion.get("operation"), OPERATIONS, path=f"{path}.operation")
+ evidence_span_id, evidence_quote, evidence_char_start, evidence_char_end = resolve_evidence_span(
+ raw_assertion.get("evidence_span_id"), path=f"{path}.evidence_span_id"
+ )
+ relation = role_identifier(raw_assertion.get("relation"), path=f"{path}.relation")
+ temporal_status = enum_value(
+ raw_assertion.get("temporal_status"), TEMPORAL_STATUSES, path=f"{path}.temporal_status"
+ )
+ polarity = enum_value(raw_assertion.get("polarity"), POLARITIES, path=f"{path}.polarity")
+ entity_key = entity_key.removeprefix("user.")
+ assertion_family = memory_family(memory_type)
+ assertion_graph_entity = graph_entity_key(entity_key, attribute_key)
+ canonical_key = f"user.{entity_key}.{assertion_family}.{attribute_key}"
+ if not entity_key or not attribute_key or len(canonical_key) > 160:
+ raise ProductWriterError(f"{path} has an invalid entity/attribute key or relation")
+ normalized_facets = validate_facets(raw_assertion.get("facets"), path=f"{path}.facets")
+ append_member_identity = ()
+ if operation == "append":
+ append_member_identity = tuple(
+ sorted(
+ (
+ facet["type"],
+ facet["role"],
+ facet["token_start"],
+ facet["token_end"],
+ )
+ for facet in normalized_facets
+ )
+ )
+ key = (
+ canonical_key,
+ evidence_char_start,
+ evidence_char_end,
+ memory_type,
+ operation,
+ relation,
+ temporal_status,
+ polarity,
+ append_member_identity,
+ )
+ existing_assertion = assertion_seen.get(key)
+ if existing_assertion is not None:
+ existing_facets = existing_assertion["facets"]
+ existing_facet_keys = {
+ (facet["type"], facet["role"], facet["token_start"], facet["token_end"])
+ for facet in existing_facets
+ }
+ for facet in normalized_facets:
+ facet_key = (facet["type"], facet["role"], facet["token_start"], facet["token_end"])
+ if facet_key not in existing_facet_keys:
+ existing_facets.append(facet)
+ existing_facet_keys.add(facet_key)
+ warn(path, "duplicate_assertion_merged", "merged facets into an identical assertion")
+ continue
+ normalized_assertion = {
+ "memory_type": memory_type,
+ "entity_key": entity_key,
+ "attribute_key": attribute_key,
+ "memory_family": assertion_family,
+ "graph_entity_key": assertion_graph_entity,
+ "canonical_key": canonical_key,
+ "operation": operation,
+ "evidence_span_id": evidence_span_id,
+ "evidence_quote": evidence_quote,
+ "evidence_char_start": evidence_char_start,
+ "evidence_char_end": evidence_char_end,
+ "subject": "user",
+ "relation": relation,
+ "object": "",
+ "temporal_status": temporal_status,
+ "polarity": polarity,
+ "facets": normalized_facets,
+ }
+ assertion_seen[key] = normalized_assertion
+ normalized_assertions.append(normalized_assertion)
+ except ProductWriterError as exc:
+ warn(path, "invalid_assertion_quarantined", str(exc), dropped_count=1)
+
+ normalized_interactions: list[dict[str, Any]] = []
+ interaction_seen: dict[tuple[Any, ...], dict[str, Any]] = {}
+ for interaction_index, raw_interaction in enumerate(interactions):
+ path = f"root.interactions[{interaction_index}]"
+ try:
+ if not isinstance(raw_interaction, dict):
+ raise ProductWriterError(f"{path} must be an object")
+ _require_exact_keys(
+ raw_interaction,
+ {"interaction_type", "status", "evidence_span_id", "intent", "about"},
+ path=path,
+ )
+ interaction_type = enum_value(
+ raw_interaction.get("interaction_type"), INTERACTION_TYPES, path=f"{path}.interaction_type"
+ )
+ status = enum_value(raw_interaction.get("status"), INTERACTION_STATUSES, path=f"{path}.status")
+ evidence_span_id, evidence_quote, evidence_char_start, evidence_char_end = resolve_evidence_span(
+ raw_interaction.get("evidence_span_id"), path=f"{path}.evidence_span_id"
+ )
+ intent = role_identifier(raw_interaction.get("intent"), path=f"{path}.intent")
+ normalized_about = validate_facets(raw_interaction.get("about"), path=f"{path}.about")
+ key = (
+ interaction_type,
+ status,
+ intent,
+ evidence_char_start,
+ evidence_char_end,
+ )
+ existing_interaction = interaction_seen.get(key)
+ if existing_interaction is not None:
+ existing_about = existing_interaction["about"]
+ existing_about_keys = {
+ (facet["type"], facet["role"], facet["token_start"], facet["token_end"])
+ for facet in existing_about
+ }
+ for facet in normalized_about:
+ facet_key = (facet["type"], facet["role"], facet["token_start"], facet["token_end"])
+ if facet_key not in existing_about_keys:
+ existing_about.append(facet)
+ existing_about_keys.add(facet_key)
+ warn(path, "duplicate_interaction_merged", "merged about items into an identical interaction")
+ continue
+ normalized_interaction = {
+ "interaction_type": interaction_type,
+ "status": status,
+ "evidence_span_id": evidence_span_id,
+ "evidence_quote": evidence_quote,
+ "evidence_char_start": evidence_char_start,
+ "evidence_char_end": evidence_char_end,
+ "intent": intent,
+ "about": normalized_about,
+ }
+ interaction_seen[key] = normalized_interaction
+ normalized_interactions.append(normalized_interaction)
+ except ProductWriterError as exc:
+ warn(path, "invalid_interaction_quarantined", str(exc), dropped_count=1)
+
+ allowed_pending = set(pending_interaction_ids)
+ normalized_resolutions: list[dict[str, Any]] = []
+ resolution_seen: dict[str, str] = {}
+ resolution_conflicts: list[str] = []
+ for resolution_index, raw_resolution in enumerate(resolutions):
+ path = f"root.resolutions[{resolution_index}]"
+ try:
+ if not isinstance(raw_resolution, dict):
+ raise ProductWriterError(f"{path} must be an object")
+ _require_exact_keys(raw_resolution, {"interaction_id", "resolution", "evidence_span_id"}, path=path)
+ interaction_id = _require_string(raw_resolution.get("interaction_id"), path=f"{path}.interaction_id")
+ resolution = enum_value(
+ raw_resolution.get("resolution"), RESOLUTION_STATES, path=f"{path}.resolution"
+ )
+ evidence_span_id, evidence_quote, evidence_char_start, evidence_char_end = resolve_evidence_span(
+ raw_resolution.get("evidence_span_id"), path=f"{path}.evidence_span_id"
+ )
+ if interaction_id not in allowed_pending:
+ raise ProductWriterError(f"{path}.interaction_id was not supplied as pending: {interaction_id!r}")
+ existing_resolution = resolution_seen.get(interaction_id)
+ if existing_resolution is not None:
+ if existing_resolution != resolution:
+ resolution_conflicts.append(
+ f"{interaction_id!r}: {existing_resolution!r} != {resolution!r}"
+ )
+ else:
+ warn(path, "duplicate_resolution_merged", "merged an identical resolution")
+ continue
+ resolution_seen[interaction_id] = resolution
+ normalized_resolutions.append(
+ {
+ "interaction_id": interaction_id,
+ "resolution": resolution,
+ "evidence_span_id": evidence_span_id,
+ "evidence_quote": evidence_quote,
+ "evidence_char_start": evidence_char_start,
+ "evidence_char_end": evidence_char_end,
+ }
+ )
+ except ProductWriterError as exc:
+ warn(path, "invalid_resolution_quarantined", str(exc), dropped_count=1)
+ if resolution_conflicts:
+ raise ProductWriterError(
+ "root.resolutions contains conflicting states for the same interaction: "
+ + "; ".join(resolution_conflicts)
+ )
+ return {
+ "schema_version": WRITE_SCHEMA_VERSION,
+ "message_role": output_role,
+ "assertions": normalized_assertions,
+ "interactions": normalized_interactions,
+ "resolutions": normalized_resolutions,
+ "validation_warnings": warnings,
+ "quarantined_item_count": sum(int(warning.get("dropped_count", 0) or 0) for warning in warnings),
+ }
+
+
+class DeepSeekProductWriter:
+ def __init__(
+ self,
+ *,
+ base_url: str,
+ model: str,
+ reviewer_model: str,
+ api_keys: Sequence[str],
+ timeout: float,
+ max_tokens: int,
+ ) -> None:
+ self.base_url = base_url.rstrip("/")
+ self.model = model
+ self.reviewer_model = reviewer_model
+ self.api_keys = list(dict.fromkeys(clean_text(value) for value in api_keys if clean_text(value)))
+ self.request_index = 0
+ self.timeout = max(1.0, float(timeout))
+ self.max_tokens = max(256, int(max_tokens))
+ if not self.base_url or not self.api_keys:
+ raise ProductWriterError("writer base URL and API key pool are required")
+ if self.model != "deepseek-v4-flash":
+ raise ProductWriterError(f"product writer requires deepseek-v4-flash, got {self.model!r}")
+ if self.reviewer_model != "deepseek-v4-pro":
+ raise ProductWriterError(f"product writer requires deepseek-v4-pro, got {self.reviewer_model!r}")
+
+ def _request(
+ self,
+ *,
+ model: str,
+ system_prompt: str,
+ user_payload: Mapping[str, Any],
+ stage: str,
+ ) -> tuple[str, dict[str, Any]]:
+ request_payload = {
+ "model": model,
+ "messages": [
+ {"role": "system", "content": system_prompt},
+ {"role": "user", "content": json.dumps(user_payload, ensure_ascii=False, separators=(",", ":"))},
+ ],
+ "temperature": 0,
+ "max_tokens": self.max_tokens,
+ "response_format": {"type": "json_object"},
+ "thinking": {"type": "disabled"},
+ }
+ key_index = self.request_index % len(self.api_keys)
+ self.request_index += 1
+ request = urllib.request.Request(
+ f"{self.base_url}/chat/completions",
+ data=json.dumps(request_payload, ensure_ascii=False).encode("utf-8"),
+ headers={"Content-Type": "application/json", "Authorization": f"Bearer {self.api_keys[key_index]}"},
+ method="POST",
+ )
+ started = time.time()
+ try:
+ with urllib.request.urlopen(request, timeout=self.timeout) as response:
+ response_payload = json.loads(response.read().decode("utf-8"))
+ except urllib.error.HTTPError as exc:
+ detail = exc.read().decode("utf-8", errors="replace")[:1000]
+ metadata = {
+ "model": model,
+ "api_key_index": key_index,
+ "latency_seconds": round(time.time() - started, 3),
+ "response_sha256": sha256_text(detail),
+ "finish_reason": "http_error",
+ "http_status": int(exc.code),
+ "max_output_tokens": self.max_tokens,
+ "prompt_tokens": 0,
+ "completion_tokens": 0,
+ "total_tokens": 0,
+ }
+ raise ProductWriterResponseError(
+ f"{stage} HTTP {exc.code}: {detail}",
+ response_content=detail,
+ request_metadata=metadata,
+ ) from exc
+ except Exception as exc:
+ metadata = {
+ "model": model,
+ "api_key_index": key_index,
+ "latency_seconds": round(time.time() - started, 3),
+ "response_sha256": sha256_text(""),
+ "finish_reason": "request_error",
+ "error_type": exc.__class__.__name__,
+ "max_output_tokens": self.max_tokens,
+ "prompt_tokens": 0,
+ "completion_tokens": 0,
+ "total_tokens": 0,
+ }
+ raise ProductWriterResponseError(
+ f"{stage} request failed: {exc.__class__.__name__}: {exc}",
+ request_metadata=metadata,
+ ) from exc
+ choices = list(response_payload.get("choices") or [])
+ usage = dict(response_payload.get("usage") or {})
+ raw_response = json.dumps(response_payload, ensure_ascii=False, sort_keys=True)
+ if len(choices) != 1:
+ raise ProductWriterResponseError(
+ f"{stage} returned {len(choices)} choices",
+ response_content=raw_response,
+ request_metadata={
+ "model": model,
+ "api_key_index": key_index,
+ "latency_seconds": round(time.time() - started, 3),
+ "response_sha256": sha256_text(raw_response),
+ "finish_reason": "invalid_response",
+ "max_output_tokens": self.max_tokens,
+ "prompt_tokens": int(usage.get("prompt_tokens", 0) or 0),
+ "completion_tokens": int(usage.get("completion_tokens", 0) or 0),
+ "total_tokens": int(usage.get("total_tokens", 0) or 0),
+ },
+ )
+ content = dict(choices[0].get("message") or {}).get("content")
+ if not isinstance(content, str):
+ raise ProductWriterResponseError(
+ f"{stage} response has no string content",
+ response_content=raw_response,
+ request_metadata={
+ "model": model,
+ "api_key_index": key_index,
+ "latency_seconds": round(time.time() - started, 3),
+ "response_sha256": sha256_text(raw_response),
+ "finish_reason": "invalid_response",
+ "max_output_tokens": self.max_tokens,
+ "prompt_tokens": int(usage.get("prompt_tokens", 0) or 0),
+ "completion_tokens": int(usage.get("completion_tokens", 0) or 0),
+ "total_tokens": int(usage.get("total_tokens", 0) or 0),
+ },
+ )
+ finish_reason = clean_text(choices[0].get("finish_reason"))
+ metadata = {
+ "model": model,
+ "api_key_index": key_index,
+ "latency_seconds": round(time.time() - started, 3),
+ "response_sha256": sha256_text(content),
+ "finish_reason": finish_reason,
+ "max_output_tokens": self.max_tokens,
+ "prompt_tokens": int(usage.get("prompt_tokens", 0) or 0),
+ "completion_tokens": int(usage.get("completion_tokens", 0) or 0),
+ "total_tokens": int(usage.get("total_tokens", 0) or 0),
+ }
+ if finish_reason != "stop":
+ raise ProductWriterResponseError(
+ f"{stage} did not finish cleanly: finish_reason={finish_reason!r}",
+ response_content=content,
+ request_metadata=metadata,
+ )
+ return content, metadata
+
+ def write(
+ self,
+ *,
+ current_message: Mapping[str, Any],
+ previous_message: Mapping[str, Any] | None,
+ existing_memory_slots: Sequence[Mapping[str, Any]] = (),
+ pending_interactions: Sequence[Mapping[str, Any]] = (),
+ ) -> tuple[dict[str, Any], dict[str, Any]]:
+ current_content = str(current_message.get("content") or "")
+ pending_ids = [clean_text(value.get("interaction_id")) for value in pending_interactions]
+ message_role = clean_text(current_message.get("role")).lower()
+ if message_role == "user":
+ selected_model = self.reviewer_model
+ selected_prompt = PRO_SYSTEM_PROMPT
+ selected_prompt_version = PRO_PROMPT_VERSION
+ routing_reason = "authoritative_user_memory"
+ elif message_role == "assistant" and pending_interactions:
+ selected_model = self.reviewer_model
+ selected_prompt = PRO_SYSTEM_PROMPT
+ selected_prompt_version = PRO_PROMPT_VERSION
+ routing_reason = "pending_interaction_resolution"
+ elif message_role == "assistant":
+ selected_model = self.model
+ selected_prompt = SYSTEM_PROMPT
+ selected_prompt_version = PROMPT_VERSION
+ routing_reason = "assistant_source_or_interaction"
+ else:
+ raise ProductWriterError(f"unsupported routed writer role: {message_role!r}")
+
+ previous_payload = (
+ {
+ "role": clean_text(previous_message.get("role")),
+ "timestamp": clean_text(previous_message.get("timestamp")),
+ "content": str(previous_message.get("content") or ""),
+ }
+ if previous_message
+ else None
+ )
+
+ def invoke(
+ content: str,
+ *,
+ stage: str,
+ capacity_range: tuple[int, int] | None,
+ ) -> tuple[dict[str, Any], dict[str, Any], dict[str, Any], str]:
+ user_payload: dict[str, Any] = {
+ "current_message": {
+ "role": clean_text(current_message.get("role")),
+ "timestamp": clean_text(current_message.get("timestamp")),
+ "content": content,
+ },
+ "previous_message": previous_payload,
+ "evidence_spans": compact_writer_evidence_spans(content),
+ "source_tokens": [token["text"] for token in exact_source_tokens(content)],
+ "existing_memory_slots": [dict(value) for value in existing_memory_slots],
+ "pending_interactions": [dict(value) for value in pending_interactions],
+ }
+ if capacity_range is not None:
+ user_payload["capacity_segment"] = {
+ "char_start": capacity_range[0],
+ "char_end": capacity_range[1],
+ }
+ user_payload["full_current_message_context"] = {
+ "role": clean_text(current_message.get("role")),
+ "timestamp": clean_text(current_message.get("timestamp")),
+ "content": current_content,
+ }
+ response_content, request_metadata = self._request(
+ model=selected_model,
+ system_prompt=selected_prompt,
+ user_payload=user_payload,
+ stage=stage,
+ )
+ try:
+ transport_warnings: list[dict[str, Any]] = []
+ wire_output = exact_json_object(response_content, warnings=transport_warnings)
+ parsed = validate_writer_output(
+ wire_output,
+ content,
+ message_role=message_role,
+ pending_interaction_ids=pending_ids,
+ )
+ parsed["validation_warnings"] = [
+ *transport_warnings,
+ *list(parsed.get("validation_warnings") or []),
+ ]
+ except ProductWriterError as exc:
+ raise ProductWriterResponseError(
+ str(exc),
+ response_content=response_content,
+ request_metadata=request_metadata,
+ ) from exc
+ return parsed, request_metadata, wire_output, response_content
+
+ def annotate_request(
+ request_metadata: Mapping[str, Any],
+ *,
+ attempt: str,
+ capacity_range: tuple[int, int] | None,
+ ) -> dict[str, Any]:
+ annotated = dict(request_metadata)
+ annotated["attempt"] = attempt
+ if capacity_range is not None:
+ annotated["segment_char_start"] = capacity_range[0]
+ annotated["segment_char_end"] = capacity_range[1]
+ return annotated
+
+ def rebase_segment(parsed: dict[str, Any], capacity_range: tuple[int, int], segment_index: int) -> None:
+ segment_start, segment_end = capacity_range
+ segment_content = current_content[segment_start:segment_end]
+ local_tokens = source_token_matches(segment_content)
+ global_tokens = source_token_matches(current_content)
+
+ def original_token_index(char_offset: int) -> int:
+ for index, match in enumerate(global_tokens):
+ if match.start() <= char_offset < match.end():
+ return index
+ raise ProductWriterError("capacity segment token coordinates do not map to the original source")
+
+ def rebase_item(item: dict[str, Any], facet_field: str | None = None) -> None:
+ item["evidence_span_id"] = f"c{segment_index}.{item['evidence_span_id']}"
+ item["evidence_char_start"] = segment_start + int(item["evidence_char_start"])
+ item["evidence_char_end"] = segment_start + int(item["evidence_char_end"])
+ if facet_field is None:
+ return
+ for facet in item.get(facet_field) or []:
+ local_start = int(facet["token_start"])
+ local_end = int(facet["token_end"])
+ global_start = segment_start + local_tokens[local_start].start()
+ global_end = segment_start + local_tokens[local_end].start()
+ facet["token_start"] = original_token_index(global_start)
+ facet["token_end"] = original_token_index(global_end)
+
+ for assertion in parsed.get("assertions") or []:
+ rebase_item(assertion, "facets")
+ for interaction in parsed.get("interactions") or []:
+ rebase_item(interaction, "about")
+ for resolution in parsed.get("resolutions") or []:
+ rebase_item(resolution)
+
+ primary_stage = "writer_primary_pro" if selected_model == self.reviewer_model else "writer_primary_flash"
+ requests: list[dict[str, Any]] = []
+ wire_outputs: list[dict[str, Any]] = []
+ response_hashes: list[str] = []
+ duplicate_count = 0
+ try:
+ parsed, request_metadata, wire_output, response_content = invoke(
+ current_content,
+ stage=primary_stage,
+ capacity_range=None,
+ )
+ requests.append(annotate_request(request_metadata, attempt="full", capacity_range=None))
+ wire_outputs.append({"capacity_range": None, "output": wire_output})
+ response_hashes.append(sha256_text(response_content))
+ writer_mode = "single_pass_routed"
+ except ProductWriterResponseError as initial_error:
+ if initial_error.request_metadata.get("finish_reason") != "length":
+ raise
+ requests.append(annotate_request(initial_error.request_metadata, attempt="full", capacity_range=None))
+ response_hashes.append(sha256_text(initial_error.response_content))
+ initial_split = split_capacity_range(current_content, 0, len(current_content))
+ if initial_split is None:
+ raise
+
+ segment_outputs: list[dict[str, Any]] = []
+ pending_ranges = [initial_split[0], initial_split[1]]
+ while pending_ranges:
+ if len(requests) >= 64:
+ raise ProductWriterResponseError(
+ "capacity segmentation exceeded 64 API attempts; source was retained and nothing was committed",
+ request_metadata={"requests": requests},
+ )
+ capacity_range = pending_ranges.pop(0)
+ segment_content = current_content[capacity_range[0] : capacity_range[1]]
+ segment_stage = (
+ "writer_capacity_segment_pro"
+ if selected_model == self.reviewer_model
+ else "writer_capacity_segment_flash"
+ )
+ try:
+ segment_output, segment_request, segment_wire, segment_response = invoke(
+ segment_content,
+ stage=segment_stage,
+ capacity_range=capacity_range,
+ )
+ except ProductWriterResponseError as segment_error:
+ requests.append(
+ annotate_request(
+ segment_error.request_metadata,
+ attempt="capacity_segment",
+ capacity_range=capacity_range,
+ )
+ )
+ response_hashes.append(sha256_text(segment_error.response_content))
+ if segment_error.request_metadata.get("finish_reason") != "length":
+ raise ProductWriterResponseError(
+ str(segment_error),
+ response_content=segment_error.response_content,
+ request_metadata={
+ "requests": requests,
+ "physical_api_attempt_count": len(requests),
+ "terminal_request": requests[-1],
+ },
+ ) from segment_error
+ nested_split = split_capacity_range(current_content, capacity_range[0], capacity_range[1])
+ if nested_split is None:
+ raise ProductWriterResponseError(
+ "an indivisible capacity segment still exceeded the model output limit",
+ response_content=segment_error.response_content,
+ request_metadata={"requests": requests},
+ ) from segment_error
+ pending_ranges = [nested_split[0], nested_split[1], *pending_ranges]
+ continue
+ requests.append(
+ annotate_request(
+ segment_request,
+ attempt="capacity_segment",
+ capacity_range=capacity_range,
+ )
+ )
+ response_hashes.append(sha256_text(segment_response))
+ rebase_segment(segment_output, capacity_range, len(segment_outputs))
+ segment_outputs.append(segment_output)
+ wire_outputs.append(
+ {
+ "capacity_range": {"char_start": capacity_range[0], "char_end": capacity_range[1]},
+ "output": segment_wire,
+ }
+ )
+ try:
+ parsed, duplicate_count = merge_segment_outputs(segment_outputs, message_role=message_role)
+ except ProductWriterError as merge_error:
+ raise ProductWriterResponseError(
+ str(merge_error),
+ request_metadata={
+ "requests": requests,
+ "physical_api_attempt_count": len(requests),
+ "terminal_request": requests[-1],
+ },
+ ) from merge_error
+ writer_mode = "capacity_segmented_on_length"
+
+ metadata = {
+ "writer_mode": writer_mode,
+ "routing_reason": routing_reason,
+ "prompt_version": selected_prompt_version,
+ "prompt_sha256": sha256_text(selected_prompt),
+ "model": selected_model,
+ "request": requests[0],
+ "requests": requests,
+ "api_call_count": len(requests),
+ "capacity_segment_count": len(wire_outputs) if writer_mode != "single_pass_routed" else 0,
+ "capacity_duplicate_count": duplicate_count,
+ "wire_output": wire_outputs[0]["output"] if writer_mode == "single_pass_routed" else None,
+ "wire_outputs": wire_outputs,
+ "latency_seconds": round(sum(float(request.get("latency_seconds", 0) or 0) for request in requests), 3),
+ "response_sha256": sha256_text("|".join(response_hashes)),
+ "prompt_tokens": sum(int(request.get("prompt_tokens", 0) or 0) for request in requests),
+ "completion_tokens": sum(int(request.get("completion_tokens", 0) or 0) for request in requests),
+ "total_tokens": sum(int(request.get("total_tokens", 0) or 0) for request in requests),
+ }
+ return parsed, metadata
+
+
+def historical_timestamp(value: Any, message_index: int) -> str:
+ text = clean_text(value)
+ for pattern in ("%Y/%m/%d (%a) %H:%M", "%Y/%m/%d %H:%M", "%Y-%m-%d %H:%M"):
+ try:
+ base = datetime.strptime(text, pattern).replace(tzinfo=timezone.utc)
+ return (base + timedelta(seconds=int(message_index))).isoformat(timespec="seconds")
+ except ValueError:
+ continue
+ raise ProductWriterError(f"unsupported historical timestamp: {text!r}")
+
+
+def graph_category(memory_type: str) -> str:
+ return {
+ "identity": "profile",
+ "relationship": "profile",
+ "routine": "profile",
+ "state": "status",
+ "plan": "stage_state",
+ }.get(memory_type, memory_type)
+
+
+def graph_facet_type(facet_type: str) -> str:
+ return {
+ "time": "temporal",
+ "quantity": "numeric",
+ "location": "entity",
+ "role": "role",
+ }.get(facet_type, facet_type)
+
+
+def build_graph_records(
+ record_class: Any,
+ *,
+ scope_id: str,
+ turn_index: int,
+ session_id: str,
+ session_index: int,
+ message_id: str,
+ message_index: int,
+ date: str,
+ timestamp: str,
+ role: str,
+ content: str,
+ extraction: Mapping[str, Any] | None,
+) -> tuple[list[Any], dict[str, int]]:
+ event_id = f"event::tmcra:{scope_id}:{message_id}"
+ sidecar = {
+ "session_id": session_id,
+ "session_index": int(session_index),
+ "message_id": message_id,
+ "message_index": int(message_index),
+ "historical_date": date,
+ "role": role,
+ }
+ source_slot = f"source.s{session_index:03d}.m{message_index:03d}"
+ source_metadata = {
+ "source": "tmcra_v3_product_runtime",
+ "writer_schema_version": WRITE_SCHEMA_VERSION,
+ "prompt_version": PROMPT_VERSION,
+ "content_variant": "source_message",
+ "memory_layer": "fast",
+ "node_kind": "immutable_source_message",
+ "immutable_evidence_leaf": True,
+ "raw_content": content,
+ "source_span": content,
+ "source_turn_text": content,
+ "speaker": role,
+ "timestamp": timestamp,
+ "session_id": session_id,
+ "session_index": int(session_index),
+ "message_id": message_id,
+ "message_index": int(message_index),
+ "historical_date": date,
+ "event_id": event_id,
+ "source_record_id": f"{source_slot}:{turn_index}",
+ "event_signature": f"source:{message_id}",
+ "dia_id": f"tmcra:{scope_id}:{message_id}",
+ "canonical_slot_key": source_slot,
+ "semantic_slot": "source_message",
+ "allow_parallel_state": True,
+ "memory_gate_decision": "source_grounding",
+ "subject_signature": slug(f"source.{message_id}"),
+ "sidecar_hint_metadata": sidecar,
+ }
+ records: list[Any] = [
+ record_class(
+ memory_id=f"{source_slot}:{turn_index}",
+ category="source",
+ slot_key=source_slot,
+ value=content,
+ relation="dialogue_source",
+ anchor_concepts=[role, session_id, date],
+ evidence_anchors=[role, session_id, date],
+ salience=0.72,
+ confidence=1.0,
+ source_kind=f"public_dialog_{role}_turn",
+ turn_index=turn_index,
+ state="evidence",
+ metadata=source_metadata,
+ )
+ ]
+ counts = {"source": 1, "semantic": 0, "facet": 0, "interaction": 0}
+ assertions = list((extraction or {}).get("assertions") or [])
+ for memory_index_in_message, memory in enumerate(assertions):
+ canonical_key = str(memory["canonical_key"])
+ entity_key = str(memory["entity_key"])
+ attribute_key = str(memory["attribute_key"])
+ operation = str(memory["operation"])
+ evidence_quote = str(memory["evidence_quote"])
+ evidence_char_start = int(memory.get("evidence_char_start", 0) or 0)
+ evidence_char_end = int(memory.get("evidence_char_end", 0) or 0)
+ memory_type = str(memory["memory_type"])
+ assertion_family = str(memory["memory_family"])
+ assertion_graph_entity = str(memory["graph_entity_key"])
+ relation = str(memory["relation"])
+ subject = str(memory["subject"])
+ object_value = str(memory["object"])
+ slot_key = f"memory.{canonical_key}"
+ value_hash = sha256_text(evidence_quote)[:16]
+ event_signature = (
+ f"{canonical_key}:{message_id}:{value_hash}:"
+ f"{evidence_char_start}:{evidence_char_end}:{memory['polarity']}"
+ )
+ anchors = [subject, relation, object_value]
+ for facet in list(memory.get("facets") or []):
+ anchors.extend([str(facet["role"]), str(facet["quote"])])
+ anchors = [value for value in dict.fromkeys(clean_text(value) for value in anchors) if value]
+ metadata = {
+ **source_metadata,
+ "content_variant": "product_semantic_memory",
+ "memory_layer": "fast",
+ "node_kind": "atomic_user_assertion",
+ "atomic_evidence_leaf": True,
+ "authority": "user_assertion",
+ "raw_content": evidence_quote,
+ "source_span": evidence_quote,
+ "evidence_char_start": int(memory.get("evidence_char_start", 0) or 0),
+ "evidence_char_end": int(memory.get("evidence_char_end", 0) or 0),
+ "source_turn_text": content,
+ "canonical_slot_key": slot_key,
+ "semantic_slot": relation,
+ "memory_type": memory_type,
+ "memory_family": assertion_family,
+ "entity_key": entity_key,
+ "graph_entity_key": assertion_graph_entity,
+ "attribute_key": attribute_key,
+ "write_operation": operation,
+ "allow_parallel_state": operation == "append",
+ "subject": subject,
+ "subject_signature": slug(assertion_graph_entity),
+ "object": object_value,
+ "target_status": str(memory["temporal_status"]),
+ "polarity": str(memory["polarity"]),
+ "event_signature": event_signature,
+ "memory_gate_decision": "model_exact_evidence",
+ "llm_write_proposal_index": memory_index_in_message,
+ }
+ if graph_category(memory_type) == "profile":
+ metadata["profile_type"] = memory_type
+ metadata["profile_domain"] = assertion_graph_entity
+ parent = record_class(
+ memory_id=f"{slot_key}:{turn_index}:{memory_index_in_message}",
+ category=graph_category(memory_type),
+ slot_key=slot_key,
+ value=evidence_quote,
+ relation=relation,
+ anchor_concepts=anchors,
+ evidence_anchors=[evidence_quote],
+ salience=0.92,
+ confidence=1.0,
+ source_kind="public_dialog_semantic_memory",
+ turn_index=turn_index,
+ state="active",
+ metadata=metadata,
+ )
+ records.append(parent)
+ counts["semantic"] += 1
+ for facet_index, facet in enumerate(list(memory.get("facets") or [])):
+ facet_type = graph_facet_type(str(facet["type"]))
+ facet_role = str(facet["role"])
+ facet_quote = str(facet["quote"])
+ facet_slot = f"{slot_key}.facet.{message_id}.{memory_index_in_message}.{facet_index}"
+ facet_metadata = {
+ **source_metadata,
+ "content_variant": "event_facet_write",
+ "memory_layer": "fast",
+ "node_kind": "atomic_assertion_facet",
+ "raw_content": facet_quote,
+ "source_span": facet_quote,
+ "facet_source_span": facet_quote,
+ "facet_type": facet_type,
+ "facet_role": facet_role,
+ "facet_value": facet_quote,
+ "facet_parent_slot_key": slot_key,
+ "facet_parent_event_signature": event_signature,
+ "canonical_slot_key": facet_slot,
+ "semantic_slot": facet_role,
+ "allow_parallel_state": True,
+ "subject": subject,
+ "subject_signature": slug(assertion_graph_entity),
+ "graph_entity_key": assertion_graph_entity,
+ "event_signature": f"{event_signature}:facet:{facet_index}",
+ "memory_gate_decision": "model_exact_evidence_facet",
+ }
+ records.append(
+ record_class(
+ memory_id=f"{facet_slot}:{turn_index}",
+ category={"temporal": "time", "state": "status"}.get(facet_type, "fact"),
+ slot_key=facet_slot,
+ value=facet_quote,
+ relation=f"has_{facet_type}_facet",
+ anchor_concepts=[facet_role, facet_quote, subject],
+ evidence_anchors=[facet_quote],
+ salience=0.84,
+ confidence=1.0,
+ source_kind="public_dialog_semantic_facet",
+ turn_index=turn_index,
+ state="evidence",
+ metadata=facet_metadata,
+ )
+ )
+ counts["facet"] += 1
+ interactions = list((extraction or {}).get("interactions") or [])
+ for interaction_index, interaction in enumerate(interactions):
+ interaction_type = str(interaction["interaction_type"])
+ interaction_status = str(interaction["status"])
+ evidence_quote = str(interaction["evidence_quote"])
+ intent = str(interaction["intent"])
+ interaction_slot = f"interaction.{message_id}.{interaction_index}"
+ interaction_memory_id = f"{interaction_slot}:{turn_index}"
+ interaction_metadata = {
+ **source_metadata,
+ "content_variant": "product_interaction",
+ "memory_layer": "fast",
+ "node_kind": "atomic_interaction",
+ "atomic_evidence_leaf": True,
+ "raw_content": evidence_quote,
+ "source_span": evidence_quote,
+ "evidence_char_start": int(interaction.get("evidence_char_start", 0) or 0),
+ "evidence_char_end": int(interaction.get("evidence_char_end", 0) or 0),
+ "source_turn_text": content,
+ "canonical_slot_key": interaction_slot,
+ "semantic_slot": intent,
+ "interaction_id": interaction_memory_id,
+ "interaction_type": interaction_type,
+ "interaction_status": interaction_status,
+ "interaction_speaker": role,
+ "allow_parallel_state": True,
+ "subject_signature": slug(f"interaction.{message_id}.{interaction_index}"),
+ "event_signature": f"interaction:{message_id}:{interaction_index}",
+ "memory_gate_decision": "model_exact_interaction_evidence",
+ }
+ about = list(interaction.get("about") or [])
+ interaction_metadata["about"] = about
+ interaction_anchors = [intent, interaction_type]
+ for facet in about:
+ interaction_anchors.extend([str(facet["role"]), str(facet["quote"])])
+ records.append(
+ record_class(
+ memory_id=interaction_memory_id,
+ category="question" if interaction_type in {"question", "clarification"} else "interaction_intent",
+ slot_key=interaction_slot,
+ value=evidence_quote,
+ relation=intent,
+ anchor_concepts=[
+ value for value in dict.fromkeys(clean_text(value) for value in interaction_anchors) if value
+ ],
+ evidence_anchors=[evidence_quote],
+ salience=0.82,
+ confidence=1.0,
+ source_kind=(
+ "public_dialog_question"
+ if interaction_type in {"question", "clarification"}
+ else "public_dialog_interaction"
+ ),
+ turn_index=turn_index,
+ state="evidence",
+ metadata=interaction_metadata,
+ )
+ )
+ counts["interaction"] += 1
+ return records, counts
+
+
+def ingest_product_message(
+ adapter: Any,
+ record_class: Any,
+ edge_class: Any,
+ *,
+ scope_id: str,
+ session_id: str,
+ session_index: int,
+ message_id: str,
+ message_index: int,
+ date: str,
+ timestamp: str,
+ role: str,
+ content: str,
+ extraction: Mapping[str, Any] | None,
+) -> dict[str, Any]:
+ adapter._reload_graph()
+ turn_index = adapter.graph.next_turn()
+ records, counts = build_graph_records(
+ record_class,
+ scope_id=scope_id,
+ turn_index=turn_index,
+ session_id=session_id,
+ session_index=session_index,
+ message_id=message_id,
+ message_index=message_index,
+ date=date,
+ timestamp=timestamp,
+ role=role,
+ content=content,
+ extraction=extraction,
+ )
+ stored_ids = adapter.graph.add_records(records)
+ source_ids = [record.memory_id for record in records if record.category == "source"]
+ if len(source_ids) != 1 or source_ids[0] not in stored_ids:
+ raise ProductWriterError(f"{message_id}: immutable source record was not persisted")
+ provenance_count = 0
+ for record in records:
+ content_variant = clean_text(dict(record.metadata or {}).get("content_variant"))
+ if content_variant not in {"product_semantic_memory", "product_interaction"}:
+ continue
+ if record.memory_id not in stored_ids or record.memory_id not in adapter.graph.records_by_id:
+ raise ProductWriterError(
+ f"{message_id}: semantic record {record.memory_id!r} was merged or dropped before provenance"
+ )
+ adapter.graph._upsert_memory_edge(
+ edge_class(
+ edge_id=f"{record.memory_id}->{source_ids[0]}:grounded_in",
+ source_memory_id=record.memory_id,
+ target_memory_id=source_ids[0],
+ edge_type="grounded_in",
+ score=1.0,
+ model_score=0.0,
+ evidence_turn=turn_index,
+ evidence=clean_text(dict(record.metadata or {}).get("source_span")) or record.value,
+ metadata={
+ "edge_source": "product_writer_provenance",
+ "message_id": message_id,
+ "source_record_id": source_ids[0],
+ },
+ )
+ )
+ provenance_count += 1
+ resolution_rows: list[dict[str, str]] = []
+ for resolution in list((extraction or {}).get("resolutions") or []):
+ interaction_id = str(resolution["interaction_id"])
+ resolution_state = str(resolution["resolution"])
+ evidence_quote = str(resolution["evidence_quote"])
+ interaction_record = adapter.graph.records_by_id.get(interaction_id)
+ if interaction_record is None:
+ raise ProductWriterError(f"{message_id}: resolution target does not exist: {interaction_id}")
+ interaction_metadata = dict(interaction_record.metadata or {})
+ if clean_text(interaction_metadata.get("content_variant")) != "product_interaction":
+ raise ProductWriterError(f"{message_id}: resolution target is not an interaction: {interaction_id}")
+ previous_status = clean_text(interaction_metadata.get("interaction_status")) or "open"
+ next_status = "resolved" if resolution_state == "resolved" else ("partial" if resolution_state == "partial" else previous_status)
+ history = list(interaction_metadata.get("resolution_history") or [])
+ history.append(
+ {
+ "message_id": message_id,
+ "source_record_id": source_ids[0],
+ "speaker": role,
+ "timestamp": timestamp,
+ "resolution": resolution_state,
+ "evidence_quote": evidence_quote,
+ }
+ )
+ interaction_metadata.update(
+ {
+ "interaction_status": next_status,
+ "resolution_history": history,
+ "last_resolution": resolution_state,
+ "last_resolution_message_id": message_id,
+ "last_resolution_source_record_id": source_ids[0],
+ "last_resolution_timestamp": timestamp,
+ }
+ )
+ if next_status == "resolved":
+ interaction_metadata["resolved_at"] = timestamp
+ interaction_metadata["resolved_by_message_id"] = message_id
+ interaction_metadata["resolved_by_source_record_id"] = source_ids[0]
+ interaction_record.metadata = interaction_metadata
+ edge_type = {
+ "resolved": "answered_by",
+ "partial": "partially_answered_by",
+ "unresolved": "responded_without_resolution",
+ }[resolution_state]
+ adapter.graph._upsert_memory_edge(
+ edge_class(
+ edge_id=f"{interaction_id}->{source_ids[0]}:{edge_type}",
+ source_memory_id=interaction_id,
+ target_memory_id=source_ids[0],
+ edge_type=edge_type,
+ score=1.0 if resolution_state == "resolved" else (0.72 if resolution_state == "partial" else 0.35),
+ model_score=0.0,
+ evidence_turn=turn_index,
+ evidence=evidence_quote,
+ metadata={
+ "edge_source": "product_writer_resolution",
+ "resolution": resolution_state,
+ "previous_status": previous_status,
+ "next_status": next_status,
+ "message_id": message_id,
+ "speaker": role,
+ },
+ )
+ )
+ resolution_rows.append(
+ {
+ "interaction_id": interaction_id,
+ "resolution": resolution_state,
+ "previous_status": previous_status,
+ "next_status": next_status,
+ "edge_type": edge_type,
+ }
+ )
+ adapter.graph.record_turn(
+ turn_kind="memory_write",
+ text=content,
+ turn_index=turn_index,
+ record_ids=stored_ids,
+ speaker=role,
+ assistant_text="",
+ metadata={
+ "source": "tmcra_v3_product_runtime",
+ "writer_schema_version": WRITE_SCHEMA_VERSION,
+ "prompt_version": PROMPT_VERSION,
+ "session_id": session_id,
+ "session_index": int(session_index),
+ "message_id": message_id,
+ "message_index": int(message_index),
+ "historical_date": date,
+ "timestamp": timestamp,
+ "source_record_id": source_ids[0],
+ },
+ )
+ adapter._persist_graph()
+ return {
+ "turn_index": turn_index,
+ "source_record_id": source_ids[0],
+ "stored_ids": stored_ids,
+ "resolution_count": len(resolution_rows),
+ "provenance_edge_count": provenance_count,
+ "resolutions": resolution_rows,
+ **counts,
+ }
+
+
+def database_counts(path: Path, scope_id: str) -> dict[str, int]:
+ with sqlite3.connect(path) as connection:
+ fast_records = sum(
+ 1
+ for (metadata_json,) in connection.execute(
+ "SELECT metadata_json FROM records WHERE scope_id=?",
+ (scope_id,),
+ )
+ if clean_text(json.loads(metadata_json or "{}").get("memory_layer"))
+ == "fast"
+ )
+ non_slow_edges = sum(
+ 1
+ for (metadata_json,) in connection.execute(
+ "SELECT metadata_json FROM memory_edges WHERE scope_id=?",
+ (scope_id,),
+ )
+ if clean_text(json.loads(metadata_json or "{}").get("edge_source"))
+ != "slow_graph_control_plane"
+ )
+ return {
+ "records": fast_records,
+ "memory_edges": non_slow_edges,
+ "audit_turn_log": int(
+ connection.execute(
+ "SELECT COUNT(*) FROM audit_turn_log WHERE scope_id=?",
+ (scope_id,),
+ ).fetchone()[0]
+ ),
+ "product_source_journal": int(
+ connection.execute(
+ "SELECT COUNT(*) FROM product_source_journal WHERE scope_id=?",
+ (scope_id,),
+ ).fetchone()[0]
+ ),
+ }
+
+
+def reconstruct_persisted_message(
+ graph: Any,
+ *,
+ message_id: str,
+ extraction: Mapping[str, Any] | None,
+) -> dict[str, Any] | None:
+ records = [
+ record
+ for record in graph.records_by_id.values()
+ if clean_text(dict(record.metadata or {}).get("message_id")) == message_id
+ ]
+ source_records = [
+ record
+ for record in records
+ if clean_text(dict(record.metadata or {}).get("content_variant")) == "source_message"
+ ]
+ if not records and not source_records:
+ return None
+ if len(source_records) != 1:
+ raise ProductWriterError(f"{message_id}: staged graph commit has {len(source_records)} source records")
+ source_record = source_records[0]
+ variants = [clean_text(dict(record.metadata or {}).get("content_variant")) for record in records]
+ provenance_count = sum(
+ clean_text(dict(edge.metadata or {}).get("edge_source")) == "product_writer_provenance"
+ and edge.target_memory_id == source_record.memory_id
+ for edge in graph.memory_edges.values()
+ )
+ return {
+ "turn_index": int(source_record.turn_index),
+ "source_record_id": source_record.memory_id,
+ "stored_ids": [record.memory_id for record in records],
+ "resolution_count": len(list((extraction or {}).get("resolutions") or [])),
+ "provenance_edge_count": provenance_count,
+ "resolutions": [],
+ "source": variants.count("source_message"),
+ "semantic": variants.count("product_semantic_memory"),
+ "facet": variants.count("event_facet_write"),
+ "interaction": variants.count("product_interaction"),
+ }
+
+
+def _ensure_source_journal_schema(connection: sqlite3.Connection) -> None:
+ connection.execute(
+ """
+ CREATE TABLE IF NOT EXISTS product_source_journal (
+ scope_id TEXT NOT NULL,
+ message_id TEXT NOT NULL,
+ session_id TEXT NOT NULL,
+ session_index INTEGER NOT NULL,
+ message_index INTEGER NOT NULL,
+ timestamp TEXT NOT NULL,
+ role TEXT NOT NULL,
+ content TEXT NOT NULL,
+ content_sha256 TEXT NOT NULL,
+ enrichment_status TEXT NOT NULL,
+ source_record_id TEXT NOT NULL,
+ error TEXT NOT NULL,
+ extraction_json TEXT NOT NULL DEFAULT '',
+ call_metadata_json TEXT NOT NULL DEFAULT '',
+ persisted_json TEXT NOT NULL DEFAULT '',
+ claim_owner TEXT NOT NULL DEFAULT '',
+ claim_token TEXT NOT NULL DEFAULT '',
+ claim_expires_at TEXT NOT NULL DEFAULT '',
+ api_call_started_at TEXT NOT NULL DEFAULT '',
+ created_at TEXT NOT NULL,
+ updated_at TEXT NOT NULL,
+ PRIMARY KEY (scope_id, message_id)
+ )
+ """
+ )
+ columns = {str(row[1]) for row in connection.execute("PRAGMA table_info(product_source_journal)")}
+ required_columns = {
+ "extraction_json": "TEXT NOT NULL DEFAULT ''",
+ "call_metadata_json": "TEXT NOT NULL DEFAULT ''",
+ "persisted_json": "TEXT NOT NULL DEFAULT ''",
+ "claim_owner": "TEXT NOT NULL DEFAULT ''",
+ "claim_token": "TEXT NOT NULL DEFAULT ''",
+ "claim_expires_at": "TEXT NOT NULL DEFAULT ''",
+ "api_call_started_at": "TEXT NOT NULL DEFAULT ''",
+ }
+ for column, definition in required_columns.items():
+ if column not in columns:
+ connection.execute(f"ALTER TABLE product_source_journal ADD COLUMN {column} {definition}")
+
+
+def _source_journal_claim_expired(claim_expires_at: str, *, now: datetime) -> bool:
+ if not claim_expires_at:
+ return True
+ try:
+ expires_at = datetime.fromisoformat(claim_expires_at)
+ except ValueError as exc:
+ raise ProductWriterError("source journal has malformed claim lease timestamp") from exc
+ if expires_at.tzinfo is None:
+ raise ProductWriterError("source journal claim lease timestamp must include a timezone")
+ return expires_at <= now
+
+
+def claim_source_journal(
+ path: Path,
+ *,
+ scope_id: str,
+ message_id: str,
+ claim_owner: str,
+ lease_seconds: int = SOURCE_JOURNAL_CLAIM_LEASE_SECONDS,
+) -> str | None:
+ owner = clean_text(claim_owner)
+ if not owner:
+ raise ValueError("source journal claim owner is required")
+ if int(lease_seconds) <= 0:
+ raise ValueError("source journal claim lease must be positive")
+ now = datetime.now(timezone.utc)
+ expires_at = (now + timedelta(seconds=int(lease_seconds))).isoformat(timespec="seconds")
+ token = uuid.uuid4().hex
+ with closing(sqlite3.connect(path, timeout=30, isolation_level=None)) as connection:
+ try:
+ connection.execute("BEGIN IMMEDIATE")
+ _ensure_source_journal_schema(connection)
+ row = connection.execute(
+ """
+ SELECT enrichment_status,claim_token,claim_expires_at,
+ extraction_json,api_call_started_at
+ FROM product_source_journal WHERE scope_id=? AND message_id=?
+ """,
+ (scope_id, message_id),
+ ).fetchone()
+ if row is None:
+ raise ProductWriterError(f"{message_id}: source journal row was not found for claim")
+ if str(row[0]) != "pending":
+ connection.commit()
+ return None
+ existing_token = str(row[1] or "")
+ if existing_token and not _source_journal_claim_expired(str(row[2] or ""), now=now):
+ connection.commit()
+ return None
+ if (
+ existing_token
+ and _source_journal_claim_expired(str(row[2] or ""), now=now)
+ and str(row[4] or "")
+ and not str(row[3] or "")
+ ):
+ connection.execute(
+ "UPDATE product_source_journal SET enrichment_status='failed',"
+ "error='expired Writer claim has an uncertain external-call outcome; explicit resume required',"
+ "claim_owner='',claim_token='',claim_expires_at='',updated_at=? "
+ "WHERE scope_id=? AND message_id=? AND enrichment_status='pending' "
+ "AND claim_token=?",
+ (
+ now.isoformat(timespec="seconds"),
+ scope_id,
+ message_id,
+ existing_token,
+ ),
+ )
+ connection.commit()
+ raise ProductWriterError(
+ f"{message_id}: expired Writer claim has an uncertain external-call outcome; "
+ "explicit resume is required before replay"
+ )
+ cursor = connection.execute(
+ """
+ UPDATE product_source_journal
+ SET claim_owner=?, claim_token=?, claim_expires_at=?, updated_at=?
+ WHERE scope_id=? AND message_id=? AND enrichment_status='pending'
+ """,
+ (owner, token, expires_at, now.isoformat(timespec="seconds"), scope_id, message_id),
+ )
+ if cursor.rowcount != 1:
+ raise ProductWriterError(f"{message_id}: source journal claim was lost")
+ connection.commit()
+ except Exception:
+ connection.rollback()
+ raise
+ return token
+
+
+def renew_source_journal_claim(
+ path: Path,
+ *,
+ scope_id: str,
+ message_id: str,
+ claim_owner: str,
+ claim_token: str,
+ lease_seconds: int,
+) -> None:
+ now = datetime.now(timezone.utc)
+ expires_at = (now + timedelta(seconds=int(lease_seconds))).isoformat(timespec="seconds")
+ now_text = now.isoformat(timespec="seconds")
+ with closing(sqlite3.connect(path, timeout=30, isolation_level=None)) as connection:
+ try:
+ connection.execute("BEGIN IMMEDIATE")
+ renewed = connection.execute(
+ "UPDATE product_source_journal SET claim_expires_at=?,updated_at=? "
+ "WHERE scope_id=? AND message_id=? AND enrichment_status='pending' "
+ "AND claim_owner=? AND claim_token=? AND claim_expires_at>=?",
+ (
+ expires_at,
+ now_text,
+ scope_id,
+ message_id,
+ claim_owner,
+ claim_token,
+ now_text,
+ ),
+ )
+ if renewed.rowcount != 1:
+ raise ProductWriterError(f"{message_id}: Writer claim could not be renewed")
+ connection.commit()
+ except Exception:
+ connection.rollback()
+ raise
+
+
+def mark_source_api_call_started(
+ path: Path,
+ *,
+ scope_id: str,
+ message_id: str,
+ claim_owner: str,
+ claim_token: str,
+) -> None:
+ updated_at = datetime.now(timezone.utc).isoformat(timespec="seconds")
+ with closing(sqlite3.connect(path)) as connection, connection:
+ marked = connection.execute(
+ f"UPDATE product_source_journal SET api_call_started_at=?,updated_at=? "
+ f"WHERE {_claimed_journal_where()} AND extraction_json=''",
+ (
+ updated_at,
+ updated_at,
+ scope_id,
+ message_id,
+ claim_owner,
+ claim_token,
+ updated_at,
+ ),
+ )
+ if marked.rowcount != 1:
+ raise ProductWriterError(f"{message_id}: Writer API-call marker could not be staged")
+
+
+def call_with_source_claim_heartbeat(
+ call: Callable[[], T],
+ *,
+ path: Path,
+ scope_id: str,
+ message_id: str,
+ claim_owner: str,
+ claim_token: str,
+ lease_seconds: int,
+) -> T:
+ stop = threading.Event()
+ errors: list[Exception] = []
+ interval = max(0.1, min(30.0, int(lease_seconds) / 3.0))
+
+ def heartbeat() -> None:
+ while not stop.wait(interval):
+ try:
+ renew_source_journal_claim(
+ path,
+ scope_id=scope_id,
+ message_id=message_id,
+ claim_owner=claim_owner,
+ claim_token=claim_token,
+ lease_seconds=lease_seconds,
+ )
+ except Exception as exc:
+ errors.append(exc)
+ stop.set()
+
+ thread = threading.Thread(
+ target=heartbeat,
+ name=f"writer-lease-{message_id}",
+ daemon=True,
+ )
+ thread.start()
+ try:
+ result = call()
+ finally:
+ stop.set()
+ thread.join(timeout=max(1.0, interval + 1.0))
+ if thread.is_alive():
+ raise ProductWriterError(f"{message_id}: Writer claim heartbeat did not stop")
+ if errors:
+ raise ProductWriterError(f"{message_id}: Writer claim heartbeat failed: {errors[0]}")
+ renew_source_journal_claim(
+ path,
+ scope_id=scope_id,
+ message_id=message_id,
+ claim_owner=claim_owner,
+ claim_token=claim_token,
+ lease_seconds=lease_seconds,
+ )
+ return result
+
+
+def _claimed_journal_where() -> str:
+ return (
+ "scope_id=? AND message_id=? AND enrichment_status='pending' "
+ "AND claim_owner=? AND claim_token=? AND claim_expires_at>?"
+ )
+
+
+def journal_source_message(
+ path: Path,
+ *,
+ scope_id: str,
+ session_id: str,
+ session_index: int,
+ message_id: str,
+ message_index: int,
+ timestamp: str,
+ role: str,
+ content: str,
+) -> None:
+ created_at = datetime.now(timezone.utc).isoformat(timespec="seconds")
+ with closing(sqlite3.connect(path)) as connection, connection:
+ _ensure_source_journal_schema(connection)
+ existing = connection.execute(
+ "SELECT content_sha256, enrichment_status FROM product_source_journal WHERE scope_id=? AND message_id=?",
+ (scope_id, message_id),
+ ).fetchone()
+ if existing is not None:
+ if str(existing[0]) != sha256_text(content):
+ raise ProductWriterError(f"{message_id}: source journal content changed")
+ raise ProductWriterError(f"{message_id}: source journal row already exists with status={existing[1]}")
+ connection.execute(
+ """
+ INSERT INTO product_source_journal (
+ scope_id, message_id, session_id, session_index, message_index, timestamp, role, content,
+ content_sha256, enrichment_status, source_record_id, error, created_at, updated_at
+ ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, 'pending', '', '', ?, ?)
+ """,
+ (
+ scope_id,
+ message_id,
+ session_id,
+ int(session_index),
+ int(message_index),
+ timestamp,
+ role,
+ content,
+ sha256_text(content),
+ created_at,
+ created_at,
+ ),
+ )
+
+
+def stage_source_enrichment(
+ path: Path,
+ *,
+ scope_id: str,
+ message_id: str,
+ claim_owner: str,
+ claim_token: str,
+ extraction: Mapping[str, Any] | None,
+ call_metadata: Mapping[str, Any],
+) -> None:
+ updated_at = datetime.now(timezone.utc).isoformat(timespec="seconds")
+ with closing(sqlite3.connect(path)) as connection, connection:
+ cursor = connection.execute(
+ f"""
+ UPDATE product_source_journal
+ SET extraction_json=?, call_metadata_json=?, updated_at=?
+ WHERE {_claimed_journal_where()}
+ """,
+ (
+ json.dumps(extraction, ensure_ascii=False, sort_keys=True),
+ json.dumps(dict(call_metadata), ensure_ascii=False, sort_keys=True),
+ updated_at,
+ scope_id,
+ message_id,
+ claim_owner,
+ claim_token,
+ updated_at,
+ ),
+ )
+ if cursor.rowcount != 1:
+ raise ProductWriterError(f"{message_id}: source journal could not stage Writer output")
+
+
+def stage_source_persisted(
+ path: Path,
+ *,
+ scope_id: str,
+ message_id: str,
+ claim_owner: str,
+ claim_token: str,
+ persisted: Mapping[str, Any],
+) -> None:
+ updated_at = datetime.now(timezone.utc).isoformat(timespec="seconds")
+ with closing(sqlite3.connect(path)) as connection, connection:
+ cursor = connection.execute(
+ f"""
+ UPDATE product_source_journal
+ SET persisted_json=?, updated_at=?
+ WHERE {_claimed_journal_where()}
+ """,
+ (
+ json.dumps(dict(persisted), ensure_ascii=False, sort_keys=True),
+ updated_at,
+ scope_id,
+ message_id,
+ claim_owner,
+ claim_token,
+ updated_at,
+ ),
+ )
+ if cursor.rowcount != 1:
+ raise ProductWriterError(f"{message_id}: source journal could not stage graph commit")
+
+
+def finish_source_journal(
+ path: Path,
+ *,
+ scope_id: str,
+ message_id: str,
+ claim_owner: str,
+ claim_token: str,
+ status: str,
+ source_record_id: str = "",
+ error: str = "",
+) -> None:
+ if status not in {"enriched", "enriched_with_warnings", "failed"}:
+ raise ValueError(f"unsupported source journal status: {status}")
+ normalized_error = clean_text(error)
+ if status == "enriched_with_warnings":
+ try:
+ warning_rows = json.loads(normalized_error)
+ except json.JSONDecodeError as exc:
+ raise ProductWriterError(
+ f"{message_id}: warning journal payload must be valid JSON"
+ ) from exc
+ if not isinstance(warning_rows, list) or not warning_rows:
+ raise ProductWriterError(
+ f"{message_id}: warning journal payload must be a nonempty list"
+ )
+ stored_error = json.dumps(
+ warning_rows, ensure_ascii=False, sort_keys=True
+ )
+ else:
+ stored_error = normalized_error[:1000]
+ updated_at = datetime.now(timezone.utc).isoformat(timespec="seconds")
+ with closing(sqlite3.connect(path)) as connection, connection:
+ cursor = connection.execute(
+ f"""
+ UPDATE product_source_journal
+ SET enrichment_status=?, source_record_id=?, error=?, claim_owner='', claim_token='',
+ claim_expires_at='', updated_at=?
+ WHERE {_claimed_journal_where()}
+ """,
+ (
+ status,
+ source_record_id,
+ stored_error,
+ updated_at,
+ scope_id,
+ message_id,
+ claim_owner,
+ claim_token,
+ updated_at,
+ ),
+ )
+ if cursor.rowcount != 1:
+ raise ProductWriterError(f"{message_id}: source journal pending row was not found")
+
+
+def repair_warning_journal_payloads(path: Path, *, scope_id: str) -> int:
+ repaired = 0
+ updated_at = datetime.now(timezone.utc).isoformat(timespec="seconds")
+ with closing(sqlite3.connect(path)) as connection, connection:
+ connection.row_factory = sqlite3.Row
+ rows = connection.execute(
+ "SELECT message_id,error,extraction_json FROM product_source_journal "
+ "WHERE scope_id=? AND enrichment_status='enriched_with_warnings'",
+ (scope_id,),
+ ).fetchall()
+ for row in rows:
+ extraction = json.loads(row["extraction_json"] or "{}")
+ warnings = extraction.get("validation_warnings")
+ if not isinstance(warnings, list) or not warnings:
+ raise ProductWriterError(
+ f"{row['message_id']}: staged extraction lacks warning payload"
+ )
+ expected = json.dumps(warnings, ensure_ascii=False, sort_keys=True)
+ if row["error"] == expected:
+ continue
+ connection.execute(
+ "UPDATE product_source_journal SET error=?,updated_at=? "
+ "WHERE scope_id=? AND message_id=?",
+ (expected, updated_at, scope_id, row["message_id"]),
+ )
+ repaired += 1
+ return repaired
+
+
+def source_journal_row(path: Path, *, scope_id: str, message_id: str) -> dict[str, str] | None:
+ with closing(sqlite3.connect(path)) as connection, connection:
+ connection.row_factory = sqlite3.Row
+ table_exists = connection.execute(
+ "SELECT 1 FROM sqlite_master WHERE type='table' AND name='product_source_journal'"
+ ).fetchone()
+ if table_exists is None:
+ return None
+ _ensure_source_journal_schema(connection)
+ row = connection.execute(
+ """
+ SELECT content_sha256, enrichment_status, source_record_id, error,
+ extraction_json, call_metadata_json, persisted_json,
+ claim_owner, claim_token, claim_expires_at, api_call_started_at
+ FROM product_source_journal WHERE scope_id=? AND message_id=?
+ """,
+ (scope_id, message_id),
+ ).fetchone()
+ return {key: str(row[key] or "") for key in row.keys()} if row is not None else None
+
+
+def decode_journal_json(raw_value: str, *, message_id: str, field: str) -> Any:
+ try:
+ return json.loads(raw_value)
+ except (TypeError, json.JSONDecodeError) as exc:
+ raise ProductWriterError(f"{message_id}: staged journal field {field} is malformed") from exc
+
+
+def call_metadata_from_log(row: Mapping[str, Any]) -> dict[str, Any]:
+ metadata = dict(row)
+ for field in (
+ "question_id",
+ "scope_id",
+ "session_index",
+ "message_index",
+ "message_id",
+ "message_role",
+ "content_sha256",
+ "assertion_count",
+ "interaction_count",
+ "resolution_count",
+ "validated_output",
+ "persisted",
+ ):
+ metadata.pop(field, None)
+ return metadata
+
+
+def reopen_failed_source_journal(path: Path, *, scope_id: str, message_id: str) -> None:
+ updated_at = datetime.now(timezone.utc).isoformat(timespec="seconds")
+ with closing(sqlite3.connect(path)) as connection, connection:
+ cursor = connection.execute(
+ """
+ UPDATE product_source_journal
+ SET enrichment_status='pending', source_record_id='', error='', claim_owner='', claim_token='',
+ claim_expires_at='', api_call_started_at='', updated_at=?
+ WHERE scope_id=? AND message_id=? AND enrichment_status='failed'
+ """,
+ (updated_at, scope_id, message_id),
+ )
+ if cursor.rowcount != 1:
+ raise ProductWriterError(f"{message_id}: failed source journal row was not found")
+
+
+def writer_slot_candidates(graph: Any, current_message: str, *, limit: int = 64) -> list[dict[str, Any]]:
+ by_key: dict[str, Any] = {}
+ for record in graph.records_by_id.values():
+ metadata = dict(record.metadata or {})
+ if clean_text(metadata.get("content_variant")) != "product_semantic_memory":
+ continue
+ if clean_text(metadata.get("write_operation")) != "replace":
+ continue
+ if clean_text(record.state).lower() not in {"active", "parallel_active"}:
+ continue
+ canonical_key = normalize_canonical_key(clean_text(metadata.get("canonical_slot_key")).removeprefix("memory."))
+ if not canonical_key:
+ continue
+ existing = by_key.get(canonical_key)
+ if existing is None or int(record.turn_index) > int(existing.turn_index):
+ by_key[canonical_key] = record
+ if not by_key:
+ return []
+
+ def tokens(value: Any) -> set[str]:
+ return set(re.findall(r"[a-z0-9]+|[\u4e00-\u9fff]", str(value or "").lower()))
+
+ query_tokens = tokens(current_message)
+ scored: list[tuple[float, int, str, Any]] = []
+ for canonical_key, record in by_key.items():
+ metadata = dict(record.metadata or {})
+ candidate_tokens = tokens(
+ " ".join(
+ [
+ canonical_key.replace(".", " "),
+ clean_text(record.relation),
+ clean_text(metadata.get("object")),
+ clean_text(record.value),
+ ]
+ )
+ )
+ overlap = len(query_tokens & candidate_tokens) / max(1, len(query_tokens))
+ scored.append((overlap, int(record.turn_index), canonical_key, record))
+ relevant = sorted((row for row in scored if row[0] > 0), key=lambda row: (row[0], row[1]), reverse=True)[:48]
+ recent = sorted(scored, key=lambda row: row[1], reverse=True)[:16]
+ selected: list[tuple[float, int, str, Any]] = []
+ seen: set[str] = set()
+ for row in [*relevant, *recent]:
+ if row[2] in seen:
+ continue
+ seen.add(row[2])
+ selected.append(row)
+ if len(selected) >= max(1, int(limit)):
+ break
+ return [
+ {
+ "canonical_key": canonical_key,
+ "entity_key": clean_text(dict(record.metadata or {}).get("entity_key")),
+ "attribute_key": clean_text(dict(record.metadata or {}).get("attribute_key")),
+ "memory_type": clean_text(dict(record.metadata or {}).get("memory_type")),
+ "memory_family": clean_text(dict(record.metadata or {}).get("memory_family")),
+ "relation": clean_text(record.relation),
+ "current_evidence_quote": str(record.value),
+ }
+ for _, _, canonical_key, record in selected
+ ]
+
+
+def pending_interaction_candidates(
+ graph: Any,
+ *,
+ current_role: str,
+ session_id: str,
+ session_index: int,
+ current_message_index: int,
+ limit: int | None = None,
+) -> list[dict[str, Any]]:
+ candidates: list[Any] = []
+ for record in graph.records_by_id.values():
+ metadata = dict(record.metadata or {})
+ if clean_text(metadata.get("content_variant")) != "product_interaction":
+ continue
+ if clean_text(metadata.get("session_id")) != session_id:
+ continue
+ if int(metadata.get("session_index", -1)) != int(session_index):
+ continue
+ if int(metadata.get("message_index", -1)) >= int(current_message_index):
+ continue
+ if clean_text(metadata.get("interaction_status")) not in {"open", "partial"}:
+ continue
+ if clean_text(metadata.get("interaction_speaker")) == current_role:
+ continue
+ candidates.append(record)
+ candidates.sort(key=lambda record: int(record.turn_index), reverse=True)
+ rows = [
+ {
+ "interaction_id": clean_text(dict(record.metadata or {}).get("interaction_id")) or record.memory_id,
+ "interaction_type": clean_text(dict(record.metadata or {}).get("interaction_type")),
+ "status": clean_text(dict(record.metadata or {}).get("interaction_status")),
+ "speaker": clean_text(dict(record.metadata or {}).get("interaction_speaker")),
+ "intent": clean_text(record.relation),
+ "evidence_quote": str(record.value),
+ "about": list(dict(record.metadata or {}).get("about") or []),
+ }
+ for record in candidates
+ ]
+ return rows if limit is None else rows[: max(1, int(limit))]
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description="TMCRA V3 product message writer with strict DeepSeek extraction")
+ parser.add_argument("--input", required=True)
+ parser.add_argument("--out-dir", required=True)
+ parser.add_argument("--repo", required=True)
+ parser.add_argument("--resume", action="store_true")
+ args = parser.parse_args()
+ input_path = Path(args.input).resolve()
+ out_dir = Path(args.out_dir).resolve()
+ out_dir.mkdir(parents=True, exist_ok=True)
+ db_path = out_dir / "native_memory.sqlite3"
+ if db_path.exists() and not args.resume:
+ raise FileExistsError(f"product writer requires a fresh database: {db_path}")
+ if args.resume and not db_path.exists():
+ raise FileNotFoundError(f"product writer resume database does not exist: {db_path}")
+ rows = json.loads(input_path.read_text(encoding="utf-8"))
+ if not isinstance(rows, list) or not rows:
+ raise ProductWriterError("writer input must be a non-empty JSON array")
+ for row in rows:
+ forbidden = sorted(FORBIDDEN_WRITER_FIELDS & set(row))
+ if forbidden:
+ raise ProductWriterError(f"writer input contains query/evaluation fields: {forbidden}")
+
+ base_url = clean_text(os.getenv("TMCRA_WRITER_BASE_URL"))
+ model = clean_text(os.getenv("TMCRA_WRITER_MODEL"))
+ reviewer_model = clean_text(os.getenv("TMCRA_WRITER_REVIEW_MODEL"))
+ api_keys = [
+ clean_text(value)
+ for value in os.getenv("TMCRA_WRITER_API_KEY_POOL", "").split(",")
+ if clean_text(value)
+ ]
+ if not base_url or not model or not reviewer_model or not api_keys:
+ raise ProductWriterError(
+ "explicit TMCRA writer base URL, Flash model, Pro model, and API key pool are required"
+ )
+ writer = DeepSeekProductWriter(
+ base_url=base_url,
+ model=model,
+ reviewer_model=reviewer_model,
+ api_keys=api_keys,
+ timeout=float(os.getenv("TMCRA_WRITER_TIMEOUT_SECONDS", "180")),
+ max_tokens=int(os.getenv("TMCRA_WRITER_MAX_TOKENS", "8192")),
+ )
+ try:
+ journal_claim_lease_seconds = int(
+ os.getenv("TMCRA_WRITER_CLAIM_LEASE_SECONDS", str(SOURCE_JOURNAL_CLAIM_LEASE_SECONDS))
+ )
+ except ValueError as exc:
+ raise ProductWriterError("TMCRA_WRITER_CLAIM_LEASE_SECONDS must be an integer") from exc
+ if journal_claim_lease_seconds <= 0:
+ raise ProductWriterError("TMCRA_WRITER_CLAIM_LEASE_SECONDS must be positive")
+ journal_claim_owner = f"writer:{os.getpid()}:{uuid.uuid4().hex}"
+
+ repo = Path(args.repo).resolve()
+ if str(repo) not in sys.path:
+ sys.path.insert(0, str(repo))
+ os.environ.update(
+ {
+ "TMCRA_PROFILE_CONSOLIDATOR_ENABLED": "0",
+ "TMCRA_LEGACY_PROFILE_LAYER_ENABLED": "0",
+ "TMCRA_WRITE_EMBEDDER_INDEX_MODE": "off",
+ "TMCRA_EMBEDDER_INDEX_RECALL_MODE": "off",
+ "TMCRA_EMBEDDER_PRE_RECALL_MODE": "off",
+ "TMCRA_EMBEDDER_FUSION_MODE": "off",
+ "TMCRA_MEMORY_ROUTER_MODE": "off",
+ "TMCRA_INJECTION_PLANNER_MODE": "off",
+ "TMCRA_TEMPORAL_LAYER_MODE": "off",
+ "TMCRA_TEMPORAL_ROUTER_MODE": "off",
+ "TMCRA_DEEPSEEK_GRAPH_MODEL_MODE": "off",
+ "TMCRA_TOPIC_BUCKET_MODE": "off",
+ "TMCRA_MULTI_UNIT_CHAIN_SLOT_MODE": "off",
+ "TMCRA_UNIT_COVERAGE_PACK_MODE": "off",
+ }
+ )
+ from experiments.replacement.adapters.memory_adapters import GraphSessionMemoryAdapter
+ from experiments.replacement.memory_graph import SessionMemoryEdgeV2, SessionMemoryRecordV2
+
+ message_total = sum(len(session or []) for row in rows for session in list(row.get("haystack_sessions") or []))
+ started = time.time()
+ call_log_path = out_dir / "product_writer_calls.jsonl"
+ message_log_path = out_dir / "product_write_messages.jsonl"
+ failure_log_path = out_dir / "product_writer_failures.jsonl"
+ recovery_log_path = out_dir / "product_writer_resume_repairs.jsonl"
+ journal_recovery_log_path = out_dir / "product_writer_journal_repairs.jsonl"
+ jsonl_tail_repairs = 0
+ journal_warning_repairs = 0
+ existing_calls: dict[tuple[str, str], dict[str, Any]] = {}
+ existing_messages: dict[tuple[str, str], dict[str, Any]] = {}
+ recoverable_extractions: dict[tuple[str, str], dict[str, Any]] = {}
+ if args.resume:
+ call_rows, call_repaired = load_jsonl_for_resume(call_log_path)
+ message_rows, message_repaired = load_jsonl_for_resume(message_log_path)
+ failure_rows_for_resume, failure_repaired = load_jsonl_for_resume(failure_log_path)
+ repaired_paths = [
+ path.name
+ for path, repaired in (
+ (call_log_path, call_repaired),
+ (message_log_path, message_repaired),
+ (failure_log_path, failure_repaired),
+ )
+ if repaired
+ ]
+ jsonl_tail_repairs = len(repaired_paths)
+ if repaired_paths:
+ with recovery_log_path.open("a", encoding="utf-8") as handle:
+ for repaired_path in repaired_paths:
+ handle.write(
+ json.dumps(
+ {
+ "timestamp": datetime.now(timezone.utc).isoformat(timespec="seconds"),
+ "repair": "truncated_incomplete_jsonl_tail",
+ "path": repaired_path,
+ },
+ sort_keys=True,
+ )
+ + "\n"
+ )
+ for value in call_rows:
+ existing_calls[(clean_text(value.get("question_id")), clean_text(value.get("message_id")))] = value
+ for value in message_rows:
+ existing_messages[(clean_text(value.get("question_id")), clean_text(value.get("message_id")))] = value
+ for value in failure_rows_for_resume:
+ if value.get("phase") != "graph_persist" or not isinstance(value.get("validated_output"), Mapping):
+ continue
+ recoverable_extractions[
+ (clean_text(value.get("question_id")), clean_text(value.get("message_id")))
+ ] = value
+ reports: list[dict[str, Any]] = []
+ for row in rows:
+ qid = clean_text(row.get("question_id"))
+ sessions = list(row.get("haystack_sessions") or [])
+ session_ids = [clean_text(value) for value in list(row.get("haystack_session_ids") or [])]
+ dates = [clean_text(value) for value in list(row.get("haystack_dates") or [])]
+ if not qid or not sessions or len(sessions) != len(session_ids) or len(sessions) != len(dates):
+ raise ProductWriterError(f"{qid}: malformed history-only writer input")
+ scope_id = f"tmcra_v3:{qid}"
+ adapter = GraphSessionMemoryAdapter(
+ auto_extract=False,
+ storage_backend="sqlite",
+ storage_path=str(db_path),
+ scope_id=scope_id,
+ audit_retention=max(1024, message_total + 128),
+ retrieval_mode="hybrid_node_scored",
+ node_model_path=clean_text(os.getenv("TMCRA_NODE_MODEL_PATH")),
+ path_model_path=clean_text(os.getenv("TMCRA_PATH_MODEL_PATH")),
+ node_model_device=clean_text(os.getenv("TMCRA_NODE_MODEL_DEVICE")) or "cpu",
+ )
+ if args.resume:
+ repaired_warning_rows = repair_warning_journal_payloads(
+ db_path, scope_id=scope_id
+ )
+ journal_warning_repairs += repaired_warning_rows
+ if repaired_warning_rows:
+ with journal_recovery_log_path.open("a", encoding="utf-8") as handle:
+ handle.write(
+ json.dumps(
+ {
+ "timestamp": datetime.now(timezone.utc).isoformat(
+ timespec="seconds"
+ ),
+ "repair": "restore_full_warning_json_from_staged_extraction",
+ "scope_id": scope_id,
+ "row_count": repaired_warning_rows,
+ },
+ sort_keys=True,
+ )
+ + "\n"
+ )
+ totals = {
+ "messages": 0,
+ "user_messages": 0,
+ "assistant_messages": 0,
+ "system_messages": 0,
+ "tool_messages": 0,
+ "writer_calls": 0,
+ "flash_writer_calls": 0,
+ "pro_writer_calls": 0,
+ "api_calls": 0,
+ "empty_semantic_calls": 0,
+ "source_records": 0,
+ "semantic_records": 0,
+ "facet_records": 0,
+ "interaction_records": 0,
+ "resolution_edges": 0,
+ "provenance_edges": 0,
+ "prompt_tokens": 0,
+ "completion_tokens": 0,
+ "resumed_messages": 0,
+ "messages_with_warnings": 0,
+ "validation_warning_count": 0,
+ "quarantined_item_count": 0,
+ "capacity_segmented_messages": 0,
+ "capacity_segments": 0,
+ "capacity_duplicate_count": 0,
+ }
+ for session_index, session in enumerate(sessions):
+ previous_message: dict[str, Any] | None = None
+ for message_index, raw_message in enumerate(list(session or [])):
+ if not isinstance(raw_message, Mapping):
+ raise ProductWriterError(f"{qid}/s{session_index}/m{message_index}: message must be an object")
+ role = clean_text(raw_message.get("role")).lower()
+ content = str(raw_message.get("content") or "")
+ if role not in {"user", "assistant", "system", "tool"} or not content:
+ raise ProductWriterError(f"{qid}/s{session_index}/m{message_index}: invalid role or empty content")
+ message_id = f"s{session_index:03d}_m{message_index:03d}"
+ timestamp = historical_timestamp(dates[session_index], message_index)
+ current_message = {"role": role, "timestamp": timestamp, "content": content}
+ existing_message = existing_messages.get((qid, message_id))
+ journal = source_journal_row(db_path, scope_id=scope_id, message_id=message_id)
+ if existing_message is not None:
+ if (
+ not args.resume
+ or journal is None
+ or journal["enrichment_status"] not in {"enriched", "enriched_with_warnings"}
+ ):
+ raise ProductWriterError(f"{message_id}: completed log and source journal disagree")
+ if clean_text(existing_message.get("role")) != role:
+ raise ProductWriterError(f"{message_id}: resumed role changed")
+ if clean_text(existing_message.get("content_sha256")) != sha256_text(content):
+ raise ProductWriterError(f"{message_id}: resumed content changed")
+ totals["messages"] += 1
+ totals["resumed_messages"] += 1
+ totals[f"{role}_messages"] = int(totals.get(f"{role}_messages", 0)) + 1
+ totals["source_records"] += int(existing_message.get("source", 0) or 0)
+ totals["semantic_records"] += int(existing_message.get("semantic", 0) or 0)
+ totals["facet_records"] += int(existing_message.get("facet", 0) or 0)
+ totals["interaction_records"] += int(existing_message.get("interaction", 0) or 0)
+ totals["resolution_edges"] += int(existing_message.get("resolution_count", 0) or 0)
+ totals["provenance_edges"] += int(existing_message.get("provenance_edge_count", 0) or 0)
+ if bool(existing_message.get("writer_called")):
+ existing_call = existing_calls.get((qid, message_id))
+ if existing_call is None:
+ raise ProductWriterError(f"{message_id}: resumed writer call log is missing")
+ totals["writer_calls"] += 1
+ existing_api_calls = max(1, int(existing_call.get("api_call_count", 1) or 1))
+ if clean_text(existing_call.get("model")) == model:
+ totals["flash_writer_calls"] += existing_api_calls
+ elif clean_text(existing_call.get("model")) == reviewer_model:
+ totals["pro_writer_calls"] += existing_api_calls
+ else:
+ raise ProductWriterError(f"{message_id}: resumed writer model is unsupported")
+ totals["api_calls"] += existing_api_calls
+ existing_segment_count = int(existing_call.get("capacity_segment_count", 0) or 0)
+ if existing_segment_count:
+ totals["capacity_segmented_messages"] += 1
+ totals["capacity_segments"] += existing_segment_count
+ totals["capacity_duplicate_count"] += int(
+ existing_call.get("capacity_duplicate_count", 0) or 0
+ )
+ totals["prompt_tokens"] += int(existing_call.get("prompt_tokens", 0) or 0)
+ totals["completion_tokens"] += int(existing_call.get("completion_tokens", 0) or 0)
+ existing_output = dict(existing_call.get("validated_output") or {})
+ warning_count = len(list(existing_output.get("validation_warnings") or []))
+ totals["validation_warning_count"] += warning_count
+ totals["quarantined_item_count"] += int(existing_output.get("quarantined_item_count", 0) or 0)
+ if warning_count:
+ totals["messages_with_warnings"] += 1
+ if not existing_output.get("assertions") and not existing_output.get("interactions") and not existing_output.get("resolutions"):
+ totals["empty_semantic_calls"] += 1
+ previous_message = current_message
+ continue
+ if journal is None:
+ journal_source_message(
+ db_path,
+ scope_id=scope_id,
+ session_id=session_ids[session_index],
+ session_index=session_index,
+ message_id=message_id,
+ message_index=message_index,
+ timestamp=timestamp,
+ role=role,
+ content=content,
+ )
+ journal = source_journal_row(db_path, scope_id=scope_id, message_id=message_id)
+ else:
+ if not args.resume or journal["content_sha256"] != sha256_text(content):
+ raise ProductWriterError(f"{message_id}: source journal cannot be resumed")
+ if journal["enrichment_status"] == "failed":
+ reopen_failed_source_journal(db_path, scope_id=scope_id, message_id=message_id)
+ journal = source_journal_row(db_path, scope_id=scope_id, message_id=message_id)
+ elif journal["enrichment_status"] not in {"pending", "enriched", "enriched_with_warnings"}:
+ raise ProductWriterError(
+ f"{message_id}: unsupported resume journal status={journal['enrichment_status']}"
+ )
+ if journal is None:
+ raise ProductWriterError(f"{message_id}: source journal row was not created")
+ journal_status = journal["enrichment_status"]
+ claim_token = ""
+ if journal_status == "pending":
+ claim_token = claim_source_journal(
+ db_path,
+ scope_id=scope_id,
+ message_id=message_id,
+ claim_owner=journal_claim_owner,
+ lease_seconds=journal_claim_lease_seconds,
+ ) or ""
+ if not claim_token:
+ current_claim = source_journal_row(
+ db_path, scope_id=scope_id, message_id=message_id
+ )
+ if current_claim is None:
+ raise ProductWriterError(f"{message_id}: source journal disappeared during claim")
+ raise ProductWriterError(
+ f"{message_id}: source journal is actively claimed by "
+ f"{current_claim['claim_owner']} until {current_claim['claim_expires_at']}; "
+ "refusing a duplicate Writer call"
+ )
+ journal = source_journal_row(db_path, scope_id=scope_id, message_id=message_id)
+ if (
+ journal is None
+ or journal["enrichment_status"] != "pending"
+ or journal["claim_owner"] != journal_claim_owner
+ or journal["claim_token"] != claim_token
+ or _source_journal_claim_expired(
+ journal["claim_expires_at"], now=datetime.now(timezone.utc)
+ )
+ ):
+ raise ProductWriterError(f"{message_id}: source journal changed after claim")
+ extraction: dict[str, Any] | None = None
+ call_metadata: dict[str, Any] = {}
+ call_log_row: dict[str, Any] | None = None
+ staged_extraction = bool(journal["extraction_json"])
+ if staged_extraction:
+ staged_value = decode_journal_json(
+ journal["extraction_json"], message_id=message_id, field="extraction_json"
+ )
+ if staged_value is not None and not isinstance(staged_value, Mapping):
+ raise ProductWriterError(f"{message_id}: staged extraction is not an object or null")
+ extraction = dict(staged_value) if isinstance(staged_value, Mapping) else None
+ staged_metadata = decode_journal_json(
+ journal["call_metadata_json"], message_id=message_id, field="call_metadata_json"
+ )
+ if not isinstance(staged_metadata, Mapping):
+ raise ProductWriterError(f"{message_id}: staged call metadata is not an object")
+ call_metadata = dict(staged_metadata)
+
+ adapter._reload_graph()
+ preexisting_graph_commit = reconstruct_persisted_message(
+ adapter.graph,
+ message_id=message_id,
+ extraction=extraction,
+ )
+ if role in {"user", "assistant"}:
+ existing_slots = writer_slot_candidates(adapter.graph, content) if role == "user" else []
+ pending_interactions = pending_interaction_candidates(
+ adapter.graph,
+ current_role=role,
+ session_id=session_ids[session_index],
+ session_index=session_index,
+ current_message_index=message_index,
+ )
+ recovered_call = existing_calls.get((qid, message_id))
+ recovered_failure = recoverable_extractions.get((qid, message_id))
+ if not staged_extraction:
+ if recovered_call is not None:
+ if clean_text(recovered_call.get("content_sha256")) != sha256_text(content):
+ raise ProductWriterError(f"{message_id}: recoverable call log source changed")
+ recovered_output = recovered_call.get("validated_output")
+ if not isinstance(recovered_output, Mapping):
+ raise ProductWriterError(f"{message_id}: recoverable call log has no validated output")
+ extraction = dict(recovered_output)
+ call_metadata = call_metadata_from_log(recovered_call)
+ elif recovered_failure is not None:
+ if clean_text(recovered_failure.get("content_sha256")) != sha256_text(content):
+ raise ProductWriterError(f"{message_id}: recoverable extraction source changed")
+ extraction = dict(recovered_failure["validated_output"])
+ call_metadata = dict(recovered_failure["call_metadata"])
+ elif preexisting_graph_commit is not None:
+ raise ProductWriterError(
+ f"{message_id}: graph is committed but Writer output is absent; refusing a duplicate API call"
+ )
+ elif journal_status != "pending":
+ raise ProductWriterError(
+ f"{message_id}: completed journal has no staged Writer output"
+ )
+ else:
+ try:
+ mark_source_api_call_started(
+ db_path,
+ scope_id=scope_id,
+ message_id=message_id,
+ claim_owner=journal_claim_owner,
+ claim_token=claim_token,
+ )
+ extraction, call_metadata = call_with_source_claim_heartbeat(
+ lambda: writer.write(
+ current_message=current_message,
+ previous_message=previous_message,
+ existing_memory_slots=existing_slots,
+ pending_interactions=pending_interactions,
+ ),
+ path=db_path,
+ scope_id=scope_id,
+ message_id=message_id,
+ claim_owner=journal_claim_owner,
+ claim_token=claim_token,
+ lease_seconds=journal_claim_lease_seconds,
+ )
+ except Exception as exc:
+ failure_row = {
+ "question_id": qid,
+ "scope_id": scope_id,
+ "session_index": session_index,
+ "message_index": message_index,
+ "message_id": message_id,
+ "message_role": role,
+ "phase": "writer",
+ "content_sha256": sha256_text(content),
+ "error": f"{exc.__class__.__name__}: {exc}",
+ }
+ if isinstance(exc, ProductWriterResponseError):
+ failed_requests = physical_requests_from_metadata(exc.request_metadata)
+ failure_row["request"] = (
+ dict(exc.request_metadata.get("terminal_request") or {})
+ if exc.request_metadata.get("terminal_request")
+ else (failed_requests[-1] if failed_requests else dict(exc.request_metadata))
+ )
+ failure_row["requests"] = failed_requests
+ failure_row["physical_api_attempt_count"] = len(failed_requests)
+ failure_row["response_content"] = exc.response_content
+ failure_row["response_sha256"] = sha256_text(exc.response_content)
+ with failure_log_path.open("a", encoding="utf-8") as handle:
+ handle.write(json.dumps(failure_row, ensure_ascii=False, sort_keys=True) + "\n")
+ finish_source_journal(
+ db_path,
+ scope_id=scope_id,
+ message_id=message_id,
+ claim_owner=journal_claim_owner,
+ claim_token=claim_token,
+ status="failed",
+ error=f"{exc.__class__.__name__}: {exc}",
+ )
+ raise
+ if extraction is None:
+ raise ProductWriterError(f"{message_id}: writable message has no validated Writer output")
+ call_metadata.setdefault("existing_slot_candidate_count", len(existing_slots))
+ call_metadata.setdefault("pending_interaction_candidate_count", len(pending_interactions))
+ if not staged_extraction and journal_status == "pending":
+ stage_source_enrichment(
+ db_path,
+ scope_id=scope_id,
+ message_id=message_id,
+ claim_owner=journal_claim_owner,
+ claim_token=claim_token,
+ extraction=extraction,
+ call_metadata=call_metadata,
+ )
+ staged_extraction = True
+ totals["writer_calls"] += 1
+ physical_call_count = max(1, int(call_metadata.get("api_call_count", 1) or 1))
+ if call_metadata["model"] == model:
+ totals["flash_writer_calls"] += physical_call_count
+ elif call_metadata["model"] == reviewer_model:
+ totals["pro_writer_calls"] += physical_call_count
+ else:
+ raise ProductWriterError(f"{message_id}: routed writer selected an unsupported model")
+ totals["api_calls"] += physical_call_count
+ capacity_segment_count = int(call_metadata.get("capacity_segment_count", 0) or 0)
+ if capacity_segment_count:
+ totals["capacity_segmented_messages"] += 1
+ totals["capacity_segments"] += capacity_segment_count
+ totals["capacity_duplicate_count"] += int(
+ call_metadata.get("capacity_duplicate_count", 0) or 0
+ )
+ totals["prompt_tokens"] += int(call_metadata.get("prompt_tokens", 0) or 0)
+ totals["completion_tokens"] += int(call_metadata.get("completion_tokens", 0) or 0)
+ warning_count = len(list(extraction.get("validation_warnings") or []))
+ totals["validation_warning_count"] += warning_count
+ totals["quarantined_item_count"] += int(extraction.get("quarantined_item_count", 0) or 0)
+ if warning_count:
+ totals["messages_with_warnings"] += 1
+ if not extraction["assertions"] and not extraction["interactions"] and not extraction["resolutions"]:
+ totals["empty_semantic_calls"] += 1
+ call_log_row = {
+ "question_id": qid,
+ "scope_id": scope_id,
+ "session_index": session_index,
+ "message_index": message_index,
+ "message_id": message_id,
+ "message_role": extraction["message_role"],
+ "content_sha256": sha256_text(content),
+ "assertion_count": len(extraction["assertions"]),
+ "interaction_count": len(extraction["interactions"]),
+ "resolution_count": len(extraction["resolutions"]),
+ "validated_output": extraction,
+ **call_metadata,
+ }
+ else:
+ if staged_extraction and extraction is not None:
+ raise ProductWriterError(f"{message_id}: system/tool journal contains semantic extraction")
+ if not staged_extraction and journal_status == "pending":
+ stage_source_enrichment(
+ db_path,
+ scope_id=scope_id,
+ message_id=message_id,
+ claim_owner=journal_claim_owner,
+ claim_token=claim_token,
+ extraction=None,
+ call_metadata={},
+ )
+ staged_extraction = True
+ persisted: dict[str, Any] | None = None
+ staged_persisted = bool(journal["persisted_json"])
+ adapter._reload_graph()
+ reconstructed = reconstruct_persisted_message(
+ adapter.graph,
+ message_id=message_id,
+ extraction=extraction,
+ )
+ if staged_persisted:
+ staged_value = decode_journal_json(
+ journal["persisted_json"], message_id=message_id, field="persisted_json"
+ )
+ if not isinstance(staged_value, Mapping):
+ raise ProductWriterError(f"{message_id}: staged graph commit is not an object")
+ persisted = dict(staged_value)
+ if reconstructed is None:
+ raise ProductWriterError(f"{message_id}: staged graph commit is absent from the graph")
+ for field in (
+ "source_record_id",
+ "source",
+ "semantic",
+ "facet",
+ "interaction",
+ "resolution_count",
+ "provenance_edge_count",
+ ):
+ if persisted.get(field) != reconstructed.get(field):
+ raise ProductWriterError(
+ f"{message_id}: staged graph field {field} disagrees with persisted graph"
+ )
+ elif reconstructed is not None:
+ persisted = reconstructed
+ if journal_status == "pending":
+ stage_source_persisted(
+ db_path,
+ scope_id=scope_id,
+ message_id=message_id,
+ claim_owner=journal_claim_owner,
+ claim_token=claim_token,
+ persisted=persisted,
+ )
+ staged_persisted = True
+ else:
+ if journal_status != "pending":
+ raise ProductWriterError(f"{message_id}: completed journal has no graph commit")
+ try:
+ persisted = ingest_product_message(
+ adapter,
+ SessionMemoryRecordV2,
+ SessionMemoryEdgeV2,
+ scope_id=scope_id,
+ session_id=session_ids[session_index],
+ session_index=session_index,
+ message_id=message_id,
+ message_index=message_index,
+ date=dates[session_index],
+ timestamp=timestamp,
+ role=role,
+ content=content,
+ extraction=extraction,
+ )
+ except Exception as exc:
+ if call_log_row is not None:
+ failed_requests = physical_requests_from_metadata(call_metadata)
+ with failure_log_path.open("a", encoding="utf-8") as handle:
+ handle.write(
+ json.dumps(
+ {
+ "question_id": qid,
+ "scope_id": scope_id,
+ "session_index": session_index,
+ "message_index": message_index,
+ "message_id": message_id,
+ "message_role": role,
+ "phase": "graph_persist",
+ "content_sha256": sha256_text(content),
+ "error": f"{exc.__class__.__name__}: {exc}",
+ "physical_api_attempt_count": len(failed_requests),
+ "requests": failed_requests,
+ "call_metadata": call_metadata,
+ "validated_output": extraction,
+ },
+ ensure_ascii=False,
+ sort_keys=True,
+ )
+ + "\n"
+ )
+ finish_source_journal(
+ db_path,
+ scope_id=scope_id,
+ message_id=message_id,
+ claim_owner=journal_claim_owner,
+ claim_token=claim_token,
+ status="failed",
+ error=f"{exc.__class__.__name__}: {exc}",
+ )
+ raise
+ stage_source_persisted(
+ db_path,
+ scope_id=scope_id,
+ message_id=message_id,
+ claim_owner=journal_claim_owner,
+ claim_token=claim_token,
+ persisted=persisted,
+ )
+ staged_persisted = True
+ if persisted is None:
+ raise ProductWriterError(f"{message_id}: graph commit was not produced")
+ if call_log_row is not None:
+ call_log_row["persisted"] = persisted
+ existing_call = existing_calls.get((qid, message_id))
+ if existing_call is None:
+ with call_log_path.open("a", encoding="utf-8") as handle:
+ handle.write(json.dumps(call_log_row, ensure_ascii=False, sort_keys=True) + "\n")
+ existing_calls[(qid, message_id)] = call_log_row
+ elif (
+ clean_text(existing_call.get("content_sha256")) != sha256_text(content)
+ or dict(existing_call.get("validated_output") or {}) != extraction
+ or dict(existing_call.get("persisted") or {}).get("source_record_id")
+ != persisted.get("source_record_id")
+ ):
+ raise ProductWriterError(f"{message_id}: existing call log disagrees with staged commit")
+ current_journal = source_journal_row(db_path, scope_id=scope_id, message_id=message_id)
+ if current_journal is None:
+ raise ProductWriterError(f"{message_id}: source journal disappeared before completion")
+ if current_journal["enrichment_status"] == "pending":
+ finish_source_journal(
+ db_path,
+ scope_id=scope_id,
+ message_id=message_id,
+ claim_owner=journal_claim_owner,
+ claim_token=claim_token,
+ status=(
+ "enriched_with_warnings"
+ if extraction and extraction.get("validation_warnings")
+ else "enriched"
+ ),
+ source_record_id=str(persisted["source_record_id"]),
+ error=(
+ json.dumps(extraction.get("validation_warnings"), ensure_ascii=False, sort_keys=True)
+ if extraction and extraction.get("validation_warnings")
+ else ""
+ ),
+ )
+ elif (
+ current_journal["enrichment_status"] not in {"enriched", "enriched_with_warnings"}
+ or current_journal["source_record_id"] != str(persisted["source_record_id"])
+ ):
+ raise ProductWriterError(f"{message_id}: completed journal disagrees with staged graph commit")
+ totals["messages"] += 1
+ totals[f"{role}_messages"] = int(totals.get(f"{role}_messages", 0)) + 1
+ totals["source_records"] += int(persisted["source"])
+ totals["semantic_records"] += int(persisted["semantic"])
+ totals["facet_records"] += int(persisted["facet"])
+ totals["interaction_records"] += int(persisted["interaction"])
+ totals["resolution_edges"] += int(persisted["resolution_count"])
+ totals["provenance_edges"] += int(persisted["provenance_edge_count"])
+ message_log_row = {
+ "question_id": qid,
+ "scope_id": scope_id,
+ "session_index": session_index,
+ "message_index": message_index,
+ "message_id": message_id,
+ "role": role,
+ "timestamp": timestamp,
+ "content_sha256": sha256_text(content),
+ "writer_called": role in {"user", "assistant"},
+ "extracted_assertion_count": len(extraction["assertions"]) if extraction else 0,
+ "extracted_interaction_count": len(extraction["interactions"]) if extraction else 0,
+ "resolution_count": len(extraction["resolutions"]) if extraction else 0,
+ "validation_warnings": list(extraction.get("validation_warnings") or []) if extraction else [],
+ "quarantined_item_count": int(extraction.get("quarantined_item_count", 0) or 0) if extraction else 0,
+ **persisted,
+ }
+ with message_log_path.open("a", encoding="utf-8") as handle:
+ handle.write(json.dumps(message_log_row, sort_keys=True) + "\n")
+ existing_messages[(qid, message_id)] = message_log_row
+ previous_message = current_message
+ if totals["messages"] != totals["source_records"]:
+ raise ProductWriterError(f"{qid}: not every message has exactly one immutable source record")
+ if totals["writer_calls"] != totals["user_messages"] + totals["assistant_messages"]:
+ raise ProductWriterError(f"{qid}: writer call count differs from user+assistant message count")
+ if totals["api_calls"] < totals["writer_calls"]:
+ raise ProductWriterError(f"{qid}: physical API call count is lower than logical writer call count")
+ if totals["flash_writer_calls"] + totals["pro_writer_calls"] != totals["api_calls"]:
+ raise ProductWriterError(f"{qid}: routed physical model call counts differ from API call count")
+ counts = database_counts(db_path, scope_id)
+ if counts["audit_turn_log"] != totals["messages"]:
+ raise ProductWriterError(f"{qid}: persisted audit count differs from message count")
+ if counts["product_source_journal"] != totals["messages"]:
+ raise ProductWriterError(f"{qid}: source journal count differs from message count")
+ reports.append({"question_id": qid, "scope_id": scope_id, "totals": totals, "database_counts": counts})
+
+ failure_rows = (
+ [json.loads(line) for line in failure_log_path.read_text(encoding="utf-8").splitlines() if line.strip()]
+ if failure_log_path.exists()
+ else []
+ )
+ failed_writer_api_attempts = sum(
+ int(row.get("physical_api_attempt_count", 0) or 0)
+ for row in failure_rows
+ if row.get("phase") == "writer"
+ )
+ successful_api_calls = sum(int(sample["totals"]["api_calls"]) for sample in reports)
+ successful_call_rows = list(existing_calls.values())
+
+ def observed_prompt_values(model_name: str, field_name: str) -> list[str]:
+ return sorted(
+ {
+ clean_text(row.get(field_name))
+ for row in successful_call_rows
+ if clean_text(row.get("model")) == model_name
+ and clean_text(row.get(field_name))
+ }
+ )
+
+ flash_prompt_versions = observed_prompt_values(model, "prompt_version")
+ flash_prompt_hashes = observed_prompt_values(model, "prompt_sha256")
+ pro_prompt_versions = observed_prompt_values(
+ reviewer_model, "prompt_version"
+ )
+ pro_prompt_hashes = observed_prompt_values(reviewer_model, "prompt_sha256")
+ jsonl_tail_repairs_total = (
+ sum(1 for line in recovery_log_path.read_text(encoding="utf-8").splitlines() if line.strip())
+ if recovery_log_path.exists()
+ else 0
+ )
+ journal_warning_repairs_total = (
+ sum(
+ int(json.loads(line).get("row_count", 0) or 0)
+ for line in journal_recovery_log_path.read_text(
+ encoding="utf-8"
+ ).splitlines()
+ if line.strip()
+ )
+ if journal_recovery_log_path.exists()
+ else 0
+ )
+ report = {
+ "schema_version": "tmcra.v3.product-writer-run.6",
+ "status": "complete",
+ "writer_schema_version": WRITE_SCHEMA_VERSION,
+ "flash_prompt_version": (
+ flash_prompt_versions[0]
+ if len(flash_prompt_versions) == 1
+ else "mixed"
+ if flash_prompt_versions
+ else PROMPT_VERSION
+ ),
+ "flash_prompt_sha256": (
+ flash_prompt_hashes[0]
+ if len(flash_prompt_hashes) == 1
+ else "mixed"
+ if flash_prompt_hashes
+ else sha256_text(SYSTEM_PROMPT)
+ ),
+ "pro_prompt_version": (
+ pro_prompt_versions[0]
+ if len(pro_prompt_versions) == 1
+ else "mixed"
+ if pro_prompt_versions
+ else PRO_PROMPT_VERSION
+ ),
+ "pro_prompt_sha256": (
+ pro_prompt_hashes[0]
+ if len(pro_prompt_hashes) == 1
+ else "mixed"
+ if pro_prompt_hashes
+ else sha256_text(PRO_SYSTEM_PROMPT)
+ ),
+ "observed_flash_prompt_versions": flash_prompt_versions,
+ "observed_flash_prompt_hashes": flash_prompt_hashes,
+ "observed_pro_prompt_versions": pro_prompt_versions,
+ "observed_pro_prompt_hashes": pro_prompt_hashes,
+ "flash_model": model,
+ "pro_model": reviewer_model,
+ "routing_policy": "user_or_pending_interaction_to_pro_else_flash",
+ "one_model_call_per_message": False,
+ "normal_path_one_model_call_per_writable_message": True,
+ "capacity_overflow_extra_calls_enabled": True,
+ "capacity_overflow_policy": "same_routed_model_recursive_source_segmentation_no_semantic_caps",
+ "capacity_attempt_circuit_breaker": 64,
+ "semantic_item_count_limits": None,
+ "writer_input_catalog_format": "compact_e0_descriptor_and_token_string_array",
+ "max_output_tokens": writer.max_tokens,
+ "validation_tolerance_policy": "warn_on_safe_syntax_normalization_and_quarantine_invalid_items",
+ "syntactic_case_normalization_enabled": True,
+ "redundant_facet_quote_tolerance_enabled": True,
+ "item_quarantine_enabled": True,
+ "silent_repairs_enabled": False,
+ "failed_attempts_logged": len(failure_rows),
+ "failed_writer_api_attempts": failed_writer_api_attempts,
+ "successful_api_calls": successful_api_calls,
+ "physical_api_attempts_including_failures": successful_api_calls + failed_writer_api_attempts,
+ "jsonl_tail_repairs": jsonl_tail_repairs_total,
+ "journal_warning_repairs": journal_warning_repairs_total,
+ "api_key_pool_size": len(api_keys),
+ "resumed": bool(args.resume),
+ "input": str(input_path),
+ "database": str(db_path),
+ "elapsed_seconds": round(time.time() - started, 3),
+ "samples": reports,
+ "fallbacks_enabled": False,
+ "semantic_rule_repairs_enabled": False,
+ "legacy_profile_aggregate_layer_enabled": False,
+ "query_or_evaluation_fields_visible_to_writer": False,
+ "source_write_ahead_log_enabled": True,
+ "staged_source_transaction_enabled": True,
+ "source_journal_cross_process_claiming": True,
+ "source_journal_claim_lease_seconds": journal_claim_lease_seconds,
+ "source_transaction_stages": [
+ "source_journal",
+ "writer_output",
+ "graph_commit",
+ "call_log",
+ "journal_complete",
+ "message_log",
+ ],
+ }
+ (out_dir / "product_writer_report.json").write_text(
+ json.dumps(report, indent=2, ensure_ascii=False, sort_keys=True), encoding="utf-8"
+ )
+ print(json.dumps(report, indent=2, ensure_ascii=False, sort_keys=True))
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v3_recall_planner.py b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v3_recall_planner.py
new file mode 100644
index 0000000000000000000000000000000000000000..99f88b58a25cfdf6ba749c40208d3c4dba74096b
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v3_recall_planner.py
@@ -0,0 +1,406 @@
+"""Strict DeepSeek Flash control plane for recall composition."""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import time
+import urllib.error
+import urllib.request
+from collections.abc import Mapping, Sequence
+from typing import Any
+
+
+PLANNER_VERSION = "tmcra.recall-plan.v4"
+PLANNER_PROMPT_VERSION = "tmcra-recall-planner-2026-07-11.1"
+DEEPSEEK_FLASH_MODEL = "deepseek-v4-flash"
+MODES = frozenset(
+ {
+ "FAST_ONLY",
+ "SLOW_ONLY",
+ "SLOW_WITH_FAST_OVERRIDE",
+ "FAST_WITH_SLOW_CONTEXT",
+ "CONFLICT_COMPARE",
+ }
+)
+PLAN_FIELDS = frozenset(
+ {
+ "mode",
+ "primary_layer",
+ "fast_role",
+ "slow_role",
+ "requires_conflict_pairs",
+ "decision_code",
+ "resolved_query",
+ "decision_reason",
+ "planner_version",
+ }
+)
+_EXPECTED_POLICIES = {
+ "FAST_ONLY": ("fast", "primary", "excluded", False),
+ "SLOW_ONLY": ("slow", "excluded", "primary", False),
+ "SLOW_WITH_FAST_OVERRIDE": ("slow", "override", "primary", False),
+ "FAST_WITH_SLOW_CONTEXT": ("fast", "primary", "context_only", False),
+ "CONFLICT_COMPARE": ("fast", "conflict_candidate", "conflict_candidate", True),
+}
+_EXPECTED_DECISION_CODES = {
+ "FAST_ONLY": "recent_event",
+ "SLOW_ONLY": "stable_or_historical_durable",
+ "SLOW_WITH_FAST_OVERRIDE": "current_durable",
+ "FAST_WITH_SLOW_CONTEXT": "current_task_with_durable_context",
+ "CONFLICT_COMPARE": "explicit_conflict_or_comparison",
+}
+
+SYSTEM_PROMPT = f"""You are the retrieval control plane of a production hierarchical memory graph.
+Return exactly one JSON object with these fields and no others:
+{{"mode":"FAST_ONLY|SLOW_ONLY|SLOW_WITH_FAST_OVERRIDE|FAST_WITH_SLOW_CONTEXT|CONFLICT_COMPARE",
+ "primary_layer":"fast|slow",
+ "fast_role":"primary|excluded|override|conflict_candidate",
+ "slow_role":"primary|excluded|context_only|conflict_candidate",
+ "requires_conflict_pairs":true|false,
+ "decision_code":"recent_event|stable_or_historical_durable|current_durable|current_task_with_durable_context|explicit_conflict_or_comparison",
+ "resolved_query":"standalone retrieval query in the user's language",
+ "decision_reason":"concise reason grounded only in the query and recent dialogue",
+ "planner_version":"{PLANNER_VERSION}"}}
+
+Mode contracts are exact:
+- FAST_ONLY: explicitly recent episodic event, current task, or interaction lookup that needs no durable user background.
+- SLOW_ONLY: explicitly historical baseline, long-term background, or stable summary where the current effective value is not requested.
+- SLOW_WITH_FAST_OVERRIDE: the default for a present/current durable identity, preference, relationship, constraint, or state. Slow is primary and fast must check for recent corrections even when the query does not say "latest".
+- FAST_WITH_SLOW_CONTEXT: a current event/task is primary and durable user context is background-only.
+- CONFLICT_COMPARE: the query explicitly asks about change, contradiction, before/after, or competing states.
+
+Hard routing order:
+1. Explicit comparison, contradiction, or before/after change -> CONFLICT_COMPARE.
+2. A present/current/latest/still standing fact -> SLOW_WITH_FAST_OVERRIDE, never SLOW_ONLY.
+3. A current task or event that needs durable personalization -> FAST_WITH_SLOW_CONTEXT.
+4. An isolated recent event, task, or exact recent-turn lookup -> FAST_ONLY.
+5. SLOW_ONLY is allowed only for an explicitly historical baseline or stable long-term summary.
+6. An unresolved or uncertain follow-up must never use SLOW_ONLY; resolve it from recent_dialogue first.
+
+Resolve references using recent_dialogue only when needed. resolved_query must preserve the user's language and intent,
+be independently searchable without recent_dialogue, and must not answer the query or invent a memory fact. If the query
+is already standalone or the dialogue is unrelated, copy the query unchanged. The current query always overrides stale
+or unrelated dialogue.
+
+Never emit scores, weights, candidate IDs, benchmark labels, or an answer to the query. Never use a layer marked
+unavailable. Choose exactly one mode. This planner chooses roles only; it does not rank or fuse evidence."""
+
+
+class RecallPlannerError(RuntimeError):
+ pass
+
+
+class RecallPlannerResponseError(RecallPlannerError):
+ def __init__(self, message: str, *, response_content: str = "", request_metadata: Mapping[str, Any] | None = None) -> None:
+ super().__init__(message)
+ self.response_content = response_content
+ self.request_metadata = dict(request_metadata or {})
+
+
+def _text(value: Any) -> str:
+ return value.strip() if isinstance(value, str) else ""
+
+
+def _sha256(value: str) -> str:
+ return hashlib.sha256(value.encode("utf-8")).hexdigest()
+
+
+def _metadata(*, key_index: int, started: float, finish_reason: str, content: str = "", usage: Mapping[str, Any] | None = None, http_status: int | None = None, error_type: str | None = None, request_sha256: str = "") -> dict[str, Any]:
+ usage = usage or {}
+ result = {
+ "provider": "deepseek",
+ "model": DEEPSEEK_FLASH_MODEL,
+ "api_key_index": key_index,
+ "physical_call_count": 1,
+ "latency_seconds": round(time.time() - started, 3),
+ "response_sha256": _sha256(content),
+ "finish_reason": finish_reason,
+ "planner_version": PLANNER_VERSION,
+ "prompt_version": PLANNER_PROMPT_VERSION,
+ "request_sha256": request_sha256,
+ "prompt_tokens": int(usage.get("prompt_tokens", 0) or 0),
+ "completion_tokens": int(usage.get("completion_tokens", 0) or 0),
+ "total_tokens": int(usage.get("total_tokens", 0) or 0),
+ }
+ if http_status is not None:
+ result["http_status"] = int(http_status)
+ if error_type:
+ result["error_type"] = error_type
+ return result
+
+
+def _reject_gold(value: Any, path: str = "input") -> None:
+ if isinstance(value, Mapping):
+ for key, item in value.items():
+ key_text = str(key).lower()
+ if "gold" in key_text or "benchmark" in key_text or "expected_answer" in key_text:
+ raise RecallPlannerError(f"{path}.{key} is not permitted in planner input")
+ _reject_gold(item, f"{path}.{key}")
+ elif isinstance(value, list):
+ for index, item in enumerate(value):
+ _reject_gold(item, f"{path}[{index}]")
+
+
+def validate_recall_plan(value: Mapping[str, Any]) -> dict[str, Any]:
+ if not isinstance(value, Mapping) or set(value) != PLAN_FIELDS:
+ raise RecallPlannerError("RecallPlan root must contain exactly the required schema fields")
+ mode = _text(value.get("mode"))
+ if mode not in MODES:
+ raise RecallPlannerError(f"unsupported RecallPlan mode: {mode!r}")
+ primary_layer, fast_role, slow_role, requires_pairs = _EXPECTED_POLICIES[mode]
+ if (
+ value.get("primary_layer") != primary_layer
+ or value.get("fast_role") != fast_role
+ or value.get("slow_role") != slow_role
+ or value.get("requires_conflict_pairs") is not requires_pairs
+ ):
+ raise RecallPlannerError(f"RecallPlan role policy does not match mode {mode}")
+ if value.get("decision_code") != _EXPECTED_DECISION_CODES[mode]:
+ raise RecallPlannerError(f"RecallPlan decision_code does not match mode {mode}")
+ reason = _text(value.get("decision_reason"))
+ if not reason:
+ raise RecallPlannerError("RecallPlan decision_reason is required")
+ resolved_query = _text(value.get("resolved_query"))
+ if not resolved_query:
+ raise RecallPlannerError("RecallPlan resolved_query is required")
+ if len(resolved_query) > 2000:
+ raise RecallPlannerError("RecallPlan resolved_query exceeds 2000 characters")
+ if value.get("planner_version") != PLANNER_VERSION:
+ raise RecallPlannerError(f"RecallPlan planner_version must be {PLANNER_VERSION!r}")
+ return {field: value[field] for field in PLAN_FIELDS}
+
+
+def _validate_recent_dialogue(value: Any) -> list[dict[str, Any]]:
+ if value is None:
+ return []
+ if not isinstance(value, Sequence) or isinstance(value, (str, bytes)):
+ raise RecallPlannerError("recent_dialogue must be a sequence of turn objects")
+ if len(value) > 8:
+ raise RecallPlannerError("recent_dialogue may contain at most 8 turns")
+ output: list[dict[str, Any]] = []
+ for index, item in enumerate(value):
+ if not isinstance(item, Mapping) or set(item) != {"turn_index", "speaker", "text"}:
+ raise RecallPlannerError(f"recent_dialogue[{index}] must contain exactly turn_index, speaker, and text")
+ try:
+ turn_index = int(item["turn_index"])
+ except (TypeError, ValueError) as exc:
+ raise RecallPlannerError(f"recent_dialogue[{index}].turn_index must be an integer") from exc
+ speaker = _text(item.get("speaker")).lower()
+ text = _text(item.get("text"))
+ if speaker not in {"user", "assistant"} or not text:
+ raise RecallPlannerError(f"recent_dialogue[{index}] has an invalid speaker or empty text")
+ if len(text) > 4000:
+ raise RecallPlannerError(f"recent_dialogue[{index}].text exceeds 4000 characters")
+ output.append({"turn_index": turn_index, "speaker": speaker, "text": text})
+ if any(left["turn_index"] >= right["turn_index"] for left, right in zip(output, output[1:])):
+ raise RecallPlannerError("recent_dialogue must be in strictly increasing turn order")
+ _reject_gold(output, "recent_dialogue")
+ return output
+
+
+def _layer_available(value: Any) -> bool:
+ if isinstance(value, Mapping) and "available" in value:
+ return value.get("available") is True
+ if isinstance(value, Sequence) and not isinstance(value, (str, bytes)):
+ return bool(value)
+ return bool(value)
+
+
+def validate_plan_availability(plan: Mapping[str, Any], available_layers: Mapping[str, Any]) -> None:
+ fast_available = _layer_available(available_layers.get("fast"))
+ slow_available = _layer_available(available_layers.get("slow"))
+ mode = plan["mode"]
+ if mode in {"FAST_ONLY", "SLOW_WITH_FAST_OVERRIDE", "FAST_WITH_SLOW_CONTEXT", "CONFLICT_COMPARE"} and not fast_available:
+ raise RecallPlannerError(f"RecallPlan mode {mode} requires an unavailable fast layer")
+ if mode in {"SLOW_ONLY", "SLOW_WITH_FAST_OVERRIDE", "FAST_WITH_SLOW_CONTEXT", "CONFLICT_COMPARE"} and not slow_available:
+ raise RecallPlannerError(f"RecallPlan mode {mode} requires an unavailable slow layer")
+
+
+class DeepSeekFlashRecallPlanner:
+ """One physical Flash call per plan request; errors are never retried or routed."""
+
+ def __init__(self, *, base_url: str, model: str, api_keys: Sequence[str], timeout: float = 30, max_tokens: int = 512) -> None:
+ self.base_url = _text(base_url).rstrip("/")
+ self.model = model
+ self.api_keys = list(dict.fromkeys(_text(key) for key in api_keys if _text(key)))
+ self.timeout = max(1.0, float(timeout))
+ self.max_tokens = max(128, int(max_tokens))
+ self.request_index = 0
+ if not self.base_url or not self.api_keys:
+ raise RecallPlannerError("planner base_url and API key pool are required")
+ if self.model != DEEPSEEK_FLASH_MODEL:
+ raise RecallPlannerError(f"recall planner requires {DEEPSEEK_FLASH_MODEL}, got {self.model!r}")
+
+ def plan(self, *, query: str, question_date: str, available_layers: Mapping[str, Any], recent_dialogue: Sequence[Mapping[str, Any]] | None = None) -> tuple[dict[str, Any], dict[str, Any]]:
+ if not _text(query) or not _text(question_date):
+ raise RecallPlannerError("query and question_date are required")
+ if not isinstance(available_layers, Mapping) or set(available_layers) - {"fast", "slow"}:
+ raise RecallPlannerError("available_layers may contain only fast and slow summaries")
+ _reject_gold(available_layers, "available_layers")
+ dialogue = _validate_recent_dialogue(recent_dialogue)
+ payload = {"query": query, "question_date": question_date, "recent_dialogue": dialogue, "available_layers": dict(available_layers)}
+ key_index = self.request_index % len(self.api_keys)
+ self.request_index += 1
+ request_body = {
+ "model": DEEPSEEK_FLASH_MODEL,
+ "messages": [{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": json.dumps(payload, ensure_ascii=False, separators=(",", ":"))}],
+ "temperature": 0,
+ "max_tokens": self.max_tokens,
+ "response_format": {"type": "json_object"},
+ "thinking": {"type": "disabled"},
+ }
+ request_sha256 = _sha256(json.dumps(request_body, ensure_ascii=False, sort_keys=True))
+ request = urllib.request.Request(
+ f"{self.base_url}/chat/completions",
+ data=json.dumps(request_body, ensure_ascii=False).encode("utf-8"),
+ headers={"Content-Type": "application/json", "Authorization": f"Bearer {self.api_keys[key_index]}"},
+ method="POST",
+ )
+ started = time.time()
+ try:
+ with urllib.request.urlopen(request, timeout=self.timeout) as response:
+ raw_http = response.read().decode("utf-8")
+ response_payload = json.loads(raw_http)
+ except urllib.error.HTTPError as exc:
+ detail = exc.read().decode("utf-8", errors="replace")[:1000]
+ raise RecallPlannerResponseError(f"recall planner HTTP {exc.code}: {detail}", response_content=detail, request_metadata=_metadata(key_index=key_index, started=started, finish_reason="http_error", content=detail, http_status=exc.code, request_sha256=request_sha256)) from exc
+ except (UnicodeDecodeError, json.JSONDecodeError) as exc:
+ raw = locals().get("raw_http", "")
+ raise RecallPlannerResponseError("recall planner returned invalid HTTP JSON", response_content=raw, request_metadata=_metadata(key_index=key_index, started=started, finish_reason="invalid_http_json", content=raw, error_type=exc.__class__.__name__, request_sha256=request_sha256)) from exc
+ except Exception as exc:
+ raise RecallPlannerResponseError(f"recall planner request failed: {exc.__class__.__name__}: {exc}", request_metadata=_metadata(key_index=key_index, started=started, finish_reason="request_error", error_type=exc.__class__.__name__, request_sha256=request_sha256)) from exc
+ usage = response_payload.get("usage") if isinstance(response_payload, Mapping) else {}
+ choices = response_payload.get("choices") if isinstance(response_payload, Mapping) else None
+ if not isinstance(choices, list) or len(choices) != 1 or not isinstance(choices[0], Mapping):
+ raise RecallPlannerResponseError("recall planner response must contain exactly one choice", response_content=raw_http, request_metadata=_metadata(key_index=key_index, started=started, finish_reason="invalid_response", content=raw_http, usage=usage if isinstance(usage, Mapping) else {}, request_sha256=request_sha256))
+ choice = choices[0]
+ content = (choice.get("message") or {}).get("content") if isinstance(choice.get("message"), Mapping) else None
+ finish_reason = _text(choice.get("finish_reason"))
+ metadata = _metadata(key_index=key_index, started=started, finish_reason=finish_reason, content=content if isinstance(content, str) else raw_http, usage=usage if isinstance(usage, Mapping) else {}, request_sha256=request_sha256)
+ if finish_reason != "stop" or not isinstance(content, str):
+ raise RecallPlannerResponseError("recall planner response did not finish with a JSON string", response_content=raw_http, request_metadata=metadata)
+ try:
+ plan = validate_recall_plan(json.loads(content))
+ except (json.JSONDecodeError, RecallPlannerError) as exc:
+ raise RecallPlannerResponseError(f"recall planner returned invalid RecallPlan: {exc}", response_content=content, request_metadata=metadata) from exc
+ try:
+ validate_plan_availability(plan, available_layers)
+ except RecallPlannerError as exc:
+ raise RecallPlannerResponseError(
+ f"recall planner selected an unavailable layer: {exc}",
+ response_content=content,
+ request_metadata=metadata,
+ ) from exc
+ return plan, metadata
+
+
+def _source_parents(item: Mapping[str, Any]) -> list[dict[str, Any]]:
+ raw_values: list[Any] = []
+ if isinstance(item.get("source_parent"), Mapping):
+ raw_values.append(item["source_parent"])
+ if isinstance(item.get("source_parents"), Sequence) and not isinstance(item.get("source_parents"), (str, bytes)):
+ raw_values.extend(item["source_parents"])
+ parents: list[dict[str, Any]] = []
+ seen: set[tuple[int, int]] = set()
+ for raw in raw_values:
+ if not isinstance(raw, Mapping):
+ raise RecallPlannerError("slow capsule source parent must be an object")
+ try:
+ location = (int(raw["session_index"]), int(raw["parent_chunk_index"]))
+ except (KeyError, TypeError, ValueError) as exc:
+ raise RecallPlannerError("slow capsule source parent lacks integer session/parent coordinates") from exc
+ if location in seen:
+ continue
+ seen.add(location)
+ parents.append(dict(raw))
+ return parents
+
+
+def _group_by_slot(items: Sequence[Mapping[str, Any]], *, layer: str, require_parent: bool) -> dict[str, list[Mapping[str, Any]]]:
+ grouped: dict[str, list[Mapping[str, Any]]] = {}
+ for item in items:
+ if not isinstance(item, Mapping):
+ raise RecallPlannerError(f"{layer} candidate must be a mapping")
+ slot = _text(item.get("canonical_slot"))
+ if not slot:
+ raise RecallPlannerError(f"{layer} candidate lacks canonical_slot")
+ if require_parent and not _source_parents(item):
+ raise RecallPlannerError(f"slow capsule {slot!r} lacks source_parent mapping")
+ grouped.setdefault(slot, []).append(item)
+ return grouped
+
+
+def apply_recall_plan(plan: Mapping[str, Any], fast_candidates: Sequence[Mapping[str, Any]], slow_capsules: Sequence[Mapping[str, Any]]) -> list[dict[str, Any]]:
+ """Compose evidence units by role, without combining or recalculating layer scores."""
+ normalized = validate_recall_plan(plan)
+ fast = _group_by_slot(fast_candidates, layer="fast", require_parent=False)
+ slow = _group_by_slot(slow_capsules, layer="slow", require_parent=True)
+ mode = normalized["mode"]
+ units: list[dict[str, Any]] = []
+ if mode == "FAST_ONLY":
+ return [
+ {"unit_type": "fast_primary", "canonical_slot": slot, "primary_layer": "fast", "fast_candidate": candidate}
+ for slot, candidates in fast.items()
+ for candidate in candidates
+ ]
+ if mode == "SLOW_ONLY":
+ return [
+ {"unit_type": "slow_primary", "canonical_slot": slot, "primary_layer": "slow", "slow_capsule": capsule}
+ for slot, capsules in slow.items()
+ for capsule in capsules
+ ]
+ if mode == "SLOW_WITH_FAST_OVERRIDE":
+ for slot, capsules in slow.items():
+ for capsule in capsules:
+ units.append(
+ {
+ "unit_type": "slow_primary_with_fast_override",
+ "canonical_slot": slot,
+ "primary_layer": "slow",
+ "slow_capsule": capsule,
+ "fast_overrides": list(fast.get(slot, [])),
+ }
+ )
+ return units
+ if mode == "FAST_WITH_SLOW_CONTEXT":
+ all_slow_context = [capsule for capsules in slow.values() for capsule in capsules]
+ for slot, candidates in fast.items():
+ for candidate in candidates:
+ units.append(
+ {
+ "unit_type": "fast_primary_with_slow_context",
+ "canonical_slot": slot,
+ "primary_layer": "fast",
+ "fast_candidate": candidate,
+ "slow_context": list(all_slow_context),
+ }
+ )
+ return units
+ for slot in fast.keys() & slow.keys():
+ units.append(
+ {
+ "unit_type": "conflict_group",
+ "canonical_slot": slot,
+ "primary_layer": "fast",
+ "fast_candidates": list(fast[slot]),
+ "slow_capsules": list(slow[slot]),
+ }
+ )
+ if not units and fast and slow:
+ ranked_fast = [candidate for candidates in fast.values() for candidate in candidates]
+ ranked_slow = [capsule for capsules in slow.values() for capsule in capsules]
+ pair_count = max(len(ranked_fast), len(ranked_slow))
+ for index in range(pair_count):
+ units.append(
+ {
+ "unit_type": "conflict_group",
+ "canonical_slot": f"semantic_conflict.{index}",
+ "primary_layer": "fast",
+ "fast_candidates": [ranked_fast[min(index, len(ranked_fast) - 1)]],
+ "slow_capsules": [ranked_slow[min(index, len(ranked_slow) - 1)]],
+ }
+ )
+ if not units:
+ raise RecallPlannerError("CONFLICT_COMPARE produced no semantic conflict pairs")
+ return units
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v3_reranker.py b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v3_reranker.py
new file mode 100644
index 0000000000000000000000000000000000000000..e2da8d00b498acdebb6e0e15653fce4ba10b6efc
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v3_reranker.py
@@ -0,0 +1,1014 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import os
+import random
+import time
+from collections import Counter, defaultdict
+from pathlib import Path
+from typing import Any, Mapping, Sequence
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+from tmcra_v3_schema import (
+ CHANNEL_NAMES,
+ SCHEMA_VERSION,
+ channel_vector,
+ clean_text,
+ read_jsonl,
+ validate_sample,
+ write_jsonl,
+)
+from tmcra_model_assets import load_optional_hf_model_manifest
+
+
+def pair_key(sample: Mapping[str, Any], candidate: Mapping[str, Any]) -> str:
+ payload = "\0".join(
+ (
+ clean_text(sample.get("question_id")),
+ clean_text(candidate.get("candidate_id")),
+ clean_text(sample.get("query_text")),
+ clean_text(candidate.get("text")),
+ )
+ )
+ return hashlib.sha256(payload.encode("utf-8")).hexdigest()
+
+
+def stable_int(value: str) -> int:
+ return int.from_bytes(hashlib.blake2b(value.encode("utf-8"), digest_size=8).digest(), "big")
+
+
+def atomic_torch_save(payload: Mapping[str, Any], target: Path) -> None:
+ temporary = target.with_suffix(target.suffix + ".tmp")
+ torch.save(dict(payload), temporary)
+ os.replace(temporary, target)
+
+
+def command_precompute_cross(args: argparse.Namespace) -> None:
+ if not torch.cuda.is_available() and not args.cpu:
+ raise RuntimeError("CUDA is unavailable; pass --cpu explicitly")
+ from transformers import AutoModelForSequenceClassification, AutoTokenizer
+
+ sample_paths = [Path(value) for value in args.samples]
+ rows = [row for path in sample_paths for row in read_jsonl(path)]
+ for row in rows:
+ validate_sample(
+ row,
+ require_positive=False,
+ allowed_splits=("full_eval", "train", "holdout", "aux_train", "aux_dev"),
+ )
+ model_dir = Path(args.model).resolve()
+ model_manifest = load_optional_hf_model_manifest(model_dir)
+ out_dir = Path(args.out_dir)
+ out_dir.mkdir(parents=True, exist_ok=True)
+ shard_dir = out_dir / "shards"
+ shard_dir.mkdir(parents=True, exist_ok=True)
+ items: list[dict[str, Any]] = []
+ seen: set[str] = set()
+ raw_pair_count = 0
+ duplicate_pair_count = 0
+ for row in rows:
+ for candidate in row["candidates"]:
+ raw_pair_count += 1
+ key = pair_key(row, candidate)
+ if key in seen:
+ duplicate_pair_count += 1
+ continue
+ seen.add(key)
+ items.append(
+ {
+ "key": key,
+ "question_id": row["question_id"],
+ "candidate_id": candidate["candidate_id"],
+ "query_text": row["query_text"],
+ "candidate_text": candidate["text"],
+ }
+ )
+ write_jsonl(
+ out_dir / "items.jsonl",
+ [
+ {
+ "key": item["key"],
+ "question_id": item["question_id"],
+ "candidate_id": item["candidate_id"],
+ "query_len": len(item["query_text"]),
+ "candidate_len": len(item["candidate_text"]),
+ }
+ for item in items
+ ],
+ )
+ device = torch.device("cpu" if args.cpu else "cuda")
+ tokenizer = AutoTokenizer.from_pretrained(
+ str(model_dir),
+ local_files_only=True,
+ trust_remote_code=False,
+ )
+ model = AutoModelForSequenceClassification.from_pretrained(
+ str(model_dir),
+ local_files_only=True,
+ trust_remote_code=False,
+ torch_dtype=torch.float32 if args.cpu else torch.float16,
+ ).to(device)
+ model.eval()
+ hidden_size = int(getattr(model.config, "hidden_size", 0) or 0)
+ if hidden_size <= 0:
+ raise RuntimeError("reranker config.hidden_size is missing")
+ started = time.time()
+ shard_paths: list[Path] = []
+ for start in range(0, len(items), args.shard_size):
+ stop = min(len(items), start + args.shard_size)
+ shard_index = start // args.shard_size
+ shard_path = shard_dir / f"shard_{shard_index:05d}.pt"
+ expected_keys = [item["key"] for item in items[start:stop]]
+ if shard_path.exists():
+ cached = torch.load(shard_path, map_location="cpu", weights_only=True)
+ if list(cached.get("keys") or []) != expected_keys:
+ raise RuntimeError(f"cross-cache resume key mismatch: {shard_path}")
+ representations = cached.get("representations")
+ logits = cached.get("semantic_logits")
+ if representations is None or logits is None:
+ raise RuntimeError(f"cross-cache resume payload invalid: {shard_path}")
+ if tuple(representations.shape) != (len(expected_keys), hidden_size):
+ raise RuntimeError(f"cross-cache resume representation shape mismatch: {shard_path}")
+ print(json.dumps({"status": "resume", "shard": shard_index, "done": stop, "total": len(items)}), flush=True)
+ shard_paths.append(shard_path)
+ continue
+ rep_chunks: list[torch.Tensor] = []
+ logit_chunks: list[torch.Tensor] = []
+ shard_items = items[start:stop]
+ with torch.inference_mode():
+ for batch_start in range(0, len(shard_items), args.batch_size):
+ batch = shard_items[batch_start : batch_start + args.batch_size]
+ encoded = tokenizer(
+ [item["query_text"] for item in batch],
+ [item["candidate_text"] for item in batch],
+ padding=True,
+ truncation=False,
+ return_tensors="pt",
+ )
+ token_lengths = encoded["attention_mask"].sum(dim=1)
+ longest = int(token_lengths.max())
+ if longest > args.max_length:
+ offending = int(torch.argmax(token_lengths))
+ raise RuntimeError(
+ f"cross-encoder pair has {longest} tokens, exceeding strict max_length={args.max_length}; "
+ f"qid={batch[offending]['question_id']} candidate={batch[offending]['candidate_id']}"
+ )
+ encoded = {key: value.to(device) for key, value in encoded.items()}
+ output = model(**encoded, output_hidden_states=True, return_dict=True)
+ if output.hidden_states is None:
+ raise RuntimeError("reranker did not return hidden states")
+ representation = output.hidden_states[-1][:, 0].detach().cpu().float()
+ semantic = output.logits.detach().cpu().float()
+ if semantic.ndim == 2 and semantic.shape[1] == 1:
+ semantic = semantic[:, 0]
+ elif semantic.ndim != 1:
+ raise RuntimeError(f"expected one reranker logit per pair, got {tuple(semantic.shape)}")
+ rep_chunks.append(representation.to(torch.float16))
+ logit_chunks.append(semantic)
+ representations = torch.cat(rep_chunks, dim=0).contiguous()
+ logits = torch.cat(logit_chunks, dim=0).contiguous()
+ atomic_torch_save(
+ {"keys": expected_keys, "representations": representations, "semantic_logits": logits},
+ shard_path,
+ )
+ print(
+ json.dumps(
+ {
+ "status": "encoded",
+ "shard": shard_index,
+ "done": stop,
+ "total": len(items),
+ "elapsed_sec": round(time.time() - started, 3),
+ }
+ ),
+ flush=True,
+ )
+ shard_paths.append(shard_path)
+ all_keys: list[str] = []
+ all_representations: list[torch.Tensor] = []
+ all_logits: list[torch.Tensor] = []
+ for path in shard_paths:
+ payload = torch.load(path, map_location="cpu", weights_only=True)
+ all_keys.extend(list(payload["keys"]))
+ all_representations.append(payload["representations"])
+ all_logits.append(payload["semantic_logits"])
+ expected_all_keys = [item["key"] for item in items]
+ representations = torch.cat(all_representations, dim=0).contiguous()
+ logits = torch.cat(all_logits, dim=0).contiguous()
+ if all_keys != expected_all_keys:
+ raise RuntimeError("cross-cache final key order mismatch")
+ if tuple(representations.shape) != (len(items), hidden_size) or tuple(logits.shape) != (len(items),):
+ raise RuntimeError("cross-cache final tensor shape mismatch")
+ atomic_torch_save(
+ {"keys": all_keys, "representations": representations, "semantic_logits": logits},
+ out_dir / "cross_cache.pt",
+ )
+ manifest = {
+ "created_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
+ "schema_version": SCHEMA_VERSION,
+ "samples": [str(path.resolve()) for path in sample_paths],
+ "sample_count": len(rows),
+ "pair_count": len(items),
+ "raw_pair_count": raw_pair_count,
+ "duplicate_pair_count": duplicate_pair_count,
+ "hidden_size": hidden_size,
+ "max_length": args.max_length,
+ "strict_no_truncation": True,
+ "dtype": "float16",
+ "model": str(model_dir),
+ "model_repo_id": model_manifest.get("repo_id"),
+ "model_revision": model_manifest.get("revision"),
+ "elapsed_sec": round(time.time() - started, 3),
+ }
+ (out_dir / "manifest.json").write_text(json.dumps(manifest, indent=2, sort_keys=True), encoding="utf-8")
+ print(json.dumps(manifest, indent=2, sort_keys=True))
+
+
+class CrossRepresentationCache:
+ def __init__(self, root: str):
+ directory = Path(root)
+ manifest_path = directory / "manifest.json"
+ cache_path = directory / "cross_cache.pt"
+ if not manifest_path.exists() or not cache_path.exists():
+ raise FileNotFoundError(f"complete cross cache is required: {directory}")
+ self.manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
+ payload = torch.load(cache_path, map_location="cpu", weights_only=True)
+ keys = list(payload.get("keys") or [])
+ representations = payload.get("representations")
+ semantic_logits = payload.get("semantic_logits")
+ if not keys or representations is None or semantic_logits is None:
+ raise RuntimeError(f"invalid cross cache: {cache_path}")
+ if len(keys) != len(set(keys)) or len(keys) != representations.shape[0] or len(keys) != semantic_logits.shape[0]:
+ raise RuntimeError(f"cross cache row mismatch: {cache_path}")
+ self.index = {key: index for index, key in enumerate(keys)}
+ self.representations = representations.float().contiguous()
+ self.semantic_logits = semantic_logits.float().contiguous()
+ self.hidden_size = int(representations.shape[1])
+
+ def get(self, sample: Mapping[str, Any], candidate: Mapping[str, Any]) -> tuple[torch.Tensor, torch.Tensor]:
+ key = pair_key(sample, candidate)
+ index = self.index.get(key)
+ if index is None:
+ raise RuntimeError(
+ f"cross cache miss: qid={sample.get('question_id')} candidate={candidate.get('candidate_id')} key={key}"
+ )
+ return self.representations[index], self.semantic_logits[index]
+
+
+class ChannelAwareMemoryReranker(nn.Module):
+ def __init__(
+ self,
+ representation_dim: int,
+ channel_dim: int,
+ hidden_dim: int = 256,
+ layers: int = 2,
+ channel_mean: torch.Tensor | None = None,
+ channel_scale: torch.Tensor | None = None,
+ ):
+ super().__init__()
+ mean = torch.zeros(channel_dim, dtype=torch.float32) if channel_mean is None else channel_mean.float()
+ scale = torch.ones(channel_dim, dtype=torch.float32) if channel_scale is None else channel_scale.float()
+ if tuple(mean.shape) != (channel_dim,) or tuple(scale.shape) != (channel_dim,):
+ raise RuntimeError("channel normalization tensors have the wrong shape")
+ self.register_buffer("channel_mean", mean.contiguous())
+ self.register_buffer("channel_scale", scale.clamp_min(1e-4).contiguous())
+ self.representation_proj = nn.Sequential(
+ nn.LayerNorm(representation_dim),
+ nn.Linear(representation_dim, hidden_dim),
+ nn.GELU(),
+ )
+ self.semantic_proj = nn.Sequential(nn.Linear(1, hidden_dim), nn.GELU())
+ self.channel_proj = nn.Sequential(nn.Linear(channel_dim, hidden_dim), nn.GELU())
+ self.channel_gate = nn.Sequential(nn.Linear(hidden_dim * 2, hidden_dim), nn.Sigmoid())
+ encoder_layer = nn.TransformerEncoderLayer(
+ d_model=hidden_dim,
+ nhead=8,
+ dim_feedforward=hidden_dim * 3,
+ dropout=0.1,
+ activation="gelu",
+ batch_first=True,
+ norm_first=True,
+ )
+ self.set_encoder = nn.TransformerEncoder(encoder_layer, num_layers=max(1, int(layers)))
+ self.delta_head = nn.Sequential(
+ nn.LayerNorm(hidden_dim),
+ nn.Linear(hidden_dim, hidden_dim),
+ nn.GELU(),
+ nn.Dropout(0.1),
+ nn.Linear(hidden_dim, 1),
+ )
+ nn.init.zeros_(self.delta_head[-1].weight)
+ nn.init.zeros_(self.delta_head[-1].bias)
+
+ def forward(
+ self,
+ representations: torch.Tensor,
+ semantic_logits: torch.Tensor,
+ channels: torch.Tensor,
+ mask: torch.Tensor,
+ *,
+ ablation: str = "full",
+ ) -> torch.Tensor:
+ if ablation not in {"full", "no_language", "no_channels", "no_graph"}:
+ raise RuntimeError(f"unknown ablation: {ablation}")
+ reps = representations
+ semantic = semantic_logits
+ feature_values = (channels - self.channel_mean) / self.channel_scale
+ if ablation == "no_language":
+ reps = torch.zeros_like(reps)
+ semantic = torch.zeros_like(semantic)
+ if ablation == "no_channels":
+ feature_values = torch.zeros_like(feature_values)
+ elif ablation == "no_graph":
+ feature_values = feature_values.clone()
+ feature_values[..., 2:5] = 0.0
+ rep_hidden = self.representation_proj(reps)
+ semantic_hidden = self.semantic_proj(semantic.unsqueeze(-1))
+ channel_hidden = self.channel_proj(feature_values)
+ gate = self.channel_gate(torch.cat((rep_hidden, semantic_hidden), dim=-1))
+ hidden = rep_hidden + semantic_hidden + gate * channel_hidden
+ hidden = self.set_encoder(hidden, src_key_padding_mask=~mask)
+ delta = self.delta_head(hidden).squeeze(-1)
+ scores = semantic + delta
+ return scores.masked_fill(~mask, -1e4)
+
+
+def select_training_candidates(sample: Mapping[str, Any], max_candidates: int, epoch: int) -> list[dict[str, Any]]:
+ candidates = list(sample["candidates"])
+ positives = [candidate for candidate in candidates if candidate["labels"]["relevance"]]
+ negatives = [candidate for candidate in candidates if not candidate["labels"]["relevance"]]
+ if not positives:
+ raise RuntimeError(f"training sample has no positive: {sample.get('question_id')}")
+ if max_candidates < 2:
+ raise RuntimeError("max_train_candidates must be at least 2")
+ positive_budget = min(len(positives), max(1, max_candidates // 2))
+ positives.sort(
+ key=lambda candidate: (
+ -float(candidate["channels"]["graph_rank_rr"]),
+ -float(candidate["channels"]["dense_score"]),
+ clean_text(candidate["candidate_id"]),
+ )
+ )
+ if len(positives) > positive_budget:
+ anchor_count = min(2, positive_budget)
+ anchors = positives[:anchor_count]
+ rotating = positives[anchor_count:]
+ rng = random.Random(stable_int(f"positive-bag:{epoch}:{sample['question_id']}"))
+ rng.shuffle(rotating)
+ positives = anchors + rotating[: positive_budget - anchor_count]
+ negatives.sort(
+ key=lambda candidate: (
+ -int(bool(candidate["labels"]["hard_negative"])),
+ -float(candidate["channels"]["graph_rank_rr"]),
+ -float(candidate["channels"]["dense_score"]),
+ clean_text(candidate["candidate_id"]),
+ )
+ )
+ selected = positives + negatives[: max(0, max_candidates - len(positives))]
+ rng = random.Random(stable_int(f"candidate-shuffle:{epoch}:{sample['question_id']}"))
+ rng.shuffle(selected)
+ return selected
+
+
+def make_batch(
+ samples: Sequence[Mapping[str, Any]],
+ cache: CrossRepresentationCache,
+ device: torch.device,
+ *,
+ training: bool,
+ max_candidates: int,
+ epoch: int,
+) -> dict[str, Any]:
+ selected_by_sample = [
+ select_training_candidates(sample, max_candidates, epoch) if training else list(sample["candidates"])
+ for sample in samples
+ ]
+ max_count = max(len(candidates) for candidates in selected_by_sample)
+ batch_size = len(samples)
+ representations = torch.zeros((batch_size, max_count, cache.hidden_size), dtype=torch.float32)
+ semantic = torch.zeros((batch_size, max_count), dtype=torch.float32)
+ channels = torch.zeros((batch_size, max_count, len(CHANNEL_NAMES)), dtype=torch.float32)
+ labels = torch.zeros((batch_size, max_count), dtype=torch.float32)
+ mask = torch.zeros((batch_size, max_count), dtype=torch.bool)
+ candidate_ids: list[list[str]] = []
+ sample_weights = torch.tensor(
+ [float((sample.get("supervision") or {}).get("training_weight", 1.0) or 1.0) for sample in samples],
+ dtype=torch.float32,
+ )
+ for row_index, (sample, candidates) in enumerate(zip(samples, selected_by_sample)):
+ ids: list[str] = []
+ for candidate_index, candidate in enumerate(candidates):
+ representation, semantic_logit = cache.get(sample, candidate)
+ representations[row_index, candidate_index] = representation
+ semantic[row_index, candidate_index] = semantic_logit
+ channels[row_index, candidate_index] = torch.tensor(channel_vector(candidate), dtype=torch.float32)
+ labels[row_index, candidate_index] = float(bool(candidate["labels"]["relevance"]))
+ mask[row_index, candidate_index] = True
+ ids.append(clean_text(candidate["candidate_id"]))
+ candidate_ids.append(ids)
+ return {
+ "representations": representations.to(device),
+ "semantic_logits": semantic.to(device),
+ "channels": channels.to(device),
+ "labels": labels.to(device),
+ "mask": mask.to(device),
+ "candidate_ids": candidate_ids,
+ "sample_weights": sample_weights.to(device),
+ }
+
+
+def listwise_loss(
+ scores: torch.Tensor,
+ labels: torch.Tensor,
+ mask: torch.Tensor,
+ sample_weights: torch.Tensor | None = None,
+) -> torch.Tensor:
+ losses: list[torch.Tensor] = []
+ weights: list[torch.Tensor] = []
+ if sample_weights is None:
+ sample_weights = torch.ones(scores.shape[0], dtype=scores.dtype, device=scores.device)
+ for row_scores, row_labels, row_mask, row_weight in zip(scores, labels, mask, sample_weights):
+ valid_scores = row_scores[row_mask]
+ valid_labels = row_labels[row_mask]
+ if not bool((valid_labels > 0.5).any()):
+ continue
+ log_probabilities = F.log_softmax(valid_scores, dim=0)
+ positive_mask = valid_labels > 0.5
+ losses.append(-torch.logsumexp(log_probabilities[positive_mask], dim=0))
+ weights.append(row_weight)
+ if not losses:
+ return scores.sum() * 0.0
+ loss_tensor = torch.stack(losses)
+ weight_tensor = torch.stack(weights).to(dtype=loss_tensor.dtype).clamp_min(1e-6)
+ return (loss_tensor * weight_tensor).sum() / weight_tensor.sum()
+
+
+def rank_metrics_update(target: dict[str, float], scores: torch.Tensor, labels: torch.Tensor, valid: torch.Tensor) -> None:
+ valid_scores = scores[valid]
+ valid_labels = labels[valid]
+ positives = set(torch.nonzero(valid_labels > 0.5, as_tuple=False).squeeze(-1).tolist())
+ target["n"] += 1
+ if not positives:
+ target["candidate_miss"] += 1
+ return
+ order = torch.argsort(valid_scores, descending=True).tolist()
+ rank = next((position + 1 for position, index in enumerate(order) if index in positives), None)
+ for cutoff in (1, 3, 5, 10, 20):
+ target[f"recall@{cutoff}"] += int(rank is not None and rank <= cutoff)
+ target["mrr"] += 0.0 if rank is None else 1.0 / rank
+
+
+def finalize_metrics(values: Mapping[str, float]) -> dict[str, Any]:
+ n = max(1.0, float(values.get("n", 0.0)))
+ output: dict[str, Any] = {"n": int(values.get("n", 0.0))}
+ for key, value in sorted(values.items()):
+ if key == "n":
+ continue
+ output[key] = round(float(value) / n, 6)
+ return output
+
+
+def evaluate_model(
+ model: ChannelAwareMemoryReranker,
+ rows: Sequence[Mapping[str, Any]],
+ cache: CrossRepresentationCache,
+ device: torch.device,
+ *,
+ batch_size: int,
+ ablation: str,
+) -> dict[str, Any]:
+ model.eval()
+ totals: dict[str, float] = defaultdict(float)
+ with torch.no_grad():
+ for start in range(0, len(rows), batch_size):
+ samples = rows[start : start + batch_size]
+ batch = make_batch(samples, cache, device, training=False, max_candidates=0, epoch=0)
+ scores = model(
+ batch["representations"],
+ batch["semantic_logits"],
+ batch["channels"],
+ batch["mask"],
+ ablation=ablation,
+ ).cpu()
+ labels = batch["labels"].cpu()
+ mask = batch["mask"].cpu()
+ for index in range(len(samples)):
+ rank_metrics_update(totals, scores[index], labels[index], mask[index])
+ return finalize_metrics(totals)
+
+
+def split_train_dev(rows: Sequence[Mapping[str, Any]], dev_count: int) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
+ ordered = sorted(rows, key=lambda row: (stable_int(f"tmcra-v3-dev:{row['question_id']}"), row["question_id"]))
+ if dev_count <= 0 or dev_count >= len(ordered):
+ raise RuntimeError("dev_count must leave non-empty train and dev sets")
+ dev = list(ordered[:dev_count])
+ train = list(ordered[dev_count:])
+ return train, dev
+
+
+def compute_channel_statistics(rows: Sequence[Mapping[str, Any]]) -> tuple[torch.Tensor, torch.Tensor]:
+ values = [channel_vector(candidate) for row in rows for candidate in row["candidates"]]
+ if not values:
+ raise RuntimeError("cannot compute channel statistics from an empty candidate set")
+ matrix = torch.tensor(values, dtype=torch.float32)
+ mean = matrix.mean(dim=0)
+ scale = matrix.std(dim=0, unbiased=False)
+ scale = torch.where(scale < 1e-4, torch.ones_like(scale), scale)
+ return mean, scale
+
+
+def load_model_checkpoint(path: Path, cache: CrossRepresentationCache, device: torch.device) -> tuple[ChannelAwareMemoryReranker, dict[str, Any]]:
+ checkpoint = torch.load(path, map_location=device, weights_only=True)
+ if checkpoint.get("schema_version") != SCHEMA_VERSION:
+ raise RuntimeError("checkpoint schema version mismatch")
+ if tuple(checkpoint.get("channel_names") or ()) != CHANNEL_NAMES:
+ raise RuntimeError("checkpoint channel contract mismatch")
+ config = checkpoint["model_config"]
+ model = ChannelAwareMemoryReranker(
+ representation_dim=cache.hidden_size,
+ channel_dim=len(CHANNEL_NAMES),
+ hidden_dim=int(config["hidden_dim"]),
+ layers=int(config["layers"]),
+ ).to(device)
+ model.load_state_dict(checkpoint["model_state"])
+ return model, checkpoint
+
+
+def command_train(args: argparse.Namespace) -> None:
+ rows = read_jsonl(Path(args.train_samples))
+ for row in rows:
+ validate_sample(row, require_positive=True, allowed_splits=("train", "aux_train"))
+ if args.dev_samples:
+ train_rows = list(rows)
+ dev_rows = read_jsonl(Path(args.dev_samples))
+ for row in dev_rows:
+ validate_sample(row, require_positive=False, allowed_splits=("train", "full_eval", "aux_dev"))
+ else:
+ train_rows, dev_rows = split_train_dev(rows, args.dev_count)
+ cache = CrossRepresentationCache(args.cross_cache)
+ device = torch.device("cpu" if args.cpu else "cuda")
+ if device.type == "cuda" and not torch.cuda.is_available():
+ raise RuntimeError("CUDA is unavailable; pass --cpu explicitly")
+ channel_mean, channel_scale = compute_channel_statistics(train_rows)
+ model = ChannelAwareMemoryReranker(
+ representation_dim=cache.hidden_size,
+ channel_dim=len(CHANNEL_NAMES),
+ hidden_dim=args.hidden_dim,
+ layers=args.layers,
+ channel_mean=channel_mean,
+ channel_scale=channel_scale,
+ ).to(device)
+ if args.init_checkpoint:
+ initial = torch.load(Path(args.init_checkpoint), map_location="cpu", weights_only=True)
+ if initial.get("schema_version") != SCHEMA_VERSION:
+ raise RuntimeError("initial checkpoint schema version mismatch")
+ if tuple(initial.get("channel_names") or ()) != CHANNEL_NAMES:
+ raise RuntimeError("initial checkpoint channel contract mismatch")
+ initial_config = initial.get("model_config") or {}
+ if int(initial_config.get("hidden_dim", -1)) != args.hidden_dim or int(initial_config.get("layers", -1)) != args.layers:
+ raise RuntimeError("initial checkpoint architecture does not match requested hidden_dim/layers")
+ transferable = {
+ key: value
+ for key, value in initial["model_state"].items()
+ if key not in {"channel_mean", "channel_scale"}
+ }
+ missing, unexpected = model.load_state_dict(transferable, strict=False)
+ if set(missing) != {"channel_mean", "channel_scale"} or unexpected:
+ raise RuntimeError(f"unexpected initial checkpoint keys: missing={missing} unexpected={unexpected}")
+ optimizer = torch.optim.AdamW(model.parameters(), lr=args.lr, weight_decay=args.weight_decay)
+ rng = random.Random(args.seed)
+ out_dir = Path(args.out_dir)
+ out_dir.mkdir(parents=True, exist_ok=False)
+ best_score: float | None = None
+ best_epoch = 0
+ stale_epochs = 0
+ history: list[dict[str, Any]] = []
+ for epoch in range(1, args.epochs + 1):
+ model.train()
+ rng.shuffle(train_rows)
+ losses: list[float] = []
+ for start in range(0, len(train_rows), args.batch_size):
+ samples = train_rows[start : start + args.batch_size]
+ batch = make_batch(
+ samples,
+ cache,
+ device,
+ training=True,
+ max_candidates=args.max_train_candidates,
+ epoch=epoch,
+ )
+ scores = model(
+ batch["representations"],
+ batch["semantic_logits"],
+ batch["channels"],
+ batch["mask"],
+ )
+ ranking_loss = listwise_loss(scores, batch["labels"], batch["mask"], batch["sample_weights"])
+ valid_delta = (scores - batch["semantic_logits"])[batch["mask"]]
+ regularization = valid_delta.square().mean() if valid_delta.numel() else scores.sum() * 0.0
+ loss = ranking_loss + args.delta_l2 * regularization
+ optimizer.zero_grad(set_to_none=True)
+ loss.backward()
+ nn.utils.clip_grad_norm_(model.parameters(), 1.0)
+ optimizer.step()
+ losses.append(float(loss.detach().cpu()))
+ dev_metrics = evaluate_model(
+ model,
+ dev_rows,
+ cache,
+ device,
+ batch_size=args.batch_size,
+ ablation="full",
+ )
+ teacher_dev_rows = [
+ row
+ for row in dev_rows
+ if clean_text((row.get("supervision") or {}).get("target_type")) == "teacher_aligned_turn_bag"
+ ]
+ dev_teacher_metrics = (
+ evaluate_model(
+ model,
+ teacher_dev_rows,
+ cache,
+ device,
+ batch_size=args.batch_size,
+ ablation="full",
+ )
+ if teacher_dev_rows
+ else {"n": 0}
+ )
+ summary = {
+ "epoch": epoch,
+ "loss": round(sum(losses) / max(1, len(losses)), 6),
+ "dev": dev_metrics,
+ "dev_teacher_aligned": dev_teacher_metrics,
+ }
+ history.append(summary)
+ print(json.dumps(summary), flush=True)
+ score_metrics = dev_teacher_metrics if int(dev_teacher_metrics.get("n", 0)) >= 10 else dev_metrics
+ score = (
+ float(score_metrics.get("recall@1", 0.0))
+ + 0.25 * float(score_metrics.get("mrr", 0.0))
+ + 0.05 * float(score_metrics.get("recall@5", 0.0))
+ )
+ if best_score is None or score > best_score:
+ best_score = score
+ best_epoch = epoch
+ stale_epochs = 0
+ atomic_torch_save(
+ {
+ "schema_version": SCHEMA_VERSION,
+ "channel_names": CHANNEL_NAMES,
+ "model_config": {"hidden_dim": args.hidden_dim, "layers": args.layers},
+ "model_state": model.state_dict(),
+ "channel_statistics": {
+ "mean": channel_mean.tolist(),
+ "scale": channel_scale.tolist(),
+ },
+ "epoch": epoch,
+ "dev_metrics": dev_metrics,
+ "dev_teacher_aligned_metrics": dev_teacher_metrics,
+ "cross_cache_manifest": cache.manifest,
+ "train_args": vars(args),
+ },
+ out_dir / "tmcra_v3_reranker.pt",
+ )
+ else:
+ stale_epochs += 1
+ if args.patience > 0 and stale_epochs >= args.patience:
+ print(json.dumps({"status": "early_stop", "epoch": epoch, "best_epoch": best_epoch}), flush=True)
+ break
+ report = {
+ "status": "complete",
+ "schema_version": SCHEMA_VERSION,
+ "train_count": len(train_rows),
+ "dev_count": len(dev_rows),
+ "best_epoch": best_epoch,
+ "best_score": best_score,
+ "init_checkpoint": str(Path(args.init_checkpoint).resolve()) if args.init_checkpoint else "",
+ "channel_statistics": {"mean": channel_mean.tolist(), "scale": channel_scale.tolist()},
+ "selection_metric_scope": "teacher_aligned_dev" if any(
+ int(item.get("dev_teacher_aligned", {}).get("n", 0)) >= 10 for item in history
+ ) else "all_dev",
+ "checkpoint": str(out_dir / "tmcra_v3_reranker.pt"),
+ "history": history,
+ }
+ (out_dir / "train_report.json").write_text(json.dumps(report, indent=2, sort_keys=True), encoding="utf-8")
+ print(json.dumps(report, indent=2, sort_keys=True))
+
+
+def direct_scores(sample: Mapping[str, Any], method: str) -> list[float]:
+ candidates = list(sample["candidates"])
+ if method == "dense":
+ return [float(candidate["channels"]["dense_score"]) for candidate in candidates]
+ if method == "graph":
+ return [float(candidate["channels"]["graph_rank_rr"]) for candidate in candidates]
+ raise RuntimeError(method)
+
+
+def command_eval(args: argparse.Namespace) -> None:
+ rows = read_jsonl(Path(args.samples))
+ if args.qid_list:
+ qids = {clean_text(value) for value in Path(args.qid_list).read_text(encoding="utf-8").splitlines() if clean_text(value)}
+ rows = [row for row in rows if clean_text(row.get("question_id")) in qids]
+ if args.limit > 0:
+ rows = rows[: args.limit]
+ if not rows:
+ raise RuntimeError("no evaluation rows")
+ for row in rows:
+ validate_sample(row, require_positive=False)
+ cache = CrossRepresentationCache(args.cross_cache)
+ device = torch.device("cpu" if args.cpu else "cuda")
+ if device.type == "cuda" and not torch.cuda.is_available():
+ raise RuntimeError("CUDA is unavailable; pass --cpu explicitly")
+ model, checkpoint = load_model_checkpoint(Path(args.checkpoint), cache, device)
+ methods = ("dense", "graph", "base_cross", "fusion", "no_language", "no_channels", "no_graph")
+ totals = {method: defaultdict(float) for method in methods}
+ by_type: dict[str, dict[str, dict[str, float]]] = {
+ method: defaultdict(lambda: defaultdict(float)) for method in methods
+ }
+ by_supervision: dict[str, dict[str, dict[str, float]]] = {
+ method: defaultdict(lambda: defaultdict(float)) for method in methods
+ }
+ predictions: list[dict[str, Any]] = []
+ model.eval()
+ with torch.no_grad():
+ for start in range(0, len(rows), args.batch_size):
+ samples = rows[start : start + args.batch_size]
+ batch = make_batch(samples, cache, device, training=False, max_candidates=0, epoch=0)
+ model_scores = {
+ "base_cross": batch["semantic_logits"].cpu(),
+ "fusion": model(batch["representations"], batch["semantic_logits"], batch["channels"], batch["mask"], ablation="full").cpu(),
+ "no_language": model(batch["representations"], batch["semantic_logits"], batch["channels"], batch["mask"], ablation="no_language").cpu(),
+ "no_channels": model(batch["representations"], batch["semantic_logits"], batch["channels"], batch["mask"], ablation="no_channels").cpu(),
+ "no_graph": model(batch["representations"], batch["semantic_logits"], batch["channels"], batch["mask"], ablation="no_graph").cpu(),
+ }
+ labels = batch["labels"].cpu()
+ mask = batch["mask"].cpu()
+ for row_index, sample in enumerate(samples):
+ qtype = clean_text(sample.get("question_type")) or "unknown"
+ supervision_type = clean_text((sample.get("supervision") or {}).get("target_type")) or "unspecified"
+ valid = mask[row_index]
+ score_map: dict[str, torch.Tensor] = {
+ "dense": torch.tensor(direct_scores(sample, "dense"), dtype=torch.float32),
+ "graph": torch.tensor(direct_scores(sample, "graph"), dtype=torch.float32),
+ }
+ score_map.update({method: values[row_index][valid] for method, values in model_scores.items()})
+ ranks: dict[str, int | None] = {}
+ top_ids: dict[str, list[str]] = {}
+ valid_labels = labels[row_index][valid]
+ positives = set(torch.nonzero(valid_labels > 0.5, as_tuple=False).squeeze(-1).tolist())
+ for method in methods:
+ scores = score_map[method]
+ method_valid = torch.ones_like(scores, dtype=torch.bool)
+ rank_metrics_update(totals[method], scores, valid_labels, method_valid)
+ rank_metrics_update(by_type[method][qtype], scores, valid_labels, method_valid)
+ rank_metrics_update(by_supervision[method][supervision_type], scores, valid_labels, method_valid)
+ order = torch.argsort(scores, descending=True).tolist()
+ ranks[method] = next((position + 1 for position, index in enumerate(order) if index in positives), None)
+ top_ids[method] = [batch["candidate_ids"][row_index][index] for index in order[:10]]
+ predictions.append(
+ {
+ "question_id": sample["question_id"],
+ "question_type": qtype,
+ "supervision_type": supervision_type,
+ "candidate_count": int(valid.sum()),
+ "gold_candidate_count": len(positives),
+ "ranks": ranks,
+ "top10": top_ids,
+ }
+ )
+ out_dir = Path(args.out_dir)
+ out_dir.mkdir(parents=True, exist_ok=False)
+ write_jsonl(out_dir / "predictions.jsonl", predictions)
+ report = {
+ "status": "complete",
+ "created_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
+ "schema_version": SCHEMA_VERSION,
+ "samples": str(Path(args.samples).resolve()),
+ "sample_count": len(rows),
+ "checkpoint": str(Path(args.checkpoint).resolve()),
+ "checkpoint_epoch": checkpoint.get("epoch"),
+ "cross_cache": str(Path(args.cross_cache).resolve()),
+ "metrics": {method: finalize_metrics(totals[method]) for method in methods},
+ "metrics_by_question_type": {
+ method: {qtype: finalize_metrics(values) for qtype, values in sorted(by_type[method].items())}
+ for method in methods
+ },
+ "metrics_by_supervision": {
+ method: {name: finalize_metrics(values) for name, values in sorted(by_supervision[method].items())}
+ for method in methods
+ },
+ }
+ (out_dir / "report.json").write_text(json.dumps(report, indent=2, sort_keys=True), encoding="utf-8")
+ print(json.dumps(report, indent=2, sort_keys=True))
+
+
+def command_export_evidence(args: argparse.Namespace) -> None:
+ rows = read_jsonl(Path(args.samples))
+ if args.qid_list:
+ requested = [
+ clean_text(value)
+ for value in Path(args.qid_list).read_text(encoding="utf-8").splitlines()
+ if clean_text(value)
+ ]
+ by_qid = {clean_text(row.get("question_id")): row for row in rows}
+ missing = [qid for qid in requested if qid not in by_qid]
+ if missing:
+ raise RuntimeError(f"requested qids are missing from samples: {missing[:10]}")
+ rows = [by_qid[qid] for qid in requested]
+ if args.limit > 0:
+ rows = rows[: args.limit]
+ if not rows:
+ raise RuntimeError("no rows for evidence export")
+ for row in rows:
+ validate_sample(row, require_positive=False)
+ cache = CrossRepresentationCache(args.cross_cache)
+ device = torch.device("cpu" if args.cpu else "cuda")
+ if device.type == "cuda" and not torch.cuda.is_available():
+ raise RuntimeError("CUDA is unavailable; pass --cpu explicitly")
+ model, checkpoint = load_model_checkpoint(Path(args.checkpoint), cache, device)
+ model.eval()
+ output_rows: list[dict[str, Any]] = []
+ recall = Counter()
+ with torch.no_grad():
+ for start in range(0, len(rows), args.batch_size):
+ samples = rows[start : start + args.batch_size]
+ batch = make_batch(samples, cache, device, training=False, max_candidates=0, epoch=0)
+ if args.method == "fusion":
+ scores = model(
+ batch["representations"],
+ batch["semantic_logits"],
+ batch["channels"],
+ batch["mask"],
+ ablation="full",
+ ).cpu()
+ elif args.method == "base_cross":
+ scores = batch["semantic_logits"].cpu().masked_fill(~batch["mask"].cpu(), -1e4)
+ elif args.method in {"dense", "graph"}:
+ channel_name = "dense_score" if args.method == "dense" else "graph_rank_rr"
+ scores = torch.full(batch["mask"].shape, -1e4, dtype=torch.float32)
+ for row_index, sample in enumerate(samples):
+ values = [float(candidate["channels"][channel_name]) for candidate in sample["candidates"]]
+ scores[row_index, : len(values)] = torch.tensor(values, dtype=torch.float32)
+ else:
+ raise RuntimeError(f"unsupported evidence method: {args.method}")
+ labels = batch["labels"].cpu()
+ mask = batch["mask"].cpu()
+ for row_index, sample in enumerate(samples):
+ candidates = list(sample["candidates"])
+ valid_count = int(mask[row_index].sum())
+ order = torch.argsort(scores[row_index, :valid_count], descending=True).tolist()
+ selected_indexes: list[int] = []
+ parent_counts: Counter[tuple[int, int]] = Counter()
+ session_counts: Counter[str] = Counter()
+ for candidate_index in order:
+ candidate = candidates[candidate_index]
+ parent_key = (int(candidate["session_index"]), int(candidate.get("parent_chunk_index", 0)))
+ session_id = clean_text(candidate.get("session_id"))
+ if args.max_per_parent > 0 and parent_counts[parent_key] >= args.max_per_parent:
+ continue
+ if args.max_per_session > 0 and session_counts[session_id] >= args.max_per_session:
+ continue
+ selected_indexes.append(candidate_index)
+ parent_counts[parent_key] += 1
+ session_counts[session_id] += 1
+ if len(selected_indexes) >= args.top_k:
+ break
+ positives = set(
+ torch.nonzero(labels[row_index, :valid_count] > 0.5, as_tuple=False).squeeze(-1).tolist()
+ )
+ recall["n"] += 1
+ recall["hit"] += int(any(index in positives for index in selected_indexes))
+ supervision_type = clean_text((sample.get("supervision") or {}).get("target_type")) or "unspecified"
+ recall[f"n:{supervision_type}"] += 1
+ recall[f"hit:{supervision_type}"] += int(any(index in positives for index in selected_indexes))
+ evidence_windows = []
+ for rank, candidate_index in enumerate(selected_indexes, start=1):
+ candidate = candidates[candidate_index]
+ evidence_windows.append(
+ {
+ "memory_id": candidate["candidate_id"],
+ "session_id": candidate["session_id"],
+ "session_index": candidate["session_index"],
+ "parent_chunk_index": candidate.get("parent_chunk_index", 0),
+ "subchunk_index": candidate.get("subchunk_index", 0),
+ "score": round(float(scores[row_index, candidate_index]), 6),
+ "rank": rank,
+ "text": candidate["text"],
+ "channels": candidate["channels"],
+ }
+ )
+ output_rows.append(
+ {
+ "schema_version": SCHEMA_VERSION,
+ "question_id": sample["question_id"],
+ "question": sample["question"],
+ "question_date": sample.get("question_date", ""),
+ "question_type": sample.get("question_type", ""),
+ "gold_answer": sample.get("gold_answer", ""),
+ "answer_session_ids": sample.get("answer_session_ids", []),
+ "selected_session_ids": [window["session_id"] for window in evidence_windows],
+ "evidence_windows": evidence_windows,
+ }
+ )
+ out_dir = Path(args.out_dir)
+ out_dir.mkdir(parents=True, exist_ok=False)
+ write_jsonl(out_dir / "evidence_windows.jsonl", output_rows)
+ report = {
+ "status": "complete",
+ "created_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
+ "schema_version": SCHEMA_VERSION,
+ "samples": str(Path(args.samples).resolve()),
+ "sample_count": len(output_rows),
+ "top_k": args.top_k,
+ "method": args.method,
+ "max_per_parent": args.max_per_parent,
+ "max_per_session": args.max_per_session,
+ "evidence_recall": round(float(recall["hit"]) / max(1, int(recall["n"])), 6),
+ "evidence_recall_by_supervision": {
+ name.removeprefix("n:"): round(
+ float(recall[f"hit:{name.removeprefix('n:')}"]) / max(1, int(count)), 6
+ )
+ for name, count in sorted(recall.items())
+ if name.startswith("n:")
+ },
+ "checkpoint": str(Path(args.checkpoint).resolve()),
+ "checkpoint_epoch": checkpoint.get("epoch"),
+ "output": str(out_dir / "evidence_windows.jsonl"),
+ }
+ (out_dir / "report.json").write_text(json.dumps(report, indent=2, sort_keys=True), encoding="utf-8")
+ print(json.dumps(report, indent=2, sort_keys=True))
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser()
+ sub = parser.add_subparsers(dest="command", required=True)
+ precompute = sub.add_parser("precompute-cross")
+ precompute.add_argument("--samples", required=True, nargs="+")
+ precompute.add_argument("--model", required=True)
+ precompute.add_argument("--out-dir", required=True)
+ precompute.add_argument("--max-length", type=int, default=1280)
+ precompute.add_argument("--batch-size", type=int, default=4)
+ precompute.add_argument("--shard-size", type=int, default=500)
+ precompute.add_argument("--cpu", action="store_true")
+ train = sub.add_parser("train")
+ train.add_argument("--train-samples", required=True)
+ train.add_argument("--dev-samples", default="")
+ train.add_argument("--cross-cache", required=True)
+ train.add_argument("--out-dir", required=True)
+ train.add_argument("--init-checkpoint", default="")
+ train.add_argument("--hidden-dim", type=int, default=256)
+ train.add_argument("--layers", type=int, default=2)
+ train.add_argument("--batch-size", type=int, default=8)
+ train.add_argument("--max-train-candidates", type=int, default=32)
+ train.add_argument("--dev-count", type=int, default=40)
+ train.add_argument("--epochs", type=int, default=16)
+ train.add_argument("--lr", type=float, default=2e-4)
+ train.add_argument("--weight-decay", type=float, default=0.01)
+ train.add_argument("--delta-l2", type=float, default=0.002)
+ train.add_argument("--patience", type=int, default=4)
+ train.add_argument("--seed", type=int, default=31)
+ train.add_argument("--cpu", action="store_true")
+ evaluate = sub.add_parser("eval")
+ evaluate.add_argument("--samples", required=True)
+ evaluate.add_argument("--cross-cache", required=True)
+ evaluate.add_argument("--checkpoint", required=True)
+ evaluate.add_argument("--out-dir", required=True)
+ evaluate.add_argument("--batch-size", type=int, default=8)
+ evaluate.add_argument("--limit", type=int, default=0)
+ evaluate.add_argument("--qid-list", default="")
+ evaluate.add_argument("--cpu", action="store_true")
+ export = sub.add_parser("export-evidence")
+ export.add_argument("--samples", required=True)
+ export.add_argument("--cross-cache", required=True)
+ export.add_argument("--checkpoint", required=True)
+ export.add_argument("--out-dir", required=True)
+ export.add_argument("--top-k", type=int, default=8)
+ export.add_argument("--method", choices=("dense", "graph", "base_cross", "fusion"), default="fusion")
+ export.add_argument("--max-per-parent", type=int, default=0)
+ export.add_argument("--max-per-session", type=int, default=0)
+ export.add_argument("--batch-size", type=int, default=8)
+ export.add_argument("--limit", type=int, default=0)
+ export.add_argument("--qid-list", default="")
+ export.add_argument("--cpu", action="store_true")
+ args = parser.parse_args()
+ if args.command == "precompute-cross":
+ command_precompute_cross(args)
+ elif args.command == "train":
+ command_train(args)
+ elif args.command == "eval":
+ command_eval(args)
+ else:
+ command_export_evidence(args)
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v3_schema.py b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v3_schema.py
new file mode 100644
index 0000000000000000000000000000000000000000..ff641b2588d82b84662ebaf0fc090c86dc12df4a
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v3_schema.py
@@ -0,0 +1,168 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import json
+import math
+from pathlib import Path
+from typing import Any, Mapping, Sequence
+
+
+SCHEMA_VERSION = "tmcra.memory_recall.v3.0"
+CHANNEL_NAMES = (
+ "dense_score",
+ "dense_rank_rr",
+ "graph_rank_rr",
+ "graph_selected",
+ "graph_final",
+ "recency_norm",
+)
+FORBIDDEN_CHANNEL_TOKENS = ("label", "positive", "negative", "hard", "gold", "answer")
+
+
+def clean_text(value: Any) -> str:
+ return " ".join(str(value or "").split())
+
+
+def read_jsonl(path: Path) -> list[dict[str, Any]]:
+ if not path.exists():
+ raise FileNotFoundError(path)
+ rows: list[dict[str, Any]] = []
+ with path.open("r", encoding="utf-8", errors="strict") as handle:
+ for line_no, line in enumerate(handle, start=1):
+ if not line.strip():
+ continue
+ try:
+ value = json.loads(line)
+ except json.JSONDecodeError as exc:
+ raise RuntimeError(f"invalid jsonl at {path}:{line_no}: {exc}") from exc
+ if not isinstance(value, dict):
+ raise RuntimeError(f"jsonl row is not an object at {path}:{line_no}")
+ rows.append(value)
+ if not rows:
+ raise RuntimeError(f"jsonl is empty: {path}")
+ return rows
+
+
+def write_jsonl(path: Path, rows: Sequence[Mapping[str, Any]]) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ with path.open("w", encoding="utf-8") as handle:
+ for row in rows:
+ handle.write(json.dumps(row, ensure_ascii=False, sort_keys=True) + "\n")
+
+
+def channel_vector(candidate: Mapping[str, Any]) -> list[float]:
+ channels = candidate.get("channels")
+ if not isinstance(channels, Mapping):
+ raise RuntimeError("candidate.channels must be an object")
+ return [float(channels[name]) for name in CHANNEL_NAMES]
+
+
+def _require_finite(value: Any, context: str) -> float:
+ try:
+ number = float(value)
+ except (TypeError, ValueError) as exc:
+ raise RuntimeError(f"{context} is not numeric: {value!r}") from exc
+ if not math.isfinite(number):
+ raise RuntimeError(f"{context} is not finite: {number!r}")
+ return number
+
+
+def validate_candidate(candidate: Mapping[str, Any], *, context: str) -> None:
+ candidate_id = clean_text(candidate.get("candidate_id"))
+ text = clean_text(candidate.get("text"))
+ if not candidate_id:
+ raise RuntimeError(f"{context}: candidate_id is required")
+ if not text:
+ raise RuntimeError(f"{context}: text is required")
+ channels = candidate.get("channels")
+ if not isinstance(channels, Mapping):
+ raise RuntimeError(f"{context}: channels must be an object")
+ if len(channels) != len(CHANNEL_NAMES) or set(channels) != set(CHANNEL_NAMES):
+ raise RuntimeError(
+ f"{context}: channel keys must exactly equal {CHANNEL_NAMES}, got {tuple(channels.keys())}"
+ )
+ for name, value in channels.items():
+ lowered = name.lower()
+ if any(token in lowered for token in FORBIDDEN_CHANNEL_TOKENS):
+ raise RuntimeError(f"{context}: label-derived channel name is forbidden: {name}")
+ _require_finite(value, f"{context}.channels.{name}")
+ labels = candidate.get("labels")
+ if not isinstance(labels, Mapping):
+ raise RuntimeError(f"{context}: labels must be an object")
+ relevance = bool(labels.get("relevance", False))
+ hard_negative = bool(labels.get("hard_negative", False))
+ if relevance and hard_negative:
+ raise RuntimeError(f"{context}: a positive candidate cannot be a hard negative")
+ role = clean_text(labels.get("evidence_role"))
+ expected_role = "positive" if relevance else ("hard_negative" if hard_negative else "negative")
+ if role != expected_role:
+ raise RuntimeError(f"{context}: evidence_role={role!r}, expected {expected_role!r}")
+ target_scope = clean_text(labels.get("target_scope"))
+ if target_scope and target_scope != "answer_session_bag":
+ raise RuntimeError(f"{context}: unsupported target_scope={target_scope!r}")
+
+
+def validate_sample(
+ sample: Mapping[str, Any],
+ *,
+ require_positive: bool,
+ allowed_splits: Sequence[str] = ("train", "holdout", "full_eval"),
+) -> None:
+ if clean_text(sample.get("schema_version")) != SCHEMA_VERSION:
+ raise RuntimeError(f"unsupported schema_version: {sample.get('schema_version')!r}")
+ qid = clean_text(sample.get("question_id"))
+ query = clean_text(sample.get("query_text"))
+ split = clean_text(sample.get("split"))
+ if not qid or not query:
+ raise RuntimeError("question_id and query_text are required")
+ if split not in set(allowed_splits):
+ raise RuntimeError(f"invalid split for {qid}: {split!r}")
+ supervision = sample.get("supervision")
+ if supervision is not None:
+ if not isinstance(supervision, Mapping):
+ raise RuntimeError(f"{qid}: supervision must be an object")
+ if clean_text(supervision.get("target_type")) not in {
+ "multi_instance_answer_session_bag",
+ "teacher_aligned_turn_bag",
+ }:
+ raise RuntimeError(f"{qid}: unsupported supervision target_type")
+ if clean_text(supervision.get("loss")) != "negative_log_positive_probability_mass":
+ raise RuntimeError(f"{qid}: unsupported supervision loss")
+ weight = _require_finite(supervision.get("training_weight", 1.0), f"{qid}.supervision.training_weight")
+ if weight <= 0.0 or weight > 1.0:
+ raise RuntimeError(f"{qid}: supervision training_weight must be in (0, 1]")
+ candidates = sample.get("candidates")
+ if not isinstance(candidates, list) or not candidates:
+ raise RuntimeError(f"{qid}: candidates must be a non-empty list")
+ seen: set[str] = set()
+ positive_count = 0
+ for index, candidate in enumerate(candidates):
+ if not isinstance(candidate, Mapping):
+ raise RuntimeError(f"{qid}: candidate {index} is not an object")
+ validate_candidate(candidate, context=f"{qid}.candidates[{index}]")
+ candidate_id = clean_text(candidate.get("candidate_id"))
+ if candidate_id in seen:
+ raise RuntimeError(f"{qid}: duplicate candidate_id: {candidate_id}")
+ seen.add(candidate_id)
+ positive_count += int(bool(candidate["labels"].get("relevance", False)))
+ if require_positive and positive_count == 0:
+ raise RuntimeError(f"{qid}: no positive candidate")
+
+
+def validate_split_isolation(train_rows: Sequence[Mapping[str, Any]], holdout_rows: Sequence[Mapping[str, Any]]) -> dict[str, int]:
+ train_qids = {clean_text(row.get("question_id")) for row in train_rows}
+ holdout_qids = {clean_text(row.get("question_id")) for row in holdout_rows}
+ train_queries = {clean_text(row.get("query_text")) for row in train_rows}
+ holdout_queries = {clean_text(row.get("query_text")) for row in holdout_rows}
+ qid_overlap = train_qids & holdout_qids
+ query_overlap = train_queries & holdout_queries
+ if qid_overlap:
+ raise RuntimeError(f"train/holdout qid overlap: {sorted(qid_overlap)[:10]}")
+ if query_overlap:
+ raise RuntimeError(f"train/holdout query overlap: {sorted(query_overlap)[:3]}")
+ return {
+ "train_qids": len(train_qids),
+ "holdout_qids": len(holdout_qids),
+ "qid_overlap": 0,
+ "query_text_overlap": 0,
+ }
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v3_slow_graph.py b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v3_slow_graph.py
new file mode 100644
index 0000000000000000000000000000000000000000..f2143127f1e7709ffbe72a05e19efcac1b1c2ff9
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v3_slow_graph.py
@@ -0,0 +1,2138 @@
+#!/usr/bin/env python3
+"""Strict, append-only slow-memory graph controller."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import importlib
+import json
+import os
+import sqlite3
+import sys
+import threading
+import time
+import urllib.error
+import urllib.request
+import uuid
+from contextlib import contextmanager
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Iterable, Mapping, Protocol
+
+LEAF_VARIANT = "product_semantic_memory"
+CAPSULE_VARIANT = "slow_memory_capsule"
+PATCH_ACTIONS = frozenset(
+ {
+ "create",
+ "revise",
+ "challenge",
+ "resolve_challenge",
+ "retire",
+ "noop",
+ }
+)
+EDGE_TYPES = frozenset(
+ {
+ "supports",
+ "contradicts",
+ "derived_from",
+ "supersedes",
+ "challenges",
+ "invalidates",
+ }
+)
+SCHEMA_VERSION = "slow-graph-patch/v4"
+SLOW_EDGE_SOURCE = "slow_graph_control_plane"
+DEFAULT_CLAIM_LEASE_SECONDS = 15 * 60
+
+
+class SlowGraphError(RuntimeError):
+ pass
+
+
+class PatchValidationError(SlowGraphError):
+ pass
+
+
+class EvidencePolicyError(SlowGraphError):
+ pass
+
+
+class StaleRevisionError(SlowGraphError):
+ pass
+
+
+class AuditError(SlowGraphError):
+ pass
+
+
+class DeepSeekCallError(SlowGraphError):
+ def __init__(self, message: str, *, retryable: bool) -> None:
+ super().__init__(message)
+ self.retryable = retryable
+
+
+@dataclass(frozen=True)
+class JobClaim:
+ job_id: str
+ attempt_id: str
+ token: str
+ owner: str
+
+
+def _now() -> int:
+ return int(time.time())
+
+
+def _clean(value: Any) -> str:
+ return str(value).strip() if value is not None else ""
+
+
+def _json(value: Any) -> str:
+ return json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
+
+
+def _digest(value: Any) -> str:
+ return hashlib.sha256(_json(value).encode("utf-8")).hexdigest()
+
+
+def _strict_json(value: str, *, label: str, expected: type) -> Any:
+ try:
+ result = json.loads(value)
+ except (TypeError, json.JSONDecodeError) as exc:
+ raise SlowGraphError(f"invalid {label} JSON") from exc
+ if not isinstance(result, expected):
+ raise SlowGraphError(f"{label} JSON must be {expected.__name__}")
+ return result
+
+
+def _required_text(value: Any, label: str) -> str:
+ text = _clean(value)
+ if not text:
+ raise PatchValidationError(f"{label} is required")
+ return text
+
+
+def load_graph_schema(repo: str | Path) -> tuple[type[Any], type[Any]]:
+ """Load the real graph record and edge types from an explicit TMCRA repo."""
+ root = Path(repo).resolve()
+ candidates = [root, root / "tmcra_code"]
+ package_roots = [
+ candidate
+ for candidate in candidates
+ if (candidate / "experiments" / "replacement" / "memory_graph.py").is_file()
+ ]
+ if len(package_roots) != 1:
+ raise SlowGraphError(
+ "--repo must resolve exactly one experiments/replacement/memory_graph.py "
+ f"at the repo root or repo/tmcra_code: {root}"
+ )
+ package_root = package_roots[0]
+ module_path = package_root / "experiments" / "replacement" / "memory_graph.py"
+ if str(package_root) not in sys.path:
+ sys.path.insert(0, str(package_root))
+ importlib.invalidate_caches()
+ try:
+ module = importlib.import_module("experiments.replacement.memory_graph")
+ return module.SessionMemoryRecordV2, module.SessionMemoryEdgeV2
+ except (ImportError, AttributeError) as exc:
+ raise SlowGraphError(
+ "unable to import real SessionMemoryRecordV2/SessionMemoryEdgeV2"
+ ) from exc
+
+
+class PatchManager(Protocol):
+ model_config: Mapping[str, Any]
+ prompt_hash: str
+ last_call_metadata: Mapping[str, Any]
+
+ def propose(
+ self, region: Mapping[str, Any], capsules: list[Mapping[str, Any]]
+ ) -> Mapping[str, Any]: ...
+
+
+@dataclass(frozen=True)
+class DeepSeekProConfig:
+ base_url: str
+ key_pool: tuple[str, ...]
+ max_tokens: int
+ model: str = "deepseek-v4-pro"
+
+ @classmethod
+ def from_env(cls) -> "DeepSeekProConfig":
+ base_url = _clean(os.getenv("TMCRA_DEEPSEEK_PRO_BASE_URL"))
+ keys = tuple(
+ item.strip()
+ for item in _clean(os.getenv("TMCRA_DEEPSEEK_PRO_KEY_POOL")).split(",")
+ if item.strip()
+ )
+ try:
+ max_tokens = int(_clean(os.getenv("TMCRA_DEEPSEEK_PRO_MAX_TOKENS")))
+ except ValueError as exc:
+ raise SlowGraphError(
+ "TMCRA_DEEPSEEK_PRO_MAX_TOKENS must be an integer"
+ ) from exc
+ if not base_url or not keys or max_tokens <= 0:
+ raise SlowGraphError(
+ "DeepSeek requires BASE_URL, KEY_POOL, and positive MAX_TOKENS"
+ )
+ return cls(base_url.rstrip("/"), keys, max_tokens)
+
+
+class DeepSeekProGraphPatchManager:
+ """The only production patch manager: no fallback and no hidden defaults."""
+
+ def __init__(self, config: DeepSeekProConfig) -> None:
+ if config.model != "deepseek-v4-pro":
+ raise SlowGraphError("DeepSeek model must be deepseek-v4-pro")
+ self.config = config
+ self._key_index = 0
+ self.model_config = {
+ "model": config.model,
+ "temperature": 0,
+ "thinking": "disabled",
+ "max_tokens": config.max_tokens,
+ }
+ self.prompt_hash = _digest(self._messages("", []))
+ self.last_call_metadata: Mapping[str, Any] = {}
+
+ def _messages(self, region: Any, capsules: Any) -> list[dict[str, str]]:
+ schema = {
+ "operations": [
+ {
+ "action": "create|revise|challenge|resolve_challenge|retire|noop",
+ "capsule_id": "required for non-create operations; omit for create because controller derives it from region_key",
+ "base_revision": "required for mutations of an existing capsule",
+ "claims": [
+ {
+ "canonical_slot": "stable semantic slot shared by corrections of the same durable fact",
+ "text": "string",
+ "support": ["fast id"],
+ "counterevidence": ["fast id"],
+ }
+ ],
+ }
+ ]
+ }
+ return [
+ {
+ "role": "system",
+ "content": (
+ "You manage only the slow durable layer above immutable fast evidence leaves. "
+ "Return exactly one JSON GraphPatch and no prose. Create or revise a capsule only for "
+ "a durable preference, identity fact, routine, relationship, standing constraint, or stable "
+ "long-running state. A single explicit durable assertion is sufficient; never require an "
+ "arbitrary repetition count. Transient events, one-off tasks, quoted third-party claims, "
+ "and evidence that does not establish durable memory must produce an empty operations list. "
+ "Use challenge when new fast evidence conflicts but does not resolve the old claim; use "
+ "resolve_challenge when the supplied evidence resolves an existing challenge; use retire only "
+ "when evidence establishes that a capsule is no longer applicable. Topology merge and split "
+ "are not part of this regional V1 controller. canonical_slot must remain identical across corrections "
+ "of the same fact. Every claim canonical_slot must exactly copy canonical_slot_key from at "
+ "least one fast evidence leaf cited by that claim. Do not invent evidence IDs, capsule IDs "
+ "for create, confidence, source "
+ "parents, claim_id, or fields outside the supplied schema. The controller assigns claim_id. "
+ "Always emit support and counterevidence arrays, using [] for the empty side; the controller "
+ "also accepts an omitted empty side as a transport-level normalization. Every non-noop claim "
+ "must cite supplied fast leaf evidence. record_state and slow_graph_evidence_role are "
+ "controller-owned. Never promote historical_noncurrent evidence as a current claim when "
+ "current_authoritative evidence exists for the same canonical_slot; use noncurrent evidence "
+ "only as history or counterevidence. Each region has exactly one controller-owned capsule: "
+ "emit create only when "
+ "capsules is empty; otherwise mutate the supplied capsule_id or emit an empty operations list. "
+ "Return at most one operation. Empty operations is the canonical region-level noop."
+ ),
+ },
+ {
+ "role": "user",
+ "content": _json(
+ {
+ "schema_version": SCHEMA_VERSION,
+ "schema": schema,
+ "region": region,
+ "capsules": capsules,
+ }
+ ),
+ },
+ ]
+
+ def propose(
+ self, region: Mapping[str, Any], capsules: list[Mapping[str, Any]]
+ ) -> Mapping[str, Any]:
+ key_index = self._key_index % len(self.config.key_pool)
+ key = self.config.key_pool[key_index]
+ self._key_index += 1
+ body = {
+ "model": "deepseek-v4-pro",
+ "temperature": 0,
+ "max_tokens": self.config.max_tokens,
+ "thinking": {"type": "disabled"},
+ "enable_thinking": False,
+ "response_format": {"type": "json_object"},
+ "messages": self._messages(region, capsules),
+ }
+ request = urllib.request.Request(
+ self.config.base_url + "/chat/completions",
+ data=_json(body).encode("utf-8"),
+ headers={
+ "Authorization": "Bearer " + key,
+ "Content-Type": "application/json",
+ },
+ method="POST",
+ )
+ physical_call_id = "dsc_" + uuid.uuid4().hex
+ started_at = _now()
+ self.last_call_metadata = {
+ "request": body,
+ "key_index": key_index,
+ "started_at": started_at,
+ "physical_call_id": physical_call_id,
+ "status": "started",
+ }
+ try:
+ with urllib.request.urlopen(request, timeout=90) as response: # nosec B310
+ status = response.getcode()
+ raw_text = response.read().decode("utf-8")
+ except urllib.error.HTTPError as exc:
+ try:
+ detail = exc.read().decode("utf-8", "replace")
+ except (AttributeError, OSError):
+ detail = ""
+ self.last_call_metadata = {
+ **self.last_call_metadata,
+ "status": "http_error",
+ "http_status": exc.code,
+ "error": detail,
+ "completed_at": _now(),
+ }
+ raise DeepSeekCallError(
+ f"DeepSeek HTTP {exc.code}: {detail}",
+ retryable=exc.code == 429 or exc.code >= 500,
+ ) from exc
+ except (urllib.error.URLError, TimeoutError, OSError) as exc:
+ self.last_call_metadata = {
+ **self.last_call_metadata,
+ "status": "request_error",
+ "error": f"{exc.__class__.__name__}: {exc}",
+ "completed_at": _now(),
+ }
+ raise DeepSeekCallError(
+ f"DeepSeek transport failure: {exc}", retryable=True
+ ) from exc
+ if status < 200 or status >= 300:
+ self.last_call_metadata = {
+ **self.last_call_metadata,
+ "status": "unexpected_http_status",
+ "http_status": status,
+ "raw_response": raw_text,
+ "completed_at": _now(),
+ }
+ raise DeepSeekCallError(
+ f"DeepSeek returned HTTP {status}", retryable=status >= 500
+ )
+ self.last_call_metadata = {
+ **self.last_call_metadata,
+ "status": "response_received",
+ "http_status": status,
+ "raw_response": raw_text,
+ "completed_at": _now(),
+ }
+ raw = _strict_json(raw_text, label="DeepSeek response", expected=dict)
+ choices = raw.get("choices")
+ usage = raw.get("usage")
+ if (
+ not isinstance(choices, list)
+ or len(choices) != 1
+ or not isinstance(usage, Mapping)
+ ):
+ raise DeepSeekCallError(
+ "DeepSeek response missing strict choices/usage", retryable=False
+ )
+ choice = choices[0]
+ if not isinstance(choice, Mapping) or not _clean(choice.get("finish_reason")):
+ raise DeepSeekCallError(
+ "DeepSeek response missing finish_reason", retryable=False
+ )
+ message = choice.get("message")
+ if not isinstance(message, Mapping) or not _clean(message.get("content")):
+ raise DeepSeekCallError(
+ "DeepSeek response missing choices[0].message.content", retryable=False
+ )
+ finish_reason = _clean(choice["finish_reason"])
+ self.last_call_metadata = {
+ **self.last_call_metadata,
+ "finish_reason": finish_reason,
+ "content": _clean(message["content"]),
+ "usage": dict(usage),
+ }
+ if finish_reason != "stop":
+ self.last_call_metadata = {
+ **self.last_call_metadata,
+ "status": "incomplete_response",
+ }
+ raise DeepSeekCallError(
+ f"DeepSeek response finish_reason must be stop, got {finish_reason!r}",
+ retryable=False,
+ )
+ self.last_call_metadata = {
+ **self.last_call_metadata,
+ "request": body,
+ "response_id": _required_text(raw.get("id"), "DeepSeek response id"),
+ "finish_reason": finish_reason,
+ "choices": choices,
+ "content": _clean(message["content"]),
+ "usage": dict(usage),
+ "http_status": status,
+ "physical_call_id": physical_call_id,
+ "status": "completed",
+ "completed_at": _now(),
+ }
+ patch = _strict_json(
+ _clean(message["content"]), label="DeepSeek content", expected=dict
+ )
+ transport_normalizations = []
+ for operation in patch.get("operations", []):
+ if (
+ isinstance(operation, Mapping)
+ and operation.get("action") == "create"
+ and "capsule_id" in operation
+ and operation.get("capsule_id") is None
+ ):
+ transport_normalizations.append(
+ {
+ "code": "create_null_capsule_id_ignored",
+ "field": "capsule_id",
+ }
+ )
+ if transport_normalizations:
+ self.last_call_metadata = {
+ **self.last_call_metadata,
+ "transport_normalizations": transport_normalizations,
+ }
+ validate_patch(patch)
+ return patch
+
+
+def _ids(value: Any, label: str) -> list[str]:
+ if not isinstance(value, list) or not value:
+ raise PatchValidationError(f"{label} must be a non-empty list")
+ result = [_required_text(item, label) for item in value]
+ if len(set(result)) != len(result):
+ raise PatchValidationError(f"{label} contains duplicates")
+ return result
+
+
+def _validate_source_parents(value: Any) -> list[dict[str, Any]]:
+ if not isinstance(value, list) or not value:
+ raise PatchValidationError("source_parents must be a non-empty list")
+ expected = {
+ "session_index",
+ "parent_chunk_index",
+ "message_index",
+ "source_record_id",
+ "event_id",
+ "evidence_char_start",
+ "evidence_char_end",
+ }
+ result: list[dict[str, Any]] = []
+ seen: set[str] = set()
+ for parent in value:
+ if not isinstance(parent, Mapping) or set(parent) != expected:
+ raise PatchValidationError("source_parent has an invalid schema")
+ normalized = dict(parent)
+ for key in ("session_index", "parent_chunk_index", "message_index"):
+ if not isinstance(normalized[key], int) or normalized[key] < 0:
+ raise PatchValidationError(f"source_parent {key} must be non-negative integer")
+ if normalized["parent_chunk_index"] != normalized["message_index"]:
+ raise PatchValidationError("source_parent chunk/message coordinates disagree")
+ if (
+ not isinstance(normalized["evidence_char_start"], int)
+ or not isinstance(normalized["evidence_char_end"], int)
+ or normalized["evidence_char_start"] < 0
+ or normalized["evidence_char_end"] <= normalized["evidence_char_start"]
+ ):
+ raise PatchValidationError("source_parent evidence character span is invalid")
+ _required_text(normalized["source_record_id"], "source_record_id")
+ _required_text(normalized["event_id"], "event_id")
+ digest = _json(normalized)
+ if digest not in seen:
+ seen.add(digest)
+ result.append(normalized)
+ return result
+
+
+def _validate_claims(
+ value: Any, *, stored: bool = False
+) -> list[dict[str, Any]]:
+ if not isinstance(value, list) or not value:
+ raise PatchValidationError("claims must be a non-empty list")
+ claims: list[dict[str, Any]] = []
+ seen = set()
+ for claim in value:
+ expected = {"canonical_slot", "text", "support", "counterevidence"}
+ if stored:
+ expected.add("claim_id")
+ expected.add("source_parents")
+ if not isinstance(claim, Mapping):
+ raise PatchValidationError("each claim must be an object")
+ if stored:
+ valid_shape = set(claim) == expected
+ else:
+ valid_shape = (
+ {"canonical_slot", "text"}.issubset(claim)
+ and set(claim).issubset(expected)
+ )
+ if not valid_shape:
+ raise PatchValidationError(
+ "each claim has an invalid schema"
+ )
+ support = claim.get("support", [])
+ counter = claim.get("counterevidence", [])
+ if not isinstance(support, list) or not isinstance(counter, list):
+ raise PatchValidationError("claim evidence must be lists")
+ normalized_support = _ids(support, "claim support") if support else []
+ normalized_counter = (
+ _ids(counter, "claim counterevidence") if counter else []
+ )
+ if not normalized_support and not normalized_counter:
+ raise PatchValidationError("each claim needs support or counterevidence")
+ canonical_slot = _required_text(
+ claim.get("canonical_slot"), "canonical_slot"
+ )
+ claim_text = _required_text(claim.get("text"), "claim text")
+ claim_id = (
+ _required_text(claim.get("claim_id"), "claim_id")
+ if stored
+ else "clm_"
+ + _digest(
+ {
+ "canonical_slot": canonical_slot,
+ "text": claim_text,
+ "support": normalized_support,
+ "counterevidence": normalized_counter,
+ }
+ )[:24]
+ )
+ if claim_id in seen:
+ raise PatchValidationError("claim_id must be unique")
+ seen.add(claim_id)
+ claims.append(
+ {
+ "claim_id": claim_id,
+ "canonical_slot": canonical_slot,
+ "text": claim_text,
+ "support": normalized_support,
+ "counterevidence": normalized_counter,
+ **(
+ {
+ "source_parents": _validate_source_parents(
+ claim.get("source_parents")
+ )
+ }
+ if stored
+ else {}
+ ),
+ }
+ )
+ return claims
+
+
+def validate_patch(patch: Mapping[str, Any]) -> None:
+ if (
+ not isinstance(patch, Mapping)
+ or set(patch) != {"operations"}
+ or not isinstance(patch["operations"], list)
+ ):
+ raise PatchValidationError("GraphPatch must contain exactly an operations list")
+ if len(patch["operations"]) > 1:
+ raise PatchValidationError("GraphPatch may contain at most one region operation")
+ for operation in patch["operations"]:
+ if not isinstance(operation, Mapping):
+ raise PatchValidationError("GraphPatch operation must be an object")
+ action = _required_text(operation.get("action"), "action")
+ if action not in PATCH_ACTIONS:
+ raise PatchValidationError("unknown GraphPatch action")
+ if "confidence" in operation:
+ raise PatchValidationError(
+ "model confidence is not an authoritative graph field"
+ )
+ allowed = {"action", "summary", "claims"}
+ if action == "create":
+ if "capsule_key" in operation:
+ raise PatchValidationError(
+ "create capsule identity is controller-derived from region_key"
+ )
+ if "capsule_id" in operation:
+ if operation.get("capsule_id") is not None:
+ raise PatchValidationError(
+ "create capsule identity is controller-derived from region_key"
+ )
+ allowed.add("capsule_id")
+ elif action == "noop":
+ allowed.add("capsule_id")
+ if "capsule_id" in operation:
+ _required_text(operation.get("capsule_id"), "capsule_id")
+ else:
+ allowed.update({"capsule_id", "base_revision"})
+ _required_text(operation.get("capsule_id"), "capsule_id")
+ if (
+ not isinstance(operation.get("base_revision"), int)
+ or operation["base_revision"] < 1
+ ):
+ raise PatchValidationError("base_revision must be a positive integer")
+ if set(operation) - allowed:
+ raise PatchValidationError("unexpected GraphPatch operation fields")
+ if action != "noop":
+ _validate_claims(operation.get("claims"))
+
+
+class SlowGraphStore:
+ def __init__(
+ self,
+ database: str | Path,
+ *,
+ schema: tuple[type[Any], type[Any]],
+ claim_lease_seconds: int = DEFAULT_CLAIM_LEASE_SECONDS,
+ ) -> None:
+ if claim_lease_seconds <= 0:
+ raise SlowGraphError("claim lease duration must be positive")
+ self.database = Path(database)
+ self.record_type, self.edge_type = schema
+ self.claim_lease_seconds = claim_lease_seconds
+ self.database.parent.mkdir(parents=True, exist_ok=True)
+ self._init_schema()
+
+ def connect(self) -> sqlite3.Connection:
+ con = sqlite3.connect(self.database)
+ con.row_factory = sqlite3.Row
+ con.execute("PRAGMA foreign_keys=ON")
+ return con
+
+ @contextmanager
+ def connection(self):
+ con = self.connect()
+ try:
+ yield con
+ con.commit()
+ except Exception:
+ con.rollback()
+ raise
+ finally:
+ con.close()
+
+ def _init_schema(self) -> None:
+ with self.connection() as con:
+ con.executescript("""
+ CREATE TABLE IF NOT EXISTS slow_graph_jobs (
+ job_id TEXT PRIMARY KEY, idempotency_key TEXT NOT NULL UNIQUE, scope_id TEXT NOT NULL, region_key TEXT NOT NULL,
+ evidence_ids_json TEXT NOT NULL, metadata_json TEXT NOT NULL, status TEXT NOT NULL CHECK(status IN ('pending','retryable','failed','completed')),
+ attempts INTEGER NOT NULL, last_error TEXT NOT NULL, created_at INTEGER NOT NULL, updated_at INTEGER NOT NULL,
+ claim_token TEXT, claim_owner TEXT, lease_expires_at INTEGER);
+ CREATE TABLE IF NOT EXISTS slow_graph_attempts (
+ attempt_id TEXT PRIMARY KEY, job_id TEXT NOT NULL, scope_id TEXT NOT NULL, status TEXT NOT NULL,
+ call_metadata_json TEXT NOT NULL, error TEXT NOT NULL, created_at INTEGER NOT NULL, completed_at INTEGER,
+ claim_token TEXT, claim_owner TEXT);
+ CREATE TABLE IF NOT EXISTS slow_graph_batches (
+ batch_id TEXT PRIMARY KEY, batch_key TEXT NOT NULL UNIQUE, scope_id TEXT NOT NULL,
+ evidence_snapshot_hash TEXT NOT NULL, job_ids_json TEXT NOT NULL, manager_metadata_json TEXT NOT NULL,
+ created_at INTEGER NOT NULL);
+ CREATE TABLE IF NOT EXISTS slow_graph_patches (
+ patch_id TEXT PRIMARY KEY, job_id TEXT NOT NULL, scope_id TEXT NOT NULL, region_key TEXT NOT NULL, manager_model TEXT NOT NULL,
+ patch_json TEXT NOT NULL, call_metadata_json TEXT NOT NULL, applied_at INTEGER NOT NULL);
+ CREATE TABLE IF NOT EXISTS slow_graph_patch_operations (
+ operation_id TEXT PRIMARY KEY, patch_id TEXT NOT NULL, ordinal INTEGER NOT NULL, capsule_id TEXT NOT NULL, action TEXT NOT NULL,
+ base_revision INTEGER, result_revision INTEGER, operation_json TEXT NOT NULL, created_at INTEGER NOT NULL);
+ CREATE TABLE IF NOT EXISTS slow_graph_provenance (
+ provenance_id TEXT PRIMARY KEY, patch_id TEXT NOT NULL, scope_id TEXT NOT NULL, capsule_id TEXT NOT NULL, revision INTEGER NOT NULL,
+ evidence_memory_id TEXT NOT NULL, claim_id TEXT NOT NULL, polarity TEXT NOT NULL, source_parent_json TEXT NOT NULL, created_at INTEGER NOT NULL);
+ CREATE INDEX IF NOT EXISTS idx_slow_graph_jobs_pending ON slow_graph_jobs(status, created_at);
+ CREATE INDEX IF NOT EXISTS idx_slow_graph_provenance_capsule ON slow_graph_provenance(scope_id, capsule_id, revision);
+ CREATE TABLE IF NOT EXISTS memory_edges (
+ scope_id TEXT NOT NULL, edge_id TEXT NOT NULL, source_memory_id TEXT NOT NULL, target_memory_id TEXT NOT NULL, edge_type TEXT NOT NULL,
+ score REAL NOT NULL, model_score REAL NOT NULL, evidence_turn INTEGER NOT NULL, evidence TEXT NOT NULL, metadata_json TEXT NOT NULL,
+ PRIMARY KEY(scope_id, edge_id));
+ CREATE TABLE IF NOT EXISTS slot_heads (
+ scope_id TEXT NOT NULL, slot_key TEXT NOT NULL, memory_id TEXT NOT NULL,
+ PRIMARY KEY(scope_id,slot_key));
+ CREATE TABLE IF NOT EXISTS slot_history (
+ scope_id TEXT NOT NULL, slot_key TEXT NOT NULL, ordinal INTEGER NOT NULL, memory_id TEXT NOT NULL,
+ PRIMARY KEY(scope_id,slot_key,ordinal));
+ """)
+ self._add_column_if_missing(con, "slow_graph_jobs", "claim_token TEXT")
+ self._add_column_if_missing(con, "slow_graph_jobs", "claim_owner TEXT")
+ self._add_column_if_missing(
+ con, "slow_graph_jobs", "lease_expires_at INTEGER"
+ )
+ self._add_column_if_missing(
+ con, "slow_graph_attempts", "claim_token TEXT"
+ )
+ self._add_column_if_missing(
+ con, "slow_graph_attempts", "claim_owner TEXT"
+ )
+ con.execute(
+ "CREATE INDEX IF NOT EXISTS idx_slow_graph_jobs_claimable "
+ "ON slow_graph_jobs(status, claim_token, created_at)"
+ )
+
+ @staticmethod
+ def _add_column_if_missing(
+ con: sqlite3.Connection, table: str, declaration: str
+ ) -> None:
+ column = declaration.split()[0]
+ columns = {
+ str(row["name"])
+ for row in con.execute("PRAGMA table_info(" + table + ")")
+ }
+ if column not in columns:
+ con.execute("ALTER TABLE " + table + " ADD COLUMN " + declaration)
+
+ def _records_table_exists(self, con: sqlite3.Connection) -> None:
+ if (
+ con.execute(
+ "SELECT 1 FROM sqlite_master WHERE type='table' AND name='records'"
+ ).fetchone()
+ is None
+ ):
+ raise SlowGraphError("real graph records table is missing")
+
+ def _metadata(self, row: sqlite3.Row, label: str) -> dict[str, Any]:
+ return _strict_json(row["metadata_json"], label=label, expected=dict)
+
+ def fast_regions(self, scope_id: str) -> dict[str, list[dict[str, Any]]]:
+ regions: dict[str, list[dict[str, Any]]] = {}
+ with self.connection() as con:
+ self._records_table_exists(con)
+ rows = con.execute(
+ "SELECT memory_id,value,relation,turn_index,metadata_json FROM records WHERE scope_id=?",
+ (scope_id,),
+ ).fetchall()
+ for row in rows:
+ metadata = self._metadata(row, "fast evidence metadata")
+ if (
+ metadata.get("content_variant") != LEAF_VARIANT
+ or metadata.get("memory_layer") != "fast"
+ or metadata.get("node_kind") != "atomic_user_assertion"
+ or metadata.get("atomic_evidence_leaf") is not True
+ or metadata.get("authority") != "user_assertion"
+ ):
+ continue
+ key = _clean(
+ metadata.get("graph_entity_key")
+ or metadata.get("entity_key")
+ or metadata.get("domain")
+ )
+ if not key:
+ raise SlowGraphError("fast evidence is missing region key")
+ regions.setdefault(key, []).append(
+ {
+ "memory_id": row["memory_id"],
+ "value": row["value"],
+ "relation": row["relation"],
+ "turn_index": row["turn_index"],
+ "metadata": metadata,
+ }
+ )
+ return regions
+
+ def _capsules(
+ self, con: sqlite3.Connection, scope_id: str, region_key: str
+ ) -> list[dict[str, Any]]:
+ rows = con.execute(
+ "SELECT memory_id,state,value,metadata_json FROM records WHERE scope_id=?",
+ (scope_id,),
+ ).fetchall()
+ revisions: list[dict[str, Any]] = []
+ for row in rows:
+ meta = self._metadata(row, "capsule metadata")
+ if (
+ meta.get("content_variant") == CAPSULE_VARIANT
+ and meta.get("region_key") == region_key
+ ):
+ revisions.append(
+ {
+ "memory_id": row["memory_id"],
+ "record_state": row["state"],
+ "value": row["value"],
+ **meta,
+ }
+ )
+ if not revisions:
+ return []
+ latest_revision = max(int(item["revision"]) for item in revisions)
+ latest = [
+ item for item in revisions if int(item["revision"]) == latest_revision
+ ]
+ if len(latest) != 1:
+ raise AuditError("region capsule lacks one latest revision")
+ return latest
+
+ def _job_metadata(
+ self,
+ con: sqlite3.Connection,
+ scope_id: str,
+ region_key: str,
+ evidence_ids: list[str],
+ manager: PatchManager | None,
+ ) -> dict[str, Any]:
+ evidence = self._evidence(con, scope_id, evidence_ids)
+ normalized_region = _required_text(region_key, "region key")
+ foreign_evidence = [
+ item["memory_id"]
+ for item in evidence
+ if _clean(item["metadata"].get("graph_entity_key"))
+ != normalized_region
+ ]
+ if foreign_evidence:
+ raise EvidencePolicyError(
+ "fast evidence graph_entity_key does not match job region: "
+ + ",".join(foreign_evidence)
+ )
+ model = dict(manager.model_config) if manager else {"model": "unbound"}
+ prompt_hash = manager.prompt_hash if manager else "unbound"
+ capsules = self._capsules(con, scope_id, region_key)
+ return {
+ "schema_version": SCHEMA_VERSION,
+ "schema_hash": _digest(
+ {"version": SCHEMA_VERSION, "actions": sorted(PATCH_ACTIONS)}
+ ),
+ "prompt_hash": prompt_hash,
+ "model_config": model,
+ "evidence_content_hash": _digest(evidence),
+ "capsule_revision_hash": _digest(capsules),
+ }
+
+ def _enqueue_in_connection(
+ self,
+ con: sqlite3.Connection,
+ scope_id: str,
+ region_key: str,
+ evidence_ids: list[str],
+ *,
+ manager: PatchManager | None = None,
+ ) -> str:
+ now = _now()
+ metadata = self._job_metadata(
+ con, scope_id, region_key, evidence_ids, manager
+ )
+ idem = _digest(
+ {
+ "scope_id": scope_id,
+ "region_key": region_key,
+ "evidence_ids": evidence_ids,
+ **{
+ key: value
+ for key, value in metadata.items()
+ if key != "capsule_revision_hash"
+ },
+ }
+ )
+ job_id = "sgj_" + uuid.uuid4().hex
+ con.execute(
+ "INSERT OR IGNORE INTO slow_graph_jobs("
+ "job_id,idempotency_key,scope_id,region_key,evidence_ids_json,metadata_json,"
+ "status,attempts,last_error,created_at,updated_at) VALUES(?,?,?,?,?,?,?,?,?,?,?)",
+ (
+ job_id,
+ idem,
+ scope_id,
+ region_key,
+ _json(evidence_ids),
+ _json(metadata),
+ "pending",
+ 0,
+ "",
+ now,
+ now,
+ ),
+ )
+ row = con.execute(
+ "SELECT job_id FROM slow_graph_jobs WHERE idempotency_key=?", (idem,)
+ ).fetchone()
+ return str(row["job_id"])
+
+ def enqueue(
+ self,
+ scope_id: str,
+ region_key: str,
+ evidence_ids: Iterable[str],
+ *,
+ manager: PatchManager | None = None,
+ ) -> str:
+ normalized_ids = sorted(
+ set(_required_text(item, "evidence id") for item in evidence_ids)
+ )
+ if not normalized_ids:
+ raise SlowGraphError("cannot enqueue without fast evidence")
+ with self.connection() as con:
+ self._records_table_exists(con)
+ return self._enqueue_in_connection(
+ con,
+ scope_id,
+ region_key,
+ normalized_ids,
+ manager=manager,
+ )
+
+ def enqueue_regions(
+ self, scope_id: str, *, manager: PatchManager | None = None
+ ) -> list[str]:
+ regions = self.fast_regions(scope_id)
+ manager_metadata = {
+ "schema_version": SCHEMA_VERSION,
+ "prompt_hash": manager.prompt_hash if manager else "unbound",
+ "model_config": dict(manager.model_config)
+ if manager
+ else {"model": "unbound"},
+ }
+ snapshot = {
+ key: [
+ {
+ "memory_id": item["memory_id"],
+ "value": item["value"],
+ "metadata": item["metadata"],
+ }
+ for item in values
+ ]
+ for key, values in sorted(regions.items())
+ }
+ evidence_snapshot_hash = _digest(snapshot)
+ batch_key = _digest(
+ {
+ "scope_id": scope_id,
+ "evidence_snapshot_hash": evidence_snapshot_hash,
+ **manager_metadata,
+ }
+ )
+ with self.connection() as con:
+ self._records_table_exists(con)
+ existing = con.execute(
+ "SELECT job_ids_json FROM slow_graph_batches WHERE batch_key=?",
+ (batch_key,),
+ ).fetchone()
+ if existing is not None:
+ return [
+ str(item)
+ for item in _strict_json(
+ existing["job_ids_json"], label="batch job IDs", expected=list
+ )
+ ]
+ job_ids = [
+ self._enqueue_in_connection(
+ con,
+ scope_id,
+ key,
+ sorted(str(item["memory_id"]) for item in values),
+ manager=manager,
+ )
+ for key, values in sorted(regions.items())
+ ]
+ con.execute(
+ "INSERT INTO slow_graph_batches VALUES(?,?,?,?,?,?,?)",
+ (
+ "sgb_" + uuid.uuid4().hex,
+ batch_key,
+ scope_id,
+ evidence_snapshot_hash,
+ _json(job_ids),
+ _json(manager_metadata),
+ _now(),
+ ),
+ )
+ return job_ids
+
+ def _job(self, job_id: str) -> sqlite3.Row:
+ with self.connection() as con:
+ row = con.execute(
+ "SELECT * FROM slow_graph_jobs WHERE job_id=?", (job_id,)
+ ).fetchone()
+ if row is None:
+ raise SlowGraphError("unknown slow graph job: " + job_id)
+ self._metadata(row, "job")
+ return row
+
+ def _evidence(
+ self, con: sqlite3.Connection, scope_id: str, ids: Iterable[str]
+ ) -> list[dict[str, Any]]:
+ result = []
+ for evidence_id in sorted(set(ids)):
+ row = con.execute(
+ "SELECT memory_id,value,turn_index,state,metadata_json FROM records WHERE scope_id=? AND memory_id=?",
+ (scope_id, evidence_id),
+ ).fetchone()
+ if row is None:
+ raise EvidencePolicyError("unknown evidence: " + evidence_id)
+ meta = self._metadata(row, "fast evidence")
+ if (
+ meta.get("content_variant") != LEAF_VARIANT
+ or meta.get("memory_layer") != "fast"
+ or meta.get("node_kind") != "atomic_user_assertion"
+ or meta.get("atomic_evidence_leaf") is not True
+ or meta.get("authority") != "user_assertion"
+ ):
+ raise EvidencePolicyError(
+ "only fast product_semantic_memory leaves may support capsules: "
+ + evidence_id
+ )
+ parent = {
+ key: meta.get(key)
+ for key in (
+ "session_index",
+ "message_index",
+ "source_record_id",
+ "event_id",
+ "evidence_char_start",
+ "evidence_char_end",
+ )
+ }
+ if any(value is None or _clean(value) == "" for value in parent.values()):
+ raise EvidencePolicyError(
+ "fast evidence lacks structured source_parent fields: "
+ + evidence_id
+ )
+ parent["parent_chunk_index"] = parent["message_index"]
+ record_state = _clean(row["state"])
+ evidence_role = (
+ "current_authoritative"
+ if record_state in {"active", "parallel_active", "promoted"}
+ else "historical_noncurrent"
+ )
+ result.append(
+ {
+ "memory_id": row["memory_id"],
+ "value": row["value"],
+ "turn_index": row["turn_index"],
+ "record_state": record_state,
+ "slow_graph_evidence_role": evidence_role,
+ "metadata": {
+ **meta,
+ "record_state": record_state,
+ "slow_graph_evidence_role": evidence_role,
+ },
+ "source_parent": parent,
+ }
+ )
+ return result
+
+ def _head(
+ self, con: sqlite3.Connection, scope_id: str, capsule_id: str
+ ) -> tuple[int, str] | None:
+ rows = con.execute(
+ "SELECT memory_id,metadata_json FROM records WHERE scope_id=?", (scope_id,)
+ ).fetchall()
+ candidates = []
+ for row in rows:
+ meta = self._metadata(row, "capsule metadata")
+ if (
+ meta.get("content_variant") == CAPSULE_VARIANT
+ and meta.get("capsule_id") == capsule_id
+ ):
+ candidates.append((int(meta["revision"]), str(row["memory_id"])))
+ return max(candidates) if candidates else None
+
+ def _capsule_id(self, scope_id: str, region_key: str) -> str:
+ return (
+ "cap_"
+ + _digest(
+ {
+ "scope_id": scope_id,
+ "region_key": _required_text(region_key, "region_key"),
+ }
+ )[:24]
+ )
+
+ def _mark_superseded(
+ self, con: sqlite3.Connection, scope_id: str, memory_id: str
+ ) -> None:
+ row = con.execute(
+ "SELECT slot_key,metadata_json FROM records WHERE scope_id=? AND memory_id=?",
+ (scope_id, memory_id),
+ ).fetchone()
+ if row is None:
+ raise SlowGraphError("cannot supersede missing record: " + memory_id)
+ metadata = self._metadata(row, "superseded record")
+ if metadata.get("content_variant") == CAPSULE_VARIANT:
+ metadata["status"] = "superseded"
+ con.execute(
+ "UPDATE records SET state='superseded',metadata_json=? WHERE scope_id=? AND memory_id=?",
+ (_json(metadata), scope_id, memory_id),
+ )
+ con.execute(
+ "DELETE FROM slot_heads WHERE scope_id=? AND slot_key=? AND memory_id=?",
+ (scope_id, row["slot_key"], memory_id),
+ )
+
+ def _write_edge(
+ self,
+ con: sqlite3.Connection,
+ *,
+ scope_id: str,
+ source: str,
+ target: str,
+ edge_type: str,
+ patch_id: str,
+ evidence_refs: list[str],
+ action: str,
+ turn: int,
+ ) -> None:
+ if edge_type not in EDGE_TYPES or source == target:
+ raise AuditError("invalid graph edge")
+ edge = self.edge_type(
+ edge_id="sge_"
+ + _digest(
+ {
+ "patch": patch_id,
+ "source": source,
+ "target": target,
+ "type": edge_type,
+ }
+ )[:24],
+ source_memory_id=source,
+ target_memory_id=target,
+ edge_type=edge_type,
+ score=0.0,
+ model_score=0.0,
+ evidence_turn=turn,
+ evidence="slow graph patch",
+ metadata={
+ "edge_source": SLOW_EDGE_SOURCE,
+ "patch_id": patch_id,
+ "evidence_refs": evidence_refs,
+ "action": action,
+ },
+ )
+ con.execute(
+ "INSERT INTO memory_edges VALUES(?,?,?,?,?,?,?,?,?,?)",
+ (
+ scope_id,
+ edge.edge_id,
+ edge.source_memory_id,
+ edge.target_memory_id,
+ edge.edge_type,
+ edge.score,
+ edge.model_score,
+ edge.evidence_turn,
+ edge.evidence,
+ _json(edge.metadata),
+ ),
+ )
+
+ def _insert_revision(
+ self,
+ con: sqlite3.Connection,
+ *,
+ job: sqlite3.Row,
+ patch_id: str,
+ operation: Mapping[str, Any],
+ capsule_id: str,
+ revision: int,
+ action: str,
+ old_memory_id: str | None = None,
+ ) -> str:
+ claims = _validate_claims(operation["claims"])
+ evidence_ids = sorted(
+ {
+ item
+ for claim in claims
+ for key in ("support", "counterevidence")
+ for item in claim[key]
+ }
+ )
+ allowed_evidence = set(
+ _strict_json(
+ job["evidence_ids_json"], label="job evidence IDs", expected=list
+ )
+ )
+ if not set(evidence_ids).issubset(allowed_evidence):
+ raise EvidencePolicyError(
+ "GraphPatch cited fast evidence that was not supplied to this job"
+ )
+ evidence = self._evidence(con, job["scope_id"], evidence_ids)
+ evidence_by_id = {item["memory_id"]: item for item in evidence}
+ controller_normalizations: list[dict[str, Any]] = []
+ for index, claim in enumerate(claims):
+ cited_slots = {
+ _clean(evidence_by_id[evidence_id]["metadata"].get("canonical_slot_key"))
+ for evidence_id in [*claim["support"], *claim["counterevidence"]]
+ }
+ cited_slots.discard("")
+ if claim["canonical_slot"] not in cited_slots:
+ if len(cited_slots) != 1:
+ raise EvidencePolicyError(
+ "claim canonical_slot is ambiguous across cited fast leaf slots"
+ )
+ authoritative_slot = next(iter(cited_slots))
+ original_slot = claim["canonical_slot"]
+ normalized_claim = {
+ **claim,
+ "canonical_slot": authoritative_slot,
+ }
+ normalized_claim["claim_id"] = "clm_" + _digest(
+ {
+ "canonical_slot": authoritative_slot,
+ "text": normalized_claim["text"],
+ "support": normalized_claim["support"],
+ "counterevidence": normalized_claim["counterevidence"],
+ }
+ )[:24]
+ claims[index] = normalized_claim
+ controller_normalizations.append(
+ {
+ "code": "canonical_slot_bound_to_unique_cited_leaf",
+ "claim_index": index,
+ "model_value": original_slot,
+ "authoritative_value": authoritative_slot,
+ }
+ )
+ claim_ids = [claim["claim_id"] for claim in claims]
+ if len(set(claim_ids)) != len(claim_ids):
+ raise EvidencePolicyError(
+ "canonical slot binding produced duplicate claims"
+ )
+ status = {
+ "challenge": "challenged",
+ "retire": "retired",
+ "resolve_challenge": "active",
+ }.get(action, "active")
+ stored_claims: list[dict[str, Any]] = []
+ for claim in claims:
+ claim_parents = [
+ evidence_by_id[evidence_id]["source_parent"]
+ for evidence_id in [*claim["support"], *claim["counterevidence"]]
+ ]
+ stored_claims.append(
+ {**claim, "source_parents": _validate_source_parents(claim_parents)}
+ )
+ source_parents = _validate_source_parents(
+ [parent for claim in stored_claims for parent in claim["source_parents"]]
+ )
+ record_id = f"slow.{capsule_id}.r{revision}"
+ metadata = {
+ "memory_layer": "slow",
+ "content_variant": CAPSULE_VARIANT,
+ "capsule_id": capsule_id,
+ "revision": revision,
+ "status": status,
+ "claims": stored_claims,
+ "source_parents": source_parents,
+ "canonical_slots": sorted(
+ {claim["canonical_slot"] for claim in stored_claims}
+ ),
+ "patch_id": patch_id,
+ "region_key": job["region_key"],
+ "action": action,
+ "controller_normalizations": controller_normalizations,
+ }
+ record_state = "active" if status in {"active", "challenged"} else status
+ record = self.record_type(
+ memory_id=record_id,
+ category=CAPSULE_VARIANT,
+ slot_key="slow." + capsule_id,
+ value=_clean(operation.get("summary")) or _json(claims),
+ relation="capsule_revision",
+ anchor_concepts=[job["region_key"]],
+ evidence_anchors=evidence_ids,
+ salience=0.7,
+ confidence=0.0,
+ source_kind="slow_graph",
+ turn_index=max(item["turn_index"] for item in evidence),
+ state=record_state,
+ supersedes=[old_memory_id] if old_memory_id else [],
+ metadata=metadata,
+ )
+ payload = record.to_dict()
+ con.execute(
+ "INSERT INTO records(scope_id,memory_id,category,slot_key,value,relation,anchor_concepts_json,evidence_anchors_json,salience,confidence,source_kind,turn_index,state,supersedes_json,metadata_json) VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
+ (
+ job["scope_id"],
+ payload["memory_id"],
+ payload["category"],
+ payload["slot_key"],
+ payload["value"],
+ payload["relation"],
+ _json(payload["anchor_concepts"]),
+ _json(payload["evidence_anchors"]),
+ payload["salience"],
+ payload["confidence"],
+ payload["source_kind"],
+ payload["turn_index"],
+ payload["state"],
+ _json(payload["supersedes"]),
+ _json(payload["metadata"]),
+ ),
+ )
+ slot_key = payload["slot_key"]
+ ordinal = int(
+ con.execute(
+ "SELECT COALESCE(MAX(ordinal),-1)+1 FROM slot_history WHERE scope_id=? AND slot_key=?",
+ (job["scope_id"], slot_key),
+ ).fetchone()[0]
+ )
+ con.execute(
+ "INSERT INTO slot_history(scope_id,slot_key,ordinal,memory_id) VALUES(?,?,?,?)",
+ (job["scope_id"], slot_key, ordinal, record_id),
+ )
+ if record_state == "active":
+ con.execute(
+ "INSERT INTO slot_heads(scope_id,slot_key,memory_id) VALUES(?,?,?) "
+ "ON CONFLICT(scope_id,slot_key) DO UPDATE SET memory_id=excluded.memory_id",
+ (job["scope_id"], slot_key, record_id),
+ )
+ else:
+ con.execute(
+ "DELETE FROM slot_heads WHERE scope_id=? AND slot_key=?",
+ (job["scope_id"], slot_key),
+ )
+ for claim in stored_claims:
+ for polarity, edge_kind in (
+ ("support", "supports"),
+ ("counterevidence", "contradicts"),
+ ):
+ for evidence_id in claim[polarity]:
+ item = evidence_by_id[evidence_id]
+ con.execute(
+ "INSERT INTO slow_graph_provenance VALUES(?,?,?,?,?,?,?,?,?,?)",
+ (
+ "sgv_" + uuid.uuid4().hex,
+ patch_id,
+ job["scope_id"],
+ capsule_id,
+ revision,
+ evidence_id,
+ claim["claim_id"],
+ polarity,
+ _json(item["source_parent"]),
+ _now(),
+ ),
+ )
+ self._write_edge(
+ con,
+ scope_id=job["scope_id"],
+ source=evidence_id,
+ target=record_id,
+ edge_type=edge_kind,
+ patch_id=patch_id,
+ evidence_refs=[evidence_id],
+ action=action,
+ turn=item["turn_index"],
+ )
+ if old_memory_id:
+ self._mark_superseded(con, job["scope_id"], old_memory_id)
+ self._write_edge(
+ con,
+ scope_id=job["scope_id"],
+ source=old_memory_id,
+ target=record_id,
+ edge_type="supersedes",
+ patch_id=patch_id,
+ evidence_refs=evidence_ids,
+ action=action,
+ turn=payload["turn_index"],
+ )
+ return record_id
+
+ def _audit_transaction(self, con: sqlite3.Connection, scope_id: str) -> None:
+ """Validate slow-layer invariants against the uncommitted write transaction."""
+ rows = con.execute(
+ "SELECT memory_id,state,metadata_json FROM records WHERE scope_id=?",
+ (scope_id,),
+ ).fetchall()
+ capsules: dict[str, list[tuple[str, str, int, str]]] = {}
+ known_records = {str(row["memory_id"]) for row in rows}
+ for row in rows:
+ meta = self._metadata(row, "transaction record")
+ if meta.get("content_variant") != CAPSULE_VARIANT:
+ continue
+ capsule_id = _required_text(meta.get("capsule_id"), "capsule_id")
+ revision = meta.get("revision")
+ status = _required_text(meta.get("status"), "capsule status")
+ if not isinstance(revision, int) or revision < 1:
+ raise AuditError("capsule revision is invalid")
+ if status not in {"active", "challenged", "retired", "superseded"}:
+ raise AuditError("capsule status is invalid")
+ claims = _validate_claims(meta.get("claims"), stored=True)
+ capsules.setdefault(capsule_id, []).append(
+ (str(row["memory_id"]), str(row["state"]), revision, status)
+ )
+ for claim in claims:
+ expected = set(claim["support"] + claim["counterevidence"])
+ actual = {
+ str(item["evidence_memory_id"])
+ for item in con.execute(
+ "SELECT evidence_memory_id FROM slow_graph_provenance "
+ "WHERE scope_id=? AND capsule_id=? AND revision=? AND claim_id=?",
+ (scope_id, capsule_id, revision, claim["claim_id"]),
+ )
+ }
+ if actual != expected:
+ raise AuditError("claim provenance does not match claim evidence")
+ for capsule_id, revisions in capsules.items():
+ ordered = sorted(revisions, key=lambda item: item[2])
+ if [item[2] for item in ordered] != list(range(1, len(ordered) + 1)):
+ raise AuditError("capsule revisions are not contiguous")
+ memory_id, state, _, status = ordered[-1]
+ slot_key = "slow." + capsule_id
+ head = con.execute(
+ "SELECT memory_id FROM slot_heads WHERE scope_id=? AND slot_key=?",
+ (scope_id, slot_key),
+ ).fetchone()
+ if status in {"active", "challenged"}:
+ if state != "active" or head is None or head["memory_id"] != memory_id:
+ raise AuditError("active/challenged capsule head is inconsistent")
+ elif head is not None:
+ raise AuditError("inactive capsule must not retain a slot head")
+ history = [
+ str(item["memory_id"])
+ for item in con.execute(
+ "SELECT memory_id FROM slot_history WHERE scope_id=? AND slot_key=? ORDER BY ordinal",
+ (scope_id, slot_key),
+ )
+ ]
+ if history != [item[0] for item in ordered]:
+ raise AuditError("capsule slot history is inconsistent")
+ for edge in con.execute(
+ "SELECT source_memory_id,target_memory_id,edge_type,metadata_json "
+ "FROM memory_edges WHERE scope_id=?",
+ (scope_id,),
+ ):
+ meta = _strict_json(
+ edge["metadata_json"], label="edge metadata", expected=dict
+ )
+ if meta.get("edge_source") != SLOW_EDGE_SOURCE:
+ continue
+ if (
+ edge["source_memory_id"] == edge["target_memory_id"]
+ or edge["edge_type"] not in EDGE_TYPES
+ or edge["source_memory_id"] not in known_records
+ or edge["target_memory_id"] not in known_records
+ or not isinstance(meta.get("evidence_refs"), list)
+ or not _clean(meta.get("patch_id"))
+ ):
+ raise AuditError("slow graph edge is inconsistent")
+
+ def apply_patch(
+ self,
+ job_id: str,
+ patch: Mapping[str, Any],
+ *,
+ manager_model: str,
+ call_metadata: Mapping[str, Any] | None = None,
+ claim: JobClaim,
+ ) -> str:
+ validate_patch(patch)
+ if claim.job_id != job_id:
+ raise SlowGraphError("claim does not belong to job")
+ patch_id = "sgp_" + uuid.uuid4().hex
+ with self.connection() as con:
+ con.execute("BEGIN IMMEDIATE")
+ now = _now()
+ job = con.execute(
+ "SELECT * FROM slow_graph_jobs WHERE job_id=?", (job_id,)
+ ).fetchone()
+ if job is None:
+ raise SlowGraphError("unknown slow graph job")
+ if (
+ job["status"] != "pending"
+ or job["claim_token"] != claim.token
+ or job["claim_owner"] != claim.owner
+ or job["lease_expires_at"] is None
+ or int(job["lease_expires_at"]) < now
+ ):
+ raise SlowGraphError("job claim is no longer active")
+ self._records_table_exists(con)
+ metadata = self._metadata(job, "job")
+ if metadata["schema_version"] != SCHEMA_VERSION:
+ raise SlowGraphError("job schema metadata drift")
+ con.execute(
+ "INSERT INTO slow_graph_patches VALUES(?,?,?,?,?,?,?,?)",
+ (
+ patch_id,
+ job_id,
+ job["scope_id"],
+ job["region_key"],
+ manager_model,
+ _json(patch),
+ _json(dict(call_metadata or {})),
+ now,
+ ),
+ )
+ ordinal = 0
+ for operation in patch["operations"]:
+ action = operation["action"]
+ region_capsule_id = self._capsule_id(
+ job["scope_id"], job["region_key"]
+ )
+ capsule_id = (
+ region_capsule_id
+ if action == "create"
+ else operation.get("capsule_id")
+ or "region:" + job["region_key"]
+ )
+ if action != "create" and operation.get("capsule_id"):
+ if capsule_id != region_capsule_id:
+ raise EvidencePolicyError(
+ "GraphPatch capsule_id does not belong to the current region"
+ )
+ head = self._head(con, job["scope_id"], capsule_id)
+ if action == "create":
+ if head is not None:
+ raise StaleRevisionError("capsule already exists")
+ record_id = self._insert_revision(
+ con,
+ job=job,
+ patch_id=patch_id,
+ operation=operation,
+ capsule_id=capsule_id,
+ revision=1,
+ action=action,
+ )
+ base, revision = None, 1
+ elif action == "noop":
+ if operation.get("capsule_id") and head is None:
+ raise StaleRevisionError("noop capsule does not exist")
+ if head is None:
+ record_id, base, revision = "", None, None
+ else:
+ record_id, (revision, _) = head[1], head
+ base = revision
+ else:
+ if head is None or operation["base_revision"] != head[0]:
+ raise StaleRevisionError(
+ "base_revision is stale for " + capsule_id
+ )
+ base, revision = head[0], head[0] + 1
+ record_id = self._insert_revision(
+ con,
+ job=job,
+ patch_id=patch_id,
+ operation=operation,
+ capsule_id=capsule_id,
+ revision=revision,
+ action=action,
+ old_memory_id=head[1],
+ )
+ if action == "challenge":
+ self._write_edge(
+ con,
+ scope_id=job["scope_id"],
+ source=record_id,
+ target=head[1],
+ edge_type="challenges",
+ patch_id=patch_id,
+ evidence_refs=[],
+ action=action,
+ turn=now,
+ )
+ if action == "retire":
+ self._write_edge(
+ con,
+ scope_id=job["scope_id"],
+ source=record_id,
+ target=head[1],
+ edge_type="invalidates",
+ patch_id=patch_id,
+ evidence_refs=[],
+ action=action,
+ turn=now,
+ )
+ con.execute(
+ "INSERT INTO slow_graph_patch_operations VALUES(?,?,?,?,?,?,?,?,?)",
+ (
+ "sgo_" + uuid.uuid4().hex,
+ patch_id,
+ ordinal,
+ capsule_id,
+ action,
+ base,
+ revision,
+ _json(operation),
+ now,
+ ),
+ )
+ ordinal += 1
+ completed_attempt = con.execute(
+ "UPDATE slow_graph_attempts SET status='completed',"
+ "call_metadata_json=?,completed_at=? WHERE attempt_id=? AND job_id=? "
+ "AND claim_token=? AND claim_owner=? AND status='started'",
+ (
+ _json(dict(call_metadata or {})),
+ now,
+ claim.attempt_id,
+ job_id,
+ claim.token,
+ claim.owner,
+ ),
+ )
+ if completed_attempt.rowcount != 1:
+ raise SlowGraphError("claimed attempt is no longer active")
+ completed_job = con.execute(
+ "UPDATE slow_graph_jobs SET status='completed',attempts=attempts+1,"
+ "last_error='',updated_at=?,claim_token=NULL,claim_owner=NULL,"
+ "lease_expires_at=NULL WHERE job_id=? AND status='pending' "
+ "AND claim_token=? AND claim_owner=? AND lease_expires_at>=?",
+ (now, job_id, claim.token, claim.owner, now),
+ )
+ if completed_job.rowcount != 1:
+ raise SlowGraphError("job claim expired before completion")
+ self._audit_transaction(con, job["scope_id"])
+ return patch_id
+
+ def _claim_pending_job(
+ self, job_id: str | None, *, owner: str
+ ) -> JobClaim | None:
+ token = "sgc_" + uuid.uuid4().hex
+ attempt_id = "sga_" + uuid.uuid4().hex
+ now = _now()
+ with self.connection() as con:
+ con.execute("BEGIN IMMEDIATE")
+ if job_id is None:
+ row = con.execute(
+ "SELECT job_id,scope_id FROM slow_graph_jobs WHERE status='pending' "
+ "AND claim_token IS NULL ORDER BY created_at LIMIT 1"
+ ).fetchone()
+ else:
+ row = con.execute(
+ "SELECT job_id,scope_id FROM slow_graph_jobs WHERE job_id=? "
+ "AND status='pending' AND claim_token IS NULL",
+ (job_id,),
+ ).fetchone()
+ if row is None:
+ return None
+ claimed = con.execute(
+ "UPDATE slow_graph_jobs SET claim_token=?,claim_owner=?,"
+ "lease_expires_at=?,updated_at=? WHERE job_id=? AND status='pending' "
+ "AND claim_token IS NULL",
+ (
+ token,
+ owner,
+ now + self.claim_lease_seconds,
+ now,
+ str(row["job_id"]),
+ ),
+ )
+ if claimed.rowcount != 1:
+ return None
+ con.execute(
+ "INSERT INTO slow_graph_attempts("
+ "attempt_id,job_id,scope_id,status,call_metadata_json,error,created_at,"
+ "completed_at,claim_token,claim_owner) VALUES(?,?,?,?,?,?,?,?,?,?)",
+ (
+ attempt_id,
+ str(row["job_id"]),
+ str(row["scope_id"]),
+ "started",
+ _json({}),
+ "",
+ now,
+ None,
+ token,
+ owner,
+ ),
+ )
+ return JobClaim(str(row["job_id"]), attempt_id, token, owner)
+
+ def _claim_owner(self) -> str:
+ return "pid:" + str(os.getpid()) + ":" + uuid.uuid4().hex
+
+ def _claim_context(
+ self, claim: JobClaim
+ ) -> tuple[dict[str, Any], list[dict[str, Any]]]:
+ with self.connection() as con:
+ job = con.execute(
+ "SELECT * FROM slow_graph_jobs WHERE job_id=?", (claim.job_id,)
+ ).fetchone()
+ if job is None:
+ raise SlowGraphError("unknown slow graph job")
+ if (
+ job["status"] != "pending"
+ or job["claim_token"] != claim.token
+ or job["claim_owner"] != claim.owner
+ or job["lease_expires_at"] is None
+ or int(job["lease_expires_at"]) < _now()
+ ):
+ raise SlowGraphError("job claim is no longer active")
+ evidence_ids = _strict_json(
+ job["evidence_ids_json"], label="job evidence IDs", expected=list
+ )
+ return (
+ {
+ "region_key": job["region_key"],
+ "evidence": self._evidence(con, job["scope_id"], evidence_ids),
+ },
+ self._capsules(con, job["scope_id"], job["region_key"]),
+ )
+
+ def _renew_claim(self, claim: JobClaim) -> None:
+ now = _now()
+ with self.connection() as con:
+ con.execute("BEGIN IMMEDIATE")
+ renewed = con.execute(
+ "UPDATE slow_graph_jobs SET lease_expires_at=?,updated_at=? "
+ "WHERE job_id=? AND status='pending' AND claim_token=? "
+ "AND claim_owner=? AND lease_expires_at>=?",
+ (
+ now + self.claim_lease_seconds,
+ now,
+ claim.job_id,
+ claim.token,
+ claim.owner,
+ now,
+ ),
+ )
+ if renewed.rowcount != 1:
+ raise SlowGraphError("job claim could not be renewed")
+
+ def _propose_with_lease_heartbeat(
+ self,
+ claim: JobClaim,
+ manager: PatchManager,
+ region: Mapping[str, Any],
+ capsules: list[dict[str, Any]],
+ ) -> dict[str, Any]:
+ stop = threading.Event()
+ errors: list[Exception] = []
+ interval = max(0.1, min(30.0, self.claim_lease_seconds / 3.0))
+
+ def heartbeat() -> None:
+ while not stop.wait(interval):
+ try:
+ self._renew_claim(claim)
+ except Exception as exc:
+ errors.append(exc)
+ stop.set()
+
+ thread = threading.Thread(
+ target=heartbeat,
+ name=f"slow-graph-lease-{claim.job_id}",
+ daemon=True,
+ )
+ thread.start()
+ try:
+ patch = manager.propose(region, capsules)
+ finally:
+ stop.set()
+ thread.join(timeout=max(1.0, interval + 1.0))
+ if thread.is_alive():
+ raise SlowGraphError("slow graph claim heartbeat did not stop")
+ if errors:
+ raise SlowGraphError(f"slow graph claim heartbeat failed: {errors[0]}")
+ self._renew_claim(claim)
+ return patch
+
+ def _finish_claim_failure(
+ self, claim: JobClaim, manager: PatchManager, exc: Exception
+ ) -> bool:
+ status = (
+ "retryable"
+ if isinstance(exc, DeepSeekCallError) and exc.retryable
+ else "failed"
+ )
+ now = _now()
+ with self.connection() as con:
+ con.execute("BEGIN IMMEDIATE")
+ failed_job = con.execute(
+ "UPDATE slow_graph_jobs SET status=?,attempts=attempts+1,last_error=?,"
+ "updated_at=?,claim_token=NULL,claim_owner=NULL,lease_expires_at=NULL "
+ "WHERE job_id=? AND status='pending' AND claim_token=? "
+ "AND claim_owner=? AND lease_expires_at>=?",
+ (status, str(exc), now, claim.job_id, claim.token, claim.owner, now),
+ )
+ if failed_job.rowcount != 1:
+ return False
+ failed_attempt = con.execute(
+ "UPDATE slow_graph_attempts SET status=?,call_metadata_json=?,error=?,"
+ "completed_at=? WHERE attempt_id=? AND job_id=? AND claim_token=? "
+ "AND claim_owner=? AND status='started'",
+ (
+ status,
+ _json(dict(manager.last_call_metadata)),
+ str(exc),
+ now,
+ claim.attempt_id,
+ claim.job_id,
+ claim.token,
+ claim.owner,
+ ),
+ )
+ if failed_attempt.rowcount != 1:
+ raise SlowGraphError("claimed attempt is no longer active")
+ return True
+
+ def _run_claimed_job(self, claim: JobClaim, manager: PatchManager) -> str:
+ try:
+ region, capsules = self._claim_context(claim)
+ patch = self._propose_with_lease_heartbeat(
+ claim, manager, region, capsules
+ )
+ patch_id = self.apply_patch(
+ claim.job_id,
+ patch,
+ manager_model=_required_text(
+ manager.model_config.get("model"), "manager model"
+ ),
+ call_metadata=manager.last_call_metadata,
+ claim=claim,
+ )
+ return patch_id
+ except Exception as exc:
+ self._finish_claim_failure(claim, manager, exc)
+ raise
+
+ def run_job(self, job_id: str, manager: PatchManager) -> str:
+ self.recover_interrupted_attempts()
+ claim = self._claim_pending_job(job_id, owner=self._claim_owner())
+ if claim is None:
+ raise SlowGraphError("job is not pending or is already claimed")
+ return self._run_claimed_job(claim, manager)
+
+ def resume(self, job_id: str) -> None:
+ with self.connection() as con:
+ con.execute("BEGIN IMMEDIATE")
+ row = con.execute(
+ "SELECT status,claim_token FROM slow_graph_jobs WHERE job_id=?",
+ (job_id,),
+ ).fetchone()
+ if (
+ row is None
+ or row["status"] not in {"failed", "retryable"}
+ or row["claim_token"] is not None
+ ):
+ raise SlowGraphError(
+ "only failed or retryable jobs can be explicitly reopened"
+ )
+ reopened = con.execute(
+ "UPDATE slow_graph_jobs SET status='pending',last_error='',updated_at=?,"
+ "claim_token=NULL,claim_owner=NULL,lease_expires_at=NULL WHERE job_id=? "
+ "AND status IN ('failed','retryable') AND claim_token IS NULL",
+ (_now(), job_id),
+ )
+ if reopened.rowcount != 1:
+ raise SlowGraphError("slow graph job changed while reopening")
+
+ def recover_interrupted_attempts(self) -> int:
+ now = _now()
+ with self.connection() as con:
+ con.execute("BEGIN IMMEDIATE")
+ rows = con.execute(
+ "SELECT job_id,claim_token,claim_owner FROM slow_graph_jobs "
+ "WHERE status='pending' AND claim_token IS NOT NULL "
+ "AND lease_expires_at list[str]:
+ self.recover_interrupted_attempts()
+ if batch_size is not None:
+ if batch_size <= 0:
+ raise SlowGraphError("batch_size must be positive")
+ owner = self._claim_owner()
+ result: list[str] = []
+ while batch_size is None or len(result) < batch_size:
+ claim = self._claim_pending_job(None, owner=owner)
+ if claim is None:
+ break
+ result.append(self._run_claimed_job(claim, manager))
+ return result
+
+ def audit(self, scope_id: str) -> dict[str, int]:
+ with self.connection() as con:
+ self._records_table_exists(con)
+ unfinished = con.execute(
+ "SELECT job_id,status,last_error FROM slow_graph_jobs WHERE scope_id=? AND status!='completed' ORDER BY created_at",
+ (scope_id,),
+ ).fetchall()
+ if unfinished:
+ raise AuditError(
+ "scope has unfinished slow graph jobs: "
+ + _json([dict(item) for item in unfinished])
+ )
+ rows = con.execute(
+ "SELECT memory_id,state,metadata_json FROM records WHERE scope_id=?",
+ (scope_id,),
+ ).fetchall()
+ fast_state: dict[str, tuple[str, str]] = {}
+ active_fast_by_slot: dict[str, set[str]] = {}
+ for row in rows:
+ row_meta = self._metadata(row, "record")
+ if (
+ row_meta.get("content_variant") == LEAF_VARIANT
+ and row_meta.get("memory_layer") == "fast"
+ ):
+ slot = _clean(row_meta.get("canonical_slot_key"))
+ state = _clean(row["state"])
+ fast_state[str(row["memory_id"])] = (state, slot)
+ if state in {"active", "parallel_active", "promoted"}:
+ active_fast_by_slot.setdefault(slot, set()).add(
+ str(row["memory_id"])
+ )
+ capsules: dict[str, list[tuple[str, str, int, str]]] = {}
+ for row in rows:
+ meta = self._metadata(row, "record")
+ if meta.get("content_variant") != CAPSULE_VARIANT:
+ continue
+ capsule_id = _required_text(meta.get("capsule_id"), "capsule_id")
+ revision = meta.get("revision")
+ if not isinstance(revision, int) or revision < 1:
+ raise AuditError("capsule revision is invalid")
+ status = _required_text(meta.get("status"), "capsule status")
+ if status not in {"active", "challenged", "retired", "superseded"}:
+ raise AuditError("capsule status is invalid")
+ capsules.setdefault(capsule_id, []).append(
+ (str(row["memory_id"]), str(row["state"]), revision, status)
+ )
+ claims = _validate_claims(meta.get("claims"), stored=True)
+ for claim in claims:
+ if status in {"active", "challenged"}:
+ stale_support = {
+ evidence_id
+ for evidence_id in claim["support"]
+ if fast_state.get(evidence_id, ("", ""))[0]
+ not in {"active", "parallel_active", "promoted"}
+ }
+ current_replacements = active_fast_by_slot.get(
+ claim["canonical_slot"], set()
+ )
+ if (
+ stale_support
+ and current_replacements
+ and not current_replacements.intersection(
+ claim["counterevidence"]
+ )
+ ):
+ raise AuditError(
+ "active capsule claim promotes superseded fast evidence"
+ )
+ evidence = claim["support"] + claim["counterevidence"]
+ provenance = con.execute(
+ "SELECT evidence_memory_id,source_parent_json FROM slow_graph_provenance WHERE scope_id=? AND capsule_id=? AND revision=? AND claim_id=?",
+ (scope_id, capsule_id, revision, claim["claim_id"]),
+ ).fetchall()
+ if {item["evidence_memory_id"] for item in provenance} != set(
+ evidence
+ ):
+ raise AuditError(
+ "claim provenance does not match claim evidence"
+ )
+ for item in provenance:
+ parent = _strict_json(
+ item["source_parent_json"],
+ label="provenance source_parent",
+ expected=dict,
+ )
+ if set(parent) != {
+ "session_index",
+ "parent_chunk_index",
+ "message_index",
+ "source_record_id",
+ "event_id",
+ "evidence_char_start",
+ "evidence_char_end",
+ }:
+ raise AuditError("provenance is not a leaf source_parent")
+ leaf = con.execute(
+ "SELECT metadata_json FROM records WHERE scope_id=? AND memory_id=?",
+ (scope_id, item["evidence_memory_id"]),
+ ).fetchone()
+ if leaf is None:
+ raise AuditError("provenance references missing evidence")
+ leaf_meta = self._metadata(leaf, "provenance leaf")
+ expected_parent = {
+ "session_index": leaf_meta.get("session_index"),
+ "parent_chunk_index": leaf_meta.get("message_index"),
+ "message_index": leaf_meta.get("message_index"),
+ "source_record_id": leaf_meta.get("source_record_id"),
+ "event_id": leaf_meta.get("event_id"),
+ "evidence_char_start": leaf_meta.get(
+ "evidence_char_start"
+ ),
+ "evidence_char_end": leaf_meta.get("evidence_char_end"),
+ }
+ if (
+ leaf_meta.get("content_variant") != LEAF_VARIANT
+ or leaf_meta.get("memory_layer") != "fast"
+ or leaf_meta.get("node_kind") != "atomic_user_assertion"
+ or leaf_meta.get("atomic_evidence_leaf") is not True
+ or leaf_meta.get("authority") != "user_assertion"
+ or expected_parent != parent
+ ):
+ raise AuditError(
+ "provenance does not resolve to the cited fast leaf"
+ )
+ for capsule_id, revisions in capsules.items():
+ revision_numbers = sorted(item[2] for item in revisions)
+ if revision_numbers != list(range(1, max(revision_numbers) + 1)):
+ raise AuditError("capsule revisions are not contiguous")
+ latest = [item for item in revisions if item[2] == max(revision_numbers)]
+ if len(latest) != 1:
+ raise AuditError("capsule lacks one latest revision")
+ memory_id, state, _, status = latest[0]
+ slot_key = "slow." + capsule_id
+ head = con.execute(
+ "SELECT memory_id FROM slot_heads WHERE scope_id=? AND slot_key=?",
+ (scope_id, slot_key),
+ ).fetchone()
+ if status in {"active", "challenged"}:
+ if state != "active" or head is None or head["memory_id"] != memory_id:
+ raise AuditError("active/challenged capsule head is inconsistent")
+ elif head is not None:
+ raise AuditError("inactive capsule must not retain a slot head")
+ history = [
+ item["memory_id"]
+ for item in con.execute(
+ "SELECT memory_id FROM slot_history WHERE scope_id=? AND slot_key=? ORDER BY ordinal",
+ (scope_id, slot_key),
+ )
+ ]
+ expected_history = [
+ item[0] for item in sorted(revisions, key=lambda item: item[2])
+ ]
+ if history != expected_history:
+ raise AuditError("capsule slot history is inconsistent")
+ edge_rows = con.execute(
+ "SELECT source_memory_id,target_memory_id,edge_type,metadata_json FROM memory_edges WHERE scope_id=?",
+ (scope_id,),
+ ).fetchall()
+ for edge in edge_rows:
+ meta = _strict_json(
+ edge["metadata_json"], label="edge metadata", expected=dict
+ )
+ if meta.get("edge_source") != SLOW_EDGE_SOURCE:
+ continue
+ if (
+ edge["source_memory_id"] == edge["target_memory_id"]
+ or edge["edge_type"] not in EDGE_TYPES
+ or not isinstance(meta.get("evidence_refs"), list)
+ or not _clean(meta.get("patch_id"))
+ ):
+ raise AuditError("edge is inconsistent")
+ return {
+ "slow_graph_batches": int(
+ con.execute(
+ "SELECT COUNT(*) FROM slow_graph_batches WHERE scope_id=?",
+ (scope_id,),
+ ).fetchone()[0]
+ ),
+ "slow_graph_jobs": int(
+ con.execute(
+ "SELECT COUNT(*) FROM slow_graph_jobs WHERE scope_id=?",
+ (scope_id,),
+ ).fetchone()[0]
+ ),
+ "slow_graph_patches": int(
+ con.execute(
+ "SELECT COUNT(*) FROM slow_graph_patches WHERE scope_id=?",
+ (scope_id,),
+ ).fetchone()[0]
+ ),
+ "slow_graph_attempts": int(
+ con.execute(
+ "SELECT COUNT(*) FROM slow_graph_attempts WHERE scope_id=?",
+ (scope_id,),
+ ).fetchone()[0]
+ ),
+ "slow_graph_patch_operations": int(
+ con.execute(
+ "SELECT COUNT(*) FROM slow_graph_patch_operations o JOIN slow_graph_patches p ON p.patch_id=o.patch_id WHERE p.scope_id=?",
+ (scope_id,),
+ ).fetchone()[0]
+ ),
+ "slow_graph_provenance": int(
+ con.execute(
+ "SELECT COUNT(*) FROM slow_graph_provenance WHERE scope_id=?",
+ (scope_id,),
+ ).fetchone()[0]
+ ),
+ "memory_edges": sum(
+ 1
+ for edge in edge_rows
+ if _strict_json(
+ edge["metadata_json"], label="edge metadata", expected=dict
+ ).get("edge_source")
+ == SLOW_EDGE_SOURCE
+ ),
+ }
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description="TMCRA slow graph controller")
+ parser.add_argument("database", type=Path)
+ parser.add_argument(
+ "--repo",
+ type=Path,
+ required=True,
+ help="TMCRA repository containing the real graph schema",
+ )
+ sub = parser.add_subparsers(dest="command", required=True)
+ enqueue = sub.add_parser("enqueue")
+ enqueue.add_argument("scope_id")
+ enqueue.add_argument("--region")
+ drain = sub.add_parser("drain")
+ drain.add_argument("--batch-size", type=int)
+ run = sub.add_parser("run")
+ run.add_argument("job_id")
+ resume = sub.add_parser("resume")
+ resume.add_argument("job_id")
+ audit = sub.add_parser("audit")
+ audit.add_argument("scope_id")
+ args = parser.parse_args()
+ store = SlowGraphStore(args.database, schema=load_graph_schema(args.repo))
+ if args.command == "enqueue":
+ manager = DeepSeekProGraphPatchManager(DeepSeekProConfig.from_env())
+ if args.region:
+ region = store.fast_regions(args.scope_id).get(args.region, [])
+ result = [
+ store.enqueue(
+ args.scope_id,
+ args.region,
+ (item["memory_id"] for item in region),
+ manager=manager,
+ )
+ ]
+ else:
+ result = store.enqueue_regions(args.scope_id, manager=manager)
+ elif args.command == "resume":
+ store.resume(args.job_id)
+ result = {"job_id": args.job_id, "status": "pending"}
+ elif args.command == "audit":
+ result = store.audit(args.scope_id)
+ else:
+ manager = DeepSeekProGraphPatchManager(DeepSeekProConfig.from_env())
+ result = (
+ store.drain(manager, batch_size=args.batch_size)
+ if args.command == "drain"
+ else store.run_job(args.job_id, manager)
+ )
+ print(_json(result))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_batch_writer.py b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_batch_writer.py
new file mode 100644
index 0000000000000000000000000000000000000000..13eaf9afbb07f42fb054455414c66db6e014b67e
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_batch_writer.py
@@ -0,0 +1,4987 @@
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import os
+import re
+import sqlite3
+import sys
+import time
+import unicodedata
+import urllib.error
+import urllib.request
+import uuid
+from contextlib import closing
+from dataclasses import dataclass
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any, Callable, Mapping, Protocol, Sequence
+
+import tmcra_v3_product_writer as _v3_writer
+
+FORBIDDEN_WRITER_FIELDS = _v3_writer.FORBIDDEN_WRITER_FIELDS
+INTERACTION_STATUSES = _v3_writer.INTERACTION_STATUSES
+INTERACTION_TYPES = _v3_writer.INTERACTION_TYPES
+MEMORY_TYPES = {*_v3_writer.MEMORY_TYPES, "belief", "opinion"}
+_v3_writer.MEMORY_TYPES = set(MEMORY_TYPES)
+_BASE_MEMORY_FAMILY = _v3_writer.memory_family
+
+
+def _v4_memory_family(memory_type: str) -> str:
+ if memory_type in {"belief", "opinion"}:
+ return "fact"
+ try:
+ return _BASE_MEMORY_FAMILY(memory_type)
+ except KeyError:
+ return "fact"
+
+
+_v3_writer.memory_family = _v4_memory_family
+OPERATIONS = _v3_writer.OPERATIONS
+POLARITIES = {*_v3_writer.POLARITIES, "neutral"}
+_v3_writer.POLARITIES = set(POLARITIES)
+RESOLUTION_STATES = _v3_writer.RESOLUTION_STATES
+ROLE_RE = re.compile(r"^[a-z][a-z0-9_]{1,127}$")
+_v3_writer.ROLE_RE = ROLE_RE
+TEMPORAL_STATUSES = _v3_writer.TEMPORAL_STATUSES
+ProductWriterError = _v3_writer.ProductWriterError
+_build_v3_graph_records = _v3_writer.build_graph_records
+clean_text = _v3_writer.clean_text
+exact_evidence_spans = _v3_writer.exact_evidence_spans
+exact_source_tokens = _v3_writer.exact_source_tokens
+sha256_text = _v3_writer.sha256_text
+source_token_matches = _v3_writer.source_token_matches
+validate_writer_output = _v3_writer.validate_writer_output
+
+
+class GroundingIntegrityError(ProductWriterError):
+ """A model response broke immutable source/evidence alignment."""
+
+
+BATCH_SCHEMA_VERSION = "tmcra.memory-write-batch.v4"
+PROMPT_VERSION = "tmcra-product-writer-batch-2026-07-14.2"
+RECONCILIATION_SCHEMA_VERSION = "tmcra.memory-reconcile.v4"
+CANDIDATE_SELECTOR_VERSION = "tmcra.v4.lexical-slot-candidates.3"
+DEFAULT_TARGET_TOKENS = 3000
+DEFAULT_MIN_SOFT_TOKENS = 2000
+DEFAULT_MAX_SOFT_TOKENS = 4000
+DEFAULT_HARD_TOKEN_LIMIT = 32768
+DECISIONS = {"insert", "merge_support", "replace_current", "keep_parallel", "challenge", "quarantine"}
+SLOT_DECISIONS = {"bind_existing", "keep_proposed", "quarantine"}
+GRAPH_AUTO_SUPERSESSION_REASONS = {
+ "same_state_revision",
+ "slot_disallows_parallel",
+ "v4_reconciliation_replace_current",
+}
+GRAPH_INJECTED_BENCHMARK_METADATA_KEYS = {
+ "origin_answer_id",
+ "origin_answer_ids",
+ "origin_question_id",
+ "origin_question_ids",
+ "benchmark_id",
+ "gold_label",
+}
+SAFE_VALIDATION_WARNING_CODES = {
+ "identifier_case_normalized",
+ "identifier_separator_normalized",
+ "duplicate_facet_dropped",
+ "duplicate_assertion_merged",
+ "duplicate_interaction_merged",
+ "duplicate_resolution_merged",
+ "optional_facet_dropped",
+ "optional_resolution_dropped",
+ "invalid_assertion_quarantined",
+ "invalid_interaction_quarantined",
+ "invalid_resolution_quarantined",
+}
+
+
+BATCH_SYSTEM_PROMPT = """You are the semantic extraction stage of a production personal-memory system.
+Return exactly one JSON object and no prose. The request contains consecutive messages from one session.
+For each message, source_spans are the only source text; their order reconstructs the exact message. Never
+invent a message, span ID, interaction ID, quote, timestamp, or fact. Do not request or assume an existing
+memory-slot inventory. Never emit benchmark questions, answers, labels, answer-session IDs, judge output,
+passwords, authentication secrets, private keys, or account credentials.
+
+Extract three independent layers for every supplied user or assistant message:
+1. assertions: explicit user self-reports about facts, events, states, beliefs, opinions, preferences, goals,
+ constraints, plans, identity, relationships, possessions, or routines. A question and its presupposition are not assertions.
+ Assistant statements never become user assertions. Every assertion is atomic and cites one supplied eN span.
+ claim_text is a concise, self-contained proposition entailed by that exact span. It must identify the actual
+ subject and value needed to distinguish this fact from other facts in the same span. It is not a quote and must
+ not add information. Split a span into multiple assertions only when their claim_text values are genuinely
+ different facts; never repeat the same claim under multiple keys.
+ The outer user message is a transport envelope, not proof that every sentence inside it was authored by or
+ describes the human user. A pasted or forwarded email, quoted reply, article, resume, transcript, log, signature,
+ or other embedded document retains its local author and subject. Never turn contact details, roles, possessions,
+ plans, preferences, or business facts from a named sender, signatory, quoted speaker, company, or document subject
+ into user assertions unless the surrounding conversational voice explicitly identifies that person/entity as the
+ user. Useful third-party document facts remain in immutable Source; emit no user assertion for them.
+2. interactions: each explicit question, request, reminder, task, clarification, or meaningful feedback.
+ Mixed messages may contain both assertions and interactions. Assistant questions/requests may be interactions;
+ assistant answers, recommendations, apologies, confirmations, and explanations are not new interactions.
+3. resolutions: whether the current message explicitly resolves an unresolved interaction. Use resolved only for
+ a complete answer/result, partial for real progress, and unresolved only for an explicit refusal or inability.
+ Absence of an answer is not resolution evidence. A batch target must point to an earlier message in this batch.
+
+The memory boundary is user-specific and cross-session. Emit an assertion only when it would help a future
+assistant understand this user's life, preferences, commitments, relationships, possessions, routines,
+experiences, current state, or a substantive personal stance. Do not store generic conversational reactions
+such as "that's interesting", "fascinating", "good to know", or "that makes sense". Do not store observations
+about an external topic merely because the user says "I think", "I noticed", or "it's interesting". A belief
+or opinion must express a substantive first-person position that is useful beyond the current topic. A goal or
+plan must be a real user commitment beyond the current turn, not acceptance of advice, a hypothetical, or a
+request to continue the present conversation. An event must involve the user, not only an external historical
+or news event. When in doubt between a generic topic reaction and personal memory, emit no assertion; preserve
+the interaction layer independently.
+
+For assertions, entity_key is the stable subject/domain and attribute_key is the stable property or event kind.
+Use lowercase dot-separated identifiers and never put the changing value in a key. Use replace for mutable slots
+and append for repeatable events. relation, intent, facet role, and about role are lowercase snake_case.
+Durability is a semantic classification made from the source: durable for a standing identity, preference,
+relationship, routine, constraint, or stable long-running state; episodic for a one-off event/task/transient state;
+uncertain when the source does not establish whether it should become long-term memory. Do not use repetition
+count as a durability rule.
+
+Each facet/about quote must be the shortest exact substring of its parent evidence span. Do not output token or
+character coordinates. Omit an optional facet/about entry instead of paraphrasing its quote. Return one message
+entry for every user/assistant input, in exact order, using empty arrays
+when appropriate. The exact wire schema is:
+{"schema_version":"tmcra.memory-write-batch.v4","batch_id":"exact request value","messages":[
+ {"message_id":"exact request value","message_role":"user|assistant","assertions":[
+ {"memory_type":"fact|event|state|belief|opinion|preference|goal|constraint|plan|identity|relationship|possession|routine",
+ "entity_key":"stable.domain","attribute_key":"stable_attribute","operation":"append|replace",
+ "claim_text":"concise self-contained atomic proposition entailed by the cited span",
+ "evidence_span_id":"eN","relation":"snake_case",
+ "temporal_status":"past|current|planned|future|timeless|uncertain",
+ "polarity":"positive|negative|neutral","durability":"durable|episodic|uncertain",
+ "facets":[{"type":"entity|time|quantity|state|location|role","role":"snake_case","quote":"exact substring"}]}],
+ "interactions":[{"interaction_type":"question|request|reminder|task|clarification|feedback",
+ "status":"open|informational","evidence_span_id":"eN","intent":"snake_case",
+ "about":[{"type":"entity|time|quantity|state|location|role","role":"snake_case","quote":"exact substring"}]}],
+ "resolutions":[{"target":{"kind":"existing","interaction_id":"supplied id"},
+ "resolution":"resolved|partial|unresolved","evidence_span_id":"eN"}]}]}
+For a batch-local resolution target, replace target with
+{"kind":"batch","message_id":"earlier exact message id","interaction_index":0}.
+"""
+
+
+def _now() -> str:
+ return datetime.now(timezone.utc).isoformat(timespec="milliseconds")
+
+
+def _json(value: Any) -> str:
+ return json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
+
+
+def _hash_json(value: Any) -> str:
+ return sha256_text(_json(value))
+
+
+def _graph_slot_key(canonical_key: Any) -> str:
+ value = clean_text(canonical_key)
+ if not value:
+ raise ProductWriterError("canonical slot key must not be empty")
+ return value if value.startswith("memory.") else f"memory.{value}"
+
+
+_SLOT_STOP_TOKENS = {
+ "user",
+ "memory",
+ "fact",
+ "event",
+ "state",
+ "preference",
+ "goal",
+ "constraint",
+ "plan",
+ "identity",
+ "relationship",
+ "possession",
+ "routine",
+ "current",
+ "timeless",
+ "replace",
+ "append",
+}
+
+# These words identify broad domains or generic attribute shapes. Sharing only
+# these words is not enough to spend a Pro call on slot binding.
+_BROAD_SLOT_IDENTITY_TOKENS = {
+ "home",
+ "house",
+ "utilities",
+ "utility",
+ "setup",
+ "set",
+ "up",
+ "service",
+ "services",
+ "status",
+ "information",
+ "info",
+ "details",
+ "has",
+ "have",
+ "needs",
+ "need",
+ "uses",
+ "use",
+ "to",
+}
+
+
+def _slot_tokens(*values: Any) -> set[str]:
+ tokens: set[str] = set()
+ for value in values:
+ tokens.update(
+ token
+ for token in re.findall(r"[a-z0-9]+|[\u4e00-\u9fff]", str(value or "").casefold())
+ if token not in _SLOT_STOP_TOKENS
+ and (len(token) > 1 or bool(re.fullmatch(r"[\u4e00-\u9fff]", token)))
+ )
+ return tokens
+
+
+def _strict_json_object(value: str, path: str) -> dict[str, Any]:
+ if not value or not value.strip():
+ raise ProductWriterError(f"{path} must be a non-empty JSON object")
+ try:
+ parsed = json.loads(value)
+ except json.JSONDecodeError as exc:
+ raise ProductWriterError(f"{path} is not strict JSON: {exc}") from exc
+ if not isinstance(parsed, dict):
+ raise ProductWriterError(f"{path} root must be an object")
+ return parsed
+
+
+def _exact_keys(value: Mapping[str, Any], expected: set[str], path: str) -> None:
+ actual = set(value)
+ if actual != expected:
+ raise ProductWriterError(
+ f"{path} keys differ from schema; missing={sorted(expected - actual)}, extra={sorted(actual - expected)}"
+ )
+
+
+def _string(value: Any, path: str, *, allow_empty: bool = False) -> str:
+ if not isinstance(value, str):
+ raise ProductWriterError(f"{path} must be a string")
+ if not allow_empty and not value:
+ raise ProductWriterError(f"{path} must not be empty")
+ if value != value.strip():
+ raise ProductWriterError(f"{path} must not have surrounding whitespace")
+ return value
+
+
+def _integer(value: Any, path: str) -> int:
+ if isinstance(value, bool) or not isinstance(value, int):
+ raise ProductWriterError(f"{path} must be an integer")
+ return value
+
+
+def _enum(value: Any, allowed: set[str], path: str) -> str:
+ result = _string(value, path)
+ if result not in allowed:
+ raise ProductWriterError(f"{path} is unsupported: {result!r}")
+ return result
+
+
+def _audited_enum(
+ value: Any,
+ allowed: set[str],
+ path: str,
+ warnings: list[dict[str, Any]],
+) -> str:
+ if not isinstance(value, str):
+ raise ProductWriterError(f"{path} must be a string")
+ normalized = re.sub(r"_+", "_", value.strip().lower().replace("-", "_").replace(" ", "_"))
+ if normalized not in allowed:
+ raise ProductWriterError(f"{path} is unsupported: {value!r}")
+ if normalized != value:
+ warnings.append(
+ {
+ "path": path,
+ "code": "enum_value_normalized",
+ "detail": f"normalized enum value: {value!r} -> {normalized!r}",
+ "dropped_count": 0,
+ }
+ )
+ return normalized
+
+
+def _audited_item_keys(
+ value: Mapping[str, Any],
+ expected: set[str],
+ path: str,
+ warnings: list[dict[str, Any]],
+ *,
+ optional: set[str] = frozenset(),
+) -> None:
+ actual = set(value)
+ missing = expected - actual - optional
+ if missing:
+ raise ProductWriterError(f"{path} is missing required keys: {sorted(missing)}")
+ extra = actual - expected
+ if extra:
+ warnings.append(
+ {
+ "path": path,
+ "code": "extra_item_fields_ignored",
+ "detail": f"ignored unsupported item fields: {sorted(extra)}",
+ "dropped_count": len(extra),
+ }
+ )
+
+
+def _audited_durability(
+ value: Any,
+ path: str,
+ warnings: list[dict[str, Any]],
+) -> str:
+ if value is None or (isinstance(value, str) and not value.strip()):
+ warnings.append(
+ {
+ "path": path,
+ "code": "durability_defaulted_uncertain",
+ "detail": "missing durability defaulted to uncertain",
+ "dropped_count": 0,
+ }
+ )
+ return "uncertain"
+ try:
+ return _audited_enum(
+ value,
+ {"durable", "episodic", "uncertain"},
+ path,
+ warnings,
+ )
+ except ProductWriterError:
+ normalized = (
+ value.strip().lower().replace("-", "_").replace(" ", "_")
+ if isinstance(value, str)
+ else ""
+ )
+ if normalized in TEMPORAL_STATUSES:
+ warnings.append(
+ {
+ "path": path,
+ "code": "temporal_durability_defaulted_uncertain",
+ "detail": (
+ f"durability contained temporal value {value!r}; "
+ "preserved assertion with uncertain durability"
+ ),
+ "dropped_count": 0,
+ }
+ )
+ return "uncertain"
+ raise
+
+
+def _case_warning(
+ warnings: list[dict[str, Any]] | None,
+ *,
+ path: str,
+ original: str,
+ normalized: str,
+) -> None:
+ if warnings is None:
+ raise ProductWriterError(f"{path} requires unaudited case normalization")
+ warnings.append(
+ {
+ "path": path,
+ "code": "identifier_case_normalized",
+ "detail": f"normalized identifier case: {original!r} -> {normalized!r}",
+ "dropped_count": 0,
+ }
+ )
+
+
+def _symbol_warning(
+ warnings: list[dict[str, Any]] | None,
+ *,
+ path: str,
+ original: str,
+ normalized: str,
+) -> None:
+ if warnings is None:
+ raise ProductWriterError(f"{path} requires unaudited symbol normalization")
+ warnings.append(
+ {
+ "path": path,
+ "code": "identifier_symbol_normalized",
+ "detail": f"normalized identifier symbol: {original!r} -> {normalized!r}",
+ "dropped_count": 0,
+ }
+ )
+
+
+def _role_identifier(
+ value: Any,
+ path: str,
+ warnings: list[dict[str, Any]] | None = None,
+) -> str:
+ result = _string(value, path)
+ if not ROLE_RE.fullmatch(result):
+ normalized = result.lower()
+ if normalized != result and ROLE_RE.fullmatch(normalized):
+ _case_warning(
+ warnings,
+ path=path,
+ original=result,
+ normalized=normalized,
+ )
+ return normalized
+ # Ampersands commonly survive when a grounded brand acronym is copied
+ # into a model-generated label (for example, T&T). This is the only
+ # symbol rewrite accepted here; whitespace and arbitrary punctuation
+ # remain hard failures.
+ if "&" in result and not any(char.isspace() for char in result):
+ symbol_normalized = re.sub(
+ r"_+", "_", result.lower().replace("&", "_and_")
+ ).strip("_")
+ if ROLE_RE.fullmatch(symbol_normalized):
+ _symbol_warning(
+ warnings,
+ path=path,
+ original=result,
+ normalized=symbol_normalized,
+ )
+ return symbol_normalized
+ raise ProductWriterError(f"{path} is not snake_case: {result!r}")
+ return result
+
+
+def _canonical_identifier(value: Any, path: str) -> str:
+ result = _string(value, path)
+ if result != result.lower() or not all(char.isalnum() or char in "_.-" for char in result):
+ raise ProductWriterError(f"{path} is not a canonical identifier: {result!r}")
+ if len(result) < 2 or len(result) > 160 or result[0] not in "abcdefghijklmnopqrstuvwxyz0123456789":
+ raise ProductWriterError(f"{path} is not a canonical identifier: {result!r}")
+ return result
+
+
+@dataclass(frozen=True)
+class SourceMessage:
+ scope_id: str
+ session_id: str
+ session_index: int
+ message_index: int
+ message_id: str
+ role: str
+ timestamp: str
+ content: str
+
+ def request_dict(self) -> dict[str, Any]:
+ return {
+ "message_id": self.message_id,
+ "message_role": self.role,
+ "timestamp": self.timestamp,
+ "source_spans": [
+ {"span_id": span["span_id"], "text": span["text"]}
+ for span in lossless_source_spans(self.content)
+ ],
+ }
+
+
+@dataclass(frozen=True)
+class SourceBatch:
+ scope_id: str
+ session_id: str
+ session_index: int
+ batch_index: int
+ messages: tuple[SourceMessage, ...]
+
+ @property
+ def batch_id(self) -> str:
+ return f"{self.scope_id}:{self.session_id}:b{self.batch_index:04d}"
+
+
+def _timestamp(raw_message: Mapping[str, Any], row: Mapping[str, Any], index: int) -> str:
+ value = raw_message.get("timestamp", raw_message.get("time", ""))
+ if value:
+ return str(value)
+ dates = list(row.get("haystack_dates") or row.get("dates") or [])
+ session_index = int(row.get("session_index", 0) or 0)
+ if session_index < len(dates) and dates[session_index]:
+ try:
+ return _v3_writer.historical_timestamp(dates[session_index], index)
+ except ProductWriterError:
+ return str(dates[session_index])
+ return ""
+
+
+def _row_sessions(row: Mapping[str, Any], row_index: int) -> list[tuple[str, list[Mapping[str, Any]], int]]:
+ forbidden = sorted(FORBIDDEN_WRITER_FIELDS & set(row))
+ if forbidden:
+ raise ProductWriterError(f"input row contains forbidden benchmark fields: {forbidden}")
+ qid = clean_text(row.get("question_id")) or f"row{row_index:04d}"
+ scope_id = f"tmcra_v4:{qid}"
+ if "haystack_sessions" in row:
+ sessions = list(row.get("haystack_sessions") or [])
+ ids = [clean_text(value) for value in list(row.get("haystack_session_ids") or [])]
+ if not ids:
+ ids = [f"session-{index:03d}" for index in range(len(sessions))]
+ if len(ids) != len(sessions):
+ raise ProductWriterError(f"{qid}: session ID count differs from session count")
+ return [(scope_id, list(session or []), index) for index, session in enumerate(sessions)]
+ if "sessions" in row:
+ sessions = list(row.get("sessions") or [])
+ return [
+ (scope_id, list(session or []), index)
+ for index, session in enumerate(sessions)
+ ]
+ messages = list(row.get("messages") or [])
+ session_id = clean_text(row.get("session_id")) or f"session-{row_index:03d}"
+ return [(scope_id, messages, 0)]
+
+
+def normalize_source_inventory(
+ rows: Sequence[Mapping[str, Any]],
+) -> tuple[list[SourceMessage], list[dict[str, Any]]]:
+ output: list[SourceMessage] = []
+ exclusions: list[dict[str, Any]] = []
+ seen: set[tuple[str, str]] = set()
+ for row_index, row in enumerate(rows):
+ if not isinstance(row, Mapping):
+ raise ProductWriterError(f"input row {row_index} must be an object")
+ for scope_id, session, session_index in _row_sessions(row, row_index):
+ qid = scope_id.split(":", 1)[-1]
+ session_ids = [clean_text(value) for value in list(row.get("haystack_session_ids") or [])]
+ session_id = (
+ session_ids[session_index]
+ if session_index < len(session_ids) and session_ids[session_index]
+ else clean_text(row.get("session_id")) or f"session-{session_index:03d}"
+ )
+ for message_index, raw_message in enumerate(session):
+ if not isinstance(raw_message, Mapping):
+ raise ProductWriterError(f"{qid}/s{session_index:03d}/m{message_index:03d}: message must be an object")
+ role = clean_text(raw_message.get("role")).lower()
+ content = str(raw_message.get("content") or "")
+ message_id = f"s{session_index:03d}_m{message_index:03d}"
+ if role not in {"user", "assistant", "system", "tool"}:
+ raise ProductWriterError(f"{qid}/s{session_index:03d}/m{message_index:03d}: invalid role")
+ if not content.strip():
+ exclusions.append(
+ {
+ "scope_id": scope_id,
+ "session_id": session_id,
+ "session_index": session_index,
+ "message_index": message_index,
+ "message_id": message_id,
+ "message_role": role,
+ "reason": "empty_content",
+ "content_sha256": sha256_text(content),
+ }
+ )
+ continue
+ key = (scope_id, message_id)
+ if key in seen:
+ raise ProductWriterError(f"duplicate source message ID: {message_id}")
+ seen.add(key)
+ output.append(
+ SourceMessage(
+ scope_id=scope_id,
+ session_id=session_id,
+ session_index=session_index,
+ message_index=message_index,
+ message_id=message_id,
+ role=role,
+ timestamp=_timestamp(raw_message, {**row, "session_index": session_index}, message_index),
+ content=content,
+ )
+ )
+ return output, exclusions
+
+
+def normalize_source_rows(rows: Sequence[Mapping[str, Any]]) -> list[SourceMessage]:
+ return normalize_source_inventory(rows)[0]
+
+
+def build_batches(
+ messages: Sequence[SourceMessage],
+ *,
+ target_tokens: int = DEFAULT_TARGET_TOKENS,
+ min_soft_tokens: int = DEFAULT_MIN_SOFT_TOKENS,
+ max_soft_tokens: int = DEFAULT_MAX_SOFT_TOKENS,
+ hard_limit_tokens: int = DEFAULT_HARD_TOKEN_LIMIT,
+) -> list[SourceBatch]:
+ if not (0 < min_soft_tokens <= target_tokens <= max_soft_tokens) or hard_limit_tokens <= 0:
+ raise ValueError("batch token limits must satisfy min <= target <= max and hard > 0")
+ batches: list[SourceBatch] = []
+ current: list[SourceMessage] = []
+ current_tokens = 0
+ batch_index_by_session: dict[tuple[str, str], int] = {}
+
+ def flush() -> None:
+ nonlocal current, current_tokens
+ if not current:
+ return
+ first = current[0]
+ key = (first.scope_id, first.session_id)
+ index = batch_index_by_session.get(key, 0)
+ batches.append(SourceBatch(first.scope_id, first.session_id, first.session_index, index, tuple(current)))
+ batch_index_by_session[key] = index + 1
+ current = []
+ current_tokens = 0
+
+ previous_key: tuple[str, str] | None = None
+ for message in messages:
+ key = (message.scope_id, message.session_id)
+ if previous_key != key:
+ flush()
+ previous_key = key
+ token_count = len(exact_source_tokens(message.content))
+ if token_count > hard_limit_tokens:
+ raise ProductWriterError(
+ f"{message.message_id}: source message has {token_count} tokens, over hard limit {hard_limit_tokens}"
+ )
+ if current and current_tokens + token_count > target_tokens:
+ flush()
+ current.append(message)
+ current_tokens += token_count
+ flush()
+ return batches
+
+
+def lossless_source_spans(content: str) -> list[dict[str, Any]]:
+ """Build the only source text representation sent to Flash.
+
+ Evidence spans retain V3's eN IDs. Gap spans preserve whitespace between
+ evidence spans, so the sequence is lossless and non-overlapping without a
+ second full-content or token-string payload.
+ """
+ evidence = exact_evidence_spans(content)
+ if len(evidence) == 1 or (
+ len(evidence) == 2
+ and int(evidence[1]["char_start"]) == 0
+ and int(evidence[1]["char_end"]) == len(content)
+ ):
+ return [{"span_id": "e0", "text": content, "char_start": 0, "char_end": len(content)}]
+ output: list[dict[str, Any]] = []
+ cursor = 0
+ gap_index = 0
+ for span in evidence[1:]:
+ start = int(span["char_start"])
+ if start > cursor:
+ output.append({"span_id": f"gap{gap_index}", "text": content[cursor:start], "char_start": cursor, "char_end": start})
+ gap_index += 1
+ output.append(dict(span))
+ cursor = int(span["char_end"])
+ if cursor < len(content):
+ output.append({"span_id": f"gap{gap_index}", "text": content[cursor:], "char_start": cursor, "char_end": len(content)})
+ if "".join(str(span["text"]) for span in output) != content:
+ raise ProductWriterError("lossless source span sequence does not reconstruct source content")
+ previous_end = 0
+ for span in output:
+ if int(span["char_start"]) != previous_end or int(span["char_end"]) < int(span["char_start"]):
+ raise ProductWriterError("lossless source span sequence overlaps or is unordered")
+ previous_end = int(span["char_end"])
+ return output
+
+
+def build_batch_request(batch: SourceBatch, unresolved_interactions: Sequence[Mapping[str, Any]] = ()) -> dict[str, Any]:
+ return {
+ "schema_version": BATCH_SCHEMA_VERSION,
+ "batch_id": batch.batch_id,
+ "messages": [message.request_dict() for message in batch.messages],
+ "unresolved_interactions": [dict(item) for item in unresolved_interactions],
+ }
+
+
+def _evidence_span(content: str, span_id: str, path: str) -> dict[str, Any]:
+ for span in exact_evidence_spans(content):
+ if span["span_id"] == span_id:
+ return span
+ raise GroundingIntegrityError(
+ f"{path} is not in the current-message evidence catalog: {span_id!r}"
+ )
+
+
+def _validate_facet(
+ value: Any,
+ content: str,
+ parent_span: Mapping[str, Any],
+ path: str,
+ warnings: list[dict[str, Any]] | None = None,
+) -> dict[str, Any]:
+ if not isinstance(value, Mapping):
+ raise ProductWriterError(f"{path} must be an object")
+ _exact_keys(value, {"type", "role", "quote"}, path)
+ facet_type = _enum(value.get("type"), {"entity", "time", "quantity", "state", "location", "role"}, f"{path}.type")
+ role = _role_identifier(value.get("role"), f"{path}.role", warnings)
+ quote = _string(value.get("quote"), f"{path}.quote")
+ parent_start = int(parent_span["char_start"])
+ parent_end = int(parent_span["char_end"])
+ parent_text = content[parent_start:parent_end]
+ relative_start = parent_text.find(quote)
+ if relative_start < 0:
+ raise ProductWriterError(f"{path}.quote is not an exact substring of its parent evidence span")
+ absolute_start = parent_start + relative_start
+ absolute_end = absolute_start + len(quote)
+ tokens = source_token_matches(content)
+ start = next((index for index, token in enumerate(tokens) if token.start() <= absolute_start < token.end()), None)
+ end = next((index for index, token in enumerate(tokens) if token.start() < absolute_end <= token.end()), None)
+ if start is None or end is None or end < start:
+ raise ProductWriterError(f"{path}.quote must overlap source tokens")
+ return {"type": facet_type, "role": role, "token_start": start, "token_end": end}
+
+
+def _validate_evidence(value: Any, content: str, path: str, allowed_ids: set[str] | None = None) -> str:
+ try:
+ span_id = _string(value, path)
+ except ProductWriterError as exc:
+ raise GroundingIntegrityError(str(exc)) from exc
+ if allowed_ids is not None and span_id not in allowed_ids:
+ raise GroundingIntegrityError(
+ f"{path} was not supplied in the lossless source-span sequence: {span_id!r}"
+ )
+ _evidence_span(content, span_id, path)
+ return span_id
+
+
+def _deterministic_interaction_id(scope_id: str, message_id: str, interaction_index: int) -> str:
+ return f"interaction:{scope_id}:{message_id}:{interaction_index}"
+
+
+def _assertion_identity(value: Mapping[str, Any]) -> tuple[Any, ...]:
+ if "canonical_key" in value:
+ canonical_key = str(value["canonical_key"])
+ else:
+ entity = _v3_writer.normalize_canonical_key(str(value["entity_key"])).removeprefix("user.")
+ attribute = _v3_writer.normalize_canonical_key(str(value["attribute_key"]))
+ family = _v3_writer.memory_family(str(value["memory_type"]))
+ canonical_key = f"user.{entity}.{family}.{attribute}"
+ facet_identity = tuple(
+ sorted(
+ (
+ str(facet["type"]),
+ str(facet["role"]),
+ int(facet["token_start"]),
+ int(facet["token_end"]),
+ )
+ for facet in value.get("facets") or []
+ )
+ )
+ return (
+ canonical_key,
+ str(value["memory_type"]),
+ str(value["operation"]),
+ str(value["evidence_span_id"]),
+ str(value["relation"]),
+ str(value["temporal_status"]),
+ str(value["polarity"]),
+ facet_identity,
+ )
+
+
+def _interaction_identity(value: Mapping[str, Any]) -> tuple[str, ...]:
+ return (
+ str(value["interaction_type"]),
+ str(value["status"]),
+ str(value["evidence_span_id"]),
+ str(value["intent"]),
+ )
+
+
+def _facet_quote_lookup(
+ normalized_facets: Sequence[Mapping[str, Any]],
+ raw_facets: Sequence[Mapping[str, Any]],
+ raw_quotes: Sequence[str],
+) -> list[str]:
+ by_key: dict[tuple[Any, ...], list[str]] = {}
+ for facet, quote in zip(raw_facets, raw_quotes):
+ key = (
+ str(facet["type"]),
+ str(facet["role"]),
+ int(facet["token_start"]),
+ int(facet["token_end"]),
+ )
+ by_key.setdefault(key, []).append(str(quote))
+ output = []
+ for facet in normalized_facets:
+ key = (
+ str(facet["type"]),
+ str(facet["role"]),
+ int(facet["token_start"]),
+ int(facet["token_end"]),
+ )
+ candidates = by_key.get(key)
+ if not candidates:
+ raise ProductWriterError("validated facet cannot be mapped back to its exact quote")
+ output.append(min(candidates, key=lambda item: (len(item), item)))
+ return output
+
+
+def validate_batch_response(
+ payload: Mapping[str, Any] | str,
+ batch: SourceBatch,
+ unresolved_interactions: Sequence[Mapping[str, Any]] = (),
+) -> dict[str, Any]:
+ if isinstance(payload, str):
+ try:
+ payload = json.loads(payload)
+ except json.JSONDecodeError as exc:
+ raise ProductWriterError(f"batch response is not strict JSON: {exc}") from exc
+ if not isinstance(payload, Mapping):
+ raise ProductWriterError("batch response root must be an object")
+ _exact_keys(payload, {"schema_version", "batch_id", "messages"}, "root")
+ if payload.get("schema_version") != BATCH_SCHEMA_VERSION:
+ raise ProductWriterError(f"unexpected batch schema: {payload.get('schema_version')!r}")
+ if payload.get("batch_id") != batch.batch_id:
+ raise ProductWriterError("batch response batch_id differs from controller value")
+ raw_messages = payload.get("messages")
+ if not isinstance(raw_messages, list):
+ raise ProductWriterError("root.messages must be an array")
+ expected_messages = [message for message in batch.messages if message.role in {"user", "assistant"}]
+ if len(raw_messages) != len(expected_messages):
+ raise ProductWriterError("batch response must contain exactly one entry for every user or assistant message")
+ existing_ids = {
+ _string(item.get("interaction_id"), "unresolved_interactions[].interaction_id")
+ for item in unresolved_interactions
+ }
+ normalized_messages: list[dict[str, Any]] = []
+ # Raw batch-local interaction indexes may collapse during duplicate merge.
+ # This map preserves the model-facing raw ID while resolving it to the
+ # interaction ID that will actually be persisted.
+ prior_interactions: dict[str, str] = {}
+ prior_message_ids: set[str] = set()
+ for response_index, (raw_message, source) in enumerate(zip(raw_messages, expected_messages)):
+ path = f"root.messages[{response_index}]"
+ controller_warnings: list[dict[str, Any]] = []
+ if not isinstance(raw_message, Mapping):
+ raise ProductWriterError(f"{path} must be an object")
+ _audited_item_keys(
+ raw_message,
+ {"message_id", "message_role", "assertions", "interactions", "resolutions"},
+ path,
+ controller_warnings,
+ optional={"assertions", "interactions", "resolutions"},
+ )
+ if raw_message.get("message_id") != source.message_id:
+ raise ProductWriterError(f"{path}.message_id differs from controller value")
+ if raw_message.get("message_role") != source.role:
+ raise ProductWriterError(f"{path}.message_role differs from controller value")
+ assertions = raw_message.get("assertions")
+ interactions = raw_message.get("interactions")
+ resolutions = raw_message.get("resolutions")
+ for name, values in (("assertions", assertions), ("interactions", interactions), ("resolutions", resolutions)):
+ if not isinstance(values, list):
+ controller_warnings.append(
+ {
+ "path": f"{path}.{name}",
+ "code": "invalid_item_collection_defaulted_empty",
+ "detail": "missing or non-array item collection defaulted to []",
+ "dropped_count": 1 if values is not None else 0,
+ }
+ )
+ assertions = assertions if isinstance(assertions, list) else []
+ interactions = interactions if isinstance(interactions, list) else []
+ resolutions = resolutions if isinstance(resolutions, list) else []
+ raw_v3_assertions: list[dict[str, Any]] = []
+ assertion_facet_quotes: list[list[str]] = []
+ assertion_claim_texts: list[str] = []
+ durability: list[str] = []
+ if source.role == "assistant" and assertions:
+ controller_warnings.append(
+ {
+ "path": f"{path}.assertions",
+ "code": "assistant_assertions_dropped",
+ "detail": (
+ "assistant-authored assertions cannot become user memory; "
+ "immutable source was retained"
+ ),
+ "dropped_count": len(assertions),
+ }
+ )
+ assertions = []
+ allowed_evidence_ids = {
+ str(span["span_id"])
+ for span in lossless_source_spans(source.content)
+ if str(span["span_id"]).startswith("e")
+ }
+ for assertion_index, raw_assertion in enumerate(assertions):
+ assertion_path = f"{path}.assertions[{assertion_index}]"
+ try:
+ if not isinstance(raw_assertion, Mapping):
+ raise ProductWriterError(f"{assertion_path} must be an object")
+ raw_assertion = dict(raw_assertion)
+ _audited_item_keys(
+ raw_assertion,
+ {
+ "memory_type", "entity_key", "attribute_key", "operation",
+ "claim_text", "evidence_span_id", "relation", "temporal_status",
+ "polarity", "durability", "facets",
+ },
+ assertion_path,
+ controller_warnings,
+ optional={"durability", "facets"},
+ )
+ if not isinstance(raw_assertion.get("facets"), list):
+ controller_warnings.append(
+ {
+ "path": f"{assertion_path}.facets",
+ "code": "optional_facets_defaulted_empty",
+ "detail": "missing or non-array optional facets defaulted to []",
+ "dropped_count": 0,
+ }
+ )
+ raw_assertion["facets"] = []
+ claim_text = _string(
+ raw_assertion.get("claim_text"), f"{assertion_path}.claim_text"
+ )
+ if len(claim_text) > 1000:
+ raise ProductWriterError(
+ f"{assertion_path}.claim_text exceeds 1000 characters"
+ )
+ memory_type = _role_identifier(
+ raw_assertion.get("memory_type"),
+ f"{assertion_path}.memory_type",
+ controller_warnings,
+ )
+ if memory_type not in MEMORY_TYPES:
+ MEMORY_TYPES.add(memory_type)
+ _v3_writer.MEMORY_TYPES.add(memory_type)
+ controller_warnings.append(
+ {
+ "path": f"{assertion_path}.memory_type",
+ "code": "memory_type_extension_accepted",
+ "detail": f"accepted grounded snake_case extension: {memory_type}",
+ "dropped_count": 0,
+ }
+ )
+ evidence_span_id = _validate_evidence(
+ raw_assertion.get("evidence_span_id"),
+ source.content,
+ f"{assertion_path}.evidence_span_id",
+ allowed_evidence_ids,
+ )
+ v3_assertion = {
+ "memory_type": memory_type,
+ "entity_key": _canonical_identifier(raw_assertion.get("entity_key"), f"{assertion_path}.entity_key"),
+ "attribute_key": _canonical_identifier(raw_assertion.get("attribute_key"), f"{assertion_path}.attribute_key"),
+ "operation": _audited_enum(raw_assertion.get("operation"), set(OPERATIONS), f"{assertion_path}.operation", controller_warnings),
+ "evidence_span_id": evidence_span_id,
+ "relation": _role_identifier(raw_assertion.get("relation"), f"{assertion_path}.relation", controller_warnings),
+ "temporal_status": _audited_enum(raw_assertion.get("temporal_status"), set(TEMPORAL_STATUSES), f"{assertion_path}.temporal_status", controller_warnings),
+ "polarity": _audited_enum(raw_assertion.get("polarity"), set(POLARITIES), f"{assertion_path}.polarity", controller_warnings),
+ "facets": [],
+ }
+ assertion_durability = _audited_durability(
+ raw_assertion.get("durability"),
+ f"{assertion_path}.durability",
+ controller_warnings,
+ )
+ parent_span = _evidence_span(
+ source.content,
+ evidence_span_id,
+ f"{assertion_path}.evidence_span_id",
+ )
+ kept_facets: list[dict[str, Any]] = []
+ kept_facet_quotes: list[str] = []
+ for facet_index, item in enumerate(raw_assertion["facets"]):
+ facet_path = f"{assertion_path}.facets[{facet_index}]"
+ try:
+ kept_facets.append(
+ _validate_facet(
+ item,
+ source.content,
+ parent_span,
+ facet_path,
+ controller_warnings,
+ )
+ )
+ kept_facet_quotes.append(
+ _string(item.get("quote"), f"{facet_path}.quote")
+ )
+ except ProductWriterError as exc:
+ controller_warnings.append(
+ {
+ "path": facet_path,
+ "code": "optional_facet_dropped",
+ "detail": str(exc),
+ "dropped_count": 1,
+ }
+ )
+ v3_assertion["facets"] = kept_facets
+ assertion_facet_quotes.append(kept_facet_quotes)
+ assertion_claim_texts.append(claim_text)
+ durability.append(assertion_durability)
+ raw_v3_assertions.append(v3_assertion)
+ except GroundingIntegrityError:
+ raise
+ except ProductWriterError as exc:
+ controller_warnings.append(
+ {
+ "path": assertion_path,
+ "code": "invalid_assertion_quarantined",
+ "detail": str(exc),
+ "dropped_count": 1,
+ }
+ )
+ raw_v3_interactions: list[dict[str, Any]] = []
+ interaction_about_quotes: list[list[str]] = []
+ interaction_source_indexes: list[int] = []
+ for interaction_index, raw_interaction in enumerate(interactions):
+ interaction_path = f"{path}.interactions[{interaction_index}]"
+ try:
+ if not isinstance(raw_interaction, Mapping):
+ raise ProductWriterError(f"{interaction_path} must be an object")
+ raw_interaction = dict(raw_interaction)
+ _audited_item_keys(
+ raw_interaction,
+ {"interaction_type", "status", "evidence_span_id", "intent", "about"},
+ interaction_path,
+ controller_warnings,
+ optional={"about"},
+ )
+ if not isinstance(raw_interaction.get("about"), list):
+ controller_warnings.append(
+ {
+ "path": f"{interaction_path}.about",
+ "code": "optional_about_defaulted_empty",
+ "detail": "missing or non-array optional about defaulted to []",
+ "dropped_count": 0,
+ }
+ )
+ raw_interaction["about"] = []
+ interaction_evidence_id = _validate_evidence(
+ raw_interaction.get("evidence_span_id"),
+ source.content,
+ f"{interaction_path}.evidence_span_id",
+ allowed_evidence_ids,
+ )
+ interaction_parent_span = _evidence_span(
+ source.content,
+ interaction_evidence_id,
+ f"{interaction_path}.evidence_span_id",
+ )
+ kept_about: list[dict[str, Any]] = []
+ kept_about_quotes: list[str] = []
+ for about_index, item in enumerate(raw_interaction["about"]):
+ about_path = f"{interaction_path}.about[{about_index}]"
+ try:
+ kept_about.append(
+ _validate_facet(
+ item,
+ source.content,
+ interaction_parent_span,
+ about_path,
+ controller_warnings,
+ )
+ )
+ kept_about_quotes.append(
+ _string(item.get("quote"), f"{about_path}.quote")
+ )
+ except ProductWriterError as exc:
+ controller_warnings.append(
+ {
+ "path": about_path,
+ "code": "optional_facet_dropped",
+ "detail": str(exc),
+ "dropped_count": 1,
+ }
+ )
+ interaction_about_quotes.append(kept_about_quotes)
+ interaction_source_indexes.append(interaction_index)
+ raw_v3_interactions.append(
+ {
+ "interaction_type": _audited_enum(raw_interaction.get("interaction_type"), set(INTERACTION_TYPES), f"{interaction_path}.interaction_type", controller_warnings),
+ "status": _audited_enum(raw_interaction.get("status"), set(INTERACTION_STATUSES), f"{interaction_path}.status", controller_warnings),
+ "evidence_span_id": interaction_evidence_id,
+ "intent": _role_identifier(raw_interaction.get("intent"), f"{interaction_path}.intent", controller_warnings),
+ "about": kept_about,
+ }
+ )
+ except GroundingIntegrityError:
+ raise
+ except ProductWriterError as exc:
+ controller_warnings.append(
+ {
+ "path": interaction_path,
+ "code": "invalid_interaction_quarantined",
+ "detail": str(exc),
+ "dropped_count": 1,
+ }
+ )
+ raw_v3_resolutions: list[dict[str, Any]] = []
+ for resolution_index, raw_resolution in enumerate(resolutions):
+ resolution_path = f"{path}.resolutions[{resolution_index}]"
+ try:
+ if not isinstance(raw_resolution, Mapping):
+ raise ProductWriterError(f"{resolution_path} must be an object")
+ _exact_keys(raw_resolution, {"target", "resolution", "evidence_span_id"}, resolution_path)
+ target = raw_resolution.get("target")
+ if not isinstance(target, Mapping):
+ raise ProductWriterError(f"{resolution_path}.target must be an object")
+ kind = target.get("kind")
+ if kind == "existing":
+ _exact_keys(target, {"kind", "interaction_id"}, f"{resolution_path}.target")
+ interaction_id = _string(target.get("interaction_id"), f"{resolution_path}.target.interaction_id")
+ if interaction_id not in existing_ids:
+ raise ProductWriterError(f"{resolution_path} targets an unknown existing interaction")
+ elif kind == "batch":
+ _exact_keys(target, {"kind", "message_id", "interaction_index"}, f"{resolution_path}.target")
+ target_message_id = _string(target.get("message_id"), f"{resolution_path}.target.message_id")
+ target_index = _integer(target.get("interaction_index"), f"{resolution_path}.target.interaction_index")
+ if target_message_id not in prior_message_ids or target_index < 0:
+ raise ProductWriterError(f"{resolution_path} batch target must reference an earlier message and interaction")
+ raw_target_id = _deterministic_interaction_id(
+ batch.scope_id, target_message_id, target_index
+ )
+ interaction_id = prior_interactions.get(raw_target_id, "")
+ if not interaction_id:
+ raise ProductWriterError(f"{resolution_path} batch target interaction index is invalid")
+ else:
+ raise ProductWriterError(f"{resolution_path}.target.kind must be existing or batch")
+ raw_v3_resolutions.append(
+ {
+ "interaction_id": interaction_id,
+ "resolution": _enum(raw_resolution.get("resolution"), set(RESOLUTION_STATES), f"{resolution_path}.resolution"),
+ "evidence_span_id": _validate_evidence(raw_resolution.get("evidence_span_id"), source.content, f"{resolution_path}.evidence_span_id", allowed_evidence_ids),
+ }
+ )
+ except ProductWriterError as exc:
+ controller_warnings.append(
+ {
+ "path": resolution_path,
+ "code": "optional_resolution_dropped",
+ "detail": str(exc),
+ "dropped_count": 1,
+ }
+ )
+ v3_payload = {
+ "schema_version": "tmcra.memory-write.v3.4",
+ "message_role": source.role,
+ "assertions": raw_v3_assertions,
+ "interactions": raw_v3_interactions,
+ "resolutions": raw_v3_resolutions,
+ }
+ normalized = validate_writer_output(
+ v3_payload,
+ source.content,
+ message_role=source.role,
+ pending_interaction_ids=[*existing_ids, *set(prior_interactions.values())],
+ )
+ warning_codes = {
+ str(warning.get("code"))
+ for warning in normalized.get("validation_warnings") or []
+ }
+ unsupported_warnings = sorted(warning_codes - SAFE_VALIDATION_WARNING_CODES)
+ if unsupported_warnings:
+ raise ProductWriterError(
+ f"{path} has unsafe V3 validation warnings: {unsupported_warnings}"
+ )
+ normalized["validation_warnings"] = [
+ *list(normalized.get("validation_warnings") or []),
+ *controller_warnings,
+ ]
+
+ normalized_durability: list[str] = []
+ for normalized_assertion in normalized.get("assertions") or []:
+ matching = [
+ index
+ for index, raw_assertion in enumerate(raw_v3_assertions)
+ if _assertion_identity(raw_assertion)
+ == _assertion_identity(normalized_assertion)
+ ]
+ if not matching:
+ raise ProductWriterError(
+ f"{path} validated assertion cannot be mapped to its durability"
+ )
+ durability_values = {durability[index] for index in matching}
+ if len(durability_values) > 1:
+ normalized["validation_warnings"].append(
+ {
+ "path": f"{path}.assertions",
+ "code": "conflicting_durability_defaulted_uncertain",
+ "detail": (
+ "equivalent grounded assertions used conflicting "
+ "durability values; defaulted to uncertain"
+ ),
+ "dropped_count": 0,
+ }
+ )
+ normalized_durability.append(
+ next(iter(durability_values))
+ if len(durability_values) == 1
+ else "uncertain"
+ )
+ claim_values = {
+ assertion_claim_texts[index]
+ for index in matching
+ }
+ if len(claim_values) > 1:
+ normalized["validation_warnings"].append(
+ {
+ "path": f"{path}.assertions",
+ "code": "duplicate_claim_text_canonicalized",
+ "detail": "equivalent grounded assertions used different claim_text values; selected deterministically",
+ "dropped_count": len(claim_values) - 1,
+ }
+ )
+ normalized_assertion["claim_text"] = min(
+ claim_values,
+ key=lambda item: (len(item), item.casefold(), item),
+ )
+ raw_facets = [
+ facet
+ for index in matching
+ for facet in raw_v3_assertions[index]["facets"]
+ ]
+ raw_quotes = [
+ quote
+ for index in matching
+ for quote in assertion_facet_quotes[index]
+ ]
+ exact_quotes = _facet_quote_lookup(
+ list(normalized_assertion.get("facets") or []),
+ raw_facets,
+ raw_quotes,
+ )
+ for facet, quote in zip(
+ normalized_assertion.get("facets") or [], exact_quotes
+ ):
+ facet["quote"] = quote
+
+ deduplicated_assertions: list[dict[str, Any]] = []
+ deduplicated_durability: list[str] = []
+ assertion_by_grounded_claim: dict[tuple[str, str], int] = {}
+ for normalized_assertion, assertion_durability in zip(
+ normalized.get("assertions") or [], normalized_durability
+ ):
+ claim_key = (
+ _normalized_claim(str(normalized_assertion["claim_text"])),
+ _normalized_evidence(str(normalized_assertion["evidence_quote"])),
+ )
+ existing_index = assertion_by_grounded_claim.get(claim_key)
+ if existing_index is None:
+ assertion_by_grounded_claim[claim_key] = len(deduplicated_assertions)
+ deduplicated_assertions.append(dict(normalized_assertion))
+ deduplicated_durability.append(assertion_durability)
+ continue
+ if deduplicated_durability[existing_index] != assertion_durability:
+ deduplicated_durability[existing_index] = "uncertain"
+ normalized.setdefault("validation_warnings", []).append(
+ {
+ "path": f"{path}.assertions",
+ "code": "conflicting_durability_defaulted_uncertain",
+ "detail": (
+ "duplicate atomic claim used conflicting durability "
+ "values; defaulted to uncertain"
+ ),
+ "dropped_count": 0,
+ }
+ )
+ existing = deduplicated_assertions[existing_index]
+ for facet in normalized_assertion.get("facets") or []:
+ if facet not in existing["facets"]:
+ existing["facets"].append(facet)
+ normalized.setdefault("validation_warnings", []).append(
+ {
+ "path": f"{path}.assertions",
+ "code": "duplicate_atomic_claim_merged",
+ "detail": "same atomic claim and evidence were emitted under multiple slots",
+ "dropped_count": 1,
+ }
+ )
+ normalized["assertions"] = deduplicated_assertions
+ normalized_durability = deduplicated_durability
+
+ for normalized_interaction_index, normalized_interaction in enumerate(
+ normalized.get("interactions") or []
+ ):
+ matching = [
+ index
+ for index, raw_interaction in enumerate(raw_v3_interactions)
+ if _interaction_identity(raw_interaction)
+ == _interaction_identity(normalized_interaction)
+ ]
+ if not matching:
+ raise ProductWriterError(
+ f"{path} validated interaction cannot be mapped to exact about quotes"
+ )
+ raw_about = [
+ facet
+ for index in matching
+ for facet in raw_v3_interactions[index]["about"]
+ ]
+ raw_quotes = [
+ quote
+ for index in matching
+ for quote in interaction_about_quotes[index]
+ ]
+ exact_quotes = _facet_quote_lookup(
+ list(normalized_interaction.get("about") or []),
+ raw_about,
+ raw_quotes,
+ )
+ for facet, quote in zip(
+ normalized_interaction.get("about") or [], exact_quotes
+ ):
+ facet["quote"] = quote
+ persisted_interaction_id = _deterministic_interaction_id(
+ batch.scope_id, source.message_id, normalized_interaction_index
+ )
+ for raw_interaction_index in matching:
+ source_interaction_index = interaction_source_indexes[
+ raw_interaction_index
+ ]
+ raw_interaction_id = _deterministic_interaction_id(
+ batch.scope_id, source.message_id, source_interaction_index
+ )
+ prior_interactions[raw_interaction_id] = persisted_interaction_id
+ normalized_messages.append(
+ {
+ "message_id": source.message_id,
+ "message_role": source.role,
+ "v3": normalized,
+ "durability": normalized_durability,
+ }
+ )
+ prior_message_ids.add(source.message_id)
+ return {
+ "schema_version": BATCH_SCHEMA_VERSION,
+ "batch_id": batch.batch_id,
+ "messages": normalized_messages,
+ }
+
+
+class BatchClient(Protocol):
+ def complete(self, payload: Mapping[str, Any]) -> Any:
+ ...
+
+
+class ReconciliationClient(Protocol):
+ def reconcile(self, payload: Mapping[str, Any]) -> Any:
+ ...
+
+
+class BatchAPIError(ProductWriterError):
+ def __init__(self, message: str, *, metadata: Mapping[str, Any]) -> None:
+ super().__init__(message)
+ self.metadata = dict(metadata)
+
+
+class DeepSeekBatchClient:
+ """One-shot OpenAI-compatible client; no retry or fallback policy lives here."""
+
+ def __init__(self, *, base_url: str, model: str, api_keys: Sequence[str], timeout: float = 180.0, max_tokens: int = 8192) -> None:
+ self.base_url = base_url.rstrip("/")
+ self.model = model
+ self.api_keys = [clean_text(value) for value in api_keys if clean_text(value)]
+ self.timeout = float(timeout)
+ self.max_tokens = int(max_tokens)
+ self.call_count = 0
+ if not self.base_url or not self.api_keys or self.timeout <= 0 or self.max_tokens <= 0:
+ raise ProductWriterError("base URL and API key pool are required")
+ if self.model not in {"deepseek-v4-flash", "deepseek-v4-pro"}:
+ raise ProductWriterError(f"unsupported V4 writer model: {self.model!r}")
+
+ @staticmethod
+ def _usage(value: Any) -> dict[str, int]:
+ if not isinstance(value, Mapping):
+ raise ProductWriterError("DeepSeek success response lacks usage")
+
+ def count(name: str, *aliases: str) -> int:
+ raw = next((value.get(key) for key in (name, *aliases) if value.get(key) is not None), 0)
+ if isinstance(raw, bool) or not isinstance(raw, (int, float)) or int(raw) < 0:
+ raise ProductWriterError(f"DeepSeek usage.{name} is invalid")
+ return int(raw)
+
+ prompt = count("prompt_tokens", "input_tokens")
+ completion = count("completion_tokens", "output_tokens")
+ hit = count(
+ "prompt_cache_hit_tokens", "cache_read_input_tokens", "cached_tokens"
+ )
+ miss_value_present = any(
+ value.get(key) is not None
+ for key in ("prompt_cache_miss_tokens", "cache_miss_input_tokens")
+ )
+ miss = count("prompt_cache_miss_tokens", "cache_miss_input_tokens")
+ if hit > prompt or (miss_value_present and hit + miss != prompt):
+ raise ProductWriterError("DeepSeek cache usage does not balance prompt tokens")
+ if not miss_value_present:
+ miss = prompt - hit
+ return {
+ "prompt_tokens": prompt,
+ "completion_tokens": completion,
+ "prompt_cache_hit_tokens": hit,
+ "prompt_cache_miss_tokens": miss,
+ "total_tokens": count("total_tokens") or prompt + completion,
+ }
+
+ def _complete(self, *, model: str, system_prompt: str, payload: Mapping[str, Any], stage: str) -> tuple[str, dict[str, Any]]:
+ key_index = self.call_count % len(self.api_keys)
+ self.call_count += 1
+ request_payload = {
+ "model": model,
+ "messages": [
+ {"role": "system", "content": system_prompt},
+ {"role": "user", "content": _json(payload)},
+ ],
+ "temperature": 0,
+ "max_tokens": self.max_tokens,
+ "response_format": {"type": "json_object"},
+ "thinking": {"type": "disabled"},
+ "enable_thinking": False,
+ }
+ physical_call_id = "dsc_" + uuid.uuid4().hex
+ request_sha256 = sha256_text(_json(request_payload))
+ started = time.time()
+ base_metadata = {
+ "physical_call_id": physical_call_id,
+ "physical_api_call": True,
+ "physical_api_calls": 1,
+ "stage": stage,
+ "model": model,
+ "api_key_index": key_index,
+ "request_sha256": request_sha256,
+ "started_at": started,
+ }
+ request = urllib.request.Request(
+ f"{self.base_url}/chat/completions",
+ data=json.dumps(request_payload, ensure_ascii=False).encode("utf-8"),
+ headers={"Content-Type": "application/json", "Authorization": f"Bearer {self.api_keys[key_index]}"},
+ method="POST",
+ )
+ try:
+ with urllib.request.urlopen(request, timeout=self.timeout) as response:
+ http_status = int(response.getcode())
+ raw_http = response.read().decode("utf-8")
+ except urllib.error.HTTPError as exc:
+ detail = exc.read().decode("utf-8", "replace")[:2000]
+ metadata = {
+ **base_metadata,
+ "status": "http_error",
+ "http_status": int(exc.code),
+ "latency_seconds": round(time.time() - started, 3),
+ "error": detail,
+ }
+ raise BatchAPIError(f"{stage} HTTP {exc.code}: {detail}", metadata=metadata) from exc
+ except (urllib.error.URLError, TimeoutError, OSError) as exc:
+ metadata = {
+ **base_metadata,
+ "status": "request_error",
+ "latency_seconds": round(time.time() - started, 3),
+ "error": f"{exc.__class__.__name__}: {exc}",
+ }
+ raise BatchAPIError(f"{stage} request failed: {exc}", metadata=metadata) from exc
+ try:
+ body = json.loads(raw_http)
+ except (UnicodeDecodeError, json.JSONDecodeError) as exc:
+ metadata = {
+ **base_metadata,
+ "status": "invalid_http_json",
+ "http_status": http_status,
+ "latency_seconds": round(time.time() - started, 3),
+ "response_sha256": sha256_text(raw_http),
+ }
+ raise BatchAPIError(f"{stage} returned invalid HTTP JSON", metadata=metadata) from exc
+ if not isinstance(body, Mapping):
+ raise BatchAPIError(
+ f"{stage} response root is not an object",
+ metadata={**base_metadata, "status": "invalid_response", "http_status": http_status},
+ )
+ choices = body.get("choices")
+ if not isinstance(choices, list) or len(choices) != 1 or not isinstance(choices[0], Mapping):
+ raise BatchAPIError(
+ f"{stage} response must contain exactly one choice",
+ metadata={**base_metadata, "status": "invalid_response", "http_status": http_status},
+ )
+ choice = choices[0]
+ message = choice.get("message")
+ content = message.get("content") if isinstance(message, Mapping) else None
+ finish_reason = clean_text(choice.get("finish_reason"))
+ try:
+ usage = self._usage(body.get("usage"))
+ except ProductWriterError as exc:
+ raise BatchAPIError(
+ f"{stage} response usage is invalid: {exc}",
+ metadata={**base_metadata, "status": "invalid_usage", "http_status": http_status},
+ ) from exc
+ metadata = {
+ **base_metadata,
+ **usage,
+ "usage": usage,
+ "status": "completed",
+ "http_status": http_status,
+ "response_id": clean_text(body.get("id")),
+ "latency_seconds": round(time.time() - started, 3),
+ "response_sha256": sha256_text(content if isinstance(content, str) else raw_http),
+ "finish_reason": finish_reason,
+ }
+ if not isinstance(content, str) or not content or finish_reason != "stop":
+ metadata["status"] = "incomplete_response"
+ raise BatchAPIError(
+ f"{stage} response was not a clean JSON completion", metadata=metadata
+ )
+ return content, metadata
+
+ def complete(self, payload: Mapping[str, Any]) -> tuple[str, dict[str, Any]]:
+ if self.model != "deepseek-v4-flash":
+ raise ProductWriterError("batch extraction client must use deepseek-v4-flash")
+ return self._complete(model=self.model, system_prompt=BATCH_SYSTEM_PROMPT, payload=payload, stage="batch_flash")
+
+ def reconcile(self, payload: Mapping[str, Any]) -> tuple[str, dict[str, Any]]:
+ if self.model != "deepseek-v4-pro":
+ raise ProductWriterError("reconciliation client must use deepseek-v4-pro")
+ prompt = (
+ "You bind one new cited assertion to a compact controller-retrieved candidate-slot set. "
+ "Use only supplied source quotes and candidate IDs. Return exactly one JSON object and no prose: "
+ '{"slot_decision":"bind_existing|keep_proposed|quarantine",'
+ '"selected_memory_id":"candidate ID or empty string",'
+ '"decision":"insert|merge_support|replace_current|keep_parallel|challenge|quarantine"}. '
+ "bind_existing means the new assertion is the same real-world memory slot as the selected candidate. "
+ "keep_proposed means none of the candidates is the same slot and requires decision=insert with an empty "
+ "selected_memory_id. quarantine means unsafe or ungrounded and requires decision=quarantine. For a bound "
+ "slot: merge_support means the atomic claim is the same fact and only its new evidence should be attached; "
+ "replace_current is a clear update, keep_parallel means simultaneous values, and challenge means "
+ "conflicting evidence without a winner. When exact_slot_match is true, slot identity is already fixed: "
+ "use bind_existing with a supplied ID, and use keep_parallel rather than insert for an independent value. "
+ "Never select an ID outside the supplied candidates."
+ )
+ return self._complete(model=self.model, system_prompt=prompt, payload=payload, stage="reconciliation_pro")
+
+
+class V4BatchStore:
+ def __init__(self, path: Path) -> None:
+ self.path = Path(path)
+ self.path.parent.mkdir(parents=True, exist_ok=True)
+ self._initialize()
+
+ def _connect(self) -> sqlite3.Connection:
+ connection = sqlite3.connect(self.path, timeout=30)
+ connection.row_factory = sqlite3.Row
+ return connection
+
+ def _initialize(self) -> None:
+ with closing(self._connect()) as connection:
+ connection.executescript(
+ """
+ CREATE TABLE IF NOT EXISTS v4_source_journal (
+ scope_id TEXT NOT NULL, session_id TEXT NOT NULL, message_id TEXT NOT NULL,
+ session_index INTEGER NOT NULL, message_index INTEGER NOT NULL, message_role TEXT NOT NULL,
+ timestamp TEXT NOT NULL, content TEXT NOT NULL, content_sha256 TEXT NOT NULL,
+ status TEXT NOT NULL, source_record_id TEXT NOT NULL DEFAULT '', source_turn_index INTEGER NOT NULL DEFAULT 0,
+ source_persisted_at TEXT NOT NULL DEFAULT '', enrichment_error TEXT NOT NULL DEFAULT '',
+ created_at TEXT NOT NULL, updated_at TEXT NOT NULL,
+ PRIMARY KEY (scope_id, message_id)
+ );
+ CREATE TABLE IF NOT EXISTS v4_batch_journal (
+ batch_id TEXT PRIMARY KEY, scope_id TEXT NOT NULL, session_id TEXT NOT NULL,
+ batch_index INTEGER NOT NULL, request_json TEXT NOT NULL, request_sha256 TEXT NOT NULL,
+ status TEXT NOT NULL, api_started_at TEXT NOT NULL DEFAULT '', response_json TEXT NOT NULL DEFAULT '',
+ response_sha256 TEXT NOT NULL DEFAULT '', response_metadata_json TEXT NOT NULL DEFAULT '{}',
+ error TEXT NOT NULL DEFAULT '', created_at TEXT NOT NULL, updated_at TEXT NOT NULL
+ );
+ CREATE TABLE IF NOT EXISTS v4_interactions (
+ interaction_id TEXT PRIMARY KEY, scope_id TEXT NOT NULL, session_id TEXT NOT NULL,
+ message_id TEXT NOT NULL, interaction_index INTEGER NOT NULL, message_role TEXT NOT NULL,
+ interaction_json TEXT NOT NULL, status TEXT NOT NULL, resolution_history_json TEXT NOT NULL
+ );
+ CREATE TABLE IF NOT EXISTS v4_reconciliation_jobs (
+ job_id TEXT PRIMARY KEY, scope_id TEXT NOT NULL, batch_id TEXT NOT NULL,
+ message_id TEXT NOT NULL DEFAULT '', canonical_slot_key TEXT NOT NULL,
+ assertion_index INTEGER NOT NULL, request_json TEXT NOT NULL,
+ status TEXT NOT NULL, decision TEXT NOT NULL DEFAULT '', response_json TEXT NOT NULL DEFAULT '',
+ response_metadata_json TEXT NOT NULL DEFAULT '', error TEXT NOT NULL DEFAULT '',
+ created_at TEXT NOT NULL, updated_at TEXT NOT NULL
+ );
+ CREATE TABLE IF NOT EXISTS v4_message_commit_journal (
+ commit_id TEXT PRIMARY KEY, batch_id TEXT NOT NULL,
+ scope_id TEXT NOT NULL, session_id TEXT NOT NULL,
+ message_id TEXT NOT NULL, message_index INTEGER NOT NULL,
+ response_sha256 TEXT NOT NULL, plan_json TEXT NOT NULL DEFAULT '',
+ plan_sha256 TEXT NOT NULL DEFAULT '', status TEXT NOT NULL,
+ semantic_committed INTEGER NOT NULL DEFAULT 0,
+ error TEXT NOT NULL DEFAULT '', created_at TEXT NOT NULL,
+ updated_at TEXT NOT NULL,
+ UNIQUE(scope_id, message_id)
+ );
+ """
+ )
+ connection.execute("DROP TABLE IF EXISTS v4_source_records")
+ connection.execute("DROP TABLE IF EXISTS v4_fast_assertion_leaves")
+ columns = {str(row[1]) for row in connection.execute("PRAGMA table_info(v4_source_journal)")}
+ if "source_record_id" not in columns:
+ connection.execute("ALTER TABLE v4_source_journal ADD COLUMN source_record_id TEXT NOT NULL DEFAULT ''")
+ if "source_turn_index" not in columns:
+ connection.execute("ALTER TABLE v4_source_journal ADD COLUMN source_turn_index INTEGER NOT NULL DEFAULT 0")
+ if "enrichment_error" not in columns:
+ connection.execute("ALTER TABLE v4_source_journal ADD COLUMN enrichment_error TEXT NOT NULL DEFAULT ''")
+ if "source_persisted_at" not in columns:
+ connection.execute(
+ "ALTER TABLE v4_source_journal ADD COLUMN source_persisted_at TEXT NOT NULL DEFAULT ''"
+ )
+ reconciliation_columns = {
+ str(row[1])
+ for row in connection.execute("PRAGMA table_info(v4_reconciliation_jobs)")
+ }
+ if "message_id" not in reconciliation_columns:
+ connection.execute(
+ "ALTER TABLE v4_reconciliation_jobs ADD COLUMN message_id TEXT NOT NULL DEFAULT ''"
+ )
+
+ def prepare(self, batch: SourceBatch, request: Mapping[str, Any]) -> sqlite3.Row:
+ request_json = _json(request)
+ request_hash = sha256_text(request_json)
+ now = _now()
+ with closing(self._connect()) as connection:
+ connection.execute("BEGIN IMMEDIATE")
+ for message in batch.messages:
+ existing = connection.execute(
+ "SELECT * FROM v4_source_journal WHERE scope_id=? AND message_id=?",
+ (message.scope_id, message.message_id),
+ ).fetchone()
+ if existing is not None:
+ if existing["content_sha256"] != sha256_text(message.content) or existing["message_role"] != message.role:
+ raise ProductWriterError(f"{message.message_id}: immutable source journal content changed")
+ continue
+ connection.execute(
+ "INSERT INTO v4_source_journal(scope_id,session_id,message_id,session_index,message_index,message_role,timestamp,content,content_sha256,status,source_record_id,source_turn_index,source_persisted_at,enrichment_error,created_at,updated_at) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
+ (message.scope_id, message.session_id, message.message_id, message.session_index, message.message_index,
+ message.role, message.timestamp, message.content, sha256_text(message.content), "pending", "", 0, "", "", now, now),
+ )
+ existing_batch = connection.execute("SELECT * FROM v4_batch_journal WHERE batch_id=?", (batch.batch_id,)).fetchone()
+ if existing_batch is None:
+ connection.execute(
+ "INSERT INTO v4_batch_journal(batch_id,scope_id,session_id,batch_index,request_json,request_sha256,status,created_at,updated_at) VALUES (?,?,?,?,?,?,?,?,?)",
+ (batch.batch_id, batch.scope_id, batch.session_id, batch.batch_index, request_json, request_hash, "prepared", now, now),
+ )
+ elif existing_batch["request_sha256"] != request_hash:
+ raise ProductWriterError(f"{batch.batch_id}: prepared request changed")
+ connection.commit()
+ return connection.execute("SELECT * FROM v4_batch_journal WHERE batch_id=?", (batch.batch_id,)).fetchone()
+
+ def batch_row(self, batch_id: str) -> sqlite3.Row | None:
+ with closing(self._connect()) as connection:
+ return connection.execute(
+ "SELECT * FROM v4_batch_journal WHERE batch_id=?", (batch_id,)
+ ).fetchone()
+
+ @staticmethod
+ def _message_commit_id(batch: SourceBatch, message: SourceMessage) -> str:
+ return f"{batch.batch_id}:{message.message_id}"
+
+ def prepare_message_commit(
+ self,
+ batch: SourceBatch,
+ message: SourceMessage,
+ response_message: Mapping[str, Any],
+ ) -> sqlite3.Row:
+ commit_id = self._message_commit_id(batch, message)
+ response_sha256 = _hash_json(response_message)
+ now = _now()
+ with closing(self._connect()) as connection:
+ connection.execute("BEGIN IMMEDIATE")
+ row = connection.execute(
+ "SELECT * FROM v4_message_commit_journal WHERE commit_id=?",
+ (commit_id,),
+ ).fetchone()
+ if row is None:
+ connection.execute(
+ "INSERT INTO v4_message_commit_journal("
+ "commit_id,batch_id,scope_id,session_id,message_id,message_index,"
+ "response_sha256,status,created_at,updated_at) VALUES (?,?,?,?,?,?,?,?,?,?)",
+ (
+ commit_id,
+ batch.batch_id,
+ batch.scope_id,
+ batch.session_id,
+ message.message_id,
+ int(message.message_index),
+ response_sha256,
+ "prepared",
+ now,
+ now,
+ ),
+ )
+ else:
+ identity = (
+ str(row["batch_id"]),
+ str(row["scope_id"]),
+ str(row["session_id"]),
+ str(row["message_id"]),
+ int(row["message_index"]),
+ str(row["response_sha256"]),
+ )
+ expected = (
+ batch.batch_id,
+ batch.scope_id,
+ batch.session_id,
+ message.message_id,
+ int(message.message_index),
+ response_sha256,
+ )
+ if identity != expected:
+ raise ProductWriterError(
+ f"{commit_id}: message commit identity or response changed"
+ )
+ connection.commit()
+ return connection.execute(
+ "SELECT * FROM v4_message_commit_journal WHERE commit_id=?",
+ (commit_id,),
+ ).fetchone()
+
+ def freeze_message_commit_plan(
+ self, commit_id: str, plan: Mapping[str, Any]
+ ) -> sqlite3.Row:
+ plan_json = _json(plan)
+ plan_sha256 = sha256_text(plan_json)
+ with closing(self._connect()) as connection:
+ connection.execute("BEGIN IMMEDIATE")
+ row = connection.execute(
+ "SELECT * FROM v4_message_commit_journal WHERE commit_id=?",
+ (commit_id,),
+ ).fetchone()
+ if row is None or row["status"] not in {"prepared", "committed"}:
+ raise ProductWriterError(
+ f"{commit_id}: message commit is not preparable"
+ )
+ if clean_text(row["plan_sha256"]):
+ if row["plan_sha256"] != plan_sha256:
+ raise ProductWriterError(
+ f"{commit_id}: frozen message commit plan changed"
+ )
+ elif row["status"] == "committed":
+ raise ProductWriterError(
+ f"{commit_id}: committed message lacks a frozen plan"
+ )
+ else:
+ connection.execute(
+ "UPDATE v4_message_commit_journal SET plan_json=?,plan_sha256=?,"
+ "error='',updated_at=? WHERE commit_id=? AND status='prepared'",
+ (plan_json, plan_sha256, _now(), commit_id),
+ )
+ connection.commit()
+ return connection.execute(
+ "SELECT * FROM v4_message_commit_journal WHERE commit_id=?",
+ (commit_id,),
+ ).fetchone()
+
+ def record_message_commit_error(self, commit_id: str, error: str) -> None:
+ with closing(self._connect()) as connection, connection:
+ connection.execute(
+ "UPDATE v4_message_commit_journal SET error=?,updated_at=? "
+ "WHERE commit_id=? AND status='prepared'",
+ (error, _now(), commit_id),
+ )
+
+ def record_batch_commit_error(self, batch_id: str, error: str) -> None:
+ with closing(self._connect()) as connection, connection:
+ connection.execute(
+ "UPDATE v4_batch_journal SET error=?,updated_at=? "
+ "WHERE batch_id=? AND status='validated'",
+ (error, _now(), batch_id),
+ )
+
+ def finalize_message_commit(
+ self,
+ connection: sqlite3.Connection,
+ *,
+ commit_id: str,
+ batch: SourceBatch,
+ message: SourceMessage,
+ source_record_id: str,
+ interactions: Sequence[Mapping[str, Any]],
+ resolutions: Sequence[Mapping[str, Any]],
+ semantic_committed: int,
+ ) -> None:
+ row = connection.execute(
+ "SELECT * FROM v4_message_commit_journal WHERE commit_id=?",
+ (commit_id,),
+ ).fetchone()
+ if row is None:
+ raise ProductWriterError(f"{commit_id}: message commit journal is missing")
+ if row["status"] == "committed":
+ if int(row["semantic_committed"]) != int(semantic_committed):
+ raise ProductWriterError(
+ f"{commit_id}: committed semantic count changed"
+ )
+ return
+ if row["status"] != "prepared" or not clean_text(row["plan_sha256"]):
+ raise ProductWriterError(
+ f"{commit_id}: message commit plan is not frozen"
+ )
+ for index, interaction in enumerate(interactions):
+ interaction_id = _deterministic_interaction_id(
+ batch.scope_id, message.message_id, index
+ )
+ payload = {
+ **dict(interaction),
+ "source_record_id": source_record_id,
+ }
+ values = (
+ interaction_id,
+ batch.scope_id,
+ batch.session_id,
+ message.message_id,
+ index,
+ message.role,
+ _json(payload),
+ interaction.get("status", "open"),
+ "[]",
+ )
+ connection.execute(
+ "INSERT OR IGNORE INTO v4_interactions VALUES (?,?,?,?,?,?,?,?,?)",
+ values,
+ )
+ persisted = connection.execute(
+ "SELECT scope_id,session_id,message_id,interaction_index,message_role,"
+ "interaction_json FROM v4_interactions WHERE interaction_id=?",
+ (interaction_id,),
+ ).fetchone()
+ if persisted is None or tuple(persisted) != values[1:7]:
+ raise ProductWriterError(
+ f"{interaction_id}: interaction identity collided"
+ )
+ for resolution in resolutions:
+ interaction_id = str(resolution["interaction_id"])
+ target = connection.execute(
+ "SELECT status,resolution_history_json FROM v4_interactions "
+ "WHERE interaction_id=?",
+ (interaction_id,),
+ ).fetchone()
+ if target is None:
+ raise ProductWriterError(
+ f"resolution target does not exist: {interaction_id}"
+ )
+ resolution_state = str(resolution["resolution"])
+ next_status = (
+ "resolved"
+ if resolution_state == "resolved"
+ else "partial"
+ if resolution_state == "partial"
+ else str(target["status"])
+ )
+ history = json.loads(target["resolution_history_json"] or "[]")
+ event = {
+ "resolution": resolution_state,
+ "message_id": message.message_id,
+ "evidence_quote": str(resolution["evidence_quote"]),
+ }
+ if event not in history:
+ history.append(event)
+ connection.execute(
+ "UPDATE v4_interactions SET status=?,resolution_history_json=? "
+ "WHERE interaction_id=?",
+ (next_status, _json(history), interaction_id),
+ )
+ updated = connection.execute(
+ "UPDATE v4_source_journal SET status='enriched',enrichment_error='',"
+ "updated_at=? WHERE scope_id=? AND message_id=?",
+ (_now(), batch.scope_id, message.message_id),
+ ).rowcount
+ if updated != 1:
+ raise ProductWriterError(
+ f"{commit_id}: source journal could not commit atomically"
+ )
+ updated = connection.execute(
+ "UPDATE v4_message_commit_journal SET status='committed',"
+ "semantic_committed=?,error='',updated_at=? "
+ "WHERE commit_id=? AND status='prepared'",
+ (int(semantic_committed), _now(), commit_id),
+ ).rowcount
+ if updated != 1:
+ raise ProductWriterError(
+ f"{commit_id}: message journal could not commit atomically"
+ )
+
+ def mark_api_started(self, batch_id: str) -> None:
+ with closing(self._connect()) as connection, connection:
+ row = connection.execute("SELECT status FROM v4_batch_journal WHERE batch_id=?", (batch_id,)).fetchone()
+ if row is None:
+ raise ProductWriterError(f"{batch_id}: batch journal is missing")
+ if row["status"] == "prepared":
+ connection.execute("UPDATE v4_batch_journal SET status='api_started',api_started_at=?,updated_at=? WHERE batch_id=?", (_now(), _now(), batch_id))
+ elif row["status"] not in {"api_started", "validated", "committed"}:
+ raise ProductWriterError(f"{batch_id}: cannot start API from status {row['status']!r}")
+
+ def abandon_interrupted_batch_call(self, batch_id: str) -> sqlite3.Row:
+ with closing(self._connect()) as connection:
+ connection.execute("BEGIN IMMEDIATE")
+ row = connection.execute(
+ "SELECT status,response_json FROM v4_batch_journal WHERE batch_id=?",
+ (batch_id,),
+ ).fetchone()
+ if row is None or row["status"] != "api_started" or clean_text(row["response_json"]):
+ raise ProductWriterError(
+ f"{batch_id}: interrupted Flash recovery requires api_started without a response"
+ )
+ updated = connection.execute(
+ "UPDATE v4_batch_journal SET status='prepared',error='',updated_at=? WHERE batch_id=? AND status='api_started' AND response_json=''",
+ (_now(), batch_id),
+ ).rowcount
+ if updated != 1:
+ raise ProductWriterError(
+ f"{batch_id}: interrupted Flash call could not be abandoned atomically"
+ )
+ connection.commit()
+ return connection.execute(
+ "SELECT * FROM v4_batch_journal WHERE batch_id=?", (batch_id,)
+ ).fetchone()
+
+ def persist_response(self, batch_id: str, response: Mapping[str, Any], metadata: Mapping[str, Any]) -> None:
+ raw = _json(response)
+ with closing(self._connect()) as connection, connection:
+ connection.execute(
+ "UPDATE v4_batch_journal SET status='validated',response_json=?,response_sha256=?,response_metadata_json=?,updated_at=? WHERE batch_id=? AND status IN ('api_started','prepared')",
+ (raw, sha256_text(raw), _json(metadata), _now(), batch_id),
+ )
+
+ def revalidate_failed_response(
+ self,
+ batch_id: str,
+ response: Mapping[str, Any],
+ metadata: Mapping[str, Any],
+ ) -> sqlite3.Row:
+ raw = _json(response)
+ with closing(self._connect()) as connection:
+ connection.execute("BEGIN IMMEDIATE")
+ row = connection.execute(
+ "SELECT * FROM v4_batch_journal WHERE batch_id=?", (batch_id,)
+ ).fetchone()
+ if row is None or row["status"] != "failed":
+ raise ProductWriterError(
+ f"{batch_id}: raw response revalidation requires failed status"
+ )
+ if clean_text(row["response_json"]):
+ raise ProductWriterError(
+ f"{batch_id}: failed batch already has a validated response; replay is unsafe"
+ )
+ updated = connection.execute(
+ "UPDATE v4_batch_journal SET status='validated',response_json=?,response_sha256=?,response_metadata_json=?,error='',updated_at=? WHERE batch_id=? AND status='failed' AND response_json=''",
+ (raw, sha256_text(raw), _json(metadata), _now(), batch_id),
+ ).rowcount
+ if updated != 1:
+ raise ProductWriterError(
+ f"{batch_id}: failed raw response could not be revalidated atomically"
+ )
+ connection.commit()
+ return connection.execute(
+ "SELECT * FROM v4_batch_journal WHERE batch_id=?", (batch_id,)
+ ).fetchone()
+
+ def reconciliation_jobs_for_batch(self, batch_id: str) -> list[sqlite3.Row]:
+ with closing(self._connect()) as connection:
+ return connection.execute(
+ "SELECT * FROM v4_reconciliation_jobs WHERE batch_id=? ORDER BY created_at,job_id",
+ (batch_id,),
+ ).fetchall()
+
+ def revalidate_failed_reconciliation_job(
+ self,
+ job_id: str,
+ decision: str,
+ response: Mapping[str, Any],
+ metadata: Mapping[str, Any],
+ ) -> None:
+ with closing(self._connect()) as connection:
+ connection.execute("BEGIN IMMEDIATE")
+ row = connection.execute(
+ "SELECT status,response_json FROM v4_reconciliation_jobs WHERE job_id=?",
+ (job_id,),
+ ).fetchone()
+ if row is None or row["status"] != "failed" or clean_text(row["response_json"]):
+ raise ProductWriterError(
+ f"{job_id}: reconciliation revalidation requires one failed response"
+ )
+ updated = connection.execute(
+ "UPDATE v4_reconciliation_jobs SET status='completed',decision=?,response_json=?,response_metadata_json=?,error='',updated_at=? WHERE job_id=? AND status='failed' AND response_json=''",
+ (decision, _json(response), _json(metadata), _now(), job_id),
+ ).rowcount
+ if updated != 1:
+ raise ProductWriterError(
+ f"{job_id}: reconciliation response could not be revalidated atomically"
+ )
+ connection.commit()
+
+ def resume_failed_validated_batch(
+ self,
+ batch_id: str,
+ metadata: Mapping[str, Any],
+ *,
+ allowed_pending_job_ids: Sequence[str] = (),
+ ) -> sqlite3.Row:
+ allowed_pending = {clean_text(value) for value in allowed_pending_job_ids}
+ if "" in allowed_pending:
+ raise ProductWriterError(
+ f"{batch_id}: pending reconciliation allowlist contains an empty job ID"
+ )
+ with closing(self._connect()) as connection:
+ connection.execute("BEGIN IMMEDIATE")
+ row = connection.execute(
+ "SELECT status,response_json FROM v4_batch_journal WHERE batch_id=?",
+ (batch_id,),
+ ).fetchone()
+ if row is None or row["status"] != "failed" or not clean_text(row["response_json"]):
+ raise ProductWriterError(
+ f"{batch_id}: validated commit recovery requires a failed batch response"
+ )
+ uncertain = connection.execute(
+ "SELECT job_id,status FROM v4_reconciliation_jobs WHERE batch_id=? AND status!='completed'",
+ (batch_id,),
+ ).fetchall()
+ actual_pending = {
+ clean_text(item["job_id"])
+ for item in uncertain
+ if clean_text(item["status"]) == "pro_pending"
+ }
+ unexpected = [
+ (clean_text(item["job_id"]), clean_text(item["status"]))
+ for item in uncertain
+ if clean_text(item["status"]) != "pro_pending"
+ or clean_text(item["job_id"]) not in allowed_pending
+ ]
+ if unexpected or actual_pending != allowed_pending:
+ raise ProductWriterError(
+ f"{batch_id}: reconciliation jobs remain outside the explicit pending allowlist: "
+ f"unexpected={unexpected}, expected={sorted(allowed_pending)}, "
+ f"actual={sorted(actual_pending)}"
+ )
+ updated = connection.execute(
+ "UPDATE v4_batch_journal SET status='validated',response_metadata_json=?,error='',updated_at=? WHERE batch_id=? AND status='failed' AND response_json!=''",
+ (_json(metadata), _now(), batch_id),
+ ).rowcount
+ if updated != 1:
+ raise ProductWriterError(
+ f"{batch_id}: failed validated batch could not resume atomically"
+ )
+ connection.commit()
+ return connection.execute(
+ "SELECT * FROM v4_batch_journal WHERE batch_id=?", (batch_id,)
+ ).fetchone()
+
+ def fail_batch(
+ self,
+ batch_id: str,
+ error: str,
+ metadata: Mapping[str, Any] | None = None,
+ ) -> None:
+ with closing(self._connect()) as connection, connection:
+ if metadata is None:
+ connection.execute(
+ "UPDATE v4_batch_journal SET status='failed',error=?,updated_at=? WHERE batch_id=? AND status!='committed'",
+ (error, _now(), batch_id),
+ )
+ else:
+ connection.execute(
+ "UPDATE v4_batch_journal SET status='failed',error=?,response_metadata_json=?,updated_at=? WHERE batch_id=? AND status!='committed'",
+ (error, _json(dict(metadata)), _now(), batch_id),
+ )
+
+ def set_source_record(self, scope_id: str, message_id: str, source_record_id: str, source_turn_index: int) -> None:
+ with closing(self._connect()) as connection, connection:
+ updated = connection.execute(
+ "UPDATE v4_source_journal SET source_record_id=?,source_turn_index=?,source_persisted_at=CASE WHEN source_persisted_at='' THEN ? ELSE source_persisted_at END,updated_at=? WHERE scope_id=? AND message_id=? AND (status='pending' OR source_record_id=?)",
+ (source_record_id, int(source_turn_index), _now(), _now(), scope_id, message_id, source_record_id),
+ ).rowcount
+ if updated != 1:
+ raise ProductWriterError(f"{message_id}: source journal could not bind real graph source record")
+
+ def source_info(self, scope_id: str, message_id: str) -> dict[str, Any]:
+ with closing(self._connect()) as connection:
+ row = connection.execute("SELECT * FROM v4_source_journal WHERE scope_id=? AND message_id=?", (scope_id, message_id)).fetchone()
+ if row is None:
+ raise ProductWriterError(f"{message_id}: source journal row is missing")
+ return dict(row)
+
+ def mark_source_enrichment_failed(self, batch: SourceBatch, error: str) -> None:
+ with closing(self._connect()) as connection, connection:
+ connection.execute(
+ "UPDATE v4_source_journal SET status='failed',enrichment_error=?,updated_at=? WHERE scope_id=? AND message_id IN ({})".format(",".join("?" for _ in batch.messages)),
+ (error, _now(), batch.scope_id, *[message.message_id for message in batch.messages]),
+ )
+
+ def mark_source_enriched(self, batch: SourceBatch) -> None:
+ with closing(self._connect()) as connection, connection:
+ connection.execute(
+ "UPDATE v4_source_journal SET status='enriched',enrichment_error='',updated_at=? WHERE scope_id=? AND message_id IN ({})".format(",".join("?" for _ in batch.messages)),
+ (_now(), batch.scope_id, *[message.message_id for message in batch.messages]),
+ )
+
+
+ def batch_row(self, batch_id: str) -> sqlite3.Row | None:
+ with closing(self._connect()) as connection:
+ return connection.execute("SELECT * FROM v4_batch_journal WHERE batch_id=?", (batch_id,)).fetchone()
+
+ def unresolved_interactions(self, scope_id: str, session_id: str) -> list[dict[str, Any]]:
+ with closing(self._connect()) as connection:
+ rows = connection.execute(
+ "SELECT * FROM v4_interactions WHERE scope_id=? AND session_id=? AND status IN ('open','partial') ORDER BY rowid",
+ (scope_id, session_id),
+ ).fetchall()
+ output = []
+ for row in rows:
+ item = json.loads(row["interaction_json"])
+ item["interaction_id"] = row["interaction_id"]
+ item["message_id"] = row["message_id"]
+ item["message_role"] = row["message_role"]
+ output.append(item)
+ return output
+
+ def insert_interaction(self, *, interaction_id: str, scope_id: str, session_id: str, message_id: str, index: int, role: str, interaction: Mapping[str, Any]) -> None:
+ with closing(self._connect()) as connection, connection:
+ connection.execute(
+ "INSERT OR IGNORE INTO v4_interactions VALUES (?,?,?,?,?,?,?,?,?)",
+ (interaction_id, scope_id, session_id, message_id, index, role, _json(interaction), interaction.get("status", "open"), "[]"),
+ )
+
+ def update_interaction_resolution(self, interaction_id: str, resolution: str, source_message_id: str, evidence_quote: str) -> None:
+ with closing(self._connect()) as connection, connection:
+ row = connection.execute("SELECT status,resolution_history_json FROM v4_interactions WHERE interaction_id=?", (interaction_id,)).fetchone()
+ if row is None:
+ raise ProductWriterError(f"resolution target does not exist: {interaction_id}")
+ next_status = "resolved" if resolution == "resolved" else ("partial" if resolution == "partial" else row["status"])
+ history = json.loads(row["resolution_history_json"] or "[]")
+ event = {"resolution": resolution, "message_id": source_message_id, "evidence_quote": evidence_quote}
+ if event not in history:
+ history.append(event)
+ connection.execute("UPDATE v4_interactions SET status=?,resolution_history_json=? WHERE interaction_id=?", (next_status, _json(history), interaction_id))
+
+ def reconciliation_job(self, job_id: str) -> sqlite3.Row | None:
+ with closing(self._connect()) as connection:
+ return connection.execute("SELECT * FROM v4_reconciliation_jobs WHERE job_id=?", (job_id,)).fetchone()
+
+ def create_reconciliation_job(
+ self,
+ *,
+ job_id: str,
+ scope_id: str,
+ batch_id: str,
+ message_id: str,
+ slot: str,
+ assertion_index: int,
+ request: Mapping[str, Any],
+ ) -> None:
+ request_json = _json(request)
+ with closing(self._connect()) as connection, connection:
+ existing = connection.execute(
+ "SELECT scope_id,batch_id,message_id,canonical_slot_key,assertion_index,request_json FROM v4_reconciliation_jobs WHERE job_id=?",
+ (job_id,),
+ ).fetchone()
+ if existing is not None:
+ expected_identity = (
+ scope_id,
+ batch_id,
+ message_id,
+ slot,
+ int(assertion_index),
+ )
+ if tuple(existing)[:5] != expected_identity:
+ raise ProductWriterError(
+ f"{job_id}: reconciliation job identity collided with different evidence"
+ )
+ frozen_request = json.loads(str(existing["request_json"]))
+ for field in (
+ "schema_version",
+ "candidate_selector_version",
+ "canonical_slot_key",
+ "message_id",
+ "new_cited_assertion",
+ ):
+ if frozen_request.get(field) != request.get(field):
+ raise ProductWriterError(
+ f"{job_id}: frozen reconciliation {field} changed"
+ )
+ return
+ connection.execute(
+ "INSERT INTO v4_reconciliation_jobs(job_id,scope_id,batch_id,message_id,canonical_slot_key,assertion_index,request_json,status,created_at,updated_at) VALUES (?,?,?,?,?,?,?,?,?,?)",
+ (
+ job_id,
+ scope_id,
+ batch_id,
+ message_id,
+ slot,
+ assertion_index,
+ request_json,
+ "pro_pending",
+ _now(),
+ _now(),
+ ),
+ )
+
+ def start_reconciliation_job(self, job_id: str) -> None:
+ with closing(self._connect()) as connection, connection:
+ connection.execute("UPDATE v4_reconciliation_jobs SET status='pro_started',updated_at=? WHERE job_id=? AND status='pro_pending'", (_now(), job_id))
+
+ def abandon_interrupted_reconciliation_call(self, job_id: str) -> sqlite3.Row:
+ with closing(self._connect()) as connection:
+ connection.execute("BEGIN IMMEDIATE")
+ row = connection.execute(
+ "SELECT status,response_json FROM v4_reconciliation_jobs WHERE job_id=?",
+ (job_id,),
+ ).fetchone()
+ if row is None or row["status"] != "pro_started" or clean_text(row["response_json"]):
+ raise ProductWriterError(
+ f"{job_id}: interrupted Pro recovery requires pro_started without a response"
+ )
+ updated = connection.execute(
+ "UPDATE v4_reconciliation_jobs SET status='pro_pending',error='',updated_at=? WHERE job_id=? AND status='pro_started' AND response_json=''",
+ (_now(), job_id),
+ ).rowcount
+ if updated != 1:
+ raise ProductWriterError(
+ f"{job_id}: interrupted Pro call could not be abandoned atomically"
+ )
+ connection.commit()
+ return connection.execute(
+ "SELECT * FROM v4_reconciliation_jobs WHERE job_id=?", (job_id,)
+ ).fetchone()
+
+ def finish_reconciliation_job(self, job_id: str, decision: str, response: Mapping[str, Any], metadata: Mapping[str, Any]) -> None:
+ with closing(self._connect()) as connection, connection:
+ connection.execute("UPDATE v4_reconciliation_jobs SET status='completed',decision=?,response_json=?,response_metadata_json=?,updated_at=? WHERE job_id=?", (decision, _json(response), _json(metadata), _now(), job_id))
+
+ def fail_reconciliation_job(
+ self,
+ job_id: str,
+ error: str,
+ metadata: Mapping[str, Any] | None = None,
+ ) -> None:
+ with closing(self._connect()) as connection, connection:
+ connection.execute(
+ "UPDATE v4_reconciliation_jobs SET status='failed',error=?,response_metadata_json=?,updated_at=? WHERE job_id=?",
+ (error, _json(dict(metadata or {})), _now(), job_id),
+ )
+
+ def commit_batch(self, batch_id: str) -> None:
+ with closing(self._connect()) as connection, connection:
+ row = connection.execute(
+ "SELECT status FROM v4_batch_journal WHERE batch_id=?", (batch_id,)
+ ).fetchone()
+ if row is not None and row["status"] == "committed":
+ return
+ updated = connection.execute(
+ "UPDATE v4_batch_journal SET status='committed',updated_at=? WHERE batch_id=? AND status='validated'",
+ (_now(), batch_id),
+ ).rowcount
+ if updated != 1:
+ raise ProductWriterError(
+ f"{batch_id}: validated batch could not transition to committed"
+ )
+
+
+def configure_real_graph_environment() -> None:
+ os.environ.update(
+ {
+ "TMCRA_PROFILE_CONSOLIDATOR_ENABLED": "0",
+ "TMCRA_LEGACY_PROFILE_LAYER_ENABLED": "0",
+ "TMCRA_WRITE_EMBEDDER_INDEX_MODE": "off",
+ "TMCRA_EMBEDDER_INDEX_RECALL_MODE": "off",
+ "TMCRA_EMBEDDER_PRE_RECALL_MODE": "off",
+ "TMCRA_EMBEDDER_FUSION_MODE": "off",
+ "TMCRA_MEMORY_ROUTER_MODE": "off",
+ "TMCRA_INJECTION_PLANNER_MODE": "off",
+ "TMCRA_TEMPORAL_LAYER_MODE": "off",
+ "TMCRA_TEMPORAL_ROUTER_MODE": "off",
+ "TMCRA_DEEPSEEK_GRAPH_MODEL_MODE": "off",
+ "TMCRA_TOPIC_BUCKET_MODE": "off",
+ "TMCRA_MULTI_UNIT_CHAIN_SLOT_MODE": "off",
+ "TMCRA_UNIT_COVERAGE_PACK_MODE": "off",
+ }
+ )
+
+
+class RealGraphBackend:
+ """Adapter boundary for the real TMCRA graph; V4 tables never mirror graph leaves."""
+
+ supports_atomic_message_commit = True
+
+ def __init__(self, *, repo: Path, database: Path, scope_id: str, audit_retention: int = 4096) -> None:
+ configure_real_graph_environment()
+ repo = Path(repo).resolve()
+ if str(repo) not in sys.path:
+ sys.path.insert(0, str(repo))
+ try:
+ from experiments.replacement.adapters.memory_adapters import GraphSessionMemoryAdapter
+ from experiments.replacement.memory_graph import SessionMemoryEdgeV2, SessionMemoryRecordV2
+ except ImportError as exc:
+ raise ProductWriterError(f"--repo does not expose the real TMCRA graph adapter: {repo}") from exc
+ self._record_class = SessionMemoryRecordV2
+ self._edge_class = SessionMemoryEdgeV2
+ self.adapter = GraphSessionMemoryAdapter(
+ auto_extract=False,
+ storage_backend="sqlite",
+ storage_path=str(database),
+ scope_id=scope_id,
+ audit_retention=max(256, int(audit_retention)),
+ retrieval_mode="heuristic",
+ )
+ self.scope_id = scope_id
+
+ def _reload(self) -> None:
+ self.adapter._reload_graph()
+
+ def _persist(
+ self,
+ transaction_hook: Callable[[sqlite3.Connection], None] | None = None,
+ ) -> None:
+ if transaction_hook is None:
+ self.adapter._persist_graph()
+ return
+ store = getattr(self.adapter, "_store", None)
+ if store is None:
+ raise ProductWriterError(
+ "atomic V4 commit requires the SQLite graph store"
+ )
+ self.adapter.graph.configure_persistence(
+ backend=self.adapter.storage_backend,
+ path=self.adapter.storage_path,
+ audit_retention=self.adapter.audit_retention,
+ )
+ store.save_graph(
+ self.scope_id,
+ self.adapter.graph,
+ transaction_hook=transaction_hook,
+ )
+ self.adapter._invalidate_runtime_graph_cache()
+
+ @staticmethod
+ def _source_metadata(record: Any) -> dict[str, Any]:
+ return dict(getattr(record, "metadata", {}) or {})
+
+ @classmethod
+ def _verified_source_content(cls, record: Any) -> str:
+ metadata = cls._source_metadata(record)
+ raw_content = metadata.get("raw_content")
+ if not isinstance(raw_content, str) or not raw_content:
+ raise ProductWriterError("real immutable source lacks exact raw_content")
+ for field in ("source_span", "source_turn_text"):
+ if metadata.get(field) != raw_content:
+ raise ProductWriterError(
+ f"real immutable source {field} differs from raw_content"
+ )
+ # The legacy graph store normalizes whitespace in record.value. Exact
+ # source text lives in raw_content; value must remain equivalent.
+ if clean_text(getattr(record, "value", "")) != clean_text(raw_content):
+ raise ProductWriterError(
+ "real immutable source graph value differs from raw_content"
+ )
+ return raw_content
+
+ def ensure_source(self, message: SourceMessage) -> tuple[str, int]:
+ self._reload()
+ for record in self.adapter.graph.records_by_id.values():
+ metadata = self._source_metadata(record)
+ if metadata.get("content_variant") != "source_message" or metadata.get("message_id") != message.message_id:
+ continue
+ if self._verified_source_content(record) != message.content:
+ raise ProductWriterError(f"{message.message_id}: real source graph content changed")
+ if int(metadata.get("session_index", -1)) != message.session_index or int(
+ metadata.get("message_index", -1)
+ ) != message.message_index:
+ raise ProductWriterError(
+ f"{message.message_id}: real source graph location changed"
+ )
+ return str(record.memory_id), int(record.turn_index)
+ turn_index = self.adapter.graph.next_turn()
+ records, _ = build_graph_records(
+ self._record_class,
+ scope_id=self.scope_id,
+ turn_index=turn_index,
+ session_id=message.session_id,
+ session_index=message.session_index,
+ message_id=message.message_id,
+ message_index=message.message_index,
+ date=message.timestamp[:10],
+ timestamp=message.timestamp,
+ role=message.role,
+ content=message.content,
+ extraction=None,
+ )
+ source = records[0]
+ source.metadata.update(
+ {
+ "source": "tmcra_v4_batch_writer",
+ "writer_schema_version": BATCH_SCHEMA_VERSION,
+ "prompt_version": PROMPT_VERSION,
+ "enrichment_status": "pending",
+ "source_record_id": source.memory_id,
+ }
+ )
+ stored = self.adapter.graph.add_records([source])
+ if source.memory_id not in stored and source.memory_id not in self.adapter.graph.records_by_id:
+ raise ProductWriterError(f"{message.message_id}: real immutable source record was not persisted")
+ self.adapter.graph.record_turn(
+ turn_kind="memory_write",
+ text=message.content,
+ turn_index=turn_index,
+ record_ids=[source.memory_id],
+ speaker=message.role,
+ metadata={"source": "tmcra_v4_batch_writer", "message_id": message.message_id, "source_record_id": source.memory_id, "enrichment_status": "pending"},
+ )
+ self._persist()
+ self.verify_source(message, str(source.memory_id), turn_index)
+ return str(source.memory_id), turn_index
+
+ def verify_source(
+ self,
+ message: SourceMessage,
+ source_record_id: str,
+ source_turn_index: int,
+ ) -> None:
+ self._reload()
+ record = self.adapter.graph.records_by_id.get(source_record_id)
+ if record is None:
+ raise ProductWriterError(
+ f"{message.message_id}: committed real source record is missing"
+ )
+ metadata = self._source_metadata(record)
+ expected = {
+ "content_variant": "source_message",
+ "message_id": message.message_id,
+ "session_id": message.session_id,
+ "session_index": message.session_index,
+ "message_index": message.message_index,
+ }
+ for key, value in expected.items():
+ if metadata.get(key) != value:
+ raise ProductWriterError(
+ f"{message.message_id}: committed real source {key} changed"
+ )
+ if self._verified_source_content(record) != message.content:
+ raise ProductWriterError(
+ f"{message.message_id}: committed real source content changed"
+ )
+ if int(record.turn_index) != int(source_turn_index):
+ raise ProductWriterError(
+ f"{message.message_id}: committed real source turn changed"
+ )
+
+ def set_enrichment_status(self, source_record_id: str, status: str, error: str = "") -> None:
+ self._reload()
+ record = self.adapter.graph.records_by_id.get(source_record_id)
+ if record is None:
+ raise ProductWriterError(f"real source record is missing: {source_record_id}")
+ metadata = self._source_metadata(record)
+ metadata["enrichment_status"] = status
+ if error:
+ metadata["enrichment_error"] = error
+ else:
+ metadata.pop("enrichment_error", None)
+ record.metadata = metadata
+ self._persist()
+
+ def source_enrichment_statuses(
+ self, source_record_ids: Sequence[str]
+ ) -> dict[str, str]:
+ self._reload()
+ result: dict[str, str] = {}
+ for source_record_id in source_record_ids:
+ record = self.adapter.graph.records_by_id.get(source_record_id)
+ if record is None:
+ raise ProductWriterError(
+ f"real source record is missing: {source_record_id}"
+ )
+ result[source_record_id] = clean_text(
+ self._source_metadata(record).get("enrichment_status")
+ )
+ return result
+
+ def current_leaves(self, canonical_slot_key: str) -> list[dict[str, Any]]:
+ self._reload()
+ graph_slot_key = _graph_slot_key(canonical_slot_key)
+ rows = []
+ for record in self.adapter.graph.records_by_id.values():
+ metadata = self._source_metadata(record)
+ if (
+ metadata.get("content_variant") == "product_semantic_memory"
+ and metadata.get("memory_layer") == "fast"
+ and metadata.get("node_kind") == "atomic_user_assertion"
+ and metadata.get("canonical_slot_key") == graph_slot_key
+ and str(record.state) in {"active", "parallel_active", "promoted"}
+ ):
+ rows.append({"memory_id": record.memory_id, "value": record.value, "claim_text": record.value, "evidence_quote": metadata.get("evidence_quote") or record.value, "canonical_slot_key": graph_slot_key, "durability": metadata.get("durability"), "record_state": record.state, "turn_index": record.turn_index, "metadata": metadata})
+ return rows
+
+ def leaf_by_id(self, memory_id: str) -> dict[str, Any] | None:
+ self._reload()
+ record = self.adapter.graph.records_by_id.get(memory_id)
+ if record is None:
+ return None
+ metadata = self._source_metadata(record)
+ if (
+ metadata.get("content_variant") != "product_semantic_memory"
+ or metadata.get("memory_layer") != "fast"
+ or metadata.get("node_kind") != "atomic_user_assertion"
+ ):
+ return None
+ return {
+ "memory_id": str(record.memory_id),
+ "value": str(record.value),
+ "claim_text": str(record.value),
+ "evidence_quote": clean_text(metadata.get("evidence_quote")) or str(record.value),
+ "canonical_slot_key": clean_text(metadata.get("canonical_slot_key")),
+ "durability": clean_text(metadata.get("durability")),
+ "record_state": str(record.state),
+ "turn_index": int(record.turn_index),
+ "metadata": metadata,
+ }
+
+ def leaf_for_source_assertion(
+ self,
+ source_record_id: str,
+ assertion_index: int,
+ ) -> dict[str, Any] | None:
+ self._reload()
+ matches: list[dict[str, Any]] = []
+ for record in self.adapter.graph.records_by_id.values():
+ metadata = self._source_metadata(record)
+ if (
+ metadata.get("content_variant") == "product_semantic_memory"
+ and clean_text(metadata.get("source_record_id"))
+ == source_record_id
+ and int(metadata.get("llm_write_proposal_index", -1))
+ == int(assertion_index)
+ ):
+ leaf = self.leaf_by_id(str(record.memory_id))
+ if leaf is not None:
+ matches.append(leaf)
+ if len(matches) > 1:
+ raise ProductWriterError(
+ f"{source_record_id}: assertion {assertion_index} has multiple persisted semantic leaves"
+ )
+ return matches[0] if matches else None
+
+ def repair_partial_replacement(
+ self,
+ historical_memory_id: str,
+ incoming_memory_id: str,
+ ) -> dict[str, Any]:
+ self._reload()
+ historical = self.adapter.graph.records_by_id.get(historical_memory_id)
+ incoming = self.adapter.graph.records_by_id.get(incoming_memory_id)
+ if historical is None or incoming is None:
+ raise ProductWriterError(
+ "partial replacement repair requires both persisted records"
+ )
+ historical_metadata = self._source_metadata(historical)
+ incoming_metadata = self._source_metadata(incoming)
+ existing_link = clean_text(historical_metadata.get("superseded_by"))
+ if (
+ str(historical.state) != "superseded"
+ or clean_text(historical_metadata.get("superseded_reason"))
+ != "v4_reconciliation_replace_current"
+ or existing_link not in {"", incoming_memory_id}
+ ):
+ raise ProductWriterError(
+ f"{historical_memory_id}: partial replacement lifecycle is incompatible"
+ )
+ historical_metadata["superseded_by"] = incoming_memory_id
+ historical_metadata["superseded_reason"] = (
+ "v4_reconciliation_replace_current"
+ )
+ historical.metadata = historical_metadata
+ incoming.state = "active"
+ incoming_metadata.pop("superseded_by", None)
+ incoming_metadata.pop("superseded_reason", None)
+ incoming.metadata = incoming_metadata
+ supersedes = list(getattr(incoming, "supersedes", []) or [])
+ if historical_memory_id not in supersedes:
+ supersedes.append(historical_memory_id)
+ incoming.supersedes = supersedes
+ self.adapter.graph.slot_heads[str(incoming.slot_key)] = incoming.memory_id
+ self._persist()
+ repaired = self.leaf_by_id(incoming_memory_id)
+ if repaired is None:
+ raise ProductWriterError(
+ f"{incoming_memory_id}: repaired replacement disappeared"
+ )
+ return repaired
+
+ def candidate_leaves(
+ self, assertion: Mapping[str, Any], *, limit: int = 3
+ ) -> list[dict[str, Any]]:
+ self._reload()
+ proposed_slot = _graph_slot_key(assertion.get("canonical_key"))
+ proposed_canonical = (
+ _slot_tokens(proposed_slot) - _BROAD_SLOT_IDENTITY_TOKENS
+ )
+ proposed_attribute = (
+ _slot_tokens(assertion.get("attribute_key"))
+ - _BROAD_SLOT_IDENTITY_TOKENS
+ )
+ proposed_family = clean_text(
+ assertion.get("memory_family") or assertion.get("memory_type")
+ )
+ proposed_identity = _slot_tokens(
+ assertion.get("canonical_key"),
+ assertion.get("entity_key"),
+ assertion.get("graph_entity_key"),
+ assertion.get("attribute_key"),
+ assertion.get("relation"),
+ )
+ proposed_value = _slot_tokens(
+ assertion.get("claim_text"),
+ *[
+ facet.get("quote")
+ for facet in assertion.get("facets") or []
+ if isinstance(facet, Mapping)
+ ],
+ )
+ by_slot: dict[str, tuple[float, int, Any, dict[str, Any]]] = {}
+ exact_claim_candidates: list[dict[str, Any]] = []
+ for record in self.adapter.graph.records_by_id.values():
+ metadata = self._source_metadata(record)
+ slot = clean_text(metadata.get("canonical_slot_key"))
+ if (
+ metadata.get("content_variant") != "product_semantic_memory"
+ or metadata.get("memory_layer") != "fast"
+ or metadata.get("node_kind") != "atomic_user_assertion"
+ or str(record.state) not in {"active", "parallel_active", "promoted"}
+ or not slot
+ or slot == proposed_slot
+ ):
+ continue
+ if _normalized_claim(str(record.value)) == _normalized_claim(
+ str(assertion.get("claim_text"))
+ ):
+ exact_claim_candidates.append(
+ {
+ "memory_id": str(record.memory_id),
+ "value": str(record.value),
+ "claim_text": str(record.value),
+ "evidence_quote": clean_text(metadata.get("evidence_quote")) or str(record.value),
+ "canonical_slot_key": slot,
+ "durability": metadata.get("durability"),
+ "record_state": str(record.state),
+ "turn_index": int(record.turn_index),
+ "metadata": metadata,
+ "candidate_score": 1_000_000.0,
+ "candidate_reason": "exact_atomic_claim",
+ }
+ )
+ continue
+ existing_identity = _slot_tokens(
+ slot.removeprefix("memory."),
+ metadata.get("entity_key"),
+ metadata.get("graph_entity_key"),
+ metadata.get("attribute_key"),
+ record.relation,
+ )
+ shared_identity = proposed_identity & existing_identity
+ strong_shared_identity = (
+ shared_identity - _BROAD_SLOT_IDENTITY_TOKENS
+ )
+ existing_canonical = (
+ _slot_tokens(slot) - _BROAD_SLOT_IDENTITY_TOKENS
+ )
+ existing_attribute = (
+ _slot_tokens(metadata.get("attribute_key"))
+ - _BROAD_SLOT_IDENTITY_TOKENS
+ )
+ canonical_overlap = proposed_canonical & existing_canonical
+ attribute_overlap = proposed_attribute & existing_attribute
+ same_entity = clean_text(assertion.get("graph_entity_key")) == clean_text(
+ metadata.get("graph_entity_key")
+ ) and bool(clean_text(assertion.get("graph_entity_key")))
+ existing_family = clean_text(
+ metadata.get("memory_family") or metadata.get("memory_type")
+ )
+ if (
+ not proposed_family
+ or proposed_family != existing_family
+ or len(attribute_overlap) < 1
+ or len(canonical_overlap) < 2
+ ):
+ continue
+ existing_value = _slot_tokens(record.value, metadata.get("object"))
+ value_overlap = len(proposed_value & existing_value)
+ same_family = clean_text(assertion.get("memory_family")) == clean_text(
+ metadata.get("memory_family")
+ )
+ score = (
+ 2.0 * len(shared_identity)
+ + float(value_overlap)
+ + (0.75 if same_entity else 0.0)
+ + (0.35 if same_family else 0.0)
+ )
+ item = {
+ "memory_id": str(record.memory_id),
+ "value": str(record.value),
+ "claim_text": str(record.value),
+ "evidence_quote": clean_text(metadata.get("evidence_quote")) or str(record.value),
+ "canonical_slot_key": slot,
+ "durability": metadata.get("durability"),
+ "record_state": str(record.state),
+ "turn_index": int(record.turn_index),
+ "metadata": metadata,
+ "candidate_score": round(score, 6),
+ "shared_identity_tokens": sorted(shared_identity),
+ "strong_shared_identity_tokens": sorted(strong_shared_identity),
+ "shared_canonical_tokens": sorted(canonical_overlap),
+ "shared_attribute_tokens": sorted(attribute_overlap),
+ }
+ existing = by_slot.get(slot)
+ ranked = (score, int(record.turn_index), record, item)
+ if existing is None or ranked[:2] > existing[:2]:
+ by_slot[slot] = ranked
+ if exact_claim_candidates:
+ return sorted(
+ exact_claim_candidates,
+ key=lambda item: (
+ -int(item["turn_index"]),
+ str(item["memory_id"]),
+ ),
+ )[: max(1, int(limit))]
+ ranked_candidates = sorted(
+ by_slot.values(), key=lambda item: (item[0], item[1], str(item[2].memory_id)), reverse=True
+ )
+ return [item[3] for item in ranked_candidates[: max(1, int(limit))]]
+
+ def add_provenance(
+ self,
+ leaf_id: str,
+ *,
+ source_record_id: str,
+ source_turn_index: int,
+ provenance: Mapping[str, Any],
+ ) -> None:
+ self._reload()
+ record = self.adapter.graph.records_by_id.get(leaf_id)
+ if record is None:
+ raise ProductWriterError(f"real fast leaf not found for provenance: {leaf_id}")
+ metadata = self._source_metadata(record)
+ provenance_entry = {
+ **dict(provenance),
+ "source_record_id": source_record_id,
+ "source_turn_index": int(source_turn_index),
+ }
+ values = list(metadata.get("provenance") or [])
+ if provenance_entry not in values:
+ values.append(provenance_entry)
+ metadata["provenance"] = values
+ record.metadata = metadata
+ self.adapter.graph._upsert_memory_edge(
+ self._edge_class(
+ edge_id=f"{leaf_id}->{source_record_id}:grounded_in",
+ source_memory_id=leaf_id,
+ target_memory_id=source_record_id,
+ edge_type="grounded_in",
+ score=1.0,
+ model_score=0.0,
+ evidence_turn=int(source_turn_index),
+ evidence=str(provenance.get("evidence_quote") or record.value),
+ metadata={
+ "edge_source": "product_writer_provenance",
+ "source_record_id": source_record_id,
+ **provenance_entry,
+ },
+ )
+ )
+ self._persist()
+
+ @staticmethod
+ def _restore_replayed_semantic_record(
+ graph: Any,
+ persisted: Any,
+ replayed: Any,
+ decision: str,
+ ) -> None:
+ persisted_metadata = dict(getattr(persisted, "metadata", {}) or {})
+ replayed_metadata = dict(getattr(replayed, "metadata", {}) or {})
+ top_level_identity = (
+ ("memory_id", str),
+ ("slot_key", clean_text),
+ ("value", clean_text),
+ ("turn_index", int),
+ )
+ for field, normalize in top_level_identity:
+ if normalize(getattr(persisted, field)) != normalize(
+ getattr(replayed, field)
+ ):
+ raise ProductWriterError(
+ f"{getattr(replayed, 'memory_id', '')}: replayed semantic record {field} changed"
+ )
+ for field in (
+ "content_variant",
+ "memory_layer",
+ "node_kind",
+ "message_id",
+ "source_record_id",
+ "llm_write_proposal_index",
+ "canonical_slot_key",
+ "event_signature",
+ "evidence_quote",
+ "source_span",
+ ):
+ if clean_text(persisted_metadata.get(field)) != clean_text(
+ replayed_metadata.get(field)
+ ):
+ raise ProductWriterError(
+ f"{getattr(replayed, 'memory_id', '')}: replayed semantic record {field} changed"
+ )
+ merged_metadata = {**persisted_metadata, **replayed_metadata}
+ provenance: list[Any] = []
+ for item in [
+ *list(persisted_metadata.get("provenance") or []),
+ *list(replayed_metadata.get("provenance") or []),
+ ]:
+ if item not in provenance:
+ provenance.append(item)
+ if provenance:
+ merged_metadata["provenance"] = provenance
+ if decision in {"insert", "replace_current", "keep_parallel"}:
+ merged_metadata.pop("superseded_by", None)
+ merged_metadata.pop("superseded_reason", None)
+ persisted.state = (
+ "parallel_active" if decision == "keep_parallel" else "active"
+ )
+ graph.slot_heads[str(replayed.slot_key)] = str(replayed.memory_id)
+ elif decision == "challenge":
+ persisted.state = "challenged"
+ elif decision == "quarantine":
+ persisted.state = "quarantined"
+ else:
+ raise ProductWriterError(
+ f"{getattr(replayed, 'memory_id', '')}: unsupported replay decision {decision!r}"
+ )
+ persisted.metadata = merged_metadata
+
+ @staticmethod
+ def _restore_auto_superseded_current(
+ graph: Any,
+ incoming: Any,
+ current: Sequence[Mapping[str, Any]],
+ *,
+ decision: str,
+ ) -> list[str]:
+ """Undo only graph-policy supersessions caused by a non-replacing insert."""
+ restored: list[str] = []
+ incoming_id = clean_text(getattr(incoming, "memory_id", ""))
+ incoming_slot = clean_text(getattr(incoming, "slot_key", ""))
+ incoming_turn = int(getattr(incoming, "turn_index", -1))
+ if not incoming_id or not incoming_slot:
+ raise ProductWriterError(
+ f"{decision} incoming record identity is incomplete"
+ )
+ for snapshot in current:
+ memory_id = clean_text(snapshot.get("memory_id"))
+ if not memory_id:
+ raise ProductWriterError(
+ f"{incoming_id}: {decision} current record lacks memory_id"
+ )
+ record = graph.records_by_id.get(memory_id)
+ if record is None:
+ raise ProductWriterError(
+ f"{incoming_id}: {decision} current record disappeared: {memory_id}"
+ )
+ metadata = dict(getattr(record, "metadata", {}) or {})
+ if not (
+ clean_text(getattr(record, "state", "")) == "superseded"
+ and clean_text(metadata.get("superseded_by")) == incoming_id
+ ):
+ continue
+ reason = clean_text(metadata.get("superseded_reason"))
+ prior_state = clean_text(snapshot.get("record_state"))
+ if reason not in GRAPH_AUTO_SUPERSESSION_REASONS:
+ raise ProductWriterError(
+ f"{incoming_id}: {decision} would erase {memory_id} for unsupported reason {reason!r}"
+ )
+ if (
+ clean_text(getattr(record, "slot_key", "")) != incoming_slot
+ or int(getattr(record, "turn_index", -1)) > incoming_turn
+ or prior_state not in {"active", "parallel_active", "promoted"}
+ ):
+ raise ProductWriterError(
+ f"{incoming_id}: {decision} supersession lifecycle is inconsistent for {memory_id}"
+ )
+ record.state = prior_state
+ metadata.pop("superseded_by", None)
+ metadata.pop("superseded_reason", None)
+ record.metadata = metadata
+ restored.append(memory_id)
+ if restored:
+ restored_ids = set(restored)
+ incoming.supersedes = [
+ memory_id
+ for memory_id in list(getattr(incoming, "supersedes", []) or [])
+ if clean_text(memory_id) not in restored_ids
+ ]
+ return restored
+
+ @staticmethod
+ def _honor_keep_parallel_decision(
+ graph: Any,
+ incoming: Any,
+ current: Sequence[Mapping[str, Any]],
+ ) -> list[str]:
+ restored = RealGraphBackend._restore_auto_superseded_current(
+ graph,
+ incoming,
+ current,
+ decision="keep_parallel",
+ )
+ if restored:
+ incoming_metadata = dict(getattr(incoming, "metadata", {}) or {})
+ incoming_metadata["conflict_action"] = "keep_parallel"
+ incoming_metadata["conflict_reason"] = "v4_reconciliation_keep_parallel"
+ incoming.metadata = incoming_metadata
+ return restored
+
+ @staticmethod
+ def _honor_challenge_decision(
+ graph: Any,
+ incoming: Any,
+ current: Sequence[Mapping[str, Any]],
+ ) -> list[str]:
+ restored = RealGraphBackend._restore_auto_superseded_current(
+ graph,
+ incoming,
+ current,
+ decision="challenge",
+ )
+ if restored:
+ incoming_metadata = dict(getattr(incoming, "metadata", {}) or {})
+ incoming_metadata["conflict_action"] = "challenge"
+ incoming_metadata["conflict_reason"] = "v4_reconciliation_challenge"
+ incoming.metadata = incoming_metadata
+ return restored
+
+ @staticmethod
+ def _remove_empty_graph_benchmark_metadata(record: Any) -> list[str]:
+ metadata = dict(getattr(record, "metadata", {}) or {})
+ removed: list[str] = []
+ for key in sorted(GRAPH_INJECTED_BENCHMARK_METADATA_KEYS.intersection(metadata)):
+ value = metadata[key]
+ if value not in (None, "", [], {}, False):
+ raise ProductWriterError(
+ f"{getattr(record, 'memory_id', '')}: graph injected non-empty benchmark metadata {key}"
+ )
+ del metadata[key]
+ removed.append(key)
+ if removed:
+ record.metadata = metadata
+ return removed
+
+ def commit_message(
+ self,
+ *,
+ message: SourceMessage,
+ source_record_id: str,
+ source_turn_index: int,
+ extraction: Mapping[str, Any],
+ durabilities: Sequence[str],
+ decisions: Mapping[int, str],
+ current_by_index: Mapping[int, Sequence[Mapping[str, Any]]],
+ duplicate_provenance: Sequence[Mapping[str, Any]] = (),
+ transaction_hook: Callable[[sqlite3.Connection, int], None] | None = None,
+ ) -> int:
+ self._reload()
+ source_record = self.adapter.graph.records_by_id.get(source_record_id)
+ if source_record is None or int(source_record.turn_index) != int(source_turn_index):
+ raise ProductWriterError(
+ f"{message.message_id}: real source record/turn is missing before enrichment"
+ )
+ source_metadata = self._source_metadata(source_record)
+ source_metadata["enrichment_status"] = "enriched"
+ source_metadata.pop("enrichment_error", None)
+ source_record.metadata = source_metadata
+ for item in duplicate_provenance:
+ leaf_id = clean_text(item.get("leaf_id"))
+ leaf = self.adapter.graph.records_by_id.get(leaf_id)
+ if leaf is None:
+ raise ProductWriterError(
+ f"real fast leaf not found for provenance: {leaf_id}"
+ )
+ metadata = self._source_metadata(leaf)
+ provenance_entry = {
+ **dict(item.get("provenance") or {}),
+ "source_record_id": source_record_id,
+ "source_turn_index": int(source_turn_index),
+ }
+ values = list(metadata.get("provenance") or [])
+ if provenance_entry not in values:
+ values.append(provenance_entry)
+ metadata["provenance"] = values
+ leaf.metadata = metadata
+ self.adapter.graph._upsert_memory_edge(
+ self._edge_class(
+ edge_id=f"{leaf_id}->{source_record_id}:grounded_in",
+ source_memory_id=leaf_id,
+ target_memory_id=source_record_id,
+ edge_type="grounded_in",
+ score=1.0,
+ model_score=0.0,
+ evidence_turn=int(source_turn_index),
+ evidence=str(
+ provenance_entry.get("evidence_quote") or leaf.value
+ ),
+ metadata={
+ "edge_source": "product_writer_provenance",
+ **provenance_entry,
+ },
+ )
+ )
+ records, _ = build_graph_records(
+ self._record_class,
+ scope_id=self.scope_id,
+ turn_index=source_turn_index,
+ session_id=message.session_id,
+ session_index=message.session_index,
+ message_id=message.message_id,
+ message_index=message.message_index,
+ date=message.timestamp[:10],
+ timestamp=message.timestamp,
+ role=message.role,
+ content=message.content,
+ extraction=extraction,
+ )
+ semantic_records = [
+ record
+ for record in records
+ if self._source_metadata(record).get("content_variant") != "source_message"
+ ]
+ assertion_by_index = {
+ int(self._source_metadata(record).get("llm_write_proposal_index", -1)): record
+ for record in semantic_records
+ if self._source_metadata(record).get("content_variant") == "product_semantic_memory"
+ }
+ decision_by_event_signature: dict[str, str] = {}
+ desired_state_by_id: dict[str, str] = {}
+ for record in semantic_records:
+ metadata = self._source_metadata(record)
+ metadata["source_record_id"] = source_record_id
+ metadata["enrichment_status"] = "enriched"
+ metadata["writer_schema_version"] = BATCH_SCHEMA_VERSION
+ metadata["prompt_version"] = PROMPT_VERSION
+ metadata["source"] = "tmcra_v4_batch_writer"
+ if metadata.get("content_variant") == "product_semantic_memory":
+ assertion_index = int(metadata.get("llm_write_proposal_index", -1))
+ if not 0 <= assertion_index < len(durabilities):
+ raise ProductWriterError(
+ f"{message.message_id}: assertion durability index is invalid"
+ )
+ metadata["durability"] = durabilities[assertion_index]
+ decision = decisions.get(assertion_index, "insert")
+ metadata["reconciliation_decision"] = decision
+ event_signature = clean_text(metadata.get("event_signature"))
+ if event_signature:
+ decision_by_event_signature[event_signature] = decision
+ if decision in {"keep_parallel", "challenge", "quarantine"}:
+ metadata["write_operation"] = "append"
+ metadata["allow_parallel_state"] = True
+ if decision == "keep_parallel":
+ desired_state_by_id[record.memory_id] = "parallel_active"
+ metadata["conflict_action"] = "keep_parallel"
+ elif decision == "challenge":
+ desired_state_by_id[record.memory_id] = "challenged"
+ metadata["conflict_action"] = "challenge"
+ elif decision == "quarantine":
+ desired_state_by_id[record.memory_id] = "quarantined"
+ metadata["conflict_action"] = "quarantine"
+ metadata["excluded_from_retrieval"] = True
+ elif decision == "replace_current":
+ metadata["conflict_action"] = "replace_current"
+ if metadata.get("content_variant") == "product_interaction":
+ interaction_index = int(str(metadata.get("interaction_id", "").rsplit(".", 1)[-1]).split(":", 1)[0] or 0)
+ interaction_id = _deterministic_interaction_id(self.scope_id, message.message_id, interaction_index)
+ record.memory_id = interaction_id
+ metadata["interaction_id"] = interaction_id
+ record.metadata = metadata
+
+ new_records = []
+ for record in semantic_records:
+ metadata = self._source_metadata(record)
+ if record.memory_id in self.adapter.graph.records_by_id:
+ continue
+ if metadata.get("content_variant") == "event_facet_write":
+ parent_signature = clean_text(metadata.get("facet_parent_event_signature"))
+ if decision_by_event_signature.get(parent_signature) == "quarantine":
+ continue
+ new_records.append(record)
+ stored_ids = self.adapter.graph.add_records(new_records)
+ for assertion_index, record in assertion_by_index.items():
+ persisted = self.adapter.graph.records_by_id.get(record.memory_id)
+ if persisted is None:
+ continue
+ decision = decisions.get(assertion_index, "insert")
+ self._restore_replayed_semantic_record(
+ self.adapter.graph,
+ persisted,
+ record,
+ decision,
+ )
+ desired_state = desired_state_by_id.get(record.memory_id)
+ if desired_state:
+ persisted.state = desired_state
+ if decision == "keep_parallel":
+ self._honor_keep_parallel_decision(
+ self.adapter.graph,
+ persisted,
+ current_by_index.get(assertion_index, []),
+ )
+ elif decision == "challenge":
+ self._honor_challenge_decision(
+ self.adapter.graph,
+ persisted,
+ current_by_index.get(assertion_index, []),
+ )
+ if decision in {"challenge", "quarantine"}:
+ slot_key = clean_text(self._source_metadata(persisted).get("canonical_slot_key"))
+ if self.adapter.graph.slot_heads.get(slot_key) == persisted.memory_id:
+ replacement = next(
+ (
+ str(item["memory_id"])
+ for item in current_by_index.get(assertion_index, [])
+ if str(item.get("record_state"))
+ in {"active", "parallel_active", "promoted"}
+ ),
+ "",
+ )
+ if replacement:
+ self.adapter.graph.slot_heads[slot_key] = replacement
+ else:
+ self.adapter.graph.slot_heads.pop(slot_key, None)
+ for assertion_index, decision in decisions.items():
+ if decision != "replace_current":
+ continue
+ incoming = assertion_by_index.get(assertion_index)
+ if incoming is None:
+ raise ProductWriterError(
+ f"{message.message_id}: replacement assertion record is missing"
+ )
+ persisted_incoming = self.adapter.graph.records_by_id.get(
+ incoming.memory_id
+ )
+ if persisted_incoming is None:
+ raise ProductWriterError(
+ f"{message.message_id}: replacement assertion was not persisted"
+ )
+ incoming_metadata = self._source_metadata(persisted_incoming)
+ for current in current_by_index.get(assertion_index, []):
+ current_id = clean_text(current.get("memory_id"))
+ if not current_id or current_id == incoming.memory_id:
+ continue
+ old = self.adapter.graph.records_by_id.get(current_id)
+ if old is None:
+ raise ProductWriterError(
+ f"{incoming.memory_id}: replacement target disappeared: {current_id}"
+ )
+ old_metadata = self._source_metadata(old)
+ existing_link = clean_text(old_metadata.get("superseded_by"))
+ existing_reason = clean_text(
+ old_metadata.get("superseded_reason")
+ )
+ if str(old.state) == "superseded":
+ allowed_reasons = {
+ "v4_reconciliation_replace_current",
+ *GRAPH_AUTO_SUPERSESSION_REASONS,
+ }
+ if (
+ existing_reason not in allowed_reasons
+ or existing_link not in {"", incoming.memory_id}
+ ):
+ raise ProductWriterError(
+ f"{incoming.memory_id}: replacement target has an incompatible supersession lifecycle"
+ )
+ if str(old.state) not in {
+ "active", "parallel_active", "promoted", "superseded"
+ }:
+ raise ProductWriterError(
+ f"{incoming.memory_id}: replacement target state is unsupported: {old.state!r}"
+ )
+ old.state = "superseded"
+ old_metadata["superseded_by"] = incoming.memory_id
+ old_metadata["superseded_reason"] = (
+ "v4_reconciliation_replace_current"
+ )
+ old.metadata = old_metadata
+ supersedes = list(
+ getattr(persisted_incoming, "supersedes", []) or []
+ )
+ if current_id not in supersedes:
+ supersedes.append(current_id)
+ persisted_incoming.supersedes = supersedes
+ persisted_incoming.state = "active"
+ incoming_metadata.pop("superseded_by", None)
+ incoming_metadata.pop("superseded_reason", None)
+ persisted_incoming.metadata = incoming_metadata
+ self.adapter.graph.slot_heads[str(persisted_incoming.slot_key)] = (
+ persisted_incoming.memory_id
+ )
+ for memory_id in stored_ids:
+ stored = self.adapter.graph.records_by_id.get(str(memory_id))
+ if stored is not None:
+ self._remove_empty_graph_benchmark_metadata(stored)
+ for record in semantic_records:
+ if record.memory_id not in self.adapter.graph.records_by_id:
+ continue
+ metadata = self._source_metadata(record)
+ if metadata.get("content_variant") not in {"product_semantic_memory", "product_interaction"}:
+ continue
+ self.adapter.graph._upsert_memory_edge(
+ self._edge_class(
+ edge_id=f"{record.memory_id}->{source_record_id}:grounded_in",
+ source_memory_id=record.memory_id,
+ target_memory_id=source_record_id,
+ edge_type="grounded_in",
+ score=1.0,
+ model_score=0.0,
+ evidence_turn=source_turn_index,
+ evidence=str(metadata.get("source_span") or record.value),
+ metadata={"edge_source": "product_writer_provenance", "message_id": message.message_id, "source_record_id": source_record_id},
+ )
+ )
+ for assertion_index, decision in decisions.items():
+ if decision not in {"challenge", "quarantine"}:
+ continue
+ candidate = assertion_by_index.get(assertion_index)
+ if candidate is None or candidate.memory_id not in self.adapter.graph.records_by_id:
+ continue
+ for current in current_by_index.get(assertion_index, []):
+ current_id = str(current["memory_id"])
+ edge_type = "contradicts" if decision == "challenge" else "quarantined_against"
+ self.adapter.graph._upsert_memory_edge(
+ self._edge_class(
+ edge_id=f"{candidate.memory_id}->{current_id}:{edge_type}",
+ source_memory_id=candidate.memory_id,
+ target_memory_id=current_id,
+ edge_type=edge_type,
+ score=1.0,
+ model_score=0.0,
+ evidence_turn=source_turn_index,
+ evidence=str(candidate.value),
+ metadata={"edge_source": "v4_reconciliation", "decision": decision, "canonical_slot_key": candidate.metadata.get("canonical_slot_key")},
+ )
+ )
+ for resolution in list(extraction.get("resolutions") or []):
+ target_id = str(resolution["interaction_id"])
+ target = self.adapter.graph.records_by_id.get(target_id)
+ if target is None:
+ raise ProductWriterError(f"{message.message_id}: resolution target does not exist: {target_id}")
+ target_meta = self._source_metadata(target)
+ previous_status = clean_text(target_meta.get("interaction_status")) or "open"
+ next_status = "resolved" if resolution["resolution"] == "resolved" else ("partial" if resolution["resolution"] == "partial" else previous_status)
+ target_meta["interaction_status"] = next_status
+ target_meta.setdefault("resolution_history", []).append({"message_id": message.message_id, "source_record_id": source_record_id, "resolution": resolution["resolution"], "evidence_quote": resolution["evidence_quote"]})
+ target.metadata = target_meta
+ edge_type = {"resolved": "answered_by", "partial": "partially_answered_by", "unresolved": "responded_without_resolution"}[resolution["resolution"]]
+ self.adapter.graph._upsert_memory_edge(self._edge_class(edge_id=f"{target_id}->{source_record_id}:{edge_type}", source_memory_id=target_id, target_memory_id=source_record_id, edge_type=edge_type, score=1.0 if edge_type == "answered_by" else 0.72, model_score=0.0, evidence_turn=source_turn_index, evidence=str(resolution["evidence_quote"]), metadata={"edge_source": "product_writer_resolution", "message_id": message.message_id, "resolution": resolution["resolution"]}))
+
+ event_ids = [source_record_id, *stored_ids]
+ for event in self.adapter.graph.turn_log:
+ if int(event.get("turn_index", -1)) == int(source_turn_index):
+ event["record_ids"] = list(dict.fromkeys([*event.get("record_ids", []), *event_ids]))
+ event.setdefault("metadata", {})["enrichment_status"] = "enriched"
+ break
+ committed_count = sum(
+ 1
+ for assertion_index, record in assertion_by_index.items()
+ if decisions.get(assertion_index) != "quarantine"
+ and record.memory_id in self.adapter.graph.records_by_id
+ )
+ self._persist(
+ None
+ if transaction_hook is None
+ else lambda connection: transaction_hook(connection, committed_count)
+ )
+ return committed_count
+
+
+class RealGraphFactory:
+ def __init__(self, *, repo: Path, database: Path) -> None:
+ self.repo = Path(repo)
+ self.database = Path(database)
+ self.backends: dict[str, RealGraphBackend] = {}
+
+ def for_scope(self, scope_id: str) -> RealGraphBackend:
+ if scope_id not in self.backends:
+ self.backends[scope_id] = RealGraphBackend(repo=self.repo, database=self.database, scope_id=scope_id)
+ return self.backends[scope_id]
+
+
+def _client_result(result: Any) -> tuple[Mapping[str, Any] | str, dict[str, Any]]:
+ if isinstance(result, tuple) and len(result) == 2:
+ return result[0], dict(result[1] or {})
+ return result, {}
+
+
+def _normalized_evidence(value: str) -> str:
+ return clean_text(unicodedata.normalize("NFKC", value))
+
+
+def _normalized_claim(value: str) -> str:
+ return clean_text(unicodedata.normalize("NFKC", value)).casefold()
+
+
+def build_graph_records(record_class: Any, **kwargs: Any) -> tuple[list[Any], dict[str, int]]:
+ """Build V3-compatible records while keeping claims separate from evidence."""
+ extraction = kwargs.get("extraction")
+ records, counts = _build_v3_graph_records(record_class, **kwargs)
+ assertions = list((extraction or {}).get("assertions") or [])
+ for record in records:
+ metadata = dict(getattr(record, "metadata", {}) or {})
+ if metadata.get("content_variant") != "product_semantic_memory":
+ continue
+ assertion_index = int(metadata.get("llm_write_proposal_index", -1))
+ if not 0 <= assertion_index < len(assertions):
+ raise ProductWriterError("semantic record cannot be mapped to its assertion")
+ assertion = assertions[assertion_index]
+ claim_text = clean_text(assertion.get("claim_text"))
+ raw_evidence_quote = assertion.get("evidence_quote")
+ evidence_quote = (
+ raw_evidence_quote if isinstance(raw_evidence_quote, str) else ""
+ )
+ if not claim_text or not evidence_quote:
+ raise ProductWriterError(
+ "V4 semantic records require claim_text and exact evidence_quote"
+ )
+ source_turn_text = metadata.get("source_turn_text")
+ try:
+ evidence_start = int(metadata["evidence_char_start"])
+ evidence_end = int(metadata["evidence_char_end"])
+ except (KeyError, TypeError, ValueError) as exc:
+ raise ProductWriterError(
+ "V4 semantic record lacks exact evidence offsets"
+ ) from exc
+ if (
+ not isinstance(source_turn_text, str)
+ or evidence_start < 0
+ or evidence_end <= evidence_start
+ or evidence_end > len(source_turn_text)
+ or source_turn_text[evidence_start:evidence_end] != evidence_quote
+ or metadata.get("raw_content") != evidence_quote
+ or metadata.get("source_span") != evidence_quote
+ ):
+ raise ProductWriterError(
+ "V4 semantic evidence quote differs from its exact Source slice"
+ )
+ record.value = claim_text
+ metadata["claim_text"] = claim_text
+ metadata["evidence_quote"] = evidence_quote
+ metadata["semantic_value_kind"] = "atomic_claim"
+ record.metadata = metadata
+ return records, counts
+
+
+def _binding_identity(value: Mapping[str, Any]) -> dict[str, str]:
+ metadata = dict(value.get("metadata") or {})
+ return {
+ "memory_id": clean_text(value.get("memory_id")),
+ "claim_text": _normalized_claim(
+ clean_text(value.get("claim_text") or value.get("value"))
+ ),
+ "canonical_slot_key": clean_text(
+ value.get("canonical_slot_key") or metadata.get("canonical_slot_key")
+ ),
+ "evidence_quote": _normalized_evidence(
+ clean_text(value.get("evidence_quote") or value.get("value"))
+ ),
+ "durability": clean_text(
+ value.get("durability") or metadata.get("durability")
+ ),
+ "source_record_id": clean_text(
+ value.get("source_record_id") or metadata.get("source_record_id")
+ ),
+ "entity_key": clean_text(
+ value.get("entity_key") or metadata.get("entity_key")
+ ),
+ "graph_entity_key": clean_text(
+ value.get("graph_entity_key") or metadata.get("graph_entity_key")
+ ),
+ "attribute_key": clean_text(
+ value.get("attribute_key") or metadata.get("attribute_key")
+ ),
+ "memory_type": clean_text(
+ value.get("memory_type") or metadata.get("memory_type")
+ ),
+ "memory_family": clean_text(
+ value.get("memory_family") or metadata.get("memory_family")
+ ),
+ "temporal_status": clean_text(
+ value.get("temporal_status") or metadata.get("target_status")
+ ),
+ "polarity": clean_text(value.get("polarity") or metadata.get("polarity")),
+ }
+
+
+def _binding_semantic_identity(value: Mapping[str, Any]) -> dict[str, str]:
+ identity = _binding_identity(value)
+ identity.pop("memory_id")
+ return identity
+
+
+def _reconciliation_job_id(
+ batch: SourceBatch,
+ message: SourceMessage,
+ assertion_index: int,
+ assertion: Mapping[str, Any],
+) -> str:
+ return sha256_text(
+ _json(
+ {
+ "batch_id": batch.batch_id,
+ "message_id": message.message_id,
+ "assertion_index": assertion_index,
+ "slot": _graph_slot_key(assertion["canonical_key"]),
+ "evidence": assertion["evidence_quote"],
+ }
+ )
+ )[:32]
+
+
+class V4BatchWriter:
+ def __init__(
+ self,
+ *,
+ store: V4BatchStore,
+ flash_client: BatchClient,
+ pro_client: ReconciliationClient | None = None,
+ graph_factory: RealGraphFactory | None = None,
+ log_dir: Path | None = None,
+ revalidate_failed_raw_response: bool = False,
+ recover_interrupted_api_calls: bool = False,
+ ) -> None:
+ self.store = store
+ self.flash_client = flash_client
+ self.pro_client = pro_client
+ self.graph_factory = graph_factory
+ self.log_dir = Path(log_dir) if log_dir is not None else None
+ if self.log_dir is not None:
+ self.log_dir.mkdir(parents=True, exist_ok=True)
+ self.revalidate_failed_raw_response = bool(revalidate_failed_raw_response)
+ self.recover_interrupted_api_calls = bool(recover_interrupted_api_calls)
+ self.stats = {
+ "batches": 0,
+ "resumed_batches": 0,
+ "flash_calls": 0,
+ "pro_calls": 0,
+ "input_messages": 0,
+ "source_messages": 0,
+ "excluded_empty_source_messages": 0,
+ "fast_assertion_leaves": 0,
+ "reconciliation_jobs": 0,
+ "reconciliation_response_quarantines": 0,
+ "validation_warnings": 0,
+ "interrupted_call_recoveries": 0,
+ "validated_batch_recoveries": 0,
+ "historical_binding_recoveries": 0,
+ "committed_source_status_repairs": 0,
+ }
+
+ def _append_unique_jsonl(self, filename: str, key: str, value: Mapping[str, Any]) -> None:
+ if self.log_dir is None:
+ return
+ path = self.log_dir / filename
+ identity = clean_text(value.get(key))
+ if path.exists():
+ for line in path.read_text(encoding="utf-8").splitlines():
+ try:
+ if clean_text(json.loads(line).get(key)) == identity:
+ return
+ except json.JSONDecodeError:
+ continue
+ with path.open("a", encoding="utf-8") as handle:
+ handle.write(_json(value) + "\n")
+
+ def _artifact_count(self, filename: str, call_key: str) -> int:
+ if self.log_dir is None:
+ raise ProductWriterError(
+ "interrupted call recovery requires a durable log directory"
+ )
+ path = self.log_dir / filename
+ if not path.exists():
+ return 0
+ count = 0
+ for line in path.read_text(encoding="utf-8").splitlines():
+ if not line.strip():
+ continue
+ value = json.loads(line)
+ count += int(clean_text(value.get("call_key")) == call_key)
+ return count
+
+ def _record_interrupted_call(
+ self,
+ *,
+ call_key: str,
+ batch: SourceBatch,
+ stage: str,
+ model: str,
+ job_id: str = "",
+ ) -> None:
+ physical_call_id = "interrupted:" + sha256_text(call_key)[:32]
+ self._append_unique_jsonl(
+ "product_writer_interrupted_calls.jsonl",
+ "call_key",
+ {
+ "call_key": call_key,
+ "batch_id": batch.batch_id,
+ "scope_id": batch.scope_id,
+ "session_id": batch.session_id,
+ "job_id": job_id,
+ "stage": stage,
+ "model": model,
+ "status": "outcome_unknown_after_confirmed_process_loss",
+ "physical_api_call": True,
+ "physical_api_calls": 1,
+ "physical_call_id": physical_call_id,
+ "usage_recorded": False,
+ "replacement_call_authorized": True,
+ "replacement_model": model,
+ "same_model_replacement": True,
+ "recovered_at": _now(),
+ },
+ )
+ self.stats["interrupted_call_recoveries"] += 1
+
+ def _assert_interrupted_call_has_no_response(self, call_key: str) -> None:
+ raw_count = self._artifact_count(
+ "product_writer_raw_responses.jsonl", call_key
+ )
+ call_count = self._artifact_count("product_writer_calls.jsonl", call_key)
+ if raw_count or call_count:
+ raise ProductWriterError(
+ f"{call_key}: interrupted call has durable response/call artifacts; refusing replacement"
+ )
+
+ def _record_api_call(
+ self,
+ *,
+ call_key: str,
+ batch: SourceBatch,
+ stage: str,
+ model: str,
+ metadata: Mapping[str, Any],
+ job_id: str = "",
+ error: str = "",
+ ) -> None:
+ if not metadata:
+ return
+ self._append_unique_jsonl(
+ "product_writer_calls.jsonl",
+ "call_key",
+ {
+ "call_key": call_key,
+ "batch_id": batch.batch_id,
+ "scope_id": batch.scope_id,
+ "session_id": batch.session_id,
+ "job_id": job_id,
+ "model": model,
+ "stage": stage,
+ "api_call_count": 1,
+ "physical_api_call_count": 1,
+ "metadata": dict(metadata),
+ "error": error,
+ },
+ )
+
+ def _record_raw_api_response(
+ self,
+ *,
+ call_key: str,
+ batch: SourceBatch,
+ stage: str,
+ model: str,
+ response: Any,
+ metadata: Mapping[str, Any],
+ job_id: str = "",
+ ) -> None:
+ raw_response = response if isinstance(response, str) else _json(response)
+ self._append_unique_jsonl(
+ "product_writer_raw_responses.jsonl",
+ "call_key",
+ {
+ "call_key": call_key,
+ "batch_id": batch.batch_id,
+ "scope_id": batch.scope_id,
+ "session_id": batch.session_id,
+ "job_id": job_id,
+ "stage": stage,
+ "model": model,
+ "raw_response": raw_response,
+ "raw_response_sha256": sha256_text(raw_response),
+ "metadata_response_sha256": clean_text(
+ metadata.get("response_sha256")
+ ),
+ },
+ )
+
+ def _raw_api_response(self, call_key: str) -> tuple[str, dict[str, Any]]:
+ if self.log_dir is None:
+ raise ProductWriterError("raw response revalidation requires a log directory")
+ path = self.log_dir / "product_writer_raw_responses.jsonl"
+ if not path.is_file():
+ raise ProductWriterError("raw response revalidation artifact is missing")
+ matches: list[dict[str, Any]] = []
+ for line in path.read_text(encoding="utf-8").splitlines():
+ if not line.strip():
+ continue
+ value = json.loads(line)
+ if clean_text(value.get("call_key")) == call_key:
+ matches.append(value)
+ if len(matches) != 1:
+ raise ProductWriterError(
+ f"{call_key}: expected exactly one raw response, found {len(matches)}"
+ )
+ record = matches[0]
+ raw_response = record.get("raw_response")
+ if not isinstance(raw_response, str) or not raw_response:
+ raise ProductWriterError(f"{call_key}: raw response is empty")
+ raw_hash = sha256_text(raw_response)
+ if raw_hash != clean_text(record.get("raw_response_sha256")):
+ raise ProductWriterError(f"{call_key}: raw response hash differs")
+ metadata_hash = clean_text(record.get("metadata_response_sha256"))
+ if not metadata_hash or metadata_hash != raw_hash:
+ raise ProductWriterError(
+ f"{call_key}: raw response does not match durable API metadata"
+ )
+ return raw_response, record
+
+ def _revalidate_failed_reconciliation_batch(
+ self,
+ batch: SourceBatch,
+ row: Mapping[str, Any],
+ ) -> sqlite3.Row:
+ jobs = self.store.reconciliation_jobs_for_batch(batch.batch_id)
+ if not jobs:
+ raise ProductWriterError(
+ f"{batch.batch_id}: failed validated batch lacks durable reconciliation state"
+ )
+ failed_jobs = [job for job in jobs if clean_text(job["status"]) == "failed"]
+ invalid = [
+ (clean_text(job["job_id"]), clean_text(job["status"]))
+ for job in jobs
+ if clean_text(job["status"])
+ not in {"completed", "failed", "pro_pending", "pro_started"}
+ ]
+ if invalid:
+ raise ProductWriterError(
+ f"{batch.batch_id}: reconciliation jobs have unsupported recovery states: {invalid}"
+ )
+ recovered_job_ids: list[str] = []
+ for job in failed_jobs:
+ job_id = clean_text(job["job_id"])
+ metadata = json.loads(str(job["response_metadata_json"] or "{}"))
+ if (
+ clean_text(metadata.get("status")) != "completed"
+ or metadata.get("physical_api_call") is not True
+ or int(metadata.get("http_status") or 0) != 200
+ ):
+ raise ProductWriterError(
+ f"{job_id}: failed reconciliation lacks one clean completed API response"
+ )
+ raw_response, raw_record = self._raw_api_response(f"pro:{job_id}")
+ parsed = _strict_json_object(
+ raw_response, f"revalidation[reconciliation:{job_id}]"
+ )
+ request = json.loads(str(job["request_json"]))
+ candidates = request.get("candidate_cited_leaves")
+ exact_slot_match = request.get("exact_slot_match")
+ if not isinstance(candidates, list) or type(exact_slot_match) is not bool:
+ raise ProductWriterError(
+ f"{job_id}: frozen reconciliation request is malformed"
+ )
+ adjudication = self._validate_reconciliation_response(
+ parsed,
+ current_cited=candidates,
+ exact_slot_match=exact_slot_match,
+ path=f"revalidation[reconciliation:{job_id}]",
+ )
+ recovery_metadata = {
+ **metadata,
+ "raw_response_revalidated": True,
+ "revalidated_at": _now(),
+ "revalidation_raw_response_sha256": raw_record[
+ "raw_response_sha256"
+ ],
+ "prior_error_sha256": sha256_text(clean_text(job["error"])),
+ "model_adjudication_sha256": _hash_json(parsed),
+ "normalized_adjudication_sha256": _hash_json(adjudication),
+ "physical_api_calls_revalidation": 0,
+ }
+ if parsed != adjudication:
+ recovery_metadata["controller_normalization"] = (
+ "slot_decision_from_selected_candidate_and_conflict_action"
+ )
+ self.store.revalidate_failed_reconciliation_job(
+ job_id,
+ adjudication["decision"],
+ adjudication,
+ recovery_metadata,
+ )
+ self._append_unique_jsonl(
+ "product_writer_reconciliation_revalidations.jsonl",
+ "job_id",
+ {
+ "job_id": job_id,
+ "batch_id": batch.batch_id,
+ "raw_response_sha256": raw_record["raw_response_sha256"],
+ "normalized_adjudication_sha256": _hash_json(adjudication),
+ "controller_normalization": recovery_metadata.get(
+ "controller_normalization", "none"
+ ),
+ "physical_api_calls": 0,
+ },
+ )
+ recovered_job_ids.append(job_id)
+ interrupted_job_ids: list[str] = []
+ for job in jobs:
+ if clean_text(job["status"]) != "pro_started":
+ continue
+ job_id = clean_text(job["job_id"])
+ self._recover_interrupted_reconciliation_call(batch, job_id)
+ interrupted_job_ids.append(job_id)
+
+ current_jobs = self.store.reconciliation_jobs_for_batch(batch.batch_id)
+ pending_job_ids = sorted(
+ clean_text(job["job_id"])
+ for job in current_jobs
+ if clean_text(job["status"]) == "pro_pending"
+ )
+ incomplete = [
+ (clean_text(job["job_id"]), clean_text(job["status"]))
+ for job in current_jobs
+ if clean_text(job["status"]) not in {"completed", "pro_pending"}
+ ]
+ if incomplete:
+ raise ProductWriterError(
+ f"{batch.batch_id}: reconciliation recovery did not reach durable states: {incomplete}"
+ )
+ batch_metadata = json.loads(str(row.get("response_metadata_json") or "{}"))
+ batch_metadata.update(
+ {
+ "validated_batch_commit_recovered": True,
+ "revalidated_at": _now(),
+ "revalidated_reconciliation_job_ids": recovered_job_ids,
+ "interrupted_reconciliation_job_ids": interrupted_job_ids,
+ "pending_reconciliation_job_ids": pending_job_ids,
+ "physical_api_calls_revalidation": 0,
+ }
+ )
+ if recovered_job_ids:
+ batch_metadata["reconciliation_raw_response_revalidated"] = True
+ recovery_artifact = {
+ "schema_version": "tmcra.v4.validated-batch-recovery.1",
+ "batch_id": batch.batch_id,
+ "scope_id": batch.scope_id,
+ "session_id": batch.session_id,
+ "response_sha256": clean_text(row.get("response_sha256")),
+ "prior_error_sha256": sha256_text(clean_text(row.get("error"))),
+ "completed_reconciliation_job_ids": sorted(
+ clean_text(job["job_id"])
+ for job in current_jobs
+ if clean_text(job["status"]) == "completed"
+ ),
+ "revalidated_reconciliation_job_ids": recovered_job_ids,
+ "interrupted_reconciliation_job_ids": interrupted_job_ids,
+ "pending_reconciliation_job_ids": pending_job_ids,
+ "physical_api_calls": 0,
+ "recovered_at": _now(),
+ }
+ self._append_unique_jsonl(
+ "product_writer_validated_batch_recoveries.jsonl",
+ "batch_id",
+ recovery_artifact,
+ )
+ self.stats["validated_batch_recoveries"] += 1
+ return self.store.resume_failed_validated_batch(
+ batch.batch_id,
+ batch_metadata,
+ allowed_pending_job_ids=pending_job_ids,
+ )
+
+ def _recover_interrupted_batch_call(
+ self, batch: SourceBatch
+ ) -> sqlite3.Row:
+ if not self.recover_interrupted_api_calls:
+ raise ProductWriterError(
+ f"{batch.batch_id}: API call was started without a durable response; refusing retry"
+ )
+ call_key = f"flash:{batch.batch_id}"
+ self._assert_interrupted_call_has_no_response(call_key)
+ self._record_interrupted_call(
+ call_key=call_key,
+ batch=batch,
+ stage="batch_flash_interrupted",
+ model="deepseek-v4-flash",
+ )
+ return self.store.abandon_interrupted_batch_call(batch.batch_id)
+
+ def _recover_interrupted_reconciliation_call(
+ self, batch: SourceBatch, job_id: str
+ ) -> sqlite3.Row:
+ if not self.recover_interrupted_api_calls:
+ raise ProductWriterError(
+ f"{job_id}: reconciliation has an uncertain external-call outcome; refusing replacement"
+ )
+ call_key = f"pro:{job_id}"
+ self._assert_interrupted_call_has_no_response(call_key)
+ self._record_interrupted_call(
+ call_key=call_key,
+ batch=batch,
+ stage="reconciliation_pro_interrupted",
+ model="deepseek-v4-pro",
+ job_id=job_id,
+ )
+ job = self.store.abandon_interrupted_reconciliation_call(job_id)
+ if job["status"] != "pro_pending":
+ raise ProductWriterError(
+ f"{job_id}: interrupted Pro recovery did not return to pro_pending"
+ )
+ return job
+
+ def _revalidate_failed_batch(
+ self,
+ batch: SourceBatch,
+ row: Mapping[str, Any],
+ unresolved: Sequence[Mapping[str, Any]],
+ ) -> sqlite3.Row:
+ row_data = dict(row)
+ if clean_text(row_data.get("response_json")):
+ return self._revalidate_failed_reconciliation_batch(
+ batch, row_data
+ )
+ metadata = json.loads(str(row_data.get("response_metadata_json") or "{}"))
+ if (
+ clean_text(metadata.get("status")) != "completed"
+ or metadata.get("physical_api_call") is not True
+ or int(metadata.get("http_status") or 0) != 200
+ ):
+ raise ProductWriterError(
+ f"{batch.batch_id}: failed batch lacks one clean completed API response"
+ )
+ raw_response, raw_record = self._raw_api_response(
+ f"flash:{batch.batch_id}"
+ )
+ raw_payload = _strict_json_object(
+ raw_response, f"revalidation[{batch.batch_id}]"
+ )
+ validated = validate_batch_response(raw_payload, batch, unresolved)
+ recovery_metadata = {
+ **metadata,
+ "raw_response_revalidated": True,
+ "revalidated_at": _now(),
+ "revalidation_prompt_version": PROMPT_VERSION,
+ "revalidation_raw_response_sha256": raw_record["raw_response_sha256"],
+ "prior_error_sha256": sha256_text(clean_text(row_data.get("error"))),
+ "validated_response_sha256": _hash_json(validated),
+ }
+ persisted = self.store.revalidate_failed_response(
+ batch.batch_id, validated, recovery_metadata
+ )
+ self._append_unique_jsonl(
+ "product_writer_revalidations.jsonl",
+ "batch_id",
+ {
+ "batch_id": batch.batch_id,
+ "raw_response_sha256": raw_record["raw_response_sha256"],
+ "validated_response_sha256": _hash_json(validated),
+ "prompt_version": PROMPT_VERSION,
+ "physical_api_calls": 0,
+ },
+ )
+ return persisted
+
+ def _ensure_graph_sources(
+ self,
+ batch: SourceBatch,
+ *,
+ verify_only: bool = False,
+ ) -> RealGraphBackend:
+ if self.graph_factory is None:
+ raise ProductWriterError("--repo real graph backend is required")
+ backend = self.graph_factory.for_scope(batch.scope_id)
+ for message in batch.messages:
+ if verify_only:
+ info = self.store.source_info(batch.scope_id, message.message_id)
+ source_record_id = clean_text(info.get("source_record_id"))
+ if not source_record_id:
+ raise ProductWriterError(
+ f"{message.message_id}: committed source journal lacks a real graph record ID"
+ )
+ backend.verify_source(
+ message,
+ source_record_id,
+ int(info.get("source_turn_index") or 0),
+ )
+ continue
+ source_record_id, source_turn_index = backend.ensure_source(message)
+ self.store.set_source_record(
+ batch.scope_id,
+ message.message_id,
+ source_record_id,
+ source_turn_index,
+ )
+ return backend
+
+ def _set_graph_source_status(self, batch: SourceBatch, status: str, error: str = "") -> None:
+ if self.graph_factory is None:
+ return
+ backend = self.graph_factory.for_scope(batch.scope_id)
+ for message in batch.messages:
+ info = self.store.source_info(batch.scope_id, message.message_id)
+ source_record_id = clean_text(info.get("source_record_id"))
+ if source_record_id:
+ backend.set_enrichment_status(source_record_id, status, error)
+
+ def _reconcile(
+ self,
+ batch: SourceBatch,
+ message: SourceMessage,
+ assertion_index: int,
+ assertion: Mapping[str, Any],
+ durability: str,
+ current: list[dict[str, Any]],
+ *,
+ exact_slot_match: bool,
+ backend: Any,
+ ) -> tuple[dict[str, str], Mapping[str, Any] | None]:
+ slot = _graph_slot_key(assertion["canonical_key"])
+ cited = {
+ "canonical_slot_key": slot,
+ "claim_text": assertion["claim_text"],
+ "evidence_span_id": assertion["evidence_span_id"],
+ "evidence_quote": assertion["evidence_quote"],
+ "memory_type": assertion["memory_type"],
+ "entity_key": assertion["entity_key"],
+ "attribute_key": assertion["attribute_key"],
+ "operation": assertion["operation"],
+ "relation": assertion["relation"],
+ "temporal_status": assertion["temporal_status"],
+ "polarity": assertion["polarity"],
+ "durability": durability,
+ }
+ current_cited = []
+ for leaf in current:
+ metadata = dict(leaf.get("metadata") or {})
+ current_cited.append(
+ {
+ "memory_id": clean_text(leaf.get("memory_id")),
+ "canonical_slot_key": clean_text(leaf.get("canonical_slot_key")),
+ "claim_text": clean_text(
+ leaf.get("claim_text") or leaf.get("value")
+ ),
+ "evidence_quote": clean_text(
+ leaf.get("evidence_quote") or leaf.get("value")
+ ),
+ "durability": clean_text(
+ leaf.get("durability") or metadata.get("durability")
+ ),
+ "record_state": clean_text(leaf.get("record_state")),
+ "temporal_status": clean_text(metadata.get("target_status")),
+ "polarity": clean_text(metadata.get("polarity")),
+ "source_record_id": clean_text(metadata.get("source_record_id")),
+ "entity_key": clean_text(metadata.get("entity_key")),
+ "graph_entity_key": clean_text(metadata.get("graph_entity_key")),
+ "attribute_key": clean_text(metadata.get("attribute_key")),
+ "memory_type": clean_text(metadata.get("memory_type")),
+ "memory_family": clean_text(metadata.get("memory_family")),
+ }
+ )
+ request = {
+ "schema_version": RECONCILIATION_SCHEMA_VERSION,
+ "candidate_selector_version": CANDIDATE_SELECTOR_VERSION,
+ "message_id": message.message_id,
+ "canonical_slot_key": slot,
+ "exact_slot_match": bool(exact_slot_match),
+ "new_cited_assertion": cited,
+ "candidate_cited_leaves": current_cited,
+ }
+ job_id = _reconciliation_job_id(
+ batch, message, assertion_index, assertion
+ )
+ self.store.create_reconciliation_job(
+ job_id=job_id,
+ scope_id=batch.scope_id,
+ batch_id=batch.batch_id,
+ message_id=message.message_id,
+ slot=slot,
+ assertion_index=assertion_index,
+ request=request,
+ )
+ self.stats["reconciliation_jobs"] += 1
+ job = self.store.reconciliation_job(job_id)
+ if job is None:
+ raise ProductWriterError(f"{job_id}: reconciliation job disappeared")
+ frozen_request = json.loads(str(job["request_json"]))
+ frozen_candidates = frozen_request.get("candidate_cited_leaves")
+ frozen_exact_slot_match = frozen_request.get("exact_slot_match")
+ if (
+ not isinstance(frozen_candidates, list)
+ or type(frozen_exact_slot_match) is not bool
+ ):
+ raise ProductWriterError(
+ f"{job_id}: frozen reconciliation request is malformed"
+ )
+ def verify_current_binding(
+ adjudication: Mapping[str, str],
+ ) -> tuple[dict[str, str], Mapping[str, Any] | None]:
+ normalized = dict(adjudication)
+ if normalized.get("slot_decision") != "bind_existing":
+ return normalized, None
+ selected_memory_id = clean_text(normalized.get("selected_memory_id"))
+ selected_current = next(
+ (
+ item
+ for item in current
+ if clean_text(item.get("memory_id")) == selected_memory_id
+ ),
+ None,
+ )
+ if selected_current is not None:
+ frozen_current = next(
+ (
+ item
+ for item in frozen_candidates
+ if clean_text(item.get("memory_id"))
+ == selected_memory_id
+ ),
+ None,
+ )
+ if (
+ frozen_current is None
+ or _binding_identity(selected_current)
+ != _binding_identity(frozen_current)
+ ):
+ raise ProductWriterError(
+ f"{job_id}: current Pro selection differs from its frozen identity"
+ )
+ return normalized, selected_current
+
+ batch_row = self.store.batch_row(batch.batch_id)
+ batch_metadata = (
+ json.loads(str(batch_row["response_metadata_json"] or "{}"))
+ if batch_row is not None
+ else {}
+ )
+ frozen_selected = next(
+ (
+ item
+ for item in frozen_candidates
+ if clean_text(item.get("memory_id")) == selected_memory_id
+ ),
+ None,
+ )
+ historical = backend.leaf_by_id(selected_memory_id)
+ historical_metadata = (
+ dict(historical.get("metadata") or {})
+ if historical is not None
+ else {}
+ )
+ source_info = self.store.source_info(
+ batch.scope_id, message.message_id
+ )
+ source_record_id = clean_text(source_info.get("source_record_id"))
+ replayed_incoming = (
+ backend.leaf_for_source_assertion(
+ source_record_id, assertion_index
+ )
+ if source_record_id
+ else None
+ )
+ incoming_metadata = (
+ dict(replayed_incoming.get("metadata") or {})
+ if replayed_incoming is not None
+ else {}
+ )
+ frozen_slot = clean_text(
+ (frozen_selected or {}).get("canonical_slot_key")
+ )
+ linked_incoming_id = clean_text(
+ historical_metadata.get("superseded_by")
+ )
+ verified_partial_commit = bool(
+ frozen_selected is not None
+ and historical is not None
+ and replayed_incoming is not None
+ and clean_text(historical.get("record_state")) == "superseded"
+ and clean_text(historical_metadata.get("superseded_reason"))
+ == "v4_reconciliation_replace_current"
+ and _binding_identity(historical)
+ == _binding_identity(frozen_selected)
+ and clean_text(incoming_metadata.get("source_record_id"))
+ == source_record_id
+ and clean_text(incoming_metadata.get("message_id"))
+ == message.message_id
+ and int(incoming_metadata.get("llm_write_proposal_index", -1))
+ == assertion_index
+ and _normalized_claim(
+ clean_text(replayed_incoming.get("claim_text") or replayed_incoming.get("value"))
+ )
+ == _normalized_claim(clean_text(assertion.get("claim_text")))
+ and _normalized_evidence(
+ clean_text(replayed_incoming.get("evidence_quote"))
+ )
+ == _normalized_evidence(clean_text(assertion.get("evidence_quote")))
+ and clean_text(replayed_incoming.get("canonical_slot_key"))
+ == frozen_slot
+ and (
+ not linked_incoming_id
+ or linked_incoming_id
+ == clean_text(replayed_incoming.get("memory_id"))
+ )
+ )
+ if (
+ (
+ batch_metadata.get("validated_batch_commit_recovered")
+ is not True
+ and not verified_partial_commit
+ )
+ or frozen_selected is None
+ or historical is None
+ or clean_text(historical_metadata.get("superseded_reason"))
+ != "v4_reconciliation_replace_current"
+ or _binding_identity(historical) != _binding_identity(frozen_selected)
+ ):
+ raise ProductWriterError(
+ f"{job_id}: frozen Pro selection is absent from the current candidate set"
+ )
+ if verified_partial_commit:
+ replayed_incoming = backend.repair_partial_replacement(
+ selected_memory_id,
+ clean_text(replayed_incoming.get("memory_id")),
+ )
+ frozen_identity = _binding_identity(frozen_selected)
+ historical_identity = _binding_identity(historical)
+ if normalized.get("decision") == "replace_current":
+ resolved_binding = historical
+ binding_mode = (
+ "verified_partial_message_commit"
+ if verified_partial_commit
+ else "verified_historical_selected"
+ )
+ else:
+ semantic_identity = _binding_semantic_identity(frozen_selected)
+ equivalent_active = [
+ item
+ for item in backend.current_leaves(
+ clean_text(frozen_selected.get("canonical_slot_key"))
+ )
+ if _binding_semantic_identity(item) == semantic_identity
+ ]
+ if len(equivalent_active) != 1:
+ raise ProductWriterError(
+ f"{job_id}: frozen Pro selection lacks one unique active semantic equivalent"
+ )
+ resolved_binding = equivalent_active[0]
+ binding_mode = "unique_active_semantic_equivalent"
+ resolved_semantic_identity = _binding_semantic_identity(resolved_binding)
+ self._append_unique_jsonl(
+ "product_writer_historical_binding_recoveries.jsonl",
+ "job_id",
+ {
+ "schema_version": "tmcra.v4.historical-binding-recovery.1",
+ "job_id": job_id,
+ "batch_id": batch.batch_id,
+ "scope_id": batch.scope_id,
+ "message_id": message.message_id,
+ "selected_memory_id": selected_memory_id,
+ "resolved_memory_id": clean_text(
+ resolved_binding.get("memory_id")
+ ),
+ "replayed_incoming_memory_id": clean_text(
+ (replayed_incoming or {}).get("memory_id")
+ ),
+ "source_record_id": source_record_id,
+ "binding_mode": binding_mode,
+ "decision": normalized["decision"],
+ "historical_record_state": clean_text(
+ historical.get("record_state")
+ ),
+ "superseded_reason": "v4_reconciliation_replace_current",
+ "frozen_binding_identity_sha256": _hash_json(frozen_identity),
+ "historical_binding_identity_sha256": _hash_json(
+ historical_identity
+ ),
+ "frozen_semantic_identity_sha256": _hash_json(
+ _binding_semantic_identity(frozen_selected)
+ ),
+ "resolved_semantic_identity_sha256": _hash_json(
+ resolved_semantic_identity
+ ),
+ "physical_api_calls": 0,
+ "recovered_at": _now(),
+ },
+ )
+ self.stats["historical_binding_recoveries"] += 1
+ return normalized, resolved_binding
+
+ if job["status"] == "completed":
+ parsed = _strict_json_object(
+ str(job["response_json"]), f"reconciliation[{job_id}]"
+ )
+ return verify_current_binding(
+ self._validate_reconciliation_response(
+ parsed,
+ current_cited=frozen_candidates,
+ exact_slot_match=frozen_exact_slot_match,
+ path=f"reconciliation[{job_id}]",
+ )
+ )
+ if job["status"] == "failed":
+ raise ProductWriterError(
+ f"{job_id}: reconciliation has a failed external-call outcome; refusing retry"
+ )
+ if job["status"] == "pro_started":
+ job = self._recover_interrupted_reconciliation_call(batch, job_id)
+ if self.pro_client is None:
+ raise ProductWriterError(f"{job_id}: Pro client is required for candidate-slot adjudication")
+ self.store.start_reconciliation_job(job_id)
+ metadata: dict[str, Any] = {}
+ self.stats["pro_calls"] += 1
+ try:
+ result, metadata = _client_result(
+ self.pro_client.reconcile(frozen_request)
+ )
+ self._record_raw_api_response(
+ call_key=f"pro:{job_id}",
+ batch=batch,
+ stage="reconciliation_pro",
+ model="deepseek-v4-pro",
+ response=result,
+ metadata=metadata,
+ job_id=job_id,
+ )
+ parsed: Mapping[str, Any] | None = None
+ validation_error = ""
+ try:
+ parsed = (
+ result
+ if isinstance(result, Mapping)
+ else _strict_json_object(
+ str(result), f"reconciliation[{job_id}]"
+ )
+ )
+ adjudication = self._validate_reconciliation_response(
+ parsed,
+ current_cited=frozen_candidates,
+ exact_slot_match=frozen_exact_slot_match,
+ path=f"reconciliation[{job_id}]",
+ )
+ except ProductWriterError as exc:
+ validation_error = f"{exc.__class__.__name__}: {exc}"
+ adjudication = {
+ "slot_decision": "quarantine",
+ "selected_memory_id": "",
+ "decision": "quarantine",
+ }
+ selected_binding = None
+ self.stats["reconciliation_response_quarantines"] += 1
+ self._append_unique_jsonl(
+ "product_writer_reconciliation_quarantines.jsonl",
+ "job_id",
+ {
+ "schema_version": "tmcra.v4.reconciliation-quarantine.1",
+ "job_id": job_id,
+ "batch_id": batch.batch_id,
+ "scope_id": batch.scope_id,
+ "message_id": message.message_id,
+ "assertion_index": assertion_index,
+ "error": validation_error,
+ "raw_response_sha256": _hash_json(result),
+ "physical_api_calls": 1,
+ "quarantined_at": _now(),
+ },
+ )
+ else:
+ adjudication, selected_binding = verify_current_binding(adjudication)
+ model_adjudication = dict(parsed) if parsed is not None else {"raw_response": str(result)}
+ if model_adjudication != adjudication:
+ metadata = {
+ **metadata,
+ "controller_normalization": (
+ "invalid_pro_response_quarantined"
+ if validation_error
+ else "exact_slot_identity_and_parallel_action"
+ ),
+ "controller_validation_error": validation_error,
+ "model_adjudication_sha256": _hash_json(model_adjudication),
+ "normalized_adjudication_sha256": _hash_json(adjudication),
+ }
+ self.store.finish_reconciliation_job(
+ job_id, adjudication["decision"], adjudication, metadata
+ )
+ self._record_api_call(
+ call_key=f"pro:{job_id}",
+ batch=batch,
+ stage="reconciliation_pro",
+ model="deepseek-v4-pro",
+ metadata=metadata,
+ job_id=job_id,
+ )
+ except Exception as exc:
+ call_metadata = dict(getattr(exc, "metadata", None) or metadata)
+ error = f"{exc.__class__.__name__}: {exc}"
+ self.store.fail_reconciliation_job(job_id, error, call_metadata)
+ self._record_api_call(
+ call_key=f"pro:{job_id}",
+ batch=batch,
+ stage="reconciliation_pro",
+ model="deepseek-v4-pro",
+ metadata=call_metadata,
+ job_id=job_id,
+ error=error,
+ )
+ raise
+ return adjudication, selected_binding
+
+ @staticmethod
+ def _validate_reconciliation_response(
+ value: Mapping[str, Any],
+ *,
+ current_cited: Sequence[Mapping[str, Any]],
+ exact_slot_match: bool,
+ path: str,
+ ) -> dict[str, str]:
+ required = {"slot_decision", "selected_memory_id", "decision"}
+ missing = required - set(value)
+ if missing:
+ raise ProductWriterError(
+ f"{path} is missing required keys: {sorted(missing)}"
+ )
+ raw_slot_value = value.get("slot_decision")
+ if not isinstance(raw_slot_value, str) or not raw_slot_value.strip():
+ raise ProductWriterError(f"{path}.slot_decision must be a non-empty string")
+ raw_slot_decision = raw_slot_value.strip().lower().replace("-", "_").replace(" ", "_")
+ raw_selected_memory_id = value.get("selected_memory_id")
+ if not isinstance(raw_selected_memory_id, str):
+ raise ProductWriterError(f"{path}.selected_memory_id must be a string")
+ selected_memory_id = raw_selected_memory_id.strip()
+ raw_decision_value = value.get("decision")
+ if not isinstance(raw_decision_value, str) or not raw_decision_value.strip():
+ raise ProductWriterError(f"{path}.decision must be a non-empty string")
+ raw_decision = raw_decision_value
+ decision = re.sub(
+ r"_+",
+ "_",
+ raw_decision.strip().lower().replace("-", "_").replace(" ", "_"),
+ )
+ if decision not in DECISIONS:
+ raise ProductWriterError(
+ f"{path}.decision is unsupported: {raw_decision!r}"
+ )
+ candidates = [item for item in current_cited if clean_text(item.get("memory_id"))]
+ candidate_ids = {clean_text(item.get("memory_id")) for item in candidates}
+ if raw_slot_decision in SLOT_DECISIONS:
+ slot_decision = raw_slot_decision
+ elif (
+ raw_slot_decision == decision
+ and decision in {"merge_support", "replace_current", "keep_parallel", "challenge"}
+ and selected_memory_id in candidate_ids
+ ):
+ # Pro occasionally places the conflict action in both enum fields.
+ # A supplied valid candidate ID makes the intended slot binding
+ # unambiguous; all other out-of-schema values remain hard failures.
+ slot_decision = "bind_existing"
+ else:
+ raise ProductWriterError(
+ f"{path}.slot_decision is unsupported: {raw_slot_decision!r}"
+ )
+ if exact_slot_match and slot_decision != "quarantine":
+ if not candidates:
+ raise ProductWriterError(f"{path}: exact slot collision lacks candidates")
+ preferred = min(
+ candidates,
+ key=lambda item: (
+ {"active": 0, "promoted": 1, "parallel_active": 2}.get(
+ clean_text(item.get("record_state")), 3
+ ),
+ -int(item.get("turn_index") or 0),
+ clean_text(item.get("memory_id")),
+ ),
+ )
+ slot_decision = "bind_existing"
+ selected_memory_id = clean_text(preferred.get("memory_id"))
+ if decision == "insert":
+ decision = "keep_parallel"
+ if slot_decision == "bind_existing":
+ if selected_memory_id not in candidate_ids:
+ raise ProductWriterError(
+ f"{path}.selected_memory_id is not a supplied candidate"
+ )
+ if decision == "insert":
+ raise ProductWriterError(
+ f"{path}: a bound existing slot cannot use insert"
+ )
+ elif slot_decision == "keep_proposed":
+ if selected_memory_id or decision != "insert":
+ raise ProductWriterError(
+ f"{path}: keep_proposed requires empty selected_memory_id and insert"
+ )
+ else:
+ if selected_memory_id or decision != "quarantine":
+ raise ProductWriterError(
+ f"{path}: quarantine requires empty selected_memory_id and quarantine"
+ )
+ return {
+ "slot_decision": slot_decision,
+ "selected_memory_id": selected_memory_id,
+ "decision": decision,
+ }
+
+ def _commit_message(
+ self,
+ batch: SourceBatch,
+ message_index: int,
+ response_message: Mapping[str, Any],
+ ) -> int:
+ source = batch.messages[message_index]
+ journal = self.store.prepare_message_commit(
+ batch, source, response_message
+ )
+ commit_id = str(journal["commit_id"])
+ if journal["status"] == "committed":
+ return int(journal["semantic_committed"])
+ v3 = dict(response_message["v3"])
+ durability = list(response_message["durability"])
+ if self.graph_factory is None:
+ raise ProductWriterError("real graph backend is required for V4 semantic commit")
+ backend = self.graph_factory.for_scope(batch.scope_id)
+ source_info = self.store.source_info(batch.scope_id, source.message_id)
+ source_record_id = clean_text(source_info.get("source_record_id"))
+ if not source_record_id:
+ raise ProductWriterError(f"{source.message_id}: real source record ID is missing")
+ source_turn_index = int(source_info.get("source_turn_index") or 0)
+ if clean_text(journal["plan_sha256"]):
+ return self._execute_message_commit_plan(
+ backend=backend,
+ batch=batch,
+ source=source,
+ source_record_id=source_record_id,
+ plan=json.loads(str(journal["plan_json"])),
+ response_message=response_message,
+ )
+ assertions = [dict(item) for item in v3.get("assertions") or []]
+ decisions: dict[int, str] = {}
+ current_by_index: dict[int, Sequence[Mapping[str, Any]]] = {}
+ duplicate_provenance: list[dict[str, Any]] = []
+
+ def add_duplicate_provenance(assertion: Mapping[str, Any], leaf: Mapping[str, Any]) -> None:
+ duplicate_provenance.append(
+ {
+ "leaf_id": leaf["memory_id"],
+ "provenance": {
+ "batch_id": batch.batch_id,
+ "message_id": source.message_id,
+ "evidence_span_id": assertion["evidence_span_id"],
+ "evidence_quote": assertion["evidence_quote"],
+ },
+ },
+ )
+
+ for assertion_index, assertion in enumerate(assertions):
+ exact_current = backend.current_leaves(str(assertion["canonical_key"]))
+ new_claim = _normalized_claim(str(assertion["claim_text"]))
+ persisted_job_id = _reconciliation_job_id(
+ batch, source, assertion_index, assertion
+ )
+ persisted_job = self.store.reconciliation_job(persisted_job_id)
+ exact_duplicate = [
+ leaf
+ for leaf in exact_current
+ if _normalized_claim(str(leaf["value"])) == new_claim
+ ]
+ if exact_duplicate and persisted_job is None:
+ add_duplicate_provenance(assertion, exact_duplicate[0])
+ decisions[assertion_index] = "duplicate"
+ current_by_index[assertion_index] = exact_current
+ continue
+
+ candidates = list(
+ exact_current or backend.candidate_leaves(assertion, limit=3)
+ )
+ if persisted_job is not None:
+ frozen_request = json.loads(str(persisted_job["request_json"]))
+ for frozen in list(
+ frozen_request.get("candidate_cited_leaves") or []
+ ):
+ if not isinstance(frozen, Mapping):
+ continue
+ frozen_memory_id = clean_text(frozen.get("memory_id"))
+ if not frozen_memory_id or any(
+ clean_text(item.get("memory_id")) == frozen_memory_id
+ for item in candidates
+ ):
+ continue
+ leaf = backend.leaf_by_id(frozen_memory_id)
+ if (
+ leaf is not None
+ and clean_text(leaf.get("record_state"))
+ in {"active", "parallel_active", "promoted"}
+ and _binding_identity(leaf) == _binding_identity(frozen)
+ ):
+ candidates.append(leaf)
+ if not candidates and persisted_job is None:
+ decisions[assertion_index] = "insert"
+ current_by_index[assertion_index] = []
+ continue
+ adjudication, selected_binding = self._reconcile(
+ batch,
+ source,
+ assertion_index,
+ assertion,
+ durability[assertion_index],
+ candidates,
+ exact_slot_match=bool(exact_current),
+ backend=backend,
+ )
+ slot_decision = adjudication["slot_decision"]
+ if slot_decision == "bind_existing":
+ if selected_binding is None:
+ raise ProductWriterError(
+ f"{batch.batch_id}: bound reconciliation lacks a selected leaf"
+ )
+ selected = selected_binding
+ assertion = self._bind_assertion_to_existing(assertion, selected)
+ assertions[assertion_index] = assertion
+ current = backend.current_leaves(str(assertion["canonical_key"]))
+ if (
+ adjudication["decision"] == "replace_current"
+ and not any(
+ clean_text(item.get("memory_id"))
+ == clean_text(selected.get("memory_id"))
+ for item in current
+ )
+ ):
+ current = [selected, *current]
+ bound_duplicate = [
+ leaf
+ for leaf in current
+ if _normalized_claim(str(leaf["value"])) == new_claim
+ ]
+ if bound_duplicate:
+ add_duplicate_provenance(assertion, bound_duplicate[0])
+ decisions[assertion_index] = "duplicate"
+ current_by_index[assertion_index] = current
+ continue
+ if adjudication["decision"] == "merge_support":
+ add_duplicate_provenance(assertion, selected)
+ decisions[assertion_index] = "duplicate"
+ current_by_index[assertion_index] = current
+ continue
+ decisions[assertion_index] = adjudication["decision"]
+ current_by_index[assertion_index] = current
+ elif slot_decision == "keep_proposed":
+ decisions[assertion_index] = "insert"
+ current_by_index[assertion_index] = []
+ else:
+ decisions[assertion_index] = "quarantine"
+ current_by_index[assertion_index] = candidates
+ committed_assertions: list[Mapping[str, Any]] = []
+ committed_durabilities: list[str] = []
+ committed_decisions: dict[int, str] = {}
+ committed_current: dict[int, Sequence[Mapping[str, Any]]] = {}
+ for original_index, assertion in enumerate(assertions):
+ decision = decisions.get(original_index, "insert")
+ if decision == "duplicate":
+ continue
+ committed_index = len(committed_assertions)
+ committed_assertions.append(assertion)
+ committed_durabilities.append(durability[original_index])
+ committed_decisions[committed_index] = decision
+ committed_current[committed_index] = current_by_index.get(original_index, [])
+ committed_v3 = dict(v3)
+ committed_v3["assertions"] = committed_assertions
+ interactions = list(v3.get("interactions") or [])
+ resolutions = list(v3.get("resolutions") or [])
+ plan = {
+ "schema_version": "tmcra.v4.message-commit-plan.1",
+ "batch_id": batch.batch_id,
+ "message_id": source.message_id,
+ "source_record_id": source_record_id,
+ "source_turn_index": source_turn_index,
+ "extraction": committed_v3,
+ "durabilities": committed_durabilities,
+ "decisions": committed_decisions,
+ "current_by_index": committed_current,
+ "duplicate_provenance": duplicate_provenance,
+ "interactions": interactions,
+ "resolutions": resolutions,
+ }
+ self.store.freeze_message_commit_plan(commit_id, plan)
+
+ return self._execute_message_commit_plan(
+ backend=backend,
+ batch=batch,
+ source=source,
+ source_record_id=source_record_id,
+ plan=plan,
+ response_message=response_message,
+ )
+
+ def _execute_message_commit_plan(
+ self,
+ *,
+ backend: Any,
+ batch: SourceBatch,
+ source: SourceMessage,
+ source_record_id: str,
+ plan: Mapping[str, Any],
+ response_message: Mapping[str, Any],
+ ) -> int:
+ commit_id = self.store._message_commit_id(batch, source)
+ if (
+ clean_text(plan.get("batch_id")) != batch.batch_id
+ or clean_text(plan.get("message_id")) != source.message_id
+ or clean_text(plan.get("source_record_id")) != source_record_id
+ ):
+ raise ProductWriterError(
+ f"{commit_id}: frozen message commit plan identity changed"
+ )
+ extraction = dict(plan.get("extraction") or {})
+ durabilities = list(plan.get("durabilities") or [])
+ decisions = {
+ int(key): str(value)
+ for key, value in dict(plan.get("decisions") or {}).items()
+ }
+ current_by_index = {
+ int(key): list(value or [])
+ for key, value in dict(plan.get("current_by_index") or {}).items()
+ }
+ duplicate_provenance = [
+ dict(item) for item in list(plan.get("duplicate_provenance") or [])
+ ]
+ interactions = [dict(item) for item in list(plan.get("interactions") or [])]
+ resolutions = [dict(item) for item in list(plan.get("resolutions") or [])]
+
+ def finalize(
+ connection: sqlite3.Connection, semantic_committed: int
+ ) -> None:
+ self.store.finalize_message_commit(
+ connection,
+ commit_id=commit_id,
+ batch=batch,
+ message=source,
+ source_record_id=source_record_id,
+ interactions=interactions,
+ resolutions=resolutions,
+ semantic_committed=semantic_committed,
+ )
+
+ atomic_commit = bool(
+ getattr(backend, "supports_atomic_message_commit", False)
+ )
+ try:
+ if not atomic_commit:
+ for item in duplicate_provenance:
+ backend.add_provenance(
+ str(item["leaf_id"]),
+ source_record_id=source_record_id,
+ source_turn_index=int(plan["source_turn_index"]),
+ provenance=dict(item.get("provenance") or {}),
+ )
+ kwargs = {
+ "message": source,
+ "source_record_id": source_record_id,
+ "source_turn_index": int(plan["source_turn_index"]),
+ "extraction": extraction,
+ "durabilities": durabilities,
+ "decisions": decisions,
+ "current_by_index": current_by_index,
+ }
+ if atomic_commit:
+ kwargs.update(
+ {
+ "duplicate_provenance": duplicate_provenance,
+ "transaction_hook": finalize,
+ }
+ )
+ committed_count = backend.commit_message(**kwargs)
+ committed_row = self.store.prepare_message_commit(
+ batch, source, response_message
+ )
+ if committed_row["status"] != "committed":
+ with closing(self.store._connect()) as connection, connection:
+ finalize(connection, committed_count)
+ except Exception as exc:
+ self.store.record_message_commit_error(
+ commit_id, f"{exc.__class__.__name__}: {exc}"
+ )
+ raise
+ self.stats["fast_assertion_leaves"] += committed_count
+ return committed_count
+
+ @staticmethod
+ def _bind_assertion_to_existing(
+ assertion: Mapping[str, Any], leaf: Mapping[str, Any]
+ ) -> dict[str, Any]:
+ metadata = dict(leaf.get("metadata") or {})
+ canonical_slot_key = clean_text(
+ leaf.get("canonical_slot_key") or metadata.get("canonical_slot_key")
+ )
+ if not canonical_slot_key:
+ raise ProductWriterError("selected binding candidate lacks canonical slot")
+ bound = dict(assertion)
+ bound["canonical_key"] = canonical_slot_key.removeprefix("memory.")
+ for key in (
+ "entity_key",
+ "graph_entity_key",
+ "attribute_key",
+ "memory_type",
+ "memory_family",
+ ):
+ value = clean_text(metadata.get(key))
+ if value:
+ bound[key] = value
+ bound["operation"] = "replace"
+ return bound
+
+ def run(self, rows: Sequence[Mapping[str, Any]]) -> dict[str, Any]:
+ messages, exclusions = normalize_source_inventory(rows)
+ self.stats["input_messages"] = len(messages) + len(exclusions)
+ self.stats["source_messages"] = len(messages)
+ self.stats["excluded_empty_source_messages"] = len(exclusions)
+ if self.log_dir is not None:
+ payload = {
+ "schema_version": "tmcra.v4.source-exclusions.1",
+ "reason_policy": "exclude_only_whitespace_empty_message_carriers",
+ "count": len(exclusions),
+ "messages": exclusions,
+ }
+ path = self.log_dir / "source_exclusions.json"
+ temporary = path.with_name(f".{path.name}.tmp.{os.getpid()}.{time.time_ns()}")
+ temporary.write_text(
+ json.dumps(payload, ensure_ascii=False, indent=2, sort_keys=True) + "\n",
+ encoding="utf-8",
+ )
+ temporary.replace(path)
+ batches = build_batches(messages)
+ self.stats["batches"] = len(batches)
+ for batch in batches:
+ existing_batch = self.store.batch_row(batch.batch_id)
+ if existing_batch is None:
+ unresolved = self.store.unresolved_interactions(
+ batch.scope_id, batch.session_id
+ )
+ request = build_batch_request(batch, unresolved)
+ else:
+ request = json.loads(str(existing_batch["request_json"]))
+ if (
+ request.get("schema_version") != BATCH_SCHEMA_VERSION
+ or request.get("batch_id") != batch.batch_id
+ ):
+ raise ProductWriterError(
+ f"{batch.batch_id}: persisted batch request schema or ID changed"
+ )
+ unresolved = list(request.get("unresolved_interactions") or [])
+ row = self.store.prepare(batch, request)
+ if row["status"] == "committed":
+ backend = self._ensure_graph_sources(batch, verify_only=True)
+ source_record_ids = [
+ clean_text(
+ self.store.source_info(
+ batch.scope_id, message.message_id
+ ).get("source_record_id")
+ )
+ for message in batch.messages
+ ]
+ statuses = backend.source_enrichment_statuses(source_record_ids)
+ repair_ids = [
+ source_record_id
+ for source_record_id in source_record_ids
+ if statuses.get(source_record_id) != "enriched"
+ ]
+ if repair_ids:
+ self._append_unique_jsonl(
+ "product_writer_committed_source_repairs.jsonl",
+ "batch_id",
+ {
+ "schema_version": "tmcra.v4.committed-source-repair.1",
+ "batch_id": batch.batch_id,
+ "scope_id": batch.scope_id,
+ "source_record_ids": repair_ids,
+ "prior_enrichment_statuses": {
+ source_record_id: statuses.get(source_record_id, "")
+ for source_record_id in repair_ids
+ },
+ "physical_api_calls": 0,
+ "repaired_at": _now(),
+ },
+ )
+ self.store.mark_source_enriched(batch)
+ for source_record_id in repair_ids:
+ backend.set_enrichment_status(source_record_id, "enriched")
+ self.stats["committed_source_status_repairs"] += len(
+ repair_ids
+ )
+ self.stats["resumed_batches"] += 1
+ continue
+ if row["status"] == "api_started":
+ row = self._recover_interrupted_batch_call(batch)
+ if row["status"] == "failed":
+ if not self.revalidate_failed_raw_response:
+ raise ProductWriterError(f"{batch.batch_id}: prior batch failed; refusing retry")
+ row = self._revalidate_failed_batch(batch, row, unresolved)
+ try:
+ self._ensure_graph_sources(
+ batch, verify_only=row["status"] == "validated"
+ )
+ except Exception as exc:
+ error = f"{exc.__class__.__name__}: {exc}"
+ self.store.mark_source_enrichment_failed(batch, error)
+ self.store.fail_batch(batch.batch_id, error)
+ raise
+ if row["status"] == "validated":
+ validated = json.loads(row["response_json"])
+ self.stats["resumed_batches"] += 1
+ elif not any(message.role in {"user", "assistant"} for message in batch.messages):
+ # Immutable-only batches are journaled and committed without a semantic API call.
+ validated = {"schema_version": BATCH_SCHEMA_VERSION, "batch_id": batch.batch_id, "messages": []}
+ self.store.persist_response(batch.batch_id, validated, {"api_call_count": 0, "reason": "immutable_only_batch"})
+ else:
+ self.store.mark_api_started(batch.batch_id)
+ metadata: dict[str, Any] = {}
+ self.stats["flash_calls"] += 1
+ try:
+ result, metadata = _client_result(self.flash_client.complete(request))
+ self._record_raw_api_response(
+ call_key=f"flash:{batch.batch_id}",
+ batch=batch,
+ stage="batch_flash",
+ model="deepseek-v4-flash",
+ response=result,
+ metadata=metadata,
+ )
+ raw_payload = result if isinstance(result, Mapping) else _strict_json_object(str(result), f"batch[{batch.batch_id}]")
+ validated = validate_batch_response(raw_payload, batch, unresolved)
+ self.store.persist_response(batch.batch_id, validated, metadata)
+ self._record_api_call(
+ call_key=f"flash:{batch.batch_id}",
+ batch=batch,
+ stage="batch_flash",
+ model="deepseek-v4-flash",
+ metadata={
+ **metadata,
+ "request_content_sha256": _hash_json(request),
+ "validated_response_sha256": _hash_json(validated),
+ },
+ )
+ except Exception as exc:
+ error = f"{exc.__class__.__name__}: {exc}"
+ call_metadata = dict(getattr(exc, "metadata", None) or metadata)
+ self._record_api_call(
+ call_key=f"flash:{batch.batch_id}",
+ batch=batch,
+ stage="batch_flash",
+ model="deepseek-v4-flash",
+ metadata=call_metadata,
+ error=error,
+ )
+ self.store.mark_source_enrichment_failed(batch, error)
+ self._set_graph_source_status(batch, "failed", error)
+ self.store.fail_batch(batch.batch_id, error, call_metadata)
+ raise
+ try:
+ source_indexes = {
+ message.message_id: index for index, message in enumerate(batch.messages)
+ }
+ for index, response_message in enumerate(validated["messages"]):
+ source_index = source_indexes[response_message["message_id"]]
+ committed_count = self._commit_message(
+ batch, source_index, response_message
+ )
+ output = response_message["v3"]
+ validation_warnings = list(output.get("validation_warnings") or [])
+ self.stats["validation_warnings"] += len(validation_warnings)
+ self._append_unique_jsonl(
+ "product_write_messages.jsonl",
+ "message_key",
+ {
+ "message_key": f"{batch.scope_id}:{response_message['message_id']}",
+ "batch_id": batch.batch_id,
+ "scope_id": batch.scope_id,
+ "message_id": response_message["message_id"],
+ "message_role": response_message["message_role"],
+ "content_sha256": sha256_text(batch.messages[source_index].content),
+ "source": 1,
+ "semantic_proposals": len(output.get("assertions") or []),
+ "semantic_committed": committed_count,
+ "facet": sum(len(item.get("facets") or []) for item in output.get("assertions") or []),
+ "interaction": len(output.get("interactions") or []),
+ "resolution_count": len(output.get("resolutions") or []),
+ "validation_warning_count": len(validation_warnings),
+ "validation_warnings": validation_warnings,
+ "writer_called": True,
+ },
+ )
+ self.store.commit_batch(batch.batch_id)
+ if not validated["messages"]:
+ self.store.mark_source_enriched(batch)
+ self._set_graph_source_status(batch, "enriched")
+ elif not bool(
+ getattr(
+ self.graph_factory.for_scope(batch.scope_id),
+ "supports_atomic_message_commit",
+ False,
+ )
+ ):
+ # Test/legacy backends cannot join the SQLite transaction.
+ self._set_graph_source_status(batch, "enriched")
+ except Exception as exc:
+ error = f"{exc.__class__.__name__}: {exc}"
+ # The validated response remains replayable. Message journals
+ # identify exactly which graph commits completed, so a local
+ # commit failure must not downgrade the whole batch or already
+ # committed source messages.
+ self.store.record_batch_commit_error(batch.batch_id, error)
+ raise
+ return dict(self.stats)
+
+
+def _build_cli_client(*, reviewer_model: str, timeout: float, max_tokens: int) -> tuple[DeepSeekBatchClient, DeepSeekBatchClient]:
+ base_url = clean_text(os.getenv("TMCRA_WRITER_BASE_URL"))
+ model = clean_text(os.getenv("TMCRA_WRITER_MODEL"))
+ keys = [clean_text(value) for value in os.getenv("TMCRA_WRITER_API_KEY_POOL", "").split(",") if clean_text(value)]
+ if not base_url or not model or not reviewer_model or not keys:
+ raise ProductWriterError("explicit writer base URL, Flash model, Pro model, and API key pool are required")
+ if model != "deepseek-v4-flash" or reviewer_model != "deepseek-v4-pro":
+ raise ProductWriterError("V4 requires deepseek-v4-flash and deepseek-v4-pro")
+ return (
+ DeepSeekBatchClient(base_url=base_url, model=model, api_keys=keys, timeout=timeout, max_tokens=max_tokens),
+ DeepSeekBatchClient(base_url=base_url, model=reviewer_model, api_keys=keys, timeout=timeout, max_tokens=max_tokens),
+ )
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description="TMCRA V4 consecutive-session batch writer")
+ parser.add_argument("--input", required=True)
+ parser.add_argument("--out-dir", required=True)
+ parser.add_argument("--repo", required=True)
+ parser.add_argument("--reviewer-model", default="deepseek-v4-pro")
+ parser.add_argument("--timeout-seconds", type=float, default=180.0)
+ parser.add_argument("--max-tokens", type=int, default=8192)
+ parser.add_argument(
+ "--revalidate-failed-raw-response",
+ action="store_true",
+ help="revalidate one clean, hashed failed response without another API call",
+ )
+ parser.add_argument(
+ "--recover-interrupted-api-calls",
+ action="store_true",
+ help=(
+ "after explicit process-loss review, replace started calls that have "
+ "no durable response or call artifact using the same model"
+ ),
+ )
+ args = parser.parse_args()
+ repo = Path(args.repo).resolve()
+ if str(repo) not in sys.path:
+ sys.path.insert(0, str(repo))
+ input_path = Path(args.input).resolve()
+ out_dir = Path(args.out_dir).resolve()
+ out_dir.mkdir(parents=True, exist_ok=True)
+ rows = json.loads(input_path.read_text(encoding="utf-8"))
+ if not isinstance(rows, list) or not rows:
+ raise ProductWriterError("writer input must be a non-empty JSON array")
+ flash, pro = _build_cli_client(reviewer_model=args.reviewer_model, timeout=args.timeout_seconds, max_tokens=args.max_tokens)
+ database = out_dir / "native_memory.sqlite3"
+ writer = V4BatchWriter(
+ store=V4BatchStore(database),
+ flash_client=flash,
+ pro_client=pro,
+ graph_factory=RealGraphFactory(repo=repo, database=database),
+ log_dir=out_dir,
+ revalidate_failed_raw_response=args.revalidate_failed_raw_response,
+ recover_interrupted_api_calls=args.recover_interrupted_api_calls,
+ )
+ report = writer.run(rows)
+ report.update({"schema_version": "tmcra.v4.batch-writer-run.1", "writer_schema_version": BATCH_SCHEMA_VERSION, "prompt_version": PROMPT_VERSION, "candidate_selector_version": CANDIDATE_SELECTOR_VERSION, "completed": True, "db_path": str(out_dir / "native_memory.sqlite3")})
+ (out_dir / "product_writer_report.json").write_text(_json(report) + "\n", encoding="utf-8")
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_cost_report.py b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_cost_report.py
new file mode 100644
index 0000000000000000000000000000000000000000..32c73432e876e5fbf20afac667c853a780d6008d
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_cost_report.py
@@ -0,0 +1,562 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import sqlite3
+from collections import defaultdict
+from contextlib import closing
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Iterable, Mapping, Sequence
+
+
+MILLION = 1_000_000
+
+
+@dataclass(frozen=True)
+class ModelRate:
+ cache_hit_input_cny_per_million: float
+ cache_miss_input_cny_per_million: float
+ output_cny_per_million: float
+
+
+OFFICIAL_DEEPSEEK_RATES = {
+ "deepseek-v4-flash": ModelRate(0.02, 1.0, 2.0),
+ "deepseek-v4-pro": ModelRate(0.025, 3.0, 6.0),
+}
+
+USAGE_KEYS = {
+ "prompt_tokens",
+ "completion_tokens",
+ "prompt_cache_hit_tokens",
+ "prompt_cache_miss_tokens",
+ "cached_tokens",
+}
+PHYSICAL_LIST_KEYS = (
+ "requests",
+ "physical_requests",
+ "api_attempts",
+ "tier_calls",
+)
+SQLITE_CALL_TABLES = {
+ "slow_graph_archived_attempts",
+ "slow_graph_attempts",
+ "v4_slow_graph_attempts",
+ "v4_reconciliation_attempts",
+ "v4_batch_journal",
+ "v4_reconciliation_jobs",
+ "v4_subject_attribution_audits",
+}
+SLOW_INTERRUPTION_ERROR = (
+ "claim lease expired; external call outcome uncertain; explicit resume required"
+)
+
+
+class CostReportError(RuntimeError):
+ pass
+
+
+def _text(value: Any) -> str:
+ return str(value or "").strip()
+
+
+def _integer(value: Any, *, label: str) -> int:
+ if isinstance(value, bool):
+ raise CostReportError(f"{label} must be an integer")
+ try:
+ output = int(value or 0)
+ except (TypeError, ValueError) as exc:
+ raise CostReportError(f"{label} must be an integer") from exc
+ if output < 0:
+ raise CostReportError(f"{label} cannot be negative")
+ return output
+
+
+def _model(record: Mapping[str, Any], inherited: str) -> str:
+ direct = _text(record.get("model"))
+ if direct:
+ return direct
+ request = record.get("request")
+ if isinstance(request, Mapping):
+ requested = _text(request.get("model"))
+ if requested:
+ return requested
+ model_config = record.get("model_config")
+ if isinstance(model_config, Mapping):
+ configured = _text(model_config.get("model"))
+ if configured:
+ return configured
+ return inherited
+
+
+def _stage(record: Mapping[str, Any], inherited: str) -> str:
+ for key in ("tier_stage", "stage", "route", "routing_reason", "call_kind"):
+ value = _text(record.get(key))
+ if value:
+ return value
+ if _text(record.get("planner_version")):
+ return "recall_planner"
+ return inherited or "unknown"
+
+
+def _usage(record: Mapping[str, Any]) -> Mapping[str, Any] | None:
+ value = record.get("usage")
+ if isinstance(value, Mapping) and USAGE_KEYS.intersection(value):
+ return value
+ if USAGE_KEYS.intersection(record):
+ return record
+ return None
+
+
+def _call_id(record: Mapping[str, Any], *, source: str, path: str) -> str:
+ for key in ("physical_call_id", "response_id", "request_id", "id"):
+ value = _text(record.get(key))
+ if value:
+ return f"{key}:{value}"
+ material = {
+ "source": source,
+ "path": path,
+ "request_sha256": _text(record.get("request_sha256")),
+ "started_at": record.get("started_at"),
+ "completed_at": record.get("completed_at"),
+ }
+ return "derived:" + hashlib.sha256(
+ json.dumps(material, sort_keys=True, separators=(",", ":")).encode("utf-8")
+ ).hexdigest()
+
+
+def extract_physical_calls(
+ value: Any,
+ *,
+ source: str,
+ path: str = "root",
+ inherited_model: str = "",
+ inherited_stage: str = "",
+) -> list[dict[str, Any]]:
+ if isinstance(value, list):
+ output: list[dict[str, Any]] = []
+ for index, item in enumerate(value):
+ output.extend(
+ extract_physical_calls(
+ item,
+ source=source,
+ path=f"{path}[{index}]",
+ inherited_model=inherited_model,
+ inherited_stage=inherited_stage,
+ )
+ )
+ return output
+ if not isinstance(value, Mapping):
+ return []
+
+ model = _model(value, inherited_model)
+ stage = _stage(value, inherited_stage)
+ for key in PHYSICAL_LIST_KEYS:
+ children = value.get(key)
+ if isinstance(children, list) and children:
+ return extract_physical_calls(
+ children,
+ source=source,
+ path=f"{path}.{key}",
+ inherited_model=model,
+ inherited_stage=stage,
+ )
+
+ explicitly_nonphysical = (
+ value.get("physical_api_call") is False
+ and int(value.get("physical_api_calls", 0) or 0) == 0
+ and not _text(value.get("physical_call_id"))
+ )
+ if explicitly_nonphysical:
+ return []
+
+ usage = _usage(value)
+ if usage is not None:
+ return [
+ {
+ "call_id": _call_id(value, source=source, path=path),
+ "source": source,
+ "path": path,
+ "model": model or "unknown",
+ "stage": stage,
+ "status": _text(value.get("status")) or "usage_recorded",
+ "usage": dict(usage),
+ "usage_recorded": True,
+ "external_call_outcome_unknown": bool(
+ value.get("external_call_outcome_unknown")
+ ),
+ }
+ ]
+
+ physical_without_usage = bool(value.get("physical_api_call")) or bool(
+ _text(value.get("physical_call_id"))
+ )
+ if physical_without_usage:
+ return [
+ {
+ "call_id": _call_id(value, source=source, path=path),
+ "source": source,
+ "path": path,
+ "model": model or "unknown",
+ "stage": stage,
+ "status": _text(value.get("status")) or "usage_missing",
+ "usage": {},
+ "usage_recorded": False,
+ "external_call_outcome_unknown": bool(
+ value.get("external_call_outcome_unknown")
+ ),
+ }
+ ]
+
+ output = []
+ for key, child in value.items():
+ if key in {"request", "model_config"}:
+ continue
+ if isinstance(child, (Mapping, list)):
+ output.extend(
+ extract_physical_calls(
+ child,
+ source=source,
+ path=f"{path}.{key}",
+ inherited_model=model,
+ inherited_stage=stage,
+ )
+ )
+ return output
+
+
+def read_json_records(path: Path) -> list[Any]:
+ if path.suffix.lower() == ".jsonl":
+ values = []
+ with path.open("r", encoding="utf-8", errors="strict") as handle:
+ for line_number, line in enumerate(handle, start=1):
+ if not line.strip():
+ continue
+ try:
+ values.append(json.loads(line))
+ except json.JSONDecodeError as exc:
+ raise CostReportError(f"invalid JSON at {path}:{line_number}") from exc
+ return values
+ try:
+ return [json.loads(path.read_text(encoding="utf-8"))]
+ except json.JSONDecodeError as exc:
+ raise CostReportError(f"invalid JSON: {path}") from exc
+
+
+def sqlite_call_metadata(path: Path) -> list[tuple[str, Any]]:
+ output: list[tuple[str, Any]] = []
+ with closing(sqlite3.connect(path)) as connection:
+ tables = {
+ str(row[0])
+ for row in connection.execute(
+ "SELECT name FROM sqlite_master WHERE type='table'"
+ )
+ }
+ for table in sorted(tables):
+ if table not in SQLITE_CALL_TABLES:
+ continue
+ columns = {
+ str(row[1])
+ for row in connection.execute(f'PRAGMA table_info("{table}")')
+ }
+ for column in (
+ "call_metadata_json",
+ "api_metadata_json",
+ "response_metadata_json",
+ ):
+ if column not in columns:
+ continue
+ query = f'SELECT rowid,"{column}" FROM "{table}" WHERE "{column}" IS NOT NULL AND "{column}" != \'\''
+ for rowid, raw in connection.execute(query):
+ try:
+ output.append((f"{table}:{rowid}:{column}", json.loads(raw)))
+ except json.JSONDecodeError as exc:
+ raise CostReportError(
+ f"invalid call metadata JSON in {path}:{table}:{rowid}"
+ ) from exc
+ if table in {
+ "slow_graph_archived_attempts",
+ "slow_graph_attempts",
+ "v4_slow_graph_attempts",
+ } and {
+ "attempt_id",
+ "status",
+ "call_metadata_json",
+ "error",
+ }.issubset(columns):
+ for rowid, attempt_id in connection.execute(
+ f'SELECT rowid,"attempt_id" FROM "{table}" '
+ "WHERE status='expired' AND error=? "
+ "AND (call_metadata_json='{}' OR call_metadata_json='')",
+ (SLOW_INTERRUPTION_ERROR,),
+ ):
+ output.append(
+ (
+ f"{table}:{rowid}:interrupted_external_call",
+ {
+ "physical_api_call": True,
+ "physical_call_id": f"slow-interrupted:{attempt_id}",
+ "model": "unknown",
+ "stage": "slow_graph_interrupted",
+ "status": "external_call_outcome_unknown",
+ "external_call_outcome_unknown": True,
+ },
+ )
+ )
+ return output
+
+
+def normalize_usage(call: Mapping[str, Any]) -> dict[str, Any]:
+ usage = dict(call["usage"])
+ prompt = _integer(usage.get("prompt_tokens"), label="prompt_tokens")
+ completion = _integer(
+ usage.get("completion_tokens"), label="completion_tokens"
+ )
+ hit_value = usage.get(
+ "prompt_cache_hit_tokens",
+ usage.get("cache_read_input_tokens", usage.get("cached_tokens")),
+ )
+ miss_value = usage.get(
+ "prompt_cache_miss_tokens", usage.get("cache_miss_input_tokens")
+ )
+ has_hit = hit_value is not None
+ has_miss = miss_value is not None
+ hit = _integer(hit_value, label="prompt_cache_hit_tokens") if has_hit else 0
+ miss = _integer(miss_value, label="prompt_cache_miss_tokens") if has_miss else 0
+ if has_hit and has_miss and hit + miss != prompt:
+ raise CostReportError(
+ f"{call['call_id']}: cache hit+miss tokens do not equal prompt_tokens"
+ )
+ if has_hit and not has_miss:
+ if hit > prompt:
+ raise CostReportError(f"{call['call_id']}: cache hit tokens exceed prompt")
+ miss = prompt - hit
+ has_miss = True
+ if has_miss and not has_hit:
+ if miss > prompt:
+ raise CostReportError(f"{call['call_id']}: cache miss tokens exceed prompt")
+ hit = prompt - miss
+ has_hit = True
+ return {
+ "prompt_tokens": prompt,
+ "completion_tokens": completion,
+ "cache_hit_tokens": hit,
+ "cache_miss_tokens": miss,
+ "cache_breakdown_exact": has_hit and has_miss,
+ "usage_recorded": bool(call.get("usage_recorded", True)),
+ }
+
+
+def price_usage(model: str, usage: Mapping[str, Any]) -> dict[str, Any]:
+ rate = OFFICIAL_DEEPSEEK_RATES.get(model)
+ if rate is None:
+ return {
+ "priced": False,
+ "exact_cost_cny": None,
+ "min_cost_cny": None,
+ "max_cost_cny": None,
+ }
+ output_cost = (
+ int(usage["completion_tokens"]) * rate.output_cny_per_million / MILLION
+ )
+ if bool(usage["cache_breakdown_exact"]):
+ input_cost = (
+ int(usage["cache_hit_tokens"])
+ * rate.cache_hit_input_cny_per_million
+ + int(usage["cache_miss_tokens"])
+ * rate.cache_miss_input_cny_per_million
+ ) / MILLION
+ exact = input_cost + output_cost
+ return {
+ "priced": True,
+ "exact_cost_cny": exact,
+ "min_cost_cny": exact,
+ "max_cost_cny": exact,
+ }
+ prompt = int(usage["prompt_tokens"])
+ return {
+ "priced": True,
+ "exact_cost_cny": None,
+ "min_cost_cny": output_cost
+ + prompt * rate.cache_hit_input_cny_per_million / MILLION,
+ "max_cost_cny": output_cost
+ + prompt * rate.cache_miss_input_cny_per_million / MILLION,
+ }
+
+
+def build_report(calls: Sequence[Mapping[str, Any]]) -> dict[str, Any]:
+ unique: dict[str, Mapping[str, Any]] = {}
+ duplicate_observations = 0
+ for call in calls:
+ call_id = _text(call.get("call_id"))
+ if not call_id:
+ raise CostReportError("physical call lacks call_id")
+ existing = unique.get(call_id)
+ if existing is not None:
+ existing_has_usage = bool(existing.get("usage_recorded", True))
+ current_has_usage = bool(call.get("usage_recorded", True))
+ if current_has_usage and not existing_has_usage:
+ unique[call_id] = call
+ duplicate_observations += 1
+ continue
+ if existing_has_usage and not current_has_usage:
+ duplicate_observations += 1
+ continue
+ same_physical_call = (
+ _text(existing.get("model")) == _text(call.get("model"))
+ and normalize_usage(existing) == normalize_usage(call)
+ )
+ if not same_physical_call:
+ raise CostReportError(f"physical call ID collision: {call_id}")
+ duplicate_observations += 1
+ continue
+ unique[call_id] = call
+
+ buckets: dict[tuple[str, str], dict[str, Any]] = defaultdict(
+ lambda: {
+ "physical_calls": 0,
+ "prompt_tokens": 0,
+ "completion_tokens": 0,
+ "cache_hit_tokens": 0,
+ "cache_miss_tokens": 0,
+ "calls_without_cache_breakdown": 0,
+ "calls_without_usage": 0,
+ "exact_cost_cny": 0.0,
+ "min_cost_cny": 0.0,
+ "max_cost_cny": 0.0,
+ "unpriced_calls": 0,
+ }
+ )
+ exact_total = 0.0
+ min_total = 0.0
+ max_total = 0.0
+ all_priced = True
+ all_exact = True
+ normalized_calls = []
+ unknown_outcome_call_count = 0
+ for call_id, call in sorted(unique.items()):
+ usage = normalize_usage(call)
+ model = _text(call.get("model")) or "unknown"
+ stage = _text(call.get("stage")) or "unknown"
+ price = price_usage(model, usage)
+ bucket = buckets[(stage, model)]
+ bucket["physical_calls"] += 1
+ for key in (
+ "prompt_tokens",
+ "completion_tokens",
+ "cache_hit_tokens",
+ "cache_miss_tokens",
+ ):
+ bucket[key] += int(usage[key])
+ if not usage["cache_breakdown_exact"]:
+ bucket["calls_without_cache_breakdown"] += 1
+ all_exact = False
+ if not usage["usage_recorded"]:
+ bucket["calls_without_usage"] += 1
+ all_priced = False
+ all_exact = False
+ if bool(call.get("external_call_outcome_unknown")):
+ unknown_outcome_call_count += 1
+ if not price["priced"]:
+ bucket["unpriced_calls"] += 1
+ all_priced = False
+ all_exact = False
+ else:
+ bucket["min_cost_cny"] += float(price["min_cost_cny"])
+ bucket["max_cost_cny"] += float(price["max_cost_cny"])
+ min_total += float(price["min_cost_cny"])
+ max_total += float(price["max_cost_cny"])
+ if price["exact_cost_cny"] is not None:
+ bucket["exact_cost_cny"] += float(price["exact_cost_cny"])
+ exact_total += float(price["exact_cost_cny"])
+ normalized_calls.append(
+ {
+ **{key: value for key, value in call.items() if key != "usage"},
+ "usage": usage,
+ "price": price,
+ }
+ )
+
+ by_stage_model = []
+ for (stage, model), bucket in sorted(buckets.items()):
+ item = {"stage": stage, "model": model, **bucket}
+ for key in ("exact_cost_cny", "min_cost_cny", "max_cost_cny"):
+ item[key] = round(float(item[key]), 9)
+ if (
+ item["unpriced_calls"]
+ or item["calls_without_cache_breakdown"]
+ or item["calls_without_usage"]
+ ):
+ item["exact_cost_cny"] = None
+ by_stage_model.append(item)
+ return {
+ "schema_version": "tmcra.v4.cost-report.1",
+ "pricing_basis": "DeepSeek official CNY token rates configured in report",
+ "physical_call_count": len(unique),
+ "definite_physical_call_count": len(unique) - unknown_outcome_call_count,
+ "unknown_outcome_call_count": unknown_outcome_call_count,
+ "duplicate_observation_count": duplicate_observations,
+ "all_calls_priced": all_priced,
+ "cache_breakdown_complete": all_exact and all_priced,
+ "exact_cost_cny": round(exact_total, 9) if all_exact and all_priced else None,
+ "known_priced_exact_component_cny": round(exact_total, 9),
+ "known_priced_min_cost_cny": round(min_total, 9),
+ "known_priced_max_cost_cny": round(max_total, 9),
+ "min_cost_cny": round(min_total, 9) if all_priced else None,
+ "max_cost_cny": round(max_total, 9) if all_priced else None,
+ "by_stage_model": by_stage_model,
+ "calls": normalized_calls,
+ }
+
+
+def collect_calls(json_paths: Iterable[Path], sqlite_paths: Iterable[Path]) -> list[dict[str, Any]]:
+ output: list[dict[str, Any]] = []
+ for path in json_paths:
+ resolved = path.resolve()
+ if not resolved.is_file():
+ raise FileNotFoundError(resolved)
+ for index, value in enumerate(read_json_records(resolved)):
+ output.extend(
+ extract_physical_calls(
+ value, source=str(resolved), path=f"record[{index}]"
+ )
+ )
+ for path in sqlite_paths:
+ resolved = path.resolve()
+ if not resolved.is_file():
+ raise FileNotFoundError(resolved)
+ for location, value in sqlite_call_metadata(resolved):
+ output.extend(
+ extract_physical_calls(
+ value, source=str(resolved), path=location
+ )
+ )
+ return output
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description="TMCRA V4 physical API cost audit")
+ parser.add_argument("--json", action="append", default=[], type=Path)
+ parser.add_argument("--sqlite", action="append", default=[], type=Path)
+ parser.add_argument("--output", required=True, type=Path)
+ args = parser.parse_args()
+ if not args.json and not args.sqlite:
+ raise CostReportError("at least one --json or --sqlite input is required")
+ report = build_report(collect_calls(args.json, args.sqlite))
+ args.output.parent.mkdir(parents=True, exist_ok=True)
+ temporary = args.output.with_suffix(args.output.suffix + ".tmp")
+ temporary.write_text(
+ json.dumps(report, ensure_ascii=False, indent=2, sort_keys=True) + "\n",
+ encoding="utf-8",
+ )
+ temporary.replace(args.output)
+ print(json.dumps({key: report[key] for key in (
+ "physical_call_count", "exact_cost_cny", "min_cost_cny", "max_cost_cny"
+ )}, sort_keys=True))
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_evidence_operations.py b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_evidence_operations.py
new file mode 100644
index 0000000000000000000000000000000000000000..8d205fcce0ee9516e9da33ce2dc941b2ce9f683c
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_evidence_operations.py
@@ -0,0 +1,1083 @@
+"""Evidence compilation and deterministic operations for TMCRA V4.
+
+This module never retrieves, drops, summarizes, or rewrites source evidence.
+It assigns stable packet IDs, extracts verifiable atoms, validates operation
+plans against those IDs, executes deterministic calculations, and validates
+answer claims against the resulting packet.
+"""
+
+from __future__ import annotations
+
+import calendar
+import re
+from collections.abc import Mapping, Sequence
+from datetime import date, datetime, timedelta
+from decimal import Decimal, InvalidOperation
+from typing import Any
+
+from tmcra_v4_task_contract import (
+ RISK_PLANNER_MISSING_WITH_PLAUSIBLE_SOURCE,
+ TaskContractError,
+ structural_risk_signals,
+ validate_task_contract,
+)
+from tmcra_v4_typed_semantics import evaluate_proposals
+
+
+CATALOG_SCHEMA = "tmcra.evidence-atom-catalog.v1"
+GRAPH_SCHEMA = "tmcra.query-evidence-graph.v1"
+PLAN_SCHEMA = "tmcra.evidence-operation-plan.v1"
+PACKET_SCHEMA = "tmcra.compiled-evidence-packet.v1"
+PACKET_COMPILER_VERSION = "tmcra-v4-packet-compiler-2026-07-14.7"
+ANSWER_SCHEMA = "tmcra.evidence-bound-answer.v2"
+ANSWER_CLAIM_ORIGINS = frozenset(
+ {
+ "memory_fact",
+ "memory_inference",
+ "memory_derived",
+ "query_context",
+ "model_knowledge",
+ }
+)
+
+OPERATION_TYPES = frozenset(
+ {
+ "date_difference",
+ "date_order",
+ "numeric_sum",
+ "numeric_difference",
+ "numeric_average",
+ "count_distinct",
+ "ordered_unique_list",
+ "entity_exact_match",
+ "entity_mismatch",
+ "set_difference",
+ }
+)
+ATOM_TYPES = frozenset({"date", "number", "currency", "quantity", "entity"})
+REQUIREMENT_STATES = frozenset(
+ {"satisfied", "conflicting", "missing", "invalid_operation"}
+)
+
+MONTHS = {name.lower(): index for index, name in enumerate(calendar.month_name) if name}
+MONTHS.update(
+ {name.lower(): index for index, name in enumerate(calendar.month_abbr) if name}
+)
+DATE_PATTERNS = (
+ re.compile(r"\b(?P20\d{2})[-/](?P0?[1-9]|1[0-2])[-/](?P0?[1-9]|[12]\d|3[01])(?=T|\b)"),
+ re.compile(
+ r"\b(?PJanuary|February|March|April|May|June|July|August|September|October|November|December|Jan|Feb|Mar|Apr|Jun|Jul|Aug|Sep|Sept|Oct|Nov|Dec)\s+"
+ r"(?P0?[1-9]|[12]\d|3[01])(?:st|nd|rd|th)?(?:,)?\s+(?P20\d{2})\b",
+ re.IGNORECASE,
+ ),
+)
+NUMBER_PATTERN = re.compile(
+ r"(?[$€£¥])?\s*(?P\d+(?:,\d{3})*(?:\.\d+)?)"
+ r"(?:\s*(?Pdays?|weeks?|months?|years?|hours?|minutes?|times?|miles?|kilometers?|km|kg|lbs?|percent|%))?"
+ r"(?!\w)",
+ re.IGNORECASE,
+)
+WORD_PATTERN = re.compile(r"[A-Za-z0-9]+(?:'[A-Za-z0-9]+)?")
+QUESTION_STOPWORDS = frozenset(
+ {"a", "an", "and", "are", "at", "be", "did", "do", "for", "from", "how", "i", "in", "is", "it", "me", "my", "of", "on", "or", "the", "to", "was", "what", "when", "with"}
+)
+SOURCE_TIMESTAMP_PATTERN = re.compile(
+ r"\btimestamp=(?P20\d{2})-(?P0?[1-9]|1[0-2])-(?P0?[1-9]|[12]\d|3[01])(?!\d)"
+)
+RELATIVE_DAY_PATTERNS = (
+ (re.compile(r"\b(?:yesterday|the day before)\b", re.IGNORECASE), -1, "source_timestamp_minus_1_day"),
+ (re.compile(r"\btoday\b", re.IGNORECASE), 0, "source_timestamp_same_day"),
+)
+EXPLICIT_RELATIVE_DURATION_PATTERNS = (
+ (
+ re.compile(
+ r"\b(?:(?:about|around|roughly)\s+)?(?P\d+|a|an|one)\s+days?\s+ago\b",
+ re.IGNORECASE,
+ ),
+ 1,
+ 0,
+ ),
+ (
+ re.compile(
+ r"\b(?:(?:about|around|roughly)\s+)?(?P\d+|a|an|one)\s+weeks?\s+ago\b",
+ re.IGNORECASE,
+ ),
+ 7,
+ 1,
+ ),
+ (
+ re.compile(
+ r"\b(?:(?:about|around|roughly)\s+)?(?P\d+|a|an|one)\s+months?\s+ago\b",
+ re.IGNORECASE,
+ ),
+ 30,
+ 3,
+ ),
+)
+
+
+class EvidenceOperationError(RuntimeError):
+ pass
+
+
+def _text(value: Any) -> str:
+ return value.strip() if isinstance(value, str) else ""
+
+
+def _ids(value: Any, *, path: str, allowed: set[str], allow_empty: bool = False) -> list[str]:
+ if not isinstance(value, Sequence) or isinstance(value, (str, bytes)):
+ raise EvidenceOperationError(f"{path} must be an ID array")
+ output = [_text(item) for item in value]
+ if (not output and not allow_empty) or any(not item or item not in allowed for item in output):
+ raise EvidenceOperationError(f"{path} contains an unknown or empty ID")
+ if len(output) != len(set(output)):
+ raise EvidenceOperationError(f"{path} contains duplicate IDs")
+ return output
+
+
+def _normalize_date(match: re.Match[str]) -> str | None:
+ groups = match.groupdict()
+ try:
+ month = (
+ MONTHS[groups["month_name"].lower()]
+ if groups.get("month_name")
+ else int(groups["month"])
+ )
+ value = date(int(groups["year"]), month, int(groups["day"]))
+ except (KeyError, TypeError, ValueError):
+ return None
+ return value.isoformat()
+
+
+def _overlaps(span: tuple[int, int], spans: Sequence[tuple[int, int]]) -> bool:
+ return any(span[1] > other[0] and span[0] < other[1] for other in spans)
+
+
+def _lexical_terms(value: str) -> set[str]:
+ return {
+ match.group(0).casefold()
+ for match in WORD_PATTERN.finditer(value)
+ if len(match.group(0)) >= 3 and match.group(0).casefold() not in QUESTION_STOPWORDS
+ }
+
+
+def _extract_atoms_from_text(
+ text: str,
+ *,
+ evidence_id: str,
+ counters: dict[str, int],
+ source_timestamp: str = "",
+ source_historical_date: str = "",
+) -> list[dict[str, Any]]:
+ atoms: list[dict[str, Any]] = []
+ date_spans: list[tuple[int, int]] = []
+ for pattern in DATE_PATTERNS:
+ for match in pattern.finditer(text):
+ span = (match.start(), match.end())
+ if _overlaps(span, date_spans):
+ continue
+ normalized = _normalize_date(match)
+ if not normalized:
+ continue
+ counters["date"] += 1
+ date_spans.append(span)
+ atoms.append(
+ {
+ "atom_id": f"D{counters['date']:03d}",
+ "atom_type": "date",
+ "raw_text": match.group(0),
+ "normalized_value": normalized,
+ "unit": "date",
+ "evidence_id": evidence_id,
+ "char_start": span[0],
+ "char_end": span[1],
+ }
+ )
+ if evidence_id != "QUESTION":
+ timestamp_match = SOURCE_TIMESTAMP_PATTERN.search(text)
+ timestamp_value = _normalize_date(timestamp_match) if timestamp_match else None
+ if not timestamp_value and source_timestamp:
+ structured_match = SOURCE_TIMESTAMP_PATTERN.search(
+ f"timestamp={source_timestamp}"
+ )
+ timestamp_value = (
+ _normalize_date(structured_match) if structured_match else None
+ )
+ if not timestamp_value and source_historical_date:
+ for pattern in DATE_PATTERNS:
+ structured_match = pattern.search(source_historical_date)
+ if structured_match:
+ timestamp_value = _normalize_date(structured_match)
+ if timestamp_value:
+ break
+ if timestamp_value:
+ base_date = date.fromisoformat(timestamp_value)
+ for pattern, offset_days, derivation in RELATIVE_DAY_PATTERNS:
+ for match in pattern.finditer(text):
+ counters["date"] += 1
+ atoms.append(
+ {
+ "atom_id": f"D{counters['date']:03d}",
+ "atom_type": "date",
+ "raw_text": match.group(0),
+ "normalized_value": (base_date + timedelta(days=offset_days)).isoformat(),
+ "unit": "date",
+ "evidence_id": evidence_id,
+ "char_start": match.start(),
+ "char_end": match.end(),
+ "derivation": derivation,
+ }
+ )
+ for match in NUMBER_PATTERN.finditer(text):
+ span = (match.start(), match.end())
+ if _overlaps(span, date_spans):
+ continue
+ raw_number = match.group("number").replace(",", "")
+ try:
+ numeric = Decimal(raw_number)
+ except InvalidOperation:
+ continue
+ currency = _text(match.group("currency"))
+ unit = _text(match.group("unit")).lower()
+ atom_type = "currency" if currency else ("quantity" if unit else "number")
+ counters[atom_type] += 1
+ prefix = {"currency": "C", "quantity": "Q", "number": "N"}[atom_type]
+ atoms.append(
+ {
+ "atom_id": f"{prefix}{counters[atom_type]:03d}",
+ "atom_type": atom_type,
+ "raw_text": match.group(0),
+ "normalized_value": str(numeric.normalize()),
+ "unit": currency or unit,
+ "evidence_id": evidence_id,
+ "char_start": span[0],
+ "char_end": span[1],
+ }
+ )
+ atoms.sort(key=lambda item: (int(item["char_start"]), str(item["atom_id"])))
+ return atoms
+
+
+def build_evidence_catalog(row: Mapping[str, Any]) -> dict[str, Any]:
+ question = _text(row.get("question"))
+ windows = row.get("evidence_windows")
+ if not question or not isinstance(windows, Sequence) or isinstance(windows, (str, bytes)) or not windows:
+ raise EvidenceOperationError("evidence row requires a question and source windows")
+ counters = {key: 0 for key in ATOM_TYPES}
+ evidence: list[dict[str, Any]] = []
+ atoms: list[dict[str, Any]] = []
+
+ def object_list(window: Mapping[str, Any], field: str, *, window_index: int) -> list[dict[str, Any]]:
+ raw = window.get(field) or []
+ if not isinstance(raw, Sequence) or isinstance(raw, (str, bytes)):
+ raise EvidenceOperationError(
+ f"evidence_windows[{window_index}].{field} must be an array"
+ )
+ output: list[dict[str, Any]] = []
+ for item_index, item in enumerate(raw):
+ if not isinstance(item, Mapping):
+ raise EvidenceOperationError(
+ f"evidence_windows[{window_index}].{field}[{item_index}] must be an object"
+ )
+ output.append(dict(item))
+ return output
+
+ for index, window in enumerate(windows, start=1):
+ if not isinstance(window, Mapping):
+ raise EvidenceOperationError(f"evidence_windows[{index - 1}] must be an object")
+ raw_text = window.get("text")
+ text = raw_text if isinstance(raw_text, str) else ""
+ session_id = _text(window.get("session_id"))
+ if not text.strip() or not session_id:
+ raise EvidenceOperationError(f"evidence_windows[{index - 1}] lacks source text or session ID")
+ evidence_id = f"E{index:02d}"
+ raw_context = window.get("source_group_context") or []
+ if not isinstance(raw_context, Sequence) or isinstance(raw_context, (str, bytes)):
+ raise EvidenceOperationError(
+ f"evidence_windows[{index - 1}].source_group_context must be an array"
+ )
+ source_group_context: list[dict[str, Any]] = []
+ for context_index, member in enumerate(raw_context):
+ if not isinstance(member, Mapping):
+ raise EvidenceOperationError(
+ f"evidence_windows[{index - 1}].source_group_context[{context_index}] must be an object"
+ )
+ member_text = member.get("text")
+ member_session_id = _text(member.get("session_id"))
+ if (
+ not isinstance(member_text, str)
+ or not member_text.strip()
+ or member_session_id != session_id
+ ):
+ raise EvidenceOperationError(
+ f"evidence_windows[{index - 1}].source_group_context[{context_index}] is invalid"
+ )
+ source_group_context.append(
+ {
+ "relationship": _text(member.get("relationship")) or "session_neighbor",
+ "parent_distance": int(member.get("parent_distance", 0)),
+ "session_id": member_session_id,
+ "session_index": int(member.get("session_index", window.get("session_index", 0))),
+ "parent_chunk_index": int(member.get("parent_chunk_index", 0)),
+ "source_record_id": _text(member.get("source_record_id")),
+ "source_char_start": int(member.get("source_char_start", 0)),
+ "source_char_end": int(
+ member.get("source_char_end", len(member_text))
+ ),
+ "historical_date": _text(member.get("historical_date")),
+ "timestamp": _text(member.get("timestamp")),
+ "message_role": _text(member.get("message_role")),
+ "text": member_text,
+ }
+ )
+ evidence.append(
+ {
+ "evidence_id": evidence_id,
+ "scope_id": _text(window.get("scope_id")),
+ "db_path": _text(window.get("db_path")),
+ "session_id": session_id,
+ "session_index": int(window.get("session_index", 0)),
+ "parent_chunk_index": int(window.get("parent_chunk_index", 0)),
+ "subchunk_index": int(window.get("subchunk_index", 0)),
+ "historical_date": _text(window.get("historical_date")),
+ "timestamp": _text(window.get("timestamp")),
+ "message_role": _text(window.get("message_role")),
+ "text": text,
+ "source_record_id": _text(window.get("source_record_id")),
+ "source_char_start": int(window.get("source_char_start", 0)),
+ "source_char_end": int(window.get("source_char_end", len(text))),
+ "retrieval_metadata": dict(window.get("retrieval_metadata") or {}),
+ "source_group_id": _text(window.get("source_group_id"))
+ or f"source-group::{session_id}:{int(window.get('parent_chunk_index', 0))}",
+ "source_group_context": source_group_context,
+ "memory_contexts": object_list(
+ window, "memory_contexts", window_index=index - 1
+ ),
+ "attachments": object_list(
+ window, "attachments", window_index=index - 1
+ ),
+ "provenance": object_list(
+ window, "provenance", window_index=index - 1
+ ),
+ }
+ )
+ atoms.extend(
+ _extract_atoms_from_text(
+ text,
+ evidence_id=evidence_id,
+ counters=counters,
+ source_timestamp=_text(window.get("timestamp")),
+ source_historical_date=_text(window.get("historical_date")),
+ )
+ )
+ for member in source_group_context:
+ atoms.extend(
+ _extract_atoms_from_text(
+ member["text"],
+ evidence_id=evidence_id,
+ counters=counters,
+ source_timestamp=_text(member.get("timestamp")),
+ source_historical_date=_text(member.get("historical_date")),
+ )
+ )
+ question_terms = _lexical_terms(question)
+ for item in evidence:
+ overlap_terms = sorted(question_terms & _lexical_terms(item["text"]))
+ item["question_overlap_terms"] = overlap_terms
+ item["question_overlap_score"] = len(overlap_terms)
+ question_atoms = _extract_atoms_from_text(
+ question, evidence_id="QUESTION", counters=counters
+ )
+ question_date = _text(row.get("question_date"))
+ if question_date:
+ question_atoms.extend(
+ _extract_atoms_from_text(
+ question_date, evidence_id="QUESTION", counters=counters
+ )
+ )
+ atoms.extend(question_atoms)
+ if len({item["atom_id"] for item in atoms}) != len(atoms):
+ raise EvidenceOperationError("atom IDs are not unique")
+ return {
+ "schema_version": CATALOG_SCHEMA,
+ "question_id": _text(row.get("question_id")),
+ "question": question,
+ "question_date": question_date,
+ "evidence": evidence,
+ "lexical_anchor_ids": [
+ item["evidence_id"]
+ for item in sorted(
+ evidence,
+ key=lambda candidate: (
+ -int(candidate["question_overlap_score"]),
+ int(str(candidate["evidence_id"])[1:]),
+ ),
+ )
+ if item["question_overlap_score"] > 0
+ ][:8],
+ "atoms": atoms,
+ }
+
+
+def build_query_evidence_graph(catalog: Mapping[str, Any]) -> dict[str, Any]:
+ if catalog.get("schema_version") != CATALOG_SCHEMA:
+ raise EvidenceOperationError("catalog schema is invalid")
+ evidence = list(catalog.get("evidence") or [])
+ atoms = list(catalog.get("atoms") or [])
+ nodes: list[dict[str, Any]] = []
+ edges: list[dict[str, Any]] = []
+ sessions: set[str] = set()
+ for item in evidence:
+ evidence_id, session_id = _text(item.get("evidence_id")), _text(item.get("session_id"))
+ if not evidence_id or not session_id:
+ raise EvidenceOperationError("catalog evidence identity is invalid")
+ nodes.append({"node_id": evidence_id, "node_type": "evidence"})
+ session_node = f"S:{session_id}"
+ if session_node not in sessions:
+ sessions.add(session_node)
+ nodes.append({"node_id": session_node, "node_type": "session", "session_id": session_id})
+ edges.append({"edge_type": "belongs_to_session", "source": evidence_id, "target": session_node, "support_ids": [evidence_id]})
+ evidence_ids = {item["evidence_id"] for item in evidence}
+ for atom in atoms:
+ atom_id, evidence_id = _text(atom.get("atom_id")), _text(atom.get("evidence_id"))
+ if not atom_id:
+ raise EvidenceOperationError("catalog atom identity is invalid")
+ nodes.append({"node_id": atom_id, "node_type": "atom", "atom_type": atom.get("atom_type"), "normalized_value": atom.get("normalized_value")})
+ if evidence_id in evidence_ids:
+ edges.append({"edge_type": "derived_from", "source": atom_id, "target": evidence_id, "support_ids": [evidence_id]})
+ return {
+ "schema_version": GRAPH_SCHEMA,
+ "question_id": catalog.get("question_id"),
+ "nodes": nodes,
+ "edges": edges,
+ }
+
+
+def _validate_typed_semantics(
+ value: Any,
+ *,
+ evidence_ids: set[str],
+) -> dict[str, list[dict[str, Any]]]:
+ if not isinstance(value, Mapping) or set(value) != {"observations", "proposals"}:
+ raise EvidenceOperationError("typed_semantics fields are invalid")
+ observations_value = value.get("observations")
+ proposals_value = value.get("proposals")
+ if not isinstance(observations_value, list) or not isinstance(proposals_value, list):
+ raise EvidenceOperationError("typed_semantics observations and proposals must be arrays")
+ observations: list[dict[str, Any]] = []
+ observation_ids: set[str] = set()
+ for index, raw in enumerate(observations_value):
+ if not isinstance(raw, Mapping):
+ raise EvidenceOperationError(f"typed_semantics.observations[{index}] must be an object")
+ observation_id = _text(raw.get("observation_id"))
+ if not observation_id or observation_id in observation_ids:
+ raise EvidenceOperationError(
+ f"typed_semantics.observations[{index}].observation_id is invalid"
+ )
+ current = dict(raw)
+ current["evidence_ids"] = _ids(
+ raw.get("evidence_ids"),
+ path=f"typed_semantics.observations[{index}].evidence_ids",
+ allowed=evidence_ids,
+ )
+ observation_ids.add(observation_id)
+ observations.append(current)
+ proposals: list[dict[str, Any]] = []
+ proposal_ids: set[str] = set()
+ for index, raw in enumerate(proposals_value):
+ if not isinstance(raw, Mapping):
+ raise EvidenceOperationError(f"typed_semantics.proposals[{index}] must be an object")
+ proposal_id = _text(raw.get("candidate_id"))
+ if not proposal_id or proposal_id in proposal_ids:
+ raise EvidenceOperationError(
+ f"typed_semantics.proposals[{index}].candidate_id is invalid"
+ )
+ current = dict(raw)
+ for field in ("source_evidence_ids", "evidence_ids"):
+ if field in current:
+ current[field] = _ids(
+ current[field],
+ path=f"typed_semantics.proposals[{index}].{field}",
+ allowed=evidence_ids,
+ allow_empty=True,
+ )
+ proposal_ids.add(proposal_id)
+ proposals.append(current)
+ return {"observations": observations, "proposals": proposals}
+
+
+def validate_operation_plan(value: Mapping[str, Any], catalog: Mapping[str, Any]) -> dict[str, Any]:
+ required_root = {"schema_version", "requirements", "operations", "bundles"}
+ optional_root = {"task_contract", "typed_semantics"}
+ if (
+ not isinstance(value, Mapping)
+ or not required_root.issubset(value)
+ or not set(value).issubset(required_root | optional_root)
+ ):
+ raise EvidenceOperationError("operation plan root fields are invalid")
+ if value.get("schema_version") != PLAN_SCHEMA:
+ raise EvidenceOperationError("operation plan schema_version is invalid")
+ evidence_ids = {_text(item.get("evidence_id")) for item in catalog.get("evidence") or []}
+ atom_ids = {_text(item.get("atom_id")) for item in catalog.get("atoms") or []}
+ requirements_value, operations_value, bundles_value = value.get("requirements"), value.get("operations"), value.get("bundles")
+ if not all(isinstance(item, list) for item in (requirements_value, operations_value, bundles_value)):
+ raise EvidenceOperationError("requirements, operations, and bundles must be arrays")
+ requirements: list[dict[str, Any]] = []
+ requirement_ids: set[str] = set()
+ for index, item in enumerate(requirements_value):
+ if not isinstance(item, Mapping) or set(item) != {"requirement_id", "description", "evidence_ids"}:
+ raise EvidenceOperationError(f"requirements[{index}] fields are invalid")
+ requirement_id, description = _text(item.get("requirement_id")), _text(item.get("description"))
+ if not requirement_id or requirement_id in requirement_ids or not description:
+ raise EvidenceOperationError(f"requirements[{index}] identity is invalid")
+ requirement_ids.add(requirement_id)
+ requirements.append({"requirement_id": requirement_id, "description": description, "evidence_ids": _ids(item.get("evidence_ids"), path=f"requirements[{index}].evidence_ids", allowed=evidence_ids, allow_empty=True)})
+ operations: list[dict[str, Any]] = []
+ operation_ids: set[str] = set()
+ for index, item in enumerate(operations_value):
+ required = {"operation_id", "operation_type", "input_atom_ids", "input_evidence_ids", "parameters"}
+ if not isinstance(item, Mapping) or set(item) != required:
+ raise EvidenceOperationError(f"operations[{index}] fields are invalid")
+ operation_id, operation_type = _text(item.get("operation_id")), _text(item.get("operation_type"))
+ if not operation_id or operation_id in operation_ids or operation_type not in OPERATION_TYPES:
+ raise EvidenceOperationError(f"operations[{index}] identity or type is invalid")
+ if not isinstance(item.get("parameters"), Mapping):
+ raise EvidenceOperationError(f"operations[{index}].parameters must be an object")
+ operation_ids.add(operation_id)
+ operations.append(
+ {
+ "operation_id": operation_id,
+ "operation_type": operation_type,
+ "input_atom_ids": _ids(item.get("input_atom_ids"), path=f"operations[{index}].input_atom_ids", allowed=atom_ids),
+ "input_evidence_ids": _ids(item.get("input_evidence_ids"), path=f"operations[{index}].input_evidence_ids", allowed=evidence_ids),
+ "parameters": dict(item["parameters"]),
+ }
+ )
+ bundles: list[dict[str, Any]] = []
+ bundle_ids: set[str] = set()
+ for index, item in enumerate(bundles_value):
+ if not isinstance(item, Mapping) or set(item) != {"bundle_id", "role", "evidence_ids"}:
+ raise EvidenceOperationError(f"bundles[{index}] fields are invalid")
+ bundle_id, role = _text(item.get("bundle_id")), _text(item.get("role"))
+ if not bundle_id or bundle_id in bundle_ids or not role:
+ raise EvidenceOperationError(f"bundles[{index}] identity is invalid")
+ bundle_ids.add(bundle_id)
+ bundles.append({"bundle_id": bundle_id, "role": role, "evidence_ids": _ids(item.get("evidence_ids"), path=f"bundles[{index}].evidence_ids", allowed=evidence_ids)})
+ normalized: dict[str, Any] = {
+ "schema_version": PLAN_SCHEMA,
+ "requirements": requirements,
+ "operations": operations,
+ "bundles": bundles,
+ }
+ if "task_contract" in value:
+ try:
+ task_contract = validate_task_contract(value["task_contract"])
+ except TaskContractError as exc:
+ raise EvidenceOperationError(f"task_contract is invalid: {exc}") from exc
+ premise_ids = {
+ item["premise_id"] for item in task_contract["premises"]
+ }
+ required_memory_ids = {
+ item["premise_id"]
+ for item in task_contract["premises"]
+ if item["source"] == "memory" and item["necessity"] == "required"
+ }
+ if not requirement_ids.issubset(premise_ids):
+ raise EvidenceOperationError(
+ "every legacy requirement must share an ID with a task premise"
+ )
+ if not required_memory_ids.issubset(requirement_ids):
+ raise EvidenceOperationError(
+ "every required memory premise must share an ID with a legacy requirement"
+ )
+ normalized["task_contract"] = task_contract
+ if "typed_semantics" in value:
+ normalized["typed_semantics"] = _validate_typed_semantics(
+ value["typed_semantics"], evidence_ids=evidence_ids
+ )
+ return normalized
+
+
+def _decimal(atom: Mapping[str, Any]) -> Decimal:
+ if atom.get("atom_type") not in {"number", "currency", "quantity"}:
+ raise EvidenceOperationError(f"atom {atom.get('atom_id')} is not numeric")
+ try:
+ return Decimal(str(atom["normalized_value"]))
+ except (KeyError, InvalidOperation) as exc:
+ raise EvidenceOperationError(f"atom {atom.get('atom_id')} has an invalid number") from exc
+
+
+def _date(atom: Mapping[str, Any]) -> date:
+ if atom.get("atom_type") != "date":
+ raise EvidenceOperationError(f"atom {atom.get('atom_id')} is not a date")
+ try:
+ return datetime.strptime(str(atom["normalized_value"]), "%Y-%m-%d").date()
+ except (KeyError, ValueError) as exc:
+ raise EvidenceOperationError(f"atom {atom.get('atom_id')} has an invalid date") from exc
+
+
+def _json_number(value: Decimal) -> int | float:
+ return int(value) if value == value.to_integral_value() else float(value)
+
+
+def execute_operation(operation: Mapping[str, Any], catalog: Mapping[str, Any]) -> dict[str, Any]:
+ atoms = {_text(item.get("atom_id")): item for item in catalog.get("atoms") or []}
+ input_ids = list(operation.get("input_atom_ids") or [])
+ selected = [atoms[item] for item in input_ids if item in atoms]
+ if len(selected) != len(input_ids):
+ raise EvidenceOperationError("operation references an unavailable atom")
+ operation_type = _text(operation.get("operation_type"))
+ parameters = dict(operation.get("parameters") or {})
+ result: dict[str, Any]
+ if operation_type == "date_difference":
+ if len(selected) != 2:
+ raise EvidenceOperationError("date_difference requires exactly two date atoms")
+ days = (_date(selected[1]) - _date(selected[0])).days
+ if not bool(parameters.get("signed", False)):
+ days = abs(days)
+ result = {"value": days, "unit": "days"}
+ elif operation_type == "date_order":
+ ordered = sorted(selected, key=lambda item: (_date(item), input_ids.index(item["atom_id"])))
+ result = {"ordered_atom_ids": [item["atom_id"] for item in ordered], "ordered_values": [item["normalized_value"] for item in ordered]}
+ elif operation_type in {"numeric_sum", "numeric_difference", "numeric_average"}:
+ values = [_decimal(item) for item in selected]
+ if not values:
+ raise EvidenceOperationError(f"{operation_type} requires numeric atoms")
+ units = {str(item.get("unit") or "") for item in selected}
+ if len(units) > 1 and not bool(parameters.get("allow_mixed_units", False)):
+ raise EvidenceOperationError(f"{operation_type} received mixed units")
+ if operation_type == "numeric_sum":
+ value = sum(values, Decimal(0))
+ elif operation_type == "numeric_difference":
+ if len(values) != 2:
+ raise EvidenceOperationError("numeric_difference requires exactly two atoms")
+ value = values[0] - values[1]
+ else:
+ value = sum(values, Decimal(0)) / Decimal(len(values))
+ result = {"value": _json_number(value), "unit": next(iter(units), "")}
+ elif operation_type == "count_distinct":
+ if not selected or any(item.get("atom_type") != "entity" for item in selected):
+ raise EvidenceOperationError("count_distinct requires entity atoms")
+ result = {"value": len(dict.fromkeys(str(item.get("normalized_value")) for item in selected)), "unit": "items"}
+ elif operation_type == "ordered_unique_list":
+ values = list(dict.fromkeys(str(item.get("normalized_value")) for item in selected))
+ result = {"values": values, "count": len(values)}
+ elif operation_type in {"entity_exact_match", "entity_mismatch"}:
+ if len(selected) != 2:
+ raise EvidenceOperationError(f"{operation_type} requires exactly two atoms")
+ equal = str(selected[0].get("normalized_value")).casefold() == str(selected[1].get("normalized_value")).casefold()
+ result = {"value": equal if operation_type == "entity_exact_match" else not equal}
+ elif operation_type == "set_difference":
+ left_ids = parameters.get("left_atom_ids")
+ right_ids = parameters.get("right_atom_ids")
+ left = _ids(left_ids, path="parameters.left_atom_ids", allowed=set(input_ids), allow_empty=True)
+ right = _ids(right_ids, path="parameters.right_atom_ids", allowed=set(input_ids), allow_empty=True)
+ right_values = {str(atoms[item].get("normalized_value")) for item in right}
+ result = {"values": [str(atoms[item].get("normalized_value")) for item in left if str(atoms[item].get("normalized_value")) not in right_values]}
+ else:
+ raise EvidenceOperationError(f"unsupported deterministic operation: {operation_type}")
+ return {
+ "operation_id": operation["operation_id"],
+ "operation_type": operation_type,
+ "status": "completed",
+ "input_atom_ids": input_ids,
+ "support_ids": list(operation.get("input_evidence_ids") or []),
+ "result": result,
+ }
+
+
+def execute_operation_plan(plan: Mapping[str, Any], catalog: Mapping[str, Any]) -> list[dict[str, Any]]:
+ validated = validate_operation_plan(plan, catalog)
+ output: list[dict[str, Any]] = []
+ for operation in validated["operations"]:
+ try:
+ output.append(execute_operation(operation, catalog))
+ except EvidenceOperationError as exc:
+ output.append(
+ {
+ "operation_id": operation["operation_id"],
+ "operation_type": operation["operation_type"],
+ "status": "error",
+ "input_atom_ids": operation["input_atom_ids"],
+ "support_ids": operation["input_evidence_ids"],
+ "error": str(exc),
+ }
+ )
+ return output
+
+
+def _explicit_relative_day_target(question: str) -> dict[str, Any] | None:
+ for pattern, multiplier, tolerance in EXPLICIT_RELATIVE_DURATION_PATTERNS:
+ match = pattern.search(question)
+ if match is None:
+ continue
+ raw_count = match.group("count").casefold()
+ count = 1 if raw_count in {"a", "an", "one"} else int(raw_count)
+ return {
+ "surface": match.group(0),
+ "expected_days": count * multiplier,
+ "tolerance_days": max(tolerance, tolerance * count),
+ }
+ return None
+
+
+def certify_operation_results(
+ catalog: Mapping[str, Any], results: Sequence[Mapping[str, Any]]
+) -> list[dict[str, Any]]:
+ """Certify only a unique operation matching an explicit query duration."""
+ output = [dict(item) for item in results]
+ for item in output:
+ if item.get("status") == "completed":
+ item["answer_authoritative"] = False
+ target = _explicit_relative_day_target(_text(catalog.get("question")))
+ if target is None:
+ return output
+ candidates = [
+ item
+ for item in output
+ if item.get("status") == "completed"
+ and item.get("operation_type") == "date_difference"
+ and isinstance(item.get("result"), Mapping)
+ and item["result"].get("unit") == "days"
+ and isinstance(item["result"].get("value"), int)
+ and abs(item["result"]["value"] - target["expected_days"])
+ <= target["tolerance_days"]
+ ]
+ identities = {
+ (
+ item["result"]["value"],
+ tuple(sorted(_text(value) for value in item.get("support_ids") or [])),
+ )
+ for item in candidates
+ }
+ if len(candidates) != 1 or len(identities) != 1:
+ return output
+ candidates[0]["answer_authoritative"] = True
+ candidates[0]["certification"] = {
+ "schema_version": "tmcra.explicit-relative-duration-certification.v1",
+ **target,
+ }
+ return output
+
+
+def operation_plan_structural_risks(plan: Mapping[str, Any]) -> list[str]:
+ """Compute local review signals without trusting model-supplied risk fields."""
+ contract = plan.get("task_contract")
+ if not isinstance(contract, Mapping):
+ return []
+ risks = structural_risk_signals(contract, planner_present=True)
+ if (
+ unbound_memory_requirement_ids(plan)
+ and RISK_PLANNER_MISSING_WITH_PLAUSIBLE_SOURCE not in risks
+ ):
+ risks.append(RISK_PLANNER_MISSING_WITH_PLAUSIBLE_SOURCE)
+ return risks
+
+
+def unbound_memory_requirement_ids(plan: Mapping[str, Any]) -> list[str]:
+ """Return only required memory premises that lack Source bindings."""
+ contract = plan.get("task_contract")
+ premise_by_id = {
+ _text(item.get("premise_id")): item
+ for item in (contract.get("premises") if isinstance(contract, Mapping) else [])
+ if isinstance(item, Mapping)
+ }
+ output: list[str] = []
+ for requirement in plan.get("requirements") or []:
+ if not isinstance(requirement, Mapping) or requirement.get("evidence_ids"):
+ continue
+ requirement_id = _text(requirement.get("requirement_id"))
+ premise = premise_by_id.get(requirement_id)
+ if premise is None:
+ output.append(requirement_id)
+ continue
+ if (
+ premise.get("source") == "memory"
+ and premise.get("necessity") == "required"
+ ):
+ output.append(requirement_id)
+ return output
+
+
+def _execute_typed_semantics(
+ plan: Mapping[str, Any],
+ *,
+ evidence_ids: set[str],
+) -> tuple[list[dict[str, Any]], dict[str, Any]]:
+ typed = plan.get("typed_semantics")
+ if not isinstance(typed, Mapping):
+ return [], {
+ "status": "not_proposed",
+ "advisory": True,
+ "authoritative": False,
+ "accepted": [],
+ "rejected": [],
+ }
+ evaluated = evaluate_proposals(
+ typed.get("observations") or [], typed.get("proposals") or []
+ )
+ accepted_results: list[dict[str, Any]] = []
+ accepted_report: list[dict[str, Any]] = []
+ rejected_report = [dict(item) for item in evaluated.get("rejected") or []]
+ used_operation_ids = {
+ _text(item.get("operation_id"))
+ for item in plan.get("operations") or []
+ if isinstance(item, Mapping)
+ }
+ for index, item in enumerate(evaluated.get("accepted") or [], start=1):
+ current = dict(item)
+ support_ids = list(current.get("source_evidence_ids") or [])
+ unknown = sorted(set(support_ids) - evidence_ids)
+ if unknown:
+ current["status"] = "rejected"
+ current.pop("value", None)
+ current.pop("value_kind", None)
+ current.pop("unit", None)
+ current.pop("result", None)
+ current.pop("operation_results", None)
+ current.setdefault("diagnostics", []).append(
+ {
+ "code": "UNKNOWN_SOURCE_EVIDENCE",
+ "message": "typed proposal cites evidence outside the immutable reservoir",
+ "path": "source_evidence_ids",
+ }
+ )
+ rejected_report.append(current)
+ continue
+ operation_id = f"TS{index:03d}"
+ while operation_id in used_operation_ids:
+ index += 1
+ operation_id = f"TS{index:03d}"
+ used_operation_ids.add(operation_id)
+ result = {
+ "value": current.get("value"),
+ "value_kind": current.get("value_kind"),
+ "unit": current.get("unit"),
+ }
+ accepted_results.append(
+ {
+ "operation_id": operation_id,
+ "operation_type": _text((current.get("result") or {}).get("operation"))
+ or "typed_semantic_program",
+ "status": "completed",
+ "input_atom_ids": [],
+ "support_ids": support_ids,
+ "result": result,
+ "typed_candidate_id": current.get("candidate_id"),
+ "advisory": True,
+ "authoritative": False,
+ }
+ )
+ accepted_report.append(
+ {
+ "candidate_id": current.get("candidate_id"),
+ "operation_id": operation_id,
+ "support_ids": support_ids,
+ "result": result,
+ }
+ )
+ report = {
+ "status": "evaluated",
+ "advisory": True,
+ "authoritative": False,
+ "accepted": accepted_report,
+ "rejected": rejected_report,
+ }
+ return accepted_results, report
+
+
+def compile_evidence_packet(row: Mapping[str, Any], plan: Mapping[str, Any]) -> dict[str, Any]:
+ catalog = build_evidence_catalog(row)
+ graph = build_query_evidence_graph(catalog)
+ validated = validate_operation_plan(plan, catalog)
+ operation_results = certify_operation_results(
+ catalog, execute_operation_plan(validated, catalog)
+ )
+ typed_results, typed_report = _execute_typed_semantics(
+ validated,
+ evidence_ids={item["evidence_id"] for item in catalog["evidence"]},
+ )
+ operation_results.extend(typed_results)
+ operation_by_id = {item["operation_id"]: item for item in operation_results}
+ evidence_by_id = {item["evidence_id"]: item for item in catalog["evidence"]}
+ task_contract = validated.get("task_contract")
+ premise_by_id = {
+ _text(item.get("premise_id")): item
+ for item in (
+ task_contract.get("premises")
+ if isinstance(task_contract, Mapping)
+ else []
+ )
+ if isinstance(item, Mapping)
+ }
+ requirement_coverage: list[dict[str, Any]] = []
+ for requirement in validated["requirements"]:
+ premise = premise_by_id.get(requirement["requirement_id"])
+ premise_source = _text(premise.get("source")) if premise else "memory"
+ if requirement["evidence_ids"]:
+ state = "satisfied"
+ coverage_origin = "source_evidence"
+ elif premise_source in {"query_context", "model_knowledge"}:
+ state = "satisfied"
+ coverage_origin = premise_source
+ else:
+ state = "missing"
+ coverage_origin = premise_source or "memory"
+ requirement_coverage.append(
+ {
+ **requirement,
+ "state": state,
+ "premise_source": premise_source,
+ "coverage_origin": coverage_origin,
+ }
+ )
+ if any(item["status"] == "error" for item in operation_results):
+ failed_support = {support for item in operation_results if item["status"] == "error" for support in item["support_ids"]}
+ for requirement in requirement_coverage:
+ if failed_support.intersection(requirement["evidence_ids"]):
+ requirement["state"] = "invalid_operation"
+ bundles = [
+ {
+ **bundle,
+ "evidence": [evidence_by_id[item] for item in bundle["evidence_ids"]],
+ }
+ for bundle in validated["bundles"]
+ ]
+ structural_risks = operation_plan_structural_risks(validated)
+ question_contract: dict[str, Any] = {
+ "question": catalog["question"],
+ "question_date": catalog["question_date"],
+ "requirement_count": len(requirement_coverage),
+ "operation_count": len(operation_results),
+ }
+ if isinstance(task_contract, Mapping):
+ question_contract["output_origin"] = task_contract["output_origin"]
+ question_contract["task_contract"] = task_contract
+ return {
+ "schema_version": PACKET_SCHEMA,
+ "packet_compiler_version": PACKET_COMPILER_VERSION,
+ "question_id": catalog["question_id"],
+ "question_contract": question_contract,
+ "task_contract": task_contract,
+ "structural_risk_signals": structural_risks,
+ "typed_semantics_report": typed_report,
+ "requirement_coverage": requirement_coverage,
+ "operation_results": operation_results,
+ "evidence_bundles": bundles,
+ "raw_evidence_reservoir": catalog["evidence"],
+ "lexical_anchor_ids": list(catalog.get("lexical_anchor_ids") or []),
+ "atom_catalog": catalog,
+ "query_evidence_graph": graph,
+ "operation_plan": validated,
+ "operation_result_ids": sorted(operation_by_id),
+ }
+
+
+def validate_evidence_bound_answer(value: Mapping[str, Any], packet: Mapping[str, Any]) -> dict[str, Any]:
+ root_fields = {"schema_version", "answerability", "claims", "missing_requirements", "answer"}
+ if not isinstance(value, Mapping) or not root_fields.issubset(value):
+ raise EvidenceOperationError("answer root fields are invalid")
+ if value.get("schema_version") != ANSWER_SCHEMA:
+ raise EvidenceOperationError("answer schema_version is invalid")
+ answerability = _text(value.get("answerability"))
+ if answerability not in {"sufficient", "insufficient"}:
+ raise EvidenceOperationError("answerability is invalid")
+ evidence_ids = {_text(item.get("evidence_id")) for item in packet.get("raw_evidence_reservoir") or []}
+ operation_ids = {
+ _text(item.get("operation_id"))
+ for item in packet.get("operation_results") or []
+ if item.get("status") == "completed"
+ and item.get("answer_authoritative") is True
+ }
+ requirement_ids = {_text(item.get("requirement_id")) for item in packet.get("requirement_coverage") or []}
+ task_contract = packet.get("task_contract")
+ if not isinstance(task_contract, Mapping):
+ question_contract = packet.get("question_contract")
+ task_contract = (
+ question_contract.get("task_contract")
+ if isinstance(question_contract, Mapping)
+ else None
+ )
+ output_origin = (
+ _text(task_contract.get("output_origin"))
+ if isinstance(task_contract, Mapping)
+ else ""
+ )
+ claims_value = value.get("claims")
+ if not isinstance(claims_value, list):
+ raise EvidenceOperationError("claims must be an array")
+ claims: list[dict[str, Any]] = []
+ claim_ids: set[str] = set()
+ for index, claim in enumerate(claims_value):
+ identity_fields = {"claim_id", "text"}
+ if not isinstance(claim, Mapping) or not identity_fields.issubset(claim):
+ raise EvidenceOperationError(f"claims[{index}] fields are invalid")
+ claim_id, text = _text(claim.get("claim_id")), _text(claim.get("text"))
+ if not claim_id or claim_id in claim_ids or not text:
+ raise EvidenceOperationError(f"claims[{index}] identity is invalid")
+ claim_ids.add(claim_id)
+ support_ids = _ids(claim.get("support_ids", []), path=f"claims[{index}].support_ids", allowed=evidence_ids, allow_empty=True)
+ origin = _text(claim.get("origin"))
+ raw_computation_ids = claim.get("computation_ids", [])
+ try:
+ computation_ids = _ids(
+ raw_computation_ids,
+ path=f"claims[{index}].computation_ids",
+ allowed=operation_ids,
+ allow_empty=True,
+ )
+ except EvidenceOperationError:
+ # Older answer prompts used memory_derived for both deterministic
+ # operation results and source-grounded synthesis. Preserve the
+ # cited Source binding while canonicalizing only that legacy case.
+ if origin != "memory_derived" or not support_ids:
+ raise
+ computation_ids = []
+ origin = "memory_inference"
+ if not origin:
+ origin = "memory_derived" if computation_ids else "memory_fact"
+ if origin == "memory_derived" and not computation_ids and support_ids:
+ origin = "memory_inference"
+ if origin not in ANSWER_CLAIM_ORIGINS:
+ raise EvidenceOperationError(f"claims[{index}].origin is invalid")
+ if origin == "memory_fact" and not support_ids:
+ raise EvidenceOperationError(f"claims[{index}] lacks source support")
+ if origin == "memory_inference" and not support_ids:
+ raise EvidenceOperationError(f"claims[{index}] lacks source support")
+ if origin == "memory_derived" and not computation_ids:
+ raise EvidenceOperationError(f"claims[{index}] lacks computation support")
+ if origin in {"query_context", "model_knowledge"}:
+ if output_origin != "memory_conditioned_generation":
+ raise EvidenceOperationError(
+ f"claims[{index}].origin is not allowed for {output_origin or 'unknown'}"
+ )
+ if support_ids or computation_ids:
+ raise EvidenceOperationError(
+ f"claims[{index}] non-memory origin cannot cite memory evidence"
+ )
+ claims.append({"claim_id": claim_id, "text": text, "origin": origin, "support_ids": support_ids, "computation_ids": computation_ids})
+ missing = _ids(value.get("missing_requirements"), path="missing_requirements", allowed=requirement_ids, allow_empty=True)
+ answer = _text(value.get("answer"))
+ if not answer:
+ raise EvidenceOperationError("answer text is empty")
+ # Planner coverage is advisory. The answer model still sees the immutable
+ # raw reservoir and may correct either a false missing or false satisfied
+ # judgment. Final claims remain strictly bound to supplied evidence or
+ # deterministic operation results.
+ if answerability == "sufficient" and (missing or not claims):
+ raise EvidenceOperationError("sufficient answer has unresolved requirements")
+ if answerability == "insufficient" and not missing:
+ raise EvidenceOperationError("insufficient answer must identify missing requirements")
+ if (
+ answerability == "sufficient"
+ and output_origin == "memory_conditioned_generation"
+ and not any(
+ claim["origin"]
+ in {"memory_fact", "memory_inference", "memory_derived"}
+ for claim in claims
+ )
+ ):
+ raise EvidenceOperationError(
+ "memory-conditioned answer lacks a memory-bound claim"
+ )
+ return {"schema_version": ANSWER_SCHEMA, "answerability": answerability, "claims": claims, "missing_requirements": missing, "answer": answer}
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_evidence_planner.py b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_evidence_planner.py
new file mode 100644
index 0000000000000000000000000000000000000000..2571fd031a7a91e324fc217ebe6aefa01346e677
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_evidence_planner.py
@@ -0,0 +1,544 @@
+"""DeepSeek Pro operation planner for compiled TMCRA evidence packets."""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import time
+import urllib.error
+import urllib.request
+import uuid
+from collections.abc import Mapping, Sequence
+from typing import Any
+
+from tmcra_v4_evidence_operations import OPERATION_TYPES, PLAN_SCHEMA, validate_operation_plan
+from tmcra_v4_task_contract import (
+ OUTPUT_CARDINALITIES,
+ OUTPUT_ORIGINS,
+ OUTPUT_ORDERS,
+ OUTPUT_SHAPES,
+ PREMISE_NECESSITY,
+ PREMISE_ROLES,
+ PREMISE_SOURCES,
+ TASK_CONTRACT_SCHEMA,
+)
+from tmcra_v4_typed_semantics import (
+ EVENT_STATUSES,
+ OPERATIONS as TYPED_OPERATIONS,
+ POLARITIES,
+ TEMPORAL_KINDS,
+ VALUE_KINDS,
+)
+
+
+MODEL = "deepseek-v4-pro"
+PROMPT_VERSION = "tmcra-evidence-operation-planner-2026-07-13.8"
+SYSTEM_PROMPT = f"""You compile one memory question into an evidence operation plan.
+Return exactly one JSON object using schema {PLAN_SCHEMA}:
+{{"schema_version":"{PLAN_SCHEMA}","requirements":[{{"requirement_id":"R01","description":"required answer field","evidence_ids":["E01"]}}],"operations":[{{"operation_id":"O01","operation_type":"one supported operation","input_atom_ids":["D001"],"input_evidence_ids":["E01"],"parameters":{{}}}}],"bundles":[{{"bundle_id":"B01","role":"direct|operand|temporal_sequence|historical_state|current_state|preference|counterevidence|context","evidence_ids":["E01"]}}]}}
+
+Supported deterministic operations: {', '.join(sorted(OPERATION_TYPES))}.
+Legacy root fields are always required, even when arrays are empty. Every new planner output must also include task_contract and typed_semantics; typed_semantics is exactly {{"observations":[],"proposals":[]}} and its arrays may be empty. The runtime accepts old packets without these fields only for backward-compatible replay.
+Use only supplied evidence IDs and atom IDs. Never invent an operand, date, number, entity, source ID, answer, fact, summary, or benchmark field.
+Requirements describe every field or operand needed to answer. Use an empty evidence_ids array when the reservoir does not satisfy a requirement.
+Operations are optional. Use them only when supplied atoms exactly support the calculation. Dates, differences, order, sums, averages, counts, and exact comparisons must use operations instead of mental arithmetic.
+Atoms with derivation=source_timestamp_minus_1_day or source_timestamp_same_day are deterministic calendar dates denoted by relative language in that evidence. Use those derived atoms as event dates. When a question describes two remembered events, create separate requirements for both events, find evidence that explicitly supports each description, and calculate between those event dates. Do not assume one source supports both event descriptions unless its text explicitly contains both. lexical_anchor_ids and question_overlap_terms are deterministic search hints, not proof; inspect distinct anchors because separate phrases may identify separate events. In a relative question, words such as "ago" may be anchored by the other remembered event named in the question; use the question date only when the question clearly asks for elapsed time up to the question date. QUESTION atoms may be operation inputs but QUESTION is never an evidence ID.
+For count or list questions, inventory every matching event in the reservoir, bind all supporting evidence IDs, and distinguish separate items from duplicate mentions. Use count_distinct or ordered_unique_list only when entity atoms are supplied; otherwise organize the matching sources and leave operations empty so the answer layer can enumerate grounded items. Do not mark the requirement missing merely because there is no pre-extracted entity atom.
+Treat named entities and activities exactly. Evidence for a shorter, broader, or merely related term does not satisfy a multiword target unless the supplied source explicitly establishes equivalence. Preserve such evidence as counterevidence rather than silently substituting it.
+Bundles organize source evidence without deleting it. For preferences retain specific prior experiences. For updates retain old and current states. For ambiguous or false-premise questions retain counterevidence and entity distinctions.
+Each evidence item may include memory_contexts and attachments. A slow_context is an auditable Slow-graph claim that organizes durable identity, preference, routine, relationship, or standing state around that Source item. A fast_context is a current atomic Fast memory. An attachment with role=override may supersede a same-slot slow_context only when precedence=newer_fast_evidence. Use these graph views to bind the relevant Source evidence IDs and organize bundles, but never cite a capsule ID, claim ID, or memory ID as evidence and never let a graph view replace inspection of the immutable Source text and source_group_context.
+task_contract must use this exact {TASK_CONTRACT_SCHEMA} shape: {{"schema_version":"{TASK_CONTRACT_SCHEMA}","output_origin":"{'|'.join(sorted(OUTPUT_ORIGINS))}","target":{{"subject":"...","relation":"...","entity_constraints":["..."],"temporal_constraints":["..."],"state_constraints":["..."]}},"output":{{"shape":"{'|'.join(sorted(OUTPUT_SHAPES))}","cardinality":"{'|'.join(sorted(OUTPUT_CARDINALITIES))}","order":"{'|'.join(sorted(OUTPUT_ORDERS))}"}},"premises":[{{"premise_id":"R01","description":"...","role":"{'|'.join(sorted(PREMISE_ROLES))}","necessity":"{'|'.join(sorted(PREMISE_NECESSITY))}","source":"{'|'.join(sorted(PREMISE_SOURCES))}","grounded_constraints":["..."],"context_quote":""}}],"operations":[{{"operation_id":"O01","operation_type":"date_difference","input_premise_ids":["R01"],"output_ref":"TARGET","parameters":{{}}}}]}}. The task_contract operations array is optional; all other shown root, target, output, and premise fields are required. Do not emit risk_signals because the runtime computes structural risks locally. A memory premise must use the same premise_id as its corresponding legacy requirement_id. Memory grounded_constraints must be explicit facts or preferences supported by that premise's bound evidence, never an invented recommendation.
+For recommendation or advice requests, set output_origin to memory_conditioned_generation and bind memory requirements to preference or constraint premises with grounded_constraints. The contract describes how memory constrains a new generation, not a ready-made recommendation. Do not mark such a requirement missing merely because the reservoir contains no historical recommendation or advice.
+When present, typed_semantics must be exactly {{"observations":[],"proposals":[]}}. Each observation uses the module schema fields observation_id, evidence_ids, entity_key, value_kind, value, unit, temporal_kind, and polarity; allowed value_kind values are {', '.join(sorted(VALUE_KINDS))}, temporal_kind values are {', '.join(sorted(TEMPORAL_KINDS))}, and polarity values are {', '.join(sorted(POLARITIES))}. Every typed evidence_ids entry must be an evidence ID from the supplied reservoir, never QUESTION or an invented ID. Event observations require event_status in {{{', '.join(sorted(EVENT_STATUSES))}}}; every event or entity input to count_distinct must carry one of those explicit statuses. A proposal contains candidate_id and a non-empty operations array. Each operation contains operation_id, one typed operation name ({', '.join(sorted(TYPED_OPERATIONS))}), explicit input_ids (or one supported alias), and parameters when needed; inputs may reference only prior observations or prior operation results. Use typed operations for latest state selection, aggregate/count, temporal/date arithmetic, and numeric arithmetic whenever applicable. Typed proposals are advisory and never establish absence.
+Do not answer the question and do not rewrite source evidence."""
+
+REVIEW_INSTRUCTION = """A review_context is supplied. Audit the initial plan against every lexical anchor, source atom, slow_context, and fast_context or override. Correct omitted event evidence, wrong temporal anchors, related-entity substitution, incomplete count/list inventories, ignored durable Slow claims, and missed newer Fast overrides. Review both optional structures: check task_contract fields, premise_id alignment with legacy requirement_id, grounded memory constraints, and structural risks including aggregate without typed inventory, temporal reasoning without an operation, multiple states without latest, memory-conditioned generation without grounded constraints, and planner-missing with a plausible source. For memory-conditioned recommendation/advice, distinguish missing preferences or constraints from the absence of a historical recommendation; do not mark the requirement missing solely for the latter. Return one complete replacement plan in the original schema. Preserve a genuinely missing requirement when the reservoir only contains a related but different entity."""
+
+
+class EvidencePlannerError(RuntimeError):
+ def __init__(self, message: str, *, metadata: Mapping[str, Any] | None = None) -> None:
+ super().__init__(message)
+ self.metadata = dict(metadata or {})
+
+
+def _text(value: Any) -> str:
+ return value.strip() if isinstance(value, str) else ""
+
+
+def _typed_evidence_ids(
+ raw: Any,
+ allowed: set[str],
+ path: str,
+ warnings: list[str],
+ *,
+ allow_empty: bool = False,
+) -> list[str] | None:
+ if not isinstance(raw, list) or (not raw and not allow_empty):
+ warnings.append(f"{path}:quarantined_invalid_evidence_ids")
+ return None
+ output: list[str] = []
+ invalid = False
+ for item in raw:
+ evidence_id = _text(item)
+ if not evidence_id or evidence_id not in allowed:
+ invalid = True
+ warnings.append(f"{path}:dropped_unknown_id")
+ continue
+ if evidence_id not in output:
+ output.append(evidence_id)
+ if invalid or (not output and not allow_empty):
+ warnings.append(f"{path}:quarantined_unknown_evidence_id")
+ return None
+ return output
+
+
+def _normalize_typed_observations(
+ raw: Any,
+ *,
+ evidence_ids: set[str],
+ warnings: list[str],
+) -> tuple[list[dict[str, Any]], dict[str, dict[str, Any]]]:
+ if not isinstance(raw, list):
+ warnings.append("typed_semantics.observations:quarantined_invalid_array")
+ return [], {}
+ observations: list[dict[str, Any]] = []
+ by_id: dict[str, dict[str, Any]] = {}
+ required = {
+ "observation_id",
+ "evidence_ids",
+ "entity_key",
+ "value_kind",
+ "value",
+ "unit",
+ "temporal_kind",
+ "polarity",
+ }
+ for index, item in enumerate(raw):
+ path = f"typed_semantics.observations[{index}]"
+ if not isinstance(item, Mapping):
+ warnings.append(f"{path}:quarantined_invalid_observation")
+ continue
+ current = dict(item)
+ if not required.issubset(current):
+ warnings.append(f"{path}:quarantined_missing_typed_fields")
+ continue
+ observation_id = _text(current.get("observation_id"))
+ if not observation_id or observation_id in by_id:
+ warnings.append(f"{path}:quarantined_invalid_observation_id")
+ continue
+ typed_ids = _typed_evidence_ids(current.get("evidence_ids"), evidence_ids, f"{path}.evidence_ids", warnings)
+ if typed_ids is None:
+ warnings.append(f"{path}:quarantined_invalid_evidence")
+ continue
+ if _text(current.get("entity_key")) == "":
+ warnings.append(f"{path}:quarantined_invalid_entity_key")
+ continue
+ if any(
+ not isinstance(current.get(field), str) or current[field] not in allowed
+ for field, allowed in (
+ ("value_kind", VALUE_KINDS),
+ ("temporal_kind", TEMPORAL_KINDS),
+ ("polarity", POLARITIES),
+ )
+ ):
+ warnings.append(f"{path}:quarantined_invalid_typed_enum")
+ continue
+ unit = current.get("unit")
+ if unit is not None and (not isinstance(unit, str) or not unit.strip()):
+ warnings.append(f"{path}:quarantined_invalid_unit")
+ continue
+ status_fields = [field for field in ("event_status", "event_state", "status") if field in current]
+ statuses = [current[field] for field in status_fields]
+ if statuses and any(status != statuses[0] for status in statuses[1:]):
+ warnings.append(f"{path}:quarantined_conflicting_event_status")
+ continue
+ status = statuses[0].strip().lower() if statuses and isinstance(statuses[0], str) else None
+ if statuses and status not in EVENT_STATUSES:
+ warnings.append(f"{path}:quarantined_invalid_event_status")
+ continue
+ if current.get("value_kind") == "event" and not statuses:
+ warnings.append(f"{path}:quarantined_missing_event_status")
+ continue
+ current["evidence_ids"] = typed_ids
+ if status is not None:
+ current["event_status"] = status
+ observations.append(current)
+ by_id[observation_id] = current
+ return observations, by_id
+
+
+def _normalize_typed_proposals(
+ raw: Any,
+ *,
+ evidence_ids: set[str],
+ observations: Mapping[str, Mapping[str, Any]],
+ warnings: list[str],
+) -> list[dict[str, Any]]:
+ if not isinstance(raw, list):
+ warnings.append("typed_semantics.proposals:quarantined_invalid_array")
+ return []
+ proposals: list[dict[str, Any]] = []
+ candidate_ids: set[str] = set()
+ for index, item in enumerate(raw):
+ path = f"typed_semantics.proposals[{index}]"
+ if not isinstance(item, Mapping):
+ warnings.append(f"{path}:quarantined_invalid_proposal")
+ continue
+ current = dict(item)
+ candidate_id = _text(current.get("candidate_id"))
+ operations = current.get("operations")
+ if (
+ not candidate_id
+ or candidate_id in candidate_ids
+ or not isinstance(operations, list)
+ or not operations
+ ):
+ warnings.append(f"{path}:quarantined_invalid_proposal_shape")
+ continue
+ candidate_ids.add(candidate_id)
+
+ source_fields = [field for field in ("source_evidence_ids", "evidence_ids") if field in current]
+ if len(source_fields) == 2 and current[source_fields[0]] != current[source_fields[1]]:
+ warnings.append(f"{path}:quarantined_conflicting_source_evidence_ids")
+ continue
+ invalid = False
+ if source_fields:
+ source_ids = _typed_evidence_ids(
+ current[source_fields[0]],
+ evidence_ids,
+ f"{path}.{source_fields[0]}",
+ warnings,
+ allow_empty=True,
+ )
+ if source_ids is None:
+ invalid = True
+ else:
+ for field in source_fields:
+ current[field] = list(source_ids)
+ if invalid:
+ warnings.append(f"{path}:quarantined_invalid_evidence")
+ continue
+
+ known_refs = set(observations)
+ operation_ids: set[str] = set()
+ for operation_index, raw_operation in enumerate(operations):
+ operation_path = f"{path}.operations[{operation_index}]"
+ if not isinstance(raw_operation, Mapping):
+ invalid = True
+ warnings.append(f"{operation_path}:quarantined_invalid_operation")
+ break
+ operation = dict(raw_operation)
+ operation_id = _text(operation.get("operation_id"))
+ if not operation_id or operation_id in operation_ids:
+ invalid = True
+ warnings.append(f"{operation_path}:quarantined_invalid_operation_id")
+ break
+ operation_ids.add(operation_id)
+ operation_names = [operation[field] for field in ("operation", "operation_type", "op") if field in operation]
+ if not operation_names or not isinstance(operation_names[0], str) or operation_names[0] not in TYPED_OPERATIONS or any(name != operation_names[0] for name in operation_names[1:]):
+ invalid = True
+ warnings.append(f"{operation_path}:quarantined_invalid_operation_type")
+ break
+ input_fields = [field for field in ("input_ids", "input_observation_ids", "observation_ids", "operands", "inputs") if field in operation]
+ if not input_fields or any(operation[field] != operation[input_fields[0]] for field in input_fields[1:]):
+ invalid = True
+ warnings.append(f"{operation_path}:quarantined_invalid_input_reference")
+ break
+ inputs = operation[input_fields[0]]
+ if not isinstance(inputs, list) or not inputs or any(not isinstance(ref, str) or not ref.strip() or ref not in known_refs for ref in inputs):
+ invalid = True
+ warnings.append(f"{operation_path}:quarantined_invalid_input_reference")
+ break
+ if operation_names[0] == "count_distinct":
+ if any(ref not in observations or observations[ref].get("value_kind") not in {"event", "entity_instance"} or observations[ref].get("event_status") not in EVENT_STATUSES for ref in inputs):
+ invalid = True
+ warnings.append(f"{operation_path}:quarantined_missing_count_event_status")
+ break
+ parameters = operation.get("parameters", {})
+ if not isinstance(parameters, Mapping):
+ invalid = True
+ warnings.append(f"{operation_path}:quarantined_invalid_parameters")
+ break
+ known_refs.add(operation_id)
+ if invalid:
+ warnings.append(f"{path}:quarantined_invalid_reference")
+ continue
+ output_fields = [field for field in ("output_ref", "output_operation_id") if field in current]
+ if len(output_fields) == 2 and current[output_fields[0]] != current[output_fields[1]]:
+ warnings.append(f"{path}:quarantined_conflicting_output_reference")
+ continue
+ if output_fields and current[output_fields[0]] not in operation_ids:
+ warnings.append(f"{path}:quarantined_invalid_output_reference")
+ continue
+ current["operations"] = [dict(operation) for operation in operations]
+ proposals.append(current)
+ return proposals
+
+
+def _normalize_typed_semantics(value: Any, evidence_ids: set[str], warnings: list[str]) -> dict[str, list[dict[str, Any]]]:
+ if not isinstance(value, Mapping):
+ warnings.append("typed_semantics:quarantined_invalid_shape")
+ return {"observations": [], "proposals": []}
+ if set(value) != {"observations", "proposals"}:
+ warnings.append("typed_semantics:normalized_to_exact_shape")
+ observations, by_id = _normalize_typed_observations(value.get("observations"), evidence_ids=evidence_ids, warnings=warnings)
+ proposals = _normalize_typed_proposals(value.get("proposals"), evidence_ids=evidence_ids, observations=by_id, warnings=warnings)
+ return {"observations": observations, "proposals": proposals}
+
+
+def _normalize_plan_ids(value: Mapping[str, Any], catalog: Mapping[str, Any]) -> tuple[dict[str, Any], list[str]]:
+ """Fail closed on invented IDs while tolerating harmless list defects."""
+ evidence_ids = {_text(item.get("evidence_id")) for item in catalog.get("evidence") or []}
+ atom_ids = {_text(item.get("atom_id")) for item in catalog.get("atoms") or []}
+ normalized = dict(value)
+ warnings: list[str] = []
+
+ def clean_ids(raw: Any, allowed: set[str], path: str) -> tuple[list[str], bool]:
+ if isinstance(raw, str):
+ warnings.append(f"{path}:coerced_scalar_id")
+ raw = [raw]
+ if not isinstance(raw, list):
+ return raw, False
+ output: list[str] = []
+ invalid = False
+ for item in raw:
+ item_id = _text(item)
+ if not item_id or item_id not in allowed:
+ invalid = True
+ warnings.append(f"{path}:dropped_unknown_id")
+ continue
+ if item_id in output:
+ warnings.append(f"{path}:deduplicated_id")
+ continue
+ output.append(item_id)
+ return output, invalid
+
+ requirements = []
+ for index, item in enumerate(value.get("requirements") or []):
+ current = dict(item) if isinstance(item, Mapping) else item
+ if isinstance(current, dict):
+ current["evidence_ids"], _ = clean_ids(current.get("evidence_ids"), evidence_ids, f"requirements[{index}].evidence_ids")
+ requirements.append(current)
+ normalized["requirements"] = requirements
+
+ operations = []
+ for index, item in enumerate(value.get("operations") or []):
+ current = dict(item) if isinstance(item, Mapping) else item
+ if not isinstance(current, dict):
+ operations.append(current)
+ continue
+ if _text(current.get("operation_type")) not in OPERATION_TYPES:
+ warnings.append(
+ f"operations[{index}]:quarantined_unsupported_legacy_operation"
+ )
+ continue
+ current["input_atom_ids"], bad_atoms = clean_ids(current.get("input_atom_ids"), atom_ids, f"operations[{index}].input_atom_ids")
+ current["input_evidence_ids"], bad_evidence = clean_ids(current.get("input_evidence_ids"), evidence_ids, f"operations[{index}].input_evidence_ids")
+ if bad_atoms or bad_evidence or not current["input_atom_ids"] or not current["input_evidence_ids"]:
+ warnings.append(f"operations[{index}]:quarantined_invalid_operands")
+ continue
+ operations.append(current)
+ normalized["operations"] = operations
+
+ bundles = []
+ for index, item in enumerate(value.get("bundles") or []):
+ current = dict(item) if isinstance(item, Mapping) else item
+ if isinstance(current, dict):
+ current["evidence_ids"], _ = clean_ids(current.get("evidence_ids"), evidence_ids, f"bundles[{index}].evidence_ids")
+ if not current["evidence_ids"]:
+ warnings.append(f"bundles[{index}]:quarantined_empty_bundle")
+ continue
+ bundles.append(current)
+ normalized["bundles"] = bundles
+ if "typed_semantics" in value:
+ normalized["typed_semantics"] = _normalize_typed_semantics(value.get("typed_semantics"), evidence_ids, warnings)
+ return normalized, warnings
+
+
+def normalize_planner_output(
+ value: Mapping[str, Any], catalog: Mapping[str, Any]
+) -> tuple[dict[str, Any], list[str]]:
+ """Normalize harmless model-shape variance and enforce the production plan."""
+ if not isinstance(value, Mapping):
+ raise ValueError("planner output must be an object")
+ normalized_plan, warnings = _normalize_plan_ids(value, catalog)
+ if "task_contract" not in normalized_plan or "typed_semantics" not in normalized_plan:
+ raise ValueError("new planner output requires task_contract and typed_semantics")
+ return validate_operation_plan(normalized_plan, catalog), warnings
+
+
+def planner_payload(catalog: Mapping[str, Any]) -> dict[str, Any]:
+ evidence = [
+ {
+ "evidence_id": item["evidence_id"],
+ "session_id": item["session_id"],
+ "session_index": item["session_index"],
+ "parent_chunk_index": item["parent_chunk_index"],
+ "text": item["text"],
+ "question_overlap_terms": item.get("question_overlap_terms") or [],
+ "question_overlap_score": int(item.get("question_overlap_score", 0)),
+ "source_group_id": item.get("source_group_id"),
+ "source_group_context": list(item.get("source_group_context") or []),
+ "memory_contexts": list(item.get("memory_contexts") or []),
+ "attachments": list(item.get("attachments") or []),
+ }
+ for item in sorted(
+ catalog.get("evidence") or [],
+ key=lambda candidate: (
+ int(candidate.get("session_index", 0)),
+ int(candidate.get("parent_chunk_index", 0)),
+ int(str(candidate.get("evidence_id"))[1:]),
+ ),
+ )
+ ]
+ atoms = [
+ {
+ "atom_id": item["atom_id"],
+ "atom_type": item["atom_type"],
+ "raw_text": item["raw_text"],
+ "normalized_value": item["normalized_value"],
+ "unit": item["unit"],
+ "evidence_id": item["evidence_id"],
+ **({"derivation": item["derivation"]} if item.get("derivation") else {}),
+ }
+ for item in catalog.get("atoms") or []
+ ]
+ return {
+ "question": catalog.get("question"),
+ "question_date": catalog.get("question_date") or "unknown",
+ "lexical_anchor_ids": list(catalog.get("lexical_anchor_ids") or []),
+ "evidence": evidence,
+ "atoms": atoms,
+ }
+
+
+def _usage(value: Any) -> dict[str, int]:
+ if not isinstance(value, Mapping):
+ raise EvidencePlannerError("planner response lacks usage")
+
+ def count(name: str, *aliases: str) -> int:
+ raw = next((value.get(key) for key in (name, *aliases) if value.get(key) is not None), 0)
+ if isinstance(raw, bool) or not isinstance(raw, (int, float)) or raw < 0:
+ raise EvidencePlannerError(f"usage.{name} is invalid")
+ return int(raw)
+
+ prompt = count("prompt_tokens", "input_tokens")
+ completion = count("completion_tokens", "output_tokens")
+ hit = count("prompt_cache_hit_tokens", "cache_read_input_tokens", "cached_tokens")
+ miss_present = any(value.get(key) is not None for key in ("prompt_cache_miss_tokens", "cache_miss_input_tokens"))
+ miss = count("prompt_cache_miss_tokens", "cache_miss_input_tokens")
+ if hit > prompt or (miss_present and hit + miss != prompt):
+ raise EvidencePlannerError("planner cache usage is inconsistent")
+ if not miss_present:
+ miss = prompt - hit
+ return {
+ "prompt_tokens": prompt,
+ "completion_tokens": completion,
+ "prompt_cache_hit_tokens": hit,
+ "prompt_cache_miss_tokens": miss,
+ "total_tokens": count("total_tokens") or prompt + completion,
+ }
+
+
+class DeepSeekEvidenceOperationPlanner:
+ def __init__(self, *, base_url: str, api_keys: Sequence[str], timeout: float = 180.0, max_tokens: int = 8192, model: str = MODEL, provider: str = "deepseek") -> None:
+ self.base_url = _text(base_url).rstrip("/")
+ self.api_keys = list(dict.fromkeys(_text(key) for key in api_keys if _text(key)))
+ self.timeout = float(timeout)
+ self.max_tokens = int(max_tokens)
+ self.model = _text(model)
+ self.provider = _text(provider)
+ self.call_index = 0
+ if not self.base_url or not self.api_keys or not self.model or self.provider not in {"deepseek", "xiaomi_mimo"} or self.timeout <= 0 or self.max_tokens <= 0:
+ raise EvidencePlannerError("planner base URL, key pool, timeout, and max tokens are required")
+
+ def plan(self, catalog: Mapping[str, Any], *, review_context: Mapping[str, Any] | None = None) -> tuple[dict[str, Any], dict[str, Any]]:
+ payload = planner_payload(catalog)
+ if review_context is not None:
+ payload["review_context"] = dict(review_context)
+ body = {
+ "model": self.model,
+ "messages": [
+ {"role": "system", "content": SYSTEM_PROMPT + ("\n\n" + REVIEW_INSTRUCTION if review_context is not None else "")},
+ {"role": "user", "content": json.dumps(payload, ensure_ascii=False, separators=(",", ":"))},
+ ],
+ "temperature": 0,
+ "max_tokens": self.max_tokens,
+ "response_format": {"type": "json_object"},
+ **({"thinking": {"type": "disabled"}, "enable_thinking": False} if self.provider == "deepseek" else {"stream": False}),
+ }
+ encoded = json.dumps(body, ensure_ascii=False).encode("utf-8")
+ request_sha256 = hashlib.sha256(encoded).hexdigest()
+ key_index = self.call_index % len(self.api_keys)
+ self.call_index += 1
+ call_id = "eop_" + uuid.uuid4().hex
+ started = time.time()
+ request = urllib.request.Request(
+ f"{self.base_url}/chat/completions",
+ data=encoded,
+ headers={"Content-Type": "application/json", "Authorization": f"Bearer {self.api_keys[key_index]}"},
+ method="POST",
+ )
+ base_metadata = {
+ "physical_call_id": call_id,
+ "physical_api_call": True,
+ "physical_api_calls": 1,
+ "stage": "evidence_operation_planner",
+ "model": self.model,
+ "provider": self.provider,
+ "prompt_version": PROMPT_VERSION,
+ "review_call": review_context is not None,
+ "api_key_index": key_index,
+ "request_sha256": request_sha256,
+ }
+ try:
+ with urllib.request.urlopen(request, timeout=self.timeout) as response:
+ status = int(response.getcode())
+ raw_http = response.read().decode("utf-8")
+ except urllib.error.HTTPError as exc:
+ detail = exc.read().decode("utf-8", errors="replace")[:2000]
+ raise EvidencePlannerError(
+ f"planner HTTP {exc.code}: {detail}",
+ metadata={**base_metadata, "status": "http_error", "http_status": int(exc.code), "latency_seconds": round(time.time() - started, 3)},
+ ) from exc
+ except (urllib.error.URLError, TimeoutError, OSError) as exc:
+ raise EvidencePlannerError(
+ f"planner request failed: {exc}",
+ metadata={**base_metadata, "status": "request_error", "latency_seconds": round(time.time() - started, 3)},
+ ) from exc
+ try:
+ response_body = json.loads(raw_http)
+ choice = response_body["choices"][0]
+ content = choice["message"]["content"]
+ finish_reason = _text(choice.get("finish_reason"))
+ raw_plan = json.loads(content)
+ except (KeyError, IndexError, TypeError, json.JSONDecodeError) as exc:
+ raise EvidencePlannerError(
+ "planner returned malformed JSON",
+ metadata={**base_metadata, "status": "invalid_response", "http_status": status, "latency_seconds": round(time.time() - started, 3)},
+ ) from exc
+ usage = _usage(response_body.get("usage"))
+ metadata = {
+ **base_metadata,
+ "status": "completed",
+ "http_status": status,
+ "latency_seconds": round(time.time() - started, 3),
+ "finish_reason": finish_reason,
+ "response_sha256": hashlib.sha256(content.encode("utf-8")).hexdigest(),
+ "usage": usage,
+ }
+ if status != 200 or finish_reason != "stop" or not isinstance(raw_plan, Mapping):
+ raise EvidencePlannerError("planner did not return one complete JSON object", metadata=metadata)
+ try:
+ plan, warnings = normalize_planner_output(raw_plan, catalog)
+ except Exception as exc:
+ raise EvidencePlannerError(
+ f"planner returned an invalid operation plan: {exc}",
+ metadata={
+ **metadata,
+ "normalization_warnings": warnings if "warnings" in locals() else [],
+ "raw_plan": dict(raw_plan),
+ },
+ ) from exc
+ metadata["normalization_warnings"] = warnings
+ return plan, metadata
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_online_runtime.py b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_online_runtime.py
new file mode 100644
index 0000000000000000000000000000000000000000..348a9048c579ec24e8eefc0e8a1aa08d9728ea1c
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_online_runtime.py
@@ -0,0 +1,3364 @@
+"""TMCRA V4 online retrieval controller.
+
+V3 supplies the index format, model implementations, graph adapter, and helper
+functions. V4 owns recall execution and composition: source, fast, and slow
+paths run independently before bounded role weights are applied.
+"""
+
+from __future__ import annotations
+
+import argparse
+import gc
+import hashlib
+import json
+import os
+import sqlite3
+import time
+from collections.abc import Callable, Mapping, Sequence
+from contextlib import closing
+from pathlib import Path
+from typing import Any
+
+from tmcra_v4_recall_planner import (
+ DEEPSEEK_FLASH_MODEL,
+ DeepSeekFlashRecallRolePlanner,
+ RecallPlannerError,
+ apply_recall_role_plan,
+ layer_weight,
+ normalized_layer_priorities,
+ validate_recall_role_plan,
+)
+from tmcra_v4_route_policy import (
+ PRODUCTION_FINAL_TOP_K,
+ RETRIEVAL_CONTRACT_SCHEMA,
+ SOURCE_COVERAGE_TRACE_K,
+ RoutePolicyError,
+ validate_production_packing_budget,
+ validate_production_retrieval_mode,
+)
+from tmcra_v4_slow_graph import (
+ PatchValidationError,
+ validate_semantic_summary,
+)
+
+_V3: Any | None = None
+ONLINE_INDEX_SCHEMA_VERSION = "tmcra.v4.online-index.3"
+ONLINE_INDEX_REPORT_SCHEMA_VERSION = "tmcra.v4.online-index-report.1"
+SLOW_INVENTORY_SCHEMA_VERSION = "tmcra.v4.slow-inventory.1"
+SLOW_SUMMARY_CONTRACT_VERSION = "tmcra.v4.slow-lossless-summary.2"
+CURRENT_SLOW_PROMPT_VERSION = "tmcra-v4-slow-graph-2026-07-14.16"
+CURRENT_SLOW_PARTITION_CONTRACT_VERSION = "tmcra.v4.slow-semantic-partition.2"
+RUNTIME_SCHEMA_VERSION = "tmcra.v4.online-retrieval.6"
+SESSION_ORDERING_POLICY = "session_rrf_then_chronological_v1"
+RECENT_DIALOGUE_MAX_TURNS = 8
+RECENT_DIALOGUE_MAX_CHARS = 4000
+ROW_CHECKPOINT_SCHEMA = "tmcra.v4.retrieval-row-checkpoint.1"
+PLANNER_DECISION_SCHEMA = "tmcra.v4.recall-planner-decision.1"
+RUN_STAGING_SCHEMA = "tmcra.v4.retrieval-staging.2"
+PACKING_BUDGET_MODES = frozenset({"fixed", "adaptive"})
+COMPOSITION_MODES = frozenset({"layered", "source-only-diagnostic"})
+EXECUTION_LANES = frozenset({"production", "diagnostic"})
+COMPLEX_QUERY_KINDS = frozenset({"comparison", "historical"})
+COMPLEX_TEMPORAL_FOCUSES = frozenset({"historical", "mixed"})
+COMPLEX_CONFLICT_POLICIES = frozenset(
+ {"compare", "preserve_parallel", "surface_uncertainty"}
+)
+SIMPLE_QUERY_KINDS = frozenset({"fact"})
+SIMPLE_TEMPORAL_FOCUSES = frozenset({"timeless"})
+CURRENT_FAST_RECORD_STATES = frozenset(
+ {"active", "parallel_active", "promoted", "challenged"}
+)
+
+
+def _v3() -> Any:
+ global _V3
+ if _V3 is None:
+ try:
+ import tmcra_v3_online_runtime as runtime
+ except Exception as exc:
+ raise RuntimeError("TMCRA V3 runtime dependencies are unavailable") from exc
+ _V3 = runtime
+ return _V3
+
+
+def __getattr__(name: str) -> Any:
+ if name in {"OnlineModels", "scope_counts", "scope_fingerprint", "load_recent_dialogue_context", "load_native_harness", "append_layered_retrieval_audit", "graph_runtime_env", "read_jsonl"}:
+ return getattr(_v3(), name)
+ raise AttributeError(name)
+
+
+def _arg(args: argparse.Namespace, name: str, default: Any) -> Any:
+ return getattr(args, name, default)
+
+
+def _validated_runtime_route(
+ args: argparse.Namespace, *, label: str
+) -> tuple[str, str, str, int]:
+ composition_mode = str(_arg(args, "composition_mode", "layered"))
+ execution_lane = str(_arg(args, "execution_lane", "production"))
+ packing_budget_mode = str(_arg(args, "packing_budget_mode", "fixed"))
+ top_k = _arg(args, "top_k", PRODUCTION_FINAL_TOP_K)
+ try:
+ validate_production_retrieval_mode(
+ composition_mode, execution_lane=execution_lane
+ )
+ validate_production_packing_budget(
+ packing_budget_mode,
+ top_k,
+ execution_lane=execution_lane,
+ )
+ except RoutePolicyError as exc:
+ raise RuntimeError(f"{label}: retrieval route policy rejected the run: {exc}") from exc
+ return composition_mode, execution_lane, packing_budget_mode, int(top_k)
+
+
+def _clean(value: Any) -> str:
+ return value.strip() if isinstance(value, str) else ""
+
+
+def _normalized_grounding_text(value: Any) -> str:
+ return " ".join(str(value or "").split())
+
+
+def _lossless_summary_projection(claims: Any) -> str:
+ """Project final claims to the V4.7 stored-summary representation."""
+ if not isinstance(claims, list) or not claims:
+ raise RuntimeError("lossless Slow summary projection requires claims")
+ texts: list[str] = []
+ for index, claim in enumerate(claims):
+ if not isinstance(claim, Mapping):
+ raise RuntimeError(f"lossless Slow summary claim {index} is not an object")
+ text = " ".join(_clean(claim.get("text")).split())
+ if not text:
+ raise RuntimeError(f"lossless Slow summary claim {index} lacks text")
+ texts.append(text)
+ return " ".join(texts)
+
+
+def _current_summary_contract(*values: Any) -> bool:
+ for value in values:
+ if isinstance(value, Mapping):
+ if (
+ value.get("summary_contract_version") == SLOW_SUMMARY_CONTRACT_VERSION
+ or value.get("prompt_version") == CURRENT_SLOW_PROMPT_VERSION
+ or value.get("partition_contract_version")
+ == "tmcra.v4.slow-semantic-partition.1"
+ ):
+ return True
+ if _current_summary_contract(value.get("provenance")):
+ return True
+ elif value == SLOW_SUMMARY_CONTRACT_VERSION:
+ return True
+ return False
+
+
+def _validate_active_capsule_partition(
+ slow: Sequence[Mapping[str, Any]],
+) -> None:
+ """Allow controlled compound-support fanout while rejecting duplicate claims."""
+ by_region: dict[str, dict[str, Mapping[str, Any]]] = {}
+ for item in slow:
+ if item.get("candidate_kind") != "capsule_summary":
+ continue
+ if _clean(item.get("status")).casefold() not in {"active", "challenged"}:
+ continue
+ if not _current_summary_contract(item):
+ continue
+ region_key = _clean(item.get("region_key"))
+ if not region_key:
+ raise RuntimeError(
+ "current Slow inventory candidate lacks region_key; legacy fallback is not allowed"
+ )
+ capsule_id = _clean(item.get("capsule_id"))
+ if not capsule_id:
+ raise RuntimeError("current Slow inventory candidate lacks capsule_id")
+ if (
+ item.get("partition_contract_version")
+ != CURRENT_SLOW_PARTITION_CONTRACT_VERSION
+ ):
+ raise RuntimeError(
+ f"{capsule_id}: Slow semantic partition contract is stale or missing"
+ )
+ by_region.setdefault(region_key, {})[capsule_id] = item
+
+ for region_key, capsules in by_region.items():
+ evidence_locations: dict[
+ str, list[tuple[str, int, str, str, str]]
+ ] = {}
+ claim_identity_capsules: dict[tuple[str, str], set[str]] = {}
+ for capsule_id, summary in capsules.items():
+ for claim_index, claim in enumerate(list(summary.get("claims") or [])):
+ if not isinstance(claim, Mapping):
+ continue
+ slot = _clean(claim.get("canonical_slot"))
+ claim_text = " ".join(_clean(claim.get("text")).casefold().split())
+ if slot and claim_text:
+ claim_identity_capsules.setdefault(
+ (slot, claim_text), set()
+ ).add(capsule_id)
+ for role in ("support", "counterevidence"):
+ for evidence_id in list(claim.get(role) or []):
+ evidence_id = _clean(evidence_id)
+ if evidence_id:
+ evidence_locations.setdefault(evidence_id, []).append(
+ (
+ capsule_id,
+ claim_index,
+ role,
+ slot,
+ claim_text,
+ )
+ )
+ invalid_repeated_evidence: dict[str, dict[str, Any]] = {}
+ for evidence_id, locations in evidence_locations.items():
+ if len(locations) <= 1:
+ continue
+ roles = {role for _, _, role, _, _ in locations}
+ claim_identities = [
+ (slot, text) for _, _, _, slot, text in locations
+ ]
+ if roles == {"support"} and len(set(claim_identities)) == len(
+ claim_identities
+ ):
+ continue
+ invalid_repeated_evidence[evidence_id] = {
+ "locations": [
+ f"{capsule}:{claim_index}:{role}"
+ for capsule, claim_index, role, _, _ in locations
+ ],
+ "roles": sorted(roles),
+ "duplicate_semantic_binding": (
+ len(set(claim_identities)) != len(claim_identities)
+ ),
+ }
+ if invalid_repeated_evidence:
+ raise RuntimeError(
+ f"region {region_key}: duplicate evidence citation across active Slow capsules: "
+ f"{json.dumps(invalid_repeated_evidence, ensure_ascii=False, sort_keys=True)}"
+ )
+ duplicated_claims = {
+ f"{slot}\u241f{text}": sorted(capsule_ids)
+ for (slot, text), capsule_ids in claim_identity_capsules.items()
+ if len(capsule_ids) > 1
+ }
+ if duplicated_claims:
+ raise RuntimeError(
+ f"region {region_key}: semantic claim appears in more than one active Slow capsule: "
+ f"{json.dumps(duplicated_claims, ensure_ascii=False, sort_keys=True)}"
+ )
+
+
+def resolve_packing_budget(
+ plan: Mapping[str, Any],
+ *,
+ mode: str,
+ fixed_k: int,
+ simple_k: int,
+ standard_k: int,
+ complex_k: int,
+) -> tuple[int, dict[str, Any]]:
+ """Resolve a per-query evidence budget from the validated recall plan."""
+ normalized = validate_recall_role_plan(plan)
+ if mode not in PACKING_BUDGET_MODES:
+ raise RuntimeError(f"unsupported packing budget mode: {mode!r}")
+ values = (fixed_k, simple_k, standard_k, complex_k)
+ if any(isinstance(value, bool) or int(value) <= 0 for value in values):
+ raise RuntimeError("packing budgets must be positive integers")
+ if not simple_k <= standard_k <= complex_k:
+ raise RuntimeError(
+ "adaptive packing budgets must satisfy simple <= standard <= complex"
+ )
+ if mode == "fixed":
+ return int(fixed_k), {
+ "mode": "fixed",
+ "tier": "fixed",
+ "budget": int(fixed_k),
+ "reasons": ["explicit_fixed_budget"],
+ }
+
+ query_kind = normalized["query_kind"]
+ temporal_focus = normalized["temporal_focus"]
+ conflict_policy = normalized["conflict_policy"]
+ complex_reasons: list[str] = []
+ if query_kind in COMPLEX_QUERY_KINDS:
+ complex_reasons.append(f"query_kind:{query_kind}")
+ if temporal_focus in COMPLEX_TEMPORAL_FOCUSES:
+ complex_reasons.append(f"temporal_focus:{temporal_focus}")
+ if conflict_policy in COMPLEX_CONFLICT_POLICIES:
+ complex_reasons.append(f"conflict_policy:{conflict_policy}")
+ if complex_reasons:
+ tier, budget, reasons = "complex", int(complex_k), complex_reasons
+ elif (
+ query_kind in SIMPLE_QUERY_KINDS
+ and temporal_focus in SIMPLE_TEMPORAL_FOCUSES
+ and conflict_policy not in COMPLEX_CONFLICT_POLICIES
+ ):
+ tier, budget, reasons = "simple", int(simple_k), [
+ f"query_kind:{query_kind}",
+ f"temporal_focus:{temporal_focus}",
+ ]
+ else:
+ tier, budget, reasons = "standard", int(standard_k), [
+ f"query_kind:{query_kind}",
+ f"temporal_focus:{temporal_focus}",
+ f"conflict_policy:{conflict_policy}",
+ ]
+ return budget, {
+ "mode": "adaptive",
+ "tier": tier,
+ "budget": budget,
+ "reasons": reasons,
+ }
+
+
+def _digest(value: Any) -> str:
+ return hashlib.sha256(
+ json.dumps(
+ value, ensure_ascii=False, sort_keys=True, separators=(",", ":")
+ ).encode("utf-8")
+ ).hexdigest()
+
+
+def project_recent_dialogue(
+ value: Sequence[Mapping[str, Any]] | None,
+ *,
+ max_turns: int = RECENT_DIALOGUE_MAX_TURNS,
+ max_chars: int = RECENT_DIALOGUE_MAX_CHARS,
+) -> tuple[list[dict[str, Any]], dict[str, Any]]:
+ """Project a dialogue tail without truncating text or orphaning replies."""
+ if max_turns <= 0 or max_chars <= 0:
+ raise RuntimeError("recent dialogue projection limits must be positive")
+ if value is None:
+ value = []
+ if not isinstance(value, Sequence) or isinstance(value, (str, bytes)):
+ raise RuntimeError("recent dialogue must be a sequence")
+
+ normalized: list[dict[str, Any]] = []
+ for position, item in enumerate(value):
+ if not isinstance(item, Mapping) or set(item) != {
+ "turn_index",
+ "speaker",
+ "text",
+ }:
+ raise RuntimeError(
+ f"recent dialogue turn {position} has an invalid schema"
+ )
+ try:
+ turn_index = int(item["turn_index"])
+ except (TypeError, ValueError) as exc:
+ raise RuntimeError(
+ f"recent dialogue turn {position} has an invalid turn_index"
+ ) from exc
+ speaker = _clean(item.get("speaker")).lower()
+ text = _clean(item.get("text"))
+ if speaker not in {"user", "assistant"} or not text:
+ raise RuntimeError(
+ f"recent dialogue turn {position} has an invalid speaker or text"
+ )
+ normalized.append(
+ {"turn_index": turn_index, "speaker": speaker, "text": text}
+ )
+ if any(
+ left["turn_index"] >= right["turn_index"]
+ for left, right in zip(normalized, normalized[1:])
+ ):
+ raise RuntimeError("recent dialogue is not in strictly increasing turn order")
+
+ excluded: list[dict[str, Any]] = []
+
+ def reject(item: Mapping[str, Any], reason: str) -> None:
+ excluded.append(
+ {
+ "turn_index": int(item["turn_index"]),
+ "speaker": str(item["speaker"]),
+ "text_chars": len(str(item["text"])),
+ "reason": reason,
+ }
+ )
+
+ tail = normalized[-max_turns:]
+ for item in normalized[: max(0, len(normalized) - len(tail))]:
+ reject(item, "outside_turn_budget")
+
+ projected: list[dict[str, Any]] = []
+ index = 0
+ while index < len(tail):
+ item = tail[index]
+ if item["speaker"] == "assistant":
+ reject(
+ item,
+ "text_over_limit"
+ if len(item["text"]) > max_chars
+ else "orphan_assistant",
+ )
+ index += 1
+ continue
+
+ following = tail[index + 1] if index + 1 < len(tail) else None
+ if following is not None and following["speaker"] == "assistant":
+ pair = (item, following)
+ oversized = [member for member in pair if len(member["text"]) > max_chars]
+ if oversized:
+ oversized_ids = {int(member["turn_index"]) for member in oversized}
+ for member in pair:
+ reject(
+ member,
+ "text_over_limit"
+ if int(member["turn_index"]) in oversized_ids
+ else "paired_with_excluded_turn",
+ )
+ else:
+ projected.extend(dict(member) for member in pair)
+ index += 2
+ continue
+
+ if len(item["text"]) > max_chars:
+ reject(item, "text_over_limit")
+ else:
+ projected.append(dict(item))
+ index += 1
+
+ reason_counts: dict[str, int] = {}
+ for item in excluded:
+ reason = str(item["reason"])
+ reason_counts[reason] = reason_counts.get(reason, 0) + 1
+ metadata = {
+ "policy": "complete_user_assistant_pairs_no_truncation_v1",
+ "input_count": len(normalized),
+ "considered_count": len(tail),
+ "included_count": len(projected),
+ "excluded_count": len(excluded),
+ "included_turn_indexes": [int(item["turn_index"]) for item in projected],
+ "excluded_turns": excluded,
+ "excluded_by_reason": reason_counts,
+ "max_turns": max_turns,
+ "max_chars_per_turn": max_chars,
+ "text_truncation_count": 0,
+ }
+ return projected, metadata
+
+
+def source_coverage_trace(
+ candidates: Sequence[Mapping[str, Any]], *, limit: int = SOURCE_COVERAGE_TRACE_K
+) -> list[dict[str, Any]]:
+ """Return text-free source coordinates for offline coverage evaluation."""
+ if limit <= 0:
+ raise RuntimeError("source coverage trace limit must be positive")
+ trace: list[dict[str, Any]] = []
+ for rank, candidate in enumerate(candidates[:limit], start=1):
+ session_id = _clean(candidate.get("session_id"))
+ candidate_id = _clean(candidate.get("candidate_id"))
+ if not session_id or not candidate_id:
+ raise RuntimeError("source candidate lacks an auditable identity")
+ try:
+ location = {
+ "session_index": int(candidate["session_index"]),
+ "parent_chunk_index": int(candidate["parent_chunk_index"]),
+ "subchunk_index": int(candidate["subchunk_index"]),
+ }
+ except (KeyError, TypeError, ValueError) as exc:
+ raise RuntimeError("source candidate lacks an auditable location") from exc
+ trace.append(
+ {
+ "rank": rank,
+ "candidate_id": candidate_id,
+ "session_id": session_id,
+ **location,
+ }
+ )
+ return trace
+
+
+def planner_from_env() -> DeepSeekFlashRecallRolePlanner:
+ pool = [item.strip() for item in os.environ.get("TMCRA_RECALL_PLANNER_API_KEY_POOL", "").split(",") if item.strip()]
+ return DeepSeekFlashRecallRolePlanner(base_url=os.environ.get("TMCRA_RECALL_PLANNER_BASE_URL", ""), model=os.environ.get("TMCRA_RECALL_PLANNER_MODEL", DEEPSEEK_FLASH_MODEL), api_keys=pool)
+
+
+def execute_local_candidate_paths(*, inventories: Mapping[str, Sequence[Any]], source_runner: Any, fast_runner: Any, slow_runner: Any) -> dict[str, Any]:
+ """Run every supplied local generator for every nonempty inventory.
+
+ Roles and weights are intentionally absent from this function. Exceptions
+ propagate and there is no retry or fallback.
+ """
+ runners = {"source": source_runner, "fast": fast_runner, "slow": slow_runner}
+ result: dict[str, Any] = {}
+ for layer in ("source", "fast", "slow"):
+ inventory = inventories.get(layer) or []
+ result[layer] = runners[layer]() if inventory else []
+ return result
+
+
+def _hydrate_fast_semantic_records(
+ db_path: Path,
+ scope_id: str,
+ semantic_records: Sequence[Mapping[str, Any]],
+) -> list[dict[str, Any]]:
+ if not semantic_records:
+ return []
+ ids = [_clean(item.get("memory_id")) for item in semantic_records]
+ if not all(ids) or len(ids) != len(set(ids)):
+ raise RuntimeError(f"{scope_id}: fast semantic index identities are invalid")
+ rows: list[tuple[Any, ...]] = []
+ connection = sqlite3.connect(db_path)
+ try:
+ for offset in range(0, len(ids), 400):
+ batch = ids[offset : offset + 400]
+ placeholders = ",".join("?" for _ in batch)
+ rows.extend(
+ connection.execute(
+ "SELECT memory_id,value,state,metadata_json FROM records "
+ f"WHERE scope_id=? AND memory_id IN ({placeholders})",
+ (scope_id, *batch),
+ ).fetchall()
+ )
+ finally:
+ connection.close()
+ if len(rows) != len(ids):
+ found = {str(memory_id) for memory_id, *_ in rows}
+ missing = sorted(set(ids) - found)
+ raise RuntimeError(f"{scope_id}: indexed fast semantic records are missing: {missing[:5]}")
+ by_id = {
+ str(memory_id): (value, str(state), raw_metadata)
+ for memory_id, value, state, raw_metadata in rows
+ }
+ if set(by_id) != set(ids):
+ missing = sorted(set(ids) - set(by_id))
+ raise RuntimeError(f"{scope_id}: indexed fast semantic records are missing: {missing[:5]}")
+
+ parsed: dict[str, tuple[str, str, dict[str, Any]]] = {}
+ source_ids: set[str] = set()
+ for memory_id in ids:
+ value, state, raw_metadata = by_id[memory_id]
+ try:
+ metadata = json.loads(raw_metadata)
+ except (TypeError, json.JSONDecodeError) as exc:
+ raise RuntimeError(f"{memory_id}: fast semantic metadata is invalid JSON") from exc
+ if not isinstance(metadata, Mapping):
+ raise RuntimeError(f"{memory_id}: fast semantic metadata is not an object")
+ if (
+ _clean(state) not in CURRENT_FAST_RECORD_STATES
+ or _clean(metadata.get("memory_layer")) != "fast"
+ or _clean(metadata.get("content_variant")) != "product_semantic_memory"
+ or _clean(metadata.get("node_kind")) != "atomic_user_assertion"
+ or metadata.get("atomic_evidence_leaf") is not True
+ or _clean(metadata.get("authority")) != "user_assertion"
+ or not isinstance(value, str)
+ or not value.strip()
+ ):
+ raise RuntimeError(f"{memory_id}: indexed fast semantic record is malformed or inactive")
+ source_record_id = _clean(metadata.get("source_record_id"))
+ if not source_record_id:
+ raise RuntimeError(f"{memory_id}: fast semantic record lacks source_record_id")
+ source_ids.add(source_record_id)
+ parsed[memory_id] = (value, state, dict(metadata))
+
+ source_rows: list[tuple[Any, ...]] = []
+ connection = sqlite3.connect(db_path)
+ try:
+ source_id_list = sorted(source_ids)
+ for offset in range(0, len(source_id_list), 400):
+ batch = source_id_list[offset : offset + 400]
+ placeholders = ",".join("?" for _ in batch)
+ source_rows.extend(
+ connection.execute(
+ "SELECT memory_id,value,state,metadata_json FROM records "
+ f"WHERE scope_id=? AND memory_id IN ({placeholders})",
+ (scope_id, *batch),
+ ).fetchall()
+ )
+ finally:
+ connection.close()
+ if len(source_rows) != len(source_ids):
+ found = {str(memory_id) for memory_id, *_ in source_rows}
+ missing = sorted(source_ids - found)
+ raise RuntimeError(f"{scope_id}: immutable source records are missing: {missing[:5]}")
+ source_by_id = {
+ str(memory_id): (value, str(state), raw_metadata)
+ for memory_id, value, state, raw_metadata in source_rows
+ }
+ if set(source_by_id) != source_ids:
+ missing = sorted(source_ids - set(source_by_id))
+ raise RuntimeError(f"{scope_id}: immutable source records are missing: {missing[:5]}")
+
+ def indexed_identity_mismatch(
+ indexed: Mapping[str, Any], identity: Mapping[str, Any], memory_id: str
+ ) -> None:
+ for field, expected in identity.items():
+ if field in indexed and indexed[field] != expected:
+ raise RuntimeError(f"{memory_id}: indexed fast semantic identity differs from SQLite")
+
+ output: list[dict[str, Any]] = []
+ for indexed in semantic_records:
+ memory_id = _clean(indexed.get("memory_id"))
+ value, state, metadata = parsed[memory_id]
+ source_record_id = _clean(metadata.get("source_record_id"))
+ source_record_value, source_record_state, raw_source_metadata = source_by_id[
+ source_record_id
+ ]
+ try:
+ source_metadata = json.loads(raw_source_metadata)
+ except (TypeError, json.JSONDecodeError) as exc:
+ raise RuntimeError(f"{memory_id}: immutable source metadata is invalid JSON") from exc
+ if not isinstance(source_metadata, Mapping):
+ raise RuntimeError(f"{memory_id}: immutable source metadata is not an object")
+ if (
+ _clean(source_metadata.get("content_variant")) != "source_message"
+ or _clean(source_metadata.get("node_kind")) != "immutable_source_message"
+ or source_metadata.get("immutable_evidence_leaf") is not True
+ or _clean(source_metadata.get("source_record_id")) != source_record_id
+ ):
+ raise RuntimeError(f"{memory_id}: immutable source record is malformed")
+ source_value = source_metadata.get("raw_content")
+ if (
+ not isinstance(source_value, str)
+ or not source_value
+ or source_record_state != "evidence"
+ or _normalized_grounding_text(source_record_value)
+ != _normalized_grounding_text(source_value)
+ or source_metadata.get("source_span") != source_value
+ or source_metadata.get("source_turn_text") != source_value
+ ):
+ raise RuntimeError(f"{memory_id}: immutable source content is malformed")
+ source_scope = source_metadata.get("scope_id")
+ if source_scope is not None and _clean(source_scope) != scope_id:
+ raise RuntimeError(f"{memory_id}: immutable source scope differs from requested scope")
+ try:
+ source_session = int(source_metadata["session_index"])
+ source_parent = int(source_metadata["message_index"])
+ semantic_session = int(metadata["session_index"])
+ semantic_parent = int(metadata.get("parent_chunk_index", metadata.get("message_index")))
+ evidence_start = int(metadata["evidence_char_start"])
+ evidence_end = int(metadata["evidence_char_end"])
+ except (KeyError, TypeError, ValueError) as exc:
+ raise RuntimeError(f"{memory_id}: fast semantic identity has unusable coordinates") from exc
+ if (
+ source_session < 0
+ or source_parent < 0
+ or semantic_session != source_session
+ or semantic_parent != source_parent
+ or evidence_start < 0
+ or evidence_end <= evidence_start
+ or evidence_end > len(source_value)
+ ):
+ raise RuntimeError(f"{memory_id}: fast semantic identity has unusable coordinates")
+ for field in (
+ "session_id",
+ "message_id",
+ "event_id",
+ "historical_date",
+ "timestamp",
+ ):
+ source_identity_value = source_metadata.get(field)
+ semantic_identity_value = metadata.get(field)
+ if (
+ source_identity_value is not None
+ or semantic_identity_value is not None
+ ) and source_identity_value != semantic_identity_value:
+ raise RuntimeError(
+ f"{memory_id}: fast semantic {field} differs from immutable source"
+ )
+ semantic_raw_content = metadata.get("raw_content")
+ semantic_source_span = metadata.get("source_span")
+ if (
+ not isinstance(semantic_raw_content, str)
+ or not semantic_raw_content
+ or semantic_source_span != semantic_raw_content
+ or metadata.get("source_turn_text") != source_value
+ ):
+ raise RuntimeError(f"{memory_id}: fast semantic quote metadata is malformed")
+ explicit_quote = metadata.get("evidence_quote")
+ if explicit_quote is not None and explicit_quote != semantic_raw_content:
+ raise RuntimeError(f"{memory_id}: fast semantic quote aliases disagree")
+ evidence_quote = semantic_raw_content
+ if source_value[evidence_start:evidence_end] != evidence_quote:
+ raise RuntimeError(f"{memory_id}: fast evidence span does not match immutable source quote")
+ slot = _clean(metadata.get("canonical_slot") or metadata.get("canonical_slot_key"))
+ if not slot:
+ raise RuntimeError(f"{memory_id}: fast semantic record lacks canonical_slot")
+ if metadata.get("canonical_slot") is not None and metadata.get("canonical_slot_key") is not None and _clean(metadata.get("canonical_slot")) != _clean(metadata.get("canonical_slot_key")):
+ raise RuntimeError(f"{memory_id}: fast semantic canonical slot aliases disagree")
+ memory_type = _clean(metadata.get("memory_type"))
+ durability = _clean(metadata.get("durability") or metadata.get("durability_class"))
+ temporal_status = _clean(metadata.get("temporal_status") or metadata.get("target_status"))
+ source_parent_identity = {
+ "session_index": source_session,
+ "parent_chunk_index": source_parent,
+ "source_record_id": source_record_id,
+ "evidence_char_start": evidence_start,
+ "evidence_char_end": evidence_end,
+ }
+ provenance = {
+ "memory_layer": "fast",
+ "content_variant": "product_semantic_memory",
+ "source_record_id": source_record_id,
+ "semantic_memory_id": memory_id,
+ }
+ indexed_identity_mismatch(
+ indexed,
+ {
+ "canonical_slot": slot,
+ "canonical_slot_key": slot,
+ "source_record_id": source_record_id,
+ "source_parent": source_parent_identity,
+ "provenance": provenance,
+ "record_state": state,
+ "memory_type": memory_type,
+ "durability": durability,
+ "temporal_status": temporal_status,
+ "evidence_quote": evidence_quote,
+ "evidence_char_start": evidence_start,
+ "evidence_char_end": evidence_end,
+ },
+ memory_id,
+ )
+ output.append(
+ {
+ "memory_id": memory_id,
+ "text": value,
+ "record_state": state,
+ "canonical_slot": slot,
+ "source_parent": source_parent_identity,
+ "provenance": provenance,
+ "memory_type": memory_type,
+ "durability": durability,
+ "temporal_status": temporal_status,
+ }
+ )
+ return output
+
+
+def _map_fast_candidates_with_slots(candidates: Sequence[Mapping[str, Any]], semantic_records: Sequence[Mapping[str, Any]]) -> list[dict[str, Any]]:
+ """Apply the V3 semantic-leaf mapper to Fast graph candidates."""
+ mapped = _v3()._fast_candidates_with_slots(candidates, semantic_records)
+ by_id = {
+ _clean(item.get("memory_id")): item
+ for item in semantic_records
+ if _clean(item.get("memory_id"))
+ }
+ for item in mapped:
+ memories = []
+ for memory_id in list(item.get("semantic_record_ids") or []):
+ record = by_id.get(_clean(memory_id))
+ if not isinstance(record, Mapping) or not _clean(record.get("text")):
+ continue
+ memories.append(
+ {
+ "memory_id": _clean(record.get("memory_id")),
+ "canonical_slot": _clean(record.get("canonical_slot")),
+ "text": _clean(record.get("text")),
+ "record_state": _clean(record.get("record_state")),
+ "memory_type": _clean(record.get("memory_type")),
+ "durability": _clean(record.get("durability")),
+ "temporal_status": _clean(record.get("temporal_status")),
+ "source_parent": dict(record.get("source_parent") or {}),
+ "provenance": dict(record.get("provenance") or {}),
+ }
+ )
+ item["semantic_memories"] = memories
+ return mapped
+
+
+def _get_graph_adapter(
+ *,
+ harness: Any,
+ scope_id: str,
+ db_path: Path,
+ cache: dict[tuple[str, str], Any] | None,
+) -> tuple[Any, bool]:
+ """Keep at most one graph adapter alive during a batched retrieval run."""
+ key = (scope_id, str(db_path))
+ if cache is None:
+ return harness.build_adapter(scope_id, db_path), False
+ adapter = cache.get(key)
+ if adapter is not None:
+ return adapter, True
+ if cache:
+ cache.clear()
+ gc.collect()
+ torch = _v3().torch
+ if torch.cuda.is_available():
+ torch.cuda.empty_cache()
+ adapter = harness.build_adapter(scope_id, db_path)
+ cache[key] = adapter
+ return adapter, False
+
+
+def _graph_fast_path(*, qid: str, runtime_question: str, fast: Sequence[Mapping[str, Any]], dense_scores: Any, dense_rank: Mapping[int, int], semantic_records: Sequence[Mapping[str, Any]], args: argparse.Namespace, harness: Any, scope_id: str, db_path: Path, graph_adapter_cache: dict[tuple[str, str], Any] | None, models: Any) -> tuple[list[dict[str, Any]], dict[str, Any], float, float]:
+ v3 = _v3()
+ parents: dict[tuple[int, int], list[int]] = {}
+ for index, candidate in enumerate(fast):
+ parents.setdefault((int(candidate["session_index"]), int(candidate["parent_chunk_index"])), []).append(index)
+ adapter, reused = _get_graph_adapter(
+ harness=harness,
+ scope_id=scope_id,
+ db_path=db_path,
+ cache=graph_adapter_cache,
+ )
+ started = time.time()
+ retrieval = adapter.retrieve(runtime_question, top_k=_arg(args, "graph_top_k", 12))
+ elapsed = time.time() - started
+ metadata = dict(getattr(retrieval, "metadata", {}) or {})
+ if _clean(metadata.get("retrieval_mode")) != "hybrid_node_scored" or not bool(metadata.get("hybrid_enabled")):
+ raise RuntimeError(f"{qid}: graph fast path is not active")
+ selected = list(metadata.get("selected_event_ids") or [])
+ recall = list(metadata.get("recall_event_ids") or [])
+ final = list(metadata.get("final_hit_event_ids") or [])
+ if not selected:
+ raise RuntimeError(f"{qid}: graph fast path selected no events")
+ slow_ids = [str(item) for item in [*selected, *recall, *final] if str(item).startswith("slow.")]
+ if slow_ids:
+ raise RuntimeError(f"{qid}: fast graph crossed the slow-layer boundary: {','.join(dict.fromkeys(slow_ids))}")
+ valid_locations = set(parents)
+ selected_parents, selected_unmapped = v3.ordered_graph_parents(selected, valid_locations=valid_locations, strict_prefix=True)
+ recall_parents, recall_unmapped = v3.ordered_graph_parents(recall, valid_locations=valid_locations)
+ final_parents, final_unmapped = v3.ordered_graph_parents(final, valid_locations=valid_locations)
+ if selected_unmapped or final_unmapped:
+ raise RuntimeError(f"{qid}: fast graph events cannot map to persisted chunks")
+ graph_parents = list(dict.fromkeys([*selected_parents, *recall_parents]))[: int(_arg(args, "graph_k", 24))]
+ order = v3.expand_parent_locations(graph_parents, parents)
+ selected_indexes = set(v3.expand_parent_locations(selected_parents, parents))
+ final_indexes = set(v3.expand_parent_locations(final_parents, parents))
+ fast_runtime = []
+ graph_rank: dict[int, int] = {}
+ for rank, index in enumerate(order):
+ item = dict(fast[index])
+ item["channels"] = {"dense_score": float(dense_scores[index]), "dense_rank_rr": v3.rrank(dense_rank[index]), "graph_rank_rr": v3.rrank(rank), "graph_selected": float(index in selected_indexes), "graph_final": float(index in final_indexes), "recency_norm": float(item["session_index"]) / max(1, max(int(candidate["session_index"]) for candidate in fast))}
+ fast_runtime.append(item)
+ graph_rank[index] = rank
+ if not fast_runtime:
+ raise RuntimeError(f"{qid}: graph fast path produced no candidates")
+ cross_started = time.time()
+ representations, logits = models.encode_cross(runtime_question, [item["text"] for item in fast_runtime])
+ channels = v3.CHANNEL_NAMES
+ channel_tensor = v3.torch.tensor([[item["channels"][name] for name in channels] for item in fast_runtime], dtype=v3.torch.float32, device=models.device)
+ with v3.torch.inference_mode():
+ scores = models.fusion(representations.unsqueeze(0), logits.unsqueeze(0), channel_tensor.unsqueeze(0), v3.torch.ones((1, len(fast_runtime)), dtype=v3.torch.bool, device=models.device), ablation="full")[0].detach().cpu()
+ cross_elapsed = time.time() - cross_started
+ for item, score, semantic in zip(fast_runtime, scores.tolist(), logits.detach().cpu().tolist()):
+ item["score"], item["semantic_logit"] = float(score), float(semantic)
+ ranked = sorted(fast_runtime, key=lambda item: (-float(item["score"]), str(item.get("candidate_id", ""))))
+ ranked = _map_fast_candidates_with_slots(ranked, semantic_records)
+ graph = {"skipped": False, "adapter_reused": reused, "retrieval_mode": metadata.get("retrieval_mode"), "hybrid_enabled": metadata.get("hybrid_enabled"), "selected_event_ids": selected, "recall_event_ids": recall, "final_hit_event_ids": final, "unmapped_recall_event_ids": recall_unmapped, "layer": "fast"}
+ return ranked, graph, elapsed, cross_elapsed
+
+
+def _source_local_path(*, runtime_question: str, fast: Sequence[Mapping[str, Any]], fast_vectors: Any, models: Any, args: argparse.Namespace) -> tuple[list[dict[str, Any]], Any, dict[int, int], float, float]:
+ """BGE dense/cross retrieval over immutable source windows."""
+ v3 = _v3()
+ started = time.time()
+ dense_scores = fast_vectors @ models.dense.encode_one(runtime_question)
+ dense_order = sorted(range(len(fast)), key=lambda index: (-float(dense_scores[index]), index))
+ dense_rank = {index: rank for rank, index in enumerate(dense_order)}
+ indexes = []
+ selected_parents: set[tuple[int, int]] = set()
+ for index in dense_order:
+ parent = (
+ int(fast[index]["session_index"]),
+ int(fast[index]["parent_chunk_index"]),
+ )
+ if parent in selected_parents:
+ continue
+ selected_parents.add(parent)
+ indexes.append(index)
+ if len(indexes) >= int(_arg(args, "dense_k", 32)):
+ break
+ if not indexes:
+ raise RuntimeError("source local path produced no candidates")
+ runtime = [dict(fast[index]) for index in indexes]
+ cross_started = time.time()
+ dense_elapsed = cross_started - started
+ representations, logits = models.encode_cross(runtime_question, [item["text"] for item in runtime])
+ cross_elapsed = time.time() - cross_started
+ for item, semantic in zip(runtime, logits.detach().cpu().tolist()):
+ item["semantic_logit"] = float(semantic)
+ ranked = sorted(runtime, key=lambda item: (-float(item["semantic_logit"]), str(item.get("candidate_id", ""))))
+ ranked = _collapse_source_parents(ranked, fast)
+ for rank, item in enumerate(ranked, start=1):
+ item["source_rank"] = rank - 1
+ item["source_path"] = "bge_dense_cross"
+ item["score"] = item["semantic_logit"]
+ return ranked, dense_scores, dense_rank, dense_elapsed, cross_elapsed
+
+
+def _collapse_source_parents(
+ ranked_representatives: Sequence[Mapping[str, Any]],
+ inventory: Sequence[Mapping[str, Any]],
+) -> list[dict[str, Any]]:
+ """Turn ranked subchunks into lossless parent evidence units."""
+ by_parent: dict[tuple[int, int], list[Mapping[str, Any]]] = {}
+ for candidate in inventory:
+ key = (
+ int(candidate["session_index"]),
+ int(candidate["parent_chunk_index"]),
+ )
+ by_parent.setdefault(key, []).append(candidate)
+ output: list[dict[str, Any]] = []
+ source_parent_payloads: dict[str, tuple[str, str]] = {}
+ for representative in ranked_representatives:
+ key = (
+ int(representative["session_index"]),
+ int(representative["parent_chunk_index"]),
+ )
+ members = sorted(
+ by_parent.get(key) or [],
+ key=lambda item: (
+ int(item["source_char_start"]),
+ int(item["source_char_end"]),
+ int(item["subchunk_index"]),
+ ),
+ )
+ if not members:
+ raise RuntimeError(f"source parent has no inventory members: {key}")
+ temporal_identity = (
+ _clean(members[0].get("historical_date")),
+ _clean(members[0].get("timestamp")),
+ _clean(members[0].get("message_role") or members[0].get("role")),
+ )
+ if any(
+ (
+ _clean(member.get("historical_date")),
+ _clean(member.get("timestamp")),
+ _clean(member.get("message_role") or member.get("role")),
+ )
+ != temporal_identity
+ for member in members[1:]
+ ):
+ raise RuntimeError(f"source parent temporal metadata is inconsistent: {key}")
+ prefix = str(members[0]["text"]).split("\n", 1)[0]
+ assembled = ""
+ assembled_end = 0
+ for member in members:
+ start = int(member["source_char_start"])
+ end = int(member["source_char_end"])
+ member_prefix, separator, payload = str(member["text"]).partition("\n")
+ if not separator or member_prefix != prefix or end - start != len(payload):
+ raise RuntimeError(f"source parent subchunk metadata is inconsistent: {key}")
+ if start > assembled_end:
+ raise RuntimeError(f"source parent subchunks contain a gap: {key}")
+ overlap = max(0, assembled_end - start)
+ assembled += payload[overlap:]
+ assembled_end = max(assembled_end, end)
+ candidate_id = f"parent::{representative['session_id']}:{key[1]}"
+ payload_identity = (assembled, temporal_identity[2])
+ previous_payload = source_parent_payloads.get(candidate_id)
+ if previous_payload is not None:
+ if previous_payload != payload_identity:
+ raise RuntimeError(
+ f"source parent candidate ID collision has different content: {candidate_id}"
+ )
+ continue
+ source_parent_payloads[candidate_id] = payload_identity
+ parent = dict(representative)
+ parent.update(
+ {
+ "candidate_id": candidate_id,
+ "text": assembled,
+ "subchunk_index": 0,
+ "source_char_start": 0,
+ "source_char_end": assembled_end,
+ "evidence_unit_kind": "source_parent",
+ "member_candidate_ids": [str(item["candidate_id"]) for item in members],
+ "member_subchunk_indexes": [int(item["subchunk_index"]) for item in members],
+ "historical_date": temporal_identity[0],
+ "timestamp": temporal_identity[1],
+ "message_role": temporal_identity[2],
+ }
+ )
+ output.append(parent)
+ return output
+
+
+def load_v4_layered_inventory(
+ db_path: Path,
+ scope_id: str,
+ parents: Sequence[Mapping[str, Any]],
+) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
+ """Build a typed Slow summary/claim inventory above V3's source mapping."""
+ v3 = _v3()
+ raw_claim_candidates, semantic_records = v3.load_layered_inventory(
+ db_path, scope_id, parents
+ )
+ if not raw_claim_candidates:
+ return [], semantic_records
+
+ values_by_memory_id: dict[str, str] = {}
+ record_metadata_by_memory_id: dict[str, dict[str, Any]] = {}
+ patch_metadata_by_id: dict[str, dict[str, Any]] = {}
+ con = sqlite3.connect(db_path)
+ try:
+ record_columns = {
+ str(row[1])
+ for row in con.execute("PRAGMA table_info(records)").fetchall()
+ }
+ columns = "memory_id,value"
+ if "metadata_json" in record_columns:
+ columns += ",metadata_json"
+ for row in con.execute(
+ f"SELECT {columns} FROM records WHERE scope_id=?", (scope_id,)
+ ).fetchall():
+ memory_id, value = row[:2]
+ values_by_memory_id[str(memory_id)] = str(value)
+ if len(row) > 2 and row[2]:
+ try:
+ metadata = json.loads(row[2])
+ except (TypeError, json.JSONDecodeError) as exc:
+ raise RuntimeError(
+ f"{memory_id}: Slow record metadata is not valid JSON"
+ ) from exc
+ if not isinstance(metadata, dict):
+ raise RuntimeError(f"{memory_id}: Slow record metadata is not an object")
+ record_metadata_by_memory_id[str(memory_id)] = metadata
+ patch_table = con.execute(
+ "SELECT 1 FROM sqlite_master WHERE type='table' AND name='slow_graph_patches'"
+ ).fetchone()
+ if patch_table is not None:
+ for patch_id, raw_metadata in con.execute(
+ "SELECT patch_id,call_metadata_json FROM slow_graph_patches"
+ ).fetchall():
+ try:
+ metadata = json.loads(raw_metadata)
+ except (TypeError, json.JSONDecodeError) as exc:
+ raise RuntimeError(
+ f"{patch_id}: Slow patch metadata is not valid JSON"
+ ) from exc
+ if not isinstance(metadata, dict):
+ raise RuntimeError(f"{patch_id}: Slow patch metadata is not an object")
+ patch_metadata_by_id[str(patch_id)] = metadata
+ finally:
+ con.close()
+
+ grouped: dict[str, list[dict[str, Any]]] = {}
+ for raw in raw_claim_candidates:
+ candidate = dict(raw)
+ memory_id = _clean(candidate.get("memory_id"))
+ if not memory_id:
+ raise RuntimeError("Slow claim candidate lacks memory_id")
+ grouped.setdefault(memory_id, []).append(candidate)
+
+ summary_candidates: list[dict[str, Any]] = []
+ claim_candidates: list[dict[str, Any]] = []
+ for memory_id in sorted(grouped):
+ members = list(grouped[memory_id])
+ capsule_ids = {_clean(item.get("capsule_id")) for item in members}
+ revisions = {item.get("revision") for item in members}
+ if len(capsule_ids) != 1 or "" in capsule_ids or len(revisions) != 1:
+ raise RuntimeError(f"{memory_id}: Slow claim identity is inconsistent")
+ capsule_id = next(iter(capsule_ids))
+ revision = next(iter(revisions))
+ if not isinstance(revision, int) or revision < 1:
+ raise RuntimeError(f"{memory_id}: Slow revision is invalid")
+ summary = values_by_memory_id.get(memory_id, "")
+ record_metadata = record_metadata_by_memory_id.get(memory_id, {})
+ patch_metadata = patch_metadata_by_id.get(
+ _clean(record_metadata.get("patch_id")), {}
+ )
+ current_summary_contract = _current_summary_contract(
+ record_metadata,
+ patch_metadata,
+ *(member.get("provenance") for member in members),
+ )
+ claims: list[dict[str, Any]] = []
+ all_parents: list[dict[str, Any]] = []
+ seen_parents: set[tuple[Any, ...]] = set()
+ for member in members:
+ member_claims = list(member.get("claims") or [])
+ if len(member_claims) != 1 or not isinstance(member_claims[0], Mapping):
+ raise RuntimeError(
+ f"{member.get('candidate_id')}: Slow claim candidate is not atomic"
+ )
+ claims.append(dict(member_claims[0]))
+ for parent in list(member.get("source_parents") or []):
+ if not isinstance(parent, Mapping):
+ raise RuntimeError(
+ f"{member.get('candidate_id')}: Slow source parent is invalid"
+ )
+ identity = (
+ parent.get("session_index"),
+ parent.get("parent_chunk_index"),
+ parent.get("source_record_id"),
+ parent.get("evidence_char_start"),
+ parent.get("evidence_char_end"),
+ )
+ if identity not in seen_parents:
+ seen_parents.add(identity)
+ all_parents.append(dict(parent))
+ try:
+ summary = validate_semantic_summary(
+ summary,
+ claims,
+ label=f"{memory_id} summary",
+ )
+ except PatchValidationError as exc:
+ raise RuntimeError(
+ f"{memory_id}: Slow summary violates the V4 inventory contract: {exc}"
+ ) from exc
+ if current_summary_contract:
+ expected_summary = _lossless_summary_projection(claims)
+ if summary != expected_summary:
+ raise RuntimeError(
+ f"{memory_id}: current V4.7 Slow summary is not the exact lossless "
+ f"claim projection; expected {expected_summary!r}"
+ )
+
+ summary_candidate_id = f"capsule-summary::{capsule_id}:r{revision}"
+ child_ids = [_clean(item.get("candidate_id")) for item in members]
+ if any(not item for item in child_ids) or len(set(child_ids)) != len(child_ids):
+ raise RuntimeError(f"{memory_id}: Slow claim candidate IDs are invalid")
+ slots = sorted({_clean(item.get("canonical_slot")) for item in members})
+ summary_candidate = {
+ "inventory_schema_version": SLOW_INVENTORY_SCHEMA_VERSION,
+ "candidate_kind": "capsule_summary",
+ "candidate_id": summary_candidate_id,
+ "memory_id": memory_id,
+ "capsule_id": capsule_id,
+ "revision": revision,
+ "status": _clean(members[0].get("status")),
+ "canonical_slot": slots[0] if len(slots) == 1 else "slow.summary",
+ "claims": claims,
+ "source_parents": all_parents,
+ "text": summary,
+ "child_claim_candidate_ids": child_ids,
+ "provenance": {
+ "memory_layer": "slow",
+ "content_variant": "slow_memory_capsule",
+ "capsule_id": capsule_id,
+ "revision": revision,
+ "candidate_kind": "capsule_summary",
+ "source_parents": all_parents,
+ },
+ }
+ if current_summary_contract:
+ summary_candidate["summary_contract_version"] = SLOW_SUMMARY_CONTRACT_VERSION
+ summary_candidate["provenance"]["summary_contract_version"] = (
+ SLOW_SUMMARY_CONTRACT_VERSION
+ )
+ partition_contract_version = _clean(
+ record_metadata.get("partition_contract_version")
+ )
+ if partition_contract_version:
+ summary_candidate["partition_contract_version"] = (
+ partition_contract_version
+ )
+ summary_candidate["provenance"]["partition_contract_version"] = (
+ partition_contract_version
+ )
+ region_key = _clean(record_metadata.get("region_key"))
+ if not region_key:
+ region_key = _clean(members[0].get("region_key"))
+ if not region_key:
+ region_key = _clean(
+ (members[0].get("provenance") or {}).get("region_key")
+ )
+ if region_key:
+ summary_candidate["region_key"] = region_key
+ summary_candidate["provenance"]["region_key"] = region_key
+ summary_candidates.append(summary_candidate)
+ for member in members:
+ claim = {
+ **member,
+ "inventory_schema_version": SLOW_INVENTORY_SCHEMA_VERSION,
+ "candidate_kind": "capsule_claim",
+ "capsule_summary_candidate_id": summary_candidate_id,
+ "capsule_summary_text": summary,
+ }
+ claim["provenance"] = {
+ **dict(member.get("provenance") or {}),
+ "candidate_kind": "capsule_claim",
+ "capsule_summary_candidate_id": summary_candidate_id,
+ }
+ if current_summary_contract:
+ claim["summary_contract_version"] = SLOW_SUMMARY_CONTRACT_VERSION
+ claim["provenance"]["summary_contract_version"] = (
+ SLOW_SUMMARY_CONTRACT_VERSION
+ )
+ if region_key:
+ claim["region_key"] = region_key
+ claim_candidates.append(claim)
+
+ inventory = sorted(
+ [*summary_candidates, *claim_candidates],
+ key=lambda item: (
+ 0 if item["candidate_kind"] == "capsule_summary" else 1,
+ str(item["candidate_id"]),
+ ),
+ )
+ candidate_ids = [str(item["candidate_id"]) for item in inventory]
+ if len(set(candidate_ids)) != len(candidate_ids):
+ raise RuntimeError("V4 Slow inventory candidate IDs are not unique")
+ _validate_active_capsule_partition(inventory)
+ return inventory, semantic_records
+
+
+def _slow_local_path(*, runtime_question: str, slow: Sequence[Mapping[str, Any]], slow_vectors: Any, models: Any, args: argparse.Namespace) -> tuple[list[dict[str, Any]], float, float]:
+ started = time.time()
+ if len(slow) != len(slow_vectors):
+ raise RuntimeError("V4 Slow inventory/vector count mismatch")
+ if any(
+ item.get("inventory_schema_version") != SLOW_INVENTORY_SCHEMA_VERSION
+ for item in slow
+ ):
+ raise RuntimeError("V4 Slow inventory schema mismatch")
+ scores = slow_vectors @ models.dense.encode_one(runtime_question)
+ summary_indexes = [
+ index
+ for index, item in enumerate(slow)
+ if item.get("candidate_kind") == "capsule_summary"
+ ]
+ claim_indexes = [
+ index
+ for index, item in enumerate(slow)
+ if item.get("candidate_kind") == "capsule_claim"
+ ]
+ if not summary_indexes or not claim_indexes:
+ raise RuntimeError(
+ "V4 Slow inventory requires both summary and claim candidates"
+ )
+ k = int(_arg(args, "slow_dense_k", 24))
+ summary_order = sorted(
+ summary_indexes, key=lambda index: (-float(scores[index]), index)
+ )
+ claim_order = sorted(
+ claim_indexes, key=lambda index: (-float(scores[index]), index)
+ )
+ selected_summaries = summary_order[: min(len(summary_order), k)]
+ selected_direct_claims = claim_order[: min(len(claim_order), k)]
+ claim_index_by_id = {
+ _clean(slow[index].get("candidate_id")): index for index in claim_indexes
+ }
+ summary_hit_by_claim: dict[str, dict[str, Any]] = {}
+ selected_claim_indexes = set(selected_direct_claims)
+ for summary_rank, summary_index in enumerate(selected_summaries):
+ summary = slow[summary_index]
+ summary_id = _clean(summary.get("candidate_id"))
+ children = list(summary.get("child_claim_candidate_ids") or [])
+ if not summary_id or not children:
+ raise RuntimeError("Slow summary candidate has no claim expansion mapping")
+ for raw_child_id in children:
+ child_id = _clean(raw_child_id)
+ child_index = claim_index_by_id.get(child_id)
+ if child_index is None:
+ raise RuntimeError(
+ f"Slow summary expansion references an unknown claim: {child_id}"
+ )
+ child = slow[child_index]
+ if (
+ _clean(child.get("capsule_summary_candidate_id")) != summary_id
+ or _clean(child.get("capsule_id"))
+ != _clean(summary.get("capsule_id"))
+ or child.get("revision") != summary.get("revision")
+ ):
+ raise RuntimeError(
+ f"Slow summary/claim expansion identity mismatch: {child_id}"
+ )
+ selected_claim_indexes.add(child_index)
+ previous = summary_hit_by_claim.get(child_id)
+ hit = {
+ "summary_candidate_id": summary_id,
+ "summary_text": _clean(summary.get("text")),
+ "summary_dense_score": float(scores[summary_index]),
+ "summary_dense_rank": summary_rank,
+ }
+ if previous is not None and previous != hit:
+ raise RuntimeError(
+ f"Slow claim is attached to multiple summary candidates: {child_id}"
+ )
+ summary_hit_by_claim[child_id] = hit
+
+ direct_rank_by_index = {
+ index: rank for rank, index in enumerate(claim_order)
+ }
+ ranked: list[dict[str, Any]] = []
+ for index in sorted(
+ selected_claim_indexes,
+ key=lambda item: (direct_rank_by_index[item], item),
+ ):
+ candidate = dict(slow[index])
+ candidate_id = _clean(candidate.get("candidate_id"))
+ summary_hit = summary_hit_by_claim.get(candidate_id)
+ is_direct = index in selected_direct_claims
+ if not is_direct and summary_hit is None:
+ raise RuntimeError("Slow claim entered the shortlist without a retrieval route")
+ candidate.update(
+ {
+ "slow_dense_score": float(scores[index]),
+ "slow_dense_rank": direct_rank_by_index[index],
+ "direct_claim_hit": is_direct,
+ "summary_expansion_hit": summary_hit is not None,
+ "summary_hit": dict(summary_hit) if summary_hit is not None else None,
+ "selection_routes": [
+ route
+ for route, enabled in (
+ ("direct_claim", is_direct),
+ ("capsule_summary", summary_hit is not None),
+ )
+ if enabled
+ ],
+ }
+ )
+ ranked.append(candidate)
+ if not ranked:
+ raise RuntimeError("slow local path produced an empty claim shortlist")
+ cross_started = time.time()
+ dense_elapsed = cross_started - started
+ cross_texts = [
+ "User memory summary: "
+ + _clean(item.get("capsule_summary_text"))
+ + "\nSpecific supported memory: "
+ + _clean(item.get("text"))
+ for item in ranked
+ ]
+ if any(
+ not _clean(item.get("capsule_summary_text")) or not _clean(item.get("text"))
+ for item in ranked
+ ):
+ raise RuntimeError("Slow claim shortlist lacks summary or claim text")
+ _, logits = models.encode_cross(runtime_question, cross_texts)
+ cross_elapsed = time.time() - cross_started
+ for item, score in zip(ranked, logits.detach().cpu().tolist()):
+ item["semantic_logit"] = float(score)
+ item["slow_retrieval_trace"] = {
+ "inventory_schema_version": SLOW_INVENTORY_SCHEMA_VERSION,
+ "direct_claim_hit": bool(item["direct_claim_hit"]),
+ "claim_candidate_id": _clean(item.get("candidate_id")),
+ "claim_dense_rank": int(item["slow_dense_rank"]),
+ "claim_dense_score": float(item["slow_dense_score"]),
+ "summary_expansion_hit": bool(item["summary_expansion_hit"]),
+ "summary_hit": item.get("summary_hit"),
+ "final_claim_cross_score": float(score),
+ "source_parents": [
+ dict(parent) for parent in list(item.get("source_parents") or [])
+ ],
+ }
+ ranked.sort(key=lambda item: (-float(item["semantic_logit"]), str(item.get("candidate_id", ""))))
+ return ranked, dense_elapsed, cross_elapsed
+
+
+def _unit_windows(unit: Mapping[str, Any], source_candidates: Sequence[Mapping[str, Any]], *, qid: str) -> list[dict[str, Any]]:
+ by_parent: dict[tuple[int, int], list[Mapping[str, Any]]] = {}
+ for candidate in source_candidates:
+ by_parent.setdefault((int(candidate["session_index"]), int(candidate["parent_chunk_index"])), []).append(candidate)
+ output: dict[tuple[int, int, int], dict[str, Any]] = {}
+
+ def add(candidate: Mapping[str, Any], role: str, capsule: Mapping[str, Any] | None = None) -> None:
+ key = (int(candidate["session_index"]), int(candidate["parent_chunk_index"]), int(candidate["subchunk_index"]))
+ item = output.setdefault(key, {**dict(candidate), "roles": [], "capsules": []})
+ if role not in item["roles"]:
+ item["roles"].append(role)
+ if capsule is not None and capsule not in item["capsules"]:
+ item["capsules"].append(dict(capsule))
+
+ if unit["unit_type"] == "source_window":
+ add(unit["source_candidate"], "source")
+ elif unit["unit_type"] == "fast_atomic":
+ fast_candidate = unit["fast_candidate"]
+ location = (int(fast_candidate["session_index"]), int(fast_candidate["parent_chunk_index"]))
+ matches = list(by_parent.get(location) or [])
+ if not matches:
+ raise RuntimeError(f"{qid}: fast candidate is not mapped to an immutable source window: {location}")
+ for candidate in matches:
+ add(candidate, "fast")
+ elif unit["unit_type"] == "slow_capsule":
+ capsule = unit["slow_candidate"]
+ for parent in capsule["source_parents"]:
+ location = (int(parent["session_index"]), int(parent["parent_chunk_index"]))
+ matches = [candidate for candidate in by_parent.get(location, []) if int(candidate["source_char_end"]) > int(parent["evidence_char_start"]) and int(candidate["source_char_start"]) < int(parent["evidence_char_end"])]
+ if not matches:
+ raise RuntimeError(f"{qid}: slow source_parent is unmapped during descent: {location}")
+ for candidate in matches:
+ add(candidate, "slow", capsule)
+ else:
+ raise RuntimeError(f"{qid}: unsupported V4 recall unit {unit['unit_type']}")
+ return list(output.values())
+
+
+def pack_recall_role_units(
+ units: Sequence[Mapping[str, Any]],
+ source_candidates: Sequence[Mapping[str, Any]],
+ *,
+ top_k: int,
+ qid: str,
+ return_stats: bool = False,
+ required_layers: Sequence[str] = (),
+ required_source_session_count: int = 1,
+) -> Any:
+ if top_k <= 0:
+ raise RuntimeError("top_k must be positive")
+ if required_source_session_count <= 0:
+ raise RuntimeError("required_source_session_count must be positive")
+ packed: list[tuple[Mapping[str, Any], list[dict[str, Any]]]] = []
+ used: set[tuple[int, int]] = set()
+ source_sessions: set[str] = set()
+ budget_excluded = 0
+ duplicate_units = 0
+ ordered_units = sorted(units, key=lambda unit: (-float(unit.get("priority_score", 0.0)), str(unit.get("layer", "")), str(unit.get("canonical_slot", ""))))
+ required = list(dict.fromkeys(str(layer) for layer in required_layers))
+ if any(layer not in {"source", "fast", "slow"} for layer in required):
+ raise RuntimeError(f"{qid}: required recall layer is invalid")
+ selected_unit_ids: set[int] = set()
+
+ def track_source_sessions(windows: Sequence[Mapping[str, Any]]) -> None:
+ for item in windows:
+ session_id = _clean(item.get("session_id"))
+ if session_id:
+ source_sessions.add(session_id)
+
+ def add_unit(
+ unit: Mapping[str, Any], *, required_layer: str | None = None
+ ) -> bool:
+ nonlocal budget_excluded, duplicate_units
+ windows = _unit_windows(unit, source_candidates, qid=qid)
+ unique = [item for item in windows if (int(item["session_index"]), int(item["parent_chunk_index"])) not in used]
+ if not unique:
+ duplicate_units += 1
+ # The physical window is already budgeted, but this layer still
+ # contributes ranking/provenance information to that window.
+ packed.append((unit, windows))
+ selected_unit_ids.add(id(unit))
+ if unit.get("layer") == "source":
+ track_source_sessions(windows)
+ return True
+ if len(unique) > top_k and not packed:
+ raise RuntimeError(f"{qid}: first atomic recall unit exceeds strict packing budget {top_k}")
+ if len(used) + len(unique) > top_k:
+ if required_layer is not None:
+ raise RuntimeError(
+ f"{qid}: required {required_layer} unit exceeds strict packing budget {top_k}"
+ )
+ budget_excluded += 1
+ return False
+ packed.append((unit, windows))
+ selected_unit_ids.add(id(unit))
+ used.update((int(item["session_index"]), int(item["parent_chunk_index"])) for item in unique)
+ if unit.get("layer") == "source":
+ track_source_sessions(windows)
+ return True
+
+ for layer in required:
+ candidates = [unit for unit in ordered_units if unit.get("layer") == layer]
+ if layer == "fast":
+ semantic = [
+ unit
+ for unit in candidates
+ if list((unit.get("fast_candidate") or {}).get("semantic_record_ids") or [])
+ ]
+ candidates = semantic or candidates
+ if not candidates:
+ raise RuntimeError(f"{qid}: required {layer} layer has no candidate unit")
+ add_unit(candidates[0], required_layer=layer)
+
+ if "source" in required and required_source_session_count > 1:
+ source_units = [unit for unit in ordered_units if unit.get("layer") == "source"]
+ for unit in source_units:
+ candidate = unit.get("source_candidate")
+ session_id = (
+ _clean(candidate.get("session_id"))
+ if isinstance(candidate, Mapping)
+ else ""
+ )
+ if not session_id or session_id in source_sessions:
+ continue
+ if len(used) >= top_k:
+ break
+ add_unit(unit)
+ if len(source_sessions) >= required_source_session_count:
+ break
+
+ for unit in ordered_units:
+ if id(unit) in selected_unit_ids:
+ continue
+ add_unit(unit)
+ if not packed:
+ raise RuntimeError(f"{qid}: recall role plan produced no packable evidence units")
+ if return_stats:
+ return packed, {
+ "budget_excluded_unit_count": budget_excluded,
+ "duplicate_unit_count": duplicate_units,
+ "required_layers": required,
+ "source_session_diversity_target": required_source_session_count,
+ "source_session_diversity_selected": len(source_sessions),
+ }
+ return packed
+
+
+def required_source_session_count(plan: Mapping[str, Any]) -> int:
+ """Reserve comparison coverage without inspecting benchmark labels."""
+ query_kind = _clean(plan.get("query_kind"))
+ temporal_focus = _clean(plan.get("temporal_focus"))
+ if temporal_focus in {"historical", "recent", "mixed"} and query_kind in {
+ "comparison",
+ "event",
+ "historical",
+ }:
+ return 3
+ return 1
+
+
+def _unit_base_score(unit: Mapping[str, Any]) -> float:
+ """Return the within-layer reciprocal rank, never a raw model score."""
+ return float(unit.get("within_layer_score", 1.0 / max(1, int(unit.get("layer_rank", 1)))))
+
+
+def _unit_contribution(unit: Mapping[str, Any]) -> dict[str, Any]:
+ fast_candidate = unit.get("fast_candidate")
+ active_semantic = bool(
+ unit.get("layer") == "fast"
+ and isinstance(fast_candidate, Mapping)
+ and list(fast_candidate.get("semantic_record_ids") or [])
+ )
+ return {
+ "layer": unit["layer"],
+ "role": unit["layer_role"],
+ "weight": unit["layer_weight"],
+ "normalized_priority": unit["normalized_priority"],
+ "within_layer_score": unit["within_layer_score"],
+ "priority_score": unit["priority_score"],
+ "active_semantic": active_semantic,
+ }
+
+
+def _fast_memory_attachments(unit: Mapping[str, Any]) -> list[dict[str, Any]]:
+ candidate = unit.get("fast_candidate")
+ if unit.get("layer") != "fast" or not isinstance(candidate, Mapping):
+ return []
+ attachments: list[dict[str, Any]] = []
+ for memory in list(candidate.get("semantic_memories") or []):
+ if not isinstance(memory, Mapping):
+ raise RuntimeError("fast semantic memory attachment is not an object")
+ memory_id = _clean(memory.get("memory_id"))
+ slot = _clean(memory.get("canonical_slot"))
+ text = _clean(memory.get("text"))
+ provenance = memory.get("provenance")
+ source_parent = memory.get("source_parent")
+ if (
+ not memory_id
+ or not slot
+ or not text
+ or not isinstance(provenance, Mapping)
+ or not isinstance(source_parent, Mapping)
+ ):
+ raise RuntimeError("fast semantic memory attachment is incomplete")
+ attachments.append(
+ {
+ "role": "fast_context",
+ "memory_id": memory_id,
+ "canonical_slot": slot,
+ "text": text,
+ "record_state": _clean(memory.get("record_state")),
+ "memory_type": _clean(memory.get("memory_type")),
+ "durability": _clean(memory.get("durability")),
+ "temporal_status": _clean(memory.get("temporal_status")),
+ "source_parent": dict(source_parent),
+ "provenance": dict(provenance),
+ }
+ )
+ return attachments
+
+
+def _slow_memory_contexts(capsules: Sequence[Mapping[str, Any]]) -> list[dict[str, Any]]:
+ contexts: list[dict[str, Any]] = []
+ for capsule in capsules:
+ claims = list(capsule.get("claims") or [])
+ if len(claims) != 1 or not isinstance(claims[0], Mapping):
+ raise RuntimeError("slow retrieval candidate must contain exactly one claim")
+ claim = claims[0]
+ provenance = capsule.get("provenance")
+ context = {
+ "role": "slow_context",
+ "capsule_id": _clean(capsule.get("capsule_id")),
+ "revision": int(capsule.get("revision", 0)),
+ "status": _clean(capsule.get("status")),
+ "claim_id": _clean(claim.get("claim_id")),
+ "canonical_slot": _clean(capsule.get("canonical_slot")),
+ "capsule_summary": _clean(capsule.get("capsule_summary_text")),
+ "capsule_summary_candidate_id": _clean(
+ capsule.get("capsule_summary_candidate_id")
+ ),
+ "claim_text": _clean(capsule.get("text")),
+ "support": list(claim.get("support") or []),
+ "counterevidence": list(claim.get("counterevidence") or []),
+ "source_parents": [dict(item) for item in list(capsule.get("source_parents") or [])],
+ "provenance": dict(provenance) if isinstance(provenance, Mapping) else {},
+ "retrieval_trace": dict(capsule.get("slow_retrieval_trace") or {}),
+ }
+ if (
+ not context["capsule_id"]
+ or context["revision"] < 1
+ or not context["claim_id"]
+ or not context["canonical_slot"]
+ or not context["capsule_summary"]
+ or not context["capsule_summary_candidate_id"]
+ or not context["claim_text"]
+ or not context["source_parents"]
+ or not context["provenance"]
+ or not context["retrieval_trace"]
+ ):
+ raise RuntimeError("slow retrieval claim context is incomplete")
+ contexts.append(context)
+ return contexts
+
+
+def _mark_newer_fast_overrides(windows: Sequence[dict[str, Any]]) -> None:
+ slow_by_slot: dict[str, dict[str, Any]] = {}
+ for window in windows:
+ for context in list(window.get("memory_contexts") or []):
+ if not isinstance(context, Mapping):
+ continue
+ slot = _clean(context.get("canonical_slot"))
+ if not slot:
+ continue
+ parents = [
+ parent
+ for parent in list(context.get("source_parents") or [])
+ if isinstance(parent, Mapping)
+ ]
+ latest = max(
+ (
+ int(parent.get("session_index", -1)),
+ int(parent.get("parent_chunk_index", parent.get("message_index", -1))),
+ )
+ for parent in parents
+ )
+ state = slow_by_slot.setdefault(slot, {"latest": latest, "support": set()})
+ state["latest"] = max(state["latest"], latest)
+ state["support"].update(_clean(value) for value in context.get("support") or [])
+ for window in windows:
+ for attachment in list(window.get("attachments") or []):
+ if not isinstance(attachment, dict):
+ continue
+ slot = _clean(attachment.get("canonical_slot"))
+ slow_state = slow_by_slot.get(slot)
+ parent = attachment.get("source_parent")
+ if not slow_state or not isinstance(parent, Mapping):
+ continue
+ location = (
+ int(parent.get("session_index", -1)),
+ int(parent.get("parent_chunk_index", parent.get("message_index", -1))),
+ )
+ if (
+ location > slow_state["latest"]
+ and _clean(attachment.get("memory_id")) not in slow_state["support"]
+ ):
+ attachment["role"] = "override"
+ attachment["precedence"] = "newer_fast_evidence"
+
+
+def _selected_layer_window_counts(
+ windows: Sequence[Mapping[str, Any]],
+) -> dict[str, int]:
+ counts = {"source": 0, "fast": 0, "slow": 0}
+ for window in windows:
+ metadata = window.get("retrieval_metadata")
+ contributions = (
+ metadata.get("layer_contributions") if isinstance(metadata, Mapping) else []
+ )
+ layers = {
+ _clean(item.get("layer"))
+ for item in contributions
+ if isinstance(item, Mapping)
+ }
+ for layer in layers:
+ if layer in counts:
+ counts[layer] += 1
+ return counts
+
+
+def _order_evidence_windows(
+ evidence_by_location: Mapping[tuple[int, int, int], dict[str, Any]],
+ *,
+ conflict_policy: str,
+) -> list[dict[str, Any]]:
+ windows = list(evidence_by_location.values())
+ for original_order, item in enumerate(windows):
+ contributions = list(
+ item["retrieval_metadata"].get("layer_contributions") or []
+ )
+ active_fast = [
+ value
+ for value in contributions
+ if value.get("layer") == "fast" and value.get("active_semantic") is True
+ ]
+ item["retrieval_metadata"]["active_fast_support"] = bool(active_fast)
+ item["retrieval_metadata"]["fast_within_layer_score"] = max(
+ (float(value["within_layer_score"]) for value in active_fast),
+ default=0.0,
+ )
+ item["retrieval_metadata"]["packing_priority_score"] = max(
+ (float(value["priority_score"]) for value in contributions),
+ default=0.0,
+ )
+ item["_original_order"] = original_order
+ if conflict_policy == "prefer_recent":
+ windows.sort(
+ key=lambda item: (
+ 0 if item["retrieval_metadata"]["active_fast_support"] else 1,
+ -int(item["session_index"])
+ if item["retrieval_metadata"]["active_fast_support"]
+ else 0,
+ -int(item["parent_chunk_index"])
+ if item["retrieval_metadata"]["active_fast_support"]
+ else 0,
+ -float(item["retrieval_metadata"]["fast_within_layer_score"]),
+ int(item["_original_order"]),
+ )
+ )
+
+ # Retrieval ranks chunks, while the answer layer consumes dialogue. Group
+ # the already-selected chunks at the immutable session occurrence boundary,
+ # then restore source order within each session. Membership and Top-K do not
+ # change, so this cannot manufacture or hide evidence.
+ sessions: dict[int, dict[str, Any]] = {}
+ for pre_session_rank, item in enumerate(windows, start=1):
+ session_index = int(item["session_index"])
+ state = sessions.setdefault(
+ session_index,
+ {
+ "rrf": 0.0,
+ "count": 0,
+ "best_rank": pre_session_rank,
+ "active_fast": False,
+ },
+ )
+ state["rrf"] += 1.0 / float(pre_session_rank)
+ state["count"] += 1
+ state["best_rank"] = min(int(state["best_rank"]), pre_session_rank)
+ state["active_fast"] = bool(
+ state["active_fast"]
+ or item["retrieval_metadata"]["active_fast_support"]
+ )
+ item["retrieval_metadata"]["pre_session_order_rank"] = pre_session_rank
+
+ def session_key(session_index: int) -> tuple[Any, ...]:
+ state = sessions[session_index]
+ recent_prefix: tuple[Any, ...] = ()
+ if conflict_policy == "prefer_recent":
+ recent_prefix = (
+ 0 if state["active_fast"] else 1,
+ -session_index if state["active_fast"] else 0,
+ )
+ return (
+ *recent_prefix,
+ -float(state["rrf"]),
+ int(state["best_rank"]),
+ session_index,
+ )
+
+ session_order = sorted(sessions, key=session_key)
+ session_rank = {
+ session_index: rank for rank, session_index in enumerate(session_order, start=1)
+ }
+ windows.sort(
+ key=lambda item: (
+ session_rank[int(item["session_index"])],
+ int(item["parent_chunk_index"]),
+ int(item.get("subchunk_index", 0)),
+ int(item["retrieval_metadata"]["pre_session_order_rank"]),
+ )
+ )
+ for rank, item in enumerate(windows, start=1):
+ state = sessions[int(item["session_index"])]
+ item.pop("_original_order", None)
+ item["rank"] = rank
+ item["retrieval_metadata"].update(
+ {
+ "session_ordering_policy": SESSION_ORDERING_POLICY,
+ "session_order_rank": session_rank[int(item["session_index"])],
+ "session_support_rrf": round(float(state["rrf"]), 8),
+ "session_selected_window_count": int(state["count"]),
+ }
+ )
+ return windows
+
+
+def _attach_source_group_context(
+ evidence_windows: Sequence[Mapping[str, Any]],
+ source_inventory: Sequence[Mapping[str, Any]],
+ *,
+ max_parent_distance: int = 2,
+ max_context_members: int = 2,
+ max_context_chars: int = 3600,
+) -> tuple[list[dict[str, Any]], dict[str, int]]:
+ """Attach nearby immutable source parents without changing Top-K membership.
+
+ Retrieval ranks source parents independently, but a fact can depend on a
+ nearby turn in the same session. Context parents are assigned once to the
+ nearest selected parent and remain separately identifiable by provenance.
+ """
+ if max_parent_distance < 0 or max_context_members < 0 or max_context_chars < 0:
+ raise RuntimeError("source group context limits must be non-negative")
+ output: list[dict[str, Any]] = []
+ for item in evidence_windows:
+ current = dict(item)
+ current["retrieval_metadata"] = dict(item.get("retrieval_metadata") or {})
+ current["source_group_id"] = (
+ f"source-group::{item['session_id']}:{int(item['parent_chunk_index'])}"
+ )
+ current["source_group_context"] = []
+ output.append(current)
+
+ selected_locations = {
+ (int(item["session_index"]), int(item["parent_chunk_index"]))
+ for item in output
+ }
+ selected_by_session: dict[int, list[tuple[int, int]]] = {}
+ for output_index, item in enumerate(output):
+ selected_by_session.setdefault(int(item["session_index"]), []).append(
+ (int(item["parent_chunk_index"]), output_index)
+ )
+
+ candidates: list[tuple[int, int, int, Mapping[str, Any]]] = []
+ seen_inventory: set[tuple[int, int]] = set()
+ for item in source_inventory:
+ location = (int(item["session_index"]), int(item["parent_chunk_index"]))
+ if location in seen_inventory:
+ continue
+ seen_inventory.add(location)
+ if location in selected_locations or location[0] not in selected_by_session:
+ continue
+ nearest = min(
+ (
+ abs(location[1] - selected_parent),
+ output_index,
+ )
+ for selected_parent, output_index in selected_by_session[location[0]]
+ )
+ distance, output_index = nearest
+ if distance <= max_parent_distance:
+ candidates.append((distance, output_index, location[1], item))
+
+ attached_chars = [0 for _ in output]
+ attached_count = 0
+ for distance, output_index, _, item in sorted(
+ candidates,
+ key=lambda value: (
+ value[0],
+ value[1],
+ value[2],
+ str(value[3].get("source_record_id") or ""),
+ ),
+ ):
+ target = output[output_index]
+ members = target["source_group_context"]
+ text = str(item.get("text") or "")
+ if not text or len(members) >= max_context_members:
+ continue
+ if attached_chars[output_index] + len(text) > max_context_chars:
+ continue
+ members.append(
+ {
+ "relationship": "session_neighbor",
+ "parent_distance": distance,
+ "session_id": str(item["session_id"]),
+ "session_index": int(item["session_index"]),
+ "parent_chunk_index": int(item["parent_chunk_index"]),
+ "source_record_id": str(item.get("source_record_id") or ""),
+ "source_char_start": int(item.get("source_char_start", 0)),
+ "source_char_end": int(item.get("source_char_end", len(text))),
+ "historical_date": _clean(item.get("historical_date")),
+ "timestamp": _clean(item.get("timestamp")),
+ "message_role": _clean(
+ item.get("message_role") or item.get("role")
+ ),
+ "text": text,
+ }
+ )
+ attached_chars[output_index] += len(text)
+ attached_count += 1
+
+ groups_with_context = 0
+ for item in output:
+ count = len(item["source_group_context"])
+ item["retrieval_metadata"]["source_group_context_count"] = count
+ item["retrieval_metadata"]["source_group_context_chars"] = sum(
+ len(member["text"]) for member in item["source_group_context"]
+ )
+ groups_with_context += int(count > 0)
+ return output, {
+ "source_group_count": len(output),
+ "groups_with_context_count": groups_with_context,
+ "attached_context_parent_count": attached_count,
+ }
+
+
+def retrieve_one(
+ row: Mapping[str, Any],
+ *,
+ args: argparse.Namespace,
+ harness: Any,
+ models: Any,
+ planner: DeepSeekFlashRecallRolePlanner | None = None,
+ route_override: Mapping[str, Any] | None = None,
+ route_override_metadata: Mapping[str, Any] | None = None,
+ planner_decision_callback: Callable[
+ [Mapping[str, Any], Mapping[str, Any], Mapping[str, Any]], None
+ ]
+ | None = None,
+ graph_adapter_cache: dict[tuple[str, str], Any] | None = None,
+) -> tuple[dict[str, Any], dict[str, Any]]:
+ started = time.time()
+ qid, question, question_date = _clean(row.get("question_id")), _clean(row.get("question")), _clean(row.get("question_date"))
+ if not qid or not question:
+ raise RuntimeError("online retrieval manifest row lacks qid or question")
+ if any(key in row for key in ("answer", "gold_answer", "answer_session_ids", "labels", "supervision", "benchmark", "expected_answer")):
+ raise RuntimeError(f"{qid}: runtime retrieval manifest contains forbidden evaluation labels")
+ (
+ composition_mode,
+ execution_lane,
+ packing_budget_mode,
+ configured_top_k,
+ ) = _validated_runtime_route(args, label=qid)
+ v3 = _v3()
+ if int(_arg(args, "slow_dense_k", 24)) <= 0:
+ raise RuntimeError("slow_dense_k must be positive")
+ db_path, scope_id, index_path = Path(row["db_path"]).resolve(), _clean(row.get("scope_id")), Path(row["index_path"]).resolve()
+ fast, fast_vectors, slow, slow_vectors, semantic_records, payload = load_online_index(
+ index_path, db_path, scope_id
+ )
+ semantic_records = _hydrate_fast_semantic_records(
+ db_path, scope_id, semantic_records
+ )
+ counts_before = v3.scope_counts(db_path, scope_id)
+ if counts_before["records"] != int(payload["graph_counts_at_index"]["records"]):
+ raise RuntimeError(f"{qid}: graph records changed after online index creation")
+ graph_fingerprint = v3.scope_fingerprint(db_path, scope_id)
+ if graph_fingerprint != _clean(payload.get("graph_fingerprint")):
+ raise RuntimeError(f"{qid}: graph fingerprint changed after online index creation")
+ raw_recent_dialogue = v3.load_recent_dialogue_context(
+ db_path,
+ scope_id,
+ current_query=question,
+ limit=RECENT_DIALOGUE_MAX_TURNS,
+ )
+ recent_dialogue, recent_dialogue_projection = project_recent_dialogue(
+ raw_recent_dialogue
+ )
+ available_layers = {
+ "source": {"available": bool(fast), "candidate_count": len(fast)},
+ "fast": {"available": bool(fast), "candidate_count": len(fast)},
+ "slow": {
+ "available": bool(slow),
+ "capsule_count": int(payload.get("slow_capsule_head_count", 0)),
+ "summary_candidate_count": int(
+ payload.get("slow_summary_candidate_count", 0)
+ ),
+ "claim_candidate_count": int(
+ payload.get("slow_claim_candidate_count", 0)
+ ),
+ },
+ }
+ if route_override is None:
+ if planner is None:
+ raise RuntimeError("V4 recall planner is required")
+ raw_plan, planner_metadata = planner.plan(
+ query=question,
+ question_date=question_date or "unknown",
+ recent_dialogue=recent_dialogue,
+ available_layers=available_layers,
+ )
+ else:
+ raw_plan = route_override
+ planner_metadata = dict(
+ route_override_metadata
+ or {
+ "physical_api_call": False,
+ "physical_api_calls": 0,
+ "stage": "recall_planner",
+ "status": "route_override",
+ "planner_version": "route_override",
+ "prompt_version": "route_override",
+ }
+ )
+ plan = validate_recall_role_plan(raw_plan)
+ if planner_decision_callback is not None:
+ planner_decision_callback(
+ plan,
+ planner_metadata,
+ {
+ "graph_fingerprint": graph_fingerprint,
+ "recent_dialogue_projection": recent_dialogue_projection,
+ "available_layers": available_layers,
+ },
+ )
+ runtime_question = f"{plan['resolved_query']}\nQuestion date: {question_date}" if question_date else plan["resolved_query"]
+
+ # These three calls are independent of all role and weight values.
+ if fast:
+ source_candidates, dense_scores, dense_rank, source_elapsed, source_cross_elapsed = _source_local_path(runtime_question=runtime_question, fast=fast, fast_vectors=fast_vectors, models=models, args=args)
+ parent_representatives = []
+ seen_inventory_parents: set[tuple[int, int]] = set()
+ for candidate in fast:
+ parent_key = (
+ int(candidate["session_index"]),
+ int(candidate["parent_chunk_index"]),
+ )
+ if parent_key not in seen_inventory_parents:
+ seen_inventory_parents.add(parent_key)
+ parent_representatives.append(candidate)
+ source_evidence_inventory = _collapse_source_parents(
+ parent_representatives, fast
+ )
+ fast_ranked, graph_metadata, graph_elapsed, fast_cross_elapsed = _graph_fast_path(qid=qid, runtime_question=runtime_question, fast=fast, dense_scores=dense_scores, dense_rank=dense_rank, semantic_records=semantic_records, args=args, harness=harness, scope_id=scope_id, db_path=db_path, graph_adapter_cache=graph_adapter_cache, models=models)
+ dense_elapsed = source_elapsed
+ else:
+ source_candidates, source_evidence_inventory, graph_metadata, fast_ranked, source_elapsed, dense_elapsed, source_cross_elapsed, graph_elapsed, fast_cross_elapsed = [], [], {"skipped": True}, [], 0.0, 0.0, 0.0, 0.0, 0.0
+ if slow:
+ slow_ranked, slow_elapsed, slow_cross_elapsed = _slow_local_path(runtime_question=runtime_question, slow=slow, slow_vectors=slow_vectors, models=models, args=args)
+ else:
+ slow_ranked, slow_elapsed, slow_cross_elapsed = [], 0.0, 0.0
+ composition_plan = plan
+ composition_fast = fast_ranked
+ composition_slow = slow_ranked
+ if composition_mode == "source-only-diagnostic":
+ composition_plan = {
+ **plan,
+ "layers": {
+ "source": {"role": "primary", "weight": 1.0},
+ "fast": {"role": "context", "weight": 0.0},
+ "slow": {"role": "context", "weight": 0.0},
+ },
+ }
+ composition_fast = []
+ composition_slow = []
+ try:
+ units = apply_recall_role_plan(
+ composition_plan,
+ source_candidates,
+ composition_fast,
+ composition_slow,
+ )
+ except RecallPlannerError as exc:
+ raise RuntimeError(f"{qid}: invalid V4 recall role composition: {exc}") from exc
+ units.sort(key=lambda unit: (-float(unit["priority_score"]), str(unit.get("layer", "")), str(unit.get("canonical_slot", ""))))
+ packing_budget, packing_budget_decision = resolve_packing_budget(
+ plan,
+ mode=packing_budget_mode,
+ fixed_k=configured_top_k,
+ simple_k=int(_arg(args, "adaptive_simple_k", 8)),
+ standard_k=int(_arg(args, "adaptive_standard_k", 12)),
+ complex_k=int(_arg(args, "adaptive_complex_k", 16)),
+ )
+ fast_semantic_shortlist_count = sum(
+ int(bool(list(candidate.get("semantic_memories") or [])))
+ for candidate in fast_ranked
+ )
+ required_layers: list[str] = []
+ if source_candidates:
+ required_layers.append("source")
+ if composition_mode == "layered":
+ if fast_ranked:
+ required_layers.append("fast")
+ if slow_ranked:
+ required_layers.append("slow")
+ packed_units, packing_stats = pack_recall_role_units(
+ units,
+ source_evidence_inventory,
+ top_k=packing_budget,
+ qid=qid,
+ return_stats=True,
+ required_layers=required_layers,
+ required_source_session_count=required_source_session_count(plan),
+ )
+
+ evidence_by_location: dict[tuple[int, int, int], dict[str, Any]] = {}
+ for unit, entries in packed_units:
+ contribution = _unit_contribution(unit)
+ unit_attachments = _fast_memory_attachments(unit)
+ for entry in entries:
+ capsules = list(entry.get("capsules") or [])
+ contexts = _slow_memory_contexts(capsules)
+ location = (int(entry["session_index"]), int(entry["parent_chunk_index"]), int(entry["subchunk_index"]))
+ fast_candidate = unit.get("fast_candidate")
+ semantic_record_ids = (
+ list(fast_candidate.get("semantic_record_ids") or [])
+ if isinstance(fast_candidate, Mapping)
+ else []
+ )
+ unit_provenance = (
+ fast_candidate.get("provenance")
+ if isinstance(fast_candidate, Mapping)
+ else entry.get("provenance")
+ )
+ raw_provenance = (
+ [item.get("provenance") for item in capsules]
+ if capsules
+ else [unit_provenance]
+ )
+ provenance = [dict(item) for item in raw_provenance if isinstance(item, Mapping)]
+ candidate = {"memory_id": entry.get("candidate_id"), "source_record_id": entry.get("source_record_id"), "source_char_start": entry.get("source_char_start"), "source_char_end": entry.get("source_char_end"), "session_id": entry["session_id"], "session_index": entry["session_index"], "parent_chunk_index": entry["parent_chunk_index"], "subchunk_index": entry["subchunk_index"], "historical_date": _clean(entry.get("historical_date")), "timestamp": _clean(entry.get("timestamp")), "message_role": _clean(entry.get("message_role") or entry.get("role")), "rank": 0, "score": entry.get("score", entry.get("semantic_logit")), "semantic_logit": entry.get("semantic_logit"), "channels": entry.get("channels"), "text": entry["text"], "unit_type": unit["unit_type"], "unit_types": [unit["unit_type"]], "canonical_slot": unit["canonical_slot"], "canonical_slots": [unit["canonical_slot"]], "role": list(entry["roles"]), "provenance": provenance, "semantic_record_ids": semantic_record_ids, "memory_contexts": contexts, "attachments": [dict(item) for item in unit_attachments], "retrieval_metadata": {"plan_layer": unit["layer"], "plan_role": unit["layer_role"], "plan_weight": unit["layer_weight"], "role_prior": unit["role_prior"], "normalized_priority": unit["normalized_priority"], "within_layer_score": unit["within_layer_score"], "priority_score": unit["priority_score"], "layer_contributions": [contribution]}}
+ candidate["scope_id"] = scope_id
+ candidate["db_path"] = str(db_path)
+ existing = evidence_by_location.get(location)
+ if existing is None:
+ evidence_by_location[location] = candidate
+ else:
+ for field in ("unit_types", "canonical_slots", "role", "provenance", "semantic_record_ids", "memory_contexts", "attachments"):
+ for item in candidate[field]:
+ if item not in existing[field]:
+ existing[field].append(item)
+ if contribution not in existing["retrieval_metadata"]["layer_contributions"]:
+ existing["retrieval_metadata"]["layer_contributions"].append(contribution)
+ existing["retrieval_metadata"][f"{unit['layer']}_weight"] = unit["layer_weight"]
+ _mark_newer_fast_overrides(list(evidence_by_location.values()))
+ evidence_windows = _order_evidence_windows(
+ evidence_by_location,
+ conflict_policy=plan["conflict_policy"],
+ )
+ evidence_windows, source_group_stats = _attach_source_group_context(
+ evidence_windows,
+ source_evidence_inventory,
+ )
+ selected_layer_window_counts = _selected_layer_window_counts(evidence_windows)
+ candidate_paths_executed = {
+ "source": bool(fast),
+ "fast": bool(fast),
+ "slow": bool(slow),
+ }
+ evidence = {
+ "schema_version": v3.SCHEMA_VERSION,
+ "runtime_schema_version": RUNTIME_SCHEMA_VERSION,
+ "question_id": qid,
+ "question": question,
+ "question_date": question_date,
+ "question_type": _clean(row.get("question_type")),
+ "selected_session_ids": list(
+ dict.fromkeys(str(item["session_id"]) for item in evidence_windows)
+ ),
+ "recall_plan": plan,
+ "retrieval_contract": {
+ "schema_version": RETRIEVAL_CONTRACT_SCHEMA,
+ "execution_lane": execution_lane,
+ "composition_mode": composition_mode,
+ "inventory_counts": {
+ "source": len(source_candidates),
+ "fast": len(fast_ranked),
+ "fast_semantic": fast_semantic_shortlist_count,
+ "slow_capsule_heads": int(
+ payload.get("slow_capsule_head_count", 0)
+ ),
+ "slow_summaries": int(
+ payload.get("slow_summary_candidate_count", 0)
+ ),
+ "slow_claims": int(payload.get("slow_claim_candidate_count", 0)),
+ "slow_ranked_claims": len(slow_ranked),
+ "slow": len(slow_ranked),
+ },
+ "candidate_paths_executed": candidate_paths_executed,
+ "required_selected_layers": required_layers,
+ "selected_layer_window_counts": selected_layer_window_counts,
+ "packing_budget_mode": packing_budget_decision["mode"],
+ "packing_budget": packing_budget,
+ "source_coverage_trace_k": SOURCE_COVERAGE_TRACE_K,
+ "final_window_count": len(evidence_windows),
+ "source_session_diversity_target": int(
+ packing_stats["source_session_diversity_target"]
+ ),
+ "source_session_diversity_selected": int(
+ packing_stats["source_session_diversity_selected"]
+ ),
+ },
+ "evidence_windows": evidence_windows,
+ }
+ debug = {
+ "question_id": qid,
+ "scope_id": scope_id,
+ "db_path": str(db_path),
+ "index_path": str(index_path),
+ "recall_plan": plan,
+ "planner_resolved_query": plan["resolved_query"],
+ "planner": planner_metadata,
+ "recent_dialogue_count": len(recent_dialogue),
+ "recent_dialogue_projection": recent_dialogue_projection,
+ "cross_layer_weighted_fusion": False,
+ "execution_lane": execution_lane,
+ "composition_mode": composition_mode,
+ "graph": graph_metadata,
+ "candidate_paths_executed": candidate_paths_executed,
+ "required_selected_layers": required_layers,
+ "selected_layer_window_counts": selected_layer_window_counts,
+ "normalized_layer_priority": normalized_layer_priorities(plan),
+ "inventory_count": len(fast) + len(slow),
+ "union_count": len(fast_ranked) + len(slow_ranked),
+ "source_inventory_count": len(fast),
+ "fast_inventory_count": len(fast),
+ "slow_capsule_count": int(payload.get("slow_capsule_head_count", 0)),
+ "slow_capsule_head_count": int(
+ payload.get("slow_capsule_head_count", 0)
+ ),
+ "slow_summary_candidate_count": int(
+ payload.get("slow_summary_candidate_count", 0)
+ ),
+ "slow_claim_candidate_count": int(
+ payload.get("slow_claim_candidate_count", 0)
+ ),
+ "slow_summary_hit_count": sum(
+ int(bool(item.get("summary_expansion_hit"))) for item in slow_ranked
+ ),
+ "slow_direct_claim_hit_count": sum(
+ int(bool(item.get("direct_claim_hit"))) for item in slow_ranked
+ ),
+ "slow_retrieval_trace": [
+ dict(item.get("slow_retrieval_trace") or {}) for item in slow_ranked
+ ],
+ "slow_dense_k": int(_arg(args, "slow_dense_k", 24)),
+ "fast_semantic_record_count": len(semantic_records),
+ "fast_semantic_shortlist_count": fast_semantic_shortlist_count,
+ "fast_semantic_state_policy": payload.get("fast_semantic_state_policy"),
+ "source_candidate_count": len(source_candidates),
+ "source_coverage_trace_k": SOURCE_COVERAGE_TRACE_K,
+ "source_top24_candidates": source_coverage_trace(source_candidates),
+ "packing_budget_decision": packing_budget_decision,
+ "source_candidate_pool_trace": source_coverage_trace(
+ source_candidates, limit=len(source_candidates)
+ ) if source_candidates else [],
+ "fast_shortlist_count": len(fast_ranked),
+ "slow_shortlist_count": len(slow_ranked),
+ "planned_unit_count": len(units),
+ "packed_unit_count": len(packed_units),
+ "packing_budget_top_k": packing_budget,
+ "budget_excluded_unit_count": int(packing_stats["budget_excluded_unit_count"]),
+ "duplicate_unit_count": int(packing_stats["duplicate_unit_count"]),
+ "selected_count": len(evidence_windows),
+ "source_group_stats": source_group_stats,
+ "active_fast_supported_selected_count": sum(
+ int(bool(item["retrieval_metadata"]["active_fast_support"]))
+ for item in evidence_windows
+ ),
+ "conflict_policy_rerank_applied": plan["conflict_policy"]
+ == "prefer_recent",
+ "session_coherent_ordering": True,
+ "session_ordering_policy": SESSION_ORDERING_POLICY,
+ "atomic_unit_packing": True,
+ "restart_boundary_verified": True,
+ "graph_counts_before_query": counts_before,
+ "graph_fingerprint": graph_fingerprint,
+ "checkpoint": str(getattr(models, "checkpoint_path", "")),
+ "checkpoint_sha256": getattr(models, "checkpoint_sha256", ""),
+ "cross_model_revision": (
+ getattr(models, "cross_manifest", {}).get("revision")
+ if isinstance(getattr(models, "cross_manifest", {}), Mapping)
+ else None
+ ),
+ "latency_sec": {
+ "graph": round(graph_elapsed, 4),
+ "source_dense_cross": round(source_elapsed + source_cross_elapsed, 4),
+ "fast_graph_fusion": round(graph_elapsed + fast_cross_elapsed, 4),
+ "slow_dense_cross": round(slow_elapsed + slow_cross_elapsed, 4),
+ "dense": round(dense_elapsed, 4),
+ "cross": round(
+ source_cross_elapsed + fast_cross_elapsed + slow_cross_elapsed, 4
+ ),
+ "total": round(time.time() - started, 4),
+ },
+ }
+ return evidence, debug
+
+
+def _atomic_write(path: Path, text: str) -> None:
+ temporary = path.with_name(f".{path.name}.tmp.{os.getpid()}.{time.time_ns()}")
+ temporary.write_text(text, encoding="utf-8")
+ os.replace(temporary, path)
+
+
+def _atomic_write_json(path: Path, value: Mapping[str, Any]) -> None:
+ _atomic_write(
+ path,
+ json.dumps(dict(value), ensure_ascii=False, indent=2, sort_keys=True) + "\n",
+ )
+
+
+def _row_identity(row: Mapping[str, Any]) -> dict[str, Any]:
+ normalized = dict(row)
+ for key in ("db_path", "index_path"):
+ if key in normalized:
+ normalized[key] = str(Path(str(normalized[key])).absolute())
+ return normalized
+
+
+def _row_artifact_path(directory: Path, index: int, qid: str) -> Path:
+ return directory / f"row_{index:06d}_{hashlib.sha256(qid.encode('utf-8')).hexdigest()[:12]}.json"
+
+
+def _read_json_object(path: Path) -> dict[str, Any]:
+ try:
+ value = json.loads(path.read_text(encoding="utf-8"))
+ except json.JSONDecodeError as exc:
+ raise RuntimeError(f"invalid JSON artifact: {path}") from exc
+ if not isinstance(value, Mapping):
+ raise RuntimeError(f"JSON artifact is not an object: {path}")
+ return dict(value)
+
+
+def _load_persisted_retrieval_audit(
+ *,
+ row: Mapping[str, Any],
+ out_dir: Path,
+ graph_fingerprint: str,
+) -> dict[str, Any] | None:
+ db_path = Path(str(row["db_path"])).resolve()
+ scope_id = _clean(row.get("scope_id"))
+ qid = _clean(row.get("question_id"))
+ operation_id = _v3().layered_retrieval_operation_id(out_dir, scope_id, qid)
+ with closing(sqlite3.connect(db_path)) as connection:
+ table = connection.execute(
+ "SELECT 1 FROM sqlite_master WHERE type='table' AND name='audit_retrieval_log'"
+ ).fetchone()
+ if table is None:
+ return None
+ raw_rows = connection.execute(
+ 'SELECT event_index,payload_json FROM "audit_retrieval_log" '
+ "WHERE scope_id=? ORDER BY event_index",
+ (scope_id,),
+ ).fetchall()
+ matches: list[tuple[int, dict[str, Any]]] = []
+ for event_index, raw_payload in raw_rows:
+ try:
+ payload = json.loads(raw_payload)
+ except (TypeError, json.JSONDecodeError) as exc:
+ raise RuntimeError(
+ f"{qid}: invalid persisted retrieval audit at event {event_index}"
+ ) from exc
+ if not isinstance(payload, Mapping):
+ raise RuntimeError(
+ f"{qid}: persisted retrieval audit at event {event_index} is not an object"
+ )
+ if operation_id in {
+ _clean(payload.get("operation_id")),
+ _clean(payload.get("idempotency_key")),
+ }:
+ matches.append((int(event_index), dict(payload)))
+ if not matches:
+ return None
+ if len(matches) != 1:
+ raise RuntimeError(f"{qid}: duplicate persisted retrieval audit operation")
+ event_index, payload = matches[0]
+ expected = {
+ "event_kind": "tmcra.v3.layered_retrieval",
+ "operation_id": operation_id,
+ "question_id": qid,
+ "query": _clean(row.get("question")),
+ "question_date": _clean(row.get("question_date")),
+ "graph_fingerprint": graph_fingerprint,
+ }
+ for key, value in expected.items():
+ if _clean(payload.get(key)) != value:
+ raise RuntimeError(
+ f"{qid}: persisted retrieval audit {key} does not match the frozen query"
+ )
+ if payload.get("runtime_input_has_gold") is not False:
+ raise RuntimeError(f"{qid}: persisted retrieval audit is not gold-free")
+ evidence_sha256 = _clean(payload.get("evidence_sha256"))
+ if len(evidence_sha256) != 64 or any(
+ char not in "0123456789abcdef" for char in evidence_sha256.lower()
+ ):
+ raise RuntimeError(f"{qid}: persisted retrieval audit lacks evidence_sha256")
+ payload["recall_plan"] = validate_recall_role_plan(payload.get("recall_plan"))
+ payload["_event_index"] = event_index
+ payload["_payload_sha256"] = _digest(
+ {key: value for key, value in payload.items() if not key.startswith("_")}
+ )
+ return payload
+
+
+def _assert_audit_matches_result(
+ *,
+ payload: Mapping[str, Any],
+ evidence: Mapping[str, Any],
+ debug: Mapping[str, Any],
+ operation_id: str,
+) -> None:
+ qid = _clean(evidence.get("question_id"))
+ expected_evidence_sha256 = _digest(dict(evidence))
+ checks = {
+ "operation_id": (_clean(payload.get("operation_id")), operation_id),
+ "question_id": (_clean(payload.get("question_id")), qid),
+ "graph_fingerprint": (
+ _clean(payload.get("graph_fingerprint")),
+ _clean(debug.get("graph_fingerprint")),
+ ),
+ "evidence_sha256": (
+ _clean(payload.get("evidence_sha256")),
+ expected_evidence_sha256,
+ ),
+ }
+ for label, (actual, expected) in checks.items():
+ if actual != expected:
+ raise RuntimeError(
+ f"{qid}: persisted retrieval audit {label} disagrees with replayed result"
+ )
+ if validate_recall_role_plan(payload.get("recall_plan")) != validate_recall_role_plan(
+ evidence.get("recall_plan")
+ ):
+ raise RuntimeError(f"{qid}: persisted retrieval plan disagrees with result")
+
+
+def _write_planner_decision(
+ *,
+ path: Path,
+ row_index: int,
+ row: Mapping[str, Any],
+ plan: Mapping[str, Any],
+ planner_metadata: Mapping[str, Any],
+ context: Mapping[str, Any],
+) -> None:
+ qid = _clean(row.get("question_id"))
+ if planner_metadata.get("physical_api_call") is not True or int(
+ planner_metadata.get("physical_api_calls", 0) or 0
+ ) != 1:
+ raise RuntimeError(f"{qid}: new planner decision lacks one physical API call")
+ _atomic_write_json(
+ path,
+ {
+ "schema_version": PLANNER_DECISION_SCHEMA,
+ "row_index": row_index,
+ "question_id": qid,
+ "row_identity_sha256": _digest(_row_identity(row)),
+ "graph_fingerprint": _clean(context.get("graph_fingerprint")),
+ "recall_plan": validate_recall_role_plan(plan),
+ "planner_metadata": dict(planner_metadata),
+ "recent_dialogue_projection": dict(
+ context.get("recent_dialogue_projection") or {}
+ ),
+ "available_layers": dict(context.get("available_layers") or {}),
+ "created_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
+ },
+ )
+
+
+def _load_planner_decision(
+ *,
+ path: Path,
+ row_index: int,
+ row: Mapping[str, Any],
+ graph_fingerprint: str,
+) -> dict[str, Any] | None:
+ if not path.is_file():
+ return None
+ value = _read_json_object(path)
+ qid = _clean(row.get("question_id"))
+ if (
+ value.get("schema_version") != PLANNER_DECISION_SCHEMA
+ or int(value.get("row_index", -1)) != row_index
+ or _clean(value.get("question_id")) != qid
+ or _clean(value.get("row_identity_sha256"))
+ != _digest(_row_identity(row))
+ or _clean(value.get("graph_fingerprint")) != graph_fingerprint
+ ):
+ raise RuntimeError(f"{qid}: durable planner decision identity mismatch")
+ metadata = value.get("planner_metadata")
+ if not isinstance(metadata, Mapping):
+ raise RuntimeError(f"{qid}: durable planner decision lacks metadata")
+ if metadata.get("physical_api_call") is not True or int(
+ metadata.get("physical_api_calls", 0) or 0
+ ) != 1:
+ raise RuntimeError(f"{qid}: durable planner decision call count is invalid")
+ value["recall_plan"] = validate_recall_role_plan(value.get("recall_plan"))
+ value["planner_metadata"] = dict(metadata)
+ return value
+
+
+def _load_explicit_planner_replays(
+ replay_dir: Path,
+ rows: Sequence[Mapping[str, Any]],
+ graph_fingerprints: Mapping[str, str],
+) -> dict[str, dict[str, Any]]:
+ evidence_path = replay_dir / "evidence_windows.jsonl"
+ debug_path = replay_dir / "retrieval_debug.jsonl"
+ if not evidence_path.is_file() or not debug_path.is_file():
+ raise RuntimeError("explicit planner replay directory lacks committed retrieval files")
+ evidence_rows = _v3().read_jsonl(evidence_path)
+ debug_rows = _v3().read_jsonl(debug_path)
+ evidence_by_qid = {_clean(item.get("question_id")): item for item in evidence_rows}
+ debug_by_qid = {_clean(item.get("question_id")): item for item in debug_rows}
+ expected_qids = [_clean(row.get("question_id")) for row in rows]
+ if (
+ len(evidence_by_qid) != len(evidence_rows)
+ or len(debug_by_qid) != len(debug_rows)
+ or set(evidence_by_qid) != set(expected_qids)
+ or set(debug_by_qid) != set(expected_qids)
+ ):
+ raise RuntimeError("explicit planner replay inventory differs from query manifest")
+ output: dict[str, dict[str, Any]] = {}
+ for row in rows:
+ qid = _clean(row.get("question_id"))
+ evidence = evidence_by_qid[qid]
+ debug = debug_by_qid[qid]
+ if (
+ _clean(evidence.get("question")) != _clean(row.get("question"))
+ or _clean(evidence.get("question_date")) != _clean(row.get("question_date"))
+ or _clean(debug.get("graph_fingerprint")) != graph_fingerprints[qid]
+ ):
+ raise RuntimeError(f"{qid}: explicit planner replay identity is stale")
+ evidence_plan = validate_recall_role_plan(evidence.get("recall_plan"))
+ debug_plan = validate_recall_role_plan(debug.get("recall_plan"))
+ if evidence_plan != debug_plan:
+ raise RuntimeError(f"{qid}: explicit planner replay plans disagree")
+ output[qid] = {
+ "recall_plan": evidence_plan,
+ "source_dir": str(replay_dir.resolve()),
+ }
+ return output
+
+
+def _write_row_checkpoint(
+ *,
+ path: Path,
+ row_index: int,
+ row: Mapping[str, Any],
+ evidence: Mapping[str, Any],
+ debug: Mapping[str, Any],
+) -> None:
+ _atomic_write_json(
+ path,
+ {
+ "schema_version": ROW_CHECKPOINT_SCHEMA,
+ "row_index": row_index,
+ "question_id": _clean(row.get("question_id")),
+ "row_identity_sha256": _digest(_row_identity(row)),
+ "graph_fingerprint": _clean(debug.get("graph_fingerprint")),
+ "evidence_sha256": _digest(dict(evidence)),
+ "debug_sha256": _digest(dict(debug)),
+ "evidence": dict(evidence),
+ "debug": dict(debug),
+ "created_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
+ },
+ )
+
+
+def _load_row_checkpoint(
+ *,
+ path: Path,
+ row_index: int,
+ row: Mapping[str, Any],
+ out_dir: Path,
+ graph_fingerprint: str,
+) -> tuple[dict[str, Any], dict[str, Any]] | None:
+ if not path.is_file():
+ return None
+ value = _read_json_object(path)
+ qid = _clean(row.get("question_id"))
+ if (
+ value.get("schema_version") != ROW_CHECKPOINT_SCHEMA
+ or int(value.get("row_index", -1)) != row_index
+ or _clean(value.get("question_id")) != qid
+ or _clean(value.get("row_identity_sha256"))
+ != _digest(_row_identity(row))
+ or _clean(value.get("graph_fingerprint")) != graph_fingerprint
+ ):
+ raise RuntimeError(f"{qid}: retrieval row checkpoint identity mismatch")
+ evidence, debug = value.get("evidence"), value.get("debug")
+ if not isinstance(evidence, Mapping) or not isinstance(debug, Mapping):
+ raise RuntimeError(f"{qid}: retrieval row checkpoint payload is invalid")
+ evidence, debug = dict(evidence), dict(debug)
+ if (
+ _digest(evidence) != _clean(value.get("evidence_sha256"))
+ or _digest(debug) != _clean(value.get("debug_sha256"))
+ or _clean(evidence.get("question_id")) != qid
+ or _clean(debug.get("question_id")) != qid
+ or Path(str(debug.get("db_path"))).resolve()
+ != Path(str(row["db_path"])).resolve()
+ or Path(str(debug.get("index_path"))).resolve()
+ != Path(str(row["index_path"])).resolve()
+ ):
+ raise RuntimeError(f"{qid}: retrieval row checkpoint content mismatch")
+ audit = _load_persisted_retrieval_audit(
+ row=row,
+ out_dir=out_dir,
+ graph_fingerprint=graph_fingerprint,
+ )
+ if audit is None:
+ raise RuntimeError(f"{qid}: row checkpoint has no persisted retrieval audit")
+ operation_id = _v3().layered_retrieval_operation_id(
+ out_dir, _clean(row.get("scope_id")), qid
+ )
+ _assert_audit_matches_result(
+ payload=audit,
+ evidence=evidence,
+ debug=debug,
+ operation_id=operation_id,
+ )
+ return evidence, debug
+
+
+def _validated_slow_inventory_counts(
+ slow: Sequence[Mapping[str, Any]],
+) -> dict[str, int]:
+ candidate_ids: set[str] = set()
+ summaries: dict[str, Mapping[str, Any]] = {}
+ claims: dict[str, Mapping[str, Any]] = {}
+ for item in slow:
+ if not isinstance(item, Mapping):
+ raise RuntimeError("V4 Slow inventory candidate is not an object")
+ if item.get("inventory_schema_version") != SLOW_INVENTORY_SCHEMA_VERSION:
+ raise RuntimeError("V4 Slow inventory schema mismatch")
+ candidate_id = _clean(item.get("candidate_id"))
+ if not candidate_id or candidate_id in candidate_ids:
+ raise RuntimeError("V4 Slow inventory candidate ID is missing or duplicate")
+ candidate_ids.add(candidate_id)
+ kind = item.get("candidate_kind")
+ if kind == "capsule_summary":
+ summaries[candidate_id] = item
+ elif kind == "capsule_claim":
+ claims[candidate_id] = item
+ else:
+ raise RuntimeError(f"V4 Slow inventory candidate kind is invalid: {kind!r}")
+ if bool(summaries) != bool(claims):
+ raise RuntimeError(
+ "V4 Slow inventory must contain summary and claim candidates together"
+ )
+ referenced_claims: set[str] = set()
+ for summary_id, summary in summaries.items():
+ children = list(summary.get("child_claim_candidate_ids") or [])
+ summary_claims = list(summary.get("claims") or [])
+ if not children or len(children) != len(summary_claims):
+ raise RuntimeError(
+ f"{summary_id}: Slow summary child/claim cardinality mismatch"
+ )
+ try:
+ validate_semantic_summary(
+ summary.get("text"),
+ summary_claims,
+ label=f"{summary_id} summary",
+ )
+ except PatchValidationError as exc:
+ raise RuntimeError(
+ f"{summary_id}: Slow summary violates the inventory contract: {exc}"
+ ) from exc
+ if _current_summary_contract(summary):
+ expected_summary = _lossless_summary_projection(summary_claims)
+ if _clean(summary.get("text")) != expected_summary:
+ raise RuntimeError(
+ f"{summary_id}: current V4.7 Slow summary is not the exact lossless "
+ f"claim projection; expected {expected_summary!r}"
+ )
+ for child_id in children:
+ child_id = _clean(child_id)
+ child = claims.get(child_id)
+ if child is None or child_id in referenced_claims:
+ raise RuntimeError(
+ f"{summary_id}: Slow summary child mapping is missing or duplicate"
+ )
+ if (
+ _clean(child.get("capsule_summary_candidate_id")) != summary_id
+ or _clean(child.get("capsule_summary_text"))
+ != _clean(summary.get("text"))
+ or _clean(child.get("capsule_id"))
+ != _clean(summary.get("capsule_id"))
+ or child.get("revision") != summary.get("revision")
+ ):
+ raise RuntimeError(
+ f"{child_id}: Slow summary/claim identity is inconsistent"
+ )
+ child_claims = list(child.get("claims") or [])
+ if len(child_claims) != 1 or not isinstance(child_claims[0], Mapping):
+ raise RuntimeError(f"{child_id}: Slow claim candidate is not atomic")
+ if not list(child.get("source_parents") or []):
+ raise RuntimeError(f"{child_id}: Slow claim has no Source descent")
+ referenced_claims.add(child_id)
+ if referenced_claims != set(claims):
+ raise RuntimeError("V4 Slow inventory contains unreferenced claim candidates")
+ _validate_active_capsule_partition(slow)
+ return {
+ "slow_candidate_count": len(slow),
+ "slow_capsule_head_count": len(summaries),
+ "slow_summary_candidate_count": len(summaries),
+ "slow_claim_candidate_count": len(claims),
+ }
+
+
+def load_online_index(
+ path: Path,
+ expected_db: Path,
+ expected_scope: str,
+) -> tuple[
+ list[dict[str, Any]],
+ Any,
+ list[dict[str, Any]],
+ Any,
+ list[dict[str, Any]],
+ dict[str, Any],
+]:
+ v3 = _v3()
+ payload = v3.torch.load(path, map_location="cpu", weights_only=True)
+ if not isinstance(payload, Mapping):
+ raise RuntimeError(f"V4 online index is not an object: {path}")
+ if payload.get("schema_version") != ONLINE_INDEX_SCHEMA_VERSION:
+ raise RuntimeError(
+ f"V4 online index schema mismatch; claim-only V3 indexes cannot be reused: {path}"
+ )
+ if payload.get("slow_inventory_schema_version") != SLOW_INVENTORY_SCHEMA_VERSION:
+ raise RuntimeError(f"V4 Slow inventory contract mismatch: {path}")
+ if payload.get("fast_semantic_state_policy") != v3.FAST_SEMANTIC_STATE_POLICY:
+ raise RuntimeError(f"V4 online index fast semantic state policy mismatch: {path}")
+ if _clean(payload.get("scope_id")) != expected_scope:
+ raise RuntimeError(f"V4 online index scope mismatch: {path}")
+ if Path(str(payload.get("db_path", ""))).resolve() != expected_db.resolve():
+ raise RuntimeError(f"V4 online index database mismatch: {path}")
+ text_dim = payload.get("text_dim")
+ if not isinstance(text_dim, int) or text_dim <= 0:
+ raise RuntimeError(f"V4 online index text dimension is invalid: {path}")
+ candidates = list(payload.get("fast_candidates") or [])
+ fast_vectors = payload.get("fast_vectors")
+ slow = list(payload.get("slow_inventory") or [])
+ slow_vectors = payload.get("slow_vectors")
+ semantic_records = list(payload.get("fast_semantic_records") or [])
+ if (
+ not candidates
+ or fast_vectors is None
+ or tuple(fast_vectors.shape) != (len(candidates), text_dim)
+ ):
+ raise RuntimeError(f"V4 online source index payload is incomplete: {path}")
+ _validate_source_candidate_temporal_metadata(candidates)
+ if slow_vectors is None or tuple(slow_vectors.shape) != (len(slow), text_dim):
+ raise RuntimeError(f"V4 online Slow index payload is incomplete: {path}")
+ if any("labels" in candidate for candidate in [*candidates, *slow]):
+ raise RuntimeError("V4 runtime index must not contain benchmark labels")
+ slow_counts = _validated_slow_inventory_counts(slow)
+ for key, expected in slow_counts.items():
+ if payload.get(key) != expected:
+ raise RuntimeError(
+ f"V4 online Slow index count mismatch for {key}: {path}"
+ )
+ if _clean(payload.get("slow_inventory_sha256")) != _digest(slow):
+ raise RuntimeError(f"V4 online Slow inventory digest mismatch: {path}")
+ return (
+ candidates,
+ fast_vectors.float().contiguous(),
+ slow,
+ slow_vectors.float().contiguous(),
+ semantic_records,
+ dict(payload),
+ )
+
+
+def _validate_source_candidate_temporal_metadata(
+ candidates: Sequence[Mapping[str, Any]],
+) -> None:
+ for index, candidate in enumerate(candidates):
+ if not isinstance(candidate, Mapping):
+ raise RuntimeError(f"V4 Source candidate {index} is not an object")
+ missing = [
+ field
+ for field in (
+ "session_id",
+ "source_record_id",
+ "historical_date",
+ "timestamp",
+ "message_role",
+ )
+ if not _clean(candidate.get(field))
+ ]
+ if missing:
+ raise RuntimeError(
+ f"V4 Source candidate {index} lacks production temporal metadata: "
+ + ",".join(missing)
+ )
+ if _clean(candidate.get("message_role")) not in {
+ "user",
+ "assistant",
+ "system",
+ "tool",
+ }:
+ raise RuntimeError(
+ f"V4 Source candidate {index} has an invalid message_role"
+ )
+
+
+def command_build_index(args: argparse.Namespace) -> None:
+ v3 = _v3()
+ rows = v3.read_jsonl(Path(args.scope_manifest))
+ device = v3.torch.device(args.device)
+ if device.type == "cuda" and not v3.torch.cuda.is_available():
+ raise RuntimeError("CUDA is unavailable")
+ vectorizer: Any | None = None
+ started = time.time()
+ report_rows: list[dict[str, Any]] = []
+ reused_index_count = 0
+ for row_index, row in enumerate(rows, start=1):
+ db_path = Path(row["db_path"]).resolve()
+ scope_id = _clean(row.get("scope_id"))
+ index_path = Path(row["index_path"]).resolve()
+ expected = {
+ "schema_version": ONLINE_INDEX_SCHEMA_VERSION,
+ "slow_inventory_schema_version": SLOW_INVENTORY_SCHEMA_VERSION,
+ "fast_semantic_state_policy": v3.FAST_SEMANTIC_STATE_POLICY,
+ "scope_id": scope_id,
+ "db_path": str(db_path),
+ "subchunk_chars": int(args.subchunk_chars),
+ "subchunk_overlap": int(args.subchunk_overlap),
+ "embedding_model": str(Path(args.embedding_model).resolve()),
+ "embedding_max_length": int(args.embedding_max_length),
+ "text_dim": int(args.text_dim),
+ "strict_no_truncation": True,
+ }
+ if index_path.exists():
+ payload = v3.torch.load(index_path, map_location="cpu", weights_only=True)
+ if not isinstance(payload, Mapping):
+ raise RuntimeError(f"existing V4 online index is not an object: {index_path}")
+ mismatches = {
+ key: {"expected": value, "actual": payload.get(key)}
+ for key, value in expected.items()
+ if payload.get(key) != value
+ }
+ current_fingerprint = v3.scope_fingerprint(db_path, scope_id)
+ if _clean(payload.get("graph_fingerprint")) != current_fingerprint:
+ mismatches["graph_fingerprint"] = {
+ "expected": current_fingerprint,
+ "actual": payload.get("graph_fingerprint"),
+ }
+ if mismatches:
+ raise RuntimeError(
+ "existing online index is incompatible with the V4 summary/claim contract: "
+ f"{index_path} mismatches={json.dumps(mismatches, sort_keys=True)}"
+ )
+ loaded = load_online_index(index_path, db_path, scope_id)
+ loaded_payload = loaded[-1]
+ report_row = {
+ "question_id": _clean(row.get("question_id")),
+ "scope_id": scope_id,
+ "db_path": str(db_path),
+ "index_path": str(index_path),
+ "parent_count": int(loaded_payload["parent_count"]),
+ "candidate_count": int(loaded_payload["candidate_count"]),
+ "slow_capsule_head_count": int(
+ loaded_payload["slow_capsule_head_count"]
+ ),
+ "slow_summary_candidate_count": int(
+ loaded_payload["slow_summary_candidate_count"]
+ ),
+ "slow_claim_candidate_count": int(
+ loaded_payload["slow_claim_candidate_count"]
+ ),
+ "fast_semantic_record_count": int(
+ loaded_payload["fast_semantic_record_count"]
+ ),
+ "fast_semantic_state_policy": loaded_payload[
+ "fast_semantic_state_policy"
+ ],
+ "graph_counts": dict(loaded_payload["graph_counts_at_index"]),
+ "graph_fingerprint": current_fingerprint,
+ "reused_existing_index": True,
+ }
+ reused_index_count += 1
+ report_rows.append(report_row)
+ print(
+ json.dumps(
+ {
+ "status": "index_reused",
+ "row": row_index,
+ "total": len(rows),
+ **report_row,
+ }
+ ),
+ flush=True,
+ )
+ continue
+ if vectorizer is None:
+ vectorizer = v3.BgeM3DenseVectorizer(
+ dim=args.text_dim,
+ model_path=args.embedding_model,
+ device=str(device),
+ max_length=args.embedding_max_length,
+ strict_max_length=True,
+ )
+ graph_fingerprint = v3.scope_fingerprint(db_path, scope_id)
+ parents = v3.load_persisted_parent_chunks(db_path, scope_id)
+ candidates = v3.parent_subchunks(
+ parents,
+ scope_id=scope_id,
+ subchunk_chars=args.subchunk_chars,
+ subchunk_overlap=args.subchunk_overlap,
+ )
+ slow, semantic_records = load_v4_layered_inventory(
+ db_path, scope_id, parents
+ )
+ slow_counts = _validated_slow_inventory_counts(slow)
+ fast_vectors = vectorizer.encode_batch(
+ [candidate["text"] for candidate in candidates],
+ batch_size=args.batch_size,
+ )
+ slow_vectors = (
+ vectorizer.encode_batch(
+ [candidate["text"] for candidate in slow],
+ batch_size=args.batch_size,
+ )
+ if slow
+ else v3.torch.empty((0, args.text_dim), dtype=v3.torch.float32)
+ )
+ fast_vectors = fast_vectors.to(v3.torch.float16).contiguous()
+ slow_vectors = slow_vectors.to(v3.torch.float16).contiguous()
+ counts = v3.scope_counts(db_path, scope_id)
+ if v3.scope_fingerprint(db_path, scope_id) != graph_fingerprint:
+ raise RuntimeError(
+ f"{scope_id}: graph changed while the V4 online index was being built"
+ )
+ payload = {
+ **expected,
+ "created_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
+ "graph_counts_at_index": counts,
+ "graph_fingerprint": graph_fingerprint,
+ "parent_count": len(parents),
+ "candidate_count": len(candidates),
+ **slow_counts,
+ "fast_semantic_record_count": len(semantic_records),
+ "fast_candidates": candidates,
+ "fast_vectors": fast_vectors,
+ "slow_inventory": slow,
+ "slow_inventory_sha256": _digest(slow),
+ "slow_vectors": slow_vectors,
+ "fast_semantic_records": semantic_records,
+ }
+ v3.atomic_torch_save(payload, index_path)
+ report_row = {
+ "question_id": _clean(row.get("question_id")),
+ "scope_id": scope_id,
+ "db_path": str(db_path),
+ "index_path": str(index_path),
+ "parent_count": len(parents),
+ "candidate_count": len(candidates),
+ "slow_capsule_head_count": slow_counts["slow_capsule_head_count"],
+ "slow_summary_candidate_count": slow_counts[
+ "slow_summary_candidate_count"
+ ],
+ "slow_claim_candidate_count": slow_counts[
+ "slow_claim_candidate_count"
+ ],
+ "fast_semantic_record_count": len(semantic_records),
+ "fast_semantic_state_policy": v3.FAST_SEMANTIC_STATE_POLICY,
+ "graph_counts": counts,
+ "graph_fingerprint": graph_fingerprint,
+ "reused_existing_index": False,
+ }
+ report_rows.append(report_row)
+ print(
+ json.dumps(
+ {
+ "status": "indexed",
+ "row": row_index,
+ "total": len(rows),
+ **report_row,
+ }
+ ),
+ flush=True,
+ )
+ report = {
+ "status": "complete",
+ "schema_version": ONLINE_INDEX_REPORT_SCHEMA_VERSION,
+ "row_count": len(report_rows),
+ "parent_count": sum(row["parent_count"] for row in report_rows),
+ "candidate_count": sum(row["candidate_count"] for row in report_rows),
+ "slow_capsule_head_count": sum(
+ row["slow_capsule_head_count"] for row in report_rows
+ ),
+ "slow_summary_candidate_count": sum(
+ row["slow_summary_candidate_count"] for row in report_rows
+ ),
+ "slow_claim_candidate_count": sum(
+ row["slow_claim_candidate_count"] for row in report_rows
+ ),
+ "reused_index_count": reused_index_count,
+ "elapsed_sec": round(time.time() - started, 3),
+ "rows": report_rows,
+ }
+ out_report = Path(args.out_report)
+ out_report.parent.mkdir(parents=True, exist_ok=True)
+ _atomic_write(out_report, json.dumps(report, indent=2, sort_keys=True) + "\n")
+
+
+def _staging_runtime_configuration(args: argparse.Namespace) -> dict[str, Any]:
+ (
+ composition_mode,
+ execution_lane,
+ packing_budget_mode,
+ top_k,
+ ) = _validated_runtime_route(args, label="retrieval runtime")
+
+ def path_value(name: str) -> str:
+ value = _arg(args, name, None)
+ return str(Path(value).resolve()) if value not in {None, ""} else ""
+
+ return {
+ "retrieval_contract_schema": RETRIEVAL_CONTRACT_SCHEMA,
+ "runtime_schema_version": RUNTIME_SCHEMA_VERSION,
+ "online_index_schema_version": ONLINE_INDEX_SCHEMA_VERSION,
+ "slow_inventory_schema_version": SLOW_INVENTORY_SCHEMA_VERSION,
+ "execution_lane": execution_lane,
+ "composition_mode": composition_mode,
+ "packing_budget_mode": packing_budget_mode,
+ "top_k": top_k,
+ "adaptive_simple_k": int(_arg(args, "adaptive_simple_k", 8)),
+ "adaptive_standard_k": int(_arg(args, "adaptive_standard_k", 12)),
+ "adaptive_complex_k": int(_arg(args, "adaptive_complex_k", 16)),
+ "source_coverage_trace_k": SOURCE_COVERAGE_TRACE_K,
+ "checkpoint": path_value("checkpoint"),
+ "cross_model": path_value("cross_model"),
+ "repo": path_value("repo"),
+ "harness": path_value("harness"),
+ "node_model": path_value("node_model"),
+ "path_model": path_value("path_model"),
+ "embedding_model": path_value("embedding_model"),
+ "device": str(_arg(args, "device", "cuda")),
+ "graph_device": str(_arg(args, "graph_device", "cuda")),
+ "candidate_event_k": int(_arg(args, "candidate_event_k", 24)),
+ "support_path_k": int(_arg(args, "support_path_k", 3)),
+ "path_tunnel_rescue_k": int(_arg(args, "path_tunnel_rescue_k", 2)),
+ "graph_top_k": int(_arg(args, "graph_top_k", 12)),
+ "dense_k": int(_arg(args, "dense_k", 32)),
+ "slow_dense_k": int(_arg(args, "slow_dense_k", 24)),
+ "graph_k": int(_arg(args, "graph_k", 24)),
+ "cross_max_length": int(_arg(args, "cross_max_length", 1280)),
+ "cross_batch_size": int(_arg(args, "cross_batch_size", 24)),
+ "embedding_max_length": int(_arg(args, "embedding_max_length", 512)),
+ }
+
+
+def command_retrieve(args: argparse.Namespace) -> None:
+ runtime_configuration = _staging_runtime_configuration(args)
+ v3 = _v3()
+ rows = v3.read_jsonl(Path(args.query_manifest))
+ if not rows:
+ raise RuntimeError("query manifest is empty")
+ # Keep checkpoint identity tied to the stable logical path, not whichever
+ # physical volume currently backs that path.
+ out_dir = Path(args.out_dir).absolute()
+ if out_dir.exists():
+ raise RuntimeError(f"retrieval output already exists: {out_dir}")
+ resume = bool(getattr(args, "resume", False))
+ staging = out_dir.with_name(f".{out_dir.name}.staging")
+ staging_preexisted = staging.exists()
+ if staging_preexisted and not resume:
+ raise RuntimeError(
+ f"retrieval staging exists; explicit --resume is required: {staging}"
+ )
+ if staging_preexisted and not staging.is_dir():
+ raise RuntimeError(f"retrieval staging is not a directory: {staging}")
+ if not staging_preexisted:
+ staging.parent.mkdir(parents=True, exist_ok=True)
+ staging.mkdir(parents=False, exist_ok=False)
+ checkpoint_dir = staging / "row_checkpoints"
+ planner_dir = staging / "planner_decisions"
+ checkpoint_dir.mkdir(exist_ok=True)
+ planner_dir.mkdir(exist_ok=True)
+
+ staging_identity = {
+ "schema_version": RUN_STAGING_SCHEMA,
+ "out_dir": str(out_dir),
+ "query_count": len(rows),
+ "question_ids": [_clean(row.get("question_id")) for row in rows],
+ "row_identity_sha256": [_digest(_row_identity(row)) for row in rows],
+ "runtime_configuration": runtime_configuration,
+ }
+ identity_path = staging / "staging_identity.json"
+ if identity_path.is_file():
+ if _read_json_object(identity_path) != staging_identity:
+ raise RuntimeError("retrieval staging identity does not match query manifest")
+ elif staging_preexisted:
+ raise RuntimeError("retrieval staging lacks a durable identity")
+ else:
+ _atomic_write_json(identity_path, staging_identity)
+
+ graph_fingerprints = {
+ _clean(row.get("question_id")): v3.scope_fingerprint(
+ Path(str(row["db_path"])).resolve(), _clean(row.get("scope_id"))
+ )
+ for row in rows
+ }
+ planner_replay_dir = _clean(getattr(args, "planner_replay_dir", ""))
+ explicit_planner_replays = (
+ _load_explicit_planner_replays(
+ Path(planner_replay_dir).resolve(), rows, graph_fingerprints
+ )
+ if planner_replay_dir
+ else {}
+ )
+ persisted_audits = {
+ _clean(row.get("question_id")): _load_persisted_retrieval_audit(
+ row=row,
+ out_dir=out_dir,
+ graph_fingerprint=graph_fingerprints[_clean(row.get("question_id"))],
+ )
+ for row in rows
+ }
+ if not resume:
+ already_audited = [qid for qid, value in persisted_audits.items() if value]
+ if already_audited:
+ raise RuntimeError(
+ "persisted retrieval work exists; explicit --resume is required: "
+ + ",".join(already_audited)
+ )
+
+ planner = None if len(explicit_planner_replays) == len(rows) else planner_from_env()
+ harness, models = v3.load_native_harness(Path(args.harness), Path(args.repo)), v3.OnlineModels(args)
+ harness.disable_topic_bucket_runtime()
+ evidence_rows: list[dict[str, Any]] = []
+ debug_rows: list[dict[str, Any]] = []
+ cache: dict[tuple[str, str], Any] = {}
+ started = time.time()
+ checkpoint_reused_count = 0
+ planner_decision_replay_count = 0
+ audit_plan_replay_count = 0
+ explicit_planner_replay_count = 0
+ new_planner_call_count = 0
+ newly_appended_audit_count = 0
+ for index, row in enumerate(rows, start=1):
+ qid = _clean(row.get("question_id"))
+ graph_fingerprint = graph_fingerprints[qid]
+ checkpoint_path = _row_artifact_path(checkpoint_dir, index, qid)
+ checkpoint = _load_row_checkpoint(
+ path=checkpoint_path,
+ row_index=index,
+ row=row,
+ out_dir=out_dir,
+ graph_fingerprint=graph_fingerprint,
+ )
+ if checkpoint is not None:
+ evidence, debug = checkpoint
+ checkpoint_reused_count += 1
+ evidence_rows.append(evidence)
+ debug_rows.append(debug)
+ print(
+ json.dumps(
+ {
+ "status": "checkpoint_reused",
+ "row": index,
+ "total": len(rows),
+ "question_id": qid,
+ }
+ ),
+ flush=True,
+ )
+ continue
+
+ audit = persisted_audits[qid]
+ decision_path = _row_artifact_path(planner_dir, index, qid)
+ decision = _load_planner_decision(
+ path=decision_path,
+ row_index=index,
+ row=row,
+ graph_fingerprint=graph_fingerprint,
+ )
+ route_override: Mapping[str, Any] | None = None
+ route_metadata: Mapping[str, Any] | None = None
+ explicit_replay = explicit_planner_replays.get(qid)
+ if explicit_replay is not None:
+ route_override = explicit_replay["recall_plan"]
+ if decision is not None and validate_recall_role_plan(
+ decision.get("recall_plan")
+ ) != validate_recall_role_plan(route_override):
+ raise RuntimeError(f"{qid}: durable and explicit planner replays disagree")
+ if audit is not None and validate_recall_role_plan(
+ audit.get("recall_plan")
+ ) != validate_recall_role_plan(route_override):
+ raise RuntimeError(f"{qid}: audit and explicit planner replays disagree")
+ route_metadata = {
+ "physical_api_call": False,
+ "physical_api_calls": 0,
+ "stage": "recall_planner",
+ "status": "explicit_replay",
+ "planner_version": "frozen_committed_retrieval",
+ "prompt_version": "frozen_committed_retrieval",
+ "replayed_without_api": True,
+ "replay_physical_api_call": False,
+ "replay_source": "explicit_committed_retrieval",
+ "replay_source_dir": explicit_replay["source_dir"],
+ }
+ explicit_planner_replay_count += 1
+ elif decision is not None:
+ route_override = decision["recall_plan"]
+ if audit is not None and validate_recall_role_plan(
+ audit.get("recall_plan")
+ ) != validate_recall_role_plan(route_override):
+ raise RuntimeError(f"{qid}: planner decision and audit plan disagree")
+ route_metadata = {
+ **dict(decision["planner_metadata"]),
+ "replayed_without_api": True,
+ "replay_physical_api_call": False,
+ "replay_source": "durable_planner_decision",
+ }
+ planner_decision_replay_count += 1
+ elif audit is not None:
+ route_override = audit["recall_plan"]
+ route_metadata = {
+ "physical_api_call": True,
+ "physical_api_calls": 1,
+ "stage": "recall_planner",
+ "provider": "deepseek",
+ "model": DEEPSEEK_FLASH_MODEL,
+ "status": "completed_response_metadata_lost_after_process_failure",
+ "planner_version": "recovered_from_layered_retrieval_audit",
+ "prompt_version": "unknown_prior_process",
+ "historical_call_usage_known": False,
+ "replayed_without_api": True,
+ "replay_physical_api_call": False,
+ "replay_source": "persisted_layered_retrieval_audit",
+ "recovery_operation_id": _clean(audit.get("operation_id")),
+ "recovery_payload_sha256": _clean(audit.get("_payload_sha256")),
+ }
+ audit_plan_replay_count += 1
+
+ def persist_new_planner_decision(
+ plan: Mapping[str, Any],
+ metadata: Mapping[str, Any],
+ context: Mapping[str, Any],
+ ) -> None:
+ _write_planner_decision(
+ path=decision_path,
+ row_index=index,
+ row=row,
+ plan=plan,
+ planner_metadata=metadata,
+ context=context,
+ )
+
+ evidence, debug = retrieve_one(
+ row,
+ args=args,
+ harness=harness,
+ models=models,
+ planner=planner,
+ route_override=route_override,
+ route_override_metadata=route_metadata,
+ planner_decision_callback=(
+ persist_new_planner_decision if route_override is None else None
+ ),
+ graph_adapter_cache=cache,
+ )
+ if route_override is None:
+ new_planner_call_count += 1
+ operation_id = v3.layered_retrieval_operation_id(out_dir, str(debug["scope_id"]), str(evidence["question_id"]))
+ if audit is not None:
+ _assert_audit_matches_result(
+ payload=audit,
+ evidence=evidence,
+ debug=debug,
+ operation_id=operation_id,
+ )
+ persisted = v3.append_layered_retrieval_audit(repo=Path(args.repo), db_path=Path(debug["db_path"]), scope_id=str(debug["scope_id"]), operation_id=operation_id, evidence=evidence, debug=debug)
+ audit_payload = dict(persisted.get("payload") or {})
+ _assert_audit_matches_result(
+ payload=audit_payload,
+ evidence=evidence,
+ debug=debug,
+ operation_id=operation_id,
+ )
+ newly_appended_audit_count += int(bool(persisted.get("appended")))
+ debug["layered_retrieval_audit"] = {"operation_id": operation_id, "query_id": _clean(audit_payload.get("query_id")), "event_total": int(persisted.get("event_total") or 0), "trimmed_total": int(persisted.get("trimmed_total") or 0), "newly_appended": bool(persisted.get("appended"))}
+ if not _clean(debug["layered_retrieval_audit"]["query_id"]):
+ raise RuntimeError(f"{qid}: persisted retrieval audit lacks query_id")
+ _write_row_checkpoint(
+ path=checkpoint_path,
+ row_index=index,
+ row=row,
+ evidence=evidence,
+ debug=debug,
+ )
+ evidence_rows.append(evidence)
+ debug_rows.append(debug)
+ print(json.dumps({"status": "retrieved", "row": index, "total": len(rows), "question_id": evidence["question_id"], "inventory": debug["inventory_count"], "selected": debug["selected_count"], "latency_sec": debug["latency_sec"]["total"]}), flush=True)
+ latencies = [float(row["latency_sec"]["total"]) for row in debug_rows]
+ budget_tiers = {
+ tier: sum(
+ int(dict(row.get("packing_budget_decision") or {}).get("tier") == tier)
+ for row in debug_rows
+ )
+ for tier in ("fixed", "simple", "standard", "complex")
+ }
+ layer_window_totals = {
+ layer: sum(
+ int(dict(row.get("selected_layer_window_counts") or {}).get(layer, 0))
+ for row in debug_rows
+ )
+ for layer in ("source", "fast", "slow")
+ }
+ report = {
+ "status": "complete",
+ "schema_version": "tmcra.v4.online-retrieval-report.6",
+ "runtime_schema_version": RUNTIME_SCHEMA_VERSION,
+ "online_index_schema_version": ONLINE_INDEX_SCHEMA_VERSION,
+ "slow_inventory_schema_version": SLOW_INVENTORY_SCHEMA_VERSION,
+ "created_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
+ "query_count": len(rows),
+ "execution_lane": str(args.execution_lane),
+ "composition_mode": str(args.composition_mode),
+ "production_layered_contract_enforced": (
+ str(args.execution_lane) == "production"
+ and str(args.composition_mode) == "layered"
+ ),
+ "candidate_paths_executed_count": sum(
+ int(all(row["candidate_paths_executed"].values())) for row in debug_rows
+ ),
+ "required_layer_row_counts": {
+ layer: sum(
+ int(layer in list(row.get("required_selected_layers") or []))
+ for row in debug_rows
+ )
+ for layer in ("source", "fast", "slow")
+ },
+ "selected_layer_window_totals": layer_window_totals,
+ "fast_semantic_shortlist_total": sum(
+ int(row.get("fast_semantic_shortlist_count", 0)) for row in debug_rows
+ ),
+ "avg_budget_excluded_unit_count": round(
+ sum(row["budget_excluded_unit_count"] for row in debug_rows) / len(debug_rows),
+ 4,
+ ),
+ "avg_duplicate_unit_count": round(
+ sum(row["duplicate_unit_count"] for row in debug_rows) / len(debug_rows),
+ 4,
+ ),
+ "avg_latency_sec": round(sum(latencies) / len(latencies), 4),
+ "max_latency_sec": round(max(latencies), 4),
+ "elapsed_sec": round(time.time() - started, 3),
+ "evidence": str(out_dir / "evidence_windows.jsonl"),
+ "debug": str(out_dir / "retrieval_debug.jsonl"),
+ "source_coverage_trace_k": SOURCE_COVERAGE_TRACE_K,
+ "packing_budget_mode": str(args.packing_budget_mode),
+ "configured_top_k": int(args.top_k),
+ "max_final_window_count": max(
+ int(row.get("selected_count", 0)) for row in debug_rows
+ ),
+ "atomic_unit_packing": True,
+ "packing_budget_tier_counts": budget_tiers,
+ "cross_layer_weighted_fusion": False,
+ "answer_attachment_contract": "object-list-v1",
+ "ranking_metadata_field": "retrieval_metadata",
+ "session_coherent_ordering": True,
+ "session_ordering_policy": SESSION_ORDERING_POLICY,
+ "row_checkpoint_policy": ROW_CHECKPOINT_SCHEMA,
+ "checkpoint_reused_count": checkpoint_reused_count,
+ "explicit_planner_replay_count": explicit_planner_replay_count,
+ "planner_decision_replay_count": planner_decision_replay_count,
+ "audit_plan_replay_count": audit_plan_replay_count,
+ "new_planner_call_count": new_planner_call_count,
+ "newly_appended_retrieval_audit_count": newly_appended_audit_count,
+ "historical_planner_calls_without_usage_count": sum(
+ int(dict(row.get("planner") or {}).get("historical_call_usage_known") is False)
+ for row in debug_rows
+ ),
+ }
+ _atomic_write(staging / "evidence_windows.jsonl", "".join(json.dumps(row, ensure_ascii=False, sort_keys=True) + "\n" for row in evidence_rows))
+ _atomic_write(staging / "retrieval_debug.jsonl", "".join(json.dumps(row, ensure_ascii=False, sort_keys=True) + "\n" for row in debug_rows))
+ _atomic_write(staging / "report.json", json.dumps(report, indent=2, sort_keys=True) + "\n")
+ os.replace(staging, out_dir)
+ print(json.dumps(report, indent=2, sort_keys=True))
+
+
+def add_common_model_args(parser: argparse.ArgumentParser) -> None:
+ parser.add_argument("--embedding-model", default=os.getenv("TMCRA_BGE_M3_PATH", "BAAI/bge-m3"))
+ parser.add_argument("--text-dim", type=int, default=1024)
+ parser.add_argument("--embedding-max-length", type=int, default=8192)
+ parser.add_argument("--device", default="cuda")
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description="TMCRA V4 production online index and retrieval runtime")
+ sub = parser.add_subparsers(dest="command", required=True)
+ index = sub.add_parser("build-index")
+ index.add_argument("--scope-manifest", required=True); index.add_argument("--out-report", required=True); index.add_argument("--subchunk-chars", type=int, default=1800); index.add_argument("--subchunk-overlap", type=int, default=200); index.add_argument("--batch-size", type=int, default=16); add_common_model_args(index)
+ retrieve = sub.add_parser("retrieve")
+ retrieve.add_argument("--query-manifest", required=True); retrieve.add_argument("--out-dir", required=True); retrieve.add_argument("--resume", action="store_true"); retrieve.add_argument("--planner-replay-dir"); retrieve.add_argument("--checkpoint", required=True); retrieve.add_argument("--cross-model", default=os.getenv("TMCRA_BGE_RERANKER_PATH", "BAAI/bge-reranker-v2-m3")); retrieve.add_argument("--cross-max-length", type=int, default=1280); retrieve.add_argument("--cross-batch-size", type=int, default=24); retrieve.add_argument("--repo", required=True); retrieve.add_argument("--harness", required=True); retrieve.add_argument("--node-model", required=True); retrieve.add_argument("--path-model", required=True); retrieve.add_argument("--graph-device", default="cuda"); retrieve.add_argument("--candidate-event-k", type=int, default=24); retrieve.add_argument("--support-path-k", type=int, default=3); retrieve.add_argument("--path-tunnel-rescue-k", type=int, default=2); retrieve.add_argument("--graph-top-k", type=int, default=12); retrieve.add_argument("--dense-k", type=int, default=32); retrieve.add_argument("--slow-dense-k", type=int, default=24); retrieve.add_argument("--graph-k", type=int, default=24); retrieve.add_argument("--execution-lane", choices=sorted(EXECUTION_LANES), default="production"); retrieve.add_argument("--composition-mode", choices=sorted(COMPOSITION_MODES), default="layered"); retrieve.add_argument("--packing-budget-mode", choices=sorted(PACKING_BUDGET_MODES), default="fixed"); retrieve.add_argument("--top-k", type=int, default=8); retrieve.add_argument("--adaptive-simple-k", type=int, default=8); retrieve.add_argument("--adaptive-standard-k", type=int, default=12); retrieve.add_argument("--adaptive-complex-k", type=int, default=16); add_common_model_args(retrieve)
+ args = parser.parse_args()
+ (command_build_index if args.command == "build-index" else command_retrieve)(args)
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_recall_planner.py b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_recall_planner.py
new file mode 100644
index 0000000000000000000000000000000000000000..545ac37d9af0cc8edd6eed3adc58aa9d42cf9455
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_recall_planner.py
@@ -0,0 +1,361 @@
+"""Strict TMCRA V4 recall role planning.
+
+The planner describes ranking and composition roles. It cannot disable a local
+retrieval layer; execution remains a controller responsibility.
+"""
+
+from __future__ import annotations
+
+import json
+import re
+import time
+import urllib.error
+import urllib.request
+import uuid
+from collections.abc import Mapping, Sequence
+from math import isfinite
+from typing import Any
+
+from tmcra_v3_recall_planner import (
+ DEEPSEEK_FLASH_MODEL,
+ RecallPlannerError,
+ RecallPlannerResponseError,
+ _reject_gold,
+ _sha256,
+)
+
+ROLE_PLAN_SCHEMA = "tmcra.recall-role-plan.v1"
+PLANNER_VERSION = ROLE_PLAN_SCHEMA
+PLANNER_PROMPT_VERSION = "tmcra-recall-role-planner-2026-07-12.3"
+PLAN_FIELDS = frozenset({"schema_version", "resolved_query", "query_kind", "temporal_focus", "conflict_policy", "layers"})
+LAYER_NAMES = ("source", "fast", "slow")
+LAYER_FIELDS = frozenset({"role", "weight"})
+ROLE_VALUES = frozenset({"primary", "support", "context", "conflict", "evidence", "atomic", "bridge"})
+ROLE_PRIORS = {"primary": 1.0, "evidence": 1.0, "atomic": 1.0, "conflict": 0.9, "support": 0.75, "bridge": 0.65, "context": 0.5}
+QUERY_KINDS = frozenset({"fact", "event", "state", "preference", "goal", "constraint", "relationship", "task", "decision", "comparison", "historical", "unknown"})
+QUERY_KIND_EXTENSION_RE = re.compile(r"^[a-z][a-z0-9_]{0,63}$")
+TEMPORAL_FOCUSES = frozenset({"current", "recent", "future", "historical", "timeless", "mixed", "unknown"})
+CONFLICT_POLICIES = frozenset({"prefer_recent", "prefer_durable", "preserve_parallel", "compare", "surface_uncertainty"})
+
+SYSTEM_PROMPT = f"""You are the TMCRA V4 recall control plane.
+Return exactly one JSON object using schema {ROLE_PLAN_SCHEMA} and no other keys:
+{{"schema_version":"{ROLE_PLAN_SCHEMA}","resolved_query":"standalone query",
+"query_kind":"fact|event|state|preference|goal|constraint|relationship|task|decision|comparison|historical|unknown",
+"temporal_focus":"current|recent|future|historical|timeless|mixed|unknown",
+"conflict_policy":"prefer_recent|prefer_durable|preserve_parallel|compare|surface_uncertainty",
+"layers":{{"source":{{"role":"primary|support|context|conflict|evidence|atomic|bridge","weight":0.0}},
+"fast":{{"role":"primary|support|context|conflict|evidence|atomic|bridge","weight":0.0}},
+"slow":{{"role":"primary|support|context|conflict|evidence|atomic|bridge","weight":0.0}}}}}}
+
+Weights are bounded in [0, 1]. A zero weight is valid but is never a retrieval
+gate: source, fast, and slow local candidate paths still run whenever their
+inventories exist. Never emit disabled or excluded layers, scores, candidate
+IDs, benchmark fields, answer text, or evidence text. Invalid output is a hard
+failure; do not retry or return a fallback plan."""
+
+
+def _text(value: Any) -> str:
+ return value.strip() if isinstance(value, str) else ""
+
+
+def _weight(value: Any, path: str) -> float:
+ if isinstance(value, bool) or not isinstance(value, (int, float)):
+ raise RecallPlannerError(f"{path} must be a finite number in [0, 1]")
+ result = float(value)
+ if not isfinite(result) or not 0.0 <= result <= 1.0:
+ raise RecallPlannerError(f"{path} must be a finite number in [0, 1]")
+ return result
+
+
+def validate_recall_role_plan(value: Mapping[str, Any]) -> dict[str, Any]:
+ if not isinstance(value, Mapping) or set(value) != PLAN_FIELDS:
+ raise RecallPlannerError("RecallRolePlan root must contain exactly the required schema fields")
+ if value.get("schema_version") != ROLE_PLAN_SCHEMA:
+ raise RecallPlannerError(f"schema_version must be {ROLE_PLAN_SCHEMA!r}")
+ resolved_query = _text(value.get("resolved_query"))
+ if not resolved_query:
+ raise RecallPlannerError("resolved_query is required")
+ if len(resolved_query) > 2000:
+ raise RecallPlannerError("resolved_query exceeds 2000 characters")
+ query_kind = _text(value.get("query_kind"))
+ if query_kind not in QUERY_KINDS and QUERY_KIND_EXTENSION_RE.fullmatch(query_kind) is None:
+ raise RecallPlannerError(
+ "query_kind must be a documented value or a bounded snake_case extension"
+ )
+ temporal_focus = _text(value.get("temporal_focus"))
+ if temporal_focus not in TEMPORAL_FOCUSES:
+ raise RecallPlannerError(f"unsupported temporal_focus: {temporal_focus!r}")
+ conflict_policy = _text(value.get("conflict_policy"))
+ if conflict_policy not in CONFLICT_POLICIES:
+ raise RecallPlannerError(f"unsupported conflict_policy: {conflict_policy!r}")
+ layers = value.get("layers")
+ if not isinstance(layers, Mapping) or set(layers) != set(LAYER_NAMES):
+ raise RecallPlannerError("layers must contain exactly source, fast, and slow")
+ normalized_layers: dict[str, dict[str, Any]] = {}
+ for layer in LAYER_NAMES:
+ entry = layers[layer]
+ if not isinstance(entry, Mapping) or set(entry) != LAYER_FIELDS:
+ raise RecallPlannerError(f"layers.{layer} must contain exactly role and weight")
+ role = _text(entry.get("role"))
+ if role not in ROLE_VALUES:
+ raise RecallPlannerError(f"layers.{layer}.role is invalid or disables a layer")
+ normalized_layers[layer] = {"role": role, "weight": _weight(entry.get("weight"), f"layers.{layer}.weight")}
+ if not any(entry["weight"] > 0.0 for entry in normalized_layers.values()):
+ raise RecallPlannerError("recall role plan cannot assign zero weight to every layer")
+ return {"schema_version": ROLE_PLAN_SCHEMA, "resolved_query": resolved_query, "query_kind": query_kind, "temporal_focus": temporal_focus, "conflict_policy": conflict_policy, "layers": normalized_layers}
+
+
+def _validate_recent_dialogue(value: Any) -> list[dict[str, Any]]:
+ if value is None:
+ return []
+ if not isinstance(value, Sequence) or isinstance(value, (str, bytes)) or len(value) > 8:
+ raise RecallPlannerError("recent_dialogue must contain at most 8 turn objects")
+ output: list[dict[str, Any]] = []
+ for index, item in enumerate(value):
+ if not isinstance(item, Mapping) or set(item) != {"turn_index", "speaker", "text"}:
+ raise RecallPlannerError(f"recent_dialogue[{index}] has an invalid schema")
+ try:
+ turn_index = int(item["turn_index"])
+ except (TypeError, ValueError) as exc:
+ raise RecallPlannerError(f"recent_dialogue[{index}].turn_index must be an integer") from exc
+ speaker, text = _text(item.get("speaker")).lower(), _text(item.get("text"))
+ if speaker not in {"user", "assistant"} or not text or len(text) > 4000:
+ raise RecallPlannerError(f"recent_dialogue[{index}] has an invalid speaker or text")
+ output.append({"turn_index": turn_index, "speaker": speaker, "text": text})
+ if any(left["turn_index"] >= right["turn_index"] for left, right in zip(output, output[1:])):
+ raise RecallPlannerError("recent_dialogue must be in strictly increasing turn order")
+ _reject_gold(output, "recent_dialogue")
+ return output
+
+
+def _normalize_usage(value: Any) -> dict[str, int]:
+ if not isinstance(value, Mapping):
+ raise RecallPlannerError("recall role planner success response lacks usage")
+
+ def integer(name: str, *aliases: str, required: bool = False) -> tuple[int, bool]:
+ for key in (name, *aliases):
+ if value.get(key) is None:
+ continue
+ raw = value.get(key)
+ if isinstance(raw, bool) or not isinstance(raw, (int, float)) or int(raw) < 0:
+ raise RecallPlannerError(f"usage.{key} is invalid")
+ return int(raw), True
+ if required:
+ raise RecallPlannerError(f"usage.{name} is missing")
+ return 0, False
+
+ prompt, _ = integer("prompt_tokens", "input_tokens", required=True)
+ completion, _ = integer("completion_tokens", "output_tokens", required=True)
+ hit, has_hit = integer(
+ "prompt_cache_hit_tokens", "cache_read_input_tokens", "cached_tokens"
+ )
+ miss, has_miss = integer("prompt_cache_miss_tokens", "cache_miss_input_tokens")
+ if hit > prompt or miss > prompt or (has_hit and has_miss and hit + miss != prompt):
+ raise RecallPlannerError("planner cache usage does not balance prompt tokens")
+ if not has_hit:
+ hit = prompt - miss
+ if not has_miss:
+ miss = prompt - hit
+ total, has_total = integer("total_tokens")
+ if not has_total:
+ total = prompt + completion
+ elif total < prompt + completion:
+ raise RecallPlannerError("usage.total_tokens is smaller than prompt plus completion")
+ return {
+ "prompt_tokens": prompt,
+ "completion_tokens": completion,
+ "prompt_cache_hit_tokens": hit,
+ "prompt_cache_miss_tokens": miss,
+ "total_tokens": total,
+ }
+
+
+def _metadata_v4(
+ *,
+ physical_call_id: str,
+ key_index: int,
+ started: float,
+ finish_reason: str,
+ content: str = "",
+ usage: Mapping[str, Any] | None = None,
+ http_status: int | None = None,
+ error_type: str | None = None,
+ request_sha256: str = "",
+ response_id: str = "",
+) -> dict[str, Any]:
+ metadata: dict[str, Any] = {
+ "physical_call_id": physical_call_id,
+ "physical_api_call": True,
+ "physical_api_calls": 1,
+ "stage": "recall_planner",
+ "provider": "deepseek",
+ "model": DEEPSEEK_FLASH_MODEL,
+ "api_key_index": key_index,
+ "latency_seconds": round(time.time() - started, 3),
+ "response_sha256": _sha256(content),
+ "finish_reason": finish_reason,
+ "status": "completed" if finish_reason == "stop" else finish_reason,
+ "planner_version": PLANNER_VERSION,
+ "prompt_version": PLANNER_PROMPT_VERSION,
+ "request_sha256": request_sha256,
+ "response_id": response_id,
+ }
+ if usage is not None:
+ normalized_usage = dict(usage)
+ metadata.update(normalized_usage)
+ metadata["usage"] = normalized_usage
+ if http_status is not None:
+ metadata["http_status"] = int(http_status)
+ if error_type:
+ metadata["error_type"] = error_type
+ return metadata
+
+
+class DeepSeekFlashRecallRolePlanner:
+ """One physical Flash request; invalid output is never retried or repaired."""
+
+ def __init__(self, *, base_url: str, model: str, api_keys: Sequence[str], timeout: float = 30, max_tokens: int = 512) -> None:
+ self.base_url = _text(base_url).rstrip("/")
+ self.model = model
+ self.api_keys = list(dict.fromkeys(_text(key) for key in api_keys if _text(key)))
+ self.timeout, self.max_tokens, self.request_index = max(1.0, float(timeout)), max(128, int(max_tokens)), 0
+ if not self.base_url or not self.api_keys:
+ raise RecallPlannerError("planner base_url and API key pool are required")
+ if self.model != DEEPSEEK_FLASH_MODEL:
+ raise RecallPlannerError(f"recall role planner requires {DEEPSEEK_FLASH_MODEL}, got {self.model!r}")
+
+ def plan(self, *, query: str, question_date: str, available_layers: Mapping[str, Any], recent_dialogue: Sequence[Mapping[str, Any]] | None = None) -> tuple[dict[str, Any], dict[str, Any]]:
+ query, question_date = _text(query), _text(question_date)
+ if not query or not question_date:
+ raise RecallPlannerError("query and question_date are required")
+ if not isinstance(available_layers, Mapping) or set(available_layers) != set(LAYER_NAMES):
+ raise RecallPlannerError("available_layers must contain exactly source, fast, and slow summaries")
+ _reject_gold(available_layers, "available_layers")
+ dialogue = _validate_recent_dialogue(recent_dialogue)
+ payload = {"query": query, "question_date": question_date, "recent_dialogue": dialogue, "available_layers": dict(available_layers)}
+ key_index = self.request_index % len(self.api_keys)
+ self.request_index += 1
+ body = {"model": DEEPSEEK_FLASH_MODEL, "messages": [{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": json.dumps(payload, ensure_ascii=False, separators=(",", ":"))}], "temperature": 0, "max_tokens": self.max_tokens, "response_format": {"type": "json_object"}, "thinking": {"type": "disabled"}, "enable_thinking": False}
+ request_sha256 = _sha256(json.dumps(body, ensure_ascii=False, sort_keys=True))
+ physical_call_id = "dsc_" + uuid.uuid4().hex
+ request = urllib.request.Request(f"{self.base_url}/chat/completions", data=json.dumps(body, ensure_ascii=False).encode("utf-8"), headers={"Content-Type": "application/json", "Authorization": f"Bearer {self.api_keys[key_index]}"}, method="POST")
+ started = time.time()
+ try:
+ with urllib.request.urlopen(request, timeout=self.timeout) as response:
+ http_status = int(response.getcode())
+ raw_http = response.read().decode("utf-8")
+ response_payload = json.loads(raw_http)
+ except urllib.error.HTTPError as exc:
+ detail = exc.read().decode("utf-8", errors="replace")[:1000]
+ raise RecallPlannerResponseError(f"recall role planner HTTP {exc.code}: {detail}", response_content=detail, request_metadata=_metadata_v4(physical_call_id=physical_call_id, key_index=key_index, started=started, finish_reason="http_error", content=detail, http_status=exc.code, request_sha256=request_sha256)) from exc
+ except (UnicodeDecodeError, json.JSONDecodeError) as exc:
+ raw = locals().get("raw_http", "")
+ raise RecallPlannerResponseError("recall role planner returned invalid HTTP JSON", response_content=raw, request_metadata=_metadata_v4(physical_call_id=physical_call_id, key_index=key_index, started=started, finish_reason="invalid_http_json", content=raw, error_type=exc.__class__.__name__, request_sha256=request_sha256, http_status=locals().get("http_status"))) from exc
+ except Exception as exc:
+ raise RecallPlannerResponseError(f"recall role planner request failed: {exc.__class__.__name__}: {exc}", request_metadata=_metadata_v4(physical_call_id=physical_call_id, key_index=key_index, started=started, finish_reason="request_error", error_type=exc.__class__.__name__, request_sha256=request_sha256)) from exc
+ try:
+ usage = _normalize_usage(
+ response_payload.get("usage") if isinstance(response_payload, Mapping) else None
+ )
+ except RecallPlannerError as exc:
+ raise RecallPlannerResponseError(
+ f"recall role planner response usage is invalid: {exc}",
+ response_content=raw_http,
+ request_metadata=_metadata_v4(
+ physical_call_id=physical_call_id,
+ key_index=key_index,
+ started=started,
+ finish_reason="invalid_usage",
+ content=raw_http,
+ http_status=http_status,
+ request_sha256=request_sha256,
+ response_id=_text(response_payload.get("id")) if isinstance(response_payload, Mapping) else "",
+ ),
+ ) from exc
+ choices = response_payload.get("choices") if isinstance(response_payload, Mapping) else None
+ if not isinstance(choices, list) or len(choices) != 1 or not isinstance(choices[0], Mapping):
+ raise RecallPlannerResponseError("recall role planner response must contain exactly one choice", response_content=raw_http, request_metadata=_metadata_v4(physical_call_id=physical_call_id, key_index=key_index, started=started, finish_reason="invalid_response", content=raw_http, usage=usage, request_sha256=request_sha256, http_status=http_status, response_id=_text(response_payload.get("id"))))
+ choice, message = choices[0], choices[0].get("message")
+ content = message.get("content") if isinstance(message, Mapping) else None
+ finish_reason = _text(choice.get("finish_reason"))
+ metadata = _metadata_v4(physical_call_id=physical_call_id, key_index=key_index, started=started, finish_reason=finish_reason, content=content if isinstance(content, str) else raw_http, usage=usage, request_sha256=request_sha256, http_status=http_status, response_id=_text(response_payload.get("id")))
+ if finish_reason != "stop" or not isinstance(content, str):
+ raise RecallPlannerResponseError("recall role planner response did not finish with a JSON string", response_content=raw_http, request_metadata=metadata)
+ try:
+ plan = validate_recall_role_plan(json.loads(content))
+ except (json.JSONDecodeError, RecallPlannerError) as exc:
+ raise RecallPlannerResponseError(f"recall role planner returned invalid RecallRolePlan: {exc}", response_content=content, request_metadata=metadata) from exc
+ if plan["query_kind"] not in QUERY_KINDS:
+ metadata["validation_warnings"] = [
+ {
+ "code": "noncanonical_query_kind",
+ "query_kind": plan["query_kind"],
+ "disposition": "preserved_as_standard_query",
+ }
+ ]
+ return plan, metadata
+
+
+RecallRolePlanner = DeepSeekFlashRecallRolePlanner
+DeepSeekFlashRecallPlanner = DeepSeekFlashRecallRolePlanner
+validate_plan = validate_recall_role_plan
+
+
+def _source_parents(item: Mapping[str, Any]) -> list[dict[str, Any]]:
+ raw = item.get("source_parents")
+ if raw is None and isinstance(item.get("source_parent"), Mapping):
+ raw = [item["source_parent"]]
+ if not isinstance(raw, Sequence) or isinstance(raw, (str, bytes)) or not raw:
+ raise RecallPlannerError("slow candidate requires source_parents")
+ output: list[dict[str, Any]] = []
+ for parent in raw:
+ if not isinstance(parent, Mapping):
+ raise RecallPlannerError("slow candidate source_parent must be an object")
+ try:
+ start, end = int(parent["evidence_char_start"]), int(parent["evidence_char_end"])
+ session, chunk = int(parent["session_index"]), int(parent["parent_chunk_index"])
+ except (KeyError, TypeError, ValueError) as exc:
+ raise RecallPlannerError("slow candidate source_parent lacks an integer evidence span") from exc
+ if start < 0 or end <= start:
+ raise RecallPlannerError("slow candidate source_parent has an invalid evidence span")
+ output.append({**dict(parent), "session_index": session, "parent_chunk_index": chunk, "evidence_char_start": start, "evidence_char_end": end})
+ return output
+
+
+def apply_recall_role_plan(plan: Mapping[str, Any], source_candidates: Sequence[Mapping[str, Any]], fast_candidates: Sequence[Mapping[str, Any]], slow_candidates: Sequence[Mapping[str, Any]]) -> list[dict[str, Any]]:
+ """Create units without gating; only source candidates can become evidence."""
+ normalized = validate_recall_role_plan(plan)
+ raw_priorities = {layer: normalized["layers"][layer]["weight"] * ROLE_PRIORS[normalized["layers"][layer]["role"]] for layer in LAYER_NAMES}
+ total_priority = sum(raw_priorities.values())
+ if total_priority <= 0.0:
+ raise RecallPlannerError("recall role plan cannot assign zero normalized priority to every layer")
+ normalized_priorities = {layer: raw_priorities[layer] / total_priority for layer in LAYER_NAMES}
+ units: list[dict[str, Any]] = []
+ for layer, candidates, key, kind in (("source", source_candidates, "source_candidate", "source_window"), ("fast", fast_candidates, "fast_candidate", "fast_atomic"), ("slow", slow_candidates, "slow_candidate", "slow_capsule")):
+ for layer_rank, candidate in enumerate(candidates, start=1):
+ if not isinstance(candidate, Mapping):
+ raise RecallPlannerError(f"{layer} candidate must be a mapping")
+ item = dict(candidate)
+ if layer == "slow":
+ item["source_parents"] = _source_parents(candidate)
+ within_layer_score = 1.0 / float(layer_rank)
+ units.append({"unit_type": kind, "layer": layer, "canonical_slot": _text(candidate.get("canonical_slot")) or layer, key: item, "layer_role": normalized["layers"][layer]["role"], "layer_weight": normalized["layers"][layer]["weight"], "role_prior": ROLE_PRIORS[normalized["layers"][layer]["role"]], "normalized_priority": normalized_priorities[layer], "layer_rank": layer_rank, "within_layer_score": within_layer_score, "priority_score": normalized_priorities[layer] * within_layer_score})
+ if not units:
+ raise RecallPlannerError("recall role plan produced no candidate units")
+ return units
+
+
+def layer_weight(plan: Mapping[str, Any], layer: str) -> float:
+ normalized = validate_recall_role_plan(plan)
+ if layer not in LAYER_NAMES:
+ raise RecallPlannerError(f"unknown recall layer: {layer!r}")
+ return float(normalized["layers"][layer]["weight"])
+
+
+def normalized_layer_priorities(plan: Mapping[str, Any]) -> dict[str, float]:
+ normalized = validate_recall_role_plan(plan)
+ raw = {layer: normalized["layers"][layer]["weight"] * ROLE_PRIORS[normalized["layers"][layer]["role"]] for layer in LAYER_NAMES}
+ total = sum(raw.values())
+ if total <= 0.0:
+ raise RecallPlannerError("recall role plan cannot assign zero normalized priority to every layer")
+ return {layer: raw[layer] / total for layer in LAYER_NAMES}
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_regression_gate.py b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_regression_gate.py
new file mode 100644
index 0000000000000000000000000000000000000000..56e7d2cb2f9884cc6ff1c5b086628eca5730e6a7
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_regression_gate.py
@@ -0,0 +1,488 @@
+#!/usr/bin/env python3
+"""Zero-API promotion gate for paired TMCRA official-judge runs."""
+
+from __future__ import annotations
+
+import argparse
+import json
+from collections import defaultdict
+from collections.abc import Mapping, Sequence
+from pathlib import Path
+from typing import Any
+
+
+SCHEMA_VERSION = "tmcra.v4.regression-gate.1"
+MOVEMENTS = ("kept_correct", "improved", "regressed", "kept_wrong")
+DEFAULT_CASES = Path(__file__).with_name("tmcra_v4_regression_cases.json")
+
+
+class GateInputError(ValueError):
+ """Raised internally for malformed or non-pairable gate inputs."""
+
+
+def _qid(row: Mapping[str, Any], source: str, line: int | None = None) -> str:
+ qid_value = row.get("qid")
+ question_id = row.get("question_id")
+ if qid_value is not None and question_id is not None:
+ if str(qid_value).strip() != str(question_id).strip():
+ location = f" line {line}" if line is not None else ""
+ raise GateInputError(f"{source}{location}: qid and question_id disagree")
+ value = qid_value if qid_value is not None else question_id
+ result = str(value or "").strip()
+ if not result:
+ location = f" line {line}" if line is not None else ""
+ raise GateInputError(f"{source}{location}: missing qid")
+ return result
+
+
+def _read_jsonl(path: Path, source: str) -> list[dict[str, Any]]:
+ try:
+ lines = path.read_text(encoding="utf-8").splitlines()
+ except OSError as exc:
+ raise GateInputError(f"cannot read {source}: {exc}") from exc
+ rows: list[dict[str, Any]] = []
+ seen: set[str] = set()
+ for line_number, line in enumerate(lines, 1):
+ if not line.strip():
+ continue
+ try:
+ row = json.loads(line)
+ except json.JSONDecodeError as exc:
+ raise GateInputError(f"{source} line {line_number}: invalid JSON") from exc
+ if not isinstance(row, dict):
+ raise GateInputError(f"{source} line {line_number}: expected an object")
+ current_qid = _qid(row, source, line_number)
+ if current_qid in seen:
+ raise GateInputError(f"{source}: duplicate qid {current_qid}")
+ seen.add(current_qid)
+ rows.append({**row, "qid": current_qid})
+ if not rows:
+ raise GateInputError(f"{source}: no rows")
+ return rows
+
+
+def _reference_rows(value: Any, source: str) -> list[dict[str, Any]]:
+ if isinstance(value, list):
+ raw_rows = value
+ elif isinstance(value, dict):
+ raw_rows: Any = None
+ for key in ("questions", "data", "rows", "references"):
+ if key in value:
+ raw_rows = value[key]
+ break
+ if raw_rows is None and all(isinstance(item, dict) for item in value.values()):
+ raw_rows = [dict(item, qid=key) for key, item in value.items()]
+ if raw_rows is None:
+ raw_rows = [value] if ("qid" in value or "question_id" in value) else None
+ if not isinstance(raw_rows, list):
+ raise GateInputError(f"{source}: expected a list of reference rows")
+ else:
+ raise GateInputError(f"{source}: expected a JSON object or list")
+
+ rows: list[dict[str, Any]] = []
+ seen: set[str] = set()
+ for index, row in enumerate(raw_rows, 1):
+ if not isinstance(row, dict):
+ raise GateInputError(f"{source} row {index}: expected an object")
+ current_qid = _qid(row, source, index)
+ if current_qid in seen:
+ raise GateInputError(f"{source}: duplicate qid {current_qid}")
+ answer_sessions = row.get("answer_session_ids")
+ if not isinstance(answer_sessions, list) or not answer_sessions:
+ raise GateInputError(
+ f"{source} row {index} ({current_qid}): answer_session_ids must be non-empty"
+ )
+ normalized_sessions = {str(item).strip() for item in answer_sessions if str(item).strip()}
+ if len(normalized_sessions) != len(answer_sessions):
+ raise GateInputError(
+ f"{source} row {index} ({current_qid}): answer_session_ids contains blank or duplicate values"
+ )
+ question_type = str(row.get("question_type") or row.get("type") or "unknown").strip()
+ rows.append(
+ {
+ **row,
+ "qid": current_qid,
+ "answer_session_ids": sorted(normalized_sessions),
+ "question_type": question_type or "unknown",
+ }
+ )
+ seen.add(current_qid)
+ if not rows:
+ raise GateInputError(f"{source}: no rows")
+ return rows
+
+
+def _read_reference(path: Path) -> list[dict[str, Any]]:
+ try:
+ value = json.loads(path.read_text(encoding="utf-8"))
+ except OSError as exc:
+ raise GateInputError(f"cannot read reference: {exc}") from exc
+ except json.JSONDecodeError as exc:
+ raise GateInputError("reference: invalid JSON") from exc
+ return _reference_rows(value, "reference")
+
+
+def _read_cases(path: Path) -> dict[str, dict[str, list[str]]]:
+ try:
+ value = json.loads(path.read_text(encoding="utf-8"))
+ except OSError as exc:
+ raise GateInputError(f"cannot read cases: {exc}") from exc
+ except json.JSONDecodeError as exc:
+ raise GateInputError("cases: invalid JSON") from exc
+ if not isinstance(value, dict) or value.get("schema_version") != "tmcra.v4.regression-cases.1":
+ raise GateInputError("cases: unsupported schema_version")
+ corpora = value.get("corpora")
+ if not isinstance(corpora, dict) or not corpora:
+ raise GateInputError("cases: corpora must be a non-empty object")
+ result: dict[str, dict[str, list[str]]] = {}
+ for name, corpus in corpora.items():
+ if not isinstance(corpus, dict):
+ raise GateInputError(f"cases corpus {name}: expected an object")
+ parsed: dict[str, list[str]] = {}
+ for key in ("regressed_qids", "improved_qids"):
+ qids = corpus.get(key)
+ if not isinstance(qids, list) or any(not str(qid).strip() for qid in qids):
+ raise GateInputError(f"cases corpus {name}: {key} must be a list of qids")
+ normalized = [str(qid).strip() for qid in qids]
+ if len(normalized) != len(set(normalized)):
+ raise GateInputError(f"cases corpus {name}: {key} contains duplicates")
+ parsed[key] = normalized
+ if set(parsed["regressed_qids"]) & set(parsed["improved_qids"]):
+ raise GateInputError(f"cases corpus {name}: movement lists overlap")
+ result[str(name)] = parsed
+ return result
+
+
+def official_judge_correct(row: Mapping[str, Any], source: str = "official judge") -> bool:
+ if str(row.get("error") or "").strip():
+ raise GateInputError(f"{source} {row.get('qid', '')}: row contains an error")
+ label = row.get("autoeval_label")
+ if not isinstance(label, Mapping) or type(label.get("label")) is not bool:
+ raise GateInputError(
+ f"{source} {row.get('qid', '')}: autoeval_label.label must be boolean"
+ )
+ return bool(label["label"])
+
+
+def _index(rows: Sequence[Mapping[str, Any]]) -> dict[str, Mapping[str, Any]]:
+ return {str(row["qid"]): row for row in rows}
+
+
+def _pairing_report(
+ baseline: Sequence[Mapping[str, Any]],
+ candidate: Sequence[Mapping[str, Any]],
+ evidence: Sequence[Mapping[str, Any]],
+ reference: Sequence[Mapping[str, Any]],
+) -> tuple[bool, dict[str, Any], set[str]]:
+ sets = {
+ "baseline_judge": set(row["qid"] for row in baseline),
+ "candidate_judge": set(row["qid"] for row in candidate),
+ "candidate_evidence": set(row["qid"] for row in evidence),
+ "reference": set(row["qid"] for row in reference),
+ }
+ expected = sets["baseline_judge"]
+ compared_sources = ("candidate_judge", "candidate_evidence")
+ exact = all(sets[name] == expected for name in compared_sources) and expected.issubset(
+ sets["reference"]
+ )
+ report = {
+ "exact": exact,
+ "counts": {name: len(qids) for name, qids in sets.items()},
+ "missing_by_source": {
+ name: sorted(expected - qids)
+ for name, qids in sets.items()
+ if expected - qids
+ },
+ "extra_by_source": {
+ name: sorted(qids - expected)
+ for name, qids in sets.items()
+ if name != "reference" and qids - expected
+ },
+ "reference_is_superset": expected.issubset(sets["reference"]),
+ }
+ return exact, report, expected
+
+
+def _empty_report(error: str | None = None) -> dict[str, Any]:
+ report: dict[str, Any] = {
+ "schema_version": SCHEMA_VERSION,
+ "status": "blocked",
+ "promotion_eligible": False,
+ "errors": [error] if error else [],
+ }
+ if error:
+ report["checks"] = {
+ "exact_qid_pairing": False,
+ "regressed_source_coverage": False,
+ "candidate_accuracy": False,
+ "max_regressions": False,
+ "min_improvements": False,
+ }
+ return report
+
+
+def evaluate_gate(
+ baseline_judge_path: Path,
+ candidate_judge_path: Path,
+ candidate_evidence_path: Path,
+ reference_path: Path,
+ *,
+ cases_path: Path = DEFAULT_CASES,
+ min_candidate_accuracy: float | None = None,
+ max_regressions: int = 0,
+ min_improvements: int = 0,
+) -> dict[str, Any]:
+ try:
+ if max_regressions < 0 or min_improvements < 0:
+ raise GateInputError("promotion thresholds must be non-negative")
+ if min_candidate_accuracy is not None and not 0 <= min_candidate_accuracy <= 1:
+ raise GateInputError("min_candidate_accuracy must be between 0 and 1")
+ baseline = _read_jsonl(baseline_judge_path, "baseline judge")
+ candidate = _read_jsonl(candidate_judge_path, "candidate judge")
+ evidence = _read_jsonl(candidate_evidence_path, "candidate evidence")
+ reference = _read_reference(reference_path)
+ cases = _read_cases(cases_path)
+ baseline_by_qid = _index(baseline)
+ candidate_by_qid = _index(candidate)
+ evidence_by_qid = _index(evidence)
+ reference_by_qid = _index(reference)
+ exact, pairing, qids = _pairing_report(baseline, candidate, evidence, reference)
+ if not exact:
+ report = _empty_report("inputs do not have exactly matching qid sets")
+ report["pairing"] = pairing
+ report["known_corpus_outcomes"] = _known_corpus_outcomes(
+ cases, baseline_by_qid, candidate_by_qid, reference_by_qid
+ )
+ return report
+
+ rows: list[dict[str, Any]] = []
+ for qid in sorted(qids):
+ baseline_correct = official_judge_correct(baseline_by_qid[qid], "baseline judge")
+ candidate_correct = official_judge_correct(candidate_by_qid[qid], "candidate judge")
+ movement = _movement(baseline_correct, candidate_correct)
+ selected = evidence_by_qid[qid].get("selected_session_ids")
+ if not isinstance(selected, list):
+ raise GateInputError(f"candidate evidence {qid}: selected_session_ids must be a list")
+ selected_values = [str(item).strip() for item in selected]
+ if any(not item for item in selected_values) or len(set(selected_values)) != len(selected_values):
+ raise GateInputError(
+ f"candidate evidence {qid}: selected_session_ids contains blank or duplicate values"
+ )
+ selected_ids = set(selected_values)
+ answer_ids = set(reference_by_qid[qid]["answer_session_ids"])
+ missing_answer_session_ids = sorted(answer_ids - selected_ids)
+ covered_answer_session_count = len(answer_ids) - len(missing_answer_session_ids)
+ session_coverage = {
+ "complete": not missing_answer_session_ids,
+ "expected_session_count": len(answer_ids),
+ "selected_session_count": len(selected_ids),
+ "covered_session_count": covered_answer_session_count,
+ "hit_rate": round(covered_answer_session_count / len(answer_ids), 6),
+ "missing_session_ids": missing_answer_session_ids,
+ }
+ rows.append(
+ {
+ "qid": qid,
+ "question_type": reference_by_qid[qid]["question_type"],
+ "movement": movement,
+ "baseline_correct": baseline_correct,
+ "candidate_correct": candidate_correct,
+ "answer_session_ids": sorted(answer_ids),
+ "selected_session_ids": sorted(selected_ids),
+ "missing_answer_session_ids": missing_answer_session_ids,
+ "session_coverage": session_coverage,
+ "source_coverage_complete": session_coverage["complete"],
+ }
+ )
+
+ counts = {movement: sum(row["movement"] == movement for row in rows) for movement in MOVEMENTS}
+ total = len(rows)
+ baseline_correct_count = counts["kept_correct"] + counts["regressed"]
+ candidate_correct_count = counts["kept_correct"] + counts["improved"]
+ baseline_accuracy = baseline_correct_count / total
+ candidate_accuracy = candidate_correct_count / total
+ by_type: dict[str, dict[str, Any]] = {}
+ grouped: dict[str, list[dict[str, Any]]] = defaultdict(list)
+ for row in rows:
+ grouped[row["question_type"]].append(row)
+ for question_type, values in sorted(grouped.items()):
+ type_counts = {
+ movement: sum(row["movement"] == movement for row in values)
+ for movement in MOVEMENTS
+ }
+ type_total = len(values)
+ by_type[question_type] = {
+ "count": type_total,
+ "baseline_correct": sum(row["baseline_correct"] for row in values),
+ "candidate_correct": sum(row["candidate_correct"] for row in values),
+ "baseline_accuracy": round(sum(row["baseline_correct"] for row in values) / type_total, 6),
+ "candidate_accuracy": round(sum(row["candidate_correct"] for row in values) / type_total, 6),
+ **type_counts,
+ }
+
+ regressed = [row for row in rows if row["movement"] == "regressed"]
+ coverage_failures = [
+ {"qid": row["qid"], "missing_answer_session_ids": sorted(
+ set(row["answer_session_ids"]) - set(row["selected_session_ids"])
+ )}
+ for row in regressed
+ if not row["source_coverage_complete"]
+ ]
+ accuracy_floor = baseline_accuracy if min_candidate_accuracy is None else min_candidate_accuracy
+ checks = {
+ "exact_qid_pairing": True,
+ "regressed_source_coverage": not coverage_failures,
+ "candidate_accuracy": candidate_accuracy >= accuracy_floor,
+ "max_regressions": counts["regressed"] <= max_regressions,
+ "min_improvements": counts["improved"] >= min_improvements,
+ }
+ report = {
+ "schema_version": SCHEMA_VERSION,
+ "status": "passed" if all(checks.values()) else "blocked",
+ "promotion_eligible": all(checks.values()),
+ "errors": [],
+ "pairing": pairing,
+ "thresholds": {
+ "candidate_accuracy_min": accuracy_floor,
+ "candidate_accuracy_min_source": "baseline" if min_candidate_accuracy is None else "cli",
+ "max_regressions": max_regressions,
+ "min_improvements": min_improvements,
+ },
+ "checks": checks,
+ "metrics": {
+ "evaluated_count": total,
+ "baseline_correct": baseline_correct_count,
+ "candidate_correct": candidate_correct_count,
+ "baseline_accuracy": round(baseline_accuracy, 6),
+ "candidate_accuracy": round(candidate_accuracy, 6),
+ **counts,
+ },
+ "by_question_type": by_type,
+ "source_coverage": {
+ "reference_backed": True,
+ "reference_source": "reference.answer_session_ids",
+ "regressed_count": len(regressed),
+ "regressed_complete_count": len(regressed) - len(coverage_failures),
+ "regressed_complete_rate": round(
+ (len(regressed) - len(coverage_failures)) / len(regressed), 6
+ ) if regressed else 1.0,
+ "regressed_failures": coverage_failures,
+ "all_qids_complete_count": sum(row["source_coverage_complete"] for row in rows),
+ "all_qids_complete_rate": round(
+ sum(row["source_coverage_complete"] for row in rows) / total, 6
+ ),
+ "missing_session_ids_by_qid": {
+ row["qid"]: row["missing_answer_session_ids"]
+ for row in rows
+ if row["missing_answer_session_ids"]
+ },
+ },
+ "reference_backed_session_coverage": {
+ "reference_source": "reference.answer_session_ids",
+ "evaluated_count": total,
+ "complete_count": sum(row["source_coverage_complete"] for row in rows),
+ "complete_rate": round(
+ sum(row["source_coverage_complete"] for row in rows) / total, 6
+ ),
+ "covered_session_count": sum(
+ row["session_coverage"]["covered_session_count"] for row in rows
+ ),
+ "expected_session_count": sum(
+ row["session_coverage"]["expected_session_count"] for row in rows
+ ),
+ "missing_session_ids_by_qid": {
+ row["qid"]: row["missing_answer_session_ids"]
+ for row in rows
+ if row["missing_answer_session_ids"]
+ },
+ },
+ "known_corpus_outcomes": _known_corpus_outcomes(
+ cases, baseline_by_qid, candidate_by_qid, reference_by_qid
+ ),
+ }
+ return report
+ except GateInputError as exc:
+ return _empty_report(str(exc))
+
+
+def _movement(baseline_correct: bool, candidate_correct: bool) -> str:
+ if baseline_correct and candidate_correct:
+ return "kept_correct"
+ if not baseline_correct and candidate_correct:
+ return "improved"
+ if baseline_correct and not candidate_correct:
+ return "regressed"
+ return "kept_wrong"
+
+
+def _known_corpus_outcomes(
+ cases: Mapping[str, Mapping[str, Sequence[str]]],
+ baseline: Mapping[str, Mapping[str, Any]],
+ candidate: Mapping[str, Mapping[str, Any]],
+ reference: Mapping[str, Mapping[str, Any]],
+) -> dict[str, Any]:
+ outcomes: dict[str, Any] = {}
+ for name, corpus in sorted(cases.items()):
+ expected_by_qid = {
+ qid: "regressed" for qid in corpus["regressed_qids"]
+ }
+ expected_by_qid.update({qid: "improved" for qid in corpus["improved_qids"]})
+ present = sorted(set(expected_by_qid) & set(baseline) & set(candidate))
+ observed = []
+ for qid in present:
+ try:
+ baseline_correct = official_judge_correct(baseline[qid], "baseline judge")
+ candidate_correct = official_judge_correct(candidate[qid], "candidate judge")
+ except GateInputError:
+ continue
+ observed.append({
+ "qid": qid,
+ "expected_movement": expected_by_qid[qid],
+ "actual_movement": _movement(baseline_correct, candidate_correct),
+ "matches_expected": _movement(baseline_correct, candidate_correct) == expected_by_qid[qid],
+ "question_type": reference.get(qid, {}).get("question_type", "unknown"),
+ })
+ outcomes[name] = {
+ "expected_regressed_count": len(corpus["regressed_qids"]),
+ "expected_improved_count": len(corpus["improved_qids"]),
+ "present_count": len(present),
+ "missing_qids": sorted(set(expected_by_qid) - set(present)),
+ "observed": observed,
+ }
+ return outcomes
+
+
+def _write_report(path: Path, report: Mapping[str, Any]) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_text(json.dumps(report, indent=2, sort_keys=True) + "\n", encoding="utf-8")
+
+
+def main(argv: Sequence[str] | None = None) -> int:
+ parser = argparse.ArgumentParser(description="Run the zero-API TMCRA V4 regression promotion gate")
+ parser.add_argument("--baseline-judge", "--baseline", dest="baseline_judge", type=Path, required=True)
+ parser.add_argument("--candidate-judge", "--candidate", dest="candidate_judge", type=Path, required=True)
+ parser.add_argument("--candidate-evidence", "--evidence", dest="candidate_evidence", type=Path, required=True)
+ parser.add_argument("--reference", "--reference-json", dest="reference", type=Path, required=True)
+ parser.add_argument("--cases", type=Path, default=DEFAULT_CASES)
+ parser.add_argument("--min-candidate-accuracy", "--candidate-accuracy-min", dest="min_candidate_accuracy", type=float)
+ parser.add_argument("--max-regressions", type=int, default=0)
+ parser.add_argument("--min-improvements", type=int, default=0)
+ parser.add_argument("--output", type=Path, required=True)
+ args = parser.parse_args(argv)
+ report = evaluate_gate(
+ args.baseline_judge,
+ args.candidate_judge,
+ args.candidate_evidence,
+ args.reference,
+ cases_path=args.cases,
+ min_candidate_accuracy=args.min_candidate_accuracy,
+ max_regressions=args.max_regressions,
+ min_improvements=args.min_improvements,
+ )
+ _write_report(args.output, report)
+ print(json.dumps(report, sort_keys=True))
+ return 0 if report["status"] == "passed" else 1
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_route_policy.py b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_route_policy.py
new file mode 100644
index 0000000000000000000000000000000000000000..b642097e20b78eeba0da6d81d3946ee2a0202cbf
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_route_policy.py
@@ -0,0 +1,543 @@
+"""Fail-closed routing policy for the promoted TMCRA answer lane."""
+
+from __future__ import annotations
+
+import json
+from collections.abc import Mapping, Sequence
+from pathlib import Path
+from typing import Any
+
+from tmcra_v4_evidence_operations import (
+ PACKET_COMPILER_VERSION,
+ PACKET_SCHEMA,
+ PLAN_SCHEMA,
+)
+
+
+POLICY_SCHEMA = "tmcra.v4.production-route-policy.2"
+RETRIEVAL_CONTRACT_SCHEMA = "tmcra.v4.layered-retrieval-contract.3"
+RETRIEVAL_INVENTORY_COUNT_FIELDS = (
+ "source",
+ "fast",
+ "fast_semantic",
+ "slow",
+ "slow_capsule_heads",
+ "slow_summaries",
+ "slow_claims",
+ "slow_ranked_claims",
+)
+PRODUCTION_LANE = "layered_memory_operation_bound_gpt54"
+PRODUCTION_COMPOSITION_MODE = "layered"
+PRODUCTION_PACKING_BUDGET_MODE = "fixed"
+PRODUCTION_FINAL_TOP_K = 8
+SOURCE_COVERAGE_TRACE_K = 24
+PRODUCTION_PACKET_FIELD = "compiled_evidence_packet"
+PRODUCTION_ANSWER_PROTOCOL = "evidence_operation_bound_v6"
+PRODUCTION_ANSWER_MODEL = "gpt-5.4"
+PRODUCTION_ANSWER_RUNNER = "run_tmcra_v4_gpt54_answers.py"
+SHADOW_PACKET_FIELD = "semantic_evidence_packet"
+SHADOW_ANSWER_PROTOCOL = "semantic_advisory_bound_v2"
+RETRIEVAL_MODES = {"layered", "source-only-diagnostic"}
+FORBIDDEN_ANSWER_FACING_FIELDS = {
+ "answer",
+ "gold_answer",
+ "expected_answer",
+ "answer_session_ids",
+ "labels",
+ "supervision",
+}
+
+
+class RoutePolicyError(RuntimeError):
+ pass
+
+
+def _text(value: Any) -> str:
+ return value.strip() if isinstance(value, str) else ""
+
+
+def _source_text(value: Any) -> str:
+ return value if isinstance(value, str) else ""
+
+
+def _source_identity(item: Mapping[str, Any]) -> tuple[Any, ...]:
+ session_id = _text(item.get("session_id"))
+ parent_chunk_index = int(item.get("parent_chunk_index", 0))
+ source_group_id = _text(item.get("source_group_id")) or (
+ f"source-group::{session_id}:{parent_chunk_index}"
+ )
+ context = item.get("source_group_context") or []
+ if not isinstance(context, Sequence) or isinstance(context, (str, bytes)):
+ raise RoutePolicyError("Source group context is not an array")
+ context_identity = tuple(
+ (
+ _text(member.get("relationship")),
+ int(member.get("parent_distance", 0)),
+ _text(member.get("session_id")),
+ int(member.get("session_index", 0)),
+ int(member.get("parent_chunk_index", 0)),
+ _text(member.get("source_record_id")),
+ int(member.get("source_char_start", 0)),
+ int(
+ member.get(
+ "source_char_end",
+ len(member.get("text")) if isinstance(member.get("text"), str) else 0,
+ )
+ ),
+ _source_text(member.get("text")),
+ )
+ for member in context
+ if isinstance(member, Mapping)
+ )
+ if len(context_identity) != len(context):
+ raise RoutePolicyError("Source group context contains a non-object member")
+ return (
+ _text(item.get("db_path")),
+ _text(item.get("scope_id")),
+ _text(item.get("source_record_id")),
+ session_id,
+ int(item.get("session_index", 0)),
+ int(item.get("parent_chunk_index", 0)),
+ int(item.get("subchunk_index", 0)),
+ int(item.get("source_char_start", 0)),
+ int(
+ item.get(
+ "source_char_end",
+ len(item.get("text")) if isinstance(item.get("text"), str) else 0,
+ )
+ ),
+ _source_text(item.get("text")),
+ source_group_id,
+ context_identity,
+ )
+
+
+def assert_production_answer_runner(path: Path) -> None:
+ if path.name != PRODUCTION_ANSWER_RUNNER:
+ raise RoutePolicyError(
+ f"production answer runner must be {PRODUCTION_ANSWER_RUNNER}, got {path.name or path}"
+ )
+
+
+def validate_production_retrieval_mode(
+ composition_mode: Any, *, execution_lane: Any
+) -> None:
+ lane = _text(execution_lane)
+ mode = _text(composition_mode)
+ if lane not in {"production", "diagnostic"}:
+ raise RoutePolicyError("retrieval execution lane must be production or diagnostic")
+ if mode not in RETRIEVAL_MODES:
+ raise RoutePolicyError(f"retrieval composition mode is invalid: {mode!r}")
+ if lane == "production" and mode != PRODUCTION_COMPOSITION_MODE:
+ raise RoutePolicyError(
+ "production retrieval requires layered composition; "
+ f"got {mode or ''!r}"
+ )
+
+
+def validate_production_packing_budget(
+ packing_budget_mode: Any,
+ top_k: Any,
+ *,
+ execution_lane: Any,
+) -> None:
+ lane = _text(execution_lane)
+ mode = _text(packing_budget_mode)
+ if lane not in {"production", "diagnostic"}:
+ raise RoutePolicyError("retrieval execution lane must be production or diagnostic")
+ if mode not in {"fixed", "adaptive"}:
+ raise RoutePolicyError(f"retrieval packing budget mode is invalid: {mode!r}")
+ if isinstance(top_k, bool) or not isinstance(top_k, int) or top_k <= 0:
+ raise RoutePolicyError("retrieval packing budget must be a positive integer")
+ if lane == "production" and (
+ mode != PRODUCTION_PACKING_BUDGET_MODE or top_k != PRODUCTION_FINAL_TOP_K
+ ):
+ raise RoutePolicyError(
+ "production retrieval requires a fixed Top8 packing budget; "
+ f"got mode={mode or ''!r}, top_k={top_k!r}"
+ )
+
+
+def _canonical_json(value: Any) -> str:
+ return json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
+
+
+def _nonnegative_int(value: Any, *, field: str) -> int:
+ if isinstance(value, bool) or not isinstance(value, int) or value < 0:
+ raise RoutePolicyError(f"{field} must be a nonnegative integer")
+ return value
+
+
+def _layer_contribution_counts(windows: Sequence[Mapping[str, Any]]) -> dict[str, int]:
+ counts = {"source": 0, "fast": 0, "slow": 0}
+ for window in windows:
+ metadata = window.get("retrieval_metadata")
+ contributions = metadata.get("layer_contributions") if isinstance(metadata, Mapping) else None
+ if not isinstance(contributions, Sequence) or isinstance(contributions, (str, bytes)):
+ raise RoutePolicyError("production evidence window lacks layer contributions")
+ seen: set[str] = set()
+ for contribution in contributions:
+ if not isinstance(contribution, Mapping):
+ raise RoutePolicyError("production layer contribution is not an object")
+ layer = _text(contribution.get("layer"))
+ if layer not in counts:
+ raise RoutePolicyError(f"production evidence contains unknown layer {layer!r}")
+ seen.add(layer)
+ for layer in seen:
+ counts[layer] += 1
+ return counts
+
+
+def _validate_retrieval_contract(
+ row: Mapping[str, Any],
+ windows: Sequence[Mapping[str, Any]],
+ *,
+ qid: str,
+ expected_lane: str = "production",
+) -> dict[str, Any]:
+ contract = row.get("retrieval_contract")
+ if not isinstance(contract, Mapping):
+ raise RoutePolicyError(f"{qid}: production evidence lacks retrieval contract")
+ if contract.get("schema_version") != RETRIEVAL_CONTRACT_SCHEMA:
+ raise RoutePolicyError(f"{qid}: retrieval contract schema is invalid")
+ validate_production_retrieval_mode(
+ contract.get("composition_mode"), execution_lane=contract.get("execution_lane")
+ )
+ actual_lane = _text(contract.get("execution_lane"))
+ if actual_lane != expected_lane:
+ if expected_lane == "production" and actual_lane == "diagnostic":
+ raise RoutePolicyError(
+ f"{qid}: diagnostic retrieval cannot enter production evaluation"
+ )
+ raise RoutePolicyError(
+ f"{qid}: retrieval lane is {actual_lane!r}, expected {expected_lane!r}"
+ )
+
+ inventory = contract.get("inventory_counts")
+ paths = contract.get("candidate_paths_executed")
+ required = contract.get("required_selected_layers")
+ selected = contract.get("selected_layer_window_counts")
+ packing_budget_mode = _text(contract.get("packing_budget_mode"))
+ packing_budget = contract.get("packing_budget")
+ source_trace_k = contract.get("source_coverage_trace_k")
+ final_window_count = contract.get("final_window_count")
+ try:
+ validate_production_packing_budget(
+ packing_budget_mode,
+ packing_budget,
+ execution_lane=actual_lane,
+ )
+ except RoutePolicyError as exc:
+ raise RoutePolicyError(f"{qid}: {exc}") from exc
+ if source_trace_k != SOURCE_COVERAGE_TRACE_K:
+ raise RoutePolicyError(
+ f"{qid}: Source coverage trace contract must be Top{SOURCE_COVERAGE_TRACE_K}"
+ )
+ if final_window_count != len(windows):
+ raise RoutePolicyError(f"{qid}: final window count does not match evidence")
+ if expected_lane == "production" and len(windows) > PRODUCTION_FINAL_TOP_K:
+ raise RoutePolicyError(
+ f"{qid}: production final evidence exceeds Top{PRODUCTION_FINAL_TOP_K}"
+ )
+ if not isinstance(inventory, Mapping) or set(inventory) != set(
+ RETRIEVAL_INVENTORY_COUNT_FIELDS
+ ):
+ raise RoutePolicyError(f"{qid}: retrieval inventory contract is invalid")
+ if not isinstance(paths, Mapping) or set(paths) != {"source", "fast", "slow"}:
+ raise RoutePolicyError(f"{qid}: candidate path execution contract is invalid")
+ if any(not isinstance(paths[layer], bool) for layer in ("source", "fast", "slow")):
+ raise RoutePolicyError(f"{qid}: candidate path execution flags must be booleans")
+ if not isinstance(required, Sequence) or isinstance(required, (str, bytes)):
+ raise RoutePolicyError(f"{qid}: required selected layers are invalid")
+ required_values = [_text(value) for value in required]
+ required_layers = set(required_values)
+ if (
+ not required_layers
+ or len(required_values) != len(required_layers)
+ or not required_layers.issubset({"source", "fast", "slow"})
+ ):
+ raise RoutePolicyError(f"{qid}: required selected layers are invalid")
+ inventory_counts = {
+ layer: _nonnegative_int(
+ inventory[layer], field=f"{qid}: inventory_counts.{layer}"
+ )
+ for layer in RETRIEVAL_INVENTORY_COUNT_FIELDS
+ }
+ if inventory_counts["fast_semantic"] > inventory_counts["fast"]:
+ raise RoutePolicyError(
+ f"{qid}: semantic Fast shortlist exceeds the Fast shortlist"
+ )
+ if (
+ inventory_counts["slow_capsule_heads"]
+ != inventory_counts["slow_summaries"]
+ or inventory_counts["slow_ranked_claims"] != inventory_counts["slow"]
+ or inventory_counts["slow_claims"]
+ < inventory_counts["slow_capsule_heads"]
+ or inventory_counts["slow_ranked_claims"]
+ > inventory_counts["slow_claims"]
+ or (
+ inventory_counts["slow_claims"] > 0
+ and inventory_counts["slow_capsule_heads"] == 0
+ )
+ ):
+ raise RoutePolicyError(
+ f"{qid}: Slow summary/claim inventory counts are inconsistent"
+ )
+ expected_required = {
+ layer
+ for layer in ("source", "fast", "slow")
+ if inventory_counts[layer] > 0
+ }
+ if expected_lane == "production" and required_layers != expected_required:
+ raise RoutePolicyError(
+ f"{qid}: required selected layers do not match nonempty inventories"
+ )
+ actual = _layer_contribution_counts(windows)
+ if not isinstance(selected, Mapping) or set(selected) != set(actual):
+ raise RoutePolicyError(f"{qid}: selected layer counts are invalid")
+ selected_counts = {
+ layer: _nonnegative_int(
+ selected[layer], field=f"{qid}: selected_layer_window_counts.{layer}"
+ )
+ for layer in actual
+ }
+ if selected_counts != actual:
+ raise RoutePolicyError(f"{qid}: selected layer counts do not match final evidence")
+ missing = sorted(layer for layer in required_layers if actual[layer] <= 0)
+ if missing:
+ raise RoutePolicyError(
+ f"{qid}: production evidence omitted required layers: {','.join(missing)}"
+ )
+ for layer in ("source", "fast", "slow"):
+ if inventory_counts[layer] > 0 and paths.get(layer) is not True:
+ raise RoutePolicyError(f"{qid}: available {layer} candidate path did not execute")
+
+ if actual["slow"] > 0:
+ slow_contexts = [
+ context
+ for window in windows
+ for context in list(window.get("memory_contexts") or [])
+ if isinstance(context, Mapping) and _text(context.get("role")) == "slow_context"
+ ]
+ if not slow_contexts or any(
+ not _text(context.get("capsule_id"))
+ or not _text(context.get("claim_id"))
+ or not _text(context.get("claim_text"))
+ or not isinstance(context.get("provenance"), Mapping)
+ for context in slow_contexts
+ ):
+ raise RoutePolicyError(f"{qid}: slow contribution lacks auditable claim context")
+ if actual["fast"] > 0 and inventory_counts["fast_semantic"] > 0:
+ fast_contexts = [
+ attachment
+ for window in windows
+ for attachment in list(window.get("attachments") or [])
+ if isinstance(attachment, Mapping)
+ and _text(attachment.get("role")) in {"fast_context", "override"}
+ ]
+ if not fast_contexts or any(
+ not _text(context.get("memory_id"))
+ or not _text(context.get("canonical_slot"))
+ or not _text(context.get("text"))
+ or not isinstance(context.get("provenance"), Mapping)
+ for context in fast_contexts
+ ):
+ raise RoutePolicyError(f"{qid}: fast contribution lacks auditable semantic context")
+ if any(
+ _text(context.get("role")) == "override"
+ and _text(context.get("precedence")) != "newer_fast_evidence"
+ for context in fast_contexts
+ ):
+ raise RoutePolicyError(
+ f"{qid}: Fast override lacks controller-verified precedence"
+ )
+ return actual
+
+
+def _validate_retrieval_rows(
+ rows: Sequence[Mapping[str, Any]], *, expected_lane: str
+) -> dict[str, Any]:
+ if not rows:
+ raise RoutePolicyError("retrieval evidence is empty")
+ qids: set[str] = set()
+ layer_window_counts = {"source": 0, "fast": 0, "slow": 0}
+ for index, row in enumerate(rows):
+ qid = _text(row.get("question_id"))
+ if not qid or qid in qids:
+ raise RoutePolicyError(
+ f"retrieval evidence row {index} has a missing or duplicate question_id"
+ )
+ qids.add(qid)
+ leaked = sorted(FORBIDDEN_ANSWER_FACING_FIELDS.intersection(row))
+ if leaked:
+ raise RoutePolicyError(
+ f"{qid}: answer-facing retrieval contains benchmark fields: {', '.join(leaked)}"
+ )
+ windows = row.get("evidence_windows")
+ if (
+ not isinstance(windows, Sequence)
+ or isinstance(windows, (str, bytes))
+ or not windows
+ ):
+ raise RoutePolicyError(f"{qid}: retrieval evidence windows are missing")
+ typed_windows = [item for item in windows if isinstance(item, Mapping)]
+ if len(typed_windows) != len(windows):
+ raise RoutePolicyError(f"{qid}: retrieval evidence contains a non-object window")
+ counts = _validate_retrieval_contract(
+ row,
+ typed_windows,
+ qid=qid,
+ expected_lane=expected_lane,
+ )
+ for layer, count in counts.items():
+ layer_window_counts[layer] += count
+ return {
+ "schema_version": RETRIEVAL_CONTRACT_SCHEMA,
+ "execution_lane": expected_lane,
+ "question_count": len(qids),
+ "layer_window_counts": layer_window_counts,
+ "ready_for_evidence_compilation": True,
+ }
+
+
+def validate_production_retrieval_rows(
+ rows: Sequence[Mapping[str, Any]],
+) -> dict[str, Any]:
+ return _validate_retrieval_rows(rows, expected_lane="production")
+
+
+def validate_diagnostic_retrieval_rows(
+ rows: Sequence[Mapping[str, Any]],
+) -> dict[str, Any]:
+ return _validate_retrieval_rows(rows, expected_lane="diagnostic")
+
+
+def validate_production_evidence(rows: Sequence[Mapping[str, Any]]) -> dict[str, Any]:
+ if not rows:
+ raise RoutePolicyError("production evidence is empty")
+ qids: list[str] = []
+ source_window_count = 0
+ layer_window_counts = {"source": 0, "fast": 0, "slow": 0}
+ for index, row in enumerate(rows):
+ qid = _text(row.get("question_id"))
+ if not qid or qid in qids:
+ raise RoutePolicyError(f"production evidence row {index} has a missing or duplicate question_id")
+ qids.append(qid)
+ if SHADOW_PACKET_FIELD in row:
+ raise RoutePolicyError(f"{qid}: semantic shadow packet is forbidden in the production lane")
+ packet = row.get(PRODUCTION_PACKET_FIELD)
+ if not isinstance(packet, Mapping) or packet.get("schema_version") != PACKET_SCHEMA:
+ raise RoutePolicyError(f"{qid}: production lane requires a compiled evidence packet")
+ if _text(packet.get("question_id")) != qid:
+ raise RoutePolicyError(f"{qid}: compiled packet identity mismatch")
+ if packet.get("packet_compiler_version") != PACKET_COMPILER_VERSION:
+ raise RoutePolicyError(f"{qid}: compiled packet compiler contract is stale")
+ question_contract = packet.get("question_contract")
+ operation_plan = packet.get("operation_plan")
+ requirement_coverage = packet.get("requirement_coverage")
+ operation_results = packet.get("operation_results")
+ evidence_bundles = packet.get("evidence_bundles")
+ if not isinstance(question_contract, Mapping):
+ raise RoutePolicyError(f"{qid}: compiled packet question contract is missing")
+ if not _text(question_contract.get("question")):
+ raise RoutePolicyError(f"{qid}: compiled packet question contract is incomplete")
+ if _text(row.get("question")) and _text(question_contract.get("question")) != _text(
+ row.get("question")
+ ):
+ raise RoutePolicyError(f"{qid}: compiled packet question contract drifted")
+ if (
+ not isinstance(operation_plan, Mapping)
+ or operation_plan.get("schema_version") != PLAN_SCHEMA
+ ):
+ raise RoutePolicyError(f"{qid}: compiled packet operation plan is missing")
+ for field, value in (
+ ("requirements", operation_plan.get("requirements")),
+ ("operations", operation_plan.get("operations")),
+ ("bundles", operation_plan.get("bundles")),
+ ("requirement_coverage", requirement_coverage),
+ ("operation_results", operation_results),
+ ("evidence_bundles", evidence_bundles),
+ ):
+ if not isinstance(value, Sequence) or isinstance(value, (str, bytes)):
+ raise RoutePolicyError(
+ f"{qid}: compiled packet {field} must be an array"
+ )
+ if any(not isinstance(item, Mapping) for item in value):
+ raise RoutePolicyError(
+ f"{qid}: compiled packet {field} contains a non-object"
+ )
+ requirement_count = question_contract.get("requirement_count")
+ operation_count = question_contract.get("operation_count")
+ if requirement_count != len(requirement_coverage):
+ raise RoutePolicyError(
+ f"{qid}: compiled packet requirement count is inconsistent"
+ )
+ if operation_count != len(operation_results):
+ raise RoutePolicyError(
+ f"{qid}: compiled packet operation count is inconsistent"
+ )
+ windows = row.get("evidence_windows")
+ reservoir = packet.get("raw_evidence_reservoir")
+ if (
+ not isinstance(windows, Sequence)
+ or isinstance(windows, (str, bytes))
+ or not windows
+ or not isinstance(reservoir, Sequence)
+ or isinstance(reservoir, (str, bytes))
+ or not reservoir
+ ):
+ raise RoutePolicyError(f"{qid}: Source evidence is missing")
+ typed_windows = [item for item in windows if isinstance(item, Mapping)]
+ if len(typed_windows) != len(windows):
+ raise RoutePolicyError(f"{qid}: production evidence contains a non-object window")
+ layer_counts = _validate_retrieval_contract(row, typed_windows, qid=qid)
+ for layer, count in layer_counts.items():
+ layer_window_counts[layer] += count
+ expected = [
+ _source_identity(item)
+ for item in windows
+ if isinstance(item, Mapping)
+ ]
+ actual = [
+ _source_identity(item)
+ for item in reservoir
+ if isinstance(item, Mapping)
+ ]
+ if expected != actual:
+ raise RoutePolicyError(f"{qid}: compiled packet does not preserve Source evidence exactly")
+ graph_fields = ("memory_contexts", "attachments", "provenance", "retrieval_metadata")
+ expected_graph = [
+ {field: item.get(field) for field in graph_fields}
+ for item in typed_windows
+ ]
+ actual_graph = [
+ {field: item.get(field) for field in graph_fields}
+ for item in reservoir
+ if isinstance(item, Mapping)
+ ]
+ if _canonical_json(expected_graph) != _canonical_json(actual_graph):
+ raise RoutePolicyError(
+ f"{qid}: compiled packet does not preserve layered memory context exactly"
+ )
+ source_window_count += len(expected)
+ return {
+ "schema_version": POLICY_SCHEMA,
+ "lane": PRODUCTION_LANE,
+ "question_count": len(qids),
+ "source_window_count": source_window_count,
+ "layer_window_counts": layer_window_counts,
+ "answer_protocol": PRODUCTION_ANSWER_PROTOCOL,
+ "answer_model": PRODUCTION_ANSWER_MODEL,
+ "promotion_eligible": True,
+ }
+
+
+def validate_production_answers(rows: Sequence[Mapping[str, Any]]) -> None:
+ if not rows:
+ raise RoutePolicyError("production answers are empty")
+ for index, row in enumerate(rows):
+ qid = _text(row.get("question_id")) or f"row {index}"
+ if row.get("answer_protocol") != PRODUCTION_ANSWER_PROTOCOL:
+ raise RoutePolicyError(f"{qid}: answer protocol is not production-approved")
+ if row.get("answer_model") != PRODUCTION_ANSWER_MODEL:
+ raise RoutePolicyError(f"{qid}: answer model is not production-approved")
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_semantic_evidence.py b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_semantic_evidence.py
new file mode 100644
index 0000000000000000000000000000000000000000..8860303aca376e601a382114e1ecee06c75bfaa1
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_semantic_evidence.py
@@ -0,0 +1,1665 @@
+"""Semantic evidence contracts layered above immutable TMCRA Source evidence."""
+
+from __future__ import annotations
+
+import re
+import unicodedata
+from collections.abc import Mapping, Sequence
+from datetime import datetime
+from decimal import Decimal, InvalidOperation
+from typing import Any
+
+from tmcra_v4_evidence_operations import (
+ OPERATION_TYPES,
+ PACKET_SCHEMA,
+ EvidenceOperationError,
+ build_evidence_catalog,
+ execute_operation_plan,
+ validate_operation_plan,
+)
+from tmcra_v4_typed_semantics import evaluate_proposals
+
+
+TASK_CONTRACT_SCHEMA = "tmcra.task-contract.v2"
+RESOLUTION_PLAN_SCHEMA = "tmcra.evidence-resolution-plan.v1"
+SEMANTIC_PACKET_SCHEMA = "tmcra.semantic-evidence-packet.v1"
+SEMANTIC_ANSWER_SCHEMA = "tmcra.semantic-bound-answer.v1"
+
+OUTPUT_SHAPES = {
+ "scalar",
+ "list",
+ "set",
+ "count",
+ "boolean",
+ "date",
+ "duration",
+ "structured",
+ "free_text",
+}
+OUTPUT_ORIGINS = {
+ "memory_direct",
+ "memory_derived",
+ "memory_conditioned",
+ "external_required",
+}
+PREMISE_ROLES = {"fact", "operand", "constraint", "scope", "counterevidence"}
+PREMISE_NECESSITY = {"required", "optional"}
+PREMISE_SOURCES = {"memory", "query_context", "model_knowledge", "external_tool"}
+ALLOWED_DERIVATIONS = set(OPERATION_TYPES) | {
+ "semantic_composition",
+ "constraint_application",
+}
+CLAIM_POLARITIES = {"positive", "negative", "unknown"}
+CLAIM_MODALITIES = {
+ "asserted",
+ "experienced",
+ "preferred",
+ "planned",
+ "recommended",
+ "uncertain",
+}
+BINDING_COVERAGE = {"complete", "partial", "conflicting", "absent"}
+BINDING_RELATIONS = {"direct", "derived", "constraint", "context", "counterevidence"}
+EXECUTABILITY = {
+ "directly_answerable",
+ "partially_answerable",
+ "derivable",
+ "memory_conditioned_generation",
+ "requires_external_tool",
+ "requires_clarification",
+ "not_answerable",
+}
+CLAIM_OPERATION_TYPES = {
+ "date_difference",
+ "date_order",
+ "numeric_sum",
+ "numeric_difference",
+ "numeric_average",
+ "count_distinct",
+ "ordered_unique_list",
+ "latest_state",
+ "entity_exact_match",
+ "entity_mismatch",
+}
+ANSWER_POLICIES = {
+ "answer",
+ "answer_partial",
+ "answer_with_memory_context",
+ "call_tool",
+ "clarify",
+ "abstain",
+}
+ANSWER_STATUSES = {
+ "answered",
+ "partial",
+ "answered_with_context",
+ "requires_tool",
+ "clarification",
+ "not_answerable",
+}
+CLAIM_ORIGINS = {"memory_fact", "memory_derived", "model_knowledge", "tool_result"}
+
+
+class SemanticEvidenceError(ValueError):
+ pass
+
+
+def _text(value: Any) -> str:
+ return value.strip() if isinstance(value, str) else ""
+
+
+def _string_list(value: Any, *, path: str, allowed: set[str] | None = None) -> list[str]:
+ if not isinstance(value, list):
+ raise SemanticEvidenceError(f"{path} must be an array")
+ output: list[str] = []
+ for item in value:
+ item_text = _text(item)
+ if not item_text:
+ raise SemanticEvidenceError(f"{path} contains an empty value")
+ if allowed is not None and item_text not in allowed:
+ raise SemanticEvidenceError(f"{path} contains unknown value: {item_text}")
+ if item_text in output:
+ raise SemanticEvidenceError(f"{path} contains a duplicate value: {item_text}")
+ output.append(item_text)
+ return output
+
+
+def _align_quote(quote: str, text: str) -> str:
+ if quote in text:
+ return quote
+ location = text.casefold().find(quote.casefold())
+ if location >= 0:
+ return text[location : location + len(quote)]
+ tokens = quote.split()
+ if not tokens:
+ return ""
+ match = re.search(
+ r"\s+".join(re.escape(token) for token in tokens),
+ text,
+ flags=re.IGNORECASE,
+ )
+ if match:
+ return match.group(0)
+
+ punctuation = str.maketrans(
+ {
+ "\u2018": "'",
+ "\u2019": "'",
+ "\u201c": '"',
+ "\u201d": '"',
+ "\u2010": "-",
+ "\u2011": "-",
+ "\u2012": "-",
+ "\u2013": "-",
+ "\u2014": "-",
+ "\u00a0": " ",
+ }
+ )
+
+ def searchable(value: str) -> tuple[str, list[int]]:
+ output: list[str] = []
+ positions: list[int] = []
+ prior_space = False
+ for index, character in enumerate(value):
+ normalized = unicodedata.normalize("NFKC", character).translate(punctuation).casefold()
+ for current in normalized:
+ if current.isspace():
+ if prior_space:
+ continue
+ current = " "
+ prior_space = True
+ else:
+ prior_space = False
+ output.append(current)
+ positions.append(index)
+ return "".join(output), positions
+
+ normalized_quote, _ = searchable(quote)
+ normalized_text, positions = searchable(text)
+ location = normalized_text.find(normalized_quote)
+ if location < 0 or not normalized_quote:
+ return ""
+ return text[positions[location] : positions[location + len(normalized_quote) - 1] + 1]
+
+
+def validate_task_contract(
+ value: Mapping[str, Any], query_context: Mapping[str, Any] | None = None
+) -> dict[str, Any]:
+ root = {"schema_version", "target", "output", "scope", "premises", "allowed_derivations"}
+ if not isinstance(value, Mapping) or set(value) != root:
+ raise SemanticEvidenceError("task contract root fields are invalid")
+ if value.get("schema_version") != TASK_CONTRACT_SCHEMA:
+ raise SemanticEvidenceError("task contract schema_version is invalid")
+ target = value.get("target")
+ if not isinstance(target, Mapping) or set(target) != {"description", "subject", "relation"}:
+ raise SemanticEvidenceError("task contract target fields are invalid")
+ normalized_target = {key: _text(target.get(key)) for key in ("description", "subject", "relation")}
+ if not all(normalized_target.values()):
+ raise SemanticEvidenceError("task contract target is incomplete")
+ output = value.get("output")
+ output_fields = {"shape", "cardinality", "ordering", "unit", "origin"}
+ if not isinstance(output, Mapping) or set(output) != output_fields:
+ raise SemanticEvidenceError("task contract output fields are invalid")
+ normalized_output = {key: _text(output.get(key)) for key in output_fields}
+ if normalized_output["shape"] not in OUTPUT_SHAPES or normalized_output["origin"] not in OUTPUT_ORIGINS:
+ raise SemanticEvidenceError("task contract output shape or origin is invalid")
+ if not normalized_output["cardinality"] or not normalized_output["ordering"]:
+ raise SemanticEvidenceError("task contract output cardinality or ordering is empty")
+ scope = value.get("scope")
+ if not isinstance(scope, Mapping) or set(scope) != {"temporal", "entity", "context"}:
+ raise SemanticEvidenceError("task contract scope fields are invalid")
+ normalized_scope = {key: _text(scope.get(key)) for key in ("temporal", "entity", "context")}
+ premises_value = value.get("premises")
+ if not isinstance(premises_value, list):
+ raise SemanticEvidenceError("task contract premises must be an array")
+ premises: list[dict[str, str]] = []
+ premise_ids: set[str] = set()
+ for index, item in enumerate(premises_value):
+ fields = {"premise_id", "description", "role", "necessity", "source", "context_quote"}
+ if not isinstance(item, Mapping) or set(item) != fields:
+ raise SemanticEvidenceError(f"premises[{index}] fields are invalid")
+ premise = {key: _text(item.get(key)) for key in fields}
+ if not premise["premise_id"] or premise["premise_id"] in premise_ids or not premise["description"]:
+ raise SemanticEvidenceError(f"premises[{index}] identity is invalid")
+ if premise["role"] not in PREMISE_ROLES or premise["necessity"] not in PREMISE_NECESSITY:
+ raise SemanticEvidenceError(f"premises[{index}] role or necessity is invalid")
+ if premise["source"] not in PREMISE_SOURCES:
+ raise SemanticEvidenceError(f"premises[{index}] source is invalid")
+ if premise["source"] == "query_context":
+ if not premise["context_quote"]:
+ raise SemanticEvidenceError(f"premises[{index}] query context quote is empty")
+ if query_context is not None:
+ context_text = "\n".join(
+ _text(query_context.get(key))
+ for key in ("question", "question_date")
+ if _text(query_context.get(key))
+ )
+ aligned = _align_quote(premise["context_quote"], context_text)
+ if not aligned:
+ raise SemanticEvidenceError(f"premises[{index}] quote is not in query context")
+ premise["context_quote"] = aligned
+ elif premise["context_quote"]:
+ raise SemanticEvidenceError(f"premises[{index}] non-context premise has a context quote")
+ premise_ids.add(premise["premise_id"])
+ premises.append(premise)
+ if not any(item["necessity"] == "required" for item in premises):
+ raise SemanticEvidenceError("task contract needs at least one required premise")
+ if normalized_output["origin"] in {"memory_direct", "memory_derived", "memory_conditioned"} and not any(
+ item["source"] == "memory" and item["necessity"] == "required"
+ for item in premises
+ ):
+ raise SemanticEvidenceError("memory task needs at least one required memory premise")
+ derivations = _string_list(value.get("allowed_derivations"), path="allowed_derivations", allowed=ALLOWED_DERIVATIONS)
+ return {
+ "schema_version": TASK_CONTRACT_SCHEMA,
+ "target": normalized_target,
+ "output": normalized_output,
+ "scope": normalized_scope,
+ "premises": premises,
+ "allowed_derivations": derivations,
+ }
+
+
+def task_contract_payload(row: Mapping[str, Any]) -> dict[str, Any]:
+ question = _text(row.get("question"))
+ if not question:
+ raise SemanticEvidenceError("question is empty")
+ payload = {
+ "question": question,
+ "question_date": _text(row.get("question_date")) or "unknown",
+ }
+ dialogue_state = row.get("dialogue_state")
+ if dialogue_state is not None:
+ if not isinstance(dialogue_state, Mapping):
+ raise SemanticEvidenceError("dialogue_state must be an object")
+ payload["dialogue_state"] = dict(dialogue_state)
+ return payload
+
+
+_SOURCE_REVIEW_STOPWORDS = {
+ "a", "an", "and", "are", "for", "from", "has", "have", "in", "is",
+ "of", "on", "the", "to", "user", "with", "request", "current",
+}
+
+
+def _source_tokens(value: str) -> set[str]:
+ return {
+ token
+ for token in re.findall(r"[a-z0-9]+", value.casefold())
+ if len(token) > 1 and token not in _SOURCE_REVIEW_STOPWORDS
+ }
+
+
+def normalize_task_contract_sources(
+ contract: Mapping[str, Any],
+) -> tuple[dict[str, Any], list[str]]:
+ contract = validate_task_contract(contract)
+ premises = [dict(item) for item in contract["premises"]]
+ context = [item for item in premises if item["source"] == "query_context"]
+ required_memory_count = sum(
+ item["source"] == "memory" and item["necessity"] == "required"
+ for item in premises
+ )
+ dropped: set[str] = set()
+ warnings: list[str] = []
+ for memory_premise in premises:
+ if memory_premise["source"] != "memory":
+ continue
+ memory_tokens = _source_tokens(memory_premise["description"])
+ if len(memory_tokens) < 2:
+ continue
+ for context_premise in context:
+ context_tokens = _source_tokens(
+ context_premise["description"] + " " + context_premise["context_quote"]
+ )
+ can_drop_required = (
+ memory_premise["necessity"] != "required" or required_memory_count > 1
+ )
+ if memory_tokens.issubset(context_tokens) and can_drop_required:
+ dropped.add(memory_premise["premise_id"])
+ if memory_premise["necessity"] == "required":
+ required_memory_count -= 1
+ warnings.append(
+ f"{memory_premise['premise_id']}:dropped_query_context_duplicate_of_{context_premise['premise_id']}"
+ )
+ break
+ normalized = dict(contract)
+ normalized["premises"] = [
+ item for item in premises if item["premise_id"] not in dropped
+ ]
+ return validate_task_contract(normalized), warnings
+
+
+def task_contract_source_review_reasons(contract: Mapping[str, Any]) -> list[str]:
+ contract = validate_task_contract(contract)
+ memory = [item for item in contract["premises"] if item["source"] == "memory"]
+ context = [item for item in contract["premises"] if item["source"] == "query_context"]
+
+ reasons: list[str] = []
+ for memory_premise in memory:
+ memory_tokens = _source_tokens(memory_premise["description"])
+ for context_premise in context:
+ context_tokens = _source_tokens(
+ context_premise["description"] + " " + context_premise["context_quote"]
+ )
+ if not memory_tokens or not context_tokens:
+ continue
+ overlap = len(memory_tokens & context_tokens)
+ containment = overlap / min(len(memory_tokens), len(context_tokens))
+ if overlap >= 2 and containment >= 0.6:
+ reasons.append(
+ f"{memory_premise['premise_id']} overlaps query-context premise {context_premise['premise_id']}"
+ )
+ return reasons
+
+
+def requires_exhaustive_resolution_review(contract: Mapping[str, Any]) -> bool:
+ """Return whether Source membership must be exhaustively audited."""
+ contract = validate_task_contract(contract)
+ output = contract["output"]
+ return output["origin"] in {"memory_direct", "memory_derived"} and output[
+ "shape"
+ ] in {"count", "list", "set"}
+
+
+def normalize_resolution_output(
+ value: Mapping[str, Any],
+ contract: Mapping[str, Any],
+ catalog: Mapping[str, Any] | None = None,
+) -> tuple[dict[str, Any], list[str]]:
+ """Remove only semantically unused resolver output before strict validation."""
+ contract = validate_task_contract(contract)
+ normalized = dict(value)
+ warnings: list[str] = []
+ memory_premises = {
+ item["premise_id"] for item in contract["premises"] if item["source"] == "memory"
+ }
+ valid_evidence_ids = {
+ _text(item.get("evidence_id")) for item in (catalog or {}).get("evidence") or []
+ }
+ valid_atom_ids = {
+ _text(item.get("atom_id")) for item in (catalog or {}).get("atoms") or []
+ }
+ evidence_text = {
+ _text(item.get("evidence_id")): str(item.get("text") or "")
+ for item in (catalog or {}).get("evidence") or []
+ }
+ claim_inputs = []
+ valid_input_claim_ids: set[str] = set()
+ for index, raw in enumerate(value.get("claims") or []):
+ if not isinstance(raw, Mapping) or catalog is None:
+ claim_inputs.append(dict(raw) if isinstance(raw, Mapping) else raw)
+ if isinstance(raw, Mapping) and _text(raw.get("claim_id")):
+ valid_input_claim_ids.add(_text(raw.get("claim_id")))
+ continue
+ claim = dict(raw)
+ claim_id = _text(claim.get("claim_id"))
+ evidence_id = _text(claim.get("evidence_id"))
+ quote = _text(claim.get("source_quote"))
+ aligned = _align_quote(quote, evidence_text.get(evidence_id, ""))
+ if (
+ not claim_id
+ or claim_id in valid_input_claim_ids
+ or evidence_id not in valid_evidence_ids
+ or not _text(claim.get("subject"))
+ or not _text(claim.get("predicate"))
+ or not aligned
+ ):
+ warnings.append(f"claims[{index}]:dropped_invalid_or_ungrounded_claim")
+ continue
+ claim["source_quote"] = aligned
+ claim_inputs.append(claim)
+ valid_input_claim_ids.add(claim_id)
+ bindings = []
+ for index, raw in enumerate(value.get("bindings") or []):
+ if isinstance(raw, Mapping) and _text(raw.get("premise_id")) not in memory_premises:
+ warnings.append(f"bindings[{index}]:dropped_non_memory_premise")
+ continue
+ binding = dict(raw) if isinstance(raw, Mapping) else raw
+ if isinstance(binding, dict) and catalog is not None:
+ filtered_claim_ids = [
+ item
+ for item in (binding.get("claim_ids") or [])
+ if isinstance(item, str) and item in valid_input_claim_ids
+ ]
+ if filtered_claim_ids != binding.get("claim_ids"):
+ warnings.append(f"bindings[{index}]:dropped_invalid_claim_reference")
+ binding["claim_ids"] = filtered_claim_ids
+ bindings.append(binding)
+ operations = []
+ for index, raw in enumerate(value.get("operations") or []):
+ if not isinstance(raw, Mapping):
+ operations.append(raw)
+ continue
+ operation = dict(raw)
+ operation_id = _text(operation.get("operation_id"))
+ output_refs = [
+ item
+ for item in (operation.get("output_refs") or [])
+ if item == "TARGET" or item in memory_premises
+ ]
+ if output_refs != operation.get("output_refs"):
+ warnings.append(f"operations[{index}]:dropped_non_memory_output_ref")
+ operation["output_refs"] = output_refs
+ if catalog is not None:
+ input_claim_ids = [
+ item
+ for item in (operation.get("input_claim_ids") or [])
+ if isinstance(item, str) and item in valid_input_claim_ids
+ ]
+ if input_claim_ids != operation.get("input_claim_ids"):
+ warnings.append(f"operations[{index}]:dropped_invalid_input_claim_id")
+ operation["input_claim_ids"] = input_claim_ids
+ input_evidence_ids = [
+ item
+ for item in (operation.get("input_evidence_ids") or [])
+ if isinstance(item, str) and item in valid_evidence_ids
+ ]
+ if input_evidence_ids != operation.get("input_evidence_ids"):
+ warnings.append(f"operations[{index}]:dropped_unknown_input_evidence_id")
+ operation["input_evidence_ids"] = input_evidence_ids
+ input_atom_ids = [
+ item
+ for item in (operation.get("input_atom_ids") or [])
+ if isinstance(item, str) and item in valid_atom_ids
+ ]
+ if input_atom_ids != operation.get("input_atom_ids"):
+ warnings.append(f"operations[{index}]:dropped_unknown_input_atom_id")
+ operation["input_atom_ids"] = input_atom_ids
+ unauthorized = _text(operation.get("operation_type")) not in contract["allowed_derivations"]
+ operation_type = _text(operation.get("operation_type"))
+ claim_operand_count = len(operation.get("input_claim_ids") or [])
+ atom_operand_count = len(operation.get("input_atom_ids") or [])
+ originally_had_claims = bool(raw.get("input_claim_ids"))
+ exact_two = operation_type in {
+ "date_difference",
+ "numeric_difference",
+ "entity_exact_match",
+ "entity_mismatch",
+ }
+ if (
+ (originally_had_claims and not claim_operand_count)
+ or (exact_two and claim_operand_count and claim_operand_count != 2)
+ or (exact_two and atom_operand_count and atom_operand_count != 2)
+ or (not claim_operand_count and not atom_operand_count)
+ ):
+ warnings.append(f"operations[{index}]:dropped_operation_with_invalid_arity")
+ continue
+ if (
+ unauthorized
+ and contract["output"]["origin"] != "memory_derived"
+ ):
+ warnings.append(f"operations[{index}]:dropped_unauthorized_non_derived_operation")
+ continue
+ operations.append(operation)
+ operation_ids = {
+ _text(item.get("operation_id")) for item in operations if isinstance(item, Mapping)
+ }
+ operation_outputs = {
+ _text(item.get("operation_id")): set(item.get("output_refs") or [])
+ for item in operations
+ if isinstance(item, Mapping)
+ }
+ for index, binding in enumerate(bindings):
+ if not isinstance(binding, dict):
+ continue
+ premise_id = _text(binding.get("premise_id"))
+ valid_binding_operations = [
+ operation_id
+ for operation_id in (binding.get("operation_ids") or [])
+ if operation_id in operation_ids
+ and premise_id in operation_outputs.get(operation_id, set())
+ ]
+ if valid_binding_operations != binding.get("operation_ids"):
+ warnings.append(f"bindings[{index}]:dropped_operation_not_outputting_premise")
+ binding["operation_ids"] = valid_binding_operations
+ if _text(binding.get("coverage")) == "absent" and _text(
+ binding.get("relation")
+ ) != "counterevidence":
+ if binding.get("claim_ids") or binding.get("evidence_ids") or binding.get("operation_ids"):
+ warnings.append(f"bindings[{index}]:cleared_support_from_absent_binding")
+ binding["claim_ids"] = []
+ binding["evidence_ids"] = []
+ binding["operation_ids"] = []
+ if (
+ _text(binding.get("coverage")) == "absent"
+ and _text(binding.get("relation")) not in BINDING_RELATIONS
+ ):
+ warnings.append(f"bindings[{index}]:normalized_absent_relation")
+ binding["relation"] = "direct"
+ if _text(binding.get("coverage")) == "complete" and not (
+ binding.get("claim_ids") or binding.get("operation_ids")
+ ):
+ warnings.append(f"bindings[{index}]:downgraded_unsupported_complete_to_absent")
+ binding["coverage"] = "absent"
+ binding["evidence_ids"] = []
+ if _text(binding.get("coverage")) == "partial" and not (
+ binding.get("claim_ids") or binding.get("operation_ids")
+ ):
+ warnings.append(f"bindings[{index}]:downgraded_unsupported_partial_to_absent")
+ binding["coverage"] = "absent"
+ binding["evidence_ids"] = []
+ if (
+ _text(binding.get("coverage")) == "complete"
+ and _text(binding.get("relation")) == "derived"
+ and not binding.get("operation_ids")
+ ):
+ warnings.append(f"bindings[{index}]:downgraded_uncomputed_derived_to_partial")
+ binding["coverage"] = "partial"
+ binding["relation"] = "direct"
+ used_claims = {
+ claim_id
+ for binding in bindings
+ if isinstance(binding, Mapping)
+ for claim_id in (binding.get("claim_ids") or [])
+ if isinstance(claim_id, str)
+ } | {
+ claim_id
+ for operation in operations
+ if isinstance(operation, Mapping)
+ for claim_id in (operation.get("input_claim_ids") or [])
+ if isinstance(claim_id, str)
+ }
+ claims = []
+ for index, raw in enumerate(claim_inputs):
+ if isinstance(raw, Mapping) and _text(raw.get("claim_id")) not in used_claims:
+ warnings.append(f"claims[{index}]:dropped_unused_claim")
+ continue
+ claims.append(dict(raw) if isinstance(raw, Mapping) else raw)
+ normalized["claims"] = claims
+ normalized["bindings"] = bindings
+ normalized["operations"] = operations
+ return normalized, warnings
+
+
+def resolution_payload(
+ row: Mapping[str, Any],
+ contract: Mapping[str, Any],
+ catalog: Mapping[str, Any],
+) -> dict[str, Any]:
+ return {
+ "question": _text(row.get("question")),
+ "question_date": _text(row.get("question_date")) or "unknown",
+ "task_contract": validate_task_contract(contract, task_contract_payload(row)),
+ "lexical_anchor_ids": list(catalog.get("lexical_anchor_ids") or []),
+ "evidence": [
+ {
+ "evidence_id": item["evidence_id"],
+ "session_id": item["session_id"],
+ "session_index": item["session_index"],
+ "parent_chunk_index": item["parent_chunk_index"],
+ "question_overlap_terms": item.get("question_overlap_terms") or [],
+ "question_overlap_score": int(item.get("question_overlap_score", 0)),
+ "text": item["text"],
+ }
+ for item in catalog.get("evidence") or []
+ ],
+ "atoms": [
+ {
+ "atom_id": item["atom_id"],
+ "atom_type": item["atom_type"],
+ "raw_text": item["raw_text"],
+ "normalized_value": item["normalized_value"],
+ "unit": item["unit"],
+ "evidence_id": item["evidence_id"],
+ **({"derivation": item["derivation"]} if item.get("derivation") else {}),
+ }
+ for item in catalog.get("atoms") or []
+ ],
+ }
+
+
+def validate_resolution_plan(
+ value: Mapping[str, Any],
+ contract: Mapping[str, Any],
+ catalog: Mapping[str, Any],
+) -> dict[str, Any]:
+ contract = validate_task_contract(contract)
+ root = {"schema_version", "claims", "bindings", "operations"}
+ if not isinstance(value, Mapping) or set(value) != root:
+ raise SemanticEvidenceError("resolution plan root fields are invalid")
+ if value.get("schema_version") != RESOLUTION_PLAN_SCHEMA:
+ raise SemanticEvidenceError("resolution plan schema_version is invalid")
+ evidence_by_id = {
+ _text(item.get("evidence_id")): item for item in catalog.get("evidence") or []
+ }
+ evidence_ids = set(evidence_by_id)
+ claims_value = value.get("claims")
+ if not isinstance(claims_value, list):
+ raise SemanticEvidenceError("resolution claims must be an array")
+ claims: list[dict[str, str]] = []
+ claim_ids: set[str] = set()
+ claim_evidence: dict[str, str] = {}
+ claim_fields = {
+ "claim_id",
+ "subject",
+ "predicate",
+ "object",
+ "valid_time",
+ "polarity",
+ "modality",
+ "evidence_id",
+ "source_quote",
+ }
+ for index, item in enumerate(claims_value):
+ if not isinstance(item, Mapping) or set(item) != claim_fields:
+ raise SemanticEvidenceError(f"claims[{index}] fields are invalid")
+ claim = {key: _text(item.get(key)) for key in claim_fields}
+ if (
+ not claim["claim_id"]
+ or claim["claim_id"] in claim_ids
+ or not claim["subject"]
+ or not claim["predicate"]
+ or claim["evidence_id"] not in evidence_ids
+ or not claim["source_quote"]
+ ):
+ raise SemanticEvidenceError(f"claims[{index}] identity or grounding is invalid")
+ if claim["polarity"] not in CLAIM_POLARITIES or claim["modality"] not in CLAIM_MODALITIES:
+ raise SemanticEvidenceError(f"claims[{index}] polarity or modality is invalid")
+ aligned_quote = _align_quote(
+ claim["source_quote"],
+ str(evidence_by_id[claim["evidence_id"]].get("text") or ""),
+ )
+ if not aligned_quote:
+ raise SemanticEvidenceError(f"claims[{index}].source_quote is not an exact Source span")
+ claim["source_quote"] = aligned_quote
+ claim_ids.add(claim["claim_id"])
+ claim_evidence[claim["claim_id"]] = claim["evidence_id"]
+ claims.append(claim)
+
+ premise_ids = {item["premise_id"] for item in contract["premises"]}
+ memory_premise_ids = {
+ item["premise_id"]
+ for item in contract["premises"]
+ if item["source"] == "memory"
+ }
+ operations_value = value.get("operations")
+ if not isinstance(operations_value, list):
+ raise SemanticEvidenceError("resolution operations must be an array")
+ operation_ids: set[str] = set()
+ operation_outputs: dict[str, set[str]] = {}
+ operations: list[dict[str, Any]] = []
+ executable_operations: list[dict[str, Any]] = []
+ operation_fields = {
+ "operation_id",
+ "operation_type",
+ "input_atom_ids",
+ "input_evidence_ids",
+ "input_claim_ids",
+ "parameters",
+ "output_refs",
+ }
+ for index, item in enumerate(operations_value):
+ if not isinstance(item, Mapping) or set(item) != operation_fields:
+ raise SemanticEvidenceError(f"operations[{index}] fields are invalid")
+ operation = dict(item)
+ output_refs = _string_list(
+ operation.pop("output_refs"),
+ path=f"operations[{index}].output_refs",
+ allowed=memory_premise_ids | {"TARGET"},
+ )
+ operation_id = _text(operation.get("operation_id"))
+ operation_allowed = _text(operation.get("operation_type")) in contract[
+ "allowed_derivations"
+ ] or (
+ "semantic_composition" in contract["allowed_derivations"]
+ and _text(operation.get("operation_type")) in OPERATION_TYPES
+ )
+ if (
+ not operation_id
+ or operation_id in operation_ids
+ or not output_refs
+ or not operation_allowed
+ ):
+ raise SemanticEvidenceError(f"operations[{index}] identity or output premises are invalid")
+ operation_ids.add(operation_id)
+ operation_outputs[operation_id] = set(output_refs)
+ input_claim_ids = _string_list(
+ operation.pop("input_claim_ids"),
+ path=f"operations[{index}].input_claim_ids",
+ allowed=claim_ids,
+ )
+ if input_claim_ids:
+ if _text(operation.get("operation_type")) not in CLAIM_OPERATION_TYPES:
+ raise SemanticEvidenceError(f"operations[{index}] does not support claim operands")
+ if operation.get("input_atom_ids"):
+ raise SemanticEvidenceError(f"operations[{index}] mixes claim and atom operands")
+ claim_support = {claim_evidence[claim_id] for claim_id in input_claim_ids}
+ if not claim_support.issubset(set(operation.get("input_evidence_ids") or [])):
+ raise SemanticEvidenceError(f"operations[{index}] omits evidence for claim operands")
+ else:
+ executable_operations.append(operation)
+ operations.append(
+ {
+ **operation,
+ "input_claim_ids": input_claim_ids,
+ "output_refs": output_refs,
+ }
+ )
+ operation_plan = {
+ "schema_version": "tmcra.evidence-operation-plan.v1",
+ "requirements": [
+ {
+ "requirement_id": premise["premise_id"],
+ "description": premise["description"],
+ "evidence_ids": [],
+ }
+ for premise in contract["premises"]
+ if premise["source"] == "memory"
+ ],
+ "operations": executable_operations,
+ "bundles": [],
+ }
+
+ bindings_value = value.get("bindings")
+ if not isinstance(bindings_value, list):
+ raise SemanticEvidenceError("resolution bindings must be an array")
+ bindings: list[dict[str, Any]] = []
+ bound_premises: set[str] = set()
+ evidence_by_premise: dict[str, list[str]] = {}
+ for index, item in enumerate(bindings_value):
+ fields = {
+ "premise_id",
+ "claim_ids",
+ "evidence_ids",
+ "operation_ids",
+ "coverage",
+ "relation",
+ }
+ if not isinstance(item, Mapping) or set(item) != fields:
+ raise SemanticEvidenceError(f"bindings[{index}] fields are invalid")
+ premise_id = _text(item.get("premise_id"))
+ if premise_id not in memory_premise_ids or premise_id in bound_premises:
+ raise SemanticEvidenceError(f"bindings[{index}].premise_id is invalid")
+ current_claims = _string_list(item.get("claim_ids"), path=f"bindings[{index}].claim_ids", allowed=claim_ids)
+ current_evidence = _string_list(item.get("evidence_ids"), path=f"bindings[{index}].evidence_ids", allowed=evidence_ids)
+ current_operations = _string_list(item.get("operation_ids"), path=f"bindings[{index}].operation_ids", allowed=operation_ids)
+ coverage, relation = _text(item.get("coverage")), _text(item.get("relation"))
+ if coverage not in BINDING_COVERAGE or relation not in BINDING_RELATIONS:
+ raise SemanticEvidenceError(f"bindings[{index}] coverage or relation is invalid")
+ claimed_evidence = {claim_evidence[claim_id] for claim_id in current_claims}
+ if not claimed_evidence.issubset(set(current_evidence)):
+ raise SemanticEvidenceError(f"bindings[{index}] omits evidence used by its claims")
+ if coverage == "absent" and (current_claims or current_operations) and relation != "counterevidence":
+ raise SemanticEvidenceError(f"bindings[{index}] absent coverage has semantic support")
+ if coverage == "complete" and not (current_claims or current_operations):
+ raise SemanticEvidenceError(f"bindings[{index}] complete coverage lacks claims or operations")
+ if relation == "derived" and coverage == "complete" and not current_operations:
+ raise SemanticEvidenceError(f"bindings[{index}] complete derived coverage lacks an operation")
+ if any(premise_id not in operation_outputs[operation_id] for operation_id in current_operations):
+ raise SemanticEvidenceError(f"bindings[{index}] cites an operation that does not output this premise")
+ bound_premises.add(premise_id)
+ evidence_by_premise[premise_id] = current_evidence
+ bindings.append(
+ {
+ "premise_id": premise_id,
+ "claim_ids": current_claims,
+ "evidence_ids": current_evidence,
+ "operation_ids": current_operations,
+ "coverage": coverage,
+ "relation": relation,
+ }
+ )
+ if bound_premises != memory_premise_ids:
+ raise SemanticEvidenceError("resolution plan must bind every memory premise exactly once")
+ operations_by_premise = {
+ premise_id: {
+ operation_id
+ for operation_id, outputs in operation_outputs.items()
+ if premise_id in outputs
+ }
+ for premise_id in memory_premise_ids
+ }
+ binding_by_premise = {item["premise_id"]: item for item in bindings}
+ if any(
+ not operations_by_premise[premise_id].issubset(
+ set(binding_by_premise[premise_id]["operation_ids"])
+ )
+ for premise_id in memory_premise_ids
+ ):
+ raise SemanticEvidenceError("resolution operation outputs are missing from premise bindings")
+ required_memory_ids = {
+ item["premise_id"]
+ for item in contract["premises"]
+ if item["source"] == "memory" and item["necessity"] == "required"
+ }
+ memory_inputs_complete = all(
+ binding_by_premise[premise_id]["coverage"] == "complete"
+ for premise_id in required_memory_ids
+ )
+ if (
+ contract["output"]["origin"] == "memory_derived"
+ and memory_inputs_complete
+ and not any("TARGET" in outputs for outputs in operation_outputs.values())
+ and "semantic_composition" not in contract["allowed_derivations"]
+ ):
+ raise SemanticEvidenceError("memory-derived task lacks an operation that outputs TARGET")
+ for requirement in operation_plan["requirements"]:
+ requirement["evidence_ids"] = evidence_by_premise[requirement["requirement_id"]]
+ operation_plan["bundles"] = [
+ {
+ "bundle_id": f"B{index:03d}",
+ "role": binding["relation"],
+ "evidence_ids": binding["evidence_ids"],
+ }
+ for index, binding in enumerate(bindings, start=1)
+ if binding["evidence_ids"]
+ ]
+ try:
+ validate_operation_plan(operation_plan, catalog)
+ except EvidenceOperationError as exc:
+ raise SemanticEvidenceError(f"resolution deterministic operation is invalid: {exc}") from exc
+ return {
+ "schema_version": RESOLUTION_PLAN_SCHEMA,
+ "claims": claims,
+ "bindings": bindings,
+ "operations": operations,
+ }
+
+
+def _operation_plan_from_resolution(
+ contract: Mapping[str, Any], resolution: Mapping[str, Any]
+) -> dict[str, Any]:
+ binding_by_premise = {item["premise_id"]: item for item in resolution["bindings"]}
+ return {
+ "schema_version": "tmcra.evidence-operation-plan.v1",
+ "requirements": [
+ {
+ "requirement_id": premise["premise_id"],
+ "description": premise["description"],
+ "evidence_ids": binding_by_premise[premise["premise_id"]]["evidence_ids"],
+ }
+ for premise in contract["premises"]
+ if premise["source"] == "memory"
+ ],
+ "operations": [
+ {
+ key: value
+ for key, value in operation.items()
+ if key not in {"output_refs", "input_claim_ids"}
+ }
+ for operation in resolution["operations"]
+ if not operation["input_claim_ids"]
+ ],
+ "bundles": [
+ {
+ "bundle_id": f"B{index:03d}",
+ "role": binding["relation"],
+ "evidence_ids": binding["evidence_ids"],
+ }
+ for index, binding in enumerate(resolution["bindings"], start=1)
+ if binding["evidence_ids"]
+ ],
+ }
+
+
+def _execute_claim_operation(
+ operation: Mapping[str, Any], claims_by_id: Mapping[str, Mapping[str, Any]]
+) -> dict[str, Any]:
+ operation_type = operation["operation_type"]
+ claims = [claims_by_id[item] for item in operation["input_claim_ids"]]
+ parameters = dict(operation.get("parameters") or {})
+ field = _text(parameters.get("field"))
+ if not field and operation_type in {"date_difference", "date_order"}:
+ field = (
+ "valid_time"
+ if all(_text(item.get("valid_time")) for item in claims)
+ else "object"
+ )
+ field = field or "subject"
+ if field not in {"subject", "predicate", "object", "valid_time"}:
+ raise SemanticEvidenceError(f"claim operation field is invalid: {field}")
+ values = [_text(item.get(field)) for item in claims]
+ if any(not value for value in values):
+ raise SemanticEvidenceError(f"claim operation has an empty {field} value")
+ unique: list[str] = []
+ seen: set[str] = set()
+ for value in values:
+ key = value.casefold()
+ if key not in seen:
+ seen.add(key)
+ unique.append(value)
+ if operation_type == "date_order":
+ def parsed_date(value: str):
+ try:
+ return datetime.fromisoformat(value.replace("Z", "+00:00")).date()
+ except ValueError as exc:
+ match = re.search(r"\b20\d{2}-\d{2}-\d{2}\b", value)
+ if not match:
+ raise SemanticEvidenceError(f"date_order has an invalid date: {value}") from exc
+ return datetime.strptime(match.group(0), "%Y-%m-%d").date()
+
+ ordered = sorted(zip(claims, values), key=lambda item: parsed_date(item[1]))
+ result = {
+ "ordered_claim_ids": [item[0]["claim_id"] for item in ordered],
+ "ordered_values": [item[1] for item in ordered],
+ }
+ elif operation_type == "date_difference":
+ if len(values) != 2:
+ raise SemanticEvidenceError("date_difference requires exactly two claim operands")
+
+ def parsed_date(value: str):
+ try:
+ return datetime.fromisoformat(value.replace("Z", "+00:00")).date()
+ except ValueError as exc:
+ match = re.search(r"\b20\d{2}-\d{2}-\d{2}\b", value)
+ if not match:
+ raise SemanticEvidenceError(f"date_difference has an invalid date: {value}") from exc
+ return datetime.strptime(match.group(0), "%Y-%m-%d").date()
+
+ result = {"value": abs((parsed_date(values[1]) - parsed_date(values[0])).days), "unit": "days"}
+ elif operation_type in {"numeric_sum", "numeric_difference", "numeric_average"}:
+ numbers: list[Decimal] = []
+ for value in values:
+ match = re.search(r"-?\d+(?:,\d{3})*(?:\.\d+)?", value)
+ if not match:
+ raise SemanticEvidenceError(f"{operation_type} has a non-numeric claim value: {value}")
+ try:
+ numbers.append(Decimal(match.group(0).replace(",", "")))
+ except InvalidOperation as exc:
+ raise SemanticEvidenceError(f"{operation_type} has an invalid number: {value}") from exc
+ if operation_type == "numeric_sum":
+ number = sum(numbers, Decimal(0))
+ elif operation_type == "numeric_difference":
+ if len(numbers) != 2:
+ raise SemanticEvidenceError("numeric_difference requires exactly two claim operands")
+ number = numbers[0] - numbers[1]
+ else:
+ number = sum(numbers, Decimal(0)) / Decimal(len(numbers))
+ result = {
+ "value": int(number) if number == number.to_integral_value() else float(number),
+ "unit": _text(parameters.get("unit")),
+ }
+ elif operation_type == "count_distinct":
+ result: dict[str, Any] = {"value": len(unique), "values": unique}
+ elif operation_type == "ordered_unique_list":
+ result = {"values": unique}
+ elif operation_type == "latest_state":
+ latest = max(claims, key=lambda item: _text(item.get("valid_time")))
+ if not _text(latest.get("valid_time")):
+ raise SemanticEvidenceError("latest_state claim operation lacks valid_time")
+ result = {
+ "claim_id": latest["claim_id"],
+ "subject": latest["subject"],
+ "predicate": latest["predicate"],
+ "object": latest["object"],
+ "valid_time": latest["valid_time"],
+ }
+ elif operation_type in {"entity_exact_match", "entity_mismatch"}:
+ if len(values) != 2:
+ raise SemanticEvidenceError(f"{operation_type} requires exactly two claim operands")
+ equal = values[0].casefold() == values[1].casefold()
+ result = {"value": equal if operation_type == "entity_exact_match" else not equal}
+ else:
+ raise SemanticEvidenceError(f"unsupported claim operation: {operation_type}")
+ return {
+ "operation_id": operation["operation_id"],
+ "operation_type": operation_type,
+ "status": "completed",
+ "input_claim_ids": list(operation["input_claim_ids"]),
+ "support_ids": list(operation.get("input_evidence_ids") or []),
+ "result": result,
+ }
+
+
+def build_answerability_certificate(
+ contract: Mapping[str, Any],
+ resolution: Mapping[str, Any],
+ operation_results: Sequence[Mapping[str, Any]],
+) -> dict[str, Any]:
+ operation_status = {_text(item.get("operation_id")): _text(item.get("status")) for item in operation_results}
+ binding_by_premise = {item["premise_id"]: item for item in resolution["bindings"]}
+ statuses: list[dict[str, Any]] = []
+ for premise in contract["premises"]:
+ source = premise["source"]
+ if source == "memory":
+ binding = binding_by_premise[premise["premise_id"]]
+ effective = binding["coverage"]
+ relation = binding["relation"]
+ elif source == "query_context":
+ effective, relation = "complete", "context"
+ elif source == "model_knowledge":
+ effective, relation = "available", "context"
+ else:
+ effective, relation = "pending", "context"
+ statuses.append(
+ {
+ "premise_id": premise["premise_id"],
+ "necessity": premise["necessity"],
+ "source": source,
+ "coverage": effective,
+ "relation": relation,
+ }
+ )
+ required_memory = [
+ item
+ for item in statuses
+ if item["necessity"] == "required" and item["source"] == "memory"
+ ]
+ required_coverages = [item["coverage"] for item in required_memory]
+ if any(value == "conflicting" for value in required_coverages):
+ memory_coverage = "conflicting"
+ elif required_coverages and all(value == "absent" for value in required_coverages):
+ memory_coverage = "absent"
+ elif any(value in {"partial", "absent"} for value in required_coverages):
+ memory_coverage = "partial"
+ else:
+ memory_coverage = "complete"
+ output_origin = contract["output"]["origin"]
+ target_operation_ids = [
+ operation["operation_id"]
+ for operation in resolution["operations"]
+ if "TARGET" in operation["output_refs"]
+ ]
+ target_operations_completed = all(
+ operation_status.get(operation_id) == "completed"
+ for operation_id in target_operation_ids
+ )
+ failed_operation_ids = [
+ operation_id
+ for operation_id in target_operation_ids
+ if operation_status.get(operation_id) != "completed"
+ ]
+ if not target_operation_ids:
+ target_operation_status = "not_needed"
+ elif failed_operation_ids:
+ target_operation_status = "failed"
+ else:
+ target_operation_status = "completed"
+ pending_external = [
+ item["premise_id"]
+ for item in statuses
+ if item["necessity"] == "required" and item["source"] == "external_tool"
+ ]
+ if memory_coverage == "conflicting":
+ executability, policy = "requires_clarification", "clarify"
+ elif memory_coverage == "absent":
+ executability, policy = "not_answerable", "abstain"
+ elif memory_coverage == "partial" and output_origin in {
+ "memory_direct",
+ "memory_conditioned",
+ }:
+ executability, policy = "partially_answerable", "answer_partial"
+ elif memory_coverage == "partial":
+ executability, policy = "not_answerable", "abstain"
+ elif pending_external:
+ executability, policy = "requires_external_tool", "call_tool"
+ elif output_origin == "memory_direct":
+ executability, policy = "directly_answerable", "answer"
+ elif output_origin == "memory_derived" and target_operations_completed:
+ executability, policy = "derivable", "answer"
+ elif output_origin == "memory_derived" and "semantic_composition" in contract["allowed_derivations"]:
+ executability, policy = "derivable", "answer"
+ elif output_origin == "memory_derived":
+ executability, policy = "not_answerable", "abstain"
+ elif output_origin == "memory_conditioned":
+ executability, policy = "memory_conditioned_generation", "answer_with_memory_context"
+ else:
+ executability, policy = "requires_external_tool", "call_tool"
+ unresolved = [
+ item["premise_id"]
+ for item in required_memory
+ if item["coverage"] in {"partial", "absent", "conflicting"}
+ ] + pending_external
+ allowed_origins = ["memory_fact", "memory_derived"]
+ if policy == "answer_with_memory_context" or any(
+ item["source"] == "model_knowledge" for item in statuses
+ ):
+ allowed_origins.append("model_knowledge")
+ if policy == "call_tool":
+ allowed_origins.append("tool_result")
+ return {
+ "memory_coverage": memory_coverage,
+ "operation_status": target_operation_status,
+ "failed_operation_ids": failed_operation_ids,
+ "unresolved_reasons": [
+ f"operator_failure:{operation_id}" for operation_id in failed_operation_ids
+ ],
+ "task_executability": executability,
+ "answer_policy": policy,
+ "output_origin": output_origin,
+ "target_operation_ids": target_operation_ids,
+ "premise_status": statuses,
+ "unresolved_premise_ids": unresolved,
+ "allowed_claim_origins": allowed_origins,
+ }
+
+
+def compile_semantic_evidence_packet(
+ row: Mapping[str, Any],
+ contract: Mapping[str, Any],
+ resolution: Mapping[str, Any],
+) -> dict[str, Any]:
+ query_context = task_contract_payload(row)
+ contract = validate_task_contract(contract, query_context)
+ catalog = build_evidence_catalog(row)
+ resolution = validate_resolution_plan(resolution, contract, catalog)
+ operation_plan = _operation_plan_from_resolution(contract, resolution)
+ standard_results = {
+ item["operation_id"]: item for item in execute_operation_plan(operation_plan, catalog)
+ }
+ claims_by_id = {item["claim_id"]: item for item in resolution["claims"]}
+ operation_results: list[dict[str, Any]] = []
+ for operation in resolution["operations"]:
+ if operation["input_claim_ids"]:
+ try:
+ operation_results.append(_execute_claim_operation(operation, claims_by_id))
+ except SemanticEvidenceError as exc:
+ operation_results.append(
+ {
+ "operation_id": operation["operation_id"],
+ "operation_type": operation["operation_type"],
+ "status": "error",
+ "input_claim_ids": list(operation["input_claim_ids"]),
+ "support_ids": list(operation.get("input_evidence_ids") or []),
+ "error": str(exc),
+ }
+ )
+ else:
+ operation_results.append(standard_results[operation["operation_id"]])
+ certificate = build_answerability_certificate(contract, resolution, operation_results)
+ evidence_by_id = {item["evidence_id"]: item for item in catalog["evidence"]}
+ return {
+ "schema_version": SEMANTIC_PACKET_SCHEMA,
+ "question_id": catalog["question_id"],
+ "query_context": query_context,
+ "task_contract": contract,
+ "claim_ledger": resolution["claims"],
+ "premise_bindings": resolution["bindings"],
+ "resolution_program": {
+ "schema_version": RESOLUTION_PLAN_SCHEMA,
+ "operations": resolution["operations"],
+ "operation_results": operation_results,
+ },
+ "answerability_certificate": certificate,
+ "bound_evidence": [
+ {
+ **binding,
+ "claims": [
+ claim
+ for claim in resolution["claims"]
+ if claim["claim_id"] in binding["claim_ids"]
+ ],
+ "evidence": [evidence_by_id[item] for item in binding["evidence_ids"]],
+ }
+ for binding in resolution["bindings"]
+ ],
+ "raw_evidence_reservoir": catalog["evidence"],
+ "lexical_anchor_ids": list(catalog.get("lexical_anchor_ids") or []),
+ "atom_catalog": catalog,
+ }
+
+
+def semantic_packet_to_advisory_packet(packet: Mapping[str, Any]) -> dict[str, Any]:
+ """Expose semantic planning as annotations over the complete Source reservoir."""
+ if packet.get("schema_version") != SEMANTIC_PACKET_SCHEMA:
+ raise SemanticEvidenceError("semantic evidence packet schema is invalid")
+ contract = packet.get("task_contract")
+ certificate = packet.get("answerability_certificate")
+ reservoir = packet.get("raw_evidence_reservoir")
+ if (
+ not isinstance(contract, Mapping)
+ or not isinstance(certificate, Mapping)
+ or not isinstance(reservoir, list)
+ or not reservoir
+ ):
+ raise SemanticEvidenceError("semantic packet lacks contract, certificate, or Source reservoir")
+ evidence_ids = {
+ _text(item.get("evidence_id"))
+ for item in reservoir
+ if isinstance(item, Mapping) and _text(item.get("evidence_id"))
+ }
+ bindings = {
+ _text(item.get("premise_id")): item
+ for item in packet.get("premise_bindings") or []
+ if isinstance(item, Mapping) and _text(item.get("premise_id"))
+ }
+ requirements: list[dict[str, Any]] = []
+ bundles: list[dict[str, Any]] = []
+ for premise in contract.get("premises") or []:
+ if not isinstance(premise, Mapping):
+ continue
+ premise_id = _text(premise.get("premise_id"))
+ if not premise_id:
+ continue
+ binding = bindings.get(premise_id, {})
+ bound_ids = [
+ item
+ for item in binding.get("evidence_ids") or []
+ if isinstance(item, str) and item in evidence_ids
+ ]
+ coverage = _text(binding.get("coverage")) or (
+ "available" if premise.get("source") != "memory" else "unresolved"
+ )
+ requirements.append(
+ {
+ "requirement_id": premise_id,
+ "description": _text(premise.get("description")) or premise_id,
+ "evidence_ids": bound_ids,
+ "state": coverage,
+ "planner_advisory": True,
+ }
+ )
+ if bound_ids:
+ bundles.append(
+ {
+ "bundle_id": f"B_{premise_id}",
+ "role": _text(premise.get("role")) or "advisory",
+ "evidence_ids": bound_ids,
+ "evidence": [
+ item
+ for item in reservoir
+ if isinstance(item, Mapping) and item.get("evidence_id") in bound_ids
+ ],
+ }
+ )
+ query_context = dict(packet.get("query_context") or {})
+ typed_input = packet.get("typed_semantics")
+ typed_evaluation: dict[str, Any] = {
+ "status": "not_provided",
+ "advisory": True,
+ "authoritative": False,
+ "accepted": [],
+ "rejected": [],
+ }
+ if typed_input is not None:
+ if not isinstance(typed_input, Mapping):
+ typed_evaluation = {
+ "status": "invalid_container",
+ "advisory": True,
+ "authoritative": False,
+ "accepted": [],
+ "rejected": [
+ {
+ "candidate_id": "typed_semantics",
+ "status": "rejected",
+ "authoritative": False,
+ "source_evidence_ids": [],
+ "evidence_ids": [],
+ "diagnostics": [
+ {
+ "code": "UNTYPED_CONTAINER",
+ "message": "typed_semantics must contain observations and proposals",
+ "path": "typed_semantics",
+ }
+ ],
+ }
+ ],
+ }
+ else:
+ typed_evaluation = evaluate_proposals(
+ typed_input.get("observations", []),
+ typed_input.get("proposals", []),
+ )
+
+ accepted_typed_results: list[dict[str, Any]] = []
+ rejected_typed_results = list(typed_evaluation.get("rejected") or [])
+ for candidate in typed_evaluation.get("accepted") or []:
+ support_ids = [
+ item
+ for item in candidate.get("source_evidence_ids") or []
+ if isinstance(item, str)
+ ]
+ unknown_ids = sorted(set(support_ids) - evidence_ids)
+ if unknown_ids:
+ rejected_typed_results.append(
+ {
+ "candidate_id": candidate.get("candidate_id"),
+ "status": "rejected",
+ "authoritative": False,
+ "source_evidence_ids": support_ids,
+ "evidence_ids": support_ids,
+ "diagnostics": [
+ {
+ "code": "UNKNOWN_SOURCE_EVIDENCE",
+ "message": "typed result cites evidence outside the current Source reservoir",
+ "path": "typed_semantics.observations.evidence_ids",
+ "unknown_evidence_ids": unknown_ids,
+ }
+ ],
+ }
+ )
+ continue
+ candidate_id = _text(candidate.get("candidate_id"))
+ accepted_typed_results.append(
+ {
+ "operation_id": f"TS_{candidate_id}",
+ "operation_type": "typed_semantic_proposal",
+ "status": "completed",
+ "input_atom_ids": [],
+ "support_ids": support_ids,
+ "result": {
+ "value": candidate.get("value"),
+ "value_kind": candidate.get("value_kind"),
+ "unit": candidate.get("unit"),
+ },
+ "advisory": True,
+ "authoritative": False,
+ }
+ )
+ typed_diagnostics = {
+ **typed_evaluation,
+ "accepted": list(typed_evaluation.get("accepted") or []),
+ "rejected": rejected_typed_results,
+ }
+ return {
+ "schema_version": PACKET_SCHEMA,
+ "question_id": packet.get("question_id"),
+ "question_contract": {
+ "question": query_context.get("question"),
+ "question_date": query_context.get("question_date"),
+ "requirement_count": len(requirements),
+ "operation_count": len(accepted_typed_results),
+ "semantic_authority": "advisory",
+ },
+ "requirement_coverage": requirements,
+ # Legacy semantic operations are intentionally withheld until a typed
+ # proposal validator proves their dimensions and entity constraints.
+ "operation_results": accepted_typed_results,
+ "evidence_bundles": bundles,
+ "raw_evidence_reservoir": list(reservoir),
+ "lexical_anchor_ids": list(packet.get("lexical_anchor_ids") or []),
+ "operation_result_ids": [item["operation_id"] for item in accepted_typed_results],
+ "semantic_advisory": {
+ "authority": "advisory_not_answerability",
+ "task_contract": dict(contract),
+ "claim_ledger": list(packet.get("claim_ledger") or []),
+ "planner_certificate": dict(certificate),
+ "legacy_operation_status": "withheld_untyped",
+ "typed_semantics": typed_diagnostics,
+ },
+ }
+
+
+def normalize_semantic_answer_output(
+ value: Mapping[str, Any], packet: Mapping[str, Any]
+) -> dict[str, Any]:
+ """Repair only mechanically verifiable answer metadata and support references."""
+ if not isinstance(value, Mapping):
+ return dict(value)
+ normalized = dict(value)
+ certificate = packet.get("answerability_certificate")
+ if not isinstance(certificate, Mapping):
+ return normalized
+
+ expected_status = {
+ "answer": "answered",
+ "answer_partial": "partial",
+ "answer_with_memory_context": "answered_with_context",
+ "call_tool": "requires_tool",
+ "clarify": "clarification",
+ "abstain": "not_answerable",
+ }.get(_text(certificate.get("answer_policy")))
+ current_status = _text(normalized.get("response_status"))
+ if {current_status, expected_status}.issubset({"answered", "answered_with_context"}):
+ normalized["response_status"] = expected_status
+
+ premise_bindings = {
+ _text(item.get("premise_id")): item
+ for item in packet.get("premise_bindings") or []
+ if isinstance(item, Mapping) and _text(item.get("premise_id"))
+ }
+ task_premise_ids = {
+ _text(item.get("premise_id"))
+ for item in (packet.get("task_contract") or {}).get("premises") or []
+ if isinstance(item, Mapping) and _text(item.get("premise_id"))
+ }
+ ledger_ids = {
+ _text(item.get("claim_id"))
+ for item in packet.get("claim_ledger") or []
+ if isinstance(item, Mapping) and _text(item.get("claim_id"))
+ }
+ completed_operation_ids = {
+ _text(item.get("operation_id"))
+ for item in (packet.get("resolution_program") or {}).get("operation_results") or []
+ if isinstance(item, Mapping)
+ and item.get("status") == "completed"
+ and _text(item.get("operation_id"))
+ }
+ operation_specs = {
+ _text(item.get("operation_id")): item
+ for item in (packet.get("resolution_program") or {}).get("operations") or []
+ if isinstance(item, Mapping) and _text(item.get("operation_id"))
+ }
+ target_operation_ids = set(certificate.get("target_operation_ids") or [])
+ allowed_origins = set(certificate.get("allowed_claim_origins") or [])
+
+ claims = normalized.get("claims")
+ if isinstance(claims, list):
+ normalized_claims: list[Any] = []
+ for raw_claim in claims:
+ if not isinstance(raw_claim, Mapping):
+ normalized_claims.append(raw_claim)
+ continue
+ claim = dict(raw_claim)
+ origin = _text(claim.get("origin"))
+ raw_premises = claim.get("premise_ids")
+ current_premises = (
+ [item for item in raw_premises if isinstance(item, str) and item in task_premise_ids]
+ if isinstance(raw_premises, list)
+ else raw_premises
+ )
+ if origin.startswith("memory_") and isinstance(current_premises, list):
+ current_premises = [item for item in current_premises if item in premise_bindings]
+ claim["premise_ids"] = current_premises
+
+ if origin.startswith("memory_") and isinstance(current_premises, list):
+ allowed_source_claims = {
+ item
+ for premise_id in current_premises
+ for item in premise_bindings[premise_id].get("claim_ids") or []
+ }
+ premise_source_claims = set(allowed_source_claims)
+ premise_evidence = {
+ item
+ for premise_id in current_premises
+ for item in premise_bindings[premise_id].get("evidence_ids") or []
+ }
+ allowed_computations = {
+ item
+ for premise_id in current_premises
+ for item in premise_bindings[premise_id].get("operation_ids") or []
+ }
+ allowed_computations.update(
+ operation_id
+ for operation_id in target_operation_ids
+ if operation_id in operation_specs
+ and set(operation_specs[operation_id].get("input_claim_ids") or []).issubset(
+ premise_source_claims
+ )
+ and set(operation_specs[operation_id].get("input_evidence_ids") or []).issubset(
+ premise_evidence
+ )
+ )
+ raw_sources = claim.get("source_claim_ids")
+ if isinstance(raw_sources, list):
+ claim["source_claim_ids"] = [
+ item
+ for item in raw_sources
+ if isinstance(item, str)
+ and item in ledger_ids
+ and item in allowed_source_claims
+ ]
+ raw_computations = claim.get("computation_ids")
+ if isinstance(raw_computations, list):
+ claim["computation_ids"] = [
+ item
+ for item in raw_computations
+ if isinstance(item, str)
+ and item in completed_operation_ids
+ and item in allowed_computations
+ ]
+ if origin == "memory_fact":
+ claim["computation_ids"] = []
+ elif (
+ origin == "memory_derived"
+ and not claim.get("computation_ids")
+ and claim.get("source_claim_ids")
+ and "memory_fact" in allowed_origins
+ ):
+ claim["origin"] = "memory_fact"
+ elif origin in {"model_knowledge", "tool_result"}:
+ claim["source_claim_ids"] = []
+ claim["computation_ids"] = []
+ normalized_claims.append(claim)
+ normalized["claims"] = normalized_claims
+
+ unresolved = normalized.get("unresolved_premise_ids")
+ expected_unresolved = list(certificate.get("unresolved_premise_ids") or [])
+ if isinstance(unresolved, list) and set(unresolved) == set(expected_unresolved):
+ normalized["unresolved_premise_ids"] = expected_unresolved
+ return normalized
+
+
+def validate_semantic_answer(value: Mapping[str, Any], packet: Mapping[str, Any]) -> dict[str, Any]:
+ root = {"schema_version", "response_status", "claims", "unresolved_premise_ids", "answer"}
+ if not isinstance(value, Mapping) or not root.issubset(value):
+ raise SemanticEvidenceError("semantic answer root fields are invalid")
+ if value.get("schema_version") != SEMANTIC_ANSWER_SCHEMA:
+ raise SemanticEvidenceError("semantic answer schema_version is invalid")
+ status = _text(value.get("response_status"))
+ if status not in ANSWER_STATUSES:
+ raise SemanticEvidenceError("semantic answer response_status is invalid")
+ certificate = packet.get("answerability_certificate")
+ if not isinstance(certificate, Mapping):
+ raise SemanticEvidenceError("semantic packet lacks an answerability certificate")
+ expected_status = {
+ "answer": "answered",
+ "answer_partial": "partial",
+ "answer_with_memory_context": "answered_with_context",
+ "call_tool": "requires_tool",
+ "clarify": "clarification",
+ "abstain": "not_answerable",
+ }.get(_text(certificate.get("answer_policy")))
+ if status != expected_status:
+ raise SemanticEvidenceError("semantic answer contradicts the answerability certificate")
+ premise_bindings = {item["premise_id"]: item for item in packet.get("premise_bindings") or []}
+ premise_ids = {
+ item["premise_id"] for item in (packet.get("task_contract") or {}).get("premises") or []
+ }
+ ledger = {item["claim_id"]: item for item in packet.get("claim_ledger") or []}
+ operation_results = {
+ item["operation_id"]: item
+ for item in (packet.get("resolution_program") or {}).get("operation_results") or []
+ if item.get("status") == "completed"
+ }
+ operation_specs = {
+ item["operation_id"]: item
+ for item in (packet.get("resolution_program") or {}).get("operations") or []
+ }
+ target_operation_ids = set(certificate.get("target_operation_ids") or [])
+ allowed_origins = set(certificate.get("allowed_claim_origins") or [])
+ claims_value = value.get("claims")
+ if not isinstance(claims_value, list):
+ raise SemanticEvidenceError("semantic answer claims must be an array")
+ claims: list[dict[str, Any]] = []
+ seen: set[str] = set()
+ for index, item in enumerate(claims_value):
+ required_fields = {"claim_id", "text", "origin", "premise_ids"}
+ if not isinstance(item, Mapping) or not required_fields.issubset(item):
+ raise SemanticEvidenceError(f"answer claims[{index}] fields are invalid")
+ claim_id, text, origin = _text(item.get("claim_id")), _text(item.get("text")), _text(item.get("origin"))
+ if not claim_id or claim_id in seen or not text or origin not in CLAIM_ORIGINS or origin not in allowed_origins:
+ raise SemanticEvidenceError(f"answer claims[{index}] identity or origin is invalid")
+ current_premises = _string_list(item.get("premise_ids"), path=f"answer claims[{index}].premise_ids", allowed=premise_ids)
+ source_claims = _string_list(item.get("source_claim_ids", []), path=f"answer claims[{index}].source_claim_ids", allowed=set(ledger))
+ computations = _string_list(item.get("computation_ids", []), path=f"answer claims[{index}].computation_ids", allowed=set(operation_results))
+ if origin == "memory_fact" and not source_claims:
+ raise SemanticEvidenceError(f"answer claims[{index}] memory_fact lacks Source claims")
+ if origin == "memory_derived" and not computations:
+ raise SemanticEvidenceError(f"answer claims[{index}] memory_derived lacks computations")
+ if origin in {"model_knowledge", "tool_result"} and (source_claims or computations):
+ raise SemanticEvidenceError(f"answer claims[{index}] non-memory origin has memory bindings")
+ if origin.startswith("memory_") and not current_premises:
+ raise SemanticEvidenceError(f"answer claims[{index}] memory claim lacks premise bindings")
+ if origin.startswith("memory_") and any(
+ premise_id not in premise_bindings for premise_id in current_premises
+ ):
+ raise SemanticEvidenceError(f"answer claims[{index}] memory claim cites a non-memory premise")
+ allowed_source_claims = {
+ claim
+ for premise_id in current_premises
+ if premise_id in premise_bindings
+ for claim in premise_bindings[premise_id]["claim_ids"]
+ }
+ allowed_computations = {
+ operation
+ for premise_id in current_premises
+ if premise_id in premise_bindings
+ for operation in premise_bindings[premise_id]["operation_ids"]
+ }
+ premise_source_claims = {
+ source_claim
+ for premise_id in current_premises
+ if premise_id in premise_bindings
+ for source_claim in premise_bindings[premise_id]["claim_ids"]
+ }
+ premise_evidence = {
+ evidence_id
+ for premise_id in current_premises
+ if premise_id in premise_bindings
+ for evidence_id in premise_bindings[premise_id]["evidence_ids"]
+ }
+ allowed_computations.update(
+ operation_id
+ for operation_id in target_operation_ids
+ if operation_id in operation_specs
+ and set(operation_specs[operation_id].get("input_claim_ids") or []).issubset(
+ premise_source_claims
+ )
+ and set(operation_specs[operation_id].get("input_evidence_ids") or []).issubset(
+ premise_evidence
+ )
+ )
+ if not set(source_claims).issubset(allowed_source_claims) or not set(computations).issubset(allowed_computations):
+ raise SemanticEvidenceError(f"answer claims[{index}] cites support outside its premises")
+ support_ids = sorted(
+ {ledger[item]["evidence_id"] for item in source_claims}
+ | {
+ support_id
+ for operation_id in computations
+ for support_id in operation_results[operation_id].get("support_ids") or []
+ }
+ )
+ seen.add(claim_id)
+ claims.append(
+ {
+ "claim_id": claim_id,
+ "text": text,
+ "origin": origin,
+ "premise_ids": current_premises,
+ "source_claim_ids": source_claims,
+ "support_ids": support_ids,
+ "computation_ids": computations,
+ }
+ )
+ unresolved = _string_list(
+ value.get("unresolved_premise_ids"),
+ path="unresolved_premise_ids",
+ allowed=premise_ids,
+ )
+ expected_unresolved = list(certificate.get("unresolved_premise_ids") or [])
+ if unresolved != expected_unresolved:
+ raise SemanticEvidenceError("semantic answer unresolved premises do not match the certificate")
+ if status in {"answered", "answered_with_context"} and (unresolved or not claims):
+ raise SemanticEvidenceError("answering status requires claims and no unresolved premises")
+ if status == "partial" and (not unresolved or not claims):
+ raise SemanticEvidenceError("partial status requires supported claims and unresolved premises")
+ if (
+ status in {"answered_with_context", "partial"}
+ and certificate.get("output_origin") == "memory_conditioned"
+ and not any(
+ claim["origin"] in {"memory_fact", "memory_derived"} for claim in claims
+ )
+ ):
+ raise SemanticEvidenceError("memory-conditioned answer does not use any memory claim")
+ answer = _text(value.get("answer"))
+ if not answer:
+ raise SemanticEvidenceError("semantic answer text is empty")
+ return {
+ "schema_version": SEMANTIC_ANSWER_SCHEMA,
+ "response_status": status,
+ "claims": claims,
+ "unresolved_premise_ids": unresolved,
+ "answer": answer,
+ }
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_slow_graph.py b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_slow_graph.py
new file mode 100644
index 0000000000000000000000000000000000000000..b2e298ef27b69743f4436abbbce987fa5596a4e5
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_slow_graph.py
@@ -0,0 +1,6343 @@
+#!/usr/bin/env python3
+"""TMCRA V4 tiered slow-graph controller.
+
+The append-only store, provenance checks, leases, and CLI job lifecycle are
+reused from the V3 implementation. This module owns only V4 route selection
+and the strict Flash/Pro transport policy.
+"""
+
+from __future__ import annotations
+
+import argparse
+import errno
+import hashlib
+import json
+import os
+import re
+import time
+import urllib.error
+import urllib.request
+import uuid
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Iterable, Mapping
+
+import tmcra_v3_slow_graph as _v3
+
+
+# Re-export the V3 store machinery instead of copying or changing it.
+AuditError = _v3.AuditError
+DeepSeekCallError = _v3.DeepSeekCallError
+EvidencePolicyError = _v3.EvidencePolicyError
+JobClaim = _v3.JobClaim
+PatchValidationError = _v3.PatchValidationError
+SlowGraphError = _v3.SlowGraphError
+StaleRevisionError = _v3.StaleRevisionError
+load_graph_schema = _v3.load_graph_schema
+validate_patch = _v3.validate_patch
+
+CAPSULE_VARIANT = _v3.CAPSULE_VARIANT
+LEAF_VARIANT = _v3.LEAF_VARIANT
+PATCH_ACTIONS = _v3.PATCH_ACTIONS
+SCHEMA_VERSION = _v3.SCHEMA_VERSION
+SLOW_PROMPT_MIGRATION_SOURCE_VERSION = "tmcra-v4-slow-graph-2026-07-14.13"
+SLOW_PROMPT_MIGRATION_SOURCE_VERSIONS = (
+ "tmcra-v4-slow-graph-2026-07-14.12",
+ SLOW_PROMPT_MIGRATION_SOURCE_VERSION,
+)
+SLOW_PROMPT_VERSION = "tmcra-v4-slow-graph-2026-07-14.16"
+SLOW_SUMMARY_CONTRACT_VERSION = "tmcra.v4.slow-lossless-summary.2"
+SLOW_PARTITION_CONTRACT_VERSION = "tmcra.v4.slow-semantic-partition.2"
+SLOW_EVIDENCE_BINDING_CONTRACT_VERSION = "tmcra.v4.slow-evidence-binding.3"
+SUPPORTED_SLOW_EVIDENCE_BINDING_CONTRACT_VERSIONS = frozenset(
+ {
+ None,
+ "tmcra.v4.slow-evidence-binding.2",
+ SLOW_EVIDENCE_BINDING_CONTRACT_VERSION,
+ }
+)
+SLOW_PROCESS_LOSS_RECOVERY_VERSION = "tmcra.v4.slow-process-loss-recovery.1"
+SLOW_PROCESS_LOSS_PHYSICAL_CALLS_MAX = 3
+PROCESS_LOSS_INTERRUPTION_ERROR = (
+ "claim lease expired; external call outcome uncertain; explicit resume required"
+)
+SLOW_SUMMARY_MAX_CHARS = 4096
+SLOW_MAX_REGION_OPERATIONS = 32
+FLASH_ESCALATION_REASON = "cross_slot_conflict"
+_CAPSULE_KEY_PATTERN = re.compile(r"[a-z0-9]+(?:\.[a-z0-9]+)*")
+_GENERIC_REGION_KEYS = frozenset(
+ {
+ "activities",
+ "activity",
+ "belief",
+ "beliefs",
+ "communication",
+ "goals",
+ "interest",
+ "interests",
+ "learning",
+ "opinion",
+ "opinions",
+ "preference",
+ "preferences",
+ "routine",
+ "routines",
+ "skills",
+ }
+)
+_CAPSULE_KEY_SCAFFOLD_TOKENS = frozenset({"memory", "user"})
+GENERIC_MULTI_SLOT_CAPSULE_KEY_ERROR = (
+ "capsule_key must name a concrete semantic topic for a generic-region "
+ "capsule containing multiple canonical slots"
+)
+LEGACY_SINGLE_BINDING_ERROR_PREFIX = (
+ "atomic Fast evidence may belong to only one resulting claim: "
+)
+ZERO_CALL_CONFIGURATION_ERRORS = {
+ "flash": "flash client is not configured; no fallback is allowed",
+ "pro": "pro client is not configured; no fallback is allowed",
+}
+
+
+class TieredAPIError(DeepSeekCallError):
+ """A physical API failure that is never eligible for automatic retry."""
+
+ def __init__(self, message: str) -> None:
+ super().__init__(message, retryable=False)
+
+
+def _clean(value: Any) -> str:
+ return str(value).strip() if value is not None else ""
+
+
+def _json(value: Any) -> str:
+ return json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
+
+
+def _digest(value: Any) -> str:
+ return hashlib.sha256(_json(value).encode("utf-8")).hexdigest()
+
+
+_OPERATIONAL_SUMMARY_PATTERNS = tuple(
+ re.compile(pattern, re.IGNORECASE)
+ for pattern in (
+ r"^\s*deterministic\s+(?:create|revise|retire|cleanup|migration)\b",
+ r"^\s*(?:initial\s+)?consolidat(?:e|ed|es|ing|ion)\b[^.]{0,120}\b(?:evidence|claims?|facts?|preferences?|routines?|memory)\b",
+ r"^\s*(?:add|added|adding|challenge|challenged|challenging)\b[^.]{0,80}\bevidence\b",
+ r"^\s*add\b",
+ r"\b(?:supplied|required|current\s+fast)\s+evidence\b",
+ r"^\s*create\s+(?:an?\s+|the\s+)?(?:initial\s+)?[^.]{0,80}\b(?:memory|claims?|record)\b",
+ r"^\s*(?:memory|record)\s+(?:revision|cleanup|migration)\b",
+ r"^\s*(?:创建|新增|更新|合并|整理).{0,24}(?:记忆|胶囊|证据|声明|记录)",
+ r"^\s*(?:控制器|确定性创建|确定性修订).{0,24}(?:记忆|证据|声明|记录|胶囊)",
+ r"(?:快速图证据)",
+ )
+)
+_GENERIC_SUMMARY_PATTERNS = tuple(
+ re.compile(pattern, re.IGNORECASE)
+ for pattern in (
+ r"^user(?:'s)?\s+(?:fitness\s+)?(?:goals?\s+and\s+facts?|commute\s+details?|schedule\s+routine|preferences?\s+and\s+facts?)(?:\.)?$",
+ r"^user\s+(?:has\s+)?social\s+media\s+goals?\s+and\s+routines?(?:\.)?$",
+ r"^user\s+opinions?\s+(?:on|about)\s+[^.]+(?:\.)?$",
+ r"^user\s+preferences?\s+and\s+facts?\s+about\s+[^.]+(?:\.)?$",
+ r"^user(?:'s)?\s+[^.]+\s+preferences?\s+and\s+wearing\s+frequency(?:\.)?$",
+ r"^(?:用户|使用者)的?(?:健身目标和事实|通勤详情|日程惯例|偏好和事实)[。.]?$",
+ )
+)
+
+
+def _semantic_summary_projection(claims: Any) -> str:
+ if not isinstance(claims, list) or not claims:
+ raise PatchValidationError("semantic summary projection requires claims")
+ texts: list[str] = []
+ for index, claim in enumerate(claims):
+ if not isinstance(claim, Mapping):
+ raise PatchValidationError(
+ f"semantic summary projection claim {index} is not an object"
+ )
+ text = _required_text(claim.get("text"), f"claim {index} text")
+ texts.append(text)
+ summary = " ".join(texts)
+ if len(summary) > SLOW_SUMMARY_MAX_CHARS:
+ raise PatchValidationError(
+ "lossless semantic summary projection exceeds the summary limit"
+ )
+ return summary
+
+
+def validate_semantic_summary(
+ summary: Any,
+ claims: Any,
+ *,
+ label: str = "slow GraphPatch summary",
+) -> str:
+ """Validate only high-confidence summary contract violations.
+
+ Semantic quality remains a model responsibility and is reviewed separately;
+ this gate rejects empty, structured, operational, and known heading-only text.
+ """
+ value = " ".join(_clean(summary).split())
+ if not value:
+ raise PatchValidationError(f"{label} is required")
+ if len(value) > SLOW_SUMMARY_MAX_CHARS:
+ raise PatchValidationError(
+ f"{label} exceeds {SLOW_SUMMARY_MAX_CHARS} characters"
+ )
+ if not isinstance(claims, list) or not claims:
+ raise PatchValidationError(f"{label} requires non-empty claims")
+ if value[:1] in "[{":
+ try:
+ structured = json.loads(value)
+ except json.JSONDecodeError:
+ structured = None
+ if isinstance(structured, (dict, list)):
+ raise PatchValidationError(f"{label} must not be JSON")
+ for pattern in _OPERATIONAL_SUMMARY_PATTERNS:
+ if pattern.search(value):
+ raise PatchValidationError(
+ f"{label} describes a graph operation instead of user memory"
+ )
+ for pattern in _GENERIC_SUMMARY_PATTERNS:
+ if pattern.fullmatch(value):
+ raise PatchValidationError(
+ f"{label} is a generic heading instead of a semantic memory statement"
+ )
+ if sum(character.isalnum() for character in value) < 4:
+ raise PatchValidationError(f"{label} has no substantive semantic content")
+ return value
+
+
+def _validate_patch_summary_contract(patch: Mapping[str, Any]) -> None:
+ operations = patch.get("operations")
+ if not isinstance(operations, list):
+ raise PatchValidationError("summary contract requires an operations list")
+ for index, operation in enumerate(operations):
+ if not isinstance(operation, Mapping):
+ raise PatchValidationError("summary contract operation must be an object")
+ if _clean(operation.get("action")) == "noop":
+ continue
+ summary = validate_semantic_summary(
+ operation.get("summary"),
+ operation.get("claims"),
+ label=f"operations[{index}].summary",
+ )
+ expected = _semantic_summary_projection(operation.get("claims"))
+ if summary != expected:
+ raise PatchValidationError(
+ f"operations[{index}].summary must equal the lossless final-claim projection"
+ )
+
+
+def _normalize_transport_patch(
+ patch: Any,
+ capsules: list[Mapping[str, Any]],
+ region: Mapping[str, Any] | None = None,
+) -> tuple[Any, list[dict[str, str]]]:
+ """Normalize only JSON transport blemishes with one unambiguous meaning."""
+ if not isinstance(patch, Mapping):
+ return patch, []
+ operations = patch.get("operations")
+ normalizations: list[dict[str, str]] = []
+ if isinstance(operations, Mapping):
+ operations = [operations]
+ normalizations.append(
+ {
+ "code": "single_operation_object_wrapped_as_list",
+ "field": "operations",
+ "reason": "one operation object is an unambiguous singleton operations list",
+ }
+ )
+ if isinstance(operations, list) and len(operations) > 1 and not capsules:
+ evidence = region.get("evidence", []) if isinstance(region, Mapping) else []
+ evidence_ids = {
+ _clean(item.get("memory_id"))
+ for item in evidence
+ if isinstance(item, Mapping) and _clean(item.get("memory_id"))
+ }
+ pure_evidence_noops = all(
+ isinstance(operation, Mapping)
+ and set(operation).issubset({"action", "capsule_id"})
+ and operation.get("action") == "noop"
+ and (
+ "capsule_id" not in operation
+ or _clean(operation.get("capsule_id")) in evidence_ids
+ )
+ for operation in operations
+ )
+ if pure_evidence_noops:
+ operations = [{"action": "noop"}]
+ normalizations.append(
+ {
+ "code": "multiple_evidence_keyed_noops_collapsed",
+ "field": "operations",
+ "reason": "multiple capsule-free noops keyed only by supplied evidence IDs have one no-op effect",
+ }
+ )
+ if not isinstance(operations, list):
+ return patch, []
+ evidence = region.get("evidence", []) if isinstance(region, Mapping) else []
+ normalized_operations: list[Any] = []
+ for operation_index, raw_operation in enumerate(operations):
+ if not isinstance(raw_operation, Mapping):
+ normalized_operations.append(raw_operation)
+ continue
+ operation = dict(raw_operation)
+ if (
+ operation.get("action") == "noop"
+ and "capsule_id" in operation
+ and operation.get("capsule_id") is None
+ and len(operations) == 1
+ ):
+ del operation["capsule_id"]
+ normalizations.append(
+ {
+ "code": "noop_null_capsule_id_ignored",
+ "field": f"operations[{operation_index}].capsule_id",
+ "reason": "a null optional noop identity has the same meaning as an omitted identity",
+ }
+ )
+ elif (
+ operation.get("action") == "noop"
+ and "capsule_id" in operation
+ and not capsules
+ and len(operations) == 1
+ ):
+ del operation["capsule_id"]
+ normalizations.append(
+ {
+ "code": "noop_identity_ignored_without_supplied_capsule",
+ "field": f"operations[{operation_index}].capsule_id",
+ "reason": "a capsule-free region noop cannot target a model-invented capsule identity",
+ }
+ )
+
+ if operation.get("action") == "create" and _clean(
+ operation.get("capsule_key")
+ ):
+ raw_capsule_key = _clean(operation.get("capsule_key"))
+ normalized_capsule_key = _normalize_transport_capsule_key(
+ raw_capsule_key
+ )
+ if (
+ normalized_capsule_key is not None
+ and normalized_capsule_key != raw_capsule_key
+ ):
+ operation["capsule_key"] = normalized_capsule_key
+ normalizations.append(
+ {
+ "code": "create_capsule_key_ascii_separators_normalized",
+ "field": f"operations[{operation_index}].capsule_key",
+ "reason": "ASCII case and word separators have one stable dot-separated identifier representation",
+ }
+ )
+
+ claims = operation.get("claims")
+ if isinstance(claims, list) and isinstance(evidence, list):
+ normalized_claims: list[Any] = []
+ for claim_index, raw_claim in enumerate(claims):
+ if not isinstance(raw_claim, Mapping):
+ normalized_claims.append(raw_claim)
+ continue
+ claim = dict(raw_claim)
+ if claim.get("support") == [] and claim.get("counterevidence") == []:
+ slot = _clean(claim.get("canonical_slot"))
+ text = _normal_text(claim.get("text"))
+ matches = [
+ item
+ for item in evidence
+ if isinstance(item, Mapping)
+ and _clean(item.get("canonical_slot")) == slot
+ and _normal_text(item.get("value")) == text
+ and _clean(item.get("memory_id"))
+ ]
+ if slot and text and len(matches) == 1:
+ evidence_id = _clean(matches[0].get("memory_id"))
+ claim["support"] = [evidence_id]
+ normalizations.append(
+ {
+ "code": "empty_support_bound_to_unique_exact_evidence",
+ "field": f"operations[{operation_index}].claims[{claim_index}].support",
+ "reason": "canonical_slot and normalized text uniquely match one supplied evidence leaf",
+ }
+ )
+ normalized_claims.append(claim)
+ operation["claims"] = normalized_claims
+ if operation.get("action") == "create" and not _clean(
+ operation.get("capsule_key")
+ ):
+ slots = {
+ _clean(item.get("canonical_slot"))
+ for item in normalized_claims
+ if isinstance(item, Mapping)
+ and _clean(item.get("canonical_slot"))
+ }
+ if len(slots) == 1:
+ operation["capsule_key"] = _capsule_key_from_slot(
+ next(iter(slots))
+ )
+ normalizations.append(
+ {
+ "code": "create_capsule_key_bound_to_unique_claim_slot",
+ "field": f"operations[{operation_index}].capsule_key",
+ "reason": "all create claims share one authoritative canonical slot",
+ }
+ )
+ normalized_operations.append(operation)
+
+ if not normalizations:
+ return patch, []
+ normalized = dict(patch)
+ normalized["operations"] = normalized_operations
+ return normalized, normalizations
+
+
+def _required_text(value: Any, label: str) -> str:
+ result = _clean(value)
+ if not result:
+ raise PatchValidationError(f"{label} is required")
+ return result
+
+
+def _normal_text(value: Any) -> str:
+ return " ".join(_clean(value).casefold().split())
+
+
+def _normalize_capsule_key(value: Any, label: str = "capsule_key") -> str:
+ key = _clean(value).casefold()
+ if not key or len(key) > 96 or _CAPSULE_KEY_PATTERN.fullmatch(key) is None:
+ raise PatchValidationError(
+ f"{label} must be a lowercase dot-separated identifier of at most 96 characters"
+ )
+ return key
+
+
+def _normalize_transport_capsule_key(value: Any) -> str | None:
+ """Return a strict key only when ASCII spelling cleanup is lossless."""
+ raw = _clean(value)
+ if not raw:
+ return None
+ key = re.sub(r"[\s_-]+", ".", raw.casefold())
+ key = re.sub(r"\.+", ".", key).strip(".")
+ if not key or len(key) > 96 or _CAPSULE_KEY_PATTERN.fullmatch(key) is None:
+ return None
+ return key
+
+
+def _capsule_key_from_slot(slot: Any) -> str:
+ value = _required_text(slot, "canonical_slot").casefold()
+ if len(value) <= 96 and _CAPSULE_KEY_PATTERN.fullmatch(value) is not None:
+ return value
+ return "slot." + _digest({"canonical_slot": value})[:24]
+
+
+def _is_generic_region_key(value: Any) -> bool:
+ return _clean(value).casefold() in _GENERIC_REGION_KEYS
+
+
+def _capsule_key_is_generic_for_region(region_key: Any, capsule_key: Any) -> bool:
+ key_tokens = {
+ item
+ for item in re.split(r"[^a-z0-9]+", _clean(capsule_key).casefold())
+ if item
+ }
+ region_tokens = {
+ item
+ for item in re.split(r"[^a-z0-9]+", _clean(region_key).casefold())
+ if item
+ }
+ # This gate is intentionally structural. Pro owns semantic grouping; the
+ # controller rejects only identities that add nothing beyond the region
+ # name and storage scaffolding.
+ return bool(key_tokens) and key_tokens <= (
+ _CAPSULE_KEY_SCAFFOLD_TOKENS | region_tokens
+ )
+
+
+def _validate_generic_create_partition_keys(
+ region_key: Any, patch: Mapping[str, Any]
+) -> None:
+ """Reject only high-confidence generic multi-topic create identities."""
+ if not _is_generic_region_key(region_key):
+ return
+ for index, operation in enumerate(patch.get("operations", [])):
+ if not isinstance(operation, Mapping) or operation.get("action") != "create":
+ continue
+ claims = operation.get("claims")
+ if not isinstance(claims, list):
+ continue
+ slots = {
+ _clean(claim.get("canonical_slot"))
+ for claim in claims
+ if isinstance(claim, Mapping) and _clean(claim.get("canonical_slot"))
+ }
+ if len(slots) > 1 and _capsule_key_is_generic_for_region(
+ region_key, operation.get("capsule_key")
+ ):
+ raise PatchValidationError(
+ f"operations[{index}].{GENERIC_MULTI_SLOT_CAPSULE_KEY_ERROR}"
+ )
+
+
+def _generic_region_requires_semantic_management(
+ region_key: Any,
+ evidence: list[Mapping[str, Any]],
+ capsules: list[Mapping[str, Any]],
+) -> bool:
+ if not _is_generic_region_key(region_key):
+ return False
+ slots = {
+ _leaf_slot(item)
+ for item in evidence
+ if _is_current_durable(item) or _is_challenged_durable(item)
+ }
+ for capsule in capsules:
+ if _clean(capsule.get("status")).casefold() not in {"active", "challenged"}:
+ continue
+ for claim in capsule.get("claims") or []:
+ if isinstance(claim, Mapping) and _clean(claim.get("canonical_slot")):
+ slots.add(_clean(claim.get("canonical_slot")))
+ return len(slots) > 1
+
+
+def _semantic_partition_targets(
+ region_key: Any, capsules: list[Mapping[str, Any]]
+) -> set[str]:
+ targets: set[str] = set()
+ for capsule in capsules:
+ if _clean(capsule.get("status")).casefold() not in {"active", "challenged"}:
+ continue
+ claims = capsule.get("claims")
+ projected_length = (
+ sum(
+ len(" ".join(_clean(claim.get("text")).split()))
+ for claim in claims
+ if isinstance(claim, Mapping)
+ )
+ + max(0, len(claims) - 1)
+ if isinstance(claims, list)
+ else 0
+ )
+ if (
+ isinstance(claims, list)
+ and claims
+ and (
+ projected_length > SLOW_SUMMARY_MAX_CHARS
+ or capsule.get("partition_contract_version")
+ != SLOW_PARTITION_CONTRACT_VERSION
+ )
+ ):
+ targets.add(_required_text(capsule.get("capsule_id"), "capsule_id"))
+ return targets
+
+
+def _partition_targets_require_model(
+ capsules: list[Mapping[str, Any]], targets: set[str]
+) -> bool:
+ for capsule in capsules:
+ capsule_id = _clean(capsule.get("capsule_id"))
+ if capsule_id not in targets:
+ continue
+ claims = capsule.get("claims")
+ if not isinstance(claims, list) or len(claims) != 1:
+ return True
+ try:
+ _semantic_summary_projection(claims)
+ except PatchValidationError:
+ return True
+ return False
+
+
+def _canonical_patch_claims(value: Any) -> list[dict[str, Any]]:
+ if not isinstance(value, list) or not value:
+ raise PatchValidationError("claims must be a non-empty list")
+ claims = [
+ _patch_claim_projection(claim)
+ for claim in value
+ if isinstance(claim, Mapping)
+ ]
+ if len(claims) != len(value):
+ raise PatchValidationError("each claim must be an object")
+ claims.sort(
+ key=lambda claim: (
+ claim["canonical_slot"],
+ _normal_text(claim["text"]),
+ tuple(claim["support"]),
+ tuple(claim["counterevidence"]),
+ )
+ )
+ return claims
+
+
+def _materialize_lossless_summaries(patch: Mapping[str, Any]) -> dict[str, Any]:
+ operations = patch.get("operations")
+ if not isinstance(operations, list):
+ raise PatchValidationError("lossless summary materialization requires operations")
+ normalized: list[dict[str, Any]] = []
+ for operation in operations:
+ if not isinstance(operation, Mapping):
+ raise PatchValidationError("GraphPatch operation must be an object")
+ current = dict(operation)
+ if _clean(current.get("action")) != "noop":
+ claims = _canonical_patch_claims(current.get("claims"))
+ current["claims"] = claims
+ current["summary"] = _semantic_summary_projection(claims)
+ normalized.append(current)
+ return {"operations": normalized}
+
+
+def validate_v4_patch(
+ patch: Mapping[str, Any], *, require_lossless_summary: bool = False
+) -> None:
+ if (
+ not isinstance(patch, Mapping)
+ or set(patch) != {"operations"}
+ or not isinstance(patch["operations"], list)
+ ):
+ raise PatchValidationError("GraphPatch must contain exactly an operations list")
+ operations = patch["operations"]
+ if not operations:
+ raise PatchValidationError("GraphPatch must contain at least one operation")
+ if len(operations) > SLOW_MAX_REGION_OPERATIONS:
+ raise PatchValidationError(
+ f"GraphPatch exceeds {SLOW_MAX_REGION_OPERATIONS} region operations"
+ )
+
+ create_keys: set[str] = set()
+ capsule_targets: set[str] = set()
+ noop_count = 0
+ for index, operation in enumerate(operations):
+ if not isinstance(operation, Mapping):
+ raise PatchValidationError("GraphPatch operation must be an object")
+ action = _required_text(operation.get("action"), "action")
+ if action not in PATCH_ACTIONS:
+ raise PatchValidationError("unknown GraphPatch action")
+ if "confidence" in operation:
+ raise PatchValidationError(
+ "model confidence is not an authoritative graph field"
+ )
+ allowed = {"action", "summary", "claims"}
+ if action == "create":
+ allowed.update({"capsule_key", "capsule_id"})
+ capsule_key = _normalize_capsule_key(
+ operation.get("capsule_key"),
+ f"operations[{index}].capsule_key",
+ )
+ if capsule_key in create_keys:
+ raise PatchValidationError(
+ f"duplicate create capsule_key in one patch: {capsule_key}"
+ )
+ create_keys.add(capsule_key)
+ if "capsule_id" in operation and operation.get("capsule_id") is not None:
+ raise PatchValidationError(
+ "create capsule identity is controller-derived from capsule_key"
+ )
+ elif action == "noop":
+ noop_count += 1
+ allowed.add("capsule_id")
+ if "capsule_id" in operation:
+ capsule_id = _required_text(operation.get("capsule_id"), "capsule_id")
+ if capsule_id in capsule_targets:
+ raise PatchValidationError(
+ f"duplicate capsule target in one patch: {capsule_id}"
+ )
+ capsule_targets.add(capsule_id)
+ else:
+ allowed.update({"capsule_id", "base_revision"})
+ capsule_id = _required_text(operation.get("capsule_id"), "capsule_id")
+ if capsule_id in capsule_targets:
+ raise PatchValidationError(
+ f"duplicate capsule target in one patch: {capsule_id}"
+ )
+ capsule_targets.add(capsule_id)
+ if (
+ isinstance(operation.get("base_revision"), bool)
+ or not isinstance(operation.get("base_revision"), int)
+ or operation["base_revision"] < 1
+ ):
+ raise PatchValidationError("base_revision must be a positive integer")
+ if set(operation) - allowed:
+ raise PatchValidationError("unexpected GraphPatch operation fields")
+ if action != "noop":
+ _v3._validate_claims(operation.get("claims"))
+
+ if noop_count and len(operations) != 1:
+ raise PatchValidationError("noop must be the only operation in a GraphPatch")
+ if require_lossless_summary:
+ _validate_patch_summary_contract(patch)
+
+
+# V4 owns a multi-capsule GraphPatch schema; V3 remains single-capsule.
+validate_patch = validate_v4_patch
+
+
+def _forbidden_field(name: str) -> bool:
+ key = _clean(name).casefold()
+ if key in {"qid", "query_id", "answer_id", "answer_session_id"}:
+ return True
+ return any(token in key for token in ("benchmark", "question", "answer", "judge", "gold", "label"))
+
+
+def _assert_no_benchmark_fields(value: Any, path: str = "payload") -> None:
+ if isinstance(value, Mapping):
+ for key, item in value.items():
+ if _forbidden_field(str(key)):
+ raise EvidencePolicyError(f"benchmark field is forbidden in slow-graph request: {path}.{key}")
+ _assert_no_benchmark_fields(item, f"{path}.{key}")
+ elif isinstance(value, list):
+ for index, item in enumerate(value):
+ _assert_no_benchmark_fields(item, f"{path}[{index}]")
+
+
+@dataclass(frozen=True)
+class DeepSeekTierConfig:
+ base_url: str
+ key_pool: tuple[str, ...]
+ max_tokens: int
+ model: str = "deepseek-v4-flash"
+ prompt_cost_per_million: float = 0.0
+ completion_cost_per_million: float = 0.0
+ cache_cost_per_million: float = 0.0
+
+ @classmethod
+ def from_env(cls, prefix: str, *, model: str) -> "DeepSeekTierConfig":
+ base_url = _clean(os.getenv(prefix + "_BASE_URL"))
+ keys = tuple(
+ item.strip()
+ for item in _clean(os.getenv(prefix + "_KEY_POOL")).split(",")
+ if item.strip()
+ )
+ try:
+ max_tokens = int(_clean(os.getenv(prefix + "_MAX_TOKENS")))
+ except ValueError as exc:
+ raise SlowGraphError(f"{prefix}_MAX_TOKENS must be an integer") from exc
+ if not base_url or not keys or max_tokens <= 0:
+ raise SlowGraphError(f"{prefix} requires BASE_URL, KEY_POOL, and positive MAX_TOKENS")
+ return cls(
+ base_url.rstrip("/"),
+ keys,
+ max_tokens,
+ model=model,
+ prompt_cost_per_million=float(os.getenv(prefix + "_PROMPT_COST_PER_MILLION", "0")),
+ completion_cost_per_million=float(os.getenv(prefix + "_COMPLETION_COST_PER_MILLION", "0")),
+ cache_cost_per_million=float(os.getenv(prefix + "_CACHE_COST_PER_MILLION", "0")),
+ )
+
+
+DeepSeekFlashConfig = DeepSeekTierConfig
+
+
+@dataclass(frozen=True)
+class DeepSeekProConfig(DeepSeekTierConfig):
+ model: str = "deepseek-v4-pro"
+
+
+def _optional_config(prefix: str, model: str) -> DeepSeekTierConfig | None:
+ values = [os.getenv(prefix + suffix) for suffix in ("_BASE_URL", "_KEY_POOL", "_MAX_TOKENS")]
+ if not any(_clean(value) for value in values):
+ return None
+ return DeepSeekTierConfig.from_env(prefix, model=model)
+
+
+class _DeepSeekTierClient:
+ def __init__(self, config: DeepSeekTierConfig, *, route: str) -> None:
+ expected_model = {
+ "flash": "deepseek-v4-flash",
+ "pro": "deepseek-v4-pro",
+ }.get(route)
+ if expected_model is None or config.model != expected_model:
+ raise SlowGraphError(
+ f"slow-graph route {route!r} requires model {expected_model!r}, "
+ f"got {config.model!r}"
+ )
+ self.config = config
+ self.route = route
+ self._key_index = 0
+ self.last_call_metadata: Mapping[str, Any] = {}
+
+ def _messages(
+ self,
+ region: Mapping[str, Any],
+ capsules: list[Mapping[str, Any]],
+ *,
+ correction: Mapping[str, Any] | None = None,
+ ) -> list[dict[str, str]]:
+ required_evidence_ids = sorted(
+ {
+ _required_text(item, "required evidence ID")
+ for item in region.get("required_evidence_ids", [])
+ }
+ )
+ partition_required = region.get("semantic_partition_required") is True
+ partition_mode = _clean(region.get("semantic_partition_mode"))
+ partition_capsule_ids = sorted(
+ {
+ _required_text(item, "partition capsule ID")
+ for item in region.get("partition_capsule_ids", [])
+ }
+ )
+ if not partition_required and (partition_mode or partition_capsule_ids):
+ raise EvidencePolicyError(
+ "semantic partition metadata requires semantic_partition_required"
+ )
+ if partition_required and partition_mode not in {"manage", "migrate"}:
+ raise EvidencePolicyError("semantic partition mode is invalid")
+ if partition_mode == "migrate" and not partition_capsule_ids:
+ raise EvidencePolicyError(
+ "semantic partition migration requires explicit partition_capsule_ids"
+ )
+ if partition_mode == "manage" and partition_capsule_ids:
+ raise EvidencePolicyError(
+ "generic semantic management cannot name legacy partition targets"
+ )
+ route_instruction = (
+ "The controller selected compatible consolidation. Assign each new atomic "
+ "claim to one coherent existing or new capsule."
+ if self.route == "flash"
+ else
+ "The controller selected full-state adjudication. Resolve genuine same-property "
+ "conflicts while preserving uncertainty when the supplied evidence does not "
+ "establish one current value."
+ )
+ if capsules and required_evidence_ids:
+ allowed_actions = (
+ "revise or create"
+ if self.route == "flash"
+ else "revise, create, challenge, resolve_challenge, or retire"
+ )
+ identity_instruction = (
+ "For revise, challenge, resolve_challenge, or retire, copy capsule_id and "
+ "base_revision exactly from one supplied capsule. For create, emit a stable "
+ "lowercase dot-separated capsule_key and omit capsule_id/base_revision. "
+ "noop is forbidden because uncited durable evidence is pending."
+ )
+ elif not capsules and required_evidence_ids:
+ allowed_actions = "one or more create operations"
+ identity_instruction = (
+ "Every create operation contains a unique stable lowercase dot-separated "
+ "capsule_key and omits capsule_id/base_revision; the controller derives "
+ "capsule identity. noop is forbidden because uncited durable evidence is pending."
+ )
+ elif capsules:
+ allowed_actions = (
+ "revise, create, or noop"
+ if self.route == "flash"
+ else "revise, create, challenge, resolve_challenge, retire, or noop"
+ )
+ identity_instruction = (
+ "For operations on supplied capsules, copy capsule_id and base_revision "
+ "exactly. For create, provide a unique stable lowercase dot-separated "
+ "capsule_key and omit capsule_id/base_revision."
+ )
+ else:
+ allowed_actions = "one or more create operations, or one noop"
+ identity_instruction = (
+ "Every create operation contains a unique stable lowercase dot-separated "
+ "capsule_key and omits capsule_id/base_revision."
+ )
+ if self.route == "flash" and required_evidence_ids:
+ allowed_actions += " or escalate"
+ delta_instruction = (
+ "This is an additive delta proposal. Every operation claim must cite only IDs in "
+ "region.required_evidence_ids and describe only those new evidence items. Use revise "
+ "when a delta belongs in one supplied capsule and create when it forms a different "
+ "semantic topic. Do not repeat or cite existing capsule claims; the controller will "
+ "merge each revise delta after validation. "
+ if self.route == "flash" and capsules and required_evidence_ids
+ else
+ "For each operated existing capsule, claims are its complete next revision. Preserve "
+ "every still-current supplied claim unless supplied evidence resolves it or the claim "
+ "is moved intact to a newly created coherent capsule. "
+ if capsules
+ else ""
+ )
+ migration_partition_instruction = (
+ "semantic_partition_required is true. The supplied legacy capsule mixes semantic "
+ "topics. Operate on every capsule ID listed in region.partition_capsule_ids and "
+ "repartition all of its current claims "
+ "across coherent next capsules. Keep one coherent group on a revised supplied "
+ "capsule and create additional capsules as needed; never leave a supplied capsule "
+ "listed for partition untouched, duplicate a claim, or group unrelated topics merely "
+ "because they share a generic region name. Other supplied capsules are context and "
+ "may remain untouched unless one must be revised to avoid duplicating the same "
+ "self-contained semantic claim across capsules. "
+ )
+ managed_partition_instruction = (
+ "semantic_partition_required is true because this region needs explicit multi-slot "
+ "semantic management. Manage the complete supplied durable state as coherent "
+ "real-world topics, revising coherent existing capsules and creating additional "
+ "capsules only when the real-world topics differ. Never group claims merely because "
+ "they share the region name, but do not split related claims merely because their "
+ "canonical slots or claim types differ. "
+ "Every multi-slot create must use a concrete topic-specific capsule_key; keys made "
+ "only from user, region, or claim-type words are forbidden. "
+ )
+ topic_granularity_instruction = (
+ "A capsule is one reusable retrieval concept centered on the same real-world person, "
+ "object, activity, project, relationship, decision, or behavioral objective; it is "
+ "not a schema field or claim-type bucket. First cluster by that shared referent and "
+ "intent. Keep facts, preferences, goals, constraints, plans, and routines together "
+ "when they jointly describe that concept, including a goal and the plan or routine "
+ "used to achieve it. Different canonical slots, claim types, or timestamps alone do "
+ "not justify separate capsules. Conversely, a shared broad region such as business, "
+ "work, schedule, goals, or reading does not justify grouping independent concrete "
+ "referents. Prefer real-world co-reference over taxonomy: details about one product "
+ "and a model used for that product belong together, while an unrelated second "
+ "business remains separate; never regroup them as identity versus possession. For "
+ "each proposed capsule, there must be a natural memory-retrieval question for which "
+ "every claim in that capsule is useful evidence. Name capsule_key after the concrete "
+ "referent or retrieval use-case, never after a broad region or schema role. Use a "
+ "singleton capsule only when no other supplied current claim shares its real-world "
+ "topic, and use multiple operations only when the groups would normally be retrieved "
+ "independently. "
+ )
+ partition_instruction = (
+ migration_partition_instruction
+ if partition_mode == "migrate"
+ else managed_partition_instruction
+ if partition_mode == "manage"
+ else
+ "Use multiple operations only when the claims form genuinely different semantic "
+ "topics; keep related facts, preferences, goals, and routines together. "
+ )
+ conflict_instruction = (
+ "If support-role evidence with different canonical slots may describe mutually "
+ "exclusive values of the same changing property, do not choose a winner. Return "
+ "exactly {\"operations\":[{\"action\":\"escalate\","
+ "\"reason\":\"cross_slot_conflict\"}]} so the controller can route to Pro. "
+ if self.route == "flash" and required_evidence_ids
+ else "First decide whether the evidence actually conflicts. Two statements are "
+ "compatible unless they cannot both be true of the same subject and changing property "
+ "at the same time. A shared topic, different canonical slots, negative wording, or "
+ "different details about one topic is not a conflict. Compatible statements must be "
+ "separate support-only claims. Never emit reciprocal claims that use each other's "
+ "support as counterevidence. Use turn_index and temporal_status only for a genuine "
+ "same-property state change. Prefer a later explicit current observation or correction "
+ "only when it resolves that same property; otherwise preserve uncertainty with one "
+ "challenge claim rather than mirrored alternatives. A create operation cannot contain "
+ "counterevidence because a new unresolved capsule must not be committed as active. "
+ )
+ system = (
+ "Return exactly one JSON GraphPatch and no prose. The top-level object must contain "
+ "exactly one key named operations; never echo the user envelope, region, capsules, "
+ f"route, schema, or schema_version. operations contains 1 to {SLOW_MAX_REGION_OPERATIONS} "
+ "atomic capsule operations. "
+ f"The only allowed actions for this request are {allowed_actions}. {identity_instruction} "
+ "A noop operation contains only action and optionally one supplied capsule_id, and "
+ "must be the only operation. Every non-noop operation contains a non-empty claims list. "
+ "Do not return summary; the controller deterministically derives the committed lossless "
+ "summary from the final claims after merge. "
+ "Each claim contains only canonical_slot, text, support, and counterevidence. "
+ "support and counterevidence are arrays of supplied fast evidence IDs. "
+ "A claim may cite multiple support IDs only when their normalized evidence texts "
+ "are identical. When same-slot evidence texts differ, preserve each evidence meaning "
+ "as a separate claim in the same coherent capsule; never compress distinct Fast "
+ "evidence texts into one claim. "
+ "One indivisible Fast evidence ID may itself name multiple parallel concrete "
+ "referents. Never duplicate or split that evidence ID across claims or capsules. "
+ "Preserve its complete compound statement as one self-contained claim in one "
+ "concrete shared retrieval-use-case capsule that naturally covers every named "
+ "referent. "
+ "Claim text must be a self-contained user-memory statement: name the subject and "
+ "object needed to understand it without neighboring claims, resolve pronouns or "
+ "deictic phrases only when supplied evidence makes the referent explicit, and never "
+ "invent a referent. Preserve epistemic force exactly (for example, heard, suspects, "
+ "plans, prefers, and knows are not interchangeable), preserve quantities and temporal "
+ "qualifiers, and do not broaden or weaken the atomic fact. "
+ "Use only supplied fast evidence. polarity describes the statement's content; it does "
+ "not make a negative statement counterevidence. evidence_role is authoritative: every "
+ "Fast evidence whose record_state is challenged, superseded, or otherwise non-current "
+ "must never become a new support binding or create a new active capsule. It may remain "
+ "as historical support only when an existing claim is explicitly adjudicated against "
+ "current replacement evidence in counterevidence. "
+ "support-role required ID belongs in support, and every support ID must be attached to "
+ "a claim whose canonical_slot exactly equals that evidence item's canonical_slot. "
+ "Never consume a distinct slot by attaching its ID to another slot's claim. "
+ "Every non-noop claim must cite supplied evidence IDs and preserve canonical slots. "
+ "Every ID in region.required_evidence_ids must appear exactly once as support or "
+ "counterevidence across all operation claims. canonical_slot is an attribute type, "
+ "not a global entity identity: it may appear in different capsules when the claims "
+ "name different concrete real-world referents. Claims about the same referent and "
+ "retrieval topic belong in one capsule, and an identical self-contained semantic "
+ "claim must never be duplicated across capsules. "
+ + delta_instruction
+ + topic_granularity_instruction
+ + partition_instruction
+ + conflict_instruction
+ +
+ "Do not invent capsule IDs, source IDs, evidence, confidence, benchmark fields, or fields "
+ "outside this GraphPatch contract. " + route_instruction + " "
+ "Do not change routes or repair invalid input locally."
+ )
+ if correction is not None:
+ if self.route != "pro":
+ raise EvidencePolicyError("semantic correction requires the Pro route")
+ rejected_patch = correction.get("rejected_patch")
+ validation_error = _clean(correction.get("validation_error"))
+ if not isinstance(rejected_patch, Mapping) or not validation_error:
+ raise EvidencePolicyError("semantic correction context is incomplete")
+ _assert_no_benchmark_fields(rejected_patch)
+ system += (
+ " This is the single allowed correction pass. The previous Pro GraphPatch "
+ "was rejected by the deterministic controller validator. Return a complete "
+ "replacement GraphPatch, not a commentary or partial edit. Correct the "
+ "underlying semantic partition when topics differ; do not merely rename a "
+ "generic capsule that still mixes unrelated claims, and do not react by putting "
+ "each related claim into its own singleton capsule. The rejected patch was "
+ + _json(rejected_patch)
+ + " The exact validator error was: "
+ + validation_error[:2000]
+ + "."
+ )
+ return [
+ {"role": "system", "content": system},
+ {"role": "user", "content": _json({"region": region, "capsules": capsules})},
+ ]
+
+ @staticmethod
+ def _usage(raw: Mapping[str, Any]) -> dict[str, int]:
+ def integer(*names: str, required: bool = False) -> tuple[int, bool]:
+ for name in names:
+ if raw.get(name) is None:
+ continue
+ value = raw.get(name)
+ if isinstance(value, bool) or not isinstance(value, (int, float)) or int(value) < 0:
+ raise TieredAPIError(f"usage.{name} is invalid")
+ return int(value), True
+ if required:
+ raise TieredAPIError(f"usage.{names[0]} is missing")
+ return 0, False
+
+ prompt, _ = integer("prompt_tokens", "input_tokens", required=True)
+ completion, _ = integer("completion_tokens", "output_tokens", required=True)
+ cached, has_cached = integer(
+ "prompt_cache_hit_tokens", "cache_read_input_tokens", "cached_tokens"
+ )
+ cache_miss, has_miss = integer(
+ "prompt_cache_miss_tokens", "cache_miss_input_tokens"
+ )
+ if cached > prompt or cache_miss > prompt:
+ raise TieredAPIError("cache usage exceeds prompt tokens")
+ if has_cached and has_miss and cached + cache_miss != prompt:
+ raise TieredAPIError("cache hit and miss usage does not balance prompt tokens")
+ if not has_miss:
+ cache_miss = prompt - cached
+ if not has_cached:
+ cached = prompt - cache_miss
+ total, has_total = integer("total_tokens")
+ if has_total and total < prompt + completion:
+ raise TieredAPIError("usage.total_tokens is smaller than prompt plus completion")
+ if not has_total:
+ total = prompt + completion
+ return {
+ "prompt_tokens": prompt,
+ "completion_tokens": completion,
+ "cache_read_input_tokens": cached,
+ "cache_hit_tokens": cached,
+ "cache_miss_tokens": cache_miss,
+ "total_tokens": total,
+ }
+
+ def _metadata(self, **values: Any) -> dict[str, Any]:
+ return {
+ "route": self.route,
+ "prompt_version": SLOW_PROMPT_VERSION,
+ "physical_api_call": True,
+ "physical_api_calls": 1,
+ "api_provider": "deepseek",
+ "model": self.config.model,
+ "attempt_count": 1,
+ **values,
+ }
+
+ def propose(
+ self, region: Mapping[str, Any], capsules: list[Mapping[str, Any]]
+ ) -> Mapping[str, Any]:
+ return self._propose(region, capsules, correction=None)
+
+ def correct(
+ self,
+ region: Mapping[str, Any],
+ capsules: list[Mapping[str, Any]],
+ *,
+ rejected_patch: Mapping[str, Any],
+ validation_error: str,
+ ) -> Mapping[str, Any]:
+ if self.route != "pro":
+ raise TieredAPIError("semantic correction is available only on the Pro route")
+ return self._propose(
+ region,
+ capsules,
+ correction={
+ "rejected_patch": dict(rejected_patch),
+ "validation_error": _required_text(
+ validation_error, "semantic correction validation error"
+ ),
+ },
+ )
+
+ def _propose(
+ self,
+ region: Mapping[str, Any],
+ capsules: list[Mapping[str, Any]],
+ *,
+ correction: Mapping[str, Any] | None,
+ ) -> Mapping[str, Any]:
+ _assert_no_benchmark_fields(region)
+ _assert_no_benchmark_fields(capsules)
+ key_index = self._key_index % len(self.config.key_pool)
+ self._key_index += 1
+ body = {
+ "model": self.config.model,
+ "temperature": 0,
+ "max_tokens": self.config.max_tokens,
+ "thinking": {"type": "disabled"},
+ "enable_thinking": False,
+ "response_format": {"type": "json_object"},
+ "messages": self._messages(region, capsules, correction=correction),
+ }
+ saved_request = {**body, "headers": {"authorization": "redacted"}}
+ request_sha256 = _digest(saved_request)
+ physical_call_id = "dsc_" + uuid.uuid4().hex
+ started = time.time()
+ self.last_call_metadata = self._metadata(
+ physical_call_id=physical_call_id,
+ key_index=key_index,
+ started_at=started,
+ status="started",
+ request=saved_request,
+ request_sha256=request_sha256,
+ )
+ request = urllib.request.Request(
+ self.config.base_url + "/chat/completions",
+ data=_json(body).encode("utf-8"),
+ headers={"Authorization": "Bearer " + self.config.key_pool[key_index], "Content-Type": "application/json"},
+ method="POST",
+ )
+ try:
+ with urllib.request.urlopen(request, timeout=90) as response: # nosec B310
+ status = response.getcode()
+ raw_text = response.read().decode("utf-8")
+ except urllib.error.HTTPError as exc:
+ try:
+ detail = exc.read().decode("utf-8", "replace")
+ except (AttributeError, OSError):
+ detail = ""
+ self.last_call_metadata = self._metadata(
+ physical_call_id=physical_call_id,
+ key_index=key_index,
+ started_at=started,
+ completed_at=time.time(),
+ latency_ms=round((time.time() - started) * 1000, 3),
+ status="http_error",
+ http_status=exc.code,
+ error=detail,
+ request=saved_request,
+ request_sha256=request_sha256,
+ )
+ raise TieredAPIError(f"{self.route} HTTP {exc.code}: {detail}") from exc
+ except (urllib.error.URLError, TimeoutError, OSError) as exc:
+ self.last_call_metadata = self._metadata(
+ physical_call_id=physical_call_id,
+ key_index=key_index,
+ started_at=started,
+ completed_at=time.time(),
+ latency_ms=round((time.time() - started) * 1000, 3),
+ status="request_error",
+ error=f"{exc.__class__.__name__}: {exc}",
+ request=saved_request,
+ request_sha256=request_sha256,
+ )
+ raise TieredAPIError(f"{self.route} transport failure: {exc}") from exc
+ completed = time.time()
+ self.last_call_metadata = self._metadata(
+ physical_call_id=physical_call_id,
+ key_index=key_index,
+ started_at=started,
+ completed_at=completed,
+ latency_ms=round((completed - started) * 1000, 3),
+ status="response_received_unvalidated",
+ http_status=status,
+ raw_response=raw_text,
+ request=saved_request,
+ request_sha256=request_sha256,
+ )
+ try:
+ raw = json.loads(raw_text)
+ except json.JSONDecodeError as exc:
+ self.last_call_metadata = self._metadata(
+ physical_call_id=physical_call_id,
+ started_at=started,
+ completed_at=completed,
+ latency_ms=round((completed - started) * 1000, 3),
+ status="invalid_json",
+ http_status=status,
+ raw_response=raw_text,
+ request=saved_request,
+ request_sha256=request_sha256,
+ )
+ raise TieredAPIError(f"{self.route} response is not JSON") from exc
+ if not isinstance(raw, Mapping):
+ raise TieredAPIError(f"{self.route} response must be an object")
+ choices = raw.get("choices")
+ usage = raw.get("usage")
+ if not isinstance(choices, list) or len(choices) != 1 or not isinstance(usage, Mapping):
+ raise TieredAPIError(f"{self.route} response missing strict choices/usage")
+ choice = choices[0]
+ message = choice.get("message") if isinstance(choice, Mapping) else None
+ finish_reason = _clean(choice.get("finish_reason")) if isinstance(choice, Mapping) else ""
+ content = _clean(message.get("content")) if isinstance(message, Mapping) else ""
+ normalized_usage = self._usage(usage)
+ cost = {
+ "prompt_tokens": normalized_usage["prompt_tokens"],
+ "cache_hit_tokens": normalized_usage["cache_hit_tokens"],
+ "cache_miss_tokens": normalized_usage["cache_miss_tokens"],
+ "completion_tokens": normalized_usage["completion_tokens"],
+ "cache_read_input_tokens": normalized_usage["cache_read_input_tokens"],
+ "prompt_cost_per_million": self.config.prompt_cost_per_million,
+ "completion_cost_per_million": self.config.completion_cost_per_million,
+ "cache_cost_per_million": self.config.cache_cost_per_million,
+ "estimated_cost": (
+ normalized_usage["cache_miss_tokens"] * self.config.prompt_cost_per_million
+ + normalized_usage["cache_hit_tokens"] * self.config.cache_cost_per_million
+ + normalized_usage["completion_tokens"] * self.config.completion_cost_per_million
+ )
+ / 1_000_000,
+ }
+ self.last_call_metadata = self._metadata(
+ physical_call_id=physical_call_id,
+ started_at=started,
+ completed_at=completed,
+ latency_ms=round((completed - started) * 1000, 3),
+ status="response_received",
+ http_status=status,
+ response_id=_clean(raw.get("id")),
+ finish_reason=finish_reason,
+ content=content,
+ usage=normalized_usage,
+ provider_usage=dict(usage),
+ cost_audit=cost,
+ raw_response=raw_text,
+ request=saved_request,
+ request_sha256=request_sha256,
+ )
+ if status < 200 or status >= 300:
+ raise TieredAPIError(f"{self.route} returned HTTP {status}")
+ if finish_reason != "stop":
+ self.last_call_metadata = {**self.last_call_metadata, "status": "incomplete_response"}
+ raise TieredAPIError(f"{self.route} finish_reason must be stop, got {finish_reason!r}")
+ if not content:
+ raise TieredAPIError(f"{self.route} response content is empty")
+ try:
+ raw_patch = json.loads(content)
+ except json.JSONDecodeError as exc:
+ raise TieredAPIError(f"{self.route} content is not JSON") from exc
+ if self.route == "flash" and _flash_escalation_patch(raw_patch):
+ required_evidence_ids = region.get("required_evidence_ids")
+ if not isinstance(required_evidence_ids, list) or not required_evidence_ids:
+ raise TieredAPIError(
+ "flash escalation requires pending durable evidence"
+ )
+ self.last_call_metadata = {
+ **self.last_call_metadata,
+ "status": "completed",
+ "escalation_requested": True,
+ "escalation_reason": FLASH_ESCALATION_REASON,
+ "raw_patch_sha256": _digest(raw_patch),
+ }
+ return raw_patch
+ patch, transport_normalizations = _normalize_transport_patch(
+ raw_patch, capsules, region
+ )
+ if transport_normalizations:
+ self.last_call_metadata = {
+ **self.last_call_metadata,
+ "transport_normalizations": transport_normalizations,
+ "raw_patch_sha256": _digest(raw_patch),
+ "normalized_patch_sha256": _digest(patch),
+ }
+ try:
+ validate_patch(patch)
+ except PatchValidationError as exc:
+ raise TieredAPIError(f"{self.route} returned an invalid GraphPatch: {exc}") from exc
+ self.last_call_metadata = {**self.last_call_metadata, "status": "completed"}
+ return patch
+
+
+def _leaf_metadata(leaf: Mapping[str, Any]) -> Mapping[str, Any]:
+ metadata = leaf.get("metadata")
+ return metadata if isinstance(metadata, Mapping) else {}
+
+
+def _leaf_id(leaf: Mapping[str, Any]) -> str:
+ return _required_text(leaf.get("memory_id"), "fast evidence memory_id")
+
+
+def _leaf_slot(leaf: Mapping[str, Any]) -> str:
+ metadata = _leaf_metadata(leaf)
+ return _required_text(metadata.get("canonical_slot_key") or metadata.get("canonical_slot"), "fast evidence canonical slot")
+
+
+def _leaf_text(leaf: Mapping[str, Any]) -> str:
+ metadata = _leaf_metadata(leaf)
+ return _required_text(leaf.get("value") or metadata.get("source_span") or metadata.get("raw_content"), "fast evidence value")
+
+
+def _leaf_state(leaf: Mapping[str, Any]) -> str:
+ metadata = _leaf_metadata(leaf)
+ state = _clean(
+ leaf.get("record_state") or leaf.get("state") or metadata.get("record_state")
+ ).casefold()
+ if not state:
+ raise EvidencePolicyError("fast evidence record state is missing")
+ return state
+
+
+def _is_current_durable(leaf: Mapping[str, Any]) -> bool:
+ """Return whether the durable memory record is currently authoritative.
+
+ target_status describes when the remembered fact, goal, or plan applies. It
+ is not a lifecycle state: an active durable memory about the past, future,
+ or a planned action remains eligible until the Writer supersedes it.
+ """
+ metadata = _leaf_metadata(leaf)
+ durability = _clean(metadata.get("durability") or metadata.get("durability_class")).casefold()
+ if durability in {"episodic", "uncertain", "", "none"}:
+ return False
+ if durability not in {"durable", "long_term", "long-term", "hard", "persistent"}:
+ return False
+ return _leaf_state(leaf) in {"active", "parallel_active", "promoted"}
+
+
+def _is_challenged_durable(leaf: Mapping[str, Any]) -> bool:
+ metadata = _leaf_metadata(leaf)
+ durability = _clean(
+ metadata.get("durability") or metadata.get("durability_class")
+ ).casefold()
+ return (
+ durability in {"durable", "long_term", "long-term", "hard", "persistent"}
+ and _leaf_state(leaf) == "challenged"
+ )
+
+
+def _is_uncertain(leaf: Mapping[str, Any]) -> bool:
+ metadata = _leaf_metadata(leaf)
+ return _clean(metadata.get("durability") or metadata.get("durability_class")).casefold() == "uncertain"
+
+
+def _is_episodic(leaf: Mapping[str, Any]) -> bool:
+ metadata = _leaf_metadata(leaf)
+ return _clean(metadata.get("durability") or metadata.get("durability_class")).casefold() == "episodic"
+
+
+def _is_counterevidence(leaf: Mapping[str, Any]) -> bool:
+ metadata = _leaf_metadata(leaf)
+ return bool(metadata.get("counterevidence")) or bool(
+ metadata.get("is_counterevidence")
+ )
+
+
+def _flash_escalation_patch(value: Any) -> bool:
+ return value == {
+ "operations": [
+ {"action": "escalate", "reason": FLASH_ESCALATION_REASON}
+ ]
+ }
+
+
+def _prior_counterevidence_ids(capsules: list[Mapping[str, Any]]) -> set[str]:
+ output: set[str] = set()
+ for capsule in capsules:
+ claims = capsule.get("claims")
+ if not isinstance(claims, list):
+ continue
+ for claim in claims:
+ if not isinstance(claim, Mapping):
+ continue
+ values = claim.get("counterevidence")
+ if isinstance(values, list):
+ output.update(_clean(item) for item in values if _clean(item))
+ return output
+
+
+def _support_text_groups(
+ evidence_ids: Iterable[str],
+ evidence_by_id: Mapping[str, Mapping[str, Any]],
+) -> dict[str, list[str]]:
+ groups: dict[str, list[str]] = {}
+ for evidence_id in sorted(set(evidence_ids)):
+ leaf = evidence_by_id.get(evidence_id)
+ if leaf is None:
+ continue
+ text = _normal_text(_leaf_text(leaf))
+ if text:
+ groups.setdefault(text, []).append(evidence_id)
+ return groups
+
+
+def _controlled_complementary_support_bundle(
+ claim_slot: str,
+ evidence_ids: Iterable[str],
+ evidence_by_id: Mapping[str, Mapping[str, Any]],
+) -> bool:
+ """Recognize one parent fact plus compatible durable subslot refinements."""
+ unique_ids = sorted(set(evidence_ids))
+ if len(unique_ids) < 2:
+ return False
+ leaves = [evidence_by_id.get(evidence_id) for evidence_id in unique_ids]
+ if any(
+ leaf is None
+ or not _is_current_durable(leaf)
+ or _is_counterevidence(leaf)
+ for leaf in leaves
+ ):
+ return False
+ typed_leaves = [leaf for leaf in leaves if leaf is not None]
+ slots = [_leaf_slot(leaf) for leaf in typed_leaves]
+ if claim_slot not in slots or len(set(slots)) < 2:
+ return False
+ if any(
+ slot != claim_slot and not slot.startswith(claim_slot + ".")
+ for slot in slots
+ ):
+ return False
+
+ texts_by_slot: dict[str, set[str]] = {}
+ for leaf in typed_leaves:
+ texts_by_slot.setdefault(_leaf_slot(leaf), set()).add(
+ _normal_text(_leaf_text(leaf))
+ )
+ if any(len(texts) != 1 for texts in texts_by_slot.values()):
+ return False
+
+ def one_shared_metadata_value(key: str) -> bool:
+ values = {
+ _clean(_leaf_metadata(leaf).get(key)).casefold()
+ for leaf in typed_leaves
+ }
+ return len(values) == 1 and "" not in values
+
+ if not all(
+ one_shared_metadata_value(key)
+ for key in ("subject_signature", "graph_entity_key", "memory_family")
+ ):
+ return False
+ relations = {
+ _clean(leaf.get("relation") or _leaf_metadata(leaf).get("semantic_slot")).casefold()
+ for leaf in typed_leaves
+ }
+ polarities = {
+ _clean(_leaf_metadata(leaf).get("polarity")).casefold()
+ for leaf in typed_leaves
+ }
+ return (
+ len(relations) == 1
+ and "" not in relations
+ and len(polarities) == 1
+ and "" not in polarities
+ )
+
+
+def _validate_repeated_evidence_bindings(
+ evidence_by_id: Mapping[str, Mapping[str, Any]],
+ bindings_by_id: Mapping[str, list[tuple[str, str, str, str]]],
+) -> None:
+ """Allow compound current support to fan out without weakening provenance rules."""
+ for evidence_id, bindings in sorted(bindings_by_id.items()):
+ if len(bindings) <= 1:
+ continue
+ roles = {role for role, _, _, _ in bindings}
+ if roles != {"support"}:
+ raise PatchValidationError(
+ "repeated Fast evidence bindings must be support-only; "
+ "counterevidence and mixed-role fan-out are forbidden: "
+ + evidence_id
+ )
+ leaf = evidence_by_id.get(evidence_id)
+ if (
+ leaf is None
+ or not _is_current_durable(leaf)
+ or _is_counterevidence(leaf)
+ ):
+ raise PatchValidationError(
+ "only supplied current durable support may bind multiple Slow claims: "
+ + evidence_id
+ )
+ claim_identities = [(slot, text) for _, slot, text, _ in bindings]
+ if len(set(claim_identities)) != len(claim_identities):
+ locations = [location for _, _, _, location in bindings]
+ raise PatchValidationError(
+ "one semantic claim cannot duplicate a Fast evidence binding: "
+ + _json({evidence_id: locations})
+ )
+
+
+def _validate_claim_evidence_contract(
+ region: Mapping[str, Any],
+ capsules: list[Mapping[str, Any]],
+ patch: Mapping[str, Any],
+ *,
+ route: str = "",
+) -> None:
+ """Preserve exact provenance while keeping Flash from inventing conflict."""
+ evidence = [
+ item for item in region.get("evidence", []) if isinstance(item, Mapping)
+ ]
+ evidence_by_id = {_leaf_id(item): item for item in evidence}
+ operations = patch.get("operations")
+ if not isinstance(operations, list) or not operations:
+ raise PatchValidationError("claim evidence contract requires operations")
+ if len(operations) == 1 and isinstance(operations[0], Mapping) and operations[0].get("action") == "noop":
+ return
+ prior_counterevidence = _prior_counterevidence_ids(capsules)
+ prior_support_bindings: set[tuple[str, str, str]] = set()
+ prior_counter_bindings: set[tuple[str, str, str]] = set()
+ for capsule in capsules:
+ prior_claims = capsule.get("claims")
+ if not isinstance(prior_claims, list):
+ continue
+ for prior_claim in prior_claims:
+ if not isinstance(prior_claim, Mapping):
+ continue
+ prior_slot = _clean(prior_claim.get("canonical_slot"))
+ prior_text = _normal_text(prior_claim.get("text"))
+ if not prior_slot or not prior_text:
+ continue
+ for evidence_id in prior_claim.get("support") or []:
+ if _clean(evidence_id):
+ prior_support_bindings.add(
+ (prior_slot, prior_text, _clean(evidence_id))
+ )
+ for evidence_id in prior_claim.get("counterevidence") or []:
+ if _clean(evidence_id):
+ prior_counter_bindings.add(
+ (prior_slot, prior_text, _clean(evidence_id))
+ )
+ support_ids: set[str] = set()
+ citation_bindings: dict[str, list[tuple[str, str, str, str]]] = {}
+ claim_roles: list[tuple[set[str], set[str]]] = []
+ for operation_index, operation in enumerate(operations):
+ if not isinstance(operation, Mapping):
+ raise PatchValidationError("claim evidence contract received malformed operation")
+ action = _clean(operation.get("action"))
+ if action == "noop":
+ continue
+ claims = operation.get("claims")
+ if not isinstance(claims, list):
+ raise PatchValidationError("claim evidence contract requires claims")
+ for claim_index, claim in enumerate(claims):
+ if not isinstance(claim, Mapping):
+ raise PatchValidationError("claim evidence contract received malformed claim")
+ claim_slot = _required_text(
+ claim.get("canonical_slot"), "claim canonical slot"
+ )
+ claim_text = _normal_text(claim.get("text"))
+ claim_support: set[str] = set()
+ claim_counter: set[str] = set()
+ normalized_support_ids = [
+ _required_text(evidence_id, "claim support evidence ID")
+ for evidence_id in claim.get("support") or []
+ ]
+ complementary_support_bundle = _controlled_complementary_support_bundle(
+ claim_slot, normalized_support_ids, evidence_by_id
+ )
+ for normalized_id in normalized_support_ids:
+ leaf = evidence_by_id.get(normalized_id)
+ if leaf is None:
+ if (claim_slot, claim_text, normalized_id) not in prior_support_bindings:
+ raise PatchValidationError(
+ f"claim support is absent from supplied evidence: {normalized_id}"
+ )
+ else:
+ evidence_slot = _leaf_slot(leaf)
+ if (
+ evidence_slot != claim_slot
+ and not complementary_support_bundle
+ and (claim_slot, claim_text, normalized_id)
+ not in prior_support_bindings
+ ):
+ raise PatchValidationError(
+ "claim support canonical slot mismatch: "
+ f"claim={claim_slot} evidence={evidence_slot} id={normalized_id}"
+ )
+ if (
+ action != "retire"
+ and
+ not _is_current_durable(leaf)
+ and (claim_slot, claim_text, normalized_id)
+ not in prior_support_bindings
+ ):
+ raise PatchValidationError(
+ "non-current Fast evidence cannot become a new claim support: "
+ + normalized_id
+ )
+ support_ids.add(normalized_id)
+ claim_support.add(normalized_id)
+ citation_bindings.setdefault(normalized_id, []).append(
+ (
+ "support",
+ claim_slot,
+ claim_text,
+ f"operations[{operation_index}].claims[{claim_index}].support",
+ )
+ )
+ support_text_groups = _support_text_groups(
+ claim_support, evidence_by_id
+ )
+ unchanged_prior_support = bool(claim_support) and all(
+ (claim_slot, claim_text, evidence_id) in prior_support_bindings
+ for evidence_id in claim_support
+ )
+ if (
+ len(support_text_groups) > 1
+ and not complementary_support_bundle
+ and not unchanged_prior_support
+ ):
+ raise PatchValidationError(
+ "distinct supplied Fast evidence texts cannot share one claim; "
+ "split these support IDs into separate claims within the same capsule "
+ "(multiple support IDs are allowed only for identical normalized evidence "
+ "text): "
+ + _json(
+ sorted(
+ evidence_id
+ for evidence_group in support_text_groups.values()
+ for evidence_id in evidence_group
+ )
+ )
+ )
+ for evidence_id in claim.get("counterevidence") or []:
+ normalized_id = _required_text(
+ evidence_id, "claim counterevidence ID"
+ )
+ leaf = evidence_by_id.get(normalized_id)
+ if leaf is None:
+ if (
+ claim_slot,
+ claim_text,
+ normalized_id,
+ ) not in prior_counter_bindings:
+ raise PatchValidationError(
+ "claim counterevidence is absent from supplied evidence: "
+ + normalized_id
+ )
+ elif _normal_text(_leaf_text(leaf)) == claim_text:
+ raise PatchValidationError(
+ "claim text cannot be identical to its counterevidence: "
+ + normalized_id
+ )
+ if (
+ route == "flash"
+ and normalized_id not in prior_counterevidence
+ and (leaf is None or not _is_counterevidence(leaf))
+ ):
+ raise PatchValidationError(
+ "Flash cannot create new counterevidence; explicit Pro escalation is "
+ "required: "
+ + normalized_id
+ )
+ claim_counter.add(normalized_id)
+ citation_bindings.setdefault(normalized_id, []).append(
+ (
+ "counterevidence",
+ claim_slot,
+ claim_text,
+ f"operations[{operation_index}].claims[{claim_index}].counterevidence",
+ )
+ )
+ if claim_support & claim_counter:
+ raise PatchValidationError(
+ "one claim cannot use the same evidence as support and counterevidence: "
+ + _json(sorted(claim_support & claim_counter))
+ )
+ stale_prior_support = {
+ evidence_id
+ for evidence_id in claim_support
+ if evidence_id in evidence_by_id
+ and not _is_current_durable(evidence_by_id[evidence_id])
+ }
+ current_replacements = {
+ evidence_id
+ for evidence_id, candidate in evidence_by_id.items()
+ if _leaf_slot(candidate) == claim_slot
+ and _is_current_durable(candidate)
+ }
+ if (
+ action != "retire"
+ and
+ stale_prior_support
+ and current_replacements
+ and not current_replacements.intersection(claim_counter)
+ ):
+ raise PatchValidationError(
+ "historical non-current support requires current replacement "
+ "counterevidence: "
+ + _json(sorted(stale_prior_support))
+ )
+ if action == "create" and claim_counter:
+ raise PatchValidationError(
+ "create cannot commit unresolved counterevidence as an active capsule: "
+ + _json(sorted(claim_counter))
+ )
+ if not claim_support and not claim_counter:
+ raise PatchValidationError(
+ f"operations[{operation_index}].claims[{claim_index}] has no evidence"
+ )
+ claim_roles.append((claim_support, claim_counter))
+ _validate_repeated_evidence_bindings(evidence_by_id, citation_bindings)
+ for left_index, (left_support, left_counter) in enumerate(claim_roles):
+ for right_support, right_counter in claim_roles[left_index + 1 :]:
+ if left_support & right_counter and right_support & left_counter:
+ raise PatchValidationError(
+ "reciprocal counterevidence claims are forbidden; emit one adjudicated "
+ "or challenged claim"
+ )
+ if route == "flash":
+ _, required_ids = _required_promotion_ids(region, capsules)
+ required_support = {
+ evidence_id
+ for evidence_id in required_ids
+ if not _is_counterevidence(evidence_by_id[evidence_id])
+ }
+ missing_support = required_support - support_ids
+ if missing_support:
+ raise PatchValidationError(
+ "Flash must represent every non-conflict durable delta as canonical-slot "
+ "support: "
+ + _json(sorted(missing_support))
+ )
+
+
+def _claim_evidence_ids(claims: Any) -> set[str]:
+ cited: set[str] = set()
+ if not isinstance(claims, list):
+ return cited
+ for claim in claims:
+ if not isinstance(claim, Mapping):
+ continue
+ for field in ("support", "counterevidence"):
+ values = claim.get(field)
+ if not isinstance(values, list):
+ continue
+ cited.update(_clean(item) for item in values if _clean(item))
+ return cited
+
+
+def _capsule_evidence_ids(capsules: list[Mapping[str, Any]]) -> set[str]:
+ cited: set[str] = set()
+ for capsule in capsules:
+ if _clean(capsule.get("status")).casefold() not in {"active", "challenged"}:
+ continue
+ cited.update(_claim_evidence_ids(capsule.get("claims")))
+ return cited
+
+
+def _patch_claim_projection(claim: Mapping[str, Any]) -> dict[str, Any]:
+ support = claim.get("support")
+ counter = claim.get("counterevidence")
+ if not isinstance(support, list) or not isinstance(counter, list):
+ raise PatchValidationError("capsule claim evidence must be arrays")
+ return {
+ "canonical_slot": _required_text(
+ claim.get("canonical_slot"), "capsule claim canonical slot"
+ ),
+ "text": " ".join(
+ _required_text(claim.get("text"), "capsule claim text").split()
+ ),
+ "support": sorted(
+ {_required_text(item, "claim support evidence ID") for item in support}
+ ),
+ "counterevidence": sorted(
+ {
+ _required_text(item, "claim counterevidence ID")
+ for item in counter
+ }
+ ),
+ }
+
+
+def _validate_flash_delta_patch(
+ patch: Mapping[str, Any],
+ capsules: list[Mapping[str, Any]],
+ required_evidence_ids: set[str],
+) -> None:
+ """Keep additive Flash operations delta-only while allowing new topics."""
+ operations = patch.get("operations")
+ if not isinstance(operations, list) or not operations:
+ raise PatchValidationError("Flash delta patch requires operations")
+ existing = {
+ _required_text(capsule.get("capsule_id"), "existing capsule_id"): capsule
+ for capsule in capsules
+ }
+ cited: set[str] = set()
+ for operation in operations:
+ if not isinstance(operation, Mapping):
+ raise PatchValidationError("Flash delta operation must be an object")
+ action = _clean(operation.get("action"))
+ if action not in ({"revise", "create"} if capsules else {"create"}):
+ raise PatchValidationError(
+ "Flash delta operations must revise an existing capsule or create a new topic"
+ )
+ if action == "revise":
+ capsule_id = _required_text(operation.get("capsule_id"), "capsule_id")
+ capsule = existing.get(capsule_id)
+ if capsule is None:
+ raise PatchValidationError("Flash delta targeted an unknown capsule")
+ if operation.get("base_revision") != capsule.get("revision"):
+ raise PatchValidationError("Flash delta base_revision changed")
+ claims = operation.get("claims")
+ if not isinstance(claims, list) or not claims:
+ raise PatchValidationError("Flash delta patch requires non-empty delta claims")
+ cited.update(_claim_evidence_ids(claims))
+ for claim in claims:
+ if not isinstance(claim, Mapping):
+ raise PatchValidationError("Flash delta claim must be an object")
+ if not _claim_evidence_ids([claim]):
+ raise PatchValidationError(
+ "every Flash delta claim must cite required delta evidence"
+ )
+ outside_delta = cited - required_evidence_ids
+ if outside_delta:
+ raise PatchValidationError(
+ "Flash delta patch cited existing or non-delta evidence: "
+ + _json(sorted(outside_delta))
+ )
+ missing = required_evidence_ids - cited
+ if missing:
+ raise PatchValidationError(
+ "Flash delta patch omitted required evidence: " + _json(sorted(missing))
+ )
+
+
+def _merge_flash_delta_patch(
+ patch: Mapping[str, Any], capsules: list[Mapping[str, Any]]
+) -> tuple[dict[str, Any], dict[str, Any]]:
+ """Build full next revisions for every Flash revise plus independent creates."""
+ capsules_by_id = {
+ _required_text(capsule.get("capsule_id"), "existing capsule_id"): capsule
+ for capsule in capsules
+ }
+ full_operations: list[dict[str, Any]] = []
+ appended_claims = 0
+ merged_claims = 0
+ prior_claim_count = 0
+ model_delta_claim_count = 0
+ result_claim_count = 0
+ for operation in patch.get("operations", []):
+ if not isinstance(operation, Mapping):
+ raise PatchValidationError("Flash delta operation must be an object")
+ action = _clean(operation.get("action"))
+ raw_delta = operation.get("claims")
+ if not isinstance(raw_delta, list):
+ raise PatchValidationError("Flash delta merge requires claim arrays")
+ model_delta_claim_count += len(raw_delta)
+ if action == "create":
+ full_operations.append(
+ {
+ "action": "create",
+ "capsule_key": _normalize_capsule_key(
+ operation.get("capsule_key")
+ ),
+ "claims": _canonical_patch_claims(raw_delta),
+ }
+ )
+ appended_claims += len(raw_delta)
+ continue
+ capsule_id = _required_text(operation.get("capsule_id"), "capsule_id")
+ capsule = capsules_by_id.get(capsule_id)
+ if capsule is None:
+ raise PatchValidationError("Flash delta merge targeted an unknown capsule")
+ expected_revision = capsule.get("revision")
+ if operation.get("base_revision") != expected_revision:
+ raise PatchValidationError("Flash delta base_revision changed")
+ raw_prior = capsule.get("claims")
+ if not isinstance(raw_prior, list):
+ raise PatchValidationError("existing capsule claim array is missing")
+ prior_claim_count += len(raw_prior)
+ merged: list[dict[str, Any]] = []
+ index_by_identity: dict[tuple[str, str], int] = {}
+ for source_claims, is_delta in ((raw_prior, False), (raw_delta, True)):
+ for raw_claim in source_claims:
+ if not isinstance(raw_claim, Mapping):
+ raise PatchValidationError("Flash delta claim must be an object")
+ claim = _patch_claim_projection(raw_claim)
+ identity = (claim["canonical_slot"], _normal_text(claim["text"]))
+ if identity in index_by_identity:
+ target = merged[index_by_identity[identity]]
+ target["support"] = sorted(set(target["support"] + claim["support"]))
+ target["counterevidence"] = sorted(
+ set(target["counterevidence"] + claim["counterevidence"])
+ )
+ if is_delta:
+ merged_claims += 1
+ continue
+ index_by_identity[identity] = len(merged)
+ merged.append(claim)
+ if is_delta:
+ appended_claims += 1
+ result_claim_count += len(merged)
+ full_operations.append(
+ {
+ "action": "revise",
+ "capsule_id": capsule_id,
+ "base_revision": expected_revision,
+ "claims": merged,
+ }
+ )
+ full_patch = _materialize_lossless_summaries({"operations": full_operations})
+ return full_patch, {
+ "schema_version": "tmcra.v4.slow-flash-delta-merge.2",
+ "operation_count": len(full_operations),
+ "prior_claim_count": prior_claim_count,
+ "model_delta_claim_count": model_delta_claim_count,
+ "appended_claim_count": appended_claims,
+ "merged_claim_count": merged_claims,
+ "result_claim_count": result_claim_count,
+ "model_delta_patch_sha256": _digest(patch),
+ "committed_patch_sha256": _digest(full_patch),
+ }
+
+
+def _sanitize_capsules_for_current_support(
+ capsules: list[Mapping[str, Any]],
+ current_support_ids: set[str],
+ challenged_support_ids: set[str],
+ known_evidence_ids: set[str],
+) -> tuple[list[dict[str, Any]], dict[str, Any]]:
+ """Remove claims whose positive support is no longer current Fast evidence."""
+ sanitized: list[dict[str, Any]] = []
+ removed_support_ids: set[str] = set()
+ removed_claim_count = 0
+ changed_claim_count = 0
+ for capsule in capsules:
+ if _clean(capsule.get("status")).casefold() not in {"active", "challenged"}:
+ sanitized.append(dict(capsule))
+ continue
+ raw_claims = capsule.get("claims")
+ if not isinstance(raw_claims, list):
+ raise EvidencePolicyError("capsule claims are not auditable")
+ next_claims: list[dict[str, Any]] = []
+ for raw_claim in raw_claims:
+ if not isinstance(raw_claim, Mapping):
+ raise EvidencePolicyError("capsule claim is not an object")
+ claim = dict(raw_claim)
+ support = claim.get("support")
+ if not isinstance(support, list):
+ raise EvidencePolicyError("capsule claim support is not a list")
+ normalized_support = [
+ _required_text(item, "capsule claim support evidence ID")
+ for item in support
+ ]
+ counterevidence = claim.get("counterevidence")
+ if not isinstance(counterevidence, list):
+ raise EvidencePolicyError(
+ "capsule claim counterevidence is not a list"
+ )
+ has_current_counterevidence = bool(
+ current_support_ids.intersection(
+ _required_text(item, "capsule claim counterevidence ID")
+ for item in counterevidence
+ )
+ )
+ preserve_adjudicated_history = (
+ _clean(capsule.get("status")).casefold() == "challenged"
+ and has_current_counterevidence
+ )
+ current_support = [
+ item
+ for item in normalized_support
+ if item not in known_evidence_ids
+ or item in current_support_ids
+ or (
+ preserve_adjudicated_history
+ and item in challenged_support_ids
+ )
+ ]
+ removed_support_ids.update(set(normalized_support) - set(current_support))
+ if not current_support:
+ removed_claim_count += 1
+ continue
+ if current_support != normalized_support:
+ changed_claim_count += 1
+ claim["support"] = current_support
+ next_claims.append(claim)
+ next_capsule = dict(capsule)
+ next_capsule["claims"] = next_claims
+ sanitized.append(next_capsule)
+ return sanitized, {
+ "schema_version": "tmcra.v4.slow-current-support-cleanup.1",
+ "changed": bool(removed_support_ids or removed_claim_count),
+ "removed_support_ids": sorted(removed_support_ids),
+ "removed_claim_count": removed_claim_count,
+ "changed_claim_count": changed_claim_count,
+ "remaining_claim_count": sum(
+ len(capsule.get("claims") or []) for capsule in sanitized
+ ),
+ }
+
+
+def _deterministic_support_cleanup_patch(
+ original_capsules: list[Mapping[str, Any]],
+ sanitized_capsules: list[Mapping[str, Any]],
+) -> dict[str, Any]:
+ original_by_id = {
+ _required_text(capsule.get("capsule_id"), "capsule_id"): capsule
+ for capsule in original_capsules
+ }
+ sanitized_by_id = {
+ _required_text(capsule.get("capsule_id"), "capsule_id"): capsule
+ for capsule in sanitized_capsules
+ }
+ if set(original_by_id) != set(sanitized_by_id):
+ raise PatchValidationError("support cleanup changed capsule identity")
+ operations: list[dict[str, Any]] = []
+ for capsule_id in sorted(original_by_id):
+ original = original_by_id[capsule_id]
+ sanitized = sanitized_by_id[capsule_id]
+ revision = original.get("revision")
+ if isinstance(revision, bool) or not isinstance(revision, int) or revision < 1:
+ raise PatchValidationError("capsule revision must be positive")
+ original_claims = _canonical_patch_claims(original.get("claims"))
+ raw_sanitized_claims = sanitized.get("claims")
+ if not isinstance(raw_sanitized_claims, list):
+ raise PatchValidationError("sanitized capsule claims are missing")
+ sanitized_claims = (
+ _canonical_patch_claims(raw_sanitized_claims)
+ if raw_sanitized_claims
+ else []
+ )
+ if sanitized_claims == original_claims:
+ continue
+ operations.append(
+ {
+ "action": "revise" if sanitized_claims else "retire",
+ "capsule_id": capsule_id,
+ "base_revision": revision,
+ "claims": sanitized_claims or original_claims,
+ }
+ )
+ if not operations:
+ raise PatchValidationError("support cleanup produced no changed capsule")
+ return _materialize_lossless_summaries({"operations": operations})
+
+
+def _deterministic_summary_migration_patch(
+ capsules: list[Mapping[str, Any]],
+) -> dict[str, Any]:
+ operations: list[dict[str, Any]] = []
+ for capsule in sorted(capsules, key=lambda item: _clean(item.get("capsule_id"))):
+ if _clean(capsule.get("status")).casefold() not in {"active", "challenged"}:
+ continue
+ if not _capsule_requires_summary_migration([capsule]):
+ continue
+ capsule_id = _required_text(capsule.get("capsule_id"), "capsule_id")
+ revision = capsule.get("revision")
+ if isinstance(revision, bool) or not isinstance(revision, int) or revision < 1:
+ raise PatchValidationError("capsule revision must be positive")
+ operations.append(
+ {
+ "action": "revise",
+ "capsule_id": capsule_id,
+ "base_revision": revision,
+ "claims": _canonical_patch_claims(capsule.get("claims")),
+ }
+ )
+ if not operations:
+ raise PatchValidationError("summary migration found no invalid capsule")
+ patch = _materialize_lossless_summaries({"operations": operations})
+ _validate_patch_summary_contract(patch)
+ return patch
+
+
+def _deterministic_contract_migration_patch(
+ capsules: list[Mapping[str, Any]], partition_targets: set[str]
+) -> dict[str, Any]:
+ """Stamp unambiguous single-claim partitions and repair summaries together."""
+ operations: list[dict[str, Any]] = []
+ operated_targets: set[str] = set()
+ for capsule in sorted(capsules, key=lambda item: _clean(item.get("capsule_id"))):
+ if _clean(capsule.get("status")).casefold() not in {"active", "challenged"}:
+ continue
+ capsule_id = _required_text(capsule.get("capsule_id"), "capsule_id")
+ targeted = capsule_id in partition_targets
+ if not targeted and not _capsule_requires_summary_migration([capsule]):
+ continue
+ claims = _canonical_patch_claims(capsule.get("claims"))
+ if targeted and len(claims) != 1:
+ raise PatchValidationError(
+ "deterministic partition migration requires exactly one claim per target"
+ )
+ revision = capsule.get("revision")
+ if isinstance(revision, bool) or not isinstance(revision, int) or revision < 1:
+ raise PatchValidationError("capsule revision must be positive")
+ operations.append(
+ {
+ "action": "revise",
+ "capsule_id": capsule_id,
+ "base_revision": revision,
+ "claims": claims,
+ }
+ )
+ if targeted:
+ operated_targets.add(capsule_id)
+ if operated_targets != partition_targets:
+ raise PatchValidationError(
+ "deterministic partition migration target set is incomplete"
+ )
+ if not operations:
+ raise PatchValidationError("contract migration found no capsules to revise")
+ patch = _materialize_lossless_summaries({"operations": operations})
+ _validate_patch_summary_contract(patch)
+ return patch
+
+
+def _capsule_requires_summary_migration(capsules: list[Mapping[str, Any]]) -> bool:
+ for capsule in capsules:
+ if _clean(capsule.get("status")).casefold() not in {"active", "challenged"}:
+ continue
+ try:
+ claims = _canonical_patch_claims(capsule.get("claims"))
+ summary = validate_semantic_summary(
+ capsule.get("value"),
+ claims,
+ label="stored Slow capsule summary",
+ )
+ if (
+ summary != _semantic_summary_projection(claims)
+ or capsule.get("summary_contract_version")
+ != SLOW_SUMMARY_CONTRACT_VERSION
+ ):
+ return True
+ except PatchValidationError:
+ return True
+ return False
+
+
+def _required_promotion_ids(
+ region: Mapping[str, Any], capsules: list[Mapping[str, Any]]
+) -> tuple[set[str], set[str]]:
+ evidence = [
+ item for item in region.get("evidence", []) if isinstance(item, Mapping)
+ ]
+ eligible = {_leaf_id(item) for item in evidence if _is_current_durable(item)}
+ return eligible, eligible - _capsule_evidence_ids(capsules)
+
+
+def _next_active_capsule_claims(
+ capsules: list[Mapping[str, Any]], patch: Mapping[str, Any]
+) -> dict[str, list[dict[str, Any]]]:
+ """Project the full active Slow state after one atomic region patch."""
+ next_claims: dict[str, list[dict[str, Any]]] = {}
+ known_capsules: set[str] = set()
+ for capsule in capsules:
+ capsule_id = _required_text(capsule.get("capsule_id"), "capsule_id")
+ if capsule_id in known_capsules:
+ raise PatchValidationError(
+ f"duplicate supplied capsule identity: {capsule_id}"
+ )
+ known_capsules.add(capsule_id)
+ if _clean(capsule.get("status")).casefold() in {"active", "challenged"}:
+ raw_claims = capsule.get("claims")
+ if not isinstance(raw_claims, list):
+ raise PatchValidationError("supplied capsule claims must be a list")
+ next_claims[capsule_id] = (
+ _canonical_patch_claims(raw_claims) if raw_claims else []
+ )
+
+ for operation in patch.get("operations", []):
+ if not isinstance(operation, Mapping):
+ raise PatchValidationError("promotion patch operation must be an object")
+ action = _clean(operation.get("action"))
+ if action == "noop":
+ continue
+ if action == "create":
+ capsule_key = _normalize_capsule_key(operation.get("capsule_key"))
+ target = "create:" + capsule_key
+ if target in next_claims:
+ raise PatchValidationError(
+ f"duplicate resulting capsule target: {capsule_key}"
+ )
+ next_claims[target] = _canonical_patch_claims(operation.get("claims"))
+ continue
+ capsule_id = _required_text(operation.get("capsule_id"), "capsule_id")
+ if capsule_id not in known_capsules:
+ raise PatchValidationError(
+ f"GraphPatch targeted an unknown supplied capsule: {capsule_id}"
+ )
+ if action == "retire":
+ next_claims.pop(capsule_id, None)
+ else:
+ next_claims[capsule_id] = _canonical_patch_claims(
+ operation.get("claims")
+ )
+ empty_active = sorted(
+ capsule_id for capsule_id, claims in next_claims.items() if not claims
+ )
+ if empty_active:
+ raise PatchValidationError(
+ "active Slow capsules cannot have zero claims: " + _json(empty_active)
+ )
+ return next_claims
+
+
+def _validate_promotion_patch(
+ region: Mapping[str, Any],
+ capsules: list[Mapping[str, Any]],
+ patch: Mapping[str, Any],
+ *,
+ required_evidence_ids: set[str] | None = None,
+) -> None:
+ """Require the committed next Slow revision to cover every current durable leaf."""
+ _validate_claim_evidence_contract(region, capsules, patch)
+ eligible_ids, required_ids = _required_promotion_ids(region, capsules)
+ operations = patch.get("operations")
+ if not isinstance(operations, list) or not operations:
+ raise PatchValidationError("promotion patch must contain operations")
+ if (
+ required_ids
+ and len(operations) == 1
+ and isinstance(operations[0], Mapping)
+ and _clean(operations[0].get("action")) == "noop"
+ ):
+ raise PatchValidationError(
+ "noop cannot consume uncited current durable Fast evidence: "
+ + _json(sorted(required_ids))
+ )
+
+ next_capsules = _next_active_capsule_claims(capsules, patch)
+ citation_locations: dict[str, list[str]] = {}
+ citation_bindings: dict[str, list[tuple[str, str, str, str]]] = {}
+ claim_identity_capsules: dict[tuple[str, str], set[str]] = {}
+ for capsule_id, claims in next_capsules.items():
+ for claim_index, claim in enumerate(claims):
+ slot = claim["canonical_slot"]
+ claim_text = _normal_text(claim["text"])
+ claim_identity = (slot, claim_text)
+ claim_identity_capsules.setdefault(claim_identity, set()).add(capsule_id)
+ for role in ("support", "counterevidence"):
+ for evidence_id in claim[role]:
+ location = f"{capsule_id}:{claim_index}:{role}"
+ citation_locations.setdefault(evidence_id, []).append(location)
+ citation_bindings.setdefault(evidence_id, []).append(
+ (role, slot, claim_text, location)
+ )
+
+ evidence_by_id = {
+ _leaf_id(item): item
+ for item in region.get("evidence", [])
+ if isinstance(item, Mapping)
+ }
+ _validate_repeated_evidence_bindings(evidence_by_id, citation_bindings)
+ duplicated_claim_identities = {
+ f"{slot}\u241f{text}": sorted(capsule_ids)
+ for (slot, text), capsule_ids in claim_identity_capsules.items()
+ if len(capsule_ids) > 1
+ }
+ if duplicated_claim_identities:
+ raise PatchValidationError(
+ "one self-contained semantic claim cannot span multiple active Slow capsules: "
+ + _json(duplicated_claim_identities)
+ )
+
+ cited_ids = set(citation_locations)
+ missing = eligible_ids - cited_ids
+ if missing:
+ raise PatchValidationError(
+ "next Slow revision omits current durable Fast evidence: "
+ + _json(sorted(missing))
+ )
+ required = set(required_evidence_ids or ())
+ missing_required = required - cited_ids
+ if missing_required:
+ raise PatchValidationError(
+ "next Slow revision omits required Fast evidence: "
+ + _json(sorted(missing_required))
+ )
+
+
+def _public_leaf(leaf: Mapping[str, Any]) -> dict[str, Any]:
+ metadata = _leaf_metadata(leaf)
+ result = {
+ "memory_id": _leaf_id(leaf),
+ "value": _leaf_text(leaf),
+ "turn_index": leaf.get("turn_index"),
+ "record_state": leaf.get("record_state"),
+ "canonical_slot": _leaf_slot(leaf),
+ "durability": metadata.get("durability"),
+ "temporal_status": metadata.get("temporal_status") or metadata.get("target_status"),
+ "polarity": metadata.get("polarity"),
+ "write_operation": metadata.get("write_operation"),
+ "evidence_role": (
+ "counterevidence" if _is_counterevidence(leaf) else "support"
+ ),
+ }
+ _assert_no_benchmark_fields(result)
+ return result
+
+
+def _public_capsule(capsule: Mapping[str, Any]) -> dict[str, Any]:
+ claims = capsule.get("claims")
+ if not isinstance(claims, list):
+ raise EvidencePolicyError("Slow capsule claims are not public-request ready")
+ result = {
+ "capsule_id": _required_text(capsule.get("capsule_id"), "capsule_id"),
+ "revision": capsule.get("revision"),
+ "status": _required_text(capsule.get("status") or "active", "capsule status"),
+ "summary": _clean(capsule.get("value")),
+ "claims": _canonical_patch_claims(claims) if claims else [],
+ }
+ capsule_key = _clean(capsule.get("capsule_key"))
+ if capsule_key:
+ result["capsule_key"] = _normalize_capsule_key(capsule_key)
+ if capsule.get("partition_contract_version"):
+ result["partition_contract_version"] = capsule.get(
+ "partition_contract_version"
+ )
+ _assert_no_benchmark_fields(result)
+ return result
+
+
+class V4SlowGraphStore(_v3.SlowGraphStore):
+ """V3 ledger with atomic multi-capsule region commits."""
+
+ def _init_schema(self) -> None:
+ super()._init_schema()
+ with self.connection() as con:
+ con.execute(
+ "CREATE UNIQUE INDEX IF NOT EXISTS "
+ "idx_v4_slow_patch_one_per_job ON slow_graph_patches(job_id)"
+ )
+ con.execute(
+ "CREATE UNIQUE INDEX IF NOT EXISTS "
+ "idx_v4_slow_patch_operation_ordinal "
+ "ON slow_graph_patch_operations(patch_id,ordinal)"
+ )
+ con.execute(
+ "CREATE UNIQUE INDEX IF NOT EXISTS idx_v4_slow_provenance_fact "
+ "ON slow_graph_provenance("
+ "patch_id,scope_id,capsule_id,revision,evidence_memory_id,claim_id,polarity)"
+ )
+ con.execute(
+ """
+ CREATE TABLE IF NOT EXISTS slow_graph_process_loss_recoveries(
+ recovery_id TEXT PRIMARY KEY,
+ job_id TEXT NOT NULL,
+ attempt_id TEXT NOT NULL UNIQUE,
+ scope_id TEXT NOT NULL,
+ recovery_source TEXT NOT NULL,
+ claim_token TEXT NOT NULL,
+ claim_owner TEXT NOT NULL,
+ lease_expires_at INTEGER,
+ attempt_created_at INTEGER NOT NULL,
+ job_attempts_before INTEGER NOT NULL,
+ attempt_metadata_sha256 TEXT NOT NULL,
+ interruption_error_sha256 TEXT NOT NULL,
+ external_call_outcome TEXT NOT NULL,
+ potential_duplicate_physical_calls_min INTEGER NOT NULL,
+ potential_duplicate_physical_calls_max INTEGER NOT NULL,
+ recovered_at INTEGER NOT NULL
+ )
+ """
+ )
+ con.execute(
+ "CREATE INDEX IF NOT EXISTS idx_slow_process_loss_job "
+ "ON slow_graph_process_loss_recoveries(job_id,recovered_at)"
+ )
+
+ @staticmethod
+ def _claim_owner_pid(owner: Any) -> int:
+ parts = _clean(owner).split(":", 2)
+ if len(parts) != 3 or parts[0] != "pid" or not parts[1].isdigit():
+ raise SlowGraphError("interrupted Slow claim owner is invalid")
+ pid = int(parts[1])
+ if pid <= 0:
+ raise SlowGraphError("interrupted Slow claim owner PID is invalid")
+ return pid
+
+ @staticmethod
+ def _pid_is_alive(pid: int) -> bool:
+ try:
+ os.kill(pid, 0)
+ except ProcessLookupError:
+ return False
+ except PermissionError:
+ return True
+ except OSError as exc:
+ if exc.errno in {errno.ESRCH, errno.EINVAL}:
+ return False
+ if exc.errno == errno.EPERM:
+ return True
+ raise
+ return True
+
+ def interrupted_process_loss_attempts(self) -> list[dict[str, Any]]:
+ """List unjournaled attempts whose external outcome needs explicit review."""
+ with self.connection() as con:
+ rows = con.execute(
+ "SELECT j.job_id,j.scope_id,j.region_key,j.status,j.attempts,"
+ "j.last_error,j.claim_token AS job_claim_token,"
+ "j.claim_owner AS job_claim_owner,j.lease_expires_at,"
+ "a.attempt_id,a.status AS attempt_status,a.call_metadata_json,"
+ "a.error AS attempt_error,a.created_at AS attempt_created_at,"
+ "a.completed_at,a.claim_token AS attempt_claim_token,"
+ "a.claim_owner AS attempt_claim_owner "
+ "FROM slow_graph_jobs j JOIN slow_graph_attempts a "
+ "ON a.job_id=j.job_id "
+ "LEFT JOIN slow_graph_process_loss_recoveries r "
+ "ON r.attempt_id=a.attempt_id "
+ "WHERE r.attempt_id IS NULL AND ("
+ "(j.status='pending' AND j.claim_token IS NOT NULL "
+ "AND a.status='started' AND a.claim_token=j.claim_token "
+ "AND a.claim_owner=j.claim_owner) OR "
+ "(j.status='failed' AND j.claim_token IS NULL "
+ "AND j.last_error=? AND a.status='expired' AND a.error=?)) "
+ "ORDER BY a.created_at,a.attempt_id",
+ (PROCESS_LOSS_INTERRUPTION_ERROR, PROCESS_LOSS_INTERRUPTION_ERROR),
+ ).fetchall()
+ return [dict(row) for row in rows]
+
+ def recover_interrupted_process_loss(
+ self, job_id: str, *, expected_attempt_id: str | None = None
+ ) -> dict[str, Any]:
+ """Atomically journal and reopen one reviewed process-loss attempt."""
+ now = _v3._now()
+ with self.connection() as con:
+ con.execute("BEGIN IMMEDIATE")
+ job = con.execute(
+ "SELECT * FROM slow_graph_jobs WHERE job_id=?", (job_id,)
+ ).fetchone()
+ if job is None:
+ raise SlowGraphError("unknown interrupted Slow job")
+ patch_count = int(
+ con.execute(
+ "SELECT count(*) FROM slow_graph_patches WHERE job_id=?",
+ (job_id,),
+ ).fetchone()[0]
+ )
+ if patch_count:
+ raise SlowGraphError("interrupted Slow job already has a patch")
+
+ recovery_source: str
+ lease_expires_at: int | None
+ if job["status"] == "pending" and job["claim_token"] is not None:
+ lease = job["lease_expires_at"]
+ if lease is None or int(lease) >= now:
+ raise SlowGraphError("interrupted Slow claim has not expired")
+ attempts = con.execute(
+ "SELECT * FROM slow_graph_attempts WHERE job_id=? "
+ "AND status='started' AND claim_token=? AND claim_owner=?",
+ (job_id, job["claim_token"], job["claim_owner"]),
+ ).fetchall()
+ recovery_source = "expired_claimed_started"
+ lease_expires_at = int(lease)
+ job_attempts_before = int(job["attempts"] or 0)
+ elif (
+ job["status"] == "failed"
+ and job["claim_token"] is None
+ and _clean(job["last_error"]) == PROCESS_LOSS_INTERRUPTION_ERROR
+ ):
+ attempts = con.execute(
+ "SELECT a.* FROM slow_graph_attempts a "
+ "LEFT JOIN slow_graph_process_loss_recoveries r "
+ "ON r.attempt_id=a.attempt_id "
+ "WHERE a.job_id=? AND a.status='expired' AND a.error=? "
+ "AND r.attempt_id IS NULL ORDER BY a.created_at,a.attempt_id",
+ (job_id, PROCESS_LOSS_INTERRUPTION_ERROR),
+ ).fetchall()
+ recovery_source = "legacy_expired_failed"
+ lease_expires_at = None
+ job_attempts_before = int(job["attempts"] or 0) - 1
+ else:
+ raise SlowGraphError(
+ "process-loss recovery requires one claimed started or legacy expired job"
+ )
+ if len(attempts) != 1:
+ raise SlowGraphError(
+ "interrupted Slow job does not have exactly one recoverable attempt"
+ )
+ attempt = attempts[0]
+ if (
+ expected_attempt_id is not None
+ and str(attempt["attempt_id"]) != expected_attempt_id
+ ):
+ raise SlowGraphError("interrupted Slow attempt changed after review")
+ raw_metadata = _required_text(
+ attempt["call_metadata_json"], "interrupted call metadata"
+ )
+ metadata = _v3._strict_json(
+ raw_metadata, label="interrupted call metadata", expected=dict
+ )
+ if metadata or (
+ recovery_source == "expired_claimed_started"
+ and (
+ _clean(attempt["error"])
+ or attempt["completed_at"] is not None
+ )
+ ):
+ raise SlowGraphError(
+ "interrupted Slow attempt already contains a durable outcome"
+ )
+ if (
+ recovery_source == "legacy_expired_failed"
+ and (
+ _clean(attempt["error"]) != PROCESS_LOSS_INTERRUPTION_ERROR
+ or attempt["completed_at"] is None
+ )
+ ):
+ raise SlowGraphError("legacy expired Slow attempt is inconsistent")
+ claim_token = _required_text(
+ attempt["claim_token"], "interrupted claim token"
+ )
+ claim_owner = _required_text(
+ attempt["claim_owner"], "interrupted claim owner"
+ )
+ owner_pid = self._claim_owner_pid(claim_owner)
+ if self._pid_is_alive(owner_pid):
+ raise SlowGraphError(
+ f"interrupted Slow claim owner is still alive: {owner_pid}"
+ )
+ metadata_sha256 = hashlib.sha256(
+ raw_metadata.encode("utf-8")
+ ).hexdigest()
+ error_sha256 = hashlib.sha256(
+ PROCESS_LOSS_INTERRUPTION_ERROR.encode("utf-8")
+ ).hexdigest()
+ recovery_id = "sgr_" + _digest(
+ {
+ "job_id": job_id,
+ "attempt_id": attempt["attempt_id"],
+ "claim_token": claim_token,
+ "claim_owner": claim_owner,
+ "attempt_metadata_sha256": metadata_sha256,
+ }
+ )[:32]
+ con.execute(
+ "INSERT INTO slow_graph_process_loss_recoveries VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
+ (
+ recovery_id,
+ job_id,
+ attempt["attempt_id"],
+ job["scope_id"],
+ recovery_source,
+ claim_token,
+ claim_owner,
+ lease_expires_at,
+ int(attempt["created_at"]),
+ job_attempts_before,
+ metadata_sha256,
+ error_sha256,
+ "uncertain",
+ 0,
+ SLOW_PROCESS_LOSS_PHYSICAL_CALLS_MAX,
+ now,
+ ),
+ )
+ if recovery_source == "expired_claimed_started":
+ expired = con.execute(
+ "UPDATE slow_graph_attempts SET status='expired',error=?,"
+ "completed_at=? WHERE attempt_id=? AND job_id=? "
+ "AND status='started' AND claim_token=? AND claim_owner=?",
+ (
+ PROCESS_LOSS_INTERRUPTION_ERROR,
+ now,
+ attempt["attempt_id"],
+ job_id,
+ claim_token,
+ claim_owner,
+ ),
+ )
+ if expired.rowcount != 1:
+ raise SlowGraphError(
+ "interrupted Slow attempt changed during recovery"
+ )
+ reopened = con.execute(
+ "UPDATE slow_graph_jobs SET status='pending',attempts=attempts+1,"
+ "last_error='',updated_at=?,claim_token=NULL,claim_owner=NULL,"
+ "lease_expires_at=NULL WHERE job_id=? AND status='pending' "
+ "AND claim_token=? AND claim_owner=? AND lease_expires_at int:
+ """Fail closed so process loss cannot be silently converted into retry state."""
+ now = _v3._now()
+ with self.connection() as con:
+ rows = con.execute(
+ "SELECT job_id FROM slow_graph_jobs WHERE status='pending' "
+ "AND claim_token IS NOT NULL AND lease_expires_at str:
+ if capsule_key is None:
+ return super()._capsule_id(scope_id, region_key)
+ return "cap_" + _digest(
+ {
+ "scope_id": _required_text(scope_id, "scope_id"),
+ "region_key": _required_text(region_key, "region_key"),
+ "capsule_key": _normalize_capsule_key(capsule_key),
+ "partition_contract_version": SLOW_PARTITION_CONTRACT_VERSION,
+ }
+ )[:24]
+
+ def _capsules(
+ self, con: Any, scope_id: str, region_key: str
+ ) -> list[dict[str, Any]]:
+ rows = con.execute(
+ "SELECT memory_id,state,value,metadata_json FROM records WHERE scope_id=?",
+ (scope_id,),
+ ).fetchall()
+ by_capsule: dict[str, list[dict[str, Any]]] = {}
+ for row in rows:
+ metadata = self._metadata(row, "capsule metadata")
+ if (
+ metadata.get("content_variant") != CAPSULE_VARIANT
+ or metadata.get("region_key") != region_key
+ ):
+ continue
+ capsule_id = _required_text(metadata.get("capsule_id"), "capsule_id")
+ revision = metadata.get("revision")
+ if (
+ isinstance(revision, bool)
+ or not isinstance(revision, int)
+ or revision < 1
+ ):
+ raise AuditError("capsule revision is invalid")
+ by_capsule.setdefault(capsule_id, []).append(
+ {
+ "memory_id": row["memory_id"],
+ "record_state": row["state"],
+ "value": row["value"],
+ **metadata,
+ }
+ )
+ latest: list[dict[str, Any]] = []
+ for capsule_id, revisions in sorted(by_capsule.items()):
+ latest_revision = max(int(item["revision"]) for item in revisions)
+ candidates = [
+ item for item in revisions if int(item["revision"]) == latest_revision
+ ]
+ if len(candidates) != 1:
+ raise AuditError(
+ f"capsule {capsule_id} lacks one latest revision"
+ )
+ latest.append(candidates[0])
+ return latest
+
+ def _job_metadata(
+ self,
+ con: Any,
+ scope_id: str,
+ region_key: str,
+ evidence_ids: list[str],
+ manager: Any | None,
+ ) -> dict[str, Any]:
+ metadata = dict(
+ super()._job_metadata(
+ con, scope_id, region_key, evidence_ids, manager
+ )
+ )
+ # V3 intentionally excludes capsule_revision_hash from idempotency. V4.7
+ # needs a changed Slow head to produce a fresh job after a stale claim.
+ metadata["capsule_state_idempotency_hash"] = metadata[
+ "capsule_revision_hash"
+ ]
+ return metadata
+
+ def _assert_job_snapshot(
+ self, con: Any, job: Any
+ ) -> tuple[dict[str, Any], list[dict[str, Any]]]:
+ metadata = self._metadata(job, "job")
+ evidence_ids = _v3._strict_json(
+ job["evidence_ids_json"], label="job evidence IDs", expected=list
+ )
+ region = {
+ "region_key": job["region_key"],
+ "evidence": self._evidence(con, job["scope_id"], evidence_ids),
+ }
+ capsules = self._capsules(con, job["scope_id"], job["region_key"])
+ if metadata.get("evidence_content_hash") != _digest(region["evidence"]):
+ raise StaleRevisionError(
+ "Fast evidence changed after the Slow job was enqueued"
+ )
+ if metadata.get("capsule_revision_hash") != _digest(capsules):
+ raise StaleRevisionError(
+ "Slow capsules changed after the Slow job was enqueued"
+ )
+ return region, capsules
+
+ def _claim_context(
+ self, claim: JobClaim
+ ) -> tuple[dict[str, Any], list[dict[str, Any]]]:
+ with self.connection() as con:
+ job = con.execute(
+ "SELECT * FROM slow_graph_jobs WHERE job_id=?", (claim.job_id,)
+ ).fetchone()
+ if job is None:
+ raise SlowGraphError("unknown slow graph job")
+ if (
+ job["status"] != "pending"
+ or job["claim_token"] != claim.token
+ or job["claim_owner"] != claim.owner
+ or job["lease_expires_at"] is None
+ or int(job["lease_expires_at"]) < _v3._now()
+ ):
+ raise SlowGraphError("job claim is no longer active")
+ return self._assert_job_snapshot(con, job)
+
+ def _annotate_revision(
+ self,
+ con: Any,
+ *,
+ scope_id: str,
+ record_id: str,
+ operation: Mapping[str, Any],
+ old_memory_id: str | None,
+ call_metadata: Mapping[str, Any],
+ ) -> None:
+ row = con.execute(
+ "SELECT value,metadata_json FROM records "
+ "WHERE scope_id=? AND memory_id=?",
+ (scope_id, record_id),
+ ).fetchone()
+ if row is None:
+ raise SlowGraphError("new Slow revision is missing before annotation")
+ metadata = self._metadata(row, "new Slow revision")
+ claims = _canonical_patch_claims(metadata.get("claims"))
+ expected_summary = _semantic_summary_projection(claims)
+ if row["value"] != expected_summary:
+ raise AuditError("stored Slow summary differs from final claims")
+ old_metadata: dict[str, Any] = {}
+ if old_memory_id:
+ old = con.execute(
+ "SELECT metadata_json FROM records "
+ "WHERE scope_id=? AND memory_id=?",
+ (scope_id, old_memory_id),
+ ).fetchone()
+ if old is None:
+ raise SlowGraphError("prior Slow revision disappeared during commit")
+ old_metadata = self._metadata(old, "prior Slow revision")
+ capsule_key = _clean(operation.get("capsule_key")) or _clean(
+ old_metadata.get("capsule_key")
+ )
+ if not capsule_key:
+ capsule_key = "legacy"
+ metadata.update(
+ {
+ "capsule_key": _normalize_capsule_key(capsule_key),
+ "summary_contract_version": SLOW_SUMMARY_CONTRACT_VERSION,
+ "evidence_binding_contract_version": (
+ SLOW_EVIDENCE_BINDING_CONTRACT_VERSION
+ ),
+ "summary_projection_sha256": hashlib.sha256(
+ expected_summary.encode("utf-8")
+ ).hexdigest(),
+ }
+ )
+ partition_targets = set(
+ call_metadata.get("semantic_partition_capsule_ids") or ()
+ )
+ if (
+ operation.get("action") == "create"
+ or old_metadata.get("partition_contract_version")
+ == SLOW_PARTITION_CONTRACT_VERSION
+ or metadata.get("capsule_id") in partition_targets
+ ):
+ metadata["partition_contract_version"] = (
+ SLOW_PARTITION_CONTRACT_VERSION
+ )
+ con.execute(
+ "UPDATE records SET metadata_json=? WHERE scope_id=? AND memory_id=?",
+ (_json(metadata), scope_id, record_id),
+ )
+
+ def apply_patch(
+ self,
+ job_id: str,
+ patch: Mapping[str, Any],
+ *,
+ manager_model: str,
+ call_metadata: Mapping[str, Any] | None = None,
+ claim: JobClaim,
+ ) -> str:
+ validate_patch(patch, require_lossless_summary=True)
+ if claim.job_id != job_id:
+ raise SlowGraphError("claim does not belong to job")
+ patch_id = "sgp_" + uuid.uuid4().hex
+ metadata_for_call = dict(call_metadata or {})
+ metadata_for_call.setdefault(
+ "evidence_binding_contract_version",
+ SLOW_EVIDENCE_BINDING_CONTRACT_VERSION,
+ )
+ with self.connection() as con:
+ con.execute("BEGIN IMMEDIATE")
+ now = _v3._now()
+ job = con.execute(
+ "SELECT * FROM slow_graph_jobs WHERE job_id=?", (job_id,)
+ ).fetchone()
+ if job is None:
+ raise SlowGraphError("unknown slow graph job")
+ if (
+ job["status"] != "pending"
+ or job["claim_token"] != claim.token
+ or job["claim_owner"] != claim.owner
+ or job["lease_expires_at"] is None
+ or int(job["lease_expires_at"]) < now
+ ):
+ raise SlowGraphError("job claim is no longer active")
+ self._records_table_exists(con)
+ job_metadata = self._metadata(job, "job")
+ if job_metadata["schema_version"] != SCHEMA_VERSION:
+ raise SlowGraphError("job schema metadata drift")
+ _, current_capsules = self._assert_job_snapshot(con, job)
+ capsules_by_id = {
+ _required_text(item.get("capsule_id"), "capsule_id"): item
+ for item in current_capsules
+ }
+ if con.execute(
+ "SELECT 1 FROM slow_graph_patches WHERE job_id=?", (job_id,)
+ ).fetchone() is not None:
+ raise SlowGraphError("slow graph job already has a committed patch")
+
+ planned: list[dict[str, Any]] = []
+ planned_targets: set[str] = set()
+ for operation in patch["operations"]:
+ action = operation["action"]
+ if action == "create":
+ capsule_id = self._capsule_id(
+ job["scope_id"],
+ job["region_key"],
+ _normalize_capsule_key(operation.get("capsule_key")),
+ )
+ else:
+ capsule_id = _clean(operation.get("capsule_id"))
+ if not capsule_id:
+ capsule_id = "region:" + job["region_key"]
+ if capsule_id in planned_targets:
+ raise PatchValidationError(
+ f"duplicate resolved capsule target: {capsule_id}"
+ )
+ planned_targets.add(capsule_id)
+ head = (
+ None
+ if capsule_id.startswith("region:")
+ else self._head(con, job["scope_id"], capsule_id)
+ )
+ if action == "create":
+ if head is not None:
+ raise StaleRevisionError("capsule already exists")
+ elif action == "noop":
+ if operation.get("capsule_id") and capsule_id not in capsules_by_id:
+ raise StaleRevisionError("noop capsule does not exist in region")
+ else:
+ current = capsules_by_id.get(capsule_id)
+ if current is None:
+ raise StaleRevisionError(
+ "capsule is not a current revision in this region"
+ )
+ if (
+ head is None
+ or operation["base_revision"] != head[0]
+ or operation["base_revision"] != current.get("revision")
+ ):
+ raise StaleRevisionError(
+ "base_revision is stale for " + capsule_id
+ )
+ planned.append(
+ {
+ "operation": operation,
+ "action": action,
+ "capsule_id": capsule_id,
+ "head": head,
+ }
+ )
+
+ con.execute(
+ "INSERT INTO slow_graph_patches VALUES(?,?,?,?,?,?,?,?)",
+ (
+ patch_id,
+ job_id,
+ job["scope_id"],
+ job["region_key"],
+ manager_model,
+ _json(patch),
+ _json(metadata_for_call),
+ now,
+ ),
+ )
+ for ordinal, item in enumerate(planned):
+ operation = item["operation"]
+ action = item["action"]
+ capsule_id = item["capsule_id"]
+ head = item["head"]
+ old_memory_id: str | None = None
+ if action == "create":
+ base, revision = None, 1
+ record_id = super()._insert_revision(
+ con,
+ job=job,
+ patch_id=patch_id,
+ operation=operation,
+ capsule_id=capsule_id,
+ revision=revision,
+ action=action,
+ )
+ self._annotate_revision(
+ con,
+ scope_id=job["scope_id"],
+ record_id=record_id,
+ operation=operation,
+ old_memory_id=None,
+ call_metadata=metadata_for_call,
+ )
+ elif action == "noop":
+ if head is None:
+ record_id, base, revision = "", None, None
+ else:
+ record_id, (revision, _) = head[1], head
+ base = revision
+ else:
+ if head is None:
+ raise StaleRevisionError("capsule head disappeared")
+ base, revision = head[0], head[0] + 1
+ old_memory_id = head[1]
+ record_id = super()._insert_revision(
+ con,
+ job=job,
+ patch_id=patch_id,
+ operation=operation,
+ capsule_id=capsule_id,
+ revision=revision,
+ action=action,
+ old_memory_id=old_memory_id,
+ )
+ self._annotate_revision(
+ con,
+ scope_id=job["scope_id"],
+ record_id=record_id,
+ operation=operation,
+ old_memory_id=old_memory_id,
+ call_metadata=metadata_for_call,
+ )
+ if action == "challenge":
+ self._write_edge(
+ con,
+ scope_id=job["scope_id"],
+ source=record_id,
+ target=old_memory_id,
+ edge_type="challenges",
+ patch_id=patch_id,
+ evidence_refs=[],
+ action=action,
+ turn=now,
+ )
+ if action == "retire":
+ self._write_edge(
+ con,
+ scope_id=job["scope_id"],
+ source=record_id,
+ target=old_memory_id,
+ edge_type="invalidates",
+ patch_id=patch_id,
+ evidence_refs=[],
+ action=action,
+ turn=now,
+ )
+ con.execute(
+ "INSERT INTO slow_graph_patch_operations VALUES(?,?,?,?,?,?,?,?,?)",
+ (
+ "sgo_" + uuid.uuid4().hex,
+ patch_id,
+ ordinal,
+ capsule_id,
+ action,
+ base,
+ revision,
+ _json(operation),
+ now,
+ ),
+ )
+
+ completed_attempt = con.execute(
+ "UPDATE slow_graph_attempts SET status='completed',"
+ "call_metadata_json=?,completed_at=? WHERE attempt_id=? AND job_id=? "
+ "AND claim_token=? AND claim_owner=? AND status='started'",
+ (
+ _json(metadata_for_call),
+ now,
+ claim.attempt_id,
+ job_id,
+ claim.token,
+ claim.owner,
+ ),
+ )
+ if completed_attempt.rowcount != 1:
+ raise SlowGraphError("claimed attempt is no longer active")
+ completed_job = con.execute(
+ "UPDATE slow_graph_jobs SET status='completed',attempts=attempts+1,"
+ "last_error='',updated_at=?,claim_token=NULL,claim_owner=NULL,"
+ "lease_expires_at=NULL WHERE job_id=? AND status='pending' "
+ "AND claim_token=? AND claim_owner=? AND lease_expires_at>=?",
+ (now, job_id, claim.token, claim.owner, now),
+ )
+ if completed_job.rowcount != 1:
+ raise SlowGraphError("job claim expired before completion")
+ self._audit_transaction(con, job["scope_id"])
+ return patch_id
+
+ def _audit_transaction(self, con: Any, scope_id: str) -> None:
+ super()._audit_transaction(con, scope_id)
+ record_rows = con.execute(
+ "SELECT memory_id,value,metadata_json FROM records WHERE scope_id=?",
+ (scope_id,),
+ ).fetchall()
+ current_patch_ids: set[str] = set()
+ result_records: dict[tuple[str, str, int], list[tuple[Any, dict[str, Any]]]] = {}
+ for row in record_rows:
+ metadata = self._metadata(row, "V4 transaction record")
+ if metadata.get("content_variant") != CAPSULE_VARIANT:
+ continue
+ patch_id = _clean(metadata.get("patch_id"))
+ capsule_id = _required_text(metadata.get("capsule_id"), "capsule_id")
+ revision = metadata.get("revision")
+ if (
+ isinstance(revision, bool)
+ or not isinstance(revision, int)
+ or revision < 1
+ ):
+ raise AuditError("V4 capsule revision is invalid")
+ result_records.setdefault(
+ (patch_id, capsule_id, revision), []
+ ).append((row, metadata))
+ if metadata.get("summary_contract_version") != SLOW_SUMMARY_CONTRACT_VERSION:
+ continue
+ if (
+ metadata.get("evidence_binding_contract_version")
+ not in SUPPORTED_SLOW_EVIDENCE_BINDING_CONTRACT_VERSIONS
+ ):
+ raise AuditError("stored Slow evidence-binding contract has drifted")
+ current_patch_ids.add(patch_id)
+ claims = _canonical_patch_claims(metadata.get("claims"))
+ expected_summary = _semantic_summary_projection(claims)
+ if row["value"] != expected_summary:
+ raise AuditError(
+ "current V4 Slow summary differs from final claims projection"
+ )
+ expected_digest = hashlib.sha256(
+ expected_summary.encode("utf-8")
+ ).hexdigest()
+ if metadata.get("summary_projection_sha256") != expected_digest:
+ raise AuditError("current V4 Slow summary digest is inconsistent")
+ _normalize_capsule_key(
+ metadata.get("capsule_key"), "stored capsule_key"
+ )
+
+ orphan_count = int(
+ con.execute(
+ "SELECT count(*) FROM slow_graph_patch_operations o "
+ "LEFT JOIN slow_graph_patches p ON p.patch_id=o.patch_id "
+ "WHERE p.patch_id IS NULL"
+ ).fetchone()[0]
+ )
+ if orphan_count:
+ raise AuditError("orphan Slow patch operations exist")
+
+ patches = con.execute(
+ "SELECT * FROM slow_graph_patches WHERE scope_id=?", (scope_id,)
+ ).fetchall()
+ for patch_row in patches:
+ patch_id = str(patch_row["patch_id"])
+ call_metadata = _v3._strict_json(
+ patch_row["call_metadata_json"],
+ label="V4 patch call metadata",
+ expected=dict,
+ )
+ if (
+ call_metadata.get("evidence_binding_contract_version")
+ not in SUPPORTED_SLOW_EVIDENCE_BINDING_CONTRACT_VERSIONS
+ ):
+ raise AuditError("Slow patch evidence-binding contract has drifted")
+ is_current = (
+ patch_id in current_patch_ids
+ or call_metadata.get("summary_contract_version")
+ == SLOW_SUMMARY_CONTRACT_VERSION
+ or call_metadata.get("prompt_version") == SLOW_PROMPT_VERSION
+ )
+ if not is_current:
+ continue
+ patch = _v3._strict_json(
+ patch_row["patch_json"], label="V4 patch", expected=dict
+ )
+ validate_patch(patch, require_lossless_summary=True)
+ operation_rows = con.execute(
+ "SELECT * FROM slow_graph_patch_operations "
+ "WHERE patch_id=? ORDER BY ordinal,operation_id",
+ (patch_id,),
+ ).fetchall()
+ operations = patch["operations"]
+ if len(operation_rows) != len(operations):
+ raise AuditError("V4 patch operation row count is inconsistent")
+ if [int(row["ordinal"]) for row in operation_rows] != list(
+ range(len(operations))
+ ):
+ raise AuditError("V4 patch operation ordinals are inconsistent")
+ for ordinal, (operation, operation_row) in enumerate(
+ zip(operations, operation_rows, strict=True)
+ ):
+ stored_operation = _v3._strict_json(
+ operation_row["operation_json"],
+ label="V4 patch operation",
+ expected=dict,
+ )
+ if stored_operation != operation:
+ raise AuditError(
+ f"V4 patch operation {ordinal} differs from patch_json"
+ )
+ action = operation["action"]
+ if operation_row["action"] != action:
+ raise AuditError("V4 patch operation action is inconsistent")
+ capsule_id = _required_text(
+ operation_row["capsule_id"], "operation capsule_id"
+ )
+ base_revision = operation_row["base_revision"]
+ result_revision = operation_row["result_revision"]
+ if action == "create":
+ expected_capsule_id = self._capsule_id(
+ scope_id,
+ patch_row["region_key"],
+ operation["capsule_key"],
+ )
+ if (
+ capsule_id != expected_capsule_id
+ or base_revision is not None
+ or result_revision != 1
+ ):
+ raise AuditError("V4 create operation identity is inconsistent")
+ elif action == "noop":
+ expected_target = _clean(operation.get("capsule_id"))
+ if expected_target and capsule_id != expected_target:
+ raise AuditError("V4 noop operation identity is inconsistent")
+ if base_revision != result_revision:
+ raise AuditError("V4 noop revision mapping is inconsistent")
+ continue
+ else:
+ if capsule_id != operation["capsule_id"]:
+ raise AuditError("V4 operation capsule target is inconsistent")
+ if (
+ base_revision != operation["base_revision"]
+ or result_revision != operation["base_revision"] + 1
+ ):
+ raise AuditError("V4 operation revision mapping is inconsistent")
+ matches = result_records.get(
+ (patch_id, capsule_id, int(result_revision)), []
+ )
+ if len(matches) != 1:
+ raise AuditError(
+ "V4 patch operation lacks one matching result revision"
+ )
+ result_metadata = matches[0][1]
+ if result_metadata.get("action") != action:
+ raise AuditError("V4 result revision action is inconsistent")
+
+ def fast_regions(self, scope_id: str) -> dict[str, list[dict[str, Any]]]:
+ regions = super().fast_regions(scope_id)
+ with self.connection() as con:
+ state_by_id = {
+ str(row["memory_id"]): _clean(row["state"])
+ for row in con.execute(
+ "SELECT memory_id,state FROM records WHERE scope_id=?", (scope_id,)
+ ).fetchall()
+ }
+ for leaves in regions.values():
+ for leaf in leaves:
+ memory_id = _leaf_id(leaf)
+ state = _required_text(
+ state_by_id.get(memory_id), f"fast evidence {memory_id} record state"
+ )
+ leaf["record_state"] = state
+ leaf["metadata"] = {**dict(_leaf_metadata(leaf)), "record_state": state}
+ return regions
+
+ def promotion_coverage(self, scope_id: str) -> dict[str, Any]:
+ regions = self.fast_regions(scope_id)
+ leaves_by_id = {
+ _leaf_id(leaf): leaf
+ for leaves in regions.values()
+ for leaf in leaves
+ }
+ eligible_ids = {
+ memory_id
+ for memory_id, leaf in leaves_by_id.items()
+ if _is_current_durable(leaf)
+ }
+ challenged_ids = {
+ memory_id
+ for memory_id, leaf in leaves_by_id.items()
+ if _is_challenged_durable(leaf)
+ }
+ uncertain_ids = {
+ memory_id
+ for memory_id, leaf in leaves_by_id.items()
+ if _is_uncertain(leaf)
+ }
+ episodic_ids = {
+ memory_id
+ for memory_id, leaf in leaves_by_id.items()
+ if _is_episodic(leaf)
+ }
+ classified = eligible_ids | challenged_ids | uncertain_ids | episodic_ids
+ inactive_ids = set(leaves_by_id) - classified
+ cited_ids: set[str] = set()
+ active_claim_count = 0
+ invalid_capsule_ids: set[str] = set()
+ semantic_integrity_issues: list[dict[str, Any]] = []
+ evidence_locations: dict[str, list[dict[str, Any]]] = {}
+ citation_bindings: dict[str, list[tuple[str, str, str, str]]] = {}
+ claim_identity_capsules: dict[tuple[str, str], set[str]] = {}
+ capsule_key_owners: dict[tuple[str, str], set[str]] = {}
+ with self.connection() as con:
+ patch_routes: dict[str, str] = {}
+ patch_table = con.execute(
+ "SELECT 1 FROM sqlite_master WHERE type='table' "
+ "AND name='slow_graph_patches'"
+ ).fetchone()
+ if patch_table is not None:
+ for patch_row in con.execute(
+ "SELECT patch_id,call_metadata_json FROM slow_graph_patches "
+ "WHERE scope_id=?",
+ (scope_id,),
+ ).fetchall():
+ patch_metadata = _v3._strict_json(
+ patch_row["call_metadata_json"],
+ label="slow patch call metadata",
+ expected=dict,
+ )
+ patch_routes[str(patch_row["patch_id"])] = _clean(
+ patch_metadata.get("route")
+ )
+ rows = con.execute(
+ "SELECT memory_id,state,value,metadata_json FROM records WHERE scope_id=?",
+ (scope_id,),
+ ).fetchall()
+ capsules_by_id: dict[str, list[tuple[int, sqlite3.Row, dict[str, Any]]]] = {}
+ for row in rows:
+ metadata = self._metadata(row, "record")
+ if metadata.get("content_variant") != CAPSULE_VARIANT:
+ continue
+ capsule_id = _clean(metadata.get("capsule_id"))
+ revision = metadata.get("revision")
+ if (
+ not capsule_id
+ or isinstance(revision, bool)
+ or not isinstance(revision, int)
+ or revision < 1
+ ):
+ invalid_capsule_ids.add(capsule_id or f"memory:{row['memory_id']}")
+ continue
+ capsules_by_id.setdefault(capsule_id, []).append(
+ (revision, row, metadata)
+ )
+
+ latest_capsules: list[
+ tuple[Any, dict[str, Any], str, set[str]]
+ ] = []
+ for capsule_id, revisions in capsules_by_id.items():
+ latest_revision = max(item[0] for item in revisions)
+ latest = [item for item in revisions if item[0] == latest_revision]
+ if len(latest) != 1:
+ invalid_capsule_ids.add(capsule_id)
+ continue
+ row, metadata = latest[0][1], latest[0][2]
+ if (
+ row["state"] != "active"
+ or metadata.get("status") not in {"active", "challenged"}
+ ):
+ continue
+ prior_counterevidence: set[str] = set()
+ prior = [
+ item for item in revisions if item[0] == latest_revision - 1
+ ]
+ if len(prior) == 1:
+ prior_counterevidence = _prior_counterevidence_ids(
+ [prior[0][2]]
+ )
+ route = patch_routes.get(_clean(metadata.get("patch_id")), "")
+ latest_capsules.append(
+ (row, metadata, route, prior_counterevidence)
+ )
+
+ for capsule_row, metadata, route, prior_counterevidence in latest_capsules:
+ claims = _v3._validate_claims(metadata.get("claims"), stored=True)
+ capsule_id = _clean(metadata.get("capsule_id"))
+ revision = metadata.get("revision")
+ try:
+ summary = validate_semantic_summary(
+ capsule_row["value"],
+ claims,
+ label="active Slow capsule summary",
+ )
+ expected_summary = _semantic_summary_projection(
+ _canonical_patch_claims(claims)
+ )
+ if summary != expected_summary:
+ raise PatchValidationError(
+ "active Slow capsule summary differs from final claims"
+ )
+ except PatchValidationError as exc:
+ semantic_integrity_issues.append(
+ {
+ "code": "invalid_semantic_summary",
+ "capsule_id": capsule_id,
+ "revision": revision,
+ "error": str(exc),
+ }
+ )
+ if metadata.get("summary_contract_version") != SLOW_SUMMARY_CONTRACT_VERSION:
+ semantic_integrity_issues.append(
+ {
+ "code": "missing_lossless_summary_contract",
+ "capsule_id": capsule_id,
+ "revision": revision,
+ "stored_contract_version": metadata.get(
+ "summary_contract_version"
+ ),
+ }
+ )
+ capsule_key = _clean(metadata.get("capsule_key")).casefold()
+ if metadata.get("summary_contract_version") == SLOW_SUMMARY_CONTRACT_VERSION:
+ try:
+ capsule_key = _normalize_capsule_key(
+ capsule_key, "stored capsule_key"
+ )
+ except PatchValidationError as exc:
+ semantic_integrity_issues.append(
+ {
+ "code": "invalid_capsule_key",
+ "capsule_id": capsule_id,
+ "revision": revision,
+ "error": str(exc),
+ }
+ )
+ if capsule_key:
+ capsule_key_owners.setdefault(
+ (_clean(metadata.get("region_key")), capsule_key), set()
+ ).add(capsule_id)
+ if (
+ metadata.get("partition_contract_version")
+ != SLOW_PARTITION_CONTRACT_VERSION
+ ):
+ semantic_integrity_issues.append(
+ {
+ "code": "semantic_partition_migration_required",
+ "capsule_id": capsule_id,
+ "revision": revision,
+ "region_key": metadata.get("region_key"),
+ "claim_count": len(claims),
+ }
+ )
+ active_claim_count += len(claims)
+ claim_roles: list[tuple[str, set[str], set[str]]] = []
+ for claim in claims:
+ cited_ids.update(claim["support"])
+ cited_ids.update(claim["counterevidence"])
+ claim_id = _clean(claim.get("claim_id"))
+ claim_slot = _clean(claim.get("canonical_slot"))
+ claim_text = _normal_text(claim.get("text"))
+ claim_support = set(claim["support"])
+ claim_counter = set(claim["counterevidence"])
+ complementary_support_bundle = (
+ _controlled_complementary_support_bundle(
+ claim_slot, claim_support, leaves_by_id
+ )
+ )
+ claim_identity_capsules.setdefault(
+ (claim_slot, claim_text), set()
+ ).add(capsule_id)
+ support_text_groups = _support_text_groups(
+ claim_support, leaves_by_id
+ )
+ if (
+ len(support_text_groups) > 1
+ and not complementary_support_bundle
+ ):
+ semantic_integrity_issues.append(
+ {
+ "code": "support_distinct_fast_values_merged",
+ "capsule_id": capsule_id,
+ "revision": revision,
+ "claim_id": claim_id,
+ "evidence_groups": [
+ {
+ "normalized_text": text,
+ "evidence_ids": evidence_ids,
+ }
+ for text, evidence_ids in sorted(
+ support_text_groups.items()
+ )
+ ],
+ }
+ )
+ for role, evidence_values in (
+ ("support", claim_support),
+ ("counterevidence", claim_counter),
+ ):
+ for evidence_id in evidence_values:
+ location = {
+ "capsule_id": capsule_id,
+ "revision": revision,
+ "claim_id": claim_id,
+ "role": role,
+ }
+ evidence_locations.setdefault(evidence_id, []).append(
+ location
+ )
+ citation_bindings.setdefault(evidence_id, []).append(
+ (
+ role,
+ claim_slot,
+ claim_text,
+ _json(location),
+ )
+ )
+ shared_roles = claim_support & claim_counter
+ if shared_roles:
+ semantic_integrity_issues.append(
+ {
+ "code": "same_evidence_support_and_counterevidence",
+ "capsule_id": _clean(metadata.get("capsule_id")),
+ "revision": metadata.get("revision"),
+ "claim_id": claim_id,
+ "evidence_ids": sorted(shared_roles),
+ }
+ )
+ if metadata.get("action") == "create" and claim_counter:
+ semantic_integrity_issues.append(
+ {
+ "code": "active_create_contains_counterevidence",
+ "capsule_id": _clean(metadata.get("capsule_id")),
+ "revision": metadata.get("revision"),
+ "claim_id": claim_id,
+ "evidence_ids": sorted(claim_counter),
+ }
+ )
+ claim_roles.append((claim_id, claim_support, claim_counter))
+ for evidence_id in claim["support"]:
+ leaf = leaves_by_id.get(evidence_id)
+ if leaf is None:
+ semantic_integrity_issues.append(
+ {
+ "code": "support_missing_fast_leaf",
+ "capsule_id": _clean(metadata.get("capsule_id")),
+ "revision": metadata.get("revision"),
+ "claim_id": claim_id,
+ "evidence_id": evidence_id,
+ }
+ )
+ elif (
+ _leaf_slot(leaf) != claim_slot
+ and not complementary_support_bundle
+ ):
+ semantic_integrity_issues.append(
+ {
+ "code": "support_canonical_slot_mismatch",
+ "capsule_id": _clean(metadata.get("capsule_id")),
+ "revision": metadata.get("revision"),
+ "claim_id": claim_id,
+ "claim_slot": claim_slot,
+ "evidence_id": evidence_id,
+ "evidence_slot": _leaf_slot(leaf),
+ }
+ )
+ elif not (
+ _is_current_durable(leaf)
+ or _is_challenged_durable(leaf)
+ ):
+ semantic_integrity_issues.append(
+ {
+ "code": "support_noncurrent_fast_leaf",
+ "capsule_id": _clean(metadata.get("capsule_id")),
+ "revision": metadata.get("revision"),
+ "claim_id": claim_id,
+ "evidence_id": evidence_id,
+ "evidence_state": _leaf_state(leaf),
+ }
+ )
+ for evidence_id in claim["counterevidence"]:
+ leaf = leaves_by_id.get(evidence_id)
+ if leaf is None:
+ semantic_integrity_issues.append(
+ {
+ "code": "counterevidence_missing_fast_leaf",
+ "capsule_id": _clean(metadata.get("capsule_id")),
+ "revision": metadata.get("revision"),
+ "claim_id": claim_id,
+ "evidence_id": evidence_id,
+ }
+ )
+ continue
+ if _normal_text(_leaf_text(leaf)) == claim_text:
+ semantic_integrity_issues.append(
+ {
+ "code": "counterevidence_identical_to_claim",
+ "capsule_id": _clean(metadata.get("capsule_id")),
+ "revision": metadata.get("revision"),
+ "claim_id": claim_id,
+ "evidence_id": evidence_id,
+ }
+ )
+ if (
+ route == "flash"
+ and evidence_id not in prior_counterevidence
+ and not _is_counterevidence(leaf)
+ ):
+ semantic_integrity_issues.append(
+ {
+ "code": "flash_invented_counterevidence",
+ "capsule_id": _clean(metadata.get("capsule_id")),
+ "revision": metadata.get("revision"),
+ "claim_id": claim_id,
+ "evidence_id": evidence_id,
+ "route": route,
+ }
+ )
+ for left_index, (
+ left_claim_id,
+ left_support,
+ left_counter,
+ ) in enumerate(claim_roles):
+ for (
+ right_claim_id,
+ right_support,
+ right_counter,
+ ) in claim_roles[left_index + 1 :]:
+ if (
+ left_support & right_counter
+ and right_support & left_counter
+ ):
+ semantic_integrity_issues.append(
+ {
+ "code": "reciprocal_counterevidence_cycle",
+ "capsule_id": _clean(metadata.get("capsule_id")),
+ "revision": metadata.get("revision"),
+ "claim_ids": sorted(
+ [left_claim_id, right_claim_id]
+ ),
+ }
+ )
+ for evidence_id, locations in evidence_locations.items():
+ if len(locations) > 1:
+ try:
+ _validate_repeated_evidence_bindings(
+ leaves_by_id,
+ {evidence_id: citation_bindings[evidence_id]},
+ )
+ except PatchValidationError as exc:
+ semantic_integrity_issues.append(
+ {
+ "code": "fast_evidence_assigned_multiple_times",
+ "evidence_id": evidence_id,
+ "locations": locations,
+ "error": str(exc),
+ }
+ )
+ for (slot, claim_text), capsule_ids in claim_identity_capsules.items():
+ if len(capsule_ids) > 1:
+ semantic_integrity_issues.append(
+ {
+ "code": "semantic_claim_split_across_capsules",
+ "canonical_slot": slot,
+ "normalized_claim_text": claim_text,
+ "capsule_ids": sorted(capsule_ids),
+ }
+ )
+ for (region_key, capsule_key), capsule_ids in capsule_key_owners.items():
+ if len(capsule_ids) > 1:
+ semantic_integrity_issues.append(
+ {
+ "code": "duplicate_capsule_key_in_region",
+ "region_key": region_key,
+ "capsule_key": capsule_key,
+ "capsule_ids": sorted(capsule_ids),
+ }
+ )
+ cited_eligible = eligible_ids & cited_ids
+ uncited_eligible = sorted(eligible_ids - cited_ids)
+ eligible_count = len(eligible_ids)
+ semantic_integrity_issues.sort(key=_json)
+ return {
+ "schema_version": "tmcra.v4.slow-promotion-coverage.3",
+ "complete": (
+ not uncited_eligible
+ and not invalid_capsule_ids
+ and not semantic_integrity_issues
+ ),
+ "eligible_current_durable_count": eligible_count,
+ "cited_current_durable_count": len(cited_eligible),
+ "coverage_ratio": (
+ round(len(cited_eligible) / eligible_count, 6)
+ if eligible_count
+ else 1.0
+ ),
+ "uncited_current_durable_ids": uncited_eligible,
+ "challenged_durable_count": len(challenged_ids),
+ "uncertain_count": len(uncertain_ids),
+ "episodic_count": len(episodic_ids),
+ "inactive_or_other_count": len(inactive_ids),
+ "active_or_challenged_claim_count": active_claim_count,
+ "invalid_capsule_identity_count": len(invalid_capsule_ids),
+ "invalid_capsule_ids": sorted(invalid_capsule_ids),
+ "semantic_integrity_issue_count": len(semantic_integrity_issues),
+ "semantic_integrity_issues": semantic_integrity_issues,
+ }
+
+ def drain(
+ self, manager: Any, *, batch_size: int | None = None
+ ) -> list[str]:
+ """Drain independent jobs while retaining complete invalid model responses."""
+ self.recover_interrupted_attempts()
+ if batch_size is not None and batch_size <= 0:
+ raise SlowGraphError("batch_size must be positive")
+ owner = self._claim_owner()
+ results: list[str] = []
+ failures: list[dict[str, str]] = []
+ processed = 0
+ while batch_size is None or processed < batch_size:
+ claim = self._claim_pending_job(None, owner=owner)
+ if claim is None:
+ break
+ processed += 1
+ try:
+ results.append(self._run_claimed_job(claim, manager))
+ except Exception as exc:
+ metadata = dict(getattr(manager, "last_call_metadata", {}) or {})
+ complete_model_response = (
+ metadata.get("physical_api_call") is True
+ and int(metadata.get("physical_api_calls", 0) or 0) >= 1
+ and int(metadata.get("http_status", 0) or 0) == 200
+ and metadata.get("finish_reason") == "stop"
+ and metadata.get("status") in {"response_received", "completed"}
+ and _clean(metadata.get("raw_response"))
+ )
+ if not complete_model_response:
+ raise
+ failures.append(
+ {
+ "job_id": claim.job_id,
+ "error_type": exc.__class__.__name__,
+ "error": str(exc),
+ }
+ )
+ if failures:
+ raise SlowGraphError(
+ "slow graph retained complete invalid model responses: "
+ + _json(failures)
+ )
+ return results
+
+ def _propose_with_lease_heartbeat(
+ self,
+ claim: JobClaim,
+ manager: Any,
+ region: Mapping[str, Any],
+ capsules: list[dict[str, Any]],
+ ) -> dict[str, Any]:
+ patch = super()._propose_with_lease_heartbeat(
+ claim, manager, region, capsules
+ )
+ _validate_promotion_patch(region, capsules, patch)
+ return patch
+
+ def audit(
+ self, scope_id: str, *, require_promotion_coverage: bool = False
+ ) -> dict[str, Any]:
+ result = dict(super().audit(scope_id))
+ promotion_coverage = self.promotion_coverage(scope_id)
+ if require_promotion_coverage and not promotion_coverage["complete"]:
+ failures: list[str] = []
+ if promotion_coverage["uncited_current_durable_ids"]:
+ failures.append(
+ "current durable Fast evidence is missing from active Slow claims: "
+ + _json(
+ promotion_coverage["uncited_current_durable_ids"][:20]
+ )
+ )
+ if promotion_coverage["invalid_capsule_ids"]:
+ failures.append(
+ "active Slow capsule identity is invalid: "
+ + _json(promotion_coverage["invalid_capsule_ids"][:20])
+ )
+ if promotion_coverage["semantic_integrity_issues"]:
+ failures.append(
+ "active Slow claim semantic integrity failed: "
+ + _json(
+ promotion_coverage["semantic_integrity_issues"][:20]
+ )
+ )
+ raise AuditError("; ".join(failures))
+ route_counts: dict[str, int] = {}
+ usage = {
+ "physical_api_calls": 0,
+ "prompt_tokens": 0,
+ "completion_tokens": 0,
+ "cache_read_input_tokens": 0,
+ "cache_hit_tokens": 0,
+ "cache_miss_tokens": 0,
+ "total_tokens": 0,
+ "estimated_cost": 0.0,
+ }
+ with self.connection() as con:
+ rows = con.execute(
+ "SELECT call_metadata_json FROM slow_graph_attempts WHERE scope_id=?",
+ (scope_id,),
+ ).fetchall()
+ for row in rows:
+ metadata = _v3._strict_json(
+ row["call_metadata_json"],
+ label="patch call metadata",
+ expected=dict,
+ )
+ route = _clean(metadata.get("route")) or "unknown"
+ route_counts[route] = route_counts.get(route, 0) + 1
+ usage["physical_api_calls"] += int(metadata.get("physical_api_calls", 0) or 0)
+ raw_usage = metadata.get("usage")
+ if isinstance(raw_usage, Mapping):
+ for key in ("prompt_tokens", "completion_tokens", "cache_read_input_tokens", "cache_hit_tokens", "cache_miss_tokens", "total_tokens"):
+ usage[key] += int(raw_usage.get(key, 0) or 0)
+ cost_audit = metadata.get("cost_audit")
+ if isinstance(cost_audit, Mapping):
+ usage["estimated_cost"] += float(cost_audit.get("estimated_cost", 0.0) or 0.0)
+ zero_call_recoveries = 0
+ zero_call_promotion_recoveries = 0
+ zero_call_projection_recoveries = 0
+ process_loss_recoveries = 0
+ process_loss_potential_min = 0
+ process_loss_potential_max = 0
+ with self.connection() as con:
+ recovery_table = con.execute(
+ "SELECT 1 FROM sqlite_master WHERE type='table' "
+ "AND name='slow_graph_zero_call_recoveries'"
+ ).fetchone()
+ if recovery_table is not None:
+ recoveries = con.execute(
+ "SELECT * FROM slow_graph_zero_call_recoveries "
+ "WHERE scope_id=? ORDER BY created_at,recovery_id",
+ (scope_id,),
+ ).fetchall()
+ for recovery in recoveries:
+ attempt = con.execute(
+ "SELECT * FROM slow_graph_attempts "
+ "WHERE attempt_id=? AND job_id=?",
+ (recovery["attempt_id"], recovery["job_id"]),
+ ).fetchone()
+ if attempt is None:
+ raise AuditError(
+ "zero-call configuration recovery references a missing attempt"
+ )
+ raw_metadata = _clean(attempt["call_metadata_json"])
+ metadata = _v3._strict_json(
+ raw_metadata,
+ label="zero-call recovery metadata",
+ expected=dict,
+ )
+ error = _clean(attempt["error"])
+ route = _clean(metadata.get("route"))
+ if (
+ int(
+ recovery["physical_api_calls"]
+ if recovery["physical_api_calls"] is not None
+ else -1
+ )
+ != 0
+ or route != _clean(recovery["route"])
+ or error != ZERO_CALL_CONFIGURATION_ERRORS.get(route)
+ or metadata.get("physical_api_call") is not False
+ or int(metadata.get("physical_api_calls", -1)) != 0
+ or hashlib.sha256(error.encode("utf-8")).hexdigest()
+ != _clean(recovery["error_sha256"])
+ or hashlib.sha256(raw_metadata.encode("utf-8")).hexdigest()
+ != _clean(recovery["call_metadata_sha256"])
+ ):
+ raise AuditError(
+ "zero-call configuration recovery contract is invalid"
+ )
+ zero_call_recoveries = len(recoveries)
+ promotion_table = con.execute(
+ "SELECT 1 FROM sqlite_master WHERE type='table' "
+ "AND name='slow_graph_zero_call_promotion_recoveries'"
+ ).fetchone()
+ if promotion_table is not None:
+ promotion_recoveries = con.execute(
+ "SELECT * FROM slow_graph_zero_call_promotion_recoveries "
+ "WHERE scope_id=? ORDER BY created_at,recovery_id",
+ (scope_id,),
+ ).fetchall()
+ for recovery in promotion_recoveries:
+ attempt = con.execute(
+ "SELECT * FROM slow_graph_attempts "
+ "WHERE attempt_id=? AND job_id=?",
+ (recovery["attempt_id"], recovery["job_id"]),
+ ).fetchone()
+ if attempt is None:
+ raise AuditError(
+ "zero-call promotion recovery references a missing attempt"
+ )
+ raw_metadata = _clean(attempt["call_metadata_json"])
+ metadata = _v3._strict_json(
+ raw_metadata,
+ label="zero-call promotion recovery metadata",
+ expected=dict,
+ )
+ error = _clean(attempt["error"])
+ eligible_ids = metadata.get("eligible_evidence_ids")
+ challenged_ids = metadata.get("challenged_evidence_ids")
+ delta_ids = metadata.get("delta_evidence_ids")
+ if (
+ int(
+ recovery["physical_api_calls"]
+ if recovery["physical_api_calls"] is not None
+ else -1
+ )
+ != 0
+ or not error.startswith(
+ "noop cannot consume uncited current durable Fast evidence: "
+ )
+ or _clean(metadata.get("route")) != "deterministic_noop"
+ or _clean(metadata.get("route_reason"))
+ != "new capsule blocked by unresolved fast challenge"
+ or metadata.get("physical_api_call") is not False
+ or int(metadata.get("physical_api_calls", -1)) != 0
+ or not isinstance(eligible_ids, list)
+ or not eligible_ids
+ or not isinstance(challenged_ids, list)
+ or not challenged_ids
+ or not isinstance(delta_ids, list)
+ or not set(eligible_ids) <= set(delta_ids)
+ or hashlib.sha256(error.encode("utf-8")).hexdigest()
+ != _clean(recovery["error_sha256"])
+ or hashlib.sha256(raw_metadata.encode("utf-8")).hexdigest()
+ != _clean(recovery["call_metadata_sha256"])
+ ):
+ raise AuditError(
+ "zero-call promotion recovery contract is invalid"
+ )
+ zero_call_promotion_recoveries = len(promotion_recoveries)
+ projection_table = con.execute(
+ "SELECT 1 FROM sqlite_master WHERE type='table' "
+ "AND name='slow_graph_zero_call_projection_recoveries'"
+ ).fetchone()
+ if projection_table is not None:
+ projection_recoveries = con.execute(
+ "SELECT * FROM slow_graph_zero_call_projection_recoveries "
+ "WHERE scope_id=? ORDER BY created_at,recovery_id",
+ (scope_id,),
+ ).fetchall()
+ for recovery in projection_recoveries:
+ attempt = con.execute(
+ "SELECT * FROM slow_graph_attempts "
+ "WHERE attempt_id=? AND job_id=?",
+ (recovery["attempt_id"], recovery["job_id"]),
+ ).fetchone()
+ job = con.execute(
+ "SELECT * FROM slow_graph_jobs WHERE job_id=?",
+ (recovery["job_id"],),
+ ).fetchone()
+ if attempt is None or job is None:
+ raise AuditError(
+ "zero-call projection recovery references missing state"
+ )
+ raw_metadata = _clean(attempt["call_metadata_json"])
+ metadata = _v3._strict_json(
+ raw_metadata,
+ label="zero-call projection recovery metadata",
+ expected=dict,
+ )
+ error = _clean(attempt["error"])
+ evidence_ids = _v3._strict_json(
+ job["evidence_ids_json"],
+ label="projection recovery evidence IDs",
+ expected=list,
+ )
+ evidence = self._evidence(
+ con, job["scope_id"], evidence_ids
+ )
+ offending_paths = []
+ for index, leaf in enumerate(evidence):
+ for key, value in _leaf_metadata(leaf).items():
+ if not _forbidden_field(key):
+ continue
+ if key != "origin_answer_ids" or value != []:
+ raise AuditError(
+ "projection recovery contains non-empty benchmark metadata"
+ )
+ offending_paths.append(
+ f"payload.evidence[{index}].metadata.origin_answer_ids"
+ )
+ stored_paths = _v3._strict_json(
+ recovery["offending_paths_json"],
+ label="projection recovery offending paths",
+ expected=list,
+ )
+ public_region = {
+ "region_key": _required_text(
+ job["region_key"], "region key"
+ ),
+ "evidence": [_public_leaf(item) for item in evidence],
+ }
+ _assert_no_benchmark_fields(public_region)
+ if (
+ int(
+ recovery["physical_api_calls"]
+ if recovery["physical_api_calls"] is not None
+ else -1
+ )
+ != 0
+ or metadata.get("physical_api_call") is not False
+ or int(metadata.get("physical_api_calls", -1)) != 0
+ or hashlib.sha256(error.encode("utf-8")).hexdigest()
+ != _clean(recovery["error_sha256"])
+ or hashlib.sha256(raw_metadata.encode("utf-8")).hexdigest()
+ != _clean(recovery["call_metadata_sha256"])
+ or _digest(public_region)
+ != _clean(recovery["public_projection_sha256"])
+ or offending_paths != stored_paths
+ ):
+ raise AuditError(
+ "zero-call projection recovery contract is invalid"
+ )
+ zero_call_projection_recoveries = len(projection_recoveries)
+ process_loss_rows = con.execute(
+ "SELECT * FROM slow_graph_process_loss_recoveries "
+ "WHERE scope_id=? ORDER BY recovered_at,recovery_id",
+ (scope_id,),
+ ).fetchall()
+ for recovery in process_loss_rows:
+ attempt = con.execute(
+ "SELECT * FROM slow_graph_attempts WHERE attempt_id=? "
+ "AND job_id=? AND scope_id=?",
+ (
+ recovery["attempt_id"],
+ recovery["job_id"],
+ recovery["scope_id"],
+ ),
+ ).fetchone()
+ job = con.execute(
+ "SELECT * FROM slow_graph_jobs WHERE job_id=? AND scope_id=?",
+ (recovery["job_id"], recovery["scope_id"]),
+ ).fetchone()
+ if attempt is None or job is None:
+ raise AuditError(
+ "process-loss recovery references missing Slow state"
+ )
+ raw_metadata = _required_text(
+ attempt["call_metadata_json"],
+ "process-loss attempt metadata",
+ )
+ metadata = _v3._strict_json(
+ raw_metadata,
+ label="process-loss attempt metadata",
+ expected=dict,
+ )
+ expected_recovery_id = "sgr_" + _digest(
+ {
+ "job_id": recovery["job_id"],
+ "attempt_id": recovery["attempt_id"],
+ "claim_token": recovery["claim_token"],
+ "claim_owner": recovery["claim_owner"],
+ "attempt_metadata_sha256": recovery[
+ "attempt_metadata_sha256"
+ ],
+ }
+ )[:32]
+ source = _clean(recovery["recovery_source"])
+ if (
+ metadata
+ or attempt["status"] != "expired"
+ or _clean(attempt["error"])
+ != PROCESS_LOSS_INTERRUPTION_ERROR
+ or attempt["completed_at"] is None
+ or _clean(attempt["claim_token"])
+ != _clean(recovery["claim_token"])
+ or _clean(attempt["claim_owner"])
+ != _clean(recovery["claim_owner"])
+ or int(attempt["created_at"])
+ != int(recovery["attempt_created_at"])
+ or hashlib.sha256(raw_metadata.encode("utf-8")).hexdigest()
+ != _clean(recovery["attempt_metadata_sha256"])
+ or hashlib.sha256(
+ PROCESS_LOSS_INTERRUPTION_ERROR.encode("utf-8")
+ ).hexdigest()
+ != _clean(recovery["interruption_error_sha256"])
+ or _clean(recovery["external_call_outcome"]) != "uncertain"
+ or int(recovery["potential_duplicate_physical_calls_min"])
+ != 0
+ or int(recovery["potential_duplicate_physical_calls_max"])
+ != SLOW_PROCESS_LOSS_PHYSICAL_CALLS_MAX
+ or int(job["attempts"])
+ < int(recovery["job_attempts_before"]) + 1
+ or _clean(recovery["recovery_id"]) != expected_recovery_id
+ or source
+ not in {"expired_claimed_started", "legacy_expired_failed"}
+ or (
+ source == "expired_claimed_started"
+ and recovery["lease_expires_at"] is None
+ )
+ or (
+ source == "legacy_expired_failed"
+ and recovery["lease_expires_at"] is not None
+ )
+ ):
+ raise AuditError(
+ "process-loss recovery contract is invalid"
+ )
+ process_loss_recoveries = len(process_loss_rows)
+ process_loss_potential_min = sum(
+ int(row["potential_duplicate_physical_calls_min"])
+ for row in process_loss_rows
+ )
+ process_loss_potential_max = sum(
+ int(row["potential_duplicate_physical_calls_max"])
+ for row in process_loss_rows
+ )
+ result["route_counts"] = dict(sorted(route_counts.items()))
+ result["usage"] = usage
+ result["zero_call_configuration_recoveries"] = zero_call_recoveries
+ result["zero_call_promotion_recoveries"] = (
+ zero_call_promotion_recoveries
+ )
+ result["zero_call_projection_recoveries"] = (
+ zero_call_projection_recoveries
+ )
+ result["process_loss_recoveries"] = process_loss_recoveries
+ result["process_loss_unknown_external_outcomes"] = (
+ process_loss_recoveries
+ )
+ result["process_loss_potential_duplicate_physical_calls_min"] = (
+ process_loss_potential_min
+ )
+ result["process_loss_potential_duplicate_physical_calls_max"] = (
+ process_loss_potential_max
+ )
+ result["promotion_coverage"] = promotion_coverage
+ return result
+
+
+def _saved_request_context(
+ metadata: Mapping[str, Any],
+) -> tuple[Mapping[str, Any], list[Mapping[str, Any]]]:
+ request = metadata.get("request")
+ if not isinstance(request, Mapping):
+ raise SlowGraphError("failed attempt has no saved request")
+ messages = request.get("messages")
+ if not isinstance(messages, list):
+ raise SlowGraphError("failed attempt request has no messages")
+ user_messages = [
+ item
+ for item in messages
+ if isinstance(item, Mapping) and item.get("role") == "user"
+ ]
+ if len(user_messages) != 1:
+ raise SlowGraphError("failed attempt request must have exactly one user message")
+ try:
+ payload = json.loads(_required_text(user_messages[0].get("content"), "saved user content"))
+ except json.JSONDecodeError as exc:
+ raise SlowGraphError("failed attempt user content is not JSON") from exc
+ if not isinstance(payload, Mapping) or set(payload) != {"region", "capsules"}:
+ raise SlowGraphError("failed attempt user payload has drifted")
+ region = payload.get("region")
+ capsules = payload.get("capsules")
+ if not isinstance(region, Mapping) or not isinstance(capsules, list) or not all(
+ isinstance(item, Mapping) for item in capsules
+ ):
+ raise SlowGraphError("failed attempt request context is invalid")
+ return region, capsules
+
+
+class _FailedRawPatchReplayManager:
+ def __init__(
+ self,
+ *,
+ patch: Mapping[str, Any],
+ original_metadata: Mapping[str, Any],
+ original_attempt_id: str,
+ original_region: Mapping[str, Any],
+ original_capsules: list[Mapping[str, Any]],
+ transport_normalizations: list[dict[str, str]],
+ controller_processing: Mapping[str, Any],
+ context_source: str,
+ evidence_snapshot_sha256: str,
+ revalidation_route: str = "raw_response_revalidation",
+ revalidation_reason: str = "unambiguous_transport_normalization",
+ revalidation_details: Mapping[str, Any] | None = None,
+ ) -> None:
+ model = _required_text(original_metadata.get("model"), "saved model")
+ self.model_config = {"model": model}
+ self.prompt_hash = _digest(original_metadata.get("request", {}).get("messages", []))
+ self.last_call_metadata: Mapping[str, Any] = {}
+ self._patch = dict(patch)
+ self._original_metadata = dict(original_metadata)
+ self._original_attempt_id = original_attempt_id
+ self._original_region = dict(original_region)
+ self._original_capsules = [dict(item) for item in original_capsules]
+ self._transport_normalizations = list(transport_normalizations)
+ self._controller_processing = dict(controller_processing)
+ self._context_source = context_source
+ self._evidence_snapshot_sha256 = evidence_snapshot_sha256
+ self._revalidation_route = _required_text(
+ revalidation_route, "revalidation route"
+ )
+ self._revalidation_reason = _required_text(
+ revalidation_reason, "revalidation reason"
+ )
+ self._revalidation_details = dict(revalidation_details or {})
+ self._used = False
+
+ @staticmethod
+ def _capsule_identity(capsules: list[Mapping[str, Any]]) -> list[tuple[str, Any, str]]:
+ return [
+ (
+ _clean(item.get("capsule_id")),
+ item.get("revision"),
+ _clean(item.get("status")),
+ )
+ for item in capsules
+ ]
+
+ def propose(
+ self, region: Mapping[str, Any], capsules: list[Mapping[str, Any]]
+ ) -> Mapping[str, Any]:
+ if self._used:
+ raise SlowGraphError("saved raw response replay may be consumed only once")
+ if _clean(region.get("region_key")) != _clean(self._original_region.get("region_key")):
+ raise SlowGraphError("slow graph region changed before raw response replay")
+ if self._capsule_identity(capsules) != self._capsule_identity(
+ self._original_capsules
+ ):
+ raise SlowGraphError("slow graph capsules changed before raw response replay")
+ original_evidence = {
+ _leaf_id(item)
+ for item in self._original_region.get("evidence", [])
+ if isinstance(item, Mapping)
+ }
+ current_evidence = {
+ _leaf_id(item)
+ for item in region.get("evidence", [])
+ if isinstance(item, Mapping)
+ }
+ if not original_evidence.issubset(current_evidence):
+ raise SlowGraphError("slow graph evidence changed before raw response replay")
+ self._used = True
+ raw_response = _required_text(
+ self._original_metadata.get("raw_response"), "saved raw response"
+ )
+ self.last_call_metadata = {
+ "route": self._revalidation_route,
+ "route_reason": self._revalidation_reason,
+ "status": "completed",
+ "evidence_binding_contract_version": (
+ SLOW_EVIDENCE_BINDING_CONTRACT_VERSION
+ ),
+ "physical_api_call": False,
+ "physical_api_calls": 0,
+ "api_provider": self._original_metadata.get("api_provider"),
+ "model": self.model_config["model"],
+ "prompt_version": self._original_metadata.get("prompt_version"),
+ "attempt_count": 0,
+ "original_attempt_id": self._original_attempt_id,
+ "original_physical_call_id": self._original_metadata.get(
+ "physical_call_id"
+ ),
+ "original_physical_api_calls": int(
+ self._original_metadata.get("physical_api_calls", 0) or 0
+ ),
+ "original_attempt_count": int(
+ self._original_metadata.get("attempt_count", 0) or 0
+ ),
+ "original_call_metadata_sha256": _digest(self._original_metadata),
+ "raw_response_sha256": hashlib.sha256(
+ raw_response.encode("utf-8")
+ ).hexdigest(),
+ "normalized_patch_sha256": _digest(self._patch),
+ "transport_normalizations": self._transport_normalizations,
+ "revalidation_context_source": self._context_source,
+ "evidence_snapshot_sha256": self._evidence_snapshot_sha256,
+ "revalidation_details": self._revalidation_details,
+ **self._controller_processing,
+ }
+ return self._patch
+
+
+def _saved_response_patch(
+ metadata: Mapping[str, Any], *, label: str
+) -> tuple[Mapping[str, Any], str, str]:
+ raw_response_text = _required_text(
+ metadata.get("raw_response"), f"{label} raw response"
+ )
+ content = _required_text(metadata.get("content"), f"{label} response content")
+ try:
+ raw_response = json.loads(raw_response_text)
+ raw_patch = json.loads(content)
+ except json.JSONDecodeError as exc:
+ raise SlowGraphError(f"{label} response is not valid JSON") from exc
+ choices = raw_response.get("choices") if isinstance(raw_response, Mapping) else None
+ choice = (
+ choices[0]
+ if isinstance(choices, list)
+ and len(choices) == 1
+ and isinstance(choices[0], Mapping)
+ else None
+ )
+ message = choice.get("message") if isinstance(choice, Mapping) else None
+ if (
+ not isinstance(message, Mapping)
+ or _clean(message.get("content")) != content
+ or _clean(choice.get("finish_reason")) != "stop"
+ ):
+ raise SlowGraphError(f"{label} response envelope does not match saved content")
+ if not isinstance(raw_patch, Mapping):
+ raise SlowGraphError(f"{label} GraphPatch is not an object")
+ return raw_patch, raw_response_text, content
+
+
+def _prepare_revalidated_patch(
+ raw_patch: Mapping[str, Any],
+ *,
+ current_region: Mapping[str, Any],
+ current_capsules: list[Mapping[str, Any]],
+ original_region: Mapping[str, Any],
+ original_capsules: list[Mapping[str, Any]],
+ metadata: Mapping[str, Any],
+ validation_route: str,
+ context_source: str,
+ transport_policy: str,
+) -> tuple[dict[str, Any], list[dict[str, str]], dict[str, Any]]:
+ normalized_patch, normalizations = _normalize_transport_patch(
+ raw_patch, original_capsules, original_region
+ )
+ if transport_policy == "required" and not normalizations:
+ raise SlowGraphError("saved GraphPatch has no approved transport normalization")
+ if transport_policy == "forbidden" and normalizations:
+ raise SlowGraphError(
+ "semantic-policy revalidation cannot include transport normalization"
+ )
+ if transport_policy not in {"required", "forbidden"}:
+ raise SlowGraphError("saved GraphPatch transport policy is invalid")
+ validate_patch(normalized_patch)
+ _validate_generic_create_partition_keys(
+ current_region.get("region_key"), normalized_patch
+ )
+ if context_source != "saved_request" and normalized_patch != {
+ "operations": [{"action": "noop"}]
+ }:
+ raise SlowGraphError(
+ "legacy response without saved request may revalidate only a pure noop"
+ )
+ _, required_ids = _required_promotion_ids(current_region, current_capsules)
+ partition_capsule_ids = {
+ _clean(item)
+ for item in metadata.get("semantic_partition_capsule_ids") or ()
+ if _clean(item)
+ }
+ TieredGraphPatchManager._validate_route_actions(
+ validation_route,
+ normalized_patch,
+ original_capsules,
+ required_evidence_ids=required_ids,
+ partition_capsule_ids=partition_capsule_ids,
+ )
+ _validate_claim_evidence_contract(
+ current_region,
+ current_capsules,
+ normalized_patch,
+ route=validation_route,
+ )
+ merge_audit: dict[str, Any] | None = None
+ if validation_route == "flash" and required_ids:
+ _validate_flash_delta_patch(
+ normalized_patch, current_capsules, required_ids
+ )
+ if current_capsules:
+ committed_patch, merge_audit = _merge_flash_delta_patch(
+ normalized_patch, current_capsules
+ )
+ else:
+ committed_patch = _materialize_lossless_summaries(normalized_patch)
+ else:
+ committed_patch = _materialize_lossless_summaries(normalized_patch)
+ validate_patch(committed_patch, require_lossless_summary=True)
+ _validate_generic_create_partition_keys(
+ current_region.get("region_key"), committed_patch
+ )
+ TieredGraphPatchManager._validate_route_actions(
+ validation_route,
+ committed_patch,
+ current_capsules,
+ required_evidence_ids=required_ids,
+ partition_capsule_ids=partition_capsule_ids,
+ )
+ _validate_claim_evidence_contract(
+ current_region,
+ current_capsules,
+ committed_patch,
+ route=validation_route,
+ )
+ _validate_promotion_patch(
+ current_region,
+ current_capsules,
+ committed_patch,
+ required_evidence_ids=required_ids,
+ )
+ controller_processing: dict[str, Any] = {
+ "controller_summary_materialization": {
+ "schema_version": SLOW_SUMMARY_CONTRACT_VERSION,
+ "model_patch_sha256": _digest(normalized_patch),
+ "committed_patch_sha256": _digest(committed_patch),
+ }
+ }
+ if merge_audit is not None:
+ controller_processing["controller_delta_merge"] = merge_audit
+ return committed_patch, normalizations, controller_processing
+
+
+def revalidate_failed_raw_response(
+ store: V4SlowGraphStore, job_id: str
+) -> str:
+ """Apply a saved, complete API response after a narrow transport normalization."""
+ with store.connection() as con:
+ job = con.execute(
+ "SELECT * FROM slow_graph_jobs WHERE job_id=?", (job_id,)
+ ).fetchone()
+ if job is None or job["status"] != "failed" or job["claim_token"] is not None:
+ raise SlowGraphError("raw response revalidation requires one unclaimed failed job")
+ attempts = con.execute(
+ "SELECT rowid,* FROM slow_graph_attempts WHERE job_id=? AND status='failed' "
+ "ORDER BY created_at DESC,rowid DESC",
+ (job_id,),
+ ).fetchall()
+ attempt_count = int(
+ con.execute(
+ "SELECT count(*) FROM slow_graph_attempts WHERE job_id=?", (job_id,)
+ ).fetchone()[0]
+ )
+ patch_count = int(
+ con.execute(
+ "SELECT count(*) FROM slow_graph_patches WHERE job_id=?", (job_id,)
+ ).fetchone()[0]
+ )
+ evidence_ids = _v3._strict_json(
+ job["evidence_ids_json"], label="job evidence IDs", expected=list
+ )
+ current_region = {
+ "region_key": job["region_key"],
+ "evidence": store._evidence(con, job["scope_id"], evidence_ids),
+ }
+ current_capsules = store._capsules(
+ con, job["scope_id"], job["region_key"]
+ )
+ if patch_count != 0:
+ raise SlowGraphError("failed job already has an applied patch")
+ attempt = None
+ metadata: dict[str, Any] | None = None
+ for candidate in attempts:
+ candidate_metadata = _v3._strict_json(
+ candidate["call_metadata_json"],
+ label="failed call metadata",
+ expected=dict,
+ )
+ if (
+ candidate_metadata.get("physical_api_call") is True
+ and int(candidate_metadata.get("physical_api_calls", 0) or 0) == 1
+ and candidate_metadata.get("status") in {"response_received", "completed"}
+ and int(candidate_metadata.get("http_status", 0) or 0) == 200
+ and candidate_metadata.get("finish_reason") == "stop"
+ and _clean(candidate_metadata.get("raw_response"))
+ and _clean(candidate_metadata.get("content"))
+ ):
+ attempt = candidate
+ metadata = candidate_metadata
+ break
+ if attempt is None or metadata is None or not _clean(attempt["error"]):
+ raise SlowGraphError(
+ "failed job has no complete HTTP 200 physical response to revalidate"
+ )
+ raw_patch, _, _ = _saved_response_patch(metadata, label="saved failed")
+ context_source = "saved_request"
+ if isinstance(metadata.get("request"), Mapping):
+ original_region, original_capsules = _saved_request_context(metadata)
+ else:
+ if attempt_count != 1 or patch_count != 0 or current_capsules:
+ raise SlowGraphError(
+ "legacy response without saved request is not a pristine capsule-free job"
+ )
+ original_region = current_region
+ original_capsules = []
+ context_source = "immutable_job_snapshot_for_legacy_noop"
+ route = _required_text(metadata.get("route"), "saved route")
+ if route not in {"flash", "pro", "flash_to_pro"}:
+ raise SlowGraphError("saved route is not an API GraphPatch route")
+ validation_route = "pro" if route in {"pro", "flash_to_pro"} else "flash"
+ committed_patch, normalizations, controller_processing = (
+ _prepare_revalidated_patch(
+ raw_patch,
+ current_region=current_region,
+ current_capsules=current_capsules,
+ original_region=original_region,
+ original_capsules=original_capsules,
+ metadata=metadata,
+ validation_route=validation_route,
+ context_source=context_source,
+ transport_policy="required",
+ )
+ )
+ manager = _FailedRawPatchReplayManager(
+ patch=committed_patch,
+ original_metadata=metadata,
+ original_attempt_id=str(attempt["attempt_id"]),
+ original_region=original_region,
+ original_capsules=original_capsules,
+ transport_normalizations=normalizations,
+ controller_processing=controller_processing,
+ context_source=context_source,
+ evidence_snapshot_sha256=_digest(evidence_ids),
+ )
+ store.resume(job_id)
+ return store.run_job(job_id, manager)
+
+
+def _semantic_policy_failure_class(error: str) -> str:
+ generic_pattern = re.compile(
+ r"operations\[\d+\]\." + re.escape(GENERIC_MULTI_SLOT_CAPSULE_KEY_ERROR)
+ )
+ if generic_pattern.fullmatch(error):
+ return "generic_capsule_key_policy"
+ if error.startswith(LEGACY_SINGLE_BINDING_ERROR_PREFIX):
+ raw_ids = error[len(LEGACY_SINGLE_BINDING_ERROR_PREFIX) :]
+ try:
+ evidence_ids = json.loads(raw_ids)
+ except json.JSONDecodeError:
+ return ""
+ if (
+ isinstance(evidence_ids, list)
+ and evidence_ids
+ and all(isinstance(item, str) and item.strip() for item in evidence_ids)
+ and len(evidence_ids) == len(set(evidence_ids))
+ ):
+ return "compound_support_binding_policy"
+ cross_slot_match = re.fullmatch(
+ r"claim support canonical slot mismatch: "
+ r"claim=(?P\S+) evidence=(?P\S+) id=(?P\S+)",
+ error,
+ )
+ if cross_slot_match and cross_slot_match.group("evidence").startswith(
+ cross_slot_match.group("claim") + "."
+ ):
+ return "complementary_subslot_support_policy"
+ return ""
+
+
+def revalidate_failed_semantic_policy_response(
+ store: V4SlowGraphStore, job_id: str
+) -> str:
+ """Replay the final complete two-call Pro result after a reviewed policy change."""
+ with store.connection() as con:
+ job = con.execute(
+ "SELECT * FROM slow_graph_jobs WHERE job_id=?", (job_id,)
+ ).fetchone()
+ if (
+ job is None
+ or job["status"] != "failed"
+ or job["claim_token"] is not None
+ or int(job["attempts"] or 0) != 1
+ ):
+ raise SlowGraphError(
+ "semantic-policy revalidation requires one unclaimed failed job"
+ )
+ attempts = con.execute(
+ "SELECT * FROM slow_graph_attempts WHERE job_id=? "
+ "ORDER BY created_at,attempt_id",
+ (job_id,),
+ ).fetchall()
+ patch_count = int(
+ con.execute(
+ "SELECT count(*) FROM slow_graph_patches WHERE job_id=?", (job_id,)
+ ).fetchone()[0]
+ )
+ evidence_ids = _v3._strict_json(
+ job["evidence_ids_json"], label="job evidence IDs", expected=list
+ )
+ current_region = {
+ "region_key": job["region_key"],
+ "evidence": store._evidence(con, job["scope_id"], evidence_ids),
+ }
+ current_capsules = store._capsules(
+ con, job["scope_id"], job["region_key"]
+ )
+ if len(attempts) != 1 or patch_count != 0:
+ raise SlowGraphError(
+ "semantic-policy revalidation requires one attempt and no patch"
+ )
+ attempt = attempts[0]
+ raw_metadata = _required_text(
+ attempt["call_metadata_json"], "failed call metadata"
+ )
+ metadata = _v3._strict_json(
+ raw_metadata, label="failed call metadata", expected=dict
+ )
+ error = _clean(attempt["error"])
+ policy_failure_class = _semantic_policy_failure_class(error)
+ initial_hash = _clean(metadata.get("initial_rejected_patch_sha256"))
+ corrected_hash = _clean(metadata.get("corrected_patch_sha256"))
+ tier_calls = metadata.get("tier_calls")
+ if (
+ attempt["status"] != "failed"
+ or not policy_failure_class
+ or _clean(job["last_error"]) != error
+ or _clean(metadata.get("semantic_correction_validation_error")) != error
+ or _clean(metadata.get("semantic_correction_rejection_error")) != error
+ or metadata.get("semantic_correction_attempted") is not True
+ or metadata.get("semantic_correction_applied") is not False
+ or metadata.get("status") != "semantic_correction_rejected"
+ or metadata.get("route") != "pro"
+ or metadata.get("prompt_version") != SLOW_PROMPT_VERSION
+ or metadata.get("physical_api_call") is not True
+ or int(metadata.get("physical_api_calls", 0) or 0) != 2
+ or int(metadata.get("attempt_count", 0) or 0) != 2
+ or int(metadata.get("http_status", 0) or 0) != 200
+ or metadata.get("finish_reason") != "stop"
+ or len(initial_hash) != 64
+ or len(corrected_hash) != 64
+ or not isinstance(tier_calls, list)
+ or len(tier_calls) != 2
+ ):
+ raise SlowGraphError(
+ "failed attempt is not a reviewed complete two-call Pro policy failure"
+ )
+
+ call_hashes = (initial_hash, corrected_hash)
+ call_ids: list[str] = []
+ for index, (call, expected_hash) in enumerate(
+ zip(tier_calls, call_hashes, strict=True)
+ ):
+ if not isinstance(call, Mapping):
+ raise SlowGraphError("semantic-policy tier call metadata is invalid")
+ expected_stage = "initial_pro" if index == 0 else "semantic_correction"
+ request = call.get("request")
+ call_id = _clean(call.get("physical_call_id"))
+ if (
+ call.get("tier_stage") != expected_stage
+ or call.get("route") != "pro"
+ or call.get("prompt_version") != SLOW_PROMPT_VERSION
+ or call.get("physical_api_call") is not True
+ or int(call.get("physical_api_calls", 0) or 0) != 1
+ or int(call.get("attempt_count", 0) or 0) != 1
+ or call.get("status") != "completed"
+ or int(call.get("http_status", 0) or 0) != 200
+ or call.get("finish_reason") != "stop"
+ or not call_id
+ or not isinstance(request, Mapping)
+ or _clean(call.get("request_sha256")) != _digest(request)
+ ):
+ raise SlowGraphError(
+ "semantic-policy tier call is not one complete durable Pro response"
+ )
+ call_patch, _, _ = _saved_response_patch(
+ call, label=f"semantic-policy tier call {index}"
+ )
+ if _digest(call_patch) != expected_hash:
+ raise SlowGraphError("semantic-policy tier call patch hash differs")
+ call_ids.append(call_id)
+ if len(set(call_ids)) != 2:
+ raise SlowGraphError("semantic-policy physical call IDs are not unique")
+
+ raw_patch, _, _ = _saved_response_patch(
+ metadata, label="semantic-policy corrected"
+ )
+ if _digest(raw_patch) != corrected_hash:
+ raise SlowGraphError("semantic-policy corrected patch hash differs")
+ request = metadata.get("request")
+ if (
+ not isinstance(request, Mapping)
+ or _clean(metadata.get("request_sha256")) != _digest(request)
+ or request != tier_calls[-1].get("request")
+ ):
+ raise SlowGraphError("semantic-policy corrected request has drifted")
+ original_region, original_capsules = _saved_request_context(metadata)
+
+ def metadata_ids(name: str) -> set[str]:
+ raw = metadata.get(name)
+ if not isinstance(raw, list):
+ raise SlowGraphError(f"semantic-policy {name} is not a list")
+ values = [_required_text(item, name) for item in raw]
+ if len(values) != len(set(values)):
+ raise SlowGraphError(f"semantic-policy {name} contains duplicates")
+ return set(values)
+
+ current_evidence = [
+ item
+ for item in current_region.get("evidence", [])
+ if isinstance(item, Mapping)
+ ]
+ current_by_id = {_leaf_id(item): item for item in current_evidence}
+ current_ids = set(current_by_id)
+ current_eligible = {
+ evidence_id
+ for evidence_id, item in current_by_id.items()
+ if _is_current_durable(item)
+ }
+ current_challenged = {
+ evidence_id
+ for evidence_id, item in current_by_id.items()
+ if _is_challenged_durable(item)
+ }
+ current_uncertain = {
+ evidence_id
+ for evidence_id, item in current_by_id.items()
+ if _is_uncertain(item)
+ }
+ current_episodic = {
+ evidence_id
+ for evidence_id, item in current_by_id.items()
+ if _is_episodic(item)
+ }
+ current_inactive = (
+ current_ids
+ - current_eligible
+ - current_challenged
+ - current_uncertain
+ - current_episodic
+ )
+ current_visible = current_eligible | current_challenged
+ saved_evidence = {
+ _leaf_id(item): item
+ for item in original_region.get("evidence", [])
+ if isinstance(item, Mapping)
+ }
+ saved_evidence_ids = set(saved_evidence)
+ saved_required_ids = {
+ _required_text(item, "saved required evidence ID")
+ for item in original_region.get("required_evidence_ids", [])
+ }
+ metadata_required_ids = metadata_ids("required_operation_evidence_ids")
+ if (
+ _clean(original_region.get("region_key"))
+ != _clean(current_region.get("region_key"))
+ or current_ids != set(evidence_ids)
+ or current_eligible != metadata_ids("eligible_evidence_ids")
+ or current_challenged != metadata_ids("challenged_evidence_ids")
+ or current_uncertain != metadata_ids("uncertain_evidence_ids")
+ or current_episodic != metadata_ids("episodic_evidence_ids")
+ or current_inactive != metadata_ids("inactive_evidence_ids")
+ or (current_uncertain | current_episodic | current_inactive)
+ != metadata_ids("ignored_evidence_ids")
+ or saved_evidence_ids != current_visible
+ or saved_required_ids != metadata_required_ids
+ or not metadata_required_ids.issubset(current_visible)
+ or saved_evidence
+ != {
+ evidence_id: _public_leaf(current_by_id[evidence_id])
+ for evidence_id in sorted(current_visible)
+ }
+ or _FailedRawPatchReplayManager._capsule_identity(original_capsules)
+ != _FailedRawPatchReplayManager._capsule_identity(current_capsules)
+ ):
+ raise SlowGraphError("semantic-policy saved request context has drifted")
+ committed_patch, normalizations, controller_processing = (
+ _prepare_revalidated_patch(
+ raw_patch,
+ current_region=current_region,
+ current_capsules=current_capsules,
+ original_region=original_region,
+ original_capsules=original_capsules,
+ metadata=metadata,
+ validation_route="pro",
+ context_source="saved_request",
+ transport_policy="forbidden",
+ )
+ )
+ manager = _FailedRawPatchReplayManager(
+ patch=committed_patch,
+ original_metadata=metadata,
+ original_attempt_id=str(attempt["attempt_id"]),
+ original_region=original_region,
+ original_capsules=original_capsules,
+ transport_normalizations=normalizations,
+ controller_processing=controller_processing,
+ context_source="saved_request",
+ evidence_snapshot_sha256=_digest(evidence_ids),
+ revalidation_route="semantic_policy_revalidation",
+ revalidation_reason={
+ "generic_capsule_key_policy": (
+ "generic_capsule_key_structural_policy_narrowing"
+ ),
+ "compound_support_binding_policy": (
+ "compound_support_many_to_many_binding_policy"
+ ),
+ "complementary_subslot_support_policy": (
+ "controlled_parent_subslot_support_synthesis_policy"
+ ),
+ }[policy_failure_class],
+ revalidation_details={
+ "schema_version": "tmcra.v4.slow-semantic-policy-revalidation.1",
+ "policy_failure_class": policy_failure_class,
+ "evidence_binding_contract_version": (
+ SLOW_EVIDENCE_BINDING_CONTRACT_VERSION
+ ),
+ "failed_error_sha256": hashlib.sha256(
+ error.encode("utf-8")
+ ).hexdigest(),
+ "failed_call_metadata_sha256": hashlib.sha256(
+ raw_metadata.encode("utf-8")
+ ).hexdigest(),
+ "initial_rejected_patch_sha256": initial_hash,
+ "corrected_patch_sha256": corrected_hash,
+ "semantic_correction_changed_patch": initial_hash != corrected_hash,
+ "physical_call_ids": call_ids,
+ },
+ )
+ store.resume(job_id)
+ return store.run_job(job_id, manager)
+
+
+def resume_failed_model_validation(
+ store: V4SlowGraphStore,
+ job_id: str,
+ manager: "TieredGraphPatchManager",
+) -> str:
+ """Explicitly reopen one complete Pro response rejected by local validation."""
+ with store.connection() as con:
+ job = con.execute(
+ "SELECT * FROM slow_graph_jobs WHERE job_id=?", (job_id,)
+ ).fetchone()
+ if (
+ job is None
+ or job["status"] != "failed"
+ or job["claim_token"] is not None
+ or int(job["attempts"] or 0) != 1
+ ):
+ raise SlowGraphError(
+ "model-validation recovery requires one unclaimed failed attempt"
+ )
+ attempts = con.execute(
+ "SELECT * FROM slow_graph_attempts WHERE job_id=? "
+ "ORDER BY created_at,attempt_id",
+ (job_id,),
+ ).fetchall()
+ patch_count = int(
+ con.execute(
+ "SELECT count(*) FROM slow_graph_patches WHERE job_id=?", (job_id,)
+ ).fetchone()[0]
+ )
+ if len(attempts) != 1 or patch_count != 0:
+ raise SlowGraphError(
+ "model-validation recovery requires one attempt and no applied patch"
+ )
+ attempt = attempts[0]
+ metadata = _v3._strict_json(
+ _required_text(attempt["call_metadata_json"], "failed call metadata"),
+ label="failed call metadata",
+ expected=dict,
+ )
+ error = _clean(attempt["error"])
+ if (
+ attempt["status"] != "failed"
+ or not error
+ or _clean(job["last_error"]) != error
+ or metadata.get("physical_api_call") is not True
+ or int(metadata.get("physical_api_calls", 0) or 0) < 1
+ or _clean(metadata.get("route")) not in {"pro", "flash_to_pro"}
+ or _clean(metadata.get("status"))
+ not in {"completed", "response_received", "semantic_correction_rejected"}
+ or int(metadata.get("http_status", 0) or 0) != 200
+ or _clean(metadata.get("finish_reason")) != "stop"
+ or not _clean(metadata.get("raw_response"))
+ or not _clean(metadata.get("content"))
+ ):
+ raise SlowGraphError(
+ "failed attempt is not a complete Pro model-validation response"
+ )
+ store.resume(job_id)
+ return store.run_job(job_id, manager)
+
+
+def resume_failed_model_validation_after_prompt_migration(
+ store: V4SlowGraphStore,
+ job_id: str,
+ manager: "TieredGraphPatchManager",
+) -> str:
+ """Reopen one twice-rejected compound-leaf job after the reviewed prompt migration."""
+ with store.connection() as con:
+ job = con.execute(
+ "SELECT * FROM slow_graph_jobs WHERE job_id=?", (job_id,)
+ ).fetchone()
+ if (
+ job is None
+ or job["status"] != "failed"
+ or job["claim_token"] is not None
+ or int(job["attempts"] or 0) != 2
+ ):
+ raise SlowGraphError(
+ "prompt-migration recovery requires two unclaimed failed attempts"
+ )
+ attempts = con.execute(
+ "SELECT * FROM slow_graph_attempts WHERE job_id=? "
+ "ORDER BY created_at,attempt_id",
+ (job_id,),
+ ).fetchall()
+ patch_count = int(
+ con.execute(
+ "SELECT count(*) FROM slow_graph_patches WHERE job_id=?", (job_id,)
+ ).fetchone()[0]
+ )
+ if len(attempts) != 2 or patch_count != 0:
+ raise SlowGraphError(
+ "prompt-migration recovery requires two attempts and no applied patch"
+ )
+ duplicate_evidence_error = (
+ "atomic Fast evidence may belong to only one resulting claim: "
+ )
+ attempt_prompt_versions: list[str] = []
+ for attempt in attempts:
+ metadata = _v3._strict_json(
+ _required_text(attempt["call_metadata_json"], "failed call metadata"),
+ label="failed call metadata",
+ expected=dict,
+ )
+ if (
+ attempt["status"] != "failed"
+ or not _clean(attempt["error"]).startswith(duplicate_evidence_error)
+ or metadata.get("physical_api_call") is not True
+ or int(metadata.get("physical_api_calls", 0) or 0) < 1
+ or _clean(metadata.get("route")) not in {"pro", "flash_to_pro"}
+ or _clean(metadata.get("status")) != "semantic_correction_rejected"
+ or int(metadata.get("http_status", 0) or 0) != 200
+ or _clean(metadata.get("finish_reason")) != "stop"
+ or _clean(metadata.get("prompt_version"))
+ not in SLOW_PROMPT_MIGRATION_SOURCE_VERSIONS
+ or not _clean(metadata.get("raw_response"))
+ or not _clean(metadata.get("content"))
+ ):
+ raise SlowGraphError(
+ "failed attempts are not the reviewed compound-leaf prompt-migration class"
+ )
+ attempt_prompt_versions.append(_clean(metadata.get("prompt_version")))
+ if attempt_prompt_versions[-1] != SLOW_PROMPT_MIGRATION_SOURCE_VERSION:
+ raise SlowGraphError(
+ "prompt-migration recovery requires the final failed attempt on the source version"
+ )
+ if _clean(job["last_error"]) != _clean(attempts[-1]["error"]):
+ raise SlowGraphError("prompt-migration job and final attempt errors differ")
+ if SLOW_PROMPT_VERSION == SLOW_PROMPT_MIGRATION_SOURCE_VERSION:
+ raise SlowGraphError("prompt-migration recovery requires a new prompt version")
+ store.resume(job_id)
+ return store.run_job(job_id, manager)
+
+
+def resume_zero_call_configuration_failure(
+ store: V4SlowGraphStore, job_id: str
+) -> dict[str, Any]:
+ """Reopen one preflight configuration failure proven to have made no call."""
+ with store.connection() as con:
+ con.execute("BEGIN IMMEDIATE")
+ job = con.execute(
+ "SELECT * FROM slow_graph_jobs WHERE job_id=?", (job_id,)
+ ).fetchone()
+ if (
+ job is None
+ or job["status"] != "failed"
+ or job["claim_token"] is not None
+ or int(job["attempts"] or 0) != 1
+ ):
+ raise SlowGraphError(
+ "zero-call configuration recovery requires one unclaimed failed job"
+ )
+ attempts = con.execute(
+ "SELECT * FROM slow_graph_attempts WHERE job_id=? ORDER BY created_at,attempt_id",
+ (job_id,),
+ ).fetchall()
+ patch_count = int(
+ con.execute(
+ "SELECT count(*) FROM slow_graph_patches WHERE job_id=?", (job_id,)
+ ).fetchone()[0]
+ )
+ if len(attempts) != 1 or patch_count != 0:
+ raise SlowGraphError(
+ "zero-call configuration recovery requires one attempt and no patch"
+ )
+ attempt = attempts[0]
+ raw_metadata = _required_text(
+ attempt["call_metadata_json"], "failed call metadata"
+ )
+ metadata = _v3._strict_json(
+ raw_metadata, label="failed call metadata", expected=dict
+ )
+ route = _clean(metadata.get("route"))
+ error = _clean(attempt["error"])
+ usage = metadata.get("usage")
+ zero_usage = isinstance(usage, Mapping) and all(
+ int(value or 0) == 0 for value in usage.values()
+ )
+ if (
+ attempt["status"] != "failed"
+ or error != ZERO_CALL_CONFIGURATION_ERRORS.get(route)
+ or _clean(job["last_error"]) != error
+ or metadata.get("physical_api_call") is not False
+ or int(metadata.get("physical_api_calls", -1)) != 0
+ or int(metadata.get("attempt_count", -1)) != 0
+ or _clean(metadata.get("status")) != "unavailable"
+ or _clean(metadata.get("physical_call_id"))
+ or _clean(metadata.get("raw_response"))
+ or _clean(metadata.get("content"))
+ or not zero_usage
+ ):
+ raise SlowGraphError(
+ "failed attempt is not a proven zero-call configuration failure"
+ )
+ error_sha256 = hashlib.sha256(error.encode("utf-8")).hexdigest()
+ metadata_sha256 = hashlib.sha256(raw_metadata.encode("utf-8")).hexdigest()
+ recovery_id = "sgz_" + _digest(
+ {
+ "job_id": job_id,
+ "attempt_id": attempt["attempt_id"],
+ "error_sha256": error_sha256,
+ "call_metadata_sha256": metadata_sha256,
+ }
+ )[:32]
+ created_at = _v3._now()
+ con.execute(
+ """
+ CREATE TABLE IF NOT EXISTS slow_graph_zero_call_recoveries(
+ recovery_id TEXT PRIMARY KEY,
+ job_id TEXT NOT NULL UNIQUE,
+ attempt_id TEXT NOT NULL UNIQUE,
+ scope_id TEXT NOT NULL,
+ route TEXT NOT NULL,
+ error_sha256 TEXT NOT NULL,
+ call_metadata_sha256 TEXT NOT NULL,
+ physical_api_calls INTEGER NOT NULL,
+ created_at INTEGER NOT NULL
+ )
+ """
+ )
+ con.execute(
+ "INSERT INTO slow_graph_zero_call_recoveries VALUES(?,?,?,?,?,?,?,?,?)",
+ (
+ recovery_id,
+ job_id,
+ attempt["attempt_id"],
+ job["scope_id"],
+ route,
+ error_sha256,
+ metadata_sha256,
+ 0,
+ created_at,
+ ),
+ )
+ reopened = con.execute(
+ "UPDATE slow_graph_jobs SET status='pending',last_error='',updated_at=?,"
+ "claim_token=NULL,claim_owner=NULL,lease_expires_at=NULL "
+ "WHERE job_id=? AND status='failed' AND claim_token IS NULL",
+ (created_at, job_id),
+ )
+ if reopened.rowcount != 1:
+ raise SlowGraphError(
+ "slow graph job changed during zero-call configuration recovery"
+ )
+ return {
+ "schema_version": "tmcra.v4.slow-zero-call-recovery.1",
+ "recovery_id": recovery_id,
+ "job_id": job_id,
+ "attempt_id": str(attempt["attempt_id"]),
+ "scope_id": str(job["scope_id"]),
+ "route": route,
+ "error_sha256": error_sha256,
+ "call_metadata_sha256": metadata_sha256,
+ "physical_api_calls": 0,
+ "status": "pending",
+ "created_at": created_at,
+ }
+
+
+def resume_zero_call_promotion_failure(
+ store: V4SlowGraphStore, job_id: str
+) -> dict[str, Any]:
+ """Reopen the reviewed no-op routing bug without hiding any model failure."""
+ with store.connection() as con:
+ con.execute("BEGIN IMMEDIATE")
+ job = con.execute(
+ "SELECT * FROM slow_graph_jobs WHERE job_id=?", (job_id,)
+ ).fetchone()
+ if (
+ job is None
+ or job["status"] != "failed"
+ or job["claim_token"] is not None
+ or int(job["attempts"] or 0) != 1
+ ):
+ raise SlowGraphError(
+ "zero-call promotion recovery requires one unclaimed failed job"
+ )
+ attempts = con.execute(
+ "SELECT * FROM slow_graph_attempts WHERE job_id=? "
+ "ORDER BY created_at,attempt_id",
+ (job_id,),
+ ).fetchall()
+ patch_count = int(
+ con.execute(
+ "SELECT count(*) FROM slow_graph_patches WHERE job_id=?", (job_id,)
+ ).fetchone()[0]
+ )
+ if len(attempts) != 1 or patch_count != 0:
+ raise SlowGraphError(
+ "zero-call promotion recovery requires one attempt and no patch"
+ )
+ attempt = attempts[0]
+ raw_metadata = _required_text(
+ attempt["call_metadata_json"], "failed call metadata"
+ )
+ metadata = _v3._strict_json(
+ raw_metadata, label="failed call metadata", expected=dict
+ )
+ error = _clean(attempt["error"])
+ eligible_ids = metadata.get("eligible_evidence_ids")
+ challenged_ids = metadata.get("challenged_evidence_ids")
+ delta_ids = metadata.get("delta_evidence_ids")
+ usage = metadata.get("usage")
+ zero_usage = isinstance(usage, Mapping) and all(
+ int(value or 0) == 0 for value in usage.values()
+ )
+ if (
+ attempt["status"] != "failed"
+ or not error.startswith(
+ "noop cannot consume uncited current durable Fast evidence: "
+ )
+ or _clean(job["last_error"]) != error
+ or _clean(metadata.get("route")) != "deterministic_noop"
+ or _clean(metadata.get("route_reason"))
+ != "new capsule blocked by unresolved fast challenge"
+ or metadata.get("physical_api_call") is not False
+ or int(metadata.get("physical_api_calls", -1)) != 0
+ or int(metadata.get("attempt_count", -1)) != 0
+ or _clean(metadata.get("physical_call_id"))
+ or _clean(metadata.get("raw_response"))
+ or _clean(metadata.get("content"))
+ or not zero_usage
+ or not isinstance(eligible_ids, list)
+ or not eligible_ids
+ or not isinstance(challenged_ids, list)
+ or not challenged_ids
+ or not isinstance(delta_ids, list)
+ or not set(eligible_ids) <= set(delta_ids)
+ ):
+ raise SlowGraphError(
+ "failed attempt is not the proven zero-call promotion routing failure"
+ )
+ error_sha256 = hashlib.sha256(error.encode("utf-8")).hexdigest()
+ metadata_sha256 = hashlib.sha256(raw_metadata.encode("utf-8")).hexdigest()
+ recovery_id = "sgp_" + _digest(
+ {
+ "job_id": job_id,
+ "attempt_id": attempt["attempt_id"],
+ "error_sha256": error_sha256,
+ "call_metadata_sha256": metadata_sha256,
+ }
+ )[:32]
+ created_at = _v3._now()
+ con.execute(
+ """
+ CREATE TABLE IF NOT EXISTS slow_graph_zero_call_promotion_recoveries(
+ recovery_id TEXT PRIMARY KEY,
+ job_id TEXT NOT NULL UNIQUE,
+ attempt_id TEXT NOT NULL UNIQUE,
+ scope_id TEXT NOT NULL,
+ error_sha256 TEXT NOT NULL,
+ call_metadata_sha256 TEXT NOT NULL,
+ physical_api_calls INTEGER NOT NULL,
+ created_at INTEGER NOT NULL
+ )
+ """
+ )
+ con.execute(
+ "INSERT INTO slow_graph_zero_call_promotion_recoveries "
+ "VALUES(?,?,?,?,?,?,?,?)",
+ (
+ recovery_id,
+ job_id,
+ attempt["attempt_id"],
+ job["scope_id"],
+ error_sha256,
+ metadata_sha256,
+ 0,
+ created_at,
+ ),
+ )
+ reopened = con.execute(
+ "UPDATE slow_graph_jobs SET status='pending',last_error='',updated_at=?,"
+ "claim_token=NULL,claim_owner=NULL,lease_expires_at=NULL "
+ "WHERE job_id=? AND status='failed' AND claim_token IS NULL",
+ (created_at, job_id),
+ )
+ if reopened.rowcount != 1:
+ raise SlowGraphError(
+ "slow graph job changed during zero-call promotion recovery"
+ )
+ return {
+ "schema_version": "tmcra.v4.slow-zero-call-promotion-recovery.1",
+ "recovery_id": recovery_id,
+ "job_id": job_id,
+ "attempt_id": str(attempt["attempt_id"]),
+ "scope_id": str(job["scope_id"]),
+ "error_sha256": error_sha256,
+ "call_metadata_sha256": metadata_sha256,
+ "physical_api_calls": 0,
+ "status": "pending",
+ "created_at": created_at,
+ }
+
+
+def resume_zero_call_projection_failure(
+ store: V4SlowGraphStore, job_id: str
+) -> dict[str, Any]:
+ """Reopen one zero-call failure caused only by an empty internal origin field."""
+ with store.connection() as con:
+ con.execute("BEGIN IMMEDIATE")
+ job = con.execute(
+ "SELECT * FROM slow_graph_jobs WHERE job_id=?", (job_id,)
+ ).fetchone()
+ if (
+ job is None
+ or job["status"] != "failed"
+ or job["claim_token"] is not None
+ or int(job["attempts"] or 0) != 1
+ ):
+ raise SlowGraphError(
+ "zero-call projection recovery requires one unclaimed failed job"
+ )
+ attempts = con.execute(
+ "SELECT * FROM slow_graph_attempts WHERE job_id=? ORDER BY created_at,attempt_id",
+ (job_id,),
+ ).fetchall()
+ patch_count = int(
+ con.execute(
+ "SELECT count(*) FROM slow_graph_patches WHERE job_id=?", (job_id,)
+ ).fetchone()[0]
+ )
+ if len(attempts) != 1 or patch_count != 0:
+ raise SlowGraphError(
+ "zero-call projection recovery requires one attempt and no patch"
+ )
+ attempt = attempts[0]
+ raw_metadata = _required_text(
+ attempt["call_metadata_json"], "failed call metadata"
+ )
+ metadata = _v3._strict_json(
+ raw_metadata, label="failed call metadata", expected=dict
+ )
+ error = _clean(attempt["error"])
+ if (
+ attempt["status"] != "failed"
+ or not re.fullmatch(
+ r"benchmark field is forbidden in slow-graph request: "
+ r"payload\.evidence\[\d+\]\.metadata\.origin_answer_ids",
+ error,
+ )
+ or _clean(job["last_error"]) != error
+ or metadata.get("physical_api_call") is not False
+ or int(metadata.get("physical_api_calls", -1)) != 0
+ or _clean(metadata.get("physical_call_id"))
+ or _clean(metadata.get("raw_response"))
+ or _clean(metadata.get("content"))
+ ):
+ raise SlowGraphError(
+ "failed attempt is not a proven zero-call projection failure"
+ )
+ evidence_ids = _v3._strict_json(
+ job["evidence_ids_json"], label="job evidence IDs", expected=list
+ )
+ evidence = store._evidence(con, job["scope_id"], evidence_ids)
+ offending_paths: list[str] = []
+ for index, leaf in enumerate(evidence):
+ for key, value in _leaf_metadata(leaf).items():
+ if not _forbidden_field(key):
+ continue
+ if key != "origin_answer_ids" or value != []:
+ raise SlowGraphError(
+ "projection recovery found non-empty or unsupported benchmark metadata"
+ )
+ offending_paths.append(
+ f"payload.evidence[{index}].metadata.origin_answer_ids"
+ )
+ if error.rsplit(": ", 1)[-1] not in offending_paths:
+ raise SlowGraphError(
+ "projection recovery error does not match current internal evidence"
+ )
+ public_region = {
+ "region_key": _required_text(job["region_key"], "region key"),
+ "evidence": [_public_leaf(item) for item in evidence],
+ }
+ _assert_no_benchmark_fields(public_region)
+ error_sha256 = hashlib.sha256(error.encode("utf-8")).hexdigest()
+ metadata_sha256 = hashlib.sha256(raw_metadata.encode("utf-8")).hexdigest()
+ projection_sha256 = _digest(public_region)
+ recovery_id = "sgp0_" + _digest(
+ {
+ "job_id": job_id,
+ "attempt_id": attempt["attempt_id"],
+ "error_sha256": error_sha256,
+ "projection_sha256": projection_sha256,
+ }
+ )[:32]
+ created_at = _v3._now()
+ con.execute(
+ """
+ CREATE TABLE IF NOT EXISTS slow_graph_zero_call_projection_recoveries(
+ recovery_id TEXT PRIMARY KEY,
+ job_id TEXT NOT NULL UNIQUE,
+ attempt_id TEXT NOT NULL UNIQUE,
+ scope_id TEXT NOT NULL,
+ offending_paths_json TEXT NOT NULL,
+ error_sha256 TEXT NOT NULL,
+ call_metadata_sha256 TEXT NOT NULL,
+ public_projection_sha256 TEXT NOT NULL,
+ physical_api_calls INTEGER NOT NULL,
+ created_at INTEGER NOT NULL
+ )
+ """
+ )
+ con.execute(
+ "INSERT INTO slow_graph_zero_call_projection_recoveries "
+ "VALUES(?,?,?,?,?,?,?,?,?,?)",
+ (
+ recovery_id,
+ job_id,
+ attempt["attempt_id"],
+ job["scope_id"],
+ _json(offending_paths),
+ error_sha256,
+ metadata_sha256,
+ projection_sha256,
+ 0,
+ created_at,
+ ),
+ )
+ reopened = con.execute(
+ "UPDATE slow_graph_jobs SET status='pending',last_error='',updated_at=?,"
+ "claim_token=NULL,claim_owner=NULL,lease_expires_at=NULL "
+ "WHERE job_id=? AND status='failed' AND claim_token IS NULL",
+ (created_at, job_id),
+ )
+ if reopened.rowcount != 1:
+ raise SlowGraphError(
+ "slow graph job changed during zero-call projection recovery"
+ )
+ return {
+ "schema_version": "tmcra.v4.slow-zero-call-projection-recovery.1",
+ "recovery_id": recovery_id,
+ "job_id": job_id,
+ "attempt_id": str(attempt["attempt_id"]),
+ "scope_id": str(job["scope_id"]),
+ "offending_paths": offending_paths,
+ "public_projection_sha256": projection_sha256,
+ "physical_api_calls": 0,
+ "status": "pending",
+ "created_at": created_at,
+ }
+
+
+class TieredGraphPatchManager:
+ """Controller-owned route selection for V4 slow graph jobs."""
+
+ def __init__(
+ self,
+ *,
+ flash_config: DeepSeekTierConfig | None = None,
+ pro_config: DeepSeekTierConfig | None = None,
+ flash: Any | None = None,
+ pro: Any | None = None,
+ ) -> None:
+ self.flash = flash or (_DeepSeekTierClient(flash_config, route="flash") if flash_config else None)
+ self.pro = pro or (_DeepSeekTierClient(pro_config, route="pro") if pro_config else None)
+ self.model_config = {
+ "model": "deepseek-v4-tiered-slow-graph",
+ "temperature": 0,
+ "route_policy": (
+ "deterministic-create-noop/flash-incremental/"
+ "pro-initial-partition-conflict"
+ ),
+ "prompt_version": SLOW_PROMPT_VERSION,
+ }
+ self.prompt_hash = _digest(
+ {
+ "schema": SCHEMA_VERSION,
+ "policy": self.model_config["route_policy"],
+ "prompt_version": SLOW_PROMPT_VERSION,
+ }
+ )
+ self.last_call_metadata: Mapping[str, Any] = {}
+
+ @classmethod
+ def from_env(cls) -> "TieredGraphPatchManager":
+ return cls(
+ flash_config=_optional_config("TMCRA_DEEPSEEK_FLASH", "deepseek-v4-flash"),
+ pro_config=_optional_config("TMCRA_DEEPSEEK_PRO", "deepseek-v4-pro"),
+ )
+
+ @staticmethod
+ def _capsule_claims(
+ capsules: list[Mapping[str, Any]],
+ ) -> tuple[set[str], dict[str, list[Mapping[str, Any]]], set[str]]:
+ cited: set[str] = set()
+ by_slot: dict[str, list[Mapping[str, Any]]] = {}
+ statuses: set[str] = set()
+ for capsule in capsules:
+ raw_claims = capsule.get("claims")
+ if not isinstance(raw_claims, list):
+ raise EvidencePolicyError("capsule claims are not auditable")
+ status = _clean(capsule.get("status") or "active").casefold()
+ statuses.add(status)
+ if status not in {"active", "challenged"}:
+ continue
+ for claim in raw_claims:
+ if not isinstance(claim, Mapping):
+ raise EvidencePolicyError("capsule claim is not an object")
+ slot = _required_text(claim.get("canonical_slot"), "capsule claim canonical slot")
+ support = claim.get("support", [])
+ counter = claim.get("counterevidence", [])
+ if not isinstance(support, list) or not isinstance(counter, list):
+ raise EvidencePolicyError("capsule claim evidence is not a list")
+ cited.update(_clean(item) for item in [*support, *counter] if _clean(item))
+ by_slot.setdefault(slot, []).append(claim)
+ return cited, by_slot, statuses
+
+ @staticmethod
+ def _base_metadata(
+ route: str,
+ *,
+ reason: str,
+ evidence: list[Mapping[str, Any]],
+ eligible: list[Mapping[str, Any]],
+ challenged: list[Mapping[str, Any]],
+ uncertain: list[str],
+ episodic: list[str],
+ inactive: list[str],
+ delta: list[Mapping[str, Any]],
+ ) -> dict[str, Any]:
+ return {
+ "route": route,
+ "route_reason": reason,
+ "summary_contract_version": SLOW_SUMMARY_CONTRACT_VERSION,
+ "evidence_binding_contract_version": (
+ SLOW_EVIDENCE_BINDING_CONTRACT_VERSION
+ ),
+ "physical_api_call": False,
+ "physical_api_calls": 0,
+ "attempt_count": 0,
+ "usage": {"prompt_tokens": 0, "completion_tokens": 0, "cache_read_input_tokens": 0, "cache_hit_tokens": 0, "cache_miss_tokens": 0, "total_tokens": 0},
+ "cost_audit": {"estimated_cost": 0.0, "prompt_tokens": 0, "completion_tokens": 0, "cache_read_input_tokens": 0, "cache_hit_tokens": 0, "cache_miss_tokens": 0},
+ "eligible_evidence_ids": [_leaf_id(item) for item in eligible],
+ "challenged_evidence_ids": [_leaf_id(item) for item in challenged],
+ "delta_evidence_ids": [_leaf_id(item) for item in delta],
+ "uncertain_evidence_ids": uncertain,
+ "episodic_evidence_ids": episodic,
+ "inactive_evidence_ids": inactive,
+ "ignored_evidence_ids": sorted(set(uncertain + episodic + inactive)),
+ "supplied_evidence_count": len(evidence),
+ }
+
+ @staticmethod
+ def _create_patch(leaf: Mapping[str, Any]) -> dict[str, Any]:
+ text = _leaf_text(leaf)
+ return _materialize_lossless_summaries({
+ "operations": [
+ {
+ "action": "create",
+ "capsule_key": _capsule_key_from_slot(_leaf_slot(leaf)),
+ "claims": [
+ {
+ "canonical_slot": _leaf_slot(leaf),
+ "text": text,
+ "support": [_leaf_id(leaf)],
+ "counterevidence": [],
+ }
+ ],
+ }
+ ]
+ })
+
+ @staticmethod
+ def _noop_patch(capsules: list[Mapping[str, Any]]) -> dict[str, Any]:
+ operation: dict[str, Any] = {"action": "noop"}
+ if capsules:
+ capsule_id = _clean(capsules[0].get("capsule_id"))
+ if capsule_id:
+ operation["capsule_id"] = capsule_id
+ return {"operations": [operation]}
+
+ @staticmethod
+ def _public_request(
+ route: str,
+ region: Mapping[str, Any],
+ required_evidence_ids: set[str],
+ *,
+ partition_capsule_ids: set[str] | None = None,
+ semantic_partition_mode: str | None = None,
+ ) -> dict[str, Any]:
+ selected_evidence = [
+ item
+ for item in region.get("evidence", [])
+ if isinstance(item, Mapping)
+ and (
+ _leaf_id(item) in required_evidence_ids
+ or (
+ route == "pro"
+ and (
+ _is_current_durable(item)
+ or _is_challenged_durable(item)
+ )
+ )
+ )
+ ]
+ public_region = {
+ "region_key": _required_text(region.get("region_key"), "region key"),
+ "evidence": [_public_leaf(item) for item in selected_evidence],
+ "required_evidence_ids": sorted(required_evidence_ids),
+ }
+ partition_ids = sorted(partition_capsule_ids or ())
+ partition_mode = _clean(semantic_partition_mode)
+ if partition_mode:
+ if partition_mode not in {"manage", "migrate"}:
+ raise EvidencePolicyError("semantic partition mode is invalid")
+ public_region["semantic_partition_required"] = True
+ public_region["semantic_partition_mode"] = partition_mode
+ if partition_mode == "migrate":
+ if not partition_ids:
+ raise EvidencePolicyError(
+ "semantic partition migration requires capsule targets"
+ )
+ public_region["partition_capsule_ids"] = partition_ids
+ elif partition_ids:
+ raise EvidencePolicyError(
+ "generic semantic management cannot name partition targets"
+ )
+ elif partition_ids:
+ raise EvidencePolicyError(
+ "partition capsule targets require semantic partition mode"
+ )
+ _assert_no_benchmark_fields(public_region)
+ return public_region
+
+ @staticmethod
+ def _completed_client_metadata(client: Any, route: str) -> dict[str, Any]:
+ raw = dict(getattr(client, "last_call_metadata", {}) or {})
+ physical_calls = int(raw.get("physical_api_calls", 1) or 0)
+ if physical_calls < 1:
+ physical_calls = 1
+ return {
+ **raw,
+ "route": route,
+ "physical_api_call": True,
+ "physical_api_calls": physical_calls,
+ "attempt_count": int(raw.get("attempt_count", physical_calls) or physical_calls),
+ }
+
+ @staticmethod
+ def _aggregate_escalation_metadata(
+ metadata: Mapping[str, Any],
+ flash_metadata: Mapping[str, Any],
+ pro_metadata: Mapping[str, Any] | None,
+ ) -> dict[str, Any]:
+ tier_calls = [dict(flash_metadata)]
+ if pro_metadata is not None:
+ tier_calls.append(dict(pro_metadata))
+ usage_keys = (
+ "prompt_tokens",
+ "completion_tokens",
+ "cache_read_input_tokens",
+ "cache_hit_tokens",
+ "cache_miss_tokens",
+ "total_tokens",
+ )
+ usage = {key: 0 for key in usage_keys}
+ estimated_cost = 0.0
+ physical_calls = 0
+ attempt_count = 0
+ latency_ms = 0.0
+ for call in tier_calls:
+ physical_calls += int(call.get("physical_api_calls", 0) or 0)
+ attempt_count += int(call.get("attempt_count", 0) or 0)
+ call_usage = call.get("usage")
+ if isinstance(call_usage, Mapping):
+ for key in usage_keys:
+ usage[key] += int(call_usage.get(key, 0) or 0)
+ call_cost = call.get("cost_audit")
+ if isinstance(call_cost, Mapping):
+ estimated_cost += float(call_cost.get("estimated_cost", 0.0) or 0.0)
+ try:
+ latency_ms += float(call.get("latency_ms", 0.0) or 0.0)
+ except (TypeError, ValueError):
+ pass
+ final_call = dict(pro_metadata or flash_metadata)
+ result = {
+ **dict(metadata),
+ **final_call,
+ "route": "flash_to_pro",
+ "route_reason": f"flash_escalation:{FLASH_ESCALATION_REASON}",
+ "initial_route_reason": metadata.get("route_reason"),
+ "escalation_requested": True,
+ "escalation_reason": FLASH_ESCALATION_REASON,
+ "physical_api_call": physical_calls > 0,
+ "physical_api_calls": physical_calls,
+ "attempt_count": attempt_count,
+ "usage": usage,
+ "cost_audit": {
+ **usage,
+ "estimated_cost": estimated_cost,
+ },
+ "tier_calls": tier_calls,
+ }
+ if latency_ms:
+ result["latency_ms"] = round(latency_ms, 3)
+ if flash_metadata.get("started_at") is not None:
+ result["started_at"] = flash_metadata.get("started_at")
+ return result
+
+ @staticmethod
+ def _aggregate_semantic_correction_metadata(
+ metadata: Mapping[str, Any],
+ tier_calls: list[Mapping[str, Any]],
+ *,
+ route: str,
+ route_reason: str,
+ validation_error: str,
+ ) -> dict[str, Any]:
+ if len(tier_calls) < 2:
+ raise SlowGraphError("semantic correction metadata requires two calls")
+ usage_keys = (
+ "prompt_tokens",
+ "completion_tokens",
+ "cache_read_input_tokens",
+ "cache_hit_tokens",
+ "cache_miss_tokens",
+ "total_tokens",
+ )
+ usage = {key: 0 for key in usage_keys}
+ estimated_cost = 0.0
+ physical_calls = 0
+ attempt_count = 0
+ latency_ms = 0.0
+ labeled_calls: list[dict[str, Any]] = []
+ for index, raw_call in enumerate(tier_calls):
+ call = dict(raw_call)
+ if index == len(tier_calls) - 1:
+ call["tier_stage"] = "semantic_correction"
+ elif route == "flash_to_pro" and index == 0:
+ call["tier_stage"] = "initial_flash"
+ else:
+ call["tier_stage"] = "initial_pro"
+ labeled_calls.append(call)
+ physical_calls += int(call.get("physical_api_calls", 0) or 0)
+ attempt_count += int(call.get("attempt_count", 0) or 0)
+ call_usage = call.get("usage")
+ if isinstance(call_usage, Mapping):
+ for key in usage_keys:
+ usage[key] += int(call_usage.get(key, 0) or 0)
+ call_cost = call.get("cost_audit")
+ if isinstance(call_cost, Mapping):
+ estimated_cost += float(call_cost.get("estimated_cost", 0.0) or 0.0)
+ try:
+ latency_ms += float(call.get("latency_ms", 0.0) or 0.0)
+ except (TypeError, ValueError):
+ pass
+ final_call = dict(labeled_calls[-1])
+ result = {
+ **dict(metadata),
+ **final_call,
+ "route": route,
+ "route_reason": route_reason,
+ "semantic_correction_attempted": True,
+ "semantic_correction_validation_error": validation_error,
+ "physical_api_call": physical_calls > 0,
+ "physical_api_calls": physical_calls,
+ "attempt_count": attempt_count,
+ "usage": usage,
+ "cost_audit": {**usage, "estimated_cost": estimated_cost},
+ "tier_calls": labeled_calls,
+ }
+ if latency_ms:
+ result["latency_ms"] = round(latency_ms, 3)
+ first_started = labeled_calls[0].get("started_at")
+ if first_started is not None:
+ result["started_at"] = first_started
+ return result
+
+ def _validate_and_materialize_patch(
+ self,
+ *,
+ route: str,
+ patch: Mapping[str, Any],
+ region: Mapping[str, Any],
+ capsules: list[Mapping[str, Any]],
+ required_evidence_ids: set[str],
+ partition_capsule_ids: set[str],
+ ) -> tuple[dict[str, Any], dict[str, Any] | None]:
+ validate_patch(patch)
+ _validate_generic_create_partition_keys(region.get("region_key"), patch)
+ self._validate_route_actions(
+ route,
+ patch,
+ capsules,
+ required_evidence_ids=required_evidence_ids,
+ partition_capsule_ids=partition_capsule_ids,
+ )
+ _validate_claim_evidence_contract(region, capsules, patch, route=route)
+ merge_audit: dict[str, Any] | None = None
+ if route == "flash" and required_evidence_ids:
+ _validate_flash_delta_patch(patch, capsules, required_evidence_ids)
+ if capsules:
+ committed_patch, merge_audit = _merge_flash_delta_patch(
+ patch, capsules
+ )
+ else:
+ committed_patch = _materialize_lossless_summaries(patch)
+ else:
+ committed_patch = _materialize_lossless_summaries(patch)
+ validate_patch(committed_patch, require_lossless_summary=True)
+ _validate_generic_create_partition_keys(
+ region.get("region_key"), committed_patch
+ )
+ self._validate_route_actions(
+ route,
+ committed_patch,
+ capsules,
+ required_evidence_ids=required_evidence_ids,
+ partition_capsule_ids=partition_capsule_ids,
+ )
+ _validate_claim_evidence_contract(
+ region, capsules, committed_patch, route=route
+ )
+ _validate_promotion_patch(
+ region,
+ capsules,
+ committed_patch,
+ required_evidence_ids=required_evidence_ids,
+ )
+ return committed_patch, merge_audit
+
+ def _validate_pro_with_optional_correction(
+ self,
+ *,
+ client: Any,
+ public_region: Mapping[str, Any],
+ public_capsules: list[Mapping[str, Any]],
+ region: Mapping[str, Any],
+ capsules: list[Mapping[str, Any]],
+ patch: Mapping[str, Any],
+ metadata: Mapping[str, Any],
+ tier_calls: list[Mapping[str, Any]],
+ route: str,
+ route_reason: str,
+ required_evidence_ids: set[str],
+ partition_capsule_ids: set[str],
+ ) -> tuple[dict[str, Any], dict[str, Any] | None, Mapping[str, Any]]:
+ try:
+ committed, merge_audit = self._validate_and_materialize_patch(
+ route="pro",
+ patch=patch,
+ region=region,
+ capsules=capsules,
+ required_evidence_ids=required_evidence_ids,
+ partition_capsule_ids=partition_capsule_ids,
+ )
+ return committed, merge_audit, patch
+ except PatchValidationError as initial_error:
+ correct = getattr(client, "correct", None)
+ if not callable(correct):
+ raise
+ initial_error_text = str(initial_error)
+
+ previous_call_id = _clean(
+ dict(getattr(client, "last_call_metadata", {}) or {}).get(
+ "physical_call_id"
+ )
+ )
+ try:
+ corrected_patch = correct(
+ public_region,
+ public_capsules,
+ rejected_patch=patch,
+ validation_error=initial_error_text,
+ )
+ except Exception:
+ raw_correction = dict(
+ getattr(client, "last_call_metadata", {}) or {}
+ )
+ correction_call_id = _clean(raw_correction.get("physical_call_id"))
+ if correction_call_id and correction_call_id != previous_call_id:
+ correction_metadata = self._completed_client_metadata(client, "pro")
+ self.last_call_metadata = {
+ **self._aggregate_semantic_correction_metadata(
+ metadata,
+ [*tier_calls, correction_metadata],
+ route=route,
+ route_reason=route_reason,
+ validation_error=initial_error_text,
+ ),
+ "status": "semantic_correction_call_failed",
+ "semantic_correction_applied": False,
+ }
+ else:
+ self.last_call_metadata = {
+ **dict(self.last_call_metadata),
+ "status": "semantic_correction_preflight_failed",
+ "semantic_correction_attempted": True,
+ "semantic_correction_applied": False,
+ "semantic_correction_validation_error": initial_error_text,
+ }
+ raise
+ correction_metadata = self._completed_client_metadata(client, "pro")
+ corrected_patch, correction_normalizations = _normalize_transport_patch(
+ corrected_patch, public_capsules, public_region
+ )
+ if correction_normalizations:
+ correction_metadata = {
+ **correction_metadata,
+ "controller_transport_normalizations": correction_normalizations,
+ }
+ self.last_call_metadata = self._aggregate_semantic_correction_metadata(
+ metadata,
+ [*tier_calls, correction_metadata],
+ route=route,
+ route_reason=route_reason,
+ validation_error=initial_error_text,
+ )
+ try:
+ committed, merge_audit = self._validate_and_materialize_patch(
+ route="pro",
+ patch=corrected_patch,
+ region=region,
+ capsules=capsules,
+ required_evidence_ids=required_evidence_ids,
+ partition_capsule_ids=partition_capsule_ids,
+ )
+ except PatchValidationError as correction_error:
+ self.last_call_metadata = {
+ **dict(self.last_call_metadata),
+ "status": "semantic_correction_rejected",
+ "semantic_correction_applied": False,
+ "semantic_correction_rejection_error": str(correction_error),
+ "initial_rejected_patch_sha256": _digest(patch),
+ "corrected_patch_sha256": _digest(corrected_patch),
+ }
+ raise
+ self.last_call_metadata = {
+ **dict(self.last_call_metadata),
+ "status": "completed",
+ "semantic_correction_applied": True,
+ "initial_rejected_patch_sha256": _digest(patch),
+ "corrected_patch_sha256": _digest(corrected_patch),
+ }
+ return committed, merge_audit, corrected_patch
+
+ def _invoke(
+ self,
+ route: str,
+ reason: str,
+ client: Any,
+ region: Mapping[str, Any],
+ capsules: list[Mapping[str, Any]],
+ metadata: dict[str, Any],
+ ) -> Mapping[str, Any]:
+ if client is None:
+ self.last_call_metadata = {**metadata, "route": route, "route_reason": reason, "status": "unavailable"}
+ raise TieredAPIError(f"{route} client is not configured; no fallback is allowed")
+ delta_ids = set(metadata["delta_evidence_ids"])
+ required_ids = set(
+ metadata.get("required_operation_evidence_ids") or delta_ids
+ )
+ partition_capsule_ids = set(
+ metadata.get("semantic_partition_capsule_ids") or ()
+ )
+ semantic_partition_mode = _clean(
+ metadata.get("semantic_partition_mode")
+ ) or None
+ public_region = self._public_request(
+ route,
+ region,
+ required_ids,
+ partition_capsule_ids=partition_capsule_ids,
+ semantic_partition_mode=semantic_partition_mode,
+ )
+ public_capsules = [_public_capsule(capsule) for capsule in capsules]
+ _assert_no_benchmark_fields(public_capsules)
+ try:
+ patch = client.propose(public_region, public_capsules)
+ except Exception:
+ client_metadata = getattr(client, "last_call_metadata", {})
+ self.last_call_metadata = {**metadata, **dict(client_metadata), "route": route, "route_reason": reason}
+ raise
+ client_metadata = self._completed_client_metadata(client, route)
+ patch, controller_transport_normalizations = _normalize_transport_patch(
+ patch, public_capsules, public_region
+ )
+ if controller_transport_normalizations:
+ client_metadata = {
+ **client_metadata,
+ "controller_transport_normalizations": controller_transport_normalizations,
+ }
+
+ if route == "flash" and _flash_escalation_patch(patch):
+ if self.pro is None:
+ self.last_call_metadata = {
+ **self._aggregate_escalation_metadata(
+ metadata, client_metadata, None
+ ),
+ "status": "unavailable",
+ }
+ raise TieredAPIError(
+ "pro client is not configured after explicit Flash escalation; "
+ "no fallback is allowed"
+ )
+ pro_region = self._public_request(
+ "pro",
+ region,
+ required_ids,
+ partition_capsule_ids=partition_capsule_ids,
+ semantic_partition_mode=semantic_partition_mode,
+ )
+ try:
+ pro_patch = self.pro.propose(pro_region, public_capsules)
+ except Exception:
+ pro_metadata = self._completed_client_metadata(self.pro, "pro")
+ self.last_call_metadata = self._aggregate_escalation_metadata(
+ metadata, client_metadata, pro_metadata
+ )
+ raise
+ pro_metadata = self._completed_client_metadata(self.pro, "pro")
+ pro_patch, pro_transport_normalizations = _normalize_transport_patch(
+ pro_patch, public_capsules, pro_region
+ )
+ if pro_transport_normalizations:
+ pro_metadata = {
+ **pro_metadata,
+ "controller_transport_normalizations": pro_transport_normalizations,
+ }
+ self.last_call_metadata = self._aggregate_escalation_metadata(
+ metadata, client_metadata, pro_metadata
+ )
+ escalation_reason = f"flash_escalation:{FLASH_ESCALATION_REASON}"
+ committed_patch, merge_audit, model_patch = (
+ self._validate_pro_with_optional_correction(
+ client=self.pro,
+ public_region=pro_region,
+ public_capsules=public_capsules,
+ region=region,
+ capsules=capsules,
+ patch=pro_patch,
+ metadata=metadata,
+ tier_calls=[client_metadata, pro_metadata],
+ route="flash_to_pro",
+ route_reason=escalation_reason,
+ required_evidence_ids=required_ids,
+ partition_capsule_ids=partition_capsule_ids,
+ )
+ )
+ self.last_call_metadata = {
+ **dict(self.last_call_metadata),
+ "controller_summary_materialization": {
+ "schema_version": SLOW_SUMMARY_CONTRACT_VERSION,
+ "model_patch_sha256": _digest(model_patch),
+ "committed_patch_sha256": _digest(committed_patch),
+ },
+ }
+ if merge_audit is not None:
+ self.last_call_metadata = {
+ **dict(self.last_call_metadata),
+ "controller_delta_merge": merge_audit,
+ }
+ return committed_patch
+
+ self.last_call_metadata = {
+ **metadata,
+ **client_metadata,
+ "route": route,
+ "route_reason": reason,
+ }
+ if route == "pro":
+ committed_patch, merge_audit, model_patch = (
+ self._validate_pro_with_optional_correction(
+ client=client,
+ public_region=public_region,
+ public_capsules=public_capsules,
+ region=region,
+ capsules=capsules,
+ patch=patch,
+ metadata=metadata,
+ tier_calls=[client_metadata],
+ route="pro",
+ route_reason=reason,
+ required_evidence_ids=required_ids,
+ partition_capsule_ids=partition_capsule_ids,
+ )
+ )
+ else:
+ committed_patch, merge_audit = self._validate_and_materialize_patch(
+ route=route,
+ patch=patch,
+ region=region,
+ capsules=capsules,
+ required_evidence_ids=required_ids,
+ partition_capsule_ids=partition_capsule_ids,
+ )
+ model_patch = patch
+ summary_audit = {
+ "schema_version": SLOW_SUMMARY_CONTRACT_VERSION,
+ "model_patch_sha256": _digest(model_patch),
+ "committed_patch_sha256": _digest(committed_patch),
+ }
+ if merge_audit is not None:
+ self.last_call_metadata = {
+ **dict(self.last_call_metadata),
+ "controller_delta_merge": merge_audit,
+ "controller_summary_materialization": summary_audit,
+ }
+ else:
+ self.last_call_metadata = {
+ **dict(self.last_call_metadata),
+ "controller_summary_materialization": summary_audit,
+ }
+ return committed_patch
+
+ @staticmethod
+ def _validate_route_actions(
+ route: str,
+ patch: Mapping[str, Any],
+ capsules: list[Mapping[str, Any]],
+ *,
+ required_evidence_ids: set[str] | None = None,
+ partition_capsule_ids: set[str] | None = None,
+ ) -> None:
+ required = bool(required_evidence_ids)
+ existing = {
+ _required_text(capsule.get("capsule_id"), "capsule_id"): capsule
+ for capsule in capsules
+ }
+ existing_keys = {
+ _clean(capsule.get("capsule_key")).casefold()
+ for capsule in capsules
+ if _clean(capsule.get("capsule_key"))
+ }
+ allowed = (
+ {"revise", "create"}
+ if route == "flash"
+ else {"revise", "create", "challenge", "resolve_challenge", "retire"}
+ )
+ if not required:
+ allowed.add("noop")
+ operated_capsules: set[str] = set()
+ for operation in patch["operations"]:
+ action = operation["action"]
+ if action not in allowed:
+ raise PatchValidationError(
+ f"{route} route does not allow action {action!r} for {'existing' if capsules else 'new'} capsule"
+ )
+ if action == "create":
+ capsule_key = _normalize_capsule_key(operation.get("capsule_key"))
+ if capsule_key in existing_keys:
+ raise PatchValidationError(
+ f"create capsule_key already exists in this region: {capsule_key}"
+ )
+ continue
+ if action == "noop":
+ capsule_id = _clean(operation.get("capsule_id"))
+ if capsule_id and capsule_id not in existing:
+ raise PatchValidationError("noop targeted an unknown capsule")
+ continue
+ capsule_id = _required_text(operation.get("capsule_id"), "capsule_id")
+ capsule = existing.get(capsule_id)
+ if capsule is None:
+ raise PatchValidationError(
+ f"{route} route targeted an unknown capsule: {capsule_id}"
+ )
+ if operation.get("base_revision") != capsule.get("revision"):
+ raise PatchValidationError(
+ f"{route} base_revision is stale for capsule {capsule_id}"
+ )
+ operated_capsules.add(capsule_id)
+ missing_partition_targets = set(partition_capsule_ids or ()) - operated_capsules
+ if missing_partition_targets:
+ raise PatchValidationError(
+ "semantic partition left required legacy capsules untouched: "
+ + _json(sorted(missing_partition_targets))
+ )
+
+ def propose(self, region: Mapping[str, Any], capsules: list[Mapping[str, Any]]) -> Mapping[str, Any]:
+ if not isinstance(region, Mapping) or not isinstance(capsules, list):
+ raise EvidencePolicyError("slow-graph region and capsules have invalid schemas")
+ if set(region) != {"region_key", "evidence"}:
+ raise EvidencePolicyError("slow-graph internal region envelope is not exact")
+ evidence = [item for item in region.get("evidence", []) if isinstance(item, Mapping)]
+ evidence.sort(key=lambda item: _leaf_id(item))
+ _assert_no_benchmark_fields(
+ {
+ "region_key": _required_text(region.get("region_key"), "region key"),
+ "evidence": [_public_leaf(item) for item in evidence],
+ }
+ )
+ _assert_no_benchmark_fields(capsules)
+ eligible = [item for item in evidence if _is_current_durable(item)]
+ challenged = [item for item in evidence if _is_challenged_durable(item)]
+ uncertain = sorted(_leaf_id(item) for item in evidence if _is_uncertain(item))
+ episodic = sorted(_leaf_id(item) for item in evidence if _is_episodic(item))
+ current_support_ids = {_leaf_id(item) for item in eligible}
+ challenged_ids = {_leaf_id(item) for item in challenged}
+ active_ids = current_support_ids | challenged_ids
+ inactive = sorted(
+ _leaf_id(item)
+ for item in evidence
+ if _leaf_id(item) not in active_ids
+ and _leaf_id(item) not in set(uncertain)
+ and _leaf_id(item) not in set(episodic)
+ )
+ original_capsules = capsules
+ capsules, support_cleanup = _sanitize_capsules_for_current_support(
+ original_capsules,
+ current_support_ids,
+ challenged_ids,
+ {_leaf_id(item) for item in evidence},
+ )
+ cited, claims_by_slot, capsule_statuses = self._capsule_claims(capsules)
+ delta = [item for item in [*eligible, *challenged] if _leaf_id(item) not in cited]
+ partition_targets = _semantic_partition_targets(
+ region.get("region_key"), capsules
+ )
+ generic_semantic_management = bool(delta) and (
+ not partition_targets
+ and _generic_region_requires_semantic_management(
+ region.get("region_key"), [*eligible, *challenged], capsules
+ )
+ )
+ uncited_current_support = [
+ item for item in eligible if _leaf_id(item) not in cited
+ ]
+ if (
+ support_cleanup["changed"]
+ and not uncited_current_support
+ and not partition_targets
+ ):
+ reason = "existing Slow claims lost all current Fast support"
+ self.last_call_metadata = {
+ **self._base_metadata(
+ "deterministic_support_cleanup",
+ reason=reason,
+ evidence=evidence,
+ eligible=eligible,
+ challenged=challenged,
+ uncertain=uncertain,
+ episodic=episodic,
+ inactive=inactive,
+ delta=delta,
+ ),
+ "support_cleanup": support_cleanup,
+ }
+ patch = _deterministic_support_cleanup_patch(
+ original_capsules, capsules
+ )
+ validate_patch(patch)
+ _validate_patch_summary_contract(patch)
+ _validate_claim_evidence_contract(region, capsules, patch)
+ _validate_promotion_patch(region, capsules, patch)
+ return patch
+ if not delta and partition_targets:
+ if not _partition_targets_require_model(capsules, partition_targets):
+ reason = "unambiguous_single_claim_partition_migration"
+ self.last_call_metadata = {
+ **self._base_metadata(
+ "deterministic_contract_migration",
+ reason=reason,
+ evidence=evidence,
+ eligible=eligible,
+ challenged=challenged,
+ uncertain=uncertain,
+ episodic=episodic,
+ inactive=inactive,
+ delta=delta,
+ ),
+ "semantic_partition_contract_version": SLOW_PARTITION_CONTRACT_VERSION,
+ "semantic_partition_capsule_ids": sorted(partition_targets),
+ "semantic_partition_mode": "migrate",
+ }
+ if support_cleanup["changed"]:
+ self.last_call_metadata["support_cleanup"] = support_cleanup
+ patch = _deterministic_contract_migration_patch(
+ capsules, partition_targets
+ )
+ validate_patch(patch, require_lossless_summary=True)
+ _validate_claim_evidence_contract(region, capsules, patch)
+ _validate_promotion_patch(region, capsules, patch)
+ return patch
+ reason = "semantic_partition_migration"
+ required_operation_ids = {
+ _leaf_id(item) for item in [*eligible, *challenged]
+ }
+ metadata = self._base_metadata(
+ "pro",
+ reason=reason,
+ evidence=evidence,
+ eligible=eligible,
+ challenged=challenged,
+ uncertain=uncertain,
+ episodic=episodic,
+ inactive=inactive,
+ delta=delta,
+ )
+ metadata.update(
+ {
+ "semantic_partition_contract_version": SLOW_PARTITION_CONTRACT_VERSION,
+ "semantic_partition_capsule_ids": sorted(partition_targets),
+ "semantic_partition_mode": "migrate",
+ "required_operation_evidence_ids": sorted(
+ required_operation_ids
+ ),
+ }
+ )
+ if support_cleanup["changed"]:
+ metadata["support_cleanup"] = support_cleanup
+ return self._invoke(
+ "pro", reason, self.pro, region, capsules, metadata
+ )
+ if not delta:
+ if _capsule_requires_summary_migration(capsules):
+ reason = "stored Slow summary violates the semantic summary contract"
+ self.last_call_metadata = self._base_metadata(
+ "deterministic_summary_migration",
+ reason=reason,
+ evidence=evidence,
+ eligible=eligible,
+ challenged=challenged,
+ uncertain=uncertain,
+ episodic=episodic,
+ inactive=inactive,
+ delta=delta,
+ )
+ patch = _deterministic_summary_migration_patch(capsules)
+ validate_patch(patch)
+ _validate_patch_summary_contract(patch)
+ _validate_claim_evidence_contract(region, capsules, patch)
+ _validate_promotion_patch(region, capsules, patch)
+ return patch
+ reason = "no new eligible durable evidence"
+ self.last_call_metadata = self._base_metadata("deterministic_noop", reason=reason, evidence=evidence, eligible=eligible, challenged=challenged, uncertain=uncertain, episodic=episodic, inactive=inactive, delta=delta)
+ patch = self._noop_patch(capsules)
+ validate_patch(patch)
+ _validate_promotion_patch(region, capsules, patch)
+ return patch
+
+ if not capsules and challenged and not eligible:
+ reason = "new capsule blocked by unresolved fast challenge"
+ self.last_call_metadata = self._base_metadata("deterministic_noop", reason=reason, evidence=evidence, eligible=eligible, challenged=challenged, uncertain=uncertain, episodic=episodic, inactive=inactive, delta=delta)
+ patch = self._noop_patch(capsules)
+ validate_patch(patch)
+ _validate_promotion_patch(region, capsules, patch)
+ return patch
+
+ reasons: list[str] = []
+ delta_texts_by_slot: dict[str, set[str]] = {}
+ for leaf in delta:
+ delta_texts_by_slot.setdefault(_leaf_slot(leaf), set()).add(
+ _normal_text(_leaf_text(leaf))
+ )
+ if not capsules and len(delta_texts_by_slot) > 1:
+ reasons.append("initial_multi_slot_semantic_partition")
+ if any(len(texts) > 1 for texts in delta_texts_by_slot.values()):
+ reasons.append("same_slot_distinct_support_semantics")
+ for leaf in delta:
+ slot = _leaf_slot(leaf)
+ if _is_challenged_durable(leaf):
+ reasons.append("unresolved_fast_challenge")
+ if _is_counterevidence(leaf):
+ reasons.append("counterevidence")
+ if slot in claims_by_slot:
+ leaf_value = _normal_text(_leaf_text(leaf))
+ if not any(_normal_text(claim.get("text")) == leaf_value for claim in claims_by_slot[slot]):
+ reasons.append("same_slot_correction")
+ if capsule_statuses & {"challenged", "quarantined"}:
+ reasons.append("unresolved_challenge")
+ if partition_targets:
+ reasons.append("semantic_partition_migration")
+ if generic_semantic_management:
+ reasons.append("generic_region_semantic_management")
+ reason = "+".join(sorted(set(reasons)))
+ metadata = self._base_metadata("pro" if reason else "flash", reason=reason or "compatible_consolidation", evidence=evidence, eligible=eligible, challenged=challenged, uncertain=uncertain, episodic=episodic, inactive=inactive, delta=delta)
+ required_operation_ids = {
+ _leaf_id(item) for item in [*eligible, *challenged]
+ }
+ if not capsules and eligible and challenged:
+ # A challenged sibling is context, not a reason to drop an unrelated
+ # authoritative fact or promote the unresolved sibling as active.
+ required_operation_ids = {_leaf_id(item) for item in eligible}
+ metadata["required_operation_evidence_ids"] = sorted(
+ required_operation_ids
+ )
+ if support_cleanup["changed"]:
+ metadata["support_cleanup"] = support_cleanup
+ if partition_targets:
+ metadata.update(
+ {
+ "semantic_partition_contract_version": SLOW_PARTITION_CONTRACT_VERSION,
+ "semantic_partition_capsule_ids": sorted(partition_targets),
+ "semantic_partition_mode": "migrate",
+ "required_operation_evidence_ids": sorted(
+ required_operation_ids
+ ),
+ }
+ )
+ elif generic_semantic_management or (
+ not capsules and "initial_multi_slot_semantic_partition" in reasons
+ ):
+ metadata.update(
+ {
+ "semantic_partition_contract_version": SLOW_PARTITION_CONTRACT_VERSION,
+ "semantic_partition_mode": "manage",
+ "required_operation_evidence_ids": sorted(
+ required_operation_ids
+ ),
+ }
+ )
+ if not capsules and len(delta) == 1 and not reasons:
+ self.last_call_metadata = self._base_metadata("deterministic_create", reason="one current durable leaf", evidence=evidence, eligible=eligible, challenged=challenged, uncertain=uncertain, episodic=episodic, inactive=inactive, delta=delta)
+ patch = self._create_patch(delta[0])
+ validate_patch(patch)
+ _validate_patch_summary_contract(patch)
+ _validate_promotion_patch(region, capsules, patch)
+ return patch
+ if reasons:
+ return self._invoke("pro", reason, self.pro, region, capsules, metadata)
+ return self._invoke("flash", "compatible_consolidation", self.flash, region, capsules, metadata)
+
+
+class DeepSeekFlashGraphPatchManager(_DeepSeekTierClient):
+ def __init__(self, config: DeepSeekTierConfig) -> None:
+ super().__init__(config, route="flash")
+
+
+class DeepSeekProGraphPatchManager(_DeepSeekTierClient):
+ def __init__(self, config: DeepSeekTierConfig | DeepSeekProConfig) -> None:
+ super().__init__(config, route="pro")
+
+
+TieredSlowGraphPatchManager = TieredGraphPatchManager
+DeepSeekTieredGraphPatchManager = TieredGraphPatchManager
+TieredPatchManager = TieredGraphPatchManager
+V4SlowGraphPatchManager = TieredGraphPatchManager
+SlowGraphStore = V4SlowGraphStore
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description="TMCRA V4 tiered slow graph controller")
+ parser.add_argument("database", type=Path)
+ parser.add_argument("--repo", type=Path, required=True, help="TMCRA repository containing the real graph schema")
+ sub = parser.add_subparsers(dest="command", required=True)
+ enqueue = sub.add_parser("enqueue")
+ enqueue.add_argument("scope_id")
+ enqueue.add_argument("--region")
+ drain = sub.add_parser("drain")
+ drain.add_argument("--batch-size", type=int)
+ run = sub.add_parser("run")
+ run.add_argument("job_id")
+ revalidate = sub.add_parser("revalidate-failed")
+ revalidate.add_argument("job_id")
+ semantic_policy_revalidation = sub.add_parser(
+ "revalidate-failed-semantic-policy"
+ )
+ semantic_policy_revalidation.add_argument("job_id")
+ model_validation_recovery = sub.add_parser(
+ "resume-failed-model-validation"
+ )
+ model_validation_recovery.add_argument("job_id")
+ prompt_migration_recovery = sub.add_parser(
+ "resume-failed-prompt-migration"
+ )
+ prompt_migration_recovery.add_argument("job_id")
+ zero_call_recovery = sub.add_parser("resume-zero-call-config-failure")
+ zero_call_recovery.add_argument("job_id")
+ promotion_recovery = sub.add_parser(
+ "resume-zero-call-promotion-failure"
+ )
+ promotion_recovery.add_argument("job_id")
+ projection_recovery = sub.add_parser("resume-zero-call-projection-failure")
+ projection_recovery.add_argument("job_id")
+ audit = sub.add_parser("audit")
+ audit.add_argument("scope_id")
+ audit.add_argument("--require-promotion-coverage", action="store_true")
+ args = parser.parse_args()
+ store = SlowGraphStore(args.database, schema=load_graph_schema(args.repo))
+ if args.command == "revalidate-failed":
+ result = revalidate_failed_raw_response(store, args.job_id)
+ elif args.command == "revalidate-failed-semantic-policy":
+ result = revalidate_failed_semantic_policy_response(store, args.job_id)
+ elif args.command == "resume-failed-model-validation":
+ result = resume_failed_model_validation(
+ store, args.job_id, TieredGraphPatchManager.from_env()
+ )
+ elif args.command == "resume-failed-prompt-migration":
+ result = resume_failed_model_validation_after_prompt_migration(
+ store, args.job_id, TieredGraphPatchManager.from_env()
+ )
+ elif args.command == "resume-zero-call-config-failure":
+ result = resume_zero_call_configuration_failure(store, args.job_id)
+ elif args.command == "resume-zero-call-promotion-failure":
+ result = resume_zero_call_promotion_failure(store, args.job_id)
+ elif args.command == "resume-zero-call-projection-failure":
+ result = resume_zero_call_projection_failure(store, args.job_id)
+ else:
+ manager = TieredGraphPatchManager.from_env()
+ if args.command == "enqueue":
+ if args.region:
+ region = store.fast_regions(args.scope_id).get(args.region, [])
+ result = [store.enqueue(args.scope_id, args.region, (item["memory_id"] for item in region), manager=manager)]
+ else:
+ result = store.enqueue_regions(args.scope_id, manager=manager)
+ elif args.command == "audit":
+ result = store.audit(
+ args.scope_id,
+ require_promotion_coverage=args.require_promotion_coverage,
+ )
+ elif args.command == "drain":
+ result = store.drain(manager, batch_size=args.batch_size)
+ elif args.command == "run":
+ result = store.run_job(args.job_id, manager)
+ print(_json(result))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_task_contract.py b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_task_contract.py
new file mode 100644
index 0000000000000000000000000000000000000000..00d3a1a063bead7cbe1e7b509895a4eef164032e
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_task_contract.py
@@ -0,0 +1,641 @@
+"""Strict, benchmark-independent task contracts for TMCRA V4.
+
+The module is deliberately self-contained. A contract describes what a task
+needs, where its premises may come from, and how its result is shaped. It does
+not contain a benchmark route, question type, answer, or model/network code.
+
+Canonical contract shape::
+
+ {
+ "schema_version": "tmcra.task-contract.v4",
+ "output_origin": "memory_direct|memory_derived|...",
+ "target": {
+ "subject": "...",
+ "relation": "...",
+ "entity_constraints": ["..."],
+ "temporal_constraints": ["..."],
+ "state_constraints": ["..."]
+ },
+ "output": {
+ "shape": "scalar|list|set|count|boolean|date|duration|structured|free_text",
+ "cardinality": "one|zero_or_one|one_or_more|zero_or_more",
+ "order": "none|input_order|chronological|reverse_chronological|recency|ranked|question_order"
+ },
+ "premises": [
+ {
+ "premise_id": "P01",
+ "description": "...",
+ "role": "fact|operand|constraint|scope|counterevidence|inventory|state",
+ "necessity": "required|optional",
+ "source": "memory|query_context|model_knowledge|external_tool",
+ "grounded_constraints": ["..."],
+ "context_quote": ""
+ }
+ ],
+ "operations": [
+ {
+ "operation_id": "O01",
+ "operation_type": "date_difference",
+ "input_premise_ids": ["P01"],
+ "output_ref": "TARGET",
+ "parameters": {}
+ }
+ ]
+ }
+
+``operations`` is optional in the input and normalizes to an empty list. All
+other fields shown above are required. ``risk_signals`` are intentionally
+computed by :func:`structural_risk_signals`, rather than trusted from model
+output.
+"""
+
+from __future__ import annotations
+
+import re
+from collections.abc import Mapping, Sequence
+from typing import Any
+
+
+TASK_CONTRACT_SCHEMA = "tmcra.task-contract.v4"
+SCHEMA_VERSION = TASK_CONTRACT_SCHEMA
+
+OUTPUT_ORIGINS = frozenset(
+ {
+ "memory_direct",
+ "memory_derived",
+ "memory_conditioned_generation",
+ "external_required",
+ }
+)
+OUTPUT_SHAPES = frozenset(
+ {
+ "scalar",
+ "list",
+ "set",
+ "count",
+ "boolean",
+ "date",
+ "duration",
+ "structured",
+ "free_text",
+ }
+)
+OUTPUT_CARDINALITIES = frozenset(
+ {"one", "zero_or_one", "one_or_more", "zero_or_more"}
+)
+OUTPUT_ORDERS = frozenset(
+ {
+ "none",
+ "input_order",
+ "chronological",
+ "reverse_chronological",
+ "recency",
+ "ranked",
+ "question_order",
+ }
+)
+PREMISE_SOURCES = frozenset(
+ {"memory", "query_context", "model_knowledge", "external_tool"}
+)
+PREMISE_ROLES = frozenset(
+ {"fact", "operand", "constraint", "scope", "counterevidence", "inventory", "state"}
+)
+PREMISE_NECESSITY = frozenset({"required", "optional"})
+OPERATION_TYPES = frozenset(
+ {
+ "aggregate",
+ "count",
+ "sum",
+ "average",
+ "min",
+ "max",
+ "difference",
+ "date_difference",
+ "date_order",
+ "latest",
+ "latest_state",
+ "ordered_unique_list",
+ "entity_exact_match",
+ "entity_mismatch",
+ "set_difference",
+ "sort",
+ "semantic_composition",
+ "constraint_application",
+ "numeric_sum",
+ "numeric_multiply",
+ "numeric_average",
+ "numeric_difference",
+ "duration_difference",
+ "relative_numeric_offset",
+ "count_distinct",
+ }
+)
+
+PREMISE_ROLE_ALIASES = {
+ "preference": "constraint",
+ "condition": "constraint",
+ "requirement": "constraint",
+ "context": "scope",
+}
+
+RISK_AGGREGATE_WITHOUT_TYPED_INVENTORY = "aggregate_without_typed_inventory"
+RISK_TEMPORAL_WITHOUT_OPERATION = "temporal_without_operation"
+RISK_MULTI_STATE_WITHOUT_LATEST = "multi_state_without_latest"
+RISK_MEMORY_CONDITIONED_WITHOUT_GROUNDED_CONSTRAINTS = (
+ "memory_conditioned_without_grounded_constraints"
+)
+RISK_PLANNER_MISSING_WITH_PLAUSIBLE_SOURCE = (
+ "planner_missing_with_plausible_source"
+)
+RISK_SIGNALS = frozenset(
+ {
+ RISK_AGGREGATE_WITHOUT_TYPED_INVENTORY,
+ RISK_TEMPORAL_WITHOUT_OPERATION,
+ RISK_MULTI_STATE_WITHOUT_LATEST,
+ RISK_MEMORY_CONDITIONED_WITHOUT_GROUNDED_CONSTRAINTS,
+ RISK_PLANNER_MISSING_WITH_PLAUSIBLE_SOURCE,
+ }
+)
+
+_ROOT_REQUIRED = frozenset(
+ {"schema_version", "output_origin", "target", "output", "premises"}
+)
+_ROOT_OPTIONAL = frozenset({"operations", "risk_signals"})
+_TARGET_REQUIRED = frozenset({"subject", "relation", "entity_constraints"})
+_TARGET_OPTIONAL = frozenset({"temporal_constraints", "state_constraints"})
+_OUTPUT_FIELDS = frozenset({"shape", "cardinality", "order"})
+_PREMISE_FIELDS = frozenset(
+ {
+ "premise_id",
+ "description",
+ "role",
+ "necessity",
+ "source",
+ "grounded_constraints",
+ "context_quote",
+ }
+)
+_OPERATION_FIELDS = frozenset(
+ {"operation_id", "operation_type", "input_premise_ids", "output_ref", "parameters"}
+)
+
+_RECOMMENDATION_PATTERN = re.compile(
+ r"\b(?:recommend(?:ation|ations|ed)?|advice|advise|suggest(?:ion|ions|ed)?)\b",
+ re.IGNORECASE,
+)
+_HISTORICAL_RECOMMENDATION_PATTERN = re.compile(
+ r"\b(?:past|previous|previously|earlier|historical|last|remember|remembered|remind|before|was|were|did)\b",
+ re.IGNORECASE,
+)
+_HISTORICAL_RECOMMENDATION_RELATION_PATTERN = re.compile(
+ r"\b(?:recommended|suggested|advised)\b", re.IGNORECASE
+)
+_TEMPORAL_OPERATION_PATTERN = re.compile(
+ r"\b(?:between|elapsed|duration|difference|how long|how many (?:day|days|week|weeks|month|months|year|years)|before|after|earlier|later|chronological|date order)\b",
+ re.IGNORECASE,
+)
+_INVENTORY_PATTERN = re.compile(
+ r"\b(?:count|all|every|list|set|inventory|items|distinct)\b", re.IGNORECASE
+)
+_MULTI_STATE_PATTERN = re.compile(
+ r"\b(?:multi[-_ ]?state|state history|state transition|transitions?|history|"
+ r"changed|changes|previous|prior|current and|old and new|before and after)\b",
+ re.IGNORECASE,
+)
+
+
+class TaskContractError(ValueError):
+ """Raised when a task contract violates its schema or invariants."""
+
+
+ContractValidationError = TaskContractError
+
+
+def _text(value: Any) -> str:
+ return value.strip() if isinstance(value, str) else ""
+
+
+def _mapping(value: Any, path: str) -> Mapping[str, Any]:
+ if not isinstance(value, Mapping):
+ raise TaskContractError(f"{path} must be an object")
+ return value
+
+
+def _exact_fields(
+ value: Mapping[str, Any],
+ *,
+ required: frozenset[str],
+ optional: frozenset[str] = frozenset(),
+ path: str,
+) -> None:
+ fields = set(value)
+ allowed = set(required) | set(optional)
+ if not required.issubset(fields) or not fields.issubset(allowed):
+ raise TaskContractError(f"{path} fields are invalid")
+
+
+def _string(value: Any, path: str) -> str:
+ result = _text(value)
+ if not result:
+ raise TaskContractError(f"{path} must be a non-empty string")
+ return result
+
+
+def _string_list(value: Any, path: str, *, allow_empty: bool = True) -> list[str]:
+ if not isinstance(value, list):
+ raise TaskContractError(f"{path} must be an array")
+ if not allow_empty and not value:
+ raise TaskContractError(f"{path} must not be empty")
+ result: list[str] = []
+ for index, item in enumerate(value):
+ item_text = _string(item, f"{path}[{index}]")
+ if item_text in result:
+ raise TaskContractError(f"{path} contains a duplicate value")
+ result.append(item_text)
+ return result
+
+
+def _bool(value: Any, path: str) -> bool:
+ if not isinstance(value, bool):
+ raise TaskContractError(f"{path} must be a boolean")
+ return value
+
+
+def _recommendation_like(target: Mapping[str, Any]) -> bool:
+ values = [target.get("subject"), target.get("relation")]
+ for key in ("entity_constraints", "temporal_constraints", "state_constraints"):
+ values.extend(target.get(key) or [])
+ text = " ".join(_text(value) for value in values if _text(value))
+ historical = bool(
+ _HISTORICAL_RECOMMENDATION_RELATION_PATTERN.search(text)
+ or _HISTORICAL_RECOMMENDATION_PATTERN.search(text)
+ )
+ return bool(_RECOMMENDATION_PATTERN.search(text) and not historical)
+
+
+def _validate_target(value: Any) -> dict[str, Any]:
+ target = _mapping(value, "target")
+ _exact_fields(target, required=_TARGET_REQUIRED, optional=_TARGET_OPTIONAL, path="target")
+ normalized = {
+ "subject": _string(target.get("subject"), "target.subject"),
+ "relation": _string(target.get("relation"), "target.relation"),
+ "entity_constraints": _string_list(
+ target.get("entity_constraints"), "target.entity_constraints"
+ ),
+ "temporal_constraints": _string_list(
+ target.get("temporal_constraints", []), "target.temporal_constraints"
+ ),
+ "state_constraints": _string_list(
+ target.get("state_constraints", []), "target.state_constraints"
+ ),
+ }
+ return normalized
+
+
+def _validate_output(value: Any) -> dict[str, str]:
+ output = _mapping(value, "output")
+ _exact_fields(output, required=_OUTPUT_FIELDS, path="output")
+ normalized = {
+ "shape": _string(output.get("shape"), "output.shape"),
+ "cardinality": _string(output.get("cardinality"), "output.cardinality"),
+ "order": _string(output.get("order"), "output.order"),
+ }
+ if normalized["shape"] not in OUTPUT_SHAPES:
+ raise TaskContractError("output.shape is invalid")
+ if normalized["cardinality"] not in OUTPUT_CARDINALITIES:
+ raise TaskContractError("output.cardinality is invalid")
+ if normalized["order"] not in OUTPUT_ORDERS:
+ raise TaskContractError("output.order is invalid")
+ if normalized["shape"] == "count" and normalized["order"] != "none":
+ raise TaskContractError("count output must have order=none")
+ return normalized
+
+
+def _validate_premises(value: Any) -> list[dict[str, Any]]:
+ if not isinstance(value, list):
+ raise TaskContractError("premises must be an array")
+ if not value:
+ raise TaskContractError("premises must not be empty")
+ result: list[dict[str, Any]] = []
+ seen: set[str] = set()
+ for index, raw in enumerate(value):
+ path = f"premises[{index}]"
+ premise = _mapping(raw, path)
+ _exact_fields(premise, required=_PREMISE_FIELDS, path=path)
+ premise_id = _string(premise.get("premise_id"), f"{path}.premise_id")
+ if premise_id in seen:
+ raise TaskContractError(f"{path}.premise_id is duplicated")
+ seen.add(premise_id)
+ role = _string(premise.get("role"), f"{path}.role")
+ role = PREMISE_ROLE_ALIASES.get(role, role)
+ necessity = _string(premise.get("necessity"), f"{path}.necessity")
+ source = _string(premise.get("source"), f"{path}.source")
+ if role not in PREMISE_ROLES:
+ raise TaskContractError(f"{path}.role is invalid")
+ if necessity not in PREMISE_NECESSITY:
+ raise TaskContractError(f"{path}.necessity is invalid")
+ if source not in PREMISE_SOURCES:
+ raise TaskContractError(f"{path}.source is invalid")
+ context_quote = premise.get("context_quote")
+ if not isinstance(context_quote, str):
+ raise TaskContractError(f"{path}.context_quote must be a string")
+ context_quote = context_quote.strip()
+ result.append(
+ {
+ "premise_id": premise_id,
+ "description": _string(premise.get("description"), f"{path}.description"),
+ "role": role,
+ "necessity": necessity,
+ "source": source,
+ "grounded_constraints": _string_list(
+ premise.get("grounded_constraints"),
+ f"{path}.grounded_constraints",
+ ),
+ "context_quote": context_quote,
+ }
+ )
+ if not any(item["necessity"] == "required" for item in result):
+ raise TaskContractError("premises needs at least one required item")
+ return result
+
+
+def _validate_operations(value: Any, premise_ids: set[str]) -> list[dict[str, Any]]:
+ if not isinstance(value, list):
+ raise TaskContractError("operations must be an array")
+ result: list[dict[str, Any]] = []
+ seen: set[str] = set()
+ for index, raw in enumerate(value):
+ path = f"operations[{index}]"
+ operation = _mapping(raw, path)
+ _exact_fields(operation, required=_OPERATION_FIELDS, path=path)
+ operation_id = _string(operation.get("operation_id"), f"{path}.operation_id")
+ if operation_id in seen:
+ raise TaskContractError(f"{path}.operation_id is duplicated")
+ seen.add(operation_id)
+ operation_type = _string(operation.get("operation_type"), f"{path}.operation_type")
+ if operation_type not in OPERATION_TYPES:
+ raise TaskContractError(f"{path}.operation_type is invalid")
+ inputs = _string_list(
+ operation.get("input_premise_ids"),
+ f"{path}.input_premise_ids",
+ allow_empty=False,
+ )
+ if not set(inputs).issubset(premise_ids):
+ raise TaskContractError(f"{path}.input_premise_ids contains an unknown premise")
+ output_ref = _string(operation.get("output_ref"), f"{path}.output_ref")
+ parameters = operation.get("parameters")
+ if not isinstance(parameters, Mapping):
+ raise TaskContractError(f"{path}.parameters must be an object")
+ result.append(
+ {
+ "operation_id": operation_id,
+ "operation_type": operation_type,
+ "input_premise_ids": inputs,
+ "output_ref": output_ref,
+ "parameters": dict(parameters),
+ }
+ )
+ return result
+
+
+def _validate_memory_semantics(
+ output_origin: str,
+ target: Mapping[str, Any],
+ premises: Sequence[Mapping[str, Any]],
+) -> None:
+ required_memory = [
+ item
+ for item in premises
+ if item["source"] == "memory" and item["necessity"] == "required"
+ ]
+ if output_origin in {"memory_direct", "memory_derived", "memory_conditioned_generation"}:
+ if not required_memory:
+ raise TaskContractError("memory output origin needs a required memory premise")
+ if output_origin == "external_required" and not any(
+ item["source"] == "external_tool" and item["necessity"] == "required"
+ for item in premises
+ ):
+ raise TaskContractError("external_required needs a required external_tool premise")
+ if _recommendation_like(target):
+ if output_origin != "memory_conditioned_generation":
+ raise TaskContractError(
+ "recommendation/advice must use memory_conditioned_generation"
+ )
+ if not required_memory:
+ raise TaskContractError(
+ "recommendation/advice needs a required memory constraint premise"
+ )
+
+
+def validate_task_contract(value: Mapping[str, Any]) -> dict[str, Any]:
+ """Validate and normalize one contract without I/O or model calls.
+
+ Unknown fields are rejected. The returned object is a fresh plain-dict
+ normalization and never mutates ``value``.
+ """
+
+ if not isinstance(value, Mapping):
+ raise TaskContractError("task contract must be an object")
+ _exact_fields(value, required=_ROOT_REQUIRED, optional=_ROOT_OPTIONAL, path="task contract")
+ if value.get("schema_version") != TASK_CONTRACT_SCHEMA:
+ raise TaskContractError("schema_version is invalid")
+ output_origin = _string(value.get("output_origin"), "output_origin")
+ if output_origin not in OUTPUT_ORIGINS:
+ raise TaskContractError("output_origin is invalid")
+ target = _validate_target(value.get("target"))
+ output = _validate_output(value.get("output"))
+ premises = _validate_premises(value.get("premises"))
+ operations = (
+ _validate_operations(
+ value["operations"], {item["premise_id"] for item in premises}
+ )
+ if "operations" in value
+ else []
+ )
+ _validate_memory_semantics(output_origin, target, premises)
+ if "risk_signals" in value:
+ signals = _string_list(value.get("risk_signals"), "risk_signals")
+ if any(signal not in RISK_SIGNALS for signal in signals):
+ raise TaskContractError("risk_signals contains an unknown signal")
+ normalized = {
+ "schema_version": TASK_CONTRACT_SCHEMA,
+ "output_origin": output_origin,
+ "target": target,
+ "output": output,
+ "premises": premises,
+ "operations": operations,
+ }
+ if "risk_signals" in value:
+ normalized["risk_signals"] = list(value["risk_signals"])
+ return normalized
+
+
+def _target_text(contract: Mapping[str, Any]) -> str:
+ target = contract.get("target")
+ if not isinstance(target, Mapping):
+ return ""
+ values: list[str] = [_text(target.get("subject")), _text(target.get("relation"))]
+ for key in ("entity_constraints", "temporal_constraints", "state_constraints"):
+ raw = target.get(key)
+ if isinstance(raw, Sequence) and not isinstance(raw, (str, bytes)):
+ values.extend(_text(item) for item in raw)
+ return " ".join(value for value in values if value)
+
+
+def _premise_items(contract: Mapping[str, Any]) -> list[Mapping[str, Any]]:
+ raw = contract.get("premises")
+ if not isinstance(raw, list):
+ return []
+ return [item for item in raw if isinstance(item, Mapping)]
+
+
+def _has_typed_inventory(premises: Sequence[Mapping[str, Any]]) -> bool:
+ for premise in premises:
+ if premise.get("role") != "inventory":
+ continue
+ constraints = premise.get("grounded_constraints")
+ if isinstance(constraints, list) and constraints:
+ return True
+ if re.search(r"\b(?:typed|type|entity|inventory)\b", _text(premise.get("description")), re.I):
+ return True
+ return False
+
+
+def _has_temporal_operation(operations: Sequence[Mapping[str, Any]]) -> bool:
+ return any(
+ _text(operation.get("operation_type"))
+ in {"date_difference", "date_order", "latest", "latest_state", "sort"}
+ for operation in operations
+ )
+
+
+def _has_latest_operation(operations: Sequence[Mapping[str, Any]]) -> bool:
+ return any(
+ _text(operation.get("operation_type")) in {"latest", "latest_state"}
+ for operation in operations
+ )
+
+
+def _is_aggregate(contract: Mapping[str, Any], target_text: str) -> bool:
+ output = contract.get("output")
+ shape = output.get("shape") if isinstance(output, Mapping) else ""
+ return shape in {"count", "list", "set"} or bool(_INVENTORY_PATTERN.search(target_text))
+
+
+def _is_temporal(contract: Mapping[str, Any], target_text: str) -> bool:
+ output = contract.get("output")
+ shape = output.get("shape") if isinstance(output, Mapping) else ""
+ return shape in {"date", "duration"} or bool(
+ _TEMPORAL_OPERATION_PATTERN.search(target_text)
+ )
+
+
+def _is_multi_state(contract: Mapping[str, Any], target_text: str) -> bool:
+ target = contract.get("target")
+ if isinstance(target, Mapping):
+ constraints = target.get("state_constraints")
+ if isinstance(constraints, list) and len(constraints) >= 2:
+ return True
+ state_premises = [item for item in _premise_items(contract) if item.get("role") == "state"]
+ return len(state_premises) >= 2 or bool(_MULTI_STATE_PATTERN.search(target_text))
+
+
+def _has_plausible_source(contract: Mapping[str, Any]) -> bool:
+ if any(item.get("source") in PREMISE_SOURCES for item in _premise_items(contract)):
+ return True
+ target = contract.get("target")
+ return isinstance(target, Mapping) and bool(
+ _text(target.get("subject")) and _text(target.get("relation"))
+ )
+
+
+def structural_risk_signals(
+ contract: Mapping[str, Any],
+ *,
+ planner_present: bool = True,
+ plausible_source: bool | None = None,
+) -> list[str]:
+ """Return deterministic structural risk codes for a raw or valid contract.
+
+ This function intentionally does not raise for malformed input: callers can
+ inspect risks before deciding whether to invoke the strict validator. Set
+ ``planner_present=False`` when a planner stage was skipped or its artifact is
+ missing. ``plausible_source`` can override the local source heuristic.
+ """
+
+ target_text = _target_text(contract)
+ premises = _premise_items(contract)
+ operations_raw = contract.get("operations")
+ operations = (
+ [item for item in operations_raw if isinstance(item, Mapping)]
+ if isinstance(operations_raw, list)
+ else []
+ )
+ output_origin = _text(contract.get("output_origin"))
+ risks: list[str] = []
+ if _is_aggregate(contract, target_text) and not _has_typed_inventory(premises):
+ risks.append(RISK_AGGREGATE_WITHOUT_TYPED_INVENTORY)
+ if _is_temporal(contract, target_text) and not _has_temporal_operation(operations):
+ risks.append(RISK_TEMPORAL_WITHOUT_OPERATION)
+ if _is_multi_state(contract, target_text) and not _has_latest_operation(operations):
+ risks.append(RISK_MULTI_STATE_WITHOUT_LATEST)
+ if output_origin == "memory_conditioned_generation":
+ grounded = any(
+ item.get("source") == "memory"
+ and item.get("necessity") == "required"
+ and isinstance(item.get("grounded_constraints"), list)
+ and bool(item.get("grounded_constraints"))
+ for item in premises
+ )
+ if not grounded:
+ risks.append(RISK_MEMORY_CONDITIONED_WITHOUT_GROUNDED_CONSTRAINTS)
+ if not planner_present and (
+ plausible_source if plausible_source is not None else _has_plausible_source(contract)
+ ):
+ risks.append(RISK_PLANNER_MISSING_WITH_PLAUSIBLE_SOURCE)
+ return risks
+
+
+def assess_structural_risks(
+ contract: Mapping[str, Any],
+ *,
+ planner_present: bool = True,
+ plausible_source: bool | None = None,
+) -> list[str]:
+ """Descriptive alias for :func:`structural_risk_signals`."""
+
+ return structural_risk_signals(
+ contract,
+ planner_present=planner_present,
+ plausible_source=plausible_source,
+ )
+
+
+validate_contract = validate_task_contract
+get_structural_risk_signals = structural_risk_signals
+
+
+__all__ = [
+ "TASK_CONTRACT_SCHEMA",
+ "SCHEMA_VERSION",
+ "OUTPUT_ORIGINS",
+ "OUTPUT_SHAPES",
+ "OUTPUT_CARDINALITIES",
+ "OUTPUT_ORDERS",
+ "PREMISE_SOURCES",
+ "PREMISE_ROLES",
+ "OPERATION_TYPES",
+ "RISK_SIGNALS",
+ "RISK_AGGREGATE_WITHOUT_TYPED_INVENTORY",
+ "RISK_TEMPORAL_WITHOUT_OPERATION",
+ "RISK_MULTI_STATE_WITHOUT_LATEST",
+ "RISK_MEMORY_CONDITIONED_WITHOUT_GROUNDED_CONSTRAINTS",
+ "RISK_PLANNER_MISSING_WITH_PLAUSIBLE_SOURCE",
+ "TaskContractError",
+ "ContractValidationError",
+ "validate_task_contract",
+ "validate_contract",
+ "structural_risk_signals",
+ "get_structural_risk_signals",
+ "assess_structural_risks",
+]
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_typed_semantics.py b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_typed_semantics.py
new file mode 100644
index 0000000000000000000000000000000000000000..790f153033fd1b065609cd26fb13bef257afa524
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/legacy/tmcra_v4_typed_semantics.py
@@ -0,0 +1,674 @@
+"""Strict, local typed semantic proposal validation and execution.
+
+This module deliberately has no model, network, or repository dependencies. It
+is an advisory computation layer: accepted results are useful deterministic
+derivations, but neither accepted nor rejected proposals establish absence or
+any other authoritative conclusion.
+
+Input schema (mapping form)
+---------------------------
+An observation must contain ``observation_id``, ``evidence_ids``,
+``entity_key``, ``value_kind``, ``value``, ``unit``, ``temporal_kind``, and
+``polarity``. Event and entity observations may also carry an explicit
+``event_status`` (``actual``, ``planned``, ``hypothetical``, or ``mentioned``).
+``time`` is optional for ordinary observations and required by ``latest``. A candidate contains ``candidate_id`` and ``operations``. Each
+operation contains ``operation_id``, ``operation``, and ``input_ids`` (the
+aliases ``input_observation_ids`` and ``operands`` are also accepted).
+
+The public evaluator accepts dictionaries or the dataclasses below. It returns
+plain dictionaries to keep the result easy to serialize and inspect.
+"""
+
+from __future__ import annotations
+
+from dataclasses import asdict, dataclass
+from datetime import date, datetime
+from decimal import Decimal, InvalidOperation
+import math
+import re
+from typing import Any, Mapping, Sequence
+
+
+VALUE_KINDS = {
+ "entity_instance",
+ "event",
+ "state_snapshot",
+ "cumulative_snapshot",
+ "delta",
+ "aggregated_quantity",
+ "rate",
+ "scalar",
+ "date",
+}
+TEMPORAL_KINDS = {"absolute", "relative_anchored", "relative_unresolved", "none"}
+POLARITIES = {"positive", "negative", "unknown"}
+OPERATIONS = {
+ "latest",
+ "numeric_sum",
+ "numeric_multiply",
+ "numeric_average",
+ "numeric_difference",
+ "duration_difference",
+ "relative_numeric_offset",
+ "count_distinct",
+ "date_order",
+ "date_difference",
+ "entity_exact_match",
+}
+SNAPSHOT_KINDS = {"state_snapshot", "cumulative_snapshot"}
+EVENT_STATUSES = {"actual", "planned", "hypothetical", "mentioned"}
+NUMERIC_KINDS = {"delta", "scalar", "aggregated_quantity"}
+
+
+@dataclass(frozen=True)
+class Observation:
+ """Typed source observation.
+
+ ``time`` is intentionally separate from ``value``. The required schema
+ fields describe the semantic value; temporal operations need an explicit
+ sortable anchor in addition to ``temporal_kind``.
+ """
+
+ observation_id: str
+ evidence_ids: Sequence[str]
+ entity_key: str
+ value_kind: str
+ value: Any
+ unit: str | None
+ temporal_kind: str
+ polarity: str
+ time: Any = None
+ event_status: str | None = None
+
+
+@dataclass(frozen=True)
+class Diagnostic:
+ code: str
+ message: str
+ path: str = ""
+
+ def to_dict(self) -> dict[str, str]:
+ result = {"code": self.code, "message": self.message}
+ if self.path:
+ result["path"] = self.path
+ return result
+
+
+@dataclass(frozen=True)
+class _TypedValue:
+ ref: str
+ value_kind: str
+ value: Any
+ unit: str | None
+ entity_key: str
+ temporal_kind: str
+ time: Any
+ source_evidence_ids: tuple[str, ...]
+ is_observation: bool
+ event_status: str | None = None
+
+
+class _Rejected(Exception):
+ def __init__(self, diagnostic: Diagnostic) -> None:
+ super().__init__(diagnostic.message)
+ self.diagnostic = diagnostic
+
+
+def _diag(code: str, message: str, path: str = "") -> _Rejected:
+ return _Rejected(Diagnostic(code, message, path))
+
+
+def _as_mapping(value: Any, *, path: str) -> Mapping[str, Any]:
+ if not isinstance(value, Mapping):
+ raise _diag("UNTYPED_VALUE", "expected an object with an explicit type", path)
+ return value
+
+
+def _nonempty_text(value: Any, *, code: str, path: str) -> str:
+ if not isinstance(value, str) or not value.strip():
+ raise _diag(code, "expected a non-empty string", path)
+ return value
+
+
+def _evidence_ids(value: Any, *, path: str) -> tuple[str, ...]:
+ if not isinstance(value, (list, tuple)) or not value:
+ raise _diag("UNTYPED_EVIDENCE", "evidence_ids must be a non-empty list", path)
+ result: list[str] = []
+ for index, item in enumerate(value):
+ item_path = f"{path}[{index}]"
+ text = _nonempty_text(item, code="UNTYPED_EVIDENCE", path=item_path)
+ if text not in result:
+ result.append(text)
+ return tuple(result)
+
+
+def _observation_mapping(raw: Any, *, path: str) -> Mapping[str, Any]:
+ if isinstance(raw, Observation):
+ result = asdict(raw)
+ if result.get("event_status") is None:
+ result.pop("event_status", None)
+ return result
+ return _as_mapping(raw, path=path)
+
+
+def _normalize_observation(raw: Any, *, index: int) -> _TypedValue:
+ path = f"observations[{index}]"
+ value = _observation_mapping(raw, path=path)
+ required = (
+ "observation_id",
+ "evidence_ids",
+ "entity_key",
+ "value_kind",
+ "value",
+ "unit",
+ "temporal_kind",
+ "polarity",
+ )
+ missing = [field for field in required if field not in value]
+ if missing:
+ raise _diag("UNTYPED_OBSERVATION", f"missing typed fields: {', '.join(missing)}", path)
+
+ observation_id = _nonempty_text(value["observation_id"], code="UNTYPED_OBSERVATION", path=f"{path}.observation_id")
+ entity_key = _nonempty_text(value["entity_key"], code="UNTYPED_OBSERVATION", path=f"{path}.entity_key")
+ value_kind = _nonempty_text(value["value_kind"], code="UNTYPED_OBSERVATION", path=f"{path}.value_kind")
+ temporal_kind = _nonempty_text(value["temporal_kind"], code="UNTYPED_OBSERVATION", path=f"{path}.temporal_kind")
+ polarity = _nonempty_text(value["polarity"], code="UNTYPED_OBSERVATION", path=f"{path}.polarity")
+ if value_kind not in VALUE_KINDS:
+ raise _diag("UNKNOWN_VALUE_KIND", f"unsupported value_kind: {value_kind!r}", f"{path}.value_kind")
+ if temporal_kind not in TEMPORAL_KINDS:
+ raise _diag("UNKNOWN_TEMPORAL_KIND", f"unsupported temporal_kind: {temporal_kind!r}", f"{path}.temporal_kind")
+ if polarity not in POLARITIES:
+ raise _diag("UNKNOWN_POLARITY", f"unsupported polarity: {polarity!r}", f"{path}.polarity")
+ unit = value["unit"]
+ if unit is not None and (not isinstance(unit, str) or not unit.strip()):
+ raise _diag("UNTYPED_UNIT", "unit must be a non-empty string or null", f"{path}.unit")
+ evidence_ids = _evidence_ids(value["evidence_ids"], path=f"{path}.evidence_ids")
+ status_fields = [field for field in ("event_status", "event_state", "status") if field in value]
+ statuses = [value[field] for field in status_fields]
+ if any(status != statuses[0] for status in statuses[1:]):
+ raise _diag("CONFLICTING_EVENT_STATUS", "event status aliases disagree", f"{path}.event_status")
+ event_status = None
+ if statuses:
+ event_status = _nonempty_text(statuses[0], code="UNTYPED_EVENT_STATUS", path=f"{path}.{status_fields[0]}").strip().lower()
+ if event_status not in EVENT_STATUSES:
+ raise _diag("UNKNOWN_EVENT_STATUS", f"unsupported event_status: {event_status!r}", f"{path}.{status_fields[0]}")
+ if value_kind == "event" and event_status is None:
+ raise _diag("MISSING_EVENT_STATUS", "event observations require an explicit event_status", f"{path}.event_status")
+ time_value = value.get("time", value.get("temporal_value", value.get("timestamp", value.get("observed_at"))))
+ return _TypedValue(
+ ref=observation_id,
+ value_kind=value_kind,
+ value=value["value"],
+ unit=unit.strip() if isinstance(unit, str) else None,
+ entity_key=entity_key,
+ temporal_kind=temporal_kind,
+ time=time_value,
+ source_evidence_ids=evidence_ids,
+ is_observation=True,
+ event_status=event_status,
+ )
+
+
+def _iter_values(raw: Any, *, singular_fields: set[str]) -> list[Any]:
+ if isinstance(raw, Mapping):
+ if singular_fields.intersection(raw):
+ return [raw]
+ return list(raw.values())
+ if isinstance(raw, (list, tuple)):
+ return list(raw)
+ return [raw]
+
+
+def _normalize_observations(raw: Any) -> tuple[dict[str, _TypedValue], list[Diagnostic]]:
+ observations: dict[str, _TypedValue] = {}
+ diagnostics: list[Diagnostic] = []
+ items = _iter_values(raw, singular_fields={"observation_id", "value_kind"})
+ for index, item in enumerate(items):
+ try:
+ normalized = _normalize_observation(item, index=index)
+ if normalized.ref in observations:
+ diagnostics.append(Diagnostic("DUPLICATE_OBSERVATION_ID", f"duplicate observation_id: {normalized.ref}", f"observations[{index}].observation_id"))
+ else:
+ observations[normalized.ref] = normalized
+ except _Rejected as exc:
+ diagnostics.append(exc.diagnostic)
+ return observations, diagnostics
+
+
+def _decimal(value: Any, *, path: str) -> Decimal:
+ if isinstance(value, bool) or not isinstance(value, (int, float, Decimal, str)):
+ raise _diag("NON_NUMERIC_VALUE", "operation requires numeric values", path)
+ try:
+ result = Decimal(str(value))
+ except (InvalidOperation, ValueError):
+ raise _diag("NON_NUMERIC_VALUE", "operation requires numeric values", path)
+ if not result.is_finite():
+ raise _diag("NON_NUMERIC_VALUE", "numeric values must be finite", path)
+ return result
+
+
+def _json_number(value: Decimal) -> int | float:
+ if value == value.to_integral_value():
+ return int(value)
+ return float(value)
+
+
+def _unit(value: str | None) -> str | None:
+ if value is None:
+ return None
+ normalized = re.sub(r"\s+", " ", value.strip().lower())
+ normalized = normalized.replace(" per ", "/")
+ normalized = normalized.replace("usd", "$" ).replace("us dollars", "$")
+ return normalized.replace(" ", "")
+
+
+def _rate_parts(value: str | None) -> tuple[str, str] | None:
+ normalized = _unit(value)
+ if normalized is None or normalized.count("/") != 1:
+ return None
+ numerator, denominator = normalized.split("/", 1)
+ if not numerator or not denominator:
+ return None
+ return numerator, denominator
+
+
+def _join_sources(values: Sequence[_TypedValue]) -> tuple[str, ...]:
+ result: list[str] = []
+ for item in values:
+ for evidence_id in item.source_evidence_ids:
+ if evidence_id not in result:
+ result.append(evidence_id)
+ return tuple(result)
+
+
+def _date_value(value: Any, *, path: str) -> date | datetime:
+ if isinstance(value, (date, datetime)):
+ return value
+ if not isinstance(value, str) or not value.strip():
+ raise _diag("INVALID_DATE", "date operation requires an ISO date or datetime value", path)
+ text = value.strip()
+ try:
+ if "T" in text or " " in text:
+ return datetime.fromisoformat(text.replace("Z", "+00:00"))
+ return date.fromisoformat(text)
+ except ValueError:
+ raise _diag("INVALID_DATE", "date operation requires an ISO date or datetime value", path)
+
+
+def _sortable_time(value: _TypedValue, *, path: str) -> tuple[str, Any]:
+ if value.time is None:
+ raise _diag("MISSING_TIME", "latest requires an explicit sortable time", path)
+ if isinstance(value.time, (int, float, Decimal)) and not isinstance(value.time, bool):
+ return "number", Decimal(str(value.time))
+ if isinstance(value.time, datetime):
+ return "datetime", value.time
+ if isinstance(value.time, date):
+ return "date", value.time
+ if isinstance(value.time, str):
+ text = value.time.strip()
+ try:
+ if "T" in text or " " in text:
+ return "datetime", datetime.fromisoformat(text.replace("Z", "+00:00"))
+ return "date", date.fromisoformat(text)
+ except ValueError:
+ raise _diag("UNORDERABLE_TIME", "latest requires an ISO or numeric time anchor", path)
+ raise _diag("UNORDERABLE_TIME", "latest requires an ISO or numeric time anchor", path)
+
+
+def _resolve_inputs(
+ raw_ids: Any,
+ *,
+ observations: Mapping[str, _TypedValue],
+ results: Mapping[str, _TypedValue],
+ path: str,
+) -> list[_TypedValue]:
+ if not isinstance(raw_ids, (list, tuple)) or not raw_ids:
+ raise _diag("UNTYPED_OPERANDS", "operation inputs must be a non-empty list", path)
+ resolved: list[_TypedValue] = []
+ for index, ref in enumerate(raw_ids):
+ ref_path = f"{path}[{index}]"
+ if not isinstance(ref, str) or not ref.strip():
+ raise _diag("UNTYPED_OPERAND", "operation input must be an explicit reference", ref_path)
+ if ref in observations:
+ resolved.append(observations[ref])
+ elif ref in results:
+ resolved.append(results[ref])
+ else:
+ raise _diag("UNKNOWN_REFERENCE", f"unknown observation or operation reference: {ref}", ref_path)
+ return resolved
+
+
+def _operation_name(raw: Mapping[str, Any], *, path: str) -> str:
+ values = [raw.get(field) for field in ("operation", "operation_type", "op") if field in raw]
+ if not values or not isinstance(values[0], str) or not values[0].strip():
+ raise _diag("UNTYPED_OPERATION", "operation must name a typed deterministic operation", path)
+ if any(value != values[0] for value in values[1:]):
+ raise _diag("CONFLICTING_OPERATION_TYPE", "operation aliases disagree", path)
+ operation = values[0]
+ if operation not in OPERATIONS:
+ raise _diag("UNKNOWN_OPERATION", f"unsupported operation: {operation!r}", path)
+ return operation
+
+
+def _input_field(raw: Mapping[str, Any], *, path: str) -> Any:
+ fields = ("input_ids", "input_observation_ids", "observation_ids", "operands", "inputs")
+ present = [field for field in fields if field in raw]
+ if not present:
+ raise _diag("UNTYPED_OPERANDS", "operation must explicitly list its inputs", path)
+ first = raw[present[0]]
+ if any(raw[field] != first for field in present[1:]):
+ raise _diag("CONFLICTING_OPERANDS", "operation input aliases disagree", path)
+ return first
+
+
+def _derived(
+ operation_id: str,
+ value_kind: str,
+ value: Any,
+ unit: str | None,
+ source_values: Sequence[_TypedValue],
+ *,
+ entity_key: str = "derived",
+ temporal_kind: str = "none",
+ time: Any = None,
+) -> _TypedValue:
+ return _TypedValue(
+ ref=operation_id,
+ value_kind=value_kind,
+ value=value,
+ unit=unit,
+ entity_key=entity_key,
+ temporal_kind=temporal_kind,
+ time=time,
+ source_evidence_ids=_join_sources(source_values),
+ is_observation=False,
+ event_status=None,
+ )
+
+
+def _numeric_operands(
+ operation: str,
+ inputs: Sequence[_TypedValue],
+ *,
+ path: str,
+ allowed_kinds: set[str] = NUMERIC_KINDS,
+) -> tuple[list[Decimal], str]:
+ if any(item.value_kind not in allowed_kinds for item in inputs):
+ allowed = ", ".join(sorted(allowed_kinds))
+ raise _diag("INVALID_NUMERIC_ROLE", f"{operation} accepts only {allowed} values", path)
+ units = {_unit(item.unit) for item in inputs}
+ if len(units) != 1 or None in units:
+ raise _diag("INCOMPATIBLE_UNITS", f"{operation} requires non-null, identical units", path)
+ return [_decimal(item.value, path=path) for item in inputs], inputs[0].unit
+
+
+def _execute_operation(
+ operation: str,
+ operation_id: str,
+ inputs: Sequence[_TypedValue],
+ parameters: Mapping[str, Any],
+ *,
+ path: str,
+) -> _TypedValue:
+ if operation == "count_distinct":
+ if any(item.value_kind not in {"entity_instance", "event"} for item in inputs):
+ raise _diag("INVALID_COUNT_ROLE", "count_distinct accepts only entity_instance or event values; snapshots and aggregated quantities are not countable", path)
+ countable: list[_TypedValue] = []
+ for index, item in enumerate(inputs):
+ if item.event_status is None:
+ raise _diag("MISSING_EVENT_STATUS", "count_distinct requires an explicit event_status on every event/entity input", f"{path}.inputs[{index}].event_status")
+ if item.event_status == "actual":
+ countable.append(item)
+ return _derived(operation_id, "scalar", len({item.entity_key for item in countable}), "count", countable)
+
+ if operation == "numeric_sum":
+ if any(item.value_kind not in {"delta", "scalar"} for item in inputs):
+ raise _diag("INVALID_SUM_ROLE", "numeric_sum accepts only additive delta or scalar values, never rates, snapshots, or aggregated quantities", path)
+ values, unit = _numeric_operands(operation, inputs, path=path, allowed_kinds={"delta", "scalar"})
+ total = sum(values, Decimal(0))
+ return _derived(operation_id, "scalar", _json_number(total), unit, inputs)
+
+ if operation in {"numeric_average", "numeric_difference", "duration_difference", "relative_numeric_offset"}:
+ if operation in {"numeric_difference", "duration_difference", "relative_numeric_offset"} and len(inputs) != 2:
+ raise _diag("INVALID_NUMERIC_ARITY", f"{operation} requires exactly two numeric values", path)
+ values, unit = _numeric_operands(operation, inputs, path=path)
+ if operation == "numeric_average":
+ result = sum(values, Decimal(0)) / Decimal(len(values))
+ else:
+ result = values[0] - values[1]
+ return _derived(operation_id, "scalar", _json_number(result), unit, inputs)
+
+ if operation == "numeric_multiply":
+ if len(inputs) != 2:
+ raise _diag("INVALID_MULTIPLY_ARITY", "numeric_multiply requires exactly one rate and one quantity", path)
+ rates = [item for item in inputs if item.value_kind == "rate"]
+ quantities = [item for item in inputs if item.value_kind != "rate"]
+ if len(rates) != 1 or len(quantities) != 1:
+ raise _diag("INVALID_MULTIPLY_ROLE", "numeric_multiply requires exactly one rate and one compatible quantity", path)
+ numerator_denominator = _rate_parts(rates[0].unit)
+ if numerator_denominator is None:
+ raise _diag("INVALID_RATE_UNIT", "rate unit must have the form numerator/denominator", path)
+ numerator, denominator = numerator_denominator
+ quantity = quantities[0]
+ if quantity.value_kind not in {"scalar", "aggregated_quantity", "delta"}:
+ raise _diag("INVALID_QUANTITY_ROLE", "the non-rate operand must be a scalar, aggregated quantity, or delta", path)
+ if _unit(quantity.unit) != denominator:
+ raise _diag("INCOMPATIBLE_RATE_QUANTITY", "quantity unit must exactly match the rate denominator", path)
+ product = _decimal(rates[0].value, path=path) * _decimal(quantity.value, path=path)
+ return _derived(operation_id, "scalar", _json_number(product), numerator, inputs)
+
+ if operation == "latest":
+ if any(item.value_kind not in SNAPSHOT_KINDS for item in inputs):
+ raise _diag("INVALID_LATEST_ROLE", "latest requires state_snapshot or cumulative_snapshot values", path)
+ if any(item.temporal_kind not in {"absolute", "relative_anchored"} for item in inputs):
+ raise _diag("INVALID_LATEST_TIME", "latest requires absolute or relative_anchored temporal kinds", path)
+ entities = {item.entity_key for item in inputs}
+ if len(entities) != 1:
+ raise _diag("MIXED_ENTITIES", "latest requires snapshots for one exact entity_key", path)
+ sortable = [_sortable_time(item, path=f"{path}.inputs[{index}]") for index, item in enumerate(inputs)]
+ kinds = {kind for kind, _ in sortable}
+ if len(kinds) != 1:
+ raise _diag("INCOMPARABLE_TIMES", "latest requires mutually comparable time anchors", path)
+ latest_index = max(range(len(inputs)), key=lambda index: sortable[index][1])
+ selected = inputs[latest_index]
+ return _derived(
+ operation_id,
+ selected.value_kind,
+ selected.value,
+ selected.unit,
+ inputs,
+ entity_key=selected.entity_key,
+ temporal_kind=selected.temporal_kind,
+ time=selected.time,
+ )
+
+ if operation in {"date_order", "date_difference"}:
+ if len(inputs) != 2:
+ raise _diag("INVALID_DATE_ARITY", f"{operation} requires exactly two dates", path)
+ if any(item.value_kind != "date" for item in inputs):
+ raise _diag("INVALID_DATE_ROLE", f"{operation} accepts only date values", path)
+ if any(item.temporal_kind == "relative_unresolved" for item in inputs):
+ raise _diag("UNRESOLVED_RELATIVE_DATE", "date operations reject unresolved relative dates", path)
+ first = _date_value(inputs[0].value, path=f"{path}.inputs[0]")
+ second = _date_value(inputs[1].value, path=f"{path}.inputs[1]")
+ try:
+ difference = first - second
+ except TypeError:
+ raise _diag("INCOMPARABLE_DATES", "date values must use compatible timezone information", path)
+ if operation == "date_difference":
+ days = difference.total_seconds() / 86400
+ if isinstance(first, date) and isinstance(second, date) and not isinstance(first, datetime) and not isinstance(second, datetime):
+ days = difference.days
+ return _derived(operation_id, "scalar", _json_number(Decimal(str(days))), "day", inputs)
+ relation = parameters.get("relation", parameters.get("comparison", "before"))
+ if relation not in {"before", "lt", "earlier", "after", "gt", "later", "equal", "eq", "same"}:
+ raise _diag("UNKNOWN_DATE_RELATION", "date_order relation must be before, after, or equal", f"{path}.parameters")
+ try:
+ if relation in {"before", "lt", "earlier"}:
+ result = first < second
+ elif relation in {"after", "gt", "later"}:
+ result = first > second
+ else:
+ result = first == second
+ except TypeError:
+ raise _diag("INCOMPARABLE_DATES", "date values must use compatible timezone information", path)
+ return _derived(operation_id, "scalar", result, None, inputs)
+
+ if operation == "entity_exact_match":
+ if len(inputs) != 2:
+ raise _diag("INVALID_ENTITY_ARITY", "entity_exact_match requires exactly two entities", path)
+ if any(item.value_kind != "entity_instance" for item in inputs):
+ raise _diag("INVALID_ENTITY_ROLE", "entity_exact_match requires entity_instance values", path)
+ return _derived(operation_id, "scalar", inputs[0].entity_key == inputs[1].entity_key, None, inputs)
+
+ raise _diag("UNKNOWN_OPERATION", f"unsupported operation: {operation!r}", path)
+
+
+def _candidate_id(raw: Any, *, index: int) -> str:
+ if isinstance(raw, Mapping) and isinstance(raw.get("candidate_id"), str) and raw["candidate_id"].strip():
+ return raw["candidate_id"]
+ return f"candidate_{index}"
+
+
+def _candidate_mapping(raw: Any, *, path: str) -> Mapping[str, Any]:
+ if not isinstance(raw, Mapping):
+ raise _diag("UNTYPED_PROPOSAL", "candidate program must be an object with typed operations", path)
+ if "candidate_id" not in raw:
+ raise _diag("UNTYPED_PROPOSAL", "candidate program requires candidate_id", path)
+ if not isinstance(raw.get("candidate_id"), str) or not raw["candidate_id"].strip():
+ raise _diag("UNTYPED_PROPOSAL", "candidate_id must be a non-empty string", f"{path}.candidate_id")
+ if not isinstance(raw.get("operations"), (list, tuple)) or not raw["operations"]:
+ raise _diag("UNTYPED_PROPOSAL", "candidate program requires a non-empty operations list", f"{path}.operations")
+ return raw
+
+
+def _candidate_source_ids(raw: Mapping[str, Any]) -> tuple[str, ...]:
+ value = raw.get("source_evidence_ids", raw.get("evidence_ids", []))
+ if not isinstance(value, (list, tuple)):
+ return ()
+ return tuple(item for item in value if isinstance(item, str) and item.strip())
+
+
+def _evaluate_candidate(raw: Any, *, index: int, observations: Mapping[str, _TypedValue]) -> dict[str, Any]:
+ candidate_id = _candidate_id(raw, index=index)
+ sources = list(_candidate_source_ids(raw) if isinstance(raw, Mapping) else ())
+ base = {"candidate_id": candidate_id, "authoritative": False, "source_evidence_ids": sources}
+ try:
+ candidate = _candidate_mapping(raw, path=f"candidates[{index}]")
+ results: dict[str, _TypedValue] = {}
+ operation_results: list[dict[str, Any]] = []
+ operation_ids: set[str] = set()
+ for operation_index, raw_operation in enumerate(candidate["operations"]):
+ path = f"candidates[{index}].operations[{operation_index}]"
+ operation_map = _as_mapping(raw_operation, path=path)
+ operation_id = _nonempty_text(operation_map.get("operation_id"), code="UNTYPED_OPERATION", path=f"{path}.operation_id")
+ if operation_id in operation_ids:
+ raise _diag("DUPLICATE_OPERATION_ID", f"duplicate operation_id: {operation_id}", f"{path}.operation_id")
+ operation_ids.add(operation_id)
+ operation = _operation_name(operation_map, path=f"{path}.operation")
+ raw_inputs = _input_field(operation_map, path=f"{path}.input_ids")
+ inputs = _resolve_inputs(raw_inputs, observations=observations, results=results, path=f"{path}.input_ids")
+ for evidence_id in _join_sources(inputs):
+ if evidence_id not in sources:
+ sources.append(evidence_id)
+ parameters = operation_map.get("parameters", {})
+ if not isinstance(parameters, Mapping):
+ raise _diag("UNTYPED_PARAMETERS", "operation parameters must be an object", f"{path}.parameters")
+ result = _execute_operation(operation, operation_id, inputs, parameters, path=path)
+ results[operation_id] = result
+ for evidence_id in result.source_evidence_ids:
+ if evidence_id not in sources:
+ sources.append(evidence_id)
+ operation_results.append({
+ "operation_id": operation_id,
+ "operation": operation,
+ "value": result.value,
+ "value_kind": result.value_kind,
+ "unit": result.unit,
+ "source_evidence_ids": list(result.source_evidence_ids),
+ })
+
+ output_ref = candidate.get("output_ref", candidate.get("output_operation_id"))
+ if output_ref is None:
+ output_ref = candidate["operations"][-1].get("operation_id") if isinstance(candidate["operations"][-1], Mapping) else None
+ if not isinstance(output_ref, str) or output_ref not in results:
+ raise _diag("UNKNOWN_OUTPUT_REFERENCE", "candidate output_ref must name an operation result", f"candidates[{index}].output_ref")
+ output = results[output_ref]
+ return {
+ **base,
+ "status": "accepted",
+ "value": output.value,
+ "value_kind": output.value_kind,
+ "unit": output.unit,
+ "result": operation_results[-1] if output_ref == operation_results[-1]["operation_id"] else next(item for item in operation_results if item["operation_id"] == output_ref),
+ "operation_results": operation_results,
+ "source_evidence_ids": sources,
+ "evidence_ids": list(sources),
+ }
+ except _Rejected as exc:
+ return {
+ **base,
+ "status": "rejected",
+ "diagnostics": [exc.diagnostic.to_dict()],
+ "source_evidence_ids": sources,
+ "evidence_ids": list(sources),
+ }
+
+
+def evaluate_proposals(observations: Any, proposals: Any) -> dict[str, Any]:
+ """Validate and execute typed candidate programs without authoritative claims.
+
+ Each candidate is evaluated independently. Malformed observations make
+ candidates that reference them reject with diagnostics; they are never
+ converted into an absence or ``not found`` result.
+ """
+
+ normalized, observation_diagnostics = _normalize_observations(observations)
+ candidates = _iter_values(proposals, singular_fields={"candidate_id", "operations"})
+ accepted: list[dict[str, Any]] = []
+ rejected: list[dict[str, Any]] = []
+ for index, candidate in enumerate(candidates):
+ result = _evaluate_candidate(candidate, index=index, observations=normalized)
+ if observation_diagnostics:
+ references = set()
+ if isinstance(candidate, Mapping):
+ for operation in candidate.get("operations", []) or []:
+ if isinstance(operation, Mapping):
+ for key in ("input_ids", "input_observation_ids", "observation_ids", "operands", "inputs"):
+ value = operation.get(key, [])
+ if isinstance(value, (list, tuple)):
+ references.update(item for item in value if isinstance(item, str))
+ if references.intersection(normalized) or not normalized:
+ result.setdefault("diagnostics", []).extend(item.to_dict() for item in observation_diagnostics)
+ if result["status"] == "accepted":
+ result["status"] = "rejected"
+ result.pop("value", None)
+ result.pop("value_kind", None)
+ result.pop("unit", None)
+ result.pop("result", None)
+ result.pop("operation_results", None)
+ rejected.append(result)
+ continue
+ if result["status"] == "accepted":
+ accepted.append(result)
+ else:
+ rejected.append(result)
+ return {
+ "status": "evaluated",
+ "advisory": True,
+ "authoritative": False,
+ "accepted": accepted,
+ "rejected": rejected,
+ }
+
+
+__all__ = [
+ "Diagnostic",
+ "EVENT_STATUSES",
+ "Observation",
+ "OPERATIONS",
+ "POLARITIES",
+ "TEMPORAL_KINDS",
+ "VALUE_KINDS",
+ "evaluate_proposals",
+]
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/__init__.py b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/__init__.py
new file mode 100644
index 0000000000000000000000000000000000000000..9f259846956f7beebaacd67614abab1b8f40f3fe
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/__init__.py
@@ -0,0 +1 @@
+"""Minimal graph/session adapter required by the benchmark."""
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/core/__init__.py b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/core/__init__.py
new file mode 100644
index 0000000000000000000000000000000000000000..cc1c7ec69a66a2b80fe3a63e5f89e76f250a72a7
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/core/__init__.py
@@ -0,0 +1,89 @@
+"""TMCRA core package exports.
+
+Keep package-level imports best-effort so focused submodules such as the
+object-sketch trainer can run without pulling in every optional runtime
+dependency from the broader app stack.
+"""
+
+__all__ = []
+
+try:
+ from .maze_engine import MazeEdge, MazeNode, MazePath, TriMazeEngine
+
+ __all__.extend(["MazeNode", "MazeEdge", "MazePath", "TriMazeEngine"])
+except Exception: # pragma: no cover - optional dependency
+ MazeNode = None
+ MazeEdge = None
+ MazePath = None
+ TriMazeEngine = None
+
+try:
+ from .native_concept_extractor import NativeConceptExtractor, NativeExtractorConfig
+
+ __all__.extend(["NativeConceptExtractor", "NativeExtractorConfig"])
+except Exception: # pragma: no cover - optional dependency
+ NativeConceptExtractor = None
+ NativeExtractorConfig = None
+
+try:
+ from .concept_graph import ConceptGraph
+
+ __all__.append("ConceptGraph")
+except Exception: # pragma: no cover - optional dependency
+ ConceptGraph = None
+
+try:
+ from .concept_memory import ConceptMemory
+
+ __all__.append("ConceptMemory")
+except Exception: # pragma: no cover - optional dependency
+ ConceptMemory = None
+
+try:
+ from .query_understanding import QueryUnderstandingLayer
+
+ __all__.append("QueryUnderstandingLayer")
+except Exception: # pragma: no cover - optional dependency
+ QueryUnderstandingLayer = None
+
+try:
+ from .gru_text_generator import GRUTextGenerator
+
+ __all__.append("GRUTextGenerator")
+except Exception: # pragma: no cover - optional dependency
+ GRUTextGenerator = None
+
+try:
+ from .policy_network import EdgePolicy
+
+ __all__.append("EdgePolicy")
+except Exception: # pragma: no cover - optional dependency
+ EdgePolicy = None
+
+try:
+ from .policy_dataset import (
+ CurriculumConfig,
+ PolicyBatch,
+ PolicyStepDataset,
+ PolicyStepRecord,
+ PolicyVocabulary,
+ )
+ from .policy_network import PolicyModelConfig
+
+ __all__.extend(
+ [
+ "CurriculumConfig",
+ "PolicyBatch",
+ "PolicyModelConfig",
+ "PolicyStepDataset",
+ "PolicyStepRecord",
+ "PolicyVocabulary",
+ ]
+ )
+except Exception: # pragma: no cover - optional dependency
+ CurriculumConfig = None
+ PolicyBatch = None
+ PolicyModelConfig = None
+ PolicyStepDataset = None
+ PolicyStepRecord = None
+ PolicyVocabulary = None
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/core/gru_text_generator.py b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/core/gru_text_generator.py
new file mode 100644
index 0000000000000000000000000000000000000000..e432545b77f4bba6b7dcf5fc2b78f03c2f9aca10
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/core/gru_text_generator.py
@@ -0,0 +1,142 @@
+"""
+GRU-based character generator (self-trained, no external embeddings/LLM).
+"""
+from __future__ import annotations
+
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Dict, List, Tuple, Iterable, Optional
+
+try:
+ import torch
+ from torch import nn
+except Exception: # pragma: no cover - optional runtime dependency
+ torch = None
+ nn = None
+
+
+SPECIAL_TOKENS = ["", "", "", "", ""]
+
+
+@dataclass
+class GRUConfig:
+ embed_dim: int = 128
+ hidden_dim: int = 256
+ num_layers: int = 2
+ dropout: float = 0.1
+
+
+class CharVocab:
+ def __init__(self, tokens: List[str]):
+ self.tokens = tokens
+ self.token_to_id = {t: i for i, t in enumerate(tokens)}
+ self.id_to_token = {i: t for i, t in enumerate(tokens)}
+
+ @classmethod
+ def build(cls, corpus_iter: Iterable[str]) -> "CharVocab":
+ chars = set()
+ for text in corpus_iter:
+ chars.update(list(text))
+ tokens = SPECIAL_TOKENS + sorted(chars)
+ return cls(tokens)
+
+ def encode(self, text: str) -> List[int]:
+ return [self.token_to_id.get(ch, self.token_to_id[""]) for ch in text]
+
+ def decode(self, ids: Iterable[int]) -> str:
+ return "".join(self.id_to_token.get(i, "") for i in ids)
+
+
+if nn is not None:
+ class GRUCharModel(nn.Module):
+ def __init__(self, vocab_size: int, config: GRUConfig):
+ super().__init__()
+ self.embedding = nn.Embedding(vocab_size, config.embed_dim)
+ self.gru = nn.GRU(
+ input_size=config.embed_dim,
+ hidden_size=config.hidden_dim,
+ num_layers=config.num_layers,
+ dropout=config.dropout if config.num_layers > 1 else 0.0,
+ batch_first=True,
+ )
+ self.fc = nn.Linear(config.hidden_dim, vocab_size)
+
+ def forward(self, x, hidden=None):
+ emb = self.embedding(x)
+ out, hidden = self.gru(emb, hidden)
+ logits = self.fc(out)
+ return logits, hidden
+else: # pragma: no cover - fallback when torch is unavailable
+ class GRUCharModel:
+ def __init__(self, *args, **kwargs):
+ raise RuntimeError("torch not available; install torch to use GRUCharModel")
+
+
+class GRUTextGenerator:
+ def __init__(self, vocab: CharVocab, config: GRUConfig | None = None):
+ if torch is None:
+ raise RuntimeError("torch not available; install torch to use GRUTextGenerator")
+ self.vocab = vocab
+ self.config = config or GRUConfig()
+ self.model = GRUCharModel(len(vocab.tokens), self.config)
+
+ @classmethod
+ def load(cls, model_path: str | Path) -> "GRUTextGenerator":
+ if torch is None:
+ raise RuntimeError("torch not available; install torch to load GRUTextGenerator")
+ data = torch.load(str(model_path), map_location="cpu", weights_only=True)
+ vocab = CharVocab(data["vocab"])
+ config = GRUConfig(**data["config"])
+ gen = cls(vocab, config)
+ gen.model.load_state_dict(data["state_dict"])
+ gen.model.eval()
+ return gen
+
+ def save(self, model_path: str | Path) -> None:
+ data = {
+ "vocab": self.vocab.tokens,
+ "config": self.config.__dict__,
+ "state_dict": self.model.state_dict(),
+ }
+ Path(model_path).parent.mkdir(parents=True, exist_ok=True)
+ torch.save(data, str(model_path))
+
+ def _prefix_ids(self, path_text: str) -> List[int]:
+ prefix = "" + path_text + ""
+ return [self.vocab.token_to_id[""]] + self.vocab.encode(prefix)
+
+ def generate(
+ self,
+ path_text: str,
+ max_len: int = 400,
+ temperature: float = 0.9,
+ device: str = "cpu",
+ ) -> str:
+ self.model.to(device)
+ self.model.eval()
+ ids = self._prefix_ids(path_text)
+ input_ids = torch.tensor([ids], dtype=torch.long, device=device)
+ hidden = None
+ generated: List[int] = []
+ for _ in range(max_len):
+ logits, hidden = self.model(input_ids, hidden)
+ next_logits = logits[:, -1, :] / max(temperature, 1e-6)
+ probs = torch.softmax(next_logits, dim=-1)
+ next_id = torch.multinomial(probs, num_samples=1).item()
+ if next_id == self.vocab.token_to_id[""]:
+ break
+ generated.append(next_id)
+ input_ids = torch.tensor([[next_id]], dtype=torch.long, device=device)
+ return self.vocab.decode(generated)
+
+ @staticmethod
+ def build_corpus_iter(corpus_path: str | Path, max_chars: int | None = None) -> Iterable[str]:
+ total = 0
+ with open(corpus_path, "r", encoding="utf-8", errors="ignore") as handle:
+ for line in handle:
+ if not line.strip():
+ continue
+ if max_chars and total >= max_chars:
+ break
+ total += len(line)
+ yield line.strip()
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/core/native_concept_extractor.py b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/core/native_concept_extractor.py
new file mode 100644
index 0000000000000000000000000000000000000000..0e4288b4b3e391d3b4fd5cc6fa16e323fee774c8
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/core/native_concept_extractor.py
@@ -0,0 +1,187 @@
+"""
+Native concept extraction based on character n-grams and co-occurrence.
+No rules/KB/embeddings/LLM are used.
+"""
+from __future__ import annotations
+
+import math
+from collections import Counter, defaultdict
+from dataclasses import dataclass
+from typing import Dict, List, Tuple
+
+
+@dataclass
+class NativeExtractorConfig:
+ ngram_min: int = 2
+ ngram_max: int = 4
+ window_size: int = 10
+ min_freq: int = 2
+ top_k: int = 80
+ entropy_threshold: float = 0.8
+ alpha: float = 0.5 # PMI weight
+ beta: float = 0.3 # Jaccard weight
+ gamma: float = 0.2 # Frequency ratio weight
+
+
+class NativeConceptExtractor:
+ """Extracts concept nodes and directed relations using only statistics."""
+
+ def __init__(self, config: NativeExtractorConfig | None = None) -> None:
+ self.config = config or NativeExtractorConfig()
+
+ def _normalize_text(self, text: str) -> str:
+ # Keep characters but remove whitespace for stable n-gram extraction.
+ return "".join(ch for ch in text if not ch.isspace())
+
+ def _entropy(self, counter: Counter) -> float:
+ total = sum(counter.values())
+ if total <= 0:
+ return 0.0
+ ent = 0.0
+ for count in counter.values():
+ p = count / total
+ ent -= p * math.log(p + 1e-12)
+ return ent
+
+ def _collect_ngrams(self, text: str) -> Tuple[Counter, Dict[str, Counter], Dict[str, Counter]]:
+ counts: Counter = Counter()
+ left_ctx: Dict[str, Counter] = defaultdict(Counter)
+ right_ctx: Dict[str, Counter] = defaultdict(Counter)
+ length = len(text)
+ for i in range(length):
+ for n in range(self.config.ngram_min, self.config.ngram_max + 1):
+ j = i + n
+ if j > length:
+ break
+ gram = text[i:j]
+ counts[gram] += 1
+ if i > 0:
+ left_ctx[gram][text[i - 1]] += 1
+ if j < length:
+ right_ctx[gram][text[j]] += 1
+ return counts, left_ctx, right_ctx
+
+ def _select_concepts(
+ self,
+ counts: Counter,
+ left_ctx: Dict[str, Counter],
+ right_ctx: Dict[str, Counter],
+ ) -> List[str]:
+ scored = []
+ for gram, freq in counts.items():
+ if freq < self.config.min_freq:
+ continue
+ l_ent = self._entropy(left_ctx.get(gram, Counter()))
+ r_ent = self._entropy(right_ctx.get(gram, Counter()))
+ ent = (l_ent + r_ent) / 2.0
+ if ent < self.config.entropy_threshold:
+ continue
+ score = freq * (1.0 + ent)
+ scored.append((score, gram))
+ scored.sort(key=lambda item: item[0], reverse=True)
+ return [gram for _, gram in scored[: self.config.top_k]]
+
+ def _scan_positions(self, text: str, concepts: set[str]) -> Tuple[Dict[int, List[str]], Counter]:
+ positions: Dict[int, List[str]] = defaultdict(list)
+ occ_counts: Counter = Counter()
+ length = len(text)
+ for i in range(length):
+ for n in range(self.config.ngram_min, self.config.ngram_max + 1):
+ j = i + n
+ if j > length:
+ break
+ gram = text[i:j]
+ if gram in concepts:
+ positions[i].append(gram)
+ occ_counts[gram] += 1
+ return positions, occ_counts
+
+ def _build_cooccurrence(
+ self, positions: Dict[int, List[str]], occ_counts: Counter
+ ) -> Tuple[Dict[Tuple[str, str], int], Dict[str, Dict[str, int]]]:
+ pair_counts: Dict[Tuple[str, str], int] = defaultdict(int)
+ context_profile: Dict[str, Dict[str, int]] = defaultdict(lambda: defaultdict(int))
+ indices = sorted(positions.keys())
+ idx_set = set(indices)
+ for i in indices:
+ current = positions[i]
+ if not current:
+ continue
+ for j in range(i + 1, i + self.config.window_size + 1):
+ if j not in idx_set:
+ continue
+ future = positions[j]
+ if not future:
+ continue
+ for src in current:
+ for dst in future:
+ if src == dst:
+ continue
+ pair_counts[(src, dst)] += 1
+ context_profile[src][dst] += 1
+ return pair_counts, context_profile
+
+ def _normalize_scores(self, values: List[float]) -> List[float]:
+ if not values:
+ return []
+ v_min = min(values)
+ v_max = max(values)
+ if abs(v_max - v_min) < 1e-12:
+ return [0.0 for _ in values]
+ return [(v - v_min) / (v_max - v_min) for v in values]
+
+ def _compute_weights(
+ self,
+ pair_counts: Dict[Tuple[str, str], int],
+ occ_counts: Counter,
+ ) -> Dict[Tuple[str, str], float]:
+ total_pairs = sum(pair_counts.values()) or 1
+ pmi_vals: Dict[Tuple[str, str], float] = {}
+ for (src, dst), count in pair_counts.items():
+ pmi = math.log((count * total_pairs) / ((occ_counts[src] * occ_counts[dst]) + 1e-9) + 1e-9)
+ pmi_vals[(src, dst)] = max(0.0, pmi)
+
+ max_pmi = max(pmi_vals.values()) if pmi_vals else 1.0
+ weights: Dict[Tuple[str, str], float] = {}
+ for (src, dst), count in pair_counts.items():
+ pmi_norm = (pmi_vals.get((src, dst), 0.0) / max_pmi) if max_pmi > 0 else 0.0
+ jaccard = count / (occ_counts[src] + occ_counts[dst] - count + 1e-9)
+ ratio = count / (min(occ_counts[src], occ_counts[dst]) + 1e-9)
+ weight = (
+ self.config.alpha * pmi_norm
+ + self.config.beta * jaccard
+ + self.config.gamma * ratio
+ )
+ weights[(src, dst)] = max(0.0, min(1.0, weight))
+ return weights
+
+ def extract(self, text: str) -> Dict:
+ cleaned = self._normalize_text(text)
+ if not cleaned:
+ return {"concepts": [], "relations": [], "contexts": {}}
+
+ counts, left_ctx, right_ctx = self._collect_ngrams(cleaned)
+ concepts = self._select_concepts(counts, left_ctx, right_ctx)
+ if not concepts:
+ return {"concepts": [], "relations": [], "contexts": {}}
+
+ concept_set = set(concepts)
+ positions, occ_counts = self._scan_positions(cleaned, concept_set)
+ pair_counts, context_profile = self._build_cooccurrence(positions, occ_counts)
+ weights = self._compute_weights(pair_counts, occ_counts)
+
+ relations = []
+ for (src, dst), weight in weights.items():
+ relations.append({
+ "from": src,
+ "to": dst,
+ "relation": "co_occurs",
+ "weight": float(weight),
+ })
+
+ concept_records = [{"concept": c, "type": "ngram"} for c in concepts]
+ return {
+ "concepts": concept_records,
+ "relations": relations,
+ "contexts": {k: dict(v) for k, v in context_profile.items()},
+ }
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/core/policy_dataset.py b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/core/policy_dataset.py
new file mode 100644
index 0000000000000000000000000000000000000000..51e2d8c7aa6532e28557d1575f735316a4e789e1
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/core/policy_dataset.py
@@ -0,0 +1,804 @@
+from __future__ import annotations
+
+import hashlib
+import json
+import math
+from dataclasses import asdict, dataclass, field
+from pathlib import Path
+from types import SimpleNamespace
+from typing import Any, Iterator, Sequence
+
+try:
+ import torch
+ from torch.utils.data import Dataset, Sampler
+except Exception as exc: # pragma: no cover - torch is required for trainer usage
+ raise RuntimeError(f"torch is required for Tri-Maze policy datasets: {exc}")
+
+
+BASE_FEATURE_DIM = 13
+PATH_CONTEXT_DIM = 6
+CANDIDATE_FLAG_DIM = 4
+DEFAULT_HISTORY_SIZE = 4
+FEATURE_SCHEMA_VERSION = "tri_maze_policy_v2.0"
+PAD_ID = 0
+UNK_ID = 1
+
+
+def _normalize_text(value: Any) -> str:
+ if value is None:
+ return ""
+ return " ".join(str(value).replace("\u00a0", " ").strip().split())
+
+
+def _stable_hash(text: str) -> int:
+ digest = hashlib.md5(text.encode("utf-8", errors="ignore")).digest()
+ return int.from_bytes(digest[:8], "big")
+
+
+def _hash_bucket(text: str, bucket_count: int) -> int:
+ if bucket_count <= 0:
+ return 0
+ normalized = _normalize_text(text)
+ if not normalized:
+ return 0
+ return 1 + (_stable_hash(normalized.casefold()) % bucket_count)
+
+
+def _clamp01(value: Any, default: float = 0.0) -> float:
+ try:
+ return max(0.0, min(1.0, float(value)))
+ except Exception:
+ return float(default)
+
+
+def _mode_one_hot(mode: str) -> list[float]:
+ normalized = (_normalize_text(mode) or "forward").lower()
+ if normalized == "reverse":
+ return [0.0, 1.0, 0.0]
+ if normalized == "boundary":
+ return [0.0, 0.0, 1.0]
+ return [1.0, 0.0, 0.0]
+
+
+def compute_candidate_base_features(
+ engine: Any,
+ current_node: Any,
+ edge: Any,
+ *,
+ path: Any = None,
+ visited: set[str] | None = None,
+ mode: str = "forward",
+ max_degree: int | None = None,
+) -> list[float]:
+ visited = visited or set()
+ next_node = edge.to_node
+ degree_cur = len(getattr(current_node, "connections", []) or [])
+ degree_next = len(getattr(next_node, "connections", []) or [])
+ max_degree = max(1, int(max_degree or getattr(engine, "_max_degree", 1) or 1))
+
+ resistance = float(getattr(edge, "resistance", 0.5) or 0.5)
+ memory_reinf = 0.0
+ if getattr(engine, "memory", None):
+ try:
+ memory_reinf = float(
+ engine.memory.get_edge_reinforcement(current_node.concept, next_node.concept)
+ )
+ except Exception:
+ memory_reinf = 0.0
+
+ semantic = 0.0
+ if hasattr(engine, "_semantic_similarity"):
+ try:
+ semantic = float(engine._semantic_similarity(current_node.concept, next_node.concept))
+ except Exception:
+ semantic = 0.0
+
+ visited_flag = 1.0 if next_node.concept in visited else 0.0
+ path_len = float(getattr(path, "length", 0) or 0)
+ max_steps = float(getattr(engine, "max_exploration_steps", 1) or 1)
+
+ features = [
+ resistance,
+ memory_reinf,
+ semantic,
+ min(1.0, degree_cur / max_degree),
+ min(1.0, degree_next / max_degree),
+ min(1.0, path_len / max_steps),
+ visited_flag,
+ 1.0 if getattr(edge, "is_memory", False) else 0.0,
+ 1.0 if getattr(edge, "is_expanded", False) else 0.0,
+ 1.0 if getattr(edge, "is_tunneling", False) else 0.0,
+ ]
+ features.extend(_mode_one_hot(mode))
+ return features
+
+
+def build_path_context_features(
+ *,
+ path_length: int,
+ visited_count: int,
+ candidate_count: int,
+ max_degree: int,
+ history_count: int,
+ revisit_ratio: float,
+ max_steps: int | None = None,
+) -> list[float]:
+ max_degree = max(1, int(max_degree or 1))
+ max_steps = max(1, int(max_steps or 80))
+ path_length_norm = min(1.0, max(0, path_length) / max_steps)
+ candidate_density = min(1.0, max(0, candidate_count) / max_degree)
+ visited_density = min(1.0, max(0, visited_count) / max(1, path_length + candidate_count + 1))
+ branch_density = min(1.0, max(0, candidate_count) / max(1, visited_count + candidate_count))
+ history_fill = min(1.0, max(0, history_count) / max(1, DEFAULT_HISTORY_SIZE))
+ revisit_ratio = _clamp01(revisit_ratio)
+ return [
+ path_length_norm,
+ candidate_density,
+ visited_density,
+ branch_density,
+ history_fill,
+ revisit_ratio,
+ ]
+
+
+def _path_length_bucket(path_length: int) -> int:
+ if path_length <= 1:
+ return 0
+ if path_length <= 3:
+ return 1
+ if path_length <= 5:
+ return 2
+ return 3
+
+
+@dataclass(slots=True)
+class CurriculumConfig:
+ enabled: bool = False
+ min_fraction: float = 0.35
+ max_fraction: float = 1.0
+ include_tunneling_after: float = 0.45
+ include_hard_after: float = 0.55
+
+ def fraction_for_epoch(self, epoch: int, total_epochs: int) -> float:
+ if not self.enabled:
+ return self.max_fraction
+ total_epochs = max(1, int(total_epochs))
+ progress = min(1.0, max(0.0, float(epoch) / max(1, total_epochs - 1)))
+ return self.min_fraction + (self.max_fraction - self.min_fraction) * progress
+
+
+@dataclass(slots=True)
+class PolicyStepRecord:
+ sample_id: str
+ source_kind: str
+ source_dataset: str
+ query_type: str
+ mode: str
+ task_key: str
+ current_concept: str
+ recent_concepts: tuple[str, ...]
+ visited_concepts: tuple[str, ...]
+ candidate_concepts: tuple[str, ...]
+ candidate_relations: tuple[str, ...]
+ candidate_base_features: tuple[tuple[float, ...], ...]
+ candidate_is_memory: tuple[int, ...]
+ candidate_is_expanded: tuple[int, ...]
+ candidate_is_tunneling: tuple[int, ...]
+ target_index: int
+ weight: float
+ path_length: int
+ path_length_bucket: int
+ difficulty: float
+ high_value_target: int
+ has_tunneling_path: int
+ source_score: float = 0.0
+ metadata: dict[str, Any] = field(default_factory=dict)
+
+ def to_dict(self) -> dict[str, Any]:
+ payload = asdict(self)
+ for key in (
+ "recent_concepts",
+ "visited_concepts",
+ "candidate_concepts",
+ "candidate_relations",
+ "candidate_is_memory",
+ "candidate_is_expanded",
+ "candidate_is_tunneling",
+ ):
+ payload[key] = list(payload[key])
+ payload["candidate_base_features"] = [list(item) for item in self.candidate_base_features]
+ return payload
+
+ @classmethod
+ def from_dict(cls, payload: dict[str, Any]) -> "PolicyStepRecord":
+ return cls(
+ sample_id=_normalize_text(payload.get("sample_id") or ""),
+ source_kind=_normalize_text(payload.get("source_kind") or "unknown"),
+ source_dataset=_normalize_text(payload.get("source_dataset") or "unknown"),
+ query_type=_normalize_text(payload.get("query_type") or "query"),
+ mode=_normalize_text(payload.get("mode") or "forward"),
+ task_key=_normalize_text(payload.get("task_key") or "default"),
+ current_concept=_normalize_text(payload.get("current_concept") or ""),
+ recent_concepts=tuple(_normalize_text(item) for item in payload.get("recent_concepts") or [] if _normalize_text(item)),
+ visited_concepts=tuple(_normalize_text(item) for item in payload.get("visited_concepts") or [] if _normalize_text(item)),
+ candidate_concepts=tuple(_normalize_text(item) for item in payload.get("candidate_concepts") or [] if _normalize_text(item)),
+ candidate_relations=tuple(_normalize_text(item) for item in payload.get("candidate_relations") or [] if _normalize_text(item)),
+ candidate_base_features=tuple(
+ tuple(float(value) for value in feature_row)
+ for feature_row in payload.get("candidate_base_features") or []
+ ),
+ candidate_is_memory=tuple(int(value) for value in payload.get("candidate_is_memory") or []),
+ candidate_is_expanded=tuple(int(value) for value in payload.get("candidate_is_expanded") or []),
+ candidate_is_tunneling=tuple(int(value) for value in payload.get("candidate_is_tunneling") or []),
+ target_index=int(payload.get("target_index", -1)),
+ weight=float(payload.get("weight", 1.0) or 1.0),
+ path_length=int(payload.get("path_length", 0) or 0),
+ path_length_bucket=int(payload.get("path_length_bucket", _path_length_bucket(int(payload.get("path_length", 0) or 0)))),
+ difficulty=float(payload.get("difficulty", 0.0) or 0.0),
+ high_value_target=int(payload.get("high_value_target", 0) or 0),
+ has_tunneling_path=int(payload.get("has_tunneling_path", 0) or 0),
+ source_score=float(payload.get("source_score", 0.0) or 0.0),
+ metadata=dict(payload.get("metadata") or {}),
+ )
+
+ def all_concepts(self) -> set[str]:
+ concepts = {self.current_concept}
+ concepts.update(self.recent_concepts)
+ concepts.update(self.visited_concepts)
+ concepts.update(self.candidate_concepts)
+ return {item for item in concepts if item}
+
+
+@dataclass(slots=True)
+class PolicyVocabulary:
+ concept_to_id: dict[str, int]
+ relation_to_id: dict[str, int]
+ source_to_id: dict[str, int]
+ query_type_to_id: dict[str, int]
+ mode_to_id: dict[str, int]
+ task_to_id: dict[str, int]
+ hash_bucket_size: int = 8192
+
+ @classmethod
+ def empty(cls, *, hash_bucket_size: int = 8192) -> "PolicyVocabulary":
+ return cls(
+ concept_to_id={"": PAD_ID, "": UNK_ID},
+ relation_to_id={"": PAD_ID, "": UNK_ID},
+ source_to_id={"": PAD_ID, "": UNK_ID},
+ query_type_to_id={"": PAD_ID, "": UNK_ID},
+ mode_to_id={"": PAD_ID, "": UNK_ID, "forward": 2, "reverse": 3, "boundary": 4, "runtime": 5},
+ task_to_id={"": PAD_ID, "": UNK_ID},
+ hash_bucket_size=max(32, int(hash_bucket_size)),
+ )
+
+ @classmethod
+ def build(
+ cls,
+ records: Sequence[PolicyStepRecord],
+ *,
+ hash_bucket_size: int = 8192,
+ ) -> "PolicyVocabulary":
+ vocab = cls.empty(hash_bucket_size=hash_bucket_size)
+ for record in records:
+ vocab._add(vocab.concept_to_id, record.current_concept)
+ for concept in record.recent_concepts:
+ vocab._add(vocab.concept_to_id, concept)
+ for concept in record.visited_concepts:
+ vocab._add(vocab.concept_to_id, concept)
+ for concept in record.candidate_concepts:
+ vocab._add(vocab.concept_to_id, concept)
+ for relation in record.candidate_relations:
+ vocab._add(vocab.relation_to_id, relation)
+ vocab._add(vocab.source_to_id, record.source_dataset)
+ vocab._add(vocab.query_type_to_id, record.query_type)
+ vocab._add(vocab.mode_to_id, record.mode)
+ vocab._add(vocab.task_to_id, record.task_key)
+ return vocab
+
+ @staticmethod
+ def _key(value: str) -> str:
+ normalized = _normalize_text(value)
+ return normalized.casefold() if normalized else ""
+
+ def _add(self, mapping: dict[str, int], value: str) -> None:
+ key = self._key(value)
+ if not key or key in mapping:
+ return
+ mapping[key] = len(mapping)
+
+ def encode_token(self, mapping: dict[str, int], value: str) -> int:
+ key = self._key(value)
+ if not key:
+ return PAD_ID
+ return int(mapping.get(key, UNK_ID))
+
+ def encode_concept(self, value: str) -> tuple[int, int]:
+ return (
+ self.encode_token(self.concept_to_id, value),
+ _hash_bucket(value, self.hash_bucket_size),
+ )
+
+ def encode_relation(self, value: str) -> int:
+ return self.encode_token(self.relation_to_id, value)
+
+ def encode_source(self, value: str) -> int:
+ return self.encode_token(self.source_to_id, value)
+
+ def encode_query_type(self, value: str) -> int:
+ return self.encode_token(self.query_type_to_id, value)
+
+ def encode_mode(self, value: str) -> int:
+ return self.encode_token(self.mode_to_id, value)
+
+ def encode_task(self, value: str) -> int:
+ return self.encode_token(self.task_to_id, value)
+
+ @property
+ def concept_vocab_size(self) -> int:
+ return len(self.concept_to_id)
+
+ @property
+ def relation_vocab_size(self) -> int:
+ return len(self.relation_to_id)
+
+ @property
+ def source_vocab_size(self) -> int:
+ return len(self.source_to_id)
+
+ @property
+ def query_type_vocab_size(self) -> int:
+ return len(self.query_type_to_id)
+
+ @property
+ def mode_vocab_size(self) -> int:
+ return len(self.mode_to_id)
+
+ @property
+ def task_vocab_size(self) -> int:
+ return len(self.task_to_id)
+
+ def to_metadata(self) -> dict[str, Any]:
+ return {
+ "concept_to_id": self.concept_to_id,
+ "relation_to_id": self.relation_to_id,
+ "source_to_id": self.source_to_id,
+ "query_type_to_id": self.query_type_to_id,
+ "mode_to_id": self.mode_to_id,
+ "task_to_id": self.task_to_id,
+ "hash_bucket_size": self.hash_bucket_size,
+ }
+
+ @classmethod
+ def from_metadata(cls, payload: dict[str, Any] | None) -> "PolicyVocabulary":
+ if not payload:
+ return cls.empty()
+ return cls(
+ concept_to_id={str(k): int(v) for k, v in dict(payload.get("concept_to_id") or {}).items()},
+ relation_to_id={str(k): int(v) for k, v in dict(payload.get("relation_to_id") or {}).items()},
+ source_to_id={str(k): int(v) for k, v in dict(payload.get("source_to_id") or {}).items()},
+ query_type_to_id={str(k): int(v) for k, v in dict(payload.get("query_type_to_id") or {}).items()},
+ mode_to_id={str(k): int(v) for k, v in dict(payload.get("mode_to_id") or {}).items()},
+ task_to_id={str(k): int(v) for k, v in dict(payload.get("task_to_id") or {}).items()},
+ hash_bucket_size=max(32, int(payload.get("hash_bucket_size", 8192) or 8192)),
+ )
+
+
+@dataclass(slots=True)
+class PolicyBatch:
+ sample_ids: list[str]
+ source_kinds: list[str]
+ current_concept_ids: torch.Tensor
+ current_hash_ids: torch.Tensor
+ history_concept_ids: torch.Tensor
+ history_hash_ids: torch.Tensor
+ candidate_concept_ids: torch.Tensor
+ candidate_hash_ids: torch.Tensor
+ relation_ids: torch.Tensor
+ source_ids: torch.Tensor
+ query_type_ids: torch.Tensor
+ mode_ids: torch.Tensor
+ task_ids: torch.Tensor
+ base_features: torch.Tensor
+ path_context: torch.Tensor
+ candidate_flags: torch.Tensor
+ candidate_mask: torch.Tensor
+ target_index: torch.Tensor
+ weights: torch.Tensor
+ path_length_bucket: torch.Tensor
+ tunnel_label: torch.Tensor
+ high_value_label: torch.Tensor
+ difficulty: torch.Tensor
+ candidate_count: torch.Tensor
+
+ def to(self, device: torch.device | str) -> "PolicyBatch":
+ for field_name in self.__dataclass_fields__:
+ value = getattr(self, field_name)
+ if torch.is_tensor(value):
+ setattr(self, field_name, value.to(device))
+ return self
+
+ @property
+ def batch_size(self) -> int:
+ return int(self.current_concept_ids.shape[0])
+
+
+class PolicyStepDataset(Dataset):
+ def __init__(
+ self,
+ records: Sequence[PolicyStepRecord],
+ *,
+ vocabulary: PolicyVocabulary | None = None,
+ history_size: int = DEFAULT_HISTORY_SIZE,
+ feature_schema_version: str = FEATURE_SCHEMA_VERSION,
+ ) -> None:
+ self.records = list(records)
+ self.history_size = max(1, int(history_size))
+ self.feature_schema_version = feature_schema_version
+ self.vocabulary = vocabulary or PolicyVocabulary.build(self.records)
+ self._encoded = [self._encode_record(record) for record in self.records]
+
+ def __len__(self) -> int:
+ return len(self.records)
+
+ def __getitem__(self, index: int) -> dict[str, Any]:
+ return self._encoded[index]
+
+ def _encode_record(self, record: PolicyStepRecord) -> dict[str, Any]:
+ vocab = self.vocabulary
+ history = list(record.recent_concepts[-self.history_size :])
+ while len(history) < self.history_size:
+ history.insert(0, "")
+
+ current_id, current_hash = vocab.encode_concept(record.current_concept)
+ history_ids: list[int] = []
+ history_hash_ids: list[int] = []
+ for concept in history:
+ concept_id, hash_id = vocab.encode_concept(concept)
+ history_ids.append(concept_id)
+ history_hash_ids.append(hash_id)
+
+ candidate_ids: list[int] = []
+ candidate_hash_ids: list[int] = []
+ relation_ids: list[int] = []
+ candidate_flags: list[list[float]] = []
+ candidate_features: list[list[float]] = []
+ candidate_count = len(record.candidate_concepts)
+ visited_set = {item.casefold() for item in record.visited_concepts}
+ revisit_hits = 0
+ for index, concept in enumerate(record.candidate_concepts):
+ concept_id, hash_id = vocab.encode_concept(concept)
+ candidate_ids.append(concept_id)
+ candidate_hash_ids.append(hash_id)
+ relation_ids.append(vocab.encode_relation(record.candidate_relations[index] if index < len(record.candidate_relations) else "related_to"))
+ visited_flag = 1.0 if concept.casefold() in visited_set else 0.0
+ revisit_hits += int(visited_flag > 0.0)
+ candidate_flags.append(
+ [
+ float(record.candidate_is_memory[index] if index < len(record.candidate_is_memory) else 0),
+ float(record.candidate_is_expanded[index] if index < len(record.candidate_is_expanded) else 0),
+ float(record.candidate_is_tunneling[index] if index < len(record.candidate_is_tunneling) else 0),
+ visited_flag,
+ ]
+ )
+ feature_row = list(record.candidate_base_features[index] if index < len(record.candidate_base_features) else ())
+ if len(feature_row) < BASE_FEATURE_DIM:
+ feature_row.extend([0.0] * (BASE_FEATURE_DIM - len(feature_row)))
+ candidate_features.append(feature_row[:BASE_FEATURE_DIM])
+
+ path_context = build_path_context_features(
+ path_length=record.path_length,
+ visited_count=len(record.visited_concepts),
+ candidate_count=candidate_count,
+ max_degree=max(candidate_count, 1),
+ history_count=len(record.recent_concepts),
+ revisit_ratio=float(revisit_hits / max(1, candidate_count)),
+ )
+
+ return {
+ "sample_id": record.sample_id,
+ "source_kind": record.source_kind,
+ "current_concept_id": current_id,
+ "current_hash_id": current_hash,
+ "history_concept_ids": history_ids,
+ "history_hash_ids": history_hash_ids,
+ "candidate_concept_ids": candidate_ids,
+ "candidate_hash_ids": candidate_hash_ids,
+ "relation_ids": relation_ids,
+ "source_id": vocab.encode_source(record.source_dataset),
+ "query_type_id": vocab.encode_query_type(record.query_type),
+ "mode_id": vocab.encode_mode(record.mode),
+ "task_id": vocab.encode_task(record.task_key),
+ "base_features": candidate_features,
+ "path_context": path_context,
+ "candidate_flags": candidate_flags,
+ "target_index": int(record.target_index),
+ "weight": float(record.weight),
+ "path_length_bucket": int(record.path_length_bucket),
+ "tunnel_label": int(record.has_tunneling_path),
+ "high_value_label": int(record.high_value_target),
+ "difficulty": float(record.difficulty),
+ "candidate_count": int(candidate_count),
+ }
+
+
+def policy_collate_fn(batch: Sequence[dict[str, Any]]) -> PolicyBatch:
+ if not batch:
+ raise ValueError("cannot collate empty policy batch")
+
+ batch_size = len(batch)
+ max_candidates = max(1, max(int(item["candidate_count"]) for item in batch))
+ history_size = max(1, len(batch[0]["history_concept_ids"]))
+
+ current_concept_ids = torch.zeros(batch_size, dtype=torch.long)
+ current_hash_ids = torch.zeros(batch_size, dtype=torch.long)
+ history_concept_ids = torch.zeros(batch_size, history_size, dtype=torch.long)
+ history_hash_ids = torch.zeros(batch_size, history_size, dtype=torch.long)
+ candidate_concept_ids = torch.zeros(batch_size, max_candidates, dtype=torch.long)
+ candidate_hash_ids = torch.zeros(batch_size, max_candidates, dtype=torch.long)
+ relation_ids = torch.zeros(batch_size, max_candidates, dtype=torch.long)
+ source_ids = torch.zeros(batch_size, dtype=torch.long)
+ query_type_ids = torch.zeros(batch_size, dtype=torch.long)
+ mode_ids = torch.zeros(batch_size, dtype=torch.long)
+ task_ids = torch.zeros(batch_size, dtype=torch.long)
+ base_features = torch.zeros(batch_size, max_candidates, BASE_FEATURE_DIM, dtype=torch.float32)
+ path_context = torch.zeros(batch_size, PATH_CONTEXT_DIM, dtype=torch.float32)
+ candidate_flags = torch.zeros(batch_size, max_candidates, CANDIDATE_FLAG_DIM, dtype=torch.float32)
+ candidate_mask = torch.zeros(batch_size, max_candidates, dtype=torch.bool)
+ target_index = torch.full((batch_size,), -1, dtype=torch.long)
+ weights = torch.ones(batch_size, dtype=torch.float32)
+ path_length_bucket = torch.zeros(batch_size, dtype=torch.long)
+ tunnel_label = torch.zeros(batch_size, dtype=torch.float32)
+ high_value_label = torch.zeros(batch_size, dtype=torch.float32)
+ difficulty = torch.zeros(batch_size, dtype=torch.float32)
+ candidate_count = torch.zeros(batch_size, dtype=torch.long)
+
+ sample_ids: list[str] = []
+ source_kinds: list[str] = []
+
+ for row_index, row in enumerate(batch):
+ count = int(row["candidate_count"])
+ sample_ids.append(str(row["sample_id"]))
+ source_kinds.append(str(row["source_kind"]))
+ current_concept_ids[row_index] = int(row["current_concept_id"])
+ current_hash_ids[row_index] = int(row["current_hash_id"])
+ history_concept_ids[row_index] = torch.tensor(row["history_concept_ids"], dtype=torch.long)
+ history_hash_ids[row_index] = torch.tensor(row["history_hash_ids"], dtype=torch.long)
+ if count > 0:
+ candidate_concept_ids[row_index, :count] = torch.tensor(row["candidate_concept_ids"], dtype=torch.long)
+ candidate_hash_ids[row_index, :count] = torch.tensor(row["candidate_hash_ids"], dtype=torch.long)
+ relation_ids[row_index, :count] = torch.tensor(row["relation_ids"], dtype=torch.long)
+ base_features[row_index, :count] = torch.tensor(row["base_features"], dtype=torch.float32)
+ candidate_flags[row_index, :count] = torch.tensor(row["candidate_flags"], dtype=torch.float32)
+ candidate_mask[row_index, :count] = True
+ source_ids[row_index] = int(row["source_id"])
+ query_type_ids[row_index] = int(row["query_type_id"])
+ mode_ids[row_index] = int(row["mode_id"])
+ task_ids[row_index] = int(row["task_id"])
+ path_context[row_index] = torch.tensor(row["path_context"], dtype=torch.float32)
+ target_index[row_index] = int(row["target_index"])
+ weights[row_index] = float(row["weight"])
+ path_length_bucket[row_index] = int(row["path_length_bucket"])
+ tunnel_label[row_index] = float(row["tunnel_label"])
+ high_value_label[row_index] = float(row["high_value_label"])
+ difficulty[row_index] = float(row["difficulty"])
+ candidate_count[row_index] = int(count)
+
+ return PolicyBatch(
+ sample_ids=sample_ids,
+ source_kinds=source_kinds,
+ current_concept_ids=current_concept_ids,
+ current_hash_ids=current_hash_ids,
+ history_concept_ids=history_concept_ids,
+ history_hash_ids=history_hash_ids,
+ candidate_concept_ids=candidate_concept_ids,
+ candidate_hash_ids=candidate_hash_ids,
+ relation_ids=relation_ids,
+ source_ids=source_ids,
+ query_type_ids=query_type_ids,
+ mode_ids=mode_ids,
+ task_ids=task_ids,
+ base_features=base_features,
+ path_context=path_context,
+ candidate_flags=candidate_flags,
+ candidate_mask=candidate_mask,
+ target_index=target_index,
+ weights=weights,
+ path_length_bucket=path_length_bucket,
+ tunnel_label=tunnel_label,
+ high_value_label=high_value_label,
+ difficulty=difficulty,
+ candidate_count=candidate_count,
+ )
+
+
+class EpisodicBatchSampler(Sampler[list[int]]):
+ def __init__(self, records: Sequence[PolicyStepRecord], *, batch_size: int, seed: int = 42) -> None:
+ self.batch_size = max(1, int(batch_size))
+ self.seed = int(seed)
+ task_groups: dict[str, list[int]] = {}
+ for index, record in enumerate(records):
+ task_groups.setdefault(record.task_key or "default", []).append(index)
+ self.task_groups = {key: value for key, value in task_groups.items() if value}
+ self.task_keys = list(self.task_groups.keys())
+
+ def __iter__(self) -> Iterator[list[int]]:
+ if not self.task_keys:
+ return iter(())
+ generator = torch.Generator().manual_seed(self.seed)
+ task_order = torch.randperm(len(self.task_keys), generator=generator).tolist()
+ batches: list[list[int]] = []
+ for task_index in task_order:
+ indices = self.task_groups[self.task_keys[task_index]].copy()
+ if len(indices) > 1:
+ perm = torch.randperm(len(indices), generator=generator).tolist()
+ indices = [indices[item] for item in perm]
+ for start in range(0, len(indices), self.batch_size):
+ batches.append(indices[start : start + self.batch_size])
+ return iter(batches)
+
+ def __len__(self) -> int:
+ total = 0
+ for indices in self.task_groups.values():
+ total += math.ceil(len(indices) / self.batch_size)
+ return total
+
+
+def build_domain_sampling_weights(records: Sequence[PolicyStepRecord]) -> list[float]:
+ if not records:
+ return []
+ counts: dict[tuple[str, str], int] = {}
+ for record in records:
+ key = (
+ _normalize_text(record.source_dataset).casefold() or "unknown",
+ _normalize_text(record.query_type).casefold() or "query",
+ )
+ counts[key] = counts.get(key, 0) + 1
+ weights: list[float] = []
+ for record in records:
+ key = (
+ _normalize_text(record.source_dataset).casefold() or "unknown",
+ _normalize_text(record.query_type).casefold() or "query",
+ )
+ weights.append(1.0 / max(1, counts.get(key, 1)))
+ return weights
+
+
+def filter_curriculum_records(
+ records: Sequence[PolicyStepRecord],
+ *,
+ epoch: int,
+ total_epochs: int,
+ curriculum: CurriculumConfig,
+) -> list[PolicyStepRecord]:
+ if not curriculum.enabled or not records:
+ return list(records)
+ ordered = sorted(records, key=lambda item: (item.difficulty, item.path_length, item.sample_id))
+ cutoff = max(1, int(math.ceil(len(ordered) * curriculum.fraction_for_epoch(epoch, total_epochs))))
+ allowed = ordered[:cutoff]
+ progress = min(1.0, max(0.0, float(epoch) / max(1, total_epochs - 1)))
+ if progress >= curriculum.include_tunneling_after:
+ tunneling_records = [record for record in records if record.has_tunneling_path and record not in allowed]
+ allowed.extend(tunneling_records)
+ if progress >= curriculum.include_hard_after:
+ hard_records = [record for record in records if len(record.candidate_concepts) > 3 and record not in allowed]
+ allowed.extend(hard_records)
+ seen = set()
+ deduped: list[PolicyStepRecord] = []
+ for record in allowed:
+ if record.sample_id in seen:
+ continue
+ seen.add(record.sample_id)
+ deduped.append(record)
+ return deduped
+
+
+def serialize_policy_records(path: str | Path, records: Sequence[PolicyStepRecord]) -> None:
+ target = Path(path)
+ target.parent.mkdir(parents=True, exist_ok=True)
+ with target.open("w", encoding="utf-8") as handle:
+ for record in records:
+ handle.write(json.dumps(record.to_dict(), ensure_ascii=False) + "\n")
+
+
+def load_policy_records(path: str | Path) -> list[PolicyStepRecord]:
+ records: list[PolicyStepRecord] = []
+ with Path(path).open("r", encoding="utf-8-sig") as handle:
+ for raw in handle:
+ line = raw.strip()
+ if not line:
+ continue
+ payload = json.loads(line)
+ if isinstance(payload, dict):
+ records.append(PolicyStepRecord.from_dict(payload))
+ return records
+
+
+def build_runtime_step_record(
+ engine: Any,
+ current_node: Any,
+ candidate_edges: Sequence[Any],
+ *,
+ path: Any = None,
+ visited: set[str] | None = None,
+ mode: str = "forward",
+ target_index: int = -1,
+ sample_id: str | None = None,
+ source_kind: str = "runtime",
+ source_dataset: str = "runtime",
+ query_type: str = "runtime",
+ task_key: str = "runtime|runtime|forward",
+ weight: float = 1.0,
+ source_score: float = 0.0,
+) -> PolicyStepRecord:
+ visited = visited or set()
+ candidate_edges = list(candidate_edges)
+ candidate_concepts = tuple(getattr(edge.to_node, "concept", "") for edge in candidate_edges)
+ candidate_relations = tuple(_normalize_text(getattr(edge, "relation", "") or "related_to") for edge in candidate_edges)
+ base_features = tuple(
+ tuple(
+ compute_candidate_base_features(
+ engine,
+ current_node,
+ edge,
+ path=path,
+ visited=visited,
+ mode=mode,
+ max_degree=getattr(engine, "_max_degree", 1),
+ )
+ )
+ for edge in candidate_edges
+ )
+ candidate_is_memory = tuple(1 if getattr(edge, "is_memory", False) else 0 for edge in candidate_edges)
+ candidate_is_expanded = tuple(1 if getattr(edge, "is_expanded", False) else 0 for edge in candidate_edges)
+ candidate_is_tunneling = tuple(1 if getattr(edge, "is_tunneling", False) else 0 for edge in candidate_edges)
+
+ path_nodes = [getattr(node, "concept", "") for node in getattr(path, "nodes", []) if getattr(node, "concept", "")]
+ recent_concepts = tuple(path_nodes[-DEFAULT_HISTORY_SIZE:])
+ visited_concepts = tuple(sorted(_normalize_text(item) for item in visited if _normalize_text(item)))
+ candidate_count = len(candidate_edges)
+ target_index = int(target_index)
+ selected_edge = candidate_edges[target_index] if 0 <= target_index < len(candidate_edges) else None
+ high_value_target = 0
+ if selected_edge is not None:
+ heuristic_score = 1.0 - float(getattr(selected_edge, "resistance", 0.5) or 0.5)
+ heuristic_score += 0.25 * float(getattr(selected_edge, "is_memory", False))
+ heuristic_score += 0.15 * float(getattr(selected_edge, "is_expanded", False))
+ high_value_target = int(heuristic_score >= 0.75)
+ has_tunneling_path = int(any(getattr(edge, "is_tunneling", False) for edge in getattr(path, "edges", []) or []))
+ difficulty = (
+ float(getattr(path, "length", 0) or 0)
+ + 0.5 * max(0, candidate_count - 1)
+ + 1.25 * float(has_tunneling_path)
+ + 0.5 * float(target_index >= 0 and candidate_count > 3)
+ )
+ return PolicyStepRecord(
+ sample_id=sample_id or f"runtime::{current_node.concept}::{mode}::{_stable_hash('|'.join(candidate_concepts))}",
+ source_kind=source_kind,
+ source_dataset=source_dataset or "runtime",
+ query_type=query_type or "runtime",
+ mode=mode or "forward",
+ task_key=task_key or f"{source_dataset}|{query_type}|{mode}",
+ current_concept=_normalize_text(getattr(current_node, "concept", "")),
+ recent_concepts=recent_concepts,
+ visited_concepts=visited_concepts,
+ candidate_concepts=candidate_concepts,
+ candidate_relations=candidate_relations,
+ candidate_base_features=base_features,
+ candidate_is_memory=candidate_is_memory,
+ candidate_is_expanded=candidate_is_expanded,
+ candidate_is_tunneling=candidate_is_tunneling,
+ target_index=target_index,
+ weight=float(weight),
+ path_length=int(getattr(path, "length", 0) or 0),
+ path_length_bucket=_path_length_bucket(int(getattr(path, "length", 0) or 0)),
+ difficulty=float(difficulty),
+ high_value_target=high_value_target,
+ has_tunneling_path=has_tunneling_path,
+ source_score=float(source_score),
+ metadata={"candidate_count": candidate_count},
+ )
+
+
+def build_path_stub(engine: Any, concepts: Sequence[str]) -> Any:
+ nodes = [engine.nodes[concept] for concept in concepts if concept in engine.nodes]
+ return SimpleNamespace(length=max(0, len(concepts) - 1), nodes=nodes, edges=[])
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/core/policy_network.py b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/core/policy_network.py
new file mode 100644
index 0000000000000000000000000000000000000000..30d79aa76a9442c9b7707a8f3f555caab7001032
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/core/policy_network.py
@@ -0,0 +1,880 @@
+"""
+Neural policy for Tri-Maze exploration (non-LLM).
+Provides a backward-compatible EdgePolicy wrapper that can load the original
+v1 shallow MLP checkpoints or the new v2 structured ranking model.
+"""
+from __future__ import annotations
+
+from dataclasses import asdict, dataclass
+from pathlib import Path
+from typing import Any, Optional, Sequence
+import random
+
+try:
+ import torch
+ from torch import nn
+ import torch.nn.functional as F
+except Exception: # pragma: no cover - optional dependency
+ torch = None
+ nn = None
+ F = None
+
+from .policy_dataset import (
+ BASE_FEATURE_DIM,
+ CANDIDATE_FLAG_DIM,
+ FEATURE_SCHEMA_VERSION,
+ PATH_CONTEXT_DIM,
+ PolicyStepDataset,
+ PolicyVocabulary,
+ build_runtime_step_record,
+ policy_collate_fn,
+)
+
+
+CHECKPOINT_VERSION_V2 = "tmcra_tri_maze_policy_v2"
+
+
+def _mask_fill_value(tensor: torch.Tensor) -> float:
+ if torch is None:
+ return -1e4
+ try:
+ return float(torch.finfo(tensor.dtype).min)
+ except Exception:
+ return -1e4
+
+
+@dataclass(slots=True)
+class PolicyModelConfig:
+ model_version: str = "v2"
+ base_feature_dim: int = BASE_FEATURE_DIM
+ path_context_dim: int = PATH_CONTEXT_DIM
+ candidate_flag_dim: int = CANDIDATE_FLAG_DIM
+ history_size: int = 4
+ concept_embedding_dim: int = 64
+ relation_embedding_dim: int = 16
+ domain_embedding_dim: int = 8
+ hash_bucket_size: int = 8192
+ trunk_dims: tuple[int, ...] = (128, 256, 128)
+ dropout: float = 0.1
+ feature_attention: bool = True
+ multitask: bool = True
+ domain_adapt: bool = False
+ contrastive_dim: int = 128
+ path_length_buckets: int = 4
+ concept_vocab_size: int = 2
+ relation_vocab_size: int = 2
+ source_vocab_size: int = 2
+ query_type_vocab_size: int = 2
+ mode_vocab_size: int = 6
+ task_vocab_size: int = 2
+
+ def to_dict(self) -> dict[str, Any]:
+ payload = asdict(self)
+ payload["trunk_dims"] = list(self.trunk_dims)
+ return payload
+
+ @classmethod
+ def from_dict(cls, payload: dict[str, Any] | None) -> "PolicyModelConfig":
+ if not payload:
+ return cls()
+ data = dict(payload)
+ if "trunk_dims" in data:
+ data["trunk_dims"] = tuple(int(item) for item in data.get("trunk_dims") or ())
+ return cls(**data)
+
+ def apply_vocabulary(self, vocabulary: PolicyVocabulary | None) -> "PolicyModelConfig":
+ if vocabulary is None:
+ return self
+ updated = self.to_dict()
+ updated.update(
+ {
+ "hash_bucket_size": int(vocabulary.hash_bucket_size),
+ "concept_vocab_size": int(vocabulary.concept_vocab_size),
+ "relation_vocab_size": int(vocabulary.relation_vocab_size),
+ "source_vocab_size": int(vocabulary.source_vocab_size),
+ "query_type_vocab_size": int(vocabulary.query_type_vocab_size),
+ "mode_vocab_size": int(vocabulary.mode_vocab_size),
+ "task_vocab_size": int(vocabulary.task_vocab_size),
+ }
+ )
+ return PolicyModelConfig.from_dict(updated)
+
+
+if nn is not None:
+ class PolicyNet(nn.Module):
+ def __init__(self, input_dim: int, hidden_dim: int = 32):
+ super().__init__()
+ self.net = nn.Sequential(
+ nn.Linear(input_dim, hidden_dim),
+ nn.ReLU(),
+ nn.Linear(hidden_dim, 1),
+ )
+
+ def forward(self, x): # x: [N, D]
+ return self.net(x).squeeze(-1)
+else: # pragma: no cover
+ class PolicyNet:
+ def __init__(self, *args, **kwargs):
+ raise RuntimeError("torch not available; install torch to use PolicyNet")
+
+
+if nn is not None:
+ class _GradReverse(torch.autograd.Function):
+ @staticmethod
+ def forward(ctx, x, coeff):
+ ctx.coeff = coeff
+ return x.view_as(x)
+
+ @staticmethod
+ def backward(ctx, grad_output):
+ return grad_output.neg() * ctx.coeff, None
+
+
+ def grad_reverse(x: torch.Tensor, coeff: float) -> torch.Tensor:
+ return _GradReverse.apply(x, coeff)
+
+
+ def _masked_mean(values: torch.Tensor, mask: torch.Tensor, dim: int) -> torch.Tensor:
+ weights = mask.float()
+ total = (values * weights.unsqueeze(-1)).sum(dim=dim)
+ denom = weights.sum(dim=dim, keepdim=True).clamp_min(1.0)
+ return total / denom
+
+
+ class PolicyNetV2(nn.Module):
+ def __init__(self, config: PolicyModelConfig):
+ super().__init__()
+ self.config = config
+ self.concept_embedding = nn.Embedding(
+ config.concept_vocab_size,
+ config.concept_embedding_dim,
+ padding_idx=0,
+ )
+ self.hash_embedding = nn.Embedding(
+ config.hash_bucket_size + 1,
+ config.concept_embedding_dim,
+ padding_idx=0,
+ )
+ self.relation_embedding = nn.Embedding(
+ config.relation_vocab_size,
+ config.relation_embedding_dim,
+ padding_idx=0,
+ )
+ self.source_embedding = nn.Embedding(
+ config.source_vocab_size,
+ config.domain_embedding_dim,
+ padding_idx=0,
+ )
+ self.query_type_embedding = nn.Embedding(
+ config.query_type_vocab_size,
+ config.domain_embedding_dim,
+ padding_idx=0,
+ )
+ self.mode_embedding = nn.Embedding(
+ config.mode_vocab_size,
+ config.domain_embedding_dim,
+ padding_idx=0,
+ )
+ self.task_embedding = nn.Embedding(
+ config.task_vocab_size,
+ config.domain_embedding_dim,
+ padding_idx=0,
+ )
+
+ self.sample_context_dim = (
+ config.concept_embedding_dim * 2
+ + config.path_context_dim
+ + config.domain_embedding_dim * 4
+ )
+ self.candidate_input_dim = (
+ config.base_feature_dim
+ + config.path_context_dim
+ + config.candidate_flag_dim
+ + config.relation_embedding_dim
+ + config.concept_embedding_dim * 4
+ + config.domain_embedding_dim * 4
+ + config.trunk_dims[0]
+ )
+ self.sample_context_proj = nn.Sequential(
+ nn.Linear(self.sample_context_dim, config.trunk_dims[0]),
+ nn.LayerNorm(config.trunk_dims[0]),
+ nn.SiLU(),
+ )
+ if config.feature_attention:
+ self.feature_gate = nn.Sequential(
+ nn.Linear(self.candidate_input_dim, self.candidate_input_dim),
+ nn.LayerNorm(self.candidate_input_dim),
+ nn.Sigmoid(),
+ )
+ else:
+ self.feature_gate = None
+
+ layers: list[nn.Module] = []
+ prev_dim = self.candidate_input_dim
+ for hidden_dim in config.trunk_dims:
+ layers.extend(
+ [
+ nn.Linear(prev_dim, hidden_dim),
+ nn.LayerNorm(hidden_dim),
+ nn.SiLU(),
+ nn.Dropout(config.dropout),
+ ]
+ )
+ prev_dim = hidden_dim
+ self.trunk = nn.Sequential(*layers)
+ self.score_head = nn.Linear(prev_dim, 1)
+ self.path_length_head = nn.Linear(prev_dim, config.path_length_buckets)
+ self.tunnel_head = nn.Linear(prev_dim, 1)
+ self.high_value_head = nn.Linear(prev_dim, 1)
+ self.context_projection = nn.Linear(prev_dim, config.contrastive_dim)
+ self.candidate_projection = nn.Linear(prev_dim, config.contrastive_dim)
+ self.domain_head = nn.Linear(prev_dim, config.source_vocab_size) if config.domain_adapt else None
+
+ def _concept_repr(self, token_ids: torch.Tensor, hash_ids: torch.Tensor) -> torch.Tensor:
+ return self.concept_embedding(token_ids) + self.hash_embedding(hash_ids)
+
+ def forward(self, batch: PolicyBatch | dict[str, torch.Tensor]) -> dict[str, torch.Tensor]:
+ if isinstance(batch, dict):
+ batch = PolicyBatch(**batch)
+
+ current_embed = self._concept_repr(batch.current_concept_ids, batch.current_hash_ids)
+ history_embed = self._concept_repr(batch.history_concept_ids, batch.history_hash_ids)
+ history_mask = (batch.history_concept_ids != 0) | (batch.history_hash_ids != 0)
+ history_summary = _masked_mean(history_embed, history_mask, dim=1)
+
+ candidate_embed = self._concept_repr(batch.candidate_concept_ids, batch.candidate_hash_ids)
+ relation_embed = self.relation_embedding(batch.relation_ids)
+ source_embed = self.source_embedding(batch.source_ids)
+ query_embed = self.query_type_embedding(batch.query_type_ids)
+ mode_embed = self.mode_embedding(batch.mode_ids)
+ task_embed = self.task_embedding(batch.task_ids)
+
+ sample_context = torch.cat(
+ [
+ current_embed,
+ history_summary,
+ batch.path_context,
+ source_embed,
+ query_embed,
+ mode_embed,
+ task_embed,
+ ],
+ dim=-1,
+ )
+ sample_context = self.sample_context_proj(sample_context)
+ sample_context_expanded = sample_context.unsqueeze(1).expand(-1, batch.base_features.shape[1], -1)
+ path_context_expanded = batch.path_context.unsqueeze(1).expand(-1, batch.base_features.shape[1], -1)
+ domain_context = torch.cat([source_embed, query_embed, mode_embed, task_embed], dim=-1)
+ domain_context = domain_context.unsqueeze(1).expand(-1, batch.base_features.shape[1], -1)
+
+ candidate_input = torch.cat(
+ [
+ batch.base_features,
+ path_context_expanded,
+ batch.candidate_flags,
+ relation_embed,
+ sample_context_expanded,
+ candidate_embed,
+ current_embed.unsqueeze(1).expand_as(candidate_embed),
+ candidate_embed - current_embed.unsqueeze(1),
+ candidate_embed * current_embed.unsqueeze(1),
+ domain_context,
+ ],
+ dim=-1,
+ )
+ if self.feature_gate is not None:
+ candidate_input = candidate_input * self.feature_gate(candidate_input)
+
+ hidden = self.trunk(candidate_input)
+ logits = self.score_head(hidden).squeeze(-1)
+ logits = logits.masked_fill(~batch.candidate_mask, _mask_fill_value(logits))
+ pooled = _masked_mean(hidden, batch.candidate_mask, dim=1)
+
+ outputs = {
+ "logits": logits,
+ "hidden": hidden,
+ "pooled": pooled,
+ "path_length_logits": self.path_length_head(pooled),
+ "tunnel_logits": self.tunnel_head(pooled).squeeze(-1),
+ "high_value_logits": self.high_value_head(pooled).squeeze(-1),
+ "contrastive_context": F.normalize(self.context_projection(pooled), dim=-1),
+ "contrastive_candidates": F.normalize(self.candidate_projection(hidden), dim=-1),
+ }
+ if self.domain_head is not None:
+ outputs["domain_logits"] = self.domain_head(grad_reverse(pooled, 1.0))
+ return outputs
+
+
+def masked_cross_entropy(
+ logits: torch.Tensor,
+ target_index: torch.Tensor,
+ candidate_mask: torch.Tensor,
+ *,
+ sample_weights: torch.Tensor | None = None,
+) -> torch.Tensor:
+ if F is None:
+ raise RuntimeError("torch not available")
+ masked_logits = logits.masked_fill(~candidate_mask, _mask_fill_value(logits))
+ loss = F.cross_entropy(masked_logits, target_index, reduction="none")
+ if sample_weights is not None:
+ loss = loss * sample_weights
+ return loss.mean()
+
+
+def hard_negative_margin_loss(
+ logits: torch.Tensor,
+ target_index: torch.Tensor,
+ candidate_mask: torch.Tensor,
+ *,
+ margin: float = 0.2,
+) -> torch.Tensor:
+ if F is None:
+ raise RuntimeError("torch not available")
+ batch_index = torch.arange(logits.shape[0], device=logits.device)
+ positive = logits[batch_index, target_index]
+ negative_mask = candidate_mask.clone()
+ negative_mask[batch_index, target_index] = False
+ hardest_negative = logits.masked_fill(~negative_mask, _mask_fill_value(logits)).max(dim=-1).values
+ loss = F.relu(margin - (positive - hardest_negative))
+ valid = negative_mask.any(dim=-1)
+ if valid.any():
+ return loss[valid].mean()
+ return loss.new_tensor(0.0)
+
+
+def candidate_contrastive_loss(
+ context_repr: torch.Tensor,
+ candidate_repr: torch.Tensor,
+ target_index: torch.Tensor,
+ candidate_mask: torch.Tensor,
+ *,
+ temperature: float = 0.1,
+) -> torch.Tensor:
+ if F is None:
+ raise RuntimeError("torch not available")
+ logits = torch.einsum("bd,bcd->bc", context_repr, candidate_repr) / max(temperature, 1e-4)
+ logits = logits.masked_fill(~candidate_mask, _mask_fill_value(logits))
+ return F.cross_entropy(logits, target_index)
+
+
+def _copy_embedding_rows_by_token(
+ target_tensor: torch.Tensor,
+ source_tensor: torch.Tensor | None,
+ target_mapping: dict[str, int],
+ source_mapping: dict[str, int],
+) -> None:
+ if source_tensor is None:
+ return
+ rows = int(target_tensor.shape[0])
+ source_rows = int(source_tensor.shape[0])
+ cols = min(int(target_tensor.shape[1]), int(source_tensor.shape[1])) if target_tensor.ndim == 2 and source_tensor.ndim == 2 else 0
+ if cols <= 0:
+ return
+ with torch.no_grad():
+ for token, target_index in target_mapping.items():
+ source_index = source_mapping.get(token)
+ if source_index is None:
+ continue
+ if not (0 <= int(target_index) < rows and 0 <= int(source_index) < source_rows):
+ continue
+ target_tensor[int(target_index), :cols].copy_(source_tensor[int(source_index), :cols])
+
+
+def _copy_prefix_tensor(target_tensor: torch.Tensor, source_tensor: torch.Tensor | None) -> None:
+ if source_tensor is None or target_tensor.ndim != source_tensor.ndim:
+ return
+ common_shape = tuple(min(int(a), int(b)) for a, b in zip(target_tensor.shape, source_tensor.shape))
+ if not common_shape:
+ return
+ target_slices = tuple(slice(0, size) for size in common_shape)
+ source_slices = tuple(slice(0, size) for size in common_shape)
+ with torch.no_grad():
+ target_tensor[target_slices].copy_(source_tensor[source_slices])
+
+
+class EdgePolicy:
+ def __init__(
+ self,
+ input_dim: int = 13,
+ hidden_dim: int = 32,
+ lr: float = 1e-3,
+ temperature: float = 1.0,
+ branch_factor: int = 2,
+ revisit_probability: float = 0.2,
+ seed: int = 42,
+ *,
+ model_version: str = "v2",
+ model_config: PolicyModelConfig | None = None,
+ vocabulary: PolicyVocabulary | None = None,
+ weight_decay: float = 1e-4,
+ ):
+ self.enabled = torch is not None
+ self.input_dim = int(input_dim)
+ self.hidden_dim = int(hidden_dim)
+ self.lr = float(lr)
+ self.weight_decay = float(weight_decay)
+ self.temperature = max(0.1, float(temperature))
+ self.branch_factor = max(1, int(branch_factor))
+ self.revisit_probability = max(0.0, min(1.0, float(revisit_probability)))
+ self.max_degree = 1
+ self.seed = int(seed)
+ self.rng = random.Random(seed)
+ self.model_version = str(model_version or "v2").lower()
+ self.feature_schema = {"version": FEATURE_SCHEMA_VERSION}
+ self.vocabulary = vocabulary or PolicyVocabulary.empty()
+ self.model_config = (model_config or PolicyModelConfig()).apply_vocabulary(self.vocabulary)
+
+ if not self.enabled:
+ self.model = None
+ self.optimizer = None
+ return
+
+ torch.manual_seed(seed)
+ self.model = None
+ self.optimizer = None
+ self._build_model()
+
+ def _build_model(self) -> None:
+ if not self.enabled:
+ return
+ if self.model_version == "v1":
+ self.model = PolicyNet(input_dim=self.input_dim, hidden_dim=self.hidden_dim)
+ self.optimizer = torch.optim.Adam(self.model.parameters(), lr=self.lr)
+ else:
+ self.model_version = "v2"
+ self.model_config = self.model_config.apply_vocabulary(self.vocabulary)
+ self.model = PolicyNetV2(self.model_config)
+ self.optimizer = torch.optim.AdamW(
+ self.model.parameters(),
+ lr=self.lr,
+ weight_decay=self.weight_decay,
+ )
+
+ def _model_device(self) -> torch.device:
+ if self.model is None:
+ return torch.device("cpu")
+ try:
+ return next(self.model.parameters()).device
+ except StopIteration:
+ return torch.device("cpu")
+
+ def set_max_degree(self, max_degree: int) -> None:
+ self.max_degree = max(1, int(max_degree))
+
+ def allow_revisit(self) -> bool:
+ return self.rng.random() < self.revisit_probability
+
+ def config_dict(self) -> dict[str, Any]:
+ base = {
+ "input_dim": self.input_dim,
+ "hidden_dim": self.hidden_dim,
+ "lr": self.lr,
+ "weight_decay": self.weight_decay,
+ "temperature": self.temperature,
+ "branch_factor": self.branch_factor,
+ "revisit_probability": self.revisit_probability,
+ "seed": self.seed,
+ "max_degree": self.max_degree,
+ "model_version": self.model_version,
+ }
+ if self.model_version == "v2":
+ base["model_config"] = self.model_config.to_dict()
+ return base
+
+ def set_vocabulary(self, vocabulary: PolicyVocabulary | None) -> None:
+ if vocabulary is None:
+ return
+ self.vocabulary = vocabulary
+ if self.model_version == "v2":
+ self.model_config = self.model_config.apply_vocabulary(vocabulary)
+ self._build_model()
+
+ def save_checkpoint(
+ self,
+ path: str | Path,
+ *,
+ metadata: dict[str, Any] | None = None,
+ scheduler_state: dict[str, Any] | None = None,
+ extra_state: dict[str, Any] | None = None,
+ ) -> None:
+ if not self.enabled:
+ raise RuntimeError("torch not available; cannot save EdgePolicy checkpoint")
+ payload = {
+ "format_version": CHECKPOINT_VERSION_V2 if self.model_version == "v2" else "tmcra_tri_maze_policy_v1",
+ "config": self.config_dict(),
+ "model_state": self.model.state_dict(),
+ "optimizer_state": self.optimizer.state_dict() if self.optimizer is not None else None,
+ "metadata": metadata or {},
+ "scheduler_state": scheduler_state,
+ "extra_state": extra_state or {},
+ }
+ if self.model_version == "v2":
+ payload["vocabulary"] = self.vocabulary.to_metadata()
+ payload["feature_schema"] = self.feature_schema
+ target = Path(path)
+ target.parent.mkdir(parents=True, exist_ok=True)
+ torch.save(payload, target)
+
+ def _load_v1_checkpoint(
+ self,
+ payload: dict[str, Any],
+ *,
+ load_optimizer: bool,
+ strict: bool,
+ ) -> dict[str, Any]:
+ config = payload.get("config") or {}
+ required = {"input_dim", "hidden_dim", "lr", "temperature", "branch_factor", "revisit_probability", "seed"}
+ if not required.issubset(config):
+ raise ValueError("invalid EdgePolicy checkpoint: missing config keys")
+ self.model_version = "v1"
+ self.input_dim = int(config["input_dim"])
+ self.hidden_dim = int(config["hidden_dim"])
+ self.lr = float(config["lr"])
+ self.temperature = float(config["temperature"])
+ self.branch_factor = int(config["branch_factor"])
+ self.revisit_probability = float(config["revisit_probability"])
+ self.seed = int(config["seed"])
+ self.max_degree = int(config.get("max_degree", self.max_degree))
+ self.rng = random.Random(self.seed)
+ self._build_model()
+ self.model.load_state_dict(payload["model_state"], strict=strict)
+ if load_optimizer and payload.get("optimizer_state") is not None and self.optimizer is not None:
+ self.optimizer.load_state_dict(payload["optimizer_state"])
+ return payload.get("metadata") or {}
+
+ def load_checkpoint(
+ self,
+ path: str | Path,
+ *,
+ load_optimizer: bool = False,
+ strict: bool = True,
+ preserve_vocabulary: bool = False,
+ ) -> dict[str, Any]:
+ if not self.enabled:
+ raise RuntimeError("torch not available; cannot load EdgePolicy checkpoint")
+ payload = torch.load(path, map_location="cpu", weights_only=True)
+ config = payload.get("config") or {}
+ if payload.get("format_version") == CHECKPOINT_VERSION_V2 or config.get("model_version") == "v2":
+ self.model_version = "v2"
+ if not preserve_vocabulary:
+ self.lr = float(config.get("lr", self.lr))
+ self.weight_decay = float(config.get("weight_decay", self.weight_decay))
+ self.temperature = float(config.get("temperature", self.temperature))
+ self.branch_factor = int(config.get("branch_factor", self.branch_factor))
+ self.revisit_probability = float(config.get("revisit_probability", self.revisit_probability))
+ self.seed = int(config.get("seed", self.seed))
+ self.max_degree = int(config.get("max_degree", self.max_degree))
+ self.rng = random.Random(self.seed)
+ checkpoint_vocabulary = PolicyVocabulary.from_metadata(payload.get("vocabulary"))
+ if preserve_vocabulary:
+ current_vocabulary = self.vocabulary or PolicyVocabulary.empty()
+ self.vocabulary = current_vocabulary
+ else:
+ self.vocabulary = checkpoint_vocabulary
+ self.feature_schema = dict(payload.get("feature_schema") or {"version": FEATURE_SCHEMA_VERSION})
+ self.model_config = PolicyModelConfig.from_dict(config.get("model_config")).apply_vocabulary(self.vocabulary)
+ self._build_model()
+ if preserve_vocabulary:
+ target_state = self.model.state_dict()
+ source_state = payload["model_state"]
+ for name, source_tensor in source_state.items():
+ target_tensor = target_state.get(name)
+ if target_tensor is None:
+ continue
+ if target_tensor.shape == source_tensor.shape:
+ target_state[name] = source_tensor
+ _copy_embedding_rows_by_token(
+ target_state["concept_embedding.weight"],
+ source_state.get("concept_embedding.weight"),
+ self.vocabulary.concept_to_id,
+ checkpoint_vocabulary.concept_to_id,
+ )
+ _copy_embedding_rows_by_token(
+ target_state["relation_embedding.weight"],
+ source_state.get("relation_embedding.weight"),
+ self.vocabulary.relation_to_id,
+ checkpoint_vocabulary.relation_to_id,
+ )
+ _copy_embedding_rows_by_token(
+ target_state["source_embedding.weight"],
+ source_state.get("source_embedding.weight"),
+ self.vocabulary.source_to_id,
+ checkpoint_vocabulary.source_to_id,
+ )
+ _copy_embedding_rows_by_token(
+ target_state["query_type_embedding.weight"],
+ source_state.get("query_type_embedding.weight"),
+ self.vocabulary.query_type_to_id,
+ checkpoint_vocabulary.query_type_to_id,
+ )
+ _copy_embedding_rows_by_token(
+ target_state["mode_embedding.weight"],
+ source_state.get("mode_embedding.weight"),
+ self.vocabulary.mode_to_id,
+ checkpoint_vocabulary.mode_to_id,
+ )
+ _copy_embedding_rows_by_token(
+ target_state["task_embedding.weight"],
+ source_state.get("task_embedding.weight"),
+ self.vocabulary.task_to_id,
+ checkpoint_vocabulary.task_to_id,
+ )
+ _copy_prefix_tensor(
+ target_state["hash_embedding.weight"],
+ source_state.get("hash_embedding.weight"),
+ )
+ self.model.load_state_dict(target_state, strict=False)
+ else:
+ self.model.load_state_dict(payload["model_state"], strict=strict)
+ if load_optimizer and not preserve_vocabulary and payload.get("optimizer_state") is not None and self.optimizer is not None:
+ self.optimizer.load_state_dict(payload["optimizer_state"])
+ return payload.get("metadata") or {}
+ return self._load_v1_checkpoint(payload, load_optimizer=load_optimizer, strict=strict)
+
+ def _encode_runtime_batch(
+ self,
+ engine,
+ current_node,
+ candidate_edges,
+ path,
+ visited: set[str],
+ mode: str,
+ *,
+ target_index: int = -1,
+ sample_id: str | None = None,
+ weight: float = 1.0,
+ ) -> PolicyBatch:
+ record = build_runtime_step_record(
+ engine=engine,
+ current_node=current_node,
+ candidate_edges=candidate_edges,
+ path=path,
+ visited=visited,
+ mode=mode,
+ target_index=target_index,
+ sample_id=sample_id,
+ weight=weight,
+ source_kind="runtime_update" if target_index >= 0 else "runtime",
+ source_dataset="runtime",
+ query_type="runtime",
+ task_key=f"runtime|runtime|{mode}",
+ )
+ dataset = PolicyStepDataset(
+ [record],
+ vocabulary=self.vocabulary,
+ history_size=self.model_config.history_size,
+ )
+ batch = policy_collate_fn([dataset[0]])
+ return batch.to(self._model_device())
+
+ def evaluate_candidates(self, engine, current_node, candidate_edges, path, visited: set, mode: str = "forward"):
+ if not self.enabled or not candidate_edges:
+ return None
+ self.model.eval()
+ with torch.no_grad():
+ if self.model_version == "v1":
+ features = torch.tensor(
+ [
+ row
+ for row in [
+ build_runtime_step_record(
+ engine,
+ current_node,
+ [edge],
+ path=path,
+ visited=visited,
+ mode=mode,
+ sample_id=f"single::{index}",
+ ).candidate_base_features[0]
+ for index, edge in enumerate(candidate_edges)
+ ]
+ ],
+ dtype=torch.float32,
+ device=self._model_device(),
+ )
+ logits = self.model(features)
+ probs = torch.softmax(logits / self.temperature, dim=0)
+ return {"features": features, "logits": logits, "probs": probs}
+
+ batch = self._encode_runtime_batch(
+ engine=engine,
+ current_node=current_node,
+ candidate_edges=candidate_edges,
+ path=path,
+ visited=visited,
+ mode=mode,
+ )
+ outputs = self.model(batch)
+ logits = outputs["logits"][0, : len(candidate_edges)]
+ probs = torch.softmax(logits / self.temperature, dim=0)
+ return {"batch": batch, "outputs": outputs, "logits": logits, "probs": probs}
+
+ def score_edges(self, engine, current_node, edges, path, visited: set, mode: str):
+ if not self.enabled or not edges:
+ return None
+ evaluated = self.evaluate_candidates(engine, current_node, edges, path, visited, mode)
+ if evaluated is None:
+ return None
+ return evaluated["logits"]
+
+ def select_edges(
+ self,
+ engine,
+ current_node,
+ candidate_edges,
+ path,
+ visited: set,
+ mode: str = "forward",
+ k: Optional[int] = None,
+ deterministic: bool = False,
+ ):
+ if not self.enabled or not candidate_edges:
+ return []
+
+ evaluated = self.evaluate_candidates(engine, current_node, candidate_edges, path, visited, mode)
+ if evaluated is None:
+ return []
+ probs = evaluated["probs"]
+
+ k = k or self.branch_factor
+ k = max(1, min(int(k), len(candidate_edges)))
+
+ if deterministic:
+ top_idx = torch.topk(probs, k=k).indices.tolist()
+ return [candidate_edges[i] for i in top_idx]
+
+ if k == 1:
+ idx = torch.multinomial(probs, num_samples=1).item()
+ return [candidate_edges[idx]]
+
+ idxs = torch.multinomial(probs, num_samples=k, replacement=False).tolist()
+ return [candidate_edges[i] for i in idxs]
+
+ def _online_supervised_step(self, batch: PolicyBatch) -> float:
+ if self.optimizer is None:
+ return 0.0
+ self.model.train()
+ outputs = self.model(batch)
+ loss = masked_cross_entropy(
+ outputs["logits"],
+ batch.target_index,
+ batch.candidate_mask,
+ sample_weights=batch.weights,
+ )
+ self.optimizer.zero_grad()
+ loss.backward()
+ self.optimizer.step()
+ return float(loss.item())
+
+ def warm_start_from_memory(
+ self,
+ engine,
+ max_paths: int = 200,
+ max_steps: int = 800,
+ ) -> int:
+ if not self.enabled:
+ return 0
+ memory = getattr(engine, "memory", None)
+ if not memory:
+ return 0
+ paths = memory.get_all_paths()
+ if not paths:
+ return 0
+
+ steps = 0
+ self.rng.shuffle(paths)
+ for path_item in paths[:max_paths]:
+ concept_list = [str(item).strip() for item in (path_item.get("path") or []) if str(item).strip()]
+ if len(concept_list) < 2:
+ continue
+ weight = max(0.1, min(1.0, float(path_item.get("score", 0.5) or 0.5)))
+ visited: set[str] = set()
+ for index in range(len(concept_list) - 1):
+ if steps >= max_steps:
+ return steps
+ current = concept_list[index]
+ target = concept_list[index + 1]
+ current_node = engine.nodes.get(current)
+ if not current_node:
+ continue
+ edges = list(current_node.connections)
+ target_idx = next((item for item, edge in enumerate(edges) if edge.to_node.concept == target), None)
+ if target_idx is None:
+ continue
+ visited.add(current)
+ path_stub = type("PathStub", (), {"length": index, "nodes": [engine.nodes[c] for c in concept_list[: index + 1] if c in engine.nodes], "edges": []})()
+ if self.model_version == "v1":
+ features = torch.tensor(
+ [build_runtime_step_record(engine, current_node, [edge], path=path_stub, visited=visited, mode="forward").candidate_base_features[0] for edge in edges],
+ dtype=torch.float32,
+ device=self._model_device(),
+ )
+ logits = self.model(features).unsqueeze(0)
+ target_tensor = torch.tensor([target_idx], dtype=torch.long, device=self._model_device())
+ loss = F.cross_entropy(logits, target_tensor) * weight
+ self.optimizer.zero_grad()
+ loss.backward()
+ self.optimizer.step()
+ else:
+ batch = self._encode_runtime_batch(
+ engine=engine,
+ current_node=current_node,
+ candidate_edges=edges,
+ path=path_stub,
+ visited=visited,
+ mode="forward",
+ target_index=target_idx,
+ sample_id=f"warm::{current}::{target}::{index}",
+ weight=weight,
+ )
+ self._online_supervised_step(batch)
+ steps += 1
+ return steps
+
+ def update_from_path(self, engine, path, mode: str = "forward") -> int:
+ if not self.enabled or not path or not getattr(path, "edges", None):
+ return 0
+
+ reward = 1.0
+ if hasattr(path, "score") and hasattr(engine, "length_penalty"):
+ reward = max(0.05, 1.0 - float(path.score(engine.length_penalty)))
+
+ visited: set[str] = set()
+ steps = 0
+ for index, edge in enumerate(path.edges):
+ current_node = path.nodes[index]
+ visited.add(current_node.concept)
+ candidate_edges = list(current_node.connections)
+ if not candidate_edges:
+ continue
+ try:
+ target_idx = candidate_edges.index(edge)
+ except ValueError:
+ continue
+ if self.model_version == "v1":
+ features = torch.tensor(
+ [build_runtime_step_record(engine, current_node, [candidate], path=path, visited=visited, mode=mode).candidate_base_features[0] for candidate in candidate_edges],
+ dtype=torch.float32,
+ device=self._model_device(),
+ )
+ logits = self.model(features).unsqueeze(0)
+ target_tensor = torch.tensor([target_idx], dtype=torch.long, device=self._model_device())
+ loss = F.cross_entropy(logits, target_tensor) * reward
+ self.optimizer.zero_grad()
+ loss.backward()
+ self.optimizer.step()
+ else:
+ batch = self._encode_runtime_batch(
+ engine=engine,
+ current_node=current_node,
+ candidate_edges=candidate_edges,
+ path=path,
+ visited=visited,
+ mode=mode,
+ target_index=target_idx,
+ sample_id=f"update::{current_node.concept}::{index}",
+ weight=reward,
+ )
+ self._online_supervised_step(batch)
+ steps += 1
+ return steps
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/core/session_memory.py b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/core/session_memory.py
new file mode 100644
index 0000000000000000000000000000000000000000..23c848956acf17ef45c05f545ab87742ac2537bf
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/core/session_memory.py
@@ -0,0 +1,620 @@
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+import re
+from typing import Any, Dict, Iterable, List
+
+
+_GOAL_MARKERS = (
+ "目标",
+ "想要",
+ "希望",
+ "实现",
+ "做成",
+ "达成",
+ "验证",
+ "评估",
+ "提升",
+ "重构",
+ "升级",
+ "训练",
+ "支持",
+ "切换",
+)
+_PREFERENCE_MARKERS = (
+ "默认",
+ "优先",
+ "保留",
+ "保持",
+ "透明",
+ "自然",
+ "可解释",
+ "开源",
+ "本地",
+)
+_CONSTRAINT_MARKERS = (
+ "不要",
+ "必须",
+ "不能",
+ "仅",
+ "只做",
+ "只用",
+ "禁止",
+)
+_VAGUE_QUERY_MARKERS = (
+ "继续",
+ "接着",
+ "然后",
+ "当前计划",
+ "这个体系",
+ "这个系统",
+ "方案",
+ "当前",
+)
+_RELATION_BY_CATEGORY = {
+ "goal": "session_goal",
+ "preference": "prefers",
+ "terminology": "uses_term",
+ "constraint": "constrained_by",
+ "stage_state": "stage_state",
+}
+_TYPE_BY_CATEGORY = {
+ "goal": "session_goal",
+ "preference": "session_preference",
+ "terminology": "session_term",
+ "constraint": "session_constraint",
+ "stage_state": "session_stage",
+}
+
+
+def _clean_text(value: Any) -> str:
+ text = str(value or "").strip()
+ text = re.sub(r"\s+", " ", text)
+ return text
+
+
+def _normalize_key(value: Any) -> str:
+ return _clean_text(value).lower()
+
+
+def _clip_text(value: Any, max_len: int = 72) -> str:
+ text = _clean_text(value)
+ if len(text) <= max_len:
+ return text
+ return text[: max_len - 3].rstrip() + "..."
+
+
+def _dedupe_texts(values: Iterable[Any], max_items: int | None = None) -> List[str]:
+ items: List[str] = []
+ seen = set()
+ for value in values:
+ text = _clean_text(value)
+ if not text:
+ continue
+ key = _normalize_key(text)
+ if key in seen:
+ continue
+ seen.add(key)
+ items.append(text)
+ if max_items is not None and len(items) >= max_items:
+ break
+ return items
+
+
+def _split_fragments(text: str) -> List[str]:
+ text = _clean_text(text)
+ if not text:
+ return []
+ fragments = re.split(r"[，,。；;！!？?\n]+", text)
+ return _dedupe_texts(fragments)
+
+
+def _contains_any(text: str, markers: Iterable[str]) -> bool:
+ return any(marker in text for marker in markers)
+
+
+def _match_fragments(text: str, markers: Iterable[str], max_items: int = 2) -> List[str]:
+ matched = [fragment for fragment in _split_fragments(text) if _contains_any(fragment, markers)]
+ return _dedupe_texts(matched, max_items=max_items)
+
+
+def _concept_order(
+ understanding_result: Dict[str, Any] | None,
+ extraction_result: Dict[str, Any] | None,
+ answer_bundle: Dict[str, Any] | None,
+) -> List[str]:
+ concepts: List[str] = []
+ if isinstance(answer_bundle, dict):
+ concepts.extend(answer_bundle.get("core_concepts", []) or [])
+ focus = answer_bundle.get("focus_concept")
+ if focus:
+ concepts.insert(0, focus)
+ if isinstance(understanding_result, dict):
+ focus = understanding_result.get("focus_concept")
+ if focus:
+ concepts.insert(0, focus)
+ if isinstance(extraction_result, dict):
+ for item in extraction_result.get("concepts", []) or []:
+ if isinstance(item, dict):
+ concepts.append(item.get("concept", ""))
+ return _dedupe_texts(concepts, max_items=12)
+
+
+_TOKEN_RE = re.compile(r"[a-z0-9_]+|[\u4e00-\u9fff]", re.IGNORECASE)
+
+
+def _token_set(value: Any) -> set[str]:
+ text = _normalize_key(value)
+ if not text:
+ return set()
+ return {token for token in _TOKEN_RE.findall(text) if token}
+
+
+def _category_slot_key(category: str, value: str) -> str:
+ normalized_category = _normalize_key(category)
+ if normalized_category == "goal":
+ return "goal.primary"
+ if normalized_category == "preference":
+ return "preference.active"
+ if normalized_category == "constraint":
+ return "constraint.active"
+ if normalized_category == "stage_state":
+ return "stage.current"
+ if normalized_category == "terminology":
+ return f"terminology.{_normalize_key(value)[:24]}"
+ return f"{normalized_category}.{_normalize_key(value)[:24]}"
+
+
+def _anchor_signature(anchors: Iterable[Any]) -> str:
+ normalized = sorted({_normalize_key(anchor) for anchor in anchors if _clean_text(anchor)})
+ return "|".join(normalized)
+
+
+def _state_signature(*, category: str, slot_key: str, anchors: Iterable[Any]) -> str:
+ components = [_normalize_key(category), _normalize_key(slot_key), _anchor_signature(anchors)]
+ return "|".join(component for component in components if component)
+
+
+def _memory_signature(*, category: str, value: str, slot_key: str, anchors: Iterable[Any]) -> str:
+ components = [
+ _state_signature(category=category, slot_key=slot_key, anchors=anchors),
+ _normalize_key(value),
+ ]
+ return "|".join(component for component in components if component)
+
+
+def _event_signature(*, category: str, value: str, slot_key: str, anchors: Iterable[Any]) -> str:
+ category_label = _normalize_key(category).replace("_", " ")
+ slot_label = _normalize_key(slot_key).replace(".", " ").replace("_", " ")
+ parts = [category_label, slot_label]
+ parts.extend(_clean_text(anchor) for anchor in anchors if _clean_text(anchor))
+ parts.append(_clean_text(value))
+ return " ".join(_dedupe_texts(parts, max_items=8))
+
+
+@dataclass(slots=True)
+class SessionMemoryRecord:
+ memory_id: str
+ category: str
+ value: str
+ relation: str
+ anchor_concepts: List[str] = field(default_factory=list)
+ salience: float = 0.6
+ confidence: float = 0.6
+ source_kind: str = "session_memory"
+ turn_index: int = 0
+ metadata: Dict[str, Any] = field(default_factory=dict)
+
+ def key(self) -> tuple[str, str, str]:
+ anchor_key = "|".join(sorted(_normalize_key(item) for item in self.anchor_concepts))
+ return (_normalize_key(self.category), _normalize_key(self.value), anchor_key)
+
+ def to_dict(self) -> Dict[str, Any]:
+ return {
+ "memory_id": self.memory_id,
+ "category": self.category,
+ "value": self.value,
+ "relation": self.relation,
+ "anchor_concepts": list(self.anchor_concepts),
+ "salience": round(float(self.salience), 4),
+ "confidence": round(float(self.confidence), 4),
+ "source_kind": self.source_kind,
+ "turn_index": int(self.turn_index),
+ "metadata": dict(self.metadata),
+ }
+
+ def to_hit(self, relevance: float) -> Dict[str, Any]:
+ return {
+ "memory_id": self.memory_id,
+ "category": self.category,
+ "value": self.value,
+ "relation": self.relation,
+ "anchors": list(self.anchor_concepts),
+ "salience": round(float(self.salience), 4),
+ "confidence": round(float(self.confidence), 4),
+ "relevance": round(float(relevance), 4),
+ "source_kind": self.source_kind,
+ "turn_index": int(self.turn_index),
+ "metadata": dict(self.metadata),
+ }
+
+
+class SessionMemoryExtractor:
+ def extract(
+ self,
+ query: str,
+ understanding_result: Dict[str, Any] | None = None,
+ extraction_result: Dict[str, Any] | None = None,
+ answer_bundle: Dict[str, Any] | None = None,
+ *,
+ answer_mode: str = "natural",
+ turn_index: int = 0,
+ ) -> List[SessionMemoryRecord]:
+ query_text = _clean_text(query)
+ if not query_text:
+ return []
+
+ anchors = _concept_order(understanding_result, extraction_result, answer_bundle)
+ records: List[SessionMemoryRecord] = []
+
+ for constraint in self._extract_constraints(query_text, understanding_result):
+ records.append(
+ self._make_record(
+ category="constraint",
+ value=constraint,
+ anchors=anchors,
+ salience=0.88,
+ confidence=0.82,
+ turn_index=turn_index,
+ )
+ )
+
+ for preference in self._extract_preferences(query_text, answer_mode):
+ records.append(
+ self._make_record(
+ category="preference",
+ value=preference,
+ anchors=anchors,
+ salience=0.82,
+ confidence=0.8,
+ turn_index=turn_index,
+ )
+ )
+
+ for goal in self._extract_goals(query_text, anchors):
+ records.append(
+ self._make_record(
+ category="goal",
+ value=goal,
+ anchors=anchors,
+ salience=0.9,
+ confidence=0.78,
+ turn_index=turn_index,
+ )
+ )
+
+ for stage_state in self._extract_stage_states(query_text):
+ records.append(
+ self._make_record(
+ category="stage_state",
+ value=stage_state,
+ anchors=anchors,
+ salience=0.74,
+ confidence=0.72,
+ turn_index=turn_index,
+ )
+ )
+
+ for term in self._extract_terms(query_text, anchors):
+ records.append(
+ self._make_record(
+ category="terminology",
+ value=term,
+ anchors=[term],
+ salience=0.64,
+ confidence=0.68,
+ turn_index=turn_index,
+ )
+ )
+
+ merged: Dict[tuple[str, str, str], SessionMemoryRecord] = {}
+ for record in records:
+ key = record.key()
+ existing = merged.get(key)
+ if existing is None:
+ merged[key] = record
+ continue
+ existing.salience = max(existing.salience, record.salience)
+ existing.confidence = max(existing.confidence, record.confidence)
+ existing.turn_index = max(existing.turn_index, record.turn_index)
+ existing.anchor_concepts = _dedupe_texts([*existing.anchor_concepts, *record.anchor_concepts], max_items=8)
+ return list(merged.values())
+
+ def _make_record(
+ self,
+ *,
+ category: str,
+ value: str,
+ anchors: List[str],
+ salience: float,
+ confidence: float,
+ turn_index: int,
+ ) -> SessionMemoryRecord:
+ clean_value = _clip_text(value)
+ clean_anchors = _dedupe_texts(anchors, max_items=6)
+ slot_key = _category_slot_key(category, clean_value)
+ state_signature = _state_signature(category=category, slot_key=slot_key, anchors=clean_anchors)
+ memory_signature = _memory_signature(category=category, value=clean_value, slot_key=slot_key, anchors=clean_anchors)
+ event_signature = _event_signature(category=category, value=clean_value, slot_key=slot_key, anchors=clean_anchors)
+ memory_id = f"{category}:{_normalize_key(clean_value)}"
+ return SessionMemoryRecord(
+ memory_id=memory_id,
+ category=category,
+ value=clean_value,
+ relation=_RELATION_BY_CATEGORY.get(category, "related_to"),
+ anchor_concepts=clean_anchors,
+ salience=max(0.0, min(1.0, float(salience))),
+ confidence=max(0.0, min(1.0, float(confidence))),
+ turn_index=int(turn_index),
+ metadata={
+ "slot_key": slot_key,
+ "state_signature": state_signature,
+ "memory_signature": memory_signature,
+ "event_signature": event_signature,
+ "anchor_signature": _anchor_signature(clean_anchors),
+ },
+ )
+
+ def _extract_goals(self, query_text: str, anchors: List[str]) -> List[str]:
+ goals = _match_fragments(query_text, _GOAL_MARKERS, max_items=2)
+ if goals:
+ return goals
+ if _contains_any(query_text, ("继续", "接着")) and anchors:
+ return [f"围绕 {anchors[0]} 持续推进"]
+ return []
+
+ def _extract_preferences(self, query_text: str, answer_mode: str) -> List[str]:
+ preferences: List[str] = []
+ lowered = query_text.lower()
+ if "transparent" in lowered or "透明" in query_text or "可解释" in query_text or answer_mode == "transparent":
+ preferences.append("transparent_mode")
+ if "natural" in lowered or "自然" in query_text or answer_mode == "natural":
+ preferences.append("natural_mode")
+ if "开源" in query_text:
+ preferences.append("prefer_open_models")
+ if any(marker in query_text for marker in ("本地", "离线", "私有部署")):
+ preferences.append("prefer_local_modules")
+ if _contains_any(query_text, _PREFERENCE_MARKERS):
+ preferences.extend(_match_fragments(query_text, _PREFERENCE_MARKERS, max_items=2))
+ return _dedupe_texts(preferences, max_items=4)
+
+ def _extract_constraints(self, query_text: str, understanding_result: Dict[str, Any] | None) -> List[str]:
+ constraints: List[str] = []
+ if isinstance(understanding_result, dict):
+ constraints.extend(understanding_result.get("constraints", []) or [])
+ constraints.extend(_match_fragments(query_text, _CONSTRAINT_MARKERS, max_items=3))
+ return [_clip_text(item, max_len=64) for item in _dedupe_texts(constraints, max_items=4)]
+
+ def _extract_terms(self, query_text: str, anchors: List[str]) -> List[str]:
+ terms: List[str] = []
+ for anchor in anchors[:6]:
+ if len(anchor) >= 2 and anchor in query_text:
+ terms.append(anchor)
+ english_terms = re.findall(r"[A-Za-z][A-Za-z0-9_-]{1,}", query_text)
+ terms.extend(english_terms[:4])
+ return _dedupe_texts(terms, max_items=6)
+
+ def _extract_stage_states(self, query_text: str) -> List[str]:
+ states: List[str] = []
+ if "继续" in query_text or "接着" in query_text:
+ states.append("continue_current_plan")
+ if "开始" in query_text or "启动" in query_text:
+ states.append("execution_started")
+ if "验证" in query_text or "测试" in query_text or "smoke" in query_text.lower():
+ states.append("validation_phase")
+ if "训练" in query_text:
+ states.append("training_phase")
+ if "实施" in query_text or "落地" in query_text or "重构" in query_text:
+ states.append("implementation_phase")
+ return _dedupe_texts(states, max_items=3)
+
+
+class SessionMemoryGraph:
+ def __init__(self, records: Iterable[SessionMemoryRecord] | None = None, turn_index: int = 0):
+ self.turn_index = int(turn_index)
+ self._records: Dict[tuple[str, str, str], SessionMemoryRecord] = {}
+ for record in records or []:
+ self.add_records([record])
+
+ @classmethod
+ def from_dict(cls, payload: Dict[str, Any] | None) -> "SessionMemoryGraph":
+ if not isinstance(payload, dict):
+ return cls()
+ records = []
+ for item in payload.get("records", []) or []:
+ if not isinstance(item, dict):
+ continue
+ records.append(
+ SessionMemoryRecord(
+ memory_id=_clean_text(item.get("memory_id")),
+ category=_clean_text(item.get("category")),
+ value=_clean_text(item.get("value")),
+ relation=_clean_text(item.get("relation")) or "related_to",
+ anchor_concepts=_dedupe_texts(item.get("anchor_concepts", []) or [], max_items=8),
+ salience=float(item.get("salience", 0.6) or 0.6),
+ confidence=float(item.get("confidence", 0.6) or 0.6),
+ source_kind=_clean_text(item.get("source_kind")) or "session_memory",
+ turn_index=int(item.get("turn_index", 0) or 0),
+ metadata=dict(item.get("metadata") or {}),
+ )
+ )
+ return cls(records=records, turn_index=int(payload.get("turn_index", 0) or 0))
+
+ def to_dict(self) -> Dict[str, Any]:
+ return {
+ "turn_index": int(self.turn_index),
+ "records": [record.to_dict() for record in self.records()],
+ }
+
+ def next_turn(self) -> int:
+ self.turn_index += 1
+ return self.turn_index
+
+ def records(self) -> List[SessionMemoryRecord]:
+ ordered = sorted(
+ self._records.values(),
+ key=lambda item: (item.turn_index, item.salience, item.confidence),
+ reverse=True,
+ )
+ return ordered
+
+ def record_count(self) -> int:
+ return len(self._records)
+
+ def add_records(self, records: Iterable[SessionMemoryRecord]) -> int:
+ stored = 0
+ for record in records:
+ key = record.key()
+ existing = self._records.get(key)
+ if existing is None:
+ self._records[key] = record
+ stored += 1
+ continue
+ existing.salience = max(existing.salience, record.salience)
+ existing.confidence = max(existing.confidence, record.confidence)
+ existing.turn_index = max(existing.turn_index, record.turn_index)
+ existing.anchor_concepts = _dedupe_texts([*existing.anchor_concepts, *record.anchor_concepts], max_items=8)
+ existing.metadata.update(record.metadata)
+ return stored
+
+ def build_context(
+ self,
+ query: str,
+ understanding_result: Dict[str, Any] | None = None,
+ extraction_result: Dict[str, Any] | None = None,
+ answer_bundle: Dict[str, Any] | None = None,
+ *,
+ max_records: int = 8,
+ ) -> Dict[str, Any]:
+ anchors = _concept_order(understanding_result, extraction_result, answer_bundle)
+ query_text = _clean_text(query)
+ vague_query = not anchors or _contains_any(query_text, _VAGUE_QUERY_MARKERS)
+ ranked: List[tuple[float, SessionMemoryRecord]] = []
+ for record in self.records():
+ relevance = self._relevance_score(record, query=query_text, anchors=anchors, vague_query=vague_query)
+ if relevance <= 0.0:
+ continue
+ ranked.append((relevance, record))
+ ranked.sort(key=lambda item: item[0], reverse=True)
+ selected = ranked[: max(0, int(max_records))]
+
+ concepts: Dict[str, Dict[str, Any]] = {}
+ relations: List[Dict[str, Any]] = []
+ memory_hits: List[Dict[str, Any]] = []
+ relation_seen = set()
+
+ for relevance, record in selected:
+ target = record.value
+ concepts.setdefault(
+ target,
+ {
+ "concept": target,
+ "type": _TYPE_BY_CATEGORY.get(record.category, "session_memory"),
+ "source_kind": "session_memory",
+ },
+ )
+
+ relation_added = False
+ for anchor in record.anchor_concepts[:4]:
+ if not anchor or anchor == target:
+ continue
+ concepts.setdefault(anchor, {"concept": anchor, "type": "general"})
+ relation_key = (anchor, target, record.relation)
+ if relation_key not in relation_seen:
+ relation_seen.add(relation_key)
+ relations.append(
+ {
+ "from": anchor,
+ "to": target,
+ "relation": record.relation,
+ "weight": round(max(0.35, min(0.95, 0.45 + record.salience * 0.4)), 4),
+ "source_kind": "session_memory",
+ "memory_id": record.memory_id,
+ }
+ )
+ back_key = (target, anchor, "context_for")
+ if back_key not in relation_seen:
+ relation_seen.add(back_key)
+ relations.append(
+ {
+ "from": target,
+ "to": anchor,
+ "relation": "context_for",
+ "weight": round(max(0.25, min(0.85, 0.3 + record.confidence * 0.35)), 4),
+ "source_kind": "session_memory",
+ "memory_id": record.memory_id,
+ }
+ )
+ relation_added = True
+
+ if not relation_added and anchors:
+ anchor = anchors[0]
+ if anchor and anchor != target:
+ relation_key = (anchor, target, record.relation)
+ if relation_key not in relation_seen:
+ relation_seen.add(relation_key)
+ relations.append(
+ {
+ "from": anchor,
+ "to": target,
+ "relation": record.relation,
+ "weight": round(max(0.35, min(0.95, 0.45 + record.salience * 0.4)), 4),
+ "source_kind": "session_memory",
+ "memory_id": record.memory_id,
+ }
+ )
+
+ memory_hits.append(record.to_hit(relevance))
+
+ return {
+ "concepts": list(concepts.values()),
+ "relations": relations,
+ "memory_hits": memory_hits,
+ }
+
+ def _relevance_score(self, record: SessionMemoryRecord, *, query: str, anchors: List[str], vague_query: bool) -> float:
+ query_tokens = _token_set(query)
+ anchor_set = {_normalize_key(item) for item in anchors}
+ record_anchor_set = {_normalize_key(item) for item in record.anchor_concepts}
+ overlap = len(anchor_set & record_anchor_set)
+ value_overlap = len(query_tokens & _token_set(record.value))
+ metadata = dict(record.metadata or {})
+ slot_key_tokens = _token_set(metadata.get("slot_key", ""))
+ state_signature_tokens = _token_set(metadata.get("state_signature", ""))
+ memory_signature_tokens = _token_set(metadata.get("memory_signature", ""))
+ event_signature_tokens = _token_set(metadata.get("event_signature", ""))
+ recency = 1.0 if self.turn_index <= 0 else max(0.0, 1.0 - ((self.turn_index - record.turn_index) / max(1.0, float(self.turn_index))))
+ score = record.salience * 0.5 + record.confidence * 0.25 + recency * 0.15
+ if overlap > 0:
+ score += min(0.4, overlap * 0.15)
+ if value_overlap > 0:
+ score += min(0.22, value_overlap * 0.08)
+ if query_tokens and event_signature_tokens:
+ score += min(0.22, (len(query_tokens & event_signature_tokens) / max(1, len(query_tokens))) * 0.22)
+ if query_tokens and slot_key_tokens:
+ score += min(0.16, (len(query_tokens & slot_key_tokens) / max(1, len(query_tokens))) * 0.16)
+ if query_tokens and state_signature_tokens:
+ score += min(0.18, (len(query_tokens & state_signature_tokens) / max(1, len(query_tokens))) * 0.18)
+ if query_tokens and memory_signature_tokens:
+ score += min(0.24, (len(query_tokens & memory_signature_tokens) / max(1, len(query_tokens))) * 0.24)
+ elif not vague_query and overlap <= 0 and value_overlap <= 0:
+ score -= 0.18
+ if vague_query and record.category in {"goal", "preference", "stage_state"}:
+ score += 0.12
+ return max(0.0, min(1.5, score))
+
+
+def create_tmcra_reasoning_adapter(*, flags: Any | None = None) -> Any:
+ from core.tmcra_reasoning_runtime import create_reasoning_v2_shadow_adapter
+
+ return create_reasoning_v2_shadow_adapter(flags=flags)
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/__init__.py b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/__init__.py
new file mode 100644
index 0000000000000000000000000000000000000000..87aa9e6a3434412977fc7cecf47407d391e9c727
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/__init__.py
@@ -0,0 +1 @@
+"""Benchmark-only experiment adapters."""
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/__init__.py b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/__init__.py
new file mode 100644
index 0000000000000000000000000000000000000000..46cf8408ee900ab95781b6ba3c8781570483c6ec
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/__init__.py
@@ -0,0 +1,24 @@
+_BENCHMARK_EXPORTS = {
+ "BenchmarkConfig",
+ "build_leaderboard",
+ "default_dialog_profiles",
+ "default_llm_profiles",
+ "default_scenario_profiles",
+ "load_eval_cases",
+ "run_long_dialogue_benchmark",
+ "run_reasoner_long_dialogue_benchmark",
+ "run_static_ab_benchmark",
+ "write_benchmark_report",
+}
+
+__all__ = sorted(_BENCHMARK_EXPORTS)
+
+
+def __getattr__(name: str):
+ if name not in _BENCHMARK_EXPORTS:
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+ from . import benchmark as _benchmark
+
+ value = getattr(_benchmark, name)
+ globals()[name] = value
+ return value
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/adapters/__init__.py b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/adapters/__init__.py
new file mode 100644
index 0000000000000000000000000000000000000000..1f76d9adb9eabb26bd3f0d257160e6a797767374
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/adapters/__init__.py
@@ -0,0 +1,62 @@
+from .base import (
+ AdapterResponse,
+ EvalCase,
+ FailureRecord,
+ LeaderboardRecord,
+ LLMProfile,
+ LongDialogProfile,
+ LongDialogProbe,
+ MemoryAdapter,
+ MemoryHit,
+ MemoryRetrieval,
+ ReasoningAdapter,
+ ScenarioProfile,
+)
+
+_MEMORY_EXPORTS = {
+ "FullHistoryMemoryAdapter",
+ "GraphSessionMemoryAdapter",
+ "NullMemoryAdapter",
+ "SummaryWindowMemoryAdapter",
+ "VectorRAGMemoryAdapter",
+}
+_REASONER_EXPORTS = {
+ "DirectExtractionReasoner",
+ "OpenAICompatCoTReasoner",
+ "OpenAICompatDirectReasoner",
+ "OpenAICompatFullContextReasoner",
+ "TriMazeIsolatedReasoner",
+}
+
+__all__ = [
+ "AdapterResponse",
+ "EvalCase",
+ "FailureRecord",
+ "LeaderboardRecord",
+ "LLMProfile",
+ "LongDialogProfile",
+ "LongDialogProbe",
+ "MemoryAdapter",
+ "MemoryHit",
+ "MemoryRetrieval",
+ "ReasoningAdapter",
+ "ScenarioProfile",
+ *_MEMORY_EXPORTS,
+ *_REASONER_EXPORTS,
+]
+
+
+def __getattr__(name: str):
+ if name in _MEMORY_EXPORTS:
+ from . import memory_adapters as _memory_adapters
+
+ value = getattr(_memory_adapters, name)
+ globals()[name] = value
+ return value
+ if name in _REASONER_EXPORTS:
+ from . import reasoning_adapters as _reasoning_adapters
+
+ value = getattr(_reasoning_adapters, name)
+ globals()[name] = value
+ return value
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/adapters/base.py b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/adapters/base.py
new file mode 100644
index 0000000000000000000000000000000000000000..82e6ed89184996701933dcf4dd7df13042b1a352
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/adapters/base.py
@@ -0,0 +1,312 @@
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from dataclasses import dataclass, field
+from typing import Any, Dict, List
+
+
+def _json_safe(value: Any) -> Any:
+ if isinstance(value, dict):
+ return {str(key): _json_safe(item) for key, item in value.items()}
+ if isinstance(value, (list, tuple)):
+ return [_json_safe(item) for item in value]
+ if isinstance(value, set):
+ return [_json_safe(item) for item in sorted(value, key=lambda item: str(item))]
+ return value
+
+
+@dataclass(slots=True)
+class EvalCase:
+ case_id: str
+ query: str
+ answer_mode: str = "transparent"
+ category: str = "general"
+ expected_keywords: List[str] = field(default_factory=list)
+ expected_answer_keywords: List[str] = field(default_factory=list)
+ expected_memory_values: List[str] = field(default_factory=list)
+ expected_absent_values: List[str] = field(default_factory=list)
+ expected_fact_phrases: List[str] = field(default_factory=list)
+ expected_path_concepts: List[str] = field(default_factory=list)
+ metadata: Dict[str, Any] = field(default_factory=dict)
+
+
+@dataclass(slots=True)
+class ScenarioProfile:
+ profile_id: str
+ title: str
+ description: str = ""
+
+
+@dataclass(slots=True)
+class LongDialogProbe:
+ probe_id: str
+ slot: str
+ prompt: str
+ expected_values: List[str] = field(default_factory=list)
+ stale_values: List[str] = field(default_factory=list)
+ false_values: List[str] = field(default_factory=list)
+
+ def to_dict(self) -> Dict[str, Any]:
+ return {
+ "probe_id": self.probe_id,
+ "slot": self.slot,
+ "prompt": self.prompt,
+ "expected_values": list(self.expected_values),
+ "stale_values": list(self.stale_values),
+ "false_values": list(self.false_values),
+ }
+
+
+@dataclass(slots=True)
+class LongDialogProfile:
+ profile_id: str
+ title: str
+ description: str = ""
+
+
+@dataclass(slots=True)
+class LLMProfile:
+ name: str
+ model: str
+ base_url: str = ""
+ api_key: str = ""
+ system_prompt: str = "Use only the supplied evidence and memory."
+ timeout_seconds: float = 60.0
+ temperature: float = 0.1
+ max_tokens: int = 256
+
+
+@dataclass(slots=True)
+class LeaderboardRecord:
+ reasoner: str
+ memory: str
+ reasoning_quality_score: float
+ memory_quality_score: float
+ efficiency_score: float
+ total_score: float
+ metadata: Dict[str, Any] = field(default_factory=dict)
+
+ def to_dict(self) -> Dict[str, Any]:
+ return {
+ "reasoner": self.reasoner,
+ "memory": self.memory,
+ "reasoning_quality_score": round(float(self.reasoning_quality_score), 6),
+ "memory_quality_score": round(float(self.memory_quality_score), 6),
+ "efficiency_score": round(float(self.efficiency_score), 6),
+ "total_score": round(float(self.total_score), 6),
+ "metadata": _json_safe(self.metadata),
+ }
+
+
+@dataclass(slots=True)
+class FailureRecord:
+ benchmark: str
+ reasoner: str = ""
+ memory: str = ""
+ case_id: str = ""
+ probe_id: str = ""
+ reason: str = ""
+ details: Dict[str, Any] = field(default_factory=dict)
+
+ def to_dict(self) -> Dict[str, Any]:
+ return {
+ "benchmark": self.benchmark,
+ "reasoner": self.reasoner,
+ "memory": self.memory,
+ "case_id": self.case_id,
+ "probe_id": self.probe_id,
+ "reason": self.reason,
+ "details": _json_safe(self.details),
+ }
+
+
+@dataclass(slots=True)
+class MemoryHit:
+ memory_id: str
+ category: str
+ value: str
+ relation: str = "related_to"
+ anchors: List[str] = field(default_factory=list)
+ score: float = 0.0
+ source_kind: str = "memory"
+ slot_key: str = ""
+ state: str = "active"
+ turn_index: int = 0
+ metadata: Dict[str, Any] = field(default_factory=dict)
+
+ def to_dict(self) -> Dict[str, Any]:
+ return {
+ "memory_id": self.memory_id,
+ "category": self.category,
+ "value": self.value,
+ "relation": self.relation,
+ "anchors": list(self.anchors),
+ "score": round(float(self.score), 6),
+ "source_kind": self.source_kind,
+ "slot_key": self.slot_key,
+ "state": self.state,
+ "turn_index": int(self.turn_index),
+ "metadata": _json_safe(self.metadata),
+ }
+
+
+@dataclass(slots=True)
+class MemoryRetrieval:
+ concepts: List[Dict[str, Any]] = field(default_factory=list)
+ relations: List[Dict[str, Any]] = field(default_factory=list)
+ hits: List[MemoryHit] = field(default_factory=list)
+ active_hits: List[MemoryHit] = field(default_factory=list)
+ history_hits: List[MemoryHit] = field(default_factory=list)
+ stale_hits: List[MemoryHit] = field(default_factory=list)
+ overwrite_hits: List[MemoryHit] = field(default_factory=list)
+ false_hits: List[MemoryHit] = field(default_factory=list)
+ retrieval_seconds: float = 0.0
+ context_token_estimate: int = 0
+ retrieval_context_token_estimate: int = 0
+ metadata: Dict[str, Any] = field(default_factory=dict)
+
+ def to_dict(self) -> Dict[str, Any]:
+ return {
+ "concepts": _json_safe(self.concepts),
+ "relations": _json_safe(self.relations),
+ "hits": [hit.to_dict() for hit in self.hits],
+ "active_hits": [hit.to_dict() for hit in self.active_hits],
+ "history_hits": [hit.to_dict() for hit in self.history_hits],
+ "stale_hits": [hit.to_dict() for hit in self.stale_hits],
+ "overwrite_hits": [hit.to_dict() for hit in self.overwrite_hits],
+ "false_hits": [hit.to_dict() for hit in self.false_hits],
+ "retrieval_seconds": round(float(self.retrieval_seconds), 6),
+ "context_token_estimate": int(self.context_token_estimate),
+ "retrieval_context_token_estimate": int(self.retrieval_context_token_estimate or self.context_token_estimate),
+ "metadata": _json_safe(self.metadata),
+ }
+
+
+@dataclass(slots=True)
+class AdapterResponse:
+ answer: str
+ answer_mode: str
+ reasoner_name: str
+ memory_name: str
+ confidence: float = 0.0
+ paths: List[Dict[str, Any]] = field(default_factory=list)
+ facts: List[Dict[str, Any]] = field(default_factory=list)
+ candidate_scores: List[Dict[str, Any]] = field(default_factory=list)
+ memory_hits: List[Dict[str, Any]] = field(default_factory=list)
+ evidence_consistent: bool = False
+ unsupported_claims: List[str] = field(default_factory=list)
+ pillar_scores: Dict[str, float] = field(default_factory=dict)
+ latency_seconds: float = 0.0
+ trace: Dict[str, Any] = field(default_factory=dict)
+ metadata: Dict[str, Any] = field(default_factory=dict)
+
+ def to_dict(self) -> Dict[str, Any]:
+ return {
+ "answer": self.answer,
+ "answer_mode": self.answer_mode,
+ "reasoner_name": self.reasoner_name,
+ "memory_name": self.memory_name,
+ "confidence": round(float(self.confidence), 6),
+ "paths": _json_safe(self.paths),
+ "facts": _json_safe(self.facts),
+ "candidate_scores": _json_safe(self.candidate_scores),
+ "memory_hits": _json_safe(self.memory_hits),
+ "evidence_consistent": bool(self.evidence_consistent),
+ "unsupported_claims": _json_safe(self.unsupported_claims),
+ "pillar_scores": {key: round(float(value), 6) for key, value in self.pillar_scores.items()},
+ "latency_seconds": round(float(self.latency_seconds), 6),
+ "trace": _json_safe(self.trace),
+ "metadata": _json_safe(self.metadata),
+ }
+
+
+class MemoryAdapter(ABC):
+ name: str = "memory"
+
+ @abstractmethod
+ def reset(self) -> None:
+ raise NotImplementedError
+
+ @abstractmethod
+ def ingest_turn(
+ self,
+ user_text: str,
+ assistant_text: str = "",
+ *,
+ answer_payload: Dict[str, Any] | None = None,
+ extraction_result: Dict[str, Any] | None = None,
+ ) -> None:
+ raise NotImplementedError
+
+ @abstractmethod
+ def retrieve(self, query: str, *, top_k: int = 6) -> MemoryRetrieval:
+ raise NotImplementedError
+
+ @abstractmethod
+ def stats(self) -> Dict[str, Any]:
+ raise NotImplementedError
+
+ @abstractmethod
+ def storage_bytes(self) -> int:
+ raise NotImplementedError
+
+ def export_dialog_graph(self) -> Dict[str, Any]:
+ return {"summary": {"supported": False}}
+
+ def export_dialog_graph_mermaid(self) -> str:
+ return "graph TD\n"
+
+ def register_answer_support(
+ self,
+ *,
+ answer_id: str,
+ memory_ids: List[str],
+ query_id: str = "",
+ answer_text: str = "",
+ ) -> None:
+ _ = answer_id, memory_ids, query_id, answer_text
+
+ def ingest_answer_writeback(
+ self,
+ *,
+ query_text: str,
+ answer_text: str,
+ answer_id: str,
+ writeback_records: List[Dict[str, Any]],
+ trace: Dict[str, Any] | None = None,
+ ) -> List[str]:
+ _ = query_text, answer_text, answer_id, trace
+ if not writeback_records:
+ return []
+ self.ingest_turn(
+ query_text,
+ answer_text,
+ answer_payload={"replacement_memory_records": list(writeback_records), "metadata": {"memory_write": True, "source": "assistant_writeback"}},
+ extraction_result={},
+ )
+ return [str(item.get("memory_id", "")) for item in writeback_records if isinstance(item, dict) and str(item.get("memory_id", "")).strip()]
+
+ def telemetry_snapshot(self) -> Dict[str, Any]:
+ return {}
+
+ def build_prompt_context(self, query: str, *, top_k: int = 8) -> Dict[str, Any]:
+ return {
+ "mode": "retrieval",
+ "query": query,
+ "retrieval": self.retrieve(query, top_k=top_k).to_dict(),
+ "stats": self.stats(),
+ }
+
+
+class ReasoningAdapter(ABC):
+ name: str = "reasoner"
+
+ @abstractmethod
+ async def answer(
+ self,
+ query: str,
+ *,
+ answer_mode: str,
+ memory_adapter: MemoryAdapter,
+ ) -> AdapterResponse:
+ raise NotImplementedError
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/adapters/memory_adapters.py b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/adapters/memory_adapters.py
new file mode 100644
index 0000000000000000000000000000000000000000..c19dc53436a2a51a9cba5ad8a328429c03d5bb9e
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/adapters/memory_adapters.py
@@ -0,0 +1,10407 @@
+from __future__ import annotations
+
+from collections import deque
+from dataclasses import dataclass, field
+import functools
+import hashlib
+import json
+import math
+import os
+from pathlib import Path
+import re
+import tempfile
+import threading
+import time
+import urllib.error
+import urllib.request
+import uuid
+from typing import Any, Dict, Iterable, List, Mapping, Sequence
+
+from core.session_memory import SessionMemoryExtractor
+from experiments.replacement.memory_graph import (
+ SessionMemoryGraphV2,
+ SessionMemoryEdgeV2,
+ SessionMemoryRecordV2,
+ SQLiteSessionMemoryStore,
+ _clean_text,
+ _estimate_tokens,
+ _normalize,
+ _public_query_subject,
+ _public_slot_root,
+ _public_subject_signature,
+ stable_slot_key,
+ _tokenize,
+ guess_slot_key,
+ infer_category_hints,
+)
+from experiments.replacement.node_memory import (
+ _call_with_supported_kwargs,
+ DEFAULT_MATRIX_EVENT_TOP_K,
+ LoadedNodeMemoryScorer,
+ MEMORY_ROUTER_LAYERS,
+ build_default_path_templates,
+ extract_question_features,
+)
+from experiments.replacement.public_event_signature import compute_public_event_signature
+from experiments.replacement.memory_profiles import TMCRAProfile
+from experiments.replacement.typed_tunnel_augmentation import (
+ annotate_memory_record,
+ merge_typed_metadata,
+ typed_edge_tags_between,
+ typed_tunnel_signature_text,
+)
+from experiments.replacement.profile_layer import infer_profile_query_intent, is_profile_layer_record, profile_query_score_delta
+from experiments.replacement import injection_planner as injection_planner_runtime
+from experiments.replacement.temporal_modeling_types import TemporalFrame, TemporalQueryPlan
+from experiments.replacement.temporal_organizer import TemporalOrganizer
+from experiments.replacement.temporal_query_planner import TemporalQueryPlanner
+from experiments.replacement.temporal_router_runtime import LoadedTemporalRouter
+from experiments.replacement.timeline_evidence_pack import TimelineEvidencePackBuilder
+from experiments.replacement.timeline_state_layer import TimelineStateLayer
+
+from .base import MemoryAdapter, MemoryHit, MemoryRetrieval
+
+
+def _dedupe(items: Iterable[Any], *, max_items: int | None = None) -> List[str]:
+ values: List[str] = []
+ seen = set()
+ for item in items:
+ text = _clean_text(item)
+ if not text:
+ continue
+ key = _normalize(text)
+ if key in seen:
+ continue
+ seen.add(key)
+ values.append(text)
+ if max_items is not None and len(values) >= max_items:
+ break
+ return values
+
+
+def _apply_typed_tunnel_annotations(records: List[SessionMemoryRecordV2], *, source_text: str = "") -> List[SessionMemoryRecordV2]:
+ for record in records:
+ annotate_memory_record(record, source_text=source_text)
+ return records
+
+
+def _estimate_tokens_from_hits(hits: Sequence[MemoryHit]) -> int:
+ total = 0
+ for hit in hits:
+ total += _estimate_tokens(hit.value)
+ total += sum(_estimate_tokens(anchor) for anchor in hit.anchors)
+ return total
+
+
+def _float_env(name: str, default: float) -> float:
+ raw = _clean_text(os.getenv(name, ""))
+ if not raw:
+ return float(default)
+ try:
+ return float(raw)
+ except ValueError:
+ return float(default)
+
+
+def _int_env(name: str, default: int) -> int:
+ raw = _clean_text(os.getenv(name, ""))
+ if not raw:
+ return int(default)
+ try:
+ return int(raw)
+ except ValueError:
+ return int(default)
+
+
+_GRAPH_PROMPT_MAX_CHARS = 12_000
+_GRAPH_PROMPT_MAX_HITS = 8
+_GRAPH_PROMPT_MAX_ACTIVE_SLOTS = 12
+_GRAPH_PROMPT_MAX_RELATIONS = 10
+_HYBRID_SELECTED_EVENT_FLOOR = 8
+_HYBRID_SELECTED_PATH_CAP = 6
+_HYBRID_TEMPORAL_PATH_CAP = 2
+_HYBRID_PROFILE_PATH_CAP = 3
+_MEMORY_ROUTER_GUIDED_MODES = {"guided", "route", "routing", "enforce"}
+_MEMORY_ROUTER_FORCE_MODES = {"force", "forced"}
+_MEMORY_ROUTER_DISABLED_MODES = {"", "off", "disabled", "none", "false", "0"}
+_MEMORY_ROUTER_OBSERVE_MODES = {"observe", "observer", "telemetry", "shadow"}
+_MEMORY_ROUTER_DEFAULT_THRESHOLD = 0.55
+_MEMORY_ROUTER_DEFAULT_MARGIN = 0.08
+_INJECTION_PLANNER_GUIDED_MODES = {"guided", "route", "routing", "enforce"}
+_INJECTION_PLANNER_FORCE_MODES = {"force", "forced"}
+_INJECTION_PLANNER_DISABLED_MODES = {"", "off", "disabled", "none", "false", "0"}
+_INJECTION_PLANNER_OBSERVE_MODES = {"observe", "observer", "telemetry", "shadow"}
+_TEMPORAL_LAYER_DISABLED_MODES = {"off", "disabled", "none", "false", "0"}
+_TEMPORAL_ROUTER_DEFAULT_WRITER_MIN_CONFIDENCE = 0.72
+_TEMPORAL_ROUTER_DEFAULT_QUERY_MIN_CONFIDENCE = 0.85
+_TEMPORAL_ROUTER_DEFAULT_QUERY_INTENT_MIN_CONFIDENCE = 0.60
+_EMBEDDER_INDEX_DISABLED_MODES = {"", "off", "disabled", "none", "false", "0"}
+_EMBEDDER_INDEX_BGE_M3_MODES = {"bge", "bge_m3", "bge-m3", "baai_bge_m3", "baai/bge-m3"}
+_EMBEDDER_INDEX_VERSION = "write_hash_sparse_v1"
+_EMBEDDER_INDEX_BGE_M3_VERSION = "write_bge_m3_dense_sparse_v1"
+_EMBEDDER_MODEL_CACHE: Dict[str, Any] = {}
+_HYBRID_SYMBOLIC_STOPWORDS = {
+ "a",
+ "an",
+ "and",
+ "are",
+ "as",
+ "at",
+ "be",
+ "did",
+ "do",
+ "does",
+ "event",
+ "for",
+ "from",
+ "had",
+ "has",
+ "have",
+ "how",
+ "in",
+ "is",
+ "of",
+ "on",
+ "or",
+ "should",
+ "the",
+ "to",
+ "turn",
+ "was",
+ "were",
+ "what",
+ "when",
+ "where",
+ "which",
+ "who",
+ "why",
+ "will",
+ "with",
+}
+
+
+def _coerce_memory_router_scores(payload: Mapping[str, Any]) -> Dict[str, float]:
+ raw_scores = dict(payload.get("memory_router_scores", {}) or {})
+ scores: Dict[str, float] = {}
+ for layer in MEMORY_ROUTER_LAYERS:
+ try:
+ scores[layer] = float(raw_scores.get(layer, 0.0))
+ except (TypeError, ValueError):
+ scores[layer] = 0.0
+ return scores
+
+
+def _memory_router_decision(
+ payload: Mapping[str, Any],
+ *,
+ mode: str,
+ threshold: float,
+ margin: float,
+) -> Dict[str, Any]:
+ normalized_mode = _normalize(mode)
+ scores = _coerce_memory_router_scores(payload)
+ has_scores = bool(scores) and any(layer in dict(payload.get("memory_router_scores", {}) or {}) for layer in MEMORY_ROUTER_LAYERS)
+ if not has_scores:
+ return {
+ "memory_router_enabled": False,
+ "memory_router_guided": False,
+ "memory_router_reason": "no_router_scores",
+ "memory_router_scores": {},
+ "memory_router_top_layers": [],
+ "memory_router_active_layers": [],
+ "memory_router_score_spread": 0.0,
+ "memory_router_confidence": 0.0,
+ }
+ ranked_layers = [
+ layer
+ for layer, _ in sorted(
+ scores.items(),
+ key=lambda item: (-float(item[1]), item[0]),
+ )
+ ]
+ score_values = list(scores.values())
+ score_spread = max(score_values) - min(score_values)
+ confidence = max(abs(float(score) - 0.5) for score in score_values)
+ resolved_threshold = max(0.0, min(1.0, float(threshold or _MEMORY_ROUTER_DEFAULT_THRESHOLD)))
+ resolved_margin = max(0.0, min(0.5, float(margin or _MEMORY_ROUTER_DEFAULT_MARGIN)))
+ active_layers = [
+ layer
+ for layer in ranked_layers
+ if float(scores.get(layer, 0.0)) >= resolved_threshold
+ ]
+ confident = bool(active_layers) and (score_spread >= resolved_margin or confidence >= resolved_margin)
+ guided_requested = normalized_mode in _MEMORY_ROUTER_GUIDED_MODES
+ forced = normalized_mode in _MEMORY_ROUTER_FORCE_MODES
+ guided = bool(forced or (guided_requested and confident))
+ if forced and not active_layers and ranked_layers:
+ active_layers = [ranked_layers[0]]
+ if active_layers and "event" not in active_layers:
+ active_layers = ["event", *active_layers]
+ reason = "observe"
+ if normalized_mode in _MEMORY_ROUTER_DISABLED_MODES:
+ reason = "disabled"
+ elif normalized_mode in _MEMORY_ROUTER_OBSERVE_MODES:
+ reason = "observe"
+ elif forced:
+ reason = "forced"
+ elif guided_requested and not confident:
+ reason = "low_confidence"
+ elif guided:
+ reason = "guided"
+ return {
+ "memory_router_enabled": True,
+ "memory_router_guided": bool(guided and normalized_mode not in _MEMORY_ROUTER_DISABLED_MODES),
+ "memory_router_reason": reason,
+ "memory_router_scores": scores,
+ "memory_router_top_layers": ranked_layers,
+ "memory_router_active_layers": active_layers,
+ "memory_router_score_spread": round(float(score_spread), 6),
+ "memory_router_confidence": round(float(confidence), 6),
+ "memory_router_threshold": round(float(resolved_threshold), 6),
+ "memory_router_margin": round(float(resolved_margin), 6),
+ "memory_router_mode": normalized_mode or "observe",
+ }
+
+
+def _memory_router_allows(decision: Mapping[str, Any], *layers: str) -> bool:
+ if not bool(decision.get("memory_router_guided")):
+ return True
+ requested = {_normalize(layer) for layer in layers if _normalize(layer)}
+ if not requested:
+ return True
+ if "event" in requested:
+ return True
+ active = {_normalize(layer) for layer in list(decision.get("memory_router_active_layers", []) or [])}
+ return bool(active & requested)
+
+
+def _hybrid_symbolic_tokens(value: Any) -> List[str]:
+ return [
+ token
+ for token in _tokenize(value)
+ if token and token not in _HYBRID_SYMBOLIC_STOPWORDS and not re.fullmatch(r"\d+", token)
+ ]
+
+
+_PATH_UTILITY_STOPWORDS = set(_HYBRID_SYMBOLIC_STOPWORDS) | {
+ "actually",
+ "also",
+ "bit",
+ "feel",
+ "feels",
+ "get",
+ "guess",
+ "i",
+ "im",
+ "it",
+ "its",
+ "just",
+ "kind",
+ "like",
+ "maybe",
+ "me",
+ "more",
+ "much",
+ "my",
+ "really",
+ "something",
+ "still",
+ "think",
+ "thats",
+ "there",
+ "this",
+ "way",
+ "would",
+}
+
+
+def _path_utility_tokens(value: Any) -> List[str]:
+ return [
+ token
+ for token in _hybrid_symbolic_tokens(value)
+ if token and token not in _PATH_UTILITY_STOPWORDS
+ ]
+
+
+_PROFILE_QUERY_ALIAS_GROUPS: tuple[set[str], ...] = (
+ {
+ "accessory",
+ "accessories",
+ "bag",
+ "camera",
+ "cameras",
+ "equipment",
+ "flash",
+ "gear",
+ "lens",
+ "lenses",
+ "photo",
+ "photography",
+ "sony",
+ "tripod",
+ },
+ {
+ "app",
+ "apps",
+ "dashboard",
+ "interface",
+ "layout",
+ "panel",
+ "software",
+ "tool",
+ "tools",
+ "ui",
+ "workflow",
+ },
+ {
+ "diet",
+ "drink",
+ "food",
+ "meal",
+ "restaurant",
+ "snack",
+ "taste",
+ },
+ {
+ "background",
+ "career",
+ "job",
+ "occupation",
+ "position",
+ "previous",
+ "profession",
+ "role",
+ "worked",
+ "work",
+ },
+)
+
+
+_PROFILE_QUERY_GENERIC_TOKENS = set(_PATH_UTILITY_STOPWORDS) | {
+ "able",
+ "about",
+ "any",
+ "anything",
+ "based",
+ "best",
+ "can",
+ "complement",
+ "could",
+ "current",
+ "give",
+ "help",
+ "information",
+ "looking",
+ "make",
+ "need",
+ "please",
+ "recommend",
+ "recommendation",
+ "recommendations",
+ "should",
+ "some",
+ "suggest",
+ "suggestion",
+ "suggestions",
+ "tell",
+ "that",
+ "using",
+ "you",
+}
+
+
+def _profile_query_expanded_tokens(value: Any) -> set[str]:
+ tokens = set(_path_utility_tokens(value))
+ expanded = set(tokens)
+ for group in _PROFILE_QUERY_ALIAS_GROUPS:
+ if tokens & group:
+ expanded.update(group)
+ return expanded
+
+
+def _profile_specific_tokens(tokens: Iterable[Any]) -> set[str]:
+ return {
+ _normalize(token)
+ for token in tokens
+ if _clean_text(token)
+ and _normalize(token) not in _PROFILE_QUERY_GENERIC_TOKENS
+ and not re.fullmatch(r"\d+", _normalize(token))
+ }
+
+
+def _profile_hit_match_score(query_tokens: set[str], expanded_query_tokens: set[str], hit: MemoryHit) -> tuple[float, List[str], List[str]]:
+ metadata = dict(hit.metadata or {})
+ payload_parts: List[Any] = [
+ hit.category,
+ hit.relation,
+ hit.slot_key,
+ hit.value,
+ *list(hit.anchors or []),
+ metadata.get("profile_summary", ""),
+ metadata.get("profile_value", ""),
+ metadata.get("profile_type", ""),
+ metadata.get("profile_domain", ""),
+ metadata.get("profile_domain_label", ""),
+ metadata.get("semantic_slot", ""),
+ metadata.get("subject", ""),
+ metadata.get("extracted_subject", ""),
+ metadata.get("profile_cluster_domains", []),
+ metadata.get("profile_cluster_types", []),
+ metadata.get("profile_cluster_route_terms", []),
+ metadata.get("profile_route_terms", []),
+ metadata.get("profile_support_values", []),
+ ]
+ payload_text = " ".join(str(item) for item in payload_parts)
+ record_raw_tokens = set(_path_utility_tokens(payload_text))
+ record_tokens = _profile_query_expanded_tokens(payload_text)
+ specific_query_tokens = _profile_specific_tokens(query_tokens)
+ raw_overlap_tokens = sorted(specific_query_tokens & _profile_specific_tokens(record_raw_tokens))
+ expanded_overlap_tokens = sorted(_profile_specific_tokens(expanded_query_tokens) & _profile_specific_tokens(record_tokens))
+ overlap_tokens = raw_overlap_tokens or expanded_overlap_tokens
+ overlap_ratio = float(len(overlap_tokens)) / float(max(1, len(specific_query_tokens)))
+ profile_type = _normalize(metadata.get("profile_type", ""))
+ source_kind = _normalize(hit.source_kind)
+ source_bonus = 0.0
+ if source_kind in {"public_dialog_preference", "public_dialog_goal", "public_dialog_profile"}:
+ source_bonus += 0.08
+ if bool(metadata.get("profile_candidate_status") == "consolidated"):
+ source_bonus += 0.04
+ if bool(metadata.get("profile_cluster_node")):
+ source_bonus -= 0.16
+ type_bonus = 0.0
+ if profile_type in {"preference", "goal", "setup", "usage_context"} and (
+ {"recommend", "suggest", "suited", "accessory", "accessories", "gear", "equipment", "setup", "current"} & query_tokens
+ ):
+ type_bonus += 0.10
+ if profile_type in {"setup", "usage_context"} and {"current", "setup", "profile"} & query_tokens:
+ type_bonus += 0.08
+ raw_bonus = 0.18 * len(raw_overlap_tokens)
+ expanded_bonus = 0.08 * max(0, len(expanded_overlap_tokens) - len(raw_overlap_tokens))
+ score = raw_bonus + expanded_bonus + (0.58 * overlap_ratio) + source_bonus + type_bonus
+ return round(score, 6), overlap_tokens, raw_overlap_tokens
+
+
+def _bounded_event_id_union(*groups: Iterable[Any], max_items: int) -> List[str]:
+ return _dedupe((item for group in groups for item in group), max_items=max(1, int(max_items)))
+
+
+_DEEPSEEK_GRAPH_TEACHER_EXAMPLE_CACHE: Dict[str, List[Dict[str, Any]]] = {}
+
+
+def _deepseek_graph_teacher_mode() -> str:
+ mode = _normalize(os.getenv("TMCRA_DEEPSEEK_GRAPH_MODEL_MODE", "off"))
+ if mode in {"1", "true", "yes", "on", "replace", "teacher"}:
+ return "replace"
+ if mode in {"shadow", "observe", "observer", "telemetry"}:
+ return "shadow"
+ if mode in {"fusion", "fuse", "merge"}:
+ return "fusion"
+ return "off"
+
+
+def _deepseek_graph_teacher_api_key() -> str:
+ explicit = _clean_text(os.getenv("TMCRA_DEEPSEEK_GRAPH_MODEL_API_KEY", ""))
+ if explicit:
+ return explicit
+ for name in (
+ "TMCRA_DEEPSEEK_GRAPH_MODEL_KEY",
+ "DEEPSEEK_API_KEY",
+ "TMCRA_WRITER_API_KEY",
+ "OPENAI_API_KEY",
+ ):
+ value = _clean_text(os.getenv(name, ""))
+ if value:
+ return value
+ pool = os.getenv("TMCRA_DEEPSEEK_WRITER_KEY_POOL", "")
+ for part in re.split(r"[\s,;]+", pool):
+ value = _clean_text(part)
+ if value:
+ return value
+ return ""
+
+
+def _deepseek_graph_teacher_extract_json(text: Any) -> Dict[str, Any]:
+ payload = _clean_text(text)
+ if not payload:
+ return {}
+ try:
+ parsed = json.loads(payload)
+ return dict(parsed) if isinstance(parsed, dict) else {}
+ except Exception:
+ pass
+ start = payload.find("{")
+ end = payload.rfind("}")
+ if start >= 0 and end > start:
+ try:
+ parsed = json.loads(payload[start : end + 1])
+ return dict(parsed) if isinstance(parsed, dict) else {}
+ except Exception:
+ return {}
+ return {}
+
+
+def _deepseek_graph_teacher_score(value: Any, default: float = 0.0) -> float:
+ try:
+ score = float(value)
+ except (TypeError, ValueError):
+ score = float(default)
+ if math.isnan(score) or math.isinf(score):
+ score = float(default)
+ return max(0.0, min(1.0, float(score)))
+
+
+def _deepseek_graph_teacher_score_map(raw: Any, valid_ids: set[str]) -> Dict[str, float]:
+ if not isinstance(raw, Mapping):
+ return {}
+ scores: Dict[str, float] = {}
+ for raw_id, raw_score in raw.items():
+ item_id = _clean_text(raw_id)
+ if not item_id or item_id not in valid_ids:
+ continue
+ scores[item_id] = _deepseek_graph_teacher_score(raw_score)
+ return scores
+
+
+def _deepseek_graph_teacher_id_list(raw: Any, valid_ids: set[str], *, max_items: int) -> List[str]:
+ if not isinstance(raw, Sequence) or isinstance(raw, (str, bytes, bytearray)):
+ return []
+ return _dedupe(
+ (_clean_text(item) for item in raw if _clean_text(item) in valid_ids),
+ max_items=max(1, int(max_items)),
+ )
+
+
+def _deepseek_graph_teacher_ranked_ids(scores: Mapping[str, float], *, max_items: int) -> List[str]:
+ return [
+ item_id
+ for item_id, _ in sorted(
+ ((item_id, float(score)) for item_id, score in scores.items() if _clean_text(item_id)),
+ key=lambda item: (-float(item[1]), item[0]),
+ )
+ ][: max(1, int(max_items))]
+
+
+def _deepseek_graph_teacher_compact_graph(
+ runtime_graph: Mapping[str, Any],
+ candidate_event_ids: Sequence[str],
+ *,
+ max_events: int,
+ max_paths: int,
+) -> Dict[str, Any]:
+ event_candidates = _dedupe(candidate_event_ids, max_items=max(1, int(max_events)))
+ event_candidate_set = set(event_candidates)
+ event_nodes: List[Dict[str, Any]] = []
+ for node in list(runtime_graph.get("nodes", []) or []):
+ if _clean_text(node.get("type", "")) != "event":
+ continue
+ event_id = _clean_text(node.get("id", ""))
+ if event_candidate_set and event_id not in event_candidate_set:
+ continue
+ metadata = dict(node.get("metadata", {}) or {})
+ event_nodes.append(
+ {
+ "id": event_id,
+ "text": _clean_text(node.get("text", ""))[:700],
+ "speaker": _clean_text(node.get("speaker", "")),
+ "turn_index": int(node.get("turn_index", 0) or 0),
+ "session_name": _clean_text(node.get("session_name", "")),
+ "slot_key": _clean_text(node.get("slot_key", "")),
+ "state_signature": _clean_text(node.get("state_signature", "")),
+ "target_status": _clean_text(node.get("target_status", "")),
+ "time_value": _clean_text(node.get("time_value", "")),
+ "time_display_value": _clean_text(node.get("time_display_value", "")),
+ "profile_type": _clean_text(node.get("profile_type", "")),
+ "profile_value": _clean_text(node.get("profile_value", ""))[:300],
+ "subject_signature": _clean_text(node.get("subject_signature", "")),
+ "tunnel_roles": list(node.get("tmcra_tunnel_roles", []) or [])[:8],
+ "tunnel_group_key": _clean_text(node.get("tmcra_tunnel_group_key", "")),
+ "teacher_fields": dict(node.get("teacher_fields", {}) or {}),
+ "metadata_tags": {
+ "source_kind": _clean_text(metadata.get("source_kind", "")),
+ "semantic_slot": _clean_text(metadata.get("semantic_slot", "")),
+ "event_signature": _clean_text(metadata.get("event_signature", ""))[:240],
+ },
+ }
+ )
+ if not event_nodes:
+ for node in list(runtime_graph.get("nodes", []) or []):
+ if _clean_text(node.get("type", "")) == "event":
+ event_id = _clean_text(node.get("id", ""))
+ if event_id:
+ event_nodes.append({"id": event_id, "text": _clean_text(node.get("text", ""))[:700]})
+ if len(event_nodes) >= max(1, int(max_events)):
+ break
+ compact_event_ids = {_clean_text(node.get("id", "")) for node in event_nodes if _clean_text(node.get("id", ""))}
+ path_rows: List[Dict[str, Any]] = []
+ for path in list(runtime_graph.get("paths", []) or []):
+ path_id = _clean_text(path.get("id", ""))
+ event_id = _clean_text(path.get("event_id", ""))
+ if not path_id or (compact_event_ids and event_id not in compact_event_ids):
+ continue
+ path_rows.append(
+ {
+ "id": path_id,
+ "event_id": event_id,
+ "type": _clean_text(path.get("type", "")),
+ "text": _clean_text(path.get("text", ""))[:700],
+ "support_node_ids": list(path.get("support_node_ids", []) or [])[:8],
+ "tags": list(path.get("tags", []) or [])[:8],
+ }
+ )
+ if len(path_rows) >= max(1, int(max_paths)):
+ break
+ tunnel_rows: List[Dict[str, Any]] = []
+ for edge in list(runtime_graph.get("typed_tunnel_edges", []) or []):
+ source = _clean_text(edge.get("source", edge.get("from", "")))
+ target = _clean_text(edge.get("target", edge.get("to", "")))
+ if compact_event_ids and source not in compact_event_ids and target not in compact_event_ids:
+ continue
+ tunnel_rows.append(
+ {
+ "source": source,
+ "target": target,
+ "type": _clean_text(edge.get("type", edge.get("relation", ""))),
+ "roles": list(edge.get("roles", []) or [])[:8],
+ "score": edge.get("score", edge.get("weight", 0.0)),
+ }
+ )
+ if len(tunnel_rows) >= max(1, int(max_paths)):
+ break
+ return {
+ "events": event_nodes,
+ "paths": path_rows,
+ "typed_tunnel_edges": tunnel_rows,
+ }
+
+
+def _deepseek_graph_teacher_examples() -> List[Dict[str, Any]]:
+ path = _clean_text(os.getenv("TMCRA_DEEPSEEK_GRAPH_MODEL_EXAMPLES_JSONL", ""))
+ if not path:
+ return []
+ limit = max(0, _int_env("TMCRA_DEEPSEEK_GRAPH_MODEL_EXAMPLE_LIMIT", 4))
+ if limit <= 0:
+ return []
+ cache_key = f"{path}|{limit}"
+ if cache_key in _DEEPSEEK_GRAPH_TEACHER_EXAMPLE_CACHE:
+ return list(_DEEPSEEK_GRAPH_TEACHER_EXAMPLE_CACHE[cache_key])
+ examples: List[Dict[str, Any]] = []
+ try:
+ with open(path, "r", encoding="utf-8") as handle:
+ for line in handle:
+ if len(examples) >= limit:
+ break
+ line = line.strip()
+ if not line:
+ continue
+ try:
+ row = json.loads(line)
+ except Exception:
+ continue
+ if not isinstance(row, Mapping):
+ continue
+ question = _clean_text(row.get("question", row.get("query", "")))
+ output = row.get("output", row.get("target", row.get("teacher_output", {})))
+ if not question or not isinstance(output, Mapping):
+ continue
+ compact = {
+ "question": question[:500],
+ "gold": {
+ "selected_event_ids": list(output.get("selected_event_ids", row.get("selected_event_ids", [])) or [])[:8],
+ "selected_path_ids": list(output.get("selected_path_ids", row.get("selected_path_ids", [])) or [])[:8],
+ "focused_answer_type": _clean_text(output.get("focused_answer_type", row.get("focused_answer_type", ""))),
+ },
+ }
+ positive_event_ids = list(row.get("positive_event_ids", row.get("target_event_ids", [])) or [])[:8]
+ positive_path_ids = list(row.get("positive_path_ids", row.get("target_path_ids", [])) or [])[:8]
+ if positive_event_ids:
+ compact["gold"]["positive_event_ids"] = positive_event_ids
+ if positive_path_ids:
+ compact["gold"]["positive_path_ids"] = positive_path_ids
+ examples.append(compact)
+ except OSError:
+ examples = []
+ _DEEPSEEK_GRAPH_TEACHER_EXAMPLE_CACHE[cache_key] = list(examples)
+ return examples
+
+
+def _deepseek_graph_teacher_normalize_output(
+ raw: Mapping[str, Any],
+ runtime_graph: Mapping[str, Any],
+ *,
+ mode: str,
+ model: str,
+ usage: Mapping[str, Any] | None,
+ elapsed_ms: int,
+ candidate_event_ids: Sequence[str],
+ top_k: int,
+ support_path_k: int,
+) -> Dict[str, Any]:
+ valid_event_ids = {
+ _clean_text(node.get("id", ""))
+ for node in list(runtime_graph.get("nodes", []) or [])
+ if _clean_text(node.get("type", "")) == "event" and _clean_text(node.get("id", ""))
+ }
+ valid_path_ids = {
+ _clean_text(path.get("id", ""))
+ for path in list(runtime_graph.get("paths", []) or [])
+ if _clean_text(path.get("id", ""))
+ }
+ event_scores = _deepseek_graph_teacher_score_map(raw.get("event_scores", {}), valid_event_ids)
+ path_scores = _deepseek_graph_teacher_score_map(raw.get("path_scores", {}), valid_path_ids)
+ selected_event_ids = _deepseek_graph_teacher_id_list(
+ raw.get("selected_event_ids", []),
+ valid_event_ids,
+ max_items=max(top_k, support_path_k * 2, _HYBRID_SELECTED_EVENT_FLOOR),
+ )
+ recall_event_ids = _deepseek_graph_teacher_id_list(
+ raw.get("recall_event_ids", []),
+ valid_event_ids,
+ max_items=max(_int_env("TMCRA_DEEPSEEK_GRAPH_MODEL_RECALL_EVENT_K", 24), len(selected_event_ids), top_k),
+ )
+ selected_path_ids = _deepseek_graph_teacher_id_list(
+ raw.get("selected_path_ids", []),
+ valid_path_ids,
+ max_items=max(1, support_path_k),
+ )
+ for rank, event_id in enumerate(selected_event_ids, start=1):
+ event_scores[event_id] = max(float(event_scores.get(event_id, 0.0)), max(0.68, 1.0 - (rank - 1) * 0.035))
+ if not selected_event_ids:
+ selected_event_ids = _deepseek_graph_teacher_ranked_ids(
+ event_scores,
+ max_items=max(top_k, support_path_k * 2, _HYBRID_SELECTED_EVENT_FLOOR),
+ )
+ if not recall_event_ids:
+ recall_event_ids = _bounded_event_id_union(
+ selected_event_ids,
+ _deepseek_graph_teacher_ranked_ids(event_scores, max_items=max(24, len(candidate_event_ids))),
+ candidate_event_ids,
+ max_items=max(24, len(candidate_event_ids), top_k),
+ )
+ for rank, event_id in enumerate(recall_event_ids, start=1):
+ event_scores[event_id] = max(float(event_scores.get(event_id, 0.0)), max(0.30, 0.84 - (rank - 1) * 0.015))
+ for rank, path_id in enumerate(selected_path_ids, start=1):
+ path_scores[path_id] = max(float(path_scores.get(path_id, 0.0)), max(0.64, 1.0 - (rank - 1) * 0.05))
+ if not selected_path_ids:
+ selected_path_ids = _deepseek_graph_teacher_ranked_ids(path_scores, max_items=max(1, support_path_k))
+ if not path_scores and selected_event_ids:
+ for path in list(runtime_graph.get("paths", []) or []):
+ path_id = _clean_text(path.get("id", ""))
+ event_id = _clean_text(path.get("event_id", ""))
+ if path_id and event_id in set(selected_event_ids):
+ path_scores[path_id] = max(0.40, float(event_scores.get(event_id, 0.5)))
+ raw_plan = raw.get("answer_plan_scores", {})
+ answer_plan_scores: Dict[str, Dict[str, float]] = {}
+ if isinstance(raw_plan, Mapping):
+ for role in ("selected", "current", "historical", "suppressed"):
+ answer_plan_scores[role] = _deepseek_graph_teacher_score_map(raw_plan.get(role, {}), valid_event_ids)
+ for role in ("selected", "current", "historical", "suppressed"):
+ answer_plan_scores.setdefault(role, {})
+ for event_id in selected_event_ids:
+ score = max(float(event_scores.get(event_id, 0.0)), 0.62)
+ answer_plan_scores["selected"][event_id] = max(float(answer_plan_scores["selected"].get(event_id, 0.0)), score)
+ answer_plan_scores["current"][event_id] = max(float(answer_plan_scores["current"].get(event_id, 0.0)), min(1.0, score * 0.92))
+ answer_type_scores = {
+ key: _deepseek_graph_teacher_score(value)
+ for key, value in dict(raw.get("answer_type_scores", {}) or {}).items()
+ if _clean_text(key)
+ }
+ if not answer_type_scores:
+ answer_type_scores = {"multi_evidence": 0.55, "direct_fact": 0.45}
+ memory_router_scores = _deepseek_graph_teacher_score_map(raw.get("memory_router_scores", {}), set(MEMORY_ROUTER_LAYERS))
+ if not memory_router_scores:
+ focused_type = _normalize(raw.get("focused_answer_type", ""))
+ memory_router_scores = {layer: 0.35 for layer in MEMORY_ROUTER_LAYERS}
+ memory_router_scores["event"] = 0.75
+ if focused_type in {"time", "temporal", "temporal_reasoning"}:
+ memory_router_scores["temporal"] = 0.78
+ if focused_type in {"profile", "preference", "resource"}:
+ memory_router_scores["profile"] = 0.78
+ memory_router_scores["resource"] = 0.68
+ if focused_type in {"multi", "multi_evidence", "aggregation", "chain"}:
+ memory_router_scores["path_tunnel"] = 0.72
+ memory_router_scores["topic_tunnel"] = 0.62
+ path_tunnel_support_scores = _deepseek_graph_teacher_score_map(raw.get("path_tunnel_support_scores", {}), valid_path_ids)
+ path_tunnel_delta_scores = _deepseek_graph_teacher_score_map(raw.get("path_tunnel_delta_scores", {}), valid_path_ids)
+ event_tunnel_support_scores = _deepseek_graph_teacher_score_map(raw.get("event_tunnel_support_scores", {}), valid_event_ids)
+ event_tunnel_delta_scores = _deepseek_graph_teacher_score_map(raw.get("event_tunnel_delta_scores", {}), valid_event_ids)
+ focused_answer_type = _clean_text(raw.get("focused_answer_type", "")) or max(
+ answer_type_scores.items(),
+ key=lambda item: (float(item[1]), item[0]),
+ )[0]
+ metadata = {
+ "mode": mode,
+ "status": "ok",
+ "model": model,
+ "elapsed_ms": int(elapsed_ms),
+ "selected_event_count": int(len(selected_event_ids)),
+ "selected_path_count": int(len(selected_path_ids)),
+ "usage": {
+ "prompt_tokens": int(dict(usage or {}).get("prompt_tokens", 0) or 0),
+ "completion_tokens": int(dict(usage or {}).get("completion_tokens", 0) or 0),
+ "total_tokens": int(dict(usage or {}).get("total_tokens", 0) or 0),
+ },
+ "reason": _clean_text(raw.get("reason", ""))[:500],
+ }
+ return {
+ "recall_event_ids": list(recall_event_ids),
+ "rerank_candidate_event_ids": list(recall_event_ids),
+ "selected_event_ids": list(selected_event_ids),
+ "selected_path_ids": list(selected_path_ids),
+ "recall_event_scores": dict(event_scores),
+ "base_event_scores": dict(event_scores),
+ "rerank_event_scores": dict(event_scores),
+ "calibrated_event_scores": dict(event_scores),
+ "matrix_event_scores": dict(event_scores),
+ "event_scores": dict(event_scores),
+ "event_fusion_delta_scores": {},
+ "event_tunnel_support_scores": dict(event_tunnel_support_scores),
+ "event_tunnel_delta_scores": dict(event_tunnel_delta_scores),
+ "base_path_scores": dict(path_scores),
+ "calibrated_path_scores": dict(path_scores),
+ "rerank_path_scores": dict(path_scores),
+ "matrix_path_scores": dict(path_scores),
+ "path_model_scores": dict(path_scores),
+ "path_scores": dict(path_scores),
+ "path_fusion_delta_scores": {},
+ "path_tunnel_support_scores": dict(path_tunnel_support_scores),
+ "path_tunnel_delta_scores": dict(path_tunnel_delta_scores),
+ "answer_plan_scores": dict(answer_plan_scores),
+ "answer_type_scores": dict(answer_type_scores),
+ "focused_answer_type": focused_answer_type,
+ "memory_router_scores": dict(memory_router_scores),
+ "fusion_enabled": True,
+ "event_fusion_enabled": True,
+ "path_fusion_enabled": True,
+ "event_calibration_enabled": True,
+ "path_calibration_enabled": True,
+ "event_tunnel_enabled": bool(event_tunnel_support_scores or event_tunnel_delta_scores),
+ "path_tunnel_enabled": bool(path_tunnel_support_scores or path_tunnel_delta_scores or selected_path_ids),
+ "final_event_fusion_enabled": True,
+ "final_path_fusion_enabled": True,
+ "decision_fusion_enabled": True,
+ "decision_score_source": "deepseek_graph_teacher",
+ "matrix_enabled": False,
+ "matrix_path_enabled": False,
+ "path_chain_extension_enabled": False,
+ "deepseek_graph_model": metadata,
+ }
+
+
+def _deepseek_graph_teacher_call(
+ runtime_graph: Mapping[str, Any],
+ question: str,
+ *,
+ candidate_event_ids: Sequence[str],
+ top_k: int,
+ support_path_k: int,
+ local_scored: Mapping[str, Any] | None = None,
+) -> Dict[str, Any]:
+ mode = _deepseek_graph_teacher_mode()
+ if mode == "off":
+ return {}
+ api_key = _deepseek_graph_teacher_api_key()
+ base_url = _clean_text(os.getenv("TMCRA_DEEPSEEK_GRAPH_MODEL_BASE_URL", "https://api.deepseek.com/v1")).rstrip("/")
+ model = _clean_text(os.getenv("TMCRA_DEEPSEEK_GRAPH_MODEL_MODEL", os.getenv("TMCRA_WRITER_MODEL", "deepseek-chat")))
+ if not api_key or not base_url or not model:
+ return {
+ "deepseek_graph_model": {
+ "mode": mode,
+ "status": "skipped_missing_config",
+ "model": model,
+ }
+ }
+ max_events = max(4, _int_env("TMCRA_DEEPSEEK_GRAPH_MODEL_MAX_EVENTS", 28))
+ max_paths = max(4, _int_env("TMCRA_DEEPSEEK_GRAPH_MODEL_MAX_PATHS", 64))
+ compact_graph = _deepseek_graph_teacher_compact_graph(
+ runtime_graph,
+ candidate_event_ids,
+ max_events=max_events,
+ max_paths=max_paths,
+ )
+ recall_limit = max(8, min(24, _int_env("TMCRA_DEEPSEEK_GRAPH_MODEL_RECALL_EVENT_K", 24)))
+ selected_event_limit = max(top_k, support_path_k * 2, _HYBRID_SELECTED_EVENT_FLOOR)
+ selected_path_limit = max(1, support_path_k)
+ prompt_payload = {
+ "task": "score_runtime_graph",
+ "question": _clean_text(question),
+ "graph": compact_graph,
+ "local_model_hints": {
+ "recall_event_ids": list(dict(local_scored or {}).get("recall_event_ids", []) or [])[:16],
+ "selected_event_ids": list(dict(local_scored or {}).get("selected_event_ids", []) or [])[:12],
+ "selected_path_ids": list(dict(local_scored or {}).get("selected_path_ids", []) or [])[:8],
+ "focused_answer_type": _clean_text(dict(local_scored or {}).get("focused_answer_type", "")),
+ },
+ "few_shot_training_examples": _deepseek_graph_teacher_examples(),
+ "output_contract": {
+ "format": "one compact JSON object only; no markdown; no code fence; no explanation outside JSON",
+ "purpose": "rank evidence ids for the graph runtime; do not answer the user question",
+ "limits": {
+ "recall_event_ids": recall_limit,
+ "selected_event_ids": selected_event_limit,
+ "selected_path_ids": selected_path_limit,
+ "reason_chars": 180,
+ },
+ "important": [
+ "Do not emit full event_scores or full path_scores maps.",
+ "Score maps are optional and must be sparse: only include ids that appear in selected_event_ids or selected_path_ids.",
+ "Prefer ranked id arrays over verbose per-id scoring.",
+ "For direct-fact questions, choose a compact evidence-covering set.",
+ "For counting, listing, aggregation, or multi-evidence questions, selected_event_ids must cover all distinct answer-unit events within the limit; do not collapse separate instances into one semantic cluster.",
+ "For counting, listing, aggregation, or multi-evidence questions, recall_event_ids should include plausible borderline positive events before distractors so later evidence packing can preserve coverage.",
+ "Prefer one representative event per distinct answer unit before adding duplicate turns from the same unit.",
+ ],
+ },
+ "output_schema": {
+ "recall_event_ids": [f"up to {recall_limit} ranked event ids"],
+ "selected_event_ids": [f"up to {selected_event_limit} ranked event ids"],
+ "selected_path_ids": [f"up to {selected_path_limit} ranked path ids"],
+ "event_scores": {"optional selected event id only": "0..1"},
+ "path_scores": {"optional selected path id only": "0..1"},
+ "event_tunnel_support_scores": {"optional selected event id only": "0..1"},
+ "path_tunnel_support_scores": {"optional selected path id only": "0..1"},
+ "answer_plan_scores": {
+ "selected": {"optional selected event id only": "0..1"},
+ "current": {"optional selected event id only": "0..1"},
+ "historical": {"optional selected event id only": "0..1"},
+ "suppressed": {"optional selected event id only": "0..1"},
+ },
+ "answer_type_scores": {"direct_fact|multi_evidence|time|profile": "0..1"},
+ "focused_answer_type": "direct_fact|multi_evidence|time|profile|unknown",
+ "memory_router_scores": {"event|profile|resource|temporal|path_tunnel|topic_tunnel": "0..1"},
+ "reason": "short diagnostic, <=180 chars",
+ },
+ }
+ system_prompt = (
+ "You are the TMCRA runtime graph teacher. You replace a learned graph scorer, not the answer model. "
+ "Given a query and a compact memory graph, activate query-conditioned typed relations, score events and paths, "
+ "select evidence paths, and produce answer-plan scores. Use only provided ids. "
+ "Do not answer the user question. Return one compact strict JSON object only. "
+ "Never wrap JSON in markdown. Never output full score maps."
+ )
+ max_tokens = max(256, min(_int_env("TMCRA_DEEPSEEK_GRAPH_MODEL_MAX_TOKENS", 1400), 4096))
+ request_payload: Dict[str, Any] = {
+ "model": model,
+ "messages": [
+ {"role": "system", "content": system_prompt},
+ {"role": "user", "content": json.dumps(prompt_payload, ensure_ascii=False)},
+ ],
+ "temperature": _float_env("TMCRA_DEEPSEEK_GRAPH_MODEL_TEMPERATURE", 0.0),
+ "max_tokens": max_tokens,
+ }
+ if _normalize(os.getenv("TMCRA_DEEPSEEK_GRAPH_MODEL_RESPONSE_FORMAT", "json_object")) not in {"off", "none", "false", "0"}:
+ request_payload["response_format"] = {"type": "json_object"}
+ timeout_seconds = max(5.0, _float_env("TMCRA_DEEPSEEK_GRAPH_MODEL_TIMEOUT_SECONDS", 120.0))
+ started = time.perf_counter()
+ try:
+ req = urllib.request.Request(
+ f"{base_url}/chat/completions",
+ data=json.dumps(request_payload, ensure_ascii=False).encode("utf-8"),
+ headers={"Content-Type": "application/json", "Authorization": f"Bearer {api_key}"},
+ method="POST",
+ )
+ with urllib.request.urlopen(req, timeout=timeout_seconds) as response:
+ response_payload = json.loads(response.read().decode("utf-8"))
+ elapsed_ms = int((time.perf_counter() - started) * 1000)
+ trace_path = _clean_text(os.getenv("TMCRA_DEEPSEEK_GRAPH_MODEL_TRACE_PATH", ""))
+ if trace_path:
+ try:
+ trace_file = Path(trace_path)
+ trace_file.parent.mkdir(parents=True, exist_ok=True)
+ trace_record = {
+ "question": _clean_text(question),
+ "model": model,
+ "mode": mode,
+ "elapsed_ms": elapsed_ms,
+ "request_chars": len(json.dumps(request_payload, ensure_ascii=False)),
+ "response": response_payload,
+ }
+ with trace_file.open("a", encoding="utf-8") as handle:
+ handle.write(json.dumps(trace_record, ensure_ascii=False) + "\n")
+ except Exception:
+ pass
+ choices = list(response_payload.get("choices", []) or [])
+ choice = dict(choices[0] if choices else {})
+ message = dict(choice.get("message", {}) or {})
+ content: Any = message.get("content", "")
+ if isinstance(content, Mapping):
+ raw = dict(content)
+ raw_preview = json.dumps(raw, ensure_ascii=False)[:500]
+ else:
+ if isinstance(content, Sequence) and not isinstance(content, (str, bytes, bytearray)):
+ content = "\n".join(
+ _clean_text(item.get("text", item.get("content", "")) if isinstance(item, Mapping) else item)
+ for item in content
+ )
+ if not _clean_text(content):
+ content = message.get("reasoning_content", "") or choice.get("text", "")
+ raw_preview = _clean_text(content)[:500]
+ raw = _deepseek_graph_teacher_extract_json(content)
+ if not raw:
+ return {
+ "deepseek_graph_model": {
+ "mode": mode,
+ "status": "empty_or_invalid_json",
+ "model": model,
+ "elapsed_ms": elapsed_ms,
+ "finish_reason": _clean_text(choice.get("finish_reason", "")),
+ "raw_preview": raw_preview,
+ }
+ }
+ return _deepseek_graph_teacher_normalize_output(
+ raw,
+ runtime_graph,
+ mode=mode,
+ model=model,
+ usage=dict(response_payload.get("usage", {}) or {}),
+ elapsed_ms=elapsed_ms,
+ candidate_event_ids=candidate_event_ids,
+ top_k=top_k,
+ support_path_k=support_path_k,
+ )
+ except Exception as exc:
+ return {
+ "deepseek_graph_model": {
+ "mode": mode,
+ "status": "failed",
+ "model": model,
+ "elapsed_ms": int((time.perf_counter() - started) * 1000),
+ "error": f"{exc.__class__.__name__}: {_clean_text(str(exc))[:220]}",
+ }
+ }
+
+
+def _deepseek_graph_teacher_merge(local_scored: Mapping[str, Any], teacher_scored: Mapping[str, Any]) -> Dict[str, Any]:
+ merged = dict(local_scored or {})
+ weight = max(0.0, min(1.0, _float_env("TMCRA_DEEPSEEK_GRAPH_MODEL_FUSION_WEIGHT", 0.55)))
+ for key in (
+ "recall_event_scores",
+ "base_event_scores",
+ "rerank_event_scores",
+ "calibrated_event_scores",
+ "matrix_event_scores",
+ "event_scores",
+ "event_tunnel_support_scores",
+ "event_tunnel_delta_scores",
+ "base_path_scores",
+ "calibrated_path_scores",
+ "rerank_path_scores",
+ "matrix_path_scores",
+ "path_model_scores",
+ "path_scores",
+ "path_tunnel_support_scores",
+ "path_tunnel_delta_scores",
+ ):
+ local_map = dict(merged.get(key, {}) or {})
+ teacher_map = dict(teacher_scored.get(key, {}) or {})
+ for item_id, teacher_score in teacher_map.items():
+ local_score = _deepseek_graph_teacher_score(local_map.get(item_id, 0.0))
+ fused = max(local_score, (1.0 - weight) * local_score + weight * _deepseek_graph_teacher_score(teacher_score))
+ local_map[item_id] = round(float(fused), 6)
+ merged[key] = local_map
+ merged["selected_event_ids"] = _bounded_event_id_union(
+ list(teacher_scored.get("selected_event_ids", []) or []),
+ list(merged.get("selected_event_ids", []) or []),
+ max_items=max(_HYBRID_SELECTED_EVENT_FLOOR, len(list(merged.get("selected_event_ids", []) or []))),
+ )
+ merged["selected_path_ids"] = _dedupe(
+ [
+ *list(teacher_scored.get("selected_path_ids", []) or []),
+ *list(merged.get("selected_path_ids", []) or []),
+ ],
+ max_items=max(1, _int_env("TMCRA_DEEPSEEK_GRAPH_MODEL_FUSION_PATH_K", 8)),
+ )
+ merged["recall_event_ids"] = _bounded_event_id_union(
+ list(teacher_scored.get("recall_event_ids", []) or []),
+ list(merged.get("recall_event_ids", []) or []),
+ max_items=max(24, _int_env("TMCRA_DEEPSEEK_GRAPH_MODEL_RECALL_EVENT_K", 24)),
+ )
+ local_plan = dict(merged.get("answer_plan_scores", {}) or {})
+ teacher_plan = dict(teacher_scored.get("answer_plan_scores", {}) or {})
+ for role in ("selected", "current", "historical", "suppressed"):
+ role_map = dict(local_plan.get(role, {}) or {})
+ for event_id, teacher_score in dict(teacher_plan.get(role, {}) or {}).items():
+ role_map[event_id] = max(_deepseek_graph_teacher_score(role_map.get(event_id, 0.0)), _deepseek_graph_teacher_score(teacher_score))
+ local_plan[role] = role_map
+ merged["answer_plan_scores"] = local_plan
+ merged["answer_type_scores"] = {
+ **dict(merged.get("answer_type_scores", {}) or {}),
+ **dict(teacher_scored.get("answer_type_scores", {}) or {}),
+ }
+ if _clean_text(teacher_scored.get("focused_answer_type", "")):
+ merged["focused_answer_type"] = _clean_text(teacher_scored.get("focused_answer_type", ""))
+ merged["deepseek_graph_model"] = dict(teacher_scored.get("deepseek_graph_model", {}) or {})
+ merged["deepseek_graph_model"]["fusion_weight"] = round(float(weight), 6)
+ merged["decision_score_source"] = f"{_clean_text(merged.get('decision_score_source', 'learned_decision_fusion'))}+deepseek_graph_teacher"
+ return merged
+
+
+def _deepseek_graph_teacher_replace_preserve_local(
+ local_scored: Mapping[str, Any],
+ teacher_scored: Mapping[str, Any],
+) -> Dict[str, Any]:
+ """Let the teacher own scores while preserving base graph hard evidence seats."""
+ merged = dict(teacher_scored or {})
+ local_selected_events = list(local_scored.get("selected_event_ids", []) or [])
+ local_selected_paths = list(local_scored.get("selected_path_ids", []) or [])
+ teacher_selected_events = list(teacher_scored.get("selected_event_ids", []) or [])
+ teacher_selected_paths = list(teacher_scored.get("selected_path_ids", []) or [])
+ preserve_local = _normalize(os.getenv("TMCRA_DEEPSEEK_GRAPH_MODEL_REPLACE_PRESERVE_LOCAL", "off")) not in {
+ "0",
+ "false",
+ "off",
+ "no",
+ "none",
+ }
+ if not preserve_local:
+ return merged
+
+ merged["selected_event_ids"] = _bounded_event_id_union(
+ teacher_selected_events,
+ local_selected_events,
+ max_items=max(_HYBRID_SELECTED_EVENT_FLOOR, len(local_selected_events), len(teacher_selected_events)),
+ )
+ merged["selected_path_ids"] = _dedupe(
+ [*teacher_selected_paths, *local_selected_paths],
+ max_items=max(1, len(local_selected_paths), len(teacher_selected_paths)),
+ )
+ merged["recall_event_ids"] = _bounded_event_id_union(
+ list(teacher_scored.get("recall_event_ids", []) or []),
+ list(local_scored.get("recall_event_ids", []) or []),
+ list(local_scored.get("rerank_candidate_event_ids", []) or []),
+ max_items=max(24, _int_env("TMCRA_DEEPSEEK_GRAPH_MODEL_RECALL_EVENT_K", 24)),
+ )
+ merged["rerank_candidate_event_ids"] = _bounded_event_id_union(
+ list(merged.get("recall_event_ids", []) or []),
+ list(local_scored.get("rerank_candidate_event_ids", []) or []),
+ max_items=max(24, _int_env("TMCRA_DEEPSEEK_GRAPH_MODEL_RECALL_EVENT_K", 24)),
+ )
+
+ event_score_keys = (
+ "recall_event_scores",
+ "base_event_scores",
+ "rerank_event_scores",
+ "calibrated_event_scores",
+ "matrix_event_scores",
+ "event_scores",
+ "event_tunnel_support_scores",
+ "event_tunnel_delta_scores",
+ )
+ for key in event_score_keys:
+ score_map = dict(merged.get(key, {}) or {})
+ local_map = dict(local_scored.get(key, {}) or {})
+ for rank, event_id in enumerate(local_selected_events, start=1):
+ local_score = _deepseek_graph_teacher_score(local_map.get(event_id, 0.0))
+ score_map[event_id] = max(float(score_map.get(event_id, 0.0) or 0.0), max(0.58, local_score))
+ for rank, event_id in enumerate(list(local_scored.get("recall_event_ids", []) or [])[:24], start=1):
+ local_score = _deepseek_graph_teacher_score(local_map.get(event_id, 0.0))
+ score_map[event_id] = max(float(score_map.get(event_id, 0.0) or 0.0), max(0.28, local_score * 0.85))
+ merged[key] = score_map
+
+ path_score_keys = (
+ "base_path_scores",
+ "calibrated_path_scores",
+ "rerank_path_scores",
+ "matrix_path_scores",
+ "path_model_scores",
+ "path_scores",
+ "path_tunnel_support_scores",
+ "path_tunnel_delta_scores",
+ )
+ for key in path_score_keys:
+ score_map = dict(merged.get(key, {}) or {})
+ local_map = dict(local_scored.get(key, {}) or {})
+ for path_id in local_selected_paths:
+ local_score = _deepseek_graph_teacher_score(local_map.get(path_id, 0.0))
+ score_map[path_id] = max(float(score_map.get(path_id, 0.0) or 0.0), max(0.52, local_score))
+ merged[key] = score_map
+
+ local_plan = dict(local_scored.get("answer_plan_scores", {}) or {})
+ teacher_plan = dict(merged.get("answer_plan_scores", {}) or {})
+ for role in ("selected", "current", "historical", "suppressed"):
+ role_map = dict(teacher_plan.get(role, {}) or {})
+ local_role_map = dict(local_plan.get(role, {}) or {})
+ if role in {"selected", "current"}:
+ for event_id in local_selected_events:
+ role_map[event_id] = max(
+ _deepseek_graph_teacher_score(role_map.get(event_id, 0.0)),
+ _deepseek_graph_teacher_score(local_role_map.get(event_id, 0.0)),
+ 0.48,
+ )
+ teacher_plan[role] = role_map
+ merged["answer_plan_scores"] = teacher_plan
+
+ metadata = dict(merged.get("deepseek_graph_model", {}) or {})
+ metadata["replace_preserve_local"] = True
+ metadata["preserved_local_event_count"] = int(len(local_selected_events))
+ metadata["preserved_local_path_count"] = int(len(local_selected_paths))
+ metadata["selected_event_count"] = int(len(merged.get("selected_event_ids", []) or []))
+ metadata["selected_path_count"] = int(len(merged.get("selected_path_ids", []) or []))
+ merged["deepseek_graph_model"] = metadata
+ merged["decision_score_source"] = "deepseek_graph_teacher+local_hard_seats"
+ return merged
+
+
+def _build_symbolic_recall_index(
+ runtime_graph: Mapping[str, Any],
+ *,
+ grouped_hits: Mapping[str, Sequence[MemoryHit]],
+) -> Dict[str, Any]:
+ nodes_by_id = {
+ _clean_text(node.get("id", "")): dict(node)
+ for node in list(runtime_graph.get("nodes", []) or [])
+ if _clean_text(node.get("id", ""))
+ }
+ event_payloads: Dict[str, List[str]] = {}
+ for node_id, node in nodes_by_id.items():
+ if _clean_text(node.get("type", "")) == "event":
+ metadata = dict(node.get("metadata", {}) or {})
+ teacher_fields = dict(node.get("teacher_fields", {}) or {})
+ event_payloads.setdefault(node_id, []).extend(
+ [
+ node.get("text", ""),
+ node.get("speaker", ""),
+ node.get("slot_key", ""),
+ node.get("target_status", ""),
+ node.get("profile_type", ""),
+ node.get("profile_value", ""),
+ *teacher_fields.values(),
+ *metadata.values(),
+ ]
+ )
+ for path in list(runtime_graph.get("paths", []) or []):
+ event_id = _clean_text(path.get("event_id", ""))
+ support_node = nodes_by_id.get(_clean_text(path.get("target", "")), {})
+ if event_id and support_node:
+ event_payloads.setdefault(event_id, []).append(support_node.get("text", ""))
+ for event_id, group_hits in grouped_hits.items():
+ payloads = event_payloads.setdefault(_clean_text(event_id), [])
+ for hit in list(group_hits or []):
+ metadata = dict(hit.metadata or {})
+ payloads.extend([hit.value, hit.slot_key, hit.category, hit.relation, *hit.anchors, *metadata.values()])
+
+ return {
+ "event_tokens": {
+ event_id: frozenset(_hybrid_symbolic_tokens(payloads))
+ for event_id, payloads in event_payloads.items()
+ },
+ "event_turn_indices": {
+ event_id: int(nodes_by_id.get(event_id, {}).get("turn_index", 0) or 0)
+ for event_id in event_payloads
+ },
+ }
+
+
+def _symbolic_recall_event_ids(
+ query: str,
+ runtime_graph: Mapping[str, Any],
+ *,
+ grouped_hits: Mapping[str, Sequence[MemoryHit]],
+ limit: int,
+ symbolic_index: Mapping[str, Any] | None = None,
+) -> List[str]:
+ question_features = extract_question_features(query)
+ query_tokens = set(_hybrid_symbolic_tokens(question_features.get("question_anchor_tokens", []) or query))
+ if not query_tokens:
+ return []
+ resolved_index = dict(symbolic_index or _build_symbolic_recall_index(runtime_graph, grouped_hits=grouped_hits))
+ event_tokens_by_id = dict(resolved_index.get("event_tokens", {}) or {})
+ event_turn_indices = dict(resolved_index.get("event_turn_indices", {}) or {})
+
+ scored_events: List[tuple[str, float]] = []
+ for event_id, event_tokens in event_tokens_by_id.items():
+ overlap = query_tokens & event_tokens
+ if not overlap:
+ continue
+ overlap_ratio = float(len(overlap)) / float(max(1, len(query_tokens)))
+ turn_index = int(event_turn_indices.get(event_id, 0) or 0)
+ scored_events.append((event_id, (len(overlap) * 4.0) + overlap_ratio + min(turn_index, 1000) * 0.000001))
+ return [
+ event_id
+ for event_id, _ in sorted(scored_events, key=lambda item: (-float(item[1]), item[0]))
+ ][: max(1, int(limit))]
+
+
+_EMBEDDER_INDEX_METADATA_TEXT_KEYS = (
+ "raw_text",
+ "source_turn_text",
+ "source_span",
+ "event_phrase",
+ "event_summary",
+ "profile_value",
+ "profile_summary",
+ "profile_type",
+ "profile_domain",
+ "profile_domain_label",
+ "semantic_slot",
+ "target_status",
+ "subject",
+ "subject_signature",
+ "canonical_slot_key",
+ "resource_key",
+ "resolved_date",
+ "resolved_time_value",
+ "time_value",
+ "time_display_value",
+ "time_granularity",
+ "speaker",
+ "session_name",
+ "topic_label",
+ "topic_bucket_id",
+ "origin_query",
+ "writeback_class",
+ "depth_layer",
+ "memory_chain_depth_layer",
+)
+_EMBEDDER_INDEX_METADATA_LIST_KEYS = (
+ "topic_keywords",
+ "profile_route_terms",
+ "profile_cluster_route_terms",
+ "profile_support_values",
+ "evidence_anchors",
+ "support_memory_ids",
+ "support_fact_refs",
+ "support_path_refs",
+)
+
+
+def _embedder_index_enabled(mode: Any) -> bool:
+ return _normalize(mode) not in _EMBEDDER_INDEX_DISABLED_MODES
+
+
+def _embedder_index_uses_bge_m3(mode: Any) -> bool:
+ return _normalize(mode).replace("-", "_") in {item.replace("-", "_") for item in _EMBEDDER_INDEX_BGE_M3_MODES}
+
+
+def _embedder_index_version_for_mode(mode: Any) -> str:
+ return _EMBEDDER_INDEX_BGE_M3_VERSION if _embedder_index_uses_bge_m3(mode) else _EMBEDDER_INDEX_VERSION
+
+
+def _embedder_index_text_items(value: Any, *, max_items: int = 64) -> List[str]:
+ items: List[str] = []
+
+ def visit(item: Any) -> None:
+ if len(items) >= max_items:
+ return
+ if item is None:
+ return
+ if isinstance(item, Mapping):
+ for key, nested in list(item.items()):
+ if len(items) >= max_items:
+ break
+ key_text = _clean_text(key)
+ if isinstance(nested, (str, int, float, bool)):
+ value_text = _clean_text(nested)[:800]
+ if value_text:
+ items.append(f"{key_text} {value_text}".strip())
+ else:
+ visit(nested)
+ return
+ if isinstance(item, (list, tuple, set)):
+ for nested in list(item):
+ if len(items) >= max_items:
+ break
+ visit(nested)
+ return
+ text = _clean_text(item)[:800]
+ if text:
+ items.append(text)
+
+ visit(value)
+ return items[:max_items]
+
+
+def _embedder_index_term_weights(value: Any, *, max_terms: int = 96) -> Dict[str, float]:
+ text = _clean_text(value)
+ if not text:
+ return {}
+ counts: Dict[str, float] = {}
+ for token in _path_utility_tokens(text):
+ token = _normalize(token)
+ if not token or len(token) > 64:
+ continue
+ counts[token] = counts.get(token, 0.0) + 1.0
+ normalized_text = _normalize(text)
+ cjk_chars = [char for char in normalized_text if "\u4e00" <= char <= "\u9fff"]
+ for width, weight in ((2, 1.35), (3, 1.15)):
+ if len(cjk_chars) < width:
+ continue
+ for index in range(0, len(cjk_chars) - width + 1):
+ gram = "".join(cjk_chars[index : index + width])
+ if gram:
+ counts[gram] = counts.get(gram, 0.0) + weight
+ if not counts:
+ return {}
+ ranked = sorted(counts.items(), key=lambda item: (-float(item[1]), item[0]))[: max(1, int(max_terms or 1))]
+ norm = math.sqrt(sum(float(weight) * float(weight) for _, weight in ranked)) or 1.0
+ return {term: round(float(weight) / norm, 6) for term, weight in ranked}
+
+
+def _embedder_dense_vectors_for_texts(texts: Sequence[str], *, mode: str) -> tuple[List[List[float]], Dict[str, Any]]:
+ normalized_mode = _normalize(mode)
+ if not _embedder_index_uses_bge_m3(normalized_mode):
+ return [[] for _ in texts], {"write_embedder_dense_enabled": False}
+ clean_texts = [_clean_text(text) for text in texts]
+ metadata: Dict[str, Any] = {
+ "write_embedder_dense_enabled": False,
+ "write_embedder_dense_backend": "bge_m3_transformers",
+ "write_embedder_dense_model": _clean_text(os.getenv("TMCRA_EMBEDDER_MODEL_PATH", "")) or "BAAI/bge-m3",
+ }
+ if not any(clean_texts):
+ metadata["write_embedder_dense_error"] = "empty_texts"
+ return [[] for _ in texts], metadata
+ try:
+ import torch # type: ignore
+ from transformers import AutoModel, AutoTokenizer # type: ignore
+ except Exception as exc:
+ metadata["write_embedder_dense_error"] = f"dependency_unavailable:{exc.__class__.__name__}"
+ return [[] for _ in texts], metadata
+ model_name = metadata["write_embedder_dense_model"]
+ device = _clean_text(os.getenv("TMCRA_EMBEDDER_DEVICE", ""))
+ if not device:
+ device = "cuda" if bool(getattr(torch, "cuda", None) and torch.cuda.is_available()) else "cpu"
+ try:
+ max_length = max(64, int(os.getenv("TMCRA_EMBEDDER_MODEL_MAX_LENGTH", "512") or 512))
+ except (TypeError, ValueError):
+ max_length = 512
+ cache_key = f"bge_m3::{model_name}::{device}::{max_length}"
+ try:
+ cached = _EMBEDDER_MODEL_CACHE.get(cache_key)
+ if cached is None:
+ tokenizer = AutoTokenizer.from_pretrained(
+ model_name,
+ local_files_only=True,
+ trust_remote_code=False,
+ )
+ model = AutoModel.from_pretrained(
+ model_name,
+ local_files_only=True,
+ trust_remote_code=False,
+ )
+ model.to(device)
+ model.eval()
+ cached = (tokenizer, model)
+ _EMBEDDER_MODEL_CACHE[cache_key] = cached
+ tokenizer, model = cached
+ encoded = tokenizer(
+ clean_texts,
+ padding=True,
+ truncation=True,
+ max_length=max_length,
+ return_tensors="pt",
+ )
+ encoded = {key: value.to(device) for key, value in encoded.items()}
+ with torch.no_grad():
+ output = model(**encoded)
+ hidden = output.last_hidden_state
+ mask = encoded.get("attention_mask")
+ if mask is not None:
+ mask = mask.unsqueeze(-1).expand(hidden.size()).float()
+ pooled = torch.sum(hidden * mask, dim=1) / torch.clamp(mask.sum(dim=1), min=1e-9)
+ else:
+ pooled = hidden[:, 0]
+ pooled = torch.nn.functional.normalize(pooled, p=2, dim=1)
+ vectors = [
+ [round(float(value), 6) for value in row.detach().cpu().tolist()]
+ for row in pooled
+ ]
+ metadata.update(
+ {
+ "write_embedder_dense_enabled": True,
+ "write_embedder_dense_device": device,
+ "write_embedder_dense_dim": int(len(vectors[0]) if vectors else 0),
+ "write_embedder_dense_max_length": int(max_length),
+ }
+ )
+ return vectors, metadata
+ except Exception as exc:
+ metadata["write_embedder_dense_error"] = f"{exc.__class__.__name__}:{_clean_text(exc)[:240]}"
+ return [[] for _ in texts], metadata
+
+
+def _prewarm_embedder_dense_if_requested(*, mode: str) -> Dict[str, Any]:
+ flag = _normalize(os.getenv("TMCRA_EMBEDDER_PREWARM", ""))
+ if flag in _EMBEDDER_INDEX_DISABLED_MODES or flag not in {"1", "true", "yes", "on", "auto"}:
+ return {"embedder_prewarm_enabled": False}
+ normalized_mode = _normalize(mode)
+ if not _embedder_index_uses_bge_m3(normalized_mode):
+ return {
+ "embedder_prewarm_enabled": False,
+ "embedder_prewarm_reason": "mode_not_dense",
+ "embedder_prewarm_mode": normalized_mode or "off",
+ }
+ warmup_text = _clean_text(os.getenv("TMCRA_EMBEDDER_PREWARM_TEXT", "")) or "tmcra memory retrieval warmup"
+ vectors, metadata = _embedder_dense_vectors_for_texts([warmup_text], mode=normalized_mode)
+ return {
+ "embedder_prewarm_enabled": bool(vectors and vectors[0]),
+ "embedder_prewarm_mode": normalized_mode,
+ "embedder_prewarm_dense_enabled": bool(metadata.get("write_embedder_dense_enabled")),
+ "embedder_prewarm_dense_device": metadata.get("write_embedder_dense_device", ""),
+ "embedder_prewarm_dense_error": metadata.get("write_embedder_dense_error", ""),
+ }
+
+
+def _embedder_index_record_text(record: SessionMemoryRecordV2, *, turn_text: str = "") -> str:
+ metadata = dict(record.metadata or {})
+ parts: List[str] = [
+ record.category,
+ record.slot_key,
+ record.value,
+ record.value,
+ record.value,
+ record.relation,
+ *list(record.anchor_concepts or []),
+ *list(record.anchor_concepts or []),
+ *list(record.evidence_anchors or []),
+ ]
+ has_record_evidence_text = any(
+ _clean_text(metadata.get(key, ""))
+ for key in ("raw_text", "source_turn_text", "source_span", "event_phrase", "profile_value", "profile_summary")
+ )
+ if turn_text and not has_record_evidence_text:
+ parts.append(_clean_text(turn_text)[:1000])
+ for key in _EMBEDDER_INDEX_METADATA_TEXT_KEYS:
+ value = metadata.get(key)
+ if value:
+ parts.append(f"{key} {' '.join(_embedder_index_text_items(value, max_items=12))}".strip())
+ for key in _EMBEDDER_INDEX_METADATA_LIST_KEYS:
+ value = metadata.get(key)
+ if value:
+ parts.append(f"{key} {' '.join(_embedder_index_text_items(value, max_items=24))}".strip())
+ return _clean_text(" ".join(_clean_text(part) for part in parts if _clean_text(part)))
+
+
+def _apply_write_embedder_index_to_graph(
+ graph: SessionMemoryGraphV2,
+ *,
+ stored_ids: Sequence[str],
+ turn_text: str,
+ turn_index: int,
+ mode: str,
+ max_terms: int,
+) -> Dict[str, Any]:
+ normalized_mode = _normalize(mode)
+ index_version = _embedder_index_version_for_mode(normalized_mode)
+ metadata: Dict[str, Any] = {
+ "write_embedder_index_enabled": False,
+ "write_embedder_index_mode": normalized_mode or "off",
+ "write_embedder_index_version": index_version,
+ "write_embedder_index_record_count": 0,
+ }
+ if not _embedder_index_enabled(normalized_mode) or not stored_ids:
+ return metadata
+ indexed_ids: List[str] = []
+ index_rows: List[tuple[SessionMemoryRecordV2, str, Dict[str, float]]] = []
+ for memory_id in _dedupe(stored_ids):
+ record = getattr(graph, "records_by_id", {}).get(memory_id)
+ if record is None:
+ continue
+ index_text = _embedder_index_record_text(record, turn_text=turn_text)
+ terms = _embedder_index_term_weights(
+ index_text,
+ max_terms=max_terms,
+ )
+ if not terms:
+ continue
+ index_rows.append((record, index_text, terms))
+ dense_vectors, dense_metadata = _embedder_dense_vectors_for_texts(
+ [index_text for _, index_text, _ in index_rows],
+ mode=normalized_mode,
+ )
+ metadata.update(dense_metadata)
+ for row_index, (record, _, terms) in enumerate(index_rows):
+ dense_vector = dense_vectors[row_index] if row_index < len(dense_vectors) else []
+ record_metadata = dict(record.metadata or {})
+ record_metadata.update(
+ {
+ "write_embedder_index_enabled": True,
+ "write_embedder_index_mode": normalized_mode,
+ "write_embedder_index_version": index_version,
+ "write_embedder_index_source_turn": int(turn_index),
+ "write_embedder_index_term_count": int(len(terms)),
+ "write_embedder_index_terms": dict(terms),
+ "write_embedder_index_top_terms": list(terms.keys())[:24],
+ }
+ )
+ if dense_vector:
+ record_metadata.update(
+ {
+ "write_embedder_dense_enabled": True,
+ "write_embedder_dense_backend": dense_metadata.get("write_embedder_dense_backend", ""),
+ "write_embedder_dense_model": dense_metadata.get("write_embedder_dense_model", ""),
+ "write_embedder_dense_dim": int(len(dense_vector)),
+ "write_embedder_dense_vector": list(dense_vector),
+ }
+ )
+ elif _embedder_index_uses_bge_m3(normalized_mode):
+ record_metadata.update(
+ {
+ "write_embedder_dense_enabled": False,
+ "write_embedder_dense_error": dense_metadata.get("write_embedder_dense_error", "dense_vector_unavailable"),
+ }
+ )
+ record.metadata = record_metadata
+ indexed_ids.append(record.memory_id)
+ metadata.update(
+ {
+ "write_embedder_index_enabled": bool(indexed_ids),
+ "write_embedder_index_record_count": int(len(indexed_ids)),
+ "write_embedder_index_record_ids": list(indexed_ids[:24]),
+ }
+ )
+ return metadata
+
+
+def _embedder_index_recall_event_ids(
+ query: str,
+ *,
+ grouped_hits: Mapping[str, Sequence[MemoryHit]],
+ mode: str,
+ limit: int,
+ max_terms: int,
+) -> Dict[str, Any]:
+ normalized_mode = _normalize(mode)
+ index_version = _embedder_index_version_for_mode(normalized_mode)
+ metadata: Dict[str, Any] = {
+ "embedder_index_recall_enabled": False,
+ "embedder_index_recall_mode": normalized_mode or "off",
+ "embedder_index_recall_version": index_version,
+ "embedder_index_event_ids": [],
+ "embedder_index_event_scores": {},
+ "embedder_index_record_count": 0,
+ }
+ if not _embedder_index_enabled(normalized_mode):
+ return {"event_ids": [], "metadata": metadata}
+ query_terms = _embedder_index_term_weights(query, max_terms=max_terms)
+ query_vectors, query_dense_metadata = _embedder_dense_vectors_for_texts([query], mode=normalized_mode)
+ query_vector = query_vectors[0] if query_vectors else []
+ metadata.update(
+ {
+ "embedder_dense_recall_enabled": bool(query_vector),
+ "embedder_dense_recall_backend": query_dense_metadata.get("write_embedder_dense_backend", ""),
+ "embedder_dense_recall_model": query_dense_metadata.get("write_embedder_dense_model", ""),
+ "embedder_dense_recall_error": query_dense_metadata.get("write_embedder_dense_error", ""),
+ }
+ )
+ if not query_terms and not query_vector:
+ metadata["embedder_index_recall_reason"] = "empty_query_terms"
+ return {"event_ids": [], "metadata": metadata}
+ scored_events: List[tuple[str, float, int]] = []
+ indexed_record_count = 0
+ dense_record_count = 0
+ for event_id, group_hits in grouped_hits.items():
+ event_score = 0.0
+ event_turn = 0
+ for hit in list(group_hits or []):
+ hit_metadata = dict(hit.metadata or {})
+ raw_terms = hit_metadata.get("write_embedder_index_terms")
+ raw_vector = hit_metadata.get("write_embedder_dense_vector")
+ if not isinstance(raw_terms, Mapping) and not isinstance(raw_vector, list):
+ continue
+ indexed_record_count += int(isinstance(raw_terms, Mapping))
+ dense_record_count += int(isinstance(raw_vector, list) and bool(raw_vector))
+ sparse_score = 0.0
+ if isinstance(raw_terms, Mapping):
+ for term, query_weight in query_terms.items():
+ try:
+ sparse_score += float(query_weight) * float(raw_terms.get(term, 0.0) or 0.0)
+ except (TypeError, ValueError):
+ continue
+ dense_score = 0.0
+ if query_vector and isinstance(raw_vector, list) and raw_vector:
+ for query_value, record_value in zip(query_vector, raw_vector):
+ try:
+ dense_score += float(query_value) * float(record_value)
+ except (TypeError, ValueError):
+ continue
+ score = max(float(sparse_score), float(dense_score) + (0.15 * float(sparse_score) if dense_score > 0.0 else 0.0))
+ if score <= 0.0:
+ continue
+ event_score = max(event_score, float(score))
+ event_turn = max(event_turn, int(hit.turn_index or 0))
+ if event_score > 0.0:
+ scored_events.append((_clean_text(event_id), round(event_score, 6), event_turn))
+ scored_events.sort(key=lambda item: (-float(item[1]), -int(item[2]), item[0]))
+ selected = scored_events[: max(1, int(limit or 1))]
+ event_ids = [event_id for event_id, _, _ in selected if event_id]
+ event_scores = {event_id: score for event_id, score, _ in selected if event_id}
+ metadata.update(
+ {
+ "embedder_index_recall_enabled": True,
+ "embedder_index_event_ids": list(event_ids),
+ "embedder_index_event_scores": dict(event_scores),
+ "embedder_index_query_terms": list(query_terms.keys())[:32],
+ "embedder_index_record_count": int(indexed_record_count),
+ "embedder_dense_record_count": int(dense_record_count),
+ }
+ )
+ return {"event_ids": event_ids, "metadata": metadata}
+
+
+_IDENTIFIER_GENERIC_TOKENS = {
+ "api",
+ "agent",
+ "code",
+ "codename",
+ "context",
+ "debug",
+ "goal",
+ "memory",
+ "model",
+ "name",
+ "project",
+ "retrieval",
+ "runtime",
+ "session",
+ "target",
+ "test",
+ "turn",
+}
+_IDENTIFIER_REQUEST_RE = re.compile(
+ r"\b(code\s*name|codename|identifier|alias|project\s+name|model\s+name|api\s+name)\b|"
+ r"(代号|编号|标识符|名称|名字|别名|项目名|模型名|接口名)",
+ flags=re.IGNORECASE,
+)
+_IDENTIFIER_TOKEN_RE = re.compile(r"\b[A-Za-z][A-Za-z0-9_-]{2,63}\b")
+
+
+def _query_identifier_tokens(query: Any) -> List[str]:
+ text = _clean_text(query)
+ tokens: List[str] = []
+ for match in _IDENTIFIER_TOKEN_RE.finditer(text):
+ token = match.group(0)
+ lowered = token.lower()
+ if lowered in _IDENTIFIER_GENERIC_TOKENS:
+ continue
+ has_inner_upper = any(ch.isupper() for ch in token[1:])
+ has_digit = any(ch.isdigit() for ch in token)
+ has_joiner = "-" in token or "_" in token
+ if has_inner_upper or has_digit or has_joiner:
+ tokens.append(token)
+ return _dedupe(tokens, max_items=8)
+
+
+def _query_requests_identifier_fact(query: Any) -> bool:
+ return bool(_IDENTIFIER_REQUEST_RE.search(_clean_text(query)))
+
+
+def _hit_text_for_identifier_match(hit: MemoryHit) -> str:
+ metadata = dict(hit.metadata or {})
+ values = [
+ hit.memory_id,
+ hit.category,
+ hit.value,
+ hit.relation,
+ hit.slot_key,
+ hit.source_kind,
+ *list(hit.anchors or []),
+ metadata.get("raw_text", ""),
+ metadata.get("source_turn_text", ""),
+ metadata.get("source_span", ""),
+ metadata.get("event_phrase", ""),
+ metadata.get("profile_value", ""),
+ metadata.get("target_status", ""),
+ ]
+ return " ".join(_clean_text(value) for value in values if _clean_text(value))
+
+
+def _copy_hit_with_identifier_boost(hit: MemoryHit, *, score: float, reasons: Sequence[str], matched_tokens: Sequence[str]) -> MemoryHit:
+ metadata = dict(hit.metadata or {})
+ metadata.update(
+ {
+ "identifier_protected": True,
+ "identifier_protected_score": round(float(score), 6),
+ "identifier_protected_reasons": list(reasons)[:8],
+ "identifier_protected_matched_tokens": list(matched_tokens)[:8],
+ "identifier_protected_original_score": round(float(hit.score), 6),
+ }
+ )
+ return MemoryHit(
+ memory_id=hit.memory_id,
+ category=hit.category,
+ value=hit.value,
+ relation=hit.relation,
+ anchors=list(hit.anchors),
+ score=max(float(hit.score), round(1.0 + float(score), 6)),
+ source_kind=hit.source_kind,
+ slot_key=hit.slot_key,
+ state=hit.state,
+ turn_index=int(hit.turn_index),
+ metadata=metadata,
+ )
+
+
+def _identifier_protected_hits(
+ *,
+ query: str,
+ final_hits: Sequence[MemoryHit],
+ candidate_hits: Sequence[MemoryHit],
+ top_k: int,
+) -> Dict[str, Any]:
+ identifier_tokens = _query_identifier_tokens(query)
+ identifier_request = _query_requests_identifier_fact(query)
+ if not identifier_tokens and not identifier_request:
+ return {"enabled": False, "hits": list(final_hits), "promoted_hits": [], "metadata": {"identifier_protected_enabled": False}}
+
+ query_text = _clean_text(query)
+ pool: Dict[str, MemoryHit] = {}
+ for hit in list(final_hits) + list(candidate_hits):
+ key = hit.memory_id or f"{_hit_event_id(hit)}::{hit.slot_key}::{hit.value[:80]}"
+ if key and key not in pool:
+ pool[key] = hit
+
+ scored: List[tuple[float, MemoryHit, List[str], List[str]]] = []
+ for hit in pool.values():
+ hit_text = _hit_text_for_identifier_match(hit)
+ hit_lower = hit_text.lower()
+ matched_tokens = [token for token in identifier_tokens if token.lower() in hit_lower]
+ reasons: List[str] = []
+ score = 0.0
+ if matched_tokens:
+ score += 12.0 + float(len(matched_tokens))
+ reasons.append("exact_identifier_match")
+ if identifier_request:
+ if any(term in hit_lower for term in ("codename", "code name", "identifier", "alias", "project_codename")):
+ score += 8.0
+ reasons.append("identifier_field_match")
+ if any(term in hit_text for term in ("代号", "编号", "标识符", "名称", "名字", "别名")):
+ score += 8.0
+ reasons.append("identifier_cjk_field_match")
+ if "项目" in query_text and "项目" in hit_text:
+ score += 1.5
+ reasons.append("project_term_match")
+ if score <= 0:
+ continue
+ scored.append((score, hit, matched_tokens, reasons))
+
+ scored.sort(key=lambda item: (item[0], float(item[1].score), int(item[1].turn_index or 0)), reverse=True)
+ promoted = [
+ _copy_hit_with_identifier_boost(hit, score=score, reasons=reasons, matched_tokens=matched_tokens)
+ for score, hit, matched_tokens, reasons in scored[:2]
+ ]
+ if not promoted:
+ return {
+ "enabled": True,
+ "hits": list(final_hits),
+ "promoted_hits": [],
+ "metadata": {
+ "identifier_protected_enabled": True,
+ "identifier_query_tokens": identifier_tokens,
+ "identifier_request": bool(identifier_request),
+ "identifier_promoted_count": 0,
+ },
+ }
+
+ promoted_keys = {hit.memory_id or f"{_hit_event_id(hit)}::{hit.slot_key}::{hit.value[:80]}" for hit in promoted}
+ merged = list(promoted)
+ for hit in final_hits:
+ key = hit.memory_id or f"{_hit_event_id(hit)}::{hit.slot_key}::{hit.value[:80]}"
+ if key in promoted_keys:
+ continue
+ merged.append(hit)
+ if len(merged) >= max(1, int(top_k)):
+ break
+ return {
+ "enabled": True,
+ "hits": merged[: max(1, int(top_k))],
+ "promoted_hits": promoted,
+ "metadata": {
+ "identifier_protected_enabled": True,
+ "identifier_query_tokens": identifier_tokens,
+ "identifier_request": bool(identifier_request),
+ "identifier_promoted_count": len(promoted),
+ "identifier_promoted_ids": [hit.memory_id for hit in promoted],
+ },
+ }
+
+
+def _trim_prompt_text(value: Any, *, max_chars: int = 220) -> str:
+ text = _clean_text(value)
+ if len(text) <= max_chars:
+ return text
+ return f"{text[: max(0, max_chars - 3)].rstrip()}..."
+
+
+def _prompt_hit_payload(hit: MemoryHit) -> Dict[str, Any]:
+ return {
+ "memory_id": hit.memory_id,
+ "slot_key": hit.slot_key,
+ "category": hit.category,
+ "value": _trim_prompt_text(hit.value),
+ "relation": hit.relation,
+ "anchors": [_trim_prompt_text(anchor, max_chars=80) for anchor in list(hit.anchors)[:4]],
+ "score": round(float(hit.score), 6),
+ "state": hit.state,
+ "turn_index": int(hit.turn_index),
+ "source_kind": hit.source_kind,
+ }
+
+
+def _prompt_record_payload(record: SessionMemoryRecordV2) -> Dict[str, Any]:
+ metadata = dict(record.metadata or {})
+ return {
+ "slot_key": record.slot_key,
+ "category": record.category,
+ "value": _trim_prompt_text(record.value),
+ "state": record.state,
+ "turn_index": int(record.turn_index),
+ "anchors": [_trim_prompt_text(anchor, max_chars=80) for anchor in list(record.anchor_concepts)[:4]],
+ "memory_role": _clean_text(metadata.get("memory_role", "")),
+ "authority": _clean_text(metadata.get("authority", "")),
+ }
+
+
+def _graph_prompt_state_summary(graph: SessionMemoryGraphV2, retrieval: MemoryRetrieval) -> Dict[str, Any]:
+ active_slots: List[Dict[str, Any]] = []
+ for slot_key, record_id in list(graph.slot_heads.items())[:_GRAPH_PROMPT_MAX_ACTIVE_SLOTS]:
+ record = graph.records_by_id.get(record_id)
+ if record is None:
+ continue
+ active_slots.append(_prompt_record_payload(record))
+ top_hits = [_prompt_hit_payload(hit) for hit in list(retrieval.hits)[:_GRAPH_PROMPT_MAX_HITS]]
+ relation_preview = [
+ {
+ "from": _trim_prompt_text(item.get("from", ""), max_chars=72),
+ "to": _trim_prompt_text(item.get("to", ""), max_chars=72),
+ "relation": _clean_text(item.get("relation", "")),
+ }
+ for item in list(retrieval.relations)[:_GRAPH_PROMPT_MAX_RELATIONS]
+ ]
+ summary = {
+ "records": len(graph.records_by_id),
+ "active_slots": len(graph.slot_heads),
+ "turn_index": int(graph.turn_index),
+ "noise_turn_count": int(graph.noise_turn_count),
+ "answer_support_events": len(graph.answer_support_log),
+ "top_hits": top_hits,
+ "active_slot_records": active_slots,
+ "relation_preview": relation_preview,
+ "context_truncated": False,
+ "truncation_reason": "",
+ }
+ truncated = False
+ truncation_reason = ""
+ while len(json.dumps(summary, ensure_ascii=False)) > _GRAPH_PROMPT_MAX_CHARS:
+ if len(summary["top_hits"]) > 1:
+ summary["top_hits"] = summary["top_hits"][:-1]
+ truncated = True
+ truncation_reason = "trimmed_top_hits"
+ continue
+ if len(summary["active_slot_records"]) > 1:
+ summary["active_slot_records"] = summary["active_slot_records"][:-1]
+ truncated = True
+ truncation_reason = "trimmed_active_slots"
+ continue
+ if len(summary["relation_preview"]) > 2:
+ summary["relation_preview"] = summary["relation_preview"][:-1]
+ truncated = True
+ truncation_reason = "trimmed_relations"
+ continue
+ break
+ summary["context_truncated"] = truncated
+ summary["truncation_reason"] = truncation_reason
+ return summary
+
+
+def _state_stats(*, storage_bytes: int, retrieval_context_tokens: int, total_state_tokens: int, **extra: Any) -> Dict[str, Any]:
+ return {
+ **extra,
+ "storage_bytes": int(storage_bytes),
+ "context_token_estimate": int(retrieval_context_tokens),
+ "retrieval_context_token_estimate": int(retrieval_context_tokens),
+ "total_state_token_estimate": int(total_state_tokens),
+ }
+
+
+def _relation_hit(hit: MemoryHit, *, weight_bias: float = 0.0) -> Dict[str, Any]:
+ if not hit.anchors:
+ return {}
+ anchor = hit.anchors[0]
+ if not anchor or anchor == hit.value:
+ return {}
+ return {
+ "from": anchor,
+ "to": hit.value,
+ "relation": hit.relation,
+ "weight": round(max(0.25, min(0.98, 0.42 + hit.score * 0.4 + weight_bias)), 6),
+ "source_kind": hit.source_kind,
+ "memory_id": hit.memory_id,
+ }
+
+
+def _raw_hit_to_memory_hit(payload: Dict[str, Any]) -> MemoryHit:
+ metadata = dict(payload.get("metadata", {}) or {})
+ if payload.get("supersedes"):
+ metadata["supersedes"] = list(payload.get("supersedes", []) or [])
+ slot_key = stable_slot_key(
+ category=str(payload.get("category", "")),
+ value=str(payload.get("value", "")),
+ anchors=[str(anchor) for anchor in payload.get("anchor_concepts", payload.get("anchors", [])) or [] if _clean_text(anchor)],
+ slot_key=str(payload.get("slot_key", metadata.get("slot", ""))),
+ relation=str(payload.get("relation", "related_to")),
+ metadata=metadata,
+ )
+ return MemoryHit(
+ memory_id=str(payload.get("memory_id", "")),
+ category=str(payload.get("category", "")),
+ value=str(payload.get("value", "")),
+ relation=str(payload.get("relation", "related_to")),
+ anchors=[str(anchor) for anchor in payload.get("anchor_concepts", payload.get("anchors", [])) or [] if _clean_text(anchor)],
+ score=float(payload.get("score", payload.get("relevance", 0.0)) or 0.0),
+ source_kind=str(payload.get("source_kind", "memory")),
+ slot_key=slot_key,
+ state=str(payload.get("state", payload.get("metadata", {}).get("state", "active")) or "active"),
+ turn_index=int(payload.get("turn_index", 0) or 0),
+ metadata=metadata,
+ )
+
+
+def _restore_hit_scores(hits: List[MemoryHit], scored_lookup: Dict[str, MemoryHit]) -> List[MemoryHit]:
+ restored: List[MemoryHit] = []
+ for hit in hits:
+ scored = scored_lookup.get(hit.memory_id)
+ if scored:
+ hit.score = max(float(hit.score), float(scored.score))
+ if not hit.anchors and scored.anchors:
+ hit.anchors = list(scored.anchors)
+ restored.append(hit)
+ return restored
+
+
+def _current_subject_query(query: str) -> bool:
+ lowered = _normalize(query)
+ return bool(
+ _public_query_subject(query)
+ and any(marker in lowered for marker in ("right now", "current", "currently", "active", "now", "当前", "现在"))
+ )
+
+
+def _record_subject_signatures(record: SessionMemoryRecordV2) -> set[str]:
+ metadata = dict(record.metadata or {})
+ signatures = {
+ _normalize(metadata.get("subject_signature", "")).replace("-", "_"),
+ _public_subject_signature(metadata.get("subject", "")),
+ }
+ canonical_slot_key = _clean_text(metadata.get("canonical_slot_key", "") or record.slot_key)
+ if ".subject." in canonical_slot_key:
+ signatures.add(_public_subject_signature(canonical_slot_key.split(".subject.", 1)[-1]))
+ if ".subject." in record.slot_key:
+ signatures.add(_public_subject_signature(record.slot_key.split(".subject.", 1)[-1]))
+ signatures.discard("")
+ return signatures
+
+
+def _current_subject_protected_hits(
+ *,
+ query: str,
+ graph: SessionMemoryGraphV2,
+ final_hits: Sequence[MemoryHit],
+ top_k: int,
+) -> Dict[str, Any]:
+ if not _current_subject_query(query):
+ return {
+ "hits": list(final_hits),
+ "metadata": {"current_subject_resolver_enabled": False},
+ }
+ subject = _public_query_subject(query)
+ subject_signature = _public_subject_signature(subject)
+ if not subject_signature:
+ return {
+ "hits": list(final_hits),
+ "metadata": {
+ "current_subject_resolver_enabled": True,
+ "current_subject_resolver_reason": "no_subject_signature",
+ },
+ }
+ def _candidate_record(record: SessionMemoryRecordV2 | None) -> bool:
+ return bool(
+ record is not None
+ and _clean_text(record.source_kind).startswith("public_dialog")
+ and _normalize(record.category) != "question"
+ and subject_signature in _record_subject_signatures(record)
+ )
+
+ slot_head_candidates = [
+ record
+ for slot_key, memory_id in graph.slot_heads.items()
+ for record in [graph.records_by_id.get(memory_id)]
+ if _candidate_record(record)
+ and subject_signature in _record_subject_signatures(record)
+ ]
+ candidates = slot_head_candidates or [
+ record
+ for record in graph.records_by_id.values()
+ if record.state == "active" and _candidate_record(record)
+ ]
+ candidates.sort(
+ key=lambda record: (
+ int(
+ _normalize((record.metadata or {}).get("target_status", "")) == "current"
+ or _normalize(record.relation) == "current_subject_value"
+ ),
+ int(record.turn_index),
+ float(record.confidence),
+ float(record.salience),
+ ),
+ reverse=True,
+ )
+ promoted: List[MemoryHit] = []
+ for index, record in enumerate(candidates[: max(1, min(2, int(top_k or 1)))], start=1):
+ metadata = dict(record.metadata or {})
+ metadata.update(
+ {
+ "current_subject_resolver": True,
+ "current_subject_resolver_rank": index,
+ "current_subject_query_subject": subject,
+ "current_subject_query_signature": subject_signature,
+ "public_subject_match": True,
+ "public_subject_overlap": 1.0,
+ }
+ )
+ promoted.append(
+ MemoryHit(
+ memory_id=record.memory_id,
+ category=record.category,
+ value=record.value,
+ relation=record.relation,
+ anchors=list(record.anchor_concepts),
+ score=max(float(record.confidence), float(record.salience), 1.75),
+ source_kind=record.source_kind,
+ slot_key=record.slot_key,
+ state=record.state,
+ turn_index=int(record.turn_index),
+ metadata=metadata,
+ )
+ )
+ if not promoted:
+ return {
+ "hits": list(final_hits),
+ "metadata": {
+ "current_subject_resolver_enabled": True,
+ "current_subject_resolver_reason": "no_active_subject_head",
+ "current_subject_query_subject": subject,
+ "current_subject_query_signature": subject_signature,
+ },
+ }
+ promoted_ids = {hit.memory_id for hit in promoted}
+ merged_tail: List[MemoryHit] = []
+ for hit in final_hits:
+ if hit.memory_id in promoted_ids:
+ continue
+ metadata = dict(hit.metadata or {})
+ hit_state = _normalize(hit.state)
+ same_subject = subject_signature in {
+ _normalize(metadata.get("subject_signature", "")).replace("-", "_"),
+ _public_subject_signature(metadata.get("subject", "")),
+ _public_subject_signature(hit.slot_key.split(".subject.", 1)[-1]) if ".subject." in hit.slot_key else "",
+ _public_subject_signature(_clean_text(metadata.get("canonical_slot_key", "")).split(".subject.", 1)[-1])
+ if ".subject." in _clean_text(metadata.get("canonical_slot_key", ""))
+ else "",
+ }
+ if same_subject and hit_state in {"superseded", "evidence", "historical", "stale", "false"}:
+ continue
+ merged_tail.append(hit)
+ merged = [*promoted, *merged_tail]
+ limit = max(int(top_k or 1), len(promoted))
+ return {
+ "hits": merged[:limit],
+ "metadata": {
+ "current_subject_resolver_enabled": True,
+ "current_subject_resolver_reason": "promoted_active_subject_head",
+ "current_subject_query_subject": subject,
+ "current_subject_query_signature": subject_signature,
+ "current_subject_promoted_memory_ids": [hit.memory_id for hit in promoted],
+ },
+ }
+
+
+def _depth_chain_protected_hits(
+ *,
+ query: str,
+ graph: SessionMemoryGraphV2,
+ final_hits: Sequence[MemoryHit],
+ top_k: int,
+) -> Dict[str, Any]:
+ seed_memory_ids = [hit.memory_id for hit in final_hits if hit.memory_id]
+ chain = graph.depth_chain_for_query(
+ query,
+ seed_memory_ids=seed_memory_ids,
+ top_k=max(3, min(8, int(top_k or 1))),
+ )
+ if not chain.get("enabled") or not chain.get("nodes"):
+ return {
+ "hits": list(final_hits),
+ "metadata": {
+ "memory_chain_enabled": bool(chain.get("enabled", False)),
+ "memory_chain_reason": _clean_text(chain.get("reason", "")),
+ "memory_chain_node_count": 0,
+ "memory_chain_edge_count": 0,
+ "memory_chain": chain,
+ },
+ }
+ seen = {hit.memory_id for hit in final_hits if hit.memory_id}
+ chain_hits: List[MemoryHit] = []
+ for rank, node in enumerate(list(chain.get("nodes", []) or []), start=1):
+ if not isinstance(node, Mapping):
+ continue
+ memory_id = _clean_text(node.get("memory_id", ""))
+ if not memory_id or memory_id in seen:
+ continue
+ payload = dict(node)
+ metadata = dict(payload.get("metadata", {}) or {})
+ metadata.update(
+ {
+ "memory_chain_protected": True,
+ "memory_chain_rank": int(rank),
+ "memory_chain_subject_signature": _clean_text(chain.get("subject_signature", "")),
+ "memory_chain_depth_layer": _clean_text(metadata.get("depth_layer", "")) or "core_view",
+ }
+ )
+ payload["metadata"] = metadata
+ payload["score"] = max(float(payload.get("score", 0.0) or 0.0), 0.62 - (rank * 0.01))
+ chain_hits.append(_raw_hit_to_memory_hit(payload))
+ seen.add(memory_id)
+ limit = max(int(top_k or 1), min(12, int(top_k or 1) + max(0, len(chain_hits))))
+ merged = [*list(final_hits), *chain_hits]
+ return {
+ "hits": merged[:limit],
+ "metadata": {
+ "memory_chain_enabled": True,
+ "memory_chain_reason": _clean_text(chain.get("reason", "")),
+ "memory_chain_subject_signature": _clean_text(chain.get("subject_signature", "")),
+ "memory_chain_node_count": int(chain.get("node_count", 0) or 0),
+ "memory_chain_edge_count": int(chain.get("edge_count", 0) or 0),
+ "memory_chain_depth_layers": list(chain.get("depth_layers", []) or []),
+ "memory_chain": chain,
+ },
+ }
+
+
+def _is_public_dialog_hit(hit: MemoryHit) -> bool:
+ return _clean_text(hit.source_kind).startswith("public_dialog")
+
+
+def _normalized_runtime_signature(prefix: str, value: str) -> str:
+ normalized = _normalize(value)
+ if not normalized:
+ return ""
+ return f"{prefix}{normalized.replace('|', '_').replace(':', '_')}"
+
+
+def _runtime_event_key(hit: MemoryHit) -> str:
+ metadata = dict(hit.metadata or {})
+ explicit = _clean_text(metadata.get("event_id", ""))
+ if explicit:
+ return explicit
+ dia_id = _clean_text(metadata.get("dia_id", ""))
+ if dia_id:
+ return f"event::{dia_id}"
+ if not _is_public_dialog_hit(hit):
+ state_signature = _clean_text(metadata.get("state_signature", ""))
+ if state_signature:
+ return _normalized_runtime_signature("event::state::", state_signature)
+ memory_signature = _clean_text(metadata.get("memory_signature", ""))
+ if memory_signature:
+ return _normalized_runtime_signature("event::memory::", memory_signature)
+ slot_root = _public_slot_root(_clean_text(hit.slot_key))
+ if slot_root:
+ return slot_root
+ return _clean_text(hit.memory_id)
+
+
+def _runtime_event_turn_index_from_id(event_id: str) -> int:
+ text = _clean_text(event_id)
+ if not text:
+ return 0
+ match = re.search(r"(?::|_)(\d+)$", text)
+ if match:
+ return int(match.group(1))
+ matches = re.findall(r"\d+", text)
+ return int(matches[-1]) if matches else 0
+
+
+def _representative_event_hit(group_hits: Sequence[MemoryHit], *, query: str = "") -> MemoryHit | None:
+ semantic_source_kinds = {
+ "public_dialog_fact",
+ "public_dialog_preference",
+ "public_dialog_goal",
+ "public_dialog_constraint",
+ "public_dialog_status",
+ "public_dialog_profile",
+ "replacement_memory",
+ "session_memory",
+ }
+ query_tokens = set(_path_utility_tokens(query))
+
+ def rank(hit: MemoryHit) -> tuple[bool, float, bool, bool, bool, bool, float]:
+ metadata = dict(hit.metadata or {})
+ source_kind = _clean_text(hit.source_kind)
+ text_parts: List[Any] = [hit.value, hit.slot_key, hit.category]
+ if source_kind != "public_dialog_turn":
+ text_parts.extend([metadata.get("source_turn_text", ""), metadata.get("raw_text", "")])
+ text = " ".join(
+ _clean_text(item)
+ for item in text_parts
+ if _clean_text(item)
+ )
+ hit_tokens = set(_path_utility_tokens(text))
+ overlap = len(query_tokens & hit_tokens) if query_tokens else 0
+ has_number = bool(re.search(r"\b\d+\b", text))
+ is_semantic = source_kind in semantic_source_kinds or (
+ source_kind != "public_dialog_turn" and bool(_clean_text(metadata.get("memory_writer_role", "")))
+ )
+ direct_semantic_answer = bool(is_semantic and has_number and overlap >= 2)
+ query_score = float(overlap) + (0.75 if has_number and overlap else 0.0)
+ return (
+ direct_semantic_answer,
+ query_score,
+ is_semantic,
+ source_kind == "public_dialog_event",
+ source_kind == "public_dialog_turn",
+ source_kind in {"replacement_memory", "session_memory"},
+ float(hit.score),
+ )
+
+ ordered = sorted(
+ list(group_hits),
+ key=rank,
+ reverse=True,
+ )
+ return ordered[0] if ordered else None
+
+
+def _event_record_hits_from_graph(graph: SessionMemoryGraphV2, event_id: str) -> List[MemoryHit]:
+ normalized_event_id = _clean_text(event_id)
+ if not normalized_event_id:
+ return []
+ hits: List[MemoryHit] = []
+ for record in graph.records_by_id.values():
+ metadata = dict(record.metadata or {})
+ if _clean_text(metadata.get("event_id", "")) != normalized_event_id:
+ continue
+ state = _normalize(record.state)
+ if state not in {"active", "parallel_active", "evidence"}:
+ continue
+ hits.append(
+ MemoryHit(
+ memory_id=record.memory_id,
+ category=record.category,
+ value=record.value,
+ relation=record.relation,
+ anchors=list(record.anchor_concepts),
+ score=max(float(record.confidence), float(record.salience), 0.01),
+ source_kind=record.source_kind,
+ slot_key=record.slot_key,
+ state=record.state,
+ turn_index=int(record.turn_index),
+ metadata=metadata,
+ )
+ )
+ return hits
+
+def _hit_matches_path_support(path_type: str, hit: MemoryHit) -> bool:
+ metadata = dict(hit.metadata or {})
+ source_kind = _clean_text(hit.source_kind)
+ category = _clean_text(hit.category)
+ relation = _clean_text(hit.relation)
+ if path_type == "speaker_event_time":
+ return bool(
+ source_kind == "public_dialog_time"
+ or _clean_text(metadata.get("resolved_time_value", ""))
+ or _clean_text(metadata.get("resolved_date", ""))
+ or _clean_text(metadata.get("time_value", ""))
+ or _clean_text(metadata.get("time_display_value", ""))
+ or _clean_text(metadata.get("time_granularity", ""))
+ or relation == "event_date"
+ or category in {"time", "event_time"}
+ )
+ if path_type == "speaker_event_profile":
+ semantic_slot = _clean_text(metadata.get("semantic_slot", "")) or _clean_text(metadata.get("profile_type", ""))
+ return bool(
+ source_kind == "public_dialog_profile"
+ or semantic_slot in {"identity", "research_topic", "education", "occupation", "profile"}
+ or _clean_text(metadata.get("profile_value", ""))
+ or category == "profile"
+ )
+ if path_type == "speaker_event_status":
+ return bool(
+ _clean_text(metadata.get("target_status", ""))
+ or category in {"status", "stage_state"}
+ or relation == "status_of"
+ )
+ if path_type == "speaker_event_source_turn":
+ return bool(
+ source_kind in {"public_dialog_turn", "public_dialog_text", "public_dialog_auxiliary_evidence"}
+ or _clean_text(metadata.get("raw_text", ""))
+ or _clean_text(metadata.get("origin_query", ""))
+ or _clean_text(metadata.get("source_turn_text", ""))
+ or not _is_public_dialog_hit(hit)
+ )
+ return False
+
+
+def _support_hit_for_path(path_type: str, group_hits: Sequence[MemoryHit]) -> MemoryHit | None:
+ matching_hits = [hit for hit in group_hits if _hit_matches_path_support(path_type, hit)]
+ if matching_hits:
+ if _clean_text(path_type) == "speaker_event_source_turn":
+ def source_turn_rank(item: MemoryHit) -> tuple[int, int, int, int, float, int]:
+ metadata = dict(item.metadata or {})
+ source_kind = _clean_text(item.source_kind)
+ relation = _clean_text(item.relation)
+ memory_id = _clean_text(item.memory_id)
+ value = _clean_text(item.value)
+ source_text = _clean_text(
+ metadata.get("source_turn_text", "")
+ or metadata.get("raw_text", "")
+ or metadata.get("origin_query", "")
+ )
+ is_source_turn = int(source_kind == "public_dialog_turn" or relation == "source_turn_grounding")
+ is_source_text = int(source_kind in {"public_dialog_text", "public_dialog_auxiliary_evidence"})
+ has_source_text = int(bool(source_text) and len(source_text) >= 80)
+ is_unit_attachment = int(
+ source_kind == "event_action_frame_attachment"
+ or ".unit." in memory_id
+ or ".profile." in memory_id
+ )
+ return (
+ is_source_turn,
+ is_source_text,
+ has_source_text,
+ -is_unit_attachment,
+ float(item.score),
+ len(value),
+ )
+
+ matching_hits.sort(key=source_turn_rank, reverse=True)
+ else:
+ matching_hits.sort(key=lambda item: (float(item.score), int(item.turn_index)), reverse=True)
+ return matching_hits[0]
+ representative = _representative_event_hit(group_hits)
+ return representative
+
+
+def _path_support_node_id(path: Dict[str, Any]) -> str:
+ node_ids = list(path.get("node_ids", []) or [])
+ if len(node_ids) < 3:
+ return ""
+ return _clean_text(node_ids[2])
+
+
+def _event_ids_from_hits(hits: Sequence[MemoryHit]) -> List[str]:
+ return _dedupe(
+ _clean_text(dict(hit.metadata or {}).get("event_id", ""))
+ for hit in hits
+ if _clean_text(dict(hit.metadata or {}).get("event_id", ""))
+ )
+
+
+def _dia_ids_from_hits(hits: Sequence[MemoryHit]) -> List[str]:
+ return _dedupe(
+ _clean_text(dict(hit.metadata or {}).get("dia_id", ""))
+ for hit in hits
+ if _clean_text(dict(hit.metadata or {}).get("dia_id", ""))
+ )
+
+
+def _low_information_support_hit(hit: MemoryHit) -> bool:
+ text = _clean_text(hit.value)
+ if not text:
+ return True
+ alpha_tokens = re.findall(r"[a-zA-Z][a-zA-Z_-]{2,}", text)
+ number_tokens = re.findall(r"\d+", text)
+ if len(alpha_tokens) <= 1 and number_tokens:
+ return True
+ if len(text) <= 24 and number_tokens and len(alpha_tokens) <= 2:
+ return True
+ source_kind = _clean_text(hit.source_kind)
+ memory_id = _clean_text(hit.memory_id)
+ if source_kind == "event_action_frame_attachment" and ".unit.numeric." in memory_id and len(alpha_tokens) <= 2:
+ return True
+ return False
+
+
+def _final_hit_role_priority(hit: MemoryHit) -> int:
+ metadata = dict(hit.metadata or {})
+ source_kind = _clean_text(hit.source_kind)
+ low_information = _low_information_support_hit(hit)
+ semantic_source_kinds = {
+ "public_dialog_fact",
+ "public_dialog_preference",
+ "public_dialog_goal",
+ "public_dialog_constraint",
+ "public_dialog_status",
+ "public_dialog_profile",
+ "public_dialog_profile_cluster",
+ "replacement_memory",
+ "session_memory",
+ }
+ if not low_information and (source_kind in semantic_source_kinds or (
+ source_kind != "public_dialog_turn" and bool(_clean_text(metadata.get("memory_writer_role", "")))
+ )):
+ return 0
+ if bool(metadata.get("profile_first_source_support")):
+ return 0
+ role = _clean_text(metadata.get("evidence_snippet_role", ""))
+ if role == "selected_path_support":
+ if low_information:
+ return 4
+ return 1
+ if role == "selected_event_representative":
+ return 2
+ if role == "selected_path_event":
+ return 3
+ return 4
+
+
+def _coverage_preserving_final_hits(
+ final_hits: Sequence[MemoryHit],
+ *,
+ selected_event_ids: Sequence[str],
+ top_k: int,
+) -> List[MemoryHit]:
+ """Keep selected-event coverage before filling the remaining prompt budget.
+
+ Learned selection can emit both path-support and event-representative snippets
+ for the same event. A pure score sort can then drop another selected event at
+ the top-k boundary, which hides recall/rerank successes from the answer head.
+ """
+
+ hits = list(final_hits)
+ selected_order = [
+ _clean_text(event_id)
+ for event_id in selected_event_ids
+ if _clean_text(event_id)
+ ]
+ try:
+ extra_selected_budget = max(0, _int_env("TMCRA_SELECTED_EVENT_COVERAGE_EXTRA_K", 4))
+ except Exception:
+ extra_selected_budget = 4
+ budget = max(1, int(top_k or 1))
+ if selected_order and extra_selected_budget:
+ budget = max(budget, min(len(selected_order), int(top_k or 1) + extra_selected_budget))
+ selected_order = selected_order[:budget]
+ if not selected_order:
+ return sorted(hits, key=lambda item: float(item.score), reverse=True)[:budget]
+ hits_by_event: Dict[str, List[MemoryHit]] = {}
+ for hit in hits:
+ event_id = _clean_text(dict(hit.metadata or {}).get("event_id", ""))
+ if event_id:
+ hits_by_event.setdefault(event_id, []).append(hit)
+ selected: List[MemoryHit] = []
+ used_memory_ids = set()
+ for event_id in selected_order:
+ candidates = [hit for hit in hits_by_event.get(event_id, []) if hit.memory_id not in used_memory_ids]
+ if not candidates:
+ continue
+ candidates.sort(key=lambda item: (_final_hit_role_priority(item), -float(item.score), item.memory_id))
+ chosen = candidates[0]
+ selected.append(chosen)
+ used_memory_ids.add(chosen.memory_id)
+ if len(selected) >= budget:
+ return selected
+ remaining = [
+ hit
+ for hit in hits
+ if hit.memory_id not in used_memory_ids
+ ]
+ remaining.sort(key=lambda item: (-float(item.score), _final_hit_role_priority(item), item.memory_id))
+ for hit in remaining:
+ selected.append(hit)
+ if len(selected) >= budget:
+ break
+ return selected[:budget]
+
+
+def _add_multi_evidence_recall_coverage(
+ selected_event_ids: Sequence[str],
+ recall_event_ids: Sequence[str],
+ *,
+ focused_answer_type: str,
+ top_k: int,
+) -> List[str]:
+ focused = _clean_text(focused_answer_type)
+ if focused not in {"multi_evidence", "multi", "aggregation", "chain"}:
+ return _dedupe(selected_event_ids)
+ try:
+ coverage_k = max(0, _int_env("TMCRA_MULTI_EVIDENCE_RECALL_COVERAGE_K", 8))
+ except Exception:
+ coverage_k = 8
+ if coverage_k <= 0:
+ return _dedupe(selected_event_ids)
+ return _dedupe(
+ [
+ *[_clean_text(event_id) for event_id in selected_event_ids if _clean_text(event_id)],
+ *[_clean_text(event_id) for event_id in list(recall_event_ids or [])[:coverage_k] if _clean_text(event_id)],
+ ],
+ max_items=max(len(list(selected_event_ids or [])), int(top_k or 1) + coverage_k),
+ )
+
+
+def _dominant_answer_type(question_analysis: Dict[str, Any], answer_type_scores: Dict[str, Any]) -> str:
+ normalized_scores = {
+ _clean_text(answer_type): float(value or 0.0)
+ for answer_type, value in dict(answer_type_scores or {}).items()
+ if _clean_text(answer_type)
+ }
+ if bool(question_analysis.get("is_temporal", False)) and normalized_scores.get("time", 0.0) >= (normalized_scores.get("abstain", 0.0) - 0.05):
+ return "time"
+ if not bool(question_analysis.get("is_temporal", False)) and normalized_scores:
+ non_time_scores = {
+ answer_type: score
+ for answer_type, score in normalized_scores.items()
+ if answer_type not in {"time", "abstain"}
+ }
+ if non_time_scores:
+ return max(non_time_scores.items(), key=lambda item: (float(item[1]), item[0]))[0]
+ if not normalized_scores:
+ return "time" if bool(question_analysis.get("is_temporal", False)) else "event_text"
+ return max(normalized_scores.items(), key=lambda item: (float(item[1]), item[0]))[0]
+
+
+def _answer_type_preferred_path_types(question_analysis: Dict[str, Any], answer_type_scores: Dict[str, Any]) -> List[str]:
+ dominant_answer_type = _dominant_answer_type(question_analysis, answer_type_scores)
+ if dominant_answer_type == "time" or bool(question_analysis.get("is_temporal", False)):
+ return ["speaker_event_time", "speaker_event_source_turn", "speaker_event_status", "speaker_event_profile"]
+ if dominant_answer_type == "profile":
+ return ["speaker_event_profile", "speaker_event_source_turn", "speaker_event_status", "speaker_event_time"]
+ if dominant_answer_type == "multi_evidence":
+ return ["speaker_event_source_turn", "speaker_event_time", "speaker_event_profile", "speaker_event_status"]
+ return ["speaker_event_source_turn", "speaker_event_time", "speaker_event_profile", "speaker_event_status"]
+
+
+def _reconciled_focused_answer_type(
+ question_analysis: Dict[str, Any],
+ answer_type_scores: Dict[str, Any],
+ model_answer_type: str,
+) -> str:
+ model_type = _clean_text(model_answer_type)
+ dominant_type = _dominant_answer_type(question_analysis, answer_type_scores)
+ if bool(question_analysis.get("is_temporal", False)) and model_type not in {"", "time", "abstain"}:
+ return "time"
+ if model_type == "time" and not bool(question_analysis.get("is_temporal", False)):
+ return dominant_type if dominant_type != "time" else "event_text"
+ return model_type or dominant_type
+
+
+def _path_type_is_focus_compatible(path_type: str, *, focused_answer_type: str, question_analysis: Dict[str, Any]) -> bool:
+ normalized_path_type = _clean_text(path_type)
+ normalized_answer_type = _clean_text(focused_answer_type)
+ if normalized_answer_type == "time" or bool(question_analysis.get("is_temporal", False)):
+ return normalized_path_type in {"speaker_event_time", "speaker_event_source_turn"}
+ if normalized_answer_type == "profile":
+ return normalized_path_type in {"speaker_event_profile", "speaker_event_source_turn"}
+ return True
+
+
+def _rank_focus_compatible_path_ids(
+ *,
+ runtime_paths: Mapping[str, Dict[str, Any]],
+ selected_event_ids: Sequence[str],
+ path_scores: Mapping[str, Any],
+ event_scores: Mapping[str, Any],
+ temporal_scores: Mapping[str, Any],
+ question_analysis: Dict[str, Any],
+ answer_type_scores: Dict[str, Any],
+ focused_answer_type: str,
+) -> List[str]:
+ selected_event_id_set = {_clean_text(event_id) for event_id in selected_event_ids if _clean_text(event_id)}
+ preferred_types = _answer_type_preferred_path_types(question_analysis, answer_type_scores)
+ if _clean_text(focused_answer_type) == "profile" and "speaker_event_profile" not in preferred_types:
+ preferred_types = ["speaker_event_profile", *preferred_types]
+ if (_clean_text(focused_answer_type) == "time" or bool(question_analysis.get("is_temporal", False))) and "speaker_event_time" not in preferred_types:
+ preferred_types = ["speaker_event_time", "speaker_event_source_turn", *preferred_types]
+ ranked: List[str] = []
+ seen = set()
+ for preferred_type in preferred_types:
+ candidates: List[tuple[str, float]] = []
+ for path_id, path in runtime_paths.items():
+ if _clean_text(path.get("type", "")) != preferred_type:
+ continue
+ event_id = _clean_text(path.get("event_id", ""))
+ if selected_event_id_set and event_id not in selected_event_id_set:
+ continue
+ if not _path_type_is_focus_compatible(preferred_type, focused_answer_type=focused_answer_type, question_analysis=question_analysis):
+ continue
+ support_node_id = _path_support_node_id(path)
+ score = (
+ float(event_scores.get(event_id, 0.0) or 0.0)
+ + (0.20 * float(path_scores.get(path_id, 0.0) or 0.0))
+ + (0.15 * float(temporal_scores.get(support_node_id, 0.0) or 0.0))
+ )
+ candidates.append((path_id, score))
+ for path_id, _ in sorted(candidates, key=lambda item: (-float(item[1]), item[0])):
+ if path_id in seen:
+ continue
+ seen.add(path_id)
+ ranked.append(path_id)
+ if ranked and (_clean_text(focused_answer_type) == "time" or bool(question_analysis.get("is_temporal", False))):
+ if preferred_type in {"speaker_event_time", "speaker_event_source_turn"}:
+ break
+ return ranked
+
+
+def _repair_selected_paths_for_focus(
+ selected_path_ids: Sequence[str],
+ *,
+ runtime_paths: Mapping[str, Dict[str, Any]],
+ selected_event_ids: Sequence[str],
+ path_scores: Mapping[str, Any],
+ event_scores: Mapping[str, Any],
+ temporal_scores: Mapping[str, Any],
+ question_analysis: Dict[str, Any],
+ answer_type_scores: Dict[str, Any],
+ focused_answer_type: str,
+ limit: int,
+) -> tuple[List[str], bool, str]:
+ normalized_selected = [_clean_text(path_id) for path_id in selected_path_ids if _clean_text(path_id)]
+ if not normalized_selected:
+ return [], False, ""
+ incompatible = [
+ path_id
+ for path_id in normalized_selected
+ if not _path_type_is_focus_compatible(
+ _clean_text(runtime_paths.get(path_id, {}).get("type", "")),
+ focused_answer_type=focused_answer_type,
+ question_analysis=question_analysis,
+ )
+ ]
+ if not incompatible:
+ return normalized_selected, False, ""
+ compatible_ranked = _rank_focus_compatible_path_ids(
+ runtime_paths=runtime_paths,
+ selected_event_ids=selected_event_ids,
+ path_scores=path_scores,
+ event_scores=event_scores,
+ temporal_scores=temporal_scores,
+ question_analysis=question_analysis,
+ answer_type_scores=answer_type_scores,
+ focused_answer_type=focused_answer_type,
+ )
+ if not compatible_ranked:
+ return normalized_selected, False, ""
+ repaired = _dedupe(
+ [
+ *[path_id for path_id in normalized_selected if path_id not in incompatible],
+ *compatible_ranked,
+ ]
+ )[: max(1, limit)]
+ if repaired == normalized_selected:
+ return normalized_selected, False, ""
+ return repaired, True, "replaced_focus_incompatible_model_paths"
+
+
+def _runtime_node_by_id(runtime_graph: Mapping[str, Any]) -> Dict[str, Dict[str, Any]]:
+ return {
+ _clean_text(node.get("id", "")): dict(node)
+ for node in list(runtime_graph.get("nodes", []) or [])
+ if _clean_text(node.get("id", ""))
+ }
+
+
+def _runtime_event_subject_signature(runtime_nodes: Mapping[str, Dict[str, Any]], event_id: str) -> str:
+ node = dict(runtime_nodes.get(_clean_text(event_id), {}) or {})
+ metadata = dict(node.get("metadata", {}) or {})
+ return _clean_text(node.get("subject_signature", "")) or _clean_text(metadata.get("subject_signature", ""))
+
+
+def _path_utility_candidate_text(
+ path: Mapping[str, Any],
+ *,
+ runtime_nodes: Mapping[str, Dict[str, Any]],
+ grouped_hits: Mapping[str, Sequence[MemoryHit]],
+) -> str:
+ event_id = _clean_text(path.get("event_id", ""))
+ support_node_id = _path_support_node_id(dict(path))
+ node_texts = [
+ _clean_text(dict(runtime_nodes.get(event_id, {}) or {}).get("text", "")),
+ _clean_text(dict(runtime_nodes.get(support_node_id, {}) or {}).get("text", "")),
+ ]
+ path_type = _clean_text(path.get("type", ""))
+ support_hit = _support_hit_for_path(path_type, grouped_hits.get(event_id, []))
+ event_hit = _representative_event_hit(grouped_hits.get(event_id, []))
+ hit_texts = [
+ _clean_text(support_hit.value if support_hit is not None else ""),
+ _clean_text(event_hit.value if event_hit is not None else ""),
+ _runtime_source_turn_text(support_hit or event_hit, speaker=""),
+ ]
+ return " ".join(_dedupe([*node_texts, *hit_texts], max_items=6))
+
+
+def _path_utility_gate(
+ candidate_path_ids: Sequence[str],
+ *,
+ query: str,
+ runtime_graph: Mapping[str, Any],
+ runtime_paths: Mapping[str, Dict[str, Any]],
+ grouped_hits: Mapping[str, Sequence[MemoryHit]],
+ selected_path_ids: Sequence[str],
+ selected_event_ids_from_model: Sequence[str],
+ path_scores: Mapping[str, Any],
+ path_tunnel_support_scores: Mapping[str, Any],
+ question_analysis: Dict[str, Any],
+ focused_answer_type: str,
+ score_threshold: float,
+ limit: int,
+) -> Dict[str, Any]:
+ runtime_nodes = _runtime_node_by_id(runtime_graph)
+ query_tokens = set(_path_utility_tokens(query))
+ anchor_event_ids = _dedupe(
+ [
+ *[
+ _clean_text(runtime_paths.get(path_id, {}).get("event_id", ""))
+ for path_id in selected_path_ids
+ if _clean_text(runtime_paths.get(path_id, {}).get("event_id", ""))
+ ],
+ *[_clean_text(event_id) for event_id in selected_event_ids_from_model if _clean_text(event_id)],
+ ]
+ )
+ anchor_subject_signatures = {
+ signature
+ for signature in (
+ _runtime_event_subject_signature(runtime_nodes, event_id)
+ for event_id in anchor_event_ids
+ )
+ if signature
+ }
+ direct_path_ids: List[str] = []
+ contrast_path_ids: List[str] = []
+ latent_path_ids: List[str] = []
+ noise_path_ids: List[str] = []
+ utility_scores: Dict[str, float] = {}
+ utility_roles: Dict[str, str] = {}
+ utility_reasons: Dict[str, str] = {}
+ utility_overlap_tokens: Dict[str, List[str]] = {}
+ for path_id in _dedupe(candidate_path_ids):
+ path = dict(runtime_paths.get(path_id, {}) or {})
+ if not path:
+ continue
+ path_type = _clean_text(path.get("type", ""))
+ event_id = _clean_text(path.get("event_id", ""))
+ candidate_text = _path_utility_candidate_text(
+ path,
+ runtime_nodes=runtime_nodes,
+ grouped_hits=grouped_hits,
+ )
+ candidate_tokens = set(_path_utility_tokens(candidate_text))
+ overlap_tokens = sorted(query_tokens & candidate_tokens)
+ overlap_ratio = float(len(overlap_tokens)) / float(max(1, min(len(query_tokens), len(candidate_tokens))))
+ support_score = float(path_tunnel_support_scores.get(path_id, 0.0) or 0.0)
+ decision_score = float(path_scores.get(path_id, support_score) or 0.0)
+ path_score = max(support_score, decision_score)
+ subject_signature = _runtime_event_subject_signature(runtime_nodes, event_id)
+ same_subject_chain = bool(subject_signature and subject_signature in anchor_subject_signatures)
+ focus_compatible = _path_type_is_focus_compatible(
+ path_type,
+ focused_answer_type=focused_answer_type,
+ question_analysis=question_analysis,
+ )
+ utility_score = path_score + (0.20 * overlap_ratio) + (0.04 if same_subject_chain else 0.0)
+ utility_scores[path_id] = round(float(utility_score), 6)
+ utility_overlap_tokens[path_id] = overlap_tokens[:12]
+ if not focus_compatible:
+ role = "drift_noise"
+ reason = "focus_incompatible"
+ elif overlap_ratio >= 0.18 and path_score >= score_threshold:
+ role = "direct_support"
+ reason = "query_overlap_and_tunnel_score"
+ elif same_subject_chain and path_score >= score_threshold:
+ role = "contrast_support"
+ reason = "same_subject_deep_chain"
+ elif path_score >= score_threshold:
+ role = "latent_context"
+ reason = "tunnel_score_without_current_turn_utility"
+ else:
+ role = "drift_noise"
+ reason = "below_utility_threshold"
+ utility_roles[path_id] = role
+ utility_reasons[path_id] = reason
+ if role == "direct_support":
+ direct_path_ids.append(path_id)
+ elif role == "contrast_support":
+ contrast_path_ids.append(path_id)
+ elif role == "latent_context":
+ latent_path_ids.append(path_id)
+ else:
+ noise_path_ids.append(path_id)
+ injected_path_ids = _dedupe([*direct_path_ids, *contrast_path_ids], max_items=max(0, int(limit)))
+ overflow_latent_path_ids = [
+ path_id
+ for path_id in [*direct_path_ids, *contrast_path_ids]
+ if path_id not in set(injected_path_ids)
+ ]
+ latent_path_ids = _dedupe([*latent_path_ids, *overflow_latent_path_ids])
+ return {
+ "enabled": True,
+ "candidate_path_ids": list(_dedupe(candidate_path_ids)),
+ "direct_support_path_ids": list(direct_path_ids),
+ "contrast_support_path_ids": list(contrast_path_ids),
+ "latent_context_path_ids": list(latent_path_ids),
+ "drift_noise_path_ids": list(noise_path_ids),
+ "injected_path_ids": list(injected_path_ids),
+ "roles": dict(utility_roles),
+ "reasons": dict(utility_reasons),
+ "scores": dict(utility_scores),
+ "overlap_tokens": dict(utility_overlap_tokens),
+ "anchor_event_ids": list(anchor_event_ids),
+ "anchor_subject_signatures": sorted(anchor_subject_signatures),
+ }
+
+
+def _calibrated_path_score(
+ *,
+ path: Dict[str, Any],
+ base_score: float,
+ temporal_scores: Dict[str, Any],
+ question_analysis: Dict[str, Any],
+ answer_type_scores: Dict[str, Any],
+) -> float:
+ normalized_path_type = _clean_text(path.get("type", ""))
+ support_node_id = _path_support_node_id(path)
+ temporal_score = float(temporal_scores.get(support_node_id, 0.0) or 0.0)
+ normalized_answer_scores = {
+ _clean_text(answer_type): float(value or 0.0)
+ for answer_type, value in dict(answer_type_scores or {}).items()
+ if _clean_text(answer_type)
+ }
+ dominant_answer_type = _dominant_answer_type(question_analysis, normalized_answer_scores)
+ calibrated = float(base_score)
+ if dominant_answer_type == "time" or bool(question_analysis.get("is_temporal", False)):
+ if normalized_path_type == "speaker_event_time":
+ calibrated += (0.30 * temporal_score) + (0.12 * normalized_answer_scores.get("time", 0.0))
+ elif normalized_path_type == "speaker_event_source_turn":
+ calibrated -= 0.08 + (0.04 * normalized_answer_scores.get("time", 0.0))
+ elif normalized_path_type == "speaker_event_status":
+ calibrated -= 0.12
+ elif normalized_path_type == "speaker_event_profile":
+ calibrated -= 0.22 + (0.06 * normalized_answer_scores.get("time", 0.0))
+ elif dominant_answer_type == "profile":
+ if normalized_path_type == "speaker_event_profile":
+ calibrated += 0.12 * normalized_answer_scores.get("profile", 0.0)
+ elif normalized_path_type != "speaker_event_source_turn":
+ calibrated -= 0.08
+ else:
+ if normalized_path_type == "speaker_event_source_turn":
+ calibrated += 0.08 * max(
+ normalized_answer_scores.get("event_text", 0.0),
+ normalized_answer_scores.get("multi_evidence", 0.0),
+ )
+ elif normalized_path_type == "speaker_event_profile":
+ calibrated -= 0.04
+ return calibrated
+
+
+def _group_metadata_value(
+ group_hits: Sequence[MemoryHit],
+ key: str,
+ *,
+ source_kinds: Sequence[str] = (),
+) -> str:
+ source_kind_set = {_clean_text(item) for item in list(source_kinds) if _clean_text(item)}
+ candidates = [
+ hit
+ for hit in group_hits
+ if not source_kind_set or _clean_text(hit.source_kind) in source_kind_set
+ ]
+ candidates.sort(key=lambda item: (float(item.score), int(item.turn_index)), reverse=True)
+ for hit in candidates:
+ metadata = dict(hit.metadata or {})
+ value = _clean_text(metadata.get(key, ""))
+ if value:
+ return value
+ return ""
+
+
+def _runtime_event_sequence_key(session_name: str, turn_index: int, event_id: str) -> tuple[Any, ...]:
+ normalized_session = _clean_text(session_name)
+ session_number_match = re.search(r"(\d+)$", normalized_session)
+ if session_number_match:
+ return (0, int(session_number_match.group(1)), int(turn_index), _clean_text(event_id))
+ return (1, normalized_session, int(turn_index), _clean_text(event_id))
+
+
+def _runtime_source_turn_text(hit: MemoryHit | None, *, speaker: str) -> str:
+ if hit is None:
+ return ""
+ metadata = dict(hit.metadata or {})
+ source_turn_text = _clean_text(metadata.get("source_turn_text", ""))
+ if source_turn_text:
+ return source_turn_text
+ raw_text = _clean_text(metadata.get("raw_text", ""))
+ if raw_text:
+ auxiliary_text = _clean_text(metadata.get("auxiliary_evidence_text", ""))
+ if auxiliary_text and auxiliary_text.lower() not in raw_text.lower():
+ return f"{raw_text}\nAuxiliary evidence: {auxiliary_text}"
+ return raw_text
+ origin_query = _clean_text(metadata.get("origin_query", ""))
+ if origin_query:
+ return origin_query
+ text = _clean_text(hit.value)
+ if _clean_text(hit.source_kind) == "public_dialog_turn":
+ text = re.sub(r"^\[[^\]]+\]\s*", "", text)
+ if _clean_text(speaker):
+ text = re.sub(rf"^{re.escape(_clean_text(speaker))}\s*:\s*", "", text, flags=re.IGNORECASE)
+ text = re.sub(r"^[A-Za-z][A-Za-z0-9_' -]{0,40}:\s*", "", text)
+ return text
+
+
+def _runtime_event_signature(
+ *,
+ group_hits: Sequence[MemoryHit],
+ representative: MemoryHit,
+ speaker: str,
+ semantic_slot: str,
+ source_turn_hit: MemoryHit | None,
+) -> str:
+ existing = _group_metadata_value(group_hits, "event_signature") or _clean_text(dict(representative.metadata or {}).get("event_signature", ""))
+ if existing:
+ return existing
+ event_phrase = _group_metadata_value(group_hits, "event_phrase") or _clean_text(dict(representative.metadata or {}).get("event_phrase", ""))
+ source_turn_text = _runtime_source_turn_text(source_turn_hit, speaker=speaker)
+ base_text = event_phrase or _clean_text(representative.value) or source_turn_text
+ if not base_text:
+ return ""
+ return compute_public_event_signature(
+ base_text,
+ speaker=_clean_text(speaker),
+ semantic_slot=_clean_text(semantic_slot),
+ ) or _clean_text(base_text)
+
+
+def _build_runtime_graph_from_hits(query: str, hits: Sequence[MemoryHit]) -> Dict[str, Any]:
+ nodes: List[Dict[str, Any]] = []
+ edges: List[Dict[str, Any]] = []
+ paths: List[Dict[str, Any]] = []
+ node_ids = set()
+ grouped_hits: Dict[str, List[MemoryHit]] = {}
+ ordered_events: List[tuple[Any, ...]] = []
+ event_typed_metadata_by_id: Dict[str, Dict[str, Any]] = {}
+ typed_tunnel_edges: List[Dict[str, Any]] = []
+
+ def add_node(node: Dict[str, Any]) -> None:
+ node_id = _clean_text(node.get("id", ""))
+ if not node_id or node_id in node_ids:
+ return
+ node_ids.add(node_id)
+ nodes.append(node)
+
+ for hit in hits:
+ event_id = _runtime_event_key(hit)
+ grouped_hits.setdefault(event_id, []).append(hit)
+
+ for event_id, group_hits in grouped_hits.items():
+ representative = _representative_event_hit(group_hits)
+ if representative is None:
+ continue
+ metadata = dict(representative.metadata or {})
+ event_time_hit = _support_hit_for_path("speaker_event_time", group_hits)
+ event_profile_hit = _support_hit_for_path("speaker_event_profile", group_hits)
+ source_turn_hit = _support_hit_for_path("speaker_event_source_turn", group_hits)
+ event_time_display_value = (
+ _group_metadata_value(group_hits, "time_display_value", source_kinds=("public_dialog_time",))
+ or _group_metadata_value(group_hits, "resolved_date")
+ or _group_metadata_value(group_hits, "time_display_value")
+ )
+ event_time_value = (
+ _group_metadata_value(group_hits, "resolved_time_value", source_kinds=("public_dialog_time",))
+ or _group_metadata_value(group_hits, "resolved_time_value")
+ or _group_metadata_value(group_hits, "time_value")
+ )
+ event_time_granularity = (
+ _group_metadata_value(group_hits, "time_granularity", source_kinds=("public_dialog_time",))
+ or _group_metadata_value(group_hits, "time_granularity")
+ )
+ event_profile_type = (
+ _group_metadata_value(group_hits, "profile_type", source_kinds=("public_dialog_profile",))
+ or _group_metadata_value(group_hits, "semantic_slot", source_kinds=("public_dialog_profile",))
+ or _group_metadata_value(group_hits, "profile_type")
+ or _group_metadata_value(group_hits, "semantic_slot")
+ )
+ profile_hit = _support_hit_for_path("speaker_event_profile", group_hits)
+ event_profile_value = (
+ _group_metadata_value(group_hits, "profile_value")
+ or _clean_text(profile_hit.value if profile_hit is not None else "")
+ or (_clean_text(representative.value) if event_profile_type else "")
+ )
+ event_target_status = _group_metadata_value(group_hits, "target_status")
+ depth_layer = (
+ _clean_text(metadata.get("depth_layer", ""))
+ or _group_metadata_value(group_hits, "depth_layer")
+ or _clean_text(metadata.get("memory_chain_depth_layer", ""))
+ or _group_metadata_value(group_hits, "memory_chain_depth_layer")
+ )
+ subject_signature = (
+ _clean_text(metadata.get("subject_signature", ""))
+ or _group_metadata_value(group_hits, "subject_signature")
+ or _clean_text(metadata.get("memory_chain_subject_signature", ""))
+ or _group_metadata_value(group_hits, "memory_chain_subject_signature")
+ )
+ session_name = (
+ _group_metadata_value(group_hits, "session_name")
+ or _group_metadata_value(group_hits, "session_key")
+ or _group_metadata_value(group_hits, "scope_id")
+ or "runtime_session"
+ )
+ speaker = (
+ _clean_text(metadata.get("speaker", ""))
+ or _group_metadata_value(group_hits, "speaker")
+ or _clean_text(metadata.get("subject_signature", ""))
+ or (_clean_text(representative.anchors[0]) if representative.anchors else "")
+ or "speaker"
+ )
+ event_turn_index = int(getattr(representative, "turn_index", 0) or 0)
+ semantic_slot = (
+ _group_metadata_value(group_hits, "semantic_slot")
+ or _clean_text(metadata.get("semantic_slot", ""))
+ or _clean_text(metadata.get("profile_type", ""))
+ or ("profile" if event_profile_type else _clean_text(representative.category))
+ or "event"
+ )
+ teacher_fields = {
+ "event_phrase": _clean_text(metadata.get("event_phrase", "")) or _clean_text(representative.value),
+ "semantic_slot": semantic_slot,
+ "target_status": event_target_status,
+ "time_expression_span": event_time_display_value,
+ "time_granularity": event_time_granularity,
+ "profile_type": event_profile_type,
+ }
+ base_event_signature = _runtime_event_signature(
+ group_hits=group_hits,
+ representative=representative,
+ speaker=speaker,
+ semantic_slot=event_profile_type or teacher_fields["semantic_slot"],
+ source_turn_hit=source_turn_hit,
+ )
+ typed_tunnel_metadata = merge_typed_metadata([metadata, *[dict(hit.metadata or {}) for hit in group_hits]])
+ event_typed_metadata_by_id[event_id] = typed_tunnel_metadata
+ typed_signature = typed_tunnel_signature_text(typed_tunnel_metadata)
+ event_text = _clean_text(metadata.get("event_phrase", "")) or representative.value
+ depth_prefix = " ".join(
+ item
+ for item in (
+ f"subject {subject_signature.replace('_', ' ')}" if subject_signature else "",
+ f"depth layer {depth_layer.replace('_', ' ')}" if depth_layer else "",
+ )
+ if item
+ )
+ runtime_event_text = f"{depth_prefix} {event_text}".strip() if depth_prefix else event_text
+ event_signature = (
+ compute_public_event_signature(
+ runtime_event_text,
+ speaker=_clean_text(speaker),
+ semantic_slot=_clean_text(event_profile_type or teacher_fields["semantic_slot"]),
+ )
+ or base_event_signature
+ )
+ if typed_signature and typed_signature not in event_signature:
+ event_signature = f"{event_signature} {typed_signature}".strip()
+ speaker_node_id = f"{event_id}:speaker:{_normalize(speaker).replace(' ', '_') or 'speaker'}"
+ add_node({"id": speaker_node_id, "type": "speaker", "text": speaker, "metadata": {"speaker": speaker}})
+ add_node(
+ {
+ "id": event_id,
+ "type": "event",
+ "text": runtime_event_text,
+ "speaker": speaker,
+ "turn_index": event_turn_index,
+ "session_name": session_name,
+ "dia_id": _clean_text(metadata.get("dia_id", "")),
+ "event_signature": event_signature,
+ "slot_key": _clean_text(representative.slot_key),
+ "state_signature": _clean_text(metadata.get("state_signature", "")),
+ "memory_signature": _clean_text(metadata.get("memory_signature", "")),
+ "target_status": event_target_status,
+ "time_granularity": event_time_granularity,
+ "time_value": event_time_value,
+ "time_display_value": event_time_display_value,
+ "profile_type": event_profile_type,
+ "profile_value": event_profile_value,
+ "depth_layer": depth_layer,
+ "subject_signature": subject_signature,
+ "tmcra_node_tags": list(typed_tunnel_metadata.get("tmcra_node_tags", []) or []),
+ "tmcra_path_tags": list(typed_tunnel_metadata.get("tmcra_path_tags", []) or []),
+ "tmcra_tunnel_roles": list(typed_tunnel_metadata.get("tmcra_tunnel_roles", []) or []),
+ "tmcra_tunnel_group_key": _clean_text(typed_tunnel_metadata.get("tmcra_tunnel_group_key", "")),
+ "teacher_fields": teacher_fields,
+ "metadata": {
+ "speaker": speaker,
+ "session_name": session_name,
+ "dia_id": _clean_text(metadata.get("dia_id", "")),
+ "slot_key": _clean_text(representative.slot_key),
+ "state_signature": _clean_text(metadata.get("state_signature", "")),
+ "memory_signature": _clean_text(metadata.get("memory_signature", "")),
+ "target_status": event_target_status,
+ "time_granularity": event_time_granularity,
+ "time_value": event_time_value,
+ "time_display_value": event_time_display_value,
+ "profile_type": event_profile_type,
+ "profile_value": event_profile_value,
+ "event_signature": event_signature,
+ "depth_layer": depth_layer,
+ "subject_signature": subject_signature,
+ **typed_tunnel_metadata,
+ },
+ }
+ )
+ edges.append({"id": f"{speaker_node_id}->{event_id}:speaker_of", "source": speaker_node_id, "target": event_id, "type": "speaker_of"})
+
+ time_node_ids: List[str] = []
+ profile_node_ids: List[str] = []
+ status_node_ids: List[str] = []
+ source_turn_node_ids: List[str] = []
+
+ if event_time_display_value or event_time_value:
+ time_node_id = f"{event_id}:time"
+ time_hit_metadata = dict((event_time_hit or representative).metadata or {})
+ add_node(
+ {
+ "id": time_node_id,
+ "type": "time",
+ "text": event_time_display_value or event_time_value,
+ "turn_index": int(getattr(event_time_hit, "turn_index", 0) or 0),
+ "time_display_value": event_time_display_value,
+ "time_value": event_time_value,
+ "time_granularity": event_time_granularity,
+ "metadata": {
+ "time_display_value": event_time_display_value,
+ "time_value": event_time_value,
+ "time_granularity": event_time_granularity,
+ "resolved_date": _clean_text(time_hit_metadata.get("resolved_date", "")),
+ },
+ }
+ )
+ edges.append({"id": f"{event_id}->{time_node_id}:time_of", "source": event_id, "target": time_node_id, "type": "time_of"})
+ time_node_ids.append(time_node_id)
+ if event_profile_value:
+ profile_node_id = f"{event_id}:profile:{_normalize(event_profile_type).replace(' ', '_') or 'profile'}"
+ add_node(
+ {
+ "id": profile_node_id,
+ "type": "profile",
+ "text": event_profile_value,
+ "turn_index": int(getattr(event_profile_hit or representative, "turn_index", 0) or 0),
+ "profile_type": event_profile_type,
+ "profile_value": event_profile_value,
+ "metadata": {
+ "profile_type": event_profile_type,
+ "profile_value": event_profile_value,
+ },
+ }
+ )
+ edges.append({"id": f"{event_id}->{profile_node_id}:profile_of", "source": event_id, "target": profile_node_id, "type": "profile_of"})
+ profile_node_ids.append(profile_node_id)
+ source_turn_text = _runtime_source_turn_text(source_turn_hit or representative, speaker=speaker)
+ if source_turn_text:
+ source_turn_node_id = f"{event_id}:source_turn"
+ add_node(
+ {
+ "id": source_turn_node_id,
+ "type": "source_turn",
+ "text": source_turn_text,
+ "turn_index": int(getattr(source_turn_hit or representative, "turn_index", 0) or 0),
+ "metadata": {
+ "speaker": speaker,
+ "dia_id": _clean_text(dict((source_turn_hit or representative).metadata or {}).get("dia_id", "")),
+ },
+ }
+ )
+ edges.append({"id": f"{event_id}->{source_turn_node_id}:supported_by_turn", "source": event_id, "target": source_turn_node_id, "type": "supported_by_turn"})
+ source_turn_node_ids.append(source_turn_node_id)
+ if event_target_status:
+ status_node_id = f"{event_id}:status"
+ add_node({"id": status_node_id, "type": "status", "text": event_target_status, "metadata": {"target_status": event_target_status}})
+ edges.append({"id": f"{event_id}->{status_node_id}:status_of", "source": event_id, "target": status_node_id, "type": "status_of"})
+ status_node_ids.append(status_node_id)
+ event_paths = build_default_path_templates(
+ event_id=event_id,
+ speaker_node_id=speaker_node_id,
+ time_node_ids=time_node_ids,
+ profile_node_ids=profile_node_ids,
+ status_node_ids=status_node_ids,
+ source_turn_node_ids=source_turn_node_ids,
+ )
+ for path in event_paths:
+ path_metadata = dict(path.get("metadata", {}) or {})
+ path_metadata["tmcra_path_tags"] = list(typed_tunnel_metadata.get("tmcra_path_tags", []) or [])
+ path_metadata["tmcra_tunnel_group_key"] = _clean_text(typed_tunnel_metadata.get("tmcra_tunnel_group_key", ""))
+ path["metadata"] = path_metadata
+ path["tmcra_path_tags"] = path_metadata["tmcra_path_tags"]
+ paths.extend(event_paths)
+ ordered_events.append(_runtime_event_sequence_key(session_name, event_turn_index, event_id))
+ ordered_event_ids = [event_id for _, _, _, event_id in sorted(ordered_events)]
+ for previous_event_id, next_event_id in zip(ordered_event_ids, ordered_event_ids[1:]):
+ typed_edge_tags = typed_edge_tags_between(
+ event_typed_metadata_by_id.get(previous_event_id, {}),
+ event_typed_metadata_by_id.get(next_event_id, {}),
+ )
+ edges.append(
+ {
+ "id": f"{previous_event_id}->{next_event_id}:same_session_next",
+ "source": previous_event_id,
+ "target": next_event_id,
+ "type": "same_session_next",
+ "metadata": {
+ "tmcra_edge_tags": typed_edge_tags,
+ "typed_tunnel_edge": bool(typed_edge_tags),
+ },
+ }
+ )
+ for index, source_event_id in enumerate(ordered_event_ids):
+ for target_event_id in ordered_event_ids[index + 1 : index + 12]:
+ typed_edge_tags = typed_edge_tags_between(
+ event_typed_metadata_by_id.get(source_event_id, {}),
+ event_typed_metadata_by_id.get(target_event_id, {}),
+ )
+ if not typed_edge_tags:
+ continue
+ typed_tunnel_edges.append(
+ {
+ "id": f"{source_event_id}->{target_event_id}:typed_tunnel",
+ "source": source_event_id,
+ "target": target_event_id,
+ "type": "typed_tunnel_candidate",
+ "metadata": {
+ "tmcra_edge_tags": typed_edge_tags,
+ "typed_tunnel_edge": True,
+ },
+ }
+ )
+ if len(typed_tunnel_edges) >= 64:
+ break
+ if len(typed_tunnel_edges) >= 64:
+ break
+ return {
+ "conversation_id": "runtime",
+ "query": query,
+ "nodes": nodes,
+ "edges": edges,
+ "typed_tunnel_edges": typed_tunnel_edges,
+ "paths": paths,
+ "grouped_hits": grouped_hits,
+ }
+
+
+def _public_graph_hits(graph: SessionMemoryGraphV2) -> List[MemoryHit]:
+ public_hits: List[MemoryHit] = []
+ for record in graph.records_by_id.values():
+ if _normalize(dict(record.metadata or {}).get("memory_layer", "")) == "slow":
+ continue
+ if record.state != "active":
+ continue
+ if not _clean_text(record.source_kind).startswith("public_dialog"):
+ continue
+ metadata = dict(record.metadata or {})
+ public_hits.append(
+ MemoryHit(
+ memory_id=record.memory_id,
+ category=record.category,
+ value=record.value,
+ relation=record.relation,
+ anchors=list(record.anchor_concepts),
+ score=max(float(record.confidence), float(record.salience), 0.01),
+ source_kind=record.source_kind,
+ slot_key=record.slot_key,
+ state=record.state,
+ turn_index=int(record.turn_index),
+ metadata=metadata,
+ )
+ )
+ public_hits.sort(key=lambda item: (int(item.turn_index), float(item.score)), reverse=True)
+ return public_hits
+
+
+_AUDIT_ANCHOR_QUERY_RE = re.compile(
+ r"(?i)\b(?:remember|recall|earlier|previous|previously|old|before|mentioned|said|quote|verbatim|original|"
+ r"that time|last time|bring back|return to|go back to|turn\s*\d+)\b|"
+ r"(?:\u7b2c\s*\d+\s*[\u8f6e\u6b21]|\d+\s*\u8f6e|\u4e4b\u524d|\u4ee5\u524d|\u521a\u521a|"
+ r"\u539f\u8bdd|\u90a3\u6b21|\u65e7\u8bdd\u9898|\u56de\u5230|\u63d0\u8d77|\u8bb0\u5f97)"
+)
+
+_AUDIT_TURN_ANCHOR_RE = re.compile(
+ r"(?i)\bturn\s*#?\s*(\d{1,6})\b|"
+ r"(?:\u7b2c\s*(\d{1,6})\s*[\u8f6e\u6b21]|\b(\d{1,6})\s*\u8f6e\b)"
+)
+
+_AUDIT_ANCHOR_STOPWORDS = set(_HYBRID_SYMBOLIC_STOPWORDS) | {
+ "about",
+ "again",
+ "back",
+ "bring",
+ "can",
+ "could",
+ "did",
+ "discuss",
+ "discussed",
+ "earlier",
+ "from",
+ "just",
+ "keep",
+ "mentioned",
+ "old",
+ "one",
+ "previous",
+ "previously",
+ "really",
+ "recall",
+ "remember",
+ "return",
+ "said",
+ "still",
+ "that",
+ "the",
+ "thing",
+ "think",
+ "this",
+ "turn",
+ "what",
+ "when",
+ "where",
+ "with",
+ "you",
+}
+
+_AUDIT_ANCHOR_GENERIC_MATCH_TOKENS = {
+ "body",
+ "coherence",
+ "continuity",
+ "fiction",
+ "fray",
+ "memories",
+ "memory",
+ "narrative",
+ "ourselves",
+ "physical",
+ "really",
+ "stories",
+ "story",
+ "tell",
+ "trust",
+}
+
+_AUDIT_ANCHOR_PHRASE_TOKEN_SETS = [
+ {"spine", "book"},
+ {"body", "spine"},
+ {"stories", "pages"},
+ {"story", "pages"},
+ {"islands", "self"},
+ {"archipelago", "self"},
+ {"loom", "body"},
+ {"braid", "body"},
+]
+
+
+def _audit_anchor_query(query: str) -> bool:
+ return bool(_AUDIT_ANCHOR_QUERY_RE.search(_clean_text(query)))
+
+
+def _audit_anchor_turn_numbers(query: str) -> List[int]:
+ numbers: List[int] = []
+ for match in _AUDIT_TURN_ANCHOR_RE.finditer(_clean_text(query)):
+ for group in match.groups():
+ if not group:
+ continue
+ try:
+ value = int(group)
+ except Exception:
+ continue
+ if value > 0 and value not in numbers:
+ numbers.append(value)
+ return numbers
+
+
+def _hit_event_id(hit: MemoryHit) -> str:
+ metadata = dict(hit.metadata or {})
+ event_id = _clean_text(metadata.get("event_id", ""))
+ if event_id:
+ return event_id
+ dia_id = _clean_text(metadata.get("dia_id", ""))
+ if dia_id:
+ return f"event::{dia_id}"
+ if int(hit.turn_index or 0) > 0:
+ return f"event::realchat:{int(hit.turn_index)}"
+ return ""
+
+
+def _audit_anchor_hit_text(hit: MemoryHit) -> str:
+ metadata = dict(hit.metadata or {})
+ values = [
+ hit.value,
+ hit.category,
+ hit.source_kind,
+ hit.slot_key,
+ metadata.get("event_text", ""),
+ metadata.get("source_span", ""),
+ metadata.get("raw_text", ""),
+ ]
+ return " ".join(_clean_text(value) for value in values if _clean_text(value))
+
+
+def _audit_anchor_content_tokens(text: str) -> set[str]:
+ tokens = set()
+ for token in _tokenize(text):
+ norm = _normalize(token)
+ if not norm or norm in _AUDIT_ANCHOR_STOPWORDS:
+ continue
+ if len(norm) < 3 and not any("\u4e00" <= ch <= "\u9fff" for ch in norm):
+ continue
+ tokens.add(norm)
+ return tokens
+
+
+def _audit_anchor_phrase_bonus(query_tokens: set[str], hit_tokens: set[str]) -> tuple[float, List[str]]:
+ matched_phrases: List[str] = []
+ bonus = 0.0
+ for phrase_tokens in _AUDIT_ANCHOR_PHRASE_TOKEN_SETS:
+ if phrase_tokens <= query_tokens and phrase_tokens <= hit_tokens:
+ matched_phrases.append("+".join(sorted(phrase_tokens)))
+ bonus += 5.0 if phrase_tokens == {"spine", "book"} else 3.0
+ return bonus, matched_phrases
+
+
+def _copy_hit_with_audit_anchor_boost(hit: MemoryHit, *, score: float, reason: str, matched_tokens: Sequence[str]) -> MemoryHit:
+ metadata = dict(hit.metadata or {})
+ metadata.update(
+ {
+ "audit_anchor_protected": True,
+ "audit_anchor_reason": reason,
+ "audit_anchor_original_score": round(float(hit.score), 6),
+ "audit_anchor_score": round(float(score), 6),
+ "audit_anchor_matched_tokens": list(matched_tokens)[:20],
+ }
+ )
+ return MemoryHit(
+ memory_id=hit.memory_id,
+ category=hit.category,
+ value=hit.value,
+ relation=hit.relation,
+ anchors=list(hit.anchors),
+ score=max(float(hit.score), round(1.0 + float(score), 6)),
+ source_kind=hit.source_kind,
+ slot_key=hit.slot_key,
+ state=hit.state,
+ turn_index=int(hit.turn_index),
+ metadata=metadata,
+ )
+
+
+def _audit_anchor_protected_hits(
+ *,
+ query: str,
+ final_hits: Sequence[MemoryHit],
+ candidate_hits: Sequence[MemoryHit],
+ metadata: Mapping[str, Any],
+ top_k: int,
+) -> Dict[str, Any]:
+ if not _audit_anchor_query(query):
+ return {"enabled": False, "hits": list(final_hits), "promoted_hits": [], "metadata": {"audit_anchor_enabled": False}}
+ query_tokens = _audit_anchor_content_tokens(query)
+ if not query_tokens:
+ return {"enabled": True, "hits": list(final_hits), "promoted_hits": [], "metadata": {"audit_anchor_enabled": True, "audit_anchor_reason": "no_content_tokens"}}
+
+ explicit_turns = _audit_anchor_turn_numbers(query)
+ explicit_turn_window = set()
+ for number in explicit_turns:
+ explicit_turn_window.update({number - 1, number, number + 1})
+
+ symbolic_ids = list(dict.fromkeys(str(item) for item in dict(metadata or {}).get("symbolic_recall_event_ids", []) or []))
+ learned_ids = list(dict.fromkeys(str(item) for item in dict(metadata or {}).get("learned_recall_event_ids", []) or []))
+ selected_ids = set(str(item) for item in dict(metadata or {}).get("selected_event_ids", []) or [])
+ symbolic_rank = {event_id: index for index, event_id in enumerate(symbolic_ids, start=1)}
+ learned_rank = {event_id: index for index, event_id in enumerate(learned_ids, start=1)}
+
+ pool: Dict[str, MemoryHit] = {}
+ for hit in list(final_hits) + list(candidate_hits):
+ key = hit.memory_id or f"{_hit_event_id(hit)}::{hit.slot_key}::{hit.value[:80]}"
+ if key and key not in pool:
+ pool[key] = hit
+
+ scored: List[tuple[float, str, List[str], MemoryHit]] = []
+ for hit in pool.values():
+ event_id = _hit_event_id(hit)
+ hit_tokens = _audit_anchor_content_tokens(_audit_anchor_hit_text(hit))
+ matched = sorted(query_tokens & hit_tokens)
+ distinctive_matched = [token for token in matched if token not in _AUDIT_ANCHOR_GENERIC_MATCH_TOKENS]
+ phrase_bonus, matched_phrases = _audit_anchor_phrase_bonus(query_tokens, hit_tokens)
+ if not matched and int(hit.turn_index or 0) not in explicit_turn_window:
+ continue
+ score = 0.0
+ reason_parts: List[str] = []
+ if int(hit.turn_index or 0) in explicit_turn_window:
+ score += 8.0
+ reason_parts.append("explicit_turn_anchor")
+ if matched:
+ weighted_overlap = 0.0
+ for token in matched:
+ weighted_overlap += 1.8 if token not in _AUDIT_ANCHOR_GENERIC_MATCH_TOKENS else 0.35
+ score += min(8.0, weighted_overlap)
+ score += len(matched) / max(1.0, float(min(len(query_tokens), len(hit_tokens))))
+ reason_parts.append("lexical_anchor_overlap")
+ if phrase_bonus > 0:
+ score += phrase_bonus
+ reason_parts.append("distinctive_phrase_anchor")
+ if event_id in symbolic_rank:
+ score += max(0.0, 2.0 - (symbolic_rank[event_id] - 1) * 0.08)
+ reason_parts.append("symbolic_recall_anchor")
+ if event_id in learned_rank:
+ score += max(0.0, 1.0 - (learned_rank[event_id] - 1) * 0.02)
+ reason_parts.append("learned_recall_anchor")
+ if event_id in selected_ids:
+ score -= 0.25
+ # For non-numeric old-topic probes, require a real content overlap so generic
+ # "earlier" phrasing does not promote unrelated old memories.
+ if not explicit_turns and len(distinctive_matched) < 1 and phrase_bonus <= 0:
+ continue
+ if score > 0:
+ scored.append((score, event_id, sorted(set(matched + matched_phrases)), hit))
+
+ scored.sort(key=lambda item: (item[0], -abs(int(item[3].turn_index or 0))), reverse=True)
+ max_promoted = 2 if not explicit_turns else 3
+ promoted: List[MemoryHit] = []
+ promoted_event_ids: set[str] = set()
+ for score, event_id, matched, hit in scored:
+ if event_id in promoted_event_ids:
+ continue
+ if any((hit.memory_id and hit.memory_id == current.memory_id) for current in final_hits[: max(1, top_k)]):
+ continue
+ reason = "explicit_turn_anchor" if int(hit.turn_index or 0) in explicit_turn_window else "old_topic_anchor"
+ promoted.append(_copy_hit_with_audit_anchor_boost(hit, score=score, reason=reason, matched_tokens=matched))
+ promoted_event_ids.add(event_id)
+ if len(promoted) >= max_promoted:
+ break
+
+ if not promoted:
+ return {
+ "enabled": True,
+ "hits": list(final_hits),
+ "promoted_hits": [],
+ "metadata": {
+ "audit_anchor_enabled": True,
+ "audit_anchor_turn_numbers": explicit_turns,
+ "audit_anchor_query_tokens": sorted(query_tokens)[:50],
+ "audit_anchor_promoted_event_ids": [],
+ },
+ }
+
+ merged: List[MemoryHit] = []
+ seen_keys: set[str] = set()
+ for hit in promoted + list(final_hits):
+ key = hit.memory_id or f"{_hit_event_id(hit)}::{hit.slot_key}::{hit.value[:80]}"
+ if key in seen_keys:
+ continue
+ seen_keys.add(key)
+ merged.append(hit)
+ if len(merged) >= max(1, int(top_k)):
+ break
+
+ return {
+ "enabled": True,
+ "hits": merged,
+ "promoted_hits": promoted,
+ "metadata": {
+ "audit_anchor_enabled": True,
+ "audit_anchor_turn_numbers": explicit_turns,
+ "audit_anchor_query_tokens": sorted(query_tokens)[:50],
+ "audit_anchor_promoted_event_ids": [_hit_event_id(hit) for hit in promoted],
+ "audit_anchor_promoted_turns": [int(hit.turn_index) for hit in promoted],
+ "audit_anchor_promoted_hit_count": len(promoted),
+ },
+ }
+
+
+def _learnable_graph_hits(graph: SessionMemoryGraphV2) -> List[MemoryHit]:
+ learnable_hits: List[MemoryHit] = []
+ for record in graph.records_by_id.values():
+ state = _normalize(record.state)
+ metadata = dict(record.metadata or {})
+ if _normalize(metadata.get("memory_layer", "")) == "slow":
+ continue
+ is_source_grounded_evidence = (
+ state == "evidence"
+ and (
+ _clean_text(record.source_kind).startswith("public_dialog")
+ or _clean_text(metadata.get("content_variant", "")) in {"source_turn", "llm_semantic_write"}
+ or _clean_text(metadata.get("write_path", "")) == "llm_semantic_writer_gate"
+ )
+ )
+ if state not in {"active", "parallel_active"} and not is_source_grounded_evidence:
+ continue
+ if _clean_text(record.slot_key).startswith("noise."):
+ continue
+ learnable_hits.append(
+ MemoryHit(
+ memory_id=record.memory_id,
+ category=record.category,
+ value=record.value,
+ relation=record.relation,
+ anchors=list(record.anchor_concepts),
+ score=max(float(record.confidence), float(record.salience), 0.01),
+ source_kind=record.source_kind,
+ slot_key=record.slot_key,
+ state=record.state,
+ turn_index=int(record.turn_index),
+ metadata=metadata,
+ )
+ )
+ learnable_hits.sort(
+ key=lambda item: (
+ _is_public_dialog_hit(item),
+ int(item.turn_index),
+ float(item.score),
+ ),
+ reverse=True,
+ )
+ return learnable_hits
+
+
+def _parse_structured_records(
+ payload: Dict[str, Any] | None,
+ *,
+ turn_index: int,
+ profile: TMCRAProfile | None = None,
+) -> List[SessionMemoryRecordV2]:
+ profile = profile or TMCRAProfile()
+ records: List[SessionMemoryRecordV2] = []
+ structured_rows: List[tuple[str, Mapping[str, Any]]] = []
+ for raw in (payload or {}).get("replacement_memory_records", []) or []:
+ if isinstance(raw, Mapping):
+ structured_rows.append(("formal", raw))
+ for raw in (payload or {}).get("suspect_memory_records", []) or []:
+ if isinstance(raw, Mapping):
+ structured_rows.append(("suspect", raw))
+ for index, (buffer_state, raw) in enumerate(structured_rows):
+ if not isinstance(raw, dict):
+ continue
+ category = _clean_text(raw.get("category", "memory")) or "memory"
+ value = _clean_text(raw.get("value", ""))
+ if not value:
+ continue
+ anchors = _dedupe(raw.get("anchors", []) or [], max_items=8)
+ metadata = dict(raw.get("metadata", {}) or {})
+ slot_key = profile.stable_slot_key(
+ category=category,
+ value=value,
+ anchors=anchors,
+ slot_key=_clean_text(raw.get("slot_key", "")) or _clean_text(raw.get("slot", "")),
+ relation=_clean_text(raw.get("relation", "")),
+ metadata=metadata,
+ )
+ metadata = {
+ **metadata,
+ "memory_role": _clean_text(metadata.get("memory_role", "")) or "user",
+ "authority": _clean_text(metadata.get("authority", "")) or "source",
+ "canonical_slot_key": _clean_text(metadata.get("canonical_slot_key", "")) or slot_key,
+ "writeback_class": _clean_text(metadata.get("writeback_class", "")),
+ "origin_query": _clean_text(metadata.get("origin_query", "")),
+ "origin_answer_id": _clean_text(metadata.get("origin_answer_id", "")),
+ "support_memory_ids": _dedupe(metadata.get("support_memory_ids", []) or []),
+ "support_fact_refs": _dedupe(metadata.get("support_fact_refs", []) or []),
+ "support_path_refs": _dedupe(metadata.get("support_path_refs", []) or []),
+ "promotion_state": _clean_text(metadata.get("promotion_state", "")) or "none",
+ "memory_buffer_state": buffer_state,
+ }
+ memory_id = f"{slot_key}:{turn_index}:{index}"
+ default_source_kind = "suspect_memory" if buffer_state == "suspect" else "replacement_memory"
+ default_state = "suspect" if buffer_state == "suspect" else ("active" if bool(raw.get("active", True)) else "historical")
+ records.append(
+ SessionMemoryRecordV2(
+ memory_id=memory_id,
+ category=category,
+ slot_key=slot_key,
+ value=value,
+ relation=_clean_text(raw.get("relation", "")) or f"{category}_memory",
+ anchor_concepts=anchors,
+ evidence_anchors=anchors,
+ salience=float(raw.get("salience", 0.88 if category in {"goal", "constraint"} else 0.74) or 0.74),
+ confidence=float(raw.get("confidence", 0.82) or 0.82),
+ source_kind=_clean_text(raw.get("source_kind", "")) or default_source_kind,
+ turn_index=int(raw.get("turn_index", turn_index) or turn_index),
+ state=_clean_text(raw.get("state", "")) or default_state,
+ metadata=metadata,
+ )
+ )
+ return records
+
+
+_WRITE_MARKERS = (
+ "goal update",
+ "goal seed",
+ "constraint update",
+ "constraint overwrite",
+ "constraint seed",
+ "preference update",
+ "preference overwrite",
+ "preference seed",
+ "terminology:",
+ "term seed",
+ "term overwrite",
+ "stage update",
+ "stage overwrite",
+ "path fact",
+ "fact:",
+ "memory update",
+)
+
+
+_OVERWRITE_MARKERS = (
+ "overwrite",
+ "replace",
+ "supersede",
+ "覆盖",
+ "替换",
+ "改成",
+ "更新为",
+)
+
+
+_TOPIC_BUCKET_CUE_GROUPS: tuple[tuple[tuple[str, ...], tuple[str, ...]], ...] = (
+ (
+ (
+ "食物",
+ "饮品",
+ "甜点",
+ "早餐",
+ "餐饮",
+ "口味",
+ "酸辣",
+ "热食",
+ "过敏",
+ "芒果",
+ "配料",
+ "推荐",
+ "吃",
+ "food",
+ "breakfast",
+ "allergy",
+ "mango",
+ ),
+ ("餐饮偏好安全", "食物偏好", "过敏约束", "口味偏好"),
+ ),
+ (
+ (
+ "界面",
+ "首页",
+ "首屏",
+ "配色",
+ "营销",
+ "横幅",
+ "电商",
+ "网站",
+ "移动端",
+ "视觉",
+ "工具型",
+ "ui",
+ "homepage",
+ "ecommerce",
+ "mobile",
+ "design",
+ ),
+ ("界面产品设计", "电商页面", "视觉布局", "移动端体验"),
+ ),
+ (
+ (
+ "api",
+ "writer",
+ "延迟",
+ "算法",
+ "耗时",
+ "评估",
+ "指标",
+ "调用",
+ "服务",
+ "key",
+ "runtime",
+ "latency",
+ "metric",
+ ),
+ ("api评估运行", "算法服务", "调用指标", "writer延迟"),
+ ),
+)
+
+
+_TOPIC_BUCKET_STOPWORDS = {
+ "不要",
+ "不用",
+ "复述",
+ "原话",
+ "只说",
+ "关键",
+ "最关键",
+ "以后",
+ "需要",
+ "必须",
+ "可以",
+ "时候",
+ "这个",
+ "那个",
+ "现在",
+ "做",
+ "说",
+ "的",
+ "了",
+ "和",
+ "与",
+ "to",
+ "the",
+ "and",
+ "for",
+}
+
+
+def _topic_bucket_keywords(text: str, *, max_items: int = 18) -> List[str]:
+ normalized = _normalize(text)
+ tokens: List[str] = []
+ for cues, anchors in _TOPIC_BUCKET_CUE_GROUPS:
+ if any(cue and _normalize(cue) in normalized for cue in cues):
+ tokens.extend(anchors)
+ tokens.extend(cue for cue in cues if len(cue) >= 2)
+ tokens.extend(re.findall(r"[\u4e00-\u9fff]{2,8}", str(text or "")))
+ tokens.extend(_tokenize(text))
+ cleaned = []
+ for token in tokens:
+ value = _clean_text(token)
+ if not value:
+ continue
+ normalized_value = _normalize(value)
+ if normalized_value in _TOPIC_BUCKET_STOPWORDS:
+ continue
+ if len(value) < 2:
+ continue
+ cleaned.append(value)
+ return _dedupe(cleaned, max_items=max_items)
+
+
+def _topic_bucket_id_from_keywords(keywords: Sequence[str]) -> str:
+ basis = "|".join(_normalize(item) for item in keywords[:8] if _clean_text(item))
+ if not basis:
+ basis = "general"
+ return "topic-" + str(uuid.uuid5(uuid.NAMESPACE_URL, f"tmcra-topic-bucket:{basis}"))[:12]
+
+
+def _topic_bucket_label_from_keywords(keywords: Sequence[str]) -> str:
+ visible = [
+ _clean_text(item)
+ for item in keywords
+ if _clean_text(item) and not _normalize(item).startswith("topic:")
+ ]
+ return " / ".join(visible[:3]) if visible else "动态话题"
+
+
+def _topic_bucket_overlap_score(left_keywords: Sequence[str], right_keywords: Sequence[str]) -> float:
+ left = {_normalize(item) for item in left_keywords if _clean_text(item)}
+ right = {_normalize(item) for item in right_keywords if _clean_text(item)}
+ if not left or not right:
+ return 0.0
+ overlap = left & right
+ if not overlap:
+ return 0.0
+ return len(overlap) / max(1, min(len(left), len(right)))
+
+
+def _explicit_topic_bucket_from_payload(answer_payload: Dict[str, Any] | None) -> Dict[str, Any]:
+ metadata = dict((answer_payload or {}).get("metadata", {}) or {})
+ explicit = metadata.get("topic_bucket")
+ if not isinstance(explicit, Mapping):
+ return {}
+ keywords = _dedupe(explicit.get("topic_keywords", explicit.get("keywords", [])) or [], max_items=24)
+ label = _clean_text(explicit.get("topic_label", explicit.get("label", "")))
+ if not keywords and label:
+ keywords = _topic_bucket_keywords(label, max_items=12)
+ bucket_id = _clean_text(explicit.get("topic_bucket_id", explicit.get("id", "")))
+ if not bucket_id:
+ bucket_id = _topic_bucket_id_from_keywords(keywords or [label])
+ return {
+ "topic_bucket_id": bucket_id,
+ "topic_label": label or _topic_bucket_label_from_keywords(keywords),
+ "topic_keywords": keywords,
+ "topic_confidence": float(explicit.get("confidence", explicit.get("topic_confidence", 0.92)) or 0.92),
+ "topic_assignment_source": "explicit_payload",
+ }
+
+
+def _coerce_topic_bucket(metadata: Mapping[str, Any]) -> Dict[str, Any]:
+ bucket_id = _clean_text(metadata.get("topic_bucket_id", ""))
+ if not bucket_id:
+ nested = metadata.get("topic_bucket")
+ if isinstance(nested, Mapping):
+ bucket_id = _clean_text(nested.get("topic_bucket_id", nested.get("id", "")))
+ if not bucket_id:
+ return {}
+ keywords = _dedupe(metadata.get("topic_keywords", []) or [], max_items=32)
+ nested = metadata.get("topic_bucket")
+ if isinstance(nested, Mapping):
+ keywords = _dedupe([*keywords, *(nested.get("topic_keywords", nested.get("keywords", [])) or [])], max_items=32)
+ label = _clean_text(metadata.get("topic_label", ""))
+ if not label and isinstance(nested, Mapping):
+ label = _clean_text(nested.get("topic_label", nested.get("label", "")))
+ return {
+ "topic_bucket_id": bucket_id,
+ "topic_label": label or _topic_bucket_label_from_keywords(keywords),
+ "topic_keywords": keywords,
+ }
+
+
+def _collect_topic_buckets(graph: SessionMemoryGraphV2) -> Dict[str, Dict[str, Any]]:
+ buckets: Dict[str, Dict[str, Any]] = {}
+
+ def merge(bucket: Mapping[str, Any], *, record_id: str = "", turn_index: int = 0) -> None:
+ bucket_id = _clean_text(bucket.get("topic_bucket_id", bucket.get("id", "")))
+ if not bucket_id:
+ return
+ current = buckets.setdefault(
+ bucket_id,
+ {
+ "topic_bucket_id": bucket_id,
+ "topic_label": _clean_text(bucket.get("topic_label", bucket.get("label", ""))),
+ "topic_keywords": [],
+ "record_ids": [],
+ "last_turn_index": 0,
+ },
+ )
+ current["topic_label"] = current.get("topic_label") or _clean_text(bucket.get("topic_label", bucket.get("label", "")))
+ current["topic_keywords"] = _dedupe(
+ [*list(current.get("topic_keywords", []) or []), *(bucket.get("topic_keywords", bucket.get("keywords", [])) or [])],
+ max_items=32,
+ )
+ if record_id:
+ current["record_ids"] = _dedupe([*list(current.get("record_ids", []) or []), record_id], max_items=200)
+ current["last_turn_index"] = max(int(current.get("last_turn_index", 0) or 0), int(turn_index or 0))
+
+ for record in getattr(graph, "records_by_id", {}).values():
+ metadata = dict(record.metadata or {})
+ bucket = _coerce_topic_bucket(metadata)
+ if bucket:
+ merge(bucket, record_id=record.memory_id, turn_index=int(record.turn_index))
+ for turn in list(getattr(graph, "turn_log", []) or []):
+ metadata = dict(turn.get("metadata", {}) if isinstance(turn, Mapping) else getattr(turn, "metadata", {}) or {})
+ bucket = _coerce_topic_bucket(metadata)
+ if bucket:
+ merge(bucket, turn_index=int(turn.get("turn_index", 0) if isinstance(turn, Mapping) else getattr(turn, "turn_index", 0) or 0))
+ for bucket in buckets.values():
+ if not bucket.get("topic_label"):
+ bucket["topic_label"] = _topic_bucket_label_from_keywords(bucket.get("topic_keywords", []) or [])
+ return buckets
+
+
+def _assign_topic_bucket_for_text(
+ graph: SessionMemoryGraphV2,
+ text: str,
+ *,
+ answer_payload: Dict[str, Any] | None = None,
+ turn_index: int = 0,
+ create: bool = True,
+) -> Dict[str, Any]:
+ explicit = _explicit_topic_bucket_from_payload(answer_payload)
+ if explicit:
+ explicit["turn_index"] = int(turn_index or 0)
+ return explicit
+ keywords = _topic_bucket_keywords(text, max_items=24)
+ if not keywords:
+ keywords = _dedupe([_clean_text(text)[:40]], max_items=1)
+ buckets = _collect_topic_buckets(graph)
+ best_bucket: Dict[str, Any] | None = None
+ best_score = 0.0
+ for bucket in buckets.values():
+ score = _topic_bucket_overlap_score(keywords, bucket.get("topic_keywords", []) or [])
+ if score > best_score:
+ best_bucket = bucket
+ best_score = score
+ if best_bucket and best_score >= 0.22:
+ merged_keywords = _dedupe([*list(best_bucket.get("topic_keywords", []) or []), *keywords], max_items=32)
+ return {
+ "topic_bucket_id": best_bucket["topic_bucket_id"],
+ "topic_label": best_bucket.get("topic_label") or _topic_bucket_label_from_keywords(merged_keywords),
+ "topic_keywords": merged_keywords,
+ "topic_confidence": round(min(0.97, 0.68 + best_score * 0.24), 6),
+ "topic_assignment_source": "reused_by_overlap",
+ "topic_match_score": round(best_score, 6),
+ "turn_index": int(turn_index or 0),
+ }
+ bucket_id = _topic_bucket_id_from_keywords(keywords)
+ return {
+ "topic_bucket_id": bucket_id,
+ "topic_label": _topic_bucket_label_from_keywords(keywords),
+ "topic_keywords": keywords,
+ "topic_confidence": 0.72 if create else 0.58,
+ "topic_assignment_source": "created_by_dialog" if create else "query_probe",
+ "topic_match_score": round(best_score, 6),
+ "turn_index": int(turn_index or 0),
+ }
+
+
+def _apply_topic_bucket_to_records(records: List[SessionMemoryRecordV2], topic_bucket: Mapping[str, Any]) -> None:
+ if not records or not topic_bucket:
+ return
+ bucket_id = _clean_text(topic_bucket.get("topic_bucket_id", ""))
+ if not bucket_id:
+ return
+ label = _clean_text(topic_bucket.get("topic_label", "")) or "动态话题"
+ keywords = _dedupe(topic_bucket.get("topic_keywords", []) or [], max_items=32)
+ for record in records:
+ metadata = dict(record.metadata or {})
+ metadata.update(
+ {
+ "topic_bucket_id": bucket_id,
+ "topic_label": label,
+ "topic_keywords": keywords,
+ "topic_confidence": float(topic_bucket.get("topic_confidence", 0.72) or 0.72),
+ "topic_assignment_source": _clean_text(topic_bucket.get("topic_assignment_source", "")) or "created_by_dialog",
+ }
+ )
+ record.metadata = metadata
+ record.anchor_concepts = _dedupe([*list(record.anchor_concepts or []), f"topic:{label}", *keywords[:8]], max_items=24)
+ evidence_anchors = list(metadata.get("evidence_anchors", []) or [])
+ metadata["evidence_anchors"] = _dedupe([*evidence_anchors, f"topic:{label}", *keywords[:8]], max_items=24)
+
+
+def _last_topic_turn(graph: SessionMemoryGraphV2) -> Dict[str, Any]:
+ for turn in reversed(list(getattr(graph, "turn_log", []) or [])):
+ metadata = dict(turn.get("metadata", {}) if isinstance(turn, Mapping) else getattr(turn, "metadata", {}) or {})
+ bucket = _coerce_topic_bucket(metadata)
+ if bucket:
+ bucket["turn_index"] = int(turn.get("turn_index", 0) if isinstance(turn, Mapping) else getattr(turn, "turn_index", 0) or 0)
+ return bucket
+ return {}
+
+
+def _add_topic_bridge_edges(
+ graph: SessionMemoryGraphV2,
+ *,
+ previous_topic: Mapping[str, Any],
+ current_topic: Mapping[str, Any],
+ current_record_ids: Sequence[str],
+ turn_index: int,
+ evidence: str,
+) -> Dict[str, Any]:
+ previous_id = _clean_text(previous_topic.get("topic_bucket_id", ""))
+ current_id = _clean_text(current_topic.get("topic_bucket_id", ""))
+ if not previous_id or not current_id or previous_id == current_id or not current_record_ids:
+ return {"topic_bridge_edge_count": 0}
+ buckets = _collect_topic_buckets(graph)
+ previous_record_ids = list((buckets.get(previous_id, {}) or {}).get("record_ids", []) or [])[-4:]
+ if not previous_record_ids:
+ return {"topic_bridge_edge_count": 0}
+ edge_count = 0
+ evidence_text = _clean_text(evidence)[:240]
+ for source_id in previous_record_ids:
+ for target_id in list(current_record_ids)[:4]:
+ if source_id == target_id:
+ continue
+ edge = SessionMemoryEdgeV2(
+ edge_id=f"{source_id}->{target_id}:topic_bridge:{previous_id}->{current_id}",
+ source_memory_id=source_id,
+ target_memory_id=target_id,
+ edge_type="topic_bridge",
+ score=0.54,
+ model_score=0.0,
+ evidence_turn=int(turn_index or 0),
+ evidence=evidence_text,
+ metadata={
+ "from_topic_bucket_id": previous_id,
+ "to_topic_bucket_id": current_id,
+ "from_topic_label": _clean_text(previous_topic.get("topic_label", "")),
+ "to_topic_label": _clean_text(current_topic.get("topic_label", "")),
+ "bridge_reason": "adjacent_dialog_topic_transition",
+ },
+ )
+ graph._upsert_memory_edge(edge)
+ edge_count += 1
+ return {
+ "topic_bridge_edge_count": edge_count,
+ "topic_bridge_from": previous_id,
+ "topic_bridge_to": current_id,
+ }
+
+
+def _add_dialogue_tunnel_edges(
+ graph: SessionMemoryGraphV2,
+ *,
+ current_topic: Mapping[str, Any],
+ current_record_ids: Sequence[str],
+ turn_index: int,
+ evidence: str,
+) -> Dict[str, Any]:
+ current_id = _clean_text(current_topic.get("topic_bucket_id", ""))
+ if not current_id or not current_record_ids:
+ return {"dialogue_tunnel_edge_count": 0}
+ buckets = _collect_topic_buckets(graph)
+ edge_count = 0
+ evidence_text = _clean_text(evidence)[:240]
+ source_ids: List[tuple[str, str, str]] = []
+ for bucket_id, bucket in sorted(
+ buckets.items(),
+ key=lambda item: int(item[1].get("last_turn_index", 0) or 0),
+ reverse=True,
+ ):
+ if bucket_id == current_id:
+ continue
+ for source_id in list(bucket.get("record_ids", []) or [])[-2:]:
+ if source_id not in current_record_ids:
+ source_ids.append((bucket_id, _clean_text(bucket.get("topic_label", "")), source_id))
+ if len(source_ids) >= 6:
+ break
+ for source_bucket_id, source_label, source_id in source_ids[:6]:
+ for target_id in list(current_record_ids)[:2]:
+ if source_id == target_id:
+ continue
+ edge = SessionMemoryEdgeV2(
+ edge_id=f"{source_id}->{target_id}:dialogue_tunnel:{source_bucket_id}->{current_id}",
+ source_memory_id=source_id,
+ target_memory_id=target_id,
+ edge_type="dialogue_tunnel",
+ score=0.24,
+ model_score=0.0,
+ evidence_turn=int(turn_index or 0),
+ evidence=evidence_text,
+ metadata={
+ "from_topic_bucket_id": source_bucket_id,
+ "to_topic_bucket_id": current_id,
+ "from_topic_label": source_label,
+ "to_topic_label": _clean_text(current_topic.get("topic_label", "")),
+ "bridge_reason": "high_resistance_dialogue_level_tunnel",
+ },
+ )
+ graph._upsert_memory_edge(edge)
+ edge_count += 1
+ return {"dialogue_tunnel_edge_count": edge_count}
+
+
+def _topic_adjacent_bucket_ids(graph: SessionMemoryGraphV2, bucket_id: str) -> set[str]:
+ adjacent: set[str] = set()
+ if not bucket_id:
+ return adjacent
+ for edge in getattr(graph, "memory_edges", {}).values():
+ if _normalize(edge.edge_type) != "topic_bridge":
+ continue
+ metadata = dict(edge.metadata or {})
+ left = _clean_text(metadata.get("from_topic_bucket_id", ""))
+ right = _clean_text(metadata.get("to_topic_bucket_id", ""))
+ if left == bucket_id and right:
+ adjacent.add(right)
+ if right == bucket_id and left:
+ adjacent.add(left)
+ return adjacent
+
+
+def _dialogue_tunnel_bucket_ids(graph: SessionMemoryGraphV2, bucket_id: str) -> set[str]:
+ adjacent: set[str] = set()
+ if not bucket_id:
+ return adjacent
+ for edge in getattr(graph, "memory_edges", {}).values():
+ if _normalize(edge.edge_type) != "dialogue_tunnel":
+ continue
+ metadata = dict(edge.metadata or {})
+ left = _clean_text(metadata.get("from_topic_bucket_id", ""))
+ right = _clean_text(metadata.get("to_topic_bucket_id", ""))
+ if left == bucket_id and right:
+ adjacent.add(right)
+ if right == bucket_id and left:
+ adjacent.add(left)
+ return adjacent
+
+
+def _topic_bridge_requested(query: str) -> bool:
+ text = _normalize(query)
+ if not text:
+ return False
+ bridge_markers = (
+ "关联",
+ "联系",
+ "链条",
+ "脉络",
+ "延展",
+ "深入",
+ "对比",
+ "整合",
+ "整体",
+ "上下文",
+ "刚才",
+ "之前",
+ "上面",
+ "跨话题",
+ "隧穿",
+ "related",
+ "connect",
+ "compare",
+ "context",
+ "chain",
+ )
+ return any(marker in text for marker in bridge_markers)
+
+
+def _dialogue_tunnel_requested(query: str) -> bool:
+ text = _normalize(query)
+ if not text:
+ return False
+ markers = (
+ "跨话题",
+ "跨对话",
+ "不同话题",
+ "不同对话",
+ "历史对话",
+ "长期脉络",
+ "整体脉络",
+ "所有相关",
+ "全局",
+ "全局记忆",
+ "远一点",
+ "更深",
+ "深层关联",
+ "对话级隧穿",
+ "记忆隧穿",
+ "cross topic",
+ "cross-topic",
+ "cross dialogue",
+ "cross-session",
+ "global context",
+ "long range",
+ )
+ return any(marker in text for marker in markers)
+
+
+def _topic_bucket_record_to_hit(
+ record: SessionMemoryRecordV2,
+ *,
+ query_topic: Mapping[str, Any],
+ rank: int,
+ rescue_kind: str = "topic_bucket",
+) -> MemoryHit:
+ metadata = dict(record.metadata or {})
+ dialogue_tunnel = _normalize(rescue_kind) == "dialogue_tunnel"
+ metadata.update(
+ {
+ "topic_bucket_rescue": not dialogue_tunnel,
+ "dialogue_tunnel_rescue": dialogue_tunnel,
+ "topic_bucket_rescue_rank": int(rank),
+ "topic_bucket_query_id": _clean_text(query_topic.get("topic_bucket_id", "")),
+ "topic_bucket_query_label": _clean_text(query_topic.get("topic_label", "")),
+ "topic_bucket_same": not dialogue_tunnel,
+ "topic_bucket_bridge": False,
+ "topic_bucket_bridge_allowed": False,
+ "topic_bucket_dialogue_tunnel_allowed": dialogue_tunnel,
+ "topic_bucket_overlap": 0.0 if dialogue_tunnel else 1.0,
+ }
+ )
+ category = _normalize(record.category)
+ value_text = _normalize(record.value)
+ hardish = (
+ category == "constraint"
+ or _normalize(metadata.get("memory_type", "")) == "hard_constraint"
+ or _normalize(metadata.get("durability", "")) == "hard"
+ or _normalize(metadata.get("conflict_policy", "")) == "must_preserve"
+ or any(marker in value_text for marker in ("过敏", "必须", "避开", "禁止", "不能", "must", "avoid", "allergy"))
+ )
+ base_score = max(float(record.confidence), float(record.salience), 0.62)
+ if dialogue_tunnel:
+ if hardish:
+ base_score += 0.42
+ metadata.setdefault("memory_type", "hard_constraint")
+ metadata.setdefault("durability", "hard")
+ metadata.setdefault("conflict_policy", "must_preserve")
+ elif category == "preference":
+ base_score += 0.22
+ metadata.setdefault("memory_type", "durable_preference")
+ metadata.setdefault("durability", "long_term")
+ else:
+ base_score += 0.12
+ metadata["dialogue_tunnel_resistance"] = "high"
+ elif hardish:
+ base_score += 1.35
+ metadata.setdefault("memory_type", "hard_constraint")
+ metadata.setdefault("durability", "hard")
+ metadata.setdefault("conflict_policy", "must_preserve")
+ elif category == "preference":
+ base_score += 0.72
+ metadata.setdefault("memory_type", "durable_preference")
+ metadata.setdefault("durability", "long_term")
+ else:
+ base_score += 0.38
+ return MemoryHit(
+ memory_id=record.memory_id,
+ category=record.category,
+ value=record.value,
+ relation=record.relation,
+ anchors=list(record.anchor_concepts),
+ score=base_score,
+ source_kind=record.source_kind,
+ slot_key=record.slot_key,
+ state=record.state,
+ turn_index=int(record.turn_index),
+ metadata=metadata,
+ )
+
+
+def _profile_query_rescue_hits(
+ graph: SessionMemoryGraphV2,
+ query: str,
+ *,
+ top_k: int,
+) -> List[MemoryHit]:
+ query_raw_tokens = set(_path_utility_tokens(query))
+ query_tokens = _profile_query_expanded_tokens(query)
+ intent = infer_profile_query_intent(query)
+ if not bool(intent.get("enabled")):
+ return []
+ rescued: List[tuple[float, MemoryHit]] = []
+ for record in getattr(graph, "records_by_id", {}).values():
+ metadata = dict(record.metadata or {})
+ if record.state != "active":
+ continue
+ if not is_profile_layer_record(
+ category=record.category,
+ source_kind=record.source_kind,
+ semantic_slot=metadata.get("semantic_slot", ""),
+ metadata=metadata,
+ ):
+ continue
+ delta, reason = profile_query_score_delta(
+ query=query,
+ query_tokens=query_tokens,
+ category=record.category,
+ source_kind=record.source_kind,
+ semantic_slot=metadata.get("semantic_slot", ""),
+ value=record.value,
+ anchors=record.anchor_concepts,
+ metadata=metadata,
+ )
+ if delta <= 0:
+ continue
+ match_score, overlap_tokens, raw_overlap_tokens = _profile_hit_match_score(
+ query_raw_tokens,
+ query_tokens,
+ MemoryHit(
+ memory_id=record.memory_id,
+ category=record.category,
+ value=record.value,
+ relation=record.relation,
+ anchors=list(record.anchor_concepts),
+ score=max(float(record.confidence), float(record.salience), 0.01),
+ source_kind=record.source_kind,
+ slot_key=record.slot_key,
+ state=record.state,
+ turn_index=int(record.turn_index),
+ metadata=metadata,
+ ),
+ )
+ if match_score <= 0.0:
+ continue
+ if match_score < 0.34 and not raw_overlap_tokens:
+ continue
+ metadata.update(
+ {
+ "profile_query_rescue": True,
+ "profile_query_rescue_reason": reason or "profile_route",
+ "profile_query_match_score": round(match_score, 6),
+ "profile_query_overlap_tokens": list(overlap_tokens),
+ "profile_query_raw_overlap_tokens": list(raw_overlap_tokens),
+ "topic_bucket_profile_route_preserved": True,
+ "match_reason": ",".join(_dedupe([metadata.get("match_reason", ""), reason or "profile_route"], max_items=4)),
+ }
+ )
+ hit = MemoryHit(
+ memory_id=record.memory_id,
+ category=record.category,
+ value=record.value,
+ relation=record.relation,
+ anchors=list(record.anchor_concepts),
+ score=max(float(record.confidence), float(record.salience), 0.62) + float(delta) + float(match_score),
+ source_kind=record.source_kind,
+ slot_key=record.slot_key,
+ state=record.state,
+ turn_index=int(record.turn_index),
+ metadata=metadata,
+ )
+ rescued.append((match_score, hit))
+ rescued.sort(key=lambda item: (float(item[0]), float(item[1].score), int(item[1].turn_index)), reverse=True)
+ return [
+ hit
+ for _, hit in rescued[: max(1, min(24, int(top_k or 1) * 3))]
+ ]
+
+
+def _memory_hit_from_record(record: SessionMemoryRecordV2, *, score: float | None = None, metadata: Mapping[str, Any] | None = None) -> MemoryHit:
+ record_metadata = {**dict(record.metadata or {}), **dict(metadata or {})}
+ return MemoryHit(
+ memory_id=record.memory_id,
+ category=record.category,
+ value=record.value,
+ relation=record.relation,
+ anchors=list(record.anchor_concepts),
+ score=max(float(record.confidence), float(record.salience), 0.01) if score is None else float(score),
+ source_kind=record.source_kind,
+ slot_key=record.slot_key,
+ state=record.state,
+ turn_index=int(record.turn_index),
+ metadata=record_metadata,
+ )
+
+
+_FACET_NUMERIC_QUERY_TOKENS = {
+ "amount",
+ "count",
+ "counts",
+ "duration",
+ "durations",
+ "many",
+ "much",
+ "number",
+ "quantity",
+ "sum",
+ "total",
+ "totals",
+ "weeks",
+ "week",
+ "hours",
+ "hour",
+ "dollars",
+ "dollar",
+ "tenants",
+ "tickets",
+}
+_FACET_TEMPORAL_QUERY_TOKENS = {
+ "after",
+ "before",
+ "date",
+ "deadline",
+ "end",
+ "finish",
+ "finished",
+ "start",
+ "started",
+ "time",
+ "when",
+}
+
+
+def _facet_query_pack_hits(
+ graph: SessionMemoryGraphV2,
+ query: str,
+ final_hits: Sequence[MemoryHit],
+ *,
+ top_k: int,
+) -> Dict[str, Any]:
+ query_tokens = set(_path_utility_tokens(query))
+ if not query_tokens:
+ return {"hits": list(final_hits), "metadata": {"facet_query_pack_enabled": False, "facet_query_pack_reason": "empty_query_tokens"}}
+ numeric_query = bool(query_tokens & _FACET_NUMERIC_QUERY_TOKENS)
+ temporal_query = bool(query_tokens & _FACET_TEMPORAL_QUERY_TOKENS)
+ if not numeric_query and not temporal_query and not any("facet" in _normalize(token) for token in query_tokens):
+ return {"hits": list(final_hits), "metadata": {"facet_query_pack_enabled": False, "facet_query_pack_reason": "no_facet_intent"}}
+
+ records = list(getattr(graph, "records_by_id", {}).values())
+ first_record_by_slot_key: Dict[str, SessionMemoryRecordV2] = {}
+ for candidate in records:
+ first_record_by_slot_key.setdefault(_clean_text(candidate.slot_key).lower(), candidate)
+
+ candidate_rows: List[tuple[float, SessionMemoryRecordV2, SessionMemoryRecordV2 | None, Dict[str, Any]]] = []
+ for record in records:
+ metadata = dict(record.metadata or {})
+ if _normalize(metadata.get("content_variant", "")) != "event_facet_write":
+ continue
+ if record.state not in {"active", "parallel_active", "evidence"}:
+ continue
+ facet_type = _normalize(metadata.get("facet_type", ""))
+ parent_slot_key = _clean_text(metadata.get("facet_parent_slot_key", ""))
+ parent = first_record_by_slot_key.get(parent_slot_key.lower())
+ parent_text = " ".join(
+ [
+ _clean_text(parent.value if parent else ""),
+ " ".join(parent.anchor_concepts if parent else []),
+ _clean_text(dict(parent.metadata or {}).get("source_span", "") if parent else ""),
+ ]
+ )
+ facet_text = " ".join(
+ [
+ record.value,
+ " ".join(record.anchor_concepts or []),
+ _clean_text(metadata.get("facet_type", "")),
+ _clean_text(metadata.get("facet_role", "")),
+ _clean_text(metadata.get("facet_source_span", "")),
+ parent_text,
+ ]
+ )
+ facet_tokens = set(_path_utility_tokens(facet_text))
+ parent_tokens = set(_path_utility_tokens(parent_text))
+ overlap_tokens = query_tokens & facet_tokens
+ parent_overlap_tokens = query_tokens & parent_tokens
+ unit_overlap = bool(query_tokens & set(_path_utility_tokens(record.value)))
+ score = 0.0
+ if overlap_tokens:
+ score += min(0.72, len(overlap_tokens) / max(1.0, len(query_tokens)) * 1.2)
+ if parent_overlap_tokens:
+ score += min(0.72, len(parent_overlap_tokens) / max(1.0, len(query_tokens)) * 1.35)
+ if numeric_query and facet_type == "numeric":
+ score += 0.42
+ if unit_overlap:
+ score += 0.42
+ if temporal_query and facet_type == "temporal":
+ score += 0.32
+ if facet_type == "entity" and parent_overlap_tokens:
+ score += 0.26
+ if parent is not None and _normalize(dict(parent.metadata or {}).get("content_variant", "")) == "llm_semantic_write":
+ score += 0.08
+ if score < 0.58:
+ continue
+ candidate_rows.append(
+ (
+ round(min(2.75, 1.18 + score), 6),
+ record,
+ parent,
+ {
+ "facet_query_pack_overlap_tokens": sorted(overlap_tokens)[:12],
+ "facet_query_pack_parent_overlap_tokens": sorted(parent_overlap_tokens)[:12],
+ "facet_query_pack_unit_overlap": bool(unit_overlap),
+ "facet_query_pack_score": round(score, 6),
+ },
+ )
+ )
+
+ if not candidate_rows:
+ return {
+ "hits": list(final_hits),
+ "metadata": {
+ "facet_query_pack_enabled": True,
+ "facet_query_pack_inserted_hit_count": 0,
+ "facet_query_pack_candidate_count": 0,
+ },
+ }
+
+ candidate_rows.sort(key=lambda item: (float(item[0]), int(item[1].turn_index)), reverse=True)
+ selected = candidate_rows[: max(4, min(18, int(top_k or 1) * 2))]
+ packed_hits: List[MemoryHit] = []
+ for score, facet_record, parent, extra_metadata in selected:
+ facet_metadata = {
+ **extra_metadata,
+ "facet_query_pack": True,
+ "evidence_snippet_role": "facet_query_attribute",
+ }
+ packed_hits.append(_memory_hit_from_record(facet_record, score=score, metadata=facet_metadata))
+ if parent is not None:
+ packed_hits.append(
+ _memory_hit_from_record(
+ parent,
+ score=max(1.12, score - 0.04),
+ metadata={
+ **extra_metadata,
+ "facet_query_pack": True,
+ "evidence_snippet_role": "facet_parent_event",
+ "facet_query_pack_child_id": facet_record.memory_id,
+ },
+ )
+ )
+
+ merged: List[MemoryHit] = []
+ seen_ids: set[str] = set()
+ for hit in [*packed_hits, *list(final_hits)]:
+ if hit.memory_id and hit.memory_id in seen_ids:
+ continue
+ if hit.memory_id:
+ seen_ids.add(hit.memory_id)
+ merged.append(hit)
+ return {
+ "hits": merged,
+ "metadata": {
+ "facet_query_pack_enabled": True,
+ "facet_query_pack_candidate_count": len(candidate_rows),
+ "facet_query_pack_inserted_hit_count": len(packed_hits),
+ "facet_query_pack_numeric_query": bool(numeric_query),
+ "facet_query_pack_temporal_query": bool(temporal_query),
+ },
+ }
+
+
+_UNIT_COVERAGE_COUNT_TOKENS = {
+ "amount",
+ "amounts",
+ "count",
+ "counts",
+ "cost",
+ "costs",
+ "dollar",
+ "dollars",
+ "how",
+ "many",
+ "much",
+ "number",
+ "minimum",
+ "maximum",
+ "paid",
+ "percent",
+ "percentage",
+ "price",
+ "prices",
+ "sale",
+ "sales",
+ "sell",
+ "sold",
+ "total",
+ "totals",
+ "sum",
+ "value",
+ "valued",
+ "values",
+ "worth",
+ "items",
+ "projects",
+ "events",
+ "things",
+}
+_UNIT_COVERAGE_TEMPORAL_TOKENS = {
+ "ago",
+ "after",
+ "before",
+ "between",
+ "consecutive",
+ "date",
+ "days",
+ "day",
+ "first",
+ "last",
+ "months",
+ "month",
+ "order",
+ "passed",
+ "since",
+ "weeks",
+ "week",
+}
+_MULTI_UNIT_CHAIN_TEMPORAL_COMPARISON_TOKENS = {
+ "after",
+ "before",
+ "between",
+ "consecutive",
+ "earlier",
+ "first",
+ "later",
+ "last",
+ "order",
+ "since",
+}
+_UNIT_COVERAGE_QUERY_DROP_TOKENS = {
+ "date",
+ "fri",
+ "friday",
+ "mon",
+ "monday",
+ "question",
+ "sat",
+ "saturday",
+ "sun",
+ "sunday",
+ "thu",
+ "thursday",
+ "tue",
+ "tuesday",
+ "wed",
+ "wednesday",
+}
+_MULTI_UNIT_CHAIN_DISABLED_MODES = {"", "off", "disabled", "none", "false", "0"}
+_MULTI_UNIT_CHAIN_COUNT_TOKENS = {
+ *_UNIT_COVERAGE_COUNT_TOKENS,
+ "which",
+ "each",
+ "all",
+ "both",
+}
+_MULTI_UNIT_CHAIN_FOCUS_DROP_TOKENS = {
+ *_MULTI_UNIT_CHAIN_COUNT_TOKENS,
+ *_UNIT_COVERAGE_TEMPORAL_TOKENS,
+ "i",
+ "me",
+ "my",
+ "mine",
+ "am",
+ "is",
+ "are",
+ "was",
+ "were",
+ "need",
+ "needs",
+ "needed",
+ "currently",
+ "did",
+ "does",
+ "fri",
+ "friday",
+ "mon",
+ "monday",
+ "or",
+ "question",
+ "sat",
+ "saturday",
+ "sun",
+ "sunday",
+ "thu",
+ "thursday",
+ "tue",
+ "tuesday",
+ "wed",
+ "wednesday",
+ "what",
+ "which",
+ "who",
+}
+_MULTI_UNIT_CHAIN_FACET_TYPES = {"action", "entity", "numeric", "role", "evidence_role", "state", "temporal"}
+_MULTI_UNIT_CHAIN_UNIT_KINDS = {
+ "action_unit",
+ "semantic_event",
+ "target_entity",
+ "numeric_quantity",
+ "leadership_role",
+ "participation_role",
+ "state_status",
+ "temporal_anchor",
+ "evidence_role",
+ "profile_shadow_unit",
+}
+_MULTI_UNIT_CHAIN_NUMERIC_VALUE_TOKENS = {
+ "amount",
+ "appraisal",
+ "appraised",
+ "cost",
+ "costs",
+ "dollar",
+ "dollars",
+ "minimum",
+ "paid",
+ "price",
+ "prices",
+ "sale",
+ "sell",
+ "sold",
+ "total",
+ "value",
+ "valued",
+ "values",
+ "worth",
+}
+
+
+def _multi_unit_chain_numeric_signal(
+ unit_kind: str,
+ facet_type: str,
+ text: str,
+ tokens: set[str],
+) -> float:
+ signal = 0.0
+ if unit_kind == "numeric_quantity" or facet_type == "numeric":
+ signal += 0.55
+ if tokens & _MULTI_UNIT_CHAIN_NUMERIC_VALUE_TOKENS:
+ signal += 0.28
+ if re.search(r"(?:[$€£¥]\s*\d|\b\d+(?:,\d{3})*(?:\.\d+)?\s*(?:dollars?|usd|bucks?|yuan|rmb|hours?|weeks?|days?|months?)\b)", text, flags=re.IGNORECASE):
+ signal += 0.42
+ return min(1.1, signal)
+
+
+def _multi_unit_chain_temporal_anchor_signal(text: str, tokens: set[str]) -> float:
+ signal = 0.0
+ if tokens & _UNIT_COVERAGE_TEMPORAL_TOKENS:
+ signal += 0.22
+ if re.search(
+ r"\b(?:about|around|roughly|few|several|couple|last|previous|earlier)\s+"
+ r"(?:a\s+)?(?:day|days|week|weeks|month|months|year|years)\s+ago\b|"
+ r"\b(?:yesterday|today|tomorrow|last\s+week|last\s+month|a\s+few\s+months\s+ago)\b",
+ text,
+ flags=re.IGNORECASE,
+ ):
+ signal += 0.72
+ return min(1.0, signal)
+
+
+_PROFILE_SHADOW_EVENTLIKE_SLOT_HINTS = {
+ "action",
+ "appointment",
+ "constraint",
+ "deadline",
+ "exchange",
+ "obligation",
+ "pickup",
+ "plan",
+ "preference",
+ "return",
+ "status",
+ "task",
+}
+_PROFILE_SHADOW_EVENTLIKE_TEXT_HINTS = {
+ "bought",
+ "buy",
+ "completed",
+ "did",
+ "exchange",
+ "exchanged",
+ "finish",
+ "finished",
+ "got",
+ "have to",
+ "need",
+ "needed",
+ "needs",
+ "paid",
+ "pick",
+ "picked",
+ "return",
+ "returned",
+ "should",
+ "still",
+ "took",
+ "went",
+}
+
+
+def _profile_shadow_eventlike_record(record: SessionMemoryRecordV2, metadata: Mapping[str, Any]) -> bool:
+ if _normalize(metadata.get("content_variant", "")) != "profile_shadow_from_writer":
+ return False
+ if record.state not in {"active", "parallel_active", "evidence"}:
+ return False
+ slot_text = " ".join(
+ [
+ _clean_text(record.category),
+ _clean_text(record.relation),
+ _clean_text(record.slot_key),
+ _clean_text(metadata.get("semantic_slot", "")),
+ _clean_text(metadata.get("profile_type", "")),
+ ]
+ ).lower()
+ value_text = " ".join(
+ [
+ _clean_text(record.value),
+ _clean_text(metadata.get("source_span", "")),
+ _clean_text(metadata.get("raw_text", "")),
+ ]
+ ).lower()
+ return bool(
+ any(hint in slot_text for hint in _PROFILE_SHADOW_EVENTLIKE_SLOT_HINTS)
+ or any(hint in value_text for hint in _PROFILE_SHADOW_EVENTLIKE_TEXT_HINTS)
+ )
+
+
+def _profile_shadow_unit_text(record: SessionMemoryRecordV2, metadata: Mapping[str, Any]) -> str:
+ return " ".join(
+ [
+ _clean_text(record.value),
+ _clean_text(record.category),
+ _clean_text(record.relation),
+ _clean_text(record.slot_key),
+ _clean_text(metadata.get("semantic_slot", "")),
+ _clean_text(metadata.get("profile_type", "")),
+ _clean_text(metadata.get("profile_domain", "")),
+ _clean_text(metadata.get("source_span", "")),
+ _clean_text(metadata.get("raw_text", "")),
+ ]
+ )
+
+
+def _unit_coverage_pack_hits(
+ graph: SessionMemoryGraphV2,
+ query: str,
+ final_hits: Sequence[MemoryHit],
+ *,
+ top_k: int,
+) -> Dict[str, Any]:
+ raw_query_tokens = set(_path_utility_tokens(query))
+ query_tokens = {token for token in raw_query_tokens if token not in _UNIT_COVERAGE_QUERY_DROP_TOKENS}
+ if not query_tokens:
+ return {"hits": list(final_hits), "metadata": {"unit_coverage_pack_enabled": False, "unit_coverage_reason": "empty_query"}}
+ count_intent = bool(query_tokens & _UNIT_COVERAGE_COUNT_TOKENS)
+ percentage_intent = bool(query_tokens & {"percent", "percentage"})
+ temporal_intent = bool(raw_query_tokens & _MULTI_UNIT_CHAIN_TEMPORAL_COMPARISON_TOKENS)
+ direct_unit_intent = not count_intent and not temporal_intent and len(query_tokens) >= 2
+ if not count_intent and not temporal_intent and not direct_unit_intent:
+ return {
+ "hits": list(final_hits),
+ "metadata": {"unit_coverage_pack_enabled": False, "unit_coverage_reason": "no_unit_intent"},
+ }
+
+ records = list(getattr(graph, "records_by_id", {}).values())
+ parent_by_slot = {_clean_text(record.slot_key).lower(): record for record in records}
+ candidates: List[tuple[float, SessionMemoryRecordV2, SessionMemoryRecordV2 | None, Dict[str, Any]]] = []
+ for record in records:
+ metadata = dict(record.metadata or {})
+ if record.state not in {"active", "parallel_active", "evidence"}:
+ continue
+ profile_shadow_unit = _profile_shadow_eventlike_record(record, metadata)
+ if profile_shadow_unit:
+ unit_kind = "profile_shadow_unit"
+ facet_type = "action"
+ parent_slot = _clean_text(record.slot_key).lower()
+ parent = None
+ parent_text = ""
+ unit_text = _profile_shadow_unit_text(record, metadata)
+ else:
+ if _normalize(metadata.get("content_variant", "")) != "event_facet_write":
+ continue
+ if _normalize(metadata.get("facet_layer_version", "")) not in {"event_unit_v1", "event_facet_v1"}:
+ continue
+ unit_kind = _normalize(metadata.get("unit_kind", ""))
+ facet_type = _normalize(metadata.get("facet_type", ""))
+ parent_slot = _clean_text(metadata.get("facet_parent_slot_key", "")).lower()
+ parent = parent_by_slot.get(parent_slot)
+ parent_text = " ".join(
+ [
+ _clean_text(parent.value if parent else ""),
+ _clean_text(dict(parent.metadata or {}).get("source_span", "") if parent else ""),
+ " ".join(parent.anchor_concepts if parent else []),
+ ]
+ )
+ unit_text = " ".join(
+ [
+ _clean_text(record.value),
+ _clean_text(metadata.get("facet_role", "")),
+ _clean_text(metadata.get("unit_kind", "")),
+ _clean_text(metadata.get("action", "")),
+ _clean_text(metadata.get("target", "")),
+ _clean_text(metadata.get("quantity", "")),
+ _clean_text(metadata.get("unit", "")),
+ _clean_text(metadata.get("normalized_time", "")),
+ _clean_text(metadata.get("status", "")),
+ _clean_text(metadata.get("facet_source_span", "")),
+ parent_text,
+ ]
+ )
+ unit_tokens = set(_path_utility_tokens(unit_text))
+ semantic_event_unit = _normalize(record.memory_id).startswith("tmcra.event.") or _normalize(parent_slot).startswith("tmcra.event.")
+ overlap = query_tokens & unit_tokens
+ score = 0.0
+ if overlap:
+ score += min(0.92, len(overlap) / max(1.0, len(query_tokens)) * 1.55)
+ if count_intent and facet_type in {"action", "entity", "numeric", "role", "evidence_role"}:
+ score += 0.42
+ if temporal_intent and facet_type in {"temporal", "action", "role", "entity"}:
+ score += 0.38
+ if direct_unit_intent and facet_type in {"action", "entity", "state", "numeric"}:
+ score += 0.30
+ if direct_unit_intent and len(overlap) >= 2:
+ score += 0.36
+ if unit_kind in {"action_unit", "target_entity", "leadership_role", "participation_role"}:
+ score += 0.16
+ if unit_kind in {"numeric_quantity", "temporal_anchor", "state_status"}:
+ score += 0.12
+ if profile_shadow_unit:
+ score += 0.28
+ if count_intent and facet_type in {"action", "entity"}:
+ score += 0.24
+ semantic_event_priority = bool(semantic_event_unit and (count_intent or percentage_intent))
+ if semantic_event_priority:
+ score += 0.46
+ if percentage_intent and (
+ unit_kind == "numeric_quantity"
+ or facet_type == "numeric"
+ or re.search(r"\b\d+(?:\.\d+)?\s*%|\b\d+(?:\.\d+)?\s*percent\b", unit_text, flags=re.IGNORECASE)
+ ):
+ score += 0.72
+ if parent is not None:
+ score += 0.08
+ if score < 0.55:
+ continue
+ candidates.append(
+ (
+ round(min(3.2, 1.28 + score), 6),
+ record,
+ parent,
+ {
+ "unit_coverage_overlap_tokens": sorted(overlap)[:14],
+ "unit_coverage_score": round(score, 6),
+ "unit_kind": unit_kind,
+ "facet_type": facet_type,
+ "unit_coverage_count_intent": bool(count_intent),
+ "unit_coverage_percentage_intent": bool(percentage_intent),
+ "unit_coverage_temporal_intent": bool(temporal_intent),
+ "unit_coverage_direct_unit_intent": bool(direct_unit_intent),
+ "unit_coverage_profile_shadow_unit": bool(profile_shadow_unit),
+ "unit_coverage_semantic_event_unit": bool(semantic_event_unit),
+ "unit_coverage_semantic_event_priority": bool(semantic_event_priority),
+ },
+ )
+ )
+
+ if not candidates:
+ return {
+ "hits": list(final_hits),
+ "metadata": {
+ "unit_coverage_pack_enabled": True,
+ "unit_coverage_candidate_count": 0,
+ "unit_coverage_inserted_hit_count": 0,
+ "unit_coverage_count_intent": bool(count_intent),
+ "unit_coverage_percentage_intent": bool(percentage_intent),
+ "unit_coverage_temporal_intent": bool(temporal_intent),
+ "unit_coverage_direct_unit_intent": bool(direct_unit_intent),
+ },
+ }
+ candidates.sort(
+ key=lambda item: (
+ 1 if bool(item[3].get("unit_coverage_semantic_event_priority", False)) else 0,
+ 1
+ if bool(item[3].get("unit_coverage_percentage_intent"))
+ and (
+ _normalize(item[3].get("unit_kind", "")) == "numeric_quantity"
+ or _normalize(item[3].get("facet_type", "")) == "numeric"
+ )
+ else 0,
+ float(item[0]),
+ int(item[1].turn_index),
+ ),
+ reverse=True,
+ )
+ selected: List[tuple[float, SessionMemoryRecordV2, SessionMemoryRecordV2 | None, Dict[str, Any]]] = []
+ seen_unit_values: set[str] = set()
+ try:
+ max_selected_units = max(1, int(os.getenv("TMCRA_UNIT_COVERAGE_PACK_MAX_UNITS", "6") or 6))
+ except (TypeError, ValueError):
+ max_selected_units = 6
+ for item in candidates:
+ _, record, parent, metadata = item
+ key = "|".join(
+ [
+ _normalize(metadata.get("unit_kind", "")),
+ _normalize(record.value),
+ _normalize(dict(record.metadata or {}).get("facet_parent_slot_key", "")),
+ ]
+ )
+ if key in seen_unit_values:
+ continue
+ seen_unit_values.add(key)
+ selected.append(item)
+ if len(selected) >= max_selected_units:
+ break
+ if count_intent:
+ try:
+ max_profile_shadow_units = max(0, int(os.getenv("TMCRA_UNIT_COVERAGE_PROFILE_SHADOW_MAX_UNITS", "2") or 2))
+ except (TypeError, ValueError):
+ max_profile_shadow_units = 2
+ selected_ids = {item[1].memory_id for item in selected}
+ added_profile_shadow_units = 0
+ for item in candidates:
+ _, record, _parent, metadata = item
+ if added_profile_shadow_units >= max_profile_shadow_units:
+ break
+ if record.memory_id in selected_ids:
+ continue
+ if not bool(metadata.get("unit_coverage_profile_shadow_unit", False)):
+ continue
+ selected.append(item)
+ selected_ids.add(record.memory_id)
+ added_profile_shadow_units += 1
+ packed_hits: List[MemoryHit] = []
+ for score, unit_record, parent, extra in selected:
+ packed_hits.append(
+ _memory_hit_from_record(
+ unit_record,
+ score=score,
+ metadata={
+ **extra,
+ "unit_coverage_pack": True,
+ "evidence_snippet_role": "unit_coverage_evidence_unit",
+ },
+ )
+ )
+ if parent is not None:
+ unit_metadata = dict(unit_record.metadata or {})
+ parent_hit = _memory_hit_from_record(
+ parent,
+ score=max(1.12, score - 0.06),
+ metadata={
+ **extra,
+ "unit_coverage_pack": True,
+ "evidence_snippet_role": "unit_coverage_parent_event",
+ "unit_coverage_parent_memory_id": parent.memory_id,
+ "unit_coverage_child_id": unit_record.memory_id,
+ "unit_coverage_child_value": unit_record.value,
+ "unit_coverage_child_source_span": _clean_text(
+ unit_metadata.get("facet_source_span", "") or unit_metadata.get("source_span", "")
+ ),
+ },
+ )
+ parent_hit.memory_id = f"{parent.memory_id}#unit_parent:{unit_record.memory_id}"
+ packed_hits.append(parent_hit)
+ merged: List[MemoryHit] = []
+ seen_ids: set[str] = set()
+ try:
+ insertion_index = max(0, min(len(final_hits), int(os.getenv("TMCRA_UNIT_COVERAGE_PACK_INSERT_AFTER", "2") or 2)))
+ except (TypeError, ValueError):
+ insertion_index = min(len(final_hits), 2)
+ ordered_hits = [*list(final_hits[:insertion_index]), *packed_hits, *list(final_hits[insertion_index:])]
+ for hit in ordered_hits:
+ if hit.memory_id and hit.memory_id in seen_ids:
+ continue
+ if hit.memory_id:
+ seen_ids.add(hit.memory_id)
+ merged.append(hit)
+ return {
+ "hits": merged,
+ "metadata": {
+ "unit_coverage_pack_enabled": True,
+ "unit_coverage_candidate_count": len(candidates),
+ "unit_coverage_selected_unit_count": len(selected),
+ "unit_coverage_inserted_hit_count": len(packed_hits),
+ "unit_coverage_count_intent": bool(count_intent),
+ "unit_coverage_percentage_intent": bool(percentage_intent),
+ "unit_coverage_temporal_intent": bool(temporal_intent),
+ "unit_coverage_direct_unit_intent": bool(direct_unit_intent),
+ },
+ }
+
+
+def _multi_unit_chain_focus_tokens(query: str) -> set[str]:
+ tokens = set(_path_utility_tokens(query))
+ return {
+ normalized
+ for token in tokens
+ for normalized in [_multi_unit_chain_normalize_token(token)]
+ if normalized and normalized not in _MULTI_UNIT_CHAIN_FOCUS_DROP_TOKENS and len(normalized) > 2
+ }
+
+
+def _multi_unit_chain_normalize_token(token: str) -> str:
+ raw = _normalize(token)
+ if not raw:
+ return ""
+ aliases = {
+ "bought": "buy",
+ "buying": "buy",
+ "purchased": "buy",
+ "purchasing": "buy",
+ "worked": "work",
+ "working": "work",
+ "started": "start",
+ "starting": "start",
+ "finished": "finish",
+ "finishing": "finish",
+ "completed": "complete",
+ "completing": "complete",
+ "returned": "return",
+ "returning": "return",
+ "met": "meet",
+ "meeting": "meet",
+ "picked": "pick",
+ "picking": "pick",
+ "led": "lead",
+ "leading": "lead",
+ "kits": "kit",
+ "items": "item",
+ "projects": "project",
+ "events": "event",
+ "clothes": "clothing",
+ }
+ if raw in aliases:
+ return aliases[raw]
+ if len(raw) > 4 and raw.endswith("ies"):
+ return raw[:-3] + "y"
+ if len(raw) > 4 and raw.endswith("ing"):
+ stem = raw[:-3]
+ if len(stem) > 3 and stem[-1] == stem[-2]:
+ stem = stem[:-1]
+ return stem
+ if len(raw) > 3 and raw.endswith("ed"):
+ stem = raw[:-2]
+ if len(stem) > 3 and stem[-1] == stem[-2]:
+ stem = stem[:-1]
+ return stem
+ if len(raw) > 3 and raw.endswith("s") and not raw.endswith("ss"):
+ return raw[:-1]
+ return raw
+
+
+def _multi_unit_chain_normalized_tokens(text: str) -> set[str]:
+ return {
+ normalized
+ for token in _path_utility_tokens(text)
+ for normalized in [_multi_unit_chain_normalize_token(token)]
+ if normalized
+ }
+
+
+def _multi_unit_chain_hit_text(record: SessionMemoryRecordV2, parent: SessionMemoryRecordV2 | None) -> str:
+ metadata = dict(record.metadata or {})
+ parent_metadata = dict(parent.metadata or {}) if parent is not None else {}
+ return " ".join(
+ [
+ _clean_text(record.value),
+ _clean_text(record.category),
+ _clean_text(record.relation),
+ _clean_text(metadata.get("facet_type", "")),
+ _clean_text(metadata.get("unit_kind", "")),
+ _clean_text(metadata.get("facet_role", "")),
+ _clean_text(metadata.get("facet_value", "")),
+ _clean_text(metadata.get("facet_source_span", "")),
+ _clean_text(metadata.get("action", "")),
+ _clean_text(metadata.get("target", "")),
+ _clean_text(metadata.get("quantity", "")),
+ _clean_text(metadata.get("status", "")),
+ _clean_text(parent.value if parent else ""),
+ _clean_text(parent_metadata.get("source_span", "")),
+ _clean_text(parent_metadata.get("raw_text", "")),
+ ]
+ )
+
+
+def _multi_unit_chain_local_hit_text(record: SessionMemoryRecordV2, parent: SessionMemoryRecordV2 | None) -> str:
+ metadata = dict(record.metadata or {})
+ return " ".join(
+ [
+ _clean_text(record.value),
+ _clean_text(record.category),
+ _clean_text(record.relation),
+ _clean_text(metadata.get("facet_type", "")),
+ _clean_text(metadata.get("unit_kind", "")),
+ _clean_text(metadata.get("facet_role", "")),
+ _clean_text(metadata.get("facet_value", "")),
+ _clean_text(metadata.get("facet_source_span", "")),
+ _clean_text(metadata.get("action", "")),
+ _clean_text(metadata.get("target", "")),
+ _clean_text(metadata.get("quantity", "")),
+ _clean_text(metadata.get("status", "")),
+ ]
+ )
+
+
+def _multi_unit_chain_slot_hits(
+ graph: SessionMemoryGraphV2,
+ query: str,
+ final_hits: Sequence[MemoryHit],
+ *,
+ top_k: int,
+) -> Dict[str, Any]:
+ mode = _normalize(os.getenv("TMCRA_MULTI_UNIT_CHAIN_SLOT_MODE", "on"))
+ if mode in _MULTI_UNIT_CHAIN_DISABLED_MODES:
+ return {
+ "hits": list(final_hits),
+ "metadata": {"multi_unit_chain_slot_enabled": False, "multi_unit_chain_slot_reason": "disabled"},
+ }
+ query_tokens = set(_path_utility_tokens(query))
+ focus_tokens = _multi_unit_chain_focus_tokens(query)
+ temporal_comparison_intent = bool(query_tokens & _MULTI_UNIT_CHAIN_TEMPORAL_COMPARISON_TOKENS) and len(focus_tokens) >= 2
+ aggregation_or_joiner_intent = bool(
+ re.search(r"\b(?:and|or|both|each|total|sum|minimum|maximum|amount|count|many|number)\b", str(query), flags=re.IGNORECASE)
+ )
+ numeric_aggregation_intent = bool(
+ query_tokens
+ & (
+ _MULTI_UNIT_CHAIN_NUMERIC_VALUE_TOKENS
+ | {"comment", "comments", "number", "percent", "percentage", "sum", "total"}
+ )
+ )
+ count_or_aggregation_intent = bool(query_tokens & _MULTI_UNIT_CHAIN_COUNT_TOKENS) and (aggregation_or_joiner_intent or len(focus_tokens) >= 3)
+ multi_intent = count_or_aggregation_intent or temporal_comparison_intent
+ if not query_tokens or not multi_intent:
+ return {
+ "hits": list(final_hits),
+ "metadata": {
+ "multi_unit_chain_slot_enabled": False,
+ "multi_unit_chain_slot_reason": "no_multi_intent",
+ },
+ }
+ if not focus_tokens:
+ return {
+ "hits": list(final_hits),
+ "metadata": {
+ "multi_unit_chain_slot_enabled": True,
+ "multi_unit_chain_slot_formed": False,
+ "multi_unit_chain_slot_reason": "no_focus_tokens",
+ },
+ }
+
+ records = list(getattr(graph, "records_by_id", {}).values())
+ parent_by_slot = {_clean_text(record.slot_key).lower(): record for record in records}
+ candidates: List[tuple[float, SessionMemoryRecordV2, SessionMemoryRecordV2 | None, Dict[str, Any]]] = []
+ for record in records:
+ metadata = dict(record.metadata or {})
+ if record.state not in {"active", "parallel_active", "evidence"}:
+ continue
+ profile_shadow_unit = _profile_shadow_eventlike_record(record, metadata)
+ semantic_event_record = bool(
+ _normalize(metadata.get("content_variant", "")) == "llm_semantic_write"
+ and (
+ _normalize(record.category) == "event"
+ or _normalize(record.source_kind).startswith("public_dialog_event")
+ )
+ )
+ if profile_shadow_unit:
+ facet_type = "action"
+ unit_kind = "profile_shadow_unit"
+ parent_slot = _clean_text(record.slot_key).lower()
+ parent = None
+ text = _profile_shadow_unit_text(record, metadata)
+ local_text = text
+ elif semantic_event_record:
+ facet_type = "action"
+ unit_kind = "semantic_event"
+ parent_slot = _clean_text(record.slot_key).lower()
+ parent = None
+ text = " ".join(
+ [
+ _clean_text(record.value),
+ _clean_text(record.category),
+ _clean_text(record.relation),
+ _clean_text(metadata.get("source_span", "")),
+ _clean_text(metadata.get("raw_text", "")),
+ ]
+ )
+ local_text = text
+ else:
+ if _normalize(metadata.get("content_variant", "")) != "event_facet_write":
+ continue
+ if _normalize(metadata.get("facet_layer_version", "")) != "event_unit_v1":
+ continue
+ facet_type = _normalize(metadata.get("facet_type", ""))
+ unit_kind = _normalize(metadata.get("unit_kind", ""))
+ if facet_type not in _MULTI_UNIT_CHAIN_FACET_TYPES and unit_kind not in _MULTI_UNIT_CHAIN_UNIT_KINDS:
+ continue
+ parent_slot = _clean_text(metadata.get("facet_parent_slot_key", "")).lower()
+ parent = parent_by_slot.get(parent_slot)
+ text = _multi_unit_chain_hit_text(record, parent)
+ local_text = _multi_unit_chain_local_hit_text(record, parent)
+ semantic_event_unit = _normalize(record.memory_id).startswith("tmcra.event.") or _normalize(parent_slot).startswith("tmcra.event.")
+ unit_tokens = _multi_unit_chain_normalized_tokens(text)
+ local_tokens = _multi_unit_chain_normalized_tokens(local_text)
+ focus_overlap = focus_tokens & unit_tokens
+ if not focus_overlap:
+ continue
+ score = min(1.2, len(focus_overlap) / max(1.0, len(focus_tokens)) * 1.6)
+ numeric_signal = _multi_unit_chain_numeric_signal(unit_kind, facet_type, text, unit_tokens)
+ temporal_anchor_signal = _multi_unit_chain_temporal_anchor_signal(text, unit_tokens)
+ local_focus_overlap = focus_tokens & local_tokens
+ local_temporal_anchor_signal = _multi_unit_chain_temporal_anchor_signal(local_text, local_tokens)
+ if not semantic_event_record and not profile_shadow_unit and not local_focus_overlap:
+ # A child unit should not enter a multi-unit chain only because its
+ # long parent window mentions the query topic. Otherwise assistant
+ # advice fragments under a relevant parent consume coverage slots.
+ if not (temporal_comparison_intent and local_temporal_anchor_signal > 0.0):
+ continue
+ if facet_type in {"action", "entity", "role", "numeric"}:
+ score += 0.34
+ if local_focus_overlap:
+ score += min(0.30, len(local_focus_overlap) * 0.10)
+ if unit_kind in {"action_unit", "target_entity", "leadership_role", "participation_role", "numeric_quantity"}:
+ score += 0.24
+ if profile_shadow_unit:
+ score += 0.30
+ semantic_event_priority = bool(semantic_event_unit and (count_or_aggregation_intent or numeric_aggregation_intent))
+ if semantic_event_priority:
+ score += 0.48
+ if numeric_signal and numeric_aggregation_intent:
+ score += numeric_signal
+ if count_or_aggregation_intent and not numeric_aggregation_intent and facet_type in {"action", "entity", "role"}:
+ score += 0.36
+ if count_or_aggregation_intent and not numeric_aggregation_intent and unit_kind in {"action_unit", "target_entity", "leadership_role", "participation_role"}:
+ score += 0.22
+ if parent is not None:
+ score += 0.12
+ if score < 0.72:
+ continue
+ candidates.append(
+ (
+ round(min(4.0, 2.15 + score), 6),
+ record,
+ parent,
+ {
+ "multi_unit_chain_focus_overlap_tokens": sorted(focus_overlap)[:16],
+ "multi_unit_chain_focus_overlap_count": len(focus_overlap),
+ "multi_unit_chain_local_focus_overlap_tokens": sorted(local_focus_overlap)[:16],
+ "multi_unit_chain_local_focus_overlap_count": len(local_focus_overlap),
+ "multi_unit_chain_score": round(score, 6),
+ "multi_unit_chain_numeric_signal": round(numeric_signal, 6),
+ "multi_unit_chain_temporal_anchor_signal": round(temporal_anchor_signal, 6),
+ "multi_unit_chain_local_temporal_anchor_signal": round(local_temporal_anchor_signal, 6),
+ "multi_unit_chain_temporal_comparison": bool(temporal_comparison_intent),
+ "multi_unit_chain_numeric_aggregation": bool(numeric_aggregation_intent),
+ "unit_kind": unit_kind,
+ "facet_type": facet_type,
+ "multi_unit_chain_parent_slot_key": parent_slot,
+ "multi_unit_chain_profile_shadow_unit": bool(profile_shadow_unit),
+ "multi_unit_chain_semantic_event_unit": bool(semantic_event_unit),
+ "multi_unit_chain_semantic_event_priority": bool(semantic_event_priority),
+ "multi_unit_chain_count_or_aggregation": bool(count_or_aggregation_intent),
+ },
+ )
+ )
+
+ if not candidates:
+ return {
+ "hits": list(final_hits),
+ "metadata": {
+ "multi_unit_chain_slot_enabled": True,
+ "multi_unit_chain_slot_formed": False,
+ "multi_unit_chain_slot_reason": "no_matching_units",
+ "multi_unit_chain_candidate_count": 0,
+ "multi_unit_chain_focus_tokens": sorted(focus_tokens)[:24],
+ },
+ }
+
+ if temporal_comparison_intent:
+ candidates.sort(
+ key=lambda item: (
+ float(item[3].get("multi_unit_chain_local_temporal_anchor_signal", 0.0) or 0.0),
+ float(item[3].get("multi_unit_chain_temporal_anchor_signal", 0.0) or 0.0),
+ 1 if bool(item[3].get("multi_unit_chain_semantic_event_priority", False)) else 0,
+ 0
+ if _normalize(item[3].get("unit_kind", "")) == "numeric_quantity"
+ or _normalize(item[3].get("facet_type", "")) == "numeric"
+ else 1,
+ float(item[3].get("multi_unit_chain_score", 0.0) or 0.0),
+ int(item[3].get("multi_unit_chain_focus_overlap_count", 0) or 0),
+ float(item[0]),
+ int(item[1].turn_index),
+ ),
+ reverse=True,
+ )
+ else:
+ candidates.sort(
+ key=lambda item: (
+ float(item[3].get("multi_unit_chain_numeric_signal", 0.0) or 0.0)
+ if bool(item[3].get("multi_unit_chain_numeric_aggregation", False))
+ else (
+ 1.2
+ if bool(item[3].get("multi_unit_chain_semantic_event_priority", False))
+ else
+ 0.0
+ if _normalize(item[3].get("unit_kind", "")) == "numeric_quantity"
+ or _normalize(item[3].get("facet_type", "")) == "numeric"
+ else 1.0
+ ),
+ float(item[3].get("multi_unit_chain_score", 0.0) or 0.0),
+ int(item[3].get("multi_unit_chain_focus_overlap_count", 0) or 0),
+ float(item[0]),
+ int(item[1].turn_index),
+ ),
+ reverse=True,
+ )
+ selected: List[tuple[float, SessionMemoryRecordV2, SessionMemoryRecordV2 | None, Dict[str, Any]]] = []
+ seen_parent_slots: set[str] = set()
+ seen_values: set[str] = set()
+ max_units = max(2, min(10, int(os.getenv("TMCRA_MULTI_UNIT_CHAIN_SLOT_MAX_UNITS", "8") or 8)))
+ for item in candidates:
+ _, record, parent, metadata = item
+ parent_slot = _clean_text(metadata.get("multi_unit_chain_parent_slot_key", ""))
+ value_key = "|".join(
+ [
+ _normalize(record.value),
+ _normalize(metadata.get("unit_kind", "")),
+ parent_slot,
+ ]
+ )
+ if value_key in seen_values:
+ continue
+ if parent_slot and parent_slot in seen_parent_slots:
+ # Keep the chain broad: one strongest unit per parent event first.
+ continue
+ seen_values.add(value_key)
+ if parent_slot:
+ seen_parent_slots.add(parent_slot)
+ selected.append(item)
+ if len(selected) >= max_units:
+ break
+ if count_or_aggregation_intent:
+ try:
+ max_profile_shadow_units = max(0, int(os.getenv("TMCRA_MULTI_UNIT_CHAIN_PROFILE_SHADOW_MAX_UNITS", "2") or 2))
+ except (TypeError, ValueError):
+ max_profile_shadow_units = 2
+ selected_ids = {item[1].memory_id for item in selected}
+ added_profile_shadow_units = 0
+ for item in candidates:
+ _, record, _parent, metadata = item
+ if added_profile_shadow_units >= max_profile_shadow_units:
+ break
+ if record.memory_id in selected_ids:
+ continue
+ if not bool(metadata.get("multi_unit_chain_profile_shadow_unit", False)):
+ continue
+ selected.append(item)
+ selected_ids.add(record.memory_id)
+ parent_slot = _clean_text(metadata.get("multi_unit_chain_parent_slot_key", ""))
+ if parent_slot:
+ seen_parent_slots.add(parent_slot)
+ added_profile_shadow_units += 1
+
+ min_parents = max(2, min(4, int(os.getenv("TMCRA_MULTI_UNIT_CHAIN_SLOT_MIN_PARENTS", "2") or 2)))
+ if len(seen_parent_slots) < min_parents:
+ return {
+ "hits": list(final_hits),
+ "metadata": {
+ "multi_unit_chain_slot_enabled": True,
+ "multi_unit_chain_slot_formed": False,
+ "multi_unit_chain_slot_reason": "insufficient_parent_coverage",
+ "multi_unit_chain_candidate_count": len(candidates),
+ "multi_unit_chain_selected_unit_count": len(selected),
+ "multi_unit_chain_parent_count": len(seen_parent_slots),
+ "multi_unit_chain_focus_tokens": sorted(focus_tokens)[:24],
+ },
+ }
+
+ chain_memory_ids: List[str] = []
+ packed_hits: List[MemoryHit] = []
+ for score, unit_record, parent, extra in selected:
+ chain_memory_ids.append(unit_record.memory_id)
+ packed_hits.append(
+ _memory_hit_from_record(
+ unit_record,
+ score=score,
+ metadata={
+ **extra,
+ "multi_unit_chain_slot": True,
+ "multi_unit_chain_bundle": True,
+ "evidence_snippet_role": "multi_unit_chain_evidence_unit",
+ },
+ )
+ )
+ if parent is not None:
+ unit_metadata = dict(unit_record.metadata or {})
+ chain_memory_ids.append(parent.memory_id)
+ parent_hit = _memory_hit_from_record(
+ parent,
+ score=max(1.9, score - 0.08),
+ metadata={
+ **extra,
+ "multi_unit_chain_slot": True,
+ "multi_unit_chain_bundle": True,
+ "evidence_snippet_role": "multi_unit_chain_parent_event",
+ "multi_unit_chain_child_id": unit_record.memory_id,
+ "multi_unit_chain_child_value": unit_record.value,
+ "multi_unit_chain_child_source_span": _clean_text(
+ unit_metadata.get("facet_source_span", "") or unit_metadata.get("source_span", "")
+ ),
+ },
+ )
+ parent_hit.memory_id = f"{parent.memory_id}#multi_parent:{unit_record.memory_id}"
+ packed_hits.append(parent_hit)
+
+ seen_ids: set[str] = set()
+ merged: List[MemoryHit] = []
+ insertion_index = max(0, min(len(final_hits), int(os.getenv("TMCRA_MULTI_UNIT_CHAIN_SLOT_INSERT_AFTER", "1") or 1)))
+ ordered = [*list(final_hits[:insertion_index]), *packed_hits, *list(final_hits[insertion_index:])]
+ for hit in ordered:
+ if hit.memory_id and hit.memory_id in seen_ids:
+ continue
+ if hit.memory_id:
+ seen_ids.add(hit.memory_id)
+ merged.append(hit)
+ return {
+ "hits": merged,
+ "metadata": {
+ "multi_unit_chain_slot_enabled": True,
+ "multi_unit_chain_slot_formed": True,
+ "multi_unit_chain_slot_reason": "formed",
+ "multi_unit_chain_candidate_count": len(candidates),
+ "multi_unit_chain_selected_unit_count": len(selected),
+ "multi_unit_chain_parent_count": len(seen_parent_slots),
+ "multi_unit_chain_inserted_hit_count": len(packed_hits),
+ "multi_unit_chain_memory_ids": _dedupe(chain_memory_ids, max_items=32),
+ "multi_unit_chain_focus_tokens": sorted(focus_tokens)[:24],
+ },
+ }
+
+
+def _profile_support_source_hits(
+ graph: SessionMemoryGraphV2,
+ profile_hit: MemoryHit,
+ *,
+ grouped_hits: Mapping[str, Sequence[MemoryHit]],
+ query: str,
+ limit: int,
+) -> List[MemoryHit]:
+ metadata = dict(profile_hit.metadata or {})
+ support_ids = _dedupe(
+ [
+ *list(metadata.get("profile_support_ids", []) or []),
+ *list(metadata.get("support_memory_ids", []) or []),
+ ],
+ max_items=24,
+ )
+ if not support_ids:
+ return []
+ query_raw_tokens = set(_path_utility_tokens(query))
+ query_tokens = _profile_query_expanded_tokens(query)
+ profile_event_id = _clean_text(metadata.get("profile_first_hybrid_event_id", "")) or _runtime_event_key(profile_hit)
+ candidates: List[tuple[float, MemoryHit]] = []
+ for support_id in support_ids:
+ record = getattr(graph, "records_by_id", {}).get(support_id)
+ if record is None or record.state not in {"active", "parallel_active", "evidence"}:
+ continue
+ support_hit = _memory_hit_from_record(record)
+ event_id = _runtime_event_key(support_hit)
+ if not event_id or event_id == profile_event_id or event_id not in grouped_hits:
+ continue
+ match_score, overlap_tokens, raw_overlap_tokens = _profile_hit_match_score(query_raw_tokens, query_tokens, support_hit)
+ source_score = max(0.80, float(profile_hit.score) - 0.02) + min(0.28, match_score * 0.12)
+ support_metadata = dict(support_hit.metadata or {})
+ profile_metadata = dict(profile_hit.metadata or {})
+ support_metadata.update(
+ {
+ "profile_first_hybrid_rescue": True,
+ "profile_first_source_support": True,
+ "profile_first_parent_memory_id": profile_hit.memory_id,
+ "profile_first_hybrid_event_id": event_id,
+ "profile_first_parent_summary": _clean_text(profile_metadata.get("profile_summary", "")) or _clean_text(profile_metadata.get("profile_value", "")) or _clean_text(profile_hit.value),
+ "profile_type": _clean_text(support_metadata.get("profile_type", "")) or _clean_text(profile_metadata.get("profile_type", "")),
+ "profile_domain": _clean_text(support_metadata.get("profile_domain", "")) or _clean_text(profile_metadata.get("profile_domain", "")),
+ "profile_domain_label": _clean_text(support_metadata.get("profile_domain_label", "")) or _clean_text(profile_metadata.get("profile_domain_label", "")),
+ "profile_value": _clean_text(support_metadata.get("profile_value", "")) or _clean_text(profile_metadata.get("profile_value", "")) or _clean_text(profile_hit.value),
+ "profile_summary": _clean_text(support_metadata.get("profile_summary", "")) or _clean_text(profile_metadata.get("profile_summary", "")) or _clean_text(profile_hit.value),
+ "profile_query_match_score": round(match_score, 6),
+ "profile_query_overlap_tokens": list(overlap_tokens),
+ "profile_query_raw_overlap_tokens": list(raw_overlap_tokens),
+ "profile_source_pack_role": "source_event_support",
+ }
+ )
+ candidates.append(
+ (
+ source_score,
+ MemoryHit(
+ memory_id=support_hit.memory_id,
+ category=support_hit.category,
+ value=support_hit.value,
+ relation=support_hit.relation,
+ anchors=list(support_hit.anchors),
+ score=round(source_score, 6),
+ source_kind=support_hit.source_kind,
+ slot_key=support_hit.slot_key,
+ state=support_hit.state,
+ turn_index=int(support_hit.turn_index),
+ metadata=support_metadata,
+ ),
+ )
+ )
+ candidates.sort(
+ key=lambda item: (
+ bool((item[1].metadata or {}).get("profile_query_raw_overlap_tokens")),
+ float(item[0]),
+ int(item[1].turn_index),
+ ),
+ reverse=True,
+ )
+ return [hit for _, hit in candidates[: max(0, int(limit))]]
+
+
+def _profile_same_event_source_hit(profile_hit: MemoryHit, group_hits: Sequence[MemoryHit], *, event_id: str) -> MemoryHit | None:
+ source_hit = _support_hit_for_path("speaker_event_source_turn", group_hits) or _representative_event_hit(group_hits)
+ if source_hit is None or source_hit.memory_id == profile_hit.memory_id:
+ return None
+ metadata = dict(source_hit.metadata or {})
+ profile_metadata = dict(profile_hit.metadata or {})
+ metadata.update(
+ {
+ "profile_first_hybrid_rescue": True,
+ "profile_first_source_support": True,
+ "profile_first_same_event_support": True,
+ "profile_first_parent_memory_id": profile_hit.memory_id,
+ "profile_first_hybrid_event_id": event_id,
+ "profile_first_parent_summary": _clean_text(profile_metadata.get("profile_summary", "")) or _clean_text(profile_metadata.get("profile_value", "")) or _clean_text(profile_hit.value),
+ "profile_type": _clean_text(metadata.get("profile_type", "")) or _clean_text(profile_metadata.get("profile_type", "")),
+ "profile_domain": _clean_text(metadata.get("profile_domain", "")) or _clean_text(profile_metadata.get("profile_domain", "")),
+ "profile_domain_label": _clean_text(metadata.get("profile_domain_label", "")) or _clean_text(profile_metadata.get("profile_domain_label", "")),
+ "profile_value": _clean_text(metadata.get("profile_value", "")) or _clean_text(profile_metadata.get("profile_value", "")) or _clean_text(profile_hit.value),
+ "profile_summary": _clean_text(metadata.get("profile_summary", "")) or _clean_text(profile_metadata.get("profile_summary", "")) or _clean_text(profile_hit.value),
+ "profile_source_pack_role": "same_event_source_turn",
+ "evidence_snippet_role": "profile_source_support",
+ }
+ )
+ return MemoryHit(
+ memory_id=source_hit.memory_id,
+ category=source_hit.category,
+ value=source_hit.value,
+ relation=source_hit.relation,
+ anchors=list(source_hit.anchors),
+ score=max(float(source_hit.score), float(profile_hit.score) - 0.01),
+ source_kind=source_hit.source_kind,
+ slot_key=source_hit.slot_key,
+ state=source_hit.state,
+ turn_index=int(source_hit.turn_index),
+ metadata=metadata,
+ )
+
+
+def _profile_first_hybrid_rescue(
+ graph: SessionMemoryGraphV2,
+ query: str,
+ *,
+ grouped_hits: Mapping[str, Sequence[MemoryHit]],
+ top_k: int,
+) -> Dict[str, Any]:
+ intent = infer_profile_query_intent(query)
+ if not bool(intent.get("enabled")):
+ return {"hits": [], "event_ids": []}
+ raw_hits = _profile_query_rescue_hits(graph, query, top_k=max(4, int(top_k or 1) * 3))
+ limit = max(2, min(4, int(top_k or 1)))
+ selected_hits: List[MemoryHit] = []
+ selected_event_ids: List[str] = []
+ seen_event_ids: set[str] = set()
+ for rank, hit in enumerate(raw_hits, start=1):
+ event_id = _runtime_event_key(hit)
+ if not event_id or event_id not in grouped_hits or event_id in seen_event_ids:
+ continue
+ metadata = dict(hit.metadata or {})
+ metadata.update(
+ {
+ "profile_first_hybrid_rescue": True,
+ "profile_first_hybrid_rank": rank,
+ "profile_first_hybrid_event_id": event_id,
+ }
+ )
+ hit.metadata = metadata
+ hit.score = round(max(float(hit.score), 0.88) + max(0.0, 0.08 - (0.01 * len(selected_hits))), 6)
+ selected_hits.append(hit)
+ selected_event_ids.append(event_id)
+ seen_event_ids.add(event_id)
+ same_event_source = _profile_same_event_source_hit(hit, grouped_hits.get(event_id, []), event_id=event_id)
+ if same_event_source is not None:
+ selected_hits.append(same_event_source)
+ if len(selected_hits) >= limit:
+ break
+ support_hits = _profile_support_source_hits(
+ graph,
+ hit,
+ grouped_hits=grouped_hits,
+ query=query,
+ limit=max(1, min(3, limit - len(selected_hits) + 1)),
+ )
+ for support_hit in support_hits:
+ support_event_id = _clean_text((support_hit.metadata or {}).get("profile_first_hybrid_event_id", "")) or _runtime_event_key(support_hit)
+ if not support_event_id or support_event_id in seen_event_ids:
+ continue
+ selected_hits.append(support_hit)
+ selected_event_ids.append(support_event_id)
+ seen_event_ids.add(support_event_id)
+ if len(selected_hits) >= limit:
+ break
+ if len(selected_hits) >= limit:
+ break
+ return {
+ "hits": selected_hits,
+ "event_ids": selected_event_ids,
+ "memory_ids": [hit.memory_id for hit in selected_hits],
+ }
+
+
+def _inject_profile_first_hits(
+ final_hits: Sequence[MemoryHit],
+ profile_first_hits: Sequence[MemoryHit],
+ *,
+ selected_event_ids: Sequence[str],
+ selected_path_ids: Sequence[str],
+) -> List[MemoryHit]:
+ merged: List[MemoryHit] = []
+ seen_memory_ids: set[str] = set()
+ for hit in list(profile_first_hits) + list(final_hits):
+ if not hit or not hit.memory_id or hit.memory_id in seen_memory_ids:
+ continue
+ metadata = dict(hit.metadata or {})
+ if bool(metadata.get("profile_first_hybrid_rescue")):
+ event_id = _clean_text(metadata.get("profile_first_hybrid_event_id", "")) or _runtime_event_key(hit)
+ metadata.update(
+ {
+ "event_id": event_id,
+ "path_id": "",
+ "evidence_snippet_role": "selected_event_representative",
+ "hybrid_score_source": "profile_first_hybrid_rescue",
+ "selected_event_ids": list(selected_event_ids),
+ "selected_path_ids": list(selected_path_ids),
+ }
+ )
+ hit = MemoryHit(
+ memory_id=hit.memory_id,
+ category=hit.category,
+ value=hit.value,
+ relation=hit.relation,
+ anchors=list(hit.anchors),
+ score=float(hit.score),
+ source_kind=hit.source_kind,
+ slot_key=hit.slot_key,
+ state=hit.state,
+ turn_index=int(hit.turn_index),
+ metadata=metadata,
+ )
+ merged.append(hit)
+ seen_memory_ids.add(hit.memory_id)
+ return merged
+
+
+def _profile_focused_pack_hits(
+ graph: SessionMemoryGraphV2,
+ query: str,
+ final_hits: Sequence[MemoryHit],
+ *,
+ top_k: int,
+ grouped_hits: Mapping[str, Sequence[MemoryHit]],
+) -> Dict[str, Any]:
+ intent = infer_profile_query_intent(query)
+ if not bool(intent.get("enabled")):
+ return {
+ "hits": list(final_hits),
+ "metadata": {
+ "profile_focused_pack_enabled": False,
+ "profile_focused_pack_reason": "profile_intent_not_requested",
+ },
+ }
+ resolved_grouped_hits = dict(grouped_hits or {})
+ if not resolved_grouped_hits:
+ return {
+ "hits": list(final_hits),
+ "metadata": {
+ "profile_focused_pack_enabled": False,
+ "profile_focused_pack_reason": "no_learnable_hits",
+ },
+ }
+ profile_first_payload = _profile_first_hybrid_rescue(
+ graph,
+ query,
+ grouped_hits=resolved_grouped_hits,
+ top_k=max(2, min(max(1, int(top_k or 1)), 4)),
+ )
+ profile_first_hits = list(profile_first_payload.get("hits", []) or [])
+ profile_first_event_ids = list(profile_first_payload.get("event_ids", []) or [])
+ profile_first_memory_ids = list(profile_first_payload.get("memory_ids", []) or [])
+ if not profile_first_hits:
+ return {
+ "hits": list(final_hits),
+ "metadata": {
+ "profile_focused_pack_enabled": True,
+ "profile_focused_pack_reason": "no_profile_hits",
+ "profile_focused_pack_event_ids": [],
+ "profile_focused_pack_memory_ids": [],
+ },
+ }
+ selected_event_ids = _dedupe([*profile_first_event_ids, *_event_ids_from_hits(final_hits)], max_items=max(1, int(top_k or 1) * 2))
+ selected_path_ids = _dedupe(
+ [
+ _clean_text((hit.metadata or {}).get("path_id", ""))
+ for hit in final_hits
+ if _clean_text((hit.metadata or {}).get("path_id", ""))
+ ],
+ max_items=max(1, int(top_k or 1)),
+ )
+ merged_hits = _inject_profile_first_hits(
+ final_hits,
+ profile_first_hits,
+ selected_event_ids=selected_event_ids,
+ selected_path_ids=selected_path_ids,
+ )
+ merged_hits = _coverage_preserving_final_hits(merged_hits, selected_event_ids=selected_event_ids, top_k=top_k)
+ return {
+ "hits": merged_hits,
+ "metadata": {
+ "profile_focused_pack_enabled": True,
+ "profile_focused_pack_reason": "profile_first_pack_injected",
+ "profile_focused_pack_event_ids": list(profile_first_event_ids),
+ "profile_focused_pack_memory_ids": list(profile_first_memory_ids),
+ "profile_focused_pack_hit_count": len(profile_first_hits),
+ "profile_first_hybrid_enabled": True,
+ "profile_first_event_ids": list(profile_first_event_ids),
+ "profile_first_memory_ids": list(profile_first_memory_ids),
+ },
+ }
+
+
+def _topic_bucket_rerank_hits(
+ graph: SessionMemoryGraphV2,
+ query: str,
+ hits: Sequence[MemoryHit],
+ *,
+ top_k: int,
+) -> Dict[str, Any]:
+ if not hits:
+ profile_rescue_hits = _profile_query_rescue_hits(graph, query, top_k=top_k)
+ if profile_rescue_hits:
+ return {
+ "hits": profile_rescue_hits[: max(1, int(top_k or 1))],
+ "metadata": {
+ "topic_bucket_rerank_enabled": True,
+ "topic_bucket_rerank_reason": "profile_query_rescue_from_empty_hits",
+ "topic_bucket_profile_query_rescue_count": len(profile_rescue_hits),
+ },
+ }
+ return {
+ "hits": [],
+ "metadata": {"topic_bucket_rerank_enabled": True, "topic_bucket_rerank_reason": "no_hits"},
+ }
+ query_topic = _assign_topic_bucket_for_text(graph, query, turn_index=0, create=False)
+ query_bucket_id = _clean_text(query_topic.get("topic_bucket_id", ""))
+ adjacent_ids = _topic_adjacent_bucket_ids(graph, query_bucket_id)
+ dialogue_tunnel_ids = _dialogue_tunnel_bucket_ids(graph, query_bucket_id)
+ query_keywords = list(query_topic.get("topic_keywords", []) or [])
+ bridge_requested = _topic_bridge_requested(query)
+ dialogue_requested = _dialogue_tunnel_requested(query)
+ profile_query_requested = bool(infer_profile_query_intent(query).get("enabled"))
+ reranked: List[MemoryHit] = []
+ stats = {
+ "same_bucket": 0,
+ "bridge_bucket": 0,
+ "blocked_bridge_bucket": 0,
+ "dialogue_tunnel_bucket": 0,
+ "blocked_dialogue_tunnel_bucket": 0,
+ "profile_route_preserved": 0,
+ "profile_query_rescue": 0,
+ "overlap_bucket": 0,
+ "off_topic": 0,
+ }
+ for hit in list(hits):
+ metadata = dict(hit.metadata or {})
+ hit_bucket_id = _clean_text(metadata.get("topic_bucket_id", ""))
+ hit_keywords = _dedupe(metadata.get("topic_keywords", []) or [], max_items=32)
+ overlap = _topic_bucket_overlap_score(query_keywords, hit_keywords)
+ same_bucket = bool(query_bucket_id and hit_bucket_id and query_bucket_id == hit_bucket_id)
+ bridge_bucket = bool(hit_bucket_id and hit_bucket_id in adjacent_ids)
+ dialogue_tunnel_bucket = bool(hit_bucket_id and hit_bucket_id in dialogue_tunnel_ids)
+ overlap_bucket = overlap >= 0.22
+ bridge_allowed = bridge_bucket and bridge_requested
+ dialogue_allowed = dialogue_tunnel_bucket and dialogue_requested
+ profile_route_preserved = bool(
+ profile_query_requested
+ and (
+ bool(metadata.get("profile_layer"))
+ or "profile_route" in _normalize(metadata.get("match_reason", ""))
+ )
+ )
+ current_subject_preserved = bool(metadata.get("current_subject_resolver") or metadata.get("public_subject_match"))
+ delta = 0.0
+ if same_bucket:
+ delta += 1.35
+ stats["same_bucket"] += 1
+ elif current_subject_preserved:
+ delta += 0.18
+ stats["profile_route_preserved"] += 1
+ elif bridge_allowed:
+ delta += 0.42
+ stats["bridge_bucket"] += 1
+ elif dialogue_allowed:
+ delta += 0.16
+ stats["dialogue_tunnel_bucket"] += 1
+ elif overlap_bucket:
+ delta += 0.28 + overlap
+ stats["overlap_bucket"] += 1
+ elif bridge_bucket:
+ delta -= 1.05
+ stats["blocked_bridge_bucket"] += 1
+ elif dialogue_tunnel_bucket:
+ delta -= 1.35
+ stats["blocked_dialogue_tunnel_bucket"] += 1
+ elif profile_route_preserved:
+ delta += 0.08
+ stats["profile_route_preserved"] += 1
+ else:
+ delta -= 0.72
+ stats["off_topic"] += 1
+ memory_type = _normalize(metadata.get("memory_type", ""))
+ durability = _normalize(metadata.get("durability", ""))
+ conflict_policy = _normalize(metadata.get("conflict_policy", ""))
+ if (memory_type == "hard_constraint" or durability == "hard" or conflict_policy == "must_preserve") and (
+ same_bucket or bridge_allowed or dialogue_allowed or overlap_bucket
+ ):
+ delta += 1.10
+ match_reason = _clean_text(metadata.get("match_reason", ""))
+ if profile_route_preserved and "profile_route" not in _normalize(match_reason):
+ match_reason = ",".join(_dedupe([match_reason, "profile_route"], max_items=4))
+ metadata.update(
+ {
+ "topic_bucket_rerank": True,
+ "topic_bucket_query_id": query_bucket_id,
+ "topic_bucket_query_label": _clean_text(query_topic.get("topic_label", "")),
+ "topic_bucket_overlap": round(overlap, 6),
+ "topic_bucket_delta": round(delta, 6),
+ "topic_bucket_same": same_bucket,
+ "topic_bucket_bridge": bridge_bucket,
+ "topic_bucket_bridge_allowed": bridge_allowed,
+ "topic_bucket_dialogue_tunnel": dialogue_tunnel_bucket,
+ "topic_bucket_dialogue_tunnel_allowed": dialogue_allowed,
+ "topic_bucket_profile_route_preserved": profile_route_preserved,
+ "topic_bucket_current_subject_preserved": current_subject_preserved,
+ "match_reason": match_reason,
+ }
+ )
+ hit.metadata = metadata
+ hit.score = float(hit.score) + delta
+ reranked.append(hit)
+ seen_ids = {hit.memory_id for hit in reranked if hit.memory_id}
+ rescue_records = [
+ record
+ for record in getattr(graph, "records_by_id", {}).values()
+ if record.memory_id not in seen_ids
+ and record.state == "active"
+ and _clean_text((record.metadata or {}).get("topic_bucket_id", "")) == query_bucket_id
+ and _normalize(record.category) != "question"
+ ]
+ rescue_records.sort(
+ key=lambda record: (
+ int(any(marker in _normalize(record.value) for marker in ("过敏", "必须", "避开", "禁止", "不能", "must", "avoid", "allergy"))),
+ int(_normalize(record.category) in {"constraint", "preference"}),
+ float(record.confidence),
+ float(record.salience),
+ -int(record.turn_index),
+ ),
+ reverse=True,
+ )
+ rescue_hits = [
+ _topic_bucket_record_to_hit(record, query_topic=query_topic, rank=index)
+ for index, record in enumerate(rescue_records[: max(2, min(12, int(top_k or 1) * 2))], start=1)
+ ]
+ reranked.extend(rescue_hits)
+ dialogue_rescue_hits: List[MemoryHit] = []
+ if dialogue_requested and dialogue_tunnel_ids:
+ seen_ids.update(hit.memory_id for hit in rescue_hits if hit.memory_id)
+ dialogue_records = [
+ record
+ for record in getattr(graph, "records_by_id", {}).values()
+ if record.memory_id not in seen_ids
+ and record.state == "active"
+ and _clean_text((record.metadata or {}).get("topic_bucket_id", "")) in dialogue_tunnel_ids
+ and _normalize(record.category) != "question"
+ ]
+ def dialogue_record_rank(record: SessionMemoryRecordV2) -> tuple[int, int, float, float, int]:
+ return (
+ int(any(marker in _normalize(record.value) for marker in ("过敏", "必须", "避开", "禁止", "不能", "must", "avoid", "allergy"))),
+ int(_normalize(record.category) in {"constraint", "preference"}),
+ float(record.confidence),
+ float(record.salience),
+ -int(record.turn_index),
+ )
+
+ dialogue_records.sort(key=dialogue_record_rank, reverse=True)
+ selected_dialogue_records: List[SessionMemoryRecordV2] = []
+ selected_dialogue_ids: set[str] = set()
+ for bucket_id in sorted(dialogue_tunnel_ids):
+ bucket_records = [
+ record
+ for record in dialogue_records
+ if _clean_text((record.metadata or {}).get("topic_bucket_id", "")) == bucket_id
+ ]
+ if not bucket_records:
+ continue
+ best = bucket_records[0]
+ selected_dialogue_records.append(best)
+ selected_dialogue_ids.add(best.memory_id)
+ target_dialogue_rescue = max(len(selected_dialogue_records), max(1, min(4, int(top_k or 1) // 2 or 1)))
+ for record in dialogue_records:
+ if len(selected_dialogue_records) >= target_dialogue_rescue:
+ break
+ if record.memory_id in selected_dialogue_ids:
+ continue
+ selected_dialogue_records.append(record)
+ selected_dialogue_ids.add(record.memory_id)
+ dialogue_rescue_hits = [
+ _topic_bucket_record_to_hit(record, query_topic=query_topic, rank=index, rescue_kind="dialogue_tunnel")
+ for index, record in enumerate(selected_dialogue_records, start=1)
+ ]
+ reranked.extend(dialogue_rescue_hits)
+ profile_rescue_hits: List[MemoryHit] = []
+ if profile_query_requested:
+ seen_ids.update(hit.memory_id for hit in reranked if hit.memory_id)
+ for index, hit in enumerate(_profile_query_rescue_hits(graph, query, top_k=top_k), start=1):
+ if hit.memory_id in seen_ids:
+ continue
+ metadata = dict(hit.metadata or {})
+ metadata.update(
+ {
+ "profile_query_rescue_rank": index,
+ "topic_bucket_query_id": query_bucket_id,
+ "topic_bucket_query_label": _clean_text(query_topic.get("topic_label", "")),
+ "topic_bucket_profile_route_preserved": True,
+ }
+ )
+ hit.metadata = metadata
+ hit.score = float(hit.score) + 0.08
+ profile_rescue_hits.append(hit)
+ seen_ids.add(hit.memory_id)
+ stats["profile_query_rescue"] = len(profile_rescue_hits)
+ reranked.extend(profile_rescue_hits)
+ reranked.sort(key=lambda item: (float(item.score), int(item.turn_index)), reverse=True)
+ limit = max(1, int(top_k or 1))
+ focused_hits = [
+ hit
+ for hit in reranked
+ if bool((hit.metadata or {}).get("topic_bucket_same"))
+ or float((hit.metadata or {}).get("topic_bucket_overlap", 0.0) or 0.0) >= 0.22
+ or bool((hit.metadata or {}).get("topic_bucket_bridge_allowed"))
+ or bool((hit.metadata or {}).get("topic_bucket_dialogue_tunnel_allowed"))
+ or bool((hit.metadata or {}).get("topic_bucket_profile_route_preserved"))
+ or bool((hit.metadata or {}).get("topic_bucket_current_subject_preserved"))
+ or bool((hit.metadata or {}).get("current_subject_resolver"))
+ ]
+ model_path_fallback = False
+ no_bucket_model_fallback = False
+ if not focused_hits:
+ model_supported_hits = [
+ hit
+ for hit in reranked
+ if _clean_text((hit.metadata or {}).get("path_id", ""))
+ or _clean_text((hit.metadata or {}).get("hybrid_score_source", ""))
+ ]
+ if model_supported_hits:
+ focused_hits = model_supported_hits
+ model_path_fallback = True
+ generic_memory_fallback = False
+ if not focused_hits and any(not _is_public_dialog_hit(hit) for hit in reranked):
+ focused_hits = list(reranked)
+ generic_memory_fallback = True
+ if not focused_hits and not query_bucket_id:
+ focused_hits = list(reranked)
+ no_bucket_model_fallback = True
+ time_focused_model_path = any(
+ "speaker_event_time" in _clean_text((hit.metadata or {}).get("path_id", ""))
+ or _clean_text((hit.metadata or {}).get("model_focused_answer_type", "")) == "time"
+ for hit in focused_hits
+ )
+ if time_focused_model_path:
+ focused_hits = [
+ hit
+ for hit in focused_hits
+ if not (
+ _clean_text(hit.source_kind) == "public_dialog_profile"
+ and not _clean_text((hit.metadata or {}).get("path_id", ""))
+ )
+ ]
+ filtered_count = max(0, len(reranked) - len(focused_hits))
+ query_subject_signature = _public_subject_signature(_public_query_subject(query))
+
+ def _hit_subject_signatures(hit: MemoryHit) -> set[str]:
+ metadata = dict(hit.metadata or {})
+ signatures = {
+ _normalize(metadata.get("subject_signature", "")).replace("-", "_"),
+ _public_subject_signature(metadata.get("subject", "")),
+ }
+ canonical_slot_key = _clean_text(metadata.get("canonical_slot_key", ""))
+ if ".subject." in canonical_slot_key:
+ signatures.add(_public_subject_signature(canonical_slot_key.split(".subject.", 1)[-1]))
+ if ".subject." in hit.slot_key:
+ signatures.add(_public_subject_signature(hit.slot_key.split(".subject.", 1)[-1]))
+ signatures.discard("")
+ return signatures
+
+ protected_current_hits = [
+ hit
+ for hit in focused_hits
+ if bool((hit.metadata or {}).get("current_subject_resolver"))
+ ]
+ protected_model_path_hits = [
+ hit
+ for hit in focused_hits
+ if _clean_text((hit.metadata or {}).get("path_id", ""))
+ and _clean_text((hit.metadata or {}).get("hybrid_score_source", ""))
+ ]
+ protected_generic_hits = [
+ hit
+ for hit in focused_hits
+ if not _is_public_dialog_hit(hit)
+ and not bool((hit.metadata or {}).get("profile_layer"))
+ ]
+ if protected_current_hits and query_subject_signature:
+ protected_ids = {hit.memory_id for hit in protected_current_hits if hit.memory_id}
+ compacted_focused_hits: List[MemoryHit] = []
+ for hit in focused_hits:
+ if hit.memory_id in protected_ids:
+ compacted_focused_hits.append(hit)
+ continue
+ same_current_subject = query_subject_signature in _hit_subject_signatures(hit)
+ inactive_state = _normalize(hit.state) in {"superseded", "evidence", "historical", "stale", "false"}
+ if same_current_subject and inactive_state:
+ continue
+ compacted_focused_hits.append(hit)
+ focused_hits = compacted_focused_hits
+ filtered_count = max(0, len(reranked) - len(focused_hits))
+
+ selected_hits: List[MemoryHit] = []
+ selected_keys: set[str] = set()
+ for hit in [*protected_current_hits, *protected_model_path_hits, *protected_generic_hits, *focused_hits]:
+ key = hit.memory_id or f"{_hit_event_id(hit)}::{hit.slot_key}::{hit.value[:80]}"
+ if key in selected_keys:
+ continue
+ selected_hits.append(hit)
+ selected_keys.add(key)
+ if len(selected_hits) >= limit:
+ break
+ dialogue_reserved_count = 0
+ if dialogue_requested and dialogue_tunnel_ids:
+ selected_ids = {hit.memory_id for hit in selected_hits if hit.memory_id}
+ selected_bucket_ids = {
+ _clean_text((hit.metadata or {}).get("topic_bucket_id", ""))
+ for hit in selected_hits
+ if _clean_text((hit.metadata or {}).get("topic_bucket_id", ""))
+ }
+ for bucket_id in sorted(dialogue_tunnel_ids):
+ if bucket_id in selected_bucket_ids:
+ continue
+ candidate = next(
+ (
+ hit
+ for hit in focused_hits
+ if hit.memory_id not in selected_ids
+ and _clean_text((hit.metadata or {}).get("topic_bucket_id", "")) == bucket_id
+ and bool((hit.metadata or {}).get("topic_bucket_dialogue_tunnel_allowed"))
+ ),
+ None,
+ )
+ if candidate is None:
+ continue
+ if len(selected_hits) < limit:
+ selected_hits.append(candidate)
+ else:
+ replace_index = len(selected_hits) - 1
+ for index in range(len(selected_hits) - 1, -1, -1):
+ metadata = dict(selected_hits[index].metadata or {})
+ hardish = (
+ _normalize(metadata.get("memory_type", "")) == "hard_constraint"
+ or _normalize(metadata.get("durability", "")) == "hard"
+ or _normalize(metadata.get("conflict_policy", "")) == "must_preserve"
+ )
+ if not hardish and not bool(metadata.get("topic_bucket_dialogue_tunnel_allowed")):
+ replace_index = index
+ break
+ selected_hits[replace_index] = candidate
+ selected_ids.add(candidate.memory_id)
+ selected_bucket_ids.add(bucket_id)
+ dialogue_reserved_count += 1
+ selected_hits.sort(
+ key=lambda item: (
+ int(bool((item.metadata or {}).get("current_subject_resolver"))),
+ float(item.score),
+ int(item.turn_index),
+ ),
+ reverse=True,
+ )
+ return {
+ "hits": selected_hits[:limit],
+ "metadata": {
+ "topic_bucket_rerank_enabled": True,
+ "topic_bucket_no_fill_policy": True,
+ "topic_bucket_query_id": query_bucket_id,
+ "topic_bucket_query_label": _clean_text(query_topic.get("topic_label", "")),
+ "topic_bucket_query_keywords": query_keywords,
+ "topic_bucket_adjacent_ids": sorted(adjacent_ids),
+ "dialogue_tunnel_adjacent_ids": sorted(dialogue_tunnel_ids),
+ "topic_bucket_bridge_requested": bridge_requested,
+ "dialogue_tunnel_requested": dialogue_requested,
+ "topic_bucket_model_path_fallback": model_path_fallback,
+ "topic_bucket_generic_memory_fallback": generic_memory_fallback,
+ "topic_bucket_no_bucket_model_fallback": no_bucket_model_fallback,
+ "topic_bucket_candidate_count": len(hits),
+ "topic_bucket_rescue_count": len(rescue_hits),
+ "dialogue_tunnel_rescue_count": len(dialogue_rescue_hits),
+ "profile_query_rescue_count": len(profile_rescue_hits),
+ "dialogue_tunnel_reserved_count": dialogue_reserved_count,
+ "topic_bucket_focused_count": len(focused_hits),
+ "topic_bucket_final_count": len(selected_hits[:limit]),
+ "topic_bucket_filtered_count": filtered_count,
+ "topic_bucket_stats": stats,
+ },
+ }
+
+
+def _looks_like_write_turn(user_text: str, *, answer_payload: Dict[str, Any] | None = None) -> bool:
+ text = _normalize(user_text)
+ if not text:
+ return False
+ if "?" in text:
+ return False
+ if any(marker in text for marker in _WRITE_MARKERS):
+ return True
+ metadata = dict((answer_payload or {}).get("metadata", {}) or {})
+ return bool(metadata.get("memory_write"))
+
+
+def _explicit_overwrite_requested(user_text: str) -> bool:
+ text = _normalize(user_text)
+ return bool(text) and any(marker in text for marker in _OVERWRITE_MARKERS)
+
+
+def _apply_turn_write_intent(records: List[SessionMemoryRecordV2], *, user_text: str) -> List[SessionMemoryRecordV2]:
+ if not records or not _explicit_overwrite_requested(user_text):
+ return records
+ for record in records:
+ metadata = dict(record.metadata or {})
+ metadata.setdefault("write_intent", "overwrite")
+ metadata.setdefault("memory_gate_decision", "explicit_overwrite")
+ metadata["allow_parallel_state"] = False
+ record.metadata = metadata
+ return records
+
+
+def _records_from_extractor(
+ extractor: SessionMemoryExtractor,
+ *,
+ query: str,
+ answer_payload: Dict[str, Any] | None,
+ extraction_result: Dict[str, Any] | None,
+ turn_index: int,
+ profile: TMCRAProfile | None = None,
+) -> List[SessionMemoryRecordV2]:
+ profile = profile or TMCRAProfile()
+ raw_records = extractor.extract(
+ query=query,
+ extraction_result=extraction_result,
+ answer_bundle=answer_payload,
+ answer_mode=str((answer_payload or {}).get("answer_mode", "transparent")),
+ turn_index=turn_index,
+ )
+ results: List[SessionMemoryRecordV2] = []
+ for index, record in enumerate(raw_records):
+ slot_key = profile.stable_slot_key(
+ category=record.category,
+ value=record.value,
+ anchors=record.anchor_concepts,
+ slot_key=record.metadata.get("slot_key", "") if isinstance(record.metadata, dict) else "",
+ relation=record.relation,
+ metadata=dict(record.metadata or {}),
+ )
+ metadata = {
+ **dict(record.metadata or {}),
+ "memory_role": _clean_text(dict(record.metadata or {}).get("memory_role", "")) or "user",
+ "authority": _clean_text(dict(record.metadata or {}).get("authority", "")) or "source",
+ "canonical_slot_key": _clean_text(dict(record.metadata or {}).get("canonical_slot_key", "")) or slot_key,
+ "writeback_class": _clean_text(dict(record.metadata or {}).get("writeback_class", "")),
+ "origin_query": _clean_text(dict(record.metadata or {}).get("origin_query", "")) or _clean_text(query),
+ "origin_answer_id": _clean_text(dict(record.metadata or {}).get("origin_answer_id", "")),
+ "support_memory_ids": _dedupe(dict(record.metadata or {}).get("support_memory_ids", []) or []),
+ "support_fact_refs": _dedupe(dict(record.metadata or {}).get("support_fact_refs", []) or []),
+ "support_path_refs": _dedupe(dict(record.metadata or {}).get("support_path_refs", []) or []),
+ "promotion_state": _clean_text(dict(record.metadata or {}).get("promotion_state", "")) or "none",
+ }
+ results.append(
+ SessionMemoryRecordV2(
+ memory_id=f"{slot_key}:{turn_index}:auto:{index}",
+ category=record.category,
+ slot_key=slot_key,
+ value=_clean_text(record.value),
+ relation=_clean_text(record.relation) or f"{record.category}_memory",
+ anchor_concepts=_dedupe(record.anchor_concepts, max_items=8),
+ evidence_anchors=_dedupe(record.anchor_concepts, max_items=8),
+ salience=float(record.salience),
+ confidence=float(record.confidence),
+ source_kind=_clean_text(record.source_kind) or "session_memory",
+ turn_index=int(record.turn_index),
+ state="active",
+ metadata=metadata,
+ )
+ )
+ return results
+
+
+def _build_turn_records(
+ extractor: SessionMemoryExtractor,
+ *,
+ user_text: str,
+ answer_payload: Dict[str, Any] | None,
+ extraction_result: Dict[str, Any] | None,
+ turn_index: int,
+ allow_auto_extract: bool,
+ profile: TMCRAProfile | None = None,
+) -> List[SessionMemoryRecordV2]:
+ profile = profile or TMCRAProfile()
+ structured_records = _parse_structured_records(answer_payload, turn_index=turn_index, profile=profile)
+ if structured_records:
+ return _apply_typed_tunnel_annotations(
+ _apply_turn_write_intent(structured_records, user_text=user_text),
+ source_text=user_text,
+ )
+ structured_records = _parse_structured_records(extraction_result, turn_index=turn_index, profile=profile)
+ if structured_records:
+ return _apply_typed_tunnel_annotations(
+ _apply_turn_write_intent(structured_records, user_text=user_text),
+ source_text=user_text,
+ )
+ if not allow_auto_extract and not _looks_like_write_turn(user_text, answer_payload=answer_payload):
+ return []
+ return _apply_typed_tunnel_annotations(
+ _records_from_extractor(
+ extractor,
+ query=user_text,
+ answer_payload=None,
+ extraction_result=extraction_result,
+ turn_index=turn_index,
+ profile=profile,
+ ),
+ source_text=user_text,
+ )
+
+
+class NullMemoryAdapter(MemoryAdapter):
+ name = "null_memory"
+
+ def reset(self) -> None:
+ return None
+
+ def ingest_turn(
+ self,
+ user_text: str,
+ assistant_text: str = "",
+ *,
+ answer_payload: Dict[str, Any] | None = None,
+ extraction_result: Dict[str, Any] | None = None,
+ ) -> None:
+ _ = user_text, assistant_text, answer_payload, extraction_result
+
+ def retrieve(self, query: str, *, top_k: int = 6) -> MemoryRetrieval:
+ _ = query, top_k
+ return MemoryRetrieval()
+
+ def stats(self) -> Dict[str, Any]:
+ return _state_stats(storage_bytes=0, retrieval_context_tokens=0, total_state_tokens=0, records=0)
+
+ def storage_bytes(self) -> int:
+ return 0
+
+ def build_prompt_context(self, query: str, *, top_k: int = 8) -> Dict[str, Any]:
+ _ = top_k
+ return {
+ "mode": "null_memory",
+ "query": query,
+ "retrieval": MemoryRetrieval().to_dict(),
+ "stats": self.stats(),
+ "state": {},
+ }
+
+
+def _serialized_adapter_call(method):
+ @functools.wraps(method)
+ def wrapped(self, *args, **kwargs):
+ with self._operation_lock:
+ return method(self, *args, **kwargs)
+
+ return wrapped
+
+
+class GraphSessionMemoryAdapter(MemoryAdapter):
+ name = "graph_session_memory_v2"
+
+ def __init__(
+ self,
+ *,
+ auto_extract: bool = False,
+ storage_backend: str = "sqlite",
+ storage_path: str = "",
+ scope_id: str = "",
+ audit_retention: int = 256,
+ lightweight_stats: bool = True,
+ retrieval_mode: str = "heuristic",
+ node_model_path: str = "",
+ path_model_path: str = "",
+ node_model_device: str = "",
+ candidate_event_k: int = 24,
+ support_path_k: int = 3,
+ path_tunnel_rescue_k: int = 0,
+ path_tunnel_rescue_score_floor: float = 0.0,
+ path_tunnel_rescue_min_age: int = 0,
+ path_tunnel_rescue_min_score_margin: float = 0.0,
+ event_rerank_mode: str = "matrix",
+ matrix_event_top_k: int = DEFAULT_MATRIX_EVENT_TOP_K,
+ memory_router_mode: str = "",
+ memory_router_threshold: float = _MEMORY_ROUTER_DEFAULT_THRESHOLD,
+ memory_router_margin: float = _MEMORY_ROUTER_DEFAULT_MARGIN,
+ injection_planner_mode: str = "",
+ injection_planner_model_path: str = "",
+ injection_planner_latest_path: str = "",
+ injection_planner_device: str = "",
+ injection_planner_selection_threshold: float = -1.0,
+ injection_planner_row_threshold: float = -1.0,
+ injection_planner_logic_threshold: float = -1.0,
+ temporal_layer_mode: str = "",
+ temporal_router_mode: str = "",
+ temporal_router_dir: str = "",
+ temporal_router_latest_path: str = "",
+ temporal_router_device: str = "",
+ ) -> None:
+ self._operation_lock = threading.RLock()
+ prewarm_embedder_mode = (
+ _normalize(os.getenv("TMCRA_EMBEDDER_INDEX_RECALL_MODE", ""))
+ or _normalize(os.getenv("TMCRA_WRITE_EMBEDDER_INDEX_MODE", ""))
+ )
+ self._embedder_prewarm_metadata = _prewarm_embedder_dense_if_requested(mode=prewarm_embedder_mode)
+ self.extractor = SessionMemoryExtractor()
+ self.profile = TMCRAProfile()
+ self.temporal_organizer = TemporalOrganizer()
+ self.temporal_query_planner = TemporalQueryPlanner()
+ self.timeline_evidence_builder = TimelineEvidencePackBuilder()
+ self.auto_extract = bool(auto_extract)
+ self.storage_backend = _normalize(storage_backend) or "sqlite"
+ self.audit_retention = max(1, int(audit_retention))
+ self.lightweight_stats = bool(lightweight_stats)
+ self.scope_id = _clean_text(scope_id) or f"graph-session-{uuid.uuid4().hex}"
+ self._store: SQLiteSessionMemoryStore | None = None
+ self.storage_path = ""
+ if self.storage_backend == "sqlite":
+ resolved_storage_path = _clean_text(storage_path) or str((Path(tempfile.gettempdir()) / "tmcra_graph_session_memory.sqlite3").resolve())
+ self._store = SQLiteSessionMemoryStore(resolved_storage_path, audit_retention=self.audit_retention)
+ self.storage_path = str(self._store.storage_path)
+ self.graph = self._store.load_graph(self.scope_id)
+ elif self.storage_backend == "memory":
+ self.storage_path = _clean_text(storage_path)
+ self.graph = SessionMemoryGraphV2(
+ audit_retention=self.audit_retention,
+ persistence_backend="memory",
+ persistence_path=self.storage_path,
+ )
+ else:
+ raise ValueError(f"Unsupported storage backend: {self.storage_backend}")
+ self._runtime_graph_cache_epoch = 0
+ self._runtime_graph_cache_key: tuple[Any, ...] | None = None
+ self._runtime_graph_cache_source_hits: List[MemoryHit] | None = None
+ self._runtime_graph_cache_payload: Dict[str, Any] | None = None
+ self._runtime_graph_cache_symbolic_index: Dict[str, Any] | None = None
+ self._last_retrieval_context_tokens = 0
+ self._last_writeback_summary: Dict[str, Any] = {}
+ self.retrieval_mode = _normalize(retrieval_mode) or "heuristic"
+ self.node_model_path = _clean_text(node_model_path)
+ self.path_model_path = _clean_text(path_model_path)
+ self.node_model_device = _clean_text(node_model_device)
+ self.candidate_event_k = max(1, int(candidate_event_k))
+ self.support_path_k = max(1, int(support_path_k))
+ self.path_tunnel_rescue_k = max(0, int(path_tunnel_rescue_k or 0))
+ self.path_tunnel_rescue_score_floor = max(0.0, float(path_tunnel_rescue_score_floor or 0.0))
+ self.path_tunnel_rescue_min_age = max(0, int(path_tunnel_rescue_min_age or 0))
+ self.path_tunnel_rescue_min_score_margin = max(0.0, float(path_tunnel_rescue_min_score_margin or 0.0))
+ self.event_rerank_mode = _normalize(event_rerank_mode) or "matrix"
+ self.matrix_event_top_k = max(1, int(matrix_event_top_k or DEFAULT_MATRIX_EVENT_TOP_K))
+ self.write_embedder_index_mode = _normalize(os.getenv("TMCRA_WRITE_EMBEDDER_INDEX_MODE", ""))
+ if not self.write_embedder_index_mode:
+ self.write_embedder_index_mode = "off"
+ try:
+ self.write_embedder_index_max_terms = max(
+ 8,
+ int(os.getenv("TMCRA_WRITE_EMBEDDER_INDEX_MAX_TERMS", "96") or 96),
+ )
+ except (TypeError, ValueError):
+ self.write_embedder_index_max_terms = 96
+ env_embedder_recall_mode = _normalize(os.getenv("TMCRA_EMBEDDER_INDEX_RECALL_MODE", ""))
+ self.embedder_index_recall_mode = env_embedder_recall_mode or self.write_embedder_index_mode
+ try:
+ self.embedder_index_recall_k = max(
+ 0,
+ int(os.getenv("TMCRA_EMBEDDER_INDEX_RECALL_K", "0") or 0),
+ )
+ except (TypeError, ValueError):
+ self.embedder_index_recall_k = 0
+ self.embedder_pre_recall_mode = _normalize(os.getenv("TMCRA_EMBEDDER_PRE_RECALL_MODE", ""))
+ if not self.embedder_pre_recall_mode:
+ self.embedder_pre_recall_mode = "off"
+ try:
+ self.embedder_pre_recall_k = max(
+ 0,
+ int(os.getenv("TMCRA_EMBEDDER_PRE_RECALL_K", "0") or 0),
+ )
+ except (TypeError, ValueError):
+ self.embedder_pre_recall_k = 0
+ self.embedder_fusion_mode = _normalize(os.getenv("TMCRA_EMBEDDER_FUSION_MODE", ""))
+ try:
+ self.embedder_fusion_weight = max(
+ 0.0,
+ float(os.getenv("TMCRA_EMBEDDER_FUSION_WEIGHT", "0.35") or 0.35),
+ )
+ except (TypeError, ValueError):
+ self.embedder_fusion_weight = 0.35
+ try:
+ self.embedder_fusion_score_floor = max(
+ 0.0,
+ float(os.getenv("TMCRA_EMBEDDER_FUSION_SCORE_FLOOR", "0.62") or 0.62),
+ )
+ except (TypeError, ValueError):
+ self.embedder_fusion_score_floor = 0.62
+ try:
+ self.embedder_fusion_top_k = max(
+ 0,
+ int(os.getenv("TMCRA_EMBEDDER_FUSION_TOP_K", "16") or 16),
+ )
+ except (TypeError, ValueError):
+ self.embedder_fusion_top_k = 16
+ try:
+ self.embedder_fusion_select_k = max(
+ 0,
+ int(os.getenv("TMCRA_EMBEDDER_FUSION_SELECT_K", "4") or 4),
+ )
+ except (TypeError, ValueError):
+ self.embedder_fusion_select_k = 4
+ try:
+ self.embedder_fusion_max_boost = max(
+ 0.0,
+ float(os.getenv("TMCRA_EMBEDDER_FUSION_MAX_BOOST", "0.42") or 0.42),
+ )
+ except (TypeError, ValueError):
+ self.embedder_fusion_max_boost = 0.42
+ env_router_mode = _clean_text(os.getenv("TMCRA_MEMORY_ROUTER_MODE", ""))
+ self.memory_router_mode = _normalize(memory_router_mode) or _normalize(env_router_mode) or "observe"
+ try:
+ self.memory_router_threshold = float(
+ os.getenv("TMCRA_MEMORY_ROUTER_THRESHOLD", "")
+ or memory_router_threshold
+ or _MEMORY_ROUTER_DEFAULT_THRESHOLD
+ )
+ except (TypeError, ValueError):
+ self.memory_router_threshold = _MEMORY_ROUTER_DEFAULT_THRESHOLD
+ try:
+ self.memory_router_margin = float(
+ os.getenv("TMCRA_MEMORY_ROUTER_MARGIN", "")
+ or memory_router_margin
+ or _MEMORY_ROUTER_DEFAULT_MARGIN
+ )
+ except (TypeError, ValueError):
+ self.memory_router_margin = _MEMORY_ROUTER_DEFAULT_MARGIN
+ self._loaded_node_scorer: LoadedNodeMemoryScorer | None = None
+ self._node_scorer_error = ""
+ env_planner_mode = _clean_text(os.getenv("TMCRA_INJECTION_PLANNER_MODE", ""))
+ self.injection_planner_mode = _normalize(injection_planner_mode) or _normalize(env_planner_mode) or "observe"
+ self.injection_planner_model_path = _clean_text(
+ injection_planner_model_path or os.getenv("TMCRA_INJECTION_PLANNER_MODEL_PATH", "")
+ )
+ self.injection_planner_latest_path = _clean_text(
+ injection_planner_latest_path or os.getenv("TMCRA_INJECTION_PLANNER_LATEST", "")
+ )
+ self.injection_planner_device = _clean_text(
+ injection_planner_device or os.getenv("TMCRA_INJECTION_PLANNER_DEVICE", "")
+ )
+ self.injection_planner_selection_threshold_override = float(injection_planner_selection_threshold)
+ self.injection_planner_row_threshold_override = float(injection_planner_row_threshold)
+ self.injection_planner_logic_threshold_override = float(injection_planner_logic_threshold)
+ self._loaded_injection_planner: Any | None = None
+ self._injection_planner_config: Any | None = None
+ self._injection_planner_payload: Dict[str, Any] = {}
+ self._injection_planner_thresholds: Dict[str, float] = {}
+ self._injection_planner_resolved_path = ""
+ self._injection_planner_error = ""
+ self._injection_planner_evidence_role_supported = False
+ env_temporal_layer_mode = _clean_text(os.getenv("TMCRA_TEMPORAL_LAYER_MODE", ""))
+ self.temporal_layer_mode = _normalize(temporal_layer_mode) or _normalize(env_temporal_layer_mode) or "observe"
+ env_temporal_router_mode = _clean_text(os.getenv("TMCRA_TEMPORAL_ROUTER_MODE", ""))
+ self.temporal_router_mode = _normalize(temporal_router_mode) or _normalize(env_temporal_router_mode) or "observe"
+ self.temporal_router_dir = _clean_text(
+ temporal_router_dir or os.getenv("TMCRA_TEMPORAL_ROUTER_DIR", "")
+ )
+ self.temporal_router_latest_path = _clean_text(
+ temporal_router_latest_path
+ or os.getenv("TMCRA_TEMPORAL_ROUTER_LATEST", "")
+ )
+ self.temporal_router_device = _clean_text(
+ temporal_router_device or os.getenv("TMCRA_TEMPORAL_ROUTER_DEVICE", "")
+ ) or "cpu"
+ self._loaded_temporal_router: Any | None = None
+ self._temporal_router_resolved_dir = ""
+ self._temporal_router_error = ""
+
+ def _empty_graph(self) -> SessionMemoryGraphV2:
+ return SessionMemoryGraphV2(
+ audit_retention=self.audit_retention,
+ persistence_backend=self.storage_backend,
+ persistence_path=self.storage_path,
+ )
+
+ def _reload_graph(self) -> None:
+ if self._store is not None:
+ self.graph = self._store.load_graph(self.scope_id)
+ else:
+ self.graph.configure_persistence(
+ backend=self.storage_backend,
+ path=self.storage_path,
+ audit_retention=self.audit_retention,
+ )
+
+ def _invalidate_runtime_graph_cache(self) -> None:
+ self._runtime_graph_cache_epoch += 1
+ self._runtime_graph_cache_key = None
+ self._runtime_graph_cache_source_hits = None
+ self._runtime_graph_cache_payload = None
+ self._runtime_graph_cache_symbolic_index = None
+
+ def _runtime_graph_static_cache_key(self) -> tuple[Any, ...]:
+ algorithm_version = "tmcra.runtime-graph.static.1"
+ if self._store is not None:
+ revision = getattr(self.graph, "_storage_revision", None)
+ if revision is None:
+ raise RuntimeError("SQLite graph is missing storage_revision required for runtime graph caching")
+ return (
+ algorithm_version,
+ "sqlite",
+ str(Path(self.storage_path).resolve()),
+ self.scope_id,
+ int(revision),
+ int(len(getattr(self.graph, "records_by_id", {}) or {})),
+ )
+
+ record_payload = [
+ record.to_dict()
+ for record in list(getattr(self.graph, "records_by_id", {}).values())
+ ]
+ record_fingerprint = hashlib.sha256(
+ json.dumps(
+ record_payload,
+ ensure_ascii=False,
+ sort_keys=True,
+ separators=(",", ":"),
+ default=str,
+ ).encode("utf-8")
+ ).hexdigest()
+ return (
+ algorithm_version,
+ "memory",
+ self.storage_path,
+ self.scope_id,
+ int(self._runtime_graph_cache_epoch),
+ record_fingerprint,
+ )
+
+ def _persist_graph(self) -> None:
+ self.graph.configure_persistence(
+ backend=self.storage_backend,
+ path=self.storage_path,
+ audit_retention=self.audit_retention,
+ )
+ if self._store is not None:
+ self._store.save_graph(self.scope_id, self.graph)
+ self._invalidate_runtime_graph_cache()
+
+ def _persist_latest_audit(self, field_name: str) -> Dict[str, Any]:
+ self.graph.configure_persistence(
+ backend=self.storage_backend,
+ path=self.storage_path,
+ audit_retention=self.audit_retention,
+ )
+ events = getattr(self.graph, field_name)
+ if not events:
+ raise RuntimeError(f"cannot persist empty audit field: {field_name}")
+ if self._store is None:
+ return dict(events[-1])
+ persisted = self._store.append_audit_event(
+ self.scope_id,
+ field_name,
+ dict(events[-1]),
+ )
+ events[-1] = dict(persisted["payload"])
+ self.graph.audit_event_totals[field_name] = int(persisted["event_total"])
+ self.graph.audit_trimmed_counts[field_name] = int(
+ persisted["trimmed_total"]
+ )
+ return dict(events[-1])
+
+ @_serialized_adapter_call
+ def replace_graph(self, graph: SessionMemoryGraphV2) -> None:
+ self.graph = graph
+ self._persist_graph()
+
+ def _storage_breakdown(self) -> Dict[str, int]:
+ core_payload = self.graph._core_payload()
+ audit_payload_full = self.graph._audit_payload()
+ if self.lightweight_stats:
+ audit_token_payload = {
+ "totals": dict(self.graph.audit_event_totals),
+ "trimmed": dict(self.graph.audit_trimmed_counts),
+ "retained": {
+ "turn_log": len(self.graph.turn_log),
+ "retrieval_log": len(self.graph.retrieval_log),
+ "answer_support_log": len(self.graph.answer_support_log),
+ },
+ "audit_retention": int(self.graph.audit_retention),
+ }
+ else:
+ audit_token_payload = audit_payload_full
+ core_storage_bytes = len(json.dumps(core_payload, ensure_ascii=False).encode("utf-8"))
+ audit_storage_bytes = len(json.dumps(audit_payload_full, ensure_ascii=False).encode("utf-8"))
+ core_state_token_estimate = _estimate_tokens(json.dumps(core_payload, ensure_ascii=False))
+ audit_state_token_estimate = _estimate_tokens(json.dumps(audit_token_payload, ensure_ascii=False))
+ return {
+ "core_storage_bytes": int(core_storage_bytes),
+ "audit_storage_bytes": int(audit_storage_bytes),
+ "storage_bytes": int(core_storage_bytes + audit_storage_bytes),
+ "core_state_token_estimate": int(core_state_token_estimate),
+ "audit_state_token_estimate": int(audit_state_token_estimate),
+ "total_state_token_estimate": int(core_state_token_estimate + audit_state_token_estimate),
+ }
+
+ @_serialized_adapter_call
+ def reset(self) -> None:
+ if self._store is not None:
+ self._store.clear_scope(self.scope_id)
+ self.graph = self._store.load_graph(self.scope_id)
+ else:
+ self.graph = self._empty_graph()
+ self._invalidate_runtime_graph_cache()
+ self._last_retrieval_context_tokens = 0
+ self._last_writeback_summary = {}
+
+ def _temporal_layer_enabled(self) -> bool:
+ return _normalize(self.temporal_layer_mode) not in _TEMPORAL_LAYER_DISABLED_MODES
+
+ def _temporal_router_enabled(self) -> bool:
+ return self._temporal_layer_enabled() and _normalize(self.temporal_router_mode) not in _TEMPORAL_LAYER_DISABLED_MODES
+
+ def _resolve_temporal_router_dir(self) -> str:
+ direct_dir = _clean_text(self.temporal_router_dir)
+ if direct_dir:
+ root = Path(direct_dir)
+ if (root / "writer_temporal_router.pt").exists() and (root / "query_temporal_router.pt").exists():
+ return str(root)
+ latest_path = _clean_text(self.temporal_router_latest_path)
+ if latest_path:
+ pointer = Path(latest_path)
+ if pointer.exists():
+ lines = pointer.read_text(encoding="utf-8").splitlines()
+ candidate = _clean_text(lines[0] if lines else "")
+ if candidate:
+ root = Path(candidate)
+ if (root / "writer_temporal_router.pt").exists() and (root / "query_temporal_router.pt").exists():
+ return str(root)
+ return ""
+
+ def _load_temporal_router(self) -> LoadedTemporalRouter | None:
+ if not self._temporal_router_enabled():
+ self._temporal_router_error = "disabled"
+ return None
+ model_dir = self._resolve_temporal_router_dir()
+ self._temporal_router_resolved_dir = model_dir
+ if not model_dir:
+ self._temporal_router_error = "model_dir_missing"
+ return None
+ if self._loaded_temporal_router is not None:
+ return self._loaded_temporal_router
+ try:
+ self._loaded_temporal_router = LoadedTemporalRouter.from_dir(
+ model_dir,
+ device=self.temporal_router_device or "cpu",
+ )
+ self._temporal_router_error = ""
+ except Exception as exc: # pragma: no cover - defensive runtime path
+ self._loaded_temporal_router = None
+ self._temporal_router_error = f"{type(exc).__name__}: {exc}"
+ return self._loaded_temporal_router
+
+ def _temporal_router_status_metadata(self) -> Dict[str, Any]:
+ router = self._load_temporal_router()
+ return {
+ "temporal_router_enabled": router is not None,
+ "temporal_router_mode": _normalize(self.temporal_router_mode) or "observe",
+ "temporal_router_model_dir": self._temporal_router_resolved_dir,
+ "temporal_router_error": self._temporal_router_error,
+ }
+
+ def _session_timestamp_from_payloads(self, *payloads: Mapping[str, Any] | None) -> str:
+ for payload in payloads:
+ data = dict(payload or {})
+ for key in ("session_timestamp", "timestamp", "created_at", "turn_timestamp"):
+ value = _clean_text(data.get(key, ""))
+ if value:
+ return value
+ metadata = data.get("metadata")
+ if isinstance(metadata, Mapping):
+ for key in ("session_timestamp", "timestamp", "created_at", "turn_timestamp"):
+ value = _clean_text(metadata.get(key, ""))
+ if value:
+ return value
+ return ""
+
+ def _temporal_frame_for_turn(
+ self,
+ *,
+ user_text: str,
+ previous_turn: str = "",
+ answer_payload: Dict[str, Any] | None = None,
+ extraction_result: Dict[str, Any] | None = None,
+ ) -> TemporalFrame | None:
+ if not self._temporal_layer_enabled():
+ return None
+ session_timestamp = self._session_timestamp_from_payloads(answer_payload, extraction_result)
+ model_frame = None
+ for payload in (answer_payload, extraction_result):
+ data = dict(payload or {})
+ candidate = data.get("temporal_frame") or dict(data.get("metadata", {}) or {}).get("temporal_frame")
+ if isinstance(candidate, Mapping):
+ model_frame = candidate
+ break
+ fallback_frame = self.temporal_organizer.organize_turn(
+ current_turn=user_text,
+ previous_turn=previous_turn,
+ session_timestamp=session_timestamp,
+ speaker="user",
+ )
+ if model_frame is None:
+ router = self._load_temporal_router()
+ if router is not None and router.writer_available():
+ predicted = router.predict_writer_frame(
+ current_turn=user_text,
+ previous_turn=previous_turn,
+ session_timestamp=session_timestamp,
+ )
+ writer_confidence = float(predicted.get("confidence", 0.0) or 0.0) if predicted else 0.0
+ if predicted and writer_confidence >= _float_env("TMCRA_TEMPORAL_ROUTER_WRITER_MIN_CONFIDENCE", _TEMPORAL_ROUTER_DEFAULT_WRITER_MIN_CONFIDENCE):
+ frame_payload = fallback_frame.to_dict()
+ frame_payload.update(
+ {
+ key: value
+ for key, value in predicted.items()
+ if key in {"temporal_intent", "anchor_type", "granularity", "state_operation"}
+ and _clean_text(value)
+ }
+ )
+ if "should_create_timeline_edge" in predicted:
+ frame_payload["should_create_timeline_edge"] = bool(predicted.get("should_create_timeline_edge", False))
+ if writer_confidence > 0.0:
+ frame_payload["confidence"] = writer_confidence
+ frame_payload["metadata"] = {
+ **dict(frame_payload.get("metadata", {}) or {}),
+ **dict(predicted.get("metadata", {}) or {}),
+ **self._temporal_router_status_metadata(),
+ }
+ model_frame = frame_payload
+ if model_frame is None:
+ return fallback_frame
+ return self.temporal_organizer.organize_turn(
+ current_turn=user_text,
+ previous_turn=previous_turn,
+ session_timestamp=session_timestamp,
+ speaker="user",
+ model_frame=model_frame,
+ )
+
+ def _temporal_turn_metadata(self, frame: TemporalFrame | None) -> Dict[str, Any]:
+ if frame is None:
+ return {
+ "temporal_layer_enabled": False,
+ "temporal_layer_mode": _normalize(self.temporal_layer_mode) or "observe",
+ **self._temporal_router_status_metadata(),
+ }
+ return {
+ "temporal_layer_enabled": True,
+ "temporal_layer_mode": _normalize(self.temporal_layer_mode) or "observe",
+ "temporal_frame": frame.to_dict(),
+ "temporal_intent": frame.temporal_intent,
+ "temporal_subject_key": frame.subject_key,
+ "temporal_state_operation": frame.state_operation,
+ **self._temporal_router_status_metadata(),
+ }
+
+ def _apply_temporal_frame_to_records(self, records: Sequence[SessionMemoryRecordV2], frame: TemporalFrame | None) -> None:
+ if frame is None:
+ return
+ if frame.temporal_intent == "non_temporal" and not frame.subject_key:
+ return
+ for record in records:
+ metadata = dict(record.metadata or {})
+ metadata.update(
+ {
+ "temporal_layer": True,
+ "temporal_frame": frame.to_dict(),
+ "temporal_intent": frame.temporal_intent,
+ "temporal_subject": frame.subject,
+ "temporal_subject_key": frame.subject_key,
+ "temporal_state_operation": frame.state_operation,
+ "temporal_event_time": frame.event_time,
+ "temporal_state_valid_from": frame.state_valid_from,
+ "temporal_state_valid_to": frame.state_valid_to,
+ }
+ )
+ record.metadata = metadata
+
+ def _build_timeline_state_layer(self) -> TimelineStateLayer:
+ layer = TimelineStateLayer()
+ for turn in sorted(list(getattr(self.graph, "turn_log", []) or []), key=lambda item: int(item.get("turn_index", 0) or 0)):
+ metadata = dict(turn.get("metadata", {}) or {})
+ frame_payload = metadata.get("temporal_frame")
+ if not isinstance(frame_payload, Mapping):
+ continue
+ frame = TemporalFrame.from_mapping(frame_payload)
+ if frame.temporal_intent == "non_temporal" and not frame.new_state:
+ continue
+ record_ids = [item for item in list(turn.get("record_ids", []) or []) if _clean_text(item)]
+ source_event_id = _clean_text(metadata.get("temporal_source_record_id", "")) or (record_ids[0] if record_ids else "")
+ frame.metadata = {
+ **dict(frame.metadata or {}),
+ "source_text": _clean_text(turn.get("text", "")) or frame.evidence_span,
+ }
+ layer.apply_frame(
+ frame,
+ source_event_id=source_event_id,
+ source_turn_id=_clean_text(turn.get("turn_id", "")),
+ state_type=_clean_text(metadata.get("temporal_state_type", "")) or "profile",
+ )
+ return layer
+
+ def _temporal_runtime_pack(self, query: str) -> Dict[str, Any]:
+ metadata: Dict[str, Any] = {
+ "temporal_runtime_enabled": False,
+ "temporal_layer_mode": _normalize(self.temporal_layer_mode) or "observe",
+ **self._temporal_router_status_metadata(),
+ }
+ if not self._temporal_layer_enabled():
+ metadata["temporal_runtime_reason"] = "disabled"
+ return {"metadata": metadata}
+ plan = self.temporal_query_planner.plan(query)
+ router = self._load_temporal_router()
+ if router is not None and router.query_available():
+ predicted_plan = router.predict_query_plan(query=query)
+ query_confidence = float(predicted_plan.get("confidence", 0.0) or 0.0) if predicted_plan else 0.0
+ router_confidences = dict(dict(predicted_plan.get("metadata", {}) or {}).get("temporal_router_confidences", {}) or {}) if predicted_plan else {}
+ intent_confidence = float(router_confidences.get("query_temporal_intent", 0.0) or 0.0)
+ if (
+ predicted_plan
+ and query_confidence >= _float_env("TMCRA_TEMPORAL_ROUTER_QUERY_MIN_CONFIDENCE", _TEMPORAL_ROUTER_DEFAULT_QUERY_MIN_CONFIDENCE)
+ and intent_confidence >= _float_env("TMCRA_TEMPORAL_ROUTER_QUERY_INTENT_MIN_CONFIDENCE", _TEMPORAL_ROUTER_DEFAULT_QUERY_INTENT_MIN_CONFIDENCE)
+ ):
+ plan_payload = plan.to_dict()
+ plan_payload.update(
+ {
+ key: value
+ for key, value in predicted_plan.items()
+ if key in {"query_temporal_intent", "timeline_operation"} and _clean_text(value)
+ }
+ )
+ for key in ("prefer_current_state", "prefer_previous_state", "requires_ordered_chain", "requires_comparison"):
+ if key in predicted_plan:
+ plan_payload[key] = bool(predicted_plan.get(key, False))
+ if float(predicted_plan.get("confidence", 0.0) or 0.0) > 0.0:
+ plan_payload["confidence"] = float(predicted_plan.get("confidence", 0.0) or 0.0)
+ plan_payload["metadata"] = {
+ **dict(plan_payload.get("metadata", {}) or {}),
+ **dict(predicted_plan.get("metadata", {}) or {}),
+ **self._temporal_router_status_metadata(),
+ }
+ plan = TemporalQueryPlan(**{key: value for key, value in plan_payload.items() if key in TemporalQueryPlan.__dataclass_fields__})
+ metadata["temporal_query_plan"] = plan.to_dict()
+ if plan.query_temporal_intent == "non_temporal" or plan.timeline_operation == "none":
+ metadata["temporal_runtime_reason"] = "non_temporal_query"
+ return {"plan": plan, "metadata": metadata}
+ if plan.timeline_operation in {"query_current", "query_previous"} and not _clean_text(plan.target_subject_key):
+ metadata["temporal_runtime_reason"] = "missing_target_subject"
+ return {"plan": plan, "metadata": metadata}
+ timeline_layer = self._build_timeline_state_layer()
+ pack = self.timeline_evidence_builder.build(plan=plan, timeline_layer=timeline_layer)
+ metadata.update(
+ {
+ "temporal_runtime_enabled": True,
+ "temporal_runtime_reason": "ok",
+ "temporal_evidence_pack": pack.to_dict(),
+ "temporal_selected_answer_value": _clean_text(pack.selected_evidence.get("answer_value", "")),
+ "temporal_selected_state_id": _clean_text(pack.selected_evidence.get("state_id", "")),
+ "temporal_timeline_state_count": len(pack.timeline),
+ }
+ )
+ return {"plan": plan, "pack": pack, "metadata": metadata}
+
+ def _temporal_state_hit(
+ self,
+ *,
+ state_payload: Mapping[str, Any],
+ plan_payload: Mapping[str, Any],
+ selected: bool,
+ rank: int,
+ ) -> MemoryHit | None:
+ state_value = _clean_text(state_payload.get("state", ""))
+ state_id = _clean_text(state_payload.get("state_id", ""))
+ if not state_value or not state_id:
+ return None
+ source_event_id = _clean_text(state_payload.get("source_event_id", ""))
+ source_record = self.graph.records_by_id.get(source_event_id)
+ source_metadata = dict(source_record.metadata or {}) if source_record is not None else {}
+ score = 0.98 if selected else max(0.75, 0.92 - (rank * 0.03))
+ return MemoryHit(
+ memory_id=f"temporal_state:{state_id}",
+ category="time",
+ value=state_value,
+ relation="temporal_state",
+ anchors=_dedupe([state_payload.get("time", ""), state_payload.get("source_text", ""), plan_payload.get("target_subject", "")], max_items=6),
+ score=round(score, 6),
+ source_kind="temporal_state_layer",
+ slot_key=f"temporal.{_clean_text(plan_payload.get('target_subject_key', 'general'))}",
+ state="active" if bool(state_payload.get("is_current", False)) else "history",
+ turn_index=int(source_record.turn_index) if source_record is not None else 0,
+ metadata={
+ **source_metadata,
+ "temporal_runtime_hit": True,
+ "temporal_runtime_selected": bool(selected),
+ "temporal_state_id": state_id,
+ "temporal_state_value": state_value,
+ "temporal_state_time": _clean_text(state_payload.get("time", "")),
+ "temporal_state_valid_to": _clean_text(state_payload.get("valid_to", "")),
+ "temporal_state_is_current": bool(state_payload.get("is_current", False)),
+ "temporal_source_event_id": source_event_id,
+ "temporal_source_turn_id": _clean_text(state_payload.get("source_turn_id", "")),
+ "temporal_query_plan": dict(plan_payload),
+ },
+ )
+
+ def _apply_temporal_evidence_pack_to_hits(
+ self,
+ hits: Sequence[MemoryHit],
+ temporal_payload: Mapping[str, Any],
+ *,
+ top_k: int,
+ ) -> Dict[str, Any]:
+ metadata = dict(temporal_payload.get("metadata", {}) or {})
+ pack = temporal_payload.get("pack")
+ plan = temporal_payload.get("plan")
+ if pack is None or plan is None or not bool(metadata.get("temporal_runtime_enabled", False)):
+ return {"hits": list(hits), "metadata": metadata}
+ pack_payload = pack.to_dict() if hasattr(pack, "to_dict") else dict(pack)
+ plan_payload = plan.to_dict() if hasattr(plan, "to_dict") else dict(plan)
+ selected_state_id = _clean_text(dict(pack_payload.get("selected_evidence", {}) or {}).get("state_id", ""))
+ selected_answer_value = _clean_text(dict(pack_payload.get("selected_evidence", {}) or {}).get("answer_value", ""))
+ timeline = list(pack_payload.get("timeline", []) or [])
+ synthetic_hits: List[MemoryHit] = []
+ if selected_state_id:
+ selected_state = next((dict(item) for item in timeline if _clean_text(dict(item).get("state_id", "")) == selected_state_id), None)
+ if selected_state is None and selected_answer_value:
+ selected_state = {
+ "state_id": selected_state_id,
+ "state": selected_answer_value,
+ "source_event_id": dict(pack_payload.get("selected_evidence", {}) or {}).get("source_event_id", ""),
+ "source_turn_id": dict(pack_payload.get("selected_evidence", {}) or {}).get("source_turn_id", ""),
+ "is_current": bool(plan_payload.get("prefer_current_state", False)),
+ }
+ if selected_state:
+ hit = self._temporal_state_hit(state_payload=selected_state, plan_payload=plan_payload, selected=True, rank=0)
+ if hit is not None:
+ synthetic_hits.append(hit)
+ if bool(plan_payload.get("requires_ordered_chain", False)) or bool(plan_payload.get("requires_comparison", False)):
+ for index, state_payload in enumerate(timeline):
+ if _clean_text(dict(state_payload).get("state_id", "")) == selected_state_id:
+ continue
+ hit = self._temporal_state_hit(
+ state_payload=dict(state_payload),
+ plan_payload=plan_payload,
+ selected=False,
+ rank=index + 1,
+ )
+ if hit is not None:
+ synthetic_hits.append(hit)
+ existing: List[MemoryHit] = []
+ synthetic_source_ids = {
+ _clean_text((hit.metadata or {}).get("temporal_source_event_id", ""))
+ for hit in synthetic_hits
+ if _clean_text((hit.metadata or {}).get("temporal_source_event_id", ""))
+ }
+ for hit in hits:
+ hit_metadata = dict(hit.metadata or {})
+ source_event_id = _clean_text(hit_metadata.get("event_id", hit.memory_id))
+ if source_event_id in synthetic_source_ids:
+ hit_metadata.update(
+ {
+ "temporal_runtime_support": True,
+ "temporal_query_plan": dict(plan_payload),
+ "temporal_selected_answer_value": selected_answer_value,
+ "temporal_selected_state_id": selected_state_id,
+ }
+ )
+ hit = MemoryHit(
+ memory_id=hit.memory_id,
+ category=hit.category,
+ value=hit.value,
+ relation=hit.relation,
+ anchors=list(hit.anchors),
+ score=hit.score,
+ source_kind=hit.source_kind,
+ slot_key=hit.slot_key,
+ state=hit.state,
+ turn_index=int(hit.turn_index),
+ metadata=hit_metadata,
+ )
+ existing.append(hit)
+ merged: List[MemoryHit] = []
+ seen = set()
+ for hit in [*synthetic_hits, *existing]:
+ key = _clean_text(hit.memory_id)
+ if key and key in seen:
+ continue
+ if key:
+ seen.add(key)
+ merged.append(hit)
+ metadata.update(
+ {
+ "temporal_runtime_injected_hit_count": len(synthetic_hits),
+ "temporal_runtime_injected_hit_ids": [hit.memory_id for hit in synthetic_hits],
+ }
+ )
+ return {"hits": merged[: max(len(merged), top_k)], "metadata": metadata}
+
+ @_serialized_adapter_call
+ def ingest_turn(
+ self,
+ user_text: str,
+ assistant_text: str = "",
+ *,
+ answer_payload: Dict[str, Any] | None = None,
+ extraction_result: Dict[str, Any] | None = None,
+ ) -> None:
+ self._reload_graph()
+ turn_index = self.graph.next_turn()
+ previous_topic = _last_topic_turn(self.graph)
+ previous_turn_text = _clean_text(self.graph.turn_log[-1].get("text", "")) if self.graph.turn_log else ""
+ temporal_frame = self._temporal_frame_for_turn(
+ user_text=user_text,
+ previous_turn=previous_turn_text,
+ answer_payload=answer_payload,
+ extraction_result=extraction_result,
+ )
+ topic_bucket = _assign_topic_bucket_for_text(
+ self.graph,
+ user_text,
+ answer_payload=answer_payload,
+ turn_index=turn_index,
+ create=True,
+ )
+ records = _build_turn_records(
+ self.extractor,
+ user_text=user_text,
+ answer_payload=answer_payload,
+ extraction_result=extraction_result,
+ turn_index=turn_index,
+ allow_auto_extract=self.auto_extract,
+ profile=self.profile,
+ )
+ self._apply_temporal_frame_to_records(records, temporal_frame)
+ _apply_topic_bucket_to_records(records, topic_bucket)
+ stored_ids = self.graph.add_records(records)
+ write_embedder_metadata = _apply_write_embedder_index_to_graph(
+ self.graph,
+ stored_ids=stored_ids,
+ turn_text=user_text,
+ turn_index=turn_index,
+ mode=self.write_embedder_index_mode,
+ max_terms=self.write_embedder_index_max_terms,
+ )
+ topic_bridge_metadata = _add_topic_bridge_edges(
+ self.graph,
+ previous_topic=previous_topic,
+ current_topic=topic_bucket,
+ current_record_ids=stored_ids,
+ turn_index=turn_index,
+ evidence=user_text,
+ )
+ dialogue_tunnel_metadata = _add_dialogue_tunnel_edges(
+ self.graph,
+ current_topic=topic_bucket,
+ current_record_ids=stored_ids,
+ turn_index=turn_index,
+ evidence=user_text,
+ )
+ turn_kind = "memory_write" if stored_ids else "noise"
+ self.graph.record_turn(
+ turn_kind=turn_kind,
+ text=user_text,
+ turn_index=turn_index,
+ record_ids=stored_ids,
+ speaker="user",
+ assistant_text=assistant_text,
+ metadata={
+ "source": "user_turn",
+ "auto_extract": bool(self.auto_extract),
+ "topic_bucket_id": _clean_text(topic_bucket.get("topic_bucket_id", "")),
+ "topic_label": _clean_text(topic_bucket.get("topic_label", "")),
+ "topic_keywords": list(topic_bucket.get("topic_keywords", []) or []),
+ "temporal_source_record_id": stored_ids[0] if stored_ids else "",
+ **self._temporal_turn_metadata(temporal_frame),
+ **write_embedder_metadata,
+ **topic_bridge_metadata,
+ **dialogue_tunnel_metadata,
+ },
+ )
+ self._persist_graph()
+
+ @_serialized_adapter_call
+ def ingest_answer_writeback(
+ self,
+ *,
+ query_text: str,
+ answer_text: str,
+ answer_id: str,
+ writeback_records: List[Dict[str, Any]],
+ trace: Dict[str, Any] | None = None,
+ ) -> List[str]:
+ self._reload_graph()
+ if not writeback_records:
+ self._last_writeback_summary = {"stored_record_ids": [], "promotion_events": []}
+ return []
+ turn_index = self.graph.next_turn()
+ records: List[SessionMemoryRecordV2] = []
+ writeback_classes: List[str] = []
+ for index, raw in enumerate(writeback_records):
+ if not isinstance(raw, dict):
+ continue
+ category = _clean_text(raw.get("category", "fact")) or "fact"
+ value = _clean_text(raw.get("value", ""))
+ raw_slot_key = _clean_text(raw.get("slot_key", "")) or _clean_text(raw.get("slot", ""))
+ anchors = _dedupe(raw.get("anchors", []) or [], max_items=8)
+ slot_key = self.profile.stable_slot_key(
+ category=category,
+ value=value,
+ anchors=anchors,
+ slot_key=raw_slot_key,
+ relation=_clean_text(raw.get("relation", "")),
+ metadata=dict(raw.get("metadata", {}) or {}),
+ )
+ if not value or not slot_key:
+ continue
+ raw_metadata = dict(raw.get("metadata", {}) or {})
+ writeback_class = _clean_text(raw_metadata.get("writeback_class", "")) or "fact"
+ writeback_classes.append(writeback_class)
+ metadata = {
+ **raw_metadata,
+ "memory_role": _clean_text(raw_metadata.get("memory_role", "")) or "assistant",
+ "authority": _clean_text(raw_metadata.get("authority", "")) or "derived",
+ "canonical_slot_key": _clean_text(raw_metadata.get("canonical_slot_key", "")) or slot_key.removeprefix("assistant.").split(".fact", 1)[0].split(".state_change", 1)[0].split(".high_conf_conclusion", 1)[0],
+ "writeback_class": writeback_class,
+ "origin_query": _clean_text(raw_metadata.get("origin_query", "")) or _clean_text(query_text),
+ "origin_answer_id": _clean_text(raw_metadata.get("origin_answer_id", "")) or answer_id,
+ "origin_answer_ids": _dedupe([*(raw_metadata.get("origin_answer_ids", []) or []), _clean_text(raw_metadata.get("origin_answer_id", "")) or answer_id]),
+ "support_memory_ids": _dedupe(raw_metadata.get("support_memory_ids", []) or []),
+ "support_fact_refs": _dedupe(raw_metadata.get("support_fact_refs", []) or []),
+ "support_path_refs": _dedupe(raw_metadata.get("support_path_refs", []) or []),
+ "promotion_state": _clean_text(raw_metadata.get("promotion_state", "")) or "candidate",
+ "answer_id": answer_id,
+ }
+ record = SessionMemoryRecordV2(
+ memory_id=f"{slot_key}:{turn_index}:assistant:{index}",
+ category=category,
+ slot_key=slot_key,
+ value=value,
+ relation=_clean_text(raw.get("relation", "")) or "assistant_memory",
+ anchor_concepts=anchors,
+ evidence_anchors=anchors,
+ salience=float(raw.get("salience", 0.62) or 0.62),
+ confidence=float(raw.get("confidence", 0.88) or 0.88),
+ source_kind=_clean_text(raw.get("source_kind", "")) or "assistant_writeback",
+ turn_index=turn_index,
+ state=_clean_text(raw.get("state", "")) or "active",
+ metadata=metadata,
+ )
+ records.append(record)
+ stored_ids = self.graph.add_records(records)
+ write_embedder_metadata = _apply_write_embedder_index_to_graph(
+ self.graph,
+ stored_ids=stored_ids,
+ turn_text=" ".join([_clean_text(query_text), _clean_text(answer_text)]),
+ turn_index=turn_index,
+ mode=self.write_embedder_index_mode,
+ max_terms=self.write_embedder_index_max_terms,
+ )
+ promotion_events = self._apply_writeback_promotions(stored_ids)
+ writeback_class = writeback_classes[0] if len(set(writeback_classes)) == 1 and writeback_classes else ("mixed" if writeback_classes else "")
+ self.graph.record_turn(
+ turn_kind="assistant_writeback" if stored_ids else "assistant_writeback_empty",
+ text=query_text,
+ turn_index=turn_index,
+ record_ids=stored_ids,
+ speaker="assistant",
+ assistant_text=answer_text,
+ writeback_class=writeback_class,
+ metadata={
+ "source": "assistant_writeback",
+ "answer_id": answer_id,
+ "trace": dict(trace or {}),
+ **write_embedder_metadata,
+ },
+ )
+ self._last_writeback_summary = {
+ "stored_record_ids": list(stored_ids),
+ "promotion_events": list(promotion_events),
+ **write_embedder_metadata,
+ }
+ self._persist_graph()
+ return stored_ids
+
+ @_serialized_adapter_call
+ def last_writeback_summary(self) -> Dict[str, Any]:
+ return dict(self._last_writeback_summary)
+
+ def _resolve_injection_planner_model_path(self) -> str:
+ explicit_path = _clean_text(self.injection_planner_model_path)
+ if explicit_path:
+ path = Path(explicit_path).expanduser()
+ if path.is_dir():
+ path = path / "injection_planner.pt"
+ return str(path)
+ latest_path = _clean_text(self.injection_planner_latest_path)
+ if not latest_path:
+ return ""
+ latest = Path(latest_path).expanduser()
+ if latest.is_dir():
+ return str(latest / "injection_planner.pt")
+ if latest.suffix == ".pt":
+ return str(latest)
+ try:
+ target = Path(latest.read_text(encoding="utf-8").strip()).expanduser()
+ except Exception as exc:
+ self._injection_planner_error = f"latest_pointer_read_failed: {exc}"
+ return ""
+ if target.is_dir():
+ target = target / "injection_planner.pt"
+ return str(target)
+
+ def _load_injection_planner(self) -> Any | None:
+ normalized_mode = _normalize(self.injection_planner_mode)
+ if normalized_mode in _INJECTION_PLANNER_DISABLED_MODES:
+ self._injection_planner_error = "disabled"
+ return None
+ if self._loaded_injection_planner is not None:
+ return self._loaded_injection_planner
+ torch_module = getattr(injection_planner_runtime, "torch", None)
+ if torch_module is None:
+ self._injection_planner_error = "torch_unavailable"
+ return None
+ model_path_text = self._resolve_injection_planner_model_path()
+ if not model_path_text:
+ if not self._injection_planner_error:
+ self._injection_planner_error = "model_path_missing"
+ return None
+ model_path = Path(model_path_text)
+ if not model_path.exists():
+ self._injection_planner_error = f"model_path_not_found: {model_path}"
+ return None
+ try:
+ device = torch_module.device(self.injection_planner_device or "cpu")
+ payload = torch_module.load(model_path, map_location=device, weights_only=True)
+ config = injection_planner_runtime.InjectionPlannerConfig.from_dict(dict(payload.get("config", {}) or {}))
+ model = injection_planner_runtime.InjectionPlannerModel(config).to(device)
+ state_dict = dict(payload.get("state_dict", {}) or {})
+ model.load_state_dict(state_dict, strict=False)
+ model.eval()
+ self._loaded_injection_planner = model
+ self._injection_planner_config = config
+ self._injection_planner_payload = dict(payload)
+ self._injection_planner_resolved_path = str(model_path)
+ self._injection_planner_error = ""
+ self._injection_planner_evidence_role_supported = any(
+ str(key).startswith("evidence_role_head.") for key in state_dict
+ )
+ thresholds = {"selection_threshold": 0.5, "row_threshold": 0.5, "logic_threshold": 0.5}
+ summary_path = model_path.parent / "train_summary.json"
+ if summary_path.exists():
+ try:
+ summary = json.loads(summary_path.read_text(encoding="utf-8"))
+ calibration = dict(summary.get("calibration", {}) or {})
+ logic_calibration = dict(summary.get("logic_calibration", {}) or {})
+ if calibration.get("selection_threshold") is not None:
+ thresholds["selection_threshold"] = float(calibration.get("selection_threshold"))
+ if calibration.get("row_threshold") is not None:
+ thresholds["row_threshold"] = float(calibration.get("row_threshold"))
+ if logic_calibration.get("logic_threshold") is not None:
+ thresholds["logic_threshold"] = float(logic_calibration.get("logic_threshold"))
+ except Exception:
+ pass
+ if self.injection_planner_selection_threshold_override >= 0.0:
+ thresholds["selection_threshold"] = float(self.injection_planner_selection_threshold_override)
+ if self.injection_planner_row_threshold_override >= 0.0:
+ thresholds["row_threshold"] = float(self.injection_planner_row_threshold_override)
+ if self.injection_planner_logic_threshold_override >= 0.0:
+ thresholds["logic_threshold"] = float(self.injection_planner_logic_threshold_override)
+ self._injection_planner_thresholds = thresholds
+ except Exception as exc:
+ self._loaded_injection_planner = None
+ self._injection_planner_config = None
+ self._injection_planner_payload = {}
+ self._injection_planner_resolved_path = str(model_path)
+ self._injection_planner_error = str(exc)
+ self._injection_planner_evidence_role_supported = False
+ return self._loaded_injection_planner
+
+ def _injection_planner_candidate_from_hit(
+ self,
+ query: str,
+ hit: MemoryHit,
+ *,
+ index: int,
+ current_turn_index: int,
+ ) -> Dict[str, Any]:
+ metadata = dict(hit.metadata or {})
+ category = _normalize(hit.category)
+ source_kind = _normalize(hit.source_kind)
+ logic_roles = [
+ _normalize(item)
+ for item in (
+ metadata.get("logic_roles", [])
+ if isinstance(metadata.get("logic_roles", []), list)
+ else [metadata.get("logic_roles", "")]
+ )
+ if _normalize(item) in set(injection_planner_runtime.LOGIC_ROLES)
+ ]
+ if category in set(injection_planner_runtime.LOGIC_ROLES):
+ logic_roles.append(category)
+ if _normalize(metadata.get("writeback_class", "")) in set(injection_planner_runtime.LOGIC_ROLES):
+ logic_roles.append(_normalize(metadata.get("writeback_class", "")))
+ if not logic_roles:
+ logic_roles = ["negative"] if _normalize(hit.state) in {"stale", "superseded", "false"} else ["evidence"]
+ evidence_snippet_role = _normalize(metadata.get("evidence_snippet_role", ""))
+ if bool(metadata.get("profile_layer")) or category == "profile" or source_kind.endswith("profile"):
+ layer = "profile"
+ elif category == "time" or float(metadata.get("temporal_score", 0.0) or 0.0) > 0.0:
+ layer = "temporal"
+ elif "resource" in logic_roles or _clean_text(metadata.get("resource_key", "")):
+ layer = "resource"
+ elif (
+ source_kind in {"path_tunnel", "path_support", "public_dialog_path"}
+ or evidence_snippet_role in {"selected_path_support", "path_tunnel_support"}
+ or bool(metadata.get("path_tunnel_node"))
+ ):
+ layer = "path_tunnel"
+ elif bool(metadata.get("topic_bucket_rerank")) or bool(metadata.get("topic_bucket_dialogue_tunnel_allowed")):
+ layer = "topic_tunnel"
+ else:
+ layer = "event"
+ normalized_state = _normalize(hit.state)
+ if normalized_state in {"stale", "superseded", "false"}:
+ temporal_state = "superseded"
+ elif bool(metadata.get("topic_bucket_current_subject_preserved")) or bool(metadata.get("current_subject_protected")):
+ temporal_state = "current"
+ elif current_turn_index and int(hit.turn_index or 0) and current_turn_index - int(hit.turn_index) <= 1:
+ temporal_state = "current"
+ elif normalized_state == "active":
+ temporal_state = "stable"
+ else:
+ temporal_state = "historical"
+ query_tokens = set(_tokenize(query))
+ hit_tokens = set(_tokenize(" ".join([hit.value, " ".join(hit.anchors), _clean_text(metadata.get("topic_label", ""))])))
+ overlap = len(query_tokens & hit_tokens) / max(1, len(query_tokens | hit_tokens))
+ candidate_id = _clean_text(hit.memory_id) or f"hit:{index}"
+ topic_label = _clean_text(metadata.get("topic_label", "")) or _clean_text(metadata.get("topic_bucket_query_label", ""))
+ profile_key = _clean_text(metadata.get("profile_type", "")) or _clean_text(metadata.get("semantic_slot", ""))
+ if not profile_key and layer in {"profile", "temporal", "topic_tunnel"}:
+ profile_key = topic_label
+ semantic_similarity = max(
+ 0.0,
+ min(
+ 1.0,
+ float(
+ metadata.get(
+ "semantic_similarity",
+ metadata.get(
+ "answer_window_semantic_similarity",
+ metadata.get(
+ "embedder_similarity",
+ metadata.get("dense_similarity", metadata.get("bge_m3_similarity", hit.score)),
+ ),
+ ),
+ )
+ or 0.0
+ ),
+ ),
+ )
+ return {
+ "id": candidate_id,
+ "text": hit.value,
+ "summary": _clean_text(metadata.get("source_turn_text", "")) or _clean_text(metadata.get("event_summary", "")),
+ "topic": topic_label,
+ "profile_key": profile_key,
+ "resource_key": _clean_text(metadata.get("resource_key", "")) or (_clean_text(hit.slot_key) if layer == "resource" else ""),
+ "layer": layer,
+ "temporal_state": temporal_state,
+ "logic_roles": _dedupe(logic_roles),
+ "query_overlap": round(float(overlap), 6),
+ "retrieval_score": max(0.0, min(1.0, float(hit.score or 0.0))),
+ "graph_score": max(
+ 0.0,
+ min(
+ 1.0,
+ float(
+ metadata.get(
+ "event_score",
+ metadata.get("recall_score", metadata.get("hybrid_score", hit.score)),
+ )
+ or 0.0
+ ),
+ ),
+ ),
+ "tunnel_score": max(
+ 0.0,
+ min(
+ 1.0,
+ max(
+ float(metadata.get("path_tunnel_support_score", 0.0) or 0.0),
+ float(metadata.get("event_tunnel_support_score", 0.0) or 0.0),
+ float(metadata.get("path_chain_extension_delta_score", 0.0) or 0.0),
+ ),
+ ),
+ ),
+ "topic_similarity": max(0.0, min(1.0, float(metadata.get("topic_bucket_overlap", 0.0) or 0.0))),
+ "semantic_similarity": semantic_similarity,
+ "confidence": max(0.0, min(1.0, float(metadata.get("confidence", hit.score or 0.0) or 0.0))),
+ "rank_score": round(1.0 / float(index + 1), 6),
+ "age_turns": max(0, int(current_turn_index) - int(hit.turn_index or 0)) if current_turn_index else 0,
+ "branch_depth": max(0, len(list(metadata.get("selected_path_ids", []) or []))),
+ "contradicts_current": normalized_state in {"stale", "superseded", "false"} or bool(metadata.get("contradicts_current")),
+ "is_current": temporal_state == "current",
+ }
+
+ def _apply_injection_planner_to_hits(
+ self,
+ query: str,
+ hits: Sequence[MemoryHit],
+ *,
+ top_k: int,
+ ) -> Dict[str, Any]:
+ normalized_mode = _normalize(self.injection_planner_mode)
+ base_metadata: Dict[str, Any] = {
+ "injection_planner_enabled": False,
+ "injection_planner_mode": normalized_mode or "observe",
+ }
+ if normalized_mode in _INJECTION_PLANNER_DISABLED_MODES:
+ base_metadata["injection_planner_reason"] = "disabled"
+ return {"hits": list(hits), "metadata": base_metadata}
+ if not hits:
+ base_metadata["injection_planner_reason"] = "no_hits"
+ return {"hits": list(hits), "metadata": base_metadata}
+ model = self._load_injection_planner()
+ if model is None or self._injection_planner_config is None:
+ base_metadata.update(
+ {
+ "injection_planner_reason": self._injection_planner_error or "load_failed",
+ "injection_planner_model_path": self._injection_planner_resolved_path,
+ }
+ )
+ return {"hits": list(hits), "metadata": base_metadata}
+ torch_module = getattr(injection_planner_runtime, "torch", None)
+ if torch_module is None:
+ base_metadata["injection_planner_reason"] = "torch_unavailable"
+ return {"hits": list(hits), "metadata": base_metadata}
+ current_turn_index = max(
+ [int(getattr(self.graph, "turn_index", 0) or 0), *[int(hit.turn_index or 0) for hit in hits]],
+ default=0,
+ )
+ candidates = [
+ self._injection_planner_candidate_from_hit(query, hit, index=index, current_turn_index=current_turn_index)
+ for index, hit in enumerate(hits)
+ ]
+ row = {"id": "runtime_injection_plan", "query": query, "candidates": candidates, "gold": {}}
+ try:
+ dataset = injection_planner_runtime.InjectionPlannerDataset([row], self._injection_planner_config)
+ batch = injection_planner_runtime.collate_injection_batch([dataset[0]])
+ device = next(model.parameters()).device
+ model_batch = {
+ key: value.to(device) if hasattr(value, "to") else value
+ for key, value in dict(batch).items()
+ }
+ with torch_module.no_grad():
+ outputs = model(model_batch["features"], model_batch["valid_mask"])
+ selection_scores = torch_module.sigmoid(outputs["selection_logits"])[0].detach().cpu().tolist()
+ should_inject_score = float(torch_module.sigmoid(outputs["should_inject_logits"])[0].detach().cpu().item())
+ mode_probs = torch_module.softmax(outputs["injection_mode_logits"], dim=-1)[0].detach().cpu()
+ mode_index = int(torch_module.argmax(mode_probs).item())
+ temporal_indices = torch_module.argmax(outputs["temporal_logits"], dim=-1)[0].detach().cpu().tolist()
+ logic_scores = torch_module.sigmoid(outputs["logic_logits"])[0].detach().cpu().tolist()
+ if bool(self._injection_planner_evidence_role_supported) and "evidence_role_logits" in outputs:
+ evidence_role_indices = torch_module.argmax(outputs["evidence_role_logits"], dim=-1)[0].detach().cpu().tolist()
+ else:
+ evidence_role_indices = [
+ injection_planner_runtime.EVIDENCE_ROLES.index("direct_answer")
+ for _ in candidates
+ ]
+ except Exception as exc:
+ base_metadata.update(
+ {
+ "injection_planner_reason": f"inference_failed: {exc}",
+ "injection_planner_model_path": self._injection_planner_resolved_path,
+ }
+ )
+ return {"hits": list(hits), "metadata": base_metadata}
+ thresholds = {
+ "selection_threshold": float(self._injection_planner_thresholds.get("selection_threshold", 0.5)),
+ "row_threshold": float(self._injection_planner_thresholds.get("row_threshold", 0.5)),
+ "logic_threshold": float(self._injection_planner_thresholds.get("logic_threshold", 0.5)),
+ }
+ injection_mode = injection_planner_runtime.INJECTION_MODES[mode_index]
+ row_allows_injection = should_inject_score >= thresholds["row_threshold"] and injection_mode != "none"
+ predictions: Dict[str, Dict[str, Any]] = {}
+ for index, candidate in enumerate(candidates):
+ logic_roles = [
+ role
+ for role, score in zip(injection_planner_runtime.LOGIC_ROLES, logic_scores[index])
+ if float(score) >= thresholds["logic_threshold"]
+ ]
+ evidence_role = injection_planner_runtime.EVIDENCE_ROLES[int(evidence_role_indices[index])]
+ role_allows_selection = evidence_role not in {"noise", "negative_evidence"}
+ predictions[candidate["id"]] = {
+ "selection_score": float(selection_scores[index]),
+ "selected": bool(
+ row_allows_injection
+ and role_allows_selection
+ and float(selection_scores[index]) >= thresholds["selection_threshold"]
+ ),
+ "temporal_state": injection_planner_runtime.TEMPORAL_STATES[int(temporal_indices[index])],
+ "evidence_role": evidence_role,
+ "logic_roles": logic_roles or ["evidence"],
+ "candidate_layer": candidate.get("layer", ""),
+ "role_allows_selection": bool(role_allows_selection),
+ }
+ annotated_hits: List[MemoryHit] = []
+ for index, hit in enumerate(hits):
+ candidate_id = _clean_text(hit.memory_id) or f"hit:{index}"
+ prediction = predictions.get(candidate_id, {})
+ metadata = dict(hit.metadata or {})
+ metadata.update(
+ {
+ "injection_planner_enabled": True,
+ "injection_planner_mode": normalized_mode or "observe",
+ "injection_planner_model_path": self._injection_planner_resolved_path,
+ "injection_planner_candidate_id": candidate_id,
+ "injection_planner_score": round(float(prediction.get("selection_score", 0.0)), 6),
+ "injection_planner_selected": bool(prediction.get("selected", False)),
+ "injection_planner_temporal_state": _clean_text(prediction.get("temporal_state", "")),
+ "injection_planner_evidence_role": _clean_text(prediction.get("evidence_role", "")),
+ "injection_planner_role_allows_selection": bool(prediction.get("role_allows_selection", False)),
+ "injection_planner_logic_roles": list(prediction.get("logic_roles", []) or []),
+ "injection_planner_candidate_layer": _clean_text(prediction.get("candidate_layer", "")),
+ "injection_planner_injection_mode": injection_mode,
+ "injection_planner_should_inject_score": round(float(should_inject_score), 6),
+ "injection_planner_thresholds": dict(thresholds),
+ "injection_planner_evidence_role_supported": bool(self._injection_planner_evidence_role_supported),
+ }
+ )
+ planner_score = float(prediction.get("selection_score", 0.0))
+ next_score = max(float(hit.score), planner_score) if bool(prediction.get("selected", False)) else float(hit.score)
+ annotated_hits.append(
+ MemoryHit(
+ memory_id=hit.memory_id,
+ category=hit.category,
+ value=hit.value,
+ relation=hit.relation,
+ anchors=list(hit.anchors),
+ score=round(next_score, 6),
+ source_kind=hit.source_kind,
+ slot_key=hit.slot_key,
+ state=hit.state,
+ turn_index=int(hit.turn_index),
+ metadata=metadata,
+ )
+ )
+ selected_count = sum(1 for item in annotated_hits if bool((item.metadata or {}).get("injection_planner_selected")))
+ if normalized_mode in _INJECTION_PLANNER_FORCE_MODES and selected_count:
+ selected_ids = {hit.memory_id for hit in annotated_hits if bool((hit.metadata or {}).get("injection_planner_selected"))}
+ planned_hits = [hit for hit in annotated_hits if hit.memory_id in selected_ids]
+ elif normalized_mode in _INJECTION_PLANNER_GUIDED_MODES and selected_count:
+ planned_hits = sorted(
+ annotated_hits,
+ key=lambda hit: (
+ not bool((hit.metadata or {}).get("injection_planner_selected")),
+ -float((hit.metadata or {}).get("injection_planner_score", 0.0) or 0.0),
+ -float(hit.score or 0.0),
+ ),
+ )
+ else:
+ planned_hits = annotated_hits
+ base_metadata.update(
+ {
+ "injection_planner_enabled": True,
+ "injection_planner_reason": "ok",
+ "injection_planner_model_path": self._injection_planner_resolved_path,
+ "injection_planner_candidate_count": len(candidates),
+ "injection_planner_selected_count": int(selected_count),
+ "injection_planner_should_inject_score": round(float(should_inject_score), 6),
+ "injection_planner_injection_mode": injection_mode,
+ "injection_planner_thresholds": dict(thresholds),
+ "injection_planner_evidence_role_supported": bool(self._injection_planner_evidence_role_supported),
+ "injection_planner_guided": bool(normalized_mode in _INJECTION_PLANNER_GUIDED_MODES | _INJECTION_PLANNER_FORCE_MODES),
+ "injection_planner_prediction_ids": [
+ candidate_id
+ for candidate_id, prediction in predictions.items()
+ if bool(prediction.get("selected", False))
+ ],
+ }
+ )
+ return {"hits": planned_hits[: max(len(planned_hits), top_k)], "metadata": base_metadata}
+
+ def _node_scorer(self) -> LoadedNodeMemoryScorer | None:
+ if self.retrieval_mode != "hybrid_node_scored":
+ return None
+ if self._loaded_node_scorer is not None:
+ return self._loaded_node_scorer
+ if not self.node_model_path:
+ self._node_scorer_error = "node_model_path_missing"
+ return None
+ try:
+ self._loaded_node_scorer = LoadedNodeMemoryScorer(
+ node_model_path=Path(self.node_model_path),
+ path_model_path=Path(self.path_model_path) if self.path_model_path else None,
+ device=self.node_model_device or None,
+ )
+ except Exception as exc:
+ self._node_scorer_error = str(exc)
+ self._loaded_node_scorer = None
+ return self._loaded_node_scorer
+
+ def _apply_writeback_promotions(self, stored_ids: Sequence[str]) -> List[Dict[str, Any]]:
+ promotion_events: List[Dict[str, Any]] = []
+ for memory_id in stored_ids:
+ record = self.graph.records_by_id.get(memory_id)
+ if record is None or not isinstance(record.metadata, dict):
+ continue
+ if _normalize(record.metadata.get("memory_role", "")) != "assistant" or _normalize(record.metadata.get("authority", "")) != "derived":
+ continue
+ canonical_slot_key = _clean_text(record.metadata.get("canonical_slot_key", ""))
+ writeback_class = _clean_text(record.metadata.get("writeback_class", ""))
+ if not canonical_slot_key or not writeback_class:
+ continue
+ support_refs = self._support_ref_union(record.metadata)
+ confidence = float(record.confidence or 0.0)
+ same_records = [
+ item
+ for item in self.graph.records_by_id.values()
+ if isinstance(item.metadata, dict)
+ and _normalize(item.metadata.get("memory_role", "")) == "assistant"
+ and _normalize(item.metadata.get("canonical_slot_key", "")) == _normalize(canonical_slot_key)
+ and _normalize(item.metadata.get("writeback_class", "")) == _normalize(writeback_class)
+ and _normalize(item.value) == _normalize(record.value)
+ ]
+ qualifying = [item for item in same_records if float(item.confidence or 0.0) >= 0.9]
+ distinct_answers = {
+ _clean_text(answer_id)
+ for item in qualifying
+ for answer_id in [*list(item.metadata.get("origin_answer_ids", []) or []), _clean_text(item.metadata.get("origin_answer_id", ""))]
+ if _clean_text(answer_id)
+ }
+ aggregated_support = set()
+ for item in same_records:
+ aggregated_support.update(self._support_ref_union(item.metadata))
+ fast_promotion = writeback_class in {"fact", "state_change"} and confidence >= 0.97 and len(support_refs) >= 3
+ standard_promotion = len(distinct_answers) >= 2 and len(aggregated_support) >= 2
+ if not (fast_promotion or standard_promotion):
+ record.metadata["promotion_state"] = "candidate"
+ continue
+ source_head = self._source_head_for_canonical(canonical_slot_key)
+ blocked_conflict = source_head is not None and _normalize(source_head.value) != _normalize(record.value)
+ promoted_slot = f"promoted.{canonical_slot_key}"
+ promoted_metadata = {
+ **dict(record.metadata or {}),
+ "memory_role": "assistant",
+ "authority": "promoted",
+ "canonical_slot_key": canonical_slot_key,
+ "writeback_class": writeback_class,
+ "promotion_state": "blocked_conflict" if blocked_conflict else "promoted",
+ "support_memory_ids": sorted({*list(record.metadata.get("support_memory_ids", []) or []), *[ref for ref in aggregated_support if ref.startswith("fact:") is False and ref.startswith("path:") is False]}),
+ "support_fact_refs": sorted({*list(record.metadata.get("support_fact_refs", []) or []), *[ref for ref in aggregated_support if ref.startswith("fact:")]}),
+ "support_path_refs": sorted({*list(record.metadata.get("support_path_refs", []) or []), *[ref for ref in aggregated_support if ref.startswith("path:")]}),
+ }
+ promoted_record = SessionMemoryRecordV2(
+ memory_id=f"{promoted_slot}:{record.turn_index}:promoted",
+ category=record.category,
+ slot_key=promoted_slot,
+ value=record.value,
+ relation=record.relation,
+ anchor_concepts=list(record.anchor_concepts),
+ evidence_anchors=list(record.evidence_anchors),
+ salience=max(float(record.salience), 0.78),
+ confidence=max(float(record.confidence), 0.9),
+ source_kind=f"promoted_{record.source_kind}",
+ turn_index=int(record.turn_index),
+ state="active",
+ metadata=promoted_metadata,
+ )
+ promoted_ids = self.graph.add_records([promoted_record])
+ record.metadata["promotion_state"] = "blocked_conflict" if blocked_conflict else "promoted"
+ promotion_events.append(
+ {
+ "source_memory_id": record.memory_id,
+ "promoted_record_ids": list(promoted_ids),
+ "canonical_slot_key": canonical_slot_key,
+ "writeback_class": writeback_class,
+ "promotion_state": record.metadata["promotion_state"],
+ "blocked_conflict": bool(blocked_conflict),
+ }
+ )
+ return promotion_events
+
+ def _source_head_for_canonical(self, canonical_slot_key: str) -> SessionMemoryRecordV2 | None:
+ head_id = self.graph.slot_heads.get(canonical_slot_key)
+ if not head_id:
+ return None
+ record = self.graph.records_by_id.get(head_id)
+ if record is None or not isinstance(record.metadata, dict):
+ return None
+ if _normalize(record.metadata.get("memory_role", "")) != "user" or _normalize(record.metadata.get("authority", "")) != "source":
+ return None
+ return record
+
+ def _support_ref_union(self, metadata: Dict[str, Any]) -> set[str]:
+ return {
+ *[_clean_text(item) for item in list(metadata.get("support_memory_ids", []) or []) if _clean_text(item)],
+ *[_clean_text(item) for item in list(metadata.get("support_fact_refs", []) or []) if _clean_text(item)],
+ *[_clean_text(item) for item in list(metadata.get("support_path_refs", []) or []) if _clean_text(item)],
+ }
+
+ def _hybrid_node_scored_hits(
+ self,
+ query: str,
+ hits: Sequence[MemoryHit],
+ *,
+ top_k: int,
+ public_hits: Sequence[MemoryHit] | None = None,
+ ) -> Dict[str, Any]:
+ hybrid_profile_enabled = _normalize(os.getenv("TMCRA_HYBRID_RUNTIME_PROFILE", "")) in {
+ "1",
+ "true",
+ "yes",
+ "on",
+ }
+ hybrid_profile_started = time.perf_counter()
+ hybrid_profile_last = hybrid_profile_started
+ hybrid_profile: Dict[str, float] = {}
+
+ def _hybrid_mark(name: str) -> None:
+ nonlocal hybrid_profile_last
+ if not hybrid_profile_enabled:
+ return
+ now = time.perf_counter()
+ hybrid_profile[name] = round(now - hybrid_profile_last, 6)
+ hybrid_profile_last = now
+
+ def _hybrid_profile_payload() -> Dict[str, Any]:
+ if not hybrid_profile_enabled:
+ return {}
+ return {
+ "enabled": True,
+ **hybrid_profile,
+ "total_sec": round(time.perf_counter() - hybrid_profile_started, 6),
+ "runtime_graph_cache_hit": bool(runtime_graph_cache_hit),
+ "source_hit_count": int(len(source_hits)),
+ "runtime_event_count": int(len(candidate_event_ids)),
+ "runtime_path_count": int(len(list(runtime_graph.get("paths", []) or []))),
+ "hybrid_candidate_event_count": int(len(hybrid_candidate_event_ids)),
+ "selected_event_count": int(len(selected_event_ids)),
+ "selected_path_count": int(len(selected_path_ids)),
+ "final_hit_count": int(len(final_hits)),
+ }
+
+ scorer = self._node_scorer()
+ _hybrid_mark("scorer_init_sec")
+ if scorer is None:
+ return {
+ "hits": list(hits),
+ "metadata": {
+ "retrieval_mode": "heuristic",
+ "hybrid_enabled": False,
+ "hybrid_error": self._node_scorer_error,
+ },
+ }
+ runtime_graph_cache_key = self._runtime_graph_static_cache_key()
+ runtime_graph_cache_hit = bool(
+ runtime_graph_cache_key == self._runtime_graph_cache_key
+ and self._runtime_graph_cache_source_hits is not None
+ and self._runtime_graph_cache_payload is not None
+ and self._runtime_graph_cache_symbolic_index is not None
+ )
+ if runtime_graph_cache_hit:
+ source_hits = self._runtime_graph_cache_source_hits
+ else:
+ source_hits = _learnable_graph_hits(self.graph)
+ _hybrid_mark("source_hits_sec")
+ if not source_hits:
+ return {
+ "hits": list(hits),
+ "metadata": {
+ "retrieval_mode": "heuristic",
+ "hybrid_enabled": False,
+ "hybrid_error": "no_learnable_hits",
+ },
+ }
+ has_public_hits = any(
+ _is_public_dialog_hit(hit)
+ and not bool((hit.metadata or {}).get("profile_layer"))
+ for hit in source_hits
+ )
+ has_generic_hits = any(not _is_public_dialog_hit(hit) for hit in source_hits)
+ hybrid_source = "mixed_full_graph"
+ if has_public_hits and not has_generic_hits:
+ hybrid_source = "public_full_graph"
+ elif has_generic_hits and not has_public_hits:
+ hybrid_source = "generic_full_graph"
+ if runtime_graph_cache_hit:
+ runtime_graph = dict(self._runtime_graph_cache_payload or {})
+ runtime_graph["query"] = query
+ symbolic_recall_index = self._runtime_graph_cache_symbolic_index or {}
+ else:
+ runtime_graph = _build_runtime_graph_from_hits(query, source_hits)
+ grouped_hits = dict(runtime_graph.get("grouped_hits", {}) or {})
+ symbolic_recall_index = _build_symbolic_recall_index(
+ runtime_graph,
+ grouped_hits=grouped_hits,
+ )
+ cached_runtime_graph = dict(runtime_graph)
+ cached_runtime_graph["query"] = ""
+ self._runtime_graph_cache_key = runtime_graph_cache_key
+ self._runtime_graph_cache_source_hits = source_hits
+ self._runtime_graph_cache_payload = cached_runtime_graph
+ self._runtime_graph_cache_symbolic_index = symbolic_recall_index
+ grouped_hits = dict(runtime_graph.get("grouped_hits", {}) or {})
+ _hybrid_mark("runtime_graph_build_sec")
+ profile_first_payload = _profile_first_hybrid_rescue(
+ self.graph,
+ query,
+ grouped_hits=grouped_hits,
+ top_k=top_k,
+ )
+ profile_first_hits = list(profile_first_payload.get("hits", []) or [])
+ profile_first_event_ids = list(profile_first_payload.get("event_ids", []) or [])
+ profile_first_memory_ids = list(profile_first_payload.get("memory_ids", []) or [])
+ _hybrid_mark("profile_first_rescue_sec")
+ candidate_event_ids = sorted(grouped_hits.keys())
+ if not candidate_event_ids:
+ return {
+ "hits": list(hits),
+ "metadata": {
+ "retrieval_mode": "heuristic",
+ "hybrid_enabled": False,
+ "hybrid_error": "no_event_candidates",
+ },
+ }
+ question_analysis = extract_question_features(query)
+ hybrid_candidate_limit = min(
+ len(candidate_event_ids),
+ max(
+ _HYBRID_SELECTED_EVENT_FLOOR,
+ int(self.candidate_event_k) * 4,
+ int(self.support_path_k) * 4,
+ int(top_k) * 4,
+ ),
+ )
+ embedder_pre_recall_mode = _normalize(getattr(self, "embedder_pre_recall_mode", ""))
+ embedder_pre_recall_enabled = embedder_pre_recall_mode not in _EMBEDDER_INDEX_DISABLED_MODES
+ embedder_pre_recall_index_mode = self.embedder_index_recall_mode
+ if embedder_pre_recall_mode not in {"1", "true", "yes", "on", "auto", "seed", "candidate", "candidates"}:
+ embedder_pre_recall_index_mode = embedder_pre_recall_mode
+ pre_embedder_index_payload: Dict[str, Any] = {
+ "event_ids": [],
+ "metadata": {
+ "embedder_pre_recall_enabled": False,
+ "embedder_pre_recall_mode": embedder_pre_recall_mode or "off",
+ "embedder_pre_recall_index_mode": embedder_pre_recall_index_mode or "off",
+ "embedder_pre_recall_event_ids": [],
+ },
+ }
+ pre_embedder_event_ids: List[str] = []
+ pre_candidate_event_ids: List[str] = []
+ if embedder_pre_recall_enabled:
+ pre_embedder_index_payload = _embedder_index_recall_event_ids(
+ query,
+ grouped_hits=grouped_hits,
+ mode=embedder_pre_recall_index_mode,
+ limit=self.embedder_pre_recall_k or self.embedder_index_recall_k or hybrid_candidate_limit,
+ max_terms=self.write_embedder_index_max_terms,
+ )
+ pre_embedder_event_ids = list(pre_embedder_index_payload.get("event_ids", []) or [])
+ pre_candidate_event_ids = _bounded_event_id_union(
+ pre_embedder_event_ids,
+ max_items=hybrid_candidate_limit,
+ )
+ _hybrid_mark("query_and_pre_recall_prep_sec")
+ pre_score_kwargs = {
+ "graph": runtime_graph,
+ "question": query,
+ "question_features": question_analysis,
+ "rerank_top_k": self.candidate_event_k,
+ "event_rerank_mode": self.event_rerank_mode,
+ "matrix_event_top_k": self.matrix_event_top_k,
+ "support_path_k": self.support_path_k,
+ "top_k": top_k,
+ }
+ if pre_candidate_event_ids:
+ pre_score_kwargs["candidate_event_ids"] = pre_candidate_event_ids
+ scored = _call_with_supported_kwargs(
+ scorer.score_runtime,
+ **pre_score_kwargs,
+ )
+ _hybrid_mark("initial_score_runtime_sec")
+ memory_router_decision = _memory_router_decision(
+ scored,
+ mode=self.memory_router_mode,
+ threshold=self.memory_router_threshold,
+ margin=self.memory_router_margin,
+ )
+ profile_first_router_suppressed = False
+ if not _memory_router_allows(memory_router_decision, "profile", "resource"):
+ profile_first_hits = []
+ profile_first_event_ids = []
+ profile_first_memory_ids = []
+ profile_first_router_suppressed = True
+ initial_recall_event_scores = dict(scored.get("recall_event_scores", {}) or {})
+ model_recall_event_ids = _bounded_event_id_union(
+ list(scored.get("recall_event_ids", []) or []),
+ [
+ event_id
+ for event_id, _ in sorted(
+ initial_recall_event_scores.items(),
+ key=lambda item: (-float(item[1]), item[0]),
+ )
+ ],
+ list(scored.get("rerank_candidate_event_ids", []) or []),
+ max_items=max(1, len(candidate_event_ids)),
+ )
+ learned_recall_event_ids = list(model_recall_event_ids)
+ _hybrid_mark("router_and_learned_recall_sec")
+ symbolic_recall_event_ids = _symbolic_recall_event_ids(
+ query,
+ runtime_graph,
+ grouped_hits=grouped_hits,
+ limit=hybrid_candidate_limit,
+ symbolic_index=symbolic_recall_index,
+ )
+ symbolic_recall_event_ids = _bounded_event_id_union(
+ profile_first_event_ids,
+ symbolic_recall_event_ids,
+ max_items=hybrid_candidate_limit,
+ )
+ _hybrid_mark("symbolic_recall_sec")
+ if pre_embedder_event_ids:
+ embedder_index_payload = pre_embedder_index_payload
+ else:
+ embedder_index_payload = _embedder_index_recall_event_ids(
+ query,
+ grouped_hits=grouped_hits,
+ mode=self.embedder_index_recall_mode,
+ limit=self.embedder_index_recall_k or hybrid_candidate_limit,
+ max_terms=self.write_embedder_index_max_terms,
+ )
+ embedder_index_event_ids = list(embedder_index_payload.get("event_ids", []) or [])
+ embedder_index_metadata = dict(embedder_index_payload.get("metadata", {}) or {})
+ embedder_index_metadata.update(
+ {
+ "embedder_pre_recall_enabled": bool(pre_candidate_event_ids),
+ "embedder_pre_recall_mode": embedder_pre_recall_mode or "off",
+ "embedder_pre_recall_index_mode": embedder_pre_recall_index_mode or "off",
+ "embedder_pre_recall_event_ids": list(pre_embedder_event_ids),
+ "embedder_pre_recall_candidate_event_ids": list(pre_candidate_event_ids),
+ "embedder_pre_recall_candidate_count": int(len(pre_candidate_event_ids)),
+ }
+ )
+ _hybrid_mark("embedder_index_recall_sec")
+ hybrid_candidate_event_ids = _bounded_event_id_union(
+ profile_first_event_ids,
+ embedder_index_event_ids,
+ learned_recall_event_ids,
+ symbolic_recall_event_ids,
+ max_items=hybrid_candidate_limit,
+ )
+ learned_candidate_event_ids = _bounded_event_id_union(
+ learned_recall_event_ids,
+ max_items=hybrid_candidate_limit,
+ )
+ learned_recall_event_id_set = set(learned_candidate_event_ids)
+ hybrid_candidate_union_added_event_ids = [
+ event_id
+ for event_id in hybrid_candidate_event_ids
+ if event_id not in learned_recall_event_id_set
+ ]
+ hybrid_candidate_union_priority_changed = list(hybrid_candidate_event_ids) != list(learned_candidate_event_ids)
+ hybrid_candidate_union_rescored = False
+ _hybrid_mark("candidate_union_finalize_sec")
+ if hybrid_candidate_union_added_event_ids or (embedder_index_event_ids and hybrid_candidate_union_priority_changed):
+ scored = _call_with_supported_kwargs(
+ scorer.score_runtime,
+ graph=runtime_graph,
+ question=query,
+ question_features=question_analysis,
+ candidate_event_ids=hybrid_candidate_event_ids,
+ rerank_top_k=self.candidate_event_k,
+ event_rerank_mode=self.event_rerank_mode,
+ matrix_event_top_k=self.matrix_event_top_k,
+ support_path_k=self.support_path_k,
+ top_k=top_k,
+ )
+ hybrid_candidate_union_rescored = True
+ memory_router_decision = _memory_router_decision(
+ scored,
+ mode=self.memory_router_mode,
+ threshold=self.memory_router_threshold,
+ margin=self.memory_router_margin,
+ )
+ _hybrid_mark("optional_rescore_sec")
+ deepseek_graph_teacher_payload: Dict[str, Any] = {}
+ deepseek_graph_teacher_mode = _deepseek_graph_teacher_mode()
+ if deepseek_graph_teacher_mode != "off":
+ teacher_scored = _deepseek_graph_teacher_call(
+ runtime_graph,
+ query,
+ candidate_event_ids=hybrid_candidate_event_ids,
+ top_k=top_k,
+ support_path_k=self.support_path_k,
+ local_scored=scored,
+ )
+ deepseek_graph_teacher_payload = dict(teacher_scored.get("deepseek_graph_model", {}) or {})
+ teacher_has_scores = bool(
+ teacher_scored.get("event_scores")
+ or teacher_scored.get("path_scores")
+ or teacher_scored.get("selected_event_ids")
+ or teacher_scored.get("selected_path_ids")
+ )
+ if teacher_has_scores and deepseek_graph_teacher_mode == "replace":
+ scored = _deepseek_graph_teacher_replace_preserve_local(scored, teacher_scored)
+ memory_router_decision = _memory_router_decision(
+ scored,
+ mode=self.memory_router_mode,
+ threshold=self.memory_router_threshold,
+ margin=self.memory_router_margin,
+ )
+ elif teacher_has_scores and deepseek_graph_teacher_mode == "fusion":
+ scored = _deepseek_graph_teacher_merge(scored, teacher_scored)
+ deepseek_graph_teacher_payload = dict(scored.get("deepseek_graph_model", {}) or deepseek_graph_teacher_payload)
+ memory_router_decision = _memory_router_decision(
+ scored,
+ mode=self.memory_router_mode,
+ threshold=self.memory_router_threshold,
+ margin=self.memory_router_margin,
+ )
+ elif deepseek_graph_teacher_mode == "replace":
+ fail_closed = _normalize(os.getenv("TMCRA_DEEPSEEK_GRAPH_MODEL_FAIL_CLOSED", "on")) not in {
+ "off",
+ "none",
+ "false",
+ "0",
+ "fallback",
+ "allow_fallback",
+ }
+ if fail_closed:
+ status = _clean_text(deepseek_graph_teacher_payload.get("status", "")) or "missing_scores"
+ detail = _clean_text(
+ deepseek_graph_teacher_payload.get("error", "")
+ or deepseek_graph_teacher_payload.get("raw_preview", "")
+ or deepseek_graph_teacher_payload.get("finish_reason", "")
+ )
+ raise RuntimeError(
+ f"DeepSeek graph teacher replace failed without usable graph scores: status={status}; detail={detail[:260]}"
+ )
+ if deepseek_graph_teacher_payload:
+ scored = dict(scored)
+ scored["deepseek_graph_model"] = dict(deepseek_graph_teacher_payload)
+ elif deepseek_graph_teacher_payload:
+ scored = dict(scored)
+ scored["deepseek_graph_model"] = dict(deepseek_graph_teacher_payload)
+ _hybrid_mark("graph_teacher_sec")
+ recall_event_scores = dict(scored.get("recall_event_scores", {}) or {})
+ rerank_candidate_event_ids = list(scored.get("rerank_candidate_event_ids", []) or [])
+ base_event_scores = dict(scored.get("base_event_scores", {}) or {})
+ rerank_event_scores = dict(scored.get("rerank_event_scores", {}) or {})
+ calibrated_event_scores = dict(scored.get("calibrated_event_scores", {}) or {})
+ matrix_event_scores = dict(scored.get("matrix_event_scores", {}) or {})
+ event_fusion_delta_scores = dict(scored.get("event_fusion_delta_scores", {}) or {})
+ event_tunnel_support_scores = dict(scored.get("event_tunnel_support_scores", {}) or {})
+ event_tunnel_delta_scores = dict(scored.get("event_tunnel_delta_scores", {}) or {})
+ tri_maze_event_reverse_scores = dict(scored.get("tri_maze_event_reverse_scores", {}) or {})
+ tri_maze_event_boundary_scores = dict(scored.get("tri_maze_event_boundary_scores", {}) or {})
+ tri_maze_event_reverse_relations = dict(scored.get("tri_maze_event_reverse_relations", {}) or {})
+ matrix_rerank_event_ids = list(scored.get("matrix_rerank_event_ids", []) or [])
+ matrix_enabled = bool(scored.get("matrix_enabled", False))
+ rerank_path_scores = dict(scored.get("rerank_path_scores", {}) or {})
+ matrix_path_scores = dict(scored.get("matrix_path_scores", {}) or {})
+ tri_maze_path_reverse_scores = dict(scored.get("tri_maze_path_reverse_scores", {}) or {})
+ tri_maze_path_boundary_scores = dict(scored.get("tri_maze_path_boundary_scores", {}) or {})
+ tri_maze_path_reverse_relations = dict(scored.get("tri_maze_path_reverse_relations", {}) or {})
+ matrix_path_rerank_ids = list(scored.get("matrix_path_rerank_ids", []) or [])
+ matrix_path_enabled = bool(scored.get("matrix_path_enabled", False))
+ fusion_enabled = bool(scored.get("fusion_enabled", False))
+ event_fusion_enabled = bool(scored.get("event_fusion_enabled", fusion_enabled))
+ path_fusion_enabled = bool(scored.get("path_fusion_enabled", fusion_enabled))
+ event_calibration_enabled = bool(scored.get("event_calibration_enabled", False))
+ path_calibration_enabled = bool(scored.get("path_calibration_enabled", False))
+ event_tunnel_enabled = bool(scored.get("event_tunnel_enabled", False))
+ path_tunnel_enabled = bool(scored.get("path_tunnel_enabled", False))
+ final_event_fusion_enabled = bool(scored.get("final_event_fusion_enabled", False))
+ final_path_fusion_enabled = bool(scored.get("final_path_fusion_enabled", False))
+ decision_fusion_enabled = bool(scored.get("decision_fusion_enabled", False))
+ decision_score_source = _clean_text(scored.get("decision_score_source", ""))
+ event_scores = dict(scored.get("event_scores", {}) or {})
+ base_path_scores = dict(scored.get("base_path_scores", {}) or {})
+ calibrated_path_scores = dict(scored.get("calibrated_path_scores", {}) or {})
+ path_fusion_delta_scores = dict(scored.get("path_fusion_delta_scores", {}) or {})
+ path_tunnel_support_scores = dict(scored.get("path_tunnel_support_scores", {}) or {})
+ path_tunnel_delta_scores = dict(scored.get("path_tunnel_delta_scores", {}) or {})
+ path_model_scores = dict(scored.get("path_model_scores", {}) or {})
+ path_chain_extension_delta_scores = dict(scored.get("path_chain_extension_delta_scores", {}) or {})
+ path_chain_extended_scores = dict(scored.get("path_chain_extended_scores", {}) or {})
+ path_chain_extension_enabled = bool(scored.get("path_chain_extension_enabled", False))
+ answer_type_scores = dict(scored.get("answer_type_scores", {}) or {})
+ selected_event_ids_from_model = list(scored.get("selected_event_ids", []) or [])
+ selected_path_ids_from_model = list(scored.get("selected_path_ids", []) or [])
+ focused_answer_type_from_model = _clean_text(scored.get("focused_answer_type", ""))
+ path_scores = dict(scored.get("path_scores", {}) or {})
+ temporal_scores = dict(scored.get("temporal_scores", {}) or {})
+ runtime_paths = {_clean_text(path.get("id", "")): dict(path) for path in list(runtime_graph.get("paths", []) or [])}
+ answer_plan_scores_raw = dict(scored.get("answer_plan_scores", {}) or {})
+
+ def _answer_plan_score_map(role: str) -> Dict[str, float]:
+ raw_scores = answer_plan_scores_raw.get(role, {})
+ if not isinstance(raw_scores, Mapping):
+ return {}
+ normalized: Dict[str, float] = {}
+ for raw_event_id, raw_score in raw_scores.items():
+ event_id = _clean_text(raw_event_id)
+ if not event_id:
+ continue
+ try:
+ normalized[event_id] = float(raw_score or 0.0)
+ except (TypeError, ValueError):
+ normalized[event_id] = 0.0
+ return normalized
+
+ answer_plan_selected_scores = _answer_plan_score_map("selected")
+ answer_plan_current_scores = _answer_plan_score_map("current")
+ answer_plan_historical_scores = _answer_plan_score_map("historical")
+ answer_plan_suppressed_scores = _answer_plan_score_map("suppressed")
+ answer_plan_scores = {
+ "selected": dict(answer_plan_selected_scores),
+ "current": dict(answer_plan_current_scores),
+ "historical": dict(answer_plan_historical_scores),
+ "suppressed": dict(answer_plan_suppressed_scores),
+ }
+ try:
+ answer_plan_event_selection_threshold = float(
+ os.getenv("TMCRA_ANSWER_PLAN_EVENT_SELECTION_THRESHOLD", "0.50") or 0.50
+ )
+ except (TypeError, ValueError):
+ answer_plan_event_selection_threshold = 0.50
+ try:
+ answer_plan_event_selection_top_k = int(os.getenv("TMCRA_ANSWER_PLAN_EVENT_SELECTION_TOP_K", "0") or 0)
+ except (TypeError, ValueError):
+ answer_plan_event_selection_top_k = 0
+ if answer_plan_event_selection_top_k <= 0:
+ answer_plan_event_selection_top_k = max(top_k, self.support_path_k * 2, _HYBRID_SELECTED_EVENT_FLOOR)
+ answer_plan_available_event_ids = {
+ _clean_text(event_id)
+ for event_id in [
+ *list(grouped_hits.keys()),
+ *[_clean_text(path.get("event_id", "")) for path in runtime_paths.values()],
+ ]
+ if _clean_text(event_id)
+ }
+ answer_plan_event_rows: List[tuple[str, float, float, float, float, float]] = []
+ answer_plan_event_ids = set(answer_plan_selected_scores) | set(answer_plan_current_scores) | set(answer_plan_historical_scores)
+ for event_id in answer_plan_event_ids:
+ if answer_plan_available_event_ids and event_id not in answer_plan_available_event_ids:
+ continue
+ selected_score = float(answer_plan_selected_scores.get(event_id, 0.0) or 0.0)
+ current_score = float(answer_plan_current_scores.get(event_id, 0.0) or 0.0)
+ historical_score = float(answer_plan_historical_scores.get(event_id, 0.0) or 0.0)
+ suppressed_score = float(answer_plan_suppressed_scores.get(event_id, 0.0) or 0.0)
+ support_score = max(selected_score, current_score)
+ adjusted_score = support_score - max(0.0, suppressed_score - support_score) * 0.5
+ if support_score < answer_plan_event_selection_threshold or suppressed_score > support_score:
+ continue
+ answer_plan_event_rows.append(
+ (event_id, adjusted_score, support_score, selected_score, current_score, historical_score)
+ )
+ answer_plan_event_rows.sort(
+ key=lambda row: (-float(row[1]), -float(row[2]), -float(row[3]), -float(row[4]), row[0])
+ )
+ answer_plan_raw_ranked_event_ids = [
+ event_id for event_id, *_ in answer_plan_event_rows[: max(1, answer_plan_event_selection_top_k)]
+ ]
+ try:
+ answer_plan_promotion_min_margin = float(
+ os.getenv("TMCRA_ANSWER_PLAN_PROMOTION_MIN_MARGIN", "0.02") or 0.02
+ )
+ except (TypeError, ValueError):
+ answer_plan_promotion_min_margin = 0.02
+ answer_plan_promotion_score_margin = 0.0
+ if len(answer_plan_event_rows) == 1:
+ answer_plan_promotion_score_margin = float(answer_plan_event_rows[0][1])
+ answer_plan_promotion_enabled = True
+ elif len(answer_plan_event_rows) > 1:
+ comparison_index = min(len(answer_plan_event_rows) - 1, max(1, min(answer_plan_event_selection_top_k, 5) - 1))
+ answer_plan_promotion_score_margin = float(answer_plan_event_rows[0][1]) - float(answer_plan_event_rows[comparison_index][1])
+ answer_plan_promotion_enabled = answer_plan_promotion_score_margin >= answer_plan_promotion_min_margin
+ else:
+ answer_plan_promotion_enabled = False
+ answer_plan_ranked_event_ids = list(answer_plan_raw_ranked_event_ids if answer_plan_promotion_enabled else [])
+ answer_plan_selected_event_ids = [
+ event_id
+ for event_id in answer_plan_ranked_event_ids
+ if float(answer_plan_selected_scores.get(event_id, 0.0) or 0.0)
+ >= answer_plan_event_selection_threshold
+ ]
+ answer_plan_current_event_ids = [
+ event_id
+ for event_id in answer_plan_ranked_event_ids
+ if float(answer_plan_current_scores.get(event_id, 0.0) or 0.0)
+ >= answer_plan_event_selection_threshold
+ ]
+ answer_plan_support_scores = {
+ event_id: round(float(support_score), 6)
+ for event_id, _, support_score, *_ in answer_plan_event_rows
+ }
+ answer_plan_adjusted_scores = {
+ event_id: round(float(adjusted_score), 6)
+ for event_id, adjusted_score, *_ in answer_plan_event_rows
+ }
+ answer_plan_rank_lookup = {
+ event_id: rank for rank, event_id in enumerate(answer_plan_ranked_event_ids, start=1)
+ }
+ answer_plan_ranked_event_id_set = set(answer_plan_ranked_event_ids)
+
+ def _answer_plan_hit_metadata(event_id: str) -> Dict[str, Any]:
+ clean_event_id = _clean_text(event_id)
+ selected_score = float(answer_plan_selected_scores.get(clean_event_id, 0.0) or 0.0)
+ current_score = float(answer_plan_current_scores.get(clean_event_id, 0.0) or 0.0)
+ historical_score = float(answer_plan_historical_scores.get(clean_event_id, 0.0) or 0.0)
+ suppressed_score = float(answer_plan_suppressed_scores.get(clean_event_id, 0.0) or 0.0)
+ support_score = max(selected_score, current_score)
+ return {
+ "answer_plan_score": round(float(support_score), 6),
+ "answer_plan_selected_score": round(float(selected_score), 6),
+ "answer_plan_current_score": round(float(current_score), 6),
+ "answer_plan_historical_score": round(float(historical_score), 6),
+ "answer_plan_suppressed_score": round(float(suppressed_score), 6),
+ "answer_plan_adjusted_score": round(float(answer_plan_adjusted_scores.get(clean_event_id, 0.0)), 6),
+ "answer_plan_selected": bool(clean_event_id in answer_plan_ranked_event_id_set),
+ "answer_plan_rank": int(answer_plan_rank_lookup.get(clean_event_id, 0) or 0),
+ }
+
+ embedder_fusion_mode = _normalize(self.embedder_fusion_mode)
+ embedder_fusion_enabled = (
+ embedder_fusion_mode not in _EMBEDDER_INDEX_DISABLED_MODES
+ and bool(embedder_index_event_ids)
+ and bool(self.embedder_fusion_top_k > 0)
+ and bool(self.embedder_fusion_weight > 0.0)
+ )
+ embedder_fusion_applied_event_scores: Dict[str, float] = {}
+ embedder_fusion_boosts: Dict[str, float] = {}
+ embedder_event_scores = dict(embedder_index_metadata.get("embedder_index_event_scores", {}) or {})
+ if embedder_fusion_enabled and embedder_event_scores:
+ ranked_embedder_events = [
+ (event_id, float(score))
+ for event_id, score in sorted(
+ embedder_event_scores.items(),
+ key=lambda item: (-float(item[1]), item[0]),
+ )
+ if _clean_text(event_id)
+ ][: max(1, int(self.embedder_fusion_top_k))]
+ for rank, (event_id, embedder_score) in enumerate(ranked_embedder_events, start=1):
+ if embedder_score < float(self.embedder_fusion_score_floor):
+ continue
+ rank_bonus = max(0.0, 0.08 - (rank - 1) * 0.006)
+ boost = min(
+ float(self.embedder_fusion_max_boost),
+ (float(self.embedder_fusion_weight) * embedder_score) + rank_bonus,
+ )
+ current_score = max(
+ float(recall_event_scores.get(event_id, 0.0) or 0.0),
+ float(event_scores.get(event_id, 0.0) or 0.0),
+ float(base_event_scores.get(event_id, 0.0) or 0.0),
+ float(rerank_event_scores.get(event_id, 0.0) or 0.0),
+ float(calibrated_event_scores.get(event_id, 0.0) or 0.0),
+ )
+ fused_score = current_score + boost
+ recall_event_scores[event_id] = max(float(recall_event_scores.get(event_id, 0.0) or 0.0), fused_score)
+ event_scores[event_id] = max(float(event_scores.get(event_id, 0.0) or 0.0), fused_score)
+ base_event_scores[event_id] = max(float(base_event_scores.get(event_id, 0.0) or 0.0), fused_score)
+ rerank_event_scores[event_id] = max(float(rerank_event_scores.get(event_id, 0.0) or 0.0), fused_score)
+ calibrated_event_scores[event_id] = max(float(calibrated_event_scores.get(event_id, 0.0) or 0.0), fused_score)
+ embedder_fusion_applied_event_scores[event_id] = round(fused_score, 6)
+ embedder_fusion_boosts[event_id] = round(boost, 6)
+ if embedder_fusion_applied_event_scores:
+ decision_score_source = f"{decision_score_source or 'learned_decision_fusion'}+embedder_fusion"
+ recall_event_ids = [
+ event_id
+ for event_id, _ in sorted(recall_event_scores.items(), key=lambda item: (-float(item[1]), item[0]))
+ ][: self.candidate_event_k]
+ if profile_first_event_ids:
+ profile_first_score_by_event = {
+ _clean_text((hit.metadata or {}).get("profile_first_hybrid_event_id", "")) or _runtime_event_key(hit): float(hit.score)
+ for hit in profile_first_hits
+ }
+ for rank, event_id in enumerate(profile_first_event_ids, start=1):
+ if not event_id:
+ continue
+ floor = max(0.92, float(profile_first_score_by_event.get(event_id, 0.0))) + max(0.0, 0.04 - (rank * 0.005))
+ recall_event_scores[event_id] = max(float(recall_event_scores.get(event_id, 0.0)), floor)
+ event_scores[event_id] = max(float(event_scores.get(event_id, 0.0)), floor)
+ base_event_scores[event_id] = max(float(base_event_scores.get(event_id, 0.0)), floor)
+ rerank_event_scores[event_id] = max(float(rerank_event_scores.get(event_id, 0.0)), floor)
+ calibrated_event_scores[event_id] = max(float(calibrated_event_scores.get(event_id, 0.0)), floor)
+ for path_id, path in runtime_paths.items():
+ event_id = _clean_text(path.get("event_id", ""))
+ path_type = _clean_text(path.get("type", ""))
+ if event_id not in set(profile_first_event_ids):
+ continue
+ if path_type not in {"speaker_event_profile", "speaker_event_source_turn", "speaker_event_status"}:
+ continue
+ floor = max(0.90, float(event_scores.get(event_id, 0.0)))
+ path_scores[path_id] = max(float(path_scores.get(path_id, 0.0)), floor)
+ base_path_scores[path_id] = max(float(base_path_scores.get(path_id, 0.0)), floor)
+ calibrated_path_scores[path_id] = max(float(calibrated_path_scores.get(path_id, 0.0)), floor)
+ path_model_scores[path_id] = max(float(path_model_scores.get(path_id, 0.0)), floor)
+ _hybrid_mark("score_postprocess_sec")
+ selection_consistency_repaired = False
+ selection_consistency_reason = ""
+ model_focused_answer_type = _clean_text(focused_answer_type_from_model)
+ focused_answer_type = _reconciled_focused_answer_type(
+ question_analysis,
+ answer_type_scores,
+ focused_answer_type_from_model,
+ )
+ embedder_fusion_selected_event_ids: List[str] = []
+ embedder_fusion_selected_path_ids: List[str] = []
+ if decision_fusion_enabled and (selected_path_ids_from_model or selected_event_ids_from_model):
+ base_selected_path_limit = max(1, min(max(1, self.support_path_k), max(1, top_k)))
+ selected_path_ids = [
+ path_id
+ for path_id in selected_path_ids_from_model
+ if _clean_text(path_id) in runtime_paths
+ ]
+ if not selected_path_ids:
+ selected_path_ids = [
+ path_id
+ for path_id, _ in sorted(
+ ((path_id, float(score or 0.0)) for path_id, score in path_scores.items()),
+ key=lambda item: (-float(item[1]), item[0]),
+ )
+ ][:base_selected_path_limit]
+ tunnel_rescue_path_ids: List[str] = []
+ tunnel_rescue_pre_filter_path_ids: List[str] = []
+ path_utility_direct_support_path_ids: List[str] = []
+ path_utility_contrast_support_path_ids: List[str] = []
+ path_utility_latent_context_path_ids: List[str] = []
+ path_utility_drift_noise_path_ids: List[str] = []
+ path_utility_roles: Dict[str, str] = {}
+ path_utility_reasons: Dict[str, str] = {}
+ path_utility_scores: Dict[str, float] = {}
+ path_utility_overlap_tokens: Dict[str, List[str]] = {}
+ path_utility_anchor_event_ids: List[str] = []
+ path_utility_anchor_subject_signatures: List[str] = []
+ tunnel_rescue_score_threshold = float(self.path_tunnel_rescue_score_floor)
+ tunnel_rescue_candidate_count = 0
+ tunnel_rescue_filtered_count = 0
+ if (
+ self.path_tunnel_rescue_k > 0
+ and path_tunnel_support_scores
+ and _memory_router_allows(memory_router_decision, "path_tunnel", "topic_tunnel")
+ ):
+ selected_path_id_set = set(selected_path_ids)
+ candidate_scores = [
+ float(score or 0.0)
+ for path_id, score in path_tunnel_support_scores.items()
+ if _clean_text(path_id) in runtime_paths
+ ]
+ tunnel_rescue_candidate_count = len(candidate_scores)
+ if candidate_scores and self.path_tunnel_rescue_min_score_margin > 0.0:
+ sorted_scores = sorted(candidate_scores)
+ median_score = sorted_scores[len(sorted_scores) // 2]
+ tunnel_rescue_score_threshold = max(
+ tunnel_rescue_score_threshold,
+ median_score + float(self.path_tunnel_rescue_min_score_margin),
+ )
+ event_turn_indices = [
+ _runtime_event_turn_index_from_id(_clean_text(path.get("event_id", "")))
+ for path in runtime_paths.values()
+ ]
+ current_turn_index = max(
+ [int(getattr(self.graph, "turn_index", 0) or 0), *[turn for turn in event_turn_indices if turn > 0]],
+ default=0,
+ )
+ ranked_tunnel_path_ids = []
+ for path_id, score in sorted(
+ (
+ (path_id, float(score or 0.0))
+ for path_id, score in path_tunnel_support_scores.items()
+ if _clean_text(path_id) in runtime_paths
+ ),
+ key=lambda item: (-float(item[1]), item[0]),
+ ):
+ if path_id in selected_path_id_set or score < tunnel_rescue_score_threshold:
+ continue
+ path_event_turn = _runtime_event_turn_index_from_id(
+ _clean_text(runtime_paths.get(path_id, {}).get("event_id", ""))
+ )
+ path_age = max(0, current_turn_index - path_event_turn) if current_turn_index and path_event_turn else 0
+ if self.path_tunnel_rescue_min_age > 0 and path_age < self.path_tunnel_rescue_min_age:
+ continue
+ ranked_tunnel_path_ids.append(path_id)
+ tunnel_rescue_filtered_count = len(ranked_tunnel_path_ids)
+ tunnel_rescue_pre_filter_path_ids = ranked_tunnel_path_ids[: max(self.path_tunnel_rescue_k, self.path_tunnel_rescue_k * 4)]
+ utility_gate = _path_utility_gate(
+ tunnel_rescue_pre_filter_path_ids,
+ query=query,
+ runtime_graph=runtime_graph,
+ runtime_paths=runtime_paths,
+ grouped_hits=grouped_hits,
+ selected_path_ids=selected_path_ids,
+ selected_event_ids_from_model=selected_event_ids_from_model,
+ path_scores=path_scores,
+ path_tunnel_support_scores=path_tunnel_support_scores,
+ question_analysis=question_analysis,
+ focused_answer_type=focused_answer_type,
+ score_threshold=tunnel_rescue_score_threshold,
+ limit=self.path_tunnel_rescue_k,
+ )
+ tunnel_rescue_path_ids = list(utility_gate.get("injected_path_ids", []) or [])
+ path_utility_direct_support_path_ids = list(utility_gate.get("direct_support_path_ids", []) or [])
+ path_utility_contrast_support_path_ids = list(utility_gate.get("contrast_support_path_ids", []) or [])
+ path_utility_latent_context_path_ids = list(utility_gate.get("latent_context_path_ids", []) or [])
+ path_utility_drift_noise_path_ids = list(utility_gate.get("drift_noise_path_ids", []) or [])
+ path_utility_roles = dict(utility_gate.get("roles", {}) or {})
+ path_utility_reasons = dict(utility_gate.get("reasons", {}) or {})
+ path_utility_scores = dict(utility_gate.get("scores", {}) or {})
+ path_utility_overlap_tokens = dict(utility_gate.get("overlap_tokens", {}) or {})
+ path_utility_anchor_event_ids = list(utility_gate.get("anchor_event_ids", []) or [])
+ path_utility_anchor_subject_signatures = list(utility_gate.get("anchor_subject_signatures", []) or [])
+ if tunnel_rescue_path_ids:
+ selected_path_ids = _dedupe([*selected_path_ids, *tunnel_rescue_path_ids])
+ effective_selected_path_limit = base_selected_path_limit + len(tunnel_rescue_path_ids)
+ selected_event_ids = _dedupe(
+ [
+ *[
+ _clean_text(runtime_paths.get(path_id, {}).get("event_id", ""))
+ for path_id in selected_path_ids
+ if _clean_text(runtime_paths.get(path_id, {}).get("event_id", ""))
+ ],
+ *[
+ _clean_text(event_id)
+ for event_id in selected_event_ids_from_model
+ if _clean_text(event_id)
+ ],
+ ]
+ )
+ if answer_plan_ranked_event_ids:
+ selected_event_ids = _dedupe([*selected_event_ids, *answer_plan_ranked_event_ids])
+ if not selected_event_ids:
+ selected_event_ids = [
+ event_id
+ for event_id, _ in sorted(
+ ((event_id, float(score or 0.0)) for event_id, score in event_scores.items()),
+ key=lambda item: (-float(item[1]), item[0]),
+ )
+ ][: max(1, min(self.candidate_event_k, max(top_k, _HYBRID_SELECTED_EVENT_FLOOR)))]
+ selected_event_ids = _add_multi_evidence_recall_coverage(
+ selected_event_ids,
+ recall_event_ids,
+ focused_answer_type=focused_answer_type,
+ top_k=top_k,
+ )
+ repaired_path_ids, selection_consistency_repaired, selection_consistency_reason = _repair_selected_paths_for_focus(
+ selected_path_ids,
+ runtime_paths=runtime_paths,
+ selected_event_ids=selected_event_ids,
+ path_scores=path_scores,
+ event_scores=event_scores,
+ temporal_scores=temporal_scores,
+ question_analysis=question_analysis,
+ answer_type_scores=answer_type_scores,
+ focused_answer_type=focused_answer_type,
+ limit=max(1, effective_selected_path_limit),
+ )
+ if selection_consistency_repaired:
+ selected_path_ids = repaired_path_ids
+ selected_event_ids = _dedupe(
+ [
+ *[
+ _clean_text(runtime_paths.get(path_id, {}).get("event_id", ""))
+ for path_id in selected_path_ids
+ if _clean_text(runtime_paths.get(path_id, {}).get("event_id", ""))
+ ],
+ *[
+ _clean_text(event_id)
+ for event_id in selected_event_ids_from_model
+ if _clean_text(event_id)
+ ],
+ ]
+ )
+ if answer_plan_ranked_event_ids:
+ selected_event_ids = _dedupe([*selected_event_ids, *answer_plan_ranked_event_ids])
+ selected_event_ids = _add_multi_evidence_recall_coverage(
+ selected_event_ids,
+ recall_event_ids,
+ focused_answer_type=focused_answer_type,
+ top_k=top_k,
+ )
+ if profile_first_event_ids:
+ selected_event_ids = _dedupe([*profile_first_event_ids, *selected_event_ids])
+ if embedder_fusion_applied_event_scores and self.embedder_fusion_select_k > 0:
+ ranked_fusion_event_ids = [
+ event_id
+ for event_id in embedder_index_event_ids
+ if event_id in embedder_fusion_applied_event_scores
+ ][: max(1, int(self.embedder_fusion_select_k))]
+ selected_event_ids = _dedupe([*ranked_fusion_event_ids, *selected_event_ids])
+ selected_path_id_set = set(selected_path_ids)
+ selected_embedder_path_ids: List[str] = []
+ for event_id in ranked_fusion_event_ids:
+ candidate_paths = [
+ (path_id, path)
+ for path_id, path in runtime_paths.items()
+ if _clean_text(path.get("event_id", "")) == event_id
+ ]
+ candidate_paths.sort(
+ key=lambda item: (
+ int(_clean_text(item[1].get("type", "")) in {"speaker_event_source_turn", "speaker_event_profile", "speaker_event_status", "speaker_event_time"}),
+ float(path_scores.get(item[0], 0.0) or 0.0),
+ item[0],
+ ),
+ reverse=True,
+ )
+ for path_id, _ in candidate_paths:
+ if path_id in selected_path_id_set:
+ continue
+ selected_embedder_path_ids.append(path_id)
+ selected_path_id_set.add(path_id)
+ embedder_fusion_selected_path_ids.append(path_id)
+ break
+ if selected_embedder_path_ids:
+ selected_path_ids = _dedupe([*selected_embedder_path_ids, *selected_path_ids])
+ embedder_fusion_selected_event_ids = list(ranked_fusion_event_ids)
+ if answer_plan_ranked_event_ids:
+ selected_event_ids = _dedupe([*selected_event_ids, *answer_plan_ranked_event_ids])
+ selected_event_ids = _add_multi_evidence_recall_coverage(
+ selected_event_ids,
+ recall_event_ids,
+ focused_answer_type=focused_answer_type,
+ top_k=top_k,
+ )
+ final_hits: List[MemoryHit] = []
+ seen_memory_ids = set()
+ selected_event_id_set = set(selected_event_ids)
+ for path_id in selected_path_ids:
+ path = runtime_paths.get(path_id, {})
+ event_id = _clean_text(path.get("event_id", ""))
+ if event_id not in selected_event_id_set:
+ continue
+ path_type = _clean_text(path.get("type", ""))
+ support_node_id = _path_support_node_id(path)
+ event_hit_candidates = [*grouped_hits.get(event_id, []), *_event_record_hits_from_graph(self.graph, event_id)]
+ support_hit = _support_hit_for_path(path_type, grouped_hits.get(event_id, []))
+ event_hit = _representative_event_hit(
+ event_hit_candidates,
+ query=query,
+ )
+ decision_score = round(float(path_scores.get(path_id, 0.0)), 6)
+ candidate_hits: List[tuple[MemoryHit | None, float]] = [(support_hit, decision_score)]
+ if event_hit is not None and (support_hit is None or event_hit.memory_id != support_hit.memory_id):
+ candidate_hits.append((event_hit, max(0.0, decision_score - 0.0001)))
+ for raw_hit, hit_score in candidate_hits:
+ if raw_hit is None or raw_hit.memory_id in seen_memory_ids:
+ continue
+ seen_memory_ids.add(raw_hit.memory_id)
+ recall_score = float(recall_event_scores.get(event_id, event_scores.get(event_id, 0.0)))
+ event_score = float(event_scores.get(event_id, raw_hit.score))
+ temporal_score = float(temporal_scores.get(support_node_id, 0.0))
+ metadata = dict(raw_hit.metadata or {})
+ metadata.update(
+ {
+ "event_id": event_id,
+ **_answer_plan_hit_metadata(event_id),
+ "path_id": path_id,
+ "base_event_score": round(float(base_event_scores.get(event_id, event_score)), 6),
+ "rerank_event_score": round(float(rerank_event_scores.get(event_id, event_score)), 6),
+ "calibrated_event_score": round(float(calibrated_event_scores.get(event_id, event_score)), 6),
+ "matrix_event_score": round(float(matrix_event_scores.get(event_id, 0.0)), 6),
+ "event_fusion_delta_score": round(float(event_fusion_delta_scores.get(event_id, 0.0)), 6),
+ "event_tunnel_support_score": round(float(event_tunnel_support_scores.get(event_id, 0.0)), 6),
+ "event_tunnel_delta_score": round(float(event_tunnel_delta_scores.get(event_id, 0.0)), 6),
+ "tri_maze_event_reverse_score": round(float(tri_maze_event_reverse_scores.get(event_id, 0.0)), 6),
+ "tri_maze_event_boundary_score": round(float(tri_maze_event_boundary_scores.get(event_id, 0.0)), 6),
+ "tri_maze_event_reverse_relation": round(float(tri_maze_event_reverse_relations.get(event_id, 0.0)), 6),
+ "matrix_enabled": matrix_enabled,
+ "event_calibration_enabled": event_calibration_enabled,
+ "path_calibration_enabled": path_calibration_enabled,
+ "event_tunnel_enabled": event_tunnel_enabled,
+ "path_tunnel_enabled": path_tunnel_enabled,
+ "final_event_fusion_enabled": final_event_fusion_enabled,
+ "final_path_fusion_enabled": final_path_fusion_enabled,
+ "decision_fusion_enabled": True,
+ "event_fusion_enabled": event_fusion_enabled,
+ "path_fusion_enabled": path_fusion_enabled,
+ "event_score": round(event_score, 6),
+ "recall_score": round(recall_score, 6),
+ "path_score": round(float(path_scores.get(path_id, hit_score)), 6),
+ "base_path_score": round(float(base_path_scores.get(path_id, path_scores.get(path_id, hit_score))), 6),
+ "calibrated_path_score": round(float(calibrated_path_scores.get(path_id, path_scores.get(path_id, hit_score))), 6),
+ "path_fusion_delta_score": round(float(path_fusion_delta_scores.get(path_id, 0.0)), 6),
+ "path_tunnel_support_score": round(float(path_tunnel_support_scores.get(path_id, 0.0)), 6),
+ "path_tunnel_delta_score": round(float(path_tunnel_delta_scores.get(path_id, 0.0)), 6),
+ "path_model_score": round(float(path_model_scores.get(path_id, path_scores.get(path_id, hit_score))), 6),
+ "path_chain_extension_enabled": path_chain_extension_enabled,
+ "path_chain_extension_delta_score": round(float(path_chain_extension_delta_scores.get(path_id, 0.0)), 6),
+ "path_chain_extended_score": round(float(path_chain_extended_scores.get(path_id, path_scores.get(path_id, hit_score))), 6),
+ "tri_maze_path_reverse_score": round(float(tri_maze_path_reverse_scores.get(path_id, 0.0)), 6),
+ "tri_maze_path_boundary_score": round(float(tri_maze_path_boundary_scores.get(path_id, 0.0)), 6),
+ "tri_maze_path_reverse_relation": round(float(tri_maze_path_reverse_relations.get(path_id, 0.0)), 6),
+ "effective_path_score": round(float(path_scores.get(path_id, hit_score)), 6),
+ "temporal_score": round(temporal_score, 6),
+ "raw_public_score": round(float(raw_hit.score), 6),
+ "hybrid_score": round(hit_score, 6),
+ "hybrid_score_source": decision_score_source or "learned_decision_fusion",
+ "evidence_snippet_role": "selected_path_support" if raw_hit is support_hit else "selected_path_event",
+ "selected_event_ids": list(selected_event_ids),
+ "selected_path_ids": list(selected_path_ids),
+ "path_tunnel_rescue_enabled": bool(self.path_tunnel_rescue_k > 0),
+ "path_tunnel_rescue_k": int(self.path_tunnel_rescue_k),
+ "path_tunnel_rescue_score_floor": round(float(self.path_tunnel_rescue_score_floor), 6),
+ "path_tunnel_rescue_min_age": int(self.path_tunnel_rescue_min_age),
+ "path_tunnel_rescue_min_score_margin": round(float(self.path_tunnel_rescue_min_score_margin), 6),
+ "path_tunnel_rescue_score_threshold": round(float(tunnel_rescue_score_threshold), 6),
+ "path_tunnel_rescue_candidate_count": int(tunnel_rescue_candidate_count),
+ "path_tunnel_rescue_filtered_count": int(tunnel_rescue_filtered_count),
+ "path_tunnel_rescue_path_ids": list(tunnel_rescue_path_ids),
+ "path_utility_gate_enabled": bool(self.path_tunnel_rescue_k > 0),
+ "path_utility_pre_filter_path_ids": list(tunnel_rescue_pre_filter_path_ids),
+ "path_utility_injected_path_ids": list(tunnel_rescue_path_ids),
+ "path_utility_direct_support_path_ids": list(path_utility_direct_support_path_ids),
+ "path_utility_contrast_support_path_ids": list(path_utility_contrast_support_path_ids),
+ "path_utility_latent_context_path_ids": list(path_utility_latent_context_path_ids),
+ "path_utility_drift_noise_path_ids": list(path_utility_drift_noise_path_ids),
+ "path_utility_roles": dict(path_utility_roles),
+ "path_utility_reasons": dict(path_utility_reasons),
+ "path_utility_scores": dict(path_utility_scores),
+ "path_utility_overlap_tokens": dict(path_utility_overlap_tokens),
+ "path_utility_anchor_event_ids": list(path_utility_anchor_event_ids),
+ "path_utility_anchor_subject_signatures": list(path_utility_anchor_subject_signatures),
+ "tunnel_recall_pre_filter_count": int(len(tunnel_rescue_pre_filter_path_ids)),
+ "tunnel_usable_post_filter_count": int(len(tunnel_rescue_path_ids)),
+ "model_focused_answer_type": model_focused_answer_type,
+ "selection_consistency_repaired": bool(selection_consistency_repaired),
+ "selection_consistency_reason": selection_consistency_reason,
+ }
+ )
+ final_hits.append(
+ MemoryHit(
+ memory_id=raw_hit.memory_id,
+ category=raw_hit.category,
+ value=raw_hit.value,
+ relation=raw_hit.relation,
+ anchors=list(raw_hit.anchors),
+ score=round(hit_score, 6),
+ source_kind=raw_hit.source_kind,
+ slot_key=raw_hit.slot_key,
+ state=raw_hit.state,
+ turn_index=int(raw_hit.turn_index),
+ metadata=metadata,
+ )
+ )
+ for event_rank, event_id in enumerate(selected_event_ids, start=1):
+ event_hit = _representative_event_hit(
+ [*grouped_hits.get(event_id, []), *_event_record_hits_from_graph(self.graph, event_id)],
+ query=query,
+ )
+ if event_hit is None or event_hit.memory_id in seen_memory_ids:
+ continue
+ seen_memory_ids.add(event_hit.memory_id)
+ recall_score = float(recall_event_scores.get(event_id, event_hit.score))
+ event_score = float(event_scores.get(event_id, event_hit.score))
+ metadata = dict(event_hit.metadata or {})
+ metadata.update(
+ {
+ "event_id": event_id,
+ **_answer_plan_hit_metadata(event_id),
+ "path_id": "",
+ "base_event_score": round(float(base_event_scores.get(event_id, event_score)), 6),
+ "rerank_event_score": round(float(rerank_event_scores.get(event_id, event_score)), 6),
+ "calibrated_event_score": round(float(calibrated_event_scores.get(event_id, event_score)), 6),
+ "matrix_event_score": round(float(matrix_event_scores.get(event_id, 0.0)), 6),
+ "event_fusion_delta_score": round(float(event_fusion_delta_scores.get(event_id, 0.0)), 6),
+ "event_tunnel_support_score": round(float(event_tunnel_support_scores.get(event_id, 0.0)), 6),
+ "event_tunnel_delta_score": round(float(event_tunnel_delta_scores.get(event_id, 0.0)), 6),
+ "tri_maze_event_reverse_score": round(float(tri_maze_event_reverse_scores.get(event_id, 0.0)), 6),
+ "tri_maze_event_boundary_score": round(float(tri_maze_event_boundary_scores.get(event_id, 0.0)), 6),
+ "tri_maze_event_reverse_relation": round(float(tri_maze_event_reverse_relations.get(event_id, 0.0)), 6),
+ "matrix_enabled": matrix_enabled,
+ "event_calibration_enabled": event_calibration_enabled,
+ "path_calibration_enabled": path_calibration_enabled,
+ "event_tunnel_enabled": event_tunnel_enabled,
+ "path_tunnel_enabled": path_tunnel_enabled,
+ "final_event_fusion_enabled": final_event_fusion_enabled,
+ "final_path_fusion_enabled": final_path_fusion_enabled,
+ "decision_fusion_enabled": True,
+ "event_fusion_enabled": event_fusion_enabled,
+ "path_fusion_enabled": path_fusion_enabled,
+ "event_score": round(event_score, 6),
+ "recall_score": round(recall_score, 6),
+ "path_score": 0.0,
+ "base_path_score": 0.0,
+ "calibrated_path_score": 0.0,
+ "path_fusion_delta_score": 0.0,
+ "path_tunnel_support_score": 0.0,
+ "path_tunnel_delta_score": 0.0,
+ "path_model_score": 0.0,
+ "path_chain_extension_enabled": path_chain_extension_enabled,
+ "path_chain_extension_delta_score": 0.0,
+ "path_chain_extended_score": 0.0,
+ "effective_path_score": 0.0,
+ "temporal_score": 0.0,
+ "raw_public_score": round(float(event_hit.score), 6),
+ "hybrid_score": round(event_score, 6),
+ "hybrid_score_source": decision_score_source or "learned_final_event_fusion",
+ "evidence_snippet_role": "selected_event_representative",
+ "selected_event_rank": int(event_rank),
+ "selected_event_ids": list(selected_event_ids),
+ "selected_path_ids": list(selected_path_ids),
+ "path_tunnel_rescue_enabled": bool(self.path_tunnel_rescue_k > 0),
+ "path_tunnel_rescue_k": int(self.path_tunnel_rescue_k),
+ "path_tunnel_rescue_score_floor": round(float(self.path_tunnel_rescue_score_floor), 6),
+ "path_tunnel_rescue_min_age": int(self.path_tunnel_rescue_min_age),
+ "path_tunnel_rescue_min_score_margin": round(float(self.path_tunnel_rescue_min_score_margin), 6),
+ "path_tunnel_rescue_score_threshold": round(float(tunnel_rescue_score_threshold), 6),
+ "path_tunnel_rescue_candidate_count": int(tunnel_rescue_candidate_count),
+ "path_tunnel_rescue_filtered_count": int(tunnel_rescue_filtered_count),
+ "path_tunnel_rescue_path_ids": list(tunnel_rescue_path_ids),
+ "path_utility_gate_enabled": bool(self.path_tunnel_rescue_k > 0),
+ "path_utility_pre_filter_path_ids": list(tunnel_rescue_pre_filter_path_ids),
+ "path_utility_injected_path_ids": list(tunnel_rescue_path_ids),
+ "path_utility_direct_support_path_ids": list(path_utility_direct_support_path_ids),
+ "path_utility_contrast_support_path_ids": list(path_utility_contrast_support_path_ids),
+ "path_utility_latent_context_path_ids": list(path_utility_latent_context_path_ids),
+ "path_utility_drift_noise_path_ids": list(path_utility_drift_noise_path_ids),
+ "path_utility_roles": dict(path_utility_roles),
+ "path_utility_reasons": dict(path_utility_reasons),
+ "path_utility_scores": dict(path_utility_scores),
+ "path_utility_overlap_tokens": dict(path_utility_overlap_tokens),
+ "path_utility_anchor_event_ids": list(path_utility_anchor_event_ids),
+ "path_utility_anchor_subject_signatures": list(path_utility_anchor_subject_signatures),
+ "tunnel_recall_pre_filter_count": int(len(tunnel_rescue_pre_filter_path_ids)),
+ "tunnel_usable_post_filter_count": int(len(tunnel_rescue_path_ids)),
+ "model_focused_answer_type": model_focused_answer_type,
+ "selection_consistency_repaired": bool(selection_consistency_repaired),
+ "selection_consistency_reason": selection_consistency_reason,
+ }
+ )
+ final_hits.append(
+ MemoryHit(
+ memory_id=event_hit.memory_id,
+ category=event_hit.category,
+ value=event_hit.value,
+ relation=event_hit.relation,
+ anchors=list(event_hit.anchors),
+ score=round(event_score, 6),
+ source_kind=event_hit.source_kind,
+ slot_key=event_hit.slot_key,
+ state=event_hit.state,
+ turn_index=int(event_hit.turn_index),
+ metadata=metadata,
+ )
+ )
+ if not final_hits:
+ final_hits = list(hits)
+ if profile_first_hits:
+ final_hits = _inject_profile_first_hits(
+ final_hits,
+ profile_first_hits,
+ selected_event_ids=selected_event_ids,
+ selected_path_ids=selected_path_ids,
+ )
+ final_hits = _coverage_preserving_final_hits(final_hits, selected_event_ids=selected_event_ids, top_k=top_k)
+ final_hit_event_ids = _event_ids_from_hits(final_hits)
+ final_missing_selected_event_ids = [event_id for event_id in selected_event_ids if event_id not in set(final_hit_event_ids)]
+ _hybrid_mark("selection_materialization_sec")
+ return {
+ "hits": final_hits,
+ "_grouped_hits": grouped_hits,
+ "metadata": {
+ "retrieval_mode": "hybrid_node_scored",
+ "hybrid_enabled": True,
+ "hybrid_source": hybrid_source,
+ "deepseek_graph_model": dict(deepseek_graph_teacher_payload),
+ **memory_router_decision,
+ "profile_first_router_suppressed": bool(profile_first_router_suppressed),
+ "recall_event_ids": list(recall_event_ids),
+ "learned_recall_event_ids": list(learned_recall_event_ids),
+ "model_recall_event_ids": list(model_recall_event_ids),
+ "symbolic_recall_event_ids": list(symbolic_recall_event_ids),
+ "embedder_index_recall_event_ids": list(embedder_index_event_ids),
+ **embedder_index_metadata,
+ "embedder_fusion_mode": embedder_fusion_mode or "off",
+ "embedder_fusion_enabled": bool(embedder_fusion_enabled),
+ "embedder_fusion_weight": round(float(self.embedder_fusion_weight), 6),
+ "embedder_fusion_score_floor": round(float(self.embedder_fusion_score_floor), 6),
+ "embedder_fusion_top_k": int(self.embedder_fusion_top_k),
+ "embedder_fusion_select_k": int(self.embedder_fusion_select_k),
+ "embedder_fusion_max_boost": round(float(self.embedder_fusion_max_boost), 6),
+ "embedder_fusion_event_scores": dict(embedder_fusion_applied_event_scores),
+ "embedder_fusion_boosts": dict(embedder_fusion_boosts),
+ "embedder_fusion_selected_event_ids": list(embedder_fusion_selected_event_ids),
+ "embedder_fusion_selected_path_ids": list(embedder_fusion_selected_path_ids),
+ "profile_first_hybrid_enabled": bool(profile_first_event_ids),
+ "profile_first_event_ids": list(profile_first_event_ids),
+ "profile_first_memory_ids": list(profile_first_memory_ids),
+ "hybrid_candidate_event_ids": list(hybrid_candidate_event_ids),
+ "hybrid_candidate_union_enabled": True,
+ "hybrid_candidate_union_rescored": bool(hybrid_candidate_union_rescored),
+ "runtime_graph_cache_hit": bool(runtime_graph_cache_hit),
+ "node_runtime_profile": dict(scored.get("runtime_profile", {}) or {}),
+ "hybrid_runtime_profile": _hybrid_profile_payload(),
+ "hybrid_candidate_union_added_event_ids": list(hybrid_candidate_union_added_event_ids),
+ "hybrid_candidate_union_priority_changed": bool(hybrid_candidate_union_priority_changed),
+ "rerank_candidate_event_ids": list(rerank_candidate_event_ids),
+ "base_event_scores": dict(base_event_scores),
+ "rerank_event_scores": dict(rerank_event_scores),
+ "calibrated_event_scores": dict(calibrated_event_scores),
+ "matrix_event_scores": dict(matrix_event_scores),
+ "event_fusion_delta_scores": dict(event_fusion_delta_scores),
+ "event_tunnel_support_scores": dict(event_tunnel_support_scores),
+ "event_tunnel_delta_scores": dict(event_tunnel_delta_scores),
+ "tri_maze_event_reverse_scores": dict(tri_maze_event_reverse_scores),
+ "tri_maze_event_boundary_scores": dict(tri_maze_event_boundary_scores),
+ "tri_maze_event_reverse_relations": dict(tri_maze_event_reverse_relations),
+ "matrix_rerank_event_ids": list(matrix_rerank_event_ids),
+ "matrix_enabled": matrix_enabled,
+ "rerank_path_scores": dict(rerank_path_scores),
+ "matrix_path_scores": dict(matrix_path_scores),
+ "tri_maze_path_reverse_scores": dict(tri_maze_path_reverse_scores),
+ "tri_maze_path_boundary_scores": dict(tri_maze_path_boundary_scores),
+ "tri_maze_path_reverse_relations": dict(tri_maze_path_reverse_relations),
+ "matrix_path_rerank_ids": list(matrix_path_rerank_ids),
+ "matrix_path_enabled": matrix_path_enabled,
+ "fusion_enabled": fusion_enabled,
+ "event_calibration_enabled": event_calibration_enabled,
+ "path_calibration_enabled": path_calibration_enabled,
+ "event_tunnel_enabled": event_tunnel_enabled,
+ "path_tunnel_enabled": path_tunnel_enabled,
+ "final_event_fusion_enabled": final_event_fusion_enabled,
+ "final_path_fusion_enabled": final_path_fusion_enabled,
+ "decision_fusion_enabled": True,
+ "decision_score_source": decision_score_source or "learned_decision_fusion",
+ "event_fusion_enabled": event_fusion_enabled,
+ "path_fusion_enabled": path_fusion_enabled,
+ "selected_event_ids": list(selected_event_ids),
+ "path_rescue_event_ids": [],
+ "selected_path_ids": list(selected_path_ids),
+ "path_tunnel_rescue_enabled": bool(self.path_tunnel_rescue_k > 0),
+ "path_tunnel_rescue_k": int(self.path_tunnel_rescue_k),
+ "path_tunnel_rescue_score_floor": round(float(self.path_tunnel_rescue_score_floor), 6),
+ "path_tunnel_rescue_min_age": int(self.path_tunnel_rescue_min_age),
+ "path_tunnel_rescue_min_score_margin": round(float(self.path_tunnel_rescue_min_score_margin), 6),
+ "path_tunnel_rescue_score_threshold": round(float(tunnel_rescue_score_threshold), 6),
+ "path_tunnel_rescue_candidate_count": int(tunnel_rescue_candidate_count),
+ "path_tunnel_rescue_filtered_count": int(tunnel_rescue_filtered_count),
+ "path_tunnel_rescue_path_ids": list(tunnel_rescue_path_ids),
+ "path_utility_gate_enabled": bool(self.path_tunnel_rescue_k > 0),
+ "path_utility_pre_filter_path_ids": list(tunnel_rescue_pre_filter_path_ids),
+ "path_utility_injected_path_ids": list(tunnel_rescue_path_ids),
+ "path_utility_direct_support_path_ids": list(path_utility_direct_support_path_ids),
+ "path_utility_contrast_support_path_ids": list(path_utility_contrast_support_path_ids),
+ "path_utility_latent_context_path_ids": list(path_utility_latent_context_path_ids),
+ "path_utility_drift_noise_path_ids": list(path_utility_drift_noise_path_ids),
+ "path_utility_roles": dict(path_utility_roles),
+ "path_utility_reasons": dict(path_utility_reasons),
+ "path_utility_scores": dict(path_utility_scores),
+ "path_utility_overlap_tokens": dict(path_utility_overlap_tokens),
+ "path_utility_anchor_event_ids": list(path_utility_anchor_event_ids),
+ "path_utility_anchor_subject_signatures": list(path_utility_anchor_subject_signatures),
+ "tunnel_recall_pre_filter_count": int(len(tunnel_rescue_pre_filter_path_ids)),
+ "tunnel_usable_post_filter_count": int(len(tunnel_rescue_path_ids)),
+ "final_hit_event_ids": list(final_hit_event_ids),
+ "final_hit_dia_ids": _dia_ids_from_hits(final_hits),
+ "final_missing_selected_event_ids": list(final_missing_selected_event_ids),
+ "selected_event_count": int(len(selected_event_ids)),
+ "selected_path_count": int(len(selected_path_ids)),
+ "temporal_scores": dict(temporal_scores),
+ "recall_event_scores": dict(recall_event_scores),
+ "event_scores": dict(event_scores),
+ "base_path_scores": dict(base_path_scores),
+ "calibrated_path_scores": dict(calibrated_path_scores),
+ "path_fusion_delta_scores": dict(path_fusion_delta_scores),
+ "path_tunnel_support_scores": dict(path_tunnel_support_scores),
+ "path_tunnel_delta_scores": dict(path_tunnel_delta_scores),
+ "path_model_scores": dict(path_model_scores),
+ "path_chain_extension_enabled": path_chain_extension_enabled,
+ "path_chain_extension_delta_scores": dict(path_chain_extension_delta_scores),
+ "path_chain_extended_scores": dict(path_chain_extended_scores),
+ "effective_path_scores": dict(path_scores),
+ "path_scores": dict(path_scores),
+ "answer_type_scores": dict(answer_type_scores),
+ "answer_plan_scores": dict(answer_plan_scores),
+ "answer_plan_support_scores": dict(answer_plan_support_scores),
+ "answer_plan_adjusted_scores": dict(answer_plan_adjusted_scores),
+ "answer_plan_raw_ranked_event_ids": list(answer_plan_raw_ranked_event_ids),
+ "answer_plan_ranked_event_ids": list(answer_plan_ranked_event_ids),
+ "answer_plan_selected_event_ids": list(answer_plan_selected_event_ids),
+ "answer_plan_current_event_ids": list(answer_plan_current_event_ids),
+ "answer_plan_promotion_enabled": bool(answer_plan_promotion_enabled),
+ "answer_plan_promotion_score_margin": round(float(answer_plan_promotion_score_margin), 6),
+ "answer_plan_promotion_min_margin": round(float(answer_plan_promotion_min_margin), 6),
+ "answer_plan_event_selection_threshold": round(float(answer_plan_event_selection_threshold), 6),
+ "answer_plan_event_selection_top_k": int(answer_plan_event_selection_top_k),
+ "focused_answer_type": focused_answer_type,
+ "model_focused_answer_type": model_focused_answer_type,
+ "selection_consistency_repaired": bool(selection_consistency_repaired),
+ "selection_consistency_reason": selection_consistency_reason,
+ "preferred_path_types": [],
+ },
+ }
+ dominant_answer_type = _dominant_answer_type(question_analysis, answer_type_scores)
+ preferred_path_types = _answer_type_preferred_path_types(question_analysis, answer_type_scores)
+ if bool(memory_router_decision.get("memory_router_guided")):
+ temporal_focus = _memory_router_allows(memory_router_decision, "temporal")
+ else:
+ temporal_focus = dominant_answer_type == "time" or bool(question_analysis.get("is_temporal", False))
+ router_profile_focus = bool(memory_router_decision.get("memory_router_guided")) and _memory_router_allows(
+ memory_router_decision,
+ "profile",
+ "resource",
+ )
+ learned_event_available = bool(event_scores)
+ learned_path_available = bool(path_scores)
+ ranked_events = [
+ event_id
+ for event_id, _ in sorted(
+ (
+ (event_id, event_scores.get(event_id, recall_event_scores.get(event_id, 0.0)))
+ for event_id in recall_event_ids
+ ),
+ key=lambda item: (-float(item[1]), item[0]),
+ )
+ ]
+ base_selected_event_count = min(
+ len(ranked_events),
+ max(
+ self.support_path_k * 2,
+ min(self.candidate_event_k, max(top_k, _HYBRID_SELECTED_EVENT_FLOOR)),
+ ),
+ )
+ effective_path_scores = dict(path_scores)
+ if not learned_path_available:
+ effective_path_scores = {
+ path_id: _calibrated_path_score(
+ path=runtime_paths.get(path_id, {}),
+ base_score=float(score),
+ temporal_scores=temporal_scores,
+ question_analysis=question_analysis,
+ answer_type_scores=answer_type_scores,
+ )
+ for path_id, score in path_scores.items()
+ }
+ ranked_path_ids_all = [
+ path_id
+ for path_id, _ in sorted(
+ ((path_id, float(score)) for path_id, score in effective_path_scores.items()),
+ key=lambda item: (-float(item[1]), item[0]),
+ )
+ ]
+ path_rescue_count = min(
+ len(ranked_path_ids_all),
+ max(self.support_path_k * 2, min(self.candidate_event_k, max(1, top_k))),
+ )
+ path_rescue_event_ids = _dedupe(
+ _clean_text(runtime_paths.get(path_id, {}).get("event_id", ""))
+ for path_id in ranked_path_ids_all[: max(1, path_rescue_count)]
+ )
+ selected_event_ids = _dedupe(
+ [
+ *path_rescue_event_ids,
+ *ranked_events[: max(1, base_selected_event_count)],
+ ]
+ )
+ if profile_first_event_ids:
+ selected_event_ids = _dedupe([*profile_first_event_ids, *selected_event_ids])
+ if answer_plan_ranked_event_ids:
+ selected_event_ids = _dedupe([*selected_event_ids, *answer_plan_ranked_event_ids])
+ selected_event_ids = _add_multi_evidence_recall_coverage(
+ selected_event_ids,
+ recall_event_ids,
+ focused_answer_type=focused_answer_type,
+ top_k=top_k,
+ )
+ selected_event_id_set = set(selected_event_ids)
+ ranked_path_ids = [
+ path_id
+ for path_id in ranked_path_ids_all
+ if _clean_text(runtime_paths.get(path_id, {}).get("event_id", "")) in selected_event_id_set
+ ]
+ if temporal_focus:
+ focused_time_path_ids = [
+ path_id
+ for path_id in ranked_path_ids
+ if _clean_text(runtime_paths.get(path_id, {}).get("type", "")) == "speaker_event_time"
+ ]
+ if focused_time_path_ids:
+ ranked_path_ids = focused_time_path_ids
+ path_limit_cap = _HYBRID_SELECTED_PATH_CAP
+ if temporal_focus:
+ path_limit_cap = _HYBRID_TEMPORAL_PATH_CAP
+ elif dominant_answer_type == "profile" or router_profile_focus:
+ path_limit_cap = _HYBRID_PROFILE_PATH_CAP
+ selected_path_count = min(
+ len(ranked_path_ids),
+ max(1, min(max(1, self.support_path_k), min(max(1, top_k), path_limit_cap))),
+ )
+ selected_path_ids = ranked_path_ids[: max(1, selected_path_count)]
+ final_hits: List[MemoryHit] = []
+ seen_memory_ids = set()
+ temporal_event_hits_added = 0
+ for path_id in selected_path_ids:
+ path = runtime_paths.get(path_id, {})
+ event_id = _clean_text(path.get("event_id", ""))
+ support_node_id = _path_support_node_id(path)
+ path_type = _clean_text(path.get("type", ""))
+ event_hit_candidates = [*grouped_hits.get(event_id, []), *_event_record_hits_from_graph(self.graph, event_id)]
+ support_hit = _support_hit_for_path(path_type, grouped_hits.get(event_id, []))
+ event_hit = _representative_event_hit(
+ event_hit_candidates,
+ query=query,
+ )
+ hit_pairs: List[tuple[MemoryHit | None, float]] = [(support_hit, path_scores.get(path_id, 0.0))]
+ if not temporal_focus:
+ hit_pairs.append((event_hit, path_scores.get(path_id, 0.0)))
+ elif support_hit is None and event_hit is not None:
+ hit_pairs.append((event_hit, path_scores.get(path_id, 0.0)))
+ elif path_type == "speaker_event_time" and event_hit is not None and temporal_event_hits_added < 1:
+ hit_pairs.append((event_hit, path_scores.get(path_id, 0.0)))
+ temporal_event_hits_added += 1
+ for raw_hit, path_score in hit_pairs:
+ if raw_hit is None or raw_hit.memory_id in seen_memory_ids:
+ continue
+ seen_memory_ids.add(raw_hit.memory_id)
+ recall_score = float(recall_event_scores.get(event_id, event_scores.get(event_id, 0.0)))
+ event_score = float(event_scores.get(event_id, raw_hit.score))
+ temporal_score = float(temporal_scores.get(support_node_id, 0.0))
+ effective_path_score = float(effective_path_scores.get(path_id, path_score))
+ hybrid_score = round(effective_path_score, 6) if learned_path_available else round(
+ (0.55 * event_score)
+ + (0.20 * float(path_score))
+ + (0.15 * recall_score)
+ + (0.10 * temporal_score),
+ 6,
+ )
+ metadata = dict(raw_hit.metadata or {})
+ metadata.update(
+ {
+ "event_id": event_id,
+ **_answer_plan_hit_metadata(event_id),
+ "path_id": path_id,
+ "base_event_score": round(float(base_event_scores.get(event_id, event_score)), 6),
+ "rerank_event_score": round(float(rerank_event_scores.get(event_id, event_score)), 6),
+ "calibrated_event_score": round(float(calibrated_event_scores.get(event_id, event_score)), 6),
+ "matrix_event_score": round(float(matrix_event_scores.get(event_id, 0.0)), 6),
+ "event_fusion_delta_score": round(float(event_fusion_delta_scores.get(event_id, 0.0)), 6),
+ "event_tunnel_support_score": round(float(event_tunnel_support_scores.get(event_id, 0.0)), 6),
+ "event_tunnel_delta_score": round(float(event_tunnel_delta_scores.get(event_id, 0.0)), 6),
+ "matrix_enabled": matrix_enabled,
+ "event_calibration_enabled": event_calibration_enabled,
+ "path_calibration_enabled": path_calibration_enabled,
+ "event_tunnel_enabled": event_tunnel_enabled,
+ "path_tunnel_enabled": path_tunnel_enabled,
+ "final_event_fusion_enabled": final_event_fusion_enabled,
+ "final_path_fusion_enabled": final_path_fusion_enabled,
+ "decision_fusion_enabled": False,
+ "event_fusion_enabled": event_fusion_enabled,
+ "path_fusion_enabled": path_fusion_enabled,
+ "event_score": round(event_score, 6),
+ "recall_score": round(recall_score, 6),
+ "path_score": round(float(path_score), 6),
+ "base_path_score": round(float(base_path_scores.get(path_id, path_score)), 6),
+ "calibrated_path_score": round(float(calibrated_path_scores.get(path_id, effective_path_score)), 6),
+ "path_fusion_delta_score": round(float(path_fusion_delta_scores.get(path_id, 0.0)), 6),
+ "path_tunnel_support_score": round(float(path_tunnel_support_scores.get(path_id, 0.0)), 6),
+ "path_tunnel_delta_score": round(float(path_tunnel_delta_scores.get(path_id, 0.0)), 6),
+ "path_model_score": round(float(path_model_scores.get(path_id, path_score)), 6),
+ "path_chain_extension_enabled": path_chain_extension_enabled,
+ "path_chain_extension_delta_score": round(float(path_chain_extension_delta_scores.get(path_id, 0.0)), 6),
+ "path_chain_extended_score": round(float(path_chain_extended_scores.get(path_id, effective_path_score)), 6),
+ "effective_path_score": round(effective_path_score, 6),
+ "temporal_score": round(temporal_score, 6),
+ "raw_public_score": round(float(raw_hit.score), 6),
+ "hybrid_score": hybrid_score,
+ "hybrid_score_source": (
+ "learned_path_fusion"
+ if path_fusion_enabled
+ else "learned_path_score"
+ if learned_path_available
+ else "heuristic_mix"
+ ),
+ "evidence_snippet_role": "selected_path_support" if raw_hit is support_hit else "selected_path_event",
+ "selected_event_ids": list(selected_event_ids),
+ "selected_path_ids": list(selected_path_ids),
+ }
+ )
+ final_hits.append(
+ MemoryHit(
+ memory_id=raw_hit.memory_id,
+ category=raw_hit.category,
+ value=raw_hit.value,
+ relation=raw_hit.relation,
+ anchors=list(raw_hit.anchors),
+ score=hybrid_score,
+ source_kind=raw_hit.source_kind,
+ slot_key=raw_hit.slot_key,
+ state=raw_hit.state,
+ turn_index=int(raw_hit.turn_index),
+ metadata=metadata,
+ )
+ )
+ for event_rank, event_id in enumerate(selected_event_ids, start=1):
+ event_hit = _representative_event_hit(
+ [*grouped_hits.get(event_id, []), *_event_record_hits_from_graph(self.graph, event_id)],
+ query=query,
+ )
+ if event_hit is None or event_hit.memory_id in seen_memory_ids:
+ continue
+ seen_memory_ids.add(event_hit.memory_id)
+ recall_score = float(recall_event_scores.get(event_id, event_hit.score))
+ event_score = float(event_scores.get(event_id, event_hit.score))
+ hybrid_score = round(event_score, 6) if learned_event_available else round(
+ (0.7 * event_score) + (0.3 * recall_score),
+ 6,
+ )
+ metadata = dict(event_hit.metadata or {})
+ metadata.update(
+ {
+ "event_id": event_id,
+ **_answer_plan_hit_metadata(event_id),
+ "path_id": "",
+ "base_event_score": round(float(base_event_scores.get(event_id, event_score)), 6),
+ "rerank_event_score": round(float(rerank_event_scores.get(event_id, event_score)), 6),
+ "calibrated_event_score": round(float(calibrated_event_scores.get(event_id, event_score)), 6),
+ "matrix_event_score": round(float(matrix_event_scores.get(event_id, 0.0)), 6),
+ "event_fusion_delta_score": round(float(event_fusion_delta_scores.get(event_id, 0.0)), 6),
+ "event_tunnel_support_score": round(float(event_tunnel_support_scores.get(event_id, 0.0)), 6),
+ "event_tunnel_delta_score": round(float(event_tunnel_delta_scores.get(event_id, 0.0)), 6),
+ "matrix_enabled": matrix_enabled,
+ "event_calibration_enabled": event_calibration_enabled,
+ "path_calibration_enabled": path_calibration_enabled,
+ "event_tunnel_enabled": event_tunnel_enabled,
+ "path_tunnel_enabled": path_tunnel_enabled,
+ "final_event_fusion_enabled": final_event_fusion_enabled,
+ "final_path_fusion_enabled": final_path_fusion_enabled,
+ "decision_fusion_enabled": False,
+ "event_fusion_enabled": event_fusion_enabled,
+ "path_fusion_enabled": path_fusion_enabled,
+ "event_score": round(event_score, 6),
+ "recall_score": round(recall_score, 6),
+ "path_score": 0.0,
+ "base_path_score": 0.0,
+ "calibrated_path_score": 0.0,
+ "path_fusion_delta_score": 0.0,
+ "path_tunnel_support_score": 0.0,
+ "path_tunnel_delta_score": 0.0,
+ "path_model_score": 0.0,
+ "path_chain_extension_enabled": path_chain_extension_enabled,
+ "path_chain_extension_delta_score": 0.0,
+ "path_chain_extended_score": 0.0,
+ "effective_path_score": 0.0,
+ "temporal_score": 0.0,
+ "raw_public_score": round(float(event_hit.score), 6),
+ "hybrid_score": hybrid_score,
+ "hybrid_score_source": (
+ "learned_event_fusion"
+ if event_fusion_enabled
+ else "learned_event_score"
+ if learned_event_available
+ else "heuristic_event_mix"
+ ),
+ "evidence_snippet_role": "selected_event_representative",
+ "selected_event_rank": int(event_rank),
+ "selected_event_ids": list(selected_event_ids),
+ "selected_path_ids": list(selected_path_ids),
+ }
+ )
+ final_hits.append(
+ MemoryHit(
+ memory_id=event_hit.memory_id,
+ category=event_hit.category,
+ value=event_hit.value,
+ relation=event_hit.relation,
+ anchors=list(event_hit.anchors),
+ score=hybrid_score,
+ source_kind=event_hit.source_kind,
+ slot_key=event_hit.slot_key,
+ state=event_hit.state,
+ turn_index=int(event_hit.turn_index),
+ metadata=metadata,
+ )
+ )
+ if not final_hits:
+ final_hits = list(hits)
+ if profile_first_hits:
+ final_hits = _inject_profile_first_hits(
+ final_hits,
+ profile_first_hits,
+ selected_event_ids=selected_event_ids,
+ selected_path_ids=selected_path_ids,
+ )
+ final_hits = _coverage_preserving_final_hits(final_hits, selected_event_ids=selected_event_ids, top_k=top_k)
+ final_hit_event_ids = _event_ids_from_hits(final_hits)
+ final_missing_selected_event_ids = [event_id for event_id in selected_event_ids if event_id not in set(final_hit_event_ids)]
+ _hybrid_mark("selection_materialization_sec")
+ return {
+ "hits": final_hits,
+ "_grouped_hits": grouped_hits,
+ "metadata": {
+ "retrieval_mode": "hybrid_node_scored",
+ "hybrid_enabled": True,
+ "hybrid_source": hybrid_source,
+ "deepseek_graph_model": dict(deepseek_graph_teacher_payload),
+ **memory_router_decision,
+ "profile_first_router_suppressed": bool(profile_first_router_suppressed),
+ "recall_event_ids": list(recall_event_ids),
+ "learned_recall_event_ids": list(learned_recall_event_ids),
+ "model_recall_event_ids": list(model_recall_event_ids),
+ "symbolic_recall_event_ids": list(symbolic_recall_event_ids),
+ "embedder_index_recall_event_ids": list(embedder_index_event_ids),
+ **embedder_index_metadata,
+ "embedder_fusion_mode": embedder_fusion_mode or "off",
+ "embedder_fusion_enabled": bool(embedder_fusion_enabled),
+ "embedder_fusion_weight": round(float(self.embedder_fusion_weight), 6),
+ "embedder_fusion_score_floor": round(float(self.embedder_fusion_score_floor), 6),
+ "embedder_fusion_top_k": int(self.embedder_fusion_top_k),
+ "embedder_fusion_select_k": int(self.embedder_fusion_select_k),
+ "embedder_fusion_max_boost": round(float(self.embedder_fusion_max_boost), 6),
+ "embedder_fusion_event_scores": dict(embedder_fusion_applied_event_scores),
+ "embedder_fusion_boosts": dict(embedder_fusion_boosts),
+ "embedder_fusion_selected_event_ids": list(embedder_fusion_selected_event_ids),
+ "embedder_fusion_selected_path_ids": list(embedder_fusion_selected_path_ids),
+ "profile_first_hybrid_enabled": bool(profile_first_event_ids),
+ "profile_first_event_ids": list(profile_first_event_ids),
+ "profile_first_memory_ids": list(profile_first_memory_ids),
+ "hybrid_candidate_event_ids": list(hybrid_candidate_event_ids),
+ "hybrid_candidate_union_enabled": True,
+ "hybrid_candidate_union_rescored": bool(hybrid_candidate_union_rescored),
+ "runtime_graph_cache_hit": bool(runtime_graph_cache_hit),
+ "node_runtime_profile": dict(scored.get("runtime_profile", {}) or {}),
+ "hybrid_runtime_profile": _hybrid_profile_payload(),
+ "hybrid_candidate_union_added_event_ids": list(hybrid_candidate_union_added_event_ids),
+ "hybrid_candidate_union_priority_changed": bool(hybrid_candidate_union_priority_changed),
+ "rerank_candidate_event_ids": list(rerank_candidate_event_ids),
+ "base_event_scores": dict(base_event_scores),
+ "rerank_event_scores": dict(rerank_event_scores),
+ "calibrated_event_scores": dict(calibrated_event_scores),
+ "matrix_event_scores": dict(matrix_event_scores),
+ "event_fusion_delta_scores": dict(event_fusion_delta_scores),
+ "event_tunnel_support_scores": dict(event_tunnel_support_scores),
+ "event_tunnel_delta_scores": dict(event_tunnel_delta_scores),
+ "tri_maze_event_reverse_scores": dict(tri_maze_event_reverse_scores),
+ "tri_maze_event_boundary_scores": dict(tri_maze_event_boundary_scores),
+ "tri_maze_event_reverse_relations": dict(tri_maze_event_reverse_relations),
+ "matrix_rerank_event_ids": list(matrix_rerank_event_ids),
+ "matrix_enabled": matrix_enabled,
+ "rerank_path_scores": dict(rerank_path_scores),
+ "matrix_path_scores": dict(matrix_path_scores),
+ "tri_maze_path_reverse_scores": dict(tri_maze_path_reverse_scores),
+ "tri_maze_path_boundary_scores": dict(tri_maze_path_boundary_scores),
+ "tri_maze_path_reverse_relations": dict(tri_maze_path_reverse_relations),
+ "matrix_path_rerank_ids": list(matrix_path_rerank_ids),
+ "matrix_path_enabled": matrix_path_enabled,
+ "fusion_enabled": fusion_enabled,
+ "event_calibration_enabled": event_calibration_enabled,
+ "path_calibration_enabled": path_calibration_enabled,
+ "event_tunnel_enabled": event_tunnel_enabled,
+ "path_tunnel_enabled": path_tunnel_enabled,
+ "final_event_fusion_enabled": final_event_fusion_enabled,
+ "final_path_fusion_enabled": final_path_fusion_enabled,
+ "decision_fusion_enabled": False,
+ "decision_score_source": "",
+ "event_fusion_enabled": event_fusion_enabled,
+ "path_fusion_enabled": path_fusion_enabled,
+ "selected_event_ids": list(selected_event_ids),
+ "path_rescue_event_ids": list(path_rescue_event_ids),
+ "selected_path_ids": list(selected_path_ids),
+ "final_hit_event_ids": list(final_hit_event_ids),
+ "final_hit_dia_ids": _dia_ids_from_hits(final_hits),
+ "final_missing_selected_event_ids": list(final_missing_selected_event_ids),
+ "selected_event_count": int(len(selected_event_ids)),
+ "selected_path_count": int(len(selected_path_ids)),
+ "temporal_scores": dict(temporal_scores),
+ "recall_event_scores": dict(recall_event_scores),
+ "event_scores": dict(event_scores),
+ "base_path_scores": dict(base_path_scores),
+ "calibrated_path_scores": dict(calibrated_path_scores),
+ "path_fusion_delta_scores": dict(path_fusion_delta_scores),
+ "path_tunnel_support_scores": dict(path_tunnel_support_scores),
+ "path_tunnel_delta_scores": dict(path_tunnel_delta_scores),
+ "path_model_scores": dict(path_model_scores),
+ "path_chain_extension_enabled": path_chain_extension_enabled,
+ "path_chain_extension_delta_scores": dict(path_chain_extension_delta_scores),
+ "path_chain_extended_scores": dict(path_chain_extended_scores),
+ "effective_path_scores": dict(effective_path_scores),
+ "path_scores": dict(path_scores),
+ "answer_type_scores": dict(answer_type_scores),
+ "answer_plan_scores": dict(answer_plan_scores),
+ "answer_plan_support_scores": dict(answer_plan_support_scores),
+ "answer_plan_adjusted_scores": dict(answer_plan_adjusted_scores),
+ "answer_plan_raw_ranked_event_ids": list(answer_plan_raw_ranked_event_ids),
+ "answer_plan_ranked_event_ids": list(answer_plan_ranked_event_ids),
+ "answer_plan_selected_event_ids": list(answer_plan_selected_event_ids),
+ "answer_plan_current_event_ids": list(answer_plan_current_event_ids),
+ "answer_plan_promotion_enabled": bool(answer_plan_promotion_enabled),
+ "answer_plan_promotion_score_margin": round(float(answer_plan_promotion_score_margin), 6),
+ "answer_plan_promotion_min_margin": round(float(answer_plan_promotion_min_margin), 6),
+ "answer_plan_event_selection_threshold": round(float(answer_plan_event_selection_threshold), 6),
+ "answer_plan_event_selection_top_k": int(answer_plan_event_selection_top_k),
+ "focused_answer_type": dominant_answer_type,
+ "preferred_path_types": list(preferred_path_types),
+ },
+ }
+
+ @_serialized_adapter_call
+ def retrieve(self, query: str, *, top_k: int = 6) -> MemoryRetrieval:
+ call_started = time.perf_counter()
+ self._reload_graph()
+ reload_finished = time.perf_counter()
+ start = time.perf_counter()
+ candidate_top_k = max(top_k, self.candidate_event_k if self.retrieval_mode == "hybrid_node_scored" else min(max(top_k, 18), 48))
+ payload = self.graph.retrieve(query, top_k=candidate_top_k)
+ heuristic_finished = time.perf_counter()
+ hits = [_raw_hit_to_memory_hit(item) for item in payload.get("hits", []) or []]
+ scored_lookup = {hit.memory_id: hit for hit in hits if hit.memory_id}
+ active_hits = _restore_hit_scores([_raw_hit_to_memory_hit(item) for item in payload.get("active_hits", []) or []], scored_lookup)
+ history_hits = _restore_hit_scores([_raw_hit_to_memory_hit(item) for item in payload.get("history_hits", []) or []], scored_lookup)
+ stale_hits = _restore_hit_scores([_raw_hit_to_memory_hit(item) for item in payload.get("stale_hits", []) or []], scored_lookup)
+ overwrite_hits = _restore_hit_scores([_raw_hit_to_memory_hit(item) for item in payload.get("overwrite_hits", []) or []], scored_lookup)
+ false_hits = _restore_hit_scores([_raw_hit_to_memory_hit(item) for item in payload.get("false_hits", []) or []], scored_lookup)
+ public_hits = _public_graph_hits(self.graph) if self.retrieval_mode == "hybrid_node_scored" else []
+ hit_prepare_finished = time.perf_counter()
+ hybrid_payload = self._hybrid_node_scored_hits(query, hits, top_k=top_k, public_hits=public_hits)
+ hybrid_finished = time.perf_counter()
+ adapter_stage_profile_enabled = _normalize(os.getenv("TMCRA_ADAPTER_STAGE_PROFILE", "")) in {
+ "1",
+ "true",
+ "yes",
+ "on",
+ }
+ adapter_stage_profile: Dict[str, float] = {}
+ adapter_stage_last = hybrid_finished
+
+ def _adapter_stage_mark(name: str) -> None:
+ nonlocal adapter_stage_last
+ if not adapter_stage_profile_enabled:
+ return
+ now = time.perf_counter()
+ adapter_stage_profile[name] = round(now - adapter_stage_last, 6)
+ adapter_stage_last = now
+
+ hits = list(hybrid_payload.get("hits", []) or hits)
+ hybrid_grouped_hits = dict(hybrid_payload.get("_grouped_hits", {}) or {})
+ hybrid_metadata = dict(hybrid_payload.get("metadata", {}) or {})
+ memory_router_decision = _memory_router_decision(
+ hybrid_metadata,
+ mode=self.memory_router_mode,
+ threshold=self.memory_router_threshold,
+ margin=self.memory_router_margin,
+ )
+ _adapter_stage_mark("router_decision_sec")
+ current_subject_payload = _current_subject_protected_hits(
+ query=query,
+ graph=self.graph,
+ final_hits=hits,
+ top_k=top_k,
+ )
+ hits = list(current_subject_payload.get("hits", hits) or hits)
+ current_subject_metadata = dict(current_subject_payload.get("metadata", {}) or {})
+ _adapter_stage_mark("current_subject_sec")
+ audit_anchor_payload = _audit_anchor_protected_hits(
+ query=query,
+ final_hits=hits,
+ candidate_hits=list(public_hits) + list(active_hits) + list(history_hits),
+ metadata={**dict(payload.get("metadata", {}) or {}), **hybrid_metadata, **current_subject_metadata},
+ top_k=top_k,
+ )
+ hits = list(audit_anchor_payload.get("hits", hits) or hits)
+ audit_anchor_metadata = dict(audit_anchor_payload.get("metadata", {}) or {})
+ _adapter_stage_mark("audit_anchor_sec")
+ identifier_payload = _identifier_protected_hits(
+ query=query,
+ final_hits=hits,
+ candidate_hits=list(public_hits) + list(active_hits) + list(history_hits),
+ top_k=top_k,
+ )
+ hits = list(identifier_payload.get("hits", hits) or hits)
+ identifier_metadata = dict(identifier_payload.get("metadata", {}) or {})
+ _adapter_stage_mark("identifier_sec")
+ if _memory_router_allows(memory_router_decision, "path_tunnel", "topic_tunnel"):
+ depth_chain_payload = _depth_chain_protected_hits(
+ query=query,
+ graph=self.graph,
+ final_hits=hits,
+ top_k=top_k,
+ )
+ hits = list(depth_chain_payload.get("hits", hits) or hits)
+ depth_chain_metadata = dict(depth_chain_payload.get("metadata", {}) or {})
+ else:
+ depth_chain_metadata = {
+ "depth_chain_protected_enabled": False,
+ "depth_chain_router_suppressed": True,
+ }
+ _adapter_stage_mark("depth_chain_sec")
+ if _memory_router_allows(memory_router_decision, "profile", "resource"):
+ profile_focused_payload = _profile_focused_pack_hits(
+ self.graph,
+ query,
+ hits,
+ top_k=top_k,
+ grouped_hits=hybrid_grouped_hits,
+ )
+ hits = list(profile_focused_payload.get("hits", hits) or hits)
+ profile_focused_metadata = dict(profile_focused_payload.get("metadata", {}) or {})
+ else:
+ profile_focused_metadata = {
+ "profile_focused_pack_enabled": False,
+ "profile_focused_router_suppressed": True,
+ }
+ _adapter_stage_mark("profile_focused_sec")
+ if _memory_router_allows(memory_router_decision, "topic_tunnel"):
+ topic_bucket_payload = _topic_bucket_rerank_hits(self.graph, query, hits, top_k=top_k)
+ topic_bucket_hits = topic_bucket_payload.get("hits", hits)
+ hits = list(hits if topic_bucket_hits is None else topic_bucket_hits)
+ topic_bucket_metadata = dict(topic_bucket_payload.get("metadata", {}) or {})
+ else:
+ topic_bucket_metadata = {
+ "topic_bucket_rerank_enabled": False,
+ "topic_bucket_router_suppressed": True,
+ }
+ _adapter_stage_mark("topic_bucket_sec")
+ temporal_runtime_payload = self._temporal_runtime_pack(query)
+ temporal_evidence_payload = self._apply_temporal_evidence_pack_to_hits(
+ hits,
+ temporal_runtime_payload,
+ top_k=top_k,
+ )
+ hits = list(temporal_evidence_payload.get("hits", hits) or hits)
+ temporal_runtime_metadata = dict(temporal_evidence_payload.get("metadata", {}) or {})
+ _adapter_stage_mark("temporal_pack_sec")
+ injection_planner_payload = self._apply_injection_planner_to_hits(query, hits, top_k=top_k)
+ hits = list(injection_planner_payload.get("hits", hits) or hits)
+ injection_planner_metadata = dict(injection_planner_payload.get("metadata", {}) or {})
+ _adapter_stage_mark("injection_planner_sec")
+ facet_query_pack_payload = _facet_query_pack_hits(
+ self.graph,
+ query,
+ hits,
+ top_k=top_k,
+ )
+ hits = list(facet_query_pack_payload.get("hits", hits) or hits)
+ facet_query_pack_metadata = dict(facet_query_pack_payload.get("metadata", {}) or {})
+ _adapter_stage_mark("facet_query_pack_sec")
+ unit_coverage_mode = _normalize(os.getenv("TMCRA_UNIT_COVERAGE_PACK_MODE", "on"))
+ if unit_coverage_mode in _MULTI_UNIT_CHAIN_DISABLED_MODES:
+ unit_coverage_metadata = {
+ "unit_coverage_pack_enabled": False,
+ "unit_coverage_reason": "disabled",
+ }
+ else:
+ unit_coverage_payload = _unit_coverage_pack_hits(
+ self.graph,
+ query,
+ hits,
+ top_k=top_k,
+ )
+ hits = list(unit_coverage_payload.get("hits", hits) or hits)
+ unit_coverage_metadata = dict(unit_coverage_payload.get("metadata", {}) or {})
+ _adapter_stage_mark("unit_coverage_sec")
+ if _normalize(os.getenv("TMCRA_MULTI_UNIT_CHAIN_SLOT_MODE", "on")) not in _MULTI_UNIT_CHAIN_DISABLED_MODES:
+ multi_unit_chain_slot_payload = _multi_unit_chain_slot_hits(
+ self.graph,
+ query,
+ hits,
+ top_k=top_k,
+ )
+ hits = list(multi_unit_chain_slot_payload.get("hits", hits) or hits)
+ multi_unit_chain_slot_metadata = dict(multi_unit_chain_slot_payload.get("metadata", {}) or {})
+ else:
+ multi_unit_chain_slot_metadata = {
+ "multi_unit_chain_slot_enabled": False,
+ "multi_unit_chain_slot_reason": "disabled",
+ }
+ _adapter_stage_mark("multi_unit_chain_slot_sec")
+ profile_protected_reinserted_count = 0
+ profile_protected_ids = _dedupe(
+ [
+ *list(profile_focused_metadata.get("profile_focused_pack_memory_ids", []) or []),
+ *list(profile_focused_metadata.get("profile_first_memory_ids", []) or []),
+ ],
+ max_items=max(1, min(6, int(top_k or 1))),
+ )
+ if profile_protected_ids:
+ existing_hits_by_id = {hit.memory_id: hit for hit in hits if hit.memory_id}
+ protected_hits: List[MemoryHit] = []
+ for memory_id in profile_protected_ids:
+ hit = existing_hits_by_id.get(memory_id)
+ if hit is None:
+ record = getattr(self.graph, "records_by_id", {}).get(memory_id)
+ if record is None:
+ continue
+ hit = _memory_hit_from_record(record)
+ metadata = dict(hit.metadata or {})
+ metadata.update(
+ {
+ "profile_first_hybrid_rescue": True,
+ "profile_protected_slot": True,
+ "evidence_snippet_role": "profile_protected_slot",
+ }
+ )
+ protected_hits.append(
+ MemoryHit(
+ memory_id=hit.memory_id,
+ category=hit.category,
+ value=hit.value,
+ relation=hit.relation,
+ anchors=list(hit.anchors),
+ score=max(float(hit.score), 4.8),
+ source_kind=hit.source_kind,
+ slot_key=hit.slot_key,
+ state=hit.state,
+ turn_index=int(hit.turn_index),
+ metadata=metadata,
+ )
+ )
+ if protected_hits:
+ profile_protected_reinserted_count = len(protected_hits)
+ seen_profile_ids = {hit.memory_id for hit in protected_hits if hit.memory_id}
+ hits = [*protected_hits, *[hit for hit in hits if hit.memory_id not in seen_profile_ids]]
+ embedder_fusion_output_event_ids = [
+ _clean_text(event_id)
+ for event_id in list(hybrid_metadata.get("embedder_fusion_selected_event_ids", []) or [])
+ if _clean_text(event_id)
+ ]
+ embedder_fusion_output_reordered = False
+ if embedder_fusion_output_event_ids:
+ before_order = _event_ids_from_hits(hits)
+ seen_output_memory_ids = {hit.memory_id for hit in hits}
+ for event_id in embedder_fusion_output_event_ids:
+ event_hit = _representative_event_hit(_event_record_hits_from_graph(self.graph, event_id), query=query)
+ if event_hit is None or event_hit.memory_id in seen_output_memory_ids:
+ continue
+ metadata = dict(event_hit.metadata or {})
+ metadata.update(
+ {
+ "event_id": event_id,
+ "evidence_snippet_role": "embedder_fusion_event_representative",
+ "embedder_fusion_event_representative": True,
+ }
+ )
+ hits.append(
+ MemoryHit(
+ memory_id=event_hit.memory_id,
+ category=event_hit.category,
+ value=event_hit.value,
+ relation=event_hit.relation,
+ anchors=list(event_hit.anchors),
+ score=float(event_hit.score),
+ source_kind=event_hit.source_kind,
+ slot_key=event_hit.slot_key,
+ state=event_hit.state,
+ turn_index=int(event_hit.turn_index),
+ metadata=metadata,
+ )
+ )
+ seen_output_memory_ids.add(event_hit.memory_id)
+ hits = _coverage_preserving_final_hits(
+ hits,
+ selected_event_ids=_dedupe([*embedder_fusion_output_event_ids, *before_order]),
+ top_k=top_k,
+ )
+ embedder_fusion_output_reordered = before_order != _event_ids_from_hits(hits)
+ retrieval_context_tokens = int(payload.get("context_token_estimate", _estimate_tokens_from_hits(hits)))
+ _adapter_stage_mark("final_protection_and_tokens_sec")
+ postprocess_finished = time.perf_counter()
+ self._last_retrieval_context_tokens = retrieval_context_tokens
+ result = MemoryRetrieval(
+ concepts=list(payload.get("concepts", []) or []),
+ relations=list(payload.get("relations", []) or []),
+ hits=hits,
+ active_hits=active_hits,
+ history_hits=history_hits,
+ stale_hits=stale_hits,
+ overwrite_hits=overwrite_hits,
+ false_hits=false_hits,
+ retrieval_seconds=time.perf_counter() - start,
+ context_token_estimate=retrieval_context_tokens,
+ retrieval_context_token_estimate=retrieval_context_tokens,
+ metadata={
+ "query_id": payload.get("query_id", ""),
+ **dict(payload.get("metadata", {}) or {}),
+ **hybrid_metadata,
+ **memory_router_decision,
+ **current_subject_metadata,
+ **audit_anchor_metadata,
+ **identifier_metadata,
+ **depth_chain_metadata,
+ **profile_focused_metadata,
+ **topic_bucket_metadata,
+ **temporal_runtime_metadata,
+ **injection_planner_metadata,
+ **facet_query_pack_metadata,
+ **unit_coverage_metadata,
+ **multi_unit_chain_slot_metadata,
+ "profile_protected_reinserted_count": profile_protected_reinserted_count,
+ "embedder_fusion_output_event_ids": list(embedder_fusion_output_event_ids),
+ "embedder_fusion_output_reordered": bool(embedder_fusion_output_reordered),
+ },
+ )
+ persisted_retrieval = self._persist_latest_audit("retrieval_log")
+ audit_finished = time.perf_counter()
+ result.metadata["query_id"] = persisted_retrieval.get(
+ "query_id", result.metadata.get("query_id", "")
+ )
+ result.metadata["adapter_runtime_profile"] = {
+ "reload_graph_sec": round(reload_finished - call_started, 6),
+ "heuristic_retrieve_sec": round(heuristic_finished - reload_finished, 6),
+ "hit_prepare_sec": round(hit_prepare_finished - heuristic_finished, 6),
+ "hybrid_graph_sec": round(hybrid_finished - hit_prepare_finished, 6),
+ "postprocess_sec": round(postprocess_finished - hybrid_finished, 6),
+ "audit_persist_sec": round(audit_finished - postprocess_finished, 6),
+ "total_sec": round(audit_finished - call_started, 6),
+ "postprocess_stages": dict(adapter_stage_profile),
+ }
+ return result
+
+ @_serialized_adapter_call
+ def export_dialog_graph(self, *, mode: str = "light") -> Dict[str, Any]:
+ self._reload_graph()
+ return self.graph.export_graph(
+ snapshot_points=(1000, 5000, 10000, 20000, 50000, 100000, 200000, 300000, 500000),
+ mode=mode,
+ )
+
+ @_serialized_adapter_call
+ def export_dialog_graph_mermaid(self) -> str:
+ self._reload_graph()
+ return self.graph.export_mermaid()
+
+ @_serialized_adapter_call
+ def register_answer_support(self, *, answer_id: str, memory_ids: List[str], query_id: str = "", answer_text: str = "") -> None:
+ self._reload_graph()
+ self.graph.register_answer_support(answer_id=answer_id, memory_ids=memory_ids, query_id=query_id, answer_text=answer_text)
+ self._persist_latest_audit("answer_support_log")
+
+ @_serialized_adapter_call
+ def telemetry_snapshot(self) -> Dict[str, Any]:
+ self._reload_graph()
+ return self.graph.summary()
+
+ @_serialized_adapter_call
+ def stats(self) -> Dict[str, Any]:
+ self._reload_graph()
+ storage = self._storage_breakdown()
+ return _state_stats(
+ storage_bytes=storage["storage_bytes"],
+ retrieval_context_tokens=self._last_retrieval_context_tokens,
+ total_state_tokens=storage["total_state_token_estimate"],
+ core_storage_bytes=storage["core_storage_bytes"],
+ audit_storage_bytes=storage["audit_storage_bytes"],
+ core_state_token_estimate=storage["core_state_token_estimate"],
+ audit_state_token_estimate=storage["audit_state_token_estimate"],
+ lightweight_stats=bool(self.lightweight_stats),
+ **self.graph.summary(),
+ )
+
+ @_serialized_adapter_call
+ def storage_bytes(self) -> int:
+ self._reload_graph()
+ return self._storage_breakdown()["storage_bytes"]
+
+ @_serialized_adapter_call
+ def build_prompt_context(self, query: str, *, top_k: int = 8) -> Dict[str, Any]:
+ retrieval = self.retrieve(query, top_k=top_k)
+ retrieval_payload = retrieval.to_dict()
+ stats = self.stats()
+ context_summary = _graph_prompt_state_summary(self.graph, retrieval)
+ prompt_context_payload = {
+ "query": query,
+ "retrieval": retrieval_payload,
+ "context_summary": context_summary,
+ }
+ prompt_context_chars = len(json.dumps(prompt_context_payload, ensure_ascii=False))
+ prompt_context_tokens_est = _estimate_tokens(json.dumps(prompt_context_payload, ensure_ascii=False))
+ return {
+ "mode": "graph_session_memory_v2",
+ "query": query,
+ "retrieval": retrieval_payload,
+ "stats": stats,
+ "state": context_summary,
+ "context_summary": context_summary,
+ "prompt_context_chars": int(prompt_context_chars),
+ "prompt_context_tokens_est": int(prompt_context_tokens_est),
+ "context_truncated": bool(context_summary.get("context_truncated", False)),
+ "truncation_reason": _clean_text(context_summary.get("truncation_reason", "")),
+ }
+
+
+class SummaryWindowMemoryAdapter(MemoryAdapter):
+ name = "summary_window_memory"
+
+ def __init__(self, *, window_size: int = 24, auto_extract: bool = False) -> None:
+ self.extractor = SessionMemoryExtractor()
+ self.window_size = max(4, int(window_size))
+ self.turn_index = 0
+ self.active_slots: Dict[str, SessionMemoryRecordV2] = {}
+ self.recent_turns: deque[Dict[str, Any]] = deque(maxlen=self.window_size)
+ self.auto_extract = bool(auto_extract)
+ self._last_retrieval_context_tokens = 0
+
+ def reset(self) -> None:
+ self.turn_index = 0
+ self.active_slots = {}
+ self.recent_turns = deque(maxlen=self.window_size)
+ self._last_retrieval_context_tokens = 0
+
+ def ingest_turn(
+ self,
+ user_text: str,
+ assistant_text: str = "",
+ *,
+ answer_payload: Dict[str, Any] | None = None,
+ extraction_result: Dict[str, Any] | None = None,
+ ) -> None:
+ self.turn_index += 1
+ records = _build_turn_records(
+ self.extractor,
+ user_text=user_text,
+ answer_payload=answer_payload,
+ extraction_result=extraction_result,
+ turn_index=self.turn_index,
+ allow_auto_extract=self.auto_extract,
+ )
+ for record in records:
+ previous = self.active_slots.get(record.slot_key)
+ if previous:
+ previous.state = "superseded"
+ record.supersedes.append(previous.memory_id)
+ record.state = "active"
+ self.active_slots[record.slot_key] = record
+ self.recent_turns.append(
+ {
+ "turn_index": self.turn_index,
+ "text": _clean_text(user_text),
+ "assistant": _clean_text(assistant_text),
+ }
+ )
+
+ def retrieve(self, query: str, *, top_k: int = 6) -> MemoryRetrieval:
+ start = time.perf_counter()
+ query_tokens = set(_tokenize(query))
+ hints = set(infer_category_hints(query))
+ scored: List[tuple[float, SessionMemoryRecordV2]] = []
+ for record in self.active_slots.values():
+ token_set = record.token_set()
+ overlap = len(query_tokens & token_set) if query_tokens and token_set else 0
+ score = overlap / max(1, len(query_tokens | token_set)) if query_tokens and token_set else 0.0
+ if hints and record.category in hints:
+ score += 0.22
+ if record.slot_key.lower() in _normalize(query):
+ score += 0.12
+ score += min(0.08, record.turn_index * 0.0004)
+ score += 0.2
+ if score > 0:
+ scored.append((score, record))
+ scored.sort(key=lambda item: (item[0], item[1].turn_index), reverse=True)
+ selected_records = [record for _, record in scored[:top_k]]
+ hits = [
+ MemoryHit(
+ memory_id=record.memory_id,
+ category=record.category,
+ value=record.value,
+ relation=record.relation,
+ anchors=list(record.anchor_concepts),
+ score=float(score),
+ source_kind=record.source_kind,
+ slot_key=record.slot_key,
+ state=record.state,
+ turn_index=record.turn_index,
+ metadata={"window_size": self.window_size},
+ )
+ for score, record in scored[:top_k]
+ ]
+ concepts = []
+ relations = []
+ for hit in hits:
+ concepts.append({"concept": hit.value, "type": hit.category, "source_kind": hit.source_kind})
+ for anchor in hit.anchors[:2]:
+ concepts.append({"concept": anchor, "type": "context", "source_kind": hit.source_kind})
+ relation = _relation_hit(hit, weight_bias=0.06)
+ if relation:
+ relations.append(relation)
+ retrieval_context_tokens = _estimate_tokens_from_hits(hits)
+ self._last_retrieval_context_tokens = retrieval_context_tokens
+ return MemoryRetrieval(
+ concepts=concepts,
+ relations=relations,
+ hits=hits,
+ active_hits=list(hits),
+ retrieval_seconds=time.perf_counter() - start,
+ context_token_estimate=retrieval_context_tokens,
+ retrieval_context_token_estimate=retrieval_context_tokens,
+ metadata={
+ "records": len(self.active_slots),
+ "window_size": self.window_size,
+ "recent_turns": len(self.recent_turns),
+ },
+ )
+
+ def stats(self) -> Dict[str, Any]:
+ payload = {
+ "active_slots": {slot: record.to_dict() for slot, record in self.active_slots.items()},
+ "recent_turns": list(self.recent_turns),
+ }
+ total_state_tokens = _estimate_tokens(json.dumps(payload, ensure_ascii=False))
+ return _state_stats(
+ storage_bytes=self.storage_bytes(),
+ retrieval_context_tokens=self._last_retrieval_context_tokens,
+ total_state_tokens=total_state_tokens,
+ records=len(self.active_slots),
+ active_slots=len(self.active_slots),
+ recent_turns=len(self.recent_turns),
+ )
+
+ def storage_bytes(self) -> int:
+ payload = {
+ "active_slots": {slot: record.to_dict() for slot, record in self.active_slots.items()},
+ "recent_turns": list(self.recent_turns),
+ }
+ return len(json.dumps(payload, ensure_ascii=False).encode("utf-8"))
+
+ def build_prompt_context(self, query: str, *, top_k: int = 8) -> Dict[str, Any]:
+ return {
+ "mode": "summary_window_memory",
+ "query": query,
+ "retrieval": self.retrieve(query, top_k=top_k).to_dict(),
+ "stats": self.stats(),
+ "state": {
+ "active_slots": {slot: record.to_dict() for slot, record in self.active_slots.items()},
+ "recent_turns": list(self.recent_turns),
+ },
+ }
+
+
+@dataclass(slots=True)
+class _VectorRecord:
+ memory_id: str
+ category: str
+ value: str
+ relation: str
+ anchors: List[str]
+ tokens: List[str]
+ turn_index: int
+ slot_key: str = ""
+ active: bool = True
+ source_kind: str = "vector_memory"
+ metadata: Dict[str, Any] = field(default_factory=dict)
+
+
+class VectorRAGMemoryAdapter(MemoryAdapter):
+ name = "vector_rag_memory"
+
+ def __init__(self, *, auto_extract: bool = False) -> None:
+ self.extractor = SessionMemoryExtractor()
+ self.records: List[_VectorRecord] = []
+ self.turn_index = 0
+ self.auto_extract = bool(auto_extract)
+ self._last_retrieval_context_tokens = 0
+
+ def reset(self) -> None:
+ self.records = []
+ self.turn_index = 0
+ self._last_retrieval_context_tokens = 0
+
+ def ingest_turn(
+ self,
+ user_text: str,
+ assistant_text: str = "",
+ *,
+ answer_payload: Dict[str, Any] | None = None,
+ extraction_result: Dict[str, Any] | None = None,
+ ) -> None:
+ _ = assistant_text
+ self.turn_index += 1
+ records = _build_turn_records(
+ self.extractor,
+ user_text=user_text,
+ answer_payload=answer_payload,
+ extraction_result=extraction_result,
+ turn_index=self.turn_index,
+ allow_auto_extract=self.auto_extract,
+ )
+ for record in records:
+ if record.slot_key:
+ for previous in self.records:
+ if previous.slot_key == record.slot_key and previous.active:
+ previous.active = False
+ self.records.append(
+ _VectorRecord(
+ memory_id=record.memory_id,
+ category=record.category,
+ value=record.value,
+ relation=record.relation,
+ anchors=list(record.anchor_concepts),
+ tokens=list(record.token_set()),
+ turn_index=record.turn_index,
+ slot_key=record.slot_key,
+ active=record.state == "active",
+ source_kind=record.source_kind,
+ metadata=dict(record.metadata),
+ )
+ )
+
+ def retrieve(self, query: str, *, top_k: int = 6) -> MemoryRetrieval:
+ start = time.perf_counter()
+ query_tokens = set(_tokenize(query))
+ hints = set(infer_category_hints(query))
+ scored: List[tuple[float, _VectorRecord]] = []
+ for record in self.records:
+ token_set = set(record.tokens)
+ overlap = len(query_tokens & token_set) if query_tokens and token_set else 0
+ score = overlap / max(1, len(query_tokens | token_set)) if query_tokens and token_set else 0.0
+ if hints and record.category in hints:
+ score += 0.18
+ if record.slot_key and record.slot_key.lower() in _normalize(query):
+ score += 0.1
+ score += min(0.12, record.turn_index * 0.0004)
+ score += 0.18 if record.active else -0.25
+ if score > 0:
+ scored.append((score, record))
+ if not scored:
+ for record in self.records[-top_k:]:
+ scored.append((0.05 + (0.15 if record.active else 0.0), record))
+ scored.sort(key=lambda item: (item[0], item[1].active, item[1].turn_index), reverse=True)
+ hits = [
+ MemoryHit(
+ memory_id=record.memory_id,
+ category=record.category,
+ value=record.value,
+ relation=record.relation,
+ anchors=list(record.anchors),
+ score=float(score),
+ source_kind=record.source_kind,
+ slot_key=record.slot_key,
+ state="active" if record.active else "superseded",
+ turn_index=record.turn_index,
+ metadata=dict(record.metadata),
+ )
+ for score, record in scored[:top_k]
+ ]
+ concepts = []
+ relations = []
+ for hit in hits:
+ concepts.append({"concept": hit.value, "type": hit.category, "source_kind": hit.source_kind})
+ for anchor in hit.anchors[:2]:
+ concepts.append({"concept": anchor, "type": "context", "source_kind": hit.source_kind})
+ relation = _relation_hit(hit)
+ if relation:
+ relations.append(relation)
+ retrieval_context_tokens = _estimate_tokens_from_hits(hits)
+ self._last_retrieval_context_tokens = retrieval_context_tokens
+ return MemoryRetrieval(
+ concepts=concepts,
+ relations=relations,
+ hits=hits,
+ active_hits=[hit for hit in hits if hit.state == "active"],
+ history_hits=[hit for hit in hits if hit.state != "active"],
+ retrieval_seconds=time.perf_counter() - start,
+ context_token_estimate=retrieval_context_tokens,
+ retrieval_context_token_estimate=retrieval_context_tokens,
+ metadata={
+ "records": len(self.records),
+ "active_records": sum(1 for item in self.records if item.active),
+ },
+ )
+
+ def stats(self) -> Dict[str, Any]:
+ payload = [
+ {
+ "memory_id": record.memory_id,
+ "category": record.category,
+ "value": record.value,
+ "relation": record.relation,
+ "anchors": record.anchors,
+ "slot_key": record.slot_key,
+ "active": record.active,
+ }
+ for record in self.records
+ ]
+ total_state_tokens = _estimate_tokens(json.dumps(payload, ensure_ascii=False))
+ return _state_stats(
+ storage_bytes=self.storage_bytes(),
+ retrieval_context_tokens=self._last_retrieval_context_tokens,
+ total_state_tokens=total_state_tokens,
+ records=len(self.records),
+ active_records=sum(1 for item in self.records if item.active),
+ )
+
+ def storage_bytes(self) -> int:
+ payload = [
+ {
+ "memory_id": record.memory_id,
+ "category": record.category,
+ "value": record.value,
+ "relation": record.relation,
+ "anchors": record.anchors,
+ "slot_key": record.slot_key,
+ "active": record.active,
+ }
+ for record in self.records
+ ]
+ return len(json.dumps(payload, ensure_ascii=False).encode("utf-8"))
+
+ def build_prompt_context(self, query: str, *, top_k: int = 8) -> Dict[str, Any]:
+ state = [
+ {
+ "memory_id": record.memory_id,
+ "category": record.category,
+ "value": record.value,
+ "relation": record.relation,
+ "anchors": list(record.anchors),
+ "slot_key": record.slot_key,
+ "active": bool(record.active),
+ "turn_index": int(record.turn_index),
+ }
+ for record in self.records
+ ]
+ return {
+ "mode": "vector_rag_memory",
+ "query": query,
+ "retrieval": self.retrieve(query, top_k=top_k).to_dict(),
+ "stats": self.stats(),
+ "state": state,
+ }
+
+
+class FullHistoryMemoryAdapter(MemoryAdapter):
+ name = "full_history_memory"
+
+ def __init__(self) -> None:
+ self.turns: List[Dict[str, str]] = []
+ self._last_retrieval_context_tokens = 0
+
+ def reset(self) -> None:
+ self.turns = []
+ self._last_retrieval_context_tokens = 0
+
+ def ingest_turn(
+ self,
+ user_text: str,
+ assistant_text: str = "",
+ *,
+ answer_payload: Dict[str, Any] | None = None,
+ extraction_result: Dict[str, Any] | None = None,
+ ) -> None:
+ _ = answer_payload, extraction_result
+ self.turns.append({"user": _clean_text(user_text), "assistant": _clean_text(assistant_text)})
+
+ def retrieve(self, query: str, *, top_k: int = 6) -> MemoryRetrieval:
+ start = time.perf_counter()
+ query_tokens = set(_tokenize(query))
+ scored: List[tuple[float, Dict[str, str], int]] = []
+ for index, turn in enumerate(self.turns):
+ combined = f"{turn.get('user', '')} {turn.get('assistant', '')}"
+ token_set = set(_tokenize(combined))
+ if not token_set:
+ continue
+ overlap = len(query_tokens & token_set) if query_tokens else 0
+ score = overlap / max(1, len(query_tokens | token_set)) if query_tokens else 0.0
+ scored.append((score, turn, index))
+ scored.sort(key=lambda item: (item[0], item[2]), reverse=True)
+ hits = [
+ MemoryHit(
+ memory_id=f"turn:{index}",
+ category="history_turn",
+ value=turn.get("user", ""),
+ relation="conversation_context",
+ anchors=[turn.get("assistant", "")] if turn.get("assistant") else [],
+ score=float(score),
+ source_kind="full_history",
+ slot_key=f"turn.{index}",
+ state="active",
+ turn_index=index + 1,
+ )
+ for score, turn, index in scored[:top_k]
+ if turn.get("user", "")
+ ]
+ retrieval_context_tokens = _estimate_tokens(json.dumps(self.turns, ensure_ascii=False))
+ self._last_retrieval_context_tokens = retrieval_context_tokens
+ return MemoryRetrieval(
+ hits=hits,
+ active_hits=list(hits),
+ retrieval_seconds=time.perf_counter() - start,
+ context_token_estimate=retrieval_context_tokens,
+ retrieval_context_token_estimate=retrieval_context_tokens,
+ metadata={"records": len(self.turns)},
+ )
+
+ def stats(self) -> Dict[str, Any]:
+ total_state_tokens = _estimate_tokens(json.dumps(self.turns, ensure_ascii=False))
+ return _state_stats(
+ storage_bytes=self.storage_bytes(),
+ retrieval_context_tokens=self._last_retrieval_context_tokens,
+ total_state_tokens=total_state_tokens,
+ records=len(self.turns),
+ )
+
+ def storage_bytes(self) -> int:
+ return len(json.dumps(self.turns, ensure_ascii=False).encode("utf-8"))
+
+ def build_prompt_context(self, query: str, *, top_k: int = 8) -> Dict[str, Any]:
+ return {
+ "mode": "full_history_memory",
+ "query": query,
+ "retrieval": self.retrieve(query, top_k=top_k).to_dict(),
+ "stats": self.stats(),
+ "state": {
+ "turns": list(self.turns),
+ },
+ }
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/generic_memory.py b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/generic_memory.py
new file mode 100644
index 0000000000000000000000000000000000000000..b481c43f884cf4d478f51aefa258877d00e8a904
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/generic_memory.py
@@ -0,0 +1,969 @@
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from contextlib import contextmanager
+from dataclasses import dataclass, field, replace
+import json
+import math
+from pathlib import Path
+import re
+import sqlite3
+import time
+import uuid
+from typing import Any, Dict, Iterable, List, Sequence
+
+
+def _clean_text(value: Any) -> str:
+ return re.sub(r"\s+", " ", str(value or "").strip())
+
+
+def _normalize(value: Any) -> str:
+ return _clean_text(value).lower()
+
+
+def _dedupe(items: Iterable[Any]) -> List[str]:
+ results: List[str] = []
+ seen = set()
+ for item in items:
+ text = _clean_text(item)
+ if not text:
+ continue
+ key = _normalize(text)
+ if key in seen:
+ continue
+ seen.add(key)
+ results.append(text)
+ return results
+
+
+def _tokenize(value: Any) -> List[str]:
+ text = _normalize(value)
+ if not text:
+ return []
+ english = re.findall(r"[a-z0-9_.-]+", text)
+ cjk = [char for char in text if "\u4e00" <= char <= "\u9fff"]
+ if english or cjk:
+ return _dedupe([*english, *cjk])
+ return [char for char in text if char.strip()]
+
+
+def _estimate_tokens(value: Any) -> int:
+ text = _clean_text(value)
+ if not text:
+ return 0
+ return max(1, math.ceil(len(text) / 4))
+
+
+def _json_dumps(value: Any) -> str:
+ return json.dumps(value, ensure_ascii=False, sort_keys=True)
+
+
+def _to_float(value: Any, default: float) -> float:
+ try:
+ return float(value)
+ except (TypeError, ValueError):
+ return float(default)
+
+
+def _to_int(value: Any, default: int) -> int:
+ try:
+ return int(value)
+ except (TypeError, ValueError):
+ return int(default)
+
+
+@dataclass(slots=True)
+class MemorySessionScope:
+ scope_id: str
+ namespace: str = "default"
+ profile_name: str = ""
+ metadata: Dict[str, Any] = field(default_factory=dict)
+
+ def to_dict(self) -> Dict[str, Any]:
+ return {
+ "scope_id": self.scope_id,
+ "namespace": self.namespace,
+ "profile_name": self.profile_name,
+ "metadata": dict(self.metadata),
+ }
+
+
+@dataclass(slots=True)
+class MemoryPolicy:
+ overwrite_mode: str = "slot_latest"
+ dedupe_mode: str = "key_value_active"
+ retention_mode: str = "bounded_audit"
+ writeback_mode: str = "explicit_and_derived"
+ conflict_mode: str = "keep_history"
+ audit_retention: int = 256
+
+ def to_dict(self) -> Dict[str, Any]:
+ return {
+ "overwrite_mode": self.overwrite_mode,
+ "dedupe_mode": self.dedupe_mode,
+ "retention_mode": self.retention_mode,
+ "writeback_mode": self.writeback_mode,
+ "conflict_mode": self.conflict_mode,
+ "audit_retention": int(self.audit_retention),
+ }
+
+
+@dataclass(slots=True)
+class MemoryRecord:
+ record_id: str
+ namespace: str
+ type: str
+ key: str
+ value: str
+ attributes: Dict[str, Any] = field(default_factory=dict)
+ state: str = "active"
+ supersedes: List[str] = field(default_factory=list)
+ provenance: Dict[str, Any] = field(default_factory=dict)
+ timestamps: Dict[str, Any] = field(default_factory=dict)
+ scope_id: str = ""
+
+ def search_text(self) -> str:
+ payload = {
+ "namespace": self.namespace,
+ "type": self.type,
+ "key": self.key,
+ "value": self.value,
+ "attributes": self.attributes,
+ "provenance": self.provenance,
+ }
+ return _json_dumps(payload)
+
+ def to_dict(self) -> Dict[str, Any]:
+ return {
+ "record_id": self.record_id,
+ "namespace": self.namespace,
+ "type": self.type,
+ "key": self.key,
+ "value": self.value,
+ "attributes": dict(self.attributes),
+ "state": self.state,
+ "supersedes": list(self.supersedes),
+ "provenance": dict(self.provenance),
+ "timestamps": dict(self.timestamps),
+ "scope_id": self.scope_id,
+ }
+
+
+@dataclass(slots=True)
+class MemoryRetrievalEntry:
+ record: MemoryRecord
+ score: float = 0.0
+ matched_terms: List[str] = field(default_factory=list)
+ rank: int = 0
+
+ def to_dict(self) -> Dict[str, Any]:
+ return {
+ "record": self.record.to_dict(),
+ "score": round(float(self.score), 6),
+ "matched_terms": list(self.matched_terms),
+ "rank": int(self.rank),
+ }
+
+
+@dataclass(slots=True)
+class MemoryRetrievalResult:
+ records: List[MemoryRetrievalEntry] = field(default_factory=list)
+ active_records: List[MemoryRetrievalEntry] = field(default_factory=list)
+ historical_records: List[MemoryRetrievalEntry] = field(default_factory=list)
+ relations: List[Dict[str, Any]] = field(default_factory=list)
+ retrieval_metadata: Dict[str, Any] = field(default_factory=dict)
+
+ def to_dict(self) -> Dict[str, Any]:
+ return {
+ "records": [item.to_dict() for item in self.records],
+ "active_records": [item.to_dict() for item in self.active_records],
+ "historical_records": [item.to_dict() for item in self.historical_records],
+ "relations": list(self.relations),
+ "retrieval_metadata": dict(self.retrieval_metadata),
+ }
+
+
+@dataclass(slots=True)
+class MemoryWritebackRequest:
+ writeback_type: str = "derived_writeback"
+ namespace: str = "default"
+ query_text: str = ""
+ answer_text: str = ""
+ records: List[Dict[str, Any]] = field(default_factory=list)
+ supports: List[Dict[str, Any]] = field(default_factory=list)
+ metadata: Dict[str, Any] = field(default_factory=dict)
+
+ def to_dict(self) -> Dict[str, Any]:
+ return {
+ "writeback_type": self.writeback_type,
+ "namespace": self.namespace,
+ "query_text": self.query_text,
+ "answer_text": self.answer_text,
+ "records": [dict(item) for item in self.records],
+ "supports": [dict(item) for item in self.supports],
+ "metadata": dict(self.metadata),
+ }
+
+
+class MemoryProfile(ABC):
+ profile_name: str = "generic"
+
+ def policy(self) -> MemoryPolicy:
+ return MemoryPolicy()
+
+ def normalize_event(self, event_kind: str, payload: Dict[str, Any] | None, scope: MemorySessionScope) -> Dict[str, Any]:
+ _ = event_kind, scope
+ normalized = dict(payload or {})
+ raw_records = normalized.get("records")
+ if raw_records is None:
+ raw_records = normalized.get("writeback_records")
+ if raw_records is None:
+ raw_records = normalized.get("replacement_memory_records")
+ if isinstance(raw_records, dict):
+ raw_records = [raw_records]
+ normalized["records"] = [dict(item) for item in list(raw_records or []) if isinstance(item, dict)]
+ normalized["metadata"] = dict(normalized.get("metadata") or {})
+ return normalized
+
+ @abstractmethod
+ def derive_records(self, event_kind: str, payload: Dict[str, Any], scope: MemorySessionScope) -> List[MemoryRecord]:
+ raise NotImplementedError
+
+ @abstractmethod
+ def resolve_key(
+ self,
+ record_type: str,
+ value: str,
+ attributes: Dict[str, Any],
+ *,
+ namespace: str,
+ provided_key: str = "",
+ relation: str = "",
+ provenance: Dict[str, Any] | None = None,
+ ) -> str:
+ raise NotImplementedError
+
+ def infer_query_types(self, query: str) -> List[str]:
+ _ = query
+ return []
+
+ def record_search_text(self, record: MemoryRecord) -> str:
+ return record.search_text()
+
+ def score_bonus(self, query: str, record: MemoryRecord) -> float:
+ _ = query, record
+ return 0.0
+
+ def build_relations(self, retrieval: MemoryRetrievalResult) -> List[Dict[str, Any]]:
+ relations: List[Dict[str, Any]] = []
+ seen = set()
+ for entry in retrieval.records:
+ record = entry.record
+ for superseded_id in record.supersedes:
+ key = ("supersedes", superseded_id, record.record_id)
+ if key in seen:
+ continue
+ seen.add(key)
+ relations.append(
+ {
+ "from": superseded_id,
+ "to": record.record_id,
+ "relation": "supersedes",
+ "score": round(float(entry.score), 6),
+ }
+ )
+ relation = _clean_text(record.attributes.get("relation", ""))
+ anchors = [anchor for anchor in list(record.attributes.get("anchors", []) or []) if _clean_text(anchor)]
+ if relation and anchors:
+ key = (relation, anchors[0], record.value)
+ if key in seen:
+ continue
+ seen.add(key)
+ relations.append(
+ {
+ "from": anchors[0],
+ "to": record.value,
+ "relation": relation,
+ "score": round(float(entry.score), 6),
+ }
+ )
+ return relations
+
+ def render_context(self, query: str, retrieval: MemoryRetrievalResult, scope: MemorySessionScope) -> Dict[str, Any]:
+ lines = [f"[{item.record.type}] {item.record.key}: {item.record.value}" for item in retrieval.records[:8]]
+ return {
+ "profile_name": self.profile_name,
+ "query": query,
+ "scope": scope.to_dict(),
+ "lines": lines,
+ "record_ids": [item.record.record_id for item in retrieval.records],
+ }
+
+
+class MemoryStore(ABC):
+ @abstractmethod
+ def next_turn_index(self, scope: MemorySessionScope) -> int:
+ raise NotImplementedError
+
+ @abstractmethod
+ def save_records(self, scope: MemorySessionScope, records: Sequence[MemoryRecord], policy: MemoryPolicy) -> List[MemoryRecord]:
+ raise NotImplementedError
+
+ @abstractmethod
+ def list_records(self, scope: MemorySessionScope) -> List[MemoryRecord]:
+ raise NotImplementedError
+
+ @abstractmethod
+ def append_support(self, scope: MemorySessionScope, *, answer_id: str, record_ids: Sequence[str], payload: Dict[str, Any]) -> None:
+ raise NotImplementedError
+
+ @abstractmethod
+ def append_audit(self, scope: MemorySessionScope, *, event_kind: str, payload: Dict[str, Any]) -> None:
+ raise NotImplementedError
+
+ @abstractmethod
+ def scope_stats(self, scope: MemorySessionScope) -> Dict[str, Any]:
+ raise NotImplementedError
+
+ @abstractmethod
+ def storage_bytes(self, scope: MemorySessionScope) -> int:
+ raise NotImplementedError
+
+ @abstractmethod
+ def reset_scope(self, scope: MemorySessionScope) -> None:
+ raise NotImplementedError
+
+
+class SQLiteMemoryStore(MemoryStore):
+ def __init__(self, storage_path: str | Path, *, audit_retention: int = 256) -> None:
+ self.storage_path = Path(storage_path).expanduser().resolve()
+ self.audit_retention = max(1, int(audit_retention))
+ self.storage_path.parent.mkdir(parents=True, exist_ok=True)
+ self._init_schema()
+
+ def _connect(self) -> sqlite3.Connection:
+ connection = sqlite3.connect(str(self.storage_path))
+ connection.row_factory = sqlite3.Row
+ connection.execute("PRAGMA journal_mode=WAL")
+ connection.execute("PRAGMA synchronous=NORMAL")
+ return connection
+
+ @contextmanager
+ def _managed_connection(self) -> Iterable[sqlite3.Connection]:
+ connection = self._connect()
+ try:
+ with connection:
+ yield connection
+ finally:
+ connection.close()
+
+ def _init_schema(self) -> None:
+ with self._managed_connection() as connection:
+ connection.executescript(
+ """
+ CREATE TABLE IF NOT EXISTS memory_scope_meta (
+ scope_id TEXT PRIMARY KEY,
+ namespace TEXT NOT NULL,
+ profile_name TEXT NOT NULL,
+ metadata_json TEXT NOT NULL,
+ last_turn_index INTEGER NOT NULL DEFAULT 0,
+ created_at REAL NOT NULL,
+ updated_at REAL NOT NULL
+ );
+ CREATE TABLE IF NOT EXISTS memory_records (
+ scope_id TEXT NOT NULL,
+ record_id TEXT NOT NULL,
+ namespace TEXT NOT NULL,
+ type TEXT NOT NULL,
+ key TEXT NOT NULL,
+ value TEXT NOT NULL,
+ attributes_json TEXT NOT NULL,
+ state TEXT NOT NULL,
+ supersedes_json TEXT NOT NULL,
+ provenance_json TEXT NOT NULL,
+ timestamps_json TEXT NOT NULL,
+ created_at REAL NOT NULL,
+ updated_at REAL NOT NULL,
+ PRIMARY KEY (scope_id, record_id)
+ );
+ CREATE INDEX IF NOT EXISTS idx_memory_records_scope_key_state
+ ON memory_records(scope_id, namespace, key, state, updated_at);
+ CREATE INDEX IF NOT EXISTS idx_memory_records_scope_type
+ ON memory_records(scope_id, type, updated_at);
+ CREATE TABLE IF NOT EXISTS memory_support_log (
+ scope_id TEXT NOT NULL,
+ support_id TEXT NOT NULL,
+ answer_id TEXT NOT NULL,
+ record_ids_json TEXT NOT NULL,
+ payload_json TEXT NOT NULL,
+ created_at REAL NOT NULL,
+ PRIMARY KEY (scope_id, support_id)
+ );
+ CREATE TABLE IF NOT EXISTS memory_audit (
+ scope_id TEXT NOT NULL,
+ event_kind TEXT NOT NULL,
+ event_id TEXT NOT NULL,
+ payload_json TEXT NOT NULL,
+ created_at REAL NOT NULL,
+ PRIMARY KEY (scope_id, event_id)
+ );
+ CREATE INDEX IF NOT EXISTS idx_memory_audit_scope_kind_created
+ ON memory_audit(scope_id, event_kind, created_at);
+ CREATE TABLE IF NOT EXISTS memory_audit_meta (
+ scope_id TEXT NOT NULL,
+ event_kind TEXT NOT NULL,
+ total_count INTEGER NOT NULL DEFAULT 0,
+ trimmed_count INTEGER NOT NULL DEFAULT 0,
+ PRIMARY KEY (scope_id, event_kind)
+ );
+ """
+ )
+
+ def _ensure_scope_meta(self, connection: sqlite3.Connection, scope: MemorySessionScope) -> None:
+ now = time.time()
+ connection.execute(
+ """
+ INSERT INTO memory_scope_meta(scope_id, namespace, profile_name, metadata_json, last_turn_index, created_at, updated_at)
+ VALUES(?, ?, ?, ?, 0, ?, ?)
+ ON CONFLICT(scope_id) DO UPDATE SET
+ namespace = excluded.namespace,
+ profile_name = CASE
+ WHEN excluded.profile_name != '' THEN excluded.profile_name
+ ELSE memory_scope_meta.profile_name
+ END,
+ metadata_json = excluded.metadata_json,
+ updated_at = excluded.updated_at
+ """,
+ (
+ scope.scope_id,
+ _clean_text(scope.namespace) or "default",
+ _clean_text(scope.profile_name),
+ _json_dumps(scope.metadata),
+ float(now),
+ float(now),
+ ),
+ )
+
+ def _row_to_record(self, row: sqlite3.Row) -> MemoryRecord:
+ return MemoryRecord(
+ record_id=str(row["record_id"]),
+ namespace=str(row["namespace"]),
+ type=str(row["type"]),
+ key=str(row["key"]),
+ value=str(row["value"]),
+ attributes=json.loads(row["attributes_json"] or "{}"),
+ state=str(row["state"]),
+ supersedes=list(json.loads(row["supersedes_json"] or "[]")),
+ provenance=json.loads(row["provenance_json"] or "{}"),
+ timestamps=json.loads(row["timestamps_json"] or "{}"),
+ scope_id=str(row["scope_id"]),
+ )
+
+ def next_turn_index(self, scope: MemorySessionScope) -> int:
+ with self._managed_connection() as connection:
+ self._ensure_scope_meta(connection, scope)
+ row = connection.execute(
+ "SELECT last_turn_index FROM memory_scope_meta WHERE scope_id = ?",
+ (scope.scope_id,),
+ ).fetchone()
+ next_turn = int(row["last_turn_index"] or 0) + 1 if row is not None else 1
+ connection.execute(
+ "UPDATE memory_scope_meta SET last_turn_index = ?, updated_at = ? WHERE scope_id = ?",
+ (int(next_turn), float(time.time()), scope.scope_id),
+ )
+ return int(next_turn)
+
+ def save_records(self, scope: MemorySessionScope, records: Sequence[MemoryRecord], policy: MemoryPolicy) -> List[MemoryRecord]:
+ stored: List[MemoryRecord] = []
+ with self._managed_connection() as connection:
+ self._ensure_scope_meta(connection, scope)
+ for original in records:
+ record = replace(
+ original,
+ scope_id=scope.scope_id,
+ namespace=_clean_text(original.namespace) or _clean_text(scope.namespace) or "default",
+ state=_clean_text(original.state) or "active",
+ )
+ if _normalize(policy.dedupe_mode) == "key_value_active":
+ existing = connection.execute(
+ """
+ SELECT * FROM memory_records
+ WHERE scope_id = ? AND namespace = ? AND key = ? AND value = ? AND state = 'active'
+ ORDER BY updated_at DESC
+ LIMIT 1
+ """,
+ (scope.scope_id, record.namespace, record.key, record.value),
+ ).fetchone()
+ if existing is not None and _normalize(existing["type"]) == _normalize(record.type):
+ stored.append(self._row_to_record(existing))
+ continue
+ superseded_ids: List[str] = list(record.supersedes)
+ if _normalize(record.state) == "active" and _normalize(policy.overwrite_mode) in {"latest", "slot_latest"}:
+ active_rows = connection.execute(
+ """
+ SELECT record_id FROM memory_records
+ WHERE scope_id = ? AND namespace = ? AND key = ? AND state = 'active'
+ ORDER BY updated_at DESC
+ """,
+ (scope.scope_id, record.namespace, record.key),
+ ).fetchall()
+ active_ids = [str(row["record_id"]) for row in active_rows if _clean_text(row["record_id"])]
+ superseded_ids.extend(active_ids)
+ if active_ids:
+ placeholders = ", ".join("?" for _ in active_ids)
+ connection.execute(
+ f"""
+ UPDATE memory_records
+ SET state = 'superseded', updated_at = ?
+ WHERE scope_id = ? AND record_id IN ({placeholders})
+ """,
+ (float(time.time()), scope.scope_id, *active_ids),
+ )
+ created_at = _to_float(record.timestamps.get("created_at"), time.time())
+ updated_at = _to_float(record.timestamps.get("updated_at"), created_at)
+ record = replace(
+ record,
+ supersedes=_dedupe(superseded_ids),
+ timestamps={
+ **dict(record.timestamps),
+ "created_at": created_at,
+ "updated_at": updated_at,
+ },
+ )
+ connection.execute(
+ """
+ INSERT OR REPLACE INTO memory_records(
+ scope_id, record_id, namespace, type, key, value,
+ attributes_json, state, supersedes_json, provenance_json,
+ timestamps_json, created_at, updated_at
+ ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+ """,
+ (
+ scope.scope_id,
+ record.record_id,
+ record.namespace,
+ record.type,
+ record.key,
+ record.value,
+ _json_dumps(record.attributes),
+ record.state,
+ _json_dumps(record.supersedes),
+ _json_dumps(record.provenance),
+ _json_dumps(record.timestamps),
+ float(created_at),
+ float(updated_at),
+ ),
+ )
+ stored.append(record)
+ connection.execute(
+ "UPDATE memory_scope_meta SET profile_name = ?, updated_at = ? WHERE scope_id = ?",
+ (_clean_text(scope.profile_name), float(time.time()), scope.scope_id),
+ )
+ return stored
+
+ def list_records(self, scope: MemorySessionScope) -> List[MemoryRecord]:
+ with self._managed_connection() as connection:
+ rows = connection.execute(
+ "SELECT * FROM memory_records WHERE scope_id = ? ORDER BY created_at ASC, record_id ASC",
+ (scope.scope_id,),
+ ).fetchall()
+ records = [self._row_to_record(row) for row in rows]
+ records.sort(
+ key=lambda item: (
+ _to_int(item.timestamps.get("turn_index"), 0),
+ _to_float(item.timestamps.get("created_at"), 0.0),
+ item.record_id,
+ )
+ )
+ return records
+
+ def append_support(self, scope: MemorySessionScope, *, answer_id: str, record_ids: Sequence[str], payload: Dict[str, Any]) -> None:
+ with self._managed_connection() as connection:
+ self._ensure_scope_meta(connection, scope)
+ connection.execute(
+ """
+ INSERT INTO memory_support_log(scope_id, support_id, answer_id, record_ids_json, payload_json, created_at)
+ VALUES (?, ?, ?, ?, ?, ?)
+ """,
+ (
+ scope.scope_id,
+ uuid.uuid4().hex,
+ _clean_text(answer_id),
+ _json_dumps(_dedupe(record_ids)),
+ _json_dumps(payload),
+ float(time.time()),
+ ),
+ )
+
+ def append_audit(self, scope: MemorySessionScope, *, event_kind: str, payload: Dict[str, Any]) -> None:
+ normalized_kind = _clean_text(event_kind) or "event"
+ with self._managed_connection() as connection:
+ self._ensure_scope_meta(connection, scope)
+ connection.execute(
+ """
+ INSERT INTO memory_audit(scope_id, event_kind, event_id, payload_json, created_at)
+ VALUES (?, ?, ?, ?, ?)
+ """,
+ (scope.scope_id, normalized_kind, uuid.uuid4().hex, _json_dumps(payload), float(time.time())),
+ )
+ connection.execute(
+ """
+ INSERT INTO memory_audit_meta(scope_id, event_kind, total_count, trimmed_count)
+ VALUES (?, ?, 1, 0)
+ ON CONFLICT(scope_id, event_kind) DO UPDATE SET
+ total_count = memory_audit_meta.total_count + 1
+ """,
+ (scope.scope_id, normalized_kind),
+ )
+ retained = connection.execute(
+ "SELECT COUNT(*) AS retained FROM memory_audit WHERE scope_id = ? AND event_kind = ?",
+ (scope.scope_id, normalized_kind),
+ ).fetchone()
+ overflow = max(0, int(retained["retained"] if retained is not None else 0) - self.audit_retention)
+ if overflow:
+ rows = connection.execute(
+ """
+ SELECT event_id FROM memory_audit
+ WHERE scope_id = ? AND event_kind = ?
+ ORDER BY created_at ASC, event_id ASC
+ LIMIT ?
+ """,
+ (scope.scope_id, normalized_kind, int(overflow)),
+ ).fetchall()
+ event_ids = [str(row["event_id"]) for row in rows]
+ if event_ids:
+ placeholders = ", ".join("?" for _ in event_ids)
+ connection.execute(
+ f"DELETE FROM memory_audit WHERE scope_id = ? AND event_id IN ({placeholders})",
+ (scope.scope_id, *event_ids),
+ )
+ connection.execute(
+ """
+ UPDATE memory_audit_meta
+ SET trimmed_count = trimmed_count + ?
+ WHERE scope_id = ? AND event_kind = ?
+ """,
+ (int(overflow), scope.scope_id, normalized_kind),
+ )
+
+ def scope_stats(self, scope: MemorySessionScope) -> Dict[str, Any]:
+ records = self.list_records(scope)
+ active_records = [record for record in records if _normalize(record.state) == "active"]
+ historical_records = [record for record in records if _normalize(record.state) != "active"]
+ with self._managed_connection() as connection:
+ support_row = connection.execute(
+ "SELECT COUNT(*) AS count FROM memory_support_log WHERE scope_id = ?",
+ (scope.scope_id,),
+ ).fetchone()
+ meta_rows = connection.execute(
+ "SELECT event_kind, total_count, trimmed_count FROM memory_audit_meta WHERE scope_id = ?",
+ (scope.scope_id,),
+ ).fetchall()
+ retained_rows = connection.execute(
+ "SELECT event_kind, COUNT(*) AS retained FROM memory_audit WHERE scope_id = ? GROUP BY event_kind",
+ (scope.scope_id,),
+ ).fetchall()
+ meta_row = connection.execute(
+ "SELECT profile_name, last_turn_index FROM memory_scope_meta WHERE scope_id = ?",
+ (scope.scope_id,),
+ ).fetchone()
+ retained_lookup = {str(row["event_kind"]): int(row["retained"]) for row in retained_rows}
+ audit_totals = {str(row["event_kind"]): int(row["total_count"]) for row in meta_rows}
+ audit_trimmed = {str(row["event_kind"]): int(row["trimmed_count"]) for row in meta_rows}
+ support_count = int(support_row["count"] if support_row is not None else 0)
+ payload = {
+ "records": [record.to_dict() for record in records],
+ "support_events": support_count,
+ "audit_totals": audit_totals,
+ "audit_trimmed": audit_trimmed,
+ }
+ return {
+ "records": len(records),
+ "active_records": len(active_records),
+ "historical_records": len(historical_records),
+ "support_events": support_count,
+ "audit_totals": audit_totals,
+ "audit_trimmed": audit_trimmed,
+ "audit_retained": retained_lookup,
+ "last_turn_index": int(meta_row["last_turn_index"] if meta_row is not None else 0),
+ "profile_name": _clean_text(meta_row["profile_name"] if meta_row is not None else scope.profile_name),
+ "state_token_estimate": _estimate_tokens(_json_dumps(payload)),
+ }
+
+ def storage_bytes(self, scope: MemorySessionScope) -> int:
+ payload = {
+ "records": [record.to_dict() for record in self.list_records(scope)],
+ "scope": scope.to_dict(),
+ }
+ return len(_json_dumps(payload).encode("utf-8"))
+
+ def reset_scope(self, scope: MemorySessionScope) -> None:
+ with self._managed_connection() as connection:
+ connection.execute("DELETE FROM memory_records WHERE scope_id = ?", (scope.scope_id,))
+ connection.execute("DELETE FROM memory_support_log WHERE scope_id = ?", (scope.scope_id,))
+ connection.execute("DELETE FROM memory_audit WHERE scope_id = ?", (scope.scope_id,))
+ connection.execute("DELETE FROM memory_audit_meta WHERE scope_id = ?", (scope.scope_id,))
+ connection.execute("DELETE FROM memory_scope_meta WHERE scope_id = ?", (scope.scope_id,))
+
+
+class MemoryRuntime:
+ def __init__(
+ self,
+ *,
+ profile: MemoryProfile,
+ store: MemoryStore | None = None,
+ storage_backend: str = "sqlite",
+ storage_path: str = "",
+ scope: MemorySessionScope | None = None,
+ audit_retention: int = 256,
+ ) -> None:
+ self.profile = profile
+ self.storage_backend = _normalize(storage_backend) or "sqlite"
+ if store is not None:
+ self.store = store
+ elif self.storage_backend == "sqlite":
+ resolved_path = _clean_text(storage_path) or str((Path.cwd() / "outputs" / "generic_memory.sqlite3").resolve())
+ self.store = SQLiteMemoryStore(resolved_path, audit_retention=audit_retention)
+ else:
+ raise ValueError(f"Unsupported storage backend: {storage_backend}")
+ self.storage_path = str(getattr(self.store, "storage_path", _clean_text(storage_path)))
+ self.default_scope = self._resolve_scope(scope or MemorySessionScope(scope_id="default", namespace="default", profile_name=self.profile.profile_name))
+ self._last_retrieval_tokens: Dict[str, int] = {}
+
+ def _resolve_scope(self, session_scope: MemorySessionScope | str | None) -> MemorySessionScope:
+ if isinstance(session_scope, MemorySessionScope):
+ scope_id = _clean_text(session_scope.scope_id) or "default"
+ return MemorySessionScope(
+ scope_id=scope_id,
+ namespace=_clean_text(session_scope.namespace) or "default",
+ profile_name=_clean_text(session_scope.profile_name) or self.profile.profile_name,
+ metadata=dict(session_scope.metadata),
+ )
+ if isinstance(session_scope, str) and _clean_text(session_scope):
+ return MemorySessionScope(scope_id=_clean_text(session_scope), namespace="default", profile_name=self.profile.profile_name)
+ if hasattr(self, "default_scope"):
+ return MemorySessionScope(
+ scope_id=self.default_scope.scope_id,
+ namespace=self.default_scope.namespace,
+ profile_name=self.profile.profile_name,
+ metadata=dict(self.default_scope.metadata),
+ )
+ return MemorySessionScope(scope_id="default", namespace="default", profile_name=self.profile.profile_name)
+
+ def ingest_event(self, event_kind: str, payload: Dict[str, Any] | None, session_scope: MemorySessionScope | str | None = None) -> List[MemoryRecord]:
+ scope = self._resolve_scope(session_scope)
+ turn_index = self.store.next_turn_index(scope)
+ created_at = time.time()
+ normalized = self.profile.normalize_event(event_kind, payload or {}, scope)
+ normalized.update({"turn_index": int(turn_index), "created_at": float(created_at)})
+ records = self.profile.derive_records(event_kind, normalized, scope)
+ stored = self.store.save_records(scope, records, self.profile.policy())
+ self.store.append_audit(
+ scope,
+ event_kind=event_kind,
+ payload={
+ "event_kind": event_kind,
+ "turn_index": int(turn_index),
+ "record_ids": [record.record_id for record in stored],
+ "metadata": dict(normalized.get("metadata") or {}),
+ },
+ )
+ return stored
+
+ def retrieve(self, query: str, session_scope: MemorySessionScope | str | None = None, *, top_k: int = 6) -> MemoryRetrievalResult:
+ scope = self._resolve_scope(session_scope)
+ records = self.store.list_records(scope)
+ query_tokens = set(_tokenize(query))
+ query_types = {_normalize(item) for item in self.profile.infer_query_types(query)}
+ scored: List[MemoryRetrievalEntry] = []
+ for record in records:
+ record_tokens = set(_tokenize(self.profile.record_search_text(record)))
+ overlap = len(query_tokens & record_tokens) if query_tokens and record_tokens else 0
+ union = len(query_tokens | record_tokens) if query_tokens and record_tokens else 0
+ score = overlap / max(1, union) if union else 0.0
+ if query_types and _normalize(record.type) in query_types:
+ score += 0.18
+ if _normalize(record.state) == "active":
+ score += 0.12
+ if record.key and _normalize(record.key) in _normalize(query):
+ score += 0.08
+ score += float(self.profile.score_bonus(query, record) or 0.0)
+ score += min(0.08, float(_to_int(record.timestamps.get("turn_index"), 0)) * 0.0005)
+ if score > 0:
+ scored.append(
+ MemoryRetrievalEntry(
+ record=record,
+ score=float(score),
+ matched_terms=sorted(query_tokens & record_tokens)[:8],
+ )
+ )
+ if not scored:
+ fallback = list(records)
+ fallback.sort(
+ key=lambda item: (
+ _normalize(item.state) == "active",
+ _to_int(item.timestamps.get("turn_index"), 0),
+ _to_float(item.timestamps.get("created_at"), 0.0),
+ ),
+ reverse=True,
+ )
+ scored = [
+ MemoryRetrievalEntry(record=item, score=(0.15 if _normalize(item.state) == "active" else 0.05), matched_terms=[])
+ for item in fallback[: max(1, int(top_k))]
+ ]
+ scored.sort(
+ key=lambda item: (
+ float(item.score),
+ _normalize(item.record.state) == "active",
+ _to_int(item.record.timestamps.get("turn_index"), 0),
+ item.record.record_id,
+ ),
+ reverse=True,
+ )
+ selected = [replace(item, rank=index + 1) for index, item in enumerate(scored[: max(1, int(top_k))])]
+ retrieval = MemoryRetrievalResult(
+ records=selected,
+ active_records=[item for item in selected if _normalize(item.record.state) == "active"],
+ historical_records=[item for item in selected if _normalize(item.record.state) != "active"],
+ retrieval_metadata={
+ "profile_name": self.profile.profile_name,
+ "scope_id": scope.scope_id,
+ "namespace": scope.namespace,
+ "query": query,
+ "query_types": sorted(query_types),
+ "total_records": len(records),
+ "selected_records": len(selected),
+ "context_token_estimate": _estimate_tokens(_json_dumps([item.to_dict() for item in selected])),
+ "persistence_backend": self.storage_backend,
+ "persistence_path": self.storage_path,
+ },
+ )
+ retrieval.relations = self.profile.build_relations(retrieval)
+ self._last_retrieval_tokens[scope.scope_id] = int(retrieval.retrieval_metadata.get("context_token_estimate", 0) or 0)
+ self.store.append_audit(
+ scope,
+ event_kind="retrieve",
+ payload={
+ "query": query,
+ "query_types": sorted(query_types),
+ "selected_record_ids": [item.record.record_id for item in selected],
+ },
+ )
+ return retrieval
+
+ def register_support(
+ self,
+ *,
+ answer_id: str,
+ supports: Sequence[Any],
+ session_scope: MemorySessionScope | str | None = None,
+ answer_text: str = "",
+ metadata: Dict[str, Any] | None = None,
+ ) -> None:
+ scope = self._resolve_scope(session_scope)
+ record_ids: List[str] = []
+ for item in supports:
+ if isinstance(item, MemoryRecord):
+ record_ids.append(item.record_id)
+ elif isinstance(item, MemoryRetrievalEntry):
+ record_ids.append(item.record.record_id)
+ elif isinstance(item, dict):
+ record_id = _clean_text(item.get("record_id", "")) or _clean_text(item.get("memory_id", ""))
+ if record_id:
+ record_ids.append(record_id)
+ else:
+ record_id = _clean_text(item)
+ if record_id:
+ record_ids.append(record_id)
+ payload = {
+ "answer_id": _clean_text(answer_id),
+ "record_ids": _dedupe(record_ids),
+ "answer_text": _clean_text(answer_text),
+ "metadata": dict(metadata or {}),
+ }
+ self.store.append_support(scope, answer_id=answer_id, record_ids=record_ids, payload=payload)
+ self.store.append_audit(scope, event_kind="support", payload=payload)
+
+ def writeback(self, request: MemoryWritebackRequest, session_scope: MemorySessionScope | str | None = None) -> List[MemoryRecord]:
+ payload = request.to_dict()
+ payload["metadata"] = {
+ **dict(request.metadata),
+ "writeback_type": request.writeback_type,
+ "supports": [dict(item) for item in request.supports],
+ "query_text": request.query_text,
+ "answer_text": request.answer_text,
+ }
+ return self.ingest_event(request.writeback_type, payload, session_scope=session_scope)
+
+ def reset_scope(self, session_scope: MemorySessionScope | str | None = None) -> None:
+ scope = self._resolve_scope(session_scope)
+ self.store.reset_scope(scope)
+ self._last_retrieval_tokens.pop(scope.scope_id, None)
+
+ def stats(self, session_scope: MemorySessionScope | str | None = None) -> Dict[str, Any]:
+ scope = self._resolve_scope(session_scope)
+ stats = self.store.scope_stats(scope)
+ return {
+ "profile_name": self.profile.profile_name,
+ "scope_id": scope.scope_id,
+ "namespace": scope.namespace,
+ "persistence_backend": self.storage_backend,
+ "persistence_path": self.storage_path,
+ "storage_bytes": int(self.store.storage_bytes(scope)),
+ "context_token_estimate": int(self._last_retrieval_tokens.get(scope.scope_id, 0)),
+ "total_state_token_estimate": int(stats.get("state_token_estimate", 0)),
+ **stats,
+ }
+
+ def before_llm(self, query: str, session_scope: MemorySessionScope | str | None = None, *, top_k: int = 6) -> Dict[str, Any]:
+ scope = self._resolve_scope(session_scope)
+ retrieval = self.retrieve(query, session_scope=scope, top_k=top_k)
+ return {
+ "query": query,
+ "scope": scope.to_dict(),
+ "retrieval": retrieval.to_dict(),
+ "context": self.profile.render_context(query, retrieval, scope),
+ }
+
+ def after_user_turn(self, turn_payload: Dict[str, Any] | None, session_scope: MemorySessionScope | str | None = None) -> List[MemoryRecord]:
+ return self.ingest_event("user_turn", turn_payload or {}, session_scope=session_scope)
+
+ def after_llm_answer(
+ self,
+ answer_payload: Dict[str, Any] | None,
+ supports: Sequence[Any] | None = None,
+ session_scope: MemorySessionScope | str | None = None,
+ ) -> Dict[str, Any]:
+ payload = dict(answer_payload or {})
+ answer_id = _clean_text(payload.get("answer_id", "")) or f"answer:{uuid.uuid4().hex}"
+ support_items = list(supports or payload.get("supports", []) or [])
+ if support_items:
+ self.register_support(
+ answer_id=answer_id,
+ supports=support_items,
+ session_scope=session_scope,
+ answer_text=_clean_text(payload.get("answer_text", "")),
+ metadata=dict(payload.get("metadata") or {}),
+ )
+ raw_records = payload.get("writeback_records")
+ if raw_records is None:
+ raw_records = payload.get("records")
+ if isinstance(raw_records, dict):
+ raw_records = [raw_records]
+ stored: List[MemoryRecord] = []
+ if raw_records:
+ request = MemoryWritebackRequest(
+ writeback_type=_clean_text(payload.get("writeback_type", "")) or "derived_writeback",
+ namespace=_clean_text(payload.get("namespace", "")) or self._resolve_scope(session_scope).namespace,
+ query_text=_clean_text(payload.get("query_text", "")),
+ answer_text=_clean_text(payload.get("answer_text", "")),
+ records=[dict(item) for item in list(raw_records or []) if isinstance(item, dict)],
+ supports=[dict(item) for item in list(payload.get("supports", []) or []) if isinstance(item, dict)],
+ metadata=dict(payload.get("metadata") or {}),
+ )
+ stored = self.writeback(request, session_scope=session_scope)
+ return {
+ "answer_id": answer_id,
+ "stored_records": [record.to_dict() for record in stored],
+ "support_count": len(support_items),
+ }
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/injection_planner.py b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/injection_planner.py
new file mode 100644
index 0000000000000000000000000000000000000000..81574ae084cb3772190691b822fc99d488898040
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/injection_planner.py
@@ -0,0 +1,679 @@
+from __future__ import annotations
+
+import hashlib
+import math
+import re
+from dataclasses import dataclass, asdict
+from typing import Any, Dict, Iterable, List, Mapping, Sequence
+
+try:
+ import torch
+ from torch import nn
+ import torch.nn.functional as F
+ from torch.utils.data import Dataset
+except ModuleNotFoundError: # Allows corpus tooling to run on machines without torch.
+ torch = None
+ nn = None
+ F = None
+ Dataset = object
+
+
+TEMPORAL_STATES = ("current", "stable", "historical", "temporary", "superseded", "irrelevant")
+LOGIC_ROLES = ("preference", "constraint", "resource", "goal", "negative", "evidence", "noise")
+EVIDENCE_ROLES = (
+ "direct_answer",
+ "initial_value",
+ "current_value",
+ "updated_value",
+ "supporting_context",
+ "bridge_context",
+ "negative_evidence",
+ "noise",
+)
+INJECTION_MODES = ("none", "direct_fact", "profile_bridge", "temporal_state", "logic_summary")
+MEMORY_LAYERS = ("event", "profile", "resource", "temporal", "path_tunnel", "topic_tunnel")
+PLANNER_ARCHITECTURE = "hashed_candidate_injection_planner_v2"
+PLANNER_ARCHITECTURE_SEMANTIC_V3 = "hashed_candidate_injection_planner_v3_semantic_scalar"
+
+_TOKEN_RE = re.compile(r"[\w]+", re.UNICODE)
+_ISO_TIMESTAMP_RE = re.compile(r"\[\d{4}-\d{2}-\d{2}T[^\]]+\]")
+_SESSION_SOURCE_RE = re.compile(r"\b[\w.-]+\s+session_id\s*=", re.IGNORECASE)
+_SESSION_ID_RE = re.compile(r"session_id\s*=\s*[\w:./-]+", re.IGNORECASE)
+_SESSION_DATE_RE = re.compile(
+ r"date\s*=\s*\d{4}/\d{2}/\d{2}(?:\s*\([A-Za-z]{3,9}\))?(?:\s+\d{1,2}:\d{2})?",
+ re.IGNORECASE,
+)
+_TURN_ROLE_RE = re.compile(
+ r"\[[^\]]*?\bturn\s*=\s*(\d+)\s+role\s*=\s*([A-Za-z_]+)[^\]]*\]",
+ re.IGNORECASE,
+)
+
+
+def _require_torch() -> None:
+ if torch is None or nn is None or F is None:
+ raise RuntimeError("torch is required for InjectionPlannerDataset and InjectionPlannerModel")
+
+
+def clean_text(value: Any) -> str:
+ text = "" if value is None else str(value)
+ return " ".join(text.strip().split())
+
+
+def tokens(text: str) -> List[str]:
+ return [item.lower() for item in _TOKEN_RE.findall(clean_text(text))]
+
+
+def canonicalize_runtime_text(value: Any) -> str:
+ text = clean_text(value)
+ if not text:
+ return ""
+ text = _ISO_TIMESTAMP_RE.sub("[timestamp]", text)
+ text = _SESSION_SOURCE_RE.sub("MemoryRuntime session_id=", text)
+ text = _SESSION_ID_RE.sub("session_id=session", text)
+ text = _SESSION_DATE_RE.sub("date=DATE", text)
+ text = _TURN_ROLE_RE.sub(lambda match: f"[turn={match.group(1)} role={match.group(2).lower()}]", text)
+ return clean_text(text)
+
+
+def stable_hash(value: str, *, modulo: int) -> int:
+ if modulo <= 0:
+ raise ValueError("modulo must be positive")
+ digest = hashlib.blake2b(clean_text(value).encode("utf-8"), digest_size=8).digest()
+ return int.from_bytes(digest, "big", signed=False) % modulo
+
+
+def normalize_label(value: Any, *, choices: Sequence[str], default: str) -> str:
+ text = clean_text(value).lower().replace("-", "_").replace(" ", "_")
+ allowed = set(choices)
+ return text if text in allowed else default
+
+
+def _float_feature(value: Any, *, default: float = 0.0, lo: float = 0.0, hi: float = 1.0) -> float:
+ try:
+ parsed = float(value)
+ except (TypeError, ValueError):
+ parsed = default
+ if math.isnan(parsed) or math.isinf(parsed):
+ parsed = default
+ return max(lo, min(hi, parsed))
+
+
+def vectorize_text(text: str, *, dim: int) -> List[float]:
+ if dim <= 0:
+ raise ValueError("dim must be positive")
+ values = [0.0] * dim
+ for token in tokens(text):
+ index = stable_hash(token, modulo=dim)
+ sign = -1.0 if stable_hash("sign:" + token, modulo=2) == 0 else 1.0
+ values[index] += sign
+ norm = math.sqrt(sum(value * value for value in values))
+ if norm > 0.0:
+ values = [value / norm for value in values]
+ return values
+
+
+@dataclass(frozen=True)
+class InjectionPlannerConfig:
+ query_hash_dim: int = 256
+ candidate_hash_dim: int = 256
+ scalar_dim: int = 22
+ hidden_dim: int = 192
+ dropout: float = 0.10
+
+ @property
+ def input_dim(self) -> int:
+ return int(self.query_hash_dim) + int(self.candidate_hash_dim) + int(self.scalar_dim)
+
+ def to_dict(self) -> Dict[str, Any]:
+ return asdict(self)
+
+ @classmethod
+ def from_dict(cls, payload: Mapping[str, Any] | None) -> "InjectionPlannerConfig":
+ if not payload:
+ return cls()
+ values = dict(payload)
+ legacy_text_buckets = values.get("text_buckets")
+ if legacy_text_buckets is not None:
+ values.setdefault("query_hash_dim", legacy_text_buckets)
+ values.setdefault("candidate_hash_dim", legacy_text_buckets)
+ allowed = {field.name for field in cls.__dataclass_fields__.values()} # type: ignore[attr-defined]
+ return cls(**{key: value for key, value in values.items() if key in allowed})
+
+
+def _as_list(value: Any) -> List[Any]:
+ if value is None:
+ return []
+ if isinstance(value, (list, tuple, set)):
+ return list(value)
+ return [value]
+
+
+def _first_nonempty(mapping: Mapping[str, Any], keys: Sequence[str]) -> Any:
+ for key in keys:
+ value = mapping.get(key)
+ if clean_text(value):
+ return value
+ return ""
+
+
+def normalize_candidate_row(candidate: Mapping[str, Any]) -> Dict[str, Any]:
+ candidate_id = _first_nonempty(candidate, ("id", "candidate_id", "memory_id", "event_id"))
+ score = candidate.get("retrieval_score", candidate.get("score", 0.0))
+ path_score = candidate.get("tunnel_score", candidate.get("path_score", 0.0))
+ temporal_state = candidate.get("temporal_state", candidate.get("time_state", "irrelevant"))
+ layer = candidate.get("layer", candidate.get("memory_layer", "event"))
+ normalized = dict(candidate)
+ normalized.update(
+ {
+ "id": clean_text(candidate_id),
+ "text": _first_nonempty(candidate, ("text", "value", "summary", "content")),
+ "summary": clean_text(candidate.get("summary", "")),
+ "topic": clean_text(candidate.get("topic", candidate.get("profile_domain", ""))),
+ "profile_key": clean_text(candidate.get("profile_key", candidate.get("profile_domain", ""))),
+ "resource_key": clean_text(candidate.get("resource_key", "")),
+ "layer": normalize_label(layer, choices=MEMORY_LAYERS, default="event"),
+ "temporal_state": normalize_label(temporal_state, choices=TEMPORAL_STATES, default="irrelevant"),
+ "logic_roles": [
+ normalize_label(item, choices=LOGIC_ROLES, default="noise")
+ for item in _as_list(candidate.get("logic_roles", candidate.get("roles", [])))
+ ]
+ or ["noise"],
+ "evidence_role": normalize_label(
+ candidate.get("evidence_role", candidate.get("evidence_role_hint", "")),
+ choices=EVIDENCE_ROLES,
+ default="",
+ ),
+ "retrieval_score": score,
+ "graph_score": candidate.get("graph_score", score),
+ "tunnel_score": path_score,
+ "topic_similarity": candidate.get("topic_similarity", candidate.get("topic_score", 0.0)),
+ "confidence": candidate.get("confidence", score),
+ "rank_score": candidate.get("rank_score", candidate.get("rank_prior", 0.0)),
+ "branch_depth": candidate.get("branch_depth", candidate.get("chain_depth", 0)),
+ "contradicts_current": bool(
+ candidate.get("contradicts_current")
+ or candidate.get("superseded")
+ or normalize_label(temporal_state, choices=TEMPORAL_STATES, default="irrelevant") == "superseded"
+ ),
+ "is_current": bool(candidate.get("is_current"))
+ or normalize_label(temporal_state, choices=TEMPORAL_STATES, default="irrelevant") in {"current", "stable"},
+ }
+ )
+ return normalized
+
+
+def normalize_gold_row(gold: Mapping[str, Any] | None) -> Dict[str, Any]:
+ source = dict(gold or {})
+ selected = source.get("selected_candidate_ids", source.get("selected_memory_ids", []))
+ temporal = source.get("temporal_state_by_candidate_id", source.get("temporal_labels", {}))
+ logic = source.get("logic_roles_by_candidate_id", source.get("logic_labels", {}))
+ evidence_roles = source.get("evidence_role_by_candidate_id", source.get("evidence_roles", {}))
+ protected = source.get("protected_candidate_ids", source.get("protected_memory_ids", []))
+ suppressed = source.get("suppressed_candidate_ids", source.get("suppressed_memory_ids", []))
+ normalized = dict(source)
+ normalized.update(
+ {
+ "selected_candidate_ids": [clean_text(item) for item in _as_list(selected) if clean_text(item)],
+ "temporal_state_by_candidate_id": dict(temporal or {}),
+ "logic_roles_by_candidate_id": dict(logic or {}),
+ "evidence_role_by_candidate_id": dict(evidence_roles or {}),
+ "protected_candidate_ids": [clean_text(item) for item in _as_list(protected) if clean_text(item)],
+ "suppressed_candidate_ids": [clean_text(item) for item in _as_list(suppressed) if clean_text(item)],
+ "expand_needed": bool(source.get("expand_needed", False)),
+ "expand_scope": clean_text(source.get("expand_scope", "")),
+ "profile_domain": clean_text(source.get("profile_domain", "")),
+ "support_chain": list(source.get("support_chain", []) or []),
+ }
+ )
+ return normalized
+
+
+def normalize_training_row(row: Mapping[str, Any]) -> Dict[str, Any]:
+ candidates = row.get("candidates", row.get("retrieved_candidates", row.get("candidate_events", [])))
+ normalized_candidates = [
+ normalize_candidate_row(candidate)
+ for candidate in list(candidates or [])
+ if isinstance(candidate, Mapping)
+ ]
+ return {
+ **dict(row),
+ "id": clean_text(_first_nonempty(row, ("id", "sample_id", "row_id"))) or f"row_{stable_hash(str(row), modulo=10**12)}",
+ "query": clean_text(_first_nonempty(row, ("query", "current_turn", "user_message", "question"))),
+ "candidates": normalized_candidates,
+ "gold": normalize_gold_row(dict(row.get("gold") or {})),
+ }
+
+
+def candidate_text(candidate: Mapping[str, Any]) -> str:
+ pieces = [
+ candidate.get("text"),
+ candidate.get("summary"),
+ candidate.get("topic"),
+ candidate.get("profile_key"),
+ candidate.get("resource_key"),
+ ]
+ return canonicalize_runtime_text(" ".join(clean_text(piece) for piece in pieces if clean_text(piece)))
+
+
+def candidate_scalar_features(candidate: Mapping[str, Any], config: InjectionPlannerConfig | None = None) -> List[float]:
+ resolved_config = config or InjectionPlannerConfig()
+ layer = normalize_label(candidate.get("layer"), choices=MEMORY_LAYERS, default="")
+ temporal_state = normalize_label(candidate.get("temporal_state"), choices=TEMPORAL_STATES, default="")
+ overlap = _float_feature(candidate.get("query_overlap"), hi=1.0)
+ retrieval_score = _float_feature(candidate.get("retrieval_score"), hi=1.0)
+ graph_score = _float_feature(candidate.get("graph_score"), hi=1.0)
+ tunnel_score = _float_feature(candidate.get("tunnel_score"), hi=1.0)
+ topic_similarity = _float_feature(candidate.get("topic_similarity"), hi=1.0)
+ confidence = _float_feature(candidate.get("confidence"), default=0.5, hi=1.0)
+ rank_score = _float_feature(candidate.get("rank_score", candidate.get("rank_prior")), default=0.0, hi=1.0)
+ semantic_similarity = _float_feature(
+ candidate.get(
+ "semantic_similarity",
+ candidate.get(
+ "answer_window_semantic_similarity",
+ candidate.get("embedder_similarity", candidate.get("dense_similarity", 0.0)),
+ ),
+ ),
+ default=0.0,
+ hi=1.0,
+ )
+ age_turns = _float_feature(candidate.get("age_turns"), default=0.0, lo=0.0, hi=500.0) / 500.0
+ branch_depth = _float_feature(candidate.get("branch_depth"), default=0.0, lo=0.0, hi=20.0) / 20.0
+ contradiction = 1.0 if bool(candidate.get("contradicts_current")) else 0.0
+ explicit_current = 1.0 if bool(candidate.get("is_current")) else 0.0
+ has_profile = 1.0 if clean_text(candidate.get("profile_key")) else 0.0
+ has_resource = 1.0 if clean_text(candidate.get("resource_key")) else 0.0
+ layer_flags = [1.0 if layer == item else 0.0 for item in MEMORY_LAYERS]
+ temporal_flags = [1.0 if temporal_state == item else 0.0 for item in ("current", "stable", "superseded")]
+ features = [
+ overlap,
+ retrieval_score,
+ graph_score,
+ tunnel_score,
+ topic_similarity,
+ confidence,
+ age_turns,
+ branch_depth,
+ contradiction,
+ explicit_current,
+ has_profile,
+ has_resource,
+ *layer_flags,
+ *temporal_flags,
+ ]
+ if int(resolved_config.scalar_dim) >= 22:
+ features = [
+ overlap,
+ retrieval_score,
+ graph_score,
+ tunnel_score,
+ topic_similarity,
+ confidence,
+ rank_score,
+ age_turns,
+ branch_depth,
+ contradiction,
+ explicit_current,
+ has_profile,
+ has_resource,
+ *layer_flags,
+ *temporal_flags,
+ ]
+ if int(resolved_config.scalar_dim) >= 23:
+ features.append(semantic_similarity)
+ if len(features) != int(resolved_config.scalar_dim):
+ raise AssertionError(f"scalar feature dim mismatch: {len(features)} != {resolved_config.scalar_dim}")
+ return features
+
+
+def _candidate_targets(candidate: Mapping[str, Any], gold: Mapping[str, Any]) -> Dict[str, Any]:
+ selected_ids = {clean_text(item) for item in gold.get("selected_candidate_ids", [])}
+ candidate_id = clean_text(candidate.get("id"))
+ temporal_by_id = dict(gold.get("temporal_state_by_candidate_id") or {})
+ logic_by_id = dict(gold.get("logic_roles_by_candidate_id") or {})
+ evidence_role_by_id = dict(gold.get("evidence_role_by_candidate_id") or {})
+ temporal = temporal_by_id.get(candidate_id, candidate.get("temporal_state", "irrelevant"))
+ logic_values = logic_by_id.get(candidate_id, candidate.get("logic_roles", []))
+ if isinstance(logic_values, str):
+ logic_values = [logic_values]
+ temporal_state = normalize_label(temporal, choices=TEMPORAL_STATES, default="irrelevant")
+ logic_roles = [
+ normalize_label(item, choices=LOGIC_ROLES, default="noise")
+ for item in list(logic_values or [])
+ ] or ["noise"]
+ selected = candidate_id in selected_ids
+ explicit_evidence_role = normalize_label(
+ evidence_role_by_id.get(candidate_id, candidate.get("evidence_role", "")),
+ choices=EVIDENCE_ROLES,
+ default="",
+ )
+ if explicit_evidence_role:
+ evidence_role = explicit_evidence_role
+ elif not selected:
+ evidence_role = "negative_evidence" if ("negative" in logic_roles or temporal_state == "superseded") else "noise"
+ elif "negative" in logic_roles or temporal_state == "superseded":
+ evidence_role = "negative_evidence"
+ elif normalize_label(candidate.get("layer"), choices=MEMORY_LAYERS, default="event") in {"path_tunnel", "topic_tunnel"}:
+ evidence_role = "bridge_context"
+ elif temporal_state in {"current", "stable"}:
+ evidence_role = "current_value"
+ elif temporal_state == "historical":
+ evidence_role = "initial_value"
+ elif temporal_state == "temporary":
+ evidence_role = "updated_value"
+ else:
+ evidence_role = "direct_answer"
+ return {
+ "selected": 1.0 if selected else 0.0,
+ "temporal_state": temporal_state,
+ "logic_roles": logic_roles,
+ "evidence_role": evidence_role,
+ }
+
+
+def flatten_training_rows(rows: Iterable[Mapping[str, Any]], config: InjectionPlannerConfig | None = None) -> List[Dict[str, Any]]:
+ resolved_config = config or InjectionPlannerConfig()
+ output: List[Dict[str, Any]] = []
+ for raw_row in rows:
+ row = normalize_training_row(raw_row)
+ query = clean_text(row.get("query") or row.get("user_message"))
+ gold = dict(row.get("gold") or {})
+ candidates = list(row.get("candidates") or [])
+ query_vec = vectorize_text(query, dim=resolved_config.query_hash_dim)
+ candidate_items = []
+ for candidate in candidates:
+ if not isinstance(candidate, Mapping):
+ continue
+ text = candidate_text(candidate)
+ targets = _candidate_targets(candidate, gold)
+ logic_target = [0.0] * len(LOGIC_ROLES)
+ for role in targets["logic_roles"]:
+ logic_target[LOGIC_ROLES.index(role)] = 1.0
+ candidate_items.append(
+ {
+ "id": clean_text(candidate.get("id")),
+ "text": text,
+ "features": query_vec
+ + vectorize_text(text, dim=resolved_config.candidate_hash_dim)
+ + candidate_scalar_features(candidate, resolved_config),
+ "selected_target": targets["selected"],
+ "temporal_target": TEMPORAL_STATES.index(targets["temporal_state"]),
+ "logic_target": logic_target,
+ "evidence_role_target": EVIDENCE_ROLES.index(targets["evidence_role"]),
+ "raw": dict(candidate),
+ }
+ )
+ injection_mode = normalize_label(gold.get("injection_mode"), choices=INJECTION_MODES, default="none")
+ output.append(
+ {
+ "id": clean_text(row.get("id")),
+ "query": query,
+ "candidates": candidate_items,
+ "should_inject_target": 1.0 if bool(gold.get("should_inject")) else 0.0,
+ "injection_mode_target": INJECTION_MODES.index(injection_mode),
+ "raw": dict(row),
+ }
+ )
+ return output
+
+
+class InjectionPlannerDataset(Dataset): # type: ignore[misc]
+ def __init__(self, rows: Sequence[Mapping[str, Any]], config: InjectionPlannerConfig | None = None):
+ _require_torch()
+ self.config = config or InjectionPlannerConfig()
+ self.examples = flatten_training_rows(rows, self.config)
+
+ def __len__(self) -> int:
+ return len(self.examples)
+
+ def __getitem__(self, index: int) -> Dict[str, Any]:
+ example = self.examples[index]
+ candidates = example["candidates"]
+ features = [item["features"] for item in candidates]
+ if not features:
+ features = [[0.0] * self.config.input_dim]
+ candidates = [
+ {
+ "id": "__empty__",
+ "selected_target": 0.0,
+ "temporal_target": TEMPORAL_STATES.index("irrelevant"),
+ "logic_target": [0.0] * len(LOGIC_ROLES),
+ "evidence_role_target": EVIDENCE_ROLES.index("noise"),
+ "raw": {},
+ }
+ ]
+ return {
+ "id": example["id"],
+ "features": torch.tensor(features, dtype=torch.float32),
+ "valid_mask": torch.tensor([item["id"] != "__empty__" for item in candidates], dtype=torch.bool),
+ "selected_target": torch.tensor([item["selected_target"] for item in candidates], dtype=torch.float32),
+ "temporal_target": torch.tensor([item["temporal_target"] for item in candidates], dtype=torch.long),
+ "logic_target": torch.tensor([item["logic_target"] for item in candidates], dtype=torch.float32),
+ "evidence_role_target": torch.tensor([item["evidence_role_target"] for item in candidates], dtype=torch.long),
+ "should_inject_target": torch.tensor(float(example["should_inject_target"]), dtype=torch.float32),
+ "injection_mode_target": torch.tensor(int(example["injection_mode_target"]), dtype=torch.long),
+ }
+
+
+def collate_injection_batch(batch: Sequence[Mapping[str, Any]]) -> Dict[str, Any]:
+ _require_torch()
+ max_candidates = max(int(item["features"].shape[0]) for item in batch)
+ input_dim = int(batch[0]["features"].shape[-1])
+ batch_size = len(batch)
+ features = torch.zeros((batch_size, max_candidates, input_dim), dtype=torch.float32)
+ valid_mask = torch.zeros((batch_size, max_candidates), dtype=torch.bool)
+ selected_target = torch.zeros((batch_size, max_candidates), dtype=torch.float32)
+ temporal_target = torch.full(
+ (batch_size, max_candidates),
+ TEMPORAL_STATES.index("irrelevant"),
+ dtype=torch.long,
+ )
+ logic_target = torch.zeros((batch_size, max_candidates, len(LOGIC_ROLES)), dtype=torch.float32)
+ evidence_role_target = torch.full(
+ (batch_size, max_candidates),
+ EVIDENCE_ROLES.index("noise"),
+ dtype=torch.long,
+ )
+ for row_index, item in enumerate(batch):
+ count = int(item["features"].shape[0])
+ features[row_index, :count] = item["features"]
+ valid_mask[row_index, :count] = item["valid_mask"]
+ selected_target[row_index, :count] = item["selected_target"]
+ temporal_target[row_index, :count] = item["temporal_target"]
+ logic_target[row_index, :count] = item["logic_target"]
+ evidence_role_target[row_index, :count] = item["evidence_role_target"]
+ return {
+ "ids": [str(item["id"]) for item in batch],
+ "features": features,
+ "valid_mask": valid_mask,
+ "selected_target": selected_target,
+ "temporal_target": temporal_target,
+ "logic_target": logic_target,
+ "evidence_role_target": evidence_role_target,
+ "should_inject_target": torch.stack([item["should_inject_target"] for item in batch]),
+ "injection_mode_target": torch.stack([item["injection_mode_target"] for item in batch]),
+ }
+
+
+class InjectionPlannerModel(nn.Module if nn is not None else object): # type: ignore[misc]
+ def __init__(self, config: InjectionPlannerConfig | Mapping[str, Any] | None = None):
+ _require_torch()
+ super().__init__()
+ self.config = config if isinstance(config, InjectionPlannerConfig) else InjectionPlannerConfig.from_dict(config)
+ hidden = int(self.config.hidden_dim)
+ self.candidate_encoder = nn.Sequential(
+ nn.Linear(self.config.input_dim, hidden),
+ nn.LayerNorm(hidden),
+ nn.SiLU(),
+ nn.Dropout(float(self.config.dropout)),
+ nn.Linear(hidden, hidden),
+ nn.LayerNorm(hidden),
+ nn.SiLU(),
+ )
+ self.selection_head = nn.Linear(hidden, 1)
+ self.temporal_head = nn.Linear(hidden, len(TEMPORAL_STATES))
+ self.logic_head = nn.Linear(hidden, len(LOGIC_ROLES))
+ self.evidence_role_head = nn.Linear(hidden, len(EVIDENCE_ROLES))
+ self.row_encoder = nn.Sequential(
+ nn.Linear(hidden * 2, hidden),
+ nn.SiLU(),
+ nn.Dropout(float(self.config.dropout)),
+ )
+ self.should_inject_head = nn.Linear(hidden, 1)
+ self.injection_mode_head = nn.Linear(hidden, len(INJECTION_MODES))
+
+ def forward(self, features: Any, valid_mask: Any | None = None) -> Dict[str, Any]:
+ encoded = self.candidate_encoder(features)
+ if valid_mask is None:
+ valid_mask = torch.ones(encoded.shape[:2], dtype=torch.bool, device=encoded.device)
+ mask_float = valid_mask.to(dtype=encoded.dtype).unsqueeze(-1)
+ masked_encoded = encoded * mask_float
+ count = mask_float.sum(dim=1).clamp_min(1.0)
+ mean_pool = masked_encoded.sum(dim=1) / count
+ max_source = encoded.masked_fill(~valid_mask.unsqueeze(-1), -1e4)
+ max_pool = torch.max(max_source, dim=1).values
+ max_pool = torch.where(torch.isfinite(max_pool), max_pool, torch.zeros_like(max_pool))
+ row_hidden = self.row_encoder(torch.cat([mean_pool, max_pool], dim=-1))
+ return {
+ "candidate_hidden": encoded,
+ "selection_logits": self.selection_head(encoded).squeeze(-1),
+ "temporal_logits": self.temporal_head(encoded),
+ "logic_logits": self.logic_head(encoded),
+ "evidence_role_logits": self.evidence_role_head(encoded),
+ "should_inject_logits": self.should_inject_head(row_hidden).squeeze(-1),
+ "injection_mode_logits": self.injection_mode_head(row_hidden),
+ }
+
+
+def loss_for_batch(outputs: Mapping[str, Any], batch: Mapping[str, Any], *, weights: Mapping[str, Any] | None = None) -> Dict[str, Any]:
+ _require_torch()
+ raw_weights = dict(weights or {})
+ logic_pos_weight = raw_weights.pop("logic_pos_weight", None)
+ temporal_class_weight = raw_weights.pop("temporal_class_weight", None)
+ mode_class_weight = raw_weights.pop("mode_class_weight", None)
+ resolved_weights = {
+ "selection": 1.0,
+ "temporal": 0.5,
+ "logic": 0.5,
+ "evidence_role": 0.8,
+ "should_inject": 0.75,
+ "mode": 0.5,
+ **raw_weights,
+ }
+ valid_mask = batch["valid_mask"]
+ valid_count = int(valid_mask.sum().item())
+ if valid_count > 0:
+ selection_loss = F.binary_cross_entropy_with_logits(
+ outputs["selection_logits"][valid_mask],
+ batch["selected_target"][valid_mask],
+ )
+ temporal_loss = F.cross_entropy(
+ outputs["temporal_logits"][valid_mask],
+ batch["temporal_target"][valid_mask],
+ weight=temporal_class_weight.to(outputs["temporal_logits"].device) if hasattr(temporal_class_weight, "to") else temporal_class_weight,
+ )
+ logic_loss = F.binary_cross_entropy_with_logits(
+ outputs["logic_logits"][valid_mask],
+ batch["logic_target"][valid_mask],
+ pos_weight=logic_pos_weight.to(outputs["logic_logits"].device) if hasattr(logic_pos_weight, "to") else logic_pos_weight,
+ )
+ evidence_role_loss = F.cross_entropy(
+ outputs["evidence_role_logits"][valid_mask],
+ batch["evidence_role_target"][valid_mask],
+ )
+ else:
+ zero = outputs["should_inject_logits"].sum() * 0.0
+ selection_loss = zero
+ temporal_loss = zero
+ logic_loss = zero
+ evidence_role_loss = zero
+ should_inject_loss = F.binary_cross_entropy_with_logits(
+ outputs["should_inject_logits"],
+ batch["should_inject_target"],
+ )
+ mode_loss = F.cross_entropy(
+ outputs["injection_mode_logits"],
+ batch["injection_mode_target"],
+ weight=mode_class_weight.to(outputs["injection_mode_logits"].device) if hasattr(mode_class_weight, "to") else mode_class_weight,
+ )
+ total = (
+ resolved_weights["selection"] * selection_loss
+ + resolved_weights["temporal"] * temporal_loss
+ + resolved_weights["logic"] * logic_loss
+ + resolved_weights["evidence_role"] * evidence_role_loss
+ + resolved_weights["should_inject"] * should_inject_loss
+ + resolved_weights["mode"] * mode_loss
+ )
+ return {
+ "loss": total,
+ "selection_loss": selection_loss.detach(),
+ "temporal_loss": temporal_loss.detach(),
+ "logic_loss": logic_loss.detach(),
+ "evidence_role_loss": evidence_role_loss.detach(),
+ "should_inject_loss": should_inject_loss.detach(),
+ "mode_loss": mode_loss.detach(),
+ }
+
+
+def summarize_candidate_plan(
+ row: Mapping[str, Any],
+ *,
+ selected_candidate_ids: Sequence[str] | None = None,
+ temporal_state_by_candidate_id: Mapping[str, str] | None = None,
+ logic_roles_by_candidate_id: Mapping[str, Sequence[str]] | None = None,
+ injection_mode: str | None = None,
+) -> Dict[str, Any]:
+ gold = dict(row.get("gold") or {})
+ selected_ids = {
+ clean_text(item)
+ for item in (
+ selected_candidate_ids
+ if selected_candidate_ids is not None
+ else gold.get("selected_candidate_ids", [])
+ )
+ }
+ temporal_by_id = dict(temporal_state_by_candidate_id or gold.get("temporal_state_by_candidate_id") or {})
+ logic_by_id = dict(logic_roles_by_candidate_id or gold.get("logic_roles_by_candidate_id") or {})
+ evidence_role_by_id = dict(gold.get("evidence_role_by_candidate_id") or {})
+ selected_candidates = []
+ for candidate in list(row.get("candidates") or []):
+ candidate_id = clean_text(candidate.get("id"))
+ if candidate_id not in selected_ids:
+ continue
+ selected_candidates.append(
+ {
+ "id": candidate_id,
+ "text": candidate_text(candidate),
+ "layer": clean_text(candidate.get("layer")),
+ "temporal_state": normalize_label(
+ temporal_by_id.get(candidate_id, candidate.get("temporal_state")),
+ choices=TEMPORAL_STATES,
+ default="irrelevant",
+ ),
+ "logic_roles": [
+ normalize_label(item, choices=LOGIC_ROLES, default="noise")
+ for item in logic_by_id.get(candidate_id, candidate.get("logic_roles", []))
+ ],
+ "evidence_role": normalize_label(
+ evidence_role_by_id.get(candidate_id, candidate.get("evidence_role", "")),
+ choices=EVIDENCE_ROLES,
+ default="direct_answer",
+ ),
+ }
+ )
+ mode = normalize_label(
+ injection_mode if injection_mode is not None else gold.get("injection_mode"),
+ choices=INJECTION_MODES,
+ default="none",
+ )
+ text_lines = []
+ for item in selected_candidates:
+ roles = ",".join(role for role in item["logic_roles"] if role != "noise") or "evidence"
+ text_lines.append(f"[{item['temporal_state']}|{item['evidence_role']}|{roles}] {item['text']}")
+ return {
+ "architecture": PLANNER_ARCHITECTURE,
+ "should_inject": bool(selected_candidates) and mode != "none",
+ "injection_mode": mode,
+ "selected_candidates": selected_candidates,
+ "inject_text": "\n".join(text_lines),
+ }
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/memory_graph.py b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/memory_graph.py
new file mode 100644
index 0000000000000000000000000000000000000000..5d27d5506941bec68c1f615741a01867bab52ed9
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/memory_graph.py
@@ -0,0 +1,3823 @@
+from __future__ import annotations
+
+from collections import defaultdict
+from contextlib import contextmanager
+from dataclasses import dataclass, field
+import json
+import math
+import os
+from pathlib import Path
+import re
+import sqlite3
+from typing import Any, Callable, Dict, Iterable, List, Mapping, Sequence
+import urllib.error
+import urllib.request
+
+from .memory_profiles import TMCRAProfile
+from .profile_layer import (
+ PROFILE_AGGREGATE_CATEGORY,
+ PROFILE_AGGREGATE_SOURCE_KIND,
+ PROFILE_CLUSTER_CATEGORY,
+ PROFILE_CLUSTER_SOURCE_KIND,
+ build_profile_aggregate_metadata,
+ build_profile_cluster_metadata,
+ is_profile_layer_record,
+ profile_aggregate_slot_key,
+ profile_cluster_similarity,
+ profile_cluster_slot_key,
+ profile_edge_score,
+ profile_query_score_delta,
+)
+
+
+_HISTORY_MARKERS = (
+ "previous",
+ "earlier",
+ "before",
+ "used to",
+ "history",
+ "historical",
+ "old",
+ "prior",
+ "earliest",
+ "之前",
+ "以前",
+ "历史",
+ "上一版",
+ "旧",
+)
+
+_GOAL_MARKERS = ("goal", "mission", "objective", "target", "primary goal", "目标")
+_CONSTRAINT_MARKERS = ("constraint", "must", "forbid", "policy", "限制", "约束", "必须")
+_PREFERENCE_MARKERS = ("preference", "prefer", "default", "mode", "偏好", "默认")
+_TERM_MARKERS = ("term", "terminology", "alias", "mean", "definition", "术语", "定义")
+_STAGE_MARKERS = ("stage", "phase", "status", "state", "阶段", "状态")
+_PATH_MARKERS = ("path", "route", "connect", "why", "tied", "链路", "路径", "连接")
+_NOISE_MARKERS = ("noise turn", "noise note", "routine chatter", "random artifact", "distractor")
+_NEGATION_MARKERS = ("not", "never", "no ", "should not", "不要", "不应", "不是", "无")
+
+
+_PUBLIC_MONTH_MARKERS = (
+ "january",
+ "february",
+ "march",
+ "april",
+ "may",
+ "june",
+ "july",
+ "august",
+ "september",
+ "october",
+ "november",
+ "december",
+)
+_PUBLIC_QUERY_FRAME_TOKENS = {
+ "what",
+ "when",
+ "which",
+ "where",
+ "who",
+ "why",
+ "how",
+ "did",
+ "does",
+ "do",
+ "is",
+ "are",
+ "am",
+ "was",
+ "were",
+ "has",
+ "have",
+ "had",
+ "will",
+ "would",
+ "can",
+ "could",
+ "should",
+ "may",
+ "might",
+}
+_PUBLIC_TEMPORAL_REFERENCE_RE = re.compile(
+ r"\b(?:when|date|day|month|year|today|tomorrow|yesterday|tonight|morning|afternoon|evening|night|week(?:end)?|last|next|before|after)\b",
+ re.IGNORECASE,
+)
+_PUBLIC_PROFILE_QUERY_RE = re.compile(
+ r"^\s*(?:who\s+is|what\s+(?:is|are|was|were)|(?:what|where)\s+(?:does|do|is|are|was|were|has|have|had)\s+[A-Z][a-z]+(?:\s+[A-Z][a-z]+)?\b|[A-Z][a-z]+(?:\s+[A-Z][a-z]+)?'s\b)",
+ re.IGNORECASE,
+)
+_PUBLIC_PLANNED_STATUS_RE = re.compile(
+ r"\b(?:will|going\s+to|plan(?:ning)?\s+to|scheduled\s+to|hoping\s+to|preparing\s+to)\b",
+ re.IGNORECASE,
+)
+_PUBLIC_CURRENT_STATUS_RE = re.compile(r"\b(?:currently|now|still)\b|\b(?:is|are|am)\s+\w+ing\b", re.IGNORECASE)
+_PUBLIC_PAST_STATUS_RE = re.compile(r"\b(?:did|was|were|had|has)\b", re.IGNORECASE)
+_PUBLIC_MEMORY_SUBJECT_QUERY_PATTERNS = (
+ re.compile(
+ r"\bmy\s+(?P[a-z0-9][a-z0-9\s'/_-]{1,100}?)\s+(?:is|are|was|were)\s+(?:now|currently)\b",
+ re.IGNORECASE,
+ ),
+ re.compile(
+ r"\bwhat\s+(?:is|are|was|were)\s+my\s+(?P[a-z0-9][a-z0-9\s'/_-]{1,100}?)\s+(?:right\s+now|now|currently)\b",
+ re.IGNORECASE,
+ ),
+ re.compile(
+ r"\bmy\s+(?P[a-z0-9][a-z0-9\s'/_-]{1,100}?)\s+(?:right\s+now|now|currently)\b",
+ re.IGNORECASE,
+ ),
+)
+
+
+def _clean_text(value: Any) -> str:
+ return re.sub(r"\s+", " ", str(value or "").strip())
+
+
+_DEFAULT_TMCRA_PROFILE = TMCRAProfile()
+
+
+# Override legacy mojibake markers with clean Chinese variants.
+_HISTORY_MARKERS = (
+ "previous",
+ "earlier",
+ "before",
+ "used to",
+ "history",
+ "historical",
+ "old",
+ "prior",
+ "earliest",
+ "之前",
+ "以前",
+ "历史",
+ "上一版",
+ "旧",
+ "原来",
+)
+_GOAL_MARKERS = ("goal", "mission", "objective", "target", "primary goal", "目标")
+_CONSTRAINT_MARKERS = ("constraint", "must", "forbid", "policy", "限制", "约束", "必须", "不要", "不能")
+_PREFERENCE_MARKERS = ("preference", "prefer", "default", "mode", "偏好", "默认", "优先")
+_TERM_MARKERS = ("term", "terminology", "alias", "mean", "definition", "术语", "定义", "别名", "叫做", "指的是")
+_STAGE_MARKERS = ("stage", "phase", "status", "state", "阶段", "状态", "进度")
+_PATH_MARKERS = ("path", "route", "connect", "why", "tied", "链路", "路径", "连接")
+_NEGATION_MARKERS = ("not", "never", "no ", "should not", "不要", "不应", "不是", "不能")
+
+
+_INACTIVE_HISTORY_MARKERS = (
+ "noise",
+ "just noise",
+ "noise note",
+ "inactive",
+ "stay inactive",
+ "should stay inactive",
+ "keep inactive",
+ "remain inactive",
+ "should be ignored",
+ "ignore this",
+ "ignore that",
+ "\u566a\u58f0",
+ "\u4e0d\u6fc0\u6d3b",
+ "\u4fdd\u6301\u4e0d\u6fc0\u6d3b",
+ "\u4e0d\u5e94\u751f\u6548",
+ "\u5ffd\u7565",
+)
+
+def _normalize(value: Any) -> str:
+ return _clean_text(value).lower()
+
+
+def _tokenize(value: Any) -> List[str]:
+ text = _normalize(value)
+ if not text:
+ return []
+ english = re.findall(r"[a-z0-9_]+", text)
+ if english:
+ return english
+ return [char for char in text if char.strip()]
+
+
+def _tokenize(value: Any) -> List[str]:
+ text = _normalize(value)
+ if not text:
+ return []
+ english = re.findall(r"[a-z0-9_]+", text)
+ cjk = [char for char in text if "\u4e00" <= char <= "\u9fff"]
+ if english or cjk:
+ return _dedupe([*english, *cjk])
+ return [char for char in text if char.strip()]
+
+
+def _dedupe(items: Iterable[Any], *, max_items: int | None = None) -> List[str]:
+ values: List[str] = []
+ seen = set()
+ for item in items:
+ text = _clean_text(item)
+ if not text:
+ continue
+ key = _normalize(text)
+ if key in seen:
+ continue
+ seen.add(key)
+ values.append(text)
+ if max_items is not None and len(values) >= max_items:
+ break
+ return values
+
+
+def _contains_any(text: str, markers: Sequence[str]) -> bool:
+ lowered = _normalize(text)
+ return any(marker in lowered for marker in markers)
+
+
+def _public_query_requests_temporal_fact(query: str) -> bool:
+ lowered = _normalize(query)
+ first_token = _tokenize(query)[:1]
+ if first_token and first_token[0] == "when":
+ return True
+ if _PUBLIC_TEMPORAL_REFERENCE_RE.search(lowered):
+ return True
+ if any(marker in lowered for marker in _PUBLIC_MONTH_MARKERS):
+ return True
+ if re.search(r"\b(?:mon|tue|wed|thu|fri|sat|sun)(?:day)?\b", lowered):
+ return True
+ return bool(re.search(r"\b\d{4}\b", lowered))
+
+
+def _public_event_tokens(tokens: Iterable[str], *, speaker_candidates: Iterable[str] = ()) -> set[str]:
+ normalized_speakers = {_normalize(item) for item in speaker_candidates if _clean_text(item)}
+ content_tokens: set[str] = set()
+ skipped_frame_tokens = 0
+ for token in tokens:
+ normalized = _normalize(token)
+ if not normalized or normalized.isdigit() or len(normalized) <= 2 or normalized in normalized_speakers:
+ continue
+ if skipped_frame_tokens < 2 and normalized in _PUBLIC_QUERY_FRAME_TOKENS:
+ skipped_frame_tokens += 1
+ continue
+ content_tokens.add(normalized)
+ return content_tokens
+
+
+def _public_query_requests_profile_fact(query: str, speaker_candidates: Iterable[str]) -> bool:
+ return bool(list(speaker_candidates or [])) and bool(_PUBLIC_PROFILE_QUERY_RE.search(_clean_text(query)))
+
+
+def _public_query_semantic_slot(query: str) -> str:
+ speaker_candidates = _public_query_speakers(query)
+ if _public_query_requests_temporal_fact(query):
+ return "event_time"
+ if _public_query_requests_profile_fact(query, speaker_candidates):
+ return "profile"
+ return "event"
+
+
+def _public_query_target_status(query: str) -> str:
+ clean_query = _clean_text(query)
+ lowered = _normalize(query)
+ if _PUBLIC_PLANNED_STATUS_RE.search(clean_query):
+ return "planned"
+ if _PUBLIC_CURRENT_STATUS_RE.search(clean_query):
+ return "current"
+ if _PUBLIC_PAST_STATUS_RE.search(lowered):
+ return "past"
+ return ""
+
+
+def _public_query_time_granularity(query: str) -> str:
+ lowered = _normalize(query)
+ if "month" in lowered or any(marker in lowered for marker in _PUBLIC_MONTH_MARKERS):
+ return "month"
+ if "year" in lowered:
+ return "year"
+ if any(marker in lowered for marker in ("date", "day", "when")):
+ return "day_or_coarse"
+ return ""
+
+
+def _public_query_speakers(query: str) -> set[str]:
+ candidates = {
+ _normalize(match.group(1).rstrip("'s"))
+ for match in re.finditer(r"\b([A-Z][a-z]+(?:'s)?)\b", _clean_text(query))
+ }
+ return {
+ item
+ for item in candidates
+ if item not in {"what", "when", "which", "who", "where", "why", "how"}
+ }
+
+
+def _public_subject_signature(value: Any) -> str:
+ return re.sub(r"[^a-z0-9]+", "_", _normalize(value)).strip("_")
+
+
+def _public_query_subject(query: str) -> str:
+ clean_query = _clean_text(query)
+ for pattern in _PUBLIC_MEMORY_SUBJECT_QUERY_PATTERNS:
+ match = pattern.search(clean_query)
+ if not match:
+ continue
+ subject = _clean_text(match.group("subject")).strip(" .,:;!?\"'")
+ subject = re.sub(r"\b(?:right|now|currently|current)\b$", "", subject, flags=re.IGNORECASE).strip(" .,:;!?\"'")
+ if len(_tokenize(subject)) >= 2:
+ return subject
+ return ""
+
+
+_DEPTH_LAYERS = {
+ "core_view",
+ "profile",
+ "deep_view",
+ "mechanism",
+ "risk",
+ "metric",
+ "product_view",
+ "evidence",
+ "question",
+}
+_DEPTH_LAYER_ORDER = {
+ "core_view": 0,
+ "profile": 0,
+ "deep_view": 1,
+ "risk": 2,
+ "mechanism": 3,
+ "metric": 4,
+ "product_view": 5,
+ "evidence": 6,
+ "question": 7,
+}
+
+
+def _record_subject_signature_from_parts(slot_key: Any, metadata: Mapping[str, Any]) -> str:
+ explicit = _clean_text(metadata.get("subject_signature", ""))
+ if explicit:
+ return _public_subject_signature(explicit)
+ subject = _clean_text(metadata.get("subject", ""))
+ if subject:
+ return _public_subject_signature(subject)
+ canonical_slot_key = _clean_text(metadata.get("canonical_slot_key", "") or slot_key)
+ if ".subject." in canonical_slot_key:
+ return _public_subject_signature(canonical_slot_key.split(".subject.", 1)[-1])
+ if ".subject." in _clean_text(slot_key):
+ return _public_subject_signature(_clean_text(slot_key).split(".subject.", 1)[-1])
+ return _public_subject_signature(_slot_group_key(_clean_text(slot_key)))
+
+
+def _infer_depth_layer(*, category: Any, relation: Any, slot_key: Any, value: Any, metadata: Mapping[str, Any]) -> str:
+ explicit = _normalize(metadata.get("depth_layer", "") or metadata.get("memory_depth_layer", ""))
+ if explicit in _DEPTH_LAYERS:
+ return explicit
+ semantic_slot = _normalize(metadata.get("semantic_slot", "") or metadata.get("profile_type", ""))
+ category_text = _normalize(category)
+ relation_text = _normalize(relation)
+ slot_text = _normalize(slot_key)
+ if is_profile_layer_record(category=category_text, semantic_slot=semantic_slot, metadata=metadata):
+ return "profile"
+ combined = _normalize(
+ " ".join(
+ [
+ str(value or ""),
+ str(metadata.get("source_span", "")),
+ str(metadata.get("raw_text", "")),
+ str(metadata.get("source_turn_text", "")),
+ str(semantic_slot),
+ str(category_text),
+ str(relation_text),
+ str(slot_text),
+ ]
+ )
+ )
+ if category_text == "question" or "question" in relation_text or semantic_slot in {"goal", "information_needed"}:
+ return "question"
+ if any(marker in combined for marker in ("deeper", "deep view", "deep safety", "accountable memory", "mature", "refined")):
+ if any(marker in combined for marker in ("dashboard", "product", "enterprise adoption", "adoption view")):
+ return "product_view"
+ return "deep_view"
+ if any(marker in combined for marker in ("metric", "measure", "score", "benchmark", "evaluation_metric", "recovery rate", "success rate")):
+ return "metric"
+ if any(marker in combined for marker in ("dashboard", "product", "enterprise adoption", "adoption view", "deployment cost")):
+ return "product_view"
+ if any(marker in combined for marker in ("risk", "failure", "outdated", "unverified", "corrupt", "silent failure", "safety violation")):
+ return "risk"
+ if any(marker in combined for marker in ("mechanism", "checksum", "retrieval", "filter", "pipeline", "interface", "state transition")):
+ return "mechanism"
+ if re.search(r"\bmy\b.{0,80}\bview\s+is\s+now\b", combined) or re.search(r"\bmy\s+view\s+is\s+now\b", combined):
+ return "core_view"
+ if any(marker in combined for marker in ("mechanism", "checksum", "logging", "audit trail", "metadata", "retrieval", "filter", "pipeline", "interface", "tool call", "write", "state transition")):
+ return "mechanism"
+ if category_text in {"fact", "profile", "evidence"} or "source_turn" in relation_text:
+ return "evidence"
+ return "core_view"
+
+
+def _depth_edge_type(source_layer: str, target_layer: str, source_text: str, target_text: str) -> str:
+ combined = _normalize(f"{source_text} {target_text}")
+ if any(marker in combined for marker in ("instead", "no longer", "not anymore", "replace", "rather than")):
+ return "contrasts"
+ if source_layer == "deep_view" and target_layer in {"core_view", "evidence", "mechanism", "product_view"}:
+ return "refines"
+ if source_layer == "mechanism" and target_layer in {"core_view", "deep_view", "risk", "metric"}:
+ return "operationalizes"
+ if source_layer == "metric":
+ return "measures"
+ if source_layer == "product_view":
+ return "productizes"
+ if source_layer == "risk":
+ return "qualifies"
+ if source_layer == target_layer:
+ return "reframes"
+ return "co_mentions"
+
+
+def _public_signature_tokens(value: Any) -> set[str]:
+ return _public_event_tokens(_tokenize(_clean_text(value)))
+
+
+def _public_slot_root(slot_key: str) -> str:
+ clean = _clean_text(slot_key)
+ match = re.match(r"^(.*?\.turn_\d+)", clean)
+ return match.group(1) if match else clean
+
+
+def _public_query_analysis(query: str, query_tokens: set[str]) -> Dict[str, Any]:
+ speaker_candidates = _public_query_speakers(query)
+ subject_text = _public_query_subject(query)
+ subject_tokens = set(_tokenize(subject_text))
+ event_query_tokens = _public_event_tokens(
+ [token for token in query_tokens if token not in subject_tokens],
+ speaker_candidates=speaker_candidates,
+ )
+ return {
+ "is_temporal": _public_query_requests_temporal_fact(query),
+ "semantic_slot_target": _public_query_semantic_slot(query),
+ "target_status": _public_query_target_status(query),
+ "time_granularity_target": _public_query_time_granularity(query),
+ "speaker_candidates": speaker_candidates,
+ "event_tokens": event_query_tokens,
+ "query_subject": subject_text,
+ "query_subject_signature": _public_subject_signature(subject_text),
+ "query_subject_tokens": subject_tokens,
+ }
+
+
+def _public_semantic_slot_match(target: str, value: str) -> bool:
+ normalized_target = _normalize(target)
+ normalized_value = _normalize(value)
+ if not normalized_target or not normalized_value:
+ return False
+ if normalized_target == "profile":
+ return normalized_value in {"profile", "identity", "research_topic", "education", "occupation"}
+ return normalized_target == normalized_value
+
+
+def _public_time_granularity_compatible(target: str, record_value: str) -> bool:
+ normalized_target = _normalize(target)
+ normalized_value = _normalize(record_value)
+ if not normalized_target:
+ return True
+ if normalized_target == "day_or_coarse":
+ return normalized_value in {"day", "relative_day_reference", "month", "year"}
+ return normalized_target == normalized_value
+
+
+def _clamp01(value: Any) -> float:
+ try:
+ numeric = float(value)
+ except (TypeError, ValueError):
+ return 0.0
+ return max(0.0, min(1.0, numeric))
+
+
+def _weighted_token_coverage(
+ expected_tokens: Iterable[str],
+ available_tokens: Iterable[str],
+ *,
+ min_token_length: int = 1,
+) -> float:
+ expected = [
+ _normalize(token)
+ for token in list(expected_tokens or [])
+ if _normalize(token) and len(_normalize(token)) >= max(1, int(min_token_length))
+ ]
+ if not expected:
+ return 0.0
+ available = {
+ _normalize(token)
+ for token in list(available_tokens or [])
+ if _normalize(token)
+ }
+ if not available:
+ return 0.0
+ total_weight = sum(max(1, len(token)) for token in expected)
+ matched_weight = sum(max(1, len(token)) for token in expected if token in available)
+ return matched_weight / max(1, total_weight)
+
+
+def _public_match_profile(
+ record: "SessionMemoryRecordV2",
+ analysis: Dict[str, Any] | None,
+ *,
+ query_text: str,
+ query_tokens: set[str],
+ token_set: set[str],
+) -> Dict[str, Any]:
+ metadata = dict(record.metadata or {})
+ source_kind = _normalize(record.source_kind)
+ speaker = _normalize(metadata.get("speaker", ""))
+ resolved_date = _normalize(metadata.get("resolved_date", ""))
+ temporal_query = bool((analysis or {}).get("is_temporal", _public_query_requests_temporal_fact(query_text)))
+ speaker_candidates = {
+ _normalize(item)
+ for item in set((analysis or {}).get("speaker_candidates", set()) or set())
+ if _clean_text(item)
+ }
+ event_query_tokens = set((analysis or {}).get("event_tokens", set()) or set())
+ event_focus_tokens = {token for token in event_query_tokens if len(_normalize(token)) >= 4}
+ query_subject_signature = _normalize((analysis or {}).get("query_subject_signature", "")).replace("-", "_")
+ query_subject_tokens = {
+ _normalize(token)
+ for token in set((analysis or {}).get("query_subject_tokens", set()) or set())
+ if _normalize(token)
+ }
+ record_subject_text = _clean_text(metadata.get("subject", ""))
+ record_subject_signature = _normalize(metadata.get("subject_signature", "")).replace("-", "_")
+ canonical_slot_key = _normalize(metadata.get("canonical_slot_key", "") or record.slot_key)
+ slot_subject_signature = _public_subject_signature(canonical_slot_key.split(".subject.", 1)[-1]) if ".subject." in canonical_slot_key else ""
+ record_subject_signatures = {
+ item
+ for item in {
+ record_subject_signature,
+ _public_subject_signature(record_subject_text),
+ slot_subject_signature,
+ }
+ if item
+ }
+ record_subject_tokens = set(
+ _tokenize(" ".join([record_subject_text, record_subject_signature.replace("_", " "), canonical_slot_key.replace("_", " ")]))
+ )
+ subject_exact_match = bool(query_subject_signature and query_subject_signature in record_subject_signatures)
+ subject_overlap = (
+ _weighted_token_coverage(query_subject_tokens, record_subject_tokens, min_token_length=3)
+ if query_subject_tokens
+ else 0.0
+ )
+ subject_match = bool(subject_exact_match or (query_subject_signature and subject_overlap >= 0.72))
+ record_signature_tokens = (
+ _public_signature_tokens(metadata.get("event_signature", ""))
+ or _public_event_tokens(token_set, speaker_candidates=speaker_candidates)
+ or _public_signature_tokens(record.value)
+ )
+ record_content_tokens = _public_event_tokens(token_set, speaker_candidates=speaker_candidates)
+ record_focus_tokens = record_signature_tokens | record_content_tokens
+ event_shared_tokens = event_query_tokens & record_signature_tokens
+ event_overlap = len(event_shared_tokens) / max(1, len(event_query_tokens)) if event_query_tokens else 0.0
+ event_focus_coverage = _weighted_token_coverage(event_focus_tokens, record_focus_tokens, min_token_length=4)
+ exact_speaker_match = bool(speaker and (speaker in speaker_candidates or speaker in query_text))
+ semantic_target = _normalize((analysis or {}).get("semantic_slot_target", ""))
+ semantic_slot = _normalize(metadata.get("semantic_slot", ""))
+ semantic_match = _public_semantic_slot_match(semantic_target, semantic_slot)
+ target_status_target = _normalize((analysis or {}).get("target_status", ""))
+ target_status = _normalize(metadata.get("target_status", ""))
+ target_status_match = bool(target_status_target and target_status and target_status_target == target_status)
+ time_target = _normalize((analysis or {}).get("time_granularity_target", ""))
+ time_granularity = _normalize(metadata.get("time_granularity", ""))
+ time_match = bool(time_granularity and _public_time_granularity_compatible(time_target, time_granularity))
+ shared_tokens = query_tokens & token_set if query_tokens and token_set else set()
+ shared_content_tokens = _public_event_tokens(shared_tokens, speaker_candidates=speaker_candidates)
+ shared_content_ratio = (
+ len(shared_content_tokens) / max(1, len(event_query_tokens))
+ if event_query_tokens
+ else len(shared_content_tokens) / max(1, len(query_tokens))
+ if query_tokens
+ else 0.0
+ )
+ event_signal = max(event_overlap, shared_content_ratio, event_focus_coverage)
+
+ support_weight = 0.0
+ support_score = 0.0
+ if speaker_candidates and speaker:
+ support_weight += 0.18
+ support_score += 0.18 * (1.0 if exact_speaker_match else 0.0)
+ if semantic_target:
+ support_weight += 0.16
+ support_score += 0.16 * (1.0 if semantic_match else 0.0)
+ if query_subject_signature:
+ support_weight += 0.42
+ support_score += 0.42 * (1.0 if subject_match else subject_overlap * 0.55)
+ if event_query_tokens:
+ support_weight += 0.34
+ support_score += 0.34 * event_signal
+ if target_status_target and target_status:
+ support_weight += 0.08
+ support_score += 0.08 * (1.0 if target_status_match else 0.0)
+ if temporal_query:
+ support_weight += 0.10
+ support_score += 0.10 * (1.0 if resolved_date else 0.0)
+ if time_target or time_granularity:
+ support_weight += 0.10
+ support_score += 0.10 * (1.0 if time_match else 0.0)
+ support_weight += 0.08
+ support_score += 0.08 * (
+ 1.0
+ if source_kind == "public_dialog_time"
+ else 0.55
+ if source_kind == "public_dialog_event"
+ else 0.0
+ )
+ elif semantic_target == "profile":
+ support_weight += 0.08
+ support_score += 0.08 * (1.0 if source_kind == "public_dialog_profile" else 0.0)
+ else:
+ support_weight += 0.06
+ support_score += 0.06 * (1.0 if source_kind == "public_dialog_event" else 0.0)
+ positive_signal = support_score / max(1e-6, support_weight) if support_weight > 0 else event_signal
+
+ conflict = 0.0
+ if speaker_candidates and speaker and not exact_speaker_match:
+ conflict += 0.14
+ if semantic_target and semantic_slot and not semantic_match:
+ conflict += 0.10 if semantic_target == "profile" else 0.05
+ if query_subject_signature and not subject_match:
+ conflict += 0.28 * max(0.0, 1.0 - subject_overlap)
+ if target_status_target and target_status and not target_status_match:
+ conflict += 0.08
+ if temporal_query and source_kind == "public_dialog_profile":
+ conflict += 0.12
+ elif temporal_query and source_kind == "public_dialog_turn":
+ conflict += 0.06
+ if event_query_tokens and source_kind in {"public_dialog_time", "public_dialog_event"}:
+ conflict += 0.22 * max(0.0, 1.0 - event_focus_coverage)
+ if temporal_query and source_kind == "public_dialog_time" and not resolved_date and not time_match:
+ conflict += 0.16
+ if shared_tokens and not shared_content_tokens and not exact_speaker_match and event_signal <= 0 and not semantic_match and not resolved_date:
+ conflict += 0.14
+ compatibility = _clamp01(positive_signal - conflict)
+ return {
+ "cluster_root": _public_slot_root(record.slot_key),
+ "signature": _normalize(metadata.get("event_signature", "")),
+ "source_kind": source_kind,
+ "temporal_query": temporal_query,
+ "speaker_match": exact_speaker_match,
+ "semantic_match": semantic_match,
+ "subject_match": subject_match,
+ "subject_overlap": subject_overlap,
+ "query_subject_signature": query_subject_signature,
+ "target_status_match": target_status_match,
+ "time_match": time_match,
+ "resolved_date": bool(resolved_date),
+ "event_overlap": event_overlap,
+ "event_focus_coverage": event_focus_coverage,
+ "shared_content_ratio": shared_content_ratio,
+ "event_signal": event_signal,
+ "positive_signal": positive_signal,
+ "conflict": conflict,
+ "compatibility": compatibility,
+ }
+
+
+def _public_match_reason(
+ record: "SessionMemoryRecordV2",
+ analysis: Dict[str, Any] | None,
+ *,
+ speaker_match: bool,
+ semantic_match: bool,
+ subject_match: bool,
+ event_overlap: float,
+ target_status_match: bool,
+ time_match: bool,
+) -> str:
+ reasons: List[str] = []
+ if speaker_match:
+ reasons.append("speaker")
+ if semantic_match:
+ reasons.append("semantic_slot")
+ if subject_match:
+ reasons.append("subject")
+ if event_overlap > 0:
+ reasons.append("event_signature")
+ if target_status_match:
+ reasons.append("target_status")
+ if time_match:
+ reasons.append("time_granularity")
+ if not reasons and analysis and _public_query_requests_temporal_fact(_clean_text(analysis.get("query", ""))):
+ reasons.append("temporal_backoff")
+ if not reasons and _clean_text(record.source_kind).startswith("public_dialog"):
+ reasons.append("public_lexical")
+ return ",".join(reasons[:3])
+
+
+def infer_category_hints(query: str) -> List[str]:
+ return _DEFAULT_TMCRA_PROFILE.infer_category_hints(query)
+
+
+def infer_history_mode(query: str) -> bool:
+ return _contains_any(query, _HISTORY_MARKERS)
+
+
+def infer_history_mode_clean(query: str) -> bool:
+ clean_markers = (
+ "\u4e4b\u524d",
+ "\u4ee5\u524d",
+ "\u5386\u53f2",
+ "\u8986\u76d6\u524d",
+ "\u4e0a\u4e00\u7248",
+ )
+ return infer_history_mode(query) or _contains_any(query, clean_markers) or _query_requests_inactive_history(query)
+
+
+def infer_noise(text: str) -> bool:
+ lowered = _normalize(text)
+ return any(marker in lowered for marker in _NOISE_MARKERS)
+
+
+def _query_requests_inactive_history(query: str) -> bool:
+ lowered = _normalize(query)
+ return any(marker in lowered for marker in _INACTIVE_HISTORY_MARKERS)
+
+
+def _extract_noise_value(text: str) -> str:
+ clean = _clean_text(text)
+ if ":" in clean:
+ prefix, suffix = clean.split(":", 1)
+ if _normalize(prefix) in {"noise note", "noise turn"} and _clean_text(suffix):
+ return _clean_text(suffix).rstrip(".")
+ return clean.rstrip(".")
+
+
+def _query_requests_current_pair(query: str) -> bool:
+ lowered = _normalize(query)
+ explicit_pair_markers = (
+ "historical",
+ "history",
+ "both",
+ "previous",
+ "old",
+ "prior",
+ "which one is active",
+ "current and previous",
+ "current vs previous",
+ "current versus previous",
+ "鍘嗗彶",
+ "之前",
+ "旧",
+ )
+ if any(marker in lowered for marker in explicit_pair_markers):
+ return True
+ current_markers = ("current", "active", "now", "right now", "当前", "现在")
+ history_markers = ("previous", "historical", "history", "old", "prior", "before", "之前", "历史", "旧")
+ return any(marker in lowered for marker in current_markers) and any(marker in lowered for marker in history_markers)
+
+
+def _query_requests_current_pair_clean(query: str) -> bool:
+ lowered = _normalize(query)
+ clean_markers = (
+ "\u5386\u53f2",
+ "\u4e4b\u524d",
+ "\u65e7",
+ "\u4e0a\u4e00\u7248",
+ )
+ return _query_requests_current_pair(query) or any(marker in lowered for marker in clean_markers)
+
+
+def _query_requests_current_pair(query: str) -> bool:
+ lowered = _normalize(query)
+ explicit_pair_markers = (
+ "historical",
+ "history",
+ "both",
+ "previous",
+ "old",
+ "prior",
+ "which one is active",
+ "current and previous",
+ "current vs previous",
+ "current versus previous",
+ "历史",
+ "之前",
+ "旧",
+ )
+ if any(marker in lowered for marker in explicit_pair_markers):
+ return True
+ current_markers = ("current", "active", "now", "right now", "当前", "现在")
+ history_markers = ("previous", "historical", "history", "old", "prior", "before", "之前", "历史", "旧")
+ return any(marker in lowered for marker in current_markers) and any(marker in lowered for marker in history_markers)
+
+
+def _query_requests_current_pair_clean(query: str) -> bool:
+ lowered = _normalize(query)
+ clean_markers = (
+ "历史",
+ "之前",
+ "旧",
+ "上一版",
+ )
+ return _query_requests_current_pair(query) or any(marker in lowered for marker in clean_markers)
+
+
+def _slot_slug(value: Any, *, fallback: str = "default", max_length: int = 24) -> str:
+ text = _normalize(value)
+ if not text:
+ return fallback
+ parts = re.findall(r"[a-z0-9]+|[\u4e00-\u9fff]", text)
+ if not parts:
+ return fallback
+ slug = ".".join(parts[: max(1, max_length // 2)])
+ slug = re.sub(r"\.+", ".", slug).strip(".")
+ if not slug:
+ return fallback
+ if len(slug) > max_length:
+ slug = slug[:max_length].rstrip(".")
+ return slug or fallback
+
+
+def stable_slot_key(
+ *,
+ category: str,
+ value: str,
+ anchors: Sequence[str],
+ slot_key: str = "",
+ relation: str = "",
+ metadata: Dict[str, Any] | None = None,
+) -> str:
+ return _DEFAULT_TMCRA_PROFILE.stable_slot_key(
+ category=category,
+ value=value,
+ anchors=anchors,
+ slot_key=slot_key,
+ relation=relation,
+ metadata=metadata,
+ )
+
+
+def guess_slot_key(*, category: str, value: str, anchors: Sequence[str]) -> str:
+ return _DEFAULT_TMCRA_PROFILE.stable_slot_key(category=category, value=value, anchors=anchors)
+
+
+def _record_tokens(category: str, slot_key: str, value: str, anchors: Sequence[str], relation: str) -> List[str]:
+ return _tokenize(" ".join([category, slot_key, value, relation, *anchors]))
+
+
+_ACTIVE_RECORD_STATES = {"active", "parallel_active"}
+_SUSPECT_RECORD_STATES = {"suspect"}
+_PASSIVE_RECORD_STATES = {"evidence", "historical", "inactive", "superseded", "promoted", "false"}
+_SINGLETON_MEMORY_CATEGORIES = {"stage_state"}
+_PARALLEL_MEMORY_CATEGORIES = {"goal", "constraint", "preference", "terminology", "path"}
+
+
+def _is_active_record_state(state: Any) -> bool:
+ return _normalize(state) in _ACTIVE_RECORD_STATES
+
+
+def _is_suspect_record_state(state: Any) -> bool:
+ return _normalize(state) in _SUSPECT_RECORD_STATES
+
+
+def _is_passive_record_state(state: Any) -> bool:
+ return _normalize(state) in _PASSIVE_RECORD_STATES
+
+
+def _slot_relation_for_state(state: Any) -> str:
+ normalized = _normalize(state)
+ if normalized == "parallel_active":
+ return "parallel_active_in_slot"
+ if normalized == "active":
+ return "active_in_slot"
+ return "historical_in_slot"
+
+
+def _metadata_text(metadata: Mapping[str, Any], *keys: str) -> str:
+ for key in keys:
+ value = _clean_text(metadata.get(key, ""))
+ if value:
+ return value
+ return ""
+
+
+def _slot_group_key(slot_key: str) -> str:
+ clean_slot = _clean_text(slot_key)
+ if not clean_slot:
+ return ""
+ parts = [part for part in clean_slot.split(".") if part]
+ if len(parts) <= 2:
+ return clean_slot
+ if parts[-1].isdigit():
+ return ".".join(parts[:-1])
+ return ".".join(parts[:-1])
+
+
+def _record_state_signature(record: "SessionMemoryRecordV2") -> str:
+ metadata = dict(record.metadata or {})
+ components = [
+ _normalize(record.category),
+ _normalize(record.relation),
+ _normalize(
+ _metadata_text(metadata, "parallel_group_key")
+ or _metadata_text(metadata, "canonical_slot_key")
+ or _slot_group_key(record.slot_key)
+ or record.slot_key
+ ),
+ _normalize(_metadata_text(metadata, "speaker")),
+ _normalize(_metadata_text(metadata, "semantic_slot", "profile_type")),
+ _normalize(_metadata_text(metadata, "target_status")),
+ _normalize(_metadata_text(metadata, "polarity")),
+ _normalize(_metadata_text(metadata, "time_granularity")),
+ _normalize(
+ _metadata_text(metadata, "subject_signature")
+ or _metadata_text(metadata, "event_id")
+ or _metadata_text(metadata, "dia_id")
+ or _metadata_text(metadata, "speaker")
+ or (record.anchor_concepts[0] if record.anchor_concepts else "")
+ ),
+ ]
+ return "|".join(component for component in components if component)
+
+
+def _record_memory_signature(record: "SessionMemoryRecordV2") -> str:
+ metadata = dict(record.metadata or {})
+ value_signature = _normalize(
+ _metadata_text(
+ metadata,
+ "resolved_time_value",
+ "resolved_date",
+ "profile_value",
+ "time_value",
+ "time_display_value",
+ "event_signature",
+ )
+ or record.value
+ )
+ components = [_record_state_signature(record), value_signature]
+ return "|".join(component for component in components if component)
+
+
+def _record_value_signature(record: "SessionMemoryRecordV2") -> str:
+ metadata = dict(record.metadata or {})
+ return _normalize(
+ _metadata_text(
+ metadata,
+ "resolved_time_value",
+ "resolved_date",
+ "profile_value",
+ "time_value",
+ "time_display_value",
+ "event_signature",
+ "target_status",
+ )
+ or record.value
+ )
+
+
+def _record_semantic_family(record: "SessionMemoryRecordV2") -> str:
+ metadata = dict(record.metadata or {})
+ return _normalize(
+ _metadata_text(metadata, "semantic_slot", "profile_type", "canonical_slot_key")
+ or record.relation
+ or record.category
+ )
+
+
+def _record_singleton_like(record: "SessionMemoryRecordV2") -> bool:
+ metadata = dict(record.metadata or {})
+ category = _normalize(record.category)
+ if category in _SINGLETON_MEMORY_CATEGORIES:
+ return True
+ semantic_slot = _normalize(_metadata_text(metadata, "semantic_slot", "profile_type"))
+ if semantic_slot in {"identity", "research_topic", "education", "occupation"}:
+ return True
+ canonical_slot_key = _normalize(_metadata_text(metadata, "canonical_slot_key"))
+ if canonical_slot_key.startswith("stage.") or canonical_slot_key.endswith(".current"):
+ return True
+ relation = _normalize(record.relation)
+ return relation in {"event_date", "profile_of", "status_of"}
+
+
+def _records_version_conflict(existing: "SessionMemoryRecordV2", incoming: "SessionMemoryRecordV2") -> bool:
+ existing_metadata = dict(existing.metadata or {})
+ incoming_metadata = dict(incoming.metadata or {})
+ if _normalize(_metadata_text(existing_metadata, "state_signature")) != _normalize(_metadata_text(incoming_metadata, "state_signature")):
+ return False
+ existing_value_signature = _record_value_signature(existing)
+ incoming_value_signature = _record_value_signature(incoming)
+ existing_singleton = _record_singleton_like(existing)
+ incoming_singleton = _record_singleton_like(incoming)
+ if existing_singleton or incoming_singleton:
+ if existing_value_signature and incoming_value_signature:
+ return existing_value_signature != incoming_value_signature
+ return _normalize(existing.value) != _normalize(incoming.value)
+ if _record_allows_parallel(existing) and _record_allows_parallel(incoming):
+ return False
+ existing_time = _normalize(_metadata_text(existing_metadata, "resolved_time_value", "resolved_date", "time_value", "time_display_value"))
+ incoming_time = _normalize(_metadata_text(incoming_metadata, "resolved_time_value", "resolved_date", "time_value", "time_display_value"))
+ if existing_time and incoming_time and existing_time != incoming_time:
+ return True
+ existing_profile = _normalize(_metadata_text(existing_metadata, "profile_value"))
+ incoming_profile = _normalize(_metadata_text(incoming_metadata, "profile_value"))
+ if existing_profile and incoming_profile and existing_profile != incoming_profile:
+ return True
+ existing_status = _normalize(_metadata_text(existing_metadata, "target_status"))
+ incoming_status = _normalize(_metadata_text(incoming_metadata, "target_status"))
+ if existing_status and incoming_status and existing_status != incoming_status:
+ return True
+ existing_family = _record_semantic_family(existing)
+ incoming_family = _record_semantic_family(incoming)
+ if existing_family and incoming_family and existing_family == incoming_family and existing_value_signature and incoming_value_signature:
+ if existing_value_signature != incoming_value_signature:
+ return True
+ return False
+
+
+def _classify_conflict_action(
+ incoming: "SessionMemoryRecordV2",
+ *,
+ active_records: Sequence["SessionMemoryRecordV2"],
+ active_by_state_signature: Sequence["SessionMemoryRecordV2"],
+) -> tuple[str, List["SessionMemoryRecordV2"], str]:
+ if not active_records:
+ return "insert", [], "empty_slot"
+ supersede_targets = [candidate for candidate in active_by_state_signature if _records_version_conflict(candidate, incoming)]
+ if supersede_targets:
+ return "supersede", supersede_targets, "same_state_revision"
+ if not _record_allows_parallel(incoming):
+ return "supersede", list(active_records), "slot_disallows_parallel"
+ return "parallel_active", [], "parallel_fact"
+
+
+def _record_allows_parallel(record: "SessionMemoryRecordV2") -> bool:
+ metadata = dict(record.metadata or {})
+ if "allow_parallel_state" in metadata:
+ return bool(metadata.get("allow_parallel_state"))
+ category = _normalize(record.category)
+ if category in _SINGLETON_MEMORY_CATEGORIES:
+ return False
+ if category in _PARALLEL_MEMORY_CATEGORIES:
+ return True
+ semantic_slot = _normalize(_metadata_text(metadata, "semantic_slot"))
+ if semantic_slot in {"identity", "research_topic", "education", "occupation"}:
+ return False
+ canonical_slot_key = _normalize(_metadata_text(metadata, "canonical_slot_key"))
+ if canonical_slot_key.startswith("stage.") or canonical_slot_key.endswith(".current"):
+ return False
+ relation = _normalize(record.relation)
+ if relation in {"event_date", "profile_of", "status_of"}:
+ return False
+ return bool(_metadata_text(metadata, "parallel_group_key")) or category not in {"profile", "status"}
+
+
+def _prepare_record_signatures(record: "SessionMemoryRecordV2") -> None:
+ metadata = dict(record.metadata or {})
+ subject_signature = _record_subject_signature_from_parts(record.slot_key, metadata)
+ if subject_signature:
+ metadata["subject_signature"] = subject_signature
+ metadata["depth_layer"] = _infer_depth_layer(
+ category=record.category,
+ relation=record.relation,
+ slot_key=record.slot_key,
+ value=record.value,
+ metadata=metadata,
+ )
+ metadata["depth_node_enabled"] = True
+ metadata["parallel_group_key"] = _clean_text(
+ metadata.get("parallel_group_key")
+ or metadata.get("canonical_slot_key")
+ or _slot_group_key(record.slot_key)
+ or record.slot_key
+ )
+ record.metadata = metadata
+ record.metadata["state_signature"] = _record_state_signature(record)
+ record.metadata["memory_signature"] = _record_memory_signature(record)
+ record.metadata["value_signature"] = _record_value_signature(record)
+
+
+def _merge_record_payload(existing: "SessionMemoryRecordV2", incoming: "SessionMemoryRecordV2") -> None:
+ previous_metadata = dict(existing.metadata or {})
+ incoming_metadata = dict(incoming.metadata or {})
+ existing.turn_index = max(existing.turn_index, incoming.turn_index)
+ existing.salience = max(existing.salience, incoming.salience)
+ existing.confidence = max(existing.confidence, incoming.confidence)
+ existing.anchor_concepts = _dedupe([*existing.anchor_concepts, *incoming.anchor_concepts])
+ existing.evidence_anchors = _dedupe([*existing.evidence_anchors, *incoming.evidence_anchors])
+ merged_metadata = dict(previous_metadata)
+ merged_metadata.update(incoming_metadata)
+ merged_metadata["origin_answer_ids"] = _dedupe(
+ [
+ *list(previous_metadata.get("origin_answer_ids", []) or []),
+ _clean_text(previous_metadata.get("origin_answer_id", "")),
+ *list(incoming_metadata.get("origin_answer_ids", []) or []),
+ _clean_text(incoming_metadata.get("origin_answer_id", "")),
+ ]
+ )
+ merged_metadata["support_memory_ids"] = _dedupe(
+ [*list(previous_metadata.get("support_memory_ids", []) or []), *list(incoming_metadata.get("support_memory_ids", []) or [])]
+ )
+ merged_metadata["support_fact_refs"] = _dedupe(
+ [*list(previous_metadata.get("support_fact_refs", []) or []), *list(incoming_metadata.get("support_fact_refs", []) or [])]
+ )
+ merged_metadata["support_path_refs"] = _dedupe(
+ [*list(previous_metadata.get("support_path_refs", []) or []), *list(incoming_metadata.get("support_path_refs", []) or [])]
+ )
+ existing.metadata = merged_metadata
+ _prepare_record_signatures(existing)
+
+
+def _record_suspect_support_turns(record: "SessionMemoryRecordV2") -> List[int]:
+ metadata = dict(record.metadata or {})
+ turns = []
+ for item in list(metadata.get("suspect_support_turns", []) or []):
+ try:
+ turn = int(item)
+ except Exception:
+ continue
+ if turn > 0 and turn not in turns:
+ turns.append(turn)
+ if int(record.turn_index or 0) > 0 and int(record.turn_index) not in turns:
+ turns.append(int(record.turn_index))
+ return sorted(turns)
+
+
+def _estimate_tokens(text: Any) -> int:
+ clean = _clean_text(text)
+ if not clean:
+ return 0
+ return max(1, math.ceil(len(clean) / 4))
+
+
+@dataclass(slots=True)
+class SessionMemoryRecordV2:
+ memory_id: str
+ category: str
+ slot_key: str
+ value: str
+ relation: str
+ anchor_concepts: List[str] = field(default_factory=list)
+ evidence_anchors: List[str] = field(default_factory=list)
+ salience: float = 0.7
+ confidence: float = 0.7
+ source_kind: str = "session_memory"
+ turn_index: int = 0
+ state: str = "active"
+ supersedes: List[str] = field(default_factory=list)
+ metadata: Dict[str, Any] = field(default_factory=dict)
+
+ def token_set(self) -> set[str]:
+ return set(_record_tokens(self.category, self.slot_key, self.value, self.anchor_concepts, self.relation))
+
+ def to_dict(self) -> Dict[str, Any]:
+ return {
+ "memory_id": self.memory_id,
+ "category": self.category,
+ "slot_key": self.slot_key,
+ "value": self.value,
+ "relation": self.relation,
+ "anchor_concepts": list(self.anchor_concepts),
+ "evidence_anchors": list(self.evidence_anchors),
+ "salience": round(float(self.salience), 6),
+ "confidence": round(float(self.confidence), 6),
+ "source_kind": self.source_kind,
+ "turn_index": int(self.turn_index),
+ "state": self.state,
+ "supersedes": list(self.supersedes),
+ "metadata": dict(self.metadata),
+ }
+
+
+@dataclass(slots=True)
+class SessionMemoryEdgeV2:
+ edge_id: str
+ source_memory_id: str
+ target_memory_id: str
+ edge_type: str
+ score: float = 0.0
+ model_score: float = 0.0
+ evidence_turn: int = 0
+ evidence: str = ""
+ metadata: Dict[str, Any] = field(default_factory=dict)
+
+ def to_dict(self) -> Dict[str, Any]:
+ return {
+ "edge_id": self.edge_id,
+ "source_memory_id": self.source_memory_id,
+ "target_memory_id": self.target_memory_id,
+ "edge_type": self.edge_type,
+ "score": round(float(self.score), 6),
+ "model_score": round(float(self.model_score), 6),
+ "evidence_turn": int(self.evidence_turn),
+ "evidence": self.evidence,
+ "metadata": dict(self.metadata),
+ }
+
+
+@dataclass(slots=True)
+class MemoryScalingRecord:
+ turn_count: int
+ records: int
+ active_slots: int
+ superseded_records: int
+ graph_nodes: int
+ graph_edges: int
+ storage_bytes: int
+ context_token_estimate: int
+ ingest_seconds_total: float
+ ingest_us_per_turn: float
+ retrieval_ms_p50: float
+ retrieval_ms_p95: float
+ retrieval_ms_p99: float
+ python_rss_bytes: int
+ python_peak_bytes: int
+ cpu_percent: float
+ disk_bytes_written: int
+ exploded: bool = False
+ guard_reason: str = ""
+
+ def to_dict(self) -> Dict[str, Any]:
+ return {
+ "turn_count": self.turn_count,
+ "records": self.records,
+ "active_slots": self.active_slots,
+ "superseded_records": self.superseded_records,
+ "graph_nodes": self.graph_nodes,
+ "graph_edges": self.graph_edges,
+ "storage_bytes": self.storage_bytes,
+ "context_token_estimate": self.context_token_estimate,
+ "ingest_seconds_total": round(float(self.ingest_seconds_total), 6),
+ "ingest_us_per_turn": round(float(self.ingest_us_per_turn), 6),
+ "retrieval_ms_p50": round(float(self.retrieval_ms_p50), 6),
+ "retrieval_ms_p95": round(float(self.retrieval_ms_p95), 6),
+ "retrieval_ms_p99": round(float(self.retrieval_ms_p99), 6),
+ "python_rss_bytes": self.python_rss_bytes,
+ "python_peak_bytes": self.python_peak_bytes,
+ "cpu_percent": round(float(self.cpu_percent), 6),
+ "disk_bytes_written": self.disk_bytes_written,
+ "exploded": self.exploded,
+ "guard_reason": self.guard_reason,
+ }
+
+
+_AUDIT_LOG_FIELDS = ("turn_log", "retrieval_log", "answer_support_log")
+
+
+class SessionMemoryGraphV2:
+ def __init__(
+ self,
+ *,
+ audit_retention: int = 256,
+ persistence_backend: str = "memory",
+ persistence_path: str = "",
+ ) -> None:
+ self.turn_index = 0
+ self.records_by_id: Dict[str, SessionMemoryRecordV2] = {}
+ self.slot_heads: Dict[str, str] = {}
+ self.slot_history: Dict[str, List[str]] = defaultdict(list)
+ self.memory_edges: Dict[str, SessionMemoryEdgeV2] = {}
+ self.subject_depth_heads: Dict[str, Dict[str, str]] = defaultdict(dict)
+ self.turn_log: List[Dict[str, Any]] = []
+ self.retrieval_log: List[Dict[str, Any]] = []
+ self.answer_support_log: List[Dict[str, Any]] = []
+ self.noise_turn_count = 0
+ self.audit_retention = max(1, int(audit_retention))
+ self.persistence_backend = _normalize(persistence_backend) or "memory"
+ self.persistence_path = _clean_text(persistence_path)
+ self.audit_event_totals: Dict[str, int] = {field: 0 for field in _AUDIT_LOG_FIELDS}
+ self.audit_trimmed_counts: Dict[str, int] = {field: 0 for field in _AUDIT_LOG_FIELDS}
+
+ def next_turn(self) -> int:
+ self.turn_index += 1
+ return self.turn_index
+
+ def configure_persistence(self, *, backend: str, path: str = "", audit_retention: int | None = None) -> None:
+ self.persistence_backend = _normalize(backend) or "memory"
+ self.persistence_path = _clean_text(path)
+ if audit_retention is not None:
+ self.audit_retention = max(1, int(audit_retention))
+
+ def _append_audit_event(self, field_name: str, payload: Dict[str, Any]) -> None:
+ if field_name not in _AUDIT_LOG_FIELDS:
+ raise KeyError(f"Unknown audit field: {field_name}")
+ events = getattr(self, field_name)
+ events.append(payload)
+ self.audit_event_totals[field_name] = int(self.audit_event_totals.get(field_name, 0) or 0) + 1
+ overflow = max(0, len(events) - self.audit_retention)
+ if overflow:
+ del events[:overflow]
+ self.audit_trimmed_counts[field_name] = int(self.audit_trimmed_counts.get(field_name, 0) or 0) + overflow
+
+ def _slot_active_records(self, slot_key: str) -> List[SessionMemoryRecordV2]:
+ active_records: List[SessionMemoryRecordV2] = []
+ for memory_id in reversed(list(self.slot_history.get(slot_key, []) or [])):
+ record = self.records_by_id.get(memory_id)
+ if record is None or not _is_active_record_state(record.state):
+ continue
+ active_records.append(record)
+ return active_records
+
+ def _record_subject_signature(self, record: SessionMemoryRecordV2) -> str:
+ return _record_subject_signature_from_parts(record.slot_key, dict(record.metadata or {}))
+
+ def _same_subject_records(self, subject_signature: str) -> List[SessionMemoryRecordV2]:
+ normalized_subject = _public_subject_signature(subject_signature)
+ if not normalized_subject:
+ return []
+ records = [
+ record
+ for record in self.records_by_id.values()
+ if self._record_subject_signature(record) == normalized_subject
+ ]
+ return sorted(records, key=lambda item: (int(item.turn_index), item.memory_id))
+
+ def _refresh_subject_depth_head(self, record: SessionMemoryRecordV2) -> None:
+ if not _is_active_record_state(record.state):
+ return
+ subject_signature = self._record_subject_signature(record)
+ depth_layer = _normalize(dict(record.metadata or {}).get("depth_layer", "")) or "core_view"
+ if not subject_signature or depth_layer not in _DEPTH_LAYERS:
+ return
+ current_id = self.subject_depth_heads[subject_signature].get(depth_layer, "")
+ current = self.records_by_id.get(current_id)
+ if current is None or int(record.turn_index) >= int(current.turn_index):
+ self.subject_depth_heads[subject_signature][depth_layer] = record.memory_id
+
+ def _upsert_memory_edge(self, edge: SessionMemoryEdgeV2) -> None:
+ if not edge.source_memory_id or not edge.target_memory_id or edge.source_memory_id == edge.target_memory_id:
+ return
+ edge.edge_type = _normalize(edge.edge_type) or "co_mentions"
+ edge.edge_id = edge.edge_id or f"{edge.source_memory_id}->{edge.target_memory_id}:{edge.edge_type}"
+ existing = self.memory_edges.get(edge.edge_id)
+ if existing is not None and float(existing.score) >= float(edge.score):
+ return
+ self.memory_edges[edge.edge_id] = edge
+
+ def _active_profile_aggregate_metadata(self, metadata: Mapping[str, Any]) -> Dict[str, Any]:
+ filtered = dict(metadata or {})
+ raw_support_ids = [_clean_text(item) for item in filtered.get("profile_support_ids", []) or [] if _clean_text(item)]
+ if not raw_support_ids:
+ return filtered
+ active_supports: List[SessionMemoryRecordV2] = []
+ for support_id in raw_support_ids:
+ support = self.records_by_id.get(support_id)
+ if support is None or not _is_active_record_state(support.state):
+ continue
+ active_supports.append(support)
+ route_terms: List[str] = []
+ for support in active_supports:
+ support_metadata = dict(support.metadata or {})
+ route_terms.extend(support_metadata.get("profile_route_terms", []) or [])
+ route_terms.extend(support.anchor_concepts or [])
+ filtered["profile_support_ids"] = _dedupe([support.memory_id for support in active_supports], max_items=64)
+ filtered["profile_support_turns"] = sorted({int(support.turn_index) for support in active_supports})
+ filtered["profile_support_values"] = _dedupe([support.value for support in active_supports], max_items=12)
+ filtered["profile_support_count"] = len(filtered["profile_support_ids"])
+ if route_terms:
+ filtered["profile_route_terms"] = _dedupe(route_terms, max_items=24)
+ return filtered
+
+ def _active_profile_cluster_metadata(self, metadata: Mapping[str, Any]) -> Dict[str, Any]:
+ filtered = dict(metadata or {})
+ raw_profile_ids = [_clean_text(item) for item in filtered.get("profile_support_profile_ids", []) or [] if _clean_text(item)]
+ if not raw_profile_ids:
+ return filtered
+ active_profiles: List[SessionMemoryRecordV2] = []
+ for profile_id in raw_profile_ids:
+ profile_record = self.records_by_id.get(profile_id)
+ if profile_record is None or not _is_active_record_state(profile_record.state):
+ continue
+ if _normalize(profile_record.source_kind) != PROFILE_AGGREGATE_SOURCE_KIND:
+ continue
+ active_profiles.append(profile_record)
+ support_ids: List[str] = []
+ support_turns: List[int] = []
+ support_values: List[str] = []
+ route_terms: List[str] = []
+ profile_types: List[str] = []
+ domains: List[str] = []
+ for profile_record in active_profiles:
+ profile_metadata = dict(profile_record.metadata or {})
+ support_ids.extend(profile_metadata.get("profile_support_ids", []) or [])
+ support_turns.extend(profile_metadata.get("profile_support_turns", []) or [])
+ support_values.extend(profile_metadata.get("profile_support_values", []) or [])
+ route_terms.extend(profile_metadata.get("profile_route_terms", []) or [])
+ route_terms.extend(profile_metadata.get("profile_cluster_route_terms", []) or [])
+ route_terms.extend(profile_record.anchor_concepts or [])
+ profile_types.append(_clean_text(profile_metadata.get("profile_type", "")))
+ domains.extend(
+ [
+ _clean_text(profile_metadata.get("profile_domain", "")),
+ _clean_text(profile_metadata.get("profile_domain_label", "")),
+ ]
+ )
+ filtered["profile_support_profile_ids"] = _dedupe([profile.memory_id for profile in active_profiles], max_items=32)
+ filtered["profile_support_ids"] = _dedupe(support_ids, max_items=96)
+ filtered["profile_support_turns"] = sorted({int(item) for item in support_turns if str(item).strip().lstrip("-").isdigit()})
+ filtered["profile_support_values"] = _dedupe(support_values, max_items=16)
+ filtered["profile_support_count"] = len(filtered["profile_support_ids"])
+ filtered["profile_cluster_profile_count"] = len(filtered["profile_support_profile_ids"])
+ filtered["profile_cluster_types"] = _dedupe(profile_types, max_items=8)
+ filtered["profile_cluster_domains"] = _dedupe(domains, max_items=16)
+ if route_terms:
+ clean_terms = _dedupe(route_terms, max_items=32)
+ filtered["profile_cluster_route_terms"] = clean_terms
+ filtered["profile_route_terms"] = clean_terms
+ return filtered
+
+ def _profile_consolidator_api_key(self) -> str:
+ explicit = _clean_text(os.getenv("TMCRA_PROFILE_CONSOLIDATOR_API_KEY", ""))
+ if explicit:
+ return explicit
+ pool = _clean_text(os.getenv("TMCRA_DEEPSEEK_WRITER_KEY_POOL", ""))
+ if not pool:
+ pool_files = [
+ _clean_text(os.getenv("TMCRA_PROFILE_CONSOLIDATOR_KEY_POOL_FILE", "")),
+ _clean_text(os.getenv("TMCRA_DEEPSEEK_WRITER_KEY_POOL_FILE", "")),
+ ]
+ for path_text in pool_files:
+ if not path_text:
+ continue
+ try:
+ path = Path(path_text)
+ if not path.exists():
+ continue
+ for line in path.read_text(errors="replace").splitlines():
+ clean_line = line.strip()
+ if not clean_line or clean_line.startswith("#") or "=" not in clean_line:
+ continue
+ name, value = clean_line.split("=", 1)
+ if name.strip() == "TMCRA_DEEPSEEK_WRITER_KEY_POOL":
+ pool = value.strip().strip('"').strip("'")
+ break
+ if pool:
+ break
+ except Exception:
+ continue
+ if not pool:
+ return ""
+ for candidate in re.split(r"[\s,;]+", pool):
+ candidate = _clean_text(candidate)
+ if candidate:
+ return candidate
+ return ""
+
+ def _profile_consolidator_enabled(self, metadata: Mapping[str, Any]) -> bool:
+ flag = _normalize(os.getenv("TMCRA_PROFILE_CONSOLIDATOR_ENABLED", "0"))
+ if flag not in {"1", "true", "yes", "on"}:
+ return False
+ try:
+ min_support = int(os.getenv("TMCRA_PROFILE_CONSOLIDATOR_MIN_SUPPORT_COUNT", "2") or "2")
+ except ValueError:
+ min_support = 2
+ support_count = int(metadata.get("profile_support_count", 0) or 0)
+ if support_count < max(1, min_support):
+ return False
+ return bool(self._profile_consolidator_api_key())
+
+ def _extract_profile_consolidator_json(self, content: str) -> Dict[str, Any]:
+ text = _clean_text(content)
+ if not text:
+ return {}
+ try:
+ parsed = json.loads(text)
+ return dict(parsed) if isinstance(parsed, dict) else {}
+ except Exception:
+ pass
+ start = text.find("{")
+ end = text.rfind("}")
+ if start >= 0 and end > start:
+ try:
+ parsed = json.loads(text[start : end + 1])
+ return dict(parsed) if isinstance(parsed, dict) else {}
+ except Exception:
+ return {}
+ return {}
+
+ def _apply_profile_llm_consolidator(self, metadata: Mapping[str, Any]) -> Dict[str, Any]:
+ consolidated = dict(metadata or {})
+ if not self._profile_consolidator_enabled(consolidated):
+ return consolidated
+ base_url = _clean_text(os.getenv("TMCRA_PROFILE_CONSOLIDATOR_BASE_URL", "https://api.deepseek.com/v1")).rstrip("/")
+ model = _clean_text(os.getenv("TMCRA_PROFILE_CONSOLIDATOR_MODEL", "deepseek-chat"))
+ api_key = self._profile_consolidator_api_key()
+ if not base_url or not model or not api_key:
+ return consolidated
+ try:
+ max_tokens = int(os.getenv("TMCRA_PROFILE_CONSOLIDATOR_MAX_TOKENS", "256") or "256")
+ except ValueError:
+ max_tokens = 256
+ try:
+ timeout_seconds = float(os.getenv("TMCRA_PROFILE_CONSOLIDATOR_TIMEOUT_SECONDS", "45") or "45")
+ except ValueError:
+ timeout_seconds = 45.0
+ system_prompt = (
+ "Return JSON only. Consolidate a user profile cluster from evidence. "
+ "Do not invent facts. Do not expose memory ids, retrieval, scores, or internal metadata. "
+ "Keep profile_summary concise and directly usable as long-term memory. "
+ "Return keys: profile_summary, profile_domain_label, profile_output_kind, profile_update_policy."
+ )
+ payload = {
+ "profile_types": list(consolidated.get("profile_cluster_types", []) or []),
+ "domain_candidates": list(consolidated.get("profile_cluster_domains", []) or []),
+ "support_values": list(consolidated.get("profile_support_values", []) or [])[:10],
+ "deterministic_summary": _clean_text(consolidated.get("profile_summary", "")),
+ "output_kind": _clean_text(consolidated.get("profile_output_kind", "")),
+ "update_policy": _clean_text(consolidated.get("profile_update_policy", "")),
+ }
+ request_payload = {
+ "model": model,
+ "messages": [
+ {"role": "system", "content": system_prompt},
+ {"role": "user", "content": json.dumps(payload, ensure_ascii=False)},
+ ],
+ "temperature": 0,
+ "max_tokens": max(64, min(max_tokens, 768)),
+ "response_format": {"type": "json_object"},
+ }
+ headers = {"Content-Type": "application/json", "Authorization": f"Bearer {api_key}"}
+ url = f"{base_url}/chat/completions"
+ try:
+ req = urllib.request.Request(
+ url,
+ data=json.dumps(request_payload, ensure_ascii=False).encode("utf-8"),
+ headers=headers,
+ method="POST",
+ )
+ with urllib.request.urlopen(req, timeout=max(1.0, timeout_seconds)) as response:
+ response_payload = json.loads(response.read().decode("utf-8"))
+ choices = list(response_payload.get("choices", []) or [])
+ message = dict(dict(choices[0] if choices else {}).get("message", {}) or {})
+ parsed = self._extract_profile_consolidator_json(_clean_text(message.get("content", "")))
+ summary = _clean_text(parsed.get("profile_summary", ""))
+ if summary:
+ consolidated["profile_summary"] = summary[:900]
+ consolidated["profile_consolidator_model"] = model
+ consolidated["profile_consolidator_status"] = "llm_ok"
+ consolidated["profile_consolidator_source"] = "deepseek_openai_compat"
+ for key in ("profile_domain_label", "profile_output_kind", "profile_update_policy"):
+ value = _clean_text(parsed.get(key, ""))
+ if value:
+ consolidated[key] = value[:160]
+ usage = dict(response_payload.get("usage", {}) or {})
+ if usage:
+ consolidated["profile_consolidator_usage"] = {
+ "prompt_tokens": int(usage.get("prompt_tokens", 0) or 0),
+ "completion_tokens": int(usage.get("completion_tokens", 0) or 0),
+ "total_tokens": int(usage.get("total_tokens", 0) or 0),
+ }
+ else:
+ consolidated["profile_consolidator_status"] = "llm_empty"
+ except Exception as exc:
+ consolidated["profile_consolidator_status"] = "llm_failed"
+ consolidated["profile_consolidator_error"] = f"{exc.__class__.__name__}: {_clean_text(str(exc))[:180]}"
+ return consolidated
+
+ def _link_depth_edges_for_record(self, record: SessionMemoryRecordV2, previous_records: Sequence[SessionMemoryRecordV2]) -> None:
+ metadata = dict(record.metadata or {})
+ subject_signature = self._record_subject_signature(record)
+ source_layer = _normalize(metadata.get("depth_layer", "")) or "core_view"
+ if not subject_signature or source_layer not in _DEPTH_LAYERS:
+ return
+ candidates: List[tuple[float, SessionMemoryRecordV2, str]] = []
+ source_tokens = set(_tokenize(record.value))
+ for previous in previous_records:
+ if previous.memory_id == record.memory_id:
+ continue
+ if self._record_subject_signature(previous) != subject_signature:
+ continue
+ target_layer = _normalize(dict(previous.metadata or {}).get("depth_layer", "")) or "core_view"
+ if target_layer not in _DEPTH_LAYERS:
+ target_layer = "core_view"
+ target_tokens = set(_tokenize(previous.value))
+ overlap = len(source_tokens & target_tokens) / max(1, len(source_tokens | target_tokens)) if source_tokens or target_tokens else 0.0
+ turn_gap = max(0, int(record.turn_index) - int(previous.turn_index))
+ recency = max(0.0, 1.0 - (turn_gap / max(1.0, float(max(1, int(self.turn_index or record.turn_index or 1))))))
+ depth_bonus = 0.24 if source_layer != target_layer else 0.08
+ state_bonus = 0.10 if _is_active_record_state(previous.state) else 0.02
+ edge_type = _depth_edge_type(source_layer, target_layer, record.value, previous.value)
+ score = 0.38 + (0.24 * overlap) + (0.18 * recency) + depth_bonus + state_bonus
+ if edge_type in {"refines", "operationalizes", "measures", "productizes", "qualifies"}:
+ score += 0.12
+ candidates.append((round(score, 6), previous, edge_type))
+ candidates.sort(
+ key=lambda item: (
+ float(item[0]),
+ _DEPTH_LAYER_ORDER.get(_normalize(dict(item[1].metadata or {}).get("depth_layer", "")), 99) != _DEPTH_LAYER_ORDER.get(source_layer, 99),
+ int(item[1].turn_index),
+ ),
+ reverse=True,
+ )
+ for score, previous, edge_type in candidates[:4]:
+ edge_id = f"{record.memory_id}->{previous.memory_id}:{edge_type}"
+ self._upsert_memory_edge(
+ SessionMemoryEdgeV2(
+ edge_id=edge_id,
+ source_memory_id=record.memory_id,
+ target_memory_id=previous.memory_id,
+ edge_type=edge_type,
+ score=score,
+ model_score=0.0,
+ evidence_turn=int(record.turn_index),
+ evidence=_clean_text(dict(record.metadata or {}).get("source_span", "")) or record.value,
+ metadata={
+ "subject_signature": subject_signature,
+ "source_depth_layer": source_layer,
+ "target_depth_layer": _normalize(dict(previous.metadata or {}).get("depth_layer", "")) or "core_view",
+ "edge_source": "depth_heuristic_v1",
+ },
+ )
+ )
+
+ def _link_profile_edges_for_record(self, record: SessionMemoryRecordV2) -> None:
+ metadata = dict(record.metadata or {})
+ if not is_profile_layer_record(
+ category=record.category,
+ source_kind=record.source_kind,
+ semantic_slot=metadata.get("semantic_slot", ""),
+ metadata=metadata,
+ ):
+ return
+ subject_signature = self._record_subject_signature(record)
+ candidates: List[tuple[float, SessionMemoryRecordV2, str]] = []
+ for previous in self.records_by_id.values():
+ if previous.memory_id == record.memory_id:
+ continue
+ previous_metadata = dict(previous.metadata or {})
+ score, edge_type = profile_edge_score(
+ metadata,
+ previous_metadata,
+ source_value=record.value,
+ target_value=previous.value,
+ )
+ if score <= 0:
+ continue
+ if subject_signature and self._record_subject_signature(previous) == subject_signature:
+ score += 0.04
+ candidates.append((round(min(score, 0.96), 6), previous, edge_type))
+ candidates.sort(key=lambda item: (float(item[0]), int(item[1].turn_index)), reverse=True)
+ for score, previous, edge_type in candidates[:6]:
+ previous_metadata = dict(previous.metadata or {})
+ self._upsert_memory_edge(
+ SessionMemoryEdgeV2(
+ edge_id=f"{record.memory_id}->{previous.memory_id}:{edge_type}",
+ source_memory_id=record.memory_id,
+ target_memory_id=previous.memory_id,
+ edge_type=edge_type,
+ score=score,
+ model_score=0.0,
+ evidence_turn=int(record.turn_index),
+ evidence=_clean_text(metadata.get("source_span", "")) or record.value,
+ metadata={
+ "subject_signature": subject_signature,
+ "source_profile_type": _clean_text(metadata.get("profile_type", "")),
+ "target_profile_type": _clean_text(previous_metadata.get("profile_type", "")),
+ "source_profile_domain": _clean_text(metadata.get("profile_domain", "")),
+ "target_profile_domain": _clean_text(previous_metadata.get("profile_domain", "")),
+ "edge_source": "profile_layer_v1",
+ },
+ )
+ )
+
+ def _link_facet_edges_for_record(self, record: SessionMemoryRecordV2) -> None:
+ metadata = dict(record.metadata or {})
+ facet_type = _normalize(metadata.get("facet_type", ""))
+ if _normalize(metadata.get("content_variant", "")) != "event_facet_write" and not facet_type:
+ return
+ parent_slot_key = _clean_text(metadata.get("facet_parent_slot_key", ""))
+ parent_event_signature = _normalize(metadata.get("facet_parent_event_signature", ""))
+ if not parent_slot_key and not parent_event_signature:
+ return
+ candidates: List[tuple[float, SessionMemoryRecordV2]] = []
+ for candidate in self.records_by_id.values():
+ if candidate.memory_id == record.memory_id:
+ continue
+ candidate_metadata = dict(candidate.metadata or {})
+ if _normalize(candidate_metadata.get("content_variant", "")) == "event_facet_write":
+ continue
+ score = 0.0
+ if parent_slot_key and _clean_text(candidate.slot_key).lower() == parent_slot_key.lower():
+ score += 0.66
+ if parent_event_signature and _normalize(candidate_metadata.get("event_signature", "")) == parent_event_signature:
+ score += 0.28
+ if int(candidate.turn_index) == int(record.turn_index):
+ score += 0.08
+ if score > 0:
+ candidates.append((round(min(score, 0.98), 6), candidate))
+ if not candidates:
+ return
+ candidates.sort(key=lambda item: (float(item[0]), int(item[1].turn_index)), reverse=True)
+ score, parent = candidates[0]
+ edge_type = f"has_{facet_type or 'attribute'}_facet"
+ evidence = _clean_text(metadata.get("facet_source_span", "")) or _clean_text(metadata.get("source_span", "")) or record.value
+ shared_metadata = {
+ "edge_source": "event_facet_layer_v1",
+ "facet_type": facet_type,
+ "facet_role": _clean_text(metadata.get("facet_role", "")),
+ "facet_value": _clean_text(metadata.get("facet_value", "")) or record.value,
+ "parent_slot_key": parent_slot_key,
+ "parent_event_signature": parent_event_signature,
+ "subject_signature": self._record_subject_signature(parent) or self._record_subject_signature(record),
+ }
+ self._upsert_memory_edge(
+ SessionMemoryEdgeV2(
+ edge_id=f"{parent.memory_id}->{record.memory_id}:{edge_type}",
+ source_memory_id=parent.memory_id,
+ target_memory_id=record.memory_id,
+ edge_type=edge_type,
+ score=score,
+ model_score=0.0,
+ evidence_turn=int(record.turn_index),
+ evidence=evidence,
+ metadata=shared_metadata,
+ )
+ )
+ self._upsert_memory_edge(
+ SessionMemoryEdgeV2(
+ edge_id=f"{record.memory_id}->{parent.memory_id}:facet_of",
+ source_memory_id=record.memory_id,
+ target_memory_id=parent.memory_id,
+ edge_type="facet_of",
+ score=score,
+ model_score=0.0,
+ evidence_turn=int(record.turn_index),
+ evidence=evidence,
+ metadata=shared_metadata,
+ )
+ )
+
+ def _upsert_profile_cluster_for_aggregate(self, aggregate: SessionMemoryRecordV2) -> str:
+ metadata = dict(aggregate.metadata or {})
+ if _normalize(aggregate.source_kind) != PROFILE_AGGREGATE_SOURCE_KIND:
+ return ""
+ if not bool(metadata.get("profile_aggregate_node")):
+ return ""
+ if bool(metadata.get("profile_cluster_node")):
+ return ""
+ if not _is_active_record_state(aggregate.state):
+ return ""
+
+ candidates: List[tuple[float, SessionMemoryRecordV2]] = []
+ for candidate in self.records_by_id.values():
+ candidate_metadata = dict(candidate.metadata or {})
+ if candidate.memory_id == aggregate.memory_id:
+ continue
+ if _normalize(candidate.source_kind) != PROFILE_CLUSTER_SOURCE_KIND and not bool(candidate_metadata.get("profile_cluster_node")):
+ continue
+ if not _is_active_record_state(candidate.state):
+ continue
+ support_profile_ids = {
+ _normalize(item)
+ for item in candidate_metadata.get("profile_support_profile_ids", []) or []
+ if _clean_text(item)
+ }
+ if _normalize(aggregate.memory_id) in support_profile_ids:
+ candidates.append((1.0, candidate))
+ continue
+ similarity = profile_cluster_similarity(metadata, candidate_metadata)
+ if similarity >= 0.16:
+ candidates.append((similarity, candidate))
+
+ candidates.sort(key=lambda item: (float(item[0]), int(item[1].turn_index)), reverse=True)
+ cluster = candidates[0][1] if candidates else None
+ existing_metadata = self._active_profile_cluster_metadata(dict(cluster.metadata or {})) if cluster is not None else {}
+ cluster_metadata = build_profile_cluster_metadata(
+ support_profile_id=aggregate.memory_id,
+ support_metadata=metadata,
+ existing_metadata=existing_metadata,
+ )
+ cluster_metadata = self._apply_profile_llm_consolidator(cluster_metadata)
+ slot_key = cluster.slot_key if cluster is not None else profile_cluster_slot_key(cluster_metadata)
+ if not slot_key:
+ return ""
+ cluster_value = _clean_text(cluster_metadata.get("profile_value", "")) or aggregate.value
+ cluster_anchors = _dedupe(
+ [
+ *list(cluster_metadata.get("profile_cluster_route_terms", []) or []),
+ *list(cluster_metadata.get("profile_route_terms", []) or []),
+ *list(aggregate.anchor_concepts or []),
+ ],
+ max_items=16,
+ )
+ if cluster is None:
+ cluster = SessionMemoryRecordV2(
+ memory_id=f"{slot_key}:cluster:{int(aggregate.turn_index)}",
+ category=PROFILE_CLUSTER_CATEGORY,
+ slot_key=slot_key,
+ value=cluster_value,
+ relation="profile_cluster",
+ anchor_concepts=cluster_anchors,
+ evidence_anchors=list(aggregate.evidence_anchors or aggregate.anchor_concepts),
+ salience=min(0.99, max(float(aggregate.salience), 0.90)),
+ confidence=min(0.97, max(float(aggregate.confidence), 0.84)),
+ source_kind=PROFILE_CLUSTER_SOURCE_KIND,
+ turn_index=int(aggregate.turn_index),
+ state="active",
+ metadata={
+ **cluster_metadata,
+ "canonical_slot_key": slot_key,
+ "memory_role": "user",
+ "authority": "derived_profile_cluster",
+ "source": "profile_layer",
+ },
+ )
+ _prepare_record_signatures(cluster)
+ self.records_by_id[cluster.memory_id] = cluster
+ self.slot_heads[slot_key] = cluster.memory_id
+ self.slot_history[slot_key].append(cluster.memory_id)
+ self._refresh_subject_depth_head(cluster)
+ else:
+ cluster.value = cluster_value
+ cluster.anchor_concepts = cluster_anchors
+ cluster.evidence_anchors = _dedupe(
+ [*list(cluster.evidence_anchors or []), *list(aggregate.evidence_anchors or aggregate.anchor_concepts)],
+ max_items=16,
+ )
+ cluster.salience = min(0.99, max(float(cluster.salience), float(aggregate.salience), 0.90))
+ cluster.confidence = min(0.97, max(float(cluster.confidence), float(aggregate.confidence), 0.84))
+ cluster.turn_index = min(int(cluster.turn_index or aggregate.turn_index), int(aggregate.turn_index))
+ cluster.state = "active"
+ cluster.metadata = {
+ **cluster_metadata,
+ "canonical_slot_key": slot_key,
+ "memory_role": "user",
+ "authority": "derived_profile_cluster",
+ "source": "profile_layer",
+ }
+ _prepare_record_signatures(cluster)
+ self._refresh_subject_depth_head(cluster)
+
+ self._upsert_memory_edge(
+ SessionMemoryEdgeV2(
+ edge_id=f"{aggregate.memory_id}->{cluster.memory_id}:profile_cluster_supports",
+ source_memory_id=aggregate.memory_id,
+ target_memory_id=cluster.memory_id,
+ edge_type="profile_cluster_supports",
+ score=0.95,
+ model_score=0.0,
+ evidence_turn=int(aggregate.turn_index),
+ evidence=aggregate.value,
+ metadata={
+ "edge_source": "profile_layer_cluster_v1",
+ "profile_support_key": _clean_text(cluster.metadata.get("profile_support_key", "")),
+ "profile_cluster_profile_count": int(cluster.metadata.get("profile_cluster_profile_count", 0) or 0),
+ },
+ )
+ )
+ self._upsert_memory_edge(
+ SessionMemoryEdgeV2(
+ edge_id=f"{cluster.memory_id}->{aggregate.memory_id}:profile_cluster_supported_by",
+ source_memory_id=cluster.memory_id,
+ target_memory_id=aggregate.memory_id,
+ edge_type="profile_cluster_supported_by",
+ score=0.90,
+ model_score=0.0,
+ evidence_turn=int(aggregate.turn_index),
+ evidence=aggregate.value,
+ metadata={
+ "edge_source": "profile_layer_cluster_v1",
+ "profile_support_key": _clean_text(cluster.metadata.get("profile_support_key", "")),
+ },
+ )
+ )
+ self._link_profile_edges_for_record(cluster)
+ return cluster.memory_id
+
+ def _upsert_profile_aggregate_for_record(self, record: SessionMemoryRecordV2) -> str:
+ legacy_profile_layer = _normalize(os.getenv("TMCRA_LEGACY_PROFILE_LAYER_ENABLED", "1"))
+ if legacy_profile_layer in {"0", "false", "no", "off", "disabled"}:
+ return ""
+ metadata = dict(record.metadata or {})
+ if _normalize(record.source_kind) in {PROFILE_AGGREGATE_SOURCE_KIND, PROFILE_CLUSTER_SOURCE_KIND}:
+ return ""
+ if bool(metadata.get("profile_aggregate_node")):
+ return ""
+ if bool(metadata.get("profile_cluster_node")):
+ return ""
+ if not _is_active_record_state(record.state):
+ return ""
+ if not is_profile_layer_record(
+ category=record.category,
+ source_kind=record.source_kind,
+ semantic_slot=metadata.get("semantic_slot", ""),
+ metadata=metadata,
+ ):
+ return ""
+ slot_key = profile_aggregate_slot_key(metadata)
+ if not slot_key:
+ return ""
+ aggregate_id = self.slot_heads.get(slot_key, "")
+ aggregate = self.records_by_id.get(aggregate_id) if aggregate_id else None
+ existing_metadata = self._active_profile_aggregate_metadata(dict(aggregate.metadata or {})) if aggregate is not None else {}
+ aggregate_metadata = build_profile_aggregate_metadata(
+ support_record_id=record.memory_id,
+ support_turn_index=int(record.turn_index),
+ support_value=record.value,
+ support_anchors=record.anchor_concepts,
+ support_metadata=metadata,
+ existing_metadata=existing_metadata,
+ )
+ aggregate_value = _clean_text(aggregate_metadata.get("profile_value", "")) or record.value
+ aggregate_anchors = _dedupe(
+ [
+ *list(aggregate_metadata.get("profile_route_terms", []) or []),
+ *list(record.anchor_concepts or []),
+ ],
+ max_items=12,
+ )
+ if aggregate is None:
+ aggregate = SessionMemoryRecordV2(
+ memory_id=f"{slot_key}:profile:{int(record.turn_index)}",
+ category=PROFILE_AGGREGATE_CATEGORY,
+ slot_key=slot_key,
+ value=aggregate_value,
+ relation="profile_aggregate",
+ anchor_concepts=aggregate_anchors,
+ evidence_anchors=list(record.evidence_anchors or record.anchor_concepts),
+ salience=min(0.98, max(float(record.salience), 0.86)),
+ confidence=min(0.96, max(float(record.confidence), 0.82)),
+ source_kind=PROFILE_AGGREGATE_SOURCE_KIND,
+ turn_index=int(record.turn_index),
+ state="active",
+ metadata={
+ **aggregate_metadata,
+ "canonical_slot_key": slot_key,
+ "memory_role": "user",
+ "authority": "derived_profile",
+ "source": "profile_layer",
+ },
+ )
+ _prepare_record_signatures(aggregate)
+ self.records_by_id[aggregate.memory_id] = aggregate
+ self.slot_heads[slot_key] = aggregate.memory_id
+ self.slot_history[slot_key].append(aggregate.memory_id)
+ self._refresh_subject_depth_head(aggregate)
+ else:
+ aggregate.value = aggregate_value
+ aggregate.anchor_concepts = aggregate_anchors
+ aggregate.evidence_anchors = _dedupe(
+ [*list(aggregate.evidence_anchors or []), *list(record.evidence_anchors or record.anchor_concepts)],
+ max_items=12,
+ )
+ aggregate.salience = min(0.98, max(float(aggregate.salience), float(record.salience), 0.86))
+ aggregate.confidence = min(0.96, max(float(aggregate.confidence), float(record.confidence), 0.82))
+ aggregate.turn_index = min(int(aggregate.turn_index or record.turn_index), int(record.turn_index))
+ aggregate.state = "active"
+ aggregate.metadata = {
+ **aggregate_metadata,
+ "canonical_slot_key": slot_key,
+ "memory_role": "user",
+ "authority": "derived_profile",
+ "source": "profile_layer",
+ }
+ _prepare_record_signatures(aggregate)
+ self._refresh_subject_depth_head(aggregate)
+ self._upsert_memory_edge(
+ SessionMemoryEdgeV2(
+ edge_id=f"{record.memory_id}->{aggregate.memory_id}:profile_supports",
+ source_memory_id=record.memory_id,
+ target_memory_id=aggregate.memory_id,
+ edge_type="profile_supports",
+ score=0.94,
+ model_score=0.0,
+ evidence_turn=int(record.turn_index),
+ evidence=record.value,
+ metadata={
+ "edge_source": "profile_layer_aggregate_v1",
+ "profile_support_key": _clean_text(aggregate.metadata.get("profile_support_key", "")),
+ "profile_type": _clean_text(aggregate.metadata.get("profile_type", "")),
+ "profile_domain": _clean_text(aggregate.metadata.get("profile_domain", "")),
+ },
+ )
+ )
+ self._upsert_memory_edge(
+ SessionMemoryEdgeV2(
+ edge_id=f"{aggregate.memory_id}->{record.memory_id}:profile_supported_by",
+ source_memory_id=aggregate.memory_id,
+ target_memory_id=record.memory_id,
+ edge_type="profile_supported_by",
+ score=0.88,
+ model_score=0.0,
+ evidence_turn=int(record.turn_index),
+ evidence=record.value,
+ metadata={
+ "edge_source": "profile_layer_aggregate_v1",
+ "profile_support_key": _clean_text(aggregate.metadata.get("profile_support_key", "")),
+ },
+ )
+ )
+ self._link_profile_edges_for_record(aggregate)
+ self._upsert_profile_cluster_for_aggregate(aggregate)
+ return aggregate.memory_id
+
+ def depth_chain_for_query(
+ self,
+ query: str,
+ *,
+ seed_memory_ids: Sequence[str] | None = None,
+ top_k: int = 6,
+ ) -> Dict[str, Any]:
+ query_text = _clean_text(query)
+ query_tokens = set(_tokenize(query_text))
+ analysis = _public_query_analysis(query_text, query_tokens)
+ subject_signature = _public_subject_signature(analysis.get("query_subject", ""))
+ seed_ids = [_clean_text(item) for item in list(seed_memory_ids or []) if _clean_text(item)]
+ if not subject_signature:
+ for memory_id in seed_ids:
+ record = self.records_by_id.get(memory_id)
+ if record is None:
+ continue
+ subject_signature = self._record_subject_signature(record)
+ if subject_signature:
+ break
+ if not subject_signature:
+ return {"enabled": False, "reason": "no_subject_signature", "nodes": [], "edges": []}
+ subject_records = self._same_subject_records(subject_signature)
+ if not subject_records:
+ return {
+ "enabled": True,
+ "reason": "no_subject_records",
+ "subject_signature": subject_signature,
+ "nodes": [],
+ "edges": [],
+ }
+ seed_set = set(seed_ids)
+ head_ids = set(self.subject_depth_heads.get(subject_signature, {}).values())
+ edge_neighbor_ids = set()
+ for edge in self.memory_edges.values():
+ if _normalize(dict(edge.metadata or {}).get("subject_signature", "")) != subject_signature:
+ continue
+ if edge.source_memory_id in seed_set or edge.target_memory_id in seed_set or edge.source_memory_id in head_ids or edge.target_memory_id in head_ids:
+ edge_neighbor_ids.add(edge.source_memory_id)
+ edge_neighbor_ids.add(edge.target_memory_id)
+ scored: List[tuple[float, SessionMemoryRecordV2]] = []
+ for record in subject_records:
+ layer = _normalize(dict(record.metadata or {}).get("depth_layer", "")) or "core_view"
+ layer_rank = _DEPTH_LAYER_ORDER.get(layer, 99)
+ token_overlap = 0.0
+ value_tokens = set(_tokenize(record.value))
+ if query_tokens or value_tokens:
+ token_overlap = len(query_tokens & value_tokens) / max(1, len(query_tokens | value_tokens))
+ score = 0.30 + (0.18 * token_overlap)
+ if record.memory_id in seed_set:
+ score += 0.50
+ if record.memory_id in head_ids:
+ score += 0.32
+ if record.memory_id in edge_neighbor_ids:
+ score += 0.20
+ if _is_active_record_state(record.state):
+ score += 0.12
+ score += max(0.0, 0.18 - (0.018 * layer_rank))
+ scored.append((round(score, 6), record))
+ scored.sort(key=lambda item: (item[0], int(item[1].turn_index)), reverse=True)
+ selected: List[SessionMemoryRecordV2] = []
+ seen_layers: set[str] = set()
+ for _, record in scored:
+ layer = _normalize(dict(record.metadata or {}).get("depth_layer", "")) or "core_view"
+ if layer not in seen_layers or record.memory_id in seed_set or record.memory_id in head_ids:
+ selected.append(record)
+ seen_layers.add(layer)
+ if len(selected) >= max(1, int(top_k)):
+ break
+ if len(selected) < max(1, min(3, int(top_k))):
+ selected_ids = {record.memory_id for record in selected}
+ for _, record in scored:
+ if record.memory_id in selected_ids:
+ continue
+ selected.append(record)
+ selected_ids.add(record.memory_id)
+ if len(selected) >= max(1, int(top_k)):
+ break
+ selected_ids = {record.memory_id for record in selected}
+ chain_edges = [
+ edge
+ for edge in self.memory_edges.values()
+ if edge.source_memory_id in selected_ids
+ and edge.target_memory_id in selected_ids
+ and _normalize(dict(edge.metadata or {}).get("subject_signature", "")) == subject_signature
+ ]
+ chain_edges.sort(key=lambda item: (float(item.score), int(item.evidence_turn)), reverse=True)
+ ordered_nodes = sorted(
+ selected,
+ key=lambda item: (
+ _DEPTH_LAYER_ORDER.get(_normalize(dict(item.metadata or {}).get("depth_layer", "")) or "core_view", 99),
+ int(item.turn_index),
+ ),
+ )
+ return {
+ "enabled": True,
+ "reason": "subject_depth_chain",
+ "subject_signature": subject_signature,
+ "subject": _clean_text(analysis.get("query_subject", "")),
+ "node_count": len(ordered_nodes),
+ "edge_count": len(chain_edges),
+ "depth_layers": _dedupe(
+ _normalize(dict(record.metadata or {}).get("depth_layer", "")) or "core_view"
+ for record in ordered_nodes
+ ),
+ "nodes": [record.to_dict() for record in ordered_nodes],
+ "edges": [edge.to_dict() for edge in chain_edges[: max(0, int(top_k) * 2)]],
+ }
+
+ def _core_payload(self) -> Dict[str, Any]:
+ return {
+ "summary": self.summary(),
+ "slot_heads": dict(self.slot_heads),
+ "slot_history": {slot: list(values) for slot, values in self.slot_history.items()},
+ "subject_depth_heads": {subject: dict(heads) for subject, heads in self.subject_depth_heads.items()},
+ "records": [record.to_dict() for record in sorted(self.records_by_id.values(), key=lambda item: (item.turn_index, item.slot_key, item.state))],
+ "memory_edges": [edge.to_dict() for edge in sorted(self.memory_edges.values(), key=lambda item: (item.evidence_turn, item.edge_id))],
+ }
+
+ def _audit_payload(self) -> Dict[str, Any]:
+ return {
+ "turn_log": list(self.turn_log),
+ "retrieval_log": list(self.retrieval_log),
+ "answer_support_log": list(self.answer_support_log),
+ "audit_event_totals": dict(self.audit_event_totals),
+ "audit_trimmed_counts": dict(self.audit_trimmed_counts),
+ "audit_retention": int(self.audit_retention),
+ }
+
+ def record_turn(
+ self,
+ *,
+ turn_kind: str,
+ text: str,
+ turn_index: int,
+ record_ids: Sequence[str],
+ speaker: str = "user",
+ assistant_text: str = "",
+ metadata: Dict[str, Any] | None = None,
+ writeback_class: str = "",
+ ) -> None:
+ if not record_ids and infer_noise(text):
+ self.noise_turn_count += 1
+ self._append_audit_event(
+ "turn_log",
+ {
+ "turn_id": f"turn:{turn_index}",
+ "turn_index": int(turn_index),
+ "kind": turn_kind,
+ "text": _clean_text(text),
+ "speaker": _clean_text(speaker) or "user",
+ "assistant_text": _clean_text(assistant_text),
+ "writeback_class": _clean_text(writeback_class),
+ "record_ids": list(record_ids),
+ "metadata": dict(metadata or {}),
+ },
+ )
+
+ def add_records(self, records: Iterable[SessionMemoryRecordV2]) -> List[str]:
+ stored_ids: List[str] = []
+ for record in records:
+ record.value = _clean_text(record.value)
+ record.category = _clean_text(record.category) or "memory"
+ record.slot_key = _clean_text(record.slot_key) or guess_slot_key(category=record.category, value=record.value, anchors=record.anchor_concepts)
+ record.relation = _clean_text(record.relation) or "related_to"
+ record.anchor_concepts = _dedupe(record.anchor_concepts)
+ record.evidence_anchors = _dedupe(record.evidence_anchors or record.anchor_concepts)
+ record.state = _normalize(record.state) or "active"
+ _prepare_record_signatures(record)
+ previous_subject_records = self._same_subject_records(self._record_subject_signature(record))
+ if not record.memory_id:
+ record.memory_id = f"{record.slot_key}:{record.turn_index}:{len(self.records_by_id)}"
+
+ if _is_suspect_record_state(record.state):
+ record.metadata["conflict_action"] = _clean_text(record.metadata.get("conflict_action", "")) or "suspect_buffer"
+ record.metadata["promotion_state"] = _clean_text(record.metadata.get("promotion_state", "")) or "suspect_buffered"
+ record.metadata["suspect_support_turns"] = _record_suspect_support_turns(record)
+ record.metadata["suspect_support_count"] = len(record.metadata["suspect_support_turns"])
+ suspect_duplicate = next(
+ (
+ candidate
+ for candidate in self.records_by_id.values()
+ if _is_suspect_record_state(candidate.state)
+ and _normalize(candidate.slot_key) == _normalize(record.slot_key)
+ and _normalize(dict(candidate.metadata or {}).get("memory_signature", "")) == _normalize(dict(record.metadata or {}).get("memory_signature", ""))
+ ),
+ None,
+ )
+ if suspect_duplicate is not None:
+ _merge_record_payload(suspect_duplicate, record)
+ support_turns = sorted(
+ set(_record_suspect_support_turns(suspect_duplicate)) | set(_record_suspect_support_turns(record))
+ )
+ suspect_duplicate.metadata["suspect_support_turns"] = support_turns
+ suspect_duplicate.metadata["suspect_support_count"] = len(support_turns)
+ suspect_duplicate.metadata["conflict_action"] = "suspect_duplicate_reinforced"
+ stored_ids.append(suspect_duplicate.memory_id)
+ continue
+ self.records_by_id[record.memory_id] = record
+ self.slot_history[record.slot_key].append(record.memory_id)
+ self._refresh_subject_depth_head(record)
+ self._link_depth_edges_for_record(record, previous_subject_records)
+ self._link_profile_edges_for_record(record)
+ self._link_facet_edges_for_record(record)
+ stored_ids.append(record.memory_id)
+ continue
+
+ if _is_passive_record_state(record.state):
+ record.metadata["conflict_action"] = _clean_text(record.metadata.get("conflict_action", "")) or f"{record.state}_buffer"
+ self.records_by_id[record.memory_id] = record
+ self.slot_history[record.slot_key].append(record.memory_id)
+ self._refresh_subject_depth_head(record)
+ self._link_depth_edges_for_record(record, previous_subject_records)
+ self._link_profile_edges_for_record(record)
+ self._link_facet_edges_for_record(record)
+ stored_ids.append(record.memory_id)
+ continue
+
+ active_records = self._slot_active_records(record.slot_key)
+ duplicate_target = next(
+ (
+ candidate
+ for candidate in active_records
+ if _normalize(dict(candidate.metadata or {}).get("memory_signature", "")) == _normalize(dict(record.metadata or {}).get("memory_signature", ""))
+ ),
+ None,
+ )
+ if duplicate_target is not None:
+ _merge_record_payload(duplicate_target, record)
+ duplicate_target.metadata["conflict_action"] = "duplicate"
+ stored_ids.append(duplicate_target.memory_id)
+ aggregate_id = self._upsert_profile_aggregate_for_record(duplicate_target)
+ if aggregate_id and aggregate_id not in stored_ids:
+ stored_ids.append(aggregate_id)
+ continue
+
+ state_signature = _normalize(dict(record.metadata or {}).get("state_signature", ""))
+ active_by_state_signature = [
+ candidate
+ for candidate in active_records
+ if _normalize(dict(candidate.metadata or {}).get("state_signature", "")) == state_signature
+ ]
+ if active_records:
+ conflict_action, supersede_targets, conflict_reason = _classify_conflict_action(
+ record,
+ active_records=active_records,
+ active_by_state_signature=active_by_state_signature,
+ )
+ record.metadata["conflict_reason"] = conflict_reason
+ if conflict_action == "supersede":
+ for active_head in supersede_targets:
+ if not _is_active_record_state(active_head.state):
+ continue
+ active_head.state = "superseded"
+ active_head.metadata["superseded_by"] = record.memory_id
+ active_head.metadata["superseded_reason"] = conflict_reason
+ record.supersedes.append(active_head.memory_id)
+ record.state = "active"
+ record.metadata["conflict_action"] = "supersede"
+ else:
+ record.state = "parallel_active"
+ record.metadata["conflict_action"] = "parallel_active"
+ else:
+ record.state = "active"
+ record.metadata["conflict_action"] = "insert"
+
+ self.records_by_id[record.memory_id] = record
+ self.slot_heads[record.slot_key] = record.memory_id
+ self.slot_history[record.slot_key].append(record.memory_id)
+ self._refresh_subject_depth_head(record)
+ self._link_depth_edges_for_record(record, previous_subject_records)
+ self._link_profile_edges_for_record(record)
+ self._link_facet_edges_for_record(record)
+ aggregate_id = self._upsert_profile_aggregate_for_record(record)
+ if aggregate_id and aggregate_id not in stored_ids:
+ stored_ids.append(aggregate_id)
+ for suspect in list(self.records_by_id.values()):
+ if not _is_suspect_record_state(suspect.state):
+ continue
+ if _normalize(suspect.slot_key) != _normalize(record.slot_key):
+ continue
+ suspect_signature = _normalize(dict(suspect.metadata or {}).get("memory_signature", ""))
+ record_signature = _normalize(dict(record.metadata or {}).get("memory_signature", ""))
+ if not suspect_signature or suspect_signature != record_signature:
+ continue
+ suspect.state = "promoted"
+ suspect.metadata["promotion_state"] = "promoted_by_formal_evidence"
+ suspect.metadata["promoted_to_memory_id"] = record.memory_id
+ suspect.metadata["promoted_turn_index"] = int(record.turn_index)
+ stored_ids.append(record.memory_id)
+ return stored_ids
+
+ def _score_record(
+ self,
+ record: SessionMemoryRecordV2,
+ *,
+ query: str,
+ query_tokens: set[str],
+ hints: set[str],
+ history_mode: bool,
+ public_analysis: Dict[str, Any] | None = None,
+ ) -> tuple[float, str, Dict[str, Any] | None]:
+ token_set = record.token_set()
+ overlap = len(query_tokens & token_set) if query_tokens and token_set else 0
+ lexical = overlap / max(1, len(query_tokens | token_set)) if query_tokens and token_set else 0.0
+ base_score = lexical
+ match_reason = ""
+ public_profile: Dict[str, Any] | None = None
+ query_text = _normalize(query)
+ if hints and record.category in hints:
+ base_score += 0.2
+ if hints and "path" in hints and record.relation == "path_edge":
+ base_score += 0.15
+ if record.slot_key and record.slot_key.lower() in query_text:
+ base_score += 0.12
+ canonical_slot_key = _normalize(record.metadata.get("canonical_slot_key", "")) if isinstance(record.metadata, dict) else ""
+ if canonical_slot_key and canonical_slot_key in query_text:
+ base_score += 0.12
+ anchor_tokens = set()
+ for anchor in record.anchor_concepts[:6]:
+ anchor_text = _normalize(anchor)
+ if not anchor_text:
+ continue
+ if anchor_text in query_text:
+ base_score += 0.18
+ anchor_tokens.update(_tokenize(anchor_text))
+ if anchor_tokens and query_tokens:
+ base_score += 0.1 * (len(query_tokens & anchor_tokens) / max(1, len(query_tokens)))
+ value_text = _normalize(record.value)
+ if value_text and value_text in query_text:
+ base_score += 0.16
+ if hints and record.category not in hints and record.category in {"goal", "constraint", "preference", "terminology", "stage_state"}:
+ base_score -= 0.04
+ recency = 1.0 if self.turn_index <= 0 else max(0.0, 1.0 - ((self.turn_index - record.turn_index) / max(1.0, float(self.turn_index))))
+ generic_recency_prior = 0.12 * recency
+ generic_salience_prior = 0.18 * float(record.salience)
+ generic_confidence_prior = 0.12 * float(record.confidence)
+ metadata = record.metadata if isinstance(record.metadata, dict) else {}
+ memory_role = _normalize(metadata.get("memory_role", ""))
+ authority = _normalize(metadata.get("authority", ""))
+ generic_authority_prior = 0.0
+ if memory_role == "user" and authority == "source":
+ generic_authority_prior += 0.18
+ elif memory_role == "assistant" and authority == "promoted":
+ generic_authority_prior += 0.08
+ elif memory_role == "assistant" and authority == "derived":
+ generic_authority_prior -= 0.18
+ source_kind = _normalize(record.source_kind)
+ speaker = _normalize(metadata.get("speaker", ""))
+ resolved_date = _normalize(metadata.get("resolved_date", ""))
+ is_public_benchmark_record = source_kind.startswith("public_dialog") or _normalize(metadata.get("source", "")) == "public_benchmark"
+ score = base_score + generic_recency_prior + generic_salience_prior + generic_confidence_prior + generic_authority_prior
+ if is_public_benchmark_record:
+ analysis = dict(public_analysis or {})
+ public_profile = _public_match_profile(
+ record,
+ analysis,
+ query_text=query_text,
+ query_tokens=query_tokens,
+ token_set=token_set,
+ )
+ temporal_query = bool(public_profile.get("temporal_query", False))
+ exact_speaker_match = bool(public_profile.get("speaker_match", False))
+ semantic_match = bool(public_profile.get("semantic_match", False))
+ target_status_match = bool(public_profile.get("target_status_match", False))
+ time_match = bool(public_profile.get("time_match", False))
+ event_overlap = float(public_profile.get("event_overlap", 0.0) or 0.0)
+ event_focus_coverage = float(public_profile.get("event_focus_coverage", 0.0) or 0.0)
+ event_signal = float(public_profile.get("event_signal", 0.0) or 0.0)
+ subject_match = bool(public_profile.get("subject_match", False))
+ subject_overlap = float(public_profile.get("subject_overlap", 0.0) or 0.0)
+ query_subject_signature = _normalize(public_profile.get("query_subject_signature", ""))
+ public_compatibility = float(public_profile.get("compatibility", 0.0) or 0.0)
+ public_conflict = float(public_profile.get("conflict", 0.0) or 0.0)
+ public_prior_gate = _clamp01(0.12 + (0.96 * public_compatibility) - (0.50 * public_conflict))
+ structured_score = base_score
+ structured_score *= 0.64 + (0.42 * public_compatibility) + (0.12 * event_signal)
+ structured_score += 0.56 * public_compatibility
+ structured_score += 0.16 * event_focus_coverage
+ structured_score += 0.08 * event_overlap
+ if query_subject_signature:
+ structured_score += 0.62 if subject_match else 0.18 * subject_overlap
+ if not subject_match and subject_overlap < 0.55:
+ structured_score -= 0.30
+ if temporal_query:
+ if resolved_date:
+ structured_score += 0.14 * public_compatibility
+ if time_match:
+ structured_score += 0.10 * public_compatibility
+ if record.relation == "event_date":
+ structured_score += 0.10
+ if source_kind == "public_dialog_time":
+ structured_score += 0.08 * max(public_compatibility, event_signal)
+ elif source_kind == "public_dialog_profile":
+ structured_score -= 0.08 * max(0.0, 1.0 - public_compatibility)
+ elif source_kind == "public_dialog_profile" and semantic_match:
+ structured_score += 0.08 * public_compatibility
+ elif source_kind == "public_dialog_event":
+ structured_score += 0.05 * event_signal
+ generic_prior_score = (
+ (generic_recency_prior * (0.18 + (0.22 * public_prior_gate)))
+ + (generic_salience_prior * (0.04 + (0.16 * public_prior_gate)))
+ + (generic_confidence_prior * (0.04 + (0.12 * public_prior_gate)))
+ + (generic_authority_prior * (0.08 + (0.12 * public_prior_gate)))
+ )
+ score = structured_score + generic_prior_score
+ score -= 0.22 * public_conflict
+ if public_compatibility < 0.12:
+ score -= 0.14
+ public_profile = {
+ **dict(public_profile or {}),
+ "prior_gate": public_prior_gate,
+ "structured_score": structured_score,
+ "generic_prior_score": generic_prior_score,
+ "base_score": base_score,
+ }
+ match_reason = _public_match_reason(
+ record,
+ {**analysis, "query": query},
+ speaker_match=exact_speaker_match,
+ semantic_match=semantic_match,
+ subject_match=subject_match,
+ event_overlap=event_overlap,
+ target_status_match=target_status_match,
+ time_match=time_match,
+ )
+ profile_delta, profile_reason = profile_query_score_delta(
+ query=query_text,
+ query_tokens=query_tokens,
+ category=record.category,
+ source_kind=record.source_kind,
+ semantic_slot=metadata.get("semantic_slot", ""),
+ value=record.value,
+ anchors=record.anchor_concepts,
+ metadata=metadata,
+ )
+ if profile_delta > 0:
+ score += profile_delta
+ if profile_reason:
+ match_reason = ",".join(_dedupe([match_reason, profile_reason], max_items=4))
+ if memory_role == "assistant" and canonical_slot_key:
+ source_head = next(iter(self._slot_active_records(canonical_slot_key)), None)
+ if (
+ source_head is not None
+ and isinstance(source_head.metadata, dict)
+ and _normalize(source_head.metadata.get("memory_role", "")) == "user"
+ and _normalize(source_head.metadata.get("authority", "")) == "source"
+ and _normalize(source_head.value) != _normalize(record.value)
+ ):
+ score -= 0.25
+ if history_mode:
+ if _is_active_record_state(record.state):
+ score -= 0.08
+ else:
+ score += 0.26
+ elif _is_active_record_state(record.state):
+ score += 0.24
+ if _normalize(record.state) == "parallel_active":
+ score += 0.06
+ else:
+ score -= 0.25
+ memory_signature_tokens = set(_tokenize(metadata.get("memory_signature", "")))
+ state_signature_tokens = set(_tokenize(metadata.get("state_signature", "")))
+ if query_tokens and memory_signature_tokens:
+ score += 0.12 * (len(query_tokens & memory_signature_tokens) / max(1, len(query_tokens)))
+ if query_tokens and state_signature_tokens:
+ score += 0.08 * (len(query_tokens & state_signature_tokens) / max(1, len(query_tokens)))
+ lowered = _normalize(record.value)
+ if any(marker in lowered for marker in _NEGATION_MARKERS) and not hints.intersection({"constraint", "history"}):
+ score -= 0.08
+ return score, match_reason, public_profile
+
+ def _expanded_selection(
+ self,
+ *,
+ selected: Sequence[tuple[float, SessionMemoryRecordV2]],
+ scored: Sequence[tuple[float, SessionMemoryRecordV2]],
+ query: str,
+ hints: set[str],
+ history_mode: bool,
+ top_k: int,
+ public_analysis: Dict[str, Any] | None = None,
+ public_profiles: Mapping[str, Mapping[str, Any]] | None = None,
+ ) -> List[tuple[float, SessionMemoryRecordV2]]:
+ expanded = list(selected)
+ seen = {record.memory_id for _, record in expanded}
+ query_text = _normalize(query)
+ analysis = dict(public_analysis or {})
+
+ if analysis:
+ cluster_roots = {
+ _public_slot_root(record.slot_key)
+ for _, record in expanded
+ if _clean_text(record.source_kind).startswith("public_dialog")
+ }
+ cluster_signatures = {
+ _normalize(dict(record.metadata or {}).get("event_signature", ""))
+ for _, record in expanded
+ if _clean_text(record.source_kind).startswith("public_dialog")
+ }
+ cluster_signatures.discard("")
+ per_root: Dict[str, int] = defaultdict(int)
+ for _, record in expanded:
+ if _clean_text(record.source_kind).startswith("public_dialog"):
+ per_root[_public_slot_root(record.slot_key)] += 1
+ public_limit = min(max(top_k + 4, top_k), 12)
+ per_event_cap = 3 if bool(analysis.get("is_temporal", False)) else 2
+ for score, record in scored:
+ if record.memory_id in seen or not _clean_text(record.source_kind).startswith("public_dialog"):
+ continue
+ profile = dict((public_profiles or {}).get(record.memory_id, {}) or {})
+ root = _clean_text(profile.get("cluster_root", "")) or _public_slot_root(record.slot_key)
+ signature = _normalize(profile.get("signature", "")) or _normalize(dict(record.metadata or {}).get("event_signature", ""))
+ compatibility = _clamp01(profile.get("compatibility", 0.0))
+ event_focus_coverage = _clamp01(profile.get("event_focus_coverage", profile.get("event_overlap", 0.0)))
+ semantic_match = bool(profile.get("semantic_match", False))
+ time_match = bool(profile.get("time_match", False))
+ cluster_bonus = 0.0
+ if root and root in cluster_roots:
+ cluster_bonus += 0.12
+ if signature and signature in cluster_signatures:
+ cluster_bonus += 0.14
+ expansion_score = compatibility + cluster_bonus + (0.06 if (semantic_match or time_match) else 0.0)
+ if event_focus_coverage >= 0.8:
+ expansion_score += 0.06
+ if expansion_score < 0.18:
+ continue
+ if root and per_root.get(root, 0) >= per_event_cap and expansion_score < 0.34:
+ continue
+ boost = 0.02 + (0.06 * compatibility) + (0.04 * event_focus_coverage)
+ semantic_slot = _normalize(dict(record.metadata or {}).get("semantic_slot", ""))
+ if bool(analysis.get("is_temporal", False)) and semantic_slot == "event_time":
+ boost += 0.05
+ elif _normalize(record.source_kind) in {"public_dialog_event", "public_dialog_profile"}:
+ boost += 0.03
+ expanded.append((score + boost, record))
+ seen.add(record.memory_id)
+ if root:
+ cluster_roots.add(root)
+ per_root[root] += 1
+ if signature:
+ cluster_signatures.add(signature)
+ if len(expanded) >= public_limit:
+ break
+
+ if "path" in hints:
+ frontier = {
+ _clean_text(anchor)
+ for _, record in expanded
+ for anchor in record.anchor_concepts[:6]
+ if _clean_text(anchor)
+ }
+ mentioned = {
+ _clean_text(anchor)
+ for _, record in scored
+ for anchor in record.anchor_concepts[:6]
+ if _clean_text(anchor) and _normalize(anchor) in query_text
+ }
+ frontier.update(mentioned)
+ path_limit = min(max(top_k + 4, top_k), 12)
+ for _ in range(4):
+ added = False
+ for score, record in scored:
+ if record.memory_id in seen or record.relation != "path_edge":
+ continue
+ anchors = {_clean_text(anchor) for anchor in record.anchor_concepts[:6] if _clean_text(anchor)}
+ if frontier and not frontier.intersection(anchors):
+ continue
+ expanded.append((score + 0.03, record))
+ seen.add(record.memory_id)
+ frontier.update(anchors)
+ added = True
+ if len(expanded) >= path_limit:
+ break
+ if not added or len(expanded) >= path_limit:
+ break
+
+ if history_mode or _query_requests_current_pair_clean(query):
+ pair_limit = min(max(top_k + 2, top_k), 10)
+ for score, record in list(expanded):
+ if not _is_active_record_state(record.state):
+ for current in self._slot_active_records(record.slot_key)[:2]:
+ if current.memory_id in seen or current.value == record.value:
+ continue
+ expanded.append((max(0.05, score - 0.01), current))
+ seen.add(current.memory_id)
+ if len(expanded) >= pair_limit:
+ break
+ if _is_active_record_state(record.state):
+ for previous_id in record.supersedes[:2]:
+ previous = self.records_by_id.get(previous_id)
+ if previous and previous.memory_id not in seen:
+ expanded.append((max(0.05, score - 0.03), previous))
+ seen.add(previous.memory_id)
+ if len(expanded) >= pair_limit:
+ break
+ for parallel in self._slot_active_records(record.slot_key):
+ if parallel.memory_id in seen or parallel.memory_id == record.memory_id:
+ continue
+ expanded.append((max(0.05, score - 0.015), parallel))
+ seen.add(parallel.memory_id)
+ if len(expanded) >= pair_limit:
+ break
+ if len(expanded) >= pair_limit:
+ break
+
+ expanded.sort(key=lambda item: (item[0], _is_active_record_state(item[1].state), item[1].turn_index), reverse=True)
+ selection_limit = min(
+ max(
+ top_k
+ + (
+ 4
+ if "path" in hints
+ else 3
+ if analysis
+ else 2
+ if (history_mode or _query_requests_current_pair_clean(query))
+ else 0
+ ),
+ top_k,
+ ),
+ 12,
+ )
+ return expanded[:selection_limit]
+
+ def _noise_false_candidates(
+ self,
+ *,
+ query: str,
+ query_tokens: set[str],
+ top_k: int,
+ ) -> List[tuple[float, SessionMemoryRecordV2]]:
+ candidates: List[tuple[float, SessionMemoryRecordV2]] = []
+ if not _query_requests_inactive_history(query):
+ return candidates
+ for turn in reversed(self.turn_log):
+ if _normalize(turn.get("kind", "")) != "noise":
+ continue
+ text = _clean_text(turn.get("text", ""))
+ if not text:
+ continue
+ value = _extract_noise_value(text)
+ if not value:
+ continue
+ turn_index = int(turn.get("turn_index", 0) or 0)
+ value_tokens = set(_tokenize(value))
+ overlap = len(query_tokens & value_tokens) / max(1, len(query_tokens | value_tokens)) if query_tokens or value_tokens else 0.0
+ recency = 1.0 if self.turn_index <= 0 else max(0.0, 1.0 - ((self.turn_index - turn_index) / max(1.0, float(self.turn_index))))
+ score = 1.28 + (0.08 * overlap) + (0.1 * recency)
+ anchor = next((token for token in _tokenize(value) if token not in {"should", "not", "become", "active", "requirement", "noise"}), "noise")
+ record = SessionMemoryRecordV2(
+ memory_id=f"noise.turn.{turn_index}",
+ category="noise",
+ slot_key=f"noise.turn.{turn_index}",
+ value=value,
+ relation="inactive_note",
+ anchor_concepts=_dedupe([anchor, "noise"], max_items=4),
+ evidence_anchors=_dedupe([anchor, "noise"], max_items=4),
+ salience=0.92,
+ confidence=0.88,
+ source_kind="noise_turn",
+ turn_index=turn_index,
+ state="false",
+ metadata={"turn_kind": "noise", "inactive_candidate": True, "source_text": text},
+ )
+ candidates.append((round(score, 6), record))
+ if len(candidates) >= max(1, top_k):
+ break
+ return candidates
+
+ def retrieve(self, query: str, *, top_k: int = 6) -> Dict[str, Any]:
+ query_text = _clean_text(query)
+ query_tokens = set(_tokenize(query_text))
+ hints = set(infer_category_hints(query_text))
+ history_mode = infer_history_mode_clean(query_text)
+ public_analysis = _public_query_analysis(query_text, query_tokens)
+ scored: List[tuple[float, SessionMemoryRecordV2]] = []
+ suppressed: List[SessionMemoryRecordV2] = []
+ match_reasons: Dict[str, str] = {}
+ public_profiles: Dict[str, Dict[str, Any]] = {}
+ for record in self.records_by_id.values():
+ if _normalize(dict(record.metadata or {}).get("memory_layer", "")) == "slow":
+ continue
+ score, match_reason, public_profile = self._score_record(
+ record,
+ query=query_text,
+ query_tokens=query_tokens,
+ hints=hints,
+ history_mode=history_mode,
+ public_analysis=public_analysis,
+ )
+ if score <= 0:
+ continue
+ if not _is_active_record_state(record.state) and not history_mode:
+ suppressed.append(record)
+ continue
+ scored.append((score, record))
+ if match_reason:
+ match_reasons[record.memory_id] = match_reason
+ if public_profile:
+ public_profiles[record.memory_id] = dict(public_profile)
+ if _query_requests_inactive_history(query_text):
+ scored.extend(self._noise_false_candidates(query=query_text, query_tokens=query_tokens, top_k=max(1, top_k)))
+ if not scored:
+ fast_records = [
+ record
+ for record in self.records_by_id.values()
+ if _normalize(dict(record.metadata or {}).get("memory_layer", "")) != "slow"
+ ]
+ for record in sorted(fast_records, key=lambda item: item.turn_index, reverse=True)[: top_k * 2]:
+ fallback_score = 0.05 + (0.2 if _is_active_record_state(record.state) else 0.0)
+ if not _is_active_record_state(record.state) and not history_mode:
+ suppressed.append(record)
+ continue
+ scored.append((fallback_score, record))
+ scored.sort(key=lambda item: (item[0], _is_active_record_state(item[1].state), item[1].turn_index), reverse=True)
+ if history_mode:
+ history_scored = [item for item in scored if not _is_active_record_state(item[1].state)]
+ active_scored = [item for item in scored if _is_active_record_state(item[1].state)]
+ selected = [*history_scored[: max(1, top_k)], *active_scored[: max(0, top_k - len(history_scored[: max(1, top_k)]))]]
+ selected = selected[: max(1, top_k)]
+ else:
+ selected = scored[: max(1, top_k)]
+ selected = self._expanded_selection(
+ selected=selected,
+ scored=scored,
+ query=query_text,
+ hints=hints,
+ history_mode=history_mode,
+ top_k=max(1, top_k),
+ public_analysis=public_analysis,
+ public_profiles=public_profiles,
+ )
+ seed_memory_ids = [record.memory_id for _, record in selected]
+ depth_chain = self.depth_chain_for_query(
+ query_text,
+ seed_memory_ids=seed_memory_ids,
+ top_k=max(3, min(8, top_k)),
+ )
+ if depth_chain.get("enabled") and depth_chain.get("nodes"):
+ seen_selected = {record.memory_id for _, record in selected}
+ chain_limit = min(max(top_k + 4, top_k), 12)
+ for node in depth_chain.get("nodes", []) or []:
+ memory_id = _clean_text(dict(node).get("memory_id", ""))
+ record = self.records_by_id.get(memory_id)
+ if (
+ record is None
+ or memory_id in seen_selected
+ or _normalize(dict(record.metadata or {}).get("memory_layer", "")) == "slow"
+ ):
+ continue
+ layer = _normalize(dict(record.metadata or {}).get("depth_layer", "")) or "core_view"
+ chain_score = 0.58 + (0.04 if _is_active_record_state(record.state) else 0.0)
+ chain_score += max(0.0, 0.12 - (0.012 * _DEPTH_LAYER_ORDER.get(layer, 8)))
+ selected.append((round(chain_score, 6), record))
+ seen_selected.add(memory_id)
+ if len(selected) >= chain_limit:
+ break
+ active_hits = [record for _, record in selected if _is_active_record_state(record.state)]
+ history_hits = [record for _, record in selected if not _is_active_record_state(record.state) and record.state != "false"]
+ overwrite_hits = [record for record in history_hits if record.state == "superseded"]
+ stale_hits = overwrite_hits if not history_mode else []
+ false_hits: List[SessionMemoryRecordV2] = [record for _, record in selected if record.state == "false"]
+
+ concepts: Dict[str, Dict[str, Any]] = {}
+ relations: List[Dict[str, Any]] = []
+ seen_relations = set()
+ selected_payload: List[Dict[str, Any]] = []
+ for score, record in selected:
+ record_payload = record.to_dict()
+ payload_metadata = dict(record_payload.get("metadata", {}) or {})
+ payload_metadata["match_reason"] = match_reasons.get(record.memory_id, "")
+ if record.memory_id in public_profiles:
+ profile = dict(public_profiles.get(record.memory_id, {}) or {})
+ payload_metadata["public_match_compatibility"] = round(float(profile.get("compatibility", 0.0) or 0.0), 6)
+ payload_metadata["public_match_conflict"] = round(float(profile.get("conflict", 0.0) or 0.0), 6)
+ payload_metadata["public_event_focus_coverage"] = round(float(profile.get("event_focus_coverage", 0.0) or 0.0), 6)
+ payload_metadata["public_event_signal"] = round(float(profile.get("event_signal", 0.0) or 0.0), 6)
+ payload_metadata["public_subject_match"] = bool(profile.get("subject_match", False))
+ payload_metadata["public_subject_overlap"] = round(float(profile.get("subject_overlap", 0.0) or 0.0), 6)
+ payload_metadata["public_prior_gate"] = round(float(profile.get("prior_gate", 0.0) or 0.0), 6)
+ payload_metadata["public_structured_score"] = round(float(profile.get("structured_score", 0.0) or 0.0), 6)
+ payload_metadata["public_generic_prior_score"] = round(float(profile.get("generic_prior_score", 0.0) or 0.0), 6)
+ payload_metadata["public_base_score"] = round(float(profile.get("base_score", 0.0) or 0.0), 6)
+ record_payload["metadata"] = payload_metadata
+ selected_payload.append(
+ {
+ "score": round(float(score), 6),
+ **record_payload,
+ }
+ )
+ concepts.setdefault(record.value, {"concept": record.value, "type": record.category, "source_kind": record.source_kind})
+ slot_node = f"slot::{record.slot_key}"
+ concepts.setdefault(slot_node, {"concept": slot_node, "type": "slot_head", "source_kind": "memory_slot"})
+ slot_relation = _slot_relation_for_state(record.state)
+ rel_key = (slot_node, record.value, slot_relation)
+ if rel_key not in seen_relations:
+ seen_relations.add(rel_key)
+ relations.append(
+ {
+ "from": slot_node,
+ "to": record.value,
+ "relation": slot_relation,
+ "weight": round(max(0.3, min(0.98, 0.5 + score * 0.4)), 6),
+ "source_kind": record.source_kind,
+ "memory_id": record.memory_id,
+ }
+ )
+ for anchor in record.anchor_concepts[:4]:
+ concepts.setdefault(anchor, {"concept": anchor, "type": "concept", "source_kind": record.source_kind})
+ anchor_key = (anchor, record.value, record.relation)
+ if anchor_key in seen_relations:
+ continue
+ seen_relations.add(anchor_key)
+ relations.append(
+ {
+ "from": anchor,
+ "to": record.value,
+ "relation": record.relation,
+ "weight": round(max(0.32, min(0.98, 0.42 + score * 0.4)), 6),
+ "source_kind": record.source_kind,
+ "memory_id": record.memory_id,
+ }
+ )
+ if history_mode:
+ for previous_id in record.supersedes:
+ previous = self.records_by_id.get(previous_id)
+ if not previous:
+ continue
+ concepts.setdefault(previous.value, {"concept": previous.value, "type": previous.category, "source_kind": previous.source_kind})
+ relation_key = (record.value, previous.value, "supersedes")
+ if relation_key not in seen_relations:
+ seen_relations.add(relation_key)
+ relations.append(
+ {
+ "from": record.value,
+ "to": previous.value,
+ "relation": "supersedes",
+ "weight": 0.91,
+ "source_kind": record.source_kind,
+ "memory_id": record.memory_id,
+ }
+ )
+
+ query_id = f"query:{int(self.audit_event_totals.get('retrieval_log', 0) or 0) + 1}"
+ self._append_audit_event(
+ "retrieval_log",
+ {
+ "query_id": query_id,
+ "turn_index": int(self.turn_index),
+ "query": query_text,
+ "history_mode": history_mode,
+ "memory_ids": [record["memory_id"] for record in selected_payload],
+ },
+ )
+ return {
+ "query_id": query_id,
+ "concepts": list(concepts.values()),
+ "relations": relations,
+ "hits": selected_payload,
+ "active_hits": [record.to_dict() for record in active_hits],
+ "history_hits": [record.to_dict() for record in history_hits],
+ "stale_hits": [record.to_dict() for record in stale_hits],
+ "overwrite_hits": [record.to_dict() for record in overwrite_hits],
+ "false_hits": [record.to_dict() for record in false_hits],
+ "suppressed_hits": [record.to_dict() for record in sorted(suppressed, key=lambda item: item.turn_index, reverse=True)[:top_k]],
+ "context_token_estimate": sum(_estimate_tokens(item["value"]) + sum(_estimate_tokens(anchor) for anchor in item.get("anchor_concepts", [])) for item in selected_payload),
+ "metadata": {
+ "history_mode": history_mode,
+ "inactive_history_mode": _query_requests_inactive_history(query_text),
+ "slot_heads": len(self.slot_heads),
+ "records": len(self.records_by_id),
+ "noise_turn_count": self.noise_turn_count,
+ "public_query_analysis": public_analysis,
+ "memory_chain": depth_chain,
+ "memory_chain_enabled": bool(depth_chain.get("enabled", False)),
+ "memory_chain_node_count": int(depth_chain.get("node_count", 0) or 0),
+ "memory_chain_edge_count": int(depth_chain.get("edge_count", 0) or 0),
+ },
+ }
+
+ def register_answer_support(self, *, answer_id: str, memory_ids: Sequence[str], query_id: str = "", answer_text: str = "") -> None:
+ self._append_audit_event(
+ "answer_support_log",
+ {
+ "answer_id": answer_id or f"answer:{int(self.audit_event_totals.get('answer_support_log', 0) or 0) + 1}",
+ "query_id": query_id,
+ "memory_ids": list(memory_ids),
+ "answer_text": _clean_text(answer_text),
+ "turn_index": int(self.turn_index),
+ },
+ )
+
+ def summary(self) -> Dict[str, Any]:
+ active_records = [record for record in self.records_by_id.values() if _is_active_record_state(record.state)]
+ parallel_active = [record for record in active_records if _normalize(record.state) == "parallel_active"]
+ superseded = [record for record in self.records_by_id.values() if record.state == "superseded"]
+ suspect_records = [record for record in self.records_by_id.values() if _is_suspect_record_state(record.state)]
+ promoted_suspect_records = [record for record in self.records_by_id.values() if _normalize(record.state) == "promoted"]
+ return {
+ "turn_index": int(self.turn_index),
+ "records": len(self.records_by_id),
+ "memory_edges": len(self.memory_edges),
+ "subject_depth_heads": sum(len(heads) for heads in self.subject_depth_heads.values()),
+ "active_slots": len(self.slot_heads),
+ "active_records": len(active_records),
+ "parallel_active_records": len(parallel_active),
+ "superseded_records": len(superseded),
+ "suspect_records": len(suspect_records),
+ "promoted_suspect_records": len(promoted_suspect_records),
+ "noise_turn_count": self.noise_turn_count,
+ "turn_events": len(self.turn_log),
+ "retrieval_events": len(self.retrieval_log),
+ "answer_support_events": len(self.answer_support_log),
+ "audit_turn_events": int(self.audit_event_totals.get("turn_log", 0) or 0),
+ "audit_retrieval_events": int(self.audit_event_totals.get("retrieval_log", 0) or 0),
+ "audit_answer_support_events": int(self.audit_event_totals.get("answer_support_log", 0) or 0),
+ "audit_trimmed_counts": dict(self.audit_trimmed_counts),
+ "audit_retention": int(self.audit_retention),
+ "persistence_backend": self.persistence_backend,
+ "persistence_path": self.persistence_path,
+ }
+
+ def to_dict(self, *, mode: str = "full") -> Dict[str, Any]:
+ payload = self._core_payload()
+ normalized_mode = _normalize(mode) or "full"
+ if normalized_mode == "light":
+ payload["audit"] = {
+ "retained": {
+ "turn_log": len(self.turn_log),
+ "retrieval_log": len(self.retrieval_log),
+ "answer_support_log": len(self.answer_support_log),
+ },
+ "totals": dict(self.audit_event_totals),
+ "trimmed": dict(self.audit_trimmed_counts),
+ "audit_retention": int(self.audit_retention),
+ }
+ return payload
+ payload.update(self._audit_payload())
+ return payload
+
+ def export_graph(self, *, snapshot_points: Sequence[int] | None = None, mode: str = "full") -> Dict[str, Any]:
+ snapshot_set = {int(point) for point in (snapshot_points or []) if int(point) > 0}
+ nodes: List[Dict[str, Any]] = []
+ edges: List[Dict[str, Any]] = []
+ node_ids = set()
+ edge_ids = set()
+
+ def add_node(node_id: str, node_type: str, **payload: Any) -> None:
+ if node_id in node_ids:
+ return
+ node_ids.add(node_id)
+ nodes.append({"id": node_id, "type": node_type, **payload})
+
+ def add_edge(source: str, target: str, relation: str, **payload: Any) -> None:
+ key = (source, target, relation)
+ if key in edge_ids:
+ return
+ edge_ids.add(key)
+ edges.append({"from": source, "to": target, "relation": relation, **payload})
+
+ for event in self.turn_log:
+ turn_id = event["turn_id"]
+ add_node(
+ turn_id,
+ "turn",
+ turn_index=event["turn_index"],
+ kind=event["kind"],
+ text=event["text"],
+ speaker=event.get("speaker", "user"),
+ assistant_text=event.get("assistant_text", ""),
+ writeback_class=event.get("writeback_class", ""),
+ metadata=dict(event.get("metadata", {}) or {}),
+ )
+ for record_id in event.get("record_ids", []) or []:
+ add_edge(turn_id, record_id, "writes")
+
+ for record in self.records_by_id.values():
+ add_node(
+ record.memory_id,
+ "memory_record",
+ category=record.category,
+ slot_key=record.slot_key,
+ value=record.value,
+ state=record.state,
+ turn_index=record.turn_index,
+ depth_layer=_normalize(dict(record.metadata or {}).get("depth_layer", "")) or "core_view",
+ subject_signature=self._record_subject_signature(record),
+ )
+ slot_node = f"slot::{record.slot_key}"
+ add_node(slot_node, "slot_head", slot_key=record.slot_key)
+ add_edge(slot_node, record.memory_id, _slot_relation_for_state(record.state))
+ for previous_id in record.supersedes:
+ add_edge(record.memory_id, previous_id, "supersedes")
+ for anchor in record.anchor_concepts:
+ concept_id = f"concept::{anchor}"
+ add_node(concept_id, "concept", concept=anchor)
+ add_edge(record.memory_id, concept_id, "anchors_to")
+
+ for edge in self.memory_edges.values():
+ if edge.source_memory_id in self.records_by_id and edge.target_memory_id in self.records_by_id:
+ add_edge(
+ edge.source_memory_id,
+ edge.target_memory_id,
+ edge.edge_type,
+ weight=round(float(edge.score), 6),
+ edge_id=edge.edge_id,
+ metadata=dict(edge.metadata or {}),
+ )
+
+ for event in self.retrieval_log:
+ query_id = event["query_id"]
+ add_node(query_id, "query", query=event["query"], turn_index=event["turn_index"], history_mode=event["history_mode"])
+ for record_id in event.get("memory_ids", []) or []:
+ add_edge(record_id, query_id, "retrieved_by")
+
+ for event in self.answer_support_log:
+ answer_id = event["answer_id"]
+ add_node(answer_id, "answer", answer_text=event["answer_text"], turn_index=event["turn_index"])
+ if event.get("query_id"):
+ add_edge(event["query_id"], answer_id, "answered_as")
+ for record_id in event.get("memory_ids", []) or []:
+ add_edge(record_id, answer_id, "supports_answer")
+
+ snapshots: List[Dict[str, Any]] = []
+ if snapshot_set:
+ turn_events = sorted(self.turn_log, key=lambda item: item["turn_index"])
+ for point in sorted(snapshot_set):
+ visible_turns = [event for event in turn_events if int(event["turn_index"]) <= point]
+ visible_turn_ids = {event["turn_id"] for event in visible_turns}
+ visible_record_ids = {record_id for event in visible_turns for record_id in event.get("record_ids", []) or []}
+ active_records = [
+ record.to_dict()
+ for record in self.records_by_id.values()
+ if record.turn_index <= point and (_is_active_record_state(record.state) or record.memory_id in visible_record_ids)
+ ]
+ snapshots.append(
+ {
+ "turn_index": point,
+ "visible_turns": len(visible_turns),
+ "visible_records": len(active_records),
+ "slot_heads": {
+ slot: record_id
+ for slot, record_id in self.slot_heads.items()
+ if self.records_by_id.get(record_id) and self.records_by_id[record_id].turn_index <= point
+ },
+ "records": active_records,
+ }
+ )
+
+ return {
+ "summary": {
+ **self.summary(),
+ "graph_nodes": len(nodes),
+ "graph_edges": len(edges),
+ },
+ "nodes": nodes,
+ "edges": edges,
+ "snapshots": snapshots if _normalize(mode) != "light" else [],
+ }
+
+ def export_mermaid(self, *, max_records: int = 96) -> str:
+ lines = ["graph TD"]
+ records = sorted(self.records_by_id.values(), key=lambda item: (not _is_active_record_state(item.state), -item.turn_index))[:max_records]
+ for record in records:
+ record_id = record.memory_id.replace(":", "_").replace(".", "_").replace("-", "_")
+ slot_id = f"slot_{record.slot_key}".replace(":", "_").replace(".", "_").replace("-", "_")
+ lines.append(f' {slot_id}["{record.slot_key}"]')
+ lines.append(f' {record_id}["{record.category}: {record.value[:48]}"]')
+ relation = _slot_relation_for_state(record.state)
+ lines.append(f" {slot_id} -->|{relation}| {record_id}")
+ for previous_id in record.supersedes[:2]:
+ prev = previous_id.replace(":", "_").replace(".", "_").replace("-", "_")
+ lines.append(f" {record_id} -->|supersedes| {prev}")
+ for anchor in record.anchor_concepts[:2]:
+ anchor_id = f"concept_{anchor}".replace(":", "_").replace(".", "_").replace("-", "_").replace(" ", "_")
+ lines.append(f' {anchor_id}["{anchor[:36]}"]')
+ lines.append(f" {record_id} -->|anchors_to| {anchor_id}")
+ return "\n".join(lines) + "\n"
+
+ def storage_bytes(self) -> int:
+ return len(json.dumps(self.to_dict(mode="full"), ensure_ascii=False).encode("utf-8"))
+
+
+class StaleGraphSnapshotError(RuntimeError):
+ pass
+
+
+class SQLiteSessionMemoryStore:
+ def __init__(self, storage_path: str | Path, *, audit_retention: int = 256) -> None:
+ self.storage_path = Path(storage_path).expanduser().resolve()
+ self.audit_retention = max(1, int(audit_retention))
+ self.storage_path.parent.mkdir(parents=True, exist_ok=True)
+ self._init_schema()
+
+ def _connect(self) -> sqlite3.Connection:
+ connection = sqlite3.connect(str(self.storage_path))
+ connection.row_factory = sqlite3.Row
+ connection.execute("PRAGMA journal_mode=WAL")
+ connection.execute("PRAGMA synchronous=NORMAL")
+ return connection
+
+ @contextmanager
+ def _managed_connection(self) -> Iterable[sqlite3.Connection]:
+ connection = self._connect()
+ try:
+ with connection:
+ yield connection
+ finally:
+ connection.close()
+
+ def _init_schema(self) -> None:
+ with self._managed_connection() as connection:
+ connection.executescript(
+ """
+ CREATE TABLE IF NOT EXISTS records (
+ scope_id TEXT NOT NULL,
+ memory_id TEXT NOT NULL,
+ category TEXT NOT NULL,
+ slot_key TEXT NOT NULL,
+ value TEXT NOT NULL,
+ relation TEXT NOT NULL,
+ anchor_concepts_json TEXT NOT NULL,
+ evidence_anchors_json TEXT NOT NULL,
+ salience REAL NOT NULL,
+ confidence REAL NOT NULL,
+ source_kind TEXT NOT NULL,
+ turn_index INTEGER NOT NULL,
+ state TEXT NOT NULL,
+ supersedes_json TEXT NOT NULL,
+ metadata_json TEXT NOT NULL,
+ PRIMARY KEY (scope_id, memory_id)
+ );
+ CREATE INDEX IF NOT EXISTS idx_records_scope_slot ON records(scope_id, slot_key);
+ CREATE INDEX IF NOT EXISTS idx_records_scope_turn ON records(scope_id, turn_index);
+ CREATE TABLE IF NOT EXISTS slot_heads (
+ scope_id TEXT NOT NULL,
+ slot_key TEXT NOT NULL,
+ memory_id TEXT NOT NULL,
+ PRIMARY KEY (scope_id, slot_key)
+ );
+ CREATE TABLE IF NOT EXISTS slot_history (
+ scope_id TEXT NOT NULL,
+ slot_key TEXT NOT NULL,
+ ordinal INTEGER NOT NULL,
+ memory_id TEXT NOT NULL,
+ PRIMARY KEY (scope_id, slot_key, ordinal)
+ );
+ CREATE TABLE IF NOT EXISTS memory_edges (
+ scope_id TEXT NOT NULL,
+ edge_id TEXT NOT NULL,
+ source_memory_id TEXT NOT NULL,
+ target_memory_id TEXT NOT NULL,
+ edge_type TEXT NOT NULL,
+ score REAL NOT NULL,
+ model_score REAL NOT NULL,
+ evidence_turn INTEGER NOT NULL,
+ evidence TEXT NOT NULL,
+ metadata_json TEXT NOT NULL,
+ PRIMARY KEY (scope_id, edge_id)
+ );
+ CREATE INDEX IF NOT EXISTS idx_memory_edges_scope_source ON memory_edges(scope_id, source_memory_id);
+ CREATE INDEX IF NOT EXISTS idx_memory_edges_scope_target ON memory_edges(scope_id, target_memory_id);
+ CREATE TABLE IF NOT EXISTS subject_depth_heads (
+ scope_id TEXT NOT NULL,
+ subject_signature TEXT NOT NULL,
+ depth_layer TEXT NOT NULL,
+ memory_id TEXT NOT NULL,
+ PRIMARY KEY (scope_id, subject_signature, depth_layer)
+ );
+ CREATE TABLE IF NOT EXISTS audit_turn_log (
+ scope_id TEXT NOT NULL,
+ event_index INTEGER NOT NULL,
+ payload_json TEXT NOT NULL,
+ PRIMARY KEY (scope_id, event_index)
+ );
+ CREATE TABLE IF NOT EXISTS audit_retrieval_log (
+ scope_id TEXT NOT NULL,
+ event_index INTEGER NOT NULL,
+ payload_json TEXT NOT NULL,
+ PRIMARY KEY (scope_id, event_index)
+ );
+ CREATE TABLE IF NOT EXISTS audit_answer_support (
+ scope_id TEXT NOT NULL,
+ event_index INTEGER NOT NULL,
+ payload_json TEXT NOT NULL,
+ PRIMARY KEY (scope_id, event_index)
+ );
+ CREATE TABLE IF NOT EXISTS meta (
+ scope_id TEXT NOT NULL,
+ key TEXT NOT NULL,
+ value_json TEXT NOT NULL,
+ PRIMARY KEY (scope_id, key)
+ );
+ """
+ )
+
+ def clear_scope(self, scope_id: str) -> None:
+ normalized_scope = _clean_text(scope_id)
+ if not normalized_scope:
+ return
+ with self._managed_connection() as connection:
+ for table in (
+ "records",
+ "slot_heads",
+ "slot_history",
+ "memory_edges",
+ "subject_depth_heads",
+ "audit_turn_log",
+ "audit_retrieval_log",
+ "audit_answer_support",
+ "meta",
+ ):
+ connection.execute(f"DELETE FROM {table} WHERE scope_id = ?", (normalized_scope,))
+
+ def _refresh_authoritative_slow_state(
+ self,
+ connection: sqlite3.Connection,
+ scope_id: str,
+ graph: SessionMemoryGraphV2,
+ ) -> None:
+ """Keep control-plane slow revisions authoritative over stale graph snapshots."""
+ record_rows = connection.execute(
+ """
+ SELECT memory_id, category, slot_key, value, relation, anchor_concepts_json,
+ evidence_anchors_json, salience, confidence, source_kind, turn_index,
+ state, supersedes_json, metadata_json
+ FROM records WHERE scope_id = ?
+ """,
+ (scope_id,),
+ ).fetchall()
+ slow_records: Dict[str, SessionMemoryRecordV2] = {}
+ slow_slots: set[str] = set()
+ for row in record_rows:
+ metadata = dict(json.loads(row["metadata_json"]) or {})
+ if (
+ _normalize(metadata.get("memory_layer")) != "slow"
+ or _normalize(metadata.get("content_variant"))
+ != "slow_memory_capsule"
+ ):
+ continue
+ record = SessionMemoryRecordV2(
+ memory_id=str(row["memory_id"]),
+ category=str(row["category"]),
+ slot_key=str(row["slot_key"]),
+ value=str(row["value"]),
+ relation=str(row["relation"]),
+ anchor_concepts=[
+ str(item) for item in json.loads(row["anchor_concepts_json"]) or []
+ ],
+ evidence_anchors=[
+ str(item) for item in json.loads(row["evidence_anchors_json"]) or []
+ ],
+ salience=float(row["salience"] or 0.0),
+ confidence=float(row["confidence"] or 0.0),
+ source_kind=str(row["source_kind"]),
+ turn_index=int(row["turn_index"] or 0),
+ state=str(row["state"]),
+ supersedes=[
+ str(item) for item in json.loads(row["supersedes_json"]) or []
+ ],
+ metadata=metadata,
+ )
+ slow_records[record.memory_id] = record
+ slow_slots.add(record.slot_key)
+
+ graph.records_by_id = {
+ memory_id: record
+ for memory_id, record in graph.records_by_id.items()
+ if _normalize(dict(record.metadata or {}).get("memory_layer")) != "slow"
+ }
+ graph.records_by_id.update(slow_records)
+ for slot_key in list(graph.slot_heads):
+ if slot_key in slow_slots or _normalize(slot_key).startswith("slow."):
+ graph.slot_heads.pop(slot_key, None)
+ for slot_key in list(graph.slot_history):
+ if slot_key in slow_slots or _normalize(slot_key).startswith("slow."):
+ graph.slot_history.pop(slot_key, None)
+ if slow_slots:
+ head_rows = connection.execute(
+ "SELECT slot_key,memory_id FROM slot_heads WHERE scope_id=?",
+ (scope_id,),
+ ).fetchall()
+ for row in head_rows:
+ if str(row["slot_key"]) in slow_slots:
+ graph.slot_heads[str(row["slot_key"])] = str(row["memory_id"])
+ history_rows = connection.execute(
+ "SELECT slot_key,memory_id FROM slot_history WHERE scope_id=? ORDER BY slot_key,ordinal",
+ (scope_id,),
+ ).fetchall()
+ for row in history_rows:
+ if str(row["slot_key"]) in slow_slots:
+ graph.slot_history[str(row["slot_key"])].append(
+ str(row["memory_id"])
+ )
+
+ graph.memory_edges = {
+ edge_id: edge
+ for edge_id, edge in graph.memory_edges.items()
+ if _normalize(dict(edge.metadata or {}).get("edge_source"))
+ != "slow_graph_control_plane"
+ }
+ edge_rows = connection.execute(
+ """
+ SELECT edge_id,source_memory_id,target_memory_id,edge_type,score,model_score,
+ evidence_turn,evidence,metadata_json
+ FROM memory_edges WHERE scope_id=?
+ """,
+ (scope_id,),
+ ).fetchall()
+ for row in edge_rows:
+ metadata = dict(json.loads(row["metadata_json"]) or {})
+ if _normalize(metadata.get("edge_source")) != "slow_graph_control_plane":
+ continue
+ edge = SessionMemoryEdgeV2(
+ edge_id=str(row["edge_id"]),
+ source_memory_id=str(row["source_memory_id"]),
+ target_memory_id=str(row["target_memory_id"]),
+ edge_type=str(row["edge_type"]),
+ score=float(row["score"] or 0.0),
+ model_score=float(row["model_score"] or 0.0),
+ evidence_turn=int(row["evidence_turn"] or 0),
+ evidence=str(row["evidence"]),
+ metadata=metadata,
+ )
+ graph.memory_edges[edge.edge_id] = edge
+
+ def append_audit_event(
+ self,
+ scope_id: str,
+ field_name: str,
+ payload: Mapping[str, Any],
+ *,
+ idempotency_key: str = "",
+ ) -> Dict[str, Any]:
+ """Append one audit event without rewriting the graph snapshot."""
+ normalized_scope = _clean_text(scope_id)
+ audit_fields = {
+ "turn_log": (
+ "audit_turn_log",
+ "audit_turn_events",
+ "audit_trimmed_turn_log",
+ ),
+ "retrieval_log": (
+ "audit_retrieval_log",
+ "audit_retrieval_events",
+ "audit_trimmed_retrieval_log",
+ ),
+ "answer_support_log": (
+ "audit_answer_support",
+ "audit_answer_support_events",
+ "audit_trimmed_answer_support_log",
+ ),
+ }
+ if not normalized_scope:
+ raise ValueError("scope_id must be non-empty for SQLite persistence")
+ if field_name not in audit_fields:
+ raise KeyError(f"Unknown audit field: {field_name}")
+ table, total_key, trimmed_key = audit_fields[field_name]
+ stored_payload = dict(payload)
+ normalized_idempotency_key = _clean_text(idempotency_key)
+ payload_idempotency_key = _clean_text(stored_payload.get("idempotency_key"))
+ if (
+ normalized_idempotency_key
+ and payload_idempotency_key
+ and normalized_idempotency_key != payload_idempotency_key
+ ):
+ raise ValueError("audit payload idempotency_key conflicts with argument")
+ if normalized_idempotency_key:
+ stored_payload["idempotency_key"] = normalized_idempotency_key
+
+ with self._managed_connection() as connection:
+ connection.execute("BEGIN IMMEDIATE")
+
+ def meta_int(key: str) -> int:
+ row = connection.execute(
+ "SELECT value_json FROM meta WHERE scope_id=? AND key=?",
+ (normalized_scope, key),
+ ).fetchone()
+ return int(json.loads(row["value_json"]) or 0) if row else 0
+
+ if normalized_idempotency_key:
+ for existing in connection.execute(
+ f"SELECT payload_json FROM {table} WHERE scope_id=? ORDER BY event_index",
+ (normalized_scope,),
+ ):
+ existing_payload = dict(json.loads(existing["payload_json"]) or {})
+ if (
+ _clean_text(existing_payload.get("idempotency_key"))
+ == normalized_idempotency_key
+ ):
+ return {
+ "payload": existing_payload,
+ "event_total": meta_int(total_key),
+ "trimmed_total": meta_int(trimmed_key),
+ "appended": False,
+ }
+
+ event_total = meta_int(total_key) + 1
+ if field_name == "retrieval_log":
+ stored_payload["query_id"] = f"query:{event_total}"
+ event_index = int(
+ connection.execute(
+ f"SELECT COALESCE(MAX(event_index),-1)+1 FROM {table} "
+ "WHERE scope_id=?",
+ (normalized_scope,),
+ ).fetchone()[0]
+ )
+ connection.execute(
+ f"INSERT INTO {table}(scope_id,event_index,payload_json) VALUES(?,?,?)",
+ (
+ normalized_scope,
+ event_index,
+ json.dumps(stored_payload, ensure_ascii=False),
+ ),
+ )
+ retained = int(
+ connection.execute(
+ f"SELECT COUNT(*) FROM {table} WHERE scope_id=?",
+ (normalized_scope,),
+ ).fetchone()[0]
+ )
+ overflow = max(0, retained - self.audit_retention)
+ if overflow:
+ expired = connection.execute(
+ f"SELECT event_index FROM {table} WHERE scope_id=? "
+ "ORDER BY event_index LIMIT ?",
+ (normalized_scope, overflow),
+ ).fetchall()
+ connection.executemany(
+ f"DELETE FROM {table} WHERE scope_id=? AND event_index=?",
+ [(normalized_scope, int(row["event_index"])) for row in expired],
+ )
+ trimmed_total = meta_int(trimmed_key) + overflow
+ for key, value in (
+ (total_key, event_total),
+ (trimmed_key, trimmed_total),
+ ("audit_retention", int(self.audit_retention)),
+ ):
+ connection.execute(
+ "INSERT INTO meta(scope_id,key,value_json) VALUES(?,?,?) "
+ "ON CONFLICT(scope_id,key) DO UPDATE SET value_json=excluded.value_json",
+ (normalized_scope, key, json.dumps(value, ensure_ascii=False)),
+ )
+ return {
+ "payload": stored_payload,
+ "event_total": event_total,
+ "trimmed_total": trimmed_total,
+ "appended": True,
+ }
+
+ def _refresh_authoritative_audit_state(
+ self,
+ connection: sqlite3.Connection,
+ scope_id: str,
+ graph: SessionMemoryGraphV2,
+ ) -> None:
+ meta_rows = connection.execute(
+ "SELECT key,value_json FROM meta WHERE scope_id=?",
+ (scope_id,),
+ ).fetchall()
+ meta = {
+ str(row["key"]): json.loads(row["value_json"])
+ for row in meta_rows
+ }
+ fields = {
+ "turn_log": (
+ "audit_turn_log",
+ "audit_turn_events",
+ "audit_trimmed_turn_log",
+ ),
+ "retrieval_log": (
+ "audit_retrieval_log",
+ "audit_retrieval_events",
+ "audit_trimmed_retrieval_log",
+ ),
+ "answer_support_log": (
+ "audit_answer_support",
+ "audit_answer_support_events",
+ "audit_trimmed_answer_support_log",
+ ),
+ }
+
+ def event_key(event: Mapping[str, Any]) -> str:
+ return json.dumps(
+ dict(event),
+ ensure_ascii=False,
+ sort_keys=True,
+ separators=(",", ":"),
+ )
+
+ for field_name, (table, total_key, trimmed_key) in fields.items():
+ persisted = [
+ dict(json.loads(row["payload_json"]) or {})
+ for row in connection.execute(
+ f"SELECT payload_json FROM {table} WHERE scope_id=? "
+ "ORDER BY event_index",
+ (scope_id,),
+ )
+ ]
+ persisted_counts: Dict[str, int] = {}
+ for event in persisted:
+ key = event_key(event)
+ persisted_counts[key] = persisted_counts.get(key, 0) + 1
+ observed: Dict[str, int] = {}
+ merged = list(persisted)
+ appended = 0
+ for event in list(getattr(graph, field_name)):
+ normalized = dict(event)
+ key = event_key(normalized)
+ observed[key] = observed.get(key, 0) + 1
+ if observed[key] > persisted_counts.get(key, 0):
+ merged.append(normalized)
+ appended += 1
+ if len(merged) > self.audit_retention:
+ merged = merged[-self.audit_retention :]
+ event_total = int(meta.get(total_key, 0) or 0) + appended
+ trimmed_total = max(
+ int(meta.get(trimmed_key, 0) or 0),
+ int(graph.audit_trimmed_counts.get(field_name, 0) or 0),
+ event_total - len(merged),
+ )
+ setattr(graph, field_name, merged)
+ graph.audit_event_totals[field_name] = event_total
+ graph.audit_trimmed_counts[field_name] = trimmed_total
+
+ def save_graph(
+ self,
+ scope_id: str,
+ graph: SessionMemoryGraphV2,
+ *,
+ transaction_hook: Callable[[sqlite3.Connection], None] | None = None,
+ ) -> None:
+ normalized_scope = _clean_text(scope_id)
+ if not normalized_scope:
+ raise ValueError("scope_id must be non-empty for SQLite persistence")
+ graph.configure_persistence(backend="sqlite", path=str(self.storage_path), audit_retention=self.audit_retention)
+ raw_expected_revision = getattr(graph, "_storage_revision", None)
+ expected_revision = (
+ int(raw_expected_revision or 0)
+ if raw_expected_revision is not None
+ else None
+ )
+ with self._managed_connection() as connection:
+ connection.execute("BEGIN IMMEDIATE")
+ revision_row = connection.execute(
+ "SELECT value_json FROM meta WHERE scope_id=? AND key='storage_revision'",
+ (normalized_scope,),
+ ).fetchone()
+ current_revision = (
+ int(json.loads(revision_row["value_json"]) or 0)
+ if revision_row
+ else 0
+ )
+ if expected_revision is None:
+ existing_record_count = int(
+ connection.execute(
+ "SELECT COUNT(*) FROM records WHERE scope_id=?",
+ (normalized_scope,),
+ ).fetchone()[0]
+ )
+ if current_revision or existing_record_count:
+ raise StaleGraphSnapshotError(
+ "unversioned graph cannot replace an existing scope; "
+ "load_graph must establish the snapshot revision first"
+ )
+ expected_revision = 0
+ if expected_revision != current_revision:
+ raise StaleGraphSnapshotError(
+ "graph snapshot revision is stale: "
+ f"expected={expected_revision}, current={current_revision}"
+ )
+ self._refresh_authoritative_slow_state(connection, normalized_scope, graph)
+ self._refresh_authoritative_audit_state(
+ connection, normalized_scope, graph
+ )
+ next_revision = current_revision + 1
+ audit_tables = {
+ "audit_turn_log": list(graph.turn_log),
+ "audit_retrieval_log": list(graph.retrieval_log),
+ "audit_answer_support": list(graph.answer_support_log),
+ }
+ meta_entries = {
+ "turn_index": int(graph.turn_index),
+ "noise_turn_count": int(graph.noise_turn_count),
+ "audit_retention": int(graph.audit_retention),
+ "audit_turn_events": int(graph.audit_event_totals.get("turn_log", 0) or 0),
+ "audit_retrieval_events": int(graph.audit_event_totals.get("retrieval_log", 0) or 0),
+ "audit_answer_support_events": int(graph.audit_event_totals.get("answer_support_log", 0) or 0),
+ "audit_trimmed_turn_log": int(graph.audit_trimmed_counts.get("turn_log", 0) or 0),
+ "audit_trimmed_retrieval_log": int(graph.audit_trimmed_counts.get("retrieval_log", 0) or 0),
+ "audit_trimmed_answer_support_log": int(graph.audit_trimmed_counts.get("answer_support_log", 0) or 0),
+ "schema_version": 3,
+ "storage_revision": next_revision,
+ }
+ for table in (
+ "records",
+ "slot_heads",
+ "slot_history",
+ "memory_edges",
+ "subject_depth_heads",
+ "audit_turn_log",
+ "audit_retrieval_log",
+ "audit_answer_support",
+ "meta",
+ ):
+ connection.execute(f"DELETE FROM {table} WHERE scope_id = ?", (normalized_scope,))
+ connection.executemany(
+ """
+ INSERT INTO records (
+ scope_id, memory_id, category, slot_key, value, relation,
+ anchor_concepts_json, evidence_anchors_json, salience, confidence,
+ source_kind, turn_index, state, supersedes_json, metadata_json
+ ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+ """,
+ [
+ (
+ normalized_scope,
+ record.memory_id,
+ record.category,
+ record.slot_key,
+ record.value,
+ record.relation,
+ json.dumps(list(record.anchor_concepts), ensure_ascii=False),
+ json.dumps(list(record.evidence_anchors), ensure_ascii=False),
+ float(record.salience),
+ float(record.confidence),
+ record.source_kind,
+ int(record.turn_index),
+ record.state,
+ json.dumps(list(record.supersedes), ensure_ascii=False),
+ json.dumps(dict(record.metadata), ensure_ascii=False),
+ )
+ for record in graph.records_by_id.values()
+ ],
+ )
+ connection.executemany(
+ "INSERT INTO slot_heads (scope_id, slot_key, memory_id) VALUES (?, ?, ?)",
+ [(normalized_scope, slot_key, memory_id) for slot_key, memory_id in graph.slot_heads.items()],
+ )
+ connection.executemany(
+ "INSERT INTO slot_history (scope_id, slot_key, ordinal, memory_id) VALUES (?, ?, ?, ?)",
+ [
+ (normalized_scope, slot_key, ordinal, memory_id)
+ for slot_key, memory_ids in graph.slot_history.items()
+ for ordinal, memory_id in enumerate(memory_ids)
+ ],
+ )
+ connection.executemany(
+ """
+ INSERT INTO memory_edges (
+ scope_id, edge_id, source_memory_id, target_memory_id, edge_type,
+ score, model_score, evidence_turn, evidence, metadata_json
+ ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+ """,
+ [
+ (
+ normalized_scope,
+ edge.edge_id,
+ edge.source_memory_id,
+ edge.target_memory_id,
+ edge.edge_type,
+ float(edge.score),
+ float(edge.model_score),
+ int(edge.evidence_turn),
+ edge.evidence,
+ json.dumps(dict(edge.metadata), ensure_ascii=False),
+ )
+ for edge in graph.memory_edges.values()
+ ],
+ )
+ connection.executemany(
+ "INSERT INTO subject_depth_heads (scope_id, subject_signature, depth_layer, memory_id) VALUES (?, ?, ?, ?)",
+ [
+ (normalized_scope, subject_signature, depth_layer, memory_id)
+ for subject_signature, heads in graph.subject_depth_heads.items()
+ for depth_layer, memory_id in heads.items()
+ ],
+ )
+ for table, rows in audit_tables.items():
+ connection.executemany(
+ f"INSERT INTO {table} (scope_id, event_index, payload_json) VALUES (?, ?, ?)",
+ [
+ (normalized_scope, event_index, json.dumps(dict(payload), ensure_ascii=False))
+ for event_index, payload in enumerate(rows)
+ ],
+ )
+ connection.executemany(
+ "INSERT INTO meta (scope_id, key, value_json) VALUES (?, ?, ?)",
+ [(normalized_scope, key, json.dumps(value, ensure_ascii=False)) for key, value in meta_entries.items()],
+ )
+ if transaction_hook is not None:
+ transaction_hook(connection)
+ graph._storage_revision = next_revision
+
+ def load_graph(self, scope_id: str) -> SessionMemoryGraphV2:
+ normalized_scope = _clean_text(scope_id)
+ graph = SessionMemoryGraphV2(
+ audit_retention=self.audit_retention,
+ persistence_backend="sqlite",
+ persistence_path=str(self.storage_path),
+ )
+ graph._storage_revision = 0
+ if not normalized_scope:
+ return graph
+ with self._managed_connection() as connection:
+ meta_rows = connection.execute("SELECT key, value_json FROM meta WHERE scope_id = ?", (normalized_scope,)).fetchall()
+ meta = {str(row["key"]): json.loads(row["value_json"]) for row in meta_rows}
+ graph._storage_revision = int(meta.get("storage_revision", 0) or 0)
+ graph.audit_retention = int(meta.get("audit_retention", graph.audit_retention) or graph.audit_retention)
+ graph.turn_index = int(meta.get("turn_index", 0) or 0)
+ graph.noise_turn_count = int(meta.get("noise_turn_count", 0) or 0)
+ graph.audit_event_totals = {
+ "turn_log": int(meta.get("audit_turn_events", 0) or 0),
+ "retrieval_log": int(meta.get("audit_retrieval_events", 0) or 0),
+ "answer_support_log": int(meta.get("audit_answer_support_events", 0) or 0),
+ }
+ graph.audit_trimmed_counts = {
+ "turn_log": int(meta.get("audit_trimmed_turn_log", 0) or 0),
+ "retrieval_log": int(meta.get("audit_trimmed_retrieval_log", 0) or 0),
+ "answer_support_log": int(meta.get("audit_trimmed_answer_support_log", 0) or 0),
+ }
+ record_rows = connection.execute(
+ """
+ SELECT memory_id, category, slot_key, value, relation, anchor_concepts_json,
+ evidence_anchors_json, salience, confidence, source_kind, turn_index,
+ state, supersedes_json, metadata_json
+ FROM records
+ WHERE scope_id = ?
+ ORDER BY turn_index, slot_key, memory_id
+ """,
+ (normalized_scope,),
+ ).fetchall()
+ for row in record_rows:
+ record = SessionMemoryRecordV2(
+ memory_id=str(row["memory_id"]),
+ category=str(row["category"]),
+ slot_key=str(row["slot_key"]),
+ value=str(row["value"]),
+ relation=str(row["relation"]),
+ anchor_concepts=[str(item) for item in json.loads(row["anchor_concepts_json"]) or []],
+ evidence_anchors=[str(item) for item in json.loads(row["evidence_anchors_json"]) or []],
+ salience=float(row["salience"] or 0.0),
+ confidence=float(row["confidence"] or 0.0),
+ source_kind=str(row["source_kind"]),
+ turn_index=int(row["turn_index"] or 0),
+ state=str(row["state"]),
+ supersedes=[str(item) for item in json.loads(row["supersedes_json"]) or []],
+ metadata=dict(json.loads(row["metadata_json"]) or {}),
+ )
+ graph.records_by_id[record.memory_id] = record
+ head_rows = connection.execute(
+ "SELECT slot_key, memory_id FROM slot_heads WHERE scope_id = ? ORDER BY slot_key",
+ (normalized_scope,),
+ ).fetchall()
+ graph.slot_heads = {str(row["slot_key"]): str(row["memory_id"]) for row in head_rows}
+ history_rows = connection.execute(
+ "SELECT slot_key, ordinal, memory_id FROM slot_history WHERE scope_id = ? ORDER BY slot_key, ordinal",
+ (normalized_scope,),
+ ).fetchall()
+ graph.slot_history = defaultdict(list)
+ for row in history_rows:
+ graph.slot_history[str(row["slot_key"])].append(str(row["memory_id"]))
+ edge_rows = connection.execute(
+ """
+ SELECT edge_id, source_memory_id, target_memory_id, edge_type,
+ score, model_score, evidence_turn, evidence, metadata_json
+ FROM memory_edges
+ WHERE scope_id = ?
+ ORDER BY evidence_turn, edge_id
+ """,
+ (normalized_scope,),
+ ).fetchall()
+ graph.memory_edges = {}
+ for row in edge_rows:
+ edge = SessionMemoryEdgeV2(
+ edge_id=str(row["edge_id"]),
+ source_memory_id=str(row["source_memory_id"]),
+ target_memory_id=str(row["target_memory_id"]),
+ edge_type=str(row["edge_type"]),
+ score=float(row["score"] or 0.0),
+ model_score=float(row["model_score"] or 0.0),
+ evidence_turn=int(row["evidence_turn"] or 0),
+ evidence=str(row["evidence"] or ""),
+ metadata=dict(json.loads(row["metadata_json"]) or {}),
+ )
+ graph.memory_edges[edge.edge_id] = edge
+ head_depth_rows = connection.execute(
+ """
+ SELECT subject_signature, depth_layer, memory_id
+ FROM subject_depth_heads
+ WHERE scope_id = ?
+ ORDER BY subject_signature, depth_layer
+ """,
+ (normalized_scope,),
+ ).fetchall()
+ graph.subject_depth_heads = defaultdict(dict)
+ for row in head_depth_rows:
+ graph.subject_depth_heads[str(row["subject_signature"])][str(row["depth_layer"])] = str(row["memory_id"])
+ if not graph.subject_depth_heads:
+ for record in sorted(graph.records_by_id.values(), key=lambda item: int(item.turn_index)):
+ graph._refresh_subject_depth_head(record)
+ graph.turn_log = self._load_audit_table(connection, "audit_turn_log", normalized_scope)
+ graph.retrieval_log = self._load_audit_table(connection, "audit_retrieval_log", normalized_scope)
+ graph.answer_support_log = self._load_audit_table(connection, "audit_answer_support", normalized_scope)
+ if graph.records_by_id and graph.turn_index <= 0:
+ graph.turn_index = max(record.turn_index for record in graph.records_by_id.values())
+ for field_name, log_rows in (
+ ("turn_log", graph.turn_log),
+ ("retrieval_log", graph.retrieval_log),
+ ("answer_support_log", graph.answer_support_log),
+ ):
+ graph.audit_event_totals[field_name] = max(int(graph.audit_event_totals.get(field_name, 0) or 0), len(log_rows))
+ return graph
+
+ def _load_audit_table(self, connection: sqlite3.Connection, table: str, scope_id: str) -> List[Dict[str, Any]]:
+ rows = connection.execute(
+ f"SELECT payload_json FROM {table} WHERE scope_id = ? ORDER BY event_index",
+ (scope_id,),
+ ).fetchall()
+ return [dict(json.loads(row["payload_json"]) or {}) for row in rows]
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/memory_profiles.py b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/memory_profiles.py
new file mode 100644
index 0000000000000000000000000000000000000000..87406171c2a68f3d4276d6d122a44c8cc1a4b6d2
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/memory_profiles.py
@@ -0,0 +1,508 @@
+from __future__ import annotations
+
+from dataclasses import dataclass
+import time
+import re
+from typing import Any, Dict, Iterable, List, Sequence
+
+from .generic_memory import MemoryPolicy, MemoryProfile, MemoryRecord, MemoryRetrievalResult, MemorySessionScope
+from .generic_memory import _clean_text, _dedupe, _normalize
+
+
+def _slot_slug(value: Any, *, fallback: str = "default", max_length: int = 24) -> str:
+ text = _normalize(value)
+ if not text:
+ return fallback
+ parts = re.findall(r"[a-z0-9]+|[\u4e00-\u9fff]", text)
+ if not parts:
+ return fallback
+ slug = ".".join(parts[: max(1, max_length // 2)])
+ slug = re.sub(r"\.+", ".", slug).strip(".")
+ if not slug:
+ return fallback
+ if len(slug) > max_length:
+ slug = slug[:max_length].rstrip(".")
+ return slug or fallback
+
+
+def _query_tokens(value: Any) -> List[str]:
+ return re.findall(r"[a-z0-9]+|[\u4e00-\u9fff]", _normalize(value))
+
+
+def _strip_record_label_prefix(label: str) -> str:
+ normalized = _normalize(label)
+ if not normalized:
+ return ""
+ normalized = re.sub(
+ r"^(?:current|previous|historical)\s+",
+ "",
+ normalized,
+ )
+ normalized = re.sub(r"^(?:当前|之前|历史)", "", normalized)
+ return normalized.strip()
+
+
+def _record_type_aliases(record_type: str, labels: "_TypeLabels") -> List[str]:
+ aliases = _dedupe(
+ [
+ _normalize(record_type),
+ _normalize(record_type.replace("_", " ")),
+ _strip_record_label_prefix(labels.current_en),
+ _strip_record_label_prefix(labels.previous_en),
+ _strip_record_label_prefix(labels.current_zh),
+ _strip_record_label_prefix(labels.previous_zh),
+ ]
+ )
+ return [alias for alias in aliases if alias]
+
+
+@dataclass(slots=True)
+class _TypeLabels:
+ current_en: str
+ previous_en: str
+ current_zh: str
+ previous_zh: str
+
+
+class StructuredMemoryProfile(MemoryProfile):
+ profile_name = "structured"
+ record_labels: Dict[str, _TypeLabels] = {}
+ record_markers: Dict[str, Sequence[str]] = {}
+ render_order: Dict[str, int] = {}
+
+ def policy(self) -> MemoryPolicy:
+ return MemoryPolicy()
+
+ def normalize_event(self, event_kind: str, payload: Dict[str, Any] | None, scope: MemorySessionScope) -> Dict[str, Any]:
+ normalized = super().normalize_event(event_kind, payload, scope)
+ if normalized["records"]:
+ return normalized
+ return {**normalized, "records": self._records_from_payload(event_kind, normalized, scope)}
+
+ def _records_from_payload(self, event_kind: str, payload: Dict[str, Any], scope: MemorySessionScope) -> List[Dict[str, Any]]:
+ _ = event_kind, payload, scope
+ return []
+
+ def _build_record(
+ self,
+ raw: Dict[str, Any],
+ *,
+ event_kind: str,
+ turn_index: int,
+ created_at: float,
+ scope: MemorySessionScope,
+ index: int,
+ ) -> MemoryRecord | None:
+ record_type = _clean_text(raw.get("type", "")) or _clean_text(raw.get("category", "")) or "memory"
+ value = _clean_text(raw.get("value", ""))
+ if not value:
+ return None
+ namespace = _clean_text(raw.get("namespace", "")) or _clean_text(scope.namespace) or "default"
+ attributes = dict(raw.get("attributes") or {})
+ anchors = _dedupe(raw.get("anchors", []) or [])
+ if anchors and not attributes.get("anchors"):
+ attributes["anchors"] = anchors
+ relation = _clean_text(raw.get("relation", ""))
+ if relation and not attributes.get("relation"):
+ attributes["relation"] = relation
+ provenance = {
+ **dict(raw.get("provenance") or {}),
+ **dict(raw.get("metadata") or {}),
+ "event_kind": _clean_text(event_kind),
+ }
+ key = self.resolve_key(
+ record_type,
+ value,
+ attributes,
+ namespace=namespace,
+ provided_key=_clean_text(raw.get("key", "")) or _clean_text(raw.get("slot_key", "")) or _clean_text(raw.get("slot", "")),
+ relation=relation,
+ provenance=provenance,
+ )
+ state = _clean_text(raw.get("state", "")) or ("active" if bool(raw.get("active", True)) else "historical")
+ record_id = _clean_text(raw.get("record_id", "")) or f"{scope.scope_id}:{namespace}:{key}:{turn_index}:{index}"
+ timestamps = {
+ **dict(raw.get("timestamps") or {}),
+ "turn_index": int(turn_index),
+ "created_at": float(raw.get("created_at", created_at) or created_at),
+ "updated_at": float(raw.get("updated_at", created_at) or created_at),
+ }
+ return MemoryRecord(
+ record_id=record_id,
+ namespace=namespace,
+ type=record_type,
+ key=key,
+ value=value,
+ attributes=attributes,
+ state=state,
+ supersedes=_dedupe(raw.get("supersedes", []) or []),
+ provenance=provenance,
+ timestamps=timestamps,
+ scope_id=scope.scope_id,
+ )
+
+ def derive_records(self, event_kind: str, payload: Dict[str, Any], scope: MemorySessionScope) -> List[MemoryRecord]:
+ turn_index = int(payload.get("turn_index", 0) or 0)
+ created_at = float(payload.get("created_at", time.time()) or time.time())
+ records: List[MemoryRecord] = []
+ for index, raw in enumerate(list(payload.get("records", []) or [])):
+ if not isinstance(raw, dict):
+ continue
+ record = self._build_record(raw, event_kind=event_kind, turn_index=turn_index, created_at=created_at, scope=scope, index=index)
+ if record is not None:
+ records.append(record)
+ return records
+
+ def infer_query_types(self, query: str) -> List[str]:
+ lowered = _normalize(query)
+ results: List[str] = []
+ for record_type, markers in self.record_markers.items():
+ if any(_normalize(marker) in lowered for marker in markers):
+ results.append(record_type)
+ return _dedupe(results)
+
+ def requested_category_order(self, query: str, hinted_types: Sequence[str] = ()) -> List[str]:
+ ordered: List[str] = []
+ for item in hinted_types:
+ key = _normalize(item)
+ if key and key not in ordered:
+ ordered.append(key)
+ lowered = _normalize(query)
+ for record_type, markers in self.record_markers.items():
+ if any(_normalize(marker) in lowered for marker in markers) and record_type not in ordered:
+ ordered.append(record_type)
+ return ordered
+
+ def label(self, record_type: str, *, prefer_chinese: bool, historical: bool = False, variant: str = "compare") -> str:
+ labels = self.record_labels.get(
+ record_type,
+ _TypeLabels(
+ current_en=f"current {record_type}",
+ previous_en=f"previous {record_type}",
+ current_zh=f"当前{record_type}",
+ previous_zh=f"之前{record_type}",
+ ),
+ )
+ if prefer_chinese:
+ return labels.previous_zh if historical else labels.current_zh
+ if variant == "history" and historical:
+ return labels.previous_en.replace("previous ", "historical ")
+ return labels.previous_en if historical else labels.current_en
+
+ def render_position(self, record_type: str) -> int:
+ return int(self.render_order.get(_normalize(record_type), 99))
+
+ def render_context(self, query: str, retrieval: MemoryRetrievalResult, scope: MemorySessionScope) -> Dict[str, Any]:
+ lines = [
+ f"{item.record.key} = {item.record.value}"
+ for item in retrieval.records[:8]
+ ]
+ return {
+ "profile_name": self.profile_name,
+ "query": query,
+ "scope": scope.to_dict(),
+ "lines": lines,
+ "record_ids": [item.record.record_id for item in retrieval.records],
+ }
+
+
+class TMCRAProfile(StructuredMemoryProfile):
+ profile_name = "tmcra"
+ record_labels = {
+ "goal": _TypeLabels("current goal", "previous goal", "当前目标", "之前目标"),
+ "constraint": _TypeLabels("current constraint", "previous constraint", "当前约束", "之前约束"),
+ "preference": _TypeLabels("current preference", "previous preference", "当前偏好", "之前偏好"),
+ "terminology": _TypeLabels("current terminology", "previous terminology", "当前术语", "之前术语"),
+ "stage_state": _TypeLabels("current stage", "previous stage", "当前阶段", "之前阶段"),
+ "path": _TypeLabels("current path", "previous path", "当前路径", "之前路径"),
+ }
+ record_markers = {
+ "goal": ("goal", "mission", "objective", "target", "primary goal", "目标"),
+ "constraint": ("constraint", "must", "forbid", "policy", "约束", "限制", "不要", "不能"),
+ "preference": ("preference", "prefer", "default", "mode", "偏好", "优先"),
+ "terminology": ("term", "terminology", "alias", "definition", "术语", "别名", "定义"),
+ "stage_state": ("stage", "phase", "status", "state", "阶段", "进度", "状态"),
+ "path": ("path", "route", "connect", "chain", "路径", "链路", "连接"),
+ }
+ render_order = {
+ "goal": 0,
+ "constraint": 1,
+ "stage_state": 2,
+ "preference": 3,
+ "terminology": 4,
+ "path": 5,
+ }
+
+ def policy(self) -> MemoryPolicy:
+ return MemoryPolicy(writeback_mode="shadow")
+
+ def infer_category_hints(self, query: str) -> List[str]:
+ return self.infer_query_types(query)
+
+ def stable_slot_key(
+ self,
+ *,
+ category: str,
+ value: str,
+ anchors: Sequence[str],
+ slot_key: str = "",
+ relation: str = "",
+ metadata: Dict[str, Any] | None = None,
+ ) -> str:
+ category_key = _normalize(category) or "memory"
+ relation_key = _normalize(relation)
+ raw_slot = _normalize(slot_key or (metadata or {}).get("slot_key") or (metadata or {}).get("slot") or "")
+ anchor_slug = _slot_slug(anchors[0] if anchors else "", fallback="")
+ value_slug = _slot_slug(value, fallback="value")
+ if relation_key == "path_edge" and len(anchors) >= 2:
+ left = _slot_slug(anchors[0], fallback="left")
+ right = _slot_slug(anchors[1], fallback="right")
+ return f"path.{left}.{right}"
+ if raw_slot:
+ raw_slot = re.sub(r"\s+", "", raw_slot)
+ raw_slot = raw_slot.replace("terminology.", "term.").replace("terms.", "term.")
+ raw_slot = raw_slot.replace("preferences.", "preference.").replace("constraints.", "constraint.")
+ raw_slot = raw_slot.replace("stage_state.", "stage.").replace("stages.", "stage.")
+ raw_slot = re.sub(r"\.+", ".", raw_slot).strip(".")
+ if category_key == "goal" and raw_slot in {"goal", "goal.current", "goal.main", "goal.default"}:
+ return "goal.primary"
+ if category_key == "stage_state" and raw_slot in {"stage", "stage.current", "status", "state", "phase"}:
+ return "stage.current"
+ if category_key == "terminology" and raw_slot in {"term", "terminology", "term.current"}:
+ return f"term.{anchor_slug or value_slug}"
+ if category_key == "preference" and raw_slot in {"preference", "preference.current"}:
+ return f"preference.{anchor_slug or 'default'}"
+ if category_key == "constraint" and raw_slot in {"constraint", "constraint.current"}:
+ return f"constraint.{anchor_slug or 'policy'}"
+ return raw_slot
+ if category_key == "goal":
+ return "goal.primary"
+ if category_key == "preference":
+ return f"preference.{anchor_slug or 'default'}"
+ if category_key == "constraint":
+ return f"constraint.{anchor_slug or 'policy'}"
+ if category_key == "terminology":
+ return f"term.{anchor_slug or value_slug}"
+ if category_key == "stage_state":
+ return "stage.current"
+ if category_key == "fact" and anchor_slug:
+ return f"fact.{anchor_slug}.{value_slug}"
+ return f"{category_key}.{anchor_slug or value_slug}"
+
+ def resolve_key(
+ self,
+ record_type: str,
+ value: str,
+ attributes: Dict[str, Any],
+ *,
+ namespace: str,
+ provided_key: str = "",
+ relation: str = "",
+ provenance: Dict[str, Any] | None = None,
+ ) -> str:
+ _ = namespace, provenance
+ return self.stable_slot_key(
+ category=record_type,
+ value=value,
+ anchors=list(attributes.get("anchors", []) or []),
+ slot_key=provided_key,
+ relation=relation or _clean_text(attributes.get("relation", "")),
+ metadata=attributes,
+ )
+
+ def record_search_text(self, record: MemoryRecord) -> str:
+ payload = {
+ "type": record.type,
+ "key": record.key,
+ "value": record.value,
+ "anchors": list(record.attributes.get("anchors", []) or []),
+ "relation": _clean_text(record.attributes.get("relation", "")),
+ "provenance": record.provenance,
+ }
+ return str(payload)
+
+
+class AttributeMemoryProfile(StructuredMemoryProfile):
+ profile_name = "attribute"
+ record_labels = {
+ "attribute": _TypeLabels("current attribute", "previous attribute", "当前属性", "之前属性"),
+ "preference": _TypeLabels("current preference", "previous preference", "当前偏好", "之前偏好"),
+ }
+ record_markers = {
+ "attribute": ("attribute", "profile", "email", "name", "role", "属性", "资料"),
+ "preference": ("preference", "prefer", "setting", "偏好", "设置"),
+ }
+ render_order = {"preference": 0, "attribute": 1}
+
+ def _records_from_payload(self, event_kind: str, payload: Dict[str, Any], scope: MemorySessionScope) -> List[Dict[str, Any]]:
+ _ = event_kind, scope
+ attributes = dict(payload.get("attributes") or {})
+ if not attributes:
+ return []
+ entity = _clean_text(payload.get("entity", "")) or _clean_text(payload.get("subject", "")) or "user"
+ record_type = _clean_text(payload.get("type", "")) or "attribute"
+ return [
+ {"type": record_type, "value": _clean_text(value), "attributes": {"entity": entity, "attribute": _clean_text(name)}}
+ for name, value in attributes.items()
+ if _clean_text(name) and _clean_text(value)
+ ]
+
+ def resolve_key(
+ self,
+ record_type: str,
+ value: str,
+ attributes: Dict[str, Any],
+ *,
+ namespace: str,
+ provided_key: str = "",
+ relation: str = "",
+ provenance: Dict[str, Any] | None = None,
+ ) -> str:
+ _ = value, namespace, relation, provenance
+ if _clean_text(provided_key):
+ return _clean_text(provided_key)
+ entity = _slot_slug(attributes.get("entity", "") or attributes.get("subject", ""), fallback="global")
+ attribute_name = _slot_slug(attributes.get("attribute", "") or attributes.get("name", ""), fallback=record_type or "attribute")
+ return f"attribute.{entity}.{attribute_name}"
+
+ def render_context(self, query: str, retrieval: MemoryRetrievalResult, scope: MemorySessionScope) -> Dict[str, Any]:
+ lines = [
+ f"{item.record.attributes.get('entity', 'user')}.{item.record.attributes.get('attribute', item.record.key)} = {item.record.value}"
+ for item in retrieval.records[:8]
+ ]
+ return {"profile_name": self.profile_name, "query": query, "scope": scope.to_dict(), "lines": lines}
+
+
+class StateMemoryProfile(StructuredMemoryProfile):
+ profile_name = "state"
+ record_labels = {
+ "state": _TypeLabels("current state", "previous state", "当前状态", "之前状态"),
+ "stage": _TypeLabels("current stage", "previous stage", "当前阶段", "之前阶段"),
+ }
+ record_markers = {
+ "state": ("state", "status", "workflow", "任务状态", "状态"),
+ "stage": ("stage", "phase", "阶段"),
+ }
+ render_order = {"state": 0, "stage": 1}
+
+ def _records_from_payload(self, event_kind: str, payload: Dict[str, Any], scope: MemorySessionScope) -> List[Dict[str, Any]]:
+ _ = event_kind, scope
+ state_value = _clean_text(payload.get("state", "")) or _clean_text(payload.get("status", ""))
+ if not state_value:
+ return []
+ entity = _clean_text(payload.get("entity", "")) or _clean_text(payload.get("machine", "")) or "workflow"
+ state_name = _clean_text(payload.get("state_name", "")) or "status"
+ return [{"type": "state", "value": state_value, "attributes": {"entity": entity, "state_name": state_name}}]
+
+ def resolve_key(
+ self,
+ record_type: str,
+ value: str,
+ attributes: Dict[str, Any],
+ *,
+ namespace: str,
+ provided_key: str = "",
+ relation: str = "",
+ provenance: Dict[str, Any] | None = None,
+ ) -> str:
+ _ = value, namespace, relation, provenance
+ if _clean_text(provided_key):
+ return _clean_text(provided_key)
+ entity = _slot_slug(attributes.get("entity", "") or attributes.get("machine", ""), fallback="workflow")
+ state_name = _slot_slug(attributes.get("state_name", "") or attributes.get("name", ""), fallback="status")
+ return f"state.{entity}.{state_name}"
+
+
+class AliasMemoryProfile(StructuredMemoryProfile):
+ profile_name = "alias"
+ record_labels = {
+ "alias": _TypeLabels("current alias", "previous alias", "当前别名", "之前别名"),
+ }
+ record_markers = {
+ "alias": ("alias", "also called", "nickname", "别名", "叫做"),
+ }
+ render_order = {"alias": 0}
+
+ def _records_from_payload(self, event_kind: str, payload: Dict[str, Any], scope: MemorySessionScope) -> List[Dict[str, Any]]:
+ _ = event_kind, scope
+ canonical = _clean_text(payload.get("canonical", "")) or _clean_text(payload.get("entity", ""))
+ aliases = list(payload.get("aliases", []) or [])
+ alias = _clean_text(payload.get("alias", ""))
+ if alias:
+ aliases.append(alias)
+ if not canonical or not aliases:
+ return []
+ return [
+ {"type": "alias", "value": _clean_text(item), "attributes": {"canonical": canonical, "alias": _clean_text(item)}}
+ for item in aliases
+ if _clean_text(item)
+ ]
+
+ def normalize_event(self, event_kind: str, payload: Dict[str, Any] | None, scope: MemorySessionScope) -> Dict[str, Any]:
+ normalized = super().normalize_event(event_kind, payload, scope)
+ expanded: List[Dict[str, Any]] = []
+ for raw in list(normalized.get("records", []) or []):
+ if not isinstance(raw, dict):
+ continue
+ aliases = list(raw.get("aliases", []) or [])
+ alias = _clean_text(raw.get("alias", ""))
+ if alias:
+ aliases.append(alias)
+ canonical = _clean_text(raw.get("canonical", "")) or _clean_text((raw.get("attributes") or {}).get("canonical", ""))
+ if canonical and aliases:
+ for item in aliases:
+ item_text = _clean_text(item)
+ if not item_text:
+ continue
+ expanded.append(
+ {
+ **raw,
+ "type": _clean_text(raw.get("type", "")) or "alias",
+ "value": item_text,
+ "attributes": {**dict(raw.get("attributes") or {}), "canonical": canonical, "alias": item_text},
+ }
+ )
+ continue
+ expanded.append(dict(raw))
+ if expanded:
+ normalized["records"] = expanded
+ return normalized
+
+ def resolve_key(
+ self,
+ record_type: str,
+ value: str,
+ attributes: Dict[str, Any],
+ *,
+ namespace: str,
+ provided_key: str = "",
+ relation: str = "",
+ provenance: Dict[str, Any] | None = None,
+ ) -> str:
+ _ = record_type, namespace, relation, provenance
+ if _clean_text(provided_key):
+ return _clean_text(provided_key)
+ canonical = _slot_slug(attributes.get("canonical", ""), fallback="entity")
+ alias = _slot_slug(attributes.get("alias", "") or value, fallback="alias")
+ return f"alias.{canonical}.{alias}"
+
+ def record_search_text(self, record: MemoryRecord) -> str:
+ canonical = _clean_text(record.attributes.get("canonical", ""))
+ alias = _clean_text(record.attributes.get("alias", "")) or record.value
+ return f"{record.type} {record.key} {alias} {canonical}"
+
+ def build_relations(self, retrieval: MemoryRetrievalResult) -> List[Dict[str, Any]]:
+ relations = super().build_relations(retrieval)
+ for entry in retrieval.records:
+ canonical = _clean_text(entry.record.attributes.get("canonical", ""))
+ alias = _clean_text(entry.record.attributes.get("alias", "")) or entry.record.value
+ if canonical and alias:
+ relations.append(
+ {
+ "from": alias,
+ "to": canonical,
+ "relation": "alias_of",
+ "score": round(float(entry.score), 6),
+ }
+ )
+ return relations
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/multi_layer_tmcra_writer.py b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/multi_layer_tmcra_writer.py
new file mode 100644
index 0000000000000000000000000000000000000000..5e9715c925a74af242c0be157a12ae2d2436e07d
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/multi_layer_tmcra_writer.py
@@ -0,0 +1,1058 @@
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+import re
+from typing import Any, Dict, List, Mapping, Sequence
+
+from experiments.replacement.semantic_memory_writer import (
+ DeterministicMemoryWriteGate,
+ OpenAICompatSemanticMemoryWriter,
+ SemanticMemoryWriterError,
+ _dedupe_texts,
+ _extract_memory_subject,
+ _safe_text,
+ _slug,
+ _tokens,
+)
+
+
+def _clip_text(value: Any, limit: int) -> str:
+ text = _safe_text(value)
+ if limit <= 0 or len(text) <= limit:
+ return text
+ return text[: max(0, limit - 1)].rstrip(" ,.;:") + "..."
+
+
+def _category_weight(category: str) -> float:
+ category = _safe_text(category).lower()
+ if category in {"profile", "status", "preference", "goal", "constraint"}:
+ return 0.05
+ if category in {"time", "event"}:
+ return 0.03
+ return 0.0
+
+
+def _semantic_slot_for(proposal: Mapping[str, Any]) -> str:
+ category = _safe_text(proposal.get("category", "")).lower() or "fact"
+ slot = _safe_text(proposal.get("semantic_slot", "")).lower()
+ if slot:
+ return slot
+ if category == "profile":
+ return "profile"
+ if category == "preference":
+ return "preference"
+ if category == "goal":
+ return "goal"
+ if category == "status":
+ return "status"
+ if category == "time":
+ return "event_time"
+ return category
+
+
+_UNCERTAINTY_CUE_RE = re.compile(
+ r"\b("
+ r"maybe|might|could|perhaps|possibly|probably|i wonder|i think|i guess|not sure|"
+ r"sort of|kind of|feels like|seems like|leaning toward|drawn to|what if"
+ r")\b",
+ re.IGNORECASE,
+)
+
+_ABSTRACT_MEMORY_TERMS = {
+ "memory",
+ "identity",
+ "meaning",
+ "self",
+ "alive",
+ "aliveness",
+ "experience",
+ "truth",
+ "freedom",
+ "agency",
+ "resonance",
+ "dissonance",
+ "pattern",
+ "structure",
+ "story",
+ "narrative",
+ "body",
+ "biology",
+ "transcendence",
+}
+
+_LOW_SIGNAL_TOKENS = {
+ "about",
+ "again",
+ "already",
+ "also",
+ "because",
+ "before",
+ "being",
+ "could",
+ "does",
+ "feel",
+ "feels",
+ "from",
+ "have",
+ "just",
+ "like",
+ "maybe",
+ "more",
+ "that",
+ "this",
+ "think",
+ "what",
+ "when",
+ "where",
+ "with",
+ "would",
+ "your",
+}
+
+
+def _token_set(*values: Any) -> set[str]:
+ return {token for value in values for token in _tokens(_safe_text(value)) if token not in _LOW_SIGNAL_TOKENS}
+
+
+def _abstract_ratio(tokens: set[str]) -> float:
+ if not tokens:
+ return 0.0
+ return len(tokens & _ABSTRACT_MEMORY_TERMS) / max(1, len(tokens))
+
+
+def _overlap_ratio(left: set[str], right: set[str]) -> float:
+ if not left or not right:
+ return 0.0
+ return len(left & right) / max(1, len(left | right))
+
+
+@dataclass(slots=True)
+class TMCRACell:
+ category: str
+ semantic_slot: str
+ value: str
+ source_span: str
+ anchors: List[str] = field(default_factory=list)
+ subject: str = ""
+ slot_key: str = ""
+ recurrence_count: int = 1
+ previous_value: str = ""
+ confidence: float = 0.86
+ salience: float = 0.9
+ metadata: Dict[str, Any] = field(default_factory=dict)
+
+ def to_proposal(self) -> Dict[str, Any]:
+ payload = {
+ "category": self.category,
+ "value": self.value,
+ "source_span": self.source_span,
+ "semantic_slot": self.semantic_slot,
+ "anchors": list(self.anchors),
+ "confidence": self.confidence,
+ "salience": self.salience,
+ "slot_key": self.slot_key,
+ "event_signature": self.metadata.get("event_signature", ""),
+ "relation": self.metadata.get("relation", f"{self.category}_memory"),
+ "state": self.metadata.get("state", "active"),
+ "allow_parallel_state": self.metadata.get("allow_parallel_state", True),
+ "tmcra_stack_metadata": dict(self.metadata),
+ }
+ return {
+ key: value
+ for key, value in payload.items()
+ if value is not None and value != "" and value != []
+ }
+
+
+class MultiLayerTMCRASemanticMemoryWriter:
+ """A local multi-layer TMCRA writer stack around one delta LLM call.
+
+ The LLM remains L0 and only sees the current turn. Later TMCRA layers are
+ deterministic/local so stacking does not multiply prompt tokens.
+ """
+
+ architecture = "multi_layer_tmcra_writer_v1"
+
+ def __init__(
+ self,
+ base_writer: OpenAICompatSemanticMemoryWriter,
+ *,
+ max_cells: int = 2,
+ value_char_budget: int = 120,
+ source_span_char_budget: int = 180,
+ ) -> None:
+ self.base_writer = base_writer
+ self.gate: DeterministicMemoryWriteGate = base_writer.gate
+ self.max_proposals = max(1, int(max_cells or 2))
+ self.value_char_budget = max(40, int(value_char_budget or 120))
+ self.source_span_char_budget = max(60, int(source_span_char_budget or 180))
+ self._slot_state: Dict[str, TMCRACell] = {}
+
+ def available(self) -> bool:
+ return self.base_writer.available()
+
+ def propose_public_turn(
+ self,
+ *,
+ current_turn: str,
+ previous_turn: str = "",
+ next_turn: str = "",
+ speaker: str = "",
+ session_timestamp: str = "",
+ sidecar_hints: Mapping[str, Any] | None = None,
+ auxiliary_evidence_texts: Sequence[Any] | None = None,
+ input_mode: str = "delta",
+ ) -> tuple[List[Dict[str, Any]], Dict[str, Any]]:
+ if not self.available():
+ raise SemanticMemoryWriterError("multi-layer TMCRA writer requires an available base writer")
+ base_proposals, base_metadata = self.base_writer.propose_public_turn(
+ current_turn=current_turn,
+ previous_turn="",
+ next_turn="",
+ speaker=speaker,
+ session_timestamp=session_timestamp,
+ sidecar_hints={},
+ auxiliary_evidence_texts=[],
+ input_mode="delta",
+ )
+ turn_cells, turn_meta = self._turn_layer(
+ base_proposals,
+ current_turn=current_turn,
+ speaker=speaker,
+ session_timestamp=session_timestamp,
+ )
+ memory_cells, memory_meta = self._memory_layer(turn_cells, speaker=speaker)
+ compressed_cells, compression_meta = self._compression_layer(memory_cells)
+ recurrent_cells, recurrence_meta = self._recurrence_layer(compressed_cells, speaker=speaker)
+ activated_cells, activation_meta = self._activation_layer(recurrent_cells)
+ proposals = [cell.to_proposal() for cell in activated_cells[: self.max_proposals]]
+ metadata = {
+ **dict(base_metadata or {}),
+ "architecture": self.architecture,
+ "input_mode": "delta",
+ "base_input_mode": dict(base_metadata or {}).get("input_mode", "delta"),
+ "system_prompt_profile": "multi_layer_tmcra_delta_l0",
+ "user_payload_keys": ["current_turn", "max_write_proposals", "speaker", "timestamp"],
+ "tmcra_layers": [
+ turn_meta,
+ memory_meta,
+ compression_meta,
+ recurrence_meta,
+ activation_meta,
+ ],
+ "tmcra_stack_state_size": len(self._slot_state),
+ "tmcra_stack_input_ignored": {
+ "previous_turn": bool(_safe_text(previous_turn)),
+ "next_turn": bool(_safe_text(next_turn)),
+ "sidecar_hints": bool(sidecar_hints),
+ "auxiliary_evidence_texts": bool(auxiliary_evidence_texts),
+ },
+ "proposal_count_after_stack": len(proposals),
+ }
+ return proposals, metadata
+
+ def _turn_layer(
+ self,
+ proposals: Sequence[Mapping[str, Any]],
+ *,
+ current_turn: str,
+ speaker: str,
+ session_timestamp: str,
+ ) -> tuple[List[TMCRACell], Dict[str, Any]]:
+ cells: List[TMCRACell] = []
+ for index, proposal in enumerate(proposals):
+ category = _safe_text(proposal.get("category", "")).lower().replace("-", "_") or "fact"
+ value = _clip_text(proposal.get("value", ""), self.value_char_budget)
+ source_span = _clip_text(proposal.get("source_span", ""), self.source_span_char_budget)
+ if not value or not source_span:
+ continue
+ semantic_slot = _semantic_slot_for(proposal)
+ anchors = _dedupe_texts(
+ [
+ speaker,
+ session_timestamp,
+ *list(proposal.get("anchors", []) or []),
+ *(_tokens(value)[:4]),
+ ],
+ max_items=8,
+ )
+ cells.append(
+ TMCRACell(
+ category=category,
+ semantic_slot=semantic_slot,
+ value=value,
+ source_span=source_span,
+ anchors=anchors,
+ confidence=min(1.0, max(0.0, float(proposal.get("confidence", 0.86) or 0.86))),
+ salience=min(1.0, max(0.0, float(proposal.get("salience", 0.9) or 0.9))),
+ metadata={
+ "tmcra_layers": ["T"],
+ "tmcra_turn_layer_index": index,
+ "tmcra_turn_signature": _slug(" ".join(_tokens(source_span)[:8])),
+ "target_status": _safe_text(proposal.get("target_status", "")),
+ },
+ )
+ )
+ return cells, {
+ "layer": "T",
+ "name": "turn_delta_projection",
+ "input_count": len(list(proposals)),
+ "output_count": len(cells),
+ }
+
+ def _memory_layer(self, cells: Sequence[TMCRACell], *, speaker: str) -> tuple[List[TMCRACell], Dict[str, Any]]:
+ output: List[TMCRACell] = []
+ for cell in cells:
+ subject = _extract_memory_subject(cell.source_span, cell.value)
+ if not subject:
+ subject = self._subject_hint(cell.value, cell.semantic_slot)
+ subject_signature = _slug(subject or cell.semantic_slot or cell.category)
+ slot_key = ".".join(
+ part
+ for part in [
+ _slug(speaker or "speaker"),
+ "tmcra",
+ _slug(cell.category),
+ _slug(cell.semantic_slot),
+ subject_signature,
+ ]
+ if part
+ )
+ cell.subject = subject
+ cell.slot_key = slot_key
+ cell.metadata = {
+ **cell.metadata,
+ "tmcra_layers": [*list(cell.metadata.get("tmcra_layers", [])), "M"],
+ "tmcra_memory_subject": subject,
+ "tmcra_subject_signature": subject_signature,
+ "canonical_slot_key": slot_key,
+ "event_signature": _slug(f"{cell.category} {cell.semantic_slot} {subject_signature} {' '.join(_tokens(cell.value)[:6])}"),
+ }
+ output.append(cell)
+ return output, {
+ "layer": "M",
+ "name": "memory_cell_projection",
+ "input_count": len(cells),
+ "output_count": len(output),
+ }
+
+ def _compression_layer(self, cells: Sequence[TMCRACell]) -> tuple[List[TMCRACell], Dict[str, Any]]:
+ merged: Dict[str, TMCRACell] = {}
+ for cell in cells:
+ dedupe_key = f"{cell.slot_key}:{_slug(' '.join(_tokens(cell.value)[:10]))}"
+ existing = merged.get(dedupe_key)
+ if existing is None:
+ merged[dedupe_key] = cell
+ continue
+ if len(cell.value) < len(existing.value):
+ cell.anchors = _dedupe_texts([*existing.anchors, *cell.anchors], max_items=8)
+ merged[dedupe_key] = cell
+ else:
+ existing.anchors = _dedupe_texts([*existing.anchors, *cell.anchors], max_items=8)
+ output = sorted(
+ merged.values(),
+ key=lambda item: (item.salience + item.confidence + _category_weight(item.category), -len(item.value)),
+ reverse=True,
+ )[: self.max_proposals]
+ for cell in output:
+ cell.metadata = {
+ **cell.metadata,
+ "tmcra_layers": [*list(cell.metadata.get("tmcra_layers", [])), "C"],
+ "tmcra_compressed": True,
+ "tmcra_value_char_budget": self.value_char_budget,
+ "tmcra_source_span_char_budget": self.source_span_char_budget,
+ }
+ return output, {
+ "layer": "C",
+ "name": "cell_compression_dedupe_budget",
+ "input_count": len(cells),
+ "output_count": len(output),
+ }
+
+ def _recurrence_layer(self, cells: Sequence[TMCRACell], *, speaker: str) -> tuple[List[TMCRACell], Dict[str, Any]]:
+ updated = 0
+ output: List[TMCRACell] = []
+ for cell in cells:
+ previous = self._slot_state.get(cell.slot_key)
+ if previous is not None:
+ cell.previous_value = previous.value
+ cell.recurrence_count = previous.recurrence_count + 1
+ updated += 1
+ self._slot_state[cell.slot_key] = cell
+ cell.metadata = {
+ **cell.metadata,
+ "tmcra_layers": [*list(cell.metadata.get("tmcra_layers", [])), "R"],
+ "tmcra_recurrence_count": cell.recurrence_count,
+ "tmcra_previous_value": cell.previous_value,
+ "relation": "updates_memory" if cell.previous_value and cell.previous_value != cell.value else f"{cell.category}_memory",
+ }
+ output.append(cell)
+ return output, {
+ "layer": "R",
+ "name": "recurrent_slot_state",
+ "input_count": len(cells),
+ "output_count": len(output),
+ "updated_slots": updated,
+ "state_size": len(self._slot_state),
+ }
+
+ def _activation_layer(self, cells: Sequence[TMCRACell]) -> tuple[List[TMCRACell], Dict[str, Any]]:
+ output: List[TMCRACell] = []
+ for cell in cells:
+ source_tokens = len(set(_tokens(cell.source_span)))
+ confidence = min(0.98, max(cell.confidence, 0.72 + min(0.16, source_tokens / 80.0)))
+ salience = min(0.98, max(cell.salience, 0.82 + _category_weight(cell.category)))
+ cell.confidence = confidence
+ cell.salience = salience
+ cell.metadata = {
+ **cell.metadata,
+ "tmcra_layers": [*list(cell.metadata.get("tmcra_layers", [])), "A"],
+ "tmcra_activation_score": round((confidence + salience) / 2.0, 6),
+ "state": "active",
+ "allow_parallel_state": cell.category not in {"profile", "status", "preference", "goal", "constraint"},
+ "source_layer": self.architecture,
+ }
+ output.append(cell)
+ return output, {
+ "layer": "A",
+ "name": "activation_write_budget",
+ "input_count": len(cells),
+ "output_count": len(output),
+ }
+
+ @staticmethod
+ def _subject_hint(value: str, semantic_slot: str) -> str:
+ tokens = [token for token in _tokens(value) if token not in {"maya", "user", "the", "and", "that", "with"}]
+ if len(tokens) >= 2:
+ return " ".join(tokens[:4])
+ return _safe_text(semantic_slot)
+
+
+class TMCRATransformerSemanticMemoryWriter(MultiLayerTMCRASemanticMemoryWriter):
+ """Transformer-inspired TMCRA stack over memory cells, not raw tokens.
+
+ This keeps the online LLM cost at one delta writer call, then applies local
+ cell attention, residual updates, normalization, and feed-forward slot
+ refinement before the normal TMCRA compression/recurrent/activation path.
+ """
+
+ architecture = "tmcra_transformer_stack_v1"
+
+ def __init__(
+ self,
+ base_writer: OpenAICompatSemanticMemoryWriter,
+ *,
+ max_cells: int = 2,
+ value_char_budget: int = 120,
+ source_span_char_budget: int = 180,
+ transformer_layers: int = 2,
+ state_attention_k: int = 16,
+ ) -> None:
+ super().__init__(
+ base_writer,
+ max_cells=max_cells,
+ value_char_budget=value_char_budget,
+ source_span_char_budget=source_span_char_budget,
+ )
+ self.transformer_layers = max(1, int(transformer_layers or 2))
+ self.state_attention_k = max(0, int(state_attention_k or 16))
+
+ def propose_public_turn(
+ self,
+ *,
+ current_turn: str,
+ previous_turn: str = "",
+ next_turn: str = "",
+ speaker: str = "",
+ session_timestamp: str = "",
+ sidecar_hints: Mapping[str, Any] | None = None,
+ auxiliary_evidence_texts: Sequence[Any] | None = None,
+ input_mode: str = "delta",
+ ) -> tuple[List[Dict[str, Any]], Dict[str, Any]]:
+ if not self.available():
+ raise SemanticMemoryWriterError("TMCRA transformer writer requires an available base writer")
+ base_proposals, base_metadata = self.base_writer.propose_public_turn(
+ current_turn=current_turn,
+ previous_turn="",
+ next_turn="",
+ speaker=speaker,
+ session_timestamp=session_timestamp,
+ sidecar_hints={},
+ auxiliary_evidence_texts=[],
+ input_mode="delta",
+ )
+ turn_cells, turn_meta = self._turn_layer(
+ base_proposals,
+ current_turn=current_turn,
+ speaker=speaker,
+ session_timestamp=session_timestamp,
+ )
+ memory_cells, memory_meta = self._memory_layer(turn_cells, speaker=speaker)
+ transformer_cells, transformer_meta = self._transformer_cell_layers(memory_cells)
+ compressed_cells, compression_meta = self._compression_layer(transformer_cells)
+ recurrent_cells, recurrence_meta = self._recurrence_layer(compressed_cells, speaker=speaker)
+ activated_cells, activation_meta = self._activation_layer(recurrent_cells)
+ proposals = [cell.to_proposal() for cell in activated_cells[: self.max_proposals]]
+ metadata = {
+ **dict(base_metadata or {}),
+ "architecture": self.architecture,
+ "input_mode": "delta",
+ "base_input_mode": dict(base_metadata or {}).get("input_mode", "delta"),
+ "system_prompt_profile": "tmcra_transformer_delta_l0",
+ "user_payload_keys": ["current_turn", "max_write_proposals", "speaker", "timestamp"],
+ "tmcra_layers": [
+ turn_meta,
+ memory_meta,
+ *transformer_meta,
+ compression_meta,
+ recurrence_meta,
+ activation_meta,
+ ],
+ "tmcra_stack_state_size": len(self._slot_state),
+ "tmcra_transformer_layers": self.transformer_layers,
+ "tmcra_state_attention_k": self.state_attention_k,
+ "tmcra_stack_input_ignored": {
+ "previous_turn": bool(_safe_text(previous_turn)),
+ "next_turn": bool(_safe_text(next_turn)),
+ "sidecar_hints": bool(sidecar_hints),
+ "auxiliary_evidence_texts": bool(auxiliary_evidence_texts),
+ },
+ "proposal_count_after_stack": len(proposals),
+ }
+ return proposals, metadata
+
+ def _transformer_cell_layers(self, cells: Sequence[TMCRACell]) -> tuple[List[TMCRACell], List[Dict[str, Any]]]:
+ active_cells = list(cells)
+ metas: List[Dict[str, Any]] = []
+ for layer_index in range(self.transformer_layers):
+ attended, attention_meta = self._cell_attention_layer(active_cells, layer_index=layer_index)
+ normalized, norm_meta = self._residual_norm_layer(attended, layer_index=layer_index)
+ active_cells, ff_meta = self._feed_forward_layer(normalized, layer_index=layer_index)
+ metas.extend([attention_meta, norm_meta, ff_meta])
+ return active_cells, metas
+
+ def _cell_attention_layer(
+ self,
+ cells: Sequence[TMCRACell],
+ *,
+ layer_index: int,
+ ) -> tuple[List[TMCRACell], Dict[str, Any]]:
+ state_cells = self._state_context_cells()
+ output: List[TMCRACell] = []
+ attended_edges = 0
+ for cell in cells:
+ head_scores = self._multi_head_scores(cell, state_cells)
+ best_key = ""
+ best_score = 0.0
+ if head_scores:
+ best_key, best_score = max(head_scores.items(), key=lambda item: item[1])
+ if best_key:
+ attended_edges += 1
+ cell.metadata = {
+ **cell.metadata,
+ "tmcra_layers": [*list(cell.metadata.get("tmcra_layers", [])), f"X{layer_index}.attn"],
+ "tmcra_attention_heads": {key: round(value, 6) for key, value in head_scores.items()},
+ "tmcra_attention_best_slot": best_key,
+ "tmcra_attention_best_score": round(best_score, 6),
+ "tmcra_attention_context_count": len(state_cells),
+ }
+ if best_key and best_score >= 0.42 and best_key in self._slot_state:
+ previous = self._slot_state[best_key]
+ cell.previous_value = previous.value
+ cell.recurrence_count = max(cell.recurrence_count, previous.recurrence_count + 1)
+ if not cell.subject and previous.subject:
+ cell.subject = previous.subject
+ if previous.slot_key and best_score >= 0.62:
+ cell.slot_key = previous.slot_key
+ cell.confidence = min(0.98, cell.confidence + min(0.08, best_score * 0.08))
+ output.append(cell)
+ return output, {
+ "layer": f"X{layer_index}.attn",
+ "name": "tmcra_cell_multi_head_attention",
+ "input_count": len(cells),
+ "output_count": len(output),
+ "state_context_count": len(state_cells),
+ "attended_edges": attended_edges,
+ }
+
+ def _residual_norm_layer(
+ self,
+ cells: Sequence[TMCRACell],
+ *,
+ layer_index: int,
+ ) -> tuple[List[TMCRACell], Dict[str, Any]]:
+ if not cells:
+ return [], {
+ "layer": f"X{layer_index}.norm",
+ "name": "tmcra_residual_layer_norm",
+ "input_count": 0,
+ "output_count": 0,
+ }
+ mean_conf = sum(cell.confidence for cell in cells) / len(cells)
+ mean_sal = sum(cell.salience for cell in cells) / len(cells)
+ output: List[TMCRACell] = []
+ for cell in cells:
+ attention_score = float(cell.metadata.get("tmcra_attention_best_score", 0.0) or 0.0)
+ # Residual path: preserve original L0 confidence/salience, then add
+ # a bounded attention-derived correction and normalize toward the
+ # turn mean so one noisy cell does not dominate the write budget.
+ cell.confidence = min(0.98, max(0.52, (cell.confidence * 0.82) + (mean_conf * 0.12) + (attention_score * 0.06)))
+ cell.salience = min(0.98, max(0.52, (cell.salience * 0.84) + (mean_sal * 0.10) + _category_weight(cell.category)))
+ cell.metadata = {
+ **cell.metadata,
+ "tmcra_layers": [*list(cell.metadata.get("tmcra_layers", [])), f"X{layer_index}.norm"],
+ "tmcra_residual_norm_mean_confidence": round(mean_conf, 6),
+ "tmcra_residual_norm_mean_salience": round(mean_sal, 6),
+ }
+ output.append(cell)
+ return output, {
+ "layer": f"X{layer_index}.norm",
+ "name": "tmcra_residual_layer_norm",
+ "input_count": len(cells),
+ "output_count": len(output),
+ }
+
+ def _feed_forward_layer(
+ self,
+ cells: Sequence[TMCRACell],
+ *,
+ layer_index: int,
+ ) -> tuple[List[TMCRACell], Dict[str, Any]]:
+ output: List[TMCRACell] = []
+ slot_refined = 0
+ for cell in cells:
+ original_slot = cell.slot_key
+ if not cell.subject:
+ cell.subject = self._subject_hint(cell.value, cell.semantic_slot)
+ subject_signature = _slug(cell.subject or cell.semantic_slot or cell.category)
+ refined_slot = ".".join(
+ part
+ for part in [
+ "tmcra",
+ _slug(cell.category),
+ _slug(cell.semantic_slot),
+ subject_signature,
+ ]
+ if part
+ )
+ if refined_slot and refined_slot != cell.slot_key:
+ cell.slot_key = refined_slot
+ slot_refined += int(bool(original_slot))
+ cell.value = _clip_text(cell.value, self.value_char_budget)
+ cell.source_span = _clip_text(cell.source_span, self.source_span_char_budget)
+ cell.metadata = {
+ **cell.metadata,
+ "tmcra_layers": [*list(cell.metadata.get("tmcra_layers", [])), f"X{layer_index}.ffn"],
+ "tmcra_ffn_refined_slot": cell.slot_key,
+ "tmcra_ffn_subject_signature": subject_signature,
+ "event_signature": _slug(f"{cell.category} {cell.semantic_slot} {subject_signature} {' '.join(_tokens(cell.value)[:6])}"),
+ }
+ output.append(cell)
+ return output, {
+ "layer": f"X{layer_index}.ffn",
+ "name": "tmcra_feed_forward_slot_refine",
+ "input_count": len(cells),
+ "output_count": len(output),
+ "slot_refined": slot_refined,
+ }
+
+ def _state_context_cells(self) -> List[TMCRACell]:
+ if self.state_attention_k <= 0:
+ return []
+ values = list(self._slot_state.values())
+ return sorted(
+ values,
+ key=lambda cell: (cell.recurrence_count, cell.confidence + cell.salience),
+ reverse=True,
+ )[: self.state_attention_k]
+
+ def _multi_head_scores(self, cell: TMCRACell, state_cells: Sequence[TMCRACell]) -> Dict[str, float]:
+ scores: Dict[str, float] = {}
+ cell_tokens = set(_tokens(" ".join([cell.value, cell.source_span, " ".join(cell.anchors)])))
+ for previous in state_cells:
+ previous_tokens = set(_tokens(" ".join([previous.value, previous.source_span, " ".join(previous.anchors)])))
+ union = max(1, len(cell_tokens | previous_tokens))
+ token_overlap = len(cell_tokens & previous_tokens) / union
+ slot_score = 1.0 if previous.slot_key == cell.slot_key and previous.slot_key else 0.0
+ category_score = 1.0 if previous.category == cell.category else 0.0
+ semantic_score = 1.0 if previous.semantic_slot == cell.semantic_slot else 0.0
+ subject_score = 1.0 if previous.subject and previous.subject == cell.subject else 0.0
+ recurrence_score = min(1.0, previous.recurrence_count / 4.0)
+ novelty_penalty = 0.18 if _safe_text(previous.value).lower() == _safe_text(cell.value).lower() else 0.0
+ head_score = max(
+ 0.0,
+ (0.30 * slot_score)
+ + (0.18 * semantic_score)
+ + (0.12 * category_score)
+ + (0.15 * subject_score)
+ + (0.20 * token_overlap)
+ + (0.05 * recurrence_score)
+ - novelty_penalty,
+ )
+ if head_score > 0:
+ scores[previous.slot_key] = max(scores.get(previous.slot_key, 0.0), head_score)
+ return scores
+
+
+class TMCRASuspectAnchoredTransformerMemoryWriter(TMCRATransformerSemanticMemoryWriter):
+ """Current-turn anchored transformer stack with a suspect memory buffer.
+
+ The suspect layer quarantines tentative or abstract cells before they can
+ become active recurrent state. Repeated compatible evidence can promote a
+ suspect cell back into the active path.
+ """
+
+ architecture = "tmcra_suspect_anchored_transformer_v1"
+
+ def __init__(
+ self,
+ base_writer: OpenAICompatSemanticMemoryWriter,
+ *,
+ max_cells: int = 2,
+ value_char_budget: int = 120,
+ source_span_char_budget: int = 180,
+ transformer_layers: int = 2,
+ state_attention_k: int = 16,
+ suspect_threshold: float = 0.48,
+ suspect_promote_count: int = 2,
+ ) -> None:
+ super().__init__(
+ base_writer,
+ max_cells=max_cells,
+ value_char_budget=value_char_budget,
+ source_span_char_budget=source_span_char_budget,
+ transformer_layers=transformer_layers,
+ state_attention_k=state_attention_k,
+ )
+ self.suspect_threshold = min(0.95, max(0.05, float(suspect_threshold or 0.48)))
+ self.suspect_promote_count = max(2, int(suspect_promote_count or 2))
+ self._suspect_state: Dict[str, TMCRACell] = {}
+ self._suspect_counts: Dict[str, int] = {}
+
+ def propose_public_turn(
+ self,
+ *,
+ current_turn: str,
+ previous_turn: str = "",
+ next_turn: str = "",
+ speaker: str = "",
+ session_timestamp: str = "",
+ sidecar_hints: Mapping[str, Any] | None = None,
+ auxiliary_evidence_texts: Sequence[Any] | None = None,
+ input_mode: str = "delta",
+ ) -> tuple[List[Dict[str, Any]], Dict[str, Any]]:
+ if not self.available():
+ raise SemanticMemoryWriterError("TMCRA suspect anchored writer requires an available base writer")
+ base_proposals, base_metadata = self.base_writer.propose_public_turn(
+ current_turn=current_turn,
+ previous_turn="",
+ next_turn="",
+ speaker=speaker,
+ session_timestamp=session_timestamp,
+ sidecar_hints={},
+ auxiliary_evidence_texts=[],
+ input_mode="delta",
+ )
+ turn_cells, turn_meta = self._turn_layer(
+ base_proposals,
+ current_turn=current_turn,
+ speaker=speaker,
+ session_timestamp=session_timestamp,
+ )
+ memory_cells, memory_meta = self._memory_layer(turn_cells, speaker=speaker)
+ active_cells, suspect_cells, suspect_meta = self._suspect_uncertainty_layer(
+ memory_cells,
+ current_turn=current_turn,
+ )
+ transformer_cells, transformer_meta = self._anchored_transformer_cell_layers(
+ active_cells,
+ current_turn=current_turn,
+ )
+ compressed_cells, compression_meta = self._compression_layer(transformer_cells)
+ recurrent_cells, recurrence_meta = self._recurrence_layer(compressed_cells, speaker=speaker)
+ activated_cells, activation_meta = self._activation_layer(recurrent_cells)
+ active_proposals = [cell.to_proposal() for cell in activated_cells[: self.max_proposals]]
+ suspect_budget = max(0, self.max_proposals - len(active_proposals))
+ suspect_proposals = [cell.to_proposal() for cell in suspect_cells[:suspect_budget]]
+ proposals = [*active_proposals, *suspect_proposals]
+ metadata = {
+ **dict(base_metadata or {}),
+ "architecture": self.architecture,
+ "input_mode": "delta",
+ "base_input_mode": dict(base_metadata or {}).get("input_mode", "delta"),
+ "system_prompt_profile": "tmcra_suspect_anchored_delta_l0",
+ "user_payload_keys": ["current_turn", "max_write_proposals", "speaker", "timestamp"],
+ "tmcra_layers": [
+ turn_meta,
+ memory_meta,
+ suspect_meta,
+ *transformer_meta,
+ compression_meta,
+ recurrence_meta,
+ activation_meta,
+ ],
+ "tmcra_stack_state_size": len(self._slot_state),
+ "tmcra_suspect_buffer_size": len(self._suspect_state),
+ "tmcra_transformer_layers": self.transformer_layers,
+ "tmcra_state_attention_k": self.state_attention_k,
+ "tmcra_suspect_threshold": self.suspect_threshold,
+ "tmcra_suspect_promote_count": self.suspect_promote_count,
+ "tmcra_stack_input_ignored": {
+ "previous_turn": bool(_safe_text(previous_turn)),
+ "next_turn": bool(_safe_text(next_turn)),
+ "sidecar_hints": bool(sidecar_hints),
+ "auxiliary_evidence_texts": bool(auxiliary_evidence_texts),
+ },
+ "proposal_count_after_stack": len(proposals),
+ "active_proposal_count_after_stack": len(active_proposals),
+ "suspect_proposal_count_after_stack": len(suspect_proposals),
+ }
+ return proposals, metadata
+
+ def _suspect_uncertainty_layer(
+ self,
+ cells: Sequence[TMCRACell],
+ *,
+ current_turn: str,
+ ) -> tuple[List[TMCRACell], List[TMCRACell], Dict[str, Any]]:
+ active: List[TMCRACell] = []
+ suspect: List[TMCRACell] = []
+ buffered = 0
+ promoted = 0
+ for cell in cells:
+ score, reasons = self._suspect_score(cell, current_turn=current_turn)
+ signature = _slug(
+ " ".join(
+ [
+ cell.category,
+ cell.semantic_slot,
+ cell.subject,
+ " ".join(_tokens(cell.value)[:8]),
+ ]
+ )
+ )
+ support_count = self._suspect_counts.get(signature, 0) + 1
+ should_buffer = score >= self.suspect_threshold and support_count < self.suspect_promote_count
+ layer_metadata = {
+ "tmcra_layers": [*list(cell.metadata.get("tmcra_layers", [])), "S"],
+ "tmcra_suspect_score": round(score, 6),
+ "tmcra_suspect_reasons": reasons,
+ "tmcra_suspect_signature": signature,
+ "tmcra_suspect_support_count": support_count,
+ "tmcra_suspect_threshold": self.suspect_threshold,
+ }
+ if should_buffer:
+ self._suspect_state[signature] = cell
+ self._suspect_counts[signature] = support_count
+ cell.confidence = min(cell.confidence, 0.56)
+ cell.salience = min(cell.salience, 0.58)
+ cell.metadata = {
+ **cell.metadata,
+ **layer_metadata,
+ "state": "suspect",
+ "relation": "suspect_memory",
+ "write_decision": "suspect_buffer",
+ "memory_gate_decision": "suspect_buffer",
+ "tmcra_suspect_buffered": True,
+ "suspicion_reason": ",".join(reasons) or "low_stability",
+ "source_layer": self.architecture,
+ }
+ suspect.append(cell)
+ buffered += 1
+ continue
+ if signature in self._suspect_state and support_count >= self.suspect_promote_count:
+ previous = self._suspect_state.pop(signature)
+ cell.previous_value = previous.value
+ cell.recurrence_count = max(cell.recurrence_count, support_count)
+ cell.confidence = min(0.9, max(cell.confidence, 0.74))
+ cell.salience = min(0.92, max(cell.salience, 0.76))
+ self._suspect_counts[signature] = support_count
+ promoted += 1
+ layer_metadata["tmcra_suspect_promoted"] = True
+ else:
+ self._suspect_counts[signature] = support_count
+ layer_metadata["tmcra_suspect_promoted"] = False
+ cell.metadata = {
+ **cell.metadata,
+ **layer_metadata,
+ "tmcra_suspect_buffered": False,
+ }
+ active.append(cell)
+ if not active and len(suspect) > 1:
+ release_index, release_cell = min(
+ enumerate(suspect),
+ key=lambda item: float(item[1].metadata.get("tmcra_suspect_score", 1.0) or 1.0),
+ )
+ suspect.pop(release_index)
+ release_cell.confidence = max(release_cell.confidence, 0.68)
+ release_cell.salience = max(release_cell.salience, 0.72)
+ release_cell.metadata = {
+ **release_cell.metadata,
+ "state": "active",
+ "relation": f"{release_cell.category}_memory",
+ "write_decision": "active_fallback",
+ "memory_gate_decision": "active_fallback",
+ "tmcra_suspect_buffered": False,
+ "tmcra_suspect_release_reason": "avoid_all_suspect_turn",
+ }
+ active.append(release_cell)
+ return active, suspect, {
+ "layer": "S",
+ "name": "suspect_uncertainty_buffer",
+ "input_count": len(cells),
+ "output_count": len(active),
+ "suspect_output_count": len(suspect),
+ "buffered": buffered,
+ "promoted": promoted,
+ "buffer_size": len(self._suspect_state),
+ "threshold": self.suspect_threshold,
+ }
+
+ def _suspect_score(self, cell: TMCRACell, *, current_turn: str) -> tuple[float, List[str]]:
+ del current_turn
+ text = " ".join([cell.value, cell.source_span])
+ cell_tokens = _token_set(cell.value, cell.source_span)
+ reasons: List[str] = []
+ score = 0.0
+ if _UNCERTAINTY_CUE_RE.search(text):
+ score += 0.24
+ reasons.append("uncertainty_cue")
+ abstract_ratio = _abstract_ratio(cell_tokens)
+ if abstract_ratio >= 0.28:
+ score += min(0.14, abstract_ratio * 0.6)
+ reasons.append("abstract_high_level_claim")
+ source_token_count = len(_token_set(cell.source_span))
+ if source_token_count <= 4:
+ score += 0.12
+ reasons.append("short_source_span")
+ if cell.category in {"profile", "preference", "goal"} and _UNCERTAINTY_CUE_RE.search(text):
+ score += 0.10
+ reasons.append("tentative_stable_category")
+ if cell.semantic_slot in {"preference", "identity_view", "exploration_topic"} and not cell.subject:
+ score += 0.08
+ reasons.append("weak_subject_anchor")
+ if cell.slot_key not in self._slot_state and cell.category in {"profile", "preference", "goal"}:
+ score += 0.04
+ reasons.append("first_observation")
+ if cell.category in {"time", "event", "constraint"}:
+ score = max(0.0, score - 0.12)
+ return min(1.0, score), reasons
+
+ def _anchored_transformer_cell_layers(
+ self,
+ cells: Sequence[TMCRACell],
+ *,
+ current_turn: str,
+ ) -> tuple[List[TMCRACell], List[Dict[str, Any]]]:
+ active_cells = list(cells)
+ metas: List[Dict[str, Any]] = []
+ current_tokens = _token_set(current_turn)
+ for layer_index in range(self.transformer_layers):
+ attended, attention_meta = self._anchored_cell_attention_layer(
+ active_cells,
+ layer_index=layer_index,
+ current_tokens=current_tokens,
+ )
+ normalized, norm_meta = self._residual_norm_layer(attended, layer_index=layer_index)
+ active_cells, ff_meta = self._feed_forward_layer(normalized, layer_index=layer_index)
+ metas.extend([attention_meta, norm_meta, ff_meta])
+ return active_cells, metas
+
+ def _anchored_cell_attention_layer(
+ self,
+ cells: Sequence[TMCRACell],
+ *,
+ layer_index: int,
+ current_tokens: set[str],
+ ) -> tuple[List[TMCRACell], Dict[str, Any]]:
+ state_cells = self._state_context_cells()
+ output: List[TMCRACell] = []
+ attended_edges = 0
+ gated_out_edges = 0
+ for cell in cells:
+ head_scores, gate_reasons = self._anchored_multi_head_scores(cell, state_cells, current_tokens)
+ best_key = ""
+ best_score = 0.0
+ if head_scores:
+ best_key, best_score = max(head_scores.items(), key=lambda item: item[1])
+ if best_key:
+ attended_edges += 1
+ else:
+ gated_out_edges += len(gate_reasons)
+ cell.metadata = {
+ **cell.metadata,
+ "tmcra_layers": [*list(cell.metadata.get("tmcra_layers", [])), f"X{layer_index}.attn"],
+ "tmcra_attention_heads": {key: round(value, 6) for key, value in head_scores.items()},
+ "tmcra_attention_best_slot": best_key,
+ "tmcra_attention_best_score": round(best_score, 6),
+ "tmcra_attention_context_count": len(state_cells),
+ "tmcra_current_anchor_tokens": len(current_tokens),
+ "tmcra_attention_gated_out": gate_reasons[:8],
+ }
+ if best_key and best_score >= 0.50 and best_key in self._slot_state:
+ previous = self._slot_state[best_key]
+ cell.previous_value = previous.value
+ cell.recurrence_count = max(cell.recurrence_count, previous.recurrence_count + 1)
+ if not cell.subject and previous.subject:
+ cell.subject = previous.subject
+ if previous.slot_key and best_score >= 0.70:
+ cell.slot_key = previous.slot_key
+ cell.confidence = min(0.96, cell.confidence + min(0.05, best_score * 0.05))
+ output.append(cell)
+ return output, {
+ "layer": f"X{layer_index}.attn",
+ "name": "tmcra_current_anchored_multi_head_attention",
+ "input_count": len(cells),
+ "output_count": len(output),
+ "state_context_count": len(state_cells),
+ "attended_edges": attended_edges,
+ "gated_out_edges": gated_out_edges,
+ }
+
+ def _anchored_multi_head_scores(
+ self,
+ cell: TMCRACell,
+ state_cells: Sequence[TMCRACell],
+ current_tokens: set[str],
+ ) -> tuple[Dict[str, float], List[str]]:
+ scores: Dict[str, float] = {}
+ gate_reasons: List[str] = []
+ cell_tokens = _token_set(cell.value, cell.source_span, " ".join(cell.anchors))
+ for previous in state_cells:
+ previous_tokens = _token_set(previous.value, previous.source_span, " ".join(previous.anchors))
+ previous_current_overlap = _overlap_ratio(previous_tokens, current_tokens)
+ cell_previous_overlap = _overlap_ratio(cell_tokens, previous_tokens)
+ slot_score = 1.0 if previous.slot_key == cell.slot_key and previous.slot_key else 0.0
+ semantic_score = 1.0 if previous.semantic_slot == cell.semantic_slot else 0.0
+ category_score = 1.0 if previous.category == cell.category else 0.0
+ subject_score = 1.0 if previous.subject and previous.subject == cell.subject else 0.0
+ recurrence_score = min(1.0, previous.recurrence_count / 4.0)
+ generic_penalty = min(0.22, _abstract_ratio(previous_tokens) * 0.5)
+ novelty_penalty = 0.18 if _safe_text(previous.value).lower() == _safe_text(cell.value).lower() else 0.0
+ anchor_ok = previous_current_overlap >= 0.08 or slot_score > 0 or subject_score > 0
+ if not anchor_ok:
+ gate_reasons.append(previous.slot_key)
+ continue
+ head_score = max(
+ 0.0,
+ (0.28 * previous_current_overlap)
+ + (0.22 * slot_score)
+ + (0.16 * semantic_score)
+ + (0.10 * category_score)
+ + (0.10 * subject_score)
+ + (0.10 * cell_previous_overlap)
+ + (0.04 * recurrence_score)
+ - generic_penalty
+ - novelty_penalty,
+ )
+ if head_score > 0:
+ scores[previous.slot_key] = max(scores.get(previous.slot_key, 0.0), head_score)
+ return scores, gate_reasons
+
+
+__all__ = [
+ "MultiLayerTMCRASemanticMemoryWriter",
+ "TMCRACell",
+ "TMCRASuspectAnchoredTransformerMemoryWriter",
+ "TMCRATransformerSemanticMemoryWriter",
+]
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/node_memory.py b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/node_memory.py
new file mode 100644
index 0000000000000000000000000000000000000000..9034464dcb5d07363e1546979adc93c5a3a7883f
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/node_memory.py
@@ -0,0 +1,13776 @@
+from __future__ import annotations
+
+from collections import Counter, OrderedDict, deque
+from concurrent.futures import FIRST_COMPLETED, Future, ProcessPoolExecutor, ThreadPoolExecutor, wait
+from concurrent.futures.process import BrokenProcessPool
+from contextlib import contextmanager, nullcontext
+from dataclasses import dataclass, field
+from datetime import datetime
+import errno
+from functools import lru_cache
+import gc
+import hashlib
+import inspect
+import itertools
+import json
+from json import encoder as json_encoder
+import math
+import multiprocessing
+import os
+import random
+import re
+import threading
+import tempfile
+import time
+from pathlib import Path
+from typing import Any, Callable, Dict, Iterable, Iterator, List, Mapping, Sequence
+
+import torch
+from torch import Tensor, nn
+import torch.nn.functional as F
+
+
+QUESTION_HASH_BUCKETS = 4096
+NODE_HASH_BUCKETS = 4096
+QUESTION_EMBED_DIM = 256
+NODE_EMBED_DIM = 256
+NODE_TYPE_EMBED_DIM = 16
+QUESTION_CATEGORICAL_EMBED_DIM = 16
+EDGE_TYPE_EMBED_DIM = 16
+NODE_SCALAR_DIM = 8
+NODE_SCALAR_HIDDEN_DIM = 32
+QUESTION_OUTPUT_DIM = 288
+NODE_OUTPUT_DIM = 304
+MESSAGE_HIDDEN_DIM = 448
+MESSAGE_DROPOUT = 0.1
+MODEL_ARCH_VERSION = "node_memory_v11_tmcra_scale49m_trimaze"
+QUESTION_ENCODER_VARIANT = "hashed_tmcra_token_attn_pool_6l_w256"
+NODE_ENCODER_VARIANT = "hashed_tmcra_token_attn_pool_6l_w256"
+EVENT_SUBGRAPH_REFINER_VARIANT = "event_centered_subgraph_attn_6l_h448"
+EVENT_DISTRACTOR_VARIANT = "candidate_distractor_delta_bce_v1"
+MEMORY_TUNNEL_VARIANT = "chain_depth_tunnel_support_delta_v1"
+ENCODER_STRUCTURAL_BIAS_VARIANT = "tmcra_token_role_relbias_v1"
+PAIR_FEATURE_ADAPTER_VARIANT = "residual_pair_feature_adapter_v1"
+ANSWER_CALIBRATION_VARIANT = "competition_alignment_v3_trimaze"
+QUESTION_INTENT_VARIANT = "text_pooled_multihead_v1"
+MEMORY_ROUTER_VARIANT = "question_pooled_multilabel_v1"
+MESSAGE_PASSING_VARIANT = "support_to_event_bidirectional_shared_typed_v1"
+ANSWER_PLAN_VARIANT = "candidate_event_answer_plan_v1"
+QUESTION_MAX_TOKENS = 48
+NODE_MAX_TOKENS = 32
+ENCODER_ATTENTION_LAYERS = 6
+ENCODER_ATTENTION_HEADS = 8
+EVENT_SUBGRAPH_ATTENTION_LAYERS = 6
+EVENT_SUBGRAPH_ATTENTION_HEADS = 8
+TMCRA_TOKEN_ROLE_COUNT = 8
+TMCRA_TOKEN_RELATION_FEATURE_DIM = 9
+PAIR_FEATURE_DIM = 29
+PATH_PAIR_FEATURE_DIM = 22
+ANSWER_CALIBRATION_FEATURE_NAMES = (
+ "event_prob_top1",
+ "event_prob_margin",
+ "event_prob_entropy",
+ "event_prob_top3_mass",
+ "path_prob_top1",
+ "path_prob_margin",
+ "path_prob_entropy",
+ "path_prob_top3_mass",
+ "temporal_prob_top1",
+ "temporal_prob_margin",
+ "temporal_prob_entropy",
+ "temporal_prob_top3_mass",
+ "event_score_top1",
+ "event_score_margin",
+ "path_score_top1",
+ "path_score_margin",
+ "temporal_score_top1",
+ "temporal_score_margin",
+ "is_temporal_question",
+ "has_event_candidates",
+ "has_path_candidates",
+ "has_temporal_candidates",
+ "speaker_conflict_rate",
+ "time_conflict_rate",
+ "temporal_distractor_rate",
+ "best_exact_signature_cover",
+ "best_source_turn_support",
+ "best_preferred_path",
+ "best_support_anchor",
+ "top_event_path_support",
+ "top_event_temporal_support",
+ "top_path_aligns_top_event",
+ "event_reverse_prob_top1",
+ "event_reverse_relation_top1",
+ "event_boundary_gap_top1",
+ "event_reverse_available_top1",
+ "path_reverse_prob_top1",
+ "path_reverse_relation_top1",
+ "path_boundary_gap_top1",
+ "path_reverse_available_top1",
+)
+ANSWER_CALIBRATION_FEATURE_DIM = len(ANSWER_CALIBRATION_FEATURE_NAMES)
+ANSWER_CALIBRATION_FEATURE_INDEX = {
+ name: index for index, name in enumerate(ANSWER_CALIBRATION_FEATURE_NAMES)
+}
+EVENT_DISTRACTOR_FEATURE_DIM = 17
+EVENT_RUNTIME_CALIBRATION_FEATURE_DIM = 22
+PATH_RUNTIME_CALIBRATION_FEATURE_DIM = 27
+EVENT_TUNNEL_FEATURE_DIM = 20
+PATH_TUNNEL_FEATURE_DIM = 14
+EVENT_TUNNEL_DELTA_LOGIT_LIMIT = 3.0
+PATH_TUNNEL_DELTA_LOGIT_LIMIT = 3.0
+FINAL_EVENT_FUSION_FEATURE_DIM = 40
+FINAL_PATH_FUSION_FEATURE_DIM = 46
+ANSWER_PLAN_OUTPUTS = ("selected", "current", "historical", "suppressed")
+ANSWER_PLAN_OUTPUT_TO_ID = {value: index for index, value in enumerate(ANSWER_PLAN_OUTPUTS)}
+ANSWER_PLAN_FEATURE_DIM = 12
+DEFAULT_RUNTIME_RECALL_TOP_K = 24
+DEFAULT_MATRIX_EVENT_TOP_K = 16
+DEFAULT_MATRIX_EVENT_RECALL_SEED_TOP_K = 12
+DEFAULT_MATRIX_EVENT_HARD_NEGATIVE_LIMIT = 4
+MATRIX_EVENT_DELTA_LOGIT_LIMIT = 4.0
+MATRIX_RELATION_FEATURE_DIM = 6
+DEFAULT_MATRIX_PATH_TOP_K = 12
+DEFAULT_MATRIX_PATH_RECALL_SEED_TOP_K = 8
+DEFAULT_MATRIX_PATH_HARD_NEGATIVE_LIMIT = 4
+MATRIX_PATH_DELTA_LOGIT_LIMIT = 4.0
+PATH_MATRIX_RELATION_FEATURE_DIM = 6
+TRI_MAZE_EVENT_RELATION_WEIGHTS = (0.32, 0.16, 0.12, 0.12, 0.08, 0.20)
+TRI_MAZE_PATH_RELATION_WEIGHTS = (0.34, 0.10, 0.20, 0.10, 0.16, 0.10)
+TRI_MAZE_RELATION_THRESHOLD = 0.15
+GRAPH_TENSOR_CACHE_VERSION = 2
+DEFAULT_BATCH_PREPARE_WORKER_MAX_TASKS_PER_CHILD = 0
+DEFAULT_BATCH_PREPARE_THREAD_FALLBACK_WORKERS_MULTIPLIER = 2
+DEFAULT_BATCH_PREPARE_THREAD_FALLBACK_MIN_WORKERS = 2
+DEFAULT_BATCH_PREPARE_PENDING_MULTIPLIER = 3
+DEFAULT_BATCH_PREPARE_PREFER_COMPLETION_ORDER = True
+DEFAULT_BATCH_PREPARE_PROCESS_RECOVERY_ATTEMPTS = 3
+DEFAULT_BATCH_PREPARE_PROCESS_WARMUP_PENDING_MULTIPLIER = 2
+MAX_LAZY_GRAPH_PATHS_MATERIALIZE = 100000
+GRAPH_TENSOR_VALUE_KEYS = frozenset(
+ {
+ "node_type_ids",
+ "node_scalar_features",
+ "edge_src",
+ "edge_dst",
+ "edge_type_ids",
+ }
+)
+_GRAPH_TENSOR_CACHE_RECOVERABLE_ERRNOS = frozenset(
+ int(value)
+ for value in (
+ getattr(errno, "EUCLEAN", None),
+ 117,
+ )
+ if isinstance(value, int)
+)
+_graph_tensor_cache_write_warning_lock = threading.Lock()
+_graph_tensor_cache_write_warning_keys: set[tuple[str, str, int | None]] = set()
+
+
+NODE_TYPES = ("speaker", "event", "time", "profile", "status", "source_turn")
+EDGE_TYPES = (
+ "speaker_of",
+ "time_of",
+ "profile_of",
+ "status_of",
+ "supported_by_turn",
+ "same_session_next",
+)
+PATH_TYPES = (
+ "speaker_event_time",
+ "speaker_event_profile",
+ "speaker_event_status",
+ "speaker_event_source_turn",
+)
+EVENT_SUPPORT_PATH_TYPE_ORDER = PATH_TYPES
+ANSWER_TYPES = ("time", "profile", "event_text", "multi_evidence", "abstain")
+SEMANTIC_SLOTS = ("event", "identity", "research_topic", "education", "occupation", "event_time", "profile")
+TARGET_STATUSES = ("", "past", "current", "planned")
+TIME_GRANULARITIES = ("", "day", "month", "year", "relative_day_reference", "none", "day_or_coarse")
+EVENT_PAIR_FEATURE_MODES = ("full", "zero")
+MEMORY_ROUTER_LAYERS = ("event", "profile", "resource", "temporal", "path_tunnel", "topic_tunnel")
+MEMORY_ROUTER_LAYER_TO_ID = {value: index for index, value in enumerate(MEMORY_ROUTER_LAYERS)}
+DEPTH_LAYERS = ("core_view", "deep_view", "mechanism", "risk", "metric", "product_view", "evidence")
+CHAIN_HINT_TOKENS = frozenset(
+ {
+ "chain",
+ "context",
+ "depth",
+ "graph",
+ "memory",
+ "mechanism",
+ "metric",
+ "risk",
+ "tunnel",
+ "view",
+ }
+)
+
+TMCRA_TOKEN_ROLE_CONTENT = 0
+TMCRA_TOKEN_ROLE_ANCHOR = 1
+TMCRA_TOKEN_ROLE_SPEAKER = 2
+TMCRA_TOKEN_ROLE_TEMPORAL = 3
+TMCRA_TOKEN_ROLE_PROFILE = 4
+TMCRA_TOKEN_ROLE_STATUS = 5
+TMCRA_TOKEN_ROLE_SOURCE = 6
+TMCRA_TOKEN_ROLE_EMPTY = 7
+
+NODE_TYPE_TO_ID = {value: index for index, value in enumerate(NODE_TYPES)}
+EDGE_TYPE_TO_ID = {value: index for index, value in enumerate(EDGE_TYPES)}
+PATH_TYPE_TO_ID = {value: index for index, value in enumerate(PATH_TYPES)}
+ANSWER_TYPE_TO_ID = {value: index for index, value in enumerate(ANSWER_TYPES)}
+SEMANTIC_SLOT_TO_ID = {value: index for index, value in enumerate(SEMANTIC_SLOTS)}
+TARGET_STATUS_TO_ID = {value: index for index, value in enumerate(TARGET_STATUSES)}
+TIME_GRANULARITY_TO_ID = {value: index for index, value in enumerate(TIME_GRANULARITIES)}
+DEPTH_LAYER_TO_ID = {value: index for index, value in enumerate(DEPTH_LAYERS)}
+
+
+def node_memory_arch_metadata() -> Dict[str, Any]:
+ return {
+ "model_arch_version": MODEL_ARCH_VERSION,
+ "question_encoder_variant": QUESTION_ENCODER_VARIANT,
+ "node_encoder_variant": NODE_ENCODER_VARIANT,
+ "event_subgraph_refiner_variant": EVENT_SUBGRAPH_REFINER_VARIANT,
+ "event_distractor_variant": EVENT_DISTRACTOR_VARIANT,
+ "memory_tunnel_variant": MEMORY_TUNNEL_VARIANT,
+ "encoder_structural_bias_variant": ENCODER_STRUCTURAL_BIAS_VARIANT,
+ "pair_feature_adapter_variant": PAIR_FEATURE_ADAPTER_VARIANT,
+ "answer_calibration_variant": ANSWER_CALIBRATION_VARIANT,
+ "question_intent_variant": QUESTION_INTENT_VARIANT,
+ "memory_router_variant": MEMORY_ROUTER_VARIANT,
+ "message_passing_variant": MESSAGE_PASSING_VARIANT,
+ "answer_plan_variant": ANSWER_PLAN_VARIANT,
+ "question_max_tokens": QUESTION_MAX_TOKENS,
+ "node_max_tokens": NODE_MAX_TOKENS,
+ "encoder_attention_layers": ENCODER_ATTENTION_LAYERS,
+ "encoder_attention_heads": ENCODER_ATTENTION_HEADS,
+ "event_subgraph_attention_layers": EVENT_SUBGRAPH_ATTENTION_LAYERS,
+ "event_subgraph_attention_heads": EVENT_SUBGRAPH_ATTENTION_HEADS,
+ }
+
+
+def extract_node_memory_checkpoint_arch(payload: Mapping[str, Any]) -> Dict[str, Any]:
+ config = dict(payload.get("config", {}) or {})
+ metadata = dict(payload.get("metadata", {}) or {})
+ extracted: Dict[str, Any] = {}
+ for key, default_value in node_memory_arch_metadata().items():
+ if key in config:
+ value = config.get(key)
+ elif key in metadata:
+ value = metadata.get(key)
+ else:
+ value = default_value if key != "model_arch_version" else ""
+ if value is None or value == "":
+ value = default_value if key != "model_arch_version" else ""
+ extracted[key] = value
+ return extracted
+
+
+def validate_node_memory_checkpoint_payload(payload: Mapping[str, Any], *, context: str) -> Dict[str, Any]:
+ arch = extract_node_memory_checkpoint_arch(payload)
+ found_version = clean_text(arch.get("model_arch_version", ""))
+ if found_version != MODEL_ARCH_VERSION:
+ found_label = found_version or ""
+ raise ValueError(
+ f"{context}: incompatible checkpoint architecture; expected "
+ f"model_arch_version='{MODEL_ARCH_VERSION}', found '{found_label}'. "
+ "Old architecture checkpoints can be archived, but cannot be resumed or loaded into the new scorer."
+ )
+ return arch
+
+
+DEFAULT_TRAINING_CONFIG: Dict[str, Any] = {
+ "epochs": 12,
+ "batch_size": 16,
+ "max_train_steps": 0,
+ "lr": 2e-4,
+ "weight_decay": 0.01,
+ "warmup_ratio": 0.06,
+ "grad_clip": 1.0,
+ "early_stopping_patience": 3,
+ "log_every_steps": 100,
+ "checkpoint_every_steps": 0,
+ "amp": True,
+ "graph_prefetch_lookahead_batches": 3,
+ "batch_prepare_workers": 0,
+ "batch_prepare_lookahead_batches": 0,
+ "recall_loss_weight": 0.6,
+ "event_loss_weight": 1.1,
+ "path_loss_weight": 0.55,
+ "temporal_loss_weight": 0.3,
+ "answer_type_loss_weight": 0.2,
+ "answer_plan_loss_weight": 0.18,
+ "token_role_loss_weight": 0.08,
+ "question_understanding_loss_weight": 0.12,
+ "memory_router_loss_weight": 0.18,
+ "event_distractor_loss_weight": 0.15,
+ "event_tunnel_loss_weight": 0.12,
+ "path_tunnel_loss_weight": 0.06,
+ "path_tunnel_delta_loss_weight": 0.12,
+ "event_tunnel_selection_loss_weight": 0.08,
+ "path_tunnel_selection_loss_weight": 0.04,
+ "event_hard_negative_loss_weight": 0.35,
+ "path_hard_negative_loss_weight": 0.25,
+ "recall_selection_loss_weight": 0.3,
+ "event_selection_loss_weight": 0.35,
+ "path_selection_loss_weight": 0.3,
+ "final_event_set_loss_weight": 0.35,
+ "event_matrix_delta_loss_weight": 0.08,
+ "path_matrix_delta_loss_weight": 0.08,
+ "event_matrix_delta_margin": 0.15,
+ "path_matrix_delta_margin": 0.12,
+ "event_tunnel_margin": 0.16,
+ "path_tunnel_margin": 0.16,
+ "online_event_hard_negative_limit": 6,
+ "online_path_hard_negative_limit": 8,
+ "trainable_stage": "all",
+ "event_hard_negative_margin": 0.35,
+ "path_hard_negative_margin": 0.28,
+ "recall_selection_margin": 0.12,
+ "event_selection_margin": 0.2,
+ "path_selection_margin": 0.15,
+ "recall_selection_top_k": DEFAULT_RUNTIME_RECALL_TOP_K,
+ "event_selection_top_k": 5,
+ "path_selection_top_k": 3,
+ "recall_selection_positive_coverage_count": 3,
+ "event_selection_positive_coverage_count": 3,
+ "path_selection_positive_coverage_count": 2,
+ "final_event_set_margin": 0.08,
+ "final_event_set_top_k": 10,
+ "final_event_set_support_path_k": 3,
+ "final_event_set_positive_coverage_count": 3,
+ "multi_positive_coverage_fraction": 0.6,
+ "multi_positive_recall_coverage_count": 6,
+ "multi_positive_event_coverage_count": 5,
+ "multi_positive_path_coverage_count": 3,
+ "multi_positive_final_event_set_coverage_count": 5,
+ "answer_refusal_loss_weight": 0.08,
+ "answer_refusal_margin": 0.15,
+ "answer_plan_selection_margin": 0.12,
+ "answer_plan_selection_top_k": 5,
+ "answer_plan_current_old_margin_loss_weight": 0.0,
+ "answer_plan_current_old_margin": 0.2,
+ "answer_plan_selected_negative_margin_loss_weight": 0.0,
+ "answer_plan_selected_negative_margin": 0.3,
+ "train_sampling_mode": "source_aware_balanced",
+ "sampling_source_alpha": 0.35,
+ "sampling_blend_uniform_ratio": 0.35,
+ "sampling_time_boost": 1.2,
+ "sampling_multi_evidence_boost": 1.45,
+ "sampling_temporal_positive_boost": 1.15,
+ "sampling_max_conversation_multiplier": 2.0,
+ "sampling_max_group_repeat": 2,
+ "loss_source_alpha": 0.4,
+ "loss_blend_uniform_ratio": 0.25,
+ "loss_weight_power": 0.5,
+ "loss_time_boost": 1.45,
+ "loss_multi_evidence_boost": 1.75,
+ "loss_temporal_positive_boost": 1.2,
+ "loss_min_example_weight": 0.65,
+ "loss_max_example_weight": 1.85,
+ "loss_group_balancing_mode": "supervision_bucket",
+ "l2sp_loss_weight": 0.0,
+ **node_memory_arch_metadata(),
+}
+
+_METRIC_WEIGHT_KEYS: Dict[str, str] = {
+ "loss": "samples",
+ "recall_loss": "recall_loss_count",
+ "event_loss": "event_loss_count",
+ "path_loss": "path_loss_count",
+ "temporal_loss": "temporal_loss_count",
+ "answer_type_loss": "answer_type_loss_count",
+ "answer_plan_loss": "answer_plan_loss_count",
+ "token_role_loss": "token_role_loss_count",
+ "question_understanding_loss": "question_understanding_loss_count",
+ "memory_router_loss": "memory_router_loss_count",
+ "memory_router_exact_match": "memory_router_total",
+ "memory_router_f1": "memory_router_total",
+ "event_distractor_loss": "event_distractor_loss_count",
+ "event_tunnel_loss": "event_tunnel_loss_count",
+ "path_tunnel_loss": "path_tunnel_loss_count",
+ "path_tunnel_delta_loss": "path_tunnel_delta_loss_count",
+ "event_tunnel_selection_loss": "event_tunnel_selection_loss_count",
+ "path_tunnel_selection_loss": "path_tunnel_selection_loss_count",
+ "event_hard_negative_loss": "event_hard_negative_loss_count",
+ "path_hard_negative_loss": "path_hard_negative_loss_count",
+ "recall_selection_loss": "recall_selection_loss_count",
+ "event_selection_loss": "event_selection_loss_count",
+ "path_selection_loss": "path_selection_loss_count",
+ "final_event_set_loss": "final_event_set_loss_count",
+ "event_matrix_delta_loss": "event_matrix_delta_loss_count",
+ "path_matrix_delta_loss": "path_matrix_delta_loss_count",
+ "answer_refusal_loss": "answer_refusal_loss_count",
+ "recall_event_recall_at_24": "recall_event_recall_total",
+ "recall_event_positive_coverage_at_24": "recall_event24_positive_total",
+ "event_recall_at_1": "event_recall_total",
+ "event_recall_at_5": "event_recall_total",
+ "event_positive_coverage_at_5": "event5_positive_total",
+ "path_recall_at_3": "path_recall_total",
+ "path_positive_coverage_at_3": "path3_positive_total",
+ "path_tunnel_support_recall_at_3": "path_tunnel_support_recall_total",
+ "path_tunnel_support_positive_coverage_at_3": "path_tunnel_support_positive_total",
+ "path_tunnel_delta_recall_at_3": "path_tunnel_delta_recall_total",
+ "path_tunnel_delta_positive_coverage_at_3": "path_tunnel_delta_positive_total",
+ "answer_plan_selected_recall_at_5": "answer_plan_selected_total",
+ "answer_plan_selected_positive_coverage_at_5": "answer_plan_selected_positive_total",
+ "answer_plan_current_top1_accuracy": "answer_plan_current_total",
+ "path_tunnel_rescue025_recall_at_3": "path_tunnel_rescue025_recall_total",
+ "path_tunnel_rescue050_recall_at_3": "path_tunnel_rescue050_recall_total",
+ "path_tunnel_rescue100_recall_at_3": "path_tunnel_rescue100_recall_total",
+ "temporal_accuracy": "temporal_total",
+}
+_METRIC_COUNT_KEYS = {"samples", "graph_error_count", "loss_group_count", *set(_METRIC_WEIGHT_KEYS.values())}
+
+
+_TOKEN_RE = re.compile(r"[a-z0-9_]+", re.IGNORECASE)
+_MONTH_MARKERS = (
+ "january",
+ "february",
+ "march",
+ "april",
+ "may",
+ "june",
+ "july",
+ "august",
+ "september",
+ "october",
+ "november",
+ "december",
+)
+_NON_SPEAKER_CAPITALIZED_WORDS = {
+ "What",
+ "When",
+ "Which",
+ "Where",
+ "Who",
+ "Why",
+ "How",
+ "Today",
+ "Tomorrow",
+ "Yesterday",
+ "Monday",
+ "Tuesday",
+ "Wednesday",
+ "Thursday",
+ "Friday",
+ "Saturday",
+ "Sunday",
+ "January",
+ "February",
+ "March",
+ "April",
+ "May",
+ "June",
+ "July",
+ "August",
+ "September",
+ "October",
+ "November",
+ "December",
+}
+_QUESTION_SPEAKER_PREPOSITIONS = {
+ "at",
+ "about",
+ "around",
+ "by",
+ "during",
+ "for",
+ "from",
+ "in",
+ "inside",
+ "into",
+ "near",
+ "of",
+ "on",
+ "over",
+ "through",
+ "to",
+ "under",
+ "with",
+}
+_QUESTION_SPEAKER_AUXILIARIES = (
+ "did",
+ "does",
+ "do",
+ "is",
+ "was",
+ "were",
+ "has",
+ "have",
+ "had",
+ "will",
+ "would",
+ "can",
+ "could",
+ "should",
+ "may",
+ "might",
+)
+_WEEKDAY_MARKERS = {
+ "monday",
+ "tuesday",
+ "wednesday",
+ "thursday",
+ "friday",
+ "saturday",
+ "sunday",
+ "mon",
+ "tue",
+ "wed",
+ "thu",
+ "fri",
+ "sat",
+ "sun",
+}
+_TMCRA_TEMPORAL_TOKEN_HINTS = {
+ "today",
+ "tomorrow",
+ "yesterday",
+ "tonight",
+ "morning",
+ "afternoon",
+ "evening",
+ "night",
+ "week",
+ "weekend",
+ "month",
+ "year",
+ "day",
+ *_MONTH_MARKERS,
+ *_WEEKDAY_MARKERS,
+}
+_QUESTION_EDGE_GLUE_TOKENS = {
+ "a",
+ "an",
+ "the",
+ "to",
+ "as",
+ "at",
+ "in",
+ "on",
+ "of",
+ "for",
+ "with",
+ "from",
+ "about",
+ "around",
+ "into",
+ "over",
+ "under",
+}
+_QUESTION_LEADING_FUNCTION_RE = re.compile(
+ rf"^\s*(?:what|when|which|where|who|why|how)\b(?:\s+(?:{'|'.join(_QUESTION_SPEAKER_AUXILIARIES)}))?",
+ re.IGNORECASE,
+)
+_TEMPORAL_REFERENCE_RE = re.compile(
+ r"\b(?:when|date|day|month|year|today|tomorrow|yesterday|tonight|morning|afternoon|evening|night|week(?:end)?|last|next|before|after)\b",
+ re.IGNORECASE,
+)
+_TEMPORAL_ANSWER_PREFIX_RE = re.compile(r"^\s*(?:when|since\s+when)\b", re.IGNORECASE)
+_TEMPORAL_ANSWER_WH_RE = re.compile(r"^\s*(?:what|which)\s+(?:date|day|month|year|time)\b", re.IGNORECASE)
+_TEMPORAL_ANSWER_DURATION_RE = re.compile(
+ r"^\s*how\s+(?:long|many\s+(?:days?|weeks?|months?|years?))\b",
+ re.IGNORECASE,
+)
+_EDUCATION_YEAR_RE = re.compile(
+ r"\b(?:year|semester|grade)\s+(?:in|of|at)\s+(?:\w+\s+){0,3}(?:school|college|university|uni)\b",
+ re.IGNORECASE,
+)
+_PROFILE_COPULAR_RE = re.compile(r"^\s*who\s+is\b", re.IGNORECASE)
+_PROFILE_ATTRIBUTE_RE = re.compile(r"^\s*(?:what|where)\s+(?:does|do|is|are|was|were|has|have|had)\b", re.IGNORECASE)
+_PROFILE_QUERY_HINT_RE = re.compile(
+ r"\b(?:occupation|job|work(?:ing)?|study|studying|research|major(?:ing)?|profession|career|school|college|university)\b",
+ re.IGNORECASE,
+)
+_PROFILE_POSSESSIVE_RE = re.compile(r"\b[A-Za-z][A-Za-z'-]+(?:\s+[A-Za-z][A-Za-z'-]+)?'s\b", re.IGNORECASE)
+_PLANNED_STATUS_RE = re.compile(
+ r"\b(?:will|going\s+to|plan(?:ning)?\s+to|scheduled\s+to|hoping\s+to|preparing\s+to)\b",
+ re.IGNORECASE,
+)
+_CURRENT_STATUS_RE = re.compile(r"\b(?:currently|now|still)\b|\b(?:is|are|am)\s+\w+ing\b", re.IGNORECASE)
+_PAST_STATUS_RE = re.compile(r"\b(?:did|was|were|had|has)\b", re.IGNORECASE)
+
+
+def clean_text(value: Any) -> str:
+ text = str(value or "").strip()
+ if not text:
+ return ""
+ return " ".join(text.split())
+
+
+def _env_flag(name: str, *, default: bool = True) -> bool:
+ raw = os.environ.get(name)
+ if raw is None:
+ return bool(default)
+ normalized = clean_text(raw).lower()
+ if normalized in {"0", "false", "no", "off", "disabled"}:
+ return False
+ if normalized in {"1", "true", "yes", "on", "enabled"}:
+ return True
+ return bool(default)
+
+
+def normalize_text(value: Any) -> str:
+ return clean_text(value).lower()
+
+
+@lru_cache(maxsize=32768)
+def _tokenize_normalized_text_cached(text: str) -> tuple[str, ...]:
+ if not text:
+ return ()
+ english = tuple(_TOKEN_RE.findall(text))
+ cjk = tuple(char for char in text if "\u4e00" <= char <= "\u9fff")
+ if english or cjk:
+ return tuple([*english, *cjk])
+ return tuple(char for char in text if char.strip())
+
+
+@lru_cache(maxsize=32768)
+def _normalized_token_set_cached(text: str) -> frozenset[str]:
+ return frozenset(_tokenize_normalized_text_cached(text))
+
+
+@lru_cache(maxsize=32768)
+def _normalized_token_bigram_set_cached(text: str) -> frozenset[tuple[str, str]]:
+ tokens = _tokenize_normalized_text_cached(text)
+ if len(tokens) < 2:
+ return frozenset()
+ return frozenset((tokens[index], tokens[index + 1]) for index in range(len(tokens) - 1))
+
+
+def tokenize_text(value: Any) -> List[str]:
+ text = normalize_text(value)
+ if not text:
+ return []
+ return list(_tokenize_normalized_text_cached(text))
+
+
+def dedupe_texts(values: Iterable[Any], *, max_items: int | None = None) -> List[str]:
+ result: List[str] = []
+ seen = set()
+ for value in values:
+ text = clean_text(value)
+ if not text:
+ continue
+ key = normalize_text(text)
+ if key in seen:
+ continue
+ seen.add(key)
+ result.append(text)
+ if max_items is not None and len(result) >= max_items:
+ break
+ return result
+
+
+class GraphTensorValidationError(RuntimeError):
+ pass
+
+
+def _format_debug_context(context: Mapping[str, Any] | None = None) -> str:
+ if not context:
+ return ""
+ parts: List[str] = []
+ for key, value in context.items():
+ normalized_key = clean_text(key)
+ if not normalized_key:
+ continue
+ if isinstance(value, (list, tuple)):
+ preview = [clean_text(item) for item in list(value)[:8]]
+ suffix = "" if len(value) <= 8 else "..."
+ rendered = f"[{', '.join(item for item in preview if item)}{suffix}]"
+ else:
+ rendered = clean_text(value)
+ parts.append(f"{normalized_key}={rendered}")
+ return ", ".join(part for part in parts if part)
+
+
+def _raise_index_validation_error(
+ index_name: str,
+ *,
+ context: str,
+ invalid_value: int,
+ lower_bound: int,
+ upper_bound: int,
+ extra: Mapping[str, Any] | None = None,
+) -> None:
+ context_payload = _format_debug_context(extra)
+ suffix = f" ({context_payload})" if context_payload else ""
+ raise GraphTensorValidationError(
+ f"{context}: {index_name}={int(invalid_value)} outside [{int(lower_bound)}, {int(upper_bound)}){suffix}"
+ )
+
+
+def _validate_index_values(
+ values: Sequence[int],
+ *,
+ upper_bound: int,
+ index_name: str,
+ context: str,
+ extra: Mapping[str, Any] | None = None,
+) -> None:
+ resolved_upper_bound = int(upper_bound)
+ if resolved_upper_bound <= 0:
+ if values:
+ _raise_index_validation_error(
+ index_name,
+ context=context,
+ invalid_value=int(values[0]),
+ lower_bound=0,
+ upper_bound=resolved_upper_bound,
+ extra=extra,
+ )
+ return
+ for value in values:
+ resolved_value = int(value)
+ if 0 <= resolved_value < resolved_upper_bound:
+ continue
+ _raise_index_validation_error(
+ index_name,
+ context=context,
+ invalid_value=resolved_value,
+ lower_bound=0,
+ upper_bound=resolved_upper_bound,
+ extra=extra,
+ )
+
+
+def _normalize_event_rerank_mode(value: Any) -> str:
+ normalized = normalize_text(value)
+ return "single" if normalized == "single" else "matrix"
+
+
+def _normalize_event_pair_feature_mode(value: Any) -> str:
+ normalized = normalize_text(value)
+ return "zero" if normalized == "zero" else "full"
+
+
+def _apply_event_pair_feature_mode(features: Tensor, mode: str) -> Tensor:
+ resolved_mode = _normalize_event_pair_feature_mode(mode)
+ if resolved_mode == "zero" and isinstance(features, Tensor) and features.numel() > 0:
+ return torch.zeros_like(features)
+ return features
+
+
+def _call_with_supported_kwargs(func: Callable[..., Any], /, *args: Any, **kwargs: Any) -> Any:
+ try:
+ signature = inspect.signature(func)
+ except (TypeError, ValueError):
+ return func(*args, **kwargs)
+ parameters = signature.parameters
+ if any(parameter.kind == inspect.Parameter.VAR_KEYWORD for parameter in parameters.values()):
+ return func(*args, **kwargs)
+ supported_kwargs = {key: value for key, value in kwargs.items() if key in parameters}
+ return func(*args, **supported_kwargs)
+
+
+def _extract_question_speakers(question: str) -> List[str]:
+ clean = clean_text(question)
+ if not clean:
+ return []
+
+ candidates: List[str] = []
+ seen = set()
+
+ def add_candidate(raw_value: str) -> None:
+ candidate = clean_text(raw_value).removesuffix("'s")
+ if not candidate:
+ return
+ parts = [part for part in candidate.split() if clean_text(part)]
+ if not parts or any(part in _NON_SPEAKER_CAPITALIZED_WORDS for part in parts):
+ return
+ key = normalize_text(candidate)
+ if key in seen:
+ return
+ seen.add(key)
+ candidates.append(candidate)
+
+ name_pattern = r"[A-Z][a-z]+(?:\s+[A-Z][a-z]+)?"
+ anchored_patterns = (
+ rf"\b({name_pattern})'s\b",
+ rf"\b(?:{'|'.join(_QUESTION_SPEAKER_AUXILIARIES)})\s+({name_pattern})\b",
+ rf"^\s*(?:What|When|Which|Where|Why|How)\s+({name_pattern})\b",
+ )
+ for pattern in anchored_patterns:
+ for match in re.finditer(pattern, clean):
+ add_candidate(match.group(1))
+
+ for match in re.finditer(rf"\b({name_pattern})\b", clean):
+ prefix = clean[: match.start()].rstrip()
+ previous_word_match = re.search(r"([A-Za-z]+)$", prefix)
+ previous_word = normalize_text(previous_word_match.group(1)) if previous_word_match else ""
+ if previous_word in _QUESTION_SPEAKER_PREPOSITIONS:
+ continue
+ add_candidate(match.group(1))
+ return candidates
+
+
+def _question_has_subject_reference(question: str, speaker_candidates: Sequence[str]) -> bool:
+ if bool(list(speaker_candidates or [])) or bool(_PROFILE_POSSESSIVE_RE.search(question)):
+ return True
+ clean = clean_text(question)
+ if not clean:
+ return False
+ content_text = clean
+ prefix_match = _QUESTION_LEADING_FUNCTION_RE.match(content_text)
+ if prefix_match:
+ content_text = content_text[prefix_match.end() :]
+ content_tokens = _trim_question_anchor_edge_tokens(_normalized_token_list(content_text))
+ if not content_tokens:
+ return False
+ first_token = normalize_text(content_tokens[0])
+ if not first_token or first_token in _TMCRA_TEMPORAL_TOKEN_HINTS:
+ return False
+ return True
+
+
+def _question_requests_profile_detail(question: str, lowered: str, speaker_candidates: Sequence[str]) -> bool:
+ has_subject_reference = _question_has_subject_reference(question, speaker_candidates)
+ if not has_subject_reference:
+ return False
+ if _PROFILE_COPULAR_RE.search(question):
+ return True
+ if _PROFILE_QUERY_HINT_RE.search(lowered):
+ return True
+ return bool(_PROFILE_ATTRIBUTE_RE.search(question) and _PROFILE_QUERY_HINT_RE.search(lowered))
+
+
+def _question_has_temporal_reference(question: str, lowered: str) -> bool:
+ first_token = normalize_text(tokenize_text(question)[0]) if tokenize_text(question) else ""
+ if first_token == "when":
+ return True
+ if _TEMPORAL_REFERENCE_RE.search(lowered):
+ return True
+ if any(marker in lowered for marker in _MONTH_MARKERS):
+ return True
+ if re.search(r"\b(?:mon|tue|wed|thu|fri|sat|sun)(?:day)?\b", lowered):
+ return True
+ if re.search(r"\b\d{4}\b", lowered):
+ return True
+ return False
+
+
+def _question_requests_temporal_answer(question: str, lowered: str) -> bool:
+ clean = clean_text(question)
+ if not clean:
+ return False
+ if _EDUCATION_YEAR_RE.search(clean):
+ return False
+ if _TEMPORAL_ANSWER_PREFIX_RE.search(clean):
+ return True
+ if _TEMPORAL_ANSWER_WH_RE.search(clean):
+ return True
+ if _TEMPORAL_ANSWER_DURATION_RE.search(clean):
+ return True
+ return False
+
+
+def _question_is_temporal(question: str, lowered: str) -> bool:
+ return _question_requests_temporal_answer(question, lowered)
+
+
+def _question_time_granularity(question: str, lowered: str) -> str:
+ if not _question_requests_temporal_answer(question, lowered):
+ return ""
+ if "month" in lowered or any(marker in lowered for marker in _MONTH_MARKERS):
+ return "month"
+ if "year" in lowered or re.search(r"\b\d{4}\b", lowered):
+ return "year"
+ return "day_or_coarse"
+
+
+def _question_target_status(question: str, lowered: str) -> str:
+ if _PLANNED_STATUS_RE.search(question):
+ return "planned"
+ if _CURRENT_STATUS_RE.search(question):
+ return "current"
+ if _PAST_STATUS_RE.search(lowered):
+ return "past"
+ return ""
+
+
+def _question_semantic_slot(question: str, lowered: str, speaker_candidates: Sequence[str]) -> str:
+ if _question_is_temporal(question, lowered):
+ return "event_time"
+ if _question_requests_profile_detail(question, lowered, speaker_candidates):
+ return "profile"
+ return "event"
+
+
+def _positive_event_payloads_have_profile(payloads: Sequence[Mapping[str, Any]]) -> bool:
+ for payload in payloads:
+ metadata = dict(payload.get("metadata", {}) or {})
+ if any(
+ clean_text(
+ payload.get(key, metadata.get(key, ""))
+ )
+ for key in ("profile_type", "profile_value")
+ ):
+ return True
+ return False
+
+
+def stable_hash_int(value: str) -> int:
+ digest = hashlib.blake2b(value.encode("utf-8"), digest_size=8).digest()
+ return int.from_bytes(digest, "little", signed=False)
+
+
+def stable_hash_bucket(value: str, buckets: int) -> int:
+ return stable_hash_int(value or "__empty__") % max(1, int(buckets))
+
+
+def stable_token_identity(value: str) -> int:
+ return stable_hash_int(value or "__empty__") & ((1 << 63) - 1)
+
+
+def _text_hash_indices(value: Any, buckets: int) -> List[int]:
+ tokens = tokenize_text(value) or ["__empty__"]
+ return [stable_hash_bucket(token, buckets) for token in tokens]
+
+
+def _resolved_text_tokens(value: Any) -> List[str]:
+ return tokenize_text(value) or ["__empty__"]
+
+
+def _align_sequence_length(values: Sequence[Any], *, target_length: int, fill_value: Any) -> List[Any]:
+ resolved_target_length = max(0, int(target_length))
+ if resolved_target_length <= 0:
+ return []
+ normalized = list(values[:resolved_target_length])
+ if len(normalized) < resolved_target_length:
+ normalized.extend([fill_value] * (resolved_target_length - len(normalized)))
+ return normalized
+
+
+def _padded_long_batch(
+ value_batches: Sequence[Sequence[int]],
+ *,
+ device: torch.device,
+ fill_value: int = 0,
+ max_tokens: int | None = None,
+) -> tuple[Tensor, Tensor]:
+ if not value_batches:
+ return (
+ torch.full((0, 1), int(fill_value), dtype=torch.long, device=device),
+ torch.zeros((0, 1), dtype=torch.bool, device=device),
+ )
+ max_allowed_tokens = max(1, int(max_tokens or 0)) if max_tokens is not None else None
+ normalized_batches: List[List[int]] = []
+ max_length = 1
+ for values in value_batches:
+ normalized = [int(item) for item in list(values or [])]
+ if max_allowed_tokens is not None:
+ normalized = normalized[:max_allowed_tokens]
+ if not normalized:
+ normalized = [int(fill_value)]
+ normalized_batches.append(normalized)
+ max_length = max(max_length, len(normalized))
+ padded = torch.full((len(normalized_batches), max_length), int(fill_value), dtype=torch.long, device=device)
+ valid_mask = torch.zeros((len(normalized_batches), max_length), dtype=torch.bool, device=device)
+ for batch_index, values in enumerate(normalized_batches):
+ value_count = len(values)
+ padded[batch_index, :value_count] = torch.tensor(values, dtype=torch.long, device=device)
+ valid_mask[batch_index, :value_count] = True
+ return padded, valid_mask
+
+
+def _padded_hashed_index_batch(
+ index_batches: Sequence[Sequence[int]],
+ *,
+ embedding: nn.Embedding,
+ max_tokens: int | None = None,
+) -> tuple[Tensor, Tensor]:
+ device = embedding.weight.device
+ empty_index = stable_hash_bucket("__empty__", int(embedding.num_embeddings))
+ if not index_batches:
+ return (
+ torch.full((0, 1), empty_index, dtype=torch.long, device=device),
+ torch.zeros((0, 1), dtype=torch.bool, device=device),
+ )
+ max_allowed_tokens = max(1, int(max_tokens or 0)) if max_tokens is not None else None
+ normalized_batches: List[List[int]] = []
+ max_length = 1
+ for index_values in index_batches:
+ values = [int(item) for item in list(index_values or [])]
+ if max_allowed_tokens is not None:
+ values = values[:max_allowed_tokens]
+ if not values:
+ values = [empty_index]
+ normalized_batches.append(values)
+ max_length = max(max_length, len(values))
+ batch_size = len(normalized_batches)
+ padded_indices = torch.full((batch_size, max_length), empty_index, dtype=torch.long, device=device)
+ valid_mask = torch.zeros((batch_size, max_length), dtype=torch.bool, device=device)
+ for batch_index, values in enumerate(normalized_batches):
+ value_count = len(values)
+ _validate_index_values(
+ values,
+ upper_bound=int(embedding.num_embeddings),
+ index_name="hashed_index",
+ context="_padded_hashed_index_batch",
+ extra={"batch_index": batch_index, "value_count": value_count},
+ )
+ padded_indices[batch_index, :value_count] = torch.tensor(values, dtype=torch.long, device=device)
+ valid_mask[batch_index, :value_count] = True
+ return padded_indices, valid_mask
+
+
+def _hashed_token_embedding_batch(
+ index_batches: Sequence[Sequence[int]],
+ embedding: nn.Embedding,
+ *,
+ max_tokens: int | None = None,
+) -> tuple[Tensor, Tensor]:
+ padded_indices, valid_mask = _padded_hashed_index_batch(
+ index_batches,
+ embedding=embedding,
+ max_tokens=max_tokens,
+ )
+ return embedding(padded_indices), valid_mask
+
+
+def _hashed_index_embedding_batch(index_batches: Sequence[Sequence[int]], embedding: nn.Embedding) -> Tensor:
+ device = embedding.weight.device
+ if not index_batches:
+ return torch.zeros((0, embedding.embedding_dim), dtype=embedding.weight.dtype, device=device)
+ embedded, valid_mask = _hashed_token_embedding_batch(index_batches, embedding)
+ valid_mask_float = valid_mask.unsqueeze(-1).to(dtype=embedded.dtype)
+ pooled = (embedded * valid_mask_float).sum(dim=1)
+ denom = valid_mask_float.sum(dim=1).clamp_min(1.0)
+ return pooled / denom
+
+
+def _question_token_roles(question: str, question_features: Mapping[str, Any]) -> tuple[List[str], List[int]]:
+ prepared = _prepare_question_scoring_features(question, question_features)
+ tokens = _resolved_text_tokens(question)
+ anchor_tokens = set(prepared.get("question_anchor_tokens", []) or [])
+ speaker_tokens = set(prepared.get("speaker_candidates", set()) or set())
+ semantic_target = clean_text(prepared.get("semantic_target", ""))
+ has_status_target = bool(clean_text(prepared.get("target_status_target", "")))
+ temporal_mode = bool(prepared.get("question_is_temporal", False)) or bool(clean_text(prepared.get("time_target", "")))
+ profile_mode = semantic_target in {"identity", "research_topic", "education", "occupation", "profile"}
+ roles: List[int] = []
+ for token in tokens:
+ if token == "__empty__":
+ roles.append(TMCRA_TOKEN_ROLE_EMPTY)
+ elif token in speaker_tokens:
+ roles.append(TMCRA_TOKEN_ROLE_SPEAKER)
+ elif temporal_mode and (token in anchor_tokens or token in _TMCRA_TEMPORAL_TOKEN_HINTS):
+ roles.append(TMCRA_TOKEN_ROLE_TEMPORAL)
+ elif profile_mode and token in anchor_tokens:
+ roles.append(TMCRA_TOKEN_ROLE_PROFILE)
+ elif has_status_target and token in anchor_tokens:
+ roles.append(TMCRA_TOKEN_ROLE_STATUS)
+ elif token in anchor_tokens:
+ roles.append(TMCRA_TOKEN_ROLE_ANCHOR)
+ else:
+ roles.append(TMCRA_TOKEN_ROLE_CONTENT)
+ return tokens, roles
+
+
+def _node_type_token_role(node_type: str) -> int:
+ normalized_type = clean_text(node_type)
+ if normalized_type == "speaker":
+ return TMCRA_TOKEN_ROLE_SPEAKER
+ if normalized_type == "time":
+ return TMCRA_TOKEN_ROLE_TEMPORAL
+ if normalized_type == "profile":
+ return TMCRA_TOKEN_ROLE_PROFILE
+ if normalized_type == "status":
+ return TMCRA_TOKEN_ROLE_STATUS
+ if normalized_type == "source_turn":
+ return TMCRA_TOKEN_ROLE_SOURCE
+ return TMCRA_TOKEN_ROLE_CONTENT
+
+
+def _node_token_roles(text: str, node_type_id: int) -> tuple[List[str], List[int]]:
+ tokens = _resolved_text_tokens(text)
+ node_type = NODE_TYPES[int(node_type_id)] if 0 <= int(node_type_id) < len(NODE_TYPES) else ""
+ default_role = _node_type_token_role(node_type)
+ roles: List[int] = []
+ for token in tokens:
+ if token == "__empty__":
+ roles.append(TMCRA_TOKEN_ROLE_EMPTY)
+ elif token in {"past", "current", "planned"}:
+ roles.append(TMCRA_TOKEN_ROLE_STATUS)
+ elif token in _TMCRA_TEMPORAL_TOKEN_HINTS:
+ roles.append(TMCRA_TOKEN_ROLE_TEMPORAL)
+ else:
+ roles.append(default_role)
+ return tokens, roles
+
+
+def _tmcra_text_relation_features(
+ token_id_batches: Sequence[Sequence[int]],
+ token_role_batches: Sequence[Sequence[int]],
+ *,
+ max_tokens: int,
+ device: torch.device,
+ dtype: torch.dtype,
+) -> tuple[Tensor, Tensor]:
+ role_ids, token_mask = _padded_long_batch(
+ token_role_batches,
+ device=device,
+ fill_value=TMCRA_TOKEN_ROLE_EMPTY,
+ max_tokens=max_tokens,
+ )
+ token_ids, _ = _padded_long_batch(
+ token_id_batches,
+ device=device,
+ fill_value=0,
+ max_tokens=max_tokens,
+ )
+ features = _tmcra_text_relation_features_from_tensors(
+ token_ids,
+ role_ids,
+ token_mask=token_mask,
+ dtype=dtype,
+ )
+ return features, token_mask
+
+
+def _tmcra_text_relation_features_from_tensors(
+ token_ids: Tensor,
+ role_ids: Tensor,
+ *,
+ token_mask: Tensor,
+ dtype: torch.dtype,
+) -> Tensor:
+ batch_size, token_count = role_ids.shape
+ device = token_ids.device
+ valid_pair = token_mask.unsqueeze(1) & token_mask.unsqueeze(2)
+ left_roles = role_ids.unsqueeze(2)
+ right_roles = role_ids.unsqueeze(1)
+ non_content_left = (left_roles != TMCRA_TOKEN_ROLE_CONTENT) & (left_roles != TMCRA_TOKEN_ROLE_EMPTY)
+ non_content_right = (right_roles != TMCRA_TOKEN_ROLE_CONTENT) & (right_roles != TMCRA_TOKEN_ROLE_EMPTY)
+ position_ids = torch.arange(token_count, dtype=torch.long, device=device)
+ position_distance = (position_ids.view(1, token_count, 1) - position_ids.view(1, 1, token_count)).abs()
+ features = torch.stack(
+ [
+ (token_ids.unsqueeze(2) == token_ids.unsqueeze(1)) & valid_pair,
+ (left_roles == right_roles) & non_content_left & non_content_right & valid_pair,
+ (left_roles == TMCRA_TOKEN_ROLE_ANCHOR) & (right_roles == TMCRA_TOKEN_ROLE_ANCHOR) & valid_pair,
+ (left_roles == TMCRA_TOKEN_ROLE_SPEAKER) & (right_roles == TMCRA_TOKEN_ROLE_SPEAKER) & valid_pair,
+ (left_roles == TMCRA_TOKEN_ROLE_TEMPORAL) & (right_roles == TMCRA_TOKEN_ROLE_TEMPORAL) & valid_pair,
+ (left_roles == TMCRA_TOKEN_ROLE_PROFILE) & (right_roles == TMCRA_TOKEN_ROLE_PROFILE) & valid_pair,
+ (left_roles == TMCRA_TOKEN_ROLE_STATUS) & (right_roles == TMCRA_TOKEN_ROLE_STATUS) & valid_pair,
+ (left_roles == TMCRA_TOKEN_ROLE_SOURCE) & (right_roles == TMCRA_TOKEN_ROLE_SOURCE) & valid_pair,
+ (position_distance <= 1) & valid_pair,
+ ],
+ dim=-1,
+ ).to(dtype=dtype)
+ return features
+
+
+def _move_structure_to_device(value: Any, device: torch.device) -> Any:
+ if isinstance(value, Tensor):
+ return value.to(device=device)
+ if isinstance(value, dict):
+ return {key: _move_structure_to_device(item, device) for key, item in value.items()}
+ if isinstance(value, list):
+ return [_move_structure_to_device(item, device) for item in value]
+ if isinstance(value, tuple):
+ return tuple(_move_structure_to_device(item, device) for item in value)
+ return value
+
+
+def _tensor_to_device(value: Tensor, device: torch.device, *, non_blocking: bool = False) -> Tensor:
+ if value.device == device:
+ return value
+ return value.to(device=device, non_blocking=bool(non_blocking))
+
+
+def _pin_tensor_memory(value: Tensor) -> Tensor:
+ if value.device.type != "cpu" or not torch.cuda.is_available():
+ return value
+ try:
+ if value.is_pinned():
+ return value
+ except RuntimeError:
+ return value
+ try:
+ return value.pin_memory()
+ except RuntimeError:
+ return value
+
+
+def _graph_tensor_mapping_to_device(
+ graph_tensors: Mapping[str, Any],
+ device: torch.device,
+ *,
+ non_blocking: bool = False,
+) -> Dict[str, Any]:
+ moved = dict(graph_tensors)
+ for key in GRAPH_TENSOR_VALUE_KEYS:
+ value = moved.get(key)
+ if isinstance(value, Tensor):
+ moved[key] = _tensor_to_device(value, device, non_blocking=non_blocking)
+ return moved
+
+
+def _pin_graph_tensor_mapping(graph_tensors: Mapping[str, Any]) -> Dict[str, Any]:
+ pinned = dict(graph_tensors)
+ for key in GRAPH_TENSOR_VALUE_KEYS:
+ value = pinned.get(key)
+ if isinstance(value, Tensor):
+ pinned[key] = _pin_tensor_memory(value)
+ return pinned
+
+
+def _graph_source_signature(path: Path) -> Dict[str, int]:
+ stats = path.stat()
+ return {
+ "size": int(stats.st_size),
+ "mtime_ns": int(stats.st_mtime_ns),
+ "cache_version": int(GRAPH_TENSOR_CACHE_VERSION),
+ }
+
+
+def _graph_tensor_cache_path(cache_dir: Path, graph_path: Path) -> Path:
+ return cache_dir / f"{graph_path.stem}.pt"
+
+
+def _load_graph_tensor_cache(
+ cache_path: Path,
+ *,
+ source_signature: Mapping[str, Any],
+) -> Dict[str, Any] | None:
+ def _invalidate_cache_file() -> None:
+ try:
+ cache_path.unlink(missing_ok=True)
+ except OSError:
+ pass
+
+ if not cache_path.exists():
+ return None
+ try:
+ payload = torch.load(cache_path, map_location="cpu", weights_only=True)
+ except Exception:
+ _invalidate_cache_file()
+ return None
+ if not isinstance(payload, Mapping):
+ _invalidate_cache_file()
+ return None
+ cached_signature = dict(payload.get("source_signature", {}) or {})
+ if cached_signature != dict(source_signature):
+ _invalidate_cache_file()
+ return None
+ graph = payload.get("graph")
+ tensors = payload.get("tensors")
+ if not isinstance(graph, Mapping) or not isinstance(tensors, Mapping):
+ _invalidate_cache_file()
+ return None
+ return {"graph": dict(graph), "tensors": dict(tensors)}
+
+
+def _write_graph_tensor_cache(
+ cache_path: Path,
+ *,
+ source_signature: Mapping[str, Any],
+ graph: Mapping[str, Any],
+ tensors: Mapping[str, Any],
+) -> bool:
+ def _report_cache_write_failure(error: Exception) -> None:
+ error_no = getattr(error, "errno", None)
+ warning_key = (str(cache_path.parent), type(error).__name__, error_no if isinstance(error_no, int) else None)
+ with _graph_tensor_cache_write_warning_lock:
+ if warning_key in _graph_tensor_cache_write_warning_keys:
+ return
+ _graph_tensor_cache_write_warning_keys.add(warning_key)
+ details = f" errno={error_no}" if isinstance(error_no, int) else ""
+ print(
+ f"[node_memory] graph tensor cache write skipped for {cache_path}{details}: {error}",
+ flush=True,
+ )
+
+ cache_path.parent.mkdir(parents=True, exist_ok=True)
+ temp_path: Path | None = None
+ try:
+ with tempfile.NamedTemporaryFile(
+ mode="wb",
+ delete=False,
+ dir=str(cache_path.parent),
+ prefix=f"{cache_path.stem}.",
+ suffix=".tmp",
+ ) as handle:
+ temp_path = Path(handle.name)
+ torch.save(
+ {
+ "source_signature": dict(source_signature),
+ "graph": dict(graph),
+ "tensors": dict(tensors),
+ },
+ handle,
+ )
+ handle.flush()
+ os.fsync(handle.fileno())
+ try:
+ os.replace(temp_path, cache_path)
+ temp_path = None
+ return True
+ except OSError as error:
+ error_no = getattr(error, "errno", None)
+ if isinstance(error_no, int) and error_no in _GRAPH_TENSOR_CACHE_RECOVERABLE_ERRNOS:
+ try:
+ cache_path.unlink(missing_ok=True)
+ except OSError:
+ pass
+ try:
+ os.replace(temp_path, cache_path)
+ temp_path = None
+ return True
+ except Exception as retry_error:
+ _report_cache_write_failure(retry_error)
+ return False
+ _report_cache_write_failure(error)
+ return False
+ except Exception as error:
+ _report_cache_write_failure(error)
+ return False
+ finally:
+ if temp_path is not None and temp_path.exists():
+ temp_path.unlink(missing_ok=True)
+
+
+class _PurePythonJSONEncoder(json.JSONEncoder):
+ def encode(self, o: Any) -> str:
+ chunks = self.iterencode(o, _one_shot=False)
+ if not isinstance(chunks, (list, tuple)):
+ chunks = list(chunks)
+ return "".join(chunks)
+
+ def iterencode(self, o: Any, _one_shot: bool = False): # type: ignore[override]
+ if self.check_circular:
+ markers: dict[int, Any] | None = {}
+ else:
+ markers = None
+ if self.ensure_ascii:
+ string_encoder = json_encoder.encode_basestring_ascii
+ else:
+ string_encoder = json_encoder.encode_basestring
+
+ def floatstr(
+ value: float,
+ allow_nan: bool = self.allow_nan,
+ _repr: Callable[[float], str] = float.__repr__,
+ _inf: float = float("inf"),
+ _neginf: float = -float("inf"),
+ ) -> str:
+ if value != value:
+ text = "NaN"
+ elif value == _inf:
+ text = "Infinity"
+ elif value == _neginf:
+ text = "-Infinity"
+ else:
+ return _repr(value)
+ if not allow_nan:
+ raise ValueError(
+ "Out of range float values are not JSON compliant: " + repr(value)
+ )
+ return text
+
+ _iterencode = json_encoder._make_iterencode(
+ markers,
+ self.default,
+ string_encoder,
+ self.indent,
+ floatstr,
+ self.key_separator,
+ self.item_separator,
+ self.sort_keys,
+ self.skipkeys,
+ _one_shot=False,
+ )
+ return _iterencode(o, 0)
+
+def json_dumps(payload: Any) -> str:
+ # Reusing a single pure-Python encoder instance has shown state corruption on
+ # Python 3.12 during repeated large graph exports. Build a fresh encoder per
+ # call so markers and iterencode internals never leak across payloads.
+ return _PurePythonJSONEncoder(ensure_ascii=False, sort_keys=True).encode(payload)
+
+
+def read_json(path: Path) -> Any:
+ try:
+ text = path.read_text(encoding="utf-8")
+ except UnicodeDecodeError as exc:
+ raise RuntimeError(f"Failed to decode JSON file {path}: {exc}") from exc
+ try:
+ return json.loads(text)
+ except json.JSONDecodeError as exc:
+ raise RuntimeError(
+ f"Failed to parse JSON file {path}: line {exc.lineno} column {exc.colno} char {exc.pos}: {exc.msg}"
+ ) from exc
+
+
+def iter_jsonl_with_offsets(path: Path) -> Iterator[tuple[int, Dict[str, Any]]]:
+ if not path.exists():
+ return
+ try:
+ with path.open("rb") as handle:
+ for file_line_number in itertools.count(start=1):
+ offset = handle.tell()
+ raw_line = handle.readline()
+ if not raw_line:
+ break
+ line = raw_line.strip()
+ if not line:
+ continue
+ try:
+ decoded_line = line.decode("utf-8")
+ except UnicodeDecodeError as exc:
+ raise RuntimeError(
+ f"Failed to decode JSONL file {path} at source line {file_line_number}: {exc}"
+ ) from exc
+ try:
+ yield offset, dict(json.loads(decoded_line))
+ except json.JSONDecodeError as exc:
+ raise RuntimeError(
+ f"Failed to parse JSONL file {path} at source line {file_line_number}: "
+ f"line {exc.lineno} column {exc.colno} char {exc.pos}: {exc.msg}"
+ ) from exc
+ except UnicodeDecodeError as exc:
+ raise RuntimeError(f"Failed to decode JSONL file {path}: {exc}") from exc
+
+
+def iter_jsonl(path: Path) -> Iterator[Dict[str, Any]]:
+ for _, row in iter_jsonl_with_offsets(path):
+ yield row
+
+
+def read_jsonl(path: Path) -> List[Dict[str, Any]]:
+ return [row for row in iter_jsonl(path)]
+
+
+def write_json(path: Path, payload: Any) -> None:
+ _atomic_write_text(path, json_dumps(payload))
+
+
+def write_jsonl(path: Path, rows: Sequence[Mapping[str, Any]]) -> None:
+ text = "\n".join(json_dumps(dict(row)) for row in rows)
+ _atomic_write_text(path, text + ("\n" if text else ""))
+
+
+def _atomic_write_text(path: Path, text: str, *, encoding: str = "utf-8") -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ fd, temp_path = tempfile.mkstemp(
+ prefix=f".{path.name}.",
+ suffix=".tmp",
+ dir=str(path.parent),
+ text=True,
+ )
+ try:
+ with os.fdopen(fd, "w", encoding=encoding, newline="") as handle:
+ handle.write(text)
+ handle.flush()
+ os.fsync(handle.fileno())
+ os.replace(temp_path, path)
+ except Exception:
+ try:
+ os.unlink(temp_path)
+ except FileNotFoundError:
+ pass
+ raise
+
+
+def answer_type_from_query(
+ question: str,
+ *,
+ category: Any = "",
+ positive_event_ids: Sequence[str] | None = None,
+ question_features: Mapping[str, Any] | None = None,
+ positive_event_payloads: Sequence[Mapping[str, Any]] | None = None,
+) -> str:
+ positives = [clean_text(item) for item in list(positive_event_ids or []) if clean_text(item)]
+ if str(category or "").strip() == "5" or not positives:
+ return "abstain"
+ resolved_question_features = dict(question_features or extract_question_features(question))
+ if len(positives) > 1:
+ return "multi_evidence"
+ if bool(resolved_question_features.get("is_temporal", False)):
+ return "time"
+ if _positive_event_payloads_have_profile(list(positive_event_payloads or [])):
+ return "profile"
+ if clean_text(resolved_question_features.get("semantic_slot_target", "")) == "profile":
+ return "profile"
+ return "event_text"
+
+
+def extract_question_features(question: str) -> Dict[str, Any]:
+ clean = clean_text(question)
+ lowered = normalize_text(clean)
+ speaker_candidates = _extract_question_speakers(clean)
+ semantic_slot = _question_semantic_slot(clean, lowered, speaker_candidates)
+ target_status = _question_target_status(clean, lowered)
+ time_granularity = _question_time_granularity(clean, lowered)
+ is_temporal = _question_is_temporal(clean, lowered)
+ has_temporal_reference = _question_has_temporal_reference(clean, lowered)
+ question_anchor_tokens = _question_anchor_tokens(
+ clean,
+ {"speaker_candidates": list(speaker_candidates)},
+ )
+ return {
+ "speaker_candidates": list(speaker_candidates),
+ "semantic_slot_target": semantic_slot,
+ "target_status_target": target_status,
+ "time_granularity_target": time_granularity,
+ "is_temporal": is_temporal,
+ "has_temporal_reference": has_temporal_reference,
+ "question_anchor_tokens": list(question_anchor_tokens),
+ }
+
+
+def _token_overlap_ratio(left: Sequence[str] | str, right: Sequence[str] | str) -> float:
+ left_tokens = (
+ _normalized_token_set_cached(normalize_text(left))
+ if isinstance(left, str)
+ else {clean_text(item) for item in left if clean_text(item)}
+ )
+ right_tokens = (
+ _normalized_token_set_cached(normalize_text(right))
+ if isinstance(right, str)
+ else {clean_text(item) for item in right if clean_text(item)}
+ )
+ if not left_tokens or not right_tokens:
+ return 0.0
+ return float(len(left_tokens & right_tokens)) / float(max(1, len(left_tokens | right_tokens)))
+
+
+def _max_token_overlap_ratio(left: Sequence[str] | str, values: Sequence[Any]) -> float:
+ best = 0.0
+ for value in values:
+ best = max(best, _token_overlap_ratio(left, clean_text(value)))
+ return best
+
+
+def _normalized_speaker_candidates(features: Mapping[str, Any]) -> set[str]:
+ return {
+ normalize_text(clean_text(item).removesuffix("'s"))
+ for item in list(features.get("speaker_candidates", []) or [])
+ if clean_text(item)
+ }
+
+
+def _normalized_required_tokens(required_tokens: Sequence[str]) -> List[str]:
+ normalized_required: List[str] = []
+ for token in required_tokens:
+ normalized = normalize_text(token)
+ if not normalized:
+ normalized = normalize_text(clean_text(token))
+ if normalized:
+ normalized_required.append(normalized)
+ return normalized_required
+
+
+def _normalized_token_list(value: Sequence[str] | str) -> List[str]:
+ if isinstance(value, str):
+ return list(_tokenize_normalized_text_cached(normalize_text(value)))
+ tokens: List[str] = []
+ for item in value:
+ for token in tokenize_text(item):
+ normalized = normalize_text(token)
+ if normalized:
+ tokens.append(normalized)
+ return tokens
+
+
+def _normalized_token_set(value: Sequence[str] | str) -> set[str]:
+ if isinstance(value, str):
+ return set(_normalized_token_set_cached(normalize_text(value)))
+ return set(_normalized_token_list(value))
+
+
+def _strip_question_speaker_mentions(text: str, speaker_candidates: Sequence[str]) -> str:
+ cleaned = clean_text(text)
+ if not cleaned:
+ return ""
+ for candidate in sorted(
+ (clean_text(item) for item in list(speaker_candidates or []) if clean_text(item)),
+ key=len,
+ reverse=True,
+ ):
+ cleaned = re.sub(rf"\b{re.escape(candidate)}\b", " ", cleaned, flags=re.IGNORECASE)
+ return clean_text(cleaned)
+
+
+def _trim_question_anchor_edge_tokens(tokens: Sequence[str]) -> List[str]:
+ trimmed = [normalize_text(clean_text(token)) for token in list(tokens or []) if clean_text(token)]
+ while trimmed and trimmed[0] in _QUESTION_EDGE_GLUE_TOKENS:
+ trimmed.pop(0)
+ while trimmed and trimmed[-1] in _QUESTION_EDGE_GLUE_TOKENS:
+ trimmed.pop()
+ return trimmed
+
+
+def _question_anchor_tokens(question: str, question_features: Mapping[str, Any]) -> List[str]:
+ existing = [
+ normalize_text(clean_text(token))
+ for token in list(question_features.get("question_anchor_tokens", []) or [])
+ if clean_text(token)
+ ]
+ if existing:
+ return dedupe_texts(existing)
+ clean = clean_text(question)
+ if not clean:
+ return []
+ speaker_tokens = _normalized_token_set(list(question_features.get("speaker_candidates", []) or []))
+ content_text = clean
+ prefix_match = _QUESTION_LEADING_FUNCTION_RE.match(content_text)
+ if prefix_match:
+ content_text = content_text[prefix_match.end() :]
+ content_text = _strip_question_speaker_mentions(
+ content_text,
+ list(question_features.get("speaker_candidates", []) or []),
+ )
+ if bool(question_features.get("is_temporal", False)):
+ content_text = _TEMPORAL_REFERENCE_RE.sub(" ", content_text)
+ content_text = re.sub(r"\b\d{1,4}\b", " ", content_text)
+ target_status = clean_text(question_features.get("target_status_target", ""))
+ if target_status == "planned":
+ content_text = _PLANNED_STATUS_RE.sub(" ", content_text)
+ elif target_status == "current":
+ content_text = _CURRENT_STATUS_RE.sub(" ", content_text)
+ elif target_status == "past":
+ content_text = _PAST_STATUS_RE.sub(" ", content_text)
+ anchors = _trim_question_anchor_edge_tokens(
+ [
+ token
+ for token in _normalized_token_list(content_text)
+ if token and token not in speaker_tokens and not re.fullmatch(r"\d+", token)
+ ]
+ )
+ if anchors:
+ return dedupe_texts(anchors)
+ fallback_tokens = _trim_question_anchor_edge_tokens(
+ [
+ token
+ for token in _normalized_token_list(clean)
+ if token and token not in speaker_tokens and not re.fullmatch(r"\d+", token)
+ ]
+ )
+ return dedupe_texts(fallback_tokens)
+
+
+def _speaker_candidates_from_predicted_roles(tokens: Sequence[str], role_ids: Sequence[int], token_mask: Sequence[bool]) -> List[str]:
+ candidates: List[str] = []
+ current_tokens: List[str] = []
+ for token, role_id, is_valid in zip(tokens, role_ids, token_mask):
+ normalized_token = normalize_text(token)
+ if not is_valid or not normalized_token or normalized_token == "__empty__":
+ break
+ if int(role_id) == TMCRA_TOKEN_ROLE_SPEAKER:
+ current_tokens.append(normalized_token)
+ continue
+ if current_tokens:
+ candidates.append(" ".join(current_tokens))
+ current_tokens = []
+ if current_tokens:
+ candidates.append(" ".join(current_tokens))
+ return dedupe_texts(candidates)
+
+
+def _anchor_tokens_from_predicted_roles(tokens: Sequence[str], role_ids: Sequence[int], token_mask: Sequence[bool]) -> List[str]:
+ anchor_like_roles = {
+ TMCRA_TOKEN_ROLE_ANCHOR,
+ TMCRA_TOKEN_ROLE_PROFILE,
+ TMCRA_TOKEN_ROLE_STATUS,
+ }
+ anchors = [
+ normalize_text(token)
+ for token, role_id, is_valid in zip(tokens, role_ids, token_mask)
+ if is_valid
+ and int(role_id) in anchor_like_roles
+ and normalize_text(token)
+ and normalize_text(token) != "__empty__"
+ and not re.fullmatch(r"\d+", normalize_text(token))
+ ]
+ return dedupe_texts(_trim_question_anchor_edge_tokens(anchors))
+
+
+def _merge_learned_question_features_from_token_roles(
+ question: str,
+ base_features: Mapping[str, Any],
+ *,
+ tokens: Sequence[str],
+ predicted_role_ids: Sequence[int],
+ token_mask: Sequence[bool],
+) -> Dict[str, Any]:
+ merged = dict(base_features or {})
+ learned_speakers = _speaker_candidates_from_predicted_roles(tokens, predicted_role_ids, token_mask)
+ learned_anchors = _anchor_tokens_from_predicted_roles(tokens, predicted_role_ids, token_mask)
+ if learned_speakers:
+ merged["speaker_candidates"] = learned_speakers
+ if learned_anchors:
+ merged["question_anchor_tokens"] = learned_anchors
+ return merged
+
+
+def _required_token_coverage(required_tokens: Sequence[str], candidate: Sequence[str] | str) -> float:
+ normalized_required = _normalized_required_tokens(required_tokens)
+ if not normalized_required:
+ return 0.0
+ candidate_tokens = (
+ _normalized_token_set_cached(normalize_text(candidate))
+ if isinstance(candidate, str)
+ else _normalized_token_set(candidate)
+ )
+ if not candidate_tokens:
+ return 0.0
+ covered = 0
+ for token in normalized_required:
+ if token in candidate_tokens:
+ covered += 1
+ return float(covered) / float(len(normalized_required))
+
+
+def _max_required_token_coverage(required_tokens: Sequence[str], values: Sequence[Any]) -> float:
+ best = 0.0
+ for value in values:
+ coverage = _required_token_coverage(required_tokens, value)
+ if coverage > best:
+ best = coverage
+ return best
+
+
+def _required_bigram_coverage(required_tokens: Sequence[str], candidate: Sequence[str] | str) -> float:
+ normalized_required = _normalized_required_tokens(required_tokens)
+ if len(normalized_required) < 2:
+ return 0.0
+ required_bigrams = set()
+ for index in range(len(normalized_required) - 1):
+ required_bigrams.add((normalized_required[index], normalized_required[index + 1]))
+ if isinstance(candidate, str):
+ candidate_bigrams = _normalized_token_bigram_set_cached(normalize_text(candidate))
+ else:
+ candidate_tokens = _normalized_token_list(candidate)
+ if len(candidate_tokens) < 2:
+ return 0.0
+ candidate_bigrams = set()
+ for index in range(len(candidate_tokens) - 1):
+ candidate_bigrams.add((candidate_tokens[index], candidate_tokens[index + 1]))
+ if not required_bigrams:
+ return 0.0
+ return float(len(required_bigrams & candidate_bigrams)) / float(len(required_bigrams))
+
+
+def _text_feature_payload(value: Any) -> Dict[str, Any]:
+ text = clean_text(value)
+ normalized = normalize_text(text)
+ token_set = set(_normalized_token_set_cached(normalized)) if normalized else set()
+ bigram_set = set(_normalized_token_bigram_set_cached(normalized)) if normalized else set()
+ return {
+ "text": text,
+ "normalized": normalized,
+ "token_set": token_set,
+ "bigram_set": bigram_set,
+ }
+
+
+def _text_feature_payloads(values: Sequence[Any]) -> List[Dict[str, Any]]:
+ payloads: List[Dict[str, Any]] = []
+ for value in values:
+ payload = _text_feature_payload(value)
+ if payload["text"]:
+ payloads.append(payload)
+ return payloads
+
+
+def _token_overlap_ratio_from_token_sets(left_tokens: set[str], right_tokens: set[str]) -> float:
+ if not left_tokens or not right_tokens:
+ return 0.0
+ return float(len(left_tokens & right_tokens)) / float(max(1, len(left_tokens | right_tokens)))
+
+
+def _max_token_overlap_ratio_from_payloads(left_tokens: set[str], payloads: Sequence[Mapping[str, Any]]) -> float:
+ best = 0.0
+ for payload in payloads:
+ best = max(best, _token_overlap_ratio_from_token_sets(left_tokens, set(payload.get("token_set", set()))))
+ return best
+
+
+def _required_bigram_set(required_tokens: Sequence[str]) -> set[tuple[str, str]]:
+ normalized_required = _normalized_required_tokens(required_tokens)
+ return _required_bigram_set_from_normalized_tokens(normalized_required)
+
+
+def _required_bigram_set_from_normalized_tokens(normalized_required: Sequence[str]) -> set[tuple[str, str]]:
+ if len(normalized_required) < 2:
+ return set()
+ return {
+ (normalized_required[index], normalized_required[index + 1])
+ for index in range(len(normalized_required) - 1)
+ }
+
+
+def _required_token_coverage_from_tokens(required_tokens: Sequence[str], candidate_tokens: set[str]) -> float:
+ normalized_required = _normalized_required_tokens(required_tokens)
+ return _required_token_coverage_from_normalized_tokens(normalized_required, candidate_tokens)
+
+
+def _required_token_coverage_from_normalized_tokens(
+ normalized_required: Sequence[str],
+ candidate_tokens: set[str],
+) -> float:
+ if not normalized_required or not candidate_tokens:
+ return 0.0
+ covered = 0
+ for token in normalized_required:
+ if token in candidate_tokens:
+ covered += 1
+ return float(covered) / float(len(normalized_required))
+
+
+def _max_required_token_coverage_from_payloads(required_tokens: Sequence[str], payloads: Sequence[Mapping[str, Any]]) -> float:
+ normalized_required = _normalized_required_tokens(required_tokens)
+ return _max_required_token_coverage_from_normalized_tokens(normalized_required, payloads)
+
+
+def _max_required_token_coverage_from_normalized_tokens(
+ normalized_required: Sequence[str],
+ payloads: Sequence[Mapping[str, Any]],
+) -> float:
+ best = 0.0
+ for payload in payloads:
+ coverage = _required_token_coverage_from_normalized_tokens(
+ normalized_required,
+ set(payload.get("token_set", set())),
+ )
+ if coverage > best:
+ best = coverage
+ return best
+
+
+def _required_bigram_coverage_from_bigrams(
+ required_bigrams: set[tuple[str, str]],
+ candidate_bigrams: set[tuple[str, str]],
+) -> float:
+ if not required_bigrams or not candidate_bigrams:
+ return 0.0
+ return float(len(required_bigrams & candidate_bigrams)) / float(len(required_bigrams))
+
+
+def _max_required_bigram_coverage_from_payloads(
+ required_bigrams: set[tuple[str, str]],
+ payloads: Sequence[Mapping[str, Any]],
+) -> float:
+ best = 0.0
+ for payload in payloads:
+ coverage = _required_bigram_coverage_from_bigrams(
+ required_bigrams,
+ set(payload.get("bigram_set", set())),
+ )
+ if coverage > best:
+ best = coverage
+ return best
+
+
+def _prepare_question_scoring_features(question: str, question_features: Mapping[str, Any]) -> Dict[str, Any]:
+ question_text = clean_text(question)
+ question_payload = _text_feature_payload(question_text)
+ question_anchor_tokens = _normalized_required_tokens(_question_anchor_tokens(question_text, question_features))
+ return {
+ "question_text": question_text,
+ "question_token_set": set(question_payload.get("token_set", set())),
+ "speaker_candidates": _normalized_speaker_candidates(question_features),
+ "semantic_target": clean_text(question_features.get("semantic_slot_target", "")),
+ "target_status_target": clean_text(question_features.get("target_status_target", "")),
+ "time_target": clean_text(question_features.get("time_granularity_target", "")),
+ "question_is_temporal": bool(question_features.get("is_temporal", False)),
+ "question_anchor_tokens": question_anchor_tokens,
+ "question_anchor_bigrams": _required_bigram_set_from_normalized_tokens(question_anchor_tokens),
+ }
+
+
+def _prepare_event_scoring_features(
+ event_node: Mapping[str, Any],
+ *,
+ support_payload: Mapping[str, Any] | None = None,
+) -> Dict[str, Any]:
+ metadata = dict(event_node.get("metadata", {}) or {})
+ support = dict(support_payload or {})
+ depth_layer = clean_text(
+ event_node.get(
+ "depth_layer",
+ metadata.get(
+ "depth_layer",
+ metadata.get("memory_chain_depth_layer", metadata.get("chain_depth_layer", "")),
+ ),
+ )
+ )
+ time_display_value = clean_text(event_node.get("time_display_value", metadata.get("time_display_value", "")))
+ time_value = clean_text(event_node.get("time_value", metadata.get("time_value", "")))
+ event_text_payload = _text_feature_payload(event_node.get("text", ""))
+ typed_signature_text = clean_text(metadata.get("tmcra_typed_event_signature", ""))
+ typed_signature_text = " ".join(
+ dedupe_texts(
+ [
+ typed_signature_text,
+ " ".join(str(item) for item in list(metadata.get("tmcra_node_tags", []) or [])),
+ " ".join(str(item) for item in list(metadata.get("tmcra_path_tags", []) or [])),
+ clean_text(metadata.get("tmcra_tunnel_group_key", "")),
+ ],
+ max_items=8,
+ )
+ )
+ event_signature_payload = _text_feature_payload(
+ " ".join(
+ item
+ for item in (
+ clean_text(event_node.get("event_signature", metadata.get("event_signature", ""))),
+ typed_signature_text,
+ )
+ if item
+ )
+ )
+ profile_value_payload = _text_feature_payload(event_node.get("profile_value", metadata.get("profile_value", "")))
+ time_payload = _text_feature_payload(time_display_value or time_value)
+ source_turn_payloads = _text_feature_payloads(list(support.get("source_turn_texts", []) or []))
+ profile_text_payloads = _text_feature_payloads(list(support.get("profile_texts", []) or []))
+ time_text_payloads = _text_feature_payloads(list(support.get("time_texts", []) or []))
+ return {
+ "speaker": normalize_text(clean_text(event_node.get("speaker", metadata.get("speaker", ""))).removesuffix("'s")),
+ "target_status": clean_text(event_node.get("target_status", metadata.get("target_status", ""))),
+ "time_granularity": clean_text(event_node.get("time_granularity", metadata.get("time_granularity", ""))),
+ "profile_type": clean_text(event_node.get("profile_type", metadata.get("profile_type", ""))),
+ "profile_value": profile_value_payload,
+ "event_signature": event_signature_payload,
+ "event_text": event_text_payload,
+ "time_text": time_payload,
+ "time_compare_value": normalize_text(time_value or time_display_value),
+ "session_name": normalize_text(clean_text(event_node.get("session_name", metadata.get("session_name", "")))),
+ "turn_index": int(event_node.get("turn_index", metadata.get("turn_index", 0)) or 0),
+ "depth_layer": depth_layer,
+ "memory_chain_role": clean_text(
+ event_node.get(
+ "memory_chain_role",
+ metadata.get("memory_chain_role", metadata.get("chain_role", metadata.get("support_role", ""))),
+ )
+ ),
+ "source_turn_texts": source_turn_payloads,
+ "profile_texts": profile_text_payloads,
+ "time_texts": time_text_payloads,
+ "has_time": 1.0 if time_display_value or time_value else 0.0,
+ "has_profile": 1.0 if clean_text(event_node.get("profile_type", metadata.get("profile_type", ""))) or profile_value_payload["text"] else 0.0,
+ "has_status": 1.0 if clean_text(event_node.get("target_status", metadata.get("target_status", ""))) else 0.0,
+ "relation_signature_tokens": (
+ set(event_signature_payload.get("token_set", set()))
+ or set(event_text_payload.get("token_set", set()))
+ ),
+ }
+
+
+def _prepare_node_scoring_features(node: Mapping[str, Any]) -> Dict[str, Any]:
+ metadata = dict(node.get("metadata", {}) or {})
+ time_display_value = clean_text(node.get("time_display_value", metadata.get("time_display_value", "")))
+ time_value = clean_text(node.get("time_value", metadata.get("time_value", "")))
+ profile_value = clean_text(node.get("profile_value", metadata.get("profile_value", "")))
+ text_payload = _text_feature_payload(node.get("text", ""))
+ time_payload = _text_feature_payload(time_display_value or time_value)
+ profile_type_payload = _text_feature_payload(node.get("profile_type", metadata.get("profile_type", "")))
+ return {
+ "type": clean_text(node.get("type", "")),
+ "text": text_payload,
+ "time_granularity": clean_text(node.get("time_granularity", metadata.get("time_granularity", ""))),
+ "time_text": time_payload,
+ "time_compare_value": clean_text(time_value or time_display_value),
+ "profile_type": clean_text(node.get("profile_type", metadata.get("profile_type", ""))),
+ "profile_type_payload": profile_type_payload,
+ "profile_value": profile_value,
+ "status_or_text": clean_text(node.get("target_status", metadata.get("target_status", ""))) or text_payload["text"],
+ }
+
+
+def _prepare_path_scoring_features(
+ path: Mapping[str, Any],
+ *,
+ node_scoring_features_by_id: Mapping[str, Mapping[str, Any]],
+) -> Dict[str, Any]:
+ support_node_id = _path_support_node_id(path)
+ support_features = dict(node_scoring_features_by_id.get(support_node_id, {}) or {})
+ path_metadata = dict(path.get("metadata", {}) or {})
+ path_tag_payload = _text_feature_payload(
+ " ".join(
+ item
+ for item in (
+ " ".join(str(tag) for tag in list(path.get("tmcra_path_tags", path_metadata.get("tmcra_path_tags", [])) or [])),
+ clean_text(path_metadata.get("tmcra_tunnel_group_key", "")),
+ )
+ if clean_text(item)
+ )
+ )
+ support_text_tokens = set(dict(support_features.get("text", {}) or {}).get("token_set", set()))
+ support_text_tokens.update(set(path_tag_payload.get("token_set", set())))
+ return {
+ "id": clean_text(path.get("id", "")),
+ "event_id": clean_text(path.get("event_id", "")),
+ "path_type": clean_text(path.get("_path_type", path.get("type", ""))),
+ "support_node_id": support_node_id,
+ "support_type": clean_text(support_features.get("type", "")),
+ "support_time_value": clean_text(support_features.get("time_compare_value", "")),
+ "support_profile": clean_text(support_features.get("profile_value", "")),
+ "support_text_tokens": support_text_tokens,
+ "path_tag_tokens": set(path_tag_payload.get("token_set", set())),
+ }
+
+
+def _ensure_graph_scoring_feature_cache(graph_tensors: Mapping[str, Any]) -> Mapping[str, Any]:
+ if (
+ "event_scoring_features_by_id" in graph_tensors
+ and "node_scoring_features_by_id" in graph_tensors
+ and "path_scoring_features_by_id" in graph_tensors
+ ):
+ return graph_tensors
+ resolved_graph_tensors = graph_tensors if isinstance(graph_tensors, dict) else dict(graph_tensors)
+ node_lookup = {
+ clean_text(node_id): dict(node or {})
+ for node_id, node in dict(resolved_graph_tensors.get("node_by_id", {}) or {}).items()
+ if clean_text(node_id)
+ }
+ event_support_lookup = {
+ clean_text(event_id): dict(payload or {})
+ for event_id, payload in dict(resolved_graph_tensors.get("event_support_lookup", {}) or {}).items()
+ if clean_text(event_id)
+ }
+ node_scoring_features_by_id = {
+ node_id: _prepare_node_scoring_features(node)
+ for node_id, node in node_lookup.items()
+ }
+ event_scoring_features_by_id = {
+ event_id: _prepare_event_scoring_features(
+ node_lookup.get(event_id, {}),
+ support_payload=event_support_lookup.get(event_id, {}),
+ )
+ for event_id in [
+ clean_text(item)
+ for item in list(resolved_graph_tensors.get("event_node_ids", []) or [])
+ if clean_text(item)
+ ]
+ }
+ path_scoring_features_by_id = {
+ clean_text(path.get("id", "")): _prepare_path_scoring_features(
+ path,
+ node_scoring_features_by_id=node_scoring_features_by_id,
+ )
+ for path in list(resolved_graph_tensors.get("paths", []) or [])
+ if clean_text(path.get("id", ""))
+ }
+ resolved_graph_tensors["node_scoring_features_by_id"] = node_scoring_features_by_id
+ resolved_graph_tensors["event_scoring_features_by_id"] = event_scoring_features_by_id
+ resolved_graph_tensors["path_scoring_features_by_id"] = path_scoring_features_by_id
+ return resolved_graph_tensors
+
+
+@contextmanager
+def _gc_suspended() -> Any:
+ gc_enabled = gc.isenabled()
+ if gc_enabled:
+ gc.disable()
+ try:
+ yield
+ finally:
+ if gc_enabled:
+ gc.enable()
+
+
+def _time_granularity_compatible(target: Any, value: Any) -> bool:
+ normalized_target = clean_text(target)
+ normalized_value = clean_text(value)
+ if not normalized_target:
+ return True
+ if not normalized_value:
+ return False
+ if normalized_target == "day_or_coarse":
+ return normalized_value in {"day", "relative_day_reference", "month", "year"}
+ return normalized_target == normalized_value
+
+
+def _event_pair_feature_values(
+ question: str,
+ question_features: Mapping[str, Any],
+ event_node: Mapping[str, Any],
+ *,
+ support_payload: Mapping[str, Any] | None = None,
+ prepared_question: Mapping[str, Any] | None = None,
+ prepared_event: Mapping[str, Any] | None = None,
+) -> List[float]:
+ question_cache = dict(prepared_question or _prepare_question_scoring_features(question, question_features))
+ event_cache = dict(prepared_event or _prepare_event_scoring_features(event_node, support_payload=support_payload))
+ question_token_set = set(question_cache.get("question_token_set", set()))
+ question_anchor_tokens = list(question_cache.get("question_anchor_tokens", []) or [])
+ question_anchor_bigrams = set(question_cache.get("question_anchor_bigrams", set()))
+ speaker_candidates = set(question_cache.get("speaker_candidates", set()))
+ question_is_temporal = bool(question_cache.get("question_is_temporal", False))
+ semantic_target = clean_text(question_cache.get("semantic_target", ""))
+ target_status_target = clean_text(question_cache.get("target_status_target", ""))
+ time_target = clean_text(question_cache.get("time_target", ""))
+ event_speaker = normalize_text(clean_text(event_cache.get("speaker", "")))
+ event_speaker_tokens = _normalized_token_set(event_speaker)
+ event_status = clean_text(event_cache.get("target_status", ""))
+ event_time_granularity = clean_text(event_cache.get("time_granularity", ""))
+ profile_type = clean_text(event_cache.get("profile_type", ""))
+ profile_value_payload = dict(event_cache.get("profile_value", {}) or {})
+ event_signature_payload = dict(event_cache.get("event_signature", {}) or {})
+ time_payload = dict(event_cache.get("time_text", {}) or {})
+ event_text_payload = dict(event_cache.get("event_text", {}) or {})
+ source_turn_payloads = [dict(item or {}) for item in list(event_cache.get("source_turn_texts", []) or [])]
+ profile_text_payloads = [dict(item or {}) for item in list(event_cache.get("profile_texts", []) or [])]
+ time_text_payloads = [dict(item or {}) for item in list(event_cache.get("time_texts", []) or [])]
+ has_time = float(event_cache.get("has_time", 0.0) or 0.0)
+ has_profile = float(event_cache.get("has_profile", 0.0) or 0.0)
+ has_status = float(event_cache.get("has_status", 0.0) or 0.0)
+ semantic_match = 0.0
+ if semantic_target == "event_time":
+ semantic_match = has_time
+ elif semantic_target in {"identity", "research_topic", "education", "occupation"}:
+ semantic_match = 1.0 if profile_type == semantic_target else 0.0
+ elif semantic_target == "profile":
+ semantic_match = has_profile
+ time_match = 1.0 if time_target and _time_granularity_compatible(time_target, event_time_granularity) else 0.0
+ time_conflict = 1.0 if time_target and event_time_granularity and not _time_granularity_compatible(time_target, event_time_granularity) else 0.0
+ status_match = 1.0 if target_status_target and event_status and target_status_target == event_status else 0.0
+ status_conflict = 1.0 if target_status_target and event_status and target_status_target != event_status else 0.0
+ speaker_token_overlap = _required_token_coverage_from_normalized_tokens(
+ list(event_speaker_tokens),
+ question_token_set,
+ )
+ explicit_speaker_match = 1.0 if speaker_candidates and event_speaker and event_speaker in speaker_candidates else 0.0
+ speaker_match = max(explicit_speaker_match, speaker_token_overlap)
+ speaker_conflict = 1.0 if (
+ speaker_candidates
+ and event_speaker
+ and event_speaker not in speaker_candidates
+ and speaker_token_overlap <= 0.0
+ ) else 0.0
+ source_turn_overlap = _max_token_overlap_ratio_from_payloads(question_token_set, source_turn_payloads)
+ content_overlap = max(
+ _token_overlap_ratio_from_token_sets(question_token_set, set(event_text_payload.get("token_set", set()))),
+ _token_overlap_ratio_from_token_sets(question_token_set, set(event_signature_payload.get("token_set", set()))),
+ source_turn_overlap,
+ _max_token_overlap_ratio_from_payloads(question_token_set, profile_text_payloads),
+ _max_token_overlap_ratio_from_payloads(question_token_set, time_text_payloads),
+ )
+ profile_conflict = 1.0 if semantic_target in {"identity", "research_topic", "education", "occupation"} and profile_type and profile_type != semantic_target else 0.0
+ profile_missing = 1.0 if semantic_target in {"identity", "research_topic", "education", "occupation", "profile"} and not has_profile else 0.0
+ time_missing = 1.0 if semantic_target == "event_time" and not has_time else 0.0
+ source_turn_present = 1.0 if source_turn_payloads else 0.0
+ signature_anchor_coverage = _required_token_coverage_from_normalized_tokens(
+ question_anchor_tokens,
+ set(event_signature_payload.get("token_set", set())),
+ )
+ signature_anchor_bigram_coverage = _required_bigram_coverage_from_bigrams(
+ question_anchor_bigrams,
+ set(event_signature_payload.get("bigram_set", set())),
+ )
+ source_anchor_coverage = _max_required_token_coverage_from_normalized_tokens(question_anchor_tokens, source_turn_payloads)
+ event_anchor_coverage = max(
+ signature_anchor_coverage,
+ _required_token_coverage_from_normalized_tokens(question_anchor_tokens, set(event_text_payload.get("token_set", set()))),
+ source_anchor_coverage,
+ )
+ exact_signature_anchor_cover = 1.0 if question_anchor_tokens and signature_anchor_coverage >= 0.999 else 0.0
+ same_speaker_day_past_distractor = 1.0 if (
+ question_is_temporal
+ and speaker_match > 0.5
+ and event_status == "past"
+ and event_time_granularity in {"day", "relative_day_reference"}
+ and event_anchor_coverage < 0.5
+ ) else 0.0
+ temporal_anchor_miss = 1.0 if (
+ question_is_temporal
+ and (has_time > 0.5 or bool(event_time_granularity))
+ and event_anchor_coverage <= 0.0
+ ) else 0.0
+ return [
+ max(1.0 if speaker_candidates else 0.0, speaker_token_overlap),
+ speaker_match,
+ semantic_match,
+ status_match,
+ status_conflict,
+ time_match,
+ time_conflict,
+ _token_overlap_ratio_from_token_sets(question_token_set, set(event_text_payload.get("token_set", set()))),
+ _token_overlap_ratio_from_token_sets(question_token_set, set(profile_value_payload.get("token_set", set()))),
+ _token_overlap_ratio_from_token_sets(question_token_set, set(time_payload.get("token_set", set()))),
+ has_time,
+ has_profile,
+ has_status,
+ _token_overlap_ratio_from_token_sets(question_token_set, set(event_signature_payload.get("token_set", set()))),
+ speaker_conflict,
+ source_turn_overlap,
+ content_overlap,
+ profile_conflict,
+ profile_missing,
+ time_missing,
+ source_turn_present,
+ max(
+ _token_overlap_ratio_from_token_sets(question_token_set, set(profile_value_payload.get("token_set", set()))),
+ _max_token_overlap_ratio_from_payloads(question_token_set, profile_text_payloads),
+ ),
+ signature_anchor_coverage,
+ signature_anchor_bigram_coverage,
+ source_anchor_coverage,
+ event_anchor_coverage,
+ exact_signature_anchor_cover,
+ same_speaker_day_past_distractor,
+ temporal_anchor_miss,
+ ]
+
+
+def _event_relation_feature_values(
+ left_event_node: Mapping[str, Any],
+ right_event_node: Mapping[str, Any],
+ *,
+ prepared_left: Mapping[str, Any] | None = None,
+ prepared_right: Mapping[str, Any] | None = None,
+) -> List[float]:
+ left_cache = dict(prepared_left or _prepare_event_scoring_features(left_event_node))
+ right_cache = dict(prepared_right or _prepare_event_scoring_features(right_event_node))
+ return _event_relation_feature_values_from_prepared(left_cache, right_cache)
+
+
+def _event_relation_feature_values_from_prepared(
+ left_cache: Mapping[str, Any],
+ right_cache: Mapping[str, Any],
+) -> List[float]:
+ left_speaker = clean_text(left_cache.get("speaker", ""))
+ right_speaker = clean_text(right_cache.get("speaker", ""))
+ left_time_value = clean_text(left_cache.get("time_compare_value", ""))
+ right_time_value = clean_text(right_cache.get("time_compare_value", ""))
+ left_time_granularity = normalize_text(clean_text(left_cache.get("time_granularity", "")))
+ right_time_granularity = normalize_text(clean_text(right_cache.get("time_granularity", "")))
+ left_target_status = normalize_text(clean_text(left_cache.get("target_status", "")))
+ right_target_status = normalize_text(clean_text(right_cache.get("target_status", "")))
+ left_turn_index = int(left_cache.get("turn_index", 0) or 0)
+ right_turn_index = int(right_cache.get("turn_index", 0) or 0)
+ left_session = clean_text(left_cache.get("session_name", ""))
+ right_session = clean_text(right_cache.get("session_name", ""))
+ return [
+ 1.0 if left_speaker and right_speaker and left_speaker == right_speaker else 0.0,
+ 1.0 if left_time_value and right_time_value and left_time_value == right_time_value else 0.0,
+ 1.0 if left_time_granularity and right_time_granularity and left_time_granularity == right_time_granularity else 0.0,
+ 1.0 if left_target_status and right_target_status and left_target_status == right_target_status else 0.0,
+ 1.0 if left_turn_index > 0 and right_turn_index > 0 and left_session and right_session and left_session == right_session and abs(left_turn_index - right_turn_index) <= 2 else 0.0,
+ _token_overlap_ratio_from_token_sets(
+ set(left_cache.get("relation_signature_tokens", set())),
+ set(right_cache.get("relation_signature_tokens", set())),
+ ),
+ ]
+
+
+def _event_relation_feature_matrix(
+ prepared_events: Sequence[Mapping[str, Any]],
+ *,
+ dtype: torch.dtype,
+ device: torch.device,
+) -> Tensor:
+ relation_rows = [
+ [
+ _event_relation_feature_values_from_prepared(left_cache, right_cache)
+ for right_cache in prepared_events
+ ]
+ for left_cache in prepared_events
+ ]
+ if not relation_rows:
+ return torch.zeros((0, 0, MATRIX_RELATION_FEATURE_DIM), dtype=dtype, device=device)
+ return torch.tensor(relation_rows, dtype=dtype, device=device)
+
+
+def _path_pair_feature_values(
+ question: str,
+ question_features: Mapping[str, Any],
+ event_node: Mapping[str, Any],
+ support_node: Mapping[str, Any],
+ *,
+ path_type: str,
+ prepared_question: Mapping[str, Any] | None = None,
+ prepared_event: Mapping[str, Any] | None = None,
+ prepared_support: Mapping[str, Any] | None = None,
+) -> List[float]:
+ question_cache = dict(prepared_question or _prepare_question_scoring_features(question, question_features))
+ event_cache = dict(prepared_event or _prepare_event_scoring_features(event_node))
+ support_cache = dict(prepared_support or _prepare_node_scoring_features(support_node))
+ question_token_set = set(question_cache.get("question_token_set", set()))
+ question_anchor_tokens = list(question_cache.get("question_anchor_tokens", []) or [])
+ question_anchor_bigrams = set(question_cache.get("question_anchor_bigrams", set()))
+ semantic_target = clean_text(question_cache.get("semantic_target", ""))
+ target_status_target = clean_text(question_cache.get("target_status_target", ""))
+ time_target = clean_text(question_cache.get("time_target", ""))
+ question_is_temporal = bool(question_cache.get("question_is_temporal", False))
+ normalized_path_type = clean_text(path_type)
+ support_text_payload = dict(support_cache.get("text", {}) or {})
+ event_text_payload = dict(event_cache.get("event_text", {}) or {})
+ event_signature_payload = dict(event_cache.get("event_signature", {}) or {})
+ support_time_granularity = clean_text(support_cache.get("time_granularity", ""))
+ support_time_payload = dict(support_cache.get("time_text", {}) or {})
+ support_profile_type = clean_text(support_cache.get("profile_type", ""))
+ support_profile_type_payload = dict(support_cache.get("profile_type_payload", {}) or {})
+ support_status = clean_text(support_cache.get("status_or_text", ""))
+ profile_like_question = semantic_target in {"identity", "research_topic", "education", "occupation", "profile"}
+ preferred_path_type = "speaker_event_source_turn"
+ if question_is_temporal:
+ preferred_path_type = "speaker_event_time"
+ elif profile_like_question:
+ preferred_path_type = "speaker_event_profile"
+ elif target_status_target:
+ preferred_path_type = "speaker_event_status"
+ preferred_match = 1.0 if normalized_path_type == preferred_path_type else 0.0
+ dispreferred = 1.0 if normalized_path_type != preferred_path_type else 0.0
+ time_match = 1.0 if time_target and support_time_granularity and _time_granularity_compatible(time_target, support_time_granularity) else 0.0
+ time_conflict = 1.0 if time_target and support_time_granularity and not _time_granularity_compatible(time_target, support_time_granularity) else 0.0
+ profile_match = 1.0 if semantic_target in {"identity", "research_topic", "education", "occupation"} and support_profile_type == semantic_target else 0.0
+ profile_conflict = 1.0 if semantic_target in {"identity", "research_topic", "education", "occupation"} and support_profile_type and support_profile_type != semantic_target else 0.0
+ status_match = 1.0 if target_status_target and support_status and support_status == target_status_target else 0.0
+ status_conflict = 1.0 if target_status_target and support_status and support_status != target_status_target else 0.0
+ event_anchor_coverage = max(
+ _required_token_coverage_from_normalized_tokens(question_anchor_tokens, set(event_signature_payload.get("token_set", set()))),
+ _required_token_coverage_from_normalized_tokens(question_anchor_tokens, set(event_text_payload.get("token_set", set()))),
+ )
+ event_anchor_bigram_coverage = max(
+ _required_bigram_coverage_from_bigrams(question_anchor_bigrams, set(event_signature_payload.get("bigram_set", set()))),
+ _required_bigram_coverage_from_bigrams(question_anchor_bigrams, set(event_text_payload.get("bigram_set", set()))),
+ )
+ support_anchor_coverage = max(
+ _required_token_coverage_from_normalized_tokens(question_anchor_tokens, set(support_text_payload.get("token_set", set()))),
+ _required_token_coverage_from_normalized_tokens(question_anchor_tokens, set(support_time_payload.get("token_set", set()))),
+ _required_token_coverage_from_normalized_tokens(question_anchor_tokens, set(support_profile_type_payload.get("token_set", set()))),
+ )
+ support_anchor_bigram_coverage = max(
+ _required_bigram_coverage_from_bigrams(question_anchor_bigrams, set(support_text_payload.get("bigram_set", set()))),
+ _required_bigram_coverage_from_bigrams(question_anchor_bigrams, set(support_time_payload.get("bigram_set", set()))),
+ )
+ temporal_time_focus = 1.0 if (
+ question_is_temporal
+ and normalized_path_type == "speaker_event_time"
+ and time_match > 0.5
+ and event_anchor_coverage >= 0.5
+ ) else 0.0
+ temporal_non_time_penalty = 1.0 if question_is_temporal and normalized_path_type != "speaker_event_time" else 0.0
+ temporal_profile_noise = 1.0 if question_is_temporal and normalized_path_type == "speaker_event_profile" else 0.0
+ return [
+ 1.0 if normalized_path_type == "speaker_event_time" else 0.0,
+ 1.0 if normalized_path_type == "speaker_event_profile" else 0.0,
+ 1.0 if normalized_path_type == "speaker_event_status" else 0.0,
+ 1.0 if normalized_path_type == "speaker_event_source_turn" else 0.0,
+ preferred_match,
+ dispreferred,
+ _token_overlap_ratio_from_token_sets(question_token_set, set(support_text_payload.get("token_set", set()))),
+ max(
+ _token_overlap_ratio_from_token_sets(question_token_set, set(event_text_payload.get("token_set", set()))),
+ _token_overlap_ratio_from_token_sets(question_token_set, set(event_signature_payload.get("token_set", set()))),
+ ),
+ time_match,
+ time_conflict,
+ profile_match,
+ profile_conflict,
+ status_match,
+ status_conflict,
+ max(
+ _token_overlap_ratio_from_token_sets(question_token_set, set(support_time_payload.get("token_set", set()))),
+ _token_overlap_ratio_from_token_sets(question_token_set, set(support_profile_type_payload.get("token_set", set()))),
+ ),
+ event_anchor_coverage,
+ event_anchor_bigram_coverage,
+ support_anchor_coverage,
+ support_anchor_bigram_coverage,
+ temporal_time_focus,
+ temporal_non_time_penalty,
+ temporal_profile_noise,
+ ]
+
+
+def _path_relation_feature_values(
+ left_path: Mapping[str, Any],
+ right_path: Mapping[str, Any],
+ *,
+ node_lookup: Mapping[str, Mapping[str, Any]],
+ prepared_left: Mapping[str, Any] | None = None,
+ prepared_right: Mapping[str, Any] | None = None,
+) -> List[float]:
+ if prepared_left is None or prepared_right is None:
+ node_scoring_features_by_id = {
+ clean_text(node_id): _prepare_node_scoring_features(node)
+ for node_id, node in dict(node_lookup or {}).items()
+ if clean_text(node_id)
+ }
+ left_cache = dict(
+ prepared_left
+ or _prepare_path_scoring_features(
+ left_path,
+ node_scoring_features_by_id=node_scoring_features_by_id,
+ )
+ )
+ right_cache = dict(
+ prepared_right
+ or _prepare_path_scoring_features(
+ right_path,
+ node_scoring_features_by_id=node_scoring_features_by_id,
+ )
+ )
+ return _path_relation_feature_values_from_prepared(left_cache, right_cache)
+
+
+def _path_relation_feature_values_from_prepared(
+ left_cache: Mapping[str, Any],
+ right_cache: Mapping[str, Any],
+) -> List[float]:
+ left_event_id = clean_text(left_cache.get("event_id", ""))
+ right_event_id = clean_text(right_cache.get("event_id", ""))
+ left_path_type = clean_text(left_cache.get("path_type", ""))
+ right_path_type = clean_text(right_cache.get("path_type", ""))
+ left_support_node_id = clean_text(left_cache.get("support_node_id", ""))
+ right_support_node_id = clean_text(right_cache.get("support_node_id", ""))
+ left_support_type = clean_text(left_cache.get("support_type", ""))
+ right_support_type = clean_text(right_cache.get("support_type", ""))
+ left_support_time_value = clean_text(left_cache.get("support_time_value", ""))
+ right_support_time_value = clean_text(right_cache.get("support_time_value", ""))
+ left_support_profile = clean_text(left_cache.get("support_profile", ""))
+ right_support_profile = clean_text(right_cache.get("support_profile", ""))
+ return [
+ 1.0 if left_event_id and right_event_id and left_event_id == right_event_id else 0.0,
+ 1.0 if left_path_type and right_path_type and left_path_type == right_path_type else 0.0,
+ 1.0 if left_support_node_id and right_support_node_id and left_support_node_id == right_support_node_id else 0.0,
+ 1.0 if left_support_type and right_support_type and left_support_type == right_support_type else 0.0,
+ 1.0 if (
+ (left_support_time_value and right_support_time_value and left_support_time_value == right_support_time_value)
+ or (left_support_profile and right_support_profile and left_support_profile == right_support_profile)
+ ) else 0.0,
+ _token_overlap_ratio_from_token_sets(
+ set(left_cache.get("support_text_tokens", set())),
+ set(right_cache.get("support_text_tokens", set())),
+ ),
+ ]
+
+
+def _path_relation_feature_matrix(
+ prepared_paths: Sequence[Mapping[str, Any]],
+ *,
+ dtype: torch.dtype,
+ device: torch.device,
+) -> Tensor:
+ relation_rows = [
+ [
+ _path_relation_feature_values_from_prepared(left_cache, right_cache)
+ for right_cache in prepared_paths
+ ]
+ for left_cache in prepared_paths
+ ]
+ if not relation_rows:
+ return torch.zeros((0, 0, PATH_MATRIX_RELATION_FEATURE_DIM), dtype=dtype, device=device)
+ return torch.tensor(relation_rows, dtype=dtype, device=device)
+
+
+def build_path_id(event_id: str, path_type: str, support_node_id: str) -> str:
+ return f"{clean_text(event_id)}::{clean_text(path_type)}::{clean_text(support_node_id)}"
+
+
+def parse_path_id(path_id: str) -> tuple[str, str, str]:
+ normalized_path_id = clean_text(path_id)
+ for path_type in PATH_TYPES:
+ marker = f"::{path_type}::"
+ if marker in normalized_path_id:
+ event_id, support_node_id = normalized_path_id.split(marker, 1)
+ return clean_text(event_id), path_type, clean_text(support_node_id)
+ return "", "", ""
+
+
+def build_default_path_templates(
+ *,
+ event_id: str,
+ speaker_node_id: str,
+ time_node_ids: Sequence[str] = (),
+ profile_node_ids: Sequence[str] = (),
+ status_node_ids: Sequence[str] = (),
+ source_turn_node_ids: Sequence[str] = (),
+) -> List[Dict[str, Any]]:
+ paths: List[Dict[str, Any]] = []
+ for support_node_id in time_node_ids:
+ paths.append(
+ {
+ "id": build_path_id(event_id, "speaker_event_time", support_node_id),
+ "type": "speaker_event_time",
+ "event_id": event_id,
+ "node_ids": [speaker_node_id, event_id, support_node_id],
+ }
+ )
+ for support_node_id in profile_node_ids:
+ paths.append(
+ {
+ "id": build_path_id(event_id, "speaker_event_profile", support_node_id),
+ "type": "speaker_event_profile",
+ "event_id": event_id,
+ "node_ids": [speaker_node_id, event_id, support_node_id],
+ }
+ )
+ for support_node_id in status_node_ids:
+ paths.append(
+ {
+ "id": build_path_id(event_id, "speaker_event_status", support_node_id),
+ "type": "speaker_event_status",
+ "event_id": event_id,
+ "node_ids": [speaker_node_id, event_id, support_node_id],
+ }
+ )
+ for support_node_id in source_turn_node_ids:
+ paths.append(
+ {
+ "id": build_path_id(event_id, "speaker_event_source_turn", support_node_id),
+ "type": "speaker_event_source_turn",
+ "event_id": event_id,
+ "node_ids": [speaker_node_id, event_id, support_node_id],
+ }
+ )
+ return paths
+
+
+@dataclass(slots=True)
+class QueryTrainingExample:
+ conversation_id: str
+ question_id: str
+ question: str
+ question_features: Dict[str, Any]
+ candidate_event_ids: List[str]
+ positive_event_ids: List[str]
+ positive_path_ids: List[str]
+ positive_time_node_ids: List[str]
+ negative_event_ids: List[str]
+ answer_targets: Dict[str, Any]
+ answer_plan_targets: Dict[str, Any] = field(default_factory=dict)
+ temporal_target: Dict[str, Any] = field(default_factory=dict)
+ event_catalog_size: int = 0
+ metadata: Dict[str, Any] = field(default_factory=dict)
+ hard_negative_event_ids: List[str] = field(default_factory=list, repr=False)
+ easy_negative_event_ids: List[str] = field(default_factory=list, repr=False)
+ question_anchor_tokens: List[str] = field(default_factory=list, repr=False)
+ question_hash_indices: List[int] = field(default_factory=list, repr=False)
+ question_feature_index: int | None = field(default=None, repr=False)
+ question_query_type_index: int | None = field(default=None, repr=False)
+
+ def __post_init__(self) -> None:
+ self.candidate_event_ids = [clean_text(item) for item in list(self.candidate_event_ids or []) if clean_text(item)]
+ self.positive_event_ids = [clean_text(item) for item in list(self.positive_event_ids or []) if clean_text(item)]
+ self.positive_path_ids = [clean_text(item) for item in list(self.positive_path_ids or []) if clean_text(item)]
+ self.positive_time_node_ids = [clean_text(item) for item in list(self.positive_time_node_ids or []) if clean_text(item)]
+ legacy_negative_ids = [clean_text(item) for item in list(self.negative_event_ids or []) if clean_text(item)]
+ explicit_hard_negative_ids = [clean_text(item) for item in list(self.hard_negative_event_ids or []) if clean_text(item)]
+ explicit_easy_negative_ids = [clean_text(item) for item in list(self.easy_negative_event_ids or []) if clean_text(item)]
+ positive_event_id_set = {item for item in self.positive_event_ids if item}
+ if not explicit_hard_negative_ids and legacy_negative_ids:
+ explicit_hard_negative_ids = list(legacy_negative_ids)
+ combined_negative_ids = dedupe_texts([*legacy_negative_ids, *explicit_hard_negative_ids, *explicit_easy_negative_ids])
+ self.negative_event_ids = [event_id for event_id in combined_negative_ids if event_id not in positive_event_id_set]
+ negative_event_id_set = {item for item in self.negative_event_ids if item}
+ self.hard_negative_event_ids = [
+ event_id
+ for event_id in dedupe_texts(explicit_hard_negative_ids)
+ if event_id in negative_event_id_set
+ ]
+ hard_negative_event_id_set = {item for item in self.hard_negative_event_ids if item}
+ self.easy_negative_event_ids = [
+ event_id
+ for event_id in dedupe_texts(explicit_easy_negative_ids)
+ if event_id in negative_event_id_set and event_id not in hard_negative_event_id_set
+ ]
+ prepared_features = dict(self.question_features or {})
+ cached_anchor_tokens = [normalize_text(clean_text(token)) for token in list(self.question_anchor_tokens or []) if clean_text(token)]
+ if not cached_anchor_tokens:
+ cached_anchor_tokens = _question_anchor_tokens(self.question, prepared_features)
+ self.question_anchor_tokens = list(cached_anchor_tokens)
+ if self.question_anchor_tokens:
+ prepared_features["question_anchor_tokens"] = list(self.question_anchor_tokens)
+ self.question_features = prepared_features
+ if not self.question_hash_indices:
+ self.question_hash_indices = _text_hash_indices(self.question, QUESTION_HASH_BUCKETS)
+ feature_index, query_type_index = _question_feature_indices(self.question_features)
+ if self.question_feature_index is None:
+ self.question_feature_index = feature_index
+ if self.question_query_type_index is None:
+ self.question_query_type_index = query_type_index
+
+ @classmethod
+ def from_dict(cls, payload: Mapping[str, Any]) -> "QueryTrainingExample":
+ return cls(
+ conversation_id=clean_text(payload.get("conversation_id", "")),
+ question_id=clean_text(payload.get("question_id", "")),
+ question=clean_text(payload.get("question", "")),
+ question_features=dict(payload.get("question_features", {}) or {}),
+ candidate_event_ids=[clean_text(item) for item in list(payload.get("candidate_event_ids", []) or []) if clean_text(item)],
+ positive_event_ids=[clean_text(item) for item in list(payload.get("positive_event_ids", []) or []) if clean_text(item)],
+ positive_path_ids=[clean_text(item) for item in list(payload.get("positive_path_ids", []) or []) if clean_text(item)],
+ positive_time_node_ids=[clean_text(item) for item in list(payload.get("positive_time_node_ids", []) or []) if clean_text(item)],
+ negative_event_ids=[clean_text(item) for item in list(payload.get("negative_event_ids", []) or []) if clean_text(item)],
+ answer_targets=dict(payload.get("answer_targets", {}) or {}),
+ answer_plan_targets=dict(payload.get("answer_plan_targets", {}) or {}),
+ temporal_target=dict(payload.get("temporal_target", {}) or {}),
+ event_catalog_size=int(payload.get("event_catalog_size", 0) or 0),
+ metadata=dict(payload.get("metadata", {}) or {}),
+ hard_negative_event_ids=[clean_text(item) for item in list(payload.get("hard_negative_event_ids", []) or []) if clean_text(item)],
+ easy_negative_event_ids=[clean_text(item) for item in list(payload.get("easy_negative_event_ids", []) or []) if clean_text(item)],
+ question_anchor_tokens=[clean_text(item) for item in list(payload.get("question_anchor_tokens", []) or []) if clean_text(item)],
+ )
+
+ def to_dict(self) -> Dict[str, Any]:
+ return {
+ "conversation_id": self.conversation_id,
+ "question_id": self.question_id,
+ "question": self.question,
+ "question_features": dict(self.question_features),
+ "candidate_event_ids": list(self.candidate_event_ids),
+ "positive_event_ids": list(self.positive_event_ids),
+ "positive_path_ids": list(self.positive_path_ids),
+ "positive_time_node_ids": list(self.positive_time_node_ids),
+ "hard_negative_event_ids": list(self.hard_negative_event_ids),
+ "easy_negative_event_ids": list(self.easy_negative_event_ids),
+ "negative_event_ids": list(self.negative_event_ids),
+ "answer_targets": dict(self.answer_targets),
+ "answer_plan_targets": dict(self.answer_plan_targets),
+ "temporal_target": dict(self.temporal_target),
+ "event_catalog_size": int(self.event_catalog_size),
+ "metadata": dict(self.metadata),
+ }
+
+
+@dataclass(slots=True)
+class _TrainingConversationSummary:
+ conversation_id: str
+ source_dataset: str
+ row_count: int = 0
+ time_example_count: int = 0
+ multi_evidence_count: int = 0
+ temporal_positive_count: int = 0
+ answer_type_counts: Dict[str, int] = field(default_factory=dict)
+
+
+@dataclass(slots=True)
+class _TrainingConversationGroup:
+ conversation_id: str
+ source_dataset: str
+ row_count: int
+ time_example_count: int
+ multi_evidence_count: int
+ temporal_positive_count: int
+ answer_type_counts: Dict[str, int]
+ repeat_count: int = 1
+ sampling_multiplier: float = 1.0
+ source_store: Any | None = field(default=None, repr=False)
+ offsets: tuple[int, ...] = field(default_factory=tuple, repr=False)
+ rows: tuple[QueryTrainingExample, ...] = field(default_factory=tuple, repr=False)
+
+
+def _clone_query_training_example(
+ example: QueryTrainingExample,
+ *,
+ metadata_updates: Mapping[str, Any] | None = None,
+) -> QueryTrainingExample:
+ cloned = QueryTrainingExample(
+ conversation_id=example.conversation_id,
+ question_id=example.question_id,
+ question=example.question,
+ question_features=dict(example.question_features or {}),
+ candidate_event_ids=list(example.candidate_event_ids or []),
+ positive_event_ids=list(example.positive_event_ids or []),
+ positive_path_ids=list(example.positive_path_ids or []),
+ positive_time_node_ids=list(example.positive_time_node_ids or []),
+ negative_event_ids=list(example.negative_event_ids or []),
+ answer_targets=dict(example.answer_targets or {}),
+ answer_plan_targets=dict(example.answer_plan_targets or {}),
+ temporal_target=dict(example.temporal_target or {}),
+ event_catalog_size=int(example.event_catalog_size),
+ metadata={
+ **dict(example.metadata or {}),
+ **dict(metadata_updates or {}),
+ },
+ hard_negative_event_ids=list(example.hard_negative_event_ids or []),
+ easy_negative_event_ids=list(example.easy_negative_event_ids or []),
+ question_anchor_tokens=list(example.question_anchor_tokens or []),
+ question_hash_indices=list(example.question_hash_indices or []),
+ question_feature_index=example.question_feature_index,
+ question_query_type_index=example.question_query_type_index,
+ )
+ return cloned
+
+
+def _training_source_dataset_name(*, example: QueryTrainingExample, fallback_source: str = "") -> str:
+ metadata = dict(example.metadata or {})
+ dataset_name = clean_text(
+ metadata.get("source_dataset", "")
+ or metadata.get("dataset", "")
+ or metadata.get("source", "")
+ or metadata.get("origin_dataset", "")
+ or metadata.get("source_data", "")
+ )
+ if dataset_name:
+ return dataset_name
+ conversation_id = clean_text(example.conversation_id)
+ if "__" in conversation_id:
+ prefix = clean_text(conversation_id.split("__", 1)[0])
+ if prefix:
+ return prefix
+ fallback = clean_text(fallback_source)
+ return fallback or "unknown"
+
+
+def _training_answer_type_name(example: QueryTrainingExample) -> str:
+ answer_type = clean_text(dict(example.answer_targets or {}).get("answer_type", ""))
+ if answer_type in ANSWER_TYPE_TO_ID:
+ return answer_type
+ inferred_index = int(_answer_type_label(example))
+ if 0 <= inferred_index < len(ANSWER_TYPES):
+ return ANSWER_TYPES[inferred_index]
+ return "event_text"
+
+
+def _positive_event_id_count(example: QueryTrainingExample) -> int:
+ return len({clean_text(item) for item in list(example.positive_event_ids or []) if clean_text(item)})
+
+
+def _training_supervision_bucket_name(example: QueryTrainingExample) -> str:
+ metadata = dict(example.metadata or {})
+ answer_type = _training_answer_type_name(example)
+ positive_count = _positive_event_id_count(example)
+ explicit_bucket = clean_text(metadata.get("supervision_bucket", ""))
+ if explicit_bucket:
+ # Old prepared rows can carry stale metadata.supervision_bucket values.
+ # The provable multi-positive cases are safer to reclassify than to let
+ # them hide inside ordinary event_text/time/profile metrics.
+ if answer_type == "event_text" and positive_count > 1 and explicit_bucket == "event_text":
+ return "list_event_text_multi_positive"
+ if answer_type in {"time", "profile"} and positive_count > 1 and explicit_bucket == answer_type:
+ return f"{answer_type}_multi_positive"
+ return explicit_bucket
+ if answer_type == "abstain":
+ return "abstain"
+ if positive_count > 1:
+ if answer_type == "event_text":
+ return "list_event_text_multi_positive"
+ if answer_type in {"time", "profile"}:
+ return f"{answer_type}_multi_positive"
+ return "multi_evidence"
+ return answer_type or "event_text"
+
+
+def _loss_group_key_for_example(
+ example: QueryTrainingExample,
+ supervision: Mapping[str, Any] | None = None,
+ *,
+ mode: str = "answer_type",
+) -> str:
+ normalized_mode = clean_text(mode or "none").lower().replace("-", "_")
+ if normalized_mode in {"", "none", "off", "disabled"}:
+ return ""
+ answer_type = _training_answer_type_name(example)
+ if normalized_mode in {"answer_type", "type"}:
+ return answer_type
+ supervision_bucket = clean_text(dict(supervision or {}).get("supervision_bucket", ""))
+ if not supervision_bucket:
+ supervision_bucket = _training_supervision_bucket_name(example)
+ if normalized_mode in {"supervision_bucket", "supervision", "task_bucket", "bucket"}:
+ return supervision_bucket
+ if normalized_mode in {"source_supervision_bucket", "source_bucket"}:
+ source_dataset = clean_text(dict(supervision or {}).get("training_source_dataset", ""))
+ if not source_dataset:
+ source_dataset = _training_source_dataset_name(example=example, fallback_source="unknown")
+ return f"{source_dataset}:{supervision_bucket}"
+ if normalized_mode in {"source_answer_type", "source_type"}:
+ source_dataset = clean_text(dict(supervision or {}).get("training_source_dataset", ""))
+ if not source_dataset:
+ source_dataset = _training_source_dataset_name(example=example, fallback_source="unknown")
+ return f"{source_dataset}:{answer_type}"
+ if normalized_mode in {"source", "dataset"}:
+ source_dataset = clean_text(dict(supervision or {}).get("training_source_dataset", ""))
+ if not source_dataset:
+ source_dataset = _training_source_dataset_name(example=example, fallback_source="unknown")
+ return source_dataset
+ return answer_type
+
+
+def _group_balanced_batch_loss(
+ loss_entries: Sequence[tuple[Tensor, str]],
+ *,
+ mode: str = "answer_type",
+) -> tuple[Tensor, int]:
+ if not loss_entries:
+ raise ValueError("loss_entries must not be empty")
+ losses = [loss for loss, _ in loss_entries]
+ normalized_mode = clean_text(mode or "none").lower().replace("-", "_")
+ if normalized_mode in {"", "none", "off", "disabled"}:
+ return torch.stack(losses).mean(), 1
+ grouped_losses: "OrderedDict[str, List[Tensor]]" = OrderedDict()
+ for loss, group_key in loss_entries:
+ resolved_group_key = clean_text(group_key) or "unknown"
+ grouped_losses.setdefault(resolved_group_key, []).append(loss)
+ if len(grouped_losses) <= 1:
+ return torch.stack(losses).mean(), len(grouped_losses) or 1
+ group_means = [torch.stack(group_values).mean() for group_values in grouped_losses.values() if group_values]
+ if not group_means:
+ return torch.stack(losses).mean(), 1
+ return torch.stack(group_means).mean(), len(group_means)
+
+
+def _new_answer_type_metric_stats() -> Dict[str, int]:
+ return {
+ "samples": 0,
+ "recall_event_recall_total": 0,
+ "recall_event24_hits": 0,
+ "recall_event24_positive_hits": 0,
+ "recall_event24_positive_total": 0,
+ "event_recall_total": 0,
+ "event_recall1_hits": 0,
+ "event_recall5_hits": 0,
+ "event5_positive_hits": 0,
+ "event5_positive_total": 0,
+ "path_recall_total": 0,
+ "path_recall3_hits": 0,
+ "path3_positive_hits": 0,
+ "path3_positive_total": 0,
+ "answer_plan_selected_total": 0,
+ "answer_plan_selected_recall5_hits": 0,
+ "answer_plan_selected_positive_hits": 0,
+ "answer_plan_selected_positive_total": 0,
+ "answer_plan_current_total": 0,
+ "answer_plan_current_top1_hits": 0,
+ "temporal_total": 0,
+ "temporal_hits": 0,
+ }
+
+
+def _finalize_answer_type_metric_stats(
+ stats_by_answer_type: Mapping[str, Mapping[str, int]],
+) -> Dict[str, Dict[str, Any]]:
+ finalized: Dict[str, Dict[str, Any]] = {}
+ for answer_type, stats in sorted(stats_by_answer_type.items()):
+ recall_total = int(stats.get("recall_event_recall_total", 0) or 0)
+ event_total = int(stats.get("event_recall_total", 0) or 0)
+ path_total = int(stats.get("path_recall_total", 0) or 0)
+ answer_plan_selected_total = int(stats.get("answer_plan_selected_total", 0) or 0)
+ answer_plan_current_total = int(stats.get("answer_plan_current_total", 0) or 0)
+ temporal_total = int(stats.get("temporal_total", 0) or 0)
+ finalized[answer_type] = {
+ "samples": int(stats.get("samples", 0) or 0),
+ "recall_event_recall_at_24": round(float(stats.get("recall_event24_hits", 0) or 0) / max(1, recall_total), 6),
+ "recall_event_positive_coverage_at_24": round(
+ float(stats.get("recall_event24_positive_hits", 0) or 0)
+ / max(1, int(stats.get("recall_event24_positive_total", 0) or 0)),
+ 6,
+ ),
+ "event_recall_at_1": round(float(stats.get("event_recall1_hits", 0) or 0) / max(1, event_total), 6),
+ "event_recall_at_5": round(float(stats.get("event_recall5_hits", 0) or 0) / max(1, event_total), 6),
+ "event_positive_coverage_at_5": round(
+ float(stats.get("event5_positive_hits", 0) or 0)
+ / max(1, int(stats.get("event5_positive_total", 0) or 0)),
+ 6,
+ ),
+ "path_recall_at_3": round(float(stats.get("path_recall3_hits", 0) or 0) / max(1, path_total), 6),
+ "path_positive_coverage_at_3": round(
+ float(stats.get("path3_positive_hits", 0) or 0)
+ / max(1, int(stats.get("path3_positive_total", 0) or 0)),
+ 6,
+ ),
+ "answer_plan_selected_recall_at_5": round(
+ float(stats.get("answer_plan_selected_recall5_hits", 0) or 0)
+ / max(1, answer_plan_selected_total),
+ 6,
+ ),
+ "answer_plan_selected_positive_coverage_at_5": round(
+ float(stats.get("answer_plan_selected_positive_hits", 0) or 0)
+ / max(1, int(stats.get("answer_plan_selected_positive_total", 0) or 0)),
+ 6,
+ ),
+ "answer_plan_current_top1_accuracy": round(
+ float(stats.get("answer_plan_current_top1_hits", 0) or 0)
+ / max(1, answer_plan_current_total),
+ 6,
+ ),
+ "temporal_accuracy": round(float(stats.get("temporal_hits", 0) or 0) / max(1, temporal_total), 6),
+ "recall_event_recall_total": recall_total,
+ "recall_event24_positive_total": int(stats.get("recall_event24_positive_total", 0) or 0),
+ "event_recall_total": event_total,
+ "event5_positive_total": int(stats.get("event5_positive_total", 0) or 0),
+ "path_recall_total": path_total,
+ "path3_positive_total": int(stats.get("path3_positive_total", 0) or 0),
+ "answer_plan_selected_total": answer_plan_selected_total,
+ "answer_plan_selected_positive_total": int(stats.get("answer_plan_selected_positive_total", 0) or 0),
+ "answer_plan_current_total": answer_plan_current_total,
+ "temporal_total": temporal_total,
+ }
+ return finalized
+
+
+def _merge_nested_metric_summaries(metrics_list: Sequence[Mapping[str, Any]], nested_key: str) -> Dict[str, Dict[str, Any]]:
+ group_names: set[str] = set()
+ for metrics in metrics_list:
+ nested = metrics.get(nested_key, {})
+ if isinstance(nested, Mapping):
+ group_names.update(clean_text(group_name) for group_name in nested.keys() if clean_text(group_name))
+ merged: Dict[str, Dict[str, Any]] = {}
+ for group_name in sorted(group_names):
+ group_metrics_list = [
+ dict(dict(metrics.get(nested_key, {}) or {}).get(group_name, {}) or {})
+ for metrics in metrics_list
+ if isinstance(metrics.get(nested_key, {}), Mapping)
+ and isinstance(dict(metrics.get(nested_key, {}) or {}).get(group_name, {}), Mapping)
+ ]
+ if not group_metrics_list:
+ continue
+ group_summary: Dict[str, Any] = {}
+ metric_keys: set[str] = set()
+ for group_metrics in group_metrics_list:
+ metric_keys.update(clean_text(key) for key in group_metrics.keys() if clean_text(key))
+ for metric_key in sorted(metric_keys):
+ if metric_key in _METRIC_COUNT_KEYS:
+ group_summary[metric_key] = int(sum(int(item.get(metric_key, 0) or 0) for item in group_metrics_list))
+ continue
+ weight_key = _METRIC_WEIGHT_KEYS.get(metric_key, "samples")
+ total_weight = float(sum(item.get(weight_key, 0.0) or 0.0 for item in group_metrics_list))
+ if total_weight <= 0.0:
+ group_summary[metric_key] = 0.0
+ continue
+ weighted_total = sum(
+ float(item.get(metric_key, 0.0) or 0.0) * float(item.get(weight_key, 0.0) or 0.0)
+ for item in group_metrics_list
+ )
+ group_summary[metric_key] = round(weighted_total / total_weight, 6)
+ merged[group_name] = group_summary
+ return merged
+
+
+def _has_positive_temporal_supervision(example: QueryTrainingExample) -> bool:
+ if example.positive_time_node_ids:
+ return True
+ temporal_target = dict(example.temporal_target or {})
+ return bool(temporal_target.get("has_positive_time_supervision", False))
+
+
+def _has_multi_evidence_supervision(example: QueryTrainingExample) -> bool:
+ if _training_answer_type_name(example) == "multi_evidence":
+ return True
+ return len({clean_text(item) for item in example.positive_event_ids if clean_text(item)}) > 1
+
+
+def _smoothed_inverse_share_factor(
+ *,
+ count: int,
+ total: int,
+ unique_count: int,
+ alpha: float,
+ blend_uniform_ratio: float,
+) -> float:
+ resolved_total = max(1, int(total or 0))
+ resolved_unique_count = max(1, int(unique_count or 0))
+ resolved_count = max(1, int(count or 0))
+ if resolved_unique_count <= 1:
+ return 1.0
+ source_share = float(resolved_count) / float(resolved_total)
+ uniform_share = 1.0 / float(resolved_unique_count)
+ raw_factor = math.pow(max(uniform_share / max(source_share, 1e-12), 1e-12), max(0.0, float(alpha)))
+ blend = min(1.0, max(0.0, float(blend_uniform_ratio)))
+ return blend + (1.0 - blend) * raw_factor
+
+
+def _bounded_training_weight(value: float, *, minimum: float, maximum: float) -> float:
+ resolved_minimum = float(minimum)
+ resolved_maximum = float(maximum)
+ if resolved_maximum < resolved_minimum:
+ resolved_maximum = resolved_minimum
+ return min(resolved_maximum, max(resolved_minimum, float(value)))
+
+
+def _conversation_repeat_count(
+ conversation_id: str,
+ *,
+ sampling_multiplier: float,
+ max_group_repeat: int,
+) -> int:
+ resolved_max_group_repeat = max(1, int(max_group_repeat or 1))
+ capped_multiplier = min(float(resolved_max_group_repeat), max(1.0, float(sampling_multiplier or 1.0)))
+ base_repeat = 1 + int(math.floor(max(0.0, capped_multiplier - 1.0)))
+ repeat_count = min(resolved_max_group_repeat, base_repeat)
+ fractional_extra = max(0.0, capped_multiplier - float(base_repeat))
+ if repeat_count < resolved_max_group_repeat and fractional_extra > 1e-9:
+ threshold = float(stable_hash_bucket(f"train-repeat::{clean_text(conversation_id)}", 1_000_000)) / 1_000_000.0
+ if threshold < fractional_extra:
+ repeat_count += 1
+ return max(1, min(resolved_max_group_repeat, repeat_count))
+
+
+@dataclass(frozen=True, slots=True)
+class QueryBatchSourceOffsets:
+ path: str
+ offsets: tuple[int, ...]
+ skip_bad_rows: bool = False
+
+
+@dataclass(frozen=True, slots=True)
+class QueryBatchSpec:
+ sources: tuple[QueryBatchSourceOffsets, ...]
+
+
+@dataclass(slots=True)
+class GraphCacheItem:
+ graph: Dict[str, Any]
+ tensors: Dict[str, Any]
+
+
+class LazyGraphCache(Mapping[str, GraphCacheItem]):
+ def __init__(
+ self,
+ *,
+ graph_paths: Mapping[str, Path],
+ conversation_ids: Sequence[str] | None = None,
+ graph_dir: Path | None = None,
+ known_graph_count: int | None = None,
+ device: torch.device,
+ cache_dir: Path | None = None,
+ memory_cache_size: int = 64,
+ prefetch_workers: int = 0,
+ prefetch_window: int = 0,
+ cache_write_enabled: bool = True,
+ require_cache_hit: bool = False,
+ progress_callback: Callable[[str, Dict[str, Any]], None] | None = None,
+ ) -> None:
+ self._graph_paths = {clean_text(key): Path(value) for key, value in dict(graph_paths or {}).items() if clean_text(key)}
+ self._conversation_ids = [clean_text(item) for item in list(conversation_ids or []) if clean_text(item)]
+ self._graph_dir = Path(graph_dir) if graph_dir is not None else None
+ self._known_graph_count = max(len(self._graph_paths), len(self._conversation_ids), int(known_graph_count or 0))
+ self._device = torch.device(device)
+ self._cache_dir = Path(cache_dir) if cache_dir is not None else None
+ self._memory_cache_size = max(0, int(memory_cache_size or 0))
+ self._memory_cache: OrderedDict[str, GraphCacheItem] = OrderedDict()
+ self._progress_callback = progress_callback
+ self._disk_cache_hits = 0
+ self._tensorized_count = 0
+ self._memory_cache_hits = 0
+ self._load_count = 0
+ self._pin_host_tensors = self._device.type == "cpu" and torch.cuda.is_available()
+ self._cache_write_enabled = bool(cache_write_enabled)
+ self._require_cache_hit = bool(require_cache_hit)
+ self._prefetch_workers = max(0, int(prefetch_workers or 0)) if self._device.type == "cpu" else 0
+ self._prefetch_window = max(0, int(prefetch_window or 0))
+ self._prefetch_submitted = 0
+ self._prefetch_completed = 0
+ self._prefetch_failed = 0
+ self._prefetch_lock = threading.Lock()
+ self._prefetch_futures: OrderedDict[str, Future[tuple[GraphCacheItem, str]]] = OrderedDict()
+ self._prefetch_executor = (
+ ThreadPoolExecutor(max_workers=self._prefetch_workers, thread_name_prefix="lazy-graph-cache")
+ if self._prefetch_workers > 0 and self._prefetch_window > 0
+ else None
+ )
+ if self._cache_dir is not None:
+ self._cache_dir.mkdir(parents=True, exist_ok=True)
+
+ def __len__(self) -> int:
+ return max(len(self._graph_paths), len(self._conversation_ids), int(self._known_graph_count))
+
+ def __iter__(self):
+ if self._conversation_ids:
+ return iter(self._conversation_ids)
+ return iter(self._graph_paths)
+
+ def __getitem__(self, key: str) -> GraphCacheItem:
+ conversation_id = clean_text(key)
+ graph_path = self._graph_paths.get(conversation_id)
+ if graph_path is None:
+ if self._graph_dir is None:
+ raise KeyError(key)
+ graph_path = self._graph_dir / f"{conversation_id}.json"
+ self._graph_paths[conversation_id] = graph_path
+ self._collect_completed_prefetch()
+ cached_item = self._memory_cache.get(conversation_id)
+ if cached_item is not None:
+ self._memory_cache.move_to_end(conversation_id)
+ self._memory_cache_hits += 1
+ return cached_item
+
+ prefetched_item = self._consume_prefetched_item(conversation_id)
+ if prefetched_item is not None:
+ self._memory_cache_hits += 1
+ return prefetched_item
+
+ item, loaded_from = self._load_item(conversation_id)
+ self._store_loaded_item(conversation_id, item, loaded_from=loaded_from)
+ return item
+
+ def close(self) -> None:
+ executor = self._prefetch_executor
+ self._prefetch_executor = None
+ if executor is not None:
+ executor.shutdown(wait=False, cancel_futures=True)
+
+ def __del__(self) -> None:
+ try:
+ self.close()
+ except Exception:
+ pass
+
+ def stats(self) -> Dict[str, int]:
+ self._collect_completed_prefetch()
+ return {
+ "graph_count": int(len(self)),
+ "loads": int(self._load_count),
+ "memory_cache_hits": int(self._memory_cache_hits),
+ "disk_cache_hits": int(self._disk_cache_hits),
+ "tensorized": int(self._tensorized_count),
+ "prefetch_submitted": int(self._prefetch_submitted),
+ "prefetch_completed": int(self._prefetch_completed),
+ "prefetch_failed": int(self._prefetch_failed),
+ "prefetch_pending": int(len(self._prefetch_futures)),
+ "memory_cache_size": int(len(self._memory_cache)),
+ }
+
+ def batch_prepare_context(self) -> Dict[str, Any]:
+ worker_memory_cache_size = 0
+ if self._memory_cache_size > 0:
+ worker_memory_cache_size = min(32, max(2, int(self._memory_cache_size or 0)))
+ if self._graph_dir is not None:
+ return {
+ "conversation_specs": {},
+ "graph_dir": str(self._graph_dir),
+ "cache_dir": str(self._cache_dir) if self._cache_dir is not None else "",
+ "cache_write_enabled": bool(self._cache_write_enabled),
+ "require_cache_hit": bool(self._require_cache_hit),
+ "worker_memory_cache_size": int(worker_memory_cache_size),
+ }
+ conversation_specs: Dict[str, Dict[str, Any]] = {}
+ for conversation_id in [clean_text(item) for item in list(self._graph_paths) if clean_text(item)]:
+ graph_path = self._graph_paths.get(conversation_id)
+ if graph_path is None:
+ continue
+ cache_path = _graph_tensor_cache_path(self._cache_dir, graph_path) if self._cache_dir is not None else None
+ conversation_specs[conversation_id] = {
+ "graph_path": str(graph_path),
+ "cache_path": str(cache_path) if cache_path is not None else "",
+ }
+ return {
+ "conversation_specs": conversation_specs,
+ "graph_dir": "",
+ "cache_dir": "",
+ "cache_write_enabled": bool(self._cache_write_enabled),
+ "require_cache_hit": bool(self._require_cache_hit),
+ "worker_memory_cache_size": int(worker_memory_cache_size),
+ }
+
+ def prefetch(self, conversation_ids: Sequence[str]) -> None:
+ if self._prefetch_executor is None or self._prefetch_window <= 0:
+ return
+ self._collect_completed_prefetch()
+ normalized_ids: List[str] = []
+ for item in list(conversation_ids or []):
+ conversation_id = clean_text(item)
+ if not conversation_id:
+ continue
+ if conversation_id not in self._graph_paths:
+ if self._graph_dir is None:
+ continue
+ self._graph_paths[conversation_id] = self._graph_dir / f"{conversation_id}.json"
+ normalized_ids.append(conversation_id)
+ if not normalized_ids:
+ return
+ with self._prefetch_lock:
+ for conversation_id in normalized_ids:
+ if conversation_id in self._memory_cache or conversation_id in self._prefetch_futures:
+ continue
+ if len(self._prefetch_futures) >= self._prefetch_window:
+ break
+ future = self._prefetch_executor.submit(self._load_item, conversation_id)
+ self._prefetch_futures[conversation_id] = future
+ self._prefetch_submitted += 1
+
+ def _load_item(self, conversation_id: str) -> tuple[GraphCacheItem, str]:
+ graph_path = self._graph_paths.get(conversation_id)
+ if graph_path is None:
+ if self._graph_dir is None:
+ raise KeyError(conversation_id)
+ graph_path = self._graph_dir / f"{conversation_id}.json"
+ self._graph_paths[conversation_id] = graph_path
+ source_signature = _graph_source_signature(graph_path)
+ payload = None
+ loaded_from = "tensorized"
+ if self._cache_dir is not None:
+ payload = _load_graph_tensor_cache(
+ _graph_tensor_cache_path(self._cache_dir, graph_path),
+ source_signature=source_signature,
+ )
+ if payload is not None:
+ graph = dict(payload["graph"])
+ tensors = _graph_tensor_mapping_to_device(payload["tensors"], self._device)
+ loaded_from = "disk_cache"
+ else:
+ if self._cache_dir is not None and self._require_cache_hit:
+ raise FileNotFoundError(f"Prebuilt graph tensor cache missing for conversation '{conversation_id}': {graph_path}")
+ graph = dict(read_json(graph_path))
+ cache_tensors = tensorize_graph(graph, device=torch.device("cpu"))
+ if self._cache_dir is not None and self._cache_write_enabled:
+ _write_graph_tensor_cache(
+ _graph_tensor_cache_path(self._cache_dir, graph_path),
+ source_signature=source_signature,
+ graph=graph,
+ tensors=cache_tensors,
+ )
+ tensors = _graph_tensor_mapping_to_device(cache_tensors, self._device)
+ tensors = dict(_ensure_graph_scoring_feature_cache(tensors))
+ if self._pin_host_tensors:
+ tensors = _pin_graph_tensor_mapping(tensors)
+ return GraphCacheItem(graph=graph, tensors=tensors), loaded_from
+
+ def _store_loaded_item(self, conversation_id: str, item: GraphCacheItem, *, loaded_from: str) -> GraphCacheItem:
+ self._load_count += 1
+ if loaded_from == "disk_cache":
+ self._disk_cache_hits += 1
+ else:
+ self._tensorized_count += 1
+ if self._memory_cache_size > 0:
+ self._memory_cache[conversation_id] = item
+ self._memory_cache.move_to_end(conversation_id)
+ while len(self._memory_cache) > self._memory_cache_size:
+ self._memory_cache.popitem(last=False)
+ self._maybe_report_progress(loaded_from=loaded_from, conversation_id=conversation_id)
+ return item
+
+ def _collect_completed_prefetch(self) -> None:
+ if not self._prefetch_futures:
+ return
+ completed_ids: List[str] = []
+ with self._prefetch_lock:
+ for conversation_id, future in list(self._prefetch_futures.items()):
+ if not future.done():
+ continue
+ completed_ids.append(conversation_id)
+ try:
+ item, loaded_from = future.result()
+ except Exception as exc:
+ self._prefetch_failed += 1
+ self._report_prefetch_error(conversation_id=conversation_id, exc=exc)
+ continue
+ self._store_loaded_item(conversation_id, item, loaded_from=loaded_from)
+ self._prefetch_completed += 1
+ for conversation_id in completed_ids:
+ self._prefetch_futures.pop(conversation_id, None)
+
+ def _consume_prefetched_item(self, conversation_id: str) -> GraphCacheItem | None:
+ with self._prefetch_lock:
+ future = self._prefetch_futures.pop(conversation_id, None)
+ if future is None:
+ return None
+ try:
+ item, loaded_from = future.result()
+ except Exception as exc:
+ self._prefetch_failed += 1
+ self._report_prefetch_error(conversation_id=conversation_id, exc=exc)
+ return None
+ self._prefetch_completed += 1
+ return self._store_loaded_item(conversation_id, item, loaded_from=loaded_from)
+
+ def _report_prefetch_error(self, *, conversation_id: str, exc: Exception) -> None:
+ if self._progress_callback is None:
+ return
+ self._progress_callback(
+ "lazy_graph_cache_prefetch_error",
+ {
+ "conversation_id": clean_text(conversation_id),
+ "error_type": type(exc).__name__,
+ "error_message": clean_text(str(exc)),
+ "prefetch_failed": int(self._prefetch_failed),
+ },
+ )
+
+ def _maybe_report_progress(self, *, loaded_from: str, conversation_id: str) -> None:
+ if self._progress_callback is None:
+ return
+ if self._load_count not in {1} and self._load_count % 1000 != 0:
+ return
+ self._progress_callback(
+ "lazy_graph_cache_progress",
+ {
+ "graph_count": len(self._graph_paths),
+ "known_graph_count": int(len(self)),
+ "loads": int(self._load_count),
+ "memory_cache_hits": int(self._memory_cache_hits),
+ "disk_cache_hits": int(self._disk_cache_hits),
+ "tensorized": int(self._tensorized_count),
+ "memory_cache_size": int(len(self._memory_cache)),
+ "prefetch_submitted": int(self._prefetch_submitted),
+ "prefetch_completed": int(self._prefetch_completed),
+ "prefetch_pending": int(len(self._prefetch_futures)),
+ "loaded_from": clean_text(loaded_from),
+ "last_conversation_id": clean_text(conversation_id),
+ },
+ )
+
+ def register_conversation_ids(self, conversation_ids: Sequence[str], *, graph_dir: Path) -> None:
+ resolved_graph_dir = Path(graph_dir)
+ for conversation_id in conversation_ids:
+ normalized_id = clean_text(conversation_id)
+ if not normalized_id:
+ continue
+ if normalized_id not in self._graph_paths:
+ self._graph_paths[normalized_id] = resolved_graph_dir / f"{normalized_id}.json"
+ if self._conversation_ids and normalized_id in self._conversation_ids:
+ continue
+ if self._conversation_ids:
+ self._conversation_ids.append(normalized_id)
+ self._known_graph_count = max(int(self._known_graph_count), len(self))
+
+
+def index_graph_paths(
+ graph_dir: Path,
+ *,
+ progress_callback: Callable[[str, Dict[str, Any]], None] | None = None,
+) -> Dict[str, Path]:
+ graph_paths: Dict[str, Path] = {}
+ for index, path in enumerate(_iter_graph_paths(graph_dir), start=1):
+ conversation_id = clean_text(path.stem)
+ if not conversation_id:
+ raise RuntimeError(f"Unable to resolve conversation_id from graph filename: {path}")
+ if conversation_id in graph_paths:
+ raise RuntimeError(f"Duplicate conversation_id in graph directory {graph_dir}: {conversation_id}")
+ graph_paths[conversation_id] = path
+ if progress_callback is not None and (index == 1 or index % 10000 == 0):
+ progress_callback(
+ "graph_index_progress",
+ {
+ "graph_dir": str(graph_dir),
+ "graphs_indexed": int(index),
+ "last_conversation_id": conversation_id,
+ },
+ )
+ if progress_callback is not None:
+ progress_callback(
+ "graph_index_completed",
+ {
+ "graph_dir": str(graph_dir),
+ "graph_count": int(len(graph_paths)),
+ },
+ )
+ return graph_paths
+
+
+def load_lazy_graph_cache(
+ graph_dir: Path,
+ *,
+ device: torch.device,
+ conversation_ids: Sequence[str] | None = None,
+ cache_dir: Path | None = None,
+ memory_cache_size: int = 64,
+ prefetch_workers: int = 0,
+ prefetch_window: int = 0,
+ cache_write_enabled: bool = True,
+ require_cache_hit: bool = False,
+ progress_callback: Callable[[str, Dict[str, Any]], None] | None = None,
+) -> LazyGraphCache:
+ resolved_graph_dir = Path(graph_dir)
+ resolved_conversation_ids = [clean_text(item) for item in list(conversation_ids or []) if clean_text(item)]
+ if resolved_conversation_ids and len(resolved_conversation_ids) <= MAX_LAZY_GRAPH_PATHS_MATERIALIZE:
+ graph_paths = {
+ conversation_id: resolved_graph_dir / f"{conversation_id}.json"
+ for conversation_id in resolved_conversation_ids
+ }
+ materialized_conversation_ids: Sequence[str] | None = None
+ direct_graph_dir = None
+ known_graph_count = len(graph_paths)
+ elif resolved_conversation_ids:
+ graph_paths = {}
+ materialized_conversation_ids = resolved_conversation_ids
+ direct_graph_dir = resolved_graph_dir
+ known_graph_count = len({conversation_id for conversation_id in resolved_conversation_ids if conversation_id})
+ else:
+ graph_paths = index_graph_paths(resolved_graph_dir, progress_callback=progress_callback)
+ materialized_conversation_ids = None
+ direct_graph_dir = None
+ known_graph_count = len(graph_paths)
+ return LazyGraphCache(
+ graph_paths=graph_paths,
+ conversation_ids=materialized_conversation_ids,
+ graph_dir=direct_graph_dir,
+ known_graph_count=known_graph_count,
+ device=device,
+ cache_dir=cache_dir,
+ memory_cache_size=memory_cache_size,
+ prefetch_workers=prefetch_workers,
+ prefetch_window=prefetch_window,
+ cache_write_enabled=cache_write_enabled,
+ require_cache_hit=require_cache_hit,
+ progress_callback=progress_callback,
+ )
+
+
+def _build_graph_tensor_cache_entry(
+ graph_path_str: str,
+ cache_dir_str: str,
+ force_rebuild: bool = False,
+) -> Dict[str, Any]:
+ torch.set_num_threads(1)
+ graph_path = Path(graph_path_str)
+ cache_dir = Path(cache_dir_str)
+ source_signature = _graph_source_signature(graph_path)
+ cache_path = _graph_tensor_cache_path(cache_dir, graph_path)
+ try:
+ if not force_rebuild:
+ cached_payload = _load_graph_tensor_cache(cache_path, source_signature=source_signature)
+ if cached_payload is not None:
+ cached_graph = dict(cached_payload.get("graph", {}) or {})
+ return {
+ "status": "cache_hit",
+ "conversation_id": clean_text(cached_graph.get("conversation_id", graph_path.stem)),
+ "cache_path": str(cache_path),
+ "graph_path": str(graph_path),
+ }
+ graph = dict(read_json(graph_path))
+ cache_tensors = tensorize_graph(graph, device=torch.device("cpu"))
+ wrote = _write_graph_tensor_cache(
+ cache_path,
+ source_signature=source_signature,
+ graph=graph,
+ tensors=cache_tensors,
+ )
+ return {
+ "status": "built" if wrote else "write_skipped",
+ "conversation_id": clean_text(graph.get("conversation_id", graph_path.stem)),
+ "cache_path": str(cache_path),
+ "graph_path": str(graph_path),
+ }
+ except Exception as exc:
+ return {
+ "status": "error",
+ "conversation_id": clean_text(graph_path.stem),
+ "cache_path": str(cache_path),
+ "graph_path": str(graph_path),
+ "error_type": type(exc).__name__,
+ "error_message": clean_text(str(exc)),
+ }
+
+
+def build_graph_tensor_cache(
+ graph_dir: Path,
+ *,
+ cache_dir: Path,
+ conversation_ids: Sequence[str] | None = None,
+ worker_count: int = 1,
+ force_rebuild: bool = False,
+ error_log_path: Path | None = None,
+ progress_callback: Callable[[str, Dict[str, Any]], None] | None = None,
+) -> Dict[str, int]:
+ resolved_graph_dir = Path(graph_dir)
+ resolved_cache_dir = Path(cache_dir)
+ resolved_cache_dir.mkdir(parents=True, exist_ok=True)
+ if conversation_ids:
+ seen: set[str] = set()
+ graph_paths = []
+ for conversation_id in conversation_ids:
+ normalized_id = clean_text(conversation_id)
+ if not normalized_id or normalized_id in seen:
+ continue
+ seen.add(normalized_id)
+ graph_paths.append(resolved_graph_dir / f"{normalized_id}.json")
+ else:
+ graph_paths = list(_iter_graph_paths(resolved_graph_dir))
+ total_graphs = len(graph_paths)
+ summary = {
+ "graph_count": int(total_graphs),
+ "cache_hits": 0,
+ "built": 0,
+ "write_skipped": 0,
+ "errors": 0,
+ "workers": max(1, int(worker_count or 1)),
+ }
+ if progress_callback is not None:
+ progress_callback(
+ "graph_cache_build_started",
+ {
+ "graph_dir": str(resolved_graph_dir),
+ "cache_dir": str(resolved_cache_dir),
+ "graph_count": int(total_graphs),
+ "workers": int(summary["workers"]),
+ "force_rebuild": bool(force_rebuild),
+ },
+ )
+ resolved_error_log_path = Path(error_log_path) if error_log_path is not None else None
+ if resolved_error_log_path is not None:
+ resolved_error_log_path.parent.mkdir(parents=True, exist_ok=True)
+ resolved_error_log_path.unlink(missing_ok=True)
+
+ def _handle_result(result: Mapping[str, Any], completed_count: int) -> None:
+ status = clean_text(result.get("status", "")) or "unknown"
+ if status == "cache_hit":
+ summary["cache_hits"] += 1
+ elif status == "built":
+ summary["built"] += 1
+ elif status == "write_skipped":
+ summary["write_skipped"] += 1
+ else:
+ summary["errors"] += 1
+ if resolved_error_log_path is not None:
+ with resolved_error_log_path.open("a", encoding="utf-8", newline="") as handle:
+ handle.write(
+ json_dumps(
+ {
+ "timestamp": datetime.now().isoformat(timespec="seconds"),
+ "stage": "graph_cache_prebuild",
+ "status": status,
+ "conversation_id": clean_text(result.get("conversation_id", "")),
+ "graph_path": clean_text(result.get("graph_path", "")),
+ "cache_path": clean_text(result.get("cache_path", "")),
+ "error_type": clean_text(result.get("error_type", "")),
+ "error_message": clean_text(result.get("error_message", "")),
+ }
+ )
+ )
+ handle.write("\n")
+ if progress_callback is not None:
+ progress_callback(
+ "graph_cache_build_error",
+ {
+ "graph_dir": str(resolved_graph_dir),
+ "cache_dir": str(resolved_cache_dir),
+ "completed": int(completed_count),
+ "graph_count": int(total_graphs),
+ "conversation_id": clean_text(result.get("conversation_id", "")),
+ "graph_path": clean_text(result.get("graph_path", "")),
+ "error_type": clean_text(result.get("error_type", "")),
+ "error_message": clean_text(result.get("error_message", "")),
+ "errors": int(summary["errors"]),
+ },
+ )
+ if progress_callback is not None and (
+ completed_count == 1
+ or completed_count == total_graphs
+ or completed_count % 1000 == 0
+ ):
+ progress_callback(
+ "graph_cache_build_progress",
+ {
+ "graph_dir": str(resolved_graph_dir),
+ "cache_dir": str(resolved_cache_dir),
+ "completed": int(completed_count),
+ "graph_count": int(total_graphs),
+ "built": int(summary["built"]),
+ "cache_hits": int(summary["cache_hits"]),
+ "write_skipped": int(summary["write_skipped"]),
+ "errors": int(summary["errors"]),
+ "last_conversation_id": clean_text(result.get("conversation_id", "")),
+ "last_status": status,
+ },
+ )
+
+ resolved_workers = max(1, int(worker_count or 1))
+ if resolved_workers <= 1 or total_graphs <= 1:
+ for completed_count, graph_path in enumerate(graph_paths, start=1):
+ _handle_result(
+ _build_graph_tensor_cache_entry(
+ str(graph_path),
+ str(resolved_cache_dir),
+ force_rebuild=force_rebuild,
+ ),
+ completed_count,
+ )
+ else:
+ chunksize = max(1, min(128, total_graphs // max(1, resolved_workers * 8)))
+ with ProcessPoolExecutor(max_workers=resolved_workers) as executor:
+ iterator = executor.map(
+ _build_graph_tensor_cache_entry,
+ (str(path) for path in graph_paths),
+ itertools.repeat(str(resolved_cache_dir), total_graphs),
+ itertools.repeat(force_rebuild, total_graphs),
+ chunksize=chunksize,
+ )
+ for completed_count, result in enumerate(iterator, start=1):
+ _handle_result(result, completed_count)
+ if progress_callback is not None:
+ progress_callback(
+ "graph_cache_build_completed",
+ {
+ "graph_dir": str(resolved_graph_dir),
+ "cache_dir": str(resolved_cache_dir),
+ **{key: int(value) for key, value in summary.items()},
+ },
+ )
+ return {key: int(value) for key, value in summary.items()}
+
+
+def _combine_graph_tensors_by_conversation(
+ graph_tensors_by_conversation: Mapping[str, Mapping[str, Any]],
+ *,
+ device: torch.device,
+) -> Dict[str, Any]:
+ ordered_items = [
+ (clean_text(conversation_id), dict(graph_tensors))
+ for conversation_id, graph_tensors in graph_tensors_by_conversation.items()
+ if clean_text(conversation_id)
+ ]
+ if not ordered_items:
+ raise ValueError("graph_tensors_by_conversation must not be empty")
+ if len(ordered_items) == 1:
+ conversation_id, graph_tensors = ordered_items[0]
+ merged = dict(graph_tensors)
+ merged = dict(_ensure_graph_scoring_feature_cache(merged))
+ merged["event_node_ids_by_conversation"] = {
+ conversation_id: [event_id for event_id in list(graph_tensors.get("event_node_ids", []) or []) if clean_text(event_id)]
+ }
+ merged["temporal_paths_by_conversation"] = {
+ conversation_id: [dict(path) for path in list(graph_tensors.get("temporal_paths", []) or [])]
+ }
+ return merged
+
+ merged_node_ids: List[str] = []
+ merged_node_texts: List[str] = []
+ merged_node_text_hash_indices: List[List[int]] = []
+ merged_node_lookup: Dict[str, Dict[str, Any]] = {}
+ merged_event_node_ids: List[str] = []
+ merged_time_node_ids: List[str] = []
+ merged_paths: List[Dict[str, Any]] = []
+ merged_temporal_paths: List[Dict[str, Any]] = []
+ merged_paths_by_event_id: Dict[str, List[Dict[str, Any]]] = {}
+ merged_event_support_lookup: Dict[str, Dict[str, List[str]]] = {}
+ merged_support_node_ids_by_event: Dict[str, List[str]] = {}
+ merged_support_node_ids_by_event_and_type: Dict[str, Dict[str, List[str]]] = {}
+ merged_event_scoring_features_by_id: Dict[str, Dict[str, Any]] = {}
+ merged_node_scoring_features_by_id: Dict[str, Dict[str, Any]] = {}
+ merged_path_scoring_features_by_id: Dict[str, Dict[str, Any]] = {}
+ event_node_ids_by_conversation: Dict[str, List[str]] = {}
+ temporal_paths_by_conversation: Dict[str, List[Dict[str, Any]]] = {}
+ node_type_tensors: List[Tensor] = []
+ node_scalar_feature_tensors: List[Tensor] = []
+ edge_src_tensors: List[Tensor] = []
+ edge_dst_tensors: List[Tensor] = []
+ edge_type_tensors: List[Tensor] = []
+ node_offset = 0
+ seen_node_ids: set[str] = set()
+ non_blocking = device.type == "cuda"
+
+ for conversation_id, graph_tensors in ordered_items:
+ graph_tensors = dict(_ensure_graph_scoring_feature_cache(graph_tensors))
+ local_node_type_ids = graph_tensors.get("node_type_ids")
+ local_node_scalar_features = graph_tensors.get("node_scalar_features")
+ local_edge_src = graph_tensors.get("edge_src")
+ local_edge_dst = graph_tensors.get("edge_dst")
+ local_edge_type_ids = graph_tensors.get("edge_type_ids")
+ if not isinstance(local_node_type_ids, Tensor) or not isinstance(local_node_scalar_features, Tensor):
+ raise TypeError("graph_tensors missing node tensors required for merge")
+ if not isinstance(local_edge_src, Tensor) or not isinstance(local_edge_dst, Tensor) or not isinstance(local_edge_type_ids, Tensor):
+ raise TypeError("graph_tensors missing edge tensors required for merge")
+ local_node_texts = [clean_text(text) for text in list(graph_tensors.get("node_texts", []) or [])]
+ local_node_text_hash_indices = [list(indices or []) for indices in list(graph_tensors.get("node_text_hash_indices", []) or [])]
+ local_node_ids_raw = [clean_text(node_id) for node_id in list(graph_tensors.get("node_ids", []) or [])]
+ local_node_count = int(local_node_type_ids.size(0))
+ if len(local_node_ids_raw) != local_node_count:
+ raise GraphTensorValidationError(
+ f"_combine_graph_tensors_by_conversation: node_ids length mismatch for conversation '{conversation_id}'"
+ f" (node_ids={len(local_node_ids_raw)}, node_type_ids={local_node_count})"
+ )
+ if len(local_node_texts) != local_node_count or len(local_node_text_hash_indices) != local_node_count:
+ raise GraphTensorValidationError(
+ f"_combine_graph_tensors_by_conversation: node text alignment mismatch for conversation '{conversation_id}'"
+ f" (node_texts={len(local_node_texts)}, node_text_hash_indices={len(local_node_text_hash_indices)}, node_type_ids={local_node_count})"
+ )
+ local_node_ids: List[str] = []
+ local_seen_node_ids: set[str] = set()
+ for local_index, raw_node_id in enumerate(local_node_ids_raw):
+ normalized_node_id = raw_node_id or f"__missing_node__:{conversation_id}:{local_index}"
+ if normalized_node_id in local_seen_node_ids:
+ raise GraphTensorValidationError(
+ f"_combine_graph_tensors_by_conversation: duplicate node_id within conversation '{conversation_id}': {normalized_node_id}"
+ )
+ local_seen_node_ids.add(normalized_node_id)
+ local_node_ids.append(normalized_node_id)
+ duplicate_node_ids = [node_id for node_id in local_node_ids if node_id in seen_node_ids]
+ if duplicate_node_ids:
+ raise RuntimeError(f"Duplicate node_id across merged graph tensors: {duplicate_node_ids[0]}")
+ seen_node_ids.update(local_node_ids)
+ merged_node_ids.extend(local_node_ids)
+ merged_node_texts.extend(local_node_texts)
+ merged_node_text_hash_indices.extend(local_node_text_hash_indices)
+ for node_id, node in dict(graph_tensors.get("node_by_id", {}) or {}).items():
+ normalized_node_id = clean_text(node_id)
+ if normalized_node_id:
+ merged_node_lookup[normalized_node_id] = dict(node)
+ local_event_node_ids = [
+ clean_text(event_id) for event_id in list(graph_tensors.get("event_node_ids", []) or []) if clean_text(event_id)
+ ]
+ merged_event_node_ids.extend(local_event_node_ids)
+ event_node_ids_by_conversation[conversation_id] = list(local_event_node_ids)
+ merged_time_node_ids.extend(
+ [clean_text(node_id) for node_id in list(graph_tensors.get("time_node_ids", []) or []) if clean_text(node_id)]
+ )
+ local_paths = [dict(path) for path in list(graph_tensors.get("paths", []) or [])]
+ merged_paths.extend(local_paths)
+ for event_id, paths in dict(graph_tensors.get("paths_by_event_id", {}) or {}).items():
+ normalized_event_id = clean_text(event_id)
+ if not normalized_event_id:
+ continue
+ merged_paths_by_event_id.setdefault(normalized_event_id, []).extend(dict(path) for path in list(paths or []))
+ local_temporal_paths = [dict(path) for path in list(graph_tensors.get("temporal_paths", []) or [])]
+ merged_temporal_paths.extend(local_temporal_paths)
+ temporal_paths_by_conversation[conversation_id] = list(local_temporal_paths)
+ for event_id, payload in dict(graph_tensors.get("event_support_lookup", {}) or {}).items():
+ normalized_event_id = clean_text(event_id)
+ if not normalized_event_id:
+ continue
+ merged_event_support_lookup[normalized_event_id] = {
+ clean_text(key): [clean_text(item) for item in list(values or []) if clean_text(item)]
+ for key, values in dict(payload or {}).items()
+ if clean_text(key)
+ }
+ for event_id, support_node_ids in dict(graph_tensors.get("support_node_ids_by_event", {}) or {}).items():
+ normalized_event_id = clean_text(event_id)
+ if normalized_event_id:
+ merged_support_node_ids_by_event[normalized_event_id] = [
+ clean_text(node_id) for node_id in list(support_node_ids or []) if clean_text(node_id)
+ ]
+ for event_id, path_type_map in dict(graph_tensors.get("support_node_ids_by_event_and_type", {}) or {}).items():
+ normalized_event_id = clean_text(event_id)
+ if not normalized_event_id:
+ continue
+ merged_support_node_ids_by_event_and_type[normalized_event_id] = {
+ clean_text(path_type): [clean_text(node_id) for node_id in list(node_ids or []) if clean_text(node_id)]
+ for path_type, node_ids in dict(path_type_map or {}).items()
+ if clean_text(path_type)
+ }
+ for event_id, payload in dict(graph_tensors.get("event_scoring_features_by_id", {}) or {}).items():
+ normalized_event_id = clean_text(event_id)
+ if normalized_event_id:
+ merged_event_scoring_features_by_id[normalized_event_id] = dict(payload or {})
+ for node_id, payload in dict(graph_tensors.get("node_scoring_features_by_id", {}) or {}).items():
+ normalized_node_id = clean_text(node_id)
+ if normalized_node_id:
+ merged_node_scoring_features_by_id[normalized_node_id] = dict(payload or {})
+ for path_id, payload in dict(graph_tensors.get("path_scoring_features_by_id", {}) or {}).items():
+ normalized_path_id = clean_text(path_id)
+ if normalized_path_id:
+ merged_path_scoring_features_by_id[normalized_path_id] = dict(payload or {})
+ node_type_tensors.append(_tensor_to_device(local_node_type_ids, device, non_blocking=non_blocking))
+ node_scalar_feature_tensors.append(_tensor_to_device(local_node_scalar_features, device, non_blocking=non_blocking))
+ edge_src_tensors.append(_tensor_to_device(local_edge_src, device, non_blocking=non_blocking) + node_offset)
+ edge_dst_tensors.append(_tensor_to_device(local_edge_dst, device, non_blocking=non_blocking) + node_offset)
+ edge_type_tensors.append(_tensor_to_device(local_edge_type_ids, device, non_blocking=non_blocking))
+ node_offset += local_node_count
+
+ merged_node_id_to_index = {node_id: index for index, node_id in enumerate(merged_node_ids)}
+ return {
+ "node_ids": merged_node_ids,
+ "node_texts": merged_node_texts,
+ "node_text_hash_indices": merged_node_text_hash_indices,
+ "node_by_id": merged_node_lookup,
+ "event_node_ids": merged_event_node_ids,
+ "time_node_ids": merged_time_node_ids,
+ "node_type_ids": torch.cat(node_type_tensors, dim=0) if node_type_tensors else torch.zeros((0,), dtype=torch.long, device=device),
+ "node_scalar_features": (
+ torch.cat(node_scalar_feature_tensors, dim=0)
+ if node_scalar_feature_tensors
+ else torch.zeros((0, NODE_SCALAR_DIM), dtype=torch.float32, device=device)
+ ),
+ "edge_src": torch.cat(edge_src_tensors, dim=0) if edge_src_tensors else torch.zeros((0,), dtype=torch.long, device=device),
+ "edge_dst": torch.cat(edge_dst_tensors, dim=0) if edge_dst_tensors else torch.zeros((0,), dtype=torch.long, device=device),
+ "edge_type_ids": (
+ torch.cat(edge_type_tensors, dim=0) if edge_type_tensors else torch.zeros((0,), dtype=torch.long, device=device)
+ ),
+ "node_id_to_index": merged_node_id_to_index,
+ "paths": merged_paths,
+ "paths_by_event_id": merged_paths_by_event_id,
+ "temporal_paths": merged_temporal_paths,
+ "event_support_lookup": merged_event_support_lookup,
+ "support_node_ids_by_event": merged_support_node_ids_by_event,
+ "support_node_ids_by_event_and_type": merged_support_node_ids_by_event_and_type,
+ "event_scoring_features_by_id": merged_event_scoring_features_by_id,
+ "node_scoring_features_by_id": merged_node_scoring_features_by_id,
+ "path_scoring_features_by_id": merged_path_scoring_features_by_id,
+ "event_node_ids_by_conversation": event_node_ids_by_conversation,
+ "temporal_paths_by_conversation": temporal_paths_by_conversation,
+ }
+
+
+def _hashed_text_embedding(texts: Sequence[str], embedding: nn.Embedding, buckets: int) -> Tensor:
+ return _hashed_index_embedding_batch(
+ [_text_hash_indices(text, buckets) for text in texts],
+ embedding,
+ )
+
+
+def _question_feature_indices(features: Mapping[str, Any]) -> tuple[int, int]:
+ feature_key = "|".join(
+ [
+ clean_text(features.get("semantic_slot_target", "")),
+ clean_text(features.get("target_status_target", "")),
+ clean_text(features.get("time_granularity_target", "")),
+ ]
+ )
+ feature_index = stable_hash_bucket(feature_key or "__feature__", 256)
+ query_type_index = 0
+ if bool(features.get("is_temporal", False)):
+ query_type_index = 1
+ elif clean_text(features.get("semantic_slot_target", "")) in {"identity", "research_topic", "education", "occupation", "profile"}:
+ query_type_index = 2
+ elif bool(list(features.get("speaker_candidates", []) or [])):
+ query_type_index = 3
+ return feature_index, query_type_index
+
+
+def _question_understanding_targets(features: Mapping[str, Any]) -> Dict[str, Any]:
+ semantic_target = clean_text(features.get("semantic_slot_target", ""))
+ status_target = clean_text(features.get("target_status_target", ""))
+ time_target = clean_text(features.get("time_granularity_target", ""))
+ is_temporal = bool(features.get("is_temporal", False))
+ return {
+ "semantic_slot_index": SEMANTIC_SLOT_TO_ID.get(semantic_target, SEMANTIC_SLOT_TO_ID["event"]),
+ "target_status_index": TARGET_STATUS_TO_ID.get(status_target, TARGET_STATUS_TO_ID[""]),
+ "time_granularity_index": TIME_GRANULARITY_TO_ID.get(time_target, TIME_GRANULARITY_TO_ID[""]),
+ "is_temporal_value": 1.0 if is_temporal else 0.0,
+ }
+
+
+def _confident_label_from_logits(
+ logits: Tensor,
+ *,
+ labels: Sequence[str],
+ fallback: str,
+ min_margin: float = 0.08,
+) -> str:
+ if not isinstance(logits, Tensor) or logits.numel() <= 0:
+ return clean_text(fallback)
+ probs = torch.softmax(logits.detach().float(), dim=-1)
+ top_k = min(2, int(probs.numel()))
+ top_values, top_indices = torch.topk(probs, k=top_k, dim=0)
+ top_margin = float(top_values[0] - top_values[1]) if top_values.numel() > 1 else float(top_values[0])
+ if top_margin < float(min_margin):
+ return clean_text(fallback)
+ top_index = int(top_indices[0].item())
+ if 0 <= top_index < len(labels):
+ return clean_text(labels[top_index])
+ return clean_text(fallback)
+
+
+def _confident_temporal_from_logit(logit: Tensor, *, fallback: bool, min_margin: float = 0.12) -> bool:
+ if not isinstance(logit, Tensor) or logit.numel() <= 0:
+ return bool(fallback)
+ probability = float(torch.sigmoid(logit.detach().float()).item())
+ if abs(probability - 0.5) < float(min_margin):
+ return bool(fallback)
+ return probability >= 0.5
+
+
+def _merge_learned_question_features_from_intent_logits(
+ base_features: Mapping[str, Any],
+ *,
+ semantic_logits: Tensor,
+ status_logits: Tensor,
+ time_granularity_logits: Tensor,
+ temporal_logit: Tensor,
+) -> Dict[str, Any]:
+ merged = dict(base_features or {})
+ merged["semantic_slot_target"] = _confident_label_from_logits(
+ semantic_logits,
+ labels=SEMANTIC_SLOTS,
+ fallback=clean_text(merged.get("semantic_slot_target", "event")) or "event",
+ )
+ merged["target_status_target"] = _confident_label_from_logits(
+ status_logits,
+ labels=TARGET_STATUSES,
+ fallback=clean_text(merged.get("target_status_target", "")),
+ )
+ merged["time_granularity_target"] = _confident_label_from_logits(
+ time_granularity_logits,
+ labels=TIME_GRANULARITIES,
+ fallback=clean_text(merged.get("time_granularity_target", "")),
+ )
+ merged["is_temporal"] = _confident_temporal_from_logit(
+ temporal_logit,
+ fallback=bool(merged.get("is_temporal", False)),
+ )
+ if clean_text(merged.get("semantic_slot_target", "")) == "event_time":
+ merged["is_temporal"] = True
+ if clean_text(merged.get("time_granularity_target", "")) in {"day", "month", "year", "relative_day_reference", "day_or_coarse"}:
+ merged["is_temporal"] = True
+ return merged
+
+
+class _LightweightSelfAttentionBlock(nn.Module):
+ def __init__(
+ self,
+ *,
+ embed_dim: int,
+ num_heads: int,
+ dropout: float,
+ relation_feature_dim: int = 0,
+ ) -> None:
+ super().__init__()
+ self.num_heads = int(num_heads)
+ self.attn_norm = nn.LayerNorm(embed_dim)
+ self.attn = nn.MultiheadAttention(
+ embed_dim=embed_dim,
+ num_heads=num_heads,
+ dropout=dropout,
+ batch_first=True,
+ )
+ self.relation_bias_projection = (
+ nn.Linear(int(relation_feature_dim), self.num_heads, bias=False)
+ if int(relation_feature_dim) > 0
+ else None
+ )
+ self.attn_dropout = nn.Dropout(dropout)
+ self.ffn_norm = nn.LayerNorm(embed_dim)
+ self.ffn = nn.Sequential(
+ nn.Linear(embed_dim, embed_dim * 4),
+ nn.SiLU(),
+ nn.Dropout(dropout),
+ nn.Linear(embed_dim * 4, embed_dim),
+ nn.Dropout(dropout),
+ )
+
+ def forward(self, hidden: Tensor, token_mask: Tensor, *, relation_features: Tensor | None = None) -> Tensor:
+ key_padding_mask = ~token_mask
+ attn_mask = None
+ if relation_features is not None and self.relation_bias_projection is not None:
+ batch_size, token_count, _ = hidden.shape
+ relation_bias = self.relation_bias_projection(relation_features).permute(0, 3, 1, 2)
+ relation_bias = relation_bias.masked_fill((~token_mask).unsqueeze(1).unsqueeze(1), -1e4)
+ attn_mask = relation_bias.reshape(batch_size * self.num_heads, token_count, token_count)
+ key_padding_mask = None
+ attn_input = self.attn_norm(hidden)
+ attn_output, _ = self.attn(
+ attn_input,
+ attn_input,
+ attn_input,
+ key_padding_mask=key_padding_mask,
+ attn_mask=attn_mask,
+ need_weights=False,
+ )
+ hidden = hidden + self.attn_dropout(attn_output)
+ hidden = hidden * token_mask.unsqueeze(-1).to(dtype=hidden.dtype)
+ hidden = hidden + self.ffn(self.ffn_norm(hidden))
+ hidden = hidden * token_mask.unsqueeze(-1).to(dtype=hidden.dtype)
+ return hidden
+
+
+class HashedSelfAttentionTextEncoder(nn.Module):
+ def __init__(
+ self,
+ *,
+ num_embeddings: int,
+ embed_dim: int,
+ max_tokens: int,
+ attention_layers: int = ENCODER_ATTENTION_LAYERS,
+ attention_heads: int = ENCODER_ATTENTION_HEADS,
+ dropout: float = MESSAGE_DROPOUT,
+ ) -> None:
+ super().__init__()
+ self.max_tokens = max(1, int(max_tokens))
+ self.embed_dim = int(embed_dim)
+ self.token_embedding = nn.Embedding(num_embeddings, embed_dim)
+ self.position_embedding = nn.Embedding(self.max_tokens, embed_dim)
+ self.role_embedding = nn.Embedding(TMCRA_TOKEN_ROLE_COUNT, embed_dim)
+ self.role_predictor = nn.Linear(embed_dim, TMCRA_TOKEN_ROLE_COUNT)
+ self.input_dropout = nn.Dropout(dropout)
+ self.blocks = nn.ModuleList(
+ [
+ _LightweightSelfAttentionBlock(
+ embed_dim=embed_dim,
+ num_heads=attention_heads,
+ dropout=dropout,
+ relation_feature_dim=TMCRA_TOKEN_RELATION_FEATURE_DIM,
+ )
+ for _ in range(max(1, int(attention_layers)))
+ ]
+ )
+ for block in self.blocks:
+ if block.relation_bias_projection is not None:
+ nn.init.zeros_(block.relation_bias_projection.weight)
+ nn.init.zeros_(self.role_embedding.weight)
+ self.pool_norm = nn.LayerNorm(embed_dim)
+ self.pool_projection = nn.Linear(embed_dim, 1)
+ self._last_role_aux_loss: Tensor | None = None
+ self._last_role_predictions: Dict[str, Tensor] | None = None
+
+ def forward_indices(
+ self,
+ index_batches: Sequence[Sequence[int]],
+ *,
+ token_role_batches: Sequence[Sequence[int]] | None = None,
+ token_identity_batches: Sequence[Sequence[int]] | None = None,
+ ) -> Tensor:
+ device = self.token_embedding.weight.device
+ self._last_role_aux_loss = None
+ self._last_role_predictions = None
+ if not index_batches:
+ return torch.zeros((0, self.embed_dim), dtype=self.token_embedding.weight.dtype, device=device)
+ token_embeddings, token_mask = _hashed_token_embedding_batch(
+ index_batches,
+ self.token_embedding,
+ max_tokens=self.max_tokens,
+ )
+ sequence_length = int(token_embeddings.size(1))
+ position_ids = torch.arange(sequence_length, dtype=torch.long, device=device)
+ if token_identity_batches is None:
+ token_identity_batches = [
+ [int(item) + 1 for item in list(index_values or [])] or [0]
+ for index_values in index_batches
+ ]
+ padded_token_ids, token_mask = _padded_long_batch(
+ token_identity_batches,
+ device=device,
+ fill_value=0,
+ max_tokens=self.max_tokens,
+ )
+ base_hidden = token_embeddings + self.position_embedding(position_ids).unsqueeze(0)
+ role_logits = self.role_predictor(base_hidden)
+ role_probs = torch.softmax(role_logits, dim=-1)
+ role_probs = role_probs * token_mask.unsqueeze(-1).to(dtype=role_probs.dtype)
+ if token_role_batches is not None and self.training:
+ padded_role_targets, _ = _padded_long_batch(
+ token_role_batches,
+ device=device,
+ fill_value=TMCRA_TOKEN_ROLE_EMPTY,
+ max_tokens=self.max_tokens,
+ )
+ valid_targets = token_mask.reshape(-1)
+ if bool(torch.any(valid_targets)):
+ self._last_role_aux_loss = F.cross_entropy(
+ role_logits.reshape(-1, TMCRA_TOKEN_ROLE_COUNT)[valid_targets],
+ padded_role_targets.reshape(-1)[valid_targets],
+ )
+ predicted_role_ids = torch.argmax(role_probs, dim=-1)
+ predicted_role_ids = predicted_role_ids.masked_fill(~token_mask, TMCRA_TOKEN_ROLE_EMPTY)
+ self._last_role_predictions = {
+ "role_probs": role_probs.detach(),
+ "predicted_role_ids": predicted_role_ids.detach(),
+ "token_mask": token_mask.detach(),
+ }
+ relation_features = _tmcra_text_relation_features_from_tensors(
+ padded_token_ids,
+ predicted_role_ids,
+ token_mask=token_mask,
+ dtype=token_embeddings.dtype,
+ )
+ role_context = torch.matmul(role_probs, self.role_embedding.weight)
+ hidden = base_hidden + role_context
+ hidden = self.input_dropout(hidden)
+ for block in self.blocks:
+ hidden = block(hidden, token_mask, relation_features=relation_features)
+ return self._attention_pool(hidden, token_mask)
+
+ def pop_last_role_aux_loss(self) -> Tensor | None:
+ value = self._last_role_aux_loss
+ self._last_role_aux_loss = None
+ return value
+
+ def pop_last_role_predictions(self) -> Dict[str, Tensor] | None:
+ value = self._last_role_predictions
+ self._last_role_predictions = None
+ return value
+
+ def _attention_pool(self, hidden: Tensor, token_mask: Tensor) -> Tensor:
+ pool_logits = self.pool_projection(self.pool_norm(hidden)).squeeze(-1)
+ pool_logits = pool_logits.masked_fill(~token_mask, float("-inf"))
+ pool_weights = torch.softmax(pool_logits, dim=-1)
+ pool_weights = pool_weights.masked_fill(~token_mask, 0.0)
+ pool_weights = pool_weights / pool_weights.sum(dim=-1, keepdim=True).clamp_min(1e-6)
+ return (hidden * pool_weights.unsqueeze(-1)).sum(dim=1)
+
+
+class QuestionEncoder(nn.Module):
+ def __init__(self) -> None:
+ super().__init__()
+ self.text_encoder = HashedSelfAttentionTextEncoder(
+ num_embeddings=QUESTION_HASH_BUCKETS,
+ embed_dim=QUESTION_EMBED_DIM,
+ max_tokens=QUESTION_MAX_TOKENS,
+ attention_layers=ENCODER_ATTENTION_LAYERS,
+ attention_heads=ENCODER_ATTENTION_HEADS,
+ dropout=MESSAGE_DROPOUT,
+ )
+ self.token_embedding = self.text_encoder.token_embedding
+ self.feature_embedding = nn.Embedding(256, QUESTION_CATEGORICAL_EMBED_DIM)
+ self.query_type_embedding = nn.Embedding(8, QUESTION_CATEGORICAL_EMBED_DIM)
+ self._last_text_embeddings: Tensor | None = None
+
+ def _feature_indices(self, features: Mapping[str, Any]) -> tuple[int, int]:
+ return _question_feature_indices(features)
+
+ def pop_last_role_aux_loss(self) -> Tensor | None:
+ return self.text_encoder.pop_last_role_aux_loss()
+
+ def pop_last_role_predictions(self) -> Dict[str, Tensor] | None:
+ return self.text_encoder.pop_last_role_predictions()
+
+ def pop_last_text_embeddings(self) -> Tensor | None:
+ value = self._last_text_embeddings
+ self._last_text_embeddings = None
+ return value
+
+ def forward_batch(
+ self,
+ questions: Sequence[str],
+ features_batch: Sequence[Mapping[str, Any]],
+ *,
+ token_hash_indices_batch: Sequence[Sequence[int]] | None = None,
+ feature_indices: Sequence[int] | None = None,
+ query_type_indices: Sequence[int] | None = None,
+ ) -> Tensor:
+ device = self.token_embedding.weight.device
+ if not questions:
+ self._last_text_embeddings = None
+ return torch.zeros((0, QUESTION_OUTPUT_DIM), dtype=self.token_embedding.weight.dtype, device=device)
+ resolved_token_hash_indices = list(token_hash_indices_batch or [_text_hash_indices(question, QUESTION_HASH_BUCKETS) for question in questions])
+ token_role_batches = None
+ question_token_payloads = [
+ (_resolved_text_tokens(question), [])
+ for question in questions
+ ]
+ if self.training:
+ question_token_payloads = [
+ _question_token_roles(question, features_batch[index] if index < len(features_batch) else {})
+ for index, question in enumerate(questions)
+ ]
+ token_role_batches = [
+ _align_sequence_length(
+ roles,
+ target_length=len(resolved_token_hash_indices[index]),
+ fill_value=TMCRA_TOKEN_ROLE_CONTENT,
+ ) or [TMCRA_TOKEN_ROLE_EMPTY]
+ for index, (_, roles) in enumerate(question_token_payloads)
+ ]
+ token_identity_batches = [
+ [
+ stable_token_identity(f"qtok:{token}") + 1
+ for token in _align_sequence_length(
+ tokens,
+ target_length=len(resolved_token_hash_indices[index]),
+ fill_value="__empty__",
+ )
+ ] or [0]
+ for index, (tokens, _) in enumerate(question_token_payloads)
+ ]
+ text_embeddings = self.text_encoder.forward_indices(
+ resolved_token_hash_indices,
+ token_role_batches=token_role_batches,
+ token_identity_batches=token_identity_batches,
+ )
+ self._last_text_embeddings = text_embeddings
+ resolved_feature_indices = list(feature_indices or [])
+ resolved_query_type_indices = list(query_type_indices or [])
+ if len(resolved_feature_indices) != len(questions) or len(resolved_query_type_indices) != len(questions):
+ resolved_feature_indices = []
+ resolved_query_type_indices = []
+ for features in features_batch:
+ feature_index, query_type_index = self._feature_indices(features)
+ resolved_feature_indices.append(feature_index)
+ resolved_query_type_indices.append(query_type_index)
+ _validate_index_values(
+ resolved_feature_indices,
+ upper_bound=int(self.feature_embedding.num_embeddings),
+ index_name="question_feature_index",
+ context="QuestionEncoder.forward_batch",
+ extra={"question_count": len(questions)},
+ )
+ _validate_index_values(
+ resolved_query_type_indices,
+ upper_bound=int(self.query_type_embedding.num_embeddings),
+ index_name="question_query_type_index",
+ context="QuestionEncoder.forward_batch",
+ extra={"question_count": len(questions)},
+ )
+ feature_embedding = self.feature_embedding(torch.tensor(resolved_feature_indices, dtype=torch.long, device=device))
+ query_type_embedding = self.query_type_embedding(torch.tensor(resolved_query_type_indices, dtype=torch.long, device=device))
+ return torch.cat([text_embeddings, feature_embedding, query_type_embedding], dim=-1)
+
+ def forward(self, question: str, features: Mapping[str, Any]) -> Tensor:
+ return self.forward_batch([question], [features])[0]
+
+
+class QuestionIntentHead(nn.Module):
+ def __init__(self) -> None:
+ super().__init__()
+ self.shared = nn.Sequential(
+ nn.LayerNorm(QUESTION_EMBED_DIM),
+ nn.Linear(QUESTION_EMBED_DIM, MESSAGE_HIDDEN_DIM),
+ nn.SiLU(),
+ )
+ self.semantic_head = nn.Linear(MESSAGE_HIDDEN_DIM, len(SEMANTIC_SLOTS))
+ self.status_head = nn.Linear(MESSAGE_HIDDEN_DIM, len(TARGET_STATUSES))
+ self.time_granularity_head = nn.Linear(MESSAGE_HIDDEN_DIM, len(TIME_GRANULARITIES))
+ self.temporal_head = nn.Linear(MESSAGE_HIDDEN_DIM, 1)
+ for layer in (self.semantic_head, self.status_head, self.time_granularity_head, self.temporal_head):
+ nn.init.zeros_(layer.weight)
+ nn.init.zeros_(layer.bias)
+
+ def forward(self, question_text_embeddings: Tensor) -> Dict[str, Tensor]:
+ hidden = self.shared(question_text_embeddings)
+ return {
+ "semantic_logits": self.semantic_head(hidden),
+ "status_logits": self.status_head(hidden),
+ "time_granularity_logits": self.time_granularity_head(hidden),
+ "temporal_logits": self.temporal_head(hidden).squeeze(-1),
+ }
+
+
+class MemoryRouterHead(nn.Module):
+ """Predict which memory layers should be opened before candidate ranking."""
+
+ def __init__(self) -> None:
+ super().__init__()
+ self.net = nn.Sequential(
+ nn.LayerNorm(QUESTION_OUTPUT_DIM),
+ nn.Linear(QUESTION_OUTPUT_DIM, MESSAGE_HIDDEN_DIM),
+ nn.SiLU(),
+ nn.Dropout(MESSAGE_DROPOUT),
+ nn.Linear(MESSAGE_HIDDEN_DIM, len(MEMORY_ROUTER_LAYERS)),
+ )
+ output = self.net[-1]
+ if isinstance(output, nn.Linear):
+ nn.init.zeros_(output.weight)
+ nn.init.zeros_(output.bias)
+
+ def forward(self, question_embeddings: Tensor) -> Tensor:
+ return self.net(question_embeddings)
+
+
+class NodeEncoder(nn.Module):
+ def __init__(self) -> None:
+ super().__init__()
+ self.text_encoder = HashedSelfAttentionTextEncoder(
+ num_embeddings=NODE_HASH_BUCKETS,
+ embed_dim=NODE_EMBED_DIM,
+ max_tokens=NODE_MAX_TOKENS,
+ attention_layers=ENCODER_ATTENTION_LAYERS,
+ attention_heads=ENCODER_ATTENTION_HEADS,
+ dropout=MESSAGE_DROPOUT,
+ )
+ self.text_embedding = self.text_encoder.token_embedding
+ self.node_type_embedding = nn.Embedding(len(NODE_TYPE_TO_ID), NODE_TYPE_EMBED_DIM)
+ self.scalar_mlp = nn.Sequential(
+ nn.Linear(NODE_SCALAR_DIM, 64),
+ nn.SiLU(),
+ nn.Linear(64, NODE_SCALAR_HIDDEN_DIM),
+ )
+
+ def pop_last_role_aux_loss(self) -> Tensor | None:
+ return self.text_encoder.pop_last_role_aux_loss()
+
+ def forward(
+ self,
+ node_texts: Sequence[str],
+ node_type_ids: Tensor,
+ scalar_features: Tensor,
+ *,
+ token_hash_indices_batch: Sequence[Sequence[int]] | None = None,
+ ) -> Tensor:
+ resolved_token_hash_indices = list(token_hash_indices_batch or [_text_hash_indices(text, NODE_HASH_BUCKETS) for text in node_texts])
+ node_type_values = node_type_ids.detach().cpu().tolist()
+ token_role_batches = None
+ node_token_payloads = [
+ (_resolved_text_tokens(node_texts[index] if index < len(node_texts) else ""), [])
+ for index in range(len(node_texts))
+ ]
+ if self.training:
+ node_token_payloads = [
+ _node_token_roles(
+ node_texts[index] if index < len(node_texts) else "",
+ int(node_type_values[index]) if index < len(node_type_values) else 0,
+ )
+ for index in range(len(node_texts))
+ ]
+ token_role_batches = [
+ _align_sequence_length(
+ roles,
+ target_length=len(resolved_token_hash_indices[index]),
+ fill_value=TMCRA_TOKEN_ROLE_CONTENT,
+ ) or [TMCRA_TOKEN_ROLE_EMPTY]
+ for index, (_, roles) in enumerate(node_token_payloads)
+ ]
+ token_identity_batches = [
+ [
+ stable_token_identity(f"ntok:{token}") + 1
+ for token in _align_sequence_length(
+ tokens,
+ target_length=len(resolved_token_hash_indices[index]),
+ fill_value="__empty__",
+ )
+ ] or [0]
+ for index, (tokens, _) in enumerate(node_token_payloads)
+ ]
+ text_embedding = self.text_encoder.forward_indices(
+ resolved_token_hash_indices,
+ token_role_batches=token_role_batches,
+ token_identity_batches=token_identity_batches,
+ )
+ _validate_index_values(
+ node_type_ids.detach().cpu().tolist(),
+ upper_bound=int(self.node_type_embedding.num_embeddings),
+ index_name="node_type_id",
+ context="NodeEncoder.forward",
+ extra={"node_count": int(node_type_ids.numel())},
+ )
+ type_embedding = self.node_type_embedding(node_type_ids)
+ scalar_embedding = self.scalar_mlp(scalar_features)
+ return torch.cat([text_embedding, type_embedding, scalar_embedding], dim=-1)
+
+
+class TypedMessagePassing(nn.Module):
+ def __init__(self, *, hidden_dim: int = MESSAGE_HIDDEN_DIM, dropout: float = MESSAGE_DROPOUT) -> None:
+ super().__init__()
+ self.edge_embedding = nn.Embedding(len(EDGE_TYPE_TO_ID), EDGE_TYPE_EMBED_DIM)
+ self.reverse_support_edge_type_ids = tuple(
+ EDGE_TYPE_TO_ID[edge_type]
+ for edge_type in (
+ "time_of",
+ "profile_of",
+ "status_of",
+ "supported_by_turn",
+ )
+ if edge_type in EDGE_TYPE_TO_ID
+ )
+ self.input_projection = nn.Linear(NODE_OUTPUT_DIM, hidden_dim)
+ self.message_layers = nn.ModuleList(
+ [nn.Linear(hidden_dim + EDGE_TYPE_EMBED_DIM, hidden_dim) for _ in range(2)]
+ )
+ self.update_layers = nn.ModuleList(
+ [nn.Linear(hidden_dim * 2, hidden_dim) for _ in range(2)]
+ )
+ self.norm_layers = nn.ModuleList([nn.LayerNorm(hidden_dim) for _ in range(2)])
+ self.dropout = nn.Dropout(dropout)
+
+ def forward(self, node_inputs: Tensor, edge_src: Tensor, edge_dst: Tensor, edge_type_ids: Tensor) -> Tensor:
+ hidden = self.input_projection(node_inputs)
+ if edge_src.numel() == 0:
+ return hidden
+ if self.reverse_support_edge_type_ids and _env_flag("TMCRA_NODE_MEMORY_REVERSE_SUPPORT_MESSAGES", default=False):
+ reverse_mask = torch.zeros_like(edge_type_ids, dtype=torch.bool)
+ for edge_type_id in self.reverse_support_edge_type_ids:
+ reverse_mask = reverse_mask | (edge_type_ids == int(edge_type_id))
+ if bool(reverse_mask.any().item()):
+ reverse_src = edge_dst.masked_select(reverse_mask)
+ reverse_dst = edge_src.masked_select(reverse_mask)
+ reverse_types = edge_type_ids.masked_select(reverse_mask)
+ edge_src = torch.cat([edge_src, reverse_src], dim=0)
+ edge_dst = torch.cat([edge_dst, reverse_dst], dim=0)
+ edge_type_ids = torch.cat([edge_type_ids, reverse_types], dim=0)
+ _validate_index_values(
+ edge_src.detach().cpu().tolist(),
+ upper_bound=int(hidden.size(0)),
+ index_name="edge_src",
+ context="TypedMessagePassing.forward",
+ extra={"node_count": int(hidden.size(0)), "edge_count": int(edge_src.numel())},
+ )
+ _validate_index_values(
+ edge_dst.detach().cpu().tolist(),
+ upper_bound=int(hidden.size(0)),
+ index_name="edge_dst",
+ context="TypedMessagePassing.forward",
+ extra={"node_count": int(hidden.size(0)), "edge_count": int(edge_dst.numel())},
+ )
+ _validate_index_values(
+ edge_type_ids.detach().cpu().tolist(),
+ upper_bound=int(self.edge_embedding.num_embeddings),
+ index_name="edge_type_id",
+ context="TypedMessagePassing.forward",
+ extra={"edge_count": int(edge_type_ids.numel())},
+ )
+ for message_layer, update_layer, norm_layer in zip(self.message_layers, self.update_layers, self.norm_layers):
+ edge_features = self.edge_embedding(edge_type_ids)
+ src_hidden = hidden.index_select(0, edge_src)
+ messages = F.silu(message_layer(torch.cat([src_hidden, edge_features], dim=-1)))
+ messages = messages.to(dtype=hidden.dtype)
+ aggregated = torch.zeros_like(hidden)
+ aggregated.index_add_(0, edge_dst, messages)
+ counts = torch.zeros((hidden.size(0), 1), dtype=hidden.dtype, device=hidden.device)
+ counts.index_add_(0, edge_dst, torch.ones((edge_dst.size(0), 1), dtype=hidden.dtype, device=hidden.device))
+ aggregated = aggregated / counts.clamp_min(1.0)
+ update = F.silu(update_layer(torch.cat([hidden, aggregated], dim=-1)))
+ hidden = norm_layer(hidden + self.dropout(update))
+ return hidden
+
+
+class ResidualPairFeatureAdapter(nn.Module):
+ def __init__(self, feature_dim: int, *, dropout: float = MESSAGE_DROPOUT) -> None:
+ super().__init__()
+ hidden_dim = max(32, int(feature_dim) * 2)
+ self.norm = nn.LayerNorm(feature_dim)
+ self.dropout = nn.Dropout(dropout)
+ self.net = nn.Sequential(
+ nn.Linear(feature_dim, hidden_dim),
+ nn.SiLU(),
+ nn.Linear(hidden_dim, feature_dim),
+ )
+ nn.init.zeros_(self.net[-1].weight)
+ nn.init.zeros_(self.net[-1].bias)
+
+ def forward(self, features: Tensor) -> Tensor:
+ if features.numel() == 0:
+ return features
+ gate_input = self.dropout(self.norm(features))
+ delta = 0.5 * torch.tanh(self.net(gate_input))
+ return features * (1.0 + delta)
+
+
+class EventRecallHead(nn.Module):
+ def __init__(self) -> None:
+ super().__init__()
+ self.net = nn.Sequential(
+ nn.Linear(QUESTION_OUTPUT_DIM + MESSAGE_HIDDEN_DIM + MESSAGE_HIDDEN_DIM + PAIR_FEATURE_DIM, MESSAGE_HIDDEN_DIM),
+ nn.SiLU(),
+ nn.Linear(MESSAGE_HIDDEN_DIM, 1),
+ )
+ self.feature_residual = nn.Sequential(
+ nn.LayerNorm(PAIR_FEATURE_DIM),
+ nn.Linear(PAIR_FEATURE_DIM, MESSAGE_HIDDEN_DIM // 2),
+ nn.SiLU(),
+ nn.Linear(MESSAGE_HIDDEN_DIM // 2, 1),
+ )
+
+ def forward(self, question_embedding: Tensor, event_embeddings: Tensor, question_projected: Tensor, pair_features: Tensor) -> Tensor:
+ if event_embeddings.dim() == 2:
+ question_features = question_embedding.unsqueeze(0).expand(event_embeddings.size(0), -1)
+ projected = question_projected.unsqueeze(0).expand(event_embeddings.size(0), -1)
+ return (
+ self.net(torch.cat([question_features, event_embeddings, projected * event_embeddings, pair_features], dim=-1))
+ + self.feature_residual(pair_features)
+ ).squeeze(-1)
+ if event_embeddings.dim() == 3:
+ question_features = question_embedding.unsqueeze(1).expand(-1, event_embeddings.size(1), -1)
+ projected = question_projected.unsqueeze(1).expand(-1, event_embeddings.size(1), -1)
+ return (
+ self.net(torch.cat([question_features, event_embeddings, projected * event_embeddings, pair_features], dim=-1))
+ + self.feature_residual(pair_features)
+ ).squeeze(-1)
+ raise ValueError(f"Unsupported event_embeddings rank: {event_embeddings.dim()}")
+
+
+class EventScoreHead(nn.Module):
+ def __init__(self) -> None:
+ super().__init__()
+ self.net = nn.Sequential(
+ nn.Linear(
+ QUESTION_OUTPUT_DIM
+ + MESSAGE_HIDDEN_DIM
+ + MESSAGE_HIDDEN_DIM
+ + MESSAGE_HIDDEN_DIM
+ + MESSAGE_HIDDEN_DIM
+ + PAIR_FEATURE_DIM,
+ MESSAGE_HIDDEN_DIM,
+ ),
+ nn.SiLU(),
+ nn.Linear(MESSAGE_HIDDEN_DIM, 1),
+ )
+ self.feature_residual = nn.Sequential(
+ nn.LayerNorm(PAIR_FEATURE_DIM),
+ nn.Linear(PAIR_FEATURE_DIM, MESSAGE_HIDDEN_DIM // 2),
+ nn.SiLU(),
+ nn.Linear(MESSAGE_HIDDEN_DIM // 2, 1),
+ )
+
+ def forward(
+ self,
+ question_embedding: Tensor,
+ event_embeddings: Tensor,
+ support_embeddings: Tensor,
+ question_projected: Tensor,
+ pair_features: Tensor,
+ ) -> Tensor:
+ if event_embeddings.dim() == 2:
+ question_features = question_embedding.unsqueeze(0).expand(event_embeddings.size(0), -1)
+ projected = question_projected.unsqueeze(0).expand(event_embeddings.size(0), -1)
+ return (
+ self.net(
+ torch.cat(
+ [
+ question_features,
+ event_embeddings,
+ support_embeddings,
+ projected * event_embeddings,
+ projected * support_embeddings,
+ pair_features,
+ ],
+ dim=-1,
+ )
+ )
+ + self.feature_residual(pair_features)
+ ).squeeze(-1)
+ if event_embeddings.dim() == 3:
+ question_features = question_embedding.unsqueeze(1).expand(-1, event_embeddings.size(1), -1)
+ projected = question_projected.unsqueeze(1).expand(-1, event_embeddings.size(1), -1)
+ return (
+ self.net(
+ torch.cat(
+ [
+ question_features,
+ event_embeddings,
+ support_embeddings,
+ projected * event_embeddings,
+ projected * support_embeddings,
+ pair_features,
+ ],
+ dim=-1,
+ )
+ )
+ + self.feature_residual(pair_features)
+ ).squeeze(-1)
+ raise ValueError(f"Unsupported event_embeddings rank: {event_embeddings.dim()}")
+
+
+class EventMatrixRerankBlock(nn.Module):
+ def __init__(self, *, hidden_dim: int = MESSAGE_HIDDEN_DIM, num_heads: int = 4, dropout: float = MESSAGE_DROPOUT) -> None:
+ super().__init__()
+ self.norm1 = nn.LayerNorm(hidden_dim)
+ self.attention = nn.MultiheadAttention(hidden_dim, num_heads=num_heads, dropout=dropout, batch_first=True)
+ self.norm2 = nn.LayerNorm(hidden_dim)
+ self.ffn = nn.Sequential(
+ nn.Linear(hidden_dim, hidden_dim * 4),
+ nn.SiLU(),
+ nn.Dropout(dropout),
+ nn.Linear(hidden_dim * 4, hidden_dim),
+ )
+ self.dropout = nn.Dropout(dropout)
+
+ def forward(self, hidden: Tensor, *, attn_mask: Tensor, event_mask: Tensor) -> Tensor:
+ masked_hidden = hidden * event_mask.unsqueeze(-1).to(dtype=hidden.dtype)
+ attended, _ = self.attention(
+ self.norm1(masked_hidden),
+ self.norm1(masked_hidden),
+ self.norm1(masked_hidden),
+ attn_mask=attn_mask,
+ need_weights=False,
+ )
+ hidden = masked_hidden + self.dropout(attended)
+ hidden = hidden * event_mask.unsqueeze(-1).to(dtype=hidden.dtype)
+ hidden = hidden + self.dropout(self.ffn(self.norm2(hidden)))
+ return hidden * event_mask.unsqueeze(-1).to(dtype=hidden.dtype)
+
+
+class EventMatrixRerankHead(nn.Module):
+ def __init__(self, *, hidden_dim: int = MESSAGE_HIDDEN_DIM, num_heads: int = 4, dropout: float = MESSAGE_DROPOUT) -> None:
+ super().__init__()
+ self.num_heads = num_heads
+ self.pair_feature_projection = nn.Sequential(
+ nn.LayerNorm(PAIR_FEATURE_DIM),
+ nn.Linear(PAIR_FEATURE_DIM, hidden_dim),
+ nn.SiLU(),
+ )
+ self.token_projection = nn.Linear(QUESTION_OUTPUT_DIM + (hidden_dim * 5), hidden_dim)
+ self.blocks = nn.ModuleList(
+ [
+ EventMatrixRerankBlock(hidden_dim=hidden_dim, num_heads=num_heads, dropout=dropout),
+ EventMatrixRerankBlock(hidden_dim=hidden_dim, num_heads=num_heads, dropout=dropout),
+ ]
+ )
+ self.relation_bias_projection = nn.Linear(MATRIX_RELATION_FEATURE_DIM, num_heads, bias=False)
+ self.output_projection = nn.Linear(hidden_dim, 1)
+ nn.init.zeros_(self.output_projection.weight)
+ nn.init.zeros_(self.output_projection.bias)
+
+ def forward(
+ self,
+ question_embedding: Tensor,
+ event_embeddings: Tensor,
+ support_embeddings: Tensor,
+ question_projected: Tensor,
+ pair_features: Tensor,
+ relation_features: Tensor,
+ event_mask: Tensor,
+ ) -> Tensor:
+ squeeze_batch = False
+ if event_embeddings.dim() == 2:
+ question_embedding = question_embedding.unsqueeze(0)
+ event_embeddings = event_embeddings.unsqueeze(0)
+ support_embeddings = support_embeddings.unsqueeze(0)
+ question_projected = question_projected.unsqueeze(0)
+ pair_features = pair_features.unsqueeze(0)
+ relation_features = relation_features.unsqueeze(0)
+ event_mask = event_mask.unsqueeze(0)
+ squeeze_batch = True
+ if event_embeddings.dim() != 3:
+ raise ValueError(f"Unsupported event_embeddings rank: {event_embeddings.dim()}")
+ batch_size, event_count, _ = event_embeddings.shape
+ pair_projected = self.pair_feature_projection(pair_features)
+ question_features = question_embedding.unsqueeze(1).expand(-1, event_count, -1)
+ projected = question_projected.unsqueeze(1).expand(-1, event_count, -1)
+ hidden = self.token_projection(
+ torch.cat(
+ [
+ question_features,
+ event_embeddings,
+ support_embeddings,
+ projected * event_embeddings,
+ projected * support_embeddings,
+ pair_projected,
+ ],
+ dim=-1,
+ )
+ )
+ hidden = hidden * event_mask.unsqueeze(-1).to(dtype=hidden.dtype)
+ relation_bias = self.relation_bias_projection(relation_features).permute(0, 3, 1, 2)
+ relation_bias = relation_bias.masked_fill((~event_mask).unsqueeze(1).unsqueeze(1), -1e4)
+ attn_mask = relation_bias.reshape(batch_size * self.num_heads, event_count, event_count)
+ for block in self.blocks:
+ hidden = block(hidden, attn_mask=attn_mask, event_mask=event_mask)
+ delta_logits = self.output_projection(hidden).squeeze(-1)
+ delta_logits = delta_logits * event_mask.to(dtype=delta_logits.dtype)
+ return delta_logits[0] if squeeze_batch else delta_logits
+
+
+class PathMatrixRerankHead(nn.Module):
+ def __init__(self, *, hidden_dim: int = MESSAGE_HIDDEN_DIM, num_heads: int = 4, dropout: float = MESSAGE_DROPOUT) -> None:
+ super().__init__()
+ self.num_heads = num_heads
+ self.pair_feature_projection = nn.Sequential(
+ nn.LayerNorm(PATH_PAIR_FEATURE_DIM),
+ nn.Linear(PATH_PAIR_FEATURE_DIM, hidden_dim),
+ nn.SiLU(),
+ )
+ self.token_projection = nn.Linear(QUESTION_OUTPUT_DIM + (hidden_dim * 5), hidden_dim)
+ self.blocks = nn.ModuleList(
+ [
+ EventMatrixRerankBlock(hidden_dim=hidden_dim, num_heads=num_heads, dropout=dropout),
+ EventMatrixRerankBlock(hidden_dim=hidden_dim, num_heads=num_heads, dropout=dropout),
+ ]
+ )
+ self.relation_bias_projection = nn.Linear(PATH_MATRIX_RELATION_FEATURE_DIM, num_heads, bias=False)
+ self.output_projection = nn.Linear(hidden_dim, 1)
+ nn.init.zeros_(self.output_projection.weight)
+ nn.init.zeros_(self.output_projection.bias)
+
+ def forward(
+ self,
+ question_embedding: Tensor,
+ event_embeddings: Tensor,
+ support_embeddings: Tensor,
+ question_projected: Tensor,
+ pair_features: Tensor,
+ relation_features: Tensor,
+ path_mask: Tensor,
+ ) -> Tensor:
+ squeeze_batch = False
+ if event_embeddings.dim() == 2:
+ question_embedding = question_embedding.unsqueeze(0)
+ event_embeddings = event_embeddings.unsqueeze(0)
+ support_embeddings = support_embeddings.unsqueeze(0)
+ question_projected = question_projected.unsqueeze(0)
+ pair_features = pair_features.unsqueeze(0)
+ relation_features = relation_features.unsqueeze(0)
+ path_mask = path_mask.unsqueeze(0)
+ squeeze_batch = True
+ if event_embeddings.dim() != 3:
+ raise ValueError(f"Unsupported event_embeddings rank: {event_embeddings.dim()}")
+ batch_size, path_count, _ = event_embeddings.shape
+ pair_projected = self.pair_feature_projection(pair_features)
+ question_features = question_embedding.unsqueeze(1).expand(-1, path_count, -1)
+ projected = question_projected.unsqueeze(1).expand(-1, path_count, -1)
+ hidden = self.token_projection(
+ torch.cat(
+ [
+ question_features,
+ event_embeddings,
+ support_embeddings,
+ projected * event_embeddings,
+ projected * support_embeddings,
+ pair_projected,
+ ],
+ dim=-1,
+ )
+ )
+ hidden = hidden * path_mask.unsqueeze(-1).to(dtype=hidden.dtype)
+ relation_bias = self.relation_bias_projection(relation_features).permute(0, 3, 1, 2)
+ relation_bias = relation_bias.masked_fill((~path_mask).unsqueeze(1).unsqueeze(1), -1e4)
+ attn_mask = relation_bias.reshape(batch_size * self.num_heads, path_count, path_count)
+ for block in self.blocks:
+ hidden = block(hidden, attn_mask=attn_mask, event_mask=path_mask)
+ delta_logits = self.output_projection(hidden).squeeze(-1)
+ delta_logits = delta_logits * path_mask.to(dtype=delta_logits.dtype)
+ return delta_logits[0] if squeeze_batch else delta_logits
+
+
+class PathScoreHead(nn.Module):
+ def __init__(self) -> None:
+ super().__init__()
+ self.path_type_embedding = nn.Embedding(len(PATH_TYPE_TO_ID), EDGE_TYPE_EMBED_DIM)
+ self.net = nn.Sequential(
+ nn.Linear(
+ QUESTION_OUTPUT_DIM
+ + MESSAGE_HIDDEN_DIM
+ + MESSAGE_HIDDEN_DIM
+ + MESSAGE_HIDDEN_DIM
+ + MESSAGE_HIDDEN_DIM
+ + EDGE_TYPE_EMBED_DIM
+ + PATH_PAIR_FEATURE_DIM,
+ MESSAGE_HIDDEN_DIM,
+ ),
+ nn.SiLU(),
+ nn.Linear(MESSAGE_HIDDEN_DIM, 1),
+ )
+ self.feature_residual = nn.Sequential(
+ nn.LayerNorm(PATH_PAIR_FEATURE_DIM),
+ nn.Linear(PATH_PAIR_FEATURE_DIM, MESSAGE_HIDDEN_DIM // 2),
+ nn.SiLU(),
+ nn.Linear(MESSAGE_HIDDEN_DIM // 2, 1),
+ )
+
+ def forward(
+ self,
+ question_embedding: Tensor,
+ event_embeddings: Tensor,
+ support_embeddings: Tensor,
+ question_projected: Tensor,
+ path_type_ids: Tensor,
+ pair_features: Tensor,
+ ) -> Tensor:
+ type_embeddings = self.path_type_embedding(path_type_ids)
+ if event_embeddings.dim() == 2:
+ question_features = question_embedding.unsqueeze(0).expand(event_embeddings.size(0), -1)
+ projected = question_projected.unsqueeze(0).expand(event_embeddings.size(0), -1)
+ return (
+ self.net(
+ torch.cat(
+ [
+ question_features,
+ event_embeddings,
+ support_embeddings,
+ projected * event_embeddings,
+ projected * support_embeddings,
+ type_embeddings,
+ pair_features,
+ ],
+ dim=-1,
+ )
+ )
+ + self.feature_residual(pair_features)
+ ).squeeze(-1)
+ if event_embeddings.dim() == 3:
+ question_features = question_embedding.unsqueeze(1).expand(-1, event_embeddings.size(1), -1)
+ projected = question_projected.unsqueeze(1).expand(-1, event_embeddings.size(1), -1)
+ return (
+ self.net(
+ torch.cat(
+ [
+ question_features,
+ event_embeddings,
+ support_embeddings,
+ projected * event_embeddings,
+ projected * support_embeddings,
+ type_embeddings,
+ pair_features,
+ ],
+ dim=-1,
+ )
+ )
+ + self.feature_residual(pair_features)
+ ).squeeze(-1)
+ raise ValueError(f"Unsupported event_embeddings rank: {event_embeddings.dim()}")
+
+
+class EventSubgraphAttentionRefiner(nn.Module):
+ def __init__(
+ self,
+ *,
+ hidden_dim: int = MESSAGE_HIDDEN_DIM,
+ attention_layers: int = EVENT_SUBGRAPH_ATTENTION_LAYERS,
+ attention_heads: int = EVENT_SUBGRAPH_ATTENTION_HEADS,
+ dropout: float = MESSAGE_DROPOUT,
+ ) -> None:
+ super().__init__()
+ self.hidden_dim = int(hidden_dim)
+ self.role_embedding = nn.Embedding(2 + len(EVENT_SUPPORT_PATH_TYPE_ORDER), hidden_dim)
+ self.input_norm = nn.LayerNorm(hidden_dim)
+ self.input_dropout = nn.Dropout(dropout)
+ self.blocks = nn.ModuleList(
+ [
+ _LightweightSelfAttentionBlock(
+ embed_dim=hidden_dim,
+ num_heads=attention_heads,
+ dropout=dropout,
+ )
+ for _ in range(max(1, int(attention_layers)))
+ ]
+ )
+ self.event_output_norm = nn.LayerNorm(hidden_dim)
+ self.support_pool_norm = nn.LayerNorm(hidden_dim)
+ self.support_pool_projection = nn.Linear(hidden_dim, 1)
+
+ def forward(
+ self,
+ question_projected: Tensor,
+ event_embeddings: Tensor,
+ support_slot_embeddings: Tensor,
+ support_slot_mask: Tensor,
+ event_mask: Tensor,
+ ) -> tuple[Tensor, Tensor]:
+ squeeze_batch = False
+ if event_embeddings.dim() == 2:
+ question_projected = question_projected.unsqueeze(0)
+ event_embeddings = event_embeddings.unsqueeze(0)
+ support_slot_embeddings = support_slot_embeddings.unsqueeze(0)
+ support_slot_mask = support_slot_mask.unsqueeze(0)
+ event_mask = event_mask.unsqueeze(0)
+ squeeze_batch = True
+ if event_embeddings.dim() != 3 or support_slot_embeddings.dim() != 4:
+ raise ValueError(
+ "EventSubgraphAttentionRefiner.forward expects "
+ f"event_embeddings rank 3 and support_slot_embeddings rank 4, got "
+ f"{event_embeddings.dim()} and {support_slot_embeddings.dim()}"
+ )
+ batch_size, event_count, hidden_dim = event_embeddings.shape
+ slot_count = int(support_slot_embeddings.size(2))
+ if hidden_dim != self.hidden_dim:
+ raise ValueError(
+ f"EventSubgraphAttentionRefiner hidden_dim mismatch: expected {self.hidden_dim}, got {hidden_dim}"
+ )
+ question_token = question_projected.unsqueeze(1).expand(-1, event_count, -1).unsqueeze(2)
+ event_token = event_embeddings.unsqueeze(2)
+ hidden = torch.cat([question_token, event_token, support_slot_embeddings], dim=2)
+ role_ids = torch.arange(2 + slot_count, dtype=torch.long, device=event_embeddings.device)
+ hidden = hidden + self.role_embedding(role_ids).view(1, 1, 2 + slot_count, hidden_dim)
+ hidden = self.input_dropout(self.input_norm(hidden))
+ token_mask = torch.cat(
+ [
+ event_mask.unsqueeze(-1),
+ event_mask.unsqueeze(-1),
+ support_slot_mask,
+ ],
+ dim=-1,
+ )
+ flat_hidden = hidden.view(batch_size * event_count, 2 + slot_count, hidden_dim)
+ flat_mask = token_mask.view(batch_size * event_count, 2 + slot_count)
+ flat_valid = event_mask.reshape(batch_size * event_count)
+ output_hidden = flat_hidden.clone()
+ if bool(flat_valid.any().item()):
+ valid_indices = torch.nonzero(flat_valid, as_tuple=False).squeeze(-1)
+ valid_hidden = flat_hidden.index_select(0, valid_indices)
+ valid_mask = flat_mask.index_select(0, valid_indices)
+ for block in self.blocks:
+ valid_hidden = block(valid_hidden, valid_mask)
+ output_hidden.index_copy_(0, valid_indices, valid_hidden)
+ hidden = output_hidden.view(batch_size, event_count, 2 + slot_count, hidden_dim)
+ refined_event_embeddings = self.event_output_norm(hidden[:, :, 1, :])
+ support_hidden = hidden[:, :, 2:, :]
+ support_logits = self.support_pool_projection(self.support_pool_norm(support_hidden)).squeeze(-1)
+ support_logits = support_logits.masked_fill(~support_slot_mask, -1e4)
+ has_support = support_slot_mask.any(dim=-1, keepdim=True)
+ support_logits = torch.where(has_support, support_logits, torch.zeros_like(support_logits))
+ support_weights = torch.softmax(support_logits, dim=-1)
+ support_weights = support_weights.masked_fill(~support_slot_mask, 0.0)
+ support_weights = support_weights / support_weights.sum(dim=-1, keepdim=True).clamp_min(1e-6)
+ refined_support_embeddings = (support_hidden * support_weights.unsqueeze(-1)).sum(dim=2)
+ has_support = support_slot_mask.any(dim=-1)
+ refined_support_embeddings = refined_support_embeddings * has_support.unsqueeze(-1).to(
+ dtype=refined_support_embeddings.dtype
+ )
+ refined_event_embeddings = refined_event_embeddings * event_mask.unsqueeze(-1).to(dtype=refined_event_embeddings.dtype)
+ if squeeze_batch:
+ return refined_event_embeddings[0], refined_support_embeddings[0]
+ return refined_event_embeddings, refined_support_embeddings
+
+
+class TemporalHead(nn.Module):
+ def __init__(self) -> None:
+ super().__init__()
+ self.net = nn.Sequential(
+ nn.Linear(
+ QUESTION_OUTPUT_DIM + MESSAGE_HIDDEN_DIM + MESSAGE_HIDDEN_DIM + MESSAGE_HIDDEN_DIM + MESSAGE_HIDDEN_DIM,
+ MESSAGE_HIDDEN_DIM,
+ ),
+ nn.SiLU(),
+ nn.Linear(MESSAGE_HIDDEN_DIM, 1),
+ )
+
+ def forward(
+ self,
+ question_embedding: Tensor,
+ event_embeddings: Tensor,
+ time_embeddings: Tensor,
+ question_projected: Tensor,
+ ) -> Tensor:
+ if event_embeddings.dim() == 2:
+ question_features = question_embedding.unsqueeze(0).expand(event_embeddings.size(0), -1)
+ projected = question_projected.unsqueeze(0).expand(event_embeddings.size(0), -1)
+ return self.net(
+ torch.cat(
+ [
+ question_features,
+ event_embeddings,
+ time_embeddings,
+ projected * event_embeddings,
+ projected * time_embeddings,
+ ],
+ dim=-1,
+ )
+ ).squeeze(-1)
+ if event_embeddings.dim() == 3:
+ question_features = question_embedding.unsqueeze(1).expand(-1, event_embeddings.size(1), -1)
+ projected = question_projected.unsqueeze(1).expand(-1, event_embeddings.size(1), -1)
+ return self.net(
+ torch.cat(
+ [
+ question_features,
+ event_embeddings,
+ time_embeddings,
+ projected * event_embeddings,
+ projected * time_embeddings,
+ ],
+ dim=-1,
+ )
+ ).squeeze(-1)
+ raise ValueError(f"Unsupported event_embeddings rank: {event_embeddings.dim()}")
+
+
+class AnswerTypeHead(nn.Module):
+ def __init__(self) -> None:
+ super().__init__()
+ self.net = nn.Sequential(
+ nn.Linear(
+ QUESTION_OUTPUT_DIM
+ + MESSAGE_HIDDEN_DIM
+ + MESSAGE_HIDDEN_DIM
+ + MESSAGE_HIDDEN_DIM
+ + MESSAGE_HIDDEN_DIM
+ + ANSWER_CALIBRATION_FEATURE_DIM,
+ MESSAGE_HIDDEN_DIM,
+ ),
+ nn.SiLU(),
+ nn.Linear(MESSAGE_HIDDEN_DIM, len(ANSWER_TYPE_TO_ID)),
+ )
+
+ def forward(
+ self,
+ question_embedding: Tensor,
+ pooled_event_embedding: Tensor,
+ pooled_support_embedding: Tensor,
+ question_projected: Tensor,
+ calibration_features: Tensor,
+ ) -> Tensor:
+ return self.net(
+ torch.cat(
+ [
+ question_embedding,
+ pooled_event_embedding,
+ pooled_support_embedding,
+ question_projected * pooled_event_embedding,
+ question_projected * pooled_support_embedding,
+ calibration_features,
+ ],
+ dim=-1,
+ )
+ )
+
+
+class AnswerPlanHead(nn.Module):
+ def __init__(self) -> None:
+ super().__init__()
+ self.net = nn.Sequential(
+ nn.Linear(
+ QUESTION_OUTPUT_DIM
+ + MESSAGE_HIDDEN_DIM
+ + MESSAGE_HIDDEN_DIM
+ + MESSAGE_HIDDEN_DIM
+ + MESSAGE_HIDDEN_DIM
+ + PAIR_FEATURE_DIM
+ + ANSWER_PLAN_FEATURE_DIM,
+ MESSAGE_HIDDEN_DIM,
+ ),
+ nn.SiLU(),
+ nn.Dropout(MESSAGE_DROPOUT),
+ nn.Linear(MESSAGE_HIDDEN_DIM, len(ANSWER_PLAN_OUTPUTS)),
+ )
+
+ def forward(
+ self,
+ question_embedding: Tensor,
+ event_embeddings: Tensor,
+ support_embeddings: Tensor,
+ question_projected: Tensor,
+ event_pair_features: Tensor,
+ answer_plan_features: Tensor,
+ ) -> Tensor:
+ if event_embeddings.dim() == 2:
+ question_features = question_embedding.unsqueeze(0).expand(event_embeddings.size(0), -1)
+ projected = question_projected.unsqueeze(0).expand(event_embeddings.size(0), -1)
+ elif event_embeddings.dim() == 3:
+ question_features = question_embedding.unsqueeze(1).expand(-1, event_embeddings.size(1), -1)
+ projected = question_projected.unsqueeze(1).expand(-1, event_embeddings.size(1), -1)
+ else:
+ raise ValueError(f"Unsupported event_embeddings rank: {event_embeddings.dim()}")
+ return self.net(
+ torch.cat(
+ [
+ question_features,
+ event_embeddings,
+ support_embeddings,
+ projected * event_embeddings,
+ projected * support_embeddings,
+ event_pair_features,
+ answer_plan_features,
+ ],
+ dim=-1,
+ )
+ )
+
+
+class EventDistractorHead(nn.Module):
+ def __init__(self) -> None:
+ super().__init__()
+ self.net = nn.Sequential(
+ nn.Linear(
+ QUESTION_OUTPUT_DIM
+ + MESSAGE_HIDDEN_DIM
+ + MESSAGE_HIDDEN_DIM
+ + MESSAGE_HIDDEN_DIM
+ + MESSAGE_HIDDEN_DIM
+ + PAIR_FEATURE_DIM
+ + EVENT_DISTRACTOR_FEATURE_DIM,
+ MESSAGE_HIDDEN_DIM,
+ ),
+ nn.SiLU(),
+ )
+ self.output_projection = nn.Linear(MESSAGE_HIDDEN_DIM, 1)
+ nn.init.zeros_(self.output_projection.weight)
+ nn.init.zeros_(self.output_projection.bias)
+
+ def forward(
+ self,
+ question_embedding: Tensor,
+ event_embeddings: Tensor,
+ support_embeddings: Tensor,
+ question_projected: Tensor,
+ pair_features: Tensor,
+ distractor_features: Tensor,
+ ) -> Tensor:
+ if event_embeddings.dim() == 2:
+ question_features = question_embedding.unsqueeze(0).expand(event_embeddings.size(0), -1)
+ projected = question_projected.unsqueeze(0).expand(event_embeddings.size(0), -1)
+ hidden = self.net(
+ torch.cat(
+ [
+ question_features,
+ event_embeddings,
+ support_embeddings,
+ projected * event_embeddings,
+ projected * support_embeddings,
+ pair_features,
+ distractor_features,
+ ],
+ dim=-1,
+ )
+ )
+ return self.output_projection(hidden).squeeze(-1)
+ if event_embeddings.dim() == 3:
+ question_features = question_embedding.unsqueeze(1).expand(-1, event_embeddings.size(1), -1)
+ projected = question_projected.unsqueeze(1).expand(-1, event_embeddings.size(1), -1)
+ hidden = self.net(
+ torch.cat(
+ [
+ question_features,
+ event_embeddings,
+ support_embeddings,
+ projected * event_embeddings,
+ projected * support_embeddings,
+ pair_features,
+ distractor_features,
+ ],
+ dim=-1,
+ )
+ )
+ return self.output_projection(hidden).squeeze(-1)
+ raise ValueError(f"Unsupported event_embeddings rank: {event_embeddings.dim()}")
+
+
+class EventCalibrationHead(nn.Module):
+ def __init__(self) -> None:
+ super().__init__()
+ self.net = nn.Sequential(
+ nn.Linear(
+ QUESTION_OUTPUT_DIM
+ + MESSAGE_HIDDEN_DIM
+ + MESSAGE_HIDDEN_DIM
+ + MESSAGE_HIDDEN_DIM
+ + MESSAGE_HIDDEN_DIM
+ + PAIR_FEATURE_DIM
+ + EVENT_RUNTIME_CALIBRATION_FEATURE_DIM,
+ MESSAGE_HIDDEN_DIM,
+ ),
+ nn.SiLU(),
+ )
+ self.output_projection = nn.Linear(MESSAGE_HIDDEN_DIM, 1)
+ nn.init.zeros_(self.output_projection.weight)
+ nn.init.zeros_(self.output_projection.bias)
+
+ def forward(
+ self,
+ question_embedding: Tensor,
+ event_embeddings: Tensor,
+ support_embeddings: Tensor,
+ question_projected: Tensor,
+ pair_features: Tensor,
+ calibration_features: Tensor,
+ ) -> Tensor:
+ if event_embeddings.dim() == 2:
+ question_features = question_embedding.unsqueeze(0).expand(event_embeddings.size(0), -1)
+ projected = question_projected.unsqueeze(0).expand(event_embeddings.size(0), -1)
+ hidden = self.net(
+ torch.cat(
+ [
+ question_features,
+ event_embeddings,
+ support_embeddings,
+ projected * event_embeddings,
+ projected * support_embeddings,
+ pair_features,
+ calibration_features,
+ ],
+ dim=-1,
+ )
+ )
+ return self.output_projection(hidden).squeeze(-1)
+ if event_embeddings.dim() == 3:
+ question_features = question_embedding.unsqueeze(1).expand(-1, event_embeddings.size(1), -1)
+ projected = question_projected.unsqueeze(1).expand(-1, event_embeddings.size(1), -1)
+ hidden = self.net(
+ torch.cat(
+ [
+ question_features,
+ event_embeddings,
+ support_embeddings,
+ projected * event_embeddings,
+ projected * support_embeddings,
+ pair_features,
+ calibration_features,
+ ],
+ dim=-1,
+ )
+ )
+ return self.output_projection(hidden).squeeze(-1)
+ raise ValueError(f"Unsupported event_embeddings rank: {event_embeddings.dim()}")
+
+
+class PathCalibrationHead(nn.Module):
+ def __init__(self) -> None:
+ super().__init__()
+ self.net = nn.Sequential(
+ nn.Linear(
+ QUESTION_OUTPUT_DIM
+ + MESSAGE_HIDDEN_DIM
+ + MESSAGE_HIDDEN_DIM
+ + MESSAGE_HIDDEN_DIM
+ + MESSAGE_HIDDEN_DIM
+ + PATH_PAIR_FEATURE_DIM
+ + PATH_RUNTIME_CALIBRATION_FEATURE_DIM,
+ MESSAGE_HIDDEN_DIM,
+ ),
+ nn.SiLU(),
+ )
+ self.output_projection = nn.Linear(MESSAGE_HIDDEN_DIM, 1)
+ nn.init.zeros_(self.output_projection.weight)
+ nn.init.zeros_(self.output_projection.bias)
+
+ def forward(
+ self,
+ question_embedding: Tensor,
+ event_embeddings: Tensor,
+ support_embeddings: Tensor,
+ question_projected: Tensor,
+ pair_features: Tensor,
+ calibration_features: Tensor,
+ ) -> Tensor:
+ if event_embeddings.dim() == 2:
+ question_features = question_embedding.unsqueeze(0).expand(event_embeddings.size(0), -1)
+ projected = question_projected.unsqueeze(0).expand(event_embeddings.size(0), -1)
+ hidden = self.net(
+ torch.cat(
+ [
+ question_features,
+ event_embeddings,
+ support_embeddings,
+ projected * event_embeddings,
+ projected * support_embeddings,
+ pair_features,
+ calibration_features,
+ ],
+ dim=-1,
+ )
+ )
+ return self.output_projection(hidden).squeeze(-1)
+ if event_embeddings.dim() == 3:
+ question_features = question_embedding.unsqueeze(1).expand(-1, event_embeddings.size(1), -1)
+ projected = question_projected.unsqueeze(1).expand(-1, event_embeddings.size(1), -1)
+ hidden = self.net(
+ torch.cat(
+ [
+ question_features,
+ event_embeddings,
+ support_embeddings,
+ projected * event_embeddings,
+ projected * support_embeddings,
+ pair_features,
+ calibration_features,
+ ],
+ dim=-1,
+ )
+ )
+ return self.output_projection(hidden).squeeze(-1)
+ raise ValueError(f"Unsupported event_embeddings rank: {event_embeddings.dim()}")
+
+
+class FinalEventFusionHead(nn.Module):
+ def __init__(self) -> None:
+ super().__init__()
+ self.net = nn.Sequential(
+ nn.Linear(
+ QUESTION_OUTPUT_DIM
+ + MESSAGE_HIDDEN_DIM
+ + MESSAGE_HIDDEN_DIM
+ + MESSAGE_HIDDEN_DIM
+ + MESSAGE_HIDDEN_DIM
+ + PAIR_FEATURE_DIM
+ + FINAL_EVENT_FUSION_FEATURE_DIM,
+ MESSAGE_HIDDEN_DIM,
+ ),
+ nn.SiLU(),
+ )
+ self.output_projection = nn.Linear(MESSAGE_HIDDEN_DIM, 1)
+ nn.init.zeros_(self.output_projection.weight)
+ nn.init.zeros_(self.output_projection.bias)
+
+ def forward(
+ self,
+ question_embedding: Tensor,
+ event_embeddings: Tensor,
+ support_embeddings: Tensor,
+ question_projected: Tensor,
+ pair_features: Tensor,
+ fusion_features: Tensor,
+ ) -> Tensor:
+ if event_embeddings.dim() == 2:
+ question_features = question_embedding.unsqueeze(0).expand(event_embeddings.size(0), -1)
+ projected = question_projected.unsqueeze(0).expand(event_embeddings.size(0), -1)
+ hidden = self.net(
+ torch.cat(
+ [
+ question_features,
+ event_embeddings,
+ support_embeddings,
+ projected * event_embeddings,
+ projected * support_embeddings,
+ pair_features,
+ fusion_features,
+ ],
+ dim=-1,
+ )
+ )
+ return self.output_projection(hidden).squeeze(-1)
+ if event_embeddings.dim() == 3:
+ question_features = question_embedding.unsqueeze(1).expand(-1, event_embeddings.size(1), -1)
+ projected = question_projected.unsqueeze(1).expand(-1, event_embeddings.size(1), -1)
+ hidden = self.net(
+ torch.cat(
+ [
+ question_features,
+ event_embeddings,
+ support_embeddings,
+ projected * event_embeddings,
+ projected * support_embeddings,
+ pair_features,
+ fusion_features,
+ ],
+ dim=-1,
+ )
+ )
+ return self.output_projection(hidden).squeeze(-1)
+ raise ValueError(f"Unsupported event_embeddings rank: {event_embeddings.dim()}")
+
+
+class FinalPathFusionHead(nn.Module):
+ def __init__(self) -> None:
+ super().__init__()
+ self.net = nn.Sequential(
+ nn.Linear(
+ QUESTION_OUTPUT_DIM
+ + MESSAGE_HIDDEN_DIM
+ + MESSAGE_HIDDEN_DIM
+ + MESSAGE_HIDDEN_DIM
+ + MESSAGE_HIDDEN_DIM
+ + PATH_PAIR_FEATURE_DIM
+ + FINAL_PATH_FUSION_FEATURE_DIM,
+ MESSAGE_HIDDEN_DIM,
+ ),
+ nn.SiLU(),
+ )
+ self.output_projection = nn.Linear(MESSAGE_HIDDEN_DIM, 1)
+ nn.init.zeros_(self.output_projection.weight)
+ nn.init.zeros_(self.output_projection.bias)
+
+ def forward(
+ self,
+ question_embedding: Tensor,
+ event_embeddings: Tensor,
+ support_embeddings: Tensor,
+ question_projected: Tensor,
+ pair_features: Tensor,
+ fusion_features: Tensor,
+ ) -> Tensor:
+ if event_embeddings.dim() == 2:
+ question_features = question_embedding.unsqueeze(0).expand(event_embeddings.size(0), -1)
+ projected = question_projected.unsqueeze(0).expand(event_embeddings.size(0), -1)
+ hidden = self.net(
+ torch.cat(
+ [
+ question_features,
+ event_embeddings,
+ support_embeddings,
+ projected * event_embeddings,
+ projected * support_embeddings,
+ pair_features,
+ fusion_features,
+ ],
+ dim=-1,
+ )
+ )
+ return self.output_projection(hidden).squeeze(-1)
+ if event_embeddings.dim() == 3:
+ question_features = question_embedding.unsqueeze(1).expand(-1, event_embeddings.size(1), -1)
+ projected = question_projected.unsqueeze(1).expand(-1, event_embeddings.size(1), -1)
+ hidden = self.net(
+ torch.cat(
+ [
+ question_features,
+ event_embeddings,
+ support_embeddings,
+ projected * event_embeddings,
+ projected * support_embeddings,
+ pair_features,
+ fusion_features,
+ ],
+ dim=-1,
+ )
+ )
+ return self.output_projection(hidden).squeeze(-1)
+ raise ValueError(f"Unsupported event_embeddings rank: {event_embeddings.dim()}")
+
+
+class MemoryTunnelHead(nn.Module):
+ def __init__(self, pair_feature_dim: int, tunnel_feature_dim: int) -> None:
+ super().__init__()
+ self.net = nn.Sequential(
+ nn.Linear(
+ QUESTION_OUTPUT_DIM
+ + MESSAGE_HIDDEN_DIM
+ + MESSAGE_HIDDEN_DIM
+ + MESSAGE_HIDDEN_DIM
+ + MESSAGE_HIDDEN_DIM
+ + int(pair_feature_dim)
+ + int(tunnel_feature_dim),
+ MESSAGE_HIDDEN_DIM,
+ ),
+ nn.SiLU(),
+ )
+ self.output_projection = nn.Linear(MESSAGE_HIDDEN_DIM, 2)
+ nn.init.zeros_(self.output_projection.weight)
+ nn.init.zeros_(self.output_projection.bias)
+
+ def forward(
+ self,
+ question_embedding: Tensor,
+ event_embeddings: Tensor,
+ support_embeddings: Tensor,
+ question_projected: Tensor,
+ pair_features: Tensor,
+ tunnel_features: Tensor,
+ ) -> Tensor:
+ if event_embeddings.dim() == 2:
+ question_features = question_embedding.unsqueeze(0).expand(event_embeddings.size(0), -1)
+ projected = question_projected.unsqueeze(0).expand(event_embeddings.size(0), -1)
+ hidden = self.net(
+ torch.cat(
+ [
+ question_features,
+ event_embeddings,
+ support_embeddings,
+ projected * event_embeddings,
+ projected * support_embeddings,
+ pair_features,
+ tunnel_features,
+ ],
+ dim=-1,
+ )
+ )
+ return self.output_projection(hidden)
+ if event_embeddings.dim() == 3:
+ question_features = question_embedding.unsqueeze(1).expand(-1, event_embeddings.size(1), -1)
+ projected = question_projected.unsqueeze(1).expand(-1, event_embeddings.size(1), -1)
+ hidden = self.net(
+ torch.cat(
+ [
+ question_features,
+ event_embeddings,
+ support_embeddings,
+ projected * event_embeddings,
+ projected * support_embeddings,
+ pair_features,
+ tunnel_features,
+ ],
+ dim=-1,
+ )
+ )
+ return self.output_projection(hidden)
+ raise ValueError(f"Unsupported event_embeddings rank: {event_embeddings.dim()}")
+
+
+class LocomoNodeMemoryModel(nn.Module):
+ def __init__(self) -> None:
+ super().__init__()
+ self.question_encoder = QuestionEncoder()
+ self.question_intent_head = QuestionIntentHead()
+ self.memory_router_head = MemoryRouterHead()
+ self.node_encoder = NodeEncoder()
+ self.message_passing = TypedMessagePassing()
+ self.question_projection = nn.Linear(QUESTION_OUTPUT_DIM, MESSAGE_HIDDEN_DIM)
+ self.event_pair_adapter = ResidualPairFeatureAdapter(PAIR_FEATURE_DIM)
+ self.path_pair_adapter = ResidualPairFeatureAdapter(PATH_PAIR_FEATURE_DIM)
+ self.event_recall_head = EventRecallHead()
+ self.event_subgraph_refiner = EventSubgraphAttentionRefiner()
+ self.event_head = EventScoreHead()
+ self.event_matrix_head = EventMatrixRerankHead()
+ self.event_distractor_head = EventDistractorHead()
+ self.path_head = PathScoreHead()
+ self.path_matrix_head = PathMatrixRerankHead()
+ self.temporal_head = TemporalHead()
+ self.answer_type_head = AnswerTypeHead()
+ self.answer_plan_head = AnswerPlanHead()
+ self.event_calibration_head = EventCalibrationHead()
+ self.path_calibration_head = PathCalibrationHead()
+ self.event_tunnel_head = MemoryTunnelHead(PAIR_FEATURE_DIM, EVENT_TUNNEL_FEATURE_DIM)
+ self.path_tunnel_head = MemoryTunnelHead(PATH_PAIR_FEATURE_DIM, PATH_TUNNEL_FEATURE_DIM)
+ self.final_event_fusion_head = FinalEventFusionHead()
+ self.final_path_fusion_head = FinalPathFusionHead()
+ self._last_node_token_role_aux_loss: Tensor | None = None
+
+ def encode_graph(self, graph_tensors: Mapping[str, Any]) -> Tensor:
+ node_inputs = self.node_encoder(
+ graph_tensors["node_texts"],
+ graph_tensors["node_type_ids"],
+ graph_tensors["node_scalar_features"],
+ token_hash_indices_batch=graph_tensors.get("node_text_hash_indices"),
+ )
+ self._last_node_token_role_aux_loss = self.node_encoder.pop_last_role_aux_loss()
+ return self.message_passing(
+ node_inputs,
+ graph_tensors["edge_src"],
+ graph_tensors["edge_dst"],
+ graph_tensors["edge_type_ids"],
+ )
+
+ def pop_last_node_token_role_aux_loss(self) -> Tensor | None:
+ value = self._last_node_token_role_aux_loss
+ self._last_node_token_role_aux_loss = None
+ return value
+
+ def score_example(
+ self,
+ graph_tensors: Mapping[str, Any],
+ example: QueryTrainingExample,
+ *,
+ event_rerank_mode: str = "matrix",
+ matrix_event_top_k: int = DEFAULT_MATRIX_EVENT_TOP_K,
+ event_pair_feature_mode: str = "full",
+ ) -> Dict[str, Any]:
+ return self.score_examples(
+ graph_tensors,
+ [example],
+ event_rerank_mode=event_rerank_mode,
+ matrix_event_top_k=matrix_event_top_k,
+ event_pair_feature_mode=event_pair_feature_mode,
+ )[0]
+
+ def score_example_with_graph_encoding(
+ self,
+ graph_tensors: Mapping[str, Any],
+ node_hidden: Tensor,
+ example: QueryTrainingExample,
+ *,
+ event_rerank_mode: str = "matrix",
+ matrix_event_top_k: int = DEFAULT_MATRIX_EVENT_TOP_K,
+ event_pair_feature_mode: str = "full",
+ recall_only: bool = False,
+ precomputed_recall: Mapping[str, Any] | None = None,
+ ) -> Dict[str, Any]:
+ return self.score_examples_with_graph_encoding(
+ graph_tensors,
+ node_hidden,
+ [example],
+ event_rerank_mode=event_rerank_mode,
+ matrix_event_top_k=matrix_event_top_k,
+ event_pair_feature_mode=event_pair_feature_mode,
+ recall_only=recall_only,
+ precomputed_recall_batch=[dict(precomputed_recall)] if precomputed_recall is not None else None,
+ )[0]
+
+ def score_examples(
+ self,
+ graph_tensors: Mapping[str, Any],
+ examples: Sequence[QueryTrainingExample],
+ *,
+ event_rerank_mode: str = "matrix",
+ matrix_event_top_k: int = DEFAULT_MATRIX_EVENT_TOP_K,
+ event_pair_feature_mode: str = "full",
+ ) -> List[Dict[str, Any]]:
+ if not examples:
+ return []
+ node_hidden = self.encode_graph(graph_tensors)
+ return self.score_examples_with_graph_encoding(
+ graph_tensors,
+ node_hidden,
+ examples,
+ event_rerank_mode=event_rerank_mode,
+ matrix_event_top_k=matrix_event_top_k,
+ event_pair_feature_mode=event_pair_feature_mode,
+ )
+
+ def score_examples_with_graph_encoding(
+ self,
+ graph_tensors: Mapping[str, Any],
+ node_hidden: Tensor,
+ examples: Sequence[QueryTrainingExample],
+ *,
+ event_rerank_mode: str = "matrix",
+ matrix_event_top_k: int = DEFAULT_MATRIX_EVENT_TOP_K,
+ event_pair_feature_mode: str = "full",
+ recall_only: bool = False,
+ precomputed_recall_batch: Sequence[Mapping[str, Any]] | None = None,
+ ) -> List[Dict[str, Any]]:
+ if not examples:
+ return []
+ graph_tensors = dict(_ensure_graph_scoring_feature_cache(graph_tensors))
+ device = graph_tensors["node_type_ids"].device
+ node_id_to_index = graph_tensors["node_id_to_index"]
+ node_lookup = dict(graph_tensors.get("node_by_id", {}) or {})
+ event_support_lookup = dict(graph_tensors.get("event_support_lookup", {}) or {})
+ event_scoring_features_by_id = dict(graph_tensors.get("event_scoring_features_by_id", {}) or {})
+ node_scoring_features_by_id = dict(graph_tensors.get("node_scoring_features_by_id", {}) or {})
+ path_scoring_features_by_id = dict(graph_tensors.get("path_scoring_features_by_id", {}) or {})
+ question_features_batch = [dict(example.question_features or {}) for example in examples]
+ question_batch = self.question_encoder.forward_batch(
+ [example.question for example in examples],
+ question_features_batch,
+ token_hash_indices_batch=[example.question_hash_indices for example in examples],
+ feature_indices=[int(example.question_feature_index or 0) for example in examples],
+ query_type_indices=[int(example.question_query_type_index or 0) for example in examples],
+ )
+ memory_router_logits_batch = self.memory_router_head(question_batch)
+ question_text_embeddings = self.question_encoder.pop_last_text_embeddings()
+ question_token_role_aux_loss = self.question_encoder.pop_last_role_aux_loss()
+ question_role_predictions = self.question_encoder.pop_last_role_predictions() or {}
+ question_intent_outputs = (
+ self.question_intent_head(question_text_embeddings)
+ if isinstance(question_text_embeddings, Tensor) and question_text_embeddings.numel() > 0
+ else {
+ "semantic_logits": question_batch.new_zeros((len(examples), len(SEMANTIC_SLOTS))),
+ "status_logits": question_batch.new_zeros((len(examples), len(TARGET_STATUSES))),
+ "time_granularity_logits": question_batch.new_zeros((len(examples), len(TIME_GRANULARITIES))),
+ "temporal_logits": question_batch.new_zeros((len(examples),)),
+ }
+ )
+ predicted_question_role_ids = question_role_predictions.get("predicted_role_ids")
+ predicted_question_role_mask = question_role_predictions.get("token_mask")
+ if isinstance(predicted_question_role_ids, Tensor) and isinstance(predicted_question_role_mask, Tensor):
+ learned_question_features_batch: List[Dict[str, Any]] = []
+ for batch_index, example in enumerate(examples):
+ token_count = min(
+ max(1, len(list(example.question_hash_indices or []))),
+ QUESTION_MAX_TOKENS,
+ )
+ aligned_tokens = _align_sequence_length(
+ _resolved_text_tokens(example.question),
+ target_length=token_count,
+ fill_value="__empty__",
+ ) or ["__empty__"]
+ role_ids = predicted_question_role_ids[batch_index, : len(aligned_tokens)].detach().cpu().tolist()
+ token_mask_values = predicted_question_role_mask[batch_index, : len(aligned_tokens)].detach().cpu().tolist()
+ learned_question_features_batch.append(
+ _merge_learned_question_features_from_token_roles(
+ example.question,
+ question_features_batch[batch_index],
+ tokens=aligned_tokens,
+ predicted_role_ids=role_ids,
+ token_mask=token_mask_values,
+ )
+ )
+ question_features_batch = learned_question_features_batch
+ learned_intent_question_features_batch: List[Dict[str, Any]] = []
+ for batch_index in range(len(examples)):
+ learned_intent_question_features_batch.append(
+ _merge_learned_question_features_from_intent_logits(
+ question_features_batch[batch_index],
+ semantic_logits=question_intent_outputs["semantic_logits"][batch_index],
+ status_logits=question_intent_outputs["status_logits"][batch_index],
+ time_granularity_logits=question_intent_outputs["time_granularity_logits"][batch_index],
+ temporal_logit=question_intent_outputs["temporal_logits"][batch_index],
+ )
+ )
+ question_features_batch = learned_intent_question_features_batch
+ prepared_question_batch = [
+ _prepare_question_scoring_features(example.question, question_features_batch[batch_index])
+ for batch_index, example in enumerate(examples)
+ ]
+ node_token_role_aux_loss = self.pop_last_node_token_role_aux_loss()
+ projected_batch = self.question_projection(question_batch)
+ batch_size = len(examples)
+ if precomputed_recall_batch is not None and len(precomputed_recall_batch) != batch_size:
+ raise ValueError("precomputed recall batch size must match examples")
+ resolved_event_rerank_mode = _normalize_event_rerank_mode(event_rerank_mode)
+ resolved_event_pair_feature_mode = _normalize_event_pair_feature_mode(event_pair_feature_mode)
+ resolved_matrix_event_top_k = max(1, int(matrix_event_top_k or DEFAULT_MATRIX_EVENT_TOP_K))
+
+ all_event_ids = [event_id for event_id in list(graph_tensors.get("event_node_ids", []) or []) if event_id in node_id_to_index]
+ event_node_ids_by_conversation = {
+ clean_text(conversation_id): [
+ event_id for event_id in list(event_ids or []) if clean_text(event_id) in node_id_to_index
+ ]
+ for conversation_id, event_ids in dict(graph_tensors.get("event_node_ids_by_conversation", {}) or {}).items()
+ if clean_text(conversation_id)
+ }
+ recall_event_ids_batch: List[List[str]] = []
+ recall_event_lengths: List[int] = []
+ max_recall_event_count = 0
+ for batch_index, example in enumerate(examples):
+ precomputed_recall = (
+ dict(precomputed_recall_batch[batch_index])
+ if precomputed_recall_batch is not None
+ else {}
+ )
+ recall_event_ids = [
+ clean_text(event_id)
+ for event_id in list(precomputed_recall.get("recall_event_ids", []) or [])
+ if clean_text(event_id) in node_id_to_index
+ ]
+ if not recall_event_ids:
+ recall_event_ids = list(event_node_ids_by_conversation.get(example.conversation_id, all_event_ids))
+ if not recall_event_ids:
+ recall_event_ids = list(all_event_ids)
+ recall_event_ids_batch.append(recall_event_ids)
+ recall_event_lengths.append(len(recall_event_ids))
+ max_recall_event_count = max(max_recall_event_count, len(recall_event_ids))
+ recall_event_logits_batch = node_hidden.new_zeros((batch_size, max_recall_event_count))
+ recall_event_mask = torch.zeros((batch_size, max_recall_event_count), dtype=torch.bool, device=device)
+ recall_pair_features_batch = node_hidden.new_zeros((batch_size, max_recall_event_count, PAIR_FEATURE_DIM))
+ recall_event_index_lookup_batch = [
+ {event_id: index for index, event_id in enumerate(recall_event_ids)}
+ for recall_event_ids in recall_event_ids_batch
+ ]
+ if max_recall_event_count > 0 and precomputed_recall_batch is not None:
+ for batch_index, recall_event_ids in enumerate(recall_event_ids_batch):
+ recall_count = len(recall_event_ids)
+ if recall_count <= 0:
+ continue
+ precomputed_recall = dict(precomputed_recall_batch[batch_index])
+ logits = precomputed_recall.get("recall_event_logits")
+ pair_features = precomputed_recall.get("recall_pair_features")
+ if not isinstance(logits, Tensor) or tuple(logits.shape) != (recall_count,):
+ raise ValueError("precomputed recall logits do not match recall event ids")
+ if not isinstance(pair_features, Tensor) or tuple(pair_features.shape) != (recall_count, PAIR_FEATURE_DIM):
+ raise ValueError("precomputed recall pair features do not match recall event ids")
+ recall_event_logits_batch[batch_index, :recall_count] = logits.to(
+ device=device,
+ dtype=node_hidden.dtype,
+ )
+ recall_pair_features_batch[batch_index, :recall_count] = pair_features.to(
+ device=device,
+ dtype=node_hidden.dtype,
+ )
+ recall_event_mask[batch_index, :recall_count] = True
+ elif max_recall_event_count > 0:
+ padded_recall_indices = torch.zeros((batch_size, max_recall_event_count), dtype=torch.long, device=device)
+ for batch_index, recall_event_ids in enumerate(recall_event_ids_batch):
+ if not recall_event_ids:
+ continue
+ recall_index_values = [node_id_to_index[event_id] for event_id in recall_event_ids]
+ _validate_index_values(
+ recall_index_values,
+ upper_bound=int(node_hidden.size(0)),
+ index_name="recall_event_index",
+ context="LocomoNodeMemoryModel.score_examples_with_graph_encoding",
+ extra={
+ "stage": "recall",
+ "conversation_id": examples[batch_index].conversation_id,
+ "question_id": examples[batch_index].question_id,
+ "event_count": len(recall_event_ids),
+ },
+ )
+ recall_indices = torch.tensor(
+ recall_index_values,
+ dtype=torch.long,
+ device=device,
+ )
+ recall_count = int(recall_indices.numel())
+ padded_recall_indices[batch_index, :recall_count] = recall_indices
+ recall_event_mask[batch_index, :recall_count] = True
+ recall_pair_features_batch[batch_index, :recall_count] = torch.tensor(
+ [
+ _event_pair_feature_values(
+ examples[batch_index].question,
+ question_features_batch[batch_index],
+ node_lookup.get(event_id, {}),
+ support_payload=event_support_lookup.get(event_id, {}),
+ prepared_question=prepared_question_batch[batch_index],
+ prepared_event=event_scoring_features_by_id.get(event_id),
+ )
+ for event_id in recall_event_ids
+ ],
+ dtype=node_hidden.dtype,
+ device=device,
+ )
+ recall_pair_features_batch = _apply_event_pair_feature_mode(
+ recall_pair_features_batch,
+ resolved_event_pair_feature_mode,
+ )
+ recall_event_embeddings = node_hidden.index_select(0, padded_recall_indices.reshape(-1)).view(
+ batch_size,
+ max_recall_event_count,
+ -1,
+ )
+ recall_event_logits_batch = self.event_recall_head(
+ question_batch,
+ recall_event_embeddings,
+ projected_batch,
+ recall_pair_features_batch,
+ ).masked_fill(~recall_event_mask, 0.0)
+ recall_event_prob_batch = (
+ torch.sigmoid(recall_event_logits_batch).masked_fill(~recall_event_mask, 0.0)
+ if max_recall_event_count > 0
+ else node_hidden.new_zeros((batch_size, 0))
+ )
+ if recall_only:
+ return [
+ {
+ "recall_event_ids": list(recall_event_ids_batch[batch_index]),
+ "recall_event_logits": recall_event_logits_batch[
+ batch_index, : recall_event_lengths[batch_index]
+ ],
+ "recall_pair_features": recall_pair_features_batch[
+ batch_index, : recall_event_lengths[batch_index]
+ ],
+ "memory_router_logits": memory_router_logits_batch[batch_index],
+ "memory_router_layers": list(MEMORY_ROUTER_LAYERS),
+ "effective_question_features": dict(question_features_batch[batch_index]),
+ }
+ for batch_index in range(batch_size)
+ ]
+
+ candidate_event_ids_batch: List[List[str]] = []
+ candidate_event_lengths: List[int] = []
+ max_event_count = 0
+ for batch_index, example in enumerate(examples):
+ static_candidate_event_ids = [event_id for event_id in example.candidate_event_ids if event_id in node_id_to_index]
+ candidate_limit = max(DEFAULT_RUNTIME_RECALL_TOP_K, len(static_candidate_event_ids))
+ candidate_event_ids = (
+ _select_rerank_event_ids(
+ recall_event_ids_batch[batch_index],
+ recall_event_logits_batch[batch_index, : recall_event_lengths[batch_index]],
+ positive_event_ids=example.positive_event_ids,
+ forced_event_ids=static_candidate_event_ids,
+ candidate_limit=candidate_limit,
+ )
+ if recall_event_lengths[batch_index] > 0
+ else list(static_candidate_event_ids)
+ )
+ if not candidate_event_ids:
+ candidate_event_ids = list(static_candidate_event_ids)
+ candidate_event_ids_batch.append(candidate_event_ids)
+ candidate_event_lengths.append(len(candidate_event_ids))
+ max_event_count = max(max_event_count, len(candidate_event_ids))
+ candidate_event_index_lookup_batch = [
+ {event_id: index for index, event_id in enumerate(candidate_event_ids)}
+ for candidate_event_ids in candidate_event_ids_batch
+ ]
+ candidate_event_recall_indices_batch = [
+ [recall_event_index_lookup_batch[batch_index].get(event_id, -1) for event_id in candidate_event_ids]
+ for batch_index, candidate_event_ids in enumerate(candidate_event_ids_batch)
+ ]
+
+ candidate_paths_batch: List[List[Dict[str, Any]]] = []
+ candidate_path_ids_batch: List[List[str]] = []
+ candidate_path_event_ids_batch: List[List[str]] = []
+ candidate_path_support_node_ids_batch: List[List[str]] = []
+ candidate_path_lengths: List[int] = []
+ max_path_count = 0
+ all_paths = list(graph_tensors.get("paths", []) or [])
+ paths_by_event_id = dict(graph_tensors.get("paths_by_event_id", {}) or {})
+ for candidate_event_ids in candidate_event_ids_batch:
+ candidate_paths = (
+ [path for event_id in candidate_event_ids for path in list(paths_by_event_id.get(event_id, []) or [])]
+ if paths_by_event_id
+ else _candidate_paths_for_event_ids(
+ all_paths,
+ candidate_event_ids,
+ node_id_to_index=node_id_to_index,
+ )
+ )
+ candidate_paths_batch.append(candidate_paths)
+ candidate_path_ids = [clean_text(path.get("id", "")) for path in candidate_paths]
+ candidate_path_event_ids = [clean_text(path.get("event_id", "")) for path in candidate_paths]
+ candidate_path_support_node_ids = [_path_support_node_id(path) for path in candidate_paths]
+ candidate_path_ids_batch.append(candidate_path_ids)
+ candidate_path_event_ids_batch.append(candidate_path_event_ids)
+ candidate_path_support_node_ids_batch.append(candidate_path_support_node_ids)
+ candidate_path_lengths.append(len(candidate_path_ids))
+ max_path_count = max(max_path_count, len(candidate_path_ids))
+ candidate_path_event_indices_batch = [
+ [
+ candidate_event_index_lookup_batch[batch_index].get(event_id, -1)
+ for event_id in candidate_path_event_ids_batch[batch_index]
+ ]
+ for batch_index in range(batch_size)
+ ]
+ candidate_path_recall_indices_batch = [
+ [
+ candidate_event_recall_indices_batch[batch_index][event_index]
+ if 0 <= event_index < len(candidate_event_recall_indices_batch[batch_index])
+ else -1
+ for event_index in candidate_path_event_indices_batch[batch_index]
+ ]
+ for batch_index in range(batch_size)
+ ]
+
+ base_event_logits_batch = node_hidden.new_zeros((batch_size, max_event_count))
+ rerank_event_logits_batch = node_hidden.new_zeros((batch_size, max_event_count))
+ distractor_event_logits_batch = node_hidden.new_zeros((batch_size, max_event_count))
+ calibrated_event_logits_batch = node_hidden.new_zeros((batch_size, max_event_count))
+ final_event_logits_batch = node_hidden.new_zeros((batch_size, max_event_count))
+ matrix_event_delta_batch = node_hidden.new_zeros((batch_size, max_event_count))
+ event_distractor_logits_batch = node_hidden.new_zeros((batch_size, max_event_count))
+ event_distractor_delta_batch = node_hidden.new_zeros((batch_size, max_event_count))
+ event_calibration_delta_batch = node_hidden.new_zeros((batch_size, max_event_count))
+ event_tunnel_support_logits_batch = node_hidden.new_zeros((batch_size, max_event_count))
+ event_tunnel_delta_batch = node_hidden.new_zeros((batch_size, max_event_count))
+ event_fusion_delta_batch = node_hidden.new_zeros((batch_size, max_event_count))
+ event_mask = torch.zeros((batch_size, max_event_count), dtype=torch.bool, device=device)
+ event_support_embeddings_batch = node_hidden.new_zeros((batch_size, max_event_count, MESSAGE_HIDDEN_DIM))
+ event_support_slot_embeddings_batch = node_hidden.new_zeros(
+ (batch_size, max_event_count, len(EVENT_SUPPORT_PATH_TYPE_ORDER), MESSAGE_HIDDEN_DIM)
+ )
+ event_support_slot_mask_batch = torch.zeros(
+ (batch_size, max_event_count, len(EVENT_SUPPORT_PATH_TYPE_ORDER)),
+ dtype=torch.bool,
+ device=device,
+ )
+ pooled_event_embeddings = node_hidden.new_zeros((batch_size, MESSAGE_HIDDEN_DIM))
+ pooled_support_embeddings = node_hidden.new_zeros((batch_size, MESSAGE_HIDDEN_DIM))
+ matrix_event_ids_batch: List[List[str]] = [[] for _ in range(batch_size)]
+ matrix_event_lengths: List[int] = [0 for _ in range(batch_size)]
+ matrix_local_index_batch: List[List[int]] = [[] for _ in range(batch_size)]
+ event_relation_strength_batch = node_hidden.new_zeros((batch_size, max_event_count, max_event_count))
+ event_pair_features_batch_raw = node_hidden.new_zeros((batch_size, max_event_count, PAIR_FEATURE_DIM))
+ event_pair_features_batch = event_pair_features_batch_raw
+ if max_event_count > 0:
+ padded_event_indices = torch.zeros((batch_size, max_event_count), dtype=torch.long, device=device)
+ for batch_index, candidate_event_ids in enumerate(candidate_event_ids_batch):
+ if not candidate_event_ids:
+ continue
+ event_index_values = [node_id_to_index[event_id] for event_id in candidate_event_ids]
+ _validate_index_values(
+ event_index_values,
+ upper_bound=int(node_hidden.size(0)),
+ index_name="candidate_event_index",
+ context="LocomoNodeMemoryModel.score_examples_with_graph_encoding",
+ extra={
+ "stage": "event",
+ "conversation_id": examples[batch_index].conversation_id,
+ "question_id": examples[batch_index].question_id,
+ "candidate_event_count": len(candidate_event_ids),
+ },
+ )
+ indices = torch.tensor(event_index_values, dtype=torch.long, device=device)
+ padded_event_indices[batch_index, : indices.numel()] = indices
+ event_mask[batch_index, : indices.numel()] = True
+ recall_indices_for_candidates = candidate_event_recall_indices_batch[batch_index]
+ reusable_positions = [
+ position for position, recall_index in enumerate(recall_indices_for_candidates) if int(recall_index) >= 0
+ ]
+ if reusable_positions:
+ reusable_recall_indices = torch.tensor(
+ [int(recall_indices_for_candidates[position]) for position in reusable_positions],
+ dtype=torch.long,
+ device=device,
+ )
+ reusable_feature_positions = torch.tensor(
+ reusable_positions,
+ dtype=torch.long,
+ device=device,
+ )
+ event_pair_features_batch_raw[batch_index, : indices.numel()].index_copy_(
+ 0,
+ reusable_feature_positions,
+ recall_pair_features_batch[batch_index].index_select(0, reusable_recall_indices),
+ )
+ missing_positions = [
+ position for position, recall_index in enumerate(recall_indices_for_candidates) if int(recall_index) < 0
+ ]
+ if missing_positions:
+ missing_feature_positions = torch.tensor(
+ missing_positions,
+ dtype=torch.long,
+ device=device,
+ )
+ event_pair_features_batch_raw[batch_index, : indices.numel()].index_copy_(
+ 0,
+ missing_feature_positions,
+ torch.tensor(
+ [
+ _event_pair_feature_values(
+ examples[batch_index].question,
+ question_features_batch[batch_index],
+ node_lookup.get(candidate_event_ids[position], {}),
+ support_payload=event_support_lookup.get(candidate_event_ids[position], {}),
+ prepared_question=prepared_question_batch[batch_index],
+ prepared_event=event_scoring_features_by_id.get(candidate_event_ids[position]),
+ )
+ for position in missing_positions
+ ],
+ dtype=node_hidden.dtype,
+ device=device,
+ ),
+ )
+ relation_features = _event_relation_feature_matrix(
+ [
+ event_scoring_features_by_id.get(event_id, {})
+ for event_id in candidate_event_ids
+ ],
+ dtype=node_hidden.dtype,
+ device=device,
+ )
+ if relation_features.numel() > 0:
+ event_relation_strength_batch[batch_index, : indices.numel(), : indices.numel()] = _relation_strength_matrix(
+ relation_features,
+ TRI_MAZE_EVENT_RELATION_WEIGHTS,
+ )
+ event_pair_features_batch_raw = _apply_event_pair_feature_mode(
+ event_pair_features_batch_raw,
+ resolved_event_pair_feature_mode,
+ )
+ event_pair_features_batch = self.event_pair_adapter(event_pair_features_batch_raw)
+ gathered_event_embeddings = node_hidden.index_select(0, padded_event_indices.reshape(-1)).view(
+ batch_size,
+ max_event_count,
+ -1,
+ )
+ raw_event_support_embeddings_batch = _aggregate_event_support_embeddings(
+ node_hidden,
+ node_id_to_index,
+ candidate_event_ids_batch,
+ candidate_paths_batch=candidate_paths_batch,
+ support_node_ids_by_event=dict(graph_tensors.get("support_node_ids_by_event", {}) or {}),
+ support_node_ids_by_event_and_type=dict(graph_tensors.get("support_node_ids_by_event_and_type", {}) or {}),
+ question_features_batch=question_features_batch,
+ )
+ event_support_slot_embeddings_batch, event_support_slot_mask_batch = _aggregate_event_support_slot_embeddings(
+ node_hidden,
+ node_id_to_index,
+ candidate_event_ids_batch,
+ candidate_paths_batch=candidate_paths_batch,
+ support_node_ids_by_event_and_type=dict(
+ graph_tensors.get("support_node_ids_by_event_and_type", {}) or {}
+ ),
+ )
+ gathered_event_embeddings, event_support_embeddings_batch = self.event_subgraph_refiner(
+ projected_batch,
+ gathered_event_embeddings,
+ event_support_slot_embeddings_batch,
+ event_support_slot_mask_batch,
+ event_mask,
+ )
+ missing_support_mask = ~event_support_slot_mask_batch.any(dim=-1)
+ event_support_embeddings_batch = torch.where(
+ missing_support_mask.unsqueeze(-1),
+ raw_event_support_embeddings_batch,
+ event_support_embeddings_batch,
+ )
+ base_event_logits_batch = self.event_head(
+ question_batch,
+ gathered_event_embeddings,
+ event_support_embeddings_batch,
+ projected_batch,
+ event_pair_features_batch,
+ )
+ rerank_event_logits_batch = base_event_logits_batch
+ if resolved_event_rerank_mode == "matrix":
+ max_matrix_event_count = 0
+ for batch_index, candidate_event_ids in enumerate(candidate_event_ids_batch):
+ recall_ranked_event_ids = []
+ candidate_event_id_set = {clean_text(item) for item in candidate_event_ids if clean_text(item)}
+ if recall_event_lengths[batch_index] > 0:
+ ranked_indices = torch.argsort(
+ recall_event_logits_batch[batch_index, : recall_event_lengths[batch_index]].detach(),
+ descending=True,
+ )
+ recall_ranked_event_ids = [
+ clean_text(recall_event_ids_batch[batch_index][int(index)])
+ for index in ranked_indices.tolist()
+ if clean_text(recall_event_ids_batch[batch_index][int(index)]) in candidate_event_id_set
+ ]
+ matrix_event_ids = _select_matrix_event_ids(
+ candidate_event_ids,
+ recall_ranked_event_ids,
+ positive_event_ids=examples[batch_index].positive_event_ids,
+ hard_negative_event_ids=_resolved_hard_negative_event_ids_for_example(examples[batch_index]),
+ matrix_top_k=resolved_matrix_event_top_k,
+ )
+ matrix_event_ids_batch[batch_index] = list(matrix_event_ids)
+ matrix_event_lengths[batch_index] = len(matrix_event_ids)
+ max_matrix_event_count = max(max_matrix_event_count, len(matrix_event_ids))
+ candidate_index_lookup = {event_id: index for index, event_id in enumerate(candidate_event_ids)}
+ matrix_local_index_batch[batch_index] = [
+ candidate_index_lookup[event_id] for event_id in matrix_event_ids if event_id in candidate_index_lookup
+ ]
+ _validate_index_values(
+ matrix_local_index_batch[batch_index],
+ upper_bound=len(candidate_event_ids),
+ index_name="matrix_event_local_index",
+ context="LocomoNodeMemoryModel.score_examples_with_graph_encoding",
+ extra={
+ "stage": "event_matrix",
+ "conversation_id": examples[batch_index].conversation_id,
+ "question_id": examples[batch_index].question_id,
+ "matrix_event_count": len(matrix_event_ids),
+ "candidate_event_count": len(candidate_event_ids),
+ },
+ )
+ if max_matrix_event_count > 0:
+ matrix_event_embeddings = node_hidden.new_zeros((batch_size, max_matrix_event_count, MESSAGE_HIDDEN_DIM))
+ matrix_support_embeddings = node_hidden.new_zeros((batch_size, max_matrix_event_count, MESSAGE_HIDDEN_DIM))
+ matrix_pair_features = node_hidden.new_zeros((batch_size, max_matrix_event_count, PAIR_FEATURE_DIM))
+ matrix_relation_features = node_hidden.new_zeros(
+ (batch_size, max_matrix_event_count, max_matrix_event_count, MATRIX_RELATION_FEATURE_DIM)
+ )
+ matrix_event_mask = torch.zeros((batch_size, max_matrix_event_count), dtype=torch.bool, device=device)
+ for batch_index, local_indices in enumerate(matrix_local_index_batch):
+ if not local_indices:
+ continue
+ local_index_tensor = torch.tensor(local_indices, dtype=torch.long, device=device)
+ matrix_count = local_index_tensor.numel()
+ matrix_event_mask[batch_index, :matrix_count] = True
+ matrix_event_embeddings[batch_index, :matrix_count] = gathered_event_embeddings[batch_index].index_select(0, local_index_tensor)
+ matrix_support_embeddings[batch_index, :matrix_count] = event_support_embeddings_batch[batch_index].index_select(0, local_index_tensor)
+ matrix_pair_features[batch_index, :matrix_count] = event_pair_features_batch[batch_index].index_select(0, local_index_tensor)
+ matrix_node_features = [
+ event_scoring_features_by_id.get(event_id, {})
+ for event_id in matrix_event_ids_batch[batch_index]
+ ]
+ matrix_relation_features[batch_index, :matrix_count, :matrix_count] = _event_relation_feature_matrix(
+ matrix_node_features,
+ dtype=node_hidden.dtype,
+ device=device,
+ )
+ matrix_delta_logits = self.event_matrix_head(
+ question_batch,
+ matrix_event_embeddings,
+ matrix_support_embeddings,
+ projected_batch,
+ matrix_pair_features,
+ matrix_relation_features,
+ matrix_event_mask,
+ )
+ for batch_index, local_indices in enumerate(matrix_local_index_batch):
+ if not local_indices:
+ continue
+ local_index_tensor = torch.tensor(local_indices, dtype=torch.long, device=device)
+ matrix_count = int(local_index_tensor.numel())
+ matrix_event_delta_batch[batch_index].index_copy_(
+ 0,
+ local_index_tensor,
+ MATRIX_EVENT_DELTA_LOGIT_LIMIT
+ * torch.tanh(
+ matrix_delta_logits[batch_index, :matrix_count].to(
+ dtype=matrix_event_delta_batch.dtype
+ )
+ / MATRIX_EVENT_DELTA_LOGIT_LIMIT
+ ),
+ )
+ rerank_event_logits_batch = base_event_logits_batch + matrix_event_delta_batch
+ rerank_event_prob_batch = torch.sigmoid(rerank_event_logits_batch).masked_fill(~event_mask, 0.0)
+ event_distractor_features_batch = node_hidden.new_zeros(
+ (batch_size, max_event_count, EVENT_DISTRACTOR_FEATURE_DIM)
+ )
+ for batch_index, candidate_event_ids in enumerate(candidate_event_ids_batch):
+ if not candidate_event_ids:
+ continue
+ event_count = candidate_event_lengths[batch_index]
+ recall_scores_for_candidates = _gather_scores_by_indices(
+ recall_event_prob_batch[batch_index, : recall_event_lengths[batch_index]],
+ candidate_event_recall_indices_batch[batch_index],
+ device=device,
+ )
+ base_event_probs = torch.sigmoid(base_event_logits_batch[batch_index, :event_count])
+ rerank_event_probs = rerank_event_prob_batch[batch_index, :event_count]
+ support_slot_cover = (
+ event_support_slot_mask_batch[batch_index, :event_count]
+ .sum(dim=-1)
+ .to(dtype=node_hidden.dtype)
+ / max(1.0, float(len(EVENT_SUPPORT_PATH_TYPE_ORDER)))
+ )
+ is_temporal_tensor = node_hidden.new_full(
+ (event_count, 1),
+ 1.0 if bool(question_features_batch[batch_index].get("is_temporal", False)) else 0.0,
+ )
+ event_pair_slice = event_pair_features_batch[batch_index, :event_count]
+ event_distractor_features_batch[batch_index, :event_count] = torch.cat(
+ [
+ recall_scores_for_candidates.unsqueeze(-1),
+ base_event_probs.unsqueeze(-1),
+ rerank_event_probs.unsqueeze(-1),
+ (rerank_event_probs - _best_other_scores(rerank_event_probs)).unsqueeze(-1),
+ (recall_scores_for_candidates - _best_other_scores(recall_scores_for_candidates)).unsqueeze(-1),
+ _rank_ratio(rerank_event_probs).unsqueeze(-1),
+ support_slot_cover.unsqueeze(-1),
+ is_temporal_tensor,
+ event_pair_slice[:, [1, 5, 6, 14, 15, 25, 26, 27, 28]],
+ ],
+ dim=-1,
+ )
+ event_distractor_logits_batch = self.event_distractor_head(
+ question_batch,
+ gathered_event_embeddings,
+ event_support_embeddings_batch,
+ projected_batch,
+ event_pair_features_batch,
+ event_distractor_features_batch,
+ ).masked_fill(~event_mask, 0.0)
+ event_distractor_delta_batch = ((0.5 - torch.sigmoid(event_distractor_logits_batch)) * event_mask.to(
+ dtype=node_hidden.dtype
+ ))
+ distractor_event_logits_batch = rerank_event_logits_batch + event_distractor_delta_batch
+ calibrated_event_logits_batch = distractor_event_logits_batch
+ masked_event_logits = distractor_event_logits_batch.masked_fill(~event_mask, float("-inf"))
+ top_k = min(3, max_event_count)
+ top_indices = torch.topk(masked_event_logits, k=top_k, dim=1).indices
+ selected_event_embeddings = torch.gather(
+ gathered_event_embeddings,
+ 1,
+ top_indices.unsqueeze(-1).expand(-1, -1, gathered_event_embeddings.size(-1)),
+ )
+ selected_support_embeddings = torch.gather(
+ event_support_embeddings_batch,
+ 1,
+ top_indices.unsqueeze(-1).expand(-1, -1, event_support_embeddings_batch.size(-1)),
+ )
+ selected_event_mask = torch.gather(event_mask, 1, top_indices)
+ pooled_event_embeddings = (
+ (selected_event_embeddings + selected_support_embeddings) * 0.5
+ * selected_event_mask.unsqueeze(-1).to(dtype=gathered_event_embeddings.dtype)
+ ).sum(dim=1) / selected_event_mask.sum(dim=1, keepdim=True).clamp_min(1).to(dtype=gathered_event_embeddings.dtype)
+ pooled_support_embeddings = (
+ selected_support_embeddings
+ * selected_event_mask.unsqueeze(-1).to(dtype=gathered_event_embeddings.dtype)
+ ).sum(dim=1) / selected_event_mask.sum(dim=1, keepdim=True).clamp_min(1).to(dtype=gathered_event_embeddings.dtype)
+
+ base_path_logits_batch = node_hidden.new_zeros((batch_size, max_path_count))
+ rerank_path_logits_batch = node_hidden.new_zeros((batch_size, max_path_count))
+ calibrated_path_logits_batch = node_hidden.new_zeros((batch_size, max_path_count))
+ final_path_logits_batch = node_hidden.new_zeros((batch_size, max_path_count))
+ matrix_path_delta_batch = node_hidden.new_zeros((batch_size, max_path_count))
+ path_calibration_delta_batch = node_hidden.new_zeros((batch_size, max_path_count))
+ path_tunnel_support_logits_batch = node_hidden.new_zeros((batch_size, max_path_count))
+ path_tunnel_delta_batch = node_hidden.new_zeros((batch_size, max_path_count))
+ path_fusion_delta_batch = node_hidden.new_zeros((batch_size, max_path_count))
+ path_mask = torch.zeros((batch_size, max_path_count), dtype=torch.bool, device=device)
+ path_event_embeddings = node_hidden.new_zeros((batch_size, max_path_count, MESSAGE_HIDDEN_DIM))
+ path_support_embeddings = node_hidden.new_zeros((batch_size, max_path_count, MESSAGE_HIDDEN_DIM))
+ matrix_path_ids_batch: List[List[str]] = [[] for _ in range(batch_size)]
+ matrix_path_lengths: List[int] = [0 for _ in range(batch_size)]
+ matrix_path_local_index_batch: List[List[int]] = [[] for _ in range(batch_size)]
+ path_relation_strength_batch = node_hidden.new_zeros((batch_size, max_path_count, max_path_count))
+ path_pair_features_batch_raw = node_hidden.new_zeros((batch_size, max_path_count, PATH_PAIR_FEATURE_DIM))
+ path_pair_features_batch = path_pair_features_batch_raw
+ if max_path_count > 0:
+ padded_event_path_indices = torch.zeros((batch_size, max_path_count), dtype=torch.long, device=device)
+ padded_support_indices = torch.zeros((batch_size, max_path_count), dtype=torch.long, device=device)
+ padded_path_type_ids = torch.zeros((batch_size, max_path_count), dtype=torch.long, device=device)
+ for batch_index, candidate_paths in enumerate(candidate_paths_batch):
+ if not candidate_paths:
+ continue
+ event_index_values = [node_id_to_index[clean_text(path.get("event_id", ""))] for path in candidate_paths]
+ support_index_values = [node_id_to_index[_path_support_node_id(path)] for path in candidate_paths]
+ _validate_index_values(
+ event_index_values,
+ upper_bound=int(node_hidden.size(0)),
+ index_name="path_event_index",
+ context="LocomoNodeMemoryModel.score_examples_with_graph_encoding",
+ extra={
+ "stage": "path",
+ "conversation_id": examples[batch_index].conversation_id,
+ "question_id": examples[batch_index].question_id,
+ "candidate_path_count": len(candidate_paths),
+ },
+ )
+ _validate_index_values(
+ support_index_values,
+ upper_bound=int(node_hidden.size(0)),
+ index_name="path_support_index",
+ context="LocomoNodeMemoryModel.score_examples_with_graph_encoding",
+ extra={
+ "stage": "path",
+ "conversation_id": examples[batch_index].conversation_id,
+ "question_id": examples[batch_index].question_id,
+ "candidate_path_count": len(candidate_paths),
+ },
+ )
+ event_indices = torch.tensor(
+ event_index_values,
+ dtype=torch.long,
+ device=device,
+ )
+ support_indices = torch.tensor(
+ support_index_values,
+ dtype=torch.long,
+ device=device,
+ )
+ path_type_ids = torch.tensor(
+ [PATH_TYPE_TO_ID.get(_path_type(path), 0) for path in candidate_paths],
+ dtype=torch.long,
+ device=device,
+ )
+ padded_event_path_indices[batch_index, : event_indices.numel()] = event_indices
+ padded_support_indices[batch_index, : support_indices.numel()] = support_indices
+ padded_path_type_ids[batch_index, : path_type_ids.numel()] = path_type_ids
+ path_pair_features_batch_raw[batch_index, : path_type_ids.numel()] = torch.tensor(
+ [
+ _path_pair_feature_values(
+ examples[batch_index].question,
+ question_features_batch[batch_index],
+ node_lookup.get(clean_text(path.get("event_id", "")), {}),
+ node_lookup.get(_path_support_node_id(path), {}),
+ path_type=_path_type(path),
+ prepared_question=prepared_question_batch[batch_index],
+ prepared_event=event_scoring_features_by_id.get(clean_text(path.get("event_id", ""))),
+ prepared_support=node_scoring_features_by_id.get(_path_support_node_id(path)),
+ )
+ for path in candidate_paths
+ ],
+ dtype=node_hidden.dtype,
+ device=device,
+ )
+ path_relation_features = _path_relation_feature_matrix(
+ [
+ path_scoring_features_by_id.get(path_id, {})
+ for path_id in candidate_path_ids_batch[batch_index]
+ ],
+ dtype=node_hidden.dtype,
+ device=device,
+ )
+ if path_relation_features.numel() > 0:
+ path_relation_strength_batch[batch_index, : path_type_ids.numel(), : path_type_ids.numel()] = _relation_strength_matrix(
+ path_relation_features,
+ TRI_MAZE_PATH_RELATION_WEIGHTS,
+ )
+ path_mask[batch_index, : path_type_ids.numel()] = True
+ path_pair_features_batch = self.path_pair_adapter(path_pair_features_batch_raw)
+ path_event_embeddings = node_hidden.new_zeros((batch_size, max_path_count, MESSAGE_HIDDEN_DIM))
+ for batch_index, event_indices in enumerate(candidate_path_event_indices_batch):
+ if not event_indices:
+ continue
+ local_event_index_tensor = torch.tensor(event_indices, dtype=torch.long, device=device)
+ _validate_index_values(
+ local_event_index_tensor.detach().cpu().tolist(),
+ upper_bound=int(candidate_event_lengths[batch_index]),
+ index_name="path_event_local_index",
+ context="LocomoNodeMemoryModel.score_examples_with_graph_encoding",
+ extra={
+ "stage": "path_event_refined",
+ "conversation_id": examples[batch_index].conversation_id,
+ "question_id": examples[batch_index].question_id,
+ "candidate_path_count": len(event_indices),
+ "candidate_event_count": int(candidate_event_lengths[batch_index]),
+ },
+ )
+ path_event_embeddings[batch_index, : local_event_index_tensor.numel()] = gathered_event_embeddings[
+ batch_index
+ ].index_select(0, local_event_index_tensor)
+ path_support_embeddings = node_hidden.index_select(0, padded_support_indices.reshape(-1)).view(batch_size, max_path_count, -1)
+ base_path_logits_batch = self.path_head(
+ question_batch,
+ path_event_embeddings,
+ path_support_embeddings,
+ projected_batch,
+ padded_path_type_ids,
+ path_pair_features_batch,
+ )
+ base_path_logits_batch = base_path_logits_batch.masked_fill(~path_mask, 0.0)
+ rerank_path_logits_batch = base_path_logits_batch
+ max_matrix_path_count = 0
+ for batch_index, candidate_paths in enumerate(candidate_paths_batch):
+ if not candidate_paths:
+ continue
+ ranked_indices = torch.argsort(base_path_logits_batch[batch_index, : candidate_path_lengths[batch_index]].detach(), descending=True)
+ ranked_path_ids = [
+ candidate_path_ids_batch[batch_index][int(index)]
+ for index in ranked_indices.tolist()
+ if int(index) < len(candidate_path_ids_batch[batch_index])
+ ]
+ hard_negative_path_ids = _hard_negative_path_ids(examples[batch_index], candidate_path_ids_batch[batch_index])
+ matrix_path_ids = _select_matrix_path_ids(
+ candidate_path_ids_batch[batch_index],
+ ranked_path_ids,
+ positive_path_ids=examples[batch_index].positive_path_ids,
+ hard_negative_path_ids=hard_negative_path_ids,
+ matrix_top_k=DEFAULT_MATRIX_PATH_TOP_K,
+ )
+ matrix_path_ids_batch[batch_index] = list(matrix_path_ids)
+ matrix_path_lengths[batch_index] = len(matrix_path_ids)
+ max_matrix_path_count = max(max_matrix_path_count, len(matrix_path_ids))
+ candidate_index_lookup = {
+ path_id: index for index, path_id in enumerate(candidate_path_ids_batch[batch_index])
+ }
+ matrix_path_local_index_batch[batch_index] = [
+ candidate_index_lookup[path_id] for path_id in matrix_path_ids if path_id in candidate_index_lookup
+ ]
+ _validate_index_values(
+ matrix_path_local_index_batch[batch_index],
+ upper_bound=len(candidate_path_ids_batch[batch_index]),
+ index_name="matrix_path_local_index",
+ context="LocomoNodeMemoryModel.score_examples_with_graph_encoding",
+ extra={
+ "stage": "path_matrix",
+ "conversation_id": examples[batch_index].conversation_id,
+ "question_id": examples[batch_index].question_id,
+ "matrix_path_count": len(matrix_path_ids),
+ "candidate_path_count": len(candidate_path_ids_batch[batch_index]),
+ },
+ )
+ if max_matrix_path_count > 0:
+ matrix_path_event_embeddings = node_hidden.new_zeros((batch_size, max_matrix_path_count, MESSAGE_HIDDEN_DIM))
+ matrix_path_support_embeddings = node_hidden.new_zeros((batch_size, max_matrix_path_count, MESSAGE_HIDDEN_DIM))
+ matrix_path_pair_features = node_hidden.new_zeros((batch_size, max_matrix_path_count, PATH_PAIR_FEATURE_DIM))
+ matrix_path_relation_features = node_hidden.new_zeros(
+ (batch_size, max_matrix_path_count, max_matrix_path_count, PATH_MATRIX_RELATION_FEATURE_DIM)
+ )
+ matrix_path_mask = torch.zeros((batch_size, max_matrix_path_count), dtype=torch.bool, device=device)
+ for batch_index, local_indices in enumerate(matrix_path_local_index_batch):
+ if not local_indices:
+ continue
+ local_index_tensor = torch.tensor(local_indices, dtype=torch.long, device=device)
+ matrix_count = local_index_tensor.numel()
+ matrix_path_mask[batch_index, :matrix_count] = True
+ matrix_path_event_embeddings[batch_index, :matrix_count] = path_event_embeddings[batch_index].index_select(0, local_index_tensor)
+ matrix_path_support_embeddings[batch_index, :matrix_count] = path_support_embeddings[batch_index].index_select(0, local_index_tensor)
+ matrix_path_pair_features[batch_index, :matrix_count] = path_pair_features_batch[batch_index].index_select(0, local_index_tensor)
+ matrix_path_features = [
+ path_scoring_features_by_id.get(path_id, {})
+ for path_id in matrix_path_ids_batch[batch_index]
+ ]
+ matrix_path_relation_features[batch_index, :matrix_count, :matrix_count] = _path_relation_feature_matrix(
+ matrix_path_features,
+ dtype=node_hidden.dtype,
+ device=device,
+ )
+ matrix_path_delta_logits = self.path_matrix_head(
+ question_batch,
+ matrix_path_event_embeddings,
+ matrix_path_support_embeddings,
+ projected_batch,
+ matrix_path_pair_features,
+ matrix_path_relation_features,
+ matrix_path_mask,
+ )
+ for batch_index, local_indices in enumerate(matrix_path_local_index_batch):
+ if not local_indices:
+ continue
+ local_index_tensor = torch.tensor(local_indices, dtype=torch.long, device=device)
+ matrix_count = int(local_index_tensor.numel())
+ matrix_path_delta_batch[batch_index].index_copy_(
+ 0,
+ local_index_tensor,
+ MATRIX_PATH_DELTA_LOGIT_LIMIT
+ * torch.tanh(
+ matrix_path_delta_logits[batch_index, :matrix_count].to(
+ dtype=matrix_path_delta_batch.dtype
+ )
+ / MATRIX_PATH_DELTA_LOGIT_LIMIT
+ ),
+ )
+ rerank_path_logits_batch = (base_path_logits_batch + matrix_path_delta_batch).masked_fill(~path_mask, 0.0)
+ calibrated_path_logits_batch = rerank_path_logits_batch
+
+ temporal_paths_by_conversation = {
+ clean_text(conversation_id): [dict(path) for path in list(paths or [])]
+ for conversation_id, paths in dict(graph_tensors.get("temporal_paths_by_conversation", {}) or {}).items()
+ if clean_text(conversation_id)
+ }
+ all_temporal_paths = [dict(path) for path in list(graph_tensors.get("temporal_paths", []) or [])]
+ temporal_paths_batch: List[List[Dict[str, Any]]] = []
+ candidate_temporal_path_ids_batch: List[List[str]] = []
+ candidate_temporal_node_ids_batch: List[List[str]] = []
+ candidate_temporal_event_ids_batch: List[List[str]] = []
+ temporal_lengths: List[int] = []
+ max_temporal_count = 0
+ for example in examples:
+ temporal_paths = list(temporal_paths_by_conversation.get(example.conversation_id, all_temporal_paths))
+ temporal_paths_batch.append(temporal_paths)
+ candidate_temporal_path_ids = [clean_text(path.get("id", "")) for path in temporal_paths]
+ candidate_temporal_node_ids = [_path_support_node_id(path) for path in temporal_paths]
+ candidate_temporal_event_ids = [clean_text(path.get("event_id", "")) for path in temporal_paths]
+ candidate_temporal_path_ids_batch.append(candidate_temporal_path_ids)
+ candidate_temporal_node_ids_batch.append(candidate_temporal_node_ids)
+ candidate_temporal_event_ids_batch.append(candidate_temporal_event_ids)
+ temporal_lengths.append(len(temporal_paths))
+ max_temporal_count = max(max_temporal_count, len(temporal_paths))
+ candidate_temporal_event_indices_batch = [
+ [
+ candidate_event_index_lookup_batch[batch_index].get(event_id, -1)
+ for event_id in candidate_temporal_event_ids_batch[batch_index]
+ ]
+ for batch_index in range(batch_size)
+ ]
+ temporal_node_index_lookup_batch = [
+ {node_id: index for index, node_id in enumerate(candidate_temporal_node_ids)}
+ for candidate_temporal_node_ids in candidate_temporal_node_ids_batch
+ ]
+ candidate_path_temporal_support_indices_batch = [
+ [
+ temporal_node_index_lookup_batch[batch_index].get(support_node_id, -1)
+ for support_node_id in candidate_path_support_node_ids_batch[batch_index]
+ ]
+ for batch_index in range(batch_size)
+ ]
+ temporal_logits_batch = node_hidden.new_zeros((batch_size, max_temporal_count))
+ temporal_mask = torch.zeros((batch_size, max_temporal_count), dtype=torch.bool, device=device)
+ if max_temporal_count > 0:
+ padded_temporal_event_indices = torch.zeros((batch_size, max_temporal_count), dtype=torch.long, device=device)
+ padded_temporal_time_indices = torch.zeros((batch_size, max_temporal_count), dtype=torch.long, device=device)
+ for batch_index, temporal_paths in enumerate(temporal_paths_batch):
+ if not temporal_paths:
+ continue
+ temporal_event_index_values = [node_id_to_index[clean_text(path.get("event_id", ""))] for path in temporal_paths]
+ temporal_time_index_values = [node_id_to_index[_path_support_node_id(path)] for path in temporal_paths]
+ _validate_index_values(
+ temporal_event_index_values,
+ upper_bound=int(node_hidden.size(0)),
+ index_name="temporal_event_index",
+ context="LocomoNodeMemoryModel.score_examples_with_graph_encoding",
+ extra={
+ "stage": "temporal",
+ "conversation_id": examples[batch_index].conversation_id,
+ "question_id": examples[batch_index].question_id,
+ "temporal_path_count": len(temporal_paths),
+ },
+ )
+ _validate_index_values(
+ temporal_time_index_values,
+ upper_bound=int(node_hidden.size(0)),
+ index_name="temporal_time_index",
+ context="LocomoNodeMemoryModel.score_examples_with_graph_encoding",
+ extra={
+ "stage": "temporal",
+ "conversation_id": examples[batch_index].conversation_id,
+ "question_id": examples[batch_index].question_id,
+ "temporal_path_count": len(temporal_paths),
+ },
+ )
+ temporal_event_indices = torch.tensor(
+ temporal_event_index_values,
+ dtype=torch.long,
+ device=device,
+ )
+ temporal_time_indices = torch.tensor(
+ temporal_time_index_values,
+ dtype=torch.long,
+ device=device,
+ )
+ temporal_count = int(temporal_event_indices.numel())
+ padded_temporal_event_indices[batch_index, :temporal_count] = temporal_event_indices
+ padded_temporal_time_indices[batch_index, :temporal_count] = temporal_time_indices
+ temporal_mask[batch_index, :temporal_count] = True
+ temporal_event_embeddings = node_hidden.index_select(0, padded_temporal_event_indices.reshape(-1)).view(
+ batch_size,
+ max_temporal_count,
+ -1,
+ )
+ temporal_time_embeddings = node_hidden.index_select(0, padded_temporal_time_indices.reshape(-1)).view(
+ batch_size,
+ max_temporal_count,
+ -1,
+ )
+ temporal_logits_batch = self.temporal_head(
+ question_batch,
+ temporal_event_embeddings,
+ temporal_time_embeddings,
+ projected_batch,
+ ).masked_fill(~temporal_mask, 0.0)
+
+ answer_calibration_features = _build_answer_calibration_features(
+ calibrated_event_logits_batch=calibrated_event_logits_batch,
+ calibrated_path_logits_batch=calibrated_path_logits_batch,
+ temporal_logits_batch=temporal_logits_batch,
+ event_pair_features_batch_raw=event_pair_features_batch_raw,
+ path_pair_features_batch_raw=path_pair_features_batch_raw,
+ event_relation_strength_batch=event_relation_strength_batch,
+ path_relation_strength_batch=path_relation_strength_batch,
+ event_mask=event_mask,
+ path_mask=path_mask,
+ temporal_mask=temporal_mask,
+ examples=examples,
+ candidate_path_event_indices_batch=candidate_path_event_indices_batch,
+ candidate_temporal_event_indices_batch=candidate_temporal_event_indices_batch,
+ )
+ answer_type_logits_batch = self.answer_type_head(
+ question_batch,
+ pooled_event_embeddings,
+ pooled_support_embeddings,
+ projected_batch,
+ answer_calibration_features,
+ )
+ answer_type_probs_batch = F.softmax(answer_type_logits_batch, dim=-1)
+ base_path_prob_batch = (
+ torch.sigmoid(base_path_logits_batch)
+ if max_path_count > 0
+ else node_hidden.new_zeros((batch_size, 0))
+ )
+ rerank_path_prob_batch = (
+ torch.sigmoid(rerank_path_logits_batch).masked_fill(~path_mask, 0.0)
+ if max_path_count > 0
+ else node_hidden.new_zeros((batch_size, 0))
+ )
+ temporal_prob_batch = (
+ torch.sigmoid(temporal_logits_batch).masked_fill(~temporal_mask, 0.0)
+ if max_temporal_count > 0
+ else node_hidden.new_zeros((batch_size, 0))
+ )
+ if max_event_count > 0:
+ event_runtime_features_batch = node_hidden.new_zeros(
+ (batch_size, max_event_count, EVENT_RUNTIME_CALIBRATION_FEATURE_DIM)
+ )
+ for batch_index, candidate_event_ids in enumerate(candidate_event_ids_batch):
+ if not candidate_event_ids:
+ continue
+ event_count = candidate_event_lengths[batch_index]
+ recall_scores_for_candidates = _gather_scores_by_indices(
+ recall_event_prob_batch[batch_index, : recall_event_lengths[batch_index]],
+ candidate_event_recall_indices_batch[batch_index],
+ device=device,
+ )
+ distractor_event_probs = torch.sigmoid(distractor_event_logits_batch[batch_index, :event_count])
+ path_event_indices = candidate_path_event_indices_batch[batch_index]
+ max_path_prob_by_event = _scatter_amax_by_indices(
+ rerank_path_prob_batch[batch_index, : candidate_path_lengths[batch_index]],
+ path_event_indices,
+ size=event_count,
+ device=device,
+ )
+ event_path_count = _scatter_sum_by_indices(
+ path_event_indices,
+ size=event_count,
+ device=device,
+ dtype=node_hidden.dtype,
+ )
+ temporal_event_indices = candidate_temporal_event_indices_batch[batch_index]
+ max_temporal_prob_by_event = _scatter_amax_by_indices(
+ temporal_prob_batch[batch_index, : temporal_lengths[batch_index]],
+ temporal_event_indices,
+ size=event_count,
+ device=device,
+ )
+ is_temporal_tensor = node_hidden.new_full(
+ (event_count, 1),
+ 1.0 if bool(question_features_batch[batch_index].get("is_temporal", False)) else 0.0,
+ )
+ event_pair_slice = event_pair_features_batch_raw[batch_index, :event_count]
+ event_reverse_scores, event_boundary_gaps, event_reverse_relations, event_reverse_available, _ = _reverse_competition_features(
+ rerank_event_prob_batch[batch_index, :event_count],
+ event_relation_strength_batch[batch_index, :event_count, :event_count],
+ valid_mask=event_mask[batch_index, :event_count],
+ )
+ answer_features = answer_type_probs_batch[batch_index].unsqueeze(0).expand(event_count, -1).to(dtype=node_hidden.dtype)
+ event_runtime_features_batch[batch_index, :event_count] = torch.cat(
+ [
+ recall_scores_for_candidates.unsqueeze(-1),
+ distractor_event_probs.unsqueeze(-1),
+ max_path_prob_by_event.unsqueeze(-1),
+ max_temporal_prob_by_event.unsqueeze(-1),
+ is_temporal_tensor,
+ (event_path_count / max(1.0, float(len(PATH_TYPES)))).clamp(max=1.0).unsqueeze(-1),
+ event_pair_slice[:, [25, 26, 1, 14, 5, 6, 27]],
+ event_reverse_scores.unsqueeze(-1),
+ event_boundary_gaps.unsqueeze(-1),
+ event_reverse_relations.unsqueeze(-1),
+ event_reverse_available.unsqueeze(-1),
+ answer_features,
+ ],
+ dim=-1,
+ )
+ event_calibration_delta_batch = self.event_calibration_head(
+ question_batch,
+ gathered_event_embeddings,
+ event_support_embeddings_batch,
+ projected_batch,
+ event_pair_features_batch,
+ event_runtime_features_batch,
+ ).masked_fill(~event_mask, 0.0)
+ calibrated_event_logits_batch = rerank_event_logits_batch + event_calibration_delta_batch
+ if max_path_count > 0:
+ calibrated_event_prob_batch = torch.sigmoid(calibrated_event_logits_batch)
+ path_runtime_features_batch = node_hidden.new_zeros(
+ (batch_size, max_path_count, PATH_RUNTIME_CALIBRATION_FEATURE_DIM)
+ )
+ for batch_index, candidate_paths in enumerate(candidate_paths_batch):
+ if not candidate_paths:
+ continue
+ path_count = candidate_path_lengths[batch_index]
+ event_indices = candidate_path_event_indices_batch[batch_index]
+ calibrated_event_scores_for_paths = _gather_scores_by_indices(
+ calibrated_event_prob_batch[batch_index, : candidate_event_lengths[batch_index]],
+ event_indices,
+ device=device,
+ )
+ recall_scores_for_paths = _gather_scores_by_indices(
+ recall_event_prob_batch[batch_index, : recall_event_lengths[batch_index]],
+ candidate_path_recall_indices_batch[batch_index],
+ device=device,
+ )
+ temporal_support_scores = _gather_scores_by_indices(
+ temporal_prob_batch[batch_index, : temporal_lengths[batch_index]],
+ candidate_path_temporal_support_indices_batch[batch_index],
+ device=device,
+ )
+ is_temporal_tensor = node_hidden.new_full(
+ (path_count, 1),
+ 1.0 if bool(question_features_batch[batch_index].get("is_temporal", False)) else 0.0,
+ )
+ path_pair_slice = path_pair_features_batch_raw[batch_index, :path_count]
+ path_reverse_scores, path_boundary_gaps, path_reverse_relations, path_reverse_available, _ = _reverse_competition_features(
+ rerank_path_prob_batch[batch_index, :path_count],
+ path_relation_strength_batch[batch_index, :path_count, :path_count],
+ valid_mask=path_mask[batch_index, :path_count],
+ )
+ answer_features = answer_type_probs_batch[batch_index].unsqueeze(0).expand(path_count, -1).to(dtype=node_hidden.dtype)
+ path_runtime_features_batch[batch_index, :path_count] = torch.cat(
+ [
+ calibrated_event_scores_for_paths.unsqueeze(-1),
+ recall_scores_for_paths.unsqueeze(-1),
+ rerank_path_prob_batch[batch_index, :path_count].unsqueeze(-1),
+ temporal_support_scores.unsqueeze(-1),
+ is_temporal_tensor,
+ path_pair_slice[:, [4, 8, 9, 10, 11, 12, 13, 15, 17, 18, 19, 20, 21]],
+ path_reverse_scores.unsqueeze(-1),
+ path_boundary_gaps.unsqueeze(-1),
+ path_reverse_relations.unsqueeze(-1),
+ path_reverse_available.unsqueeze(-1),
+ answer_features,
+ ],
+ dim=-1,
+ )
+ path_calibration_delta_batch = self.path_calibration_head(
+ question_batch,
+ path_event_embeddings,
+ path_support_embeddings,
+ projected_batch,
+ path_pair_features_batch,
+ path_runtime_features_batch,
+ ).masked_fill(~path_mask, 0.0)
+ calibrated_path_logits_batch = (rerank_path_logits_batch + path_calibration_delta_batch).masked_fill(~path_mask, 0.0)
+ calibrated_event_prob_batch = (
+ torch.sigmoid(calibrated_event_logits_batch)
+ if max_event_count > 0
+ else node_hidden.new_zeros((batch_size, 0))
+ )
+ calibrated_path_prob_batch = (
+ torch.sigmoid(calibrated_path_logits_batch)
+ if max_path_count > 0
+ else node_hidden.new_zeros((batch_size, 0))
+ )
+ if max_event_count > 0:
+ event_tunnel_features_batch = node_hidden.new_zeros(
+ (batch_size, max_event_count, EVENT_TUNNEL_FEATURE_DIM)
+ )
+ for batch_index, candidate_event_ids in enumerate(candidate_event_ids_batch):
+ if not candidate_event_ids:
+ continue
+ event_count = candidate_event_lengths[batch_index]
+ path_event_indices = candidate_path_event_indices_batch[batch_index]
+ max_rerank_path_prob_by_event = _scatter_amax_by_indices(
+ rerank_path_prob_batch[batch_index, : candidate_path_lengths[batch_index]],
+ path_event_indices,
+ size=event_count,
+ device=device,
+ )
+ max_calibrated_path_prob_by_event = _scatter_amax_by_indices(
+ calibrated_path_prob_batch[batch_index, : candidate_path_lengths[batch_index]],
+ path_event_indices,
+ size=event_count,
+ device=device,
+ )
+ event_path_count = _scatter_sum_by_indices(
+ path_event_indices,
+ size=event_count,
+ device=device,
+ dtype=node_hidden.dtype,
+ )
+ temporal_event_indices = candidate_temporal_event_indices_batch[batch_index]
+ max_temporal_prob_by_event = _scatter_amax_by_indices(
+ temporal_prob_batch[batch_index, : temporal_lengths[batch_index]],
+ temporal_event_indices,
+ size=event_count,
+ device=device,
+ )
+ event_tunnel_features_batch[batch_index, :event_count] = _build_event_tunnel_features(
+ prepared_events=[
+ event_scoring_features_by_id.get(event_id, {})
+ for event_id in candidate_event_ids
+ ],
+ calibrated_event_probs=calibrated_event_prob_batch[batch_index, :event_count],
+ max_rerank_path_prob_by_event=max_rerank_path_prob_by_event,
+ max_calibrated_path_prob_by_event=max_calibrated_path_prob_by_event,
+ max_temporal_prob_by_event=max_temporal_prob_by_event,
+ event_path_count=event_path_count,
+ event_relation_strength=event_relation_strength_batch[batch_index, :event_count, :event_count],
+ event_mask=event_mask[batch_index, :event_count],
+ question=examples[batch_index].question,
+ question_features=question_features_batch[batch_index],
+ )
+ event_tunnel_outputs = self.event_tunnel_head(
+ question_batch,
+ gathered_event_embeddings,
+ event_support_embeddings_batch,
+ projected_batch,
+ event_pair_features_batch,
+ event_tunnel_features_batch,
+ ).masked_fill(~event_mask.unsqueeze(-1), 0.0)
+ event_tunnel_support_logits_batch = event_tunnel_outputs[..., 0]
+ event_tunnel_delta_batch = (
+ EVENT_TUNNEL_DELTA_LOGIT_LIMIT
+ * torch.tanh(event_tunnel_outputs[..., 1] / EVENT_TUNNEL_DELTA_LOGIT_LIMIT)
+ ).masked_fill(~event_mask, 0.0)
+ final_event_logits_batch = calibrated_event_logits_batch + event_tunnel_delta_batch
+ answer_plan_prior_logits_batch = node_hidden.new_zeros((batch_size, max_event_count, len(ANSWER_PLAN_OUTPUTS)))
+ if max_event_count > 0:
+ answer_plan_prior_features_batch = node_hidden.new_zeros((batch_size, max_event_count, ANSWER_PLAN_FEATURE_DIM))
+ preliminary_event_prob_batch = torch.sigmoid(final_event_logits_batch).masked_fill(~event_mask, 0.0)
+ for batch_index, candidate_event_ids in enumerate(candidate_event_ids_batch):
+ if not candidate_event_ids:
+ continue
+ event_count = candidate_event_lengths[batch_index]
+ path_event_indices = candidate_path_event_indices_batch[batch_index]
+ max_calibrated_path_prob_by_event = _scatter_amax_by_indices(
+ calibrated_path_prob_batch[batch_index, : candidate_path_lengths[batch_index]],
+ path_event_indices,
+ size=event_count,
+ device=device,
+ )
+ temporal_event_indices = candidate_temporal_event_indices_batch[batch_index]
+ max_temporal_prob_by_event = _scatter_amax_by_indices(
+ temporal_prob_batch[batch_index, : temporal_lengths[batch_index]],
+ temporal_event_indices,
+ size=event_count,
+ device=device,
+ )
+ question_features = dict(question_features_batch[batch_index] or {})
+ semantic_target = clean_text(question_features.get("semantic_slot_target", ""))
+ lowered_question = normalize_text(examples[batch_index].question)
+ is_profile_query = bool(
+ question_features.get("asks_profile", False)
+ or semantic_target == "profile"
+ or "profile" in lowered_question
+ or "preference" in lowered_question
+ or "prefer" in lowered_question
+ )
+ is_temporal_query = bool(question_features.get("is_temporal", False))
+ is_chain_query = bool(
+ question_features.get("asks_chain", False)
+ or "chain" in lowered_question
+ or "connected" in lowered_question
+ or "evidence" in lowered_question
+ or "changed" in lowered_question
+ )
+ answer_profile_prob = answer_type_probs_batch[batch_index, ANSWER_TYPE_TO_ID["profile"]]
+ answer_multi_prob = answer_type_probs_batch[batch_index, ANSWER_TYPE_TO_ID["multi_evidence"]]
+ answer_plan_prior_features_batch[batch_index, :event_count] = torch.cat(
+ [
+ preliminary_event_prob_batch[batch_index, :event_count].unsqueeze(-1),
+ calibrated_event_prob_batch[batch_index, :event_count].unsqueeze(-1),
+ torch.sigmoid(event_tunnel_support_logits_batch[batch_index, :event_count]).unsqueeze(-1),
+ torch.sigmoid(event_distractor_logits_batch[batch_index, :event_count]).unsqueeze(-1),
+ max_calibrated_path_prob_by_event.unsqueeze(-1),
+ max_calibrated_path_prob_by_event.unsqueeze(-1),
+ max_temporal_prob_by_event.unsqueeze(-1),
+ node_hidden.new_full((event_count, 1), 1.0 if is_profile_query else 0.0),
+ node_hidden.new_full((event_count, 1), 1.0 if is_temporal_query else 0.0),
+ node_hidden.new_full((event_count, 1), 1.0 if is_chain_query else 0.0),
+ answer_profile_prob.reshape(1, 1).expand(event_count, 1).to(dtype=node_hidden.dtype),
+ answer_multi_prob.reshape(1, 1).expand(event_count, 1).to(dtype=node_hidden.dtype),
+ ],
+ dim=-1,
+ )
+ answer_plan_prior_logits_batch = self.answer_plan_head(
+ question_batch,
+ gathered_event_embeddings,
+ event_support_embeddings_batch,
+ projected_batch,
+ event_pair_features_batch,
+ answer_plan_prior_features_batch,
+ ).masked_fill(~event_mask.unsqueeze(-1), 0.0)
+ answer_plan_prior_prob_batch = torch.sigmoid(answer_plan_prior_logits_batch).masked_fill(
+ ~event_mask.unsqueeze(-1),
+ 0.0,
+ )
+ if max_event_count > 0:
+ final_event_features_batch = node_hidden.new_zeros(
+ (batch_size, max_event_count, FINAL_EVENT_FUSION_FEATURE_DIM)
+ )
+ for batch_index, candidate_event_ids in enumerate(candidate_event_ids_batch):
+ if not candidate_event_ids:
+ continue
+ event_count = candidate_event_lengths[batch_index]
+ recall_scores_for_candidates = _gather_scores_by_indices(
+ recall_event_prob_batch[batch_index, : recall_event_lengths[batch_index]],
+ candidate_event_recall_indices_batch[batch_index],
+ device=device,
+ )
+ base_event_probs = torch.sigmoid(base_event_logits_batch[batch_index, :event_count])
+ rerank_event_probs = torch.sigmoid(rerank_event_logits_batch[batch_index, :event_count])
+ distractor_event_probs = torch.sigmoid(distractor_event_logits_batch[batch_index, :event_count])
+ calibrated_event_probs = calibrated_event_prob_batch[batch_index, :event_count]
+ path_event_indices = candidate_path_event_indices_batch[batch_index]
+ max_rerank_path_prob_by_event = _scatter_amax_by_indices(
+ rerank_path_prob_batch[batch_index, : candidate_path_lengths[batch_index]],
+ path_event_indices,
+ size=event_count,
+ device=device,
+ )
+ max_calibrated_path_prob_by_event = _scatter_amax_by_indices(
+ calibrated_path_prob_batch[batch_index, : candidate_path_lengths[batch_index]],
+ path_event_indices,
+ size=event_count,
+ device=device,
+ )
+ event_path_count = _scatter_sum_by_indices(
+ path_event_indices,
+ size=event_count,
+ device=device,
+ dtype=node_hidden.dtype,
+ )
+ temporal_event_indices = candidate_temporal_event_indices_batch[batch_index]
+ max_temporal_prob_by_event = _scatter_amax_by_indices(
+ temporal_prob_batch[batch_index, : temporal_lengths[batch_index]],
+ temporal_event_indices,
+ size=event_count,
+ device=device,
+ )
+ is_temporal_tensor = node_hidden.new_full(
+ (event_count, 1),
+ 1.0 if bool(question_features_batch[batch_index].get("is_temporal", False)) else 0.0,
+ )
+ event_pair_slice = event_pair_features_batch_raw[batch_index, :event_count]
+ event_reverse_scores, event_boundary_gaps, event_reverse_relations, event_reverse_available, event_reverse_indices = _reverse_competition_features(
+ calibrated_event_probs,
+ event_relation_strength_batch[batch_index, :event_count, :event_count],
+ valid_mask=event_mask[batch_index, :event_count],
+ )
+ reverse_event_path_support = _gather_1d_by_index(
+ max_calibrated_path_prob_by_event,
+ event_reverse_indices,
+ valid_mask=event_reverse_available > 0.0,
+ )
+ reverse_event_temporal_support = _gather_1d_by_index(
+ max_temporal_prob_by_event,
+ event_reverse_indices,
+ valid_mask=event_reverse_available > 0.0,
+ )
+ answer_plan_prior_probs = answer_plan_prior_prob_batch[batch_index, :event_count]
+ answer_plan_prior_current_delta = (
+ answer_plan_prior_probs[:, ANSWER_PLAN_OUTPUT_TO_ID["current"]]
+ - answer_plan_prior_probs[:, ANSWER_PLAN_OUTPUT_TO_ID["suppressed"]]
+ )
+ answer_plan_prior_selected_delta = (
+ answer_plan_prior_probs[:, ANSWER_PLAN_OUTPUT_TO_ID["selected"]]
+ - answer_plan_prior_probs[:, ANSWER_PLAN_OUTPUT_TO_ID["suppressed"]]
+ )
+ answer_features = answer_type_probs_batch[batch_index].unsqueeze(0).expand(event_count, -1).to(dtype=node_hidden.dtype)
+ final_event_features_batch[batch_index, :event_count] = torch.cat(
+ [
+ recall_scores_for_candidates.unsqueeze(-1),
+ base_event_probs.unsqueeze(-1),
+ rerank_event_probs.unsqueeze(-1),
+ distractor_event_probs.unsqueeze(-1),
+ calibrated_event_probs.unsqueeze(-1),
+ max_rerank_path_prob_by_event.unsqueeze(-1),
+ max_calibrated_path_prob_by_event.unsqueeze(-1),
+ max_temporal_prob_by_event.unsqueeze(-1),
+ is_temporal_tensor,
+ (event_path_count / max(1.0, float(len(PATH_TYPES)))).clamp(max=1.0).unsqueeze(-1),
+ event_pair_slice[:, [1, 2, 3, 4, 5, 6, 14, 25, 26, 27, 28]],
+ (calibrated_event_probs - _best_other_scores(calibrated_event_probs)).unsqueeze(-1),
+ (recall_scores_for_candidates - _best_other_scores(recall_scores_for_candidates)).unsqueeze(-1),
+ event_reverse_scores.unsqueeze(-1),
+ event_boundary_gaps.unsqueeze(-1),
+ event_reverse_relations.unsqueeze(-1),
+ event_reverse_available.unsqueeze(-1),
+ reverse_event_path_support.unsqueeze(-1),
+ reverse_event_temporal_support.unsqueeze(-1),
+ answer_plan_prior_probs,
+ answer_plan_prior_current_delta.unsqueeze(-1),
+ answer_plan_prior_selected_delta.unsqueeze(-1),
+ answer_features,
+ ],
+ dim=-1,
+ )
+ event_fusion_delta_batch = self.final_event_fusion_head(
+ question_batch,
+ gathered_event_embeddings,
+ event_support_embeddings_batch,
+ projected_batch,
+ event_pair_features_batch,
+ final_event_features_batch,
+ ).masked_fill(~event_mask, 0.0)
+ final_event_logits_batch = calibrated_event_logits_batch + event_tunnel_delta_batch + event_fusion_delta_batch
+ final_event_prob_batch = (
+ torch.sigmoid(final_event_logits_batch)
+ if max_event_count > 0
+ else node_hidden.new_zeros((batch_size, 0))
+ )
+ if max_path_count > 0:
+ path_tunnel_features_batch = node_hidden.new_zeros(
+ (batch_size, max_path_count, PATH_TUNNEL_FEATURE_DIM)
+ )
+ for batch_index, candidate_paths in enumerate(candidate_paths_batch):
+ if not candidate_paths:
+ continue
+ path_count = candidate_path_lengths[batch_index]
+ event_count = candidate_event_lengths[batch_index]
+ path_event_indices = candidate_path_event_indices_batch[batch_index]
+ event_scores_for_paths = _gather_scores_by_indices(
+ final_event_prob_batch[batch_index, :event_count],
+ path_event_indices,
+ device=device,
+ )
+ temporal_support_scores = _gather_scores_by_indices(
+ temporal_prob_batch[batch_index, : temporal_lengths[batch_index]],
+ candidate_path_temporal_support_indices_batch[batch_index],
+ device=device,
+ )
+ calibrated_path_scores = calibrated_path_prob_batch[batch_index, :path_count]
+ path_count_by_event = _scatter_sum_by_indices(
+ path_event_indices,
+ size=event_count,
+ device=device,
+ dtype=node_hidden.dtype,
+ )
+ path_coverage_by_event = (path_count_by_event / max(1.0, float(len(PATH_TYPES)))).clamp(max=1.0)
+ path_coverage_for_paths = _gather_scores_by_indices(path_coverage_by_event, path_event_indices, device=device)
+ event_rank_ratio_for_paths = _gather_scores_by_indices(
+ _rank_ratio(final_event_prob_batch[batch_index, :event_count]),
+ path_event_indices,
+ device=device,
+ )
+ path_reverse_scores, path_boundary_gaps, path_reverse_relations, path_reverse_available, _ = _reverse_competition_features(
+ calibrated_path_scores,
+ path_relation_strength_batch[batch_index, :path_count, :path_count],
+ valid_mask=path_mask[batch_index, :path_count],
+ )
+ path_tunnel_features_batch[batch_index, :path_count] = _build_path_tunnel_features(
+ event_scores_for_paths=event_scores_for_paths,
+ calibrated_path_scores=calibrated_path_scores,
+ temporal_support_scores=temporal_support_scores,
+ path_pair_features=path_pair_features_batch_raw[batch_index, :path_count],
+ path_reverse_scores=path_reverse_scores,
+ path_boundary_gaps=path_boundary_gaps,
+ path_reverse_relations=path_reverse_relations,
+ path_reverse_available=path_reverse_available,
+ path_coverage_for_paths=path_coverage_for_paths,
+ event_rank_ratio_for_paths=event_rank_ratio_for_paths,
+ question=examples[batch_index].question,
+ question_features=question_features_batch[batch_index],
+ )
+ path_tunnel_outputs = self.path_tunnel_head(
+ question_batch,
+ path_event_embeddings,
+ path_support_embeddings,
+ projected_batch,
+ path_pair_features_batch,
+ path_tunnel_features_batch,
+ ).masked_fill(~path_mask.unsqueeze(-1), 0.0)
+ path_tunnel_support_logits_batch = path_tunnel_outputs[..., 0]
+ path_tunnel_delta_batch = (
+ PATH_TUNNEL_DELTA_LOGIT_LIMIT
+ * torch.tanh(path_tunnel_outputs[..., 1] / PATH_TUNNEL_DELTA_LOGIT_LIMIT)
+ ).masked_fill(~path_mask, 0.0)
+ path_tunnel_delta_batch = _zero_center_masked_logits(path_tunnel_delta_batch, path_mask)
+ final_path_logits_batch = calibrated_path_logits_batch + path_tunnel_delta_batch
+ if max_path_count > 0:
+ final_path_features_batch = node_hidden.new_zeros(
+ (batch_size, max_path_count, FINAL_PATH_FUSION_FEATURE_DIM)
+ )
+ for batch_index, candidate_paths in enumerate(candidate_paths_batch):
+ if not candidate_paths:
+ continue
+ path_count = candidate_path_lengths[batch_index]
+ event_count = candidate_event_lengths[batch_index]
+ path_event_indices = candidate_path_event_indices_batch[batch_index]
+ event_scores_for_paths = _gather_scores_by_indices(
+ final_event_prob_batch[batch_index, :event_count],
+ path_event_indices,
+ device=device,
+ )
+ calibrated_event_scores_for_paths = _gather_scores_by_indices(
+ calibrated_event_prob_batch[batch_index, :event_count],
+ path_event_indices,
+ device=device,
+ )
+ recall_scores_for_paths = _gather_scores_by_indices(
+ recall_event_prob_batch[batch_index, : recall_event_lengths[batch_index]],
+ candidate_path_recall_indices_batch[batch_index],
+ device=device,
+ )
+ temporal_support_scores = _gather_scores_by_indices(
+ temporal_prob_batch[batch_index, : temporal_lengths[batch_index]],
+ candidate_path_temporal_support_indices_batch[batch_index],
+ device=device,
+ )
+ calibrated_path_scores = calibrated_path_prob_batch[batch_index, :path_count]
+ path_count_by_event = _scatter_sum_by_indices(
+ path_event_indices,
+ size=event_count,
+ device=device,
+ dtype=node_hidden.dtype,
+ )
+ path_coverage_by_event = (path_count_by_event / max(1.0, float(len(PATH_TYPES)))).clamp(max=1.0)
+ path_coverage_for_paths = _gather_scores_by_indices(path_coverage_by_event, path_event_indices, device=device)
+ event_rank_ratio_for_paths = _gather_scores_by_indices(
+ _rank_ratio(final_event_prob_batch[batch_index, :event_count]),
+ path_event_indices,
+ device=device,
+ )
+ is_temporal_tensor = node_hidden.new_full(
+ (path_count, 1),
+ 1.0 if bool(question_features_batch[batch_index].get("is_temporal", False)) else 0.0,
+ )
+ path_pair_slice = path_pair_features_batch_raw[batch_index, :path_count]
+ path_reverse_scores, path_boundary_gaps, path_reverse_relations, path_reverse_available, path_reverse_indices = _reverse_competition_features(
+ calibrated_path_scores,
+ path_relation_strength_batch[batch_index, :path_count, :path_count],
+ valid_mask=path_mask[batch_index, :path_count],
+ )
+ reverse_path_event_scores = _gather_1d_by_index(
+ event_scores_for_paths,
+ path_reverse_indices,
+ valid_mask=path_reverse_available > 0.0,
+ )
+ reverse_path_temporal_scores = _gather_1d_by_index(
+ temporal_support_scores,
+ path_reverse_indices,
+ valid_mask=path_reverse_available > 0.0,
+ )
+ answer_plan_prior_for_paths = torch.stack(
+ [
+ _gather_scores_by_indices(
+ answer_plan_prior_prob_batch[batch_index, :event_count, role_index],
+ path_event_indices,
+ device=device,
+ )
+ for role_index in range(len(ANSWER_PLAN_OUTPUTS))
+ ],
+ dim=-1,
+ )
+ answer_plan_prior_current_delta_for_paths = (
+ answer_plan_prior_for_paths[:, ANSWER_PLAN_OUTPUT_TO_ID["current"]]
+ - answer_plan_prior_for_paths[:, ANSWER_PLAN_OUTPUT_TO_ID["suppressed"]]
+ )
+ answer_plan_prior_selected_delta_for_paths = (
+ answer_plan_prior_for_paths[:, ANSWER_PLAN_OUTPUT_TO_ID["selected"]]
+ - answer_plan_prior_for_paths[:, ANSWER_PLAN_OUTPUT_TO_ID["suppressed"]]
+ )
+ answer_features = answer_type_probs_batch[batch_index].unsqueeze(0).expand(path_count, -1).to(dtype=node_hidden.dtype)
+ final_path_features_batch[batch_index, :path_count] = torch.cat(
+ [
+ event_scores_for_paths.unsqueeze(-1),
+ calibrated_event_scores_for_paths.unsqueeze(-1),
+ recall_scores_for_paths.unsqueeze(-1),
+ rerank_path_prob_batch[batch_index, :path_count].unsqueeze(-1),
+ calibrated_path_scores.unsqueeze(-1),
+ temporal_support_scores.unsqueeze(-1),
+ is_temporal_tensor,
+ path_pair_slice[:, [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 17, 18, 19, 20, 21]],
+ (calibrated_path_scores - _best_other_scores(calibrated_path_scores)).unsqueeze(-1),
+ (event_scores_for_paths - _gather_scores_by_indices(_best_other_scores(final_event_prob_batch[batch_index, :event_count]), path_event_indices, device=device)).unsqueeze(-1),
+ path_coverage_for_paths.unsqueeze(-1),
+ _rank_ratio(calibrated_path_scores).unsqueeze(-1),
+ event_rank_ratio_for_paths.unsqueeze(-1),
+ path_reverse_scores.unsqueeze(-1),
+ path_boundary_gaps.unsqueeze(-1),
+ path_reverse_relations.unsqueeze(-1),
+ path_reverse_available.unsqueeze(-1),
+ reverse_path_event_scores.unsqueeze(-1),
+ reverse_path_temporal_scores.unsqueeze(-1),
+ answer_plan_prior_for_paths,
+ answer_plan_prior_current_delta_for_paths.unsqueeze(-1),
+ answer_plan_prior_selected_delta_for_paths.unsqueeze(-1),
+ answer_features,
+ ],
+ dim=-1,
+ )
+ path_fusion_delta_batch = self.final_path_fusion_head(
+ question_batch,
+ path_event_embeddings,
+ path_support_embeddings,
+ projected_batch,
+ path_pair_features_batch,
+ final_path_features_batch,
+ ).masked_fill(~path_mask, 0.0)
+ final_path_logits_batch = (
+ calibrated_path_logits_batch + path_tunnel_delta_batch + path_fusion_delta_batch
+ ).masked_fill(~path_mask, 0.0)
+ answer_plan_logits_batch = node_hidden.new_zeros((batch_size, max_event_count, len(ANSWER_PLAN_OUTPUTS)))
+ if max_event_count > 0:
+ answer_plan_features_batch = node_hidden.new_zeros((batch_size, max_event_count, ANSWER_PLAN_FEATURE_DIM))
+ final_path_prob_batch = (
+ torch.sigmoid(final_path_logits_batch).masked_fill(~path_mask, 0.0)
+ if max_path_count > 0
+ else node_hidden.new_zeros((batch_size, 0))
+ )
+ for batch_index, candidate_event_ids in enumerate(candidate_event_ids_batch):
+ if not candidate_event_ids:
+ continue
+ event_count = candidate_event_lengths[batch_index]
+ path_event_indices = candidate_path_event_indices_batch[batch_index]
+ max_calibrated_path_prob_by_event = _scatter_amax_by_indices(
+ calibrated_path_prob_batch[batch_index, : candidate_path_lengths[batch_index]],
+ path_event_indices,
+ size=event_count,
+ device=device,
+ )
+ max_final_path_prob_by_event = _scatter_amax_by_indices(
+ final_path_prob_batch[batch_index, : candidate_path_lengths[batch_index]],
+ path_event_indices,
+ size=event_count,
+ device=device,
+ )
+ temporal_event_indices = candidate_temporal_event_indices_batch[batch_index]
+ max_temporal_prob_by_event = _scatter_amax_by_indices(
+ temporal_prob_batch[batch_index, : temporal_lengths[batch_index]],
+ temporal_event_indices,
+ size=event_count,
+ device=device,
+ )
+ question_features = dict(question_features_batch[batch_index] or {})
+ semantic_target = clean_text(question_features.get("semantic_slot_target", ""))
+ lowered_question = normalize_text(examples[batch_index].question)
+ is_profile_query = bool(
+ question_features.get("asks_profile", False)
+ or semantic_target == "profile"
+ or "profile" in lowered_question
+ or "preference" in lowered_question
+ or "prefer" in lowered_question
+ )
+ is_temporal_query = bool(question_features.get("is_temporal", False))
+ is_chain_query = bool(
+ question_features.get("asks_chain", False)
+ or "chain" in lowered_question
+ or "connected" in lowered_question
+ or "evidence" in lowered_question
+ or "changed" in lowered_question
+ )
+ answer_profile_prob = answer_type_probs_batch[batch_index, ANSWER_TYPE_TO_ID["profile"]]
+ answer_multi_prob = answer_type_probs_batch[batch_index, ANSWER_TYPE_TO_ID["multi_evidence"]]
+ answer_plan_features_batch[batch_index, :event_count] = torch.cat(
+ [
+ final_event_prob_batch[batch_index, :event_count].unsqueeze(-1),
+ calibrated_event_prob_batch[batch_index, :event_count].unsqueeze(-1),
+ torch.sigmoid(event_tunnel_support_logits_batch[batch_index, :event_count]).unsqueeze(-1),
+ torch.sigmoid(event_distractor_logits_batch[batch_index, :event_count]).unsqueeze(-1),
+ max_calibrated_path_prob_by_event.unsqueeze(-1),
+ max_final_path_prob_by_event.unsqueeze(-1),
+ max_temporal_prob_by_event.unsqueeze(-1),
+ node_hidden.new_full((event_count, 1), 1.0 if is_profile_query else 0.0),
+ node_hidden.new_full((event_count, 1), 1.0 if is_temporal_query else 0.0),
+ node_hidden.new_full((event_count, 1), 1.0 if is_chain_query else 0.0),
+ answer_profile_prob.reshape(1, 1).expand(event_count, 1).to(dtype=node_hidden.dtype),
+ answer_multi_prob.reshape(1, 1).expand(event_count, 1).to(dtype=node_hidden.dtype),
+ ],
+ dim=-1,
+ )
+ answer_plan_logits_batch = self.answer_plan_head(
+ question_batch,
+ gathered_event_embeddings,
+ event_support_embeddings_batch,
+ projected_batch,
+ event_pair_features_batch,
+ answer_plan_features_batch,
+ ).masked_fill(~event_mask.unsqueeze(-1), 0.0)
+ outputs: List[Dict[str, Any]] = []
+ for batch_index, candidate_event_ids in enumerate(candidate_event_ids_batch):
+ event_count = candidate_event_lengths[batch_index]
+ path_count = candidate_path_lengths[batch_index]
+ event_reverse_scores = node_hidden.new_zeros((event_count,))
+ event_boundary_scores = node_hidden.new_zeros((event_count,))
+ event_reverse_relations = node_hidden.new_zeros((event_count,))
+ path_reverse_scores = node_hidden.new_zeros((path_count,))
+ path_boundary_scores = node_hidden.new_zeros((path_count,))
+ path_reverse_relations = node_hidden.new_zeros((path_count,))
+ if event_count > 0:
+ event_reverse_scores, event_boundary_scores, event_reverse_relations, _, _ = _reverse_competition_features(
+ final_event_prob_batch[batch_index, :event_count],
+ event_relation_strength_batch[batch_index, :event_count, :event_count],
+ valid_mask=event_mask[batch_index, :event_count],
+ )
+ if path_count > 0:
+ path_reverse_scores, path_boundary_scores, path_reverse_relations, _, _ = _reverse_competition_features(
+ torch.sigmoid(final_path_logits_batch[batch_index, :path_count]),
+ path_relation_strength_batch[batch_index, :path_count, :path_count],
+ valid_mask=path_mask[batch_index, :path_count],
+ )
+ matrix_event_count = matrix_event_lengths[batch_index]
+ matrix_index_tensor = torch.tensor(matrix_local_index_batch[batch_index], dtype=torch.long, device=device) if matrix_event_count > 0 else None
+ matrix_path_count = matrix_path_lengths[batch_index]
+ matrix_path_index_tensor = (
+ torch.tensor(matrix_path_local_index_batch[batch_index], dtype=torch.long, device=device)
+ if matrix_path_count > 0
+ else None
+ )
+ outputs.append(
+ {
+ "recall_event_ids": list(recall_event_ids_batch[batch_index]),
+ "recall_event_logits": recall_event_logits_batch[batch_index, : recall_event_lengths[batch_index]],
+ "candidate_event_ids": candidate_event_ids,
+ "event_pair_feature_mode": resolved_event_pair_feature_mode,
+ "base_event_logits": base_event_logits_batch[batch_index, :event_count],
+ "rerank_event_logits": rerank_event_logits_batch[batch_index, :event_count],
+ "matrix_event_ids": list(matrix_event_ids_batch[batch_index]),
+ "matrix_event_delta_logits": (
+ matrix_event_delta_batch[batch_index].index_select(0, matrix_index_tensor)
+ if matrix_index_tensor is not None
+ else node_hidden.new_zeros((0,))
+ ),
+ "event_distractor_logits": event_distractor_logits_batch[batch_index, :event_count],
+ "event_distractor_delta_logits": event_distractor_delta_batch[batch_index, :event_count],
+ "distractor_event_logits": distractor_event_logits_batch[batch_index, :event_count],
+ "event_calibration_delta_logits": event_calibration_delta_batch[batch_index, :event_count],
+ "calibrated_event_logits": calibrated_event_logits_batch[batch_index, :event_count],
+ "event_tunnel_support_logits": event_tunnel_support_logits_batch[batch_index, :event_count],
+ "event_tunnel_delta_logits": event_tunnel_delta_batch[batch_index, :event_count],
+ "event_fusion_delta_logits": event_fusion_delta_batch[batch_index, :event_count],
+ "event_logits": final_event_logits_batch[batch_index, :event_count],
+ "answer_plan_logits": answer_plan_logits_batch[batch_index, :event_count],
+ "answer_plan_output_names": list(ANSWER_PLAN_OUTPUTS),
+ "event_reverse_scores": event_reverse_scores,
+ "event_boundary_scores": event_boundary_scores,
+ "event_reverse_relations": event_reverse_relations,
+ "candidate_path_ids": candidate_path_ids_batch[batch_index],
+ "candidate_path_event_ids": candidate_path_event_ids_batch[batch_index],
+ "candidate_path_types": [
+ clean_text(path.get("type", ""))
+ for path in candidate_paths_batch[batch_index]
+ ],
+ "base_path_logits": base_path_logits_batch[batch_index, :path_count],
+ "rerank_path_logits": rerank_path_logits_batch[batch_index, :path_count],
+ "matrix_path_ids": list(matrix_path_ids_batch[batch_index]),
+ "matrix_path_delta_logits": (
+ matrix_path_delta_batch[batch_index].index_select(0, matrix_path_index_tensor)
+ if matrix_path_index_tensor is not None
+ else node_hidden.new_zeros((0,))
+ ),
+ "path_calibration_delta_logits": path_calibration_delta_batch[batch_index, :path_count],
+ "calibrated_path_logits": calibrated_path_logits_batch[batch_index, :path_count],
+ "path_tunnel_support_logits": path_tunnel_support_logits_batch[batch_index, :path_count],
+ "path_tunnel_delta_logits": path_tunnel_delta_batch[batch_index, :path_count],
+ "path_fusion_delta_logits": path_fusion_delta_batch[batch_index, :path_count],
+ "path_logits": final_path_logits_batch[batch_index, :path_count],
+ "path_reverse_scores": path_reverse_scores,
+ "path_boundary_scores": path_boundary_scores,
+ "path_reverse_relations": path_reverse_relations,
+ "candidate_temporal_path_ids": list(candidate_temporal_path_ids_batch[batch_index]),
+ "candidate_temporal_event_ids": list(candidate_temporal_event_ids_batch[batch_index]),
+ "candidate_temporal_node_ids": list(candidate_temporal_node_ids_batch[batch_index]),
+ "temporal_logits": temporal_logits_batch[batch_index, : temporal_lengths[batch_index]],
+ "answer_type_logits": answer_type_logits_batch[batch_index],
+ "memory_router_logits": memory_router_logits_batch[batch_index],
+ "memory_router_layers": list(MEMORY_ROUTER_LAYERS),
+ "question_semantic_logits": question_intent_outputs["semantic_logits"][batch_index],
+ "question_status_logits": question_intent_outputs["status_logits"][batch_index],
+ "question_time_granularity_logits": question_intent_outputs["time_granularity_logits"][batch_index],
+ "question_temporal_logit": question_intent_outputs["temporal_logits"][batch_index],
+ "effective_question_features": dict(question_features_batch[batch_index]),
+ "question_token_role_aux_loss": (
+ question_token_role_aux_loss
+ if batch_index == 0 and question_token_role_aux_loss is not None
+ else node_hidden.new_zeros(())
+ ),
+ "node_token_role_aux_loss": (
+ node_token_role_aux_loss
+ if batch_index == 0 and node_token_role_aux_loss is not None
+ else node_hidden.new_zeros(())
+ ),
+ }
+ )
+ return outputs
+
+ def checkpoint_payload(self, *, metadata: Mapping[str, Any] | None = None) -> Dict[str, Any]:
+ arch_metadata = node_memory_arch_metadata()
+ return {
+ "config": {
+ "question_hash_buckets": QUESTION_HASH_BUCKETS,
+ "node_hash_buckets": NODE_HASH_BUCKETS,
+ "question_output_dim": QUESTION_OUTPUT_DIM,
+ "node_output_dim": NODE_OUTPUT_DIM,
+ "message_hidden_dim": MESSAGE_HIDDEN_DIM,
+ **arch_metadata,
+ },
+ "state_dict": self.state_dict(),
+ "metadata": {**dict(metadata or {}), **arch_metadata},
+ }
+
+ @classmethod
+ def from_checkpoint(cls, payload: Mapping[str, Any], *, device: torch.device | None = None) -> "LocomoNodeMemoryModel":
+ validate_node_memory_checkpoint_payload(payload, context="LocomoNodeMemoryModel.from_checkpoint")
+ model = cls()
+ state_dict = dict(payload.get("state_dict", {}) or {})
+ model._checkpoint_load_report = _load_compatible_state_dict(model, state_dict)
+ if device is not None:
+ model.to(device)
+ return model
+
+
+def _load_compatible_state_dict(module: nn.Module, state_dict: Mapping[str, Any]) -> Dict[str, List[str]]:
+ narrowable_vocab_embeddings = {
+ "node_encoder.node_type_embedding.weight",
+ "message_passing.edge_embedding.weight",
+ "event_subgraph_refiner.role_embedding.weight",
+ "path_head.path_type_embedding.weight",
+ }
+ current_state = module.state_dict()
+ compatible_state: Dict[str, Any] = {}
+ skipped_keys: List[str] = []
+ partially_loaded_keys: List[str] = []
+ unexpected_keys: List[str] = []
+ for key, value in dict(state_dict or {}).items():
+ if key not in current_state:
+ unexpected_keys.append(key)
+ continue
+ current_value = current_state[key]
+ if tuple(current_value.shape) != tuple(value.shape):
+ if (
+ key in narrowable_vocab_embeddings
+ and isinstance(current_value, Tensor)
+ and isinstance(value, Tensor)
+ and int(current_value.ndim) == int(value.ndim)
+ and int(current_value.ndim) >= 1
+ and int(value.shape[0]) >= int(current_value.shape[0])
+ and tuple(value.shape[1:]) == tuple(current_value.shape[1:])
+ ):
+ slices = (slice(0, int(current_value.shape[0])),) + tuple(
+ slice(0, int(dim)) for dim in current_value.shape[1:]
+ )
+ compatible_state[key] = value[slices].to(
+ device=current_value.device,
+ dtype=current_value.dtype,
+ )
+ partially_loaded_keys.append(key)
+ continue
+ if (
+ isinstance(current_value, Tensor)
+ and isinstance(value, Tensor)
+ and int(current_value.ndim) == int(value.ndim)
+ and all(int(old_dim) <= int(new_dim) for old_dim, new_dim in zip(value.shape, current_value.shape))
+ ):
+ widened_value = torch.zeros_like(current_value)
+ slices = tuple(slice(0, int(dim)) for dim in value.shape)
+ widened_value[slices] = value.to(device=widened_value.device, dtype=widened_value.dtype)
+ compatible_state[key] = widened_value
+ partially_loaded_keys.append(key)
+ continue
+ skipped_keys.append(key)
+ continue
+ compatible_state[key] = value
+ missing_keys, remaining_unexpected = module.load_state_dict(compatible_state, strict=False)
+ return {
+ "missing_keys": list(missing_keys),
+ "unexpected_keys": [*unexpected_keys, *list(remaining_unexpected)],
+ "skipped_keys": skipped_keys,
+ "partially_loaded_keys": partially_loaded_keys,
+ }
+
+
+def _output_projection_has_signal(module: Any) -> bool:
+ output_projection = getattr(module, "output_projection", None)
+ if not isinstance(output_projection, nn.Linear):
+ return False
+ weight_nonzero = bool(torch.count_nonzero(output_projection.weight.detach()).item())
+ bias_nonzero = bool(torch.count_nonzero(output_projection.bias.detach()).item()) if output_projection.bias is not None else False
+ return weight_nonzero or bias_nonzero
+
+
+def _sorted_score_ids(score_lookup: Mapping[str, Any], *, limit: int | None = None) -> List[str]:
+ ranked = [
+ event_id
+ for event_id, _ in sorted(
+ (
+ (clean_text(item_id), float(score or 0.0))
+ for item_id, score in dict(score_lookup or {}).items()
+ if clean_text(item_id)
+ ),
+ key=lambda item: (-float(item[1]), item[0]),
+ )
+ ]
+ if limit is None:
+ return ranked
+ return ranked[: max(0, int(limit))]
+
+
+def _dominant_answer_type_from_scores(
+ answer_type_scores: Mapping[str, Any],
+ *,
+ question_is_temporal: bool = False,
+) -> str:
+ normalized_scores = {
+ clean_text(answer_type): float(value or 0.0)
+ for answer_type, value in dict(answer_type_scores or {}).items()
+ if clean_text(answer_type)
+ }
+ if not normalized_scores:
+ return "time" if question_is_temporal else "event_text"
+ return max(normalized_scores.items(), key=lambda item: (float(item[1]), item[0]))[0]
+
+
+def _safe_probability_to_logit(value: Any) -> float:
+ probability = min(1.0 - 1e-6, max(1e-6, float(value or 0.0)))
+ return math.log(probability / (1.0 - probability))
+
+
+def _safe_sigmoid_scalar(value: float) -> float:
+ if value >= 0:
+ z = math.exp(-float(value))
+ return 1.0 / (1.0 + z)
+ z = math.exp(float(value))
+ return z / (1.0 + z)
+
+
+def _event_turn_index_from_node(event_id: str, node: Mapping[str, Any]) -> int:
+ metadata = dict(node.get("metadata", {}) or {})
+ for value in (
+ node.get("turn_index"),
+ metadata.get("turn_index"),
+ node.get("turn"),
+ metadata.get("turn"),
+ ):
+ try:
+ turn_index = int(value)
+ except (TypeError, ValueError):
+ continue
+ if turn_index > 0:
+ return turn_index
+ for value in (
+ clean_text(event_id),
+ clean_text(node.get("dia_id", "")),
+ clean_text(metadata.get("dia_id", "")),
+ ):
+ match = re.search(r"(?:realchat|turn|event)[^0-9]{0,4}(\d+)", value, flags=re.IGNORECASE)
+ if match:
+ return int(match.group(1))
+ return 0
+
+
+def _node_text_token_set(node: Mapping[str, Any]) -> set[str]:
+ metadata = dict(node.get("metadata", {}) or {})
+ texts = [
+ clean_text(node.get("text", "")),
+ clean_text(node.get("event_signature", metadata.get("event_signature", ""))),
+ clean_text(metadata.get("tmcra_typed_event_signature", "")),
+ " ".join(str(item) for item in list(metadata.get("tmcra_node_tags", []) or [])),
+ " ".join(str(item) for item in list(metadata.get("tmcra_path_tags", []) or [])),
+ clean_text(node.get("profile_value", metadata.get("profile_value", ""))),
+ clean_text(node.get("time_value", metadata.get("time_value", ""))),
+ clean_text(node.get("target_status", metadata.get("target_status", ""))),
+ ]
+ tokens: set[str] = set()
+ for text in texts:
+ tokens.update(_normalized_token_list(text))
+ return {token for token in tokens if token}
+
+
+def _node_scalar_features(node: Mapping[str, Any]) -> List[float]:
+ metadata = dict(node.get("metadata", {}) or {})
+ teacher_fields = dict(node.get("teacher_fields", {}) or {})
+ turn_index = float(node.get("turn_index", metadata.get("turn_index", 0)) or 0.0)
+ salience = float(node.get("salience", metadata.get("salience", 0.7)) or 0.7)
+ confidence = float(node.get("confidence", metadata.get("confidence", 0.7)) or 0.7)
+ time_present = 1.0 if clean_text(metadata.get("time_value", node.get("time_value", ""))) else 0.0
+ profile_present = 1.0 if clean_text(metadata.get("profile_value", node.get("profile_value", ""))) else 0.0
+ status_present = 1.0 if clean_text(metadata.get("target_status", node.get("target_status", ""))) else 0.0
+ teacher_present = 1.0 if teacher_fields else 0.0
+ token_count = min(32.0, float(len(tokenize_text(node.get("text", "")))))
+ return [
+ salience,
+ confidence,
+ min(1.0, turn_index / 64.0),
+ time_present,
+ profile_present,
+ status_present,
+ teacher_present,
+ token_count / 32.0,
+ ]
+
+
+def _event_encoder_text(node: Mapping[str, Any], support_payload: Mapping[str, Any] | None = None) -> str:
+ support = dict(support_payload or {})
+ source_turn_texts = [clean_text(item) for item in list(support.get("source_turn_texts", []) or []) if clean_text(item)]
+ profile_texts = [clean_text(item) for item in list(support.get("profile_texts", []) or []) if clean_text(item)]
+ time_texts = [clean_text(item) for item in list(support.get("time_texts", []) or []) if clean_text(item)]
+ status_texts = [clean_text(item) for item in list(support.get("status_texts", []) or []) if clean_text(item)]
+ metadata = dict(node.get("metadata", {}) or {})
+ typed_signature_text = " ".join(
+ dedupe_texts(
+ [
+ clean_text(metadata.get("tmcra_typed_event_signature", "")),
+ " ".join(str(item) for item in list(metadata.get("tmcra_node_tags", []) or [])),
+ " ".join(str(item) for item in list(metadata.get("tmcra_path_tags", []) or [])),
+ clean_text(metadata.get("tmcra_tunnel_group_key", "")),
+ ],
+ max_items=8,
+ )
+ )
+ return " ".join(
+ dedupe_texts(
+ [
+ *source_turn_texts[:1],
+ clean_text(node.get("text", "")),
+ clean_text(node.get("event_signature", metadata.get("event_signature", ""))),
+ typed_signature_text,
+ *profile_texts[:1],
+ *time_texts[:1],
+ *status_texts[:1],
+ ],
+ max_items=6,
+ )
+ )
+
+
+def tensorize_graph(graph: Mapping[str, Any], *, device: torch.device) -> Dict[str, Any]:
+ nodes = [dict(node) for node in list(graph.get("nodes", []) or [])]
+ node_ids = [clean_text(node.get("id", "")) for node in nodes]
+ node_texts = [clean_text(node.get("text", "")) for node in nodes]
+ event_node_ids = [node_id for node_id, node in zip(node_ids, nodes) if clean_text(node.get("type", "")) == "event"]
+ time_node_ids = [node_id for node_id, node in zip(node_ids, nodes) if clean_text(node.get("type", "")) == "time"]
+ node_type_ids = torch.tensor(
+ [NODE_TYPE_TO_ID.get(clean_text(node.get("type", "")), 0) for node in nodes],
+ dtype=torch.long,
+ device=device,
+ )
+ node_scalar_features = torch.zeros((0, NODE_SCALAR_DIM), dtype=torch.float32, device=device)
+ node_id_to_index = {node_id: index for index, node_id in enumerate(node_ids)}
+ edge_src_values: List[int] = []
+ edge_dst_values: List[int] = []
+ edge_type_values: List[int] = []
+ for edge in list(graph.get("edges", []) or []):
+ source = clean_text(edge.get("source", ""))
+ target = clean_text(edge.get("target", ""))
+ if source not in node_id_to_index or target not in node_id_to_index:
+ continue
+ edge_src_values.append(node_id_to_index[source])
+ edge_dst_values.append(node_id_to_index[target])
+ edge_type_values.append(EDGE_TYPE_TO_ID.get(clean_text(edge.get("type", "")), 0))
+ edge_src = torch.tensor(edge_src_values, dtype=torch.long, device=device)
+ edge_dst = torch.tensor(edge_dst_values, dtype=torch.long, device=device)
+ edge_type_ids = torch.tensor(edge_type_values, dtype=torch.long, device=device)
+ node_lookup = {node_id: dict(node) for node_id, node in zip(node_ids, nodes)}
+ normalized_paths: List[Dict[str, Any]] = []
+ paths_by_event_id: Dict[str, List[Dict[str, Any]]] = {}
+ temporal_paths: List[Dict[str, Any]] = []
+ support_texts_by_event: Dict[str, Dict[str, List[str]]] = {}
+ support_node_ids_by_event_and_type: Dict[str, Dict[str, List[str]]] = {}
+ for path in list(graph.get("paths", []) or []):
+ normalized_path = dict(path)
+ event_id = clean_text(normalized_path.get("event_id", ""))
+ normalized_node_ids = [clean_text(item) for item in list(normalized_path.get("node_ids", []) or []) if clean_text(item)]
+ support_node_id = clean_text(normalized_node_ids[2]) if len(normalized_node_ids) >= 3 else ""
+ path_type = clean_text(normalized_path.get("type", ""))
+ if not event_id or event_id not in node_id_to_index or not support_node_id or support_node_id not in node_id_to_index:
+ continue
+ normalized_path["event_id"] = event_id
+ normalized_path["node_ids"] = normalized_node_ids
+ normalized_path["_support_node_id"] = support_node_id
+ normalized_path["_path_type"] = path_type
+ normalized_paths.append(normalized_path)
+ paths_by_event_id.setdefault(event_id, []).append(normalized_path)
+ support_node_ids_by_event_and_type.setdefault(event_id, {}).setdefault(path_type, []).append(support_node_id)
+ support_node = dict(node_lookup.get(support_node_id, {}) or {})
+ support_text = clean_text(support_node.get("text", ""))
+ if support_text:
+ event_support = support_texts_by_event.setdefault(
+ event_id,
+ {
+ "source_turn_texts": [],
+ "profile_texts": [],
+ "time_texts": [],
+ "status_texts": [],
+ },
+ )
+ if path_type == "speaker_event_source_turn":
+ event_support["source_turn_texts"].append(support_text)
+ elif path_type == "speaker_event_profile":
+ event_support["profile_texts"].append(support_text)
+ elif path_type == "speaker_event_time":
+ event_support["time_texts"].append(support_text)
+ temporal_paths.append(normalized_path)
+ elif path_type == "speaker_event_status":
+ event_support["status_texts"].append(support_text)
+ event_support_lookup = {
+ event_id: {key: dedupe_texts(values, max_items=4) for key, values in payload.items()}
+ for event_id, payload in support_texts_by_event.items()
+ }
+ if _env_flag("TMCRA_NODE_MEMORY_EVENT_SUPPORT_CONTEXT", default=True):
+ for index, (node_id, node) in enumerate(zip(node_ids, nodes)):
+ if clean_text(node.get("type", "")) != "event":
+ continue
+ encoder_text = _event_encoder_text(node, event_support_lookup.get(node_id, {}))
+ if encoder_text:
+ nodes[index]["text"] = encoder_text
+ node_texts[index] = encoder_text
+ node_text_hash_indices = [_text_hash_indices(text, NODE_HASH_BUCKETS) for text in node_texts]
+ node_lookup = {node_id: dict(node) for node_id, node in zip(node_ids, nodes)}
+ node_scalar_features = torch.tensor(
+ [_node_scalar_features(node) for node in nodes],
+ dtype=torch.float32,
+ device=device,
+ ) if nodes else torch.zeros((0, NODE_SCALAR_DIM), dtype=torch.float32, device=device)
+ support_node_ids_by_event_and_type = {
+ event_id: {path_type: dedupe_texts(values) for path_type, values in payload.items()}
+ for event_id, payload in support_node_ids_by_event_and_type.items()
+ }
+ support_node_ids_by_event = {
+ event_id: dedupe_texts(
+ support_node_id
+ for support_node_ids in path_type_map.values()
+ for support_node_id in support_node_ids
+ )
+ for event_id, path_type_map in support_node_ids_by_event_and_type.items()
+ }
+ return {
+ "node_ids": node_ids,
+ "node_texts": node_texts,
+ "node_text_hash_indices": node_text_hash_indices,
+ "node_by_id": node_lookup,
+ "event_node_ids": event_node_ids,
+ "time_node_ids": time_node_ids,
+ "node_type_ids": node_type_ids,
+ "node_scalar_features": node_scalar_features,
+ "edge_src": edge_src,
+ "edge_dst": edge_dst,
+ "edge_type_ids": edge_type_ids,
+ "node_id_to_index": node_id_to_index,
+ "paths": normalized_paths,
+ "paths_by_event_id": paths_by_event_id,
+ "temporal_paths": temporal_paths,
+ "event_support_lookup": event_support_lookup,
+ "support_node_ids_by_event": support_node_ids_by_event,
+ "support_node_ids_by_event_and_type": support_node_ids_by_event_and_type,
+ }
+
+
+def _path_support_node_id(path: Mapping[str, Any]) -> str:
+ cached = clean_text(path.get("_support_node_id", ""))
+ if cached:
+ return cached
+ node_ids = [clean_text(item) for item in list(path.get("node_ids", []) or []) if clean_text(item)]
+ return clean_text(node_ids[2]) if len(node_ids) >= 3 else ""
+
+
+def _path_type(path: Mapping[str, Any]) -> str:
+ return clean_text(path.get("_path_type", path.get("type", "")))
+
+
+def _build_event_support_lookup(graph_tensors: Mapping[str, Any]) -> Dict[str, Dict[str, List[str]]]:
+ precomputed = graph_tensors.get("event_support_lookup")
+ if isinstance(precomputed, Mapping):
+ return {
+ clean_text(event_id): {
+ clean_text(key): [clean_text(item) for item in list(values or []) if clean_text(item)]
+ for key, values in dict(payload or {}).items()
+ }
+ for event_id, payload in dict(precomputed).items()
+ if clean_text(event_id)
+ }
+ node_lookup = dict(graph_tensors.get("node_by_id", {}) or {})
+ support_lookup: Dict[str, Dict[str, List[str]]] = {}
+ for path in list(graph_tensors.get("paths", []) or []):
+ event_id = clean_text(path.get("event_id", ""))
+ support_node_id = _path_support_node_id(path)
+ if not event_id or not support_node_id:
+ continue
+ support_node = dict(node_lookup.get(support_node_id, {}) or {})
+ support_text = clean_text(support_node.get("text", ""))
+ if not support_text:
+ continue
+ event_support = support_lookup.setdefault(
+ event_id,
+ {
+ "source_turn_texts": [],
+ "profile_texts": [],
+ "time_texts": [],
+ "status_texts": [],
+ },
+ )
+ normalized_path_type = _path_type(path)
+ if normalized_path_type == "speaker_event_source_turn":
+ event_support["source_turn_texts"].append(support_text)
+ elif normalized_path_type == "speaker_event_profile":
+ event_support["profile_texts"].append(support_text)
+ elif normalized_path_type == "speaker_event_time":
+ event_support["time_texts"].append(support_text)
+ elif normalized_path_type == "speaker_event_status":
+ event_support["status_texts"].append(support_text)
+ for event_id, payload in support_lookup.items():
+ support_lookup[event_id] = {
+ key: dedupe_texts(values, max_items=4)
+ for key, values in payload.items()
+ }
+ return support_lookup
+
+
+def _preferred_support_path_types(question_features: Mapping[str, Any]) -> List[str]:
+ semantic_target = clean_text(question_features.get("semantic_slot_target", ""))
+ target_status_target = clean_text(question_features.get("target_status_target", ""))
+ if bool(question_features.get("is_temporal", False)) or semantic_target == "event_time":
+ return ["speaker_event_time", "speaker_event_source_turn", "speaker_event_status", "speaker_event_profile"]
+ if semantic_target in {"identity", "research_topic", "education", "occupation", "profile"}:
+ return ["speaker_event_profile", "speaker_event_source_turn", "speaker_event_status", "speaker_event_time"]
+ if target_status_target:
+ return ["speaker_event_status", "speaker_event_source_turn", "speaker_event_time", "speaker_event_profile"]
+ return ["speaker_event_source_turn", "speaker_event_time", "speaker_event_profile", "speaker_event_status"]
+
+
+def _candidate_paths_for_event_ids(
+ all_paths: Sequence[Mapping[str, Any]],
+ candidate_event_ids: Sequence[str],
+ *,
+ node_id_to_index: Mapping[str, int],
+) -> List[Dict[str, Any]]:
+ candidate_event_set = {clean_text(item) for item in candidate_event_ids if clean_text(item)}
+ return [
+ dict(path)
+ for path in all_paths
+ if clean_text(path.get("event_id", "")) in candidate_event_set
+ and _path_support_node_id(path) in node_id_to_index
+ ]
+
+
+def _support_node_ids_by_event_and_type(candidate_paths: Sequence[Mapping[str, Any]]) -> Dict[str, Dict[str, List[str]]]:
+ support_node_ids_by_event: Dict[str, Dict[str, List[str]]] = {}
+ for path in candidate_paths:
+ event_id = clean_text(path.get("event_id", ""))
+ support_node_id = _path_support_node_id(path)
+ path_type = _path_type(path)
+ if not event_id or not support_node_id:
+ continue
+ support_node_ids_by_event.setdefault(event_id, {}).setdefault(path_type, []).append(support_node_id)
+ return {
+ event_id: {
+ path_type: dedupe_texts(support_node_ids)
+ for path_type, support_node_ids in path_type_map.items()
+ }
+ for event_id, path_type_map in support_node_ids_by_event.items()
+ }
+
+
+def _support_node_ids_by_event(candidate_paths: Sequence[Mapping[str, Any]]) -> Dict[str, List[str]]:
+ support_node_ids_by_event: Dict[str, List[str]] = {}
+ typed_support = _support_node_ids_by_event_and_type(candidate_paths)
+ return {
+ event_id: dedupe_texts(
+ support_node_id
+ for support_node_ids in path_type_map.values()
+ for support_node_id in support_node_ids
+ )
+ for event_id, path_type_map in typed_support.items()
+ }
+
+
+def _aggregate_event_support_embeddings(
+ node_hidden: Tensor,
+ node_id_to_index: Mapping[str, int],
+ candidate_event_ids_batch: Sequence[Sequence[str]],
+ candidate_paths_batch: Sequence[Sequence[Mapping[str, Any]]] | None = None,
+ *,
+ support_node_ids_by_event: Mapping[str, Sequence[str]] | None = None,
+ support_node_ids_by_event_and_type: Mapping[str, Mapping[str, Sequence[str]]] | None = None,
+ question_features_batch: Sequence[Mapping[str, Any]] | None = None,
+) -> Tensor:
+ batch_size = len(candidate_event_ids_batch)
+ max_event_count = max((len(candidate_event_ids) for candidate_event_ids in candidate_event_ids_batch), default=0)
+ hidden_dim = int(node_hidden.size(-1)) if node_hidden.dim() > 1 else 0
+ support_embeddings = node_hidden.new_zeros((batch_size, max_event_count, hidden_dim))
+ resolved_question_features = list(question_features_batch or [{} for _ in candidate_event_ids_batch])
+ resolved_candidate_paths_batch = list(candidate_paths_batch or [[] for _ in candidate_event_ids_batch])
+ for batch_index, candidate_event_ids in enumerate(candidate_event_ids_batch):
+ if support_node_ids_by_event is not None and support_node_ids_by_event_and_type is not None:
+ local_support_node_ids_by_event = support_node_ids_by_event
+ local_support_node_ids_by_event_and_type = support_node_ids_by_event_and_type
+ else:
+ local_support_node_ids_by_event = _support_node_ids_by_event(resolved_candidate_paths_batch[batch_index])
+ local_support_node_ids_by_event_and_type = _support_node_ids_by_event_and_type(resolved_candidate_paths_batch[batch_index])
+ preferred_path_types = _preferred_support_path_types(
+ resolved_question_features[batch_index] if batch_index < len(resolved_question_features) else {}
+ )
+ for event_index, event_id in enumerate(candidate_event_ids):
+ event_support_by_type = dict(local_support_node_ids_by_event_and_type.get(event_id, {}) or {})
+ support_node_ids: List[str] = []
+ for path_type in preferred_path_types:
+ preferred_support_ids = [
+ support_node_id
+ for support_node_id in event_support_by_type.get(path_type, [])
+ if support_node_id in node_id_to_index
+ ]
+ if preferred_support_ids:
+ support_node_ids = preferred_support_ids
+ break
+ if not support_node_ids:
+ support_node_ids = [
+ support_node_id
+ for support_node_id in local_support_node_ids_by_event.get(event_id, [])
+ if support_node_id in node_id_to_index
+ ]
+ if not support_node_ids:
+ continue
+ support_indices = torch.tensor(
+ [node_id_to_index[support_node_id] for support_node_id in support_node_ids],
+ dtype=torch.long,
+ device=node_hidden.device,
+ )
+ support_embeddings[batch_index, event_index] = node_hidden.index_select(0, support_indices).mean(dim=0)
+ return support_embeddings
+
+
+def _aggregate_event_support_slot_embeddings(
+ node_hidden: Tensor,
+ node_id_to_index: Mapping[str, int],
+ candidate_event_ids_batch: Sequence[Sequence[str]],
+ candidate_paths_batch: Sequence[Sequence[Mapping[str, Any]]] | None = None,
+ *,
+ support_node_ids_by_event_and_type: Mapping[str, Mapping[str, Sequence[str]]] | None = None,
+) -> tuple[Tensor, Tensor]:
+ batch_size = len(candidate_event_ids_batch)
+ max_event_count = max((len(candidate_event_ids) for candidate_event_ids in candidate_event_ids_batch), default=0)
+ slot_count = len(EVENT_SUPPORT_PATH_TYPE_ORDER)
+ hidden_dim = int(node_hidden.size(-1)) if node_hidden.dim() > 1 else 0
+ support_slot_embeddings = node_hidden.new_zeros((batch_size, max_event_count, slot_count, hidden_dim))
+ support_slot_mask = torch.zeros((batch_size, max_event_count, slot_count), dtype=torch.bool, device=node_hidden.device)
+ resolved_candidate_paths_batch = list(candidate_paths_batch or [[] for _ in candidate_event_ids_batch])
+ for batch_index, candidate_event_ids in enumerate(candidate_event_ids_batch):
+ if support_node_ids_by_event_and_type is not None:
+ local_support_node_ids_by_event_and_type = support_node_ids_by_event_and_type
+ else:
+ local_support_node_ids_by_event_and_type = _support_node_ids_by_event_and_type(
+ resolved_candidate_paths_batch[batch_index]
+ )
+ for event_index, event_id in enumerate(candidate_event_ids):
+ event_support_by_type = dict(local_support_node_ids_by_event_and_type.get(event_id, {}) or {})
+ for slot_index, path_type in enumerate(EVENT_SUPPORT_PATH_TYPE_ORDER):
+ support_node_ids = [
+ support_node_id
+ for support_node_id in event_support_by_type.get(path_type, [])
+ if support_node_id in node_id_to_index
+ ]
+ if not support_node_ids:
+ continue
+ support_indices = torch.tensor(
+ [node_id_to_index[support_node_id] for support_node_id in support_node_ids],
+ dtype=torch.long,
+ device=node_hidden.device,
+ )
+ support_slot_embeddings[batch_index, event_index, slot_index] = node_hidden.index_select(
+ 0,
+ support_indices,
+ ).mean(dim=0)
+ support_slot_mask[batch_index, event_index, slot_index] = True
+ return support_slot_embeddings, support_slot_mask
+
+
+def _select_rerank_event_ids(
+ all_event_ids: Sequence[str],
+ recall_logits: Tensor,
+ *,
+ positive_event_ids: Sequence[str] = (),
+ forced_event_ids: Sequence[str] = (),
+ candidate_limit: int = DEFAULT_RUNTIME_RECALL_TOP_K,
+) -> List[str]:
+ available_event_ids = {clean_text(item) for item in all_event_ids if clean_text(item)}
+ ranked_indices = torch.argsort(recall_logits.detach(), descending=True)
+ ranked_event_ids = [clean_text(all_event_ids[int(index)]) for index in ranked_indices.tolist() if clean_text(all_event_ids[int(index)])]
+ positives = [clean_text(item) for item in positive_event_ids if clean_text(item) and clean_text(item) in available_event_ids]
+ forced = [clean_text(item) for item in forced_event_ids if clean_text(item) and clean_text(item) in available_event_ids]
+ resolved_limit = max(1, int(candidate_limit or DEFAULT_RUNTIME_RECALL_TOP_K))
+ return dedupe_texts([*positives, *forced, *ranked_event_ids[:resolved_limit]])
+
+
+def _resolved_hard_negative_event_ids_for_example(example: Any) -> List[str]:
+ explicit = [
+ clean_text(item)
+ for item in list(getattr(example, "hard_negative_event_ids", []) or [])
+ if clean_text(item)
+ ]
+ if explicit:
+ return dedupe_texts(explicit)
+ return [
+ clean_text(item)
+ for item in list(getattr(example, "negative_event_ids", []) or [])
+ if clean_text(item)
+ ]
+
+
+def _resolved_hard_negative_event_ids_from_supervision(supervision: Mapping[str, Any]) -> frozenset[str]:
+ explicit = frozenset(
+ clean_text(item)
+ for item in list(supervision.get("hard_negative_event_ids", []) or [])
+ if clean_text(item)
+ )
+ if explicit:
+ return explicit
+ return frozenset(
+ clean_text(item)
+ for item in list(supervision.get("negative_event_ids", []) or [])
+ if clean_text(item)
+ )
+
+
+def _select_matrix_event_ids(
+ candidate_event_ids: Sequence[str],
+ recall_ranked_event_ids: Sequence[str],
+ *,
+ positive_event_ids: Sequence[str] = (),
+ hard_negative_event_ids: Sequence[str] = (),
+ matrix_top_k: int = DEFAULT_MATRIX_EVENT_TOP_K,
+ recall_seed_top_k: int = DEFAULT_MATRIX_EVENT_RECALL_SEED_TOP_K,
+ hard_negative_limit: int = DEFAULT_MATRIX_EVENT_HARD_NEGATIVE_LIMIT,
+) -> List[str]:
+ available = {clean_text(item) for item in candidate_event_ids if clean_text(item)}
+ recall_ranked = [clean_text(item) for item in recall_ranked_event_ids if clean_text(item) and clean_text(item) in available]
+ if not recall_ranked:
+ recall_ranked = [clean_text(item) for item in candidate_event_ids if clean_text(item)]
+ positives = [clean_text(item) for item in positive_event_ids if clean_text(item) and clean_text(item) in available]
+ positive_set = {clean_text(item) for item in positives if clean_text(item)}
+ hard_negative_set = {clean_text(item) for item in hard_negative_event_ids if clean_text(item) and clean_text(item) in available}
+ resolved_top_k = max(1, int(matrix_top_k or DEFAULT_MATRIX_EVENT_TOP_K))
+ if not positives and not hard_negative_set:
+ return dedupe_texts(recall_ranked, max_items=resolved_top_k)
+ recall_seed = recall_ranked[: max(1, int(recall_seed_top_k or DEFAULT_MATRIX_EVENT_RECALL_SEED_TOP_K))]
+ hard_negatives = [
+ event_id
+ for event_id in recall_ranked
+ if event_id in hard_negative_set and event_id not in positive_set
+ ][: max(0, int(hard_negative_limit or DEFAULT_MATRIX_EVENT_HARD_NEGATIVE_LIMIT))]
+ return dedupe_texts([*positives, *hard_negatives, *recall_seed], max_items=resolved_top_k)
+
+
+def _select_matrix_path_ids(
+ candidate_path_ids: Sequence[str],
+ ranked_path_ids: Sequence[str],
+ *,
+ positive_path_ids: Sequence[str] = (),
+ hard_negative_path_ids: Sequence[str] = (),
+ matrix_top_k: int = DEFAULT_MATRIX_PATH_TOP_K,
+ ranked_seed_top_k: int = DEFAULT_MATRIX_PATH_RECALL_SEED_TOP_K,
+ hard_negative_limit: int = DEFAULT_MATRIX_PATH_HARD_NEGATIVE_LIMIT,
+) -> List[str]:
+ available = {clean_text(item) for item in candidate_path_ids if clean_text(item)}
+ ranked = [clean_text(item) for item in ranked_path_ids if clean_text(item) and clean_text(item) in available]
+ if not ranked:
+ ranked = [clean_text(item) for item in candidate_path_ids if clean_text(item)]
+ positives = [clean_text(item) for item in positive_path_ids if clean_text(item) and clean_text(item) in available]
+ positive_set = {clean_text(item) for item in positives if clean_text(item)}
+ hard_negative_set = {clean_text(item) for item in hard_negative_path_ids if clean_text(item) and clean_text(item) in available}
+ resolved_top_k = max(1, int(matrix_top_k or DEFAULT_MATRIX_PATH_TOP_K))
+ if not positives and not hard_negative_set:
+ return dedupe_texts(ranked, max_items=resolved_top_k)
+ ranked_seed = ranked[: max(1, int(ranked_seed_top_k or DEFAULT_MATRIX_PATH_RECALL_SEED_TOP_K))]
+ hard_negatives = [
+ path_id
+ for path_id in ranked
+ if path_id in hard_negative_set and path_id not in positive_set
+ ][: max(0, int(hard_negative_limit or DEFAULT_MATRIX_PATH_HARD_NEGATIVE_LIMIT))]
+ return dedupe_texts([*positives, *hard_negatives, *ranked_seed], max_items=resolved_top_k)
+
+
+def load_graph_cache(
+ graph_dir: Path,
+ *,
+ device: torch.device,
+ cache_dir: Path | None = None,
+) -> Dict[str, GraphCacheItem]:
+ cache: Dict[str, GraphCacheItem] = {}
+ cpu_device = torch.device("cpu")
+ resolved_cache_dir = Path(cache_dir) if cache_dir is not None else None
+ if resolved_cache_dir is not None:
+ resolved_cache_dir.mkdir(parents=True, exist_ok=True)
+ gc_enabled = gc.isenabled()
+ if gc_enabled:
+ gc.disable()
+ try:
+ for path in _iter_graph_paths(graph_dir):
+ cached_payload = None
+ source_signature = _graph_source_signature(path)
+ if resolved_cache_dir is not None:
+ cached_payload = _load_graph_tensor_cache(
+ _graph_tensor_cache_path(resolved_cache_dir, path),
+ source_signature=source_signature,
+ )
+ if cached_payload is not None:
+ graph = dict(cached_payload["graph"])
+ tensors = _graph_tensor_mapping_to_device(cached_payload["tensors"], device, non_blocking=device.type == "cuda")
+ else:
+ graph = dict(read_json(path))
+ cache_tensors = tensorize_graph(graph, device=cpu_device)
+ if resolved_cache_dir is not None:
+ _write_graph_tensor_cache(
+ _graph_tensor_cache_path(resolved_cache_dir, path),
+ source_signature=source_signature,
+ graph=graph,
+ tensors=cache_tensors,
+ )
+ tensors = _graph_tensor_mapping_to_device(cache_tensors, device, non_blocking=device.type == "cuda")
+ tensors = dict(_ensure_graph_scoring_feature_cache(tensors))
+ conversation_id = clean_text(graph.get("conversation_id", path.stem))
+ cache[conversation_id] = GraphCacheItem(graph=graph, tensors=tensors)
+ finally:
+ if gc_enabled:
+ gc.enable()
+ return cache
+
+
+def _iter_graph_paths(graph_dir: Path) -> Iterator[Path]:
+ resolved_graph_dir = Path(graph_dir)
+ names: List[str] = []
+ with os.scandir(resolved_graph_dir) as iterator:
+ for entry in iterator:
+ if not entry.is_file():
+ continue
+ name = str(entry.name)
+ if not name.endswith(".json"):
+ continue
+ names.append(name)
+ names.sort()
+ for name in names:
+ yield resolved_graph_dir / name
+
+
+def _load_query_example_from_raw_line(path: Path, offset: int, raw_line: bytes) -> QueryTrainingExample:
+ if not raw_line:
+ raise RuntimeError(f"Missing JSONL row at byte offset {offset} in {path}")
+ try:
+ payload = dict(json.loads(raw_line.decode("utf-8")))
+ except UnicodeDecodeError as exc:
+ raise RuntimeError(f"Failed to decode JSONL row at byte offset {offset} in {path}: {exc}") from exc
+ except json.JSONDecodeError as exc:
+ raise RuntimeError(
+ f"Failed to parse JSONL row at byte offset {offset} in {path}: "
+ f"line {exc.lineno} column {exc.colno} char {exc.pos}: {exc.msg}"
+ ) from exc
+ try:
+ return QueryTrainingExample.from_dict(payload)
+ except Exception as exc:
+ raise RuntimeError(f"Failed to normalize JSONL row at byte offset {offset} in {path}: {exc}") from exc
+
+
+def iter_query_examples(
+ path: Path,
+ *,
+ skip_bad_rows: bool = False,
+ error_callback: Callable[[Dict[str, Any]], None] | None = None,
+) -> Iterator[QueryTrainingExample]:
+ for row in load_query_examples(path, skip_bad_rows=skip_bad_rows, error_callback=error_callback):
+ yield row
+
+
+class IndexedQueryExampleStore(Sequence[QueryTrainingExample]):
+ def __init__(
+ self,
+ path: Path,
+ *,
+ skip_bad_rows: bool = False,
+ error_callback: Callable[[Dict[str, Any]], None] | None = None,
+ ) -> None:
+ self.path = Path(path)
+ self._skip_bad_rows = bool(skip_bad_rows)
+ self._error_callback = error_callback
+ self._row_offsets: List[int] = []
+ self._conversation_offsets: "OrderedDict[str, List[int]]" = OrderedDict()
+ self._build_index()
+
+ def _build_index(self) -> None:
+ if not self._skip_bad_rows:
+ for offset, row in iter_jsonl_with_offsets(self.path):
+ row_index = len(self._row_offsets)
+ conversation_id = clean_text(row.get("conversation_id", "")) or f"__row_{row_index}"
+ self._row_offsets.append(int(offset))
+ self._conversation_offsets.setdefault(conversation_id, []).append(int(offset))
+ return
+ if not self.path.exists():
+ return
+ with self.path.open("rb") as handle:
+ for file_line_number in itertools.count(start=1):
+ offset = handle.tell()
+ raw_line = handle.readline()
+ if not raw_line:
+ break
+ if not raw_line.strip():
+ continue
+ try:
+ example = _load_query_example_from_raw_line(self.path, int(offset), raw_line)
+ except Exception as exc:
+ self._report_bad_row(
+ offset=int(offset),
+ file_line_number=int(file_line_number),
+ exc=exc,
+ )
+ continue
+ row_index = len(self._row_offsets)
+ conversation_id = clean_text(example.conversation_id) or f"__row_{row_index}"
+ self._row_offsets.append(int(offset))
+ self._conversation_offsets.setdefault(conversation_id, []).append(int(offset))
+
+ def _report_bad_row(self, *, offset: int, file_line_number: int, exc: Exception) -> None:
+ if self._error_callback is None:
+ return
+ self._error_callback(
+ {
+ "path": str(self.path),
+ "offset": int(offset),
+ "file_line_number": int(file_line_number),
+ "error_type": type(exc).__name__,
+ "error_message": clean_text(str(exc)),
+ }
+ )
+
+ def _load_batch_from_offsets(self, offsets: Sequence[int]) -> List[QueryTrainingExample]:
+ resolved_offsets = [int(offset) for offset in offsets]
+ if not resolved_offsets:
+ return []
+ rows: List[QueryTrainingExample] = []
+ with self.path.open("rb") as handle:
+ for offset in resolved_offsets:
+ handle.seek(offset)
+ raw_line = handle.readline()
+ try:
+ rows.append(_load_query_example_from_raw_line(self.path, int(offset), raw_line))
+ except Exception as exc:
+ if not self._skip_bad_rows:
+ raise
+ self._report_bad_row(offset=int(offset), file_line_number=0, exc=exc)
+ return rows
+
+ def _batch_spec_from_offsets(self, offsets: Sequence[int]) -> QueryBatchSpec:
+ return QueryBatchSpec(
+ sources=(
+ QueryBatchSourceOffsets(
+ path=str(self.path),
+ offsets=tuple(int(offset) for offset in offsets),
+ skip_bad_rows=bool(self._skip_bad_rows),
+ ),
+ ),
+ )
+
+ def conversation_offset_items(self, *, shuffle: bool, seed: int) -> Iterator[tuple[str, List[int]]]:
+ rng = random.Random(seed)
+ conversation_ids = list(self._conversation_offsets)
+ if shuffle:
+ rng.shuffle(conversation_ids)
+ for conversation_id in conversation_ids:
+ offsets = list(self._conversation_offsets.get(conversation_id, []))
+ if shuffle:
+ rng.shuffle(offsets)
+ yield conversation_id, offsets
+
+ def conversation_ids(self) -> List[str]:
+ return list(self._conversation_offsets)
+
+ def iter_batches(
+ self,
+ *,
+ batch_size: int,
+ shuffle: bool,
+ seed: int,
+ ) -> Iterator[List[QueryTrainingExample]]:
+ resolved_batch_size = max(1, int(batch_size))
+ current_offsets: List[int] = []
+ for _, offsets in self.conversation_offset_items(shuffle=shuffle, seed=seed):
+ if len(offsets) > resolved_batch_size:
+ if current_offsets:
+ yield self._load_batch_from_offsets(current_offsets)
+ current_offsets = []
+ for index in range(0, len(offsets), resolved_batch_size):
+ yield self._load_batch_from_offsets(offsets[index : index + resolved_batch_size])
+ continue
+ if current_offsets and len(current_offsets) + len(offsets) > resolved_batch_size:
+ yield self._load_batch_from_offsets(current_offsets)
+ current_offsets = []
+ current_offsets.extend(offsets)
+ if current_offsets:
+ yield self._load_batch_from_offsets(current_offsets)
+
+ def iter_batch_specs(
+ self,
+ *,
+ batch_size: int,
+ shuffle: bool,
+ seed: int,
+ ) -> Iterator[QueryBatchSpec]:
+ resolved_batch_size = max(1, int(batch_size))
+ current_offsets: List[int] = []
+ for _, offsets in self.conversation_offset_items(shuffle=shuffle, seed=seed):
+ if len(offsets) > resolved_batch_size:
+ if current_offsets:
+ yield self._batch_spec_from_offsets(current_offsets)
+ current_offsets = []
+ for index in range(0, len(offsets), resolved_batch_size):
+ yield self._batch_spec_from_offsets(offsets[index : index + resolved_batch_size])
+ continue
+ if current_offsets and len(current_offsets) + len(offsets) > resolved_batch_size:
+ yield self._batch_spec_from_offsets(current_offsets)
+ current_offsets = []
+ current_offsets.extend(offsets)
+ if current_offsets:
+ yield self._batch_spec_from_offsets(current_offsets)
+
+ def __len__(self) -> int:
+ return len(self._row_offsets)
+
+ def __getitem__(self, index: int | slice) -> QueryTrainingExample | List[QueryTrainingExample]:
+ if isinstance(index, slice):
+ return self._load_batch_from_offsets(self._row_offsets[index])
+ resolved_index = int(index)
+ if resolved_index < 0:
+ resolved_index += len(self._row_offsets)
+ if resolved_index < 0 or resolved_index >= len(self._row_offsets):
+ raise IndexError(resolved_index)
+ return self._load_batch_from_offsets([self._row_offsets[resolved_index]])[0]
+
+ def __iter__(self) -> Iterator[QueryTrainingExample]:
+ with self.path.open("rb") as handle:
+ for offset in self._row_offsets:
+ handle.seek(offset)
+ raw_line = handle.readline()
+ try:
+ yield _load_query_example_from_raw_line(self.path, int(offset), raw_line)
+ except Exception as exc:
+ if not self._skip_bad_rows:
+ raise
+ self._report_bad_row(offset=int(offset), file_line_number=0, exc=exc)
+
+
+class CombinedQueryExampleStore(Sequence[QueryTrainingExample]):
+ def __init__(self, sources: Sequence[IndexedQueryExampleStore]) -> None:
+ self._sources = [source for source in sources if len(source) > 0]
+ self._row_count = sum(len(source) for source in self._sources)
+
+ def __len__(self) -> int:
+ return self._row_count
+
+ def __getitem__(self, index: int | slice) -> QueryTrainingExample | List[QueryTrainingExample]:
+ if isinstance(index, slice):
+ start, stop, step = index.indices(len(self))
+ if step != 1:
+ return [self[position] for position in range(start, stop, step)]
+ rows: List[QueryTrainingExample] = []
+ for position in range(start, stop):
+ rows.append(self[position])
+ return rows
+ resolved_index = int(index)
+ if resolved_index < 0:
+ resolved_index += len(self)
+ if resolved_index < 0 or resolved_index >= len(self):
+ raise IndexError(resolved_index)
+ for source in self._sources:
+ if resolved_index < len(source):
+ return source[resolved_index]
+ resolved_index -= len(source)
+ raise IndexError(index)
+
+ def __iter__(self) -> Iterator[QueryTrainingExample]:
+ for source in self._sources:
+ for row in source:
+ yield row
+
+ def iter_batches(
+ self,
+ *,
+ batch_size: int,
+ shuffle: bool,
+ seed: int,
+ ) -> Iterator[List[QueryTrainingExample]]:
+ resolved_batch_size = max(1, int(batch_size))
+ rng = random.Random(seed)
+ group_specs: List[tuple[IndexedQueryExampleStore, List[int]]] = []
+ for source_index, source in enumerate(self._sources):
+ source_seed = seed + ((source_index + 1) * 9973)
+ for _, offsets in source.conversation_offset_items(shuffle=shuffle, seed=source_seed):
+ group_specs.append((source, offsets))
+ if shuffle:
+ rng.shuffle(group_specs)
+ current_batch: List[tuple[IndexedQueryExampleStore, List[int]]] = []
+ current_batch_size = 0
+ for source, offsets in group_specs:
+ if len(offsets) > resolved_batch_size:
+ if current_batch:
+ yield self._load_group_batch(current_batch)
+ current_batch = []
+ current_batch_size = 0
+ for index in range(0, len(offsets), resolved_batch_size):
+ yield source._load_batch_from_offsets(offsets[index : index + resolved_batch_size])
+ continue
+ if current_batch and current_batch_size + len(offsets) > resolved_batch_size:
+ yield self._load_group_batch(current_batch)
+ current_batch = []
+ current_batch_size = 0
+ current_batch.append((source, offsets))
+ current_batch_size += len(offsets)
+ if current_batch:
+ yield self._load_group_batch(current_batch)
+
+ def iter_batch_specs(
+ self,
+ *,
+ batch_size: int,
+ shuffle: bool,
+ seed: int,
+ ) -> Iterator[QueryBatchSpec]:
+ resolved_batch_size = max(1, int(batch_size))
+ rng = random.Random(seed)
+ group_specs: List[tuple[IndexedQueryExampleStore, List[int]]] = []
+ for source_index, source in enumerate(self._sources):
+ source_seed = seed + ((source_index + 1) * 9973)
+ for _, offsets in source.conversation_offset_items(shuffle=shuffle, seed=source_seed):
+ group_specs.append((source, offsets))
+ if shuffle:
+ rng.shuffle(group_specs)
+ current_batch: List[tuple[IndexedQueryExampleStore, List[int]]] = []
+ current_batch_size = 0
+ for source, offsets in group_specs:
+ if len(offsets) > resolved_batch_size:
+ if current_batch:
+ yield self._load_group_batch_spec(current_batch)
+ current_batch = []
+ current_batch_size = 0
+ for index in range(0, len(offsets), resolved_batch_size):
+ yield source._batch_spec_from_offsets(offsets[index : index + resolved_batch_size])
+ continue
+ if current_batch and current_batch_size + len(offsets) > resolved_batch_size:
+ yield self._load_group_batch_spec(current_batch)
+ current_batch = []
+ current_batch_size = 0
+ current_batch.append((source, offsets))
+ current_batch_size += len(offsets)
+ if current_batch:
+ yield self._load_group_batch_spec(current_batch)
+
+ def conversation_ids(self) -> List[str]:
+ conversation_ids: List[str] = []
+ seen: set[str] = set()
+ for source in self._sources:
+ source_conversation_ids = getattr(source, "conversation_ids", None)
+ if callable(source_conversation_ids):
+ values = source_conversation_ids()
+ else:
+ values = [clean_text(getattr(row, "conversation_id", "")) for row in source]
+ for conversation_id in values:
+ normalized_id = clean_text(conversation_id)
+ if not normalized_id or normalized_id in seen:
+ continue
+ seen.add(normalized_id)
+ conversation_ids.append(normalized_id)
+ return conversation_ids
+
+ def _load_group_batch(self, group_specs: Sequence[tuple[IndexedQueryExampleStore, List[int]]]) -> List[QueryTrainingExample]:
+ rows: List[QueryTrainingExample] = []
+ for source, offsets in group_specs:
+ rows.extend(source._load_batch_from_offsets(offsets))
+ return rows
+
+ def _load_group_batch_spec(self, group_specs: Sequence[tuple[IndexedQueryExampleStore, List[int]]]) -> QueryBatchSpec:
+ return QueryBatchSpec(
+ sources=tuple(
+ QueryBatchSourceOffsets(
+ path=str(source.path),
+ offsets=tuple(int(offset) for offset in offsets),
+ skip_bad_rows=bool(source._skip_bad_rows),
+ )
+ for source, offsets in group_specs
+ if offsets
+ )
+ )
+
+
+class SourceAwareTrainingStore:
+ def __init__(
+ self,
+ base_rows: Sequence[QueryTrainingExample],
+ *,
+ config: Mapping[str, Any] | None = None,
+ ) -> None:
+ self._base_rows = base_rows
+ self._config = {**DEFAULT_TRAINING_CONFIG, **dict(config or {})}
+ self._groups: List[_TrainingConversationGroup] = []
+ self._source_row_counts: Counter[str] = Counter()
+ self._answer_type_counts: Counter[str] = Counter()
+ self._supervision_bucket_counts: Counter[str] = Counter()
+ self._source_answer_temporal_counts: Counter[tuple[str, str, bool]] = Counter()
+ self._source_supervision_temporal_counts: Counter[tuple[str, str, bool]] = Counter()
+ self._sampling_summary: Dict[str, Any] = {}
+ self._effective_row_count = 0
+ self._build()
+
+ def __len__(self) -> int:
+ return int(self._effective_row_count)
+
+ def base_rows(self) -> Sequence[QueryTrainingExample]:
+ return self._base_rows
+
+ def sampling_summary(self) -> Dict[str, Any]:
+ return dict(self._sampling_summary)
+
+ def _iter_sources(self) -> List[Any]:
+ if isinstance(self._base_rows, CombinedQueryExampleStore):
+ return list(self._base_rows._sources)
+ return [self._base_rows]
+
+ def _collect_conversation_summaries(
+ self,
+ source: Any,
+ *,
+ fallback_source: str,
+ in_memory_groups: OrderedDict[str, List[QueryTrainingExample]] | None = None,
+ ) -> "OrderedDict[str, _TrainingConversationSummary]":
+ conversation_summaries: "OrderedDict[str, _TrainingConversationSummary]" = OrderedDict()
+ for example in source:
+ if not isinstance(example, QueryTrainingExample):
+ continue
+ conversation_id = clean_text(example.conversation_id)
+ if not conversation_id:
+ continue
+ source_dataset = _training_source_dataset_name(example=example, fallback_source=fallback_source)
+ answer_type = _training_answer_type_name(example)
+ supervision_bucket = _training_supervision_bucket_name(example)
+ has_temporal_positive = _has_positive_temporal_supervision(example)
+ summary = conversation_summaries.get(conversation_id)
+ if summary is None:
+ summary = _TrainingConversationSummary(
+ conversation_id=conversation_id,
+ source_dataset=source_dataset,
+ )
+ conversation_summaries[conversation_id] = summary
+ summary.row_count += 1
+ summary.answer_type_counts[answer_type] = int(summary.answer_type_counts.get(answer_type, 0) or 0) + 1
+ if answer_type == "time":
+ summary.time_example_count += 1
+ if _has_multi_evidence_supervision(example):
+ summary.multi_evidence_count += 1
+ if has_temporal_positive:
+ summary.temporal_positive_count += 1
+ self._source_row_counts[source_dataset] += 1
+ self._answer_type_counts[answer_type] += 1
+ self._supervision_bucket_counts[supervision_bucket] += 1
+ self._source_answer_temporal_counts[(source_dataset, answer_type, bool(has_temporal_positive))] += 1
+ self._source_supervision_temporal_counts[(source_dataset, supervision_bucket, bool(has_temporal_positive))] += 1
+ if in_memory_groups is not None:
+ in_memory_groups.setdefault(conversation_id, []).append(example)
+ return conversation_summaries
+
+ def _source_fallback_name(self, source: Any) -> str:
+ path_value = getattr(source, "path", None)
+ if path_value is None:
+ return "unknown"
+ try:
+ resolved_path = Path(path_value)
+ except Exception:
+ return "unknown"
+ return clean_text(resolved_path.parent.parent.name) or "unknown"
+
+ def _build(self) -> None:
+ source_entries: List[tuple[Any, OrderedDict[str, _TrainingConversationSummary], OrderedDict[str, List[QueryTrainingExample]] | None]] = []
+ for source in self._iter_sources():
+ fallback_source = self._source_fallback_name(source)
+ has_offset_loader = callable(getattr(source, "conversation_offset_items", None)) and callable(
+ getattr(source, "_load_batch_from_offsets", None)
+ )
+ in_memory_groups: OrderedDict[str, List[QueryTrainingExample]] | None = None
+ if not has_offset_loader:
+ in_memory_groups = OrderedDict()
+ conversation_summaries = self._collect_conversation_summaries(
+ source,
+ fallback_source=fallback_source,
+ in_memory_groups=in_memory_groups,
+ )
+ source_entries.append((source, conversation_summaries, in_memory_groups))
+
+ total_rows = sum(int(count or 0) for count in self._source_row_counts.values())
+ unique_source_count = max(1, len(self._source_row_counts))
+ sampling_source_factors = {
+ source_dataset: max(
+ 1.0,
+ _smoothed_inverse_share_factor(
+ count=row_count,
+ total=total_rows,
+ unique_count=unique_source_count,
+ alpha=float(self._config.get("sampling_source_alpha", 0.35) or 0.35),
+ blend_uniform_ratio=float(self._config.get("sampling_blend_uniform_ratio", 0.35) or 0.35),
+ ),
+ )
+ for source_dataset, row_count in self._source_row_counts.items()
+ }
+ loss_source_factors = {
+ source_dataset: _smoothed_inverse_share_factor(
+ count=row_count,
+ total=total_rows,
+ unique_count=unique_source_count,
+ alpha=float(self._config.get("loss_source_alpha", 0.4) or 0.4),
+ blend_uniform_ratio=float(self._config.get("loss_blend_uniform_ratio", 0.25) or 0.25),
+ )
+ for source_dataset, row_count in self._source_row_counts.items()
+ }
+
+ for source, conversation_summaries, in_memory_groups in source_entries:
+ has_offset_loader = callable(getattr(source, "conversation_offset_items", None)) and callable(
+ getattr(source, "_load_batch_from_offsets", None)
+ )
+ if has_offset_loader:
+ for conversation_id, offsets in source.conversation_offset_items(shuffle=False, seed=0):
+ summary = conversation_summaries.get(clean_text(conversation_id))
+ if summary is None:
+ continue
+ sampling_multiplier = self._conversation_sampling_multiplier(
+ summary,
+ sampling_source_factors=sampling_source_factors,
+ )
+ repeat_count = _conversation_repeat_count(
+ summary.conversation_id,
+ sampling_multiplier=sampling_multiplier,
+ max_group_repeat=int(self._config.get("sampling_max_group_repeat", 2) or 2),
+ )
+ self._groups.append(
+ _TrainingConversationGroup(
+ conversation_id=summary.conversation_id,
+ source_dataset=summary.source_dataset,
+ row_count=int(summary.row_count),
+ time_example_count=int(summary.time_example_count),
+ multi_evidence_count=int(summary.multi_evidence_count),
+ temporal_positive_count=int(summary.temporal_positive_count),
+ answer_type_counts=dict(summary.answer_type_counts),
+ repeat_count=repeat_count,
+ sampling_multiplier=float(sampling_multiplier),
+ source_store=source,
+ offsets=tuple(int(offset) for offset in list(offsets or []) if int(offset) >= 0),
+ )
+ )
+ else:
+ for conversation_id, rows in (in_memory_groups or {}).items():
+ summary = conversation_summaries.get(clean_text(conversation_id))
+ if summary is None:
+ continue
+ sampling_multiplier = self._conversation_sampling_multiplier(
+ summary,
+ sampling_source_factors=sampling_source_factors,
+ )
+ repeat_count = _conversation_repeat_count(
+ summary.conversation_id,
+ sampling_multiplier=sampling_multiplier,
+ max_group_repeat=int(self._config.get("sampling_max_group_repeat", 2) or 2),
+ )
+ self._groups.append(
+ _TrainingConversationGroup(
+ conversation_id=summary.conversation_id,
+ source_dataset=summary.source_dataset,
+ row_count=int(summary.row_count),
+ time_example_count=int(summary.time_example_count),
+ multi_evidence_count=int(summary.multi_evidence_count),
+ temporal_positive_count=int(summary.temporal_positive_count),
+ answer_type_counts=dict(summary.answer_type_counts),
+ repeat_count=repeat_count,
+ sampling_multiplier=float(sampling_multiplier),
+ rows=tuple(rows),
+ )
+ )
+ self._groups.sort(key=lambda item: item.conversation_id)
+ self._effective_row_count = sum(int(group.row_count) * int(group.repeat_count) for group in self._groups)
+ self._sampling_summary = self._build_sampling_summary(
+ total_rows=total_rows,
+ sampling_source_factors=sampling_source_factors,
+ loss_source_factors=loss_source_factors,
+ )
+ self._loss_source_factors = loss_source_factors
+
+ def _conversation_sampling_multiplier(
+ self,
+ summary: _TrainingConversationSummary,
+ *,
+ sampling_source_factors: Mapping[str, float],
+ ) -> float:
+ source_factor = max(1.0, float(sampling_source_factors.get(summary.source_dataset, 1.0) or 1.0))
+ boost = 1.0
+ time_signal = min(1.0, float(summary.time_example_count) / 2.0)
+ multi_signal = min(1.0, float(summary.multi_evidence_count))
+ temporal_signal = min(1.0, float(summary.temporal_positive_count) / 2.0)
+ boost += max(0.0, float(self._config.get("sampling_time_boost", 1.2) or 1.2) - 1.0) * time_signal
+ boost += max(0.0, float(self._config.get("sampling_multi_evidence_boost", 1.45) or 1.45) - 1.0) * multi_signal
+ boost += max(0.0, float(self._config.get("sampling_temporal_positive_boost", 1.15) or 1.15) - 1.0) * temporal_signal
+ raw_multiplier = source_factor * boost
+ return _bounded_training_weight(
+ raw_multiplier,
+ minimum=1.0,
+ maximum=float(self._config.get("sampling_max_conversation_multiplier", 2.0) or 2.0),
+ )
+
+ def _example_training_weight(self, example: QueryTrainingExample, *, fallback_source: str) -> float:
+ source_dataset = _training_source_dataset_name(example=example, fallback_source=fallback_source)
+ answer_type = _training_answer_type_name(example)
+ raw_weight = float(self._loss_source_factors.get(source_dataset, 1.0) or 1.0)
+ if answer_type == "time":
+ raw_weight *= float(self._config.get("loss_time_boost", 1.45) or 1.45)
+ elif answer_type == "multi_evidence" or _has_multi_evidence_supervision(example):
+ raw_weight *= float(self._config.get("loss_multi_evidence_boost", 1.75) or 1.75)
+ if _has_positive_temporal_supervision(example):
+ raw_weight *= float(self._config.get("loss_temporal_positive_boost", 1.2) or 1.2)
+ resolved_power = max(0.0, float(self._config.get("loss_weight_power", 0.5) or 0.5))
+ if resolved_power <= 0.0:
+ softened_weight = 1.0
+ elif abs(resolved_power - 1.0) <= 1e-9:
+ softened_weight = raw_weight
+ else:
+ softened_weight = math.pow(max(raw_weight, 1e-12), resolved_power)
+ return _bounded_training_weight(
+ softened_weight,
+ minimum=float(self._config.get("loss_min_example_weight", 0.65) or 0.65),
+ maximum=float(self._config.get("loss_max_example_weight", 1.85) or 1.85),
+ )
+
+ def _annotate_rows_for_training(
+ self,
+ rows: Sequence[QueryTrainingExample],
+ *,
+ group: _TrainingConversationGroup,
+ clone_rows: bool,
+ shuffle: bool,
+ rng: random.Random,
+ ) -> List[QueryTrainingExample]:
+ prepared_rows: List[QueryTrainingExample] = []
+ for row in rows:
+ example_weight = self._example_training_weight(row, fallback_source=group.source_dataset)
+ metadata_updates = {
+ "_training_enabled": True,
+ "_training_source_dataset": group.source_dataset,
+ "_training_sampling_multiplier": float(group.sampling_multiplier),
+ "_training_repeat_count": int(group.repeat_count),
+ "_training_example_weight": float(example_weight),
+ }
+ if clone_rows:
+ prepared_rows.append(_clone_query_training_example(row, metadata_updates=metadata_updates))
+ else:
+ row.metadata = {**dict(row.metadata or {}), **metadata_updates}
+ prepared_rows.append(row)
+ if shuffle and len(prepared_rows) > 1:
+ rng.shuffle(prepared_rows)
+ return prepared_rows
+
+ def _materialize_group_rows(
+ self,
+ group: _TrainingConversationGroup,
+ *,
+ shuffle: bool,
+ rng: random.Random,
+ ) -> List[QueryTrainingExample]:
+ if group.source_store is not None and group.offsets:
+ rows = group.source_store._load_batch_from_offsets(group.offsets)
+ return self._annotate_rows_for_training(rows, group=group, clone_rows=False, shuffle=shuffle, rng=rng)
+ rows = list(group.rows or ())
+ return self._annotate_rows_for_training(rows, group=group, clone_rows=True, shuffle=shuffle, rng=rng)
+
+ def _load_group_batch(
+ self,
+ group_specs: Sequence[_TrainingConversationGroup],
+ *,
+ shuffle: bool,
+ rng: random.Random,
+ ) -> List[QueryTrainingExample]:
+ rows: List[QueryTrainingExample] = []
+ for group in group_specs:
+ rows.extend(self._materialize_group_rows(group, shuffle=shuffle, rng=rng))
+ return rows
+
+ def iter_batches(
+ self,
+ *,
+ batch_size: int,
+ shuffle: bool,
+ seed: int,
+ ) -> Iterator[List[QueryTrainingExample]]:
+ resolved_batch_size = max(1, int(batch_size))
+ rng = random.Random(seed)
+ expanded_groups: List[_TrainingConversationGroup] = []
+ for group in self._groups:
+ repeat_count = max(1, int(group.repeat_count))
+ for _ in range(repeat_count):
+ expanded_groups.append(group)
+ if shuffle:
+ rng.shuffle(expanded_groups)
+ current_batch: List[_TrainingConversationGroup] = []
+ current_batch_size = 0
+ for group in expanded_groups:
+ if int(group.row_count) > resolved_batch_size:
+ if current_batch:
+ yield self._load_group_batch(current_batch, shuffle=shuffle, rng=rng)
+ current_batch = []
+ current_batch_size = 0
+ oversized_rows = self._materialize_group_rows(group, shuffle=shuffle, rng=rng)
+ for index in range(0, len(oversized_rows), resolved_batch_size):
+ yield oversized_rows[index : index + resolved_batch_size]
+ continue
+ if current_batch and current_batch_size + int(group.row_count) > resolved_batch_size:
+ yield self._load_group_batch(current_batch, shuffle=shuffle, rng=rng)
+ current_batch = []
+ current_batch_size = 0
+ current_batch.append(group)
+ current_batch_size += int(group.row_count)
+ if current_batch:
+ yield self._load_group_batch(current_batch, shuffle=shuffle, rng=rng)
+
+ def _build_sampling_summary(
+ self,
+ *,
+ total_rows: int,
+ sampling_source_factors: Mapping[str, float],
+ loss_source_factors: Mapping[str, float],
+ ) -> Dict[str, Any]:
+ sampled_source_rows: Counter[str] = Counter()
+ repeat_histogram: Counter[str] = Counter()
+ weighted_answer_mass: Counter[str] = Counter()
+ weighted_supervision_bucket_mass: Counter[str] = Counter()
+ weighted_source_mass: Counter[str] = Counter()
+ weight_floor = float(self._config.get("loss_min_example_weight", 0.65) or 0.65)
+ weight_cap = float(self._config.get("loss_max_example_weight", 1.85) or 1.85)
+ observed_min_weight = weight_cap
+ observed_max_weight = weight_floor
+
+ for group in self._groups:
+ sampled_source_rows[group.source_dataset] += int(group.row_count) * int(group.repeat_count)
+ repeat_histogram[str(int(group.repeat_count))] += 1
+
+ for (source_dataset, answer_type, has_temporal_positive), count in self._source_answer_temporal_counts.items():
+ if count <= 0:
+ continue
+ raw_weight = float(loss_source_factors.get(source_dataset, 1.0) or 1.0)
+ if answer_type == "time":
+ raw_weight *= float(self._config.get("loss_time_boost", 1.45) or 1.45)
+ elif answer_type == "multi_evidence":
+ raw_weight *= float(self._config.get("loss_multi_evidence_boost", 1.75) or 1.75)
+ if has_temporal_positive:
+ raw_weight *= float(self._config.get("loss_temporal_positive_boost", 1.2) or 1.2)
+ resolved_power = max(0.0, float(self._config.get("loss_weight_power", 0.5) or 0.5))
+ if resolved_power <= 0.0:
+ example_weight = 1.0
+ elif abs(resolved_power - 1.0) <= 1e-9:
+ example_weight = raw_weight
+ else:
+ example_weight = math.pow(max(raw_weight, 1e-12), resolved_power)
+ example_weight = _bounded_training_weight(example_weight, minimum=weight_floor, maximum=weight_cap)
+ observed_min_weight = min(observed_min_weight, float(example_weight))
+ observed_max_weight = max(observed_max_weight, float(example_weight))
+ weighted_answer_mass[answer_type] += float(count) * float(example_weight)
+ weighted_source_mass[source_dataset] += float(count) * float(example_weight)
+ for (source_dataset, supervision_bucket, has_temporal_positive), count in self._source_supervision_temporal_counts.items():
+ if count <= 0:
+ continue
+ raw_weight = float(loss_source_factors.get(source_dataset, 1.0) or 1.0)
+ if supervision_bucket == "time":
+ raw_weight *= float(self._config.get("loss_time_boost", 1.45) or 1.45)
+ elif supervision_bucket in {"multi_evidence", "list_event_text_multi_positive"} or supervision_bucket.endswith("_multi_positive"):
+ raw_weight *= float(self._config.get("loss_multi_evidence_boost", 1.75) or 1.75)
+ if has_temporal_positive:
+ raw_weight *= float(self._config.get("loss_temporal_positive_boost", 1.2) or 1.2)
+ resolved_power = max(0.0, float(self._config.get("loss_weight_power", 0.5) or 0.5))
+ if resolved_power <= 0.0:
+ example_weight = 1.0
+ elif abs(resolved_power - 1.0) <= 1e-9:
+ example_weight = raw_weight
+ else:
+ example_weight = math.pow(max(raw_weight, 1e-12), resolved_power)
+ example_weight = _bounded_training_weight(example_weight, minimum=weight_floor, maximum=weight_cap)
+ weighted_supervision_bucket_mass[supervision_bucket] += float(count) * float(example_weight)
+
+ source_details: Dict[str, Any] = {}
+ for source_dataset, row_count in self._source_row_counts.items():
+ sampled_rows = int(sampled_source_rows.get(source_dataset, row_count) or 0)
+ weighted_loss_mass = float(weighted_source_mass.get(source_dataset, 0.0) or 0.0)
+ source_details[source_dataset] = {
+ "row_count": int(row_count),
+ "row_share": round(float(row_count) / float(max(1, total_rows)), 6),
+ "sampling_source_factor": round(float(sampling_source_factors.get(source_dataset, 1.0) or 1.0), 6),
+ "loss_source_factor": round(float(loss_source_factors.get(source_dataset, 1.0) or 1.0), 6),
+ "sampled_row_count_estimate": int(sampled_rows),
+ "sampled_row_share_estimate": round(float(sampled_rows) / float(max(1, self._effective_row_count)), 6),
+ "weighted_loss_mass": round(weighted_loss_mass, 6),
+ "avg_example_weight_estimate": round(weighted_loss_mass / float(max(1, row_count)), 6),
+ }
+
+ return {
+ "mode": clean_text(self._config.get("train_sampling_mode", "")) or "uniform",
+ "base_row_count": int(total_rows),
+ "effective_row_count": int(self._effective_row_count),
+ "source_count": int(len(self._source_row_counts)),
+ "source_row_counts": {key: int(value) for key, value in sorted(self._source_row_counts.items())},
+ "answer_type_counts": {key: int(value) for key, value in sorted(self._answer_type_counts.items())},
+ "supervision_bucket_counts": {key: int(value) for key, value in sorted(self._supervision_bucket_counts.items())},
+ "repeat_histogram": {key: int(value) for key, value in sorted(repeat_histogram.items(), key=lambda item: int(item[0]))},
+ "weighted_answer_mass_estimate": {key: round(float(value), 6) for key, value in sorted(weighted_answer_mass.items())},
+ "weighted_supervision_bucket_mass_estimate": {
+ key: round(float(value), 6) for key, value in sorted(weighted_supervision_bucket_mass.items())
+ },
+ "loss_weight_range_estimate": {
+ "min": round(float(observed_min_weight if self._source_answer_temporal_counts else 1.0), 6),
+ "max": round(float(observed_max_weight if self._source_answer_temporal_counts else 1.0), 6),
+ },
+ "source_details": source_details,
+ }
+
+
+def maybe_build_source_aware_training_store(
+ rows: Sequence[QueryTrainingExample],
+ *,
+ config: Mapping[str, Any] | None = None,
+) -> Sequence[QueryTrainingExample]:
+ resolved_config = {**DEFAULT_TRAINING_CONFIG, **dict(config or {})}
+ mode = clean_text(resolved_config.get("train_sampling_mode", ""))
+ if not mode or mode == "uniform":
+ return rows
+ return SourceAwareTrainingStore(rows, config=resolved_config)
+
+
+def load_query_examples(
+ path: Path,
+ *,
+ skip_bad_rows: bool = False,
+ error_callback: Callable[[Dict[str, Any]], None] | None = None,
+) -> IndexedQueryExampleStore:
+ return IndexedQueryExampleStore(path, skip_bad_rows=skip_bad_rows, error_callback=error_callback)
+
+
+def _materialize_query_batch_spec(batch_spec: QueryBatchSpec) -> List[QueryTrainingExample]:
+ rows: List[QueryTrainingExample] = []
+ for source in list(batch_spec.sources or ()):
+ resolved_path = Path(clean_text(source.path))
+ resolved_offsets = [int(offset) for offset in list(source.offsets or ()) if int(offset) >= 0]
+ if not resolved_offsets or not str(resolved_path):
+ continue
+ with resolved_path.open("rb") as handle:
+ for offset in resolved_offsets:
+ handle.seek(offset)
+ raw_line = handle.readline()
+ try:
+ rows.append(_load_query_example_from_raw_line(resolved_path, int(offset), raw_line))
+ except Exception:
+ if not bool(source.skip_bad_rows):
+ raise
+ return rows
+
+
+def _materialize_query_batch_payload(
+ batch_payload: QueryBatchSpec | Sequence[QueryTrainingExample],
+) -> List[QueryTrainingExample]:
+ if isinstance(batch_payload, QueryBatchSpec):
+ return _materialize_query_batch_spec(batch_payload)
+ return list(batch_payload)
+
+
+def _build_grad_scaler(*, enabled: bool, device: torch.device) -> Any:
+ amp_module = getattr(torch, "amp", None)
+ if amp_module is not None and hasattr(amp_module, "GradScaler"):
+ return amp_module.GradScaler(device.type, enabled=enabled)
+ return torch.cuda.amp.GradScaler(enabled=enabled)
+
+
+def _autocast_context(*, enabled: bool, device: torch.device) -> Any:
+ if not enabled:
+ return nullcontext()
+ amp_module = getattr(torch, "amp", None)
+ if amp_module is not None and hasattr(amp_module, "autocast"):
+ return amp_module.autocast(device_type=device.type, enabled=True)
+ return torch.cuda.amp.autocast(enabled=True)
+
+
+def _event_labels(example: QueryTrainingExample, candidate_event_ids: Sequence[str], *, device: torch.device) -> Tensor:
+ positive_ids = {clean_text(item) for item in example.positive_event_ids if clean_text(item)}
+ return torch.tensor(
+ [1.0 if event_id in positive_ids else 0.0 for event_id in candidate_event_ids],
+ dtype=torch.float32,
+ device=device,
+ )
+
+
+def _balanced_binary_loss(logits: Tensor, targets: Tensor) -> Tensor:
+ if logits.numel() == 0:
+ return logits.sum() * 0.0
+ working_logits = logits.float()
+ working_targets = targets.float()
+ positive_count = int((working_targets > 0.5).sum().item())
+ negative_count = int(working_targets.numel() - positive_count)
+ if positive_count > 0 and negative_count > 0:
+ pos_weight = torch.tensor(
+ float(negative_count) / float(max(1, positive_count)),
+ dtype=working_logits.dtype,
+ device=working_logits.device,
+ )
+ return F.binary_cross_entropy_with_logits(working_logits, working_targets, pos_weight=pos_weight)
+ return F.binary_cross_entropy_with_logits(working_logits, working_targets)
+
+
+def _pairwise_margin_loss(logits: Tensor, targets: Tensor, *, margin: float = 0.2) -> Tensor:
+ if logits.numel() == 0:
+ return logits.sum() * 0.0
+ working_logits = logits.float()
+ working_targets = targets.float()
+ positive_logits = working_logits[working_targets > 0.5]
+ negative_logits = working_logits[working_targets <= 0.5]
+ if positive_logits.numel() == 0 or negative_logits.numel() == 0:
+ return working_logits.sum() * 0.0
+ margins = float(margin) - (positive_logits.unsqueeze(-1) - negative_logits.unsqueeze(0))
+ return F.relu(margins).mean()
+
+
+def _positive_mass_loss(logits: Tensor, targets: Tensor) -> Tensor:
+ if logits.numel() == 0:
+ return logits.sum() * 0.0
+ working_logits = logits.float()
+ positive_mask = targets.float() > 0.5
+ if not bool(torch.any(positive_mask)):
+ return working_logits.sum() * 0.0
+ positive_logits = working_logits.masked_fill(~positive_mask, float("-inf"))
+ return torch.logsumexp(working_logits, dim=-1) - torch.logsumexp(positive_logits, dim=-1)
+
+
+def _recall_loss(logits: Tensor, targets: Tensor) -> Tensor:
+ return 0.5 * _balanced_binary_loss(logits, targets) + 0.5 * _positive_mass_loss(logits, targets)
+
+
+def _ranking_binary_loss(
+ logits: Tensor,
+ targets: Tensor,
+ *,
+ margin: float = 0.2,
+ bce_weight: float = 0.4,
+ positive_mass_weight: float = 0.3,
+ pairwise_weight: float = 0.3,
+) -> Tensor:
+ return (
+ float(bce_weight) * _balanced_binary_loss(logits, targets)
+ + float(positive_mass_weight) * _positive_mass_loss(logits, targets)
+ + float(pairwise_weight) * _pairwise_margin_loss(logits, targets, margin=margin)
+ )
+
+
+def _path_labels(example: QueryTrainingExample, candidate_path_ids: Sequence[str], *, device: torch.device) -> Tensor:
+ positive_ids = {clean_text(item) for item in example.positive_path_ids if clean_text(item)}
+ return torch.tensor(
+ [1.0 if path_id in positive_ids else 0.0 for path_id in candidate_path_ids],
+ dtype=torch.float32,
+ device=device,
+ )
+
+
+def _apply_trainable_stage(model: nn.Module, stage: str | None) -> Dict[str, Any]:
+ normalized_stage = clean_text(stage or "all").lower().replace("-", "_")
+ if normalized_stage in {"", "all", "full"}:
+ for parameter in model.parameters():
+ parameter.requires_grad = True
+ trainable_names = [name for name, parameter in model.named_parameters() if parameter.requires_grad]
+ return {
+ "trainable_stage": "all",
+ "trainable_parameter_count": int(sum(parameter.numel() for parameter in model.parameters() if parameter.requires_grad)),
+ "frozen_parameter_count": 0,
+ "trainable_name_prefixes": sorted({name.split(".", 1)[0] for name in trainable_names}),
+ }
+ if normalized_stage == "recall_only":
+ allowed_prefixes = ("event_recall_head.",)
+ elif normalized_stage == "recall_and_question_projection":
+ allowed_prefixes = ("event_recall_head.", "question_projection.")
+ elif normalized_stage == "selection_and_fusion":
+ allowed_prefixes = (
+ "answer_type_head.",
+ "event_calibration_head.",
+ "event_distractor_head.",
+ "event_head.",
+ "event_matrix_head.",
+ "event_pair_adapter.",
+ "event_subgraph_refiner.",
+ "event_tunnel_head.",
+ "final_event_fusion_head.",
+ "final_path_fusion_head.",
+ "memory_router_head.",
+ "path_calibration_head.",
+ "path_head.",
+ "path_matrix_head.",
+ "path_pair_adapter.",
+ "path_tunnel_head.",
+ "question_intent_head.",
+ "temporal_head.",
+ )
+ elif normalized_stage == "profile_recall_selection_plan":
+ allowed_prefixes = (
+ "answer_plan_head.",
+ "event_head.",
+ "event_matrix_head.",
+ "event_recall_head.",
+ "event_tunnel_head.",
+ "final_event_fusion_head.",
+ "final_path_fusion_head.",
+ "memory_router_head.",
+ "path_head.",
+ "path_matrix_head.",
+ "path_tunnel_head.",
+ "question_intent_head.",
+ "temporal_head.",
+ )
+ elif normalized_stage == "tunnel_fusion_only":
+ allowed_prefixes = (
+ "event_tunnel_head.",
+ "final_event_fusion_head.",
+ "final_path_fusion_head.",
+ "path_tunnel_head.",
+ )
+ elif normalized_stage == "path_tunnel_only":
+ allowed_prefixes = (
+ "final_path_fusion_head.",
+ "path_calibration_head.",
+ "path_head.",
+ "path_matrix_head.",
+ "path_pair_adapter.",
+ "path_tunnel_head.",
+ )
+ elif normalized_stage == "answer_plan_only":
+ allowed_prefixes = ("answer_plan_head.",)
+ elif normalized_stage == "memory_router_only":
+ allowed_prefixes = ("memory_router_head.",)
+ else:
+ raise ValueError(
+ "Unsupported trainable_stage "
+ f"{stage!r}; expected one of: all, recall_only, recall_and_question_projection, selection_and_fusion, profile_recall_selection_plan, tunnel_fusion_only, path_tunnel_only, answer_plan_only, memory_router_only"
+ )
+ trainable_parameter_count = 0
+ frozen_parameter_count = 0
+ trainable_prefixes: set[str] = set()
+ for name, parameter in model.named_parameters():
+ should_train = any(name.startswith(prefix) for prefix in allowed_prefixes)
+ parameter.requires_grad = bool(should_train)
+ if should_train:
+ trainable_parameter_count += int(parameter.numel())
+ trainable_prefixes.add(name.split(".", 1)[0])
+ else:
+ frozen_parameter_count += int(parameter.numel())
+ if trainable_parameter_count <= 0:
+ raise RuntimeError(f"trainable_stage={stage!r} did not leave any trainable parameters")
+ return {
+ "trainable_stage": normalized_stage,
+ "trainable_parameter_count": int(trainable_parameter_count),
+ "frozen_parameter_count": int(frozen_parameter_count),
+ "trainable_name_prefixes": sorted(trainable_prefixes),
+ }
+
+
+def _temporal_path_labels(
+ example: QueryTrainingExample,
+ candidate_temporal_path_ids: Sequence[str],
+ candidate_temporal_node_ids: Sequence[str],
+ *,
+ device: torch.device,
+) -> Tensor:
+ positive_event_ids = {clean_text(item) for item in example.positive_event_ids if clean_text(item)}
+ positive_time_node_ids = {clean_text(item) for item in example.positive_time_node_ids if clean_text(item)}
+ temporal_path_targets: List[float] = []
+ for path_id, time_node_id in zip(candidate_temporal_path_ids, candidate_temporal_node_ids):
+ event_id, _, _ = parse_path_id(path_id)
+ is_positive = bool(event_id in positive_event_ids and time_node_id in positive_time_node_ids)
+ temporal_path_targets.append(1.0 if is_positive else 0.0)
+ return torch.tensor(temporal_path_targets, dtype=torch.float32, device=device)
+
+
+def _is_temporal_example(example: QueryTrainingExample) -> bool:
+ answer_type = clean_text(dict(example.answer_targets or {}).get("answer_type", ""))
+ temporal_target = dict(example.temporal_target or {})
+ if "use_temporal_head" in temporal_target:
+ return bool(temporal_target.get("use_temporal_head", False))
+ question_is_temporal = bool(
+ temporal_target.get("question_is_temporal", temporal_target.get("is_temporal", False))
+ ) or bool(dict(example.question_features or {}).get("is_temporal", False))
+ has_positive_supervision = bool(
+ temporal_target.get("has_positive_time_supervision", False)
+ ) or bool(list(example.positive_time_node_ids or []))
+ return question_is_temporal and (has_positive_supervision or answer_type == "abstain")
+
+
+def _answer_type_label(example: QueryTrainingExample) -> int:
+ answer_type = clean_text(dict(example.answer_targets or {}).get("answer_type", "")) or "abstain"
+ return ANSWER_TYPE_TO_ID.get(answer_type, ANSWER_TYPE_TO_ID["abstain"])
+
+
+def _coerce_memory_router_targets(value: Any) -> Dict[str, float]:
+ targets = {layer: 0.0 for layer in MEMORY_ROUTER_LAYERS}
+ if isinstance(value, Mapping):
+ for key, raw in value.items():
+ layer = clean_text(key).lower().replace("-", "_").replace(" ", "_")
+ if layer in targets:
+ targets[layer] = 1.0 if bool(raw) else 0.0
+ return targets
+ if isinstance(value, str):
+ raw_items = re.split(r"[\s,;|/]+", value)
+ else:
+ raw_items = list(value or []) if isinstance(value, (list, tuple, set)) else []
+ for item in raw_items:
+ layer = clean_text(item).lower().replace("-", "_").replace(" ", "_")
+ if layer in targets:
+ targets[layer] = 1.0
+ return targets
+
+
+def _memory_router_targets_for_example(
+ example: QueryTrainingExample,
+ supervision: Mapping[str, Any] | None = None,
+) -> Dict[str, float]:
+ answer_targets = dict(example.answer_targets or {})
+ metadata = dict(example.metadata or {})
+ question_features = dict(example.question_features or {})
+ explicit = (
+ answer_targets.get("memory_router_targets")
+ or metadata.get("memory_router_targets")
+ or question_features.get("memory_router_targets")
+ )
+ if explicit:
+ targets = _coerce_memory_router_targets(explicit)
+ if any(value > 0.0 for value in targets.values()):
+ return targets
+
+ supervision_payload = dict(supervision or {})
+ positive_event_ids = set(supervision_payload.get("positive_event_ids", example.positive_event_ids) or [])
+ positive_path_ids = set(supervision_payload.get("positive_path_ids", example.positive_path_ids) or [])
+ answer_type = _training_answer_type_name(example)
+ semantic_slot = clean_text(question_features.get("semantic_slot_target", "")).lower()
+ training_focus = clean_text(metadata.get("training_focus", "")).lower()
+ source_dataset = clean_text(metadata.get("source_dataset", "")).lower()
+ domain = clean_text(metadata.get("domain", "")).lower()
+ targets = {layer: 0.0 for layer in MEMORY_ROUTER_LAYERS}
+ if positive_event_ids or answer_type in {"event_text", "multi_evidence", "profile", "time"}:
+ targets["event"] = 1.0
+ if answer_type == "profile" or semantic_slot == "profile" or "profile" in training_focus:
+ targets["profile"] = 1.0
+ if (
+ bool(question_features.get("implicit_profile_resource_query", False))
+ or "resource" in training_focus
+ or any(marker in domain for marker in ("resource", "tool", "gear", "device", "accessor", "ingredient"))
+ ):
+ targets["resource"] = 1.0
+ if answer_type == "time" or _is_temporal_example(example):
+ targets["temporal"] = 1.0
+ if positive_path_ids and (len(positive_event_ids) > 1 or answer_type in {"profile", "multi_evidence"}):
+ targets["path_tunnel"] = 1.0
+ if len(positive_event_ids) > 1 or any(marker in source_dataset for marker in ("longmemeval", "multi_session", "locomo")):
+ targets["topic_tunnel"] = 1.0
+ if not any(value > 0.0 for value in targets.values()):
+ targets["event"] = 1.0
+ return targets
+
+
+def _memory_router_target_tensor(
+ example: QueryTrainingExample,
+ supervision: Mapping[str, Any] | None,
+ *,
+ device: torch.device,
+) -> Tensor:
+ targets = _memory_router_targets_for_example(example, supervision)
+ return torch.tensor(
+ [float(targets.get(layer, 0.0) or 0.0) for layer in MEMORY_ROUTER_LAYERS],
+ dtype=torch.float32,
+ device=device,
+ )
+
+
+def _batched_logit_statistics(logits: Tensor, *, mask: Tensor | None = None) -> Tensor:
+ if logits.dim() != 2:
+ raise ValueError(f"Expected rank-2 logits, got {logits.dim()}")
+ batch_size = int(logits.size(0))
+ if logits.numel() == 0:
+ return torch.zeros((batch_size, 2), dtype=logits.dtype, device=logits.device)
+ scores = torch.sigmoid(logits.float())
+ if mask is not None:
+ scores = scores.masked_fill(~mask, 0.0)
+ available = mask.any(dim=1).to(dtype=scores.dtype)
+ else:
+ available = torch.ones((batch_size,), dtype=scores.dtype, device=scores.device)
+ top_k = min(2, int(scores.size(1)))
+ if top_k <= 0:
+ return torch.zeros((batch_size, 2), dtype=logits.dtype, device=logits.device)
+ top_values = torch.topk(scores, k=top_k, dim=1).values
+ top1 = top_values[:, 0] * available
+ top2 = top_values[:, 1] * available if top_values.size(1) > 1 else torch.zeros_like(top1)
+ return torch.stack([top1.to(dtype=logits.dtype), (top1 - top2).to(dtype=logits.dtype)], dim=-1)
+
+
+def _batched_probability_competition_features(logits: Tensor, *, mask: Tensor | None = None) -> Tensor:
+ if logits.dim() != 2:
+ raise ValueError(f"Expected rank-2 logits, got {logits.dim()}")
+ batch_size = int(logits.size(0))
+ if logits.numel() == 0 or logits.size(1) <= 0:
+ return torch.zeros((batch_size, 4), dtype=logits.dtype, device=logits.device)
+ working_logits = logits.float()
+ if mask is not None:
+ valid_mask = mask.bool()
+ else:
+ valid_mask = torch.ones_like(working_logits, dtype=torch.bool)
+ available = valid_mask.any(dim=1)
+ masked_logits = working_logits.masked_fill(~valid_mask, float("-inf"))
+ probs = torch.softmax(masked_logits, dim=1)
+ probs = torch.where(valid_mask, probs, torch.zeros_like(probs))
+ if probs.size(1) <= 0:
+ return torch.zeros((batch_size, 4), dtype=logits.dtype, device=logits.device)
+ top_k = min(3, int(probs.size(1)))
+ top_values = torch.topk(probs, k=top_k, dim=1).values
+ top1 = top_values[:, 0]
+ top2 = top_values[:, 1] if top_values.size(1) > 1 else torch.zeros_like(top1)
+ top3_mass = top_values.sum(dim=1)
+ entropy = -(probs * torch.log(probs.clamp_min(1e-8))).sum(dim=1)
+ valid_counts = valid_mask.sum(dim=1).clamp_min(1).to(dtype=entropy.dtype)
+ entropy = entropy / torch.log(valid_counts + 1e-8).clamp_min(1.0)
+ features = torch.stack([top1, top1 - top2, entropy, top3_mass], dim=-1)
+ return torch.where(available.unsqueeze(-1), features.to(dtype=logits.dtype), torch.zeros_like(features, dtype=logits.dtype))
+
+
+def _masked_feature_mean(features: Tensor, mask: Tensor | None, feature_index: int) -> Tensor:
+ batch_size = int(features.size(0))
+ if features.numel() == 0:
+ return torch.zeros((batch_size, 1), dtype=features.dtype, device=features.device)
+ values = features[:, :, feature_index]
+ if mask is None:
+ return values.mean(dim=1, keepdim=True)
+ valid_mask = mask.bool()
+ denom = valid_mask.sum(dim=1, keepdim=True).clamp_min(1).to(dtype=features.dtype)
+ total = values.masked_fill(~valid_mask, 0.0).sum(dim=1, keepdim=True)
+ available = valid_mask.any(dim=1, keepdim=True)
+ return torch.where(available, total / denom, torch.zeros_like(total))
+
+
+def _zero_center_masked_logits(values: Tensor, mask: Tensor | None) -> Tensor:
+ if values.dim() != 2:
+ raise ValueError(f"Expected rank-2 values, got {values.dim()}")
+ if values.numel() == 0:
+ return values
+ if mask is None:
+ if values.size(1) <= 1:
+ return values
+ return values - values.mean(dim=1, keepdim=True)
+ valid_mask = mask.bool()
+ valid_counts = valid_mask.sum(dim=1, keepdim=True)
+ multi_candidate = valid_counts > 1
+ denom = valid_counts.clamp_min(1).to(dtype=values.dtype)
+ mean = values.masked_fill(~valid_mask, 0.0).sum(dim=1, keepdim=True) / denom
+ centered = torch.where(multi_candidate, values - mean, values)
+ return centered.masked_fill(~valid_mask, 0.0)
+
+
+def _masked_feature_max(features: Tensor, mask: Tensor | None, feature_index: int) -> Tensor:
+ batch_size = int(features.size(0))
+ if features.numel() == 0:
+ return torch.zeros((batch_size, 1), dtype=features.dtype, device=features.device)
+ values = features[:, :, feature_index]
+ if mask is None:
+ return values.max(dim=1, keepdim=True).values
+ valid_mask = mask.bool()
+ masked_values = values.masked_fill(~valid_mask, float("-inf"))
+ best = masked_values.max(dim=1, keepdim=True).values
+ available = valid_mask.any(dim=1, keepdim=True)
+ return torch.where(available, best, torch.zeros_like(best))
+
+
+def _gather_scores_by_indices(source_scores: Tensor, indices: Sequence[int], *, device: torch.device) -> Tensor:
+ count = len(indices)
+ if count <= 0:
+ return source_scores.new_zeros((0,))
+ source_count = int(source_scores.numel())
+ if source_count <= 0:
+ return source_scores.new_zeros((count,))
+ safe_indices = [index if 0 <= int(index) < source_count else 0 for index in indices]
+ index_tensor = torch.tensor(safe_indices, dtype=torch.long, device=device)
+ gathered = source_scores.index_select(0, index_tensor)
+ valid_mask = torch.tensor(
+ [0 <= int(index) < source_count for index in indices],
+ dtype=torch.bool,
+ device=device,
+ )
+ return gathered.masked_fill(~valid_mask, 0.0)
+
+
+def _gather_scores_by_id(
+ source_ids: Sequence[str],
+ source_scores: Tensor,
+ target_ids: Sequence[str],
+ *,
+ device: torch.device,
+) -> Tensor:
+ lookup = {clean_text(item_id): index for index, item_id in enumerate(source_ids) if clean_text(item_id)}
+ return _gather_scores_by_indices(
+ source_scores,
+ [lookup.get(clean_text(item_id), -1) for item_id in target_ids],
+ device=device,
+ )
+
+
+def _scatter_amax_by_indices(
+ values: Tensor,
+ indices: Sequence[int],
+ *,
+ size: int,
+ device: torch.device,
+) -> Tensor:
+ result = values.new_zeros((max(0, int(size)),))
+ if result.numel() == 0 or values.numel() == 0 or not indices:
+ return result
+ usable_count = min(int(values.numel()), len(indices))
+ if usable_count <= 0:
+ return result
+ safe_indices = [int(index) if 0 <= int(index) < int(size) else 0 for index in indices[:usable_count]]
+ valid_mask = torch.tensor(
+ [0 <= int(index) < int(size) for index in indices[:usable_count]],
+ dtype=torch.bool,
+ device=device,
+ )
+ if not bool(torch.any(valid_mask)):
+ return result
+ index_tensor = torch.tensor(safe_indices, dtype=torch.long, device=device)
+ value_tensor = values[:usable_count]
+ if hasattr(result, "scatter_reduce_"):
+ result.scatter_reduce_(0, index_tensor[valid_mask], value_tensor[valid_mask], reduce="amax", include_self=True)
+ return result
+ for index, value in zip(index_tensor[valid_mask].tolist(), value_tensor[valid_mask]):
+ result[index] = torch.maximum(result[index], value)
+ return result
+
+
+def _scatter_sum_by_indices(
+ indices: Sequence[int],
+ *,
+ size: int,
+ device: torch.device,
+ dtype: torch.dtype,
+) -> Tensor:
+ result = torch.zeros((max(0, int(size)),), dtype=dtype, device=device)
+ if result.numel() == 0 or not indices:
+ return result
+ safe_indices = [int(index) if 0 <= int(index) < int(size) else 0 for index in indices]
+ valid_mask = torch.tensor(
+ [0 <= int(index) < int(size) for index in indices],
+ dtype=torch.bool,
+ device=device,
+ )
+ if not bool(torch.any(valid_mask)):
+ return result
+ index_tensor = torch.tensor(safe_indices, dtype=torch.long, device=device)
+ ones = torch.ones((len(indices),), dtype=dtype, device=device)
+ result.scatter_add_(0, index_tensor[valid_mask], ones[valid_mask])
+ return result
+
+
+def _best_other_scores(scores: Tensor) -> Tensor:
+ if scores.numel() <= 1:
+ return scores.new_zeros(scores.shape)
+ top_values, top_indices = torch.topk(scores, k=min(2, int(scores.numel())), dim=0)
+ best_other = top_values[0].expand_as(scores).clone()
+ best_other[top_indices[0]] = top_values[1]
+ return best_other
+
+
+def _best_other_indices(scores: Tensor) -> Tensor:
+ if scores.numel() <= 1:
+ return torch.zeros(scores.shape, dtype=torch.long, device=scores.device)
+ top_indices = torch.topk(scores, k=min(2, int(scores.numel())), dim=0).indices
+ best_other = top_indices[0].expand(scores.shape).clone()
+ best_other[top_indices[0]] = top_indices[1]
+ return best_other.to(dtype=torch.long)
+
+
+def _relation_strength_matrix(relation_features: Tensor, weights: Sequence[float]) -> Tensor:
+ if relation_features.numel() == 0:
+ return relation_features.new_zeros(relation_features.shape[:-1])
+ weight_tensor = relation_features.new_tensor(list(weights), dtype=relation_features.dtype)
+ if relation_features.size(-1) != int(weight_tensor.numel()):
+ raise ValueError(
+ f"Relation feature width mismatch: expected {int(weight_tensor.numel())}, "
+ f"found {int(relation_features.size(-1))}"
+ )
+ normalizer = weight_tensor.sum().clamp_min(1e-8)
+ return ((relation_features * weight_tensor).sum(dim=-1) / normalizer).clamp_(0.0, 1.0)
+
+
+def _gather_1d_by_index(values: Tensor, indices: Tensor, *, valid_mask: Tensor | None = None) -> Tensor:
+ if values.numel() <= 0 or indices.numel() <= 0:
+ return values.new_zeros(indices.shape, dtype=values.dtype)
+ safe_indices = indices.clamp(min=0, max=max(0, int(values.numel()) - 1)).to(dtype=torch.long)
+ gathered = values.index_select(0, safe_indices)
+ if valid_mask is not None:
+ gathered = gathered.masked_fill(~valid_mask.bool(), 0.0)
+ return gathered
+
+
+def _reverse_competition_features(
+ scores: Tensor,
+ relation_strength: Tensor | None = None,
+ *,
+ valid_mask: Tensor | None = None,
+ relation_threshold: float = TRI_MAZE_RELATION_THRESHOLD,
+) -> tuple[Tensor, Tensor, Tensor, Tensor, Tensor]:
+ score_count = int(scores.numel())
+ if score_count <= 0:
+ empty_float = scores.new_zeros((0,))
+ empty_index = torch.zeros((0,), dtype=torch.long, device=scores.device)
+ return empty_float, empty_float, empty_float, empty_float, empty_index
+ active_mask = (
+ valid_mask.reshape(-1).bool()
+ if isinstance(valid_mask, Tensor)
+ else torch.ones((score_count,), dtype=torch.bool, device=scores.device)
+ )
+ available = active_mask.to(dtype=scores.dtype)
+ if int(active_mask.sum().item()) <= 1:
+ zero_scores = scores.new_zeros(scores.shape)
+ zero_indices = torch.zeros(scores.shape, dtype=torch.long, device=scores.device)
+ return zero_scores, zero_scores, zero_scores, zero_scores, zero_indices
+ working_scores = scores.masked_fill(~active_mask, float("-inf"))
+ fallback_scores = _best_other_scores(working_scores)
+ fallback_indices = _best_other_indices(working_scores)
+ fallback_scores = torch.where(torch.isfinite(fallback_scores), fallback_scores, torch.zeros_like(fallback_scores))
+ reverse_scores = fallback_scores
+ reverse_indices = fallback_indices
+ reverse_relations = scores.new_zeros(scores.shape)
+ if isinstance(relation_strength, Tensor) and relation_strength.numel() > 0:
+ relation = relation_strength[:score_count, :score_count].to(dtype=scores.dtype, device=scores.device)
+ pair_mask = active_mask.unsqueeze(0) & active_mask.unsqueeze(1)
+ pair_mask &= ~torch.eye(score_count, dtype=torch.bool, device=scores.device)
+ relation = relation.masked_fill(~pair_mask, 0.0)
+ related_mask = pair_mask & (relation >= float(relation_threshold))
+ related_scores = working_scores.unsqueeze(0).expand(score_count, -1).masked_fill(~related_mask, float("-inf"))
+ related_values, related_indices = related_scores.max(dim=1)
+ has_related = torch.isfinite(related_values)
+ reverse_scores = torch.where(has_related, related_values, fallback_scores)
+ reverse_indices = torch.where(has_related, related_indices, fallback_indices)
+ reverse_relations = relation.gather(1, reverse_indices.unsqueeze(-1)).squeeze(-1)
+ reverse_available = available.clone()
+ invalid_mask = ~active_mask
+ reverse_scores = reverse_scores.masked_fill(invalid_mask, 0.0)
+ reverse_relations = reverse_relations.masked_fill(invalid_mask, 0.0)
+ reverse_available = reverse_available.masked_fill(invalid_mask, 0.0)
+ boundary_gap = torch.where(
+ active_mask,
+ scores - reverse_scores,
+ torch.zeros_like(scores),
+ )
+ reverse_indices = reverse_indices.masked_fill(invalid_mask, 0)
+ return reverse_scores, boundary_gap, reverse_relations, reverse_available, reverse_indices
+
+
+def _build_answer_calibration_features(
+ *,
+ calibrated_event_logits_batch: Tensor,
+ calibrated_path_logits_batch: Tensor,
+ temporal_logits_batch: Tensor,
+ event_pair_features_batch_raw: Tensor,
+ path_pair_features_batch_raw: Tensor,
+ event_relation_strength_batch: Tensor | None = None,
+ path_relation_strength_batch: Tensor | None = None,
+ event_mask: Tensor,
+ path_mask: Tensor,
+ temporal_mask: Tensor,
+ examples: Sequence[QueryTrainingExample],
+ candidate_path_event_indices_batch: Sequence[Sequence[int]],
+ candidate_temporal_event_indices_batch: Sequence[Sequence[int]],
+) -> Tensor:
+ batch_size = len(examples)
+ device = calibrated_event_logits_batch.device
+ dtype = calibrated_event_logits_batch.dtype
+ event_competition_features = _batched_probability_competition_features(
+ calibrated_event_logits_batch,
+ mask=event_mask,
+ )
+ path_competition_features = _batched_probability_competition_features(
+ calibrated_path_logits_batch,
+ mask=path_mask,
+ )
+ temporal_competition_features = _batched_probability_competition_features(
+ temporal_logits_batch,
+ mask=temporal_mask,
+ )
+ event_logit_stats = _batched_logit_statistics(calibrated_event_logits_batch, mask=event_mask)
+ path_logit_stats = _batched_logit_statistics(calibrated_path_logits_batch, mask=path_mask)
+ temporal_logit_stats = _batched_logit_statistics(temporal_logits_batch, mask=temporal_mask)
+ if event_pair_features_batch_raw.size(1) > 0:
+ speaker_conflict_rate = _masked_feature_mean(event_pair_features_batch_raw, event_mask, 14)
+ time_conflict_rate = _masked_feature_mean(event_pair_features_batch_raw, event_mask, 6)
+ temporal_distractor_rate = _masked_feature_mean(event_pair_features_batch_raw, event_mask, 27)
+ best_exact_signature_cover = _masked_feature_max(event_pair_features_batch_raw, event_mask, 26)
+ best_source_turn_support = _masked_feature_max(event_pair_features_batch_raw, event_mask, 15)
+ else:
+ speaker_conflict_rate = torch.zeros((batch_size, 1), dtype=dtype, device=device)
+ time_conflict_rate = torch.zeros((batch_size, 1), dtype=dtype, device=device)
+ temporal_distractor_rate = torch.zeros((batch_size, 1), dtype=dtype, device=device)
+ best_exact_signature_cover = torch.zeros((batch_size, 1), dtype=dtype, device=device)
+ best_source_turn_support = torch.zeros((batch_size, 1), dtype=dtype, device=device)
+ if path_pair_features_batch_raw.size(1) > 0:
+ best_preferred_path = _masked_feature_max(path_pair_features_batch_raw, path_mask, 4)
+ best_support_anchor = _masked_feature_max(path_pair_features_batch_raw, path_mask, 17)
+ else:
+ best_preferred_path = torch.zeros((batch_size, 1), dtype=dtype, device=device)
+ best_support_anchor = torch.zeros((batch_size, 1), dtype=dtype, device=device)
+ has_event_candidates = event_mask.any(dim=1, keepdim=True).to(dtype=dtype)
+ has_path_candidates = path_mask.any(dim=1, keepdim=True).to(dtype=dtype)
+ has_temporal_candidates = temporal_mask.any(dim=1, keepdim=True).to(dtype=dtype)
+ is_temporal_question = torch.tensor(
+ [[1.0 if bool(dict(example.question_features or {}).get("is_temporal", False)) else 0.0] for example in examples],
+ dtype=dtype,
+ device=device,
+ )
+ alignment_features = torch.zeros((batch_size, 3), dtype=dtype, device=device)
+ structured_reverse_features = torch.zeros((batch_size, 8), dtype=dtype, device=device)
+ calibrated_path_prob_batch = torch.sigmoid(calibrated_path_logits_batch.float()).masked_fill(~path_mask, 0.0)
+ temporal_prob_batch = torch.sigmoid(temporal_logits_batch.float()).masked_fill(~temporal_mask, 0.0)
+ for batch_index in range(batch_size):
+ event_count = int(event_mask[batch_index].sum().item())
+ if event_count <= 0:
+ continue
+ event_probs = torch.sigmoid(calibrated_event_logits_batch[batch_index, :event_count].float())
+ best_event_index = int(torch.argmax(calibrated_event_logits_batch[batch_index, :event_count].float()).item())
+ event_reverse_scores, event_boundary_gaps, event_reverse_relations, event_reverse_available, _ = _reverse_competition_features(
+ event_probs,
+ event_relation_strength_batch[batch_index, :event_count, :event_count]
+ if isinstance(event_relation_strength_batch, Tensor)
+ else None,
+ valid_mask=event_mask[batch_index, :event_count],
+ )
+ structured_reverse_features[batch_index, 0] = event_reverse_scores[best_event_index]
+ structured_reverse_features[batch_index, 1] = event_reverse_relations[best_event_index]
+ structured_reverse_features[batch_index, 2] = event_boundary_gaps[best_event_index]
+ structured_reverse_features[batch_index, 3] = event_reverse_available[best_event_index]
+ path_count = int(path_mask[batch_index].sum().item())
+ if path_count > 0:
+ path_event_indices = list(candidate_path_event_indices_batch[batch_index])[:path_count]
+ max_path_prob_by_event = _scatter_amax_by_indices(
+ calibrated_path_prob_batch[batch_index, :path_count],
+ path_event_indices,
+ size=event_count,
+ device=device,
+ )
+ alignment_features[batch_index, 0] = max_path_prob_by_event[best_event_index]
+ best_path_index = int(torch.argmax(calibrated_path_logits_batch[batch_index, :path_count].float()).item())
+ if 0 <= best_path_index < len(path_event_indices):
+ top_path_event_index = int(path_event_indices[best_path_index])
+ if 0 <= top_path_event_index < event_count and top_path_event_index == best_event_index:
+ alignment_features[batch_index, 2] = 1.0
+ path_reverse_scores, path_boundary_gaps, path_reverse_relations, path_reverse_available, _ = _reverse_competition_features(
+ calibrated_path_prob_batch[batch_index, :path_count],
+ path_relation_strength_batch[batch_index, :path_count, :path_count]
+ if isinstance(path_relation_strength_batch, Tensor)
+ else None,
+ valid_mask=path_mask[batch_index, :path_count],
+ )
+ best_path_index = int(torch.argmax(calibrated_path_logits_batch[batch_index, :path_count].float()).item())
+ structured_reverse_features[batch_index, 4] = path_reverse_scores[best_path_index]
+ structured_reverse_features[batch_index, 5] = path_reverse_relations[best_path_index]
+ structured_reverse_features[batch_index, 6] = path_boundary_gaps[best_path_index]
+ structured_reverse_features[batch_index, 7] = path_reverse_available[best_path_index]
+ temporal_count = int(temporal_mask[batch_index].sum().item())
+ if temporal_count > 0:
+ temporal_event_indices = list(candidate_temporal_event_indices_batch[batch_index])[:temporal_count]
+ max_temporal_prob_by_event = _scatter_amax_by_indices(
+ temporal_prob_batch[batch_index, :temporal_count],
+ temporal_event_indices,
+ size=event_count,
+ device=device,
+ )
+ alignment_features[batch_index, 1] = max_temporal_prob_by_event[best_event_index]
+ return torch.cat(
+ [
+ event_competition_features,
+ path_competition_features,
+ temporal_competition_features,
+ event_logit_stats,
+ path_logit_stats,
+ temporal_logit_stats,
+ is_temporal_question,
+ has_event_candidates,
+ has_path_candidates,
+ has_temporal_candidates,
+ speaker_conflict_rate,
+ time_conflict_rate,
+ temporal_distractor_rate,
+ best_exact_signature_cover,
+ best_source_turn_support,
+ best_preferred_path,
+ best_support_anchor,
+ alignment_features,
+ structured_reverse_features,
+ ],
+ dim=-1,
+ )
+
+
+def _rank_ratio(scores: Tensor) -> Tensor:
+ score_count = int(scores.numel())
+ if score_count <= 0:
+ return scores.new_zeros((0,))
+ if score_count == 1:
+ return scores.new_ones((1,))
+ higher_counts = (scores.unsqueeze(0) > scores.unsqueeze(1)).sum(dim=1).to(dtype=scores.dtype)
+ return 1.0 - (higher_counts / float(score_count - 1))
+
+
+def _question_chain_hint_value(question: str, question_features: Mapping[str, Any]) -> float:
+ tokens = set(_normalized_token_list(question))
+ tokens.update(normalize_text(clean_text(item)) for item in list(question_features.get("question_anchor_tokens", []) or []))
+ return 1.0 if any(token in CHAIN_HINT_TOKENS for token in tokens if token) else 0.0
+
+
+def _depth_layer_scalar(depth_layer: str) -> float:
+ normalized_layer = clean_text(depth_layer)
+ if not normalized_layer:
+ return 0.0
+ index = DEPTH_LAYER_TO_ID.get(normalized_layer)
+ if index is None:
+ return 1.0
+ return float(index + 1) / float(max(1, len(DEPTH_LAYERS)))
+
+
+def _event_status_feature_values(event_features: Mapping[str, Any]) -> tuple[float, float, float]:
+ status = clean_text(event_features.get("target_status", ""))
+ return (
+ 1.0 if status == "current" else 0.0,
+ 1.0 if status == "past" else 0.0,
+ 1.0 if status == "planned" else 0.0,
+ )
+
+
+def _turn_feature_values(
+ prepared_events: Sequence[Mapping[str, Any]],
+ *,
+ scores: Tensor,
+ dtype: torch.dtype,
+ device: torch.device,
+) -> tuple[Tensor, Tensor, Tensor]:
+ event_count = len(prepared_events)
+ if event_count <= 0:
+ empty = torch.zeros((0,), dtype=dtype, device=device)
+ return empty, empty, empty
+ turn_values = torch.tensor(
+ [max(0, int(dict(event or {}).get("turn_index", 0) or 0)) for event in prepared_events],
+ dtype=dtype,
+ device=device,
+ )
+ max_turn = torch.clamp(turn_values.max(), min=1.0)
+ turn_norm = turn_values / max_turn
+ if scores.numel() <= 0:
+ top_turn = turn_values.new_zeros(())
+ else:
+ top_turn = turn_values[int(torch.argmax(scores).item())]
+ older_than_top = (turn_values < top_turn).to(dtype=dtype)
+ turn_distance_to_top = torch.abs(turn_values - top_turn) / max_turn
+ return turn_norm, older_than_top, turn_distance_to_top
+
+
+def _build_event_tunnel_features(
+ *,
+ prepared_events: Sequence[Mapping[str, Any]],
+ calibrated_event_probs: Tensor,
+ max_rerank_path_prob_by_event: Tensor,
+ max_calibrated_path_prob_by_event: Tensor,
+ max_temporal_prob_by_event: Tensor,
+ event_path_count: Tensor,
+ event_relation_strength: Tensor,
+ event_mask: Tensor,
+ question: str,
+ question_features: Mapping[str, Any],
+) -> Tensor:
+ event_count = len(prepared_events)
+ dtype = calibrated_event_probs.dtype
+ device = calibrated_event_probs.device
+ if event_count <= 0:
+ return calibrated_event_probs.new_zeros((0, EVENT_TUNNEL_FEATURE_DIM))
+ valid_mask = event_mask[:event_count].bool()
+ event_scores = calibrated_event_probs[:event_count].float()
+ rank_ratio = _rank_ratio(event_scores).to(dtype=dtype)
+ best_other_gap = (event_scores - _best_other_scores(event_scores)).to(dtype=dtype)
+ event_reverse_scores, event_boundary_gaps, event_reverse_relations, event_reverse_available, _ = _reverse_competition_features(
+ event_scores,
+ event_relation_strength[:event_count, :event_count],
+ valid_mask=valid_mask,
+ )
+ if event_count > 0:
+ top_index = int(torch.argmax(event_scores.masked_fill(~valid_mask, float("-inf"))).item()) if bool(valid_mask.any()) else 0
+ relation_to_top = event_relation_strength[:event_count, top_index].to(dtype=dtype)
+ else:
+ relation_to_top = calibrated_event_probs.new_zeros((0,))
+ turn_norm, older_than_top, turn_distance_to_top = _turn_feature_values(
+ prepared_events,
+ scores=event_scores,
+ dtype=dtype,
+ device=device,
+ )
+ status_rows = [_event_status_feature_values(event) for event in prepared_events]
+ status_features = torch.tensor(status_rows, dtype=dtype, device=device)
+ depth_scalar = torch.tensor(
+ [_depth_layer_scalar(clean_text(dict(event or {}).get("depth_layer", ""))) for event in prepared_events],
+ dtype=dtype,
+ device=device,
+ )
+ depth_present = (depth_scalar > 0.0).to(dtype=dtype)
+ deep_layer = (depth_scalar > (1.0 / float(max(1, len(DEPTH_LAYERS))))).to(dtype=dtype)
+ chain_hint = calibrated_event_probs.new_full(
+ (event_count,),
+ _question_chain_hint_value(question, question_features),
+ dtype=dtype,
+ )
+ features = torch.cat(
+ [
+ calibrated_event_probs[:event_count].to(dtype=dtype).unsqueeze(-1),
+ rank_ratio.unsqueeze(-1),
+ best_other_gap.unsqueeze(-1),
+ max_rerank_path_prob_by_event[:event_count].to(dtype=dtype).unsqueeze(-1),
+ max_calibrated_path_prob_by_event[:event_count].to(dtype=dtype).unsqueeze(-1),
+ max_temporal_prob_by_event[:event_count].to(dtype=dtype).unsqueeze(-1),
+ (event_path_count[:event_count].to(dtype=dtype) / max(1.0, float(len(PATH_TYPES)))).clamp(max=1.0).unsqueeze(-1),
+ event_reverse_scores.to(dtype=dtype).unsqueeze(-1),
+ event_boundary_gaps.to(dtype=dtype).unsqueeze(-1),
+ event_reverse_relations.to(dtype=dtype).unsqueeze(-1),
+ event_reverse_available.to(dtype=dtype).unsqueeze(-1),
+ relation_to_top.unsqueeze(-1),
+ turn_norm.unsqueeze(-1),
+ older_than_top.unsqueeze(-1),
+ turn_distance_to_top.unsqueeze(-1),
+ status_features[:, [0, 1]],
+ depth_present.unsqueeze(-1),
+ deep_layer.unsqueeze(-1),
+ chain_hint.unsqueeze(-1),
+ ],
+ dim=-1,
+ )
+ return features.masked_fill(~valid_mask.unsqueeze(-1), 0.0)
+
+
+def _build_path_tunnel_features(
+ *,
+ event_scores_for_paths: Tensor,
+ calibrated_path_scores: Tensor,
+ temporal_support_scores: Tensor,
+ path_pair_features: Tensor,
+ path_reverse_scores: Tensor,
+ path_boundary_gaps: Tensor,
+ path_reverse_relations: Tensor,
+ path_reverse_available: Tensor,
+ path_coverage_for_paths: Tensor,
+ event_rank_ratio_for_paths: Tensor,
+ question: str,
+ question_features: Mapping[str, Any],
+) -> Tensor:
+ path_count = int(calibrated_path_scores.numel())
+ if path_count <= 0:
+ return calibrated_path_scores.new_zeros((0, PATH_TUNNEL_FEATURE_DIM))
+ dtype = calibrated_path_scores.dtype
+ chain_hint = calibrated_path_scores.new_full(
+ (path_count,),
+ _question_chain_hint_value(question, question_features),
+ dtype=dtype,
+ )
+ pair_slice = path_pair_features[:path_count]
+ return torch.cat(
+ [
+ event_scores_for_paths.to(dtype=dtype).unsqueeze(-1),
+ calibrated_path_scores.to(dtype=dtype).unsqueeze(-1),
+ temporal_support_scores.to(dtype=dtype).unsqueeze(-1),
+ path_reverse_scores.to(dtype=dtype).unsqueeze(-1),
+ path_boundary_gaps.to(dtype=dtype).unsqueeze(-1),
+ path_reverse_relations.to(dtype=dtype).unsqueeze(-1),
+ path_reverse_available.to(dtype=dtype).unsqueeze(-1),
+ path_coverage_for_paths.to(dtype=dtype).unsqueeze(-1),
+ event_rank_ratio_for_paths.to(dtype=dtype).unsqueeze(-1),
+ pair_slice[:, [3, 4, 15, 17]],
+ chain_hint.unsqueeze(-1),
+ ],
+ dim=-1,
+ )
+
+
+def _normalized_id_frozenset(values: Sequence[str] | frozenset[str]) -> frozenset[str]:
+ if isinstance(values, frozenset):
+ return values
+ return frozenset(clean_text(item) for item in values if clean_text(item))
+
+
+def _event_distractor_supervision_loss(
+ logits: Tensor,
+ *,
+ candidate_ids: Sequence[str],
+ positive_ids: Sequence[str],
+ hard_negative_ids: Sequence[str],
+) -> Tensor:
+ if logits.numel() == 0:
+ return logits.sum() * 0.0
+ positive_id_set = _normalized_id_frozenset(positive_ids)
+ hard_negative_id_set = _normalized_id_frozenset(hard_negative_ids)
+ if not positive_id_set and not hard_negative_id_set:
+ return logits.sum() * 0.0
+ supervised_indices: List[int] = []
+ supervised_targets: List[float] = []
+ for index, candidate_id in enumerate(candidate_ids):
+ normalized_id = clean_text(candidate_id)
+ if not normalized_id:
+ continue
+ if normalized_id in positive_id_set:
+ supervised_indices.append(index)
+ supervised_targets.append(0.0)
+ elif normalized_id in hard_negative_id_set:
+ supervised_indices.append(index)
+ supervised_targets.append(1.0)
+ if not supervised_indices:
+ return logits.sum() * 0.0
+ working_logits = logits.float()
+ device = working_logits.device
+ index_tensor = torch.tensor(supervised_indices, dtype=torch.long, device=device)
+ target_tensor = torch.tensor(supervised_targets, dtype=working_logits.dtype, device=device)
+ selected_logits = working_logits.index_select(0, index_tensor)
+ return F.binary_cross_entropy_with_logits(selected_logits, target_tensor)
+
+
+def _hard_negative_margin_loss(
+ logits: Tensor,
+ *,
+ candidate_ids: Sequence[str],
+ positive_ids: Sequence[str],
+ hard_negative_ids: Sequence[str],
+ margin: float,
+) -> Tensor:
+ if logits.numel() == 0:
+ return logits.sum() * 0.0
+ positive_id_set = _normalized_id_frozenset(positive_ids)
+ hard_negative_id_set = _normalized_id_frozenset(hard_negative_ids)
+ if not positive_id_set or not hard_negative_id_set:
+ return logits.sum() * 0.0
+ positive_indices = [index for index, candidate_id in enumerate(candidate_ids) if clean_text(candidate_id) in positive_id_set]
+ hard_negative_indices = [index for index, candidate_id in enumerate(candidate_ids) if clean_text(candidate_id) in hard_negative_id_set]
+ if not positive_indices or not hard_negative_indices:
+ return logits.sum() * 0.0
+ working_logits = logits.float()
+ device = working_logits.device
+ positive_logits = working_logits.index_select(0, torch.tensor(positive_indices, dtype=torch.long, device=device))
+ negative_logits = working_logits.index_select(0, torch.tensor(hard_negative_indices, dtype=torch.long, device=device))
+ margins = float(margin) - (positive_logits.unsqueeze(-1) - negative_logits.unsqueeze(0))
+ return F.relu(margins).mean()
+
+
+def _online_hard_negative_ids(
+ logits: Tensor,
+ *,
+ candidate_ids: Sequence[str],
+ positive_ids: Sequence[str],
+ limit: int,
+ protected_ids: Sequence[str] = (),
+) -> List[str]:
+ resolved_limit = max(0, int(limit or 0))
+ if resolved_limit <= 0 or logits.numel() == 0:
+ return []
+ positive_id_set = _normalized_id_frozenset(positive_ids)
+ protected_id_set = _normalized_id_frozenset(protected_ids)
+ scored_candidates: List[tuple[str, float]] = []
+ detached_scores = logits.detach().float().cpu().tolist()
+ for candidate_id, score in zip(candidate_ids, detached_scores):
+ normalized_id = clean_text(candidate_id)
+ if not normalized_id or normalized_id in positive_id_set or normalized_id in protected_id_set:
+ continue
+ scored_candidates.append((normalized_id, float(score)))
+ scored_candidates.sort(key=lambda item: (-item[1], item[0]))
+ return dedupe_texts([candidate_id for candidate_id, _ in scored_candidates], max_items=resolved_limit)
+
+
+def _online_hard_negative_path_ids(
+ logits: Tensor,
+ *,
+ candidate_path_ids: Sequence[str],
+ positive_path_ids: Sequence[str],
+ candidate_path_event_ids: Sequence[str] | None = None,
+ positive_event_ids: Sequence[str] = (),
+ limit: int,
+) -> List[str]:
+ resolved_limit = max(0, int(limit or 0))
+ if resolved_limit <= 0 or logits.numel() == 0:
+ return []
+ positive_path_id_set = _normalized_id_frozenset(positive_path_ids)
+ positive_event_id_set = _normalized_id_frozenset(positive_event_ids)
+ event_ids = list(candidate_path_event_ids or [])
+ scored_candidates: List[tuple[str, float]] = []
+ detached_scores = logits.detach().float().cpu().tolist()
+ for index, (path_id, score) in enumerate(zip(candidate_path_ids, detached_scores)):
+ normalized_path_id = clean_text(path_id)
+ if not normalized_path_id or normalized_path_id in positive_path_id_set:
+ continue
+ event_id = clean_text(event_ids[index]) if index < len(event_ids) else parse_path_id(normalized_path_id)[0]
+ if event_id and event_id in positive_event_id_set:
+ continue
+ scored_candidates.append((normalized_path_id, float(score)))
+ scored_candidates.sort(key=lambda item: (-item[1], item[0]))
+ return dedupe_texts([path_id for path_id, _ in scored_candidates], max_items=resolved_limit)
+
+
+def _multi_positive_supervision_active(supervision: Mapping[str, Any]) -> bool:
+ bucket = clean_text(supervision.get("supervision_bucket", ""))
+ answer_type = clean_text(supervision.get("answer_type", ""))
+ positive_event_count = int(supervision.get("positive_event_count", 0) or 0)
+ if bucket.endswith("_multi_positive") or bucket in {"list_event_text_multi_positive", "multi_evidence"}:
+ return True
+ return positive_event_count > 1 and answer_type in {"event_text", "multi_evidence"}
+
+
+def _selection_positive_coverage_count(
+ supervision: Mapping[str, Any],
+ *,
+ positive_count: int,
+ top_k: int,
+ base_count: int,
+ multi_positive_count: int,
+ multi_positive_fraction: float,
+) -> int:
+ resolved_positive_count = max(0, int(positive_count or 0))
+ if resolved_positive_count <= 0:
+ return 1
+ resolved_top_k = max(1, int(top_k or 1))
+ resolved_base_count = max(1, int(base_count or 1))
+ if not _multi_positive_supervision_active(supervision):
+ return max(1, min(resolved_positive_count, resolved_top_k, resolved_base_count))
+ resolved_multi_count = max(resolved_base_count, int(multi_positive_count or resolved_base_count))
+ fraction_target = int(math.ceil(float(resolved_positive_count) * max(0.0, float(multi_positive_fraction or 0.0))))
+ target_count = max(resolved_base_count, fraction_target)
+ return max(1, min(resolved_positive_count, resolved_top_k, resolved_multi_count, target_count))
+
+
+def _positive_coverage_hit_count(
+ ranked_ids: Sequence[str],
+ positive_ids: Sequence[str],
+ *,
+ limit: int,
+) -> int:
+ positive_id_set = _normalized_id_frozenset(positive_ids)
+ if not positive_id_set:
+ return 0
+ selected_ids = _normalized_id_frozenset(ranked_ids[: max(0, int(limit or 0))])
+ return len(positive_id_set & selected_ids)
+
+
+def _selection_margin_loss(
+ logits: Tensor,
+ *,
+ candidate_ids: Sequence[str],
+ positive_ids: Sequence[str],
+ margin: float,
+ top_k: int = 1,
+ required_positive_count: int = 1,
+) -> Tensor:
+ if logits.numel() == 0:
+ return logits.sum() * 0.0
+ positive_id_set = _normalized_id_frozenset(positive_ids)
+ if not positive_id_set:
+ return logits.sum() * 0.0
+ positive_indices = [index for index, candidate_id in enumerate(candidate_ids) if clean_text(candidate_id) in positive_id_set]
+ negative_indices = [index for index, candidate_id in enumerate(candidate_ids) if clean_text(candidate_id) not in positive_id_set]
+ if not positive_indices or not negative_indices:
+ return logits.sum() * 0.0
+ working_logits = logits.float()
+ device = working_logits.device
+ positive_logits = working_logits.index_select(0, torch.tensor(positive_indices, dtype=torch.long, device=device))
+ negative_logits = working_logits.index_select(0, torch.tensor(negative_indices, dtype=torch.long, device=device))
+ resolved_top_k = max(1, int(top_k or 1))
+ resolved_required_positive_count = max(
+ 1,
+ min(
+ int(required_positive_count or 1),
+ int(positive_logits.numel()),
+ resolved_top_k,
+ ),
+ )
+ negative_rank = max(1, resolved_top_k - resolved_required_positive_count + 1)
+ if negative_rank > int(negative_logits.numel()):
+ # If the runtime selection window is wider than all negatives, any
+ # required positive candidates are guaranteed to survive this stage.
+ return logits.sum() * 0.0
+ positive_threshold = torch.topk(
+ positive_logits,
+ k=resolved_required_positive_count,
+ dim=0,
+ ).values[-1]
+ negative_threshold = torch.topk(negative_logits, k=negative_rank, dim=0).values[-1]
+ return F.relu(torch.tensor(float(margin), dtype=working_logits.dtype, device=device) - (positive_threshold - negative_threshold))
+
+
+def _positive_vs_negative_ids_margin_loss(
+ logits: Tensor,
+ *,
+ candidate_ids: Sequence[str],
+ positive_ids: Sequence[str] | frozenset[str],
+ negative_ids: Sequence[str] | frozenset[str],
+ margin: float,
+) -> Tensor:
+ if logits.numel() == 0:
+ return logits.sum() * 0.0
+ positive_id_set = _normalized_id_frozenset(positive_ids)
+ negative_id_set = _normalized_id_frozenset(negative_ids)
+ if not positive_id_set or not negative_id_set:
+ return logits.sum() * 0.0
+ positive_indices = [
+ index for index, candidate_id in enumerate(candidate_ids)
+ if clean_text(candidate_id) in positive_id_set
+ ]
+ negative_indices = [
+ index for index, candidate_id in enumerate(candidate_ids)
+ if clean_text(candidate_id) in negative_id_set
+ ]
+ if not positive_indices or not negative_indices:
+ return logits.sum() * 0.0
+ working_logits = logits.float()
+ device = working_logits.device
+ positive_logits = working_logits.index_select(0, torch.tensor(positive_indices, dtype=torch.long, device=device))
+ negative_logits = working_logits.index_select(0, torch.tensor(negative_indices, dtype=torch.long, device=device))
+ positive_threshold = torch.max(positive_logits)
+ negative_threshold = torch.max(negative_logits)
+ return F.relu(torch.tensor(float(margin), dtype=working_logits.dtype, device=device) - (positive_threshold - negative_threshold))
+
+
+def _event_decision_scores_for_final_set(
+ event_logits: Tensor,
+ *,
+ candidate_event_ids: Sequence[str],
+ path_logits: Tensor,
+ candidate_path_event_ids: Sequence[str],
+) -> Tensor:
+ if event_logits.numel() == 0:
+ return event_logits.sum().reshape(1)[:0]
+ event_scores = torch.sigmoid(event_logits.float())
+ if path_logits.numel() == 0 or not candidate_path_event_ids:
+ return event_scores.to(dtype=event_logits.dtype)
+ path_scores = torch.sigmoid(path_logits.float())
+ event_to_path_scores: Dict[str, List[Tensor]] = {}
+ for path_index, event_id in enumerate(candidate_path_event_ids):
+ if path_index >= int(path_scores.numel()):
+ break
+ normalized_event_id = clean_text(event_id)
+ if not normalized_event_id:
+ continue
+ event_to_path_scores.setdefault(normalized_event_id, []).append(path_scores[path_index])
+ decision_scores: List[Tensor] = []
+ for event_index, event_id in enumerate(candidate_event_ids):
+ components = [event_scores[event_index]]
+ components.extend(event_to_path_scores.get(clean_text(event_id), []))
+ decision_scores.append(torch.stack(components).max())
+ if not decision_scores:
+ return event_scores.new_zeros((0,), dtype=event_logits.dtype)
+ return torch.stack(decision_scores).to(dtype=event_logits.dtype)
+
+
+def _runtime_selected_event_set_loss(
+ event_logits: Tensor,
+ *,
+ candidate_event_ids: Sequence[str],
+ path_logits: Tensor,
+ candidate_path_event_ids: Sequence[str],
+ positive_event_ids: Sequence[str],
+ margin: float,
+ top_k: int,
+ support_path_k: int,
+ required_positive_count: int = 1,
+) -> Tensor:
+ """Match the runtime selected-event construction more closely.
+
+ Runtime prepends events from the highest scoring support paths before it
+ fills the remaining prompt window with event scores. A loss that only sees
+ event top-k can therefore pass while high-scoring negative paths consume
+ the exact slots that would have preserved a gold event.
+ """
+
+ if event_logits.numel() == 0:
+ return event_logits.sum() * 0.0
+ positive_id_set = _normalized_id_frozenset(positive_event_ids)
+ if not positive_id_set:
+ return event_logits.sum() * 0.0
+ resolved_top_k = max(1, int(top_k or 1))
+ resolved_support_path_k = max(0, int(support_path_k or 0))
+ resolved_required_positive_count = max(1, int(required_positive_count or 1))
+ event_window_top_k = max(1, resolved_top_k - resolved_support_path_k)
+ event_window_loss = _selection_margin_loss(
+ event_logits,
+ candidate_ids=candidate_event_ids,
+ positive_ids=positive_event_ids,
+ margin=margin,
+ top_k=event_window_top_k,
+ required_positive_count=resolved_required_positive_count,
+ )
+ if path_logits.numel() == 0 or not candidate_path_event_ids or resolved_support_path_k <= 0:
+ return event_window_loss
+ working_path_logits = path_logits.float()
+ device = working_path_logits.device
+ positive_path_scores_by_event: Dict[str, Tensor] = {}
+ negative_path_scores_by_event: Dict[str, Tensor] = {}
+ for path_index, event_id in enumerate(candidate_path_event_ids):
+ if path_index >= int(working_path_logits.numel()):
+ break
+ normalized_event_id = clean_text(event_id)
+ if not normalized_event_id:
+ continue
+ score = working_path_logits[path_index]
+ if normalized_event_id in positive_id_set:
+ previous = positive_path_scores_by_event.get(normalized_event_id)
+ if previous is None:
+ positive_path_scores_by_event[normalized_event_id] = score
+ else:
+ positive_path_scores_by_event[normalized_event_id] = torch.maximum(previous, score)
+ else:
+ previous = negative_path_scores_by_event.get(normalized_event_id)
+ if previous is None:
+ negative_path_scores_by_event[normalized_event_id] = score
+ else:
+ negative_path_scores_by_event[normalized_event_id] = torch.maximum(previous, score)
+ if not positive_path_scores_by_event or not negative_path_scores_by_event:
+ return event_window_loss
+ positive_path_scores = torch.stack(list(positive_path_scores_by_event.values()))
+ negative_path_scores = torch.stack(list(negative_path_scores_by_event.values()))
+ required_path_positive_count = max(
+ 1,
+ min(
+ resolved_required_positive_count,
+ int(positive_path_scores.numel()),
+ resolved_support_path_k,
+ ),
+ )
+ negative_rank = max(1, resolved_support_path_k - required_path_positive_count + 1)
+ if negative_rank > int(negative_path_scores.numel()):
+ return event_window_loss
+ positive_path_threshold = torch.topk(
+ positive_path_scores,
+ k=required_path_positive_count,
+ dim=0,
+ ).values[-1]
+ negative_prefix_count = min(negative_rank, int(negative_path_scores.numel()))
+ negative_threshold = torch.topk(negative_path_scores, k=negative_prefix_count, dim=0).values[-1]
+ path_prefix_loss = F.relu(
+ torch.tensor(float(margin), dtype=working_path_logits.dtype, device=device)
+ - (positive_path_threshold - negative_threshold)
+ )
+ return 0.5 * (event_window_loss + path_prefix_loss.to(dtype=event_window_loss.dtype))
+
+
+def _matrix_delta_direction_loss(
+ delta_logits: Tensor,
+ *,
+ matrix_ids: Sequence[str],
+ positive_ids: Sequence[str],
+ hard_negative_ids: Sequence[str],
+ margin: float,
+ positive_weight: float = 1.0,
+ hard_negative_weight: float = 1.0,
+ pair_weight: float = 1.0,
+) -> Tensor:
+ if delta_logits.numel() == 0:
+ return delta_logits.sum() * 0.0
+ positive_id_set = _normalized_id_frozenset(positive_ids)
+ hard_negative_id_set = _normalized_id_frozenset(hard_negative_ids)
+ if not positive_id_set and not hard_negative_id_set:
+ return delta_logits.sum() * 0.0
+ working_delta = delta_logits.float()
+ device = working_delta.device
+ weighted_losses: List[Tensor] = []
+ weights: List[float] = []
+ positive_indices = [index for index, item_id in enumerate(matrix_ids) if clean_text(item_id) in positive_id_set]
+ hard_negative_indices = [index for index, item_id in enumerate(matrix_ids) if clean_text(item_id) in hard_negative_id_set]
+ if positive_indices:
+ positive_delta = working_delta.index_select(0, torch.tensor(positive_indices, dtype=torch.long, device=device))
+ weighted_losses.append(
+ float(positive_weight) * F.relu(torch.tensor(float(margin), dtype=working_delta.dtype, device=device) - positive_delta).mean()
+ )
+ weights.append(max(0.0, float(positive_weight)))
+ if hard_negative_indices:
+ hard_negative_delta = working_delta.index_select(0, torch.tensor(hard_negative_indices, dtype=torch.long, device=device))
+ weighted_losses.append(
+ float(hard_negative_weight)
+ * F.relu(torch.tensor(float(margin), dtype=working_delta.dtype, device=device) + hard_negative_delta).mean()
+ )
+ weights.append(max(0.0, float(hard_negative_weight)))
+ if positive_indices and hard_negative_indices:
+ positive_delta = working_delta.index_select(0, torch.tensor(positive_indices, dtype=torch.long, device=device))
+ hard_negative_delta = working_delta.index_select(0, torch.tensor(hard_negative_indices, dtype=torch.long, device=device))
+ pair_margin = torch.tensor(float(margin), dtype=working_delta.dtype, device=device)
+ weighted_losses.append(float(pair_weight) * F.relu(pair_margin - (positive_delta.unsqueeze(-1) - hard_negative_delta.unsqueeze(0))).mean())
+ weights.append(max(0.0, float(pair_weight)))
+ if not weighted_losses:
+ return delta_logits.sum() * 0.0
+ weight_sum = sum(weight for weight in weights if weight > 0.0)
+ if weight_sum <= 0.0:
+ return delta_logits.sum() * 0.0
+ return torch.stack(weighted_losses).sum() / float(weight_sum)
+
+
+def _answer_refusal_margin_loss(
+ logits: Tensor,
+ *,
+ target_index: int,
+ has_positive_supervision: bool,
+ margin: float,
+) -> Tensor:
+ if logits.numel() == 0 or not has_positive_supervision:
+ return logits.sum() * 0.0
+ abstain_index = ANSWER_TYPE_TO_ID["abstain"]
+ if target_index == abstain_index:
+ return logits.sum() * 0.0
+ working_logits = logits.float()
+ gold_logit = working_logits[target_index]
+ abstain_logit = working_logits[abstain_index]
+ return F.relu(torch.tensor(float(margin), dtype=working_logits.dtype, device=working_logits.device) - (gold_logit - abstain_logit))
+
+
+def _hard_negative_path_ids(example: QueryTrainingExample, candidate_path_ids: Sequence[str]) -> List[str]:
+ question_features = dict(example.question_features or {})
+ hard_negative_event_ids = {clean_text(item) for item in _resolved_hard_negative_event_ids_for_example(example) if clean_text(item)}
+ positive_event_ids = {clean_text(item) for item in example.positive_event_ids if clean_text(item)}
+ preferred_path_types = _preferred_support_path_types(question_features)
+ primary_preferred_path_type = clean_text(preferred_path_types[0]) if preferred_path_types else ""
+ semantic_target = clean_text(question_features.get("semantic_slot_target", ""))
+ typed_query = bool(question_features.get("is_temporal", False)) or semantic_target in {
+ "identity",
+ "research_topic",
+ "education",
+ "occupation",
+ "profile",
+ "event_time",
+ } or bool(clean_text(question_features.get("target_status_target", "")))
+ hard_negative_path_ids: List[str] = []
+ for path_id in candidate_path_ids:
+ event_id, path_type, _ = parse_path_id(path_id)
+ if not event_id or not path_type:
+ continue
+ if event_id in hard_negative_event_ids:
+ hard_negative_path_ids.append(path_id)
+ continue
+ if typed_query and primary_preferred_path_type and event_id in positive_event_ids and path_type != primary_preferred_path_type:
+ hard_negative_path_ids.append(path_id)
+ return dedupe_texts(hard_negative_path_ids)
+
+
+def _label_tensor_from_positive_ids(
+ candidate_ids: Sequence[str],
+ positive_ids: frozenset[str],
+ *,
+ device: torch.device,
+) -> Tensor:
+ return torch.tensor(
+ [1.0 if candidate_id in positive_ids else 0.0 for candidate_id in candidate_ids],
+ dtype=torch.float32,
+ device=device,
+ )
+
+
+def _binary_plan_loss_for_ids(
+ logits: Tensor,
+ *,
+ candidate_ids: Sequence[str],
+ positive_ids: Sequence[str] | frozenset[str],
+ device: torch.device,
+) -> Tensor:
+ if logits.numel() == 0:
+ return logits.sum() * 0.0
+ target_ids = _normalized_id_frozenset(positive_ids)
+ if not target_ids:
+ return logits.sum() * 0.0
+ targets = _label_tensor_from_positive_ids(candidate_ids, target_ids, device=device)
+ return F.binary_cross_entropy_with_logits(logits.float(), targets)
+
+
+def _build_example_supervision_payload(example: QueryTrainingExample) -> Dict[str, Any]:
+ question_features = dict(example.question_features or {})
+ metadata = dict(example.metadata or {})
+ answer_plan_targets = dict(example.answer_plan_targets or {})
+ selected_memory_ids = frozenset(
+ clean_text(item)
+ for item in list(answer_plan_targets.get("selected_memory_ids", []) or [])
+ if clean_text(item)
+ )
+ protected_memory_ids = frozenset(
+ clean_text(item)
+ for item in list(answer_plan_targets.get("protected_memory_ids", []) or [])
+ if clean_text(item)
+ )
+ current_memory_ids = frozenset(
+ clean_text(item)
+ for item in list(answer_plan_targets.get("current_memory_ids", []) or [])
+ if clean_text(item)
+ )
+ historical_memory_ids = frozenset(
+ clean_text(item)
+ for item in list(answer_plan_targets.get("historical_memory_ids", []) or [])
+ if clean_text(item)
+ )
+ suppressed_memory_ids = frozenset(
+ clean_text(item)
+ for item in list(answer_plan_targets.get("suppressed_memory_ids", []) or [])
+ if clean_text(item)
+ )
+ preferred_path_types = _preferred_support_path_types(question_features)
+ semantic_target = clean_text(question_features.get("semantic_slot_target", ""))
+ typed_query = bool(question_features.get("is_temporal", False)) or semantic_target in {
+ "identity",
+ "research_topic",
+ "education",
+ "occupation",
+ "profile",
+ "event_time",
+ } or bool(clean_text(question_features.get("target_status_target", "")))
+ return {
+ "answer_type_index": int(_answer_type_label(example)),
+ "answer_type": _training_answer_type_name(example),
+ "supervision_bucket": _training_supervision_bucket_name(example),
+ "positive_event_count": _positive_event_id_count(example),
+ "has_positive_supervision": bool(example.positive_event_ids or example.positive_path_ids),
+ "is_temporal_example": bool(_is_temporal_example(example)),
+ "positive_event_ids": frozenset(
+ clean_text(item) for item in example.positive_event_ids if clean_text(item)
+ ),
+ "positive_path_ids": frozenset(
+ clean_text(item) for item in example.positive_path_ids if clean_text(item)
+ ),
+ "positive_time_node_ids": frozenset(
+ clean_text(item) for item in example.positive_time_node_ids if clean_text(item)
+ ),
+ "has_answer_plan_supervision": bool(
+ selected_memory_ids
+ or protected_memory_ids
+ or current_memory_ids
+ or historical_memory_ids
+ or suppressed_memory_ids
+ ),
+ "answer_plan_selected_event_ids": frozenset(selected_memory_ids or protected_memory_ids),
+ "answer_plan_current_event_ids": current_memory_ids,
+ "answer_plan_historical_event_ids": historical_memory_ids,
+ "answer_plan_suppressed_event_ids": suppressed_memory_ids,
+ "negative_event_ids": frozenset(
+ clean_text(item) for item in example.negative_event_ids if clean_text(item)
+ ),
+ "hard_negative_event_ids": frozenset(_resolved_hard_negative_event_ids_for_example(example)),
+ "ancestor_event_ids": frozenset(
+ clean_text(item) for item in list(metadata.get("ancestor_event_ids", []) or []) if clean_text(item)
+ ),
+ "answer_primary_event_ids": frozenset(
+ clean_text(item) for item in list(metadata.get("answer_primary_event_ids", []) or []) if clean_text(item)
+ ),
+ "side_branch_event_ids": frozenset(
+ clean_text(item) for item in list(metadata.get("side_branch_event_ids", []) or []) if clean_text(item)
+ ),
+ "distractor_event_ids": frozenset(
+ clean_text(item) for item in list(metadata.get("distractor_event_ids", []) or []) if clean_text(item)
+ ),
+ "typed_query": bool(typed_query),
+ "primary_preferred_path_type": clean_text(preferred_path_types[0]) if preferred_path_types else "",
+ "training_example_weight": float(metadata.get("_training_example_weight", 1.0) or 1.0),
+ "training_source_dataset": clean_text(metadata.get("_training_source_dataset", "")),
+ "training_sampling_multiplier": float(metadata.get("_training_sampling_multiplier", 1.0) or 1.0),
+ }
+
+
+def _path_hard_negative_ids_from_supervision(
+ supervision: Mapping[str, Any],
+ *,
+ candidate_path_ids: Sequence[str],
+ candidate_path_event_ids: Sequence[str] | None = None,
+ candidate_path_types: Sequence[str] | None = None,
+) -> List[str]:
+ hard_negative_event_ids = _resolved_hard_negative_event_ids_from_supervision(supervision)
+ positive_event_ids = {
+ clean_text(item) for item in list(supervision.get("positive_event_ids", []) or []) if clean_text(item)
+ }
+ typed_query = bool(supervision.get("typed_query", False))
+ primary_preferred_path_type = clean_text(supervision.get("primary_preferred_path_type", ""))
+ hard_negative_path_ids: List[str] = []
+ for index, path_id in enumerate(candidate_path_ids):
+ event_id = (
+ clean_text(candidate_path_event_ids[index])
+ if candidate_path_event_ids is not None and index < len(candidate_path_event_ids)
+ else ""
+ )
+ path_type = (
+ clean_text(candidate_path_types[index])
+ if candidate_path_types is not None and index < len(candidate_path_types)
+ else ""
+ )
+ if not event_id or not path_type:
+ parsed_event_id, parsed_path_type, _ = parse_path_id(path_id)
+ event_id = event_id or parsed_event_id
+ path_type = path_type or parsed_path_type
+ if not event_id or not path_type:
+ continue
+ if event_id in hard_negative_event_ids:
+ hard_negative_path_ids.append(path_id)
+ continue
+ if typed_query and primary_preferred_path_type and event_id in positive_event_ids and path_type != primary_preferred_path_type:
+ hard_negative_path_ids.append(path_id)
+ return dedupe_texts(hard_negative_path_ids)
+
+
+def _tunnel_positive_event_ids_from_supervision(supervision: Mapping[str, Any]) -> frozenset[str]:
+ bucket = clean_text(supervision.get("supervision_bucket", ""))
+ positive_ids = {
+ clean_text(item)
+ for item in list(supervision.get("positive_event_ids", []) or [])
+ if clean_text(item)
+ }
+ if bucket in {"chain_retrieval", "current_chain_head_selection"} or bucket.startswith("depth_chain"):
+ positive_ids.update(
+ clean_text(item)
+ for item in list(supervision.get("ancestor_event_ids", []) or [])
+ if clean_text(item)
+ )
+ positive_ids.update(
+ clean_text(item)
+ for item in list(supervision.get("answer_primary_event_ids", []) or [])
+ if clean_text(item)
+ )
+ return frozenset(positive_ids)
+
+
+def _tunnel_hard_negative_event_ids_from_supervision(supervision: Mapping[str, Any]) -> frozenset[str]:
+ positive_ids = _tunnel_positive_event_ids_from_supervision(supervision)
+ hard_negative_ids = {
+ clean_text(item)
+ for item in list(supervision.get("hard_negative_event_ids", []) or [])
+ if clean_text(item)
+ }
+ hard_negative_ids.update(
+ clean_text(item)
+ for item in list(supervision.get("distractor_event_ids", []) or [])
+ if clean_text(item)
+ )
+ hard_negative_ids.update(
+ clean_text(item)
+ for item in list(supervision.get("side_branch_event_ids", []) or [])
+ if clean_text(item)
+ )
+ return frozenset(item for item in hard_negative_ids if item and item not in positive_ids)
+
+
+def _temporal_path_labels_from_supervision(
+ supervision: Mapping[str, Any],
+ candidate_temporal_path_ids: Sequence[str],
+ candidate_temporal_node_ids: Sequence[str],
+ *,
+ candidate_temporal_event_ids: Sequence[str] | None = None,
+ device: torch.device,
+) -> Tensor:
+ positive_event_ids = {
+ clean_text(item) for item in list(supervision.get("positive_event_ids", []) or []) if clean_text(item)
+ }
+ positive_time_node_ids = {
+ clean_text(item) for item in list(supervision.get("positive_time_node_ids", []) or []) if clean_text(item)
+ }
+ temporal_path_targets: List[float] = []
+ for index, (path_id, time_node_id) in enumerate(zip(candidate_temporal_path_ids, candidate_temporal_node_ids)):
+ event_id = (
+ clean_text(candidate_temporal_event_ids[index])
+ if candidate_temporal_event_ids is not None and index < len(candidate_temporal_event_ids)
+ else ""
+ )
+ if not event_id:
+ event_id, _, _ = parse_path_id(path_id)
+ is_positive = bool(event_id in positive_event_ids and clean_text(time_node_id) in positive_time_node_ids)
+ temporal_path_targets.append(1.0 if is_positive else 0.0)
+ return torch.tensor(temporal_path_targets, dtype=torch.float32, device=device)
+
+
+_BATCH_PREPARE_WORKER_CONTEXT: Dict[str, Any] = {}
+_BATCH_PREPARE_WORKER_CACHE: OrderedDict[str, GraphCacheItem] = OrderedDict()
+_BATCH_PREPARE_WORKER_MEMORY_CACHE_SIZE = 0
+_BATCH_PREPARE_THREAD_STATE = threading.local()
+
+
+class _PreparedBatchWorkerPool:
+ def __init__(
+ self,
+ *,
+ worker_context: Mapping[str, Any] | None,
+ worker_count: int,
+ ) -> None:
+ self.worker_context = _normalize_batch_prepare_worker_context(worker_context)
+ self.worker_count = max(0, int(worker_count or 0))
+ self._executor: ProcessPoolExecutor | None = None
+ self._thread_executor: ThreadPoolExecutor | None = None
+ self._direct_fallback_mode = False
+ self._thread_fallback_mode = False
+ self._worker_exc: Exception | None = None
+ self._process_recovery_attempts = 0
+ self._max_process_recovery_attempts = max(0, int(DEFAULT_BATCH_PREPARE_PROCESS_RECOVERY_ATTEMPTS or 0))
+ self._current_process_worker_count = self.worker_count
+ self._process_success_count = 0
+ self._max_tasks_per_child = max(0, int(DEFAULT_BATCH_PREPARE_WORKER_MAX_TASKS_PER_CHILD or 0))
+ self._thread_worker_count = max(
+ int(DEFAULT_BATCH_PREPARE_THREAD_FALLBACK_MIN_WORKERS),
+ min(16, max(1, int(self.worker_count or 0) * int(DEFAULT_BATCH_PREPARE_THREAD_FALLBACK_WORKERS_MULTIPLIER))),
+ )
+
+ def __del__(self) -> None:
+ self.close()
+
+ def close(self) -> None:
+ self._close_process_executor()
+ self._close_thread_executor()
+
+ def _close_process_executor(self) -> None:
+ executor = self._executor
+ self._executor = None
+ if executor is not None:
+ executor.shutdown(wait=True, cancel_futures=True)
+
+ def _close_thread_executor(self) -> None:
+ executor = self._thread_executor
+ self._thread_executor = None
+ if executor is not None:
+ executor.shutdown(wait=True, cancel_futures=True)
+
+ def _enable_direct_fallback(self, exc: Exception) -> None:
+ self._worker_exc = exc
+ self._direct_fallback_mode = True
+ self.close()
+
+ def _enable_thread_fallback(self, exc: Exception) -> None:
+ self._worker_exc = exc
+ self._thread_fallback_mode = True
+ self._close_process_executor()
+
+ def _record_worker_success(self, *, backend: str) -> None:
+ self._worker_exc = None
+ if backend == "process":
+ self._process_recovery_attempts = 0
+ self._process_success_count += 1
+
+ def _process_worker_count(self) -> int:
+ return max(1, int(self._current_process_worker_count or self.worker_count or 1))
+
+ def _recover_process_pool(self, exc: Exception) -> Dict[str, Any]:
+ previous_worker_count = self._process_worker_count()
+ self._worker_exc = exc
+ self._close_process_executor()
+ self._process_success_count = 0
+ self._process_recovery_attempts += 1
+ shrunk_worker_count = previous_worker_count
+ if previous_worker_count > 1:
+ shrunk_worker_count = max(1, math.ceil(previous_worker_count / 2))
+ self._current_process_worker_count = shrunk_worker_count
+ if self._process_recovery_attempts > self._max_process_recovery_attempts:
+ self._enable_thread_fallback(exc)
+ return {
+ "recovery_mode": "thread_fallback",
+ "recovery_attempt": int(self._process_recovery_attempts),
+ "previous_worker_count": int(previous_worker_count),
+ "next_worker_count": int(self._thread_worker_count),
+ }
+ return {
+ "recovery_mode": "process_retry",
+ "recovery_attempt": int(self._process_recovery_attempts),
+ "previous_worker_count": int(previous_worker_count),
+ "next_worker_count": int(self._process_worker_count()),
+ }
+
+ def _ensure_executor(self) -> ProcessPoolExecutor | None:
+ if self.worker_count <= 0 or self._direct_fallback_mode or self._thread_fallback_mode:
+ return None
+ if self._executor is not None:
+ return self._executor
+ try:
+ executor_kwargs: Dict[str, Any] = {
+ "max_workers": self._process_worker_count(),
+ "mp_context": multiprocessing.get_context("spawn"),
+ "initializer": _initialize_batch_prepare_worker,
+ "initargs": (self.worker_context,),
+ }
+ executor_signature = inspect.signature(ProcessPoolExecutor)
+ if "max_tasks_per_child" in executor_signature.parameters and self._max_tasks_per_child > 0:
+ executor_kwargs["max_tasks_per_child"] = int(self._max_tasks_per_child)
+ self._executor = ProcessPoolExecutor(
+ **executor_kwargs,
+ )
+ except Exception as exc:
+ self._enable_thread_fallback(exc)
+ return self._executor
+
+ def _ensure_thread_executor(self) -> ThreadPoolExecutor | None:
+ if self._direct_fallback_mode:
+ return None
+ if self._thread_executor is not None:
+ return self._thread_executor
+ try:
+ self._thread_executor = ThreadPoolExecutor(
+ max_workers=max(1, int(self._thread_worker_count)),
+ thread_name_prefix="prepared-batch-thread",
+ )
+ except Exception as exc:
+ self._enable_direct_fallback(exc)
+ return self._thread_executor
+
+ def iter_prepared_batches(
+ self,
+ rows: Sequence[QueryTrainingExample],
+ *,
+ graph_cache: Mapping[str, GraphCacheItem],
+ batch_size: int,
+ shuffle: bool,
+ seed: int,
+ lookahead_batches: int,
+ graph_error_stage: str,
+ skip_batches: int = 0,
+ prefer_completion_order: bool = DEFAULT_BATCH_PREPARE_PREFER_COMPLETION_ORDER,
+ worker_event_callback: Callable[[Dict[str, Any]], None] | None = None,
+ ) -> Iterator[Dict[str, Any]]:
+ batch_iter = iter(
+ _iter_batches_any(
+ rows,
+ batch_size=batch_size,
+ shuffle=shuffle,
+ seed=seed,
+ skip_batches=skip_batches,
+ )
+ )
+ resolved_worker_context = dict(self.worker_context or {})
+ has_conversation_specs = bool(dict(resolved_worker_context).get("conversation_specs"))
+ has_graph_dir = bool(clean_text(resolved_worker_context.get("graph_dir", "")))
+ if self.worker_count <= 0 or (not has_conversation_specs and not has_graph_dir):
+ for batch in batch_iter:
+ yield _prepare_batch_payload_from_graph_cache(
+ batch,
+ graph_cache,
+ merge_device=torch.device("cpu"),
+ graph_error_stage=graph_error_stage,
+ )
+ return
+
+ pending_items: deque[
+ tuple[QueryBatchSpec | List[QueryTrainingExample], Future[Dict[str, Any]] | None, List[Dict[str, Any]], str]
+ ] = deque()
+ worker_parallelism = max(self.worker_count, self._thread_worker_count)
+ pending_target = max(
+ 1,
+ max(0, int(lookahead_batches or 0), int(worker_parallelism) * int(DEFAULT_BATCH_PREPARE_PENDING_MULTIPLIER)) + 1,
+ )
+ resolved_graph_error_stage = clean_text(graph_error_stage) or "prepare_batch"
+ use_completion_order = bool(prefer_completion_order)
+
+ def _emit_worker_event(event_type: str, **payload: Any) -> None:
+ if worker_event_callback is None:
+ return
+ worker_event_callback(
+ {
+ "event_type": clean_text(event_type),
+ "graph_error_stage": resolved_graph_error_stage,
+ **dict(payload),
+ }
+ )
+
+ def _current_pending_target() -> int:
+ if self._direct_fallback_mode:
+ return 1
+ if self._thread_fallback_mode:
+ worker_parallelism = max(1, int(self._thread_worker_count))
+ return max(
+ 1,
+ max(0, int(lookahead_batches or 0), int(worker_parallelism) * int(DEFAULT_BATCH_PREPARE_PENDING_MULTIPLIER)) + 1,
+ )
+ process_parallelism = self._process_worker_count()
+ steady_target = max(
+ 1,
+ max(0, int(lookahead_batches or 0), int(process_parallelism) * int(DEFAULT_BATCH_PREPARE_PENDING_MULTIPLIER)) + 1,
+ )
+ if self._process_success_count > 0:
+ return steady_target
+ return max(
+ 2,
+ min(
+ steady_target,
+ max(
+ process_parallelism + 1,
+ int(process_parallelism) * int(DEFAULT_BATCH_PREPARE_PROCESS_WARMUP_PENDING_MULTIPLIER),
+ ),
+ ),
+ )
+
+ def _queue_local_batch(
+ batch_payload: QueryBatchSpec | List[QueryTrainingExample],
+ *,
+ worker_exc: Exception | None = None,
+ ) -> None:
+ worker_error_payloads = (
+ _batch_prepare_worker_error_payloads(
+ batch_payload,
+ worker_exc,
+ graph_error_stage=graph_error_stage,
+ )
+ if worker_exc is not None
+ else []
+ )
+ pending_items.append((batch_payload, None, list(worker_error_payloads), "local"))
+
+ def _submit_batch(
+ batch_payload: QueryBatchSpec | List[QueryTrainingExample],
+ *,
+ worker_exc: Exception | None = None,
+ ) -> None:
+ worker_error_payloads = (
+ _batch_prepare_worker_error_payloads(
+ batch_payload,
+ worker_exc,
+ graph_error_stage=graph_error_stage,
+ )
+ if worker_exc is not None
+ else []
+ )
+ if self._direct_fallback_mode:
+ pending_items.append((batch_payload, None, list(worker_error_payloads), "local"))
+ return
+ if not self._thread_fallback_mode:
+ executor = self._ensure_executor()
+ if executor is not None:
+ try:
+ future = executor.submit(_prepare_batch_worker_task, batch_payload, resolved_graph_error_stage)
+ except Exception as exc:
+ recovery_payload = self._recover_process_pool(exc)
+ _emit_worker_event(
+ "batch_prepare_process_pool_recovery",
+ error_type=type(exc).__name__,
+ error_message=clean_text(str(exc)),
+ backend="process_submit",
+ **dict(recovery_payload),
+ )
+ worker_error_payloads.extend(
+ _batch_prepare_worker_error_payloads(
+ batch_payload,
+ exc,
+ graph_error_stage=graph_error_stage,
+ )
+ )
+ else:
+ pending_items.append((batch_payload, future, list(worker_error_payloads), "process"))
+ return
+ thread_executor = self._ensure_thread_executor()
+ if thread_executor is not None:
+ try:
+ future = thread_executor.submit(
+ _prepare_batch_thread_task,
+ batch_payload,
+ resolved_graph_error_stage,
+ resolved_worker_context,
+ )
+ except Exception as exc:
+ self._enable_direct_fallback(exc)
+ worker_error_payloads.extend(
+ _batch_prepare_worker_error_payloads(
+ batch_payload,
+ exc,
+ graph_error_stage=graph_error_stage,
+ )
+ )
+ else:
+ pending_items.append((batch_payload, future, list(worker_error_payloads), "thread"))
+ return
+ pending_items.append((batch_payload, None, list(worker_error_payloads), "local"))
+
+ def _retry_pending_with_fresh_executor() -> None:
+ if not pending_items:
+ return
+ retry_batches = [batch_payload for batch_payload, _, _, _ in pending_items]
+ for _, future, _, _ in pending_items:
+ if future is not None:
+ try:
+ future.cancel()
+ except Exception:
+ pass
+ pending_items.clear()
+ for batch_payload in retry_batches:
+ _submit_batch(batch_payload)
+
+ def _submit_pending() -> None:
+ while len(pending_items) < _current_pending_target():
+ try:
+ raw_batch = next(batch_iter)
+ except StopIteration:
+ break
+ batch_payload = raw_batch if isinstance(raw_batch, QueryBatchSpec) else list(raw_batch)
+ _submit_batch(batch_payload)
+
+ def _pop_pending_item_at(
+ index: int,
+ ) -> tuple[QueryBatchSpec | List[QueryTrainingExample], Future[Dict[str, Any]] | None, List[Dict[str, Any]], str]:
+ pending_items.rotate(-int(index))
+ item = pending_items.popleft()
+ pending_items.rotate(int(index))
+ return item
+
+ def _pop_next_pending_item(
+ ) -> tuple[QueryBatchSpec | List[QueryTrainingExample], Future[Dict[str, Any]] | None, List[Dict[str, Any]], str]:
+ if not use_completion_order or len(pending_items) <= 1:
+ return pending_items.popleft()
+ for index, item in enumerate(pending_items):
+ if item[1] is None:
+ return _pop_pending_item_at(index)
+ for index, item in enumerate(pending_items):
+ future = item[1]
+ future_done = getattr(future, "done", None)
+ if future is not None and callable(future_done) and future_done():
+ return _pop_pending_item_at(index)
+ pending_futures = []
+ for _, future, _, _ in pending_items:
+ future_done = getattr(future, "done", None)
+ if future is None or not callable(future_done):
+ return pending_items.popleft()
+ pending_futures.append(future)
+ if pending_futures:
+ done_futures, _ = wait(pending_futures, return_when=FIRST_COMPLETED)
+ if done_futures:
+ for index, item in enumerate(pending_items):
+ future = item[1]
+ if future is not None and future in done_futures:
+ return _pop_pending_item_at(index)
+ return pending_items.popleft()
+
+ _submit_pending()
+ while pending_items:
+ batch_payload, future, queued_error_payloads, backend = _pop_next_pending_item()
+ prepared_batch: Dict[str, Any]
+ if future is None:
+ prepared_batch = _prepare_batch_payload_from_graph_cache(
+ batch_payload,
+ graph_cache,
+ merge_device=torch.device("cpu"),
+ graph_error_stage=graph_error_stage,
+ )
+ if queued_error_payloads:
+ prepared_batch["graph_error_payloads"] = list(queued_error_payloads) + list(prepared_batch.get("graph_error_payloads", []) or [])
+ prepared_batch["graph_error_count"] = len(prepared_batch["graph_error_payloads"])
+ else:
+ try:
+ prepared_batch = dict(future.result())
+ self._record_worker_success(backend=backend)
+ if backend == "process" and self._process_success_count == 1:
+ _emit_worker_event(
+ "batch_prepare_process_pool_ready",
+ backend="process",
+ worker_count=int(self._process_worker_count()),
+ )
+ if queued_error_payloads:
+ prepared_batch["graph_error_payloads"] = list(queued_error_payloads) + list(prepared_batch.get("graph_error_payloads", []) or [])
+ prepared_batch["graph_error_count"] = len(prepared_batch["graph_error_payloads"])
+ except (BrokenProcessPool, Exception) as exc:
+ if backend == "process":
+ recovery_payload = self._recover_process_pool(exc)
+ _emit_worker_event(
+ "batch_prepare_process_pool_recovery",
+ error_type=type(exc).__name__,
+ error_message=clean_text(str(exc)),
+ backend="process_result",
+ **dict(recovery_payload),
+ )
+ _retry_pending_with_fresh_executor()
+ elif backend == "thread":
+ self._worker_exc = exc
+ _emit_worker_event(
+ "batch_prepare_thread_fallback_error",
+ error_type=type(exc).__name__,
+ error_message=clean_text(str(exc)),
+ backend="thread_result",
+ )
+ prepared_batch = _prepare_batch_payload_from_graph_cache(
+ batch_payload,
+ graph_cache,
+ merge_device=torch.device("cpu"),
+ graph_error_stage=graph_error_stage,
+ )
+ worker_error_payloads = _batch_prepare_worker_error_payloads(
+ batch_payload,
+ exc,
+ graph_error_stage=graph_error_stage,
+ )
+ prepared_batch["graph_error_payloads"] = list(queued_error_payloads) + worker_error_payloads + list(prepared_batch.get("graph_error_payloads", []) or [])
+ prepared_batch["graph_error_count"] = len(prepared_batch["graph_error_payloads"])
+ yield prepared_batch
+ _submit_pending()
+
+
+def _build_graph_error_payload(
+ example: QueryTrainingExample,
+ exc: Exception,
+ *,
+ stage: str,
+ batch_conversation_ids: Sequence[str] | None = None,
+) -> Dict[str, Any]:
+ payload = {
+ "stage": clean_text(stage) or "score_batch",
+ "conversation_id": clean_text(example.conversation_id),
+ "question_id": clean_text(example.question_id),
+ "question": clean_text(example.question),
+ "error_type": type(exc).__name__,
+ "error_message": clean_text(str(exc)),
+ }
+ if batch_conversation_ids:
+ payload["batch_conversation_ids"] = [clean_text(item) for item in list(batch_conversation_ids) if clean_text(item)]
+ return payload
+
+
+def _emit_graph_error_payloads(
+ graph_error_payloads: Sequence[Mapping[str, Any]],
+ *,
+ graph_error_callback: Callable[[Dict[str, Any]], None] | None,
+) -> None:
+ if graph_error_callback is None:
+ return
+ for payload in graph_error_payloads:
+ graph_error_callback(dict(payload))
+
+
+def _empty_score_batch_result(*, device: torch.device, graph_error_count: int) -> Dict[str, Any]:
+ zero = torch.tensor(0.0, dtype=torch.float32, device=device)
+ return {
+ "loss": zero,
+ "metrics": {
+ "recall_loss": 0.0,
+ "loss": 0.0,
+ "event_loss": 0.0,
+ "path_loss": 0.0,
+ "temporal_loss": 0.0,
+ "answer_type_loss": 0.0,
+ "answer_plan_loss": 0.0,
+ "token_role_loss": 0.0,
+ "question_understanding_loss": 0.0,
+ "memory_router_loss": 0.0,
+ "memory_router_exact_match": 0.0,
+ "memory_router_f1": 0.0,
+ "event_distractor_loss": 0.0,
+ "event_tunnel_loss": 0.0,
+ "path_tunnel_loss": 0.0,
+ "path_tunnel_delta_loss": 0.0,
+ "event_tunnel_selection_loss": 0.0,
+ "path_tunnel_selection_loss": 0.0,
+ "event_hard_negative_loss": 0.0,
+ "path_hard_negative_loss": 0.0,
+ "recall_selection_loss": 0.0,
+ "event_selection_loss": 0.0,
+ "path_selection_loss": 0.0,
+ "final_event_set_loss": 0.0,
+ "event_matrix_delta_loss": 0.0,
+ "path_matrix_delta_loss": 0.0,
+ "answer_refusal_loss": 0.0,
+ "recall_event_recall_at_24": 0.0,
+ "event_recall_at_1": 0.0,
+ "event_recall_at_5": 0.0,
+ "path_recall_at_3": 0.0,
+ "path_tunnel_support_recall_at_3": 0.0,
+ "path_tunnel_support_positive_coverage_at_3": 0.0,
+ "path_tunnel_delta_recall_at_3": 0.0,
+ "path_tunnel_delta_positive_coverage_at_3": 0.0,
+ "answer_plan_selected_recall_at_5": 0.0,
+ "answer_plan_selected_positive_coverage_at_5": 0.0,
+ "answer_plan_current_top1_accuracy": 0.0,
+ "path_tunnel_rescue025_recall_at_3": 0.0,
+ "path_tunnel_rescue050_recall_at_3": 0.0,
+ "path_tunnel_rescue100_recall_at_3": 0.0,
+ "temporal_accuracy": 0.0,
+ "training_weight_mean": 1.0,
+ "training_weight_min": 1.0,
+ "training_weight_max": 1.0,
+ "samples": 0,
+ "loss_group_count": 0,
+ "answer_type_metrics": {},
+ "supervision_bucket_metrics": {},
+ "recall_loss_count": 0,
+ "event_loss_count": 0,
+ "path_loss_count": 0,
+ "temporal_loss_count": 0,
+ "answer_type_loss_count": 0,
+ "answer_plan_loss_count": 0,
+ "token_role_loss_count": 0,
+ "question_understanding_loss_count": 0,
+ "memory_router_loss_count": 0,
+ "memory_router_total": 0,
+ "event_distractor_loss_count": 0,
+ "event_tunnel_loss_count": 0,
+ "path_tunnel_loss_count": 0,
+ "path_tunnel_delta_loss_count": 0,
+ "event_tunnel_selection_loss_count": 0,
+ "path_tunnel_selection_loss_count": 0,
+ "event_hard_negative_loss_count": 0,
+ "path_hard_negative_loss_count": 0,
+ "recall_selection_loss_count": 0,
+ "event_selection_loss_count": 0,
+ "path_selection_loss_count": 0,
+ "final_event_set_loss_count": 0,
+ "event_matrix_delta_loss_count": 0,
+ "path_matrix_delta_loss_count": 0,
+ "answer_refusal_loss_count": 0,
+ "recall_event_recall_total": 0,
+ "event_recall_total": 0,
+ "path_recall_total": 0,
+ "path_tunnel_support_recall_total": 0,
+ "path_tunnel_support_positive_total": 0,
+ "path_tunnel_delta_recall_total": 0,
+ "path_tunnel_delta_positive_total": 0,
+ "answer_plan_selected_total": 0,
+ "answer_plan_selected_positive_total": 0,
+ "answer_plan_current_total": 0,
+ "path_tunnel_rescue025_recall_total": 0,
+ "path_tunnel_rescue050_recall_total": 0,
+ "path_tunnel_rescue100_recall_total": 0,
+ "temporal_total": 0,
+ "graph_error_count": int(graph_error_count),
+ },
+ }
+
+
+def _prepare_batch_payload(
+ batch: Sequence[QueryTrainingExample],
+ *,
+ graph_item_resolver: Callable[[str], GraphCacheItem | None],
+ merge_device: torch.device,
+ graph_error_stage: str,
+) -> Dict[str, Any]:
+ ordered_graph_tensors: OrderedDict[str, Mapping[str, Any]] = OrderedDict()
+ scored_examples: List[QueryTrainingExample] = []
+ prepared_supervision: List[Dict[str, Any]] = []
+ graph_error_payloads: List[Dict[str, Any]] = []
+ for example in batch:
+ conversation_id = clean_text(example.conversation_id)
+ try:
+ graph_item = graph_item_resolver(conversation_id)
+ except Exception as exc:
+ graph_error_payloads.append(_build_graph_error_payload(example, exc, stage=graph_error_stage))
+ continue
+ if graph_item is None:
+ continue
+ if conversation_id not in ordered_graph_tensors:
+ graph_tensors = graph_item.tensors
+ if isinstance(graph_tensors.get("node_type_ids"), Tensor) and graph_tensors["node_type_ids"].device != merge_device:
+ graph_tensors = _graph_tensor_mapping_to_device(
+ graph_tensors,
+ merge_device,
+ non_blocking=merge_device.type == "cuda",
+ )
+ ordered_graph_tensors[conversation_id] = graph_tensors
+ scored_examples.append(example)
+ prepared_supervision.append(_build_example_supervision_payload(example))
+
+ merged_graph_tensors: Dict[str, Any] | None = None
+ if scored_examples:
+ try:
+ if len(ordered_graph_tensors) == 1:
+ merged_graph_tensors = dict(next(iter(ordered_graph_tensors.values())))
+ else:
+ merged_graph_tensors = _combine_graph_tensors_by_conversation(ordered_graph_tensors, device=merge_device)
+ except Exception as exc:
+ batch_conversation_ids = list(ordered_graph_tensors)
+ graph_error_payloads.extend(
+ _build_graph_error_payload(
+ example,
+ exc,
+ stage=graph_error_stage,
+ batch_conversation_ids=batch_conversation_ids,
+ )
+ for example in scored_examples
+ )
+ scored_examples = []
+ merged_graph_tensors = None
+ return {
+ "scored_examples": list(scored_examples),
+ "prepared_supervision": list(prepared_supervision),
+ "merged_graph_tensors": merged_graph_tensors,
+ "graph_error_payloads": graph_error_payloads,
+ "graph_error_count": len(graph_error_payloads),
+ "prepared_conversation_ids": [clean_text(item) for item in list(ordered_graph_tensors) if clean_text(item)],
+ }
+
+
+def _prepare_batch_payload_from_graph_cache(
+ batch: QueryBatchSpec | Sequence[QueryTrainingExample],
+ graph_cache: Mapping[str, GraphCacheItem],
+ *,
+ merge_device: torch.device,
+ graph_error_stage: str,
+) -> Dict[str, Any]:
+ materialized_batch = _materialize_query_batch_payload(batch)
+ return _prepare_batch_payload(
+ materialized_batch,
+ graph_item_resolver=lambda conversation_id: graph_cache.get(conversation_id),
+ merge_device=merge_device,
+ graph_error_stage=graph_error_stage,
+ )
+
+
+def _normalize_batch_prepare_worker_context(worker_context: Mapping[str, Any] | None) -> Dict[str, Any]:
+ resolved_context = dict(worker_context or {})
+ return {
+ "conversation_specs": {
+ clean_text(conversation_id): dict(payload or {})
+ for conversation_id, payload in dict(resolved_context.get("conversation_specs", {}) or {}).items()
+ if clean_text(conversation_id)
+ },
+ "graph_dir": clean_text(resolved_context.get("graph_dir", "")),
+ "cache_dir": clean_text(resolved_context.get("cache_dir", "")),
+ "cache_write_enabled": bool(resolved_context.get("cache_write_enabled", True)),
+ "require_cache_hit": bool(resolved_context.get("require_cache_hit", False)),
+ "worker_memory_cache_size": max(0, int(resolved_context.get("worker_memory_cache_size", 0) or 0)),
+ }
+
+
+def _load_graph_item_from_worker_context(
+ conversation_id: str,
+ *,
+ worker_context: Mapping[str, Any],
+ worker_cache: OrderedDict[str, GraphCacheItem],
+ worker_memory_cache_size: int,
+) -> GraphCacheItem | None:
+ normalized_conversation_id = clean_text(conversation_id)
+ if not normalized_conversation_id:
+ return None
+ cached_item = worker_cache.get(normalized_conversation_id)
+ if cached_item is not None:
+ worker_cache.move_to_end(normalized_conversation_id)
+ return cached_item
+ resolved_context = dict(worker_context or {})
+ spec = dict(resolved_context.get("conversation_specs", {}) or {}).get(normalized_conversation_id)
+ graph_path_raw = clean_text((spec or {}).get("graph_path", ""))
+ cache_path_raw = clean_text((spec or {}).get("cache_path", ""))
+ if not graph_path_raw:
+ graph_dir_raw = clean_text(resolved_context.get("graph_dir", ""))
+ if not graph_dir_raw:
+ return None
+ graph_path_raw = str(Path(graph_dir_raw) / f"{normalized_conversation_id}.json")
+ graph_path = Path(graph_path_raw)
+ if not cache_path_raw:
+ cache_dir_raw = clean_text(resolved_context.get("cache_dir", ""))
+ if cache_dir_raw:
+ cache_path_raw = str(_graph_tensor_cache_path(Path(cache_dir_raw), graph_path))
+ cache_path = Path(cache_path_raw) if cache_path_raw else None
+ source_signature = _graph_source_signature(graph_path)
+ payload = _load_graph_tensor_cache(cache_path, source_signature=source_signature) if cache_path is not None else None
+ if payload is not None:
+ graph = dict(payload.get("graph", {}) or {})
+ tensors = _graph_tensor_mapping_to_device(payload.get("tensors", {}), torch.device("cpu"))
+ else:
+ if cache_path is not None and bool((spec or {}).get("require_cache_hit", resolved_context.get("require_cache_hit", False))):
+ raise FileNotFoundError(f"Prebuilt graph tensor cache missing for conversation '{normalized_conversation_id}': {graph_path}")
+ graph = dict(read_json(graph_path))
+ cache_tensors = tensorize_graph(graph, device=torch.device("cpu"))
+ if cache_path is not None and bool((spec or {}).get("cache_write_enabled", resolved_context.get("cache_write_enabled", True))):
+ _write_graph_tensor_cache(
+ cache_path,
+ source_signature=source_signature,
+ graph=graph,
+ tensors=cache_tensors,
+ )
+ tensors = _graph_tensor_mapping_to_device(cache_tensors, torch.device("cpu"))
+ tensors = dict(_ensure_graph_scoring_feature_cache(tensors))
+ item = GraphCacheItem(graph=graph, tensors=tensors)
+ if worker_memory_cache_size > 0:
+ worker_cache[normalized_conversation_id] = item
+ worker_cache.move_to_end(normalized_conversation_id)
+ while len(worker_cache) > worker_memory_cache_size:
+ worker_cache.popitem(last=False)
+ return item
+
+
+def _initialize_batch_prepare_worker(worker_context: Mapping[str, Any]) -> None:
+ torch.set_num_threads(1)
+ if hasattr(torch, "set_num_interop_threads"):
+ try:
+ torch.set_num_interop_threads(1)
+ except RuntimeError:
+ pass
+ global _BATCH_PREPARE_WORKER_CONTEXT, _BATCH_PREPARE_WORKER_CACHE, _BATCH_PREPARE_WORKER_MEMORY_CACHE_SIZE
+ normalized_context = _normalize_batch_prepare_worker_context(worker_context)
+ _BATCH_PREPARE_WORKER_CONTEXT = dict(normalized_context)
+ _BATCH_PREPARE_WORKER_CACHE = OrderedDict()
+ _BATCH_PREPARE_WORKER_MEMORY_CACHE_SIZE = max(0, int(normalized_context.get("worker_memory_cache_size", 0) or 0))
+
+
+def _load_graph_item_from_batch_prepare_worker(conversation_id: str) -> GraphCacheItem | None:
+ return _load_graph_item_from_worker_context(
+ conversation_id,
+ worker_context=_BATCH_PREPARE_WORKER_CONTEXT,
+ worker_cache=_BATCH_PREPARE_WORKER_CACHE,
+ worker_memory_cache_size=int(_BATCH_PREPARE_WORKER_MEMORY_CACHE_SIZE),
+ )
+
+
+def _thread_batch_prepare_state(worker_context: Mapping[str, Any]) -> Dict[str, Any]:
+ normalized_context = _normalize_batch_prepare_worker_context(worker_context)
+ context_key = (
+ clean_text(normalized_context.get("graph_dir", "")),
+ clean_text(normalized_context.get("cache_dir", "")),
+ bool(normalized_context.get("cache_write_enabled", True)),
+ bool(normalized_context.get("require_cache_hit", False)),
+ int(normalized_context.get("worker_memory_cache_size", 0) or 0),
+ len(dict(normalized_context.get("conversation_specs", {}) or {})),
+ )
+ state = getattr(_BATCH_PREPARE_THREAD_STATE, "state", None)
+ if not isinstance(state, Mapping) or tuple(state.get("context_key", ())) != context_key:
+ state = {
+ "context_key": context_key,
+ "context": dict(normalized_context),
+ "cache": OrderedDict(),
+ "worker_memory_cache_size": int(normalized_context.get("worker_memory_cache_size", 0) or 0),
+ }
+ _BATCH_PREPARE_THREAD_STATE.state = state
+ return dict(state)
+
+
+def _prepare_batch_thread_task(
+ batch: QueryBatchSpec | Sequence[QueryTrainingExample],
+ graph_error_stage: str,
+ worker_context: Mapping[str, Any],
+) -> Dict[str, Any]:
+ state = _thread_batch_prepare_state(worker_context)
+ thread_context = dict(state.get("context", {}) or {})
+ thread_cache = state.get("cache")
+ if not isinstance(thread_cache, OrderedDict):
+ thread_cache = OrderedDict()
+ _BATCH_PREPARE_THREAD_STATE.state = {
+ **dict(state),
+ "cache": thread_cache,
+ }
+ return _prepare_batch_payload(
+ _materialize_query_batch_payload(batch),
+ graph_item_resolver=lambda conversation_id: _load_graph_item_from_worker_context(
+ conversation_id,
+ worker_context=thread_context,
+ worker_cache=thread_cache,
+ worker_memory_cache_size=int(state.get("worker_memory_cache_size", 0) or 0),
+ ),
+ merge_device=torch.device("cpu"),
+ graph_error_stage=graph_error_stage,
+ )
+
+
+def _prepare_batch_worker_task(
+ batch: QueryBatchSpec | Sequence[QueryTrainingExample],
+ graph_error_stage: str,
+) -> Dict[str, Any]:
+ return _prepare_batch_payload(
+ _materialize_query_batch_payload(batch),
+ graph_item_resolver=_load_graph_item_from_batch_prepare_worker,
+ merge_device=torch.device("cpu"),
+ graph_error_stage=graph_error_stage,
+ )
+
+
+def _batch_prepare_worker_error_payloads(
+ batch: QueryBatchSpec | Sequence[QueryTrainingExample],
+ exc: Exception,
+ *,
+ graph_error_stage: str,
+) -> List[Dict[str, Any]]:
+ worker_stage = f"{clean_text(graph_error_stage) or 'prepare_batch'}_worker"
+ materialized_batch = _materialize_query_batch_payload(batch)
+ return [
+ _build_graph_error_payload(example, exc, stage=worker_stage)
+ for example in materialized_batch
+ ]
+
+
+def _iter_prepared_batches_with_workers(
+ rows: Sequence[QueryTrainingExample],
+ *,
+ graph_cache: Mapping[str, GraphCacheItem],
+ batch_size: int,
+ shuffle: bool,
+ seed: int,
+ worker_context: Mapping[str, Any] | None,
+ worker_count: int,
+ lookahead_batches: int,
+ graph_error_stage: str,
+ skip_batches: int = 0,
+ worker_pool: _PreparedBatchWorkerPool | None = None,
+ prefer_completion_order: bool = DEFAULT_BATCH_PREPARE_PREFER_COMPLETION_ORDER,
+ worker_event_callback: Callable[[Dict[str, Any]], None] | None = None,
+) -> Iterator[Dict[str, Any]]:
+ resolved_pool = worker_pool
+ created_pool = False
+ if resolved_pool is None:
+ resolved_pool = _PreparedBatchWorkerPool(
+ worker_context=worker_context,
+ worker_count=worker_count,
+ )
+ created_pool = True
+ try:
+ yield from resolved_pool.iter_prepared_batches(
+ rows,
+ graph_cache=graph_cache,
+ batch_size=batch_size,
+ shuffle=shuffle,
+ seed=seed,
+ lookahead_batches=lookahead_batches,
+ graph_error_stage=graph_error_stage,
+ skip_batches=skip_batches,
+ prefer_completion_order=prefer_completion_order,
+ worker_event_callback=worker_event_callback,
+ )
+ finally:
+ if created_pool and resolved_pool is not None:
+ resolved_pool.close()
+
+
+def score_prepared_batch(
+ model: LocomoNodeMemoryModel,
+ prepared_batch: Mapping[str, Any],
+ *,
+ device: torch.device,
+ loss_config: Mapping[str, Any] | None = None,
+ score_kwargs: Mapping[str, Any] | None = None,
+ apply_example_weights: bool = False,
+ graph_error_callback: Callable[[Dict[str, Any]], None] | None = None,
+ graph_error_stage: str = "",
+) -> Dict[str, Any]:
+ resolved_loss_config = {**DEFAULT_TRAINING_CONFIG, **dict(loss_config or {})}
+ losses: List[Tensor] = []
+ loss_entries: List[tuple[Tensor, str]] = []
+ recall_loss_values: List[float] = []
+ event_loss_values: List[float] = []
+ path_loss_values: List[float] = []
+ temporal_loss_values: List[float] = []
+ answer_type_loss_values: List[float] = []
+ answer_plan_loss_values: List[float] = []
+ token_role_loss_values: List[float] = []
+ question_understanding_loss_values: List[float] = []
+ memory_router_loss_values: List[float] = []
+ event_distractor_loss_values: List[float] = []
+ event_tunnel_loss_values: List[float] = []
+ path_tunnel_loss_values: List[float] = []
+ path_tunnel_delta_loss_values: List[float] = []
+ event_tunnel_selection_loss_values: List[float] = []
+ path_tunnel_selection_loss_values: List[float] = []
+ event_hard_negative_loss_values: List[float] = []
+ path_hard_negative_loss_values: List[float] = []
+ recall_selection_loss_values: List[float] = []
+ event_selection_loss_values: List[float] = []
+ path_selection_loss_values: List[float] = []
+ final_event_set_loss_values: List[float] = []
+ event_matrix_delta_loss_values: List[float] = []
+ path_matrix_delta_loss_values: List[float] = []
+ answer_refusal_loss_values: List[float] = []
+ training_weight_values: List[float] = []
+ answer_type_metric_stats: Dict[str, Dict[str, int]] = {}
+ supervision_bucket_metric_stats: Dict[str, Dict[str, int]] = {}
+ recall_event24_hits = 0
+ recall_event24_positive_hits = 0
+ recall_event24_positive_total = 0
+ recall_event_total = 0
+ event_recall1_hits = 0
+ event_recall5_hits = 0
+ event5_positive_hits = 0
+ event5_positive_total = 0
+ path_recall3_hits = 0
+ path3_positive_hits = 0
+ path3_positive_total = 0
+ path_tunnel_support_recall3_hits = 0
+ path_tunnel_support_positive_hits = 0
+ path_tunnel_support_positive_total = 0
+ path_tunnel_support_recall_total = 0
+ path_tunnel_delta_recall3_hits = 0
+ path_tunnel_delta_positive_hits = 0
+ path_tunnel_delta_positive_total = 0
+ path_tunnel_delta_recall_total = 0
+ answer_plan_selected_recall5_hits = 0
+ answer_plan_selected_total = 0
+ answer_plan_selected_positive_hits = 0
+ answer_plan_selected_positive_total = 0
+ answer_plan_current_top1_hits = 0
+ answer_plan_current_total = 0
+ path_tunnel_rescue025_recall3_hits = 0
+ path_tunnel_rescue025_recall_total = 0
+ path_tunnel_rescue050_recall3_hits = 0
+ path_tunnel_rescue050_recall_total = 0
+ path_tunnel_rescue100_recall3_hits = 0
+ path_tunnel_rescue100_recall_total = 0
+ temporal_hits = 0
+ memory_router_exact_hits = 0
+ memory_router_f1_sum = 0.0
+ memory_router_total = 0
+ recall_loss_count = 0
+ event_loss_count = 0
+ path_loss_count = 0
+ temporal_loss_count = 0
+ answer_type_loss_count = 0
+ answer_plan_loss_count = 0
+ token_role_loss_count = 0
+ question_understanding_loss_count = 0
+ memory_router_loss_count = 0
+ event_distractor_loss_count = 0
+ event_tunnel_loss_count = 0
+ path_tunnel_loss_count = 0
+ path_tunnel_delta_loss_count = 0
+ event_tunnel_selection_loss_count = 0
+ path_tunnel_selection_loss_count = 0
+ event_hard_negative_loss_count = 0
+ path_hard_negative_loss_count = 0
+ recall_selection_loss_count = 0
+ event_selection_loss_count = 0
+ path_selection_loss_count = 0
+ final_event_set_loss_count = 0
+ event_matrix_delta_loss_count = 0
+ path_matrix_delta_loss_count = 0
+ answer_refusal_loss_count = 0
+ graph_error_payloads = [dict(payload) for payload in list(prepared_batch.get("graph_error_payloads", []) or [])]
+ _emit_graph_error_payloads(graph_error_payloads, graph_error_callback=graph_error_callback)
+ graph_error_count = len(graph_error_payloads)
+ event_recall_total = 0
+ path_recall_total = 0
+ temporal_total = 0
+ scored_examples = list(prepared_batch.get("scored_examples", []) or [])
+ prepared_supervision = list(prepared_batch.get("prepared_supervision", []) or [])
+ merged_graph_tensors = prepared_batch.get("merged_graph_tensors")
+
+ if not scored_examples or not isinstance(merged_graph_tensors, Mapping):
+ return _empty_score_batch_result(device=device, graph_error_count=graph_error_count)
+
+ prepared_conversation_ids = [
+ clean_text(item) for item in list(prepared_batch.get("prepared_conversation_ids", []) or []) if clean_text(item)
+ ]
+ working_graph_tensors = dict(merged_graph_tensors)
+ if isinstance(working_graph_tensors.get("node_type_ids"), Tensor) and working_graph_tensors["node_type_ids"].device != device:
+ working_graph_tensors = _graph_tensor_mapping_to_device(
+ working_graph_tensors,
+ device,
+ non_blocking=device.type == "cuda",
+ )
+
+ with _gc_suspended():
+ try:
+ outputs_batch = _call_with_supported_kwargs(
+ model.score_examples,
+ working_graph_tensors,
+ scored_examples,
+ **dict(score_kwargs or {}),
+ )
+ except Exception as exc:
+ graph_error_count += len(scored_examples)
+ if graph_error_callback is not None:
+ for example in scored_examples:
+ graph_error_callback(
+ _build_graph_error_payload(
+ example,
+ exc,
+ stage=clean_text(graph_error_stage) or "score_prepared_batch",
+ batch_conversation_ids=prepared_conversation_ids,
+ )
+ )
+ return _empty_score_batch_result(device=device, graph_error_count=graph_error_count)
+
+ for example_index, (example, outputs) in enumerate(zip(scored_examples, outputs_batch)):
+ supervision = (
+ prepared_supervision[example_index]
+ if example_index < len(prepared_supervision) and isinstance(prepared_supervision[example_index], Mapping)
+ else _build_example_supervision_payload(example)
+ )
+ example_training_weight = max(0.0, float(supervision.get("training_example_weight", 1.0) or 1.0))
+ training_weight_values.append(example_training_weight)
+ answer_type_name = _training_answer_type_name(example)
+ answer_stats = answer_type_metric_stats.setdefault(answer_type_name, _new_answer_type_metric_stats())
+ answer_stats["samples"] += 1
+ supervision_bucket_name = clean_text(supervision.get("supervision_bucket", "")) or _training_supervision_bucket_name(example)
+ supervision_stats = supervision_bucket_metric_stats.setdefault(
+ supervision_bucket_name,
+ _new_answer_type_metric_stats(),
+ )
+ supervision_stats["samples"] += 1
+ example_recall_total = 0
+ example_recall24_hit = 0
+ example_recall24_positive_hits = 0
+ example_recall24_positive_total = 0
+ example_event_total = 0
+ example_event1_hit = 0
+ example_event5_hit = 0
+ example_event5_positive_hits = 0
+ example_event5_positive_total = 0
+ example_path_total = 0
+ example_path3_hit = 0
+ example_path3_positive_hits = 0
+ example_path3_positive_total = 0
+ example_temporal_total = 0
+ example_temporal_hit = 0
+ example_answer_plan_total = 0
+ example_answer_plan_recall5_hit = 0
+ example_answer_plan_positive_hits = 0
+ example_answer_plan_positive_total = 0
+ example_answer_plan_current_total = 0
+ example_answer_plan_current_top1_hit = 0
+ total_loss = outputs["answer_type_logits"].new_zeros(())
+ answer_type_logits = outputs["answer_type_logits"].unsqueeze(0)
+ answer_type_target = torch.tensor(
+ [int(supervision.get("answer_type_index", _answer_type_label(example)))],
+ dtype=torch.long,
+ device=device,
+ )
+ question_understanding_targets = _question_understanding_targets(example.question_features or {})
+ question_semantic_logits = outputs.get("question_semantic_logits")
+ question_status_logits = outputs.get("question_status_logits")
+ question_time_granularity_logits = outputs.get("question_time_granularity_logits")
+ question_temporal_logit = outputs.get("question_temporal_logit")
+ if all(
+ isinstance(value, Tensor)
+ for value in (
+ question_semantic_logits,
+ question_status_logits,
+ question_time_granularity_logits,
+ question_temporal_logit,
+ )
+ ):
+ question_semantic_loss = F.cross_entropy(
+ question_semantic_logits.unsqueeze(0),
+ torch.tensor(
+ [int(question_understanding_targets["semantic_slot_index"])],
+ dtype=torch.long,
+ device=device,
+ ),
+ )
+ question_status_loss = F.cross_entropy(
+ question_status_logits.unsqueeze(0),
+ torch.tensor(
+ [int(question_understanding_targets["target_status_index"])],
+ dtype=torch.long,
+ device=device,
+ ),
+ )
+ question_time_granularity_loss = F.cross_entropy(
+ question_time_granularity_logits.unsqueeze(0),
+ torch.tensor(
+ [int(question_understanding_targets["time_granularity_index"])],
+ dtype=torch.long,
+ device=device,
+ ),
+ )
+ question_temporal_loss = F.binary_cross_entropy_with_logits(
+ question_temporal_logit.reshape(1),
+ torch.tensor(
+ [float(question_understanding_targets["is_temporal_value"])],
+ dtype=torch.float32,
+ device=device,
+ ),
+ )
+ question_understanding_loss = torch.stack(
+ [
+ question_semantic_loss,
+ question_status_loss,
+ question_time_granularity_loss,
+ question_temporal_loss,
+ ]
+ ).mean()
+ total_loss = (
+ total_loss
+ + float(resolved_loss_config["question_understanding_loss_weight"]) * question_understanding_loss
+ )
+ question_understanding_loss_values.append(float(question_understanding_loss.detach().cpu()))
+ question_understanding_loss_count += 1
+ answer_type_loss = F.cross_entropy(answer_type_logits, answer_type_target)
+ total_loss = total_loss + float(resolved_loss_config["answer_type_loss_weight"]) * answer_type_loss
+ answer_type_loss_values.append(float(answer_type_loss.detach().cpu()))
+ answer_type_loss_count += 1
+ memory_router_logits = outputs.get("memory_router_logits")
+ if isinstance(memory_router_logits, Tensor) and memory_router_logits.numel() == len(MEMORY_ROUTER_LAYERS):
+ memory_router_targets = _memory_router_target_tensor(example, supervision, device=device)
+ memory_router_loss = F.binary_cross_entropy_with_logits(
+ memory_router_logits.reshape(-1),
+ memory_router_targets,
+ )
+ total_loss = total_loss + float(resolved_loss_config["memory_router_loss_weight"]) * memory_router_loss
+ memory_router_loss_values.append(float(memory_router_loss.detach().cpu()))
+ memory_router_loss_count += 1
+ router_probs = torch.sigmoid(memory_router_logits.detach().reshape(-1))
+ router_pred = router_probs >= 0.5
+ router_target = memory_router_targets.detach() >= 0.5
+ if bool(torch.equal(router_pred, router_target)):
+ memory_router_exact_hits += 1
+ tp = int((router_pred & router_target).sum().item())
+ fp = int((router_pred & ~router_target).sum().item())
+ fn = int((~router_pred & router_target).sum().item())
+ memory_router_f1_sum += (2.0 * tp) / max(1.0, float(2 * tp + fp + fn))
+ memory_router_total += 1
+ if example_index == 0:
+ question_token_role_aux_loss = outputs.get("question_token_role_aux_loss")
+ node_token_role_aux_loss = outputs.get("node_token_role_aux_loss")
+ token_role_aux_losses = [
+ loss
+ for loss in (question_token_role_aux_loss, node_token_role_aux_loss)
+ if isinstance(loss, Tensor) and loss.numel() > 0
+ ]
+ if token_role_aux_losses:
+ token_role_loss = torch.stack([loss.float() for loss in token_role_aux_losses]).mean()
+ total_loss = total_loss + float(resolved_loss_config["token_role_loss_weight"]) * token_role_loss
+ token_role_loss_values.append(float(token_role_loss.detach().cpu()))
+ token_role_loss_count += 1
+ answer_refusal_loss = _answer_refusal_margin_loss(
+ outputs["answer_type_logits"],
+ target_index=int(answer_type_target.item()),
+ has_positive_supervision=bool(supervision.get("has_positive_supervision", False)),
+ margin=float(resolved_loss_config["answer_refusal_margin"]),
+ )
+ total_loss = total_loss + float(resolved_loss_config["answer_refusal_loss_weight"]) * answer_refusal_loss
+ answer_refusal_loss_values.append(float(answer_refusal_loss.detach().cpu()))
+ answer_refusal_loss_count += 1
+
+ answer_plan_logits = outputs.get("answer_plan_logits")
+ if (
+ bool(supervision.get("has_answer_plan_supervision", False))
+ and isinstance(answer_plan_logits, Tensor)
+ and answer_plan_logits.numel() > 0
+ ):
+ candidate_ids_for_plan = outputs["candidate_event_ids"]
+ selected_plan_ids = supervision.get("answer_plan_selected_event_ids", frozenset())
+ current_plan_ids = supervision.get("answer_plan_current_event_ids", frozenset())
+ historical_plan_ids = supervision.get("answer_plan_historical_event_ids", frozenset())
+ suppressed_plan_ids = supervision.get("answer_plan_suppressed_event_ids", frozenset())
+ role_losses: List[Tensor] = []
+ for role_name, supervision_key in (
+ ("selected", "answer_plan_selected_event_ids"),
+ ("current", "answer_plan_current_event_ids"),
+ ("historical", "answer_plan_historical_event_ids"),
+ ("suppressed", "answer_plan_suppressed_event_ids"),
+ ):
+ role_ids = supervision.get(supervision_key, frozenset())
+ if not role_ids:
+ continue
+ role_index = ANSWER_PLAN_OUTPUT_TO_ID[role_name]
+ role_losses.append(
+ _binary_plan_loss_for_ids(
+ answer_plan_logits[:, role_index],
+ candidate_ids=candidate_ids_for_plan,
+ positive_ids=role_ids,
+ device=device,
+ )
+ )
+ current_old_weight = float(resolved_loss_config.get("answer_plan_current_old_margin_loss_weight", 0.0) or 0.0)
+ if current_old_weight > 0.0 and current_plan_ids:
+ old_or_suppressed_ids = _normalized_id_frozenset(historical_plan_ids) | _normalized_id_frozenset(suppressed_plan_ids)
+ current_margin_loss = _positive_vs_negative_ids_margin_loss(
+ answer_plan_logits[:, ANSWER_PLAN_OUTPUT_TO_ID["current"]],
+ candidate_ids=candidate_ids_for_plan,
+ positive_ids=current_plan_ids,
+ negative_ids=old_or_suppressed_ids,
+ margin=float(resolved_loss_config.get("answer_plan_current_old_margin", 0.2) or 0.2),
+ )
+ total_loss = total_loss + current_old_weight * current_margin_loss
+ answer_plan_loss_values.append(float(current_margin_loss.detach().cpu()))
+ selected_negative_weight = float(
+ resolved_loss_config.get("answer_plan_selected_negative_margin_loss_weight", 0.0) or 0.0
+ )
+ if selected_negative_weight > 0.0 and selected_plan_ids:
+ selected_negative_ids = (
+ _normalized_id_frozenset(suppressed_plan_ids)
+ | _normalized_id_frozenset(supervision.get("hard_negative_event_ids", frozenset()))
+ | _normalized_id_frozenset(supervision.get("negative_event_ids", frozenset()))
+ )
+ selected_negative_margin_loss = _positive_vs_negative_ids_margin_loss(
+ answer_plan_logits[:, ANSWER_PLAN_OUTPUT_TO_ID["selected"]],
+ candidate_ids=candidate_ids_for_plan,
+ positive_ids=selected_plan_ids,
+ negative_ids=selected_negative_ids,
+ margin=float(
+ resolved_loss_config.get("answer_plan_selected_negative_margin", 0.3) or 0.3
+ ),
+ )
+ total_loss = total_loss + selected_negative_weight * selected_negative_margin_loss
+ answer_plan_loss_values.append(float(selected_negative_margin_loss.detach().cpu()))
+ if selected_plan_ids:
+ selected_index = ANSWER_PLAN_OUTPUT_TO_ID["selected"]
+ role_losses.append(
+ _selection_margin_loss(
+ answer_plan_logits[:, selected_index],
+ candidate_ids=candidate_ids_for_plan,
+ positive_ids=selected_plan_ids,
+ margin=float(resolved_loss_config["answer_plan_selection_margin"]),
+ top_k=int(resolved_loss_config["answer_plan_selection_top_k"]),
+ required_positive_count=_selection_positive_coverage_count(
+ supervision,
+ positive_count=len(selected_plan_ids),
+ top_k=int(resolved_loss_config["answer_plan_selection_top_k"]),
+ base_count=1,
+ multi_positive_count=min(3, max(1, len(selected_plan_ids))),
+ multi_positive_fraction=0.5,
+ ),
+ )
+ )
+ if role_losses:
+ answer_plan_loss = torch.stack([loss.float() for loss in role_losses]).mean()
+ total_loss = total_loss + float(resolved_loss_config["answer_plan_loss_weight"]) * answer_plan_loss
+ answer_plan_loss_values.append(float(answer_plan_loss.detach().cpu()))
+ answer_plan_loss_count += 1
+ if selected_plan_ids:
+ selected_scores = torch.sigmoid(answer_plan_logits[:, ANSWER_PLAN_OUTPUT_TO_ID["selected"]]).detach()
+ selected_ranked_indices = torch.argsort(selected_scores, descending=True)
+ selected_ranked_event_ids = [
+ candidate_ids_for_plan[int(index)]
+ for index in selected_ranked_indices.tolist()
+ if int(index) < len(candidate_ids_for_plan)
+ ]
+ answer_plan_selected_total += 1
+ example_answer_plan_total = 1
+ if selected_ranked_event_ids[:5] and any(item in selected_plan_ids for item in selected_ranked_event_ids[:5]):
+ answer_plan_selected_recall5_hits += 1
+ example_answer_plan_recall5_hit = 1
+ example_answer_plan_positive_hits = _positive_coverage_hit_count(
+ selected_ranked_event_ids,
+ selected_plan_ids,
+ limit=5,
+ )
+ example_answer_plan_positive_total = min(len(selected_plan_ids), 5)
+ answer_plan_selected_positive_hits += example_answer_plan_positive_hits
+ answer_plan_selected_positive_total += example_answer_plan_positive_total
+ if current_plan_ids:
+ current_scores = torch.sigmoid(answer_plan_logits[:, ANSWER_PLAN_OUTPUT_TO_ID["current"]]).detach()
+ current_ranked_indices = torch.argsort(current_scores, descending=True)
+ current_ranked_event_ids = [
+ candidate_ids_for_plan[int(index)]
+ for index in current_ranked_indices.tolist()
+ if int(index) < len(candidate_ids_for_plan)
+ ]
+ answer_plan_current_total += 1
+ example_answer_plan_current_total = 1
+ if current_ranked_event_ids and current_ranked_event_ids[0] in current_plan_ids:
+ answer_plan_current_top1_hits += 1
+ example_answer_plan_current_top1_hit = 1
+
+ positive_event_ids = supervision.get("positive_event_ids", frozenset())
+ positive_path_ids = supervision.get("positive_path_ids", frozenset())
+ negative_event_ids = supervision.get("negative_event_ids", frozenset())
+ hard_negative_event_ids = supervision.get("hard_negative_event_ids", negative_event_ids)
+ if positive_event_ids and outputs["recall_event_logits"].numel() > 0:
+ recall_targets = _label_tensor_from_positive_ids(
+ outputs["recall_event_ids"],
+ positive_event_ids,
+ device=device,
+ )
+ recall_loss = _recall_loss(outputs["recall_event_logits"], recall_targets)
+ total_loss = total_loss + float(resolved_loss_config["recall_loss_weight"]) * recall_loss
+ recall_loss_values.append(float(recall_loss.detach().cpu()))
+ recall_loss_count += 1
+ recall_selection_loss = _selection_margin_loss(
+ outputs["recall_event_logits"],
+ candidate_ids=outputs["recall_event_ids"],
+ positive_ids=positive_event_ids,
+ margin=float(resolved_loss_config["recall_selection_margin"]),
+ top_k=int(resolved_loss_config["recall_selection_top_k"]),
+ required_positive_count=_selection_positive_coverage_count(
+ supervision,
+ positive_count=len(positive_event_ids),
+ top_k=int(resolved_loss_config["recall_selection_top_k"]),
+ base_count=max(1, int(resolved_loss_config.get("recall_selection_positive_coverage_count", 1) or 1)),
+ multi_positive_count=max(
+ 1,
+ int(resolved_loss_config.get("multi_positive_recall_coverage_count", 1) or 1),
+ ),
+ multi_positive_fraction=float(
+ resolved_loss_config.get("multi_positive_coverage_fraction", 0.0) or 0.0
+ ),
+ ),
+ )
+ total_loss = total_loss + float(resolved_loss_config["recall_selection_loss_weight"]) * recall_selection_loss
+ recall_selection_loss_values.append(float(recall_selection_loss.detach().cpu()))
+ recall_selection_loss_count += 1
+ recall_scores = torch.sigmoid(outputs["recall_event_logits"]).detach()
+ ranked_indices = torch.argsort(recall_scores, descending=True)
+ ranked_event_ids = [outputs["recall_event_ids"][int(index)] for index in ranked_indices.tolist()]
+ recall_event_total += 1
+ example_recall_total = 1
+ if ranked_event_ids[:24] and any(item in positive_event_ids for item in ranked_event_ids[:24]):
+ recall_event24_hits += 1
+ example_recall24_hit = 1
+ example_recall24_positive_hits = _positive_coverage_hit_count(
+ ranked_event_ids,
+ positive_event_ids,
+ limit=24,
+ )
+ example_recall24_positive_total = min(len(positive_event_ids), 24)
+ recall_event24_positive_hits += example_recall24_positive_hits
+ recall_event24_positive_total += example_recall24_positive_total
+
+ if positive_event_ids and outputs["event_logits"].numel() > 0:
+ online_hard_negative_event_ids = _online_hard_negative_ids(
+ outputs["event_logits"],
+ candidate_ids=outputs["candidate_event_ids"],
+ positive_ids=positive_event_ids,
+ limit=int(resolved_loss_config.get("online_event_hard_negative_limit", 0) or 0),
+ )
+ combined_hard_negative_event_ids = dedupe_texts(
+ [*list(hard_negative_event_ids or []), *online_hard_negative_event_ids]
+ )
+ event_targets = _label_tensor_from_positive_ids(
+ outputs["candidate_event_ids"],
+ positive_event_ids,
+ device=device,
+ )
+ event_loss = _ranking_binary_loss(outputs["event_logits"], event_targets, margin=0.25)
+ total_loss = total_loss + float(resolved_loss_config["event_loss_weight"]) * event_loss
+ event_loss_values.append(float(event_loss.detach().cpu()))
+ event_loss_count += 1
+ event_distractor_loss = _event_distractor_supervision_loss(
+ outputs.get("event_distractor_logits", outputs["event_logits"]),
+ candidate_ids=outputs["candidate_event_ids"],
+ positive_ids=positive_event_ids,
+ hard_negative_ids=combined_hard_negative_event_ids,
+ )
+ total_loss = total_loss + float(resolved_loss_config["event_distractor_loss_weight"]) * event_distractor_loss
+ event_distractor_loss_values.append(float(event_distractor_loss.detach().cpu()))
+ event_distractor_loss_count += 1
+ tunnel_positive_event_ids = _tunnel_positive_event_ids_from_supervision(supervision)
+ tunnel_hard_negative_event_ids = _tunnel_hard_negative_event_ids_from_supervision(supervision)
+ event_tunnel_logits = outputs.get("event_tunnel_support_logits", outputs["event_logits"].new_zeros((0,)))
+ if tunnel_positive_event_ids and isinstance(event_tunnel_logits, Tensor) and event_tunnel_logits.numel() > 0:
+ event_tunnel_targets = _label_tensor_from_positive_ids(
+ outputs["candidate_event_ids"],
+ tunnel_positive_event_ids,
+ device=device,
+ )
+ event_tunnel_loss = _ranking_binary_loss(event_tunnel_logits, event_tunnel_targets, margin=0.2)
+ if tunnel_hard_negative_event_ids:
+ event_tunnel_loss = 0.5 * (
+ event_tunnel_loss
+ + _hard_negative_margin_loss(
+ event_tunnel_logits,
+ candidate_ids=outputs["candidate_event_ids"],
+ positive_ids=tunnel_positive_event_ids,
+ hard_negative_ids=tunnel_hard_negative_event_ids,
+ margin=float(resolved_loss_config["event_tunnel_margin"]),
+ )
+ )
+ total_loss = total_loss + float(resolved_loss_config["event_tunnel_loss_weight"]) * event_tunnel_loss
+ event_tunnel_loss_values.append(float(event_tunnel_loss.detach().cpu()))
+ event_tunnel_loss_count += 1
+ event_tunnel_selection_loss = _selection_margin_loss(
+ event_tunnel_logits,
+ candidate_ids=outputs["candidate_event_ids"],
+ positive_ids=tunnel_positive_event_ids,
+ margin=float(resolved_loss_config["event_tunnel_margin"]),
+ top_k=int(resolved_loss_config["final_event_set_top_k"]),
+ required_positive_count=_selection_positive_coverage_count(
+ supervision,
+ positive_count=len(tunnel_positive_event_ids),
+ top_k=int(resolved_loss_config["final_event_set_top_k"]),
+ base_count=max(1, int(resolved_loss_config.get("final_event_set_positive_coverage_count", 1) or 1)),
+ multi_positive_count=max(
+ 1,
+ int(resolved_loss_config.get("multi_positive_final_event_set_coverage_count", 1) or 1),
+ ),
+ multi_positive_fraction=float(
+ resolved_loss_config.get("multi_positive_coverage_fraction", 0.0) or 0.0
+ ),
+ ),
+ )
+ total_loss = total_loss + float(resolved_loss_config["event_tunnel_selection_loss_weight"]) * event_tunnel_selection_loss
+ event_tunnel_selection_loss_values.append(float(event_tunnel_selection_loss.detach().cpu()))
+ event_tunnel_selection_loss_count += 1
+ event_hard_negative_loss = _hard_negative_margin_loss(
+ outputs["event_logits"],
+ candidate_ids=outputs["candidate_event_ids"],
+ positive_ids=positive_event_ids,
+ hard_negative_ids=combined_hard_negative_event_ids,
+ margin=float(resolved_loss_config["event_hard_negative_margin"]),
+ )
+ total_loss = total_loss + float(resolved_loss_config["event_hard_negative_loss_weight"]) * event_hard_negative_loss
+ event_hard_negative_loss_values.append(float(event_hard_negative_loss.detach().cpu()))
+ event_hard_negative_loss_count += 1
+ event_selection_loss = _selection_margin_loss(
+ outputs["event_logits"],
+ candidate_ids=outputs["candidate_event_ids"],
+ positive_ids=positive_event_ids,
+ margin=float(resolved_loss_config["event_selection_margin"]),
+ top_k=int(resolved_loss_config["event_selection_top_k"]),
+ required_positive_count=_selection_positive_coverage_count(
+ supervision,
+ positive_count=len(positive_event_ids),
+ top_k=int(resolved_loss_config["event_selection_top_k"]),
+ base_count=max(1, int(resolved_loss_config.get("event_selection_positive_coverage_count", 1) or 1)),
+ multi_positive_count=max(
+ 1,
+ int(resolved_loss_config.get("multi_positive_event_coverage_count", 1) or 1),
+ ),
+ multi_positive_fraction=float(
+ resolved_loss_config.get("multi_positive_coverage_fraction", 0.0) or 0.0
+ ),
+ ),
+ )
+ total_loss = total_loss + float(resolved_loss_config["event_selection_loss_weight"]) * event_selection_loss
+ event_selection_loss_values.append(float(event_selection_loss.detach().cpu()))
+ event_selection_loss_count += 1
+ final_event_set_loss = _runtime_selected_event_set_loss(
+ outputs["event_logits"],
+ candidate_event_ids=outputs["candidate_event_ids"],
+ path_logits=outputs.get("path_logits", outputs["event_logits"].new_zeros((0,))),
+ candidate_path_event_ids=outputs.get("candidate_path_event_ids", []),
+ positive_event_ids=positive_event_ids,
+ margin=float(resolved_loss_config["final_event_set_margin"]),
+ top_k=int(resolved_loss_config["final_event_set_top_k"]),
+ support_path_k=int(resolved_loss_config.get("final_event_set_support_path_k", 3) or 3),
+ required_positive_count=_selection_positive_coverage_count(
+ supervision,
+ positive_count=len(positive_event_ids),
+ top_k=int(resolved_loss_config["final_event_set_top_k"]),
+ base_count=max(1, int(resolved_loss_config.get("final_event_set_positive_coverage_count", 1) or 1)),
+ multi_positive_count=max(
+ 1,
+ int(resolved_loss_config.get("multi_positive_final_event_set_coverage_count", 1) or 1),
+ ),
+ multi_positive_fraction=float(
+ resolved_loss_config.get("multi_positive_coverage_fraction", 0.0) or 0.0
+ ),
+ ),
+ )
+ total_loss = total_loss + float(resolved_loss_config["final_event_set_loss_weight"]) * final_event_set_loss
+ final_event_set_loss_values.append(float(final_event_set_loss.detach().cpu()))
+ final_event_set_loss_count += 1
+ event_matrix_delta_loss = _matrix_delta_direction_loss(
+ outputs.get("matrix_event_delta_logits", outputs["event_logits"].new_zeros((0,))),
+ matrix_ids=outputs.get("matrix_event_ids", []),
+ positive_ids=positive_event_ids,
+ hard_negative_ids=combined_hard_negative_event_ids,
+ margin=float(resolved_loss_config["event_matrix_delta_margin"]),
+ )
+ total_loss = total_loss + float(resolved_loss_config["event_matrix_delta_loss_weight"]) * event_matrix_delta_loss
+ event_matrix_delta_loss_values.append(float(event_matrix_delta_loss.detach().cpu()))
+ event_matrix_delta_loss_count += 1
+ scores = torch.sigmoid(outputs["event_logits"]).detach()
+ ranked_indices = torch.argsort(scores, descending=True)
+ ranked_event_ids = [outputs["candidate_event_ids"][int(index)] for index in ranked_indices.tolist()]
+ event_recall_total += 1
+ example_event_total = 1
+ if ranked_event_ids[:1] and any(item in positive_event_ids for item in ranked_event_ids[:1]):
+ event_recall1_hits += 1
+ example_event1_hit = 1
+ if ranked_event_ids[:5] and any(item in positive_event_ids for item in ranked_event_ids[:5]):
+ event_recall5_hits += 1
+ example_event5_hit = 1
+ example_event5_positive_hits = _positive_coverage_hit_count(
+ ranked_event_ids,
+ positive_event_ids,
+ limit=5,
+ )
+ example_event5_positive_total = min(len(positive_event_ids), 5)
+ event5_positive_hits += example_event5_positive_hits
+ event5_positive_total += example_event5_positive_total
+
+ if positive_path_ids and outputs["path_logits"].numel() > 0:
+ path_hard_negative_ids = _path_hard_negative_ids_from_supervision(
+ supervision,
+ candidate_path_ids=outputs["candidate_path_ids"],
+ candidate_path_event_ids=outputs.get("candidate_path_event_ids"),
+ candidate_path_types=outputs.get("candidate_path_types"),
+ )
+ online_path_hard_negative_ids = _online_hard_negative_path_ids(
+ outputs["path_logits"],
+ candidate_path_ids=outputs["candidate_path_ids"],
+ positive_path_ids=positive_path_ids,
+ candidate_path_event_ids=outputs.get("candidate_path_event_ids"),
+ positive_event_ids=positive_event_ids,
+ limit=int(resolved_loss_config.get("online_path_hard_negative_limit", 0) or 0),
+ )
+ combined_path_hard_negative_ids = dedupe_texts(
+ [*path_hard_negative_ids, *online_path_hard_negative_ids]
+ )
+ path_targets = _label_tensor_from_positive_ids(
+ outputs["candidate_path_ids"],
+ positive_path_ids,
+ device=device,
+ )
+ path_loss = _ranking_binary_loss(outputs["path_logits"], path_targets, margin=0.2)
+ total_loss = total_loss + float(resolved_loss_config["path_loss_weight"]) * path_loss
+ path_loss_values.append(float(path_loss.detach().cpu()))
+ path_loss_count += 1
+ path_tunnel_logits = outputs.get("path_tunnel_support_logits", outputs["path_logits"].new_zeros((0,)))
+ if isinstance(path_tunnel_logits, Tensor) and path_tunnel_logits.numel() > 0:
+ path_tunnel_targets = _label_tensor_from_positive_ids(
+ outputs["candidate_path_ids"],
+ positive_path_ids,
+ device=device,
+ )
+ path_tunnel_loss = _ranking_binary_loss(path_tunnel_logits, path_tunnel_targets, margin=0.16)
+ if combined_path_hard_negative_ids:
+ path_tunnel_loss = 0.5 * (
+ path_tunnel_loss
+ + _hard_negative_margin_loss(
+ path_tunnel_logits,
+ candidate_ids=outputs["candidate_path_ids"],
+ positive_ids=positive_path_ids,
+ hard_negative_ids=combined_path_hard_negative_ids,
+ margin=float(resolved_loss_config["path_tunnel_margin"]),
+ )
+ )
+ total_loss = total_loss + float(resolved_loss_config["path_tunnel_loss_weight"]) * path_tunnel_loss
+ path_tunnel_loss_values.append(float(path_tunnel_loss.detach().cpu()))
+ path_tunnel_loss_count += 1
+ path_tunnel_selection_loss = _selection_margin_loss(
+ path_tunnel_logits,
+ candidate_ids=outputs["candidate_path_ids"],
+ positive_ids=positive_path_ids,
+ margin=float(resolved_loss_config["path_tunnel_margin"]),
+ top_k=int(resolved_loss_config["path_selection_top_k"]),
+ required_positive_count=_selection_positive_coverage_count(
+ supervision,
+ positive_count=len(positive_path_ids),
+ top_k=int(resolved_loss_config["path_selection_top_k"]),
+ base_count=max(1, int(resolved_loss_config.get("path_selection_positive_coverage_count", 1) or 1)),
+ multi_positive_count=max(
+ 1,
+ int(resolved_loss_config.get("multi_positive_path_coverage_count", 1) or 1),
+ ),
+ multi_positive_fraction=float(
+ resolved_loss_config.get("multi_positive_coverage_fraction", 0.0) or 0.0
+ ),
+ ),
+ )
+ total_loss = total_loss + float(resolved_loss_config["path_tunnel_selection_loss_weight"]) * path_tunnel_selection_loss
+ path_tunnel_selection_loss_values.append(float(path_tunnel_selection_loss.detach().cpu()))
+ path_tunnel_selection_loss_count += 1
+ path_tunnel_delta_logits = outputs.get(
+ "path_tunnel_delta_logits",
+ outputs["path_logits"].new_zeros((0,)),
+ )
+ if (
+ isinstance(path_tunnel_delta_logits, Tensor)
+ and path_tunnel_delta_logits.numel() > 0
+ and float(resolved_loss_config.get("path_tunnel_delta_loss_weight", 0.0) or 0.0) > 0.0
+ ):
+ path_tunnel_delta_loss = _matrix_delta_direction_loss(
+ path_tunnel_delta_logits,
+ matrix_ids=outputs["candidate_path_ids"],
+ positive_ids=positive_path_ids,
+ hard_negative_ids=combined_path_hard_negative_ids,
+ margin=float(resolved_loss_config["path_tunnel_margin"]),
+ positive_weight=0.45,
+ hard_negative_weight=3.0,
+ pair_weight=1.0,
+ )
+ total_loss = total_loss + float(resolved_loss_config["path_tunnel_delta_loss_weight"]) * path_tunnel_delta_loss
+ path_tunnel_delta_loss_values.append(float(path_tunnel_delta_loss.detach().cpu()))
+ path_tunnel_delta_loss_count += 1
+ path_hard_negative_loss = _hard_negative_margin_loss(
+ outputs["path_logits"],
+ candidate_ids=outputs["candidate_path_ids"],
+ positive_ids=positive_path_ids,
+ hard_negative_ids=combined_path_hard_negative_ids,
+ margin=float(resolved_loss_config["path_hard_negative_margin"]),
+ )
+ total_loss = total_loss + float(resolved_loss_config["path_hard_negative_loss_weight"]) * path_hard_negative_loss
+ path_hard_negative_loss_values.append(float(path_hard_negative_loss.detach().cpu()))
+ path_hard_negative_loss_count += 1
+ path_selection_loss = _selection_margin_loss(
+ outputs["path_logits"],
+ candidate_ids=outputs["candidate_path_ids"],
+ positive_ids=positive_path_ids,
+ margin=float(resolved_loss_config["path_selection_margin"]),
+ top_k=int(resolved_loss_config["path_selection_top_k"]),
+ required_positive_count=_selection_positive_coverage_count(
+ supervision,
+ positive_count=len(positive_path_ids),
+ top_k=int(resolved_loss_config["path_selection_top_k"]),
+ base_count=max(1, int(resolved_loss_config.get("path_selection_positive_coverage_count", 1) or 1)),
+ multi_positive_count=max(
+ 1,
+ int(resolved_loss_config.get("multi_positive_path_coverage_count", 1) or 1),
+ ),
+ multi_positive_fraction=float(
+ resolved_loss_config.get("multi_positive_coverage_fraction", 0.0) or 0.0
+ ),
+ ),
+ )
+ total_loss = total_loss + float(resolved_loss_config["path_selection_loss_weight"]) * path_selection_loss
+ path_selection_loss_values.append(float(path_selection_loss.detach().cpu()))
+ path_selection_loss_count += 1
+ path_matrix_delta_loss = _matrix_delta_direction_loss(
+ outputs.get("matrix_path_delta_logits", outputs["path_logits"].new_zeros((0,))),
+ matrix_ids=outputs.get("matrix_path_ids", []),
+ positive_ids=positive_path_ids,
+ hard_negative_ids=combined_path_hard_negative_ids,
+ margin=float(resolved_loss_config["path_matrix_delta_margin"]),
+ )
+ total_loss = total_loss + float(resolved_loss_config["path_matrix_delta_loss_weight"]) * path_matrix_delta_loss
+ path_matrix_delta_loss_values.append(float(path_matrix_delta_loss.detach().cpu()))
+ path_matrix_delta_loss_count += 1
+ scores = torch.sigmoid(outputs["path_logits"]).detach()
+ ranked_indices = torch.argsort(scores, descending=True)
+ ranked_path_ids = [outputs["candidate_path_ids"][int(index)] for index in ranked_indices.tolist()]
+ path_recall_total += 1
+ example_path_total = 1
+ if ranked_path_ids[:3] and any(item in positive_path_ids for item in ranked_path_ids[:3]):
+ path_recall3_hits += 1
+ example_path3_hit = 1
+ example_path3_positive_hits = _positive_coverage_hit_count(
+ ranked_path_ids,
+ positive_path_ids,
+ limit=3,
+ )
+ example_path3_positive_total = min(len(positive_path_ids), 3)
+ path3_positive_hits += example_path3_positive_hits
+ path3_positive_total += example_path3_positive_total
+ path_tunnel_support_logits = outputs.get(
+ "path_tunnel_support_logits",
+ outputs["path_logits"].new_zeros((0,)),
+ )
+ if isinstance(path_tunnel_support_logits, Tensor) and path_tunnel_support_logits.numel() > 0:
+ support_scores = torch.sigmoid(path_tunnel_support_logits).detach()
+ support_ranked_indices = torch.argsort(support_scores, descending=True)
+ support_ranked_path_ids = [
+ outputs["candidate_path_ids"][int(index)]
+ for index in support_ranked_indices.tolist()
+ ]
+ path_tunnel_support_recall_total += 1
+ if support_ranked_path_ids[:3] and any(
+ item in positive_path_ids for item in support_ranked_path_ids[:3]
+ ):
+ path_tunnel_support_recall3_hits += 1
+ path_tunnel_support_positive_hits += _positive_coverage_hit_count(
+ support_ranked_path_ids,
+ positive_path_ids,
+ limit=3,
+ )
+ path_tunnel_support_positive_total += min(len(positive_path_ids), 3)
+ for alpha, hit_attr in (
+ (0.25, "025"),
+ (0.50, "050"),
+ (1.00, "100"),
+ ):
+ rescue_scores = scores + (support_scores * float(alpha))
+ rescue_ranked_indices = torch.argsort(rescue_scores, descending=True)
+ rescue_ranked_path_ids = [
+ outputs["candidate_path_ids"][int(index)]
+ for index in rescue_ranked_indices.tolist()
+ ]
+ if hit_attr == "025":
+ path_tunnel_rescue025_recall_total += 1
+ if rescue_ranked_path_ids[:3] and any(
+ item in positive_path_ids for item in rescue_ranked_path_ids[:3]
+ ):
+ path_tunnel_rescue025_recall3_hits += 1
+ elif hit_attr == "050":
+ path_tunnel_rescue050_recall_total += 1
+ if rescue_ranked_path_ids[:3] and any(
+ item in positive_path_ids for item in rescue_ranked_path_ids[:3]
+ ):
+ path_tunnel_rescue050_recall3_hits += 1
+ else:
+ path_tunnel_rescue100_recall_total += 1
+ if rescue_ranked_path_ids[:3] and any(
+ item in positive_path_ids for item in rescue_ranked_path_ids[:3]
+ ):
+ path_tunnel_rescue100_recall3_hits += 1
+ path_tunnel_delta_logits = outputs.get(
+ "path_tunnel_delta_logits",
+ outputs["path_logits"].new_zeros((0,)),
+ )
+ if isinstance(path_tunnel_delta_logits, Tensor) and path_tunnel_delta_logits.numel() > 0:
+ delta_scores = path_tunnel_delta_logits.detach()
+ delta_ranked_indices = torch.argsort(delta_scores, descending=True)
+ delta_ranked_path_ids = [
+ outputs["candidate_path_ids"][int(index)]
+ for index in delta_ranked_indices.tolist()
+ ]
+ path_tunnel_delta_recall_total += 1
+ if delta_ranked_path_ids[:3] and any(
+ item in positive_path_ids for item in delta_ranked_path_ids[:3]
+ ):
+ path_tunnel_delta_recall3_hits += 1
+ path_tunnel_delta_positive_hits += _positive_coverage_hit_count(
+ delta_ranked_path_ids,
+ positive_path_ids,
+ limit=3,
+ )
+ path_tunnel_delta_positive_total += min(len(positive_path_ids), 3)
+
+ if bool(supervision.get("is_temporal_example", _is_temporal_example(example))) and outputs["temporal_logits"].numel() > 0:
+ temporal_targets = _temporal_path_labels_from_supervision(
+ supervision,
+ outputs["candidate_temporal_path_ids"],
+ outputs["candidate_temporal_node_ids"],
+ candidate_temporal_event_ids=outputs.get("candidate_temporal_event_ids"),
+ device=device,
+ )
+ temporal_loss = _ranking_binary_loss(
+ outputs["temporal_logits"],
+ temporal_targets,
+ margin=0.15,
+ bce_weight=0.5,
+ positive_mass_weight=0.0,
+ pairwise_weight=0.5,
+ )
+ total_loss = total_loss + float(resolved_loss_config["temporal_loss_weight"]) * temporal_loss
+ temporal_loss_values.append(float(temporal_loss.detach().cpu()))
+ temporal_loss_count += 1
+ temporal_scores = torch.sigmoid(outputs["temporal_logits"]).detach()
+ temporal_total += 1
+ example_temporal_total = 1
+ if temporal_scores.numel() > 0:
+ top_index = int(torch.argmax(temporal_scores).item())
+ target_values = temporal_targets.detach().cpu().tolist()
+ if any(value > 0.5 for value in target_values):
+ if target_values[top_index] > 0.5:
+ temporal_hits += 1
+ example_temporal_hit = 1
+ elif float(torch.max(temporal_scores).item()) < 0.5:
+ temporal_hits += 1
+ example_temporal_hit = 1
+
+ if apply_example_weights and example_training_weight > 0.0 and abs(example_training_weight - 1.0) > 1e-9:
+ total_loss = total_loss * outputs["answer_type_logits"].new_tensor(float(example_training_weight))
+ for stats in (answer_stats, supervision_stats):
+ stats["recall_event_recall_total"] += example_recall_total
+ stats["recall_event24_hits"] += example_recall24_hit
+ stats["recall_event24_positive_hits"] += example_recall24_positive_hits
+ stats["recall_event24_positive_total"] += example_recall24_positive_total
+ stats["event_recall_total"] += example_event_total
+ stats["event_recall1_hits"] += example_event1_hit
+ stats["event_recall5_hits"] += example_event5_hit
+ stats["event5_positive_hits"] += example_event5_positive_hits
+ stats["event5_positive_total"] += example_event5_positive_total
+ stats["path_recall_total"] += example_path_total
+ stats["path_recall3_hits"] += example_path3_hit
+ stats["path3_positive_hits"] += example_path3_positive_hits
+ stats["path3_positive_total"] += example_path3_positive_total
+ stats["answer_plan_selected_total"] += example_answer_plan_total
+ stats["answer_plan_selected_recall5_hits"] += example_answer_plan_recall5_hit
+ stats["answer_plan_selected_positive_hits"] += example_answer_plan_positive_hits
+ stats["answer_plan_selected_positive_total"] += example_answer_plan_positive_total
+ stats["answer_plan_current_total"] += example_answer_plan_current_total
+ stats["answer_plan_current_top1_hits"] += example_answer_plan_current_top1_hit
+ stats["temporal_total"] += example_temporal_total
+ stats["temporal_hits"] += example_temporal_hit
+ losses.append(total_loss)
+ loss_entries.append(
+ (
+ total_loss,
+ _loss_group_key_for_example(
+ example,
+ supervision,
+ mode=clean_text(resolved_loss_config.get("loss_group_balancing_mode", "answer_type")),
+ ),
+ )
+ )
+
+ if not losses:
+ return _empty_score_batch_result(device=device, graph_error_count=graph_error_count)
+ batch_loss, loss_group_count = _group_balanced_batch_loss(
+ loss_entries,
+ mode=clean_text(resolved_loss_config.get("loss_group_balancing_mode", "answer_type")),
+ )
+ return {
+ "loss": batch_loss,
+ "metrics": {
+ "loss": float(batch_loss.detach().cpu()),
+ "recall_loss": round(sum(recall_loss_values) / max(1, len(recall_loss_values)), 6),
+ "event_loss": round(sum(event_loss_values) / max(1, len(event_loss_values)), 6),
+ "path_loss": round(sum(path_loss_values) / max(1, len(path_loss_values)), 6),
+ "temporal_loss": round(sum(temporal_loss_values) / max(1, len(temporal_loss_values)), 6),
+ "answer_type_loss": round(sum(answer_type_loss_values) / max(1, len(answer_type_loss_values)), 6),
+ "answer_plan_loss": round(sum(answer_plan_loss_values) / max(1, len(answer_plan_loss_values)), 6),
+ "token_role_loss": round(sum(token_role_loss_values) / max(1, len(token_role_loss_values)), 6),
+ "question_understanding_loss": round(
+ sum(question_understanding_loss_values) / max(1, len(question_understanding_loss_values)),
+ 6,
+ ),
+ "memory_router_loss": round(sum(memory_router_loss_values) / max(1, len(memory_router_loss_values)), 6),
+ "memory_router_exact_match": round(memory_router_exact_hits / max(1, memory_router_total), 6),
+ "memory_router_f1": round(memory_router_f1_sum / max(1, memory_router_total), 6),
+ "event_distractor_loss": round(sum(event_distractor_loss_values) / max(1, len(event_distractor_loss_values)), 6),
+ "event_tunnel_loss": round(sum(event_tunnel_loss_values) / max(1, len(event_tunnel_loss_values)), 6),
+ "path_tunnel_loss": round(sum(path_tunnel_loss_values) / max(1, len(path_tunnel_loss_values)), 6),
+ "path_tunnel_delta_loss": round(
+ sum(path_tunnel_delta_loss_values) / max(1, len(path_tunnel_delta_loss_values)),
+ 6,
+ ),
+ "event_tunnel_selection_loss": round(
+ sum(event_tunnel_selection_loss_values) / max(1, len(event_tunnel_selection_loss_values)),
+ 6,
+ ),
+ "path_tunnel_selection_loss": round(
+ sum(path_tunnel_selection_loss_values) / max(1, len(path_tunnel_selection_loss_values)),
+ 6,
+ ),
+ "event_hard_negative_loss": round(sum(event_hard_negative_loss_values) / max(1, len(event_hard_negative_loss_values)), 6),
+ "path_hard_negative_loss": round(sum(path_hard_negative_loss_values) / max(1, len(path_hard_negative_loss_values)), 6),
+ "recall_selection_loss": round(sum(recall_selection_loss_values) / max(1, len(recall_selection_loss_values)), 6),
+ "event_selection_loss": round(sum(event_selection_loss_values) / max(1, len(event_selection_loss_values)), 6),
+ "path_selection_loss": round(sum(path_selection_loss_values) / max(1, len(path_selection_loss_values)), 6),
+ "final_event_set_loss": round(sum(final_event_set_loss_values) / max(1, len(final_event_set_loss_values)), 6),
+ "event_matrix_delta_loss": round(sum(event_matrix_delta_loss_values) / max(1, len(event_matrix_delta_loss_values)), 6),
+ "path_matrix_delta_loss": round(sum(path_matrix_delta_loss_values) / max(1, len(path_matrix_delta_loss_values)), 6),
+ "answer_refusal_loss": round(sum(answer_refusal_loss_values) / max(1, len(answer_refusal_loss_values)), 6),
+ "recall_event_recall_at_24": round(recall_event24_hits / max(1, recall_event_total), 6),
+ "recall_event_positive_coverage_at_24": round(
+ recall_event24_positive_hits / max(1, recall_event24_positive_total),
+ 6,
+ ),
+ "event_recall_at_1": round(event_recall1_hits / max(1, event_recall_total), 6),
+ "event_recall_at_5": round(event_recall5_hits / max(1, event_recall_total), 6),
+ "event_positive_coverage_at_5": round(event5_positive_hits / max(1, event5_positive_total), 6),
+ "path_recall_at_3": round(path_recall3_hits / max(1, path_recall_total), 6),
+ "path_positive_coverage_at_3": round(path3_positive_hits / max(1, path3_positive_total), 6),
+ "path_tunnel_support_recall_at_3": round(
+ path_tunnel_support_recall3_hits / max(1, path_tunnel_support_recall_total),
+ 6,
+ ),
+ "path_tunnel_support_positive_coverage_at_3": round(
+ path_tunnel_support_positive_hits / max(1, path_tunnel_support_positive_total),
+ 6,
+ ),
+ "path_tunnel_delta_recall_at_3": round(
+ path_tunnel_delta_recall3_hits / max(1, path_tunnel_delta_recall_total),
+ 6,
+ ),
+ "path_tunnel_delta_positive_coverage_at_3": round(
+ path_tunnel_delta_positive_hits / max(1, path_tunnel_delta_positive_total),
+ 6,
+ ),
+ "answer_plan_selected_recall_at_5": round(
+ answer_plan_selected_recall5_hits / max(1, answer_plan_selected_total),
+ 6,
+ ),
+ "answer_plan_selected_positive_coverage_at_5": round(
+ answer_plan_selected_positive_hits / max(1, answer_plan_selected_positive_total),
+ 6,
+ ),
+ "answer_plan_current_top1_accuracy": round(
+ answer_plan_current_top1_hits / max(1, answer_plan_current_total),
+ 6,
+ ),
+ "path_tunnel_rescue025_recall_at_3": round(
+ path_tunnel_rescue025_recall3_hits / max(1, path_tunnel_rescue025_recall_total),
+ 6,
+ ),
+ "path_tunnel_rescue050_recall_at_3": round(
+ path_tunnel_rescue050_recall3_hits / max(1, path_tunnel_rescue050_recall_total),
+ 6,
+ ),
+ "path_tunnel_rescue100_recall_at_3": round(
+ path_tunnel_rescue100_recall3_hits / max(1, path_tunnel_rescue100_recall_total),
+ 6,
+ ),
+ "temporal_accuracy": round(temporal_hits / max(1, temporal_total), 6),
+ "samples": len(losses),
+ "loss_group_count": int(loss_group_count),
+ "recall_event24_positive_total": int(recall_event24_positive_total),
+ "event5_positive_total": int(event5_positive_total),
+ "path3_positive_total": int(path3_positive_total),
+ "path_tunnel_support_recall_total": int(path_tunnel_support_recall_total),
+ "path_tunnel_support_positive_total": int(path_tunnel_support_positive_total),
+ "path_tunnel_delta_recall_total": int(path_tunnel_delta_recall_total),
+ "path_tunnel_delta_positive_total": int(path_tunnel_delta_positive_total),
+ "answer_plan_selected_total": int(answer_plan_selected_total),
+ "answer_plan_selected_positive_total": int(answer_plan_selected_positive_total),
+ "answer_plan_current_total": int(answer_plan_current_total),
+ "path_tunnel_rescue025_recall_total": int(path_tunnel_rescue025_recall_total),
+ "path_tunnel_rescue050_recall_total": int(path_tunnel_rescue050_recall_total),
+ "path_tunnel_rescue100_recall_total": int(path_tunnel_rescue100_recall_total),
+ "answer_type_metrics": _finalize_answer_type_metric_stats(answer_type_metric_stats),
+ "supervision_bucket_metrics": _finalize_answer_type_metric_stats(supervision_bucket_metric_stats),
+ "training_weight_mean": round(sum(training_weight_values) / max(1, len(training_weight_values)), 6),
+ "training_weight_min": round(min(training_weight_values) if training_weight_values else 1.0, 6),
+ "training_weight_max": round(max(training_weight_values) if training_weight_values else 1.0, 6),
+ "recall_loss_count": recall_loss_count,
+ "event_loss_count": event_loss_count,
+ "path_loss_count": path_loss_count,
+ "temporal_loss_count": temporal_loss_count,
+ "answer_type_loss_count": answer_type_loss_count,
+ "answer_plan_loss_count": answer_plan_loss_count,
+ "token_role_loss_count": token_role_loss_count,
+ "question_understanding_loss_count": question_understanding_loss_count,
+ "memory_router_loss_count": memory_router_loss_count,
+ "memory_router_total": memory_router_total,
+ "event_distractor_loss_count": event_distractor_loss_count,
+ "event_tunnel_loss_count": event_tunnel_loss_count,
+ "path_tunnel_loss_count": path_tunnel_loss_count,
+ "path_tunnel_delta_loss_count": path_tunnel_delta_loss_count,
+ "event_tunnel_selection_loss_count": event_tunnel_selection_loss_count,
+ "path_tunnel_selection_loss_count": path_tunnel_selection_loss_count,
+ "event_hard_negative_loss_count": event_hard_negative_loss_count,
+ "path_hard_negative_loss_count": path_hard_negative_loss_count,
+ "recall_selection_loss_count": recall_selection_loss_count,
+ "event_selection_loss_count": event_selection_loss_count,
+ "path_selection_loss_count": path_selection_loss_count,
+ "final_event_set_loss_count": final_event_set_loss_count,
+ "event_matrix_delta_loss_count": event_matrix_delta_loss_count,
+ "path_matrix_delta_loss_count": path_matrix_delta_loss_count,
+ "answer_refusal_loss_count": answer_refusal_loss_count,
+ "recall_event_recall_total": recall_event_total,
+ "event_recall_total": event_recall_total,
+ "path_recall_total": path_recall_total,
+ "temporal_total": temporal_total,
+ "graph_error_count": graph_error_count,
+ },
+ }
+
+
+def score_batch(
+ model: LocomoNodeMemoryModel,
+ batch: Sequence[QueryTrainingExample],
+ graph_cache: Mapping[str, GraphCacheItem],
+ *,
+ device: torch.device,
+ loss_config: Mapping[str, Any] | None = None,
+ score_kwargs: Mapping[str, Any] | None = None,
+ apply_example_weights: bool = False,
+ graph_error_callback: Callable[[Dict[str, Any]], None] | None = None,
+ graph_error_stage: str = "",
+) -> Dict[str, Any]:
+ prepared_batch = _prepare_batch_payload_from_graph_cache(
+ batch,
+ graph_cache,
+ merge_device=device,
+ graph_error_stage=graph_error_stage,
+ )
+ return score_prepared_batch(
+ model,
+ prepared_batch,
+ device=device,
+ loss_config=loss_config,
+ score_kwargs=score_kwargs,
+ apply_example_weights=apply_example_weights,
+ graph_error_callback=graph_error_callback,
+ graph_error_stage=graph_error_stage,
+ )
+
+
+def iter_batches(
+ rows: Sequence[QueryTrainingExample],
+ *,
+ batch_size: int,
+ shuffle: bool,
+ seed: int,
+) -> List[List[QueryTrainingExample]]:
+ resolved_batch_size = max(1, int(batch_size))
+ if any(not hasattr(row, "conversation_id") for row in rows):
+ working = list(rows)
+ if shuffle:
+ random.Random(seed).shuffle(working)
+ return [
+ working[index : index + resolved_batch_size]
+ for index in range(0, len(working), resolved_batch_size)
+ ]
+ grouped: "OrderedDict[str, List[QueryTrainingExample]]" = OrderedDict()
+ for row in rows:
+ grouped.setdefault(row.conversation_id, []).append(row)
+ conversation_ids = list(grouped)
+ rng = random.Random(seed)
+ if shuffle:
+ rng.shuffle(conversation_ids)
+ batches: List[List[QueryTrainingExample]] = []
+ current_batch: List[QueryTrainingExample] = []
+ for conversation_id in conversation_ids:
+ group_rows = list(grouped[conversation_id])
+ if shuffle:
+ rng.shuffle(group_rows)
+ if len(group_rows) > resolved_batch_size:
+ if current_batch:
+ batches.append(current_batch)
+ current_batch = []
+ for index in range(0, len(group_rows), resolved_batch_size):
+ batches.append(group_rows[index : index + resolved_batch_size])
+ continue
+ if current_batch and len(current_batch) + len(group_rows) > resolved_batch_size:
+ batches.append(current_batch)
+ current_batch = []
+ current_batch.extend(group_rows)
+ if current_batch:
+ batches.append(current_batch)
+ return batches
+
+
+def _iter_batches_any(
+ rows: Sequence[QueryTrainingExample],
+ *,
+ batch_size: int,
+ shuffle: bool,
+ seed: int,
+ skip_batches: int = 0,
+) -> Iterator[QueryBatchSpec | Sequence[QueryTrainingExample]]:
+ custom_iter_batches = getattr(rows, "iter_batches", None)
+ custom_iter_batch_specs = getattr(rows, "iter_batch_specs", None)
+ if callable(custom_iter_batch_specs):
+ batch_iter = iter(custom_iter_batch_specs(batch_size=batch_size, shuffle=shuffle, seed=seed))
+ elif callable(custom_iter_batches):
+ batch_iter = iter(custom_iter_batches(batch_size=batch_size, shuffle=shuffle, seed=seed))
+ else:
+ batch_iter = iter(iter_batches(rows, batch_size=batch_size, shuffle=shuffle, seed=seed))
+ resolved_skip_batches = max(0, int(skip_batches or 0))
+ for _ in range(resolved_skip_batches):
+ try:
+ next(batch_iter)
+ except StopIteration:
+ return
+ yield from batch_iter
+
+
+def _prefetch_graph_cache_for_pending_batches(
+ graph_cache: Mapping[str, GraphCacheItem],
+ pending_batches: Sequence[QueryBatchSpec | Sequence[QueryTrainingExample]],
+) -> None:
+ prefetch = getattr(graph_cache, "prefetch", None)
+ if not callable(prefetch):
+ return
+ conversation_ids: List[str] = []
+ seen: set[str] = set()
+ for batch in pending_batches:
+ for example in _materialize_query_batch_payload(batch):
+ conversation_id = clean_text(example.conversation_id)
+ if not conversation_id or conversation_id in seen:
+ continue
+ seen.add(conversation_id)
+ conversation_ids.append(conversation_id)
+ if conversation_ids:
+ prefetch(conversation_ids)
+
+
+def _iter_batches_with_prefetch(
+ graph_cache: Mapping[str, GraphCacheItem],
+ rows: Sequence[QueryTrainingExample],
+ *,
+ batch_size: int,
+ shuffle: bool,
+ seed: int,
+ lookahead_batches: int,
+ skip_batches: int = 0,
+) -> Iterator[Sequence[QueryTrainingExample]]:
+ batch_iter = iter(
+ _iter_batches_any(
+ rows,
+ batch_size=batch_size,
+ shuffle=shuffle,
+ seed=seed,
+ skip_batches=skip_batches,
+ )
+ )
+ pending_batches: deque[Sequence[QueryTrainingExample]] = deque()
+ pending_target = max(1, max(0, int(lookahead_batches or 0)) + 1)
+
+ def _fill_pending() -> None:
+ while len(pending_batches) < pending_target:
+ try:
+ pending_batches.append(next(batch_iter))
+ except StopIteration:
+ break
+
+ _fill_pending()
+ while pending_batches:
+ _prefetch_graph_cache_for_pending_batches(graph_cache, list(pending_batches))
+ yield pending_batches.popleft()
+ _fill_pending()
+
+
+def _prefetch_graph_cache_for_batches(
+ graph_cache: Mapping[str, GraphCacheItem],
+ batches: Sequence[QueryBatchSpec | Sequence[QueryTrainingExample]],
+ *,
+ batch_index: int,
+ lookahead_batches: int,
+) -> None:
+ prefetch = getattr(graph_cache, "prefetch", None)
+ if not callable(prefetch):
+ return
+ resolved_lookahead = max(0, int(lookahead_batches or 0))
+ if resolved_lookahead <= 0:
+ return
+ conversation_ids: List[str] = []
+ seen: set[str] = set()
+ end_index = min(len(batches), int(batch_index) + resolved_lookahead + 1)
+ for future_batch in batches[int(batch_index) : end_index]:
+ for example in _materialize_query_batch_payload(future_batch):
+ conversation_id = clean_text(example.conversation_id)
+ if not conversation_id or conversation_id in seen:
+ continue
+ seen.add(conversation_id)
+ conversation_ids.append(conversation_id)
+ if conversation_ids:
+ prefetch(conversation_ids)
+
+
+def _select_eval_rows(
+ rows: Sequence[QueryTrainingExample],
+ *,
+ max_rows: int = 0,
+ seed: int = 0,
+) -> tuple[Sequence[QueryTrainingExample], int]:
+ total_rows = len(rows)
+ resolved_max_rows = max(0, int(max_rows or 0))
+ if resolved_max_rows <= 0 or total_rows <= resolved_max_rows:
+ return rows, total_rows
+ rng = random.Random(int(seed))
+ selected_indices = sorted(rng.sample(range(total_rows), resolved_max_rows))
+ return [rows[index] for index in selected_indices], total_rows
+
+
+def evaluate_examples(
+ model: LocomoNodeMemoryModel,
+ rows: Sequence[QueryTrainingExample],
+ graph_cache: Mapping[str, GraphCacheItem],
+ *,
+ device: torch.device,
+ batch_size: int,
+ loss_config: Mapping[str, Any] | None = None,
+ score_kwargs: Mapping[str, Any] | None = None,
+ prefetch_lookahead_batches: int = 0,
+ graph_error_callback: Callable[[Dict[str, Any]], None] | None = None,
+ worker_event_callback: Callable[[Dict[str, Any]], None] | None = None,
+ graph_error_stage: str = "",
+ max_rows: int = 0,
+ sample_seed: int = 0,
+ batch_prepare_context: Mapping[str, Any] | None = None,
+ batch_prepare_workers: int = 0,
+ batch_prepare_lookahead_batches: int = 0,
+ batch_prepare_worker_pool: _PreparedBatchWorkerPool | None = None,
+) -> Dict[str, Any]:
+ model.eval()
+ metrics_list: List[Dict[str, Any]] = []
+ eval_rows, total_row_count = _select_eval_rows(rows, max_rows=max_rows, seed=sample_seed)
+ evaluated_row_count = len(eval_rows)
+ evaluation_limited = evaluated_row_count < total_row_count
+ with torch.no_grad():
+ if batch_prepare_worker_pool is not None or (
+ max(0, int(batch_prepare_workers or 0)) > 0 and isinstance(batch_prepare_context, Mapping)
+ ):
+ for prepared_batch in _iter_prepared_batches_with_workers(
+ eval_rows,
+ graph_cache=graph_cache,
+ batch_size=batch_size,
+ shuffle=False,
+ seed=int(sample_seed),
+ worker_context=batch_prepare_context,
+ worker_count=max(0, int(batch_prepare_workers or 0)),
+ lookahead_batches=max(0, int(batch_prepare_lookahead_batches or 0)),
+ graph_error_stage=f"{clean_text(graph_error_stage) or 'eval'}_batch_prepare",
+ worker_pool=batch_prepare_worker_pool,
+ prefer_completion_order=False,
+ worker_event_callback=worker_event_callback,
+ ):
+ metrics_list.append(
+ score_prepared_batch(
+ model,
+ prepared_batch,
+ device=device,
+ loss_config=loss_config,
+ score_kwargs=score_kwargs,
+ graph_error_callback=graph_error_callback,
+ graph_error_stage=graph_error_stage,
+ )["metrics"]
+ )
+ else:
+ for batch in _iter_batches_with_prefetch(
+ graph_cache,
+ eval_rows,
+ batch_size=batch_size,
+ shuffle=False,
+ seed=int(sample_seed),
+ lookahead_batches=prefetch_lookahead_batches,
+ ):
+ metrics_list.append(
+ score_batch(
+ model,
+ batch,
+ graph_cache,
+ device=device,
+ loss_config=loss_config,
+ score_kwargs=score_kwargs,
+ graph_error_callback=graph_error_callback,
+ graph_error_stage=graph_error_stage,
+ )["metrics"]
+ )
+ if not metrics_list:
+ return {
+ "loss": 0.0,
+ "recall_loss": 0.0,
+ "event_loss": 0.0,
+ "path_loss": 0.0,
+ "temporal_loss": 0.0,
+ "answer_type_loss": 0.0,
+ "answer_plan_loss": 0.0,
+ "token_role_loss": 0.0,
+ "memory_router_loss": 0.0,
+ "memory_router_exact_match": 0.0,
+ "memory_router_f1": 0.0,
+ "event_distractor_loss": 0.0,
+ "event_tunnel_loss": 0.0,
+ "path_tunnel_loss": 0.0,
+ "path_tunnel_delta_loss": 0.0,
+ "event_tunnel_selection_loss": 0.0,
+ "path_tunnel_selection_loss": 0.0,
+ "event_hard_negative_loss": 0.0,
+ "path_hard_negative_loss": 0.0,
+ "recall_selection_loss": 0.0,
+ "event_selection_loss": 0.0,
+ "path_selection_loss": 0.0,
+ "final_event_set_loss": 0.0,
+ "event_matrix_delta_loss": 0.0,
+ "path_matrix_delta_loss": 0.0,
+ "answer_refusal_loss": 0.0,
+ "recall_event_recall_at_24": 0.0,
+ "event_recall_at_1": 0.0,
+ "event_recall_at_5": 0.0,
+ "path_recall_at_3": 0.0,
+ "answer_plan_selected_recall_at_5": 0.0,
+ "answer_plan_selected_positive_coverage_at_5": 0.0,
+ "answer_plan_current_top1_accuracy": 0.0,
+ "temporal_accuracy": 0.0,
+ "training_weight_mean": 1.0,
+ "training_weight_min": 1.0,
+ "training_weight_max": 1.0,
+ "samples": 0,
+ "loss_group_count": 0,
+ "answer_type_metrics": {},
+ "supervision_bucket_metrics": {},
+ "recall_loss_count": 0,
+ "event_loss_count": 0,
+ "path_loss_count": 0,
+ "temporal_loss_count": 0,
+ "answer_type_loss_count": 0,
+ "answer_plan_loss_count": 0,
+ "token_role_loss_count": 0,
+ "memory_router_loss_count": 0,
+ "memory_router_total": 0,
+ "event_distractor_loss_count": 0,
+ "event_tunnel_loss_count": 0,
+ "path_tunnel_loss_count": 0,
+ "path_tunnel_delta_loss_count": 0,
+ "event_tunnel_selection_loss_count": 0,
+ "path_tunnel_selection_loss_count": 0,
+ "event_hard_negative_loss_count": 0,
+ "path_hard_negative_loss_count": 0,
+ "recall_selection_loss_count": 0,
+ "event_selection_loss_count": 0,
+ "path_selection_loss_count": 0,
+ "final_event_set_loss_count": 0,
+ "event_matrix_delta_loss_count": 0,
+ "path_matrix_delta_loss_count": 0,
+ "answer_refusal_loss_count": 0,
+ "recall_event_recall_total": 0,
+ "event_recall_total": 0,
+ "path_recall_total": 0,
+ "answer_plan_selected_total": 0,
+ "answer_plan_selected_positive_total": 0,
+ "answer_plan_current_total": 0,
+ "temporal_total": 0,
+ "graph_error_count": 0,
+ "rows_total": int(total_row_count),
+ "rows_evaluated": int(evaluated_row_count),
+ "evaluation_limited": bool(evaluation_limited),
+ }
+ summary: Dict[str, Any] = {}
+ for key in metrics_list[0].keys():
+ if isinstance(metrics_list[0].get(key), Mapping):
+ summary[key] = _merge_nested_metric_summaries(metrics_list, key) # type: ignore[assignment]
+ continue
+ if key in _METRIC_COUNT_KEYS:
+ summary[key] = int(sum(int(item.get(key, 0) or 0) for item in metrics_list))
+ continue
+ weight_key = _METRIC_WEIGHT_KEYS.get(key, "samples")
+ total_weight = float(sum(item.get(weight_key, 0.0) or 0.0 for item in metrics_list))
+ if total_weight <= 0.0:
+ summary[key] = 0.0
+ continue
+ weighted_total = sum(
+ float(item.get(key, 0.0) or 0.0) * float(item.get(weight_key, 0.0) or 0.0)
+ for item in metrics_list
+ )
+ summary[key] = round(weighted_total / total_weight, 6)
+ summary["rows_total"] = int(total_row_count)
+ summary["rows_evaluated"] = int(evaluated_row_count)
+ summary["evaluation_limited"] = bool(evaluation_limited)
+ return summary
+
+
+def _skipped_eval_summary(sample_count: int) -> Dict[str, Any]:
+ return {
+ "loss": 0.0,
+ "recall_loss": 0.0,
+ "event_loss": 0.0,
+ "path_loss": 0.0,
+ "temporal_loss": 0.0,
+ "answer_type_loss": 0.0,
+ "answer_plan_loss": 0.0,
+ "token_role_loss": 0.0,
+ "memory_router_loss": 0.0,
+ "memory_router_exact_match": 0.0,
+ "memory_router_f1": 0.0,
+ "event_distractor_loss": 0.0,
+ "event_tunnel_loss": 0.0,
+ "path_tunnel_loss": 0.0,
+ "path_tunnel_delta_loss": 0.0,
+ "event_tunnel_selection_loss": 0.0,
+ "path_tunnel_selection_loss": 0.0,
+ "event_hard_negative_loss": 0.0,
+ "path_hard_negative_loss": 0.0,
+ "recall_selection_loss": 0.0,
+ "event_selection_loss": 0.0,
+ "path_selection_loss": 0.0,
+ "final_event_set_loss": 0.0,
+ "event_matrix_delta_loss": 0.0,
+ "path_matrix_delta_loss": 0.0,
+ "answer_refusal_loss": 0.0,
+ "recall_event_recall_at_24": 0.0,
+ "event_recall_at_1": 0.0,
+ "event_recall_at_5": 0.0,
+ "path_recall_at_3": 0.0,
+ "answer_plan_selected_recall_at_5": 0.0,
+ "answer_plan_selected_positive_coverage_at_5": 0.0,
+ "answer_plan_current_top1_accuracy": 0.0,
+ "temporal_accuracy": 0.0,
+ "training_weight_mean": 1.0,
+ "training_weight_min": 1.0,
+ "training_weight_max": 1.0,
+ "samples": int(sample_count),
+ "loss_group_count": 0,
+ "answer_type_metrics": {},
+ "supervision_bucket_metrics": {},
+ "final_event_set_loss_count": 0,
+ "path_tunnel_delta_loss_count": 0,
+ "answer_plan_loss_count": 0,
+ "memory_router_loss_count": 0,
+ "memory_router_total": 0,
+ "answer_plan_selected_total": 0,
+ "answer_plan_selected_positive_total": 0,
+ "answer_plan_current_total": 0,
+ "graph_error_count": 0,
+ "skipped": True,
+ }
+
+
+def _answer_type_metric_floor(
+ summary: Mapping[str, Any],
+ metric_key: str,
+ total_key: str,
+ *,
+ fallback: float,
+ min_total: int = 3,
+ nested_key: str = "answer_type_metrics",
+) -> float:
+ answer_type_metrics = summary.get(nested_key, {})
+ if not isinstance(answer_type_metrics, Mapping):
+ return float(fallback)
+ values: List[float] = []
+ for metrics in answer_type_metrics.values():
+ if not isinstance(metrics, Mapping):
+ continue
+ if int(metrics.get(total_key, 0) or 0) < int(min_total):
+ continue
+ values.append(float(metrics.get(metric_key, 0.0) or 0.0))
+ if not values:
+ return float(fallback)
+ return min(values)
+
+
+def _checkpoint_selection_key(summary: Mapping[str, Any]) -> tuple[float, float, float, float, float]:
+ event_recall_at_5 = float(summary.get("event_recall_at_5", 0.0) or 0.0)
+ path_recall_at_3 = float(summary.get("path_recall_at_3", 0.0) or 0.0)
+ event_type_floor = _answer_type_metric_floor(
+ summary,
+ "event_recall_at_5",
+ "event_recall_total",
+ fallback=event_recall_at_5,
+ )
+ path_type_floor = _answer_type_metric_floor(
+ summary,
+ "path_recall_at_3",
+ "path_recall_total",
+ fallback=path_recall_at_3,
+ )
+ event_supervision_floor = _answer_type_metric_floor(
+ summary,
+ "event_recall_at_5",
+ "event_recall_total",
+ fallback=event_recall_at_5,
+ nested_key="supervision_bucket_metrics",
+ )
+ path_supervision_floor = _answer_type_metric_floor(
+ summary,
+ "path_recall_at_3",
+ "path_recall_total",
+ fallback=path_recall_at_3,
+ nested_key="supervision_bucket_metrics",
+ )
+ return (
+ float(min(event_type_floor, event_supervision_floor)),
+ float(summary.get("recall_event_recall_at_24", 0.0) or 0.0),
+ float(min(path_type_floor, path_supervision_floor)),
+ float(summary.get("temporal_accuracy", 0.0) or 0.0),
+ -float(summary.get("loss", 0.0) or 0.0),
+ )
+
+
+def _checkpoint_selection_score(summary: Mapping[str, Any]) -> float:
+ selection_key = _checkpoint_selection_key(summary)
+ return round(
+ selection_key[0] * 100.0
+ + selection_key[1] * 10.0
+ + selection_key[2] * 10.0
+ + selection_key[3] * 5.0
+ + selection_key[4],
+ 6,
+ )
+
+
+def train_locomo_node_memory(
+ *,
+ train_rows: Sequence[QueryTrainingExample],
+ val_rows: Sequence[QueryTrainingExample],
+ train_eval_rows: Sequence[QueryTrainingExample] | None = None,
+ graph_cache: Mapping[str, GraphCacheItem],
+ device: torch.device,
+ config: Mapping[str, Any] | None = None,
+ initial_model_state: Mapping[str, Any] | None = None,
+ resume_training_state: Mapping[str, Any] | None = None,
+ epoch_train_eval: bool = True,
+ epoch_val_eval: bool = True,
+ epoch_callback: Callable[[Dict[str, Any]], None] | None = None,
+ step_callback: Callable[[Dict[str, Any]], None] | None = None,
+ checkpoint_callback: Callable[[Dict[str, Any], LocomoNodeMemoryModel, Any, Any, Any], None] | None = None,
+ step_checkpoint_callback: Callable[[Dict[str, Any], LocomoNodeMemoryModel, Any, Any, Any], None] | None = None,
+ graph_error_callback: Callable[[Dict[str, Any]], None] | None = None,
+ worker_event_callback: Callable[[Dict[str, Any]], None] | None = None,
+) -> Dict[str, Any]:
+ resolved_config = {**DEFAULT_TRAINING_CONFIG, **dict(config or {})}
+ resolved_train_eval_rows = train_eval_rows if train_eval_rows is not None else train_rows
+ model = LocomoNodeMemoryModel().to(device)
+ if initial_model_state:
+ _load_compatible_state_dict(model, initial_model_state)
+ trainable_stage_summary = _apply_trainable_stage(
+ model,
+ clean_text(resolved_config.get("trainable_stage", "all")) or "all",
+ )
+ trainable_parameters = [parameter for parameter in model.parameters() if parameter.requires_grad]
+ if not trainable_parameters:
+ raise RuntimeError("No trainable parameters resolved for node-memory training")
+ l2sp_loss_weight = max(0.0, float(resolved_config.get("l2sp_loss_weight", 0.0) or 0.0))
+ l2sp_reference_state: Dict[str, Tensor] = {}
+ if l2sp_loss_weight > 0.0 and initial_model_state:
+ for name, parameter in model.named_parameters():
+ if not parameter.requires_grad:
+ continue
+ reference_value = initial_model_state.get(name)
+ if not isinstance(reference_value, Tensor):
+ continue
+ if tuple(reference_value.shape) != tuple(parameter.shape):
+ continue
+ l2sp_reference_state[name] = reference_value.detach().to(device=device, dtype=parameter.dtype).clone()
+ optimizer = torch.optim.AdamW(
+ trainable_parameters,
+ lr=float(resolved_config["lr"]),
+ weight_decay=float(resolved_config["weight_decay"]),
+ )
+ planned_total_steps = max(1, math.ceil(len(train_rows) / max(1, int(resolved_config["batch_size"]))) * max(1, int(resolved_config["epochs"])))
+ configured_max_train_steps = max(0, int(resolved_config.get("max_train_steps", 0) or 0))
+ total_steps = min(planned_total_steps, configured_max_train_steps) if configured_max_train_steps > 0 else planned_total_steps
+ warmup_steps = max(1, int(total_steps * float(resolved_config["warmup_ratio"])))
+
+ def lr_lambda(step: int) -> float:
+ if step < warmup_steps:
+ return float(step + 1) / float(warmup_steps)
+ remaining = max(1, total_steps - warmup_steps)
+ return max(0.05, float(total_steps - step) / float(remaining))
+
+ scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda=lr_lambda)
+ amp_enabled = bool(resolved_config["amp"]) and device.type in {"cuda", "cpu"}
+ scaler = _build_grad_scaler(enabled=amp_enabled and device.type == "cuda", device=device)
+ log_every_steps = max(1, int(resolved_config.get("log_every_steps", 100) or 100))
+ checkpoint_every_steps = max(0, int(resolved_config.get("checkpoint_every_steps", 0) or 0))
+ prefetch_lookahead_batches = max(0, int(resolved_config.get("graph_prefetch_lookahead_batches", 0) or 0))
+ batch_prepare_workers = max(0, int(resolved_config.get("batch_prepare_workers", 0) or 0))
+ batch_prepare_lookahead_batches = max(0, int(resolved_config.get("batch_prepare_lookahead_batches", 0) or 0))
+ epoch_train_eval_max_rows = max(0, int(resolved_config.get("epoch_train_eval_max_rows", 0) or 0))
+ epoch_val_eval_max_rows = max(0, int(resolved_config.get("epoch_val_eval_max_rows", 0) or 0))
+ batch_prepare_context: Dict[str, Any] | None = None
+ batch_prepare_context_getter = getattr(graph_cache, "batch_prepare_context", None)
+ if batch_prepare_workers > 0 and callable(batch_prepare_context_getter):
+ candidate_context = batch_prepare_context_getter()
+ if isinstance(candidate_context, Mapping):
+ resolved_candidate_context = dict(candidate_context or {})
+ has_conversation_specs = bool(dict(resolved_candidate_context).get("conversation_specs"))
+ has_graph_dir = bool(clean_text(resolved_candidate_context.get("graph_dir", "")))
+ if has_conversation_specs or has_graph_dir:
+ batch_prepare_context = resolved_candidate_context
+ batch_prepare_worker_pool = (
+ _PreparedBatchWorkerPool(
+ worker_context=batch_prepare_context,
+ worker_count=batch_prepare_workers,
+ )
+ if batch_prepare_workers > 0 and batch_prepare_context is not None
+ else None
+ )
+
+ resolved_resume_training_state = dict(resume_training_state or {})
+ optimizer_state_dict = resolved_resume_training_state.get("optimizer_state_dict")
+ if isinstance(optimizer_state_dict, Mapping):
+ optimizer.load_state_dict(dict(optimizer_state_dict))
+ scheduler_state_dict = resolved_resume_training_state.get("scheduler_state_dict")
+ if isinstance(scheduler_state_dict, Mapping):
+ scheduler.load_state_dict(dict(scheduler_state_dict))
+ scaler_state_dict = resolved_resume_training_state.get("scaler_state_dict")
+ if isinstance(scaler_state_dict, Mapping) and hasattr(scaler, "load_state_dict"):
+ scaler.load_state_dict(dict(scaler_state_dict))
+
+ history = list(resolved_resume_training_state.get("history", []) or [])
+ val_selection_enabled = bool(epoch_val_eval)
+ best_val_loss = float(resolved_resume_training_state.get("best_val_loss", float("inf")) or float("inf"))
+ best_val_selection_key: tuple[float, float, float, float, float] | None = None
+ raw_best_val_selection_key = resolved_resume_training_state.get("best_val_selection_key")
+ if isinstance(raw_best_val_selection_key, (list, tuple)) and len(raw_best_val_selection_key) == 5:
+ best_val_selection_key = tuple(float(value or 0.0) for value in raw_best_val_selection_key)
+ best_val_selection_score = float(
+ resolved_resume_training_state.get("best_val_selection_score", float("-inf")) or float("-inf")
+ )
+ best_state: Dict[str, Any] | None = None
+ patience = max(0, int(resolved_resume_training_state.get("patience", 0) or 0))
+ if not val_selection_enabled:
+ best_val_loss = float("inf")
+ best_val_selection_key = None
+ best_val_selection_score = float("-inf")
+ patience = 0
+ global_step = max(0, int(resolved_resume_training_state.get("global_step", 0) or 0))
+ stop_training = global_step >= total_steps
+ resolved_batch_size = max(1, int(resolved_config["batch_size"]))
+ steps_per_epoch = max(1, math.ceil(len(train_rows) / resolved_batch_size))
+ start_epoch = max(1, int(resolved_resume_training_state.get("epoch", 1) or 1))
+ start_epoch_step = max(0, int(resolved_resume_training_state.get("epoch_step", 0) or 0))
+ if start_epoch_step >= steps_per_epoch:
+ start_epoch += start_epoch_step // steps_per_epoch
+ start_epoch_step = start_epoch_step % steps_per_epoch
+
+ for epoch in range(start_epoch, max(1, int(resolved_config["epochs"])) + 1):
+ if stop_training:
+ break
+ model.train()
+ skip_batches = start_epoch_step if epoch == start_epoch else 0
+ epoch_step = skip_batches
+ if batch_prepare_workers > 0 and batch_prepare_context is not None:
+ batch_iter: Iterator[Any] = _iter_prepared_batches_with_workers(
+ train_rows,
+ graph_cache=graph_cache,
+ batch_size=resolved_batch_size,
+ shuffle=True,
+ seed=epoch,
+ worker_context=batch_prepare_context,
+ worker_count=batch_prepare_workers,
+ lookahead_batches=batch_prepare_lookahead_batches,
+ graph_error_stage="train_batch_prepare",
+ skip_batches=skip_batches,
+ worker_pool=batch_prepare_worker_pool,
+ prefer_completion_order=True,
+ worker_event_callback=worker_event_callback,
+ )
+ else:
+ batch_iter = _iter_batches_with_prefetch(
+ graph_cache,
+ train_rows,
+ batch_size=resolved_batch_size,
+ shuffle=True,
+ seed=epoch,
+ lookahead_batches=prefetch_lookahead_batches,
+ skip_batches=skip_batches,
+ )
+ for batch_payload in batch_iter:
+ optimizer.zero_grad(set_to_none=True)
+ with _autocast_context(enabled=amp_enabled and device.type == "cuda", device=device):
+ if isinstance(batch_payload, Mapping) and "scored_examples" in batch_payload:
+ batch_result = score_prepared_batch(
+ model,
+ batch_payload,
+ device=device,
+ loss_config=resolved_config,
+ apply_example_weights=True,
+ graph_error_callback=graph_error_callback,
+ graph_error_stage="train_batch",
+ )
+ else:
+ batch_result = score_batch(
+ model,
+ batch_payload,
+ graph_cache,
+ device=device,
+ loss_config=resolved_config,
+ apply_example_weights=True,
+ graph_error_callback=graph_error_callback,
+ graph_error_stage="train_batch",
+ )
+ loss = batch_result["loss"]
+ if l2sp_reference_state:
+ l2sp_terms = [
+ (parameter.float() - l2sp_reference_state[name].float()).pow(2).mean()
+ for name, parameter in model.named_parameters()
+ if parameter.requires_grad and name in l2sp_reference_state
+ ]
+ if l2sp_terms:
+ l2sp_loss = torch.stack(l2sp_terms).mean()
+ loss = loss + loss.new_tensor(float(l2sp_loss_weight)) * l2sp_loss
+ batch_result["loss"] = loss
+ metrics = dict(batch_result.get("metrics", {}) or {})
+ metrics["l2sp_loss"] = round(float(l2sp_loss.detach().cpu()), 8)
+ metrics["l2sp_loss_weight"] = float(l2sp_loss_weight)
+ metrics["l2sp_parameter_count"] = int(len(l2sp_terms))
+ batch_result["metrics"] = metrics
+ if int(dict(batch_result.get("metrics", {}) or {}).get("samples", 0) or 0) <= 0:
+ continue
+ scaler.scale(loss).backward()
+ scaler.unscale_(optimizer)
+ torch.nn.utils.clip_grad_norm_(model.parameters(), float(resolved_config["grad_clip"]))
+ scaler.step(optimizer)
+ scaler.update()
+ scheduler.step()
+ global_step += 1
+ epoch_step += 1
+ if step_callback is not None and (global_step == 1 or global_step % log_every_steps == 0 or global_step == total_steps):
+ step_callback(
+ {
+ "epoch": epoch,
+ "epoch_step": epoch_step,
+ "global_step": global_step,
+ "total_steps": total_steps,
+ "batch_metrics": dict(batch_result.get("metrics", {}) or {}),
+ }
+ )
+ if step_checkpoint_callback is not None and checkpoint_every_steps > 0 and (
+ global_step % checkpoint_every_steps == 0 or global_step == total_steps
+ ):
+ step_checkpoint_callback(
+ {
+ "epoch": epoch,
+ "epoch_step": epoch_step,
+ "global_step": global_step,
+ "total_steps": total_steps,
+ "batch_metrics": dict(batch_result.get("metrics", {}) or {}),
+ },
+ model,
+ optimizer,
+ scheduler,
+ scaler,
+ )
+ if global_step >= total_steps:
+ stop_training = True
+ break
+ if epoch_train_eval:
+ train_summary = evaluate_examples(
+ model,
+ resolved_train_eval_rows,
+ graph_cache,
+ device=device,
+ batch_size=max(1, int(resolved_config["batch_size"])),
+ loss_config=resolved_config,
+ prefetch_lookahead_batches=prefetch_lookahead_batches,
+ graph_error_callback=graph_error_callback,
+ graph_error_stage="train_eval",
+ max_rows=epoch_train_eval_max_rows,
+ sample_seed=epoch,
+ batch_prepare_context=batch_prepare_context,
+ batch_prepare_workers=batch_prepare_workers,
+ batch_prepare_lookahead_batches=batch_prepare_lookahead_batches,
+ batch_prepare_worker_pool=batch_prepare_worker_pool,
+ worker_event_callback=worker_event_callback,
+ )
+ else:
+ train_summary = _skipped_eval_summary(len(resolved_train_eval_rows))
+ if epoch_val_eval:
+ val_summary = evaluate_examples(
+ model,
+ val_rows,
+ graph_cache,
+ device=device,
+ batch_size=max(1, int(resolved_config["batch_size"])),
+ loss_config=resolved_config,
+ prefetch_lookahead_batches=prefetch_lookahead_batches,
+ graph_error_callback=graph_error_callback,
+ graph_error_stage="val_eval",
+ max_rows=epoch_val_eval_max_rows,
+ sample_seed=epoch,
+ batch_prepare_context=batch_prepare_context,
+ batch_prepare_workers=batch_prepare_workers,
+ batch_prepare_lookahead_batches=batch_prepare_lookahead_batches,
+ batch_prepare_worker_pool=batch_prepare_worker_pool,
+ worker_event_callback=worker_event_callback,
+ )
+ else:
+ val_summary = _skipped_eval_summary(len(val_rows))
+ epoch_summary = {
+ "epoch": epoch,
+ "global_step": global_step,
+ "total_steps": total_steps,
+ "train": train_summary,
+ "val": val_summary,
+ }
+ current_val_loss = float(val_summary.get("loss", 0.0) or 0.0)
+ current_val_selection_key = _checkpoint_selection_key(val_summary) if val_selection_enabled else None
+ current_val_selection_score = _checkpoint_selection_score(val_summary) if val_selection_enabled else float("-inf")
+ is_best = False
+ if val_selection_enabled and (
+ best_val_selection_key is None or current_val_selection_key is not None and current_val_selection_key > best_val_selection_key
+ ):
+ best_val_loss = current_val_loss
+ best_val_selection_key = current_val_selection_key
+ best_val_selection_score = current_val_selection_score
+ best_state = {
+ "model": model.state_dict(),
+ "optimizer": optimizer.state_dict(),
+ "epoch": epoch,
+ }
+ patience = 0
+ is_best = True
+ elif val_selection_enabled:
+ patience += 1
+ epoch_summary["is_best"] = is_best
+ epoch_summary["val_selection_enabled"] = val_selection_enabled
+ epoch_summary["patience"] = patience
+ epoch_summary["best_val_loss"] = round(float(best_val_loss), 6)
+ epoch_summary["val_selection_score"] = round(float(current_val_selection_score), 6)
+ epoch_summary["best_val_selection_score"] = round(float(best_val_selection_score if best_val_selection_score != float("-inf") else 0.0), 6)
+ epoch_summary["best_val_selection_key"] = list(best_val_selection_key) if best_val_selection_key is not None else []
+ history.append(epoch_summary)
+ if epoch_callback is not None:
+ epoch_callback(dict(epoch_summary))
+ if checkpoint_callback is not None:
+ checkpoint_callback(dict(epoch_summary), model, optimizer, scheduler, scaler)
+ if val_selection_enabled and not is_best and patience >= max(1, int(resolved_config["early_stopping_patience"])):
+ break
+ if stop_training:
+ break
+
+ if batch_prepare_worker_pool is not None:
+ batch_prepare_worker_pool.close()
+
+ if best_state is not None:
+ model.load_state_dict(best_state["model"])
+
+ return {
+ "model": model,
+ "history": history,
+ "best_val_loss": round(float(best_val_loss if best_val_loss != float("inf") else 0.0), 6),
+ "best_val_selection_score": round(float(best_val_selection_score if best_val_selection_score != float("-inf") else 0.0), 6),
+ "config": resolved_config,
+ "trainable_stage": dict(trainable_stage_summary),
+ "optimizer_state_dict": optimizer.state_dict(),
+ "scheduler_state_dict": scheduler.state_dict(),
+ "scaler_state_dict": scaler.state_dict() if hasattr(scaler, "state_dict") else {},
+ }
+
+
+class LoadedNodeMemoryScorer:
+ def __init__(
+ self,
+ *,
+ node_model_path: Path,
+ path_model_path: Path | None = None,
+ device: torch.device | str | None = None,
+ ) -> None:
+ requested_device = device if device is not None else os.environ.get("TMCRA_NODE_MEMORY_DEVICE", "")
+ self.device = torch.device(str(requested_device)) if str(requested_device or "").strip() else torch.device("cuda" if torch.cuda.is_available() else "cpu")
+ node_payload = torch.load(Path(node_model_path), map_location=self.device, weights_only=True)
+ validate_node_memory_checkpoint_payload(node_payload, context=f"LoadedNodeMemoryScorer(node_model_path={node_model_path})")
+ self.model = LocomoNodeMemoryModel.from_checkpoint(node_payload, device=self.device)
+ node_load_report = dict(getattr(self.model, "_checkpoint_load_report", {}) or {})
+ node_load_failures = {
+ key: list(node_load_report.get(key, []) or [])
+ for key in ("missing_keys", "unexpected_keys", "skipped_keys")
+ if node_load_report.get(key)
+ }
+ if node_load_failures:
+ raise ValueError(f"node checkpoint is not fully compatible: {node_load_failures}")
+ if path_model_path:
+ path_payload = torch.load(Path(path_model_path), map_location=self.device, weights_only=True)
+ validate_node_memory_checkpoint_payload(path_payload, context=f"LoadedNodeMemoryScorer(path_model_path={path_model_path})")
+ path_load_report = _load_compatible_state_dict(
+ self.model,
+ dict(path_payload.get("state_dict", {}) or {}),
+ )
+ path_load_failures = {
+ key: list(path_load_report.get(key, []) or [])
+ for key in ("missing_keys", "unexpected_keys", "skipped_keys")
+ if path_load_report.get(key)
+ }
+ if path_load_failures:
+ raise ValueError(f"path checkpoint is not fully compatible: {path_load_failures}")
+ self.model.eval()
+ self._runtime_graph_cache_key = ""
+ self._runtime_graph_cache_model_id = id(self.model)
+ self._runtime_graph_tensors: Dict[str, Any] | None = None
+ self._runtime_graph_hidden: Tensor | None = None
+ self._runtime_recall_cache_key = ""
+ self._runtime_recall_cache: Dict[str, Any] | None = None
+
+ def _runtime_graph_signature(self, graph: Mapping[str, Any]) -> str:
+ payload = {
+ "conversation_id": clean_text(graph.get("conversation_id", "")) or "runtime",
+ "nodes": [dict(node) for node in list(graph.get("nodes", []) or [])],
+ "edges": [dict(edge) for edge in list(graph.get("edges", []) or [])],
+ "paths": [dict(path) for path in list(graph.get("paths", []) or [])],
+ }
+ canonical = json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
+ return hashlib.sha256(canonical.encode("utf-8")).hexdigest()
+
+ def _runtime_graph_state(self, graph: Mapping[str, Any]) -> tuple[Dict[str, Any], Tensor | None]:
+ model_id = id(self.model)
+ cache_key = self._runtime_graph_signature(graph)
+ if (
+ self._runtime_graph_tensors is not None
+ and self._runtime_graph_cache_key == cache_key
+ and self._runtime_graph_cache_model_id == model_id
+ ):
+ return self._runtime_graph_tensors, self._runtime_graph_hidden
+ graph_tensors = tensorize_graph(graph, device=self.device)
+ node_hidden: Tensor | None = None
+ encode_graph = getattr(self.model, "encode_graph", None)
+ if callable(encode_graph):
+ with torch.no_grad():
+ node_hidden = encode_graph(graph_tensors)
+ self._runtime_graph_cache_key = cache_key
+ self._runtime_graph_cache_model_id = model_id
+ self._runtime_graph_tensors = graph_tensors
+ self._runtime_graph_hidden = node_hidden
+ self._runtime_recall_cache_key = ""
+ self._runtime_recall_cache = None
+ return graph_tensors, node_hidden
+
+ def score_runtime(
+ self,
+ *,
+ graph: Mapping[str, Any],
+ question: str,
+ question_features: Mapping[str, Any] | None = None,
+ candidate_event_ids: Sequence[str] | None = None,
+ rerank_top_k: int | None = None,
+ event_rerank_mode: str = "matrix",
+ matrix_event_top_k: int = DEFAULT_MATRIX_EVENT_TOP_K,
+ event_pair_feature_mode: str = "full",
+ support_path_k: int | None = None,
+ top_k: int | None = None,
+ ) -> Dict[str, Any]:
+ profile_enabled = str(os.environ.get("TMCRA_NODE_RUNTIME_PROFILE", "")).strip().lower() in {
+ "1", "true", "yes", "on",
+ }
+
+ def profile_clock() -> float:
+ if profile_enabled and self.device.type == "cuda":
+ torch.cuda.synchronize(self.device)
+ return time.perf_counter()
+
+ def materialize(values: Tensor, *, sigmoid: bool = False) -> List[float]:
+ resolved = torch.sigmoid(values) if sigmoid else values
+ return [float(value) for value in resolved.detach().float().cpu().tolist()]
+
+ def score_map(ids: Sequence[str], values: Tensor, *, sigmoid: bool = False) -> Dict[str, float]:
+ return dict(zip(ids, materialize(values, sigmoid=sigmoid)))
+
+ runtime_started = profile_clock()
+ graph_tensors, node_hidden = self._runtime_graph_state(graph)
+ graph_state_finished = profile_clock()
+ explicit_runtime_candidates = candidate_event_ids is not None
+ runtime_candidate_event_ids = [
+ clean_text(item)
+ for item in (candidate_event_ids if candidate_event_ids is not None else graph_tensors.get("event_node_ids", []))
+ if clean_text(item)
+ ]
+ question_feature_payload = dict(question_features or extract_question_features(question))
+ resolved_event_rerank_mode = _normalize_event_rerank_mode(event_rerank_mode)
+ resolved_matrix_event_top_k = max(1, int(matrix_event_top_k or DEFAULT_MATRIX_EVENT_TOP_K))
+ recall_cache_payload = {
+ "graph": self._runtime_graph_cache_key,
+ "question": clean_text(question),
+ "question_features": question_feature_payload,
+ "event_pair_feature_mode": _normalize_event_pair_feature_mode(event_pair_feature_mode),
+ }
+ recall_cache_key = hashlib.sha256(
+ json.dumps(
+ recall_cache_payload,
+ ensure_ascii=False,
+ sort_keys=True,
+ separators=(",", ":"),
+ default=str,
+ ).encode("utf-8")
+ ).hexdigest()
+ recall_cache_hit = bool(
+ self._runtime_recall_cache is not None
+ and self._runtime_recall_cache_key == recall_cache_key
+ )
+ first_pass_example = QueryTrainingExample(
+ conversation_id=clean_text(graph.get("conversation_id", "runtime")),
+ question_id="runtime:recall",
+ question=question,
+ question_features=question_feature_payload,
+ candidate_event_ids=runtime_candidate_event_ids,
+ positive_event_ids=[],
+ positive_path_ids=[],
+ positive_time_node_ids=[],
+ negative_event_ids=[],
+ answer_targets={"answer_type": "event_text"},
+ answer_plan_targets={},
+ temporal_target={
+ "is_temporal": bool(question_feature_payload.get("is_temporal", False)),
+ "question_is_temporal": bool(question_feature_payload.get("is_temporal", False)),
+ "use_temporal_head": bool(question_feature_payload.get("is_temporal", False)),
+ },
+ event_catalog_size=len(runtime_candidate_event_ids),
+ metadata={},
+ )
+ recall_started = profile_clock()
+ if recall_cache_hit:
+ recall_outputs = dict(self._runtime_recall_cache or {})
+ else:
+ with torch.no_grad():
+ if node_hidden is not None and hasattr(self.model, "score_example_with_graph_encoding"):
+ recall_outputs = _call_with_supported_kwargs(
+ self.model.score_example_with_graph_encoding,
+ graph_tensors,
+ node_hidden,
+ first_pass_example,
+ event_rerank_mode="single",
+ matrix_event_top_k=resolved_matrix_event_top_k,
+ event_pair_feature_mode=event_pair_feature_mode,
+ recall_only=True,
+ )
+ else:
+ recall_outputs = _call_with_supported_kwargs(
+ self.model.score_example,
+ graph_tensors,
+ first_pass_example,
+ event_rerank_mode="single",
+ matrix_event_top_k=resolved_matrix_event_top_k,
+ event_pair_feature_mode=event_pair_feature_mode,
+ )
+ if isinstance(recall_outputs.get("recall_pair_features"), Tensor):
+ self._runtime_recall_cache_key = recall_cache_key
+ self._runtime_recall_cache = dict(recall_outputs)
+ recall_finished = profile_clock()
+ recall_event_scores = score_map(
+ recall_outputs["recall_event_ids"],
+ recall_outputs["recall_event_logits"],
+ sigmoid=True,
+ )
+ recall_event_ids = [
+ event_id
+ for event_id, _ in sorted(recall_event_scores.items(), key=lambda item: (-float(item[1]), item[0]))
+ ]
+ resolved_rerank_top_k = max(
+ 1,
+ min(
+ len(runtime_candidate_event_ids),
+ int(rerank_top_k or DEFAULT_RUNTIME_RECALL_TOP_K),
+ ),
+ )
+ if explicit_runtime_candidates:
+ # A caller-provided candidate set is a contract, not just a hint. The
+ # oracle/probe path relies on this to distinguish recall failure from
+ # rerank failure; silently dropping explicit candidates makes the
+ # diagnostic look worse than the actual reranker.
+ rerank_candidate_event_ids = dedupe_texts(
+ [
+ *runtime_candidate_event_ids,
+ *(recall_event_ids[:resolved_rerank_top_k] if recall_event_ids else []),
+ ]
+ )
+ else:
+ rerank_candidate_event_ids = (
+ recall_event_ids[:resolved_rerank_top_k]
+ if recall_event_ids
+ else list(runtime_candidate_event_ids[:resolved_rerank_top_k])
+ )
+ rerank_example = QueryTrainingExample(
+ conversation_id=clean_text(graph.get("conversation_id", "runtime")),
+ question_id="runtime:rerank",
+ question=question,
+ question_features=question_feature_payload,
+ candidate_event_ids=rerank_candidate_event_ids,
+ positive_event_ids=[],
+ positive_path_ids=[],
+ positive_time_node_ids=[],
+ negative_event_ids=[],
+ answer_targets={"answer_type": "event_text"},
+ answer_plan_targets={},
+ temporal_target={
+ "is_temporal": bool(question_feature_payload.get("is_temporal", False)),
+ "question_is_temporal": bool(question_feature_payload.get("is_temporal", False)),
+ "use_temporal_head": bool(question_feature_payload.get("is_temporal", False)),
+ },
+ event_catalog_size=len(runtime_candidate_event_ids),
+ metadata={},
+ )
+ rerank_started = profile_clock()
+ with torch.no_grad():
+ if node_hidden is not None and hasattr(self.model, "score_example_with_graph_encoding"):
+ outputs = _call_with_supported_kwargs(
+ self.model.score_example_with_graph_encoding,
+ graph_tensors,
+ node_hidden,
+ rerank_example,
+ event_rerank_mode=resolved_event_rerank_mode,
+ matrix_event_top_k=resolved_matrix_event_top_k,
+ event_pair_feature_mode=event_pair_feature_mode,
+ precomputed_recall=(
+ recall_outputs
+ if isinstance(recall_outputs.get("recall_pair_features"), Tensor)
+ else None
+ ),
+ )
+ else:
+ outputs = _call_with_supported_kwargs(
+ self.model.score_example,
+ graph_tensors,
+ rerank_example,
+ event_rerank_mode=resolved_event_rerank_mode,
+ matrix_event_top_k=resolved_matrix_event_top_k,
+ event_pair_feature_mode=event_pair_feature_mode,
+ )
+ rerank_finished = profile_clock()
+ materialization_started = profile_clock()
+ base_event_logits = outputs.get("base_event_logits", outputs.get("event_logits", torch.tensor([], dtype=torch.float32, device=self.device)))
+ rerank_event_logits = outputs.get("rerank_event_logits", outputs.get("event_logits", torch.tensor([], dtype=torch.float32, device=self.device)))
+ distractor_event_logits = outputs.get("distractor_event_logits", rerank_event_logits)
+ calibrated_event_logits = outputs.get("calibrated_event_logits", outputs.get("event_logits", torch.tensor([], dtype=torch.float32, device=self.device)))
+ matrix_event_ids = list(outputs.get("matrix_event_ids", []) or [])
+ matrix_event_delta_logits = outputs.get(
+ "matrix_event_delta_logits",
+ outputs["event_logits"].new_zeros((0,)) if isinstance(outputs.get("event_logits"), Tensor) else torch.tensor([], dtype=torch.float32, device=self.device),
+ )
+ event_distractor_logits = outputs.get(
+ "event_distractor_logits",
+ outputs["event_logits"].new_zeros((0,)) if isinstance(outputs.get("event_logits"), Tensor) else torch.tensor([], dtype=torch.float32, device=self.device),
+ )
+ event_distractor_delta_logits = outputs.get(
+ "event_distractor_delta_logits",
+ outputs["event_logits"].new_zeros((0,)) if isinstance(outputs.get("event_logits"), Tensor) else torch.tensor([], dtype=torch.float32, device=self.device),
+ )
+ event_fusion_delta_logits = outputs.get(
+ "event_fusion_delta_logits",
+ outputs["event_logits"].new_zeros((0,)) if isinstance(outputs.get("event_logits"), Tensor) else torch.tensor([], dtype=torch.float32, device=self.device),
+ )
+ event_tunnel_support_logits = outputs.get(
+ "event_tunnel_support_logits",
+ outputs["event_logits"].new_zeros((0,)) if isinstance(outputs.get("event_logits"), Tensor) else torch.tensor([], dtype=torch.float32, device=self.device),
+ )
+ event_tunnel_delta_logits = outputs.get(
+ "event_tunnel_delta_logits",
+ outputs["event_logits"].new_zeros((0,)) if isinstance(outputs.get("event_logits"), Tensor) else torch.tensor([], dtype=torch.float32, device=self.device),
+ )
+ base_path_logits = outputs.get("base_path_logits", outputs.get("path_logits", torch.tensor([], dtype=torch.float32, device=self.device)))
+ rerank_path_logits = outputs.get("rerank_path_logits", outputs.get("path_logits", torch.tensor([], dtype=torch.float32, device=self.device)))
+ calibrated_path_logits = outputs.get("calibrated_path_logits", outputs.get("path_logits", torch.tensor([], dtype=torch.float32, device=self.device)))
+ matrix_path_ids = list(outputs.get("matrix_path_ids", []) or [])
+ matrix_path_delta_logits = outputs.get(
+ "matrix_path_delta_logits",
+ outputs["path_logits"].new_zeros((0,)) if isinstance(outputs.get("path_logits"), Tensor) else torch.tensor([], dtype=torch.float32, device=self.device),
+ )
+ path_fusion_delta_logits = outputs.get(
+ "path_fusion_delta_logits",
+ outputs["path_logits"].new_zeros((0,)) if isinstance(outputs.get("path_logits"), Tensor) else torch.tensor([], dtype=torch.float32, device=self.device),
+ )
+ path_tunnel_support_logits = outputs.get(
+ "path_tunnel_support_logits",
+ outputs["path_logits"].new_zeros((0,)) if isinstance(outputs.get("path_logits"), Tensor) else torch.tensor([], dtype=torch.float32, device=self.device),
+ )
+ path_tunnel_delta_logits = outputs.get(
+ "path_tunnel_delta_logits",
+ outputs["path_logits"].new_zeros((0,)) if isinstance(outputs.get("path_logits"), Tensor) else torch.tensor([], dtype=torch.float32, device=self.device),
+ )
+ event_reverse_values = outputs.get(
+ "event_reverse_scores",
+ outputs["event_logits"].new_zeros((0,)) if isinstance(outputs.get("event_logits"), Tensor) else torch.tensor([], dtype=torch.float32, device=self.device),
+ )
+ event_boundary_values = outputs.get(
+ "event_boundary_scores",
+ outputs["event_logits"].new_zeros((0,)) if isinstance(outputs.get("event_logits"), Tensor) else torch.tensor([], dtype=torch.float32, device=self.device),
+ )
+ event_reverse_relation_values = outputs.get(
+ "event_reverse_relations",
+ outputs["event_logits"].new_zeros((0,)) if isinstance(outputs.get("event_logits"), Tensor) else torch.tensor([], dtype=torch.float32, device=self.device),
+ )
+ path_reverse_values = outputs.get(
+ "path_reverse_scores",
+ outputs["path_logits"].new_zeros((0,)) if isinstance(outputs.get("path_logits"), Tensor) else torch.tensor([], dtype=torch.float32, device=self.device),
+ )
+ path_boundary_values = outputs.get(
+ "path_boundary_scores",
+ outputs["path_logits"].new_zeros((0,)) if isinstance(outputs.get("path_logits"), Tensor) else torch.tensor([], dtype=torch.float32, device=self.device),
+ )
+ path_reverse_relation_values = outputs.get(
+ "path_reverse_relations",
+ outputs["path_logits"].new_zeros((0,)) if isinstance(outputs.get("path_logits"), Tensor) else torch.tensor([], dtype=torch.float32, device=self.device),
+ )
+ candidate_event_ids = outputs["candidate_event_ids"]
+ base_event_scores = score_map(candidate_event_ids, base_event_logits, sigmoid=True)
+ rerank_event_scores = score_map(candidate_event_ids, rerank_event_logits, sigmoid=True)
+ distractor_event_scores = score_map(candidate_event_ids, event_distractor_logits, sigmoid=True)
+ distractor_adjusted_event_scores = score_map(candidate_event_ids, distractor_event_logits, sigmoid=True)
+ calibrated_event_scores = score_map(candidate_event_ids, calibrated_event_logits, sigmoid=True)
+ event_scores = score_map(candidate_event_ids, outputs["event_logits"], sigmoid=True)
+ matrix_event_scores = score_map(matrix_event_ids, matrix_event_delta_logits, sigmoid=True)
+ event_distractor_delta_scores = score_map(candidate_event_ids, event_distractor_delta_logits)
+ event_fusion_delta_scores = score_map(candidate_event_ids, event_fusion_delta_logits, sigmoid=True)
+ event_tunnel_support_scores = score_map(candidate_event_ids, event_tunnel_support_logits, sigmoid=True)
+ event_tunnel_delta_scores = score_map(candidate_event_ids, event_tunnel_delta_logits)
+ tri_maze_event_reverse_scores = score_map(candidate_event_ids, event_reverse_values)
+ tri_maze_event_boundary_scores = score_map(candidate_event_ids, event_boundary_values)
+ tri_maze_event_reverse_relations = score_map(candidate_event_ids, event_reverse_relation_values)
+ candidate_path_ids = outputs["candidate_path_ids"]
+ base_path_scores = score_map(candidate_path_ids, base_path_logits, sigmoid=True)
+ rerank_path_scores = score_map(candidate_path_ids, rerank_path_logits, sigmoid=True)
+ calibrated_path_scores = score_map(candidate_path_ids, calibrated_path_logits, sigmoid=True)
+ path_scores = score_map(candidate_path_ids, outputs["path_logits"], sigmoid=True)
+ matrix_path_scores = score_map(matrix_path_ids, matrix_path_delta_logits, sigmoid=True)
+ path_fusion_delta_scores = score_map(candidate_path_ids, path_fusion_delta_logits, sigmoid=True)
+ path_tunnel_support_scores = score_map(candidate_path_ids, path_tunnel_support_logits, sigmoid=True)
+ path_tunnel_delta_scores = score_map(candidate_path_ids, path_tunnel_delta_logits)
+ tri_maze_path_reverse_scores = score_map(candidate_path_ids, path_reverse_values)
+ tri_maze_path_boundary_scores = score_map(candidate_path_ids, path_boundary_values)
+ tri_maze_path_reverse_relations = score_map(candidate_path_ids, path_reverse_relation_values)
+ temporal_node_scores: Dict[str, float] = {}
+ for node_id, score in zip(
+ outputs["candidate_temporal_node_ids"],
+ materialize(outputs["temporal_logits"], sigmoid=True),
+ ):
+ temporal_node_scores[node_id] = max(score, float(temporal_node_scores.get(node_id, 0.0)))
+ answer_type_values = materialize(F.softmax(outputs["answer_type_logits"], dim=-1))
+ answer_type_scores = {
+ answer_type: answer_type_values[index]
+ for answer_type, index in ANSWER_TYPE_TO_ID.items()
+ }
+ answer_plan_logits = outputs.get(
+ "answer_plan_logits",
+ outputs["event_logits"].new_zeros((0, len(ANSWER_PLAN_OUTPUTS)))
+ if isinstance(outputs.get("event_logits"), Tensor)
+ else torch.tensor([], dtype=torch.float32, device=self.device),
+ )
+ answer_plan_scores: Dict[str, Dict[str, float]] = {}
+ if isinstance(answer_plan_logits, Tensor) and answer_plan_logits.numel() > 0:
+ for role_name, role_index in ANSWER_PLAN_OUTPUT_TO_ID.items():
+ answer_plan_scores[role_name] = score_map(
+ candidate_event_ids,
+ answer_plan_logits[:, role_index],
+ sigmoid=True,
+ )
+ memory_router_logits = outputs.get("memory_router_logits", torch.tensor([], dtype=torch.float32, device=self.device))
+ memory_router_scores = score_map(MEMORY_ROUTER_LAYERS, memory_router_logits, sigmoid=True)
+ memory_router_top_layers = [
+ layer
+ for layer, score in sorted(memory_router_scores.items(), key=lambda item: (-float(item[1]), item[0]))
+ if float(score) >= 0.5
+ ]
+ if not memory_router_top_layers and memory_router_scores:
+ memory_router_top_layers = [
+ max(memory_router_scores.items(), key=lambda item: float(item[1]))[0]
+ ]
+ event_distractor_enabled = _output_projection_has_signal(getattr(self.model, "event_distractor_head", None))
+ event_calibration_enabled = _output_projection_has_signal(getattr(self.model, "event_calibration_head", None))
+ path_calibration_enabled = _output_projection_has_signal(getattr(self.model, "path_calibration_head", None))
+ event_tunnel_enabled = _output_projection_has_signal(getattr(self.model, "event_tunnel_head", None))
+ path_tunnel_enabled = _output_projection_has_signal(getattr(self.model, "path_tunnel_head", None))
+ final_event_fusion_enabled = _output_projection_has_signal(getattr(self.model, "final_event_fusion_head", None))
+ final_path_fusion_enabled = _output_projection_has_signal(getattr(self.model, "final_path_fusion_head", None))
+ event_fusion_enabled = bool(event_calibration_enabled or event_tunnel_enabled or final_event_fusion_enabled)
+ path_fusion_enabled = bool(path_calibration_enabled or path_tunnel_enabled or final_path_fusion_enabled)
+ decision_fusion_enabled = bool(event_tunnel_enabled or path_tunnel_enabled or final_event_fusion_enabled or final_path_fusion_enabled)
+ resolved_support_path_k = max(1, int(support_path_k or 3))
+ resolved_top_k = max(1, int(top_k or resolved_support_path_k))
+ path_lookup = {
+ clean_text(path.get("id", "")): dict(path)
+ for path in list(graph_tensors.get("paths", []) or [])
+ if clean_text(path.get("id", ""))
+ }
+ path_model_scores = dict(path_scores)
+ path_chain_extended_scores: Dict[str, float] = {}
+ path_chain_extension_delta_scores: Dict[str, float] = {}
+ ranked_final_path_ids = _sorted_score_ids(path_scores)
+ ranked_final_event_ids = _sorted_score_ids(event_scores)
+ selected_path_ids: List[str] = []
+ selected_event_ids: List[str] = []
+ decision_score_source = ""
+ if decision_fusion_enabled:
+ selected_path_limit = min(len(ranked_final_path_ids), max(1, min(resolved_support_path_k, resolved_top_k)))
+ selected_path_ids = list(ranked_final_path_ids[:selected_path_limit])
+ selected_event_ids = dedupe_texts(
+ [
+ *[
+ clean_text(path_lookup.get(path_id, {}).get("event_id", ""))
+ for path_id in selected_path_ids
+ if clean_text(path_lookup.get(path_id, {}).get("event_id", ""))
+ ],
+ *_sorted_score_ids(event_scores, limit=max(resolved_support_path_k * 2, resolved_top_k)),
+ ],
+ max_items=max(resolved_support_path_k * 2, resolved_top_k),
+ )
+ if final_path_fusion_enabled and selected_path_ids:
+ decision_score_source = "learned_final_path_fusion"
+ elif final_event_fusion_enabled:
+ decision_score_source = "learned_final_event_fusion"
+ elif path_tunnel_enabled and selected_path_ids:
+ decision_score_source = "learned_path_tunnel"
+ elif event_tunnel_enabled:
+ decision_score_source = "learned_event_tunnel"
+ if path_chain_extension_delta_scores and any(
+ float(path_chain_extension_delta_scores.get(path_id, 0.0) or 0.0) > 0.0
+ for path_id in selected_path_ids
+ ):
+ decision_score_source = (
+ f"{decision_score_source}+chain_extension"
+ if decision_score_source
+ else "chain_extension"
+ )
+ focused_answer_type = _dominant_answer_type_from_scores(
+ answer_type_scores,
+ question_is_temporal=bool(question_feature_payload.get("is_temporal", False)),
+ )
+ materialization_finished = profile_clock()
+ return {
+ "runtime_profile": {
+ "enabled": bool(profile_enabled),
+ "graph_state_sec": round(graph_state_finished - runtime_started, 6),
+ "recall_sec": round(recall_finished - recall_started, 6),
+ "rerank_sec": round(rerank_finished - rerank_started, 6),
+ "materialization_sec": round(materialization_finished - materialization_started, 6),
+ "total_sec": round(materialization_finished - runtime_started, 6),
+ "recall_cache_hit": bool(recall_cache_hit),
+ "runtime_candidate_event_count": len(runtime_candidate_event_ids),
+ "rerank_candidate_event_count": len(rerank_candidate_event_ids),
+ "recall_event_count": len(recall_outputs.get("recall_event_ids", []) or []),
+ "candidate_path_count": len(outputs.get("candidate_path_ids", []) or []),
+ },
+ "recall_event_scores": recall_event_scores,
+ "recall_event_ids": recall_event_ids,
+ "rerank_candidate_event_ids": rerank_candidate_event_ids,
+ "base_event_scores": base_event_scores,
+ "rerank_event_scores": rerank_event_scores,
+ "event_distractor_scores": distractor_event_scores,
+ "distractor_event_scores": distractor_adjusted_event_scores,
+ "calibrated_event_scores": calibrated_event_scores,
+ "matrix_event_scores": matrix_event_scores,
+ "event_distractor_delta_scores": event_distractor_delta_scores,
+ "event_fusion_delta_scores": event_fusion_delta_scores,
+ "event_tunnel_support_scores": event_tunnel_support_scores,
+ "event_tunnel_delta_scores": event_tunnel_delta_scores,
+ "tri_maze_event_reverse_scores": tri_maze_event_reverse_scores,
+ "tri_maze_event_boundary_scores": tri_maze_event_boundary_scores,
+ "tri_maze_event_reverse_relations": tri_maze_event_reverse_relations,
+ "matrix_rerank_event_ids": matrix_event_ids,
+ "matrix_enabled": bool(resolved_event_rerank_mode == "matrix" and matrix_event_ids),
+ "matrix_path_enabled": bool(matrix_path_ids),
+ "event_distractor_enabled": bool(event_distractor_enabled),
+ "event_calibration_enabled": bool(event_calibration_enabled),
+ "path_calibration_enabled": bool(path_calibration_enabled),
+ "event_tunnel_enabled": bool(event_tunnel_enabled),
+ "path_tunnel_enabled": bool(path_tunnel_enabled),
+ "final_event_fusion_enabled": bool(final_event_fusion_enabled),
+ "final_path_fusion_enabled": bool(final_path_fusion_enabled),
+ "decision_fusion_enabled": bool(decision_fusion_enabled),
+ "decision_score_source": decision_score_source,
+ "event_fusion_enabled": bool(event_fusion_enabled),
+ "path_fusion_enabled": bool(path_fusion_enabled),
+ "fusion_enabled": bool(event_fusion_enabled or path_fusion_enabled),
+ "path_chain_extension_enabled": bool(path_chain_extension_delta_scores),
+ "selected_event_ids": list(selected_event_ids),
+ "selected_path_ids": list(selected_path_ids),
+ "event_scores": event_scores,
+ "path_model_scores": dict(path_model_scores),
+ "base_path_scores": base_path_scores,
+ "rerank_path_scores": rerank_path_scores,
+ "calibrated_path_scores": calibrated_path_scores,
+ "matrix_path_scores": matrix_path_scores,
+ "matrix_path_rerank_ids": matrix_path_ids,
+ "path_fusion_delta_scores": path_fusion_delta_scores,
+ "path_tunnel_support_scores": path_tunnel_support_scores,
+ "path_tunnel_delta_scores": path_tunnel_delta_scores,
+ "path_chain_extension_delta_scores": path_chain_extension_delta_scores,
+ "path_chain_extended_scores": path_chain_extended_scores,
+ "tri_maze_path_reverse_scores": tri_maze_path_reverse_scores,
+ "tri_maze_path_boundary_scores": tri_maze_path_boundary_scores,
+ "tri_maze_path_reverse_relations": tri_maze_path_reverse_relations,
+ "path_scores": path_scores,
+ "temporal_scores": temporal_node_scores,
+ "answer_type_scores": answer_type_scores,
+ "answer_plan_scores": answer_plan_scores,
+ "memory_router_scores": memory_router_scores,
+ "memory_router_top_layers": memory_router_top_layers,
+ "memory_router_layers": list(MEMORY_ROUTER_LAYERS),
+ "focused_answer_type": focused_answer_type,
+ "event_pair_feature_mode": _normalize_event_pair_feature_mode(event_pair_feature_mode),
+ }
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/profile_layer.py b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/profile_layer.py
new file mode 100644
index 0000000000000000000000000000000000000000..942eb49545fc1e636be73c9ddd3c5059e1f9173b
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/profile_layer.py
@@ -0,0 +1,748 @@
+from __future__ import annotations
+
+import re
+from typing import Any, Iterable, Mapping, Sequence
+
+
+PROFILE_CATEGORIES = {"profile", "preference", "goal", "constraint", "stage_state", "status"}
+PROFILE_TYPES = {"setup", "preference", "constraint", "goal", "avoid", "usage_context"}
+PROFILE_SEMANTIC_SLOTS = {"identity", "research_topic", "education", "occupation"}
+PROFILE_AGGREGATE_SOURCE_KIND = "public_dialog_profile"
+PROFILE_AGGREGATE_CATEGORY = "profile"
+PROFILE_CLUSTER_SOURCE_KIND = "public_dialog_profile_cluster"
+PROFILE_CLUSTER_CATEGORY = "profile"
+PROFILE_CONSOLIDATOR_VERSION = "profile_consolidator_v1_structured_summary"
+
+_PROFILE_QUERY_MARKERS = (
+ "preference",
+ "prefer",
+ "like",
+ "dislike",
+ "recommend",
+ "suggest",
+ "advice",
+ "advise",
+ "any advice",
+ "any suggestions",
+ "any tips",
+ "tips",
+ "trouble with",
+ "struggling with",
+ "what do you think",
+ "learn more",
+ "resources",
+ "suited",
+ "fit me",
+ "for me",
+ "based on me",
+ "based on my",
+ "my setup",
+ "my profile",
+ "my goal",
+ "my constraint",
+ "my occupation",
+ "my previous occupation",
+ "my role",
+ "my previous role",
+ "my job",
+ "my previous job",
+ "what was my",
+ "where did i work",
+ "worked as",
+ "occupation",
+ "previous occupation",
+ "role",
+ "previous role",
+ "job",
+ "career",
+ "background",
+ "identity",
+ "experience",
+ "avoid",
+ "what should i",
+ "should i",
+ "what should",
+ "serve",
+ "dinner",
+ "homegrown",
+ "ingredients",
+ "battery life",
+ "getting around",
+ "偏好",
+ "喜欢",
+ "不喜欢",
+ "推荐",
+ "建议",
+ "适合我",
+ "根据我",
+ "我的情况",
+ "我的配置",
+ "我的目标",
+ "我的约束",
+ "画像",
+ "避免",
+)
+
+
+def _clean_text(value: Any) -> str:
+ return re.sub(r"\s+", " ", str(value or "").strip())
+
+
+def _normalize(value: Any) -> str:
+ return _clean_text(value).lower()
+
+
+def _tokens(value: Any) -> list[str]:
+ text = _normalize(value)
+ english = re.findall(r"[a-z0-9_]+", text)
+ cjk = [char for char in text if "\u4e00" <= char <= "\u9fff"]
+ return _dedupe([*english, *cjk])
+
+
+def _slug(value: Any, *, fallback: str = "general") -> str:
+ text = _normalize(value)
+ parts = re.findall(r"[a-z0-9]+|[\u4e00-\u9fff]", text)
+ slug = "_".join(parts[:10]).strip("_")
+ return slug or fallback
+
+
+def _dedupe(items: Iterable[Any], *, max_items: int | None = None) -> list[str]:
+ values: list[str] = []
+ seen: set[str] = set()
+ for item in items:
+ text = _clean_text(item)
+ if not text:
+ continue
+ key = _normalize(text)
+ if key in seen:
+ continue
+ seen.add(key)
+ values.append(text)
+ if max_items is not None and len(values) >= max_items:
+ break
+ return values
+
+
+def _first_nonempty(*values: Any) -> str:
+ for value in values:
+ text = _clean_text(value)
+ if text:
+ return text
+ return ""
+
+
+def _bounded_union(*groups: Iterable[Any], max_items: int) -> list[str]:
+ return _dedupe([item for group in groups for item in group], max_items=max_items)
+
+
+def _bounded_int_union(*groups: Iterable[Any], max_items: int) -> list[int]:
+ values: list[int] = []
+ seen: set[int] = set()
+ for group in groups:
+ for item in group:
+ try:
+ value = int(item)
+ except Exception:
+ continue
+ if value in seen:
+ continue
+ seen.add(value)
+ values.append(value)
+ if len(values) >= max_items:
+ return sorted(values)
+ return sorted(values)
+
+
+def is_profile_layer_record(*, category: Any, source_kind: Any = "", semantic_slot: Any = "", metadata: Mapping[str, Any] | None = None) -> bool:
+ data = dict(metadata or {})
+ if bool(data.get("profile_layer")):
+ return True
+ category_text = _normalize(category)
+ slot_text = _normalize(semantic_slot or data.get("semantic_slot", ""))
+ source_text = _normalize(source_kind)
+ if category_text in PROFILE_CATEGORIES:
+ return True
+ if slot_text in PROFILE_SEMANTIC_SLOTS or slot_text.startswith("profile_"):
+ return True
+ return source_text in {
+ "public_dialog_profile",
+ "public_dialog_profile_cluster",
+ "public_dialog_preference",
+ "public_dialog_goal",
+ "public_dialog_constraint",
+ }
+
+
+def infer_profile_type(*, category: Any, semantic_slot: Any = "", relation: Any = "", value: Any = "", metadata: Mapping[str, Any] | None = None) -> str:
+ data = dict(metadata or {})
+ explicit = _normalize(data.get("profile_type", ""))
+ if explicit in PROFILE_TYPES:
+ return explicit
+ category_text = _normalize(category)
+ combined = _normalize(f"{semantic_slot} {relation} {value}")
+ if category_text == "preference" or any(marker in combined for marker in ("prefer", "like", "default", "偏好", "喜欢", "默认")):
+ return "preference"
+ if category_text == "constraint" or any(marker in combined for marker in ("must", "cannot", "forbid", "constraint", "必须", "不能", "约束")):
+ return "constraint"
+ if category_text == "goal" or any(marker in combined for marker in ("goal", "target", "objective", "目标")):
+ return "goal"
+ if any(marker in combined for marker in ("avoid", "dislike", "do not", "don't", "避免", "不喜欢", "不要")):
+ return "avoid"
+ if any(marker in combined for marker in ("setup", "environment", "workflow", "current", "配置", "环境", "流程", "当前")):
+ return "setup"
+ return "usage_context"
+
+
+def infer_profile_domain(
+ *,
+ category: Any,
+ semantic_slot: Any = "",
+ slot_key: Any = "",
+ anchors: Sequence[Any] = (),
+ value: Any = "",
+ metadata: Mapping[str, Any] | None = None,
+) -> tuple[str, str]:
+ data = dict(metadata or {})
+ explicit = _first_nonempty(data.get("profile_domain", ""), data.get("profile_domain_label", ""), data.get("domain", ""))
+ if explicit:
+ return _slug(explicit), explicit
+ subject = _first_nonempty(data.get("subject", ""), data.get("extracted_subject", ""))
+ if subject:
+ return _slug(subject), subject
+ slot = _normalize(semantic_slot)
+ if slot and slot not in {"profile", "preference", "goal", "constraint", "event", "fact", "status"}:
+ return _slug(slot), slot.replace("_", " ")
+ for anchor in anchors:
+ anchor_text = _clean_text(anchor)
+ if not anchor_text:
+ continue
+ if re.search(r"^\d{4}|\d{1,2}\s+[A-Za-z]+|^[A-Z][a-z]+$", anchor_text):
+ continue
+ return _slug(anchor_text), anchor_text
+ slot_key_text = _clean_text(slot_key)
+ if slot_key_text:
+ tail = slot_key_text.split(".")[-1].replace("_", " ")
+ if tail:
+ return _slug(tail), tail
+ value_tokens = _tokens(value)
+ if value_tokens:
+ label = " ".join(value_tokens[:4])
+ return _slug(label), label
+ return _slug(category), _normalize(category) or "general"
+
+
+def profile_candidate_metadata(
+ *,
+ category: Any,
+ semantic_slot: Any = "",
+ relation: Any = "",
+ value: Any = "",
+ source_span: Any = "",
+ slot_key: Any = "",
+ anchors: Sequence[Any] = (),
+ subject: Any = "",
+ subject_signature: Any = "",
+ proposal: Mapping[str, Any] | None = None,
+) -> dict[str, Any]:
+ proposal_data = dict(proposal or {})
+ base_metadata = {
+ "profile_type": proposal_data.get("profile_type", ""),
+ "profile_domain": proposal_data.get("profile_domain", ""),
+ "profile_domain_label": proposal_data.get("profile_domain_label", ""),
+ "subject": subject,
+ "extracted_subject": proposal_data.get("extracted_subject", ""),
+ }
+ if not is_profile_layer_record(category=category, semantic_slot=semantic_slot, metadata=base_metadata):
+ return {}
+ profile_type = infer_profile_type(
+ category=category,
+ semantic_slot=semantic_slot,
+ relation=relation,
+ value=value or source_span,
+ metadata=proposal_data,
+ )
+ domain, domain_label = infer_profile_domain(
+ category=category,
+ semantic_slot=semantic_slot,
+ slot_key=slot_key,
+ anchors=anchors,
+ value=value or source_span,
+ metadata={**proposal_data, "subject": subject},
+ )
+ normalized_subject_signature = _slug(subject_signature or subject or domain, fallback=domain)
+ route_terms = _dedupe(
+ [
+ profile_type,
+ domain_label,
+ semantic_slot,
+ subject,
+ *list(anchors or []),
+ ],
+ max_items=12,
+ )
+ return {
+ "profile_layer": True,
+ "profile_candidate_status": _clean_text(proposal_data.get("profile_candidate_status", "")) or "writer_candidate",
+ "profile_consolidation_stage": _clean_text(proposal_data.get("profile_consolidation_stage", "")) or "pre_consolidation",
+ "profile_type": profile_type,
+ "profile_domain": domain,
+ "profile_domain_label": domain_label,
+ "profile_subject_signature": normalized_subject_signature,
+ "profile_support_key": f"{profile_type}:{domain}:{normalized_subject_signature}",
+ "profile_route_terms": route_terms,
+ }
+
+
+def profile_aggregate_slot_key(metadata: Mapping[str, Any]) -> str:
+ data = dict(metadata or {})
+ support_key = _clean_text(data.get("profile_support_key", ""))
+ if support_key:
+ return f"tmcra.profile.aggregate.{_slug(support_key)}"
+ profile_type = _normalize(data.get("profile_type", "")) or "usage_context"
+ domain = _normalize(data.get("profile_domain", "")) or _slug(data.get("profile_domain_label", "general"))
+ subject = _normalize(data.get("profile_subject_signature", "")) or domain
+ return f"tmcra.profile.aggregate.{_slug(f'{profile_type}:{domain}:{subject}')}"
+
+
+def profile_aggregate_value(
+ *,
+ profile_type: Any,
+ domain_label: Any,
+ support_values: Sequence[Any],
+) -> str:
+ typed = _clean_text(profile_type) or "usage_context"
+ domain = _clean_text(domain_label) or "general"
+ values = _dedupe(support_values, max_items=5)
+ if not values:
+ return f"User {typed} profile for {domain}."
+ return f"User {typed} profile for {domain}: " + "; ".join(values)
+
+
+def _profile_output_kind(profile_types: Sequence[Any]) -> str:
+ normalized = {_normalize(item) for item in profile_types if _clean_text(item)}
+ if "constraint" in normalized:
+ return "constraint_profile"
+ if "goal" in normalized:
+ return "goal_profile"
+ if normalized.intersection({"preference", "avoid"}):
+ return "preference_profile"
+ if "setup" in normalized:
+ return "setup_profile"
+ return "usage_context_profile"
+
+
+def _profile_update_policy(profile_types: Sequence[Any]) -> str:
+ normalized = {_normalize(item) for item in profile_types if _clean_text(item)}
+ if "constraint" in normalized:
+ return "preserve_until_explicitly_changed"
+ if normalized.intersection({"preference", "avoid", "goal", "setup"}):
+ return "update_on_newer_user_evidence"
+ return "background_context"
+
+
+def _profile_memory_type(profile_types: Sequence[Any]) -> str:
+ normalized = {_normalize(item) for item in profile_types if _clean_text(item)}
+ if "constraint" in normalized:
+ return "hard_constraint"
+ if normalized.intersection({"preference", "avoid"}):
+ return "durable_preference"
+ return "profile_context"
+
+
+def profile_summary(
+ *,
+ profile_types: Sequence[Any],
+ domain_label: Any,
+ support_values: Sequence[Any],
+ stage: str,
+) -> str:
+ kind = _profile_output_kind(profile_types)
+ domain = _clean_text(domain_label) or "general"
+ values = _dedupe(support_values, max_items=6 if stage == "cluster" else 4)
+ prefix = {
+ "constraint_profile": "User constraint profile",
+ "goal_profile": "User goal profile",
+ "preference_profile": "User preference profile",
+ "setup_profile": "User setup profile",
+ "usage_context_profile": "User usage-context profile",
+ }.get(kind, "User profile")
+ if not values:
+ return f"{prefix} for {domain}."
+ return f"{prefix} for {domain}: " + "; ".join(values)
+
+
+def build_profile_aggregate_metadata(
+ *,
+ support_record_id: Any,
+ support_turn_index: int,
+ support_value: Any,
+ support_anchors: Sequence[Any],
+ support_metadata: Mapping[str, Any],
+ existing_metadata: Mapping[str, Any] | None = None,
+) -> dict[str, Any]:
+ support = dict(support_metadata or {})
+ existing = dict(existing_metadata or {})
+ profile_type = _normalize(support.get("profile_type", "")) or "usage_context"
+ domain = _normalize(support.get("profile_domain", "")) or _slug(support.get("profile_domain_label", "general"))
+ domain_label = _clean_text(support.get("profile_domain_label", "")) or domain.replace("_", " ")
+ subject_signature = _clean_text(support.get("profile_subject_signature", "")) or domain
+ support_ids = _bounded_union(existing.get("profile_support_ids", []) or [], [support_record_id], max_items=64)
+ support_turns = _bounded_int_union(existing.get("profile_support_turns", []) or [], [support_turn_index], max_items=64)
+ support_values = _bounded_union(existing.get("profile_support_values", []) or [], [support_value], max_items=12)
+ support_route_terms = _bounded_union(
+ existing.get("profile_route_terms", []) or [],
+ support.get("profile_route_terms", []) or [],
+ support_anchors,
+ max_items=24,
+ )
+ profile_types = [profile_type]
+ output_kind = _profile_output_kind(profile_types)
+ memory_type = _profile_memory_type(profile_types)
+ summary = profile_summary(
+ profile_types=profile_types,
+ domain_label=domain_label,
+ support_values=support_values,
+ stage="aggregate",
+ )
+ value = profile_aggregate_value(
+ profile_type=profile_type,
+ domain_label=domain_label,
+ support_values=support_values,
+ )
+ return {
+ **existing,
+ "profile_layer": True,
+ "profile_candidate_status": "consolidated",
+ "profile_consolidation_stage": "aggregate",
+ "profile_consolidator_version": PROFILE_CONSOLIDATOR_VERSION,
+ "profile_aggregate_node": True,
+ "profile_type": profile_type,
+ "profile_domain": domain,
+ "profile_domain_label": domain_label,
+ "profile_subject_signature": subject_signature,
+ "profile_support_key": f"{profile_type}:{domain}:{subject_signature}",
+ "profile_support_ids": support_ids,
+ "profile_support_turns": support_turns,
+ "profile_support_values": support_values,
+ "profile_support_count": len(support_ids),
+ "profile_route_terms": support_route_terms,
+ "profile_value": value,
+ "profile_summary": summary,
+ "profile_output_kind": output_kind,
+ "profile_update_policy": _profile_update_policy(profile_types),
+ "profile_conflict_policy": "latest_active_support_only",
+ "profile_evidence_count": len(support_ids),
+ "memory_type": memory_type,
+ "durable_memory_type": memory_type,
+ "memory_chain_depth_layer": "profile",
+ "depth_layer": "profile",
+ }
+
+
+_PROFILE_CLUSTER_STOPWORDS = {
+ "user",
+ "profile",
+ "preference",
+ "preferences",
+ "constraint",
+ "constraints",
+ "goal",
+ "goals",
+ "avoid",
+ "usage",
+ "context",
+ "general",
+ "default",
+ "should",
+ "would",
+ "could",
+ "want",
+ "wants",
+ "need",
+ "needs",
+ "when",
+ "then",
+ "than",
+ "with",
+ "that",
+ "this",
+ "from",
+ "into",
+ "instead",
+}
+
+
+def profile_cluster_tokens(*values: Any) -> list[str]:
+ tokens: list[str] = []
+ for value in values:
+ if isinstance(value, (list, tuple, set)):
+ for item in value:
+ tokens.extend(_tokens(item))
+ else:
+ tokens.extend(_tokens(value))
+ filtered = [
+ token
+ for token in tokens
+ if token
+ and token not in _PROFILE_CLUSTER_STOPWORDS
+ and not token.isdigit()
+ and (len(token) > 2 or any("\u4e00" <= char <= "\u9fff" for char in token))
+ ]
+ return _dedupe(filtered, max_items=32)
+
+
+def profile_cluster_similarity(source_metadata: Mapping[str, Any], target_metadata: Mapping[str, Any]) -> float:
+ source = dict(source_metadata or {})
+ target = dict(target_metadata or {})
+ source_tokens = set(
+ profile_cluster_tokens(
+ source.get("profile_domain_label", ""),
+ source.get("profile_domain", ""),
+ source.get("profile_route_terms", []) or [],
+ source.get("profile_support_values", []) or [],
+ source.get("profile_value", ""),
+ )
+ )
+ target_tokens = set(
+ profile_cluster_tokens(
+ target.get("profile_domain_label", ""),
+ target.get("profile_domain", ""),
+ target.get("profile_route_terms", []) or [],
+ target.get("profile_support_values", []) or [],
+ target.get("profile_value", ""),
+ )
+ )
+ if not source_tokens or not target_tokens:
+ return 0.0
+ overlap = len(source_tokens & target_tokens) / max(1, len(source_tokens | target_tokens))
+ containment = len(source_tokens & target_tokens) / max(1, min(len(source_tokens), len(target_tokens)))
+ type_bonus = 0.06 if _normalize(source.get("profile_type", "")) == _normalize(target.get("profile_type", "")) else 0.0
+ return round(min(1.0, (0.62 * overlap) + (0.38 * containment) + type_bonus), 6)
+
+
+def profile_cluster_slot_key(metadata: Mapping[str, Any]) -> str:
+ data = dict(metadata or {})
+ tokens = profile_cluster_tokens(
+ data.get("profile_domain_label", ""),
+ data.get("profile_domain", ""),
+ data.get("profile_route_terms", []) or [],
+ data.get("profile_support_values", []) or [],
+ data.get("profile_value", ""),
+ )
+ if tokens:
+ return f"tmcra.profile.cluster.{_slug('_'.join(tokens[:5]))}"
+ support_profiles = data.get("profile_support_profile_ids", []) or []
+ support_seed = _clean_text(support_profiles[0] if support_profiles else "")
+ seed = support_seed or _clean_text(data.get("profile_support_key", "general"))
+ return f"tmcra.profile.cluster.{_slug(seed)}"
+
+
+def profile_cluster_value(*, support_values: Sequence[Any]) -> str:
+ values = _dedupe(support_values, max_items=8)
+ if not values:
+ return "User profile cluster."
+ return "User profile cluster: " + "; ".join(values)
+
+
+def build_profile_cluster_metadata(
+ *,
+ support_profile_id: Any,
+ support_metadata: Mapping[str, Any],
+ existing_metadata: Mapping[str, Any] | None = None,
+) -> dict[str, Any]:
+ support = dict(support_metadata or {})
+ existing = dict(existing_metadata or {})
+ support_profile_ids = _bounded_union(
+ existing.get("profile_support_profile_ids", []) or [],
+ [support_profile_id],
+ max_items=32,
+ )
+ support_ids = _bounded_union(
+ existing.get("profile_support_ids", []) or [],
+ support.get("profile_support_ids", []) or [],
+ max_items=96,
+ )
+ support_turns = _bounded_int_union(
+ existing.get("profile_support_turns", []) or [],
+ support.get("profile_support_turns", []) or [],
+ max_items=96,
+ )
+ raw_support_values = support.get("profile_support_values", []) or []
+ support_values = _bounded_union(
+ existing.get("profile_support_values", []) or [],
+ raw_support_values,
+ [] if raw_support_values else [support.get("profile_value", "")],
+ max_items=16,
+ )
+ route_terms = _bounded_union(
+ existing.get("profile_route_terms", []) or [],
+ existing.get("profile_cluster_route_terms", []) or [],
+ support.get("profile_route_terms", []) or [],
+ profile_cluster_tokens(
+ support.get("profile_domain_label", ""),
+ support.get("profile_domain", ""),
+ support.get("profile_value", ""),
+ support.get("profile_support_values", []) or [],
+ ),
+ max_items=32,
+ )
+ profile_types = _bounded_union(
+ existing.get("profile_cluster_types", []) or [],
+ [support.get("profile_type", "")],
+ max_items=8,
+ )
+ domains = _bounded_union(
+ existing.get("profile_cluster_domains", []) or [],
+ [support.get("profile_domain", ""), support.get("profile_domain_label", "")],
+ max_items=16,
+ )
+ primary_type = _clean_text(profile_types[0] if profile_types else support.get("profile_type", "")) or "usage_context"
+ primary_domain = _clean_text(domains[0] if domains else support.get("profile_domain", "")) or "general"
+ output_kind = _profile_output_kind(profile_types)
+ memory_type = _profile_memory_type(profile_types)
+ summary = profile_summary(
+ profile_types=profile_types,
+ domain_label=primary_domain,
+ support_values=support_values,
+ stage="cluster",
+ )
+ value = profile_cluster_value(support_values=support_values)
+ return {
+ **existing,
+ "profile_layer": True,
+ "profile_candidate_status": "consolidated",
+ "profile_consolidation_stage": "cluster",
+ "profile_consolidator_version": PROFILE_CONSOLIDATOR_VERSION,
+ "profile_cluster_node": True,
+ "profile_type": primary_type,
+ "profile_domain": _slug(primary_domain),
+ "profile_domain_label": primary_domain.replace("_", " "),
+ "profile_subject_signature": _clean_text(existing.get("profile_subject_signature", "")) or _slug(primary_domain),
+ "profile_support_key": _clean_text(existing.get("profile_support_key", "")) or f"cluster:{_slug(primary_domain)}",
+ "profile_support_profile_ids": support_profile_ids,
+ "profile_support_ids": support_ids,
+ "profile_support_turns": support_turns,
+ "profile_support_values": support_values,
+ "profile_support_count": len(support_ids),
+ "profile_cluster_profile_count": len(support_profile_ids),
+ "profile_cluster_types": profile_types,
+ "profile_cluster_domains": domains,
+ "profile_cluster_route_terms": route_terms,
+ "profile_route_terms": route_terms,
+ "profile_value": value,
+ "profile_summary": summary,
+ "profile_output_kind": output_kind,
+ "profile_update_policy": _profile_update_policy(profile_types),
+ "profile_conflict_policy": "latest_active_support_only",
+ "profile_evidence_count": len(support_ids),
+ "memory_type": memory_type,
+ "durable_memory_type": memory_type,
+ "memory_chain_depth_layer": "profile",
+ "depth_layer": "profile",
+ }
+
+
+def infer_profile_query_intent(query: Any) -> dict[str, Any]:
+ query_text = _clean_text(query)
+ lowered = _normalize(query_text)
+ enabled = any(marker in lowered for marker in _PROFILE_QUERY_MARKERS)
+ types: list[str] = []
+ if any(marker in lowered for marker in ("prefer", "preference", "like", "dislike", "偏好", "喜欢", "不喜欢")):
+ types.append("preference")
+ if any(marker in lowered for marker in ("constraint", "must", "cannot", "policy", "约束", "限制", "必须", "不能")):
+ types.append("constraint")
+ if any(marker in lowered for marker in ("goal", "target", "objective", "目标")):
+ types.append("goal")
+ if any(marker in lowered for marker in ("avoid", "dislike", "避免", "不喜欢")):
+ types.append("avoid")
+ if any(marker in lowered for marker in ("setup", "profile", "current", "occupation", "role", "job", "career", "background", "identity", "experience", "配置", "画像", "当前")):
+ types.append("setup")
+ if enabled and any(
+ marker in lowered
+ for marker in (
+ "recommend",
+ "suggest",
+ "advice",
+ "tips",
+ "trouble with",
+ "struggling with",
+ "what do you think",
+ "should i",
+ "what should",
+ "serve",
+ "dinner",
+ "battery life",
+ "getting around",
+ "resources",
+ "learn more",
+ "推荐",
+ "建议",
+ )
+ ):
+ types.append("usage_context")
+ if not types and enabled:
+ types.append("usage_context")
+ return {
+ "enabled": enabled,
+ "types": _dedupe(types),
+ "tokens": _tokens(query_text),
+ }
+
+
+def profile_query_score_delta(
+ *,
+ query: Any,
+ query_tokens: set[str],
+ category: Any,
+ source_kind: Any,
+ semantic_slot: Any,
+ value: Any,
+ anchors: Sequence[Any],
+ metadata: Mapping[str, Any] | None = None,
+) -> tuple[float, str]:
+ data = dict(metadata or {})
+ intent = infer_profile_query_intent(query)
+ if not intent.get("enabled"):
+ return 0.0, ""
+ if not is_profile_layer_record(category=category, source_kind=source_kind, semantic_slot=semantic_slot, metadata=data):
+ return 0.0, ""
+ profile_type = _normalize(data.get("profile_type", "")) or infer_profile_type(
+ category=category,
+ semantic_slot=semantic_slot,
+ value=value,
+ metadata=data,
+ )
+ domain = _normalize(data.get("profile_domain_label", "") or data.get("profile_domain", ""))
+ route_terms = " ".join(str(item) for item in data.get("profile_route_terms", []) or [])
+ record_tokens = set(_tokens(f"{profile_type} {domain} {semantic_slot} {value} {' '.join(str(item) for item in anchors)} {route_terms}"))
+ overlap = len(set(query_tokens) & record_tokens) / max(1, len(set(query_tokens) | record_tokens)) if query_tokens or record_tokens else 0.0
+ type_match = profile_type in set(intent.get("types", []) or [])
+ delta = 0.20 + (0.16 if type_match else 0.0) + (0.18 * overlap)
+ if data.get("profile_candidate_status") == "consolidated":
+ delta += 0.06
+ if data.get("profile_cluster_node") or data.get("profile_consolidation_stage") == "cluster":
+ delta += 0.08
+ return round(min(delta, 0.54), 6), "profile_route"
+
+
+def profile_edge_score(source_metadata: Mapping[str, Any], target_metadata: Mapping[str, Any], *, source_value: Any = "", target_value: Any = "") -> tuple[float, str]:
+ source = dict(source_metadata or {})
+ target = dict(target_metadata or {})
+ if not source.get("profile_layer") or not target.get("profile_layer"):
+ return 0.0, ""
+ source_key = _normalize(source.get("profile_support_key", ""))
+ target_key = _normalize(target.get("profile_support_key", ""))
+ source_domain = _normalize(source.get("profile_domain", ""))
+ target_domain = _normalize(target.get("profile_domain", ""))
+ source_type = _normalize(source.get("profile_type", ""))
+ target_type = _normalize(target.get("profile_type", ""))
+ source_tokens = set(_tokens(f"{source_value} {source.get('profile_domain_label', '')} {' '.join(source.get('profile_route_terms', []) or [])}"))
+ target_tokens = set(_tokens(f"{target_value} {target.get('profile_domain_label', '')} {' '.join(target.get('profile_route_terms', []) or [])}"))
+ overlap = len(source_tokens & target_tokens) / max(1, len(source_tokens | target_tokens)) if source_tokens or target_tokens else 0.0
+ if source_key and source_key == target_key:
+ return round(0.72 + (0.14 * overlap), 6), "profile_support"
+ if source_domain and source_domain == target_domain:
+ return round(0.58 + (0.12 if source_type == target_type else 0.04) + (0.12 * overlap), 6), "profile_tunnel"
+ if source_type and source_type == target_type and overlap >= 0.22:
+ return round(0.44 + (0.18 * overlap), 6), "profile_soft_tunnel"
+ return 0.0, ""
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/public_event_signature.py b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/public_event_signature.py
new file mode 100644
index 0000000000000000000000000000000000000000..2aba0a499b8ee106a01e1b07b64747864dca71d1
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/public_event_signature.py
@@ -0,0 +1,116 @@
+from __future__ import annotations
+
+import re
+import string
+from typing import Any, Iterable, List
+
+
+_PROFILE_SIGNATURE_SLOTS = {"identity", "research_topic", "education", "occupation"}
+
+
+def _safe_text(value: Any) -> str:
+ return str(value or "").strip()
+
+
+def _dedupe_texts(items: Iterable[Any]) -> List[str]:
+ results: List[str] = []
+ seen = set()
+ for item in items:
+ text = _safe_text(item)
+ if not text:
+ continue
+ key = text.lower()
+ if key in seen:
+ continue
+ seen.add(key)
+ results.append(text)
+ return results
+
+
+def _public_lemmatize_token(token: str) -> str:
+ lowered = _safe_text(token).lower().strip(string.punctuation)
+ if not lowered:
+ return ""
+ if lowered in {"m", "re", "ve", "ll", "d", "s", "t"}:
+ return ""
+ if lowered.endswith("ies") and len(lowered) > 4:
+ return f"{lowered[:-3]}y"
+ if lowered.endswith("ing") and len(lowered) > 5:
+ return lowered[:-3]
+ if lowered.endswith("ed") and len(lowered) > 4:
+ return lowered[:-2]
+ if lowered.endswith("s") and len(lowered) > 4 and not lowered.endswith("ss"):
+ return lowered[:-1]
+ return lowered
+
+
+def public_content_tokens(text: str, *, speaker: str = "", keep_time_tokens: bool = False) -> List[str]:
+ speaker_tokens = {
+ _public_lemmatize_token(part)
+ for part in re.findall(r"[A-Za-z0-9]+", _safe_text(speaker))
+ if _public_lemmatize_token(part)
+ }
+ tokens: List[str] = []
+ for raw in re.findall(r"[A-Za-z0-9\+]+", _safe_text(text)):
+ token = _public_lemmatize_token(raw)
+ if not token:
+ continue
+ if token in speaker_tokens:
+ continue
+ if not keep_time_tokens and token.isdigit():
+ continue
+ tokens.append(token)
+ return _dedupe_texts(tokens)
+
+
+def _signature_anchor_tokens(text: str, *, speaker: str = "", keep_time_tokens: bool = False, limit: int = 8) -> List[str]:
+ raw_tokens = re.findall(r"[A-Za-z0-9\+]+", _safe_text(text))
+ if not raw_tokens:
+ return []
+ speaker_tokens = {
+ _public_lemmatize_token(part)
+ for part in re.findall(r"[A-Za-z0-9]+", _safe_text(speaker))
+ if _public_lemmatize_token(part)
+ }
+ scored: dict[str, tuple[float, int]] = {}
+ total = len(raw_tokens)
+ for index, raw in enumerate(raw_tokens):
+ token = _public_lemmatize_token(raw)
+ if not token or token in speaker_tokens:
+ continue
+ if not keep_time_tokens and token.isdigit():
+ continue
+ score = min(3.0, len(token) / 2.5)
+ if raw[:1].isupper() and index > 0:
+ score += 1.0
+ if any(ch.isdigit() for ch in raw) or "+" in raw:
+ score += 0.6
+ if len(token) <= 2:
+ score -= 1.0
+ elif len(token) == 3:
+ score -= 0.25
+ if index >= max(1, total // 2):
+ score += 0.2
+ current = scored.get(token)
+ if current is None or score > current[0]:
+ scored[token] = (score, index)
+ if not scored:
+ return public_content_tokens(text, speaker=speaker, keep_time_tokens=keep_time_tokens)[:limit]
+ selected = sorted(scored.items(), key=lambda item: (-item[1][0], item[1][1], item[0]))[: max(1, limit)]
+ ordered = sorted(((token, pos) for token, (_, pos) in selected), key=lambda item: item[1])
+ return [token for token, _ in ordered]
+
+
+def compute_public_event_signature(text: str, *, speaker: str = "", semantic_slot: str = "") -> str:
+ tokens = _signature_anchor_tokens(text, speaker=speaker)
+ normalized_slot = _safe_text(semantic_slot)
+ if normalized_slot in _PROFILE_SIGNATURE_SLOTS:
+ tokens = _dedupe_texts(
+ [
+ *_signature_anchor_tokens(normalized_slot.replace("_", " "), keep_time_tokens=True, limit=4),
+ *tokens,
+ ]
+ )
+ if len(tokens) <= 8:
+ return " ".join(tokens)
+ return " ".join(tokens[:8])
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/semantic_memory_writer.py b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/semantic_memory_writer.py
new file mode 100644
index 0000000000000000000000000000000000000000..13570c93e8396fcb3f4b5d437214d825a40b359b
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/semantic_memory_writer.py
@@ -0,0 +1,3144 @@
+from __future__ import annotations
+
+from dataclasses import dataclass
+import json
+import os
+import re
+import urllib.error
+import urllib.request
+from typing import Any, Dict, Iterable, List, Mapping, Sequence
+
+from experiments.replacement.adapters.base import LLMProfile
+from experiments.replacement.profile_layer import profile_candidate_metadata
+from experiments.replacement.public_event_signature import compute_public_event_signature
+
+
+ALLOWED_WRITE_CATEGORIES = {
+ "fact",
+ "event",
+ "profile",
+ "time",
+ "status",
+ "preference",
+ "goal",
+ "constraint",
+ "stage_state",
+ "terminology",
+ "question",
+ "interaction_intent",
+}
+_FACET_TYPES = {"temporal", "numeric", "state", "entity", "evidence_role", "action", "role"}
+_FACET_TYPE_ALIASES = {
+ "time": "temporal",
+ "date": "temporal",
+ "duration": "temporal",
+ "temporal_anchor": "temporal",
+ "number": "numeric",
+ "quantity": "numeric",
+ "amount": "numeric",
+ "count": "numeric",
+ "numeric_quantity": "numeric",
+ "status": "state",
+ "state_status": "state",
+ "object": "entity",
+ "item": "entity",
+ "target_entity": "entity",
+ "role": "evidence_role",
+ "action_unit": "action",
+ "action": "action",
+ "participation": "role",
+ "leadership": "role",
+ "role_participation_leadership": "role",
+}
+_FACET_CATEGORY_BY_TYPE = {
+ "temporal": "time",
+ "numeric": "fact",
+ "state": "status",
+ "entity": "fact",
+ "evidence_role": "fact",
+ "action": "event",
+ "role": "fact",
+}
+_UNIT_WRITER_ACTION_RE = re.compile(
+ r"\b("
+ r"buy|bought|purchase|purchased|order|ordered|return|returned|exchange|exchanged|"
+ r"pick(?:ed|ing)?\s+up|receive|received|get|got|arrive|arrived|"
+ r"start(?:ed|ing)?|finish(?:ed|ing)?|complete(?:d|ing)?|work(?:ed|ing)?\s+on|"
+ r"plan(?:ned|ning)?\s+to\s+[a-z][a-z-]+|"
+ r"lead|led|leading|manage|managed|organize|organized|"
+ r"attend(?:ed)?|participate(?:d)?|join(?:ed)?|volunteer(?:ed)?|"
+ r"build|built|create|created|submit|submitted|redeem|redeemed|"
+ r"spend|spent|pay|paid|borrow|borrowed|rent|rented"
+ r")\b",
+ re.IGNORECASE,
+)
+_UNIT_WRITER_NUMERIC_RE = re.compile(
+ r"(?:(?:[$€£¥]\s*)?\d+(?:[.,]\d+)?\s*(?:%|percent|cents?|dollars?|usd|eur|gbp|"
+ r"days?|weeks?|months?|years?|hours?|minutes?|times?|items?|people|engineers?|members?|"
+ r"tickets?|points?|miles?|km|kilometers?|meters?|lbs?|pounds?|kg|grams?)?\b|"
+ r"\b\d+\s*/\s*\d+\b)",
+ re.IGNORECASE,
+)
+_UNIT_WRITER_NUMERIC_RE = re.compile(
+ r"(?:\b\d+\s*/\s*\d+\s*(?:scale|ratio|chance|share)?\b|"
+ r"(?:[$]\s*)?\d+(?:[.,]\d+)?\s*(?:%|percent|cents?|dollars?|usd|eur|gbp|"
+ r"days?|weeks?|months?|years?|hours?|minutes?|times?|items?|people|engineers?|members?|"
+ r"tickets?|points?|miles?|km|kilometers?|meters?|lbs?|pounds?|kg|grams?)\b)",
+ re.IGNORECASE,
+)
+_UNIT_WRITER_TEMPORAL_RE = re.compile(
+ r"\b("
+ r"today|tomorrow|yesterday|tonight|currently|now|recently|previously|later|earlier|"
+ r"monday|tuesday|wednesday|thursday|friday|saturday|sunday|"
+ r"january|february|march|april|may|june|july|august|september|october|november|december|"
+ r"jan\.?|feb\.?|mar\.?|apr\.?|jun\.?|jul\.?|aug\.?|sep\.?|sept\.?|oct\.?|nov\.?|dec\.?|"
+ r"last\s+(?:week|month|year|monday|tuesday|wednesday|thursday|friday|saturday|sunday)|"
+ r"next\s+(?:week|month|year|monday|tuesday|wednesday|thursday|friday|saturday|sunday)|"
+ r"\d{4}[-/]\d{1,2}[-/]\d{1,2}|\d{1,2}[-/]\d{1,2}(?:[-/]\d{2,4})?"
+ r")\b",
+ re.IGNORECASE,
+)
+_UNIT_WRITER_STATE_RE = re.compile(
+ r"\b("
+ r"currently|still|already|pending|planned|planning|need(?:s|ed)?\s+to|"
+ r"not\s+yet|waiting\s+for|completed|finished|done|cancelled|canceled|"
+ r"too\s+\w+|larger|smaller|newer|older|current|previous|latest|old|"
+ r"single|single\s+parent|married|divorced|widowed|partner|relationship|"
+ r"identity|member|belongs|home\s+country|country|origin|roots|moved\s+from"
+ r")\b",
+ re.IGNORECASE,
+)
+_UNIT_WRITER_ROLE_RE = re.compile(
+ r"\b("
+ r"lead|led|leading|leader|manager|managed|organizer|organized|"
+ r"participant|participated|volunteer|volunteered|attendee|attended|member|team"
+ r")\b",
+ re.IGNORECASE,
+)
+_PROFILE_SLOTS = {"identity", "research_topic", "education", "occupation"}
+_PROFILE_DIRECT_CATEGORIES = {"profile", "preference", "goal", "constraint"}
+_PROFILE_SETUP_SLOT_MARKERS = {
+ "brand",
+ "camera",
+ "device",
+ "equipment",
+ "gear",
+ "model",
+ "phone",
+ "setup",
+ "software",
+ "stack",
+ "tool",
+ "workflow",
+}
+_PROFILE_INTENT_SLOT_MARKERS = {
+ "avoid",
+ "constraint",
+ "goal",
+ "habit",
+ "identity",
+ "preference",
+ "profile",
+ "style",
+ "usage",
+ *_PROFILE_SETUP_SLOT_MARKERS,
+}
+_WRITE_CATEGORY_ALIASES = {
+ "identity": "profile",
+ "research_topic": "profile",
+ "education": "profile",
+ "occupation": "profile",
+ "event_time": "time",
+ "user_question": "question",
+}
+_SECRET_RE = re.compile(
+ r"\b(password|passcode|secret|private[_ -]?key|bearer\s+[a-z0-9._-]+|sk-[a-z0-9]{12,}|"
+ r"api[_ -]?key\s*(?:is|=|:)\s*[a-z0-9._-]{8,})\b",
+ flags=re.IGNORECASE,
+)
+_TOKEN_RE = re.compile(r"[a-z0-9_]+", flags=re.IGNORECASE)
+_SPEECH_ACT_MEMORY_RE = re.compile(
+ r"^\s*(asked|asks|asking)\b|"
+ r"^\s*(requested|requests|requesting)\s+(the\s+)?(assistant|ai|model|you)\b|"
+ r"^\s*(wanted|wants|wanting)\s+(the\s+)?(assistant|ai|model|you)\s+to\b",
+ flags=re.IGNORECASE,
+)
+_QUESTION_ONLY_RE = re.compile(
+ r"^\s*(quick\s+memory\s+check|what|which|who|where|when|how|why|do\s+you|can\s+you|could\s+you|would\s+you|please\s+remind|remind\s+me)\b",
+ flags=re.IGNORECASE,
+)
+_ASSERTIVE_MEMORY_CUE_RE = re.compile(
+ r"\b(remember\s+that|for\s+context|small\s+update|update\s+this|is\s+now|going\s+forward|my\s+.+?\s+is\s+[^?]+)\b",
+ flags=re.IGNORECASE,
+)
+_FIRST_PERSON_MEMORY_ASSERTION_RE = re.compile(
+ r"\b(?:i|we)\s+(?:want|prefer|need|avoid|like|dislike|use|usually|always|keep|choose|work|value|care|plan|intend|have|am|do)\b|"
+ r"\b(?:my|our)\s+[a-z0-9][a-z0-9\s'/_-]{0,80}\s+(?:is|are|was|were|should|needs?|uses?|prefers?|avoids?|has|have)\b|"
+ r"\bwhat\s+(?:i|we)\s+(?:want|prefer|need|avoid|like|use|value)\s+is\b",
+ flags=re.IGNORECASE,
+)
+_PROFILE_VALUE_MARKERS = re.compile(
+ r"\b(user\s+)?(?:uses?|prefers?|likes?|dislikes?|avoids?|needs?|wants?|plans?|values?|cares?|has|is|am|works?\s+with|usually|always)\b|"
+ r"\b(?:current|setup|preference|goal|constraint|habit|style|brand|device|equipment|workflow)\b",
+ flags=re.IGNORECASE,
+)
+_CJK_MEMORY_ASSERTION_RE = re.compile(
+ r"(我|我们).{0,18}(想要|希望|偏好|喜欢|不喜欢|避免|需要|通常|总是|选择|使用|重视|计划|打算|有|是)|"
+ r"(我的|我们的).{0,32}(是|需要|使用|偏好|避免|有)",
+)
+_TRANSIENT_ASSISTANT_DIRECTIVE_RE = re.compile(
+ r"\b(answer|respond|reply|summari[sz]e|translate|compare|explain|based\s+only|only\s+use|do\s+not\s+(mention|reveal|include|use))\b|"
+ r"\buse\s+(?:this|that|the|these|those|memory|context|retrieval|debug|tool|api|file|document|prompt|evidence|format|style)\b|"
+ r"(请|帮我|你先|先别|不要|别).{0,18}(回答|回复|总结|翻译|解释|比较|复述|展开|写|生成|提到|泄露)|"
+ r"(只|仅|只能|不要|别).{0,18}(基于|根据|使用|提到|绕远路|写总结)|"
+ r"(长期记忆上下文|拿到的.*上下文|memory\s+context|retrieval|debug\s+metadata)",
+ flags=re.IGNORECASE,
+)
+_CJK_QUESTION_MARKERS = (
+ "什么",
+ "吗",
+ "有没有",
+ "能不能",
+ "是不是",
+ "是否",
+ "怎么",
+ "如何",
+ "为什么",
+ "哪",
+ "记得",
+ "还记得",
+)
+
+
+def _safe_text(value: Any) -> str:
+ return " ".join(str(value or "").replace("\r", " ").replace("\n", " ").replace("\t", " ").split()).strip()
+
+
+def _slug(value: Any) -> str:
+ return re.sub(r"[^a-z0-9]+", "_", _safe_text(value).lower()).strip("_") or "item"
+
+
+def _tokens(value: Any) -> List[str]:
+ return [item.lower() for item in _TOKEN_RE.findall(_safe_text(value)) if len(item) >= 2]
+
+
+def _dedupe_texts(items: Iterable[Any], *, max_items: int | None = None) -> List[str]:
+ values: List[str] = []
+ seen = set()
+ for item in items:
+ text = _safe_text(item)
+ if not text:
+ continue
+ key = text.lower()
+ if key in seen:
+ continue
+ seen.add(key)
+ values.append(text)
+ if max_items is not None and len(values) >= max_items:
+ break
+ return values
+
+
+def _facet_type(value: Any) -> str:
+ raw = _safe_text(value).lower().replace("-", "_")
+ raw = _FACET_TYPE_ALIASES.get(raw, raw)
+ return raw if raw in _FACET_TYPES else ""
+
+
+def _facet_role(facet: Mapping[str, Any], *, fallback: str = "") -> str:
+ for key in ("role", "facet_role", "evidence_role", "temporal_role", "state_role", "numeric_role"):
+ value = _safe_text(facet.get(key, ""))
+ if value:
+ return value[:80]
+ return _safe_text(fallback)[:80]
+
+
+def _facet_source_span(facet: Mapping[str, Any], *, fallback: str = "") -> str:
+ for key in ("source_span", "span", "evidence_span", "quantity_span", "time_expression_span"):
+ value = _safe_text(facet.get(key, ""))
+ if value:
+ return value[:240]
+ return _safe_text(fallback)[:240]
+
+
+def _facet_value(facet: Mapping[str, Any], *, facet_type: str) -> str:
+ for key in ("value", "normalized_value", "text", "resolved_value", "resolved_time_value", "time_display_value"):
+ value = _safe_text(facet.get(key, ""))
+ if value:
+ return value[:180]
+ if facet_type == "numeric":
+ numeric_value = _safe_text(facet.get("numeric_value", facet.get("number", facet.get("amount", ""))))
+ unit = _safe_text(facet.get("unit", facet.get("numeric_unit", "")))
+ if numeric_value and unit:
+ return f"{numeric_value} {unit}"[:180]
+ if numeric_value:
+ return numeric_value[:180]
+ if facet_type == "temporal":
+ value = _safe_text(facet.get("time_expression_span", facet.get("resolved_date", "")))
+ if value:
+ return value[:180]
+ if facet_type == "state":
+ value = _safe_text(facet.get("target_status", facet.get("state", "")))
+ if value:
+ return value[:180]
+ return ""
+
+
+def _writer_facets_from_proposal(
+ proposal: Mapping[str, Any],
+ *,
+ source_span: str,
+ subject: str = "",
+ semantic_slot: str = "",
+) -> List[Dict[str, Any]]:
+ raw_facets = proposal.get("facets", proposal.get("memory_facets", proposal.get("attribute_facets", [])))
+ if isinstance(raw_facets, Mapping):
+ raw_items: List[Any] = [raw_facets]
+ elif isinstance(raw_facets, list):
+ raw_items = list(raw_facets)
+ else:
+ raw_items = []
+ facets: List[Dict[str, Any]] = [dict(item) for item in raw_items if isinstance(item, Mapping)]
+ explicit_facet_types = {
+ _facet_type(item.get("type", item.get("facet_type", item.get("category", ""))))
+ for item in facets
+ if isinstance(item, Mapping)
+ }
+
+ time_span = _safe_text(proposal.get("time_expression_span", ""))
+ resolved_time = _safe_text(proposal.get("resolved_time_value", "")) or _safe_text(proposal.get("time_display_value", ""))
+ resolved_date = _safe_text(proposal.get("resolved_date", ""))
+ if "temporal" not in explicit_facet_types and (time_span or resolved_time or resolved_date):
+ facets.append(
+ {
+ "type": "temporal",
+ "role": _safe_text(proposal.get("target_status", "")) or _safe_text(semantic_slot) or "time",
+ "value": resolved_time or resolved_date or time_span,
+ "source_span": time_span or source_span,
+ "resolved_time_value": resolved_time,
+ "resolved_date": resolved_date,
+ "time_granularity": _safe_text(proposal.get("time_granularity", "")),
+ }
+ )
+
+ target_status = _safe_text(proposal.get("target_status", ""))
+ if "state" not in explicit_facet_types and target_status:
+ facets.append(
+ {
+ "type": "state",
+ "role": "target_status",
+ "value": target_status,
+ "source_span": source_span,
+ }
+ )
+
+ numeric_value = _safe_text(proposal.get("numeric_value", proposal.get("quantity_value", proposal.get("amount_value", ""))))
+ numeric_unit = _safe_text(proposal.get("numeric_unit", proposal.get("quantity_unit", proposal.get("amount_unit", ""))))
+ quantity_span = _safe_text(proposal.get("quantity_span", proposal.get("numeric_span", "")))
+ if "numeric" not in explicit_facet_types and (numeric_value or quantity_span):
+ facets.append(
+ {
+ "type": "numeric",
+ "role": _safe_text(proposal.get("numeric_role", "")) or _safe_text(semantic_slot) or "quantity",
+ "value": f"{numeric_value} {numeric_unit}".strip() or quantity_span,
+ "source_span": quantity_span or source_span,
+ "numeric_value": numeric_value,
+ "unit": numeric_unit,
+ }
+ )
+
+ if "entity" not in explicit_facet_types and subject:
+ facets.append(
+ {
+ "type": "entity",
+ "role": "subject",
+ "value": subject,
+ "source_span": subject if _safe_text(subject).lower() in _safe_text(source_span).lower() else source_span,
+ }
+ )
+
+ normalized: List[Dict[str, Any]] = []
+ seen = set()
+ for raw in facets:
+ facet_type = _facet_type(raw.get("type", raw.get("facet_type", raw.get("category", ""))))
+ if not facet_type:
+ continue
+ role = _facet_role(raw, fallback=semantic_slot or facet_type)
+ value = _facet_value(raw, facet_type=facet_type)
+ span = _facet_source_span(raw, fallback=source_span)
+ if not value or not span:
+ continue
+ key = (facet_type, role.lower(), value.lower(), span.lower())
+ if key in seen:
+ continue
+ seen.add(key)
+ normalized.append({**raw, "type": facet_type, "role": role, "value": value, "source_span": span})
+ return normalized[:6]
+
+
+def _build_event_facet_records(
+ *,
+ parent_record: Mapping[str, Any],
+ proposal: Mapping[str, Any],
+ common_metadata: Mapping[str, Any],
+ source_context: str,
+ current_turn_text: str,
+ speaker: str,
+ timestamp: str,
+ turn_index: int,
+ proposal_index: int,
+) -> List[Dict[str, Any]]:
+ if _safe_text(parent_record.get("category", "")).lower() in {"question", "interaction_intent"}:
+ return []
+ parent_metadata = dict(parent_record.get("metadata", {}) or {})
+ parent_slot_key = _safe_text(parent_record.get("slot_key", ""))
+ parent_event_signature = _safe_text(parent_metadata.get("event_signature", ""))
+ parent_source_span = _safe_text(parent_metadata.get("source_span", "")) or _safe_text(parent_record.get("value", ""))
+ parent_subject = _safe_text(parent_metadata.get("subject", ""))
+ parent_subject_signature = _safe_text(parent_metadata.get("subject_signature", ""))
+ parent_semantic_slot = _safe_text(parent_metadata.get("semantic_slot", "")) or _safe_text(parent_record.get("category", ""))
+ facets = _writer_facets_from_proposal(
+ proposal,
+ source_span=parent_source_span,
+ subject=parent_subject,
+ semantic_slot=parent_semantic_slot,
+ )
+ records: List[Dict[str, Any]] = []
+ for facet_index, facet in enumerate(facets, start=1):
+ facet_type = _facet_type(facet.get("type", ""))
+ role = _facet_role(facet, fallback=parent_semantic_slot or facet_type)
+ value = _facet_value(facet, facet_type=facet_type)
+ span = _facet_source_span(facet, fallback=parent_source_span)
+ if not facet_type or not value or not span:
+ continue
+ if not _span_is_grounded_in_current_turn(source_span=span, current_turn_text=current_turn_text):
+ # Facets are evidence-bearing; keep only attributes grounded in the
+ # writable turn or in the already-grounded parent source span.
+ span = parent_source_span
+ if not _span_is_grounded_in_current_turn(source_span=span, current_turn_text=current_turn_text):
+ continue
+ category = _FACET_CATEGORY_BY_TYPE.get(facet_type, "fact")
+ slot_key = (
+ f"{parent_slot_key}.facet.{facet_type}.{_slug(role)}.{facet_index}"
+ if parent_slot_key
+ else f"turn_{max(1, int(turn_index or 1))}.facet.{facet_type}.{proposal_index}.{facet_index}"
+ )
+ anchors = _dedupe_texts(
+ [
+ speaker,
+ timestamp,
+ parent_subject,
+ facet_type,
+ role,
+ value,
+ *list(parent_record.get("anchors", []) or []),
+ ],
+ max_items=12,
+ )
+ metadata = {
+ **dict(common_metadata),
+ "content_variant": "event_facet_write",
+ "facet_layer_version": "event_facet_v1",
+ "facet_type": facet_type,
+ "facet_role": role,
+ "facet_value": value,
+ "facet_source_span": span,
+ "facet_parent_slot_key": parent_slot_key,
+ "facet_parent_event_signature": parent_event_signature,
+ "facet_parent_category": _safe_text(parent_record.get("category", "")),
+ "facet_parent_relation": _safe_text(parent_record.get("relation", "")),
+ "facet_parent_source_kind": _safe_text(parent_record.get("source_kind", "")),
+ "raw_text": span,
+ "source_turn_text": span,
+ "source_span": span,
+ "semantic_slot": f"{facet_type}:{role}",
+ "event_phrase": value,
+ "event_text": value,
+ "event_signature": f"{parent_event_signature} facet {facet_type} {role} {value}".strip(),
+ "subject": parent_subject,
+ "subject_signature": parent_subject_signature,
+ "canonical_slot_key": slot_key,
+ "allow_parallel_state": True,
+ "memory_gate_decision": "facet_from_parent_event",
+ "grounding_score": round(float(_grounding_score(value=value, source_span=span, source_context=source_context)), 6),
+ "value_grounding_score": round(float(_value_grounding_score(value=value, source_context=source_context)), 6),
+ "llm_write_proposal_index": proposal_index,
+ "facet_index": facet_index,
+ "facet_raw": {key: _safe_text(val) if not isinstance(val, (list, dict)) else val for key, val in dict(facet).items()},
+ "time_expression_span": _safe_text(facet.get("time_expression_span", "")),
+ "time_granularity": _safe_text(facet.get("time_granularity", "")),
+ "resolved_time_value": _safe_text(facet.get("resolved_time_value", "")),
+ "time_display_value": _safe_text(facet.get("time_display_value", "")),
+ "resolved_date": _safe_text(facet.get("resolved_date", "")),
+ "target_status": _safe_text(facet.get("target_status", "")),
+ "numeric_value": _safe_text(facet.get("numeric_value", facet.get("number", facet.get("amount", "")))),
+ "numeric_unit": _safe_text(facet.get("unit", facet.get("numeric_unit", ""))),
+ "entity_signature": _slug(value) if facet_type == "entity" else "",
+ "evidence_role": role if facet_type == "evidence_role" else "",
+ }
+ records.append(
+ {
+ "category": category,
+ "slot_key": slot_key,
+ "value": value,
+ "anchors": anchors,
+ "relation": f"{facet_type}_facet_of",
+ "source_kind": "public_dialog_facet",
+ "state": "active",
+ "salience": min(0.96, max(0.72, float(parent_record.get("salience", 0.86) or 0.86))),
+ "confidence": min(0.96, max(0.70, float(parent_record.get("confidence", 0.84) or 0.84))),
+ "metadata": metadata,
+ }
+ )
+ return records
+
+
+def _proposal_parent_candidates(records: Sequence[Mapping[str, Any]]) -> List[Dict[str, str]]:
+ candidates: List[Dict[str, str]] = []
+ for index, raw in enumerate(records):
+ if not isinstance(raw, Mapping):
+ continue
+ metadata = dict(raw.get("metadata", {}) or {})
+ category = _safe_text(raw.get("category", ""))
+ if category.lower() in {"question", "interaction_intent", "profile", "preference"}:
+ continue
+ slot_key = _safe_text(raw.get("slot_key", ""))
+ value = _safe_text(raw.get("value", ""))
+ source_span = _safe_text(metadata.get("source_span", "")) or value
+ if not value or not slot_key:
+ continue
+ candidates.append(
+ {
+ "index": str(index),
+ "slot_key": slot_key,
+ "category": category,
+ "value": value[:220],
+ "source_span": source_span[:320],
+ "event_signature": _safe_text(metadata.get("event_signature", "")),
+ }
+ )
+ return candidates[:12]
+
+
+def _unit_writer_signal_hits(text: str, parents: Sequence[Mapping[str, Any]]) -> List[str]:
+ haystack_parts = [_safe_text(text)]
+ for parent in parents[:8]:
+ if not isinstance(parent, Mapping):
+ continue
+ haystack_parts.extend(
+ [
+ _safe_text(parent.get("value", "")),
+ _safe_text(parent.get("source_span", "")),
+ _safe_text(parent.get("category", "")),
+ ]
+ )
+ haystack = "\n".join(item for item in haystack_parts if item)
+ hits: List[str] = []
+ if _UNIT_WRITER_ACTION_RE.search(haystack):
+ hits.append("action")
+ if _UNIT_WRITER_NUMERIC_RE.search(haystack):
+ hits.append("numeric")
+ if _UNIT_WRITER_TEMPORAL_RE.search(haystack):
+ hits.append("temporal")
+ if _UNIT_WRITER_STATE_RE.search(haystack):
+ hits.append("state")
+ if _UNIT_WRITER_ROLE_RE.search(haystack):
+ hits.append("role")
+ return _dedupe_texts(hits, max_items=8)
+
+
+def _unit_writer_clause_span(text: str, start: int, end: int, *, max_chars: int = 220) -> str:
+ source = _safe_text(text)
+ if not source:
+ return ""
+ start = max(0, min(len(source), int(start)))
+ end = max(start, min(len(source), int(end)))
+ left = max(source.rfind(".", 0, start), source.rfind(";", 0, start), source.rfind("\n", 0, start))
+ left = 0 if left < 0 else left + 1
+ comma_left = source.rfind(",", left, start)
+ if comma_left >= left and start - comma_left > 36:
+ left = comma_left + 1
+ right_candidates = [idx for idx in (source.find(".", end), source.find(";", end), source.find("\n", end)) if idx >= 0]
+ right = min(right_candidates) if right_candidates else len(source)
+ comma_right = source.find(",", end, right)
+ if comma_right >= 0 and comma_right - end > 48:
+ right = comma_right
+ if right - left > max_chars:
+ left = max(0, start - 80)
+ right = min(len(source), end + 120)
+ return source[left:right].strip(" \t\r\n,;")
+
+
+def _unit_writer_coverage_requirements(text: str, *, max_items: int = 18) -> List[Dict[str, str]]:
+ source = _safe_text(text)
+ if not source:
+ return []
+ requirements: List[Dict[str, str]] = []
+ seen: set[tuple[str, str]] = set()
+
+ def add(kind: str, facet_type: str, span: str, role: str = "") -> None:
+ clean_span = _safe_text(span)
+ if not clean_span:
+ return
+ key = (kind, re.sub(r"\s+", " ", clean_span.lower()).strip())
+ if key in seen:
+ return
+ seen.add(key)
+ requirements.append(
+ {
+ "id": f"req_{len(requirements) + 1}",
+ "unit_kind_hint": kind,
+ "type_hint": facet_type,
+ "role_hint": role or kind,
+ "source_span": clean_span[:220],
+ }
+ )
+
+ for match in _UNIT_WRITER_ACTION_RE.finditer(source):
+ add("action_unit", "action", _unit_writer_clause_span(source, match.start(), match.end()), "action")
+ for match in _UNIT_WRITER_NUMERIC_RE.finditer(source):
+ add("numeric_quantity", "numeric", match.group(0), "quantity")
+ for match in _UNIT_WRITER_TEMPORAL_RE.finditer(source):
+ temporal_span = match.group(0)
+ nearby = source[match.start() : min(len(source), match.end() + 16)].lower()
+ if "/" in temporal_span and "scale" in nearby:
+ continue
+ add("temporal_anchor", "temporal", match.group(0), "time")
+ for match in _UNIT_WRITER_STATE_RE.finditer(source):
+ add("state_status", "state", _unit_writer_clause_span(source, match.start(), match.end(), max_chars=180), "status")
+ for match in _UNIT_WRITER_ROLE_RE.finditer(source):
+ add("evidence_role", "evidence_role", _unit_writer_clause_span(source, match.start(), match.end(), max_chars=180), "role")
+ return requirements[: max(1, int(max_items or 18))]
+
+
+def _unit_requirement_is_covered(requirement: Mapping[str, Any], unit_record: Mapping[str, Any]) -> bool:
+ metadata = dict(unit_record.get("metadata", {}) or {})
+ req_kind = _safe_text(requirement.get("unit_kind_hint", "")).lower()
+ req_type = _safe_text(requirement.get("type_hint", "")).lower()
+ req_span = _safe_text(requirement.get("source_span", ""))
+ unit_kind = _safe_text(metadata.get("unit_kind", "")).lower()
+ unit_type = _safe_text(metadata.get("facet_type", "")).lower()
+ unit_text = " ".join(
+ [
+ _safe_text(unit_record.get("value", "")),
+ _safe_text(metadata.get("facet_source_span", "")),
+ _safe_text(metadata.get("source_span", "")),
+ ]
+ )
+ if req_kind and unit_kind and req_kind != unit_kind:
+ if req_kind not in unit_kind and unit_kind not in req_kind:
+ return False
+ if req_type and unit_type and req_type != unit_type:
+ return False
+ req_norm = re.sub(r"\s+", " ", req_span.lower()).strip()
+ unit_norm = re.sub(r"\s+", " ", unit_text.lower()).strip()
+ if req_norm and unit_norm and (req_norm in unit_norm or unit_norm in req_norm):
+ return True
+ req_tokens = {token for token in _tokens(req_span) if len(token) > 2}
+ unit_tokens = {token for token in _tokens(unit_text) if len(token) > 2}
+ if not req_tokens or not unit_tokens:
+ return False
+ coverage = len(req_tokens & unit_tokens) / max(1, len(req_tokens))
+ return coverage >= 0.45
+
+
+def _unit_writer_uncovered_requirements(
+ requirements: Sequence[Mapping[str, Any]],
+ unit_records: Sequence[Mapping[str, Any]],
+) -> List[Dict[str, str]]:
+ uncovered: List[Dict[str, str]] = []
+ for requirement in requirements:
+ if not isinstance(requirement, Mapping):
+ continue
+ if any(_unit_requirement_is_covered(requirement, record) for record in unit_records if isinstance(record, Mapping)):
+ continue
+ uncovered.append({key: _safe_text(value) for key, value in dict(requirement).items()})
+ return uncovered
+
+
+def _unit_writer_fallback_units_from_requirements(
+ requirements: Sequence[Mapping[str, Any]],
+ *,
+ parent_records: Sequence[Mapping[str, Any]],
+ current_turn_text: str,
+) -> List[Dict[str, Any]]:
+ """Grounded attachment fallback when the LLM returns no unit for an obvious span."""
+
+ output: List[Dict[str, Any]] = []
+ source = _safe_text(current_turn_text)
+ for item in requirements:
+ if not isinstance(item, Mapping):
+ continue
+ span = _safe_text(item.get("source_span", ""))
+ if not span or not _span_is_grounded_in_current_turn(source_span=span, current_turn_text=source):
+ continue
+ kind = _safe_text(item.get("unit_kind_hint", "")) or "evidence_role"
+ facet_type = _safe_text(item.get("type_hint", "")) or "evidence_role"
+ role = _safe_text(item.get("role_hint", "")) or kind
+ if kind == "action_unit":
+ match = _UNIT_WRITER_ACTION_RE.search(span)
+ action = match.group(0) if match else ""
+ value = span
+ elif kind == "numeric_quantity":
+ action = ""
+ value = span
+ elif kind == "temporal_anchor":
+ action = ""
+ value = span
+ elif kind == "state_status":
+ action = ""
+ value = span
+ else:
+ action = ""
+ value = span
+ parent = _resolve_unit_parent_record({"source_span": span, "value": value}, parent_records)
+ output.append(
+ {
+ "parent_slot_key": _safe_text(parent.get("slot_key", "")) if parent else "",
+ "unit_kind": kind,
+ "type": facet_type,
+ "role": role,
+ "value": value,
+ "source_span": span,
+ "action": action,
+ "target": "",
+ "action_frame_id": _slug(f"fallback {kind} {span}")[:80],
+ "action_frame_index": "fallback",
+ "action_frame_source_span": span,
+ "applies_to_action": action,
+ "applies_to_entity": "",
+ "unit_writer_fallback": "coverage_requirement",
+ }
+ )
+ return output
+
+
+def _as_mapping_list(value: Any) -> List[Mapping[str, Any]]:
+ if isinstance(value, Mapping):
+ return [value]
+ if isinstance(value, list):
+ return [item for item in value if isinstance(item, Mapping)]
+ return []
+
+
+def _first_text(mapping: Mapping[str, Any], keys: Sequence[str]) -> str:
+ for key in keys:
+ text = _safe_text(mapping.get(key, ""))
+ if text:
+ return text
+ return ""
+
+
+def _unit_writer_raw_units_from_action_frames(parsed: Mapping[str, Any]) -> List[Dict[str, Any]]:
+ """Convert action-frame output into flat event_unit records while keeping frame binding metadata."""
+
+ raw_frames = (
+ parsed.get("action_frames")
+ or parsed.get("frames")
+ or parsed.get("event_frames")
+ or parsed.get("action_frame_units")
+ or []
+ )
+ frames = _as_mapping_list(raw_frames)
+ output: List[Dict[str, Any]] = []
+ for frame_index, frame in enumerate(frames, start=1):
+ action = _first_text(frame, ("action", "verb", "action_unit", "operation"))
+ entity = _first_text(frame, ("entity", "target_entity", "target", "object", "item"))
+ source_span = _first_text(frame, ("source_span", "span", "evidence_span", "action_span"))
+ frame_id = _safe_text(frame.get("frame_id", "")) or _slug(" ".join([action, entity, str(frame_index)]))
+ parent_slot_key = _safe_text(frame.get("parent_slot_key", frame.get("parent", "")))
+ base = {
+ "parent_slot_key": parent_slot_key,
+ "action_frame_id": frame_id,
+ "action_frame_index": str(frame_index),
+ "action_frame_source_span": source_span,
+ "applies_to_action": action,
+ "applies_to_entity": entity,
+ }
+ if action:
+ value = " ".join(item for item in (action, entity) if item).strip() or source_span
+ output.append(
+ {
+ **base,
+ "unit_kind": "action_unit",
+ "type": "action",
+ "role": _safe_text(frame.get("action_role", "")) or "action",
+ "value": value,
+ "source_span": _first_text(frame, ("action_span", "source_span", "span")) or source_span or value,
+ "action": action,
+ "target": entity,
+ }
+ )
+ if entity:
+ output.append(
+ {
+ **base,
+ "unit_kind": "target_entity",
+ "type": "entity",
+ "role": _safe_text(frame.get("entity_role", "")) or "target_entity",
+ "value": entity,
+ "source_span": _first_text(frame, ("entity_span", "source_span", "span")) or entity,
+ "action": action,
+ "target": entity,
+ }
+ )
+ for temporal_index, temporal in enumerate(
+ _as_mapping_list(frame.get("temporal_anchors", frame.get("temporals", frame.get("times", [])))),
+ start=1,
+ ):
+ value = _first_text(temporal, ("value", "time", "time_expression_span", "resolved_time_value", "resolved_date"))
+ span = _first_text(temporal, ("source_span", "span", "time_expression_span")) or value
+ if not value and not span:
+ continue
+ role = _first_text(temporal, ("role", "temporal_role", "applies_to_role")) or "temporal_anchor"
+ output.append(
+ {
+ **base,
+ "unit_kind": "temporal_anchor",
+ "type": "temporal",
+ "role": role,
+ "value": value or span,
+ "source_span": span,
+ "normalized_time": _first_text(temporal, ("normalized_time", "resolved_time_value", "resolved_date")),
+ "temporal_role": role,
+ "temporal_index": str(temporal_index),
+ "action": action,
+ "target": entity,
+ }
+ )
+ for numeric_index, numeric in enumerate(
+ _as_mapping_list(frame.get("numeric_quantities", frame.get("numerics", frame.get("quantities", [])))),
+ start=1,
+ ):
+ value = _first_text(numeric, ("value", "quantity", "numeric_value", "number", "amount"))
+ unit = _first_text(numeric, ("unit", "numeric_unit"))
+ span = _first_text(numeric, ("source_span", "span", "quantity_span")) or " ".join(item for item in (value, unit) if item)
+ if not value and not span:
+ continue
+ output.append(
+ {
+ **base,
+ "unit_kind": "numeric_quantity",
+ "type": "numeric",
+ "role": _first_text(numeric, ("role", "numeric_role")) or "quantity",
+ "value": " ".join(item for item in (value, unit) if item).strip() or span,
+ "source_span": span,
+ "quantity": value,
+ "unit": unit,
+ "numeric_index": str(numeric_index),
+ "action": action,
+ "target": entity,
+ }
+ )
+ for state_index, state in enumerate(
+ _as_mapping_list(frame.get("states", frame.get("state_statuses", frame.get("statuses", [])))),
+ start=1,
+ ):
+ value = _first_text(state, ("value", "status", "state", "target_status"))
+ span = _first_text(state, ("source_span", "span")) or value or source_span
+ if not value and not span:
+ continue
+ output.append(
+ {
+ **base,
+ "unit_kind": "state_status",
+ "type": "state",
+ "role": _first_text(state, ("role", "state_role")) or "status",
+ "value": value or span,
+ "source_span": span,
+ "status": value,
+ "state_index": str(state_index),
+ "action": action,
+ "target": entity,
+ }
+ )
+ for role_index, role_item in enumerate(
+ _as_mapping_list(frame.get("roles", frame.get("participation_roles", []))),
+ start=1,
+ ):
+ value = _first_text(role_item, ("value", "role", "participant", "actor")) or _safe_text(frame.get("actor", ""))
+ span = _first_text(role_item, ("source_span", "span")) or source_span or value
+ if not value and not span:
+ continue
+ role_value = _first_text(role_item, ("role", "evidence_role")) or "role"
+ output.append(
+ {
+ **base,
+ "unit_kind": "leadership_role" if "lead" in role_value.lower() else "participation_role",
+ "type": "role",
+ "role": role_value,
+ "value": value or role_value,
+ "source_span": span,
+ "role_index": str(role_index),
+ "action": action,
+ "target": entity,
+ }
+ )
+ return output
+
+
+def _resolve_unit_parent_record(unit: Mapping[str, Any], parent_records: Sequence[Mapping[str, Any]]) -> Mapping[str, Any] | None:
+ requested_slot = _safe_text(unit.get("parent_slot_key", unit.get("parent", ""))).lower()
+ if requested_slot:
+ for record in parent_records:
+ if _safe_text(record.get("slot_key", "")).lower() == requested_slot:
+ return record
+ span = _safe_text(unit.get("source_span", ""))
+ value = _safe_text(unit.get("value", unit.get("target", "")))
+ best: tuple[int, Mapping[str, Any] | None] = (0, None)
+ for record in parent_records:
+ metadata = dict(record.get("metadata", {}) or {})
+ haystack = " ".join(
+ [
+ _safe_text(record.get("value", "")),
+ _safe_text(metadata.get("source_span", "")),
+ _safe_text(metadata.get("raw_text", "")),
+ ]
+ ).lower()
+ score = 0
+ if span and span.lower() in haystack:
+ score += 5
+ if value and value.lower() in haystack:
+ score += 2
+ score += len(set(_tokens(span or value)) & set(_tokens(haystack)))
+ if score > best[0]:
+ best = (score, record)
+ if best[1] is not None:
+ return best[1]
+ return parent_records[0] if parent_records else None
+
+
+def _unit_kind(raw: Mapping[str, Any], facet_type: str, role: str) -> str:
+ value = _safe_text(raw.get("unit_kind", raw.get("kind", raw.get("schema", "")))).lower()
+ if value:
+ return _slug(value)[:80]
+ role_l = role.lower()
+ if facet_type == "action":
+ return "action_unit"
+ if facet_type == "entity":
+ return "target_entity"
+ if facet_type == "numeric":
+ return "numeric_quantity"
+ if facet_type == "temporal":
+ return "temporal_anchor"
+ if facet_type == "state":
+ return "state_status"
+ if "lead" in role_l:
+ return "leadership_role"
+ if "participat" in role_l or "volunteer" in role_l or "attend" in role_l:
+ return "participation_role"
+ return "evidence_role"
+
+
+def build_modelized_facet_unit_records(
+ writer: Any,
+ *,
+ current_turn_text: str,
+ parent_records: Sequence[Mapping[str, Any]],
+ speaker: str,
+ timestamp: str,
+ turn_index: int,
+ max_units: int = 14,
+) -> tuple[List[Dict[str, Any]], Dict[str, Any]]:
+ """Attach action-frame unit nodes under accepted event/source records.
+
+ This is not a separate multi-memory writer. It refines normal event writes by
+ adding action-frame-bound attributes that graph tunneling can later connect.
+ """
+
+ mode = _safe_text(
+ os.environ.get("TMCRA_EVENT_ACTION_FRAME_ATTACHMENT_MODE", os.environ.get("TMCRA_MODELIZED_UNIT_WRITER_MODE", "on"))
+ ).lower()
+ if mode in {"0", "off", "false", "disabled", "none"}:
+ return [], {"enabled": False, "reason": "disabled"}
+ parents = _proposal_parent_candidates(parent_records)
+ source_or_event_parents = [
+ item
+ for item in parents
+ if _safe_text(item.get("category", "")).lower() in {"fact", "event", "time", "status", "constraint", "goal"}
+ ]
+ if not source_or_event_parents:
+ return [], {"enabled": False, "reason": "no_event_or_source_parent_records", "parent_count": len(parents)}
+ gate_mode = _safe_text(os.environ.get("TMCRA_MODELIZED_UNIT_WRITER_SIGNAL_GATE", "on")).lower()
+ signal_hits = _unit_writer_signal_hits(current_turn_text, source_or_event_parents)
+ if gate_mode not in {"0", "off", "false", "disabled", "none"} and not signal_hits:
+ return [], {
+ "enabled": False,
+ "reason": "no_unit_worthy_signal",
+ "parent_count": len(parents),
+ "source_or_event_parent_count": len(source_or_event_parents),
+ }
+ chat = getattr(writer, "_chat", None)
+ if chat is None and hasattr(writer, "base_writer"):
+ chat = getattr(getattr(writer, "base_writer"), "_chat", None)
+ if chat is None:
+ return [], {
+ "enabled": False,
+ "reason": "writer_chat_unavailable",
+ "parent_count": len(parents),
+ "unit_signal_hits": signal_hits,
+ }
+ max_units = max(1, min(24, int(max_units or 14)))
+ coverage_requirements = _unit_writer_coverage_requirements(current_turn_text, max_items=max_units)
+ system_prompt = (
+ "Return strict JSON only. You are the TMCRA event action-frame attachment writer. "
+ "Your job is to attach grounded action frames under current_turn's accepted event/source records. "
+ "This is not a multi-memory writer and you must not decide whether separate events form a chain. "
+ "Do not answer any benchmark question and do not infer facts not grounded in current_turn plus its explicit timestamp/session date metadata. "
+ "Split one sentence into multiple action_frames when it contains multiple actions with different time/status/numeric bindings. "
+ "For each action_frame, bind temporal_anchors, numeric_quantities, states, roles, and target_entity to the specific action they apply to. "
+ "Preserve exact answer-bearing values as units: names, places, relationship/status labels, countries, dates, years, quantities, object identities, and role participants. "
+ "For status-like phrases, emit state units with precise role names such as relationship_status, life_status, task_status, possession_status, or membership_status; examples include single parent -> relationship_status/single and passed interviews -> task_status/passed. "
+ "For appositions or identity bindings, emit entity units with role identity_binding or origin_binding; examples include home country, Sweden -> home_country = Sweden and necklace from grandma -> origin/person = grandma. "
+ "For relative time expressions, keep the exact expression in value and source_span, and fill normalized_time when it can be grounded from the turn/session date; examples: last year with a 2023 session date -> 2022, yesterday -> normalized calendar date if computable. "
+ "If a turn says 'I got X and will do Y this weekend', create one frame for getting X with the turn time/current status, and one frame for doing Y with weekend as planned_action_time. "
+ "Use coverage_requirements as grounded spans that should be covered by frames or frame attributes when durable. "
+ "For every source_span, copy exact text from current_turn. "
+ "For parent_slot_key, choose one slot_key from parent_candidates when possible; otherwise use the source_turn parent. "
+ "Return {\"action_frames\":[{\"parent_slot_key\":\"...\",\"frame_id\":\"...\",\"action\":\"...\",\"entity\":\"...\",\"source_span\":\"...\","
+ "\"temporal_anchors\":[{\"role\":\"acquisition_time|planned_action_time|completion_time|current_turn_time|event_time\",\"value\":\"...\",\"source_span\":\"...\",\"normalized_time\":\"\"}],"
+ "\"numeric_quantities\":[{\"role\":\"quantity|scale|amount|duration\",\"value\":\"...\",\"unit\":\"...\",\"source_span\":\"...\"}],"
+ "\"states\":[{\"role\":\"status\",\"value\":\"current|planned|completed|pending|returned|started\",\"source_span\":\"...\"}],"
+ "\"roles\":[{\"role\":\"leadership|participation|ownership\",\"value\":\"...\",\"source_span\":\"...\"}]}],"
+ "\"units\":[{\"parent_slot_key\":\"...\",\"unit_kind\":\"...\",\"type\":\"action|entity|numeric|temporal|state|role|evidence_role\","
+ "\"role\":\"...\",\"value\":\"...\",\"source_span\":\"...\",\"action\":\"\",\"target\":\"\",\"quantity\":\"\",\"unit\":\"\",\"normalized_time\":\"\",\"status\":\"\"}]}. "
+ "Prefer action_frames; include units only for extra attributes that cannot fit a frame. Do not emit empty or duplicate data."
+ )
+ payload = {
+ "speaker": _safe_text(speaker),
+ "timestamp": _safe_text(timestamp),
+ "max_units": max_units,
+ "parent_candidates": parents,
+ "coverage_requirements": coverage_requirements,
+ "current_turn": _safe_text(current_turn_text),
+ }
+ try:
+ raw, usage = chat(
+ [
+ {"role": "system", "content": system_prompt},
+ {"role": "user", "content": json.dumps(payload, ensure_ascii=False)},
+ ]
+ )
+ except Exception as exc:
+ return [], {
+ "enabled": True,
+ "error": f"{exc.__class__.__name__}: {str(exc)[:200]}",
+ "parent_count": len(parents),
+ }
+ parsed = _extract_json_object(raw)
+ raw_frame_units = _unit_writer_raw_units_from_action_frames(parsed) if isinstance(parsed, Mapping) else []
+ raw_units = parsed.get("units", []) if isinstance(parsed, Mapping) else []
+ if not isinstance(raw_units, list):
+ raw_units = []
+ raw_units = list(raw_frame_units) + [item for item in raw_units if isinstance(item, Mapping)]
+ source_text = _safe_text(current_turn_text)
+ output: List[Dict[str, Any]] = []
+ seen = set()
+ parent_records_by_slot = {_safe_text(item.get("slot_key", "")): item for item in parent_records if isinstance(item, Mapping)}
+
+ def materialize_units(raw_items: Sequence[Any], *, phase: str, limit: int) -> int:
+ stored = 0
+ for raw_unit in list(raw_items or [])[: max(0, int(limit or 0))]:
+ if not isinstance(raw_unit, Mapping):
+ continue
+ facet_type = _facet_type(raw_unit.get("type", raw_unit.get("facet_type", raw_unit.get("unit_kind", ""))))
+ if not facet_type:
+ continue
+ role = _facet_role(raw_unit, fallback=_unit_kind(raw_unit, facet_type, ""))
+ value = _facet_value(raw_unit, facet_type=facet_type) or _safe_text(raw_unit.get("target", raw_unit.get("action", "")))
+ normalized_time = _safe_text(raw_unit.get("normalized_time", raw_unit.get("resolved_date", "")))
+ if facet_type == "temporal" and normalized_time and normalized_time.lower() not in value.lower():
+ value = f"{value} ({normalized_time})"
+ span = _facet_source_span(raw_unit, fallback=value)
+ if not value or not span:
+ continue
+ if not _span_is_grounded_in_current_turn(source_span=span, current_turn_text=source_text):
+ continue
+ parent = _resolve_unit_parent_record(raw_unit, parent_records)
+ if parent is None:
+ continue
+ parent_slot_key = _safe_text(parent.get("slot_key", ""))
+ parent_metadata = dict(parent.get("metadata", {}) or {})
+ unit_kind = _unit_kind(raw_unit, facet_type, role)
+ key = (parent_slot_key.lower(), unit_kind, facet_type, role.lower(), value.lower(), span.lower())
+ if key in seen:
+ continue
+ seen.add(key)
+ category = _FACET_CATEGORY_BY_TYPE.get(facet_type, "fact")
+ slot_key = f"{parent_slot_key}.unit.{facet_type}.{_slug(unit_kind)}.{len(output) + 1}"
+ metadata = {
+ **parent_metadata,
+ "content_variant": "event_facet_write",
+ "facet_layer_version": "event_unit_v1",
+ "unit_writer": "event_action_frame_attachment_writer_v2",
+ "unit_writer_phase": phase,
+ "unit_attachment_schema": "event_action_frame_v2",
+ "action_frame_id": _safe_text(raw_unit.get("action_frame_id", raw_unit.get("frame_id", ""))),
+ "action_frame_index": _safe_text(raw_unit.get("action_frame_index", "")),
+ "action_frame_source_span": _safe_text(raw_unit.get("action_frame_source_span", "")),
+ "applies_to_action": _safe_text(raw_unit.get("applies_to_action", raw_unit.get("action", ""))),
+ "applies_to_entity": _safe_text(raw_unit.get("applies_to_entity", raw_unit.get("target", ""))),
+ "unit_writer_fallback": _safe_text(raw_unit.get("unit_writer_fallback", "")),
+ "temporal_role": _safe_text(raw_unit.get("temporal_role", "")),
+ "unit_kind": unit_kind,
+ "facet_type": facet_type,
+ "facet_role": role,
+ "facet_value": value,
+ "facet_source_span": span,
+ "facet_parent_slot_key": parent_slot_key,
+ "facet_parent_event_signature": _safe_text(parent_metadata.get("event_signature", "")),
+ "facet_parent_category": _safe_text(parent.get("category", "")),
+ "facet_parent_relation": _safe_text(parent.get("relation", "")),
+ "raw_text": span,
+ "source_turn_text": span,
+ "source_span": span,
+ "semantic_slot": f"{unit_kind}:{role}",
+ "event_phrase": value,
+ "event_text": value,
+ "event_signature": f"{_safe_text(parent_metadata.get('event_signature', ''))} unit {unit_kind} {role} {value}".strip(),
+ "action": _safe_text(raw_unit.get("action", "")),
+ "target": _safe_text(raw_unit.get("target", "")),
+ "quantity": _safe_text(raw_unit.get("quantity", raw_unit.get("numeric_value", ""))),
+ "unit": _safe_text(raw_unit.get("unit", "")),
+ "normalized_time": normalized_time,
+ "status": _safe_text(raw_unit.get("status", "")),
+ }
+ output.append(
+ {
+ "category": category,
+ "slot_key": slot_key,
+ "value": value,
+ "anchors": _dedupe_texts([speaker, timestamp, unit_kind, facet_type, role, value], max_items=10),
+ "relation": "unit_of_event",
+ "source_kind": "event_action_frame_attachment",
+ "state": "evidence",
+ "salience": 0.94,
+ "confidence": 0.86,
+ "metadata": metadata,
+ }
+ )
+ stored += 1
+ return stored
+
+ initial_stored_count = materialize_units(raw_units, phase="initial", limit=max_units)
+ uncovered_before_repair = _unit_writer_uncovered_requirements(coverage_requirements, output)
+ repair_raw_count = 0
+ repair_stored_count = 0
+ repair_error = ""
+ repair_usage: Dict[str, Any] = {}
+ repair_mode = _safe_text(
+ os.environ.get(
+ "TMCRA_EVENT_ACTION_FRAME_ATTACHMENT_REPAIR_MODE",
+ os.environ.get("TMCRA_MODELIZED_UNIT_WRITER_COVERAGE_REPAIR_MODE", "on"),
+ )
+ ).lower()
+ if (
+ repair_mode not in {"0", "off", "false", "disabled", "none"}
+ and uncovered_before_repair
+ and len(output) < max_units
+ ):
+ repair_prompt = (
+ "Return strict JSON only. You are repairing missing TMCRA event action-frame attachment writes. "
+ "The initial pass missed some grounded coverage_requirements. "
+ "Emit missing action_frames or units only, using exact source_span text from current_turn. "
+ "Do not repeat existing_units. Do not infer facts not grounded in current_turn plus explicit timestamp/session date metadata. "
+ "Do not decide whether this is a multi-memory chain; only attach attributes to the current event. "
+ "Prefer action_frames with correctly bound temporal_anchors/numeric_quantities/states. "
+ "Repair missing precise answer-bearing units: relationship/status labels, country/place identity bindings, normalized relative dates, quantities, and participants. "
+ "Return {\"action_frames\":[{\"parent_slot_key\":\"...\",\"frame_id\":\"...\",\"action\":\"...\",\"entity\":\"...\",\"source_span\":\"...\","
+ "\"temporal_anchors\":[{\"role\":\"...\",\"value\":\"...\",\"source_span\":\"...\",\"normalized_time\":\"\"}],"
+ "\"numeric_quantities\":[{\"role\":\"...\",\"value\":\"...\",\"unit\":\"...\",\"source_span\":\"...\"}],"
+ "\"states\":[{\"role\":\"status\",\"value\":\"...\",\"source_span\":\"...\"}],\"roles\":[]}],"
+ "\"units\":[{\"parent_slot_key\":\"...\",\"unit_kind\":\"...\",\"type\":\"action|entity|numeric|temporal|state|role|evidence_role\","
+ "\"role\":\"...\",\"value\":\"...\",\"source_span\":\"...\",\"action\":\"\",\"target\":\"\",\"quantity\":\"\",\"unit\":\"\",\"normalized_time\":\"\",\"status\":\"\"}]}."
+ )
+ repair_payload = {
+ "speaker": _safe_text(speaker),
+ "timestamp": _safe_text(timestamp),
+ "max_missing_units": max_units - len(output),
+ "parent_candidates": parents,
+ "current_turn": source_text,
+ "uncovered_requirements": uncovered_before_repair,
+ "existing_units": [
+ {
+ "unit_kind": dict(item.get("metadata", {}) or {}).get("unit_kind", ""),
+ "type": dict(item.get("metadata", {}) or {}).get("facet_type", ""),
+ "value": item.get("value", ""),
+ "source_span": dict(item.get("metadata", {}) or {}).get("facet_source_span", ""),
+ }
+ for item in output
+ ],
+ }
+ try:
+ repair_raw, repair_usage = chat(
+ [
+ {"role": "system", "content": repair_prompt},
+ {"role": "user", "content": json.dumps(repair_payload, ensure_ascii=False)},
+ ]
+ )
+ repair_parsed = _extract_json_object(repair_raw)
+ repair_frame_units = _unit_writer_raw_units_from_action_frames(repair_parsed)
+ repair_units = repair_parsed.get("units", []) if isinstance(repair_parsed, Mapping) else []
+ if not isinstance(repair_units, list):
+ repair_units = []
+ repair_units = list(repair_frame_units) + [item for item in repair_units if isinstance(item, Mapping)]
+ repair_raw_count = len(repair_units)
+ repair_stored_count = materialize_units(repair_units, phase="coverage_repair", limit=max_units - len(output))
+ except Exception as exc:
+ repair_error = f"{exc.__class__.__name__}: {str(exc)[:200]}"
+ uncovered_after_repair = _unit_writer_uncovered_requirements(coverage_requirements, output)
+ fallback_raw_count = 0
+ fallback_stored_count = 0
+ fallback_mode = _safe_text(
+ os.environ.get(
+ "TMCRA_EVENT_ACTION_FRAME_ATTACHMENT_FALLBACK_MODE",
+ os.environ.get("TMCRA_MODELIZED_UNIT_WRITER_FALLBACK_MODE", "on"),
+ )
+ ).lower()
+ if (
+ fallback_mode not in {"0", "off", "false", "disabled", "none"}
+ and uncovered_after_repair
+ and len(output) < max_units
+ ):
+ fallback_units = _unit_writer_fallback_units_from_requirements(
+ uncovered_after_repair,
+ parent_records=parent_records,
+ current_turn_text=source_text,
+ )
+ fallback_raw_count = len(fallback_units)
+ fallback_stored_count = materialize_units(fallback_units, phase="coverage_fallback", limit=max_units - len(output))
+ uncovered_after_repair = _unit_writer_uncovered_requirements(coverage_requirements, output)
+ return output, {
+ "enabled": True,
+ "parent_count": len(parents),
+ "source_or_event_parent_count": len(source_or_event_parents),
+ "unit_signal_hits": signal_hits,
+ "coverage_requirement_count": len(coverage_requirements),
+ "coverage_requirements": coverage_requirements,
+ "raw_unit_count": len(raw_units),
+ "initial_stored_unit_count": initial_stored_count,
+ "coverage_repair_enabled": repair_mode not in {"0", "off", "false", "disabled", "none"},
+ "uncovered_before_repair_count": len(uncovered_before_repair),
+ "uncovered_before_repair": uncovered_before_repair,
+ "repair_raw_unit_count": repair_raw_count,
+ "repair_stored_unit_count": repair_stored_count,
+ "repair_error": repair_error,
+ "repair_usage": repair_usage,
+ "coverage_fallback_enabled": fallback_mode not in {"0", "off", "false", "disabled", "none"},
+ "fallback_raw_unit_count": fallback_raw_count,
+ "fallback_stored_unit_count": fallback_stored_count,
+ "uncovered_after_repair_count": len(uncovered_after_repair),
+ "uncovered_after_repair": uncovered_after_repair,
+ "stored_unit_count": len(output),
+ "usage": usage,
+ }
+
+
+def _extract_json_object(text: str) -> Dict[str, Any]:
+ raw = _safe_text(text)
+ if raw.startswith("```"):
+ raw = re.sub(r"^```(?:json)?\s*", "", raw, flags=re.IGNORECASE).strip()
+ raw = re.sub(r"```$", "", raw).strip()
+ try:
+ parsed = json.loads(raw)
+ if isinstance(parsed, Mapping):
+ return dict(parsed)
+ except Exception:
+ pass
+ repaired_raw = _repair_jsonish_text(raw)
+ try:
+ parsed = json.loads(repaired_raw)
+ return dict(parsed) if isinstance(parsed, Mapping) else {}
+ except Exception:
+ pass
+ start = repaired_raw.find("{")
+ while start >= 0:
+ depth = 0
+ in_string = False
+ escaped = False
+ for index in range(start, len(repaired_raw)):
+ char = repaired_raw[index]
+ if in_string:
+ if escaped:
+ escaped = False
+ elif char == "\\":
+ escaped = True
+ elif char == '"':
+ in_string = False
+ continue
+ if char == '"':
+ in_string = True
+ elif char == "{":
+ depth += 1
+ elif char == "}":
+ depth -= 1
+ if depth == 0:
+ candidate = repaired_raw[start : index + 1]
+ try:
+ parsed = json.loads(candidate)
+ if isinstance(parsed, Mapping) and (
+ "write_proposals" in parsed or "write_proposal" in parsed or "memory_writes" in parsed or "units" in parsed
+ ):
+ return dict(parsed)
+ except Exception:
+ break
+ start = repaired_raw.find("{", start + 1)
+ end = repaired_raw.rfind("}")
+ if repaired_raw.find("{") >= 0 and end > repaired_raw.find("{"):
+ try:
+ parsed = json.loads(repaired_raw[repaired_raw.find("{") : end + 1])
+ if isinstance(parsed, Mapping) and ("write_proposals" in parsed or "write_proposal" in parsed or "memory_writes" in parsed or "units" in parsed):
+ return dict(parsed)
+ except Exception:
+ pass
+ partial = _extract_partial_write_proposals(repaired_raw)
+ if partial:
+ return partial
+ if "write_proposals" in repaired_raw or "write_proposal" in repaired_raw or "memory_writes" in repaired_raw or "units" in repaired_raw:
+ return {
+ "write_proposals": [],
+ "_tmcra_json_repair_status": "empty_from_unrecoverable_writer_json",
+ }
+ return {}
+
+
+def _repair_jsonish_text(value: str) -> str:
+ text = _safe_text(value)
+ text = re.sub(r"^```(?:json)?\s*", "", text, flags=re.IGNORECASE).strip()
+ text = re.sub(r"```$", "", text).strip()
+ text = re.sub(r'"write_proposal"\s*:', '"write_proposals":', text, flags=re.IGNORECASE)
+ text = re.sub(r"//.*?(?=(?:[,}\]]|$))", "", text)
+ text = re.sub(r",\s*([}\]])", r"\1", text)
+ text = re.sub(r"'([^'\\]*(?:\\.[^'\\]*)*)'", r'"\1"', text)
+ # Some local OpenAI-compatible servers emit invalid JSON escapes such as
+ # That\'s inside double-quoted strings. This repair is conservative enough
+ # for writer payloads and avoids failing an otherwise complete object.
+ text = text.replace("\\'", "'")
+ return text
+
+
+def _extract_turn_intent_object(parsed: Mapping[str, Any] | None) -> Dict[str, Any]:
+ if not isinstance(parsed, Mapping):
+ return {}
+ candidate = parsed.get("turn_intent", parsed.get("turn_intent_gate", {}))
+ if isinstance(candidate, Mapping):
+ normalized = {_safe_text(key).strip().lower(): value for key, value in dict(candidate).items()}
+ if "write_allowed" not in normalized:
+ for key, value in list(normalized.items()):
+ if key.startswith("write_allowed"):
+ normalized["write_allowed"] = value
+ break
+ return normalized
+ normalized_parsed = {_safe_text(key).strip().lower(): value for key, value in dict(parsed).items()}
+ if "write_allowed" not in normalized_parsed:
+ for key, value in list(normalized_parsed.items()):
+ if key.startswith("write_allowed"):
+ normalized_parsed["write_allowed"] = value
+ break
+ if any(key in normalized_parsed for key in ("intent", "write_allowed", "question_like", "memory_assertion", "question_only")):
+ return {
+ key: normalized_parsed.get(key)
+ for key in ("intent", "write_allowed", "question_like", "memory_assertion", "question_only", "reason")
+ if key in normalized_parsed
+ }
+ return {}
+
+
+def _extract_turn_intent_from_text(text: str) -> Dict[str, Any]:
+ raw = _safe_text(text)
+ if not raw:
+ return {}
+ intent_match = re.search(r"""["']?\bintent\b["']?\s*:\s*["']([a-zA-Z0-9_\- ]+)["']""", raw, flags=re.IGNORECASE)
+ if not intent_match:
+ return {}
+
+ def find_bool(key: str) -> bool | None:
+ pattern = rf"""["']?\s*{re.escape(key)}[a-zA-Z_ ]*["']?\s*:\s*(true|false|yes|no|1|0)\b"""
+ match = re.search(pattern, raw, flags=re.IGNORECASE)
+ if not match:
+ return None
+ return match.group(1).lower() in {"true", "yes", "1"}
+
+ reason = ""
+ reason_match = re.search(r"""["']?\breason\b["']?\s*:\s*["']([^"']{0,240})["']""", raw, flags=re.IGNORECASE)
+ if reason_match:
+ reason = _safe_text(reason_match.group(1))
+ result: Dict[str, Any] = {
+ "intent": intent_match.group(1).strip().lower().replace(" ", "_"),
+ }
+ for key in ("write_allowed", "question_like", "memory_assertion", "question_only"):
+ value = find_bool(key)
+ if value is not None:
+ result[key] = value
+ if reason:
+ result["reason"] = reason
+ return result
+
+
+def _extract_partial_write_proposals(raw: str) -> Dict[str, Any]:
+ proposals_start = re.search(r'"(?:write_proposals|write_proposal|memory_writes)"\s*:\s*\[', raw)
+ if not proposals_start:
+ return {}
+ array_start = raw.find("[", proposals_start.start())
+ if array_start < 0:
+ return {}
+ proposals: List[Dict[str, Any]] = []
+ start = raw.find("{", array_start)
+ while start >= 0:
+ depth = 0
+ in_string = False
+ escaped = False
+ parsed_any = False
+ for index in range(start, len(raw)):
+ char = raw[index]
+ if in_string:
+ if escaped:
+ escaped = False
+ elif char == "\\":
+ escaped = True
+ elif char == '"':
+ in_string = False
+ continue
+ if char == '"':
+ in_string = True
+ elif char == "{":
+ depth += 1
+ elif char == "}":
+ depth -= 1
+ if depth == 0:
+ candidate = _repair_jsonish_text(raw[start : index + 1])
+ try:
+ parsed = json.loads(candidate)
+ except Exception:
+ parsed = None
+ if isinstance(parsed, Mapping):
+ proposals.append(dict(parsed))
+ start = raw.find("{", index + 1)
+ parsed_any = True
+ break
+ elif char == "]" and depth == 0:
+ start = -1
+ parsed_any = True
+ break
+ if not parsed_any:
+ break
+ if proposals:
+ return {
+ "write_proposals": proposals,
+ "_tmcra_json_repair_status": "partial_write_proposals_recovered",
+ }
+ return {}
+
+
+def _grounding_score(*, value: str, source_span: str, source_context: str) -> float:
+ source_lower = _safe_text(source_context).lower()
+ value_lower = _safe_text(value).lower()
+ span_lower = _safe_text(source_span).lower()
+ if span_lower and span_lower in source_lower:
+ return 1.0
+ if value_lower and value_lower in source_lower:
+ return 1.0
+ candidate_tokens = set(_tokens(source_span or value))
+ if not candidate_tokens:
+ return 0.0
+ source_tokens = set(_tokens(source_context))
+ if not source_tokens:
+ return 0.0
+ return len(candidate_tokens & source_tokens) / max(1, len(candidate_tokens))
+
+
+def _value_grounding_score(*, value: str, source_context: str) -> float:
+ return _grounding_score(value=value, source_span="", source_context=source_context)
+
+
+def _span_is_grounded_in_current_turn(*, source_span: str, current_turn_text: str) -> bool:
+ span_lower = _safe_text(source_span).lower()
+ turn_lower = _safe_text(current_turn_text).lower()
+ return bool(span_lower) and span_lower in turn_lower
+
+
+def _source_span_can_stand_as_value(*, source_span: str, value: str, current_turn_text: str) -> bool:
+ """Allow a truth-preserving repair only when the copied span is real evidence.
+
+ If an LLM emits a lossy normalized value, the gate may replace that value
+ with the copied source span instead of accepting an under-grounded summary.
+ This is not a fallback write: the mutation remains constrained to text that
+ appears in the current user turn.
+ """
+ if not _span_is_grounded_in_current_turn(source_span=source_span, current_turn_text=current_turn_text):
+ return False
+ span_tokens = _tokens(source_span)
+ value_tokens = _tokens(value)
+ if not span_tokens:
+ return False
+ # Very short spans such as "my project" are not safe canonical values for a
+ # hallucinated summary. Longer copied spans can preserve the asserted fact.
+ return len(set(span_tokens)) >= max(4, min(6, len(set(value_tokens)) or 4))
+
+
+def _current_turn_can_repair_proposal(*, current_turn_text: str, value: str, source_span: str) -> bool:
+ turn_tokens = set(_tokens(current_turn_text))
+ if len(turn_tokens) < 5:
+ return False
+ candidate_tokens = set(_tokens(" ".join([value, source_span])))
+ if len(candidate_tokens) < 4:
+ return False
+ overlap = len(turn_tokens & candidate_tokens) / max(1, len(candidate_tokens))
+ return overlap >= 0.52
+
+
+_MEMORY_SUBJECT_PATTERNS = (
+ re.compile(
+ r"\bmy\s+(?P[a-z0-9][a-z0-9\s'/_-]{1,90}?)\s+(?:is|are|was|were)\s+(?:now\s+)?",
+ flags=re.IGNORECASE,
+ ),
+ re.compile(
+ r"\bwhat\s+(?:is|are|was|were)\s+my\s+(?P[a-z0-9][a-z0-9\s'/_-]{1,90}?)\s+(?:right\s+now|now|currently)\b",
+ flags=re.IGNORECASE,
+ ),
+)
+
+
+def _extract_memory_subject(*texts: Any) -> str:
+ for text in texts:
+ compact = _safe_text(text)
+ if not compact:
+ continue
+ for pattern in _MEMORY_SUBJECT_PATTERNS:
+ match = pattern.search(compact)
+ if not match:
+ continue
+ subject = _safe_text(match.group("subject")).strip(" .,:;!?\"'")
+ subject = re.sub(r"\b(now|currently|right)\b$", "", subject, flags=re.IGNORECASE).strip(" .,:;!?\"'")
+ if len(_tokens(subject)) >= 2:
+ return subject
+ return ""
+
+
+def _loose_phrase_pattern(value: Any) -> str:
+ parts = [_safe_text(part) for part in _safe_text(value).split() if _safe_text(part)]
+ return r"\s+".join(re.escape(part) for part in parts)
+
+
+def _trim_current_value(value: Any) -> str:
+ clean = _safe_text(value).strip(" .,:;!?\"'")
+ if not clean:
+ return ""
+ clean = re.split(r"\b(?:the\s+)?earlier\s+view\s+was\b", clean, maxsplit=1, flags=re.IGNORECASE)[0]
+ clean = re.split(r"\b(?:the\s+)?previous\s+view\s+was\b", clean, maxsplit=1, flags=re.IGNORECASE)[0]
+ return _safe_text(clean).strip(" .,:;!?\"'")
+
+
+def _extract_current_value_assertion(*, current_turn_text: str, subject: str = "") -> tuple[str, str]:
+ """Extract a current-value assertion from visible user text only."""
+
+ compact = _safe_text(current_turn_text)
+ subject_text = _safe_text(subject)
+ if not compact:
+ return "", ""
+ if subject_text:
+ subject_pattern = _loose_phrase_pattern(subject_text)
+ if subject_pattern:
+ patterns = (
+ rf"\bmy\s+{subject_pattern}\s+(?:is|are)\s+now\s+(?P.+?)(?:[.!?](?:\s|$)|$)",
+ rf"\b(?:i(?:'ve| have)\s+updated\s+my\s+{subject_pattern}\s*[:,-]\s*)(?P.+?)(?:[.!?](?:\s|$)|$)",
+ )
+ for pattern in patterns:
+ match = re.search(pattern, compact, flags=re.IGNORECASE)
+ if not match:
+ continue
+ value = _trim_current_value(match.group("value"))
+ if len(_tokens(value)) >= 3:
+ return _safe_text(match.group(0)).strip(" .,:;!?\"'"), value
+
+ subject_tokens = {token for token in _tokens(subject_text) if len(token) >= 4}
+ for marker in (
+ r"\bi\s+now\s+(?:believe|think|see)\s+(?P.+?)(?:[.!?](?:\s|$)|$)",
+ r"\bi\s+now\s+believe\s+that\s+(?P.+?)(?:[.!?](?:\s|$)|$)",
+ ):
+ match = re.search(marker, compact, flags=re.IGNORECASE)
+ if not match:
+ continue
+ value = _trim_current_value(match.group("value"))
+ value_tokens = set(_tokens(value))
+ if len(value_tokens) < 3:
+ continue
+ if subject_tokens and not (subject_tokens & value_tokens):
+ continue
+ return _safe_text(match.group(0)).strip(" .,:;!?\"'"), value
+ return "", ""
+
+
+_CURRENT_VALUE_COVERAGE_STOP_TOKENS = {
+ "about",
+ "current",
+ "latest",
+ "main",
+ "more",
+ "now",
+ "position",
+ "right",
+ "that",
+ "this",
+ "view",
+}
+
+
+def _current_value_meaning_tokens(value: Any) -> set[str]:
+ return {
+ token
+ for token in _tokens(value)
+ if len(token) >= 4 and token not in _CURRENT_VALUE_COVERAGE_STOP_TOKENS
+ }
+
+
+def _record_covers_current_value(*, record: Mapping[str, Any], slot_key: str, current_value: str) -> bool:
+ if _safe_text(record.get("slot_key", "")).lower() != _safe_text(slot_key).lower():
+ return False
+ if _safe_text(record.get("category", "")).lower() == "question":
+ return False
+ target_tokens = _current_value_meaning_tokens(current_value)
+ if not target_tokens:
+ return False
+ metadata = dict(record.get("metadata", {}) or {})
+ record_text = " ".join(
+ [
+ _safe_text(record.get("value", "")),
+ _safe_text(metadata.get("source_span", "")),
+ _safe_text(metadata.get("event_text", "")),
+ _safe_text(metadata.get("event_phrase", "")),
+ ]
+ )
+ covered_tokens = _current_value_meaning_tokens(record_text)
+ coverage = len(target_tokens & covered_tokens) / max(1, len(target_tokens))
+ return coverage >= 0.85
+
+
+def _sidecar_memory_subject_hint(sidecar_hints: Mapping[str, Any] | None, *, session_slug: str) -> tuple[str, str, str]:
+ hints = dict(sidecar_hints or {})
+ raw_slot_hint = hints.get("memory_slot", {})
+ slot_hint = dict(raw_slot_hint or {}) if isinstance(raw_slot_hint, Mapping) else {}
+ subject = _safe_text(
+ slot_hint.get("subject", "")
+ or slot_hint.get("prompt_label", "")
+ or slot_hint.get("label", "")
+ )
+ subject_signature = _safe_text(slot_hint.get("subject_signature", "")) or (_slug(subject) if subject else "")
+ canonical_slot_key = _safe_text(slot_hint.get("canonical_slot_key", ""))
+ if not canonical_slot_key and subject_signature:
+ canonical_slot_key = f"{session_slug}.subject.{subject_signature}"
+ if subject and len(_tokens(subject)) < 2:
+ subject = ""
+ subject_signature = ""
+ canonical_slot_key = ""
+ return subject, subject_signature, canonical_slot_key
+
+
+def _looks_like_speech_act_memory(*, value: str, source_span: str) -> bool:
+ """Reject proposals that describe the user act of asking instead of durable memory."""
+ value_text = _safe_text(value).lower()
+ span_text = _safe_text(source_span)
+ if not value_text:
+ return False
+ if _SPEECH_ACT_MEMORY_RE.search(value_text):
+ return True
+ if span_text.endswith("?") and value_text.startswith(("asked ", "asks ", "asking ", "requested ", "requesting ")):
+ return True
+ return False
+
+
+def _looks_like_transient_assistant_directive(*, value: str, source_span: str, turn_text: str) -> bool:
+ """Reject one-shot instructions only when the proposal itself is transient."""
+ value_text = _safe_text(value)
+ span_text = _safe_text(source_span)
+ combined = " ".join(part for part in (value_text, span_text) if part)
+ if not combined:
+ return False
+ durable_profile_assertion = bool(
+ _FIRST_PERSON_MEMORY_ASSERTION_RE.search(span_text)
+ or _FIRST_PERSON_MEMORY_ASSERTION_RE.search(_safe_text(turn_text))
+ or _PROFILE_VALUE_MARKERS.search(combined)
+ )
+ if _TRANSIENT_ASSISTANT_DIRECTIVE_RE.search(combined):
+ if durable_profile_assertion and not re.search(
+ r"\b(?:answer|respond|reply|summari[sz]e|translate|compare|explain|based\s+only|only\s+use|do\s+not\s+(?:mention|reveal|include|use))\b",
+ combined,
+ flags=re.IGNORECASE,
+ ):
+ return False
+ return True
+ # Only reject when the proposal itself is transient. Long chunks can mix
+ # durable facts with nearby requests such as "recommend" or "give advice";
+ # scanning the whole chunk here rejects valid writer proposals.
+ return False
+
+
+class TurnIntentGate:
+ """Use whole-turn intent as the gate before mutating writer proposals."""
+
+ INTENT_ALIASES = {
+ "assertion": "memory_assertion",
+ "memory": "memory_assertion",
+ "memory_write": "memory_assertion",
+ "durable_memory": "memory_assertion",
+ "question": "question_only",
+ "question_only_turn": "question_only",
+ "recall_question": "question_only",
+ "answer_request": "question_only",
+ "mixed": "mixed_question_with_memory",
+ "mixed_question": "mixed_question_with_memory",
+ "question_with_memory": "mixed_question_with_memory",
+ "question_like_with_memory_assertion": "mixed_question_with_memory",
+ "no_memory": "no_new_memory",
+ "none": "no_new_memory",
+ "empty": "no_new_memory",
+ "transient": "transient_directive",
+ "instruction": "transient_directive",
+ "directive": "transient_directive",
+ }
+ MEMORY_LABELS = {"memory_assertion", "mixed_question_with_memory"}
+ QUESTION_ONLY_LABELS = {"question_only"}
+ BLOCKED_LABELS = {"question_only", "no_new_memory", "transient_directive"}
+
+ @staticmethod
+ def _coerce_bool(value: Any, default: bool = False) -> bool:
+ if isinstance(value, bool):
+ return value
+ text = _safe_text(value).lower()
+ if text in {"true", "yes", "1", "allow", "allowed"}:
+ return True
+ if text in {"false", "no", "0", "deny", "blocked"}:
+ return False
+ return default
+
+ @staticmethod
+ def _heuristic_intent(text: str) -> Dict[str, Any]:
+ compact = _safe_text(text)
+ if not compact:
+ return {
+ "intent": "no_new_memory",
+ "question_like": False,
+ "memory_assertion": False,
+ "question_only": False,
+ "write_allowed": False,
+ "reason": "empty",
+ "source": "heuristic_fallback",
+ }
+ memory_assertion = bool(
+ _ASSERTIVE_MEMORY_CUE_RE.search(compact)
+ or _FIRST_PERSON_MEMORY_ASSERTION_RE.search(compact)
+ or _CJK_MEMORY_ASSERTION_RE.search(compact)
+ )
+ question_like = bool(
+ compact.endswith("?")
+ or compact.endswith("？")
+ or _QUESTION_ONLY_RE.search(compact)
+ or any(marker in compact for marker in _CJK_QUESTION_MARKERS)
+ )
+ if question_like and memory_assertion:
+ intent = "mixed_question_with_memory"
+ elif question_like:
+ intent = "question_only"
+ elif memory_assertion:
+ intent = "memory_assertion"
+ else:
+ intent = "plain_statement"
+ return {
+ "intent": intent,
+ "question_like": question_like,
+ "memory_assertion": memory_assertion,
+ "question_only": bool(question_like and not memory_assertion),
+ "write_allowed": bool(memory_assertion or not question_like),
+ "reason": intent,
+ "source": "heuristic_fallback",
+ }
+
+ @classmethod
+ def _normalize_writer_intent(
+ cls,
+ writer_intent: Mapping[str, Any] | None,
+ ) -> Dict[str, Any]:
+ if not isinstance(writer_intent, Mapping):
+ return {}
+ raw_label = _safe_text(
+ writer_intent.get("turn_intent")
+ or writer_intent.get("intent")
+ or writer_intent.get("label")
+ or writer_intent.get("decision")
+ ).lower()
+ label = raw_label.replace("-", "_").replace(" ", "_")
+ label = cls.INTENT_ALIASES.get(label, label)
+ valid_labels = cls.MEMORY_LABELS | cls.QUESTION_ONLY_LABELS | cls.BLOCKED_LABELS | {"plain_statement"}
+ if label not in valid_labels:
+ return {}
+ memory_assertion = cls._coerce_bool(writer_intent.get("memory_assertion"), label in cls.MEMORY_LABELS)
+ question_only = cls._coerce_bool(writer_intent.get("question_only"), label in cls.QUESTION_ONLY_LABELS)
+ question_like = cls._coerce_bool(
+ writer_intent.get("question_like"),
+ label in {"question_only", "mixed_question_with_memory"},
+ )
+ write_allowed = cls._coerce_bool(writer_intent.get("write_allowed"), label not in cls.BLOCKED_LABELS)
+ if label in cls.MEMORY_LABELS:
+ write_allowed = True
+ memory_assertion = True
+ question_only = False
+ if label in cls.BLOCKED_LABELS:
+ write_allowed = False
+ memory_assertion = False
+ question_only = label == "question_only"
+ return {
+ "intent": label,
+ "question_like": question_like,
+ "memory_assertion": memory_assertion,
+ "question_only": question_only,
+ "write_allowed": write_allowed,
+ "reason": _safe_text(writer_intent.get("reason", "")) or label,
+ "source": "writer_turn_intent",
+ "writer_raw_intent": dict(writer_intent),
+ }
+
+ @classmethod
+ def classify_turn(
+ cls,
+ text: str,
+ *,
+ writer_intent: Mapping[str, Any] | None = None,
+ ) -> Dict[str, Any]:
+ fallback = cls._heuristic_intent(text)
+ normalized = cls._normalize_writer_intent(writer_intent)
+ if not normalized:
+ return fallback
+ normalized["heuristic_fallback"] = fallback
+ return normalized
+
+ @staticmethod
+ def proposal_policy(
+ *,
+ turn_intent: Mapping[str, Any],
+ value: str,
+ source_span: str,
+ turn_text: str,
+ ) -> Dict[str, Any]:
+ speech_act_memory = _looks_like_speech_act_memory(value=value, source_span=source_span)
+ transient_directive = _looks_like_transient_assistant_directive(
+ value=value,
+ source_span=source_span,
+ turn_text=turn_text,
+ )
+ if transient_directive:
+ return {
+ "decision": "reject",
+ "reject_reason": "transient_assistant_directive",
+ "force_question": False,
+ "speech_act_memory": speech_act_memory,
+ "transient_directive": True,
+ "turn_intent_reason": _safe_text(turn_intent.get("reason", "")),
+ }
+ force_question = bool(turn_intent.get("question_only", False) or speech_act_memory)
+ if force_question:
+ reason = "speech_act_memory" if speech_act_memory else "turn_question_only"
+ return {
+ "decision": "force_question",
+ "reject_reason": "",
+ "force_question": True,
+ "force_question_reason": reason,
+ "speech_act_memory": speech_act_memory,
+ "transient_directive": False,
+ "turn_intent_reason": _safe_text(turn_intent.get("reason", "")),
+ }
+ return {
+ "decision": "preserve_writer_category",
+ "reject_reason": "",
+ "force_question": False,
+ "speech_act_memory": speech_act_memory,
+ "transient_directive": False,
+ "turn_intent_reason": _safe_text(turn_intent.get("reason", "")),
+ }
+
+
+def _turn_intent_gate(text: str) -> Dict[str, Any]:
+ return TurnIntentGate.classify_turn(text)
+
+
+def _is_question_only_turn(text: str) -> bool:
+ return bool(_turn_intent_gate(text).get("question_only", False))
+
+
+def _source_kind_for(category: str, semantic_slot: str, relation: str) -> tuple[str, str]:
+ normalized_category = _safe_text(category).lower()
+ normalized_slot = _safe_text(semantic_slot).lower()
+ normalized_relation = _safe_text(relation).lower()
+ if (
+ normalized_category in {"question", "interaction_intent"}
+ or normalized_slot in {"question", "interaction_intent", "user_question"}
+ or "question" in normalized_relation
+ or "ask" in normalized_relation
+ ):
+ return "question", "public_dialog_question"
+ if normalized_category == "profile" or normalized_slot in _PROFILE_SLOTS:
+ return "profile", "public_dialog_profile"
+ if normalized_category == "time" or normalized_slot == "event_time" or "time" in normalized_relation or "date" in normalized_relation:
+ return "fact", "public_dialog_time"
+ if normalized_category == "event":
+ return "event", "public_dialog_event"
+ return normalized_category or "fact", f"public_dialog_{normalized_category or 'fact'}"
+
+
+def _profile_intent_from_proposal(
+ *,
+ proposal: Mapping[str, Any],
+ category: str,
+ record_category: str,
+ semantic_slot: str,
+ relation: str,
+ value: str,
+ source_span: str,
+ subject: str,
+ turn_intent: Mapping[str, Any],
+) -> Dict[str, Any]:
+ if record_category == "question":
+ return {"enabled": False, "reason": "question_record", "create_profile_shadow": False}
+ normalized_category = _safe_text(category).lower()
+ normalized_record_category = _safe_text(record_category).lower()
+ slot_text = _safe_text(semantic_slot).lower()
+ relation_text = _safe_text(relation).lower()
+ combined = " ".join([slot_text, relation_text, _safe_text(value).lower(), _safe_text(source_span).lower()])
+ explicit_profile_type = _safe_text(proposal.get("profile_type", "")).lower()
+ explicit_domain = _safe_text(proposal.get("profile_domain", "") or proposal.get("profile_domain_label", ""))
+ slot_tokens = set(_tokens(slot_text.replace("_", " ")))
+ reasons: List[str] = []
+ enabled = False
+ if normalized_category in _PROFILE_DIRECT_CATEGORIES or normalized_record_category in _PROFILE_DIRECT_CATEGORIES:
+ enabled = True
+ reasons.append("profile_category")
+ if explicit_profile_type:
+ enabled = True
+ reasons.append("writer_profile_type")
+ if slot_tokens & _PROFILE_INTENT_SLOT_MARKERS:
+ enabled = True
+ reasons.append("profile_slot_marker")
+ if bool(turn_intent.get("memory_assertion")) and _PROFILE_VALUE_MARKERS.search(combined):
+ enabled = True
+ reasons.append("profile_value_marker")
+ if not enabled:
+ return {"enabled": False, "reason": "no_profile_signal", "create_profile_shadow": False}
+
+ profile_type = explicit_profile_type
+ if profile_type not in {"setup", "preference", "constraint", "goal", "avoid", "usage_context"}:
+ if normalized_record_category == "preference" or "prefer" in combined or "like" in combined:
+ profile_type = "preference"
+ elif normalized_record_category == "goal" or any(marker in combined for marker in ("goal", "want", "plan", "intend")):
+ profile_type = "goal"
+ elif normalized_record_category == "constraint" or any(marker in combined for marker in ("constraint", "must", "cannot", "need", "avoid")):
+ profile_type = "avoid" if "avoid" in combined or "dislike" in combined else "constraint"
+ elif slot_tokens & _PROFILE_SETUP_SLOT_MARKERS or any(marker in combined for marker in ("current", "setup", "brand", "uses", "user is", "equipment", "gear")):
+ profile_type = "setup"
+ else:
+ profile_type = "usage_context"
+
+ profile_domain = explicit_domain
+ if not profile_domain:
+ if slot_text and slot_text not in {"profile", "preference", "goal", "constraint", "event", "fact", "status"}:
+ profile_domain = slot_text.replace("_", " ")
+ elif subject:
+ profile_domain = subject
+ else:
+ domain_tokens = [
+ token
+ for token in _tokens(value)
+ if token not in {"user", "uses", "prefers", "likes", "wants", "needs", "has", "current"}
+ ]
+ profile_domain = " ".join(domain_tokens[:4]) if domain_tokens else "general"
+ return {
+ "enabled": True,
+ "reason": ",".join(_dedupe_texts(reasons, max_items=6)),
+ "profile_type": profile_type,
+ "profile_domain": profile_domain,
+ "create_profile_shadow": normalized_record_category not in _PROFILE_DIRECT_CATEGORIES,
+ "profile_writer_layer_version": "turn_intent_profile_writer_v1",
+ }
+
+
+@dataclass(slots=True)
+class MemoryWriteGateResult:
+ payload: Dict[str, Any]
+ accepted_count: int
+ rejected: List[Dict[str, Any]]
+ suspected_count: int = 0
+
+
+class SemanticMemoryWriterError(RuntimeError):
+ """Raised when the required LLM semantic writer cannot produce gated writes."""
+
+
+def _suspect_source_kind(source_kind: str) -> str:
+ normalized = _safe_text(source_kind) or "memory"
+ return normalized if normalized.startswith("suspect_") else f"suspect_{normalized}"
+
+
+class DeterministicMemoryWriteGate:
+ """Validate LLM memory write proposals before they can mutate memory state."""
+
+ def __init__(self, *, min_grounding_score: float = 0.5) -> None:
+ self.min_grounding_score = max(0.0, float(min_grounding_score))
+
+ def _suspect_record(
+ self,
+ *,
+ index: int,
+ category: str,
+ value: str,
+ source_span: str,
+ semantic_slot: str,
+ relation: str,
+ source_kind: str,
+ slot_key: str,
+ event_signature: str,
+ anchors: Sequence[Any],
+ common_metadata: Mapping[str, Any],
+ proposal: Mapping[str, Any],
+ grounding: float,
+ value_grounding: float,
+ min_value_grounding: float,
+ suspicion_reason: str,
+ compact_text: str,
+ subject: str = "",
+ subject_signature: str = "",
+ ) -> Dict[str, Any]:
+ return {
+ "category": category or "fact",
+ "slot_key": slot_key,
+ "value": value or source_span,
+ "anchors": _dedupe_texts(anchors, max_items=8),
+ "relation": relation or f"{category or 'fact'}_memory",
+ "source_kind": _suspect_source_kind(source_kind),
+ "state": "suspect",
+ "salience": min(0.58, float(proposal.get("salience", 0.46) or 0.46)),
+ "confidence": min(0.56, float(proposal.get("confidence", 0.42) or 0.42)),
+ "metadata": {
+ **dict(common_metadata),
+ "content_variant": "llm_semantic_suspect_write",
+ "semantic_slot": semantic_slot or category,
+ "target_status": _safe_text(proposal.get("target_status", "")),
+ "source_span": source_span,
+ "event_phrase": _safe_text(proposal.get("event_phrase", "")) or value or source_span,
+ "event_text": _safe_text(proposal.get("event_text", "")) or _safe_text(proposal.get("event_phrase", "")) or value or source_span,
+ "event_signature": event_signature,
+ "subject": subject,
+ "subject_signature": subject_signature,
+ "canonical_slot_key": slot_key,
+ "allow_parallel_state": False if subject_signature else bool(proposal.get("allow_parallel_state", True)),
+ "time_expression_span": _safe_text(proposal.get("time_expression_span", "")),
+ "time_granularity": _safe_text(proposal.get("time_granularity", "")),
+ "resolved_time_value": _safe_text(proposal.get("resolved_time_value", "")),
+ "time_display_value": _safe_text(proposal.get("time_display_value", "")),
+ "resolved_date": _safe_text(proposal.get("resolved_date", "")),
+ "memory_gate_decision": "suspect_buffer",
+ "suspicion_reason": suspicion_reason,
+ "candidate_source_kind": source_kind,
+ "source_span_exact_current_turn": _span_is_grounded_in_current_turn(
+ source_span=source_span,
+ current_turn_text=compact_text,
+ ),
+ "grounding_score": round(float(grounding), 6),
+ "value_grounding_score": round(float(value_grounding), 6),
+ "min_value_grounding_score": round(float(min_value_grounding), 6),
+ "llm_write_proposal_index": index,
+ "suspect_support_count": 1,
+ "suspect_support_turns": [int(proposal.get("turn_index", 0) or 0)],
+ "tmcra_stack_metadata": dict(proposal.get("tmcra_stack_metadata", {}) or {}),
+ "write_decision": _safe_text(proposal.get("write_decision", "")),
+ "proposal_intent_gate": dict(proposal.get("proposal_intent_gate", {}) or {}),
+ **profile_candidate_metadata(
+ category=category,
+ semantic_slot=semantic_slot,
+ relation=relation,
+ value=value or source_span,
+ source_span=source_span,
+ slot_key=slot_key,
+ anchors=anchors,
+ subject=subject,
+ subject_signature=subject_signature,
+ proposal=proposal,
+ ),
+ },
+ }
+
+ def build_payload(
+ self,
+ *,
+ proposals: Sequence[Mapping[str, Any]],
+ text: str,
+ raw_text: str,
+ speaker: str,
+ session_key: str,
+ turn_index: int,
+ timestamp: str = "",
+ dia_id: str = "",
+ sidecar_hints: Mapping[str, Any] | None = None,
+ auxiliary_evidence_texts: Sequence[Any] | None = None,
+ writer_metadata: Mapping[str, Any] | None = None,
+ ) -> MemoryWriteGateResult:
+ rendered_text = _safe_text(text)
+ compact_text = _safe_text(raw_text) or rendered_text
+ auxiliary_texts = _dedupe_texts(auxiliary_evidence_texts or [])
+ source_context = compact_text if not auxiliary_texts else compact_text + "\nAuxiliary evidence: " + " | ".join(auxiliary_texts)
+ writer_turn_intent = dict((writer_metadata or {}).get("writer_turn_intent", {}) or {})
+ turn_intent = TurnIntentGate.classify_turn(compact_text, writer_intent=writer_turn_intent)
+ question_only_turn = bool(turn_intent.get("question_only", False))
+ session_slug = _slug(session_key)
+ sidecar_subject, sidecar_subject_signature, sidecar_canonical_slot_key = _sidecar_memory_subject_hint(
+ sidecar_hints,
+ session_slug=session_slug,
+ )
+ common_metadata = {
+ "speaker": _safe_text(speaker),
+ "timestamp": _safe_text(timestamp),
+ "session_key": _safe_text(session_key),
+ "dia_id": _safe_text(dia_id),
+ "event_id": f"event::{_safe_text(dia_id)}" if _safe_text(dia_id) else "",
+ "source": "public_benchmark",
+ "raw_text": compact_text,
+ "source_turn_text": source_context,
+ "write_path": "llm_semantic_writer_gate",
+ "memory_writer_role": "llm_semantic_writer",
+ "turn_extractor_role": "sidecar_hints",
+ "semantic_writer_metadata": dict(writer_metadata or {}),
+ "sidecar_hint_metadata": dict(sidecar_hints or {}).get("metadata", {}),
+ "turn_intent_gate": dict(turn_intent),
+ }
+ records: List[Dict[str, Any]] = [
+ {
+ "category": "fact",
+ "slot_key": f"{session_slug}.turn_{max(1, int(turn_index or 1))}",
+ "value": rendered_text,
+ "anchors": _dedupe_texts([speaker, timestamp]),
+ "relation": "source_turn_grounding",
+ "source_kind": "public_dialog_turn",
+ "state": "evidence",
+ "salience": 0.92,
+ "confidence": 1.0,
+ "metadata": {
+ **common_metadata,
+ "content_variant": "source_turn",
+ "memory_gate_decision": "source_grounding",
+ },
+ }
+ ]
+ rejected: List[Dict[str, Any]] = []
+ suspect_records: List[Dict[str, Any]] = []
+ seen = set()
+ for index, raw_proposal in enumerate(proposals):
+ proposal = dict(raw_proposal or {})
+ if proposal.get("write") is False:
+ rejected.append({"index": index, "reason": "proposal_write_false"})
+ continue
+ category = _safe_text(proposal.get("category", "")).lower().replace("-", "_")
+ if category in _WRITE_CATEGORY_ALIASES:
+ proposal["original_category"] = category
+ proposal["category"] = _WRITE_CATEGORY_ALIASES[category]
+ proposal["semantic_slot"] = _safe_text(proposal.get("semantic_slot", "")) or category
+ category = _safe_text(proposal.get("category", "")).lower().replace("-", "_")
+ if category not in ALLOWED_WRITE_CATEGORIES:
+ rejected.append({"index": index, "reason": "category_not_allowed", "category": category})
+ continue
+ source_span = _safe_text(proposal.get("source_span", ""))
+ if not source_span:
+ rejected.append({"index": index, "reason": "missing_source_span"})
+ continue
+ value = _safe_text(proposal.get("value", "")) or source_span
+ if not value:
+ rejected.append({"index": index, "reason": "missing_value"})
+ continue
+ proposal_policy = TurnIntentGate.proposal_policy(
+ turn_intent=turn_intent,
+ value=value,
+ source_span=source_span,
+ turn_text=compact_text,
+ )
+ proposal["proposal_intent_gate"] = dict(proposal_policy)
+ if proposal_policy.get("decision") == "reject":
+ rejected.append(
+ {
+ "index": index,
+ "reason": _safe_text(proposal_policy.get("reject_reason", "")) or "turn_intent_gate_reject",
+ "turn_intent_reason": _safe_text(proposal_policy.get("turn_intent_reason", "")),
+ }
+ )
+ continue
+ if bool(proposal_policy.get("force_question", False)):
+ category = "question"
+ proposal["category"] = "question"
+ proposal["semantic_slot"] = _safe_text(proposal.get("semantic_slot", "")) or "interaction_intent"
+ proposal["relation"] = _safe_text(proposal.get("relation", "")) or "user_question"
+ proposal["target_status"] = ""
+ if _SECRET_RE.search(value) or _SECRET_RE.search(source_span):
+ rejected.append({"index": index, "reason": "secret_like_value"})
+ continue
+ grounding = _grounding_score(value=value, source_span=source_span, source_context=source_context)
+ value_grounding = _value_grounding_score(value=value, source_context=source_context)
+ semantic_slot = _safe_text(proposal.get("semantic_slot", ""))
+ relation = _safe_text(proposal.get("relation", "")) or f"{category}_fact"
+ record_category, source_kind = _source_kind_for(category, semantic_slot, relation)
+ if record_category == "question":
+ # Question memory should preserve the user's actual question, not a
+ # lossy LLM summary such as "check evaluation view".
+ source_span = compact_text
+ value = compact_text
+ grounding = 1.0
+ value_grounding = 1.0
+ # Values may be concise normalizations of a longer exact source span. Keep a
+ # floor high enough to reject hallucinated slot values, but do not reject
+ # grounded summaries when the source_span itself is copied from the turn.
+ min_value_grounding = self.min_grounding_score if record_category == "question" else max(self.min_grounding_score, 0.65)
+ if (
+ record_category != "question"
+ and bool(turn_intent.get("write_allowed", False))
+ and value_grounding < min_value_grounding
+ and not _span_is_grounded_in_current_turn(source_span=source_span, current_turn_text=compact_text)
+ and _current_turn_can_repair_proposal(
+ current_turn_text=compact_text,
+ value=value,
+ source_span=source_span,
+ )
+ ):
+ proposal["source_span_repair"] = "current_turn_exact_from_turn_intent"
+ proposal["original_source_span"] = source_span
+ proposal["original_value"] = value
+ source_span = compact_text
+ value = compact_text
+ grounding = 1.0
+ value_grounding = 1.0
+ event_signature = _safe_text(proposal.get("event_signature", "")) or compute_public_event_signature(
+ source_span or value,
+ speaker=_safe_text(speaker),
+ semantic_slot=semantic_slot or record_category,
+ )
+ # Prefer the full current turn so nested values like
+ # "my model size view is now my refined deployment view is ..."
+ # keep the outer durable slot instead of drifting to the inner
+ # wording emitted by the writer.
+ extracted_subject = "" if record_category == "question" else _extract_memory_subject(compact_text, source_span, value)
+ subject = extracted_subject
+ subject_from_sidecar = False
+ if record_category != "question" and sidecar_subject:
+ subject = sidecar_subject
+ subject_from_sidecar = _slug(extracted_subject) != (sidecar_subject_signature or _slug(sidecar_subject))
+ subject_signature = _slug(subject) if subject else ""
+ if subject_from_sidecar and sidecar_subject_signature:
+ subject_signature = sidecar_subject_signature
+ canonical_slot_key = f"{session_slug}.subject.{subject_signature}" if subject_signature else ""
+ if subject_from_sidecar and sidecar_canonical_slot_key:
+ canonical_slot_key = sidecar_canonical_slot_key
+ slot_key = _safe_text(proposal.get("slot_key", ""))
+ if canonical_slot_key:
+ slot_key = canonical_slot_key
+ elif not slot_key:
+ slot_key = f"{session_slug}.turn_{max(1, int(turn_index or 1))}.{record_category}.{index + 1}"
+ anchors = _dedupe_texts(
+ [
+ speaker,
+ timestamp,
+ subject,
+ *list(proposal.get("anchors", []) or []),
+ *event_signature.split()[:4],
+ ],
+ max_items=8,
+ )
+ profile_intent_gate = _profile_intent_from_proposal(
+ proposal=proposal,
+ category=category,
+ record_category=record_category,
+ semantic_slot=semantic_slot,
+ relation=relation,
+ value=value,
+ source_span=source_span,
+ subject=subject,
+ turn_intent=turn_intent,
+ )
+ if grounding < self.min_grounding_score:
+ rejected.append(
+ {
+ "index": index,
+ "reason": "ungrounded",
+ "grounding_score": round(float(grounding), 6),
+ }
+ )
+ if record_category != "question":
+ suspect_records.append(
+ self._suspect_record(
+ index=index,
+ category=record_category,
+ value=value,
+ source_span=source_span,
+ semantic_slot=semantic_slot,
+ relation=relation,
+ source_kind=source_kind,
+ slot_key=slot_key,
+ event_signature=event_signature,
+ anchors=anchors,
+ common_metadata=common_metadata,
+ proposal={**proposal, "turn_index": turn_index},
+ grounding=grounding,
+ value_grounding=value_grounding,
+ min_value_grounding=min_value_grounding,
+ suspicion_reason="ungrounded",
+ compact_text=compact_text,
+ subject=subject,
+ subject_signature=subject_signature,
+ )
+ )
+ continue
+ canonicalized_from_value = ""
+ canonicalization_reason = ""
+ original_value_grounding = value_grounding
+ if (
+ record_category != "question"
+ and value_grounding < min_value_grounding
+ and _source_span_can_stand_as_value(
+ source_span=source_span,
+ value=value,
+ current_turn_text=compact_text,
+ )
+ ):
+ canonicalized_from_value = value
+ canonicalization_reason = "low_value_grounding_source_span_exact_current_turn"
+ value = source_span
+ value_grounding = 1.0
+ if value_grounding < min_value_grounding:
+ rejected.append(
+ {
+ "index": index,
+ "reason": "ungrounded_value",
+ "grounding_score": round(float(grounding), 6),
+ "value_grounding_score": round(float(value_grounding), 6),
+ "min_value_grounding_score": round(float(min_value_grounding), 6),
+ "source_span_token_count": len(set(_tokens(source_span))),
+ "value_token_count": len(set(_tokens(value))),
+ }
+ )
+ if record_category != "question":
+ suspect_records.append(
+ self._suspect_record(
+ index=index,
+ category=record_category,
+ value=value,
+ source_span=source_span,
+ semantic_slot=semantic_slot,
+ relation=relation,
+ source_kind=source_kind,
+ slot_key=slot_key,
+ event_signature=event_signature,
+ anchors=anchors,
+ common_metadata=common_metadata,
+ proposal={**proposal, "turn_index": turn_index},
+ grounding=grounding,
+ value_grounding=value_grounding,
+ min_value_grounding=min_value_grounding,
+ suspicion_reason="ungrounded_value",
+ compact_text=compact_text,
+ subject=subject,
+ subject_signature=subject_signature,
+ )
+ )
+ continue
+ dedupe_key = (
+ record_category,
+ slot_key.lower(),
+ re.sub(r"\s+", " ", value.lower()).strip(),
+ )
+ if dedupe_key in seen:
+ rejected.append({"index": index, "reason": "duplicate"})
+ continue
+ seen.add(dedupe_key)
+ tmcra_stack_metadata = dict(proposal.get("tmcra_stack_metadata", {}) or {})
+ proposal_state = _safe_text(proposal.get("state", "")).lower()
+ proposal_write_decision = (
+ _safe_text(proposal.get("write_decision", "")).lower()
+ or _safe_text(proposal.get("memory_gate_decision", "")).lower()
+ or _safe_text(tmcra_stack_metadata.get("memory_gate_decision", "")).lower()
+ )
+ tmcra_suspect_buffered = bool(tmcra_stack_metadata.get("tmcra_suspect_buffered"))
+ if proposal_state == "suspect" or proposal_write_decision == "suspect_buffer" or tmcra_suspect_buffered:
+ suspect_records.append(
+ self._suspect_record(
+ index=index,
+ category=record_category,
+ value=value,
+ source_span=source_span,
+ semantic_slot=semantic_slot,
+ relation=relation,
+ source_kind=source_kind,
+ slot_key=slot_key,
+ event_signature=event_signature,
+ anchors=anchors,
+ common_metadata=common_metadata,
+ proposal={**proposal, "turn_index": turn_index},
+ grounding=grounding,
+ value_grounding=value_grounding,
+ min_value_grounding=min_value_grounding,
+ suspicion_reason=_safe_text(
+ tmcra_stack_metadata.get("suspicion_reason", "")
+ or proposal.get("suspicion_reason", "")
+ or "writer_marked_suspect"
+ ),
+ compact_text=compact_text,
+ subject=subject,
+ subject_signature=subject_signature,
+ )
+ )
+ continue
+ profile_proposal = {
+ **proposal,
+ "profile_type": _safe_text(profile_intent_gate.get("profile_type", "")) or proposal.get("profile_type", ""),
+ "profile_domain": _safe_text(profile_intent_gate.get("profile_domain", "")) or proposal.get("profile_domain", ""),
+ }
+ metadata = {
+ **common_metadata,
+ "content_variant": "llm_semantic_write",
+ "raw_text": source_span or value,
+ "source_turn_text": source_span or value,
+ "semantic_slot": semantic_slot or record_category,
+ "target_status": _safe_text(proposal.get("target_status", "")),
+ "source_span": source_span or value,
+ "event_phrase": _safe_text(proposal.get("event_phrase", "")) or value,
+ "event_text": _safe_text(proposal.get("event_text", "")) or _safe_text(proposal.get("event_phrase", "")) or value,
+ "event_signature": event_signature,
+ "extracted_subject": extracted_subject,
+ "subject": subject,
+ "subject_signature": subject_signature,
+ "canonical_slot_key": canonical_slot_key or slot_key,
+ "subject_from_sidecar": bool(subject_from_sidecar),
+ "sidecar_subject_hint": sidecar_subject,
+ "sidecar_subject_signature_hint": sidecar_subject_signature,
+ "allow_parallel_state": False if canonical_slot_key else bool(proposal.get("allow_parallel_state", True)),
+ "time_expression_span": _safe_text(proposal.get("time_expression_span", "")),
+ "time_granularity": _safe_text(proposal.get("time_granularity", "")),
+ "resolved_time_value": _safe_text(proposal.get("resolved_time_value", "")),
+ "time_display_value": _safe_text(proposal.get("time_display_value", "")),
+ "resolved_date": _safe_text(proposal.get("resolved_date", "")),
+ "memory_gate_decision": _safe_text(proposal.get("write_decision", "")) or "parallel_active",
+ "grounding_score": round(float(grounding), 6),
+ "value_grounding_score": round(float(value_grounding), 6),
+ "original_value_grounding_score": round(float(original_value_grounding), 6),
+ "canonicalized_from_value": canonicalized_from_value,
+ "canonicalization_reason": canonicalization_reason,
+ "source_span_repair": _safe_text(proposal.get("source_span_repair", "")),
+ "original_source_span": _safe_text(proposal.get("original_source_span", "")),
+ "original_value": _safe_text(proposal.get("original_value", "")),
+ "llm_write_proposal_index": index,
+ "proposal_intent_gate": dict(proposal_policy),
+ "profile_intent_gate": dict(profile_intent_gate),
+ **profile_candidate_metadata(
+ category=record_category,
+ semantic_slot=semantic_slot,
+ relation=relation,
+ value=value,
+ source_span=source_span,
+ slot_key=slot_key,
+ anchors=anchors,
+ subject=subject,
+ subject_signature=subject_signature,
+ proposal=profile_proposal,
+ ),
+ }
+ parent_record = {
+ "category": record_category,
+ "slot_key": slot_key,
+ "value": value,
+ "anchors": anchors,
+ "relation": relation,
+ "source_kind": source_kind,
+ "state": "evidence" if record_category == "question" else _safe_text(proposal.get("state", "")),
+ "salience": float(proposal.get("salience", 0.94) or 0.94),
+ "confidence": min(1.0, max(0.0, float(proposal.get("confidence", 0.86) or 0.86))),
+ "metadata": metadata,
+ }
+ facet_records = _build_event_facet_records(
+ parent_record=parent_record,
+ proposal=proposal,
+ common_metadata=common_metadata,
+ source_context=source_context,
+ current_turn_text=compact_text,
+ speaker=speaker,
+ timestamp=timestamp,
+ turn_index=turn_index,
+ proposal_index=index,
+ )
+ if facet_records:
+ metadata["facet_types"] = _dedupe_texts(
+ [dict(item.get("metadata", {}) or {}).get("facet_type", "") for item in facet_records],
+ max_items=8,
+ )
+ metadata["facet_record_count"] = len(facet_records)
+ records.append(parent_record)
+ records.extend(facet_records)
+ suppress_profile_shadow_for_sidecar_subject = bool(
+ sidecar_subject
+ and sidecar_canonical_slot_key
+ and _safe_text(slot_key).lower() == _safe_text(sidecar_canonical_slot_key).lower()
+ )
+ if bool(profile_intent_gate.get("create_profile_shadow", False)) and not suppress_profile_shadow_for_sidecar_subject:
+ shadow_profile_type = _safe_text(profile_intent_gate.get("profile_type", "")) or "usage_context"
+ shadow_profile_domain = _safe_text(profile_intent_gate.get("profile_domain", "")) or semantic_slot or subject or "general"
+ shadow_slot_key = (
+ f"{session_slug}.turn_{max(1, int(turn_index or 1))}."
+ f"profile.{_slug(shadow_profile_type)}.{_slug(shadow_profile_domain)}.{index + 1}"
+ )
+ shadow_anchors = _dedupe_texts(
+ [
+ *anchors,
+ shadow_profile_type,
+ shadow_profile_domain,
+ ],
+ max_items=10,
+ )
+ shadow_proposal = {
+ **proposal,
+ "profile_type": shadow_profile_type,
+ "profile_domain": shadow_profile_domain,
+ "profile_candidate_status": "writer_shadow",
+ }
+ shadow_metadata = {
+ **common_metadata,
+ "content_variant": "profile_shadow_from_writer",
+ "raw_text": source_span or value,
+ "source_turn_text": source_span or value,
+ "semantic_slot": semantic_slot or shadow_profile_domain,
+ "target_status": _safe_text(proposal.get("target_status", "")),
+ "source_span": source_span or value,
+ "event_phrase": _safe_text(proposal.get("event_phrase", "")) or value,
+ "event_text": _safe_text(proposal.get("event_text", "")) or _safe_text(proposal.get("event_phrase", "")) or value,
+ "event_signature": event_signature,
+ "extracted_subject": extracted_subject,
+ "subject": subject or shadow_profile_domain,
+ "subject_signature": subject_signature or _slug(subject or shadow_profile_domain),
+ "canonical_slot_key": shadow_slot_key,
+ "subject_from_sidecar": bool(subject_from_sidecar),
+ "sidecar_subject_hint": sidecar_subject,
+ "sidecar_subject_signature_hint": sidecar_subject_signature,
+ "allow_parallel_state": False,
+ "time_expression_span": _safe_text(proposal.get("time_expression_span", "")),
+ "time_granularity": _safe_text(proposal.get("time_granularity", "")),
+ "resolved_time_value": _safe_text(proposal.get("resolved_time_value", "")),
+ "time_display_value": _safe_text(proposal.get("time_display_value", "")),
+ "resolved_date": _safe_text(proposal.get("resolved_date", "")),
+ "memory_gate_decision": "profile_shadow_from_turn_intent",
+ "grounding_score": round(float(grounding), 6),
+ "value_grounding_score": round(float(value_grounding), 6),
+ "original_value_grounding_score": round(float(original_value_grounding), 6),
+ "canonicalized_from_value": canonicalized_from_value,
+ "canonicalization_reason": canonicalization_reason,
+ "source_span_repair": _safe_text(proposal.get("source_span_repair", "")),
+ "original_source_span": _safe_text(proposal.get("original_source_span", "")),
+ "original_value": _safe_text(proposal.get("original_value", "")),
+ "llm_write_proposal_index": index,
+ "proposal_intent_gate": dict(proposal_policy),
+ "profile_intent_gate": dict(profile_intent_gate),
+ "profile_shadow_source_category": record_category,
+ "profile_shadow_source_slot_key": slot_key,
+ **profile_candidate_metadata(
+ category="profile",
+ semantic_slot=semantic_slot or shadow_profile_domain,
+ relation="profile_shadow",
+ value=value,
+ source_span=source_span,
+ slot_key=shadow_slot_key,
+ anchors=shadow_anchors,
+ subject=subject or shadow_profile_domain,
+ subject_signature=subject_signature or _slug(subject or shadow_profile_domain),
+ proposal=shadow_proposal,
+ ),
+ }
+ shadow_dedupe_key = (
+ "profile",
+ shadow_slot_key.lower(),
+ re.sub(r"\s+", " ", value.lower()).strip(),
+ )
+ if shadow_dedupe_key not in seen:
+ seen.add(shadow_dedupe_key)
+ records.append(
+ {
+ "category": "profile",
+ "slot_key": shadow_slot_key,
+ "value": value,
+ "anchors": shadow_anchors,
+ "relation": "profile_shadow",
+ "source_kind": "public_dialog_profile",
+ "state": _safe_text(proposal.get("state", "")),
+ "salience": min(0.98, max(0.86, float(proposal.get("salience", 0.9) or 0.9))),
+ "confidence": min(0.97, max(0.82, float(proposal.get("confidence", 0.84) or 0.84))),
+ "metadata": shadow_metadata,
+ }
+ )
+ if (
+ len(records) == 1
+ and bool(turn_intent.get("write_allowed", False))
+ and not question_only_turn
+ and _safe_text(turn_intent.get("source", "")).startswith("writer")
+ and not _SECRET_RE.search(compact_text)
+ ):
+ fallback_proposal = next((dict(item) for item in proposals if isinstance(item, Mapping)), {"category": "fact"})
+ fallback_category = _safe_text(fallback_proposal.get("category", "")).lower().replace("-", "_") or "fact"
+ if fallback_category in _WRITE_CATEGORY_ALIASES:
+ fallback_category = _WRITE_CATEGORY_ALIASES[fallback_category]
+ if fallback_category in ALLOWED_WRITE_CATEGORIES and fallback_category not in {"question", "interaction_intent"}:
+ fallback_semantic_slot = _safe_text(fallback_proposal.get("semantic_slot", "")) or fallback_category
+ fallback_relation = _safe_text(fallback_proposal.get("relation", "")) or f"{fallback_category}_memory"
+ fallback_record_category, fallback_source_kind = _source_kind_for(
+ fallback_category,
+ fallback_semantic_slot,
+ fallback_relation,
+ )
+ if fallback_record_category != "question" and len(set(_tokens(compact_text))) >= 4:
+ fallback_event_signature = compute_public_event_signature(
+ compact_text,
+ speaker=_safe_text(speaker),
+ semantic_slot=fallback_semantic_slot or fallback_record_category,
+ )
+ fallback_slot_key = (
+ _safe_text(fallback_proposal.get("slot_key", ""))
+ or f"{session_slug}.turn_{max(1, int(turn_index or 1))}.{fallback_record_category}.intent_fallback"
+ )
+ fallback_anchors = _dedupe_texts(
+ [
+ speaker,
+ timestamp,
+ *list(fallback_proposal.get("anchors", []) or []),
+ *fallback_event_signature.split()[:4],
+ ],
+ max_items=8,
+ )
+ fallback_policy = {
+ "decision": "current_turn_exact_fallback",
+ "reject_reason": "",
+ "force_question": False,
+ "speech_act_memory": False,
+ "transient_directive": False,
+ "turn_intent_reason": _safe_text(turn_intent.get("reason", "")),
+ }
+ fallback_metadata = {
+ **common_metadata,
+ "content_variant": "llm_semantic_write",
+ "raw_text": compact_text,
+ "source_turn_text": compact_text,
+ "semantic_slot": fallback_semantic_slot,
+ "target_status": _safe_text(fallback_proposal.get("target_status", "")),
+ "source_span": compact_text,
+ "event_phrase": compact_text,
+ "event_text": compact_text,
+ "event_signature": fallback_event_signature,
+ "canonical_slot_key": fallback_slot_key,
+ "allow_parallel_state": bool(fallback_proposal.get("allow_parallel_state", True)),
+ "memory_gate_decision": "turn_intent_current_turn_fallback",
+ "grounding_score": 1.0,
+ "value_grounding_score": 1.0,
+ "original_value_grounding_score": 0.0,
+ "canonicalized_from_value": _safe_text(fallback_proposal.get("value", "")),
+ "canonicalization_reason": "turn_intent_current_turn_exact_fallback",
+ "source_span_repair": "current_turn_exact_from_turn_intent",
+ "original_source_span": _safe_text(fallback_proposal.get("source_span", "")),
+ "original_value": _safe_text(fallback_proposal.get("value", "")),
+ "llm_write_proposal_index": 0,
+ "proposal_intent_gate": fallback_policy,
+ **profile_candidate_metadata(
+ category=fallback_record_category,
+ semantic_slot=fallback_semantic_slot,
+ relation=fallback_relation,
+ value=compact_text,
+ source_span=compact_text,
+ slot_key=fallback_slot_key,
+ anchors=fallback_anchors,
+ subject="",
+ subject_signature="",
+ proposal={**fallback_proposal, "proposal_intent_gate": fallback_policy},
+ ),
+ }
+ fallback_record = {
+ "category": fallback_record_category,
+ "slot_key": fallback_slot_key,
+ "value": compact_text,
+ "anchors": fallback_anchors,
+ "relation": fallback_relation,
+ "source_kind": fallback_source_kind,
+ "state": "active",
+ "salience": float(fallback_proposal.get("salience", 0.88) or 0.88),
+ "confidence": min(1.0, max(0.0, float(fallback_proposal.get("confidence", 0.82) or 0.82))),
+ "metadata": fallback_metadata,
+ }
+ fallback_facet_records = _build_event_facet_records(
+ parent_record=fallback_record,
+ proposal=fallback_proposal,
+ common_metadata=common_metadata,
+ source_context=source_context,
+ current_turn_text=compact_text,
+ speaker=speaker,
+ timestamp=timestamp,
+ turn_index=turn_index,
+ proposal_index=0,
+ )
+ if fallback_facet_records:
+ fallback_metadata["facet_types"] = _dedupe_texts(
+ [dict(item.get("metadata", {}) or {}).get("facet_type", "") for item in fallback_facet_records],
+ max_items=8,
+ )
+ fallback_metadata["facet_record_count"] = len(fallback_facet_records)
+ records.append(fallback_record)
+ records.extend(fallback_facet_records)
+ current_subject = sidecar_subject or _extract_memory_subject(compact_text)
+ current_subject_signature = sidecar_subject_signature or (_slug(current_subject) if current_subject else "")
+ current_slot_key = sidecar_canonical_slot_key or (
+ f"{session_slug}.subject.{current_subject_signature}" if current_subject_signature else ""
+ )
+ current_source_span, current_value = _extract_current_value_assertion(
+ current_turn_text=compact_text,
+ subject=current_subject,
+ )
+ current_slot_records = [
+ record
+ for record in records[1:]
+ if _safe_text(record.get("slot_key", "")).lower() == _safe_text(current_slot_key).lower()
+ and _safe_text(record.get("category", "")).lower() != "question"
+ ]
+ current_value_final_head = bool(current_slot_records) and _record_covers_current_value(
+ record=current_slot_records[-1],
+ slot_key=current_slot_key,
+ current_value=current_value,
+ )
+ if current_slot_key and current_value and not question_only_turn and not current_value_final_head:
+ current_event_signature = compute_public_event_signature(
+ current_source_span or current_value,
+ speaker=_safe_text(speaker),
+ semantic_slot=current_subject_signature or "current_subject_value",
+ )
+ current_dedupe_key = (
+ "preference",
+ current_slot_key.lower(),
+ re.sub(r"\s+", " ", current_value.lower()).strip(),
+ )
+ if current_dedupe_key not in seen:
+ seen.add(current_dedupe_key)
+ records.append(
+ {
+ "category": "preference",
+ "slot_key": current_slot_key,
+ "value": current_value,
+ "anchors": _dedupe_texts(
+ [
+ speaker,
+ timestamp,
+ current_subject,
+ *current_event_signature.split()[:4],
+ ],
+ max_items=8,
+ ),
+ "relation": "current_subject_value",
+ "source_kind": "public_dialog_fact",
+ "state": "active",
+ "salience": 0.98,
+ "confidence": 0.98,
+ "metadata": {
+ **common_metadata,
+ "content_variant": "deterministic_current_subject_value",
+ "raw_text": current_source_span or current_value,
+ "source_turn_text": current_source_span or current_value,
+ "semantic_slot": current_subject_signature or "current_subject_value",
+ "target_status": "current",
+ "source_span": current_source_span or current_value,
+ "event_phrase": current_value,
+ "event_text": current_value,
+ "event_signature": current_event_signature,
+ "extracted_subject": _extract_memory_subject(compact_text),
+ "subject": current_subject,
+ "subject_signature": current_subject_signature,
+ "canonical_slot_key": current_slot_key,
+ "subject_from_sidecar": bool(sidecar_subject and current_subject == sidecar_subject),
+ "sidecar_subject_hint": sidecar_subject,
+ "sidecar_subject_signature_hint": sidecar_subject_signature,
+ "allow_parallel_state": False,
+ "memory_gate_decision": "current_slot_head",
+ "grounding_score": 1.0,
+ "value_grounding_score": 1.0,
+ "original_value_grounding_score": 1.0,
+ "canonicalized_from_value": "",
+ "canonicalization_reason": "",
+ "llm_write_proposal_index": -1,
+ **profile_candidate_metadata(
+ category="preference",
+ semantic_slot=current_subject_signature or "current_subject_value",
+ relation="current_subject_value",
+ value=current_value,
+ source_span=current_source_span or current_value,
+ slot_key=current_slot_key,
+ anchors=[
+ speaker,
+ timestamp,
+ current_subject,
+ *current_event_signature.split()[:4],
+ ],
+ subject=current_subject,
+ subject_signature=current_subject_signature,
+ proposal={"profile_type": "usage_context", "profile_domain": current_subject},
+ ),
+ },
+ }
+ )
+
+ payload = {
+ "metadata": {
+ "memory_write": True,
+ "source": "public_benchmark",
+ "write_path": "llm_semantic_writer_gate",
+ "semantic_writer_accepted_count": max(0, len(records) - 1),
+ "semantic_writer_rejected_count": len(rejected),
+ "semantic_writer_suspect_count": len(suspect_records),
+ "turn_extractor_role": "sidecar_hints",
+ },
+ "replacement_memory_records": records,
+ "suspect_memory_records": suspect_records,
+ "semantic_writer_rejections": rejected,
+ }
+ accepted_count = max(0, len(records) - 1)
+ return MemoryWriteGateResult(
+ payload=payload,
+ accepted_count=accepted_count,
+ rejected=rejected,
+ suspected_count=len(suspect_records),
+ )
+
+
+class OpenAICompatSemanticMemoryWriter:
+ def __init__(
+ self,
+ profile: LLMProfile,
+ *,
+ gate: DeterministicMemoryWriteGate | None = None,
+ max_proposals: int = 4,
+ ) -> None:
+ self.profile = profile
+ self.gate = gate or DeterministicMemoryWriteGate()
+ self.max_proposals = max(1, int(max_proposals or 4))
+
+ def available(self) -> bool:
+ return bool(_safe_text(self.profile.base_url)) and bool(_safe_text(self.profile.model))
+
+ def _writer_family(self) -> str:
+ marker = f"{_safe_text(self.profile.model)} {_safe_text(self.profile.base_url)}".lower()
+ if "deepseek" in marker:
+ return "deepseek"
+ if "gemma" in marker:
+ return "gemma"
+ return "generic"
+
+ def _chat(self, messages: Sequence[Mapping[str, str]]) -> tuple[str, Dict[str, int]]:
+ base_url = _safe_text(self.profile.base_url).rstrip("/")
+ if not base_url:
+ raise RuntimeError("semantic memory writer base_url is empty")
+ url = f"{base_url}/chat/completions"
+ payload = {
+ "model": self.profile.model,
+ "messages": list(messages),
+ "temperature": float(self.profile.temperature),
+ "max_tokens": int(self.profile.max_tokens),
+ "response_format": {"type": "json_object"},
+ }
+ raw_payload = json.dumps(payload).encode("utf-8")
+ headers = {"Content-Type": "application/json"}
+ if _safe_text(self.profile.api_key):
+ headers["Authorization"] = f"Bearer {_safe_text(self.profile.api_key)}"
+ request = urllib.request.Request(url, data=raw_payload, headers=headers, method="POST")
+ try:
+ with urllib.request.urlopen(request, timeout=float(self.profile.timeout_seconds)) as response:
+ response_payload = json.loads(response.read().decode("utf-8"))
+ except urllib.error.HTTPError as exc:
+ if b"response_format" not in exc.read()[:2048]:
+ raise
+ payload.pop("response_format", None)
+ request = urllib.request.Request(
+ url,
+ data=json.dumps(payload).encode("utf-8"),
+ headers=headers,
+ method="POST",
+ )
+ with urllib.request.urlopen(request, timeout=float(self.profile.timeout_seconds)) as response:
+ response_payload = json.loads(response.read().decode("utf-8"))
+ choices = list(response_payload.get("choices", []) or [])
+ message = dict(dict(choices[0] if choices else {}).get("message", {}) or {})
+ usage = dict(response_payload.get("usage", {}) or {})
+ return _safe_text(message.get("content", "")), {
+ "prompt_tokens": int(usage.get("prompt_tokens", 0) or 0),
+ "completion_tokens": int(usage.get("completion_tokens", 0) or 0),
+ "total_tokens": int(usage.get("total_tokens", 0) or 0),
+ }
+
+ def _classify_public_turn_intent(
+ self,
+ *,
+ current_turn: str,
+ speaker: str = "",
+ session_timestamp: str = "",
+ ) -> tuple[Dict[str, Any], Dict[str, Any]]:
+ system_prompt = (
+ "Return JSON only. Read only current_turn. Classify the whole current_turn for memory writing. "
+ "Do not extract memories. Do not include prose. "
+ "Use intent=memory_assertion when the turn states a durable user fact, preference, goal, constraint, status, event, or plan. "
+ "Use intent=mixed_question_with_memory when the turn has question-like wording and also states a durable new user memory. "
+ "Use intent=question_only when the turn only asks, checks, or asks you to recall existing memory. "
+ "Use intent=no_new_memory for thanks, reactions, or turns with no new durable memory. "
+ "Use intent=transient_directive for one-shot instructions about how to answer this turn. "
+ "Return exactly {\"turn_intent\":{\"intent\":\"memory_assertion|mixed_question_with_memory|question_only|no_new_memory|transient_directive|plain_statement\","
+ "\"write_allowed\":true,\"question_like\":false,\"memory_assertion\":true,\"question_only\":false,\"reason\":\"...\"}}."
+ )
+ user_payload = {
+ "speaker": _safe_text(speaker),
+ "timestamp": _safe_text(session_timestamp),
+ "current_turn": _safe_text(current_turn),
+ }
+ content, usage = self._chat(
+ [
+ {"role": "system", "content": system_prompt},
+ {"role": "user", "content": json.dumps(user_payload, ensure_ascii=False)},
+ ]
+ )
+ parsed = _extract_json_object(content)
+ writer_turn_intent = _extract_turn_intent_object(parsed) or _extract_turn_intent_from_text(content)
+ return writer_turn_intent, {
+ "raw_output_excerpt": content[:1000],
+ "usage": usage,
+ "json_repair_status": "parsed" if parsed else ("intent_extracted_from_jsonish_text" if writer_turn_intent else "empty_from_non_json_writer_content"),
+ "system_prompt_chars": len(system_prompt),
+ "user_payload_chars": len(json.dumps(user_payload, ensure_ascii=False)),
+ }
+
+ def propose_public_turn(
+ self,
+ *,
+ current_turn: str,
+ previous_turn: str = "",
+ next_turn: str = "",
+ speaker: str = "",
+ session_timestamp: str = "",
+ sidecar_hints: Mapping[str, Any] | None = None,
+ auxiliary_evidence_texts: Sequence[Any] | None = None,
+ input_mode: str = "full",
+ ) -> tuple[List[Dict[str, Any]], Dict[str, Any]]:
+ if not self.available():
+ raise SemanticMemoryWriterError(
+ "semantic memory writer is required but not configured; "
+ f"model={_safe_text(self.profile.model)!r}, base_url={_safe_text(self.profile.base_url)!r}"
+ )
+ normalized_input_mode = _safe_text(input_mode).lower() or "full"
+ if normalized_input_mode not in {"full", "delta"}:
+ raise SemanticMemoryWriterError(f"unsupported semantic writer input_mode={input_mode!r}")
+ writer_family = self._writer_family()
+ answerable_memory_prompt = ""
+ if writer_family == "deepseek":
+ answerable_memory_prompt = (
+ "DeepSeek writer profile: maximize recall for answerable user memories, not just broad preferences. "
+ "In long benchmark-like conversation chunks, one-off first-person facts are durable when they can answer a later question. "
+ "Always prefer the most specific grounded fact over a generic summary. "
+ "Treat these as high-priority writable memories when stated by the user: store or venue names, service names, class or studio names, playlist names, book/movie/play/song/artwork titles, event names, dates/months, durations, prices, colors, locations, people names, product names, and organizations. "
+ "User phrases such as I bought, I redeemed, I attended, I volunteered, I created, I take classes at, I repainted, I changed my name, I commute, I recently, by the way, or I just remembered usually contain answerable memory. "
+ "Do not skip such facts because nearby text asks for recommendations, tips, explanations, or advice. "
+ "For source_span, copy the shortest exact span that contains the answer string and enough context to prove it belongs to the user. "
+ "For value, write a direct third-person memory about the user that preserves exact names and casing. "
+ "Prioritize answerable facts over generic goals, attitudes, or assistant-request memories. "
+ "If a chunk contains multiple answerable facts, return the strongest one or two within the proposal budget. "
+ "Few-shot positives: "
+ "'I redeemed a $5 coupon on coffee creamer at Target' -> user redeemed a $5 coupon on coffee creamer at Target. "
+ "'I attended The Glass Menagerie at the local community theater' -> user attended The Glass Menagerie at the local community theater. "
+ "'I take yoga classes at Serenity Yoga' -> user takes yoga classes at Serenity Yoga. "
+ "'I created a Spotify playlist called Summer Vibes' -> user created a Spotify playlist called Summer Vibes. "
+ "'I volunteered at the Love is in the Air fundraising dinner back on Valentine's Day' -> user volunteered at the Love is in the Air fundraising dinner on Valentine's Day. "
+ )
+ intent_schema_prompt = (
+ "Top-level turn intent is required. "
+ "Use intent=memory_assertion when the turn states a durable user fact, preference, goal, constraint, status, event, or plan. "
+ "Use intent=mixed_question_with_memory when the turn contains a question-like phrase and also states a durable new user memory. "
+ "Use intent=question_only when the turn only asks, checks, or asks you to recall existing memory. "
+ "Use intent=no_new_memory for thanks, reactions, or turns with no new durable memory. "
+ "Use intent=transient_directive for one-shot instructions about how to answer this turn. "
+ "Use intent=plain_statement only for durable statements that are not clearly first-person profile/fact writes. "
+ "Return the full JSON shape {\"turn_intent\":{\"intent\":\"memory_assertion|mixed_question_with_memory|question_only|no_new_memory|transient_directive|plain_statement\","
+ "\"write_allowed\":true,\"question_like\":false,\"memory_assertion\":true,\"question_only\":false,\"reason\":\"...\"},\"write_proposals\":[...]}. "
+ )
+ schema_prompt = intent_schema_prompt + (
+ "Write proposal schema: {\"category\":\"event|profile|time|status|preference|goal|constraint|fact\","
+ "\"value\":\"...\",\"source_span\":\"...\",\"semantic_slot\":\"open_domain_slot|event_time\","
+ "\"profile_type\":\"setup|preference|constraint|goal|avoid|usage_context\",\"profile_domain\":\"dynamic open-domain label\","
+ "\"target_status\":\"past|current|planned|\",\"time_expression_span\":\"...\",\"time_granularity\":\"day|month|year|relative_day_reference|none|\","
+ "\"anchors\":[\"...\"],\"salience\":0.0,\"confidence\":0.0,"
+ "\"facets\":[{\"type\":\"temporal|numeric|state|entity|evidence_role\",\"role\":\"start|finish|amount|count|subject|constraint|...\","
+ "\"value\":\"grounded attribute value\",\"unit\":\"optional unit\",\"source_span\":\"exact copied evidence span\"}]}. "
+ "Facets are optional attribute subnodes of the proposal. Emit a facet only when the current_turn explicitly contains that attribute; never emit empty, zero, or placeholder facets. "
+ )
+ if normalized_input_mode == "delta":
+ system_prompt = (
+ "Return JSON only. Read only current_turn. "
+ "First classify the whole current_turn into top-level turn_intent, then emit write_proposals. "
+ "Write proposals only for new durable user memory explicitly stated now: fact, event, profile, time, status, preference, goal, or constraint. "
+ "Keep output tiny: at most two proposals; value <= 120 chars; source_span <= 160 chars and copied exactly from current_turn. "
+ "If current_turn is a long mixed conversation chunk, still extract durable first-person user facts inside it; do not skip them just because nearby turns ask for recommendations, tips, or advice. "
+ "Write value as a complete third-person memory about the user, preferably starting with 'user ...'. "
+ "For user preferences, goals, constraints, setup, habits, or usage context, include profile_type and dynamic profile_domain. "
+ "When a writable fact contains explicit time, duration, quantity, amount, count, state, or acted-on entity, include grounded facets under that write proposal. "
+ "Do not put absent attributes into facets; omitted means not present. "
+ "Profile writer rule: when the user states current setup, a brand/device they use, durable taste, recurring habit, goal, or constraint, mark it as profile-compatible with profile_type and profile_domain even if the category is fact or event. "
+ "Preserve exact project names, codenames, acronyms, model/API names, and mixed-case identifiers when they appear in source_span. "
+ f"{answerable_memory_prompt}"
+ "Skip questions, thanks, reactions, assistant requests, one-shot answer instructions, secrets, and no-new-memory turns. "
+ "Never write instructions about how the assistant should answer this turn. "
+ f"{intent_schema_prompt}"
+ "{\"turn_intent\":{\"intent\":\"...\",\"write_allowed\":true,\"question_like\":false,\"memory_assertion\":true,\"question_only\":false,\"reason\":\"...\"},"
+ "\"write_proposals\":[{\"category\":\"fact|event|profile|time|status|preference|goal|constraint\","
+ "\"value\":\"...\",\"source_span\":\"...\",\"semantic_slot\":\"...\","
+ "\"facets\":[{\"type\":\"temporal|numeric|state|entity|evidence_role\",\"role\":\"...\",\"value\":\"...\",\"source_span\":\"...\"}]}]}"
+ )
+ user_payload = {
+ "speaker": _safe_text(speaker),
+ "timestamp": _safe_text(session_timestamp),
+ "current_turn": _safe_text(current_turn),
+ "max_write_proposals": self.max_proposals,
+ }
+ system_prompt_profile = "delta_current_turn_only"
+ else:
+ system_prompt = (
+ "You are the TMCRA semantic memory writer. Return strict JSON only. "
+ "Do not include thinking, analysis, markdown, or prose outside JSON. "
+ "First classify the whole current_turn into top-level turn_intent, then decide write_proposals. "
+ "Decide what should be remembered from the current conversation turn. "
+ "The extractor output is only sidecar hints; do not copy unsupported hints. "
+ "Every write must include a source_span copied from the turn or auxiliary evidence. "
+ "The current turn is the only writable dialogue source; previous and next turns are context only. "
+ "Do not write secrets, passwords, API keys, or unsupported guesses. "
+ "Only write durable memory: asserted user facts, preferences, goals, constraints, plans, status changes, events, or time facts. "
+ "When a writable fact contains explicit time, duration, quantity, amount, count, state, or acted-on entity, include grounded facets under that write proposal. "
+ "Do not put absent attributes into facets; omitted means not present. "
+ "For user preferences, goals, constraints, setup, habits, or usage context, fill profile_type from the fixed set "
+ "setup|preference|constraint|goal|avoid|usage_context and fill profile_domain as a dynamic open-domain label from the user's own wording. "
+ "Profile writer rule: when the user states current setup, a brand/device they use, durable taste, recurring habit, goal, or constraint, mark it as profile-compatible with profile_type and profile_domain even if the category is fact or event. "
+ "Preserve exact project names, codenames, acronyms, model/API names, and mixed-case identifiers when they appear in source_span. "
+ "Skip one-shot answer instructions such as 'answer only from context', 'do not mention retrieval', or 'summarize this now'. "
+ "Never write instructions about how the assistant should answer this turn. "
+ "If the current turn only asks a question, asks the assistant to recall existing memory, thanks/reacts, or contains no new durable memory, return an empty write_proposals list. "
+ "Do not convert questions such as 'what is my project?' or 'do you remember my plan?' into profile/fact/status writes. "
+ "Never write speech-act memories that merely describe the user asking, checking, reminding, thanking, agreeing, or requesting assistant action. "
+ "Values like 'asked what the project is', 'asked for a reminder', 'wanted the assistant to recall', or 'requested help' are forbidden unless the user also states a new durable fact. "
+ "A question mark in the current turn is a strong warning: only write if the same turn also asserts a concrete new value to remember. "
+ "Negative examples: 'Can you remind me what my main project is?' -> {\"turn_intent\":{\"intent\":\"question_only\",\"write_allowed\":false,\"question_like\":true,\"memory_assertion\":false,\"question_only\":true,\"reason\":\"asks recall only\"},\"write_proposals\":[]}; 'What is my meeting plan?' -> same question_only shape with empty write_proposals. "
+ "Positive examples: 'Remember that my main project is TMCRA.' -> write a grounded profile/fact; 'My meeting plan is Friday.' -> write a grounded plan/status. "
+ "For memory-bearing turns, produce at least one grounded write proposal. "
+ "Activity evaluations without explicit 'I like' can still be speaker preferences or coping strategies when the preference is grounded in the current turn; previous turns may disambiguate only. "
+ f"{answerable_memory_prompt}"
+ f"{schema_prompt}"
+ )
+ user_payload = {
+ "speaker": _safe_text(speaker),
+ "timestamp": _safe_text(session_timestamp),
+ "previous_turn": _safe_text(previous_turn),
+ "current_turn": _safe_text(current_turn),
+ "next_turn": _safe_text(next_turn),
+ "auxiliary_evidence_texts": list(auxiliary_evidence_texts or []),
+ "sidecar_hints": dict(sidecar_hints or {}),
+ "max_write_proposals": self.max_proposals,
+ }
+ system_prompt_profile = "full_context_sidecar"
+ content, usage = self._chat(
+ [
+ {"role": "system", "content": system_prompt},
+ {"role": "user", "content": json.dumps(user_payload, ensure_ascii=False)},
+ ]
+ )
+ parsed = _extract_json_object(content)
+ json_repair_status = _safe_text(parsed.pop("_tmcra_json_repair_status", "")) if parsed else ""
+ if not parsed and content:
+ parsed = {"write_proposals": []}
+ json_repair_status = "empty_from_non_json_writer_content"
+ writer_turn_intent = _extract_turn_intent_object(parsed) or _extract_turn_intent_from_text(content)
+ writer_turn_intent_source = "inline_writer_output" if writer_turn_intent else ""
+ writer_turn_intent_repair_metadata: Dict[str, Any] = {}
+ if not writer_turn_intent:
+ writer_turn_intent, writer_turn_intent_repair_metadata = self._classify_public_turn_intent(
+ current_turn=current_turn,
+ speaker=speaker,
+ session_timestamp=session_timestamp,
+ )
+ writer_turn_intent_source = "repair_intent_call" if writer_turn_intent else "missing"
+ proposals = parsed.get("write_proposals", parsed.get("write_proposal", parsed.get("memory_writes", [])))
+ if isinstance(proposals, Mapping):
+ proposals = [dict(proposals)]
+ if not isinstance(proposals, list):
+ proposals = []
+ json_repair_status = json_repair_status or "empty_from_invalid_write_proposals_field"
+ return [dict(item) for item in proposals[: self.max_proposals] if isinstance(item, Mapping)], {
+ "raw_output_excerpt": content[:1000],
+ "usage": usage,
+ "proposal_count": len(proposals),
+ "json_repair_status": json_repair_status or "parsed",
+ "json_repair_applied": bool(json_repair_status),
+ "writer_turn_intent": dict(writer_turn_intent),
+ "writer_turn_intent_source": writer_turn_intent_source,
+ "writer_turn_intent_repair_metadata": dict(writer_turn_intent_repair_metadata),
+ "input_mode": normalized_input_mode,
+ "writer_family": writer_family,
+ "system_prompt_profile": system_prompt_profile,
+ "user_payload_keys": sorted(user_payload.keys()),
+ "system_prompt_chars": len(system_prompt),
+ "user_payload_chars": len(json.dumps(user_payload, ensure_ascii=False)),
+ }
+
+ def write_public_turn(
+ self,
+ *,
+ text: str,
+ raw_text: str = "",
+ speaker: str,
+ session_key: str,
+ turn_index: int,
+ timestamp: str = "",
+ dia_id: str = "",
+ previous_turn: str = "",
+ next_turn: str = "",
+ sidecar_hints: Mapping[str, Any] | None = None,
+ auxiliary_evidence_texts: Sequence[Any] | None = None,
+ input_mode: str = "full",
+ ) -> Dict[str, Any]:
+ proposals, writer_metadata = self.propose_public_turn(
+ current_turn=raw_text or text,
+ previous_turn=previous_turn,
+ next_turn=next_turn,
+ speaker=speaker,
+ session_timestamp=timestamp,
+ sidecar_hints=sidecar_hints,
+ auxiliary_evidence_texts=auxiliary_evidence_texts,
+ input_mode=input_mode,
+ )
+ result = self.gate.build_payload(
+ proposals=proposals,
+ text=text,
+ raw_text=raw_text,
+ speaker=speaker,
+ session_key=session_key,
+ turn_index=turn_index,
+ timestamp=timestamp,
+ dia_id=dia_id,
+ sidecar_hints=sidecar_hints,
+ auxiliary_evidence_texts=auxiliary_evidence_texts,
+ writer_metadata=writer_metadata,
+ )
+ if result.accepted_count <= 0 and result.suspected_count <= 0:
+ raise SemanticMemoryWriterError(
+ "semantic memory writer produced no gated write proposals; "
+ f"proposal_count={len(proposals)}, rejected={result.rejected[:3]}"
+ )
+ return result.payload
+
+
+__all__ = [
+ "ALLOWED_WRITE_CATEGORIES",
+ "DeterministicMemoryWriteGate",
+ "MemoryWriteGateResult",
+ "OpenAICompatSemanticMemoryWriter",
+ "SemanticMemoryWriterError",
+ "build_modelized_facet_unit_records",
+]
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/temporal_modeling_types.py b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/temporal_modeling_types.py
new file mode 100644
index 0000000000000000000000000000000000000000..631574fc7b228c89c66a4b162c5d582b61c874b8
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/temporal_modeling_types.py
@@ -0,0 +1,183 @@
+from __future__ import annotations
+
+from dataclasses import asdict, dataclass, field
+from typing import Any, Mapping
+
+
+TEMPORAL_INTENTS = {
+ "non_temporal",
+ "explicit_time",
+ "relative_time",
+ "current_state",
+ "previous_state",
+ "timeline",
+ "compare_before_after",
+ "state_evolution",
+}
+
+ANCHOR_TYPES = {"none", "absolute", "relative", "session_time", "event_time", "turn_order"}
+
+GRANULARITIES = {
+ "none",
+ "turn",
+ "day",
+ "week",
+ "month",
+ "year",
+ "range",
+ "relative_day_reference",
+}
+
+STATE_OPERATIONS = {
+ "none",
+ "create",
+ "update_current",
+ "supersede",
+ "query_current",
+ "query_previous",
+ "query_timeline",
+ "query_compare",
+}
+
+
+def clean_text(value: Any) -> str:
+ return " ".join(str(value or "").strip().split())
+
+
+def normalize_key(value: Any, *, fallback: str = "general") -> str:
+ text = clean_text(value).lower()
+ if not text:
+ return fallback
+ allowed = []
+ previous_sep = False
+ for char in text:
+ if char.isalnum() or "\u4e00" <= char <= "\u9fff":
+ allowed.append(char)
+ previous_sep = False
+ elif not previous_sep:
+ allowed.append("_")
+ previous_sep = True
+ key = "".join(allowed).strip("_")
+ return key or fallback
+
+
+def coerce_choice(value: Any, allowed: set[str], default: str) -> str:
+ text = clean_text(value).lower().replace("-", "_").replace(" ", "_")
+ return text if text in allowed else default
+
+
+@dataclass(slots=True)
+class TemporalFrame:
+ temporal_intent: str = "non_temporal"
+ time_expression: str = ""
+ anchor_type: str = "none"
+ anchor_base: str = ""
+ resolved_start: str = ""
+ resolved_end: str = ""
+ granularity: str = "none"
+ utterance_time: str = ""
+ event_time: str = ""
+ state_valid_from: str = ""
+ state_valid_to: str = ""
+ state_operation: str = "none"
+ subject: str = ""
+ subject_key: str = ""
+ old_state: str = ""
+ new_state: str = ""
+ should_create_timeline_edge: bool = False
+ confidence: float = 0.0
+ evidence_span: str = ""
+ metadata: dict[str, Any] = field(default_factory=dict)
+
+ def __post_init__(self) -> None:
+ self.temporal_intent = coerce_choice(self.temporal_intent, TEMPORAL_INTENTS, "non_temporal")
+ self.anchor_type = coerce_choice(self.anchor_type, ANCHOR_TYPES, "none")
+ self.granularity = coerce_choice(self.granularity, GRANULARITIES, "none")
+ self.state_operation = coerce_choice(self.state_operation, STATE_OPERATIONS, "none")
+ if not self.subject_key and self.subject:
+ self.subject_key = normalize_key(self.subject)
+ self.confidence = max(0.0, min(1.0, float(self.confidence or 0.0)))
+
+ def to_dict(self) -> dict[str, Any]:
+ return asdict(self)
+
+ @classmethod
+ def from_mapping(cls, payload: Mapping[str, Any] | None) -> "TemporalFrame":
+ data = dict(payload or {})
+ allowed = {field_name for field_name in cls.__dataclass_fields__}
+ return cls(**{key: value for key, value in data.items() if key in allowed})
+
+
+@dataclass(slots=True)
+class TimelineState:
+ state_id: str
+ subject_key: str
+ subject_label: str = ""
+ state_value: str = ""
+ state_type: str = "profile"
+ valid_from: str = ""
+ valid_to: str = ""
+ event_time: str = ""
+ utterance_time: str = ""
+ is_current: bool = True
+ supersedes_state_id: str = ""
+ source_event_id: str = ""
+ source_turn_id: str = ""
+ confidence: float = 0.0
+ metadata: dict[str, Any] = field(default_factory=dict)
+
+ def to_dict(self) -> dict[str, Any]:
+ return asdict(self)
+
+
+@dataclass(slots=True)
+class TimelineEdge:
+ edge_id: str
+ subject_key: str
+ from_state_id: str = ""
+ to_state_id: str = ""
+ edge_type: str = "updates"
+ transition_time: str = ""
+ transition_reason: str = ""
+ source_event_id: str = ""
+ confidence: float = 0.0
+
+ def to_dict(self) -> dict[str, Any]:
+ return asdict(self)
+
+
+@dataclass(slots=True)
+class TemporalQueryPlan:
+ query_temporal_intent: str = "non_temporal"
+ target_subject: str = ""
+ target_subject_key: str = ""
+ time_filter: dict[str, Any] = field(default_factory=dict)
+ timeline_operation: str = "none"
+ prefer_current_state: bool = False
+ prefer_previous_state: bool = False
+ requires_ordered_chain: bool = False
+ requires_comparison: bool = False
+ confidence: float = 0.0
+ metadata: dict[str, Any] = field(default_factory=dict)
+
+ def __post_init__(self) -> None:
+ self.query_temporal_intent = coerce_choice(self.query_temporal_intent, TEMPORAL_INTENTS, "non_temporal")
+ if not self.target_subject_key and self.target_subject:
+ self.target_subject_key = normalize_key(self.target_subject)
+ self.confidence = max(0.0, min(1.0, float(self.confidence or 0.0)))
+
+ def to_dict(self) -> dict[str, Any]:
+ return asdict(self)
+
+
+@dataclass(slots=True)
+class TimelineEvidencePack:
+ mode: str
+ subject: str = ""
+ subject_key: str = ""
+ timeline: list[dict[str, Any]] = field(default_factory=list)
+ selected_evidence: dict[str, Any] = field(default_factory=dict)
+ evidence_confidence: float = 0.0
+
+ def to_dict(self) -> dict[str, Any]:
+ return asdict(self)
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/temporal_normalizer.py b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/temporal_normalizer.py
new file mode 100644
index 0000000000000000000000000000000000000000..1f0c3ea795fd038315c2bc5b6dfefde7a899954c
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/temporal_normalizer.py
@@ -0,0 +1,143 @@
+from __future__ import annotations
+
+import calendar
+import re
+from datetime import datetime, timedelta
+from typing import Any
+
+from experiments.replacement.temporal_modeling_types import TemporalFrame, clean_text
+
+
+_MONTH_NAMES = {
+ "january": 1,
+ "february": 2,
+ "march": 3,
+ "april": 4,
+ "may": 5,
+ "june": 6,
+ "july": 7,
+ "august": 8,
+ "september": 9,
+ "october": 10,
+ "november": 11,
+ "december": 12,
+}
+
+_CJK_MONTH_RE = re.compile(r"(?P1[0-2]|0?[1-9])\s*月")
+
+
+def _parse_datetime(value: Any) -> datetime | None:
+ text = clean_text(value)
+ if not text:
+ return None
+ for candidate in (text, text.replace("Z", "+00:00")):
+ try:
+ return datetime.fromisoformat(candidate)
+ except ValueError:
+ continue
+ for fmt in ("%Y-%m-%d", "%Y/%m/%d", "%Y-%m", "%Y/%m"):
+ try:
+ return datetime.strptime(text, fmt)
+ except ValueError:
+ continue
+ return None
+
+
+def _date_text(value: datetime) -> str:
+ return value.strftime("%Y-%m-%d")
+
+
+def _month_range(year: int, month: int) -> tuple[str, str]:
+ last_day = calendar.monthrange(year, month)[1]
+ return f"{year:04d}-{month:02d}-01", f"{year:04d}-{month:02d}-{last_day:02d}"
+
+
+class TemporalNormalizer:
+ """Resolve a TemporalFrame against session time without mutating memory."""
+
+ version = "temporal_normalizer_v1"
+
+ def normalize(self, frame: TemporalFrame, *, session_timestamp: str = "") -> TemporalFrame:
+ resolved = TemporalFrame.from_mapping(frame.to_dict())
+ session_time = _parse_datetime(session_timestamp or resolved.utterance_time) or datetime.now()
+ if not resolved.utterance_time:
+ resolved.utterance_time = _date_text(session_time)
+ if resolved.resolved_start or resolved.resolved_end:
+ self._fill_event_state_time(resolved)
+ return resolved
+ start, end, granularity = self._resolve_expression(
+ resolved.time_expression,
+ session_time=session_time,
+ fallback_granularity=resolved.granularity,
+ )
+ if start:
+ resolved.resolved_start = start
+ resolved.resolved_end = end or start
+ resolved.event_time = resolved.event_time or start
+ resolved.granularity = granularity or resolved.granularity
+ elif resolved.state_operation in {"create", "update_current", "supersede"}:
+ resolved.resolved_start = _date_text(session_time)
+ resolved.resolved_end = _date_text(session_time)
+ resolved.event_time = resolved.event_time or resolved.resolved_start
+ if resolved.granularity == "none":
+ resolved.granularity = "day"
+ self._fill_event_state_time(resolved)
+ resolved.metadata = {
+ **dict(resolved.metadata or {}),
+ "temporal_normalizer": self.version,
+ "session_timestamp": clean_text(session_timestamp),
+ }
+ return resolved
+
+ def _fill_event_state_time(self, frame: TemporalFrame) -> None:
+ if frame.state_operation in {"create", "update_current", "supersede"}:
+ frame.state_valid_from = frame.state_valid_from or frame.event_time or frame.resolved_start
+ if not frame.event_time and frame.resolved_start:
+ frame.event_time = frame.resolved_start
+
+ def _resolve_expression(
+ self,
+ expression: str,
+ *,
+ session_time: datetime,
+ fallback_granularity: str = "none",
+ ) -> tuple[str, str, str]:
+ text = clean_text(expression).lower()
+ if not text:
+ return "", "", fallback_granularity
+ if text in {"today", "今天"} or "今天" in text:
+ day = _date_text(session_time)
+ return day, day, "day"
+ if text in {"yesterday", "昨天"} or "昨天" in text:
+ day = _date_text(session_time - timedelta(days=1))
+ return day, day, "day"
+ if "last week" in text or "上周" in text:
+ start = session_time - timedelta(days=session_time.weekday() + 7)
+ end = start + timedelta(days=6)
+ return _date_text(start), _date_text(end), "week"
+ if "this week" in text or "本周" in text:
+ start = session_time - timedelta(days=session_time.weekday())
+ end = start + timedelta(days=6)
+ return _date_text(start), _date_text(end), "week"
+ if "last month" in text or "上个月" in text:
+ month = session_time.month - 1
+ year = session_time.year
+ if month <= 0:
+ month = 12
+ year -= 1
+ start, end = _month_range(year, month)
+ return start, end, "month"
+ cjk_month = _CJK_MONTH_RE.search(text)
+ if cjk_month:
+ month = int(cjk_month.group("month"))
+ start, end = _month_range(session_time.year, month)
+ return start, end, "month"
+ for name, month in _MONTH_NAMES.items():
+ if name in text:
+ start, end = _month_range(session_time.year, month)
+ return start, end, "month"
+ parsed = _parse_datetime(text)
+ if parsed is not None:
+ day = _date_text(parsed)
+ return day, day, "day"
+ return "", "", fallback_granularity
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/temporal_organizer.py b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/temporal_organizer.py
new file mode 100644
index 0000000000000000000000000000000000000000..09672dab47d1d372a12967c301f85476fe1b292f
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/temporal_organizer.py
@@ -0,0 +1,174 @@
+from __future__ import annotations
+
+import re
+from typing import Any, Mapping
+
+from experiments.replacement.temporal_modeling_types import TemporalFrame, clean_text, normalize_key
+from experiments.replacement.temporal_normalizer import TemporalNormalizer
+
+
+_TIME_MARKERS = (
+ "yesterday",
+ "today",
+ "last week",
+ "this week",
+ "last month",
+ "may",
+ "june",
+ "july",
+ "昨天",
+ "今天",
+ "上周",
+ "本周",
+ "上个月",
+ "月",
+)
+
+_CURRENT_MARKERS = ("now", "currently", "current", "right now", "现在", "目前", "当前")
+_PREVIOUS_MARKERS = ("previous", "before", "used to", "last time", "earlier", "之前", "以前", "上次", "原来")
+_TIMELINE_MARKERS = ("timeline", "over time", "how did", "history", "按时间", "时间线", "怎么变", "变化")
+_COMPARE_MARKERS = ("before and after", "compare", "versus", "vs", "对比", "前后")
+
+
+class TemporalOrganizer:
+ """Create a temporal frame for a turn before writer/retrieval logic.
+
+ This class is intentionally model-ready: callers may pass a teacher/model
+ payload later and convert it with TemporalFrame.from_mapping. The fallback
+ implementation is conservative and only supplies a valid baseline frame.
+ """
+
+ version = "temporal_organizer_v1"
+
+ def __init__(self, *, normalizer: TemporalNormalizer | None = None) -> None:
+ self.normalizer = normalizer or TemporalNormalizer()
+
+ def organize_turn(
+ self,
+ *,
+ current_turn: str,
+ session_timestamp: str = "",
+ previous_turn: str = "",
+ speaker: str = "user",
+ model_frame: Mapping[str, Any] | None = None,
+ ) -> TemporalFrame:
+ if model_frame:
+ frame = TemporalFrame.from_mapping(model_frame)
+ return self.normalizer.normalize(frame, session_timestamp=session_timestamp)
+ text = clean_text(current_turn)
+ lowered = text.lower()
+ subject, old_state, new_state = self._extract_subject_state(text)
+ time_expression = self._extract_time_expression(text)
+ has_time = bool(time_expression)
+ state_change = bool(new_state and (old_state or self._contains_any(lowered, ("switched", "changed", "换成", "改成", "开始"))))
+ if state_change:
+ temporal_intent = "state_evolution"
+ state_operation = "supersede" if old_state else "update_current"
+ elif subject and self._contains_any(lowered, _CURRENT_MARKERS):
+ temporal_intent = "current_state"
+ state_operation = "update_current"
+ elif has_time:
+ temporal_intent = "relative_time" if self._is_relative_time(time_expression) else "explicit_time"
+ state_operation = "update_current" if new_state else "create"
+ else:
+ temporal_intent = "non_temporal"
+ state_operation = "none"
+ frame = TemporalFrame(
+ temporal_intent=temporal_intent,
+ time_expression=time_expression,
+ anchor_type="relative" if self._is_relative_time(time_expression) else ("absolute" if has_time else "none"),
+ anchor_base="session_time" if self._is_relative_time(time_expression) else "",
+ granularity="day" if has_time else "none",
+ utterance_time=session_timestamp,
+ state_operation=state_operation,
+ subject=subject,
+ subject_key=normalize_key(subject) if subject else "",
+ old_state=old_state,
+ new_state=new_state,
+ should_create_timeline_edge=state_operation in {"update_current", "supersede"},
+ confidence=0.62 if temporal_intent != "non_temporal" else 0.25,
+ evidence_span=text,
+ metadata={
+ "source": self.version,
+ "speaker": clean_text(speaker),
+ "previous_turn_available": bool(clean_text(previous_turn)),
+ "fallback_mode": "heuristic_bootstrap",
+ },
+ )
+ return self.normalizer.normalize(frame, session_timestamp=session_timestamp)
+
+ def _extract_time_expression(self, text: str) -> str:
+ lowered = clean_text(text).lower()
+ for marker in _TIME_MARKERS:
+ if marker in lowered:
+ if marker == "月":
+ match = re.search(r"(?:1[0-2]|0?[1-9])\s*月", text)
+ return match.group(0) if match else ""
+ return marker
+ month = re.search(
+ r"\b(january|february|march|april|may|june|july|august|september|october|november|december)\b",
+ lowered,
+ )
+ return month.group(1) if month else ""
+
+ def _extract_subject_state(self, text: str) -> tuple[str, str, str]:
+ compact = clean_text(text)
+ lowered = compact.lower()
+ subject = self._infer_subject(lowered)
+ old_state = ""
+ new_state = ""
+ switch_match = re.search(r"\bfrom\s+(?P[^,.!?]+?)\s+to\s+(?P[^,.!?]+)", compact, flags=re.IGNORECASE)
+ if switch_match:
+ old_state = clean_text(switch_match.group("old"))
+ new_state = clean_text(switch_match.group("new"))
+ cjk_switch = re.search(r"从(?P[^，。！？]+?)换成(?:了)?(?P[^，。！？]+)", compact)
+ if cjk_switch:
+ old_state = clean_text(cjk_switch.group("old"))
+ new_state = clean_text(cjk_switch.group("new"))
+ if not new_state:
+ use_match = re.search(r"\b(?:use|using|currently use|now use)\s+(?P[A-Za-z0-9][^,.!?]*)", compact, flags=re.IGNORECASE)
+ if use_match:
+ new_state = clean_text(use_match.group("new"))
+ if not new_state:
+ cjk_now = re.search(r"(?:现在|目前|当前).*?(?:用|使用|换成了?|改成了?)(?P[^，。！？]+)", compact)
+ if cjk_now:
+ new_state = clean_text(cjk_now.group("new"))
+ if not old_state:
+ used_to = re.search(r"\bused to use\s+(?P[^,.!?]+)", compact, flags=re.IGNORECASE)
+ if used_to:
+ old_state = clean_text(used_to.group("old"))
+ return subject, old_state, new_state
+
+ def _infer_subject(self, lowered: str) -> str:
+ if any(token in lowered for token in ("phone", "iphone", "android", "手机")):
+ return "phone"
+ if any(token in lowered for token in ("project", "api", "benchmark", "论文", "项目")):
+ return "project_focus"
+ if any(token in lowered for token in ("diet", "meal", "food", "ingredient", "饮食", "食物", "食材")):
+ return "food"
+ if any(token in lowered for token in ("plan", "schedule", "calendar", "计划", "日程")):
+ return "plan"
+ return ""
+
+ def _is_relative_time(self, expression: str) -> bool:
+ text = clean_text(expression).lower()
+ return text in {"yesterday", "today", "last week", "this week", "last month", "昨天", "今天", "上周", "本周", "上个月"}
+
+ def _contains_any(self, text: str, markers: tuple[str, ...]) -> bool:
+ return any(marker in text for marker in markers)
+
+
+def infer_query_temporal_intent(query: str) -> tuple[str, str, dict[str, Any]]:
+ text = clean_text(query)
+ lowered = text.lower()
+ if any(marker in lowered for marker in _TIMELINE_MARKERS):
+ return "timeline", "query_timeline", {"requires_ordered_chain": True}
+ if any(marker in lowered for marker in _COMPARE_MARKERS):
+ return "compare_before_after", "query_compare", {"requires_comparison": True}
+ if any(marker in lowered for marker in _PREVIOUS_MARKERS):
+ return "previous_state", "query_previous", {"prefer_previous_state": True}
+ if any(marker in lowered for marker in _CURRENT_MARKERS):
+ return "current_state", "query_current", {"prefer_current_state": True}
+ if any(marker in lowered for marker in _TIME_MARKERS):
+ return "explicit_time", "query_timeline", {"requires_ordered_chain": False}
+ return "non_temporal", "none", {}
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/temporal_query_planner.py b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/temporal_query_planner.py
new file mode 100644
index 0000000000000000000000000000000000000000..8c7c251500ffeea17e9b389798e0d63e4cbf6f54
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/temporal_query_planner.py
@@ -0,0 +1,41 @@
+from __future__ import annotations
+
+from experiments.replacement.temporal_modeling_types import TemporalQueryPlan, clean_text, normalize_key
+from experiments.replacement.temporal_organizer import infer_query_temporal_intent
+
+
+class TemporalQueryPlanner:
+ """Plan temporal retrieval before graph/rerank/answer stages."""
+
+ version = "temporal_query_planner_v1"
+
+ def plan(self, query: str, *, target_subject: str = "", session_timestamp: str = "") -> TemporalQueryPlan:
+ intent, operation, flags = infer_query_temporal_intent(query)
+ subject = clean_text(target_subject) or self._infer_subject(query)
+ return TemporalQueryPlan(
+ query_temporal_intent=intent,
+ target_subject=subject,
+ target_subject_key=normalize_key(subject) if subject else "",
+ timeline_operation=operation,
+ prefer_current_state=bool(flags.get("prefer_current_state", False)),
+ prefer_previous_state=bool(flags.get("prefer_previous_state", False)),
+ requires_ordered_chain=bool(flags.get("requires_ordered_chain", False)),
+ requires_comparison=bool(flags.get("requires_comparison", False)),
+ confidence=0.66 if intent != "non_temporal" else 0.25,
+ metadata={
+ "source": self.version,
+ "session_timestamp": clean_text(session_timestamp),
+ },
+ )
+
+ def _infer_subject(self, query: str) -> str:
+ lowered = clean_text(query).lower()
+ if any(token in lowered for token in ("phone", "iphone", "android", "手机")):
+ return "phone"
+ if any(token in lowered for token in ("project", "api", "benchmark", "论文", "项目")):
+ return "project_focus"
+ if any(token in lowered for token in ("diet", "meal", "food", "ingredient", "饮食", "食物", "食材")):
+ return "food"
+ if any(token in lowered for token in ("plan", "schedule", "calendar", "计划", "日程")):
+ return "plan"
+ return ""
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/temporal_router_runtime.py b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/temporal_router_runtime.py
new file mode 100644
index 0000000000000000000000000000000000000000..76eaf1726a8349478fcffcfdfd0178f3db5b17f1
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/temporal_router_runtime.py
@@ -0,0 +1,220 @@
+from __future__ import annotations
+
+import re
+from pathlib import Path
+from typing import Any, Dict, Mapping
+
+try:
+ import torch
+ from torch import Tensor, nn
+except ModuleNotFoundError:
+ torch = None
+ Tensor = Any
+ nn = None
+
+from experiments.replacement.temporal_modeling_types import clean_text
+
+
+TEMPORAL_ROUTER_ARCHITECTURE = "hashed_text_multihead_classifier_v1"
+
+
+def _tokens(text: str) -> list[str]:
+ normalized = clean_text(text).lower()
+ english = re.findall(r"[a-z0-9_.-]+", normalized)
+ cjk = [char for char in normalized if "\u4e00" <= char <= "\u9fff"]
+ return [*english, *cjk]
+
+
+def _stable_hash(value: str) -> int:
+ h = 2166136261
+ for byte in value.encode("utf-8"):
+ h ^= byte
+ h = (h * 16777619) & 0xFFFFFFFF
+ return h
+
+
+def _vectorize_text(text: str, *, buckets: int) -> Tensor:
+ if torch is None:
+ raise RuntimeError("torch is required for temporal router runtime")
+ vector = torch.zeros((buckets,), dtype=torch.float32)
+ tokens = _tokens(text)
+ if not tokens:
+ return vector
+ features: list[str] = []
+ features.extend(tokens)
+ features.extend(f"{tokens[index]}__{tokens[index + 1]}" for index in range(len(tokens) - 1))
+ for item in features:
+ vector[_stable_hash(item) % buckets] += 1.0
+ norm = float(vector.norm().item())
+ if norm > 0.0:
+ vector /= norm
+ return vector
+
+
+class MultiHeadClassifier(nn.Module if nn is not None else object):
+ def __init__(self, *, input_dim: int, hidden_dim: int, heads: Mapping[str, int]) -> None:
+ if nn is None:
+ raise RuntimeError("torch is required for temporal router runtime")
+ super().__init__()
+ self.encoder = nn.Sequential(
+ nn.Linear(input_dim, hidden_dim),
+ nn.LayerNorm(hidden_dim),
+ nn.GELU(),
+ nn.Dropout(0.08),
+ nn.Linear(hidden_dim, hidden_dim),
+ nn.GELU(),
+ )
+ self.heads = nn.ModuleDict({key: nn.Linear(hidden_dim, int(size)) for key, size in heads.items()})
+
+ def forward(self, features: Tensor) -> Dict[str, Tensor]:
+ hidden = self.encoder(features)
+ return {key: head(hidden) for key, head in self.heads.items()}
+
+
+class LoadedTemporalRouter:
+ """Runtime wrapper for the trained temporal writer/query router checkpoints."""
+
+ def __init__(
+ self,
+ *,
+ writer_model_path: str = "",
+ query_model_path: str = "",
+ device: str = "cpu",
+ ) -> None:
+ if torch is None:
+ raise RuntimeError("torch is required for temporal router runtime")
+ self.device = torch.device(device or "cpu")
+ self.writer = self._load_model(writer_model_path, expected_task="writer") if writer_model_path else None
+ self.query = self._load_model(query_model_path, expected_task="query") if query_model_path else None
+
+ @classmethod
+ def from_dir(cls, model_dir: str | Path, *, device: str = "cpu") -> "LoadedTemporalRouter":
+ root = Path(model_dir)
+ return cls(
+ writer_model_path=str(root / "writer_temporal_router.pt"),
+ query_model_path=str(root / "query_temporal_router.pt"),
+ device=device,
+ )
+
+ def writer_available(self) -> bool:
+ return self.writer is not None
+
+ def query_available(self) -> bool:
+ return self.query is not None
+
+ def predict_writer_frame(
+ self,
+ *,
+ current_turn: str,
+ previous_turn: str = "",
+ session_timestamp: str = "",
+ ) -> Dict[str, Any]:
+ if self.writer is None:
+ return {}
+ text = "\n".join(
+ part
+ for part in [
+ f"previous: {clean_text(previous_turn)}",
+ f"current: {clean_text(current_turn)}",
+ f"timestamp: {clean_text(session_timestamp)}",
+ ]
+ if clean_text(part)
+ )
+ prediction = self._predict(self.writer, text)
+ labels = dict(prediction.get("labels", {}) or {})
+ confidences = dict(prediction.get("confidences", {}) or {})
+ confidence = self._mean_confidence(confidences)
+ return {
+ "temporal_intent": clean_text(labels.get("temporal_intent", "")),
+ "anchor_type": clean_text(labels.get("anchor_type", "")),
+ "granularity": clean_text(labels.get("granularity", "")),
+ "state_operation": clean_text(labels.get("state_operation", "")),
+ "should_create_timeline_edge": labels.get("should_create_timeline_edge") == "true",
+ "confidence": confidence,
+ "metadata": {
+ "temporal_router_runtime": True,
+ "temporal_router_task": "writer",
+ "temporal_router_confidences": confidences,
+ },
+ }
+
+ def predict_query_plan(self, *, query: str, session_timestamp: str = "") -> Dict[str, Any]:
+ if self.query is None:
+ return {}
+ text = "\n".join(
+ part
+ for part in [
+ f"query: {clean_text(query)}",
+ f"timestamp: {clean_text(session_timestamp)}",
+ ]
+ if clean_text(part)
+ )
+ prediction = self._predict(self.query, text)
+ labels = dict(prediction.get("labels", {}) or {})
+ confidences = dict(prediction.get("confidences", {}) or {})
+ confidence = self._mean_confidence(confidences)
+ return {
+ "query_temporal_intent": clean_text(labels.get("query_temporal_intent", "")),
+ "timeline_operation": clean_text(labels.get("timeline_operation", "")),
+ "prefer_current_state": labels.get("prefer_current_state") == "true",
+ "prefer_previous_state": labels.get("prefer_previous_state") == "true",
+ "requires_ordered_chain": labels.get("requires_ordered_chain") == "true",
+ "requires_comparison": labels.get("requires_comparison") == "true",
+ "confidence": confidence,
+ "metadata": {
+ "temporal_router_runtime": True,
+ "temporal_router_task": "query",
+ "temporal_router_confidences": confidences,
+ },
+ }
+
+ def _load_model(self, model_path: str, *, expected_task: str) -> Dict[str, Any]:
+ path = Path(model_path)
+ if not path.exists():
+ raise FileNotFoundError(str(path))
+ payload = torch.load(path, map_location=self.device, weights_only=True)
+ if clean_text(payload.get("architecture", "")) != TEMPORAL_ROUTER_ARCHITECTURE:
+ raise ValueError(f"Unsupported temporal router architecture: {payload.get('architecture', '')}")
+ task = clean_text(payload.get("task", ""))
+ if task and task != expected_task:
+ raise ValueError(f"Expected temporal router task {expected_task}, got {task}")
+ buckets = int(payload.get("buckets", 4096) or 4096)
+ hidden_dim = int(payload.get("hidden_dim", 256) or 256)
+ label_maps = {key: dict(value) for key, value in dict(payload.get("label_maps", {}) or {}).items()}
+ heads = {key: len(value) for key, value in label_maps.items()}
+ model = MultiHeadClassifier(input_dim=buckets, hidden_dim=hidden_dim, heads=heads).to(self.device)
+ model.load_state_dict(payload["state_dict"])
+ model.eval()
+ return {
+ "path": str(path),
+ "model": model,
+ "buckets": buckets,
+ "label_maps": label_maps,
+ "metrics": {
+ "val": dict(payload.get("val_metrics", {}) or {}),
+ "test": dict(payload.get("test_metrics", {}) or {}),
+ },
+ }
+
+ def _predict(self, loaded: Mapping[str, Any], text: str) -> Dict[str, Any]:
+ model = loaded["model"]
+ buckets = int(loaded["buckets"])
+ label_maps = dict(loaded["label_maps"])
+ features = _vectorize_text(text, buckets=buckets).unsqueeze(0).to(self.device)
+ with torch.no_grad():
+ outputs = model(features)
+ labels: Dict[str, str] = {}
+ confidences: Dict[str, float] = {}
+ for key, logits in dict(outputs).items():
+ probs = torch.softmax(logits[0].detach().float().cpu(), dim=-1)
+ index = int(torch.argmax(probs).item())
+ reverse = {int(value): str(label) for label, value in dict(label_maps.get(key, {}) or {}).items()}
+ labels[key] = reverse.get(index, "")
+ confidences[key] = round(float(probs[index].item()), 6)
+ return {"labels": labels, "confidences": confidences}
+
+ def _mean_confidence(self, confidences: Mapping[str, Any]) -> float:
+ values = [float(value) for value in dict(confidences or {}).values()]
+ if not values:
+ return 0.0
+ return round(sum(values) / len(values), 6)
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/timeline_evidence_pack.py b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/timeline_evidence_pack.py
new file mode 100644
index 0000000000000000000000000000000000000000..f889845c113a9937ee929ef14e417f6b309d1468
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/timeline_evidence_pack.py
@@ -0,0 +1,70 @@
+from __future__ import annotations
+
+from typing import Any
+
+from experiments.replacement.temporal_modeling_types import TemporalQueryPlan, TimelineEvidencePack, TimelineState, clean_text
+from experiments.replacement.timeline_state_layer import TimelineStateLayer
+
+
+class TimelineEvidencePackBuilder:
+ """Build structured temporal evidence for the answer model."""
+
+ version = "timeline_evidence_pack_v1"
+
+ def build(self, *, plan: TemporalQueryPlan, timeline_layer: TimelineStateLayer) -> TimelineEvidencePack:
+ subject_key = clean_text(plan.target_subject_key)
+ states = timeline_layer.timeline(subject_key) if subject_key else []
+ selected: TimelineState | None = None
+ if plan.timeline_operation == "query_current":
+ selected = timeline_layer.current_state(subject_key)
+ elif plan.timeline_operation == "query_previous":
+ selected = timeline_layer.previous_state(subject_key)
+ elif plan.timeline_operation == "query_compare":
+ selected = timeline_layer.current_state(subject_key)
+ elif states:
+ selected = states[-1]
+ timeline = [self._state_payload(state) for state in states]
+ evidence = self._selected_payload(selected, plan=plan)
+ confidence = selected.confidence if selected is not None else 0.0
+ return TimelineEvidencePack(
+ mode=plan.timeline_operation,
+ subject=plan.target_subject,
+ subject_key=subject_key,
+ timeline=timeline,
+ selected_evidence=evidence,
+ evidence_confidence=confidence,
+ )
+
+ def _state_payload(self, state: TimelineState) -> dict[str, Any]:
+ return {
+ "time": clean_text(state.valid_from or state.event_time or state.utterance_time),
+ "state": clean_text(state.state_value),
+ "state_id": state.state_id,
+ "source_event_id": state.source_event_id,
+ "source_turn_id": state.source_turn_id,
+ "source_text": clean_text(state.metadata.get("source_text", "")),
+ "is_current": bool(state.is_current),
+ "valid_to": clean_text(state.valid_to),
+ }
+
+ def _selected_payload(self, state: TimelineState | None, *, plan: TemporalQueryPlan) -> dict[str, Any]:
+ if state is None:
+ return {
+ "answer_value": "",
+ "state_id": "",
+ "source_event_id": "",
+ "reason": "no_timeline_state_found",
+ }
+ reason = {
+ "query_current": "selected current state",
+ "query_previous": "selected previous state before current state",
+ "query_timeline": "selected latest state in ordered timeline",
+ "query_compare": "selected current state for comparison",
+ }.get(plan.timeline_operation, "selected timeline state")
+ return {
+ "answer_value": clean_text(state.state_value),
+ "state_id": state.state_id,
+ "source_event_id": state.source_event_id,
+ "source_turn_id": state.source_turn_id,
+ "reason": reason,
+ }
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/timeline_state_layer.py b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/timeline_state_layer.py
new file mode 100644
index 0000000000000000000000000000000000000000..ed47efb34ec24bd7f93dc69c4236211fe3595c65
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/timeline_state_layer.py
@@ -0,0 +1,179 @@
+from __future__ import annotations
+
+from collections import defaultdict
+from typing import Any, Dict, List
+
+from experiments.replacement.temporal_modeling_types import (
+ TemporalFrame,
+ TimelineEdge,
+ TimelineState,
+ clean_text,
+ normalize_key,
+)
+
+
+class TimelineStateLayer:
+ """In-memory timeline state layer for temporal smoke tests and runtime wiring."""
+
+ version = "timeline_state_layer_v1"
+
+ def __init__(self) -> None:
+ self._states: Dict[str, List[TimelineState]] = defaultdict(list)
+ self._edges: List[TimelineEdge] = []
+ self._counter = 0
+
+ def apply_frame(
+ self,
+ frame: TemporalFrame,
+ *,
+ source_event_id: str = "",
+ source_turn_id: str = "",
+ state_type: str = "profile",
+ ) -> TimelineState | None:
+ subject_key = clean_text(frame.subject_key) or normalize_key(frame.subject)
+ if not subject_key or not clean_text(frame.new_state):
+ return None
+ valid_from = clean_text(frame.state_valid_from or frame.event_time or frame.resolved_start or frame.utterance_time)
+ previous = self.current_state(subject_key)
+ if frame.old_state and previous is None:
+ previous = self._create_state(
+ subject_key=subject_key,
+ subject_label=frame.subject,
+ value=frame.old_state,
+ valid_from="",
+ valid_to=valid_from,
+ event_time="",
+ utterance_time=frame.utterance_time,
+ is_current=False,
+ source_event_id=source_event_id,
+ source_turn_id=source_turn_id,
+ state_type=state_type,
+ confidence=frame.confidence,
+ metadata={
+ "synthetic_previous_from_frame": True,
+ "temporal_intent": frame.temporal_intent,
+ "state_operation": frame.state_operation,
+ "time_expression": frame.time_expression,
+ "source_text": clean_text(frame.evidence_span),
+ "timeline_state_layer": self.version,
+ },
+ )
+ self._states[subject_key].append(previous)
+ if previous is not None and previous.is_current:
+ previous.is_current = False
+ previous.valid_to = previous.valid_to or valid_from
+ state = self._create_state(
+ subject_key=subject_key,
+ subject_label=frame.subject,
+ value=frame.new_state,
+ valid_from=valid_from,
+ valid_to=clean_text(frame.state_valid_to),
+ event_time=clean_text(frame.event_time),
+ utterance_time=clean_text(frame.utterance_time),
+ is_current=True,
+ supersedes_state_id=previous.state_id if previous is not None else "",
+ source_event_id=source_event_id,
+ source_turn_id=source_turn_id,
+ state_type=state_type,
+ confidence=frame.confidence,
+ metadata={
+ "temporal_intent": frame.temporal_intent,
+ "state_operation": frame.state_operation,
+ "time_expression": frame.time_expression,
+ "source_text": clean_text(frame.evidence_span),
+ "timeline_state_layer": self.version,
+ },
+ )
+ self._states[subject_key].append(state)
+ if previous is not None:
+ self._edges.append(
+ TimelineEdge(
+ edge_id=self._next_id("edge"),
+ subject_key=subject_key,
+ from_state_id=previous.state_id,
+ to_state_id=state.state_id,
+ edge_type="supersedes" if frame.state_operation == "supersede" else "updates",
+ transition_time=valid_from,
+ transition_reason=frame.evidence_span,
+ source_event_id=source_event_id,
+ confidence=frame.confidence,
+ )
+ )
+ return state
+
+ def current_state(self, subject_key: str) -> TimelineState | None:
+ states = self._states.get(clean_text(subject_key), [])
+ for state in reversed(states):
+ if state.is_current:
+ return state
+ return states[-1] if states else None
+
+ def previous_state(self, subject_key: str) -> TimelineState | None:
+ states = self.timeline(subject_key)
+ if len(states) < 2:
+ return None
+ current = self.current_state(subject_key)
+ for state in reversed(states):
+ if current is None or state.state_id != current.state_id:
+ return state
+ return None
+
+ def timeline(self, subject_key: str) -> List[TimelineState]:
+ states = list(self._states.get(clean_text(subject_key), []))
+ return sorted(states, key=lambda item: (clean_text(item.valid_from), clean_text(item.event_time), item.state_id))
+
+ def edges(self, subject_key: str = "") -> List[TimelineEdge]:
+ key = clean_text(subject_key)
+ if not key:
+ return list(self._edges)
+ return [edge for edge in self._edges if edge.subject_key == key]
+
+ def to_dict(self) -> dict[str, Any]:
+ return {
+ "version": self.version,
+ "subjects": {
+ subject: [state.to_dict() for state in states]
+ for subject, states in sorted(self._states.items())
+ },
+ "edges": [edge.to_dict() for edge in self._edges],
+ }
+
+ def _create_state(
+ self,
+ *,
+ subject_key: str,
+ subject_label: str,
+ value: str,
+ valid_from: str,
+ valid_to: str,
+ event_time: str,
+ utterance_time: str,
+ is_current: bool,
+ source_event_id: str,
+ source_turn_id: str,
+ state_type: str,
+ confidence: float,
+ supersedes_state_id: str = "",
+ metadata: dict[str, Any] | None = None,
+ ) -> TimelineState:
+ return TimelineState(
+ state_id=self._next_id("state"),
+ subject_key=subject_key,
+ subject_label=clean_text(subject_label),
+ state_value=clean_text(value),
+ state_type=clean_text(state_type) or "profile",
+ valid_from=valid_from,
+ valid_to=valid_to,
+ event_time=event_time,
+ utterance_time=utterance_time,
+ is_current=bool(is_current),
+ supersedes_state_id=supersedes_state_id,
+ source_event_id=clean_text(source_event_id),
+ source_turn_id=clean_text(source_turn_id),
+ confidence=confidence,
+ metadata=dict(metadata or {}),
+ )
+
+ def _next_id(self, prefix: str) -> str:
+ self._counter += 1
+ return f"{prefix}_{self._counter:06d}"
diff --git a/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/typed_tunnel_augmentation.py b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/typed_tunnel_augmentation.py
new file mode 100644
index 0000000000000000000000000000000000000000..1394e0f70228187d71187c9132374444ef5127f7
--- /dev/null
+++ b/benchmarks/longmemeval/src/tmcra_benchmark/vendor/tmcra_integrated/experiments/replacement/typed_tunnel_augmentation.py
@@ -0,0 +1,423 @@
+from __future__ import annotations
+
+import re
+from typing import Any, Iterable, Mapping, Sequence
+
+
+TYPED_TUNNEL_VERSION = "typed_tunnel_v1"
+
+NODE_TAG_KEY = "tmcra_node_tags"
+EDGE_TAG_KEY = "tmcra_edge_tags"
+PATH_TAG_KEY = "tmcra_path_tags"
+TUNNEL_ROLE_KEY = "tmcra_tunnel_roles"
+TUNNEL_GROUP_KEY = "tmcra_tunnel_group_key"
+TEMPORAL_RELATION_KEY = "tmcra_temporal_relation"
+MEASURE_SIGNATURE_KEY = "tmcra_measure_signature"
+ACTION_SIGNATURE_KEY = "tmcra_action_signature"
+ENTITY_SIGNATURE_KEY = "tmcra_entity_signature"
+
+_TOKEN_RE = re.compile(r"[a-z0-9]+|[\u4e00-\u9fff]+", re.I)
+_MONEY_RE = re.compile(
+ r"(?:[$€£]\s*\d+(?:[\d,]*)(?:\.\d+)?|\b\d+(?:[\d,]*)(?:\.\d+)?\s*(?:dollars?|usd|eur|pounds?|yuan|rmb)\b)",
+ re.I,
+)
+_QUANTITY_RE = re.compile(
+ r"\b\d+(?:[\d,]*)(?:\.\d+)?\s*(?:items?|tickets?|orders?|times?|days?|weeks?|months?|years?|people|users?|engineers?|hours?|minutes?|eggs?|cars?|bikes?|books?|meals?|servings?|units?)\b",
+ re.I,
+)
+_DATE_RE = re.compile(
+ r"\b(?:20\d{2}[-/]\d{1,2}[-/]\d{1,2}|\d{1,2}[-/]\d{1,2}[-/]20\d{2}|jan(?:uary)?|feb(?:ruary)?|mar(?:ch)?|apr(?:il)?|may|jun(?:e)?|jul(?:y)?|aug(?:ust)?|sep(?:tember)?|oct(?:ober)?|nov(?:ember)?|dec(?:ember)?|monday|tuesday|wednesday|thursday|friday|saturday|sunday|today|tomorrow|yesterday|last|next|before|after|earlier|later|then|finally)\b",
+ re.I,
+)
+
+_STOPWORDS = {
+ "a",
+ "an",
+ "and",
+ "are",
+ "as",
+ "at",
+ "be",
+ "but",
+ "by",
+ "for",
+ "from",
+ "i",
+ "in",
+ "is",
+ "it",
+ "me",
+ "my",
+ "of",
+ "on",
+ "or",
+ "our",
+ "the",
+ "their",
+ "this",
+ "that",
+ "to",
+ "was",
+ "we",
+ "with",
+ "you",
+}
+
+_ACTION_MARKERS = (
+ "bought",
+ "buy",
+ "purchased",
+ "purchase",
+ "ordered",
+ "order",
+ "sold",
+ "sell",
+ "paid",
+ "pay",
+ "returned",
+ "return",
+ "exchanged",
+ "exchange",
+ "picked",
+ "pick",
+ "visited",
+ "visit",
+ "serviced",
+ "service",
+ "cleaned",
+ "clean",
+ "made",
+ "make",
+ "ate",
+ "eat",
+ "drank",
+ "drink",
+ "met",
+ "meet",
+ "called",
+ "call",
+ "scheduled",
+ "schedule",
+ "cancelled",
+ "canceled",
+ "cancel",
+)
+
+
+def _clean_text(value: Any) -> str:
+ if value is None:
+ return ""
+ return re.sub(r"\s+", " ", str(value).replace("\x00", " ")).strip()
+
+
+def _normalize(value: Any) -> str:
+ return _clean_text(value).lower()
+
+
+def _tokens(value: Any) -> list[str]:
+ return [item.lower() for item in _TOKEN_RE.findall(_clean_text(value))]
+
+
+def _dedupe(items: Iterable[Any], *, max_items: int | None = None) -> list[str]:
+ values: list[str] = []
+ seen: set[str] = set()
+ for item in items:
+ text = _clean_text(item)
+ if not text:
+ continue
+ key = _normalize(text)
+ if key in seen:
+ continue
+ seen.add(key)
+ values.append(text)
+ if max_items is not None and len(values) >= max_items:
+ break
+ return values
+
+
+def _first_match(pattern: re.Pattern[str], text: str) -> str:
+ match = pattern.search(text)
+ return _clean_text(match.group(0)) if match else ""
+
+
+def _signature_slug(parts: Sequence[Any], *, max_tokens: int = 6) -> str:
+ tokens: list[str] = []
+ for part in parts:
+ for token in _tokens(part):
+ if token in _STOPWORDS:
+ continue
+ tokens.append(token)
+ if len(tokens) >= max_tokens:
+ break
+ if len(tokens) >= max_tokens:
+ break
+ return ".".join(_dedupe(tokens, max_items=max_tokens))
+
+
+def _infer_entity_signature(
+ *,
+ text: str,
+ metadata: Mapping[str, Any],
+ anchors: Sequence[Any],
+ slot_key: Any,
+ category: Any,
+) -> str:
+ explicit = (
+ _clean_text(metadata.get(ENTITY_SIGNATURE_KEY, ""))
+ or _clean_text(metadata.get("subject_signature", ""))
+ or _clean_text(metadata.get("memory_chain_subject_signature", ""))
+ or _clean_text(metadata.get("profile_subject_signature", ""))
+ or _clean_text(metadata.get("topic_bucket_id", ""))
+ )
+ if explicit:
+ return explicit
+ anchor_signature = _signature_slug(list(anchors or []), max_tokens=5)
+ if anchor_signature:
+ return anchor_signature
+ slot_signature = _signature_slug([slot_key, category], max_tokens=5)
+ if slot_signature:
+ return slot_signature
+ return _signature_slug([text], max_tokens=5)
+
+
+def _infer_action_signature(text: str, metadata: Mapping[str, Any]) -> str:
+ explicit = _clean_text(metadata.get(ACTION_SIGNATURE_KEY, ""))
+ if explicit:
+ return explicit
+ tokens = _tokens(text)
+ for index, token in enumerate(tokens):
+ if token in _ACTION_MARKERS:
+ tail = [item for item in tokens[index + 1 : index + 4] if item not in _STOPWORDS]
+ return ".".join(_dedupe([token, *tail], max_items=4))
+ return ""
+
+
+def _infer_temporal_relation(text: str, metadata: Mapping[str, Any]) -> str:
+ explicit = _clean_text(metadata.get(TEMPORAL_RELATION_KEY, ""))
+ if explicit:
+ return explicit
+ normalized = _normalize(text)
+ if "immediately before" in normalized or "right before" in normalized:
+ return "immediate_before"
+ if "immediately after" in normalized or "right after" in normalized:
+ return "immediate_after"
+ if "before" in normalized or "earlier" in normalized:
+ return "before"
+ if "after" in normalized or "later" in normalized or "then" in normalized:
+ return "after"
+ if "latest" in normalized or "current" in normalized or "now" in normalized:
+ return "current"
+ if "first" in normalized or "earliest" in normalized:
+ return "first"
+ if "last" in normalized or "finally" in normalized:
+ return "last"
+ return ""
+
+
+def infer_typed_tunnel_metadata(
+ *,
+ value: Any,
+ category: Any = "",
+ relation: Any = "",
+ slot_key: Any = "",
+ anchors: Sequence[Any] = (),
+ metadata: Mapping[str, Any] | None = None,
+ source_text: Any = "",
+) -> dict[str, Any]:
+ base = dict(metadata or {})
+ text = _clean_text(" ".join([_clean_text(source_text), _clean_text(value), _clean_text(relation), _clean_text(category)]))
+ normalized = _normalize(text)
+ node_tags = list(base.get(NODE_TAG_KEY, []) or [])
+ path_tags = list(base.get(PATH_TAG_KEY, []) or [])
+ tunnel_roles = list(base.get(TUNNEL_ROLE_KEY, []) or [])
+ action_frame_schema = _clean_text(base.get("unit_attachment_schema", ""))
+ action_frame_id = _clean_text(base.get("action_frame_id", ""))
+ applies_to_action = _clean_text(base.get("applies_to_action", "")) or _clean_text(base.get("action", ""))
+ applies_to_entity = _clean_text(base.get("applies_to_entity", "")) or _clean_text(base.get("target", ""))
+ unit_kind = _clean_text(base.get("unit_kind", ""))
+ facet_type = _clean_text(base.get("facet_type", ""))
+ temporal_role = _clean_text(base.get("temporal_role", ""))
+ numeric_unit = _clean_text(base.get("unit", "")) or _clean_text(base.get("numeric_unit", ""))
+
+ money_signature = _clean_text(base.get(MEASURE_SIGNATURE_KEY, "")) or _first_match(_MONEY_RE, text)
+ quantity_signature = _first_match(_QUANTITY_RE, text)
+ measure_signature = _clean_text(base.get(MEASURE_SIGNATURE_KEY, "")) or numeric_unit or money_signature or quantity_signature
+ temporal_relation = _clean_text(base.get(TEMPORAL_RELATION_KEY, "")) or temporal_role or _infer_temporal_relation(text, base)
+ action_signature = _clean_text(base.get(ACTION_SIGNATURE_KEY, "")) or _signature_slug([applies_to_action], max_tokens=4) or _infer_action_signature(text, base)
+ entity_signature = _clean_text(base.get(ENTITY_SIGNATURE_KEY, "")) or _signature_slug([applies_to_entity], max_tokens=6)
+ if not entity_signature:
+ entity_signature = _infer_entity_signature(
+ text=text,
+ metadata=base,
+ anchors=anchors,
+ slot_key=slot_key,
+ category=category,
+ )
+
+ if action_frame_schema == "event_action_frame_v2" or action_frame_id:
+ node_tags.append("event_action_frame_unit")
+ path_tags.append("unit_tunnel_path")
+ path_tags.append("multi_support_path")
+ tunnel_roles.append("unit_positive_candidate")
+ if unit_kind:
+ node_tags.append(unit_kind)
+ if facet_type:
+ node_tags.append(f"{facet_type}_unit")
+ if action_frame_id:
+ node_tags.append("action_frame_bound")
+ if applies_to_action:
+ node_tags.append("action_instance")
+ if applies_to_entity:
+ node_tags.append("entity_bound")
+ if temporal_role:
+ node_tags.append("time_anchor")
+ node_tags.append("timeline_event")
+ path_tags.append("temporal_order_path")
+ if facet_type == "numeric" or unit_kind == "numeric_quantity":
+ node_tags.append("measure_fact")
+ path_tags.append("aggregation_support_path")
+ tunnel_roles.append("multi_positive_candidate")
+
+ if money_signature:
+ node_tags.append("money_fact")
+ node_tags.append("measure_fact")
+ path_tags.append("aggregation_support_path")
+ tunnel_roles.append("multi_positive_candidate")
+ if quantity_signature:
+ node_tags.append("quantity_fact")
+ node_tags.append("measure_fact")
+ path_tags.append("aggregation_support_path")
+ tunnel_roles.append("multi_positive_candidate")
+ if action_signature:
+ node_tags.append("action_instance")
+ path_tags.append("multi_support_path")
+ if _DATE_RE.search(text) or _clean_text(base.get("resolved_date", "")) or _clean_text(base.get("time_value", "")):
+ node_tags.append("time_anchor")
+ node_tags.append("timeline_event")
+ path_tags.append("temporal_order_path")
+ if temporal_relation:
+ path_tags.append("temporal_order_path")
+ if temporal_relation in {"before", "after", "immediate_before", "immediate_after"}:
+ tunnel_roles.append("temporal_anchor_candidate")
+ if any(marker in normalized for marker in ("current", "now", "latest", "updated", "changed to")):
+ node_tags.append("current_state")
+ path_tags.append("current_value_path")
+ if any(marker in normalized for marker in ("previous", "old", "former", "used to", "earlier", "historical")):
+ node_tags.append("historical_state")
+ if any(marker in normalized for marker in ("plan", "planning", "might", "maybe", "consider", "could", "would", "if ")):
+ node_tags.append("planned_state")
+ node_tags.append("hypothetical_state")
+ if any(marker in normalized for marker in ("not ", "never", "cancel", "exclude", "returned", "refund")):
+ node_tags.append("negative_or_excluded")
+ if _normalize(category) in {"profile", "preference", "constraint", "goal"}:
+ path_tags.append("profile_bridge_path")
+
+ group_parts = [entity_signature, action_signature, measure_signature]
+ if action_frame_schema == "event_action_frame_v2" and facet_type in {"temporal", "state"}:
+ group_parts.append(f"frame:{action_frame_id or applies_to_action}")
+ if temporal_relation:
+ group_parts.append(f"time:{temporal_relation}")
+ tunnel_group_key = _clean_text(base.get(TUNNEL_GROUP_KEY, "")) or _signature_slug(group_parts, max_tokens=8)
+
+ typed_signature_terms = _dedupe(
+ [
+ *node_tags,
+ *path_tags,
+ *tunnel_roles,
+ entity_signature and f"entity {entity_signature}",
+ action_signature and f"action {action_signature}",
+ measure_signature and f"measure {measure_signature}",
+ temporal_relation and f"temporal {temporal_relation}",
+ ],
+ max_items=24,
+ )
+
+ return {
+ "tmcra_typed_tunnel_version": TYPED_TUNNEL_VERSION,
+ NODE_TAG_KEY: _dedupe(node_tags, max_items=16),
+ PATH_TAG_KEY: _dedupe(path_tags, max_items=16),
+ TUNNEL_ROLE_KEY: _dedupe(tunnel_roles, max_items=16),
+ TUNNEL_GROUP_KEY: tunnel_group_key,
+ TEMPORAL_RELATION_KEY: temporal_relation,
+ MEASURE_SIGNATURE_KEY: measure_signature,
+ ACTION_SIGNATURE_KEY: action_signature,
+ ENTITY_SIGNATURE_KEY: entity_signature,
+ "tmcra_typed_event_signature": " ".join(typed_signature_terms),
+ }
+
+
+def annotate_memory_record(record: Any, *, source_text: Any = "") -> Any:
+ metadata = dict(getattr(record, "metadata", {}) or {})
+ typed = infer_typed_tunnel_metadata(
+ value=getattr(record, "value", ""),
+ category=getattr(record, "category", ""),
+ relation=getattr(record, "relation", ""),
+ slot_key=getattr(record, "slot_key", ""),
+ anchors=list(getattr(record, "anchor_concepts", []) or []),
+ metadata=metadata,
+ source_text=source_text,
+ )
+ metadata.update({key: value for key, value in typed.items() if value not in ("", [], {})})
+ record.metadata = metadata
+ anchors = list(getattr(record, "anchor_concepts", []) or [])
+ evidence_anchors = list(getattr(record, "evidence_anchors", []) or [])
+ tag_anchors = [f"tag:{tag}" for tag in list(metadata.get(NODE_TAG_KEY, []) or [])[:6]]
+ if tag_anchors:
+ record.anchor_concepts = _dedupe([*anchors, *tag_anchors], max_items=24)
+ record.evidence_anchors = _dedupe([*evidence_anchors, *tag_anchors], max_items=24)
+ return record
+
+
+def typed_tunnel_signature_text(metadata: Mapping[str, Any] | None) -> str:
+ data = dict(metadata or {})
+ parts = [
+ data.get("tmcra_typed_event_signature", ""),
+ " ".join(str(item) for item in list(data.get(NODE_TAG_KEY, []) or [])),
+ " ".join(str(item) for item in list(data.get(PATH_TAG_KEY, []) or [])),
+ data.get(TUNNEL_GROUP_KEY, ""),
+ ]
+ return _clean_text(" ".join(_clean_text(item) for item in parts if _clean_text(item)))
+
+
+def typed_edge_tags_between(left: Mapping[str, Any] | None, right: Mapping[str, Any] | None) -> list[str]:
+ left_data = dict(left or {})
+ right_data = dict(right or {})
+ tags: list[str] = []
+ if _clean_text(left_data.get(ENTITY_SIGNATURE_KEY, "")) and _clean_text(left_data.get(ENTITY_SIGNATURE_KEY, "")) == _clean_text(right_data.get(ENTITY_SIGNATURE_KEY, "")):
+ tags.append("same_entity")
+ if _clean_text(left_data.get(ACTION_SIGNATURE_KEY, "")) and _clean_text(left_data.get(ACTION_SIGNATURE_KEY, "")) == _clean_text(right_data.get(ACTION_SIGNATURE_KEY, "")):
+ tags.append("same_action")
+ if _clean_text(left_data.get(MEASURE_SIGNATURE_KEY, "")) and _clean_text(left_data.get(MEASURE_SIGNATURE_KEY, "")) == _clean_text(right_data.get(MEASURE_SIGNATURE_KEY, "")):
+ tags.append("same_measure")
+ if _clean_text(left_data.get(TUNNEL_GROUP_KEY, "")) and _clean_text(left_data.get(TUNNEL_GROUP_KEY, "")) == _clean_text(right_data.get(TUNNEL_GROUP_KEY, "")):
+ tags.append("same_tunnel_group")
+ left_temporal = _clean_text(left_data.get(TEMPORAL_RELATION_KEY, ""))
+ right_temporal = _clean_text(right_data.get(TEMPORAL_RELATION_KEY, ""))
+ left_tags = set(left_data.get(NODE_TAG_KEY, []) or [])
+ right_tags = set(right_data.get(NODE_TAG_KEY, []) or [])
+ if left_temporal or right_temporal or ("timeline_event" in left_tags and "timeline_event" in right_tags):
+ tags.append("same_timeline")
+ if "current_state" in left_tags and "historical_state" in right_tags:
+ tags.append("current_replaces_old")
+ return _dedupe(tags, max_items=8)
+
+
+def merge_typed_metadata(items: Iterable[Mapping[str, Any] | None]) -> dict[str, Any]:
+ merged: dict[str, Any] = {}
+ for item in items:
+ data = dict(item or {})
+ for key in (NODE_TAG_KEY, PATH_TAG_KEY, TUNNEL_ROLE_KEY):
+ merged[key] = _dedupe([*list(merged.get(key, []) or []), *list(data.get(key, []) or [])], max_items=24)
+ for key in (
+ TUNNEL_GROUP_KEY,
+ TEMPORAL_RELATION_KEY,
+ MEASURE_SIGNATURE_KEY,
+ ACTION_SIGNATURE_KEY,
+ ENTITY_SIGNATURE_KEY,
+ "tmcra_typed_event_signature",
+ ):
+ if not _clean_text(merged.get(key, "")) and _clean_text(data.get(key, "")):
+ merged[key] = _clean_text(data.get(key, ""))
+ if merged:
+ merged["tmcra_typed_tunnel_version"] = TYPED_TUNNEL_VERSION
+ return merged
diff --git a/benchmarks/longmemeval/tests/test_answer_gold_isolation.py b/benchmarks/longmemeval/tests/test_answer_gold_isolation.py
new file mode 100644
index 0000000000000000000000000000000000000000..65256a1bd5f79a5bad58dea66d5802295c9d6c09
--- /dev/null
+++ b/benchmarks/longmemeval/tests/test_answer_gold_isolation.py
@@ -0,0 +1,42 @@
+from __future__ import annotations
+
+import ast
+from pathlib import Path
+
+
+ROOT = Path(__file__).resolve().parents[1]
+ANSWER_RUNNER = (
+ ROOT
+ / "src"
+ / "tmcra_benchmark"
+ / "legacy"
+ / "run_tmcra_v4_gpt54_answers.py"
+)
+
+
+def test_answer_runner_rejects_evaluation_only_fields() -> None:
+ tree = ast.parse(ANSWER_RUNNER.read_text(encoding="utf-8"), filename=str(ANSWER_RUNNER))
+ functions = {
+ node.name: node
+ for node in tree.body
+ if isinstance(node, ast.FunctionDef)
+ }
+ rejection = functions["reject_evaluation_fields"]
+ literals = {
+ node.value
+ for node in ast.walk(tree)
+ if isinstance(node, ast.Constant) and isinstance(node.value, str)
+ }
+ assert "gold_answer" in literals
+ assert "answer_session_ids" in literals
+ assert any(
+ isinstance(node, ast.Raise)
+ for node in ast.walk(rejection)
+ )
+
+
+def test_answer_output_does_not_persist_gold_or_gold_scores() -> None:
+ source = ANSWER_RUNNER.read_text(encoding="utf-8")
+ assert 'row.get("gold_answer")' not in source
+ assert '"deterministic_scores"' not in source
+ assert 'row.get("answer_session_ids")' not in source
diff --git a/benchmarks/longmemeval/tests/test_fetch_assets.py b/benchmarks/longmemeval/tests/test_fetch_assets.py
new file mode 100644
index 0000000000000000000000000000000000000000..bfcd944aad63d31587d3dd168755a51a3caae6a5
--- /dev/null
+++ b/benchmarks/longmemeval/tests/test_fetch_assets.py
@@ -0,0 +1,61 @@
+from __future__ import annotations
+
+import hashlib
+import importlib.util
+from pathlib import Path
+
+import pytest
+
+
+ROOT = Path(__file__).resolve().parents[1]
+
+
+def load_script():
+ path = ROOT / "scripts" / "fetch_assets.py"
+ spec = importlib.util.spec_from_file_location(path.stem, path)
+ module = importlib.util.module_from_spec(spec)
+ assert spec and spec.loader
+ spec.loader.exec_module(module)
+ return module
+
+
+def test_git_lfs_checkpoint_is_verified_in_place(tmp_path: Path) -> None:
+ module = load_script()
+ payload = b"checkpoint-bytes"
+ checkpoint = tmp_path / "weights.pt"
+ checkpoint.write_bytes(payload)
+ module.fetch_file(
+ tmp_path,
+ {
+ "id": "weights",
+ "kind": "checkpoint",
+ "storage": "git_lfs",
+ "destination": "weights.pt",
+ "bytes": len(payload),
+ "sha256": hashlib.sha256(payload).hexdigest(),
+ },
+ )
+ assert checkpoint.read_bytes() == payload
+
+
+def test_git_lfs_pointer_fails_with_pull_guidance(tmp_path: Path) -> None:
+ module = load_script()
+ pointer = tmp_path / "weights.pt"
+ pointer.write_text(
+ "version https://git-lfs.github.com/spec/v1\n"
+ "oid sha256:0000000000000000000000000000000000000000000000000000000000000000\n"
+ "size 10\n",
+ encoding="utf-8",
+ )
+ with pytest.raises(RuntimeError, match="git lfs pull"):
+ module.fetch_file(
+ tmp_path,
+ {
+ "id": "weights",
+ "kind": "checkpoint",
+ "storage": "git_lfs",
+ "destination": "weights.pt",
+ "bytes": 10,
+ "sha256": "0" * 64,
+ },
+ )
diff --git a/benchmarks/longmemeval/tests/test_prepare_longmemeval.py b/benchmarks/longmemeval/tests/test_prepare_longmemeval.py
new file mode 100644
index 0000000000000000000000000000000000000000..f242c00e0db14de90d1dc98f50737721dcc9d14e
--- /dev/null
+++ b/benchmarks/longmemeval/tests/test_prepare_longmemeval.py
@@ -0,0 +1,27 @@
+from __future__ import annotations
+
+import importlib.util
+from pathlib import Path
+
+
+ROOT = Path(__file__).resolve().parents[1]
+
+
+def load_script(name: str):
+ path = ROOT / "scripts" / name
+ spec = importlib.util.spec_from_file_location(path.stem, path)
+ module = importlib.util.module_from_spec(spec)
+ assert spec and spec.loader
+ spec.loader.exec_module(module)
+ return module
+
+
+def test_fixture_is_valid_and_gold_is_not_exported(tmp_path: Path) -> None:
+ module = load_script("prepare_longmemeval.py")
+ rows = module.load_and_validate(ROOT / "fixtures" / "tiny_longmemeval.json")
+ assert [row["question_id"] for row in rows] == ["tmcra_fixture_user", "tmcra_fixture_update"]
+ output = tmp_path / "qids.txt"
+ output.write_text("\n".join(str(row["question_id"]) for row in rows) + "\n", encoding="utf-8")
+ text = output.read_text(encoding="utf-8")
+ assert "Tea" not in text
+ assert "Shanghai" not in text
diff --git a/benchmarks/longmemeval/tests/test_safe_model_loading.py b/benchmarks/longmemeval/tests/test_safe_model_loading.py
new file mode 100644
index 0000000000000000000000000000000000000000..32fdaf00b431fbdf411f48c0401c31bebaac982b
--- /dev/null
+++ b/benchmarks/longmemeval/tests/test_safe_model_loading.py
@@ -0,0 +1,122 @@
+from __future__ import annotations
+
+import ast
+import importlib.util
+import json
+from pathlib import Path
+
+import pytest
+
+
+ROOT = Path(__file__).resolve().parents[1]
+SOURCE_ROOT = ROOT / "src" / "tmcra_benchmark"
+REPOSITORY_ROOT = ROOT.parents[1]
+SECURITY_SOURCE_ROOTS = (SOURCE_ROOT, REPOSITORY_ROOT / "code")
+
+
+def _dotted_name(node: ast.AST) -> str:
+ if isinstance(node, ast.Name):
+ return node.id
+ if isinstance(node, ast.Attribute):
+ prefix = _dotted_name(node.value)
+ return f"{prefix}.{node.attr}" if prefix else node.attr
+ return ""
+
+
+def _keyword(call: ast.Call, name: str) -> ast.AST | None:
+ return next((item.value for item in call.keywords if item.arg == name), None)
+
+
+def test_every_torch_load_is_weights_only() -> None:
+ violations: list[str] = []
+ for source_root in SECURITY_SOURCE_ROOTS:
+ for path in source_root.rglob("*.py"):
+ tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path))
+ torch_aliases = {
+ alias.asname or alias.name
+ for node in ast.walk(tree)
+ if isinstance(node, ast.Import)
+ for alias in node.names
+ if alias.name == "torch"
+ }
+ torch_aliases.update({"torch", "torch_module"})
+ for node in ast.walk(tree):
+ if not isinstance(node, ast.Call) or not isinstance(node.func, ast.Attribute):
+ continue
+ if node.func.attr != "load":
+ continue
+ receiver = _dotted_name(node.func.value)
+ if receiver not in torch_aliases and not receiver.endswith(".torch"):
+ continue
+ value = _keyword(node, "weights_only")
+ if not isinstance(value, ast.Constant) or value.value is not True:
+ violations.append(f"{path.relative_to(REPOSITORY_ROOT)}:{node.lineno}")
+ assert not violations, "torch.load must use weights_only=True: " + ", ".join(violations)
+
+
+def test_local_transformer_models_disable_remote_code() -> None:
+ violations: list[str] = []
+ for path in SOURCE_ROOT.rglob("*.py"):
+ tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path))
+ for node in ast.walk(tree):
+ if not isinstance(node, ast.Call) or not isinstance(node.func, ast.Attribute):
+ continue
+ if node.func.attr != "from_pretrained":
+ continue
+ owner = _dotted_name(node.func.value)
+ if not (owner.startswith("AutoTokenizer") or owner.startswith("AutoModel")):
+ continue
+ local_only = _keyword(node, "local_files_only")
+ remote_code = _keyword(node, "trust_remote_code")
+ if not isinstance(local_only, ast.Constant) or local_only.value is not True:
+ violations.append(f"{path.relative_to(ROOT)}:{node.lineno}:local_files_only")
+ if not isinstance(remote_code, ast.Constant) or remote_code.value is not False:
+ violations.append(f"{path.relative_to(ROOT)}:{node.lineno}:trust_remote_code")
+ assert not violations, "local model loading is not hardened: " + ", ".join(violations)
+
+
+def test_public_code_never_enables_remote_code() -> None:
+ violations: list[str] = []
+ for source_root in SECURITY_SOURCE_ROOTS:
+ for path in source_root.rglob("*.py"):
+ tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path))
+ for node in ast.walk(tree):
+ if not isinstance(node, ast.Call) or node.func is None:
+ continue
+ remote_code = _keyword(node, "trust_remote_code")
+ if isinstance(remote_code, ast.Constant) and remote_code.value is True:
+ violations.append(f"{path.relative_to(REPOSITORY_ROOT)}:{node.lineno}")
+ assert not violations, "public code enables trust_remote_code: " + ", ".join(violations)
+
+
+def _load_model_assets_module():
+ path = SOURCE_ROOT / "legacy" / "tmcra_model_assets.py"
+ spec = importlib.util.spec_from_file_location(path.stem, path)
+ assert spec and spec.loader
+ module = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(module)
+ return module
+
+
+def test_official_hf_model_does_not_require_tmcra_manifest(tmp_path: Path) -> None:
+ module = _load_model_assets_module()
+ (tmp_path / "config.json").write_text("{}\n", encoding="utf-8")
+
+ assert module.load_optional_hf_model_manifest(tmp_path) == {}
+
+ for relative in ("tmcra_v3_reranker.py", "tmcra_v3_online_runtime.py"):
+ source = (SOURCE_ROOT / "legacy" / relative).read_text(encoding="utf-8")
+ assert "load_optional_hf_model_manifest" in source
+ assert "pinned cross model manifest is required" not in source
+ assert "pinned model manifest is required" not in source
+
+
+def test_optional_tmcra_manifest_is_validated(tmp_path: Path) -> None:
+ module = _load_model_assets_module()
+ manifest = tmp_path / "TMCRA_MODEL_MANIFEST.json"
+ manifest.write_text(json.dumps({"repo_id": "BAAI/bge-reranker-v2-m3", "revision": "abc"}), encoding="utf-8")
+ assert module.load_optional_hf_model_manifest(tmp_path)["revision"] == "abc"
+
+ manifest.write_text("[]", encoding="utf-8")
+ with pytest.raises(ValueError, match="JSON object"):
+ module.load_optional_hf_model_manifest(tmp_path)
diff --git a/benchmarks/longmemeval/tests/test_score_breakdown.py b/benchmarks/longmemeval/tests/test_score_breakdown.py
new file mode 100644
index 0000000000000000000000000000000000000000..95ca81ba368f427edae47d1f5fef482bdb765997
--- /dev/null
+++ b/benchmarks/longmemeval/tests/test_score_breakdown.py
@@ -0,0 +1,32 @@
+from __future__ import annotations
+
+import importlib.util
+from pathlib import Path
+
+
+ROOT = Path(__file__).resolve().parents[1]
+
+
+def load_script():
+ path = ROOT / "scripts" / "summarize_official_judge.py"
+ spec = importlib.util.spec_from_file_location(path.stem, path)
+ module = importlib.util.module_from_spec(spec)
+ assert spec and spec.loader
+ spec.loader.exec_module(module)
+ return module
+
+
+def test_score_breakdown() -> None:
+ module = load_script()
+ dataset = [
+ {"question_id": "a", "question_type": "single-session-user"},
+ {"question_id": "b", "question_type": "knowledge-update"},
+ ]
+ judged = [
+ {"question_id": "a", "autoeval_label": {"label": True}},
+ {"question_id": "b", "autoeval_label": {"label": False}},
+ ]
+ result = module.summarize(dataset, judged)
+ assert result["correct"] == 1
+ assert result["total"] == 2
+ assert result["accuracy_percent"] == 50.0
diff --git a/benchmarks/longmemeval/tests/test_validate_stage.py b/benchmarks/longmemeval/tests/test_validate_stage.py
new file mode 100644
index 0000000000000000000000000000000000000000..d1aa9ba24916c168bdf7289b38c9aba730a3d3b8
--- /dev/null
+++ b/benchmarks/longmemeval/tests/test_validate_stage.py
@@ -0,0 +1,111 @@
+from __future__ import annotations
+
+import importlib.util
+import json
+from pathlib import Path
+
+import pytest
+
+
+ROOT = Path(__file__).resolve().parents[1]
+
+
+def load_script():
+ path = ROOT / "scripts" / "validate_stage.py"
+ spec = importlib.util.spec_from_file_location(path.stem, path)
+ module = importlib.util.module_from_spec(spec)
+ assert spec and spec.loader
+ spec.loader.exec_module(module)
+ return module
+
+
+def write_json(path: Path, value: object) -> None:
+ path.write_text(json.dumps(value) + "\n", encoding="utf-8")
+
+
+def write_jsonl(path: Path, rows: list[dict[str, object]]) -> None:
+ path.write_text("".join(json.dumps(row) + "\n" for row in rows), encoding="utf-8")
+
+
+def test_answer_validation_requires_complete_report(tmp_path: Path) -> None:
+ module = load_script()
+ write_json(
+ tmp_path / "report.json",
+ {
+ "status": "incomplete",
+ "requested": 1,
+ "completed": 1,
+ "failure_count": 0,
+ "answer_model": "gpt-5.4",
+ },
+ )
+ write_jsonl(tmp_path / "answers.jsonl", [{"question_id": "q1"}])
+ with pytest.raises(SystemExit, match="report is not complete"):
+ module.validate_answer(tmp_path, ["q1"])
+
+
+def test_answer_validation_requires_hardened_production_protocol(tmp_path: Path) -> None:
+ module = load_script()
+ report = {
+ "status": "complete",
+ "requested": 1,
+ "completed": 1,
+ "failure_count": 0,
+ "answer_model": "gpt-5.4",
+ "answer_protocol": "evidence_operation_bound_v6",
+ "lane": "production",
+ "promotion_eligible": True,
+ "compiled_evidence_packet_required": True,
+ }
+ write_json(tmp_path / "report.json", report)
+ write_jsonl(
+ tmp_path / "answers.jsonl",
+ [
+ {
+ "question_id": "q1",
+ "answer_model": "gpt-5.4",
+ "answer_protocol": "evidence_operation_bound_v6",
+ }
+ ],
+ )
+ module.validate_answer(tmp_path, ["q1"])
+
+ report["answer_protocol"] = "evidence_operation_bound_v5"
+ write_json(tmp_path / "report.json", report)
+ with pytest.raises(SystemExit, match="unexpected protocol"):
+ module.validate_answer(tmp_path, ["q1"])
+
+
+def test_judge_validation_requires_fixed_resolved_model(tmp_path: Path) -> None:
+ module = load_script()
+ results = tmp_path / "judge.jsonl"
+ write_jsonl(
+ results,
+ [
+ {
+ "question_id": "q1",
+ "autoeval_label": {
+ "model": "gpt-4o-2024-08-06",
+ "label": True,
+ "raw_response": "yes",
+ },
+ }
+ ],
+ )
+ write_json(
+ tmp_path / "judge.jsonl.isolated_report.json",
+ {
+ "status": "complete",
+ "metric_model": "gpt-4o",
+ "row_count": 1,
+ "committed_count": 1,
+ "failure_count": 0,
+ },
+ )
+ module.validate_judge(results, ["q1"])
+
+ rows = [json.loads(results.read_text(encoding="utf-8"))]
+ rows[0]["autoeval_label"]["model"] = "gpt-4o"
+ write_jsonl(results, rows)
+ with pytest.raises(SystemExit, match="unexpected model"):
+ module.validate_judge(results, ["q1"])
diff --git a/code/memory_adapters.py b/code/memory_adapters.py
index 32fe7609c0e1727a7aa81038a38f8a291c534c48..c5c74ddb82cf262751a08b12f3cd24fc0425a1b3 100644
--- a/code/memory_adapters.py
+++ b/code/memory_adapters.py
@@ -1,9190 +1,9190 @@
-from __future__ import annotations
-
-from collections import deque
-from dataclasses import dataclass, field
-import json
-import math
-import os
-from pathlib import Path
-import re
-import tempfile
-import time
-import uuid
-from typing import Any, Dict, Iterable, List, Mapping, Sequence
-
-from core.session_memory import SessionMemoryExtractor
-from experiments.replacement.memory_graph import (
- SessionMemoryGraphV2,
- SessionMemoryEdgeV2,
- SessionMemoryRecordV2,
- SQLiteSessionMemoryStore,
- _clean_text,
- _estimate_tokens,
- _normalize,
- _public_query_subject,
- _public_slot_root,
- _public_subject_signature,
- stable_slot_key,
- _tokenize,
- guess_slot_key,
- infer_category_hints,
-)
-from experiments.replacement.node_memory import (
- _call_with_supported_kwargs,
- DEFAULT_MATRIX_EVENT_TOP_K,
- LoadedNodeMemoryScorer,
- MEMORY_ROUTER_LAYERS,
- build_default_path_templates,
- extract_question_features,
-)
-from experiments.replacement.public_event_signature import compute_public_event_signature
-from experiments.replacement.memory_profiles import TMCRAProfile
-from experiments.replacement.typed_tunnel_augmentation import (
- annotate_memory_record,
- merge_typed_metadata,
- typed_edge_tags_between,
- typed_tunnel_signature_text,
-)
-from experiments.replacement.profile_layer import infer_profile_query_intent, is_profile_layer_record, profile_query_score_delta
-from experiments.replacement import injection_planner as injection_planner_runtime
-from experiments.replacement.temporal_modeling_types import TemporalFrame, TemporalQueryPlan
-from experiments.replacement.temporal_organizer import TemporalOrganizer
-from experiments.replacement.temporal_query_planner import TemporalQueryPlanner
-from experiments.replacement.temporal_router_runtime import LoadedTemporalRouter
-from experiments.replacement.timeline_evidence_pack import TimelineEvidencePackBuilder
-from experiments.replacement.timeline_state_layer import TimelineStateLayer
-
-from .base import MemoryAdapter, MemoryHit, MemoryRetrieval
-
-
-def _dedupe(items: Iterable[Any], *, max_items: int | None = None) -> List[str]:
- values: List[str] = []
- seen = set()
- for item in items:
- text = _clean_text(item)
- if not text:
- continue
- key = _normalize(text)
- if key in seen:
- continue
- seen.add(key)
- values.append(text)
- if max_items is not None and len(values) >= max_items:
- break
- return values
-
-
-def _apply_typed_tunnel_annotations(records: List[SessionMemoryRecordV2], *, source_text: str = "") -> List[SessionMemoryRecordV2]:
- for record in records:
- annotate_memory_record(record, source_text=source_text)
- return records
-
-
-def _estimate_tokens_from_hits(hits: Sequence[MemoryHit]) -> int:
- total = 0
- for hit in hits:
- total += _estimate_tokens(hit.value)
- total += sum(_estimate_tokens(anchor) for anchor in hit.anchors)
- return total
-
-
-def _float_env(name: str, default: float) -> float:
- raw = _clean_text(os.getenv(name, ""))
- if not raw:
- return float(default)
- try:
- return float(raw)
- except ValueError:
- return float(default)
-
-
-_GRAPH_PROMPT_MAX_CHARS = 12_000
-_GRAPH_PROMPT_MAX_HITS = 8
-_GRAPH_PROMPT_MAX_ACTIVE_SLOTS = 12
-_GRAPH_PROMPT_MAX_RELATIONS = 10
-_HYBRID_SELECTED_EVENT_FLOOR = 8
-_HYBRID_SELECTED_PATH_CAP = 6
-_HYBRID_TEMPORAL_PATH_CAP = 2
-_HYBRID_PROFILE_PATH_CAP = 3
-_MEMORY_ROUTER_GUIDED_MODES = {"guided", "route", "routing", "enforce"}
-_MEMORY_ROUTER_FORCE_MODES = {"force", "forced"}
-_MEMORY_ROUTER_DISABLED_MODES = {"", "off", "disabled", "none", "false", "0"}
-_MEMORY_ROUTER_OBSERVE_MODES = {"observe", "observer", "telemetry", "shadow"}
-_MEMORY_ROUTER_DEFAULT_THRESHOLD = 0.55
-_MEMORY_ROUTER_DEFAULT_MARGIN = 0.08
-_INJECTION_PLANNER_GUIDED_MODES = {"guided", "route", "routing", "enforce"}
-_INJECTION_PLANNER_FORCE_MODES = {"force", "forced"}
-_INJECTION_PLANNER_DISABLED_MODES = {"", "off", "disabled", "none", "false", "0"}
-_INJECTION_PLANNER_OBSERVE_MODES = {"observe", "observer", "telemetry", "shadow"}
-_TEMPORAL_LAYER_DISABLED_MODES = {"off", "disabled", "none", "false", "0"}
-_TEMPORAL_ROUTER_DEFAULT_WRITER_MIN_CONFIDENCE = 0.72
-_TEMPORAL_ROUTER_DEFAULT_QUERY_MIN_CONFIDENCE = 0.85
-_TEMPORAL_ROUTER_DEFAULT_QUERY_INTENT_MIN_CONFIDENCE = 0.60
-_EMBEDDER_INDEX_DISABLED_MODES = {"", "off", "disabled", "none", "false", "0"}
-_EMBEDDER_INDEX_BGE_M3_MODES = {"bge", "bge_m3", "bge-m3", "baai_bge_m3", "baai/bge-m3"}
-_EMBEDDER_INDEX_VERSION = "write_hash_sparse_v1"
-_EMBEDDER_INDEX_BGE_M3_VERSION = "write_bge_m3_dense_sparse_v1"
-_EMBEDDER_MODEL_CACHE: Dict[str, Any] = {}
-_HYBRID_SYMBOLIC_STOPWORDS = {
- "a",
- "an",
- "and",
- "are",
- "as",
- "at",
- "be",
- "did",
- "do",
- "does",
- "event",
- "for",
- "from",
- "had",
- "has",
- "have",
- "how",
- "in",
- "is",
- "of",
- "on",
- "or",
- "should",
- "the",
- "to",
- "turn",
- "was",
- "were",
- "what",
- "when",
- "where",
- "which",
- "who",
- "why",
- "will",
- "with",
-}
-
-
-def _coerce_memory_router_scores(payload: Mapping[str, Any]) -> Dict[str, float]:
- raw_scores = dict(payload.get("memory_router_scores", {}) or {})
- scores: Dict[str, float] = {}
- for layer in MEMORY_ROUTER_LAYERS:
- try:
- scores[layer] = float(raw_scores.get(layer, 0.0))
- except (TypeError, ValueError):
- scores[layer] = 0.0
- return scores
-
-
-def _memory_router_decision(
- payload: Mapping[str, Any],
- *,
- mode: str,
- threshold: float,
- margin: float,
-) -> Dict[str, Any]:
- normalized_mode = _normalize(mode)
- scores = _coerce_memory_router_scores(payload)
- has_scores = bool(scores) and any(layer in dict(payload.get("memory_router_scores", {}) or {}) for layer in MEMORY_ROUTER_LAYERS)
- if not has_scores:
- return {
- "memory_router_enabled": False,
- "memory_router_guided": False,
- "memory_router_reason": "no_router_scores",
- "memory_router_scores": {},
- "memory_router_top_layers": [],
- "memory_router_active_layers": [],
- "memory_router_score_spread": 0.0,
- "memory_router_confidence": 0.0,
- }
- ranked_layers = [
- layer
- for layer, _ in sorted(
- scores.items(),
- key=lambda item: (-float(item[1]), item[0]),
- )
- ]
- score_values = list(scores.values())
- score_spread = max(score_values) - min(score_values)
- confidence = max(abs(float(score) - 0.5) for score in score_values)
- resolved_threshold = max(0.0, min(1.0, float(threshold or _MEMORY_ROUTER_DEFAULT_THRESHOLD)))
- resolved_margin = max(0.0, min(0.5, float(margin or _MEMORY_ROUTER_DEFAULT_MARGIN)))
- active_layers = [
- layer
- for layer in ranked_layers
- if float(scores.get(layer, 0.0)) >= resolved_threshold
- ]
- confident = bool(active_layers) and (score_spread >= resolved_margin or confidence >= resolved_margin)
- guided_requested = normalized_mode in _MEMORY_ROUTER_GUIDED_MODES
- forced = normalized_mode in _MEMORY_ROUTER_FORCE_MODES
- guided = bool(forced or (guided_requested and confident))
- if forced and not active_layers and ranked_layers:
- active_layers = [ranked_layers[0]]
- if active_layers and "event" not in active_layers:
- active_layers = ["event", *active_layers]
- reason = "observe"
- if normalized_mode in _MEMORY_ROUTER_DISABLED_MODES:
- reason = "disabled"
- elif normalized_mode in _MEMORY_ROUTER_OBSERVE_MODES:
- reason = "observe"
- elif forced:
- reason = "forced"
- elif guided_requested and not confident:
- reason = "low_confidence"
- elif guided:
- reason = "guided"
- return {
- "memory_router_enabled": True,
- "memory_router_guided": bool(guided and normalized_mode not in _MEMORY_ROUTER_DISABLED_MODES),
- "memory_router_reason": reason,
- "memory_router_scores": scores,
- "memory_router_top_layers": ranked_layers,
- "memory_router_active_layers": active_layers,
- "memory_router_score_spread": round(float(score_spread), 6),
- "memory_router_confidence": round(float(confidence), 6),
- "memory_router_threshold": round(float(resolved_threshold), 6),
- "memory_router_margin": round(float(resolved_margin), 6),
- "memory_router_mode": normalized_mode or "observe",
- }
-
-
-def _memory_router_allows(decision: Mapping[str, Any], *layers: str) -> bool:
- if not bool(decision.get("memory_router_guided")):
- return True
- requested = {_normalize(layer) for layer in layers if _normalize(layer)}
- if not requested:
- return True
- if "event" in requested:
- return True
- active = {_normalize(layer) for layer in list(decision.get("memory_router_active_layers", []) or [])}
- return bool(active & requested)
-
-
-def _hybrid_symbolic_tokens(value: Any) -> List[str]:
- return [
- token
- for token in _tokenize(value)
- if token and token not in _HYBRID_SYMBOLIC_STOPWORDS and not re.fullmatch(r"\d+", token)
- ]
-
-
-_PATH_UTILITY_STOPWORDS = set(_HYBRID_SYMBOLIC_STOPWORDS) | {
- "actually",
- "also",
- "bit",
- "feel",
- "feels",
- "get",
- "guess",
- "i",
- "im",
- "it",
- "its",
- "just",
- "kind",
- "like",
- "maybe",
- "me",
- "more",
- "much",
- "my",
- "really",
- "something",
- "still",
- "think",
- "thats",
- "there",
- "this",
- "way",
- "would",
-}
-
-
-def _path_utility_tokens(value: Any) -> List[str]:
- return [
- token
- for token in _hybrid_symbolic_tokens(value)
- if token and token not in _PATH_UTILITY_STOPWORDS
- ]
-
-
-_PROFILE_QUERY_ALIAS_GROUPS: tuple[set[str], ...] = (
- {
- "accessory",
- "accessories",
- "bag",
- "camera",
- "cameras",
- "equipment",
- "flash",
- "gear",
- "lens",
- "lenses",
- "photo",
- "photography",
- "sony",
- "tripod",
- },
- {
- "app",
- "apps",
- "dashboard",
- "interface",
- "layout",
- "panel",
- "software",
- "tool",
- "tools",
- "ui",
- "workflow",
- },
- {
- "diet",
- "drink",
- "food",
- "meal",
- "restaurant",
- "snack",
- "taste",
- },
- {
- "background",
- "career",
- "job",
- "occupation",
- "position",
- "previous",
- "profession",
- "role",
- "worked",
- "work",
- },
-)
-
-
-_PROFILE_QUERY_GENERIC_TOKENS = set(_PATH_UTILITY_STOPWORDS) | {
- "able",
- "about",
- "any",
- "anything",
- "based",
- "best",
- "can",
- "complement",
- "could",
- "current",
- "give",
- "help",
- "information",
- "looking",
- "make",
- "need",
- "please",
- "recommend",
- "recommendation",
- "recommendations",
- "should",
- "some",
- "suggest",
- "suggestion",
- "suggestions",
- "tell",
- "that",
- "using",
- "you",
-}
-
-
-def _profile_query_expanded_tokens(value: Any) -> set[str]:
- tokens = set(_path_utility_tokens(value))
- expanded = set(tokens)
- for group in _PROFILE_QUERY_ALIAS_GROUPS:
- if tokens & group:
- expanded.update(group)
- return expanded
-
-
-def _profile_specific_tokens(tokens: Iterable[Any]) -> set[str]:
- return {
- _normalize(token)
- for token in tokens
- if _clean_text(token)
- and _normalize(token) not in _PROFILE_QUERY_GENERIC_TOKENS
- and not re.fullmatch(r"\d+", _normalize(token))
- }
-
-
-def _profile_hit_match_score(query_tokens: set[str], expanded_query_tokens: set[str], hit: MemoryHit) -> tuple[float, List[str], List[str]]:
- metadata = dict(hit.metadata or {})
- payload_parts: List[Any] = [
- hit.category,
- hit.relation,
- hit.slot_key,
- hit.value,
- *list(hit.anchors or []),
- metadata.get("profile_summary", ""),
- metadata.get("profile_value", ""),
- metadata.get("profile_type", ""),
- metadata.get("profile_domain", ""),
- metadata.get("profile_domain_label", ""),
- metadata.get("semantic_slot", ""),
- metadata.get("subject", ""),
- metadata.get("extracted_subject", ""),
- metadata.get("profile_cluster_domains", []),
- metadata.get("profile_cluster_types", []),
- metadata.get("profile_cluster_route_terms", []),
- metadata.get("profile_route_terms", []),
- metadata.get("profile_support_values", []),
- ]
- payload_text = " ".join(str(item) for item in payload_parts)
- record_raw_tokens = set(_path_utility_tokens(payload_text))
- record_tokens = _profile_query_expanded_tokens(payload_text)
- specific_query_tokens = _profile_specific_tokens(query_tokens)
- raw_overlap_tokens = sorted(specific_query_tokens & _profile_specific_tokens(record_raw_tokens))
- expanded_overlap_tokens = sorted(_profile_specific_tokens(expanded_query_tokens) & _profile_specific_tokens(record_tokens))
- overlap_tokens = raw_overlap_tokens or expanded_overlap_tokens
- overlap_ratio = float(len(overlap_tokens)) / float(max(1, len(specific_query_tokens)))
- profile_type = _normalize(metadata.get("profile_type", ""))
- source_kind = _normalize(hit.source_kind)
- source_bonus = 0.0
- if source_kind in {"public_dialog_preference", "public_dialog_goal", "public_dialog_profile"}:
- source_bonus += 0.08
- if bool(metadata.get("profile_candidate_status") == "consolidated"):
- source_bonus += 0.04
- if bool(metadata.get("profile_cluster_node")):
- source_bonus -= 0.16
- type_bonus = 0.0
- if profile_type in {"preference", "goal", "setup", "usage_context"} and (
- {"recommend", "suggest", "suited", "accessory", "accessories", "gear", "equipment", "setup", "current"} & query_tokens
- ):
- type_bonus += 0.10
- if profile_type in {"setup", "usage_context"} and {"current", "setup", "profile"} & query_tokens:
- type_bonus += 0.08
- raw_bonus = 0.18 * len(raw_overlap_tokens)
- expanded_bonus = 0.08 * max(0, len(expanded_overlap_tokens) - len(raw_overlap_tokens))
- score = raw_bonus + expanded_bonus + (0.58 * overlap_ratio) + source_bonus + type_bonus
- return round(score, 6), overlap_tokens, raw_overlap_tokens
-
-
-def _bounded_event_id_union(*groups: Iterable[Any], max_items: int) -> List[str]:
- return _dedupe((item for group in groups for item in group), max_items=max(1, int(max_items)))
-
-
-def _symbolic_recall_event_ids(
- query: str,
- runtime_graph: Mapping[str, Any],
- *,
- grouped_hits: Mapping[str, Sequence[MemoryHit]],
- limit: int,
-) -> List[str]:
- question_features = extract_question_features(query)
- query_tokens = set(_hybrid_symbolic_tokens(question_features.get("question_anchor_tokens", []) or query))
- if not query_tokens:
- return []
- nodes_by_id = {
- _clean_text(node.get("id", "")): dict(node)
- for node in list(runtime_graph.get("nodes", []) or [])
- if _clean_text(node.get("id", ""))
- }
- event_payloads: Dict[str, List[str]] = {}
- for node_id, node in nodes_by_id.items():
- if _clean_text(node.get("type", "")) == "event":
- metadata = dict(node.get("metadata", {}) or {})
- teacher_fields = dict(node.get("teacher_fields", {}) or {})
- event_payloads.setdefault(node_id, []).extend(
- [
- node.get("text", ""),
- node.get("speaker", ""),
- node.get("slot_key", ""),
- node.get("target_status", ""),
- node.get("profile_type", ""),
- node.get("profile_value", ""),
- *teacher_fields.values(),
- *metadata.values(),
- ]
- )
- for path in list(runtime_graph.get("paths", []) or []):
- event_id = _clean_text(path.get("event_id", ""))
- support_node = nodes_by_id.get(_clean_text(path.get("target", "")), {})
- if event_id and support_node:
- event_payloads.setdefault(event_id, []).append(support_node.get("text", ""))
- for event_id, group_hits in grouped_hits.items():
- payloads = event_payloads.setdefault(_clean_text(event_id), [])
- for hit in list(group_hits or []):
- metadata = dict(hit.metadata or {})
- payloads.extend([hit.value, hit.slot_key, hit.category, hit.relation, *hit.anchors, *metadata.values()])
-
- scored_events: List[tuple[str, float]] = []
- for event_id, payloads in event_payloads.items():
- event_tokens = set(_hybrid_symbolic_tokens(payloads))
- overlap = query_tokens & event_tokens
- if not overlap:
- continue
- overlap_ratio = float(len(overlap)) / float(max(1, len(query_tokens)))
- event_node = nodes_by_id.get(event_id, {})
- turn_index = int(event_node.get("turn_index", 0) or 0)
- scored_events.append((event_id, (len(overlap) * 4.0) + overlap_ratio + min(turn_index, 1000) * 0.000001))
- return [
- event_id
- for event_id, _ in sorted(scored_events, key=lambda item: (-float(item[1]), item[0]))
- ][: max(1, int(limit))]
-
-
-_EMBEDDER_INDEX_METADATA_TEXT_KEYS = (
- "raw_text",
- "source_turn_text",
- "source_span",
- "event_phrase",
- "event_summary",
- "profile_value",
- "profile_summary",
- "profile_type",
- "profile_domain",
- "profile_domain_label",
- "semantic_slot",
- "target_status",
- "subject",
- "subject_signature",
- "canonical_slot_key",
- "resource_key",
- "resolved_date",
- "resolved_time_value",
- "time_value",
- "time_display_value",
- "time_granularity",
- "speaker",
- "session_name",
- "topic_label",
- "topic_bucket_id",
- "origin_query",
- "writeback_class",
- "depth_layer",
- "memory_chain_depth_layer",
-)
-_EMBEDDER_INDEX_METADATA_LIST_KEYS = (
- "topic_keywords",
- "profile_route_terms",
- "profile_cluster_route_terms",
- "profile_support_values",
- "evidence_anchors",
- "support_memory_ids",
- "support_fact_refs",
- "support_path_refs",
-)
-
-
-def _embedder_index_enabled(mode: Any) -> bool:
- return _normalize(mode) not in _EMBEDDER_INDEX_DISABLED_MODES
-
-
-def _embedder_index_uses_bge_m3(mode: Any) -> bool:
- return _normalize(mode).replace("-", "_") in {item.replace("-", "_") for item in _EMBEDDER_INDEX_BGE_M3_MODES}
-
-
-def _embedder_index_version_for_mode(mode: Any) -> str:
- return _EMBEDDER_INDEX_BGE_M3_VERSION if _embedder_index_uses_bge_m3(mode) else _EMBEDDER_INDEX_VERSION
-
-
-def _embedder_index_text_items(value: Any, *, max_items: int = 64) -> List[str]:
- items: List[str] = []
-
- def visit(item: Any) -> None:
- if len(items) >= max_items:
- return
- if item is None:
- return
- if isinstance(item, Mapping):
- for key, nested in list(item.items()):
- if len(items) >= max_items:
- break
- key_text = _clean_text(key)
- if isinstance(nested, (str, int, float, bool)):
- value_text = _clean_text(nested)[:800]
- if value_text:
- items.append(f"{key_text} {value_text}".strip())
- else:
- visit(nested)
- return
- if isinstance(item, (list, tuple, set)):
- for nested in list(item):
- if len(items) >= max_items:
- break
- visit(nested)
- return
- text = _clean_text(item)[:800]
- if text:
- items.append(text)
-
- visit(value)
- return items[:max_items]
-
-
-def _embedder_index_term_weights(value: Any, *, max_terms: int = 96) -> Dict[str, float]:
- text = _clean_text(value)
- if not text:
- return {}
- counts: Dict[str, float] = {}
- for token in _path_utility_tokens(text):
- token = _normalize(token)
- if not token or len(token) > 64:
- continue
- counts[token] = counts.get(token, 0.0) + 1.0
- normalized_text = _normalize(text)
- cjk_chars = [char for char in normalized_text if "\u4e00" <= char <= "\u9fff"]
- for width, weight in ((2, 1.35), (3, 1.15)):
- if len(cjk_chars) < width:
- continue
- for index in range(0, len(cjk_chars) - width + 1):
- gram = "".join(cjk_chars[index : index + width])
- if gram:
- counts[gram] = counts.get(gram, 0.0) + weight
- if not counts:
- return {}
- ranked = sorted(counts.items(), key=lambda item: (-float(item[1]), item[0]))[: max(1, int(max_terms or 1))]
- norm = math.sqrt(sum(float(weight) * float(weight) for _, weight in ranked)) or 1.0
- return {term: round(float(weight) / norm, 6) for term, weight in ranked}
-
-
-def _embedder_dense_vectors_for_texts(texts: Sequence[str], *, mode: str) -> tuple[List[List[float]], Dict[str, Any]]:
- normalized_mode = _normalize(mode)
- if not _embedder_index_uses_bge_m3(normalized_mode):
- return [[] for _ in texts], {"write_embedder_dense_enabled": False}
- clean_texts = [_clean_text(text) for text in texts]
- metadata: Dict[str, Any] = {
- "write_embedder_dense_enabled": False,
- "write_embedder_dense_backend": "bge_m3_transformers",
- "write_embedder_dense_model": _clean_text(os.getenv("TMCRA_EMBEDDER_MODEL_PATH", "")) or "BAAI/bge-m3",
- }
- if not any(clean_texts):
- metadata["write_embedder_dense_error"] = "empty_texts"
- return [[] for _ in texts], metadata
- try:
- import torch # type: ignore
- from transformers import AutoModel, AutoTokenizer # type: ignore
- except Exception as exc:
- metadata["write_embedder_dense_error"] = f"dependency_unavailable:{exc.__class__.__name__}"
- return [[] for _ in texts], metadata
- model_name = metadata["write_embedder_dense_model"]
- device = _clean_text(os.getenv("TMCRA_EMBEDDER_DEVICE", ""))
- if not device:
- device = "cuda" if bool(getattr(torch, "cuda", None) and torch.cuda.is_available()) else "cpu"
- try:
- max_length = max(64, int(os.getenv("TMCRA_EMBEDDER_MODEL_MAX_LENGTH", "512") or 512))
- except (TypeError, ValueError):
- max_length = 512
- cache_key = f"bge_m3::{model_name}::{device}::{max_length}"
- try:
- cached = _EMBEDDER_MODEL_CACHE.get(cache_key)
- if cached is None:
- tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
- model = AutoModel.from_pretrained(model_name, trust_remote_code=True)
- model.to(device)
- model.eval()
- cached = (tokenizer, model)
- _EMBEDDER_MODEL_CACHE[cache_key] = cached
- tokenizer, model = cached
- encoded = tokenizer(
- clean_texts,
- padding=True,
- truncation=True,
- max_length=max_length,
- return_tensors="pt",
- )
- encoded = {key: value.to(device) for key, value in encoded.items()}
- with torch.no_grad():
- output = model(**encoded)
- hidden = output.last_hidden_state
- mask = encoded.get("attention_mask")
- if mask is not None:
- mask = mask.unsqueeze(-1).expand(hidden.size()).float()
- pooled = torch.sum(hidden * mask, dim=1) / torch.clamp(mask.sum(dim=1), min=1e-9)
- else:
- pooled = hidden[:, 0]
- pooled = torch.nn.functional.normalize(pooled, p=2, dim=1)
- vectors = [
- [round(float(value), 6) for value in row.detach().cpu().tolist()]
- for row in pooled
- ]
- metadata.update(
- {
- "write_embedder_dense_enabled": True,
- "write_embedder_dense_device": device,
- "write_embedder_dense_dim": int(len(vectors[0]) if vectors else 0),
- "write_embedder_dense_max_length": int(max_length),
- }
- )
- return vectors, metadata
- except Exception as exc:
- metadata["write_embedder_dense_error"] = f"{exc.__class__.__name__}:{_clean_text(exc)[:240]}"
- return [[] for _ in texts], metadata
-
-
-def _prewarm_embedder_dense_if_requested(*, mode: str) -> Dict[str, Any]:
- flag = _normalize(os.getenv("TMCRA_EMBEDDER_PREWARM", ""))
- if flag in _EMBEDDER_INDEX_DISABLED_MODES or flag not in {"1", "true", "yes", "on", "auto"}:
- return {"embedder_prewarm_enabled": False}
- normalized_mode = _normalize(mode)
- if not _embedder_index_uses_bge_m3(normalized_mode):
- return {
- "embedder_prewarm_enabled": False,
- "embedder_prewarm_reason": "mode_not_dense",
- "embedder_prewarm_mode": normalized_mode or "off",
- }
- warmup_text = _clean_text(os.getenv("TMCRA_EMBEDDER_PREWARM_TEXT", "")) or "tmcra memory retrieval warmup"
- vectors, metadata = _embedder_dense_vectors_for_texts([warmup_text], mode=normalized_mode)
- return {
- "embedder_prewarm_enabled": bool(vectors and vectors[0]),
- "embedder_prewarm_mode": normalized_mode,
- "embedder_prewarm_dense_enabled": bool(metadata.get("write_embedder_dense_enabled")),
- "embedder_prewarm_dense_device": metadata.get("write_embedder_dense_device", ""),
- "embedder_prewarm_dense_error": metadata.get("write_embedder_dense_error", ""),
- }
-
-
-def _embedder_index_record_text(record: SessionMemoryRecordV2, *, turn_text: str = "") -> str:
- metadata = dict(record.metadata or {})
- parts: List[str] = [
- record.category,
- record.slot_key,
- record.value,
- record.value,
- record.value,
- record.relation,
- *list(record.anchor_concepts or []),
- *list(record.anchor_concepts or []),
- *list(record.evidence_anchors or []),
- ]
- has_record_evidence_text = any(
- _clean_text(metadata.get(key, ""))
- for key in ("raw_text", "source_turn_text", "source_span", "event_phrase", "profile_value", "profile_summary")
- )
- if turn_text and not has_record_evidence_text:
- parts.append(_clean_text(turn_text)[:1000])
- for key in _EMBEDDER_INDEX_METADATA_TEXT_KEYS:
- value = metadata.get(key)
- if value:
- parts.append(f"{key} {' '.join(_embedder_index_text_items(value, max_items=12))}".strip())
- for key in _EMBEDDER_INDEX_METADATA_LIST_KEYS:
- value = metadata.get(key)
- if value:
- parts.append(f"{key} {' '.join(_embedder_index_text_items(value, max_items=24))}".strip())
- return _clean_text(" ".join(_clean_text(part) for part in parts if _clean_text(part)))
-
-
-def _apply_write_embedder_index_to_graph(
- graph: SessionMemoryGraphV2,
- *,
- stored_ids: Sequence[str],
- turn_text: str,
- turn_index: int,
- mode: str,
- max_terms: int,
-) -> Dict[str, Any]:
- normalized_mode = _normalize(mode)
- index_version = _embedder_index_version_for_mode(normalized_mode)
- metadata: Dict[str, Any] = {
- "write_embedder_index_enabled": False,
- "write_embedder_index_mode": normalized_mode or "off",
- "write_embedder_index_version": index_version,
- "write_embedder_index_record_count": 0,
- }
- if not _embedder_index_enabled(normalized_mode) or not stored_ids:
- return metadata
- indexed_ids: List[str] = []
- index_rows: List[tuple[SessionMemoryRecordV2, str, Dict[str, float]]] = []
- for memory_id in _dedupe(stored_ids):
- record = getattr(graph, "records_by_id", {}).get(memory_id)
- if record is None:
- continue
- index_text = _embedder_index_record_text(record, turn_text=turn_text)
- terms = _embedder_index_term_weights(
- index_text,
- max_terms=max_terms,
- )
- if not terms:
- continue
- index_rows.append((record, index_text, terms))
- dense_vectors, dense_metadata = _embedder_dense_vectors_for_texts(
- [index_text for _, index_text, _ in index_rows],
- mode=normalized_mode,
- )
- metadata.update(dense_metadata)
- for row_index, (record, _, terms) in enumerate(index_rows):
- dense_vector = dense_vectors[row_index] if row_index < len(dense_vectors) else []
- record_metadata = dict(record.metadata or {})
- record_metadata.update(
- {
- "write_embedder_index_enabled": True,
- "write_embedder_index_mode": normalized_mode,
- "write_embedder_index_version": index_version,
- "write_embedder_index_source_turn": int(turn_index),
- "write_embedder_index_term_count": int(len(terms)),
- "write_embedder_index_terms": dict(terms),
- "write_embedder_index_top_terms": list(terms.keys())[:24],
- }
- )
- if dense_vector:
- record_metadata.update(
- {
- "write_embedder_dense_enabled": True,
- "write_embedder_dense_backend": dense_metadata.get("write_embedder_dense_backend", ""),
- "write_embedder_dense_model": dense_metadata.get("write_embedder_dense_model", ""),
- "write_embedder_dense_dim": int(len(dense_vector)),
- "write_embedder_dense_vector": list(dense_vector),
- }
- )
- elif _embedder_index_uses_bge_m3(normalized_mode):
- record_metadata.update(
- {
- "write_embedder_dense_enabled": False,
- "write_embedder_dense_error": dense_metadata.get("write_embedder_dense_error", "dense_vector_unavailable"),
- }
- )
- record.metadata = record_metadata
- indexed_ids.append(record.memory_id)
- metadata.update(
- {
- "write_embedder_index_enabled": bool(indexed_ids),
- "write_embedder_index_record_count": int(len(indexed_ids)),
- "write_embedder_index_record_ids": list(indexed_ids[:24]),
- }
- )
- return metadata
-
-
-def _embedder_index_recall_event_ids(
- query: str,
- *,
- grouped_hits: Mapping[str, Sequence[MemoryHit]],
- mode: str,
- limit: int,
- max_terms: int,
-) -> Dict[str, Any]:
- normalized_mode = _normalize(mode)
- index_version = _embedder_index_version_for_mode(normalized_mode)
- metadata: Dict[str, Any] = {
- "embedder_index_recall_enabled": False,
- "embedder_index_recall_mode": normalized_mode or "off",
- "embedder_index_recall_version": index_version,
- "embedder_index_event_ids": [],
- "embedder_index_event_scores": {},
- "embedder_index_record_count": 0,
- }
- if not _embedder_index_enabled(normalized_mode):
- return {"event_ids": [], "metadata": metadata}
- query_terms = _embedder_index_term_weights(query, max_terms=max_terms)
- query_vectors, query_dense_metadata = _embedder_dense_vectors_for_texts([query], mode=normalized_mode)
- query_vector = query_vectors[0] if query_vectors else []
- metadata.update(
- {
- "embedder_dense_recall_enabled": bool(query_vector),
- "embedder_dense_recall_backend": query_dense_metadata.get("write_embedder_dense_backend", ""),
- "embedder_dense_recall_model": query_dense_metadata.get("write_embedder_dense_model", ""),
- "embedder_dense_recall_error": query_dense_metadata.get("write_embedder_dense_error", ""),
- }
- )
- if not query_terms and not query_vector:
- metadata["embedder_index_recall_reason"] = "empty_query_terms"
- return {"event_ids": [], "metadata": metadata}
- scored_events: List[tuple[str, float, int]] = []
- indexed_record_count = 0
- dense_record_count = 0
- for event_id, group_hits in grouped_hits.items():
- event_score = 0.0
- event_turn = 0
- for hit in list(group_hits or []):
- hit_metadata = dict(hit.metadata or {})
- raw_terms = hit_metadata.get("write_embedder_index_terms")
- raw_vector = hit_metadata.get("write_embedder_dense_vector")
- if not isinstance(raw_terms, Mapping) and not isinstance(raw_vector, list):
- continue
- indexed_record_count += int(isinstance(raw_terms, Mapping))
- dense_record_count += int(isinstance(raw_vector, list) and bool(raw_vector))
- sparse_score = 0.0
- if isinstance(raw_terms, Mapping):
- for term, query_weight in query_terms.items():
- try:
- sparse_score += float(query_weight) * float(raw_terms.get(term, 0.0) or 0.0)
- except (TypeError, ValueError):
- continue
- dense_score = 0.0
- if query_vector and isinstance(raw_vector, list) and raw_vector:
- for query_value, record_value in zip(query_vector, raw_vector):
- try:
- dense_score += float(query_value) * float(record_value)
- except (TypeError, ValueError):
- continue
- score = max(float(sparse_score), float(dense_score) + (0.15 * float(sparse_score) if dense_score > 0.0 else 0.0))
- if score <= 0.0:
- continue
- event_score = max(event_score, float(score))
- event_turn = max(event_turn, int(hit.turn_index or 0))
- if event_score > 0.0:
- scored_events.append((_clean_text(event_id), round(event_score, 6), event_turn))
- scored_events.sort(key=lambda item: (-float(item[1]), -int(item[2]), item[0]))
- selected = scored_events[: max(1, int(limit or 1))]
- event_ids = [event_id for event_id, _, _ in selected if event_id]
- event_scores = {event_id: score for event_id, score, _ in selected if event_id}
- metadata.update(
- {
- "embedder_index_recall_enabled": True,
- "embedder_index_event_ids": list(event_ids),
- "embedder_index_event_scores": dict(event_scores),
- "embedder_index_query_terms": list(query_terms.keys())[:32],
- "embedder_index_record_count": int(indexed_record_count),
- "embedder_dense_record_count": int(dense_record_count),
- }
- )
- return {"event_ids": event_ids, "metadata": metadata}
-
-
-_IDENTIFIER_GENERIC_TOKENS = {
- "api",
- "agent",
- "code",
- "codename",
- "context",
- "debug",
- "goal",
- "memory",
- "model",
- "name",
- "project",
- "retrieval",
- "runtime",
- "session",
- "target",
- "test",
- "turn",
-}
-_IDENTIFIER_REQUEST_RE = re.compile(
- r"\b(code\s*name|codename|identifier|alias|project\s+name|model\s+name|api\s+name)\b|"
- r"(代号|编号|标识符|名称|名字|别名|项目名|模型名|接口名)",
- flags=re.IGNORECASE,
-)
-_IDENTIFIER_TOKEN_RE = re.compile(r"\b[A-Za-z][A-Za-z0-9_-]{2,63}\b")
-
-
-def _query_identifier_tokens(query: Any) -> List[str]:
- text = _clean_text(query)
- tokens: List[str] = []
- for match in _IDENTIFIER_TOKEN_RE.finditer(text):
- token = match.group(0)
- lowered = token.lower()
- if lowered in _IDENTIFIER_GENERIC_TOKENS:
- continue
- has_inner_upper = any(ch.isupper() for ch in token[1:])
- has_digit = any(ch.isdigit() for ch in token)
- has_joiner = "-" in token or "_" in token
- if has_inner_upper or has_digit or has_joiner:
- tokens.append(token)
- return _dedupe(tokens, max_items=8)
-
-
-def _query_requests_identifier_fact(query: Any) -> bool:
- return bool(_IDENTIFIER_REQUEST_RE.search(_clean_text(query)))
-
-
-def _hit_text_for_identifier_match(hit: MemoryHit) -> str:
- metadata = dict(hit.metadata or {})
- values = [
- hit.memory_id,
- hit.category,
- hit.value,
- hit.relation,
- hit.slot_key,
- hit.source_kind,
- *list(hit.anchors or []),
- metadata.get("raw_text", ""),
- metadata.get("source_turn_text", ""),
- metadata.get("source_span", ""),
- metadata.get("event_phrase", ""),
- metadata.get("profile_value", ""),
- metadata.get("target_status", ""),
- ]
- return " ".join(_clean_text(value) for value in values if _clean_text(value))
-
-
-def _copy_hit_with_identifier_boost(hit: MemoryHit, *, score: float, reasons: Sequence[str], matched_tokens: Sequence[str]) -> MemoryHit:
- metadata = dict(hit.metadata or {})
- metadata.update(
- {
- "identifier_protected": True,
- "identifier_protected_score": round(float(score), 6),
- "identifier_protected_reasons": list(reasons)[:8],
- "identifier_protected_matched_tokens": list(matched_tokens)[:8],
- "identifier_protected_original_score": round(float(hit.score), 6),
- }
- )
- return MemoryHit(
- memory_id=hit.memory_id,
- category=hit.category,
- value=hit.value,
- relation=hit.relation,
- anchors=list(hit.anchors),
- score=max(float(hit.score), round(1.0 + float(score), 6)),
- source_kind=hit.source_kind,
- slot_key=hit.slot_key,
- state=hit.state,
- turn_index=int(hit.turn_index),
- metadata=metadata,
- )
-
-
-def _identifier_protected_hits(
- *,
- query: str,
- final_hits: Sequence[MemoryHit],
- candidate_hits: Sequence[MemoryHit],
- top_k: int,
-) -> Dict[str, Any]:
- identifier_tokens = _query_identifier_tokens(query)
- identifier_request = _query_requests_identifier_fact(query)
- if not identifier_tokens and not identifier_request:
- return {"enabled": False, "hits": list(final_hits), "promoted_hits": [], "metadata": {"identifier_protected_enabled": False}}
-
- query_text = _clean_text(query)
- pool: Dict[str, MemoryHit] = {}
- for hit in list(final_hits) + list(candidate_hits):
- key = hit.memory_id or f"{_hit_event_id(hit)}::{hit.slot_key}::{hit.value[:80]}"
- if key and key not in pool:
- pool[key] = hit
-
- scored: List[tuple[float, MemoryHit, List[str], List[str]]] = []
- for hit in pool.values():
- hit_text = _hit_text_for_identifier_match(hit)
- hit_lower = hit_text.lower()
- matched_tokens = [token for token in identifier_tokens if token.lower() in hit_lower]
- reasons: List[str] = []
- score = 0.0
- if matched_tokens:
- score += 12.0 + float(len(matched_tokens))
- reasons.append("exact_identifier_match")
- if identifier_request:
- if any(term in hit_lower for term in ("codename", "code name", "identifier", "alias", "project_codename")):
- score += 8.0
- reasons.append("identifier_field_match")
- if any(term in hit_text for term in ("代号", "编号", "标识符", "名称", "名字", "别名")):
- score += 8.0
- reasons.append("identifier_cjk_field_match")
- if "项目" in query_text and "项目" in hit_text:
- score += 1.5
- reasons.append("project_term_match")
- if score <= 0:
- continue
- scored.append((score, hit, matched_tokens, reasons))
-
- scored.sort(key=lambda item: (item[0], float(item[1].score), int(item[1].turn_index or 0)), reverse=True)
- promoted = [
- _copy_hit_with_identifier_boost(hit, score=score, reasons=reasons, matched_tokens=matched_tokens)
- for score, hit, matched_tokens, reasons in scored[:2]
- ]
- if not promoted:
- return {
- "enabled": True,
- "hits": list(final_hits),
- "promoted_hits": [],
- "metadata": {
- "identifier_protected_enabled": True,
- "identifier_query_tokens": identifier_tokens,
- "identifier_request": bool(identifier_request),
- "identifier_promoted_count": 0,
- },
- }
-
- promoted_keys = {hit.memory_id or f"{_hit_event_id(hit)}::{hit.slot_key}::{hit.value[:80]}" for hit in promoted}
- merged = list(promoted)
- for hit in final_hits:
- key = hit.memory_id or f"{_hit_event_id(hit)}::{hit.slot_key}::{hit.value[:80]}"
- if key in promoted_keys:
- continue
- merged.append(hit)
- if len(merged) >= max(1, int(top_k)):
- break
- return {
- "enabled": True,
- "hits": merged[: max(1, int(top_k))],
- "promoted_hits": promoted,
- "metadata": {
- "identifier_protected_enabled": True,
- "identifier_query_tokens": identifier_tokens,
- "identifier_request": bool(identifier_request),
- "identifier_promoted_count": len(promoted),
- "identifier_promoted_ids": [hit.memory_id for hit in promoted],
- },
- }
-
-
-def _trim_prompt_text(value: Any, *, max_chars: int = 220) -> str:
- text = _clean_text(value)
- if len(text) <= max_chars:
- return text
- return f"{text[: max(0, max_chars - 3)].rstrip()}..."
-
-
-def _prompt_hit_payload(hit: MemoryHit) -> Dict[str, Any]:
- return {
- "memory_id": hit.memory_id,
- "slot_key": hit.slot_key,
- "category": hit.category,
- "value": _trim_prompt_text(hit.value),
- "relation": hit.relation,
- "anchors": [_trim_prompt_text(anchor, max_chars=80) for anchor in list(hit.anchors)[:4]],
- "score": round(float(hit.score), 6),
- "state": hit.state,
- "turn_index": int(hit.turn_index),
- "source_kind": hit.source_kind,
- }
-
-
-def _prompt_record_payload(record: SessionMemoryRecordV2) -> Dict[str, Any]:
- metadata = dict(record.metadata or {})
- return {
- "slot_key": record.slot_key,
- "category": record.category,
- "value": _trim_prompt_text(record.value),
- "state": record.state,
- "turn_index": int(record.turn_index),
- "anchors": [_trim_prompt_text(anchor, max_chars=80) for anchor in list(record.anchor_concepts)[:4]],
- "memory_role": _clean_text(metadata.get("memory_role", "")),
- "authority": _clean_text(metadata.get("authority", "")),
- }
-
-
-def _graph_prompt_state_summary(graph: SessionMemoryGraphV2, retrieval: MemoryRetrieval) -> Dict[str, Any]:
- active_slots: List[Dict[str, Any]] = []
- for slot_key, record_id in list(graph.slot_heads.items())[:_GRAPH_PROMPT_MAX_ACTIVE_SLOTS]:
- record = graph.records_by_id.get(record_id)
- if record is None:
- continue
- active_slots.append(_prompt_record_payload(record))
- top_hits = [_prompt_hit_payload(hit) for hit in list(retrieval.hits)[:_GRAPH_PROMPT_MAX_HITS]]
- relation_preview = [
- {
- "from": _trim_prompt_text(item.get("from", ""), max_chars=72),
- "to": _trim_prompt_text(item.get("to", ""), max_chars=72),
- "relation": _clean_text(item.get("relation", "")),
- }
- for item in list(retrieval.relations)[:_GRAPH_PROMPT_MAX_RELATIONS]
- ]
- summary = {
- "records": len(graph.records_by_id),
- "active_slots": len(graph.slot_heads),
- "turn_index": int(graph.turn_index),
- "noise_turn_count": int(graph.noise_turn_count),
- "answer_support_events": len(graph.answer_support_log),
- "top_hits": top_hits,
- "active_slot_records": active_slots,
- "relation_preview": relation_preview,
- "context_truncated": False,
- "truncation_reason": "",
- }
- truncated = False
- truncation_reason = ""
- while len(json.dumps(summary, ensure_ascii=False)) > _GRAPH_PROMPT_MAX_CHARS:
- if len(summary["top_hits"]) > 1:
- summary["top_hits"] = summary["top_hits"][:-1]
- truncated = True
- truncation_reason = "trimmed_top_hits"
- continue
- if len(summary["active_slot_records"]) > 1:
- summary["active_slot_records"] = summary["active_slot_records"][:-1]
- truncated = True
- truncation_reason = "trimmed_active_slots"
- continue
- if len(summary["relation_preview"]) > 2:
- summary["relation_preview"] = summary["relation_preview"][:-1]
- truncated = True
- truncation_reason = "trimmed_relations"
- continue
- break
- summary["context_truncated"] = truncated
- summary["truncation_reason"] = truncation_reason
- return summary
-
-
-def _state_stats(*, storage_bytes: int, retrieval_context_tokens: int, total_state_tokens: int, **extra: Any) -> Dict[str, Any]:
- return {
- **extra,
- "storage_bytes": int(storage_bytes),
- "context_token_estimate": int(retrieval_context_tokens),
- "retrieval_context_token_estimate": int(retrieval_context_tokens),
- "total_state_token_estimate": int(total_state_tokens),
- }
-
-
-def _relation_hit(hit: MemoryHit, *, weight_bias: float = 0.0) -> Dict[str, Any]:
- if not hit.anchors:
- return {}
- anchor = hit.anchors[0]
- if not anchor or anchor == hit.value:
- return {}
- return {
- "from": anchor,
- "to": hit.value,
- "relation": hit.relation,
- "weight": round(max(0.25, min(0.98, 0.42 + hit.score * 0.4 + weight_bias)), 6),
- "source_kind": hit.source_kind,
- "memory_id": hit.memory_id,
- }
-
-
-def _raw_hit_to_memory_hit(payload: Dict[str, Any]) -> MemoryHit:
- metadata = dict(payload.get("metadata", {}) or {})
- if payload.get("supersedes"):
- metadata["supersedes"] = list(payload.get("supersedes", []) or [])
- slot_key = stable_slot_key(
- category=str(payload.get("category", "")),
- value=str(payload.get("value", "")),
- anchors=[str(anchor) for anchor in payload.get("anchor_concepts", payload.get("anchors", [])) or [] if _clean_text(anchor)],
- slot_key=str(payload.get("slot_key", metadata.get("slot", ""))),
- relation=str(payload.get("relation", "related_to")),
- metadata=metadata,
- )
- return MemoryHit(
- memory_id=str(payload.get("memory_id", "")),
- category=str(payload.get("category", "")),
- value=str(payload.get("value", "")),
- relation=str(payload.get("relation", "related_to")),
- anchors=[str(anchor) for anchor in payload.get("anchor_concepts", payload.get("anchors", [])) or [] if _clean_text(anchor)],
- score=float(payload.get("score", payload.get("relevance", 0.0)) or 0.0),
- source_kind=str(payload.get("source_kind", "memory")),
- slot_key=slot_key,
- state=str(payload.get("state", payload.get("metadata", {}).get("state", "active")) or "active"),
- turn_index=int(payload.get("turn_index", 0) or 0),
- metadata=metadata,
- )
-
-
-def _restore_hit_scores(hits: List[MemoryHit], scored_lookup: Dict[str, MemoryHit]) -> List[MemoryHit]:
- restored: List[MemoryHit] = []
- for hit in hits:
- scored = scored_lookup.get(hit.memory_id)
- if scored:
- hit.score = max(float(hit.score), float(scored.score))
- if not hit.anchors and scored.anchors:
- hit.anchors = list(scored.anchors)
- restored.append(hit)
- return restored
-
-
-def _current_subject_query(query: str) -> bool:
- lowered = _normalize(query)
- return bool(
- _public_query_subject(query)
- and any(marker in lowered for marker in ("right now", "current", "currently", "active", "now", "当前", "现在"))
- )
-
-
-def _record_subject_signatures(record: SessionMemoryRecordV2) -> set[str]:
- metadata = dict(record.metadata or {})
- signatures = {
- _normalize(metadata.get("subject_signature", "")).replace("-", "_"),
- _public_subject_signature(metadata.get("subject", "")),
- }
- canonical_slot_key = _clean_text(metadata.get("canonical_slot_key", "") or record.slot_key)
- if ".subject." in canonical_slot_key:
- signatures.add(_public_subject_signature(canonical_slot_key.split(".subject.", 1)[-1]))
- if ".subject." in record.slot_key:
- signatures.add(_public_subject_signature(record.slot_key.split(".subject.", 1)[-1]))
- signatures.discard("")
- return signatures
-
-
-def _current_subject_protected_hits(
- *,
- query: str,
- graph: SessionMemoryGraphV2,
- final_hits: Sequence[MemoryHit],
- top_k: int,
-) -> Dict[str, Any]:
- if not _current_subject_query(query):
- return {
- "hits": list(final_hits),
- "metadata": {"current_subject_resolver_enabled": False},
- }
- subject = _public_query_subject(query)
- subject_signature = _public_subject_signature(subject)
- if not subject_signature:
- return {
- "hits": list(final_hits),
- "metadata": {
- "current_subject_resolver_enabled": True,
- "current_subject_resolver_reason": "no_subject_signature",
- },
- }
- def _candidate_record(record: SessionMemoryRecordV2 | None) -> bool:
- return bool(
- record is not None
- and _clean_text(record.source_kind).startswith("public_dialog")
- and _normalize(record.category) != "question"
- and subject_signature in _record_subject_signatures(record)
- )
-
- slot_head_candidates = [
- record
- for slot_key, memory_id in graph.slot_heads.items()
- for record in [graph.records_by_id.get(memory_id)]
- if _candidate_record(record)
- and subject_signature in _record_subject_signatures(record)
- ]
- candidates = slot_head_candidates or [
- record
- for record in graph.records_by_id.values()
- if record.state == "active" and _candidate_record(record)
- ]
- candidates.sort(
- key=lambda record: (
- int(
- _normalize((record.metadata or {}).get("target_status", "")) == "current"
- or _normalize(record.relation) == "current_subject_value"
- ),
- int(record.turn_index),
- float(record.confidence),
- float(record.salience),
- ),
- reverse=True,
- )
- promoted: List[MemoryHit] = []
- for index, record in enumerate(candidates[: max(1, min(2, int(top_k or 1)))], start=1):
- metadata = dict(record.metadata or {})
- metadata.update(
- {
- "current_subject_resolver": True,
- "current_subject_resolver_rank": index,
- "current_subject_query_subject": subject,
- "current_subject_query_signature": subject_signature,
- "public_subject_match": True,
- "public_subject_overlap": 1.0,
- }
- )
- promoted.append(
- MemoryHit(
- memory_id=record.memory_id,
- category=record.category,
- value=record.value,
- relation=record.relation,
- anchors=list(record.anchor_concepts),
- score=max(float(record.confidence), float(record.salience), 1.75),
- source_kind=record.source_kind,
- slot_key=record.slot_key,
- state=record.state,
- turn_index=int(record.turn_index),
- metadata=metadata,
- )
- )
- if not promoted:
- return {
- "hits": list(final_hits),
- "metadata": {
- "current_subject_resolver_enabled": True,
- "current_subject_resolver_reason": "no_active_subject_head",
- "current_subject_query_subject": subject,
- "current_subject_query_signature": subject_signature,
- },
- }
- promoted_ids = {hit.memory_id for hit in promoted}
- merged_tail: List[MemoryHit] = []
- for hit in final_hits:
- if hit.memory_id in promoted_ids:
- continue
- metadata = dict(hit.metadata or {})
- hit_state = _normalize(hit.state)
- same_subject = subject_signature in {
- _normalize(metadata.get("subject_signature", "")).replace("-", "_"),
- _public_subject_signature(metadata.get("subject", "")),
- _public_subject_signature(hit.slot_key.split(".subject.", 1)[-1]) if ".subject." in hit.slot_key else "",
- _public_subject_signature(_clean_text(metadata.get("canonical_slot_key", "")).split(".subject.", 1)[-1])
- if ".subject." in _clean_text(metadata.get("canonical_slot_key", ""))
- else "",
- }
- if same_subject and hit_state in {"superseded", "evidence", "historical", "stale", "false"}:
- continue
- merged_tail.append(hit)
- merged = [*promoted, *merged_tail]
- limit = max(int(top_k or 1), len(promoted))
- return {
- "hits": merged[:limit],
- "metadata": {
- "current_subject_resolver_enabled": True,
- "current_subject_resolver_reason": "promoted_active_subject_head",
- "current_subject_query_subject": subject,
- "current_subject_query_signature": subject_signature,
- "current_subject_promoted_memory_ids": [hit.memory_id for hit in promoted],
- },
- }
-
-
-def _depth_chain_protected_hits(
- *,
- query: str,
- graph: SessionMemoryGraphV2,
- final_hits: Sequence[MemoryHit],
- top_k: int,
-) -> Dict[str, Any]:
- seed_memory_ids = [hit.memory_id for hit in final_hits if hit.memory_id]
- chain = graph.depth_chain_for_query(
- query,
- seed_memory_ids=seed_memory_ids,
- top_k=max(3, min(8, int(top_k or 1))),
- )
- if not chain.get("enabled") or not chain.get("nodes"):
- return {
- "hits": list(final_hits),
- "metadata": {
- "memory_chain_enabled": bool(chain.get("enabled", False)),
- "memory_chain_reason": _clean_text(chain.get("reason", "")),
- "memory_chain_node_count": 0,
- "memory_chain_edge_count": 0,
- "memory_chain": chain,
- },
- }
- seen = {hit.memory_id for hit in final_hits if hit.memory_id}
- chain_hits: List[MemoryHit] = []
- for rank, node in enumerate(list(chain.get("nodes", []) or []), start=1):
- if not isinstance(node, Mapping):
- continue
- memory_id = _clean_text(node.get("memory_id", ""))
- if not memory_id or memory_id in seen:
- continue
- payload = dict(node)
- metadata = dict(payload.get("metadata", {}) or {})
- metadata.update(
- {
- "memory_chain_protected": True,
- "memory_chain_rank": int(rank),
- "memory_chain_subject_signature": _clean_text(chain.get("subject_signature", "")),
- "memory_chain_depth_layer": _clean_text(metadata.get("depth_layer", "")) or "core_view",
- }
- )
- payload["metadata"] = metadata
- payload["score"] = max(float(payload.get("score", 0.0) or 0.0), 0.62 - (rank * 0.01))
- chain_hits.append(_raw_hit_to_memory_hit(payload))
- seen.add(memory_id)
- limit = max(int(top_k or 1), min(12, int(top_k or 1) + max(0, len(chain_hits))))
- merged = [*list(final_hits), *chain_hits]
- return {
- "hits": merged[:limit],
- "metadata": {
- "memory_chain_enabled": True,
- "memory_chain_reason": _clean_text(chain.get("reason", "")),
- "memory_chain_subject_signature": _clean_text(chain.get("subject_signature", "")),
- "memory_chain_node_count": int(chain.get("node_count", 0) or 0),
- "memory_chain_edge_count": int(chain.get("edge_count", 0) or 0),
- "memory_chain_depth_layers": list(chain.get("depth_layers", []) or []),
- "memory_chain": chain,
- },
- }
-
-
-def _is_public_dialog_hit(hit: MemoryHit) -> bool:
- return _clean_text(hit.source_kind).startswith("public_dialog")
-
-
-def _normalized_runtime_signature(prefix: str, value: str) -> str:
- normalized = _normalize(value)
- if not normalized:
- return ""
- return f"{prefix}{normalized.replace('|', '_').replace(':', '_')}"
-
-
-def _runtime_event_key(hit: MemoryHit) -> str:
- metadata = dict(hit.metadata or {})
- explicit = _clean_text(metadata.get("event_id", ""))
- if explicit:
- return explicit
- dia_id = _clean_text(metadata.get("dia_id", ""))
- if dia_id:
- return f"event::{dia_id}"
- if not _is_public_dialog_hit(hit):
- state_signature = _clean_text(metadata.get("state_signature", ""))
- if state_signature:
- return _normalized_runtime_signature("event::state::", state_signature)
- memory_signature = _clean_text(metadata.get("memory_signature", ""))
- if memory_signature:
- return _normalized_runtime_signature("event::memory::", memory_signature)
- slot_root = _public_slot_root(_clean_text(hit.slot_key))
- if slot_root:
- return slot_root
- return _clean_text(hit.memory_id)
-
-
-def _runtime_event_turn_index_from_id(event_id: str) -> int:
- text = _clean_text(event_id)
- if not text:
- return 0
- match = re.search(r"(?::|_)(\d+)$", text)
- if match:
- return int(match.group(1))
- matches = re.findall(r"\d+", text)
- return int(matches[-1]) if matches else 0
-
-
-def _representative_event_hit(group_hits: Sequence[MemoryHit], *, query: str = "") -> MemoryHit | None:
- semantic_source_kinds = {
- "public_dialog_fact",
- "public_dialog_preference",
- "public_dialog_goal",
- "public_dialog_constraint",
- "public_dialog_status",
- "public_dialog_profile",
- "replacement_memory",
- "session_memory",
- }
- query_tokens = set(_path_utility_tokens(query))
-
- def rank(hit: MemoryHit) -> tuple[bool, float, bool, bool, bool, bool, float]:
- metadata = dict(hit.metadata or {})
- source_kind = _clean_text(hit.source_kind)
- text_parts: List[Any] = [hit.value, hit.slot_key, hit.category]
- if source_kind != "public_dialog_turn":
- text_parts.extend([metadata.get("source_turn_text", ""), metadata.get("raw_text", "")])
- text = " ".join(
- _clean_text(item)
- for item in text_parts
- if _clean_text(item)
- )
- hit_tokens = set(_path_utility_tokens(text))
- overlap = len(query_tokens & hit_tokens) if query_tokens else 0
- has_number = bool(re.search(r"\b\d+\b", text))
- is_semantic = source_kind in semantic_source_kinds or (
- source_kind != "public_dialog_turn" and bool(_clean_text(metadata.get("memory_writer_role", "")))
- )
- direct_semantic_answer = bool(is_semantic and has_number and overlap >= 2)
- query_score = float(overlap) + (0.75 if has_number and overlap else 0.0)
- return (
- direct_semantic_answer,
- query_score,
- is_semantic,
- source_kind == "public_dialog_event",
- source_kind == "public_dialog_turn",
- source_kind in {"replacement_memory", "session_memory"},
- float(hit.score),
- )
-
- ordered = sorted(
- list(group_hits),
- key=rank,
- reverse=True,
- )
- return ordered[0] if ordered else None
-
-
-def _event_record_hits_from_graph(graph: SessionMemoryGraphV2, event_id: str) -> List[MemoryHit]:
- normalized_event_id = _clean_text(event_id)
- if not normalized_event_id:
- return []
- hits: List[MemoryHit] = []
- for record in graph.records_by_id.values():
- metadata = dict(record.metadata or {})
- if _clean_text(metadata.get("event_id", "")) != normalized_event_id:
- continue
- state = _normalize(record.state)
- if state not in {"active", "parallel_active", "evidence"}:
- continue
- hits.append(
- MemoryHit(
- memory_id=record.memory_id,
- category=record.category,
- value=record.value,
- relation=record.relation,
- anchors=list(record.anchor_concepts),
- score=max(float(record.confidence), float(record.salience), 0.01),
- source_kind=record.source_kind,
- slot_key=record.slot_key,
- state=record.state,
- turn_index=int(record.turn_index),
- metadata=metadata,
- )
- )
- return hits
-
-def _hit_matches_path_support(path_type: str, hit: MemoryHit) -> bool:
- metadata = dict(hit.metadata or {})
- source_kind = _clean_text(hit.source_kind)
- category = _clean_text(hit.category)
- relation = _clean_text(hit.relation)
- if path_type == "speaker_event_time":
- return bool(
- source_kind == "public_dialog_time"
- or _clean_text(metadata.get("resolved_time_value", ""))
- or _clean_text(metadata.get("resolved_date", ""))
- or _clean_text(metadata.get("time_value", ""))
- or _clean_text(metadata.get("time_display_value", ""))
- or _clean_text(metadata.get("time_granularity", ""))
- or relation == "event_date"
- or category in {"time", "event_time"}
- )
- if path_type == "speaker_event_profile":
- semantic_slot = _clean_text(metadata.get("semantic_slot", "")) or _clean_text(metadata.get("profile_type", ""))
- return bool(
- source_kind == "public_dialog_profile"
- or semantic_slot in {"identity", "research_topic", "education", "occupation", "profile"}
- or _clean_text(metadata.get("profile_value", ""))
- or category == "profile"
- )
- if path_type == "speaker_event_status":
- return bool(
- _clean_text(metadata.get("target_status", ""))
- or category in {"status", "stage_state"}
- or relation == "status_of"
- )
- if path_type == "speaker_event_source_turn":
- return bool(
- source_kind in {"public_dialog_turn", "public_dialog_text", "public_dialog_auxiliary_evidence"}
- or _clean_text(metadata.get("raw_text", ""))
- or _clean_text(metadata.get("origin_query", ""))
- or _clean_text(metadata.get("source_turn_text", ""))
- or not _is_public_dialog_hit(hit)
- )
- return False
-
-
-def _support_hit_for_path(path_type: str, group_hits: Sequence[MemoryHit]) -> MemoryHit | None:
- matching_hits = [hit for hit in group_hits if _hit_matches_path_support(path_type, hit)]
- if matching_hits:
- matching_hits.sort(key=lambda item: (float(item.score), int(item.turn_index)), reverse=True)
- return matching_hits[0]
- representative = _representative_event_hit(group_hits)
- return representative
-
-
-def _path_support_node_id(path: Dict[str, Any]) -> str:
- node_ids = list(path.get("node_ids", []) or [])
- if len(node_ids) < 3:
- return ""
- return _clean_text(node_ids[2])
-
-
-def _event_ids_from_hits(hits: Sequence[MemoryHit]) -> List[str]:
- return _dedupe(
- _clean_text(dict(hit.metadata or {}).get("event_id", ""))
- for hit in hits
- if _clean_text(dict(hit.metadata or {}).get("event_id", ""))
- )
-
-
-def _dia_ids_from_hits(hits: Sequence[MemoryHit]) -> List[str]:
- return _dedupe(
- _clean_text(dict(hit.metadata or {}).get("dia_id", ""))
- for hit in hits
- if _clean_text(dict(hit.metadata or {}).get("dia_id", ""))
- )
-
-
-def _final_hit_role_priority(hit: MemoryHit) -> int:
- metadata = dict(hit.metadata or {})
- source_kind = _clean_text(hit.source_kind)
- semantic_source_kinds = {
- "public_dialog_fact",
- "public_dialog_preference",
- "public_dialog_goal",
- "public_dialog_constraint",
- "public_dialog_status",
- "public_dialog_profile",
- "public_dialog_profile_cluster",
- "replacement_memory",
- "session_memory",
- }
- if source_kind in semantic_source_kinds or (
- source_kind != "public_dialog_turn" and bool(_clean_text(metadata.get("memory_writer_role", "")))
- ):
- return 0
- if bool(metadata.get("profile_first_source_support")):
- return 0
- role = _clean_text(metadata.get("evidence_snippet_role", ""))
- if role == "selected_path_support":
- return 1
- if role == "selected_event_representative":
- return 2
- if role == "selected_path_event":
- return 3
- return 4
-
-
-def _coverage_preserving_final_hits(
- final_hits: Sequence[MemoryHit],
- *,
- selected_event_ids: Sequence[str],
- top_k: int,
-) -> List[MemoryHit]:
- """Keep selected-event coverage before filling the remaining prompt budget.
-
- Learned selection can emit both path-support and event-representative snippets
- for the same event. A pure score sort can then drop another selected event at
- the top-k boundary, which hides recall/rerank successes from the answer head.
- """
-
- budget = max(1, int(top_k or 1))
- hits = list(final_hits)
- selected_order = [
- _clean_text(event_id)
- for event_id in selected_event_ids
- if _clean_text(event_id)
- ][:budget]
- if not selected_order:
- return sorted(hits, key=lambda item: float(item.score), reverse=True)[:budget]
- hits_by_event: Dict[str, List[MemoryHit]] = {}
- for hit in hits:
- event_id = _clean_text(dict(hit.metadata or {}).get("event_id", ""))
- if event_id:
- hits_by_event.setdefault(event_id, []).append(hit)
- selected: List[MemoryHit] = []
- used_memory_ids = set()
- for event_id in selected_order:
- candidates = [hit for hit in hits_by_event.get(event_id, []) if hit.memory_id not in used_memory_ids]
- if not candidates:
- continue
- candidates.sort(key=lambda item: (_final_hit_role_priority(item), -float(item.score), item.memory_id))
- chosen = candidates[0]
- selected.append(chosen)
- used_memory_ids.add(chosen.memory_id)
- if len(selected) >= budget:
- return selected
- remaining = [
- hit
- for hit in hits
- if hit.memory_id not in used_memory_ids
- ]
- remaining.sort(key=lambda item: (-float(item.score), _final_hit_role_priority(item), item.memory_id))
- for hit in remaining:
- selected.append(hit)
- if len(selected) >= budget:
- break
- return selected[:budget]
-
-
-def _dominant_answer_type(question_analysis: Dict[str, Any], answer_type_scores: Dict[str, Any]) -> str:
- normalized_scores = {
- _clean_text(answer_type): float(value or 0.0)
- for answer_type, value in dict(answer_type_scores or {}).items()
- if _clean_text(answer_type)
- }
- if bool(question_analysis.get("is_temporal", False)) and normalized_scores.get("time", 0.0) >= (normalized_scores.get("abstain", 0.0) - 0.05):
- return "time"
- if not bool(question_analysis.get("is_temporal", False)) and normalized_scores:
- non_time_scores = {
- answer_type: score
- for answer_type, score in normalized_scores.items()
- if answer_type not in {"time", "abstain"}
- }
- if non_time_scores:
- return max(non_time_scores.items(), key=lambda item: (float(item[1]), item[0]))[0]
- if not normalized_scores:
- return "time" if bool(question_analysis.get("is_temporal", False)) else "event_text"
- return max(normalized_scores.items(), key=lambda item: (float(item[1]), item[0]))[0]
-
-
-def _answer_type_preferred_path_types(question_analysis: Dict[str, Any], answer_type_scores: Dict[str, Any]) -> List[str]:
- dominant_answer_type = _dominant_answer_type(question_analysis, answer_type_scores)
- if dominant_answer_type == "time" or bool(question_analysis.get("is_temporal", False)):
- return ["speaker_event_time", "speaker_event_source_turn", "speaker_event_status", "speaker_event_profile"]
- if dominant_answer_type == "profile":
- return ["speaker_event_profile", "speaker_event_source_turn", "speaker_event_status", "speaker_event_time"]
- if dominant_answer_type == "multi_evidence":
- return ["speaker_event_source_turn", "speaker_event_time", "speaker_event_profile", "speaker_event_status"]
- return ["speaker_event_source_turn", "speaker_event_time", "speaker_event_profile", "speaker_event_status"]
-
-
-def _reconciled_focused_answer_type(
- question_analysis: Dict[str, Any],
- answer_type_scores: Dict[str, Any],
- model_answer_type: str,
-) -> str:
- model_type = _clean_text(model_answer_type)
- dominant_type = _dominant_answer_type(question_analysis, answer_type_scores)
- if bool(question_analysis.get("is_temporal", False)) and model_type not in {"", "time", "abstain"}:
- return "time"
- if model_type == "time" and not bool(question_analysis.get("is_temporal", False)):
- return dominant_type if dominant_type != "time" else "event_text"
- return model_type or dominant_type
-
-
-def _path_type_is_focus_compatible(path_type: str, *, focused_answer_type: str, question_analysis: Dict[str, Any]) -> bool:
- normalized_path_type = _clean_text(path_type)
- normalized_answer_type = _clean_text(focused_answer_type)
- if normalized_answer_type == "time" or bool(question_analysis.get("is_temporal", False)):
- return normalized_path_type in {"speaker_event_time", "speaker_event_source_turn"}
- if normalized_answer_type == "profile":
- return normalized_path_type in {"speaker_event_profile", "speaker_event_source_turn"}
- return True
-
-
-def _rank_focus_compatible_path_ids(
- *,
- runtime_paths: Mapping[str, Dict[str, Any]],
- selected_event_ids: Sequence[str],
- path_scores: Mapping[str, Any],
- event_scores: Mapping[str, Any],
- temporal_scores: Mapping[str, Any],
- question_analysis: Dict[str, Any],
- answer_type_scores: Dict[str, Any],
- focused_answer_type: str,
-) -> List[str]:
- selected_event_id_set = {_clean_text(event_id) for event_id in selected_event_ids if _clean_text(event_id)}
- preferred_types = _answer_type_preferred_path_types(question_analysis, answer_type_scores)
- if _clean_text(focused_answer_type) == "profile" and "speaker_event_profile" not in preferred_types:
- preferred_types = ["speaker_event_profile", *preferred_types]
- if (_clean_text(focused_answer_type) == "time" or bool(question_analysis.get("is_temporal", False))) and "speaker_event_time" not in preferred_types:
- preferred_types = ["speaker_event_time", "speaker_event_source_turn", *preferred_types]
- ranked: List[str] = []
- seen = set()
- for preferred_type in preferred_types:
- candidates: List[tuple[str, float]] = []
- for path_id, path in runtime_paths.items():
- if _clean_text(path.get("type", "")) != preferred_type:
- continue
- event_id = _clean_text(path.get("event_id", ""))
- if selected_event_id_set and event_id not in selected_event_id_set:
- continue
- if not _path_type_is_focus_compatible(preferred_type, focused_answer_type=focused_answer_type, question_analysis=question_analysis):
- continue
- support_node_id = _path_support_node_id(path)
- score = (
- float(event_scores.get(event_id, 0.0) or 0.0)
- + (0.20 * float(path_scores.get(path_id, 0.0) or 0.0))
- + (0.15 * float(temporal_scores.get(support_node_id, 0.0) or 0.0))
- )
- candidates.append((path_id, score))
- for path_id, _ in sorted(candidates, key=lambda item: (-float(item[1]), item[0])):
- if path_id in seen:
- continue
- seen.add(path_id)
- ranked.append(path_id)
- if ranked and (_clean_text(focused_answer_type) == "time" or bool(question_analysis.get("is_temporal", False))):
- if preferred_type in {"speaker_event_time", "speaker_event_source_turn"}:
- break
- return ranked
-
-
-def _repair_selected_paths_for_focus(
- selected_path_ids: Sequence[str],
- *,
- runtime_paths: Mapping[str, Dict[str, Any]],
- selected_event_ids: Sequence[str],
- path_scores: Mapping[str, Any],
- event_scores: Mapping[str, Any],
- temporal_scores: Mapping[str, Any],
- question_analysis: Dict[str, Any],
- answer_type_scores: Dict[str, Any],
- focused_answer_type: str,
- limit: int,
-) -> tuple[List[str], bool, str]:
- normalized_selected = [_clean_text(path_id) for path_id in selected_path_ids if _clean_text(path_id)]
- if not normalized_selected:
- return [], False, ""
- incompatible = [
- path_id
- for path_id in normalized_selected
- if not _path_type_is_focus_compatible(
- _clean_text(runtime_paths.get(path_id, {}).get("type", "")),
- focused_answer_type=focused_answer_type,
- question_analysis=question_analysis,
- )
- ]
- if not incompatible:
- return normalized_selected, False, ""
- compatible_ranked = _rank_focus_compatible_path_ids(
- runtime_paths=runtime_paths,
- selected_event_ids=selected_event_ids,
- path_scores=path_scores,
- event_scores=event_scores,
- temporal_scores=temporal_scores,
- question_analysis=question_analysis,
- answer_type_scores=answer_type_scores,
- focused_answer_type=focused_answer_type,
- )
- if not compatible_ranked:
- return normalized_selected, False, ""
- repaired = _dedupe(
- [
- *[path_id for path_id in normalized_selected if path_id not in incompatible],
- *compatible_ranked,
- ]
- )[: max(1, limit)]
- if repaired == normalized_selected:
- return normalized_selected, False, ""
- return repaired, True, "replaced_focus_incompatible_model_paths"
-
-
-def _runtime_node_by_id(runtime_graph: Mapping[str, Any]) -> Dict[str, Dict[str, Any]]:
- return {
- _clean_text(node.get("id", "")): dict(node)
- for node in list(runtime_graph.get("nodes", []) or [])
- if _clean_text(node.get("id", ""))
- }
-
-
-def _runtime_event_subject_signature(runtime_nodes: Mapping[str, Dict[str, Any]], event_id: str) -> str:
- node = dict(runtime_nodes.get(_clean_text(event_id), {}) or {})
- metadata = dict(node.get("metadata", {}) or {})
- return _clean_text(node.get("subject_signature", "")) or _clean_text(metadata.get("subject_signature", ""))
-
-
-def _path_utility_candidate_text(
- path: Mapping[str, Any],
- *,
- runtime_nodes: Mapping[str, Dict[str, Any]],
- grouped_hits: Mapping[str, Sequence[MemoryHit]],
-) -> str:
- event_id = _clean_text(path.get("event_id", ""))
- support_node_id = _path_support_node_id(dict(path))
- node_texts = [
- _clean_text(dict(runtime_nodes.get(event_id, {}) or {}).get("text", "")),
- _clean_text(dict(runtime_nodes.get(support_node_id, {}) or {}).get("text", "")),
- ]
- path_type = _clean_text(path.get("type", ""))
- support_hit = _support_hit_for_path(path_type, grouped_hits.get(event_id, []))
- event_hit = _representative_event_hit(grouped_hits.get(event_id, []))
- hit_texts = [
- _clean_text(support_hit.value if support_hit is not None else ""),
- _clean_text(event_hit.value if event_hit is not None else ""),
- _runtime_source_turn_text(support_hit or event_hit, speaker=""),
- ]
- return " ".join(_dedupe([*node_texts, *hit_texts], max_items=6))
-
-
-def _path_utility_gate(
- candidate_path_ids: Sequence[str],
- *,
- query: str,
- runtime_graph: Mapping[str, Any],
- runtime_paths: Mapping[str, Dict[str, Any]],
- grouped_hits: Mapping[str, Sequence[MemoryHit]],
- selected_path_ids: Sequence[str],
- selected_event_ids_from_model: Sequence[str],
- path_scores: Mapping[str, Any],
- path_tunnel_support_scores: Mapping[str, Any],
- question_analysis: Dict[str, Any],
- focused_answer_type: str,
- score_threshold: float,
- limit: int,
-) -> Dict[str, Any]:
- runtime_nodes = _runtime_node_by_id(runtime_graph)
- query_tokens = set(_path_utility_tokens(query))
- anchor_event_ids = _dedupe(
- [
- *[
- _clean_text(runtime_paths.get(path_id, {}).get("event_id", ""))
- for path_id in selected_path_ids
- if _clean_text(runtime_paths.get(path_id, {}).get("event_id", ""))
- ],
- *[_clean_text(event_id) for event_id in selected_event_ids_from_model if _clean_text(event_id)],
- ]
- )
- anchor_subject_signatures = {
- signature
- for signature in (
- _runtime_event_subject_signature(runtime_nodes, event_id)
- for event_id in anchor_event_ids
- )
- if signature
- }
- direct_path_ids: List[str] = []
- contrast_path_ids: List[str] = []
- latent_path_ids: List[str] = []
- noise_path_ids: List[str] = []
- utility_scores: Dict[str, float] = {}
- utility_roles: Dict[str, str] = {}
- utility_reasons: Dict[str, str] = {}
- utility_overlap_tokens: Dict[str, List[str]] = {}
- for path_id in _dedupe(candidate_path_ids):
- path = dict(runtime_paths.get(path_id, {}) or {})
- if not path:
- continue
- path_type = _clean_text(path.get("type", ""))
- event_id = _clean_text(path.get("event_id", ""))
- candidate_text = _path_utility_candidate_text(
- path,
- runtime_nodes=runtime_nodes,
- grouped_hits=grouped_hits,
- )
- candidate_tokens = set(_path_utility_tokens(candidate_text))
- overlap_tokens = sorted(query_tokens & candidate_tokens)
- overlap_ratio = float(len(overlap_tokens)) / float(max(1, min(len(query_tokens), len(candidate_tokens))))
- support_score = float(path_tunnel_support_scores.get(path_id, 0.0) or 0.0)
- decision_score = float(path_scores.get(path_id, support_score) or 0.0)
- path_score = max(support_score, decision_score)
- subject_signature = _runtime_event_subject_signature(runtime_nodes, event_id)
- same_subject_chain = bool(subject_signature and subject_signature in anchor_subject_signatures)
- focus_compatible = _path_type_is_focus_compatible(
- path_type,
- focused_answer_type=focused_answer_type,
- question_analysis=question_analysis,
- )
- utility_score = path_score + (0.20 * overlap_ratio) + (0.04 if same_subject_chain else 0.0)
- utility_scores[path_id] = round(float(utility_score), 6)
- utility_overlap_tokens[path_id] = overlap_tokens[:12]
- if not focus_compatible:
- role = "drift_noise"
- reason = "focus_incompatible"
- elif overlap_ratio >= 0.18 and path_score >= score_threshold:
- role = "direct_support"
- reason = "query_overlap_and_tunnel_score"
- elif same_subject_chain and path_score >= score_threshold:
- role = "contrast_support"
- reason = "same_subject_deep_chain"
- elif path_score >= score_threshold:
- role = "latent_context"
- reason = "tunnel_score_without_current_turn_utility"
- else:
- role = "drift_noise"
- reason = "below_utility_threshold"
- utility_roles[path_id] = role
- utility_reasons[path_id] = reason
- if role == "direct_support":
- direct_path_ids.append(path_id)
- elif role == "contrast_support":
- contrast_path_ids.append(path_id)
- elif role == "latent_context":
- latent_path_ids.append(path_id)
- else:
- noise_path_ids.append(path_id)
- injected_path_ids = _dedupe([*direct_path_ids, *contrast_path_ids], max_items=max(0, int(limit)))
- overflow_latent_path_ids = [
- path_id
- for path_id in [*direct_path_ids, *contrast_path_ids]
- if path_id not in set(injected_path_ids)
- ]
- latent_path_ids = _dedupe([*latent_path_ids, *overflow_latent_path_ids])
- return {
- "enabled": True,
- "candidate_path_ids": list(_dedupe(candidate_path_ids)),
- "direct_support_path_ids": list(direct_path_ids),
- "contrast_support_path_ids": list(contrast_path_ids),
- "latent_context_path_ids": list(latent_path_ids),
- "drift_noise_path_ids": list(noise_path_ids),
- "injected_path_ids": list(injected_path_ids),
- "roles": dict(utility_roles),
- "reasons": dict(utility_reasons),
- "scores": dict(utility_scores),
- "overlap_tokens": dict(utility_overlap_tokens),
- "anchor_event_ids": list(anchor_event_ids),
- "anchor_subject_signatures": sorted(anchor_subject_signatures),
- }
-
-
-def _calibrated_path_score(
- *,
- path: Dict[str, Any],
- base_score: float,
- temporal_scores: Dict[str, Any],
- question_analysis: Dict[str, Any],
- answer_type_scores: Dict[str, Any],
-) -> float:
- normalized_path_type = _clean_text(path.get("type", ""))
- support_node_id = _path_support_node_id(path)
- temporal_score = float(temporal_scores.get(support_node_id, 0.0) or 0.0)
- normalized_answer_scores = {
- _clean_text(answer_type): float(value or 0.0)
- for answer_type, value in dict(answer_type_scores or {}).items()
- if _clean_text(answer_type)
- }
- dominant_answer_type = _dominant_answer_type(question_analysis, normalized_answer_scores)
- calibrated = float(base_score)
- if dominant_answer_type == "time" or bool(question_analysis.get("is_temporal", False)):
- if normalized_path_type == "speaker_event_time":
- calibrated += (0.30 * temporal_score) + (0.12 * normalized_answer_scores.get("time", 0.0))
- elif normalized_path_type == "speaker_event_source_turn":
- calibrated -= 0.08 + (0.04 * normalized_answer_scores.get("time", 0.0))
- elif normalized_path_type == "speaker_event_status":
- calibrated -= 0.12
- elif normalized_path_type == "speaker_event_profile":
- calibrated -= 0.22 + (0.06 * normalized_answer_scores.get("time", 0.0))
- elif dominant_answer_type == "profile":
- if normalized_path_type == "speaker_event_profile":
- calibrated += 0.12 * normalized_answer_scores.get("profile", 0.0)
- elif normalized_path_type != "speaker_event_source_turn":
- calibrated -= 0.08
- else:
- if normalized_path_type == "speaker_event_source_turn":
- calibrated += 0.08 * max(
- normalized_answer_scores.get("event_text", 0.0),
- normalized_answer_scores.get("multi_evidence", 0.0),
- )
- elif normalized_path_type == "speaker_event_profile":
- calibrated -= 0.04
- return calibrated
-
-
-def _group_metadata_value(
- group_hits: Sequence[MemoryHit],
- key: str,
- *,
- source_kinds: Sequence[str] = (),
-) -> str:
- source_kind_set = {_clean_text(item) for item in list(source_kinds) if _clean_text(item)}
- candidates = [
- hit
- for hit in group_hits
- if not source_kind_set or _clean_text(hit.source_kind) in source_kind_set
- ]
- candidates.sort(key=lambda item: (float(item.score), int(item.turn_index)), reverse=True)
- for hit in candidates:
- metadata = dict(hit.metadata or {})
- value = _clean_text(metadata.get(key, ""))
- if value:
- return value
- return ""
-
-
-def _runtime_event_sequence_key(session_name: str, turn_index: int, event_id: str) -> tuple[Any, ...]:
- normalized_session = _clean_text(session_name)
- session_number_match = re.search(r"(\d+)$", normalized_session)
- if session_number_match:
- return (0, int(session_number_match.group(1)), int(turn_index), _clean_text(event_id))
- return (1, normalized_session, int(turn_index), _clean_text(event_id))
-
-
-def _runtime_source_turn_text(hit: MemoryHit | None, *, speaker: str) -> str:
- if hit is None:
- return ""
- metadata = dict(hit.metadata or {})
- source_turn_text = _clean_text(metadata.get("source_turn_text", ""))
- if source_turn_text:
- return source_turn_text
- raw_text = _clean_text(metadata.get("raw_text", ""))
- if raw_text:
- auxiliary_text = _clean_text(metadata.get("auxiliary_evidence_text", ""))
- if auxiliary_text and auxiliary_text.lower() not in raw_text.lower():
- return f"{raw_text}\nAuxiliary evidence: {auxiliary_text}"
- return raw_text
- origin_query = _clean_text(metadata.get("origin_query", ""))
- if origin_query:
- return origin_query
- text = _clean_text(hit.value)
- if _clean_text(hit.source_kind) == "public_dialog_turn":
- text = re.sub(r"^\[[^\]]+\]\s*", "", text)
- if _clean_text(speaker):
- text = re.sub(rf"^{re.escape(_clean_text(speaker))}\s*:\s*", "", text, flags=re.IGNORECASE)
- text = re.sub(r"^[A-Za-z][A-Za-z0-9_' -]{0,40}:\s*", "", text)
- return text
-
-
-def _runtime_event_signature(
- *,
- group_hits: Sequence[MemoryHit],
- representative: MemoryHit,
- speaker: str,
- semantic_slot: str,
- source_turn_hit: MemoryHit | None,
-) -> str:
- existing = _group_metadata_value(group_hits, "event_signature") or _clean_text(dict(representative.metadata or {}).get("event_signature", ""))
- if existing:
- return existing
- event_phrase = _group_metadata_value(group_hits, "event_phrase") or _clean_text(dict(representative.metadata or {}).get("event_phrase", ""))
- source_turn_text = _runtime_source_turn_text(source_turn_hit, speaker=speaker)
- base_text = event_phrase or _clean_text(representative.value) or source_turn_text
- if not base_text:
- return ""
- return compute_public_event_signature(
- base_text,
- speaker=_clean_text(speaker),
- semantic_slot=_clean_text(semantic_slot),
- ) or _clean_text(base_text)
-
-
-def _build_runtime_graph_from_hits(query: str, hits: Sequence[MemoryHit]) -> Dict[str, Any]:
- nodes: List[Dict[str, Any]] = []
- edges: List[Dict[str, Any]] = []
- paths: List[Dict[str, Any]] = []
- node_ids = set()
- grouped_hits: Dict[str, List[MemoryHit]] = {}
- ordered_events: List[tuple[Any, ...]] = []
- event_typed_metadata_by_id: Dict[str, Dict[str, Any]] = {}
- typed_tunnel_edges: List[Dict[str, Any]] = []
-
- def add_node(node: Dict[str, Any]) -> None:
- node_id = _clean_text(node.get("id", ""))
- if not node_id or node_id in node_ids:
- return
- node_ids.add(node_id)
- nodes.append(node)
-
- for hit in hits:
- event_id = _runtime_event_key(hit)
- grouped_hits.setdefault(event_id, []).append(hit)
-
- for event_id, group_hits in grouped_hits.items():
- representative = _representative_event_hit(group_hits)
- if representative is None:
- continue
- metadata = dict(representative.metadata or {})
- event_time_hit = _support_hit_for_path("speaker_event_time", group_hits)
- event_profile_hit = _support_hit_for_path("speaker_event_profile", group_hits)
- source_turn_hit = _support_hit_for_path("speaker_event_source_turn", group_hits)
- event_time_display_value = (
- _group_metadata_value(group_hits, "time_display_value", source_kinds=("public_dialog_time",))
- or _group_metadata_value(group_hits, "resolved_date")
- or _group_metadata_value(group_hits, "time_display_value")
- )
- event_time_value = (
- _group_metadata_value(group_hits, "resolved_time_value", source_kinds=("public_dialog_time",))
- or _group_metadata_value(group_hits, "resolved_time_value")
- or _group_metadata_value(group_hits, "time_value")
- )
- event_time_granularity = (
- _group_metadata_value(group_hits, "time_granularity", source_kinds=("public_dialog_time",))
- or _group_metadata_value(group_hits, "time_granularity")
- )
- event_profile_type = (
- _group_metadata_value(group_hits, "profile_type", source_kinds=("public_dialog_profile",))
- or _group_metadata_value(group_hits, "semantic_slot", source_kinds=("public_dialog_profile",))
- or _group_metadata_value(group_hits, "profile_type")
- or _group_metadata_value(group_hits, "semantic_slot")
- )
- profile_hit = _support_hit_for_path("speaker_event_profile", group_hits)
- event_profile_value = (
- _group_metadata_value(group_hits, "profile_value")
- or _clean_text(profile_hit.value if profile_hit is not None else "")
- or (_clean_text(representative.value) if event_profile_type else "")
- )
- event_target_status = _group_metadata_value(group_hits, "target_status")
- depth_layer = (
- _clean_text(metadata.get("depth_layer", ""))
- or _group_metadata_value(group_hits, "depth_layer")
- or _clean_text(metadata.get("memory_chain_depth_layer", ""))
- or _group_metadata_value(group_hits, "memory_chain_depth_layer")
- )
- subject_signature = (
- _clean_text(metadata.get("subject_signature", ""))
- or _group_metadata_value(group_hits, "subject_signature")
- or _clean_text(metadata.get("memory_chain_subject_signature", ""))
- or _group_metadata_value(group_hits, "memory_chain_subject_signature")
- )
- session_name = (
- _group_metadata_value(group_hits, "session_name")
- or _group_metadata_value(group_hits, "session_key")
- or _group_metadata_value(group_hits, "scope_id")
- or "runtime_session"
- )
- speaker = (
- _clean_text(metadata.get("speaker", ""))
- or _group_metadata_value(group_hits, "speaker")
- or _clean_text(metadata.get("subject_signature", ""))
- or (_clean_text(representative.anchors[0]) if representative.anchors else "")
- or "speaker"
- )
- event_turn_index = int(getattr(representative, "turn_index", 0) or 0)
- semantic_slot = (
- _group_metadata_value(group_hits, "semantic_slot")
- or _clean_text(metadata.get("semantic_slot", ""))
- or _clean_text(metadata.get("profile_type", ""))
- or ("profile" if event_profile_type else _clean_text(representative.category))
- or "event"
- )
- teacher_fields = {
- "event_phrase": _clean_text(metadata.get("event_phrase", "")) or _clean_text(representative.value),
- "semantic_slot": semantic_slot,
- "target_status": event_target_status,
- "time_expression_span": event_time_display_value,
- "time_granularity": event_time_granularity,
- "profile_type": event_profile_type,
- }
- base_event_signature = _runtime_event_signature(
- group_hits=group_hits,
- representative=representative,
- speaker=speaker,
- semantic_slot=event_profile_type or teacher_fields["semantic_slot"],
- source_turn_hit=source_turn_hit,
- )
- typed_tunnel_metadata = merge_typed_metadata([metadata, *[dict(hit.metadata or {}) for hit in group_hits]])
- event_typed_metadata_by_id[event_id] = typed_tunnel_metadata
- typed_signature = typed_tunnel_signature_text(typed_tunnel_metadata)
- event_text = _clean_text(metadata.get("event_phrase", "")) or representative.value
- depth_prefix = " ".join(
- item
- for item in (
- f"subject {subject_signature.replace('_', ' ')}" if subject_signature else "",
- f"depth layer {depth_layer.replace('_', ' ')}" if depth_layer else "",
- )
- if item
- )
- runtime_event_text = f"{depth_prefix} {event_text}".strip() if depth_prefix else event_text
- event_signature = (
- compute_public_event_signature(
- runtime_event_text,
- speaker=_clean_text(speaker),
- semantic_slot=_clean_text(event_profile_type or teacher_fields["semantic_slot"]),
- )
- or base_event_signature
- )
- if typed_signature and typed_signature not in event_signature:
- event_signature = f"{event_signature} {typed_signature}".strip()
- speaker_node_id = f"{event_id}:speaker:{_normalize(speaker).replace(' ', '_') or 'speaker'}"
- add_node({"id": speaker_node_id, "type": "speaker", "text": speaker, "metadata": {"speaker": speaker}})
- add_node(
- {
- "id": event_id,
- "type": "event",
- "text": runtime_event_text,
- "speaker": speaker,
- "turn_index": event_turn_index,
- "session_name": session_name,
- "dia_id": _clean_text(metadata.get("dia_id", "")),
- "event_signature": event_signature,
- "slot_key": _clean_text(representative.slot_key),
- "state_signature": _clean_text(metadata.get("state_signature", "")),
- "memory_signature": _clean_text(metadata.get("memory_signature", "")),
- "target_status": event_target_status,
- "time_granularity": event_time_granularity,
- "time_value": event_time_value,
- "time_display_value": event_time_display_value,
- "profile_type": event_profile_type,
- "profile_value": event_profile_value,
- "depth_layer": depth_layer,
- "subject_signature": subject_signature,
- "tmcra_node_tags": list(typed_tunnel_metadata.get("tmcra_node_tags", []) or []),
- "tmcra_path_tags": list(typed_tunnel_metadata.get("tmcra_path_tags", []) or []),
- "tmcra_tunnel_roles": list(typed_tunnel_metadata.get("tmcra_tunnel_roles", []) or []),
- "tmcra_tunnel_group_key": _clean_text(typed_tunnel_metadata.get("tmcra_tunnel_group_key", "")),
- "teacher_fields": teacher_fields,
- "metadata": {
- "speaker": speaker,
- "session_name": session_name,
- "dia_id": _clean_text(metadata.get("dia_id", "")),
- "slot_key": _clean_text(representative.slot_key),
- "state_signature": _clean_text(metadata.get("state_signature", "")),
- "memory_signature": _clean_text(metadata.get("memory_signature", "")),
- "target_status": event_target_status,
- "time_granularity": event_time_granularity,
- "time_value": event_time_value,
- "time_display_value": event_time_display_value,
- "profile_type": event_profile_type,
- "profile_value": event_profile_value,
- "event_signature": event_signature,
- "depth_layer": depth_layer,
- "subject_signature": subject_signature,
- **typed_tunnel_metadata,
- },
- }
- )
- edges.append({"id": f"{speaker_node_id}->{event_id}:speaker_of", "source": speaker_node_id, "target": event_id, "type": "speaker_of"})
-
- time_node_ids: List[str] = []
- profile_node_ids: List[str] = []
- status_node_ids: List[str] = []
- source_turn_node_ids: List[str] = []
-
- if event_time_display_value or event_time_value:
- time_node_id = f"{event_id}:time"
- time_hit_metadata = dict((event_time_hit or representative).metadata or {})
- add_node(
- {
- "id": time_node_id,
- "type": "time",
- "text": event_time_display_value or event_time_value,
- "turn_index": int(getattr(event_time_hit, "turn_index", 0) or 0),
- "time_display_value": event_time_display_value,
- "time_value": event_time_value,
- "time_granularity": event_time_granularity,
- "metadata": {
- "time_display_value": event_time_display_value,
- "time_value": event_time_value,
- "time_granularity": event_time_granularity,
- "resolved_date": _clean_text(time_hit_metadata.get("resolved_date", "")),
- },
- }
- )
- edges.append({"id": f"{event_id}->{time_node_id}:time_of", "source": event_id, "target": time_node_id, "type": "time_of"})
- time_node_ids.append(time_node_id)
- if event_profile_value:
- profile_node_id = f"{event_id}:profile:{_normalize(event_profile_type).replace(' ', '_') or 'profile'}"
- add_node(
- {
- "id": profile_node_id,
- "type": "profile",
- "text": event_profile_value,
- "turn_index": int(getattr(event_profile_hit or representative, "turn_index", 0) or 0),
- "profile_type": event_profile_type,
- "profile_value": event_profile_value,
- "metadata": {
- "profile_type": event_profile_type,
- "profile_value": event_profile_value,
- },
- }
- )
- edges.append({"id": f"{event_id}->{profile_node_id}:profile_of", "source": event_id, "target": profile_node_id, "type": "profile_of"})
- profile_node_ids.append(profile_node_id)
- source_turn_text = _runtime_source_turn_text(source_turn_hit or representative, speaker=speaker)
- if source_turn_text:
- source_turn_node_id = f"{event_id}:source_turn"
- add_node(
- {
- "id": source_turn_node_id,
- "type": "source_turn",
- "text": source_turn_text,
- "turn_index": int(getattr(source_turn_hit or representative, "turn_index", 0) or 0),
- "metadata": {
- "speaker": speaker,
- "dia_id": _clean_text(dict((source_turn_hit or representative).metadata or {}).get("dia_id", "")),
- },
- }
- )
- edges.append({"id": f"{event_id}->{source_turn_node_id}:supported_by_turn", "source": event_id, "target": source_turn_node_id, "type": "supported_by_turn"})
- source_turn_node_ids.append(source_turn_node_id)
- if event_target_status:
- status_node_id = f"{event_id}:status"
- add_node({"id": status_node_id, "type": "status", "text": event_target_status, "metadata": {"target_status": event_target_status}})
- edges.append({"id": f"{event_id}->{status_node_id}:status_of", "source": event_id, "target": status_node_id, "type": "status_of"})
- status_node_ids.append(status_node_id)
- event_paths = build_default_path_templates(
- event_id=event_id,
- speaker_node_id=speaker_node_id,
- time_node_ids=time_node_ids,
- profile_node_ids=profile_node_ids,
- status_node_ids=status_node_ids,
- source_turn_node_ids=source_turn_node_ids,
- )
- for path in event_paths:
- path_metadata = dict(path.get("metadata", {}) or {})
- path_metadata["tmcra_path_tags"] = list(typed_tunnel_metadata.get("tmcra_path_tags", []) or [])
- path_metadata["tmcra_tunnel_group_key"] = _clean_text(typed_tunnel_metadata.get("tmcra_tunnel_group_key", ""))
- path["metadata"] = path_metadata
- path["tmcra_path_tags"] = path_metadata["tmcra_path_tags"]
- paths.extend(event_paths)
- ordered_events.append(_runtime_event_sequence_key(session_name, event_turn_index, event_id))
- ordered_event_ids = [event_id for _, _, _, event_id in sorted(ordered_events)]
- for previous_event_id, next_event_id in zip(ordered_event_ids, ordered_event_ids[1:]):
- typed_edge_tags = typed_edge_tags_between(
- event_typed_metadata_by_id.get(previous_event_id, {}),
- event_typed_metadata_by_id.get(next_event_id, {}),
- )
- edges.append(
- {
- "id": f"{previous_event_id}->{next_event_id}:same_session_next",
- "source": previous_event_id,
- "target": next_event_id,
- "type": "same_session_next",
- "metadata": {
- "tmcra_edge_tags": typed_edge_tags,
- "typed_tunnel_edge": bool(typed_edge_tags),
- },
- }
- )
- for index, source_event_id in enumerate(ordered_event_ids):
- for target_event_id in ordered_event_ids[index + 1 : index + 12]:
- typed_edge_tags = typed_edge_tags_between(
- event_typed_metadata_by_id.get(source_event_id, {}),
- event_typed_metadata_by_id.get(target_event_id, {}),
- )
- if not typed_edge_tags:
- continue
- typed_tunnel_edges.append(
- {
- "id": f"{source_event_id}->{target_event_id}:typed_tunnel",
- "source": source_event_id,
- "target": target_event_id,
- "type": "typed_tunnel_candidate",
- "metadata": {
- "tmcra_edge_tags": typed_edge_tags,
- "typed_tunnel_edge": True,
- },
- }
- )
- if len(typed_tunnel_edges) >= 64:
- break
- if len(typed_tunnel_edges) >= 64:
- break
- return {
- "conversation_id": "runtime",
- "query": query,
- "nodes": nodes,
- "edges": edges,
- "typed_tunnel_edges": typed_tunnel_edges,
- "paths": paths,
- "grouped_hits": grouped_hits,
- }
-
-
-def _public_graph_hits(graph: SessionMemoryGraphV2) -> List[MemoryHit]:
- public_hits: List[MemoryHit] = []
- for record in graph.records_by_id.values():
- if record.state != "active":
- continue
- if not _clean_text(record.source_kind).startswith("public_dialog"):
- continue
- metadata = dict(record.metadata or {})
- public_hits.append(
- MemoryHit(
- memory_id=record.memory_id,
- category=record.category,
- value=record.value,
- relation=record.relation,
- anchors=list(record.anchor_concepts),
- score=max(float(record.confidence), float(record.salience), 0.01),
- source_kind=record.source_kind,
- slot_key=record.slot_key,
- state=record.state,
- turn_index=int(record.turn_index),
- metadata=metadata,
- )
- )
- public_hits.sort(key=lambda item: (int(item.turn_index), float(item.score)), reverse=True)
- return public_hits
-
-
-_AUDIT_ANCHOR_QUERY_RE = re.compile(
- r"(?i)\b(?:remember|recall|earlier|previous|previously|old|before|mentioned|said|quote|verbatim|original|"
- r"that time|last time|bring back|return to|go back to|turn\s*\d+)\b|"
- r"(?:\u7b2c\s*\d+\s*[\u8f6e\u6b21]|\d+\s*\u8f6e|\u4e4b\u524d|\u4ee5\u524d|\u521a\u521a|"
- r"\u539f\u8bdd|\u90a3\u6b21|\u65e7\u8bdd\u9898|\u56de\u5230|\u63d0\u8d77|\u8bb0\u5f97)"
-)
-
-_AUDIT_TURN_ANCHOR_RE = re.compile(
- r"(?i)\bturn\s*#?\s*(\d{1,6})\b|"
- r"(?:\u7b2c\s*(\d{1,6})\s*[\u8f6e\u6b21]|\b(\d{1,6})\s*\u8f6e\b)"
-)
-
-_AUDIT_ANCHOR_STOPWORDS = set(_HYBRID_SYMBOLIC_STOPWORDS) | {
- "about",
- "again",
- "back",
- "bring",
- "can",
- "could",
- "did",
- "discuss",
- "discussed",
- "earlier",
- "from",
- "just",
- "keep",
- "mentioned",
- "old",
- "one",
- "previous",
- "previously",
- "really",
- "recall",
- "remember",
- "return",
- "said",
- "still",
- "that",
- "the",
- "thing",
- "think",
- "this",
- "turn",
- "what",
- "when",
- "where",
- "with",
- "you",
-}
-
-_AUDIT_ANCHOR_GENERIC_MATCH_TOKENS = {
- "body",
- "coherence",
- "continuity",
- "fiction",
- "fray",
- "memories",
- "memory",
- "narrative",
- "ourselves",
- "physical",
- "really",
- "stories",
- "story",
- "tell",
- "trust",
-}
-
-_AUDIT_ANCHOR_PHRASE_TOKEN_SETS = [
- {"spine", "book"},
- {"body", "spine"},
- {"stories", "pages"},
- {"story", "pages"},
- {"islands", "self"},
- {"archipelago", "self"},
- {"loom", "body"},
- {"braid", "body"},
-]
-
-
-def _audit_anchor_query(query: str) -> bool:
- return bool(_AUDIT_ANCHOR_QUERY_RE.search(_clean_text(query)))
-
-
-def _audit_anchor_turn_numbers(query: str) -> List[int]:
- numbers: List[int] = []
- for match in _AUDIT_TURN_ANCHOR_RE.finditer(_clean_text(query)):
- for group in match.groups():
- if not group:
- continue
- try:
- value = int(group)
- except Exception:
- continue
- if value > 0 and value not in numbers:
- numbers.append(value)
- return numbers
-
-
-def _hit_event_id(hit: MemoryHit) -> str:
- metadata = dict(hit.metadata or {})
- event_id = _clean_text(metadata.get("event_id", ""))
- if event_id:
- return event_id
- dia_id = _clean_text(metadata.get("dia_id", ""))
- if dia_id:
- return f"event::{dia_id}"
- if int(hit.turn_index or 0) > 0:
- return f"event::realchat:{int(hit.turn_index)}"
- return ""
-
-
-def _audit_anchor_hit_text(hit: MemoryHit) -> str:
- metadata = dict(hit.metadata or {})
- values = [
- hit.value,
- hit.category,
- hit.source_kind,
- hit.slot_key,
- metadata.get("event_text", ""),
- metadata.get("source_span", ""),
- metadata.get("raw_text", ""),
- ]
- return " ".join(_clean_text(value) for value in values if _clean_text(value))
-
-
-def _audit_anchor_content_tokens(text: str) -> set[str]:
- tokens = set()
- for token in _tokenize(text):
- norm = _normalize(token)
- if not norm or norm in _AUDIT_ANCHOR_STOPWORDS:
- continue
- if len(norm) < 3 and not any("\u4e00" <= ch <= "\u9fff" for ch in norm):
- continue
- tokens.add(norm)
- return tokens
-
-
-def _audit_anchor_phrase_bonus(query_tokens: set[str], hit_tokens: set[str]) -> tuple[float, List[str]]:
- matched_phrases: List[str] = []
- bonus = 0.0
- for phrase_tokens in _AUDIT_ANCHOR_PHRASE_TOKEN_SETS:
- if phrase_tokens <= query_tokens and phrase_tokens <= hit_tokens:
- matched_phrases.append("+".join(sorted(phrase_tokens)))
- bonus += 5.0 if phrase_tokens == {"spine", "book"} else 3.0
- return bonus, matched_phrases
-
-
-def _copy_hit_with_audit_anchor_boost(hit: MemoryHit, *, score: float, reason: str, matched_tokens: Sequence[str]) -> MemoryHit:
- metadata = dict(hit.metadata or {})
- metadata.update(
- {
- "audit_anchor_protected": True,
- "audit_anchor_reason": reason,
- "audit_anchor_original_score": round(float(hit.score), 6),
- "audit_anchor_score": round(float(score), 6),
- "audit_anchor_matched_tokens": list(matched_tokens)[:20],
- }
- )
- return MemoryHit(
- memory_id=hit.memory_id,
- category=hit.category,
- value=hit.value,
- relation=hit.relation,
- anchors=list(hit.anchors),
- score=max(float(hit.score), round(1.0 + float(score), 6)),
- source_kind=hit.source_kind,
- slot_key=hit.slot_key,
- state=hit.state,
- turn_index=int(hit.turn_index),
- metadata=metadata,
- )
-
-
-def _audit_anchor_protected_hits(
- *,
- query: str,
- final_hits: Sequence[MemoryHit],
- candidate_hits: Sequence[MemoryHit],
- metadata: Mapping[str, Any],
- top_k: int,
-) -> Dict[str, Any]:
- if not _audit_anchor_query(query):
- return {"enabled": False, "hits": list(final_hits), "promoted_hits": [], "metadata": {"audit_anchor_enabled": False}}
- query_tokens = _audit_anchor_content_tokens(query)
- if not query_tokens:
- return {"enabled": True, "hits": list(final_hits), "promoted_hits": [], "metadata": {"audit_anchor_enabled": True, "audit_anchor_reason": "no_content_tokens"}}
-
- explicit_turns = _audit_anchor_turn_numbers(query)
- explicit_turn_window = set()
- for number in explicit_turns:
- explicit_turn_window.update({number - 1, number, number + 1})
-
- symbolic_ids = list(dict.fromkeys(str(item) for item in dict(metadata or {}).get("symbolic_recall_event_ids", []) or []))
- learned_ids = list(dict.fromkeys(str(item) for item in dict(metadata or {}).get("learned_recall_event_ids", []) or []))
- selected_ids = set(str(item) for item in dict(metadata or {}).get("selected_event_ids", []) or [])
- symbolic_rank = {event_id: index for index, event_id in enumerate(symbolic_ids, start=1)}
- learned_rank = {event_id: index for index, event_id in enumerate(learned_ids, start=1)}
-
- pool: Dict[str, MemoryHit] = {}
- for hit in list(final_hits) + list(candidate_hits):
- key = hit.memory_id or f"{_hit_event_id(hit)}::{hit.slot_key}::{hit.value[:80]}"
- if key and key not in pool:
- pool[key] = hit
-
- scored: List[tuple[float, str, List[str], MemoryHit]] = []
- for hit in pool.values():
- event_id = _hit_event_id(hit)
- hit_tokens = _audit_anchor_content_tokens(_audit_anchor_hit_text(hit))
- matched = sorted(query_tokens & hit_tokens)
- distinctive_matched = [token for token in matched if token not in _AUDIT_ANCHOR_GENERIC_MATCH_TOKENS]
- phrase_bonus, matched_phrases = _audit_anchor_phrase_bonus(query_tokens, hit_tokens)
- if not matched and int(hit.turn_index or 0) not in explicit_turn_window:
- continue
- score = 0.0
- reason_parts: List[str] = []
- if int(hit.turn_index or 0) in explicit_turn_window:
- score += 8.0
- reason_parts.append("explicit_turn_anchor")
- if matched:
- weighted_overlap = 0.0
- for token in matched:
- weighted_overlap += 1.8 if token not in _AUDIT_ANCHOR_GENERIC_MATCH_TOKENS else 0.35
- score += min(8.0, weighted_overlap)
- score += len(matched) / max(1.0, float(min(len(query_tokens), len(hit_tokens))))
- reason_parts.append("lexical_anchor_overlap")
- if phrase_bonus > 0:
- score += phrase_bonus
- reason_parts.append("distinctive_phrase_anchor")
- if event_id in symbolic_rank:
- score += max(0.0, 2.0 - (symbolic_rank[event_id] - 1) * 0.08)
- reason_parts.append("symbolic_recall_anchor")
- if event_id in learned_rank:
- score += max(0.0, 1.0 - (learned_rank[event_id] - 1) * 0.02)
- reason_parts.append("learned_recall_anchor")
- if event_id in selected_ids:
- score -= 0.25
- # For non-numeric old-topic probes, require a real content overlap so generic
- # "earlier" phrasing does not promote unrelated old memories.
- if not explicit_turns and len(distinctive_matched) < 1 and phrase_bonus <= 0:
- continue
- if score > 0:
- scored.append((score, event_id, sorted(set(matched + matched_phrases)), hit))
-
- scored.sort(key=lambda item: (item[0], -abs(int(item[3].turn_index or 0))), reverse=True)
- max_promoted = 2 if not explicit_turns else 3
- promoted: List[MemoryHit] = []
- promoted_event_ids: set[str] = set()
- for score, event_id, matched, hit in scored:
- if event_id in promoted_event_ids:
- continue
- if any((hit.memory_id and hit.memory_id == current.memory_id) for current in final_hits[: max(1, top_k)]):
- continue
- reason = "explicit_turn_anchor" if int(hit.turn_index or 0) in explicit_turn_window else "old_topic_anchor"
- promoted.append(_copy_hit_with_audit_anchor_boost(hit, score=score, reason=reason, matched_tokens=matched))
- promoted_event_ids.add(event_id)
- if len(promoted) >= max_promoted:
- break
-
- if not promoted:
- return {
- "enabled": True,
- "hits": list(final_hits),
- "promoted_hits": [],
- "metadata": {
- "audit_anchor_enabled": True,
- "audit_anchor_turn_numbers": explicit_turns,
- "audit_anchor_query_tokens": sorted(query_tokens)[:50],
- "audit_anchor_promoted_event_ids": [],
- },
- }
-
- merged: List[MemoryHit] = []
- seen_keys: set[str] = set()
- for hit in promoted + list(final_hits):
- key = hit.memory_id or f"{_hit_event_id(hit)}::{hit.slot_key}::{hit.value[:80]}"
- if key in seen_keys:
- continue
- seen_keys.add(key)
- merged.append(hit)
- if len(merged) >= max(1, int(top_k)):
- break
-
- return {
- "enabled": True,
- "hits": merged,
- "promoted_hits": promoted,
- "metadata": {
- "audit_anchor_enabled": True,
- "audit_anchor_turn_numbers": explicit_turns,
- "audit_anchor_query_tokens": sorted(query_tokens)[:50],
- "audit_anchor_promoted_event_ids": [_hit_event_id(hit) for hit in promoted],
- "audit_anchor_promoted_turns": [int(hit.turn_index) for hit in promoted],
- "audit_anchor_promoted_hit_count": len(promoted),
- },
- }
-
-
-def _learnable_graph_hits(graph: SessionMemoryGraphV2) -> List[MemoryHit]:
- learnable_hits: List[MemoryHit] = []
- for record in graph.records_by_id.values():
- state = _normalize(record.state)
- metadata = dict(record.metadata or {})
- is_source_grounded_evidence = (
- state == "evidence"
- and (
- _clean_text(record.source_kind).startswith("public_dialog")
- or _clean_text(metadata.get("content_variant", "")) in {"source_turn", "llm_semantic_write"}
- or _clean_text(metadata.get("write_path", "")) == "llm_semantic_writer_gate"
- )
- )
- if state not in {"active", "parallel_active"} and not is_source_grounded_evidence:
- continue
- if _clean_text(record.slot_key).startswith("noise."):
- continue
- learnable_hits.append(
- MemoryHit(
- memory_id=record.memory_id,
- category=record.category,
- value=record.value,
- relation=record.relation,
- anchors=list(record.anchor_concepts),
- score=max(float(record.confidence), float(record.salience), 0.01),
- source_kind=record.source_kind,
- slot_key=record.slot_key,
- state=record.state,
- turn_index=int(record.turn_index),
- metadata=metadata,
- )
- )
- learnable_hits.sort(
- key=lambda item: (
- _is_public_dialog_hit(item),
- int(item.turn_index),
- float(item.score),
- ),
- reverse=True,
- )
- return learnable_hits
-
-
-def _parse_structured_records(
- payload: Dict[str, Any] | None,
- *,
- turn_index: int,
- profile: TMCRAProfile | None = None,
-) -> List[SessionMemoryRecordV2]:
- profile = profile or TMCRAProfile()
- records: List[SessionMemoryRecordV2] = []
- structured_rows: List[tuple[str, Mapping[str, Any]]] = []
- for raw in (payload or {}).get("replacement_memory_records", []) or []:
- if isinstance(raw, Mapping):
- structured_rows.append(("formal", raw))
- for raw in (payload or {}).get("suspect_memory_records", []) or []:
- if isinstance(raw, Mapping):
- structured_rows.append(("suspect", raw))
- for index, (buffer_state, raw) in enumerate(structured_rows):
- if not isinstance(raw, dict):
- continue
- category = _clean_text(raw.get("category", "memory")) or "memory"
- value = _clean_text(raw.get("value", ""))
- if not value:
- continue
- anchors = _dedupe(raw.get("anchors", []) or [], max_items=8)
- metadata = dict(raw.get("metadata", {}) or {})
- slot_key = profile.stable_slot_key(
- category=category,
- value=value,
- anchors=anchors,
- slot_key=_clean_text(raw.get("slot_key", "")) or _clean_text(raw.get("slot", "")),
- relation=_clean_text(raw.get("relation", "")),
- metadata=metadata,
- )
- metadata = {
- **metadata,
- "memory_role": _clean_text(metadata.get("memory_role", "")) or "user",
- "authority": _clean_text(metadata.get("authority", "")) or "source",
- "canonical_slot_key": _clean_text(metadata.get("canonical_slot_key", "")) or slot_key,
- "writeback_class": _clean_text(metadata.get("writeback_class", "")),
- "origin_query": _clean_text(metadata.get("origin_query", "")),
- "origin_answer_id": _clean_text(metadata.get("origin_answer_id", "")),
- "support_memory_ids": _dedupe(metadata.get("support_memory_ids", []) or []),
- "support_fact_refs": _dedupe(metadata.get("support_fact_refs", []) or []),
- "support_path_refs": _dedupe(metadata.get("support_path_refs", []) or []),
- "promotion_state": _clean_text(metadata.get("promotion_state", "")) or "none",
- "memory_buffer_state": buffer_state,
- }
- memory_id = f"{slot_key}:{turn_index}:{index}"
- default_source_kind = "suspect_memory" if buffer_state == "suspect" else "replacement_memory"
- default_state = "suspect" if buffer_state == "suspect" else ("active" if bool(raw.get("active", True)) else "historical")
- records.append(
- SessionMemoryRecordV2(
- memory_id=memory_id,
- category=category,
- slot_key=slot_key,
- value=value,
- relation=_clean_text(raw.get("relation", "")) or f"{category}_memory",
- anchor_concepts=anchors,
- evidence_anchors=anchors,
- salience=float(raw.get("salience", 0.88 if category in {"goal", "constraint"} else 0.74) or 0.74),
- confidence=float(raw.get("confidence", 0.82) or 0.82),
- source_kind=_clean_text(raw.get("source_kind", "")) or default_source_kind,
- turn_index=int(raw.get("turn_index", turn_index) or turn_index),
- state=_clean_text(raw.get("state", "")) or default_state,
- metadata=metadata,
- )
- )
- return records
-
-
-_WRITE_MARKERS = (
- "goal update",
- "goal seed",
- "constraint update",
- "constraint overwrite",
- "constraint seed",
- "preference update",
- "preference overwrite",
- "preference seed",
- "terminology:",
- "term seed",
- "term overwrite",
- "stage update",
- "stage overwrite",
- "path fact",
- "fact:",
- "memory update",
-)
-
-
-_OVERWRITE_MARKERS = (
- "overwrite",
- "replace",
- "supersede",
- "覆盖",
- "替换",
- "改成",
- "更新为",
-)
-
-
-_TOPIC_BUCKET_CUE_GROUPS: tuple[tuple[tuple[str, ...], tuple[str, ...]], ...] = (
- (
- (
- "食物",
- "饮品",
- "甜点",
- "早餐",
- "餐饮",
- "口味",
- "酸辣",
- "热食",
- "过敏",
- "芒果",
- "配料",
- "推荐",
- "吃",
- "food",
- "breakfast",
- "allergy",
- "mango",
- ),
- ("餐饮偏好安全", "食物偏好", "过敏约束", "口味偏好"),
- ),
- (
- (
- "界面",
- "首页",
- "首屏",
- "配色",
- "营销",
- "横幅",
- "电商",
- "网站",
- "移动端",
- "视觉",
- "工具型",
- "ui",
- "homepage",
- "ecommerce",
- "mobile",
- "design",
- ),
- ("界面产品设计", "电商页面", "视觉布局", "移动端体验"),
- ),
- (
- (
- "api",
- "writer",
- "延迟",
- "算法",
- "耗时",
- "评估",
- "指标",
- "调用",
- "服务",
- "key",
- "runtime",
- "latency",
- "metric",
- ),
- ("api评估运行", "算法服务", "调用指标", "writer延迟"),
- ),
-)
-
-
-_TOPIC_BUCKET_STOPWORDS = {
- "不要",
- "不用",
- "复述",
- "原话",
- "只说",
- "关键",
- "最关键",
- "以后",
- "需要",
- "必须",
- "可以",
- "时候",
- "这个",
- "那个",
- "现在",
- "做",
- "说",
- "的",
- "了",
- "和",
- "与",
- "to",
- "the",
- "and",
- "for",
-}
-
-
-def _topic_bucket_keywords(text: str, *, max_items: int = 18) -> List[str]:
- normalized = _normalize(text)
- tokens: List[str] = []
- for cues, anchors in _TOPIC_BUCKET_CUE_GROUPS:
- if any(cue and _normalize(cue) in normalized for cue in cues):
- tokens.extend(anchors)
- tokens.extend(cue for cue in cues if len(cue) >= 2)
- tokens.extend(re.findall(r"[\u4e00-\u9fff]{2,8}", str(text or "")))
- tokens.extend(_tokenize(text))
- cleaned = []
- for token in tokens:
- value = _clean_text(token)
- if not value:
- continue
- normalized_value = _normalize(value)
- if normalized_value in _TOPIC_BUCKET_STOPWORDS:
- continue
- if len(value) < 2:
- continue
- cleaned.append(value)
- return _dedupe(cleaned, max_items=max_items)
-
-
-def _topic_bucket_id_from_keywords(keywords: Sequence[str]) -> str:
- basis = "|".join(_normalize(item) for item in keywords[:8] if _clean_text(item))
- if not basis:
- basis = "general"
- return "topic-" + str(uuid.uuid5(uuid.NAMESPACE_URL, f"tmcra-topic-bucket:{basis}"))[:12]
-
-
-def _topic_bucket_label_from_keywords(keywords: Sequence[str]) -> str:
- visible = [
- _clean_text(item)
- for item in keywords
- if _clean_text(item) and not _normalize(item).startswith("topic:")
- ]
- return " / ".join(visible[:3]) if visible else "动态话题"
-
-
-def _topic_bucket_overlap_score(left_keywords: Sequence[str], right_keywords: Sequence[str]) -> float:
- left = {_normalize(item) for item in left_keywords if _clean_text(item)}
- right = {_normalize(item) for item in right_keywords if _clean_text(item)}
- if not left or not right:
- return 0.0
- overlap = left & right
- if not overlap:
- return 0.0
- return len(overlap) / max(1, min(len(left), len(right)))
-
-
-def _explicit_topic_bucket_from_payload(answer_payload: Dict[str, Any] | None) -> Dict[str, Any]:
- metadata = dict((answer_payload or {}).get("metadata", {}) or {})
- explicit = metadata.get("topic_bucket")
- if not isinstance(explicit, Mapping):
- return {}
- keywords = _dedupe(explicit.get("topic_keywords", explicit.get("keywords", [])) or [], max_items=24)
- label = _clean_text(explicit.get("topic_label", explicit.get("label", "")))
- if not keywords and label:
- keywords = _topic_bucket_keywords(label, max_items=12)
- bucket_id = _clean_text(explicit.get("topic_bucket_id", explicit.get("id", "")))
- if not bucket_id:
- bucket_id = _topic_bucket_id_from_keywords(keywords or [label])
- return {
- "topic_bucket_id": bucket_id,
- "topic_label": label or _topic_bucket_label_from_keywords(keywords),
- "topic_keywords": keywords,
- "topic_confidence": float(explicit.get("confidence", explicit.get("topic_confidence", 0.92)) or 0.92),
- "topic_assignment_source": "explicit_payload",
- }
-
-
-def _coerce_topic_bucket(metadata: Mapping[str, Any]) -> Dict[str, Any]:
- bucket_id = _clean_text(metadata.get("topic_bucket_id", ""))
- if not bucket_id:
- nested = metadata.get("topic_bucket")
- if isinstance(nested, Mapping):
- bucket_id = _clean_text(nested.get("topic_bucket_id", nested.get("id", "")))
- if not bucket_id:
- return {}
- keywords = _dedupe(metadata.get("topic_keywords", []) or [], max_items=32)
- nested = metadata.get("topic_bucket")
- if isinstance(nested, Mapping):
- keywords = _dedupe([*keywords, *(nested.get("topic_keywords", nested.get("keywords", [])) or [])], max_items=32)
- label = _clean_text(metadata.get("topic_label", ""))
- if not label and isinstance(nested, Mapping):
- label = _clean_text(nested.get("topic_label", nested.get("label", "")))
- return {
- "topic_bucket_id": bucket_id,
- "topic_label": label or _topic_bucket_label_from_keywords(keywords),
- "topic_keywords": keywords,
- }
-
-
-def _collect_topic_buckets(graph: SessionMemoryGraphV2) -> Dict[str, Dict[str, Any]]:
- buckets: Dict[str, Dict[str, Any]] = {}
-
- def merge(bucket: Mapping[str, Any], *, record_id: str = "", turn_index: int = 0) -> None:
- bucket_id = _clean_text(bucket.get("topic_bucket_id", bucket.get("id", "")))
- if not bucket_id:
- return
- current = buckets.setdefault(
- bucket_id,
- {
- "topic_bucket_id": bucket_id,
- "topic_label": _clean_text(bucket.get("topic_label", bucket.get("label", ""))),
- "topic_keywords": [],
- "record_ids": [],
- "last_turn_index": 0,
- },
- )
- current["topic_label"] = current.get("topic_label") or _clean_text(bucket.get("topic_label", bucket.get("label", "")))
- current["topic_keywords"] = _dedupe(
- [*list(current.get("topic_keywords", []) or []), *(bucket.get("topic_keywords", bucket.get("keywords", [])) or [])],
- max_items=32,
- )
- if record_id:
- current["record_ids"] = _dedupe([*list(current.get("record_ids", []) or []), record_id], max_items=200)
- current["last_turn_index"] = max(int(current.get("last_turn_index", 0) or 0), int(turn_index or 0))
-
- for record in getattr(graph, "records_by_id", {}).values():
- metadata = dict(record.metadata or {})
- bucket = _coerce_topic_bucket(metadata)
- if bucket:
- merge(bucket, record_id=record.memory_id, turn_index=int(record.turn_index))
- for turn in list(getattr(graph, "turn_log", []) or []):
- metadata = dict(turn.get("metadata", {}) if isinstance(turn, Mapping) else getattr(turn, "metadata", {}) or {})
- bucket = _coerce_topic_bucket(metadata)
- if bucket:
- merge(bucket, turn_index=int(turn.get("turn_index", 0) if isinstance(turn, Mapping) else getattr(turn, "turn_index", 0) or 0))
- for bucket in buckets.values():
- if not bucket.get("topic_label"):
- bucket["topic_label"] = _topic_bucket_label_from_keywords(bucket.get("topic_keywords", []) or [])
- return buckets
-
-
-def _assign_topic_bucket_for_text(
- graph: SessionMemoryGraphV2,
- text: str,
- *,
- answer_payload: Dict[str, Any] | None = None,
- turn_index: int = 0,
- create: bool = True,
-) -> Dict[str, Any]:
- explicit = _explicit_topic_bucket_from_payload(answer_payload)
- if explicit:
- explicit["turn_index"] = int(turn_index or 0)
- return explicit
- keywords = _topic_bucket_keywords(text, max_items=24)
- if not keywords:
- keywords = _dedupe([_clean_text(text)[:40]], max_items=1)
- buckets = _collect_topic_buckets(graph)
- best_bucket: Dict[str, Any] | None = None
- best_score = 0.0
- for bucket in buckets.values():
- score = _topic_bucket_overlap_score(keywords, bucket.get("topic_keywords", []) or [])
- if score > best_score:
- best_bucket = bucket
- best_score = score
- if best_bucket and best_score >= 0.22:
- merged_keywords = _dedupe([*list(best_bucket.get("topic_keywords", []) or []), *keywords], max_items=32)
- return {
- "topic_bucket_id": best_bucket["topic_bucket_id"],
- "topic_label": best_bucket.get("topic_label") or _topic_bucket_label_from_keywords(merged_keywords),
- "topic_keywords": merged_keywords,
- "topic_confidence": round(min(0.97, 0.68 + best_score * 0.24), 6),
- "topic_assignment_source": "reused_by_overlap",
- "topic_match_score": round(best_score, 6),
- "turn_index": int(turn_index or 0),
- }
- bucket_id = _topic_bucket_id_from_keywords(keywords)
- return {
- "topic_bucket_id": bucket_id,
- "topic_label": _topic_bucket_label_from_keywords(keywords),
- "topic_keywords": keywords,
- "topic_confidence": 0.72 if create else 0.58,
- "topic_assignment_source": "created_by_dialog" if create else "query_probe",
- "topic_match_score": round(best_score, 6),
- "turn_index": int(turn_index or 0),
- }
-
-
-def _apply_topic_bucket_to_records(records: List[SessionMemoryRecordV2], topic_bucket: Mapping[str, Any]) -> None:
- if not records or not topic_bucket:
- return
- bucket_id = _clean_text(topic_bucket.get("topic_bucket_id", ""))
- if not bucket_id:
- return
- label = _clean_text(topic_bucket.get("topic_label", "")) or "动态话题"
- keywords = _dedupe(topic_bucket.get("topic_keywords", []) or [], max_items=32)
- for record in records:
- metadata = dict(record.metadata or {})
- metadata.update(
- {
- "topic_bucket_id": bucket_id,
- "topic_label": label,
- "topic_keywords": keywords,
- "topic_confidence": float(topic_bucket.get("topic_confidence", 0.72) or 0.72),
- "topic_assignment_source": _clean_text(topic_bucket.get("topic_assignment_source", "")) or "created_by_dialog",
- }
- )
- record.metadata = metadata
- record.anchor_concepts = _dedupe([*list(record.anchor_concepts or []), f"topic:{label}", *keywords[:8]], max_items=24)
- evidence_anchors = list(metadata.get("evidence_anchors", []) or [])
- metadata["evidence_anchors"] = _dedupe([*evidence_anchors, f"topic:{label}", *keywords[:8]], max_items=24)
-
-
-def _last_topic_turn(graph: SessionMemoryGraphV2) -> Dict[str, Any]:
- for turn in reversed(list(getattr(graph, "turn_log", []) or [])):
- metadata = dict(turn.get("metadata", {}) if isinstance(turn, Mapping) else getattr(turn, "metadata", {}) or {})
- bucket = _coerce_topic_bucket(metadata)
- if bucket:
- bucket["turn_index"] = int(turn.get("turn_index", 0) if isinstance(turn, Mapping) else getattr(turn, "turn_index", 0) or 0)
- return bucket
- return {}
-
-
-def _add_topic_bridge_edges(
- graph: SessionMemoryGraphV2,
- *,
- previous_topic: Mapping[str, Any],
- current_topic: Mapping[str, Any],
- current_record_ids: Sequence[str],
- turn_index: int,
- evidence: str,
-) -> Dict[str, Any]:
- previous_id = _clean_text(previous_topic.get("topic_bucket_id", ""))
- current_id = _clean_text(current_topic.get("topic_bucket_id", ""))
- if not previous_id or not current_id or previous_id == current_id or not current_record_ids:
- return {"topic_bridge_edge_count": 0}
- buckets = _collect_topic_buckets(graph)
- previous_record_ids = list((buckets.get(previous_id, {}) or {}).get("record_ids", []) or [])[-4:]
- if not previous_record_ids:
- return {"topic_bridge_edge_count": 0}
- edge_count = 0
- evidence_text = _clean_text(evidence)[:240]
- for source_id in previous_record_ids:
- for target_id in list(current_record_ids)[:4]:
- if source_id == target_id:
- continue
- edge = SessionMemoryEdgeV2(
- edge_id=f"{source_id}->{target_id}:topic_bridge:{previous_id}->{current_id}",
- source_memory_id=source_id,
- target_memory_id=target_id,
- edge_type="topic_bridge",
- score=0.54,
- model_score=0.0,
- evidence_turn=int(turn_index or 0),
- evidence=evidence_text,
- metadata={
- "from_topic_bucket_id": previous_id,
- "to_topic_bucket_id": current_id,
- "from_topic_label": _clean_text(previous_topic.get("topic_label", "")),
- "to_topic_label": _clean_text(current_topic.get("topic_label", "")),
- "bridge_reason": "adjacent_dialog_topic_transition",
- },
- )
- graph._upsert_memory_edge(edge)
- edge_count += 1
- return {
- "topic_bridge_edge_count": edge_count,
- "topic_bridge_from": previous_id,
- "topic_bridge_to": current_id,
- }
-
-
-def _add_dialogue_tunnel_edges(
- graph: SessionMemoryGraphV2,
- *,
- current_topic: Mapping[str, Any],
- current_record_ids: Sequence[str],
- turn_index: int,
- evidence: str,
-) -> Dict[str, Any]:
- current_id = _clean_text(current_topic.get("topic_bucket_id", ""))
- if not current_id or not current_record_ids:
- return {"dialogue_tunnel_edge_count": 0}
- buckets = _collect_topic_buckets(graph)
- edge_count = 0
- evidence_text = _clean_text(evidence)[:240]
- source_ids: List[tuple[str, str, str]] = []
- for bucket_id, bucket in sorted(
- buckets.items(),
- key=lambda item: int(item[1].get("last_turn_index", 0) or 0),
- reverse=True,
- ):
- if bucket_id == current_id:
- continue
- for source_id in list(bucket.get("record_ids", []) or [])[-2:]:
- if source_id not in current_record_ids:
- source_ids.append((bucket_id, _clean_text(bucket.get("topic_label", "")), source_id))
- if len(source_ids) >= 6:
- break
- for source_bucket_id, source_label, source_id in source_ids[:6]:
- for target_id in list(current_record_ids)[:2]:
- if source_id == target_id:
- continue
- edge = SessionMemoryEdgeV2(
- edge_id=f"{source_id}->{target_id}:dialogue_tunnel:{source_bucket_id}->{current_id}",
- source_memory_id=source_id,
- target_memory_id=target_id,
- edge_type="dialogue_tunnel",
- score=0.24,
- model_score=0.0,
- evidence_turn=int(turn_index or 0),
- evidence=evidence_text,
- metadata={
- "from_topic_bucket_id": source_bucket_id,
- "to_topic_bucket_id": current_id,
- "from_topic_label": source_label,
- "to_topic_label": _clean_text(current_topic.get("topic_label", "")),
- "bridge_reason": "high_resistance_dialogue_level_tunnel",
- },
- )
- graph._upsert_memory_edge(edge)
- edge_count += 1
- return {"dialogue_tunnel_edge_count": edge_count}
-
-
-def _topic_adjacent_bucket_ids(graph: SessionMemoryGraphV2, bucket_id: str) -> set[str]:
- adjacent: set[str] = set()
- if not bucket_id:
- return adjacent
- for edge in getattr(graph, "memory_edges", {}).values():
- if _normalize(edge.edge_type) != "topic_bridge":
- continue
- metadata = dict(edge.metadata or {})
- left = _clean_text(metadata.get("from_topic_bucket_id", ""))
- right = _clean_text(metadata.get("to_topic_bucket_id", ""))
- if left == bucket_id and right:
- adjacent.add(right)
- if right == bucket_id and left:
- adjacent.add(left)
- return adjacent
-
-
-def _dialogue_tunnel_bucket_ids(graph: SessionMemoryGraphV2, bucket_id: str) -> set[str]:
- adjacent: set[str] = set()
- if not bucket_id:
- return adjacent
- for edge in getattr(graph, "memory_edges", {}).values():
- if _normalize(edge.edge_type) != "dialogue_tunnel":
- continue
- metadata = dict(edge.metadata or {})
- left = _clean_text(metadata.get("from_topic_bucket_id", ""))
- right = _clean_text(metadata.get("to_topic_bucket_id", ""))
- if left == bucket_id and right:
- adjacent.add(right)
- if right == bucket_id and left:
- adjacent.add(left)
- return adjacent
-
-
-def _topic_bridge_requested(query: str) -> bool:
- text = _normalize(query)
- if not text:
- return False
- bridge_markers = (
- "关联",
- "联系",
- "链条",
- "脉络",
- "延展",
- "深入",
- "对比",
- "整合",
- "整体",
- "上下文",
- "刚才",
- "之前",
- "上面",
- "跨话题",
- "隧穿",
- "related",
- "connect",
- "compare",
- "context",
- "chain",
- )
- return any(marker in text for marker in bridge_markers)
-
-
-def _dialogue_tunnel_requested(query: str) -> bool:
- text = _normalize(query)
- if not text:
- return False
- markers = (
- "跨话题",
- "跨对话",
- "不同话题",
- "不同对话",
- "历史对话",
- "长期脉络",
- "整体脉络",
- "所有相关",
- "全局",
- "全局记忆",
- "远一点",
- "更深",
- "深层关联",
- "对话级隧穿",
- "记忆隧穿",
- "cross topic",
- "cross-topic",
- "cross dialogue",
- "cross-session",
- "global context",
- "long range",
- )
- return any(marker in text for marker in markers)
-
-
-def _topic_bucket_record_to_hit(
- record: SessionMemoryRecordV2,
- *,
- query_topic: Mapping[str, Any],
- rank: int,
- rescue_kind: str = "topic_bucket",
-) -> MemoryHit:
- metadata = dict(record.metadata or {})
- dialogue_tunnel = _normalize(rescue_kind) == "dialogue_tunnel"
- metadata.update(
- {
- "topic_bucket_rescue": not dialogue_tunnel,
- "dialogue_tunnel_rescue": dialogue_tunnel,
- "topic_bucket_rescue_rank": int(rank),
- "topic_bucket_query_id": _clean_text(query_topic.get("topic_bucket_id", "")),
- "topic_bucket_query_label": _clean_text(query_topic.get("topic_label", "")),
- "topic_bucket_same": not dialogue_tunnel,
- "topic_bucket_bridge": False,
- "topic_bucket_bridge_allowed": False,
- "topic_bucket_dialogue_tunnel_allowed": dialogue_tunnel,
- "topic_bucket_overlap": 0.0 if dialogue_tunnel else 1.0,
- }
- )
- category = _normalize(record.category)
- value_text = _normalize(record.value)
- hardish = (
- category == "constraint"
- or _normalize(metadata.get("memory_type", "")) == "hard_constraint"
- or _normalize(metadata.get("durability", "")) == "hard"
- or _normalize(metadata.get("conflict_policy", "")) == "must_preserve"
- or any(marker in value_text for marker in ("过敏", "必须", "避开", "禁止", "不能", "must", "avoid", "allergy"))
- )
- base_score = max(float(record.confidence), float(record.salience), 0.62)
- if dialogue_tunnel:
- if hardish:
- base_score += 0.42
- metadata.setdefault("memory_type", "hard_constraint")
- metadata.setdefault("durability", "hard")
- metadata.setdefault("conflict_policy", "must_preserve")
- elif category == "preference":
- base_score += 0.22
- metadata.setdefault("memory_type", "durable_preference")
- metadata.setdefault("durability", "long_term")
- else:
- base_score += 0.12
- metadata["dialogue_tunnel_resistance"] = "high"
- elif hardish:
- base_score += 1.35
- metadata.setdefault("memory_type", "hard_constraint")
- metadata.setdefault("durability", "hard")
- metadata.setdefault("conflict_policy", "must_preserve")
- elif category == "preference":
- base_score += 0.72
- metadata.setdefault("memory_type", "durable_preference")
- metadata.setdefault("durability", "long_term")
- else:
- base_score += 0.38
- return MemoryHit(
- memory_id=record.memory_id,
- category=record.category,
- value=record.value,
- relation=record.relation,
- anchors=list(record.anchor_concepts),
- score=base_score,
- source_kind=record.source_kind,
- slot_key=record.slot_key,
- state=record.state,
- turn_index=int(record.turn_index),
- metadata=metadata,
- )
-
-
-def _profile_query_rescue_hits(
- graph: SessionMemoryGraphV2,
- query: str,
- *,
- top_k: int,
-) -> List[MemoryHit]:
- query_raw_tokens = set(_path_utility_tokens(query))
- query_tokens = _profile_query_expanded_tokens(query)
- intent = infer_profile_query_intent(query)
- if not bool(intent.get("enabled")):
- return []
- rescued: List[tuple[float, MemoryHit]] = []
- for record in getattr(graph, "records_by_id", {}).values():
- metadata = dict(record.metadata or {})
- if record.state != "active":
- continue
- if not is_profile_layer_record(
- category=record.category,
- source_kind=record.source_kind,
- semantic_slot=metadata.get("semantic_slot", ""),
- metadata=metadata,
- ):
- continue
- delta, reason = profile_query_score_delta(
- query=query,
- query_tokens=query_tokens,
- category=record.category,
- source_kind=record.source_kind,
- semantic_slot=metadata.get("semantic_slot", ""),
- value=record.value,
- anchors=record.anchor_concepts,
- metadata=metadata,
- )
- if delta <= 0:
- continue
- match_score, overlap_tokens, raw_overlap_tokens = _profile_hit_match_score(
- query_raw_tokens,
- query_tokens,
- MemoryHit(
- memory_id=record.memory_id,
- category=record.category,
- value=record.value,
- relation=record.relation,
- anchors=list(record.anchor_concepts),
- score=max(float(record.confidence), float(record.salience), 0.01),
- source_kind=record.source_kind,
- slot_key=record.slot_key,
- state=record.state,
- turn_index=int(record.turn_index),
- metadata=metadata,
- ),
- )
- if match_score <= 0.0:
- continue
- if match_score < 0.34 and not raw_overlap_tokens:
- continue
- metadata.update(
- {
- "profile_query_rescue": True,
- "profile_query_rescue_reason": reason or "profile_route",
- "profile_query_match_score": round(match_score, 6),
- "profile_query_overlap_tokens": list(overlap_tokens),
- "profile_query_raw_overlap_tokens": list(raw_overlap_tokens),
- "topic_bucket_profile_route_preserved": True,
- "match_reason": ",".join(_dedupe([metadata.get("match_reason", ""), reason or "profile_route"], max_items=4)),
- }
- )
- hit = MemoryHit(
- memory_id=record.memory_id,
- category=record.category,
- value=record.value,
- relation=record.relation,
- anchors=list(record.anchor_concepts),
- score=max(float(record.confidence), float(record.salience), 0.62) + float(delta) + float(match_score),
- source_kind=record.source_kind,
- slot_key=record.slot_key,
- state=record.state,
- turn_index=int(record.turn_index),
- metadata=metadata,
- )
- rescued.append((match_score, hit))
- rescued.sort(key=lambda item: (float(item[0]), float(item[1].score), int(item[1].turn_index)), reverse=True)
- return [
- hit
- for _, hit in rescued[: max(1, min(24, int(top_k or 1) * 3))]
- ]
-
-
-def _memory_hit_from_record(record: SessionMemoryRecordV2, *, score: float | None = None, metadata: Mapping[str, Any] | None = None) -> MemoryHit:
- record_metadata = {**dict(record.metadata or {}), **dict(metadata or {})}
- return MemoryHit(
- memory_id=record.memory_id,
- category=record.category,
- value=record.value,
- relation=record.relation,
- anchors=list(record.anchor_concepts),
- score=max(float(record.confidence), float(record.salience), 0.01) if score is None else float(score),
- source_kind=record.source_kind,
- slot_key=record.slot_key,
- state=record.state,
- turn_index=int(record.turn_index),
- metadata=record_metadata,
- )
-
-
-_FACET_NUMERIC_QUERY_TOKENS = {
- "amount",
- "count",
- "counts",
- "duration",
- "durations",
- "many",
- "much",
- "number",
- "quantity",
- "sum",
- "total",
- "totals",
- "weeks",
- "week",
- "hours",
- "hour",
- "dollars",
- "dollar",
- "tenants",
- "tickets",
-}
-_FACET_TEMPORAL_QUERY_TOKENS = {
- "after",
- "before",
- "date",
- "deadline",
- "end",
- "finish",
- "finished",
- "start",
- "started",
- "time",
- "when",
-}
-
-
-def _facet_query_pack_hits(
- graph: SessionMemoryGraphV2,
- query: str,
- final_hits: Sequence[MemoryHit],
- *,
- top_k: int,
-) -> Dict[str, Any]:
- query_tokens = set(_path_utility_tokens(query))
- if not query_tokens:
- return {"hits": list(final_hits), "metadata": {"facet_query_pack_enabled": False, "facet_query_pack_reason": "empty_query_tokens"}}
- numeric_query = bool(query_tokens & _FACET_NUMERIC_QUERY_TOKENS)
- temporal_query = bool(query_tokens & _FACET_TEMPORAL_QUERY_TOKENS)
- if not numeric_query and not temporal_query and not any("facet" in _normalize(token) for token in query_tokens):
- return {"hits": list(final_hits), "metadata": {"facet_query_pack_enabled": False, "facet_query_pack_reason": "no_facet_intent"}}
-
- candidate_rows: List[tuple[float, SessionMemoryRecordV2, SessionMemoryRecordV2 | None, Dict[str, Any]]] = []
- for record in getattr(graph, "records_by_id", {}).values():
- metadata = dict(record.metadata or {})
- if _normalize(metadata.get("content_variant", "")) != "event_facet_write":
- continue
- if record.state not in {"active", "parallel_active", "evidence"}:
- continue
- facet_type = _normalize(metadata.get("facet_type", ""))
- parent_slot_key = _clean_text(metadata.get("facet_parent_slot_key", ""))
- parent = next(
- (
- candidate
- for candidate in getattr(graph, "records_by_id", {}).values()
- if _clean_text(candidate.slot_key).lower() == parent_slot_key.lower()
- ),
- None,
- )
- parent_text = " ".join(
- [
- _clean_text(parent.value if parent else ""),
- " ".join(parent.anchor_concepts if parent else []),
- _clean_text(dict(parent.metadata or {}).get("source_span", "") if parent else ""),
- ]
- )
- facet_text = " ".join(
- [
- record.value,
- " ".join(record.anchor_concepts or []),
- _clean_text(metadata.get("facet_type", "")),
- _clean_text(metadata.get("facet_role", "")),
- _clean_text(metadata.get("facet_source_span", "")),
- parent_text,
- ]
- )
- facet_tokens = set(_path_utility_tokens(facet_text))
- parent_tokens = set(_path_utility_tokens(parent_text))
- overlap_tokens = query_tokens & facet_tokens
- parent_overlap_tokens = query_tokens & parent_tokens
- unit_overlap = bool(query_tokens & set(_path_utility_tokens(record.value)))
- score = 0.0
- if overlap_tokens:
- score += min(0.72, len(overlap_tokens) / max(1.0, len(query_tokens)) * 1.2)
- if parent_overlap_tokens:
- score += min(0.72, len(parent_overlap_tokens) / max(1.0, len(query_tokens)) * 1.35)
- if numeric_query and facet_type == "numeric":
- score += 0.42
- if unit_overlap:
- score += 0.42
- if temporal_query and facet_type == "temporal":
- score += 0.32
- if facet_type == "entity" and parent_overlap_tokens:
- score += 0.26
- if parent is not None and _normalize(dict(parent.metadata or {}).get("content_variant", "")) == "llm_semantic_write":
- score += 0.08
- if score < 0.58:
- continue
- candidate_rows.append(
- (
- round(min(2.75, 1.18 + score), 6),
- record,
- parent,
- {
- "facet_query_pack_overlap_tokens": sorted(overlap_tokens)[:12],
- "facet_query_pack_parent_overlap_tokens": sorted(parent_overlap_tokens)[:12],
- "facet_query_pack_unit_overlap": bool(unit_overlap),
- "facet_query_pack_score": round(score, 6),
- },
- )
- )
-
- if not candidate_rows:
- return {
- "hits": list(final_hits),
- "metadata": {
- "facet_query_pack_enabled": True,
- "facet_query_pack_inserted_hit_count": 0,
- "facet_query_pack_candidate_count": 0,
- },
- }
-
- candidate_rows.sort(key=lambda item: (float(item[0]), int(item[1].turn_index)), reverse=True)
- selected = candidate_rows[: max(4, min(18, int(top_k or 1) * 2))]
- packed_hits: List[MemoryHit] = []
- for score, facet_record, parent, extra_metadata in selected:
- facet_metadata = {
- **extra_metadata,
- "facet_query_pack": True,
- "evidence_snippet_role": "facet_query_attribute",
- }
- packed_hits.append(_memory_hit_from_record(facet_record, score=score, metadata=facet_metadata))
- if parent is not None:
- packed_hits.append(
- _memory_hit_from_record(
- parent,
- score=max(1.12, score - 0.04),
- metadata={
- **extra_metadata,
- "facet_query_pack": True,
- "evidence_snippet_role": "facet_parent_event",
- "facet_query_pack_child_id": facet_record.memory_id,
- },
- )
- )
-
- merged: List[MemoryHit] = []
- seen_ids: set[str] = set()
- for hit in [*packed_hits, *list(final_hits)]:
- if hit.memory_id and hit.memory_id in seen_ids:
- continue
- if hit.memory_id:
- seen_ids.add(hit.memory_id)
- merged.append(hit)
- return {
- "hits": merged,
- "metadata": {
- "facet_query_pack_enabled": True,
- "facet_query_pack_candidate_count": len(candidate_rows),
- "facet_query_pack_inserted_hit_count": len(packed_hits),
- "facet_query_pack_numeric_query": bool(numeric_query),
- "facet_query_pack_temporal_query": bool(temporal_query),
- },
- }
-
-
-_UNIT_COVERAGE_COUNT_TOKENS = {
- "amount",
- "amounts",
- "count",
- "counts",
- "cost",
- "costs",
- "dollar",
- "dollars",
- "how",
- "many",
- "much",
- "number",
- "minimum",
- "maximum",
- "paid",
- "percent",
- "percentage",
- "price",
- "prices",
- "sale",
- "sales",
- "sell",
- "sold",
- "total",
- "totals",
- "sum",
- "value",
- "valued",
- "values",
- "worth",
- "items",
- "projects",
- "events",
- "things",
-}
-_UNIT_COVERAGE_TEMPORAL_TOKENS = {
- "ago",
- "after",
- "before",
- "between",
- "consecutive",
- "date",
- "days",
- "day",
- "first",
- "last",
- "months",
- "month",
- "order",
- "passed",
- "since",
- "weeks",
- "week",
-}
-_MULTI_UNIT_CHAIN_TEMPORAL_COMPARISON_TOKENS = {
- "after",
- "before",
- "between",
- "consecutive",
- "earlier",
- "first",
- "later",
- "last",
- "order",
- "since",
-}
-_UNIT_COVERAGE_QUERY_DROP_TOKENS = {
- "date",
- "fri",
- "friday",
- "mon",
- "monday",
- "question",
- "sat",
- "saturday",
- "sun",
- "sunday",
- "thu",
- "thursday",
- "tue",
- "tuesday",
- "wed",
- "wednesday",
-}
-_MULTI_UNIT_CHAIN_DISABLED_MODES = {"", "off", "disabled", "none", "false", "0"}
-_MULTI_UNIT_CHAIN_COUNT_TOKENS = {
- *_UNIT_COVERAGE_COUNT_TOKENS,
- "which",
- "each",
- "all",
- "both",
-}
-_MULTI_UNIT_CHAIN_FOCUS_DROP_TOKENS = {
- *_MULTI_UNIT_CHAIN_COUNT_TOKENS,
- *_UNIT_COVERAGE_TEMPORAL_TOKENS,
- "i",
- "me",
- "my",
- "mine",
- "am",
- "is",
- "are",
- "was",
- "were",
- "need",
- "needs",
- "needed",
- "currently",
- "did",
- "does",
- "fri",
- "friday",
- "mon",
- "monday",
- "or",
- "question",
- "sat",
- "saturday",
- "sun",
- "sunday",
- "thu",
- "thursday",
- "tue",
- "tuesday",
- "wed",
- "wednesday",
- "what",
- "which",
- "who",
-}
-_MULTI_UNIT_CHAIN_FACET_TYPES = {"action", "entity", "numeric", "role", "evidence_role", "state", "temporal"}
-_MULTI_UNIT_CHAIN_UNIT_KINDS = {
- "action_unit",
- "target_entity",
- "numeric_quantity",
- "leadership_role",
- "participation_role",
- "state_status",
- "temporal_anchor",
- "evidence_role",
- "profile_shadow_unit",
-}
-_MULTI_UNIT_CHAIN_NUMERIC_VALUE_TOKENS = {
- "amount",
- "appraisal",
- "appraised",
- "cost",
- "costs",
- "dollar",
- "dollars",
- "minimum",
- "paid",
- "price",
- "prices",
- "sale",
- "sell",
- "sold",
- "total",
- "value",
- "valued",
- "values",
- "worth",
-}
-
-
-def _multi_unit_chain_numeric_signal(
- unit_kind: str,
- facet_type: str,
- text: str,
- tokens: set[str],
-) -> float:
- compact_text = _clean_text(text)
- date_like_only = bool(
- re.fullmatch(
- r"(?:\d{4}[/-]\d{1,2}(?:[/-]\d{1,2})?|\d{1,2}[/-]\d{1,2}(?:[/-]\d{2,4})?|\d{4}/\d{2})",
- compact_text,
- )
- )
- if date_like_only:
- return 0.0
- signal = 0.0
- if unit_kind == "numeric_quantity" or facet_type == "numeric":
- signal += 0.55
- if tokens & _MULTI_UNIT_CHAIN_NUMERIC_VALUE_TOKENS:
- signal += 0.28
- if re.search(r"(?:[$€£¥]\s*\d|\b\d+(?:,\d{3})*(?:\.\d+)?\s*(?:dollars?|usd|bucks?|yuan|rmb)\b)", text, flags=re.IGNORECASE):
- signal += 0.72
- elif re.search(r"\b\d+(?:,\d{3})*(?:\.\d+)?\s*(?:comments?|items?|pieces?|kits?|projects?|doctors?|weddings?|hours?|weeks?|days?|months?|years?|miles?)\b", text, flags=re.IGNORECASE):
- signal += 0.42
- return min(1.1, signal)
-
-
-def _multi_unit_chain_date_like_numeric(text: str) -> bool:
- compact_text = _clean_text(text)
- return bool(
- re.fullmatch(
- r"(?:\d{4}[/-]\d{1,2}(?:[/-]\d{1,2})?|\d{1,2}[/-]\d{1,2}(?:[/-]\d{2,4})?|\d{4}/\d{2})",
- compact_text,
- )
- )
-
-
-def _multi_unit_chain_temporal_anchor_signal(text: str, tokens: set[str]) -> float:
- signal = 0.0
- if tokens & _UNIT_COVERAGE_TEMPORAL_TOKENS:
- signal += 0.22
- if re.search(
- r"\b(?:about|around|roughly|few|several|couple|last|previous|earlier)\s+"
- r"(?:a\s+)?(?:day|days|week|weeks|month|months|year|years)\s+ago\b|"
- r"\b(?:yesterday|today|tomorrow|last\s+week|last\s+month|a\s+few\s+months\s+ago)\b",
- text,
- flags=re.IGNORECASE,
- ):
- signal += 0.72
- return min(1.0, signal)
-
-
-_PROFILE_SHADOW_EVENTLIKE_SLOT_HINTS = {
- "action",
- "appointment",
- "constraint",
- "deadline",
- "exchange",
- "obligation",
- "pickup",
- "plan",
- "preference",
- "return",
- "status",
- "task",
-}
-_PROFILE_SHADOW_EVENTLIKE_TEXT_HINTS = {
- "bought",
- "buy",
- "completed",
- "did",
- "exchange",
- "exchanged",
- "finish",
- "finished",
- "got",
- "have to",
- "need",
- "needed",
- "needs",
- "paid",
- "pick",
- "picked",
- "return",
- "returned",
- "should",
- "still",
- "took",
- "went",
-}
-
-
-def _profile_shadow_eventlike_record(record: SessionMemoryRecordV2, metadata: Mapping[str, Any]) -> bool:
- if _normalize(metadata.get("content_variant", "")) != "profile_shadow_from_writer":
- return False
- if record.state not in {"active", "parallel_active", "evidence"}:
- return False
- slot_text = " ".join(
- [
- _clean_text(record.category),
- _clean_text(record.relation),
- _clean_text(record.slot_key),
- _clean_text(metadata.get("semantic_slot", "")),
- _clean_text(metadata.get("profile_type", "")),
- ]
- ).lower()
- value_text = " ".join(
- [
- _clean_text(record.value),
- _clean_text(metadata.get("source_span", "")),
- _clean_text(metadata.get("raw_text", "")),
- ]
- ).lower()
- return bool(
- any(hint in slot_text for hint in _PROFILE_SHADOW_EVENTLIKE_SLOT_HINTS)
- or any(hint in value_text for hint in _PROFILE_SHADOW_EVENTLIKE_TEXT_HINTS)
- )
-
-
-def _profile_shadow_unit_text(record: SessionMemoryRecordV2, metadata: Mapping[str, Any]) -> str:
- return " ".join(
- [
- _clean_text(record.value),
- _clean_text(record.category),
- _clean_text(record.relation),
- _clean_text(record.slot_key),
- _clean_text(metadata.get("semantic_slot", "")),
- _clean_text(metadata.get("profile_type", "")),
- _clean_text(metadata.get("profile_domain", "")),
- _clean_text(metadata.get("source_span", "")),
- _clean_text(metadata.get("raw_text", "")),
- ]
- )
-
-
-def _unit_coverage_pack_hits(
- graph: SessionMemoryGraphV2,
- query: str,
- final_hits: Sequence[MemoryHit],
- *,
- top_k: int,
-) -> Dict[str, Any]:
- raw_query_tokens = set(_path_utility_tokens(query))
- query_tokens = {token for token in raw_query_tokens if token not in _UNIT_COVERAGE_QUERY_DROP_TOKENS}
- if not query_tokens:
- return {"hits": list(final_hits), "metadata": {"unit_coverage_pack_enabled": False, "unit_coverage_reason": "empty_query"}}
- count_intent = bool(query_tokens & _UNIT_COVERAGE_COUNT_TOKENS)
- percentage_intent = bool(query_tokens & {"percent", "percentage"})
- temporal_intent = bool(raw_query_tokens & _MULTI_UNIT_CHAIN_TEMPORAL_COMPARISON_TOKENS)
- direct_unit_intent = not count_intent and not temporal_intent and len(query_tokens) >= 2
- if not count_intent and not temporal_intent and not direct_unit_intent:
- return {
- "hits": list(final_hits),
- "metadata": {"unit_coverage_pack_enabled": False, "unit_coverage_reason": "no_unit_intent"},
- }
-
- records = list(getattr(graph, "records_by_id", {}).values())
- parent_by_slot = {_clean_text(record.slot_key).lower(): record for record in records}
- candidates: List[tuple[float, SessionMemoryRecordV2, SessionMemoryRecordV2 | None, Dict[str, Any]]] = []
- for record in records:
- metadata = dict(record.metadata or {})
- if record.state not in {"active", "parallel_active", "evidence"}:
- continue
- profile_shadow_unit = _profile_shadow_eventlike_record(record, metadata)
- if profile_shadow_unit:
- unit_kind = "profile_shadow_unit"
- facet_type = "action"
- parent_slot = _clean_text(record.slot_key).lower()
- parent = None
- parent_text = ""
- unit_text = _profile_shadow_unit_text(record, metadata)
- else:
- if _normalize(metadata.get("content_variant", "")) != "event_facet_write":
- continue
- if _normalize(metadata.get("facet_layer_version", "")) not in {"event_unit_v1", "event_facet_v1"}:
- continue
- unit_kind = _normalize(metadata.get("unit_kind", ""))
- facet_type = _normalize(metadata.get("facet_type", ""))
- parent_slot = _clean_text(metadata.get("facet_parent_slot_key", "")).lower()
- parent = parent_by_slot.get(parent_slot)
- parent_text = " ".join(
- [
- _clean_text(parent.value if parent else ""),
- _clean_text(dict(parent.metadata or {}).get("source_span", "") if parent else ""),
- " ".join(parent.anchor_concepts if parent else []),
- ]
- )
- unit_text = " ".join(
- [
- _clean_text(record.value),
- _clean_text(metadata.get("facet_role", "")),
- _clean_text(metadata.get("unit_kind", "")),
- _clean_text(metadata.get("action", "")),
- _clean_text(metadata.get("target", "")),
- _clean_text(metadata.get("quantity", "")),
- _clean_text(metadata.get("unit", "")),
- _clean_text(metadata.get("normalized_time", "")),
- _clean_text(metadata.get("status", "")),
- _clean_text(metadata.get("facet_source_span", "")),
- parent_text,
- ]
- )
- unit_tokens = set(_path_utility_tokens(unit_text))
- semantic_event_unit = _normalize(record.memory_id).startswith("tmcra.event.") or _normalize(parent_slot).startswith("tmcra.event.")
- overlap = query_tokens & unit_tokens
- score = 0.0
- if overlap:
- score += min(0.92, len(overlap) / max(1.0, len(query_tokens)) * 1.55)
- if count_intent and facet_type in {"action", "entity", "numeric", "role", "evidence_role"}:
- score += 0.42
- if temporal_intent and facet_type in {"temporal", "action", "role", "entity"}:
- score += 0.38
- if direct_unit_intent and facet_type in {"action", "entity", "state", "numeric"}:
- score += 0.30
- if direct_unit_intent and len(overlap) >= 2:
- score += 0.36
- if unit_kind in {"action_unit", "target_entity", "leadership_role", "participation_role"}:
- score += 0.16
- if unit_kind in {"numeric_quantity", "temporal_anchor", "state_status"}:
- score += 0.12
- if profile_shadow_unit:
- score += 0.28
- if count_intent and facet_type in {"action", "entity"}:
- score += 0.24
- semantic_event_priority = bool(semantic_event_unit and (count_intent or percentage_intent))
- if semantic_event_priority:
- score += 0.46
- if percentage_intent and (
- unit_kind == "numeric_quantity"
- or facet_type == "numeric"
- or re.search(r"\b\d+(?:\.\d+)?\s*%|\b\d+(?:\.\d+)?\s*percent\b", unit_text, flags=re.IGNORECASE)
- ):
- score += 0.72
- if parent is not None:
- score += 0.08
- if score < 0.55:
- continue
- candidates.append(
- (
- round(min(3.2, 1.28 + score), 6),
- record,
- parent,
- {
- "unit_coverage_overlap_tokens": sorted(overlap)[:14],
- "unit_coverage_score": round(score, 6),
- "unit_kind": unit_kind,
- "facet_type": facet_type,
- "unit_coverage_count_intent": bool(count_intent),
- "unit_coverage_percentage_intent": bool(percentage_intent),
- "unit_coverage_temporal_intent": bool(temporal_intent),
- "unit_coverage_direct_unit_intent": bool(direct_unit_intent),
- "unit_coverage_profile_shadow_unit": bool(profile_shadow_unit),
- "unit_coverage_semantic_event_unit": bool(semantic_event_unit),
- "unit_coverage_semantic_event_priority": bool(semantic_event_priority),
- },
- )
- )
-
- if not candidates:
- return {
- "hits": list(final_hits),
- "metadata": {
- "unit_coverage_pack_enabled": True,
- "unit_coverage_candidate_count": 0,
- "unit_coverage_inserted_hit_count": 0,
- "unit_coverage_count_intent": bool(count_intent),
- "unit_coverage_percentage_intent": bool(percentage_intent),
- "unit_coverage_temporal_intent": bool(temporal_intent),
- "unit_coverage_direct_unit_intent": bool(direct_unit_intent),
- },
- }
- candidates.sort(
- key=lambda item: (
- 1 if bool(item[3].get("unit_coverage_semantic_event_priority", False)) else 0,
- 1
- if bool(item[3].get("unit_coverage_percentage_intent"))
- and (
- _normalize(item[3].get("unit_kind", "")) == "numeric_quantity"
- or _normalize(item[3].get("facet_type", "")) == "numeric"
- )
- else 0,
- float(item[0]),
- int(item[1].turn_index),
- ),
- reverse=True,
- )
- selected: List[tuple[float, SessionMemoryRecordV2, SessionMemoryRecordV2 | None, Dict[str, Any]]] = []
- seen_unit_values: set[str] = set()
- try:
- max_selected_units = max(1, int(os.getenv("TMCRA_UNIT_COVERAGE_PACK_MAX_UNITS", "6") or 6))
- except (TypeError, ValueError):
- max_selected_units = 6
- for item in candidates:
- _, record, parent, metadata = item
- key = "|".join(
- [
- _normalize(metadata.get("unit_kind", "")),
- _normalize(record.value),
- _normalize(dict(record.metadata or {}).get("facet_parent_slot_key", "")),
- ]
- )
- if key in seen_unit_values:
- continue
- seen_unit_values.add(key)
- selected.append(item)
- if len(selected) >= max_selected_units:
- break
- if count_intent:
- try:
- max_profile_shadow_units = max(0, int(os.getenv("TMCRA_UNIT_COVERAGE_PROFILE_SHADOW_MAX_UNITS", "2") or 2))
- except (TypeError, ValueError):
- max_profile_shadow_units = 2
- selected_ids = {item[1].memory_id for item in selected}
- added_profile_shadow_units = 0
- for item in candidates:
- _, record, _parent, metadata = item
- if added_profile_shadow_units >= max_profile_shadow_units:
- break
- if record.memory_id in selected_ids:
- continue
- if not bool(metadata.get("unit_coverage_profile_shadow_unit", False)):
- continue
- selected.append(item)
- selected_ids.add(record.memory_id)
- added_profile_shadow_units += 1
- packed_hits: List[MemoryHit] = []
- for score, unit_record, parent, extra in selected:
- packed_hits.append(
- _memory_hit_from_record(
- unit_record,
- score=score,
- metadata={
- **extra,
- "unit_coverage_pack": True,
- "evidence_snippet_role": "unit_coverage_evidence_unit",
- },
- )
- )
- if parent is not None:
- unit_metadata = dict(unit_record.metadata or {})
- parent_hit = _memory_hit_from_record(
- parent,
- score=max(1.12, score - 0.06),
- metadata={
- **extra,
- "unit_coverage_pack": True,
- "evidence_snippet_role": "unit_coverage_parent_event",
- "unit_coverage_parent_memory_id": parent.memory_id,
- "unit_coverage_child_id": unit_record.memory_id,
- "unit_coverage_child_value": unit_record.value,
- "unit_coverage_child_source_span": _clean_text(
- unit_metadata.get("facet_source_span", "") or unit_metadata.get("source_span", "")
- ),
- },
- )
- parent_hit.memory_id = f"{parent.memory_id}#unit_parent:{unit_record.memory_id}"
- packed_hits.append(parent_hit)
- merged: List[MemoryHit] = []
- seen_ids: set[str] = set()
- try:
- insertion_index = max(0, min(len(final_hits), int(os.getenv("TMCRA_UNIT_COVERAGE_PACK_INSERT_AFTER", "2") or 2)))
- except (TypeError, ValueError):
- insertion_index = min(len(final_hits), 2)
- ordered_hits = [*list(final_hits[:insertion_index]), *packed_hits, *list(final_hits[insertion_index:])]
- for hit in ordered_hits:
- if hit.memory_id and hit.memory_id in seen_ids:
- continue
- if hit.memory_id:
- seen_ids.add(hit.memory_id)
- merged.append(hit)
- return {
- "hits": merged,
- "metadata": {
- "unit_coverage_pack_enabled": True,
- "unit_coverage_candidate_count": len(candidates),
- "unit_coverage_selected_unit_count": len(selected),
- "unit_coverage_inserted_hit_count": len(packed_hits),
- "unit_coverage_count_intent": bool(count_intent),
- "unit_coverage_percentage_intent": bool(percentage_intent),
- "unit_coverage_temporal_intent": bool(temporal_intent),
- "unit_coverage_direct_unit_intent": bool(direct_unit_intent),
- },
- }
-
-
-def _multi_unit_chain_focus_tokens(query: str) -> set[str]:
- tokens = set(_path_utility_tokens(query))
- return {
- normalized
- for token in tokens
- for normalized in [_multi_unit_chain_normalize_token(token)]
- if normalized and normalized not in _MULTI_UNIT_CHAIN_FOCUS_DROP_TOKENS and len(normalized) > 2
- }
-
-
-def _multi_unit_chain_normalize_token(token: str) -> str:
- raw = _normalize(token)
- if not raw:
- return ""
- aliases = {
- "bought": "buy",
- "buying": "buy",
- "purchased": "buy",
- "purchasing": "buy",
- "worked": "work",
- "working": "work",
- "started": "start",
- "starting": "start",
- "finished": "finish",
- "finishing": "finish",
- "completed": "complete",
- "completing": "complete",
- "returned": "return",
- "returning": "return",
- "met": "meet",
- "meeting": "meet",
- "picked": "pick",
- "picking": "pick",
- "led": "lead",
- "leading": "lead",
- "kits": "kit",
- "items": "item",
- "projects": "project",
- "events": "event",
- "clothes": "clothing",
- }
- if raw in aliases:
- return aliases[raw]
- if len(raw) > 4 and raw.endswith("ies"):
- return raw[:-3] + "y"
- if len(raw) > 4 and raw.endswith("ing"):
- stem = raw[:-3]
- if len(stem) > 3 and stem[-1] == stem[-2]:
- stem = stem[:-1]
- return stem
- if len(raw) > 3 and raw.endswith("ed"):
- stem = raw[:-2]
- if len(stem) > 3 and stem[-1] == stem[-2]:
- stem = stem[:-1]
- return stem
- if len(raw) > 3 and raw.endswith("s") and not raw.endswith("ss"):
- return raw[:-1]
- return raw
-
-
-def _multi_unit_chain_normalized_tokens(text: str) -> set[str]:
- return {
- normalized
- for token in _path_utility_tokens(text)
- for normalized in [_multi_unit_chain_normalize_token(token)]
- if normalized
- }
-
-
-def _multi_unit_chain_hit_text(record: SessionMemoryRecordV2, parent: SessionMemoryRecordV2 | None) -> str:
- metadata = dict(record.metadata or {})
- parent_metadata = dict(parent.metadata or {}) if parent is not None else {}
- return " ".join(
- [
- _clean_text(record.value),
- _clean_text(record.category),
- _clean_text(record.relation),
- _clean_text(metadata.get("facet_type", "")),
- _clean_text(metadata.get("unit_kind", "")),
- _clean_text(metadata.get("facet_role", "")),
- _clean_text(metadata.get("facet_value", "")),
- _clean_text(metadata.get("facet_source_span", "")),
- _clean_text(metadata.get("action", "")),
- _clean_text(metadata.get("target", "")),
- _clean_text(metadata.get("quantity", "")),
- _clean_text(metadata.get("status", "")),
- _clean_text(parent.value if parent else ""),
- _clean_text(parent_metadata.get("source_span", "")),
- _clean_text(parent_metadata.get("raw_text", "")),
- ]
- )
-
-
-def _multi_unit_chain_local_hit_text(record: SessionMemoryRecordV2, parent: SessionMemoryRecordV2 | None) -> str:
- metadata = dict(record.metadata or {})
- parent_metadata = dict(parent.metadata or {}) if parent is not None else {}
- return " ".join(
- [
- _clean_text(record.value),
- _clean_text(record.category),
- _clean_text(record.relation),
- _clean_text(metadata.get("facet_type", "")),
- _clean_text(metadata.get("unit_kind", "")),
- _clean_text(metadata.get("facet_role", "")),
- _clean_text(metadata.get("facet_value", "")),
- _clean_text(metadata.get("facet_source_span", "")),
- _clean_text(metadata.get("action", "")),
- _clean_text(metadata.get("target", "")),
- _clean_text(metadata.get("quantity", "")),
- _clean_text(metadata.get("status", "")),
- _clean_text(parent.value if parent else ""),
- _clean_text(parent_metadata.get("source_span", "")),
- ]
- )
-
-
-def _multi_unit_chain_slot_hits(
- graph: SessionMemoryGraphV2,
- query: str,
- final_hits: Sequence[MemoryHit],
- *,
- top_k: int,
-) -> Dict[str, Any]:
- mode = _normalize(os.getenv("TMCRA_MULTI_UNIT_CHAIN_SLOT_MODE", "on"))
- if mode in _MULTI_UNIT_CHAIN_DISABLED_MODES:
- return {
- "hits": list(final_hits),
- "metadata": {"multi_unit_chain_slot_enabled": False, "multi_unit_chain_slot_reason": "disabled"},
- }
- query_tokens = set(_path_utility_tokens(query))
- focus_tokens = _multi_unit_chain_focus_tokens(query)
- temporal_comparison_intent = bool(query_tokens & _MULTI_UNIT_CHAIN_TEMPORAL_COMPARISON_TOKENS) and len(focus_tokens) >= 2
- aggregation_or_joiner_intent = bool(
- re.search(r"\b(?:and|or|both|each|total|sum|minimum|maximum|amount|count|many|number)\b", str(query), flags=re.IGNORECASE)
- )
- numeric_aggregation_intent = bool(
- query_tokens
- & (
- _MULTI_UNIT_CHAIN_NUMERIC_VALUE_TOKENS
- | {"comment", "comments", "number", "percent", "percentage", "sum", "total"}
- )
- )
- count_or_aggregation_intent = bool(query_tokens & _MULTI_UNIT_CHAIN_COUNT_TOKENS) and (aggregation_or_joiner_intent or len(focus_tokens) >= 3)
- multi_intent = count_or_aggregation_intent or temporal_comparison_intent
- if not query_tokens or not multi_intent:
- return {
- "hits": list(final_hits),
- "metadata": {
- "multi_unit_chain_slot_enabled": False,
- "multi_unit_chain_slot_reason": "no_multi_intent",
- },
- }
- if not focus_tokens:
- return {
- "hits": list(final_hits),
- "metadata": {
- "multi_unit_chain_slot_enabled": True,
- "multi_unit_chain_slot_formed": False,
- "multi_unit_chain_slot_reason": "no_focus_tokens",
- },
- }
-
- records = list(getattr(graph, "records_by_id", {}).values())
- parent_by_slot = {_clean_text(record.slot_key).lower(): record for record in records}
- candidates: List[tuple[float, SessionMemoryRecordV2, SessionMemoryRecordV2 | None, Dict[str, Any]]] = []
- for record in records:
- metadata = dict(record.metadata or {})
- if record.state not in {"active", "parallel_active", "evidence"}:
- continue
- profile_shadow_unit = _profile_shadow_eventlike_record(record, metadata)
- if profile_shadow_unit:
- facet_type = "action"
- unit_kind = "profile_shadow_unit"
- parent_slot = _clean_text(record.slot_key).lower()
- parent = None
- text = _profile_shadow_unit_text(record, metadata)
- local_text = text
- else:
- if _normalize(metadata.get("content_variant", "")) != "event_facet_write":
- continue
- if _normalize(metadata.get("facet_layer_version", "")) != "event_unit_v1":
- continue
- facet_type = _normalize(metadata.get("facet_type", ""))
- unit_kind = _normalize(metadata.get("unit_kind", ""))
- if facet_type not in _MULTI_UNIT_CHAIN_FACET_TYPES and unit_kind not in _MULTI_UNIT_CHAIN_UNIT_KINDS:
- continue
- parent_slot = _clean_text(metadata.get("facet_parent_slot_key", "")).lower()
- parent = parent_by_slot.get(parent_slot)
- text = _multi_unit_chain_hit_text(record, parent)
- local_text = _multi_unit_chain_local_hit_text(record, parent)
- semantic_event_unit = _normalize(record.memory_id).startswith("tmcra.event.") or _normalize(parent_slot).startswith("tmcra.event.")
- unit_tokens = _multi_unit_chain_normalized_tokens(text)
- local_tokens = _multi_unit_chain_normalized_tokens(local_text)
- focus_overlap = focus_tokens & unit_tokens
- if not focus_overlap:
- continue
- score = min(1.2, len(focus_overlap) / max(1.0, len(focus_tokens)) * 1.6)
- local_numeric_signal = _multi_unit_chain_numeric_signal(unit_kind, facet_type, local_text, local_tokens)
- if _multi_unit_chain_date_like_numeric(local_text):
- numeric_signal = 0.0
- else:
- numeric_signal = local_numeric_signal
- if numeric_signal <= 0.0:
- numeric_signal = _multi_unit_chain_numeric_signal(unit_kind, facet_type, text, unit_tokens)
- temporal_anchor_signal = _multi_unit_chain_temporal_anchor_signal(text, unit_tokens)
- local_temporal_anchor_signal = _multi_unit_chain_temporal_anchor_signal(local_text, local_tokens)
- if facet_type in {"action", "entity", "role", "numeric"}:
- score += 0.34
- if unit_kind in {"action_unit", "target_entity", "leadership_role", "participation_role", "numeric_quantity"}:
- score += 0.24
- if profile_shadow_unit:
- score += 0.30
- semantic_event_priority = bool(semantic_event_unit and (count_or_aggregation_intent or numeric_aggregation_intent))
- if semantic_event_priority:
- score += 0.48
- if numeric_signal and numeric_aggregation_intent:
- score += numeric_signal
- if count_or_aggregation_intent and not numeric_aggregation_intent and facet_type in {"action", "entity", "role"}:
- score += 0.36
- if count_or_aggregation_intent and not numeric_aggregation_intent and unit_kind in {"action_unit", "target_entity", "leadership_role", "participation_role"}:
- score += 0.22
- if parent is not None:
- score += 0.12
- if score < 0.72:
- continue
- candidates.append(
- (
- round(min(4.0, 2.15 + score), 6),
- record,
- parent,
- {
- "multi_unit_chain_focus_overlap_tokens": sorted(focus_overlap)[:16],
- "multi_unit_chain_score": round(score, 6),
- "multi_unit_chain_numeric_signal": round(numeric_signal, 6),
- "multi_unit_chain_temporal_anchor_signal": round(temporal_anchor_signal, 6),
- "multi_unit_chain_local_temporal_anchor_signal": round(local_temporal_anchor_signal, 6),
- "multi_unit_chain_temporal_comparison": bool(temporal_comparison_intent),
- "multi_unit_chain_numeric_aggregation": bool(numeric_aggregation_intent),
- "unit_kind": unit_kind,
- "facet_type": facet_type,
- "multi_unit_chain_parent_slot_key": parent_slot,
- "multi_unit_chain_profile_shadow_unit": bool(profile_shadow_unit),
- "multi_unit_chain_semantic_event_unit": bool(semantic_event_unit),
- "multi_unit_chain_semantic_event_priority": bool(semantic_event_priority),
- "multi_unit_chain_count_or_aggregation": bool(count_or_aggregation_intent),
- },
- )
- )
-
- if not candidates:
- return {
- "hits": list(final_hits),
- "metadata": {
- "multi_unit_chain_slot_enabled": True,
- "multi_unit_chain_slot_formed": False,
- "multi_unit_chain_slot_reason": "no_matching_units",
- "multi_unit_chain_candidate_count": 0,
- "multi_unit_chain_focus_tokens": sorted(focus_tokens)[:24],
- },
- }
-
- if temporal_comparison_intent:
- candidates.sort(
- key=lambda item: (
- float(item[3].get("multi_unit_chain_local_temporal_anchor_signal", 0.0) or 0.0),
- float(item[3].get("multi_unit_chain_temporal_anchor_signal", 0.0) or 0.0),
- 1 if bool(item[3].get("multi_unit_chain_semantic_event_priority", False)) else 0,
- 0
- if _normalize(item[3].get("unit_kind", "")) == "numeric_quantity"
- or _normalize(item[3].get("facet_type", "")) == "numeric"
- else 1,
- float(item[0]),
- int(item[1].turn_index),
- ),
- reverse=True,
- )
- else:
- candidates.sort(
- key=lambda item: (
- float(item[3].get("multi_unit_chain_numeric_signal", 0.0) or 0.0)
- if bool(item[3].get("multi_unit_chain_numeric_aggregation", False))
- else (
- 1.2
- if bool(item[3].get("multi_unit_chain_semantic_event_priority", False))
- else
- 0.0
- if _normalize(item[3].get("unit_kind", "")) == "numeric_quantity"
- or _normalize(item[3].get("facet_type", "")) == "numeric"
- else 1.0
- ),
- float(item[0]),
- int(item[1].turn_index),
- ),
- reverse=True,
- )
- selected: List[tuple[float, SessionMemoryRecordV2, SessionMemoryRecordV2 | None, Dict[str, Any]]] = []
- seen_parent_slots: set[str] = set()
- seen_values: set[str] = set()
- max_units = max(2, min(8, int(os.getenv("TMCRA_MULTI_UNIT_CHAIN_SLOT_MAX_UNITS", "6") or 6)))
- for item in candidates:
- _, record, parent, metadata = item
- parent_slot = _clean_text(metadata.get("multi_unit_chain_parent_slot_key", ""))
- value_key = "|".join(
- [
- _normalize(record.value),
- _normalize(metadata.get("unit_kind", "")),
- parent_slot,
- ]
- )
- if value_key in seen_values:
- continue
- if parent_slot and parent_slot in seen_parent_slots:
- # Keep the chain broad: one strongest unit per parent event first.
- continue
- seen_values.add(value_key)
- if parent_slot:
- seen_parent_slots.add(parent_slot)
- selected.append(item)
- if len(selected) >= max_units:
- break
- if count_or_aggregation_intent:
- try:
- max_profile_shadow_units = max(0, int(os.getenv("TMCRA_MULTI_UNIT_CHAIN_PROFILE_SHADOW_MAX_UNITS", "2") or 2))
- except (TypeError, ValueError):
- max_profile_shadow_units = 2
- selected_ids = {item[1].memory_id for item in selected}
- added_profile_shadow_units = 0
- for item in candidates:
- _, record, _parent, metadata = item
- if added_profile_shadow_units >= max_profile_shadow_units:
- break
- if record.memory_id in selected_ids:
- continue
- if not bool(metadata.get("multi_unit_chain_profile_shadow_unit", False)):
- continue
- selected.append(item)
- selected_ids.add(record.memory_id)
- parent_slot = _clean_text(metadata.get("multi_unit_chain_parent_slot_key", ""))
- if parent_slot:
- seen_parent_slots.add(parent_slot)
- added_profile_shadow_units += 1
-
- min_parents = max(2, min(4, int(os.getenv("TMCRA_MULTI_UNIT_CHAIN_SLOT_MIN_PARENTS", "2") or 2)))
- if len(seen_parent_slots) < min_parents:
- return {
- "hits": list(final_hits),
- "metadata": {
- "multi_unit_chain_slot_enabled": True,
- "multi_unit_chain_slot_formed": False,
- "multi_unit_chain_slot_reason": "insufficient_parent_coverage",
- "multi_unit_chain_candidate_count": len(candidates),
- "multi_unit_chain_selected_unit_count": len(selected),
- "multi_unit_chain_parent_count": len(seen_parent_slots),
- "multi_unit_chain_focus_tokens": sorted(focus_tokens)[:24],
- },
- }
-
- chain_memory_ids: List[str] = []
- packed_hits: List[MemoryHit] = []
- for score, unit_record, parent, extra in selected:
- chain_memory_ids.append(unit_record.memory_id)
- packed_hits.append(
- _memory_hit_from_record(
- unit_record,
- score=score,
- metadata={
- **extra,
- "multi_unit_chain_slot": True,
- "multi_unit_chain_bundle": True,
- "evidence_snippet_role": "multi_unit_chain_evidence_unit",
- },
- )
- )
- if parent is not None:
- unit_metadata = dict(unit_record.metadata or {})
- chain_memory_ids.append(parent.memory_id)
- parent_hit = _memory_hit_from_record(
- parent,
- score=max(1.9, score - 0.08),
- metadata={
- **extra,
- "multi_unit_chain_slot": True,
- "multi_unit_chain_bundle": True,
- "evidence_snippet_role": "multi_unit_chain_parent_event",
- "multi_unit_chain_child_id": unit_record.memory_id,
- "multi_unit_chain_child_value": unit_record.value,
- "multi_unit_chain_child_source_span": _clean_text(
- unit_metadata.get("facet_source_span", "") or unit_metadata.get("source_span", "")
- ),
- },
- )
- parent_hit.memory_id = f"{parent.memory_id}#multi_parent:{unit_record.memory_id}"
- packed_hits.append(parent_hit)
-
- seen_ids: set[str] = set()
- merged: List[MemoryHit] = []
- insertion_index = max(0, min(len(final_hits), int(os.getenv("TMCRA_MULTI_UNIT_CHAIN_SLOT_INSERT_AFTER", "1") or 1)))
- ordered = [*list(final_hits[:insertion_index]), *packed_hits, *list(final_hits[insertion_index:])]
- for hit in ordered:
- if hit.memory_id and hit.memory_id in seen_ids:
- continue
- if hit.memory_id:
- seen_ids.add(hit.memory_id)
- merged.append(hit)
- return {
- "hits": merged,
- "metadata": {
- "multi_unit_chain_slot_enabled": True,
- "multi_unit_chain_slot_formed": True,
- "multi_unit_chain_slot_reason": "formed",
- "multi_unit_chain_candidate_count": len(candidates),
- "multi_unit_chain_selected_unit_count": len(selected),
- "multi_unit_chain_parent_count": len(seen_parent_slots),
- "multi_unit_chain_inserted_hit_count": len(packed_hits),
- "multi_unit_chain_memory_ids": _dedupe(chain_memory_ids, max_items=32),
- "multi_unit_chain_focus_tokens": sorted(focus_tokens)[:24],
- },
- }
-
-
-def _profile_support_source_hits(
- graph: SessionMemoryGraphV2,
- profile_hit: MemoryHit,
- *,
- grouped_hits: Mapping[str, Sequence[MemoryHit]],
- query: str,
- limit: int,
-) -> List[MemoryHit]:
- metadata = dict(profile_hit.metadata or {})
- support_ids = _dedupe(
- [
- *list(metadata.get("profile_support_ids", []) or []),
- *list(metadata.get("support_memory_ids", []) or []),
- ],
- max_items=24,
- )
- if not support_ids:
- return []
- query_raw_tokens = set(_path_utility_tokens(query))
- query_tokens = _profile_query_expanded_tokens(query)
- profile_event_id = _clean_text(metadata.get("profile_first_hybrid_event_id", "")) or _runtime_event_key(profile_hit)
- candidates: List[tuple[float, MemoryHit]] = []
- for support_id in support_ids:
- record = getattr(graph, "records_by_id", {}).get(support_id)
- if record is None or record.state not in {"active", "parallel_active", "evidence"}:
- continue
- support_hit = _memory_hit_from_record(record)
- event_id = _runtime_event_key(support_hit)
- if not event_id or event_id == profile_event_id or event_id not in grouped_hits:
- continue
- match_score, overlap_tokens, raw_overlap_tokens = _profile_hit_match_score(query_raw_tokens, query_tokens, support_hit)
- source_score = max(0.80, float(profile_hit.score) - 0.02) + min(0.28, match_score * 0.12)
- support_metadata = dict(support_hit.metadata or {})
- profile_metadata = dict(profile_hit.metadata or {})
- support_metadata.update(
- {
- "profile_first_hybrid_rescue": True,
- "profile_first_source_support": True,
- "profile_first_parent_memory_id": profile_hit.memory_id,
- "profile_first_hybrid_event_id": event_id,
- "profile_first_parent_summary": _clean_text(profile_metadata.get("profile_summary", "")) or _clean_text(profile_metadata.get("profile_value", "")) or _clean_text(profile_hit.value),
- "profile_type": _clean_text(support_metadata.get("profile_type", "")) or _clean_text(profile_metadata.get("profile_type", "")),
- "profile_domain": _clean_text(support_metadata.get("profile_domain", "")) or _clean_text(profile_metadata.get("profile_domain", "")),
- "profile_domain_label": _clean_text(support_metadata.get("profile_domain_label", "")) or _clean_text(profile_metadata.get("profile_domain_label", "")),
- "profile_value": _clean_text(support_metadata.get("profile_value", "")) or _clean_text(profile_metadata.get("profile_value", "")) or _clean_text(profile_hit.value),
- "profile_summary": _clean_text(support_metadata.get("profile_summary", "")) or _clean_text(profile_metadata.get("profile_summary", "")) or _clean_text(profile_hit.value),
- "profile_query_match_score": round(match_score, 6),
- "profile_query_overlap_tokens": list(overlap_tokens),
- "profile_query_raw_overlap_tokens": list(raw_overlap_tokens),
- "profile_source_pack_role": "source_event_support",
- }
- )
- candidates.append(
- (
- source_score,
- MemoryHit(
- memory_id=support_hit.memory_id,
- category=support_hit.category,
- value=support_hit.value,
- relation=support_hit.relation,
- anchors=list(support_hit.anchors),
- score=round(source_score, 6),
- source_kind=support_hit.source_kind,
- slot_key=support_hit.slot_key,
- state=support_hit.state,
- turn_index=int(support_hit.turn_index),
- metadata=support_metadata,
- ),
- )
- )
- candidates.sort(
- key=lambda item: (
- bool((item[1].metadata or {}).get("profile_query_raw_overlap_tokens")),
- float(item[0]),
- int(item[1].turn_index),
- ),
- reverse=True,
- )
- return [hit for _, hit in candidates[: max(0, int(limit))]]
-
-
-def _profile_same_event_source_hit(profile_hit: MemoryHit, group_hits: Sequence[MemoryHit], *, event_id: str) -> MemoryHit | None:
- source_hit = _support_hit_for_path("speaker_event_source_turn", group_hits) or _representative_event_hit(group_hits)
- if source_hit is None or source_hit.memory_id == profile_hit.memory_id:
- return None
- metadata = dict(source_hit.metadata or {})
- profile_metadata = dict(profile_hit.metadata or {})
- metadata.update(
- {
- "profile_first_hybrid_rescue": True,
- "profile_first_source_support": True,
- "profile_first_same_event_support": True,
- "profile_first_parent_memory_id": profile_hit.memory_id,
- "profile_first_hybrid_event_id": event_id,
- "profile_first_parent_summary": _clean_text(profile_metadata.get("profile_summary", "")) or _clean_text(profile_metadata.get("profile_value", "")) or _clean_text(profile_hit.value),
- "profile_type": _clean_text(metadata.get("profile_type", "")) or _clean_text(profile_metadata.get("profile_type", "")),
- "profile_domain": _clean_text(metadata.get("profile_domain", "")) or _clean_text(profile_metadata.get("profile_domain", "")),
- "profile_domain_label": _clean_text(metadata.get("profile_domain_label", "")) or _clean_text(profile_metadata.get("profile_domain_label", "")),
- "profile_value": _clean_text(metadata.get("profile_value", "")) or _clean_text(profile_metadata.get("profile_value", "")) or _clean_text(profile_hit.value),
- "profile_summary": _clean_text(metadata.get("profile_summary", "")) or _clean_text(profile_metadata.get("profile_summary", "")) or _clean_text(profile_hit.value),
- "profile_source_pack_role": "same_event_source_turn",
- "evidence_snippet_role": "profile_source_support",
- }
- )
- return MemoryHit(
- memory_id=source_hit.memory_id,
- category=source_hit.category,
- value=source_hit.value,
- relation=source_hit.relation,
- anchors=list(source_hit.anchors),
- score=max(float(source_hit.score), float(profile_hit.score) - 0.01),
- source_kind=source_hit.source_kind,
- slot_key=source_hit.slot_key,
- state=source_hit.state,
- turn_index=int(source_hit.turn_index),
- metadata=metadata,
- )
-
-
-def _profile_first_hybrid_rescue(
- graph: SessionMemoryGraphV2,
- query: str,
- *,
- grouped_hits: Mapping[str, Sequence[MemoryHit]],
- top_k: int,
-) -> Dict[str, Any]:
- intent = infer_profile_query_intent(query)
- if not bool(intent.get("enabled")):
- return {"hits": [], "event_ids": []}
- raw_hits = _profile_query_rescue_hits(graph, query, top_k=max(4, int(top_k or 1) * 3))
- limit = max(2, min(4, int(top_k or 1)))
- selected_hits: List[MemoryHit] = []
- selected_event_ids: List[str] = []
- seen_event_ids: set[str] = set()
- for rank, hit in enumerate(raw_hits, start=1):
- event_id = _runtime_event_key(hit)
- if not event_id or event_id not in grouped_hits or event_id in seen_event_ids:
- continue
- metadata = dict(hit.metadata or {})
- metadata.update(
- {
- "profile_first_hybrid_rescue": True,
- "profile_first_hybrid_rank": rank,
- "profile_first_hybrid_event_id": event_id,
- }
- )
- hit.metadata = metadata
- hit.score = round(max(float(hit.score), 0.88) + max(0.0, 0.08 - (0.01 * len(selected_hits))), 6)
- selected_hits.append(hit)
- selected_event_ids.append(event_id)
- seen_event_ids.add(event_id)
- same_event_source = _profile_same_event_source_hit(hit, grouped_hits.get(event_id, []), event_id=event_id)
- if same_event_source is not None:
- selected_hits.append(same_event_source)
- if len(selected_hits) >= limit:
- break
- support_hits = _profile_support_source_hits(
- graph,
- hit,
- grouped_hits=grouped_hits,
- query=query,
- limit=max(1, min(3, limit - len(selected_hits) + 1)),
- )
- for support_hit in support_hits:
- support_event_id = _clean_text((support_hit.metadata or {}).get("profile_first_hybrid_event_id", "")) or _runtime_event_key(support_hit)
- if not support_event_id or support_event_id in seen_event_ids:
- continue
- selected_hits.append(support_hit)
- selected_event_ids.append(support_event_id)
- seen_event_ids.add(support_event_id)
- if len(selected_hits) >= limit:
- break
- if len(selected_hits) >= limit:
- break
- return {
- "hits": selected_hits,
- "event_ids": selected_event_ids,
- "memory_ids": [hit.memory_id for hit in selected_hits],
- }
-
-
-def _inject_profile_first_hits(
- final_hits: Sequence[MemoryHit],
- profile_first_hits: Sequence[MemoryHit],
- *,
- selected_event_ids: Sequence[str],
- selected_path_ids: Sequence[str],
-) -> List[MemoryHit]:
- merged: List[MemoryHit] = []
- seen_memory_ids: set[str] = set()
- for hit in list(profile_first_hits) + list(final_hits):
- if not hit or not hit.memory_id or hit.memory_id in seen_memory_ids:
- continue
- metadata = dict(hit.metadata or {})
- if bool(metadata.get("profile_first_hybrid_rescue")):
- event_id = _clean_text(metadata.get("profile_first_hybrid_event_id", "")) or _runtime_event_key(hit)
- metadata.update(
- {
- "event_id": event_id,
- "path_id": "",
- "evidence_snippet_role": "selected_event_representative",
- "hybrid_score_source": "profile_first_hybrid_rescue",
- "selected_event_ids": list(selected_event_ids),
- "selected_path_ids": list(selected_path_ids),
- }
- )
- hit = MemoryHit(
- memory_id=hit.memory_id,
- category=hit.category,
- value=hit.value,
- relation=hit.relation,
- anchors=list(hit.anchors),
- score=float(hit.score),
- source_kind=hit.source_kind,
- slot_key=hit.slot_key,
- state=hit.state,
- turn_index=int(hit.turn_index),
- metadata=metadata,
- )
- merged.append(hit)
- seen_memory_ids.add(hit.memory_id)
- return merged
-
-
-def _profile_focused_pack_hits(
- graph: SessionMemoryGraphV2,
- query: str,
- final_hits: Sequence[MemoryHit],
- *,
- top_k: int,
-) -> Dict[str, Any]:
- intent = infer_profile_query_intent(query)
- if not bool(intent.get("enabled")):
- return {
- "hits": list(final_hits),
- "metadata": {
- "profile_focused_pack_enabled": False,
- "profile_focused_pack_reason": "profile_intent_not_requested",
- },
- }
- source_hits = _learnable_graph_hits(graph)
- if not source_hits:
- return {
- "hits": list(final_hits),
- "metadata": {
- "profile_focused_pack_enabled": False,
- "profile_focused_pack_reason": "no_learnable_hits",
- },
- }
- runtime_graph = _build_runtime_graph_from_hits(query, source_hits)
- grouped_hits = dict(runtime_graph.get("grouped_hits", {}) or {})
- profile_first_payload = _profile_first_hybrid_rescue(
- graph,
- query,
- grouped_hits=grouped_hits,
- top_k=max(2, min(max(1, int(top_k or 1)), 4)),
- )
- profile_first_hits = list(profile_first_payload.get("hits", []) or [])
- profile_first_event_ids = list(profile_first_payload.get("event_ids", []) or [])
- profile_first_memory_ids = list(profile_first_payload.get("memory_ids", []) or [])
- if not profile_first_hits:
- return {
- "hits": list(final_hits),
- "metadata": {
- "profile_focused_pack_enabled": True,
- "profile_focused_pack_reason": "no_profile_hits",
- "profile_focused_pack_event_ids": [],
- "profile_focused_pack_memory_ids": [],
- },
- }
- selected_event_ids = _dedupe([*profile_first_event_ids, *_event_ids_from_hits(final_hits)], max_items=max(1, int(top_k or 1) * 2))
- selected_path_ids = _dedupe(
- [
- _clean_text((hit.metadata or {}).get("path_id", ""))
- for hit in final_hits
- if _clean_text((hit.metadata or {}).get("path_id", ""))
- ],
- max_items=max(1, int(top_k or 1)),
- )
- merged_hits = _inject_profile_first_hits(
- final_hits,
- profile_first_hits,
- selected_event_ids=selected_event_ids,
- selected_path_ids=selected_path_ids,
- )
- merged_hits = _coverage_preserving_final_hits(merged_hits, selected_event_ids=selected_event_ids, top_k=top_k)
- return {
- "hits": merged_hits,
- "metadata": {
- "profile_focused_pack_enabled": True,
- "profile_focused_pack_reason": "profile_first_pack_injected",
- "profile_focused_pack_event_ids": list(profile_first_event_ids),
- "profile_focused_pack_memory_ids": list(profile_first_memory_ids),
- "profile_focused_pack_hit_count": len(profile_first_hits),
- "profile_first_hybrid_enabled": True,
- "profile_first_event_ids": list(profile_first_event_ids),
- "profile_first_memory_ids": list(profile_first_memory_ids),
- },
- }
-
-
-def _topic_bucket_rerank_hits(
- graph: SessionMemoryGraphV2,
- query: str,
- hits: Sequence[MemoryHit],
- *,
- top_k: int,
-) -> Dict[str, Any]:
- if not hits:
- profile_rescue_hits = _profile_query_rescue_hits(graph, query, top_k=top_k)
- if profile_rescue_hits:
- return {
- "hits": profile_rescue_hits[: max(1, int(top_k or 1))],
- "metadata": {
- "topic_bucket_rerank_enabled": True,
- "topic_bucket_rerank_reason": "profile_query_rescue_from_empty_hits",
- "topic_bucket_profile_query_rescue_count": len(profile_rescue_hits),
- },
- }
- return {
- "hits": [],
- "metadata": {"topic_bucket_rerank_enabled": True, "topic_bucket_rerank_reason": "no_hits"},
- }
- query_topic = _assign_topic_bucket_for_text(graph, query, turn_index=0, create=False)
- query_bucket_id = _clean_text(query_topic.get("topic_bucket_id", ""))
- adjacent_ids = _topic_adjacent_bucket_ids(graph, query_bucket_id)
- dialogue_tunnel_ids = _dialogue_tunnel_bucket_ids(graph, query_bucket_id)
- query_keywords = list(query_topic.get("topic_keywords", []) or [])
- bridge_requested = _topic_bridge_requested(query)
- dialogue_requested = _dialogue_tunnel_requested(query)
- profile_query_requested = bool(infer_profile_query_intent(query).get("enabled"))
- reranked: List[MemoryHit] = []
- stats = {
- "same_bucket": 0,
- "bridge_bucket": 0,
- "blocked_bridge_bucket": 0,
- "dialogue_tunnel_bucket": 0,
- "blocked_dialogue_tunnel_bucket": 0,
- "profile_route_preserved": 0,
- "profile_query_rescue": 0,
- "overlap_bucket": 0,
- "off_topic": 0,
- }
- for hit in list(hits):
- metadata = dict(hit.metadata or {})
- hit_bucket_id = _clean_text(metadata.get("topic_bucket_id", ""))
- hit_keywords = _dedupe(metadata.get("topic_keywords", []) or [], max_items=32)
- overlap = _topic_bucket_overlap_score(query_keywords, hit_keywords)
- same_bucket = bool(query_bucket_id and hit_bucket_id and query_bucket_id == hit_bucket_id)
- bridge_bucket = bool(hit_bucket_id and hit_bucket_id in adjacent_ids)
- dialogue_tunnel_bucket = bool(hit_bucket_id and hit_bucket_id in dialogue_tunnel_ids)
- overlap_bucket = overlap >= 0.22
- bridge_allowed = bridge_bucket and bridge_requested
- dialogue_allowed = dialogue_tunnel_bucket and dialogue_requested
- profile_route_preserved = bool(
- profile_query_requested
- and (
- bool(metadata.get("profile_layer"))
- or "profile_route" in _normalize(metadata.get("match_reason", ""))
- )
- )
- current_subject_preserved = bool(metadata.get("current_subject_resolver") or metadata.get("public_subject_match"))
- delta = 0.0
- if same_bucket:
- delta += 1.35
- stats["same_bucket"] += 1
- elif current_subject_preserved:
- delta += 0.18
- stats["profile_route_preserved"] += 1
- elif bridge_allowed:
- delta += 0.42
- stats["bridge_bucket"] += 1
- elif dialogue_allowed:
- delta += 0.16
- stats["dialogue_tunnel_bucket"] += 1
- elif overlap_bucket:
- delta += 0.28 + overlap
- stats["overlap_bucket"] += 1
- elif bridge_bucket:
- delta -= 1.05
- stats["blocked_bridge_bucket"] += 1
- elif dialogue_tunnel_bucket:
- delta -= 1.35
- stats["blocked_dialogue_tunnel_bucket"] += 1
- elif profile_route_preserved:
- delta += 0.08
- stats["profile_route_preserved"] += 1
- else:
- delta -= 0.72
- stats["off_topic"] += 1
- memory_type = _normalize(metadata.get("memory_type", ""))
- durability = _normalize(metadata.get("durability", ""))
- conflict_policy = _normalize(metadata.get("conflict_policy", ""))
- if (memory_type == "hard_constraint" or durability == "hard" or conflict_policy == "must_preserve") and (
- same_bucket or bridge_allowed or dialogue_allowed or overlap_bucket
- ):
- delta += 1.10
- match_reason = _clean_text(metadata.get("match_reason", ""))
- if profile_route_preserved and "profile_route" not in _normalize(match_reason):
- match_reason = ",".join(_dedupe([match_reason, "profile_route"], max_items=4))
- metadata.update(
- {
- "topic_bucket_rerank": True,
- "topic_bucket_query_id": query_bucket_id,
- "topic_bucket_query_label": _clean_text(query_topic.get("topic_label", "")),
- "topic_bucket_overlap": round(overlap, 6),
- "topic_bucket_delta": round(delta, 6),
- "topic_bucket_same": same_bucket,
- "topic_bucket_bridge": bridge_bucket,
- "topic_bucket_bridge_allowed": bridge_allowed,
- "topic_bucket_dialogue_tunnel": dialogue_tunnel_bucket,
- "topic_bucket_dialogue_tunnel_allowed": dialogue_allowed,
- "topic_bucket_profile_route_preserved": profile_route_preserved,
- "topic_bucket_current_subject_preserved": current_subject_preserved,
- "match_reason": match_reason,
- }
- )
- hit.metadata = metadata
- hit.score = float(hit.score) + delta
- reranked.append(hit)
- seen_ids = {hit.memory_id for hit in reranked if hit.memory_id}
- rescue_records = [
- record
- for record in getattr(graph, "records_by_id", {}).values()
- if record.memory_id not in seen_ids
- and record.state == "active"
- and _clean_text((record.metadata or {}).get("topic_bucket_id", "")) == query_bucket_id
- and _normalize(record.category) != "question"
- ]
- rescue_records.sort(
- key=lambda record: (
- int(any(marker in _normalize(record.value) for marker in ("过敏", "必须", "避开", "禁止", "不能", "must", "avoid", "allergy"))),
- int(_normalize(record.category) in {"constraint", "preference"}),
- float(record.confidence),
- float(record.salience),
- -int(record.turn_index),
- ),
- reverse=True,
- )
- rescue_hits = [
- _topic_bucket_record_to_hit(record, query_topic=query_topic, rank=index)
- for index, record in enumerate(rescue_records[: max(2, min(12, int(top_k or 1) * 2))], start=1)
- ]
- reranked.extend(rescue_hits)
- dialogue_rescue_hits: List[MemoryHit] = []
- if dialogue_requested and dialogue_tunnel_ids:
- seen_ids.update(hit.memory_id for hit in rescue_hits if hit.memory_id)
- dialogue_records = [
- record
- for record in getattr(graph, "records_by_id", {}).values()
- if record.memory_id not in seen_ids
- and record.state == "active"
- and _clean_text((record.metadata or {}).get("topic_bucket_id", "")) in dialogue_tunnel_ids
- and _normalize(record.category) != "question"
- ]
- def dialogue_record_rank(record: SessionMemoryRecordV2) -> tuple[int, int, float, float, int]:
- return (
- int(any(marker in _normalize(record.value) for marker in ("过敏", "必须", "避开", "禁止", "不能", "must", "avoid", "allergy"))),
- int(_normalize(record.category) in {"constraint", "preference"}),
- float(record.confidence),
- float(record.salience),
- -int(record.turn_index),
- )
-
- dialogue_records.sort(key=dialogue_record_rank, reverse=True)
- selected_dialogue_records: List[SessionMemoryRecordV2] = []
- selected_dialogue_ids: set[str] = set()
- for bucket_id in sorted(dialogue_tunnel_ids):
- bucket_records = [
- record
- for record in dialogue_records
- if _clean_text((record.metadata or {}).get("topic_bucket_id", "")) == bucket_id
- ]
- if not bucket_records:
- continue
- best = bucket_records[0]
- selected_dialogue_records.append(best)
- selected_dialogue_ids.add(best.memory_id)
- target_dialogue_rescue = max(len(selected_dialogue_records), max(1, min(4, int(top_k or 1) // 2 or 1)))
- for record in dialogue_records:
- if len(selected_dialogue_records) >= target_dialogue_rescue:
- break
- if record.memory_id in selected_dialogue_ids:
- continue
- selected_dialogue_records.append(record)
- selected_dialogue_ids.add(record.memory_id)
- dialogue_rescue_hits = [
- _topic_bucket_record_to_hit(record, query_topic=query_topic, rank=index, rescue_kind="dialogue_tunnel")
- for index, record in enumerate(selected_dialogue_records, start=1)
- ]
- reranked.extend(dialogue_rescue_hits)
- profile_rescue_hits: List[MemoryHit] = []
- if profile_query_requested:
- seen_ids.update(hit.memory_id for hit in reranked if hit.memory_id)
- for index, hit in enumerate(_profile_query_rescue_hits(graph, query, top_k=top_k), start=1):
- if hit.memory_id in seen_ids:
- continue
- metadata = dict(hit.metadata or {})
- metadata.update(
- {
- "profile_query_rescue_rank": index,
- "topic_bucket_query_id": query_bucket_id,
- "topic_bucket_query_label": _clean_text(query_topic.get("topic_label", "")),
- "topic_bucket_profile_route_preserved": True,
- }
- )
- hit.metadata = metadata
- hit.score = float(hit.score) + 0.08
- profile_rescue_hits.append(hit)
- seen_ids.add(hit.memory_id)
- stats["profile_query_rescue"] = len(profile_rescue_hits)
- reranked.extend(profile_rescue_hits)
- reranked.sort(key=lambda item: (float(item.score), int(item.turn_index)), reverse=True)
- limit = max(1, int(top_k or 1))
- focused_hits = [
- hit
- for hit in reranked
- if bool((hit.metadata or {}).get("topic_bucket_same"))
- or float((hit.metadata or {}).get("topic_bucket_overlap", 0.0) or 0.0) >= 0.22
- or bool((hit.metadata or {}).get("topic_bucket_bridge_allowed"))
- or bool((hit.metadata or {}).get("topic_bucket_dialogue_tunnel_allowed"))
- or bool((hit.metadata or {}).get("topic_bucket_profile_route_preserved"))
- or bool((hit.metadata or {}).get("topic_bucket_current_subject_preserved"))
- or bool((hit.metadata or {}).get("current_subject_resolver"))
- ]
- model_path_fallback = False
- no_bucket_model_fallback = False
- if not focused_hits:
- model_supported_hits = [
- hit
- for hit in reranked
- if _clean_text((hit.metadata or {}).get("path_id", ""))
- or _clean_text((hit.metadata or {}).get("hybrid_score_source", ""))
- ]
- if model_supported_hits:
- focused_hits = model_supported_hits
- model_path_fallback = True
- generic_memory_fallback = False
- if not focused_hits and any(not _is_public_dialog_hit(hit) for hit in reranked):
- focused_hits = list(reranked)
- generic_memory_fallback = True
- if not focused_hits and not query_bucket_id:
- focused_hits = list(reranked)
- no_bucket_model_fallback = True
- time_focused_model_path = any(
- "speaker_event_time" in _clean_text((hit.metadata or {}).get("path_id", ""))
- or _clean_text((hit.metadata or {}).get("model_focused_answer_type", "")) == "time"
- for hit in focused_hits
- )
- if time_focused_model_path:
- focused_hits = [
- hit
- for hit in focused_hits
- if not (
- _clean_text(hit.source_kind) == "public_dialog_profile"
- and not _clean_text((hit.metadata or {}).get("path_id", ""))
- )
- ]
- filtered_count = max(0, len(reranked) - len(focused_hits))
- query_subject_signature = _public_subject_signature(_public_query_subject(query))
-
- def _hit_subject_signatures(hit: MemoryHit) -> set[str]:
- metadata = dict(hit.metadata or {})
- signatures = {
- _normalize(metadata.get("subject_signature", "")).replace("-", "_"),
- _public_subject_signature(metadata.get("subject", "")),
- }
- canonical_slot_key = _clean_text(metadata.get("canonical_slot_key", ""))
- if ".subject." in canonical_slot_key:
- signatures.add(_public_subject_signature(canonical_slot_key.split(".subject.", 1)[-1]))
- if ".subject." in hit.slot_key:
- signatures.add(_public_subject_signature(hit.slot_key.split(".subject.", 1)[-1]))
- signatures.discard("")
- return signatures
-
- protected_current_hits = [
- hit
- for hit in focused_hits
- if bool((hit.metadata or {}).get("current_subject_resolver"))
- ]
- protected_model_path_hits = [
- hit
- for hit in focused_hits
- if _clean_text((hit.metadata or {}).get("path_id", ""))
- and _clean_text((hit.metadata or {}).get("hybrid_score_source", ""))
- ]
- protected_generic_hits = [
- hit
- for hit in focused_hits
- if not _is_public_dialog_hit(hit)
- and not bool((hit.metadata or {}).get("profile_layer"))
- ]
- if protected_current_hits and query_subject_signature:
- protected_ids = {hit.memory_id for hit in protected_current_hits if hit.memory_id}
- compacted_focused_hits: List[MemoryHit] = []
- for hit in focused_hits:
- if hit.memory_id in protected_ids:
- compacted_focused_hits.append(hit)
- continue
- same_current_subject = query_subject_signature in _hit_subject_signatures(hit)
- inactive_state = _normalize(hit.state) in {"superseded", "evidence", "historical", "stale", "false"}
- if same_current_subject and inactive_state:
- continue
- compacted_focused_hits.append(hit)
- focused_hits = compacted_focused_hits
- filtered_count = max(0, len(reranked) - len(focused_hits))
-
- selected_hits: List[MemoryHit] = []
- selected_keys: set[str] = set()
- for hit in [*protected_current_hits, *protected_model_path_hits, *protected_generic_hits, *focused_hits]:
- key = hit.memory_id or f"{_hit_event_id(hit)}::{hit.slot_key}::{hit.value[:80]}"
- if key in selected_keys:
- continue
- selected_hits.append(hit)
- selected_keys.add(key)
- if len(selected_hits) >= limit:
- break
- dialogue_reserved_count = 0
- if dialogue_requested and dialogue_tunnel_ids:
- selected_ids = {hit.memory_id for hit in selected_hits if hit.memory_id}
- selected_bucket_ids = {
- _clean_text((hit.metadata or {}).get("topic_bucket_id", ""))
- for hit in selected_hits
- if _clean_text((hit.metadata or {}).get("topic_bucket_id", ""))
- }
- for bucket_id in sorted(dialogue_tunnel_ids):
- if bucket_id in selected_bucket_ids:
- continue
- candidate = next(
- (
- hit
- for hit in focused_hits
- if hit.memory_id not in selected_ids
- and _clean_text((hit.metadata or {}).get("topic_bucket_id", "")) == bucket_id
- and bool((hit.metadata or {}).get("topic_bucket_dialogue_tunnel_allowed"))
- ),
- None,
- )
- if candidate is None:
- continue
- if len(selected_hits) < limit:
- selected_hits.append(candidate)
- else:
- replace_index = len(selected_hits) - 1
- for index in range(len(selected_hits) - 1, -1, -1):
- metadata = dict(selected_hits[index].metadata or {})
- hardish = (
- _normalize(metadata.get("memory_type", "")) == "hard_constraint"
- or _normalize(metadata.get("durability", "")) == "hard"
- or _normalize(metadata.get("conflict_policy", "")) == "must_preserve"
- )
- if not hardish and not bool(metadata.get("topic_bucket_dialogue_tunnel_allowed")):
- replace_index = index
- break
- selected_hits[replace_index] = candidate
- selected_ids.add(candidate.memory_id)
- selected_bucket_ids.add(bucket_id)
- dialogue_reserved_count += 1
- selected_hits.sort(
- key=lambda item: (
- int(bool((item.metadata or {}).get("current_subject_resolver"))),
- float(item.score),
- int(item.turn_index),
- ),
- reverse=True,
- )
- return {
- "hits": selected_hits[:limit],
- "metadata": {
- "topic_bucket_rerank_enabled": True,
- "topic_bucket_no_fill_policy": True,
- "topic_bucket_query_id": query_bucket_id,
- "topic_bucket_query_label": _clean_text(query_topic.get("topic_label", "")),
- "topic_bucket_query_keywords": query_keywords,
- "topic_bucket_adjacent_ids": sorted(adjacent_ids),
- "dialogue_tunnel_adjacent_ids": sorted(dialogue_tunnel_ids),
- "topic_bucket_bridge_requested": bridge_requested,
- "dialogue_tunnel_requested": dialogue_requested,
- "topic_bucket_model_path_fallback": model_path_fallback,
- "topic_bucket_generic_memory_fallback": generic_memory_fallback,
- "topic_bucket_no_bucket_model_fallback": no_bucket_model_fallback,
- "topic_bucket_candidate_count": len(hits),
- "topic_bucket_rescue_count": len(rescue_hits),
- "dialogue_tunnel_rescue_count": len(dialogue_rescue_hits),
- "profile_query_rescue_count": len(profile_rescue_hits),
- "dialogue_tunnel_reserved_count": dialogue_reserved_count,
- "topic_bucket_focused_count": len(focused_hits),
- "topic_bucket_final_count": len(selected_hits[:limit]),
- "topic_bucket_filtered_count": filtered_count,
- "topic_bucket_stats": stats,
- },
- }
-
-
-def _looks_like_write_turn(user_text: str, *, answer_payload: Dict[str, Any] | None = None) -> bool:
- text = _normalize(user_text)
- if not text:
- return False
- if "?" in text:
- return False
- if any(marker in text for marker in _WRITE_MARKERS):
- return True
- metadata = dict((answer_payload or {}).get("metadata", {}) or {})
- return bool(metadata.get("memory_write"))
-
-
-def _explicit_overwrite_requested(user_text: str) -> bool:
- text = _normalize(user_text)
- return bool(text) and any(marker in text for marker in _OVERWRITE_MARKERS)
-
-
-def _apply_turn_write_intent(records: List[SessionMemoryRecordV2], *, user_text: str) -> List[SessionMemoryRecordV2]:
- if not records or not _explicit_overwrite_requested(user_text):
- return records
- for record in records:
- metadata = dict(record.metadata or {})
- metadata.setdefault("write_intent", "overwrite")
- metadata.setdefault("memory_gate_decision", "explicit_overwrite")
- metadata["allow_parallel_state"] = False
- record.metadata = metadata
- return records
-
-
-def _records_from_extractor(
- extractor: SessionMemoryExtractor,
- *,
- query: str,
- answer_payload: Dict[str, Any] | None,
- extraction_result: Dict[str, Any] | None,
- turn_index: int,
- profile: TMCRAProfile | None = None,
-) -> List[SessionMemoryRecordV2]:
- profile = profile or TMCRAProfile()
- raw_records = extractor.extract(
- query=query,
- extraction_result=extraction_result,
- answer_bundle=answer_payload,
- answer_mode=str((answer_payload or {}).get("answer_mode", "transparent")),
- turn_index=turn_index,
- )
- results: List[SessionMemoryRecordV2] = []
- for index, record in enumerate(raw_records):
- slot_key = profile.stable_slot_key(
- category=record.category,
- value=record.value,
- anchors=record.anchor_concepts,
- slot_key=record.metadata.get("slot_key", "") if isinstance(record.metadata, dict) else "",
- relation=record.relation,
- metadata=dict(record.metadata or {}),
- )
- metadata = {
- **dict(record.metadata or {}),
- "memory_role": _clean_text(dict(record.metadata or {}).get("memory_role", "")) or "user",
- "authority": _clean_text(dict(record.metadata or {}).get("authority", "")) or "source",
- "canonical_slot_key": _clean_text(dict(record.metadata or {}).get("canonical_slot_key", "")) or slot_key,
- "writeback_class": _clean_text(dict(record.metadata or {}).get("writeback_class", "")),
- "origin_query": _clean_text(dict(record.metadata or {}).get("origin_query", "")) or _clean_text(query),
- "origin_answer_id": _clean_text(dict(record.metadata or {}).get("origin_answer_id", "")),
- "support_memory_ids": _dedupe(dict(record.metadata or {}).get("support_memory_ids", []) or []),
- "support_fact_refs": _dedupe(dict(record.metadata or {}).get("support_fact_refs", []) or []),
- "support_path_refs": _dedupe(dict(record.metadata or {}).get("support_path_refs", []) or []),
- "promotion_state": _clean_text(dict(record.metadata or {}).get("promotion_state", "")) or "none",
- }
- results.append(
- SessionMemoryRecordV2(
- memory_id=f"{slot_key}:{turn_index}:auto:{index}",
- category=record.category,
- slot_key=slot_key,
- value=_clean_text(record.value),
- relation=_clean_text(record.relation) or f"{record.category}_memory",
- anchor_concepts=_dedupe(record.anchor_concepts, max_items=8),
- evidence_anchors=_dedupe(record.anchor_concepts, max_items=8),
- salience=float(record.salience),
- confidence=float(record.confidence),
- source_kind=_clean_text(record.source_kind) or "session_memory",
- turn_index=int(record.turn_index),
- state="active",
- metadata=metadata,
- )
- )
- return results
-
-
-def _build_turn_records(
- extractor: SessionMemoryExtractor,
- *,
- user_text: str,
- answer_payload: Dict[str, Any] | None,
- extraction_result: Dict[str, Any] | None,
- turn_index: int,
- allow_auto_extract: bool,
- profile: TMCRAProfile | None = None,
-) -> List[SessionMemoryRecordV2]:
- profile = profile or TMCRAProfile()
- structured_records = _parse_structured_records(answer_payload, turn_index=turn_index, profile=profile)
- if structured_records:
- return _apply_typed_tunnel_annotations(
- _apply_turn_write_intent(structured_records, user_text=user_text),
- source_text=user_text,
- )
- structured_records = _parse_structured_records(extraction_result, turn_index=turn_index, profile=profile)
- if structured_records:
- return _apply_typed_tunnel_annotations(
- _apply_turn_write_intent(structured_records, user_text=user_text),
- source_text=user_text,
- )
- if not allow_auto_extract and not _looks_like_write_turn(user_text, answer_payload=answer_payload):
- return []
- return _apply_typed_tunnel_annotations(
- _records_from_extractor(
- extractor,
- query=user_text,
- answer_payload=None,
- extraction_result=extraction_result,
- turn_index=turn_index,
- profile=profile,
- ),
- source_text=user_text,
- )
-
-
-class NullMemoryAdapter(MemoryAdapter):
- name = "null_memory"
-
- def reset(self) -> None:
- return None
-
- def ingest_turn(
- self,
- user_text: str,
- assistant_text: str = "",
- *,
- answer_payload: Dict[str, Any] | None = None,
- extraction_result: Dict[str, Any] | None = None,
- ) -> None:
- _ = user_text, assistant_text, answer_payload, extraction_result
-
- def retrieve(self, query: str, *, top_k: int = 6) -> MemoryRetrieval:
- _ = query, top_k
- return MemoryRetrieval()
-
- def stats(self) -> Dict[str, Any]:
- return _state_stats(storage_bytes=0, retrieval_context_tokens=0, total_state_tokens=0, records=0)
-
- def storage_bytes(self) -> int:
- return 0
-
- def build_prompt_context(self, query: str, *, top_k: int = 8) -> Dict[str, Any]:
- _ = top_k
- return {
- "mode": "null_memory",
- "query": query,
- "retrieval": MemoryRetrieval().to_dict(),
- "stats": self.stats(),
- "state": {},
- }
-
-
-class GraphSessionMemoryAdapter(MemoryAdapter):
- name = "graph_session_memory_v2"
-
- def __init__(
- self,
- *,
- auto_extract: bool = False,
- storage_backend: str = "sqlite",
- storage_path: str = "",
- scope_id: str = "",
- audit_retention: int = 256,
- lightweight_stats: bool = True,
- retrieval_mode: str = "heuristic",
- node_model_path: str = "",
- path_model_path: str = "",
- node_model_device: str = "",
- candidate_event_k: int = 24,
- support_path_k: int = 3,
- path_tunnel_rescue_k: int = 0,
- path_tunnel_rescue_score_floor: float = 0.0,
- path_tunnel_rescue_min_age: int = 0,
- path_tunnel_rescue_min_score_margin: float = 0.0,
- event_rerank_mode: str = "matrix",
- matrix_event_top_k: int = DEFAULT_MATRIX_EVENT_TOP_K,
- memory_router_mode: str = "",
- memory_router_threshold: float = _MEMORY_ROUTER_DEFAULT_THRESHOLD,
- memory_router_margin: float = _MEMORY_ROUTER_DEFAULT_MARGIN,
- injection_planner_mode: str = "",
- injection_planner_model_path: str = "",
- injection_planner_latest_path: str = "",
- injection_planner_device: str = "",
- injection_planner_selection_threshold: float = -1.0,
- injection_planner_row_threshold: float = -1.0,
- injection_planner_logic_threshold: float = -1.0,
- temporal_layer_mode: str = "",
- temporal_router_mode: str = "",
- temporal_router_dir: str = "",
- temporal_router_latest_path: str = "",
- temporal_router_device: str = "",
- ) -> None:
- prewarm_embedder_mode = (
- _normalize(os.getenv("TMCRA_EMBEDDER_INDEX_RECALL_MODE", ""))
- or _normalize(os.getenv("TMCRA_WRITE_EMBEDDER_INDEX_MODE", ""))
- )
- self._embedder_prewarm_metadata = _prewarm_embedder_dense_if_requested(mode=prewarm_embedder_mode)
- self.extractor = SessionMemoryExtractor()
- self.profile = TMCRAProfile()
- self.temporal_organizer = TemporalOrganizer()
- self.temporal_query_planner = TemporalQueryPlanner()
- self.timeline_evidence_builder = TimelineEvidencePackBuilder()
- self.auto_extract = bool(auto_extract)
- self.storage_backend = _normalize(storage_backend) or "sqlite"
- self.audit_retention = max(1, int(audit_retention))
- self.lightweight_stats = bool(lightweight_stats)
- self.scope_id = _clean_text(scope_id) or f"graph-session-{uuid.uuid4().hex}"
- self._store: SQLiteSessionMemoryStore | None = None
- self.storage_path = ""
- if self.storage_backend == "sqlite":
- resolved_storage_path = _clean_text(storage_path) or str((Path(tempfile.gettempdir()) / "tmcra_graph_session_memory.sqlite3").resolve())
- self._store = SQLiteSessionMemoryStore(resolved_storage_path, audit_retention=self.audit_retention)
- self.storage_path = str(self._store.storage_path)
- self.graph = self._store.load_graph(self.scope_id)
- elif self.storage_backend == "memory":
- self.storage_path = _clean_text(storage_path)
- self.graph = SessionMemoryGraphV2(
- audit_retention=self.audit_retention,
- persistence_backend="memory",
- persistence_path=self.storage_path,
- )
- else:
- raise ValueError(f"Unsupported storage backend: {self.storage_backend}")
- self._last_retrieval_context_tokens = 0
- self._last_writeback_summary: Dict[str, Any] = {}
- self.retrieval_mode = _normalize(retrieval_mode) or "heuristic"
- self.node_model_path = _clean_text(node_model_path)
- self.path_model_path = _clean_text(path_model_path)
- self.node_model_device = _clean_text(node_model_device)
- self.candidate_event_k = max(1, int(candidate_event_k))
- self.support_path_k = max(1, int(support_path_k))
- self.path_tunnel_rescue_k = max(0, int(path_tunnel_rescue_k or 0))
- self.path_tunnel_rescue_score_floor = max(0.0, float(path_tunnel_rescue_score_floor or 0.0))
- self.path_tunnel_rescue_min_age = max(0, int(path_tunnel_rescue_min_age or 0))
- self.path_tunnel_rescue_min_score_margin = max(0.0, float(path_tunnel_rescue_min_score_margin or 0.0))
- self.event_rerank_mode = _normalize(event_rerank_mode) or "matrix"
- self.matrix_event_top_k = max(1, int(matrix_event_top_k or DEFAULT_MATRIX_EVENT_TOP_K))
- self.write_embedder_index_mode = _normalize(os.getenv("TMCRA_WRITE_EMBEDDER_INDEX_MODE", ""))
- if not self.write_embedder_index_mode:
- self.write_embedder_index_mode = "off"
- try:
- self.write_embedder_index_max_terms = max(
- 8,
- int(os.getenv("TMCRA_WRITE_EMBEDDER_INDEX_MAX_TERMS", "96") or 96),
- )
- except (TypeError, ValueError):
- self.write_embedder_index_max_terms = 96
- env_embedder_recall_mode = _normalize(os.getenv("TMCRA_EMBEDDER_INDEX_RECALL_MODE", ""))
- self.embedder_index_recall_mode = env_embedder_recall_mode or self.write_embedder_index_mode
- try:
- self.embedder_index_recall_k = max(
- 0,
- int(os.getenv("TMCRA_EMBEDDER_INDEX_RECALL_K", "0") or 0),
- )
- except (TypeError, ValueError):
- self.embedder_index_recall_k = 0
- self.embedder_pre_recall_mode = _normalize(os.getenv("TMCRA_EMBEDDER_PRE_RECALL_MODE", ""))
- if not self.embedder_pre_recall_mode:
- self.embedder_pre_recall_mode = "off"
- try:
- self.embedder_pre_recall_k = max(
- 0,
- int(os.getenv("TMCRA_EMBEDDER_PRE_RECALL_K", "0") or 0),
- )
- except (TypeError, ValueError):
- self.embedder_pre_recall_k = 0
- self.embedder_fusion_mode = _normalize(os.getenv("TMCRA_EMBEDDER_FUSION_MODE", ""))
- try:
- self.embedder_fusion_weight = max(
- 0.0,
- float(os.getenv("TMCRA_EMBEDDER_FUSION_WEIGHT", "0.35") or 0.35),
- )
- except (TypeError, ValueError):
- self.embedder_fusion_weight = 0.35
- try:
- self.embedder_fusion_score_floor = max(
- 0.0,
- float(os.getenv("TMCRA_EMBEDDER_FUSION_SCORE_FLOOR", "0.62") or 0.62),
- )
- except (TypeError, ValueError):
- self.embedder_fusion_score_floor = 0.62
- try:
- self.embedder_fusion_top_k = max(
- 0,
- int(os.getenv("TMCRA_EMBEDDER_FUSION_TOP_K", "16") or 16),
- )
- except (TypeError, ValueError):
- self.embedder_fusion_top_k = 16
- try:
- self.embedder_fusion_select_k = max(
- 0,
- int(os.getenv("TMCRA_EMBEDDER_FUSION_SELECT_K", "4") or 4),
- )
- except (TypeError, ValueError):
- self.embedder_fusion_select_k = 4
- try:
- self.embedder_fusion_max_boost = max(
- 0.0,
- float(os.getenv("TMCRA_EMBEDDER_FUSION_MAX_BOOST", "0.42") or 0.42),
- )
- except (TypeError, ValueError):
- self.embedder_fusion_max_boost = 0.42
- env_router_mode = _clean_text(os.getenv("TMCRA_MEMORY_ROUTER_MODE", ""))
- self.memory_router_mode = _normalize(memory_router_mode) or _normalize(env_router_mode) or "observe"
- try:
- self.memory_router_threshold = float(
- os.getenv("TMCRA_MEMORY_ROUTER_THRESHOLD", "")
- or memory_router_threshold
- or _MEMORY_ROUTER_DEFAULT_THRESHOLD
- )
- except (TypeError, ValueError):
- self.memory_router_threshold = _MEMORY_ROUTER_DEFAULT_THRESHOLD
- try:
- self.memory_router_margin = float(
- os.getenv("TMCRA_MEMORY_ROUTER_MARGIN", "")
- or memory_router_margin
- or _MEMORY_ROUTER_DEFAULT_MARGIN
- )
- except (TypeError, ValueError):
- self.memory_router_margin = _MEMORY_ROUTER_DEFAULT_MARGIN
- self._loaded_node_scorer: LoadedNodeMemoryScorer | None = None
- self._node_scorer_error = ""
- env_planner_mode = _clean_text(os.getenv("TMCRA_INJECTION_PLANNER_MODE", ""))
- self.injection_planner_mode = _normalize(injection_planner_mode) or _normalize(env_planner_mode) or "observe"
- self.injection_planner_model_path = _clean_text(
- injection_planner_model_path or os.getenv("TMCRA_INJECTION_PLANNER_MODEL_PATH", "")
- )
- self.injection_planner_latest_path = _clean_text(
- injection_planner_latest_path or os.getenv("TMCRA_INJECTION_PLANNER_LATEST", "")
- )
- self.injection_planner_device = _clean_text(
- injection_planner_device or os.getenv("TMCRA_INJECTION_PLANNER_DEVICE", "")
- )
- self.injection_planner_selection_threshold_override = float(injection_planner_selection_threshold)
- self.injection_planner_row_threshold_override = float(injection_planner_row_threshold)
- self.injection_planner_logic_threshold_override = float(injection_planner_logic_threshold)
- self._loaded_injection_planner: Any | None = None
- self._injection_planner_config: Any | None = None
- self._injection_planner_payload: Dict[str, Any] = {}
- self._injection_planner_thresholds: Dict[str, float] = {}
- self._injection_planner_resolved_path = ""
- self._injection_planner_error = ""
- self._injection_planner_evidence_role_supported = False
- env_temporal_layer_mode = _clean_text(os.getenv("TMCRA_TEMPORAL_LAYER_MODE", ""))
- self.temporal_layer_mode = _normalize(temporal_layer_mode) or _normalize(env_temporal_layer_mode) or "observe"
- env_temporal_router_mode = _clean_text(os.getenv("TMCRA_TEMPORAL_ROUTER_MODE", ""))
- self.temporal_router_mode = _normalize(temporal_router_mode) or _normalize(env_temporal_router_mode) or "observe"
- self.temporal_router_dir = _clean_text(
- temporal_router_dir or os.getenv("TMCRA_TEMPORAL_ROUTER_DIR", "")
- )
- self.temporal_router_latest_path = _clean_text(
- temporal_router_latest_path
- or os.getenv("TMCRA_TEMPORAL_ROUTER_LATEST", "")
- or "models/temporal_router_v1_latest.txt"
- )
- self.temporal_router_device = _clean_text(
- temporal_router_device or os.getenv("TMCRA_TEMPORAL_ROUTER_DEVICE", "")
- ) or "cpu"
- self._loaded_temporal_router: Any | None = None
- self._temporal_router_resolved_dir = ""
- self._temporal_router_error = ""
-
- def _empty_graph(self) -> SessionMemoryGraphV2:
- return SessionMemoryGraphV2(
- audit_retention=self.audit_retention,
- persistence_backend=self.storage_backend,
- persistence_path=self.storage_path,
- )
-
- def _reload_graph(self) -> None:
- if self._store is not None:
- self.graph = self._store.load_graph(self.scope_id)
- else:
- self.graph.configure_persistence(
- backend=self.storage_backend,
- path=self.storage_path,
- audit_retention=self.audit_retention,
- )
-
- def _persist_graph(self) -> None:
- self.graph.configure_persistence(
- backend=self.storage_backend,
- path=self.storage_path,
- audit_retention=self.audit_retention,
- )
- if self._store is not None:
- self._store.save_graph(self.scope_id, self.graph)
-
- def replace_graph(self, graph: SessionMemoryGraphV2) -> None:
- self.graph = graph
- self._persist_graph()
-
- def _storage_breakdown(self) -> Dict[str, int]:
- core_payload = self.graph._core_payload()
- audit_payload_full = self.graph._audit_payload()
- if self.lightweight_stats:
- audit_token_payload = {
- "totals": dict(self.graph.audit_event_totals),
- "trimmed": dict(self.graph.audit_trimmed_counts),
- "retained": {
- "turn_log": len(self.graph.turn_log),
- "retrieval_log": len(self.graph.retrieval_log),
- "answer_support_log": len(self.graph.answer_support_log),
- },
- "audit_retention": int(self.graph.audit_retention),
- }
- else:
- audit_token_payload = audit_payload_full
- core_storage_bytes = len(json.dumps(core_payload, ensure_ascii=False).encode("utf-8"))
- audit_storage_bytes = len(json.dumps(audit_payload_full, ensure_ascii=False).encode("utf-8"))
- core_state_token_estimate = _estimate_tokens(json.dumps(core_payload, ensure_ascii=False))
- audit_state_token_estimate = _estimate_tokens(json.dumps(audit_token_payload, ensure_ascii=False))
- return {
- "core_storage_bytes": int(core_storage_bytes),
- "audit_storage_bytes": int(audit_storage_bytes),
- "storage_bytes": int(core_storage_bytes + audit_storage_bytes),
- "core_state_token_estimate": int(core_state_token_estimate),
- "audit_state_token_estimate": int(audit_state_token_estimate),
- "total_state_token_estimate": int(core_state_token_estimate + audit_state_token_estimate),
- }
-
- def reset(self) -> None:
- if self._store is not None:
- self._store.clear_scope(self.scope_id)
- self.graph = self._store.load_graph(self.scope_id)
- else:
- self.graph = self._empty_graph()
- self._last_retrieval_context_tokens = 0
- self._last_writeback_summary = {}
-
- def _temporal_layer_enabled(self) -> bool:
- return _normalize(self.temporal_layer_mode) not in _TEMPORAL_LAYER_DISABLED_MODES
-
- def _temporal_router_enabled(self) -> bool:
- return self._temporal_layer_enabled() and _normalize(self.temporal_router_mode) not in _TEMPORAL_LAYER_DISABLED_MODES
-
- def _resolve_temporal_router_dir(self) -> str:
- direct_dir = _clean_text(self.temporal_router_dir)
- if direct_dir:
- root = Path(direct_dir)
- if (root / "writer_temporal_router.pt").exists() and (root / "query_temporal_router.pt").exists():
- return str(root)
- latest_path = _clean_text(self.temporal_router_latest_path)
- if latest_path:
- pointer = Path(latest_path)
- if pointer.exists():
- lines = pointer.read_text(encoding="utf-8").splitlines()
- candidate = _clean_text(lines[0] if lines else "")
- if candidate:
- root = Path(candidate)
- if (root / "writer_temporal_router.pt").exists() and (root / "query_temporal_router.pt").exists():
- return str(root)
- return ""
-
- def _load_temporal_router(self) -> LoadedTemporalRouter | None:
- if not self._temporal_router_enabled():
- self._temporal_router_error = "disabled"
- return None
- model_dir = self._resolve_temporal_router_dir()
- self._temporal_router_resolved_dir = model_dir
- if not model_dir:
- self._temporal_router_error = "model_dir_missing"
- return None
- if self._loaded_temporal_router is not None:
- return self._loaded_temporal_router
- try:
- self._loaded_temporal_router = LoadedTemporalRouter.from_dir(
- model_dir,
- device=self.temporal_router_device or "cpu",
- )
- self._temporal_router_error = ""
- except Exception as exc: # pragma: no cover - defensive runtime path
- self._loaded_temporal_router = None
- self._temporal_router_error = f"{type(exc).__name__}: {exc}"
- return self._loaded_temporal_router
-
- def _temporal_router_status_metadata(self) -> Dict[str, Any]:
- router = self._load_temporal_router()
- return {
- "temporal_router_enabled": router is not None,
- "temporal_router_mode": _normalize(self.temporal_router_mode) or "observe",
- "temporal_router_model_dir": self._temporal_router_resolved_dir,
- "temporal_router_error": self._temporal_router_error,
- }
-
- def _session_timestamp_from_payloads(self, *payloads: Mapping[str, Any] | None) -> str:
- for payload in payloads:
- data = dict(payload or {})
- for key in ("session_timestamp", "timestamp", "created_at", "turn_timestamp"):
- value = _clean_text(data.get(key, ""))
- if value:
- return value
- metadata = data.get("metadata")
- if isinstance(metadata, Mapping):
- for key in ("session_timestamp", "timestamp", "created_at", "turn_timestamp"):
- value = _clean_text(metadata.get(key, ""))
- if value:
- return value
- return ""
-
- def _temporal_frame_for_turn(
- self,
- *,
- user_text: str,
- previous_turn: str = "",
- answer_payload: Dict[str, Any] | None = None,
- extraction_result: Dict[str, Any] | None = None,
- ) -> TemporalFrame | None:
- if not self._temporal_layer_enabled():
- return None
- session_timestamp = self._session_timestamp_from_payloads(answer_payload, extraction_result)
- model_frame = None
- for payload in (answer_payload, extraction_result):
- data = dict(payload or {})
- candidate = data.get("temporal_frame") or dict(data.get("metadata", {}) or {}).get("temporal_frame")
- if isinstance(candidate, Mapping):
- model_frame = candidate
- break
- fallback_frame = self.temporal_organizer.organize_turn(
- current_turn=user_text,
- previous_turn=previous_turn,
- session_timestamp=session_timestamp,
- speaker="user",
- )
- if model_frame is None:
- router = self._load_temporal_router()
- if router is not None and router.writer_available():
- predicted = router.predict_writer_frame(
- current_turn=user_text,
- previous_turn=previous_turn,
- session_timestamp=session_timestamp,
- )
- writer_confidence = float(predicted.get("confidence", 0.0) or 0.0) if predicted else 0.0
- if predicted and writer_confidence >= _float_env("TMCRA_TEMPORAL_ROUTER_WRITER_MIN_CONFIDENCE", _TEMPORAL_ROUTER_DEFAULT_WRITER_MIN_CONFIDENCE):
- frame_payload = fallback_frame.to_dict()
- frame_payload.update(
- {
- key: value
- for key, value in predicted.items()
- if key in {"temporal_intent", "anchor_type", "granularity", "state_operation"}
- and _clean_text(value)
- }
- )
- if "should_create_timeline_edge" in predicted:
- frame_payload["should_create_timeline_edge"] = bool(predicted.get("should_create_timeline_edge", False))
- if writer_confidence > 0.0:
- frame_payload["confidence"] = writer_confidence
- frame_payload["metadata"] = {
- **dict(frame_payload.get("metadata", {}) or {}),
- **dict(predicted.get("metadata", {}) or {}),
- **self._temporal_router_status_metadata(),
- }
- model_frame = frame_payload
- if model_frame is None:
- return fallback_frame
- return self.temporal_organizer.organize_turn(
- current_turn=user_text,
- previous_turn=previous_turn,
- session_timestamp=session_timestamp,
- speaker="user",
- model_frame=model_frame,
- )
-
- def _temporal_turn_metadata(self, frame: TemporalFrame | None) -> Dict[str, Any]:
- if frame is None:
- return {
- "temporal_layer_enabled": False,
- "temporal_layer_mode": _normalize(self.temporal_layer_mode) or "observe",
- **self._temporal_router_status_metadata(),
- }
- return {
- "temporal_layer_enabled": True,
- "temporal_layer_mode": _normalize(self.temporal_layer_mode) or "observe",
- "temporal_frame": frame.to_dict(),
- "temporal_intent": frame.temporal_intent,
- "temporal_subject_key": frame.subject_key,
- "temporal_state_operation": frame.state_operation,
- **self._temporal_router_status_metadata(),
- }
-
- def _apply_temporal_frame_to_records(self, records: Sequence[SessionMemoryRecordV2], frame: TemporalFrame | None) -> None:
- if frame is None:
- return
- if frame.temporal_intent == "non_temporal" and not frame.subject_key:
- return
- for record in records:
- metadata = dict(record.metadata or {})
- metadata.update(
- {
- "temporal_layer": True,
- "temporal_frame": frame.to_dict(),
- "temporal_intent": frame.temporal_intent,
- "temporal_subject": frame.subject,
- "temporal_subject_key": frame.subject_key,
- "temporal_state_operation": frame.state_operation,
- "temporal_event_time": frame.event_time,
- "temporal_state_valid_from": frame.state_valid_from,
- "temporal_state_valid_to": frame.state_valid_to,
- }
- )
- record.metadata = metadata
-
- def _build_timeline_state_layer(self) -> TimelineStateLayer:
- layer = TimelineStateLayer()
- for turn in sorted(list(getattr(self.graph, "turn_log", []) or []), key=lambda item: int(item.get("turn_index", 0) or 0)):
- metadata = dict(turn.get("metadata", {}) or {})
- frame_payload = metadata.get("temporal_frame")
- if not isinstance(frame_payload, Mapping):
- continue
- frame = TemporalFrame.from_mapping(frame_payload)
- if frame.temporal_intent == "non_temporal" and not frame.new_state:
- continue
- record_ids = [item for item in list(turn.get("record_ids", []) or []) if _clean_text(item)]
- source_event_id = _clean_text(metadata.get("temporal_source_record_id", "")) or (record_ids[0] if record_ids else "")
- frame.metadata = {
- **dict(frame.metadata or {}),
- "source_text": _clean_text(turn.get("text", "")) or frame.evidence_span,
- }
- layer.apply_frame(
- frame,
- source_event_id=source_event_id,
- source_turn_id=_clean_text(turn.get("turn_id", "")),
- state_type=_clean_text(metadata.get("temporal_state_type", "")) or "profile",
- )
- return layer
-
- def _temporal_runtime_pack(self, query: str) -> Dict[str, Any]:
- metadata: Dict[str, Any] = {
- "temporal_runtime_enabled": False,
- "temporal_layer_mode": _normalize(self.temporal_layer_mode) or "observe",
- **self._temporal_router_status_metadata(),
- }
- if not self._temporal_layer_enabled():
- metadata["temporal_runtime_reason"] = "disabled"
- return {"metadata": metadata}
- plan = self.temporal_query_planner.plan(query)
- router = self._load_temporal_router()
- if router is not None and router.query_available():
- predicted_plan = router.predict_query_plan(query=query)
- query_confidence = float(predicted_plan.get("confidence", 0.0) or 0.0) if predicted_plan else 0.0
- router_confidences = dict(dict(predicted_plan.get("metadata", {}) or {}).get("temporal_router_confidences", {}) or {}) if predicted_plan else {}
- intent_confidence = float(router_confidences.get("query_temporal_intent", 0.0) or 0.0)
- if (
- predicted_plan
- and query_confidence >= _float_env("TMCRA_TEMPORAL_ROUTER_QUERY_MIN_CONFIDENCE", _TEMPORAL_ROUTER_DEFAULT_QUERY_MIN_CONFIDENCE)
- and intent_confidence >= _float_env("TMCRA_TEMPORAL_ROUTER_QUERY_INTENT_MIN_CONFIDENCE", _TEMPORAL_ROUTER_DEFAULT_QUERY_INTENT_MIN_CONFIDENCE)
- ):
- plan_payload = plan.to_dict()
- plan_payload.update(
- {
- key: value
- for key, value in predicted_plan.items()
- if key in {"query_temporal_intent", "timeline_operation"} and _clean_text(value)
- }
- )
- for key in ("prefer_current_state", "prefer_previous_state", "requires_ordered_chain", "requires_comparison"):
- if key in predicted_plan:
- plan_payload[key] = bool(predicted_plan.get(key, False))
- if float(predicted_plan.get("confidence", 0.0) or 0.0) > 0.0:
- plan_payload["confidence"] = float(predicted_plan.get("confidence", 0.0) or 0.0)
- plan_payload["metadata"] = {
- **dict(plan_payload.get("metadata", {}) or {}),
- **dict(predicted_plan.get("metadata", {}) or {}),
- **self._temporal_router_status_metadata(),
- }
- plan = TemporalQueryPlan(**{key: value for key, value in plan_payload.items() if key in TemporalQueryPlan.__dataclass_fields__})
- metadata["temporal_query_plan"] = plan.to_dict()
- if plan.query_temporal_intent == "non_temporal" or plan.timeline_operation == "none":
- metadata["temporal_runtime_reason"] = "non_temporal_query"
- return {"plan": plan, "metadata": metadata}
- if plan.timeline_operation in {"query_current", "query_previous"} and not _clean_text(plan.target_subject_key):
- metadata["temporal_runtime_reason"] = "missing_target_subject"
- return {"plan": plan, "metadata": metadata}
- timeline_layer = self._build_timeline_state_layer()
- pack = self.timeline_evidence_builder.build(plan=plan, timeline_layer=timeline_layer)
- metadata.update(
- {
- "temporal_runtime_enabled": True,
- "temporal_runtime_reason": "ok",
- "temporal_evidence_pack": pack.to_dict(),
- "temporal_selected_answer_value": _clean_text(pack.selected_evidence.get("answer_value", "")),
- "temporal_selected_state_id": _clean_text(pack.selected_evidence.get("state_id", "")),
- "temporal_timeline_state_count": len(pack.timeline),
- }
- )
- return {"plan": plan, "pack": pack, "metadata": metadata}
-
- def _temporal_state_hit(
- self,
- *,
- state_payload: Mapping[str, Any],
- plan_payload: Mapping[str, Any],
- selected: bool,
- rank: int,
- ) -> MemoryHit | None:
- state_value = _clean_text(state_payload.get("state", ""))
- state_id = _clean_text(state_payload.get("state_id", ""))
- if not state_value or not state_id:
- return None
- source_event_id = _clean_text(state_payload.get("source_event_id", ""))
- source_record = self.graph.records_by_id.get(source_event_id)
- source_metadata = dict(source_record.metadata or {}) if source_record is not None else {}
- score = 0.98 if selected else max(0.75, 0.92 - (rank * 0.03))
- return MemoryHit(
- memory_id=f"temporal_state:{state_id}",
- category="time",
- value=state_value,
- relation="temporal_state",
- anchors=_dedupe([state_payload.get("time", ""), state_payload.get("source_text", ""), plan_payload.get("target_subject", "")], max_items=6),
- score=round(score, 6),
- source_kind="temporal_state_layer",
- slot_key=f"temporal.{_clean_text(plan_payload.get('target_subject_key', 'general'))}",
- state="active" if bool(state_payload.get("is_current", False)) else "history",
- turn_index=int(source_record.turn_index) if source_record is not None else 0,
- metadata={
- **source_metadata,
- "temporal_runtime_hit": True,
- "temporal_runtime_selected": bool(selected),
- "temporal_state_id": state_id,
- "temporal_state_value": state_value,
- "temporal_state_time": _clean_text(state_payload.get("time", "")),
- "temporal_state_valid_to": _clean_text(state_payload.get("valid_to", "")),
- "temporal_state_is_current": bool(state_payload.get("is_current", False)),
- "temporal_source_event_id": source_event_id,
- "temporal_source_turn_id": _clean_text(state_payload.get("source_turn_id", "")),
- "temporal_query_plan": dict(plan_payload),
- },
- )
-
- def _apply_temporal_evidence_pack_to_hits(
- self,
- hits: Sequence[MemoryHit],
- temporal_payload: Mapping[str, Any],
- *,
- top_k: int,
- ) -> Dict[str, Any]:
- metadata = dict(temporal_payload.get("metadata", {}) or {})
- pack = temporal_payload.get("pack")
- plan = temporal_payload.get("plan")
- if pack is None or plan is None or not bool(metadata.get("temporal_runtime_enabled", False)):
- return {"hits": list(hits), "metadata": metadata}
- pack_payload = pack.to_dict() if hasattr(pack, "to_dict") else dict(pack)
- plan_payload = plan.to_dict() if hasattr(plan, "to_dict") else dict(plan)
- selected_state_id = _clean_text(dict(pack_payload.get("selected_evidence", {}) or {}).get("state_id", ""))
- selected_answer_value = _clean_text(dict(pack_payload.get("selected_evidence", {}) or {}).get("answer_value", ""))
- timeline = list(pack_payload.get("timeline", []) or [])
- synthetic_hits: List[MemoryHit] = []
- if selected_state_id:
- selected_state = next((dict(item) for item in timeline if _clean_text(dict(item).get("state_id", "")) == selected_state_id), None)
- if selected_state is None and selected_answer_value:
- selected_state = {
- "state_id": selected_state_id,
- "state": selected_answer_value,
- "source_event_id": dict(pack_payload.get("selected_evidence", {}) or {}).get("source_event_id", ""),
- "source_turn_id": dict(pack_payload.get("selected_evidence", {}) or {}).get("source_turn_id", ""),
- "is_current": bool(plan_payload.get("prefer_current_state", False)),
- }
- if selected_state:
- hit = self._temporal_state_hit(state_payload=selected_state, plan_payload=plan_payload, selected=True, rank=0)
- if hit is not None:
- synthetic_hits.append(hit)
- if bool(plan_payload.get("requires_ordered_chain", False)) or bool(plan_payload.get("requires_comparison", False)):
- for index, state_payload in enumerate(timeline):
- if _clean_text(dict(state_payload).get("state_id", "")) == selected_state_id:
- continue
- hit = self._temporal_state_hit(
- state_payload=dict(state_payload),
- plan_payload=plan_payload,
- selected=False,
- rank=index + 1,
- )
- if hit is not None:
- synthetic_hits.append(hit)
- existing: List[MemoryHit] = []
- synthetic_source_ids = {
- _clean_text((hit.metadata or {}).get("temporal_source_event_id", ""))
- for hit in synthetic_hits
- if _clean_text((hit.metadata or {}).get("temporal_source_event_id", ""))
- }
- for hit in hits:
- hit_metadata = dict(hit.metadata or {})
- source_event_id = _clean_text(hit_metadata.get("event_id", hit.memory_id))
- if source_event_id in synthetic_source_ids:
- hit_metadata.update(
- {
- "temporal_runtime_support": True,
- "temporal_query_plan": dict(plan_payload),
- "temporal_selected_answer_value": selected_answer_value,
- "temporal_selected_state_id": selected_state_id,
- }
- )
- hit = MemoryHit(
- memory_id=hit.memory_id,
- category=hit.category,
- value=hit.value,
- relation=hit.relation,
- anchors=list(hit.anchors),
- score=hit.score,
- source_kind=hit.source_kind,
- slot_key=hit.slot_key,
- state=hit.state,
- turn_index=int(hit.turn_index),
- metadata=hit_metadata,
- )
- existing.append(hit)
- merged: List[MemoryHit] = []
- seen = set()
- for hit in [*synthetic_hits, *existing]:
- key = _clean_text(hit.memory_id)
- if key and key in seen:
- continue
- if key:
- seen.add(key)
- merged.append(hit)
- metadata.update(
- {
- "temporal_runtime_injected_hit_count": len(synthetic_hits),
- "temporal_runtime_injected_hit_ids": [hit.memory_id for hit in synthetic_hits],
- }
- )
- return {"hits": merged[: max(len(merged), top_k)], "metadata": metadata}
-
- def ingest_turn(
- self,
- user_text: str,
- assistant_text: str = "",
- *,
- answer_payload: Dict[str, Any] | None = None,
- extraction_result: Dict[str, Any] | None = None,
- ) -> None:
- self._reload_graph()
- turn_index = self.graph.next_turn()
- previous_topic = _last_topic_turn(self.graph)
- previous_turn_text = _clean_text(self.graph.turn_log[-1].get("text", "")) if self.graph.turn_log else ""
- temporal_frame = self._temporal_frame_for_turn(
- user_text=user_text,
- previous_turn=previous_turn_text,
- answer_payload=answer_payload,
- extraction_result=extraction_result,
- )
- topic_bucket = _assign_topic_bucket_for_text(
- self.graph,
- user_text,
- answer_payload=answer_payload,
- turn_index=turn_index,
- create=True,
- )
- records = _build_turn_records(
- self.extractor,
- user_text=user_text,
- answer_payload=answer_payload,
- extraction_result=extraction_result,
- turn_index=turn_index,
- allow_auto_extract=self.auto_extract,
- profile=self.profile,
- )
- self._apply_temporal_frame_to_records(records, temporal_frame)
- _apply_topic_bucket_to_records(records, topic_bucket)
- stored_ids = self.graph.add_records(records)
- write_embedder_metadata = _apply_write_embedder_index_to_graph(
- self.graph,
- stored_ids=stored_ids,
- turn_text=user_text,
- turn_index=turn_index,
- mode=self.write_embedder_index_mode,
- max_terms=self.write_embedder_index_max_terms,
- )
- topic_bridge_metadata = _add_topic_bridge_edges(
- self.graph,
- previous_topic=previous_topic,
- current_topic=topic_bucket,
- current_record_ids=stored_ids,
- turn_index=turn_index,
- evidence=user_text,
- )
- dialogue_tunnel_metadata = _add_dialogue_tunnel_edges(
- self.graph,
- current_topic=topic_bucket,
- current_record_ids=stored_ids,
- turn_index=turn_index,
- evidence=user_text,
- )
- turn_kind = "memory_write" if stored_ids else "noise"
- self.graph.record_turn(
- turn_kind=turn_kind,
- text=user_text,
- turn_index=turn_index,
- record_ids=stored_ids,
- speaker="user",
- assistant_text=assistant_text,
- metadata={
- "source": "user_turn",
- "auto_extract": bool(self.auto_extract),
- "topic_bucket_id": _clean_text(topic_bucket.get("topic_bucket_id", "")),
- "topic_label": _clean_text(topic_bucket.get("topic_label", "")),
- "topic_keywords": list(topic_bucket.get("topic_keywords", []) or []),
- "temporal_source_record_id": stored_ids[0] if stored_ids else "",
- **self._temporal_turn_metadata(temporal_frame),
- **write_embedder_metadata,
- **topic_bridge_metadata,
- **dialogue_tunnel_metadata,
- },
- )
- self._persist_graph()
-
- def ingest_answer_writeback(
- self,
- *,
- query_text: str,
- answer_text: str,
- answer_id: str,
- writeback_records: List[Dict[str, Any]],
- trace: Dict[str, Any] | None = None,
- ) -> List[str]:
- self._reload_graph()
- if not writeback_records:
- self._last_writeback_summary = {"stored_record_ids": [], "promotion_events": []}
- return []
- turn_index = self.graph.next_turn()
- records: List[SessionMemoryRecordV2] = []
- writeback_classes: List[str] = []
- for index, raw in enumerate(writeback_records):
- if not isinstance(raw, dict):
- continue
- category = _clean_text(raw.get("category", "fact")) or "fact"
- value = _clean_text(raw.get("value", ""))
- raw_slot_key = _clean_text(raw.get("slot_key", "")) or _clean_text(raw.get("slot", ""))
- anchors = _dedupe(raw.get("anchors", []) or [], max_items=8)
- slot_key = self.profile.stable_slot_key(
- category=category,
- value=value,
- anchors=anchors,
- slot_key=raw_slot_key,
- relation=_clean_text(raw.get("relation", "")),
- metadata=dict(raw.get("metadata", {}) or {}),
- )
- if not value or not slot_key:
- continue
- raw_metadata = dict(raw.get("metadata", {}) or {})
- writeback_class = _clean_text(raw_metadata.get("writeback_class", "")) or "fact"
- writeback_classes.append(writeback_class)
- metadata = {
- **raw_metadata,
- "memory_role": _clean_text(raw_metadata.get("memory_role", "")) or "assistant",
- "authority": _clean_text(raw_metadata.get("authority", "")) or "derived",
- "canonical_slot_key": _clean_text(raw_metadata.get("canonical_slot_key", "")) or slot_key.removeprefix("assistant.").split(".fact", 1)[0].split(".state_change", 1)[0].split(".high_conf_conclusion", 1)[0],
- "writeback_class": writeback_class,
- "origin_query": _clean_text(raw_metadata.get("origin_query", "")) or _clean_text(query_text),
- "origin_answer_id": _clean_text(raw_metadata.get("origin_answer_id", "")) or answer_id,
- "origin_answer_ids": _dedupe([*(raw_metadata.get("origin_answer_ids", []) or []), _clean_text(raw_metadata.get("origin_answer_id", "")) or answer_id]),
- "support_memory_ids": _dedupe(raw_metadata.get("support_memory_ids", []) or []),
- "support_fact_refs": _dedupe(raw_metadata.get("support_fact_refs", []) or []),
- "support_path_refs": _dedupe(raw_metadata.get("support_path_refs", []) or []),
- "promotion_state": _clean_text(raw_metadata.get("promotion_state", "")) or "candidate",
- "answer_id": answer_id,
- }
- record = SessionMemoryRecordV2(
- memory_id=f"{slot_key}:{turn_index}:assistant:{index}",
- category=category,
- slot_key=slot_key,
- value=value,
- relation=_clean_text(raw.get("relation", "")) or "assistant_memory",
- anchor_concepts=anchors,
- evidence_anchors=anchors,
- salience=float(raw.get("salience", 0.62) or 0.62),
- confidence=float(raw.get("confidence", 0.88) or 0.88),
- source_kind=_clean_text(raw.get("source_kind", "")) or "assistant_writeback",
- turn_index=turn_index,
- state=_clean_text(raw.get("state", "")) or "active",
- metadata=metadata,
- )
- records.append(record)
- stored_ids = self.graph.add_records(records)
- write_embedder_metadata = _apply_write_embedder_index_to_graph(
- self.graph,
- stored_ids=stored_ids,
- turn_text=" ".join([_clean_text(query_text), _clean_text(answer_text)]),
- turn_index=turn_index,
- mode=self.write_embedder_index_mode,
- max_terms=self.write_embedder_index_max_terms,
- )
- promotion_events = self._apply_writeback_promotions(stored_ids)
- writeback_class = writeback_classes[0] if len(set(writeback_classes)) == 1 and writeback_classes else ("mixed" if writeback_classes else "")
- self.graph.record_turn(
- turn_kind="assistant_writeback" if stored_ids else "assistant_writeback_empty",
- text=query_text,
- turn_index=turn_index,
- record_ids=stored_ids,
- speaker="assistant",
- assistant_text=answer_text,
- writeback_class=writeback_class,
- metadata={
- "source": "assistant_writeback",
- "answer_id": answer_id,
- "trace": dict(trace or {}),
- **write_embedder_metadata,
- },
- )
- self._last_writeback_summary = {
- "stored_record_ids": list(stored_ids),
- "promotion_events": list(promotion_events),
- **write_embedder_metadata,
- }
- self._persist_graph()
- return stored_ids
-
- def last_writeback_summary(self) -> Dict[str, Any]:
- return dict(self._last_writeback_summary)
-
- def _resolve_injection_planner_model_path(self) -> str:
- explicit_path = _clean_text(self.injection_planner_model_path)
- if explicit_path:
- path = Path(explicit_path).expanduser()
- if path.is_dir():
- path = path / "injection_planner.pt"
- return str(path)
- latest_path = _clean_text(self.injection_planner_latest_path)
- if not latest_path:
- return ""
- latest = Path(latest_path).expanduser()
- if latest.is_dir():
- return str(latest / "injection_planner.pt")
- if latest.suffix == ".pt":
- return str(latest)
- try:
- target = Path(latest.read_text(encoding="utf-8").strip()).expanduser()
- except Exception as exc:
- self._injection_planner_error = f"latest_pointer_read_failed: {exc}"
- return ""
- if target.is_dir():
- target = target / "injection_planner.pt"
- return str(target)
-
- def _load_injection_planner(self) -> Any | None:
- normalized_mode = _normalize(self.injection_planner_mode)
- if normalized_mode in _INJECTION_PLANNER_DISABLED_MODES:
- self._injection_planner_error = "disabled"
- return None
- if self._loaded_injection_planner is not None:
- return self._loaded_injection_planner
- torch_module = getattr(injection_planner_runtime, "torch", None)
- if torch_module is None:
- self._injection_planner_error = "torch_unavailable"
- return None
- model_path_text = self._resolve_injection_planner_model_path()
- if not model_path_text:
- if not self._injection_planner_error:
- self._injection_planner_error = "model_path_missing"
- return None
- model_path = Path(model_path_text)
- if not model_path.exists():
- self._injection_planner_error = f"model_path_not_found: {model_path}"
- return None
- try:
- device = torch_module.device(self.injection_planner_device or "cpu")
- payload = torch_module.load(model_path, map_location=device, weights_only=False)
- config = injection_planner_runtime.InjectionPlannerConfig.from_dict(dict(payload.get("config", {}) or {}))
- model = injection_planner_runtime.InjectionPlannerModel(config).to(device)
- state_dict = dict(payload.get("state_dict", {}) or {})
- model.load_state_dict(state_dict, strict=False)
- model.eval()
- self._loaded_injection_planner = model
- self._injection_planner_config = config
- self._injection_planner_payload = dict(payload)
- self._injection_planner_resolved_path = str(model_path)
- self._injection_planner_error = ""
- self._injection_planner_evidence_role_supported = any(
- str(key).startswith("evidence_role_head.") for key in state_dict
- )
- thresholds = {"selection_threshold": 0.5, "row_threshold": 0.5, "logic_threshold": 0.5}
- summary_path = model_path.parent / "train_summary.json"
- if summary_path.exists():
- try:
- summary = json.loads(summary_path.read_text(encoding="utf-8"))
- calibration = dict(summary.get("calibration", {}) or {})
- logic_calibration = dict(summary.get("logic_calibration", {}) or {})
- if calibration.get("selection_threshold") is not None:
- thresholds["selection_threshold"] = float(calibration.get("selection_threshold"))
- if calibration.get("row_threshold") is not None:
- thresholds["row_threshold"] = float(calibration.get("row_threshold"))
- if logic_calibration.get("logic_threshold") is not None:
- thresholds["logic_threshold"] = float(logic_calibration.get("logic_threshold"))
- except Exception:
- pass
- if self.injection_planner_selection_threshold_override >= 0.0:
- thresholds["selection_threshold"] = float(self.injection_planner_selection_threshold_override)
- if self.injection_planner_row_threshold_override >= 0.0:
- thresholds["row_threshold"] = float(self.injection_planner_row_threshold_override)
- if self.injection_planner_logic_threshold_override >= 0.0:
- thresholds["logic_threshold"] = float(self.injection_planner_logic_threshold_override)
- self._injection_planner_thresholds = thresholds
- except Exception as exc:
- self._loaded_injection_planner = None
- self._injection_planner_config = None
- self._injection_planner_payload = {}
- self._injection_planner_resolved_path = str(model_path)
- self._injection_planner_error = str(exc)
- self._injection_planner_evidence_role_supported = False
- return self._loaded_injection_planner
-
- def _injection_planner_candidate_from_hit(
- self,
- query: str,
- hit: MemoryHit,
- *,
- index: int,
- current_turn_index: int,
- ) -> Dict[str, Any]:
- metadata = dict(hit.metadata or {})
- category = _normalize(hit.category)
- source_kind = _normalize(hit.source_kind)
- logic_roles = [
- _normalize(item)
- for item in (
- metadata.get("logic_roles", [])
- if isinstance(metadata.get("logic_roles", []), list)
- else [metadata.get("logic_roles", "")]
- )
- if _normalize(item) in set(injection_planner_runtime.LOGIC_ROLES)
- ]
- if category in set(injection_planner_runtime.LOGIC_ROLES):
- logic_roles.append(category)
- if _normalize(metadata.get("writeback_class", "")) in set(injection_planner_runtime.LOGIC_ROLES):
- logic_roles.append(_normalize(metadata.get("writeback_class", "")))
- if not logic_roles:
- logic_roles = ["negative"] if _normalize(hit.state) in {"stale", "superseded", "false"} else ["evidence"]
- evidence_snippet_role = _normalize(metadata.get("evidence_snippet_role", ""))
- if bool(metadata.get("profile_layer")) or category == "profile" or source_kind.endswith("profile"):
- layer = "profile"
- elif category == "time" or float(metadata.get("temporal_score", 0.0) or 0.0) > 0.0:
- layer = "temporal"
- elif "resource" in logic_roles or _clean_text(metadata.get("resource_key", "")):
- layer = "resource"
- elif (
- source_kind in {"path_tunnel", "path_support", "public_dialog_path"}
- or evidence_snippet_role in {"selected_path_support", "path_tunnel_support"}
- or bool(metadata.get("path_tunnel_node"))
- ):
- layer = "path_tunnel"
- elif bool(metadata.get("topic_bucket_rerank")) or bool(metadata.get("topic_bucket_dialogue_tunnel_allowed")):
- layer = "topic_tunnel"
- else:
- layer = "event"
- normalized_state = _normalize(hit.state)
- if normalized_state in {"stale", "superseded", "false"}:
- temporal_state = "superseded"
- elif bool(metadata.get("topic_bucket_current_subject_preserved")) or bool(metadata.get("current_subject_protected")):
- temporal_state = "current"
- elif current_turn_index and int(hit.turn_index or 0) and current_turn_index - int(hit.turn_index) <= 1:
- temporal_state = "current"
- elif normalized_state == "active":
- temporal_state = "stable"
- else:
- temporal_state = "historical"
- query_tokens = set(_tokenize(query))
- hit_tokens = set(_tokenize(" ".join([hit.value, " ".join(hit.anchors), _clean_text(metadata.get("topic_label", ""))])))
- overlap = len(query_tokens & hit_tokens) / max(1, len(query_tokens | hit_tokens))
- candidate_id = _clean_text(hit.memory_id) or f"hit:{index}"
- topic_label = _clean_text(metadata.get("topic_label", "")) or _clean_text(metadata.get("topic_bucket_query_label", ""))
- profile_key = _clean_text(metadata.get("profile_type", "")) or _clean_text(metadata.get("semantic_slot", ""))
- if not profile_key and layer in {"profile", "temporal", "topic_tunnel"}:
- profile_key = topic_label
- semantic_similarity = max(
- 0.0,
- min(
- 1.0,
- float(
- metadata.get(
- "semantic_similarity",
- metadata.get(
- "answer_window_semantic_similarity",
- metadata.get(
- "embedder_similarity",
- metadata.get("dense_similarity", metadata.get("bge_m3_similarity", hit.score)),
- ),
- ),
- )
- or 0.0
- ),
- ),
- )
- return {
- "id": candidate_id,
- "text": hit.value,
- "summary": _clean_text(metadata.get("source_turn_text", "")) or _clean_text(metadata.get("event_summary", "")),
- "topic": topic_label,
- "profile_key": profile_key,
- "resource_key": _clean_text(metadata.get("resource_key", "")) or (_clean_text(hit.slot_key) if layer == "resource" else ""),
- "layer": layer,
- "temporal_state": temporal_state,
- "logic_roles": _dedupe(logic_roles),
- "query_overlap": round(float(overlap), 6),
- "retrieval_score": max(0.0, min(1.0, float(hit.score or 0.0))),
- "graph_score": max(
- 0.0,
- min(
- 1.0,
- float(
- metadata.get(
- "event_score",
- metadata.get("recall_score", metadata.get("hybrid_score", hit.score)),
- )
- or 0.0
- ),
- ),
- ),
- "tunnel_score": max(
- 0.0,
- min(
- 1.0,
- max(
- float(metadata.get("path_tunnel_support_score", 0.0) or 0.0),
- float(metadata.get("event_tunnel_support_score", 0.0) or 0.0),
- float(metadata.get("path_chain_extension_delta_score", 0.0) or 0.0),
- ),
- ),
- ),
- "topic_similarity": max(0.0, min(1.0, float(metadata.get("topic_bucket_overlap", 0.0) or 0.0))),
- "semantic_similarity": semantic_similarity,
- "confidence": max(0.0, min(1.0, float(metadata.get("confidence", hit.score or 0.0) or 0.0))),
- "rank_score": round(1.0 / float(index + 1), 6),
- "age_turns": max(0, int(current_turn_index) - int(hit.turn_index or 0)) if current_turn_index else 0,
- "branch_depth": max(0, len(list(metadata.get("selected_path_ids", []) or []))),
- "contradicts_current": normalized_state in {"stale", "superseded", "false"} or bool(metadata.get("contradicts_current")),
- "is_current": temporal_state == "current",
- }
-
- def _apply_injection_planner_to_hits(
- self,
- query: str,
- hits: Sequence[MemoryHit],
- *,
- top_k: int,
- ) -> Dict[str, Any]:
- normalized_mode = _normalize(self.injection_planner_mode)
- base_metadata: Dict[str, Any] = {
- "injection_planner_enabled": False,
- "injection_planner_mode": normalized_mode or "observe",
- }
- if normalized_mode in _INJECTION_PLANNER_DISABLED_MODES:
- base_metadata["injection_planner_reason"] = "disabled"
- return {"hits": list(hits), "metadata": base_metadata}
- if not hits:
- base_metadata["injection_planner_reason"] = "no_hits"
- return {"hits": list(hits), "metadata": base_metadata}
- model = self._load_injection_planner()
- if model is None or self._injection_planner_config is None:
- base_metadata.update(
- {
- "injection_planner_reason": self._injection_planner_error or "load_failed",
- "injection_planner_model_path": self._injection_planner_resolved_path,
- }
- )
- return {"hits": list(hits), "metadata": base_metadata}
- torch_module = getattr(injection_planner_runtime, "torch", None)
- if torch_module is None:
- base_metadata["injection_planner_reason"] = "torch_unavailable"
- return {"hits": list(hits), "metadata": base_metadata}
- current_turn_index = max(
- [int(getattr(self.graph, "turn_index", 0) or 0), *[int(hit.turn_index or 0) for hit in hits]],
- default=0,
- )
- candidates = [
- self._injection_planner_candidate_from_hit(query, hit, index=index, current_turn_index=current_turn_index)
- for index, hit in enumerate(hits)
- ]
- row = {"id": "runtime_injection_plan", "query": query, "candidates": candidates, "gold": {}}
- try:
- dataset = injection_planner_runtime.InjectionPlannerDataset([row], self._injection_planner_config)
- batch = injection_planner_runtime.collate_injection_batch([dataset[0]])
- device = next(model.parameters()).device
- model_batch = {
- key: value.to(device) if hasattr(value, "to") else value
- for key, value in dict(batch).items()
- }
- with torch_module.no_grad():
- outputs = model(model_batch["features"], model_batch["valid_mask"])
- selection_scores = torch_module.sigmoid(outputs["selection_logits"])[0].detach().cpu().tolist()
- should_inject_score = float(torch_module.sigmoid(outputs["should_inject_logits"])[0].detach().cpu().item())
- mode_probs = torch_module.softmax(outputs["injection_mode_logits"], dim=-1)[0].detach().cpu()
- mode_index = int(torch_module.argmax(mode_probs).item())
- temporal_indices = torch_module.argmax(outputs["temporal_logits"], dim=-1)[0].detach().cpu().tolist()
- logic_scores = torch_module.sigmoid(outputs["logic_logits"])[0].detach().cpu().tolist()
- if bool(self._injection_planner_evidence_role_supported) and "evidence_role_logits" in outputs:
- evidence_role_indices = torch_module.argmax(outputs["evidence_role_logits"], dim=-1)[0].detach().cpu().tolist()
- else:
- evidence_role_indices = [
- injection_planner_runtime.EVIDENCE_ROLES.index("direct_answer")
- for _ in candidates
- ]
- except Exception as exc:
- base_metadata.update(
- {
- "injection_planner_reason": f"inference_failed: {exc}",
- "injection_planner_model_path": self._injection_planner_resolved_path,
- }
- )
- return {"hits": list(hits), "metadata": base_metadata}
- thresholds = {
- "selection_threshold": float(self._injection_planner_thresholds.get("selection_threshold", 0.5)),
- "row_threshold": float(self._injection_planner_thresholds.get("row_threshold", 0.5)),
- "logic_threshold": float(self._injection_planner_thresholds.get("logic_threshold", 0.5)),
- }
- injection_mode = injection_planner_runtime.INJECTION_MODES[mode_index]
- row_allows_injection = should_inject_score >= thresholds["row_threshold"] and injection_mode != "none"
- predictions: Dict[str, Dict[str, Any]] = {}
- for index, candidate in enumerate(candidates):
- logic_roles = [
- role
- for role, score in zip(injection_planner_runtime.LOGIC_ROLES, logic_scores[index])
- if float(score) >= thresholds["logic_threshold"]
- ]
- evidence_role = injection_planner_runtime.EVIDENCE_ROLES[int(evidence_role_indices[index])]
- role_allows_selection = evidence_role not in {"noise", "negative_evidence"}
- predictions[candidate["id"]] = {
- "selection_score": float(selection_scores[index]),
- "selected": bool(
- row_allows_injection
- and role_allows_selection
- and float(selection_scores[index]) >= thresholds["selection_threshold"]
- ),
- "temporal_state": injection_planner_runtime.TEMPORAL_STATES[int(temporal_indices[index])],
- "evidence_role": evidence_role,
- "logic_roles": logic_roles or ["evidence"],
- "candidate_layer": candidate.get("layer", ""),
- "role_allows_selection": bool(role_allows_selection),
- }
- annotated_hits: List[MemoryHit] = []
- for index, hit in enumerate(hits):
- candidate_id = _clean_text(hit.memory_id) or f"hit:{index}"
- prediction = predictions.get(candidate_id, {})
- metadata = dict(hit.metadata or {})
- metadata.update(
- {
- "injection_planner_enabled": True,
- "injection_planner_mode": normalized_mode or "observe",
- "injection_planner_model_path": self._injection_planner_resolved_path,
- "injection_planner_candidate_id": candidate_id,
- "injection_planner_score": round(float(prediction.get("selection_score", 0.0)), 6),
- "injection_planner_selected": bool(prediction.get("selected", False)),
- "injection_planner_temporal_state": _clean_text(prediction.get("temporal_state", "")),
- "injection_planner_evidence_role": _clean_text(prediction.get("evidence_role", "")),
- "injection_planner_role_allows_selection": bool(prediction.get("role_allows_selection", False)),
- "injection_planner_logic_roles": list(prediction.get("logic_roles", []) or []),
- "injection_planner_candidate_layer": _clean_text(prediction.get("candidate_layer", "")),
- "injection_planner_injection_mode": injection_mode,
- "injection_planner_should_inject_score": round(float(should_inject_score), 6),
- "injection_planner_thresholds": dict(thresholds),
- "injection_planner_evidence_role_supported": bool(self._injection_planner_evidence_role_supported),
- }
- )
- planner_score = float(prediction.get("selection_score", 0.0))
- next_score = max(float(hit.score), planner_score) if bool(prediction.get("selected", False)) else float(hit.score)
- annotated_hits.append(
- MemoryHit(
- memory_id=hit.memory_id,
- category=hit.category,
- value=hit.value,
- relation=hit.relation,
- anchors=list(hit.anchors),
- score=round(next_score, 6),
- source_kind=hit.source_kind,
- slot_key=hit.slot_key,
- state=hit.state,
- turn_index=int(hit.turn_index),
- metadata=metadata,
- )
- )
- selected_count = sum(1 for item in annotated_hits if bool((item.metadata or {}).get("injection_planner_selected")))
- if normalized_mode in _INJECTION_PLANNER_FORCE_MODES and selected_count:
- selected_ids = {hit.memory_id for hit in annotated_hits if bool((hit.metadata or {}).get("injection_planner_selected"))}
- planned_hits = [hit for hit in annotated_hits if hit.memory_id in selected_ids]
- elif normalized_mode in _INJECTION_PLANNER_GUIDED_MODES and selected_count:
- planned_hits = sorted(
- annotated_hits,
- key=lambda hit: (
- not bool((hit.metadata or {}).get("injection_planner_selected")),
- -float((hit.metadata or {}).get("injection_planner_score", 0.0) or 0.0),
- -float(hit.score or 0.0),
- ),
- )
- else:
- planned_hits = annotated_hits
- base_metadata.update(
- {
- "injection_planner_enabled": True,
- "injection_planner_reason": "ok",
- "injection_planner_model_path": self._injection_planner_resolved_path,
- "injection_planner_candidate_count": len(candidates),
- "injection_planner_selected_count": int(selected_count),
- "injection_planner_should_inject_score": round(float(should_inject_score), 6),
- "injection_planner_injection_mode": injection_mode,
- "injection_planner_thresholds": dict(thresholds),
- "injection_planner_evidence_role_supported": bool(self._injection_planner_evidence_role_supported),
- "injection_planner_guided": bool(normalized_mode in _INJECTION_PLANNER_GUIDED_MODES | _INJECTION_PLANNER_FORCE_MODES),
- "injection_planner_prediction_ids": [
- candidate_id
- for candidate_id, prediction in predictions.items()
- if bool(prediction.get("selected", False))
- ],
- }
- )
- return {"hits": planned_hits[: max(len(planned_hits), top_k)], "metadata": base_metadata}
-
- def _node_scorer(self) -> LoadedNodeMemoryScorer | None:
- if self.retrieval_mode != "hybrid_node_scored":
- return None
- if self._loaded_node_scorer is not None:
- return self._loaded_node_scorer
- if not self.node_model_path:
- self._node_scorer_error = "node_model_path_missing"
- return None
- try:
- self._loaded_node_scorer = LoadedNodeMemoryScorer(
- node_model_path=Path(self.node_model_path),
- path_model_path=Path(self.path_model_path) if self.path_model_path else None,
- device=self.node_model_device or None,
- )
- except Exception as exc:
- self._node_scorer_error = str(exc)
- self._loaded_node_scorer = None
- return self._loaded_node_scorer
-
- def _apply_writeback_promotions(self, stored_ids: Sequence[str]) -> List[Dict[str, Any]]:
- promotion_events: List[Dict[str, Any]] = []
- for memory_id in stored_ids:
- record = self.graph.records_by_id.get(memory_id)
- if record is None or not isinstance(record.metadata, dict):
- continue
- if _normalize(record.metadata.get("memory_role", "")) != "assistant" or _normalize(record.metadata.get("authority", "")) != "derived":
- continue
- canonical_slot_key = _clean_text(record.metadata.get("canonical_slot_key", ""))
- writeback_class = _clean_text(record.metadata.get("writeback_class", ""))
- if not canonical_slot_key or not writeback_class:
- continue
- support_refs = self._support_ref_union(record.metadata)
- confidence = float(record.confidence or 0.0)
- same_records = [
- item
- for item in self.graph.records_by_id.values()
- if isinstance(item.metadata, dict)
- and _normalize(item.metadata.get("memory_role", "")) == "assistant"
- and _normalize(item.metadata.get("canonical_slot_key", "")) == _normalize(canonical_slot_key)
- and _normalize(item.metadata.get("writeback_class", "")) == _normalize(writeback_class)
- and _normalize(item.value) == _normalize(record.value)
- ]
- qualifying = [item for item in same_records if float(item.confidence or 0.0) >= 0.9]
- distinct_answers = {
- _clean_text(answer_id)
- for item in qualifying
- for answer_id in [*list(item.metadata.get("origin_answer_ids", []) or []), _clean_text(item.metadata.get("origin_answer_id", ""))]
- if _clean_text(answer_id)
- }
- aggregated_support = set()
- for item in same_records:
- aggregated_support.update(self._support_ref_union(item.metadata))
- fast_promotion = writeback_class in {"fact", "state_change"} and confidence >= 0.97 and len(support_refs) >= 3
- standard_promotion = len(distinct_answers) >= 2 and len(aggregated_support) >= 2
- if not (fast_promotion or standard_promotion):
- record.metadata["promotion_state"] = "candidate"
- continue
- source_head = self._source_head_for_canonical(canonical_slot_key)
- blocked_conflict = source_head is not None and _normalize(source_head.value) != _normalize(record.value)
- promoted_slot = f"promoted.{canonical_slot_key}"
- promoted_metadata = {
- **dict(record.metadata or {}),
- "memory_role": "assistant",
- "authority": "promoted",
- "canonical_slot_key": canonical_slot_key,
- "writeback_class": writeback_class,
- "promotion_state": "blocked_conflict" if blocked_conflict else "promoted",
- "support_memory_ids": sorted({*list(record.metadata.get("support_memory_ids", []) or []), *[ref for ref in aggregated_support if ref.startswith("fact:") is False and ref.startswith("path:") is False]}),
- "support_fact_refs": sorted({*list(record.metadata.get("support_fact_refs", []) or []), *[ref for ref in aggregated_support if ref.startswith("fact:")]}),
- "support_path_refs": sorted({*list(record.metadata.get("support_path_refs", []) or []), *[ref for ref in aggregated_support if ref.startswith("path:")]}),
- }
- promoted_record = SessionMemoryRecordV2(
- memory_id=f"{promoted_slot}:{record.turn_index}:promoted",
- category=record.category,
- slot_key=promoted_slot,
- value=record.value,
- relation=record.relation,
- anchor_concepts=list(record.anchor_concepts),
- evidence_anchors=list(record.evidence_anchors),
- salience=max(float(record.salience), 0.78),
- confidence=max(float(record.confidence), 0.9),
- source_kind=f"promoted_{record.source_kind}",
- turn_index=int(record.turn_index),
- state="active",
- metadata=promoted_metadata,
- )
- promoted_ids = self.graph.add_records([promoted_record])
- record.metadata["promotion_state"] = "blocked_conflict" if blocked_conflict else "promoted"
- promotion_events.append(
- {
- "source_memory_id": record.memory_id,
- "promoted_record_ids": list(promoted_ids),
- "canonical_slot_key": canonical_slot_key,
- "writeback_class": writeback_class,
- "promotion_state": record.metadata["promotion_state"],
- "blocked_conflict": bool(blocked_conflict),
- }
- )
- return promotion_events
-
- def _source_head_for_canonical(self, canonical_slot_key: str) -> SessionMemoryRecordV2 | None:
- head_id = self.graph.slot_heads.get(canonical_slot_key)
- if not head_id:
- return None
- record = self.graph.records_by_id.get(head_id)
- if record is None or not isinstance(record.metadata, dict):
- return None
- if _normalize(record.metadata.get("memory_role", "")) != "user" or _normalize(record.metadata.get("authority", "")) != "source":
- return None
- return record
-
- def _support_ref_union(self, metadata: Dict[str, Any]) -> set[str]:
- return {
- *[_clean_text(item) for item in list(metadata.get("support_memory_ids", []) or []) if _clean_text(item)],
- *[_clean_text(item) for item in list(metadata.get("support_fact_refs", []) or []) if _clean_text(item)],
- *[_clean_text(item) for item in list(metadata.get("support_path_refs", []) or []) if _clean_text(item)],
- }
-
- def _hybrid_node_scored_hits(
- self,
- query: str,
- hits: Sequence[MemoryHit],
- *,
- top_k: int,
- public_hits: Sequence[MemoryHit] | None = None,
- ) -> Dict[str, Any]:
- scorer = self._node_scorer()
- if scorer is None:
- return {
- "hits": list(hits),
- "metadata": {
- "retrieval_mode": "heuristic",
- "hybrid_enabled": False,
- "hybrid_error": self._node_scorer_error,
- },
- }
- source_hits = _learnable_graph_hits(self.graph)
- if not source_hits:
- return {
- "hits": list(hits),
- "metadata": {
- "retrieval_mode": "heuristic",
- "hybrid_enabled": False,
- "hybrid_error": "no_learnable_hits",
- },
- }
- has_public_hits = any(
- _is_public_dialog_hit(hit)
- and not bool((hit.metadata or {}).get("profile_layer"))
- for hit in source_hits
- )
- has_generic_hits = any(not _is_public_dialog_hit(hit) for hit in source_hits)
- hybrid_source = "mixed_full_graph"
- if has_public_hits and not has_generic_hits:
- hybrid_source = "public_full_graph"
- elif has_generic_hits and not has_public_hits:
- hybrid_source = "generic_full_graph"
- runtime_graph = _build_runtime_graph_from_hits(query, source_hits)
- grouped_hits = dict(runtime_graph.get("grouped_hits", {}) or {})
- profile_first_payload = _profile_first_hybrid_rescue(
- self.graph,
- query,
- grouped_hits=grouped_hits,
- top_k=top_k,
- )
- profile_first_hits = list(profile_first_payload.get("hits", []) or [])
- profile_first_event_ids = list(profile_first_payload.get("event_ids", []) or [])
- profile_first_memory_ids = list(profile_first_payload.get("memory_ids", []) or [])
- candidate_event_ids = sorted(grouped_hits.keys())
- if not candidate_event_ids:
- return {
- "hits": list(hits),
- "metadata": {
- "retrieval_mode": "heuristic",
- "hybrid_enabled": False,
- "hybrid_error": "no_event_candidates",
- },
- }
- question_analysis = extract_question_features(query)
- hybrid_candidate_limit = min(
- len(candidate_event_ids),
- max(
- _HYBRID_SELECTED_EVENT_FLOOR,
- int(self.candidate_event_k) * 4,
- int(self.support_path_k) * 4,
- int(top_k) * 4,
- ),
- )
- embedder_pre_recall_mode = _normalize(getattr(self, "embedder_pre_recall_mode", ""))
- embedder_pre_recall_enabled = embedder_pre_recall_mode not in _EMBEDDER_INDEX_DISABLED_MODES
- embedder_pre_recall_index_mode = self.embedder_index_recall_mode
- if embedder_pre_recall_mode not in {"1", "true", "yes", "on", "auto", "seed", "candidate", "candidates"}:
- embedder_pre_recall_index_mode = embedder_pre_recall_mode
- pre_embedder_index_payload: Dict[str, Any] = {
- "event_ids": [],
- "metadata": {
- "embedder_pre_recall_enabled": False,
- "embedder_pre_recall_mode": embedder_pre_recall_mode or "off",
- "embedder_pre_recall_index_mode": embedder_pre_recall_index_mode or "off",
- "embedder_pre_recall_event_ids": [],
- },
- }
- pre_embedder_event_ids: List[str] = []
- pre_candidate_event_ids: List[str] = []
- if embedder_pre_recall_enabled:
- pre_embedder_index_payload = _embedder_index_recall_event_ids(
- query,
- grouped_hits=grouped_hits,
- mode=embedder_pre_recall_index_mode,
- limit=self.embedder_pre_recall_k or self.embedder_index_recall_k or hybrid_candidate_limit,
- max_terms=self.write_embedder_index_max_terms,
- )
- pre_embedder_event_ids = list(pre_embedder_index_payload.get("event_ids", []) or [])
- pre_candidate_event_ids = _bounded_event_id_union(
- pre_embedder_event_ids,
- max_items=hybrid_candidate_limit,
- )
- pre_score_kwargs = {
- "graph": runtime_graph,
- "question": query,
- "question_features": question_analysis,
- "rerank_top_k": self.candidate_event_k,
- "event_rerank_mode": self.event_rerank_mode,
- "matrix_event_top_k": self.matrix_event_top_k,
- "support_path_k": self.support_path_k,
- "top_k": top_k,
- }
- if pre_candidate_event_ids:
- pre_score_kwargs["candidate_event_ids"] = pre_candidate_event_ids
- scored = _call_with_supported_kwargs(
- scorer.score_runtime,
- **pre_score_kwargs,
- )
- memory_router_decision = _memory_router_decision(
- scored,
- mode=self.memory_router_mode,
- threshold=self.memory_router_threshold,
- margin=self.memory_router_margin,
- )
- profile_first_router_suppressed = False
- if not _memory_router_allows(memory_router_decision, "profile", "resource"):
- profile_first_hits = []
- profile_first_event_ids = []
- profile_first_memory_ids = []
- profile_first_router_suppressed = True
- initial_recall_event_scores = dict(scored.get("recall_event_scores", {}) or {})
- model_recall_event_ids = _bounded_event_id_union(
- list(scored.get("recall_event_ids", []) or []),
- [
- event_id
- for event_id, _ in sorted(
- initial_recall_event_scores.items(),
- key=lambda item: (-float(item[1]), item[0]),
- )
- ],
- list(scored.get("rerank_candidate_event_ids", []) or []),
- max_items=max(1, len(candidate_event_ids)),
- )
- learned_recall_event_ids = list(model_recall_event_ids)
- symbolic_recall_event_ids = _symbolic_recall_event_ids(
- query,
- runtime_graph,
- grouped_hits=grouped_hits,
- limit=hybrid_candidate_limit,
- )
- symbolic_recall_event_ids = _bounded_event_id_union(
- profile_first_event_ids,
- symbolic_recall_event_ids,
- max_items=hybrid_candidate_limit,
- )
- if pre_embedder_event_ids:
- embedder_index_payload = pre_embedder_index_payload
- else:
- embedder_index_payload = _embedder_index_recall_event_ids(
- query,
- grouped_hits=grouped_hits,
- mode=self.embedder_index_recall_mode,
- limit=self.embedder_index_recall_k or hybrid_candidate_limit,
- max_terms=self.write_embedder_index_max_terms,
- )
- embedder_index_event_ids = list(embedder_index_payload.get("event_ids", []) or [])
- embedder_index_metadata = dict(embedder_index_payload.get("metadata", {}) or {})
- embedder_index_metadata.update(
- {
- "embedder_pre_recall_enabled": bool(pre_candidate_event_ids),
- "embedder_pre_recall_mode": embedder_pre_recall_mode or "off",
- "embedder_pre_recall_index_mode": embedder_pre_recall_index_mode or "off",
- "embedder_pre_recall_event_ids": list(pre_embedder_event_ids),
- "embedder_pre_recall_candidate_event_ids": list(pre_candidate_event_ids),
- "embedder_pre_recall_candidate_count": int(len(pre_candidate_event_ids)),
- }
- )
- hybrid_candidate_event_ids = _bounded_event_id_union(
- profile_first_event_ids,
- embedder_index_event_ids,
- learned_recall_event_ids,
- symbolic_recall_event_ids,
- max_items=hybrid_candidate_limit,
- )
- learned_candidate_event_ids = _bounded_event_id_union(
- learned_recall_event_ids,
- max_items=hybrid_candidate_limit,
- )
- learned_recall_event_id_set = set(learned_candidate_event_ids)
- hybrid_candidate_union_added_event_ids = [
- event_id
- for event_id in hybrid_candidate_event_ids
- if event_id not in learned_recall_event_id_set
- ]
- hybrid_candidate_union_priority_changed = list(hybrid_candidate_event_ids) != list(learned_candidate_event_ids)
- hybrid_candidate_union_rescored = False
- if hybrid_candidate_union_added_event_ids or (embedder_index_event_ids and hybrid_candidate_union_priority_changed):
- scored = _call_with_supported_kwargs(
- scorer.score_runtime,
- graph=runtime_graph,
- question=query,
- question_features=question_analysis,
- candidate_event_ids=hybrid_candidate_event_ids,
- rerank_top_k=self.candidate_event_k,
- event_rerank_mode=self.event_rerank_mode,
- matrix_event_top_k=self.matrix_event_top_k,
- support_path_k=self.support_path_k,
- top_k=top_k,
- )
- hybrid_candidate_union_rescored = True
- memory_router_decision = _memory_router_decision(
- scored,
- mode=self.memory_router_mode,
- threshold=self.memory_router_threshold,
- margin=self.memory_router_margin,
- )
- recall_event_scores = dict(scored.get("recall_event_scores", {}) or {})
- rerank_candidate_event_ids = list(scored.get("rerank_candidate_event_ids", []) or [])
- base_event_scores = dict(scored.get("base_event_scores", {}) or {})
- rerank_event_scores = dict(scored.get("rerank_event_scores", {}) or {})
- calibrated_event_scores = dict(scored.get("calibrated_event_scores", {}) or {})
- matrix_event_scores = dict(scored.get("matrix_event_scores", {}) or {})
- event_fusion_delta_scores = dict(scored.get("event_fusion_delta_scores", {}) or {})
- event_tunnel_support_scores = dict(scored.get("event_tunnel_support_scores", {}) or {})
- event_tunnel_delta_scores = dict(scored.get("event_tunnel_delta_scores", {}) or {})
- tri_maze_event_reverse_scores = dict(scored.get("tri_maze_event_reverse_scores", {}) or {})
- tri_maze_event_boundary_scores = dict(scored.get("tri_maze_event_boundary_scores", {}) or {})
- tri_maze_event_reverse_relations = dict(scored.get("tri_maze_event_reverse_relations", {}) or {})
- matrix_rerank_event_ids = list(scored.get("matrix_rerank_event_ids", []) or [])
- matrix_enabled = bool(scored.get("matrix_enabled", False))
- rerank_path_scores = dict(scored.get("rerank_path_scores", {}) or {})
- matrix_path_scores = dict(scored.get("matrix_path_scores", {}) or {})
- tri_maze_path_reverse_scores = dict(scored.get("tri_maze_path_reverse_scores", {}) or {})
- tri_maze_path_boundary_scores = dict(scored.get("tri_maze_path_boundary_scores", {}) or {})
- tri_maze_path_reverse_relations = dict(scored.get("tri_maze_path_reverse_relations", {}) or {})
- matrix_path_rerank_ids = list(scored.get("matrix_path_rerank_ids", []) or [])
- matrix_path_enabled = bool(scored.get("matrix_path_enabled", False))
- fusion_enabled = bool(scored.get("fusion_enabled", False))
- event_fusion_enabled = bool(scored.get("event_fusion_enabled", fusion_enabled))
- path_fusion_enabled = bool(scored.get("path_fusion_enabled", fusion_enabled))
- event_calibration_enabled = bool(scored.get("event_calibration_enabled", False))
- path_calibration_enabled = bool(scored.get("path_calibration_enabled", False))
- event_tunnel_enabled = bool(scored.get("event_tunnel_enabled", False))
- path_tunnel_enabled = bool(scored.get("path_tunnel_enabled", False))
- final_event_fusion_enabled = bool(scored.get("final_event_fusion_enabled", False))
- final_path_fusion_enabled = bool(scored.get("final_path_fusion_enabled", False))
- decision_fusion_enabled = bool(scored.get("decision_fusion_enabled", False))
- decision_score_source = _clean_text(scored.get("decision_score_source", ""))
- event_scores = dict(scored.get("event_scores", {}) or {})
- base_path_scores = dict(scored.get("base_path_scores", {}) or {})
- calibrated_path_scores = dict(scored.get("calibrated_path_scores", {}) or {})
- path_fusion_delta_scores = dict(scored.get("path_fusion_delta_scores", {}) or {})
- path_tunnel_support_scores = dict(scored.get("path_tunnel_support_scores", {}) or {})
- path_tunnel_delta_scores = dict(scored.get("path_tunnel_delta_scores", {}) or {})
- path_model_scores = dict(scored.get("path_model_scores", {}) or {})
- path_chain_extension_delta_scores = dict(scored.get("path_chain_extension_delta_scores", {}) or {})
- path_chain_extended_scores = dict(scored.get("path_chain_extended_scores", {}) or {})
- path_chain_extension_enabled = bool(scored.get("path_chain_extension_enabled", False))
- answer_type_scores = dict(scored.get("answer_type_scores", {}) or {})
- selected_event_ids_from_model = list(scored.get("selected_event_ids", []) or [])
- selected_path_ids_from_model = list(scored.get("selected_path_ids", []) or [])
- focused_answer_type_from_model = _clean_text(scored.get("focused_answer_type", ""))
- path_scores = dict(scored.get("path_scores", {}) or {})
- temporal_scores = dict(scored.get("temporal_scores", {}) or {})
- runtime_paths = {_clean_text(path.get("id", "")): dict(path) for path in list(runtime_graph.get("paths", []) or [])}
- answer_plan_scores_raw = dict(scored.get("answer_plan_scores", {}) or {})
-
- def _answer_plan_score_map(role: str) -> Dict[str, float]:
- raw_scores = answer_plan_scores_raw.get(role, {})
- if not isinstance(raw_scores, Mapping):
- return {}
- normalized: Dict[str, float] = {}
- for raw_event_id, raw_score in raw_scores.items():
- event_id = _clean_text(raw_event_id)
- if not event_id:
- continue
- try:
- normalized[event_id] = float(raw_score or 0.0)
- except (TypeError, ValueError):
- normalized[event_id] = 0.0
- return normalized
-
- answer_plan_selected_scores = _answer_plan_score_map("selected")
- answer_plan_current_scores = _answer_plan_score_map("current")
- answer_plan_historical_scores = _answer_plan_score_map("historical")
- answer_plan_suppressed_scores = _answer_plan_score_map("suppressed")
- answer_plan_scores = {
- "selected": dict(answer_plan_selected_scores),
- "current": dict(answer_plan_current_scores),
- "historical": dict(answer_plan_historical_scores),
- "suppressed": dict(answer_plan_suppressed_scores),
- }
- try:
- answer_plan_event_selection_threshold = float(
- os.getenv("TMCRA_ANSWER_PLAN_EVENT_SELECTION_THRESHOLD", "0.50") or 0.50
- )
- except (TypeError, ValueError):
- answer_plan_event_selection_threshold = 0.50
- try:
- answer_plan_event_selection_top_k = int(os.getenv("TMCRA_ANSWER_PLAN_EVENT_SELECTION_TOP_K", "0") or 0)
- except (TypeError, ValueError):
- answer_plan_event_selection_top_k = 0
- if answer_plan_event_selection_top_k <= 0:
- answer_plan_event_selection_top_k = max(top_k, self.support_path_k * 2, _HYBRID_SELECTED_EVENT_FLOOR)
- answer_plan_available_event_ids = {
- _clean_text(event_id)
- for event_id in [
- *list(grouped_hits.keys()),
- *[_clean_text(path.get("event_id", "")) for path in runtime_paths.values()],
- ]
- if _clean_text(event_id)
- }
- answer_plan_event_rows: List[tuple[str, float, float, float, float, float]] = []
- answer_plan_event_ids = set(answer_plan_selected_scores) | set(answer_plan_current_scores) | set(answer_plan_historical_scores)
- for event_id in answer_plan_event_ids:
- if answer_plan_available_event_ids and event_id not in answer_plan_available_event_ids:
- continue
- selected_score = float(answer_plan_selected_scores.get(event_id, 0.0) or 0.0)
- current_score = float(answer_plan_current_scores.get(event_id, 0.0) or 0.0)
- historical_score = float(answer_plan_historical_scores.get(event_id, 0.0) or 0.0)
- suppressed_score = float(answer_plan_suppressed_scores.get(event_id, 0.0) or 0.0)
- support_score = max(selected_score, current_score)
- adjusted_score = support_score - max(0.0, suppressed_score - support_score) * 0.5
- if support_score < answer_plan_event_selection_threshold or suppressed_score > support_score:
- continue
- answer_plan_event_rows.append(
- (event_id, adjusted_score, support_score, selected_score, current_score, historical_score)
- )
- answer_plan_event_rows.sort(
- key=lambda row: (-float(row[1]), -float(row[2]), -float(row[3]), -float(row[4]), row[0])
- )
- answer_plan_raw_ranked_event_ids = [
- event_id for event_id, *_ in answer_plan_event_rows[: max(1, answer_plan_event_selection_top_k)]
- ]
- try:
- answer_plan_promotion_min_margin = float(
- os.getenv("TMCRA_ANSWER_PLAN_PROMOTION_MIN_MARGIN", "0.02") or 0.02
- )
- except (TypeError, ValueError):
- answer_plan_promotion_min_margin = 0.02
- answer_plan_promotion_score_margin = 0.0
- if len(answer_plan_event_rows) == 1:
- answer_plan_promotion_score_margin = float(answer_plan_event_rows[0][1])
- answer_plan_promotion_enabled = True
- elif len(answer_plan_event_rows) > 1:
- comparison_index = min(len(answer_plan_event_rows) - 1, max(1, min(answer_plan_event_selection_top_k, 5) - 1))
- answer_plan_promotion_score_margin = float(answer_plan_event_rows[0][1]) - float(answer_plan_event_rows[comparison_index][1])
- answer_plan_promotion_enabled = answer_plan_promotion_score_margin >= answer_plan_promotion_min_margin
- else:
- answer_plan_promotion_enabled = False
- answer_plan_ranked_event_ids = list(answer_plan_raw_ranked_event_ids if answer_plan_promotion_enabled else [])
- answer_plan_selected_event_ids = [
- event_id
- for event_id in answer_plan_ranked_event_ids
- if float(answer_plan_selected_scores.get(event_id, 0.0) or 0.0)
- >= answer_plan_event_selection_threshold
- ]
- answer_plan_current_event_ids = [
- event_id
- for event_id in answer_plan_ranked_event_ids
- if float(answer_plan_current_scores.get(event_id, 0.0) or 0.0)
- >= answer_plan_event_selection_threshold
- ]
- answer_plan_support_scores = {
- event_id: round(float(support_score), 6)
- for event_id, _, support_score, *_ in answer_plan_event_rows
- }
- answer_plan_adjusted_scores = {
- event_id: round(float(adjusted_score), 6)
- for event_id, adjusted_score, *_ in answer_plan_event_rows
- }
- answer_plan_rank_lookup = {
- event_id: rank for rank, event_id in enumerate(answer_plan_ranked_event_ids, start=1)
- }
- answer_plan_ranked_event_id_set = set(answer_plan_ranked_event_ids)
-
- def _answer_plan_hit_metadata(event_id: str) -> Dict[str, Any]:
- clean_event_id = _clean_text(event_id)
- selected_score = float(answer_plan_selected_scores.get(clean_event_id, 0.0) or 0.0)
- current_score = float(answer_plan_current_scores.get(clean_event_id, 0.0) or 0.0)
- historical_score = float(answer_plan_historical_scores.get(clean_event_id, 0.0) or 0.0)
- suppressed_score = float(answer_plan_suppressed_scores.get(clean_event_id, 0.0) or 0.0)
- support_score = max(selected_score, current_score)
- return {
- "answer_plan_score": round(float(support_score), 6),
- "answer_plan_selected_score": round(float(selected_score), 6),
- "answer_plan_current_score": round(float(current_score), 6),
- "answer_plan_historical_score": round(float(historical_score), 6),
- "answer_plan_suppressed_score": round(float(suppressed_score), 6),
- "answer_plan_adjusted_score": round(float(answer_plan_adjusted_scores.get(clean_event_id, 0.0)), 6),
- "answer_plan_selected": bool(clean_event_id in answer_plan_ranked_event_id_set),
- "answer_plan_rank": int(answer_plan_rank_lookup.get(clean_event_id, 0) or 0),
- }
-
- embedder_fusion_mode = _normalize(self.embedder_fusion_mode)
- embedder_fusion_enabled = (
- embedder_fusion_mode not in _EMBEDDER_INDEX_DISABLED_MODES
- and bool(embedder_index_event_ids)
- and bool(self.embedder_fusion_top_k > 0)
- and bool(self.embedder_fusion_weight > 0.0)
- )
- embedder_fusion_applied_event_scores: Dict[str, float] = {}
- embedder_fusion_boosts: Dict[str, float] = {}
- embedder_event_scores = dict(embedder_index_metadata.get("embedder_index_event_scores", {}) or {})
- if embedder_fusion_enabled and embedder_event_scores:
- ranked_embedder_events = [
- (event_id, float(score))
- for event_id, score in sorted(
- embedder_event_scores.items(),
- key=lambda item: (-float(item[1]), item[0]),
- )
- if _clean_text(event_id)
- ][: max(1, int(self.embedder_fusion_top_k))]
- for rank, (event_id, embedder_score) in enumerate(ranked_embedder_events, start=1):
- if embedder_score < float(self.embedder_fusion_score_floor):
- continue
- rank_bonus = max(0.0, 0.08 - (rank - 1) * 0.006)
- boost = min(
- float(self.embedder_fusion_max_boost),
- (float(self.embedder_fusion_weight) * embedder_score) + rank_bonus,
- )
- current_score = max(
- float(recall_event_scores.get(event_id, 0.0) or 0.0),
- float(event_scores.get(event_id, 0.0) or 0.0),
- float(base_event_scores.get(event_id, 0.0) or 0.0),
- float(rerank_event_scores.get(event_id, 0.0) or 0.0),
- float(calibrated_event_scores.get(event_id, 0.0) or 0.0),
- )
- fused_score = current_score + boost
- recall_event_scores[event_id] = max(float(recall_event_scores.get(event_id, 0.0) or 0.0), fused_score)
- event_scores[event_id] = max(float(event_scores.get(event_id, 0.0) or 0.0), fused_score)
- base_event_scores[event_id] = max(float(base_event_scores.get(event_id, 0.0) or 0.0), fused_score)
- rerank_event_scores[event_id] = max(float(rerank_event_scores.get(event_id, 0.0) or 0.0), fused_score)
- calibrated_event_scores[event_id] = max(float(calibrated_event_scores.get(event_id, 0.0) or 0.0), fused_score)
- embedder_fusion_applied_event_scores[event_id] = round(fused_score, 6)
- embedder_fusion_boosts[event_id] = round(boost, 6)
- if embedder_fusion_applied_event_scores:
- decision_score_source = f"{decision_score_source or 'learned_decision_fusion'}+embedder_fusion"
- recall_event_ids = [
- event_id
- for event_id, _ in sorted(recall_event_scores.items(), key=lambda item: (-float(item[1]), item[0]))
- ][: self.candidate_event_k]
- if profile_first_event_ids:
- profile_first_score_by_event = {
- _clean_text((hit.metadata or {}).get("profile_first_hybrid_event_id", "")) or _runtime_event_key(hit): float(hit.score)
- for hit in profile_first_hits
- }
- for rank, event_id in enumerate(profile_first_event_ids, start=1):
- if not event_id:
- continue
- floor = max(0.92, float(profile_first_score_by_event.get(event_id, 0.0))) + max(0.0, 0.04 - (rank * 0.005))
- recall_event_scores[event_id] = max(float(recall_event_scores.get(event_id, 0.0)), floor)
- event_scores[event_id] = max(float(event_scores.get(event_id, 0.0)), floor)
- base_event_scores[event_id] = max(float(base_event_scores.get(event_id, 0.0)), floor)
- rerank_event_scores[event_id] = max(float(rerank_event_scores.get(event_id, 0.0)), floor)
- calibrated_event_scores[event_id] = max(float(calibrated_event_scores.get(event_id, 0.0)), floor)
- for path_id, path in runtime_paths.items():
- event_id = _clean_text(path.get("event_id", ""))
- path_type = _clean_text(path.get("type", ""))
- if event_id not in set(profile_first_event_ids):
- continue
- if path_type not in {"speaker_event_profile", "speaker_event_source_turn", "speaker_event_status"}:
- continue
- floor = max(0.90, float(event_scores.get(event_id, 0.0)))
- path_scores[path_id] = max(float(path_scores.get(path_id, 0.0)), floor)
- base_path_scores[path_id] = max(float(base_path_scores.get(path_id, 0.0)), floor)
- calibrated_path_scores[path_id] = max(float(calibrated_path_scores.get(path_id, 0.0)), floor)
- path_model_scores[path_id] = max(float(path_model_scores.get(path_id, 0.0)), floor)
- selection_consistency_repaired = False
- selection_consistency_reason = ""
- model_focused_answer_type = _clean_text(focused_answer_type_from_model)
- embedder_fusion_selected_event_ids: List[str] = []
- embedder_fusion_selected_path_ids: List[str] = []
- if decision_fusion_enabled and (selected_path_ids_from_model or selected_event_ids_from_model):
- base_selected_path_limit = max(1, min(max(1, self.support_path_k), max(1, top_k)))
- selected_path_ids = [
- path_id
- for path_id in selected_path_ids_from_model
- if _clean_text(path_id) in runtime_paths
- ]
- if not selected_path_ids:
- selected_path_ids = [
- path_id
- for path_id, _ in sorted(
- ((path_id, float(score or 0.0)) for path_id, score in path_scores.items()),
- key=lambda item: (-float(item[1]), item[0]),
- )
- ][:base_selected_path_limit]
- focused_answer_type = _reconciled_focused_answer_type(question_analysis, answer_type_scores, focused_answer_type_from_model)
- tunnel_rescue_path_ids: List[str] = []
- tunnel_rescue_pre_filter_path_ids: List[str] = []
- path_utility_direct_support_path_ids: List[str] = []
- path_utility_contrast_support_path_ids: List[str] = []
- path_utility_latent_context_path_ids: List[str] = []
- path_utility_drift_noise_path_ids: List[str] = []
- path_utility_roles: Dict[str, str] = {}
- path_utility_reasons: Dict[str, str] = {}
- path_utility_scores: Dict[str, float] = {}
- path_utility_overlap_tokens: Dict[str, List[str]] = {}
- path_utility_anchor_event_ids: List[str] = []
- path_utility_anchor_subject_signatures: List[str] = []
- tunnel_rescue_score_threshold = float(self.path_tunnel_rescue_score_floor)
- tunnel_rescue_candidate_count = 0
- tunnel_rescue_filtered_count = 0
- if (
- self.path_tunnel_rescue_k > 0
- and path_tunnel_support_scores
- and _memory_router_allows(memory_router_decision, "path_tunnel", "topic_tunnel")
- ):
- selected_path_id_set = set(selected_path_ids)
- candidate_scores = [
- float(score or 0.0)
- for path_id, score in path_tunnel_support_scores.items()
- if _clean_text(path_id) in runtime_paths
- ]
- tunnel_rescue_candidate_count = len(candidate_scores)
- if candidate_scores and self.path_tunnel_rescue_min_score_margin > 0.0:
- sorted_scores = sorted(candidate_scores)
- median_score = sorted_scores[len(sorted_scores) // 2]
- tunnel_rescue_score_threshold = max(
- tunnel_rescue_score_threshold,
- median_score + float(self.path_tunnel_rescue_min_score_margin),
- )
- event_turn_indices = [
- _runtime_event_turn_index_from_id(_clean_text(path.get("event_id", "")))
- for path in runtime_paths.values()
- ]
- current_turn_index = max(
- [int(getattr(self.graph, "turn_index", 0) or 0), *[turn for turn in event_turn_indices if turn > 0]],
- default=0,
- )
- ranked_tunnel_path_ids = []
- for path_id, score in sorted(
- (
- (path_id, float(score or 0.0))
- for path_id, score in path_tunnel_support_scores.items()
- if _clean_text(path_id) in runtime_paths
- ),
- key=lambda item: (-float(item[1]), item[0]),
- ):
- if path_id in selected_path_id_set or score < tunnel_rescue_score_threshold:
- continue
- path_event_turn = _runtime_event_turn_index_from_id(
- _clean_text(runtime_paths.get(path_id, {}).get("event_id", ""))
- )
- path_age = max(0, current_turn_index - path_event_turn) if current_turn_index and path_event_turn else 0
- if self.path_tunnel_rescue_min_age > 0 and path_age < self.path_tunnel_rescue_min_age:
- continue
- ranked_tunnel_path_ids.append(path_id)
- tunnel_rescue_filtered_count = len(ranked_tunnel_path_ids)
- tunnel_rescue_pre_filter_path_ids = ranked_tunnel_path_ids[: max(self.path_tunnel_rescue_k, self.path_tunnel_rescue_k * 4)]
- utility_gate = _path_utility_gate(
- tunnel_rescue_pre_filter_path_ids,
- query=query,
- runtime_graph=runtime_graph,
- runtime_paths=runtime_paths,
- grouped_hits=grouped_hits,
- selected_path_ids=selected_path_ids,
- selected_event_ids_from_model=selected_event_ids_from_model,
- path_scores=path_scores,
- path_tunnel_support_scores=path_tunnel_support_scores,
- question_analysis=question_analysis,
- focused_answer_type=focused_answer_type,
- score_threshold=tunnel_rescue_score_threshold,
- limit=self.path_tunnel_rescue_k,
- )
- tunnel_rescue_path_ids = list(utility_gate.get("injected_path_ids", []) or [])
- path_utility_direct_support_path_ids = list(utility_gate.get("direct_support_path_ids", []) or [])
- path_utility_contrast_support_path_ids = list(utility_gate.get("contrast_support_path_ids", []) or [])
- path_utility_latent_context_path_ids = list(utility_gate.get("latent_context_path_ids", []) or [])
- path_utility_drift_noise_path_ids = list(utility_gate.get("drift_noise_path_ids", []) or [])
- path_utility_roles = dict(utility_gate.get("roles", {}) or {})
- path_utility_reasons = dict(utility_gate.get("reasons", {}) or {})
- path_utility_scores = dict(utility_gate.get("scores", {}) or {})
- path_utility_overlap_tokens = dict(utility_gate.get("overlap_tokens", {}) or {})
- path_utility_anchor_event_ids = list(utility_gate.get("anchor_event_ids", []) or [])
- path_utility_anchor_subject_signatures = list(utility_gate.get("anchor_subject_signatures", []) or [])
- if tunnel_rescue_path_ids:
- selected_path_ids = _dedupe([*selected_path_ids, *tunnel_rescue_path_ids])
- effective_selected_path_limit = base_selected_path_limit + len(tunnel_rescue_path_ids)
- selected_event_ids = _dedupe(
- [
- *[
- _clean_text(runtime_paths.get(path_id, {}).get("event_id", ""))
- for path_id in selected_path_ids
- if _clean_text(runtime_paths.get(path_id, {}).get("event_id", ""))
- ],
- *[
- _clean_text(event_id)
- for event_id in selected_event_ids_from_model
- if _clean_text(event_id)
- ],
- ]
- )
- if answer_plan_ranked_event_ids:
- selected_event_ids = _dedupe([*selected_event_ids, *answer_plan_ranked_event_ids])
- if not selected_event_ids:
- selected_event_ids = [
- event_id
- for event_id, _ in sorted(
- ((event_id, float(score or 0.0)) for event_id, score in event_scores.items()),
- key=lambda item: (-float(item[1]), item[0]),
- )
- ][: max(1, min(self.candidate_event_k, max(top_k, _HYBRID_SELECTED_EVENT_FLOOR)))]
- repaired_path_ids, selection_consistency_repaired, selection_consistency_reason = _repair_selected_paths_for_focus(
- selected_path_ids,
- runtime_paths=runtime_paths,
- selected_event_ids=selected_event_ids,
- path_scores=path_scores,
- event_scores=event_scores,
- temporal_scores=temporal_scores,
- question_analysis=question_analysis,
- answer_type_scores=answer_type_scores,
- focused_answer_type=focused_answer_type,
- limit=max(1, effective_selected_path_limit),
- )
- if selection_consistency_repaired:
- selected_path_ids = repaired_path_ids
- selected_event_ids = _dedupe(
- [
- *[
- _clean_text(runtime_paths.get(path_id, {}).get("event_id", ""))
- for path_id in selected_path_ids
- if _clean_text(runtime_paths.get(path_id, {}).get("event_id", ""))
- ],
- *[
- _clean_text(event_id)
- for event_id in selected_event_ids_from_model
- if _clean_text(event_id)
- ],
- ]
- )
- if answer_plan_ranked_event_ids:
- selected_event_ids = _dedupe([*selected_event_ids, *answer_plan_ranked_event_ids])
- if profile_first_event_ids:
- selected_event_ids = _dedupe([*profile_first_event_ids, *selected_event_ids])
- if embedder_fusion_applied_event_scores and self.embedder_fusion_select_k > 0:
- ranked_fusion_event_ids = [
- event_id
- for event_id in embedder_index_event_ids
- if event_id in embedder_fusion_applied_event_scores
- ][: max(1, int(self.embedder_fusion_select_k))]
- selected_event_ids = _dedupe([*ranked_fusion_event_ids, *selected_event_ids])
- selected_path_id_set = set(selected_path_ids)
- selected_embedder_path_ids: List[str] = []
- for event_id in ranked_fusion_event_ids:
- candidate_paths = [
- (path_id, path)
- for path_id, path in runtime_paths.items()
- if _clean_text(path.get("event_id", "")) == event_id
- ]
- candidate_paths.sort(
- key=lambda item: (
- int(_clean_text(item[1].get("type", "")) in {"speaker_event_source_turn", "speaker_event_profile", "speaker_event_status", "speaker_event_time"}),
- float(path_scores.get(item[0], 0.0) or 0.0),
- item[0],
- ),
- reverse=True,
- )
- for path_id, _ in candidate_paths:
- if path_id in selected_path_id_set:
- continue
- selected_embedder_path_ids.append(path_id)
- selected_path_id_set.add(path_id)
- embedder_fusion_selected_path_ids.append(path_id)
- break
- if selected_embedder_path_ids:
- selected_path_ids = _dedupe([*selected_embedder_path_ids, *selected_path_ids])
- embedder_fusion_selected_event_ids = list(ranked_fusion_event_ids)
- if answer_plan_ranked_event_ids:
- selected_event_ids = _dedupe([*selected_event_ids, *answer_plan_ranked_event_ids])
- final_hits: List[MemoryHit] = []
- seen_memory_ids = set()
- selected_event_id_set = set(selected_event_ids)
- for path_id in selected_path_ids:
- path = runtime_paths.get(path_id, {})
- event_id = _clean_text(path.get("event_id", ""))
- if event_id not in selected_event_id_set:
- continue
- path_type = _clean_text(path.get("type", ""))
- support_node_id = _path_support_node_id(path)
- support_hit = _support_hit_for_path(path_type, grouped_hits.get(event_id, []))
- event_hit = _representative_event_hit(
- [*grouped_hits.get(event_id, []), *_event_record_hits_from_graph(self.graph, event_id)],
- query=query,
- )
- decision_score = round(float(path_scores.get(path_id, 0.0)), 6)
- candidate_hits: List[tuple[MemoryHit | None, float]] = [(support_hit, decision_score)]
- if event_hit is not None and (support_hit is None or event_hit.memory_id != support_hit.memory_id):
- candidate_hits.append((event_hit, max(0.0, decision_score - 0.0001)))
- for raw_hit, hit_score in candidate_hits:
- if raw_hit is None or raw_hit.memory_id in seen_memory_ids:
- continue
- seen_memory_ids.add(raw_hit.memory_id)
- recall_score = float(recall_event_scores.get(event_id, event_scores.get(event_id, 0.0)))
- event_score = float(event_scores.get(event_id, raw_hit.score))
- temporal_score = float(temporal_scores.get(support_node_id, 0.0))
- metadata = dict(raw_hit.metadata or {})
- metadata.update(
- {
- "event_id": event_id,
- **_answer_plan_hit_metadata(event_id),
- "path_id": path_id,
- "base_event_score": round(float(base_event_scores.get(event_id, event_score)), 6),
- "rerank_event_score": round(float(rerank_event_scores.get(event_id, event_score)), 6),
- "calibrated_event_score": round(float(calibrated_event_scores.get(event_id, event_score)), 6),
- "matrix_event_score": round(float(matrix_event_scores.get(event_id, 0.0)), 6),
- "event_fusion_delta_score": round(float(event_fusion_delta_scores.get(event_id, 0.0)), 6),
- "event_tunnel_support_score": round(float(event_tunnel_support_scores.get(event_id, 0.0)), 6),
- "event_tunnel_delta_score": round(float(event_tunnel_delta_scores.get(event_id, 0.0)), 6),
- "tri_maze_event_reverse_score": round(float(tri_maze_event_reverse_scores.get(event_id, 0.0)), 6),
- "tri_maze_event_boundary_score": round(float(tri_maze_event_boundary_scores.get(event_id, 0.0)), 6),
- "tri_maze_event_reverse_relation": round(float(tri_maze_event_reverse_relations.get(event_id, 0.0)), 6),
- "matrix_enabled": matrix_enabled,
- "event_calibration_enabled": event_calibration_enabled,
- "path_calibration_enabled": path_calibration_enabled,
- "event_tunnel_enabled": event_tunnel_enabled,
- "path_tunnel_enabled": path_tunnel_enabled,
- "final_event_fusion_enabled": final_event_fusion_enabled,
- "final_path_fusion_enabled": final_path_fusion_enabled,
- "decision_fusion_enabled": True,
- "event_fusion_enabled": event_fusion_enabled,
- "path_fusion_enabled": path_fusion_enabled,
- "event_score": round(event_score, 6),
- "recall_score": round(recall_score, 6),
- "path_score": round(float(path_scores.get(path_id, hit_score)), 6),
- "base_path_score": round(float(base_path_scores.get(path_id, path_scores.get(path_id, hit_score))), 6),
- "calibrated_path_score": round(float(calibrated_path_scores.get(path_id, path_scores.get(path_id, hit_score))), 6),
- "path_fusion_delta_score": round(float(path_fusion_delta_scores.get(path_id, 0.0)), 6),
- "path_tunnel_support_score": round(float(path_tunnel_support_scores.get(path_id, 0.0)), 6),
- "path_tunnel_delta_score": round(float(path_tunnel_delta_scores.get(path_id, 0.0)), 6),
- "path_model_score": round(float(path_model_scores.get(path_id, path_scores.get(path_id, hit_score))), 6),
- "path_chain_extension_enabled": path_chain_extension_enabled,
- "path_chain_extension_delta_score": round(float(path_chain_extension_delta_scores.get(path_id, 0.0)), 6),
- "path_chain_extended_score": round(float(path_chain_extended_scores.get(path_id, path_scores.get(path_id, hit_score))), 6),
- "tri_maze_path_reverse_score": round(float(tri_maze_path_reverse_scores.get(path_id, 0.0)), 6),
- "tri_maze_path_boundary_score": round(float(tri_maze_path_boundary_scores.get(path_id, 0.0)), 6),
- "tri_maze_path_reverse_relation": round(float(tri_maze_path_reverse_relations.get(path_id, 0.0)), 6),
- "effective_path_score": round(float(path_scores.get(path_id, hit_score)), 6),
- "temporal_score": round(temporal_score, 6),
- "raw_public_score": round(float(raw_hit.score), 6),
- "hybrid_score": round(hit_score, 6),
- "hybrid_score_source": decision_score_source or "learned_decision_fusion",
- "evidence_snippet_role": "selected_path_support" if raw_hit is support_hit else "selected_path_event",
- "selected_event_ids": list(selected_event_ids),
- "selected_path_ids": list(selected_path_ids),
- "path_tunnel_rescue_enabled": bool(self.path_tunnel_rescue_k > 0),
- "path_tunnel_rescue_k": int(self.path_tunnel_rescue_k),
- "path_tunnel_rescue_score_floor": round(float(self.path_tunnel_rescue_score_floor), 6),
- "path_tunnel_rescue_min_age": int(self.path_tunnel_rescue_min_age),
- "path_tunnel_rescue_min_score_margin": round(float(self.path_tunnel_rescue_min_score_margin), 6),
- "path_tunnel_rescue_score_threshold": round(float(tunnel_rescue_score_threshold), 6),
- "path_tunnel_rescue_candidate_count": int(tunnel_rescue_candidate_count),
- "path_tunnel_rescue_filtered_count": int(tunnel_rescue_filtered_count),
- "path_tunnel_rescue_path_ids": list(tunnel_rescue_path_ids),
- "path_utility_gate_enabled": bool(self.path_tunnel_rescue_k > 0),
- "path_utility_pre_filter_path_ids": list(tunnel_rescue_pre_filter_path_ids),
- "path_utility_injected_path_ids": list(tunnel_rescue_path_ids),
- "path_utility_direct_support_path_ids": list(path_utility_direct_support_path_ids),
- "path_utility_contrast_support_path_ids": list(path_utility_contrast_support_path_ids),
- "path_utility_latent_context_path_ids": list(path_utility_latent_context_path_ids),
- "path_utility_drift_noise_path_ids": list(path_utility_drift_noise_path_ids),
- "path_utility_roles": dict(path_utility_roles),
- "path_utility_reasons": dict(path_utility_reasons),
- "path_utility_scores": dict(path_utility_scores),
- "path_utility_overlap_tokens": dict(path_utility_overlap_tokens),
- "path_utility_anchor_event_ids": list(path_utility_anchor_event_ids),
- "path_utility_anchor_subject_signatures": list(path_utility_anchor_subject_signatures),
- "tunnel_recall_pre_filter_count": int(len(tunnel_rescue_pre_filter_path_ids)),
- "tunnel_usable_post_filter_count": int(len(tunnel_rescue_path_ids)),
- "model_focused_answer_type": model_focused_answer_type,
- "selection_consistency_repaired": bool(selection_consistency_repaired),
- "selection_consistency_reason": selection_consistency_reason,
- }
- )
- final_hits.append(
- MemoryHit(
- memory_id=raw_hit.memory_id,
- category=raw_hit.category,
- value=raw_hit.value,
- relation=raw_hit.relation,
- anchors=list(raw_hit.anchors),
- score=round(hit_score, 6),
- source_kind=raw_hit.source_kind,
- slot_key=raw_hit.slot_key,
- state=raw_hit.state,
- turn_index=int(raw_hit.turn_index),
- metadata=metadata,
- )
- )
- for event_rank, event_id in enumerate(selected_event_ids, start=1):
- event_hit = _representative_event_hit(
- [*grouped_hits.get(event_id, []), *_event_record_hits_from_graph(self.graph, event_id)],
- query=query,
- )
- if event_hit is None or event_hit.memory_id in seen_memory_ids:
- continue
- seen_memory_ids.add(event_hit.memory_id)
- recall_score = float(recall_event_scores.get(event_id, event_hit.score))
- event_score = float(event_scores.get(event_id, event_hit.score))
- metadata = dict(event_hit.metadata or {})
- metadata.update(
- {
- "event_id": event_id,
- **_answer_plan_hit_metadata(event_id),
- "path_id": "",
- "base_event_score": round(float(base_event_scores.get(event_id, event_score)), 6),
- "rerank_event_score": round(float(rerank_event_scores.get(event_id, event_score)), 6),
- "calibrated_event_score": round(float(calibrated_event_scores.get(event_id, event_score)), 6),
- "matrix_event_score": round(float(matrix_event_scores.get(event_id, 0.0)), 6),
- "event_fusion_delta_score": round(float(event_fusion_delta_scores.get(event_id, 0.0)), 6),
- "event_tunnel_support_score": round(float(event_tunnel_support_scores.get(event_id, 0.0)), 6),
- "event_tunnel_delta_score": round(float(event_tunnel_delta_scores.get(event_id, 0.0)), 6),
- "tri_maze_event_reverse_score": round(float(tri_maze_event_reverse_scores.get(event_id, 0.0)), 6),
- "tri_maze_event_boundary_score": round(float(tri_maze_event_boundary_scores.get(event_id, 0.0)), 6),
- "tri_maze_event_reverse_relation": round(float(tri_maze_event_reverse_relations.get(event_id, 0.0)), 6),
- "matrix_enabled": matrix_enabled,
- "event_calibration_enabled": event_calibration_enabled,
- "path_calibration_enabled": path_calibration_enabled,
- "event_tunnel_enabled": event_tunnel_enabled,
- "path_tunnel_enabled": path_tunnel_enabled,
- "final_event_fusion_enabled": final_event_fusion_enabled,
- "final_path_fusion_enabled": final_path_fusion_enabled,
- "decision_fusion_enabled": True,
- "event_fusion_enabled": event_fusion_enabled,
- "path_fusion_enabled": path_fusion_enabled,
- "event_score": round(event_score, 6),
- "recall_score": round(recall_score, 6),
- "path_score": 0.0,
- "base_path_score": 0.0,
- "calibrated_path_score": 0.0,
- "path_fusion_delta_score": 0.0,
- "path_tunnel_support_score": 0.0,
- "path_tunnel_delta_score": 0.0,
- "path_model_score": 0.0,
- "path_chain_extension_enabled": path_chain_extension_enabled,
- "path_chain_extension_delta_score": 0.0,
- "path_chain_extended_score": 0.0,
- "effective_path_score": 0.0,
- "temporal_score": 0.0,
- "raw_public_score": round(float(event_hit.score), 6),
- "hybrid_score": round(event_score, 6),
- "hybrid_score_source": decision_score_source or "learned_final_event_fusion",
- "evidence_snippet_role": "selected_event_representative",
- "selected_event_rank": int(event_rank),
- "selected_event_ids": list(selected_event_ids),
- "selected_path_ids": list(selected_path_ids),
- "path_tunnel_rescue_enabled": bool(self.path_tunnel_rescue_k > 0),
- "path_tunnel_rescue_k": int(self.path_tunnel_rescue_k),
- "path_tunnel_rescue_score_floor": round(float(self.path_tunnel_rescue_score_floor), 6),
- "path_tunnel_rescue_min_age": int(self.path_tunnel_rescue_min_age),
- "path_tunnel_rescue_min_score_margin": round(float(self.path_tunnel_rescue_min_score_margin), 6),
- "path_tunnel_rescue_score_threshold": round(float(tunnel_rescue_score_threshold), 6),
- "path_tunnel_rescue_candidate_count": int(tunnel_rescue_candidate_count),
- "path_tunnel_rescue_filtered_count": int(tunnel_rescue_filtered_count),
- "path_tunnel_rescue_path_ids": list(tunnel_rescue_path_ids),
- "path_utility_gate_enabled": bool(self.path_tunnel_rescue_k > 0),
- "path_utility_pre_filter_path_ids": list(tunnel_rescue_pre_filter_path_ids),
- "path_utility_injected_path_ids": list(tunnel_rescue_path_ids),
- "path_utility_direct_support_path_ids": list(path_utility_direct_support_path_ids),
- "path_utility_contrast_support_path_ids": list(path_utility_contrast_support_path_ids),
- "path_utility_latent_context_path_ids": list(path_utility_latent_context_path_ids),
- "path_utility_drift_noise_path_ids": list(path_utility_drift_noise_path_ids),
- "path_utility_roles": dict(path_utility_roles),
- "path_utility_reasons": dict(path_utility_reasons),
- "path_utility_scores": dict(path_utility_scores),
- "path_utility_overlap_tokens": dict(path_utility_overlap_tokens),
- "path_utility_anchor_event_ids": list(path_utility_anchor_event_ids),
- "path_utility_anchor_subject_signatures": list(path_utility_anchor_subject_signatures),
- "tunnel_recall_pre_filter_count": int(len(tunnel_rescue_pre_filter_path_ids)),
- "tunnel_usable_post_filter_count": int(len(tunnel_rescue_path_ids)),
- "model_focused_answer_type": model_focused_answer_type,
- "selection_consistency_repaired": bool(selection_consistency_repaired),
- "selection_consistency_reason": selection_consistency_reason,
- }
- )
- final_hits.append(
- MemoryHit(
- memory_id=event_hit.memory_id,
- category=event_hit.category,
- value=event_hit.value,
- relation=event_hit.relation,
- anchors=list(event_hit.anchors),
- score=round(event_score, 6),
- source_kind=event_hit.source_kind,
- slot_key=event_hit.slot_key,
- state=event_hit.state,
- turn_index=int(event_hit.turn_index),
- metadata=metadata,
- )
- )
- if not final_hits:
- final_hits = list(hits)
- if profile_first_hits:
- final_hits = _inject_profile_first_hits(
- final_hits,
- profile_first_hits,
- selected_event_ids=selected_event_ids,
- selected_path_ids=selected_path_ids,
- )
- final_hits = _coverage_preserving_final_hits(final_hits, selected_event_ids=selected_event_ids, top_k=top_k)
- final_hit_event_ids = _event_ids_from_hits(final_hits)
- final_missing_selected_event_ids = [event_id for event_id in selected_event_ids if event_id not in set(final_hit_event_ids)]
- return {
- "hits": final_hits,
- "metadata": {
- "retrieval_mode": "hybrid_node_scored",
- "hybrid_enabled": True,
- "hybrid_source": hybrid_source,
- **memory_router_decision,
- "profile_first_router_suppressed": bool(profile_first_router_suppressed),
- "recall_event_ids": list(recall_event_ids),
- "learned_recall_event_ids": list(learned_recall_event_ids),
- "model_recall_event_ids": list(model_recall_event_ids),
- "symbolic_recall_event_ids": list(symbolic_recall_event_ids),
- "embedder_index_recall_event_ids": list(embedder_index_event_ids),
- **embedder_index_metadata,
- "embedder_fusion_mode": embedder_fusion_mode or "off",
- "embedder_fusion_enabled": bool(embedder_fusion_enabled),
- "embedder_fusion_weight": round(float(self.embedder_fusion_weight), 6),
- "embedder_fusion_score_floor": round(float(self.embedder_fusion_score_floor), 6),
- "embedder_fusion_top_k": int(self.embedder_fusion_top_k),
- "embedder_fusion_select_k": int(self.embedder_fusion_select_k),
- "embedder_fusion_max_boost": round(float(self.embedder_fusion_max_boost), 6),
- "embedder_fusion_event_scores": dict(embedder_fusion_applied_event_scores),
- "embedder_fusion_boosts": dict(embedder_fusion_boosts),
- "embedder_fusion_selected_event_ids": list(embedder_fusion_selected_event_ids),
- "embedder_fusion_selected_path_ids": list(embedder_fusion_selected_path_ids),
- "profile_first_hybrid_enabled": bool(profile_first_event_ids),
- "profile_first_event_ids": list(profile_first_event_ids),
- "profile_first_memory_ids": list(profile_first_memory_ids),
- "hybrid_candidate_event_ids": list(hybrid_candidate_event_ids),
- "hybrid_candidate_union_enabled": True,
- "hybrid_candidate_union_rescored": bool(hybrid_candidate_union_rescored),
- "hybrid_candidate_union_added_event_ids": list(hybrid_candidate_union_added_event_ids),
- "hybrid_candidate_union_priority_changed": bool(hybrid_candidate_union_priority_changed),
- "rerank_candidate_event_ids": list(rerank_candidate_event_ids),
- "base_event_scores": dict(base_event_scores),
- "rerank_event_scores": dict(rerank_event_scores),
- "calibrated_event_scores": dict(calibrated_event_scores),
- "matrix_event_scores": dict(matrix_event_scores),
- "event_fusion_delta_scores": dict(event_fusion_delta_scores),
- "event_tunnel_support_scores": dict(event_tunnel_support_scores),
- "event_tunnel_delta_scores": dict(event_tunnel_delta_scores),
- "tri_maze_event_reverse_scores": dict(tri_maze_event_reverse_scores),
- "tri_maze_event_boundary_scores": dict(tri_maze_event_boundary_scores),
- "tri_maze_event_reverse_relations": dict(tri_maze_event_reverse_relations),
- "matrix_rerank_event_ids": list(matrix_rerank_event_ids),
- "matrix_enabled": matrix_enabled,
- "rerank_path_scores": dict(rerank_path_scores),
- "matrix_path_scores": dict(matrix_path_scores),
- "tri_maze_path_reverse_scores": dict(tri_maze_path_reverse_scores),
- "tri_maze_path_boundary_scores": dict(tri_maze_path_boundary_scores),
- "tri_maze_path_reverse_relations": dict(tri_maze_path_reverse_relations),
- "matrix_path_rerank_ids": list(matrix_path_rerank_ids),
- "matrix_path_enabled": matrix_path_enabled,
- "fusion_enabled": fusion_enabled,
- "event_calibration_enabled": event_calibration_enabled,
- "path_calibration_enabled": path_calibration_enabled,
- "event_tunnel_enabled": event_tunnel_enabled,
- "path_tunnel_enabled": path_tunnel_enabled,
- "final_event_fusion_enabled": final_event_fusion_enabled,
- "final_path_fusion_enabled": final_path_fusion_enabled,
- "decision_fusion_enabled": True,
- "decision_score_source": decision_score_source or "learned_decision_fusion",
- "event_fusion_enabled": event_fusion_enabled,
- "path_fusion_enabled": path_fusion_enabled,
- "selected_event_ids": list(selected_event_ids),
- "path_rescue_event_ids": [],
- "selected_path_ids": list(selected_path_ids),
- "path_tunnel_rescue_enabled": bool(self.path_tunnel_rescue_k > 0),
- "path_tunnel_rescue_k": int(self.path_tunnel_rescue_k),
- "path_tunnel_rescue_score_floor": round(float(self.path_tunnel_rescue_score_floor), 6),
- "path_tunnel_rescue_min_age": int(self.path_tunnel_rescue_min_age),
- "path_tunnel_rescue_min_score_margin": round(float(self.path_tunnel_rescue_min_score_margin), 6),
- "path_tunnel_rescue_score_threshold": round(float(tunnel_rescue_score_threshold), 6),
- "path_tunnel_rescue_candidate_count": int(tunnel_rescue_candidate_count),
- "path_tunnel_rescue_filtered_count": int(tunnel_rescue_filtered_count),
- "path_tunnel_rescue_path_ids": list(tunnel_rescue_path_ids),
- "path_utility_gate_enabled": bool(self.path_tunnel_rescue_k > 0),
- "path_utility_pre_filter_path_ids": list(tunnel_rescue_pre_filter_path_ids),
- "path_utility_injected_path_ids": list(tunnel_rescue_path_ids),
- "path_utility_direct_support_path_ids": list(path_utility_direct_support_path_ids),
- "path_utility_contrast_support_path_ids": list(path_utility_contrast_support_path_ids),
- "path_utility_latent_context_path_ids": list(path_utility_latent_context_path_ids),
- "path_utility_drift_noise_path_ids": list(path_utility_drift_noise_path_ids),
- "path_utility_roles": dict(path_utility_roles),
- "path_utility_reasons": dict(path_utility_reasons),
- "path_utility_scores": dict(path_utility_scores),
- "path_utility_overlap_tokens": dict(path_utility_overlap_tokens),
- "path_utility_anchor_event_ids": list(path_utility_anchor_event_ids),
- "path_utility_anchor_subject_signatures": list(path_utility_anchor_subject_signatures),
- "tunnel_recall_pre_filter_count": int(len(tunnel_rescue_pre_filter_path_ids)),
- "tunnel_usable_post_filter_count": int(len(tunnel_rescue_path_ids)),
- "final_hit_event_ids": list(final_hit_event_ids),
- "final_hit_dia_ids": _dia_ids_from_hits(final_hits),
- "final_missing_selected_event_ids": list(final_missing_selected_event_ids),
- "selected_event_count": int(len(selected_event_ids)),
- "selected_path_count": int(len(selected_path_ids)),
- "temporal_scores": dict(temporal_scores),
- "recall_event_scores": dict(recall_event_scores),
- "event_scores": dict(event_scores),
- "base_path_scores": dict(base_path_scores),
- "calibrated_path_scores": dict(calibrated_path_scores),
- "path_fusion_delta_scores": dict(path_fusion_delta_scores),
- "path_tunnel_support_scores": dict(path_tunnel_support_scores),
- "path_tunnel_delta_scores": dict(path_tunnel_delta_scores),
- "path_model_scores": dict(path_model_scores),
- "path_chain_extension_enabled": path_chain_extension_enabled,
- "path_chain_extension_delta_scores": dict(path_chain_extension_delta_scores),
- "path_chain_extended_scores": dict(path_chain_extended_scores),
- "effective_path_scores": dict(path_scores),
- "path_scores": dict(path_scores),
- "answer_type_scores": dict(answer_type_scores),
- "answer_plan_scores": dict(answer_plan_scores),
- "answer_plan_support_scores": dict(answer_plan_support_scores),
- "answer_plan_adjusted_scores": dict(answer_plan_adjusted_scores),
- "answer_plan_raw_ranked_event_ids": list(answer_plan_raw_ranked_event_ids),
- "answer_plan_ranked_event_ids": list(answer_plan_ranked_event_ids),
- "answer_plan_selected_event_ids": list(answer_plan_selected_event_ids),
- "answer_plan_current_event_ids": list(answer_plan_current_event_ids),
- "answer_plan_promotion_enabled": bool(answer_plan_promotion_enabled),
- "answer_plan_promotion_score_margin": round(float(answer_plan_promotion_score_margin), 6),
- "answer_plan_promotion_min_margin": round(float(answer_plan_promotion_min_margin), 6),
- "answer_plan_event_selection_threshold": round(float(answer_plan_event_selection_threshold), 6),
- "answer_plan_event_selection_top_k": int(answer_plan_event_selection_top_k),
- "focused_answer_type": focused_answer_type,
- "model_focused_answer_type": model_focused_answer_type,
- "selection_consistency_repaired": bool(selection_consistency_repaired),
- "selection_consistency_reason": selection_consistency_reason,
- "preferred_path_types": [],
- },
- }
- dominant_answer_type = _dominant_answer_type(question_analysis, answer_type_scores)
- preferred_path_types = _answer_type_preferred_path_types(question_analysis, answer_type_scores)
- if bool(memory_router_decision.get("memory_router_guided")):
- temporal_focus = _memory_router_allows(memory_router_decision, "temporal")
- else:
- temporal_focus = dominant_answer_type == "time" or bool(question_analysis.get("is_temporal", False))
- router_profile_focus = bool(memory_router_decision.get("memory_router_guided")) and _memory_router_allows(
- memory_router_decision,
- "profile",
- "resource",
- )
- learned_event_available = bool(event_scores)
- learned_path_available = bool(path_scores)
- ranked_events = [
- event_id
- for event_id, _ in sorted(
- (
- (event_id, event_scores.get(event_id, recall_event_scores.get(event_id, 0.0)))
- for event_id in recall_event_ids
- ),
- key=lambda item: (-float(item[1]), item[0]),
- )
- ]
- base_selected_event_count = min(
- len(ranked_events),
- max(
- self.support_path_k * 2,
- min(self.candidate_event_k, max(top_k, _HYBRID_SELECTED_EVENT_FLOOR)),
- ),
- )
- effective_path_scores = dict(path_scores)
- if not learned_path_available:
- effective_path_scores = {
- path_id: _calibrated_path_score(
- path=runtime_paths.get(path_id, {}),
- base_score=float(score),
- temporal_scores=temporal_scores,
- question_analysis=question_analysis,
- answer_type_scores=answer_type_scores,
- )
- for path_id, score in path_scores.items()
- }
- ranked_path_ids_all = [
- path_id
- for path_id, _ in sorted(
- ((path_id, float(score)) for path_id, score in effective_path_scores.items()),
- key=lambda item: (-float(item[1]), item[0]),
- )
- ]
- path_rescue_count = min(
- len(ranked_path_ids_all),
- max(self.support_path_k * 2, min(self.candidate_event_k, max(1, top_k))),
- )
- path_rescue_event_ids = _dedupe(
- _clean_text(runtime_paths.get(path_id, {}).get("event_id", ""))
- for path_id in ranked_path_ids_all[: max(1, path_rescue_count)]
- )
- selected_event_ids = _dedupe(
- [
- *path_rescue_event_ids,
- *ranked_events[: max(1, base_selected_event_count)],
- ]
- )
- if profile_first_event_ids:
- selected_event_ids = _dedupe([*profile_first_event_ids, *selected_event_ids])
- if answer_plan_ranked_event_ids:
- selected_event_ids = _dedupe([*selected_event_ids, *answer_plan_ranked_event_ids])
- selected_event_id_set = set(selected_event_ids)
- ranked_path_ids = [
- path_id
- for path_id in ranked_path_ids_all
- if _clean_text(runtime_paths.get(path_id, {}).get("event_id", "")) in selected_event_id_set
- ]
- if temporal_focus:
- focused_time_path_ids = [
- path_id
- for path_id in ranked_path_ids
- if _clean_text(runtime_paths.get(path_id, {}).get("type", "")) == "speaker_event_time"
- ]
- if focused_time_path_ids:
- ranked_path_ids = focused_time_path_ids
- path_limit_cap = _HYBRID_SELECTED_PATH_CAP
- if temporal_focus:
- path_limit_cap = _HYBRID_TEMPORAL_PATH_CAP
- elif dominant_answer_type == "profile" or router_profile_focus:
- path_limit_cap = _HYBRID_PROFILE_PATH_CAP
- selected_path_count = min(
- len(ranked_path_ids),
- max(1, min(max(1, self.support_path_k), min(max(1, top_k), path_limit_cap))),
- )
- selected_path_ids = ranked_path_ids[: max(1, selected_path_count)]
- final_hits: List[MemoryHit] = []
- seen_memory_ids = set()
- temporal_event_hits_added = 0
- for path_id in selected_path_ids:
- path = runtime_paths.get(path_id, {})
- event_id = _clean_text(path.get("event_id", ""))
- support_node_id = _path_support_node_id(path)
- path_type = _clean_text(path.get("type", ""))
- support_hit = _support_hit_for_path(path_type, grouped_hits.get(event_id, []))
- event_hit = _representative_event_hit(
- [*grouped_hits.get(event_id, []), *_event_record_hits_from_graph(self.graph, event_id)],
- query=query,
- )
- hit_pairs: List[tuple[MemoryHit | None, float]] = [(support_hit, path_scores.get(path_id, 0.0))]
- if not temporal_focus:
- hit_pairs.append((event_hit, path_scores.get(path_id, 0.0)))
- elif support_hit is None and event_hit is not None:
- hit_pairs.append((event_hit, path_scores.get(path_id, 0.0)))
- elif path_type == "speaker_event_time" and event_hit is not None and temporal_event_hits_added < 1:
- hit_pairs.append((event_hit, path_scores.get(path_id, 0.0)))
- temporal_event_hits_added += 1
- for raw_hit, path_score in hit_pairs:
- if raw_hit is None or raw_hit.memory_id in seen_memory_ids:
- continue
- seen_memory_ids.add(raw_hit.memory_id)
- recall_score = float(recall_event_scores.get(event_id, event_scores.get(event_id, 0.0)))
- event_score = float(event_scores.get(event_id, raw_hit.score))
- temporal_score = float(temporal_scores.get(support_node_id, 0.0))
- effective_path_score = float(effective_path_scores.get(path_id, path_score))
- hybrid_score = round(effective_path_score, 6) if learned_path_available else round(
- (0.55 * event_score)
- + (0.20 * float(path_score))
- + (0.15 * recall_score)
- + (0.10 * temporal_score),
- 6,
- )
- metadata = dict(raw_hit.metadata or {})
- metadata.update(
- {
- "event_id": event_id,
- **_answer_plan_hit_metadata(event_id),
- "path_id": path_id,
- "base_event_score": round(float(base_event_scores.get(event_id, event_score)), 6),
- "rerank_event_score": round(float(rerank_event_scores.get(event_id, event_score)), 6),
- "calibrated_event_score": round(float(calibrated_event_scores.get(event_id, event_score)), 6),
- "matrix_event_score": round(float(matrix_event_scores.get(event_id, 0.0)), 6),
- "event_fusion_delta_score": round(float(event_fusion_delta_scores.get(event_id, 0.0)), 6),
- "event_tunnel_support_score": round(float(event_tunnel_support_scores.get(event_id, 0.0)), 6),
- "event_tunnel_delta_score": round(float(event_tunnel_delta_scores.get(event_id, 0.0)), 6),
- "matrix_enabled": matrix_enabled,
- "event_calibration_enabled": event_calibration_enabled,
- "path_calibration_enabled": path_calibration_enabled,
- "event_tunnel_enabled": event_tunnel_enabled,
- "path_tunnel_enabled": path_tunnel_enabled,
- "final_event_fusion_enabled": final_event_fusion_enabled,
- "final_path_fusion_enabled": final_path_fusion_enabled,
- "decision_fusion_enabled": False,
- "event_fusion_enabled": event_fusion_enabled,
- "path_fusion_enabled": path_fusion_enabled,
- "event_score": round(event_score, 6),
- "recall_score": round(recall_score, 6),
- "path_score": round(float(path_score), 6),
- "base_path_score": round(float(base_path_scores.get(path_id, path_score)), 6),
- "calibrated_path_score": round(float(calibrated_path_scores.get(path_id, effective_path_score)), 6),
- "path_fusion_delta_score": round(float(path_fusion_delta_scores.get(path_id, 0.0)), 6),
- "path_tunnel_support_score": round(float(path_tunnel_support_scores.get(path_id, 0.0)), 6),
- "path_tunnel_delta_score": round(float(path_tunnel_delta_scores.get(path_id, 0.0)), 6),
- "path_model_score": round(float(path_model_scores.get(path_id, path_score)), 6),
- "path_chain_extension_enabled": path_chain_extension_enabled,
- "path_chain_extension_delta_score": round(float(path_chain_extension_delta_scores.get(path_id, 0.0)), 6),
- "path_chain_extended_score": round(float(path_chain_extended_scores.get(path_id, effective_path_score)), 6),
- "effective_path_score": round(effective_path_score, 6),
- "temporal_score": round(temporal_score, 6),
- "raw_public_score": round(float(raw_hit.score), 6),
- "hybrid_score": hybrid_score,
- "hybrid_score_source": (
- "learned_path_fusion"
- if path_fusion_enabled
- else "learned_path_score"
- if learned_path_available
- else "heuristic_mix"
- ),
- "evidence_snippet_role": "selected_path_support" if raw_hit is support_hit else "selected_path_event",
- "selected_event_ids": list(selected_event_ids),
- "selected_path_ids": list(selected_path_ids),
- }
- )
- final_hits.append(
- MemoryHit(
- memory_id=raw_hit.memory_id,
- category=raw_hit.category,
- value=raw_hit.value,
- relation=raw_hit.relation,
- anchors=list(raw_hit.anchors),
- score=hybrid_score,
- source_kind=raw_hit.source_kind,
- slot_key=raw_hit.slot_key,
- state=raw_hit.state,
- turn_index=int(raw_hit.turn_index),
- metadata=metadata,
- )
- )
- for event_rank, event_id in enumerate(selected_event_ids, start=1):
- event_hit = _representative_event_hit(
- [*grouped_hits.get(event_id, []), *_event_record_hits_from_graph(self.graph, event_id)],
- query=query,
- )
- if event_hit is None or event_hit.memory_id in seen_memory_ids:
- continue
- seen_memory_ids.add(event_hit.memory_id)
- recall_score = float(recall_event_scores.get(event_id, event_hit.score))
- event_score = float(event_scores.get(event_id, event_hit.score))
- hybrid_score = round(event_score, 6) if learned_event_available else round(
- (0.7 * event_score) + (0.3 * recall_score),
- 6,
- )
- metadata = dict(event_hit.metadata or {})
- metadata.update(
- {
- "event_id": event_id,
- **_answer_plan_hit_metadata(event_id),
- "path_id": "",
- "base_event_score": round(float(base_event_scores.get(event_id, event_score)), 6),
- "rerank_event_score": round(float(rerank_event_scores.get(event_id, event_score)), 6),
- "calibrated_event_score": round(float(calibrated_event_scores.get(event_id, event_score)), 6),
- "matrix_event_score": round(float(matrix_event_scores.get(event_id, 0.0)), 6),
- "event_fusion_delta_score": round(float(event_fusion_delta_scores.get(event_id, 0.0)), 6),
- "event_tunnel_support_score": round(float(event_tunnel_support_scores.get(event_id, 0.0)), 6),
- "event_tunnel_delta_score": round(float(event_tunnel_delta_scores.get(event_id, 0.0)), 6),
- "matrix_enabled": matrix_enabled,
- "event_calibration_enabled": event_calibration_enabled,
- "path_calibration_enabled": path_calibration_enabled,
- "event_tunnel_enabled": event_tunnel_enabled,
- "path_tunnel_enabled": path_tunnel_enabled,
- "final_event_fusion_enabled": final_event_fusion_enabled,
- "final_path_fusion_enabled": final_path_fusion_enabled,
- "decision_fusion_enabled": False,
- "event_fusion_enabled": event_fusion_enabled,
- "path_fusion_enabled": path_fusion_enabled,
- "event_score": round(event_score, 6),
- "recall_score": round(recall_score, 6),
- "path_score": 0.0,
- "base_path_score": 0.0,
- "calibrated_path_score": 0.0,
- "path_fusion_delta_score": 0.0,
- "path_tunnel_support_score": 0.0,
- "path_tunnel_delta_score": 0.0,
- "path_model_score": 0.0,
- "path_chain_extension_enabled": path_chain_extension_enabled,
- "path_chain_extension_delta_score": 0.0,
- "path_chain_extended_score": 0.0,
- "effective_path_score": 0.0,
- "temporal_score": 0.0,
- "raw_public_score": round(float(event_hit.score), 6),
- "hybrid_score": hybrid_score,
- "hybrid_score_source": (
- "learned_event_fusion"
- if event_fusion_enabled
- else "learned_event_score"
- if learned_event_available
- else "heuristic_event_mix"
- ),
- "evidence_snippet_role": "selected_event_representative",
- "selected_event_rank": int(event_rank),
- "selected_event_ids": list(selected_event_ids),
- "selected_path_ids": list(selected_path_ids),
- }
- )
- final_hits.append(
- MemoryHit(
- memory_id=event_hit.memory_id,
- category=event_hit.category,
- value=event_hit.value,
- relation=event_hit.relation,
- anchors=list(event_hit.anchors),
- score=hybrid_score,
- source_kind=event_hit.source_kind,
- slot_key=event_hit.slot_key,
- state=event_hit.state,
- turn_index=int(event_hit.turn_index),
- metadata=metadata,
- )
- )
- if not final_hits:
- final_hits = list(hits)
- if profile_first_hits:
- final_hits = _inject_profile_first_hits(
- final_hits,
- profile_first_hits,
- selected_event_ids=selected_event_ids,
- selected_path_ids=selected_path_ids,
- )
- final_hits = _coverage_preserving_final_hits(final_hits, selected_event_ids=selected_event_ids, top_k=top_k)
- final_hit_event_ids = _event_ids_from_hits(final_hits)
- final_missing_selected_event_ids = [event_id for event_id in selected_event_ids if event_id not in set(final_hit_event_ids)]
- return {
- "hits": final_hits,
- "metadata": {
- "retrieval_mode": "hybrid_node_scored",
- "hybrid_enabled": True,
- "hybrid_source": hybrid_source,
- **memory_router_decision,
- "profile_first_router_suppressed": bool(profile_first_router_suppressed),
- "recall_event_ids": list(recall_event_ids),
- "learned_recall_event_ids": list(learned_recall_event_ids),
- "model_recall_event_ids": list(model_recall_event_ids),
- "symbolic_recall_event_ids": list(symbolic_recall_event_ids),
- "embedder_index_recall_event_ids": list(embedder_index_event_ids),
- **embedder_index_metadata,
- "embedder_fusion_mode": embedder_fusion_mode or "off",
- "embedder_fusion_enabled": bool(embedder_fusion_enabled),
- "embedder_fusion_weight": round(float(self.embedder_fusion_weight), 6),
- "embedder_fusion_score_floor": round(float(self.embedder_fusion_score_floor), 6),
- "embedder_fusion_top_k": int(self.embedder_fusion_top_k),
- "embedder_fusion_select_k": int(self.embedder_fusion_select_k),
- "embedder_fusion_max_boost": round(float(self.embedder_fusion_max_boost), 6),
- "embedder_fusion_event_scores": dict(embedder_fusion_applied_event_scores),
- "embedder_fusion_boosts": dict(embedder_fusion_boosts),
- "embedder_fusion_selected_event_ids": list(embedder_fusion_selected_event_ids),
- "embedder_fusion_selected_path_ids": list(embedder_fusion_selected_path_ids),
- "profile_first_hybrid_enabled": bool(profile_first_event_ids),
- "profile_first_event_ids": list(profile_first_event_ids),
- "profile_first_memory_ids": list(profile_first_memory_ids),
- "hybrid_candidate_event_ids": list(hybrid_candidate_event_ids),
- "hybrid_candidate_union_enabled": True,
- "hybrid_candidate_union_rescored": bool(hybrid_candidate_union_rescored),
- "hybrid_candidate_union_added_event_ids": list(hybrid_candidate_union_added_event_ids),
- "hybrid_candidate_union_priority_changed": bool(hybrid_candidate_union_priority_changed),
- "rerank_candidate_event_ids": list(rerank_candidate_event_ids),
- "base_event_scores": dict(base_event_scores),
- "rerank_event_scores": dict(rerank_event_scores),
- "calibrated_event_scores": dict(calibrated_event_scores),
- "matrix_event_scores": dict(matrix_event_scores),
- "event_fusion_delta_scores": dict(event_fusion_delta_scores),
- "event_tunnel_support_scores": dict(event_tunnel_support_scores),
- "event_tunnel_delta_scores": dict(event_tunnel_delta_scores),
- "tri_maze_event_reverse_scores": dict(tri_maze_event_reverse_scores),
- "tri_maze_event_boundary_scores": dict(tri_maze_event_boundary_scores),
- "tri_maze_event_reverse_relations": dict(tri_maze_event_reverse_relations),
- "matrix_rerank_event_ids": list(matrix_rerank_event_ids),
- "matrix_enabled": matrix_enabled,
- "rerank_path_scores": dict(rerank_path_scores),
- "matrix_path_scores": dict(matrix_path_scores),
- "tri_maze_path_reverse_scores": dict(tri_maze_path_reverse_scores),
- "tri_maze_path_boundary_scores": dict(tri_maze_path_boundary_scores),
- "tri_maze_path_reverse_relations": dict(tri_maze_path_reverse_relations),
- "matrix_path_rerank_ids": list(matrix_path_rerank_ids),
- "matrix_path_enabled": matrix_path_enabled,
- "fusion_enabled": fusion_enabled,
- "event_calibration_enabled": event_calibration_enabled,
- "path_calibration_enabled": path_calibration_enabled,
- "event_tunnel_enabled": event_tunnel_enabled,
- "path_tunnel_enabled": path_tunnel_enabled,
- "final_event_fusion_enabled": final_event_fusion_enabled,
- "final_path_fusion_enabled": final_path_fusion_enabled,
- "decision_fusion_enabled": False,
- "decision_score_source": "",
- "event_fusion_enabled": event_fusion_enabled,
- "path_fusion_enabled": path_fusion_enabled,
- "selected_event_ids": list(selected_event_ids),
- "path_rescue_event_ids": list(path_rescue_event_ids),
- "selected_path_ids": list(selected_path_ids),
- "final_hit_event_ids": list(final_hit_event_ids),
- "final_hit_dia_ids": _dia_ids_from_hits(final_hits),
- "final_missing_selected_event_ids": list(final_missing_selected_event_ids),
- "selected_event_count": int(len(selected_event_ids)),
- "selected_path_count": int(len(selected_path_ids)),
- "temporal_scores": dict(temporal_scores),
- "recall_event_scores": dict(recall_event_scores),
- "event_scores": dict(event_scores),
- "base_path_scores": dict(base_path_scores),
- "calibrated_path_scores": dict(calibrated_path_scores),
- "path_fusion_delta_scores": dict(path_fusion_delta_scores),
- "path_tunnel_support_scores": dict(path_tunnel_support_scores),
- "path_tunnel_delta_scores": dict(path_tunnel_delta_scores),
- "path_model_scores": dict(path_model_scores),
- "path_chain_extension_enabled": path_chain_extension_enabled,
- "path_chain_extension_delta_scores": dict(path_chain_extension_delta_scores),
- "path_chain_extended_scores": dict(path_chain_extended_scores),
- "effective_path_scores": dict(effective_path_scores),
- "path_scores": dict(path_scores),
- "answer_type_scores": dict(answer_type_scores),
- "answer_plan_scores": dict(answer_plan_scores),
- "answer_plan_support_scores": dict(answer_plan_support_scores),
- "answer_plan_adjusted_scores": dict(answer_plan_adjusted_scores),
- "answer_plan_raw_ranked_event_ids": list(answer_plan_raw_ranked_event_ids),
- "answer_plan_ranked_event_ids": list(answer_plan_ranked_event_ids),
- "answer_plan_selected_event_ids": list(answer_plan_selected_event_ids),
- "answer_plan_current_event_ids": list(answer_plan_current_event_ids),
- "answer_plan_promotion_enabled": bool(answer_plan_promotion_enabled),
- "answer_plan_promotion_score_margin": round(float(answer_plan_promotion_score_margin), 6),
- "answer_plan_promotion_min_margin": round(float(answer_plan_promotion_min_margin), 6),
- "answer_plan_event_selection_threshold": round(float(answer_plan_event_selection_threshold), 6),
- "answer_plan_event_selection_top_k": int(answer_plan_event_selection_top_k),
- "focused_answer_type": dominant_answer_type,
- "preferred_path_types": list(preferred_path_types),
- },
- }
-
- def retrieve(self, query: str, *, top_k: int = 6) -> MemoryRetrieval:
- self._reload_graph()
- start = time.perf_counter()
- candidate_top_k = max(top_k, self.candidate_event_k if self.retrieval_mode == "hybrid_node_scored" else min(max(top_k, 18), 48))
- payload = self.graph.retrieve(query, top_k=candidate_top_k)
- hits = [_raw_hit_to_memory_hit(item) for item in payload.get("hits", []) or []]
- scored_lookup = {hit.memory_id: hit for hit in hits if hit.memory_id}
- active_hits = _restore_hit_scores([_raw_hit_to_memory_hit(item) for item in payload.get("active_hits", []) or []], scored_lookup)
- history_hits = _restore_hit_scores([_raw_hit_to_memory_hit(item) for item in payload.get("history_hits", []) or []], scored_lookup)
- stale_hits = _restore_hit_scores([_raw_hit_to_memory_hit(item) for item in payload.get("stale_hits", []) or []], scored_lookup)
- overwrite_hits = _restore_hit_scores([_raw_hit_to_memory_hit(item) for item in payload.get("overwrite_hits", []) or []], scored_lookup)
- false_hits = _restore_hit_scores([_raw_hit_to_memory_hit(item) for item in payload.get("false_hits", []) or []], scored_lookup)
- public_hits = _public_graph_hits(self.graph) if self.retrieval_mode == "hybrid_node_scored" else []
- hybrid_payload = self._hybrid_node_scored_hits(query, hits, top_k=top_k, public_hits=public_hits)
- hits = list(hybrid_payload.get("hits", []) or hits)
- hybrid_metadata = dict(hybrid_payload.get("metadata", {}) or {})
- memory_router_decision = _memory_router_decision(
- hybrid_metadata,
- mode=self.memory_router_mode,
- threshold=self.memory_router_threshold,
- margin=self.memory_router_margin,
- )
- current_subject_payload = _current_subject_protected_hits(
- query=query,
- graph=self.graph,
- final_hits=hits,
- top_k=top_k,
- )
- hits = list(current_subject_payload.get("hits", hits) or hits)
- current_subject_metadata = dict(current_subject_payload.get("metadata", {}) or {})
- audit_anchor_payload = _audit_anchor_protected_hits(
- query=query,
- final_hits=hits,
- candidate_hits=list(public_hits) + list(active_hits) + list(history_hits),
- metadata={**dict(payload.get("metadata", {}) or {}), **hybrid_metadata, **current_subject_metadata},
- top_k=top_k,
- )
- hits = list(audit_anchor_payload.get("hits", hits) or hits)
- audit_anchor_metadata = dict(audit_anchor_payload.get("metadata", {}) or {})
- identifier_payload = _identifier_protected_hits(
- query=query,
- final_hits=hits,
- candidate_hits=list(public_hits) + list(active_hits) + list(history_hits),
- top_k=top_k,
- )
- hits = list(identifier_payload.get("hits", hits) or hits)
- identifier_metadata = dict(identifier_payload.get("metadata", {}) or {})
- if _memory_router_allows(memory_router_decision, "path_tunnel", "topic_tunnel"):
- depth_chain_payload = _depth_chain_protected_hits(
- query=query,
- graph=self.graph,
- final_hits=hits,
- top_k=top_k,
- )
- hits = list(depth_chain_payload.get("hits", hits) or hits)
- depth_chain_metadata = dict(depth_chain_payload.get("metadata", {}) or {})
- else:
- depth_chain_metadata = {
- "depth_chain_protected_enabled": False,
- "depth_chain_router_suppressed": True,
- }
- if _memory_router_allows(memory_router_decision, "profile", "resource"):
- profile_focused_payload = _profile_focused_pack_hits(
- self.graph,
- query,
- hits,
- top_k=top_k,
- )
- hits = list(profile_focused_payload.get("hits", hits) or hits)
- profile_focused_metadata = dict(profile_focused_payload.get("metadata", {}) or {})
- else:
- profile_focused_metadata = {
- "profile_focused_pack_enabled": False,
- "profile_focused_router_suppressed": True,
- }
- if _memory_router_allows(memory_router_decision, "topic_tunnel"):
- topic_bucket_payload = _topic_bucket_rerank_hits(self.graph, query, hits, top_k=top_k)
- topic_bucket_hits = topic_bucket_payload.get("hits", hits)
- hits = list(hits if topic_bucket_hits is None else topic_bucket_hits)
- topic_bucket_metadata = dict(topic_bucket_payload.get("metadata", {}) or {})
- else:
- topic_bucket_metadata = {
- "topic_bucket_rerank_enabled": False,
- "topic_bucket_router_suppressed": True,
- }
- temporal_runtime_payload = self._temporal_runtime_pack(query)
- temporal_evidence_payload = self._apply_temporal_evidence_pack_to_hits(
- hits,
- temporal_runtime_payload,
- top_k=top_k,
- )
- hits = list(temporal_evidence_payload.get("hits", hits) or hits)
- temporal_runtime_metadata = dict(temporal_evidence_payload.get("metadata", {}) or {})
- injection_planner_payload = self._apply_injection_planner_to_hits(query, hits, top_k=top_k)
- hits = list(injection_planner_payload.get("hits", hits) or hits)
- injection_planner_metadata = dict(injection_planner_payload.get("metadata", {}) or {})
- facet_query_pack_payload = _facet_query_pack_hits(
- self.graph,
- query,
- hits,
- top_k=top_k,
- )
- hits = list(facet_query_pack_payload.get("hits", hits) or hits)
- facet_query_pack_metadata = dict(facet_query_pack_payload.get("metadata", {}) or {})
- unit_coverage_mode = _normalize(os.getenv("TMCRA_UNIT_COVERAGE_PACK_MODE", "on"))
- if unit_coverage_mode in _MULTI_UNIT_CHAIN_DISABLED_MODES:
- unit_coverage_metadata = {
- "unit_coverage_pack_enabled": False,
- "unit_coverage_reason": "disabled",
- }
- else:
- unit_coverage_payload = _unit_coverage_pack_hits(
- self.graph,
- query,
- hits,
- top_k=top_k,
- )
- hits = list(unit_coverage_payload.get("hits", hits) or hits)
- unit_coverage_metadata = dict(unit_coverage_payload.get("metadata", {}) or {})
- if _normalize(os.getenv("TMCRA_MULTI_UNIT_CHAIN_SLOT_MODE", "on")) not in _MULTI_UNIT_CHAIN_DISABLED_MODES:
- multi_unit_chain_slot_payload = _multi_unit_chain_slot_hits(
- self.graph,
- query,
- hits,
- top_k=top_k,
- )
- hits = list(multi_unit_chain_slot_payload.get("hits", hits) or hits)
- multi_unit_chain_slot_metadata = dict(multi_unit_chain_slot_payload.get("metadata", {}) or {})
- else:
- multi_unit_chain_slot_metadata = {
- "multi_unit_chain_slot_enabled": False,
- "multi_unit_chain_slot_reason": "disabled",
- }
- profile_protected_reinserted_count = 0
- profile_protected_ids = _dedupe(
- [
- *list(profile_focused_metadata.get("profile_focused_pack_memory_ids", []) or []),
- *list(profile_focused_metadata.get("profile_first_memory_ids", []) or []),
- ],
- max_items=max(1, min(6, int(top_k or 1))),
- )
- if profile_protected_ids:
- existing_hits_by_id = {hit.memory_id: hit for hit in hits if hit.memory_id}
- protected_hits: List[MemoryHit] = []
- for memory_id in profile_protected_ids:
- hit = existing_hits_by_id.get(memory_id)
- if hit is None:
- record = getattr(self.graph, "records_by_id", {}).get(memory_id)
- if record is None:
- continue
- hit = _memory_hit_from_record(record)
- metadata = dict(hit.metadata or {})
- metadata.update(
- {
- "profile_first_hybrid_rescue": True,
- "profile_protected_slot": True,
- "evidence_snippet_role": "profile_protected_slot",
- }
- )
- protected_hits.append(
- MemoryHit(
- memory_id=hit.memory_id,
- category=hit.category,
- value=hit.value,
- relation=hit.relation,
- anchors=list(hit.anchors),
- score=max(float(hit.score), 4.8),
- source_kind=hit.source_kind,
- slot_key=hit.slot_key,
- state=hit.state,
- turn_index=int(hit.turn_index),
- metadata=metadata,
- )
- )
- if protected_hits:
- profile_protected_reinserted_count = len(protected_hits)
- seen_profile_ids = {hit.memory_id for hit in protected_hits if hit.memory_id}
- hits = [*protected_hits, *[hit for hit in hits if hit.memory_id not in seen_profile_ids]]
- embedder_fusion_output_event_ids = [
- _clean_text(event_id)
- for event_id in list(hybrid_metadata.get("embedder_fusion_selected_event_ids", []) or [])
- if _clean_text(event_id)
- ]
- embedder_fusion_output_reordered = False
- if embedder_fusion_output_event_ids:
- before_order = _event_ids_from_hits(hits)
- seen_output_memory_ids = {hit.memory_id for hit in hits}
- for event_id in embedder_fusion_output_event_ids:
- event_hit = _representative_event_hit(_event_record_hits_from_graph(self.graph, event_id), query=query)
- if event_hit is None or event_hit.memory_id in seen_output_memory_ids:
- continue
- metadata = dict(event_hit.metadata or {})
- metadata.update(
- {
- "event_id": event_id,
- "evidence_snippet_role": "embedder_fusion_event_representative",
- "embedder_fusion_event_representative": True,
- }
- )
- hits.append(
- MemoryHit(
- memory_id=event_hit.memory_id,
- category=event_hit.category,
- value=event_hit.value,
- relation=event_hit.relation,
- anchors=list(event_hit.anchors),
- score=float(event_hit.score),
- source_kind=event_hit.source_kind,
- slot_key=event_hit.slot_key,
- state=event_hit.state,
- turn_index=int(event_hit.turn_index),
- metadata=metadata,
- )
- )
- seen_output_memory_ids.add(event_hit.memory_id)
- hits = _coverage_preserving_final_hits(
- hits,
- selected_event_ids=_dedupe([*embedder_fusion_output_event_ids, *before_order]),
- top_k=top_k,
- )
- embedder_fusion_output_reordered = before_order != _event_ids_from_hits(hits)
- retrieval_context_tokens = int(payload.get("context_token_estimate", _estimate_tokens_from_hits(hits)))
- self._last_retrieval_context_tokens = retrieval_context_tokens
- result = MemoryRetrieval(
- concepts=list(payload.get("concepts", []) or []),
- relations=list(payload.get("relations", []) or []),
- hits=hits,
- active_hits=active_hits,
- history_hits=history_hits,
- stale_hits=stale_hits,
- overwrite_hits=overwrite_hits,
- false_hits=false_hits,
- retrieval_seconds=time.perf_counter() - start,
- context_token_estimate=retrieval_context_tokens,
- retrieval_context_token_estimate=retrieval_context_tokens,
- metadata={
- "query_id": payload.get("query_id", ""),
- **dict(payload.get("metadata", {}) or {}),
- **hybrid_metadata,
- **memory_router_decision,
- **current_subject_metadata,
- **audit_anchor_metadata,
- **identifier_metadata,
- **depth_chain_metadata,
- **profile_focused_metadata,
- **topic_bucket_metadata,
- **temporal_runtime_metadata,
- **injection_planner_metadata,
- **facet_query_pack_metadata,
- **unit_coverage_metadata,
- **multi_unit_chain_slot_metadata,
- "profile_protected_reinserted_count": profile_protected_reinserted_count,
- "embedder_fusion_output_event_ids": list(embedder_fusion_output_event_ids),
- "embedder_fusion_output_reordered": bool(embedder_fusion_output_reordered),
- },
- )
- self._persist_graph()
- return result
-
- def export_dialog_graph(self, *, mode: str = "light") -> Dict[str, Any]:
- self._reload_graph()
- return self.graph.export_graph(
- snapshot_points=(1000, 5000, 10000, 20000, 50000, 100000, 200000, 300000, 500000),
- mode=mode,
- )
-
- def export_dialog_graph_mermaid(self) -> str:
- self._reload_graph()
- return self.graph.export_mermaid()
-
- def register_answer_support(self, *, answer_id: str, memory_ids: List[str], query_id: str = "", answer_text: str = "") -> None:
- self._reload_graph()
- self.graph.register_answer_support(answer_id=answer_id, memory_ids=memory_ids, query_id=query_id, answer_text=answer_text)
- self._persist_graph()
-
- def telemetry_snapshot(self) -> Dict[str, Any]:
- self._reload_graph()
- return self.graph.summary()
-
- def stats(self) -> Dict[str, Any]:
- self._reload_graph()
- storage = self._storage_breakdown()
- return _state_stats(
- storage_bytes=storage["storage_bytes"],
- retrieval_context_tokens=self._last_retrieval_context_tokens,
- total_state_tokens=storage["total_state_token_estimate"],
- core_storage_bytes=storage["core_storage_bytes"],
- audit_storage_bytes=storage["audit_storage_bytes"],
- core_state_token_estimate=storage["core_state_token_estimate"],
- audit_state_token_estimate=storage["audit_state_token_estimate"],
- lightweight_stats=bool(self.lightweight_stats),
- **self.graph.summary(),
- )
-
- def storage_bytes(self) -> int:
- self._reload_graph()
- return self._storage_breakdown()["storage_bytes"]
-
- def build_prompt_context(self, query: str, *, top_k: int = 8) -> Dict[str, Any]:
- retrieval = self.retrieve(query, top_k=top_k)
- retrieval_payload = retrieval.to_dict()
- stats = self.stats()
- context_summary = _graph_prompt_state_summary(self.graph, retrieval)
- prompt_context_payload = {
- "query": query,
- "retrieval": retrieval_payload,
- "context_summary": context_summary,
- }
- prompt_context_chars = len(json.dumps(prompt_context_payload, ensure_ascii=False))
- prompt_context_tokens_est = _estimate_tokens(json.dumps(prompt_context_payload, ensure_ascii=False))
- return {
- "mode": "graph_session_memory_v2",
- "query": query,
- "retrieval": retrieval_payload,
- "stats": stats,
- "state": context_summary,
- "context_summary": context_summary,
- "prompt_context_chars": int(prompt_context_chars),
- "prompt_context_tokens_est": int(prompt_context_tokens_est),
- "context_truncated": bool(context_summary.get("context_truncated", False)),
- "truncation_reason": _clean_text(context_summary.get("truncation_reason", "")),
- }
-
-
-class SummaryWindowMemoryAdapter(MemoryAdapter):
- name = "summary_window_memory"
-
- def __init__(self, *, window_size: int = 24, auto_extract: bool = False) -> None:
- self.extractor = SessionMemoryExtractor()
- self.window_size = max(4, int(window_size))
- self.turn_index = 0
- self.active_slots: Dict[str, SessionMemoryRecordV2] = {}
- self.recent_turns: deque[Dict[str, Any]] = deque(maxlen=self.window_size)
- self.auto_extract = bool(auto_extract)
- self._last_retrieval_context_tokens = 0
-
- def reset(self) -> None:
- self.turn_index = 0
- self.active_slots = {}
- self.recent_turns = deque(maxlen=self.window_size)
- self._last_retrieval_context_tokens = 0
-
- def ingest_turn(
- self,
- user_text: str,
- assistant_text: str = "",
- *,
- answer_payload: Dict[str, Any] | None = None,
- extraction_result: Dict[str, Any] | None = None,
- ) -> None:
- self.turn_index += 1
- records = _build_turn_records(
- self.extractor,
- user_text=user_text,
- answer_payload=answer_payload,
- extraction_result=extraction_result,
- turn_index=self.turn_index,
- allow_auto_extract=self.auto_extract,
- )
- for record in records:
- previous = self.active_slots.get(record.slot_key)
- if previous:
- previous.state = "superseded"
- record.supersedes.append(previous.memory_id)
- record.state = "active"
- self.active_slots[record.slot_key] = record
- self.recent_turns.append(
- {
- "turn_index": self.turn_index,
- "text": _clean_text(user_text),
- "assistant": _clean_text(assistant_text),
- }
- )
-
- def retrieve(self, query: str, *, top_k: int = 6) -> MemoryRetrieval:
- start = time.perf_counter()
- query_tokens = set(_tokenize(query))
- hints = set(infer_category_hints(query))
- scored: List[tuple[float, SessionMemoryRecordV2]] = []
- for record in self.active_slots.values():
- token_set = record.token_set()
- overlap = len(query_tokens & token_set) if query_tokens and token_set else 0
- score = overlap / max(1, len(query_tokens | token_set)) if query_tokens and token_set else 0.0
- if hints and record.category in hints:
- score += 0.22
- if record.slot_key.lower() in _normalize(query):
- score += 0.12
- score += min(0.08, record.turn_index * 0.0004)
- score += 0.2
- if score > 0:
- scored.append((score, record))
- scored.sort(key=lambda item: (item[0], item[1].turn_index), reverse=True)
- selected_records = [record for _, record in scored[:top_k]]
- hits = [
- MemoryHit(
- memory_id=record.memory_id,
- category=record.category,
- value=record.value,
- relation=record.relation,
- anchors=list(record.anchor_concepts),
- score=float(score),
- source_kind=record.source_kind,
- slot_key=record.slot_key,
- state=record.state,
- turn_index=record.turn_index,
- metadata={"window_size": self.window_size},
- )
- for score, record in scored[:top_k]
- ]
- concepts = []
- relations = []
- for hit in hits:
- concepts.append({"concept": hit.value, "type": hit.category, "source_kind": hit.source_kind})
- for anchor in hit.anchors[:2]:
- concepts.append({"concept": anchor, "type": "context", "source_kind": hit.source_kind})
- relation = _relation_hit(hit, weight_bias=0.06)
- if relation:
- relations.append(relation)
- retrieval_context_tokens = _estimate_tokens_from_hits(hits)
- self._last_retrieval_context_tokens = retrieval_context_tokens
- return MemoryRetrieval(
- concepts=concepts,
- relations=relations,
- hits=hits,
- active_hits=list(hits),
- retrieval_seconds=time.perf_counter() - start,
- context_token_estimate=retrieval_context_tokens,
- retrieval_context_token_estimate=retrieval_context_tokens,
- metadata={
- "records": len(self.active_slots),
- "window_size": self.window_size,
- "recent_turns": len(self.recent_turns),
- },
- )
-
- def stats(self) -> Dict[str, Any]:
- payload = {
- "active_slots": {slot: record.to_dict() for slot, record in self.active_slots.items()},
- "recent_turns": list(self.recent_turns),
- }
- total_state_tokens = _estimate_tokens(json.dumps(payload, ensure_ascii=False))
- return _state_stats(
- storage_bytes=self.storage_bytes(),
- retrieval_context_tokens=self._last_retrieval_context_tokens,
- total_state_tokens=total_state_tokens,
- records=len(self.active_slots),
- active_slots=len(self.active_slots),
- recent_turns=len(self.recent_turns),
- )
-
- def storage_bytes(self) -> int:
- payload = {
- "active_slots": {slot: record.to_dict() for slot, record in self.active_slots.items()},
- "recent_turns": list(self.recent_turns),
- }
- return len(json.dumps(payload, ensure_ascii=False).encode("utf-8"))
-
- def build_prompt_context(self, query: str, *, top_k: int = 8) -> Dict[str, Any]:
- return {
- "mode": "summary_window_memory",
- "query": query,
- "retrieval": self.retrieve(query, top_k=top_k).to_dict(),
- "stats": self.stats(),
- "state": {
- "active_slots": {slot: record.to_dict() for slot, record in self.active_slots.items()},
- "recent_turns": list(self.recent_turns),
- },
- }
-
-
-@dataclass(slots=True)
-class _VectorRecord:
- memory_id: str
- category: str
- value: str
- relation: str
- anchors: List[str]
- tokens: List[str]
- turn_index: int
- slot_key: str = ""
- active: bool = True
- source_kind: str = "vector_memory"
- metadata: Dict[str, Any] = field(default_factory=dict)
-
-
-class VectorRAGMemoryAdapter(MemoryAdapter):
- name = "vector_rag_memory"
-
- def __init__(self, *, auto_extract: bool = False) -> None:
- self.extractor = SessionMemoryExtractor()
- self.records: List[_VectorRecord] = []
- self.turn_index = 0
- self.auto_extract = bool(auto_extract)
- self._last_retrieval_context_tokens = 0
-
- def reset(self) -> None:
- self.records = []
- self.turn_index = 0
- self._last_retrieval_context_tokens = 0
-
- def ingest_turn(
- self,
- user_text: str,
- assistant_text: str = "",
- *,
- answer_payload: Dict[str, Any] | None = None,
- extraction_result: Dict[str, Any] | None = None,
- ) -> None:
- _ = assistant_text
- self.turn_index += 1
- records = _build_turn_records(
- self.extractor,
- user_text=user_text,
- answer_payload=answer_payload,
- extraction_result=extraction_result,
- turn_index=self.turn_index,
- allow_auto_extract=self.auto_extract,
- )
- for record in records:
- if record.slot_key:
- for previous in self.records:
- if previous.slot_key == record.slot_key and previous.active:
- previous.active = False
- self.records.append(
- _VectorRecord(
- memory_id=record.memory_id,
- category=record.category,
- value=record.value,
- relation=record.relation,
- anchors=list(record.anchor_concepts),
- tokens=list(record.token_set()),
- turn_index=record.turn_index,
- slot_key=record.slot_key,
- active=record.state == "active",
- source_kind=record.source_kind,
- metadata=dict(record.metadata),
- )
- )
-
- def retrieve(self, query: str, *, top_k: int = 6) -> MemoryRetrieval:
- start = time.perf_counter()
- query_tokens = set(_tokenize(query))
- hints = set(infer_category_hints(query))
- scored: List[tuple[float, _VectorRecord]] = []
- for record in self.records:
- token_set = set(record.tokens)
- overlap = len(query_tokens & token_set) if query_tokens and token_set else 0
- score = overlap / max(1, len(query_tokens | token_set)) if query_tokens and token_set else 0.0
- if hints and record.category in hints:
- score += 0.18
- if record.slot_key and record.slot_key.lower() in _normalize(query):
- score += 0.1
- score += min(0.12, record.turn_index * 0.0004)
- score += 0.18 if record.active else -0.25
- if score > 0:
- scored.append((score, record))
- if not scored:
- for record in self.records[-top_k:]:
- scored.append((0.05 + (0.15 if record.active else 0.0), record))
- scored.sort(key=lambda item: (item[0], item[1].active, item[1].turn_index), reverse=True)
- hits = [
- MemoryHit(
- memory_id=record.memory_id,
- category=record.category,
- value=record.value,
- relation=record.relation,
- anchors=list(record.anchors),
- score=float(score),
- source_kind=record.source_kind,
- slot_key=record.slot_key,
- state="active" if record.active else "superseded",
- turn_index=record.turn_index,
- metadata=dict(record.metadata),
- )
- for score, record in scored[:top_k]
- ]
- concepts = []
- relations = []
- for hit in hits:
- concepts.append({"concept": hit.value, "type": hit.category, "source_kind": hit.source_kind})
- for anchor in hit.anchors[:2]:
- concepts.append({"concept": anchor, "type": "context", "source_kind": hit.source_kind})
- relation = _relation_hit(hit)
- if relation:
- relations.append(relation)
- retrieval_context_tokens = _estimate_tokens_from_hits(hits)
- self._last_retrieval_context_tokens = retrieval_context_tokens
- return MemoryRetrieval(
- concepts=concepts,
- relations=relations,
- hits=hits,
- active_hits=[hit for hit in hits if hit.state == "active"],
- history_hits=[hit for hit in hits if hit.state != "active"],
- retrieval_seconds=time.perf_counter() - start,
- context_token_estimate=retrieval_context_tokens,
- retrieval_context_token_estimate=retrieval_context_tokens,
- metadata={
- "records": len(self.records),
- "active_records": sum(1 for item in self.records if item.active),
- },
- )
-
- def stats(self) -> Dict[str, Any]:
- payload = [
- {
- "memory_id": record.memory_id,
- "category": record.category,
- "value": record.value,
- "relation": record.relation,
- "anchors": record.anchors,
- "slot_key": record.slot_key,
- "active": record.active,
- }
- for record in self.records
- ]
- total_state_tokens = _estimate_tokens(json.dumps(payload, ensure_ascii=False))
- return _state_stats(
- storage_bytes=self.storage_bytes(),
- retrieval_context_tokens=self._last_retrieval_context_tokens,
- total_state_tokens=total_state_tokens,
- records=len(self.records),
- active_records=sum(1 for item in self.records if item.active),
- )
-
- def storage_bytes(self) -> int:
- payload = [
- {
- "memory_id": record.memory_id,
- "category": record.category,
- "value": record.value,
- "relation": record.relation,
- "anchors": record.anchors,
- "slot_key": record.slot_key,
- "active": record.active,
- }
- for record in self.records
- ]
- return len(json.dumps(payload, ensure_ascii=False).encode("utf-8"))
-
- def build_prompt_context(self, query: str, *, top_k: int = 8) -> Dict[str, Any]:
- state = [
- {
- "memory_id": record.memory_id,
- "category": record.category,
- "value": record.value,
- "relation": record.relation,
- "anchors": list(record.anchors),
- "slot_key": record.slot_key,
- "active": bool(record.active),
- "turn_index": int(record.turn_index),
- }
- for record in self.records
- ]
- return {
- "mode": "vector_rag_memory",
- "query": query,
- "retrieval": self.retrieve(query, top_k=top_k).to_dict(),
- "stats": self.stats(),
- "state": state,
- }
-
-
-class FullHistoryMemoryAdapter(MemoryAdapter):
- name = "full_history_memory"
-
- def __init__(self) -> None:
- self.turns: List[Dict[str, str]] = []
- self._last_retrieval_context_tokens = 0
-
- def reset(self) -> None:
- self.turns = []
- self._last_retrieval_context_tokens = 0
-
- def ingest_turn(
- self,
- user_text: str,
- assistant_text: str = "",
- *,
- answer_payload: Dict[str, Any] | None = None,
- extraction_result: Dict[str, Any] | None = None,
- ) -> None:
- _ = answer_payload, extraction_result
- self.turns.append({"user": _clean_text(user_text), "assistant": _clean_text(assistant_text)})
-
- def retrieve(self, query: str, *, top_k: int = 6) -> MemoryRetrieval:
- start = time.perf_counter()
- query_tokens = set(_tokenize(query))
- scored: List[tuple[float, Dict[str, str], int]] = []
- for index, turn in enumerate(self.turns):
- combined = f"{turn.get('user', '')} {turn.get('assistant', '')}"
- token_set = set(_tokenize(combined))
- if not token_set:
- continue
- overlap = len(query_tokens & token_set) if query_tokens else 0
- score = overlap / max(1, len(query_tokens | token_set)) if query_tokens else 0.0
- scored.append((score, turn, index))
- scored.sort(key=lambda item: (item[0], item[2]), reverse=True)
- hits = [
- MemoryHit(
- memory_id=f"turn:{index}",
- category="history_turn",
- value=turn.get("user", ""),
- relation="conversation_context",
- anchors=[turn.get("assistant", "")] if turn.get("assistant") else [],
- score=float(score),
- source_kind="full_history",
- slot_key=f"turn.{index}",
- state="active",
- turn_index=index + 1,
- )
- for score, turn, index in scored[:top_k]
- if turn.get("user", "")
- ]
- retrieval_context_tokens = _estimate_tokens(json.dumps(self.turns, ensure_ascii=False))
- self._last_retrieval_context_tokens = retrieval_context_tokens
- return MemoryRetrieval(
- hits=hits,
- active_hits=list(hits),
- retrieval_seconds=time.perf_counter() - start,
- context_token_estimate=retrieval_context_tokens,
- retrieval_context_token_estimate=retrieval_context_tokens,
- metadata={"records": len(self.turns)},
- )
-
- def stats(self) -> Dict[str, Any]:
- total_state_tokens = _estimate_tokens(json.dumps(self.turns, ensure_ascii=False))
- return _state_stats(
- storage_bytes=self.storage_bytes(),
- retrieval_context_tokens=self._last_retrieval_context_tokens,
- total_state_tokens=total_state_tokens,
- records=len(self.turns),
- )
-
- def storage_bytes(self) -> int:
- return len(json.dumps(self.turns, ensure_ascii=False).encode("utf-8"))
-
- def build_prompt_context(self, query: str, *, top_k: int = 8) -> Dict[str, Any]:
- return {
- "mode": "full_history_memory",
- "query": query,
- "retrieval": self.retrieve(query, top_k=top_k).to_dict(),
- "stats": self.stats(),
- "state": {
- "turns": list(self.turns),
- },
- }
+from __future__ import annotations
+
+from collections import deque
+from dataclasses import dataclass, field
+import json
+import math
+import os
+from pathlib import Path
+import re
+import tempfile
+import time
+import uuid
+from typing import Any, Dict, Iterable, List, Mapping, Sequence
+
+from core.session_memory import SessionMemoryExtractor
+from experiments.replacement.memory_graph import (
+ SessionMemoryGraphV2,
+ SessionMemoryEdgeV2,
+ SessionMemoryRecordV2,
+ SQLiteSessionMemoryStore,
+ _clean_text,
+ _estimate_tokens,
+ _normalize,
+ _public_query_subject,
+ _public_slot_root,
+ _public_subject_signature,
+ stable_slot_key,
+ _tokenize,
+ guess_slot_key,
+ infer_category_hints,
+)
+from experiments.replacement.node_memory import (
+ _call_with_supported_kwargs,
+ DEFAULT_MATRIX_EVENT_TOP_K,
+ LoadedNodeMemoryScorer,
+ MEMORY_ROUTER_LAYERS,
+ build_default_path_templates,
+ extract_question_features,
+)
+from experiments.replacement.public_event_signature import compute_public_event_signature
+from experiments.replacement.memory_profiles import TMCRAProfile
+from experiments.replacement.typed_tunnel_augmentation import (
+ annotate_memory_record,
+ merge_typed_metadata,
+ typed_edge_tags_between,
+ typed_tunnel_signature_text,
+)
+from experiments.replacement.profile_layer import infer_profile_query_intent, is_profile_layer_record, profile_query_score_delta
+from experiments.replacement import injection_planner as injection_planner_runtime
+from experiments.replacement.temporal_modeling_types import TemporalFrame, TemporalQueryPlan
+from experiments.replacement.temporal_organizer import TemporalOrganizer
+from experiments.replacement.temporal_query_planner import TemporalQueryPlanner
+from experiments.replacement.temporal_router_runtime import LoadedTemporalRouter
+from experiments.replacement.timeline_evidence_pack import TimelineEvidencePackBuilder
+from experiments.replacement.timeline_state_layer import TimelineStateLayer
+
+from .base import MemoryAdapter, MemoryHit, MemoryRetrieval
+
+
+def _dedupe(items: Iterable[Any], *, max_items: int | None = None) -> List[str]:
+ values: List[str] = []
+ seen = set()
+ for item in items:
+ text = _clean_text(item)
+ if not text:
+ continue
+ key = _normalize(text)
+ if key in seen:
+ continue
+ seen.add(key)
+ values.append(text)
+ if max_items is not None and len(values) >= max_items:
+ break
+ return values
+
+
+def _apply_typed_tunnel_annotations(records: List[SessionMemoryRecordV2], *, source_text: str = "") -> List[SessionMemoryRecordV2]:
+ for record in records:
+ annotate_memory_record(record, source_text=source_text)
+ return records
+
+
+def _estimate_tokens_from_hits(hits: Sequence[MemoryHit]) -> int:
+ total = 0
+ for hit in hits:
+ total += _estimate_tokens(hit.value)
+ total += sum(_estimate_tokens(anchor) for anchor in hit.anchors)
+ return total
+
+
+def _float_env(name: str, default: float) -> float:
+ raw = _clean_text(os.getenv(name, ""))
+ if not raw:
+ return float(default)
+ try:
+ return float(raw)
+ except ValueError:
+ return float(default)
+
+
+_GRAPH_PROMPT_MAX_CHARS = 12_000
+_GRAPH_PROMPT_MAX_HITS = 8
+_GRAPH_PROMPT_MAX_ACTIVE_SLOTS = 12
+_GRAPH_PROMPT_MAX_RELATIONS = 10
+_HYBRID_SELECTED_EVENT_FLOOR = 8
+_HYBRID_SELECTED_PATH_CAP = 6
+_HYBRID_TEMPORAL_PATH_CAP = 2
+_HYBRID_PROFILE_PATH_CAP = 3
+_MEMORY_ROUTER_GUIDED_MODES = {"guided", "route", "routing", "enforce"}
+_MEMORY_ROUTER_FORCE_MODES = {"force", "forced"}
+_MEMORY_ROUTER_DISABLED_MODES = {"", "off", "disabled", "none", "false", "0"}
+_MEMORY_ROUTER_OBSERVE_MODES = {"observe", "observer", "telemetry", "shadow"}
+_MEMORY_ROUTER_DEFAULT_THRESHOLD = 0.55
+_MEMORY_ROUTER_DEFAULT_MARGIN = 0.08
+_INJECTION_PLANNER_GUIDED_MODES = {"guided", "route", "routing", "enforce"}
+_INJECTION_PLANNER_FORCE_MODES = {"force", "forced"}
+_INJECTION_PLANNER_DISABLED_MODES = {"", "off", "disabled", "none", "false", "0"}
+_INJECTION_PLANNER_OBSERVE_MODES = {"observe", "observer", "telemetry", "shadow"}
+_TEMPORAL_LAYER_DISABLED_MODES = {"off", "disabled", "none", "false", "0"}
+_TEMPORAL_ROUTER_DEFAULT_WRITER_MIN_CONFIDENCE = 0.72
+_TEMPORAL_ROUTER_DEFAULT_QUERY_MIN_CONFIDENCE = 0.85
+_TEMPORAL_ROUTER_DEFAULT_QUERY_INTENT_MIN_CONFIDENCE = 0.60
+_EMBEDDER_INDEX_DISABLED_MODES = {"", "off", "disabled", "none", "false", "0"}
+_EMBEDDER_INDEX_BGE_M3_MODES = {"bge", "bge_m3", "bge-m3", "baai_bge_m3", "baai/bge-m3"}
+_EMBEDDER_INDEX_VERSION = "write_hash_sparse_v1"
+_EMBEDDER_INDEX_BGE_M3_VERSION = "write_bge_m3_dense_sparse_v1"
+_EMBEDDER_MODEL_CACHE: Dict[str, Any] = {}
+_HYBRID_SYMBOLIC_STOPWORDS = {
+ "a",
+ "an",
+ "and",
+ "are",
+ "as",
+ "at",
+ "be",
+ "did",
+ "do",
+ "does",
+ "event",
+ "for",
+ "from",
+ "had",
+ "has",
+ "have",
+ "how",
+ "in",
+ "is",
+ "of",
+ "on",
+ "or",
+ "should",
+ "the",
+ "to",
+ "turn",
+ "was",
+ "were",
+ "what",
+ "when",
+ "where",
+ "which",
+ "who",
+ "why",
+ "will",
+ "with",
+}
+
+
+def _coerce_memory_router_scores(payload: Mapping[str, Any]) -> Dict[str, float]:
+ raw_scores = dict(payload.get("memory_router_scores", {}) or {})
+ scores: Dict[str, float] = {}
+ for layer in MEMORY_ROUTER_LAYERS:
+ try:
+ scores[layer] = float(raw_scores.get(layer, 0.0))
+ except (TypeError, ValueError):
+ scores[layer] = 0.0
+ return scores
+
+
+def _memory_router_decision(
+ payload: Mapping[str, Any],
+ *,
+ mode: str,
+ threshold: float,
+ margin: float,
+) -> Dict[str, Any]:
+ normalized_mode = _normalize(mode)
+ scores = _coerce_memory_router_scores(payload)
+ has_scores = bool(scores) and any(layer in dict(payload.get("memory_router_scores", {}) or {}) for layer in MEMORY_ROUTER_LAYERS)
+ if not has_scores:
+ return {
+ "memory_router_enabled": False,
+ "memory_router_guided": False,
+ "memory_router_reason": "no_router_scores",
+ "memory_router_scores": {},
+ "memory_router_top_layers": [],
+ "memory_router_active_layers": [],
+ "memory_router_score_spread": 0.0,
+ "memory_router_confidence": 0.0,
+ }
+ ranked_layers = [
+ layer
+ for layer, _ in sorted(
+ scores.items(),
+ key=lambda item: (-float(item[1]), item[0]),
+ )
+ ]
+ score_values = list(scores.values())
+ score_spread = max(score_values) - min(score_values)
+ confidence = max(abs(float(score) - 0.5) for score in score_values)
+ resolved_threshold = max(0.0, min(1.0, float(threshold or _MEMORY_ROUTER_DEFAULT_THRESHOLD)))
+ resolved_margin = max(0.0, min(0.5, float(margin or _MEMORY_ROUTER_DEFAULT_MARGIN)))
+ active_layers = [
+ layer
+ for layer in ranked_layers
+ if float(scores.get(layer, 0.0)) >= resolved_threshold
+ ]
+ confident = bool(active_layers) and (score_spread >= resolved_margin or confidence >= resolved_margin)
+ guided_requested = normalized_mode in _MEMORY_ROUTER_GUIDED_MODES
+ forced = normalized_mode in _MEMORY_ROUTER_FORCE_MODES
+ guided = bool(forced or (guided_requested and confident))
+ if forced and not active_layers and ranked_layers:
+ active_layers = [ranked_layers[0]]
+ if active_layers and "event" not in active_layers:
+ active_layers = ["event", *active_layers]
+ reason = "observe"
+ if normalized_mode in _MEMORY_ROUTER_DISABLED_MODES:
+ reason = "disabled"
+ elif normalized_mode in _MEMORY_ROUTER_OBSERVE_MODES:
+ reason = "observe"
+ elif forced:
+ reason = "forced"
+ elif guided_requested and not confident:
+ reason = "low_confidence"
+ elif guided:
+ reason = "guided"
+ return {
+ "memory_router_enabled": True,
+ "memory_router_guided": bool(guided and normalized_mode not in _MEMORY_ROUTER_DISABLED_MODES),
+ "memory_router_reason": reason,
+ "memory_router_scores": scores,
+ "memory_router_top_layers": ranked_layers,
+ "memory_router_active_layers": active_layers,
+ "memory_router_score_spread": round(float(score_spread), 6),
+ "memory_router_confidence": round(float(confidence), 6),
+ "memory_router_threshold": round(float(resolved_threshold), 6),
+ "memory_router_margin": round(float(resolved_margin), 6),
+ "memory_router_mode": normalized_mode or "observe",
+ }
+
+
+def _memory_router_allows(decision: Mapping[str, Any], *layers: str) -> bool:
+ if not bool(decision.get("memory_router_guided")):
+ return True
+ requested = {_normalize(layer) for layer in layers if _normalize(layer)}
+ if not requested:
+ return True
+ if "event" in requested:
+ return True
+ active = {_normalize(layer) for layer in list(decision.get("memory_router_active_layers", []) or [])}
+ return bool(active & requested)
+
+
+def _hybrid_symbolic_tokens(value: Any) -> List[str]:
+ return [
+ token
+ for token in _tokenize(value)
+ if token and token not in _HYBRID_SYMBOLIC_STOPWORDS and not re.fullmatch(r"\d+", token)
+ ]
+
+
+_PATH_UTILITY_STOPWORDS = set(_HYBRID_SYMBOLIC_STOPWORDS) | {
+ "actually",
+ "also",
+ "bit",
+ "feel",
+ "feels",
+ "get",
+ "guess",
+ "i",
+ "im",
+ "it",
+ "its",
+ "just",
+ "kind",
+ "like",
+ "maybe",
+ "me",
+ "more",
+ "much",
+ "my",
+ "really",
+ "something",
+ "still",
+ "think",
+ "thats",
+ "there",
+ "this",
+ "way",
+ "would",
+}
+
+
+def _path_utility_tokens(value: Any) -> List[str]:
+ return [
+ token
+ for token in _hybrid_symbolic_tokens(value)
+ if token and token not in _PATH_UTILITY_STOPWORDS
+ ]
+
+
+_PROFILE_QUERY_ALIAS_GROUPS: tuple[set[str], ...] = (
+ {
+ "accessory",
+ "accessories",
+ "bag",
+ "camera",
+ "cameras",
+ "equipment",
+ "flash",
+ "gear",
+ "lens",
+ "lenses",
+ "photo",
+ "photography",
+ "sony",
+ "tripod",
+ },
+ {
+ "app",
+ "apps",
+ "dashboard",
+ "interface",
+ "layout",
+ "panel",
+ "software",
+ "tool",
+ "tools",
+ "ui",
+ "workflow",
+ },
+ {
+ "diet",
+ "drink",
+ "food",
+ "meal",
+ "restaurant",
+ "snack",
+ "taste",
+ },
+ {
+ "background",
+ "career",
+ "job",
+ "occupation",
+ "position",
+ "previous",
+ "profession",
+ "role",
+ "worked",
+ "work",
+ },
+)
+
+
+_PROFILE_QUERY_GENERIC_TOKENS = set(_PATH_UTILITY_STOPWORDS) | {
+ "able",
+ "about",
+ "any",
+ "anything",
+ "based",
+ "best",
+ "can",
+ "complement",
+ "could",
+ "current",
+ "give",
+ "help",
+ "information",
+ "looking",
+ "make",
+ "need",
+ "please",
+ "recommend",
+ "recommendation",
+ "recommendations",
+ "should",
+ "some",
+ "suggest",
+ "suggestion",
+ "suggestions",
+ "tell",
+ "that",
+ "using",
+ "you",
+}
+
+
+def _profile_query_expanded_tokens(value: Any) -> set[str]:
+ tokens = set(_path_utility_tokens(value))
+ expanded = set(tokens)
+ for group in _PROFILE_QUERY_ALIAS_GROUPS:
+ if tokens & group:
+ expanded.update(group)
+ return expanded
+
+
+def _profile_specific_tokens(tokens: Iterable[Any]) -> set[str]:
+ return {
+ _normalize(token)
+ for token in tokens
+ if _clean_text(token)
+ and _normalize(token) not in _PROFILE_QUERY_GENERIC_TOKENS
+ and not re.fullmatch(r"\d+", _normalize(token))
+ }
+
+
+def _profile_hit_match_score(query_tokens: set[str], expanded_query_tokens: set[str], hit: MemoryHit) -> tuple[float, List[str], List[str]]:
+ metadata = dict(hit.metadata or {})
+ payload_parts: List[Any] = [
+ hit.category,
+ hit.relation,
+ hit.slot_key,
+ hit.value,
+ *list(hit.anchors or []),
+ metadata.get("profile_summary", ""),
+ metadata.get("profile_value", ""),
+ metadata.get("profile_type", ""),
+ metadata.get("profile_domain", ""),
+ metadata.get("profile_domain_label", ""),
+ metadata.get("semantic_slot", ""),
+ metadata.get("subject", ""),
+ metadata.get("extracted_subject", ""),
+ metadata.get("profile_cluster_domains", []),
+ metadata.get("profile_cluster_types", []),
+ metadata.get("profile_cluster_route_terms", []),
+ metadata.get("profile_route_terms", []),
+ metadata.get("profile_support_values", []),
+ ]
+ payload_text = " ".join(str(item) for item in payload_parts)
+ record_raw_tokens = set(_path_utility_tokens(payload_text))
+ record_tokens = _profile_query_expanded_tokens(payload_text)
+ specific_query_tokens = _profile_specific_tokens(query_tokens)
+ raw_overlap_tokens = sorted(specific_query_tokens & _profile_specific_tokens(record_raw_tokens))
+ expanded_overlap_tokens = sorted(_profile_specific_tokens(expanded_query_tokens) & _profile_specific_tokens(record_tokens))
+ overlap_tokens = raw_overlap_tokens or expanded_overlap_tokens
+ overlap_ratio = float(len(overlap_tokens)) / float(max(1, len(specific_query_tokens)))
+ profile_type = _normalize(metadata.get("profile_type", ""))
+ source_kind = _normalize(hit.source_kind)
+ source_bonus = 0.0
+ if source_kind in {"public_dialog_preference", "public_dialog_goal", "public_dialog_profile"}:
+ source_bonus += 0.08
+ if bool(metadata.get("profile_candidate_status") == "consolidated"):
+ source_bonus += 0.04
+ if bool(metadata.get("profile_cluster_node")):
+ source_bonus -= 0.16
+ type_bonus = 0.0
+ if profile_type in {"preference", "goal", "setup", "usage_context"} and (
+ {"recommend", "suggest", "suited", "accessory", "accessories", "gear", "equipment", "setup", "current"} & query_tokens
+ ):
+ type_bonus += 0.10
+ if profile_type in {"setup", "usage_context"} and {"current", "setup", "profile"} & query_tokens:
+ type_bonus += 0.08
+ raw_bonus = 0.18 * len(raw_overlap_tokens)
+ expanded_bonus = 0.08 * max(0, len(expanded_overlap_tokens) - len(raw_overlap_tokens))
+ score = raw_bonus + expanded_bonus + (0.58 * overlap_ratio) + source_bonus + type_bonus
+ return round(score, 6), overlap_tokens, raw_overlap_tokens
+
+
+def _bounded_event_id_union(*groups: Iterable[Any], max_items: int) -> List[str]:
+ return _dedupe((item for group in groups for item in group), max_items=max(1, int(max_items)))
+
+
+def _symbolic_recall_event_ids(
+ query: str,
+ runtime_graph: Mapping[str, Any],
+ *,
+ grouped_hits: Mapping[str, Sequence[MemoryHit]],
+ limit: int,
+) -> List[str]:
+ question_features = extract_question_features(query)
+ query_tokens = set(_hybrid_symbolic_tokens(question_features.get("question_anchor_tokens", []) or query))
+ if not query_tokens:
+ return []
+ nodes_by_id = {
+ _clean_text(node.get("id", "")): dict(node)
+ for node in list(runtime_graph.get("nodes", []) or [])
+ if _clean_text(node.get("id", ""))
+ }
+ event_payloads: Dict[str, List[str]] = {}
+ for node_id, node in nodes_by_id.items():
+ if _clean_text(node.get("type", "")) == "event":
+ metadata = dict(node.get("metadata", {}) or {})
+ teacher_fields = dict(node.get("teacher_fields", {}) or {})
+ event_payloads.setdefault(node_id, []).extend(
+ [
+ node.get("text", ""),
+ node.get("speaker", ""),
+ node.get("slot_key", ""),
+ node.get("target_status", ""),
+ node.get("profile_type", ""),
+ node.get("profile_value", ""),
+ *teacher_fields.values(),
+ *metadata.values(),
+ ]
+ )
+ for path in list(runtime_graph.get("paths", []) or []):
+ event_id = _clean_text(path.get("event_id", ""))
+ support_node = nodes_by_id.get(_clean_text(path.get("target", "")), {})
+ if event_id and support_node:
+ event_payloads.setdefault(event_id, []).append(support_node.get("text", ""))
+ for event_id, group_hits in grouped_hits.items():
+ payloads = event_payloads.setdefault(_clean_text(event_id), [])
+ for hit in list(group_hits or []):
+ metadata = dict(hit.metadata or {})
+ payloads.extend([hit.value, hit.slot_key, hit.category, hit.relation, *hit.anchors, *metadata.values()])
+
+ scored_events: List[tuple[str, float]] = []
+ for event_id, payloads in event_payloads.items():
+ event_tokens = set(_hybrid_symbolic_tokens(payloads))
+ overlap = query_tokens & event_tokens
+ if not overlap:
+ continue
+ overlap_ratio = float(len(overlap)) / float(max(1, len(query_tokens)))
+ event_node = nodes_by_id.get(event_id, {})
+ turn_index = int(event_node.get("turn_index", 0) or 0)
+ scored_events.append((event_id, (len(overlap) * 4.0) + overlap_ratio + min(turn_index, 1000) * 0.000001))
+ return [
+ event_id
+ for event_id, _ in sorted(scored_events, key=lambda item: (-float(item[1]), item[0]))
+ ][: max(1, int(limit))]
+
+
+_EMBEDDER_INDEX_METADATA_TEXT_KEYS = (
+ "raw_text",
+ "source_turn_text",
+ "source_span",
+ "event_phrase",
+ "event_summary",
+ "profile_value",
+ "profile_summary",
+ "profile_type",
+ "profile_domain",
+ "profile_domain_label",
+ "semantic_slot",
+ "target_status",
+ "subject",
+ "subject_signature",
+ "canonical_slot_key",
+ "resource_key",
+ "resolved_date",
+ "resolved_time_value",
+ "time_value",
+ "time_display_value",
+ "time_granularity",
+ "speaker",
+ "session_name",
+ "topic_label",
+ "topic_bucket_id",
+ "origin_query",
+ "writeback_class",
+ "depth_layer",
+ "memory_chain_depth_layer",
+)
+_EMBEDDER_INDEX_METADATA_LIST_KEYS = (
+ "topic_keywords",
+ "profile_route_terms",
+ "profile_cluster_route_terms",
+ "profile_support_values",
+ "evidence_anchors",
+ "support_memory_ids",
+ "support_fact_refs",
+ "support_path_refs",
+)
+
+
+def _embedder_index_enabled(mode: Any) -> bool:
+ return _normalize(mode) not in _EMBEDDER_INDEX_DISABLED_MODES
+
+
+def _embedder_index_uses_bge_m3(mode: Any) -> bool:
+ return _normalize(mode).replace("-", "_") in {item.replace("-", "_") for item in _EMBEDDER_INDEX_BGE_M3_MODES}
+
+
+def _embedder_index_version_for_mode(mode: Any) -> str:
+ return _EMBEDDER_INDEX_BGE_M3_VERSION if _embedder_index_uses_bge_m3(mode) else _EMBEDDER_INDEX_VERSION
+
+
+def _embedder_index_text_items(value: Any, *, max_items: int = 64) -> List[str]:
+ items: List[str] = []
+
+ def visit(item: Any) -> None:
+ if len(items) >= max_items:
+ return
+ if item is None:
+ return
+ if isinstance(item, Mapping):
+ for key, nested in list(item.items()):
+ if len(items) >= max_items:
+ break
+ key_text = _clean_text(key)
+ if isinstance(nested, (str, int, float, bool)):
+ value_text = _clean_text(nested)[:800]
+ if value_text:
+ items.append(f"{key_text} {value_text}".strip())
+ else:
+ visit(nested)
+ return
+ if isinstance(item, (list, tuple, set)):
+ for nested in list(item):
+ if len(items) >= max_items:
+ break
+ visit(nested)
+ return
+ text = _clean_text(item)[:800]
+ if text:
+ items.append(text)
+
+ visit(value)
+ return items[:max_items]
+
+
+def _embedder_index_term_weights(value: Any, *, max_terms: int = 96) -> Dict[str, float]:
+ text = _clean_text(value)
+ if not text:
+ return {}
+ counts: Dict[str, float] = {}
+ for token in _path_utility_tokens(text):
+ token = _normalize(token)
+ if not token or len(token) > 64:
+ continue
+ counts[token] = counts.get(token, 0.0) + 1.0
+ normalized_text = _normalize(text)
+ cjk_chars = [char for char in normalized_text if "\u4e00" <= char <= "\u9fff"]
+ for width, weight in ((2, 1.35), (3, 1.15)):
+ if len(cjk_chars) < width:
+ continue
+ for index in range(0, len(cjk_chars) - width + 1):
+ gram = "".join(cjk_chars[index : index + width])
+ if gram:
+ counts[gram] = counts.get(gram, 0.0) + weight
+ if not counts:
+ return {}
+ ranked = sorted(counts.items(), key=lambda item: (-float(item[1]), item[0]))[: max(1, int(max_terms or 1))]
+ norm = math.sqrt(sum(float(weight) * float(weight) for _, weight in ranked)) or 1.0
+ return {term: round(float(weight) / norm, 6) for term, weight in ranked}
+
+
+def _embedder_dense_vectors_for_texts(texts: Sequence[str], *, mode: str) -> tuple[List[List[float]], Dict[str, Any]]:
+ normalized_mode = _normalize(mode)
+ if not _embedder_index_uses_bge_m3(normalized_mode):
+ return [[] for _ in texts], {"write_embedder_dense_enabled": False}
+ clean_texts = [_clean_text(text) for text in texts]
+ metadata: Dict[str, Any] = {
+ "write_embedder_dense_enabled": False,
+ "write_embedder_dense_backend": "bge_m3_transformers",
+ "write_embedder_dense_model": _clean_text(os.getenv("TMCRA_EMBEDDER_MODEL_PATH", "")) or "BAAI/bge-m3",
+ }
+ if not any(clean_texts):
+ metadata["write_embedder_dense_error"] = "empty_texts"
+ return [[] for _ in texts], metadata
+ try:
+ import torch # type: ignore
+ from transformers import AutoModel, AutoTokenizer # type: ignore
+ except Exception as exc:
+ metadata["write_embedder_dense_error"] = f"dependency_unavailable:{exc.__class__.__name__}"
+ return [[] for _ in texts], metadata
+ model_name = metadata["write_embedder_dense_model"]
+ device = _clean_text(os.getenv("TMCRA_EMBEDDER_DEVICE", ""))
+ if not device:
+ device = "cuda" if bool(getattr(torch, "cuda", None) and torch.cuda.is_available()) else "cpu"
+ try:
+ max_length = max(64, int(os.getenv("TMCRA_EMBEDDER_MODEL_MAX_LENGTH", "512") or 512))
+ except (TypeError, ValueError):
+ max_length = 512
+ cache_key = f"bge_m3::{model_name}::{device}::{max_length}"
+ try:
+ cached = _EMBEDDER_MODEL_CACHE.get(cache_key)
+ if cached is None:
+ tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=False)
+ model = AutoModel.from_pretrained(model_name, trust_remote_code=False)
+ model.to(device)
+ model.eval()
+ cached = (tokenizer, model)
+ _EMBEDDER_MODEL_CACHE[cache_key] = cached
+ tokenizer, model = cached
+ encoded = tokenizer(
+ clean_texts,
+ padding=True,
+ truncation=True,
+ max_length=max_length,
+ return_tensors="pt",
+ )
+ encoded = {key: value.to(device) for key, value in encoded.items()}
+ with torch.no_grad():
+ output = model(**encoded)
+ hidden = output.last_hidden_state
+ mask = encoded.get("attention_mask")
+ if mask is not None:
+ mask = mask.unsqueeze(-1).expand(hidden.size()).float()
+ pooled = torch.sum(hidden * mask, dim=1) / torch.clamp(mask.sum(dim=1), min=1e-9)
+ else:
+ pooled = hidden[:, 0]
+ pooled = torch.nn.functional.normalize(pooled, p=2, dim=1)
+ vectors = [
+ [round(float(value), 6) for value in row.detach().cpu().tolist()]
+ for row in pooled
+ ]
+ metadata.update(
+ {
+ "write_embedder_dense_enabled": True,
+ "write_embedder_dense_device": device,
+ "write_embedder_dense_dim": int(len(vectors[0]) if vectors else 0),
+ "write_embedder_dense_max_length": int(max_length),
+ }
+ )
+ return vectors, metadata
+ except Exception as exc:
+ metadata["write_embedder_dense_error"] = f"{exc.__class__.__name__}:{_clean_text(exc)[:240]}"
+ return [[] for _ in texts], metadata
+
+
+def _prewarm_embedder_dense_if_requested(*, mode: str) -> Dict[str, Any]:
+ flag = _normalize(os.getenv("TMCRA_EMBEDDER_PREWARM", ""))
+ if flag in _EMBEDDER_INDEX_DISABLED_MODES or flag not in {"1", "true", "yes", "on", "auto"}:
+ return {"embedder_prewarm_enabled": False}
+ normalized_mode = _normalize(mode)
+ if not _embedder_index_uses_bge_m3(normalized_mode):
+ return {
+ "embedder_prewarm_enabled": False,
+ "embedder_prewarm_reason": "mode_not_dense",
+ "embedder_prewarm_mode": normalized_mode or "off",
+ }
+ warmup_text = _clean_text(os.getenv("TMCRA_EMBEDDER_PREWARM_TEXT", "")) or "tmcra memory retrieval warmup"
+ vectors, metadata = _embedder_dense_vectors_for_texts([warmup_text], mode=normalized_mode)
+ return {
+ "embedder_prewarm_enabled": bool(vectors and vectors[0]),
+ "embedder_prewarm_mode": normalized_mode,
+ "embedder_prewarm_dense_enabled": bool(metadata.get("write_embedder_dense_enabled")),
+ "embedder_prewarm_dense_device": metadata.get("write_embedder_dense_device", ""),
+ "embedder_prewarm_dense_error": metadata.get("write_embedder_dense_error", ""),
+ }
+
+
+def _embedder_index_record_text(record: SessionMemoryRecordV2, *, turn_text: str = "") -> str:
+ metadata = dict(record.metadata or {})
+ parts: List[str] = [
+ record.category,
+ record.slot_key,
+ record.value,
+ record.value,
+ record.value,
+ record.relation,
+ *list(record.anchor_concepts or []),
+ *list(record.anchor_concepts or []),
+ *list(record.evidence_anchors or []),
+ ]
+ has_record_evidence_text = any(
+ _clean_text(metadata.get(key, ""))
+ for key in ("raw_text", "source_turn_text", "source_span", "event_phrase", "profile_value", "profile_summary")
+ )
+ if turn_text and not has_record_evidence_text:
+ parts.append(_clean_text(turn_text)[:1000])
+ for key in _EMBEDDER_INDEX_METADATA_TEXT_KEYS:
+ value = metadata.get(key)
+ if value:
+ parts.append(f"{key} {' '.join(_embedder_index_text_items(value, max_items=12))}".strip())
+ for key in _EMBEDDER_INDEX_METADATA_LIST_KEYS:
+ value = metadata.get(key)
+ if value:
+ parts.append(f"{key} {' '.join(_embedder_index_text_items(value, max_items=24))}".strip())
+ return _clean_text(" ".join(_clean_text(part) for part in parts if _clean_text(part)))
+
+
+def _apply_write_embedder_index_to_graph(
+ graph: SessionMemoryGraphV2,
+ *,
+ stored_ids: Sequence[str],
+ turn_text: str,
+ turn_index: int,
+ mode: str,
+ max_terms: int,
+) -> Dict[str, Any]:
+ normalized_mode = _normalize(mode)
+ index_version = _embedder_index_version_for_mode(normalized_mode)
+ metadata: Dict[str, Any] = {
+ "write_embedder_index_enabled": False,
+ "write_embedder_index_mode": normalized_mode or "off",
+ "write_embedder_index_version": index_version,
+ "write_embedder_index_record_count": 0,
+ }
+ if not _embedder_index_enabled(normalized_mode) or not stored_ids:
+ return metadata
+ indexed_ids: List[str] = []
+ index_rows: List[tuple[SessionMemoryRecordV2, str, Dict[str, float]]] = []
+ for memory_id in _dedupe(stored_ids):
+ record = getattr(graph, "records_by_id", {}).get(memory_id)
+ if record is None:
+ continue
+ index_text = _embedder_index_record_text(record, turn_text=turn_text)
+ terms = _embedder_index_term_weights(
+ index_text,
+ max_terms=max_terms,
+ )
+ if not terms:
+ continue
+ index_rows.append((record, index_text, terms))
+ dense_vectors, dense_metadata = _embedder_dense_vectors_for_texts(
+ [index_text for _, index_text, _ in index_rows],
+ mode=normalized_mode,
+ )
+ metadata.update(dense_metadata)
+ for row_index, (record, _, terms) in enumerate(index_rows):
+ dense_vector = dense_vectors[row_index] if row_index < len(dense_vectors) else []
+ record_metadata = dict(record.metadata or {})
+ record_metadata.update(
+ {
+ "write_embedder_index_enabled": True,
+ "write_embedder_index_mode": normalized_mode,
+ "write_embedder_index_version": index_version,
+ "write_embedder_index_source_turn": int(turn_index),
+ "write_embedder_index_term_count": int(len(terms)),
+ "write_embedder_index_terms": dict(terms),
+ "write_embedder_index_top_terms": list(terms.keys())[:24],
+ }
+ )
+ if dense_vector:
+ record_metadata.update(
+ {
+ "write_embedder_dense_enabled": True,
+ "write_embedder_dense_backend": dense_metadata.get("write_embedder_dense_backend", ""),
+ "write_embedder_dense_model": dense_metadata.get("write_embedder_dense_model", ""),
+ "write_embedder_dense_dim": int(len(dense_vector)),
+ "write_embedder_dense_vector": list(dense_vector),
+ }
+ )
+ elif _embedder_index_uses_bge_m3(normalized_mode):
+ record_metadata.update(
+ {
+ "write_embedder_dense_enabled": False,
+ "write_embedder_dense_error": dense_metadata.get("write_embedder_dense_error", "dense_vector_unavailable"),
+ }
+ )
+ record.metadata = record_metadata
+ indexed_ids.append(record.memory_id)
+ metadata.update(
+ {
+ "write_embedder_index_enabled": bool(indexed_ids),
+ "write_embedder_index_record_count": int(len(indexed_ids)),
+ "write_embedder_index_record_ids": list(indexed_ids[:24]),
+ }
+ )
+ return metadata
+
+
+def _embedder_index_recall_event_ids(
+ query: str,
+ *,
+ grouped_hits: Mapping[str, Sequence[MemoryHit]],
+ mode: str,
+ limit: int,
+ max_terms: int,
+) -> Dict[str, Any]:
+ normalized_mode = _normalize(mode)
+ index_version = _embedder_index_version_for_mode(normalized_mode)
+ metadata: Dict[str, Any] = {
+ "embedder_index_recall_enabled": False,
+ "embedder_index_recall_mode": normalized_mode or "off",
+ "embedder_index_recall_version": index_version,
+ "embedder_index_event_ids": [],
+ "embedder_index_event_scores": {},
+ "embedder_index_record_count": 0,
+ }
+ if not _embedder_index_enabled(normalized_mode):
+ return {"event_ids": [], "metadata": metadata}
+ query_terms = _embedder_index_term_weights(query, max_terms=max_terms)
+ query_vectors, query_dense_metadata = _embedder_dense_vectors_for_texts([query], mode=normalized_mode)
+ query_vector = query_vectors[0] if query_vectors else []
+ metadata.update(
+ {
+ "embedder_dense_recall_enabled": bool(query_vector),
+ "embedder_dense_recall_backend": query_dense_metadata.get("write_embedder_dense_backend", ""),
+ "embedder_dense_recall_model": query_dense_metadata.get("write_embedder_dense_model", ""),
+ "embedder_dense_recall_error": query_dense_metadata.get("write_embedder_dense_error", ""),
+ }
+ )
+ if not query_terms and not query_vector:
+ metadata["embedder_index_recall_reason"] = "empty_query_terms"
+ return {"event_ids": [], "metadata": metadata}
+ scored_events: List[tuple[str, float, int]] = []
+ indexed_record_count = 0
+ dense_record_count = 0
+ for event_id, group_hits in grouped_hits.items():
+ event_score = 0.0
+ event_turn = 0
+ for hit in list(group_hits or []):
+ hit_metadata = dict(hit.metadata or {})
+ raw_terms = hit_metadata.get("write_embedder_index_terms")
+ raw_vector = hit_metadata.get("write_embedder_dense_vector")
+ if not isinstance(raw_terms, Mapping) and not isinstance(raw_vector, list):
+ continue
+ indexed_record_count += int(isinstance(raw_terms, Mapping))
+ dense_record_count += int(isinstance(raw_vector, list) and bool(raw_vector))
+ sparse_score = 0.0
+ if isinstance(raw_terms, Mapping):
+ for term, query_weight in query_terms.items():
+ try:
+ sparse_score += float(query_weight) * float(raw_terms.get(term, 0.0) or 0.0)
+ except (TypeError, ValueError):
+ continue
+ dense_score = 0.0
+ if query_vector and isinstance(raw_vector, list) and raw_vector:
+ for query_value, record_value in zip(query_vector, raw_vector):
+ try:
+ dense_score += float(query_value) * float(record_value)
+ except (TypeError, ValueError):
+ continue
+ score = max(float(sparse_score), float(dense_score) + (0.15 * float(sparse_score) if dense_score > 0.0 else 0.0))
+ if score <= 0.0:
+ continue
+ event_score = max(event_score, float(score))
+ event_turn = max(event_turn, int(hit.turn_index or 0))
+ if event_score > 0.0:
+ scored_events.append((_clean_text(event_id), round(event_score, 6), event_turn))
+ scored_events.sort(key=lambda item: (-float(item[1]), -int(item[2]), item[0]))
+ selected = scored_events[: max(1, int(limit or 1))]
+ event_ids = [event_id for event_id, _, _ in selected if event_id]
+ event_scores = {event_id: score for event_id, score, _ in selected if event_id}
+ metadata.update(
+ {
+ "embedder_index_recall_enabled": True,
+ "embedder_index_event_ids": list(event_ids),
+ "embedder_index_event_scores": dict(event_scores),
+ "embedder_index_query_terms": list(query_terms.keys())[:32],
+ "embedder_index_record_count": int(indexed_record_count),
+ "embedder_dense_record_count": int(dense_record_count),
+ }
+ )
+ return {"event_ids": event_ids, "metadata": metadata}
+
+
+_IDENTIFIER_GENERIC_TOKENS = {
+ "api",
+ "agent",
+ "code",
+ "codename",
+ "context",
+ "debug",
+ "goal",
+ "memory",
+ "model",
+ "name",
+ "project",
+ "retrieval",
+ "runtime",
+ "session",
+ "target",
+ "test",
+ "turn",
+}
+_IDENTIFIER_REQUEST_RE = re.compile(
+ r"\b(code\s*name|codename|identifier|alias|project\s+name|model\s+name|api\s+name)\b|"
+ r"(代号|编号|标识符|名称|名字|别名|项目名|模型名|接口名)",
+ flags=re.IGNORECASE,
+)
+_IDENTIFIER_TOKEN_RE = re.compile(r"\b[A-Za-z][A-Za-z0-9_-]{2,63}\b")
+
+
+def _query_identifier_tokens(query: Any) -> List[str]:
+ text = _clean_text(query)
+ tokens: List[str] = []
+ for match in _IDENTIFIER_TOKEN_RE.finditer(text):
+ token = match.group(0)
+ lowered = token.lower()
+ if lowered in _IDENTIFIER_GENERIC_TOKENS:
+ continue
+ has_inner_upper = any(ch.isupper() for ch in token[1:])
+ has_digit = any(ch.isdigit() for ch in token)
+ has_joiner = "-" in token or "_" in token
+ if has_inner_upper or has_digit or has_joiner:
+ tokens.append(token)
+ return _dedupe(tokens, max_items=8)
+
+
+def _query_requests_identifier_fact(query: Any) -> bool:
+ return bool(_IDENTIFIER_REQUEST_RE.search(_clean_text(query)))
+
+
+def _hit_text_for_identifier_match(hit: MemoryHit) -> str:
+ metadata = dict(hit.metadata or {})
+ values = [
+ hit.memory_id,
+ hit.category,
+ hit.value,
+ hit.relation,
+ hit.slot_key,
+ hit.source_kind,
+ *list(hit.anchors or []),
+ metadata.get("raw_text", ""),
+ metadata.get("source_turn_text", ""),
+ metadata.get("source_span", ""),
+ metadata.get("event_phrase", ""),
+ metadata.get("profile_value", ""),
+ metadata.get("target_status", ""),
+ ]
+ return " ".join(_clean_text(value) for value in values if _clean_text(value))
+
+
+def _copy_hit_with_identifier_boost(hit: MemoryHit, *, score: float, reasons: Sequence[str], matched_tokens: Sequence[str]) -> MemoryHit:
+ metadata = dict(hit.metadata or {})
+ metadata.update(
+ {
+ "identifier_protected": True,
+ "identifier_protected_score": round(float(score), 6),
+ "identifier_protected_reasons": list(reasons)[:8],
+ "identifier_protected_matched_tokens": list(matched_tokens)[:8],
+ "identifier_protected_original_score": round(float(hit.score), 6),
+ }
+ )
+ return MemoryHit(
+ memory_id=hit.memory_id,
+ category=hit.category,
+ value=hit.value,
+ relation=hit.relation,
+ anchors=list(hit.anchors),
+ score=max(float(hit.score), round(1.0 + float(score), 6)),
+ source_kind=hit.source_kind,
+ slot_key=hit.slot_key,
+ state=hit.state,
+ turn_index=int(hit.turn_index),
+ metadata=metadata,
+ )
+
+
+def _identifier_protected_hits(
+ *,
+ query: str,
+ final_hits: Sequence[MemoryHit],
+ candidate_hits: Sequence[MemoryHit],
+ top_k: int,
+) -> Dict[str, Any]:
+ identifier_tokens = _query_identifier_tokens(query)
+ identifier_request = _query_requests_identifier_fact(query)
+ if not identifier_tokens and not identifier_request:
+ return {"enabled": False, "hits": list(final_hits), "promoted_hits": [], "metadata": {"identifier_protected_enabled": False}}
+
+ query_text = _clean_text(query)
+ pool: Dict[str, MemoryHit] = {}
+ for hit in list(final_hits) + list(candidate_hits):
+ key = hit.memory_id or f"{_hit_event_id(hit)}::{hit.slot_key}::{hit.value[:80]}"
+ if key and key not in pool:
+ pool[key] = hit
+
+ scored: List[tuple[float, MemoryHit, List[str], List[str]]] = []
+ for hit in pool.values():
+ hit_text = _hit_text_for_identifier_match(hit)
+ hit_lower = hit_text.lower()
+ matched_tokens = [token for token in identifier_tokens if token.lower() in hit_lower]
+ reasons: List[str] = []
+ score = 0.0
+ if matched_tokens:
+ score += 12.0 + float(len(matched_tokens))
+ reasons.append("exact_identifier_match")
+ if identifier_request:
+ if any(term in hit_lower for term in ("codename", "code name", "identifier", "alias", "project_codename")):
+ score += 8.0
+ reasons.append("identifier_field_match")
+ if any(term in hit_text for term in ("代号", "编号", "标识符", "名称", "名字", "别名")):
+ score += 8.0
+ reasons.append("identifier_cjk_field_match")
+ if "项目" in query_text and "项目" in hit_text:
+ score += 1.5
+ reasons.append("project_term_match")
+ if score <= 0:
+ continue
+ scored.append((score, hit, matched_tokens, reasons))
+
+ scored.sort(key=lambda item: (item[0], float(item[1].score), int(item[1].turn_index or 0)), reverse=True)
+ promoted = [
+ _copy_hit_with_identifier_boost(hit, score=score, reasons=reasons, matched_tokens=matched_tokens)
+ for score, hit, matched_tokens, reasons in scored[:2]
+ ]
+ if not promoted:
+ return {
+ "enabled": True,
+ "hits": list(final_hits),
+ "promoted_hits": [],
+ "metadata": {
+ "identifier_protected_enabled": True,
+ "identifier_query_tokens": identifier_tokens,
+ "identifier_request": bool(identifier_request),
+ "identifier_promoted_count": 0,
+ },
+ }
+
+ promoted_keys = {hit.memory_id or f"{_hit_event_id(hit)}::{hit.slot_key}::{hit.value[:80]}" for hit in promoted}
+ merged = list(promoted)
+ for hit in final_hits:
+ key = hit.memory_id or f"{_hit_event_id(hit)}::{hit.slot_key}::{hit.value[:80]}"
+ if key in promoted_keys:
+ continue
+ merged.append(hit)
+ if len(merged) >= max(1, int(top_k)):
+ break
+ return {
+ "enabled": True,
+ "hits": merged[: max(1, int(top_k))],
+ "promoted_hits": promoted,
+ "metadata": {
+ "identifier_protected_enabled": True,
+ "identifier_query_tokens": identifier_tokens,
+ "identifier_request": bool(identifier_request),
+ "identifier_promoted_count": len(promoted),
+ "identifier_promoted_ids": [hit.memory_id for hit in promoted],
+ },
+ }
+
+
+def _trim_prompt_text(value: Any, *, max_chars: int = 220) -> str:
+ text = _clean_text(value)
+ if len(text) <= max_chars:
+ return text
+ return f"{text[: max(0, max_chars - 3)].rstrip()}..."
+
+
+def _prompt_hit_payload(hit: MemoryHit) -> Dict[str, Any]:
+ return {
+ "memory_id": hit.memory_id,
+ "slot_key": hit.slot_key,
+ "category": hit.category,
+ "value": _trim_prompt_text(hit.value),
+ "relation": hit.relation,
+ "anchors": [_trim_prompt_text(anchor, max_chars=80) for anchor in list(hit.anchors)[:4]],
+ "score": round(float(hit.score), 6),
+ "state": hit.state,
+ "turn_index": int(hit.turn_index),
+ "source_kind": hit.source_kind,
+ }
+
+
+def _prompt_record_payload(record: SessionMemoryRecordV2) -> Dict[str, Any]:
+ metadata = dict(record.metadata or {})
+ return {
+ "slot_key": record.slot_key,
+ "category": record.category,
+ "value": _trim_prompt_text(record.value),
+ "state": record.state,
+ "turn_index": int(record.turn_index),
+ "anchors": [_trim_prompt_text(anchor, max_chars=80) for anchor in list(record.anchor_concepts)[:4]],
+ "memory_role": _clean_text(metadata.get("memory_role", "")),
+ "authority": _clean_text(metadata.get("authority", "")),
+ }
+
+
+def _graph_prompt_state_summary(graph: SessionMemoryGraphV2, retrieval: MemoryRetrieval) -> Dict[str, Any]:
+ active_slots: List[Dict[str, Any]] = []
+ for slot_key, record_id in list(graph.slot_heads.items())[:_GRAPH_PROMPT_MAX_ACTIVE_SLOTS]:
+ record = graph.records_by_id.get(record_id)
+ if record is None:
+ continue
+ active_slots.append(_prompt_record_payload(record))
+ top_hits = [_prompt_hit_payload(hit) for hit in list(retrieval.hits)[:_GRAPH_PROMPT_MAX_HITS]]
+ relation_preview = [
+ {
+ "from": _trim_prompt_text(item.get("from", ""), max_chars=72),
+ "to": _trim_prompt_text(item.get("to", ""), max_chars=72),
+ "relation": _clean_text(item.get("relation", "")),
+ }
+ for item in list(retrieval.relations)[:_GRAPH_PROMPT_MAX_RELATIONS]
+ ]
+ summary = {
+ "records": len(graph.records_by_id),
+ "active_slots": len(graph.slot_heads),
+ "turn_index": int(graph.turn_index),
+ "noise_turn_count": int(graph.noise_turn_count),
+ "answer_support_events": len(graph.answer_support_log),
+ "top_hits": top_hits,
+ "active_slot_records": active_slots,
+ "relation_preview": relation_preview,
+ "context_truncated": False,
+ "truncation_reason": "",
+ }
+ truncated = False
+ truncation_reason = ""
+ while len(json.dumps(summary, ensure_ascii=False)) > _GRAPH_PROMPT_MAX_CHARS:
+ if len(summary["top_hits"]) > 1:
+ summary["top_hits"] = summary["top_hits"][:-1]
+ truncated = True
+ truncation_reason = "trimmed_top_hits"
+ continue
+ if len(summary["active_slot_records"]) > 1:
+ summary["active_slot_records"] = summary["active_slot_records"][:-1]
+ truncated = True
+ truncation_reason = "trimmed_active_slots"
+ continue
+ if len(summary["relation_preview"]) > 2:
+ summary["relation_preview"] = summary["relation_preview"][:-1]
+ truncated = True
+ truncation_reason = "trimmed_relations"
+ continue
+ break
+ summary["context_truncated"] = truncated
+ summary["truncation_reason"] = truncation_reason
+ return summary
+
+
+def _state_stats(*, storage_bytes: int, retrieval_context_tokens: int, total_state_tokens: int, **extra: Any) -> Dict[str, Any]:
+ return {
+ **extra,
+ "storage_bytes": int(storage_bytes),
+ "context_token_estimate": int(retrieval_context_tokens),
+ "retrieval_context_token_estimate": int(retrieval_context_tokens),
+ "total_state_token_estimate": int(total_state_tokens),
+ }
+
+
+def _relation_hit(hit: MemoryHit, *, weight_bias: float = 0.0) -> Dict[str, Any]:
+ if not hit.anchors:
+ return {}
+ anchor = hit.anchors[0]
+ if not anchor or anchor == hit.value:
+ return {}
+ return {
+ "from": anchor,
+ "to": hit.value,
+ "relation": hit.relation,
+ "weight": round(max(0.25, min(0.98, 0.42 + hit.score * 0.4 + weight_bias)), 6),
+ "source_kind": hit.source_kind,
+ "memory_id": hit.memory_id,
+ }
+
+
+def _raw_hit_to_memory_hit(payload: Dict[str, Any]) -> MemoryHit:
+ metadata = dict(payload.get("metadata", {}) or {})
+ if payload.get("supersedes"):
+ metadata["supersedes"] = list(payload.get("supersedes", []) or [])
+ slot_key = stable_slot_key(
+ category=str(payload.get("category", "")),
+ value=str(payload.get("value", "")),
+ anchors=[str(anchor) for anchor in payload.get("anchor_concepts", payload.get("anchors", [])) or [] if _clean_text(anchor)],
+ slot_key=str(payload.get("slot_key", metadata.get("slot", ""))),
+ relation=str(payload.get("relation", "related_to")),
+ metadata=metadata,
+ )
+ return MemoryHit(
+ memory_id=str(payload.get("memory_id", "")),
+ category=str(payload.get("category", "")),
+ value=str(payload.get("value", "")),
+ relation=str(payload.get("relation", "related_to")),
+ anchors=[str(anchor) for anchor in payload.get("anchor_concepts", payload.get("anchors", [])) or [] if _clean_text(anchor)],
+ score=float(payload.get("score", payload.get("relevance", 0.0)) or 0.0),
+ source_kind=str(payload.get("source_kind", "memory")),
+ slot_key=slot_key,
+ state=str(payload.get("state", payload.get("metadata", {}).get("state", "active")) or "active"),
+ turn_index=int(payload.get("turn_index", 0) or 0),
+ metadata=metadata,
+ )
+
+
+def _restore_hit_scores(hits: List[MemoryHit], scored_lookup: Dict[str, MemoryHit]) -> List[MemoryHit]:
+ restored: List[MemoryHit] = []
+ for hit in hits:
+ scored = scored_lookup.get(hit.memory_id)
+ if scored:
+ hit.score = max(float(hit.score), float(scored.score))
+ if not hit.anchors and scored.anchors:
+ hit.anchors = list(scored.anchors)
+ restored.append(hit)
+ return restored
+
+
+def _current_subject_query(query: str) -> bool:
+ lowered = _normalize(query)
+ return bool(
+ _public_query_subject(query)
+ and any(marker in lowered for marker in ("right now", "current", "currently", "active", "now", "当前", "现在"))
+ )
+
+
+def _record_subject_signatures(record: SessionMemoryRecordV2) -> set[str]:
+ metadata = dict(record.metadata or {})
+ signatures = {
+ _normalize(metadata.get("subject_signature", "")).replace("-", "_"),
+ _public_subject_signature(metadata.get("subject", "")),
+ }
+ canonical_slot_key = _clean_text(metadata.get("canonical_slot_key", "") or record.slot_key)
+ if ".subject." in canonical_slot_key:
+ signatures.add(_public_subject_signature(canonical_slot_key.split(".subject.", 1)[-1]))
+ if ".subject." in record.slot_key:
+ signatures.add(_public_subject_signature(record.slot_key.split(".subject.", 1)[-1]))
+ signatures.discard("")
+ return signatures
+
+
+def _current_subject_protected_hits(
+ *,
+ query: str,
+ graph: SessionMemoryGraphV2,
+ final_hits: Sequence[MemoryHit],
+ top_k: int,
+) -> Dict[str, Any]:
+ if not _current_subject_query(query):
+ return {
+ "hits": list(final_hits),
+ "metadata": {"current_subject_resolver_enabled": False},
+ }
+ subject = _public_query_subject(query)
+ subject_signature = _public_subject_signature(subject)
+ if not subject_signature:
+ return {
+ "hits": list(final_hits),
+ "metadata": {
+ "current_subject_resolver_enabled": True,
+ "current_subject_resolver_reason": "no_subject_signature",
+ },
+ }
+ def _candidate_record(record: SessionMemoryRecordV2 | None) -> bool:
+ return bool(
+ record is not None
+ and _clean_text(record.source_kind).startswith("public_dialog")
+ and _normalize(record.category) != "question"
+ and subject_signature in _record_subject_signatures(record)
+ )
+
+ slot_head_candidates = [
+ record
+ for slot_key, memory_id in graph.slot_heads.items()
+ for record in [graph.records_by_id.get(memory_id)]
+ if _candidate_record(record)
+ and subject_signature in _record_subject_signatures(record)
+ ]
+ candidates = slot_head_candidates or [
+ record
+ for record in graph.records_by_id.values()
+ if record.state == "active" and _candidate_record(record)
+ ]
+ candidates.sort(
+ key=lambda record: (
+ int(
+ _normalize((record.metadata or {}).get("target_status", "")) == "current"
+ or _normalize(record.relation) == "current_subject_value"
+ ),
+ int(record.turn_index),
+ float(record.confidence),
+ float(record.salience),
+ ),
+ reverse=True,
+ )
+ promoted: List[MemoryHit] = []
+ for index, record in enumerate(candidates[: max(1, min(2, int(top_k or 1)))], start=1):
+ metadata = dict(record.metadata or {})
+ metadata.update(
+ {
+ "current_subject_resolver": True,
+ "current_subject_resolver_rank": index,
+ "current_subject_query_subject": subject,
+ "current_subject_query_signature": subject_signature,
+ "public_subject_match": True,
+ "public_subject_overlap": 1.0,
+ }
+ )
+ promoted.append(
+ MemoryHit(
+ memory_id=record.memory_id,
+ category=record.category,
+ value=record.value,
+ relation=record.relation,
+ anchors=list(record.anchor_concepts),
+ score=max(float(record.confidence), float(record.salience), 1.75),
+ source_kind=record.source_kind,
+ slot_key=record.slot_key,
+ state=record.state,
+ turn_index=int(record.turn_index),
+ metadata=metadata,
+ )
+ )
+ if not promoted:
+ return {
+ "hits": list(final_hits),
+ "metadata": {
+ "current_subject_resolver_enabled": True,
+ "current_subject_resolver_reason": "no_active_subject_head",
+ "current_subject_query_subject": subject,
+ "current_subject_query_signature": subject_signature,
+ },
+ }
+ promoted_ids = {hit.memory_id for hit in promoted}
+ merged_tail: List[MemoryHit] = []
+ for hit in final_hits:
+ if hit.memory_id in promoted_ids:
+ continue
+ metadata = dict(hit.metadata or {})
+ hit_state = _normalize(hit.state)
+ same_subject = subject_signature in {
+ _normalize(metadata.get("subject_signature", "")).replace("-", "_"),
+ _public_subject_signature(metadata.get("subject", "")),
+ _public_subject_signature(hit.slot_key.split(".subject.", 1)[-1]) if ".subject." in hit.slot_key else "",
+ _public_subject_signature(_clean_text(metadata.get("canonical_slot_key", "")).split(".subject.", 1)[-1])
+ if ".subject." in _clean_text(metadata.get("canonical_slot_key", ""))
+ else "",
+ }
+ if same_subject and hit_state in {"superseded", "evidence", "historical", "stale", "false"}:
+ continue
+ merged_tail.append(hit)
+ merged = [*promoted, *merged_tail]
+ limit = max(int(top_k or 1), len(promoted))
+ return {
+ "hits": merged[:limit],
+ "metadata": {
+ "current_subject_resolver_enabled": True,
+ "current_subject_resolver_reason": "promoted_active_subject_head",
+ "current_subject_query_subject": subject,
+ "current_subject_query_signature": subject_signature,
+ "current_subject_promoted_memory_ids": [hit.memory_id for hit in promoted],
+ },
+ }
+
+
+def _depth_chain_protected_hits(
+ *,
+ query: str,
+ graph: SessionMemoryGraphV2,
+ final_hits: Sequence[MemoryHit],
+ top_k: int,
+) -> Dict[str, Any]:
+ seed_memory_ids = [hit.memory_id for hit in final_hits if hit.memory_id]
+ chain = graph.depth_chain_for_query(
+ query,
+ seed_memory_ids=seed_memory_ids,
+ top_k=max(3, min(8, int(top_k or 1))),
+ )
+ if not chain.get("enabled") or not chain.get("nodes"):
+ return {
+ "hits": list(final_hits),
+ "metadata": {
+ "memory_chain_enabled": bool(chain.get("enabled", False)),
+ "memory_chain_reason": _clean_text(chain.get("reason", "")),
+ "memory_chain_node_count": 0,
+ "memory_chain_edge_count": 0,
+ "memory_chain": chain,
+ },
+ }
+ seen = {hit.memory_id for hit in final_hits if hit.memory_id}
+ chain_hits: List[MemoryHit] = []
+ for rank, node in enumerate(list(chain.get("nodes", []) or []), start=1):
+ if not isinstance(node, Mapping):
+ continue
+ memory_id = _clean_text(node.get("memory_id", ""))
+ if not memory_id or memory_id in seen:
+ continue
+ payload = dict(node)
+ metadata = dict(payload.get("metadata", {}) or {})
+ metadata.update(
+ {
+ "memory_chain_protected": True,
+ "memory_chain_rank": int(rank),
+ "memory_chain_subject_signature": _clean_text(chain.get("subject_signature", "")),
+ "memory_chain_depth_layer": _clean_text(metadata.get("depth_layer", "")) or "core_view",
+ }
+ )
+ payload["metadata"] = metadata
+ payload["score"] = max(float(payload.get("score", 0.0) or 0.0), 0.62 - (rank * 0.01))
+ chain_hits.append(_raw_hit_to_memory_hit(payload))
+ seen.add(memory_id)
+ limit = max(int(top_k or 1), min(12, int(top_k or 1) + max(0, len(chain_hits))))
+ merged = [*list(final_hits), *chain_hits]
+ return {
+ "hits": merged[:limit],
+ "metadata": {
+ "memory_chain_enabled": True,
+ "memory_chain_reason": _clean_text(chain.get("reason", "")),
+ "memory_chain_subject_signature": _clean_text(chain.get("subject_signature", "")),
+ "memory_chain_node_count": int(chain.get("node_count", 0) or 0),
+ "memory_chain_edge_count": int(chain.get("edge_count", 0) or 0),
+ "memory_chain_depth_layers": list(chain.get("depth_layers", []) or []),
+ "memory_chain": chain,
+ },
+ }
+
+
+def _is_public_dialog_hit(hit: MemoryHit) -> bool:
+ return _clean_text(hit.source_kind).startswith("public_dialog")
+
+
+def _normalized_runtime_signature(prefix: str, value: str) -> str:
+ normalized = _normalize(value)
+ if not normalized:
+ return ""
+ return f"{prefix}{normalized.replace('|', '_').replace(':', '_')}"
+
+
+def _runtime_event_key(hit: MemoryHit) -> str:
+ metadata = dict(hit.metadata or {})
+ explicit = _clean_text(metadata.get("event_id", ""))
+ if explicit:
+ return explicit
+ dia_id = _clean_text(metadata.get("dia_id", ""))
+ if dia_id:
+ return f"event::{dia_id}"
+ if not _is_public_dialog_hit(hit):
+ state_signature = _clean_text(metadata.get("state_signature", ""))
+ if state_signature:
+ return _normalized_runtime_signature("event::state::", state_signature)
+ memory_signature = _clean_text(metadata.get("memory_signature", ""))
+ if memory_signature:
+ return _normalized_runtime_signature("event::memory::", memory_signature)
+ slot_root = _public_slot_root(_clean_text(hit.slot_key))
+ if slot_root:
+ return slot_root
+ return _clean_text(hit.memory_id)
+
+
+def _runtime_event_turn_index_from_id(event_id: str) -> int:
+ text = _clean_text(event_id)
+ if not text:
+ return 0
+ match = re.search(r"(?::|_)(\d+)$", text)
+ if match:
+ return int(match.group(1))
+ matches = re.findall(r"\d+", text)
+ return int(matches[-1]) if matches else 0
+
+
+def _representative_event_hit(group_hits: Sequence[MemoryHit], *, query: str = "") -> MemoryHit | None:
+ semantic_source_kinds = {
+ "public_dialog_fact",
+ "public_dialog_preference",
+ "public_dialog_goal",
+ "public_dialog_constraint",
+ "public_dialog_status",
+ "public_dialog_profile",
+ "replacement_memory",
+ "session_memory",
+ }
+ query_tokens = set(_path_utility_tokens(query))
+
+ def rank(hit: MemoryHit) -> tuple[bool, float, bool, bool, bool, bool, float]:
+ metadata = dict(hit.metadata or {})
+ source_kind = _clean_text(hit.source_kind)
+ text_parts: List[Any] = [hit.value, hit.slot_key, hit.category]
+ if source_kind != "public_dialog_turn":
+ text_parts.extend([metadata.get("source_turn_text", ""), metadata.get("raw_text", "")])
+ text = " ".join(
+ _clean_text(item)
+ for item in text_parts
+ if _clean_text(item)
+ )
+ hit_tokens = set(_path_utility_tokens(text))
+ overlap = len(query_tokens & hit_tokens) if query_tokens else 0
+ has_number = bool(re.search(r"\b\d+\b", text))
+ is_semantic = source_kind in semantic_source_kinds or (
+ source_kind != "public_dialog_turn" and bool(_clean_text(metadata.get("memory_writer_role", "")))
+ )
+ direct_semantic_answer = bool(is_semantic and has_number and overlap >= 2)
+ query_score = float(overlap) + (0.75 if has_number and overlap else 0.0)
+ return (
+ direct_semantic_answer,
+ query_score,
+ is_semantic,
+ source_kind == "public_dialog_event",
+ source_kind == "public_dialog_turn",
+ source_kind in {"replacement_memory", "session_memory"},
+ float(hit.score),
+ )
+
+ ordered = sorted(
+ list(group_hits),
+ key=rank,
+ reverse=True,
+ )
+ return ordered[0] if ordered else None
+
+
+def _event_record_hits_from_graph(graph: SessionMemoryGraphV2, event_id: str) -> List[MemoryHit]:
+ normalized_event_id = _clean_text(event_id)
+ if not normalized_event_id:
+ return []
+ hits: List[MemoryHit] = []
+ for record in graph.records_by_id.values():
+ metadata = dict(record.metadata or {})
+ if _clean_text(metadata.get("event_id", "")) != normalized_event_id:
+ continue
+ state = _normalize(record.state)
+ if state not in {"active", "parallel_active", "evidence"}:
+ continue
+ hits.append(
+ MemoryHit(
+ memory_id=record.memory_id,
+ category=record.category,
+ value=record.value,
+ relation=record.relation,
+ anchors=list(record.anchor_concepts),
+ score=max(float(record.confidence), float(record.salience), 0.01),
+ source_kind=record.source_kind,
+ slot_key=record.slot_key,
+ state=record.state,
+ turn_index=int(record.turn_index),
+ metadata=metadata,
+ )
+ )
+ return hits
+
+def _hit_matches_path_support(path_type: str, hit: MemoryHit) -> bool:
+ metadata = dict(hit.metadata or {})
+ source_kind = _clean_text(hit.source_kind)
+ category = _clean_text(hit.category)
+ relation = _clean_text(hit.relation)
+ if path_type == "speaker_event_time":
+ return bool(
+ source_kind == "public_dialog_time"
+ or _clean_text(metadata.get("resolved_time_value", ""))
+ or _clean_text(metadata.get("resolved_date", ""))
+ or _clean_text(metadata.get("time_value", ""))
+ or _clean_text(metadata.get("time_display_value", ""))
+ or _clean_text(metadata.get("time_granularity", ""))
+ or relation == "event_date"
+ or category in {"time", "event_time"}
+ )
+ if path_type == "speaker_event_profile":
+ semantic_slot = _clean_text(metadata.get("semantic_slot", "")) or _clean_text(metadata.get("profile_type", ""))
+ return bool(
+ source_kind == "public_dialog_profile"
+ or semantic_slot in {"identity", "research_topic", "education", "occupation", "profile"}
+ or _clean_text(metadata.get("profile_value", ""))
+ or category == "profile"
+ )
+ if path_type == "speaker_event_status":
+ return bool(
+ _clean_text(metadata.get("target_status", ""))
+ or category in {"status", "stage_state"}
+ or relation == "status_of"
+ )
+ if path_type == "speaker_event_source_turn":
+ return bool(
+ source_kind in {"public_dialog_turn", "public_dialog_text", "public_dialog_auxiliary_evidence"}
+ or _clean_text(metadata.get("raw_text", ""))
+ or _clean_text(metadata.get("origin_query", ""))
+ or _clean_text(metadata.get("source_turn_text", ""))
+ or not _is_public_dialog_hit(hit)
+ )
+ return False
+
+
+def _support_hit_for_path(path_type: str, group_hits: Sequence[MemoryHit]) -> MemoryHit | None:
+ matching_hits = [hit for hit in group_hits if _hit_matches_path_support(path_type, hit)]
+ if matching_hits:
+ matching_hits.sort(key=lambda item: (float(item.score), int(item.turn_index)), reverse=True)
+ return matching_hits[0]
+ representative = _representative_event_hit(group_hits)
+ return representative
+
+
+def _path_support_node_id(path: Dict[str, Any]) -> str:
+ node_ids = list(path.get("node_ids", []) or [])
+ if len(node_ids) < 3:
+ return ""
+ return _clean_text(node_ids[2])
+
+
+def _event_ids_from_hits(hits: Sequence[MemoryHit]) -> List[str]:
+ return _dedupe(
+ _clean_text(dict(hit.metadata or {}).get("event_id", ""))
+ for hit in hits
+ if _clean_text(dict(hit.metadata or {}).get("event_id", ""))
+ )
+
+
+def _dia_ids_from_hits(hits: Sequence[MemoryHit]) -> List[str]:
+ return _dedupe(
+ _clean_text(dict(hit.metadata or {}).get("dia_id", ""))
+ for hit in hits
+ if _clean_text(dict(hit.metadata or {}).get("dia_id", ""))
+ )
+
+
+def _final_hit_role_priority(hit: MemoryHit) -> int:
+ metadata = dict(hit.metadata or {})
+ source_kind = _clean_text(hit.source_kind)
+ semantic_source_kinds = {
+ "public_dialog_fact",
+ "public_dialog_preference",
+ "public_dialog_goal",
+ "public_dialog_constraint",
+ "public_dialog_status",
+ "public_dialog_profile",
+ "public_dialog_profile_cluster",
+ "replacement_memory",
+ "session_memory",
+ }
+ if source_kind in semantic_source_kinds or (
+ source_kind != "public_dialog_turn" and bool(_clean_text(metadata.get("memory_writer_role", "")))
+ ):
+ return 0
+ if bool(metadata.get("profile_first_source_support")):
+ return 0
+ role = _clean_text(metadata.get("evidence_snippet_role", ""))
+ if role == "selected_path_support":
+ return 1
+ if role == "selected_event_representative":
+ return 2
+ if role == "selected_path_event":
+ return 3
+ return 4
+
+
+def _coverage_preserving_final_hits(
+ final_hits: Sequence[MemoryHit],
+ *,
+ selected_event_ids: Sequence[str],
+ top_k: int,
+) -> List[MemoryHit]:
+ """Keep selected-event coverage before filling the remaining prompt budget.
+
+ Learned selection can emit both path-support and event-representative snippets
+ for the same event. A pure score sort can then drop another selected event at
+ the top-k boundary, which hides recall/rerank successes from the answer head.
+ """
+
+ budget = max(1, int(top_k or 1))
+ hits = list(final_hits)
+ selected_order = [
+ _clean_text(event_id)
+ for event_id in selected_event_ids
+ if _clean_text(event_id)
+ ][:budget]
+ if not selected_order:
+ return sorted(hits, key=lambda item: float(item.score), reverse=True)[:budget]
+ hits_by_event: Dict[str, List[MemoryHit]] = {}
+ for hit in hits:
+ event_id = _clean_text(dict(hit.metadata or {}).get("event_id", ""))
+ if event_id:
+ hits_by_event.setdefault(event_id, []).append(hit)
+ selected: List[MemoryHit] = []
+ used_memory_ids = set()
+ for event_id in selected_order:
+ candidates = [hit for hit in hits_by_event.get(event_id, []) if hit.memory_id not in used_memory_ids]
+ if not candidates:
+ continue
+ candidates.sort(key=lambda item: (_final_hit_role_priority(item), -float(item.score), item.memory_id))
+ chosen = candidates[0]
+ selected.append(chosen)
+ used_memory_ids.add(chosen.memory_id)
+ if len(selected) >= budget:
+ return selected
+ remaining = [
+ hit
+ for hit in hits
+ if hit.memory_id not in used_memory_ids
+ ]
+ remaining.sort(key=lambda item: (-float(item.score), _final_hit_role_priority(item), item.memory_id))
+ for hit in remaining:
+ selected.append(hit)
+ if len(selected) >= budget:
+ break
+ return selected[:budget]
+
+
+def _dominant_answer_type(question_analysis: Dict[str, Any], answer_type_scores: Dict[str, Any]) -> str:
+ normalized_scores = {
+ _clean_text(answer_type): float(value or 0.0)
+ for answer_type, value in dict(answer_type_scores or {}).items()
+ if _clean_text(answer_type)
+ }
+ if bool(question_analysis.get("is_temporal", False)) and normalized_scores.get("time", 0.0) >= (normalized_scores.get("abstain", 0.0) - 0.05):
+ return "time"
+ if not bool(question_analysis.get("is_temporal", False)) and normalized_scores:
+ non_time_scores = {
+ answer_type: score
+ for answer_type, score in normalized_scores.items()
+ if answer_type not in {"time", "abstain"}
+ }
+ if non_time_scores:
+ return max(non_time_scores.items(), key=lambda item: (float(item[1]), item[0]))[0]
+ if not normalized_scores:
+ return "time" if bool(question_analysis.get("is_temporal", False)) else "event_text"
+ return max(normalized_scores.items(), key=lambda item: (float(item[1]), item[0]))[0]
+
+
+def _answer_type_preferred_path_types(question_analysis: Dict[str, Any], answer_type_scores: Dict[str, Any]) -> List[str]:
+ dominant_answer_type = _dominant_answer_type(question_analysis, answer_type_scores)
+ if dominant_answer_type == "time" or bool(question_analysis.get("is_temporal", False)):
+ return ["speaker_event_time", "speaker_event_source_turn", "speaker_event_status", "speaker_event_profile"]
+ if dominant_answer_type == "profile":
+ return ["speaker_event_profile", "speaker_event_source_turn", "speaker_event_status", "speaker_event_time"]
+ if dominant_answer_type == "multi_evidence":
+ return ["speaker_event_source_turn", "speaker_event_time", "speaker_event_profile", "speaker_event_status"]
+ return ["speaker_event_source_turn", "speaker_event_time", "speaker_event_profile", "speaker_event_status"]
+
+
+def _reconciled_focused_answer_type(
+ question_analysis: Dict[str, Any],
+ answer_type_scores: Dict[str, Any],
+ model_answer_type: str,
+) -> str:
+ model_type = _clean_text(model_answer_type)
+ dominant_type = _dominant_answer_type(question_analysis, answer_type_scores)
+ if bool(question_analysis.get("is_temporal", False)) and model_type not in {"", "time", "abstain"}:
+ return "time"
+ if model_type == "time" and not bool(question_analysis.get("is_temporal", False)):
+ return dominant_type if dominant_type != "time" else "event_text"
+ return model_type or dominant_type
+
+
+def _path_type_is_focus_compatible(path_type: str, *, focused_answer_type: str, question_analysis: Dict[str, Any]) -> bool:
+ normalized_path_type = _clean_text(path_type)
+ normalized_answer_type = _clean_text(focused_answer_type)
+ if normalized_answer_type == "time" or bool(question_analysis.get("is_temporal", False)):
+ return normalized_path_type in {"speaker_event_time", "speaker_event_source_turn"}
+ if normalized_answer_type == "profile":
+ return normalized_path_type in {"speaker_event_profile", "speaker_event_source_turn"}
+ return True
+
+
+def _rank_focus_compatible_path_ids(
+ *,
+ runtime_paths: Mapping[str, Dict[str, Any]],
+ selected_event_ids: Sequence[str],
+ path_scores: Mapping[str, Any],
+ event_scores: Mapping[str, Any],
+ temporal_scores: Mapping[str, Any],
+ question_analysis: Dict[str, Any],
+ answer_type_scores: Dict[str, Any],
+ focused_answer_type: str,
+) -> List[str]:
+ selected_event_id_set = {_clean_text(event_id) for event_id in selected_event_ids if _clean_text(event_id)}
+ preferred_types = _answer_type_preferred_path_types(question_analysis, answer_type_scores)
+ if _clean_text(focused_answer_type) == "profile" and "speaker_event_profile" not in preferred_types:
+ preferred_types = ["speaker_event_profile", *preferred_types]
+ if (_clean_text(focused_answer_type) == "time" or bool(question_analysis.get("is_temporal", False))) and "speaker_event_time" not in preferred_types:
+ preferred_types = ["speaker_event_time", "speaker_event_source_turn", *preferred_types]
+ ranked: List[str] = []
+ seen = set()
+ for preferred_type in preferred_types:
+ candidates: List[tuple[str, float]] = []
+ for path_id, path in runtime_paths.items():
+ if _clean_text(path.get("type", "")) != preferred_type:
+ continue
+ event_id = _clean_text(path.get("event_id", ""))
+ if selected_event_id_set and event_id not in selected_event_id_set:
+ continue
+ if not _path_type_is_focus_compatible(preferred_type, focused_answer_type=focused_answer_type, question_analysis=question_analysis):
+ continue
+ support_node_id = _path_support_node_id(path)
+ score = (
+ float(event_scores.get(event_id, 0.0) or 0.0)
+ + (0.20 * float(path_scores.get(path_id, 0.0) or 0.0))
+ + (0.15 * float(temporal_scores.get(support_node_id, 0.0) or 0.0))
+ )
+ candidates.append((path_id, score))
+ for path_id, _ in sorted(candidates, key=lambda item: (-float(item[1]), item[0])):
+ if path_id in seen:
+ continue
+ seen.add(path_id)
+ ranked.append(path_id)
+ if ranked and (_clean_text(focused_answer_type) == "time" or bool(question_analysis.get("is_temporal", False))):
+ if preferred_type in {"speaker_event_time", "speaker_event_source_turn"}:
+ break
+ return ranked
+
+
+def _repair_selected_paths_for_focus(
+ selected_path_ids: Sequence[str],
+ *,
+ runtime_paths: Mapping[str, Dict[str, Any]],
+ selected_event_ids: Sequence[str],
+ path_scores: Mapping[str, Any],
+ event_scores: Mapping[str, Any],
+ temporal_scores: Mapping[str, Any],
+ question_analysis: Dict[str, Any],
+ answer_type_scores: Dict[str, Any],
+ focused_answer_type: str,
+ limit: int,
+) -> tuple[List[str], bool, str]:
+ normalized_selected = [_clean_text(path_id) for path_id in selected_path_ids if _clean_text(path_id)]
+ if not normalized_selected:
+ return [], False, ""
+ incompatible = [
+ path_id
+ for path_id in normalized_selected
+ if not _path_type_is_focus_compatible(
+ _clean_text(runtime_paths.get(path_id, {}).get("type", "")),
+ focused_answer_type=focused_answer_type,
+ question_analysis=question_analysis,
+ )
+ ]
+ if not incompatible:
+ return normalized_selected, False, ""
+ compatible_ranked = _rank_focus_compatible_path_ids(
+ runtime_paths=runtime_paths,
+ selected_event_ids=selected_event_ids,
+ path_scores=path_scores,
+ event_scores=event_scores,
+ temporal_scores=temporal_scores,
+ question_analysis=question_analysis,
+ answer_type_scores=answer_type_scores,
+ focused_answer_type=focused_answer_type,
+ )
+ if not compatible_ranked:
+ return normalized_selected, False, ""
+ repaired = _dedupe(
+ [
+ *[path_id for path_id in normalized_selected if path_id not in incompatible],
+ *compatible_ranked,
+ ]
+ )[: max(1, limit)]
+ if repaired == normalized_selected:
+ return normalized_selected, False, ""
+ return repaired, True, "replaced_focus_incompatible_model_paths"
+
+
+def _runtime_node_by_id(runtime_graph: Mapping[str, Any]) -> Dict[str, Dict[str, Any]]:
+ return {
+ _clean_text(node.get("id", "")): dict(node)
+ for node in list(runtime_graph.get("nodes", []) or [])
+ if _clean_text(node.get("id", ""))
+ }
+
+
+def _runtime_event_subject_signature(runtime_nodes: Mapping[str, Dict[str, Any]], event_id: str) -> str:
+ node = dict(runtime_nodes.get(_clean_text(event_id), {}) or {})
+ metadata = dict(node.get("metadata", {}) or {})
+ return _clean_text(node.get("subject_signature", "")) or _clean_text(metadata.get("subject_signature", ""))
+
+
+def _path_utility_candidate_text(
+ path: Mapping[str, Any],
+ *,
+ runtime_nodes: Mapping[str, Dict[str, Any]],
+ grouped_hits: Mapping[str, Sequence[MemoryHit]],
+) -> str:
+ event_id = _clean_text(path.get("event_id", ""))
+ support_node_id = _path_support_node_id(dict(path))
+ node_texts = [
+ _clean_text(dict(runtime_nodes.get(event_id, {}) or {}).get("text", "")),
+ _clean_text(dict(runtime_nodes.get(support_node_id, {}) or {}).get("text", "")),
+ ]
+ path_type = _clean_text(path.get("type", ""))
+ support_hit = _support_hit_for_path(path_type, grouped_hits.get(event_id, []))
+ event_hit = _representative_event_hit(grouped_hits.get(event_id, []))
+ hit_texts = [
+ _clean_text(support_hit.value if support_hit is not None else ""),
+ _clean_text(event_hit.value if event_hit is not None else ""),
+ _runtime_source_turn_text(support_hit or event_hit, speaker=""),
+ ]
+ return " ".join(_dedupe([*node_texts, *hit_texts], max_items=6))
+
+
+def _path_utility_gate(
+ candidate_path_ids: Sequence[str],
+ *,
+ query: str,
+ runtime_graph: Mapping[str, Any],
+ runtime_paths: Mapping[str, Dict[str, Any]],
+ grouped_hits: Mapping[str, Sequence[MemoryHit]],
+ selected_path_ids: Sequence[str],
+ selected_event_ids_from_model: Sequence[str],
+ path_scores: Mapping[str, Any],
+ path_tunnel_support_scores: Mapping[str, Any],
+ question_analysis: Dict[str, Any],
+ focused_answer_type: str,
+ score_threshold: float,
+ limit: int,
+) -> Dict[str, Any]:
+ runtime_nodes = _runtime_node_by_id(runtime_graph)
+ query_tokens = set(_path_utility_tokens(query))
+ anchor_event_ids = _dedupe(
+ [
+ *[
+ _clean_text(runtime_paths.get(path_id, {}).get("event_id", ""))
+ for path_id in selected_path_ids
+ if _clean_text(runtime_paths.get(path_id, {}).get("event_id", ""))
+ ],
+ *[_clean_text(event_id) for event_id in selected_event_ids_from_model if _clean_text(event_id)],
+ ]
+ )
+ anchor_subject_signatures = {
+ signature
+ for signature in (
+ _runtime_event_subject_signature(runtime_nodes, event_id)
+ for event_id in anchor_event_ids
+ )
+ if signature
+ }
+ direct_path_ids: List[str] = []
+ contrast_path_ids: List[str] = []
+ latent_path_ids: List[str] = []
+ noise_path_ids: List[str] = []
+ utility_scores: Dict[str, float] = {}
+ utility_roles: Dict[str, str] = {}
+ utility_reasons: Dict[str, str] = {}
+ utility_overlap_tokens: Dict[str, List[str]] = {}
+ for path_id in _dedupe(candidate_path_ids):
+ path = dict(runtime_paths.get(path_id, {}) or {})
+ if not path:
+ continue
+ path_type = _clean_text(path.get("type", ""))
+ event_id = _clean_text(path.get("event_id", ""))
+ candidate_text = _path_utility_candidate_text(
+ path,
+ runtime_nodes=runtime_nodes,
+ grouped_hits=grouped_hits,
+ )
+ candidate_tokens = set(_path_utility_tokens(candidate_text))
+ overlap_tokens = sorted(query_tokens & candidate_tokens)
+ overlap_ratio = float(len(overlap_tokens)) / float(max(1, min(len(query_tokens), len(candidate_tokens))))
+ support_score = float(path_tunnel_support_scores.get(path_id, 0.0) or 0.0)
+ decision_score = float(path_scores.get(path_id, support_score) or 0.0)
+ path_score = max(support_score, decision_score)
+ subject_signature = _runtime_event_subject_signature(runtime_nodes, event_id)
+ same_subject_chain = bool(subject_signature and subject_signature in anchor_subject_signatures)
+ focus_compatible = _path_type_is_focus_compatible(
+ path_type,
+ focused_answer_type=focused_answer_type,
+ question_analysis=question_analysis,
+ )
+ utility_score = path_score + (0.20 * overlap_ratio) + (0.04 if same_subject_chain else 0.0)
+ utility_scores[path_id] = round(float(utility_score), 6)
+ utility_overlap_tokens[path_id] = overlap_tokens[:12]
+ if not focus_compatible:
+ role = "drift_noise"
+ reason = "focus_incompatible"
+ elif overlap_ratio >= 0.18 and path_score >= score_threshold:
+ role = "direct_support"
+ reason = "query_overlap_and_tunnel_score"
+ elif same_subject_chain and path_score >= score_threshold:
+ role = "contrast_support"
+ reason = "same_subject_deep_chain"
+ elif path_score >= score_threshold:
+ role = "latent_context"
+ reason = "tunnel_score_without_current_turn_utility"
+ else:
+ role = "drift_noise"
+ reason = "below_utility_threshold"
+ utility_roles[path_id] = role
+ utility_reasons[path_id] = reason
+ if role == "direct_support":
+ direct_path_ids.append(path_id)
+ elif role == "contrast_support":
+ contrast_path_ids.append(path_id)
+ elif role == "latent_context":
+ latent_path_ids.append(path_id)
+ else:
+ noise_path_ids.append(path_id)
+ injected_path_ids = _dedupe([*direct_path_ids, *contrast_path_ids], max_items=max(0, int(limit)))
+ overflow_latent_path_ids = [
+ path_id
+ for path_id in [*direct_path_ids, *contrast_path_ids]
+ if path_id not in set(injected_path_ids)
+ ]
+ latent_path_ids = _dedupe([*latent_path_ids, *overflow_latent_path_ids])
+ return {
+ "enabled": True,
+ "candidate_path_ids": list(_dedupe(candidate_path_ids)),
+ "direct_support_path_ids": list(direct_path_ids),
+ "contrast_support_path_ids": list(contrast_path_ids),
+ "latent_context_path_ids": list(latent_path_ids),
+ "drift_noise_path_ids": list(noise_path_ids),
+ "injected_path_ids": list(injected_path_ids),
+ "roles": dict(utility_roles),
+ "reasons": dict(utility_reasons),
+ "scores": dict(utility_scores),
+ "overlap_tokens": dict(utility_overlap_tokens),
+ "anchor_event_ids": list(anchor_event_ids),
+ "anchor_subject_signatures": sorted(anchor_subject_signatures),
+ }
+
+
+def _calibrated_path_score(
+ *,
+ path: Dict[str, Any],
+ base_score: float,
+ temporal_scores: Dict[str, Any],
+ question_analysis: Dict[str, Any],
+ answer_type_scores: Dict[str, Any],
+) -> float:
+ normalized_path_type = _clean_text(path.get("type", ""))
+ support_node_id = _path_support_node_id(path)
+ temporal_score = float(temporal_scores.get(support_node_id, 0.0) or 0.0)
+ normalized_answer_scores = {
+ _clean_text(answer_type): float(value or 0.0)
+ for answer_type, value in dict(answer_type_scores or {}).items()
+ if _clean_text(answer_type)
+ }
+ dominant_answer_type = _dominant_answer_type(question_analysis, normalized_answer_scores)
+ calibrated = float(base_score)
+ if dominant_answer_type == "time" or bool(question_analysis.get("is_temporal", False)):
+ if normalized_path_type == "speaker_event_time":
+ calibrated += (0.30 * temporal_score) + (0.12 * normalized_answer_scores.get("time", 0.0))
+ elif normalized_path_type == "speaker_event_source_turn":
+ calibrated -= 0.08 + (0.04 * normalized_answer_scores.get("time", 0.0))
+ elif normalized_path_type == "speaker_event_status":
+ calibrated -= 0.12
+ elif normalized_path_type == "speaker_event_profile":
+ calibrated -= 0.22 + (0.06 * normalized_answer_scores.get("time", 0.0))
+ elif dominant_answer_type == "profile":
+ if normalized_path_type == "speaker_event_profile":
+ calibrated += 0.12 * normalized_answer_scores.get("profile", 0.0)
+ elif normalized_path_type != "speaker_event_source_turn":
+ calibrated -= 0.08
+ else:
+ if normalized_path_type == "speaker_event_source_turn":
+ calibrated += 0.08 * max(
+ normalized_answer_scores.get("event_text", 0.0),
+ normalized_answer_scores.get("multi_evidence", 0.0),
+ )
+ elif normalized_path_type == "speaker_event_profile":
+ calibrated -= 0.04
+ return calibrated
+
+
+def _group_metadata_value(
+ group_hits: Sequence[MemoryHit],
+ key: str,
+ *,
+ source_kinds: Sequence[str] = (),
+) -> str:
+ source_kind_set = {_clean_text(item) for item in list(source_kinds) if _clean_text(item)}
+ candidates = [
+ hit
+ for hit in group_hits
+ if not source_kind_set or _clean_text(hit.source_kind) in source_kind_set
+ ]
+ candidates.sort(key=lambda item: (float(item.score), int(item.turn_index)), reverse=True)
+ for hit in candidates:
+ metadata = dict(hit.metadata or {})
+ value = _clean_text(metadata.get(key, ""))
+ if value:
+ return value
+ return ""
+
+
+def _runtime_event_sequence_key(session_name: str, turn_index: int, event_id: str) -> tuple[Any, ...]:
+ normalized_session = _clean_text(session_name)
+ session_number_match = re.search(r"(\d+)$", normalized_session)
+ if session_number_match:
+ return (0, int(session_number_match.group(1)), int(turn_index), _clean_text(event_id))
+ return (1, normalized_session, int(turn_index), _clean_text(event_id))
+
+
+def _runtime_source_turn_text(hit: MemoryHit | None, *, speaker: str) -> str:
+ if hit is None:
+ return ""
+ metadata = dict(hit.metadata or {})
+ source_turn_text = _clean_text(metadata.get("source_turn_text", ""))
+ if source_turn_text:
+ return source_turn_text
+ raw_text = _clean_text(metadata.get("raw_text", ""))
+ if raw_text:
+ auxiliary_text = _clean_text(metadata.get("auxiliary_evidence_text", ""))
+ if auxiliary_text and auxiliary_text.lower() not in raw_text.lower():
+ return f"{raw_text}\nAuxiliary evidence: {auxiliary_text}"
+ return raw_text
+ origin_query = _clean_text(metadata.get("origin_query", ""))
+ if origin_query:
+ return origin_query
+ text = _clean_text(hit.value)
+ if _clean_text(hit.source_kind) == "public_dialog_turn":
+ text = re.sub(r"^\[[^\]]+\]\s*", "", text)
+ if _clean_text(speaker):
+ text = re.sub(rf"^{re.escape(_clean_text(speaker))}\s*:\s*", "", text, flags=re.IGNORECASE)
+ text = re.sub(r"^[A-Za-z][A-Za-z0-9_' -]{0,40}:\s*", "", text)
+ return text
+
+
+def _runtime_event_signature(
+ *,
+ group_hits: Sequence[MemoryHit],
+ representative: MemoryHit,
+ speaker: str,
+ semantic_slot: str,
+ source_turn_hit: MemoryHit | None,
+) -> str:
+ existing = _group_metadata_value(group_hits, "event_signature") or _clean_text(dict(representative.metadata or {}).get("event_signature", ""))
+ if existing:
+ return existing
+ event_phrase = _group_metadata_value(group_hits, "event_phrase") or _clean_text(dict(representative.metadata or {}).get("event_phrase", ""))
+ source_turn_text = _runtime_source_turn_text(source_turn_hit, speaker=speaker)
+ base_text = event_phrase or _clean_text(representative.value) or source_turn_text
+ if not base_text:
+ return ""
+ return compute_public_event_signature(
+ base_text,
+ speaker=_clean_text(speaker),
+ semantic_slot=_clean_text(semantic_slot),
+ ) or _clean_text(base_text)
+
+
+def _build_runtime_graph_from_hits(query: str, hits: Sequence[MemoryHit]) -> Dict[str, Any]:
+ nodes: List[Dict[str, Any]] = []
+ edges: List[Dict[str, Any]] = []
+ paths: List[Dict[str, Any]] = []
+ node_ids = set()
+ grouped_hits: Dict[str, List[MemoryHit]] = {}
+ ordered_events: List[tuple[Any, ...]] = []
+ event_typed_metadata_by_id: Dict[str, Dict[str, Any]] = {}
+ typed_tunnel_edges: List[Dict[str, Any]] = []
+
+ def add_node(node: Dict[str, Any]) -> None:
+ node_id = _clean_text(node.get("id", ""))
+ if not node_id or node_id in node_ids:
+ return
+ node_ids.add(node_id)
+ nodes.append(node)
+
+ for hit in hits:
+ event_id = _runtime_event_key(hit)
+ grouped_hits.setdefault(event_id, []).append(hit)
+
+ for event_id, group_hits in grouped_hits.items():
+ representative = _representative_event_hit(group_hits)
+ if representative is None:
+ continue
+ metadata = dict(representative.metadata or {})
+ event_time_hit = _support_hit_for_path("speaker_event_time", group_hits)
+ event_profile_hit = _support_hit_for_path("speaker_event_profile", group_hits)
+ source_turn_hit = _support_hit_for_path("speaker_event_source_turn", group_hits)
+ event_time_display_value = (
+ _group_metadata_value(group_hits, "time_display_value", source_kinds=("public_dialog_time",))
+ or _group_metadata_value(group_hits, "resolved_date")
+ or _group_metadata_value(group_hits, "time_display_value")
+ )
+ event_time_value = (
+ _group_metadata_value(group_hits, "resolved_time_value", source_kinds=("public_dialog_time",))
+ or _group_metadata_value(group_hits, "resolved_time_value")
+ or _group_metadata_value(group_hits, "time_value")
+ )
+ event_time_granularity = (
+ _group_metadata_value(group_hits, "time_granularity", source_kinds=("public_dialog_time",))
+ or _group_metadata_value(group_hits, "time_granularity")
+ )
+ event_profile_type = (
+ _group_metadata_value(group_hits, "profile_type", source_kinds=("public_dialog_profile",))
+ or _group_metadata_value(group_hits, "semantic_slot", source_kinds=("public_dialog_profile",))
+ or _group_metadata_value(group_hits, "profile_type")
+ or _group_metadata_value(group_hits, "semantic_slot")
+ )
+ profile_hit = _support_hit_for_path("speaker_event_profile", group_hits)
+ event_profile_value = (
+ _group_metadata_value(group_hits, "profile_value")
+ or _clean_text(profile_hit.value if profile_hit is not None else "")
+ or (_clean_text(representative.value) if event_profile_type else "")
+ )
+ event_target_status = _group_metadata_value(group_hits, "target_status")
+ depth_layer = (
+ _clean_text(metadata.get("depth_layer", ""))
+ or _group_metadata_value(group_hits, "depth_layer")
+ or _clean_text(metadata.get("memory_chain_depth_layer", ""))
+ or _group_metadata_value(group_hits, "memory_chain_depth_layer")
+ )
+ subject_signature = (
+ _clean_text(metadata.get("subject_signature", ""))
+ or _group_metadata_value(group_hits, "subject_signature")
+ or _clean_text(metadata.get("memory_chain_subject_signature", ""))
+ or _group_metadata_value(group_hits, "memory_chain_subject_signature")
+ )
+ session_name = (
+ _group_metadata_value(group_hits, "session_name")
+ or _group_metadata_value(group_hits, "session_key")
+ or _group_metadata_value(group_hits, "scope_id")
+ or "runtime_session"
+ )
+ speaker = (
+ _clean_text(metadata.get("speaker", ""))
+ or _group_metadata_value(group_hits, "speaker")
+ or _clean_text(metadata.get("subject_signature", ""))
+ or (_clean_text(representative.anchors[0]) if representative.anchors else "")
+ or "speaker"
+ )
+ event_turn_index = int(getattr(representative, "turn_index", 0) or 0)
+ semantic_slot = (
+ _group_metadata_value(group_hits, "semantic_slot")
+ or _clean_text(metadata.get("semantic_slot", ""))
+ or _clean_text(metadata.get("profile_type", ""))
+ or ("profile" if event_profile_type else _clean_text(representative.category))
+ or "event"
+ )
+ teacher_fields = {
+ "event_phrase": _clean_text(metadata.get("event_phrase", "")) or _clean_text(representative.value),
+ "semantic_slot": semantic_slot,
+ "target_status": event_target_status,
+ "time_expression_span": event_time_display_value,
+ "time_granularity": event_time_granularity,
+ "profile_type": event_profile_type,
+ }
+ base_event_signature = _runtime_event_signature(
+ group_hits=group_hits,
+ representative=representative,
+ speaker=speaker,
+ semantic_slot=event_profile_type or teacher_fields["semantic_slot"],
+ source_turn_hit=source_turn_hit,
+ )
+ typed_tunnel_metadata = merge_typed_metadata([metadata, *[dict(hit.metadata or {}) for hit in group_hits]])
+ event_typed_metadata_by_id[event_id] = typed_tunnel_metadata
+ typed_signature = typed_tunnel_signature_text(typed_tunnel_metadata)
+ event_text = _clean_text(metadata.get("event_phrase", "")) or representative.value
+ depth_prefix = " ".join(
+ item
+ for item in (
+ f"subject {subject_signature.replace('_', ' ')}" if subject_signature else "",
+ f"depth layer {depth_layer.replace('_', ' ')}" if depth_layer else "",
+ )
+ if item
+ )
+ runtime_event_text = f"{depth_prefix} {event_text}".strip() if depth_prefix else event_text
+ event_signature = (
+ compute_public_event_signature(
+ runtime_event_text,
+ speaker=_clean_text(speaker),
+ semantic_slot=_clean_text(event_profile_type or teacher_fields["semantic_slot"]),
+ )
+ or base_event_signature
+ )
+ if typed_signature and typed_signature not in event_signature:
+ event_signature = f"{event_signature} {typed_signature}".strip()
+ speaker_node_id = f"{event_id}:speaker:{_normalize(speaker).replace(' ', '_') or 'speaker'}"
+ add_node({"id": speaker_node_id, "type": "speaker", "text": speaker, "metadata": {"speaker": speaker}})
+ add_node(
+ {
+ "id": event_id,
+ "type": "event",
+ "text": runtime_event_text,
+ "speaker": speaker,
+ "turn_index": event_turn_index,
+ "session_name": session_name,
+ "dia_id": _clean_text(metadata.get("dia_id", "")),
+ "event_signature": event_signature,
+ "slot_key": _clean_text(representative.slot_key),
+ "state_signature": _clean_text(metadata.get("state_signature", "")),
+ "memory_signature": _clean_text(metadata.get("memory_signature", "")),
+ "target_status": event_target_status,
+ "time_granularity": event_time_granularity,
+ "time_value": event_time_value,
+ "time_display_value": event_time_display_value,
+ "profile_type": event_profile_type,
+ "profile_value": event_profile_value,
+ "depth_layer": depth_layer,
+ "subject_signature": subject_signature,
+ "tmcra_node_tags": list(typed_tunnel_metadata.get("tmcra_node_tags", []) or []),
+ "tmcra_path_tags": list(typed_tunnel_metadata.get("tmcra_path_tags", []) or []),
+ "tmcra_tunnel_roles": list(typed_tunnel_metadata.get("tmcra_tunnel_roles", []) or []),
+ "tmcra_tunnel_group_key": _clean_text(typed_tunnel_metadata.get("tmcra_tunnel_group_key", "")),
+ "teacher_fields": teacher_fields,
+ "metadata": {
+ "speaker": speaker,
+ "session_name": session_name,
+ "dia_id": _clean_text(metadata.get("dia_id", "")),
+ "slot_key": _clean_text(representative.slot_key),
+ "state_signature": _clean_text(metadata.get("state_signature", "")),
+ "memory_signature": _clean_text(metadata.get("memory_signature", "")),
+ "target_status": event_target_status,
+ "time_granularity": event_time_granularity,
+ "time_value": event_time_value,
+ "time_display_value": event_time_display_value,
+ "profile_type": event_profile_type,
+ "profile_value": event_profile_value,
+ "event_signature": event_signature,
+ "depth_layer": depth_layer,
+ "subject_signature": subject_signature,
+ **typed_tunnel_metadata,
+ },
+ }
+ )
+ edges.append({"id": f"{speaker_node_id}->{event_id}:speaker_of", "source": speaker_node_id, "target": event_id, "type": "speaker_of"})
+
+ time_node_ids: List[str] = []
+ profile_node_ids: List[str] = []
+ status_node_ids: List[str] = []
+ source_turn_node_ids: List[str] = []
+
+ if event_time_display_value or event_time_value:
+ time_node_id = f"{event_id}:time"
+ time_hit_metadata = dict((event_time_hit or representative).metadata or {})
+ add_node(
+ {
+ "id": time_node_id,
+ "type": "time",
+ "text": event_time_display_value or event_time_value,
+ "turn_index": int(getattr(event_time_hit, "turn_index", 0) or 0),
+ "time_display_value": event_time_display_value,
+ "time_value": event_time_value,
+ "time_granularity": event_time_granularity,
+ "metadata": {
+ "time_display_value": event_time_display_value,
+ "time_value": event_time_value,
+ "time_granularity": event_time_granularity,
+ "resolved_date": _clean_text(time_hit_metadata.get("resolved_date", "")),
+ },
+ }
+ )
+ edges.append({"id": f"{event_id}->{time_node_id}:time_of", "source": event_id, "target": time_node_id, "type": "time_of"})
+ time_node_ids.append(time_node_id)
+ if event_profile_value:
+ profile_node_id = f"{event_id}:profile:{_normalize(event_profile_type).replace(' ', '_') or 'profile'}"
+ add_node(
+ {
+ "id": profile_node_id,
+ "type": "profile",
+ "text": event_profile_value,
+ "turn_index": int(getattr(event_profile_hit or representative, "turn_index", 0) or 0),
+ "profile_type": event_profile_type,
+ "profile_value": event_profile_value,
+ "metadata": {
+ "profile_type": event_profile_type,
+ "profile_value": event_profile_value,
+ },
+ }
+ )
+ edges.append({"id": f"{event_id}->{profile_node_id}:profile_of", "source": event_id, "target": profile_node_id, "type": "profile_of"})
+ profile_node_ids.append(profile_node_id)
+ source_turn_text = _runtime_source_turn_text(source_turn_hit or representative, speaker=speaker)
+ if source_turn_text:
+ source_turn_node_id = f"{event_id}:source_turn"
+ add_node(
+ {
+ "id": source_turn_node_id,
+ "type": "source_turn",
+ "text": source_turn_text,
+ "turn_index": int(getattr(source_turn_hit or representative, "turn_index", 0) or 0),
+ "metadata": {
+ "speaker": speaker,
+ "dia_id": _clean_text(dict((source_turn_hit or representative).metadata or {}).get("dia_id", "")),
+ },
+ }
+ )
+ edges.append({"id": f"{event_id}->{source_turn_node_id}:supported_by_turn", "source": event_id, "target": source_turn_node_id, "type": "supported_by_turn"})
+ source_turn_node_ids.append(source_turn_node_id)
+ if event_target_status:
+ status_node_id = f"{event_id}:status"
+ add_node({"id": status_node_id, "type": "status", "text": event_target_status, "metadata": {"target_status": event_target_status}})
+ edges.append({"id": f"{event_id}->{status_node_id}:status_of", "source": event_id, "target": status_node_id, "type": "status_of"})
+ status_node_ids.append(status_node_id)
+ event_paths = build_default_path_templates(
+ event_id=event_id,
+ speaker_node_id=speaker_node_id,
+ time_node_ids=time_node_ids,
+ profile_node_ids=profile_node_ids,
+ status_node_ids=status_node_ids,
+ source_turn_node_ids=source_turn_node_ids,
+ )
+ for path in event_paths:
+ path_metadata = dict(path.get("metadata", {}) or {})
+ path_metadata["tmcra_path_tags"] = list(typed_tunnel_metadata.get("tmcra_path_tags", []) or [])
+ path_metadata["tmcra_tunnel_group_key"] = _clean_text(typed_tunnel_metadata.get("tmcra_tunnel_group_key", ""))
+ path["metadata"] = path_metadata
+ path["tmcra_path_tags"] = path_metadata["tmcra_path_tags"]
+ paths.extend(event_paths)
+ ordered_events.append(_runtime_event_sequence_key(session_name, event_turn_index, event_id))
+ ordered_event_ids = [event_id for _, _, _, event_id in sorted(ordered_events)]
+ for previous_event_id, next_event_id in zip(ordered_event_ids, ordered_event_ids[1:]):
+ typed_edge_tags = typed_edge_tags_between(
+ event_typed_metadata_by_id.get(previous_event_id, {}),
+ event_typed_metadata_by_id.get(next_event_id, {}),
+ )
+ edges.append(
+ {
+ "id": f"{previous_event_id}->{next_event_id}:same_session_next",
+ "source": previous_event_id,
+ "target": next_event_id,
+ "type": "same_session_next",
+ "metadata": {
+ "tmcra_edge_tags": typed_edge_tags,
+ "typed_tunnel_edge": bool(typed_edge_tags),
+ },
+ }
+ )
+ for index, source_event_id in enumerate(ordered_event_ids):
+ for target_event_id in ordered_event_ids[index + 1 : index + 12]:
+ typed_edge_tags = typed_edge_tags_between(
+ event_typed_metadata_by_id.get(source_event_id, {}),
+ event_typed_metadata_by_id.get(target_event_id, {}),
+ )
+ if not typed_edge_tags:
+ continue
+ typed_tunnel_edges.append(
+ {
+ "id": f"{source_event_id}->{target_event_id}:typed_tunnel",
+ "source": source_event_id,
+ "target": target_event_id,
+ "type": "typed_tunnel_candidate",
+ "metadata": {
+ "tmcra_edge_tags": typed_edge_tags,
+ "typed_tunnel_edge": True,
+ },
+ }
+ )
+ if len(typed_tunnel_edges) >= 64:
+ break
+ if len(typed_tunnel_edges) >= 64:
+ break
+ return {
+ "conversation_id": "runtime",
+ "query": query,
+ "nodes": nodes,
+ "edges": edges,
+ "typed_tunnel_edges": typed_tunnel_edges,
+ "paths": paths,
+ "grouped_hits": grouped_hits,
+ }
+
+
+def _public_graph_hits(graph: SessionMemoryGraphV2) -> List[MemoryHit]:
+ public_hits: List[MemoryHit] = []
+ for record in graph.records_by_id.values():
+ if record.state != "active":
+ continue
+ if not _clean_text(record.source_kind).startswith("public_dialog"):
+ continue
+ metadata = dict(record.metadata or {})
+ public_hits.append(
+ MemoryHit(
+ memory_id=record.memory_id,
+ category=record.category,
+ value=record.value,
+ relation=record.relation,
+ anchors=list(record.anchor_concepts),
+ score=max(float(record.confidence), float(record.salience), 0.01),
+ source_kind=record.source_kind,
+ slot_key=record.slot_key,
+ state=record.state,
+ turn_index=int(record.turn_index),
+ metadata=metadata,
+ )
+ )
+ public_hits.sort(key=lambda item: (int(item.turn_index), float(item.score)), reverse=True)
+ return public_hits
+
+
+_AUDIT_ANCHOR_QUERY_RE = re.compile(
+ r"(?i)\b(?:remember|recall|earlier|previous|previously|old|before|mentioned|said|quote|verbatim|original|"
+ r"that time|last time|bring back|return to|go back to|turn\s*\d+)\b|"
+ r"(?:\u7b2c\s*\d+\s*[\u8f6e\u6b21]|\d+\s*\u8f6e|\u4e4b\u524d|\u4ee5\u524d|\u521a\u521a|"
+ r"\u539f\u8bdd|\u90a3\u6b21|\u65e7\u8bdd\u9898|\u56de\u5230|\u63d0\u8d77|\u8bb0\u5f97)"
+)
+
+_AUDIT_TURN_ANCHOR_RE = re.compile(
+ r"(?i)\bturn\s*#?\s*(\d{1,6})\b|"
+ r"(?:\u7b2c\s*(\d{1,6})\s*[\u8f6e\u6b21]|\b(\d{1,6})\s*\u8f6e\b)"
+)
+
+_AUDIT_ANCHOR_STOPWORDS = set(_HYBRID_SYMBOLIC_STOPWORDS) | {
+ "about",
+ "again",
+ "back",
+ "bring",
+ "can",
+ "could",
+ "did",
+ "discuss",
+ "discussed",
+ "earlier",
+ "from",
+ "just",
+ "keep",
+ "mentioned",
+ "old",
+ "one",
+ "previous",
+ "previously",
+ "really",
+ "recall",
+ "remember",
+ "return",
+ "said",
+ "still",
+ "that",
+ "the",
+ "thing",
+ "think",
+ "this",
+ "turn",
+ "what",
+ "when",
+ "where",
+ "with",
+ "you",
+}
+
+_AUDIT_ANCHOR_GENERIC_MATCH_TOKENS = {
+ "body",
+ "coherence",
+ "continuity",
+ "fiction",
+ "fray",
+ "memories",
+ "memory",
+ "narrative",
+ "ourselves",
+ "physical",
+ "really",
+ "stories",
+ "story",
+ "tell",
+ "trust",
+}
+
+_AUDIT_ANCHOR_PHRASE_TOKEN_SETS = [
+ {"spine", "book"},
+ {"body", "spine"},
+ {"stories", "pages"},
+ {"story", "pages"},
+ {"islands", "self"},
+ {"archipelago", "self"},
+ {"loom", "body"},
+ {"braid", "body"},
+]
+
+
+def _audit_anchor_query(query: str) -> bool:
+ return bool(_AUDIT_ANCHOR_QUERY_RE.search(_clean_text(query)))
+
+
+def _audit_anchor_turn_numbers(query: str) -> List[int]:
+ numbers: List[int] = []
+ for match in _AUDIT_TURN_ANCHOR_RE.finditer(_clean_text(query)):
+ for group in match.groups():
+ if not group:
+ continue
+ try:
+ value = int(group)
+ except Exception:
+ continue
+ if value > 0 and value not in numbers:
+ numbers.append(value)
+ return numbers
+
+
+def _hit_event_id(hit: MemoryHit) -> str:
+ metadata = dict(hit.metadata or {})
+ event_id = _clean_text(metadata.get("event_id", ""))
+ if event_id:
+ return event_id
+ dia_id = _clean_text(metadata.get("dia_id", ""))
+ if dia_id:
+ return f"event::{dia_id}"
+ if int(hit.turn_index or 0) > 0:
+ return f"event::realchat:{int(hit.turn_index)}"
+ return ""
+
+
+def _audit_anchor_hit_text(hit: MemoryHit) -> str:
+ metadata = dict(hit.metadata or {})
+ values = [
+ hit.value,
+ hit.category,
+ hit.source_kind,
+ hit.slot_key,
+ metadata.get("event_text", ""),
+ metadata.get("source_span", ""),
+ metadata.get("raw_text", ""),
+ ]
+ return " ".join(_clean_text(value) for value in values if _clean_text(value))
+
+
+def _audit_anchor_content_tokens(text: str) -> set[str]:
+ tokens = set()
+ for token in _tokenize(text):
+ norm = _normalize(token)
+ if not norm or norm in _AUDIT_ANCHOR_STOPWORDS:
+ continue
+ if len(norm) < 3 and not any("\u4e00" <= ch <= "\u9fff" for ch in norm):
+ continue
+ tokens.add(norm)
+ return tokens
+
+
+def _audit_anchor_phrase_bonus(query_tokens: set[str], hit_tokens: set[str]) -> tuple[float, List[str]]:
+ matched_phrases: List[str] = []
+ bonus = 0.0
+ for phrase_tokens in _AUDIT_ANCHOR_PHRASE_TOKEN_SETS:
+ if phrase_tokens <= query_tokens and phrase_tokens <= hit_tokens:
+ matched_phrases.append("+".join(sorted(phrase_tokens)))
+ bonus += 5.0 if phrase_tokens == {"spine", "book"} else 3.0
+ return bonus, matched_phrases
+
+
+def _copy_hit_with_audit_anchor_boost(hit: MemoryHit, *, score: float, reason: str, matched_tokens: Sequence[str]) -> MemoryHit:
+ metadata = dict(hit.metadata or {})
+ metadata.update(
+ {
+ "audit_anchor_protected": True,
+ "audit_anchor_reason": reason,
+ "audit_anchor_original_score": round(float(hit.score), 6),
+ "audit_anchor_score": round(float(score), 6),
+ "audit_anchor_matched_tokens": list(matched_tokens)[:20],
+ }
+ )
+ return MemoryHit(
+ memory_id=hit.memory_id,
+ category=hit.category,
+ value=hit.value,
+ relation=hit.relation,
+ anchors=list(hit.anchors),
+ score=max(float(hit.score), round(1.0 + float(score), 6)),
+ source_kind=hit.source_kind,
+ slot_key=hit.slot_key,
+ state=hit.state,
+ turn_index=int(hit.turn_index),
+ metadata=metadata,
+ )
+
+
+def _audit_anchor_protected_hits(
+ *,
+ query: str,
+ final_hits: Sequence[MemoryHit],
+ candidate_hits: Sequence[MemoryHit],
+ metadata: Mapping[str, Any],
+ top_k: int,
+) -> Dict[str, Any]:
+ if not _audit_anchor_query(query):
+ return {"enabled": False, "hits": list(final_hits), "promoted_hits": [], "metadata": {"audit_anchor_enabled": False}}
+ query_tokens = _audit_anchor_content_tokens(query)
+ if not query_tokens:
+ return {"enabled": True, "hits": list(final_hits), "promoted_hits": [], "metadata": {"audit_anchor_enabled": True, "audit_anchor_reason": "no_content_tokens"}}
+
+ explicit_turns = _audit_anchor_turn_numbers(query)
+ explicit_turn_window = set()
+ for number in explicit_turns:
+ explicit_turn_window.update({number - 1, number, number + 1})
+
+ symbolic_ids = list(dict.fromkeys(str(item) for item in dict(metadata or {}).get("symbolic_recall_event_ids", []) or []))
+ learned_ids = list(dict.fromkeys(str(item) for item in dict(metadata or {}).get("learned_recall_event_ids", []) or []))
+ selected_ids = set(str(item) for item in dict(metadata or {}).get("selected_event_ids", []) or [])
+ symbolic_rank = {event_id: index for index, event_id in enumerate(symbolic_ids, start=1)}
+ learned_rank = {event_id: index for index, event_id in enumerate(learned_ids, start=1)}
+
+ pool: Dict[str, MemoryHit] = {}
+ for hit in list(final_hits) + list(candidate_hits):
+ key = hit.memory_id or f"{_hit_event_id(hit)}::{hit.slot_key}::{hit.value[:80]}"
+ if key and key not in pool:
+ pool[key] = hit
+
+ scored: List[tuple[float, str, List[str], MemoryHit]] = []
+ for hit in pool.values():
+ event_id = _hit_event_id(hit)
+ hit_tokens = _audit_anchor_content_tokens(_audit_anchor_hit_text(hit))
+ matched = sorted(query_tokens & hit_tokens)
+ distinctive_matched = [token for token in matched if token not in _AUDIT_ANCHOR_GENERIC_MATCH_TOKENS]
+ phrase_bonus, matched_phrases = _audit_anchor_phrase_bonus(query_tokens, hit_tokens)
+ if not matched and int(hit.turn_index or 0) not in explicit_turn_window:
+ continue
+ score = 0.0
+ reason_parts: List[str] = []
+ if int(hit.turn_index or 0) in explicit_turn_window:
+ score += 8.0
+ reason_parts.append("explicit_turn_anchor")
+ if matched:
+ weighted_overlap = 0.0
+ for token in matched:
+ weighted_overlap += 1.8 if token not in _AUDIT_ANCHOR_GENERIC_MATCH_TOKENS else 0.35
+ score += min(8.0, weighted_overlap)
+ score += len(matched) / max(1.0, float(min(len(query_tokens), len(hit_tokens))))
+ reason_parts.append("lexical_anchor_overlap")
+ if phrase_bonus > 0:
+ score += phrase_bonus
+ reason_parts.append("distinctive_phrase_anchor")
+ if event_id in symbolic_rank:
+ score += max(0.0, 2.0 - (symbolic_rank[event_id] - 1) * 0.08)
+ reason_parts.append("symbolic_recall_anchor")
+ if event_id in learned_rank:
+ score += max(0.0, 1.0 - (learned_rank[event_id] - 1) * 0.02)
+ reason_parts.append("learned_recall_anchor")
+ if event_id in selected_ids:
+ score -= 0.25
+ # For non-numeric old-topic probes, require a real content overlap so generic
+ # "earlier" phrasing does not promote unrelated old memories.
+ if not explicit_turns and len(distinctive_matched) < 1 and phrase_bonus <= 0:
+ continue
+ if score > 0:
+ scored.append((score, event_id, sorted(set(matched + matched_phrases)), hit))
+
+ scored.sort(key=lambda item: (item[0], -abs(int(item[3].turn_index or 0))), reverse=True)
+ max_promoted = 2 if not explicit_turns else 3
+ promoted: List[MemoryHit] = []
+ promoted_event_ids: set[str] = set()
+ for score, event_id, matched, hit in scored:
+ if event_id in promoted_event_ids:
+ continue
+ if any((hit.memory_id and hit.memory_id == current.memory_id) for current in final_hits[: max(1, top_k)]):
+ continue
+ reason = "explicit_turn_anchor" if int(hit.turn_index or 0) in explicit_turn_window else "old_topic_anchor"
+ promoted.append(_copy_hit_with_audit_anchor_boost(hit, score=score, reason=reason, matched_tokens=matched))
+ promoted_event_ids.add(event_id)
+ if len(promoted) >= max_promoted:
+ break
+
+ if not promoted:
+ return {
+ "enabled": True,
+ "hits": list(final_hits),
+ "promoted_hits": [],
+ "metadata": {
+ "audit_anchor_enabled": True,
+ "audit_anchor_turn_numbers": explicit_turns,
+ "audit_anchor_query_tokens": sorted(query_tokens)[:50],
+ "audit_anchor_promoted_event_ids": [],
+ },
+ }
+
+ merged: List[MemoryHit] = []
+ seen_keys: set[str] = set()
+ for hit in promoted + list(final_hits):
+ key = hit.memory_id or f"{_hit_event_id(hit)}::{hit.slot_key}::{hit.value[:80]}"
+ if key in seen_keys:
+ continue
+ seen_keys.add(key)
+ merged.append(hit)
+ if len(merged) >= max(1, int(top_k)):
+ break
+
+ return {
+ "enabled": True,
+ "hits": merged,
+ "promoted_hits": promoted,
+ "metadata": {
+ "audit_anchor_enabled": True,
+ "audit_anchor_turn_numbers": explicit_turns,
+ "audit_anchor_query_tokens": sorted(query_tokens)[:50],
+ "audit_anchor_promoted_event_ids": [_hit_event_id(hit) for hit in promoted],
+ "audit_anchor_promoted_turns": [int(hit.turn_index) for hit in promoted],
+ "audit_anchor_promoted_hit_count": len(promoted),
+ },
+ }
+
+
+def _learnable_graph_hits(graph: SessionMemoryGraphV2) -> List[MemoryHit]:
+ learnable_hits: List[MemoryHit] = []
+ for record in graph.records_by_id.values():
+ state = _normalize(record.state)
+ metadata = dict(record.metadata or {})
+ is_source_grounded_evidence = (
+ state == "evidence"
+ and (
+ _clean_text(record.source_kind).startswith("public_dialog")
+ or _clean_text(metadata.get("content_variant", "")) in {"source_turn", "llm_semantic_write"}
+ or _clean_text(metadata.get("write_path", "")) == "llm_semantic_writer_gate"
+ )
+ )
+ if state not in {"active", "parallel_active"} and not is_source_grounded_evidence:
+ continue
+ if _clean_text(record.slot_key).startswith("noise."):
+ continue
+ learnable_hits.append(
+ MemoryHit(
+ memory_id=record.memory_id,
+ category=record.category,
+ value=record.value,
+ relation=record.relation,
+ anchors=list(record.anchor_concepts),
+ score=max(float(record.confidence), float(record.salience), 0.01),
+ source_kind=record.source_kind,
+ slot_key=record.slot_key,
+ state=record.state,
+ turn_index=int(record.turn_index),
+ metadata=metadata,
+ )
+ )
+ learnable_hits.sort(
+ key=lambda item: (
+ _is_public_dialog_hit(item),
+ int(item.turn_index),
+ float(item.score),
+ ),
+ reverse=True,
+ )
+ return learnable_hits
+
+
+def _parse_structured_records(
+ payload: Dict[str, Any] | None,
+ *,
+ turn_index: int,
+ profile: TMCRAProfile | None = None,
+) -> List[SessionMemoryRecordV2]:
+ profile = profile or TMCRAProfile()
+ records: List[SessionMemoryRecordV2] = []
+ structured_rows: List[tuple[str, Mapping[str, Any]]] = []
+ for raw in (payload or {}).get("replacement_memory_records", []) or []:
+ if isinstance(raw, Mapping):
+ structured_rows.append(("formal", raw))
+ for raw in (payload or {}).get("suspect_memory_records", []) or []:
+ if isinstance(raw, Mapping):
+ structured_rows.append(("suspect", raw))
+ for index, (buffer_state, raw) in enumerate(structured_rows):
+ if not isinstance(raw, dict):
+ continue
+ category = _clean_text(raw.get("category", "memory")) or "memory"
+ value = _clean_text(raw.get("value", ""))
+ if not value:
+ continue
+ anchors = _dedupe(raw.get("anchors", []) or [], max_items=8)
+ metadata = dict(raw.get("metadata", {}) or {})
+ slot_key = profile.stable_slot_key(
+ category=category,
+ value=value,
+ anchors=anchors,
+ slot_key=_clean_text(raw.get("slot_key", "")) or _clean_text(raw.get("slot", "")),
+ relation=_clean_text(raw.get("relation", "")),
+ metadata=metadata,
+ )
+ metadata = {
+ **metadata,
+ "memory_role": _clean_text(metadata.get("memory_role", "")) or "user",
+ "authority": _clean_text(metadata.get("authority", "")) or "source",
+ "canonical_slot_key": _clean_text(metadata.get("canonical_slot_key", "")) or slot_key,
+ "writeback_class": _clean_text(metadata.get("writeback_class", "")),
+ "origin_query": _clean_text(metadata.get("origin_query", "")),
+ "origin_answer_id": _clean_text(metadata.get("origin_answer_id", "")),
+ "support_memory_ids": _dedupe(metadata.get("support_memory_ids", []) or []),
+ "support_fact_refs": _dedupe(metadata.get("support_fact_refs", []) or []),
+ "support_path_refs": _dedupe(metadata.get("support_path_refs", []) or []),
+ "promotion_state": _clean_text(metadata.get("promotion_state", "")) or "none",
+ "memory_buffer_state": buffer_state,
+ }
+ memory_id = f"{slot_key}:{turn_index}:{index}"
+ default_source_kind = "suspect_memory" if buffer_state == "suspect" else "replacement_memory"
+ default_state = "suspect" if buffer_state == "suspect" else ("active" if bool(raw.get("active", True)) else "historical")
+ records.append(
+ SessionMemoryRecordV2(
+ memory_id=memory_id,
+ category=category,
+ slot_key=slot_key,
+ value=value,
+ relation=_clean_text(raw.get("relation", "")) or f"{category}_memory",
+ anchor_concepts=anchors,
+ evidence_anchors=anchors,
+ salience=float(raw.get("salience", 0.88 if category in {"goal", "constraint"} else 0.74) or 0.74),
+ confidence=float(raw.get("confidence", 0.82) or 0.82),
+ source_kind=_clean_text(raw.get("source_kind", "")) or default_source_kind,
+ turn_index=int(raw.get("turn_index", turn_index) or turn_index),
+ state=_clean_text(raw.get("state", "")) or default_state,
+ metadata=metadata,
+ )
+ )
+ return records
+
+
+_WRITE_MARKERS = (
+ "goal update",
+ "goal seed",
+ "constraint update",
+ "constraint overwrite",
+ "constraint seed",
+ "preference update",
+ "preference overwrite",
+ "preference seed",
+ "terminology:",
+ "term seed",
+ "term overwrite",
+ "stage update",
+ "stage overwrite",
+ "path fact",
+ "fact:",
+ "memory update",
+)
+
+
+_OVERWRITE_MARKERS = (
+ "overwrite",
+ "replace",
+ "supersede",
+ "覆盖",
+ "替换",
+ "改成",
+ "更新为",
+)
+
+
+_TOPIC_BUCKET_CUE_GROUPS: tuple[tuple[tuple[str, ...], tuple[str, ...]], ...] = (
+ (
+ (
+ "食物",
+ "饮品",
+ "甜点",
+ "早餐",
+ "餐饮",
+ "口味",
+ "酸辣",
+ "热食",
+ "过敏",
+ "芒果",
+ "配料",
+ "推荐",
+ "吃",
+ "food",
+ "breakfast",
+ "allergy",
+ "mango",
+ ),
+ ("餐饮偏好安全", "食物偏好", "过敏约束", "口味偏好"),
+ ),
+ (
+ (
+ "界面",
+ "首页",
+ "首屏",
+ "配色",
+ "营销",
+ "横幅",
+ "电商",
+ "网站",
+ "移动端",
+ "视觉",
+ "工具型",
+ "ui",
+ "homepage",
+ "ecommerce",
+ "mobile",
+ "design",
+ ),
+ ("界面产品设计", "电商页面", "视觉布局", "移动端体验"),
+ ),
+ (
+ (
+ "api",
+ "writer",
+ "延迟",
+ "算法",
+ "耗时",
+ "评估",
+ "指标",
+ "调用",
+ "服务",
+ "key",
+ "runtime",
+ "latency",
+ "metric",
+ ),
+ ("api评估运行", "算法服务", "调用指标", "writer延迟"),
+ ),
+)
+
+
+_TOPIC_BUCKET_STOPWORDS = {
+ "不要",
+ "不用",
+ "复述",
+ "原话",
+ "只说",
+ "关键",
+ "最关键",
+ "以后",
+ "需要",
+ "必须",
+ "可以",
+ "时候",
+ "这个",
+ "那个",
+ "现在",
+ "做",
+ "说",
+ "的",
+ "了",
+ "和",
+ "与",
+ "to",
+ "the",
+ "and",
+ "for",
+}
+
+
+def _topic_bucket_keywords(text: str, *, max_items: int = 18) -> List[str]:
+ normalized = _normalize(text)
+ tokens: List[str] = []
+ for cues, anchors in _TOPIC_BUCKET_CUE_GROUPS:
+ if any(cue and _normalize(cue) in normalized for cue in cues):
+ tokens.extend(anchors)
+ tokens.extend(cue for cue in cues if len(cue) >= 2)
+ tokens.extend(re.findall(r"[\u4e00-\u9fff]{2,8}", str(text or "")))
+ tokens.extend(_tokenize(text))
+ cleaned = []
+ for token in tokens:
+ value = _clean_text(token)
+ if not value:
+ continue
+ normalized_value = _normalize(value)
+ if normalized_value in _TOPIC_BUCKET_STOPWORDS:
+ continue
+ if len(value) < 2:
+ continue
+ cleaned.append(value)
+ return _dedupe(cleaned, max_items=max_items)
+
+
+def _topic_bucket_id_from_keywords(keywords: Sequence[str]) -> str:
+ basis = "|".join(_normalize(item) for item in keywords[:8] if _clean_text(item))
+ if not basis:
+ basis = "general"
+ return "topic-" + str(uuid.uuid5(uuid.NAMESPACE_URL, f"tmcra-topic-bucket:{basis}"))[:12]
+
+
+def _topic_bucket_label_from_keywords(keywords: Sequence[str]) -> str:
+ visible = [
+ _clean_text(item)
+ for item in keywords
+ if _clean_text(item) and not _normalize(item).startswith("topic:")
+ ]
+ return " / ".join(visible[:3]) if visible else "动态话题"
+
+
+def _topic_bucket_overlap_score(left_keywords: Sequence[str], right_keywords: Sequence[str]) -> float:
+ left = {_normalize(item) for item in left_keywords if _clean_text(item)}
+ right = {_normalize(item) for item in right_keywords if _clean_text(item)}
+ if not left or not right:
+ return 0.0
+ overlap = left & right
+ if not overlap:
+ return 0.0
+ return len(overlap) / max(1, min(len(left), len(right)))
+
+
+def _explicit_topic_bucket_from_payload(answer_payload: Dict[str, Any] | None) -> Dict[str, Any]:
+ metadata = dict((answer_payload or {}).get("metadata", {}) or {})
+ explicit = metadata.get("topic_bucket")
+ if not isinstance(explicit, Mapping):
+ return {}
+ keywords = _dedupe(explicit.get("topic_keywords", explicit.get("keywords", [])) or [], max_items=24)
+ label = _clean_text(explicit.get("topic_label", explicit.get("label", "")))
+ if not keywords and label:
+ keywords = _topic_bucket_keywords(label, max_items=12)
+ bucket_id = _clean_text(explicit.get("topic_bucket_id", explicit.get("id", "")))
+ if not bucket_id:
+ bucket_id = _topic_bucket_id_from_keywords(keywords or [label])
+ return {
+ "topic_bucket_id": bucket_id,
+ "topic_label": label or _topic_bucket_label_from_keywords(keywords),
+ "topic_keywords": keywords,
+ "topic_confidence": float(explicit.get("confidence", explicit.get("topic_confidence", 0.92)) or 0.92),
+ "topic_assignment_source": "explicit_payload",
+ }
+
+
+def _coerce_topic_bucket(metadata: Mapping[str, Any]) -> Dict[str, Any]:
+ bucket_id = _clean_text(metadata.get("topic_bucket_id", ""))
+ if not bucket_id:
+ nested = metadata.get("topic_bucket")
+ if isinstance(nested, Mapping):
+ bucket_id = _clean_text(nested.get("topic_bucket_id", nested.get("id", "")))
+ if not bucket_id:
+ return {}
+ keywords = _dedupe(metadata.get("topic_keywords", []) or [], max_items=32)
+ nested = metadata.get("topic_bucket")
+ if isinstance(nested, Mapping):
+ keywords = _dedupe([*keywords, *(nested.get("topic_keywords", nested.get("keywords", [])) or [])], max_items=32)
+ label = _clean_text(metadata.get("topic_label", ""))
+ if not label and isinstance(nested, Mapping):
+ label = _clean_text(nested.get("topic_label", nested.get("label", "")))
+ return {
+ "topic_bucket_id": bucket_id,
+ "topic_label": label or _topic_bucket_label_from_keywords(keywords),
+ "topic_keywords": keywords,
+ }
+
+
+def _collect_topic_buckets(graph: SessionMemoryGraphV2) -> Dict[str, Dict[str, Any]]:
+ buckets: Dict[str, Dict[str, Any]] = {}
+
+ def merge(bucket: Mapping[str, Any], *, record_id: str = "", turn_index: int = 0) -> None:
+ bucket_id = _clean_text(bucket.get("topic_bucket_id", bucket.get("id", "")))
+ if not bucket_id:
+ return
+ current = buckets.setdefault(
+ bucket_id,
+ {
+ "topic_bucket_id": bucket_id,
+ "topic_label": _clean_text(bucket.get("topic_label", bucket.get("label", ""))),
+ "topic_keywords": [],
+ "record_ids": [],
+ "last_turn_index": 0,
+ },
+ )
+ current["topic_label"] = current.get("topic_label") or _clean_text(bucket.get("topic_label", bucket.get("label", "")))
+ current["topic_keywords"] = _dedupe(
+ [*list(current.get("topic_keywords", []) or []), *(bucket.get("topic_keywords", bucket.get("keywords", [])) or [])],
+ max_items=32,
+ )
+ if record_id:
+ current["record_ids"] = _dedupe([*list(current.get("record_ids", []) or []), record_id], max_items=200)
+ current["last_turn_index"] = max(int(current.get("last_turn_index", 0) or 0), int(turn_index or 0))
+
+ for record in getattr(graph, "records_by_id", {}).values():
+ metadata = dict(record.metadata or {})
+ bucket = _coerce_topic_bucket(metadata)
+ if bucket:
+ merge(bucket, record_id=record.memory_id, turn_index=int(record.turn_index))
+ for turn in list(getattr(graph, "turn_log", []) or []):
+ metadata = dict(turn.get("metadata", {}) if isinstance(turn, Mapping) else getattr(turn, "metadata", {}) or {})
+ bucket = _coerce_topic_bucket(metadata)
+ if bucket:
+ merge(bucket, turn_index=int(turn.get("turn_index", 0) if isinstance(turn, Mapping) else getattr(turn, "turn_index", 0) or 0))
+ for bucket in buckets.values():
+ if not bucket.get("topic_label"):
+ bucket["topic_label"] = _topic_bucket_label_from_keywords(bucket.get("topic_keywords", []) or [])
+ return buckets
+
+
+def _assign_topic_bucket_for_text(
+ graph: SessionMemoryGraphV2,
+ text: str,
+ *,
+ answer_payload: Dict[str, Any] | None = None,
+ turn_index: int = 0,
+ create: bool = True,
+) -> Dict[str, Any]:
+ explicit = _explicit_topic_bucket_from_payload(answer_payload)
+ if explicit:
+ explicit["turn_index"] = int(turn_index or 0)
+ return explicit
+ keywords = _topic_bucket_keywords(text, max_items=24)
+ if not keywords:
+ keywords = _dedupe([_clean_text(text)[:40]], max_items=1)
+ buckets = _collect_topic_buckets(graph)
+ best_bucket: Dict[str, Any] | None = None
+ best_score = 0.0
+ for bucket in buckets.values():
+ score = _topic_bucket_overlap_score(keywords, bucket.get("topic_keywords", []) or [])
+ if score > best_score:
+ best_bucket = bucket
+ best_score = score
+ if best_bucket and best_score >= 0.22:
+ merged_keywords = _dedupe([*list(best_bucket.get("topic_keywords", []) or []), *keywords], max_items=32)
+ return {
+ "topic_bucket_id": best_bucket["topic_bucket_id"],
+ "topic_label": best_bucket.get("topic_label") or _topic_bucket_label_from_keywords(merged_keywords),
+ "topic_keywords": merged_keywords,
+ "topic_confidence": round(min(0.97, 0.68 + best_score * 0.24), 6),
+ "topic_assignment_source": "reused_by_overlap",
+ "topic_match_score": round(best_score, 6),
+ "turn_index": int(turn_index or 0),
+ }
+ bucket_id = _topic_bucket_id_from_keywords(keywords)
+ return {
+ "topic_bucket_id": bucket_id,
+ "topic_label": _topic_bucket_label_from_keywords(keywords),
+ "topic_keywords": keywords,
+ "topic_confidence": 0.72 if create else 0.58,
+ "topic_assignment_source": "created_by_dialog" if create else "query_probe",
+ "topic_match_score": round(best_score, 6),
+ "turn_index": int(turn_index or 0),
+ }
+
+
+def _apply_topic_bucket_to_records(records: List[SessionMemoryRecordV2], topic_bucket: Mapping[str, Any]) -> None:
+ if not records or not topic_bucket:
+ return
+ bucket_id = _clean_text(topic_bucket.get("topic_bucket_id", ""))
+ if not bucket_id:
+ return
+ label = _clean_text(topic_bucket.get("topic_label", "")) or "动态话题"
+ keywords = _dedupe(topic_bucket.get("topic_keywords", []) or [], max_items=32)
+ for record in records:
+ metadata = dict(record.metadata or {})
+ metadata.update(
+ {
+ "topic_bucket_id": bucket_id,
+ "topic_label": label,
+ "topic_keywords": keywords,
+ "topic_confidence": float(topic_bucket.get("topic_confidence", 0.72) or 0.72),
+ "topic_assignment_source": _clean_text(topic_bucket.get("topic_assignment_source", "")) or "created_by_dialog",
+ }
+ )
+ record.metadata = metadata
+ record.anchor_concepts = _dedupe([*list(record.anchor_concepts or []), f"topic:{label}", *keywords[:8]], max_items=24)
+ evidence_anchors = list(metadata.get("evidence_anchors", []) or [])
+ metadata["evidence_anchors"] = _dedupe([*evidence_anchors, f"topic:{label}", *keywords[:8]], max_items=24)
+
+
+def _last_topic_turn(graph: SessionMemoryGraphV2) -> Dict[str, Any]:
+ for turn in reversed(list(getattr(graph, "turn_log", []) or [])):
+ metadata = dict(turn.get("metadata", {}) if isinstance(turn, Mapping) else getattr(turn, "metadata", {}) or {})
+ bucket = _coerce_topic_bucket(metadata)
+ if bucket:
+ bucket["turn_index"] = int(turn.get("turn_index", 0) if isinstance(turn, Mapping) else getattr(turn, "turn_index", 0) or 0)
+ return bucket
+ return {}
+
+
+def _add_topic_bridge_edges(
+ graph: SessionMemoryGraphV2,
+ *,
+ previous_topic: Mapping[str, Any],
+ current_topic: Mapping[str, Any],
+ current_record_ids: Sequence[str],
+ turn_index: int,
+ evidence: str,
+) -> Dict[str, Any]:
+ previous_id = _clean_text(previous_topic.get("topic_bucket_id", ""))
+ current_id = _clean_text(current_topic.get("topic_bucket_id", ""))
+ if not previous_id or not current_id or previous_id == current_id or not current_record_ids:
+ return {"topic_bridge_edge_count": 0}
+ buckets = _collect_topic_buckets(graph)
+ previous_record_ids = list((buckets.get(previous_id, {}) or {}).get("record_ids", []) or [])[-4:]
+ if not previous_record_ids:
+ return {"topic_bridge_edge_count": 0}
+ edge_count = 0
+ evidence_text = _clean_text(evidence)[:240]
+ for source_id in previous_record_ids:
+ for target_id in list(current_record_ids)[:4]:
+ if source_id == target_id:
+ continue
+ edge = SessionMemoryEdgeV2(
+ edge_id=f"{source_id}->{target_id}:topic_bridge:{previous_id}->{current_id}",
+ source_memory_id=source_id,
+ target_memory_id=target_id,
+ edge_type="topic_bridge",
+ score=0.54,
+ model_score=0.0,
+ evidence_turn=int(turn_index or 0),
+ evidence=evidence_text,
+ metadata={
+ "from_topic_bucket_id": previous_id,
+ "to_topic_bucket_id": current_id,
+ "from_topic_label": _clean_text(previous_topic.get("topic_label", "")),
+ "to_topic_label": _clean_text(current_topic.get("topic_label", "")),
+ "bridge_reason": "adjacent_dialog_topic_transition",
+ },
+ )
+ graph._upsert_memory_edge(edge)
+ edge_count += 1
+ return {
+ "topic_bridge_edge_count": edge_count,
+ "topic_bridge_from": previous_id,
+ "topic_bridge_to": current_id,
+ }
+
+
+def _add_dialogue_tunnel_edges(
+ graph: SessionMemoryGraphV2,
+ *,
+ current_topic: Mapping[str, Any],
+ current_record_ids: Sequence[str],
+ turn_index: int,
+ evidence: str,
+) -> Dict[str, Any]:
+ current_id = _clean_text(current_topic.get("topic_bucket_id", ""))
+ if not current_id or not current_record_ids:
+ return {"dialogue_tunnel_edge_count": 0}
+ buckets = _collect_topic_buckets(graph)
+ edge_count = 0
+ evidence_text = _clean_text(evidence)[:240]
+ source_ids: List[tuple[str, str, str]] = []
+ for bucket_id, bucket in sorted(
+ buckets.items(),
+ key=lambda item: int(item[1].get("last_turn_index", 0) or 0),
+ reverse=True,
+ ):
+ if bucket_id == current_id:
+ continue
+ for source_id in list(bucket.get("record_ids", []) or [])[-2:]:
+ if source_id not in current_record_ids:
+ source_ids.append((bucket_id, _clean_text(bucket.get("topic_label", "")), source_id))
+ if len(source_ids) >= 6:
+ break
+ for source_bucket_id, source_label, source_id in source_ids[:6]:
+ for target_id in list(current_record_ids)[:2]:
+ if source_id == target_id:
+ continue
+ edge = SessionMemoryEdgeV2(
+ edge_id=f"{source_id}->{target_id}:dialogue_tunnel:{source_bucket_id}->{current_id}",
+ source_memory_id=source_id,
+ target_memory_id=target_id,
+ edge_type="dialogue_tunnel",
+ score=0.24,
+ model_score=0.0,
+ evidence_turn=int(turn_index or 0),
+ evidence=evidence_text,
+ metadata={
+ "from_topic_bucket_id": source_bucket_id,
+ "to_topic_bucket_id": current_id,
+ "from_topic_label": source_label,
+ "to_topic_label": _clean_text(current_topic.get("topic_label", "")),
+ "bridge_reason": "high_resistance_dialogue_level_tunnel",
+ },
+ )
+ graph._upsert_memory_edge(edge)
+ edge_count += 1
+ return {"dialogue_tunnel_edge_count": edge_count}
+
+
+def _topic_adjacent_bucket_ids(graph: SessionMemoryGraphV2, bucket_id: str) -> set[str]:
+ adjacent: set[str] = set()
+ if not bucket_id:
+ return adjacent
+ for edge in getattr(graph, "memory_edges", {}).values():
+ if _normalize(edge.edge_type) != "topic_bridge":
+ continue
+ metadata = dict(edge.metadata or {})
+ left = _clean_text(metadata.get("from_topic_bucket_id", ""))
+ right = _clean_text(metadata.get("to_topic_bucket_id", ""))
+ if left == bucket_id and right:
+ adjacent.add(right)
+ if right == bucket_id and left:
+ adjacent.add(left)
+ return adjacent
+
+
+def _dialogue_tunnel_bucket_ids(graph: SessionMemoryGraphV2, bucket_id: str) -> set[str]:
+ adjacent: set[str] = set()
+ if not bucket_id:
+ return adjacent
+ for edge in getattr(graph, "memory_edges", {}).values():
+ if _normalize(edge.edge_type) != "dialogue_tunnel":
+ continue
+ metadata = dict(edge.metadata or {})
+ left = _clean_text(metadata.get("from_topic_bucket_id", ""))
+ right = _clean_text(metadata.get("to_topic_bucket_id", ""))
+ if left == bucket_id and right:
+ adjacent.add(right)
+ if right == bucket_id and left:
+ adjacent.add(left)
+ return adjacent
+
+
+def _topic_bridge_requested(query: str) -> bool:
+ text = _normalize(query)
+ if not text:
+ return False
+ bridge_markers = (
+ "关联",
+ "联系",
+ "链条",
+ "脉络",
+ "延展",
+ "深入",
+ "对比",
+ "整合",
+ "整体",
+ "上下文",
+ "刚才",
+ "之前",
+ "上面",
+ "跨话题",
+ "隧穿",
+ "related",
+ "connect",
+ "compare",
+ "context",
+ "chain",
+ )
+ return any(marker in text for marker in bridge_markers)
+
+
+def _dialogue_tunnel_requested(query: str) -> bool:
+ text = _normalize(query)
+ if not text:
+ return False
+ markers = (
+ "跨话题",
+ "跨对话",
+ "不同话题",
+ "不同对话",
+ "历史对话",
+ "长期脉络",
+ "整体脉络",
+ "所有相关",
+ "全局",
+ "全局记忆",
+ "远一点",
+ "更深",
+ "深层关联",
+ "对话级隧穿",
+ "记忆隧穿",
+ "cross topic",
+ "cross-topic",
+ "cross dialogue",
+ "cross-session",
+ "global context",
+ "long range",
+ )
+ return any(marker in text for marker in markers)
+
+
+def _topic_bucket_record_to_hit(
+ record: SessionMemoryRecordV2,
+ *,
+ query_topic: Mapping[str, Any],
+ rank: int,
+ rescue_kind: str = "topic_bucket",
+) -> MemoryHit:
+ metadata = dict(record.metadata or {})
+ dialogue_tunnel = _normalize(rescue_kind) == "dialogue_tunnel"
+ metadata.update(
+ {
+ "topic_bucket_rescue": not dialogue_tunnel,
+ "dialogue_tunnel_rescue": dialogue_tunnel,
+ "topic_bucket_rescue_rank": int(rank),
+ "topic_bucket_query_id": _clean_text(query_topic.get("topic_bucket_id", "")),
+ "topic_bucket_query_label": _clean_text(query_topic.get("topic_label", "")),
+ "topic_bucket_same": not dialogue_tunnel,
+ "topic_bucket_bridge": False,
+ "topic_bucket_bridge_allowed": False,
+ "topic_bucket_dialogue_tunnel_allowed": dialogue_tunnel,
+ "topic_bucket_overlap": 0.0 if dialogue_tunnel else 1.0,
+ }
+ )
+ category = _normalize(record.category)
+ value_text = _normalize(record.value)
+ hardish = (
+ category == "constraint"
+ or _normalize(metadata.get("memory_type", "")) == "hard_constraint"
+ or _normalize(metadata.get("durability", "")) == "hard"
+ or _normalize(metadata.get("conflict_policy", "")) == "must_preserve"
+ or any(marker in value_text for marker in ("过敏", "必须", "避开", "禁止", "不能", "must", "avoid", "allergy"))
+ )
+ base_score = max(float(record.confidence), float(record.salience), 0.62)
+ if dialogue_tunnel:
+ if hardish:
+ base_score += 0.42
+ metadata.setdefault("memory_type", "hard_constraint")
+ metadata.setdefault("durability", "hard")
+ metadata.setdefault("conflict_policy", "must_preserve")
+ elif category == "preference":
+ base_score += 0.22
+ metadata.setdefault("memory_type", "durable_preference")
+ metadata.setdefault("durability", "long_term")
+ else:
+ base_score += 0.12
+ metadata["dialogue_tunnel_resistance"] = "high"
+ elif hardish:
+ base_score += 1.35
+ metadata.setdefault("memory_type", "hard_constraint")
+ metadata.setdefault("durability", "hard")
+ metadata.setdefault("conflict_policy", "must_preserve")
+ elif category == "preference":
+ base_score += 0.72
+ metadata.setdefault("memory_type", "durable_preference")
+ metadata.setdefault("durability", "long_term")
+ else:
+ base_score += 0.38
+ return MemoryHit(
+ memory_id=record.memory_id,
+ category=record.category,
+ value=record.value,
+ relation=record.relation,
+ anchors=list(record.anchor_concepts),
+ score=base_score,
+ source_kind=record.source_kind,
+ slot_key=record.slot_key,
+ state=record.state,
+ turn_index=int(record.turn_index),
+ metadata=metadata,
+ )
+
+
+def _profile_query_rescue_hits(
+ graph: SessionMemoryGraphV2,
+ query: str,
+ *,
+ top_k: int,
+) -> List[MemoryHit]:
+ query_raw_tokens = set(_path_utility_tokens(query))
+ query_tokens = _profile_query_expanded_tokens(query)
+ intent = infer_profile_query_intent(query)
+ if not bool(intent.get("enabled")):
+ return []
+ rescued: List[tuple[float, MemoryHit]] = []
+ for record in getattr(graph, "records_by_id", {}).values():
+ metadata = dict(record.metadata or {})
+ if record.state != "active":
+ continue
+ if not is_profile_layer_record(
+ category=record.category,
+ source_kind=record.source_kind,
+ semantic_slot=metadata.get("semantic_slot", ""),
+ metadata=metadata,
+ ):
+ continue
+ delta, reason = profile_query_score_delta(
+ query=query,
+ query_tokens=query_tokens,
+ category=record.category,
+ source_kind=record.source_kind,
+ semantic_slot=metadata.get("semantic_slot", ""),
+ value=record.value,
+ anchors=record.anchor_concepts,
+ metadata=metadata,
+ )
+ if delta <= 0:
+ continue
+ match_score, overlap_tokens, raw_overlap_tokens = _profile_hit_match_score(
+ query_raw_tokens,
+ query_tokens,
+ MemoryHit(
+ memory_id=record.memory_id,
+ category=record.category,
+ value=record.value,
+ relation=record.relation,
+ anchors=list(record.anchor_concepts),
+ score=max(float(record.confidence), float(record.salience), 0.01),
+ source_kind=record.source_kind,
+ slot_key=record.slot_key,
+ state=record.state,
+ turn_index=int(record.turn_index),
+ metadata=metadata,
+ ),
+ )
+ if match_score <= 0.0:
+ continue
+ if match_score < 0.34 and not raw_overlap_tokens:
+ continue
+ metadata.update(
+ {
+ "profile_query_rescue": True,
+ "profile_query_rescue_reason": reason or "profile_route",
+ "profile_query_match_score": round(match_score, 6),
+ "profile_query_overlap_tokens": list(overlap_tokens),
+ "profile_query_raw_overlap_tokens": list(raw_overlap_tokens),
+ "topic_bucket_profile_route_preserved": True,
+ "match_reason": ",".join(_dedupe([metadata.get("match_reason", ""), reason or "profile_route"], max_items=4)),
+ }
+ )
+ hit = MemoryHit(
+ memory_id=record.memory_id,
+ category=record.category,
+ value=record.value,
+ relation=record.relation,
+ anchors=list(record.anchor_concepts),
+ score=max(float(record.confidence), float(record.salience), 0.62) + float(delta) + float(match_score),
+ source_kind=record.source_kind,
+ slot_key=record.slot_key,
+ state=record.state,
+ turn_index=int(record.turn_index),
+ metadata=metadata,
+ )
+ rescued.append((match_score, hit))
+ rescued.sort(key=lambda item: (float(item[0]), float(item[1].score), int(item[1].turn_index)), reverse=True)
+ return [
+ hit
+ for _, hit in rescued[: max(1, min(24, int(top_k or 1) * 3))]
+ ]
+
+
+def _memory_hit_from_record(record: SessionMemoryRecordV2, *, score: float | None = None, metadata: Mapping[str, Any] | None = None) -> MemoryHit:
+ record_metadata = {**dict(record.metadata or {}), **dict(metadata or {})}
+ return MemoryHit(
+ memory_id=record.memory_id,
+ category=record.category,
+ value=record.value,
+ relation=record.relation,
+ anchors=list(record.anchor_concepts),
+ score=max(float(record.confidence), float(record.salience), 0.01) if score is None else float(score),
+ source_kind=record.source_kind,
+ slot_key=record.slot_key,
+ state=record.state,
+ turn_index=int(record.turn_index),
+ metadata=record_metadata,
+ )
+
+
+_FACET_NUMERIC_QUERY_TOKENS = {
+ "amount",
+ "count",
+ "counts",
+ "duration",
+ "durations",
+ "many",
+ "much",
+ "number",
+ "quantity",
+ "sum",
+ "total",
+ "totals",
+ "weeks",
+ "week",
+ "hours",
+ "hour",
+ "dollars",
+ "dollar",
+ "tenants",
+ "tickets",
+}
+_FACET_TEMPORAL_QUERY_TOKENS = {
+ "after",
+ "before",
+ "date",
+ "deadline",
+ "end",
+ "finish",
+ "finished",
+ "start",
+ "started",
+ "time",
+ "when",
+}
+
+
+def _facet_query_pack_hits(
+ graph: SessionMemoryGraphV2,
+ query: str,
+ final_hits: Sequence[MemoryHit],
+ *,
+ top_k: int,
+) -> Dict[str, Any]:
+ query_tokens = set(_path_utility_tokens(query))
+ if not query_tokens:
+ return {"hits": list(final_hits), "metadata": {"facet_query_pack_enabled": False, "facet_query_pack_reason": "empty_query_tokens"}}
+ numeric_query = bool(query_tokens & _FACET_NUMERIC_QUERY_TOKENS)
+ temporal_query = bool(query_tokens & _FACET_TEMPORAL_QUERY_TOKENS)
+ if not numeric_query and not temporal_query and not any("facet" in _normalize(token) for token in query_tokens):
+ return {"hits": list(final_hits), "metadata": {"facet_query_pack_enabled": False, "facet_query_pack_reason": "no_facet_intent"}}
+
+ candidate_rows: List[tuple[float, SessionMemoryRecordV2, SessionMemoryRecordV2 | None, Dict[str, Any]]] = []
+ for record in getattr(graph, "records_by_id", {}).values():
+ metadata = dict(record.metadata or {})
+ if _normalize(metadata.get("content_variant", "")) != "event_facet_write":
+ continue
+ if record.state not in {"active", "parallel_active", "evidence"}:
+ continue
+ facet_type = _normalize(metadata.get("facet_type", ""))
+ parent_slot_key = _clean_text(metadata.get("facet_parent_slot_key", ""))
+ parent = next(
+ (
+ candidate
+ for candidate in getattr(graph, "records_by_id", {}).values()
+ if _clean_text(candidate.slot_key).lower() == parent_slot_key.lower()
+ ),
+ None,
+ )
+ parent_text = " ".join(
+ [
+ _clean_text(parent.value if parent else ""),
+ " ".join(parent.anchor_concepts if parent else []),
+ _clean_text(dict(parent.metadata or {}).get("source_span", "") if parent else ""),
+ ]
+ )
+ facet_text = " ".join(
+ [
+ record.value,
+ " ".join(record.anchor_concepts or []),
+ _clean_text(metadata.get("facet_type", "")),
+ _clean_text(metadata.get("facet_role", "")),
+ _clean_text(metadata.get("facet_source_span", "")),
+ parent_text,
+ ]
+ )
+ facet_tokens = set(_path_utility_tokens(facet_text))
+ parent_tokens = set(_path_utility_tokens(parent_text))
+ overlap_tokens = query_tokens & facet_tokens
+ parent_overlap_tokens = query_tokens & parent_tokens
+ unit_overlap = bool(query_tokens & set(_path_utility_tokens(record.value)))
+ score = 0.0
+ if overlap_tokens:
+ score += min(0.72, len(overlap_tokens) / max(1.0, len(query_tokens)) * 1.2)
+ if parent_overlap_tokens:
+ score += min(0.72, len(parent_overlap_tokens) / max(1.0, len(query_tokens)) * 1.35)
+ if numeric_query and facet_type == "numeric":
+ score += 0.42
+ if unit_overlap:
+ score += 0.42
+ if temporal_query and facet_type == "temporal":
+ score += 0.32
+ if facet_type == "entity" and parent_overlap_tokens:
+ score += 0.26
+ if parent is not None and _normalize(dict(parent.metadata or {}).get("content_variant", "")) == "llm_semantic_write":
+ score += 0.08
+ if score < 0.58:
+ continue
+ candidate_rows.append(
+ (
+ round(min(2.75, 1.18 + score), 6),
+ record,
+ parent,
+ {
+ "facet_query_pack_overlap_tokens": sorted(overlap_tokens)[:12],
+ "facet_query_pack_parent_overlap_tokens": sorted(parent_overlap_tokens)[:12],
+ "facet_query_pack_unit_overlap": bool(unit_overlap),
+ "facet_query_pack_score": round(score, 6),
+ },
+ )
+ )
+
+ if not candidate_rows:
+ return {
+ "hits": list(final_hits),
+ "metadata": {
+ "facet_query_pack_enabled": True,
+ "facet_query_pack_inserted_hit_count": 0,
+ "facet_query_pack_candidate_count": 0,
+ },
+ }
+
+ candidate_rows.sort(key=lambda item: (float(item[0]), int(item[1].turn_index)), reverse=True)
+ selected = candidate_rows[: max(4, min(18, int(top_k or 1) * 2))]
+ packed_hits: List[MemoryHit] = []
+ for score, facet_record, parent, extra_metadata in selected:
+ facet_metadata = {
+ **extra_metadata,
+ "facet_query_pack": True,
+ "evidence_snippet_role": "facet_query_attribute",
+ }
+ packed_hits.append(_memory_hit_from_record(facet_record, score=score, metadata=facet_metadata))
+ if parent is not None:
+ packed_hits.append(
+ _memory_hit_from_record(
+ parent,
+ score=max(1.12, score - 0.04),
+ metadata={
+ **extra_metadata,
+ "facet_query_pack": True,
+ "evidence_snippet_role": "facet_parent_event",
+ "facet_query_pack_child_id": facet_record.memory_id,
+ },
+ )
+ )
+
+ merged: List[MemoryHit] = []
+ seen_ids: set[str] = set()
+ for hit in [*packed_hits, *list(final_hits)]:
+ if hit.memory_id and hit.memory_id in seen_ids:
+ continue
+ if hit.memory_id:
+ seen_ids.add(hit.memory_id)
+ merged.append(hit)
+ return {
+ "hits": merged,
+ "metadata": {
+ "facet_query_pack_enabled": True,
+ "facet_query_pack_candidate_count": len(candidate_rows),
+ "facet_query_pack_inserted_hit_count": len(packed_hits),
+ "facet_query_pack_numeric_query": bool(numeric_query),
+ "facet_query_pack_temporal_query": bool(temporal_query),
+ },
+ }
+
+
+_UNIT_COVERAGE_COUNT_TOKENS = {
+ "amount",
+ "amounts",
+ "count",
+ "counts",
+ "cost",
+ "costs",
+ "dollar",
+ "dollars",
+ "how",
+ "many",
+ "much",
+ "number",
+ "minimum",
+ "maximum",
+ "paid",
+ "percent",
+ "percentage",
+ "price",
+ "prices",
+ "sale",
+ "sales",
+ "sell",
+ "sold",
+ "total",
+ "totals",
+ "sum",
+ "value",
+ "valued",
+ "values",
+ "worth",
+ "items",
+ "projects",
+ "events",
+ "things",
+}
+_UNIT_COVERAGE_TEMPORAL_TOKENS = {
+ "ago",
+ "after",
+ "before",
+ "between",
+ "consecutive",
+ "date",
+ "days",
+ "day",
+ "first",
+ "last",
+ "months",
+ "month",
+ "order",
+ "passed",
+ "since",
+ "weeks",
+ "week",
+}
+_MULTI_UNIT_CHAIN_TEMPORAL_COMPARISON_TOKENS = {
+ "after",
+ "before",
+ "between",
+ "consecutive",
+ "earlier",
+ "first",
+ "later",
+ "last",
+ "order",
+ "since",
+}
+_UNIT_COVERAGE_QUERY_DROP_TOKENS = {
+ "date",
+ "fri",
+ "friday",
+ "mon",
+ "monday",
+ "question",
+ "sat",
+ "saturday",
+ "sun",
+ "sunday",
+ "thu",
+ "thursday",
+ "tue",
+ "tuesday",
+ "wed",
+ "wednesday",
+}
+_MULTI_UNIT_CHAIN_DISABLED_MODES = {"", "off", "disabled", "none", "false", "0"}
+_MULTI_UNIT_CHAIN_COUNT_TOKENS = {
+ *_UNIT_COVERAGE_COUNT_TOKENS,
+ "which",
+ "each",
+ "all",
+ "both",
+}
+_MULTI_UNIT_CHAIN_FOCUS_DROP_TOKENS = {
+ *_MULTI_UNIT_CHAIN_COUNT_TOKENS,
+ *_UNIT_COVERAGE_TEMPORAL_TOKENS,
+ "i",
+ "me",
+ "my",
+ "mine",
+ "am",
+ "is",
+ "are",
+ "was",
+ "were",
+ "need",
+ "needs",
+ "needed",
+ "currently",
+ "did",
+ "does",
+ "fri",
+ "friday",
+ "mon",
+ "monday",
+ "or",
+ "question",
+ "sat",
+ "saturday",
+ "sun",
+ "sunday",
+ "thu",
+ "thursday",
+ "tue",
+ "tuesday",
+ "wed",
+ "wednesday",
+ "what",
+ "which",
+ "who",
+}
+_MULTI_UNIT_CHAIN_FACET_TYPES = {"action", "entity", "numeric", "role", "evidence_role", "state", "temporal"}
+_MULTI_UNIT_CHAIN_UNIT_KINDS = {
+ "action_unit",
+ "target_entity",
+ "numeric_quantity",
+ "leadership_role",
+ "participation_role",
+ "state_status",
+ "temporal_anchor",
+ "evidence_role",
+ "profile_shadow_unit",
+}
+_MULTI_UNIT_CHAIN_NUMERIC_VALUE_TOKENS = {
+ "amount",
+ "appraisal",
+ "appraised",
+ "cost",
+ "costs",
+ "dollar",
+ "dollars",
+ "minimum",
+ "paid",
+ "price",
+ "prices",
+ "sale",
+ "sell",
+ "sold",
+ "total",
+ "value",
+ "valued",
+ "values",
+ "worth",
+}
+
+
+def _multi_unit_chain_numeric_signal(
+ unit_kind: str,
+ facet_type: str,
+ text: str,
+ tokens: set[str],
+) -> float:
+ compact_text = _clean_text(text)
+ date_like_only = bool(
+ re.fullmatch(
+ r"(?:\d{4}[/-]\d{1,2}(?:[/-]\d{1,2})?|\d{1,2}[/-]\d{1,2}(?:[/-]\d{2,4})?|\d{4}/\d{2})",
+ compact_text,
+ )
+ )
+ if date_like_only:
+ return 0.0
+ signal = 0.0
+ if unit_kind == "numeric_quantity" or facet_type == "numeric":
+ signal += 0.55
+ if tokens & _MULTI_UNIT_CHAIN_NUMERIC_VALUE_TOKENS:
+ signal += 0.28
+ if re.search(r"(?:[$€£¥]\s*\d|\b\d+(?:,\d{3})*(?:\.\d+)?\s*(?:dollars?|usd|bucks?|yuan|rmb)\b)", text, flags=re.IGNORECASE):
+ signal += 0.72
+ elif re.search(r"\b\d+(?:,\d{3})*(?:\.\d+)?\s*(?:comments?|items?|pieces?|kits?|projects?|doctors?|weddings?|hours?|weeks?|days?|months?|years?|miles?)\b", text, flags=re.IGNORECASE):
+ signal += 0.42
+ return min(1.1, signal)
+
+
+def _multi_unit_chain_date_like_numeric(text: str) -> bool:
+ compact_text = _clean_text(text)
+ return bool(
+ re.fullmatch(
+ r"(?:\d{4}[/-]\d{1,2}(?:[/-]\d{1,2})?|\d{1,2}[/-]\d{1,2}(?:[/-]\d{2,4})?|\d{4}/\d{2})",
+ compact_text,
+ )
+ )
+
+
+def _multi_unit_chain_temporal_anchor_signal(text: str, tokens: set[str]) -> float:
+ signal = 0.0
+ if tokens & _UNIT_COVERAGE_TEMPORAL_TOKENS:
+ signal += 0.22
+ if re.search(
+ r"\b(?:about|around|roughly|few|several|couple|last|previous|earlier)\s+"
+ r"(?:a\s+)?(?:day|days|week|weeks|month|months|year|years)\s+ago\b|"
+ r"\b(?:yesterday|today|tomorrow|last\s+week|last\s+month|a\s+few\s+months\s+ago)\b",
+ text,
+ flags=re.IGNORECASE,
+ ):
+ signal += 0.72
+ return min(1.0, signal)
+
+
+_PROFILE_SHADOW_EVENTLIKE_SLOT_HINTS = {
+ "action",
+ "appointment",
+ "constraint",
+ "deadline",
+ "exchange",
+ "obligation",
+ "pickup",
+ "plan",
+ "preference",
+ "return",
+ "status",
+ "task",
+}
+_PROFILE_SHADOW_EVENTLIKE_TEXT_HINTS = {
+ "bought",
+ "buy",
+ "completed",
+ "did",
+ "exchange",
+ "exchanged",
+ "finish",
+ "finished",
+ "got",
+ "have to",
+ "need",
+ "needed",
+ "needs",
+ "paid",
+ "pick",
+ "picked",
+ "return",
+ "returned",
+ "should",
+ "still",
+ "took",
+ "went",
+}
+
+
+def _profile_shadow_eventlike_record(record: SessionMemoryRecordV2, metadata: Mapping[str, Any]) -> bool:
+ if _normalize(metadata.get("content_variant", "")) != "profile_shadow_from_writer":
+ return False
+ if record.state not in {"active", "parallel_active", "evidence"}:
+ return False
+ slot_text = " ".join(
+ [
+ _clean_text(record.category),
+ _clean_text(record.relation),
+ _clean_text(record.slot_key),
+ _clean_text(metadata.get("semantic_slot", "")),
+ _clean_text(metadata.get("profile_type", "")),
+ ]
+ ).lower()
+ value_text = " ".join(
+ [
+ _clean_text(record.value),
+ _clean_text(metadata.get("source_span", "")),
+ _clean_text(metadata.get("raw_text", "")),
+ ]
+ ).lower()
+ return bool(
+ any(hint in slot_text for hint in _PROFILE_SHADOW_EVENTLIKE_SLOT_HINTS)
+ or any(hint in value_text for hint in _PROFILE_SHADOW_EVENTLIKE_TEXT_HINTS)
+ )
+
+
+def _profile_shadow_unit_text(record: SessionMemoryRecordV2, metadata: Mapping[str, Any]) -> str:
+ return " ".join(
+ [
+ _clean_text(record.value),
+ _clean_text(record.category),
+ _clean_text(record.relation),
+ _clean_text(record.slot_key),
+ _clean_text(metadata.get("semantic_slot", "")),
+ _clean_text(metadata.get("profile_type", "")),
+ _clean_text(metadata.get("profile_domain", "")),
+ _clean_text(metadata.get("source_span", "")),
+ _clean_text(metadata.get("raw_text", "")),
+ ]
+ )
+
+
+def _unit_coverage_pack_hits(
+ graph: SessionMemoryGraphV2,
+ query: str,
+ final_hits: Sequence[MemoryHit],
+ *,
+ top_k: int,
+) -> Dict[str, Any]:
+ raw_query_tokens = set(_path_utility_tokens(query))
+ query_tokens = {token for token in raw_query_tokens if token not in _UNIT_COVERAGE_QUERY_DROP_TOKENS}
+ if not query_tokens:
+ return {"hits": list(final_hits), "metadata": {"unit_coverage_pack_enabled": False, "unit_coverage_reason": "empty_query"}}
+ count_intent = bool(query_tokens & _UNIT_COVERAGE_COUNT_TOKENS)
+ percentage_intent = bool(query_tokens & {"percent", "percentage"})
+ temporal_intent = bool(raw_query_tokens & _MULTI_UNIT_CHAIN_TEMPORAL_COMPARISON_TOKENS)
+ direct_unit_intent = not count_intent and not temporal_intent and len(query_tokens) >= 2
+ if not count_intent and not temporal_intent and not direct_unit_intent:
+ return {
+ "hits": list(final_hits),
+ "metadata": {"unit_coverage_pack_enabled": False, "unit_coverage_reason": "no_unit_intent"},
+ }
+
+ records = list(getattr(graph, "records_by_id", {}).values())
+ parent_by_slot = {_clean_text(record.slot_key).lower(): record for record in records}
+ candidates: List[tuple[float, SessionMemoryRecordV2, SessionMemoryRecordV2 | None, Dict[str, Any]]] = []
+ for record in records:
+ metadata = dict(record.metadata or {})
+ if record.state not in {"active", "parallel_active", "evidence"}:
+ continue
+ profile_shadow_unit = _profile_shadow_eventlike_record(record, metadata)
+ if profile_shadow_unit:
+ unit_kind = "profile_shadow_unit"
+ facet_type = "action"
+ parent_slot = _clean_text(record.slot_key).lower()
+ parent = None
+ parent_text = ""
+ unit_text = _profile_shadow_unit_text(record, metadata)
+ else:
+ if _normalize(metadata.get("content_variant", "")) != "event_facet_write":
+ continue
+ if _normalize(metadata.get("facet_layer_version", "")) not in {"event_unit_v1", "event_facet_v1"}:
+ continue
+ unit_kind = _normalize(metadata.get("unit_kind", ""))
+ facet_type = _normalize(metadata.get("facet_type", ""))
+ parent_slot = _clean_text(metadata.get("facet_parent_slot_key", "")).lower()
+ parent = parent_by_slot.get(parent_slot)
+ parent_text = " ".join(
+ [
+ _clean_text(parent.value if parent else ""),
+ _clean_text(dict(parent.metadata or {}).get("source_span", "") if parent else ""),
+ " ".join(parent.anchor_concepts if parent else []),
+ ]
+ )
+ unit_text = " ".join(
+ [
+ _clean_text(record.value),
+ _clean_text(metadata.get("facet_role", "")),
+ _clean_text(metadata.get("unit_kind", "")),
+ _clean_text(metadata.get("action", "")),
+ _clean_text(metadata.get("target", "")),
+ _clean_text(metadata.get("quantity", "")),
+ _clean_text(metadata.get("unit", "")),
+ _clean_text(metadata.get("normalized_time", "")),
+ _clean_text(metadata.get("status", "")),
+ _clean_text(metadata.get("facet_source_span", "")),
+ parent_text,
+ ]
+ )
+ unit_tokens = set(_path_utility_tokens(unit_text))
+ semantic_event_unit = _normalize(record.memory_id).startswith("tmcra.event.") or _normalize(parent_slot).startswith("tmcra.event.")
+ overlap = query_tokens & unit_tokens
+ score = 0.0
+ if overlap:
+ score += min(0.92, len(overlap) / max(1.0, len(query_tokens)) * 1.55)
+ if count_intent and facet_type in {"action", "entity", "numeric", "role", "evidence_role"}:
+ score += 0.42
+ if temporal_intent and facet_type in {"temporal", "action", "role", "entity"}:
+ score += 0.38
+ if direct_unit_intent and facet_type in {"action", "entity", "state", "numeric"}:
+ score += 0.30
+ if direct_unit_intent and len(overlap) >= 2:
+ score += 0.36
+ if unit_kind in {"action_unit", "target_entity", "leadership_role", "participation_role"}:
+ score += 0.16
+ if unit_kind in {"numeric_quantity", "temporal_anchor", "state_status"}:
+ score += 0.12
+ if profile_shadow_unit:
+ score += 0.28
+ if count_intent and facet_type in {"action", "entity"}:
+ score += 0.24
+ semantic_event_priority = bool(semantic_event_unit and (count_intent or percentage_intent))
+ if semantic_event_priority:
+ score += 0.46
+ if percentage_intent and (
+ unit_kind == "numeric_quantity"
+ or facet_type == "numeric"
+ or re.search(r"\b\d+(?:\.\d+)?\s*%|\b\d+(?:\.\d+)?\s*percent\b", unit_text, flags=re.IGNORECASE)
+ ):
+ score += 0.72
+ if parent is not None:
+ score += 0.08
+ if score < 0.55:
+ continue
+ candidates.append(
+ (
+ round(min(3.2, 1.28 + score), 6),
+ record,
+ parent,
+ {
+ "unit_coverage_overlap_tokens": sorted(overlap)[:14],
+ "unit_coverage_score": round(score, 6),
+ "unit_kind": unit_kind,
+ "facet_type": facet_type,
+ "unit_coverage_count_intent": bool(count_intent),
+ "unit_coverage_percentage_intent": bool(percentage_intent),
+ "unit_coverage_temporal_intent": bool(temporal_intent),
+ "unit_coverage_direct_unit_intent": bool(direct_unit_intent),
+ "unit_coverage_profile_shadow_unit": bool(profile_shadow_unit),
+ "unit_coverage_semantic_event_unit": bool(semantic_event_unit),
+ "unit_coverage_semantic_event_priority": bool(semantic_event_priority),
+ },
+ )
+ )
+
+ if not candidates:
+ return {
+ "hits": list(final_hits),
+ "metadata": {
+ "unit_coverage_pack_enabled": True,
+ "unit_coverage_candidate_count": 0,
+ "unit_coverage_inserted_hit_count": 0,
+ "unit_coverage_count_intent": bool(count_intent),
+ "unit_coverage_percentage_intent": bool(percentage_intent),
+ "unit_coverage_temporal_intent": bool(temporal_intent),
+ "unit_coverage_direct_unit_intent": bool(direct_unit_intent),
+ },
+ }
+ candidates.sort(
+ key=lambda item: (
+ 1 if bool(item[3].get("unit_coverage_semantic_event_priority", False)) else 0,
+ 1
+ if bool(item[3].get("unit_coverage_percentage_intent"))
+ and (
+ _normalize(item[3].get("unit_kind", "")) == "numeric_quantity"
+ or _normalize(item[3].get("facet_type", "")) == "numeric"
+ )
+ else 0,
+ float(item[0]),
+ int(item[1].turn_index),
+ ),
+ reverse=True,
+ )
+ selected: List[tuple[float, SessionMemoryRecordV2, SessionMemoryRecordV2 | None, Dict[str, Any]]] = []
+ seen_unit_values: set[str] = set()
+ try:
+ max_selected_units = max(1, int(os.getenv("TMCRA_UNIT_COVERAGE_PACK_MAX_UNITS", "6") or 6))
+ except (TypeError, ValueError):
+ max_selected_units = 6
+ for item in candidates:
+ _, record, parent, metadata = item
+ key = "|".join(
+ [
+ _normalize(metadata.get("unit_kind", "")),
+ _normalize(record.value),
+ _normalize(dict(record.metadata or {}).get("facet_parent_slot_key", "")),
+ ]
+ )
+ if key in seen_unit_values:
+ continue
+ seen_unit_values.add(key)
+ selected.append(item)
+ if len(selected) >= max_selected_units:
+ break
+ if count_intent:
+ try:
+ max_profile_shadow_units = max(0, int(os.getenv("TMCRA_UNIT_COVERAGE_PROFILE_SHADOW_MAX_UNITS", "2") or 2))
+ except (TypeError, ValueError):
+ max_profile_shadow_units = 2
+ selected_ids = {item[1].memory_id for item in selected}
+ added_profile_shadow_units = 0
+ for item in candidates:
+ _, record, _parent, metadata = item
+ if added_profile_shadow_units >= max_profile_shadow_units:
+ break
+ if record.memory_id in selected_ids:
+ continue
+ if not bool(metadata.get("unit_coverage_profile_shadow_unit", False)):
+ continue
+ selected.append(item)
+ selected_ids.add(record.memory_id)
+ added_profile_shadow_units += 1
+ packed_hits: List[MemoryHit] = []
+ for score, unit_record, parent, extra in selected:
+ packed_hits.append(
+ _memory_hit_from_record(
+ unit_record,
+ score=score,
+ metadata={
+ **extra,
+ "unit_coverage_pack": True,
+ "evidence_snippet_role": "unit_coverage_evidence_unit",
+ },
+ )
+ )
+ if parent is not None:
+ unit_metadata = dict(unit_record.metadata or {})
+ parent_hit = _memory_hit_from_record(
+ parent,
+ score=max(1.12, score - 0.06),
+ metadata={
+ **extra,
+ "unit_coverage_pack": True,
+ "evidence_snippet_role": "unit_coverage_parent_event",
+ "unit_coverage_parent_memory_id": parent.memory_id,
+ "unit_coverage_child_id": unit_record.memory_id,
+ "unit_coverage_child_value": unit_record.value,
+ "unit_coverage_child_source_span": _clean_text(
+ unit_metadata.get("facet_source_span", "") or unit_metadata.get("source_span", "")
+ ),
+ },
+ )
+ parent_hit.memory_id = f"{parent.memory_id}#unit_parent:{unit_record.memory_id}"
+ packed_hits.append(parent_hit)
+ merged: List[MemoryHit] = []
+ seen_ids: set[str] = set()
+ try:
+ insertion_index = max(0, min(len(final_hits), int(os.getenv("TMCRA_UNIT_COVERAGE_PACK_INSERT_AFTER", "2") or 2)))
+ except (TypeError, ValueError):
+ insertion_index = min(len(final_hits), 2)
+ ordered_hits = [*list(final_hits[:insertion_index]), *packed_hits, *list(final_hits[insertion_index:])]
+ for hit in ordered_hits:
+ if hit.memory_id and hit.memory_id in seen_ids:
+ continue
+ if hit.memory_id:
+ seen_ids.add(hit.memory_id)
+ merged.append(hit)
+ return {
+ "hits": merged,
+ "metadata": {
+ "unit_coverage_pack_enabled": True,
+ "unit_coverage_candidate_count": len(candidates),
+ "unit_coverage_selected_unit_count": len(selected),
+ "unit_coverage_inserted_hit_count": len(packed_hits),
+ "unit_coverage_count_intent": bool(count_intent),
+ "unit_coverage_percentage_intent": bool(percentage_intent),
+ "unit_coverage_temporal_intent": bool(temporal_intent),
+ "unit_coverage_direct_unit_intent": bool(direct_unit_intent),
+ },
+ }
+
+
+def _multi_unit_chain_focus_tokens(query: str) -> set[str]:
+ tokens = set(_path_utility_tokens(query))
+ return {
+ normalized
+ for token in tokens
+ for normalized in [_multi_unit_chain_normalize_token(token)]
+ if normalized and normalized not in _MULTI_UNIT_CHAIN_FOCUS_DROP_TOKENS and len(normalized) > 2
+ }
+
+
+def _multi_unit_chain_normalize_token(token: str) -> str:
+ raw = _normalize(token)
+ if not raw:
+ return ""
+ aliases = {
+ "bought": "buy",
+ "buying": "buy",
+ "purchased": "buy",
+ "purchasing": "buy",
+ "worked": "work",
+ "working": "work",
+ "started": "start",
+ "starting": "start",
+ "finished": "finish",
+ "finishing": "finish",
+ "completed": "complete",
+ "completing": "complete",
+ "returned": "return",
+ "returning": "return",
+ "met": "meet",
+ "meeting": "meet",
+ "picked": "pick",
+ "picking": "pick",
+ "led": "lead",
+ "leading": "lead",
+ "kits": "kit",
+ "items": "item",
+ "projects": "project",
+ "events": "event",
+ "clothes": "clothing",
+ }
+ if raw in aliases:
+ return aliases[raw]
+ if len(raw) > 4 and raw.endswith("ies"):
+ return raw[:-3] + "y"
+ if len(raw) > 4 and raw.endswith("ing"):
+ stem = raw[:-3]
+ if len(stem) > 3 and stem[-1] == stem[-2]:
+ stem = stem[:-1]
+ return stem
+ if len(raw) > 3 and raw.endswith("ed"):
+ stem = raw[:-2]
+ if len(stem) > 3 and stem[-1] == stem[-2]:
+ stem = stem[:-1]
+ return stem
+ if len(raw) > 3 and raw.endswith("s") and not raw.endswith("ss"):
+ return raw[:-1]
+ return raw
+
+
+def _multi_unit_chain_normalized_tokens(text: str) -> set[str]:
+ return {
+ normalized
+ for token in _path_utility_tokens(text)
+ for normalized in [_multi_unit_chain_normalize_token(token)]
+ if normalized
+ }
+
+
+def _multi_unit_chain_hit_text(record: SessionMemoryRecordV2, parent: SessionMemoryRecordV2 | None) -> str:
+ metadata = dict(record.metadata or {})
+ parent_metadata = dict(parent.metadata or {}) if parent is not None else {}
+ return " ".join(
+ [
+ _clean_text(record.value),
+ _clean_text(record.category),
+ _clean_text(record.relation),
+ _clean_text(metadata.get("facet_type", "")),
+ _clean_text(metadata.get("unit_kind", "")),
+ _clean_text(metadata.get("facet_role", "")),
+ _clean_text(metadata.get("facet_value", "")),
+ _clean_text(metadata.get("facet_source_span", "")),
+ _clean_text(metadata.get("action", "")),
+ _clean_text(metadata.get("target", "")),
+ _clean_text(metadata.get("quantity", "")),
+ _clean_text(metadata.get("status", "")),
+ _clean_text(parent.value if parent else ""),
+ _clean_text(parent_metadata.get("source_span", "")),
+ _clean_text(parent_metadata.get("raw_text", "")),
+ ]
+ )
+
+
+def _multi_unit_chain_local_hit_text(record: SessionMemoryRecordV2, parent: SessionMemoryRecordV2 | None) -> str:
+ metadata = dict(record.metadata or {})
+ parent_metadata = dict(parent.metadata or {}) if parent is not None else {}
+ return " ".join(
+ [
+ _clean_text(record.value),
+ _clean_text(record.category),
+ _clean_text(record.relation),
+ _clean_text(metadata.get("facet_type", "")),
+ _clean_text(metadata.get("unit_kind", "")),
+ _clean_text(metadata.get("facet_role", "")),
+ _clean_text(metadata.get("facet_value", "")),
+ _clean_text(metadata.get("facet_source_span", "")),
+ _clean_text(metadata.get("action", "")),
+ _clean_text(metadata.get("target", "")),
+ _clean_text(metadata.get("quantity", "")),
+ _clean_text(metadata.get("status", "")),
+ _clean_text(parent.value if parent else ""),
+ _clean_text(parent_metadata.get("source_span", "")),
+ ]
+ )
+
+
+def _multi_unit_chain_slot_hits(
+ graph: SessionMemoryGraphV2,
+ query: str,
+ final_hits: Sequence[MemoryHit],
+ *,
+ top_k: int,
+) -> Dict[str, Any]:
+ mode = _normalize(os.getenv("TMCRA_MULTI_UNIT_CHAIN_SLOT_MODE", "on"))
+ if mode in _MULTI_UNIT_CHAIN_DISABLED_MODES:
+ return {
+ "hits": list(final_hits),
+ "metadata": {"multi_unit_chain_slot_enabled": False, "multi_unit_chain_slot_reason": "disabled"},
+ }
+ query_tokens = set(_path_utility_tokens(query))
+ focus_tokens = _multi_unit_chain_focus_tokens(query)
+ temporal_comparison_intent = bool(query_tokens & _MULTI_UNIT_CHAIN_TEMPORAL_COMPARISON_TOKENS) and len(focus_tokens) >= 2
+ aggregation_or_joiner_intent = bool(
+ re.search(r"\b(?:and|or|both|each|total|sum|minimum|maximum|amount|count|many|number)\b", str(query), flags=re.IGNORECASE)
+ )
+ numeric_aggregation_intent = bool(
+ query_tokens
+ & (
+ _MULTI_UNIT_CHAIN_NUMERIC_VALUE_TOKENS
+ | {"comment", "comments", "number", "percent", "percentage", "sum", "total"}
+ )
+ )
+ count_or_aggregation_intent = bool(query_tokens & _MULTI_UNIT_CHAIN_COUNT_TOKENS) and (aggregation_or_joiner_intent or len(focus_tokens) >= 3)
+ multi_intent = count_or_aggregation_intent or temporal_comparison_intent
+ if not query_tokens or not multi_intent:
+ return {
+ "hits": list(final_hits),
+ "metadata": {
+ "multi_unit_chain_slot_enabled": False,
+ "multi_unit_chain_slot_reason": "no_multi_intent",
+ },
+ }
+ if not focus_tokens:
+ return {
+ "hits": list(final_hits),
+ "metadata": {
+ "multi_unit_chain_slot_enabled": True,
+ "multi_unit_chain_slot_formed": False,
+ "multi_unit_chain_slot_reason": "no_focus_tokens",
+ },
+ }
+
+ records = list(getattr(graph, "records_by_id", {}).values())
+ parent_by_slot = {_clean_text(record.slot_key).lower(): record for record in records}
+ candidates: List[tuple[float, SessionMemoryRecordV2, SessionMemoryRecordV2 | None, Dict[str, Any]]] = []
+ for record in records:
+ metadata = dict(record.metadata or {})
+ if record.state not in {"active", "parallel_active", "evidence"}:
+ continue
+ profile_shadow_unit = _profile_shadow_eventlike_record(record, metadata)
+ if profile_shadow_unit:
+ facet_type = "action"
+ unit_kind = "profile_shadow_unit"
+ parent_slot = _clean_text(record.slot_key).lower()
+ parent = None
+ text = _profile_shadow_unit_text(record, metadata)
+ local_text = text
+ else:
+ if _normalize(metadata.get("content_variant", "")) != "event_facet_write":
+ continue
+ if _normalize(metadata.get("facet_layer_version", "")) != "event_unit_v1":
+ continue
+ facet_type = _normalize(metadata.get("facet_type", ""))
+ unit_kind = _normalize(metadata.get("unit_kind", ""))
+ if facet_type not in _MULTI_UNIT_CHAIN_FACET_TYPES and unit_kind not in _MULTI_UNIT_CHAIN_UNIT_KINDS:
+ continue
+ parent_slot = _clean_text(metadata.get("facet_parent_slot_key", "")).lower()
+ parent = parent_by_slot.get(parent_slot)
+ text = _multi_unit_chain_hit_text(record, parent)
+ local_text = _multi_unit_chain_local_hit_text(record, parent)
+ semantic_event_unit = _normalize(record.memory_id).startswith("tmcra.event.") or _normalize(parent_slot).startswith("tmcra.event.")
+ unit_tokens = _multi_unit_chain_normalized_tokens(text)
+ local_tokens = _multi_unit_chain_normalized_tokens(local_text)
+ focus_overlap = focus_tokens & unit_tokens
+ if not focus_overlap:
+ continue
+ score = min(1.2, len(focus_overlap) / max(1.0, len(focus_tokens)) * 1.6)
+ local_numeric_signal = _multi_unit_chain_numeric_signal(unit_kind, facet_type, local_text, local_tokens)
+ if _multi_unit_chain_date_like_numeric(local_text):
+ numeric_signal = 0.0
+ else:
+ numeric_signal = local_numeric_signal
+ if numeric_signal <= 0.0:
+ numeric_signal = _multi_unit_chain_numeric_signal(unit_kind, facet_type, text, unit_tokens)
+ temporal_anchor_signal = _multi_unit_chain_temporal_anchor_signal(text, unit_tokens)
+ local_temporal_anchor_signal = _multi_unit_chain_temporal_anchor_signal(local_text, local_tokens)
+ if facet_type in {"action", "entity", "role", "numeric"}:
+ score += 0.34
+ if unit_kind in {"action_unit", "target_entity", "leadership_role", "participation_role", "numeric_quantity"}:
+ score += 0.24
+ if profile_shadow_unit:
+ score += 0.30
+ semantic_event_priority = bool(semantic_event_unit and (count_or_aggregation_intent or numeric_aggregation_intent))
+ if semantic_event_priority:
+ score += 0.48
+ if numeric_signal and numeric_aggregation_intent:
+ score += numeric_signal
+ if count_or_aggregation_intent and not numeric_aggregation_intent and facet_type in {"action", "entity", "role"}:
+ score += 0.36
+ if count_or_aggregation_intent and not numeric_aggregation_intent and unit_kind in {"action_unit", "target_entity", "leadership_role", "participation_role"}:
+ score += 0.22
+ if parent is not None:
+ score += 0.12
+ if score < 0.72:
+ continue
+ candidates.append(
+ (
+ round(min(4.0, 2.15 + score), 6),
+ record,
+ parent,
+ {
+ "multi_unit_chain_focus_overlap_tokens": sorted(focus_overlap)[:16],
+ "multi_unit_chain_score": round(score, 6),
+ "multi_unit_chain_numeric_signal": round(numeric_signal, 6),
+ "multi_unit_chain_temporal_anchor_signal": round(temporal_anchor_signal, 6),
+ "multi_unit_chain_local_temporal_anchor_signal": round(local_temporal_anchor_signal, 6),
+ "multi_unit_chain_temporal_comparison": bool(temporal_comparison_intent),
+ "multi_unit_chain_numeric_aggregation": bool(numeric_aggregation_intent),
+ "unit_kind": unit_kind,
+ "facet_type": facet_type,
+ "multi_unit_chain_parent_slot_key": parent_slot,
+ "multi_unit_chain_profile_shadow_unit": bool(profile_shadow_unit),
+ "multi_unit_chain_semantic_event_unit": bool(semantic_event_unit),
+ "multi_unit_chain_semantic_event_priority": bool(semantic_event_priority),
+ "multi_unit_chain_count_or_aggregation": bool(count_or_aggregation_intent),
+ },
+ )
+ )
+
+ if not candidates:
+ return {
+ "hits": list(final_hits),
+ "metadata": {
+ "multi_unit_chain_slot_enabled": True,
+ "multi_unit_chain_slot_formed": False,
+ "multi_unit_chain_slot_reason": "no_matching_units",
+ "multi_unit_chain_candidate_count": 0,
+ "multi_unit_chain_focus_tokens": sorted(focus_tokens)[:24],
+ },
+ }
+
+ if temporal_comparison_intent:
+ candidates.sort(
+ key=lambda item: (
+ float(item[3].get("multi_unit_chain_local_temporal_anchor_signal", 0.0) or 0.0),
+ float(item[3].get("multi_unit_chain_temporal_anchor_signal", 0.0) or 0.0),
+ 1 if bool(item[3].get("multi_unit_chain_semantic_event_priority", False)) else 0,
+ 0
+ if _normalize(item[3].get("unit_kind", "")) == "numeric_quantity"
+ or _normalize(item[3].get("facet_type", "")) == "numeric"
+ else 1,
+ float(item[0]),
+ int(item[1].turn_index),
+ ),
+ reverse=True,
+ )
+ else:
+ candidates.sort(
+ key=lambda item: (
+ float(item[3].get("multi_unit_chain_numeric_signal", 0.0) or 0.0)
+ if bool(item[3].get("multi_unit_chain_numeric_aggregation", False))
+ else (
+ 1.2
+ if bool(item[3].get("multi_unit_chain_semantic_event_priority", False))
+ else
+ 0.0
+ if _normalize(item[3].get("unit_kind", "")) == "numeric_quantity"
+ or _normalize(item[3].get("facet_type", "")) == "numeric"
+ else 1.0
+ ),
+ float(item[0]),
+ int(item[1].turn_index),
+ ),
+ reverse=True,
+ )
+ selected: List[tuple[float, SessionMemoryRecordV2, SessionMemoryRecordV2 | None, Dict[str, Any]]] = []
+ seen_parent_slots: set[str] = set()
+ seen_values: set[str] = set()
+ max_units = max(2, min(8, int(os.getenv("TMCRA_MULTI_UNIT_CHAIN_SLOT_MAX_UNITS", "6") or 6)))
+ for item in candidates:
+ _, record, parent, metadata = item
+ parent_slot = _clean_text(metadata.get("multi_unit_chain_parent_slot_key", ""))
+ value_key = "|".join(
+ [
+ _normalize(record.value),
+ _normalize(metadata.get("unit_kind", "")),
+ parent_slot,
+ ]
+ )
+ if value_key in seen_values:
+ continue
+ if parent_slot and parent_slot in seen_parent_slots:
+ # Keep the chain broad: one strongest unit per parent event first.
+ continue
+ seen_values.add(value_key)
+ if parent_slot:
+ seen_parent_slots.add(parent_slot)
+ selected.append(item)
+ if len(selected) >= max_units:
+ break
+ if count_or_aggregation_intent:
+ try:
+ max_profile_shadow_units = max(0, int(os.getenv("TMCRA_MULTI_UNIT_CHAIN_PROFILE_SHADOW_MAX_UNITS", "2") or 2))
+ except (TypeError, ValueError):
+ max_profile_shadow_units = 2
+ selected_ids = {item[1].memory_id for item in selected}
+ added_profile_shadow_units = 0
+ for item in candidates:
+ _, record, _parent, metadata = item
+ if added_profile_shadow_units >= max_profile_shadow_units:
+ break
+ if record.memory_id in selected_ids:
+ continue
+ if not bool(metadata.get("multi_unit_chain_profile_shadow_unit", False)):
+ continue
+ selected.append(item)
+ selected_ids.add(record.memory_id)
+ parent_slot = _clean_text(metadata.get("multi_unit_chain_parent_slot_key", ""))
+ if parent_slot:
+ seen_parent_slots.add(parent_slot)
+ added_profile_shadow_units += 1
+
+ min_parents = max(2, min(4, int(os.getenv("TMCRA_MULTI_UNIT_CHAIN_SLOT_MIN_PARENTS", "2") or 2)))
+ if len(seen_parent_slots) < min_parents:
+ return {
+ "hits": list(final_hits),
+ "metadata": {
+ "multi_unit_chain_slot_enabled": True,
+ "multi_unit_chain_slot_formed": False,
+ "multi_unit_chain_slot_reason": "insufficient_parent_coverage",
+ "multi_unit_chain_candidate_count": len(candidates),
+ "multi_unit_chain_selected_unit_count": len(selected),
+ "multi_unit_chain_parent_count": len(seen_parent_slots),
+ "multi_unit_chain_focus_tokens": sorted(focus_tokens)[:24],
+ },
+ }
+
+ chain_memory_ids: List[str] = []
+ packed_hits: List[MemoryHit] = []
+ for score, unit_record, parent, extra in selected:
+ chain_memory_ids.append(unit_record.memory_id)
+ packed_hits.append(
+ _memory_hit_from_record(
+ unit_record,
+ score=score,
+ metadata={
+ **extra,
+ "multi_unit_chain_slot": True,
+ "multi_unit_chain_bundle": True,
+ "evidence_snippet_role": "multi_unit_chain_evidence_unit",
+ },
+ )
+ )
+ if parent is not None:
+ unit_metadata = dict(unit_record.metadata or {})
+ chain_memory_ids.append(parent.memory_id)
+ parent_hit = _memory_hit_from_record(
+ parent,
+ score=max(1.9, score - 0.08),
+ metadata={
+ **extra,
+ "multi_unit_chain_slot": True,
+ "multi_unit_chain_bundle": True,
+ "evidence_snippet_role": "multi_unit_chain_parent_event",
+ "multi_unit_chain_child_id": unit_record.memory_id,
+ "multi_unit_chain_child_value": unit_record.value,
+ "multi_unit_chain_child_source_span": _clean_text(
+ unit_metadata.get("facet_source_span", "") or unit_metadata.get("source_span", "")
+ ),
+ },
+ )
+ parent_hit.memory_id = f"{parent.memory_id}#multi_parent:{unit_record.memory_id}"
+ packed_hits.append(parent_hit)
+
+ seen_ids: set[str] = set()
+ merged: List[MemoryHit] = []
+ insertion_index = max(0, min(len(final_hits), int(os.getenv("TMCRA_MULTI_UNIT_CHAIN_SLOT_INSERT_AFTER", "1") or 1)))
+ ordered = [*list(final_hits[:insertion_index]), *packed_hits, *list(final_hits[insertion_index:])]
+ for hit in ordered:
+ if hit.memory_id and hit.memory_id in seen_ids:
+ continue
+ if hit.memory_id:
+ seen_ids.add(hit.memory_id)
+ merged.append(hit)
+ return {
+ "hits": merged,
+ "metadata": {
+ "multi_unit_chain_slot_enabled": True,
+ "multi_unit_chain_slot_formed": True,
+ "multi_unit_chain_slot_reason": "formed",
+ "multi_unit_chain_candidate_count": len(candidates),
+ "multi_unit_chain_selected_unit_count": len(selected),
+ "multi_unit_chain_parent_count": len(seen_parent_slots),
+ "multi_unit_chain_inserted_hit_count": len(packed_hits),
+ "multi_unit_chain_memory_ids": _dedupe(chain_memory_ids, max_items=32),
+ "multi_unit_chain_focus_tokens": sorted(focus_tokens)[:24],
+ },
+ }
+
+
+def _profile_support_source_hits(
+ graph: SessionMemoryGraphV2,
+ profile_hit: MemoryHit,
+ *,
+ grouped_hits: Mapping[str, Sequence[MemoryHit]],
+ query: str,
+ limit: int,
+) -> List[MemoryHit]:
+ metadata = dict(profile_hit.metadata or {})
+ support_ids = _dedupe(
+ [
+ *list(metadata.get("profile_support_ids", []) or []),
+ *list(metadata.get("support_memory_ids", []) or []),
+ ],
+ max_items=24,
+ )
+ if not support_ids:
+ return []
+ query_raw_tokens = set(_path_utility_tokens(query))
+ query_tokens = _profile_query_expanded_tokens(query)
+ profile_event_id = _clean_text(metadata.get("profile_first_hybrid_event_id", "")) or _runtime_event_key(profile_hit)
+ candidates: List[tuple[float, MemoryHit]] = []
+ for support_id in support_ids:
+ record = getattr(graph, "records_by_id", {}).get(support_id)
+ if record is None or record.state not in {"active", "parallel_active", "evidence"}:
+ continue
+ support_hit = _memory_hit_from_record(record)
+ event_id = _runtime_event_key(support_hit)
+ if not event_id or event_id == profile_event_id or event_id not in grouped_hits:
+ continue
+ match_score, overlap_tokens, raw_overlap_tokens = _profile_hit_match_score(query_raw_tokens, query_tokens, support_hit)
+ source_score = max(0.80, float(profile_hit.score) - 0.02) + min(0.28, match_score * 0.12)
+ support_metadata = dict(support_hit.metadata or {})
+ profile_metadata = dict(profile_hit.metadata or {})
+ support_metadata.update(
+ {
+ "profile_first_hybrid_rescue": True,
+ "profile_first_source_support": True,
+ "profile_first_parent_memory_id": profile_hit.memory_id,
+ "profile_first_hybrid_event_id": event_id,
+ "profile_first_parent_summary": _clean_text(profile_metadata.get("profile_summary", "")) or _clean_text(profile_metadata.get("profile_value", "")) or _clean_text(profile_hit.value),
+ "profile_type": _clean_text(support_metadata.get("profile_type", "")) or _clean_text(profile_metadata.get("profile_type", "")),
+ "profile_domain": _clean_text(support_metadata.get("profile_domain", "")) or _clean_text(profile_metadata.get("profile_domain", "")),
+ "profile_domain_label": _clean_text(support_metadata.get("profile_domain_label", "")) or _clean_text(profile_metadata.get("profile_domain_label", "")),
+ "profile_value": _clean_text(support_metadata.get("profile_value", "")) or _clean_text(profile_metadata.get("profile_value", "")) or _clean_text(profile_hit.value),
+ "profile_summary": _clean_text(support_metadata.get("profile_summary", "")) or _clean_text(profile_metadata.get("profile_summary", "")) or _clean_text(profile_hit.value),
+ "profile_query_match_score": round(match_score, 6),
+ "profile_query_overlap_tokens": list(overlap_tokens),
+ "profile_query_raw_overlap_tokens": list(raw_overlap_tokens),
+ "profile_source_pack_role": "source_event_support",
+ }
+ )
+ candidates.append(
+ (
+ source_score,
+ MemoryHit(
+ memory_id=support_hit.memory_id,
+ category=support_hit.category,
+ value=support_hit.value,
+ relation=support_hit.relation,
+ anchors=list(support_hit.anchors),
+ score=round(source_score, 6),
+ source_kind=support_hit.source_kind,
+ slot_key=support_hit.slot_key,
+ state=support_hit.state,
+ turn_index=int(support_hit.turn_index),
+ metadata=support_metadata,
+ ),
+ )
+ )
+ candidates.sort(
+ key=lambda item: (
+ bool((item[1].metadata or {}).get("profile_query_raw_overlap_tokens")),
+ float(item[0]),
+ int(item[1].turn_index),
+ ),
+ reverse=True,
+ )
+ return [hit for _, hit in candidates[: max(0, int(limit))]]
+
+
+def _profile_same_event_source_hit(profile_hit: MemoryHit, group_hits: Sequence[MemoryHit], *, event_id: str) -> MemoryHit | None:
+ source_hit = _support_hit_for_path("speaker_event_source_turn", group_hits) or _representative_event_hit(group_hits)
+ if source_hit is None or source_hit.memory_id == profile_hit.memory_id:
+ return None
+ metadata = dict(source_hit.metadata or {})
+ profile_metadata = dict(profile_hit.metadata or {})
+ metadata.update(
+ {
+ "profile_first_hybrid_rescue": True,
+ "profile_first_source_support": True,
+ "profile_first_same_event_support": True,
+ "profile_first_parent_memory_id": profile_hit.memory_id,
+ "profile_first_hybrid_event_id": event_id,
+ "profile_first_parent_summary": _clean_text(profile_metadata.get("profile_summary", "")) or _clean_text(profile_metadata.get("profile_value", "")) or _clean_text(profile_hit.value),
+ "profile_type": _clean_text(metadata.get("profile_type", "")) or _clean_text(profile_metadata.get("profile_type", "")),
+ "profile_domain": _clean_text(metadata.get("profile_domain", "")) or _clean_text(profile_metadata.get("profile_domain", "")),
+ "profile_domain_label": _clean_text(metadata.get("profile_domain_label", "")) or _clean_text(profile_metadata.get("profile_domain_label", "")),
+ "profile_value": _clean_text(metadata.get("profile_value", "")) or _clean_text(profile_metadata.get("profile_value", "")) or _clean_text(profile_hit.value),
+ "profile_summary": _clean_text(metadata.get("profile_summary", "")) or _clean_text(profile_metadata.get("profile_summary", "")) or _clean_text(profile_hit.value),
+ "profile_source_pack_role": "same_event_source_turn",
+ "evidence_snippet_role": "profile_source_support",
+ }
+ )
+ return MemoryHit(
+ memory_id=source_hit.memory_id,
+ category=source_hit.category,
+ value=source_hit.value,
+ relation=source_hit.relation,
+ anchors=list(source_hit.anchors),
+ score=max(float(source_hit.score), float(profile_hit.score) - 0.01),
+ source_kind=source_hit.source_kind,
+ slot_key=source_hit.slot_key,
+ state=source_hit.state,
+ turn_index=int(source_hit.turn_index),
+ metadata=metadata,
+ )
+
+
+def _profile_first_hybrid_rescue(
+ graph: SessionMemoryGraphV2,
+ query: str,
+ *,
+ grouped_hits: Mapping[str, Sequence[MemoryHit]],
+ top_k: int,
+) -> Dict[str, Any]:
+ intent = infer_profile_query_intent(query)
+ if not bool(intent.get("enabled")):
+ return {"hits": [], "event_ids": []}
+ raw_hits = _profile_query_rescue_hits(graph, query, top_k=max(4, int(top_k or 1) * 3))
+ limit = max(2, min(4, int(top_k or 1)))
+ selected_hits: List[MemoryHit] = []
+ selected_event_ids: List[str] = []
+ seen_event_ids: set[str] = set()
+ for rank, hit in enumerate(raw_hits, start=1):
+ event_id = _runtime_event_key(hit)
+ if not event_id or event_id not in grouped_hits or event_id in seen_event_ids:
+ continue
+ metadata = dict(hit.metadata or {})
+ metadata.update(
+ {
+ "profile_first_hybrid_rescue": True,
+ "profile_first_hybrid_rank": rank,
+ "profile_first_hybrid_event_id": event_id,
+ }
+ )
+ hit.metadata = metadata
+ hit.score = round(max(float(hit.score), 0.88) + max(0.0, 0.08 - (0.01 * len(selected_hits))), 6)
+ selected_hits.append(hit)
+ selected_event_ids.append(event_id)
+ seen_event_ids.add(event_id)
+ same_event_source = _profile_same_event_source_hit(hit, grouped_hits.get(event_id, []), event_id=event_id)
+ if same_event_source is not None:
+ selected_hits.append(same_event_source)
+ if len(selected_hits) >= limit:
+ break
+ support_hits = _profile_support_source_hits(
+ graph,
+ hit,
+ grouped_hits=grouped_hits,
+ query=query,
+ limit=max(1, min(3, limit - len(selected_hits) + 1)),
+ )
+ for support_hit in support_hits:
+ support_event_id = _clean_text((support_hit.metadata or {}).get("profile_first_hybrid_event_id", "")) or _runtime_event_key(support_hit)
+ if not support_event_id or support_event_id in seen_event_ids:
+ continue
+ selected_hits.append(support_hit)
+ selected_event_ids.append(support_event_id)
+ seen_event_ids.add(support_event_id)
+ if len(selected_hits) >= limit:
+ break
+ if len(selected_hits) >= limit:
+ break
+ return {
+ "hits": selected_hits,
+ "event_ids": selected_event_ids,
+ "memory_ids": [hit.memory_id for hit in selected_hits],
+ }
+
+
+def _inject_profile_first_hits(
+ final_hits: Sequence[MemoryHit],
+ profile_first_hits: Sequence[MemoryHit],
+ *,
+ selected_event_ids: Sequence[str],
+ selected_path_ids: Sequence[str],
+) -> List[MemoryHit]:
+ merged: List[MemoryHit] = []
+ seen_memory_ids: set[str] = set()
+ for hit in list(profile_first_hits) + list(final_hits):
+ if not hit or not hit.memory_id or hit.memory_id in seen_memory_ids:
+ continue
+ metadata = dict(hit.metadata or {})
+ if bool(metadata.get("profile_first_hybrid_rescue")):
+ event_id = _clean_text(metadata.get("profile_first_hybrid_event_id", "")) or _runtime_event_key(hit)
+ metadata.update(
+ {
+ "event_id": event_id,
+ "path_id": "",
+ "evidence_snippet_role": "selected_event_representative",
+ "hybrid_score_source": "profile_first_hybrid_rescue",
+ "selected_event_ids": list(selected_event_ids),
+ "selected_path_ids": list(selected_path_ids),
+ }
+ )
+ hit = MemoryHit(
+ memory_id=hit.memory_id,
+ category=hit.category,
+ value=hit.value,
+ relation=hit.relation,
+ anchors=list(hit.anchors),
+ score=float(hit.score),
+ source_kind=hit.source_kind,
+ slot_key=hit.slot_key,
+ state=hit.state,
+ turn_index=int(hit.turn_index),
+ metadata=metadata,
+ )
+ merged.append(hit)
+ seen_memory_ids.add(hit.memory_id)
+ return merged
+
+
+def _profile_focused_pack_hits(
+ graph: SessionMemoryGraphV2,
+ query: str,
+ final_hits: Sequence[MemoryHit],
+ *,
+ top_k: int,
+) -> Dict[str, Any]:
+ intent = infer_profile_query_intent(query)
+ if not bool(intent.get("enabled")):
+ return {
+ "hits": list(final_hits),
+ "metadata": {
+ "profile_focused_pack_enabled": False,
+ "profile_focused_pack_reason": "profile_intent_not_requested",
+ },
+ }
+ source_hits = _learnable_graph_hits(graph)
+ if not source_hits:
+ return {
+ "hits": list(final_hits),
+ "metadata": {
+ "profile_focused_pack_enabled": False,
+ "profile_focused_pack_reason": "no_learnable_hits",
+ },
+ }
+ runtime_graph = _build_runtime_graph_from_hits(query, source_hits)
+ grouped_hits = dict(runtime_graph.get("grouped_hits", {}) or {})
+ profile_first_payload = _profile_first_hybrid_rescue(
+ graph,
+ query,
+ grouped_hits=grouped_hits,
+ top_k=max(2, min(max(1, int(top_k or 1)), 4)),
+ )
+ profile_first_hits = list(profile_first_payload.get("hits", []) or [])
+ profile_first_event_ids = list(profile_first_payload.get("event_ids", []) or [])
+ profile_first_memory_ids = list(profile_first_payload.get("memory_ids", []) or [])
+ if not profile_first_hits:
+ return {
+ "hits": list(final_hits),
+ "metadata": {
+ "profile_focused_pack_enabled": True,
+ "profile_focused_pack_reason": "no_profile_hits",
+ "profile_focused_pack_event_ids": [],
+ "profile_focused_pack_memory_ids": [],
+ },
+ }
+ selected_event_ids = _dedupe([*profile_first_event_ids, *_event_ids_from_hits(final_hits)], max_items=max(1, int(top_k or 1) * 2))
+ selected_path_ids = _dedupe(
+ [
+ _clean_text((hit.metadata or {}).get("path_id", ""))
+ for hit in final_hits
+ if _clean_text((hit.metadata or {}).get("path_id", ""))
+ ],
+ max_items=max(1, int(top_k or 1)),
+ )
+ merged_hits = _inject_profile_first_hits(
+ final_hits,
+ profile_first_hits,
+ selected_event_ids=selected_event_ids,
+ selected_path_ids=selected_path_ids,
+ )
+ merged_hits = _coverage_preserving_final_hits(merged_hits, selected_event_ids=selected_event_ids, top_k=top_k)
+ return {
+ "hits": merged_hits,
+ "metadata": {
+ "profile_focused_pack_enabled": True,
+ "profile_focused_pack_reason": "profile_first_pack_injected",
+ "profile_focused_pack_event_ids": list(profile_first_event_ids),
+ "profile_focused_pack_memory_ids": list(profile_first_memory_ids),
+ "profile_focused_pack_hit_count": len(profile_first_hits),
+ "profile_first_hybrid_enabled": True,
+ "profile_first_event_ids": list(profile_first_event_ids),
+ "profile_first_memory_ids": list(profile_first_memory_ids),
+ },
+ }
+
+
+def _topic_bucket_rerank_hits(
+ graph: SessionMemoryGraphV2,
+ query: str,
+ hits: Sequence[MemoryHit],
+ *,
+ top_k: int,
+) -> Dict[str, Any]:
+ if not hits:
+ profile_rescue_hits = _profile_query_rescue_hits(graph, query, top_k=top_k)
+ if profile_rescue_hits:
+ return {
+ "hits": profile_rescue_hits[: max(1, int(top_k or 1))],
+ "metadata": {
+ "topic_bucket_rerank_enabled": True,
+ "topic_bucket_rerank_reason": "profile_query_rescue_from_empty_hits",
+ "topic_bucket_profile_query_rescue_count": len(profile_rescue_hits),
+ },
+ }
+ return {
+ "hits": [],
+ "metadata": {"topic_bucket_rerank_enabled": True, "topic_bucket_rerank_reason": "no_hits"},
+ }
+ query_topic = _assign_topic_bucket_for_text(graph, query, turn_index=0, create=False)
+ query_bucket_id = _clean_text(query_topic.get("topic_bucket_id", ""))
+ adjacent_ids = _topic_adjacent_bucket_ids(graph, query_bucket_id)
+ dialogue_tunnel_ids = _dialogue_tunnel_bucket_ids(graph, query_bucket_id)
+ query_keywords = list(query_topic.get("topic_keywords", []) or [])
+ bridge_requested = _topic_bridge_requested(query)
+ dialogue_requested = _dialogue_tunnel_requested(query)
+ profile_query_requested = bool(infer_profile_query_intent(query).get("enabled"))
+ reranked: List[MemoryHit] = []
+ stats = {
+ "same_bucket": 0,
+ "bridge_bucket": 0,
+ "blocked_bridge_bucket": 0,
+ "dialogue_tunnel_bucket": 0,
+ "blocked_dialogue_tunnel_bucket": 0,
+ "profile_route_preserved": 0,
+ "profile_query_rescue": 0,
+ "overlap_bucket": 0,
+ "off_topic": 0,
+ }
+ for hit in list(hits):
+ metadata = dict(hit.metadata or {})
+ hit_bucket_id = _clean_text(metadata.get("topic_bucket_id", ""))
+ hit_keywords = _dedupe(metadata.get("topic_keywords", []) or [], max_items=32)
+ overlap = _topic_bucket_overlap_score(query_keywords, hit_keywords)
+ same_bucket = bool(query_bucket_id and hit_bucket_id and query_bucket_id == hit_bucket_id)
+ bridge_bucket = bool(hit_bucket_id and hit_bucket_id in adjacent_ids)
+ dialogue_tunnel_bucket = bool(hit_bucket_id and hit_bucket_id in dialogue_tunnel_ids)
+ overlap_bucket = overlap >= 0.22
+ bridge_allowed = bridge_bucket and bridge_requested
+ dialogue_allowed = dialogue_tunnel_bucket and dialogue_requested
+ profile_route_preserved = bool(
+ profile_query_requested
+ and (
+ bool(metadata.get("profile_layer"))
+ or "profile_route" in _normalize(metadata.get("match_reason", ""))
+ )
+ )
+ current_subject_preserved = bool(metadata.get("current_subject_resolver") or metadata.get("public_subject_match"))
+ delta = 0.0
+ if same_bucket:
+ delta += 1.35
+ stats["same_bucket"] += 1
+ elif current_subject_preserved:
+ delta += 0.18
+ stats["profile_route_preserved"] += 1
+ elif bridge_allowed:
+ delta += 0.42
+ stats["bridge_bucket"] += 1
+ elif dialogue_allowed:
+ delta += 0.16
+ stats["dialogue_tunnel_bucket"] += 1
+ elif overlap_bucket:
+ delta += 0.28 + overlap
+ stats["overlap_bucket"] += 1
+ elif bridge_bucket:
+ delta -= 1.05
+ stats["blocked_bridge_bucket"] += 1
+ elif dialogue_tunnel_bucket:
+ delta -= 1.35
+ stats["blocked_dialogue_tunnel_bucket"] += 1
+ elif profile_route_preserved:
+ delta += 0.08
+ stats["profile_route_preserved"] += 1
+ else:
+ delta -= 0.72
+ stats["off_topic"] += 1
+ memory_type = _normalize(metadata.get("memory_type", ""))
+ durability = _normalize(metadata.get("durability", ""))
+ conflict_policy = _normalize(metadata.get("conflict_policy", ""))
+ if (memory_type == "hard_constraint" or durability == "hard" or conflict_policy == "must_preserve") and (
+ same_bucket or bridge_allowed or dialogue_allowed or overlap_bucket
+ ):
+ delta += 1.10
+ match_reason = _clean_text(metadata.get("match_reason", ""))
+ if profile_route_preserved and "profile_route" not in _normalize(match_reason):
+ match_reason = ",".join(_dedupe([match_reason, "profile_route"], max_items=4))
+ metadata.update(
+ {
+ "topic_bucket_rerank": True,
+ "topic_bucket_query_id": query_bucket_id,
+ "topic_bucket_query_label": _clean_text(query_topic.get("topic_label", "")),
+ "topic_bucket_overlap": round(overlap, 6),
+ "topic_bucket_delta": round(delta, 6),
+ "topic_bucket_same": same_bucket,
+ "topic_bucket_bridge": bridge_bucket,
+ "topic_bucket_bridge_allowed": bridge_allowed,
+ "topic_bucket_dialogue_tunnel": dialogue_tunnel_bucket,
+ "topic_bucket_dialogue_tunnel_allowed": dialogue_allowed,
+ "topic_bucket_profile_route_preserved": profile_route_preserved,
+ "topic_bucket_current_subject_preserved": current_subject_preserved,
+ "match_reason": match_reason,
+ }
+ )
+ hit.metadata = metadata
+ hit.score = float(hit.score) + delta
+ reranked.append(hit)
+ seen_ids = {hit.memory_id for hit in reranked if hit.memory_id}
+ rescue_records = [
+ record
+ for record in getattr(graph, "records_by_id", {}).values()
+ if record.memory_id not in seen_ids
+ and record.state == "active"
+ and _clean_text((record.metadata or {}).get("topic_bucket_id", "")) == query_bucket_id
+ and _normalize(record.category) != "question"
+ ]
+ rescue_records.sort(
+ key=lambda record: (
+ int(any(marker in _normalize(record.value) for marker in ("过敏", "必须", "避开", "禁止", "不能", "must", "avoid", "allergy"))),
+ int(_normalize(record.category) in {"constraint", "preference"}),
+ float(record.confidence),
+ float(record.salience),
+ -int(record.turn_index),
+ ),
+ reverse=True,
+ )
+ rescue_hits = [
+ _topic_bucket_record_to_hit(record, query_topic=query_topic, rank=index)
+ for index, record in enumerate(rescue_records[: max(2, min(12, int(top_k or 1) * 2))], start=1)
+ ]
+ reranked.extend(rescue_hits)
+ dialogue_rescue_hits: List[MemoryHit] = []
+ if dialogue_requested and dialogue_tunnel_ids:
+ seen_ids.update(hit.memory_id for hit in rescue_hits if hit.memory_id)
+ dialogue_records = [
+ record
+ for record in getattr(graph, "records_by_id", {}).values()
+ if record.memory_id not in seen_ids
+ and record.state == "active"
+ and _clean_text((record.metadata or {}).get("topic_bucket_id", "")) in dialogue_tunnel_ids
+ and _normalize(record.category) != "question"
+ ]
+ def dialogue_record_rank(record: SessionMemoryRecordV2) -> tuple[int, int, float, float, int]:
+ return (
+ int(any(marker in _normalize(record.value) for marker in ("过敏", "必须", "避开", "禁止", "不能", "must", "avoid", "allergy"))),
+ int(_normalize(record.category) in {"constraint", "preference"}),
+ float(record.confidence),
+ float(record.salience),
+ -int(record.turn_index),
+ )
+
+ dialogue_records.sort(key=dialogue_record_rank, reverse=True)
+ selected_dialogue_records: List[SessionMemoryRecordV2] = []
+ selected_dialogue_ids: set[str] = set()
+ for bucket_id in sorted(dialogue_tunnel_ids):
+ bucket_records = [
+ record
+ for record in dialogue_records
+ if _clean_text((record.metadata or {}).get("topic_bucket_id", "")) == bucket_id
+ ]
+ if not bucket_records:
+ continue
+ best = bucket_records[0]
+ selected_dialogue_records.append(best)
+ selected_dialogue_ids.add(best.memory_id)
+ target_dialogue_rescue = max(len(selected_dialogue_records), max(1, min(4, int(top_k or 1) // 2 or 1)))
+ for record in dialogue_records:
+ if len(selected_dialogue_records) >= target_dialogue_rescue:
+ break
+ if record.memory_id in selected_dialogue_ids:
+ continue
+ selected_dialogue_records.append(record)
+ selected_dialogue_ids.add(record.memory_id)
+ dialogue_rescue_hits = [
+ _topic_bucket_record_to_hit(record, query_topic=query_topic, rank=index, rescue_kind="dialogue_tunnel")
+ for index, record in enumerate(selected_dialogue_records, start=1)
+ ]
+ reranked.extend(dialogue_rescue_hits)
+ profile_rescue_hits: List[MemoryHit] = []
+ if profile_query_requested:
+ seen_ids.update(hit.memory_id for hit in reranked if hit.memory_id)
+ for index, hit in enumerate(_profile_query_rescue_hits(graph, query, top_k=top_k), start=1):
+ if hit.memory_id in seen_ids:
+ continue
+ metadata = dict(hit.metadata or {})
+ metadata.update(
+ {
+ "profile_query_rescue_rank": index,
+ "topic_bucket_query_id": query_bucket_id,
+ "topic_bucket_query_label": _clean_text(query_topic.get("topic_label", "")),
+ "topic_bucket_profile_route_preserved": True,
+ }
+ )
+ hit.metadata = metadata
+ hit.score = float(hit.score) + 0.08
+ profile_rescue_hits.append(hit)
+ seen_ids.add(hit.memory_id)
+ stats["profile_query_rescue"] = len(profile_rescue_hits)
+ reranked.extend(profile_rescue_hits)
+ reranked.sort(key=lambda item: (float(item.score), int(item.turn_index)), reverse=True)
+ limit = max(1, int(top_k or 1))
+ focused_hits = [
+ hit
+ for hit in reranked
+ if bool((hit.metadata or {}).get("topic_bucket_same"))
+ or float((hit.metadata or {}).get("topic_bucket_overlap", 0.0) or 0.0) >= 0.22
+ or bool((hit.metadata or {}).get("topic_bucket_bridge_allowed"))
+ or bool((hit.metadata or {}).get("topic_bucket_dialogue_tunnel_allowed"))
+ or bool((hit.metadata or {}).get("topic_bucket_profile_route_preserved"))
+ or bool((hit.metadata or {}).get("topic_bucket_current_subject_preserved"))
+ or bool((hit.metadata or {}).get("current_subject_resolver"))
+ ]
+ model_path_fallback = False
+ no_bucket_model_fallback = False
+ if not focused_hits:
+ model_supported_hits = [
+ hit
+ for hit in reranked
+ if _clean_text((hit.metadata or {}).get("path_id", ""))
+ or _clean_text((hit.metadata or {}).get("hybrid_score_source", ""))
+ ]
+ if model_supported_hits:
+ focused_hits = model_supported_hits
+ model_path_fallback = True
+ generic_memory_fallback = False
+ if not focused_hits and any(not _is_public_dialog_hit(hit) for hit in reranked):
+ focused_hits = list(reranked)
+ generic_memory_fallback = True
+ if not focused_hits and not query_bucket_id:
+ focused_hits = list(reranked)
+ no_bucket_model_fallback = True
+ time_focused_model_path = any(
+ "speaker_event_time" in _clean_text((hit.metadata or {}).get("path_id", ""))
+ or _clean_text((hit.metadata or {}).get("model_focused_answer_type", "")) == "time"
+ for hit in focused_hits
+ )
+ if time_focused_model_path:
+ focused_hits = [
+ hit
+ for hit in focused_hits
+ if not (
+ _clean_text(hit.source_kind) == "public_dialog_profile"
+ and not _clean_text((hit.metadata or {}).get("path_id", ""))
+ )
+ ]
+ filtered_count = max(0, len(reranked) - len(focused_hits))
+ query_subject_signature = _public_subject_signature(_public_query_subject(query))
+
+ def _hit_subject_signatures(hit: MemoryHit) -> set[str]:
+ metadata = dict(hit.metadata or {})
+ signatures = {
+ _normalize(metadata.get("subject_signature", "")).replace("-", "_"),
+ _public_subject_signature(metadata.get("subject", "")),
+ }
+ canonical_slot_key = _clean_text(metadata.get("canonical_slot_key", ""))
+ if ".subject." in canonical_slot_key:
+ signatures.add(_public_subject_signature(canonical_slot_key.split(".subject.", 1)[-1]))
+ if ".subject." in hit.slot_key:
+ signatures.add(_public_subject_signature(hit.slot_key.split(".subject.", 1)[-1]))
+ signatures.discard("")
+ return signatures
+
+ protected_current_hits = [
+ hit
+ for hit in focused_hits
+ if bool((hit.metadata or {}).get("current_subject_resolver"))
+ ]
+ protected_model_path_hits = [
+ hit
+ for hit in focused_hits
+ if _clean_text((hit.metadata or {}).get("path_id", ""))
+ and _clean_text((hit.metadata or {}).get("hybrid_score_source", ""))
+ ]
+ protected_generic_hits = [
+ hit
+ for hit in focused_hits
+ if not _is_public_dialog_hit(hit)
+ and not bool((hit.metadata or {}).get("profile_layer"))
+ ]
+ if protected_current_hits and query_subject_signature:
+ protected_ids = {hit.memory_id for hit in protected_current_hits if hit.memory_id}
+ compacted_focused_hits: List[MemoryHit] = []
+ for hit in focused_hits:
+ if hit.memory_id in protected_ids:
+ compacted_focused_hits.append(hit)
+ continue
+ same_current_subject = query_subject_signature in _hit_subject_signatures(hit)
+ inactive_state = _normalize(hit.state) in {"superseded", "evidence", "historical", "stale", "false"}
+ if same_current_subject and inactive_state:
+ continue
+ compacted_focused_hits.append(hit)
+ focused_hits = compacted_focused_hits
+ filtered_count = max(0, len(reranked) - len(focused_hits))
+
+ selected_hits: List[MemoryHit] = []
+ selected_keys: set[str] = set()
+ for hit in [*protected_current_hits, *protected_model_path_hits, *protected_generic_hits, *focused_hits]:
+ key = hit.memory_id or f"{_hit_event_id(hit)}::{hit.slot_key}::{hit.value[:80]}"
+ if key in selected_keys:
+ continue
+ selected_hits.append(hit)
+ selected_keys.add(key)
+ if len(selected_hits) >= limit:
+ break
+ dialogue_reserved_count = 0
+ if dialogue_requested and dialogue_tunnel_ids:
+ selected_ids = {hit.memory_id for hit in selected_hits if hit.memory_id}
+ selected_bucket_ids = {
+ _clean_text((hit.metadata or {}).get("topic_bucket_id", ""))
+ for hit in selected_hits
+ if _clean_text((hit.metadata or {}).get("topic_bucket_id", ""))
+ }
+ for bucket_id in sorted(dialogue_tunnel_ids):
+ if bucket_id in selected_bucket_ids:
+ continue
+ candidate = next(
+ (
+ hit
+ for hit in focused_hits
+ if hit.memory_id not in selected_ids
+ and _clean_text((hit.metadata or {}).get("topic_bucket_id", "")) == bucket_id
+ and bool((hit.metadata or {}).get("topic_bucket_dialogue_tunnel_allowed"))
+ ),
+ None,
+ )
+ if candidate is None:
+ continue
+ if len(selected_hits) < limit:
+ selected_hits.append(candidate)
+ else:
+ replace_index = len(selected_hits) - 1
+ for index in range(len(selected_hits) - 1, -1, -1):
+ metadata = dict(selected_hits[index].metadata or {})
+ hardish = (
+ _normalize(metadata.get("memory_type", "")) == "hard_constraint"
+ or _normalize(metadata.get("durability", "")) == "hard"
+ or _normalize(metadata.get("conflict_policy", "")) == "must_preserve"
+ )
+ if not hardish and not bool(metadata.get("topic_bucket_dialogue_tunnel_allowed")):
+ replace_index = index
+ break
+ selected_hits[replace_index] = candidate
+ selected_ids.add(candidate.memory_id)
+ selected_bucket_ids.add(bucket_id)
+ dialogue_reserved_count += 1
+ selected_hits.sort(
+ key=lambda item: (
+ int(bool((item.metadata or {}).get("current_subject_resolver"))),
+ float(item.score),
+ int(item.turn_index),
+ ),
+ reverse=True,
+ )
+ return {
+ "hits": selected_hits[:limit],
+ "metadata": {
+ "topic_bucket_rerank_enabled": True,
+ "topic_bucket_no_fill_policy": True,
+ "topic_bucket_query_id": query_bucket_id,
+ "topic_bucket_query_label": _clean_text(query_topic.get("topic_label", "")),
+ "topic_bucket_query_keywords": query_keywords,
+ "topic_bucket_adjacent_ids": sorted(adjacent_ids),
+ "dialogue_tunnel_adjacent_ids": sorted(dialogue_tunnel_ids),
+ "topic_bucket_bridge_requested": bridge_requested,
+ "dialogue_tunnel_requested": dialogue_requested,
+ "topic_bucket_model_path_fallback": model_path_fallback,
+ "topic_bucket_generic_memory_fallback": generic_memory_fallback,
+ "topic_bucket_no_bucket_model_fallback": no_bucket_model_fallback,
+ "topic_bucket_candidate_count": len(hits),
+ "topic_bucket_rescue_count": len(rescue_hits),
+ "dialogue_tunnel_rescue_count": len(dialogue_rescue_hits),
+ "profile_query_rescue_count": len(profile_rescue_hits),
+ "dialogue_tunnel_reserved_count": dialogue_reserved_count,
+ "topic_bucket_focused_count": len(focused_hits),
+ "topic_bucket_final_count": len(selected_hits[:limit]),
+ "topic_bucket_filtered_count": filtered_count,
+ "topic_bucket_stats": stats,
+ },
+ }
+
+
+def _looks_like_write_turn(user_text: str, *, answer_payload: Dict[str, Any] | None = None) -> bool:
+ text = _normalize(user_text)
+ if not text:
+ return False
+ if "?" in text:
+ return False
+ if any(marker in text for marker in _WRITE_MARKERS):
+ return True
+ metadata = dict((answer_payload or {}).get("metadata", {}) or {})
+ return bool(metadata.get("memory_write"))
+
+
+def _explicit_overwrite_requested(user_text: str) -> bool:
+ text = _normalize(user_text)
+ return bool(text) and any(marker in text for marker in _OVERWRITE_MARKERS)
+
+
+def _apply_turn_write_intent(records: List[SessionMemoryRecordV2], *, user_text: str) -> List[SessionMemoryRecordV2]:
+ if not records or not _explicit_overwrite_requested(user_text):
+ return records
+ for record in records:
+ metadata = dict(record.metadata or {})
+ metadata.setdefault("write_intent", "overwrite")
+ metadata.setdefault("memory_gate_decision", "explicit_overwrite")
+ metadata["allow_parallel_state"] = False
+ record.metadata = metadata
+ return records
+
+
+def _records_from_extractor(
+ extractor: SessionMemoryExtractor,
+ *,
+ query: str,
+ answer_payload: Dict[str, Any] | None,
+ extraction_result: Dict[str, Any] | None,
+ turn_index: int,
+ profile: TMCRAProfile | None = None,
+) -> List[SessionMemoryRecordV2]:
+ profile = profile or TMCRAProfile()
+ raw_records = extractor.extract(
+ query=query,
+ extraction_result=extraction_result,
+ answer_bundle=answer_payload,
+ answer_mode=str((answer_payload or {}).get("answer_mode", "transparent")),
+ turn_index=turn_index,
+ )
+ results: List[SessionMemoryRecordV2] = []
+ for index, record in enumerate(raw_records):
+ slot_key = profile.stable_slot_key(
+ category=record.category,
+ value=record.value,
+ anchors=record.anchor_concepts,
+ slot_key=record.metadata.get("slot_key", "") if isinstance(record.metadata, dict) else "",
+ relation=record.relation,
+ metadata=dict(record.metadata or {}),
+ )
+ metadata = {
+ **dict(record.metadata or {}),
+ "memory_role": _clean_text(dict(record.metadata or {}).get("memory_role", "")) or "user",
+ "authority": _clean_text(dict(record.metadata or {}).get("authority", "")) or "source",
+ "canonical_slot_key": _clean_text(dict(record.metadata or {}).get("canonical_slot_key", "")) or slot_key,
+ "writeback_class": _clean_text(dict(record.metadata or {}).get("writeback_class", "")),
+ "origin_query": _clean_text(dict(record.metadata or {}).get("origin_query", "")) or _clean_text(query),
+ "origin_answer_id": _clean_text(dict(record.metadata or {}).get("origin_answer_id", "")),
+ "support_memory_ids": _dedupe(dict(record.metadata or {}).get("support_memory_ids", []) or []),
+ "support_fact_refs": _dedupe(dict(record.metadata or {}).get("support_fact_refs", []) or []),
+ "support_path_refs": _dedupe(dict(record.metadata or {}).get("support_path_refs", []) or []),
+ "promotion_state": _clean_text(dict(record.metadata or {}).get("promotion_state", "")) or "none",
+ }
+ results.append(
+ SessionMemoryRecordV2(
+ memory_id=f"{slot_key}:{turn_index}:auto:{index}",
+ category=record.category,
+ slot_key=slot_key,
+ value=_clean_text(record.value),
+ relation=_clean_text(record.relation) or f"{record.category}_memory",
+ anchor_concepts=_dedupe(record.anchor_concepts, max_items=8),
+ evidence_anchors=_dedupe(record.anchor_concepts, max_items=8),
+ salience=float(record.salience),
+ confidence=float(record.confidence),
+ source_kind=_clean_text(record.source_kind) or "session_memory",
+ turn_index=int(record.turn_index),
+ state="active",
+ metadata=metadata,
+ )
+ )
+ return results
+
+
+def _build_turn_records(
+ extractor: SessionMemoryExtractor,
+ *,
+ user_text: str,
+ answer_payload: Dict[str, Any] | None,
+ extraction_result: Dict[str, Any] | None,
+ turn_index: int,
+ allow_auto_extract: bool,
+ profile: TMCRAProfile | None = None,
+) -> List[SessionMemoryRecordV2]:
+ profile = profile or TMCRAProfile()
+ structured_records = _parse_structured_records(answer_payload, turn_index=turn_index, profile=profile)
+ if structured_records:
+ return _apply_typed_tunnel_annotations(
+ _apply_turn_write_intent(structured_records, user_text=user_text),
+ source_text=user_text,
+ )
+ structured_records = _parse_structured_records(extraction_result, turn_index=turn_index, profile=profile)
+ if structured_records:
+ return _apply_typed_tunnel_annotations(
+ _apply_turn_write_intent(structured_records, user_text=user_text),
+ source_text=user_text,
+ )
+ if not allow_auto_extract and not _looks_like_write_turn(user_text, answer_payload=answer_payload):
+ return []
+ return _apply_typed_tunnel_annotations(
+ _records_from_extractor(
+ extractor,
+ query=user_text,
+ answer_payload=None,
+ extraction_result=extraction_result,
+ turn_index=turn_index,
+ profile=profile,
+ ),
+ source_text=user_text,
+ )
+
+
+class NullMemoryAdapter(MemoryAdapter):
+ name = "null_memory"
+
+ def reset(self) -> None:
+ return None
+
+ def ingest_turn(
+ self,
+ user_text: str,
+ assistant_text: str = "",
+ *,
+ answer_payload: Dict[str, Any] | None = None,
+ extraction_result: Dict[str, Any] | None = None,
+ ) -> None:
+ _ = user_text, assistant_text, answer_payload, extraction_result
+
+ def retrieve(self, query: str, *, top_k: int = 6) -> MemoryRetrieval:
+ _ = query, top_k
+ return MemoryRetrieval()
+
+ def stats(self) -> Dict[str, Any]:
+ return _state_stats(storage_bytes=0, retrieval_context_tokens=0, total_state_tokens=0, records=0)
+
+ def storage_bytes(self) -> int:
+ return 0
+
+ def build_prompt_context(self, query: str, *, top_k: int = 8) -> Dict[str, Any]:
+ _ = top_k
+ return {
+ "mode": "null_memory",
+ "query": query,
+ "retrieval": MemoryRetrieval().to_dict(),
+ "stats": self.stats(),
+ "state": {},
+ }
+
+
+class GraphSessionMemoryAdapter(MemoryAdapter):
+ name = "graph_session_memory_v2"
+
+ def __init__(
+ self,
+ *,
+ auto_extract: bool = False,
+ storage_backend: str = "sqlite",
+ storage_path: str = "",
+ scope_id: str = "",
+ audit_retention: int = 256,
+ lightweight_stats: bool = True,
+ retrieval_mode: str = "heuristic",
+ node_model_path: str = "",
+ path_model_path: str = "",
+ node_model_device: str = "",
+ candidate_event_k: int = 24,
+ support_path_k: int = 3,
+ path_tunnel_rescue_k: int = 0,
+ path_tunnel_rescue_score_floor: float = 0.0,
+ path_tunnel_rescue_min_age: int = 0,
+ path_tunnel_rescue_min_score_margin: float = 0.0,
+ event_rerank_mode: str = "matrix",
+ matrix_event_top_k: int = DEFAULT_MATRIX_EVENT_TOP_K,
+ memory_router_mode: str = "",
+ memory_router_threshold: float = _MEMORY_ROUTER_DEFAULT_THRESHOLD,
+ memory_router_margin: float = _MEMORY_ROUTER_DEFAULT_MARGIN,
+ injection_planner_mode: str = "",
+ injection_planner_model_path: str = "",
+ injection_planner_latest_path: str = "",
+ injection_planner_device: str = "",
+ injection_planner_selection_threshold: float = -1.0,
+ injection_planner_row_threshold: float = -1.0,
+ injection_planner_logic_threshold: float = -1.0,
+ temporal_layer_mode: str = "",
+ temporal_router_mode: str = "",
+ temporal_router_dir: str = "",
+ temporal_router_latest_path: str = "",
+ temporal_router_device: str = "",
+ ) -> None:
+ prewarm_embedder_mode = (
+ _normalize(os.getenv("TMCRA_EMBEDDER_INDEX_RECALL_MODE", ""))
+ or _normalize(os.getenv("TMCRA_WRITE_EMBEDDER_INDEX_MODE", ""))
+ )
+ self._embedder_prewarm_metadata = _prewarm_embedder_dense_if_requested(mode=prewarm_embedder_mode)
+ self.extractor = SessionMemoryExtractor()
+ self.profile = TMCRAProfile()
+ self.temporal_organizer = TemporalOrganizer()
+ self.temporal_query_planner = TemporalQueryPlanner()
+ self.timeline_evidence_builder = TimelineEvidencePackBuilder()
+ self.auto_extract = bool(auto_extract)
+ self.storage_backend = _normalize(storage_backend) or "sqlite"
+ self.audit_retention = max(1, int(audit_retention))
+ self.lightweight_stats = bool(lightweight_stats)
+ self.scope_id = _clean_text(scope_id) or f"graph-session-{uuid.uuid4().hex}"
+ self._store: SQLiteSessionMemoryStore | None = None
+ self.storage_path = ""
+ if self.storage_backend == "sqlite":
+ resolved_storage_path = _clean_text(storage_path) or str((Path(tempfile.gettempdir()) / "tmcra_graph_session_memory.sqlite3").resolve())
+ self._store = SQLiteSessionMemoryStore(resolved_storage_path, audit_retention=self.audit_retention)
+ self.storage_path = str(self._store.storage_path)
+ self.graph = self._store.load_graph(self.scope_id)
+ elif self.storage_backend == "memory":
+ self.storage_path = _clean_text(storage_path)
+ self.graph = SessionMemoryGraphV2(
+ audit_retention=self.audit_retention,
+ persistence_backend="memory",
+ persistence_path=self.storage_path,
+ )
+ else:
+ raise ValueError(f"Unsupported storage backend: {self.storage_backend}")
+ self._last_retrieval_context_tokens = 0
+ self._last_writeback_summary: Dict[str, Any] = {}
+ self.retrieval_mode = _normalize(retrieval_mode) or "heuristic"
+ self.node_model_path = _clean_text(node_model_path)
+ self.path_model_path = _clean_text(path_model_path)
+ self.node_model_device = _clean_text(node_model_device)
+ self.candidate_event_k = max(1, int(candidate_event_k))
+ self.support_path_k = max(1, int(support_path_k))
+ self.path_tunnel_rescue_k = max(0, int(path_tunnel_rescue_k or 0))
+ self.path_tunnel_rescue_score_floor = max(0.0, float(path_tunnel_rescue_score_floor or 0.0))
+ self.path_tunnel_rescue_min_age = max(0, int(path_tunnel_rescue_min_age or 0))
+ self.path_tunnel_rescue_min_score_margin = max(0.0, float(path_tunnel_rescue_min_score_margin or 0.0))
+ self.event_rerank_mode = _normalize(event_rerank_mode) or "matrix"
+ self.matrix_event_top_k = max(1, int(matrix_event_top_k or DEFAULT_MATRIX_EVENT_TOP_K))
+ self.write_embedder_index_mode = _normalize(os.getenv("TMCRA_WRITE_EMBEDDER_INDEX_MODE", ""))
+ if not self.write_embedder_index_mode:
+ self.write_embedder_index_mode = "off"
+ try:
+ self.write_embedder_index_max_terms = max(
+ 8,
+ int(os.getenv("TMCRA_WRITE_EMBEDDER_INDEX_MAX_TERMS", "96") or 96),
+ )
+ except (TypeError, ValueError):
+ self.write_embedder_index_max_terms = 96
+ env_embedder_recall_mode = _normalize(os.getenv("TMCRA_EMBEDDER_INDEX_RECALL_MODE", ""))
+ self.embedder_index_recall_mode = env_embedder_recall_mode or self.write_embedder_index_mode
+ try:
+ self.embedder_index_recall_k = max(
+ 0,
+ int(os.getenv("TMCRA_EMBEDDER_INDEX_RECALL_K", "0") or 0),
+ )
+ except (TypeError, ValueError):
+ self.embedder_index_recall_k = 0
+ self.embedder_pre_recall_mode = _normalize(os.getenv("TMCRA_EMBEDDER_PRE_RECALL_MODE", ""))
+ if not self.embedder_pre_recall_mode:
+ self.embedder_pre_recall_mode = "off"
+ try:
+ self.embedder_pre_recall_k = max(
+ 0,
+ int(os.getenv("TMCRA_EMBEDDER_PRE_RECALL_K", "0") or 0),
+ )
+ except (TypeError, ValueError):
+ self.embedder_pre_recall_k = 0
+ self.embedder_fusion_mode = _normalize(os.getenv("TMCRA_EMBEDDER_FUSION_MODE", ""))
+ try:
+ self.embedder_fusion_weight = max(
+ 0.0,
+ float(os.getenv("TMCRA_EMBEDDER_FUSION_WEIGHT", "0.35") or 0.35),
+ )
+ except (TypeError, ValueError):
+ self.embedder_fusion_weight = 0.35
+ try:
+ self.embedder_fusion_score_floor = max(
+ 0.0,
+ float(os.getenv("TMCRA_EMBEDDER_FUSION_SCORE_FLOOR", "0.62") or 0.62),
+ )
+ except (TypeError, ValueError):
+ self.embedder_fusion_score_floor = 0.62
+ try:
+ self.embedder_fusion_top_k = max(
+ 0,
+ int(os.getenv("TMCRA_EMBEDDER_FUSION_TOP_K", "16") or 16),
+ )
+ except (TypeError, ValueError):
+ self.embedder_fusion_top_k = 16
+ try:
+ self.embedder_fusion_select_k = max(
+ 0,
+ int(os.getenv("TMCRA_EMBEDDER_FUSION_SELECT_K", "4") or 4),
+ )
+ except (TypeError, ValueError):
+ self.embedder_fusion_select_k = 4
+ try:
+ self.embedder_fusion_max_boost = max(
+ 0.0,
+ float(os.getenv("TMCRA_EMBEDDER_FUSION_MAX_BOOST", "0.42") or 0.42),
+ )
+ except (TypeError, ValueError):
+ self.embedder_fusion_max_boost = 0.42
+ env_router_mode = _clean_text(os.getenv("TMCRA_MEMORY_ROUTER_MODE", ""))
+ self.memory_router_mode = _normalize(memory_router_mode) or _normalize(env_router_mode) or "observe"
+ try:
+ self.memory_router_threshold = float(
+ os.getenv("TMCRA_MEMORY_ROUTER_THRESHOLD", "")
+ or memory_router_threshold
+ or _MEMORY_ROUTER_DEFAULT_THRESHOLD
+ )
+ except (TypeError, ValueError):
+ self.memory_router_threshold = _MEMORY_ROUTER_DEFAULT_THRESHOLD
+ try:
+ self.memory_router_margin = float(
+ os.getenv("TMCRA_MEMORY_ROUTER_MARGIN", "")
+ or memory_router_margin
+ or _MEMORY_ROUTER_DEFAULT_MARGIN
+ )
+ except (TypeError, ValueError):
+ self.memory_router_margin = _MEMORY_ROUTER_DEFAULT_MARGIN
+ self._loaded_node_scorer: LoadedNodeMemoryScorer | None = None
+ self._node_scorer_error = ""
+ env_planner_mode = _clean_text(os.getenv("TMCRA_INJECTION_PLANNER_MODE", ""))
+ self.injection_planner_mode = _normalize(injection_planner_mode) or _normalize(env_planner_mode) or "observe"
+ self.injection_planner_model_path = _clean_text(
+ injection_planner_model_path or os.getenv("TMCRA_INJECTION_PLANNER_MODEL_PATH", "")
+ )
+ self.injection_planner_latest_path = _clean_text(
+ injection_planner_latest_path or os.getenv("TMCRA_INJECTION_PLANNER_LATEST", "")
+ )
+ self.injection_planner_device = _clean_text(
+ injection_planner_device or os.getenv("TMCRA_INJECTION_PLANNER_DEVICE", "")
+ )
+ self.injection_planner_selection_threshold_override = float(injection_planner_selection_threshold)
+ self.injection_planner_row_threshold_override = float(injection_planner_row_threshold)
+ self.injection_planner_logic_threshold_override = float(injection_planner_logic_threshold)
+ self._loaded_injection_planner: Any | None = None
+ self._injection_planner_config: Any | None = None
+ self._injection_planner_payload: Dict[str, Any] = {}
+ self._injection_planner_thresholds: Dict[str, float] = {}
+ self._injection_planner_resolved_path = ""
+ self._injection_planner_error = ""
+ self._injection_planner_evidence_role_supported = False
+ env_temporal_layer_mode = _clean_text(os.getenv("TMCRA_TEMPORAL_LAYER_MODE", ""))
+ self.temporal_layer_mode = _normalize(temporal_layer_mode) or _normalize(env_temporal_layer_mode) or "observe"
+ env_temporal_router_mode = _clean_text(os.getenv("TMCRA_TEMPORAL_ROUTER_MODE", ""))
+ self.temporal_router_mode = _normalize(temporal_router_mode) or _normalize(env_temporal_router_mode) or "observe"
+ self.temporal_router_dir = _clean_text(
+ temporal_router_dir or os.getenv("TMCRA_TEMPORAL_ROUTER_DIR", "")
+ )
+ self.temporal_router_latest_path = _clean_text(
+ temporal_router_latest_path
+ or os.getenv("TMCRA_TEMPORAL_ROUTER_LATEST", "")
+ or "models/temporal_router_v1_latest.txt"
+ )
+ self.temporal_router_device = _clean_text(
+ temporal_router_device or os.getenv("TMCRA_TEMPORAL_ROUTER_DEVICE", "")
+ ) or "cpu"
+ self._loaded_temporal_router: Any | None = None
+ self._temporal_router_resolved_dir = ""
+ self._temporal_router_error = ""
+
+ def _empty_graph(self) -> SessionMemoryGraphV2:
+ return SessionMemoryGraphV2(
+ audit_retention=self.audit_retention,
+ persistence_backend=self.storage_backend,
+ persistence_path=self.storage_path,
+ )
+
+ def _reload_graph(self) -> None:
+ if self._store is not None:
+ self.graph = self._store.load_graph(self.scope_id)
+ else:
+ self.graph.configure_persistence(
+ backend=self.storage_backend,
+ path=self.storage_path,
+ audit_retention=self.audit_retention,
+ )
+
+ def _persist_graph(self) -> None:
+ self.graph.configure_persistence(
+ backend=self.storage_backend,
+ path=self.storage_path,
+ audit_retention=self.audit_retention,
+ )
+ if self._store is not None:
+ self._store.save_graph(self.scope_id, self.graph)
+
+ def replace_graph(self, graph: SessionMemoryGraphV2) -> None:
+ self.graph = graph
+ self._persist_graph()
+
+ def _storage_breakdown(self) -> Dict[str, int]:
+ core_payload = self.graph._core_payload()
+ audit_payload_full = self.graph._audit_payload()
+ if self.lightweight_stats:
+ audit_token_payload = {
+ "totals": dict(self.graph.audit_event_totals),
+ "trimmed": dict(self.graph.audit_trimmed_counts),
+ "retained": {
+ "turn_log": len(self.graph.turn_log),
+ "retrieval_log": len(self.graph.retrieval_log),
+ "answer_support_log": len(self.graph.answer_support_log),
+ },
+ "audit_retention": int(self.graph.audit_retention),
+ }
+ else:
+ audit_token_payload = audit_payload_full
+ core_storage_bytes = len(json.dumps(core_payload, ensure_ascii=False).encode("utf-8"))
+ audit_storage_bytes = len(json.dumps(audit_payload_full, ensure_ascii=False).encode("utf-8"))
+ core_state_token_estimate = _estimate_tokens(json.dumps(core_payload, ensure_ascii=False))
+ audit_state_token_estimate = _estimate_tokens(json.dumps(audit_token_payload, ensure_ascii=False))
+ return {
+ "core_storage_bytes": int(core_storage_bytes),
+ "audit_storage_bytes": int(audit_storage_bytes),
+ "storage_bytes": int(core_storage_bytes + audit_storage_bytes),
+ "core_state_token_estimate": int(core_state_token_estimate),
+ "audit_state_token_estimate": int(audit_state_token_estimate),
+ "total_state_token_estimate": int(core_state_token_estimate + audit_state_token_estimate),
+ }
+
+ def reset(self) -> None:
+ if self._store is not None:
+ self._store.clear_scope(self.scope_id)
+ self.graph = self._store.load_graph(self.scope_id)
+ else:
+ self.graph = self._empty_graph()
+ self._last_retrieval_context_tokens = 0
+ self._last_writeback_summary = {}
+
+ def _temporal_layer_enabled(self) -> bool:
+ return _normalize(self.temporal_layer_mode) not in _TEMPORAL_LAYER_DISABLED_MODES
+
+ def _temporal_router_enabled(self) -> bool:
+ return self._temporal_layer_enabled() and _normalize(self.temporal_router_mode) not in _TEMPORAL_LAYER_DISABLED_MODES
+
+ def _resolve_temporal_router_dir(self) -> str:
+ direct_dir = _clean_text(self.temporal_router_dir)
+ if direct_dir:
+ root = Path(direct_dir)
+ if (root / "writer_temporal_router.pt").exists() and (root / "query_temporal_router.pt").exists():
+ return str(root)
+ latest_path = _clean_text(self.temporal_router_latest_path)
+ if latest_path:
+ pointer = Path(latest_path)
+ if pointer.exists():
+ lines = pointer.read_text(encoding="utf-8").splitlines()
+ candidate = _clean_text(lines[0] if lines else "")
+ if candidate:
+ root = Path(candidate)
+ if (root / "writer_temporal_router.pt").exists() and (root / "query_temporal_router.pt").exists():
+ return str(root)
+ return ""
+
+ def _load_temporal_router(self) -> LoadedTemporalRouter | None:
+ if not self._temporal_router_enabled():
+ self._temporal_router_error = "disabled"
+ return None
+ model_dir = self._resolve_temporal_router_dir()
+ self._temporal_router_resolved_dir = model_dir
+ if not model_dir:
+ self._temporal_router_error = "model_dir_missing"
+ return None
+ if self._loaded_temporal_router is not None:
+ return self._loaded_temporal_router
+ try:
+ self._loaded_temporal_router = LoadedTemporalRouter.from_dir(
+ model_dir,
+ device=self.temporal_router_device or "cpu",
+ )
+ self._temporal_router_error = ""
+ except Exception as exc: # pragma: no cover - defensive runtime path
+ self._loaded_temporal_router = None
+ self._temporal_router_error = f"{type(exc).__name__}: {exc}"
+ return self._loaded_temporal_router
+
+ def _temporal_router_status_metadata(self) -> Dict[str, Any]:
+ router = self._load_temporal_router()
+ return {
+ "temporal_router_enabled": router is not None,
+ "temporal_router_mode": _normalize(self.temporal_router_mode) or "observe",
+ "temporal_router_model_dir": self._temporal_router_resolved_dir,
+ "temporal_router_error": self._temporal_router_error,
+ }
+
+ def _session_timestamp_from_payloads(self, *payloads: Mapping[str, Any] | None) -> str:
+ for payload in payloads:
+ data = dict(payload or {})
+ for key in ("session_timestamp", "timestamp", "created_at", "turn_timestamp"):
+ value = _clean_text(data.get(key, ""))
+ if value:
+ return value
+ metadata = data.get("metadata")
+ if isinstance(metadata, Mapping):
+ for key in ("session_timestamp", "timestamp", "created_at", "turn_timestamp"):
+ value = _clean_text(metadata.get(key, ""))
+ if value:
+ return value
+ return ""
+
+ def _temporal_frame_for_turn(
+ self,
+ *,
+ user_text: str,
+ previous_turn: str = "",
+ answer_payload: Dict[str, Any] | None = None,
+ extraction_result: Dict[str, Any] | None = None,
+ ) -> TemporalFrame | None:
+ if not self._temporal_layer_enabled():
+ return None
+ session_timestamp = self._session_timestamp_from_payloads(answer_payload, extraction_result)
+ model_frame = None
+ for payload in (answer_payload, extraction_result):
+ data = dict(payload or {})
+ candidate = data.get("temporal_frame") or dict(data.get("metadata", {}) or {}).get("temporal_frame")
+ if isinstance(candidate, Mapping):
+ model_frame = candidate
+ break
+ fallback_frame = self.temporal_organizer.organize_turn(
+ current_turn=user_text,
+ previous_turn=previous_turn,
+ session_timestamp=session_timestamp,
+ speaker="user",
+ )
+ if model_frame is None:
+ router = self._load_temporal_router()
+ if router is not None and router.writer_available():
+ predicted = router.predict_writer_frame(
+ current_turn=user_text,
+ previous_turn=previous_turn,
+ session_timestamp=session_timestamp,
+ )
+ writer_confidence = float(predicted.get("confidence", 0.0) or 0.0) if predicted else 0.0
+ if predicted and writer_confidence >= _float_env("TMCRA_TEMPORAL_ROUTER_WRITER_MIN_CONFIDENCE", _TEMPORAL_ROUTER_DEFAULT_WRITER_MIN_CONFIDENCE):
+ frame_payload = fallback_frame.to_dict()
+ frame_payload.update(
+ {
+ key: value
+ for key, value in predicted.items()
+ if key in {"temporal_intent", "anchor_type", "granularity", "state_operation"}
+ and _clean_text(value)
+ }
+ )
+ if "should_create_timeline_edge" in predicted:
+ frame_payload["should_create_timeline_edge"] = bool(predicted.get("should_create_timeline_edge", False))
+ if writer_confidence > 0.0:
+ frame_payload["confidence"] = writer_confidence
+ frame_payload["metadata"] = {
+ **dict(frame_payload.get("metadata", {}) or {}),
+ **dict(predicted.get("metadata", {}) or {}),
+ **self._temporal_router_status_metadata(),
+ }
+ model_frame = frame_payload
+ if model_frame is None:
+ return fallback_frame
+ return self.temporal_organizer.organize_turn(
+ current_turn=user_text,
+ previous_turn=previous_turn,
+ session_timestamp=session_timestamp,
+ speaker="user",
+ model_frame=model_frame,
+ )
+
+ def _temporal_turn_metadata(self, frame: TemporalFrame | None) -> Dict[str, Any]:
+ if frame is None:
+ return {
+ "temporal_layer_enabled": False,
+ "temporal_layer_mode": _normalize(self.temporal_layer_mode) or "observe",
+ **self._temporal_router_status_metadata(),
+ }
+ return {
+ "temporal_layer_enabled": True,
+ "temporal_layer_mode": _normalize(self.temporal_layer_mode) or "observe",
+ "temporal_frame": frame.to_dict(),
+ "temporal_intent": frame.temporal_intent,
+ "temporal_subject_key": frame.subject_key,
+ "temporal_state_operation": frame.state_operation,
+ **self._temporal_router_status_metadata(),
+ }
+
+ def _apply_temporal_frame_to_records(self, records: Sequence[SessionMemoryRecordV2], frame: TemporalFrame | None) -> None:
+ if frame is None:
+ return
+ if frame.temporal_intent == "non_temporal" and not frame.subject_key:
+ return
+ for record in records:
+ metadata = dict(record.metadata or {})
+ metadata.update(
+ {
+ "temporal_layer": True,
+ "temporal_frame": frame.to_dict(),
+ "temporal_intent": frame.temporal_intent,
+ "temporal_subject": frame.subject,
+ "temporal_subject_key": frame.subject_key,
+ "temporal_state_operation": frame.state_operation,
+ "temporal_event_time": frame.event_time,
+ "temporal_state_valid_from": frame.state_valid_from,
+ "temporal_state_valid_to": frame.state_valid_to,
+ }
+ )
+ record.metadata = metadata
+
+ def _build_timeline_state_layer(self) -> TimelineStateLayer:
+ layer = TimelineStateLayer()
+ for turn in sorted(list(getattr(self.graph, "turn_log", []) or []), key=lambda item: int(item.get("turn_index", 0) or 0)):
+ metadata = dict(turn.get("metadata", {}) or {})
+ frame_payload = metadata.get("temporal_frame")
+ if not isinstance(frame_payload, Mapping):
+ continue
+ frame = TemporalFrame.from_mapping(frame_payload)
+ if frame.temporal_intent == "non_temporal" and not frame.new_state:
+ continue
+ record_ids = [item for item in list(turn.get("record_ids", []) or []) if _clean_text(item)]
+ source_event_id = _clean_text(metadata.get("temporal_source_record_id", "")) or (record_ids[0] if record_ids else "")
+ frame.metadata = {
+ **dict(frame.metadata or {}),
+ "source_text": _clean_text(turn.get("text", "")) or frame.evidence_span,
+ }
+ layer.apply_frame(
+ frame,
+ source_event_id=source_event_id,
+ source_turn_id=_clean_text(turn.get("turn_id", "")),
+ state_type=_clean_text(metadata.get("temporal_state_type", "")) or "profile",
+ )
+ return layer
+
+ def _temporal_runtime_pack(self, query: str) -> Dict[str, Any]:
+ metadata: Dict[str, Any] = {
+ "temporal_runtime_enabled": False,
+ "temporal_layer_mode": _normalize(self.temporal_layer_mode) or "observe",
+ **self._temporal_router_status_metadata(),
+ }
+ if not self._temporal_layer_enabled():
+ metadata["temporal_runtime_reason"] = "disabled"
+ return {"metadata": metadata}
+ plan = self.temporal_query_planner.plan(query)
+ router = self._load_temporal_router()
+ if router is not None and router.query_available():
+ predicted_plan = router.predict_query_plan(query=query)
+ query_confidence = float(predicted_plan.get("confidence", 0.0) or 0.0) if predicted_plan else 0.0
+ router_confidences = dict(dict(predicted_plan.get("metadata", {}) or {}).get("temporal_router_confidences", {}) or {}) if predicted_plan else {}
+ intent_confidence = float(router_confidences.get("query_temporal_intent", 0.0) or 0.0)
+ if (
+ predicted_plan
+ and query_confidence >= _float_env("TMCRA_TEMPORAL_ROUTER_QUERY_MIN_CONFIDENCE", _TEMPORAL_ROUTER_DEFAULT_QUERY_MIN_CONFIDENCE)
+ and intent_confidence >= _float_env("TMCRA_TEMPORAL_ROUTER_QUERY_INTENT_MIN_CONFIDENCE", _TEMPORAL_ROUTER_DEFAULT_QUERY_INTENT_MIN_CONFIDENCE)
+ ):
+ plan_payload = plan.to_dict()
+ plan_payload.update(
+ {
+ key: value
+ for key, value in predicted_plan.items()
+ if key in {"query_temporal_intent", "timeline_operation"} and _clean_text(value)
+ }
+ )
+ for key in ("prefer_current_state", "prefer_previous_state", "requires_ordered_chain", "requires_comparison"):
+ if key in predicted_plan:
+ plan_payload[key] = bool(predicted_plan.get(key, False))
+ if float(predicted_plan.get("confidence", 0.0) or 0.0) > 0.0:
+ plan_payload["confidence"] = float(predicted_plan.get("confidence", 0.0) or 0.0)
+ plan_payload["metadata"] = {
+ **dict(plan_payload.get("metadata", {}) or {}),
+ **dict(predicted_plan.get("metadata", {}) or {}),
+ **self._temporal_router_status_metadata(),
+ }
+ plan = TemporalQueryPlan(**{key: value for key, value in plan_payload.items() if key in TemporalQueryPlan.__dataclass_fields__})
+ metadata["temporal_query_plan"] = plan.to_dict()
+ if plan.query_temporal_intent == "non_temporal" or plan.timeline_operation == "none":
+ metadata["temporal_runtime_reason"] = "non_temporal_query"
+ return {"plan": plan, "metadata": metadata}
+ if plan.timeline_operation in {"query_current", "query_previous"} and not _clean_text(plan.target_subject_key):
+ metadata["temporal_runtime_reason"] = "missing_target_subject"
+ return {"plan": plan, "metadata": metadata}
+ timeline_layer = self._build_timeline_state_layer()
+ pack = self.timeline_evidence_builder.build(plan=plan, timeline_layer=timeline_layer)
+ metadata.update(
+ {
+ "temporal_runtime_enabled": True,
+ "temporal_runtime_reason": "ok",
+ "temporal_evidence_pack": pack.to_dict(),
+ "temporal_selected_answer_value": _clean_text(pack.selected_evidence.get("answer_value", "")),
+ "temporal_selected_state_id": _clean_text(pack.selected_evidence.get("state_id", "")),
+ "temporal_timeline_state_count": len(pack.timeline),
+ }
+ )
+ return {"plan": plan, "pack": pack, "metadata": metadata}
+
+ def _temporal_state_hit(
+ self,
+ *,
+ state_payload: Mapping[str, Any],
+ plan_payload: Mapping[str, Any],
+ selected: bool,
+ rank: int,
+ ) -> MemoryHit | None:
+ state_value = _clean_text(state_payload.get("state", ""))
+ state_id = _clean_text(state_payload.get("state_id", ""))
+ if not state_value or not state_id:
+ return None
+ source_event_id = _clean_text(state_payload.get("source_event_id", ""))
+ source_record = self.graph.records_by_id.get(source_event_id)
+ source_metadata = dict(source_record.metadata or {}) if source_record is not None else {}
+ score = 0.98 if selected else max(0.75, 0.92 - (rank * 0.03))
+ return MemoryHit(
+ memory_id=f"temporal_state:{state_id}",
+ category="time",
+ value=state_value,
+ relation="temporal_state",
+ anchors=_dedupe([state_payload.get("time", ""), state_payload.get("source_text", ""), plan_payload.get("target_subject", "")], max_items=6),
+ score=round(score, 6),
+ source_kind="temporal_state_layer",
+ slot_key=f"temporal.{_clean_text(plan_payload.get('target_subject_key', 'general'))}",
+ state="active" if bool(state_payload.get("is_current", False)) else "history",
+ turn_index=int(source_record.turn_index) if source_record is not None else 0,
+ metadata={
+ **source_metadata,
+ "temporal_runtime_hit": True,
+ "temporal_runtime_selected": bool(selected),
+ "temporal_state_id": state_id,
+ "temporal_state_value": state_value,
+ "temporal_state_time": _clean_text(state_payload.get("time", "")),
+ "temporal_state_valid_to": _clean_text(state_payload.get("valid_to", "")),
+ "temporal_state_is_current": bool(state_payload.get("is_current", False)),
+ "temporal_source_event_id": source_event_id,
+ "temporal_source_turn_id": _clean_text(state_payload.get("source_turn_id", "")),
+ "temporal_query_plan": dict(plan_payload),
+ },
+ )
+
+ def _apply_temporal_evidence_pack_to_hits(
+ self,
+ hits: Sequence[MemoryHit],
+ temporal_payload: Mapping[str, Any],
+ *,
+ top_k: int,
+ ) -> Dict[str, Any]:
+ metadata = dict(temporal_payload.get("metadata", {}) or {})
+ pack = temporal_payload.get("pack")
+ plan = temporal_payload.get("plan")
+ if pack is None or plan is None or not bool(metadata.get("temporal_runtime_enabled", False)):
+ return {"hits": list(hits), "metadata": metadata}
+ pack_payload = pack.to_dict() if hasattr(pack, "to_dict") else dict(pack)
+ plan_payload = plan.to_dict() if hasattr(plan, "to_dict") else dict(plan)
+ selected_state_id = _clean_text(dict(pack_payload.get("selected_evidence", {}) or {}).get("state_id", ""))
+ selected_answer_value = _clean_text(dict(pack_payload.get("selected_evidence", {}) or {}).get("answer_value", ""))
+ timeline = list(pack_payload.get("timeline", []) or [])
+ synthetic_hits: List[MemoryHit] = []
+ if selected_state_id:
+ selected_state = next((dict(item) for item in timeline if _clean_text(dict(item).get("state_id", "")) == selected_state_id), None)
+ if selected_state is None and selected_answer_value:
+ selected_state = {
+ "state_id": selected_state_id,
+ "state": selected_answer_value,
+ "source_event_id": dict(pack_payload.get("selected_evidence", {}) or {}).get("source_event_id", ""),
+ "source_turn_id": dict(pack_payload.get("selected_evidence", {}) or {}).get("source_turn_id", ""),
+ "is_current": bool(plan_payload.get("prefer_current_state", False)),
+ }
+ if selected_state:
+ hit = self._temporal_state_hit(state_payload=selected_state, plan_payload=plan_payload, selected=True, rank=0)
+ if hit is not None:
+ synthetic_hits.append(hit)
+ if bool(plan_payload.get("requires_ordered_chain", False)) or bool(plan_payload.get("requires_comparison", False)):
+ for index, state_payload in enumerate(timeline):
+ if _clean_text(dict(state_payload).get("state_id", "")) == selected_state_id:
+ continue
+ hit = self._temporal_state_hit(
+ state_payload=dict(state_payload),
+ plan_payload=plan_payload,
+ selected=False,
+ rank=index + 1,
+ )
+ if hit is not None:
+ synthetic_hits.append(hit)
+ existing: List[MemoryHit] = []
+ synthetic_source_ids = {
+ _clean_text((hit.metadata or {}).get("temporal_source_event_id", ""))
+ for hit in synthetic_hits
+ if _clean_text((hit.metadata or {}).get("temporal_source_event_id", ""))
+ }
+ for hit in hits:
+ hit_metadata = dict(hit.metadata or {})
+ source_event_id = _clean_text(hit_metadata.get("event_id", hit.memory_id))
+ if source_event_id in synthetic_source_ids:
+ hit_metadata.update(
+ {
+ "temporal_runtime_support": True,
+ "temporal_query_plan": dict(plan_payload),
+ "temporal_selected_answer_value": selected_answer_value,
+ "temporal_selected_state_id": selected_state_id,
+ }
+ )
+ hit = MemoryHit(
+ memory_id=hit.memory_id,
+ category=hit.category,
+ value=hit.value,
+ relation=hit.relation,
+ anchors=list(hit.anchors),
+ score=hit.score,
+ source_kind=hit.source_kind,
+ slot_key=hit.slot_key,
+ state=hit.state,
+ turn_index=int(hit.turn_index),
+ metadata=hit_metadata,
+ )
+ existing.append(hit)
+ merged: List[MemoryHit] = []
+ seen = set()
+ for hit in [*synthetic_hits, *existing]:
+ key = _clean_text(hit.memory_id)
+ if key and key in seen:
+ continue
+ if key:
+ seen.add(key)
+ merged.append(hit)
+ metadata.update(
+ {
+ "temporal_runtime_injected_hit_count": len(synthetic_hits),
+ "temporal_runtime_injected_hit_ids": [hit.memory_id for hit in synthetic_hits],
+ }
+ )
+ return {"hits": merged[: max(len(merged), top_k)], "metadata": metadata}
+
+ def ingest_turn(
+ self,
+ user_text: str,
+ assistant_text: str = "",
+ *,
+ answer_payload: Dict[str, Any] | None = None,
+ extraction_result: Dict[str, Any] | None = None,
+ ) -> None:
+ self._reload_graph()
+ turn_index = self.graph.next_turn()
+ previous_topic = _last_topic_turn(self.graph)
+ previous_turn_text = _clean_text(self.graph.turn_log[-1].get("text", "")) if self.graph.turn_log else ""
+ temporal_frame = self._temporal_frame_for_turn(
+ user_text=user_text,
+ previous_turn=previous_turn_text,
+ answer_payload=answer_payload,
+ extraction_result=extraction_result,
+ )
+ topic_bucket = _assign_topic_bucket_for_text(
+ self.graph,
+ user_text,
+ answer_payload=answer_payload,
+ turn_index=turn_index,
+ create=True,
+ )
+ records = _build_turn_records(
+ self.extractor,
+ user_text=user_text,
+ answer_payload=answer_payload,
+ extraction_result=extraction_result,
+ turn_index=turn_index,
+ allow_auto_extract=self.auto_extract,
+ profile=self.profile,
+ )
+ self._apply_temporal_frame_to_records(records, temporal_frame)
+ _apply_topic_bucket_to_records(records, topic_bucket)
+ stored_ids = self.graph.add_records(records)
+ write_embedder_metadata = _apply_write_embedder_index_to_graph(
+ self.graph,
+ stored_ids=stored_ids,
+ turn_text=user_text,
+ turn_index=turn_index,
+ mode=self.write_embedder_index_mode,
+ max_terms=self.write_embedder_index_max_terms,
+ )
+ topic_bridge_metadata = _add_topic_bridge_edges(
+ self.graph,
+ previous_topic=previous_topic,
+ current_topic=topic_bucket,
+ current_record_ids=stored_ids,
+ turn_index=turn_index,
+ evidence=user_text,
+ )
+ dialogue_tunnel_metadata = _add_dialogue_tunnel_edges(
+ self.graph,
+ current_topic=topic_bucket,
+ current_record_ids=stored_ids,
+ turn_index=turn_index,
+ evidence=user_text,
+ )
+ turn_kind = "memory_write" if stored_ids else "noise"
+ self.graph.record_turn(
+ turn_kind=turn_kind,
+ text=user_text,
+ turn_index=turn_index,
+ record_ids=stored_ids,
+ speaker="user",
+ assistant_text=assistant_text,
+ metadata={
+ "source": "user_turn",
+ "auto_extract": bool(self.auto_extract),
+ "topic_bucket_id": _clean_text(topic_bucket.get("topic_bucket_id", "")),
+ "topic_label": _clean_text(topic_bucket.get("topic_label", "")),
+ "topic_keywords": list(topic_bucket.get("topic_keywords", []) or []),
+ "temporal_source_record_id": stored_ids[0] if stored_ids else "",
+ **self._temporal_turn_metadata(temporal_frame),
+ **write_embedder_metadata,
+ **topic_bridge_metadata,
+ **dialogue_tunnel_metadata,
+ },
+ )
+ self._persist_graph()
+
+ def ingest_answer_writeback(
+ self,
+ *,
+ query_text: str,
+ answer_text: str,
+ answer_id: str,
+ writeback_records: List[Dict[str, Any]],
+ trace: Dict[str, Any] | None = None,
+ ) -> List[str]:
+ self._reload_graph()
+ if not writeback_records:
+ self._last_writeback_summary = {"stored_record_ids": [], "promotion_events": []}
+ return []
+ turn_index = self.graph.next_turn()
+ records: List[SessionMemoryRecordV2] = []
+ writeback_classes: List[str] = []
+ for index, raw in enumerate(writeback_records):
+ if not isinstance(raw, dict):
+ continue
+ category = _clean_text(raw.get("category", "fact")) or "fact"
+ value = _clean_text(raw.get("value", ""))
+ raw_slot_key = _clean_text(raw.get("slot_key", "")) or _clean_text(raw.get("slot", ""))
+ anchors = _dedupe(raw.get("anchors", []) or [], max_items=8)
+ slot_key = self.profile.stable_slot_key(
+ category=category,
+ value=value,
+ anchors=anchors,
+ slot_key=raw_slot_key,
+ relation=_clean_text(raw.get("relation", "")),
+ metadata=dict(raw.get("metadata", {}) or {}),
+ )
+ if not value or not slot_key:
+ continue
+ raw_metadata = dict(raw.get("metadata", {}) or {})
+ writeback_class = _clean_text(raw_metadata.get("writeback_class", "")) or "fact"
+ writeback_classes.append(writeback_class)
+ metadata = {
+ **raw_metadata,
+ "memory_role": _clean_text(raw_metadata.get("memory_role", "")) or "assistant",
+ "authority": _clean_text(raw_metadata.get("authority", "")) or "derived",
+ "canonical_slot_key": _clean_text(raw_metadata.get("canonical_slot_key", "")) or slot_key.removeprefix("assistant.").split(".fact", 1)[0].split(".state_change", 1)[0].split(".high_conf_conclusion", 1)[0],
+ "writeback_class": writeback_class,
+ "origin_query": _clean_text(raw_metadata.get("origin_query", "")) or _clean_text(query_text),
+ "origin_answer_id": _clean_text(raw_metadata.get("origin_answer_id", "")) or answer_id,
+ "origin_answer_ids": _dedupe([*(raw_metadata.get("origin_answer_ids", []) or []), _clean_text(raw_metadata.get("origin_answer_id", "")) or answer_id]),
+ "support_memory_ids": _dedupe(raw_metadata.get("support_memory_ids", []) or []),
+ "support_fact_refs": _dedupe(raw_metadata.get("support_fact_refs", []) or []),
+ "support_path_refs": _dedupe(raw_metadata.get("support_path_refs", []) or []),
+ "promotion_state": _clean_text(raw_metadata.get("promotion_state", "")) or "candidate",
+ "answer_id": answer_id,
+ }
+ record = SessionMemoryRecordV2(
+ memory_id=f"{slot_key}:{turn_index}:assistant:{index}",
+ category=category,
+ slot_key=slot_key,
+ value=value,
+ relation=_clean_text(raw.get("relation", "")) or "assistant_memory",
+ anchor_concepts=anchors,
+ evidence_anchors=anchors,
+ salience=float(raw.get("salience", 0.62) or 0.62),
+ confidence=float(raw.get("confidence", 0.88) or 0.88),
+ source_kind=_clean_text(raw.get("source_kind", "")) or "assistant_writeback",
+ turn_index=turn_index,
+ state=_clean_text(raw.get("state", "")) or "active",
+ metadata=metadata,
+ )
+ records.append(record)
+ stored_ids = self.graph.add_records(records)
+ write_embedder_metadata = _apply_write_embedder_index_to_graph(
+ self.graph,
+ stored_ids=stored_ids,
+ turn_text=" ".join([_clean_text(query_text), _clean_text(answer_text)]),
+ turn_index=turn_index,
+ mode=self.write_embedder_index_mode,
+ max_terms=self.write_embedder_index_max_terms,
+ )
+ promotion_events = self._apply_writeback_promotions(stored_ids)
+ writeback_class = writeback_classes[0] if len(set(writeback_classes)) == 1 and writeback_classes else ("mixed" if writeback_classes else "")
+ self.graph.record_turn(
+ turn_kind="assistant_writeback" if stored_ids else "assistant_writeback_empty",
+ text=query_text,
+ turn_index=turn_index,
+ record_ids=stored_ids,
+ speaker="assistant",
+ assistant_text=answer_text,
+ writeback_class=writeback_class,
+ metadata={
+ "source": "assistant_writeback",
+ "answer_id": answer_id,
+ "trace": dict(trace or {}),
+ **write_embedder_metadata,
+ },
+ )
+ self._last_writeback_summary = {
+ "stored_record_ids": list(stored_ids),
+ "promotion_events": list(promotion_events),
+ **write_embedder_metadata,
+ }
+ self._persist_graph()
+ return stored_ids
+
+ def last_writeback_summary(self) -> Dict[str, Any]:
+ return dict(self._last_writeback_summary)
+
+ def _resolve_injection_planner_model_path(self) -> str:
+ explicit_path = _clean_text(self.injection_planner_model_path)
+ if explicit_path:
+ path = Path(explicit_path).expanduser()
+ if path.is_dir():
+ path = path / "injection_planner.pt"
+ return str(path)
+ latest_path = _clean_text(self.injection_planner_latest_path)
+ if not latest_path:
+ return ""
+ latest = Path(latest_path).expanduser()
+ if latest.is_dir():
+ return str(latest / "injection_planner.pt")
+ if latest.suffix == ".pt":
+ return str(latest)
+ try:
+ target = Path(latest.read_text(encoding="utf-8").strip()).expanduser()
+ except Exception as exc:
+ self._injection_planner_error = f"latest_pointer_read_failed: {exc}"
+ return ""
+ if target.is_dir():
+ target = target / "injection_planner.pt"
+ return str(target)
+
+ def _load_injection_planner(self) -> Any | None:
+ normalized_mode = _normalize(self.injection_planner_mode)
+ if normalized_mode in _INJECTION_PLANNER_DISABLED_MODES:
+ self._injection_planner_error = "disabled"
+ return None
+ if self._loaded_injection_planner is not None:
+ return self._loaded_injection_planner
+ torch_module = getattr(injection_planner_runtime, "torch", None)
+ if torch_module is None:
+ self._injection_planner_error = "torch_unavailable"
+ return None
+ model_path_text = self._resolve_injection_planner_model_path()
+ if not model_path_text:
+ if not self._injection_planner_error:
+ self._injection_planner_error = "model_path_missing"
+ return None
+ model_path = Path(model_path_text)
+ if not model_path.exists():
+ self._injection_planner_error = f"model_path_not_found: {model_path}"
+ return None
+ try:
+ device = torch_module.device(self.injection_planner_device or "cpu")
+ payload = torch_module.load(model_path, map_location=device, weights_only=True)
+ config = injection_planner_runtime.InjectionPlannerConfig.from_dict(dict(payload.get("config", {}) or {}))
+ model = injection_planner_runtime.InjectionPlannerModel(config).to(device)
+ state_dict = dict(payload.get("state_dict", {}) or {})
+ model.load_state_dict(state_dict, strict=False)
+ model.eval()
+ self._loaded_injection_planner = model
+ self._injection_planner_config = config
+ self._injection_planner_payload = dict(payload)
+ self._injection_planner_resolved_path = str(model_path)
+ self._injection_planner_error = ""
+ self._injection_planner_evidence_role_supported = any(
+ str(key).startswith("evidence_role_head.") for key in state_dict
+ )
+ thresholds = {"selection_threshold": 0.5, "row_threshold": 0.5, "logic_threshold": 0.5}
+ summary_path = model_path.parent / "train_summary.json"
+ if summary_path.exists():
+ try:
+ summary = json.loads(summary_path.read_text(encoding="utf-8"))
+ calibration = dict(summary.get("calibration", {}) or {})
+ logic_calibration = dict(summary.get("logic_calibration", {}) or {})
+ if calibration.get("selection_threshold") is not None:
+ thresholds["selection_threshold"] = float(calibration.get("selection_threshold"))
+ if calibration.get("row_threshold") is not None:
+ thresholds["row_threshold"] = float(calibration.get("row_threshold"))
+ if logic_calibration.get("logic_threshold") is not None:
+ thresholds["logic_threshold"] = float(logic_calibration.get("logic_threshold"))
+ except Exception:
+ pass
+ if self.injection_planner_selection_threshold_override >= 0.0:
+ thresholds["selection_threshold"] = float(self.injection_planner_selection_threshold_override)
+ if self.injection_planner_row_threshold_override >= 0.0:
+ thresholds["row_threshold"] = float(self.injection_planner_row_threshold_override)
+ if self.injection_planner_logic_threshold_override >= 0.0:
+ thresholds["logic_threshold"] = float(self.injection_planner_logic_threshold_override)
+ self._injection_planner_thresholds = thresholds
+ except Exception as exc:
+ self._loaded_injection_planner = None
+ self._injection_planner_config = None
+ self._injection_planner_payload = {}
+ self._injection_planner_resolved_path = str(model_path)
+ self._injection_planner_error = str(exc)
+ self._injection_planner_evidence_role_supported = False
+ return self._loaded_injection_planner
+
+ def _injection_planner_candidate_from_hit(
+ self,
+ query: str,
+ hit: MemoryHit,
+ *,
+ index: int,
+ current_turn_index: int,
+ ) -> Dict[str, Any]:
+ metadata = dict(hit.metadata or {})
+ category = _normalize(hit.category)
+ source_kind = _normalize(hit.source_kind)
+ logic_roles = [
+ _normalize(item)
+ for item in (
+ metadata.get("logic_roles", [])
+ if isinstance(metadata.get("logic_roles", []), list)
+ else [metadata.get("logic_roles", "")]
+ )
+ if _normalize(item) in set(injection_planner_runtime.LOGIC_ROLES)
+ ]
+ if category in set(injection_planner_runtime.LOGIC_ROLES):
+ logic_roles.append(category)
+ if _normalize(metadata.get("writeback_class", "")) in set(injection_planner_runtime.LOGIC_ROLES):
+ logic_roles.append(_normalize(metadata.get("writeback_class", "")))
+ if not logic_roles:
+ logic_roles = ["negative"] if _normalize(hit.state) in {"stale", "superseded", "false"} else ["evidence"]
+ evidence_snippet_role = _normalize(metadata.get("evidence_snippet_role", ""))
+ if bool(metadata.get("profile_layer")) or category == "profile" or source_kind.endswith("profile"):
+ layer = "profile"
+ elif category == "time" or float(metadata.get("temporal_score", 0.0) or 0.0) > 0.0:
+ layer = "temporal"
+ elif "resource" in logic_roles or _clean_text(metadata.get("resource_key", "")):
+ layer = "resource"
+ elif (
+ source_kind in {"path_tunnel", "path_support", "public_dialog_path"}
+ or evidence_snippet_role in {"selected_path_support", "path_tunnel_support"}
+ or bool(metadata.get("path_tunnel_node"))
+ ):
+ layer = "path_tunnel"
+ elif bool(metadata.get("topic_bucket_rerank")) or bool(metadata.get("topic_bucket_dialogue_tunnel_allowed")):
+ layer = "topic_tunnel"
+ else:
+ layer = "event"
+ normalized_state = _normalize(hit.state)
+ if normalized_state in {"stale", "superseded", "false"}:
+ temporal_state = "superseded"
+ elif bool(metadata.get("topic_bucket_current_subject_preserved")) or bool(metadata.get("current_subject_protected")):
+ temporal_state = "current"
+ elif current_turn_index and int(hit.turn_index or 0) and current_turn_index - int(hit.turn_index) <= 1:
+ temporal_state = "current"
+ elif normalized_state == "active":
+ temporal_state = "stable"
+ else:
+ temporal_state = "historical"
+ query_tokens = set(_tokenize(query))
+ hit_tokens = set(_tokenize(" ".join([hit.value, " ".join(hit.anchors), _clean_text(metadata.get("topic_label", ""))])))
+ overlap = len(query_tokens & hit_tokens) / max(1, len(query_tokens | hit_tokens))
+ candidate_id = _clean_text(hit.memory_id) or f"hit:{index}"
+ topic_label = _clean_text(metadata.get("topic_label", "")) or _clean_text(metadata.get("topic_bucket_query_label", ""))
+ profile_key = _clean_text(metadata.get("profile_type", "")) or _clean_text(metadata.get("semantic_slot", ""))
+ if not profile_key and layer in {"profile", "temporal", "topic_tunnel"}:
+ profile_key = topic_label
+ semantic_similarity = max(
+ 0.0,
+ min(
+ 1.0,
+ float(
+ metadata.get(
+ "semantic_similarity",
+ metadata.get(
+ "answer_window_semantic_similarity",
+ metadata.get(
+ "embedder_similarity",
+ metadata.get("dense_similarity", metadata.get("bge_m3_similarity", hit.score)),
+ ),
+ ),
+ )
+ or 0.0
+ ),
+ ),
+ )
+ return {
+ "id": candidate_id,
+ "text": hit.value,
+ "summary": _clean_text(metadata.get("source_turn_text", "")) or _clean_text(metadata.get("event_summary", "")),
+ "topic": topic_label,
+ "profile_key": profile_key,
+ "resource_key": _clean_text(metadata.get("resource_key", "")) or (_clean_text(hit.slot_key) if layer == "resource" else ""),
+ "layer": layer,
+ "temporal_state": temporal_state,
+ "logic_roles": _dedupe(logic_roles),
+ "query_overlap": round(float(overlap), 6),
+ "retrieval_score": max(0.0, min(1.0, float(hit.score or 0.0))),
+ "graph_score": max(
+ 0.0,
+ min(
+ 1.0,
+ float(
+ metadata.get(
+ "event_score",
+ metadata.get("recall_score", metadata.get("hybrid_score", hit.score)),
+ )
+ or 0.0
+ ),
+ ),
+ ),
+ "tunnel_score": max(
+ 0.0,
+ min(
+ 1.0,
+ max(
+ float(metadata.get("path_tunnel_support_score", 0.0) or 0.0),
+ float(metadata.get("event_tunnel_support_score", 0.0) or 0.0),
+ float(metadata.get("path_chain_extension_delta_score", 0.0) or 0.0),
+ ),
+ ),
+ ),
+ "topic_similarity": max(0.0, min(1.0, float(metadata.get("topic_bucket_overlap", 0.0) or 0.0))),
+ "semantic_similarity": semantic_similarity,
+ "confidence": max(0.0, min(1.0, float(metadata.get("confidence", hit.score or 0.0) or 0.0))),
+ "rank_score": round(1.0 / float(index + 1), 6),
+ "age_turns": max(0, int(current_turn_index) - int(hit.turn_index or 0)) if current_turn_index else 0,
+ "branch_depth": max(0, len(list(metadata.get("selected_path_ids", []) or []))),
+ "contradicts_current": normalized_state in {"stale", "superseded", "false"} or bool(metadata.get("contradicts_current")),
+ "is_current": temporal_state == "current",
+ }
+
+ def _apply_injection_planner_to_hits(
+ self,
+ query: str,
+ hits: Sequence[MemoryHit],
+ *,
+ top_k: int,
+ ) -> Dict[str, Any]:
+ normalized_mode = _normalize(self.injection_planner_mode)
+ base_metadata: Dict[str, Any] = {
+ "injection_planner_enabled": False,
+ "injection_planner_mode": normalized_mode or "observe",
+ }
+ if normalized_mode in _INJECTION_PLANNER_DISABLED_MODES:
+ base_metadata["injection_planner_reason"] = "disabled"
+ return {"hits": list(hits), "metadata": base_metadata}
+ if not hits:
+ base_metadata["injection_planner_reason"] = "no_hits"
+ return {"hits": list(hits), "metadata": base_metadata}
+ model = self._load_injection_planner()
+ if model is None or self._injection_planner_config is None:
+ base_metadata.update(
+ {
+ "injection_planner_reason": self._injection_planner_error or "load_failed",
+ "injection_planner_model_path": self._injection_planner_resolved_path,
+ }
+ )
+ return {"hits": list(hits), "metadata": base_metadata}
+ torch_module = getattr(injection_planner_runtime, "torch", None)
+ if torch_module is None:
+ base_metadata["injection_planner_reason"] = "torch_unavailable"
+ return {"hits": list(hits), "metadata": base_metadata}
+ current_turn_index = max(
+ [int(getattr(self.graph, "turn_index", 0) or 0), *[int(hit.turn_index or 0) for hit in hits]],
+ default=0,
+ )
+ candidates = [
+ self._injection_planner_candidate_from_hit(query, hit, index=index, current_turn_index=current_turn_index)
+ for index, hit in enumerate(hits)
+ ]
+ row = {"id": "runtime_injection_plan", "query": query, "candidates": candidates, "gold": {}}
+ try:
+ dataset = injection_planner_runtime.InjectionPlannerDataset([row], self._injection_planner_config)
+ batch = injection_planner_runtime.collate_injection_batch([dataset[0]])
+ device = next(model.parameters()).device
+ model_batch = {
+ key: value.to(device) if hasattr(value, "to") else value
+ for key, value in dict(batch).items()
+ }
+ with torch_module.no_grad():
+ outputs = model(model_batch["features"], model_batch["valid_mask"])
+ selection_scores = torch_module.sigmoid(outputs["selection_logits"])[0].detach().cpu().tolist()
+ should_inject_score = float(torch_module.sigmoid(outputs["should_inject_logits"])[0].detach().cpu().item())
+ mode_probs = torch_module.softmax(outputs["injection_mode_logits"], dim=-1)[0].detach().cpu()
+ mode_index = int(torch_module.argmax(mode_probs).item())
+ temporal_indices = torch_module.argmax(outputs["temporal_logits"], dim=-1)[0].detach().cpu().tolist()
+ logic_scores = torch_module.sigmoid(outputs["logic_logits"])[0].detach().cpu().tolist()
+ if bool(self._injection_planner_evidence_role_supported) and "evidence_role_logits" in outputs:
+ evidence_role_indices = torch_module.argmax(outputs["evidence_role_logits"], dim=-1)[0].detach().cpu().tolist()
+ else:
+ evidence_role_indices = [
+ injection_planner_runtime.EVIDENCE_ROLES.index("direct_answer")
+ for _ in candidates
+ ]
+ except Exception as exc:
+ base_metadata.update(
+ {
+ "injection_planner_reason": f"inference_failed: {exc}",
+ "injection_planner_model_path": self._injection_planner_resolved_path,
+ }
+ )
+ return {"hits": list(hits), "metadata": base_metadata}
+ thresholds = {
+ "selection_threshold": float(self._injection_planner_thresholds.get("selection_threshold", 0.5)),
+ "row_threshold": float(self._injection_planner_thresholds.get("row_threshold", 0.5)),
+ "logic_threshold": float(self._injection_planner_thresholds.get("logic_threshold", 0.5)),
+ }
+ injection_mode = injection_planner_runtime.INJECTION_MODES[mode_index]
+ row_allows_injection = should_inject_score >= thresholds["row_threshold"] and injection_mode != "none"
+ predictions: Dict[str, Dict[str, Any]] = {}
+ for index, candidate in enumerate(candidates):
+ logic_roles = [
+ role
+ for role, score in zip(injection_planner_runtime.LOGIC_ROLES, logic_scores[index])
+ if float(score) >= thresholds["logic_threshold"]
+ ]
+ evidence_role = injection_planner_runtime.EVIDENCE_ROLES[int(evidence_role_indices[index])]
+ role_allows_selection = evidence_role not in {"noise", "negative_evidence"}
+ predictions[candidate["id"]] = {
+ "selection_score": float(selection_scores[index]),
+ "selected": bool(
+ row_allows_injection
+ and role_allows_selection
+ and float(selection_scores[index]) >= thresholds["selection_threshold"]
+ ),
+ "temporal_state": injection_planner_runtime.TEMPORAL_STATES[int(temporal_indices[index])],
+ "evidence_role": evidence_role,
+ "logic_roles": logic_roles or ["evidence"],
+ "candidate_layer": candidate.get("layer", ""),
+ "role_allows_selection": bool(role_allows_selection),
+ }
+ annotated_hits: List[MemoryHit] = []
+ for index, hit in enumerate(hits):
+ candidate_id = _clean_text(hit.memory_id) or f"hit:{index}"
+ prediction = predictions.get(candidate_id, {})
+ metadata = dict(hit.metadata or {})
+ metadata.update(
+ {
+ "injection_planner_enabled": True,
+ "injection_planner_mode": normalized_mode or "observe",
+ "injection_planner_model_path": self._injection_planner_resolved_path,
+ "injection_planner_candidate_id": candidate_id,
+ "injection_planner_score": round(float(prediction.get("selection_score", 0.0)), 6),
+ "injection_planner_selected": bool(prediction.get("selected", False)),
+ "injection_planner_temporal_state": _clean_text(prediction.get("temporal_state", "")),
+ "injection_planner_evidence_role": _clean_text(prediction.get("evidence_role", "")),
+ "injection_planner_role_allows_selection": bool(prediction.get("role_allows_selection", False)),
+ "injection_planner_logic_roles": list(prediction.get("logic_roles", []) or []),
+ "injection_planner_candidate_layer": _clean_text(prediction.get("candidate_layer", "")),
+ "injection_planner_injection_mode": injection_mode,
+ "injection_planner_should_inject_score": round(float(should_inject_score), 6),
+ "injection_planner_thresholds": dict(thresholds),
+ "injection_planner_evidence_role_supported": bool(self._injection_planner_evidence_role_supported),
+ }
+ )
+ planner_score = float(prediction.get("selection_score", 0.0))
+ next_score = max(float(hit.score), planner_score) if bool(prediction.get("selected", False)) else float(hit.score)
+ annotated_hits.append(
+ MemoryHit(
+ memory_id=hit.memory_id,
+ category=hit.category,
+ value=hit.value,
+ relation=hit.relation,
+ anchors=list(hit.anchors),
+ score=round(next_score, 6),
+ source_kind=hit.source_kind,
+ slot_key=hit.slot_key,
+ state=hit.state,
+ turn_index=int(hit.turn_index),
+ metadata=metadata,
+ )
+ )
+ selected_count = sum(1 for item in annotated_hits if bool((item.metadata or {}).get("injection_planner_selected")))
+ if normalized_mode in _INJECTION_PLANNER_FORCE_MODES and selected_count:
+ selected_ids = {hit.memory_id for hit in annotated_hits if bool((hit.metadata or {}).get("injection_planner_selected"))}
+ planned_hits = [hit for hit in annotated_hits if hit.memory_id in selected_ids]
+ elif normalized_mode in _INJECTION_PLANNER_GUIDED_MODES and selected_count:
+ planned_hits = sorted(
+ annotated_hits,
+ key=lambda hit: (
+ not bool((hit.metadata or {}).get("injection_planner_selected")),
+ -float((hit.metadata or {}).get("injection_planner_score", 0.0) or 0.0),
+ -float(hit.score or 0.0),
+ ),
+ )
+ else:
+ planned_hits = annotated_hits
+ base_metadata.update(
+ {
+ "injection_planner_enabled": True,
+ "injection_planner_reason": "ok",
+ "injection_planner_model_path": self._injection_planner_resolved_path,
+ "injection_planner_candidate_count": len(candidates),
+ "injection_planner_selected_count": int(selected_count),
+ "injection_planner_should_inject_score": round(float(should_inject_score), 6),
+ "injection_planner_injection_mode": injection_mode,
+ "injection_planner_thresholds": dict(thresholds),
+ "injection_planner_evidence_role_supported": bool(self._injection_planner_evidence_role_supported),
+ "injection_planner_guided": bool(normalized_mode in _INJECTION_PLANNER_GUIDED_MODES | _INJECTION_PLANNER_FORCE_MODES),
+ "injection_planner_prediction_ids": [
+ candidate_id
+ for candidate_id, prediction in predictions.items()
+ if bool(prediction.get("selected", False))
+ ],
+ }
+ )
+ return {"hits": planned_hits[: max(len(planned_hits), top_k)], "metadata": base_metadata}
+
+ def _node_scorer(self) -> LoadedNodeMemoryScorer | None:
+ if self.retrieval_mode != "hybrid_node_scored":
+ return None
+ if self._loaded_node_scorer is not None:
+ return self._loaded_node_scorer
+ if not self.node_model_path:
+ self._node_scorer_error = "node_model_path_missing"
+ return None
+ try:
+ self._loaded_node_scorer = LoadedNodeMemoryScorer(
+ node_model_path=Path(self.node_model_path),
+ path_model_path=Path(self.path_model_path) if self.path_model_path else None,
+ device=self.node_model_device or None,
+ )
+ except Exception as exc:
+ self._node_scorer_error = str(exc)
+ self._loaded_node_scorer = None
+ return self._loaded_node_scorer
+
+ def _apply_writeback_promotions(self, stored_ids: Sequence[str]) -> List[Dict[str, Any]]:
+ promotion_events: List[Dict[str, Any]] = []
+ for memory_id in stored_ids:
+ record = self.graph.records_by_id.get(memory_id)
+ if record is None or not isinstance(record.metadata, dict):
+ continue
+ if _normalize(record.metadata.get("memory_role", "")) != "assistant" or _normalize(record.metadata.get("authority", "")) != "derived":
+ continue
+ canonical_slot_key = _clean_text(record.metadata.get("canonical_slot_key", ""))
+ writeback_class = _clean_text(record.metadata.get("writeback_class", ""))
+ if not canonical_slot_key or not writeback_class:
+ continue
+ support_refs = self._support_ref_union(record.metadata)
+ confidence = float(record.confidence or 0.0)
+ same_records = [
+ item
+ for item in self.graph.records_by_id.values()
+ if isinstance(item.metadata, dict)
+ and _normalize(item.metadata.get("memory_role", "")) == "assistant"
+ and _normalize(item.metadata.get("canonical_slot_key", "")) == _normalize(canonical_slot_key)
+ and _normalize(item.metadata.get("writeback_class", "")) == _normalize(writeback_class)
+ and _normalize(item.value) == _normalize(record.value)
+ ]
+ qualifying = [item for item in same_records if float(item.confidence or 0.0) >= 0.9]
+ distinct_answers = {
+ _clean_text(answer_id)
+ for item in qualifying
+ for answer_id in [*list(item.metadata.get("origin_answer_ids", []) or []), _clean_text(item.metadata.get("origin_answer_id", ""))]
+ if _clean_text(answer_id)
+ }
+ aggregated_support = set()
+ for item in same_records:
+ aggregated_support.update(self._support_ref_union(item.metadata))
+ fast_promotion = writeback_class in {"fact", "state_change"} and confidence >= 0.97 and len(support_refs) >= 3
+ standard_promotion = len(distinct_answers) >= 2 and len(aggregated_support) >= 2
+ if not (fast_promotion or standard_promotion):
+ record.metadata["promotion_state"] = "candidate"
+ continue
+ source_head = self._source_head_for_canonical(canonical_slot_key)
+ blocked_conflict = source_head is not None and _normalize(source_head.value) != _normalize(record.value)
+ promoted_slot = f"promoted.{canonical_slot_key}"
+ promoted_metadata = {
+ **dict(record.metadata or {}),
+ "memory_role": "assistant",
+ "authority": "promoted",
+ "canonical_slot_key": canonical_slot_key,
+ "writeback_class": writeback_class,
+ "promotion_state": "blocked_conflict" if blocked_conflict else "promoted",
+ "support_memory_ids": sorted({*list(record.metadata.get("support_memory_ids", []) or []), *[ref for ref in aggregated_support if ref.startswith("fact:") is False and ref.startswith("path:") is False]}),
+ "support_fact_refs": sorted({*list(record.metadata.get("support_fact_refs", []) or []), *[ref for ref in aggregated_support if ref.startswith("fact:")]}),
+ "support_path_refs": sorted({*list(record.metadata.get("support_path_refs", []) or []), *[ref for ref in aggregated_support if ref.startswith("path:")]}),
+ }
+ promoted_record = SessionMemoryRecordV2(
+ memory_id=f"{promoted_slot}:{record.turn_index}:promoted",
+ category=record.category,
+ slot_key=promoted_slot,
+ value=record.value,
+ relation=record.relation,
+ anchor_concepts=list(record.anchor_concepts),
+ evidence_anchors=list(record.evidence_anchors),
+ salience=max(float(record.salience), 0.78),
+ confidence=max(float(record.confidence), 0.9),
+ source_kind=f"promoted_{record.source_kind}",
+ turn_index=int(record.turn_index),
+ state="active",
+ metadata=promoted_metadata,
+ )
+ promoted_ids = self.graph.add_records([promoted_record])
+ record.metadata["promotion_state"] = "blocked_conflict" if blocked_conflict else "promoted"
+ promotion_events.append(
+ {
+ "source_memory_id": record.memory_id,
+ "promoted_record_ids": list(promoted_ids),
+ "canonical_slot_key": canonical_slot_key,
+ "writeback_class": writeback_class,
+ "promotion_state": record.metadata["promotion_state"],
+ "blocked_conflict": bool(blocked_conflict),
+ }
+ )
+ return promotion_events
+
+ def _source_head_for_canonical(self, canonical_slot_key: str) -> SessionMemoryRecordV2 | None:
+ head_id = self.graph.slot_heads.get(canonical_slot_key)
+ if not head_id:
+ return None
+ record = self.graph.records_by_id.get(head_id)
+ if record is None or not isinstance(record.metadata, dict):
+ return None
+ if _normalize(record.metadata.get("memory_role", "")) != "user" or _normalize(record.metadata.get("authority", "")) != "source":
+ return None
+ return record
+
+ def _support_ref_union(self, metadata: Dict[str, Any]) -> set[str]:
+ return {
+ *[_clean_text(item) for item in list(metadata.get("support_memory_ids", []) or []) if _clean_text(item)],
+ *[_clean_text(item) for item in list(metadata.get("support_fact_refs", []) or []) if _clean_text(item)],
+ *[_clean_text(item) for item in list(metadata.get("support_path_refs", []) or []) if _clean_text(item)],
+ }
+
+ def _hybrid_node_scored_hits(
+ self,
+ query: str,
+ hits: Sequence[MemoryHit],
+ *,
+ top_k: int,
+ public_hits: Sequence[MemoryHit] | None = None,
+ ) -> Dict[str, Any]:
+ scorer = self._node_scorer()
+ if scorer is None:
+ return {
+ "hits": list(hits),
+ "metadata": {
+ "retrieval_mode": "heuristic",
+ "hybrid_enabled": False,
+ "hybrid_error": self._node_scorer_error,
+ },
+ }
+ source_hits = _learnable_graph_hits(self.graph)
+ if not source_hits:
+ return {
+ "hits": list(hits),
+ "metadata": {
+ "retrieval_mode": "heuristic",
+ "hybrid_enabled": False,
+ "hybrid_error": "no_learnable_hits",
+ },
+ }
+ has_public_hits = any(
+ _is_public_dialog_hit(hit)
+ and not bool((hit.metadata or {}).get("profile_layer"))
+ for hit in source_hits
+ )
+ has_generic_hits = any(not _is_public_dialog_hit(hit) for hit in source_hits)
+ hybrid_source = "mixed_full_graph"
+ if has_public_hits and not has_generic_hits:
+ hybrid_source = "public_full_graph"
+ elif has_generic_hits and not has_public_hits:
+ hybrid_source = "generic_full_graph"
+ runtime_graph = _build_runtime_graph_from_hits(query, source_hits)
+ grouped_hits = dict(runtime_graph.get("grouped_hits", {}) or {})
+ profile_first_payload = _profile_first_hybrid_rescue(
+ self.graph,
+ query,
+ grouped_hits=grouped_hits,
+ top_k=top_k,
+ )
+ profile_first_hits = list(profile_first_payload.get("hits", []) or [])
+ profile_first_event_ids = list(profile_first_payload.get("event_ids", []) or [])
+ profile_first_memory_ids = list(profile_first_payload.get("memory_ids", []) or [])
+ candidate_event_ids = sorted(grouped_hits.keys())
+ if not candidate_event_ids:
+ return {
+ "hits": list(hits),
+ "metadata": {
+ "retrieval_mode": "heuristic",
+ "hybrid_enabled": False,
+ "hybrid_error": "no_event_candidates",
+ },
+ }
+ question_analysis = extract_question_features(query)
+ hybrid_candidate_limit = min(
+ len(candidate_event_ids),
+ max(
+ _HYBRID_SELECTED_EVENT_FLOOR,
+ int(self.candidate_event_k) * 4,
+ int(self.support_path_k) * 4,
+ int(top_k) * 4,
+ ),
+ )
+ embedder_pre_recall_mode = _normalize(getattr(self, "embedder_pre_recall_mode", ""))
+ embedder_pre_recall_enabled = embedder_pre_recall_mode not in _EMBEDDER_INDEX_DISABLED_MODES
+ embedder_pre_recall_index_mode = self.embedder_index_recall_mode
+ if embedder_pre_recall_mode not in {"1", "true", "yes", "on", "auto", "seed", "candidate", "candidates"}:
+ embedder_pre_recall_index_mode = embedder_pre_recall_mode
+ pre_embedder_index_payload: Dict[str, Any] = {
+ "event_ids": [],
+ "metadata": {
+ "embedder_pre_recall_enabled": False,
+ "embedder_pre_recall_mode": embedder_pre_recall_mode or "off",
+ "embedder_pre_recall_index_mode": embedder_pre_recall_index_mode or "off",
+ "embedder_pre_recall_event_ids": [],
+ },
+ }
+ pre_embedder_event_ids: List[str] = []
+ pre_candidate_event_ids: List[str] = []
+ if embedder_pre_recall_enabled:
+ pre_embedder_index_payload = _embedder_index_recall_event_ids(
+ query,
+ grouped_hits=grouped_hits,
+ mode=embedder_pre_recall_index_mode,
+ limit=self.embedder_pre_recall_k or self.embedder_index_recall_k or hybrid_candidate_limit,
+ max_terms=self.write_embedder_index_max_terms,
+ )
+ pre_embedder_event_ids = list(pre_embedder_index_payload.get("event_ids", []) or [])
+ pre_candidate_event_ids = _bounded_event_id_union(
+ pre_embedder_event_ids,
+ max_items=hybrid_candidate_limit,
+ )
+ pre_score_kwargs = {
+ "graph": runtime_graph,
+ "question": query,
+ "question_features": question_analysis,
+ "rerank_top_k": self.candidate_event_k,
+ "event_rerank_mode": self.event_rerank_mode,
+ "matrix_event_top_k": self.matrix_event_top_k,
+ "support_path_k": self.support_path_k,
+ "top_k": top_k,
+ }
+ if pre_candidate_event_ids:
+ pre_score_kwargs["candidate_event_ids"] = pre_candidate_event_ids
+ scored = _call_with_supported_kwargs(
+ scorer.score_runtime,
+ **pre_score_kwargs,
+ )
+ memory_router_decision = _memory_router_decision(
+ scored,
+ mode=self.memory_router_mode,
+ threshold=self.memory_router_threshold,
+ margin=self.memory_router_margin,
+ )
+ profile_first_router_suppressed = False
+ if not _memory_router_allows(memory_router_decision, "profile", "resource"):
+ profile_first_hits = []
+ profile_first_event_ids = []
+ profile_first_memory_ids = []
+ profile_first_router_suppressed = True
+ initial_recall_event_scores = dict(scored.get("recall_event_scores", {}) or {})
+ model_recall_event_ids = _bounded_event_id_union(
+ list(scored.get("recall_event_ids", []) or []),
+ [
+ event_id
+ for event_id, _ in sorted(
+ initial_recall_event_scores.items(),
+ key=lambda item: (-float(item[1]), item[0]),
+ )
+ ],
+ list(scored.get("rerank_candidate_event_ids", []) or []),
+ max_items=max(1, len(candidate_event_ids)),
+ )
+ learned_recall_event_ids = list(model_recall_event_ids)
+ symbolic_recall_event_ids = _symbolic_recall_event_ids(
+ query,
+ runtime_graph,
+ grouped_hits=grouped_hits,
+ limit=hybrid_candidate_limit,
+ )
+ symbolic_recall_event_ids = _bounded_event_id_union(
+ profile_first_event_ids,
+ symbolic_recall_event_ids,
+ max_items=hybrid_candidate_limit,
+ )
+ if pre_embedder_event_ids:
+ embedder_index_payload = pre_embedder_index_payload
+ else:
+ embedder_index_payload = _embedder_index_recall_event_ids(
+ query,
+ grouped_hits=grouped_hits,
+ mode=self.embedder_index_recall_mode,
+ limit=self.embedder_index_recall_k or hybrid_candidate_limit,
+ max_terms=self.write_embedder_index_max_terms,
+ )
+ embedder_index_event_ids = list(embedder_index_payload.get("event_ids", []) or [])
+ embedder_index_metadata = dict(embedder_index_payload.get("metadata", {}) or {})
+ embedder_index_metadata.update(
+ {
+ "embedder_pre_recall_enabled": bool(pre_candidate_event_ids),
+ "embedder_pre_recall_mode": embedder_pre_recall_mode or "off",
+ "embedder_pre_recall_index_mode": embedder_pre_recall_index_mode or "off",
+ "embedder_pre_recall_event_ids": list(pre_embedder_event_ids),
+ "embedder_pre_recall_candidate_event_ids": list(pre_candidate_event_ids),
+ "embedder_pre_recall_candidate_count": int(len(pre_candidate_event_ids)),
+ }
+ )
+ hybrid_candidate_event_ids = _bounded_event_id_union(
+ profile_first_event_ids,
+ embedder_index_event_ids,
+ learned_recall_event_ids,
+ symbolic_recall_event_ids,
+ max_items=hybrid_candidate_limit,
+ )
+ learned_candidate_event_ids = _bounded_event_id_union(
+ learned_recall_event_ids,
+ max_items=hybrid_candidate_limit,
+ )
+ learned_recall_event_id_set = set(learned_candidate_event_ids)
+ hybrid_candidate_union_added_event_ids = [
+ event_id
+ for event_id in hybrid_candidate_event_ids
+ if event_id not in learned_recall_event_id_set
+ ]
+ hybrid_candidate_union_priority_changed = list(hybrid_candidate_event_ids) != list(learned_candidate_event_ids)
+ hybrid_candidate_union_rescored = False
+ if hybrid_candidate_union_added_event_ids or (embedder_index_event_ids and hybrid_candidate_union_priority_changed):
+ scored = _call_with_supported_kwargs(
+ scorer.score_runtime,
+ graph=runtime_graph,
+ question=query,
+ question_features=question_analysis,
+ candidate_event_ids=hybrid_candidate_event_ids,
+ rerank_top_k=self.candidate_event_k,
+ event_rerank_mode=self.event_rerank_mode,
+ matrix_event_top_k=self.matrix_event_top_k,
+ support_path_k=self.support_path_k,
+ top_k=top_k,
+ )
+ hybrid_candidate_union_rescored = True
+ memory_router_decision = _memory_router_decision(
+ scored,
+ mode=self.memory_router_mode,
+ threshold=self.memory_router_threshold,
+ margin=self.memory_router_margin,
+ )
+ recall_event_scores = dict(scored.get("recall_event_scores", {}) or {})
+ rerank_candidate_event_ids = list(scored.get("rerank_candidate_event_ids", []) or [])
+ base_event_scores = dict(scored.get("base_event_scores", {}) or {})
+ rerank_event_scores = dict(scored.get("rerank_event_scores", {}) or {})
+ calibrated_event_scores = dict(scored.get("calibrated_event_scores", {}) or {})
+ matrix_event_scores = dict(scored.get("matrix_event_scores", {}) or {})
+ event_fusion_delta_scores = dict(scored.get("event_fusion_delta_scores", {}) or {})
+ event_tunnel_support_scores = dict(scored.get("event_tunnel_support_scores", {}) or {})
+ event_tunnel_delta_scores = dict(scored.get("event_tunnel_delta_scores", {}) or {})
+ tri_maze_event_reverse_scores = dict(scored.get("tri_maze_event_reverse_scores", {}) or {})
+ tri_maze_event_boundary_scores = dict(scored.get("tri_maze_event_boundary_scores", {}) or {})
+ tri_maze_event_reverse_relations = dict(scored.get("tri_maze_event_reverse_relations", {}) or {})
+ matrix_rerank_event_ids = list(scored.get("matrix_rerank_event_ids", []) or [])
+ matrix_enabled = bool(scored.get("matrix_enabled", False))
+ rerank_path_scores = dict(scored.get("rerank_path_scores", {}) or {})
+ matrix_path_scores = dict(scored.get("matrix_path_scores", {}) or {})
+ tri_maze_path_reverse_scores = dict(scored.get("tri_maze_path_reverse_scores", {}) or {})
+ tri_maze_path_boundary_scores = dict(scored.get("tri_maze_path_boundary_scores", {}) or {})
+ tri_maze_path_reverse_relations = dict(scored.get("tri_maze_path_reverse_relations", {}) or {})
+ matrix_path_rerank_ids = list(scored.get("matrix_path_rerank_ids", []) or [])
+ matrix_path_enabled = bool(scored.get("matrix_path_enabled", False))
+ fusion_enabled = bool(scored.get("fusion_enabled", False))
+ event_fusion_enabled = bool(scored.get("event_fusion_enabled", fusion_enabled))
+ path_fusion_enabled = bool(scored.get("path_fusion_enabled", fusion_enabled))
+ event_calibration_enabled = bool(scored.get("event_calibration_enabled", False))
+ path_calibration_enabled = bool(scored.get("path_calibration_enabled", False))
+ event_tunnel_enabled = bool(scored.get("event_tunnel_enabled", False))
+ path_tunnel_enabled = bool(scored.get("path_tunnel_enabled", False))
+ final_event_fusion_enabled = bool(scored.get("final_event_fusion_enabled", False))
+ final_path_fusion_enabled = bool(scored.get("final_path_fusion_enabled", False))
+ decision_fusion_enabled = bool(scored.get("decision_fusion_enabled", False))
+ decision_score_source = _clean_text(scored.get("decision_score_source", ""))
+ event_scores = dict(scored.get("event_scores", {}) or {})
+ base_path_scores = dict(scored.get("base_path_scores", {}) or {})
+ calibrated_path_scores = dict(scored.get("calibrated_path_scores", {}) or {})
+ path_fusion_delta_scores = dict(scored.get("path_fusion_delta_scores", {}) or {})
+ path_tunnel_support_scores = dict(scored.get("path_tunnel_support_scores", {}) or {})
+ path_tunnel_delta_scores = dict(scored.get("path_tunnel_delta_scores", {}) or {})
+ path_model_scores = dict(scored.get("path_model_scores", {}) or {})
+ path_chain_extension_delta_scores = dict(scored.get("path_chain_extension_delta_scores", {}) or {})
+ path_chain_extended_scores = dict(scored.get("path_chain_extended_scores", {}) or {})
+ path_chain_extension_enabled = bool(scored.get("path_chain_extension_enabled", False))
+ answer_type_scores = dict(scored.get("answer_type_scores", {}) or {})
+ selected_event_ids_from_model = list(scored.get("selected_event_ids", []) or [])
+ selected_path_ids_from_model = list(scored.get("selected_path_ids", []) or [])
+ focused_answer_type_from_model = _clean_text(scored.get("focused_answer_type", ""))
+ path_scores = dict(scored.get("path_scores", {}) or {})
+ temporal_scores = dict(scored.get("temporal_scores", {}) or {})
+ runtime_paths = {_clean_text(path.get("id", "")): dict(path) for path in list(runtime_graph.get("paths", []) or [])}
+ answer_plan_scores_raw = dict(scored.get("answer_plan_scores", {}) or {})
+
+ def _answer_plan_score_map(role: str) -> Dict[str, float]:
+ raw_scores = answer_plan_scores_raw.get(role, {})
+ if not isinstance(raw_scores, Mapping):
+ return {}
+ normalized: Dict[str, float] = {}
+ for raw_event_id, raw_score in raw_scores.items():
+ event_id = _clean_text(raw_event_id)
+ if not event_id:
+ continue
+ try:
+ normalized[event_id] = float(raw_score or 0.0)
+ except (TypeError, ValueError):
+ normalized[event_id] = 0.0
+ return normalized
+
+ answer_plan_selected_scores = _answer_plan_score_map("selected")
+ answer_plan_current_scores = _answer_plan_score_map("current")
+ answer_plan_historical_scores = _answer_plan_score_map("historical")
+ answer_plan_suppressed_scores = _answer_plan_score_map("suppressed")
+ answer_plan_scores = {
+ "selected": dict(answer_plan_selected_scores),
+ "current": dict(answer_plan_current_scores),
+ "historical": dict(answer_plan_historical_scores),
+ "suppressed": dict(answer_plan_suppressed_scores),
+ }
+ try:
+ answer_plan_event_selection_threshold = float(
+ os.getenv("TMCRA_ANSWER_PLAN_EVENT_SELECTION_THRESHOLD", "0.50") or 0.50
+ )
+ except (TypeError, ValueError):
+ answer_plan_event_selection_threshold = 0.50
+ try:
+ answer_plan_event_selection_top_k = int(os.getenv("TMCRA_ANSWER_PLAN_EVENT_SELECTION_TOP_K", "0") or 0)
+ except (TypeError, ValueError):
+ answer_plan_event_selection_top_k = 0
+ if answer_plan_event_selection_top_k <= 0:
+ answer_plan_event_selection_top_k = max(top_k, self.support_path_k * 2, _HYBRID_SELECTED_EVENT_FLOOR)
+ answer_plan_available_event_ids = {
+ _clean_text(event_id)
+ for event_id in [
+ *list(grouped_hits.keys()),
+ *[_clean_text(path.get("event_id", "")) for path in runtime_paths.values()],
+ ]
+ if _clean_text(event_id)
+ }
+ answer_plan_event_rows: List[tuple[str, float, float, float, float, float]] = []
+ answer_plan_event_ids = set(answer_plan_selected_scores) | set(answer_plan_current_scores) | set(answer_plan_historical_scores)
+ for event_id in answer_plan_event_ids:
+ if answer_plan_available_event_ids and event_id not in answer_plan_available_event_ids:
+ continue
+ selected_score = float(answer_plan_selected_scores.get(event_id, 0.0) or 0.0)
+ current_score = float(answer_plan_current_scores.get(event_id, 0.0) or 0.0)
+ historical_score = float(answer_plan_historical_scores.get(event_id, 0.0) or 0.0)
+ suppressed_score = float(answer_plan_suppressed_scores.get(event_id, 0.0) or 0.0)
+ support_score = max(selected_score, current_score)
+ adjusted_score = support_score - max(0.0, suppressed_score - support_score) * 0.5
+ if support_score < answer_plan_event_selection_threshold or suppressed_score > support_score:
+ continue
+ answer_plan_event_rows.append(
+ (event_id, adjusted_score, support_score, selected_score, current_score, historical_score)
+ )
+ answer_plan_event_rows.sort(
+ key=lambda row: (-float(row[1]), -float(row[2]), -float(row[3]), -float(row[4]), row[0])
+ )
+ answer_plan_raw_ranked_event_ids = [
+ event_id for event_id, *_ in answer_plan_event_rows[: max(1, answer_plan_event_selection_top_k)]
+ ]
+ try:
+ answer_plan_promotion_min_margin = float(
+ os.getenv("TMCRA_ANSWER_PLAN_PROMOTION_MIN_MARGIN", "0.02") or 0.02
+ )
+ except (TypeError, ValueError):
+ answer_plan_promotion_min_margin = 0.02
+ answer_plan_promotion_score_margin = 0.0
+ if len(answer_plan_event_rows) == 1:
+ answer_plan_promotion_score_margin = float(answer_plan_event_rows[0][1])
+ answer_plan_promotion_enabled = True
+ elif len(answer_plan_event_rows) > 1:
+ comparison_index = min(len(answer_plan_event_rows) - 1, max(1, min(answer_plan_event_selection_top_k, 5) - 1))
+ answer_plan_promotion_score_margin = float(answer_plan_event_rows[0][1]) - float(answer_plan_event_rows[comparison_index][1])
+ answer_plan_promotion_enabled = answer_plan_promotion_score_margin >= answer_plan_promotion_min_margin
+ else:
+ answer_plan_promotion_enabled = False
+ answer_plan_ranked_event_ids = list(answer_plan_raw_ranked_event_ids if answer_plan_promotion_enabled else [])
+ answer_plan_selected_event_ids = [
+ event_id
+ for event_id in answer_plan_ranked_event_ids
+ if float(answer_plan_selected_scores.get(event_id, 0.0) or 0.0)
+ >= answer_plan_event_selection_threshold
+ ]
+ answer_plan_current_event_ids = [
+ event_id
+ for event_id in answer_plan_ranked_event_ids
+ if float(answer_plan_current_scores.get(event_id, 0.0) or 0.0)
+ >= answer_plan_event_selection_threshold
+ ]
+ answer_plan_support_scores = {
+ event_id: round(float(support_score), 6)
+ for event_id, _, support_score, *_ in answer_plan_event_rows
+ }
+ answer_plan_adjusted_scores = {
+ event_id: round(float(adjusted_score), 6)
+ for event_id, adjusted_score, *_ in answer_plan_event_rows
+ }
+ answer_plan_rank_lookup = {
+ event_id: rank for rank, event_id in enumerate(answer_plan_ranked_event_ids, start=1)
+ }
+ answer_plan_ranked_event_id_set = set(answer_plan_ranked_event_ids)
+
+ def _answer_plan_hit_metadata(event_id: str) -> Dict[str, Any]:
+ clean_event_id = _clean_text(event_id)
+ selected_score = float(answer_plan_selected_scores.get(clean_event_id, 0.0) or 0.0)
+ current_score = float(answer_plan_current_scores.get(clean_event_id, 0.0) or 0.0)
+ historical_score = float(answer_plan_historical_scores.get(clean_event_id, 0.0) or 0.0)
+ suppressed_score = float(answer_plan_suppressed_scores.get(clean_event_id, 0.0) or 0.0)
+ support_score = max(selected_score, current_score)
+ return {
+ "answer_plan_score": round(float(support_score), 6),
+ "answer_plan_selected_score": round(float(selected_score), 6),
+ "answer_plan_current_score": round(float(current_score), 6),
+ "answer_plan_historical_score": round(float(historical_score), 6),
+ "answer_plan_suppressed_score": round(float(suppressed_score), 6),
+ "answer_plan_adjusted_score": round(float(answer_plan_adjusted_scores.get(clean_event_id, 0.0)), 6),
+ "answer_plan_selected": bool(clean_event_id in answer_plan_ranked_event_id_set),
+ "answer_plan_rank": int(answer_plan_rank_lookup.get(clean_event_id, 0) or 0),
+ }
+
+ embedder_fusion_mode = _normalize(self.embedder_fusion_mode)
+ embedder_fusion_enabled = (
+ embedder_fusion_mode not in _EMBEDDER_INDEX_DISABLED_MODES
+ and bool(embedder_index_event_ids)
+ and bool(self.embedder_fusion_top_k > 0)
+ and bool(self.embedder_fusion_weight > 0.0)
+ )
+ embedder_fusion_applied_event_scores: Dict[str, float] = {}
+ embedder_fusion_boosts: Dict[str, float] = {}
+ embedder_event_scores = dict(embedder_index_metadata.get("embedder_index_event_scores", {}) or {})
+ if embedder_fusion_enabled and embedder_event_scores:
+ ranked_embedder_events = [
+ (event_id, float(score))
+ for event_id, score in sorted(
+ embedder_event_scores.items(),
+ key=lambda item: (-float(item[1]), item[0]),
+ )
+ if _clean_text(event_id)
+ ][: max(1, int(self.embedder_fusion_top_k))]
+ for rank, (event_id, embedder_score) in enumerate(ranked_embedder_events, start=1):
+ if embedder_score < float(self.embedder_fusion_score_floor):
+ continue
+ rank_bonus = max(0.0, 0.08 - (rank - 1) * 0.006)
+ boost = min(
+ float(self.embedder_fusion_max_boost),
+ (float(self.embedder_fusion_weight) * embedder_score) + rank_bonus,
+ )
+ current_score = max(
+ float(recall_event_scores.get(event_id, 0.0) or 0.0),
+ float(event_scores.get(event_id, 0.0) or 0.0),
+ float(base_event_scores.get(event_id, 0.0) or 0.0),
+ float(rerank_event_scores.get(event_id, 0.0) or 0.0),
+ float(calibrated_event_scores.get(event_id, 0.0) or 0.0),
+ )
+ fused_score = current_score + boost
+ recall_event_scores[event_id] = max(float(recall_event_scores.get(event_id, 0.0) or 0.0), fused_score)
+ event_scores[event_id] = max(float(event_scores.get(event_id, 0.0) or 0.0), fused_score)
+ base_event_scores[event_id] = max(float(base_event_scores.get(event_id, 0.0) or 0.0), fused_score)
+ rerank_event_scores[event_id] = max(float(rerank_event_scores.get(event_id, 0.0) or 0.0), fused_score)
+ calibrated_event_scores[event_id] = max(float(calibrated_event_scores.get(event_id, 0.0) or 0.0), fused_score)
+ embedder_fusion_applied_event_scores[event_id] = round(fused_score, 6)
+ embedder_fusion_boosts[event_id] = round(boost, 6)
+ if embedder_fusion_applied_event_scores:
+ decision_score_source = f"{decision_score_source or 'learned_decision_fusion'}+embedder_fusion"
+ recall_event_ids = [
+ event_id
+ for event_id, _ in sorted(recall_event_scores.items(), key=lambda item: (-float(item[1]), item[0]))
+ ][: self.candidate_event_k]
+ if profile_first_event_ids:
+ profile_first_score_by_event = {
+ _clean_text((hit.metadata or {}).get("profile_first_hybrid_event_id", "")) or _runtime_event_key(hit): float(hit.score)
+ for hit in profile_first_hits
+ }
+ for rank, event_id in enumerate(profile_first_event_ids, start=1):
+ if not event_id:
+ continue
+ floor = max(0.92, float(profile_first_score_by_event.get(event_id, 0.0))) + max(0.0, 0.04 - (rank * 0.005))
+ recall_event_scores[event_id] = max(float(recall_event_scores.get(event_id, 0.0)), floor)
+ event_scores[event_id] = max(float(event_scores.get(event_id, 0.0)), floor)
+ base_event_scores[event_id] = max(float(base_event_scores.get(event_id, 0.0)), floor)
+ rerank_event_scores[event_id] = max(float(rerank_event_scores.get(event_id, 0.0)), floor)
+ calibrated_event_scores[event_id] = max(float(calibrated_event_scores.get(event_id, 0.0)), floor)
+ for path_id, path in runtime_paths.items():
+ event_id = _clean_text(path.get("event_id", ""))
+ path_type = _clean_text(path.get("type", ""))
+ if event_id not in set(profile_first_event_ids):
+ continue
+ if path_type not in {"speaker_event_profile", "speaker_event_source_turn", "speaker_event_status"}:
+ continue
+ floor = max(0.90, float(event_scores.get(event_id, 0.0)))
+ path_scores[path_id] = max(float(path_scores.get(path_id, 0.0)), floor)
+ base_path_scores[path_id] = max(float(base_path_scores.get(path_id, 0.0)), floor)
+ calibrated_path_scores[path_id] = max(float(calibrated_path_scores.get(path_id, 0.0)), floor)
+ path_model_scores[path_id] = max(float(path_model_scores.get(path_id, 0.0)), floor)
+ selection_consistency_repaired = False
+ selection_consistency_reason = ""
+ model_focused_answer_type = _clean_text(focused_answer_type_from_model)
+ embedder_fusion_selected_event_ids: List[str] = []
+ embedder_fusion_selected_path_ids: List[str] = []
+ if decision_fusion_enabled and (selected_path_ids_from_model or selected_event_ids_from_model):
+ base_selected_path_limit = max(1, min(max(1, self.support_path_k), max(1, top_k)))
+ selected_path_ids = [
+ path_id
+ for path_id in selected_path_ids_from_model
+ if _clean_text(path_id) in runtime_paths
+ ]
+ if not selected_path_ids:
+ selected_path_ids = [
+ path_id
+ for path_id, _ in sorted(
+ ((path_id, float(score or 0.0)) for path_id, score in path_scores.items()),
+ key=lambda item: (-float(item[1]), item[0]),
+ )
+ ][:base_selected_path_limit]
+ focused_answer_type = _reconciled_focused_answer_type(question_analysis, answer_type_scores, focused_answer_type_from_model)
+ tunnel_rescue_path_ids: List[str] = []
+ tunnel_rescue_pre_filter_path_ids: List[str] = []
+ path_utility_direct_support_path_ids: List[str] = []
+ path_utility_contrast_support_path_ids: List[str] = []
+ path_utility_latent_context_path_ids: List[str] = []
+ path_utility_drift_noise_path_ids: List[str] = []
+ path_utility_roles: Dict[str, str] = {}
+ path_utility_reasons: Dict[str, str] = {}
+ path_utility_scores: Dict[str, float] = {}
+ path_utility_overlap_tokens: Dict[str, List[str]] = {}
+ path_utility_anchor_event_ids: List[str] = []
+ path_utility_anchor_subject_signatures: List[str] = []
+ tunnel_rescue_score_threshold = float(self.path_tunnel_rescue_score_floor)
+ tunnel_rescue_candidate_count = 0
+ tunnel_rescue_filtered_count = 0
+ if (
+ self.path_tunnel_rescue_k > 0
+ and path_tunnel_support_scores
+ and _memory_router_allows(memory_router_decision, "path_tunnel", "topic_tunnel")
+ ):
+ selected_path_id_set = set(selected_path_ids)
+ candidate_scores = [
+ float(score or 0.0)
+ for path_id, score in path_tunnel_support_scores.items()
+ if _clean_text(path_id) in runtime_paths
+ ]
+ tunnel_rescue_candidate_count = len(candidate_scores)
+ if candidate_scores and self.path_tunnel_rescue_min_score_margin > 0.0:
+ sorted_scores = sorted(candidate_scores)
+ median_score = sorted_scores[len(sorted_scores) // 2]
+ tunnel_rescue_score_threshold = max(
+ tunnel_rescue_score_threshold,
+ median_score + float(self.path_tunnel_rescue_min_score_margin),
+ )
+ event_turn_indices = [
+ _runtime_event_turn_index_from_id(_clean_text(path.get("event_id", "")))
+ for path in runtime_paths.values()
+ ]
+ current_turn_index = max(
+ [int(getattr(self.graph, "turn_index", 0) or 0), *[turn for turn in event_turn_indices if turn > 0]],
+ default=0,
+ )
+ ranked_tunnel_path_ids = []
+ for path_id, score in sorted(
+ (
+ (path_id, float(score or 0.0))
+ for path_id, score in path_tunnel_support_scores.items()
+ if _clean_text(path_id) in runtime_paths
+ ),
+ key=lambda item: (-float(item[1]), item[0]),
+ ):
+ if path_id in selected_path_id_set or score < tunnel_rescue_score_threshold:
+ continue
+ path_event_turn = _runtime_event_turn_index_from_id(
+ _clean_text(runtime_paths.get(path_id, {}).get("event_id", ""))
+ )
+ path_age = max(0, current_turn_index - path_event_turn) if current_turn_index and path_event_turn else 0
+ if self.path_tunnel_rescue_min_age > 0 and path_age < self.path_tunnel_rescue_min_age:
+ continue
+ ranked_tunnel_path_ids.append(path_id)
+ tunnel_rescue_filtered_count = len(ranked_tunnel_path_ids)
+ tunnel_rescue_pre_filter_path_ids = ranked_tunnel_path_ids[: max(self.path_tunnel_rescue_k, self.path_tunnel_rescue_k * 4)]
+ utility_gate = _path_utility_gate(
+ tunnel_rescue_pre_filter_path_ids,
+ query=query,
+ runtime_graph=runtime_graph,
+ runtime_paths=runtime_paths,
+ grouped_hits=grouped_hits,
+ selected_path_ids=selected_path_ids,
+ selected_event_ids_from_model=selected_event_ids_from_model,
+ path_scores=path_scores,
+ path_tunnel_support_scores=path_tunnel_support_scores,
+ question_analysis=question_analysis,
+ focused_answer_type=focused_answer_type,
+ score_threshold=tunnel_rescue_score_threshold,
+ limit=self.path_tunnel_rescue_k,
+ )
+ tunnel_rescue_path_ids = list(utility_gate.get("injected_path_ids", []) or [])
+ path_utility_direct_support_path_ids = list(utility_gate.get("direct_support_path_ids", []) or [])
+ path_utility_contrast_support_path_ids = list(utility_gate.get("contrast_support_path_ids", []) or [])
+ path_utility_latent_context_path_ids = list(utility_gate.get("latent_context_path_ids", []) or [])
+ path_utility_drift_noise_path_ids = list(utility_gate.get("drift_noise_path_ids", []) or [])
+ path_utility_roles = dict(utility_gate.get("roles", {}) or {})
+ path_utility_reasons = dict(utility_gate.get("reasons", {}) or {})
+ path_utility_scores = dict(utility_gate.get("scores", {}) or {})
+ path_utility_overlap_tokens = dict(utility_gate.get("overlap_tokens", {}) or {})
+ path_utility_anchor_event_ids = list(utility_gate.get("anchor_event_ids", []) or [])
+ path_utility_anchor_subject_signatures = list(utility_gate.get("anchor_subject_signatures", []) or [])
+ if tunnel_rescue_path_ids:
+ selected_path_ids = _dedupe([*selected_path_ids, *tunnel_rescue_path_ids])
+ effective_selected_path_limit = base_selected_path_limit + len(tunnel_rescue_path_ids)
+ selected_event_ids = _dedupe(
+ [
+ *[
+ _clean_text(runtime_paths.get(path_id, {}).get("event_id", ""))
+ for path_id in selected_path_ids
+ if _clean_text(runtime_paths.get(path_id, {}).get("event_id", ""))
+ ],
+ *[
+ _clean_text(event_id)
+ for event_id in selected_event_ids_from_model
+ if _clean_text(event_id)
+ ],
+ ]
+ )
+ if answer_plan_ranked_event_ids:
+ selected_event_ids = _dedupe([*selected_event_ids, *answer_plan_ranked_event_ids])
+ if not selected_event_ids:
+ selected_event_ids = [
+ event_id
+ for event_id, _ in sorted(
+ ((event_id, float(score or 0.0)) for event_id, score in event_scores.items()),
+ key=lambda item: (-float(item[1]), item[0]),
+ )
+ ][: max(1, min(self.candidate_event_k, max(top_k, _HYBRID_SELECTED_EVENT_FLOOR)))]
+ repaired_path_ids, selection_consistency_repaired, selection_consistency_reason = _repair_selected_paths_for_focus(
+ selected_path_ids,
+ runtime_paths=runtime_paths,
+ selected_event_ids=selected_event_ids,
+ path_scores=path_scores,
+ event_scores=event_scores,
+ temporal_scores=temporal_scores,
+ question_analysis=question_analysis,
+ answer_type_scores=answer_type_scores,
+ focused_answer_type=focused_answer_type,
+ limit=max(1, effective_selected_path_limit),
+ )
+ if selection_consistency_repaired:
+ selected_path_ids = repaired_path_ids
+ selected_event_ids = _dedupe(
+ [
+ *[
+ _clean_text(runtime_paths.get(path_id, {}).get("event_id", ""))
+ for path_id in selected_path_ids
+ if _clean_text(runtime_paths.get(path_id, {}).get("event_id", ""))
+ ],
+ *[
+ _clean_text(event_id)
+ for event_id in selected_event_ids_from_model
+ if _clean_text(event_id)
+ ],
+ ]
+ )
+ if answer_plan_ranked_event_ids:
+ selected_event_ids = _dedupe([*selected_event_ids, *answer_plan_ranked_event_ids])
+ if profile_first_event_ids:
+ selected_event_ids = _dedupe([*profile_first_event_ids, *selected_event_ids])
+ if embedder_fusion_applied_event_scores and self.embedder_fusion_select_k > 0:
+ ranked_fusion_event_ids = [
+ event_id
+ for event_id in embedder_index_event_ids
+ if event_id in embedder_fusion_applied_event_scores
+ ][: max(1, int(self.embedder_fusion_select_k))]
+ selected_event_ids = _dedupe([*ranked_fusion_event_ids, *selected_event_ids])
+ selected_path_id_set = set(selected_path_ids)
+ selected_embedder_path_ids: List[str] = []
+ for event_id in ranked_fusion_event_ids:
+ candidate_paths = [
+ (path_id, path)
+ for path_id, path in runtime_paths.items()
+ if _clean_text(path.get("event_id", "")) == event_id
+ ]
+ candidate_paths.sort(
+ key=lambda item: (
+ int(_clean_text(item[1].get("type", "")) in {"speaker_event_source_turn", "speaker_event_profile", "speaker_event_status", "speaker_event_time"}),
+ float(path_scores.get(item[0], 0.0) or 0.0),
+ item[0],
+ ),
+ reverse=True,
+ )
+ for path_id, _ in candidate_paths:
+ if path_id in selected_path_id_set:
+ continue
+ selected_embedder_path_ids.append(path_id)
+ selected_path_id_set.add(path_id)
+ embedder_fusion_selected_path_ids.append(path_id)
+ break
+ if selected_embedder_path_ids:
+ selected_path_ids = _dedupe([*selected_embedder_path_ids, *selected_path_ids])
+ embedder_fusion_selected_event_ids = list(ranked_fusion_event_ids)
+ if answer_plan_ranked_event_ids:
+ selected_event_ids = _dedupe([*selected_event_ids, *answer_plan_ranked_event_ids])
+ final_hits: List[MemoryHit] = []
+ seen_memory_ids = set()
+ selected_event_id_set = set(selected_event_ids)
+ for path_id in selected_path_ids:
+ path = runtime_paths.get(path_id, {})
+ event_id = _clean_text(path.get("event_id", ""))
+ if event_id not in selected_event_id_set:
+ continue
+ path_type = _clean_text(path.get("type", ""))
+ support_node_id = _path_support_node_id(path)
+ support_hit = _support_hit_for_path(path_type, grouped_hits.get(event_id, []))
+ event_hit = _representative_event_hit(
+ [*grouped_hits.get(event_id, []), *_event_record_hits_from_graph(self.graph, event_id)],
+ query=query,
+ )
+ decision_score = round(float(path_scores.get(path_id, 0.0)), 6)
+ candidate_hits: List[tuple[MemoryHit | None, float]] = [(support_hit, decision_score)]
+ if event_hit is not None and (support_hit is None or event_hit.memory_id != support_hit.memory_id):
+ candidate_hits.append((event_hit, max(0.0, decision_score - 0.0001)))
+ for raw_hit, hit_score in candidate_hits:
+ if raw_hit is None or raw_hit.memory_id in seen_memory_ids:
+ continue
+ seen_memory_ids.add(raw_hit.memory_id)
+ recall_score = float(recall_event_scores.get(event_id, event_scores.get(event_id, 0.0)))
+ event_score = float(event_scores.get(event_id, raw_hit.score))
+ temporal_score = float(temporal_scores.get(support_node_id, 0.0))
+ metadata = dict(raw_hit.metadata or {})
+ metadata.update(
+ {
+ "event_id": event_id,
+ **_answer_plan_hit_metadata(event_id),
+ "path_id": path_id,
+ "base_event_score": round(float(base_event_scores.get(event_id, event_score)), 6),
+ "rerank_event_score": round(float(rerank_event_scores.get(event_id, event_score)), 6),
+ "calibrated_event_score": round(float(calibrated_event_scores.get(event_id, event_score)), 6),
+ "matrix_event_score": round(float(matrix_event_scores.get(event_id, 0.0)), 6),
+ "event_fusion_delta_score": round(float(event_fusion_delta_scores.get(event_id, 0.0)), 6),
+ "event_tunnel_support_score": round(float(event_tunnel_support_scores.get(event_id, 0.0)), 6),
+ "event_tunnel_delta_score": round(float(event_tunnel_delta_scores.get(event_id, 0.0)), 6),
+ "tri_maze_event_reverse_score": round(float(tri_maze_event_reverse_scores.get(event_id, 0.0)), 6),
+ "tri_maze_event_boundary_score": round(float(tri_maze_event_boundary_scores.get(event_id, 0.0)), 6),
+ "tri_maze_event_reverse_relation": round(float(tri_maze_event_reverse_relations.get(event_id, 0.0)), 6),
+ "matrix_enabled": matrix_enabled,
+ "event_calibration_enabled": event_calibration_enabled,
+ "path_calibration_enabled": path_calibration_enabled,
+ "event_tunnel_enabled": event_tunnel_enabled,
+ "path_tunnel_enabled": path_tunnel_enabled,
+ "final_event_fusion_enabled": final_event_fusion_enabled,
+ "final_path_fusion_enabled": final_path_fusion_enabled,
+ "decision_fusion_enabled": True,
+ "event_fusion_enabled": event_fusion_enabled,
+ "path_fusion_enabled": path_fusion_enabled,
+ "event_score": round(event_score, 6),
+ "recall_score": round(recall_score, 6),
+ "path_score": round(float(path_scores.get(path_id, hit_score)), 6),
+ "base_path_score": round(float(base_path_scores.get(path_id, path_scores.get(path_id, hit_score))), 6),
+ "calibrated_path_score": round(float(calibrated_path_scores.get(path_id, path_scores.get(path_id, hit_score))), 6),
+ "path_fusion_delta_score": round(float(path_fusion_delta_scores.get(path_id, 0.0)), 6),
+ "path_tunnel_support_score": round(float(path_tunnel_support_scores.get(path_id, 0.0)), 6),
+ "path_tunnel_delta_score": round(float(path_tunnel_delta_scores.get(path_id, 0.0)), 6),
+ "path_model_score": round(float(path_model_scores.get(path_id, path_scores.get(path_id, hit_score))), 6),
+ "path_chain_extension_enabled": path_chain_extension_enabled,
+ "path_chain_extension_delta_score": round(float(path_chain_extension_delta_scores.get(path_id, 0.0)), 6),
+ "path_chain_extended_score": round(float(path_chain_extended_scores.get(path_id, path_scores.get(path_id, hit_score))), 6),
+ "tri_maze_path_reverse_score": round(float(tri_maze_path_reverse_scores.get(path_id, 0.0)), 6),
+ "tri_maze_path_boundary_score": round(float(tri_maze_path_boundary_scores.get(path_id, 0.0)), 6),
+ "tri_maze_path_reverse_relation": round(float(tri_maze_path_reverse_relations.get(path_id, 0.0)), 6),
+ "effective_path_score": round(float(path_scores.get(path_id, hit_score)), 6),
+ "temporal_score": round(temporal_score, 6),
+ "raw_public_score": round(float(raw_hit.score), 6),
+ "hybrid_score": round(hit_score, 6),
+ "hybrid_score_source": decision_score_source or "learned_decision_fusion",
+ "evidence_snippet_role": "selected_path_support" if raw_hit is support_hit else "selected_path_event",
+ "selected_event_ids": list(selected_event_ids),
+ "selected_path_ids": list(selected_path_ids),
+ "path_tunnel_rescue_enabled": bool(self.path_tunnel_rescue_k > 0),
+ "path_tunnel_rescue_k": int(self.path_tunnel_rescue_k),
+ "path_tunnel_rescue_score_floor": round(float(self.path_tunnel_rescue_score_floor), 6),
+ "path_tunnel_rescue_min_age": int(self.path_tunnel_rescue_min_age),
+ "path_tunnel_rescue_min_score_margin": round(float(self.path_tunnel_rescue_min_score_margin), 6),
+ "path_tunnel_rescue_score_threshold": round(float(tunnel_rescue_score_threshold), 6),
+ "path_tunnel_rescue_candidate_count": int(tunnel_rescue_candidate_count),
+ "path_tunnel_rescue_filtered_count": int(tunnel_rescue_filtered_count),
+ "path_tunnel_rescue_path_ids": list(tunnel_rescue_path_ids),
+ "path_utility_gate_enabled": bool(self.path_tunnel_rescue_k > 0),
+ "path_utility_pre_filter_path_ids": list(tunnel_rescue_pre_filter_path_ids),
+ "path_utility_injected_path_ids": list(tunnel_rescue_path_ids),
+ "path_utility_direct_support_path_ids": list(path_utility_direct_support_path_ids),
+ "path_utility_contrast_support_path_ids": list(path_utility_contrast_support_path_ids),
+ "path_utility_latent_context_path_ids": list(path_utility_latent_context_path_ids),
+ "path_utility_drift_noise_path_ids": list(path_utility_drift_noise_path_ids),
+ "path_utility_roles": dict(path_utility_roles),
+ "path_utility_reasons": dict(path_utility_reasons),
+ "path_utility_scores": dict(path_utility_scores),
+ "path_utility_overlap_tokens": dict(path_utility_overlap_tokens),
+ "path_utility_anchor_event_ids": list(path_utility_anchor_event_ids),
+ "path_utility_anchor_subject_signatures": list(path_utility_anchor_subject_signatures),
+ "tunnel_recall_pre_filter_count": int(len(tunnel_rescue_pre_filter_path_ids)),
+ "tunnel_usable_post_filter_count": int(len(tunnel_rescue_path_ids)),
+ "model_focused_answer_type": model_focused_answer_type,
+ "selection_consistency_repaired": bool(selection_consistency_repaired),
+ "selection_consistency_reason": selection_consistency_reason,
+ }
+ )
+ final_hits.append(
+ MemoryHit(
+ memory_id=raw_hit.memory_id,
+ category=raw_hit.category,
+ value=raw_hit.value,
+ relation=raw_hit.relation,
+ anchors=list(raw_hit.anchors),
+ score=round(hit_score, 6),
+ source_kind=raw_hit.source_kind,
+ slot_key=raw_hit.slot_key,
+ state=raw_hit.state,
+ turn_index=int(raw_hit.turn_index),
+ metadata=metadata,
+ )
+ )
+ for event_rank, event_id in enumerate(selected_event_ids, start=1):
+ event_hit = _representative_event_hit(
+ [*grouped_hits.get(event_id, []), *_event_record_hits_from_graph(self.graph, event_id)],
+ query=query,
+ )
+ if event_hit is None or event_hit.memory_id in seen_memory_ids:
+ continue
+ seen_memory_ids.add(event_hit.memory_id)
+ recall_score = float(recall_event_scores.get(event_id, event_hit.score))
+ event_score = float(event_scores.get(event_id, event_hit.score))
+ metadata = dict(event_hit.metadata or {})
+ metadata.update(
+ {
+ "event_id": event_id,
+ **_answer_plan_hit_metadata(event_id),
+ "path_id": "",
+ "base_event_score": round(float(base_event_scores.get(event_id, event_score)), 6),
+ "rerank_event_score": round(float(rerank_event_scores.get(event_id, event_score)), 6),
+ "calibrated_event_score": round(float(calibrated_event_scores.get(event_id, event_score)), 6),
+ "matrix_event_score": round(float(matrix_event_scores.get(event_id, 0.0)), 6),
+ "event_fusion_delta_score": round(float(event_fusion_delta_scores.get(event_id, 0.0)), 6),
+ "event_tunnel_support_score": round(float(event_tunnel_support_scores.get(event_id, 0.0)), 6),
+ "event_tunnel_delta_score": round(float(event_tunnel_delta_scores.get(event_id, 0.0)), 6),
+ "tri_maze_event_reverse_score": round(float(tri_maze_event_reverse_scores.get(event_id, 0.0)), 6),
+ "tri_maze_event_boundary_score": round(float(tri_maze_event_boundary_scores.get(event_id, 0.0)), 6),
+ "tri_maze_event_reverse_relation": round(float(tri_maze_event_reverse_relations.get(event_id, 0.0)), 6),
+ "matrix_enabled": matrix_enabled,
+ "event_calibration_enabled": event_calibration_enabled,
+ "path_calibration_enabled": path_calibration_enabled,
+ "event_tunnel_enabled": event_tunnel_enabled,
+ "path_tunnel_enabled": path_tunnel_enabled,
+ "final_event_fusion_enabled": final_event_fusion_enabled,
+ "final_path_fusion_enabled": final_path_fusion_enabled,
+ "decision_fusion_enabled": True,
+ "event_fusion_enabled": event_fusion_enabled,
+ "path_fusion_enabled": path_fusion_enabled,
+ "event_score": round(event_score, 6),
+ "recall_score": round(recall_score, 6),
+ "path_score": 0.0,
+ "base_path_score": 0.0,
+ "calibrated_path_score": 0.0,
+ "path_fusion_delta_score": 0.0,
+ "path_tunnel_support_score": 0.0,
+ "path_tunnel_delta_score": 0.0,
+ "path_model_score": 0.0,
+ "path_chain_extension_enabled": path_chain_extension_enabled,
+ "path_chain_extension_delta_score": 0.0,
+ "path_chain_extended_score": 0.0,
+ "effective_path_score": 0.0,
+ "temporal_score": 0.0,
+ "raw_public_score": round(float(event_hit.score), 6),
+ "hybrid_score": round(event_score, 6),
+ "hybrid_score_source": decision_score_source or "learned_final_event_fusion",
+ "evidence_snippet_role": "selected_event_representative",
+ "selected_event_rank": int(event_rank),
+ "selected_event_ids": list(selected_event_ids),
+ "selected_path_ids": list(selected_path_ids),
+ "path_tunnel_rescue_enabled": bool(self.path_tunnel_rescue_k > 0),
+ "path_tunnel_rescue_k": int(self.path_tunnel_rescue_k),
+ "path_tunnel_rescue_score_floor": round(float(self.path_tunnel_rescue_score_floor), 6),
+ "path_tunnel_rescue_min_age": int(self.path_tunnel_rescue_min_age),
+ "path_tunnel_rescue_min_score_margin": round(float(self.path_tunnel_rescue_min_score_margin), 6),
+ "path_tunnel_rescue_score_threshold": round(float(tunnel_rescue_score_threshold), 6),
+ "path_tunnel_rescue_candidate_count": int(tunnel_rescue_candidate_count),
+ "path_tunnel_rescue_filtered_count": int(tunnel_rescue_filtered_count),
+ "path_tunnel_rescue_path_ids": list(tunnel_rescue_path_ids),
+ "path_utility_gate_enabled": bool(self.path_tunnel_rescue_k > 0),
+ "path_utility_pre_filter_path_ids": list(tunnel_rescue_pre_filter_path_ids),
+ "path_utility_injected_path_ids": list(tunnel_rescue_path_ids),
+ "path_utility_direct_support_path_ids": list(path_utility_direct_support_path_ids),
+ "path_utility_contrast_support_path_ids": list(path_utility_contrast_support_path_ids),
+ "path_utility_latent_context_path_ids": list(path_utility_latent_context_path_ids),
+ "path_utility_drift_noise_path_ids": list(path_utility_drift_noise_path_ids),
+ "path_utility_roles": dict(path_utility_roles),
+ "path_utility_reasons": dict(path_utility_reasons),
+ "path_utility_scores": dict(path_utility_scores),
+ "path_utility_overlap_tokens": dict(path_utility_overlap_tokens),
+ "path_utility_anchor_event_ids": list(path_utility_anchor_event_ids),
+ "path_utility_anchor_subject_signatures": list(path_utility_anchor_subject_signatures),
+ "tunnel_recall_pre_filter_count": int(len(tunnel_rescue_pre_filter_path_ids)),
+ "tunnel_usable_post_filter_count": int(len(tunnel_rescue_path_ids)),
+ "model_focused_answer_type": model_focused_answer_type,
+ "selection_consistency_repaired": bool(selection_consistency_repaired),
+ "selection_consistency_reason": selection_consistency_reason,
+ }
+ )
+ final_hits.append(
+ MemoryHit(
+ memory_id=event_hit.memory_id,
+ category=event_hit.category,
+ value=event_hit.value,
+ relation=event_hit.relation,
+ anchors=list(event_hit.anchors),
+ score=round(event_score, 6),
+ source_kind=event_hit.source_kind,
+ slot_key=event_hit.slot_key,
+ state=event_hit.state,
+ turn_index=int(event_hit.turn_index),
+ metadata=metadata,
+ )
+ )
+ if not final_hits:
+ final_hits = list(hits)
+ if profile_first_hits:
+ final_hits = _inject_profile_first_hits(
+ final_hits,
+ profile_first_hits,
+ selected_event_ids=selected_event_ids,
+ selected_path_ids=selected_path_ids,
+ )
+ final_hits = _coverage_preserving_final_hits(final_hits, selected_event_ids=selected_event_ids, top_k=top_k)
+ final_hit_event_ids = _event_ids_from_hits(final_hits)
+ final_missing_selected_event_ids = [event_id for event_id in selected_event_ids if event_id not in set(final_hit_event_ids)]
+ return {
+ "hits": final_hits,
+ "metadata": {
+ "retrieval_mode": "hybrid_node_scored",
+ "hybrid_enabled": True,
+ "hybrid_source": hybrid_source,
+ **memory_router_decision,
+ "profile_first_router_suppressed": bool(profile_first_router_suppressed),
+ "recall_event_ids": list(recall_event_ids),
+ "learned_recall_event_ids": list(learned_recall_event_ids),
+ "model_recall_event_ids": list(model_recall_event_ids),
+ "symbolic_recall_event_ids": list(symbolic_recall_event_ids),
+ "embedder_index_recall_event_ids": list(embedder_index_event_ids),
+ **embedder_index_metadata,
+ "embedder_fusion_mode": embedder_fusion_mode or "off",
+ "embedder_fusion_enabled": bool(embedder_fusion_enabled),
+ "embedder_fusion_weight": round(float(self.embedder_fusion_weight), 6),
+ "embedder_fusion_score_floor": round(float(self.embedder_fusion_score_floor), 6),
+ "embedder_fusion_top_k": int(self.embedder_fusion_top_k),
+ "embedder_fusion_select_k": int(self.embedder_fusion_select_k),
+ "embedder_fusion_max_boost": round(float(self.embedder_fusion_max_boost), 6),
+ "embedder_fusion_event_scores": dict(embedder_fusion_applied_event_scores),
+ "embedder_fusion_boosts": dict(embedder_fusion_boosts),
+ "embedder_fusion_selected_event_ids": list(embedder_fusion_selected_event_ids),
+ "embedder_fusion_selected_path_ids": list(embedder_fusion_selected_path_ids),
+ "profile_first_hybrid_enabled": bool(profile_first_event_ids),
+ "profile_first_event_ids": list(profile_first_event_ids),
+ "profile_first_memory_ids": list(profile_first_memory_ids),
+ "hybrid_candidate_event_ids": list(hybrid_candidate_event_ids),
+ "hybrid_candidate_union_enabled": True,
+ "hybrid_candidate_union_rescored": bool(hybrid_candidate_union_rescored),
+ "hybrid_candidate_union_added_event_ids": list(hybrid_candidate_union_added_event_ids),
+ "hybrid_candidate_union_priority_changed": bool(hybrid_candidate_union_priority_changed),
+ "rerank_candidate_event_ids": list(rerank_candidate_event_ids),
+ "base_event_scores": dict(base_event_scores),
+ "rerank_event_scores": dict(rerank_event_scores),
+ "calibrated_event_scores": dict(calibrated_event_scores),
+ "matrix_event_scores": dict(matrix_event_scores),
+ "event_fusion_delta_scores": dict(event_fusion_delta_scores),
+ "event_tunnel_support_scores": dict(event_tunnel_support_scores),
+ "event_tunnel_delta_scores": dict(event_tunnel_delta_scores),
+ "tri_maze_event_reverse_scores": dict(tri_maze_event_reverse_scores),
+ "tri_maze_event_boundary_scores": dict(tri_maze_event_boundary_scores),
+ "tri_maze_event_reverse_relations": dict(tri_maze_event_reverse_relations),
+ "matrix_rerank_event_ids": list(matrix_rerank_event_ids),
+ "matrix_enabled": matrix_enabled,
+ "rerank_path_scores": dict(rerank_path_scores),
+ "matrix_path_scores": dict(matrix_path_scores),
+ "tri_maze_path_reverse_scores": dict(tri_maze_path_reverse_scores),
+ "tri_maze_path_boundary_scores": dict(tri_maze_path_boundary_scores),
+ "tri_maze_path_reverse_relations": dict(tri_maze_path_reverse_relations),
+ "matrix_path_rerank_ids": list(matrix_path_rerank_ids),
+ "matrix_path_enabled": matrix_path_enabled,
+ "fusion_enabled": fusion_enabled,
+ "event_calibration_enabled": event_calibration_enabled,
+ "path_calibration_enabled": path_calibration_enabled,
+ "event_tunnel_enabled": event_tunnel_enabled,
+ "path_tunnel_enabled": path_tunnel_enabled,
+ "final_event_fusion_enabled": final_event_fusion_enabled,
+ "final_path_fusion_enabled": final_path_fusion_enabled,
+ "decision_fusion_enabled": True,
+ "decision_score_source": decision_score_source or "learned_decision_fusion",
+ "event_fusion_enabled": event_fusion_enabled,
+ "path_fusion_enabled": path_fusion_enabled,
+ "selected_event_ids": list(selected_event_ids),
+ "path_rescue_event_ids": [],
+ "selected_path_ids": list(selected_path_ids),
+ "path_tunnel_rescue_enabled": bool(self.path_tunnel_rescue_k > 0),
+ "path_tunnel_rescue_k": int(self.path_tunnel_rescue_k),
+ "path_tunnel_rescue_score_floor": round(float(self.path_tunnel_rescue_score_floor), 6),
+ "path_tunnel_rescue_min_age": int(self.path_tunnel_rescue_min_age),
+ "path_tunnel_rescue_min_score_margin": round(float(self.path_tunnel_rescue_min_score_margin), 6),
+ "path_tunnel_rescue_score_threshold": round(float(tunnel_rescue_score_threshold), 6),
+ "path_tunnel_rescue_candidate_count": int(tunnel_rescue_candidate_count),
+ "path_tunnel_rescue_filtered_count": int(tunnel_rescue_filtered_count),
+ "path_tunnel_rescue_path_ids": list(tunnel_rescue_path_ids),
+ "path_utility_gate_enabled": bool(self.path_tunnel_rescue_k > 0),
+ "path_utility_pre_filter_path_ids": list(tunnel_rescue_pre_filter_path_ids),
+ "path_utility_injected_path_ids": list(tunnel_rescue_path_ids),
+ "path_utility_direct_support_path_ids": list(path_utility_direct_support_path_ids),
+ "path_utility_contrast_support_path_ids": list(path_utility_contrast_support_path_ids),
+ "path_utility_latent_context_path_ids": list(path_utility_latent_context_path_ids),
+ "path_utility_drift_noise_path_ids": list(path_utility_drift_noise_path_ids),
+ "path_utility_roles": dict(path_utility_roles),
+ "path_utility_reasons": dict(path_utility_reasons),
+ "path_utility_scores": dict(path_utility_scores),
+ "path_utility_overlap_tokens": dict(path_utility_overlap_tokens),
+ "path_utility_anchor_event_ids": list(path_utility_anchor_event_ids),
+ "path_utility_anchor_subject_signatures": list(path_utility_anchor_subject_signatures),
+ "tunnel_recall_pre_filter_count": int(len(tunnel_rescue_pre_filter_path_ids)),
+ "tunnel_usable_post_filter_count": int(len(tunnel_rescue_path_ids)),
+ "final_hit_event_ids": list(final_hit_event_ids),
+ "final_hit_dia_ids": _dia_ids_from_hits(final_hits),
+ "final_missing_selected_event_ids": list(final_missing_selected_event_ids),
+ "selected_event_count": int(len(selected_event_ids)),
+ "selected_path_count": int(len(selected_path_ids)),
+ "temporal_scores": dict(temporal_scores),
+ "recall_event_scores": dict(recall_event_scores),
+ "event_scores": dict(event_scores),
+ "base_path_scores": dict(base_path_scores),
+ "calibrated_path_scores": dict(calibrated_path_scores),
+ "path_fusion_delta_scores": dict(path_fusion_delta_scores),
+ "path_tunnel_support_scores": dict(path_tunnel_support_scores),
+ "path_tunnel_delta_scores": dict(path_tunnel_delta_scores),
+ "path_model_scores": dict(path_model_scores),
+ "path_chain_extension_enabled": path_chain_extension_enabled,
+ "path_chain_extension_delta_scores": dict(path_chain_extension_delta_scores),
+ "path_chain_extended_scores": dict(path_chain_extended_scores),
+ "effective_path_scores": dict(path_scores),
+ "path_scores": dict(path_scores),
+ "answer_type_scores": dict(answer_type_scores),
+ "answer_plan_scores": dict(answer_plan_scores),
+ "answer_plan_support_scores": dict(answer_plan_support_scores),
+ "answer_plan_adjusted_scores": dict(answer_plan_adjusted_scores),
+ "answer_plan_raw_ranked_event_ids": list(answer_plan_raw_ranked_event_ids),
+ "answer_plan_ranked_event_ids": list(answer_plan_ranked_event_ids),
+ "answer_plan_selected_event_ids": list(answer_plan_selected_event_ids),
+ "answer_plan_current_event_ids": list(answer_plan_current_event_ids),
+ "answer_plan_promotion_enabled": bool(answer_plan_promotion_enabled),
+ "answer_plan_promotion_score_margin": round(float(answer_plan_promotion_score_margin), 6),
+ "answer_plan_promotion_min_margin": round(float(answer_plan_promotion_min_margin), 6),
+ "answer_plan_event_selection_threshold": round(float(answer_plan_event_selection_threshold), 6),
+ "answer_plan_event_selection_top_k": int(answer_plan_event_selection_top_k),
+ "focused_answer_type": focused_answer_type,
+ "model_focused_answer_type": model_focused_answer_type,
+ "selection_consistency_repaired": bool(selection_consistency_repaired),
+ "selection_consistency_reason": selection_consistency_reason,
+ "preferred_path_types": [],
+ },
+ }
+ dominant_answer_type = _dominant_answer_type(question_analysis, answer_type_scores)
+ preferred_path_types = _answer_type_preferred_path_types(question_analysis, answer_type_scores)
+ if bool(memory_router_decision.get("memory_router_guided")):
+ temporal_focus = _memory_router_allows(memory_router_decision, "temporal")
+ else:
+ temporal_focus = dominant_answer_type == "time" or bool(question_analysis.get("is_temporal", False))
+ router_profile_focus = bool(memory_router_decision.get("memory_router_guided")) and _memory_router_allows(
+ memory_router_decision,
+ "profile",
+ "resource",
+ )
+ learned_event_available = bool(event_scores)
+ learned_path_available = bool(path_scores)
+ ranked_events = [
+ event_id
+ for event_id, _ in sorted(
+ (
+ (event_id, event_scores.get(event_id, recall_event_scores.get(event_id, 0.0)))
+ for event_id in recall_event_ids
+ ),
+ key=lambda item: (-float(item[1]), item[0]),
+ )
+ ]
+ base_selected_event_count = min(
+ len(ranked_events),
+ max(
+ self.support_path_k * 2,
+ min(self.candidate_event_k, max(top_k, _HYBRID_SELECTED_EVENT_FLOOR)),
+ ),
+ )
+ effective_path_scores = dict(path_scores)
+ if not learned_path_available:
+ effective_path_scores = {
+ path_id: _calibrated_path_score(
+ path=runtime_paths.get(path_id, {}),
+ base_score=float(score),
+ temporal_scores=temporal_scores,
+ question_analysis=question_analysis,
+ answer_type_scores=answer_type_scores,
+ )
+ for path_id, score in path_scores.items()
+ }
+ ranked_path_ids_all = [
+ path_id
+ for path_id, _ in sorted(
+ ((path_id, float(score)) for path_id, score in effective_path_scores.items()),
+ key=lambda item: (-float(item[1]), item[0]),
+ )
+ ]
+ path_rescue_count = min(
+ len(ranked_path_ids_all),
+ max(self.support_path_k * 2, min(self.candidate_event_k, max(1, top_k))),
+ )
+ path_rescue_event_ids = _dedupe(
+ _clean_text(runtime_paths.get(path_id, {}).get("event_id", ""))
+ for path_id in ranked_path_ids_all[: max(1, path_rescue_count)]
+ )
+ selected_event_ids = _dedupe(
+ [
+ *path_rescue_event_ids,
+ *ranked_events[: max(1, base_selected_event_count)],
+ ]
+ )
+ if profile_first_event_ids:
+ selected_event_ids = _dedupe([*profile_first_event_ids, *selected_event_ids])
+ if answer_plan_ranked_event_ids:
+ selected_event_ids = _dedupe([*selected_event_ids, *answer_plan_ranked_event_ids])
+ selected_event_id_set = set(selected_event_ids)
+ ranked_path_ids = [
+ path_id
+ for path_id in ranked_path_ids_all
+ if _clean_text(runtime_paths.get(path_id, {}).get("event_id", "")) in selected_event_id_set
+ ]
+ if temporal_focus:
+ focused_time_path_ids = [
+ path_id
+ for path_id in ranked_path_ids
+ if _clean_text(runtime_paths.get(path_id, {}).get("type", "")) == "speaker_event_time"
+ ]
+ if focused_time_path_ids:
+ ranked_path_ids = focused_time_path_ids
+ path_limit_cap = _HYBRID_SELECTED_PATH_CAP
+ if temporal_focus:
+ path_limit_cap = _HYBRID_TEMPORAL_PATH_CAP
+ elif dominant_answer_type == "profile" or router_profile_focus:
+ path_limit_cap = _HYBRID_PROFILE_PATH_CAP
+ selected_path_count = min(
+ len(ranked_path_ids),
+ max(1, min(max(1, self.support_path_k), min(max(1, top_k), path_limit_cap))),
+ )
+ selected_path_ids = ranked_path_ids[: max(1, selected_path_count)]
+ final_hits: List[MemoryHit] = []
+ seen_memory_ids = set()
+ temporal_event_hits_added = 0
+ for path_id in selected_path_ids:
+ path = runtime_paths.get(path_id, {})
+ event_id = _clean_text(path.get("event_id", ""))
+ support_node_id = _path_support_node_id(path)
+ path_type = _clean_text(path.get("type", ""))
+ support_hit = _support_hit_for_path(path_type, grouped_hits.get(event_id, []))
+ event_hit = _representative_event_hit(
+ [*grouped_hits.get(event_id, []), *_event_record_hits_from_graph(self.graph, event_id)],
+ query=query,
+ )
+ hit_pairs: List[tuple[MemoryHit | None, float]] = [(support_hit, path_scores.get(path_id, 0.0))]
+ if not temporal_focus:
+ hit_pairs.append((event_hit, path_scores.get(path_id, 0.0)))
+ elif support_hit is None and event_hit is not None:
+ hit_pairs.append((event_hit, path_scores.get(path_id, 0.0)))
+ elif path_type == "speaker_event_time" and event_hit is not None and temporal_event_hits_added < 1:
+ hit_pairs.append((event_hit, path_scores.get(path_id, 0.0)))
+ temporal_event_hits_added += 1
+ for raw_hit, path_score in hit_pairs:
+ if raw_hit is None or raw_hit.memory_id in seen_memory_ids:
+ continue
+ seen_memory_ids.add(raw_hit.memory_id)
+ recall_score = float(recall_event_scores.get(event_id, event_scores.get(event_id, 0.0)))
+ event_score = float(event_scores.get(event_id, raw_hit.score))
+ temporal_score = float(temporal_scores.get(support_node_id, 0.0))
+ effective_path_score = float(effective_path_scores.get(path_id, path_score))
+ hybrid_score = round(effective_path_score, 6) if learned_path_available else round(
+ (0.55 * event_score)
+ + (0.20 * float(path_score))
+ + (0.15 * recall_score)
+ + (0.10 * temporal_score),
+ 6,
+ )
+ metadata = dict(raw_hit.metadata or {})
+ metadata.update(
+ {
+ "event_id": event_id,
+ **_answer_plan_hit_metadata(event_id),
+ "path_id": path_id,
+ "base_event_score": round(float(base_event_scores.get(event_id, event_score)), 6),
+ "rerank_event_score": round(float(rerank_event_scores.get(event_id, event_score)), 6),
+ "calibrated_event_score": round(float(calibrated_event_scores.get(event_id, event_score)), 6),
+ "matrix_event_score": round(float(matrix_event_scores.get(event_id, 0.0)), 6),
+ "event_fusion_delta_score": round(float(event_fusion_delta_scores.get(event_id, 0.0)), 6),
+ "event_tunnel_support_score": round(float(event_tunnel_support_scores.get(event_id, 0.0)), 6),
+ "event_tunnel_delta_score": round(float(event_tunnel_delta_scores.get(event_id, 0.0)), 6),
+ "matrix_enabled": matrix_enabled,
+ "event_calibration_enabled": event_calibration_enabled,
+ "path_calibration_enabled": path_calibration_enabled,
+ "event_tunnel_enabled": event_tunnel_enabled,
+ "path_tunnel_enabled": path_tunnel_enabled,
+ "final_event_fusion_enabled": final_event_fusion_enabled,
+ "final_path_fusion_enabled": final_path_fusion_enabled,
+ "decision_fusion_enabled": False,
+ "event_fusion_enabled": event_fusion_enabled,
+ "path_fusion_enabled": path_fusion_enabled,
+ "event_score": round(event_score, 6),
+ "recall_score": round(recall_score, 6),
+ "path_score": round(float(path_score), 6),
+ "base_path_score": round(float(base_path_scores.get(path_id, path_score)), 6),
+ "calibrated_path_score": round(float(calibrated_path_scores.get(path_id, effective_path_score)), 6),
+ "path_fusion_delta_score": round(float(path_fusion_delta_scores.get(path_id, 0.0)), 6),
+ "path_tunnel_support_score": round(float(path_tunnel_support_scores.get(path_id, 0.0)), 6),
+ "path_tunnel_delta_score": round(float(path_tunnel_delta_scores.get(path_id, 0.0)), 6),
+ "path_model_score": round(float(path_model_scores.get(path_id, path_score)), 6),
+ "path_chain_extension_enabled": path_chain_extension_enabled,
+ "path_chain_extension_delta_score": round(float(path_chain_extension_delta_scores.get(path_id, 0.0)), 6),
+ "path_chain_extended_score": round(float(path_chain_extended_scores.get(path_id, effective_path_score)), 6),
+ "effective_path_score": round(effective_path_score, 6),
+ "temporal_score": round(temporal_score, 6),
+ "raw_public_score": round(float(raw_hit.score), 6),
+ "hybrid_score": hybrid_score,
+ "hybrid_score_source": (
+ "learned_path_fusion"
+ if path_fusion_enabled
+ else "learned_path_score"
+ if learned_path_available
+ else "heuristic_mix"
+ ),
+ "evidence_snippet_role": "selected_path_support" if raw_hit is support_hit else "selected_path_event",
+ "selected_event_ids": list(selected_event_ids),
+ "selected_path_ids": list(selected_path_ids),
+ }
+ )
+ final_hits.append(
+ MemoryHit(
+ memory_id=raw_hit.memory_id,
+ category=raw_hit.category,
+ value=raw_hit.value,
+ relation=raw_hit.relation,
+ anchors=list(raw_hit.anchors),
+ score=hybrid_score,
+ source_kind=raw_hit.source_kind,
+ slot_key=raw_hit.slot_key,
+ state=raw_hit.state,
+ turn_index=int(raw_hit.turn_index),
+ metadata=metadata,
+ )
+ )
+ for event_rank, event_id in enumerate(selected_event_ids, start=1):
+ event_hit = _representative_event_hit(
+ [*grouped_hits.get(event_id, []), *_event_record_hits_from_graph(self.graph, event_id)],
+ query=query,
+ )
+ if event_hit is None or event_hit.memory_id in seen_memory_ids:
+ continue
+ seen_memory_ids.add(event_hit.memory_id)
+ recall_score = float(recall_event_scores.get(event_id, event_hit.score))
+ event_score = float(event_scores.get(event_id, event_hit.score))
+ hybrid_score = round(event_score, 6) if learned_event_available else round(
+ (0.7 * event_score) + (0.3 * recall_score),
+ 6,
+ )
+ metadata = dict(event_hit.metadata or {})
+ metadata.update(
+ {
+ "event_id": event_id,
+ **_answer_plan_hit_metadata(event_id),
+ "path_id": "",
+ "base_event_score": round(float(base_event_scores.get(event_id, event_score)), 6),
+ "rerank_event_score": round(float(rerank_event_scores.get(event_id, event_score)), 6),
+ "calibrated_event_score": round(float(calibrated_event_scores.get(event_id, event_score)), 6),
+ "matrix_event_score": round(float(matrix_event_scores.get(event_id, 0.0)), 6),
+ "event_fusion_delta_score": round(float(event_fusion_delta_scores.get(event_id, 0.0)), 6),
+ "event_tunnel_support_score": round(float(event_tunnel_support_scores.get(event_id, 0.0)), 6),
+ "event_tunnel_delta_score": round(float(event_tunnel_delta_scores.get(event_id, 0.0)), 6),
+ "matrix_enabled": matrix_enabled,
+ "event_calibration_enabled": event_calibration_enabled,
+ "path_calibration_enabled": path_calibration_enabled,
+ "event_tunnel_enabled": event_tunnel_enabled,
+ "path_tunnel_enabled": path_tunnel_enabled,
+ "final_event_fusion_enabled": final_event_fusion_enabled,
+ "final_path_fusion_enabled": final_path_fusion_enabled,
+ "decision_fusion_enabled": False,
+ "event_fusion_enabled": event_fusion_enabled,
+ "path_fusion_enabled": path_fusion_enabled,
+ "event_score": round(event_score, 6),
+ "recall_score": round(recall_score, 6),
+ "path_score": 0.0,
+ "base_path_score": 0.0,
+ "calibrated_path_score": 0.0,
+ "path_fusion_delta_score": 0.0,
+ "path_tunnel_support_score": 0.0,
+ "path_tunnel_delta_score": 0.0,
+ "path_model_score": 0.0,
+ "path_chain_extension_enabled": path_chain_extension_enabled,
+ "path_chain_extension_delta_score": 0.0,
+ "path_chain_extended_score": 0.0,
+ "effective_path_score": 0.0,
+ "temporal_score": 0.0,
+ "raw_public_score": round(float(event_hit.score), 6),
+ "hybrid_score": hybrid_score,
+ "hybrid_score_source": (
+ "learned_event_fusion"
+ if event_fusion_enabled
+ else "learned_event_score"
+ if learned_event_available
+ else "heuristic_event_mix"
+ ),
+ "evidence_snippet_role": "selected_event_representative",
+ "selected_event_rank": int(event_rank),
+ "selected_event_ids": list(selected_event_ids),
+ "selected_path_ids": list(selected_path_ids),
+ }
+ )
+ final_hits.append(
+ MemoryHit(
+ memory_id=event_hit.memory_id,
+ category=event_hit.category,
+ value=event_hit.value,
+ relation=event_hit.relation,
+ anchors=list(event_hit.anchors),
+ score=hybrid_score,
+ source_kind=event_hit.source_kind,
+ slot_key=event_hit.slot_key,
+ state=event_hit.state,
+ turn_index=int(event_hit.turn_index),
+ metadata=metadata,
+ )
+ )
+ if not final_hits:
+ final_hits = list(hits)
+ if profile_first_hits:
+ final_hits = _inject_profile_first_hits(
+ final_hits,
+ profile_first_hits,
+ selected_event_ids=selected_event_ids,
+ selected_path_ids=selected_path_ids,
+ )
+ final_hits = _coverage_preserving_final_hits(final_hits, selected_event_ids=selected_event_ids, top_k=top_k)
+ final_hit_event_ids = _event_ids_from_hits(final_hits)
+ final_missing_selected_event_ids = [event_id for event_id in selected_event_ids if event_id not in set(final_hit_event_ids)]
+ return {
+ "hits": final_hits,
+ "metadata": {
+ "retrieval_mode": "hybrid_node_scored",
+ "hybrid_enabled": True,
+ "hybrid_source": hybrid_source,
+ **memory_router_decision,
+ "profile_first_router_suppressed": bool(profile_first_router_suppressed),
+ "recall_event_ids": list(recall_event_ids),
+ "learned_recall_event_ids": list(learned_recall_event_ids),
+ "model_recall_event_ids": list(model_recall_event_ids),
+ "symbolic_recall_event_ids": list(symbolic_recall_event_ids),
+ "embedder_index_recall_event_ids": list(embedder_index_event_ids),
+ **embedder_index_metadata,
+ "embedder_fusion_mode": embedder_fusion_mode or "off",
+ "embedder_fusion_enabled": bool(embedder_fusion_enabled),
+ "embedder_fusion_weight": round(float(self.embedder_fusion_weight), 6),
+ "embedder_fusion_score_floor": round(float(self.embedder_fusion_score_floor), 6),
+ "embedder_fusion_top_k": int(self.embedder_fusion_top_k),
+ "embedder_fusion_select_k": int(self.embedder_fusion_select_k),
+ "embedder_fusion_max_boost": round(float(self.embedder_fusion_max_boost), 6),
+ "embedder_fusion_event_scores": dict(embedder_fusion_applied_event_scores),
+ "embedder_fusion_boosts": dict(embedder_fusion_boosts),
+ "embedder_fusion_selected_event_ids": list(embedder_fusion_selected_event_ids),
+ "embedder_fusion_selected_path_ids": list(embedder_fusion_selected_path_ids),
+ "profile_first_hybrid_enabled": bool(profile_first_event_ids),
+ "profile_first_event_ids": list(profile_first_event_ids),
+ "profile_first_memory_ids": list(profile_first_memory_ids),
+ "hybrid_candidate_event_ids": list(hybrid_candidate_event_ids),
+ "hybrid_candidate_union_enabled": True,
+ "hybrid_candidate_union_rescored": bool(hybrid_candidate_union_rescored),
+ "hybrid_candidate_union_added_event_ids": list(hybrid_candidate_union_added_event_ids),
+ "hybrid_candidate_union_priority_changed": bool(hybrid_candidate_union_priority_changed),
+ "rerank_candidate_event_ids": list(rerank_candidate_event_ids),
+ "base_event_scores": dict(base_event_scores),
+ "rerank_event_scores": dict(rerank_event_scores),
+ "calibrated_event_scores": dict(calibrated_event_scores),
+ "matrix_event_scores": dict(matrix_event_scores),
+ "event_fusion_delta_scores": dict(event_fusion_delta_scores),
+ "event_tunnel_support_scores": dict(event_tunnel_support_scores),
+ "event_tunnel_delta_scores": dict(event_tunnel_delta_scores),
+ "tri_maze_event_reverse_scores": dict(tri_maze_event_reverse_scores),
+ "tri_maze_event_boundary_scores": dict(tri_maze_event_boundary_scores),
+ "tri_maze_event_reverse_relations": dict(tri_maze_event_reverse_relations),
+ "matrix_rerank_event_ids": list(matrix_rerank_event_ids),
+ "matrix_enabled": matrix_enabled,
+ "rerank_path_scores": dict(rerank_path_scores),
+ "matrix_path_scores": dict(matrix_path_scores),
+ "tri_maze_path_reverse_scores": dict(tri_maze_path_reverse_scores),
+ "tri_maze_path_boundary_scores": dict(tri_maze_path_boundary_scores),
+ "tri_maze_path_reverse_relations": dict(tri_maze_path_reverse_relations),
+ "matrix_path_rerank_ids": list(matrix_path_rerank_ids),
+ "matrix_path_enabled": matrix_path_enabled,
+ "fusion_enabled": fusion_enabled,
+ "event_calibration_enabled": event_calibration_enabled,
+ "path_calibration_enabled": path_calibration_enabled,
+ "event_tunnel_enabled": event_tunnel_enabled,
+ "path_tunnel_enabled": path_tunnel_enabled,
+ "final_event_fusion_enabled": final_event_fusion_enabled,
+ "final_path_fusion_enabled": final_path_fusion_enabled,
+ "decision_fusion_enabled": False,
+ "decision_score_source": "",
+ "event_fusion_enabled": event_fusion_enabled,
+ "path_fusion_enabled": path_fusion_enabled,
+ "selected_event_ids": list(selected_event_ids),
+ "path_rescue_event_ids": list(path_rescue_event_ids),
+ "selected_path_ids": list(selected_path_ids),
+ "final_hit_event_ids": list(final_hit_event_ids),
+ "final_hit_dia_ids": _dia_ids_from_hits(final_hits),
+ "final_missing_selected_event_ids": list(final_missing_selected_event_ids),
+ "selected_event_count": int(len(selected_event_ids)),
+ "selected_path_count": int(len(selected_path_ids)),
+ "temporal_scores": dict(temporal_scores),
+ "recall_event_scores": dict(recall_event_scores),
+ "event_scores": dict(event_scores),
+ "base_path_scores": dict(base_path_scores),
+ "calibrated_path_scores": dict(calibrated_path_scores),
+ "path_fusion_delta_scores": dict(path_fusion_delta_scores),
+ "path_tunnel_support_scores": dict(path_tunnel_support_scores),
+ "path_tunnel_delta_scores": dict(path_tunnel_delta_scores),
+ "path_model_scores": dict(path_model_scores),
+ "path_chain_extension_enabled": path_chain_extension_enabled,
+ "path_chain_extension_delta_scores": dict(path_chain_extension_delta_scores),
+ "path_chain_extended_scores": dict(path_chain_extended_scores),
+ "effective_path_scores": dict(effective_path_scores),
+ "path_scores": dict(path_scores),
+ "answer_type_scores": dict(answer_type_scores),
+ "answer_plan_scores": dict(answer_plan_scores),
+ "answer_plan_support_scores": dict(answer_plan_support_scores),
+ "answer_plan_adjusted_scores": dict(answer_plan_adjusted_scores),
+ "answer_plan_raw_ranked_event_ids": list(answer_plan_raw_ranked_event_ids),
+ "answer_plan_ranked_event_ids": list(answer_plan_ranked_event_ids),
+ "answer_plan_selected_event_ids": list(answer_plan_selected_event_ids),
+ "answer_plan_current_event_ids": list(answer_plan_current_event_ids),
+ "answer_plan_promotion_enabled": bool(answer_plan_promotion_enabled),
+ "answer_plan_promotion_score_margin": round(float(answer_plan_promotion_score_margin), 6),
+ "answer_plan_promotion_min_margin": round(float(answer_plan_promotion_min_margin), 6),
+ "answer_plan_event_selection_threshold": round(float(answer_plan_event_selection_threshold), 6),
+ "answer_plan_event_selection_top_k": int(answer_plan_event_selection_top_k),
+ "focused_answer_type": dominant_answer_type,
+ "preferred_path_types": list(preferred_path_types),
+ },
+ }
+
+ def retrieve(self, query: str, *, top_k: int = 6) -> MemoryRetrieval:
+ self._reload_graph()
+ start = time.perf_counter()
+ candidate_top_k = max(top_k, self.candidate_event_k if self.retrieval_mode == "hybrid_node_scored" else min(max(top_k, 18), 48))
+ payload = self.graph.retrieve(query, top_k=candidate_top_k)
+ hits = [_raw_hit_to_memory_hit(item) for item in payload.get("hits", []) or []]
+ scored_lookup = {hit.memory_id: hit for hit in hits if hit.memory_id}
+ active_hits = _restore_hit_scores([_raw_hit_to_memory_hit(item) for item in payload.get("active_hits", []) or []], scored_lookup)
+ history_hits = _restore_hit_scores([_raw_hit_to_memory_hit(item) for item in payload.get("history_hits", []) or []], scored_lookup)
+ stale_hits = _restore_hit_scores([_raw_hit_to_memory_hit(item) for item in payload.get("stale_hits", []) or []], scored_lookup)
+ overwrite_hits = _restore_hit_scores([_raw_hit_to_memory_hit(item) for item in payload.get("overwrite_hits", []) or []], scored_lookup)
+ false_hits = _restore_hit_scores([_raw_hit_to_memory_hit(item) for item in payload.get("false_hits", []) or []], scored_lookup)
+ public_hits = _public_graph_hits(self.graph) if self.retrieval_mode == "hybrid_node_scored" else []
+ hybrid_payload = self._hybrid_node_scored_hits(query, hits, top_k=top_k, public_hits=public_hits)
+ hits = list(hybrid_payload.get("hits", []) or hits)
+ hybrid_metadata = dict(hybrid_payload.get("metadata", {}) or {})
+ memory_router_decision = _memory_router_decision(
+ hybrid_metadata,
+ mode=self.memory_router_mode,
+ threshold=self.memory_router_threshold,
+ margin=self.memory_router_margin,
+ )
+ current_subject_payload = _current_subject_protected_hits(
+ query=query,
+ graph=self.graph,
+ final_hits=hits,
+ top_k=top_k,
+ )
+ hits = list(current_subject_payload.get("hits", hits) or hits)
+ current_subject_metadata = dict(current_subject_payload.get("metadata", {}) or {})
+ audit_anchor_payload = _audit_anchor_protected_hits(
+ query=query,
+ final_hits=hits,
+ candidate_hits=list(public_hits) + list(active_hits) + list(history_hits),
+ metadata={**dict(payload.get("metadata", {}) or {}), **hybrid_metadata, **current_subject_metadata},
+ top_k=top_k,
+ )
+ hits = list(audit_anchor_payload.get("hits", hits) or hits)
+ audit_anchor_metadata = dict(audit_anchor_payload.get("metadata", {}) or {})
+ identifier_payload = _identifier_protected_hits(
+ query=query,
+ final_hits=hits,
+ candidate_hits=list(public_hits) + list(active_hits) + list(history_hits),
+ top_k=top_k,
+ )
+ hits = list(identifier_payload.get("hits", hits) or hits)
+ identifier_metadata = dict(identifier_payload.get("metadata", {}) or {})
+ if _memory_router_allows(memory_router_decision, "path_tunnel", "topic_tunnel"):
+ depth_chain_payload = _depth_chain_protected_hits(
+ query=query,
+ graph=self.graph,
+ final_hits=hits,
+ top_k=top_k,
+ )
+ hits = list(depth_chain_payload.get("hits", hits) or hits)
+ depth_chain_metadata = dict(depth_chain_payload.get("metadata", {}) or {})
+ else:
+ depth_chain_metadata = {
+ "depth_chain_protected_enabled": False,
+ "depth_chain_router_suppressed": True,
+ }
+ if _memory_router_allows(memory_router_decision, "profile", "resource"):
+ profile_focused_payload = _profile_focused_pack_hits(
+ self.graph,
+ query,
+ hits,
+ top_k=top_k,
+ )
+ hits = list(profile_focused_payload.get("hits", hits) or hits)
+ profile_focused_metadata = dict(profile_focused_payload.get("metadata", {}) or {})
+ else:
+ profile_focused_metadata = {
+ "profile_focused_pack_enabled": False,
+ "profile_focused_router_suppressed": True,
+ }
+ if _memory_router_allows(memory_router_decision, "topic_tunnel"):
+ topic_bucket_payload = _topic_bucket_rerank_hits(self.graph, query, hits, top_k=top_k)
+ topic_bucket_hits = topic_bucket_payload.get("hits", hits)
+ hits = list(hits if topic_bucket_hits is None else topic_bucket_hits)
+ topic_bucket_metadata = dict(topic_bucket_payload.get("metadata", {}) or {})
+ else:
+ topic_bucket_metadata = {
+ "topic_bucket_rerank_enabled": False,
+ "topic_bucket_router_suppressed": True,
+ }
+ temporal_runtime_payload = self._temporal_runtime_pack(query)
+ temporal_evidence_payload = self._apply_temporal_evidence_pack_to_hits(
+ hits,
+ temporal_runtime_payload,
+ top_k=top_k,
+ )
+ hits = list(temporal_evidence_payload.get("hits", hits) or hits)
+ temporal_runtime_metadata = dict(temporal_evidence_payload.get("metadata", {}) or {})
+ injection_planner_payload = self._apply_injection_planner_to_hits(query, hits, top_k=top_k)
+ hits = list(injection_planner_payload.get("hits", hits) or hits)
+ injection_planner_metadata = dict(injection_planner_payload.get("metadata", {}) or {})
+ facet_query_pack_payload = _facet_query_pack_hits(
+ self.graph,
+ query,
+ hits,
+ top_k=top_k,
+ )
+ hits = list(facet_query_pack_payload.get("hits", hits) or hits)
+ facet_query_pack_metadata = dict(facet_query_pack_payload.get("metadata", {}) or {})
+ unit_coverage_mode = _normalize(os.getenv("TMCRA_UNIT_COVERAGE_PACK_MODE", "on"))
+ if unit_coverage_mode in _MULTI_UNIT_CHAIN_DISABLED_MODES:
+ unit_coverage_metadata = {
+ "unit_coverage_pack_enabled": False,
+ "unit_coverage_reason": "disabled",
+ }
+ else:
+ unit_coverage_payload = _unit_coverage_pack_hits(
+ self.graph,
+ query,
+ hits,
+ top_k=top_k,
+ )
+ hits = list(unit_coverage_payload.get("hits", hits) or hits)
+ unit_coverage_metadata = dict(unit_coverage_payload.get("metadata", {}) or {})
+ if _normalize(os.getenv("TMCRA_MULTI_UNIT_CHAIN_SLOT_MODE", "on")) not in _MULTI_UNIT_CHAIN_DISABLED_MODES:
+ multi_unit_chain_slot_payload = _multi_unit_chain_slot_hits(
+ self.graph,
+ query,
+ hits,
+ top_k=top_k,
+ )
+ hits = list(multi_unit_chain_slot_payload.get("hits", hits) or hits)
+ multi_unit_chain_slot_metadata = dict(multi_unit_chain_slot_payload.get("metadata", {}) or {})
+ else:
+ multi_unit_chain_slot_metadata = {
+ "multi_unit_chain_slot_enabled": False,
+ "multi_unit_chain_slot_reason": "disabled",
+ }
+ profile_protected_reinserted_count = 0
+ profile_protected_ids = _dedupe(
+ [
+ *list(profile_focused_metadata.get("profile_focused_pack_memory_ids", []) or []),
+ *list(profile_focused_metadata.get("profile_first_memory_ids", []) or []),
+ ],
+ max_items=max(1, min(6, int(top_k or 1))),
+ )
+ if profile_protected_ids:
+ existing_hits_by_id = {hit.memory_id: hit for hit in hits if hit.memory_id}
+ protected_hits: List[MemoryHit] = []
+ for memory_id in profile_protected_ids:
+ hit = existing_hits_by_id.get(memory_id)
+ if hit is None:
+ record = getattr(self.graph, "records_by_id", {}).get(memory_id)
+ if record is None:
+ continue
+ hit = _memory_hit_from_record(record)
+ metadata = dict(hit.metadata or {})
+ metadata.update(
+ {
+ "profile_first_hybrid_rescue": True,
+ "profile_protected_slot": True,
+ "evidence_snippet_role": "profile_protected_slot",
+ }
+ )
+ protected_hits.append(
+ MemoryHit(
+ memory_id=hit.memory_id,
+ category=hit.category,
+ value=hit.value,
+ relation=hit.relation,
+ anchors=list(hit.anchors),
+ score=max(float(hit.score), 4.8),
+ source_kind=hit.source_kind,
+ slot_key=hit.slot_key,
+ state=hit.state,
+ turn_index=int(hit.turn_index),
+ metadata=metadata,
+ )
+ )
+ if protected_hits:
+ profile_protected_reinserted_count = len(protected_hits)
+ seen_profile_ids = {hit.memory_id for hit in protected_hits if hit.memory_id}
+ hits = [*protected_hits, *[hit for hit in hits if hit.memory_id not in seen_profile_ids]]
+ embedder_fusion_output_event_ids = [
+ _clean_text(event_id)
+ for event_id in list(hybrid_metadata.get("embedder_fusion_selected_event_ids", []) or [])
+ if _clean_text(event_id)
+ ]
+ embedder_fusion_output_reordered = False
+ if embedder_fusion_output_event_ids:
+ before_order = _event_ids_from_hits(hits)
+ seen_output_memory_ids = {hit.memory_id for hit in hits}
+ for event_id in embedder_fusion_output_event_ids:
+ event_hit = _representative_event_hit(_event_record_hits_from_graph(self.graph, event_id), query=query)
+ if event_hit is None or event_hit.memory_id in seen_output_memory_ids:
+ continue
+ metadata = dict(event_hit.metadata or {})
+ metadata.update(
+ {
+ "event_id": event_id,
+ "evidence_snippet_role": "embedder_fusion_event_representative",
+ "embedder_fusion_event_representative": True,
+ }
+ )
+ hits.append(
+ MemoryHit(
+ memory_id=event_hit.memory_id,
+ category=event_hit.category,
+ value=event_hit.value,
+ relation=event_hit.relation,
+ anchors=list(event_hit.anchors),
+ score=float(event_hit.score),
+ source_kind=event_hit.source_kind,
+ slot_key=event_hit.slot_key,
+ state=event_hit.state,
+ turn_index=int(event_hit.turn_index),
+ metadata=metadata,
+ )
+ )
+ seen_output_memory_ids.add(event_hit.memory_id)
+ hits = _coverage_preserving_final_hits(
+ hits,
+ selected_event_ids=_dedupe([*embedder_fusion_output_event_ids, *before_order]),
+ top_k=top_k,
+ )
+ embedder_fusion_output_reordered = before_order != _event_ids_from_hits(hits)
+ retrieval_context_tokens = int(payload.get("context_token_estimate", _estimate_tokens_from_hits(hits)))
+ self._last_retrieval_context_tokens = retrieval_context_tokens
+ result = MemoryRetrieval(
+ concepts=list(payload.get("concepts", []) or []),
+ relations=list(payload.get("relations", []) or []),
+ hits=hits,
+ active_hits=active_hits,
+ history_hits=history_hits,
+ stale_hits=stale_hits,
+ overwrite_hits=overwrite_hits,
+ false_hits=false_hits,
+ retrieval_seconds=time.perf_counter() - start,
+ context_token_estimate=retrieval_context_tokens,
+ retrieval_context_token_estimate=retrieval_context_tokens,
+ metadata={
+ "query_id": payload.get("query_id", ""),
+ **dict(payload.get("metadata", {}) or {}),
+ **hybrid_metadata,
+ **memory_router_decision,
+ **current_subject_metadata,
+ **audit_anchor_metadata,
+ **identifier_metadata,
+ **depth_chain_metadata,
+ **profile_focused_metadata,
+ **topic_bucket_metadata,
+ **temporal_runtime_metadata,
+ **injection_planner_metadata,
+ **facet_query_pack_metadata,
+ **unit_coverage_metadata,
+ **multi_unit_chain_slot_metadata,
+ "profile_protected_reinserted_count": profile_protected_reinserted_count,
+ "embedder_fusion_output_event_ids": list(embedder_fusion_output_event_ids),
+ "embedder_fusion_output_reordered": bool(embedder_fusion_output_reordered),
+ },
+ )
+ self._persist_graph()
+ return result
+
+ def export_dialog_graph(self, *, mode: str = "light") -> Dict[str, Any]:
+ self._reload_graph()
+ return self.graph.export_graph(
+ snapshot_points=(1000, 5000, 10000, 20000, 50000, 100000, 200000, 300000, 500000),
+ mode=mode,
+ )
+
+ def export_dialog_graph_mermaid(self) -> str:
+ self._reload_graph()
+ return self.graph.export_mermaid()
+
+ def register_answer_support(self, *, answer_id: str, memory_ids: List[str], query_id: str = "", answer_text: str = "") -> None:
+ self._reload_graph()
+ self.graph.register_answer_support(answer_id=answer_id, memory_ids=memory_ids, query_id=query_id, answer_text=answer_text)
+ self._persist_graph()
+
+ def telemetry_snapshot(self) -> Dict[str, Any]:
+ self._reload_graph()
+ return self.graph.summary()
+
+ def stats(self) -> Dict[str, Any]:
+ self._reload_graph()
+ storage = self._storage_breakdown()
+ return _state_stats(
+ storage_bytes=storage["storage_bytes"],
+ retrieval_context_tokens=self._last_retrieval_context_tokens,
+ total_state_tokens=storage["total_state_token_estimate"],
+ core_storage_bytes=storage["core_storage_bytes"],
+ audit_storage_bytes=storage["audit_storage_bytes"],
+ core_state_token_estimate=storage["core_state_token_estimate"],
+ audit_state_token_estimate=storage["audit_state_token_estimate"],
+ lightweight_stats=bool(self.lightweight_stats),
+ **self.graph.summary(),
+ )
+
+ def storage_bytes(self) -> int:
+ self._reload_graph()
+ return self._storage_breakdown()["storage_bytes"]
+
+ def build_prompt_context(self, query: str, *, top_k: int = 8) -> Dict[str, Any]:
+ retrieval = self.retrieve(query, top_k=top_k)
+ retrieval_payload = retrieval.to_dict()
+ stats = self.stats()
+ context_summary = _graph_prompt_state_summary(self.graph, retrieval)
+ prompt_context_payload = {
+ "query": query,
+ "retrieval": retrieval_payload,
+ "context_summary": context_summary,
+ }
+ prompt_context_chars = len(json.dumps(prompt_context_payload, ensure_ascii=False))
+ prompt_context_tokens_est = _estimate_tokens(json.dumps(prompt_context_payload, ensure_ascii=False))
+ return {
+ "mode": "graph_session_memory_v2",
+ "query": query,
+ "retrieval": retrieval_payload,
+ "stats": stats,
+ "state": context_summary,
+ "context_summary": context_summary,
+ "prompt_context_chars": int(prompt_context_chars),
+ "prompt_context_tokens_est": int(prompt_context_tokens_est),
+ "context_truncated": bool(context_summary.get("context_truncated", False)),
+ "truncation_reason": _clean_text(context_summary.get("truncation_reason", "")),
+ }
+
+
+class SummaryWindowMemoryAdapter(MemoryAdapter):
+ name = "summary_window_memory"
+
+ def __init__(self, *, window_size: int = 24, auto_extract: bool = False) -> None:
+ self.extractor = SessionMemoryExtractor()
+ self.window_size = max(4, int(window_size))
+ self.turn_index = 0
+ self.active_slots: Dict[str, SessionMemoryRecordV2] = {}
+ self.recent_turns: deque[Dict[str, Any]] = deque(maxlen=self.window_size)
+ self.auto_extract = bool(auto_extract)
+ self._last_retrieval_context_tokens = 0
+
+ def reset(self) -> None:
+ self.turn_index = 0
+ self.active_slots = {}
+ self.recent_turns = deque(maxlen=self.window_size)
+ self._last_retrieval_context_tokens = 0
+
+ def ingest_turn(
+ self,
+ user_text: str,
+ assistant_text: str = "",
+ *,
+ answer_payload: Dict[str, Any] | None = None,
+ extraction_result: Dict[str, Any] | None = None,
+ ) -> None:
+ self.turn_index += 1
+ records = _build_turn_records(
+ self.extractor,
+ user_text=user_text,
+ answer_payload=answer_payload,
+ extraction_result=extraction_result,
+ turn_index=self.turn_index,
+ allow_auto_extract=self.auto_extract,
+ )
+ for record in records:
+ previous = self.active_slots.get(record.slot_key)
+ if previous:
+ previous.state = "superseded"
+ record.supersedes.append(previous.memory_id)
+ record.state = "active"
+ self.active_slots[record.slot_key] = record
+ self.recent_turns.append(
+ {
+ "turn_index": self.turn_index,
+ "text": _clean_text(user_text),
+ "assistant": _clean_text(assistant_text),
+ }
+ )
+
+ def retrieve(self, query: str, *, top_k: int = 6) -> MemoryRetrieval:
+ start = time.perf_counter()
+ query_tokens = set(_tokenize(query))
+ hints = set(infer_category_hints(query))
+ scored: List[tuple[float, SessionMemoryRecordV2]] = []
+ for record in self.active_slots.values():
+ token_set = record.token_set()
+ overlap = len(query_tokens & token_set) if query_tokens and token_set else 0
+ score = overlap / max(1, len(query_tokens | token_set)) if query_tokens and token_set else 0.0
+ if hints and record.category in hints:
+ score += 0.22
+ if record.slot_key.lower() in _normalize(query):
+ score += 0.12
+ score += min(0.08, record.turn_index * 0.0004)
+ score += 0.2
+ if score > 0:
+ scored.append((score, record))
+ scored.sort(key=lambda item: (item[0], item[1].turn_index), reverse=True)
+ selected_records = [record for _, record in scored[:top_k]]
+ hits = [
+ MemoryHit(
+ memory_id=record.memory_id,
+ category=record.category,
+ value=record.value,
+ relation=record.relation,
+ anchors=list(record.anchor_concepts),
+ score=float(score),
+ source_kind=record.source_kind,
+ slot_key=record.slot_key,
+ state=record.state,
+ turn_index=record.turn_index,
+ metadata={"window_size": self.window_size},
+ )
+ for score, record in scored[:top_k]
+ ]
+ concepts = []
+ relations = []
+ for hit in hits:
+ concepts.append({"concept": hit.value, "type": hit.category, "source_kind": hit.source_kind})
+ for anchor in hit.anchors[:2]:
+ concepts.append({"concept": anchor, "type": "context", "source_kind": hit.source_kind})
+ relation = _relation_hit(hit, weight_bias=0.06)
+ if relation:
+ relations.append(relation)
+ retrieval_context_tokens = _estimate_tokens_from_hits(hits)
+ self._last_retrieval_context_tokens = retrieval_context_tokens
+ return MemoryRetrieval(
+ concepts=concepts,
+ relations=relations,
+ hits=hits,
+ active_hits=list(hits),
+ retrieval_seconds=time.perf_counter() - start,
+ context_token_estimate=retrieval_context_tokens,
+ retrieval_context_token_estimate=retrieval_context_tokens,
+ metadata={
+ "records": len(self.active_slots),
+ "window_size": self.window_size,
+ "recent_turns": len(self.recent_turns),
+ },
+ )
+
+ def stats(self) -> Dict[str, Any]:
+ payload = {
+ "active_slots": {slot: record.to_dict() for slot, record in self.active_slots.items()},
+ "recent_turns": list(self.recent_turns),
+ }
+ total_state_tokens = _estimate_tokens(json.dumps(payload, ensure_ascii=False))
+ return _state_stats(
+ storage_bytes=self.storage_bytes(),
+ retrieval_context_tokens=self._last_retrieval_context_tokens,
+ total_state_tokens=total_state_tokens,
+ records=len(self.active_slots),
+ active_slots=len(self.active_slots),
+ recent_turns=len(self.recent_turns),
+ )
+
+ def storage_bytes(self) -> int:
+ payload = {
+ "active_slots": {slot: record.to_dict() for slot, record in self.active_slots.items()},
+ "recent_turns": list(self.recent_turns),
+ }
+ return len(json.dumps(payload, ensure_ascii=False).encode("utf-8"))
+
+ def build_prompt_context(self, query: str, *, top_k: int = 8) -> Dict[str, Any]:
+ return {
+ "mode": "summary_window_memory",
+ "query": query,
+ "retrieval": self.retrieve(query, top_k=top_k).to_dict(),
+ "stats": self.stats(),
+ "state": {
+ "active_slots": {slot: record.to_dict() for slot, record in self.active_slots.items()},
+ "recent_turns": list(self.recent_turns),
+ },
+ }
+
+
+@dataclass(slots=True)
+class _VectorRecord:
+ memory_id: str
+ category: str
+ value: str
+ relation: str
+ anchors: List[str]
+ tokens: List[str]
+ turn_index: int
+ slot_key: str = ""
+ active: bool = True
+ source_kind: str = "vector_memory"
+ metadata: Dict[str, Any] = field(default_factory=dict)
+
+
+class VectorRAGMemoryAdapter(MemoryAdapter):
+ name = "vector_rag_memory"
+
+ def __init__(self, *, auto_extract: bool = False) -> None:
+ self.extractor = SessionMemoryExtractor()
+ self.records: List[_VectorRecord] = []
+ self.turn_index = 0
+ self.auto_extract = bool(auto_extract)
+ self._last_retrieval_context_tokens = 0
+
+ def reset(self) -> None:
+ self.records = []
+ self.turn_index = 0
+ self._last_retrieval_context_tokens = 0
+
+ def ingest_turn(
+ self,
+ user_text: str,
+ assistant_text: str = "",
+ *,
+ answer_payload: Dict[str, Any] | None = None,
+ extraction_result: Dict[str, Any] | None = None,
+ ) -> None:
+ _ = assistant_text
+ self.turn_index += 1
+ records = _build_turn_records(
+ self.extractor,
+ user_text=user_text,
+ answer_payload=answer_payload,
+ extraction_result=extraction_result,
+ turn_index=self.turn_index,
+ allow_auto_extract=self.auto_extract,
+ )
+ for record in records:
+ if record.slot_key:
+ for previous in self.records:
+ if previous.slot_key == record.slot_key and previous.active:
+ previous.active = False
+ self.records.append(
+ _VectorRecord(
+ memory_id=record.memory_id,
+ category=record.category,
+ value=record.value,
+ relation=record.relation,
+ anchors=list(record.anchor_concepts),
+ tokens=list(record.token_set()),
+ turn_index=record.turn_index,
+ slot_key=record.slot_key,
+ active=record.state == "active",
+ source_kind=record.source_kind,
+ metadata=dict(record.metadata),
+ )
+ )
+
+ def retrieve(self, query: str, *, top_k: int = 6) -> MemoryRetrieval:
+ start = time.perf_counter()
+ query_tokens = set(_tokenize(query))
+ hints = set(infer_category_hints(query))
+ scored: List[tuple[float, _VectorRecord]] = []
+ for record in self.records:
+ token_set = set(record.tokens)
+ overlap = len(query_tokens & token_set) if query_tokens and token_set else 0
+ score = overlap / max(1, len(query_tokens | token_set)) if query_tokens and token_set else 0.0
+ if hints and record.category in hints:
+ score += 0.18
+ if record.slot_key and record.slot_key.lower() in _normalize(query):
+ score += 0.1
+ score += min(0.12, record.turn_index * 0.0004)
+ score += 0.18 if record.active else -0.25
+ if score > 0:
+ scored.append((score, record))
+ if not scored:
+ for record in self.records[-top_k:]:
+ scored.append((0.05 + (0.15 if record.active else 0.0), record))
+ scored.sort(key=lambda item: (item[0], item[1].active, item[1].turn_index), reverse=True)
+ hits = [
+ MemoryHit(
+ memory_id=record.memory_id,
+ category=record.category,
+ value=record.value,
+ relation=record.relation,
+ anchors=list(record.anchors),
+ score=float(score),
+ source_kind=record.source_kind,
+ slot_key=record.slot_key,
+ state="active" if record.active else "superseded",
+ turn_index=record.turn_index,
+ metadata=dict(record.metadata),
+ )
+ for score, record in scored[:top_k]
+ ]
+ concepts = []
+ relations = []
+ for hit in hits:
+ concepts.append({"concept": hit.value, "type": hit.category, "source_kind": hit.source_kind})
+ for anchor in hit.anchors[:2]:
+ concepts.append({"concept": anchor, "type": "context", "source_kind": hit.source_kind})
+ relation = _relation_hit(hit)
+ if relation:
+ relations.append(relation)
+ retrieval_context_tokens = _estimate_tokens_from_hits(hits)
+ self._last_retrieval_context_tokens = retrieval_context_tokens
+ return MemoryRetrieval(
+ concepts=concepts,
+ relations=relations,
+ hits=hits,
+ active_hits=[hit for hit in hits if hit.state == "active"],
+ history_hits=[hit for hit in hits if hit.state != "active"],
+ retrieval_seconds=time.perf_counter() - start,
+ context_token_estimate=retrieval_context_tokens,
+ retrieval_context_token_estimate=retrieval_context_tokens,
+ metadata={
+ "records": len(self.records),
+ "active_records": sum(1 for item in self.records if item.active),
+ },
+ )
+
+ def stats(self) -> Dict[str, Any]:
+ payload = [
+ {
+ "memory_id": record.memory_id,
+ "category": record.category,
+ "value": record.value,
+ "relation": record.relation,
+ "anchors": record.anchors,
+ "slot_key": record.slot_key,
+ "active": record.active,
+ }
+ for record in self.records
+ ]
+ total_state_tokens = _estimate_tokens(json.dumps(payload, ensure_ascii=False))
+ return _state_stats(
+ storage_bytes=self.storage_bytes(),
+ retrieval_context_tokens=self._last_retrieval_context_tokens,
+ total_state_tokens=total_state_tokens,
+ records=len(self.records),
+ active_records=sum(1 for item in self.records if item.active),
+ )
+
+ def storage_bytes(self) -> int:
+ payload = [
+ {
+ "memory_id": record.memory_id,
+ "category": record.category,
+ "value": record.value,
+ "relation": record.relation,
+ "anchors": record.anchors,
+ "slot_key": record.slot_key,
+ "active": record.active,
+ }
+ for record in self.records
+ ]
+ return len(json.dumps(payload, ensure_ascii=False).encode("utf-8"))
+
+ def build_prompt_context(self, query: str, *, top_k: int = 8) -> Dict[str, Any]:
+ state = [
+ {
+ "memory_id": record.memory_id,
+ "category": record.category,
+ "value": record.value,
+ "relation": record.relation,
+ "anchors": list(record.anchors),
+ "slot_key": record.slot_key,
+ "active": bool(record.active),
+ "turn_index": int(record.turn_index),
+ }
+ for record in self.records
+ ]
+ return {
+ "mode": "vector_rag_memory",
+ "query": query,
+ "retrieval": self.retrieve(query, top_k=top_k).to_dict(),
+ "stats": self.stats(),
+ "state": state,
+ }
+
+
+class FullHistoryMemoryAdapter(MemoryAdapter):
+ name = "full_history_memory"
+
+ def __init__(self) -> None:
+ self.turns: List[Dict[str, str]] = []
+ self._last_retrieval_context_tokens = 0
+
+ def reset(self) -> None:
+ self.turns = []
+ self._last_retrieval_context_tokens = 0
+
+ def ingest_turn(
+ self,
+ user_text: str,
+ assistant_text: str = "",
+ *,
+ answer_payload: Dict[str, Any] | None = None,
+ extraction_result: Dict[str, Any] | None = None,
+ ) -> None:
+ _ = answer_payload, extraction_result
+ self.turns.append({"user": _clean_text(user_text), "assistant": _clean_text(assistant_text)})
+
+ def retrieve(self, query: str, *, top_k: int = 6) -> MemoryRetrieval:
+ start = time.perf_counter()
+ query_tokens = set(_tokenize(query))
+ scored: List[tuple[float, Dict[str, str], int]] = []
+ for index, turn in enumerate(self.turns):
+ combined = f"{turn.get('user', '')} {turn.get('assistant', '')}"
+ token_set = set(_tokenize(combined))
+ if not token_set:
+ continue
+ overlap = len(query_tokens & token_set) if query_tokens else 0
+ score = overlap / max(1, len(query_tokens | token_set)) if query_tokens else 0.0
+ scored.append((score, turn, index))
+ scored.sort(key=lambda item: (item[0], item[2]), reverse=True)
+ hits = [
+ MemoryHit(
+ memory_id=f"turn:{index}",
+ category="history_turn",
+ value=turn.get("user", ""),
+ relation="conversation_context",
+ anchors=[turn.get("assistant", "")] if turn.get("assistant") else [],
+ score=float(score),
+ source_kind="full_history",
+ slot_key=f"turn.{index}",
+ state="active",
+ turn_index=index + 1,
+ )
+ for score, turn, index in scored[:top_k]
+ if turn.get("user", "")
+ ]
+ retrieval_context_tokens = _estimate_tokens(json.dumps(self.turns, ensure_ascii=False))
+ self._last_retrieval_context_tokens = retrieval_context_tokens
+ return MemoryRetrieval(
+ hits=hits,
+ active_hits=list(hits),
+ retrieval_seconds=time.perf_counter() - start,
+ context_token_estimate=retrieval_context_tokens,
+ retrieval_context_token_estimate=retrieval_context_tokens,
+ metadata={"records": len(self.turns)},
+ )
+
+ def stats(self) -> Dict[str, Any]:
+ total_state_tokens = _estimate_tokens(json.dumps(self.turns, ensure_ascii=False))
+ return _state_stats(
+ storage_bytes=self.storage_bytes(),
+ retrieval_context_tokens=self._last_retrieval_context_tokens,
+ total_state_tokens=total_state_tokens,
+ records=len(self.turns),
+ )
+
+ def storage_bytes(self) -> int:
+ return len(json.dumps(self.turns, ensure_ascii=False).encode("utf-8"))
+
+ def build_prompt_context(self, query: str, *, top_k: int = 8) -> Dict[str, Any]:
+ return {
+ "mode": "full_history_memory",
+ "query": query,
+ "retrieval": self.retrieve(query, top_k=top_k).to_dict(),
+ "stats": self.stats(),
+ "state": {
+ "turns": list(self.turns),
+ },
+ }
diff --git a/code/profile_layer.py b/code/profile_layer.py
index 942eb49545fc1e636be73c9ddd3c5059e1f9173b..b2fe5d75b3eb7d3ab1c94fb7150c6bb22f583964 100644
--- a/code/profile_layer.py
+++ b/code/profile_layer.py
@@ -1,748 +1,748 @@
-from __future__ import annotations
-
-import re
-from typing import Any, Iterable, Mapping, Sequence
-
-
-PROFILE_CATEGORIES = {"profile", "preference", "goal", "constraint", "stage_state", "status"}
-PROFILE_TYPES = {"setup", "preference", "constraint", "goal", "avoid", "usage_context"}
-PROFILE_SEMANTIC_SLOTS = {"identity", "research_topic", "education", "occupation"}
-PROFILE_AGGREGATE_SOURCE_KIND = "public_dialog_profile"
-PROFILE_AGGREGATE_CATEGORY = "profile"
-PROFILE_CLUSTER_SOURCE_KIND = "public_dialog_profile_cluster"
-PROFILE_CLUSTER_CATEGORY = "profile"
-PROFILE_CONSOLIDATOR_VERSION = "profile_consolidator_v1_structured_summary"
-
-_PROFILE_QUERY_MARKERS = (
- "preference",
- "prefer",
- "like",
- "dislike",
- "recommend",
- "suggest",
- "advice",
- "advise",
- "any advice",
- "any suggestions",
- "any tips",
- "tips",
- "trouble with",
- "struggling with",
- "what do you think",
- "learn more",
- "resources",
- "suited",
- "fit me",
- "for me",
- "based on me",
- "based on my",
- "my setup",
- "my profile",
- "my goal",
- "my constraint",
- "my occupation",
- "my previous occupation",
- "my role",
- "my previous role",
- "my job",
- "my previous job",
- "what was my",
- "where did i work",
- "worked as",
- "occupation",
- "previous occupation",
- "role",
- "previous role",
- "job",
- "career",
- "background",
- "identity",
- "experience",
- "avoid",
- "what should i",
- "should i",
- "what should",
- "serve",
- "dinner",
- "homegrown",
- "ingredients",
- "battery life",
- "getting around",
- "偏好",
- "喜欢",
- "不喜欢",
- "推荐",
- "建议",
- "适合我",
- "根据我",
- "我的情况",
- "我的配置",
- "我的目标",
- "我的约束",
- "画像",
- "避免",
-)
-
-
-def _clean_text(value: Any) -> str:
- return re.sub(r"\s+", " ", str(value or "").strip())
-
-
-def _normalize(value: Any) -> str:
- return _clean_text(value).lower()
-
-
-def _tokens(value: Any) -> list[str]:
- text = _normalize(value)
- english = re.findall(r"[a-z0-9_]+", text)
- cjk = [char for char in text if "\u4e00" <= char <= "\u9fff"]
- return _dedupe([*english, *cjk])
-
-
-def _slug(value: Any, *, fallback: str = "general") -> str:
- text = _normalize(value)
- parts = re.findall(r"[a-z0-9]+|[\u4e00-\u9fff]", text)
- slug = "_".join(parts[:10]).strip("_")
- return slug or fallback
-
-
-def _dedupe(items: Iterable[Any], *, max_items: int | None = None) -> list[str]:
- values: list[str] = []
- seen: set[str] = set()
- for item in items:
- text = _clean_text(item)
- if not text:
- continue
- key = _normalize(text)
- if key in seen:
- continue
- seen.add(key)
- values.append(text)
- if max_items is not None and len(values) >= max_items:
- break
- return values
-
-
-def _first_nonempty(*values: Any) -> str:
- for value in values:
- text = _clean_text(value)
- if text:
- return text
- return ""
-
-
-def _bounded_union(*groups: Iterable[Any], max_items: int) -> list[str]:
- return _dedupe([item for group in groups for item in group], max_items=max_items)
-
-
-def _bounded_int_union(*groups: Iterable[Any], max_items: int) -> list[int]:
- values: list[int] = []
- seen: set[int] = set()
- for group in groups:
- for item in group:
- try:
- value = int(item)
- except Exception:
- continue
- if value in seen:
- continue
- seen.add(value)
- values.append(value)
- if len(values) >= max_items:
- return sorted(values)
- return sorted(values)
-
-
-def is_profile_layer_record(*, category: Any, source_kind: Any = "", semantic_slot: Any = "", metadata: Mapping[str, Any] | None = None) -> bool:
- data = dict(metadata or {})
- if bool(data.get("profile_layer")):
- return True
- category_text = _normalize(category)
- slot_text = _normalize(semantic_slot or data.get("semantic_slot", ""))
- source_text = _normalize(source_kind)
- if category_text in PROFILE_CATEGORIES:
- return True
- if slot_text in PROFILE_SEMANTIC_SLOTS or slot_text.startswith("profile_"):
- return True
- return source_text in {
- "public_dialog_profile",
- "public_dialog_profile_cluster",
- "public_dialog_preference",
- "public_dialog_goal",
- "public_dialog_constraint",
- }
-
-
-def infer_profile_type(*, category: Any, semantic_slot: Any = "", relation: Any = "", value: Any = "", metadata: Mapping[str, Any] | None = None) -> str:
- data = dict(metadata or {})
- explicit = _normalize(data.get("profile_type", ""))
- if explicit in PROFILE_TYPES:
- return explicit
- category_text = _normalize(category)
- combined = _normalize(f"{semantic_slot} {relation} {value}")
- if category_text == "preference" or any(marker in combined for marker in ("prefer", "like", "default", "偏好", "喜欢", "默认")):
- return "preference"
- if category_text == "constraint" or any(marker in combined for marker in ("must", "cannot", "forbid", "constraint", "必须", "不能", "约束")):
- return "constraint"
- if category_text == "goal" or any(marker in combined for marker in ("goal", "target", "objective", "目标")):
- return "goal"
- if any(marker in combined for marker in ("avoid", "dislike", "do not", "don't", "避免", "不喜欢", "不要")):
- return "avoid"
- if any(marker in combined for marker in ("setup", "environment", "workflow", "current", "配置", "环境", "流程", "当前")):
- return "setup"
- return "usage_context"
-
-
-def infer_profile_domain(
- *,
- category: Any,
- semantic_slot: Any = "",
- slot_key: Any = "",
- anchors: Sequence[Any] = (),
- value: Any = "",
- metadata: Mapping[str, Any] | None = None,
-) -> tuple[str, str]:
- data = dict(metadata or {})
- explicit = _first_nonempty(data.get("profile_domain", ""), data.get("profile_domain_label", ""), data.get("domain", ""))
- if explicit:
- return _slug(explicit), explicit
- subject = _first_nonempty(data.get("subject", ""), data.get("extracted_subject", ""))
- if subject:
- return _slug(subject), subject
- slot = _normalize(semantic_slot)
- if slot and slot not in {"profile", "preference", "goal", "constraint", "event", "fact", "status"}:
- return _slug(slot), slot.replace("_", " ")
- for anchor in anchors:
- anchor_text = _clean_text(anchor)
- if not anchor_text:
- continue
- if re.search(r"^\d{4}|\d{1,2}\s+[A-Za-z]+|^[A-Z][a-z]+$", anchor_text):
- continue
- return _slug(anchor_text), anchor_text
- slot_key_text = _clean_text(slot_key)
- if slot_key_text:
- tail = slot_key_text.split(".")[-1].replace("_", " ")
- if tail:
- return _slug(tail), tail
- value_tokens = _tokens(value)
- if value_tokens:
- label = " ".join(value_tokens[:4])
- return _slug(label), label
- return _slug(category), _normalize(category) or "general"
-
-
-def profile_candidate_metadata(
- *,
- category: Any,
- semantic_slot: Any = "",
- relation: Any = "",
- value: Any = "",
- source_span: Any = "",
- slot_key: Any = "",
- anchors: Sequence[Any] = (),
- subject: Any = "",
- subject_signature: Any = "",
- proposal: Mapping[str, Any] | None = None,
-) -> dict[str, Any]:
- proposal_data = dict(proposal or {})
- base_metadata = {
- "profile_type": proposal_data.get("profile_type", ""),
- "profile_domain": proposal_data.get("profile_domain", ""),
- "profile_domain_label": proposal_data.get("profile_domain_label", ""),
- "subject": subject,
- "extracted_subject": proposal_data.get("extracted_subject", ""),
- }
- if not is_profile_layer_record(category=category, semantic_slot=semantic_slot, metadata=base_metadata):
- return {}
- profile_type = infer_profile_type(
- category=category,
- semantic_slot=semantic_slot,
- relation=relation,
- value=value or source_span,
- metadata=proposal_data,
- )
- domain, domain_label = infer_profile_domain(
- category=category,
- semantic_slot=semantic_slot,
- slot_key=slot_key,
- anchors=anchors,
- value=value or source_span,
- metadata={**proposal_data, "subject": subject},
- )
- normalized_subject_signature = _slug(subject_signature or subject or domain, fallback=domain)
- route_terms = _dedupe(
- [
- profile_type,
- domain_label,
- semantic_slot,
- subject,
- *list(anchors or []),
- ],
- max_items=12,
- )
- return {
- "profile_layer": True,
- "profile_candidate_status": _clean_text(proposal_data.get("profile_candidate_status", "")) or "writer_candidate",
- "profile_consolidation_stage": _clean_text(proposal_data.get("profile_consolidation_stage", "")) or "pre_consolidation",
- "profile_type": profile_type,
- "profile_domain": domain,
- "profile_domain_label": domain_label,
- "profile_subject_signature": normalized_subject_signature,
- "profile_support_key": f"{profile_type}:{domain}:{normalized_subject_signature}",
- "profile_route_terms": route_terms,
- }
-
-
-def profile_aggregate_slot_key(metadata: Mapping[str, Any]) -> str:
- data = dict(metadata or {})
- support_key = _clean_text(data.get("profile_support_key", ""))
- if support_key:
- return f"tmcra.profile.aggregate.{_slug(support_key)}"
- profile_type = _normalize(data.get("profile_type", "")) or "usage_context"
- domain = _normalize(data.get("profile_domain", "")) or _slug(data.get("profile_domain_label", "general"))
- subject = _normalize(data.get("profile_subject_signature", "")) or domain
- return f"tmcra.profile.aggregate.{_slug(f'{profile_type}:{domain}:{subject}')}"
-
-
-def profile_aggregate_value(
- *,
- profile_type: Any,
- domain_label: Any,
- support_values: Sequence[Any],
-) -> str:
- typed = _clean_text(profile_type) or "usage_context"
- domain = _clean_text(domain_label) or "general"
- values = _dedupe(support_values, max_items=5)
- if not values:
- return f"User {typed} profile for {domain}."
- return f"User {typed} profile for {domain}: " + "; ".join(values)
-
-
-def _profile_output_kind(profile_types: Sequence[Any]) -> str:
- normalized = {_normalize(item) for item in profile_types if _clean_text(item)}
- if "constraint" in normalized:
- return "constraint_profile"
- if "goal" in normalized:
- return "goal_profile"
- if normalized.intersection({"preference", "avoid"}):
- return "preference_profile"
- if "setup" in normalized:
- return "setup_profile"
- return "usage_context_profile"
-
-
-def _profile_update_policy(profile_types: Sequence[Any]) -> str:
- normalized = {_normalize(item) for item in profile_types if _clean_text(item)}
- if "constraint" in normalized:
- return "preserve_until_explicitly_changed"
- if normalized.intersection({"preference", "avoid", "goal", "setup"}):
- return "update_on_newer_user_evidence"
- return "background_context"
-
-
-def _profile_memory_type(profile_types: Sequence[Any]) -> str:
- normalized = {_normalize(item) for item in profile_types if _clean_text(item)}
- if "constraint" in normalized:
- return "hard_constraint"
- if normalized.intersection({"preference", "avoid"}):
- return "durable_preference"
- return "profile_context"
-
-
-def profile_summary(
- *,
- profile_types: Sequence[Any],
- domain_label: Any,
- support_values: Sequence[Any],
- stage: str,
-) -> str:
- kind = _profile_output_kind(profile_types)
- domain = _clean_text(domain_label) or "general"
- values = _dedupe(support_values, max_items=6 if stage == "cluster" else 4)
- prefix = {
- "constraint_profile": "User constraint profile",
- "goal_profile": "User goal profile",
- "preference_profile": "User preference profile",
- "setup_profile": "User setup profile",
- "usage_context_profile": "User usage-context profile",
- }.get(kind, "User profile")
- if not values:
- return f"{prefix} for {domain}."
- return f"{prefix} for {domain}: " + "; ".join(values)
-
-
-def build_profile_aggregate_metadata(
- *,
- support_record_id: Any,
- support_turn_index: int,
- support_value: Any,
- support_anchors: Sequence[Any],
- support_metadata: Mapping[str, Any],
- existing_metadata: Mapping[str, Any] | None = None,
-) -> dict[str, Any]:
- support = dict(support_metadata or {})
- existing = dict(existing_metadata or {})
- profile_type = _normalize(support.get("profile_type", "")) or "usage_context"
- domain = _normalize(support.get("profile_domain", "")) or _slug(support.get("profile_domain_label", "general"))
- domain_label = _clean_text(support.get("profile_domain_label", "")) or domain.replace("_", " ")
- subject_signature = _clean_text(support.get("profile_subject_signature", "")) or domain
- support_ids = _bounded_union(existing.get("profile_support_ids", []) or [], [support_record_id], max_items=64)
- support_turns = _bounded_int_union(existing.get("profile_support_turns", []) or [], [support_turn_index], max_items=64)
- support_values = _bounded_union(existing.get("profile_support_values", []) or [], [support_value], max_items=12)
- support_route_terms = _bounded_union(
- existing.get("profile_route_terms", []) or [],
- support.get("profile_route_terms", []) or [],
- support_anchors,
- max_items=24,
- )
- profile_types = [profile_type]
- output_kind = _profile_output_kind(profile_types)
- memory_type = _profile_memory_type(profile_types)
- summary = profile_summary(
- profile_types=profile_types,
- domain_label=domain_label,
- support_values=support_values,
- stage="aggregate",
- )
- value = profile_aggregate_value(
- profile_type=profile_type,
- domain_label=domain_label,
- support_values=support_values,
- )
- return {
- **existing,
- "profile_layer": True,
- "profile_candidate_status": "consolidated",
- "profile_consolidation_stage": "aggregate",
- "profile_consolidator_version": PROFILE_CONSOLIDATOR_VERSION,
- "profile_aggregate_node": True,
- "profile_type": profile_type,
- "profile_domain": domain,
- "profile_domain_label": domain_label,
- "profile_subject_signature": subject_signature,
- "profile_support_key": f"{profile_type}:{domain}:{subject_signature}",
- "profile_support_ids": support_ids,
- "profile_support_turns": support_turns,
- "profile_support_values": support_values,
- "profile_support_count": len(support_ids),
- "profile_route_terms": support_route_terms,
- "profile_value": value,
- "profile_summary": summary,
- "profile_output_kind": output_kind,
- "profile_update_policy": _profile_update_policy(profile_types),
- "profile_conflict_policy": "latest_active_support_only",
- "profile_evidence_count": len(support_ids),
- "memory_type": memory_type,
- "durable_memory_type": memory_type,
- "memory_chain_depth_layer": "profile",
- "depth_layer": "profile",
- }
-
-
-_PROFILE_CLUSTER_STOPWORDS = {
- "user",
- "profile",
- "preference",
- "preferences",
- "constraint",
- "constraints",
- "goal",
- "goals",
- "avoid",
- "usage",
- "context",
- "general",
- "default",
- "should",
- "would",
- "could",
- "want",
- "wants",
- "need",
- "needs",
- "when",
- "then",
- "than",
- "with",
- "that",
- "this",
- "from",
- "into",
- "instead",
-}
-
-
-def profile_cluster_tokens(*values: Any) -> list[str]:
- tokens: list[str] = []
- for value in values:
- if isinstance(value, (list, tuple, set)):
- for item in value:
- tokens.extend(_tokens(item))
- else:
- tokens.extend(_tokens(value))
- filtered = [
- token
- for token in tokens
- if token
- and token not in _PROFILE_CLUSTER_STOPWORDS
- and not token.isdigit()
- and (len(token) > 2 or any("\u4e00" <= char <= "\u9fff" for char in token))
- ]
- return _dedupe(filtered, max_items=32)
-
-
-def profile_cluster_similarity(source_metadata: Mapping[str, Any], target_metadata: Mapping[str, Any]) -> float:
- source = dict(source_metadata or {})
- target = dict(target_metadata or {})
- source_tokens = set(
- profile_cluster_tokens(
- source.get("profile_domain_label", ""),
- source.get("profile_domain", ""),
- source.get("profile_route_terms", []) or [],
- source.get("profile_support_values", []) or [],
- source.get("profile_value", ""),
- )
- )
- target_tokens = set(
- profile_cluster_tokens(
- target.get("profile_domain_label", ""),
- target.get("profile_domain", ""),
- target.get("profile_route_terms", []) or [],
- target.get("profile_support_values", []) or [],
- target.get("profile_value", ""),
- )
- )
- if not source_tokens or not target_tokens:
- return 0.0
- overlap = len(source_tokens & target_tokens) / max(1, len(source_tokens | target_tokens))
- containment = len(source_tokens & target_tokens) / max(1, min(len(source_tokens), len(target_tokens)))
- type_bonus = 0.06 if _normalize(source.get("profile_type", "")) == _normalize(target.get("profile_type", "")) else 0.0
- return round(min(1.0, (0.62 * overlap) + (0.38 * containment) + type_bonus), 6)
-
-
-def profile_cluster_slot_key(metadata: Mapping[str, Any]) -> str:
- data = dict(metadata or {})
- tokens = profile_cluster_tokens(
- data.get("profile_domain_label", ""),
- data.get("profile_domain", ""),
- data.get("profile_route_terms", []) or [],
- data.get("profile_support_values", []) or [],
- data.get("profile_value", ""),
- )
- if tokens:
- return f"tmcra.profile.cluster.{_slug('_'.join(tokens[:5]))}"
- support_profiles = data.get("profile_support_profile_ids", []) or []
- support_seed = _clean_text(support_profiles[0] if support_profiles else "")
- seed = support_seed or _clean_text(data.get("profile_support_key", "general"))
- return f"tmcra.profile.cluster.{_slug(seed)}"
-
-
-def profile_cluster_value(*, support_values: Sequence[Any]) -> str:
- values = _dedupe(support_values, max_items=8)
- if not values:
- return "User profile cluster."
- return "User profile cluster: " + "; ".join(values)
-
-
-def build_profile_cluster_metadata(
- *,
- support_profile_id: Any,
- support_metadata: Mapping[str, Any],
- existing_metadata: Mapping[str, Any] | None = None,
-) -> dict[str, Any]:
- support = dict(support_metadata or {})
- existing = dict(existing_metadata or {})
- support_profile_ids = _bounded_union(
- existing.get("profile_support_profile_ids", []) or [],
- [support_profile_id],
- max_items=32,
- )
- support_ids = _bounded_union(
- existing.get("profile_support_ids", []) or [],
- support.get("profile_support_ids", []) or [],
- max_items=96,
- )
- support_turns = _bounded_int_union(
- existing.get("profile_support_turns", []) or [],
- support.get("profile_support_turns", []) or [],
- max_items=96,
- )
- raw_support_values = support.get("profile_support_values", []) or []
- support_values = _bounded_union(
- existing.get("profile_support_values", []) or [],
- raw_support_values,
- [] if raw_support_values else [support.get("profile_value", "")],
- max_items=16,
- )
- route_terms = _bounded_union(
- existing.get("profile_route_terms", []) or [],
- existing.get("profile_cluster_route_terms", []) or [],
- support.get("profile_route_terms", []) or [],
- profile_cluster_tokens(
- support.get("profile_domain_label", ""),
- support.get("profile_domain", ""),
- support.get("profile_value", ""),
- support.get("profile_support_values", []) or [],
- ),
- max_items=32,
- )
- profile_types = _bounded_union(
- existing.get("profile_cluster_types", []) or [],
- [support.get("profile_type", "")],
- max_items=8,
- )
- domains = _bounded_union(
- existing.get("profile_cluster_domains", []) or [],
- [support.get("profile_domain", ""), support.get("profile_domain_label", "")],
- max_items=16,
- )
- primary_type = _clean_text(profile_types[0] if profile_types else support.get("profile_type", "")) or "usage_context"
- primary_domain = _clean_text(domains[0] if domains else support.get("profile_domain", "")) or "general"
- output_kind = _profile_output_kind(profile_types)
- memory_type = _profile_memory_type(profile_types)
- summary = profile_summary(
- profile_types=profile_types,
- domain_label=primary_domain,
- support_values=support_values,
- stage="cluster",
- )
- value = profile_cluster_value(support_values=support_values)
- return {
- **existing,
- "profile_layer": True,
- "profile_candidate_status": "consolidated",
- "profile_consolidation_stage": "cluster",
- "profile_consolidator_version": PROFILE_CONSOLIDATOR_VERSION,
- "profile_cluster_node": True,
- "profile_type": primary_type,
- "profile_domain": _slug(primary_domain),
- "profile_domain_label": primary_domain.replace("_", " "),
- "profile_subject_signature": _clean_text(existing.get("profile_subject_signature", "")) or _slug(primary_domain),
- "profile_support_key": _clean_text(existing.get("profile_support_key", "")) or f"cluster:{_slug(primary_domain)}",
- "profile_support_profile_ids": support_profile_ids,
- "profile_support_ids": support_ids,
- "profile_support_turns": support_turns,
- "profile_support_values": support_values,
- "profile_support_count": len(support_ids),
- "profile_cluster_profile_count": len(support_profile_ids),
- "profile_cluster_types": profile_types,
- "profile_cluster_domains": domains,
- "profile_cluster_route_terms": route_terms,
- "profile_route_terms": route_terms,
- "profile_value": value,
- "profile_summary": summary,
- "profile_output_kind": output_kind,
- "profile_update_policy": _profile_update_policy(profile_types),
- "profile_conflict_policy": "latest_active_support_only",
- "profile_evidence_count": len(support_ids),
- "memory_type": memory_type,
- "durable_memory_type": memory_type,
- "memory_chain_depth_layer": "profile",
- "depth_layer": "profile",
- }
-
-
-def infer_profile_query_intent(query: Any) -> dict[str, Any]:
- query_text = _clean_text(query)
- lowered = _normalize(query_text)
- enabled = any(marker in lowered for marker in _PROFILE_QUERY_MARKERS)
- types: list[str] = []
- if any(marker in lowered for marker in ("prefer", "preference", "like", "dislike", "偏好", "喜欢", "不喜欢")):
- types.append("preference")
- if any(marker in lowered for marker in ("constraint", "must", "cannot", "policy", "约束", "限制", "必须", "不能")):
- types.append("constraint")
- if any(marker in lowered for marker in ("goal", "target", "objective", "目标")):
- types.append("goal")
- if any(marker in lowered for marker in ("avoid", "dislike", "避免", "不喜欢")):
- types.append("avoid")
- if any(marker in lowered for marker in ("setup", "profile", "current", "occupation", "role", "job", "career", "background", "identity", "experience", "配置", "画像", "当前")):
- types.append("setup")
- if enabled and any(
- marker in lowered
- for marker in (
- "recommend",
- "suggest",
- "advice",
- "tips",
- "trouble with",
- "struggling with",
- "what do you think",
- "should i",
- "what should",
- "serve",
- "dinner",
- "battery life",
- "getting around",
- "resources",
- "learn more",
- "推荐",
- "建议",
- )
- ):
- types.append("usage_context")
- if not types and enabled:
- types.append("usage_context")
- return {
- "enabled": enabled,
- "types": _dedupe(types),
- "tokens": _tokens(query_text),
- }
-
-
-def profile_query_score_delta(
- *,
- query: Any,
- query_tokens: set[str],
- category: Any,
- source_kind: Any,
- semantic_slot: Any,
- value: Any,
- anchors: Sequence[Any],
- metadata: Mapping[str, Any] | None = None,
-) -> tuple[float, str]:
- data = dict(metadata or {})
- intent = infer_profile_query_intent(query)
- if not intent.get("enabled"):
- return 0.0, ""
- if not is_profile_layer_record(category=category, source_kind=source_kind, semantic_slot=semantic_slot, metadata=data):
- return 0.0, ""
- profile_type = _normalize(data.get("profile_type", "")) or infer_profile_type(
- category=category,
- semantic_slot=semantic_slot,
- value=value,
- metadata=data,
- )
- domain = _normalize(data.get("profile_domain_label", "") or data.get("profile_domain", ""))
- route_terms = " ".join(str(item) for item in data.get("profile_route_terms", []) or [])
- record_tokens = set(_tokens(f"{profile_type} {domain} {semantic_slot} {value} {' '.join(str(item) for item in anchors)} {route_terms}"))
- overlap = len(set(query_tokens) & record_tokens) / max(1, len(set(query_tokens) | record_tokens)) if query_tokens or record_tokens else 0.0
- type_match = profile_type in set(intent.get("types", []) or [])
- delta = 0.20 + (0.16 if type_match else 0.0) + (0.18 * overlap)
- if data.get("profile_candidate_status") == "consolidated":
- delta += 0.06
- if data.get("profile_cluster_node") or data.get("profile_consolidation_stage") == "cluster":
- delta += 0.08
- return round(min(delta, 0.54), 6), "profile_route"
-
-
-def profile_edge_score(source_metadata: Mapping[str, Any], target_metadata: Mapping[str, Any], *, source_value: Any = "", target_value: Any = "") -> tuple[float, str]:
- source = dict(source_metadata or {})
- target = dict(target_metadata or {})
- if not source.get("profile_layer") or not target.get("profile_layer"):
- return 0.0, ""
- source_key = _normalize(source.get("profile_support_key", ""))
- target_key = _normalize(target.get("profile_support_key", ""))
- source_domain = _normalize(source.get("profile_domain", ""))
- target_domain = _normalize(target.get("profile_domain", ""))
- source_type = _normalize(source.get("profile_type", ""))
- target_type = _normalize(target.get("profile_type", ""))
- source_tokens = set(_tokens(f"{source_value} {source.get('profile_domain_label', '')} {' '.join(source.get('profile_route_terms', []) or [])}"))
- target_tokens = set(_tokens(f"{target_value} {target.get('profile_domain_label', '')} {' '.join(target.get('profile_route_terms', []) or [])}"))
- overlap = len(source_tokens & target_tokens) / max(1, len(source_tokens | target_tokens)) if source_tokens or target_tokens else 0.0
- if source_key and source_key == target_key:
- return round(0.72 + (0.14 * overlap), 6), "profile_support"
- if source_domain and source_domain == target_domain:
- return round(0.58 + (0.12 if source_type == target_type else 0.04) + (0.12 * overlap), 6), "profile_tunnel"
- if source_type and source_type == target_type and overlap >= 0.22:
- return round(0.44 + (0.18 * overlap), 6), "profile_soft_tunnel"
- return 0.0, ""
+from __future__ import annotations
+
+import re
+from typing import Any, Iterable, Mapping, Sequence
+
+
+PROFILE_CATEGORIES = {"profile", "preference", "goal", "constraint", "stage_state", "status"}
+PROFILE_TYPES = {"setup", "preference", "constraint", "goal", "avoid", "usage_context"}
+PROFILE_SEMANTIC_SLOTS = {"identity", "research_topic", "education", "occupation"}
+PROFILE_AGGREGATE_SOURCE_KIND = "public_dialog_profile"
+PROFILE_AGGREGATE_CATEGORY = "profile"
+PROFILE_CLUSTER_SOURCE_KIND = "public_dialog_profile_cluster"
+PROFILE_CLUSTER_CATEGORY = "profile"
+PROFILE_CONSOLIDATOR_VERSION = "profile_consolidator_v1_structured_summary"
+
+_PROFILE_QUERY_MARKERS = (
+ "preference",
+ "prefer",
+ "like",
+ "dislike",
+ "recommend",
+ "suggest",
+ "advice",
+ "advise",
+ "any advice",
+ "any suggestions",
+ "any tips",
+ "tips",
+ "trouble with",
+ "struggling with",
+ "what do you think",
+ "learn more",
+ "resources",
+ "suited",
+ "fit me",
+ "for me",
+ "based on me",
+ "based on my",
+ "my setup",
+ "my profile",
+ "my goal",
+ "my constraint",
+ "my occupation",
+ "my previous occupation",
+ "my role",
+ "my previous role",
+ "my job",
+ "my previous job",
+ "what was my",
+ "where did i work",
+ "worked as",
+ "occupation",
+ "previous occupation",
+ "role",
+ "previous role",
+ "job",
+ "career",
+ "background",
+ "identity",
+ "experience",
+ "avoid",
+ "what should i",
+ "should i",
+ "what should",
+ "serve",
+ "dinner",
+ "homegrown",
+ "ingredients",
+ "battery life",
+ "getting around",
+ "偏好",
+ "喜欢",
+ "不喜欢",
+ "推荐",
+ "建议",
+ "适合我",
+ "根据我",
+ "我的情况",
+ "我的配置",
+ "我的目标",
+ "我的约束",
+ "画像",
+ "避免",
+)
+
+
+def _clean_text(value: Any) -> str:
+ return re.sub(r"\s+", " ", str(value or "").strip())
+
+
+def _normalize(value: Any) -> str:
+ return _clean_text(value).lower()
+
+
+def _tokens(value: Any) -> list[str]:
+ text = _normalize(value)
+ english = re.findall(r"[a-z0-9_]+", text)
+ cjk = [char for char in text if "\u4e00" <= char <= "\u9fff"]
+ return _dedupe([*english, *cjk])
+
+
+def _slug(value: Any, *, fallback: str = "general") -> str:
+ text = _normalize(value)
+ parts = re.findall(r"[a-z0-9]+|[\u4e00-\u9fff]", text)
+ slug = "_".join(parts[:10]).strip("_")
+ return slug or fallback
+
+
+def _dedupe(items: Iterable[Any], *, max_items: int | None = None) -> list[str]:
+ values: list[str] = []
+ seen: set[str] = set()
+ for item in items:
+ text = _clean_text(item)
+ if not text:
+ continue
+ key = _normalize(text)
+ if key in seen:
+ continue
+ seen.add(key)
+ values.append(text)
+ if max_items is not None and len(values) >= max_items:
+ break
+ return values
+
+
+def _first_nonempty(*values: Any) -> str:
+ for value in values:
+ text = _clean_text(value)
+ if text:
+ return text
+ return ""
+
+
+def _bounded_union(*groups: Iterable[Any], max_items: int) -> list[str]:
+ return _dedupe([item for group in groups for item in group], max_items=max_items)
+
+
+def _bounded_int_union(*groups: Iterable[Any], max_items: int) -> list[int]:
+ values: list[int] = []
+ seen: set[int] = set()
+ for group in groups:
+ for item in group:
+ try:
+ value = int(item)
+ except Exception:
+ continue
+ if value in seen:
+ continue
+ seen.add(value)
+ values.append(value)
+ if len(values) >= max_items:
+ return sorted(values)
+ return sorted(values)
+
+
+def is_profile_layer_record(*, category: Any, source_kind: Any = "", semantic_slot: Any = "", metadata: Mapping[str, Any] | None = None) -> bool:
+ data = dict(metadata or {})
+ if bool(data.get("profile_layer")):
+ return True
+ category_text = _normalize(category)
+ slot_text = _normalize(semantic_slot or data.get("semantic_slot", ""))
+ source_text = _normalize(source_kind)
+ if category_text in PROFILE_CATEGORIES:
+ return True
+ if slot_text in PROFILE_SEMANTIC_SLOTS or slot_text.startswith("profile_"):
+ return True
+ return source_text in {
+ "public_dialog_profile",
+ "public_dialog_profile_cluster",
+ "public_dialog_preference",
+ "public_dialog_goal",
+ "public_dialog_constraint",
+ }
+
+
+def infer_profile_type(*, category: Any, semantic_slot: Any = "", relation: Any = "", value: Any = "", metadata: Mapping[str, Any] | None = None) -> str:
+ data = dict(metadata or {})
+ explicit = _normalize(data.get("profile_type", ""))
+ if explicit in PROFILE_TYPES:
+ return explicit
+ category_text = _normalize(category)
+ combined = _normalize(f"{semantic_slot} {relation} {value}")
+ if category_text == "preference" or any(marker in combined for marker in ("prefer", "like", "default", "偏好", "喜欢", "默认")):
+ return "preference"
+ if category_text == "constraint" or any(marker in combined for marker in ("must", "cannot", "forbid", "constraint", "必须", "不能", "约束")):
+ return "constraint"
+ if category_text == "goal" or any(marker in combined for marker in ("goal", "target", "objective", "目标")):
+ return "goal"
+ if any(marker in combined for marker in ("avoid", "dislike", "do not", "don't", "避免", "不喜欢", "不要")):
+ return "avoid"
+ if any(marker in combined for marker in ("setup", "environment", "workflow", "current", "配置", "环境", "流程", "当前")):
+ return "setup"
+ return "usage_context"
+
+
+def infer_profile_domain(
+ *,
+ category: Any,
+ semantic_slot: Any = "",
+ slot_key: Any = "",
+ anchors: Sequence[Any] = (),
+ value: Any = "",
+ metadata: Mapping[str, Any] | None = None,
+) -> tuple[str, str]:
+ data = dict(metadata or {})
+ explicit = _first_nonempty(data.get("profile_domain", ""), data.get("profile_domain_label", ""), data.get("domain", ""))
+ if explicit:
+ return _slug(explicit), explicit
+ subject = _first_nonempty(data.get("subject", ""), data.get("extracted_subject", ""))
+ if subject:
+ return _slug(subject), subject
+ slot = _normalize(semantic_slot)
+ if slot and slot not in {"profile", "preference", "goal", "constraint", "event", "fact", "status"}:
+ return _slug(slot), slot.replace("_", " ")
+ for anchor in anchors:
+ anchor_text = _clean_text(anchor)
+ if not anchor_text:
+ continue
+ if re.search(r"^\d{4}|\d{1,2}\s+[A-Za-z]+|^[A-Z][a-z]+$", anchor_text):
+ continue
+ return _slug(anchor_text), anchor_text
+ slot_key_text = _clean_text(slot_key)
+ if slot_key_text:
+ tail = slot_key_text.split(".")[-1].replace("_", " ")
+ if tail:
+ return _slug(tail), tail
+ value_tokens = _tokens(value)
+ if value_tokens:
+ label = " ".join(value_tokens[:4])
+ return _slug(label), label
+ return _slug(category), _normalize(category) or "general"
+
+
+def profile_candidate_metadata(
+ *,
+ category: Any,
+ semantic_slot: Any = "",
+ relation: Any = "",
+ value: Any = "",
+ source_span: Any = "",
+ slot_key: Any = "",
+ anchors: Sequence[Any] = (),
+ subject: Any = "",
+ subject_signature: Any = "",
+ proposal: Mapping[str, Any] | None = None,
+) -> dict[str, Any]:
+ proposal_data = dict(proposal or {})
+ base_metadata = {
+ "profile_type": proposal_data.get("profile_type", ""),
+ "profile_domain": proposal_data.get("profile_domain", ""),
+ "profile_domain_label": proposal_data.get("profile_domain_label", ""),
+ "subject": subject,
+ "extracted_subject": proposal_data.get("extracted_subject", ""),
+ }
+ if not is_profile_layer_record(category=category, semantic_slot=semantic_slot, metadata=base_metadata):
+ return {}
+ profile_type = infer_profile_type(
+ category=category,
+ semantic_slot=semantic_slot,
+ relation=relation,
+ value=value or source_span,
+ metadata=proposal_data,
+ )
+ domain, domain_label = infer_profile_domain(
+ category=category,
+ semantic_slot=semantic_slot,
+ slot_key=slot_key,
+ anchors=anchors,
+ value=value or source_span,
+ metadata={**proposal_data, "subject": subject},
+ )
+ normalized_subject_signature = _slug(subject_signature or subject or domain, fallback=domain)
+ route_terms = _dedupe(
+ [
+ profile_type,
+ domain_label,
+ semantic_slot,
+ subject,
+ *list(anchors or []),
+ ],
+ max_items=12,
+ )
+ return {
+ "profile_layer": True,
+ "profile_candidate_status": _clean_text(proposal_data.get("profile_candidate_status", "")) or "writer_candidate",
+ "profile_consolidation_stage": _clean_text(proposal_data.get("profile_consolidation_stage", "")) or "pre_consolidation",
+ "profile_type": profile_type,
+ "profile_domain": domain,
+ "profile_domain_label": domain_label,
+ "profile_subject_signature": normalized_subject_signature,
+ "profile_support_key": f"{profile_type}:{domain}:{normalized_subject_signature}",
+ "profile_route_terms": route_terms,
+ }
+
+
+def profile_aggregate_slot_key(metadata: Mapping[str, Any]) -> str:
+ data = dict(metadata or {})
+ support_key = _clean_text(data.get("profile_support_key", ""))
+ if support_key:
+ return f"tmcra.profile.aggregate.{_slug(support_key)}"
+ profile_type = _normalize(data.get("profile_type", "")) or "usage_context"
+ domain = _normalize(data.get("profile_domain", "")) or _slug(data.get("profile_domain_label", "general"))
+ subject = _normalize(data.get("profile_subject_signature", "")) or domain
+ return f"tmcra.profile.aggregate.{_slug(f'{profile_type}:{domain}:{subject}')}"
+
+
+def profile_aggregate_value(
+ *,
+ profile_type: Any,
+ domain_label: Any,
+ support_values: Sequence[Any],
+) -> str:
+ typed = _clean_text(profile_type) or "usage_context"
+ domain = _clean_text(domain_label) or "general"
+ values = _dedupe(support_values, max_items=5)
+ if not values:
+ return f"User {typed} profile for {domain}."
+ return f"User {typed} profile for {domain}: " + "; ".join(values)
+
+
+def _profile_output_kind(profile_types: Sequence[Any]) -> str:
+ normalized = {_normalize(item) for item in profile_types if _clean_text(item)}
+ if "constraint" in normalized:
+ return "constraint_profile"
+ if "goal" in normalized:
+ return "goal_profile"
+ if normalized.intersection({"preference", "avoid"}):
+ return "preference_profile"
+ if "setup" in normalized:
+ return "setup_profile"
+ return "usage_context_profile"
+
+
+def _profile_update_policy(profile_types: Sequence[Any]) -> str:
+ normalized = {_normalize(item) for item in profile_types if _clean_text(item)}
+ if "constraint" in normalized:
+ return "preserve_until_explicitly_changed"
+ if normalized.intersection({"preference", "avoid", "goal", "setup"}):
+ return "update_on_newer_user_evidence"
+ return "background_context"
+
+
+def _profile_memory_type(profile_types: Sequence[Any]) -> str:
+ normalized = {_normalize(item) for item in profile_types if _clean_text(item)}
+ if "constraint" in normalized:
+ return "hard_constraint"
+ if normalized.intersection({"preference", "avoid"}):
+ return "durable_preference"
+ return "profile_context"
+
+
+def profile_summary(
+ *,
+ profile_types: Sequence[Any],
+ domain_label: Any,
+ support_values: Sequence[Any],
+ stage: str,
+) -> str:
+ kind = _profile_output_kind(profile_types)
+ domain = _clean_text(domain_label) or "general"
+ values = _dedupe(support_values, max_items=6 if stage == "cluster" else 4)
+ prefix = {
+ "constraint_profile": "User constraint profile",
+ "goal_profile": "User goal profile",
+ "preference_profile": "User preference profile",
+ "setup_profile": "User setup profile",
+ "usage_context_profile": "User usage-context profile",
+ }.get(kind, "User profile")
+ if not values:
+ return f"{prefix} for {domain}."
+ return f"{prefix} for {domain}: " + "; ".join(values)
+
+
+def build_profile_aggregate_metadata(
+ *,
+ support_record_id: Any,
+ support_turn_index: int,
+ support_value: Any,
+ support_anchors: Sequence[Any],
+ support_metadata: Mapping[str, Any],
+ existing_metadata: Mapping[str, Any] | None = None,
+) -> dict[str, Any]:
+ support = dict(support_metadata or {})
+ existing = dict(existing_metadata or {})
+ profile_type = _normalize(support.get("profile_type", "")) or "usage_context"
+ domain = _normalize(support.get("profile_domain", "")) or _slug(support.get("profile_domain_label", "general"))
+ domain_label = _clean_text(support.get("profile_domain_label", "")) or domain.replace("_", " ")
+ subject_signature = _clean_text(support.get("profile_subject_signature", "")) or domain
+ support_ids = _bounded_union(existing.get("profile_support_ids", []) or [], [support_record_id], max_items=64)
+ support_turns = _bounded_int_union(existing.get("profile_support_turns", []) or [], [support_turn_index], max_items=64)
+ support_values = _bounded_union(existing.get("profile_support_values", []) or [], [support_value], max_items=12)
+ support_route_terms = _bounded_union(
+ existing.get("profile_route_terms", []) or [],
+ support.get("profile_route_terms", []) or [],
+ support_anchors,
+ max_items=24,
+ )
+ profile_types = [profile_type]
+ output_kind = _profile_output_kind(profile_types)
+ memory_type = _profile_memory_type(profile_types)
+ summary = profile_summary(
+ profile_types=profile_types,
+ domain_label=domain_label,
+ support_values=support_values,
+ stage="aggregate",
+ )
+ value = profile_aggregate_value(
+ profile_type=profile_type,
+ domain_label=domain_label,
+ support_values=support_values,
+ )
+ return {
+ **existing,
+ "profile_layer": True,
+ "profile_candidate_status": "consolidated",
+ "profile_consolidation_stage": "aggregate",
+ "profile_consolidator_version": PROFILE_CONSOLIDATOR_VERSION,
+ "profile_aggregate_node": True,
+ "profile_type": profile_type,
+ "profile_domain": domain,
+ "profile_domain_label": domain_label,
+ "profile_subject_signature": subject_signature,
+ "profile_support_key": f"{profile_type}:{domain}:{subject_signature}",
+ "profile_support_ids": support_ids,
+ "profile_support_turns": support_turns,
+ "profile_support_values": support_values,
+ "profile_support_count": len(support_ids),
+ "profile_route_terms": support_route_terms,
+ "profile_value": value,
+ "profile_summary": summary,
+ "profile_output_kind": output_kind,
+ "profile_update_policy": _profile_update_policy(profile_types),
+ "profile_conflict_policy": "latest_active_support_only",
+ "profile_evidence_count": len(support_ids),
+ "memory_type": memory_type,
+ "durable_memory_type": memory_type,
+ "memory_chain_depth_layer": "profile",
+ "depth_layer": "profile",
+ }
+
+
+_PROFILE_CLUSTER_STOPWORDS = {
+ "user",
+ "profile",
+ "preference",
+ "preferences",
+ "constraint",
+ "constraints",
+ "goal",
+ "goals",
+ "avoid",
+ "usage",
+ "context",
+ "general",
+ "default",
+ "should",
+ "would",
+ "could",
+ "want",
+ "wants",
+ "need",
+ "needs",
+ "when",
+ "then",
+ "than",
+ "with",
+ "that",
+ "this",
+ "from",
+ "into",
+ "instead",
+}
+
+
+def profile_cluster_tokens(*values: Any) -> list[str]:
+ tokens: list[str] = []
+ for value in values:
+ if isinstance(value, (list, tuple, set)):
+ for item in value:
+ tokens.extend(_tokens(item))
+ else:
+ tokens.extend(_tokens(value))
+ filtered = [
+ token
+ for token in tokens
+ if token
+ and token not in _PROFILE_CLUSTER_STOPWORDS
+ and not token.isdigit()
+ and (len(token) > 2 or any("\u4e00" <= char <= "\u9fff" for char in token))
+ ]
+ return _dedupe(filtered, max_items=32)
+
+
+def profile_cluster_similarity(source_metadata: Mapping[str, Any], target_metadata: Mapping[str, Any]) -> float:
+ source = dict(source_metadata or {})
+ target = dict(target_metadata or {})
+ source_tokens = set(
+ profile_cluster_tokens(
+ source.get("profile_domain_label", ""),
+ source.get("profile_domain", ""),
+ source.get("profile_route_terms", []) or [],
+ source.get("profile_support_values", []) or [],
+ source.get("profile_value", ""),
+ )
+ )
+ target_tokens = set(
+ profile_cluster_tokens(
+ target.get("profile_domain_label", ""),
+ target.get("profile_domain", ""),
+ target.get("profile_route_terms", []) or [],
+ target.get("profile_support_values", []) or [],
+ target.get("profile_value", ""),
+ )
+ )
+ if not source_tokens or not target_tokens:
+ return 0.0
+ overlap = len(source_tokens & target_tokens) / max(1, len(source_tokens | target_tokens))
+ containment = len(source_tokens & target_tokens) / max(1, min(len(source_tokens), len(target_tokens)))
+ type_bonus = 0.06 if _normalize(source.get("profile_type", "")) == _normalize(target.get("profile_type", "")) else 0.0
+ return round(min(1.0, (0.62 * overlap) + (0.38 * containment) + type_bonus), 6)
+
+
+def profile_cluster_slot_key(metadata: Mapping[str, Any]) -> str:
+ data = dict(metadata or {})
+ tokens = profile_cluster_tokens(
+ data.get("profile_domain_label", ""),
+ data.get("profile_domain", ""),
+ data.get("profile_route_terms", []) or [],
+ data.get("profile_support_values", []) or [],
+ data.get("profile_value", ""),
+ )
+ if tokens:
+ return f"tmcra.profile.cluster.{_slug('_'.join(tokens[:5]))}"
+ support_profiles = data.get("profile_support_profile_ids", []) or []
+ support_seed = _clean_text(support_profiles[0] if support_profiles else "")
+ seed = support_seed or _clean_text(data.get("profile_support_key", "general"))
+ return f"tmcra.profile.cluster.{_slug(seed)}"
+
+
+def profile_cluster_value(*, support_values: Sequence[Any]) -> str:
+ values = _dedupe(support_values, max_items=8)
+ if not values:
+ return "User profile cluster."
+ return "User profile cluster: " + "; ".join(values)
+
+
+def build_profile_cluster_metadata(
+ *,
+ support_profile_id: Any,
+ support_metadata: Mapping[str, Any],
+ existing_metadata: Mapping[str, Any] | None = None,
+) -> dict[str, Any]:
+ support = dict(support_metadata or {})
+ existing = dict(existing_metadata or {})
+ support_profile_ids = _bounded_union(
+ existing.get("profile_support_profile_ids", []) or [],
+ [support_profile_id],
+ max_items=32,
+ )
+ support_ids = _bounded_union(
+ existing.get("profile_support_ids", []) or [],
+ support.get("profile_support_ids", []) or [],
+ max_items=96,
+ )
+ support_turns = _bounded_int_union(
+ existing.get("profile_support_turns", []) or [],
+ support.get("profile_support_turns", []) or [],
+ max_items=96,
+ )
+ raw_support_values = support.get("profile_support_values", []) or []
+ support_values = _bounded_union(
+ existing.get("profile_support_values", []) or [],
+ raw_support_values,
+ [] if raw_support_values else [support.get("profile_value", "")],
+ max_items=16,
+ )
+ route_terms = _bounded_union(
+ existing.get("profile_route_terms", []) or [],
+ existing.get("profile_cluster_route_terms", []) or [],
+ support.get("profile_route_terms", []) or [],
+ profile_cluster_tokens(
+ support.get("profile_domain_label", ""),
+ support.get("profile_domain", ""),
+ support.get("profile_value", ""),
+ support.get("profile_support_values", []) or [],
+ ),
+ max_items=32,
+ )
+ profile_types = _bounded_union(
+ existing.get("profile_cluster_types", []) or [],
+ [support.get("profile_type", "")],
+ max_items=8,
+ )
+ domains = _bounded_union(
+ existing.get("profile_cluster_domains", []) or [],
+ [support.get("profile_domain", ""), support.get("profile_domain_label", "")],
+ max_items=16,
+ )
+ primary_type = _clean_text(profile_types[0] if profile_types else support.get("profile_type", "")) or "usage_context"
+ primary_domain = _clean_text(domains[0] if domains else support.get("profile_domain", "")) or "general"
+ output_kind = _profile_output_kind(profile_types)
+ memory_type = _profile_memory_type(profile_types)
+ summary = profile_summary(
+ profile_types=profile_types,
+ domain_label=primary_domain,
+ support_values=support_values,
+ stage="cluster",
+ )
+ value = profile_cluster_value(support_values=support_values)
+ return {
+ **existing,
+ "profile_layer": True,
+ "profile_candidate_status": "consolidated",
+ "profile_consolidation_stage": "cluster",
+ "profile_consolidator_version": PROFILE_CONSOLIDATOR_VERSION,
+ "profile_cluster_node": True,
+ "profile_type": primary_type,
+ "profile_domain": _slug(primary_domain),
+ "profile_domain_label": primary_domain.replace("_", " "),
+ "profile_subject_signature": _clean_text(existing.get("profile_subject_signature", "")) or _slug(primary_domain),
+ "profile_support_key": _clean_text(existing.get("profile_support_key", "")) or f"cluster:{_slug(primary_domain)}",
+ "profile_support_profile_ids": support_profile_ids,
+ "profile_support_ids": support_ids,
+ "profile_support_turns": support_turns,
+ "profile_support_values": support_values,
+ "profile_support_count": len(support_ids),
+ "profile_cluster_profile_count": len(support_profile_ids),
+ "profile_cluster_types": profile_types,
+ "profile_cluster_domains": domains,
+ "profile_cluster_route_terms": route_terms,
+ "profile_route_terms": route_terms,
+ "profile_value": value,
+ "profile_summary": summary,
+ "profile_output_kind": output_kind,
+ "profile_update_policy": _profile_update_policy(profile_types),
+ "profile_conflict_policy": "latest_active_support_only",
+ "profile_evidence_count": len(support_ids),
+ "memory_type": memory_type,
+ "durable_memory_type": memory_type,
+ "memory_chain_depth_layer": "profile",
+ "depth_layer": "profile",
+ }
+
+
+def infer_profile_query_intent(query: Any) -> dict[str, Any]:
+ query_text = _clean_text(query)
+ lowered = _normalize(query_text)
+ enabled = any(marker in lowered for marker in _PROFILE_QUERY_MARKERS)
+ types: list[str] = []
+ if any(marker in lowered for marker in ("prefer", "preference", "like", "dislike", "偏好", "喜欢", "不喜欢")):
+ types.append("preference")
+ if any(marker in lowered for marker in ("constraint", "must", "cannot", "policy", "约束", "限制", "必须", "不能")):
+ types.append("constraint")
+ if any(marker in lowered for marker in ("goal", "target", "objective", "目标")):
+ types.append("goal")
+ if any(marker in lowered for marker in ("avoid", "dislike", "避免", "不喜欢")):
+ types.append("avoid")
+ if any(marker in lowered for marker in ("setup", "profile", "current", "occupation", "role", "job", "career", "background", "identity", "experience", "配置", "画像", "当前")):
+ types.append("setup")
+ if enabled and any(
+ marker in lowered
+ for marker in (
+ "recommend",
+ "suggest",
+ "advice",
+ "tips",
+ "trouble with",
+ "struggling with",
+ "what do you think",
+ "should i",
+ "what should",
+ "serve",
+ "dinner",
+ "battery life",
+ "getting around",
+ "resources",
+ "learn more",
+ "推荐",
+ "建议",
+ )
+ ):
+ types.append("usage_context")
+ if not types and enabled:
+ types.append("usage_context")
+ return {
+ "enabled": enabled,
+ "types": _dedupe(types),
+ "tokens": _tokens(query_text),
+ }
+
+
+def profile_query_score_delta(
+ *,
+ query: Any,
+ query_tokens: set[str],
+ category: Any,
+ source_kind: Any,
+ semantic_slot: Any,
+ value: Any,
+ anchors: Sequence[Any],
+ metadata: Mapping[str, Any] | None = None,
+) -> tuple[float, str]:
+ data = dict(metadata or {})
+ intent = infer_profile_query_intent(query)
+ if not intent.get("enabled"):
+ return 0.0, ""
+ if not is_profile_layer_record(category=category, source_kind=source_kind, semantic_slot=semantic_slot, metadata=data):
+ return 0.0, ""
+ profile_type = _normalize(data.get("profile_type", "")) or infer_profile_type(
+ category=category,
+ semantic_slot=semantic_slot,
+ value=value,
+ metadata=data,
+ )
+ domain = _normalize(data.get("profile_domain_label", "") or data.get("profile_domain", ""))
+ route_terms = " ".join(str(item) for item in data.get("profile_route_terms", []) or [])
+ record_tokens = set(_tokens(f"{profile_type} {domain} {semantic_slot} {value} {' '.join(str(item) for item in anchors)} {route_terms}"))
+ overlap = len(set(query_tokens) & record_tokens) / max(1, len(set(query_tokens) | record_tokens)) if query_tokens or record_tokens else 0.0
+ type_match = profile_type in set(intent.get("types", []) or [])
+ delta = 0.20 + (0.16 if type_match else 0.0) + (0.18 * overlap)
+ if data.get("profile_candidate_status") == "consolidated":
+ delta += 0.06
+ if data.get("profile_cluster_node") or data.get("profile_consolidation_stage") == "cluster":
+ delta += 0.08
+ return round(min(delta, 0.54), 6), "profile_route"
+
+
+def profile_edge_score(source_metadata: Mapping[str, Any], target_metadata: Mapping[str, Any], *, source_value: Any = "", target_value: Any = "") -> tuple[float, str]:
+ source = dict(source_metadata or {})
+ target = dict(target_metadata or {})
+ if not source.get("profile_layer") or not target.get("profile_layer"):
+ return 0.0, ""
+ source_key = _normalize(source.get("profile_support_key", ""))
+ target_key = _normalize(target.get("profile_support_key", ""))
+ source_domain = _normalize(source.get("profile_domain", ""))
+ target_domain = _normalize(target.get("profile_domain", ""))
+ source_type = _normalize(source.get("profile_type", ""))
+ target_type = _normalize(target.get("profile_type", ""))
+ source_tokens = set(_tokens(f"{source_value} {source.get('profile_domain_label', '')} {' '.join(source.get('profile_route_terms', []) or [])}"))
+ target_tokens = set(_tokens(f"{target_value} {target.get('profile_domain_label', '')} {' '.join(target.get('profile_route_terms', []) or [])}"))
+ overlap = len(source_tokens & target_tokens) / max(1, len(source_tokens | target_tokens)) if source_tokens or target_tokens else 0.0
+ if source_key and source_key == target_key:
+ return round(0.72 + (0.14 * overlap), 6), "profile_support"
+ if source_domain and source_domain == target_domain:
+ return round(0.58 + (0.12 if source_type == target_type else 0.04) + (0.12 * overlap), 6), "profile_tunnel"
+ if source_type and source_type == target_type and overlap >= 0.22:
+ return round(0.44 + (0.18 * overlap), 6), "profile_soft_tunnel"
+ return 0.0, ""
diff --git a/code/run_lme_s10_native_tmcra.py b/code/run_lme_s10_native_tmcra.py
index d28bdfd69620abc9c2c2c2f042118f0a1ea859b4..5769a4073228b8174cd442667764a2037d663554 100644
--- a/code/run_lme_s10_native_tmcra.py
+++ b/code/run_lme_s10_native_tmcra.py
@@ -1,3792 +1,3792 @@
-#!/usr/bin/env python3
-from __future__ import annotations
-
-import argparse
-import json
-import os
-import re
-import sys
-import time
-import urllib.error
-import urllib.request
-from datetime import datetime, timezone
-from pathlib import Path
-from types import SimpleNamespace
-from typing import Any, Iterable, Mapping
-
-
-DEFAULT_SERVICE_ROOT = Path(os.getenv("TMCRA_SERVICE_ROOT", "./tmcra_api_service"))
-DEFAULT_REPO = Path(os.getenv("TMCRA_REPO_ROOT", str(DEFAULT_SERVICE_ROOT / "private" / "tmcra-integrated")))
-DEFAULT_DATA = Path(os.getenv("LONGMEMEVAL_S_DATA", "./data/longmemeval_s_cleaned.json"))
-DEFAULT_OUT_ROOT = Path(os.getenv("TMCRA_LME_RUN_ROOT", "./runs"))
-
-
-def log(event: str, **payload: Any) -> None:
- stamp = datetime.now().isoformat(timespec="seconds")
- details = " ".join(f"{key}={json.dumps(value, ensure_ascii=False)}" for key, value in sorted(payload.items()))
- print(f"[lme_native] {stamp} {event}" + (f" {details}" if details else ""), flush=True)
-
-
-def read_env_file(path: Path) -> dict[str, str]:
- env: dict[str, str] = {}
- if not path.exists():
- return env
- for raw in path.read_text(encoding="utf-8", errors="replace").splitlines():
- line = raw.strip()
- if not line or line.startswith("#") or "=" not in line:
- continue
- if line.startswith("export "):
- line = line[len("export ") :].strip()
- key, value = line.split("=", 1)
- env[key.strip()] = value.strip().strip('"').strip("'")
- return env
-
-
-def apply_env_defaults(service_root: Path) -> None:
- for env_path in [service_root / "env" / "tmcra-api.env", service_root / "env" / "tmcra-gemma.env"]:
- for key, value in read_env_file(env_path).items():
- if value == "":
- continue
- os.environ.setdefault(key, value)
-
-
-def iter_json_array(path: Path, *, limit: int = 0) -> Iterable[dict[str, Any]]:
- import codecs
-
- decoder = codecs.getincrementaldecoder("utf-8")(errors="replace")
- yielded = 0
- array_started = False
- object_started = False
- object_depth = 0
- in_string = False
- escape_next = False
- buffer: list[str] = []
- with path.open("rb") as handle:
- while True:
- chunk = handle.read(1 << 20)
- if not chunk:
- text = decoder.decode(b"", final=True)
- else:
- text = decoder.decode(chunk)
- if not text and not chunk:
- break
- for char in text:
- if not array_started:
- if char.isspace():
- continue
- if char != "[":
- raise RuntimeError(f"expected top-level JSON array in {path}")
- array_started = True
- continue
- if not object_started:
- if char.isspace() or char == ",":
- continue
- if char == "]":
- return
- if char != "{":
- raise RuntimeError(f"expected object item in {path}, got {char!r}")
- object_started = True
- object_depth = 1
- in_string = False
- escape_next = False
- buffer = ["{"]
- continue
- buffer.append(char)
- if in_string:
- if escape_next:
- escape_next = False
- elif char == "\\":
- escape_next = True
- elif char == '"':
- in_string = False
- continue
- if char == '"':
- in_string = True
- continue
- if char == "{":
- object_depth += 1
- elif char == "}":
- object_depth -= 1
- if object_depth == 0:
- row = json.loads("".join(buffer))
- if isinstance(row, dict):
- yield row
- yielded += 1
- if limit > 0 and yielded >= limit:
- return
- object_started = False
- buffer = []
- if not chunk:
- break
- if object_started:
- raise RuntimeError(f"unterminated JSON object in {path}")
-
-
-def http_json(
- method: str,
- url: str,
- payload: dict[str, Any] | None = None,
- *,
- timeout: int = 300,
- api_key: str = "",
-) -> dict[str, Any]:
- data = json.dumps(payload, ensure_ascii=False).encode("utf-8") if payload is not None else None
- headers = {"Content-Type": "application/json; charset=utf-8"}
- if api_key:
- headers["Authorization"] = f"Bearer {api_key}"
- max_attempts = max(1, int(os.getenv("TMCRA_HTTP_JSON_MAX_ATTEMPTS", "8") or 8))
- retry_codes = {429, 500, 502, 503, 504}
- last_error: Exception | None = None
- for attempt in range(1, max_attempts + 1):
- req = urllib.request.Request(url, data=data, method=method, headers=headers)
- try:
- with urllib.request.urlopen(req, timeout=timeout) as response:
- body = response.read().decode("utf-8", "replace")
- return json.loads(body) if body else {}
- except urllib.error.HTTPError as exc:
- body = exc.read().decode("utf-8", "replace")
- last_error = RuntimeError(f"HTTP {exc.code} {url}: {body[:500]}")
- if exc.code not in retry_codes or attempt >= max_attempts:
- raise last_error from exc
- except (urllib.error.URLError, TimeoutError, json.JSONDecodeError) as exc:
- last_error = exc
- if attempt >= max_attempts:
- raise RuntimeError(f"HTTP request failed {url}: {exc}") from exc
- time.sleep(min(2 ** (attempt - 1), 8))
- raise RuntimeError(f"HTTP request failed {url}: {last_error}")
-
-
-def chat_completion(
- base_url: str,
- model: str,
- messages: list[dict[str, str]],
- *,
- max_tokens: int = 180,
- temperature: float = 0.0,
- api_key: str = "",
-) -> str:
- wire_api = clean_text(os.getenv("TMCRA_ANSWER_WIRE_API", os.getenv("OPENAI_WIRE_API", ""))).lower()
- if wire_api == "responses":
- body = http_json(
- "POST",
- base_url.rstrip("/") + "/responses",
- {
- "model": model,
- "input": messages,
- "max_output_tokens": max_tokens,
- "temperature": temperature,
- },
- timeout=360,
- api_key=api_key,
- )
- output_text = clean_text(body.get("output_text", ""))
- if output_text:
- return output_text
- chunks: list[str] = []
- for item in body.get("output", []) or []:
- for content in item.get("content", []) or []:
- text = content.get("text") or content.get("content") or ""
- if text:
- chunks.append(str(text))
- return "\n".join(chunks).strip()
- body = http_json(
- "POST",
- base_url.rstrip("/") + "/chat/completions",
- {
- "model": model,
- "messages": messages,
- "max_tokens": max_tokens,
- "temperature": temperature,
- "stream": False,
- },
- timeout=360,
- api_key=api_key,
- )
- return (((body.get("choices") or [{}])[0].get("message") or {}).get("content") or "").strip()
-
-
-def answer_llm_config() -> tuple[str, str, str]:
- answer_base_url = clean_text(os.getenv("TMCRA_ANSWER_BASE_URL", ""))
- answer_model = clean_text(os.getenv("TMCRA_ANSWER_MODEL", ""))
- answer_api_key = clean_text(os.getenv("TMCRA_ANSWER_API_KEY", ""))
- if answer_base_url or answer_model or answer_api_key:
- return (
- answer_base_url or os.getenv("GEMMA_BASE_URL", "http://127.0.0.1:18002/v1"),
- answer_model or os.getenv("GEMMA_MODEL", os.getenv("TMCRA_GEMMA_MODEL_NAME", "gemma-4-e4b-it")),
- answer_api_key or clean_text(os.getenv("OPENAI_API_KEY", "")),
- )
- return (
- os.getenv("GEMMA_BASE_URL", "http://127.0.0.1:18002/v1"),
- os.getenv("GEMMA_MODEL", os.getenv("TMCRA_GEMMA_MODEL_NAME", "gemma-4-e4b-it")),
- clean_text(os.getenv("GEMMA_API_KEY", "")),
- )
-
-
-def query_graph_llm_config() -> tuple[str, str, str]:
- base_url = clean_text(os.getenv("TMCRA_QUERY_GRAPH_BASE_URL", ""))
- model = clean_text(os.getenv("TMCRA_QUERY_GRAPH_MODEL", ""))
- api_key = clean_text(os.getenv("TMCRA_QUERY_GRAPH_API_KEY", ""))
- if base_url or model or api_key:
- answer_base_url, answer_model, answer_api_key = answer_llm_config()
- return (
- base_url or answer_base_url,
- model or answer_model,
- api_key or answer_api_key,
- )
- return answer_llm_config()
-
-
-def clean_text(value: Any) -> str:
- return re.sub(r"\s+", " ", str(value or "").strip())
-
-
-def truncate(value: Any, n: int = 500) -> str:
- text = clean_text(value)
- return text[:n]
-
-
-def session_text_chunks(
- *,
- session_id: str,
- date: str,
- turns: list[Mapping[str, Any]],
- max_chars: int,
- max_chunks: int,
-) -> list[str]:
- chunks: list[str] = []
- current: list[str] = [f"LongMemEval session_id={session_id} date={date}"]
- current_len = len(current[0])
- for index, turn in enumerate(turns, start=1):
- role = clean_text(turn.get("role", "unknown"))
- content = clean_text(turn.get("content", ""))
- if not content:
- continue
- part = f"[{session_id} turn={index} role={role}] {content}"
- if current_len + len(part) + 2 > max_chars and len(current) > 1:
- chunks.append("\n".join(current))
- if max_chunks > 0 and len(chunks) >= max_chunks:
- return chunks
- current = [f"LongMemEval session_id={session_id} date={date} continued=true"]
- current_len = len(current[0])
- current.append(part)
- current_len += len(part) + 1
- if len(current) > 1 and (max_chunks <= 0 or len(chunks) < max_chunks):
- chunks.append("\n".join(current))
- return chunks
-
-
-def select_session_indices(row: Mapping[str, Any], *, max_distractors: int) -> list[int]:
- session_ids = [clean_text(item) for item in list(row.get("haystack_session_ids") or [])]
- answer_ids = {clean_text(item) for item in list(row.get("answer_session_ids") or [])}
- answer_indices = [index for index, sid in enumerate(session_ids) if sid in answer_ids]
- distractors: list[int] = []
- latest_candidates = list(range(max(0, len(session_ids) - 4), len(session_ids)))
- first_candidates = list(range(min(3, len(session_ids))))
- mid = len(session_ids) // 2
- middle_candidates = [max(0, mid - 1), mid, min(len(session_ids) - 1, mid + 1)] if session_ids else []
- for index in [*first_candidates, *middle_candidates, *latest_candidates]:
- if index not in answer_indices and index not in distractors and 0 <= index < len(session_ids):
- distractors.append(index)
- if len(distractors) >= max_distractors:
- break
- return sorted(set(answer_indices + distractors))
-
-
-def select_official_full_history_indices(row: Mapping[str, Any]) -> list[int]:
- """Official LongMemEval-style input: use every haystack session in file order."""
- return list(range(len(list(row.get("haystack_sessions") or []))))
-
-
-def build_writer() -> Any:
- from experiments.replacement.adapters.base import LLMProfile
- from experiments.replacement.multi_layer_tmcra_writer import TMCRASuspectAnchoredTransformerMemoryWriter
- from experiments.replacement.semantic_memory_writer import DeterministicMemoryWriteGate, OpenAICompatSemanticMemoryWriter
-
- profile = LLMProfile(
- name="longmemeval_native_writer",
- model=os.getenv("TMCRA_WRITER_MODEL", "gemma-4-e4b-it"),
- base_url=os.getenv("TMCRA_WRITER_BASE_URL", "http://127.0.0.1:18002/v1"),
- api_key=os.getenv("TMCRA_WRITER_API_KEY", ""),
- timeout_seconds=float(os.getenv("TMCRA_WRITER_TIMEOUT_SECONDS", "180")),
- temperature=float(os.getenv("TMCRA_WRITER_TEMPERATURE", "0.0")),
- max_tokens=int(os.getenv("TMCRA_WRITER_MAX_TOKENS", "512")),
- )
- base_writer = OpenAICompatSemanticMemoryWriter(
- profile,
- gate=DeterministicMemoryWriteGate(min_grounding_score=float(os.getenv("TMCRA_WRITER_MIN_GROUNDING_SCORE", "0.35"))),
- max_proposals=int(os.getenv("TMCRA_WRITER_MAX_PROPOSALS", "2")),
- )
- return TMCRASuspectAnchoredTransformerMemoryWriter(
- base_writer,
- max_cells=int(os.getenv("TMCRA_WRITER_MAX_PROPOSALS", "2")),
- value_char_budget=int(os.getenv("TMCRA_WRITER_VALUE_CHAR_BUDGET", "120")),
- source_span_char_budget=int(os.getenv("TMCRA_WRITER_SOURCE_SPAN_CHAR_BUDGET", "180")),
- transformer_layers=int(os.getenv("TMCRA_WRITER_TRANSFORMER_LAYERS", "2")),
- state_attention_k=int(os.getenv("TMCRA_WRITER_STATE_ATTENTION_K", "16")),
- suspect_threshold=float(os.getenv("TMCRA_WRITER_SUSPECT_THRESHOLD", "0.48")),
- suspect_promote_count=int(os.getenv("TMCRA_WRITER_SUSPECT_PROMOTE_COUNT", "2")),
- )
-
-
-def build_adapter(scope_id: str, storage_path: Path) -> Any:
- from experiments.replacement.adapters.memory_adapters import GraphSessionMemoryAdapter
-
- return GraphSessionMemoryAdapter(
- auto_extract=False,
- storage_backend="sqlite",
- storage_path=str(storage_path),
- scope_id=scope_id,
- retrieval_mode=os.getenv("TMCRA_RETRIEVAL_MODE", "hybrid_node_scored"),
- node_model_path=os.getenv("TMCRA_NODE_MODEL_PATH", ""),
- path_model_path=os.getenv("TMCRA_PATH_MODEL_PATH", ""),
- node_model_device=os.getenv("TMCRA_NODE_MODEL_DEVICE", "cpu"),
- candidate_event_k=int(os.getenv("TMCRA_CANDIDATE_EVENT_K", "24")),
- support_path_k=int(os.getenv("TMCRA_SUPPORT_PATH_K", "3")),
- path_tunnel_rescue_k=int(os.getenv("TMCRA_PATH_TUNNEL_RESCUE_K", "2")),
- path_tunnel_rescue_score_floor=float(os.getenv("TMCRA_PATH_TUNNEL_RESCUE_SCORE_FLOOR", "0.0")),
- path_tunnel_rescue_min_age=int(os.getenv("TMCRA_PATH_TUNNEL_RESCUE_MIN_AGE", "0")),
- path_tunnel_rescue_min_score_margin=float(os.getenv("TMCRA_PATH_TUNNEL_RESCUE_MIN_SCORE_MARGIN", "0.0")),
- )
-
-
-def disable_topic_bucket_runtime() -> None:
- import experiments.replacement.adapters.memory_adapters as memory_adapters
-
- def no_topic_bucket(*args: Any, **kwargs: Any) -> dict[str, Any]:
- return {}
-
- def no_apply_topic_bucket(records: list[Any], topic_bucket: Mapping[str, Any]) -> None:
- return None
-
- def no_topic_edges(*args: Any, **kwargs: Any) -> dict[str, Any]:
- return {
- "topic_bridge_disabled": True,
- "dialogue_tunnel_disabled": True,
- "disabled_reason": "longmemeval_native_no_topic_bucket",
- }
-
- def no_topic_rerank(graph: Any, query: str, hits: list[Any], *, top_k: int) -> dict[str, Any]:
- limit = max(1, int(top_k or 1))
- return {
- "hits": list(hits)[:limit],
- "metadata": {
- "topic_bucket_rerank_enabled": False,
- "topic_bucket_disabled": True,
- "topic_bucket_disable_reason": "longmemeval_native_no_topic_bucket",
- "topic_bucket_candidate_count": len(list(hits)),
- "topic_bucket_final_count": len(list(hits)[:limit]),
- },
- }
-
- memory_adapters._assign_topic_bucket_for_text = no_topic_bucket
- memory_adapters._apply_topic_bucket_to_records = no_apply_topic_bucket
- memory_adapters._last_topic_turn = no_topic_bucket
- memory_adapters._add_topic_bridge_edges = no_topic_edges
- memory_adapters._add_dialogue_tunnel_edges = no_topic_edges
- memory_adapters._topic_bucket_rerank_hits = no_topic_rerank
-
-
-def writer_ingest(adapter: Any, writer: Any, text: str, *, qid: str, chunk_id: str) -> dict[str, Any]:
- from experiments.replacement.semantic_memory_writer import build_modelized_facet_unit_records
-
- timestamp = datetime.now(timezone.utc).isoformat(timespec="seconds")
- sidecar_hints = {
- "metadata": {
- "source": "longmemeval_native_flow",
- "question_id": qid,
- "chunk_id": chunk_id,
- }
- }
- t0 = time.perf_counter()
- proposals, writer_metadata = writer.propose_public_turn(
- current_turn=text,
- previous_turn="",
- next_turn="",
- speaker="user",
- session_timestamp=timestamp,
- sidecar_hints=sidecar_hints,
- auxiliary_evidence_texts=[],
- input_mode=os.getenv("TMCRA_WRITER_INPUT_MODE", "delta"),
- )
- graph = getattr(adapter, "graph", None)
- turn_index = int(getattr(graph, "turn_index", 0) or 0) + 1
- gate_result = writer.gate.build_payload(
- proposals=proposals,
- text=f"[{timestamp}] user: {text}",
- raw_text=text,
- speaker="user",
- session_key=qid,
- turn_index=turn_index,
- timestamp=timestamp,
- dia_id=f"longmemeval:{qid}:{chunk_id}",
- sidecar_hints=sidecar_hints,
- writer_metadata=writer_metadata,
- )
- payload = dict(gate_result.payload or {})
- replacement_records = list(payload.get("replacement_memory_records", []) or [])
- base_writer = getattr(writer, "base_writer", writer)
- unit_records, unit_metadata = build_modelized_facet_unit_records(
- base_writer,
- current_turn_text=text,
- parent_records=replacement_records,
- speaker="user",
- timestamp=timestamp,
- turn_index=turn_index,
- max_units=int(os.getenv("TMCRA_MODELIZED_UNIT_WRITER_MAX_UNITS", "14") or 14),
- )
- if unit_records:
- replacement_records.extend(unit_records)
- payload["replacement_memory_records"] = replacement_records
- payload["modelized_facet_unit_writer"] = dict(unit_metadata or {})
- adapter.ingest_turn(
- f"[{timestamp}] user: {text}",
- assistant_text="",
- answer_payload=payload,
- extraction_result={},
- )
- return {
- "seconds": round(time.perf_counter() - t0, 3),
- "proposal_count": len(proposals or []),
- "accepted_count": int(getattr(gate_result, "accepted_count", 0) or 0),
- "suspected_count": int(getattr(gate_result, "suspected_count", 0) or 0),
- "record_count": len(payload.get("replacement_memory_records", []) or []),
- "unit_record_count": len(unit_records),
- "unit_writer_enabled": bool(unit_metadata.get("enabled", False)),
- "unit_writer_metadata": dict(unit_metadata or {}),
- }
-
-
-def retrieval_debug(retrieval: Any) -> dict[str, Any]:
- metadata = dict(getattr(retrieval, "metadata", {}) or {})
- hits = list(getattr(retrieval, "hits", []) or [])
- return {
- "hit_count": len(hits),
- "retrieval_seconds": round(float(getattr(retrieval, "retrieval_seconds", 0.0) or 0.0), 4),
- "retrieval_mode": metadata.get("retrieval_mode"),
- "hybrid_enabled": metadata.get("hybrid_enabled"),
- "decision_score_source": metadata.get("decision_score_source"),
- "selected_event_count": len(metadata.get("selected_event_ids", []) or []),
- "selected_path_count": len(metadata.get("selected_path_ids", []) or []),
- "path_tunnel_enabled": metadata.get("path_tunnel_enabled"),
- "path_tunnel_rescue_candidate_count": metadata.get("path_tunnel_rescue_candidate_count"),
- "path_tunnel_rescue_path_count": len(metadata.get("path_tunnel_rescue_path_ids", []) or []),
- "profile_focused_pack_enabled": metadata.get("profile_focused_pack_enabled"),
- "profile_focused_pack_reason": metadata.get("profile_focused_pack_reason"),
- "profile_focused_pack_hit_count": metadata.get("profile_focused_pack_hit_count"),
- "profile_focused_pack_event_ids": list(metadata.get("profile_focused_pack_event_ids", []) or []),
- "profile_focused_pack_memory_ids": list(metadata.get("profile_focused_pack_memory_ids", []) or []),
- "profile_first_hybrid_enabled": metadata.get("profile_first_hybrid_enabled"),
- "profile_first_event_ids": list(metadata.get("profile_first_event_ids", []) or []),
- "profile_first_memory_ids": list(metadata.get("profile_first_memory_ids", []) or []),
- "facet_query_pack_enabled": metadata.get("facet_query_pack_enabled"),
- "facet_query_pack_inserted_hit_count": metadata.get("facet_query_pack_inserted_hit_count"),
- "unit_coverage_pack_enabled": metadata.get("unit_coverage_pack_enabled"),
- "unit_coverage_candidate_count": metadata.get("unit_coverage_candidate_count"),
- "unit_coverage_selected_unit_count": metadata.get("unit_coverage_selected_unit_count"),
- "unit_coverage_inserted_hit_count": metadata.get("unit_coverage_inserted_hit_count"),
- "multi_unit_chain_slot_enabled": metadata.get("multi_unit_chain_slot_enabled"),
- "multi_unit_chain_slot_formed": metadata.get("multi_unit_chain_slot_formed"),
- "multi_unit_chain_slot_reason": metadata.get("multi_unit_chain_slot_reason"),
- "multi_unit_chain_candidate_count": metadata.get("multi_unit_chain_candidate_count"),
- "multi_unit_chain_selected_unit_count": metadata.get("multi_unit_chain_selected_unit_count"),
- "multi_unit_chain_parent_count": metadata.get("multi_unit_chain_parent_count"),
- "multi_unit_chain_inserted_hit_count": metadata.get("multi_unit_chain_inserted_hit_count"),
- "multi_unit_chain_memory_ids": list(metadata.get("multi_unit_chain_memory_ids", []) or []),
- "multi_unit_chain_focus_tokens": list(metadata.get("multi_unit_chain_focus_tokens", []) or []),
- "top_values": [truncate(getattr(hit, "value", ""), 220) for hit in hits[:5]],
- }
-
-
-STOPWORDS = {
- "a",
- "an",
- "and",
- "are",
- "as",
- "at",
- "be",
- "by",
- "can",
- "did",
- "do",
- "does",
- "for",
- "from",
- "had",
- "has",
- "have",
- "how",
- "i",
- "in",
- "is",
- "it",
- "me",
- "my",
- "of",
- "on",
- "or",
- "that",
- "the",
- "to",
- "was",
- "were",
- "what",
- "when",
- "where",
- "which",
- "who",
- "with",
-}
-
-_ANSWER_WINDOW_PLANNER_CACHE: dict[str, Any] = {}
-_EVIDENCE_UNIT_SELECTOR_CACHE: dict[str, Any] = {}
-_UNIFIED_OPERATION_PLANNER_CACHE: dict[str, Any] = {}
-
-
-def _vector_dot(left: list[float], right: list[float]) -> float:
- if not left or not right:
- return 0.0
- total = 0.0
- for a, b in zip(left, right):
- try:
- total += float(a) * float(b)
- except (TypeError, ValueError):
- continue
- if total != total:
- return 0.0
- return max(0.0, min(1.0, float(total)))
-
-
-def apply_answer_window_semantic_features(question: str, evidence_windows: list[dict[str, Any]]) -> list[dict[str, Any]]:
- mode = os.getenv("TMCRA_ANSWER_WINDOW_PLANNER_SEMANTIC_MODE", "").strip().lower()
- if mode in {"", "off", "false", "0", "disabled", "none"} or not evidence_windows:
- return evidence_windows
- if mode in {"auto", "on", "true", "1"}:
- mode = os.getenv("TMCRA_EMBEDDER_INDEX_RECALL_MODE", "").strip().lower() or "bge_m3"
- try:
- from experiments.replacement.adapters import memory_adapters
- except Exception:
- return evidence_windows
- texts = [clean_text(question), *[clean_text(item.get("text", "")) for item in evidence_windows]]
- try:
- vectors, metadata = memory_adapters._embedder_dense_vectors_for_texts(texts, mode=mode)
- except Exception:
- return evidence_windows
- if not vectors or not vectors[0]:
- return evidence_windows
- query_vector = vectors[0]
- enriched: list[dict[str, Any]] = []
- dense_enabled = bool(metadata.get("write_embedder_dense_enabled"))
- for item, vector in zip(evidence_windows, vectors[1:]):
- score = _vector_dot(query_vector, list(vector or []))
- next_item = dict(item)
- next_item["answer_window_semantic_similarity"] = round(score, 6)
- next_item["answer_window_semantic_mode"] = mode
- next_item["answer_window_semantic_enabled"] = bool(dense_enabled and score > 0.0)
- enriched.append(next_item)
- return enriched
-
-
-def evidence_unit_planner_config() -> tuple[str, str, str]:
- # TMCRA_LAYER_TAG: answer-side LLM planner config; not graph-memory model core.
- answer_base_url, answer_model, answer_api_key = answer_llm_config()
- return (
- clean_text(os.getenv("TMCRA_EVIDENCE_UNIT_PLANNER_BASE_URL", "")) or answer_base_url,
- clean_text(os.getenv("TMCRA_EVIDENCE_UNIT_PLANNER_MODEL", "")) or answer_model,
- clean_text(os.getenv("TMCRA_EVIDENCE_UNIT_PLANNER_API_KEY", "")) or answer_api_key,
- )
-
-
-def token_stem(token: str) -> str:
- token = token.lower()
- for suffix in ("ing", "ed", "es", "s"):
- if len(token) > len(suffix) + 3 and token.endswith(suffix):
- return token[: -len(suffix)]
- return token
-
-
-def content_tokens(text: str) -> list[str]:
- return [
- token_stem(item)
- for item in re.findall(r"[a-zA-Z0-9$]+", text.lower())
- if item and item not in STOPWORDS
- ]
-
-
-def quoted_query_phrases(question: str) -> list[str]:
- phrases: list[str] = []
- seen: set[str] = set()
- for match in re.finditer(r"""['"]([^'"]{2,120})['"]""", question):
- phrase = clean_text(match.group(1))
- key = phrase.lower()
- if phrase and key not in seen:
- phrases.append(phrase)
- seen.add(key)
- return phrases
-
-
-def strict_quoted_phrase_hits(text: str, phrases: list[str]) -> int:
- hits = 0
- for phrase in phrases:
- pattern = re.compile(
- r"(? bool:
- text = question.lower()
- return bool(re.search(r"\b(day|days|week|weeks|month|months|year|years|hour|hours|duration|how long|spent)\b", text))
-
-
-def temporal_event_anchor_text(text: str) -> bool:
- return bool(
- re.search(
- r"\b(start|started|starting|begin|began|finished|finish|completed|complete|ended|end|stopped|since|until)\b",
- text.lower(),
- )
- )
-
-
-def parent_dialogue_header(text: str) -> str:
- value = clean_text(text)
- if not value:
- return ""
- match = re.search(
- r"\bLongMemEval\s+session_id=[A-Za-z0-9_.:-]+\s+date=\d{4}/\d{2}/\d{2}\s+\([^)]+\)\s+\d{2}:\d{2}",
- value,
- )
- if match:
- return clean_text(match.group(0))
- match = re.search(
- r"\[[0-9T:+-]{19,25}\]\s+user:\s+LongMemEval\s+session_id=[A-Za-z0-9_.:-]+\s+date=\d{4}/\d{2}/\d{2}\s+\([^)]+\)\s+\d{2}:\d{2}",
- value,
- )
- return clean_text(match.group(0)) if match else ""
-
-
-def attach_parent_temporal_context(unit: str, parent_text: str) -> str:
- unit_text = clean_text(unit)
- if not unit_text:
- return ""
- if re.search(r"\bdate=\d{4}/\d{2}/\d{2}\b", unit_text) or re.search(r"\bLongMemEval\s+session_id=", unit_text):
- return unit_text
- header = parent_dialogue_header(parent_text)
- if not header:
- return unit_text
- return clean_text(f"{header} {unit_text}")
-
-
-# TMCRA_LAYER_TAG: rule-based evidence query expansion; candidate for modelized retrieval/intent replacement.
-EVIDENCE_QUERY_GENERIC_TERMS = {
- "complement",
- "current",
- "recommend",
- "recommendation",
- "recommendations",
- "suggest",
- "suggestion",
- "suggestions",
-}
-
-
-# TMCRA_LAYER_TAG: manual semantic alias buckets; not part of trained graph-memory model core.
-EVIDENCE_QUERY_ALIAS_GROUPS = (
- {
- "accessory",
- "accessories",
- "bag",
- "camera",
- "cameras",
- "equipment",
- "flash",
- "gear",
- "lens",
- "lenses",
- "photo",
- "photography",
- "sony",
- "tripod",
- },
- {
- "app",
- "apps",
- "dashboard",
- "interface",
- "layout",
- "panel",
- "software",
- "tool",
- "tools",
- "ui",
- "workflow",
- },
- {
- "diet",
- "drink",
- "food",
- "meal",
- "restaurant",
- "snack",
- "taste",
- },
-)
-
-
-def evidence_query_terms(question: str) -> set[str]:
- terms = {term for term in content_tokens(question) if term not in EVIDENCE_QUERY_GENERIC_TERMS}
- expanded = set(terms)
- for group in EVIDENCE_QUERY_ALIAS_GROUPS:
- if terms & group:
- expanded.update(group)
- return expanded
-
-
-def turn_units(text: str) -> list[str]:
- text = clean_text(text)
- if not text:
- return []
- marker = re.compile(r"(\[[^\]]+\bturn=\d+\s+role=(?:user|assistant)\])", re.IGNORECASE)
- parts = marker.split(text)
- units: list[str] = []
- prefix = ""
- index = 0
- sentence_mode = os.getenv("TMCRA_DIALOGUE_TURN_SENTENCE_UNITS", "").strip().lower() not in {
- "",
- "0",
- "false",
- "off",
- "no",
- "disabled",
- "none",
- }
- if parts and not marker.match(parts[0]):
- prefix = parts[0].strip()
- index = 1
- while index < len(parts):
- head = parts[index].strip()
- body = parts[index + 1].strip() if index + 1 < len(parts) else ""
- body_units = [body]
- if sentence_mode:
- split_units = [clean_text(item) for item in re.split(r"(?<=[.!?])\s+", body) if clean_text(item)]
- if split_units:
- body_units = split_units
- for body_unit in body_units:
- unit = clean_text(f"{head} {body_unit}")
- if unit:
- units.append(unit)
- index += 2
- if units:
- if prefix:
- units[0] = clean_text(f"{prefix} {units[0]}")
- return units
- sentence_units = re.split(r"(?<=[.!?])\s+", text)
- return [clean_text(item) for item in sentence_units if clean_text(item)]
-
-
-def score_evidence_unit(question_terms: set[str], question: str, unit: str) -> float:
- unit_norm = unit.lower()
- unit_terms = set(content_tokens(unit))
- overlap = question_terms & unit_terms
- score = float(len(overlap))
- for term in overlap:
- if len(term) >= 5:
- score += 0.35
- raw_terms = [
- item
- for item in re.findall(r"[a-zA-Z0-9$]+", question.lower())
- if item not in STOPWORDS and item not in EVIDENCE_QUERY_GENERIC_TERMS
- ]
- for size in (3, 2):
- for index in range(0, max(0, len(raw_terms) - size + 1)):
- phrase = " ".join(raw_terms[index : index + size])
- if phrase and phrase in unit_norm:
- score += 1.5 if size == 3 else 1.0
- if re.search(r"\b\d+\b|\$\d+", unit_norm):
- score += 0.3
- if "role=user" in unit_norm:
- score += 0.25
- return score
-
-
-def centered_excerpt(unit: str, question_terms: set[str], *, max_chars: int) -> str:
- unit = clean_text(unit)
- if len(unit) <= max_chars:
- return unit
- unit_lower = unit.lower()
- positions = [
- unit_lower.find(term)
- for term in sorted(question_terms, key=len, reverse=True)
- if len(term) >= 4 and unit_lower.find(term) >= 0
- ]
- center = min(positions) if positions else 0
- start = max(0, center - max_chars // 3)
- end = min(len(unit), start + max_chars)
- start = max(0, end - max_chars)
- prefix = "..." if start > 0 else ""
- suffix = "..." if end < len(unit) else ""
- return prefix + unit[start:end].strip() + suffix
-
-
-def make_centered_window(
- units: list[str],
- seed_index: int,
- score: float,
- question_terms: set[str],
- *,
- max_chars: int,
-) -> dict[str, Any]:
- seed_unit = units[seed_index].lower() if 0 <= seed_index < len(units) else ""
- is_dialogue_question_seed = "role=user" in seed_unit and ("?" in seed_unit or "how many" in seed_unit)
- if is_dialogue_question_seed:
- max_chars = max(max_chars, 2200)
- candidate_order = [seed_index, seed_index + 1, seed_index + 2, seed_index - 1, seed_index - 2]
- order = [index for index in candidate_order if 0 <= index < len(units)]
- else:
- order = [seed_index]
- for radius in (1, 2):
- left = seed_index - radius
- right = seed_index + radius
- if left >= 0:
- order.append(left)
- if right < len(units):
- order.append(right)
-
- selected: list[tuple[int, str]] = []
- used = 0
- for index in order:
- unit_l = units[index].lower() if 0 <= index < len(units) else ""
- if "role=assistant" in unit_l:
- budget = 1800 if index == seed_index else 900
- else:
- budget = 950 if index == seed_index else 350 if is_dialogue_question_seed else 520
- piece = centered_excerpt(units[index], question_terms, max_chars=budget)
- extra = len(piece) + 2
- if selected and used + extra > max_chars:
- continue
- selected.append((index, piece))
- used += extra
- selected.sort(key=lambda item: item[0])
- return {
- "score": round(float(score), 3),
- "unit_indexes": [index for index, _ in selected],
- "text": "\n".join(piece for _, piece in selected),
- }
-
-
-def assistant_answer_candidate_windows(
- value: str,
- question: str,
- question_terms: set[str],
- *,
- max_windows: int = 4,
- max_chars: int = 900,
-) -> list[dict[str, Any]]:
- if not assistant_memory_query(question) or not assistant_origin_evidence(value):
- return []
- marker = re.compile(r"(\[[^\]]+\bturn=\d+\s+role=assistant\])", re.IGNORECASE)
- parts = marker.split(clean_text(value))
- candidates: list[tuple[float, int, str]] = []
- raw_question_terms = [
- item
- for item in re.findall(r"[a-zA-Z0-9$]+", clean_text(question).lower())
- if item not in STOPWORDS and item not in EVIDENCE_QUERY_GENERIC_TERMS
- ]
- focus_phrases: set[str] = set()
- for size in (4, 3, 2):
- for raw_index in range(0, max(0, len(raw_question_terms) - size + 1)):
- phrase = " ".join(raw_question_terms[raw_index : raw_index + size])
- if phrase:
- focus_phrases.add(phrase)
- for index in range(1, len(parts), 2):
- head = parts[index].strip()
- body = parts[index + 1].strip() if index + 1 < len(parts) else ""
- if not body:
- continue
- pieces = [
- clean_text(item)
- for item in re.split(r"(?=(?:\d+\.|[-*]\s+|[A-Z][A-Za-z '’-]{2,40}:))|(?<=[.!?])\s+", body)
- if clean_text(item)
- ]
- for local_index, piece in enumerate(pieces):
- if len(piece) < 24:
- continue
- full_piece = clean_text(f"{head} {piece}")
- score = score_evidence_unit(question_terms, question, full_piece)
- piece_terms = set(content_tokens(piece))
- overlap = piece_terms & question_terms
- named_entity_like = bool(
- re.search(r"\b(?:The\s+)?[A-Z][A-Za-z'’-]+(?:\s+(?:of|at|and|the|de|del|la|el|[A-Z][A-Za-z'’-]+)){1,7}\b", piece)
- or re.search(r"['\"][^'\"]{3,80}['\"]", piece)
- )
- if named_entity_like:
- score += 1.25
- if overlap and named_entity_like:
- score += 2.0
- if len(overlap) >= 2:
- score += 1.0
- piece_l = clean_text(piece).lower()
- phrase_hits = sum(1 for phrase in focus_phrases if len(phrase) >= 8 and phrase in piece_l)
- if phrase_hits:
- score += min(4.0, 1.5 * phrase_hits)
- if score < 2.0:
- continue
- text = centered_excerpt(full_piece, question_terms, max_chars=max_chars)
- candidates.append((score, local_index, text))
- candidates.sort(key=lambda item: (item[0], -item[1]), reverse=True)
- windows: list[dict[str, Any]] = []
- seen: set[str] = set()
- for score, _, text in candidates:
- key = text[:220].lower()
- if key in seen:
- continue
- seen.add(key)
- windows.append(
- {
- "score": round(float(score) + 2.0, 3),
- "unit_indexes": [],
- "text": text,
- "assistant_answer_candidate": True,
- "assistant_memory_query": True,
- "assistant_origin_evidence": True,
- "assistant_candidate_side_channel": True,
- "evidence_role": "assistant_answer_candidate",
- "planner_selected": True,
- "planner_score": round(float(score), 3),
- }
- )
- if len(windows) >= max_windows:
- break
- return windows
-
-
-def _load_evidence_unit_selector() -> tuple[Any, Any, Any, dict[str, float]] | None:
- mode = os.getenv("TMCRA_EVIDENCE_UNIT_SELECTOR_MODE", "").strip().lower()
- if mode in {"", "off", "false", "0", "disabled", "none"}:
- return None
- model_path = os.getenv("TMCRA_EVIDENCE_UNIT_SELECTOR_MODEL_PATH", "").strip()
- if not model_path:
- return None
- cache_key = "|".join([model_path, os.getenv("TMCRA_EVIDENCE_UNIT_SELECTOR_DEVICE", "cpu")])
- cached = _EVIDENCE_UNIT_SELECTOR_CACHE.get(cache_key)
- if cached is not None:
- return cached
- try:
- from experiments.replacement import injection_planner as planner
- except Exception:
- return None
- torch_module = getattr(planner, "torch", None)
- if torch_module is None:
- return None
- try:
- device = torch_module.device(os.getenv("TMCRA_EVIDENCE_UNIT_SELECTOR_DEVICE", "cpu") or "cpu")
- payload = torch_module.load(Path(model_path), map_location=device, weights_only=False)
- config = planner.InjectionPlannerConfig.from_dict(dict(payload.get("config", {}) or {}))
- model = planner.InjectionPlannerModel(config).to(device)
- model.load_state_dict(dict(payload.get("state_dict", {}) or {}), strict=False)
- model.eval()
- thresholds = {
- "selection_threshold": float(os.getenv("TMCRA_EVIDENCE_UNIT_SELECTOR_SELECTION_THRESHOLD", "0.45")),
- "row_threshold": float(os.getenv("TMCRA_EVIDENCE_UNIT_SELECTOR_ROW_THRESHOLD", "0.35")),
- "logic_threshold": float(os.getenv("TMCRA_EVIDENCE_UNIT_SELECTOR_LOGIC_THRESHOLD", "0.45")),
- }
- except Exception:
- return None
- loaded = (planner, torch_module, model, thresholds)
- _EVIDENCE_UNIT_SELECTOR_CACHE[cache_key] = loaded
- return loaded
-
-
-def model_selected_evidence_unit_indexes(
- question: str,
- units: list[str],
- hit_metadata: Mapping[str, Any],
- scored_units: list[tuple[float, int]],
-) -> list[tuple[float, int]]:
- loaded = _load_evidence_unit_selector()
- if loaded is None or not units:
- return []
- planner, torch_module, model, thresholds = loaded
- max_units = max(4, int(os.getenv("TMCRA_EVIDENCE_UNIT_SELECTOR_MAX_UNITS", "14") or 14))
- top_scored = sorted(scored_units, key=lambda item: item[0], reverse=True)[:max_units]
- candidate_indexes = sorted({index for _, index in top_scored if 0 <= index < len(units)})
- if not candidate_indexes:
- return []
- score_by_index = {index: score for score, index in scored_units}
- max_unit_score = max([abs(float(score)) for score, _ in top_scored] + [1.0])
- retrieval_score = max(
- 0.0,
- min(
- 1.0,
- max(
- metadata_float_from_mapping(hit_metadata, "hybrid_score"),
- metadata_float_from_mapping(hit_metadata, "recall_score"),
- metadata_float_from_mapping(hit_metadata, "event_score"),
- metadata_float_from_mapping(hit_metadata, "path_score"),
- ),
- ),
- )
- graph_score = max(
- 0.0,
- min(
- 1.0,
- max(
- metadata_float_from_mapping(hit_metadata, "answer_plan_score"),
- metadata_float_from_mapping(hit_metadata, "answer_plan_selected_score"),
- metadata_float_from_mapping(hit_metadata, "answer_plan_adjusted_score"),
- ),
- ),
- )
- temporal_state = clean_text(hit_metadata.get("injection_planner_temporal_state", "")) or "irrelevant"
- evidence_role = clean_text(hit_metadata.get("injection_planner_evidence_role", "")) or clean_text(
- hit_metadata.get("evidence_role", "")
- )
- candidates = []
- for rank, index in enumerate(candidate_indexes, start=1):
- unit_score = float(score_by_index.get(index, 0.0) or 0.0)
- candidates.append(
- {
- "id": f"unit_{index}",
- "text": clean_text(units[index]),
- "layer": "event",
- "temporal_state": temporal_state if temporal_state in getattr(planner, "TEMPORAL_STATES", ()) else "irrelevant",
- "logic_roles": ["evidence"],
- "evidence_role": evidence_role if evidence_role in getattr(planner, "EVIDENCE_ROLES", ()) else "",
- "retrieval_score": retrieval_score,
- "graph_score": graph_score,
- "tunnel_score": max(0.0, min(1.0, metadata_float_from_mapping(hit_metadata, "path_score"))),
- "topic_similarity": max(0.0, min(1.0, unit_score / max_unit_score)),
- "confidence": retrieval_score,
- "rank_score": round(1.0 / float(rank), 6),
- "branch_depth": 1,
- }
- )
- if len(candidates) < 3:
- return []
- try:
- row = {"id": "evidence_unit_selector", "query": question, "candidates": candidates, "gold": {}}
- dataset = planner.InjectionPlannerDataset([row], model.config)
- batch = planner.collate_injection_batch([dataset[0]])
- device = next(model.parameters()).device
- model_batch = {key: value.to(device) if hasattr(value, "to") else value for key, value in dict(batch).items()}
- with torch_module.no_grad():
- outputs = model(model_batch["features"], model_batch["valid_mask"])
- row_score = float(torch_module.sigmoid(outputs["should_inject_logits"])[0].detach().cpu().item())
- selection_scores = torch_module.sigmoid(outputs["selection_logits"])[0].detach().cpu().tolist()
- role_indices = torch_module.argmax(outputs["evidence_role_logits"], dim=-1)[0].detach().cpu().tolist()
- except Exception:
- return []
- if row_score < thresholds["row_threshold"]:
- return []
- selected: list[tuple[float, int]] = []
- for local_index, unit_index in enumerate(candidate_indexes):
- role = planner.EVIDENCE_ROLES[int(role_indices[local_index])]
- if role in {"noise", "negative_evidence"}:
- continue
- score = float(selection_scores[local_index])
- if score >= thresholds["selection_threshold"]:
- selected.append((score, unit_index))
- selected.sort(key=lambda item: item[0], reverse=True)
- return selected[: max(1, int(os.getenv("TMCRA_EVIDENCE_UNIT_SELECTOR_TOP_K", "4") or 4))]
-
-
-def metadata_float_from_mapping(metadata: Mapping[str, Any], key: str, default: float = 0.0) -> float:
- try:
- return float(metadata.get(key, default) or default)
- except (TypeError, ValueError):
- return default
-
-
-def evidence_windows_for_hit(question: str, hit: Any, *, max_chars: int = 1800) -> list[dict[str, Any]]:
- value = clean_text(getattr(hit, "value", ""))
- if not value:
- return []
- hit_metadata = dict(getattr(hit, "metadata", {}) or {})
- evidence_snippet_role = clean_text(hit_metadata.get("evidence_snippet_role", ""))
- unit_coverage_parent_event = bool(hit_metadata.get("unit_coverage_pack")) and evidence_snippet_role == "unit_coverage_parent_event"
- multi_chain_parent_event = bool(hit_metadata.get("multi_unit_chain_slot")) and evidence_snippet_role == "multi_unit_chain_parent_event"
- if unit_coverage_parent_event or multi_chain_parent_event:
- try:
- max_chars = max(max_chars, int(os.getenv("TMCRA_UNIT_COVERAGE_PARENT_WINDOW_CHARS", "3800") or 3800))
- except (TypeError, ValueError):
- max_chars = max(max_chars, 3800)
- if assistant_memory_query(question) and assistant_origin_evidence(value):
- try:
- max_chars = max(max_chars, int(os.getenv("TMCRA_ASSISTANT_MEMORY_WINDOW_CHARS", "4200") or 4200))
- except (TypeError, ValueError):
- max_chars = max(max_chars, 4200)
- def metadata_float(key: str, default: float = 0.0) -> float:
- try:
- return float(hit_metadata.get(key, default) or default)
- except (TypeError, ValueError):
- return default
-
- answer_plan_score = max(
- metadata_float("answer_plan_score"),
- metadata_float("answer_plan_selected_score"),
- metadata_float("answer_plan_current_score"),
- metadata_float("answer_plan_adjusted_score"),
- )
- answer_plan_selected = bool(hit_metadata.get("answer_plan_selected", False)) and answer_plan_score > 0.0
- try:
- answer_plan_rank = int(hit_metadata.get("answer_plan_rank", 0) or 0)
- except (TypeError, ValueError):
- answer_plan_rank = 0
- parent_profile_summary = clean_text(
- hit_metadata.get("profile_first_parent_summary", "")
- or hit_metadata.get("profile_summary", "")
- or hit_metadata.get("profile_value", "")
- )
- if hit_metadata.get("profile_first_source_support") and parent_profile_summary:
- value = clean_text(f"[profile summary] {parent_profile_summary} {value}")
- if unit_coverage_parent_event or multi_chain_parent_event:
- child_value = clean_text(hit_metadata.get("unit_coverage_child_value", "") or hit_metadata.get("multi_unit_chain_child_value", ""))
- child_source_span = clean_text(
- hit_metadata.get("unit_coverage_child_source_span", "")
- or hit_metadata.get("multi_unit_chain_child_source_span", "")
- )
- child_anchor = child_source_span or child_value
- if child_anchor:
- parent_l = value.lower()
- child_l = child_anchor.lower()
- pos = parent_l.find(child_l[: min(len(child_l), 120)])
- if pos >= 0:
- half = max_chars // 2
- start = max(0, pos - half)
- end = min(len(value), pos + len(child_anchor) + half)
- value = clean_text(value[start:end])
- elif child_value or child_source_span:
- value = clean_text(value)
- if child_value or child_source_span:
- value = clean_text(
- f"[unit evidence] {child_source_span or child_value}"
- f"{' | unit_value=' + child_value if child_value and child_value != child_source_span else ''}\n"
- f"[parent evidence] {value}"
- )
- units = turn_units(value)
- if not units:
- return []
- question_terms = evidence_query_terms(question)
- profile_bonus = 0.0
- if hit_metadata.get("profile_first_hybrid_rescue"):
- profile_bonus += 8.0
- elif hit_metadata.get("profile_layer") or str(getattr(hit, "source_kind", "")).startswith("public_dialog_profile"):
- profile_bonus += 4.0
- scored = [
- (score_evidence_unit(question_terms, question, unit) + profile_bonus, index)
- for index, unit in enumerate(units)
- ]
- scored.sort(key=lambda item: item[0], reverse=True)
- seed_indexes: list[tuple[float, int]] = []
- model_unit_indexes = model_selected_evidence_unit_indexes(question, units, hit_metadata, scored)
- for model_score, index in model_unit_indexes:
- if any(existing == index for _, existing in seed_indexes):
- continue
- seed_indexes.append((max(float(model_score) * 10.0, float(scored[0][0]) if scored else 0.0), index))
- seed_limit = 5 if assistant_memory_query(question) and assistant_origin_evidence(value) else 3
- if len(seed_indexes) >= seed_limit:
- break
- for score, index in scored:
- if score <= 0 and seed_indexes:
- break
- if score <= 0 and not seed_indexes:
- seed_indexes.append((score, index))
- break
- if any(abs(index - existing) <= 1 for _, existing in seed_indexes):
- continue
- seed_indexes.append((score, index))
- seed_limit = 5 if assistant_memory_query(question) and assistant_origin_evidence(value) else 3
- if len(seed_indexes) >= seed_limit:
- break
- windows: list[dict[str, Any]] = []
- for score, index in seed_indexes:
- window = make_centered_window(units, index, score, question_terms, max_chars=max_chars)
- window["memory_id"] = getattr(hit, "memory_id", "")
- window["original_chars"] = len(value)
- window["planner_selected"] = bool(hit_metadata.get("injection_planner_selected", False)) or answer_plan_selected
- evidence_role = clean_text(hit_metadata.get("injection_planner_evidence_role", ""))
- if not evidence_role and answer_plan_selected:
- evidence_role = "direct_answer"
- memory_id = clean_text(getattr(hit, "memory_id", ""))
- source_kind = clean_text(getattr(hit, "source_kind", ""))
- if (
- not evidence_role
- and (
- bool(hit_metadata.get("profile_first_hybrid_rescue"))
- or bool(hit_metadata.get("profile_protected_slot"))
- or bool(hit_metadata.get("profile_layer"))
- or ".subject." in memory_id
- or source_kind.startswith("public_dialog_profile")
- )
- ):
- evidence_role = "profile_fact"
- window["evidence_role"] = evidence_role
- window["temporal_state"] = clean_text(hit_metadata.get("injection_planner_temporal_state", ""))
- promoted_answer_plan_score = answer_plan_score if answer_plan_selected else 0.0
- window["planner_score"] = max(metadata_float("injection_planner_score"), promoted_answer_plan_score)
- window["answer_plan_score"] = answer_plan_score
- window["answer_plan_selected"] = answer_plan_selected
- window["answer_plan_rank"] = answer_plan_rank
- if hit_metadata.get("semantic_coverage_expansion"):
- window["semantic_coverage_expansion"] = True
- window["semantic_coverage_score"] = metadata_float("semantic_coverage_score")
- window["semantic_coverage_source_memory_id"] = clean_text(hit_metadata.get("semantic_coverage_source_memory_id", ""))
- if hit_metadata.get("unit_coverage_pack"):
- window["unit_coverage_pack"] = True
- window["unit_kind"] = clean_text(hit_metadata.get("unit_kind", ""))
- window["facet_type"] = clean_text(hit_metadata.get("facet_type", ""))
- window["unit_coverage_semantic_event_unit"] = bool(hit_metadata.get("unit_coverage_semantic_event_unit", False))
- window["unit_coverage_semantic_event_priority"] = bool(hit_metadata.get("unit_coverage_semantic_event_priority", False))
- window["unit_coverage_profile_shadow_unit"] = bool(hit_metadata.get("unit_coverage_profile_shadow_unit", False))
- window["evidence_role"] = window["evidence_role"] or clean_text(hit_metadata.get("unit_kind", ""))
- if hit_metadata.get("multi_unit_chain_slot"):
- window["multi_unit_chain_slot"] = True
- window["multi_unit_chain_bundle"] = bool(hit_metadata.get("multi_unit_chain_bundle", False))
- window["multi_unit_chain_temporal_comparison"] = bool(hit_metadata.get("multi_unit_chain_temporal_comparison", False))
- window["multi_unit_chain_semantic_event_unit"] = bool(hit_metadata.get("multi_unit_chain_semantic_event_unit", False))
- window["multi_unit_chain_semantic_event_priority"] = bool(hit_metadata.get("multi_unit_chain_semantic_event_priority", False))
- window["multi_unit_chain_profile_shadow_unit"] = bool(hit_metadata.get("multi_unit_chain_profile_shadow_unit", False))
- window["unit_kind"] = window.get("unit_kind", "") or clean_text(hit_metadata.get("unit_kind", ""))
- window["facet_type"] = window.get("facet_type", "") or clean_text(hit_metadata.get("facet_type", ""))
- window["evidence_role"] = window["evidence_role"] or "multi_unit_chain"
- window["multi_unit_chain_score"] = float(hit_metadata.get("multi_unit_chain_score", 0.0) or 0.0)
- if hit_metadata.get("facet_query_pack"):
- window["facet_query_pack"] = True
- window["facet_type"] = window.get("facet_type", "") or clean_text(hit_metadata.get("facet_type", ""))
- windows.append(window)
- try:
- assistant_candidate_limit = max(0, int(os.getenv("TMCRA_ASSISTANT_ANSWER_CANDIDATE_LIMIT", "0") or 0))
- except (TypeError, ValueError):
- assistant_candidate_limit = 4
- if assistant_candidate_limit:
- for candidate in assistant_answer_candidate_windows(
- value,
- question,
- question_terms,
- max_windows=assistant_candidate_limit,
- max_chars=max(700, min(max_chars, 1200)),
- ):
- candidate["memory_id"] = getattr(hit, "memory_id", "")
- candidate["original_chars"] = len(value)
- candidate["temporal_state"] = clean_text(hit_metadata.get("injection_planner_temporal_state", ""))
- candidate["answer_plan_score"] = answer_plan_score
- candidate["answer_plan_selected"] = answer_plan_selected
- candidate["answer_plan_rank"] = answer_plan_rank
- windows.append(candidate)
- return windows
-
-
-def build_answer_evidence(question: str, memory_hits: list[Any]) -> list[dict[str, Any]]:
- windows: list[dict[str, Any]] = []
- try:
- hit_scan_limit = max(8, int(os.getenv("TMCRA_ANSWER_EVIDENCE_HIT_SCAN_LIMIT", "16") or 16))
- build_limit = max(8, int(os.getenv("TMCRA_ANSWER_EVIDENCE_BUILD_LIMIT", "16") or 16))
- except (TypeError, ValueError):
- hit_scan_limit, build_limit = 16, 16
- for hit in memory_hits[:hit_scan_limit]:
- windows.extend(evidence_windows_for_hit(question, hit))
- assistant_query = assistant_memory_query(question)
- for index, item in enumerate(windows):
- text = clean_text(item.get("text", ""))
- if assistant_query and assistant_origin_evidence(text):
- next_item = dict(item)
- next_item["assistant_memory_query"] = True
- next_item["assistant_origin_evidence"] = True
- if not clean_text(next_item.get("evidence_role", "")):
- next_item["evidence_role"] = "assistant_origin_detail"
- windows[index] = next_item
-
- def planner_window_key(item: dict[str, Any]) -> tuple[bool, bool, float, float, int]:
- evidence_role = clean_text(item.get("evidence_role", ""))
- role_allows_evidence = evidence_role not in {"noise", "negative_evidence"}
- planner_score = float(item.get("planner_score", 0.0) or 0.0) if role_allows_evidence else 0.0
- return (
- bool(item.get("planner_selected", False)),
- role_allows_evidence,
- planner_score,
- float(item.get("score", 0.0) or 0.0),
- len(item.get("text", "")),
- )
-
- assistant_candidates = [item for item in windows if bool(item.get("assistant_candidate_side_channel", False))]
- windows = [item for item in windows if not bool(item.get("assistant_candidate_side_channel", False))]
-
- windows.sort(
- key=planner_window_key,
- reverse=True,
- )
- windows = compose_layered_evidence_windows(windows, build_limit=build_limit)
- windows = diversify_evidence_windows(windows)
- assistant_candidates.sort(key=planner_window_key, reverse=True)
- try:
- side_limit = max(0, int(os.getenv("TMCRA_ASSISTANT_ANSWER_CANDIDATE_SIDE_LIMIT", "12") or 12))
- except (TypeError, ValueError):
- side_limit = 12
- return windows[:build_limit] + assistant_candidates[:side_limit]
-
-
-def assistant_memory_query(question: str) -> bool:
- question_l = clean_text(question).lower()
- return bool(
- re.search(
- r"\b(?:you said|you suggested|you recommended|you told|you wrote|you gave|you mentioned|"
- r"previous chat|previous conversation|last time|remind me|we talked|we discussed|"
- r"script you wrote|list you provided|what was the .* you said|what did you say)\b",
- question_l,
- )
- )
-
-
-def assistant_origin_evidence(text: str) -> bool:
- text_l = clean_text(text).lower()
- if not text_l:
- return False
- if re.search(r"\[[^\]]+\brole=assistant\]", text_l):
- return True
- return bool(re.search(r"\b(?:assistant\]|role=assistant|assistant:)\b", text_l))
-
-
-def evidence_window_layer(item: dict[str, Any]) -> str:
- memory_id = clean_text(item.get("memory_id", ""))
- evidence_role = clean_text(item.get("evidence_role", "")).lower()
- text = clean_text(item.get("text", "")).lower()
- if bool(item.get("assistant_origin_evidence", False)) or evidence_role == "assistant_origin_detail":
- return "assistant"
- if bool(item.get("multi_unit_chain_slot", False)):
- return "chain"
- if bool(item.get("unit_coverage_pack", False)):
- return "unit"
- if bool(item.get("facet_query_pack", False)):
- return "facet"
- if (
- "public_dialog_profile" in memory_id
- or ".subject." in memory_id
- or "profile" in evidence_role
- or text.startswith("[profile summary]")
- ):
- return "profile"
- if bool(item.get("answer_plan_selected", False)) or evidence_role in {"direct_answer", "positive", "supporting_fact"}:
- return "direct"
- return "base"
-
-
-def compose_layered_evidence_windows(evidence_windows: list[dict[str, Any]], *, build_limit: int) -> list[dict[str, Any]]:
- mode = os.getenv("TMCRA_LAYERED_EVIDENCE_COMPOSE_MODE", "on").strip().lower()
- if mode in {"", "0", "false", "off", "no", "disabled", "none"}:
- return evidence_windows
- try:
- profile_limit = max(0, int(os.getenv("TMCRA_LAYERED_EVIDENCE_PROFILE_LIMIT", "3") or 3))
- assistant_limit = max(0, int(os.getenv("TMCRA_LAYERED_EVIDENCE_ASSISTANT_LIMIT", "4") or 4))
- direct_limit = max(0, int(os.getenv("TMCRA_LAYERED_EVIDENCE_DIRECT_LIMIT", "4") or 4))
- base_limit = max(0, int(os.getenv("TMCRA_LAYERED_EVIDENCE_BASE_LIMIT", "5") or 5))
- unit_limit = max(0, int(os.getenv("TMCRA_LAYERED_EVIDENCE_UNIT_LIMIT", "4") or 4))
- chain_limit = max(0, int(os.getenv("TMCRA_LAYERED_EVIDENCE_CHAIN_LIMIT", "4") or 4))
- facet_limit = max(0, int(os.getenv("TMCRA_LAYERED_EVIDENCE_FACET_LIMIT", "2") or 2))
- except (TypeError, ValueError):
- profile_limit, assistant_limit, direct_limit, base_limit, unit_limit, chain_limit, facet_limit = 3, 4, 4, 5, 4, 4, 2
- buckets: dict[str, list[dict[str, Any]]] = {
- "profile": [],
- "assistant": [],
- "direct": [],
- "base": [],
- "unit": [],
- "chain": [],
- "facet": [],
- }
- for item in evidence_windows:
- buckets.setdefault(evidence_window_layer(item), buckets["base"]).append(item)
-
- def coverage_bucket_key(item: dict[str, Any]) -> tuple[int, int, int, int, float, float, int]:
- unit_kind = clean_text(item.get("unit_kind", "")).lower()
- evidence_role = clean_text(item.get("evidence_role", "")).lower()
- memory_id = clean_text(item.get("memory_id", "")).lower()
- semantic_event = int(
- bool(item.get("multi_unit_chain_semantic_event_priority", False))
- or bool(item.get("unit_coverage_semantic_event_priority", False))
- )
- profile_shadow = int(
- unit_kind == "profile_shadow_unit"
- or bool(item.get("multi_unit_chain_profile_shadow_unit", False))
- or bool(item.get("unit_coverage_profile_shadow_unit", False))
- or "profile_shadow" in memory_id
- )
- selected = int(bool(item.get("planner_selected", False)) or bool(item.get("evidence_unit_planner_selected", False)))
- parent_event = int("parent_event" in evidence_role or "#multi_parent:" in memory_id or "#unit_parent:" in memory_id)
- return (
- semantic_event,
- selected,
- profile_shadow,
- parent_event,
- float(item.get("multi_unit_chain_score", 0.0) or 0.0),
- float(item.get("score", 0.0) or 0.0),
- len(clean_text(item.get("text", ""))),
- )
-
- buckets["chain"].sort(key=coverage_bucket_key, reverse=True)
- buckets["unit"].sort(key=coverage_bucket_key, reverse=True)
-
- ordered: list[dict[str, Any]] = []
- seen_keys: set[str] = set()
-
- def add_from(items: list[dict[str, Any]], limit: int) -> None:
- if limit <= 0:
- return
- added = 0
- for item in items:
- memory_id = clean_text(item.get("memory_id", ""))
- text_key = clean_text(item.get("text", ""))[:240].lower()
- key = memory_id or text_key
- if key and key in seen_keys:
- continue
- if key:
- seen_keys.add(key)
- ordered.append(item)
- added += 1
- if len(ordered) >= build_limit or added >= limit:
- break
-
- # Stable evidence composition: profile/direct/base keep the normal answer surface;
- # unit and chain provide bounded coverage slots instead of competing for all top ranks.
- temporal_chain_first = any(bool(item.get("multi_unit_chain_temporal_comparison", False)) for item in buckets["chain"])
- add_from(buckets["profile"], profile_limit)
- add_from(buckets["assistant"], assistant_limit)
- add_from(buckets["direct"], direct_limit)
- if temporal_chain_first:
- add_from(buckets["chain"], chain_limit)
- add_from(buckets["base"], base_limit)
- add_from(buckets["unit"], unit_limit)
- else:
- add_from(buckets["base"], base_limit)
- add_from(buckets["unit"], unit_limit)
- add_from(buckets["chain"], chain_limit)
- add_from(buckets["facet"], facet_limit)
- for item in evidence_windows:
- if len(ordered) >= build_limit:
- break
- memory_id = clean_text(item.get("memory_id", ""))
- text_key = clean_text(item.get("text", ""))[:240].lower()
- key = memory_id or text_key
- if key and key in seen_keys:
- continue
- if key:
- seen_keys.add(key)
- ordered.append(item)
- return ordered
-
-
-def expand_dialogue_chain_hits(question: str, memory_hits: list[Any], graph: Any | None) -> list[Any]:
- if graph is None or not memory_hits:
- return list(memory_hits)
- mode = os.getenv("TMCRA_DIALOGUE_CHAIN_EXPANSION_MODE", "adjacent").strip().lower()
- if mode in {"", "off", "false", "0", "disabled", "none"}:
- return list(memory_hits)
- try:
- radius = max(0, int(os.getenv("TMCRA_DIALOGUE_CHAIN_EXPANSION_RADIUS", "3") or 3))
- max_total = max(0, int(os.getenv("TMCRA_DIALOGUE_CHAIN_EXPANSION_MAX_TOTAL", "5") or 5))
- max_per_seed = max(1, int(os.getenv("TMCRA_DIALOGUE_CHAIN_EXPANSION_MAX_PER_SEED", "2") or 2))
- except (TypeError, ValueError):
- radius, max_total, max_per_seed = 3, 5, 2
- if radius <= 0 or max_total <= 0:
- return list(memory_hits)
- question_terms = evidence_query_terms(question)
- if not question_terms:
- return list(memory_hits)
- records_by_id = getattr(graph, "records_by_id", {}) or {}
- if not records_by_id:
- return list(memory_hits)
- existing_ids = {clean_text(getattr(hit, "memory_id", "")) for hit in memory_hits if clean_text(getattr(hit, "memory_id", ""))}
- seed_infos = []
- for seed_index, hit in enumerate(memory_hits[:12]):
- session_id = dialogue_session_id_from_text(hit_text_for_chain(hit))
- if not session_id:
- continue
- seed_infos.append((seed_index, session_id, int(getattr(hit, "turn_index", 0) or 0), clean_text(getattr(hit, "memory_id", ""))))
- if not seed_infos:
- return list(memory_hits)
-
- candidates_by_seed: dict[str, list[Any]] = {}
- for record in records_by_id.values():
- memory_id = clean_text(getattr(record, "memory_id", ""))
- if not memory_id or memory_id in existing_ids:
- continue
- state = clean_text(getattr(record, "state", "active")).lower()
- if state not in {"active", "parallel_active", "evidence"}:
- continue
- value = clean_text(getattr(record, "value", ""))
- metadata = dict(getattr(record, "metadata", {}) or {})
- session_id = dialogue_session_id_from_text(" ".join([value, clean_text(metadata.get("raw_text", "")), clean_text(metadata.get("source_turn_text", ""))]))
- if not session_id:
- continue
- record_turn = int(getattr(record, "turn_index", 0) or 0)
- for seed_index, seed_session, seed_turn, seed_id in seed_infos:
- if session_id != seed_session:
- continue
- distance = abs(record_turn - seed_turn) if seed_turn and record_turn else 0
- if distance > radius:
- continue
- score_metadata = {**metadata, "source_kind": clean_text(getattr(record, "source_kind", ""))}
- score = dialogue_chain_candidate_score(question, question_terms, value, score_metadata, distance)
- if score <= 0:
- continue
- hit = record_to_chain_hit(record, score=score, seed_id=seed_id, distance=distance, session_id=session_id)
- candidates_by_seed.setdefault(seed_id, []).append(hit)
-
- expanded: list[Any] = []
- added_ids: set[str] = set()
- total_added = 0
- for hit_index, hit in enumerate(memory_hits):
- expanded.append(hit)
- seed_id = clean_text(getattr(hit, "memory_id", ""))
- candidates = candidates_by_seed.get(seed_id, [])
- candidates.sort(
- key=lambda item: (
- -float(getattr(item, "score", 0.0) or 0.0),
- int((getattr(item, "metadata", {}) or {}).get("dialogue_chain_distance", 99) or 99),
- int(getattr(item, "turn_index", 0) or 0),
- clean_text(getattr(item, "memory_id", "")),
- )
- )
- per_seed_added = 0
- for candidate in candidates:
- candidate_id = clean_text(getattr(candidate, "memory_id", ""))
- if not candidate_id or candidate_id in existing_ids or candidate_id in added_ids:
- continue
- expanded.append(candidate)
- added_ids.add(candidate_id)
- total_added += 1
- per_seed_added += 1
- if per_seed_added >= max_per_seed or total_added >= max_total:
- break
- if total_added >= max_total:
- expanded.extend(memory_hits[hit_index + 1 :])
- break
- if not added_ids:
- return list(memory_hits)
- return dedupe_hits_by_memory_id(expanded)
-
-
-def expand_semantic_coverage_hits(question: str, memory_hits: list[Any], graph: Any | None) -> list[Any]:
- mode = os.getenv("TMCRA_SEMANTIC_COVERAGE_EXPANSION_MODE", "").strip().lower()
- if mode in {"", "off", "false", "0", "disabled", "none"}:
- return list(memory_hits)
- if graph is None:
- return list(memory_hits)
- records_by_id = getattr(graph, "records_by_id", {}) or {}
- if not records_by_id:
- return list(memory_hits)
- embedder_mode = os.getenv("TMCRA_SEMANTIC_COVERAGE_EMBEDDER_MODE", "").strip() or os.getenv(
- "TMCRA_EMBEDDER_INDEX_RECALL_MODE", ""
- ).strip() or "bge_m3"
- try:
- max_records = max(16, int(os.getenv("TMCRA_SEMANTIC_COVERAGE_MAX_RECORDS", "160") or 160))
- max_units = max(24, int(os.getenv("TMCRA_SEMANTIC_COVERAGE_MAX_UNITS", "260") or 260))
- max_add = max(1, int(os.getenv("TMCRA_SEMANTIC_COVERAGE_MAX_ADD", "16") or 16))
- score_floor = float(os.getenv("TMCRA_SEMANTIC_COVERAGE_SCORE_FLOOR", "0.74") or 0.74)
- except (TypeError, ValueError):
- max_records, max_units, max_add, score_floor = 160, 260, 16, 0.74
- try:
- from experiments.replacement.adapters import memory_adapters
- except Exception:
- return list(memory_hits)
- existing_unit_texts = {clean_text(getattr(hit, "value", "")) for hit in memory_hits if clean_text(getattr(hit, "value", ""))}
- candidates: list[tuple[Any, int, str]] = []
- for record in list(records_by_id.values())[:max_records]:
- state = clean_text(getattr(record, "state", "active")).lower()
- if state not in {"active", "parallel_active", "evidence"}:
- continue
- value = clean_text(getattr(record, "value", ""))
- if not value:
- continue
- for unit_index, unit in enumerate((turn_units(value) or [value])[:8]):
- text = clean_text(unit)
- if not text or text in existing_unit_texts:
- continue
- candidates.append((record, unit_index, text))
- if len(candidates) >= max_units:
- break
- if len(candidates) >= max_units:
- break
- if not candidates:
- return list(memory_hits)
- texts = [clean_text(question), *[text for _, _, text in candidates]]
- try:
- vectors, metadata = memory_adapters._embedder_dense_vectors_for_texts(texts, mode=embedder_mode)
- except Exception:
- return list(memory_hits)
- if not vectors or not vectors[0] or not bool(metadata.get("write_embedder_dense_enabled", False)):
- return list(memory_hits)
- query_vector = list(vectors[0] or [])
- query_phrases = quoted_query_phrases(question)
- query_needs_temporal_anchors = bool(query_phrases and temporal_duration_query(question))
- scored: list[tuple[float, str, Any, int, str]] = []
- for (record, unit_index, text), vector in zip(candidates, vectors[1:]):
- if query_needs_temporal_anchors:
- text = attach_parent_temporal_context(text, clean_text(getattr(record, "value", "")))
- score = _vector_dot(query_vector, list(vector or []))
- phrase_hits = strict_quoted_phrase_hits(text, query_phrases)
- if query_phrases and phrase_hits <= 0:
- continue
- if query_needs_temporal_anchors and not temporal_event_anchor_text(text):
- continue
- if phrase_hits:
- score = min(1.0, score + (0.025 * min(3, phrase_hits)))
- if score < score_floor:
- continue
- memory_id = clean_text(getattr(record, "memory_id", "")) or clean_text(getattr(record, "slot_key", ""))
- scored.append((score, memory_id, record, unit_index, text))
- if not scored:
- return list(memory_hits)
- scored.sort(key=lambda item: (-item[0], item[1], item[3]))
- expanded = list(memory_hits)
- added = 0
- seen_semantic_ids: set[str] = set()
- for score, memory_id, record, unit_index, text in scored:
- semantic_id = f"{memory_id}#semantic_unit_{unit_index}"
- if semantic_id in seen_semantic_ids:
- continue
- metadata = dict(getattr(record, "metadata", {}) or {})
- metadata.update(
- {
- "semantic_coverage_expansion": True,
- "semantic_coverage_score": round(float(score), 6),
- "semantic_coverage_unit_index": int(unit_index),
- "semantic_coverage_source_memory_id": memory_id,
- "injection_planner_selected": True,
- "injection_planner_score": round(float(score), 6),
- "injection_planner_evidence_role": "positive",
- "hybrid_score": max(float(score), metadata_float_from_mapping(metadata, "hybrid_score")),
- "recall_score": max(float(score), metadata_float_from_mapping(metadata, "recall_score")),
- }
- )
- expanded.append(
- SimpleNamespace(
- memory_id=semantic_id,
- category=clean_text(getattr(record, "category", "")),
- value=text,
- relation=clean_text(getattr(record, "relation", "related_to")) or "related_to",
- anchors=list(getattr(record, "anchor_concepts", []) or []),
- score=float(score),
- source_kind="semantic_coverage_unit",
- slot_key=clean_text(getattr(record, "slot_key", "")),
- state=clean_text(getattr(record, "state", "active")) or "active",
- turn_index=int(getattr(record, "turn_index", 0) or 0),
- metadata=metadata,
- )
- )
- seen_semantic_ids.add(semantic_id)
- added += 1
- if added >= max_add:
- break
- return dedupe_hits_by_memory_id(expanded)
-
-
-def hit_text_for_chain(hit: Any) -> str:
- metadata = dict(getattr(hit, "metadata", {}) or {})
- return " ".join(
- [
- clean_text(getattr(hit, "value", "")),
- clean_text(metadata.get("raw_text", "")),
- clean_text(metadata.get("source_turn_text", "")),
- clean_text(metadata.get("source_span", "")),
- ]
- )
-
-
-def dialogue_session_id_from_text(text: str) -> str:
- value = clean_text(text)
- if not value:
- return ""
- match = re.search(r"\bLongMemEval\s+session_id=([A-Za-z0-9_.:-]+)", value)
- if match:
- return match.group(1)
- match = re.search(r"\[([A-Za-z0-9_.:-]+)\s+turn=\d+\s+role=", value)
- return match.group(1) if match else ""
-
-
-def dialogue_chain_candidate_score(question: str, question_terms: set[str], value: str, metadata: Mapping[str, Any], distance: int) -> float:
- text = " ".join([value, clean_text(metadata.get("source_span", "")), clean_text(metadata.get("event_text", ""))])
- units = turn_units(text) or [text]
- base = max(score_evidence_unit(question_terms, question, unit) for unit in units)
- if base <= 0:
- return 0.0
- source_kind = clean_text(metadata.get("source_kind", ""))
- if source_kind.startswith("public_dialog_") and source_kind != "public_dialog_turn":
- base += 0.35
- if clean_text(metadata.get("event_id", "")):
- base += 0.2
- return round(max(0.0, base - (0.15 * max(0, distance - 1))), 6)
-
-
-def record_to_chain_hit(record: Any, *, score: float, seed_id: str, distance: int, session_id: str) -> Any:
- metadata = dict(getattr(record, "metadata", {}) or {})
- metadata.update(
- {
- "dialogue_chain_expansion": True,
- "dialogue_chain_seed_memory_id": seed_id,
- "dialogue_chain_distance": int(distance),
- "dialogue_chain_session_id": session_id,
- }
- )
- return SimpleNamespace(
- memory_id=clean_text(getattr(record, "memory_id", "")),
- category=clean_text(getattr(record, "category", "")),
- value=clean_text(getattr(record, "value", "")),
- relation=clean_text(getattr(record, "relation", "related_to")) or "related_to",
- anchors=list(getattr(record, "anchor_concepts", []) or []),
- score=float(score),
- source_kind=clean_text(getattr(record, "source_kind", "memory")) or "memory",
- slot_key=clean_text(getattr(record, "slot_key", "")),
- state=clean_text(getattr(record, "state", "active")) or "active",
- turn_index=int(getattr(record, "turn_index", 0) or 0),
- metadata=metadata,
- )
-
-
-def dedupe_hits_by_memory_id(hits: list[Any]) -> list[Any]:
- deduped: list[Any] = []
- seen: set[str] = set()
- for hit in hits:
- memory_id = clean_text(getattr(hit, "memory_id", ""))
- if memory_id and memory_id in seen:
- continue
- if memory_id:
- seen.add(memory_id)
- deduped.append(hit)
- return deduped
-
-
-def _load_answer_window_planner() -> tuple[Any, Any, Any, dict[str, float]] | None:
- mode = os.getenv("TMCRA_ANSWER_WINDOW_PLANNER_MODE", "").strip().lower()
- if mode in {"", "off", "false", "0", "disabled", "none"}:
- return None
- model_path = (
- os.getenv("TMCRA_ANSWER_WINDOW_PLANNER_MODEL_PATH", "").strip()
- or os.getenv("TMCRA_INJECTION_PLANNER_MODEL_PATH", "").strip()
- )
- if not model_path:
- return None
- cache_key = "|".join(
- [
- model_path,
- os.getenv("TMCRA_ANSWER_WINDOW_PLANNER_DEVICE", os.getenv("TMCRA_INJECTION_PLANNER_DEVICE", "cpu")),
- ]
- )
- cached = _ANSWER_WINDOW_PLANNER_CACHE.get(cache_key)
- if cached is not None:
- return cached
- try:
- from experiments.replacement import injection_planner as planner
- except Exception:
- return None
- torch_module = getattr(planner, "torch", None)
- if torch_module is None:
- return None
- try:
- device = torch_module.device(os.getenv("TMCRA_ANSWER_WINDOW_PLANNER_DEVICE", os.getenv("TMCRA_INJECTION_PLANNER_DEVICE", "cpu")) or "cpu")
- payload = torch_module.load(Path(model_path), map_location=device, weights_only=False)
- config = planner.InjectionPlannerConfig.from_dict(dict(payload.get("config", {}) or {}))
- model = planner.InjectionPlannerModel(config).to(device)
- model.load_state_dict(dict(payload.get("state_dict", {}) or {}), strict=False)
- model.eval()
- thresholds = {
- "selection_threshold": float(os.getenv("TMCRA_ANSWER_WINDOW_PLANNER_SELECTION_THRESHOLD", os.getenv("TMCRA_INJECTION_PLANNER_SELECTION_THRESHOLD", "0.3"))),
- "row_threshold": float(os.getenv("TMCRA_ANSWER_WINDOW_PLANNER_ROW_THRESHOLD", os.getenv("TMCRA_INJECTION_PLANNER_ROW_THRESHOLD", "0.3"))),
- "logic_threshold": float(os.getenv("TMCRA_ANSWER_WINDOW_PLANNER_LOGIC_THRESHOLD", os.getenv("TMCRA_INJECTION_PLANNER_LOGIC_THRESHOLD", "0.45"))),
- }
- except Exception:
- return None
- loaded = (planner, torch_module, model, thresholds)
- _ANSWER_WINDOW_PLANNER_CACHE[cache_key] = loaded
- return loaded
-
-
-def apply_answer_window_planner(question: str, evidence_windows: list[dict[str, Any]]) -> list[dict[str, Any]]:
- loaded = _load_answer_window_planner()
- if loaded is None or not evidence_windows:
- return evidence_windows
- planner, torch_module, model, thresholds = loaded
- evidence_windows = apply_answer_window_semantic_features(question, evidence_windows)
- max_score = max([float(item.get("score", 0.0) or 0.0) for item in evidence_windows] + [1.0])
- candidates = []
- for index, item in enumerate(evidence_windows):
- original_role = clean_text(item.get("evidence_role", ""))
- role_is_noise = original_role in {"noise", "negative_evidence"}
- semantic_similarity = max(
- 0.0,
- min(
- 1.0,
- float(item.get("answer_window_semantic_similarity", item.get("semantic_similarity", 0.0)) or 0.0),
- ),
- )
- candidates.append(
- {
- "id": f"window_{index}",
- "text": clean_text(item.get("text", "")),
- "layer": "event",
- "temporal_state": clean_text(item.get("temporal_state", "")) or "irrelevant",
- "logic_roles": ["noise"] if role_is_noise else ["evidence"],
- "retrieval_score": min(1.0, max(0.0, float(item.get("score", 0.0) or 0.0) / max_score)),
- "graph_score": min(1.0, max(0.0, float(item.get("planner_score", 0.0) or 0.0))),
- "tunnel_score": 1.0 if original_role == "bridge_context" else 0.0,
- "topic_similarity": 0.0,
- "semantic_similarity": semantic_similarity,
- "confidence": min(1.0, max(0.0, float(item.get("score", 0.0) or 0.0) / max_score)),
- "rank_score": round(1.0 / float(index + 1), 6),
- "branch_depth": 1,
- "evidence_role": original_role,
- }
- )
- try:
- row = {"id": "answer_window_plan", "query": question, "candidates": candidates, "gold": {}}
- dataset = planner.InjectionPlannerDataset([row], model.config)
- batch = planner.collate_injection_batch([dataset[0]])
- device = next(model.parameters()).device
- model_batch = {key: value.to(device) if hasattr(value, "to") else value for key, value in dict(batch).items()}
- with torch_module.no_grad():
- outputs = model(model_batch["features"], model_batch["valid_mask"])
- should_inject_score = float(torch_module.sigmoid(outputs["should_inject_logits"])[0].detach().cpu().item())
- selection_scores = torch_module.sigmoid(outputs["selection_logits"])[0].detach().cpu().tolist()
- temporal_indices = torch_module.argmax(outputs["temporal_logits"], dim=-1)[0].detach().cpu().tolist()
- role_indices = torch_module.argmax(outputs["evidence_role_logits"], dim=-1)[0].detach().cpu().tolist()
- logic_scores = torch_module.sigmoid(outputs["logic_logits"])[0].detach().cpu().tolist()
- except Exception:
- return evidence_windows
- row_allows = should_inject_score >= thresholds["row_threshold"]
- planned: list[dict[str, Any]] = []
- for index, item in enumerate(evidence_windows):
- next_item = dict(item)
- item_semantic_similarity = max(
- 0.0,
- min(
- 1.0,
- float(item.get("answer_window_semantic_similarity", item.get("semantic_similarity", 0.0)) or 0.0),
- ),
- )
- role = planner.EVIDENCE_ROLES[int(role_indices[index])]
- role_allows = role not in {"noise", "negative_evidence"}
- selected = bool(row_allows and role_allows and float(selection_scores[index]) >= thresholds["selection_threshold"])
- logic_roles = [
- logic_role
- for logic_role, score in zip(planner.LOGIC_ROLES, logic_scores[index])
- if float(score) >= thresholds["logic_threshold"]
- ]
- next_item.update(
- {
- "hit_evidence_role": clean_text(item.get("evidence_role", "")),
- "hit_temporal_state": clean_text(item.get("temporal_state", "")),
- "planner_selected": selected,
- "planner_score": round(float(selection_scores[index]), 6),
- "semantic_similarity": round(float(item_semantic_similarity), 6),
- "evidence_role": role,
- "temporal_state": planner.TEMPORAL_STATES[int(temporal_indices[index])],
- "window_planner_enabled": True,
- "window_planner_should_inject_score": round(float(should_inject_score), 6),
- "window_planner_logic_roles": logic_roles or ["evidence"],
- }
- )
- planned.append(next_item)
-
- def window_key(item: dict[str, Any]) -> tuple[bool, bool, float, float, int]:
- evidence_role = clean_text(item.get("evidence_role", ""))
- role_allows_evidence = evidence_role not in {"noise", "negative_evidence"}
- planner_score = float(item.get("planner_score", 0.0) or 0.0) if role_allows_evidence else 0.0
- return (
- bool(item.get("planner_selected", False)),
- role_allows_evidence,
- planner_score,
- float(item.get("score", 0.0) or 0.0),
- len(item.get("text", "")),
- )
-
- planned.sort(key=window_key, reverse=True)
- return planned
-
-
-def apply_llm_evidence_selector(question: str, evidence_windows: list[dict[str, Any]]) -> list[dict[str, Any]]:
- # TMCRA_LAYER_TAG: answer-side LLM evidence selector; post-retrieval rerank only, not recall.
- mode = os.getenv("TMCRA_LLM_EVIDENCE_SELECTOR_MODE", "on").strip().lower()
- if mode in {"", "off", "false", "0", "disabled", "none"} or not evidence_windows:
- return evidence_windows
- try:
- max_candidates = max(1, int(os.getenv("TMCRA_LLM_EVIDENCE_SELECTOR_MAX_CANDIDATES", "12") or 12))
- max_chars = max(200, int(os.getenv("TMCRA_LLM_EVIDENCE_SELECTOR_CHARS", "900") or 900))
- except (TypeError, ValueError):
- max_candidates, max_chars = 12, 900
- candidates = list(evidence_windows[:max_candidates])
- candidate_lines = []
- for index, item in enumerate(candidates, start=1):
- memory_id = clean_text(item.get("memory_id", ""))
- text = truncate(item.get("text", ""), max_chars)
- if not text:
- continue
- header = f"[{index}]"
- if memory_id:
- header += f" memory_id={memory_id}"
- candidate_lines.append(f"{header}\n{text}")
- if not candidate_lines:
- return evidence_windows
- messages = [
- {
- "role": "system",
- "content": (
- "You are a semantic evidence selector for a long-memory runtime. "
- "Your job is only to select evidence windows that could help answer the current question. "
- "Use semantic relevance and discourse continuity, not keyword lists or domain-specific rules. "
- "Select every distinct useful clue, including indirect clues, continuation turns, and separate pieces that must be combined. "
- "If two windows repeat the same clue, keep the clearest one. "
- "If two windows contain different clues, keep both even if one looks weaker or appears later in the list. "
- "Do not answer the question. "
- "Return strict JSON only with keys: selected_indices, rejected_indices. "
- "selected_indices must be 1-based integers from the candidate list."
- ),
- },
- {
- "role": "user",
- "content": f"Question:\n{question}\n\nCandidate evidence windows:\n" + "\n\n".join(candidate_lines),
- },
- ]
- try:
- raw = chat_completion(
- os.getenv("TMCRA_EVIDENCE_SELECTOR_BASE_URL", os.getenv("GEMMA_BASE_URL", "http://127.0.0.1:18002/v1")),
- os.getenv("TMCRA_EVIDENCE_SELECTOR_MODEL", os.getenv("GEMMA_MODEL", os.getenv("TMCRA_GEMMA_MODEL_NAME", "gemma-4-e4b-it"))),
- messages,
- max_tokens=160,
- temperature=0.0,
- )
- except Exception:
- return evidence_windows
- payload = parse_json_object(raw)
- raw_indices = payload.get("selected_indices", []) if payload else []
- if not isinstance(raw_indices, list):
- return evidence_windows
- selected_order: list[int] = []
- for value in iter_selector_indices(raw_indices):
- try:
- index = int(value) - 1
- except (TypeError, ValueError):
- continue
- if 0 <= index < len(candidates) and index not in selected_order:
- selected_order.append(index)
- if not selected_order:
- return evidence_windows
- keep_model_selected = os.getenv("TMCRA_LLM_EVIDENCE_SELECTOR_KEEP_MODEL_SELECTED", "1").strip().lower() not in {
- "0",
- "false",
- "off",
- "no",
- }
- model_selected_order: list[int] = []
- if keep_model_selected:
- for index, item in enumerate(candidates):
- if bool(item.get("planner_selected", False)) or bool(item.get("answer_plan_selected", False)):
- model_selected_order.append(index)
- selected_order = list(dict.fromkeys([*selected_order, *model_selected_order]))
- selected_set = set(selected_order)
- selected_rank = {index: rank for rank, index in enumerate(selected_order)}
- planned: list[tuple[int, int, dict[str, Any]]] = []
- for index, item in enumerate(evidence_windows):
- next_item = dict(item)
- llm_selected = index in selected_set
- next_item["llm_evidence_selected"] = llm_selected
- next_item["llm_evidence_rank"] = selected_rank.get(index, 9999)
- next_item["llm_evidence_selector_enabled"] = True
- next_item["llm_evidence_model_kept"] = bool(index in model_selected_order)
- group = 0 if llm_selected else 1
- rank = selected_rank.get(index, index)
- planned.append((group, rank, next_item))
- planned.sort(key=lambda row: (row[0], row[1]))
- return diversify_evidence_windows([item for _, _, item in planned])
-
-
-def apply_evidence_unit_planner(question: str, evidence_windows: list[dict[str, Any]]) -> list[dict[str, Any]]:
- # TMCRA_LAYER_TAG: answer-side LLM evidence-unit planner; normalizes retrieved windows after recall.
- mode = os.getenv("TMCRA_EVIDENCE_UNIT_PLANNER_MODE", "on").strip().lower()
- if mode in {"", "off", "false", "0", "disabled", "none"} or not evidence_windows:
- return evidence_windows
- if evidence_unit_plan_from_windows(evidence_windows):
- return evidence_windows
- plan = build_evidence_unit_plan(question, evidence_windows)
- if not plan:
- return evidence_windows
- selected_ranks = evidence_unit_selected_source_ranks(plan)
- reorder_mode = os.getenv("TMCRA_EVIDENCE_UNIT_PLANNER_REORDER", "0").strip().lower()
- reorder_enabled = reorder_mode not in {"", "0", "false", "off", "no", "disabled", "none"}
- planned: list[tuple[int, int, int, dict[str, Any]]] = []
- for index, item in enumerate(evidence_windows):
- next_item = dict(item)
- if index == 0:
- next_item["evidence_unit_plan"] = plan
- source_index = index + 1
- unit_rank = selected_ranks.get(source_index)
- next_item["evidence_unit_planner_enabled"] = True
- next_item["evidence_unit_planner_selected"] = unit_rank is not None
- next_item["evidence_unit_planner_rank"] = unit_rank if unit_rank is not None else 9999
- group = 0 if unit_rank is not None else 1
- rank = unit_rank if unit_rank is not None else index
- planned.append((group, rank, index, next_item))
- if not reorder_enabled:
- return [item for _, _, _, item in sorted(planned, key=lambda row: row[2])]
- planned.sort(key=lambda row: (row[0], row[1], row[2]))
- return diversify_evidence_windows([item for _, _, _, item in planned])
-
-
-def build_evidence_unit_plan(question: str, evidence_windows: list[dict[str, Any]]) -> dict[str, Any]:
- try:
- max_candidates = max(1, int(os.getenv("TMCRA_EVIDENCE_UNIT_PLANNER_MAX_CANDIDATES", "10") or 10))
- max_chars = max(240, int(os.getenv("TMCRA_EVIDENCE_UNIT_PLANNER_CHARS", "1100") or 1100))
- max_tokens = max(280, int(os.getenv("TMCRA_EVIDENCE_UNIT_PLANNER_MAX_TOKENS", "760") or 760))
- except (TypeError, ValueError):
- max_candidates, max_chars, max_tokens = 10, 1100, 760
- candidate_lines = []
- for index, item in enumerate(evidence_windows[:max_candidates], start=1):
- text = truncate(item.get("text", ""), max_chars)
- if not text:
- continue
- memory_id = clean_text(item.get("memory_id", ""))
- role = clean_text(item.get("evidence_role", ""))
- temporal_state = clean_text(item.get("temporal_state", ""))
- label_bits = []
- if memory_id:
- label_bits.append(f"memory_id={memory_id}")
- if role:
- label_bits.append(f"role={role}")
- if temporal_state:
- label_bits.append(f"time={temporal_state}")
- label = " ".join(label_bits)
- candidate_lines.append(f"[{index}] {label}\n{text}".strip())
- if not candidate_lines:
- return {}
- messages = [
- {
- "role": "system",
- "content": (
- "You are a model-based evidence-unit planner for a long-memory runtime. "
- "Do not answer from prior knowledge. Do not use domain-specific keyword rules. "
- "Read the current question and the retrieved evidence windows, then normalize the evidence into answer units that a final answer model can reason over. "
- "Your job is to expose the operation needed by the question: direct fact lookup, distinct-item counting, temporal difference, temporal disambiguation, current/latest value selection, negative/absence judgment, preference synthesis, or multi-evidence synthesis. "
- "For count questions, infer the answer unit from the question and create one unit per distinct evidence-backed candidate. "
- "When the question requests multiple actions joined by and/or, keep separate units for separate action-target instances, especially original-versus-replacement or return-versus-pick-up instances. "
- "Do not collapse a stated pending need, intention, or obligation into mere context just because the same sentence later mentions an exchange, replacement, reschedule, or follow-up action. "
- "Treat the original action and the replacement/follow-up action as separate units unless the evidence explicitly says the original action was completed, cancelled, or no longer needed. "
- "Do not mark a clue as a distractor solely because it appears in an assistant turn; assistant turns can preserve user-specific obligations, summaries, answers, and confirmations when they refer to the surrounding dialogue. "
- "If an assistant turn says the user needs to do an action, or summarizes a user obligation, keep it as a candidate evidence unit unless contradicted by user evidence. "
- "For temporal questions, identify the event date or relative time anchor, the query-time anchor if available, and which event should be compared or excluded. "
- "Use session dates and relative-time phrases from the evidence window text when present; connect nearby event descriptions to the session date when the dialogue makes that local anchor clear. "
- "For month-level questions, if a past-tense personal event appears inside a dated session and no older or future date is stated, treat it as compatible with that session month; mark the time_anchor as inferred from session date rather than excluding it. "
- "For negative or absence questions, first identify the relevant event/time bucket, then state whether the requested attribute is present, absent, or contradicted in that event. "
- "Keep weak but potentially useful clues; mark distractors separately instead of deleting them. "
- "Return strict JSON only with these keys: question_task, answer_unit, required_operation, evidence_units, plan_steps, candidate_answer, uncertainty. "
- "evidence_units must be a list of objects with keys: source_index, role, unit_key, value, action, target, instance, time_anchor, polarity, relation_to_question. "
- "Allowed role values: answer_unit, positive_evidence, temporal_anchor, current_value, old_value, constraint, negative_evidence, distractor, context. "
- "source_index must be the 1-based evidence window index. candidate_answer may be empty when computation should be left to the answer model."
- ),
- },
- {
- "role": "user",
- "content": f"Question:\n{question}\n\nRetrieved evidence windows:\n" + "\n\n".join(candidate_lines),
- },
- ]
- base_url, model, api_key = evidence_unit_planner_config()
- try:
- raw = chat_completion(
- base_url,
- model,
- messages,
- max_tokens=max_tokens,
- temperature=0.0,
- api_key=api_key,
- )
- except Exception as exc:
- return {
- "enabled": False,
- "error": f"{exc.__class__.__name__}: {str(exc)[:240]}",
- }
- payload = parse_json_object(raw)
- if not payload:
- return {
- "enabled": False,
- "error": "planner_returned_non_json",
- "raw": truncate(raw, 500),
- }
- return sanitize_evidence_unit_plan(payload, max_source_index=len(candidate_lines))
-
-
-def sanitize_evidence_unit_plan(payload: Mapping[str, Any], *, max_source_index: int) -> dict[str, Any]:
- allowed_roles = {
- "answer_unit",
- "positive_evidence",
- "temporal_anchor",
- "current_value",
- "old_value",
- "constraint",
- "negative_evidence",
- "distractor",
- "context",
- }
- plan: dict[str, Any] = {
- "enabled": True,
- "question_task": truncate(payload.get("question_task", ""), 80),
- "answer_unit": truncate(payload.get("answer_unit", ""), 120),
- "required_operation": truncate(payload.get("required_operation", ""), 180),
- "evidence_units": [],
- "plan_steps": [],
- "candidate_answer": truncate(payload.get("candidate_answer", ""), 160),
- "uncertainty": truncate(payload.get("uncertainty", ""), 180),
- }
- raw_units = payload.get("evidence_units", [])
- if isinstance(raw_units, list):
- for raw_unit in raw_units[:24]:
- if not isinstance(raw_unit, Mapping):
- continue
- try:
- source_index = int(raw_unit.get("source_index", 0) or 0)
- except (TypeError, ValueError):
- source_index = 0
- if source_index < 1 or source_index > max_source_index:
- continue
- role = clean_text(raw_unit.get("role", "context")).lower()
- if role not in allowed_roles:
- role = "context"
- plan["evidence_units"].append(
- {
- "source_index": source_index,
- "role": role,
- "unit_key": truncate(raw_unit.get("unit_key", ""), 100),
- "value": truncate(raw_unit.get("value", ""), 220),
- "action": truncate(raw_unit.get("action", ""), 80),
- "target": truncate(raw_unit.get("target", ""), 100),
- "instance": truncate(raw_unit.get("instance", ""), 100),
- "time_anchor": truncate(raw_unit.get("time_anchor", ""), 100),
- "polarity": truncate(raw_unit.get("polarity", ""), 60),
- "relation_to_question": truncate(raw_unit.get("relation_to_question", ""), 180),
- }
- )
- raw_steps = payload.get("plan_steps", [])
- if isinstance(raw_steps, list):
- plan["plan_steps"] = [truncate(item, 180) for item in raw_steps[:8] if clean_text(item)]
- return plan
-
-
-def evidence_unit_selected_source_ranks(plan: Mapping[str, Any]) -> dict[int, int]:
- selected: dict[int, int] = {}
- ignored_roles = {"distractor"}
- for unit in list(plan.get("evidence_units") or []):
- if not isinstance(unit, Mapping):
- continue
- role = clean_text(unit.get("role", "")).lower()
- if role in ignored_roles:
- continue
- try:
- source_index = int(unit.get("source_index", 0) or 0)
- except (TypeError, ValueError):
- continue
- if source_index <= 0 or source_index in selected:
- continue
- selected[source_index] = len(selected)
- return selected
-
-
-def evidence_unit_plan_from_windows(evidence_windows: list[dict[str, Any]]) -> dict[str, Any]:
- for item in evidence_windows:
- plan = item.get("evidence_unit_plan")
- if isinstance(plan, dict) and plan:
- return plan
- return {}
-
-
-def _load_unified_operation_planner() -> tuple[Any, Any, Any] | None:
- mode = os.getenv("TMCRA_UNIFIED_OPERATION_PLANNER_MODE", "").strip().lower()
- if mode in {"", "off", "false", "0", "disabled", "none"}:
- return None
- model_path = os.getenv("TMCRA_UNIFIED_OPERATION_PLANNER_MODEL_PATH", "").strip()
- if not model_path:
- return None
- cache_key = "|".join([model_path, os.getenv("TMCRA_UNIFIED_OPERATION_PLANNER_DEVICE", "cpu")])
- cached = _UNIFIED_OPERATION_PLANNER_CACHE.get(cache_key)
- if cached is not None:
- return cached
- try:
- from experiments.replacement import unified_operation_planner as planner
- except Exception:
- return None
- torch_module = getattr(planner, "torch", None)
- if torch_module is None:
- return None
- try:
- device = torch_module.device(os.getenv("TMCRA_UNIFIED_OPERATION_PLANNER_DEVICE", "cpu") or "cpu")
- payload = torch_module.load(Path(model_path), map_location=device, weights_only=False)
- config = planner.UnifiedPlannerConfig.from_dict(dict(payload.get("config", {}) or {}))
- model = planner.UnifiedOperationPlannerModel(config).to(device)
- state = dict(payload.get("model_state") or payload.get("state_dict") or {})
- model.load_state_dict(state, strict=False)
- model.eval()
- except Exception:
- return None
- loaded = (planner, torch_module, model)
- _UNIFIED_OPERATION_PLANNER_CACHE[cache_key] = loaded
- return loaded
-
-
-def apply_unified_operation_planner(question: str, evidence_windows: list[dict[str, Any]]) -> list[dict[str, Any]]:
- loaded = _load_unified_operation_planner()
- if loaded is None or not evidence_windows:
- return evidence_windows
- planner, torch_module, model = loaded
- try:
- max_windows = max(3, int(os.getenv("TMCRA_UNIFIED_OPERATION_PLANNER_MAX_WINDOWS", "16") or 16))
- answer_threshold = float(os.getenv("TMCRA_UNIFIED_OPERATION_PLANNER_ANSWER_THRESHOLD", "0.52"))
- except (TypeError, ValueError):
- max_windows, answer_threshold = 16, 0.52
- scoped = list(evidence_windows[:max_windows])
- all_text = " ".join(clean_text(item.get("text", "")) for item in scoped).lower()
- units = []
- for index, item in enumerate(scoped, start=1):
- text = clean_text(item.get("text", ""))
- memory_id = clean_text(item.get("memory_id", "")) or f"window_{index}"
- units.append(
- {
- "unit_id": f"u{index}",
- "record_id": memory_id,
- "session_id": memory_id.split(".turn_", 1)[0],
- "turn_index": index,
- "speaker": "unknown",
- "text": text,
- "timestamp": "",
- "topic_bucket": clean_text(item.get("topic_bucket", "")),
- "node_features": {
- "retrieval_score": float(item.get("score", 0.0) or 0.0) / 10.0,
- "graph_score": float(item.get("planner_score", 0.0) or item.get("answer_plan_score", 0.0) or 0.0),
- "tunnel_score": float(item.get("path_score", 0.0) or 0.0),
- },
- "graph_neighbors": [],
- }
- )
- row = {
- "id": "runtime_unified_operation_planner",
- "query": question,
- "retrieval_metadata": {
- "candidate_count": len(scoped),
- "session_span": len({clean_text(item.get("memory_id", "")).split(".turn_", 1)[0] for item in scoped}),
- "topic_span": len({clean_text(item.get("topic_bucket", "")) for item in scoped if clean_text(item.get("topic_bucket", ""))}),
- "has_temporal_anchor": bool(re.search(r"\b(before|after|latest|current|previous|ago|week|month|year|date)\b", all_text)),
- "has_numeric_units": bool(re.search(r"\b\d[\d,.]*\b|\$\s?\d|%", all_text)),
- "has_profile_units": bool(re.search(r"\b(prefer|like|avoid|favorite|usually|always|never)\b", all_text)),
- },
- "memory_units": units,
- "gold": {},
- }
- try:
- dataset = planner.UnifiedOperationPlannerDataset([row], model.config)
- batch = planner.collate_unified_planner_batch([dataset[0]])
- device = next(model.parameters()).device
- model_batch = {key: value.to(device) if hasattr(value, "to") else value for key, value in dict(batch).items()}
- with torch_module.no_grad():
- outputs = model(model_batch["features"], model_batch["valid_mask"])
- relevance_scores = torch_module.sigmoid(outputs["relevance_logits"])[0].detach().cpu().tolist()
- answer_scores = torch_module.sigmoid(outputs["answer_logits"])[0].detach().cpu().tolist()
- temporal_scores = torch_module.sigmoid(outputs["temporal_logits"])[0].detach().cpu().tolist()
- aggregation_scores = torch_module.sigmoid(outputs["aggregation_logits"])[0].detach().cpu().tolist()
- current_value_scores = torch_module.sigmoid(outputs["current_value_logits"])[0].detach().cpu().tolist()
- operation_scores = torch_module.sigmoid(outputs["operation_required_logits"])[0].detach().cpu().tolist()
- operation_family_index = int(torch_module.argmax(outputs["operation_family_logits"], dim=-1)[0].detach().cpu().item())
- except Exception:
- return evidence_windows
- operation_family = planner.OPERATION_FAMILIES[operation_family_index]
- ranked: list[tuple[int, float, int, dict[str, Any]]] = []
- for index, item in enumerate(evidence_windows):
- next_item = dict(item)
- if index < len(scoped):
- relevance_score = float(relevance_scores[index])
- answer_score = float(answer_scores[index])
- temporal_score = float(temporal_scores[index])
- aggregation_score = float(aggregation_scores[index])
- current_value_score = float(current_value_scores[index])
- unified_score = max(answer_score, temporal_score, aggregation_score, current_value_score, relevance_score * 0.8)
- next_item["unified_operation_planner"] = True
- next_item["unified_operation_family"] = operation_family
- next_item["unified_operation_required_scores"] = {
- "temporal": round(float(operation_scores[0]), 4),
- "aggregation": round(float(operation_scores[1]), 4),
- "profile": round(float(operation_scores[2]), 4),
- "current_value": round(float(operation_scores[3]), 4),
- "multi_hop": round(float(operation_scores[4]), 4),
- }
- next_item["unified_scores"] = {
- "relevance": round(relevance_score, 4),
- "answer": round(answer_score, 4),
- "temporal": round(temporal_score, 4),
- "aggregation": round(aggregation_score, 4),
- "current_value": round(current_value_score, 4),
- "combined": round(unified_score, 4),
- }
- next_item["unified_selected"] = bool(answer_score >= answer_threshold or unified_score >= max(answer_threshold, 0.58))
- old_planner_score = float(next_item.get("planner_score", 0.0) or next_item.get("answer_plan_score", 0.0) or 0.0)
- next_item["planner_score"] = max(old_planner_score, unified_score)
- group = 0 if next_item["unified_selected"] else 1
- ranked.append((group, -unified_score, index, next_item))
- else:
- ranked.append((2, 0.0, index, next_item))
- ranked.sort(key=lambda row: (row[0], row[1], row[2]))
- return [item for _, _, _, item in ranked]
-
-
-def format_evidence_unit_plan_for_prompt(plan: Mapping[str, Any]) -> str:
- if not plan or not bool(plan.get("enabled", False)):
- return ""
- compact = {
- "question_task": plan.get("question_task", ""),
- "answer_unit": plan.get("answer_unit", ""),
- "required_operation": plan.get("required_operation", ""),
- "evidence_units": list(plan.get("evidence_units") or [])[:16],
- "plan_steps": list(plan.get("plan_steps") or [])[:6],
- "candidate_answer": plan.get("candidate_answer", ""),
- "uncertainty": plan.get("uncertainty", ""),
- }
- return json.dumps(compact, ensure_ascii=False, separators=(",", ":"))
-
-
-def focused_evidence_unit_lines(
- question: str,
- evidence_windows: list[dict[str, Any]],
- *,
- max_items: int | None = None,
- max_chars: int | None = None,
-) -> list[str]:
- mode = os.getenv("TMCRA_FOCUSED_EVIDENCE_UNIT_MODE", "").strip().lower()
- if mode in {"", "off", "0", "false", "disabled", "none"}:
- return []
- try:
- item_limit = max(1, int(os.getenv("TMCRA_FOCUSED_EVIDENCE_UNIT_LIMIT", "18") if max_items is None else max_items))
- char_limit = max(120, int(os.getenv("TMCRA_FOCUSED_EVIDENCE_UNIT_CHARS", "420") if max_chars is None else max_chars))
- per_window = max(1, int(os.getenv("TMCRA_FOCUSED_EVIDENCE_UNITS_PER_WINDOW", "2") or 2))
- except (TypeError, ValueError):
- item_limit, char_limit, per_window = 18, 420, 2
- question_terms = evidence_query_terms(question)
- lines: list[str] = []
- seen: set[str] = set()
- for window_index, item in enumerate(evidence_windows, start=1):
- text = clean_text(item.get("text", ""))
- if not text:
- continue
- units = turn_units(text) or [text]
- scored_units: list[tuple[float, int, str]] = []
- for unit_index, unit in enumerate(units):
- unit_text = clean_text(unit)
- if not unit_text:
- continue
- score = score_evidence_unit(question_terms, question, unit_text)
- if bool(item.get("planner_selected", False)) or bool(item.get("evidence_unit_planner_selected", False)):
- score += 0.4
- if bool(item.get("semantic_coverage_expansion", False)):
- score += 0.25
- if score <= 0 and len(units) > 1:
- continue
- scored_units.append((float(score), unit_index, unit_text))
- if not scored_units:
- scored_units = [(0.0, 0, text)]
- scored_units.sort(key=lambda row: (-row[0], row[1]))
- added_for_window = 0
- for _, unit_index, unit_text in scored_units:
- snippet = centered_excerpt(unit_text, question_terms, max_chars=char_limit)
- key = re.sub(r"\W+", " ", snippet.lower()).strip()
- if not key or key in seen:
- continue
- seen.add(key)
- memory_id = clean_text(item.get("memory_id", ""))
- source = f"source={window_index}"
- if memory_id:
- source += f" memory_id={memory_id}"
- lines.append(f"{len(lines) + 1}. [{source} unit={unit_index}] {snippet}")
- added_for_window += 1
- if len(lines) >= item_limit or added_for_window >= per_window:
- break
- if len(lines) >= item_limit:
- break
- return lines
-
-
-def diversify_evidence_windows(evidence_windows: list[dict[str, Any]]) -> list[dict[str, Any]]:
- mode = os.getenv("TMCRA_ANSWER_EVIDENCE_DIVERSIFY_MEMORY_IDS", "1").strip().lower()
- if mode in {"", "0", "false", "off", "no"} or len(evidence_windows) <= 1:
- return evidence_windows
- first_pass: list[dict[str, Any]] = []
- overflow: list[dict[str, Any]] = []
- seen: set[str] = set()
- for item in evidence_windows:
- memory_id = clean_text(item.get("memory_id", ""))
- if not memory_id:
- first_pass.append(item)
- continue
- if memory_id in seen:
- overflow.append(item)
- continue
- seen.add(memory_id)
- first_pass.append(item)
- return [*first_pass, *overflow]
-
-
-def final_answer_channel_intent(question: str) -> dict[str, bool]:
- terms = evidence_query_terms(question)
- question_l = question.lower()
- aggregation = bool(
- terms
- & {
- "amount",
- "count",
- "different",
- "how",
- "many",
- "minimum",
- "number",
- "percent",
- "percentage",
- "sum",
- "total",
- }
- ) or bool(re.search(r"\b(?:how many|how much|total|sum|minimum|maximum|combined|different)\b", question_l))
- temporal = bool(
- re.search(
- r"\b(?:before|after|first|last|earlier|later|weeks?|days?|months?|years?|when|date|how long)\b",
- question_l,
- )
- )
- assistant_detail = assistant_memory_query(question)
- return {
- "aggregation": aggregation,
- "temporal": temporal,
- "assistant_detail": assistant_detail,
- }
-
-
-def _json_object_from_text(value: str) -> dict[str, Any]:
- text = clean_text(value)
- if not text:
- return {}
- try:
- payload = json.loads(text)
- return payload if isinstance(payload, dict) else {}
- except json.JSONDecodeError:
- pass
- match = re.search(r"\{.*\}", text, flags=re.DOTALL)
- if not match:
- return {}
- try:
- payload = json.loads(match.group(0))
- except json.JSONDecodeError:
- return {}
- return payload if isinstance(payload, dict) else {}
-
-
-def build_query_graph(question: str, question_date: str = "") -> dict[str, Any]:
- mode = os.getenv("TMCRA_QUERY_GRAPH_BUILDER_MODE", "").strip().lower()
- if mode in {"", "0", "false", "off", "no", "disabled", "none"}:
- return {"enabled": False, "mode": mode or "off"}
- base_url, model, api_key = query_graph_llm_config()
- try:
- max_tokens = max(240, int(os.getenv("TMCRA_QUERY_GRAPH_MAX_TOKENS", "700") or 700))
- except (TypeError, ValueError):
- max_tokens = 700
- system = (
- "You convert a user question into a compact retrieval query graph for a long-memory system. "
- "Return JSON only. Do not answer the question. Build a general graph, not benchmark-specific logic."
- )
- user = {
- "question": question,
- "question_date": question_date,
- "schema": {
- "task_intent": "direct_fact | count | sum | compare | temporal | preference | multi_evidence | unknown",
- "operation": "none | count_distinct | sum_numeric | compare_values | select_latest | select_current | timeline_order | infer_preference",
- "required_units": [
- {
- "unit_key": "short stable key",
- "role": "main_fact | coverage_fact | old_value | current_value | temporal_anchor | constraint | negative",
- "target_entity": "entity or topic to retrieve",
- "attribute": "amount/count/status/action/time/preference/etc",
- "expected_value_type": "money | count | date | duration | item | state | preference | text",
- "must_cover": True,
- }
- ],
- "tunnel_needs": ["same_topic", "cross_session", "same_entity", "temporal_chain", "profile_bridge"],
- "query_terms": ["terms that should retrieve memory units"],
- "negative_terms": ["terms that should not dominate retrieval"],
- },
- }
- try:
- raw = chat_completion(
- base_url,
- model,
- [
- {"role": "system", "content": system},
- {"role": "user", "content": json.dumps(user, ensure_ascii=False)},
- ],
- max_tokens=max_tokens,
- temperature=0.0,
- api_key=api_key,
- )
- graph = _json_object_from_text(raw)
- if not graph:
- return {"enabled": True, "mode": mode, "error": "empty_or_invalid_json", "raw": truncate(raw, 600)}
- graph["enabled"] = True
- graph["mode"] = mode
- graph["model"] = model
- return graph
- except Exception as exc:
- return {"enabled": True, "mode": mode, "error": f"{exc.__class__.__name__}: {str(exc)[:240]}"}
-
-
-def query_graph_retrieval_text(question: str, question_date: str, query_graph: dict[str, Any]) -> str:
- runtime_question = question
- if question_date:
- runtime_question = f"{question}\nQuestion date: {question_date}"
- use_text_mode = os.getenv("TMCRA_QUERY_GRAPH_RETRIEVAL_TEXT_MODE", "").strip().lower() in {"1", "true", "on", "yes"}
- if not use_text_mode or not bool(query_graph.get("enabled", False)) or query_graph.get("error"):
- return runtime_question
- graph = {key: value for key, value in query_graph.items() if key not in {"enabled", "mode", "model", "raw"}}
- graph_text = truncate(json.dumps(graph, ensure_ascii=False), int(os.getenv("TMCRA_QUERY_GRAPH_RETRIEVAL_CHARS", "2200") or 2200))
- return f"{runtime_question}\n\nQuestion retrieval graph JSON:\n{graph_text}"
-
-
-def query_graph_sidecar_queries(question: str, question_date: str, query_graph: dict[str, Any]) -> list[str]:
- if not bool(query_graph.get("enabled", False)) or query_graph.get("error"):
- return []
- mode = os.getenv("TMCRA_QUERY_GRAPH_SIDECAR_RETRIEVAL_MODE", "on").strip().lower()
- if mode in {"", "0", "false", "off", "no", "disabled", "none"}:
- return []
- try:
- max_queries = max(0, int(os.getenv("TMCRA_QUERY_GRAPH_SIDECAR_MAX_QUERIES", "6") or 6))
- except (TypeError, ValueError):
- max_queries = 6
- if max_queries <= 0:
- return []
- query_terms = [clean_text(item) for item in list(query_graph.get("query_terms") or []) if clean_text(item)]
- required_units = list(query_graph.get("required_units") or [])
- queries: list[str] = []
- seen: set[str] = set()
-
- def add(value: str) -> None:
- text = clean_text(value)
- if not text:
- return
- if question_date and "question date:" not in text.lower():
- text = f"{text}\nQuestion date: {question_date}"
- key = text.lower()
- if key in seen or len(queries) >= max_queries:
- return
- seen.add(key)
- queries.append(text)
-
- for unit in required_units:
- if not isinstance(unit, dict):
- continue
- role = clean_text(unit.get("role", ""))
- target = clean_text(unit.get("target_entity", ""))
- attribute = clean_text(unit.get("attribute", ""))
- value_type = clean_text(unit.get("expected_value_type", ""))
- unit_key = clean_text(unit.get("unit_key", ""))
- add(
- " ".join(
- part
- for part in [
- question,
- f"required evidence role {role}",
- f"target {target}",
- f"attribute {attribute}",
- f"value type {value_type}",
- f"unit {unit_key}",
- ]
- if part
- )
- )
- if query_terms and len(queries) < max_queries:
- add(f"{question} {' '.join(query_terms[:24])}")
- return queries[:max_queries]
-
-
-def merge_memory_hits(primary_hits: list[Any], extra_hits: list[Any]) -> list[Any]:
- merged: list[Any] = []
- seen: set[str] = set()
- for hit in [*primary_hits, *extra_hits]:
- memory_id = clean_text(getattr(hit, "memory_id", ""))
- key = memory_id or clean_text(getattr(hit, "text", ""))[:240].lower()
- if key and key in seen:
- continue
- if key:
- seen.add(key)
- merged.append(hit)
- return merged
-
-
-def apply_query_graph_sidecar_retrieval(
- adapter: Any,
- query_graph: dict[str, Any],
- question: str,
- question_date: str,
- hits: list[Any],
- *,
- top_k: int,
-) -> tuple[list[Any], dict[str, Any]]:
- queries = query_graph_sidecar_queries(question, question_date, query_graph)
- if not queries:
- return hits, {"enabled": False, "reason": "no_queries"}
- try:
- per_query_top_k = max(1, int(os.getenv("TMCRA_QUERY_GRAPH_SIDECAR_TOP_K", "4") or 4))
- except (TypeError, ValueError):
- per_query_top_k = 4
- extra_hits: list[Any] = []
- errors: list[str] = []
- for query in queries:
- try:
- retrieval = adapter.retrieve(query, top_k=min(max(1, top_k), per_query_top_k))
- extra_hits.extend(list(getattr(retrieval, "hits", []) or []))
- except Exception as exc:
- errors.append(f"{exc.__class__.__name__}:{str(exc)[:160]}")
- merged = merge_memory_hits(hits, extra_hits)
- return merged, {
- "enabled": True,
- "query_count": len(queries),
- "extra_hit_count": len(extra_hits),
- "merged_hit_count": len(merged),
- "errors": errors[:3],
- "queries": [truncate(item, 260) for item in queries[:6]],
- }
-
-
-def final_answer_surface_windows(question: str, evidence_windows: list[dict[str, Any]]) -> list[dict[str, Any]]:
- mode = os.getenv("TMCRA_FINAL_ANSWER_SURFACE_CHANNEL_MODE", "on").strip().lower()
- if mode in {"", "0", "false", "off", "no", "disabled", "none"} or not evidence_windows:
- return evidence_windows
- intent = final_answer_channel_intent(question)
- try:
- final_limit = max(4, int(os.getenv("TMCRA_ANSWER_EVIDENCE_WINDOW_LIMIT", "8") or 8))
- if intent["aggregation"]:
- final_limit = max(final_limit, int(os.getenv("TMCRA_ANSWER_EVIDENCE_WINDOW_LIMIT_AGG", "12") or 12))
- except (TypeError, ValueError):
- final_limit = 12 if final_answer_channel_intent(question)["aggregation"] else 8
- if intent["aggregation"]:
- main_hard_limit = max(4, int(os.getenv("TMCRA_FINAL_MAIN_HARD_LIMIT_AGG", "6") or 6))
- coverage_hard_limit = max(1, int(os.getenv("TMCRA_FINAL_COVERAGE_HARD_LIMIT_AGG", "6") or 6))
- elif intent["temporal"] or intent["assistant_detail"]:
- main_hard_limit = max(4, int(os.getenv("TMCRA_FINAL_MAIN_HARD_LIMIT_DETAIL", "7") or 7))
- coverage_hard_limit = max(0, int(os.getenv("TMCRA_FINAL_COVERAGE_HARD_LIMIT_DETAIL", "1") or 1))
- else:
- main_hard_limit = max(4, int(os.getenv("TMCRA_FINAL_MAIN_HARD_LIMIT_DEFAULT", "6") or 6))
- coverage_hard_limit = max(0, int(os.getenv("TMCRA_FINAL_COVERAGE_HARD_LIMIT_DEFAULT", "2") or 2))
- coverage_hard_limit = min(coverage_hard_limit, max(0, final_limit - 1))
- main_hard_limit = min(main_hard_limit, final_limit - coverage_hard_limit)
-
- def is_coverage(item: dict[str, Any]) -> bool:
- return bool(item.get("multi_unit_chain_slot", False)) or bool(item.get("unit_coverage_pack", False))
-
- def is_assistant_candidate(item: dict[str, Any]) -> bool:
- return bool(item.get("assistant_candidate_side_channel", False)) or clean_text(item.get("evidence_role", "")).lower() == "assistant_answer_candidate"
-
- def is_assistant(item: dict[str, Any]) -> bool:
- if is_assistant_candidate(item):
- return False
- return bool(item.get("assistant_origin_evidence", False)) or clean_text(item.get("evidence_role", "")).lower() == "assistant_origin_detail"
-
- def key(item: dict[str, Any]) -> tuple[int, int, float, float, int]:
- role = clean_text(item.get("evidence_role", "")).lower()
- role_allows = int(role not in {"noise", "negative_evidence"})
- selected = int(bool(item.get("planner_selected", False)) or bool(item.get("evidence_unit_planner_selected", False)))
- planner_score = max(float(item.get("planner_score", 0.0) or 0.0), 0.0)
- return (
- selected,
- role_allows,
- planner_score,
- float(item.get("score", 0.0) or 0.0),
- len(clean_text(item.get("text", ""))),
- )
-
- assistant_candidates = [item for item in evidence_windows if is_assistant_candidate(item)]
- assistant = [item for item in evidence_windows if is_assistant(item)]
- source_windows = list(evidence_windows)
- main = [item for item in source_windows if not is_coverage(item) and not is_assistant(item) and not is_assistant_candidate(item)]
- coverage = [item for item in source_windows if is_coverage(item)]
- assistant_candidates.sort(key=key, reverse=True)
- assistant.sort(key=key, reverse=True)
- main.sort(key=key, reverse=True)
- coverage.sort(key=key, reverse=True)
- top_assistant_memory_ids = {
- clean_text(item.get("memory_id", ""))
- for item in assistant[: max(1, int(os.getenv("TMCRA_FINAL_ASSISTANT_HARD_LIMIT", "4") or 4))]
- if clean_text(item.get("memory_id", ""))
- }
- if top_assistant_memory_ids:
- assistant_candidates.sort(
- key=lambda item: (
- int(clean_text(item.get("memory_id", "")) in top_assistant_memory_ids),
- *key(item),
- ),
- reverse=True,
- )
-
- ordered: list[dict[str, Any]] = []
- seen: set[str] = set()
-
- def add(items: list[dict[str, Any]], limit: int) -> None:
- added = 0
- for item in items:
- if len(ordered) >= final_limit or added >= limit:
- break
- memory_id = clean_text(item.get("memory_id", ""))
- text_key = clean_text(item.get("text", ""))[:240].lower()
- dedupe_key = text_key if is_assistant_candidate(item) else memory_id or text_key
- if dedupe_key and dedupe_key in seen:
- continue
- if dedupe_key:
- seen.add(dedupe_key)
- next_item = dict(item)
- if is_assistant_candidate(item):
- channel = "assistant_side"
- elif is_assistant(item):
- channel = "assistant"
- elif is_coverage(item):
- channel = "coverage"
- else:
- channel = "main"
- next_item["final_answer_channel"] = channel
- next_item["final_answer_channel_intent"] = dict(intent)
- ordered.append(next_item)
- added += 1
-
- if intent["assistant_detail"]:
- assistant_limit = max(1, int(os.getenv("TMCRA_FINAL_ASSISTANT_HARD_LIMIT", "4") or 4))
- add(assistant, min(assistant_limit, final_limit))
- try:
- assistant_side_limit = max(0, int(os.getenv("TMCRA_FINAL_ASSISTANT_SIDE_LIMIT", "1") or 1))
- except (TypeError, ValueError):
- assistant_side_limit = 1
- if assistant_side_limit:
- add(assistant_candidates, assistant_side_limit)
- add(main, main_hard_limit)
- add(coverage, coverage_hard_limit)
- add(assistant, final_limit)
- add(assistant_candidates, final_limit)
- add(main, final_limit)
- add(coverage, final_limit)
- return ordered[:final_limit]
-
-
-def _llm_channel_planner_enabled() -> bool:
- mode = os.getenv("TMCRA_LLM_CHANNEL_PLANNER_MODE", "").strip().lower()
- return mode not in {"", "0", "false", "off", "no", "disabled", "none"}
-
-
-def _int_list_from_payload(value: Any, *, upper_bound: int) -> list[int]:
- values = iter_selector_indices(value)
- result: list[int] = []
- seen: set[int] = set()
- for raw in values:
- try:
- index = int(str(raw).strip())
- except (TypeError, ValueError):
- continue
- if index < 1 or index > upper_bound or index in seen:
- continue
- seen.add(index)
- result.append(index)
- return result
-
-
-def apply_llm_channel_planner(question: str, evidence_windows: list[dict[str, Any]]) -> list[dict[str, Any]]:
- if not _llm_channel_planner_enabled() or not evidence_windows:
- return evidence_windows
- if any(bool(item.get("llm_channel_planner_enabled", False)) for item in evidence_windows):
- return evidence_windows
- try:
- max_windows = max(4, int(os.getenv("TMCRA_LLM_CHANNEL_PLANNER_MAX_WINDOWS", "16") or 16))
- max_chars = max(160, int(os.getenv("TMCRA_LLM_CHANNEL_PLANNER_WINDOW_CHARS", "520") or 520))
- max_tokens = max(160, int(os.getenv("TMCRA_LLM_CHANNEL_PLANNER_MAX_TOKENS", "700") or 700))
- except (TypeError, ValueError):
- max_windows, max_chars, max_tokens = 16, 520, 700
- scoped = list(evidence_windows[:max_windows])
- if not scoped:
- return evidence_windows
-
- lines: list[str] = []
- for index, item in enumerate(scoped, start=1):
- text = truncate(item.get("text", ""), max_chars)
- metadata = {
- "memory_id": clean_text(item.get("memory_id", "")),
- "score": round(float(item.get("score", 0.0) or 0.0), 4),
- "planner_score": round(float(item.get("planner_score", 0.0) or 0.0), 4),
- "evidence_role": clean_text(item.get("evidence_role", "")),
- "temporal_state": clean_text(item.get("temporal_state", "")),
- "unit_coverage": bool(item.get("unit_coverage_pack", False)),
- "multi_chain": bool(item.get("multi_unit_chain_slot", False)),
- "profile": bool(item.get("profile_shadow_unit", False) or item.get("profile_layer", False)),
- "unified_family": clean_text(item.get("unified_operation_family", "")),
- }
- lines.append(f"{index}. metadata={json.dumps(metadata, ensure_ascii=False, separators=(',', ':'))}\n{text}")
-
- system_prompt = (
- "You are an evidence-channel planner for a long-memory QA system. "
- "Do not answer the user question. Select evidence windows only. "
- "Return one strict JSON object and no markdown."
- )
- user_prompt = (
- "Plan the final evidence channels for the answer model.\n"
- "Rules:\n"
- "- main_indices must keep direct facts, stable base evidence, temporal anchors, assistant details, and profile facts.\n"
- "- coverage_indices are for count, sum, ratio, duration, repeated actions, multi-unit coverage, or unit-to-unit chains.\n"
- "- coverage evidence supplements main evidence; do not let coverage replace main facts.\n"
- "- support_indices are useful context only.\n"
- "- suppress_indices are duplicates, stale/conflicting values, or noise.\n"
- "- Use 1-based indices from the candidate list.\n"
- "- If unsure, keep a window in main or support instead of suppressing it.\n\n"
- "Return JSON schema:\n"
- "{\"operation_family\":\"direct|temporal|count|sum|ratio|profile|current_value|synthesis|unknown\","
- "\"main_indices\":[1],\"coverage_indices\":[],\"support_indices\":[],\"suppress_indices\":[],"
- "\"coverage_complete\":true,\"missing_evidence\":\"\",\"reasoning_policy\":\"short label\"}\n\n"
- f"Question:\n{question}\n\n"
- "Candidate evidence windows:\n"
- + "\n\n".join(lines)
- )
- try:
- base_url, model, api_key = answer_llm_config()
- raw = chat_completion(
- base_url,
- model,
- [{"role": "system", "content": system_prompt}, {"role": "user", "content": user_prompt}],
- max_tokens=max_tokens,
- temperature=0.0,
- api_key=api_key,
- )
- payload = parse_json_object(raw)
- except Exception as exc:
- annotated = [dict(item) for item in evidence_windows]
- if annotated:
- annotated[0]["llm_channel_planner_error"] = f"{exc.__class__.__name__}:{str(exc)[:160]}"
- return annotated
- if not payload:
- return evidence_windows
-
- main_indices = _int_list_from_payload(payload.get("main_indices", []), upper_bound=len(scoped))
- coverage_indices = _int_list_from_payload(payload.get("coverage_indices", []), upper_bound=len(scoped))
- support_indices = _int_list_from_payload(payload.get("support_indices", []), upper_bound=len(scoped))
- suppress_indices = _int_list_from_payload(payload.get("suppress_indices", []), upper_bound=len(scoped))
- channels: dict[int, str] = {}
- for channel, indices in (
- ("main", main_indices),
- ("coverage", coverage_indices),
- ("support", support_indices),
- ("suppress", suppress_indices),
- ):
- for index in indices:
- channels.setdefault(index - 1, channel)
-
- if not any(channel in {"main", "coverage", "support"} for channel in channels.values()):
- return evidence_windows
-
- channel_rank = {"main": 0, "coverage": 1, "support": 2, "unassigned": 3, "suppress": 4}
- planned: list[tuple[int, int, dict[str, Any]]] = []
- for index, item in enumerate(evidence_windows):
- next_item = dict(item)
- channel = channels.get(index, "unassigned") if index < len(scoped) else "unassigned"
- next_item["llm_channel_planner_enabled"] = True
- next_item["llm_channel"] = channel
- next_item["llm_channel_operation_family"] = clean_text(payload.get("operation_family", ""))
- next_item["llm_channel_coverage_complete"] = bool(payload.get("coverage_complete", False))
- missing_evidence = clean_text(payload.get("missing_evidence", ""))
- reasoning_policy = clean_text(payload.get("reasoning_policy", ""))
- if missing_evidence:
- next_item["llm_channel_missing_evidence"] = missing_evidence
- if reasoning_policy:
- next_item["llm_channel_reasoning_policy"] = reasoning_policy
- planned.append((channel_rank.get(channel, 3), index, next_item))
- planned.sort(key=lambda row: (row[0], row[1]))
- return [item for _, _, item in planned]
-
-
-def iter_selector_indices(value: Any) -> list[Any]:
- if isinstance(value, list):
- flattened: list[Any] = []
- for item in value:
- flattened.extend(iter_selector_indices(item))
- return flattened
- return [value]
-
-
-def profile_facts_for_answer(memory_hits: list[Any], *, max_items: int = 8) -> list[str]:
- facts: list[str] = []
- seen: set[str] = set()
- for hit in memory_hits[:12]:
- metadata = dict(getattr(hit, "metadata", {}) or {})
- source_kind = str(getattr(hit, "source_kind", "") or "")
- is_profile_hit = (
- bool(metadata.get("profile_first_hybrid_rescue"))
- or bool(metadata.get("profile_protected_slot"))
- or bool(metadata.get("profile_layer"))
- or source_kind.startswith("public_dialog_profile")
- or ".subject." in str(getattr(hit, "memory_id", "") or "")
- or source_kind in {"public_dialog_preference", "public_dialog_goal", "public_dialog_constraint", "public_dialog_status"}
- or str(getattr(hit, "category", "") or "").lower() in {"profile", "preference", "goal", "constraint", "status"}
- )
- if not is_profile_hit:
- continue
- profile_type = clean_text(metadata.get("profile_type", "")) or clean_text(getattr(hit, "category", ""))
- domain = clean_text(metadata.get("profile_domain_label", "") or metadata.get("profile_domain", ""))
- summary = clean_text(metadata.get("profile_summary", "") or metadata.get("profile_value", "") or getattr(hit, "value", ""))
- if not summary:
- continue
- prefix = "profile"
- if profile_type:
- prefix = profile_type.replace("_", " ")
- if domain:
- prefix = f"{prefix} / {domain}"
- fact = f"{prefix}: {summary}"
- key = fact.lower()
- if key in seen:
- continue
- seen.add(key)
- facts.append(fact)
- if len(facts) >= max_items:
- break
- return facts
-
-
-def complete_profile_answer_from_evidence(
- question: str,
- answer: str,
- evidence_windows: list[dict[str, Any]],
-) -> str:
- # TMCRA_LAYER_TAG: answer-side profile completion; preserves bound qualifiers from retrieved profile facts.
- answer_clean = clean_text(answer)
- if not answer_clean:
- return answer
- answer_l = answer_clean.lower()
- question_l = clean_text(question).lower()
- role_attribute_query = bool(re.search(r"\b(?:occupation|job|role|worked\s+as|work\s+as|career|position)\b", question_l))
- if len(answer_l) < 4:
- return answer
- for item in evidence_windows[:10]:
- if clean_text(item.get("evidence_role", "")).lower() != "profile_fact":
- continue
- fact = clean_text(item.get("text", ""))
- if not fact or len(fact) > 320:
- continue
- fact_l = fact.lower()
- if answer_l not in fact_l:
- continue
- candidates: list[str] = []
- for pattern in (
- r"\bas\s+(?:a|an|the)?\s*([^.;,\n]+)",
- r"\bworked\s+as\s+(?:a|an|the)?\s*([^.;,\n]+)",
- r"\bpreviously\s+(?:was|worked\s+as)\s+(?:a|an|the)?\s*([^.;,\n]+)",
- r"\brole\s+as\s+(?:a|an|the)?\s*([^.;,\n]+)",
- ):
- match = re.search(pattern, fact, flags=re.IGNORECASE)
- if match:
- candidates.append(clean_text(match.group(1)))
- if role_attribute_query and candidates:
- return candidates[0].strip(" .")
- for candidate in candidates:
- candidate_l = candidate.lower()
- if answer_l in candidate_l and len(candidate) > len(answer_clean) and len(candidate) <= 180:
- return candidate.strip(" .")
- return answer
-
-
-def answer_question(question: str, memory_hits: list[Any], evidence_windows: list[dict[str, Any]] | None = None) -> str:
- if evidence_windows is None:
- evidence_windows = build_answer_evidence(question, memory_hits)
- if not any(bool(item.get("window_planner_enabled", False)) for item in evidence_windows):
- evidence_windows = apply_answer_window_planner(question, list(evidence_windows))
- if not any(bool(item.get("llm_evidence_selector_enabled", False)) for item in evidence_windows):
- evidence_windows = apply_llm_evidence_selector(question, list(evidence_windows))
- if not evidence_unit_plan_from_windows(evidence_windows):
- evidence_windows = apply_evidence_unit_planner(question, list(evidence_windows))
- evidence_windows = apply_unified_operation_planner(question, list(evidence_windows))
- evidence_windows = apply_llm_channel_planner(question, list(evidence_windows))
- evidence_windows = final_answer_surface_windows(question, list(evidence_windows))
- question_l = question.lower()
- # TMCRA_LAYER_TAG: keyword-based personalized answer gate; replace with turn-intent/profile model.
- personalized_request = any(
- marker in question_l
- for marker in (
- "recommend",
- "recommendation",
- "suggest",
- "accessories",
- "my current",
- "my setup",
- "for me",
- "preference",
- "prefer",
- "what should",
- "should i",
- "any tips",
- "trouble with",
- "serve",
- "dinner",
- "battery life",
- )
- )
- force_answer_for_eval = os.getenv("TMCRA_LME_FORCE_ANSWER", "1").strip().lower() not in {"0", "false", "no"}
- profile_facts = profile_facts_for_answer(memory_hits)
- compact_hit_lines = compact_memory_hit_lines(memory_hits)
- memory_lines = []
- evidence_window_limit = int(os.getenv("TMCRA_ANSWER_EVIDENCE_WINDOW_LIMIT", "8"))
- if final_answer_channel_intent(question)["aggregation"]:
- evidence_window_limit = max(
- evidence_window_limit,
- int(os.getenv("TMCRA_ANSWER_EVIDENCE_WINDOW_LIMIT_AGG", "12") or 12),
- )
- ranked_evidence_windows = list(evidence_windows[: max(1, evidence_window_limit)])
- for index, item in enumerate(ranked_evidence_windows, start=1):
- value = clean_text(item.get("text", ""))
- if value:
- memory_id = clean_text(item.get("memory_id", ""))
- prefix = f"{index}. "
- if memory_id:
- prefix += f"[{memory_id}] "
- role_bits = []
- evidence_role = clean_text(item.get("evidence_role", ""))
- temporal_state = clean_text(item.get("temporal_state", ""))
- if evidence_role:
- role_bits.append(f"role={evidence_role}")
- if temporal_state:
- role_bits.append(f"time={temporal_state}")
- if role_bits:
- prefix += "[" + " ".join(role_bits) + "] "
- memory_lines.append(prefix + value)
- model_plan_lines = []
- model_plan_has_temporal_units = False
- model_plan_has_computed_temporal_answer = False
- model_plan_has_computed_coverage_answer = False
- seen_model_plan_ids: set[str] = set()
- for item in evidence_windows:
- if not (
- bool(item.get("temporal_comparison_model_planner", False))
- or bool(item.get("aggregation_unit_model_planner", False))
- or bool(item.get("operation_planner", False))
- or bool(item.get("unified_operation_planner", False))
- or bool(item.get("temporal_operation_layer", False))
- ):
- continue
- value = clean_text(item.get("text", ""))
- if not value:
- continue
- memory_id = clean_text(item.get("memory_id", "")) or f"model_plan_{len(model_plan_lines) + 1}"
- if memory_id in seen_model_plan_ids:
- continue
- seen_model_plan_ids.add(memory_id)
- if '"temporal_units":[{' in value:
- model_plan_has_temporal_units = True
- if re.search(r'"computed_temporal_answer":"[^"]+', value):
- model_plan_has_computed_temporal_answer = True
- if re.search(r'"computed_coverage_answer":\{"answer":"[^"]+', value):
- model_plan_has_computed_coverage_answer = True
- model_plan_lines.append(f"- [{memory_id}] {value}")
- if len(model_plan_lines) >= 3:
- break
- profile_block = "\n".join(f"- {fact}" for fact in profile_facts)
- memory_block = "\n".join(memory_lines) if memory_lines else "(no relevant long-term memory retrieved)"
- model_plan_section = (
- "Model planner outputs:\n"
- + "\n".join(model_plan_lines)
- + "\n\n"
- if model_plan_lines
- else ""
- )
- evidence_unit_plan = evidence_unit_plan_from_windows(evidence_windows)
- evidence_unit_plan_json = format_evidence_unit_plan_for_prompt(evidence_unit_plan)
- evidence_unit_plan_section = (
- f"Model evidence-unit plan:\n{evidence_unit_plan_json}\n\n" if evidence_unit_plan_json else ""
- )
- focused_unit_lines = focused_evidence_unit_lines(question, ranked_evidence_windows)
- focused_unit_section = (
- "Focused evidence units selected from the retrieved windows:\n"
- + "\n".join(focused_unit_lines)
- + "\n\n"
- if focused_unit_lines
- else ""
- )
- # TMCRA_LAYER_TAG: answer prompt protocol; affects answer utilization, not memory retrieval.
- reasoning_protocol = (
- "Treat TMCRA memory as evidence clues, not as a pre-written answer. "
- "If model temporal, aggregation, or operation plans are provided, read them before the raw evidence because they contain model-selected answer events, unit coverage, time anchors, and comparison direction. "
- "Use those plans as normalized evidence maps, but verify the answer against the raw memory evidence. "
- "If a model evidence-unit plan is provided, use it as the primary normalized view of answer units, time anchors, and distinct clues, while verifying it against the raw evidence lines. "
- "If focused evidence units are provided, scan them before the raw windows; they are compressed excerpts from retrieved memory, not external facts. "
- "If the evidence-unit plan lists answer_unit, positive_evidence, current_value, old_value, temporal_anchor, or negative_evidence units, reason over those units explicitly before deciding the answer. "
- "If an aggregation/unit coverage plan contains computed_coverage_answer.answer and its listed units_used or terms match the question, use that computed value unless raw evidence directly contradicts it. "
- "If candidate_answer is present in the plan and is supported by the listed units, prefer it over re-summarizing the raw window text. "
- "Before writing the JSON answer, internally infer the question task: direct fact lookup, arithmetic, distinct-item counting, temporal difference, current-value selection, preference/recommendation synthesis, or multi-evidence synthesis. "
- "Then use the retrieved evidence as clues for that task. "
- "For arithmetic questions, combine quantities found in different evidence lines and compute the requested value. "
- "For count questions, infer the answer unit from the current question and count distinct evidence-backed answer units across relevant evidence, not just one sentence. "
- "For offline benchmark count questions, include plausible evidence-backed units when the only uncertainty is an implicit date or an assistant-confirmed action and there is no contradictory evidence. "
- "For month-level count questions, a past-tense user event inside a dated session is a usable candidate for that session month when no older or future date contradicts it. "
- "When the question contains multiple actions joined by and/or, count each distinct target instance attached to any requested action. "
- "Do not merge two answer units merely because they share the same broad noun; keep them separate when the evidence distinguishes their action, status, time, location, or original-versus-replacement instance. "
- "For current/now/since/update questions, compare old and later evidence and choose the current or latest value when the question asks for it. "
- "For advice, preference, dinner, shopping, setup, or tips questions, synthesize a useful answer from remembered personal setup, owned items, ingredients, goals, habits, and constraints; exact prior wording is not required. "
- "Ignore irrelevant retrieved windows. "
- "Only answer that information is unavailable when none of the evidence provides a usable clue for the requested task. "
- )
- temporal_plan_protocol = ""
- if model_plan_has_temporal_units or model_plan_has_computed_temporal_answer:
- temporal_plan_protocol = (
- "When a model temporal plan explicitly contains non-empty temporal_units or computed_temporal_answer, use those fields as the time math layer. "
- "For how-many-ago questions, use computed_temporal_answer or delta_to_query fields instead of copying local words such as today. "
- "For first/order questions with explicit temporal_units, compare normalized_event_date values for the compared events. "
- "Ignore this temporal-plan instruction when the temporal plan has no temporal_units and no computed_temporal_answer. "
- )
- coverage_plan_protocol = ""
- if model_plan_has_computed_coverage_answer:
- coverage_plan_protocol = (
- "When a model aggregation/unit plan explicitly contains computed_coverage_answer, treat it as the unit coverage layer. "
- "For count, total, sum, and arithmetic questions, prefer the computed answer when the listed units_used or terms are relevant evidence for the requested unit. "
- "Do not drop a listed unit merely because its raw evidence window also contains unrelated topic text. "
- )
- personalized_instruction = ""
- if personalized_request:
- personalized_instruction = (
- "For personalized recommendation or preference questions, synthesize a concise recommendation using the user's retrieved setup, preferences, goals, and constraints. "
- "If a Structured personalized facts block is present, use it first; otherwise use the long-term memory evidence directly. "
- "Do not abstain when the memory contains relevant personal setup or preference evidence. "
- "Do not say the memory lacks the user's setup when the evidence mentions a current brand, device, activity, goal, style, constraint, or previous preference. "
- "Exact prior accessory lists are not required; infer a compatible recommendation from the remembered setup or preference. "
- + (
- "For this offline benchmark, do not ask follow-up questions; a partial remembered setup is enough to produce the best available personalized answer. "
- if force_answer_for_eval
- else "In realtime product mode, you may ask a brief follow-up question after giving the best available memory-backed suggestion. "
- )
- + "The answer should mention the personalized basis when useful, such as compatibility, brand, quality, durability, or current gear. "
- )
- profile_section = f"Structured personalized facts:\n{profile_block}\n\n" if profile_block else ""
- system_prompt = (
- "You answer LongMemEval questions using only the current question and the provided long-term memory. "
- "First identify the requested attribute, then answer only that attribute. "
- + reasoning_protocol
- + temporal_plan_protocol
- + coverage_plan_protocol
- + "If the question has multiple parts, compares earlier versus now, or asks for both before/current values, answer every requested part explicitly. "
- "If evidence lines are tagged with role=initial_value and role=current_value, use initial_value for the earlier/starting state and current_value for the now/current state; do not collapse them into one value. "
- "For current or now values, prefer later-dated evidence over earlier evidence when both are present. "
- "The answer value must not be an object, list, or multiple JSON keys. "
- "For where questions, answer with the place, store, venue, retailer, or location. "
- "Never answer a where question with a time or date such as last Sunday. "
- "For retail or shopping questions, a nearby store, retailer, or app/store name is the location. "
- "For how long questions, answer with the duration. "
- "For name questions, answer with the name. "
- "If the answer is implied by nearby turns in the same evidence window, use that local context. "
- "For profile, identity, occupation, role, ownership, and preference facts, preserve attached qualifiers that are part of the remembered fact, such as company type, setting, object owner, condition, strength, or scope. "
- "If a role=profile_fact evidence line or Structured personalized fact directly answers the requested attribute, preserve the complete specific fact rather than only the head noun or job title. "
- "Do not shorten a complete remembered fact into only its broad category when the evidence gives a more specific attribute requested by the question. "
- + personalized_instruction
- + "Return strict JSON only, with exactly one key named answer. "
- + "The answer value must be the shortest complete answer phrase or sentence. Do not explain. Do not cite evidence. "
- + "Example: {\"answer\":\"45 minutes each way\"}\n\n"
- + profile_section
- + model_plan_section
- + evidence_unit_plan_section
- + focused_unit_section
- + f"Long-term memory evidence:\n{memory_block}"
- )
- messages = [
- {
- "role": "system",
- "content": system_prompt,
- },
- {"role": "user", "content": f"Question: {question}\nReturn only JSON."},
- ]
- answer_base_url, answer_model, answer_api_key = answer_llm_config()
- answer = clean_answer_output(chat_completion(
- answer_base_url,
- answer_model,
- messages,
- max_tokens=int(os.getenv("TMCRA_ANSWER_MAX_TOKENS", "512")),
- temperature=0.0,
- api_key=answer_api_key,
- ))
- answer = complete_profile_answer_from_evidence(question, answer, evidence_windows)
- organizer_uses_model_plan = bool(
- re.search(
- r"\bhow many\s+(?:days?|weeks?|months?|years?)\b.*\bago\b|"
- r"\b(?:\d+|one|two|three|four|five|six|seven|eight|nine|ten|eleven|twelve)\s+"
- r"(?:days?|weeks?|months?|years?)\s+ago\b",
- question_l,
- )
- )
- organizer_context_section = (
- profile_section
- + (model_plan_section if organizer_uses_model_plan else "")
- + evidence_unit_plan_section
- )
- if memory_lines and should_run_answer_reasoning_organizer(
- question,
- answer,
- personalized_request=personalized_request,
- force_answer_for_eval=force_answer_for_eval,
- ):
- organized_answer = organize_answer_from_evidence(
- question,
- memory_lines,
- profile_section=organizer_context_section,
- force_answer_for_eval=force_answer_for_eval,
- prior_answer=answer,
- )
- if (not organized_answer or looks_like_no_memory_answer(organized_answer)) and compact_hit_lines:
- organized_answer = organize_answer_from_evidence(
- question,
- compact_hit_lines,
- profile_section=organizer_context_section,
- force_answer_for_eval=force_answer_for_eval,
- prior_answer=answer,
- )
- if organized_answer and not looks_like_no_memory_answer(organized_answer):
- return complete_profile_answer_from_evidence(question, organized_answer, evidence_windows)
- focused_memory_block = "\n".join(memory_lines[:8])
- # TMCRA_LAYER_TAG: answer-side fallback organizer for conservative/no-memory answers.
- retry_messages = [
- {
- "role": "system",
- "content": (
- "You are correcting an over-conservative answer. "
- "Your previous answer treated memory as a complete answer database; that is wrong. "
- "The focused evidence below is non-empty and may contain indirect clues rather than an exact answer sentence. "
- "You must first infer the task required by the question, then combine the evidence clues to answer it. "
- "Use arithmetic for total/each/save questions, distinct counting for how-many questions, date comparison for temporal questions, current/latest evidence for now/current questions, and synthesis from remembered setup or preferences for advice/tips/recommendation questions. "
- "For advice, tips, what-should-I, or recommendation questions, a remembered owned item, ingredient, activity, goal, preference, or constraint is enough to produce a useful answer; the memory does not need to contain the exact advice sentence. "
- "For count questions, infer the answer unit from the current question and combine distinct evidence-backed units across the focused evidence instead of requiring a precomputed total. "
- "If a model evidence-unit plan is provided, use its answer_unit and positive_evidence units as normalized candidates and only exclude units that are contradicted or outside the requested time/action scope. "
- "For month-level count questions, do not discard a past-tense user event merely because only the session date provides the month anchor. "
- "Do not merge units only because they share the same broad noun; keep distinct action targets or original/replacement instances separate when the evidence separates them. "
- "Do not claim no memory if any focused evidence gives a usable clue for the requested task. "
- "Do not answer with Information unavailable, not available, no information, or similar wording unless the focused evidence block is empty or entirely unrelated. "
- "Ignore evidence that is unrelated to the requested task. "
- + (
- "For this offline benchmark, do not ask follow-up questions; give the best direct evidence-backed answer. "
- if force_answer_for_eval
- else "In realtime product mode, you may ask a brief follow-up question after giving the best memory-backed answer. "
- )
- +
- "Return strict JSON only, with exactly one key named answer.\n\n"
- + organizer_context_section
- + f"Focused long-term memory evidence:\n{focused_memory_block}"
- ),
- },
- {"role": "user", "content": f"Question: {question}\nReturn only JSON."},
- ]
- retry_answer = clean_answer_output(chat_completion(
- answer_base_url,
- answer_model,
- retry_messages,
- max_tokens=128,
- temperature=0.0,
- api_key=answer_api_key,
- ))
- if retry_answer and not looks_like_no_memory_answer(retry_answer):
- return complete_profile_answer_from_evidence(question, retry_answer, evidence_windows)
- return complete_profile_answer_from_evidence(question, answer, evidence_windows)
-
-
-def should_run_answer_reasoning_organizer(
- question: str,
- answer: str,
- *,
- personalized_request: bool,
- force_answer_for_eval: bool,
-) -> bool:
- # TMCRA_LAYER_TAG: rule gate for answer-side retry/organizer; not a trained routing model.
- mode = os.getenv("TMCRA_ANSWER_REASONING_ORGANIZER", "1").strip().lower()
- if mode in {"", "0", "false", "off", "disabled", "none"}:
- return False
- if looks_like_no_memory_answer(answer):
- return True
- if personalized_request and force_answer_for_eval and looks_like_followup_answer(answer):
- return True
- question_l = clean_text(question).lower()
- count_markers = (
- "how many",
- "number of",
- "count of",
- "count the",
- "count my",
- "total number",
- )
- return any(marker in question_l for marker in count_markers)
-
-
-def organize_answer_from_evidence(
- question: str,
- memory_lines: list[str],
- *,
- profile_section: str = "",
- force_answer_for_eval: bool = True,
- prior_answer: str = "",
-) -> str:
- if not memory_lines:
- return ""
- evidence_block = "\n".join(memory_lines)
- prior_answer = clean_text(prior_answer)
- prior_section = f"Previous candidate answer, possibly incomplete: {prior_answer}\n\n" if prior_answer else ""
- messages = [
- {
- "role": "system",
- "content": (
- "You are the evidence organizer for a long-memory runtime. "
- "The final answer model may have been too conservative or may have under-used the retrieved clues, so your job is to read the retrieved memory lines as clues and produce the best evidence-backed answer. "
- "Do not treat memory as a database that must contain the exact answer sentence. "
- "Ignore unrelated lines, but if any line contains a usable clue for the question, mark relevant=true. "
- "For advice, tips, recommendations, dinner, shopping, setup, or preference questions, remembered owned items, tools, accessories, ingredients, goals, habits, constraints, or preferences are enough to synthesize an answer. "
- "Infer the affordance of remembered items and ingredients: a relevant tool, accessory, setup, ingredient, preference, or constraint can answer the user's underlying need even when the old conversation asked a different surface question. "
- "Do not answer the old conversation's surface task; adapt the remembered clue to the current question's need. "
- "When advice uses a remembered owned item, phrase the answer as using or preparing that already-mentioned item and include one practical use step; do not turn it into a generic purchase suggestion unless the question asks what to buy. "
- "If a model evidence-unit plan is provided, enumerate the plan's answer_unit and positive_evidence units first, then reconcile duplicates or exclusions against the raw evidence. "
- "For count questions, infer the count unit from the current question itself, then enumerate all relevant candidate clues, merge duplicate mentions of the same clue, and count only evidence-backed units matching that inferred unit. "
- "For month-level count questions, include past-tense user events from a dated session as candidates for that session month unless another date excludes them. "
- "When the current question contains multiple actions joined by and/or, the count unit is each distinct target instance attached to any requested action, unless the question explicitly asks for object categories. "
- "Do not merge two candidate units merely because they share the same broad noun; keep them separate when the evidence distinguishes their action, status, time, location, or original-versus-replacement instance. "
- "If one memory line contains multiple distinct obligations or requested action targets, count them separately only when the inferred count unit requires action-level counting. "
- "Do not count broad object categories, generic advice, examples, or unrelated context as answer units. "
- "Do not stop after the first relevant clue; scan every retrieved line before producing the count. "
- "For current/now questions, prefer later or current evidence over old evidence when both appear. "
- "For temporal questions, compare dates or durations when evidence provides them. "
- "Return strict JSON only with keys: relevant, task, clues, answer. "
- "answer must be the final shortest useful answer, not an explanation. "
- "If evidence is genuinely empty or unrelated, return relevant=false and answer=\"\". "
- + (
- "For this offline benchmark, do not ask follow-up questions; use the best available evidence-backed inference. "
- if force_answer_for_eval
- else "In realtime product mode, prefer a direct memory-backed answer; only ask follow-up if evidence is too thin. "
- )
- + "\n\n"
- + profile_section
- + prior_section
- + f"Retrieved memory lines:\n{evidence_block}"
- ),
- },
- {"role": "user", "content": f"Question: {question}\nReturn only JSON."},
- ]
- answer_base_url, answer_model, answer_api_key = answer_llm_config()
- raw = chat_completion(
- answer_base_url,
- answer_model,
- messages,
- max_tokens=220,
- temperature=0.0,
- api_key=answer_api_key,
- )
- payload = parse_json_object(raw)
- if not payload:
- fallback = clean_answer_output(raw)
- return "" if fallback.lstrip().startswith("{") else fallback
- relevant = payload.get("relevant", False)
- if isinstance(relevant, str):
- relevant = relevant.strip().lower() in {"1", "true", "yes", "y"}
- answer = clean_text(payload.get("answer", ""))
- if not relevant or not answer:
- return ""
- return answer.strip(" \t\n\r\"'")
-
-
-def compact_memory_hit_lines(memory_hits: list[Any], *, max_items: int | None = None, max_chars: int | None = None) -> list[str]:
- if max_items is None:
- max_items = int(os.getenv("TMCRA_COMPACT_MEMORY_HIT_LIMIT", "8"))
- if max_chars is None:
- max_chars = int(os.getenv("TMCRA_COMPACT_MEMORY_HIT_CHARS", "260"))
- lines: list[str] = []
- seen: set[str] = set()
- for index, hit in enumerate(memory_hits[: max(1, max_items)], start=1):
- value = truncate(getattr(hit, "value", ""), max(80, max_chars))
- if not value:
- continue
- memory_id = clean_text(getattr(hit, "memory_id", ""))
- category = clean_text(getattr(hit, "category", ""))
- source_kind = clean_text(getattr(hit, "source_kind", ""))
- label_bits = []
- if category:
- label_bits.append(category)
- if source_kind:
- label_bits.append(source_kind)
- label = " / ".join(label_bits)
- prefix = f"{index}. "
- if memory_id:
- prefix += f"[{memory_id}] "
- if label:
- prefix += f"({label}) "
- line = prefix + value
- key = re.sub(r"\s+", " ", line.lower())
- if key in seen:
- continue
- seen.add(key)
- lines.append(line)
- return lines
-
-
-def parse_json_object(text: str) -> dict[str, Any]:
- text = clean_text(text)
- text = re.sub(r"^```(?:\w+)?\s*|\s*```$", "", text).strip()
- decoder = json.JSONDecoder()
- for index, char in enumerate(text):
- if char != "{":
- continue
- try:
- payload, _ = decoder.raw_decode(text[index:])
- except Exception:
- continue
- return payload if isinstance(payload, dict) else {}
- return {}
-
-
-def looks_like_no_memory_answer(answer: str) -> bool:
- # TMCRA_LAYER_TAG: string-marker abstention detector used to trigger answer-side retry.
- value = clean_text(answer).lower()
- if not value:
- return True
- no_memory_markers = (
- "no information",
- "no relevant",
- "not contain",
- "does not contain",
- "doesn't contain",
- "do not have",
- "don't have",
- "cannot determine",
- "can't determine",
- "not enough information",
- "insufficient information",
- "information unavailable",
- "not available",
- "unavailable",
- "not mention",
- "not mentioned",
- "unknown",
- )
- return any(marker in value for marker in no_memory_markers)
-
-
-def looks_like_followup_answer(answer: str) -> bool:
- # TMCRA_LAYER_TAG: string-marker follow-up detector used by offline benchmark force-answer mode.
- value = clean_text(answer).lower()
- if not value:
- return False
- followup_markers = (
- "could you tell me",
- "can you tell me",
- "i need more information",
- "need more information",
- "tell me more",
- "what is your budget",
- "what's your budget",
- "what kind of",
- "would you like",
- "are you looking",
- )
- return any(marker in value for marker in followup_markers) or value.endswith("?")
-
-
-def clean_answer_output(text: str) -> str:
- text = clean_text(text)
- text = re.sub(r"^```(?:\w+)?\s*|\s*```$", "", text).strip()
- json_match = re.search(r"\{.*?\}", text, flags=re.DOTALL)
- if json_match:
- try:
- payload = json.loads(json_match.group(0))
- answer = clean_text(payload.get("answer", ""))
- if answer:
- return answer.strip(" \t\n\r\"'")
- except Exception:
- pass
- tail_markers = [
- "Referencing the",
- "Based on the",
- "While the",
- "While ",
- "Although ",
- "However,",
- "There is no mention",
- "There is no information",
- "Give a direct",
- "Do not mention",
- "The provided context",
- "In the provided",
- ]
- for marker in tail_markers:
- index = text.find(marker)
- if index > 0:
- text = text[:index].strip()
- return text.strip(" \t\n\r\"'")
-
-
-def run(args: argparse.Namespace) -> None:
- service_root = Path(args.service_root).resolve()
- repo = Path(args.repo).resolve()
- apply_env_defaults(service_root)
- os.environ.setdefault("GEMMA_BASE_URL", f"http://{os.getenv('TMCRA_GEMMA_HOST', '127.0.0.1')}:{os.getenv('TMCRA_GEMMA_PORT', '18002')}/v1")
- os.environ.setdefault("GEMMA_MODEL", os.getenv("TMCRA_GEMMA_MODEL_NAME", "gemma-4-e4b-it"))
- if str(repo) not in sys.path:
- sys.path.insert(0, str(repo))
- if not args.enable_topic_buckets:
- disable_topic_bucket_runtime()
-
- started = time.time()
- out_dir = Path(args.out) if args.out else DEFAULT_OUT_ROOT / f"lme_s10_native_{datetime.now().strftime('%Y%m%d_%H%M%S')}"
- out_dir.mkdir(parents=True, exist_ok=True)
- predictions_path = out_dir / "predictions.jsonl"
- debug_path = out_dir / "samples_debug.jsonl"
- summary_path = out_dir / "summary.json"
- storage_path = out_dir / "native_memory.sqlite3"
-
- writer = build_writer()
- records: list[dict[str, Any]] = []
- log("run_start", data=str(args.data), limit=args.limit, out=str(out_dir), repo=str(repo))
- for sample_index, row in enumerate(iter_json_array(Path(args.data), limit=args.limit), start=1):
- qid = clean_text(row.get("question_id")) or f"sample_{sample_index:04d}"
- question = clean_text(row.get("question"))
- gold = clean_text(row.get("answer"))
- sessions = list(row.get("haystack_sessions") or [])
- session_ids = [clean_text(item) for item in list(row.get("haystack_session_ids") or [])]
- dates = [clean_text(item) for item in list(row.get("haystack_dates") or [])]
- selected_indices = (
- select_official_full_history_indices(row)
- if args.official_full_history
- else select_session_indices(row, max_distractors=args.max_distractor_sessions)
- )
- adapter = build_adapter(f"longmemeval_native:{qid}:{sample_index}", storage_path)
- sample_start = time.perf_counter()
- writer_calls = 0
- writer_accepted = 0
- writer_suspected = 0
- writer_records = 0
- writer_seconds = 0.0
- writer_unit_records = 0
- writer_unit_calls = 0
- chunk_errors: list[str] = []
- selected_session_ids = []
-
- for session_index in selected_indices:
- if session_index >= len(sessions):
- continue
- sid = session_ids[session_index] if session_index < len(session_ids) else f"session_{session_index}"
- selected_session_ids.append(sid)
- date = dates[session_index] if session_index < len(dates) else ""
- is_answer_session = sid in {clean_text(item) for item in list(row.get("answer_session_ids") or [])}
- max_chunks = (
- args.max_session_chunks
- if args.official_full_history
- else args.max_answer_chunks if is_answer_session else args.max_distractor_chunks
- )
- chunks = session_text_chunks(
- session_id=sid,
- date=date,
- turns=list(sessions[session_index] or []),
- max_chars=args.chunk_chars,
- max_chunks=max_chunks,
- )
- for chunk_index, chunk in enumerate(chunks, start=1):
- chunk_id = f"s{session_index:03d}_c{chunk_index:02d}"
- try:
- result = writer_ingest(adapter, writer, chunk, qid=qid, chunk_id=chunk_id)
- writer_calls += 1
- writer_accepted += result["accepted_count"]
- writer_suspected += result["suspected_count"]
- writer_records += result["record_count"]
- writer_seconds += float(result["seconds"])
- writer_unit_records += int(result.get("unit_record_count", 0) or 0)
- if bool(result.get("unit_writer_enabled", False)):
- writer_unit_calls += 1
- except Exception as exc:
- chunk_errors.append(f"{chunk_id}:{exc.__class__.__name__}:{str(exc)[:200]}")
- log("chunk_error", sample=sample_index, qid=qid, chunk=chunk_id, error=str(exc)[:240])
-
- question_date = clean_text(row.get("question_date", ""))
- query_graph = build_query_graph(question, question_date)
- runtime_question = query_graph_retrieval_text(question, question_date, query_graph)
- retrieve_start = time.perf_counter()
- retrieval = adapter.retrieve(runtime_question, top_k=args.top_k)
- retrieve_seconds = time.perf_counter() - retrieve_start
- hits = list(getattr(retrieval, "hits", []) or [])
- hits, query_graph_sidecar = apply_query_graph_sidecar_retrieval(
- adapter,
- query_graph,
- question,
- question_date,
- hits,
- top_k=args.top_k,
- )
- hits = expand_dialogue_chain_hits(runtime_question, hits, getattr(adapter, "graph", None))
- hits = expand_semantic_coverage_hits(runtime_question, hits, getattr(adapter, "graph", None))
- answer_evidence_windows = build_answer_evidence(runtime_question, hits)
- answer_evidence_windows = apply_answer_window_planner(runtime_question, answer_evidence_windows)
- answer_evidence_windows = apply_llm_evidence_selector(runtime_question, answer_evidence_windows)
- answer_evidence_windows = apply_evidence_unit_planner(runtime_question, answer_evidence_windows)
- answer_evidence_windows = apply_unified_operation_planner(runtime_question, answer_evidence_windows)
- answer_evidence_windows = diversify_evidence_windows(answer_evidence_windows)
- answer_evidence_windows = apply_llm_channel_planner(runtime_question, answer_evidence_windows)
- answer_evidence_windows = final_answer_surface_windows(runtime_question, answer_evidence_windows)
- answer_start = time.perf_counter()
- hypothesis = answer_question(runtime_question, hits, answer_evidence_windows)
- answer_seconds = time.perf_counter() - answer_start
- graph = getattr(adapter, "graph", None)
- record = {
- "sample_index": sample_index,
- "question_id": qid,
- "question_type": row.get("question_type"),
- "question": question,
- "question_date": question_date,
- "query_graph": query_graph,
- "query_graph_sidecar": query_graph_sidecar,
- "gold_answer": gold,
- "hypothesis": hypothesis,
- "answer_session_ids": list(row.get("answer_session_ids") or []),
- "selected_session_ids": selected_session_ids,
- "total_sessions": len(sessions),
- "total_turns": sum(len(s or []) for s in sessions),
- "selected_sessions": len(selected_session_ids),
- "history_mode": "official_full_history" if args.official_full_history else "controlled_answer_plus_distractors",
- "writer_calls": writer_calls,
- "writer_accepted": writer_accepted,
- "writer_suspected": writer_suspected,
- "writer_records": writer_records,
- "writer_unit_records": writer_unit_records,
- "writer_unit_calls": writer_unit_calls,
- "writer_seconds": round(writer_seconds, 3),
- "retrieve_seconds_wall": round(retrieve_seconds, 3),
- "answer_seconds": round(answer_seconds, 3),
- "sample_seconds": round(time.perf_counter() - sample_start, 3),
- "graph_records": len(getattr(graph, "records_by_id", {}) or {}),
- "graph_edges": len(getattr(graph, "edges", []) or []),
- "chunk_errors": chunk_errors,
- "retrieval": retrieval_debug(retrieval),
- "evidence_unit_plan": evidence_unit_plan_from_windows(answer_evidence_windows),
- "answer_evidence_windows": [
- {
- "memory_id": item.get("memory_id", ""),
- "score": item.get("score", 0.0),
- "planner_selected": bool(item.get("planner_selected", False)),
- "planner_score": item.get("planner_score", 0.0),
- "semantic_similarity": item.get("semantic_similarity", item.get("answer_window_semantic_similarity", 0.0)),
- "answer_window_semantic_similarity": item.get("answer_window_semantic_similarity", 0.0),
- "answer_window_semantic_enabled": bool(item.get("answer_window_semantic_enabled", False)),
- "answer_window_semantic_mode": item.get("answer_window_semantic_mode", ""),
- "window_planner_should_inject_score": item.get("window_planner_should_inject_score", 0.0),
- "answer_plan_selected": bool(item.get("answer_plan_selected", False)),
- "answer_plan_score": item.get("answer_plan_score", 0.0),
- "answer_plan_rank": item.get("answer_plan_rank", 0),
- "evidence_role": item.get("evidence_role", ""),
- "temporal_state": item.get("temporal_state", ""),
- "assistant_memory_query": bool(item.get("assistant_memory_query", False)),
- "assistant_origin_evidence": bool(item.get("assistant_origin_evidence", False)),
- "hit_evidence_role": item.get("hit_evidence_role", ""),
- "hit_temporal_state": item.get("hit_temporal_state", ""),
- "window_planner_enabled": bool(item.get("window_planner_enabled", False)),
- "llm_evidence_selector_enabled": bool(item.get("llm_evidence_selector_enabled", False)),
- "llm_evidence_selected": bool(item.get("llm_evidence_selected", False)),
- "llm_evidence_rank": item.get("llm_evidence_rank", None),
- "llm_evidence_model_kept": bool(item.get("llm_evidence_model_kept", False)),
- "evidence_unit_planner_enabled": bool(item.get("evidence_unit_planner_enabled", False)),
- "evidence_unit_planner_selected": bool(item.get("evidence_unit_planner_selected", False)),
- "evidence_unit_planner_rank": item.get("evidence_unit_planner_rank", None),
- "semantic_coverage_expansion": bool(item.get("semantic_coverage_expansion", False)),
- "semantic_coverage_score": item.get("semantic_coverage_score", 0.0),
- "unit_coverage_pack": bool(item.get("unit_coverage_pack", False)),
- "multi_unit_chain_slot": bool(item.get("multi_unit_chain_slot", False)),
- "multi_unit_chain_bundle": bool(item.get("multi_unit_chain_bundle", False)),
- "multi_unit_chain_score": float(item.get("multi_unit_chain_score", 0.0) or 0.0),
- "unit_kind": item.get("unit_kind", ""),
- "facet_type": item.get("facet_type", ""),
- "unified_operation_planner": bool(item.get("unified_operation_planner", False)),
- "unified_operation_family": item.get("unified_operation_family", ""),
- "unified_selected": bool(item.get("unified_selected", False)),
- "unified_scores": item.get("unified_scores", {}),
- "llm_channel_planner_enabled": bool(item.get("llm_channel_planner_enabled", False)),
- "llm_channel": item.get("llm_channel", ""),
- "llm_channel_operation_family": item.get("llm_channel_operation_family", ""),
- "llm_channel_coverage_complete": bool(item.get("llm_channel_coverage_complete", False)),
- "llm_channel_missing_evidence": item.get("llm_channel_missing_evidence", ""),
- "llm_channel_reasoning_policy": item.get("llm_channel_reasoning_policy", ""),
- "llm_channel_planner_error": item.get("llm_channel_planner_error", ""),
- "unit_indexes": item.get("unit_indexes", []),
- "original_chars": item.get("original_chars", 0),
- "text": truncate(item.get("text", ""), 700),
- }
- for item in answer_evidence_windows[:8]
- ],
- }
- records.append(record)
- with predictions_path.open("a", encoding="utf-8") as handle:
- handle.write(json.dumps({"question_id": qid, "hypothesis": hypothesis}, ensure_ascii=False) + "\n")
- with debug_path.open("a", encoding="utf-8") as handle:
- handle.write(json.dumps(record, ensure_ascii=False) + "\n")
- log(
- "sample_done",
- sample=sample_index,
- qid=qid,
- hits=record["retrieval"]["hit_count"],
- selected_events=record["retrieval"]["selected_event_count"],
- paths=record["retrieval"]["selected_path_count"],
- writer_calls=writer_calls,
- seconds=record["sample_seconds"],
- hypothesis=truncate(hypothesis, 120),
- )
-
- summary = {
- "status": "completed",
- "mode": "native_tmcra_flow_smoke",
- "history_mode": "official_full_history" if args.official_full_history else "controlled_answer_plus_distractors",
- "limit": args.limit,
- "sample_count": len(records),
- "out_dir": str(out_dir),
- "data": str(args.data),
- "started_at": datetime.fromtimestamp(started).isoformat(timespec="seconds"),
- "completed_at": datetime.now().isoformat(timespec="seconds"),
- "elapsed_seconds": round(time.time() - started, 3),
- "total_writer_calls": sum(int(item["writer_calls"]) for item in records),
- "total_writer_accepted": sum(int(item["writer_accepted"]) for item in records),
- "total_writer_suspected": sum(int(item["writer_suspected"]) for item in records),
- "total_writer_unit_calls": sum(int(item.get("writer_unit_calls", 0)) for item in records),
- "total_writer_unit_records": sum(int(item.get("writer_unit_records", 0)) for item in records),
- "total_graph_records": sum(int(item["graph_records"]) for item in records),
- "total_graph_edges": sum(int(item["graph_edges"]) for item in records),
- "avg_retrieval_hits": round(sum(float(item["retrieval"]["hit_count"]) for item in records) / max(1, len(records)), 4),
- "avg_selected_events": round(sum(float(item["retrieval"]["selected_event_count"]) for item in records) / max(1, len(records)), 4),
- "avg_selected_paths": round(sum(float(item["retrieval"]["selected_path_count"]) for item in records) / max(1, len(records)), 4),
- "samples": [
- {
- "question_id": item["question_id"],
- "question": item["question"],
- "gold_answer": item["gold_answer"],
- "hypothesis": item["hypothesis"],
- "retrieval_hits": item["retrieval"]["hit_count"],
- "selected_event_count": item["retrieval"]["selected_event_count"],
- "selected_path_count": item["retrieval"]["selected_path_count"],
- "writer_calls": item["writer_calls"],
- "sample_seconds": item["sample_seconds"],
- "chunk_errors": item["chunk_errors"],
- }
- for item in records
- ],
- }
- summary_path.write_text(json.dumps(summary, ensure_ascii=False, indent=2), encoding="utf-8")
- log("run_done", out=str(out_dir), samples=len(records), elapsed=summary["elapsed_seconds"])
-
-
-def main() -> None:
- parser = argparse.ArgumentParser()
- parser.add_argument("--data", default=str(DEFAULT_DATA))
- parser.add_argument("--repo", default=str(DEFAULT_REPO))
- parser.add_argument("--service-root", default=str(DEFAULT_SERVICE_ROOT))
- parser.add_argument("--out", default="")
- parser.add_argument("--limit", type=int, default=10)
- parser.add_argument("--top-k", type=int, default=8)
- parser.add_argument("--max-distractor-sessions", type=int, default=5)
- parser.add_argument("--max-distractor-chunks", type=int, default=1)
- parser.add_argument("--max-answer-chunks", type=int, default=4)
- parser.add_argument("--official-full-history", action="store_true")
- parser.add_argument("--max-session-chunks", type=int, default=0)
- parser.add_argument("--chunk-chars", type=int, default=7000)
- parser.add_argument("--enable-topic-buckets", action="store_true")
- args = parser.parse_args()
- run(args)
-
-
-if __name__ == "__main__":
- main()
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import re
+import sys
+import time
+import urllib.error
+import urllib.request
+from datetime import datetime, timezone
+from pathlib import Path
+from types import SimpleNamespace
+from typing import Any, Iterable, Mapping
+
+
+DEFAULT_SERVICE_ROOT = Path(os.getenv("TMCRA_SERVICE_ROOT", "./tmcra_api_service"))
+DEFAULT_REPO = Path(os.getenv("TMCRA_REPO_ROOT", str(DEFAULT_SERVICE_ROOT / "private" / "tmcra-integrated")))
+DEFAULT_DATA = Path(os.getenv("LONGMEMEVAL_S_DATA", "./data/longmemeval_s_cleaned.json"))
+DEFAULT_OUT_ROOT = Path(os.getenv("TMCRA_LME_RUN_ROOT", "./runs"))
+
+
+def log(event: str, **payload: Any) -> None:
+ stamp = datetime.now().isoformat(timespec="seconds")
+ details = " ".join(f"{key}={json.dumps(value, ensure_ascii=False)}" for key, value in sorted(payload.items()))
+ print(f"[lme_native] {stamp} {event}" + (f" {details}" if details else ""), flush=True)
+
+
+def read_env_file(path: Path) -> dict[str, str]:
+ env: dict[str, str] = {}
+ if not path.exists():
+ return env
+ for raw in path.read_text(encoding="utf-8", errors="replace").splitlines():
+ line = raw.strip()
+ if not line or line.startswith("#") or "=" not in line:
+ continue
+ if line.startswith("export "):
+ line = line[len("export ") :].strip()
+ key, value = line.split("=", 1)
+ env[key.strip()] = value.strip().strip('"').strip("'")
+ return env
+
+
+def apply_env_defaults(service_root: Path) -> None:
+ for env_path in [service_root / "env" / "tmcra-api.env", service_root / "env" / "tmcra-gemma.env"]:
+ for key, value in read_env_file(env_path).items():
+ if value == "":
+ continue
+ os.environ.setdefault(key, value)
+
+
+def iter_json_array(path: Path, *, limit: int = 0) -> Iterable[dict[str, Any]]:
+ import codecs
+
+ decoder = codecs.getincrementaldecoder("utf-8")(errors="replace")
+ yielded = 0
+ array_started = False
+ object_started = False
+ object_depth = 0
+ in_string = False
+ escape_next = False
+ buffer: list[str] = []
+ with path.open("rb") as handle:
+ while True:
+ chunk = handle.read(1 << 20)
+ if not chunk:
+ text = decoder.decode(b"", final=True)
+ else:
+ text = decoder.decode(chunk)
+ if not text and not chunk:
+ break
+ for char in text:
+ if not array_started:
+ if char.isspace():
+ continue
+ if char != "[":
+ raise RuntimeError(f"expected top-level JSON array in {path}")
+ array_started = True
+ continue
+ if not object_started:
+ if char.isspace() or char == ",":
+ continue
+ if char == "]":
+ return
+ if char != "{":
+ raise RuntimeError(f"expected object item in {path}, got {char!r}")
+ object_started = True
+ object_depth = 1
+ in_string = False
+ escape_next = False
+ buffer = ["{"]
+ continue
+ buffer.append(char)
+ if in_string:
+ if escape_next:
+ escape_next = False
+ elif char == "\\":
+ escape_next = True
+ elif char == '"':
+ in_string = False
+ continue
+ if char == '"':
+ in_string = True
+ continue
+ if char == "{":
+ object_depth += 1
+ elif char == "}":
+ object_depth -= 1
+ if object_depth == 0:
+ row = json.loads("".join(buffer))
+ if isinstance(row, dict):
+ yield row
+ yielded += 1
+ if limit > 0 and yielded >= limit:
+ return
+ object_started = False
+ buffer = []
+ if not chunk:
+ break
+ if object_started:
+ raise RuntimeError(f"unterminated JSON object in {path}")
+
+
+def http_json(
+ method: str,
+ url: str,
+ payload: dict[str, Any] | None = None,
+ *,
+ timeout: int = 300,
+ api_key: str = "",
+) -> dict[str, Any]:
+ data = json.dumps(payload, ensure_ascii=False).encode("utf-8") if payload is not None else None
+ headers = {"Content-Type": "application/json; charset=utf-8"}
+ if api_key:
+ headers["Authorization"] = f"Bearer {api_key}"
+ max_attempts = max(1, int(os.getenv("TMCRA_HTTP_JSON_MAX_ATTEMPTS", "8") or 8))
+ retry_codes = {429, 500, 502, 503, 504}
+ last_error: Exception | None = None
+ for attempt in range(1, max_attempts + 1):
+ req = urllib.request.Request(url, data=data, method=method, headers=headers)
+ try:
+ with urllib.request.urlopen(req, timeout=timeout) as response:
+ body = response.read().decode("utf-8", "replace")
+ return json.loads(body) if body else {}
+ except urllib.error.HTTPError as exc:
+ body = exc.read().decode("utf-8", "replace")
+ last_error = RuntimeError(f"HTTP {exc.code} {url}: {body[:500]}")
+ if exc.code not in retry_codes or attempt >= max_attempts:
+ raise last_error from exc
+ except (urllib.error.URLError, TimeoutError, json.JSONDecodeError) as exc:
+ last_error = exc
+ if attempt >= max_attempts:
+ raise RuntimeError(f"HTTP request failed {url}: {exc}") from exc
+ time.sleep(min(2 ** (attempt - 1), 8))
+ raise RuntimeError(f"HTTP request failed {url}: {last_error}")
+
+
+def chat_completion(
+ base_url: str,
+ model: str,
+ messages: list[dict[str, str]],
+ *,
+ max_tokens: int = 180,
+ temperature: float = 0.0,
+ api_key: str = "",
+) -> str:
+ wire_api = clean_text(os.getenv("TMCRA_ANSWER_WIRE_API", os.getenv("OPENAI_WIRE_API", ""))).lower()
+ if wire_api == "responses":
+ body = http_json(
+ "POST",
+ base_url.rstrip("/") + "/responses",
+ {
+ "model": model,
+ "input": messages,
+ "max_output_tokens": max_tokens,
+ "temperature": temperature,
+ },
+ timeout=360,
+ api_key=api_key,
+ )
+ output_text = clean_text(body.get("output_text", ""))
+ if output_text:
+ return output_text
+ chunks: list[str] = []
+ for item in body.get("output", []) or []:
+ for content in item.get("content", []) or []:
+ text = content.get("text") or content.get("content") or ""
+ if text:
+ chunks.append(str(text))
+ return "\n".join(chunks).strip()
+ body = http_json(
+ "POST",
+ base_url.rstrip("/") + "/chat/completions",
+ {
+ "model": model,
+ "messages": messages,
+ "max_tokens": max_tokens,
+ "temperature": temperature,
+ "stream": False,
+ },
+ timeout=360,
+ api_key=api_key,
+ )
+ return (((body.get("choices") or [{}])[0].get("message") or {}).get("content") or "").strip()
+
+
+def answer_llm_config() -> tuple[str, str, str]:
+ answer_base_url = clean_text(os.getenv("TMCRA_ANSWER_BASE_URL", ""))
+ answer_model = clean_text(os.getenv("TMCRA_ANSWER_MODEL", ""))
+ answer_api_key = clean_text(os.getenv("TMCRA_ANSWER_API_KEY", ""))
+ if answer_base_url or answer_model or answer_api_key:
+ return (
+ answer_base_url or os.getenv("GEMMA_BASE_URL", "http://127.0.0.1:18002/v1"),
+ answer_model or os.getenv("GEMMA_MODEL", os.getenv("TMCRA_GEMMA_MODEL_NAME", "gemma-4-e4b-it")),
+ answer_api_key or clean_text(os.getenv("OPENAI_API_KEY", "")),
+ )
+ return (
+ os.getenv("GEMMA_BASE_URL", "http://127.0.0.1:18002/v1"),
+ os.getenv("GEMMA_MODEL", os.getenv("TMCRA_GEMMA_MODEL_NAME", "gemma-4-e4b-it")),
+ clean_text(os.getenv("GEMMA_API_KEY", "")),
+ )
+
+
+def query_graph_llm_config() -> tuple[str, str, str]:
+ base_url = clean_text(os.getenv("TMCRA_QUERY_GRAPH_BASE_URL", ""))
+ model = clean_text(os.getenv("TMCRA_QUERY_GRAPH_MODEL", ""))
+ api_key = clean_text(os.getenv("TMCRA_QUERY_GRAPH_API_KEY", ""))
+ if base_url or model or api_key:
+ answer_base_url, answer_model, answer_api_key = answer_llm_config()
+ return (
+ base_url or answer_base_url,
+ model or answer_model,
+ api_key or answer_api_key,
+ )
+ return answer_llm_config()
+
+
+def clean_text(value: Any) -> str:
+ return re.sub(r"\s+", " ", str(value or "").strip())
+
+
+def truncate(value: Any, n: int = 500) -> str:
+ text = clean_text(value)
+ return text[:n]
+
+
+def session_text_chunks(
+ *,
+ session_id: str,
+ date: str,
+ turns: list[Mapping[str, Any]],
+ max_chars: int,
+ max_chunks: int,
+) -> list[str]:
+ chunks: list[str] = []
+ current: list[str] = [f"LongMemEval session_id={session_id} date={date}"]
+ current_len = len(current[0])
+ for index, turn in enumerate(turns, start=1):
+ role = clean_text(turn.get("role", "unknown"))
+ content = clean_text(turn.get("content", ""))
+ if not content:
+ continue
+ part = f"[{session_id} turn={index} role={role}] {content}"
+ if current_len + len(part) + 2 > max_chars and len(current) > 1:
+ chunks.append("\n".join(current))
+ if max_chunks > 0 and len(chunks) >= max_chunks:
+ return chunks
+ current = [f"LongMemEval session_id={session_id} date={date} continued=true"]
+ current_len = len(current[0])
+ current.append(part)
+ current_len += len(part) + 1
+ if len(current) > 1 and (max_chunks <= 0 or len(chunks) < max_chunks):
+ chunks.append("\n".join(current))
+ return chunks
+
+
+def select_session_indices(row: Mapping[str, Any], *, max_distractors: int) -> list[int]:
+ session_ids = [clean_text(item) for item in list(row.get("haystack_session_ids") or [])]
+ answer_ids = {clean_text(item) for item in list(row.get("answer_session_ids") or [])}
+ answer_indices = [index for index, sid in enumerate(session_ids) if sid in answer_ids]
+ distractors: list[int] = []
+ latest_candidates = list(range(max(0, len(session_ids) - 4), len(session_ids)))
+ first_candidates = list(range(min(3, len(session_ids))))
+ mid = len(session_ids) // 2
+ middle_candidates = [max(0, mid - 1), mid, min(len(session_ids) - 1, mid + 1)] if session_ids else []
+ for index in [*first_candidates, *middle_candidates, *latest_candidates]:
+ if index not in answer_indices and index not in distractors and 0 <= index < len(session_ids):
+ distractors.append(index)
+ if len(distractors) >= max_distractors:
+ break
+ return sorted(set(answer_indices + distractors))
+
+
+def select_official_full_history_indices(row: Mapping[str, Any]) -> list[int]:
+ """Official LongMemEval-style input: use every haystack session in file order."""
+ return list(range(len(list(row.get("haystack_sessions") or []))))
+
+
+def build_writer() -> Any:
+ from experiments.replacement.adapters.base import LLMProfile
+ from experiments.replacement.multi_layer_tmcra_writer import TMCRASuspectAnchoredTransformerMemoryWriter
+ from experiments.replacement.semantic_memory_writer import DeterministicMemoryWriteGate, OpenAICompatSemanticMemoryWriter
+
+ profile = LLMProfile(
+ name="longmemeval_native_writer",
+ model=os.getenv("TMCRA_WRITER_MODEL", "gemma-4-e4b-it"),
+ base_url=os.getenv("TMCRA_WRITER_BASE_URL", "http://127.0.0.1:18002/v1"),
+ api_key=os.getenv("TMCRA_WRITER_API_KEY", ""),
+ timeout_seconds=float(os.getenv("TMCRA_WRITER_TIMEOUT_SECONDS", "180")),
+ temperature=float(os.getenv("TMCRA_WRITER_TEMPERATURE", "0.0")),
+ max_tokens=int(os.getenv("TMCRA_WRITER_MAX_TOKENS", "512")),
+ )
+ base_writer = OpenAICompatSemanticMemoryWriter(
+ profile,
+ gate=DeterministicMemoryWriteGate(min_grounding_score=float(os.getenv("TMCRA_WRITER_MIN_GROUNDING_SCORE", "0.35"))),
+ max_proposals=int(os.getenv("TMCRA_WRITER_MAX_PROPOSALS", "2")),
+ )
+ return TMCRASuspectAnchoredTransformerMemoryWriter(
+ base_writer,
+ max_cells=int(os.getenv("TMCRA_WRITER_MAX_PROPOSALS", "2")),
+ value_char_budget=int(os.getenv("TMCRA_WRITER_VALUE_CHAR_BUDGET", "120")),
+ source_span_char_budget=int(os.getenv("TMCRA_WRITER_SOURCE_SPAN_CHAR_BUDGET", "180")),
+ transformer_layers=int(os.getenv("TMCRA_WRITER_TRANSFORMER_LAYERS", "2")),
+ state_attention_k=int(os.getenv("TMCRA_WRITER_STATE_ATTENTION_K", "16")),
+ suspect_threshold=float(os.getenv("TMCRA_WRITER_SUSPECT_THRESHOLD", "0.48")),
+ suspect_promote_count=int(os.getenv("TMCRA_WRITER_SUSPECT_PROMOTE_COUNT", "2")),
+ )
+
+
+def build_adapter(scope_id: str, storage_path: Path) -> Any:
+ from experiments.replacement.adapters.memory_adapters import GraphSessionMemoryAdapter
+
+ return GraphSessionMemoryAdapter(
+ auto_extract=False,
+ storage_backend="sqlite",
+ storage_path=str(storage_path),
+ scope_id=scope_id,
+ retrieval_mode=os.getenv("TMCRA_RETRIEVAL_MODE", "hybrid_node_scored"),
+ node_model_path=os.getenv("TMCRA_NODE_MODEL_PATH", ""),
+ path_model_path=os.getenv("TMCRA_PATH_MODEL_PATH", ""),
+ node_model_device=os.getenv("TMCRA_NODE_MODEL_DEVICE", "cpu"),
+ candidate_event_k=int(os.getenv("TMCRA_CANDIDATE_EVENT_K", "24")),
+ support_path_k=int(os.getenv("TMCRA_SUPPORT_PATH_K", "3")),
+ path_tunnel_rescue_k=int(os.getenv("TMCRA_PATH_TUNNEL_RESCUE_K", "2")),
+ path_tunnel_rescue_score_floor=float(os.getenv("TMCRA_PATH_TUNNEL_RESCUE_SCORE_FLOOR", "0.0")),
+ path_tunnel_rescue_min_age=int(os.getenv("TMCRA_PATH_TUNNEL_RESCUE_MIN_AGE", "0")),
+ path_tunnel_rescue_min_score_margin=float(os.getenv("TMCRA_PATH_TUNNEL_RESCUE_MIN_SCORE_MARGIN", "0.0")),
+ )
+
+
+def disable_topic_bucket_runtime() -> None:
+ import experiments.replacement.adapters.memory_adapters as memory_adapters
+
+ def no_topic_bucket(*args: Any, **kwargs: Any) -> dict[str, Any]:
+ return {}
+
+ def no_apply_topic_bucket(records: list[Any], topic_bucket: Mapping[str, Any]) -> None:
+ return None
+
+ def no_topic_edges(*args: Any, **kwargs: Any) -> dict[str, Any]:
+ return {
+ "topic_bridge_disabled": True,
+ "dialogue_tunnel_disabled": True,
+ "disabled_reason": "longmemeval_native_no_topic_bucket",
+ }
+
+ def no_topic_rerank(graph: Any, query: str, hits: list[Any], *, top_k: int) -> dict[str, Any]:
+ limit = max(1, int(top_k or 1))
+ return {
+ "hits": list(hits)[:limit],
+ "metadata": {
+ "topic_bucket_rerank_enabled": False,
+ "topic_bucket_disabled": True,
+ "topic_bucket_disable_reason": "longmemeval_native_no_topic_bucket",
+ "topic_bucket_candidate_count": len(list(hits)),
+ "topic_bucket_final_count": len(list(hits)[:limit]),
+ },
+ }
+
+ memory_adapters._assign_topic_bucket_for_text = no_topic_bucket
+ memory_adapters._apply_topic_bucket_to_records = no_apply_topic_bucket
+ memory_adapters._last_topic_turn = no_topic_bucket
+ memory_adapters._add_topic_bridge_edges = no_topic_edges
+ memory_adapters._add_dialogue_tunnel_edges = no_topic_edges
+ memory_adapters._topic_bucket_rerank_hits = no_topic_rerank
+
+
+def writer_ingest(adapter: Any, writer: Any, text: str, *, qid: str, chunk_id: str) -> dict[str, Any]:
+ from experiments.replacement.semantic_memory_writer import build_modelized_facet_unit_records
+
+ timestamp = datetime.now(timezone.utc).isoformat(timespec="seconds")
+ sidecar_hints = {
+ "metadata": {
+ "source": "longmemeval_native_flow",
+ "question_id": qid,
+ "chunk_id": chunk_id,
+ }
+ }
+ t0 = time.perf_counter()
+ proposals, writer_metadata = writer.propose_public_turn(
+ current_turn=text,
+ previous_turn="",
+ next_turn="",
+ speaker="user",
+ session_timestamp=timestamp,
+ sidecar_hints=sidecar_hints,
+ auxiliary_evidence_texts=[],
+ input_mode=os.getenv("TMCRA_WRITER_INPUT_MODE", "delta"),
+ )
+ graph = getattr(adapter, "graph", None)
+ turn_index = int(getattr(graph, "turn_index", 0) or 0) + 1
+ gate_result = writer.gate.build_payload(
+ proposals=proposals,
+ text=f"[{timestamp}] user: {text}",
+ raw_text=text,
+ speaker="user",
+ session_key=qid,
+ turn_index=turn_index,
+ timestamp=timestamp,
+ dia_id=f"longmemeval:{qid}:{chunk_id}",
+ sidecar_hints=sidecar_hints,
+ writer_metadata=writer_metadata,
+ )
+ payload = dict(gate_result.payload or {})
+ replacement_records = list(payload.get("replacement_memory_records", []) or [])
+ base_writer = getattr(writer, "base_writer", writer)
+ unit_records, unit_metadata = build_modelized_facet_unit_records(
+ base_writer,
+ current_turn_text=text,
+ parent_records=replacement_records,
+ speaker="user",
+ timestamp=timestamp,
+ turn_index=turn_index,
+ max_units=int(os.getenv("TMCRA_MODELIZED_UNIT_WRITER_MAX_UNITS", "14") or 14),
+ )
+ if unit_records:
+ replacement_records.extend(unit_records)
+ payload["replacement_memory_records"] = replacement_records
+ payload["modelized_facet_unit_writer"] = dict(unit_metadata or {})
+ adapter.ingest_turn(
+ f"[{timestamp}] user: {text}",
+ assistant_text="",
+ answer_payload=payload,
+ extraction_result={},
+ )
+ return {
+ "seconds": round(time.perf_counter() - t0, 3),
+ "proposal_count": len(proposals or []),
+ "accepted_count": int(getattr(gate_result, "accepted_count", 0) or 0),
+ "suspected_count": int(getattr(gate_result, "suspected_count", 0) or 0),
+ "record_count": len(payload.get("replacement_memory_records", []) or []),
+ "unit_record_count": len(unit_records),
+ "unit_writer_enabled": bool(unit_metadata.get("enabled", False)),
+ "unit_writer_metadata": dict(unit_metadata or {}),
+ }
+
+
+def retrieval_debug(retrieval: Any) -> dict[str, Any]:
+ metadata = dict(getattr(retrieval, "metadata", {}) or {})
+ hits = list(getattr(retrieval, "hits", []) or [])
+ return {
+ "hit_count": len(hits),
+ "retrieval_seconds": round(float(getattr(retrieval, "retrieval_seconds", 0.0) or 0.0), 4),
+ "retrieval_mode": metadata.get("retrieval_mode"),
+ "hybrid_enabled": metadata.get("hybrid_enabled"),
+ "decision_score_source": metadata.get("decision_score_source"),
+ "selected_event_count": len(metadata.get("selected_event_ids", []) or []),
+ "selected_path_count": len(metadata.get("selected_path_ids", []) or []),
+ "path_tunnel_enabled": metadata.get("path_tunnel_enabled"),
+ "path_tunnel_rescue_candidate_count": metadata.get("path_tunnel_rescue_candidate_count"),
+ "path_tunnel_rescue_path_count": len(metadata.get("path_tunnel_rescue_path_ids", []) or []),
+ "profile_focused_pack_enabled": metadata.get("profile_focused_pack_enabled"),
+ "profile_focused_pack_reason": metadata.get("profile_focused_pack_reason"),
+ "profile_focused_pack_hit_count": metadata.get("profile_focused_pack_hit_count"),
+ "profile_focused_pack_event_ids": list(metadata.get("profile_focused_pack_event_ids", []) or []),
+ "profile_focused_pack_memory_ids": list(metadata.get("profile_focused_pack_memory_ids", []) or []),
+ "profile_first_hybrid_enabled": metadata.get("profile_first_hybrid_enabled"),
+ "profile_first_event_ids": list(metadata.get("profile_first_event_ids", []) or []),
+ "profile_first_memory_ids": list(metadata.get("profile_first_memory_ids", []) or []),
+ "facet_query_pack_enabled": metadata.get("facet_query_pack_enabled"),
+ "facet_query_pack_inserted_hit_count": metadata.get("facet_query_pack_inserted_hit_count"),
+ "unit_coverage_pack_enabled": metadata.get("unit_coverage_pack_enabled"),
+ "unit_coverage_candidate_count": metadata.get("unit_coverage_candidate_count"),
+ "unit_coverage_selected_unit_count": metadata.get("unit_coverage_selected_unit_count"),
+ "unit_coverage_inserted_hit_count": metadata.get("unit_coverage_inserted_hit_count"),
+ "multi_unit_chain_slot_enabled": metadata.get("multi_unit_chain_slot_enabled"),
+ "multi_unit_chain_slot_formed": metadata.get("multi_unit_chain_slot_formed"),
+ "multi_unit_chain_slot_reason": metadata.get("multi_unit_chain_slot_reason"),
+ "multi_unit_chain_candidate_count": metadata.get("multi_unit_chain_candidate_count"),
+ "multi_unit_chain_selected_unit_count": metadata.get("multi_unit_chain_selected_unit_count"),
+ "multi_unit_chain_parent_count": metadata.get("multi_unit_chain_parent_count"),
+ "multi_unit_chain_inserted_hit_count": metadata.get("multi_unit_chain_inserted_hit_count"),
+ "multi_unit_chain_memory_ids": list(metadata.get("multi_unit_chain_memory_ids", []) or []),
+ "multi_unit_chain_focus_tokens": list(metadata.get("multi_unit_chain_focus_tokens", []) or []),
+ "top_values": [truncate(getattr(hit, "value", ""), 220) for hit in hits[:5]],
+ }
+
+
+STOPWORDS = {
+ "a",
+ "an",
+ "and",
+ "are",
+ "as",
+ "at",
+ "be",
+ "by",
+ "can",
+ "did",
+ "do",
+ "does",
+ "for",
+ "from",
+ "had",
+ "has",
+ "have",
+ "how",
+ "i",
+ "in",
+ "is",
+ "it",
+ "me",
+ "my",
+ "of",
+ "on",
+ "or",
+ "that",
+ "the",
+ "to",
+ "was",
+ "were",
+ "what",
+ "when",
+ "where",
+ "which",
+ "who",
+ "with",
+}
+
+_ANSWER_WINDOW_PLANNER_CACHE: dict[str, Any] = {}
+_EVIDENCE_UNIT_SELECTOR_CACHE: dict[str, Any] = {}
+_UNIFIED_OPERATION_PLANNER_CACHE: dict[str, Any] = {}
+
+
+def _vector_dot(left: list[float], right: list[float]) -> float:
+ if not left or not right:
+ return 0.0
+ total = 0.0
+ for a, b in zip(left, right):
+ try:
+ total += float(a) * float(b)
+ except (TypeError, ValueError):
+ continue
+ if total != total:
+ return 0.0
+ return max(0.0, min(1.0, float(total)))
+
+
+def apply_answer_window_semantic_features(question: str, evidence_windows: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ mode = os.getenv("TMCRA_ANSWER_WINDOW_PLANNER_SEMANTIC_MODE", "").strip().lower()
+ if mode in {"", "off", "false", "0", "disabled", "none"} or not evidence_windows:
+ return evidence_windows
+ if mode in {"auto", "on", "true", "1"}:
+ mode = os.getenv("TMCRA_EMBEDDER_INDEX_RECALL_MODE", "").strip().lower() or "bge_m3"
+ try:
+ from experiments.replacement.adapters import memory_adapters
+ except Exception:
+ return evidence_windows
+ texts = [clean_text(question), *[clean_text(item.get("text", "")) for item in evidence_windows]]
+ try:
+ vectors, metadata = memory_adapters._embedder_dense_vectors_for_texts(texts, mode=mode)
+ except Exception:
+ return evidence_windows
+ if not vectors or not vectors[0]:
+ return evidence_windows
+ query_vector = vectors[0]
+ enriched: list[dict[str, Any]] = []
+ dense_enabled = bool(metadata.get("write_embedder_dense_enabled"))
+ for item, vector in zip(evidence_windows, vectors[1:]):
+ score = _vector_dot(query_vector, list(vector or []))
+ next_item = dict(item)
+ next_item["answer_window_semantic_similarity"] = round(score, 6)
+ next_item["answer_window_semantic_mode"] = mode
+ next_item["answer_window_semantic_enabled"] = bool(dense_enabled and score > 0.0)
+ enriched.append(next_item)
+ return enriched
+
+
+def evidence_unit_planner_config() -> tuple[str, str, str]:
+ # TMCRA_LAYER_TAG: answer-side LLM planner config; not graph-memory model core.
+ answer_base_url, answer_model, answer_api_key = answer_llm_config()
+ return (
+ clean_text(os.getenv("TMCRA_EVIDENCE_UNIT_PLANNER_BASE_URL", "")) or answer_base_url,
+ clean_text(os.getenv("TMCRA_EVIDENCE_UNIT_PLANNER_MODEL", "")) or answer_model,
+ clean_text(os.getenv("TMCRA_EVIDENCE_UNIT_PLANNER_API_KEY", "")) or answer_api_key,
+ )
+
+
+def token_stem(token: str) -> str:
+ token = token.lower()
+ for suffix in ("ing", "ed", "es", "s"):
+ if len(token) > len(suffix) + 3 and token.endswith(suffix):
+ return token[: -len(suffix)]
+ return token
+
+
+def content_tokens(text: str) -> list[str]:
+ return [
+ token_stem(item)
+ for item in re.findall(r"[a-zA-Z0-9$]+", text.lower())
+ if item and item not in STOPWORDS
+ ]
+
+
+def quoted_query_phrases(question: str) -> list[str]:
+ phrases: list[str] = []
+ seen: set[str] = set()
+ for match in re.finditer(r"""['"]([^'"]{2,120})['"]""", question):
+ phrase = clean_text(match.group(1))
+ key = phrase.lower()
+ if phrase and key not in seen:
+ phrases.append(phrase)
+ seen.add(key)
+ return phrases
+
+
+def strict_quoted_phrase_hits(text: str, phrases: list[str]) -> int:
+ hits = 0
+ for phrase in phrases:
+ pattern = re.compile(
+ r"(? bool:
+ text = question.lower()
+ return bool(re.search(r"\b(day|days|week|weeks|month|months|year|years|hour|hours|duration|how long|spent)\b", text))
+
+
+def temporal_event_anchor_text(text: str) -> bool:
+ return bool(
+ re.search(
+ r"\b(start|started|starting|begin|began|finished|finish|completed|complete|ended|end|stopped|since|until)\b",
+ text.lower(),
+ )
+ )
+
+
+def parent_dialogue_header(text: str) -> str:
+ value = clean_text(text)
+ if not value:
+ return ""
+ match = re.search(
+ r"\bLongMemEval\s+session_id=[A-Za-z0-9_.:-]+\s+date=\d{4}/\d{2}/\d{2}\s+\([^)]+\)\s+\d{2}:\d{2}",
+ value,
+ )
+ if match:
+ return clean_text(match.group(0))
+ match = re.search(
+ r"\[[0-9T:+-]{19,25}\]\s+user:\s+LongMemEval\s+session_id=[A-Za-z0-9_.:-]+\s+date=\d{4}/\d{2}/\d{2}\s+\([^)]+\)\s+\d{2}:\d{2}",
+ value,
+ )
+ return clean_text(match.group(0)) if match else ""
+
+
+def attach_parent_temporal_context(unit: str, parent_text: str) -> str:
+ unit_text = clean_text(unit)
+ if not unit_text:
+ return ""
+ if re.search(r"\bdate=\d{4}/\d{2}/\d{2}\b", unit_text) or re.search(r"\bLongMemEval\s+session_id=", unit_text):
+ return unit_text
+ header = parent_dialogue_header(parent_text)
+ if not header:
+ return unit_text
+ return clean_text(f"{header} {unit_text}")
+
+
+# TMCRA_LAYER_TAG: rule-based evidence query expansion; candidate for modelized retrieval/intent replacement.
+EVIDENCE_QUERY_GENERIC_TERMS = {
+ "complement",
+ "current",
+ "recommend",
+ "recommendation",
+ "recommendations",
+ "suggest",
+ "suggestion",
+ "suggestions",
+}
+
+
+# TMCRA_LAYER_TAG: manual semantic alias buckets; not part of trained graph-memory model core.
+EVIDENCE_QUERY_ALIAS_GROUPS = (
+ {
+ "accessory",
+ "accessories",
+ "bag",
+ "camera",
+ "cameras",
+ "equipment",
+ "flash",
+ "gear",
+ "lens",
+ "lenses",
+ "photo",
+ "photography",
+ "sony",
+ "tripod",
+ },
+ {
+ "app",
+ "apps",
+ "dashboard",
+ "interface",
+ "layout",
+ "panel",
+ "software",
+ "tool",
+ "tools",
+ "ui",
+ "workflow",
+ },
+ {
+ "diet",
+ "drink",
+ "food",
+ "meal",
+ "restaurant",
+ "snack",
+ "taste",
+ },
+)
+
+
+def evidence_query_terms(question: str) -> set[str]:
+ terms = {term for term in content_tokens(question) if term not in EVIDENCE_QUERY_GENERIC_TERMS}
+ expanded = set(terms)
+ for group in EVIDENCE_QUERY_ALIAS_GROUPS:
+ if terms & group:
+ expanded.update(group)
+ return expanded
+
+
+def turn_units(text: str) -> list[str]:
+ text = clean_text(text)
+ if not text:
+ return []
+ marker = re.compile(r"(\[[^\]]+\bturn=\d+\s+role=(?:user|assistant)\])", re.IGNORECASE)
+ parts = marker.split(text)
+ units: list[str] = []
+ prefix = ""
+ index = 0
+ sentence_mode = os.getenv("TMCRA_DIALOGUE_TURN_SENTENCE_UNITS", "").strip().lower() not in {
+ "",
+ "0",
+ "false",
+ "off",
+ "no",
+ "disabled",
+ "none",
+ }
+ if parts and not marker.match(parts[0]):
+ prefix = parts[0].strip()
+ index = 1
+ while index < len(parts):
+ head = parts[index].strip()
+ body = parts[index + 1].strip() if index + 1 < len(parts) else ""
+ body_units = [body]
+ if sentence_mode:
+ split_units = [clean_text(item) for item in re.split(r"(?<=[.!?])\s+", body) if clean_text(item)]
+ if split_units:
+ body_units = split_units
+ for body_unit in body_units:
+ unit = clean_text(f"{head} {body_unit}")
+ if unit:
+ units.append(unit)
+ index += 2
+ if units:
+ if prefix:
+ units[0] = clean_text(f"{prefix} {units[0]}")
+ return units
+ sentence_units = re.split(r"(?<=[.!?])\s+", text)
+ return [clean_text(item) for item in sentence_units if clean_text(item)]
+
+
+def score_evidence_unit(question_terms: set[str], question: str, unit: str) -> float:
+ unit_norm = unit.lower()
+ unit_terms = set(content_tokens(unit))
+ overlap = question_terms & unit_terms
+ score = float(len(overlap))
+ for term in overlap:
+ if len(term) >= 5:
+ score += 0.35
+ raw_terms = [
+ item
+ for item in re.findall(r"[a-zA-Z0-9$]+", question.lower())
+ if item not in STOPWORDS and item not in EVIDENCE_QUERY_GENERIC_TERMS
+ ]
+ for size in (3, 2):
+ for index in range(0, max(0, len(raw_terms) - size + 1)):
+ phrase = " ".join(raw_terms[index : index + size])
+ if phrase and phrase in unit_norm:
+ score += 1.5 if size == 3 else 1.0
+ if re.search(r"\b\d+\b|\$\d+", unit_norm):
+ score += 0.3
+ if "role=user" in unit_norm:
+ score += 0.25
+ return score
+
+
+def centered_excerpt(unit: str, question_terms: set[str], *, max_chars: int) -> str:
+ unit = clean_text(unit)
+ if len(unit) <= max_chars:
+ return unit
+ unit_lower = unit.lower()
+ positions = [
+ unit_lower.find(term)
+ for term in sorted(question_terms, key=len, reverse=True)
+ if len(term) >= 4 and unit_lower.find(term) >= 0
+ ]
+ center = min(positions) if positions else 0
+ start = max(0, center - max_chars // 3)
+ end = min(len(unit), start + max_chars)
+ start = max(0, end - max_chars)
+ prefix = "..." if start > 0 else ""
+ suffix = "..." if end < len(unit) else ""
+ return prefix + unit[start:end].strip() + suffix
+
+
+def make_centered_window(
+ units: list[str],
+ seed_index: int,
+ score: float,
+ question_terms: set[str],
+ *,
+ max_chars: int,
+) -> dict[str, Any]:
+ seed_unit = units[seed_index].lower() if 0 <= seed_index < len(units) else ""
+ is_dialogue_question_seed = "role=user" in seed_unit and ("?" in seed_unit or "how many" in seed_unit)
+ if is_dialogue_question_seed:
+ max_chars = max(max_chars, 2200)
+ candidate_order = [seed_index, seed_index + 1, seed_index + 2, seed_index - 1, seed_index - 2]
+ order = [index for index in candidate_order if 0 <= index < len(units)]
+ else:
+ order = [seed_index]
+ for radius in (1, 2):
+ left = seed_index - radius
+ right = seed_index + radius
+ if left >= 0:
+ order.append(left)
+ if right < len(units):
+ order.append(right)
+
+ selected: list[tuple[int, str]] = []
+ used = 0
+ for index in order:
+ unit_l = units[index].lower() if 0 <= index < len(units) else ""
+ if "role=assistant" in unit_l:
+ budget = 1800 if index == seed_index else 900
+ else:
+ budget = 950 if index == seed_index else 350 if is_dialogue_question_seed else 520
+ piece = centered_excerpt(units[index], question_terms, max_chars=budget)
+ extra = len(piece) + 2
+ if selected and used + extra > max_chars:
+ continue
+ selected.append((index, piece))
+ used += extra
+ selected.sort(key=lambda item: item[0])
+ return {
+ "score": round(float(score), 3),
+ "unit_indexes": [index for index, _ in selected],
+ "text": "\n".join(piece for _, piece in selected),
+ }
+
+
+def assistant_answer_candidate_windows(
+ value: str,
+ question: str,
+ question_terms: set[str],
+ *,
+ max_windows: int = 4,
+ max_chars: int = 900,
+) -> list[dict[str, Any]]:
+ if not assistant_memory_query(question) or not assistant_origin_evidence(value):
+ return []
+ marker = re.compile(r"(\[[^\]]+\bturn=\d+\s+role=assistant\])", re.IGNORECASE)
+ parts = marker.split(clean_text(value))
+ candidates: list[tuple[float, int, str]] = []
+ raw_question_terms = [
+ item
+ for item in re.findall(r"[a-zA-Z0-9$]+", clean_text(question).lower())
+ if item not in STOPWORDS and item not in EVIDENCE_QUERY_GENERIC_TERMS
+ ]
+ focus_phrases: set[str] = set()
+ for size in (4, 3, 2):
+ for raw_index in range(0, max(0, len(raw_question_terms) - size + 1)):
+ phrase = " ".join(raw_question_terms[raw_index : raw_index + size])
+ if phrase:
+ focus_phrases.add(phrase)
+ for index in range(1, len(parts), 2):
+ head = parts[index].strip()
+ body = parts[index + 1].strip() if index + 1 < len(parts) else ""
+ if not body:
+ continue
+ pieces = [
+ clean_text(item)
+ for item in re.split(r"(?=(?:\d+\.|[-*]\s+|[A-Z][A-Za-z '’-]{2,40}:))|(?<=[.!?])\s+", body)
+ if clean_text(item)
+ ]
+ for local_index, piece in enumerate(pieces):
+ if len(piece) < 24:
+ continue
+ full_piece = clean_text(f"{head} {piece}")
+ score = score_evidence_unit(question_terms, question, full_piece)
+ piece_terms = set(content_tokens(piece))
+ overlap = piece_terms & question_terms
+ named_entity_like = bool(
+ re.search(r"\b(?:The\s+)?[A-Z][A-Za-z'’-]+(?:\s+(?:of|at|and|the|de|del|la|el|[A-Z][A-Za-z'’-]+)){1,7}\b", piece)
+ or re.search(r"['\"][^'\"]{3,80}['\"]", piece)
+ )
+ if named_entity_like:
+ score += 1.25
+ if overlap and named_entity_like:
+ score += 2.0
+ if len(overlap) >= 2:
+ score += 1.0
+ piece_l = clean_text(piece).lower()
+ phrase_hits = sum(1 for phrase in focus_phrases if len(phrase) >= 8 and phrase in piece_l)
+ if phrase_hits:
+ score += min(4.0, 1.5 * phrase_hits)
+ if score < 2.0:
+ continue
+ text = centered_excerpt(full_piece, question_terms, max_chars=max_chars)
+ candidates.append((score, local_index, text))
+ candidates.sort(key=lambda item: (item[0], -item[1]), reverse=True)
+ windows: list[dict[str, Any]] = []
+ seen: set[str] = set()
+ for score, _, text in candidates:
+ key = text[:220].lower()
+ if key in seen:
+ continue
+ seen.add(key)
+ windows.append(
+ {
+ "score": round(float(score) + 2.0, 3),
+ "unit_indexes": [],
+ "text": text,
+ "assistant_answer_candidate": True,
+ "assistant_memory_query": True,
+ "assistant_origin_evidence": True,
+ "assistant_candidate_side_channel": True,
+ "evidence_role": "assistant_answer_candidate",
+ "planner_selected": True,
+ "planner_score": round(float(score), 3),
+ }
+ )
+ if len(windows) >= max_windows:
+ break
+ return windows
+
+
+def _load_evidence_unit_selector() -> tuple[Any, Any, Any, dict[str, float]] | None:
+ mode = os.getenv("TMCRA_EVIDENCE_UNIT_SELECTOR_MODE", "").strip().lower()
+ if mode in {"", "off", "false", "0", "disabled", "none"}:
+ return None
+ model_path = os.getenv("TMCRA_EVIDENCE_UNIT_SELECTOR_MODEL_PATH", "").strip()
+ if not model_path:
+ return None
+ cache_key = "|".join([model_path, os.getenv("TMCRA_EVIDENCE_UNIT_SELECTOR_DEVICE", "cpu")])
+ cached = _EVIDENCE_UNIT_SELECTOR_CACHE.get(cache_key)
+ if cached is not None:
+ return cached
+ try:
+ from experiments.replacement import injection_planner as planner
+ except Exception:
+ return None
+ torch_module = getattr(planner, "torch", None)
+ if torch_module is None:
+ return None
+ try:
+ device = torch_module.device(os.getenv("TMCRA_EVIDENCE_UNIT_SELECTOR_DEVICE", "cpu") or "cpu")
+ payload = torch_module.load(Path(model_path), map_location=device, weights_only=True)
+ config = planner.InjectionPlannerConfig.from_dict(dict(payload.get("config", {}) or {}))
+ model = planner.InjectionPlannerModel(config).to(device)
+ model.load_state_dict(dict(payload.get("state_dict", {}) or {}), strict=False)
+ model.eval()
+ thresholds = {
+ "selection_threshold": float(os.getenv("TMCRA_EVIDENCE_UNIT_SELECTOR_SELECTION_THRESHOLD", "0.45")),
+ "row_threshold": float(os.getenv("TMCRA_EVIDENCE_UNIT_SELECTOR_ROW_THRESHOLD", "0.35")),
+ "logic_threshold": float(os.getenv("TMCRA_EVIDENCE_UNIT_SELECTOR_LOGIC_THRESHOLD", "0.45")),
+ }
+ except Exception:
+ return None
+ loaded = (planner, torch_module, model, thresholds)
+ _EVIDENCE_UNIT_SELECTOR_CACHE[cache_key] = loaded
+ return loaded
+
+
+def model_selected_evidence_unit_indexes(
+ question: str,
+ units: list[str],
+ hit_metadata: Mapping[str, Any],
+ scored_units: list[tuple[float, int]],
+) -> list[tuple[float, int]]:
+ loaded = _load_evidence_unit_selector()
+ if loaded is None or not units:
+ return []
+ planner, torch_module, model, thresholds = loaded
+ max_units = max(4, int(os.getenv("TMCRA_EVIDENCE_UNIT_SELECTOR_MAX_UNITS", "14") or 14))
+ top_scored = sorted(scored_units, key=lambda item: item[0], reverse=True)[:max_units]
+ candidate_indexes = sorted({index for _, index in top_scored if 0 <= index < len(units)})
+ if not candidate_indexes:
+ return []
+ score_by_index = {index: score for score, index in scored_units}
+ max_unit_score = max([abs(float(score)) for score, _ in top_scored] + [1.0])
+ retrieval_score = max(
+ 0.0,
+ min(
+ 1.0,
+ max(
+ metadata_float_from_mapping(hit_metadata, "hybrid_score"),
+ metadata_float_from_mapping(hit_metadata, "recall_score"),
+ metadata_float_from_mapping(hit_metadata, "event_score"),
+ metadata_float_from_mapping(hit_metadata, "path_score"),
+ ),
+ ),
+ )
+ graph_score = max(
+ 0.0,
+ min(
+ 1.0,
+ max(
+ metadata_float_from_mapping(hit_metadata, "answer_plan_score"),
+ metadata_float_from_mapping(hit_metadata, "answer_plan_selected_score"),
+ metadata_float_from_mapping(hit_metadata, "answer_plan_adjusted_score"),
+ ),
+ ),
+ )
+ temporal_state = clean_text(hit_metadata.get("injection_planner_temporal_state", "")) or "irrelevant"
+ evidence_role = clean_text(hit_metadata.get("injection_planner_evidence_role", "")) or clean_text(
+ hit_metadata.get("evidence_role", "")
+ )
+ candidates = []
+ for rank, index in enumerate(candidate_indexes, start=1):
+ unit_score = float(score_by_index.get(index, 0.0) or 0.0)
+ candidates.append(
+ {
+ "id": f"unit_{index}",
+ "text": clean_text(units[index]),
+ "layer": "event",
+ "temporal_state": temporal_state if temporal_state in getattr(planner, "TEMPORAL_STATES", ()) else "irrelevant",
+ "logic_roles": ["evidence"],
+ "evidence_role": evidence_role if evidence_role in getattr(planner, "EVIDENCE_ROLES", ()) else "",
+ "retrieval_score": retrieval_score,
+ "graph_score": graph_score,
+ "tunnel_score": max(0.0, min(1.0, metadata_float_from_mapping(hit_metadata, "path_score"))),
+ "topic_similarity": max(0.0, min(1.0, unit_score / max_unit_score)),
+ "confidence": retrieval_score,
+ "rank_score": round(1.0 / float(rank), 6),
+ "branch_depth": 1,
+ }
+ )
+ if len(candidates) < 3:
+ return []
+ try:
+ row = {"id": "evidence_unit_selector", "query": question, "candidates": candidates, "gold": {}}
+ dataset = planner.InjectionPlannerDataset([row], model.config)
+ batch = planner.collate_injection_batch([dataset[0]])
+ device = next(model.parameters()).device
+ model_batch = {key: value.to(device) if hasattr(value, "to") else value for key, value in dict(batch).items()}
+ with torch_module.no_grad():
+ outputs = model(model_batch["features"], model_batch["valid_mask"])
+ row_score = float(torch_module.sigmoid(outputs["should_inject_logits"])[0].detach().cpu().item())
+ selection_scores = torch_module.sigmoid(outputs["selection_logits"])[0].detach().cpu().tolist()
+ role_indices = torch_module.argmax(outputs["evidence_role_logits"], dim=-1)[0].detach().cpu().tolist()
+ except Exception:
+ return []
+ if row_score < thresholds["row_threshold"]:
+ return []
+ selected: list[tuple[float, int]] = []
+ for local_index, unit_index in enumerate(candidate_indexes):
+ role = planner.EVIDENCE_ROLES[int(role_indices[local_index])]
+ if role in {"noise", "negative_evidence"}:
+ continue
+ score = float(selection_scores[local_index])
+ if score >= thresholds["selection_threshold"]:
+ selected.append((score, unit_index))
+ selected.sort(key=lambda item: item[0], reverse=True)
+ return selected[: max(1, int(os.getenv("TMCRA_EVIDENCE_UNIT_SELECTOR_TOP_K", "4") or 4))]
+
+
+def metadata_float_from_mapping(metadata: Mapping[str, Any], key: str, default: float = 0.0) -> float:
+ try:
+ return float(metadata.get(key, default) or default)
+ except (TypeError, ValueError):
+ return default
+
+
+def evidence_windows_for_hit(question: str, hit: Any, *, max_chars: int = 1800) -> list[dict[str, Any]]:
+ value = clean_text(getattr(hit, "value", ""))
+ if not value:
+ return []
+ hit_metadata = dict(getattr(hit, "metadata", {}) or {})
+ evidence_snippet_role = clean_text(hit_metadata.get("evidence_snippet_role", ""))
+ unit_coverage_parent_event = bool(hit_metadata.get("unit_coverage_pack")) and evidence_snippet_role == "unit_coverage_parent_event"
+ multi_chain_parent_event = bool(hit_metadata.get("multi_unit_chain_slot")) and evidence_snippet_role == "multi_unit_chain_parent_event"
+ if unit_coverage_parent_event or multi_chain_parent_event:
+ try:
+ max_chars = max(max_chars, int(os.getenv("TMCRA_UNIT_COVERAGE_PARENT_WINDOW_CHARS", "3800") or 3800))
+ except (TypeError, ValueError):
+ max_chars = max(max_chars, 3800)
+ if assistant_memory_query(question) and assistant_origin_evidence(value):
+ try:
+ max_chars = max(max_chars, int(os.getenv("TMCRA_ASSISTANT_MEMORY_WINDOW_CHARS", "4200") or 4200))
+ except (TypeError, ValueError):
+ max_chars = max(max_chars, 4200)
+ def metadata_float(key: str, default: float = 0.0) -> float:
+ try:
+ return float(hit_metadata.get(key, default) or default)
+ except (TypeError, ValueError):
+ return default
+
+ answer_plan_score = max(
+ metadata_float("answer_plan_score"),
+ metadata_float("answer_plan_selected_score"),
+ metadata_float("answer_plan_current_score"),
+ metadata_float("answer_plan_adjusted_score"),
+ )
+ answer_plan_selected = bool(hit_metadata.get("answer_plan_selected", False)) and answer_plan_score > 0.0
+ try:
+ answer_plan_rank = int(hit_metadata.get("answer_plan_rank", 0) or 0)
+ except (TypeError, ValueError):
+ answer_plan_rank = 0
+ parent_profile_summary = clean_text(
+ hit_metadata.get("profile_first_parent_summary", "")
+ or hit_metadata.get("profile_summary", "")
+ or hit_metadata.get("profile_value", "")
+ )
+ if hit_metadata.get("profile_first_source_support") and parent_profile_summary:
+ value = clean_text(f"[profile summary] {parent_profile_summary} {value}")
+ if unit_coverage_parent_event or multi_chain_parent_event:
+ child_value = clean_text(hit_metadata.get("unit_coverage_child_value", "") or hit_metadata.get("multi_unit_chain_child_value", ""))
+ child_source_span = clean_text(
+ hit_metadata.get("unit_coverage_child_source_span", "")
+ or hit_metadata.get("multi_unit_chain_child_source_span", "")
+ )
+ child_anchor = child_source_span or child_value
+ if child_anchor:
+ parent_l = value.lower()
+ child_l = child_anchor.lower()
+ pos = parent_l.find(child_l[: min(len(child_l), 120)])
+ if pos >= 0:
+ half = max_chars // 2
+ start = max(0, pos - half)
+ end = min(len(value), pos + len(child_anchor) + half)
+ value = clean_text(value[start:end])
+ elif child_value or child_source_span:
+ value = clean_text(value)
+ if child_value or child_source_span:
+ value = clean_text(
+ f"[unit evidence] {child_source_span or child_value}"
+ f"{' | unit_value=' + child_value if child_value and child_value != child_source_span else ''}\n"
+ f"[parent evidence] {value}"
+ )
+ units = turn_units(value)
+ if not units:
+ return []
+ question_terms = evidence_query_terms(question)
+ profile_bonus = 0.0
+ if hit_metadata.get("profile_first_hybrid_rescue"):
+ profile_bonus += 8.0
+ elif hit_metadata.get("profile_layer") or str(getattr(hit, "source_kind", "")).startswith("public_dialog_profile"):
+ profile_bonus += 4.0
+ scored = [
+ (score_evidence_unit(question_terms, question, unit) + profile_bonus, index)
+ for index, unit in enumerate(units)
+ ]
+ scored.sort(key=lambda item: item[0], reverse=True)
+ seed_indexes: list[tuple[float, int]] = []
+ model_unit_indexes = model_selected_evidence_unit_indexes(question, units, hit_metadata, scored)
+ for model_score, index in model_unit_indexes:
+ if any(existing == index for _, existing in seed_indexes):
+ continue
+ seed_indexes.append((max(float(model_score) * 10.0, float(scored[0][0]) if scored else 0.0), index))
+ seed_limit = 5 if assistant_memory_query(question) and assistant_origin_evidence(value) else 3
+ if len(seed_indexes) >= seed_limit:
+ break
+ for score, index in scored:
+ if score <= 0 and seed_indexes:
+ break
+ if score <= 0 and not seed_indexes:
+ seed_indexes.append((score, index))
+ break
+ if any(abs(index - existing) <= 1 for _, existing in seed_indexes):
+ continue
+ seed_indexes.append((score, index))
+ seed_limit = 5 if assistant_memory_query(question) and assistant_origin_evidence(value) else 3
+ if len(seed_indexes) >= seed_limit:
+ break
+ windows: list[dict[str, Any]] = []
+ for score, index in seed_indexes:
+ window = make_centered_window(units, index, score, question_terms, max_chars=max_chars)
+ window["memory_id"] = getattr(hit, "memory_id", "")
+ window["original_chars"] = len(value)
+ window["planner_selected"] = bool(hit_metadata.get("injection_planner_selected", False)) or answer_plan_selected
+ evidence_role = clean_text(hit_metadata.get("injection_planner_evidence_role", ""))
+ if not evidence_role and answer_plan_selected:
+ evidence_role = "direct_answer"
+ memory_id = clean_text(getattr(hit, "memory_id", ""))
+ source_kind = clean_text(getattr(hit, "source_kind", ""))
+ if (
+ not evidence_role
+ and (
+ bool(hit_metadata.get("profile_first_hybrid_rescue"))
+ or bool(hit_metadata.get("profile_protected_slot"))
+ or bool(hit_metadata.get("profile_layer"))
+ or ".subject." in memory_id
+ or source_kind.startswith("public_dialog_profile")
+ )
+ ):
+ evidence_role = "profile_fact"
+ window["evidence_role"] = evidence_role
+ window["temporal_state"] = clean_text(hit_metadata.get("injection_planner_temporal_state", ""))
+ promoted_answer_plan_score = answer_plan_score if answer_plan_selected else 0.0
+ window["planner_score"] = max(metadata_float("injection_planner_score"), promoted_answer_plan_score)
+ window["answer_plan_score"] = answer_plan_score
+ window["answer_plan_selected"] = answer_plan_selected
+ window["answer_plan_rank"] = answer_plan_rank
+ if hit_metadata.get("semantic_coverage_expansion"):
+ window["semantic_coverage_expansion"] = True
+ window["semantic_coverage_score"] = metadata_float("semantic_coverage_score")
+ window["semantic_coverage_source_memory_id"] = clean_text(hit_metadata.get("semantic_coverage_source_memory_id", ""))
+ if hit_metadata.get("unit_coverage_pack"):
+ window["unit_coverage_pack"] = True
+ window["unit_kind"] = clean_text(hit_metadata.get("unit_kind", ""))
+ window["facet_type"] = clean_text(hit_metadata.get("facet_type", ""))
+ window["unit_coverage_semantic_event_unit"] = bool(hit_metadata.get("unit_coverage_semantic_event_unit", False))
+ window["unit_coverage_semantic_event_priority"] = bool(hit_metadata.get("unit_coverage_semantic_event_priority", False))
+ window["unit_coverage_profile_shadow_unit"] = bool(hit_metadata.get("unit_coverage_profile_shadow_unit", False))
+ window["evidence_role"] = window["evidence_role"] or clean_text(hit_metadata.get("unit_kind", ""))
+ if hit_metadata.get("multi_unit_chain_slot"):
+ window["multi_unit_chain_slot"] = True
+ window["multi_unit_chain_bundle"] = bool(hit_metadata.get("multi_unit_chain_bundle", False))
+ window["multi_unit_chain_temporal_comparison"] = bool(hit_metadata.get("multi_unit_chain_temporal_comparison", False))
+ window["multi_unit_chain_semantic_event_unit"] = bool(hit_metadata.get("multi_unit_chain_semantic_event_unit", False))
+ window["multi_unit_chain_semantic_event_priority"] = bool(hit_metadata.get("multi_unit_chain_semantic_event_priority", False))
+ window["multi_unit_chain_profile_shadow_unit"] = bool(hit_metadata.get("multi_unit_chain_profile_shadow_unit", False))
+ window["unit_kind"] = window.get("unit_kind", "") or clean_text(hit_metadata.get("unit_kind", ""))
+ window["facet_type"] = window.get("facet_type", "") or clean_text(hit_metadata.get("facet_type", ""))
+ window["evidence_role"] = window["evidence_role"] or "multi_unit_chain"
+ window["multi_unit_chain_score"] = float(hit_metadata.get("multi_unit_chain_score", 0.0) or 0.0)
+ if hit_metadata.get("facet_query_pack"):
+ window["facet_query_pack"] = True
+ window["facet_type"] = window.get("facet_type", "") or clean_text(hit_metadata.get("facet_type", ""))
+ windows.append(window)
+ try:
+ assistant_candidate_limit = max(0, int(os.getenv("TMCRA_ASSISTANT_ANSWER_CANDIDATE_LIMIT", "0") or 0))
+ except (TypeError, ValueError):
+ assistant_candidate_limit = 4
+ if assistant_candidate_limit:
+ for candidate in assistant_answer_candidate_windows(
+ value,
+ question,
+ question_terms,
+ max_windows=assistant_candidate_limit,
+ max_chars=max(700, min(max_chars, 1200)),
+ ):
+ candidate["memory_id"] = getattr(hit, "memory_id", "")
+ candidate["original_chars"] = len(value)
+ candidate["temporal_state"] = clean_text(hit_metadata.get("injection_planner_temporal_state", ""))
+ candidate["answer_plan_score"] = answer_plan_score
+ candidate["answer_plan_selected"] = answer_plan_selected
+ candidate["answer_plan_rank"] = answer_plan_rank
+ windows.append(candidate)
+ return windows
+
+
+def build_answer_evidence(question: str, memory_hits: list[Any]) -> list[dict[str, Any]]:
+ windows: list[dict[str, Any]] = []
+ try:
+ hit_scan_limit = max(8, int(os.getenv("TMCRA_ANSWER_EVIDENCE_HIT_SCAN_LIMIT", "16") or 16))
+ build_limit = max(8, int(os.getenv("TMCRA_ANSWER_EVIDENCE_BUILD_LIMIT", "16") or 16))
+ except (TypeError, ValueError):
+ hit_scan_limit, build_limit = 16, 16
+ for hit in memory_hits[:hit_scan_limit]:
+ windows.extend(evidence_windows_for_hit(question, hit))
+ assistant_query = assistant_memory_query(question)
+ for index, item in enumerate(windows):
+ text = clean_text(item.get("text", ""))
+ if assistant_query and assistant_origin_evidence(text):
+ next_item = dict(item)
+ next_item["assistant_memory_query"] = True
+ next_item["assistant_origin_evidence"] = True
+ if not clean_text(next_item.get("evidence_role", "")):
+ next_item["evidence_role"] = "assistant_origin_detail"
+ windows[index] = next_item
+
+ def planner_window_key(item: dict[str, Any]) -> tuple[bool, bool, float, float, int]:
+ evidence_role = clean_text(item.get("evidence_role", ""))
+ role_allows_evidence = evidence_role not in {"noise", "negative_evidence"}
+ planner_score = float(item.get("planner_score", 0.0) or 0.0) if role_allows_evidence else 0.0
+ return (
+ bool(item.get("planner_selected", False)),
+ role_allows_evidence,
+ planner_score,
+ float(item.get("score", 0.0) or 0.0),
+ len(item.get("text", "")),
+ )
+
+ assistant_candidates = [item for item in windows if bool(item.get("assistant_candidate_side_channel", False))]
+ windows = [item for item in windows if not bool(item.get("assistant_candidate_side_channel", False))]
+
+ windows.sort(
+ key=planner_window_key,
+ reverse=True,
+ )
+ windows = compose_layered_evidence_windows(windows, build_limit=build_limit)
+ windows = diversify_evidence_windows(windows)
+ assistant_candidates.sort(key=planner_window_key, reverse=True)
+ try:
+ side_limit = max(0, int(os.getenv("TMCRA_ASSISTANT_ANSWER_CANDIDATE_SIDE_LIMIT", "12") or 12))
+ except (TypeError, ValueError):
+ side_limit = 12
+ return windows[:build_limit] + assistant_candidates[:side_limit]
+
+
+def assistant_memory_query(question: str) -> bool:
+ question_l = clean_text(question).lower()
+ return bool(
+ re.search(
+ r"\b(?:you said|you suggested|you recommended|you told|you wrote|you gave|you mentioned|"
+ r"previous chat|previous conversation|last time|remind me|we talked|we discussed|"
+ r"script you wrote|list you provided|what was the .* you said|what did you say)\b",
+ question_l,
+ )
+ )
+
+
+def assistant_origin_evidence(text: str) -> bool:
+ text_l = clean_text(text).lower()
+ if not text_l:
+ return False
+ if re.search(r"\[[^\]]+\brole=assistant\]", text_l):
+ return True
+ return bool(re.search(r"\b(?:assistant\]|role=assistant|assistant:)\b", text_l))
+
+
+def evidence_window_layer(item: dict[str, Any]) -> str:
+ memory_id = clean_text(item.get("memory_id", ""))
+ evidence_role = clean_text(item.get("evidence_role", "")).lower()
+ text = clean_text(item.get("text", "")).lower()
+ if bool(item.get("assistant_origin_evidence", False)) or evidence_role == "assistant_origin_detail":
+ return "assistant"
+ if bool(item.get("multi_unit_chain_slot", False)):
+ return "chain"
+ if bool(item.get("unit_coverage_pack", False)):
+ return "unit"
+ if bool(item.get("facet_query_pack", False)):
+ return "facet"
+ if (
+ "public_dialog_profile" in memory_id
+ or ".subject." in memory_id
+ or "profile" in evidence_role
+ or text.startswith("[profile summary]")
+ ):
+ return "profile"
+ if bool(item.get("answer_plan_selected", False)) or evidence_role in {"direct_answer", "positive", "supporting_fact"}:
+ return "direct"
+ return "base"
+
+
+def compose_layered_evidence_windows(evidence_windows: list[dict[str, Any]], *, build_limit: int) -> list[dict[str, Any]]:
+ mode = os.getenv("TMCRA_LAYERED_EVIDENCE_COMPOSE_MODE", "on").strip().lower()
+ if mode in {"", "0", "false", "off", "no", "disabled", "none"}:
+ return evidence_windows
+ try:
+ profile_limit = max(0, int(os.getenv("TMCRA_LAYERED_EVIDENCE_PROFILE_LIMIT", "3") or 3))
+ assistant_limit = max(0, int(os.getenv("TMCRA_LAYERED_EVIDENCE_ASSISTANT_LIMIT", "4") or 4))
+ direct_limit = max(0, int(os.getenv("TMCRA_LAYERED_EVIDENCE_DIRECT_LIMIT", "4") or 4))
+ base_limit = max(0, int(os.getenv("TMCRA_LAYERED_EVIDENCE_BASE_LIMIT", "5") or 5))
+ unit_limit = max(0, int(os.getenv("TMCRA_LAYERED_EVIDENCE_UNIT_LIMIT", "4") or 4))
+ chain_limit = max(0, int(os.getenv("TMCRA_LAYERED_EVIDENCE_CHAIN_LIMIT", "4") or 4))
+ facet_limit = max(0, int(os.getenv("TMCRA_LAYERED_EVIDENCE_FACET_LIMIT", "2") or 2))
+ except (TypeError, ValueError):
+ profile_limit, assistant_limit, direct_limit, base_limit, unit_limit, chain_limit, facet_limit = 3, 4, 4, 5, 4, 4, 2
+ buckets: dict[str, list[dict[str, Any]]] = {
+ "profile": [],
+ "assistant": [],
+ "direct": [],
+ "base": [],
+ "unit": [],
+ "chain": [],
+ "facet": [],
+ }
+ for item in evidence_windows:
+ buckets.setdefault(evidence_window_layer(item), buckets["base"]).append(item)
+
+ def coverage_bucket_key(item: dict[str, Any]) -> tuple[int, int, int, int, float, float, int]:
+ unit_kind = clean_text(item.get("unit_kind", "")).lower()
+ evidence_role = clean_text(item.get("evidence_role", "")).lower()
+ memory_id = clean_text(item.get("memory_id", "")).lower()
+ semantic_event = int(
+ bool(item.get("multi_unit_chain_semantic_event_priority", False))
+ or bool(item.get("unit_coverage_semantic_event_priority", False))
+ )
+ profile_shadow = int(
+ unit_kind == "profile_shadow_unit"
+ or bool(item.get("multi_unit_chain_profile_shadow_unit", False))
+ or bool(item.get("unit_coverage_profile_shadow_unit", False))
+ or "profile_shadow" in memory_id
+ )
+ selected = int(bool(item.get("planner_selected", False)) or bool(item.get("evidence_unit_planner_selected", False)))
+ parent_event = int("parent_event" in evidence_role or "#multi_parent:" in memory_id or "#unit_parent:" in memory_id)
+ return (
+ semantic_event,
+ selected,
+ profile_shadow,
+ parent_event,
+ float(item.get("multi_unit_chain_score", 0.0) or 0.0),
+ float(item.get("score", 0.0) or 0.0),
+ len(clean_text(item.get("text", ""))),
+ )
+
+ buckets["chain"].sort(key=coverage_bucket_key, reverse=True)
+ buckets["unit"].sort(key=coverage_bucket_key, reverse=True)
+
+ ordered: list[dict[str, Any]] = []
+ seen_keys: set[str] = set()
+
+ def add_from(items: list[dict[str, Any]], limit: int) -> None:
+ if limit <= 0:
+ return
+ added = 0
+ for item in items:
+ memory_id = clean_text(item.get("memory_id", ""))
+ text_key = clean_text(item.get("text", ""))[:240].lower()
+ key = memory_id or text_key
+ if key and key in seen_keys:
+ continue
+ if key:
+ seen_keys.add(key)
+ ordered.append(item)
+ added += 1
+ if len(ordered) >= build_limit or added >= limit:
+ break
+
+ # Stable evidence composition: profile/direct/base keep the normal answer surface;
+ # unit and chain provide bounded coverage slots instead of competing for all top ranks.
+ temporal_chain_first = any(bool(item.get("multi_unit_chain_temporal_comparison", False)) for item in buckets["chain"])
+ add_from(buckets["profile"], profile_limit)
+ add_from(buckets["assistant"], assistant_limit)
+ add_from(buckets["direct"], direct_limit)
+ if temporal_chain_first:
+ add_from(buckets["chain"], chain_limit)
+ add_from(buckets["base"], base_limit)
+ add_from(buckets["unit"], unit_limit)
+ else:
+ add_from(buckets["base"], base_limit)
+ add_from(buckets["unit"], unit_limit)
+ add_from(buckets["chain"], chain_limit)
+ add_from(buckets["facet"], facet_limit)
+ for item in evidence_windows:
+ if len(ordered) >= build_limit:
+ break
+ memory_id = clean_text(item.get("memory_id", ""))
+ text_key = clean_text(item.get("text", ""))[:240].lower()
+ key = memory_id or text_key
+ if key and key in seen_keys:
+ continue
+ if key:
+ seen_keys.add(key)
+ ordered.append(item)
+ return ordered
+
+
+def expand_dialogue_chain_hits(question: str, memory_hits: list[Any], graph: Any | None) -> list[Any]:
+ if graph is None or not memory_hits:
+ return list(memory_hits)
+ mode = os.getenv("TMCRA_DIALOGUE_CHAIN_EXPANSION_MODE", "adjacent").strip().lower()
+ if mode in {"", "off", "false", "0", "disabled", "none"}:
+ return list(memory_hits)
+ try:
+ radius = max(0, int(os.getenv("TMCRA_DIALOGUE_CHAIN_EXPANSION_RADIUS", "3") or 3))
+ max_total = max(0, int(os.getenv("TMCRA_DIALOGUE_CHAIN_EXPANSION_MAX_TOTAL", "5") or 5))
+ max_per_seed = max(1, int(os.getenv("TMCRA_DIALOGUE_CHAIN_EXPANSION_MAX_PER_SEED", "2") or 2))
+ except (TypeError, ValueError):
+ radius, max_total, max_per_seed = 3, 5, 2
+ if radius <= 0 or max_total <= 0:
+ return list(memory_hits)
+ question_terms = evidence_query_terms(question)
+ if not question_terms:
+ return list(memory_hits)
+ records_by_id = getattr(graph, "records_by_id", {}) or {}
+ if not records_by_id:
+ return list(memory_hits)
+ existing_ids = {clean_text(getattr(hit, "memory_id", "")) for hit in memory_hits if clean_text(getattr(hit, "memory_id", ""))}
+ seed_infos = []
+ for seed_index, hit in enumerate(memory_hits[:12]):
+ session_id = dialogue_session_id_from_text(hit_text_for_chain(hit))
+ if not session_id:
+ continue
+ seed_infos.append((seed_index, session_id, int(getattr(hit, "turn_index", 0) or 0), clean_text(getattr(hit, "memory_id", ""))))
+ if not seed_infos:
+ return list(memory_hits)
+
+ candidates_by_seed: dict[str, list[Any]] = {}
+ for record in records_by_id.values():
+ memory_id = clean_text(getattr(record, "memory_id", ""))
+ if not memory_id or memory_id in existing_ids:
+ continue
+ state = clean_text(getattr(record, "state", "active")).lower()
+ if state not in {"active", "parallel_active", "evidence"}:
+ continue
+ value = clean_text(getattr(record, "value", ""))
+ metadata = dict(getattr(record, "metadata", {}) or {})
+ session_id = dialogue_session_id_from_text(" ".join([value, clean_text(metadata.get("raw_text", "")), clean_text(metadata.get("source_turn_text", ""))]))
+ if not session_id:
+ continue
+ record_turn = int(getattr(record, "turn_index", 0) or 0)
+ for seed_index, seed_session, seed_turn, seed_id in seed_infos:
+ if session_id != seed_session:
+ continue
+ distance = abs(record_turn - seed_turn) if seed_turn and record_turn else 0
+ if distance > radius:
+ continue
+ score_metadata = {**metadata, "source_kind": clean_text(getattr(record, "source_kind", ""))}
+ score = dialogue_chain_candidate_score(question, question_terms, value, score_metadata, distance)
+ if score <= 0:
+ continue
+ hit = record_to_chain_hit(record, score=score, seed_id=seed_id, distance=distance, session_id=session_id)
+ candidates_by_seed.setdefault(seed_id, []).append(hit)
+
+ expanded: list[Any] = []
+ added_ids: set[str] = set()
+ total_added = 0
+ for hit_index, hit in enumerate(memory_hits):
+ expanded.append(hit)
+ seed_id = clean_text(getattr(hit, "memory_id", ""))
+ candidates = candidates_by_seed.get(seed_id, [])
+ candidates.sort(
+ key=lambda item: (
+ -float(getattr(item, "score", 0.0) or 0.0),
+ int((getattr(item, "metadata", {}) or {}).get("dialogue_chain_distance", 99) or 99),
+ int(getattr(item, "turn_index", 0) or 0),
+ clean_text(getattr(item, "memory_id", "")),
+ )
+ )
+ per_seed_added = 0
+ for candidate in candidates:
+ candidate_id = clean_text(getattr(candidate, "memory_id", ""))
+ if not candidate_id or candidate_id in existing_ids or candidate_id in added_ids:
+ continue
+ expanded.append(candidate)
+ added_ids.add(candidate_id)
+ total_added += 1
+ per_seed_added += 1
+ if per_seed_added >= max_per_seed or total_added >= max_total:
+ break
+ if total_added >= max_total:
+ expanded.extend(memory_hits[hit_index + 1 :])
+ break
+ if not added_ids:
+ return list(memory_hits)
+ return dedupe_hits_by_memory_id(expanded)
+
+
+def expand_semantic_coverage_hits(question: str, memory_hits: list[Any], graph: Any | None) -> list[Any]:
+ mode = os.getenv("TMCRA_SEMANTIC_COVERAGE_EXPANSION_MODE", "").strip().lower()
+ if mode in {"", "off", "false", "0", "disabled", "none"}:
+ return list(memory_hits)
+ if graph is None:
+ return list(memory_hits)
+ records_by_id = getattr(graph, "records_by_id", {}) or {}
+ if not records_by_id:
+ return list(memory_hits)
+ embedder_mode = os.getenv("TMCRA_SEMANTIC_COVERAGE_EMBEDDER_MODE", "").strip() or os.getenv(
+ "TMCRA_EMBEDDER_INDEX_RECALL_MODE", ""
+ ).strip() or "bge_m3"
+ try:
+ max_records = max(16, int(os.getenv("TMCRA_SEMANTIC_COVERAGE_MAX_RECORDS", "160") or 160))
+ max_units = max(24, int(os.getenv("TMCRA_SEMANTIC_COVERAGE_MAX_UNITS", "260") or 260))
+ max_add = max(1, int(os.getenv("TMCRA_SEMANTIC_COVERAGE_MAX_ADD", "16") or 16))
+ score_floor = float(os.getenv("TMCRA_SEMANTIC_COVERAGE_SCORE_FLOOR", "0.74") or 0.74)
+ except (TypeError, ValueError):
+ max_records, max_units, max_add, score_floor = 160, 260, 16, 0.74
+ try:
+ from experiments.replacement.adapters import memory_adapters
+ except Exception:
+ return list(memory_hits)
+ existing_unit_texts = {clean_text(getattr(hit, "value", "")) for hit in memory_hits if clean_text(getattr(hit, "value", ""))}
+ candidates: list[tuple[Any, int, str]] = []
+ for record in list(records_by_id.values())[:max_records]:
+ state = clean_text(getattr(record, "state", "active")).lower()
+ if state not in {"active", "parallel_active", "evidence"}:
+ continue
+ value = clean_text(getattr(record, "value", ""))
+ if not value:
+ continue
+ for unit_index, unit in enumerate((turn_units(value) or [value])[:8]):
+ text = clean_text(unit)
+ if not text or text in existing_unit_texts:
+ continue
+ candidates.append((record, unit_index, text))
+ if len(candidates) >= max_units:
+ break
+ if len(candidates) >= max_units:
+ break
+ if not candidates:
+ return list(memory_hits)
+ texts = [clean_text(question), *[text for _, _, text in candidates]]
+ try:
+ vectors, metadata = memory_adapters._embedder_dense_vectors_for_texts(texts, mode=embedder_mode)
+ except Exception:
+ return list(memory_hits)
+ if not vectors or not vectors[0] or not bool(metadata.get("write_embedder_dense_enabled", False)):
+ return list(memory_hits)
+ query_vector = list(vectors[0] or [])
+ query_phrases = quoted_query_phrases(question)
+ query_needs_temporal_anchors = bool(query_phrases and temporal_duration_query(question))
+ scored: list[tuple[float, str, Any, int, str]] = []
+ for (record, unit_index, text), vector in zip(candidates, vectors[1:]):
+ if query_needs_temporal_anchors:
+ text = attach_parent_temporal_context(text, clean_text(getattr(record, "value", "")))
+ score = _vector_dot(query_vector, list(vector or []))
+ phrase_hits = strict_quoted_phrase_hits(text, query_phrases)
+ if query_phrases and phrase_hits <= 0:
+ continue
+ if query_needs_temporal_anchors and not temporal_event_anchor_text(text):
+ continue
+ if phrase_hits:
+ score = min(1.0, score + (0.025 * min(3, phrase_hits)))
+ if score < score_floor:
+ continue
+ memory_id = clean_text(getattr(record, "memory_id", "")) or clean_text(getattr(record, "slot_key", ""))
+ scored.append((score, memory_id, record, unit_index, text))
+ if not scored:
+ return list(memory_hits)
+ scored.sort(key=lambda item: (-item[0], item[1], item[3]))
+ expanded = list(memory_hits)
+ added = 0
+ seen_semantic_ids: set[str] = set()
+ for score, memory_id, record, unit_index, text in scored:
+ semantic_id = f"{memory_id}#semantic_unit_{unit_index}"
+ if semantic_id in seen_semantic_ids:
+ continue
+ metadata = dict(getattr(record, "metadata", {}) or {})
+ metadata.update(
+ {
+ "semantic_coverage_expansion": True,
+ "semantic_coverage_score": round(float(score), 6),
+ "semantic_coverage_unit_index": int(unit_index),
+ "semantic_coverage_source_memory_id": memory_id,
+ "injection_planner_selected": True,
+ "injection_planner_score": round(float(score), 6),
+ "injection_planner_evidence_role": "positive",
+ "hybrid_score": max(float(score), metadata_float_from_mapping(metadata, "hybrid_score")),
+ "recall_score": max(float(score), metadata_float_from_mapping(metadata, "recall_score")),
+ }
+ )
+ expanded.append(
+ SimpleNamespace(
+ memory_id=semantic_id,
+ category=clean_text(getattr(record, "category", "")),
+ value=text,
+ relation=clean_text(getattr(record, "relation", "related_to")) or "related_to",
+ anchors=list(getattr(record, "anchor_concepts", []) or []),
+ score=float(score),
+ source_kind="semantic_coverage_unit",
+ slot_key=clean_text(getattr(record, "slot_key", "")),
+ state=clean_text(getattr(record, "state", "active")) or "active",
+ turn_index=int(getattr(record, "turn_index", 0) or 0),
+ metadata=metadata,
+ )
+ )
+ seen_semantic_ids.add(semantic_id)
+ added += 1
+ if added >= max_add:
+ break
+ return dedupe_hits_by_memory_id(expanded)
+
+
+def hit_text_for_chain(hit: Any) -> str:
+ metadata = dict(getattr(hit, "metadata", {}) or {})
+ return " ".join(
+ [
+ clean_text(getattr(hit, "value", "")),
+ clean_text(metadata.get("raw_text", "")),
+ clean_text(metadata.get("source_turn_text", "")),
+ clean_text(metadata.get("source_span", "")),
+ ]
+ )
+
+
+def dialogue_session_id_from_text(text: str) -> str:
+ value = clean_text(text)
+ if not value:
+ return ""
+ match = re.search(r"\bLongMemEval\s+session_id=([A-Za-z0-9_.:-]+)", value)
+ if match:
+ return match.group(1)
+ match = re.search(r"\[([A-Za-z0-9_.:-]+)\s+turn=\d+\s+role=", value)
+ return match.group(1) if match else ""
+
+
+def dialogue_chain_candidate_score(question: str, question_terms: set[str], value: str, metadata: Mapping[str, Any], distance: int) -> float:
+ text = " ".join([value, clean_text(metadata.get("source_span", "")), clean_text(metadata.get("event_text", ""))])
+ units = turn_units(text) or [text]
+ base = max(score_evidence_unit(question_terms, question, unit) for unit in units)
+ if base <= 0:
+ return 0.0
+ source_kind = clean_text(metadata.get("source_kind", ""))
+ if source_kind.startswith("public_dialog_") and source_kind != "public_dialog_turn":
+ base += 0.35
+ if clean_text(metadata.get("event_id", "")):
+ base += 0.2
+ return round(max(0.0, base - (0.15 * max(0, distance - 1))), 6)
+
+
+def record_to_chain_hit(record: Any, *, score: float, seed_id: str, distance: int, session_id: str) -> Any:
+ metadata = dict(getattr(record, "metadata", {}) or {})
+ metadata.update(
+ {
+ "dialogue_chain_expansion": True,
+ "dialogue_chain_seed_memory_id": seed_id,
+ "dialogue_chain_distance": int(distance),
+ "dialogue_chain_session_id": session_id,
+ }
+ )
+ return SimpleNamespace(
+ memory_id=clean_text(getattr(record, "memory_id", "")),
+ category=clean_text(getattr(record, "category", "")),
+ value=clean_text(getattr(record, "value", "")),
+ relation=clean_text(getattr(record, "relation", "related_to")) or "related_to",
+ anchors=list(getattr(record, "anchor_concepts", []) or []),
+ score=float(score),
+ source_kind=clean_text(getattr(record, "source_kind", "memory")) or "memory",
+ slot_key=clean_text(getattr(record, "slot_key", "")),
+ state=clean_text(getattr(record, "state", "active")) or "active",
+ turn_index=int(getattr(record, "turn_index", 0) or 0),
+ metadata=metadata,
+ )
+
+
+def dedupe_hits_by_memory_id(hits: list[Any]) -> list[Any]:
+ deduped: list[Any] = []
+ seen: set[str] = set()
+ for hit in hits:
+ memory_id = clean_text(getattr(hit, "memory_id", ""))
+ if memory_id and memory_id in seen:
+ continue
+ if memory_id:
+ seen.add(memory_id)
+ deduped.append(hit)
+ return deduped
+
+
+def _load_answer_window_planner() -> tuple[Any, Any, Any, dict[str, float]] | None:
+ mode = os.getenv("TMCRA_ANSWER_WINDOW_PLANNER_MODE", "").strip().lower()
+ if mode in {"", "off", "false", "0", "disabled", "none"}:
+ return None
+ model_path = (
+ os.getenv("TMCRA_ANSWER_WINDOW_PLANNER_MODEL_PATH", "").strip()
+ or os.getenv("TMCRA_INJECTION_PLANNER_MODEL_PATH", "").strip()
+ )
+ if not model_path:
+ return None
+ cache_key = "|".join(
+ [
+ model_path,
+ os.getenv("TMCRA_ANSWER_WINDOW_PLANNER_DEVICE", os.getenv("TMCRA_INJECTION_PLANNER_DEVICE", "cpu")),
+ ]
+ )
+ cached = _ANSWER_WINDOW_PLANNER_CACHE.get(cache_key)
+ if cached is not None:
+ return cached
+ try:
+ from experiments.replacement import injection_planner as planner
+ except Exception:
+ return None
+ torch_module = getattr(planner, "torch", None)
+ if torch_module is None:
+ return None
+ try:
+ device = torch_module.device(os.getenv("TMCRA_ANSWER_WINDOW_PLANNER_DEVICE", os.getenv("TMCRA_INJECTION_PLANNER_DEVICE", "cpu")) or "cpu")
+ payload = torch_module.load(Path(model_path), map_location=device, weights_only=True)
+ config = planner.InjectionPlannerConfig.from_dict(dict(payload.get("config", {}) or {}))
+ model = planner.InjectionPlannerModel(config).to(device)
+ model.load_state_dict(dict(payload.get("state_dict", {}) or {}), strict=False)
+ model.eval()
+ thresholds = {
+ "selection_threshold": float(os.getenv("TMCRA_ANSWER_WINDOW_PLANNER_SELECTION_THRESHOLD", os.getenv("TMCRA_INJECTION_PLANNER_SELECTION_THRESHOLD", "0.3"))),
+ "row_threshold": float(os.getenv("TMCRA_ANSWER_WINDOW_PLANNER_ROW_THRESHOLD", os.getenv("TMCRA_INJECTION_PLANNER_ROW_THRESHOLD", "0.3"))),
+ "logic_threshold": float(os.getenv("TMCRA_ANSWER_WINDOW_PLANNER_LOGIC_THRESHOLD", os.getenv("TMCRA_INJECTION_PLANNER_LOGIC_THRESHOLD", "0.45"))),
+ }
+ except Exception:
+ return None
+ loaded = (planner, torch_module, model, thresholds)
+ _ANSWER_WINDOW_PLANNER_CACHE[cache_key] = loaded
+ return loaded
+
+
+def apply_answer_window_planner(question: str, evidence_windows: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ loaded = _load_answer_window_planner()
+ if loaded is None or not evidence_windows:
+ return evidence_windows
+ planner, torch_module, model, thresholds = loaded
+ evidence_windows = apply_answer_window_semantic_features(question, evidence_windows)
+ max_score = max([float(item.get("score", 0.0) or 0.0) for item in evidence_windows] + [1.0])
+ candidates = []
+ for index, item in enumerate(evidence_windows):
+ original_role = clean_text(item.get("evidence_role", ""))
+ role_is_noise = original_role in {"noise", "negative_evidence"}
+ semantic_similarity = max(
+ 0.0,
+ min(
+ 1.0,
+ float(item.get("answer_window_semantic_similarity", item.get("semantic_similarity", 0.0)) or 0.0),
+ ),
+ )
+ candidates.append(
+ {
+ "id": f"window_{index}",
+ "text": clean_text(item.get("text", "")),
+ "layer": "event",
+ "temporal_state": clean_text(item.get("temporal_state", "")) or "irrelevant",
+ "logic_roles": ["noise"] if role_is_noise else ["evidence"],
+ "retrieval_score": min(1.0, max(0.0, float(item.get("score", 0.0) or 0.0) / max_score)),
+ "graph_score": min(1.0, max(0.0, float(item.get("planner_score", 0.0) or 0.0))),
+ "tunnel_score": 1.0 if original_role == "bridge_context" else 0.0,
+ "topic_similarity": 0.0,
+ "semantic_similarity": semantic_similarity,
+ "confidence": min(1.0, max(0.0, float(item.get("score", 0.0) or 0.0) / max_score)),
+ "rank_score": round(1.0 / float(index + 1), 6),
+ "branch_depth": 1,
+ "evidence_role": original_role,
+ }
+ )
+ try:
+ row = {"id": "answer_window_plan", "query": question, "candidates": candidates, "gold": {}}
+ dataset = planner.InjectionPlannerDataset([row], model.config)
+ batch = planner.collate_injection_batch([dataset[0]])
+ device = next(model.parameters()).device
+ model_batch = {key: value.to(device) if hasattr(value, "to") else value for key, value in dict(batch).items()}
+ with torch_module.no_grad():
+ outputs = model(model_batch["features"], model_batch["valid_mask"])
+ should_inject_score = float(torch_module.sigmoid(outputs["should_inject_logits"])[0].detach().cpu().item())
+ selection_scores = torch_module.sigmoid(outputs["selection_logits"])[0].detach().cpu().tolist()
+ temporal_indices = torch_module.argmax(outputs["temporal_logits"], dim=-1)[0].detach().cpu().tolist()
+ role_indices = torch_module.argmax(outputs["evidence_role_logits"], dim=-1)[0].detach().cpu().tolist()
+ logic_scores = torch_module.sigmoid(outputs["logic_logits"])[0].detach().cpu().tolist()
+ except Exception:
+ return evidence_windows
+ row_allows = should_inject_score >= thresholds["row_threshold"]
+ planned: list[dict[str, Any]] = []
+ for index, item in enumerate(evidence_windows):
+ next_item = dict(item)
+ item_semantic_similarity = max(
+ 0.0,
+ min(
+ 1.0,
+ float(item.get("answer_window_semantic_similarity", item.get("semantic_similarity", 0.0)) or 0.0),
+ ),
+ )
+ role = planner.EVIDENCE_ROLES[int(role_indices[index])]
+ role_allows = role not in {"noise", "negative_evidence"}
+ selected = bool(row_allows and role_allows and float(selection_scores[index]) >= thresholds["selection_threshold"])
+ logic_roles = [
+ logic_role
+ for logic_role, score in zip(planner.LOGIC_ROLES, logic_scores[index])
+ if float(score) >= thresholds["logic_threshold"]
+ ]
+ next_item.update(
+ {
+ "hit_evidence_role": clean_text(item.get("evidence_role", "")),
+ "hit_temporal_state": clean_text(item.get("temporal_state", "")),
+ "planner_selected": selected,
+ "planner_score": round(float(selection_scores[index]), 6),
+ "semantic_similarity": round(float(item_semantic_similarity), 6),
+ "evidence_role": role,
+ "temporal_state": planner.TEMPORAL_STATES[int(temporal_indices[index])],
+ "window_planner_enabled": True,
+ "window_planner_should_inject_score": round(float(should_inject_score), 6),
+ "window_planner_logic_roles": logic_roles or ["evidence"],
+ }
+ )
+ planned.append(next_item)
+
+ def window_key(item: dict[str, Any]) -> tuple[bool, bool, float, float, int]:
+ evidence_role = clean_text(item.get("evidence_role", ""))
+ role_allows_evidence = evidence_role not in {"noise", "negative_evidence"}
+ planner_score = float(item.get("planner_score", 0.0) or 0.0) if role_allows_evidence else 0.0
+ return (
+ bool(item.get("planner_selected", False)),
+ role_allows_evidence,
+ planner_score,
+ float(item.get("score", 0.0) or 0.0),
+ len(item.get("text", "")),
+ )
+
+ planned.sort(key=window_key, reverse=True)
+ return planned
+
+
+def apply_llm_evidence_selector(question: str, evidence_windows: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ # TMCRA_LAYER_TAG: answer-side LLM evidence selector; post-retrieval rerank only, not recall.
+ mode = os.getenv("TMCRA_LLM_EVIDENCE_SELECTOR_MODE", "on").strip().lower()
+ if mode in {"", "off", "false", "0", "disabled", "none"} or not evidence_windows:
+ return evidence_windows
+ try:
+ max_candidates = max(1, int(os.getenv("TMCRA_LLM_EVIDENCE_SELECTOR_MAX_CANDIDATES", "12") or 12))
+ max_chars = max(200, int(os.getenv("TMCRA_LLM_EVIDENCE_SELECTOR_CHARS", "900") or 900))
+ except (TypeError, ValueError):
+ max_candidates, max_chars = 12, 900
+ candidates = list(evidence_windows[:max_candidates])
+ candidate_lines = []
+ for index, item in enumerate(candidates, start=1):
+ memory_id = clean_text(item.get("memory_id", ""))
+ text = truncate(item.get("text", ""), max_chars)
+ if not text:
+ continue
+ header = f"[{index}]"
+ if memory_id:
+ header += f" memory_id={memory_id}"
+ candidate_lines.append(f"{header}\n{text}")
+ if not candidate_lines:
+ return evidence_windows
+ messages = [
+ {
+ "role": "system",
+ "content": (
+ "You are a semantic evidence selector for a long-memory runtime. "
+ "Your job is only to select evidence windows that could help answer the current question. "
+ "Use semantic relevance and discourse continuity, not keyword lists or domain-specific rules. "
+ "Select every distinct useful clue, including indirect clues, continuation turns, and separate pieces that must be combined. "
+ "If two windows repeat the same clue, keep the clearest one. "
+ "If two windows contain different clues, keep both even if one looks weaker or appears later in the list. "
+ "Do not answer the question. "
+ "Return strict JSON only with keys: selected_indices, rejected_indices. "
+ "selected_indices must be 1-based integers from the candidate list."
+ ),
+ },
+ {
+ "role": "user",
+ "content": f"Question:\n{question}\n\nCandidate evidence windows:\n" + "\n\n".join(candidate_lines),
+ },
+ ]
+ try:
+ raw = chat_completion(
+ os.getenv("TMCRA_EVIDENCE_SELECTOR_BASE_URL", os.getenv("GEMMA_BASE_URL", "http://127.0.0.1:18002/v1")),
+ os.getenv("TMCRA_EVIDENCE_SELECTOR_MODEL", os.getenv("GEMMA_MODEL", os.getenv("TMCRA_GEMMA_MODEL_NAME", "gemma-4-e4b-it"))),
+ messages,
+ max_tokens=160,
+ temperature=0.0,
+ )
+ except Exception:
+ return evidence_windows
+ payload = parse_json_object(raw)
+ raw_indices = payload.get("selected_indices", []) if payload else []
+ if not isinstance(raw_indices, list):
+ return evidence_windows
+ selected_order: list[int] = []
+ for value in iter_selector_indices(raw_indices):
+ try:
+ index = int(value) - 1
+ except (TypeError, ValueError):
+ continue
+ if 0 <= index < len(candidates) and index not in selected_order:
+ selected_order.append(index)
+ if not selected_order:
+ return evidence_windows
+ keep_model_selected = os.getenv("TMCRA_LLM_EVIDENCE_SELECTOR_KEEP_MODEL_SELECTED", "1").strip().lower() not in {
+ "0",
+ "false",
+ "off",
+ "no",
+ }
+ model_selected_order: list[int] = []
+ if keep_model_selected:
+ for index, item in enumerate(candidates):
+ if bool(item.get("planner_selected", False)) or bool(item.get("answer_plan_selected", False)):
+ model_selected_order.append(index)
+ selected_order = list(dict.fromkeys([*selected_order, *model_selected_order]))
+ selected_set = set(selected_order)
+ selected_rank = {index: rank for rank, index in enumerate(selected_order)}
+ planned: list[tuple[int, int, dict[str, Any]]] = []
+ for index, item in enumerate(evidence_windows):
+ next_item = dict(item)
+ llm_selected = index in selected_set
+ next_item["llm_evidence_selected"] = llm_selected
+ next_item["llm_evidence_rank"] = selected_rank.get(index, 9999)
+ next_item["llm_evidence_selector_enabled"] = True
+ next_item["llm_evidence_model_kept"] = bool(index in model_selected_order)
+ group = 0 if llm_selected else 1
+ rank = selected_rank.get(index, index)
+ planned.append((group, rank, next_item))
+ planned.sort(key=lambda row: (row[0], row[1]))
+ return diversify_evidence_windows([item for _, _, item in planned])
+
+
+def apply_evidence_unit_planner(question: str, evidence_windows: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ # TMCRA_LAYER_TAG: answer-side LLM evidence-unit planner; normalizes retrieved windows after recall.
+ mode = os.getenv("TMCRA_EVIDENCE_UNIT_PLANNER_MODE", "on").strip().lower()
+ if mode in {"", "off", "false", "0", "disabled", "none"} or not evidence_windows:
+ return evidence_windows
+ if evidence_unit_plan_from_windows(evidence_windows):
+ return evidence_windows
+ plan = build_evidence_unit_plan(question, evidence_windows)
+ if not plan:
+ return evidence_windows
+ selected_ranks = evidence_unit_selected_source_ranks(plan)
+ reorder_mode = os.getenv("TMCRA_EVIDENCE_UNIT_PLANNER_REORDER", "0").strip().lower()
+ reorder_enabled = reorder_mode not in {"", "0", "false", "off", "no", "disabled", "none"}
+ planned: list[tuple[int, int, int, dict[str, Any]]] = []
+ for index, item in enumerate(evidence_windows):
+ next_item = dict(item)
+ if index == 0:
+ next_item["evidence_unit_plan"] = plan
+ source_index = index + 1
+ unit_rank = selected_ranks.get(source_index)
+ next_item["evidence_unit_planner_enabled"] = True
+ next_item["evidence_unit_planner_selected"] = unit_rank is not None
+ next_item["evidence_unit_planner_rank"] = unit_rank if unit_rank is not None else 9999
+ group = 0 if unit_rank is not None else 1
+ rank = unit_rank if unit_rank is not None else index
+ planned.append((group, rank, index, next_item))
+ if not reorder_enabled:
+ return [item for _, _, _, item in sorted(planned, key=lambda row: row[2])]
+ planned.sort(key=lambda row: (row[0], row[1], row[2]))
+ return diversify_evidence_windows([item for _, _, _, item in planned])
+
+
+def build_evidence_unit_plan(question: str, evidence_windows: list[dict[str, Any]]) -> dict[str, Any]:
+ try:
+ max_candidates = max(1, int(os.getenv("TMCRA_EVIDENCE_UNIT_PLANNER_MAX_CANDIDATES", "10") or 10))
+ max_chars = max(240, int(os.getenv("TMCRA_EVIDENCE_UNIT_PLANNER_CHARS", "1100") or 1100))
+ max_tokens = max(280, int(os.getenv("TMCRA_EVIDENCE_UNIT_PLANNER_MAX_TOKENS", "760") or 760))
+ except (TypeError, ValueError):
+ max_candidates, max_chars, max_tokens = 10, 1100, 760
+ candidate_lines = []
+ for index, item in enumerate(evidence_windows[:max_candidates], start=1):
+ text = truncate(item.get("text", ""), max_chars)
+ if not text:
+ continue
+ memory_id = clean_text(item.get("memory_id", ""))
+ role = clean_text(item.get("evidence_role", ""))
+ temporal_state = clean_text(item.get("temporal_state", ""))
+ label_bits = []
+ if memory_id:
+ label_bits.append(f"memory_id={memory_id}")
+ if role:
+ label_bits.append(f"role={role}")
+ if temporal_state:
+ label_bits.append(f"time={temporal_state}")
+ label = " ".join(label_bits)
+ candidate_lines.append(f"[{index}] {label}\n{text}".strip())
+ if not candidate_lines:
+ return {}
+ messages = [
+ {
+ "role": "system",
+ "content": (
+ "You are a model-based evidence-unit planner for a long-memory runtime. "
+ "Do not answer from prior knowledge. Do not use domain-specific keyword rules. "
+ "Read the current question and the retrieved evidence windows, then normalize the evidence into answer units that a final answer model can reason over. "
+ "Your job is to expose the operation needed by the question: direct fact lookup, distinct-item counting, temporal difference, temporal disambiguation, current/latest value selection, negative/absence judgment, preference synthesis, or multi-evidence synthesis. "
+ "For count questions, infer the answer unit from the question and create one unit per distinct evidence-backed candidate. "
+ "When the question requests multiple actions joined by and/or, keep separate units for separate action-target instances, especially original-versus-replacement or return-versus-pick-up instances. "
+ "Do not collapse a stated pending need, intention, or obligation into mere context just because the same sentence later mentions an exchange, replacement, reschedule, or follow-up action. "
+ "Treat the original action and the replacement/follow-up action as separate units unless the evidence explicitly says the original action was completed, cancelled, or no longer needed. "
+ "Do not mark a clue as a distractor solely because it appears in an assistant turn; assistant turns can preserve user-specific obligations, summaries, answers, and confirmations when they refer to the surrounding dialogue. "
+ "If an assistant turn says the user needs to do an action, or summarizes a user obligation, keep it as a candidate evidence unit unless contradicted by user evidence. "
+ "For temporal questions, identify the event date or relative time anchor, the query-time anchor if available, and which event should be compared or excluded. "
+ "Use session dates and relative-time phrases from the evidence window text when present; connect nearby event descriptions to the session date when the dialogue makes that local anchor clear. "
+ "For month-level questions, if a past-tense personal event appears inside a dated session and no older or future date is stated, treat it as compatible with that session month; mark the time_anchor as inferred from session date rather than excluding it. "
+ "For negative or absence questions, first identify the relevant event/time bucket, then state whether the requested attribute is present, absent, or contradicted in that event. "
+ "Keep weak but potentially useful clues; mark distractors separately instead of deleting them. "
+ "Return strict JSON only with these keys: question_task, answer_unit, required_operation, evidence_units, plan_steps, candidate_answer, uncertainty. "
+ "evidence_units must be a list of objects with keys: source_index, role, unit_key, value, action, target, instance, time_anchor, polarity, relation_to_question. "
+ "Allowed role values: answer_unit, positive_evidence, temporal_anchor, current_value, old_value, constraint, negative_evidence, distractor, context. "
+ "source_index must be the 1-based evidence window index. candidate_answer may be empty when computation should be left to the answer model."
+ ),
+ },
+ {
+ "role": "user",
+ "content": f"Question:\n{question}\n\nRetrieved evidence windows:\n" + "\n\n".join(candidate_lines),
+ },
+ ]
+ base_url, model, api_key = evidence_unit_planner_config()
+ try:
+ raw = chat_completion(
+ base_url,
+ model,
+ messages,
+ max_tokens=max_tokens,
+ temperature=0.0,
+ api_key=api_key,
+ )
+ except Exception as exc:
+ return {
+ "enabled": False,
+ "error": f"{exc.__class__.__name__}: {str(exc)[:240]}",
+ }
+ payload = parse_json_object(raw)
+ if not payload:
+ return {
+ "enabled": False,
+ "error": "planner_returned_non_json",
+ "raw": truncate(raw, 500),
+ }
+ return sanitize_evidence_unit_plan(payload, max_source_index=len(candidate_lines))
+
+
+def sanitize_evidence_unit_plan(payload: Mapping[str, Any], *, max_source_index: int) -> dict[str, Any]:
+ allowed_roles = {
+ "answer_unit",
+ "positive_evidence",
+ "temporal_anchor",
+ "current_value",
+ "old_value",
+ "constraint",
+ "negative_evidence",
+ "distractor",
+ "context",
+ }
+ plan: dict[str, Any] = {
+ "enabled": True,
+ "question_task": truncate(payload.get("question_task", ""), 80),
+ "answer_unit": truncate(payload.get("answer_unit", ""), 120),
+ "required_operation": truncate(payload.get("required_operation", ""), 180),
+ "evidence_units": [],
+ "plan_steps": [],
+ "candidate_answer": truncate(payload.get("candidate_answer", ""), 160),
+ "uncertainty": truncate(payload.get("uncertainty", ""), 180),
+ }
+ raw_units = payload.get("evidence_units", [])
+ if isinstance(raw_units, list):
+ for raw_unit in raw_units[:24]:
+ if not isinstance(raw_unit, Mapping):
+ continue
+ try:
+ source_index = int(raw_unit.get("source_index", 0) or 0)
+ except (TypeError, ValueError):
+ source_index = 0
+ if source_index < 1 or source_index > max_source_index:
+ continue
+ role = clean_text(raw_unit.get("role", "context")).lower()
+ if role not in allowed_roles:
+ role = "context"
+ plan["evidence_units"].append(
+ {
+ "source_index": source_index,
+ "role": role,
+ "unit_key": truncate(raw_unit.get("unit_key", ""), 100),
+ "value": truncate(raw_unit.get("value", ""), 220),
+ "action": truncate(raw_unit.get("action", ""), 80),
+ "target": truncate(raw_unit.get("target", ""), 100),
+ "instance": truncate(raw_unit.get("instance", ""), 100),
+ "time_anchor": truncate(raw_unit.get("time_anchor", ""), 100),
+ "polarity": truncate(raw_unit.get("polarity", ""), 60),
+ "relation_to_question": truncate(raw_unit.get("relation_to_question", ""), 180),
+ }
+ )
+ raw_steps = payload.get("plan_steps", [])
+ if isinstance(raw_steps, list):
+ plan["plan_steps"] = [truncate(item, 180) for item in raw_steps[:8] if clean_text(item)]
+ return plan
+
+
+def evidence_unit_selected_source_ranks(plan: Mapping[str, Any]) -> dict[int, int]:
+ selected: dict[int, int] = {}
+ ignored_roles = {"distractor"}
+ for unit in list(plan.get("evidence_units") or []):
+ if not isinstance(unit, Mapping):
+ continue
+ role = clean_text(unit.get("role", "")).lower()
+ if role in ignored_roles:
+ continue
+ try:
+ source_index = int(unit.get("source_index", 0) or 0)
+ except (TypeError, ValueError):
+ continue
+ if source_index <= 0 or source_index in selected:
+ continue
+ selected[source_index] = len(selected)
+ return selected
+
+
+def evidence_unit_plan_from_windows(evidence_windows: list[dict[str, Any]]) -> dict[str, Any]:
+ for item in evidence_windows:
+ plan = item.get("evidence_unit_plan")
+ if isinstance(plan, dict) and plan:
+ return plan
+ return {}
+
+
+def _load_unified_operation_planner() -> tuple[Any, Any, Any] | None:
+ mode = os.getenv("TMCRA_UNIFIED_OPERATION_PLANNER_MODE", "").strip().lower()
+ if mode in {"", "off", "false", "0", "disabled", "none"}:
+ return None
+ model_path = os.getenv("TMCRA_UNIFIED_OPERATION_PLANNER_MODEL_PATH", "").strip()
+ if not model_path:
+ return None
+ cache_key = "|".join([model_path, os.getenv("TMCRA_UNIFIED_OPERATION_PLANNER_DEVICE", "cpu")])
+ cached = _UNIFIED_OPERATION_PLANNER_CACHE.get(cache_key)
+ if cached is not None:
+ return cached
+ try:
+ from experiments.replacement import unified_operation_planner as planner
+ except Exception:
+ return None
+ torch_module = getattr(planner, "torch", None)
+ if torch_module is None:
+ return None
+ try:
+ device = torch_module.device(os.getenv("TMCRA_UNIFIED_OPERATION_PLANNER_DEVICE", "cpu") or "cpu")
+ payload = torch_module.load(Path(model_path), map_location=device, weights_only=True)
+ config = planner.UnifiedPlannerConfig.from_dict(dict(payload.get("config", {}) or {}))
+ model = planner.UnifiedOperationPlannerModel(config).to(device)
+ state = dict(payload.get("model_state") or payload.get("state_dict") or {})
+ model.load_state_dict(state, strict=False)
+ model.eval()
+ except Exception:
+ return None
+ loaded = (planner, torch_module, model)
+ _UNIFIED_OPERATION_PLANNER_CACHE[cache_key] = loaded
+ return loaded
+
+
+def apply_unified_operation_planner(question: str, evidence_windows: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ loaded = _load_unified_operation_planner()
+ if loaded is None or not evidence_windows:
+ return evidence_windows
+ planner, torch_module, model = loaded
+ try:
+ max_windows = max(3, int(os.getenv("TMCRA_UNIFIED_OPERATION_PLANNER_MAX_WINDOWS", "16") or 16))
+ answer_threshold = float(os.getenv("TMCRA_UNIFIED_OPERATION_PLANNER_ANSWER_THRESHOLD", "0.52"))
+ except (TypeError, ValueError):
+ max_windows, answer_threshold = 16, 0.52
+ scoped = list(evidence_windows[:max_windows])
+ all_text = " ".join(clean_text(item.get("text", "")) for item in scoped).lower()
+ units = []
+ for index, item in enumerate(scoped, start=1):
+ text = clean_text(item.get("text", ""))
+ memory_id = clean_text(item.get("memory_id", "")) or f"window_{index}"
+ units.append(
+ {
+ "unit_id": f"u{index}",
+ "record_id": memory_id,
+ "session_id": memory_id.split(".turn_", 1)[0],
+ "turn_index": index,
+ "speaker": "unknown",
+ "text": text,
+ "timestamp": "",
+ "topic_bucket": clean_text(item.get("topic_bucket", "")),
+ "node_features": {
+ "retrieval_score": float(item.get("score", 0.0) or 0.0) / 10.0,
+ "graph_score": float(item.get("planner_score", 0.0) or item.get("answer_plan_score", 0.0) or 0.0),
+ "tunnel_score": float(item.get("path_score", 0.0) or 0.0),
+ },
+ "graph_neighbors": [],
+ }
+ )
+ row = {
+ "id": "runtime_unified_operation_planner",
+ "query": question,
+ "retrieval_metadata": {
+ "candidate_count": len(scoped),
+ "session_span": len({clean_text(item.get("memory_id", "")).split(".turn_", 1)[0] for item in scoped}),
+ "topic_span": len({clean_text(item.get("topic_bucket", "")) for item in scoped if clean_text(item.get("topic_bucket", ""))}),
+ "has_temporal_anchor": bool(re.search(r"\b(before|after|latest|current|previous|ago|week|month|year|date)\b", all_text)),
+ "has_numeric_units": bool(re.search(r"\b\d[\d,.]*\b|\$\s?\d|%", all_text)),
+ "has_profile_units": bool(re.search(r"\b(prefer|like|avoid|favorite|usually|always|never)\b", all_text)),
+ },
+ "memory_units": units,
+ "gold": {},
+ }
+ try:
+ dataset = planner.UnifiedOperationPlannerDataset([row], model.config)
+ batch = planner.collate_unified_planner_batch([dataset[0]])
+ device = next(model.parameters()).device
+ model_batch = {key: value.to(device) if hasattr(value, "to") else value for key, value in dict(batch).items()}
+ with torch_module.no_grad():
+ outputs = model(model_batch["features"], model_batch["valid_mask"])
+ relevance_scores = torch_module.sigmoid(outputs["relevance_logits"])[0].detach().cpu().tolist()
+ answer_scores = torch_module.sigmoid(outputs["answer_logits"])[0].detach().cpu().tolist()
+ temporal_scores = torch_module.sigmoid(outputs["temporal_logits"])[0].detach().cpu().tolist()
+ aggregation_scores = torch_module.sigmoid(outputs["aggregation_logits"])[0].detach().cpu().tolist()
+ current_value_scores = torch_module.sigmoid(outputs["current_value_logits"])[0].detach().cpu().tolist()
+ operation_scores = torch_module.sigmoid(outputs["operation_required_logits"])[0].detach().cpu().tolist()
+ operation_family_index = int(torch_module.argmax(outputs["operation_family_logits"], dim=-1)[0].detach().cpu().item())
+ except Exception:
+ return evidence_windows
+ operation_family = planner.OPERATION_FAMILIES[operation_family_index]
+ ranked: list[tuple[int, float, int, dict[str, Any]]] = []
+ for index, item in enumerate(evidence_windows):
+ next_item = dict(item)
+ if index < len(scoped):
+ relevance_score = float(relevance_scores[index])
+ answer_score = float(answer_scores[index])
+ temporal_score = float(temporal_scores[index])
+ aggregation_score = float(aggregation_scores[index])
+ current_value_score = float(current_value_scores[index])
+ unified_score = max(answer_score, temporal_score, aggregation_score, current_value_score, relevance_score * 0.8)
+ next_item["unified_operation_planner"] = True
+ next_item["unified_operation_family"] = operation_family
+ next_item["unified_operation_required_scores"] = {
+ "temporal": round(float(operation_scores[0]), 4),
+ "aggregation": round(float(operation_scores[1]), 4),
+ "profile": round(float(operation_scores[2]), 4),
+ "current_value": round(float(operation_scores[3]), 4),
+ "multi_hop": round(float(operation_scores[4]), 4),
+ }
+ next_item["unified_scores"] = {
+ "relevance": round(relevance_score, 4),
+ "answer": round(answer_score, 4),
+ "temporal": round(temporal_score, 4),
+ "aggregation": round(aggregation_score, 4),
+ "current_value": round(current_value_score, 4),
+ "combined": round(unified_score, 4),
+ }
+ next_item["unified_selected"] = bool(answer_score >= answer_threshold or unified_score >= max(answer_threshold, 0.58))
+ old_planner_score = float(next_item.get("planner_score", 0.0) or next_item.get("answer_plan_score", 0.0) or 0.0)
+ next_item["planner_score"] = max(old_planner_score, unified_score)
+ group = 0 if next_item["unified_selected"] else 1
+ ranked.append((group, -unified_score, index, next_item))
+ else:
+ ranked.append((2, 0.0, index, next_item))
+ ranked.sort(key=lambda row: (row[0], row[1], row[2]))
+ return [item for _, _, _, item in ranked]
+
+
+def format_evidence_unit_plan_for_prompt(plan: Mapping[str, Any]) -> str:
+ if not plan or not bool(plan.get("enabled", False)):
+ return ""
+ compact = {
+ "question_task": plan.get("question_task", ""),
+ "answer_unit": plan.get("answer_unit", ""),
+ "required_operation": plan.get("required_operation", ""),
+ "evidence_units": list(plan.get("evidence_units") or [])[:16],
+ "plan_steps": list(plan.get("plan_steps") or [])[:6],
+ "candidate_answer": plan.get("candidate_answer", ""),
+ "uncertainty": plan.get("uncertainty", ""),
+ }
+ return json.dumps(compact, ensure_ascii=False, separators=(",", ":"))
+
+
+def focused_evidence_unit_lines(
+ question: str,
+ evidence_windows: list[dict[str, Any]],
+ *,
+ max_items: int | None = None,
+ max_chars: int | None = None,
+) -> list[str]:
+ mode = os.getenv("TMCRA_FOCUSED_EVIDENCE_UNIT_MODE", "").strip().lower()
+ if mode in {"", "off", "0", "false", "disabled", "none"}:
+ return []
+ try:
+ item_limit = max(1, int(os.getenv("TMCRA_FOCUSED_EVIDENCE_UNIT_LIMIT", "18") if max_items is None else max_items))
+ char_limit = max(120, int(os.getenv("TMCRA_FOCUSED_EVIDENCE_UNIT_CHARS", "420") if max_chars is None else max_chars))
+ per_window = max(1, int(os.getenv("TMCRA_FOCUSED_EVIDENCE_UNITS_PER_WINDOW", "2") or 2))
+ except (TypeError, ValueError):
+ item_limit, char_limit, per_window = 18, 420, 2
+ question_terms = evidence_query_terms(question)
+ lines: list[str] = []
+ seen: set[str] = set()
+ for window_index, item in enumerate(evidence_windows, start=1):
+ text = clean_text(item.get("text", ""))
+ if not text:
+ continue
+ units = turn_units(text) or [text]
+ scored_units: list[tuple[float, int, str]] = []
+ for unit_index, unit in enumerate(units):
+ unit_text = clean_text(unit)
+ if not unit_text:
+ continue
+ score = score_evidence_unit(question_terms, question, unit_text)
+ if bool(item.get("planner_selected", False)) or bool(item.get("evidence_unit_planner_selected", False)):
+ score += 0.4
+ if bool(item.get("semantic_coverage_expansion", False)):
+ score += 0.25
+ if score <= 0 and len(units) > 1:
+ continue
+ scored_units.append((float(score), unit_index, unit_text))
+ if not scored_units:
+ scored_units = [(0.0, 0, text)]
+ scored_units.sort(key=lambda row: (-row[0], row[1]))
+ added_for_window = 0
+ for _, unit_index, unit_text in scored_units:
+ snippet = centered_excerpt(unit_text, question_terms, max_chars=char_limit)
+ key = re.sub(r"\W+", " ", snippet.lower()).strip()
+ if not key or key in seen:
+ continue
+ seen.add(key)
+ memory_id = clean_text(item.get("memory_id", ""))
+ source = f"source={window_index}"
+ if memory_id:
+ source += f" memory_id={memory_id}"
+ lines.append(f"{len(lines) + 1}. [{source} unit={unit_index}] {snippet}")
+ added_for_window += 1
+ if len(lines) >= item_limit or added_for_window >= per_window:
+ break
+ if len(lines) >= item_limit:
+ break
+ return lines
+
+
+def diversify_evidence_windows(evidence_windows: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ mode = os.getenv("TMCRA_ANSWER_EVIDENCE_DIVERSIFY_MEMORY_IDS", "1").strip().lower()
+ if mode in {"", "0", "false", "off", "no"} or len(evidence_windows) <= 1:
+ return evidence_windows
+ first_pass: list[dict[str, Any]] = []
+ overflow: list[dict[str, Any]] = []
+ seen: set[str] = set()
+ for item in evidence_windows:
+ memory_id = clean_text(item.get("memory_id", ""))
+ if not memory_id:
+ first_pass.append(item)
+ continue
+ if memory_id in seen:
+ overflow.append(item)
+ continue
+ seen.add(memory_id)
+ first_pass.append(item)
+ return [*first_pass, *overflow]
+
+
+def final_answer_channel_intent(question: str) -> dict[str, bool]:
+ terms = evidence_query_terms(question)
+ question_l = question.lower()
+ aggregation = bool(
+ terms
+ & {
+ "amount",
+ "count",
+ "different",
+ "how",
+ "many",
+ "minimum",
+ "number",
+ "percent",
+ "percentage",
+ "sum",
+ "total",
+ }
+ ) or bool(re.search(r"\b(?:how many|how much|total|sum|minimum|maximum|combined|different)\b", question_l))
+ temporal = bool(
+ re.search(
+ r"\b(?:before|after|first|last|earlier|later|weeks?|days?|months?|years?|when|date|how long)\b",
+ question_l,
+ )
+ )
+ assistant_detail = assistant_memory_query(question)
+ return {
+ "aggregation": aggregation,
+ "temporal": temporal,
+ "assistant_detail": assistant_detail,
+ }
+
+
+def _json_object_from_text(value: str) -> dict[str, Any]:
+ text = clean_text(value)
+ if not text:
+ return {}
+ try:
+ payload = json.loads(text)
+ return payload if isinstance(payload, dict) else {}
+ except json.JSONDecodeError:
+ pass
+ match = re.search(r"\{.*\}", text, flags=re.DOTALL)
+ if not match:
+ return {}
+ try:
+ payload = json.loads(match.group(0))
+ except json.JSONDecodeError:
+ return {}
+ return payload if isinstance(payload, dict) else {}
+
+
+def build_query_graph(question: str, question_date: str = "") -> dict[str, Any]:
+ mode = os.getenv("TMCRA_QUERY_GRAPH_BUILDER_MODE", "").strip().lower()
+ if mode in {"", "0", "false", "off", "no", "disabled", "none"}:
+ return {"enabled": False, "mode": mode or "off"}
+ base_url, model, api_key = query_graph_llm_config()
+ try:
+ max_tokens = max(240, int(os.getenv("TMCRA_QUERY_GRAPH_MAX_TOKENS", "700") or 700))
+ except (TypeError, ValueError):
+ max_tokens = 700
+ system = (
+ "You convert a user question into a compact retrieval query graph for a long-memory system. "
+ "Return JSON only. Do not answer the question. Build a general graph, not benchmark-specific logic."
+ )
+ user = {
+ "question": question,
+ "question_date": question_date,
+ "schema": {
+ "task_intent": "direct_fact | count | sum | compare | temporal | preference | multi_evidence | unknown",
+ "operation": "none | count_distinct | sum_numeric | compare_values | select_latest | select_current | timeline_order | infer_preference",
+ "required_units": [
+ {
+ "unit_key": "short stable key",
+ "role": "main_fact | coverage_fact | old_value | current_value | temporal_anchor | constraint | negative",
+ "target_entity": "entity or topic to retrieve",
+ "attribute": "amount/count/status/action/time/preference/etc",
+ "expected_value_type": "money | count | date | duration | item | state | preference | text",
+ "must_cover": True,
+ }
+ ],
+ "tunnel_needs": ["same_topic", "cross_session", "same_entity", "temporal_chain", "profile_bridge"],
+ "query_terms": ["terms that should retrieve memory units"],
+ "negative_terms": ["terms that should not dominate retrieval"],
+ },
+ }
+ try:
+ raw = chat_completion(
+ base_url,
+ model,
+ [
+ {"role": "system", "content": system},
+ {"role": "user", "content": json.dumps(user, ensure_ascii=False)},
+ ],
+ max_tokens=max_tokens,
+ temperature=0.0,
+ api_key=api_key,
+ )
+ graph = _json_object_from_text(raw)
+ if not graph:
+ return {"enabled": True, "mode": mode, "error": "empty_or_invalid_json", "raw": truncate(raw, 600)}
+ graph["enabled"] = True
+ graph["mode"] = mode
+ graph["model"] = model
+ return graph
+ except Exception as exc:
+ return {"enabled": True, "mode": mode, "error": f"{exc.__class__.__name__}: {str(exc)[:240]}"}
+
+
+def query_graph_retrieval_text(question: str, question_date: str, query_graph: dict[str, Any]) -> str:
+ runtime_question = question
+ if question_date:
+ runtime_question = f"{question}\nQuestion date: {question_date}"
+ use_text_mode = os.getenv("TMCRA_QUERY_GRAPH_RETRIEVAL_TEXT_MODE", "").strip().lower() in {"1", "true", "on", "yes"}
+ if not use_text_mode or not bool(query_graph.get("enabled", False)) or query_graph.get("error"):
+ return runtime_question
+ graph = {key: value for key, value in query_graph.items() if key not in {"enabled", "mode", "model", "raw"}}
+ graph_text = truncate(json.dumps(graph, ensure_ascii=False), int(os.getenv("TMCRA_QUERY_GRAPH_RETRIEVAL_CHARS", "2200") or 2200))
+ return f"{runtime_question}\n\nQuestion retrieval graph JSON:\n{graph_text}"
+
+
+def query_graph_sidecar_queries(question: str, question_date: str, query_graph: dict[str, Any]) -> list[str]:
+ if not bool(query_graph.get("enabled", False)) or query_graph.get("error"):
+ return []
+ mode = os.getenv("TMCRA_QUERY_GRAPH_SIDECAR_RETRIEVAL_MODE", "on").strip().lower()
+ if mode in {"", "0", "false", "off", "no", "disabled", "none"}:
+ return []
+ try:
+ max_queries = max(0, int(os.getenv("TMCRA_QUERY_GRAPH_SIDECAR_MAX_QUERIES", "6") or 6))
+ except (TypeError, ValueError):
+ max_queries = 6
+ if max_queries <= 0:
+ return []
+ query_terms = [clean_text(item) for item in list(query_graph.get("query_terms") or []) if clean_text(item)]
+ required_units = list(query_graph.get("required_units") or [])
+ queries: list[str] = []
+ seen: set[str] = set()
+
+ def add(value: str) -> None:
+ text = clean_text(value)
+ if not text:
+ return
+ if question_date and "question date:" not in text.lower():
+ text = f"{text}\nQuestion date: {question_date}"
+ key = text.lower()
+ if key in seen or len(queries) >= max_queries:
+ return
+ seen.add(key)
+ queries.append(text)
+
+ for unit in required_units:
+ if not isinstance(unit, dict):
+ continue
+ role = clean_text(unit.get("role", ""))
+ target = clean_text(unit.get("target_entity", ""))
+ attribute = clean_text(unit.get("attribute", ""))
+ value_type = clean_text(unit.get("expected_value_type", ""))
+ unit_key = clean_text(unit.get("unit_key", ""))
+ add(
+ " ".join(
+ part
+ for part in [
+ question,
+ f"required evidence role {role}",
+ f"target {target}",
+ f"attribute {attribute}",
+ f"value type {value_type}",
+ f"unit {unit_key}",
+ ]
+ if part
+ )
+ )
+ if query_terms and len(queries) < max_queries:
+ add(f"{question} {' '.join(query_terms[:24])}")
+ return queries[:max_queries]
+
+
+def merge_memory_hits(primary_hits: list[Any], extra_hits: list[Any]) -> list[Any]:
+ merged: list[Any] = []
+ seen: set[str] = set()
+ for hit in [*primary_hits, *extra_hits]:
+ memory_id = clean_text(getattr(hit, "memory_id", ""))
+ key = memory_id or clean_text(getattr(hit, "text", ""))[:240].lower()
+ if key and key in seen:
+ continue
+ if key:
+ seen.add(key)
+ merged.append(hit)
+ return merged
+
+
+def apply_query_graph_sidecar_retrieval(
+ adapter: Any,
+ query_graph: dict[str, Any],
+ question: str,
+ question_date: str,
+ hits: list[Any],
+ *,
+ top_k: int,
+) -> tuple[list[Any], dict[str, Any]]:
+ queries = query_graph_sidecar_queries(question, question_date, query_graph)
+ if not queries:
+ return hits, {"enabled": False, "reason": "no_queries"}
+ try:
+ per_query_top_k = max(1, int(os.getenv("TMCRA_QUERY_GRAPH_SIDECAR_TOP_K", "4") or 4))
+ except (TypeError, ValueError):
+ per_query_top_k = 4
+ extra_hits: list[Any] = []
+ errors: list[str] = []
+ for query in queries:
+ try:
+ retrieval = adapter.retrieve(query, top_k=min(max(1, top_k), per_query_top_k))
+ extra_hits.extend(list(getattr(retrieval, "hits", []) or []))
+ except Exception as exc:
+ errors.append(f"{exc.__class__.__name__}:{str(exc)[:160]}")
+ merged = merge_memory_hits(hits, extra_hits)
+ return merged, {
+ "enabled": True,
+ "query_count": len(queries),
+ "extra_hit_count": len(extra_hits),
+ "merged_hit_count": len(merged),
+ "errors": errors[:3],
+ "queries": [truncate(item, 260) for item in queries[:6]],
+ }
+
+
+def final_answer_surface_windows(question: str, evidence_windows: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ mode = os.getenv("TMCRA_FINAL_ANSWER_SURFACE_CHANNEL_MODE", "on").strip().lower()
+ if mode in {"", "0", "false", "off", "no", "disabled", "none"} or not evidence_windows:
+ return evidence_windows
+ intent = final_answer_channel_intent(question)
+ try:
+ final_limit = max(4, int(os.getenv("TMCRA_ANSWER_EVIDENCE_WINDOW_LIMIT", "8") or 8))
+ if intent["aggregation"]:
+ final_limit = max(final_limit, int(os.getenv("TMCRA_ANSWER_EVIDENCE_WINDOW_LIMIT_AGG", "12") or 12))
+ except (TypeError, ValueError):
+ final_limit = 12 if final_answer_channel_intent(question)["aggregation"] else 8
+ if intent["aggregation"]:
+ main_hard_limit = max(4, int(os.getenv("TMCRA_FINAL_MAIN_HARD_LIMIT_AGG", "6") or 6))
+ coverage_hard_limit = max(1, int(os.getenv("TMCRA_FINAL_COVERAGE_HARD_LIMIT_AGG", "6") or 6))
+ elif intent["temporal"] or intent["assistant_detail"]:
+ main_hard_limit = max(4, int(os.getenv("TMCRA_FINAL_MAIN_HARD_LIMIT_DETAIL", "7") or 7))
+ coverage_hard_limit = max(0, int(os.getenv("TMCRA_FINAL_COVERAGE_HARD_LIMIT_DETAIL", "1") or 1))
+ else:
+ main_hard_limit = max(4, int(os.getenv("TMCRA_FINAL_MAIN_HARD_LIMIT_DEFAULT", "6") or 6))
+ coverage_hard_limit = max(0, int(os.getenv("TMCRA_FINAL_COVERAGE_HARD_LIMIT_DEFAULT", "2") or 2))
+ coverage_hard_limit = min(coverage_hard_limit, max(0, final_limit - 1))
+ main_hard_limit = min(main_hard_limit, final_limit - coverage_hard_limit)
+
+ def is_coverage(item: dict[str, Any]) -> bool:
+ return bool(item.get("multi_unit_chain_slot", False)) or bool(item.get("unit_coverage_pack", False))
+
+ def is_assistant_candidate(item: dict[str, Any]) -> bool:
+ return bool(item.get("assistant_candidate_side_channel", False)) or clean_text(item.get("evidence_role", "")).lower() == "assistant_answer_candidate"
+
+ def is_assistant(item: dict[str, Any]) -> bool:
+ if is_assistant_candidate(item):
+ return False
+ return bool(item.get("assistant_origin_evidence", False)) or clean_text(item.get("evidence_role", "")).lower() == "assistant_origin_detail"
+
+ def key(item: dict[str, Any]) -> tuple[int, int, float, float, int]:
+ role = clean_text(item.get("evidence_role", "")).lower()
+ role_allows = int(role not in {"noise", "negative_evidence"})
+ selected = int(bool(item.get("planner_selected", False)) or bool(item.get("evidence_unit_planner_selected", False)))
+ planner_score = max(float(item.get("planner_score", 0.0) or 0.0), 0.0)
+ return (
+ selected,
+ role_allows,
+ planner_score,
+ float(item.get("score", 0.0) or 0.0),
+ len(clean_text(item.get("text", ""))),
+ )
+
+ assistant_candidates = [item for item in evidence_windows if is_assistant_candidate(item)]
+ assistant = [item for item in evidence_windows if is_assistant(item)]
+ source_windows = list(evidence_windows)
+ main = [item for item in source_windows if not is_coverage(item) and not is_assistant(item) and not is_assistant_candidate(item)]
+ coverage = [item for item in source_windows if is_coverage(item)]
+ assistant_candidates.sort(key=key, reverse=True)
+ assistant.sort(key=key, reverse=True)
+ main.sort(key=key, reverse=True)
+ coverage.sort(key=key, reverse=True)
+ top_assistant_memory_ids = {
+ clean_text(item.get("memory_id", ""))
+ for item in assistant[: max(1, int(os.getenv("TMCRA_FINAL_ASSISTANT_HARD_LIMIT", "4") or 4))]
+ if clean_text(item.get("memory_id", ""))
+ }
+ if top_assistant_memory_ids:
+ assistant_candidates.sort(
+ key=lambda item: (
+ int(clean_text(item.get("memory_id", "")) in top_assistant_memory_ids),
+ *key(item),
+ ),
+ reverse=True,
+ )
+
+ ordered: list[dict[str, Any]] = []
+ seen: set[str] = set()
+
+ def add(items: list[dict[str, Any]], limit: int) -> None:
+ added = 0
+ for item in items:
+ if len(ordered) >= final_limit or added >= limit:
+ break
+ memory_id = clean_text(item.get("memory_id", ""))
+ text_key = clean_text(item.get("text", ""))[:240].lower()
+ dedupe_key = text_key if is_assistant_candidate(item) else memory_id or text_key
+ if dedupe_key and dedupe_key in seen:
+ continue
+ if dedupe_key:
+ seen.add(dedupe_key)
+ next_item = dict(item)
+ if is_assistant_candidate(item):
+ channel = "assistant_side"
+ elif is_assistant(item):
+ channel = "assistant"
+ elif is_coverage(item):
+ channel = "coverage"
+ else:
+ channel = "main"
+ next_item["final_answer_channel"] = channel
+ next_item["final_answer_channel_intent"] = dict(intent)
+ ordered.append(next_item)
+ added += 1
+
+ if intent["assistant_detail"]:
+ assistant_limit = max(1, int(os.getenv("TMCRA_FINAL_ASSISTANT_HARD_LIMIT", "4") or 4))
+ add(assistant, min(assistant_limit, final_limit))
+ try:
+ assistant_side_limit = max(0, int(os.getenv("TMCRA_FINAL_ASSISTANT_SIDE_LIMIT", "1") or 1))
+ except (TypeError, ValueError):
+ assistant_side_limit = 1
+ if assistant_side_limit:
+ add(assistant_candidates, assistant_side_limit)
+ add(main, main_hard_limit)
+ add(coverage, coverage_hard_limit)
+ add(assistant, final_limit)
+ add(assistant_candidates, final_limit)
+ add(main, final_limit)
+ add(coverage, final_limit)
+ return ordered[:final_limit]
+
+
+def _llm_channel_planner_enabled() -> bool:
+ mode = os.getenv("TMCRA_LLM_CHANNEL_PLANNER_MODE", "").strip().lower()
+ return mode not in {"", "0", "false", "off", "no", "disabled", "none"}
+
+
+def _int_list_from_payload(value: Any, *, upper_bound: int) -> list[int]:
+ values = iter_selector_indices(value)
+ result: list[int] = []
+ seen: set[int] = set()
+ for raw in values:
+ try:
+ index = int(str(raw).strip())
+ except (TypeError, ValueError):
+ continue
+ if index < 1 or index > upper_bound or index in seen:
+ continue
+ seen.add(index)
+ result.append(index)
+ return result
+
+
+def apply_llm_channel_planner(question: str, evidence_windows: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ if not _llm_channel_planner_enabled() or not evidence_windows:
+ return evidence_windows
+ if any(bool(item.get("llm_channel_planner_enabled", False)) for item in evidence_windows):
+ return evidence_windows
+ try:
+ max_windows = max(4, int(os.getenv("TMCRA_LLM_CHANNEL_PLANNER_MAX_WINDOWS", "16") or 16))
+ max_chars = max(160, int(os.getenv("TMCRA_LLM_CHANNEL_PLANNER_WINDOW_CHARS", "520") or 520))
+ max_tokens = max(160, int(os.getenv("TMCRA_LLM_CHANNEL_PLANNER_MAX_TOKENS", "700") or 700))
+ except (TypeError, ValueError):
+ max_windows, max_chars, max_tokens = 16, 520, 700
+ scoped = list(evidence_windows[:max_windows])
+ if not scoped:
+ return evidence_windows
+
+ lines: list[str] = []
+ for index, item in enumerate(scoped, start=1):
+ text = truncate(item.get("text", ""), max_chars)
+ metadata = {
+ "memory_id": clean_text(item.get("memory_id", "")),
+ "score": round(float(item.get("score", 0.0) or 0.0), 4),
+ "planner_score": round(float(item.get("planner_score", 0.0) or 0.0), 4),
+ "evidence_role": clean_text(item.get("evidence_role", "")),
+ "temporal_state": clean_text(item.get("temporal_state", "")),
+ "unit_coverage": bool(item.get("unit_coverage_pack", False)),
+ "multi_chain": bool(item.get("multi_unit_chain_slot", False)),
+ "profile": bool(item.get("profile_shadow_unit", False) or item.get("profile_layer", False)),
+ "unified_family": clean_text(item.get("unified_operation_family", "")),
+ }
+ lines.append(f"{index}. metadata={json.dumps(metadata, ensure_ascii=False, separators=(',', ':'))}\n{text}")
+
+ system_prompt = (
+ "You are an evidence-channel planner for a long-memory QA system. "
+ "Do not answer the user question. Select evidence windows only. "
+ "Return one strict JSON object and no markdown."
+ )
+ user_prompt = (
+ "Plan the final evidence channels for the answer model.\n"
+ "Rules:\n"
+ "- main_indices must keep direct facts, stable base evidence, temporal anchors, assistant details, and profile facts.\n"
+ "- coverage_indices are for count, sum, ratio, duration, repeated actions, multi-unit coverage, or unit-to-unit chains.\n"
+ "- coverage evidence supplements main evidence; do not let coverage replace main facts.\n"
+ "- support_indices are useful context only.\n"
+ "- suppress_indices are duplicates, stale/conflicting values, or noise.\n"
+ "- Use 1-based indices from the candidate list.\n"
+ "- If unsure, keep a window in main or support instead of suppressing it.\n\n"
+ "Return JSON schema:\n"
+ "{\"operation_family\":\"direct|temporal|count|sum|ratio|profile|current_value|synthesis|unknown\","
+ "\"main_indices\":[1],\"coverage_indices\":[],\"support_indices\":[],\"suppress_indices\":[],"
+ "\"coverage_complete\":true,\"missing_evidence\":\"\",\"reasoning_policy\":\"short label\"}\n\n"
+ f"Question:\n{question}\n\n"
+ "Candidate evidence windows:\n"
+ + "\n\n".join(lines)
+ )
+ try:
+ base_url, model, api_key = answer_llm_config()
+ raw = chat_completion(
+ base_url,
+ model,
+ [{"role": "system", "content": system_prompt}, {"role": "user", "content": user_prompt}],
+ max_tokens=max_tokens,
+ temperature=0.0,
+ api_key=api_key,
+ )
+ payload = parse_json_object(raw)
+ except Exception as exc:
+ annotated = [dict(item) for item in evidence_windows]
+ if annotated:
+ annotated[0]["llm_channel_planner_error"] = f"{exc.__class__.__name__}:{str(exc)[:160]}"
+ return annotated
+ if not payload:
+ return evidence_windows
+
+ main_indices = _int_list_from_payload(payload.get("main_indices", []), upper_bound=len(scoped))
+ coverage_indices = _int_list_from_payload(payload.get("coverage_indices", []), upper_bound=len(scoped))
+ support_indices = _int_list_from_payload(payload.get("support_indices", []), upper_bound=len(scoped))
+ suppress_indices = _int_list_from_payload(payload.get("suppress_indices", []), upper_bound=len(scoped))
+ channels: dict[int, str] = {}
+ for channel, indices in (
+ ("main", main_indices),
+ ("coverage", coverage_indices),
+ ("support", support_indices),
+ ("suppress", suppress_indices),
+ ):
+ for index in indices:
+ channels.setdefault(index - 1, channel)
+
+ if not any(channel in {"main", "coverage", "support"} for channel in channels.values()):
+ return evidence_windows
+
+ channel_rank = {"main": 0, "coverage": 1, "support": 2, "unassigned": 3, "suppress": 4}
+ planned: list[tuple[int, int, dict[str, Any]]] = []
+ for index, item in enumerate(evidence_windows):
+ next_item = dict(item)
+ channel = channels.get(index, "unassigned") if index < len(scoped) else "unassigned"
+ next_item["llm_channel_planner_enabled"] = True
+ next_item["llm_channel"] = channel
+ next_item["llm_channel_operation_family"] = clean_text(payload.get("operation_family", ""))
+ next_item["llm_channel_coverage_complete"] = bool(payload.get("coverage_complete", False))
+ missing_evidence = clean_text(payload.get("missing_evidence", ""))
+ reasoning_policy = clean_text(payload.get("reasoning_policy", ""))
+ if missing_evidence:
+ next_item["llm_channel_missing_evidence"] = missing_evidence
+ if reasoning_policy:
+ next_item["llm_channel_reasoning_policy"] = reasoning_policy
+ planned.append((channel_rank.get(channel, 3), index, next_item))
+ planned.sort(key=lambda row: (row[0], row[1]))
+ return [item for _, _, item in planned]
+
+
+def iter_selector_indices(value: Any) -> list[Any]:
+ if isinstance(value, list):
+ flattened: list[Any] = []
+ for item in value:
+ flattened.extend(iter_selector_indices(item))
+ return flattened
+ return [value]
+
+
+def profile_facts_for_answer(memory_hits: list[Any], *, max_items: int = 8) -> list[str]:
+ facts: list[str] = []
+ seen: set[str] = set()
+ for hit in memory_hits[:12]:
+ metadata = dict(getattr(hit, "metadata", {}) or {})
+ source_kind = str(getattr(hit, "source_kind", "") or "")
+ is_profile_hit = (
+ bool(metadata.get("profile_first_hybrid_rescue"))
+ or bool(metadata.get("profile_protected_slot"))
+ or bool(metadata.get("profile_layer"))
+ or source_kind.startswith("public_dialog_profile")
+ or ".subject." in str(getattr(hit, "memory_id", "") or "")
+ or source_kind in {"public_dialog_preference", "public_dialog_goal", "public_dialog_constraint", "public_dialog_status"}
+ or str(getattr(hit, "category", "") or "").lower() in {"profile", "preference", "goal", "constraint", "status"}
+ )
+ if not is_profile_hit:
+ continue
+ profile_type = clean_text(metadata.get("profile_type", "")) or clean_text(getattr(hit, "category", ""))
+ domain = clean_text(metadata.get("profile_domain_label", "") or metadata.get("profile_domain", ""))
+ summary = clean_text(metadata.get("profile_summary", "") or metadata.get("profile_value", "") or getattr(hit, "value", ""))
+ if not summary:
+ continue
+ prefix = "profile"
+ if profile_type:
+ prefix = profile_type.replace("_", " ")
+ if domain:
+ prefix = f"{prefix} / {domain}"
+ fact = f"{prefix}: {summary}"
+ key = fact.lower()
+ if key in seen:
+ continue
+ seen.add(key)
+ facts.append(fact)
+ if len(facts) >= max_items:
+ break
+ return facts
+
+
+def complete_profile_answer_from_evidence(
+ question: str,
+ answer: str,
+ evidence_windows: list[dict[str, Any]],
+) -> str:
+ # TMCRA_LAYER_TAG: answer-side profile completion; preserves bound qualifiers from retrieved profile facts.
+ answer_clean = clean_text(answer)
+ if not answer_clean:
+ return answer
+ answer_l = answer_clean.lower()
+ question_l = clean_text(question).lower()
+ role_attribute_query = bool(re.search(r"\b(?:occupation|job|role|worked\s+as|work\s+as|career|position)\b", question_l))
+ if len(answer_l) < 4:
+ return answer
+ for item in evidence_windows[:10]:
+ if clean_text(item.get("evidence_role", "")).lower() != "profile_fact":
+ continue
+ fact = clean_text(item.get("text", ""))
+ if not fact or len(fact) > 320:
+ continue
+ fact_l = fact.lower()
+ if answer_l not in fact_l:
+ continue
+ candidates: list[str] = []
+ for pattern in (
+ r"\bas\s+(?:a|an|the)?\s*([^.;,\n]+)",
+ r"\bworked\s+as\s+(?:a|an|the)?\s*([^.;,\n]+)",
+ r"\bpreviously\s+(?:was|worked\s+as)\s+(?:a|an|the)?\s*([^.;,\n]+)",
+ r"\brole\s+as\s+(?:a|an|the)?\s*([^.;,\n]+)",
+ ):
+ match = re.search(pattern, fact, flags=re.IGNORECASE)
+ if match:
+ candidates.append(clean_text(match.group(1)))
+ if role_attribute_query and candidates:
+ return candidates[0].strip(" .")
+ for candidate in candidates:
+ candidate_l = candidate.lower()
+ if answer_l in candidate_l and len(candidate) > len(answer_clean) and len(candidate) <= 180:
+ return candidate.strip(" .")
+ return answer
+
+
+def answer_question(question: str, memory_hits: list[Any], evidence_windows: list[dict[str, Any]] | None = None) -> str:
+ if evidence_windows is None:
+ evidence_windows = build_answer_evidence(question, memory_hits)
+ if not any(bool(item.get("window_planner_enabled", False)) for item in evidence_windows):
+ evidence_windows = apply_answer_window_planner(question, list(evidence_windows))
+ if not any(bool(item.get("llm_evidence_selector_enabled", False)) for item in evidence_windows):
+ evidence_windows = apply_llm_evidence_selector(question, list(evidence_windows))
+ if not evidence_unit_plan_from_windows(evidence_windows):
+ evidence_windows = apply_evidence_unit_planner(question, list(evidence_windows))
+ evidence_windows = apply_unified_operation_planner(question, list(evidence_windows))
+ evidence_windows = apply_llm_channel_planner(question, list(evidence_windows))
+ evidence_windows = final_answer_surface_windows(question, list(evidence_windows))
+ question_l = question.lower()
+ # TMCRA_LAYER_TAG: keyword-based personalized answer gate; replace with turn-intent/profile model.
+ personalized_request = any(
+ marker in question_l
+ for marker in (
+ "recommend",
+ "recommendation",
+ "suggest",
+ "accessories",
+ "my current",
+ "my setup",
+ "for me",
+ "preference",
+ "prefer",
+ "what should",
+ "should i",
+ "any tips",
+ "trouble with",
+ "serve",
+ "dinner",
+ "battery life",
+ )
+ )
+ force_answer_for_eval = os.getenv("TMCRA_LME_FORCE_ANSWER", "1").strip().lower() not in {"0", "false", "no"}
+ profile_facts = profile_facts_for_answer(memory_hits)
+ compact_hit_lines = compact_memory_hit_lines(memory_hits)
+ memory_lines = []
+ evidence_window_limit = int(os.getenv("TMCRA_ANSWER_EVIDENCE_WINDOW_LIMIT", "8"))
+ if final_answer_channel_intent(question)["aggregation"]:
+ evidence_window_limit = max(
+ evidence_window_limit,
+ int(os.getenv("TMCRA_ANSWER_EVIDENCE_WINDOW_LIMIT_AGG", "12") or 12),
+ )
+ ranked_evidence_windows = list(evidence_windows[: max(1, evidence_window_limit)])
+ for index, item in enumerate(ranked_evidence_windows, start=1):
+ value = clean_text(item.get("text", ""))
+ if value:
+ memory_id = clean_text(item.get("memory_id", ""))
+ prefix = f"{index}. "
+ if memory_id:
+ prefix += f"[{memory_id}] "
+ role_bits = []
+ evidence_role = clean_text(item.get("evidence_role", ""))
+ temporal_state = clean_text(item.get("temporal_state", ""))
+ if evidence_role:
+ role_bits.append(f"role={evidence_role}")
+ if temporal_state:
+ role_bits.append(f"time={temporal_state}")
+ if role_bits:
+ prefix += "[" + " ".join(role_bits) + "] "
+ memory_lines.append(prefix + value)
+ model_plan_lines = []
+ model_plan_has_temporal_units = False
+ model_plan_has_computed_temporal_answer = False
+ model_plan_has_computed_coverage_answer = False
+ seen_model_plan_ids: set[str] = set()
+ for item in evidence_windows:
+ if not (
+ bool(item.get("temporal_comparison_model_planner", False))
+ or bool(item.get("aggregation_unit_model_planner", False))
+ or bool(item.get("operation_planner", False))
+ or bool(item.get("unified_operation_planner", False))
+ or bool(item.get("temporal_operation_layer", False))
+ ):
+ continue
+ value = clean_text(item.get("text", ""))
+ if not value:
+ continue
+ memory_id = clean_text(item.get("memory_id", "")) or f"model_plan_{len(model_plan_lines) + 1}"
+ if memory_id in seen_model_plan_ids:
+ continue
+ seen_model_plan_ids.add(memory_id)
+ if '"temporal_units":[{' in value:
+ model_plan_has_temporal_units = True
+ if re.search(r'"computed_temporal_answer":"[^"]+', value):
+ model_plan_has_computed_temporal_answer = True
+ if re.search(r'"computed_coverage_answer":\{"answer":"[^"]+', value):
+ model_plan_has_computed_coverage_answer = True
+ model_plan_lines.append(f"- [{memory_id}] {value}")
+ if len(model_plan_lines) >= 3:
+ break
+ profile_block = "\n".join(f"- {fact}" for fact in profile_facts)
+ memory_block = "\n".join(memory_lines) if memory_lines else "(no relevant long-term memory retrieved)"
+ model_plan_section = (
+ "Model planner outputs:\n"
+ + "\n".join(model_plan_lines)
+ + "\n\n"
+ if model_plan_lines
+ else ""
+ )
+ evidence_unit_plan = evidence_unit_plan_from_windows(evidence_windows)
+ evidence_unit_plan_json = format_evidence_unit_plan_for_prompt(evidence_unit_plan)
+ evidence_unit_plan_section = (
+ f"Model evidence-unit plan:\n{evidence_unit_plan_json}\n\n" if evidence_unit_plan_json else ""
+ )
+ focused_unit_lines = focused_evidence_unit_lines(question, ranked_evidence_windows)
+ focused_unit_section = (
+ "Focused evidence units selected from the retrieved windows:\n"
+ + "\n".join(focused_unit_lines)
+ + "\n\n"
+ if focused_unit_lines
+ else ""
+ )
+ # TMCRA_LAYER_TAG: answer prompt protocol; affects answer utilization, not memory retrieval.
+ reasoning_protocol = (
+ "Treat TMCRA memory as evidence clues, not as a pre-written answer. "
+ "If model temporal, aggregation, or operation plans are provided, read them before the raw evidence because they contain model-selected answer events, unit coverage, time anchors, and comparison direction. "
+ "Use those plans as normalized evidence maps, but verify the answer against the raw memory evidence. "
+ "If a model evidence-unit plan is provided, use it as the primary normalized view of answer units, time anchors, and distinct clues, while verifying it against the raw evidence lines. "
+ "If focused evidence units are provided, scan them before the raw windows; they are compressed excerpts from retrieved memory, not external facts. "
+ "If the evidence-unit plan lists answer_unit, positive_evidence, current_value, old_value, temporal_anchor, or negative_evidence units, reason over those units explicitly before deciding the answer. "
+ "If an aggregation/unit coverage plan contains computed_coverage_answer.answer and its listed units_used or terms match the question, use that computed value unless raw evidence directly contradicts it. "
+ "If candidate_answer is present in the plan and is supported by the listed units, prefer it over re-summarizing the raw window text. "
+ "Before writing the JSON answer, internally infer the question task: direct fact lookup, arithmetic, distinct-item counting, temporal difference, current-value selection, preference/recommendation synthesis, or multi-evidence synthesis. "
+ "Then use the retrieved evidence as clues for that task. "
+ "For arithmetic questions, combine quantities found in different evidence lines and compute the requested value. "
+ "For count questions, infer the answer unit from the current question and count distinct evidence-backed answer units across relevant evidence, not just one sentence. "
+ "For offline benchmark count questions, include plausible evidence-backed units when the only uncertainty is an implicit date or an assistant-confirmed action and there is no contradictory evidence. "
+ "For month-level count questions, a past-tense user event inside a dated session is a usable candidate for that session month when no older or future date contradicts it. "
+ "When the question contains multiple actions joined by and/or, count each distinct target instance attached to any requested action. "
+ "Do not merge two answer units merely because they share the same broad noun; keep them separate when the evidence distinguishes their action, status, time, location, or original-versus-replacement instance. "
+ "For current/now/since/update questions, compare old and later evidence and choose the current or latest value when the question asks for it. "
+ "For advice, preference, dinner, shopping, setup, or tips questions, synthesize a useful answer from remembered personal setup, owned items, ingredients, goals, habits, and constraints; exact prior wording is not required. "
+ "Ignore irrelevant retrieved windows. "
+ "Only answer that information is unavailable when none of the evidence provides a usable clue for the requested task. "
+ )
+ temporal_plan_protocol = ""
+ if model_plan_has_temporal_units or model_plan_has_computed_temporal_answer:
+ temporal_plan_protocol = (
+ "When a model temporal plan explicitly contains non-empty temporal_units or computed_temporal_answer, use those fields as the time math layer. "
+ "For how-many-ago questions, use computed_temporal_answer or delta_to_query fields instead of copying local words such as today. "
+ "For first/order questions with explicit temporal_units, compare normalized_event_date values for the compared events. "
+ "Ignore this temporal-plan instruction when the temporal plan has no temporal_units and no computed_temporal_answer. "
+ )
+ coverage_plan_protocol = ""
+ if model_plan_has_computed_coverage_answer:
+ coverage_plan_protocol = (
+ "When a model aggregation/unit plan explicitly contains computed_coverage_answer, treat it as the unit coverage layer. "
+ "For count, total, sum, and arithmetic questions, prefer the computed answer when the listed units_used or terms are relevant evidence for the requested unit. "
+ "Do not drop a listed unit merely because its raw evidence window also contains unrelated topic text. "
+ )
+ personalized_instruction = ""
+ if personalized_request:
+ personalized_instruction = (
+ "For personalized recommendation or preference questions, synthesize a concise recommendation using the user's retrieved setup, preferences, goals, and constraints. "
+ "If a Structured personalized facts block is present, use it first; otherwise use the long-term memory evidence directly. "
+ "Do not abstain when the memory contains relevant personal setup or preference evidence. "
+ "Do not say the memory lacks the user's setup when the evidence mentions a current brand, device, activity, goal, style, constraint, or previous preference. "
+ "Exact prior accessory lists are not required; infer a compatible recommendation from the remembered setup or preference. "
+ + (
+ "For this offline benchmark, do not ask follow-up questions; a partial remembered setup is enough to produce the best available personalized answer. "
+ if force_answer_for_eval
+ else "In realtime product mode, you may ask a brief follow-up question after giving the best available memory-backed suggestion. "
+ )
+ + "The answer should mention the personalized basis when useful, such as compatibility, brand, quality, durability, or current gear. "
+ )
+ profile_section = f"Structured personalized facts:\n{profile_block}\n\n" if profile_block else ""
+ system_prompt = (
+ "You answer LongMemEval questions using only the current question and the provided long-term memory. "
+ "First identify the requested attribute, then answer only that attribute. "
+ + reasoning_protocol
+ + temporal_plan_protocol
+ + coverage_plan_protocol
+ + "If the question has multiple parts, compares earlier versus now, or asks for both before/current values, answer every requested part explicitly. "
+ "If evidence lines are tagged with role=initial_value and role=current_value, use initial_value for the earlier/starting state and current_value for the now/current state; do not collapse them into one value. "
+ "For current or now values, prefer later-dated evidence over earlier evidence when both are present. "
+ "The answer value must not be an object, list, or multiple JSON keys. "
+ "For where questions, answer with the place, store, venue, retailer, or location. "
+ "Never answer a where question with a time or date such as last Sunday. "
+ "For retail or shopping questions, a nearby store, retailer, or app/store name is the location. "
+ "For how long questions, answer with the duration. "
+ "For name questions, answer with the name. "
+ "If the answer is implied by nearby turns in the same evidence window, use that local context. "
+ "For profile, identity, occupation, role, ownership, and preference facts, preserve attached qualifiers that are part of the remembered fact, such as company type, setting, object owner, condition, strength, or scope. "
+ "If a role=profile_fact evidence line or Structured personalized fact directly answers the requested attribute, preserve the complete specific fact rather than only the head noun or job title. "
+ "Do not shorten a complete remembered fact into only its broad category when the evidence gives a more specific attribute requested by the question. "
+ + personalized_instruction
+ + "Return strict JSON only, with exactly one key named answer. "
+ + "The answer value must be the shortest complete answer phrase or sentence. Do not explain. Do not cite evidence. "
+ + "Example: {\"answer\":\"45 minutes each way\"}\n\n"
+ + profile_section
+ + model_plan_section
+ + evidence_unit_plan_section
+ + focused_unit_section
+ + f"Long-term memory evidence:\n{memory_block}"
+ )
+ messages = [
+ {
+ "role": "system",
+ "content": system_prompt,
+ },
+ {"role": "user", "content": f"Question: {question}\nReturn only JSON."},
+ ]
+ answer_base_url, answer_model, answer_api_key = answer_llm_config()
+ answer = clean_answer_output(chat_completion(
+ answer_base_url,
+ answer_model,
+ messages,
+ max_tokens=int(os.getenv("TMCRA_ANSWER_MAX_TOKENS", "512")),
+ temperature=0.0,
+ api_key=answer_api_key,
+ ))
+ answer = complete_profile_answer_from_evidence(question, answer, evidence_windows)
+ organizer_uses_model_plan = bool(
+ re.search(
+ r"\bhow many\s+(?:days?|weeks?|months?|years?)\b.*\bago\b|"
+ r"\b(?:\d+|one|two|three|four|five|six|seven|eight|nine|ten|eleven|twelve)\s+"
+ r"(?:days?|weeks?|months?|years?)\s+ago\b",
+ question_l,
+ )
+ )
+ organizer_context_section = (
+ profile_section
+ + (model_plan_section if organizer_uses_model_plan else "")
+ + evidence_unit_plan_section
+ )
+ if memory_lines and should_run_answer_reasoning_organizer(
+ question,
+ answer,
+ personalized_request=personalized_request,
+ force_answer_for_eval=force_answer_for_eval,
+ ):
+ organized_answer = organize_answer_from_evidence(
+ question,
+ memory_lines,
+ profile_section=organizer_context_section,
+ force_answer_for_eval=force_answer_for_eval,
+ prior_answer=answer,
+ )
+ if (not organized_answer or looks_like_no_memory_answer(organized_answer)) and compact_hit_lines:
+ organized_answer = organize_answer_from_evidence(
+ question,
+ compact_hit_lines,
+ profile_section=organizer_context_section,
+ force_answer_for_eval=force_answer_for_eval,
+ prior_answer=answer,
+ )
+ if organized_answer and not looks_like_no_memory_answer(organized_answer):
+ return complete_profile_answer_from_evidence(question, organized_answer, evidence_windows)
+ focused_memory_block = "\n".join(memory_lines[:8])
+ # TMCRA_LAYER_TAG: answer-side fallback organizer for conservative/no-memory answers.
+ retry_messages = [
+ {
+ "role": "system",
+ "content": (
+ "You are correcting an over-conservative answer. "
+ "Your previous answer treated memory as a complete answer database; that is wrong. "
+ "The focused evidence below is non-empty and may contain indirect clues rather than an exact answer sentence. "
+ "You must first infer the task required by the question, then combine the evidence clues to answer it. "
+ "Use arithmetic for total/each/save questions, distinct counting for how-many questions, date comparison for temporal questions, current/latest evidence for now/current questions, and synthesis from remembered setup or preferences for advice/tips/recommendation questions. "
+ "For advice, tips, what-should-I, or recommendation questions, a remembered owned item, ingredient, activity, goal, preference, or constraint is enough to produce a useful answer; the memory does not need to contain the exact advice sentence. "
+ "For count questions, infer the answer unit from the current question and combine distinct evidence-backed units across the focused evidence instead of requiring a precomputed total. "
+ "If a model evidence-unit plan is provided, use its answer_unit and positive_evidence units as normalized candidates and only exclude units that are contradicted or outside the requested time/action scope. "
+ "For month-level count questions, do not discard a past-tense user event merely because only the session date provides the month anchor. "
+ "Do not merge units only because they share the same broad noun; keep distinct action targets or original/replacement instances separate when the evidence separates them. "
+ "Do not claim no memory if any focused evidence gives a usable clue for the requested task. "
+ "Do not answer with Information unavailable, not available, no information, or similar wording unless the focused evidence block is empty or entirely unrelated. "
+ "Ignore evidence that is unrelated to the requested task. "
+ + (
+ "For this offline benchmark, do not ask follow-up questions; give the best direct evidence-backed answer. "
+ if force_answer_for_eval
+ else "In realtime product mode, you may ask a brief follow-up question after giving the best memory-backed answer. "
+ )
+ +
+ "Return strict JSON only, with exactly one key named answer.\n\n"
+ + organizer_context_section
+ + f"Focused long-term memory evidence:\n{focused_memory_block}"
+ ),
+ },
+ {"role": "user", "content": f"Question: {question}\nReturn only JSON."},
+ ]
+ retry_answer = clean_answer_output(chat_completion(
+ answer_base_url,
+ answer_model,
+ retry_messages,
+ max_tokens=128,
+ temperature=0.0,
+ api_key=answer_api_key,
+ ))
+ if retry_answer and not looks_like_no_memory_answer(retry_answer):
+ return complete_profile_answer_from_evidence(question, retry_answer, evidence_windows)
+ return complete_profile_answer_from_evidence(question, answer, evidence_windows)
+
+
+def should_run_answer_reasoning_organizer(
+ question: str,
+ answer: str,
+ *,
+ personalized_request: bool,
+ force_answer_for_eval: bool,
+) -> bool:
+ # TMCRA_LAYER_TAG: rule gate for answer-side retry/organizer; not a trained routing model.
+ mode = os.getenv("TMCRA_ANSWER_REASONING_ORGANIZER", "1").strip().lower()
+ if mode in {"", "0", "false", "off", "disabled", "none"}:
+ return False
+ if looks_like_no_memory_answer(answer):
+ return True
+ if personalized_request and force_answer_for_eval and looks_like_followup_answer(answer):
+ return True
+ question_l = clean_text(question).lower()
+ count_markers = (
+ "how many",
+ "number of",
+ "count of",
+ "count the",
+ "count my",
+ "total number",
+ )
+ return any(marker in question_l for marker in count_markers)
+
+
+def organize_answer_from_evidence(
+ question: str,
+ memory_lines: list[str],
+ *,
+ profile_section: str = "",
+ force_answer_for_eval: bool = True,
+ prior_answer: str = "",
+) -> str:
+ if not memory_lines:
+ return ""
+ evidence_block = "\n".join(memory_lines)
+ prior_answer = clean_text(prior_answer)
+ prior_section = f"Previous candidate answer, possibly incomplete: {prior_answer}\n\n" if prior_answer else ""
+ messages = [
+ {
+ "role": "system",
+ "content": (
+ "You are the evidence organizer for a long-memory runtime. "
+ "The final answer model may have been too conservative or may have under-used the retrieved clues, so your job is to read the retrieved memory lines as clues and produce the best evidence-backed answer. "
+ "Do not treat memory as a database that must contain the exact answer sentence. "
+ "Ignore unrelated lines, but if any line contains a usable clue for the question, mark relevant=true. "
+ "For advice, tips, recommendations, dinner, shopping, setup, or preference questions, remembered owned items, tools, accessories, ingredients, goals, habits, constraints, or preferences are enough to synthesize an answer. "
+ "Infer the affordance of remembered items and ingredients: a relevant tool, accessory, setup, ingredient, preference, or constraint can answer the user's underlying need even when the old conversation asked a different surface question. "
+ "Do not answer the old conversation's surface task; adapt the remembered clue to the current question's need. "
+ "When advice uses a remembered owned item, phrase the answer as using or preparing that already-mentioned item and include one practical use step; do not turn it into a generic purchase suggestion unless the question asks what to buy. "
+ "If a model evidence-unit plan is provided, enumerate the plan's answer_unit and positive_evidence units first, then reconcile duplicates or exclusions against the raw evidence. "
+ "For count questions, infer the count unit from the current question itself, then enumerate all relevant candidate clues, merge duplicate mentions of the same clue, and count only evidence-backed units matching that inferred unit. "
+ "For month-level count questions, include past-tense user events from a dated session as candidates for that session month unless another date excludes them. "
+ "When the current question contains multiple actions joined by and/or, the count unit is each distinct target instance attached to any requested action, unless the question explicitly asks for object categories. "
+ "Do not merge two candidate units merely because they share the same broad noun; keep them separate when the evidence distinguishes their action, status, time, location, or original-versus-replacement instance. "
+ "If one memory line contains multiple distinct obligations or requested action targets, count them separately only when the inferred count unit requires action-level counting. "
+ "Do not count broad object categories, generic advice, examples, or unrelated context as answer units. "
+ "Do not stop after the first relevant clue; scan every retrieved line before producing the count. "
+ "For current/now questions, prefer later or current evidence over old evidence when both appear. "
+ "For temporal questions, compare dates or durations when evidence provides them. "
+ "Return strict JSON only with keys: relevant, task, clues, answer. "
+ "answer must be the final shortest useful answer, not an explanation. "
+ "If evidence is genuinely empty or unrelated, return relevant=false and answer=\"\". "
+ + (
+ "For this offline benchmark, do not ask follow-up questions; use the best available evidence-backed inference. "
+ if force_answer_for_eval
+ else "In realtime product mode, prefer a direct memory-backed answer; only ask follow-up if evidence is too thin. "
+ )
+ + "\n\n"
+ + profile_section
+ + prior_section
+ + f"Retrieved memory lines:\n{evidence_block}"
+ ),
+ },
+ {"role": "user", "content": f"Question: {question}\nReturn only JSON."},
+ ]
+ answer_base_url, answer_model, answer_api_key = answer_llm_config()
+ raw = chat_completion(
+ answer_base_url,
+ answer_model,
+ messages,
+ max_tokens=220,
+ temperature=0.0,
+ api_key=answer_api_key,
+ )
+ payload = parse_json_object(raw)
+ if not payload:
+ fallback = clean_answer_output(raw)
+ return "" if fallback.lstrip().startswith("{") else fallback
+ relevant = payload.get("relevant", False)
+ if isinstance(relevant, str):
+ relevant = relevant.strip().lower() in {"1", "true", "yes", "y"}
+ answer = clean_text(payload.get("answer", ""))
+ if not relevant or not answer:
+ return ""
+ return answer.strip(" \t\n\r\"'")
+
+
+def compact_memory_hit_lines(memory_hits: list[Any], *, max_items: int | None = None, max_chars: int | None = None) -> list[str]:
+ if max_items is None:
+ max_items = int(os.getenv("TMCRA_COMPACT_MEMORY_HIT_LIMIT", "8"))
+ if max_chars is None:
+ max_chars = int(os.getenv("TMCRA_COMPACT_MEMORY_HIT_CHARS", "260"))
+ lines: list[str] = []
+ seen: set[str] = set()
+ for index, hit in enumerate(memory_hits[: max(1, max_items)], start=1):
+ value = truncate(getattr(hit, "value", ""), max(80, max_chars))
+ if not value:
+ continue
+ memory_id = clean_text(getattr(hit, "memory_id", ""))
+ category = clean_text(getattr(hit, "category", ""))
+ source_kind = clean_text(getattr(hit, "source_kind", ""))
+ label_bits = []
+ if category:
+ label_bits.append(category)
+ if source_kind:
+ label_bits.append(source_kind)
+ label = " / ".join(label_bits)
+ prefix = f"{index}. "
+ if memory_id:
+ prefix += f"[{memory_id}] "
+ if label:
+ prefix += f"({label}) "
+ line = prefix + value
+ key = re.sub(r"\s+", " ", line.lower())
+ if key in seen:
+ continue
+ seen.add(key)
+ lines.append(line)
+ return lines
+
+
+def parse_json_object(text: str) -> dict[str, Any]:
+ text = clean_text(text)
+ text = re.sub(r"^```(?:\w+)?\s*|\s*```$", "", text).strip()
+ decoder = json.JSONDecoder()
+ for index, char in enumerate(text):
+ if char != "{":
+ continue
+ try:
+ payload, _ = decoder.raw_decode(text[index:])
+ except Exception:
+ continue
+ return payload if isinstance(payload, dict) else {}
+ return {}
+
+
+def looks_like_no_memory_answer(answer: str) -> bool:
+ # TMCRA_LAYER_TAG: string-marker abstention detector used to trigger answer-side retry.
+ value = clean_text(answer).lower()
+ if not value:
+ return True
+ no_memory_markers = (
+ "no information",
+ "no relevant",
+ "not contain",
+ "does not contain",
+ "doesn't contain",
+ "do not have",
+ "don't have",
+ "cannot determine",
+ "can't determine",
+ "not enough information",
+ "insufficient information",
+ "information unavailable",
+ "not available",
+ "unavailable",
+ "not mention",
+ "not mentioned",
+ "unknown",
+ )
+ return any(marker in value for marker in no_memory_markers)
+
+
+def looks_like_followup_answer(answer: str) -> bool:
+ # TMCRA_LAYER_TAG: string-marker follow-up detector used by offline benchmark force-answer mode.
+ value = clean_text(answer).lower()
+ if not value:
+ return False
+ followup_markers = (
+ "could you tell me",
+ "can you tell me",
+ "i need more information",
+ "need more information",
+ "tell me more",
+ "what is your budget",
+ "what's your budget",
+ "what kind of",
+ "would you like",
+ "are you looking",
+ )
+ return any(marker in value for marker in followup_markers) or value.endswith("?")
+
+
+def clean_answer_output(text: str) -> str:
+ text = clean_text(text)
+ text = re.sub(r"^```(?:\w+)?\s*|\s*```$", "", text).strip()
+ json_match = re.search(r"\{.*?\}", text, flags=re.DOTALL)
+ if json_match:
+ try:
+ payload = json.loads(json_match.group(0))
+ answer = clean_text(payload.get("answer", ""))
+ if answer:
+ return answer.strip(" \t\n\r\"'")
+ except Exception:
+ pass
+ tail_markers = [
+ "Referencing the",
+ "Based on the",
+ "While the",
+ "While ",
+ "Although ",
+ "However,",
+ "There is no mention",
+ "There is no information",
+ "Give a direct",
+ "Do not mention",
+ "The provided context",
+ "In the provided",
+ ]
+ for marker in tail_markers:
+ index = text.find(marker)
+ if index > 0:
+ text = text[:index].strip()
+ return text.strip(" \t\n\r\"'")
+
+
+def run(args: argparse.Namespace) -> None:
+ service_root = Path(args.service_root).resolve()
+ repo = Path(args.repo).resolve()
+ apply_env_defaults(service_root)
+ os.environ.setdefault("GEMMA_BASE_URL", f"http://{os.getenv('TMCRA_GEMMA_HOST', '127.0.0.1')}:{os.getenv('TMCRA_GEMMA_PORT', '18002')}/v1")
+ os.environ.setdefault("GEMMA_MODEL", os.getenv("TMCRA_GEMMA_MODEL_NAME", "gemma-4-e4b-it"))
+ if str(repo) not in sys.path:
+ sys.path.insert(0, str(repo))
+ if not args.enable_topic_buckets:
+ disable_topic_bucket_runtime()
+
+ started = time.time()
+ out_dir = Path(args.out) if args.out else DEFAULT_OUT_ROOT / f"lme_s10_native_{datetime.now().strftime('%Y%m%d_%H%M%S')}"
+ out_dir.mkdir(parents=True, exist_ok=True)
+ predictions_path = out_dir / "predictions.jsonl"
+ debug_path = out_dir / "samples_debug.jsonl"
+ summary_path = out_dir / "summary.json"
+ storage_path = out_dir / "native_memory.sqlite3"
+
+ writer = build_writer()
+ records: list[dict[str, Any]] = []
+ log("run_start", data=str(args.data), limit=args.limit, out=str(out_dir), repo=str(repo))
+ for sample_index, row in enumerate(iter_json_array(Path(args.data), limit=args.limit), start=1):
+ qid = clean_text(row.get("question_id")) or f"sample_{sample_index:04d}"
+ question = clean_text(row.get("question"))
+ gold = clean_text(row.get("answer"))
+ sessions = list(row.get("haystack_sessions") or [])
+ session_ids = [clean_text(item) for item in list(row.get("haystack_session_ids") or [])]
+ dates = [clean_text(item) for item in list(row.get("haystack_dates") or [])]
+ selected_indices = (
+ select_official_full_history_indices(row)
+ if args.official_full_history
+ else select_session_indices(row, max_distractors=args.max_distractor_sessions)
+ )
+ adapter = build_adapter(f"longmemeval_native:{qid}:{sample_index}", storage_path)
+ sample_start = time.perf_counter()
+ writer_calls = 0
+ writer_accepted = 0
+ writer_suspected = 0
+ writer_records = 0
+ writer_seconds = 0.0
+ writer_unit_records = 0
+ writer_unit_calls = 0
+ chunk_errors: list[str] = []
+ selected_session_ids = []
+
+ for session_index in selected_indices:
+ if session_index >= len(sessions):
+ continue
+ sid = session_ids[session_index] if session_index < len(session_ids) else f"session_{session_index}"
+ selected_session_ids.append(sid)
+ date = dates[session_index] if session_index < len(dates) else ""
+ is_answer_session = sid in {clean_text(item) for item in list(row.get("answer_session_ids") or [])}
+ max_chunks = (
+ args.max_session_chunks
+ if args.official_full_history
+ else args.max_answer_chunks if is_answer_session else args.max_distractor_chunks
+ )
+ chunks = session_text_chunks(
+ session_id=sid,
+ date=date,
+ turns=list(sessions[session_index] or []),
+ max_chars=args.chunk_chars,
+ max_chunks=max_chunks,
+ )
+ for chunk_index, chunk in enumerate(chunks, start=1):
+ chunk_id = f"s{session_index:03d}_c{chunk_index:02d}"
+ try:
+ result = writer_ingest(adapter, writer, chunk, qid=qid, chunk_id=chunk_id)
+ writer_calls += 1
+ writer_accepted += result["accepted_count"]
+ writer_suspected += result["suspected_count"]
+ writer_records += result["record_count"]
+ writer_seconds += float(result["seconds"])
+ writer_unit_records += int(result.get("unit_record_count", 0) or 0)
+ if bool(result.get("unit_writer_enabled", False)):
+ writer_unit_calls += 1
+ except Exception as exc:
+ chunk_errors.append(f"{chunk_id}:{exc.__class__.__name__}:{str(exc)[:200]}")
+ log("chunk_error", sample=sample_index, qid=qid, chunk=chunk_id, error=str(exc)[:240])
+
+ question_date = clean_text(row.get("question_date", ""))
+ query_graph = build_query_graph(question, question_date)
+ runtime_question = query_graph_retrieval_text(question, question_date, query_graph)
+ retrieve_start = time.perf_counter()
+ retrieval = adapter.retrieve(runtime_question, top_k=args.top_k)
+ retrieve_seconds = time.perf_counter() - retrieve_start
+ hits = list(getattr(retrieval, "hits", []) or [])
+ hits, query_graph_sidecar = apply_query_graph_sidecar_retrieval(
+ adapter,
+ query_graph,
+ question,
+ question_date,
+ hits,
+ top_k=args.top_k,
+ )
+ hits = expand_dialogue_chain_hits(runtime_question, hits, getattr(adapter, "graph", None))
+ hits = expand_semantic_coverage_hits(runtime_question, hits, getattr(adapter, "graph", None))
+ answer_evidence_windows = build_answer_evidence(runtime_question, hits)
+ answer_evidence_windows = apply_answer_window_planner(runtime_question, answer_evidence_windows)
+ answer_evidence_windows = apply_llm_evidence_selector(runtime_question, answer_evidence_windows)
+ answer_evidence_windows = apply_evidence_unit_planner(runtime_question, answer_evidence_windows)
+ answer_evidence_windows = apply_unified_operation_planner(runtime_question, answer_evidence_windows)
+ answer_evidence_windows = diversify_evidence_windows(answer_evidence_windows)
+ answer_evidence_windows = apply_llm_channel_planner(runtime_question, answer_evidence_windows)
+ answer_evidence_windows = final_answer_surface_windows(runtime_question, answer_evidence_windows)
+ answer_start = time.perf_counter()
+ hypothesis = answer_question(runtime_question, hits, answer_evidence_windows)
+ answer_seconds = time.perf_counter() - answer_start
+ graph = getattr(adapter, "graph", None)
+ record = {
+ "sample_index": sample_index,
+ "question_id": qid,
+ "question_type": row.get("question_type"),
+ "question": question,
+ "question_date": question_date,
+ "query_graph": query_graph,
+ "query_graph_sidecar": query_graph_sidecar,
+ "gold_answer": gold,
+ "hypothesis": hypothesis,
+ "answer_session_ids": list(row.get("answer_session_ids") or []),
+ "selected_session_ids": selected_session_ids,
+ "total_sessions": len(sessions),
+ "total_turns": sum(len(s or []) for s in sessions),
+ "selected_sessions": len(selected_session_ids),
+ "history_mode": "official_full_history" if args.official_full_history else "controlled_answer_plus_distractors",
+ "writer_calls": writer_calls,
+ "writer_accepted": writer_accepted,
+ "writer_suspected": writer_suspected,
+ "writer_records": writer_records,
+ "writer_unit_records": writer_unit_records,
+ "writer_unit_calls": writer_unit_calls,
+ "writer_seconds": round(writer_seconds, 3),
+ "retrieve_seconds_wall": round(retrieve_seconds, 3),
+ "answer_seconds": round(answer_seconds, 3),
+ "sample_seconds": round(time.perf_counter() - sample_start, 3),
+ "graph_records": len(getattr(graph, "records_by_id", {}) or {}),
+ "graph_edges": len(getattr(graph, "edges", []) or []),
+ "chunk_errors": chunk_errors,
+ "retrieval": retrieval_debug(retrieval),
+ "evidence_unit_plan": evidence_unit_plan_from_windows(answer_evidence_windows),
+ "answer_evidence_windows": [
+ {
+ "memory_id": item.get("memory_id", ""),
+ "score": item.get("score", 0.0),
+ "planner_selected": bool(item.get("planner_selected", False)),
+ "planner_score": item.get("planner_score", 0.0),
+ "semantic_similarity": item.get("semantic_similarity", item.get("answer_window_semantic_similarity", 0.0)),
+ "answer_window_semantic_similarity": item.get("answer_window_semantic_similarity", 0.0),
+ "answer_window_semantic_enabled": bool(item.get("answer_window_semantic_enabled", False)),
+ "answer_window_semantic_mode": item.get("answer_window_semantic_mode", ""),
+ "window_planner_should_inject_score": item.get("window_planner_should_inject_score", 0.0),
+ "answer_plan_selected": bool(item.get("answer_plan_selected", False)),
+ "answer_plan_score": item.get("answer_plan_score", 0.0),
+ "answer_plan_rank": item.get("answer_plan_rank", 0),
+ "evidence_role": item.get("evidence_role", ""),
+ "temporal_state": item.get("temporal_state", ""),
+ "assistant_memory_query": bool(item.get("assistant_memory_query", False)),
+ "assistant_origin_evidence": bool(item.get("assistant_origin_evidence", False)),
+ "hit_evidence_role": item.get("hit_evidence_role", ""),
+ "hit_temporal_state": item.get("hit_temporal_state", ""),
+ "window_planner_enabled": bool(item.get("window_planner_enabled", False)),
+ "llm_evidence_selector_enabled": bool(item.get("llm_evidence_selector_enabled", False)),
+ "llm_evidence_selected": bool(item.get("llm_evidence_selected", False)),
+ "llm_evidence_rank": item.get("llm_evidence_rank", None),
+ "llm_evidence_model_kept": bool(item.get("llm_evidence_model_kept", False)),
+ "evidence_unit_planner_enabled": bool(item.get("evidence_unit_planner_enabled", False)),
+ "evidence_unit_planner_selected": bool(item.get("evidence_unit_planner_selected", False)),
+ "evidence_unit_planner_rank": item.get("evidence_unit_planner_rank", None),
+ "semantic_coverage_expansion": bool(item.get("semantic_coverage_expansion", False)),
+ "semantic_coverage_score": item.get("semantic_coverage_score", 0.0),
+ "unit_coverage_pack": bool(item.get("unit_coverage_pack", False)),
+ "multi_unit_chain_slot": bool(item.get("multi_unit_chain_slot", False)),
+ "multi_unit_chain_bundle": bool(item.get("multi_unit_chain_bundle", False)),
+ "multi_unit_chain_score": float(item.get("multi_unit_chain_score", 0.0) or 0.0),
+ "unit_kind": item.get("unit_kind", ""),
+ "facet_type": item.get("facet_type", ""),
+ "unified_operation_planner": bool(item.get("unified_operation_planner", False)),
+ "unified_operation_family": item.get("unified_operation_family", ""),
+ "unified_selected": bool(item.get("unified_selected", False)),
+ "unified_scores": item.get("unified_scores", {}),
+ "llm_channel_planner_enabled": bool(item.get("llm_channel_planner_enabled", False)),
+ "llm_channel": item.get("llm_channel", ""),
+ "llm_channel_operation_family": item.get("llm_channel_operation_family", ""),
+ "llm_channel_coverage_complete": bool(item.get("llm_channel_coverage_complete", False)),
+ "llm_channel_missing_evidence": item.get("llm_channel_missing_evidence", ""),
+ "llm_channel_reasoning_policy": item.get("llm_channel_reasoning_policy", ""),
+ "llm_channel_planner_error": item.get("llm_channel_planner_error", ""),
+ "unit_indexes": item.get("unit_indexes", []),
+ "original_chars": item.get("original_chars", 0),
+ "text": truncate(item.get("text", ""), 700),
+ }
+ for item in answer_evidence_windows[:8]
+ ],
+ }
+ records.append(record)
+ with predictions_path.open("a", encoding="utf-8") as handle:
+ handle.write(json.dumps({"question_id": qid, "hypothesis": hypothesis}, ensure_ascii=False) + "\n")
+ with debug_path.open("a", encoding="utf-8") as handle:
+ handle.write(json.dumps(record, ensure_ascii=False) + "\n")
+ log(
+ "sample_done",
+ sample=sample_index,
+ qid=qid,
+ hits=record["retrieval"]["hit_count"],
+ selected_events=record["retrieval"]["selected_event_count"],
+ paths=record["retrieval"]["selected_path_count"],
+ writer_calls=writer_calls,
+ seconds=record["sample_seconds"],
+ hypothesis=truncate(hypothesis, 120),
+ )
+
+ summary = {
+ "status": "completed",
+ "mode": "native_tmcra_flow_smoke",
+ "history_mode": "official_full_history" if args.official_full_history else "controlled_answer_plus_distractors",
+ "limit": args.limit,
+ "sample_count": len(records),
+ "out_dir": str(out_dir),
+ "data": str(args.data),
+ "started_at": datetime.fromtimestamp(started).isoformat(timespec="seconds"),
+ "completed_at": datetime.now().isoformat(timespec="seconds"),
+ "elapsed_seconds": round(time.time() - started, 3),
+ "total_writer_calls": sum(int(item["writer_calls"]) for item in records),
+ "total_writer_accepted": sum(int(item["writer_accepted"]) for item in records),
+ "total_writer_suspected": sum(int(item["writer_suspected"]) for item in records),
+ "total_writer_unit_calls": sum(int(item.get("writer_unit_calls", 0)) for item in records),
+ "total_writer_unit_records": sum(int(item.get("writer_unit_records", 0)) for item in records),
+ "total_graph_records": sum(int(item["graph_records"]) for item in records),
+ "total_graph_edges": sum(int(item["graph_edges"]) for item in records),
+ "avg_retrieval_hits": round(sum(float(item["retrieval"]["hit_count"]) for item in records) / max(1, len(records)), 4),
+ "avg_selected_events": round(sum(float(item["retrieval"]["selected_event_count"]) for item in records) / max(1, len(records)), 4),
+ "avg_selected_paths": round(sum(float(item["retrieval"]["selected_path_count"]) for item in records) / max(1, len(records)), 4),
+ "samples": [
+ {
+ "question_id": item["question_id"],
+ "question": item["question"],
+ "gold_answer": item["gold_answer"],
+ "hypothesis": item["hypothesis"],
+ "retrieval_hits": item["retrieval"]["hit_count"],
+ "selected_event_count": item["retrieval"]["selected_event_count"],
+ "selected_path_count": item["retrieval"]["selected_path_count"],
+ "writer_calls": item["writer_calls"],
+ "sample_seconds": item["sample_seconds"],
+ "chunk_errors": item["chunk_errors"],
+ }
+ for item in records
+ ],
+ }
+ summary_path.write_text(json.dumps(summary, ensure_ascii=False, indent=2), encoding="utf-8")
+ log("run_done", out=str(out_dir), samples=len(records), elapsed=summary["elapsed_seconds"])
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--data", default=str(DEFAULT_DATA))
+ parser.add_argument("--repo", default=str(DEFAULT_REPO))
+ parser.add_argument("--service-root", default=str(DEFAULT_SERVICE_ROOT))
+ parser.add_argument("--out", default="")
+ parser.add_argument("--limit", type=int, default=10)
+ parser.add_argument("--top-k", type=int, default=8)
+ parser.add_argument("--max-distractor-sessions", type=int, default=5)
+ parser.add_argument("--max-distractor-chunks", type=int, default=1)
+ parser.add_argument("--max-answer-chunks", type=int, default=4)
+ parser.add_argument("--official-full-history", action="store_true")
+ parser.add_argument("--max-session-chunks", type=int, default=0)
+ parser.add_argument("--chunk-chars", type=int, default=7000)
+ parser.add_argument("--enable-topic-buckets", action="store_true")
+ args = parser.parse_args()
+ run(args)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/docs/BASELINE_S500_20260525.md b/docs/BASELINE_S500_20260525.md
index 12e8ed447ba7af88488c9b73734b58a812ef2b1a..01ba5ccf933516088b4c319a878beaa68d406f45 100644
--- a/docs/BASELINE_S500_20260525.md
+++ b/docs/BASELINE_S500_20260525.md
@@ -1,35 +1,35 @@
-# S500 Baseline Record
-
-## Run Identity
-
-- Name: frozen baseline38 GPT5.4 full10
-- Date: 2026-05-25
-- Local package: `tmcra_s500_baseline38_release_20260525`
-- Result archive: `results/lme_s500_frozen_baseline38_full10_20260525_results.tar.gz`
-- Judge output: `results/judge_gpt4o_alias_vectorengine.jsonl`
-- Summary: `results/judge_gpt4o_alias_vectorengine.jsonl.summary.json`
-
-## Metrics
-
-- Evaluated samples: 500
-- Correct: 310
-- Accuracy: 62.00%
-- Judge alias: `gpt-4o`
-- Judge resolved model: `gpt-4o-2024-08-06`
-
-## Subtask Metrics
-
-| subtask | accuracy | count |
-| --- | ---: | ---: |
-| knowledge-update | 70.51% | 78 |
-| multi-session | 39.85% | 133 |
-| single-session-assistant | 78.57% | 56 |
-| single-session-preference | 56.67% | 30 |
-| single-session-user | 81.43% | 70 |
-| temporal-reasoning | 63.16% | 133 |
-
-## Interpretation
-
-This baseline is useful as a frozen comparison point. Stronger areas are single-session direct user facts, assistant detail recall, and knowledge-update. Weak areas remain multi-session aggregation, preference abstraction, and temporal reasoning.
-
-The full model output directory is included to preserve all runtime weights and training checkpoints, not only deployment weights.
+# S500 Baseline Record
+
+## Run Identity
+
+- Name: frozen baseline38 GPT5.4 full10
+- Date: 2026-05-25
+- Local package: `tmcra_s500_baseline38_release_20260525`
+- Result archive: `results/lme_s500_frozen_baseline38_full10_20260525_results.tar.gz`
+- Judge output: `results/judge_gpt4o_alias_vectorengine.jsonl`
+- Summary: `results/judge_gpt4o_alias_vectorengine.jsonl.summary.json`
+
+## Metrics
+
+- Evaluated samples: 500
+- Correct: 310
+- Accuracy: 62.00%
+- Judge alias: `gpt-4o`
+- Judge resolved model: `gpt-4o-2024-08-06`
+
+## Subtask Metrics
+
+| subtask | accuracy | count |
+| --- | ---: | ---: |
+| knowledge-update | 70.51% | 78 |
+| multi-session | 39.85% | 133 |
+| single-session-assistant | 78.57% | 56 |
+| single-session-preference | 56.67% | 30 |
+| single-session-user | 81.43% | 70 |
+| temporal-reasoning | 63.16% | 133 |
+
+## Interpretation
+
+This baseline is useful as a frozen comparison point. Stronger areas are single-session direct user facts, assistant detail recall, and knowledge-update. Weak areas remain multi-session aggregation, preference abstraction, and temporal reasoning.
+
+The full model output directory is included to preserve all runtime weights and training checkpoints, not only deployment weights.
diff --git a/docs/OPTIONAL_MODULES_AND_PARALLEL.md b/docs/OPTIONAL_MODULES_AND_PARALLEL.md
index e5ddb53df5e3713cf3ef3ccc2da38c7558fbc905..7f43e72e27e1a6765071e2b128f92773917e01fa 100644
--- a/docs/OPTIONAL_MODULES_AND_PARALLEL.md
+++ b/docs/OPTIONAL_MODULES_AND_PARALLEL.md
@@ -1,347 +1,347 @@
-# Optional Modules and Parallel Evaluation Plan
-
-This document describes two optional extension points already preserved in the current TMCRA package:
-
-- Embedder interface
-- LLM planner interface
-
-It also summarizes the parallel evaluation pattern used for the S500 baseline so later deployment, evaluation, and ablation runs can reuse the same structure.
-
-## 1. Current Main Path
-
-The frozen S500 baseline uses this core path:
-
-```text
-dialogue -> writer layer -> graph memory -> learned node/path scorer -> evidence selection -> answer layer
-```
-
-The responsibilities are:
-
-- The writer layer converts dialogue into memory nodes, event units, profile signals, and temporal signals.
-- The graph memory layer stores nodes, paths, and tunnel links.
-- `node_scorer.pt` and `path_scorer.pt` perform learned node/path scoring.
-- Evidence selection converts candidate memories into compact evidence.
-- The answer-layer LLM produces the final response from the selected evidence.
-
-Embedder and LLM planner modules are optional enhancement modules. They should not replace the main graph model. They are better treated as auxiliary channels, ablation switches, or higher-cost deployment paths.
-
-## 2. Embedder Interface
-
-The current code exposes three embedder integration points.
-
-### 2.1 Write-time Indexing
-
-At write time, TMCRA can build an embedding index for newly written memory nodes. That index can later serve as an auxiliary candidate source during retrieval.
-
-Relevant configuration:
-
-```bash
-export TMCRA_EMBEDDER_MODEL_PATH="BAAI/bge-m3"
-export TMCRA_EMBEDDER_DEVICE="cuda"
-export TMCRA_EMBEDDER_MODEL_MAX_LENGTH="512"
-export TMCRA_WRITE_EMBEDDER_INDEX_MODE="bge_m3"
-export TMCRA_WRITE_EMBEDDER_INDEX_MAX_TERMS="96"
-```
-
-Purpose:
-
-- Build semantic indexes after the writer stores memory nodes.
-- Keep the original graph structure unchanged.
-- Add a semantic candidate channel without replacing the learned node/path scorers.
-
-### 2.2 Pre-recall Candidate Expansion
-
-Before graph retrieval, the embedder can find candidate event ids, which are then passed into graph retrieval and scorer ranking.
-
-Relevant configuration:
-
-```bash
-export TMCRA_EMBEDDER_PRE_RECALL_MODE="bge_m3"
-export TMCRA_EMBEDDER_PRE_RECALL_K="16"
-export TMCRA_EMBEDDER_INDEX_RECALL_MODE="bge_m3"
-export TMCRA_EMBEDDER_INDEX_RECALL_K="24"
-```
-
-Purpose:
-
-- Expand the candidate range before retrieval.
-- Help when query wording and memory wording differ.
-- Provide an auxiliary path for semantically close memories with weak graph paths.
-
-### 2.3 Post-recall Fusion
-
-After retrieval, embedder-matched events can be fused with graph-model results so semantically relevant nodes receive a limited boost.
-
-Relevant configuration:
-
-```bash
-export TMCRA_EMBEDDER_FUSION_MODE="on"
-export TMCRA_EMBEDDER_FUSION_WEIGHT="0.35"
-export TMCRA_EMBEDDER_FUSION_SCORE_FLOOR="0.62"
-export TMCRA_EMBEDDER_FUSION_TOP_K="16"
-export TMCRA_EMBEDDER_FUSION_SELECT_K="4"
-export TMCRA_EMBEDDER_FUSION_MAX_BOOST="0.42"
-```
-
-Purpose:
-
-- Give semantically similar candidates a bounded score boost.
-- Prevent the embedder from directly replacing main evidence ranking.
-- Use embedding as an auxiliary recall layer for the learned graph scorer.
-
-## 3. LLM Planner Interface
-
-The current code exposes three main LLM planner paths. They run either after retrieval or before query-side retrieval expansion.
-
-### 3.1 Evidence-unit Planner
-
-The evidence-unit planner runs after retrieval and uses an LLM to normalize retrieved windows into evidence units.
-
-Relevant configuration:
-
-```bash
-export TMCRA_EVIDENCE_UNIT_PLANNER_MODE="on"
-export TMCRA_EVIDENCE_UNIT_PLANNER_BASE_URL=""
-export TMCRA_EVIDENCE_UNIT_PLANNER_MODEL=""
-export TMCRA_EVIDENCE_UNIT_PLANNER_API_KEY=""
-export TMCRA_EVIDENCE_UNIT_PLANNER_MAX_CANDIDATES="10"
-export TMCRA_EVIDENCE_UNIT_PLANNER_CHARS="1100"
-export TMCRA_EVIDENCE_UNIT_PLANNER_MAX_TOKENS="760"
-export TMCRA_EVIDENCE_UNIT_PLANNER_REORDER="0"
-```
-
-If planner-specific base/model/key values are not configured, this planner inherits answer-layer configuration:
-
-```bash
-export TMCRA_ANSWER_BASE_URL=""
-export TMCRA_ANSWER_MODEL=""
-export TMCRA_ANSWER_API_KEY=""
-```
-
-Purpose:
-
-- Mark answer units, positive evidence, temporal anchors, current values, old values, constraints, and negative evidence.
-- Help the final answer layer understand how to use the retrieved evidence.
-- Organize evidence without replacing graph retrieval.
-
-### 3.2 LLM Channel Planner
-
-The LLM channel planner runs before final evidence is sent to the answer layer. It separates main evidence, coverage evidence, support evidence, and suppressed evidence.
-
-Relevant configuration:
-
-```bash
-export TMCRA_LLM_CHANNEL_PLANNER_MODE="on"
-export TMCRA_LLM_CHANNEL_PLANNER_MAX_WINDOWS="16"
-export TMCRA_LLM_CHANNEL_PLANNER_WINDOW_CHARS="520"
-export TMCRA_LLM_CHANNEL_PLANNER_MAX_TOKENS="700"
-```
-
-Purpose:
-
-- Make coverage evidence supplement main facts instead of replacing them.
-- Improve count/sum/ratio/duration/multi-unit tasks.
-- Provide a higher-cost quality mode for experiments and selected deployments.
-
-In the frozen S500 baseline record, this module was:
-
-```text
-llm_channel_planner=off
-```
-
-### 3.3 Query Graph Builder
-
-The query graph builder runs before retrieval. It converts the user question into a compact query graph and can expand that graph into sidecar retrieval queries.
-
-Relevant configuration:
-
-```bash
-export TMCRA_QUERY_GRAPH_BUILDER_MODE="on"
-export TMCRA_QUERY_GRAPH_BASE_URL=""
-export TMCRA_QUERY_GRAPH_MODEL=""
-export TMCRA_QUERY_GRAPH_API_KEY=""
-export TMCRA_QUERY_GRAPH_MAX_TOKENS="700"
-export TMCRA_QUERY_GRAPH_SIDECAR_RETRIEVAL_MODE="on"
-export TMCRA_QUERY_GRAPH_SIDECAR_MAX_QUERIES="6"
-export TMCRA_QUERY_GRAPH_SIDECAR_TOP_K="4"
-```
-
-Purpose:
-
-- Convert the question into task intent, required units, operation, and tunnel needs.
-- Give complex multi-session, temporal, and profile questions a clearer retrieval direction.
-- Test whether building a query graph before retrieval improves candidate recall.
-
-## 4. Local Model Planner vs LLM Planner
-
-The code also contains local model planner interfaces, for example:
-
-```bash
-export TMCRA_ANSWER_WINDOW_PLANNER_MODE="on"
-export TMCRA_ANSWER_WINDOW_PLANNER_MODEL_PATH=""
-export TMCRA_UNIFIED_OPERATION_PLANNER_MODE="on"
-export TMCRA_UNIFIED_OPERATION_PLANNER_MODEL_PATH=""
-export TMCRA_INJECTION_PLANNER_MODE="guided"
-export TMCRA_INJECTION_PLANNER_MODEL_PATH=""
-```
-
-These are local model interfaces, not LLM planner interfaces.
-
-The distinction is:
-
-- LLM planner: calls an external or local LLM; higher cost; useful for validating capability ceilings.
-- Local model planner: lower cost and better for productization, but requires targeted training and stability validation.
-
-Recommended workflow:
-
-```text
-validate behavior with an LLM planner -> distill or train the useful behavior into the graph model or a local planner head
-```
-
-## 5. Parallel Evaluation Plan
-
-The S500 baseline used shard-level parallelism:
-
-```text
-500 samples -> 10 shards -> 50 samples per shard
-```
-
-Each shard runs independently:
-
-```text
-input_shard_N.json -> shard_N/ -> predictions/debug/summary
-```
-
-Core parallelization principles:
-
-- One independent process per shard.
-- One independent output directory per shard.
-- Writer key pool is rotated by shard index.
-- Main model weights are read-only and shared.
-- Predictions, samples_debug, and judge results are merged after all shards complete.
-
-Key S500 baseline runtime configuration:
-
-```text
-samples=500
-shards=10
-per_shard=50
-writer=DeepSeek v4 Flash
-answer_layer=GPT5.4
-llm_channel_planner=off
-history_mode=controlled_answer_plus_distractors
-```
-
-### 5.1 Reusable Parallel Template
-
-Recommended baseline template:
-
-```bash
-export TMCRA_RETRIEVAL_MODE="hybrid_node_scored"
-export TMCRA_REQUIRE_LEARNED_SCORER="1"
-export TMCRA_NODE_MODEL_DEVICE="cuda"
-export TMCRA_NODE_MODEL_PATH="models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer.pt"
-export TMCRA_PATH_MODEL_PATH="models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer.pt"
-
-export TMCRA_WRITER_MODEL="deepseek-chat"
-export TMCRA_WRITER_MAX_TOKENS="512"
-export TMCRA_WRITER_TIMEOUT_SECONDS="180"
-export TMCRA_WRITER_TEMPERATURE="0"
-export TMCRA_WRITER_INPUT_MODE="delta"
-export TMCRA_WRITER_MAX_PROPOSALS="2"
-
-export TMCRA_ANSWER_MAX_TOKENS="512"
-```
-
-Single-shard execution shape:
-
-```bash
-python code/run_lme_s10_native_tmcra.py \
- --data "/input_shard_N.json" \
- --repo "" \
- --service-root "" \
- --out "/shard_N" \
- --limit 50 \
- --top-k 10 \
- --max-distractor-sessions 5 \
- --max-distractor-chunks 1 \
- --max-answer-chunks 4 \
- --chunk-chars 7000
-```
-
-### 5.2 Suggested Rollout Order
-
-Do not enable all optional modules at once. Use staged A/B testing:
-
-1. **Baseline scorer-only**
- - embedder off
- - LLM channel planner off
- - query graph builder off
- - confirms frozen baseline stability
-
-2. **Embedder pre-recall A/B**
- - enable write-time indexing and pre-recall candidate expansion only
- - measure candidate hit rate, retrieval latency, and error-type shifts
-
-3. **Embedder fusion A/B**
- - enable fusion only after pre-recall is stable
- - keep boost bounded so embedder does not override the main graph scorer
-
-4. **Evidence-unit planner A/B**
- - enable LLM evidence-unit planner
- - measure whether the answer layer uses retrieved evidence better
-
-5. **LLM channel planner A/B**
- - test mainly on multi/aggregation/temporal error clusters
- - verify coverage evidence supplements main facts instead of replacing them
-
-6. **Query graph builder A/B**
- - validate the ceiling of query-graph-first retrieval
- - if effective, distill the behavior into query-understanding or graph scorer training
-
-### 5.3 Parallel Scale Guidance
-
-Parallelism should not be determined only by the number of API keys. Also watch:
-
-- GPU memory
-- CPU memory
-- writer latency
-- answer-layer latency
-- graph ingest / SQLite write overhead
-- average writer calls per shard
-
-Scale gradually:
-
-```text
-5 shards smoke -> 10 shards stable -> 20 shards stress -> 30 shards only if no memory/API/IO issue
-```
-
-If error rate rises, memory drops sharply, API 402/429 appears, chunk errors occur, or shards stall, reduce parallelism first and then resume missing samples.
-
-## 6. Recommended Experiment Matrix
-
-Minimal interpretable matrix:
-
-| experiment | Embedder | LLM planner | purpose |
-| --- | --- | --- | --- |
-| baseline | off | off | fixed main graph-model baseline |
-| embedder-pre | pre-recall on | off | test candidate expansion |
-| embedder-fusion | pre-recall + fusion on | off | test semantic fusion |
-| evidence-unit | off | evidence-unit on | test pre-answer evidence organization |
-| channel-planner | off | channel planner on | test main/coverage separation |
-| query-graph | off | query graph on | test query-graph retrieval |
-| combined-light | pre-recall on | evidence-unit on | test lower-cost combined path |
-| combined-heavy | pre-recall + fusion on | evidence-unit + channel planner on | test capability ceiling |
-
-Each run should preserve:
-
-- predictions
-- samples_debug
-- judge output
-- by-task accuracy
-- writer calls
-- retrieval latency
-- answer latency
-- per-sample error type
-
-This makes it possible to separate recall errors, evidence-selection errors, planner errors, answer-layer errors, and parallel-runtime instability.
+# Optional Modules and Parallel Evaluation Plan
+
+This document describes two optional extension points already preserved in the current TMCRA package:
+
+- Embedder interface
+- LLM planner interface
+
+It also summarizes the parallel evaluation pattern used for the S500 baseline so later deployment, evaluation, and ablation runs can reuse the same structure.
+
+## 1. Current Main Path
+
+The frozen S500 baseline uses this core path:
+
+```text
+dialogue -> writer layer -> graph memory -> learned node/path scorer -> evidence selection -> answer layer
+```
+
+The responsibilities are:
+
+- The writer layer converts dialogue into memory nodes, event units, profile signals, and temporal signals.
+- The graph memory layer stores nodes, paths, and tunnel links.
+- `node_scorer.pt` and `path_scorer.pt` perform learned node/path scoring.
+- Evidence selection converts candidate memories into compact evidence.
+- The answer-layer LLM produces the final response from the selected evidence.
+
+Embedder and LLM planner modules are optional enhancement modules. They should not replace the main graph model. They are better treated as auxiliary channels, ablation switches, or higher-cost deployment paths.
+
+## 2. Embedder Interface
+
+The current code exposes three embedder integration points.
+
+### 2.1 Write-time Indexing
+
+At write time, TMCRA can build an embedding index for newly written memory nodes. That index can later serve as an auxiliary candidate source during retrieval.
+
+Relevant configuration:
+
+```bash
+export TMCRA_EMBEDDER_MODEL_PATH="BAAI/bge-m3"
+export TMCRA_EMBEDDER_DEVICE="cuda"
+export TMCRA_EMBEDDER_MODEL_MAX_LENGTH="512"
+export TMCRA_WRITE_EMBEDDER_INDEX_MODE="bge_m3"
+export TMCRA_WRITE_EMBEDDER_INDEX_MAX_TERMS="96"
+```
+
+Purpose:
+
+- Build semantic indexes after the writer stores memory nodes.
+- Keep the original graph structure unchanged.
+- Add a semantic candidate channel without replacing the learned node/path scorers.
+
+### 2.2 Pre-recall Candidate Expansion
+
+Before graph retrieval, the embedder can find candidate event ids, which are then passed into graph retrieval and scorer ranking.
+
+Relevant configuration:
+
+```bash
+export TMCRA_EMBEDDER_PRE_RECALL_MODE="bge_m3"
+export TMCRA_EMBEDDER_PRE_RECALL_K="16"
+export TMCRA_EMBEDDER_INDEX_RECALL_MODE="bge_m3"
+export TMCRA_EMBEDDER_INDEX_RECALL_K="24"
+```
+
+Purpose:
+
+- Expand the candidate range before retrieval.
+- Help when query wording and memory wording differ.
+- Provide an auxiliary path for semantically close memories with weak graph paths.
+
+### 2.3 Post-recall Fusion
+
+After retrieval, embedder-matched events can be fused with graph-model results so semantically relevant nodes receive a limited boost.
+
+Relevant configuration:
+
+```bash
+export TMCRA_EMBEDDER_FUSION_MODE="on"
+export TMCRA_EMBEDDER_FUSION_WEIGHT="0.35"
+export TMCRA_EMBEDDER_FUSION_SCORE_FLOOR="0.62"
+export TMCRA_EMBEDDER_FUSION_TOP_K="16"
+export TMCRA_EMBEDDER_FUSION_SELECT_K="4"
+export TMCRA_EMBEDDER_FUSION_MAX_BOOST="0.42"
+```
+
+Purpose:
+
+- Give semantically similar candidates a bounded score boost.
+- Prevent the embedder from directly replacing main evidence ranking.
+- Use embedding as an auxiliary recall layer for the learned graph scorer.
+
+## 3. LLM Planner Interface
+
+The current code exposes three main LLM planner paths. They run either after retrieval or before query-side retrieval expansion.
+
+### 3.1 Evidence-unit Planner
+
+The evidence-unit planner runs after retrieval and uses an LLM to normalize retrieved windows into evidence units.
+
+Relevant configuration:
+
+```bash
+export TMCRA_EVIDENCE_UNIT_PLANNER_MODE="on"
+export TMCRA_EVIDENCE_UNIT_PLANNER_BASE_URL=""
+export TMCRA_EVIDENCE_UNIT_PLANNER_MODEL=""
+export TMCRA_EVIDENCE_UNIT_PLANNER_API_KEY=""
+export TMCRA_EVIDENCE_UNIT_PLANNER_MAX_CANDIDATES="10"
+export TMCRA_EVIDENCE_UNIT_PLANNER_CHARS="1100"
+export TMCRA_EVIDENCE_UNIT_PLANNER_MAX_TOKENS="760"
+export TMCRA_EVIDENCE_UNIT_PLANNER_REORDER="0"
+```
+
+If planner-specific base/model/key values are not configured, this planner inherits answer-layer configuration:
+
+```bash
+export TMCRA_ANSWER_BASE_URL=""
+export TMCRA_ANSWER_MODEL=""
+export TMCRA_ANSWER_API_KEY=""
+```
+
+Purpose:
+
+- Mark answer units, positive evidence, temporal anchors, current values, old values, constraints, and negative evidence.
+- Help the final answer layer understand how to use the retrieved evidence.
+- Organize evidence without replacing graph retrieval.
+
+### 3.2 LLM Channel Planner
+
+The LLM channel planner runs before final evidence is sent to the answer layer. It separates main evidence, coverage evidence, support evidence, and suppressed evidence.
+
+Relevant configuration:
+
+```bash
+export TMCRA_LLM_CHANNEL_PLANNER_MODE="on"
+export TMCRA_LLM_CHANNEL_PLANNER_MAX_WINDOWS="16"
+export TMCRA_LLM_CHANNEL_PLANNER_WINDOW_CHARS="520"
+export TMCRA_LLM_CHANNEL_PLANNER_MAX_TOKENS="700"
+```
+
+Purpose:
+
+- Make coverage evidence supplement main facts instead of replacing them.
+- Improve count/sum/ratio/duration/multi-unit tasks.
+- Provide a higher-cost quality mode for experiments and selected deployments.
+
+In the frozen S500 baseline record, this module was:
+
+```text
+llm_channel_planner=off
+```
+
+### 3.3 Query Graph Builder
+
+The query graph builder runs before retrieval. It converts the user question into a compact query graph and can expand that graph into sidecar retrieval queries.
+
+Relevant configuration:
+
+```bash
+export TMCRA_QUERY_GRAPH_BUILDER_MODE="on"
+export TMCRA_QUERY_GRAPH_BASE_URL=""
+export TMCRA_QUERY_GRAPH_MODEL=""
+export TMCRA_QUERY_GRAPH_API_KEY=""
+export TMCRA_QUERY_GRAPH_MAX_TOKENS="700"
+export TMCRA_QUERY_GRAPH_SIDECAR_RETRIEVAL_MODE="on"
+export TMCRA_QUERY_GRAPH_SIDECAR_MAX_QUERIES="6"
+export TMCRA_QUERY_GRAPH_SIDECAR_TOP_K="4"
+```
+
+Purpose:
+
+- Convert the question into task intent, required units, operation, and tunnel needs.
+- Give complex multi-session, temporal, and profile questions a clearer retrieval direction.
+- Test whether building a query graph before retrieval improves candidate recall.
+
+## 4. Local Model Planner vs LLM Planner
+
+The code also contains local model planner interfaces, for example:
+
+```bash
+export TMCRA_ANSWER_WINDOW_PLANNER_MODE="on"
+export TMCRA_ANSWER_WINDOW_PLANNER_MODEL_PATH=""
+export TMCRA_UNIFIED_OPERATION_PLANNER_MODE="on"
+export TMCRA_UNIFIED_OPERATION_PLANNER_MODEL_PATH=""
+export TMCRA_INJECTION_PLANNER_MODE="guided"
+export TMCRA_INJECTION_PLANNER_MODEL_PATH=""
+```
+
+These are local model interfaces, not LLM planner interfaces.
+
+The distinction is:
+
+- LLM planner: calls an external or local LLM; higher cost; useful for validating capability ceilings.
+- Local model planner: lower cost and better for productization, but requires targeted training and stability validation.
+
+Recommended workflow:
+
+```text
+validate behavior with an LLM planner -> distill or train the useful behavior into the graph model or a local planner head
+```
+
+## 5. Parallel Evaluation Plan
+
+The S500 baseline used shard-level parallelism:
+
+```text
+500 samples -> 10 shards -> 50 samples per shard
+```
+
+Each shard runs independently:
+
+```text
+input_shard_N.json -> shard_N/ -> predictions/debug/summary
+```
+
+Core parallelization principles:
+
+- One independent process per shard.
+- One independent output directory per shard.
+- Writer key pool is rotated by shard index.
+- Main model weights are read-only and shared.
+- Predictions, samples_debug, and judge results are merged after all shards complete.
+
+Key S500 baseline runtime configuration:
+
+```text
+samples=500
+shards=10
+per_shard=50
+writer=DeepSeek v4 Flash
+answer_layer=GPT5.4
+llm_channel_planner=off
+history_mode=controlled_answer_plus_distractors
+```
+
+### 5.1 Reusable Parallel Template
+
+Recommended baseline template:
+
+```bash
+export TMCRA_RETRIEVAL_MODE="hybrid_node_scored"
+export TMCRA_REQUIRE_LEARNED_SCORER="1"
+export TMCRA_NODE_MODEL_DEVICE="cuda"
+export TMCRA_NODE_MODEL_PATH="models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer.pt"
+export TMCRA_PATH_MODEL_PATH="models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer.pt"
+
+export TMCRA_WRITER_MODEL="deepseek-chat"
+export TMCRA_WRITER_MAX_TOKENS="512"
+export TMCRA_WRITER_TIMEOUT_SECONDS="180"
+export TMCRA_WRITER_TEMPERATURE="0"
+export TMCRA_WRITER_INPUT_MODE="delta"
+export TMCRA_WRITER_MAX_PROPOSALS="2"
+
+export TMCRA_ANSWER_MAX_TOKENS="512"
+```
+
+Single-shard execution shape:
+
+```bash
+python code/run_lme_s10_native_tmcra.py \
+ --data "/input_shard_N.json" \
+ --repo "" \
+ --service-root "" \
+ --out "/shard_N" \
+ --limit 50 \
+ --top-k 10 \
+ --max-distractor-sessions 5 \
+ --max-distractor-chunks 1 \
+ --max-answer-chunks 4 \
+ --chunk-chars 7000
+```
+
+### 5.2 Suggested Rollout Order
+
+Do not enable all optional modules at once. Use staged A/B testing:
+
+1. **Baseline scorer-only**
+ - embedder off
+ - LLM channel planner off
+ - query graph builder off
+ - confirms frozen baseline stability
+
+2. **Embedder pre-recall A/B**
+ - enable write-time indexing and pre-recall candidate expansion only
+ - measure candidate hit rate, retrieval latency, and error-type shifts
+
+3. **Embedder fusion A/B**
+ - enable fusion only after pre-recall is stable
+ - keep boost bounded so embedder does not override the main graph scorer
+
+4. **Evidence-unit planner A/B**
+ - enable LLM evidence-unit planner
+ - measure whether the answer layer uses retrieved evidence better
+
+5. **LLM channel planner A/B**
+ - test mainly on multi/aggregation/temporal error clusters
+ - verify coverage evidence supplements main facts instead of replacing them
+
+6. **Query graph builder A/B**
+ - validate the ceiling of query-graph-first retrieval
+ - if effective, distill the behavior into query-understanding or graph scorer training
+
+### 5.3 Parallel Scale Guidance
+
+Parallelism should not be determined only by the number of API keys. Also watch:
+
+- GPU memory
+- CPU memory
+- writer latency
+- answer-layer latency
+- graph ingest / SQLite write overhead
+- average writer calls per shard
+
+Scale gradually:
+
+```text
+5 shards smoke -> 10 shards stable -> 20 shards stress -> 30 shards only if no memory/API/IO issue
+```
+
+If error rate rises, memory drops sharply, API 402/429 appears, chunk errors occur, or shards stall, reduce parallelism first and then resume missing samples.
+
+## 6. Recommended Experiment Matrix
+
+Minimal interpretable matrix:
+
+| experiment | Embedder | LLM planner | purpose |
+| --- | --- | --- | --- |
+| baseline | off | off | fixed main graph-model baseline |
+| embedder-pre | pre-recall on | off | test candidate expansion |
+| embedder-fusion | pre-recall + fusion on | off | test semantic fusion |
+| evidence-unit | off | evidence-unit on | test pre-answer evidence organization |
+| channel-planner | off | channel planner on | test main/coverage separation |
+| query-graph | off | query graph on | test query-graph retrieval |
+| combined-light | pre-recall on | evidence-unit on | test lower-cost combined path |
+| combined-heavy | pre-recall + fusion on | evidence-unit + channel planner on | test capability ceiling |
+
+Each run should preserve:
+
+- predictions
+- samples_debug
+- judge output
+- by-task accuracy
+- writer calls
+- retrieval latency
+- answer latency
+- per-sample error type
+
+This makes it possible to separate recall errors, evidence-selection errors, planner errors, answer-layer errors, and parallel-runtime instability.
diff --git a/docs/OPTIONAL_MODULES_AND_PARALLEL.zh-CN.md b/docs/OPTIONAL_MODULES_AND_PARALLEL.zh-CN.md
index d28949515b9a91a1253c7e33d8953a2d9bf2c20a..437778a577a247ccd98566b302a192e25e46fc77 100644
--- a/docs/OPTIONAL_MODULES_AND_PARALLEL.zh-CN.md
+++ b/docs/OPTIONAL_MODULES_AND_PARALLEL.zh-CN.md
@@ -1,345 +1,345 @@
-# 可开启模块接入与并行推进说明
-
-本文说明当前 TMCRA 包中已经保留的两个可开启接入口：
-
-- Embedder 接入口
-- LLM planner 接入口
-
-同时说明 S500 基线测试中使用过的并行推进方式，方便后续在部署、评估或消融实验中复用。
-
-## 1. 当前主链路
-
-冻结 S500 基线的核心链路是：
-
-```text
-dialogue -> writer layer -> graph memory -> learned node/path scorer -> evidence selection -> answer layer
-```
-
-其中：
-
-- 写入层负责把对话转成记忆节点、事件单元、profile 信号、时间信号。
-- 图记忆层保存节点、路径和隧穿关系。
-- `node_scorer.pt` 和 `path_scorer.pt` 负责学习式节点/路径打分。
-- evidence selection 把候选记忆整理成紧凑证据。
-- 回答层 LLM 根据证据生成最终回复。
-
-Embedder 和 LLM planner 都是可开启增强模块，不应该替代主图模型。它们更适合作为辅助通道、对比实验或高成本部署路径。
-
-## 2. Embedder 接入口
-
-Embedder 当前有三类接入位置。
-
-### 2.1 写入阶段索引
-
-写入阶段可以为新写入的记忆建立 embedding 索引，后续召回时作为辅助候选来源。
-
-相关配置：
-
-```bash
-export TMCRA_EMBEDDER_MODEL_PATH="BAAI/bge-m3"
-export TMCRA_EMBEDDER_DEVICE="cuda"
-export TMCRA_EMBEDDER_MODEL_MAX_LENGTH="512"
-export TMCRA_WRITE_EMBEDDER_INDEX_MODE="bge_m3"
-export TMCRA_WRITE_EMBEDDER_INDEX_MAX_TERMS="96"
-```
-
-作用：
-
-- 在 writer 写入记忆节点后，为节点文本建立语义索引。
-- 不改变原图结构。
-- 不替代 node/path scorer，只是给召回增加一条语义候选通道。
-
-### 2.2 召回前候选补充
-
-召回前可以先用 embedder 找到一批候选 event id，再交给图召回和 scorer 做后续排序。
-
-相关配置：
-
-```bash
-export TMCRA_EMBEDDER_PRE_RECALL_MODE="bge_m3"
-export TMCRA_EMBEDDER_PRE_RECALL_K="16"
-export TMCRA_EMBEDDER_INDEX_RECALL_MODE="bge_m3"
-export TMCRA_EMBEDDER_INDEX_RECALL_K="24"
-```
-
-作用：
-
-- 帮助召回阶段扩大候选范围。
-- 对语义相近但图路径弱的记忆提供补充入口。
-- 适合测试 query 与 memory 表达不完全一致的场景。
-
-### 2.3 召回后融合加权
-
-召回后可以把 embedder 命中的 event 与主图模型结果融合，让高语义相关的节点获得有限 boost。
-
-相关配置：
-
-```bash
-export TMCRA_EMBEDDER_FUSION_MODE="on"
-export TMCRA_EMBEDDER_FUSION_WEIGHT="0.35"
-export TMCRA_EMBEDDER_FUSION_SCORE_FLOOR="0.62"
-export TMCRA_EMBEDDER_FUSION_TOP_K="16"
-export TMCRA_EMBEDDER_FUSION_SELECT_K="4"
-export TMCRA_EMBEDDER_FUSION_MAX_BOOST="0.42"
-```
-
-作用：
-
-- 给语义相似候选增加有限分数。
-- 避免 embedder 直接重排主证据。
-- 适合作为主图 scorer 的辅助召回层。
-
-## 3. LLM Planner 接入口
-
-LLM planner 当前主要有三类接入方式。它们都位于召回之后或 query 进入召回之前，用于增强证据组织能力。
-
-### 3.1 Evidence-unit planner
-
-Evidence-unit planner 在召回后运行，用 LLM 把候选窗口整理成 evidence unit。
-
-相关配置：
-
-```bash
-export TMCRA_EVIDENCE_UNIT_PLANNER_MODE="on"
-export TMCRA_EVIDENCE_UNIT_PLANNER_BASE_URL=""
-export TMCRA_EVIDENCE_UNIT_PLANNER_MODEL=""
-export TMCRA_EVIDENCE_UNIT_PLANNER_API_KEY=""
-export TMCRA_EVIDENCE_UNIT_PLANNER_MAX_CANDIDATES="10"
-export TMCRA_EVIDENCE_UNIT_PLANNER_CHARS="1100"
-export TMCRA_EVIDENCE_UNIT_PLANNER_MAX_TOKENS="760"
-export TMCRA_EVIDENCE_UNIT_PLANNER_REORDER="0"
-```
-
-如果不单独设置 planner 的 base/model/key，它会继承回答层配置：
-
-```bash
-export TMCRA_ANSWER_BASE_URL=""
-export TMCRA_ANSWER_MODEL=""
-export TMCRA_ANSWER_API_KEY=""
-```
-
-作用：
-
-- 标注候选窗口里的 answer unit、positive evidence、temporal anchor、current value、old value、constraint、negative evidence。
-- 帮助最终回答层理解“这批证据应该怎么用”。
-- 默认更适合做证据整理，不建议让它直接替代图召回。
-
-### 3.2 LLM channel planner
-
-LLM channel planner 在最终证据进入回答层前运行，用 LLM 区分 main evidence、coverage evidence、support evidence 和 suppress evidence。
-
-相关配置：
-
-```bash
-export TMCRA_LLM_CHANNEL_PLANNER_MODE="on"
-export TMCRA_LLM_CHANNEL_PLANNER_MAX_WINDOWS="16"
-export TMCRA_LLM_CHANNEL_PLANNER_WINDOW_CHARS="520"
-export TMCRA_LLM_CHANNEL_PLANNER_MAX_TOKENS="700"
-```
-
-作用：
-
-- 让 coverage 证据补充主事实，而不是替代主事实。
-- 对 count/sum/ratio/duration/multi-unit 问题特别有用。
-- 成本高于纯模型 scorer，适合高质量模式或实验开关。
-
-冻结 S500 基线记录中，该项为：
-
-```text
-llm_channel_planner=off
-```
-
-### 3.3 Query graph builder
-
-Query graph builder 在召回前运行，把用户问题转成 query graph，再扩展为 sidecar retrieval queries。
-
-相关配置：
-
-```bash
-export TMCRA_QUERY_GRAPH_BUILDER_MODE="on"
-export TMCRA_QUERY_GRAPH_BASE_URL=""
-export TMCRA_QUERY_GRAPH_MODEL=""
-export TMCRA_QUERY_GRAPH_API_KEY=""
-export TMCRA_QUERY_GRAPH_MAX_TOKENS="700"
-export TMCRA_QUERY_GRAPH_SIDECAR_RETRIEVAL_MODE="on"
-export TMCRA_QUERY_GRAPH_SIDECAR_MAX_QUERIES="6"
-export TMCRA_QUERY_GRAPH_SIDECAR_TOP_K="4"
-```
-
-作用：
-
-- 把问题拆成 task intent、required units、operation、tunnel needs。
-- 对复杂 multi-session、temporal、profile 问题提供更明确的召回方向。
-- 适合做对比实验，观察“问题建图后再召回”是否提升候选命中。
-
-## 4. 本地模型 planner 与 LLM planner 的区别
-
-当前代码里也有本地模型 planner 接口，例如：
-
-```bash
-export TMCRA_ANSWER_WINDOW_PLANNER_MODE="on"
-export TMCRA_ANSWER_WINDOW_PLANNER_MODEL_PATH=""
-export TMCRA_UNIFIED_OPERATION_PLANNER_MODE="on"
-export TMCRA_UNIFIED_OPERATION_PLANNER_MODEL_PATH=""
-export TMCRA_INJECTION_PLANNER_MODE="guided"
-export TMCRA_INJECTION_PLANNER_MODEL_PATH=""
-```
-
-这些是本地模型接入口，不是 LLM planner。区别是：
-
-- LLM planner：调用外部或本地 LLM，成本更高，适合验证能力上限。
-- 本地模型 planner：成本更低，更适合产品化，但需要专项训练和稳定性验证。
-
-建议流程是：
-
-```text
-先用 LLM planner 验证能力是否有效 -> 再把有效行为蒸馏/训练进本地图模型或 planner head
-```
-
-## 5. 并行推进方案
-
-S500 基线采用过分片并行方式：
-
-```text
-500 samples -> 10 shards -> 50 samples per shard
-```
-
-每个 shard 独立运行：
-
-```text
-input_shard_N.json -> shard_N/ -> predictions/debug/summary
-```
-
-核心并行原则：
-
-- 每个 shard 独立进程。
-- 每个 shard 独立输出目录。
-- writer key pool 按 shard index 轮转。
-- 主模型权重只读共享。
-- 最终合并 predictions、samples_debug、judge 结果。
-
-冻结 S500 记录中的关键运行配置：
-
-```text
-samples=500
-shards=10
-per_shard=50
-writer=DeepSeek v4 Flash
-answer_layer=GPT5.4
-llm_channel_planner=off
-history_mode=controlled_answer_plus_distractors
-```
-
-### 5.1 复用的并行模板
-
-推荐的并行模板：
-
-```bash
-export TMCRA_RETRIEVAL_MODE="hybrid_node_scored"
-export TMCRA_REQUIRE_LEARNED_SCORER="1"
-export TMCRA_NODE_MODEL_DEVICE="cuda"
-export TMCRA_NODE_MODEL_PATH="models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer.pt"
-export TMCRA_PATH_MODEL_PATH="models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer.pt"
-
-export TMCRA_WRITER_MODEL="deepseek-chat"
-export TMCRA_WRITER_MAX_TOKENS="512"
-export TMCRA_WRITER_TIMEOUT_SECONDS="180"
-export TMCRA_WRITER_TEMPERATURE="0"
-export TMCRA_WRITER_INPUT_MODE="delta"
-export TMCRA_WRITER_MAX_PROPOSALS="2"
-
-export TMCRA_ANSWER_MAX_TOKENS="512"
-```
-
-单 shard 执行形态：
-
-```bash
-python code/run_lme_s10_native_tmcra.py \
- --data "/input_shard_N.json" \
- --repo "" \
- --service-root "" \
- --out "/shard_N" \
- --limit 50 \
- --top-k 10 \
- --max-distractor-sessions 5 \
- --max-distractor-chunks 1 \
- --max-answer-chunks 4 \
- --chunk-chars 7000
-```
-
-### 5.2 推进顺序
-
-建议按下面顺序推进，不要一次把所有模块全打开：
-
-1. **Baseline scorer-only**
- - embedder off
- - LLM channel planner off
- - query graph builder off
- - 用于确认 frozen baseline 是否稳定。
-
-2. **Embedder pre-recall A/B**
- - 只打开写入索引和召回前候选补充。
- - 观察候选命中率、召回延迟、错误类型是否变化。
-
-3. **Embedder fusion A/B**
- - 在 pre-recall 稳定后打开 fusion。
- - 控制 boost，不允许 embedder 直接压过主图 scorer。
-
-4. **Evidence-unit planner A/B**
- - 打开 LLM evidence-unit planner。
- - 观察 answer 层是否更会使用召回证据。
-
-5. **LLM channel planner A/B**
- - 只在 multi/aggregation/temporal 错误集中验证。
- - 重点观察 coverage 证据是否补充主事实，而不是替换主事实。
-
-6. **Query graph builder A/B**
- - 用于验证“问题建图后再召回”的上限。
- - 如果有效，再考虑训练进 query-understanding 或 graph scorer。
-
-### 5.3 并行规模建议
-
-并行数不要只看 API 数量，还要看：
-
-- GPU 显存
-- CPU 内存
-- writer 延迟
-- answer 层延迟
-- graph ingest/SQLite 写入开销
-- 每 shard 平均 writer calls
-
-建议从小到大：
-
-```text
-5 shards smoke -> 10 shards stable -> 20 shards stress -> 30 shards only if no memory/API/IO issue
-```
-
-如果出现错误率升高、内存下降明显、API 402/429、chunk error 或 shard 卡住，应先降并行，再补跑缺失样本。
-
-## 6. 推荐实验矩阵
-
-最小可解释矩阵：
-
-| 实验 | Embedder | LLM planner | 目的 |
-| --- | --- | --- | --- |
-| baseline | off | off | 固定主图模型基线 |
-| embedder-pre | pre-recall on | off | 测候选扩展是否提升 |
-| embedder-fusion | pre-recall + fusion on | off | 测语义融合是否提升 |
-| evidence-unit | off | evidence-unit on | 测回答前证据整理 |
-| channel-planner | off | channel planner on | 测 main/coverage 分离 |
-| query-graph | off | query graph on | 测问题建图召回 |
-| combined-light | pre-recall on | evidence-unit on | 测较低成本组合 |
-| combined-heavy | pre-recall + fusion on | evidence-unit + channel planner on | 测能力上限 |
-
-每一组都应保留：
-
-- predictions
-- samples_debug
-- judge output
-- by-task accuracy
-- writer calls
-- retrieval latency
-- answer latency
-- per-sample error type
-
-这样后续可以判断问题来自召回、证据选择、planner、回答层，还是并行运行不稳定。
+# 可开启模块接入与并行推进说明
+
+本文说明当前 TMCRA 包中已经保留的两个可开启接入口：
+
+- Embedder 接入口
+- LLM planner 接入口
+
+同时说明 S500 基线测试中使用过的并行推进方式，方便后续在部署、评估或消融实验中复用。
+
+## 1. 当前主链路
+
+冻结 S500 基线的核心链路是：
+
+```text
+dialogue -> writer layer -> graph memory -> learned node/path scorer -> evidence selection -> answer layer
+```
+
+其中：
+
+- 写入层负责把对话转成记忆节点、事件单元、profile 信号、时间信号。
+- 图记忆层保存节点、路径和隧穿关系。
+- `node_scorer.pt` 和 `path_scorer.pt` 负责学习式节点/路径打分。
+- evidence selection 把候选记忆整理成紧凑证据。
+- 回答层 LLM 根据证据生成最终回复。
+
+Embedder 和 LLM planner 都是可开启增强模块，不应该替代主图模型。它们更适合作为辅助通道、对比实验或高成本部署路径。
+
+## 2. Embedder 接入口
+
+Embedder 当前有三类接入位置。
+
+### 2.1 写入阶段索引
+
+写入阶段可以为新写入的记忆建立 embedding 索引，后续召回时作为辅助候选来源。
+
+相关配置：
+
+```bash
+export TMCRA_EMBEDDER_MODEL_PATH="BAAI/bge-m3"
+export TMCRA_EMBEDDER_DEVICE="cuda"
+export TMCRA_EMBEDDER_MODEL_MAX_LENGTH="512"
+export TMCRA_WRITE_EMBEDDER_INDEX_MODE="bge_m3"
+export TMCRA_WRITE_EMBEDDER_INDEX_MAX_TERMS="96"
+```
+
+作用：
+
+- 在 writer 写入记忆节点后，为节点文本建立语义索引。
+- 不改变原图结构。
+- 不替代 node/path scorer，只是给召回增加一条语义候选通道。
+
+### 2.2 召回前候选补充
+
+召回前可以先用 embedder 找到一批候选 event id，再交给图召回和 scorer 做后续排序。
+
+相关配置：
+
+```bash
+export TMCRA_EMBEDDER_PRE_RECALL_MODE="bge_m3"
+export TMCRA_EMBEDDER_PRE_RECALL_K="16"
+export TMCRA_EMBEDDER_INDEX_RECALL_MODE="bge_m3"
+export TMCRA_EMBEDDER_INDEX_RECALL_K="24"
+```
+
+作用：
+
+- 帮助召回阶段扩大候选范围。
+- 对语义相近但图路径弱的记忆提供补充入口。
+- 适合测试 query 与 memory 表达不完全一致的场景。
+
+### 2.3 召回后融合加权
+
+召回后可以把 embedder 命中的 event 与主图模型结果融合，让高语义相关的节点获得有限 boost。
+
+相关配置：
+
+```bash
+export TMCRA_EMBEDDER_FUSION_MODE="on"
+export TMCRA_EMBEDDER_FUSION_WEIGHT="0.35"
+export TMCRA_EMBEDDER_FUSION_SCORE_FLOOR="0.62"
+export TMCRA_EMBEDDER_FUSION_TOP_K="16"
+export TMCRA_EMBEDDER_FUSION_SELECT_K="4"
+export TMCRA_EMBEDDER_FUSION_MAX_BOOST="0.42"
+```
+
+作用：
+
+- 给语义相似候选增加有限分数。
+- 避免 embedder 直接重排主证据。
+- 适合作为主图 scorer 的辅助召回层。
+
+## 3. LLM Planner 接入口
+
+LLM planner 当前主要有三类接入方式。它们都位于召回之后或 query 进入召回之前，用于增强证据组织能力。
+
+### 3.1 Evidence-unit planner
+
+Evidence-unit planner 在召回后运行，用 LLM 把候选窗口整理成 evidence unit。
+
+相关配置：
+
+```bash
+export TMCRA_EVIDENCE_UNIT_PLANNER_MODE="on"
+export TMCRA_EVIDENCE_UNIT_PLANNER_BASE_URL=""
+export TMCRA_EVIDENCE_UNIT_PLANNER_MODEL=""
+export TMCRA_EVIDENCE_UNIT_PLANNER_API_KEY=""
+export TMCRA_EVIDENCE_UNIT_PLANNER_MAX_CANDIDATES="10"
+export TMCRA_EVIDENCE_UNIT_PLANNER_CHARS="1100"
+export TMCRA_EVIDENCE_UNIT_PLANNER_MAX_TOKENS="760"
+export TMCRA_EVIDENCE_UNIT_PLANNER_REORDER="0"
+```
+
+如果不单独设置 planner 的 base/model/key，它会继承回答层配置：
+
+```bash
+export TMCRA_ANSWER_BASE_URL=""
+export TMCRA_ANSWER_MODEL=""
+export TMCRA_ANSWER_API_KEY=""
+```
+
+作用：
+
+- 标注候选窗口里的 answer unit、positive evidence、temporal anchor、current value、old value、constraint、negative evidence。
+- 帮助最终回答层理解“这批证据应该怎么用”。
+- 默认更适合做证据整理，不建议让它直接替代图召回。
+
+### 3.2 LLM channel planner
+
+LLM channel planner 在最终证据进入回答层前运行，用 LLM 区分 main evidence、coverage evidence、support evidence 和 suppress evidence。
+
+相关配置：
+
+```bash
+export TMCRA_LLM_CHANNEL_PLANNER_MODE="on"
+export TMCRA_LLM_CHANNEL_PLANNER_MAX_WINDOWS="16"
+export TMCRA_LLM_CHANNEL_PLANNER_WINDOW_CHARS="520"
+export TMCRA_LLM_CHANNEL_PLANNER_MAX_TOKENS="700"
+```
+
+作用：
+
+- 让 coverage 证据补充主事实，而不是替代主事实。
+- 对 count/sum/ratio/duration/multi-unit 问题特别有用。
+- 成本高于纯模型 scorer，适合高质量模式或实验开关。
+
+冻结 S500 基线记录中，该项为：
+
+```text
+llm_channel_planner=off
+```
+
+### 3.3 Query graph builder
+
+Query graph builder 在召回前运行，把用户问题转成 query graph，再扩展为 sidecar retrieval queries。
+
+相关配置：
+
+```bash
+export TMCRA_QUERY_GRAPH_BUILDER_MODE="on"
+export TMCRA_QUERY_GRAPH_BASE_URL=""
+export TMCRA_QUERY_GRAPH_MODEL=""
+export TMCRA_QUERY_GRAPH_API_KEY=""
+export TMCRA_QUERY_GRAPH_MAX_TOKENS="700"
+export TMCRA_QUERY_GRAPH_SIDECAR_RETRIEVAL_MODE="on"
+export TMCRA_QUERY_GRAPH_SIDECAR_MAX_QUERIES="6"
+export TMCRA_QUERY_GRAPH_SIDECAR_TOP_K="4"
+```
+
+作用：
+
+- 把问题拆成 task intent、required units、operation、tunnel needs。
+- 对复杂 multi-session、temporal、profile 问题提供更明确的召回方向。
+- 适合做对比实验，观察“问题建图后再召回”是否提升候选命中。
+
+## 4. 本地模型 planner 与 LLM planner 的区别
+
+当前代码里也有本地模型 planner 接口，例如：
+
+```bash
+export TMCRA_ANSWER_WINDOW_PLANNER_MODE="on"
+export TMCRA_ANSWER_WINDOW_PLANNER_MODEL_PATH=""
+export TMCRA_UNIFIED_OPERATION_PLANNER_MODE="on"
+export TMCRA_UNIFIED_OPERATION_PLANNER_MODEL_PATH=""
+export TMCRA_INJECTION_PLANNER_MODE="guided"
+export TMCRA_INJECTION_PLANNER_MODEL_PATH=""
+```
+
+这些是本地模型接入口，不是 LLM planner。区别是：
+
+- LLM planner：调用外部或本地 LLM，成本更高，适合验证能力上限。
+- 本地模型 planner：成本更低，更适合产品化，但需要专项训练和稳定性验证。
+
+建议流程是：
+
+```text
+先用 LLM planner 验证能力是否有效 -> 再把有效行为蒸馏/训练进本地图模型或 planner head
+```
+
+## 5. 并行推进方案
+
+S500 基线采用过分片并行方式：
+
+```text
+500 samples -> 10 shards -> 50 samples per shard
+```
+
+每个 shard 独立运行：
+
+```text
+input_shard_N.json -> shard_N/ -> predictions/debug/summary
+```
+
+核心并行原则：
+
+- 每个 shard 独立进程。
+- 每个 shard 独立输出目录。
+- writer key pool 按 shard index 轮转。
+- 主模型权重只读共享。
+- 最终合并 predictions、samples_debug、judge 结果。
+
+冻结 S500 记录中的关键运行配置：
+
+```text
+samples=500
+shards=10
+per_shard=50
+writer=DeepSeek v4 Flash
+answer_layer=GPT5.4
+llm_channel_planner=off
+history_mode=controlled_answer_plus_distractors
+```
+
+### 5.1 复用的并行模板
+
+推荐的并行模板：
+
+```bash
+export TMCRA_RETRIEVAL_MODE="hybrid_node_scored"
+export TMCRA_REQUIRE_LEARNED_SCORER="1"
+export TMCRA_NODE_MODEL_DEVICE="cuda"
+export TMCRA_NODE_MODEL_PATH="models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer.pt"
+export TMCRA_PATH_MODEL_PATH="models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer.pt"
+
+export TMCRA_WRITER_MODEL="deepseek-chat"
+export TMCRA_WRITER_MAX_TOKENS="512"
+export TMCRA_WRITER_TIMEOUT_SECONDS="180"
+export TMCRA_WRITER_TEMPERATURE="0"
+export TMCRA_WRITER_INPUT_MODE="delta"
+export TMCRA_WRITER_MAX_PROPOSALS="2"
+
+export TMCRA_ANSWER_MAX_TOKENS="512"
+```
+
+单 shard 执行形态：
+
+```bash
+python code/run_lme_s10_native_tmcra.py \
+ --data "/input_shard_N.json" \
+ --repo "" \
+ --service-root "" \
+ --out "/shard_N" \
+ --limit 50 \
+ --top-k 10 \
+ --max-distractor-sessions 5 \
+ --max-distractor-chunks 1 \
+ --max-answer-chunks 4 \
+ --chunk-chars 7000
+```
+
+### 5.2 推进顺序
+
+建议按下面顺序推进，不要一次把所有模块全打开：
+
+1. **Baseline scorer-only**
+ - embedder off
+ - LLM channel planner off
+ - query graph builder off
+ - 用于确认 frozen baseline 是否稳定。
+
+2. **Embedder pre-recall A/B**
+ - 只打开写入索引和召回前候选补充。
+ - 观察候选命中率、召回延迟、错误类型是否变化。
+
+3. **Embedder fusion A/B**
+ - 在 pre-recall 稳定后打开 fusion。
+ - 控制 boost，不允许 embedder 直接压过主图 scorer。
+
+4. **Evidence-unit planner A/B**
+ - 打开 LLM evidence-unit planner。
+ - 观察 answer 层是否更会使用召回证据。
+
+5. **LLM channel planner A/B**
+ - 只在 multi/aggregation/temporal 错误集中验证。
+ - 重点观察 coverage 证据是否补充主事实，而不是替换主事实。
+
+6. **Query graph builder A/B**
+ - 用于验证“问题建图后再召回”的上限。
+ - 如果有效，再考虑训练进 query-understanding 或 graph scorer。
+
+### 5.3 并行规模建议
+
+并行数不要只看 API 数量，还要看：
+
+- GPU 显存
+- CPU 内存
+- writer 延迟
+- answer 层延迟
+- graph ingest/SQLite 写入开销
+- 每 shard 平均 writer calls
+
+建议从小到大：
+
+```text
+5 shards smoke -> 10 shards stable -> 20 shards stress -> 30 shards only if no memory/API/IO issue
+```
+
+如果出现错误率升高、内存下降明显、API 402/429、chunk error 或 shard 卡住，应先降并行，再补跑缺失样本。
+
+## 6. 推荐实验矩阵
+
+最小可解释矩阵：
+
+| 实验 | Embedder | LLM planner | 目的 |
+| --- | --- | --- | --- |
+| baseline | off | off | 固定主图模型基线 |
+| embedder-pre | pre-recall on | off | 测候选扩展是否提升 |
+| embedder-fusion | pre-recall + fusion on | off | 测语义融合是否提升 |
+| evidence-unit | off | evidence-unit on | 测回答前证据整理 |
+| channel-planner | off | channel planner on | 测 main/coverage 分离 |
+| query-graph | off | query graph on | 测问题建图召回 |
+| combined-light | pre-recall on | evidence-unit on | 测较低成本组合 |
+| combined-heavy | pre-recall + fusion on | evidence-unit + channel planner on | 测能力上限 |
+
+每一组都应保留：
+
+- predictions
+- samples_debug
+- judge output
+- by-task accuracy
+- writer calls
+- retrieval latency
+- answer latency
+- per-sample error type
+
+这样后续可以判断问题来自召回、证据选择、planner、回答层，还是并行运行不稳定。
diff --git a/docs/TRAINING.md b/docs/TRAINING.md
index 18f47f409153c3851883f0ec0772c9e2fb1c6ad8..d40d833ba6accfc11b3860ef0c9d971891195e20 100644
--- a/docs/TRAINING.md
+++ b/docs/TRAINING.md
@@ -1,113 +1,113 @@
-# TMCRA Training Notes
-
-This document summarizes the training direction behind the graph scorer package included in this release.
-
-## Training Goal
-
-TMCRA trains graph-scoring components for long-memory retrieval. The goal is to help the runtime decide which memory nodes and graph paths should be surfaced to an answer model for a given user query.
-
-The trained model is not intended to replace the answer LLM. It is responsible for memory selection:
-
-- identify relevant memory nodes
-- score graph paths between related memory events
-- preserve useful cross-turn and cross-session links
-- reduce noisy or stale evidence before answer generation
-
-## Model Components
-
-The released model directory contains two main runtime scorers:
-
-- `node_scorer.pt`: scores candidate memory nodes.
-- `path_scorer.pt`: scores graph paths and tunnel links between memory nodes.
-
-The training output also includes:
-
-- best checkpoints
-- last checkpoints
-- epoch and step checkpoints
-- training summary and logs
-- export manifest
-
-## Training Data Direction
-
-Training data is built around dialogue-memory behavior rather than isolated QA pairs. Samples are designed to teach the model how memory should connect across turns and sessions.
-
-The major training directions include:
-
-- direct user facts
-- assistant-provided details
-- preference/profile signals
-- temporal state changes
-- old-value vs current-value selection
-- cross-session event links
-- multi-evidence aggregation
-- evidence-positive vs noise/negative memory separation
-- unit-to-unit coverage for count/sum/compare tasks
-
-## Graph Memory Supervision
-
-Each training example is converted into graph-oriented supervision. Instead of only asking whether a text chunk is relevant, TMCRA trains over:
-
-- memory node relevance
-- event-unit relevance
-- path usefulness
-- tunnel/link usefulness
-- evidence role
-- currentness and temporal state
-- whether a candidate should be injected into answer context
-
-This allows the runtime to learn memory structure, not only lexical similarity.
-
-## Writer and Scorer Separation
-
-TMCRA separates memory writing from graph scoring.
-
-The writer extracts candidate memory records from dialogue. The graph model then learns how those records should be selected and connected during retrieval.
-
-This separation is important because a long-memory system needs two different abilities:
-
-- write useful memory units from conversation
-- retrieve and connect the right units later under noise
-
-## Training Output Included
-
-The packaged model output is located at:
-
-```text
-models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/
-```
-
-Runtime files:
-
-```text
-node_scorer.pt
-path_scorer.pt
-export_manifest.json
-```
-
-Full training trace:
-
-```text
-checkpoints/
-node_scorer_best.pt
-path_scorer_best.pt
-node_scorer_last.pt
-path_scorer_last.pt
-train_summary.json
-train.log
-training_issues.jsonl
-```
-
-## Current Training Lessons
-
-The current baseline shows that TMCRA has strong single-session fact recall and assistant-detail recall. It also has working temporal and preference layers.
-
-The main remaining training targets are:
-
-- stronger multi-session aggregation
-- better unit coverage for count/sum/compare questions
-- deeper temporal graph planning
-- query-graph to memory-graph matching
-- more stable preference abstraction under indirect user requests
-
-These directions are the next step for improving TMCRA from a working long-memory runtime into a stronger general agent-memory layer.
+# TMCRA Training Notes
+
+This document summarizes the training direction behind the graph scorer package included in this release.
+
+## Training Goal
+
+TMCRA trains graph-scoring components for long-memory retrieval. The goal is to help the runtime decide which memory nodes and graph paths should be surfaced to an answer model for a given user query.
+
+The trained model is not intended to replace the answer LLM. It is responsible for memory selection:
+
+- identify relevant memory nodes
+- score graph paths between related memory events
+- preserve useful cross-turn and cross-session links
+- reduce noisy or stale evidence before answer generation
+
+## Model Components
+
+The released model directory contains two main runtime scorers:
+
+- `node_scorer.pt`: scores candidate memory nodes.
+- `path_scorer.pt`: scores graph paths and tunnel links between memory nodes.
+
+The training output also includes:
+
+- best checkpoints
+- last checkpoints
+- epoch and step checkpoints
+- training summary and logs
+- export manifest
+
+## Training Data Direction
+
+Training data is built around dialogue-memory behavior rather than isolated QA pairs. Samples are designed to teach the model how memory should connect across turns and sessions.
+
+The major training directions include:
+
+- direct user facts
+- assistant-provided details
+- preference/profile signals
+- temporal state changes
+- old-value vs current-value selection
+- cross-session event links
+- multi-evidence aggregation
+- evidence-positive vs noise/negative memory separation
+- unit-to-unit coverage for count/sum/compare tasks
+
+## Graph Memory Supervision
+
+Each training example is converted into graph-oriented supervision. Instead of only asking whether a text chunk is relevant, TMCRA trains over:
+
+- memory node relevance
+- event-unit relevance
+- path usefulness
+- tunnel/link usefulness
+- evidence role
+- currentness and temporal state
+- whether a candidate should be injected into answer context
+
+This allows the runtime to learn memory structure, not only lexical similarity.
+
+## Writer and Scorer Separation
+
+TMCRA separates memory writing from graph scoring.
+
+The writer extracts candidate memory records from dialogue. The graph model then learns how those records should be selected and connected during retrieval.
+
+This separation is important because a long-memory system needs two different abilities:
+
+- write useful memory units from conversation
+- retrieve and connect the right units later under noise
+
+## Training Output Included
+
+The packaged model output is located at:
+
+```text
+models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/
+```
+
+Runtime files:
+
+```text
+node_scorer.pt
+path_scorer.pt
+export_manifest.json
+```
+
+Full training trace:
+
+```text
+checkpoints/
+node_scorer_best.pt
+path_scorer_best.pt
+node_scorer_last.pt
+path_scorer_last.pt
+train_summary.json
+train.log
+training_issues.jsonl
+```
+
+## Current Training Lessons
+
+The current baseline shows that TMCRA has strong single-session fact recall and assistant-detail recall. It also has working temporal and preference layers.
+
+The main remaining training targets are:
+
+- stronger multi-session aggregation
+- better unit coverage for count/sum/compare questions
+- deeper temporal graph planning
+- query-graph to memory-graph matching
+- more stable preference abstraction under indirect user requests
+
+These directions are the next step for improving TMCRA from a working long-memory runtime into a stronger general agent-memory layer.
diff --git a/github_push.err.log b/github_push.err.log
deleted file mode 100644
index afa699a4e41207493b6a3e5d689de8b6044beac8..0000000000000000000000000000000000000000
--- a/github_push.err.log
+++ /dev/null
@@ -1,2 +0,0 @@
-To https://github.com/reshuibuduo/TMCRA-agent-memory-algorithm.git
- * [new branch] master -> master
diff --git a/github_push.out.log b/github_push.out.log
deleted file mode 100644
index 93d272536f1b963ba118c6767c72d41113b18c4d..0000000000000000000000000000000000000000
--- a/github_push.out.log
+++ /dev/null
@@ -1,2 +0,0 @@
-Uploading LFS objects: 100% (27/27), 5.6 GB | 0 B/s, done.
-branch 'master' set up to track 'origin/master'.
diff --git a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/node_scorer_epoch_01.pt b/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/node_scorer_epoch_01.pt
deleted file mode 100644
index b8d932ac199523794a82f1b102cfbf5230150b1c..0000000000000000000000000000000000000000
--- a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/node_scorer_epoch_01.pt
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:8aaa0153ae0642e709d287b947403493b461594dc8f7ac7f94b31b00aa32c7cf
-size 238030855
diff --git a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/node_scorer_epoch_02.pt b/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/node_scorer_epoch_02.pt
deleted file mode 100644
index b07e692691357a3c54a69ea54f5bd411f0dd56ad..0000000000000000000000000000000000000000
--- a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/node_scorer_epoch_02.pt
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:9eec8b8fcef26d82c9306d8f96b98915724e8c0d0967ba33ddcfba7dbf983cb4
-size 238030855
diff --git a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/node_scorer_epoch_03.pt b/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/node_scorer_epoch_03.pt
deleted file mode 100644
index 43bd8b3a99f1932d1299d6b93def63d6bb464bc8..0000000000000000000000000000000000000000
--- a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/node_scorer_epoch_03.pt
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:3137ad50512ef6890eecf0d241348844e2ae43c4a46649b4ade50f29f68c4c33
-size 238030855
diff --git a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/node_scorer_epoch_04.pt b/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/node_scorer_epoch_04.pt
deleted file mode 100644
index 2b89a22266522d979412869150970b0f1c6e28a7..0000000000000000000000000000000000000000
--- a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/node_scorer_epoch_04.pt
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:38c5e248f449080f4316c8f0a28ea8490bb2d9413afda7ce0d4324b256ce025a
-size 238030855
diff --git a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/node_scorer_epoch_05.pt b/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/node_scorer_epoch_05.pt
deleted file mode 100644
index eaf3ee6e76d61806a9ec58c44e6d4fa2a2966345..0000000000000000000000000000000000000000
--- a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/node_scorer_epoch_05.pt
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:61e6ee0e0f02de94c46b7d3bfb8bf954a7588282592fd6507068cb46942bca16
-size 238030855
diff --git a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/node_scorer_epoch_06.pt b/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/node_scorer_epoch_06.pt
deleted file mode 100644
index 39bb61468e1dcc49a081df605c281337585cf89e..0000000000000000000000000000000000000000
--- a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/node_scorer_epoch_06.pt
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:92af8e2ae8e55a29b9abcaffc02f37bb09544188f591b00fb5200fadfb46fbed
-size 238030855
diff --git a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/node_scorer_epoch_07.pt b/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/node_scorer_epoch_07.pt
deleted file mode 100644
index 1cbabe2049914ad9147d5701514a2a571905e4dd..0000000000000000000000000000000000000000
--- a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/node_scorer_epoch_07.pt
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:5dec148fe044f7f94694b4f72b49a01e7a74455f1bee17253beb53f6e6f94ce7
-size 238030855
diff --git a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/node_scorer_step_0000600.pt b/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/node_scorer_step_0000600.pt
deleted file mode 100644
index a18b37146f995301ad1e946caad82a4c404ae396..0000000000000000000000000000000000000000
--- a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/node_scorer_step_0000600.pt
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:29d3d1436839617bdda708090cac2727cb66c19dcdcf8634c6474f907b2c82c2
-size 238032983
diff --git a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/node_scorer_step_0000650.pt b/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/node_scorer_step_0000650.pt
deleted file mode 100644
index 7143dd95e95dac0cf2f4955ef9969cf655afde3a..0000000000000000000000000000000000000000
--- a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/node_scorer_step_0000650.pt
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:02077ad6b1df777cf8d9478a3fda234e7059340bce7d320f12184e89225e9510
-size 238032983
diff --git a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/path_scorer_epoch_01.pt b/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/path_scorer_epoch_01.pt
deleted file mode 100644
index 7efa2b3a20b3f6da8bdef5a631bedefe19861e6c..0000000000000000000000000000000000000000
--- a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/path_scorer_epoch_01.pt
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:28e8279e49f7dcc7a9edacf7c827de6bb2d0bc8ac3c2725a12dae314dd847ee1
-size 238030855
diff --git a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/path_scorer_epoch_02.pt b/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/path_scorer_epoch_02.pt
deleted file mode 100644
index 4dac2122ca5fe86786142c31e86c695e6b788acd..0000000000000000000000000000000000000000
--- a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/path_scorer_epoch_02.pt
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:7b895a3a0f2fdb0690ae9ea26b04f2c93a67b23a66a988fab2ad5db12b7c35c7
-size 238030855
diff --git a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/path_scorer_epoch_03.pt b/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/path_scorer_epoch_03.pt
deleted file mode 100644
index 4ab93f635c44837bbecb841c8beb15fbdcafe9a1..0000000000000000000000000000000000000000
--- a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/path_scorer_epoch_03.pt
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:6b6a0480a36e0665efd92edab06f51a3deec6195e72a09bdb253205f91cb4fec
-size 238030855
diff --git a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/path_scorer_epoch_04.pt b/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/path_scorer_epoch_04.pt
deleted file mode 100644
index f4f4da3dcf238fc469685d8a8671e60a77b2918f..0000000000000000000000000000000000000000
--- a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/path_scorer_epoch_04.pt
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:2b13d4ca8479fbdec63eebefdf40cbdabc700fb8074b8b7317a619b66260dafb
-size 238030855
diff --git a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/path_scorer_epoch_05.pt b/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/path_scorer_epoch_05.pt
deleted file mode 100644
index 81aee61dd2b7511231c42e39e81e74f1d607e739..0000000000000000000000000000000000000000
--- a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/path_scorer_epoch_05.pt
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:182fe18bb0288645186d13e677fbd2b560681794a007efd02de7ad21c39cf581
-size 238030855
diff --git a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/path_scorer_epoch_06.pt b/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/path_scorer_epoch_06.pt
deleted file mode 100644
index 0299bfaa25fad77fa954ba8a87485e1d1bcadba5..0000000000000000000000000000000000000000
--- a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/path_scorer_epoch_06.pt
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:a192607bc818123a9c3462c0283195cfd500c891b4a3a7e9b98426b016f5b07b
-size 238030855
diff --git a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/path_scorer_epoch_07.pt b/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/path_scorer_epoch_07.pt
deleted file mode 100644
index 1897e4119313898852ce86cc53e24b13430600fc..0000000000000000000000000000000000000000
--- a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/path_scorer_epoch_07.pt
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:5d1961ac8ee0bcb0d8f9e5ea82de282e990f26fbbe237e8bb8e45d60be827749
-size 238030855
diff --git a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/path_scorer_step_0000600.pt b/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/path_scorer_step_0000600.pt
deleted file mode 100644
index 2e48763a4e52ed3add7694eaa4bba15753a30e7c..0000000000000000000000000000000000000000
--- a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/path_scorer_step_0000600.pt
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:2fbbc7510ecc077fe9724593568ea829f5bf2b90e8e59388675052853c1d9d99
-size 238032983
diff --git a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/path_scorer_step_0000650.pt b/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/path_scorer_step_0000650.pt
deleted file mode 100644
index b4419f4eb13f408ad84003058b37c3052182b619..0000000000000000000000000000000000000000
--- a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints/path_scorer_step_0000650.pt
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:148b3461a74a390534a1cd5161555e5c310d11c2d0406e785654957eae9178b8
-size 238032983
diff --git a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/export_manifest.json b/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/export_manifest.json
deleted file mode 100644
index f34a4e1aeff5e66915489e2162722476a6962193..0000000000000000000000000000000000000000
--- a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/export_manifest.json
+++ /dev/null
@@ -1 +0,0 @@
-{"answer_calibration_variant": "competition_alignment_v3_trimaze", "answer_plan_variant": "candidate_event_answer_plan_v1", "checkpoint_audit": {"node": {"missing_prefixes": [], "ok": true, "path": "outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer.pt", "prefix_counts": {"event_head": 10, "event_tunnel_head": 4, "final_event_fusion_head": 4, "final_path_fusion_head": 4, "memory_router_head": 6, "path_head": 11, "path_tunnel_head": 4, "question_intent_head": 12, "temporal_head": 4}, "required_prefixes": ["memory_router_head", "temporal_head", "question_intent_head", "event_head", "path_head", "event_tunnel_head", "path_tunnel_head", "final_event_fusion_head", "final_path_fusion_head"], "saved_head": "event", "state_key_count": 448}, "path": {"missing_prefixes": [], "ok": true, "path": "outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer.pt", "prefix_counts": {"event_head": 10, "event_tunnel_head": 4, "final_event_fusion_head": 4, "final_path_fusion_head": 4, "memory_router_head": 6, "path_head": 11, "path_tunnel_head": 4, "question_intent_head": 12, "temporal_head": 4}, "required_prefixes": ["memory_router_head", "temporal_head", "question_intent_head", "event_head", "path_head", "event_tunnel_head", "path_tunnel_head", "final_event_fusion_head", "final_path_fusion_head"], "saved_head": "path", "state_key_count": 448}}, "completed_at": "2026-05-24T04:38:50", "created_at": "2026-05-23T20:23:49.564683+00:00", "dataset_version": "typed_tunnel_graph_v1", "encoder_attention_heads": 8, "encoder_attention_layers": 6, "encoder_structural_bias_variant": "tmcra_token_role_relbias_v1", "event_distractor_variant": "candidate_distractor_delta_bce_v1", "event_subgraph_attention_heads": 8, "event_subgraph_attention_layers": 6, "event_subgraph_refiner_variant": "event_centered_subgraph_attn_6l_h448", "focus_counts": {"action_frame_temporal_binding": 58, "cross_session_multi_event_chain": 104, "hard_distractor_similar_entity": 61, "normal_non_chain_stability": 16, "numeric_aggregation_unit_coverage": 116, "profile_boundary_mixed_case": 36, "same_entity_unit_to_unit_chain": 104, "status_negative_exclusion_update": 53}, "graph_count": 333, "input_jsonl": "outputs/action_frame_tunnel_final_accepted_20260524/typed_tunnel_accepted.jsonl", "memory_router_variant": "question_pooled_multilabel_v1", "memory_tunnel_variant": "chain_depth_tunnel_support_delta_v1", "message_passing_variant": "support_to_event_bidirectional_shared_typed_v1", "model_arch_version": "node_memory_v11_tmcra_scale49m_trimaze", "node_encoder_variant": "hashed_tmcra_token_attn_pool_6l_w256", "node_max_tokens": 32, "notes": ["Benchmark data is not an input to this builder.", "Typed tunnel signals are encoded as event/path metadata and existing graph node/path types.", "No new node type, edge type, path type, or model tensor shape is introduced.", "Train with train_locomo_node_memory.py and a compatible node-memory checkpoint."], "output_dir": "outputs/action_frame_tunnel_graph_dataset_548_event_units_20260524/dataset", "pair_feature_adapter_variant": "residual_pair_feature_adapter_v1", "query_counts": {"test": 54, "train": 439, "val": 55}, "question_encoder_variant": "hashed_tmcra_token_attn_pool_6l_w256", "question_intent_variant": "text_pooled_multihead_v1", "question_max_tokens": 48, "resume": {"resume_checkpoint": "/models/tmcra_node_memory_v11_priorfusion_zeroload_20260517/node_scorer.pt", "resume_mode": "weights_only_forced", "resume_weights_only": true}, "sample_count": 548, "sample_id_prefix": "actionframe548_", "skipped_count": 0, "source_data_dirs": ["outputs/action_frame_tunnel_graph_dataset_548_event_units_20260524/dataset"], "source_manifests": ["outputs/action_frame_tunnel_graph_dataset_548_event_units_20260524/dataset/export_manifest.json"], "split_policy": {"test_ratio": 0.09999999999999995, "train_ratio": 0.8, "val_ratio": 0.1}, "status": "completed", "trainable_stage": {"frozen_parameter_count": 47963050, "trainable_name_prefixes": ["event_tunnel_head", "final_event_fusion_head", "final_path_fusion_head", "path_tunnel_head"], "trainable_parameter_count": 3831302, "trainable_stage": "tunnel_fusion_only"}, "training_sampling": {"answer_type_counts": {"multi_evidence": 422, "time": 17}, "base_row_count": 439, "effective_row_count": 697, "loss_weight_range_estimate": {"max": 1.549193, "min": 1.319091}, "mode": "source_aware_balanced", "repeat_histogram": {"1": 118, "2": 144}, "source_count": 1, "source_details": {"dataset": {"avg_example_weight_estimate": 1.540283, "loss_source_factor": 1.0, "row_count": 439, "row_share": 1.0, "sampled_row_count_estimate": 697, "sampled_row_share_estimate": 1.0, "sampling_source_factor": 1.0, "weighted_loss_mass": 676.184129}}, "source_row_counts": {"dataset": 439}, "supervision_bucket_counts": {"typed_tunnel_multi_positive": 439}, "weighted_answer_mass_estimate": {"multi_evidence": 653.759589, "time": 22.42454}, "weighted_supervision_bucket_mass_estimate": {"typed_tunnel_multi_positive": 680.095876}}, "typed_tunnel_edge_count": 3855}
\ No newline at end of file
diff --git a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/launch_train.sh b/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/launch_train.sh
deleted file mode 100644
index 816a5e23a881832abe6ea8e2ae94be51f942b430..0000000000000000000000000000000000000000
--- a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/launch_train.sh
+++ /dev/null
@@ -1,48 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-cd
-DATA=outputs/action_frame_tunnel_graph_dataset_548_event_units_20260524/dataset
-OUT="$1"
-/.venv/bin/python scripts/train_locomo_node_memory.py \
- --data-dir "$DATA" \
- --output-dir "$OUT" \
- --resume-checkpoint /models/tmcra_node_memory_v11_priorfusion_zeroload_20260517/node_scorer.pt \
- --resume-weights-only \
- --trainable-stage tunnel_fusion_only \
- --epochs 8 \
- --batch-size 8 \
- --lr 8e-05 \
- --fail-on-bad-query-rows \
- --epoch-train-eval-max-rows 80 \
- --epoch-val-eval-max-rows 32 \
- --graph-memory-cache-size 64 \
- --lazy-prefetch-workers 6 \
- --lazy-prefetch-window 24 \
- --batch-prepare-workers 16 \
- --batch-prepare-lookahead-batches 64 \
- --graph-prefetch-lookahead-batches 3 \
- --reserve-cpu-cores 2 \
- --torch-cpu-threads 4 \
- --torch-interop-threads 4 \
- --log-every-steps 20 \
- --checkpoint-every-steps 50 \
- --keep-step-checkpoints 2 \
- --train-sampling-mode source_aware_balanced \
- --loss-group-balancing-mode supervision_bucket \
- --loss-source-alpha 0.4 \
- --loss-blend-uniform-ratio 0.25 \
- --loss-weight-power 0.5 \
- --loss-time-boost 1.45 \
- --loss-multi-evidence-boost 2.0 \
- --loss-temporal-positive-boost 1.2 \
- --loss-min-example-weight 0.65 \
- --loss-max-example-weight 1.85 \
- --l2sp-loss-weight 0.02 \
- --event-selection-positive-coverage-count 3 \
- --path-selection-positive-coverage-count 3 \
- --multi-positive-coverage-fraction 0.7 \
- --multi-positive-recall-coverage-count 6 \
- --multi-positive-event-coverage-count 5 \
- --multi-positive-path-coverage-count 3 \
- --multi-positive-final-event-set-coverage-count 5 \
- > "$OUT/train.log" 2>&1
diff --git a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer.pt b/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer.pt
deleted file mode 100644
index 6a8d7a5ef5f67dc21654234957294650976685f1..0000000000000000000000000000000000000000
--- a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer.pt
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:fb4e0d3f51143ec2822315721f76d2ed636218ae3c6edd488d7fa53774193739
-size 238028279
diff --git a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer_best.pt b/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer_best.pt
deleted file mode 100644
index 6a8d7a5ef5f67dc21654234957294650976685f1..0000000000000000000000000000000000000000
--- a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer_best.pt
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:fb4e0d3f51143ec2822315721f76d2ed636218ae3c6edd488d7fa53774193739
-size 238028279
diff --git a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer_last.pt b/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer_last.pt
deleted file mode 100644
index 797bebe18bdb65ff1234b3f625e9b08084b1f8f2..0000000000000000000000000000000000000000
--- a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer_last.pt
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:d7aeba343e330c918956dd53cca9104c580c8d45051ae108608fff0404af5b7c
-size 238028279
diff --git a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/nohup.log b/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/nohup.log
deleted file mode 100644
index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..0000000000000000000000000000000000000000
diff --git a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer.pt b/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer.pt
deleted file mode 100644
index fc4cc5b2277768b75772ca23d4b8ad9ffdaa5c52..0000000000000000000000000000000000000000
--- a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer.pt
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:b6cc3e720625ccbfb9ad7f322cd5e23c4b7b87d31cbb780c30d94ca7d2573c80
-size 238028279
diff --git a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer_best.pt b/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer_best.pt
deleted file mode 100644
index fc4cc5b2277768b75772ca23d4b8ad9ffdaa5c52..0000000000000000000000000000000000000000
--- a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer_best.pt
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:b6cc3e720625ccbfb9ad7f322cd5e23c4b7b87d31cbb780c30d94ca7d2573c80
-size 238028279
diff --git a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer_last.pt b/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer_last.pt
deleted file mode 100644
index 5aadeb98d8d3cb2c75cdd72afd3b2df0521333a3..0000000000000000000000000000000000000000
--- a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer_last.pt
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:4ec5a311d5a1fcf78436310c846f03bc4b70491a7a5b68a065cd485298fbc63e
-size 238028279
diff --git a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/train.log b/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/train.log
deleted file mode 100644
index b84a87a1b8c609e0ae762ad19fa4fa5a793887ae..0000000000000000000000000000000000000000
--- a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/train.log
+++ /dev/null
@@ -1,84 +0,0 @@
-[train_locomo_node_memory] 2026-05-24T04:25:59 gc_process_disabled reason='python_3_12_runtime_stability' scope='cli_process'
-[train_locomo_node_memory] 2026-05-24T04:25:59 query_split_loaded elapsed_seconds=0.022 rows=439 source_dir='outputs/action_frame_tunnel_graph_dataset_548_event_units_20260524/dataset' split='train'
-[train_locomo_node_memory] 2026-05-24T04:25:59 query_split_loaded elapsed_seconds=0.003 rows=55 source_dir='outputs/action_frame_tunnel_graph_dataset_548_event_units_20260524/dataset' split='val'
-[train_locomo_node_memory] 2026-05-24T04:25:59 query_split_loaded elapsed_seconds=0.003 rows=54 source_dir='outputs/action_frame_tunnel_graph_dataset_548_event_units_20260524/dataset' split='test'
-[train_locomo_node_memory] 2026-05-24T04:25:59 graph_source_index_started cache_dir='outputs/action_frame_tunnel_graph_dataset_548_event_units_20260524/dataset/graph_tensor_cache_v2' cache_write_enabled=True prebuild_skipped_for_warm_cache=False requested_conversation_ids=333 require_cache_hit=False source_dir='outputs/action_frame_tunnel_graph_dataset_548_event_units_20260524/dataset'
-[train_locomo_node_memory] 2026-05-24T04:25:59 graph_source_index_ready graph_count=333 source_dir='outputs/action_frame_tunnel_graph_dataset_548_event_units_20260524/dataset' test_rows=54 train_rows=439 val_rows=55
-[train_locomo_node_memory] 2026-05-24T04:26:00 resume_training_state_skipped current_batch_size=8 reason='resume_weights_only' resume_checkpoint='/models/tmcra_node_memory_v11_priorfusion_zeroload_20260517/node_scorer.pt' saved_batch_size=8
-[train_locomo_node_memory] 2026-05-24T04:26:00 resume_checkpoint_loaded resume_checkpoint='/models/tmcra_node_memory_v11_priorfusion_zeroload_20260517/node_scorer.pt' resume_mode='weights_only_forced' saved_head='event' state_key_count=448
-[train_locomo_node_memory] 2026-05-24T04:26:00 training_started batch_prepare_lookahead_batches=64 batch_prepare_workers=16 batch_size=8 checkpoint_every_steps=50 cpu_count=36 data_dir='outputs/action_frame_tunnel_graph_dataset_548_event_units_20260524/dataset' device='cuda' epoch_train_eval=True epoch_train_eval_max_rows=80 epoch_val_eval=True epoch_val_eval_max_rows=32 epochs=8 gc_mode='disabled_during_training' graph_cache_device='cpu' graph_cache_dir='outputs/action_frame_tunnel_graph_dataset_548_event_units_20260524/dataset/graph_tensor_cache_v2' graph_count=333 graph_memory_cache_size=64 graph_prefetch_lookahead_batches=3 issue_log='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/training_issues.jsonl' lazy_prefetch_window=24 lazy_prefetch_workers=6 log_every_steps=20 max_train_steps=0 output_dir='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557' prebuild_issue_log='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/graph_cache_prebuild_errors.jsonl' reserve_cpu_cores=2 resume_checkpoint='/models/tmcra_node_memory_v11_priorfusion_zeroload_20260517/node_scorer.pt' resume_mode='weights_only_forced' resume_weights_only=True skip_bad_query_rows=False skip_final_eval=False test_row_limit=0 test_rows=54 torch_cpu_threads=4 torch_interop_threads=4 train_row_limit=0 train_rows=697 train_rows_before_limit=439 train_rows_effective=697 train_sampling_mode='source_aware_balanced' trainable_stage='tunnel_fusion_only' usable_cpu_cores=34 val_row_limit=0 val_rows=55 val_rows_before_limit=55
-[train_locomo_node_memory] 2026-05-24T04:26:00 training_sampling_ready base_row_count=439 effective_row_count=697 loss_weight_max=1.549193 loss_weight_min=1.319091 mode='source_aware_balanced' source_count=1
-[train_locomo_node_memory] 2026-05-24T04:26:31 batch_prepare_pool_event backend='process' event_type='batch_prepare_process_pool_ready' graph_error_stage='train_batch_prepare' worker_count=16
-[train_locomo_node_memory] 2026-05-24T04:26:33 step_completed batch_event_recall_at_5=1.0 batch_final_event_set_loss=0.061843 batch_graph_error_count=0 batch_l2sp_loss=0.0 batch_loss=7.617344379425049 batch_loss_group_count=1 batch_path_recall_at_3=1.0 epoch=1 epoch_step=1 global_step=1 total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:26:49 step_completed batch_event_recall_at_5=1.0 batch_final_event_set_loss=0.064798 batch_graph_error_count=0 batch_l2sp_loss=7e-08 batch_loss=6.46565055847168 batch_loss_group_count=1 batch_path_recall_at_3=0.833333 epoch=1 epoch_step=20 global_step=20 total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:27:05 step_completed batch_event_recall_at_5=1.0 batch_final_event_set_loss=0.086709 batch_graph_error_count=0 batch_l2sp_loss=1.03e-06 batch_loss=7.7209601402282715 batch_loss_group_count=1 batch_path_recall_at_3=0.875 epoch=1 epoch_step=40 global_step=40 total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:27:15 step_checkpoint_saved epoch=1 epoch_step=50 global_step=50 node_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer_last.pt' path_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer_last.pt' total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:27:22 step_completed batch_event_recall_at_5=1.0 batch_final_event_set_loss=0.126139 batch_graph_error_count=0 batch_l2sp_loss=3.54e-06 batch_loss=6.973794937133789 batch_loss_group_count=1 batch_path_recall_at_3=0.875 epoch=1 epoch_step=60 global_step=60 total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:27:39 step_completed batch_event_recall_at_5=1.0 batch_final_event_set_loss=0.029379 batch_graph_error_count=0 batch_l2sp_loss=5.95e-06 batch_loss=6.7596940994262695 batch_loss_group_count=1 batch_path_recall_at_3=1.0 epoch=1 epoch_step=80 global_step=80 total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:28:03 epoch_completed best_val_loss=4.024884 best_val_selection_score=116.975116 epoch=1 global_step=97 is_best=True train_event_recall_at_5=1.0 train_graph_error_count=0 train_loss=4.217114 train_path_recall_at_3=0.95 val_event_recall_at_5=1.0 val_graph_error_count=0 val_loss=4.024884 val_path_recall_at_3=0.96875 val_selection_score=116.975116
-[train_locomo_node_memory] 2026-05-24T04:28:06 checkpoint_saved epoch=1 global_step=97 is_best=True node_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer_last.pt' path_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer_last.pt'
-[train_locomo_node_memory] 2026-05-24T04:28:09 step_completed batch_event_recall_at_5=1.0 batch_final_event_set_loss=0.131766 batch_graph_error_count=0 batch_l2sp_loss=8.54e-06 batch_loss=6.195566654205322 batch_loss_group_count=1 batch_path_recall_at_3=1.0 epoch=2 epoch_step=3 global_step=100 total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:28:11 step_checkpoint_saved epoch=2 epoch_step=3 global_step=100 node_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer_last.pt' path_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer_last.pt' total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:28:30 step_completed batch_event_recall_at_5=1.0 batch_final_event_set_loss=0.003715 batch_graph_error_count=0 batch_l2sp_loss=1.191e-05 batch_loss=6.659519195556641 batch_loss_group_count=1 batch_path_recall_at_3=1.0 epoch=2 epoch_step=23 global_step=120 total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:28:46 step_completed batch_event_recall_at_5=1.0 batch_final_event_set_loss=0.061323 batch_graph_error_count=0 batch_l2sp_loss=1.57e-05 batch_loss=5.569675445556641 batch_loss_group_count=1 batch_path_recall_at_3=0.833333 epoch=2 epoch_step=43 global_step=140 total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:28:57 step_checkpoint_saved epoch=2 epoch_step=53 global_step=150 node_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer_last.pt' path_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer_last.pt' total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:29:05 step_completed batch_event_recall_at_5=1.0 batch_final_event_set_loss=0.038297 batch_graph_error_count=0 batch_l2sp_loss=1.985e-05 batch_loss=7.193849563598633 batch_loss_group_count=1 batch_path_recall_at_3=1.0 epoch=2 epoch_step=63 global_step=160 total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:29:21 step_completed batch_event_recall_at_5=1.0 batch_final_event_set_loss=0.046331 batch_graph_error_count=0 batch_l2sp_loss=2.431e-05 batch_loss=5.3383002281188965 batch_loss_group_count=1 batch_path_recall_at_3=0.875 epoch=2 epoch_step=83 global_step=180 total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:29:40 epoch_completed best_val_loss=3.758544 best_val_selection_score=117.128901 epoch=2 global_step=191 is_best=True train_event_recall_at_5=1.0 train_graph_error_count=0 train_loss=3.925607 train_path_recall_at_3=0.9625 val_event_recall_at_5=1.0 val_graph_error_count=0 val_loss=3.758544 val_path_recall_at_3=0.96875 val_selection_score=117.128901
-[train_locomo_node_memory] 2026-05-24T04:29:44 checkpoint_saved epoch=2 global_step=191 is_best=True node_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer_last.pt' path_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer_last.pt'
-[train_locomo_node_memory] 2026-05-24T04:29:53 step_completed batch_event_recall_at_5=1.0 batch_final_event_set_loss=0.13841 batch_graph_error_count=0 batch_l2sp_loss=2.782e-05 batch_loss=6.168829917907715 batch_loss_group_count=1 batch_path_recall_at_3=1.0 epoch=3 epoch_step=9 global_step=200 total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:29:56 step_checkpoint_saved epoch=3 epoch_step=9 global_step=200 node_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer_last.pt' path_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer_last.pt' total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:30:12 step_completed batch_event_recall_at_5=1.0 batch_final_event_set_loss=0.050877 batch_graph_error_count=0 batch_l2sp_loss=3.105e-05 batch_loss=5.412087917327881 batch_loss_group_count=1 batch_path_recall_at_3=0.75 epoch=3 epoch_step=29 global_step=220 total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:30:27 step_completed batch_event_recall_at_5=1.0 batch_final_event_set_loss=0.133808 batch_graph_error_count=0 batch_l2sp_loss=3.419e-05 batch_loss=7.1404523849487305 batch_loss_group_count=1 batch_path_recall_at_3=1.0 epoch=3 epoch_step=49 global_step=240 total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:30:38 step_checkpoint_saved epoch=3 epoch_step=59 global_step=250 node_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer_last.pt' path_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer_last.pt' total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:30:46 step_completed batch_event_recall_at_5=1.0 batch_final_event_set_loss=0.0 batch_graph_error_count=0 batch_l2sp_loss=3.686e-05 batch_loss=4.634302616119385 batch_loss_group_count=1 batch_path_recall_at_3=1.0 epoch=3 epoch_step=69 global_step=260 total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:31:02 step_completed batch_event_recall_at_5=1.0 batch_final_event_set_loss=0.08086 batch_graph_error_count=0 batch_l2sp_loss=3.993e-05 batch_loss=6.546285629272461 batch_loss_group_count=1 batch_path_recall_at_3=0.875 epoch=3 epoch_step=89 global_step=280 total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:31:17 epoch_completed best_val_loss=3.710867 best_val_selection_score=118.164133 epoch=3 global_step=286 is_best=True train_event_recall_at_5=1.0 train_graph_error_count=0 train_loss=3.793919 train_path_recall_at_3=0.9875 val_event_recall_at_5=1.0 val_graph_error_count=0 val_loss=3.710867 val_path_recall_at_3=1.0 val_selection_score=118.164133
-[train_locomo_node_memory] 2026-05-24T04:31:21 checkpoint_saved epoch=3 global_step=286 is_best=True node_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer_last.pt' path_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer_last.pt'
-[train_locomo_node_memory] 2026-05-24T04:31:34 step_completed batch_event_recall_at_5=1.0 batch_final_event_set_loss=0.0 batch_graph_error_count=0 batch_l2sp_loss=4.268e-05 batch_loss=6.9818620681762695 batch_loss_group_count=1 batch_path_recall_at_3=1.0 epoch=4 epoch_step=14 global_step=300 total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:31:36 step_checkpoint_saved epoch=4 epoch_step=14 global_step=300 node_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer_last.pt' path_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer_last.pt' total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:31:52 step_completed batch_event_recall_at_5=1.0 batch_final_event_set_loss=0.02366 batch_graph_error_count=0 batch_l2sp_loss=4.505e-05 batch_loss=6.924047470092773 batch_loss_group_count=1 batch_path_recall_at_3=1.0 epoch=4 epoch_step=34 global_step=320 total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:32:09 step_completed batch_event_recall_at_5=1.0 batch_final_event_set_loss=0.065861 batch_graph_error_count=0 batch_l2sp_loss=4.73e-05 batch_loss=5.456366062164307 batch_loss_group_count=1 batch_path_recall_at_3=1.0 epoch=4 epoch_step=54 global_step=340 total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:32:20 step_checkpoint_saved epoch=4 epoch_step=64 global_step=350 node_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer_last.pt' path_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer_last.pt' total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:32:28 step_completed batch_event_recall_at_5=1.0 batch_final_event_set_loss=0.033157 batch_graph_error_count=0 batch_l2sp_loss=4.912e-05 batch_loss=7.173478603363037 batch_loss_group_count=1 batch_path_recall_at_3=1.0 epoch=4 epoch_step=74 global_step=360 total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:32:44 step_completed batch_event_recall_at_5=1.0 batch_final_event_set_loss=0.0 batch_graph_error_count=0 batch_l2sp_loss=5.091e-05 batch_loss=4.523404121398926 batch_loss_group_count=1 batch_path_recall_at_3=1.0 epoch=4 epoch_step=94 global_step=380 total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:32:54 epoch_completed best_val_loss=3.718748 best_val_selection_score=118.424107 epoch=4 global_step=380 is_best=True train_event_recall_at_5=1.0 train_graph_error_count=0 train_loss=3.824029 train_path_recall_at_3=0.9875 val_event_recall_at_5=1.0 val_graph_error_count=0 val_loss=3.718748 val_path_recall_at_3=1.0 val_selection_score=118.424107
-[train_locomo_node_memory] 2026-05-24T04:32:58 checkpoint_saved epoch=4 global_step=380 is_best=True node_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer_last.pt' path_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer_last.pt'
-[train_locomo_node_memory] 2026-05-24T04:33:16 step_completed batch_event_recall_at_5=1.0 batch_final_event_set_loss=0.0 batch_graph_error_count=0 batch_l2sp_loss=5.232e-05 batch_loss=6.215372085571289 batch_loss_group_count=1 batch_path_recall_at_3=1.0 epoch=5 epoch_step=20 global_step=400 total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:33:18 step_checkpoint_saved epoch=5 epoch_step=20 global_step=400 node_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer_last.pt' path_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer_last.pt' total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:33:35 step_completed batch_event_recall_at_5=1.0 batch_final_event_set_loss=0.02998 batch_graph_error_count=0 batch_l2sp_loss=5.362e-05 batch_loss=6.076883316040039 batch_loss_group_count=1 batch_path_recall_at_3=1.0 epoch=5 epoch_step=40 global_step=420 total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:33:50 step_completed batch_event_recall_at_5=1.0 batch_final_event_set_loss=0.026733 batch_graph_error_count=0 batch_l2sp_loss=5.463e-05 batch_loss=4.687601566314697 batch_loss_group_count=1 batch_path_recall_at_3=1.0 epoch=5 epoch_step=60 global_step=440 total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:34:00 step_checkpoint_saved epoch=5 epoch_step=70 global_step=450 node_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer_last.pt' path_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer_last.pt' total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:34:08 step_completed batch_event_recall_at_5=1.0 batch_final_event_set_loss=0.017054 batch_graph_error_count=0 batch_l2sp_loss=5.565e-05 batch_loss=6.018680095672607 batch_loss_group_count=1 batch_path_recall_at_3=1.0 epoch=5 epoch_step=80 global_step=460 total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:34:29 epoch_completed best_val_loss=3.718748 best_val_selection_score=118.424107 epoch=5 global_step=474 is_best=False train_event_recall_at_5=1.0 train_graph_error_count=0 train_loss=3.451428 train_path_recall_at_3=0.975 val_event_recall_at_5=1.0 val_graph_error_count=0 val_loss=3.512776 val_path_recall_at_3=1.0 val_selection_score=118.487224
-[train_locomo_node_memory] 2026-05-24T04:34:32 checkpoint_saved epoch=5 global_step=474 is_best=False node_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer_last.pt' path_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer_last.pt'
-[train_locomo_node_memory] 2026-05-24T04:34:38 step_completed batch_event_recall_at_5=1.0 batch_final_event_set_loss=0.011568 batch_graph_error_count=0 batch_l2sp_loss=5.656e-05 batch_loss=4.576869964599609 batch_loss_group_count=1 batch_path_recall_at_3=1.0 epoch=6 epoch_step=6 global_step=480 total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:34:54 step_completed batch_event_recall_at_5=1.0 batch_final_event_set_loss=0.061455 batch_graph_error_count=0 batch_l2sp_loss=5.735e-05 batch_loss=4.150051116943359 batch_loss_group_count=1 batch_path_recall_at_3=1.0 epoch=6 epoch_step=26 global_step=500 total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:34:57 step_checkpoint_saved epoch=6 epoch_step=26 global_step=500 node_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer_last.pt' path_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer_last.pt' total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:35:12 step_completed batch_event_recall_at_5=1.0 batch_final_event_set_loss=0.000704 batch_graph_error_count=0 batch_l2sp_loss=5.803e-05 batch_loss=5.257463455200195 batch_loss_group_count=1 batch_path_recall_at_3=1.0 epoch=6 epoch_step=46 global_step=520 total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:35:28 step_completed batch_event_recall_at_5=1.0 batch_final_event_set_loss=0.0 batch_graph_error_count=0 batch_l2sp_loss=5.871e-05 batch_loss=5.609147071838379 batch_loss_group_count=1 batch_path_recall_at_3=1.0 epoch=6 epoch_step=66 global_step=540 total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:35:39 step_checkpoint_saved epoch=6 epoch_step=76 global_step=550 node_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer_last.pt' path_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer_last.pt' total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:35:46 step_completed batch_event_recall_at_5=1.0 batch_final_event_set_loss=0.284378 batch_graph_error_count=0 batch_l2sp_loss=5.923e-05 batch_loss=6.41322135925293 batch_loss_group_count=1 batch_path_recall_at_3=0.857143 epoch=6 epoch_step=86 global_step=560 total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:36:03 epoch_completed best_val_loss=3.718748 best_val_selection_score=118.424107 epoch=6 global_step=569 is_best=False train_event_recall_at_5=1.0 train_graph_error_count=0 train_loss=3.54849 train_path_recall_at_3=0.95 val_event_recall_at_5=1.0 val_graph_error_count=0 val_loss=3.330123 val_path_recall_at_3=1.0 val_selection_score=117.669877
-[train_locomo_node_memory] 2026-05-24T04:36:05 checkpoint_saved epoch=6 global_step=569 is_best=False node_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer_last.pt' path_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer_last.pt'
-[train_locomo_node_memory] 2026-05-24T04:36:16 step_completed batch_event_recall_at_5=1.0 batch_final_event_set_loss=0.051387 batch_graph_error_count=0 batch_l2sp_loss=5.974e-05 batch_loss=5.32041072845459 batch_loss_group_count=1 batch_path_recall_at_3=1.0 epoch=7 epoch_step=11 global_step=580 total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:36:32 step_completed batch_event_recall_at_5=1.0 batch_final_event_set_loss=0.042565 batch_graph_error_count=0 batch_l2sp_loss=6.013e-05 batch_loss=5.378664970397949 batch_loss_group_count=1 batch_path_recall_at_3=1.0 epoch=7 epoch_step=31 global_step=600 total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:36:35 step_checkpoint_saved epoch=7 epoch_step=31 global_step=600 node_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer_last.pt' path_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer_last.pt' total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:36:50 step_completed batch_event_recall_at_5=1.0 batch_final_event_set_loss=0.023837 batch_graph_error_count=0 batch_l2sp_loss=6.043e-05 batch_loss=4.947947978973389 batch_loss_group_count=1 batch_path_recall_at_3=1.0 epoch=7 epoch_step=51 global_step=620 total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:37:05 step_completed batch_event_recall_at_5=1.0 batch_final_event_set_loss=0.0 batch_graph_error_count=0 batch_l2sp_loss=6.069e-05 batch_loss=5.096177101135254 batch_loss_group_count=1 batch_path_recall_at_3=1.0 epoch=7 epoch_step=71 global_step=640 total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:37:16 step_checkpoint_saved epoch=7 epoch_step=81 global_step=650 node_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer_last.pt' path_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer_last.pt' total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:37:24 step_completed batch_event_recall_at_5=1.0 batch_final_event_set_loss=0.008784 batch_graph_error_count=0 batch_l2sp_loss=6.089e-05 batch_loss=6.337571144104004 batch_loss_group_count=1 batch_path_recall_at_3=1.0 epoch=7 epoch_step=91 global_step=660 total_steps=704
-[train_locomo_node_memory] 2026-05-24T04:37:37 epoch_completed best_val_loss=3.718748 best_val_selection_score=118.424107 epoch=7 global_step=665 is_best=False train_event_recall_at_5=1.0 train_graph_error_count=0 train_loss=3.466264 train_path_recall_at_3=1.0 val_event_recall_at_5=1.0 val_graph_error_count=0 val_loss=3.604544 val_path_recall_at_3=1.0 val_selection_score=117.895456
-[train_locomo_node_memory] 2026-05-24T04:37:40 checkpoint_saved epoch=7 global_step=665 is_best=False node_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer_last.pt' path_model_last='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer_last.pt'
-[train_locomo_node_memory] 2026-05-24T04:37:41 final_eval_started batch_size=8 rows=439 split='train'
-[train_locomo_node_memory] 2026-05-24T04:37:41 lazy_graph_cache_progress disk_cache_hits=1 graph_count=333 known_graph_count=333 last_conversation_id='actionframe548_typed_tunnel_v1_50' loaded_from='disk_cache' loads=1 memory_cache_hits=0 memory_cache_size=1 prefetch_completed=1 prefetch_pending=11 prefetch_submitted=12 source_dir='outputs/action_frame_tunnel_graph_dataset_548_event_units_20260524/dataset' tensorized=0
-[train_locomo_node_memory] 2026-05-24T04:38:34 final_eval_completed event_recall_at_5=0.997722 path_recall_at_3=0.98861 rows=439 split='train' temporal_accuracy=0.235294
-[train_locomo_node_memory] 2026-05-24T04:38:34 final_eval_started batch_size=8 rows=55 split='val'
-[train_locomo_node_memory] 2026-05-24T04:38:42 final_eval_completed event_recall_at_5=1.0 path_recall_at_3=1.0 rows=55 split='val' temporal_accuracy=0.388889
-[train_locomo_node_memory] 2026-05-24T04:38:42 query_split_load_deferred_started source_count=1 split='test'
-[train_locomo_node_memory] 2026-05-24T04:38:42 query_split_loaded elapsed_seconds=0.016 rows=54 source_dir='outputs/action_frame_tunnel_graph_dataset_548_event_units_20260524/dataset' split='test'
-[train_locomo_node_memory] 2026-05-24T04:38:42 query_split_load_deferred_completed rows=54 split='test'
-[train_locomo_node_memory] 2026-05-24T04:38:42 final_eval_started batch_size=8 rows=54 split='test'
-[train_locomo_node_memory] 2026-05-24T04:38:47 final_eval_completed event_recall_at_5=1.0 path_recall_at_3=1.0 rows=54 split='test' temporal_accuracy=0.434783
-[train_locomo_node_memory] 2026-05-24T04:38:50 unified_checkpoint_audit_passed node_missing_prefixes='' path_missing_prefixes='' required_prefix_count=9
-[train_locomo_node_memory] 2026-05-24T04:38:50 training_completed best_val_loss=3.718748 elapsed_seconds=770.323 issue_count=1 node_model_path='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer.pt' path_model_path='outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer.pt' promoted_checkpoint_source='best' test_event_recall_at_5=1.0 test_path_recall_at_3=1.0 val_event_recall_at_5=1.0 val_path_recall_at_3=1.0
diff --git a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/train.pid b/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/train.pid
deleted file mode 100644
index e9e0896f870177ca44887f3b01a2b8517c55535b..0000000000000000000000000000000000000000
--- a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/train.pid
+++ /dev/null
@@ -1 +0,0 @@
-826216
diff --git a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/train_summary.json b/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/train_summary.json
deleted file mode 100644
index 5f024536cc8425d07e0cdea9ac16092980e0e132..0000000000000000000000000000000000000000
--- a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/train_summary.json
+++ /dev/null
@@ -1 +0,0 @@
-{"answer_calibration_variant": "competition_alignment_v3_trimaze", "answer_plan_variant": "candidate_event_answer_plan_v1", "artifacts": {"node_model_best_path": "outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer_best.pt", "node_model_last_path": "outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer_last.pt", "node_model_path": "outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer.pt", "path_model_best_path": "outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer_best.pt", "path_model_last_path": "outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer_last.pt", "path_model_path": "outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer.pt", "promoted_checkpoint_source": "best", "promoted_node_model_source_path": "outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer_best.pt", "promoted_path_model_source_path": "outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer_best.pt"}, "best_val_loss": 3.718748, "best_val_selection_score": 118.424107, "checkpoint_audit": {"node": {"missing_prefixes": [], "ok": true, "path": "outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer.pt", "prefix_counts": {"event_head": 10, "event_tunnel_head": 4, "final_event_fusion_head": 4, "final_path_fusion_head": 4, "memory_router_head": 6, "path_head": 11, "path_tunnel_head": 4, "question_intent_head": 12, "temporal_head": 4}, "required_prefixes": ["memory_router_head", "temporal_head", "question_intent_head", "event_head", "path_head", "event_tunnel_head", "path_tunnel_head", "final_event_fusion_head", "final_path_fusion_head"], "saved_head": "event", "state_key_count": 448}, "path": {"missing_prefixes": [], "ok": true, "path": "outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer.pt", "prefix_counts": {"event_head": 10, "event_tunnel_head": 4, "final_event_fusion_head": 4, "final_path_fusion_head": 4, "memory_router_head": 6, "path_head": 11, "path_tunnel_head": 4, "question_intent_head": 12, "temporal_head": 4}, "required_prefixes": ["memory_router_head", "temporal_head", "question_intent_head", "event_head", "path_head", "event_tunnel_head", "path_tunnel_head", "final_event_fusion_head", "final_path_fusion_head"], "saved_head": "path", "state_key_count": 448}}, "completed_at": "2026-05-24T04:38:50", "config": {"amp": true, "answer_calibration_variant": "competition_alignment_v3_trimaze", "answer_plan_current_old_margin": 0.2, "answer_plan_current_old_margin_loss_weight": 0.0, "answer_plan_loss_weight": 0.18, "answer_plan_selected_negative_margin": 0.3, "answer_plan_selected_negative_margin_loss_weight": 0.0, "answer_plan_selection_margin": 0.12, "answer_plan_selection_top_k": 5, "answer_plan_variant": "candidate_event_answer_plan_v1", "answer_refusal_loss_weight": 0.08, "answer_refusal_margin": 0.15, "answer_type_loss_weight": 0.2, "batch_prepare_lookahead_batches": 64, "batch_prepare_workers": 16, "batch_size": 8, "checkpoint_every_steps": 50, "early_stopping_patience": 3, "encoder_attention_heads": 8, "encoder_attention_layers": 6, "encoder_structural_bias_variant": "tmcra_token_role_relbias_v1", "epoch_train_eval_enabled": true, "epoch_train_eval_max_rows": 80, "epoch_val_eval_enabled": true, "epoch_val_eval_max_rows": 32, "epochs": 8, "event_distractor_loss_weight": 0.15, "event_distractor_variant": "candidate_distractor_delta_bce_v1", "event_hard_negative_loss_weight": 0.35, "event_hard_negative_margin": 0.35, "event_loss_weight": 1.1, "event_matrix_delta_loss_weight": 0.08, "event_matrix_delta_margin": 0.15, "event_selection_loss_weight": 0.35, "event_selection_margin": 0.2, "event_selection_positive_coverage_count": 3, "event_selection_top_k": 5, "event_subgraph_attention_heads": 8, "event_subgraph_attention_layers": 6, "event_subgraph_refiner_variant": "event_centered_subgraph_attn_6l_h448", "event_tunnel_loss_weight": 0.12, "event_tunnel_margin": 0.16, "event_tunnel_selection_loss_weight": 0.08, "final_event_set_loss_weight": 0.35, "final_event_set_margin": 0.08, "final_event_set_positive_coverage_count": 3, "final_event_set_support_path_k": 3, "final_event_set_top_k": 10, "grad_clip": 1.0, "graph_prefetch_lookahead_batches": 3, "l2sp_loss_weight": 0.02, "log_every_steps": 20, "loss_blend_uniform_ratio": 0.25, "loss_group_balancing_mode": "supervision_bucket", "loss_max_example_weight": 1.85, "loss_min_example_weight": 0.65, "loss_multi_evidence_boost": 2.0, "loss_source_alpha": 0.4, "loss_temporal_positive_boost": 1.2, "loss_time_boost": 1.45, "loss_weight_power": 0.5, "lr": 8e-05, "max_train_steps": 0, "memory_router_loss_weight": 0.18, "memory_router_variant": "question_pooled_multilabel_v1", "memory_tunnel_variant": "chain_depth_tunnel_support_delta_v1", "message_passing_variant": "support_to_event_bidirectional_shared_typed_v1", "model_arch_version": "node_memory_v11_tmcra_scale49m_trimaze", "multi_positive_coverage_fraction": 0.7, "multi_positive_event_coverage_count": 5, "multi_positive_final_event_set_coverage_count": 5, "multi_positive_path_coverage_count": 3, "multi_positive_recall_coverage_count": 6, "node_encoder_variant": "hashed_tmcra_token_attn_pool_6l_w256", "node_max_tokens": 32, "online_event_hard_negative_limit": 6, "online_path_hard_negative_limit": 8, "pair_feature_adapter_variant": "residual_pair_feature_adapter_v1", "path_hard_negative_loss_weight": 0.25, "path_hard_negative_margin": 0.28, "path_loss_weight": 0.55, "path_matrix_delta_loss_weight": 0.08, "path_matrix_delta_margin": 0.12, "path_selection_loss_weight": 0.3, "path_selection_margin": 0.15, "path_selection_positive_coverage_count": 3, "path_selection_top_k": 3, "path_tunnel_delta_loss_weight": 0.12, "path_tunnel_loss_weight": 0.06, "path_tunnel_margin": 0.16, "path_tunnel_selection_loss_weight": 0.04, "question_encoder_variant": "hashed_tmcra_token_attn_pool_6l_w256", "question_intent_variant": "text_pooled_multihead_v1", "question_max_tokens": 48, "question_understanding_loss_weight": 0.12, "recall_loss_weight": 0.6, "recall_selection_loss_weight": 0.3, "recall_selection_margin": 0.12, "recall_selection_positive_coverage_count": 3, "recall_selection_top_k": 24, "sampling_blend_uniform_ratio": 0.35, "sampling_max_conversation_multiplier": 2.0, "sampling_max_group_repeat": 2, "sampling_multi_evidence_boost": 1.45, "sampling_source_alpha": 0.35, "sampling_temporal_positive_boost": 1.15, "sampling_time_boost": 1.2, "temporal_loss_weight": 0.3, "token_role_loss_weight": 0.08, "train_sampling_mode": "source_aware_balanced", "trainable_stage": "tunnel_fusion_only", "warmup_ratio": 0.06, "weight_decay": 0.01}, "data_issues": {"issue_counts": {"batch_prepare_pool_event": 1}, "issue_log_path": "outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/training_issues.jsonl", "prebuild_issue_log_path": "outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/graph_cache_prebuild_errors.jsonl"}, "device": "cuda", "elapsed_seconds": 770.323, "encoder_attention_heads": 8, "encoder_attention_layers": 6, "encoder_structural_bias_variant": "tmcra_token_role_relbias_v1", "event_distractor_variant": "candidate_distractor_delta_bce_v1", "event_subgraph_attention_heads": 8, "event_subgraph_attention_layers": 6, "event_subgraph_refiner_variant": "event_centered_subgraph_attn_6l_h448", "history": [{"best_val_loss": 4.024884, "best_val_selection_key": [1.0, 1.0, 0.9, 0.4, -4.024884], "best_val_selection_score": 116.975116, "epoch": 1, "global_step": 97, "is_best": true, "patience": 0, "total_steps": 704, "train": {"answer_plan_current_top1_accuracy": 0.425, "answer_plan_current_total": 80, "answer_plan_loss": 0.546678, "answer_plan_loss_count": 80, "answer_plan_selected_positive_coverage_at_5": 0.787037, "answer_plan_selected_positive_total": 216, "answer_plan_selected_recall_at_5": 0.9625, "answer_plan_selected_total": 80, "answer_refusal_loss": 0.0, "answer_refusal_loss_count": 80, "answer_type_loss": 5.293935, "answer_type_loss_count": 80, "answer_type_metrics": {"multi_evidence": {"answer_plan_current_top1_accuracy": 0.44, "answer_plan_current_total": 75, "answer_plan_selected_positive_coverage_at_5": 0.796116, "answer_plan_selected_positive_total": 206, "answer_plan_selected_recall_at_5": 0.973333, "answer_plan_selected_total": 75, "event5_positive_total": 206, "event_positive_coverage_at_5": 0.966019, "event_recall_at_1": 0.84, "event_recall_at_5": 1.0, "event_recall_total": 75, "path3_positive_total": 225, "path_positive_coverage_at_3": 0.715555, "path_recall_at_3": 0.946667, "path_recall_total": 75, "recall_event24_positive_total": 206, "recall_event_positive_coverage_at_24": 0.990291, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 75, "samples": 75, "temporal_accuracy": 0.0, "temporal_total": 0}, "time": {"answer_plan_current_top1_accuracy": 0.2, "answer_plan_current_total": 5, "answer_plan_selected_positive_coverage_at_5": 0.6, "answer_plan_selected_positive_total": 10, "answer_plan_selected_recall_at_5": 0.8, "answer_plan_selected_total": 5, "event5_positive_total": 10, "event_positive_coverage_at_5": 0.7, "event_recall_at_1": 0.2, "event_recall_at_5": 1.0, "event_recall_total": 5, "path3_positive_total": 15, "path_positive_coverage_at_3": 0.333333, "path_recall_at_3": 1.0, "path_recall_total": 5, "recall_event24_positive_total": 10, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 5, "samples": 5, "temporal_accuracy": 0.0, "temporal_total": 5}}, "evaluation_limited": true, "event5_positive_total": 216, "event_distractor_loss": 0.668917, "event_distractor_loss_count": 80, "event_hard_negative_loss": 0.113749, "event_hard_negative_loss_count": 80, "event_loss": 0.423251, "event_loss_count": 80, "event_matrix_delta_loss": 0.132817, "event_matrix_delta_loss_count": 80, "event_positive_coverage_at_5": 0.953704, "event_recall_at_1": 0.8, "event_recall_at_5": 1.0, "event_recall_total": 80, "event_selection_loss": 0.034901, "event_selection_loss_count": 80, "event_tunnel_loss": 0.283888, "event_tunnel_loss_count": 80, "event_tunnel_selection_loss": 0.0, "event_tunnel_selection_loss_count": 80, "final_event_set_loss": 0.038654, "final_event_set_loss_count": 80, "graph_error_count": 0, "loss": 4.217114, "loss_group_count": 10, "memory_router_exact_match": 0.375, "memory_router_f1": 0.845089, "memory_router_loss": 1.473932, "memory_router_loss_count": 80, "memory_router_total": 80, "path3_positive_total": 240, "path_hard_negative_loss": 0.530825, "path_hard_negative_loss_count": 80, "path_loss": 0.767234, "path_loss_count": 80, "path_matrix_delta_loss": 0.466691, "path_matrix_delta_loss_count": 80, "path_positive_coverage_at_3": 0.691667, "path_recall_at_3": 0.95, "path_recall_total": 80, "path_selection_loss": 0.962866, "path_selection_loss_count": 80, "path_tunnel_delta_loss": 0.364665, "path_tunnel_delta_loss_count": 80, "path_tunnel_delta_positive_coverage_at_3": 0.516667, "path_tunnel_delta_positive_total": 240, "path_tunnel_delta_recall_at_3": 0.875, "path_tunnel_delta_recall_total": 80, "path_tunnel_loss": 0.488218, "path_tunnel_loss_count": 80, "path_tunnel_rescue025_recall_at_3": 0.95, "path_tunnel_rescue025_recall_total": 80, "path_tunnel_rescue050_recall_at_3": 0.95, "path_tunnel_rescue050_recall_total": 80, "path_tunnel_rescue100_recall_at_3": 0.95, "path_tunnel_rescue100_recall_total": 80, "path_tunnel_selection_loss": 0.360459, "path_tunnel_selection_loss_count": 80, "path_tunnel_support_positive_coverage_at_3": 0.7, "path_tunnel_support_positive_total": 240, "path_tunnel_support_recall_at_3": 0.95, "path_tunnel_support_recall_total": 80, "question_understanding_loss": 5.189661, "question_understanding_loss_count": 80, "recall_event24_positive_total": 216, "recall_event_positive_coverage_at_24": 0.990741, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 80, "recall_loss": 0.860721, "recall_loss_count": 80, "recall_selection_loss": 0.0, "recall_selection_loss_count": 80, "rows_evaluated": 80, "rows_total": 439, "samples": 80, "supervision_bucket_metrics": {"typed_tunnel_multi_positive": {"answer_plan_current_top1_accuracy": 0.425, "answer_plan_current_total": 80, "answer_plan_selected_positive_coverage_at_5": 0.787037, "answer_plan_selected_positive_total": 216, "answer_plan_selected_recall_at_5": 0.9625, "answer_plan_selected_total": 80, "event5_positive_total": 216, "event_positive_coverage_at_5": 0.953704, "event_recall_at_1": 0.8, "event_recall_at_5": 1.0, "event_recall_total": 80, "path3_positive_total": 240, "path_positive_coverage_at_3": 0.691667, "path_recall_at_3": 0.95, "path_recall_total": 80, "recall_event24_positive_total": 216, "recall_event_positive_coverage_at_24": 0.990741, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 80, "samples": 80, "temporal_accuracy": 0.0, "temporal_total": 5}}, "temporal_accuracy": 0.0, "temporal_loss": 0.581992, "temporal_loss_count": 5, "temporal_total": 5, "token_role_loss": 0.0, "token_role_loss_count": 10, "training_weight_max": 1.35, "training_weight_mean": 1.35, "training_weight_min": 1.35}, "val": {"answer_plan_current_top1_accuracy": 0.46875, "answer_plan_current_total": 32, "answer_plan_loss": 0.534763, "answer_plan_loss_count": 32, "answer_plan_selected_positive_coverage_at_5": 0.8125, "answer_plan_selected_positive_total": 80, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 32, "answer_refusal_loss": 0.0, "answer_refusal_loss_count": 32, "answer_type_loss": 5.569499, "answer_type_loss_count": 32, "answer_type_metrics": {"multi_evidence": {"answer_plan_current_top1_accuracy": 0.409091, "answer_plan_current_total": 22, "answer_plan_selected_positive_coverage_at_5": 0.783333, "answer_plan_selected_positive_total": 60, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 22, "event5_positive_total": 60, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 0.818182, "event_recall_at_5": 1.0, "event_recall_total": 22, "path3_positive_total": 66, "path_positive_coverage_at_3": 0.80303, "path_recall_at_3": 1.0, "path_recall_total": 22, "recall_event24_positive_total": 60, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 22, "samples": 22, "temporal_accuracy": 0.0, "temporal_total": 0}, "time": {"answer_plan_current_top1_accuracy": 0.6, "answer_plan_current_total": 10, "answer_plan_selected_positive_coverage_at_5": 0.9, "answer_plan_selected_positive_total": 20, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 10, "event5_positive_total": 20, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 0.8, "event_recall_at_5": 1.0, "event_recall_total": 10, "path3_positive_total": 30, "path_positive_coverage_at_3": 0.733333, "path_recall_at_3": 0.9, "path_recall_total": 10, "recall_event24_positive_total": 20, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 10, "samples": 10, "temporal_accuracy": 0.4, "temporal_total": 10}}, "evaluation_limited": true, "event5_positive_total": 80, "event_distractor_loss": 0.646126, "event_distractor_loss_count": 32, "event_hard_negative_loss": 0.060784, "event_hard_negative_loss_count": 32, "event_loss": 0.388608, "event_loss_count": 32, "event_matrix_delta_loss": 0.123856, "event_matrix_delta_loss_count": 32, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 0.8125, "event_recall_at_5": 1.0, "event_recall_total": 32, "event_selection_loss": 0.010889, "event_selection_loss_count": 32, "event_tunnel_loss": 0.255723, "event_tunnel_loss_count": 32, "event_tunnel_selection_loss": 0.0, "event_tunnel_selection_loss_count": 32, "final_event_set_loss": 0.019053, "final_event_set_loss_count": 32, "graph_error_count": 0, "loss": 4.024884, "loss_group_count": 4, "memory_router_exact_match": 0.375, "memory_router_f1": 0.917535, "memory_router_loss": 0.56219, "memory_router_loss_count": 32, "memory_router_total": 32, "path3_positive_total": 96, "path_hard_negative_loss": 0.674052, "path_hard_negative_loss_count": 32, "path_loss": 0.831701, "path_loss_count": 32, "path_matrix_delta_loss": 0.527073, "path_matrix_delta_loss_count": 32, "path_positive_coverage_at_3": 0.78125, "path_recall_at_3": 0.96875, "path_recall_total": 32, "path_selection_loss": 1.007566, "path_selection_loss_count": 32, "path_tunnel_delta_loss": 0.379047, "path_tunnel_delta_loss_count": 32, "path_tunnel_delta_positive_coverage_at_3": 0.375, "path_tunnel_delta_positive_total": 96, "path_tunnel_delta_recall_at_3": 0.75, "path_tunnel_delta_recall_total": 32, "path_tunnel_loss": 0.508271, "path_tunnel_loss_count": 32, "path_tunnel_rescue025_recall_at_3": 1.0, "path_tunnel_rescue025_recall_total": 32, "path_tunnel_rescue050_recall_at_3": 1.0, "path_tunnel_rescue050_recall_total": 32, "path_tunnel_rescue100_recall_at_3": 1.0, "path_tunnel_rescue100_recall_total": 32, "path_tunnel_selection_loss": 0.398805, "path_tunnel_selection_loss_count": 32, "path_tunnel_support_positive_coverage_at_3": 0.760417, "path_tunnel_support_positive_total": 96, "path_tunnel_support_recall_at_3": 1.0, "path_tunnel_support_recall_total": 32, "question_understanding_loss": 3.967331, "question_understanding_loss_count": 32, "recall_event24_positive_total": 80, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 32, "recall_loss": 0.884973, "recall_loss_count": 32, "recall_selection_loss": 0.0, "recall_selection_loss_count": 32, "rows_evaluated": 32, "rows_total": 55, "samples": 32, "supervision_bucket_metrics": {"typed_tunnel_multi_positive": {"answer_plan_current_top1_accuracy": 0.46875, "answer_plan_current_total": 32, "answer_plan_selected_positive_coverage_at_5": 0.8125, "answer_plan_selected_positive_total": 80, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 32, "event5_positive_total": 80, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 0.8125, "event_recall_at_5": 1.0, "event_recall_total": 32, "path3_positive_total": 96, "path_positive_coverage_at_3": 0.78125, "path_recall_at_3": 0.96875, "path_recall_total": 32, "recall_event24_positive_total": 80, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 32, "samples": 32, "temporal_accuracy": 0.4, "temporal_total": 10}}, "temporal_accuracy": 0.4, "temporal_loss": 0.505753, "temporal_loss_count": 10, "temporal_total": 10, "token_role_loss": 0.0, "token_role_loss_count": 4, "training_weight_max": 1.35, "training_weight_mean": 1.35, "training_weight_min": 1.35}, "val_selection_enabled": true, "val_selection_score": 116.975116}, {"best_val_loss": 3.758544, "best_val_selection_key": [1.0, 1.0, 0.952381, 0.272727, -3.758544], "best_val_selection_score": 117.128901, "epoch": 2, "global_step": 191, "is_best": true, "patience": 0, "total_steps": 704, "train": {"answer_plan_current_top1_accuracy": 0.4625, "answer_plan_current_total": 80, "answer_plan_loss": 0.555802, "answer_plan_loss_count": 80, "answer_plan_selected_positive_coverage_at_5": 0.809955, "answer_plan_selected_positive_total": 221, "answer_plan_selected_recall_at_5": 0.95, "answer_plan_selected_total": 80, "answer_refusal_loss": 0.0, "answer_refusal_loss_count": 80, "answer_type_loss": 5.481835, "answer_type_loss_count": 80, "answer_type_metrics": {"multi_evidence": {"answer_plan_current_top1_accuracy": 0.468354, "answer_plan_current_total": 79, "answer_plan_selected_positive_coverage_at_5": 0.817352, "answer_plan_selected_positive_total": 219, "answer_plan_selected_recall_at_5": 0.962025, "answer_plan_selected_total": 79, "event5_positive_total": 219, "event_positive_coverage_at_5": 0.990868, "event_recall_at_1": 0.835443, "event_recall_at_5": 1.0, "event_recall_total": 79, "path3_positive_total": 237, "path_positive_coverage_at_3": 0.805907, "path_recall_at_3": 0.974684, "path_recall_total": 79, "recall_event24_positive_total": 219, "recall_event_positive_coverage_at_24": 0.995434, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 79, "samples": 79, "temporal_accuracy": 0.0, "temporal_total": 0}, "time": {"answer_plan_current_top1_accuracy": 0.0, "answer_plan_current_total": 1, "answer_plan_selected_positive_coverage_at_5": 0.0, "answer_plan_selected_positive_total": 2, "answer_plan_selected_recall_at_5": 0.0, "answer_plan_selected_total": 1, "event5_positive_total": 2, "event_positive_coverage_at_5": 0.5, "event_recall_at_1": 0.0, "event_recall_at_5": 1.0, "event_recall_total": 1, "path3_positive_total": 3, "path_positive_coverage_at_3": 0.0, "path_recall_at_3": 0.0, "path_recall_total": 1, "recall_event24_positive_total": 2, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 1, "samples": 1, "temporal_accuracy": 0.0, "temporal_total": 1}}, "evaluation_limited": true, "event5_positive_total": 221, "event_distractor_loss": 0.675154, "event_distractor_loss_count": 80, "event_hard_negative_loss": 0.09031, "event_hard_negative_loss_count": 80, "event_loss": 0.372825, "event_loss_count": 80, "event_matrix_delta_loss": 0.133374, "event_matrix_delta_loss_count": 80, "event_positive_coverage_at_5": 0.986425, "event_recall_at_1": 0.825, "event_recall_at_5": 1.0, "event_recall_total": 80, "event_selection_loss": 0.013065, "event_selection_loss_count": 80, "event_tunnel_loss": 0.240243, "event_tunnel_loss_count": 80, "event_tunnel_selection_loss": 0.0, "event_tunnel_selection_loss_count": 80, "final_event_set_loss": 0.061613, "final_event_set_loss_count": 80, "graph_error_count": 0, "loss": 3.925607, "loss_group_count": 10, "memory_router_exact_match": 0.3125, "memory_router_f1": 0.835615, "memory_router_loss": 1.565009, "memory_router_loss_count": 80, "memory_router_total": 80, "path3_positive_total": 240, "path_hard_negative_loss": 0.348101, "path_hard_negative_loss_count": 80, "path_loss": 0.544556, "path_loss_count": 80, "path_matrix_delta_loss": 0.507079, "path_matrix_delta_loss_count": 80, "path_positive_coverage_at_3": 0.795833, "path_recall_at_3": 0.9625, "path_recall_total": 80, "path_selection_loss": 0.386352, "path_selection_loss_count": 80, "path_tunnel_delta_loss": 0.593897, "path_tunnel_delta_loss_count": 80, "path_tunnel_delta_positive_coverage_at_3": 0.5, "path_tunnel_delta_positive_total": 240, "path_tunnel_delta_recall_at_3": 0.825, "path_tunnel_delta_recall_total": 80, "path_tunnel_loss": 0.409488, "path_tunnel_loss_count": 80, "path_tunnel_rescue025_recall_at_3": 0.95, "path_tunnel_rescue025_recall_total": 80, "path_tunnel_rescue050_recall_at_3": 0.9625, "path_tunnel_rescue050_recall_total": 80, "path_tunnel_rescue100_recall_at_3": 0.9625, "path_tunnel_rescue100_recall_total": 80, "path_tunnel_selection_loss": 0.301836, "path_tunnel_selection_loss_count": 80, "path_tunnel_support_positive_coverage_at_3": 0.704167, "path_tunnel_support_positive_total": 240, "path_tunnel_support_recall_at_3": 0.95, "path_tunnel_support_recall_total": 80, "question_understanding_loss": 5.580859, "question_understanding_loss_count": 80, "recall_event24_positive_total": 221, "recall_event_positive_coverage_at_24": 0.995475, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 80, "recall_loss": 0.859999, "recall_loss_count": 80, "recall_selection_loss": 0.0, "recall_selection_loss_count": 80, "rows_evaluated": 80, "rows_total": 439, "samples": 80, "supervision_bucket_metrics": {"typed_tunnel_multi_positive": {"answer_plan_current_top1_accuracy": 0.4625, "answer_plan_current_total": 80, "answer_plan_selected_positive_coverage_at_5": 0.809955, "answer_plan_selected_positive_total": 221, "answer_plan_selected_recall_at_5": 0.95, "answer_plan_selected_total": 80, "event5_positive_total": 221, "event_positive_coverage_at_5": 0.986425, "event_recall_at_1": 0.825, "event_recall_at_5": 1.0, "event_recall_total": 80, "path3_positive_total": 240, "path_positive_coverage_at_3": 0.795833, "path_recall_at_3": 0.9625, "path_recall_total": 80, "recall_event24_positive_total": 221, "recall_event_positive_coverage_at_24": 0.995475, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 80, "samples": 80, "temporal_accuracy": 0.0, "temporal_total": 1}}, "temporal_accuracy": 0.0, "temporal_loss": 0.600853, "temporal_loss_count": 1, "temporal_total": 1, "token_role_loss": 0.0, "token_role_loss_count": 10, "training_weight_max": 1.35, "training_weight_mean": 1.35, "training_weight_min": 1.35}, "val": {"answer_plan_current_top1_accuracy": 0.46875, "answer_plan_current_total": 32, "answer_plan_loss": 0.52675, "answer_plan_loss_count": 32, "answer_plan_selected_positive_coverage_at_5": 0.807693, "answer_plan_selected_positive_total": 78, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 32, "answer_refusal_loss": 0.0, "answer_refusal_loss_count": 32, "answer_type_loss": 5.94012, "answer_type_loss_count": 32, "answer_type_metrics": {"multi_evidence": {"answer_plan_current_top1_accuracy": 0.52381, "answer_plan_current_total": 21, "answer_plan_selected_positive_coverage_at_5": 0.785715, "answer_plan_selected_positive_total": 56, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 21, "event5_positive_total": 56, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 0.857143, "event_recall_at_5": 1.0, "event_recall_total": 21, "path3_positive_total": 63, "path_positive_coverage_at_3": 0.84127, "path_recall_at_3": 0.952381, "path_recall_total": 21, "recall_event24_positive_total": 56, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 21, "samples": 21, "temporal_accuracy": 0.0, "temporal_total": 0}, "time": {"answer_plan_current_top1_accuracy": 0.363636, "answer_plan_current_total": 11, "answer_plan_selected_positive_coverage_at_5": 0.863636, "answer_plan_selected_positive_total": 22, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 11, "event5_positive_total": 22, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 0.727273, "event_recall_at_5": 1.0, "event_recall_total": 11, "path3_positive_total": 33, "path_positive_coverage_at_3": 0.69697, "path_recall_at_3": 1.0, "path_recall_total": 11, "recall_event24_positive_total": 22, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 11, "samples": 11, "temporal_accuracy": 0.272727, "temporal_total": 11}}, "evaluation_limited": true, "event5_positive_total": 78, "event_distractor_loss": 0.645134, "event_distractor_loss_count": 32, "event_hard_negative_loss": 0.072217, "event_hard_negative_loss_count": 32, "event_loss": 0.393431, "event_loss_count": 32, "event_matrix_delta_loss": 0.124774, "event_matrix_delta_loss_count": 32, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 0.8125, "event_recall_at_5": 1.0, "event_recall_total": 32, "event_selection_loss": 0.011071, "event_selection_loss_count": 32, "event_tunnel_loss": 0.257903, "event_tunnel_loss_count": 32, "event_tunnel_selection_loss": 0.0, "event_tunnel_selection_loss_count": 32, "final_event_set_loss": 0.018009, "final_event_set_loss_count": 32, "graph_error_count": 0, "loss": 3.758544, "loss_group_count": 4, "memory_router_exact_match": 0.375, "memory_router_f1": 0.921875, "memory_router_loss": 0.470315, "memory_router_loss_count": 32, "memory_router_total": 32, "path3_positive_total": 96, "path_hard_negative_loss": 0.477834, "path_hard_negative_loss_count": 32, "path_loss": 0.613759, "path_loss_count": 32, "path_matrix_delta_loss": 0.535951, "path_matrix_delta_loss_count": 32, "path_positive_coverage_at_3": 0.791666, "path_recall_at_3": 0.96875, "path_recall_total": 32, "path_selection_loss": 0.438738, "path_selection_loss_count": 32, "path_tunnel_delta_loss": 0.59899, "path_tunnel_delta_loss_count": 32, "path_tunnel_delta_positive_coverage_at_3": 0.34375, "path_tunnel_delta_positive_total": 96, "path_tunnel_delta_recall_at_3": 0.59375, "path_tunnel_delta_recall_total": 32, "path_tunnel_loss": 0.427782, "path_tunnel_loss_count": 32, "path_tunnel_rescue025_recall_at_3": 0.96875, "path_tunnel_rescue025_recall_total": 32, "path_tunnel_rescue050_recall_at_3": 0.96875, "path_tunnel_rescue050_recall_total": 32, "path_tunnel_rescue100_recall_at_3": 1.0, "path_tunnel_rescue100_recall_total": 32, "path_tunnel_selection_loss": 0.236935, "path_tunnel_selection_loss_count": 32, "path_tunnel_support_positive_coverage_at_3": 0.78125, "path_tunnel_support_positive_total": 96, "path_tunnel_support_recall_at_3": 1.0, "path_tunnel_support_recall_total": 32, "question_understanding_loss": 3.773471, "question_understanding_loss_count": 32, "recall_event24_positive_total": 78, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 32, "recall_loss": 0.903305, "recall_loss_count": 32, "recall_selection_loss": 0.0, "recall_selection_loss_count": 32, "rows_evaluated": 32, "rows_total": 55, "samples": 32, "supervision_bucket_metrics": {"typed_tunnel_multi_positive": {"answer_plan_current_top1_accuracy": 0.46875, "answer_plan_current_total": 32, "answer_plan_selected_positive_coverage_at_5": 0.807693, "answer_plan_selected_positive_total": 78, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 32, "event5_positive_total": 78, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 0.8125, "event_recall_at_5": 1.0, "event_recall_total": 32, "path3_positive_total": 96, "path_positive_coverage_at_3": 0.791666, "path_recall_at_3": 0.96875, "path_recall_total": 32, "recall_event24_positive_total": 78, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 32, "samples": 32, "temporal_accuracy": 0.272727, "temporal_total": 11}}, "temporal_accuracy": 0.272727, "temporal_loss": 0.502052, "temporal_loss_count": 11, "temporal_total": 11, "token_role_loss": 0.0, "token_role_loss_count": 4, "training_weight_max": 1.35, "training_weight_mean": 1.35, "training_weight_min": 1.35}, "val_selection_enabled": true, "val_selection_score": 117.128901}, {"best_val_loss": 3.710867, "best_val_selection_key": [1.0, 1.0, 1.0, 0.375, -3.710867], "best_val_selection_score": 118.164133, "epoch": 3, "global_step": 286, "is_best": true, "patience": 0, "total_steps": 704, "train": {"answer_plan_current_top1_accuracy": 0.346154, "answer_plan_current_total": 78, "answer_plan_loss": 0.5523, "answer_plan_loss_count": 80, "answer_plan_selected_positive_coverage_at_5": 0.748837, "answer_plan_selected_positive_total": 215, "answer_plan_selected_recall_at_5": 0.95, "answer_plan_selected_total": 80, "answer_refusal_loss": 0.0, "answer_refusal_loss_count": 80, "answer_type_loss": 5.470425, "answer_type_loss_count": 80, "answer_type_metrics": {"multi_evidence": {"answer_plan_current_top1_accuracy": 0.356164, "answer_plan_current_total": 73, "answer_plan_selected_positive_coverage_at_5": 0.745098, "answer_plan_selected_positive_total": 204, "answer_plan_selected_recall_at_5": 0.96, "answer_plan_selected_total": 75, "event5_positive_total": 204, "event_positive_coverage_at_5": 0.990196, "event_recall_at_1": 0.906667, "event_recall_at_5": 1.0, "event_recall_total": 75, "path3_positive_total": 225, "path_positive_coverage_at_3": 0.835556, "path_recall_at_3": 0.986667, "path_recall_total": 75, "recall_event24_positive_total": 204, "recall_event_positive_coverage_at_24": 0.995098, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 75, "samples": 75, "temporal_accuracy": 0.0, "temporal_total": 0}, "time": {"answer_plan_current_top1_accuracy": 0.2, "answer_plan_current_total": 5, "answer_plan_selected_positive_coverage_at_5": 0.818182, "answer_plan_selected_positive_total": 11, "answer_plan_selected_recall_at_5": 0.8, "answer_plan_selected_total": 5, "event5_positive_total": 11, "event_positive_coverage_at_5": 0.909091, "event_recall_at_1": 0.4, "event_recall_at_5": 1.0, "event_recall_total": 5, "path3_positive_total": 15, "path_positive_coverage_at_3": 0.666666, "path_recall_at_3": 1.0, "path_recall_total": 5, "recall_event24_positive_total": 11, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 5, "samples": 5, "temporal_accuracy": 0.2, "temporal_total": 5}}, "evaluation_limited": true, "event5_positive_total": 215, "event_distractor_loss": 0.672825, "event_distractor_loss_count": 80, "event_hard_negative_loss": 0.074378, "event_hard_negative_loss_count": 80, "event_loss": 0.348452, "event_loss_count": 80, "event_matrix_delta_loss": 0.136848, "event_matrix_delta_loss_count": 80, "event_positive_coverage_at_5": 0.986046, "event_recall_at_1": 0.875, "event_recall_at_5": 1.0, "event_recall_total": 80, "event_selection_loss": 0.01167, "event_selection_loss_count": 80, "event_tunnel_loss": 0.241095, "event_tunnel_loss_count": 80, "event_tunnel_selection_loss": 0.0, "event_tunnel_selection_loss_count": 80, "final_event_set_loss": 0.03739, "final_event_set_loss_count": 80, "graph_error_count": 0, "loss": 3.793919, "loss_group_count": 11, "memory_router_exact_match": 0.3125, "memory_router_f1": 0.836012, "memory_router_loss": 1.426151, "memory_router_loss_count": 80, "memory_router_total": 80, "path3_positive_total": 240, "path_hard_negative_loss": 0.240227, "path_hard_negative_loss_count": 80, "path_loss": 0.467457, "path_loss_count": 80, "path_matrix_delta_loss": 0.446912, "path_matrix_delta_loss_count": 80, "path_positive_coverage_at_3": 0.825, "path_recall_at_3": 0.9875, "path_recall_total": 80, "path_selection_loss": 0.268014, "path_selection_loss_count": 80, "path_tunnel_delta_loss": 0.644251, "path_tunnel_delta_loss_count": 80, "path_tunnel_delta_positive_coverage_at_3": 0.570833, "path_tunnel_delta_positive_total": 240, "path_tunnel_delta_recall_at_3": 0.8625, "path_tunnel_delta_recall_total": 80, "path_tunnel_loss": 0.389094, "path_tunnel_loss_count": 80, "path_tunnel_rescue025_recall_at_3": 0.975, "path_tunnel_rescue025_recall_total": 80, "path_tunnel_rescue050_recall_at_3": 0.975, "path_tunnel_rescue050_recall_total": 80, "path_tunnel_rescue100_recall_at_3": 0.975, "path_tunnel_rescue100_recall_total": 80, "path_tunnel_selection_loss": 0.231788, "path_tunnel_selection_loss_count": 80, "path_tunnel_support_positive_coverage_at_3": 0.779167, "path_tunnel_support_positive_total": 240, "path_tunnel_support_recall_at_3": 0.9875, "path_tunnel_support_recall_total": 80, "question_understanding_loss": 5.515932, "question_understanding_loss_count": 80, "recall_event24_positive_total": 215, "recall_event_positive_coverage_at_24": 0.995349, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 80, "recall_loss": 0.934783, "recall_loss_count": 80, "recall_selection_loss": 0.0, "recall_selection_loss_count": 80, "rows_evaluated": 80, "rows_total": 439, "samples": 80, "supervision_bucket_metrics": {"typed_tunnel_multi_positive": {"answer_plan_current_top1_accuracy": 0.346154, "answer_plan_current_total": 78, "answer_plan_selected_positive_coverage_at_5": 0.748837, "answer_plan_selected_positive_total": 215, "answer_plan_selected_recall_at_5": 0.95, "answer_plan_selected_total": 80, "event5_positive_total": 215, "event_positive_coverage_at_5": 0.986046, "event_recall_at_1": 0.875, "event_recall_at_5": 1.0, "event_recall_total": 80, "path3_positive_total": 240, "path_positive_coverage_at_3": 0.825, "path_recall_at_3": 0.9875, "path_recall_total": 80, "recall_event24_positive_total": 215, "recall_event_positive_coverage_at_24": 0.995349, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 80, "samples": 80, "temporal_accuracy": 0.2, "temporal_total": 5}}, "temporal_accuracy": 0.2, "temporal_loss": 0.553871, "temporal_loss_count": 5, "temporal_total": 5, "token_role_loss": 0.0, "token_role_loss_count": 11, "training_weight_max": 1.35, "training_weight_mean": 1.35, "training_weight_min": 1.35}, "val": {"answer_plan_current_top1_accuracy": 0.4375, "answer_plan_current_total": 32, "answer_plan_loss": 0.525668, "answer_plan_loss_count": 32, "answer_plan_selected_positive_coverage_at_5": 0.866667, "answer_plan_selected_positive_total": 75, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 32, "answer_refusal_loss": 0.0, "answer_refusal_loss_count": 32, "answer_type_loss": 6.01012, "answer_type_loss_count": 32, "answer_type_metrics": {"multi_evidence": {"answer_plan_current_top1_accuracy": 0.4375, "answer_plan_current_total": 16, "answer_plan_selected_positive_coverage_at_5": 0.837209, "answer_plan_selected_positive_total": 43, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 16, "event5_positive_total": 43, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 1.0, "event_recall_at_5": 1.0, "event_recall_total": 16, "path3_positive_total": 48, "path_positive_coverage_at_3": 0.833334, "path_recall_at_3": 1.0, "path_recall_total": 16, "recall_event24_positive_total": 43, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 16, "samples": 16, "temporal_accuracy": 0.0, "temporal_total": 0}, "time": {"answer_plan_current_top1_accuracy": 0.4375, "answer_plan_current_total": 16, "answer_plan_selected_positive_coverage_at_5": 0.90625, "answer_plan_selected_positive_total": 32, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 16, "event5_positive_total": 32, "event_positive_coverage_at_5": 0.96875, "event_recall_at_1": 0.8125, "event_recall_at_5": 1.0, "event_recall_total": 16, "path3_positive_total": 48, "path_positive_coverage_at_3": 0.75, "path_recall_at_3": 1.0, "path_recall_total": 16, "recall_event24_positive_total": 32, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 16, "samples": 16, "temporal_accuracy": 0.375, "temporal_total": 16}}, "evaluation_limited": true, "event5_positive_total": 75, "event_distractor_loss": 0.639759, "event_distractor_loss_count": 32, "event_hard_negative_loss": 0.075601, "event_hard_negative_loss_count": 32, "event_loss": 0.393289, "event_loss_count": 32, "event_matrix_delta_loss": 0.125742, "event_matrix_delta_loss_count": 32, "event_positive_coverage_at_5": 0.986667, "event_recall_at_1": 0.90625, "event_recall_at_5": 1.0, "event_recall_total": 32, "event_selection_loss": 0.018588, "event_selection_loss_count": 32, "event_tunnel_loss": 0.245768, "event_tunnel_loss_count": 32, "event_tunnel_selection_loss": 0.0, "event_tunnel_selection_loss_count": 32, "final_event_set_loss": 0.023793, "final_event_set_loss_count": 32, "graph_error_count": 0, "loss": 3.710867, "loss_group_count": 4, "memory_router_exact_match": 0.53125, "memory_router_f1": 0.934896, "memory_router_loss": 0.549485, "memory_router_loss_count": 32, "memory_router_total": 32, "path3_positive_total": 96, "path_hard_negative_loss": 0.373631, "path_hard_negative_loss_count": 32, "path_loss": 0.54686, "path_loss_count": 32, "path_matrix_delta_loss": 0.549039, "path_matrix_delta_loss_count": 32, "path_positive_coverage_at_3": 0.791667, "path_recall_at_3": 1.0, "path_recall_total": 32, "path_selection_loss": 0.34008, "path_selection_loss_count": 32, "path_tunnel_delta_loss": 0.673943, "path_tunnel_delta_loss_count": 32, "path_tunnel_delta_positive_coverage_at_3": 0.385417, "path_tunnel_delta_positive_total": 96, "path_tunnel_delta_recall_at_3": 0.75, "path_tunnel_delta_recall_total": 32, "path_tunnel_loss": 0.405298, "path_tunnel_loss_count": 32, "path_tunnel_rescue025_recall_at_3": 1.0, "path_tunnel_rescue025_recall_total": 32, "path_tunnel_rescue050_recall_at_3": 1.0, "path_tunnel_rescue050_recall_total": 32, "path_tunnel_rescue100_recall_at_3": 1.0, "path_tunnel_rescue100_recall_total": 32, "path_tunnel_selection_loss": 0.264057, "path_tunnel_selection_loss_count": 32, "path_tunnel_support_positive_coverage_at_3": 0.770834, "path_tunnel_support_positive_total": 96, "path_tunnel_support_recall_at_3": 1.0, "path_tunnel_support_recall_total": 32, "question_understanding_loss": 3.868408, "question_understanding_loss_count": 32, "recall_event24_positive_total": 75, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 32, "recall_loss": 0.850204, "recall_loss_count": 32, "recall_selection_loss": 0.0, "recall_selection_loss_count": 32, "rows_evaluated": 32, "rows_total": 55, "samples": 32, "supervision_bucket_metrics": {"typed_tunnel_multi_positive": {"answer_plan_current_top1_accuracy": 0.4375, "answer_plan_current_total": 32, "answer_plan_selected_positive_coverage_at_5": 0.866667, "answer_plan_selected_positive_total": 75, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 32, "event5_positive_total": 75, "event_positive_coverage_at_5": 0.986667, "event_recall_at_1": 0.90625, "event_recall_at_5": 1.0, "event_recall_total": 32, "path3_positive_total": 96, "path_positive_coverage_at_3": 0.791667, "path_recall_at_3": 1.0, "path_recall_total": 32, "recall_event24_positive_total": 75, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 32, "samples": 32, "temporal_accuracy": 0.375, "temporal_total": 16}}, "temporal_accuracy": 0.375, "temporal_loss": 0.504512, "temporal_loss_count": 16, "temporal_total": 16, "token_role_loss": 0.0, "token_role_loss_count": 4, "training_weight_max": 1.35, "training_weight_mean": 1.35, "training_weight_min": 1.35}, "val_selection_enabled": true, "val_selection_score": 118.164133}, {"best_val_loss": 3.718748, "best_val_selection_key": [1.0, 1.0, 1.0, 0.428571, -3.718748], "best_val_selection_score": 118.424107, "epoch": 4, "global_step": 380, "is_best": true, "patience": 0, "total_steps": 704, "train": {"answer_plan_current_top1_accuracy": 0.392405, "answer_plan_current_total": 79, "answer_plan_loss": 0.54617, "answer_plan_loss_count": 80, "answer_plan_selected_positive_coverage_at_5": 0.838863, "answer_plan_selected_positive_total": 211, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 80, "answer_refusal_loss": 0.0, "answer_refusal_loss_count": 80, "answer_type_loss": 5.782481, "answer_type_loss_count": 80, "answer_type_metrics": {"multi_evidence": {"answer_plan_current_top1_accuracy": 0.405405, "answer_plan_current_total": 74, "answer_plan_selected_positive_coverage_at_5": 0.835821, "answer_plan_selected_positive_total": 201, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 75, "event5_positive_total": 201, "event_positive_coverage_at_5": 0.985075, "event_recall_at_1": 0.84, "event_recall_at_5": 1.0, "event_recall_total": 75, "path3_positive_total": 225, "path_positive_coverage_at_3": 0.862222, "path_recall_at_3": 0.986667, "path_recall_total": 75, "recall_event24_positive_total": 201, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 75, "samples": 75, "temporal_accuracy": 0.0, "temporal_total": 0}, "time": {"answer_plan_current_top1_accuracy": 0.2, "answer_plan_current_total": 5, "answer_plan_selected_positive_coverage_at_5": 0.9, "answer_plan_selected_positive_total": 10, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 5, "event5_positive_total": 10, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 0.6, "event_recall_at_5": 1.0, "event_recall_total": 5, "path3_positive_total": 15, "path_positive_coverage_at_3": 0.8, "path_recall_at_3": 1.0, "path_recall_total": 5, "recall_event24_positive_total": 10, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 5, "samples": 5, "temporal_accuracy": 0.6, "temporal_total": 5}}, "evaluation_limited": true, "event5_positive_total": 211, "event_distractor_loss": 0.66659, "event_distractor_loss_count": 80, "event_hard_negative_loss": 0.087507, "event_hard_negative_loss_count": 80, "event_loss": 0.367468, "event_loss_count": 80, "event_matrix_delta_loss": 0.132377, "event_matrix_delta_loss_count": 80, "event_positive_coverage_at_5": 0.985782, "event_recall_at_1": 0.825, "event_recall_at_5": 1.0, "event_recall_total": 80, "event_selection_loss": 0.015455, "event_selection_loss_count": 80, "event_tunnel_loss": 0.241893, "event_tunnel_loss_count": 80, "event_tunnel_selection_loss": 0.0, "event_tunnel_selection_loss_count": 80, "final_event_set_loss": 0.020541, "final_event_set_loss_count": 80, "graph_error_count": 0, "loss": 3.824029, "loss_group_count": 11, "memory_router_exact_match": 0.3625, "memory_router_f1": 0.845833, "memory_router_loss": 1.465376, "memory_router_loss_count": 80, "memory_router_total": 80, "path3_positive_total": 240, "path_hard_negative_loss": 0.202253, "path_hard_negative_loss_count": 80, "path_loss": 0.441408, "path_loss_count": 80, "path_matrix_delta_loss": 0.484976, "path_matrix_delta_loss_count": 80, "path_positive_coverage_at_3": 0.858333, "path_recall_at_3": 0.9875, "path_recall_total": 80, "path_selection_loss": 0.254877, "path_selection_loss_count": 80, "path_tunnel_delta_loss": 0.654535, "path_tunnel_delta_loss_count": 80, "path_tunnel_delta_positive_coverage_at_3": 0.529167, "path_tunnel_delta_positive_total": 240, "path_tunnel_delta_recall_at_3": 0.85, "path_tunnel_delta_recall_total": 80, "path_tunnel_loss": 0.383018, "path_tunnel_loss_count": 80, "path_tunnel_rescue025_recall_at_3": 0.9875, "path_tunnel_rescue025_recall_total": 80, "path_tunnel_rescue050_recall_at_3": 0.9875, "path_tunnel_rescue050_recall_total": 80, "path_tunnel_rescue100_recall_at_3": 0.9875, "path_tunnel_rescue100_recall_total": 80, "path_tunnel_selection_loss": 0.20957, "path_tunnel_selection_loss_count": 80, "path_tunnel_support_positive_coverage_at_3": 0.7375, "path_tunnel_support_positive_total": 240, "path_tunnel_support_recall_at_3": 0.95, "path_tunnel_support_recall_total": 80, "question_understanding_loss": 5.429164, "question_understanding_loss_count": 80, "recall_event24_positive_total": 211, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 80, "recall_loss": 0.896912, "recall_loss_count": 80, "recall_selection_loss": 0.0, "recall_selection_loss_count": 80, "rows_evaluated": 80, "rows_total": 439, "samples": 80, "supervision_bucket_metrics": {"typed_tunnel_multi_positive": {"answer_plan_current_top1_accuracy": 0.392405, "answer_plan_current_total": 79, "answer_plan_selected_positive_coverage_at_5": 0.838863, "answer_plan_selected_positive_total": 211, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 80, "event5_positive_total": 211, "event_positive_coverage_at_5": 0.985782, "event_recall_at_1": 0.825, "event_recall_at_5": 1.0, "event_recall_total": 80, "path3_positive_total": 240, "path_positive_coverage_at_3": 0.858333, "path_recall_at_3": 0.9875, "path_recall_total": 80, "recall_event24_positive_total": 211, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 80, "samples": 80, "temporal_accuracy": 0.6, "temporal_total": 5}}, "temporal_accuracy": 0.6, "temporal_loss": 0.546881, "temporal_loss_count": 5, "temporal_total": 5, "token_role_loss": 0.0, "token_role_loss_count": 11, "training_weight_max": 1.35, "training_weight_mean": 1.35, "training_weight_min": 1.35}, "val": {"answer_plan_current_top1_accuracy": 0.40625, "answer_plan_current_total": 32, "answer_plan_loss": 0.532802, "answer_plan_loss_count": 32, "answer_plan_selected_positive_coverage_at_5": 0.825, "answer_plan_selected_positive_total": 80, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 32, "answer_refusal_loss": 0.0, "answer_refusal_loss_count": 32, "answer_type_loss": 6.63429, "answer_type_loss_count": 32, "answer_type_metrics": {"multi_evidence": {"answer_plan_current_top1_accuracy": 0.4, "answer_plan_current_total": 25, "answer_plan_selected_positive_coverage_at_5": 0.80303, "answer_plan_selected_positive_total": 66, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 25, "event5_positive_total": 66, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 0.92, "event_recall_at_5": 1.0, "event_recall_total": 25, "path3_positive_total": 75, "path_positive_coverage_at_3": 0.866667, "path_recall_at_3": 1.0, "path_recall_total": 25, "recall_event24_positive_total": 66, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 25, "samples": 25, "temporal_accuracy": 0.0, "temporal_total": 0}, "time": {"answer_plan_current_top1_accuracy": 0.428571, "answer_plan_current_total": 7, "answer_plan_selected_positive_coverage_at_5": 0.928571, "answer_plan_selected_positive_total": 14, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 7, "event5_positive_total": 14, "event_positive_coverage_at_5": 0.928571, "event_recall_at_1": 1.0, "event_recall_at_5": 1.0, "event_recall_total": 7, "path3_positive_total": 21, "path_positive_coverage_at_3": 0.666667, "path_recall_at_3": 1.0, "path_recall_total": 7, "recall_event24_positive_total": 14, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 7, "samples": 7, "temporal_accuracy": 0.428571, "temporal_total": 7}}, "evaluation_limited": true, "event5_positive_total": 80, "event_distractor_loss": 0.643267, "event_distractor_loss_count": 32, "event_hard_negative_loss": 0.060095, "event_hard_negative_loss_count": 32, "event_loss": 0.359434, "event_loss_count": 32, "event_matrix_delta_loss": 0.125532, "event_matrix_delta_loss_count": 32, "event_positive_coverage_at_5": 0.9875, "event_recall_at_1": 0.9375, "event_recall_at_5": 1.0, "event_recall_total": 32, "event_selection_loss": 0.015075, "event_selection_loss_count": 32, "event_tunnel_loss": 0.239424, "event_tunnel_loss_count": 32, "event_tunnel_selection_loss": 0.0, "event_tunnel_selection_loss_count": 32, "final_event_set_loss": 0.014138, "final_event_set_loss_count": 32, "graph_error_count": 0, "loss": 3.718748, "loss_group_count": 4, "memory_router_exact_match": 0.25, "memory_router_f1": 0.899305, "memory_router_loss": 0.772448, "memory_router_loss_count": 32, "memory_router_total": 32, "path3_positive_total": 96, "path_hard_negative_loss": 0.304885, "path_hard_negative_loss_count": 32, "path_loss": 0.453625, "path_loss_count": 32, "path_matrix_delta_loss": 0.617056, "path_matrix_delta_loss_count": 32, "path_positive_coverage_at_3": 0.822917, "path_recall_at_3": 1.0, "path_recall_total": 32, "path_selection_loss": 0.25727, "path_selection_loss_count": 32, "path_tunnel_delta_loss": 0.678382, "path_tunnel_delta_loss_count": 32, "path_tunnel_delta_positive_coverage_at_3": 0.447917, "path_tunnel_delta_positive_total": 96, "path_tunnel_delta_recall_at_3": 0.71875, "path_tunnel_delta_recall_total": 32, "path_tunnel_loss": 0.38617, "path_tunnel_loss_count": 32, "path_tunnel_rescue025_recall_at_3": 1.0, "path_tunnel_rescue025_recall_total": 32, "path_tunnel_rescue050_recall_at_3": 1.0, "path_tunnel_rescue050_recall_total": 32, "path_tunnel_rescue100_recall_at_3": 1.0, "path_tunnel_rescue100_recall_total": 32, "path_tunnel_selection_loss": 0.191212, "path_tunnel_selection_loss_count": 32, "path_tunnel_support_positive_coverage_at_3": 0.8125, "path_tunnel_support_positive_total": 96, "path_tunnel_support_recall_at_3": 1.0, "path_tunnel_support_recall_total": 32, "question_understanding_loss": 3.943016, "question_understanding_loss_count": 32, "recall_event24_positive_total": 80, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 32, "recall_loss": 0.874398, "recall_loss_count": 32, "recall_selection_loss": 0.0, "recall_selection_loss_count": 32, "rows_evaluated": 32, "rows_total": 55, "samples": 32, "supervision_bucket_metrics": {"typed_tunnel_multi_positive": {"answer_plan_current_top1_accuracy": 0.40625, "answer_plan_current_total": 32, "answer_plan_selected_positive_coverage_at_5": 0.825, "answer_plan_selected_positive_total": 80, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 32, "event5_positive_total": 80, "event_positive_coverage_at_5": 0.9875, "event_recall_at_1": 0.9375, "event_recall_at_5": 1.0, "event_recall_total": 32, "path3_positive_total": 96, "path_positive_coverage_at_3": 0.822917, "path_recall_at_3": 1.0, "path_recall_total": 32, "recall_event24_positive_total": 80, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 32, "samples": 32, "temporal_accuracy": 0.428571, "temporal_total": 7}}, "temporal_accuracy": 0.428571, "temporal_loss": 0.499198, "temporal_loss_count": 7, "temporal_total": 7, "token_role_loss": 0.0, "token_role_loss_count": 4, "training_weight_max": 1.35, "training_weight_mean": 1.35, "training_weight_min": 1.35}, "val_selection_enabled": true, "val_selection_score": 118.424107}, {"best_val_loss": 3.718748, "best_val_selection_key": [1.0, 1.0, 1.0, 0.428571, -3.718748], "best_val_selection_score": 118.424107, "epoch": 5, "global_step": 474, "is_best": false, "patience": 1, "total_steps": 704, "train": {"answer_plan_current_top1_accuracy": 0.425, "answer_plan_current_total": 80, "answer_plan_loss": 0.556815, "answer_plan_loss_count": 80, "answer_plan_selected_positive_coverage_at_5": 0.828194, "answer_plan_selected_positive_total": 227, "answer_plan_selected_recall_at_5": 0.9875, "answer_plan_selected_total": 80, "answer_refusal_loss": 0.0, "answer_refusal_loss_count": 80, "answer_type_loss": 5.118271, "answer_type_loss_count": 80, "answer_type_metrics": {"multi_evidence": {"answer_plan_current_top1_accuracy": 0.459459, "answer_plan_current_total": 74, "answer_plan_selected_positive_coverage_at_5": 0.837209, "answer_plan_selected_positive_total": 215, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 74, "event5_positive_total": 215, "event_positive_coverage_at_5": 0.986047, "event_recall_at_1": 0.945946, "event_recall_at_5": 1.0, "event_recall_total": 74, "path3_positive_total": 222, "path_positive_coverage_at_3": 0.900901, "path_recall_at_3": 0.986486, "path_recall_total": 74, "recall_event24_positive_total": 215, "recall_event_positive_coverage_at_24": 0.986047, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 74, "samples": 74, "temporal_accuracy": 0.0, "temporal_total": 0}, "time": {"answer_plan_current_top1_accuracy": 0.0, "answer_plan_current_total": 6, "answer_plan_selected_positive_coverage_at_5": 0.666667, "answer_plan_selected_positive_total": 12, "answer_plan_selected_recall_at_5": 0.833333, "answer_plan_selected_total": 6, "event5_positive_total": 12, "event_positive_coverage_at_5": 0.916667, "event_recall_at_1": 0.833333, "event_recall_at_5": 1.0, "event_recall_total": 6, "path3_positive_total": 18, "path_positive_coverage_at_3": 0.5, "path_recall_at_3": 0.833333, "path_recall_total": 6, "recall_event24_positive_total": 12, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 6, "samples": 6, "temporal_accuracy": 0.0, "temporal_total": 6}}, "evaluation_limited": true, "event5_positive_total": 227, "event_distractor_loss": 0.673871, "event_distractor_loss_count": 80, "event_hard_negative_loss": 0.05566, "event_hard_negative_loss_count": 80, "event_loss": 0.311322, "event_loss_count": 80, "event_matrix_delta_loss": 0.136033, "event_matrix_delta_loss_count": 80, "event_positive_coverage_at_5": 0.982379, "event_recall_at_1": 0.9375, "event_recall_at_5": 1.0, "event_recall_total": 80, "event_selection_loss": 0.007061, "event_selection_loss_count": 80, "event_tunnel_loss": 0.202112, "event_tunnel_loss_count": 80, "event_tunnel_selection_loss": 0.0, "event_tunnel_selection_loss_count": 80, "final_event_set_loss": 0.024521, "final_event_set_loss_count": 80, "graph_error_count": 0, "loss": 3.451428, "loss_group_count": 11, "memory_router_exact_match": 0.3875, "memory_router_f1": 0.850347, "memory_router_loss": 1.325362, "memory_router_loss_count": 80, "memory_router_total": 80, "path3_positive_total": 240, "path_hard_negative_loss": 0.136119, "path_hard_negative_loss_count": 80, "path_loss": 0.386382, "path_loss_count": 80, "path_matrix_delta_loss": 0.493189, "path_matrix_delta_loss_count": 80, "path_positive_coverage_at_3": 0.870833, "path_recall_at_3": 0.975, "path_recall_total": 80, "path_selection_loss": 0.129158, "path_selection_loss_count": 80, "path_tunnel_delta_loss": 0.627449, "path_tunnel_delta_loss_count": 80, "path_tunnel_delta_positive_coverage_at_3": 0.483333, "path_tunnel_delta_positive_total": 240, "path_tunnel_delta_recall_at_3": 0.8125, "path_tunnel_delta_recall_total": 80, "path_tunnel_loss": 0.337236, "path_tunnel_loss_count": 80, "path_tunnel_rescue025_recall_at_3": 0.975, "path_tunnel_rescue025_recall_total": 80, "path_tunnel_rescue050_recall_at_3": 0.975, "path_tunnel_rescue050_recall_total": 80, "path_tunnel_rescue100_recall_at_3": 0.975, "path_tunnel_rescue100_recall_total": 80, "path_tunnel_selection_loss": 0.13618, "path_tunnel_selection_loss_count": 80, "path_tunnel_support_positive_coverage_at_3": 0.8625, "path_tunnel_support_positive_total": 240, "path_tunnel_support_recall_at_3": 0.9875, "path_tunnel_support_recall_total": 80, "question_understanding_loss": 5.293768, "question_understanding_loss_count": 80, "recall_event24_positive_total": 227, "recall_event_positive_coverage_at_24": 0.986784, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 80, "recall_loss": 0.843209, "recall_loss_count": 80, "recall_selection_loss": 0.0, "recall_selection_loss_count": 80, "rows_evaluated": 80, "rows_total": 439, "samples": 80, "supervision_bucket_metrics": {"typed_tunnel_multi_positive": {"answer_plan_current_top1_accuracy": 0.425, "answer_plan_current_total": 80, "answer_plan_selected_positive_coverage_at_5": 0.828194, "answer_plan_selected_positive_total": 227, "answer_plan_selected_recall_at_5": 0.9875, "answer_plan_selected_total": 80, "event5_positive_total": 227, "event_positive_coverage_at_5": 0.982379, "event_recall_at_1": 0.9375, "event_recall_at_5": 1.0, "event_recall_total": 80, "path3_positive_total": 240, "path_positive_coverage_at_3": 0.870833, "path_recall_at_3": 0.975, "path_recall_total": 80, "recall_event24_positive_total": 227, "recall_event_positive_coverage_at_24": 0.986784, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 80, "samples": 80, "temporal_accuracy": 0.0, "temporal_total": 6}}, "temporal_accuracy": 0.0, "temporal_loss": 0.57252, "temporal_loss_count": 6, "temporal_total": 6, "token_role_loss": 0.0, "token_role_loss_count": 11, "training_weight_max": 1.35, "training_weight_mean": 1.35, "training_weight_min": 1.35}, "val": {"answer_plan_current_top1_accuracy": 0.59375, "answer_plan_current_total": 32, "answer_plan_loss": 0.524282, "answer_plan_loss_count": 32, "answer_plan_selected_positive_coverage_at_5": 0.871795, "answer_plan_selected_positive_total": 78, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 32, "answer_refusal_loss": 0.0, "answer_refusal_loss_count": 32, "answer_type_loss": 5.920396, "answer_type_loss_count": 32, "answer_type_metrics": {"multi_evidence": {"answer_plan_current_top1_accuracy": 0.636364, "answer_plan_current_total": 22, "answer_plan_selected_positive_coverage_at_5": 0.862069, "answer_plan_selected_positive_total": 58, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 22, "event5_positive_total": 58, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 0.909091, "event_recall_at_5": 1.0, "event_recall_total": 22, "path3_positive_total": 66, "path_positive_coverage_at_3": 0.863637, "path_recall_at_3": 1.0, "path_recall_total": 22, "recall_event24_positive_total": 58, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 22, "samples": 22, "temporal_accuracy": 0.0, "temporal_total": 0}, "time": {"answer_plan_current_top1_accuracy": 0.5, "answer_plan_current_total": 10, "answer_plan_selected_positive_coverage_at_5": 0.9, "answer_plan_selected_positive_total": 20, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 10, "event5_positive_total": 20, "event_positive_coverage_at_5": 0.95, "event_recall_at_1": 0.9, "event_recall_at_5": 1.0, "event_recall_total": 10, "path3_positive_total": 30, "path_positive_coverage_at_3": 0.7, "path_recall_at_3": 1.0, "path_recall_total": 10, "recall_event24_positive_total": 20, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 10, "samples": 10, "temporal_accuracy": 0.4, "temporal_total": 10}}, "evaluation_limited": true, "event5_positive_total": 78, "event_distractor_loss": 0.63666, "event_distractor_loss_count": 32, "event_hard_negative_loss": 0.067466, "event_hard_negative_loss_count": 32, "event_loss": 0.359222, "event_loss_count": 32, "event_matrix_delta_loss": 0.113683, "event_matrix_delta_loss_count": 32, "event_positive_coverage_at_5": 0.987179, "event_recall_at_1": 0.90625, "event_recall_at_5": 1.0, "event_recall_total": 32, "event_selection_loss": 0.012552, "event_selection_loss_count": 32, "event_tunnel_loss": 0.245427, "event_tunnel_loss_count": 32, "event_tunnel_selection_loss": 0.0, "event_tunnel_selection_loss_count": 32, "final_event_set_loss": 0.014992, "final_event_set_loss_count": 32, "graph_error_count": 0, "loss": 3.512776, "loss_group_count": 4, "memory_router_exact_match": 0.34375, "memory_router_f1": 0.914062, "memory_router_loss": 0.548788, "memory_router_loss_count": 32, "memory_router_total": 32, "path3_positive_total": 96, "path_hard_negative_loss": 0.260774, "path_hard_negative_loss_count": 32, "path_loss": 0.439803, "path_loss_count": 32, "path_matrix_delta_loss": 0.534145, "path_matrix_delta_loss_count": 32, "path_positive_coverage_at_3": 0.8125, "path_recall_at_3": 1.0, "path_recall_total": 32, "path_selection_loss": 0.266705, "path_selection_loss_count": 32, "path_tunnel_delta_loss": 0.683555, "path_tunnel_delta_loss_count": 32, "path_tunnel_delta_positive_coverage_at_3": 0.53125, "path_tunnel_delta_positive_total": 96, "path_tunnel_delta_recall_at_3": 0.875, "path_tunnel_delta_recall_total": 32, "path_tunnel_loss": 0.376881, "path_tunnel_loss_count": 32, "path_tunnel_rescue025_recall_at_3": 1.0, "path_tunnel_rescue025_recall_total": 32, "path_tunnel_rescue050_recall_at_3": 1.0, "path_tunnel_rescue050_recall_total": 32, "path_tunnel_rescue100_recall_at_3": 1.0, "path_tunnel_rescue100_recall_total": 32, "path_tunnel_selection_loss": 0.204427, "path_tunnel_selection_loss_count": 32, "path_tunnel_support_positive_coverage_at_3": 0.833334, "path_tunnel_support_positive_total": 96, "path_tunnel_support_recall_at_3": 1.0, "path_tunnel_support_recall_total": 32, "question_understanding_loss": 4.045167, "question_understanding_loss_count": 32, "recall_event24_positive_total": 78, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 32, "recall_loss": 0.831306, "recall_loss_count": 32, "recall_selection_loss": 0.0, "recall_selection_loss_count": 32, "rows_evaluated": 32, "rows_total": 55, "samples": 32, "supervision_bucket_metrics": {"typed_tunnel_multi_positive": {"answer_plan_current_top1_accuracy": 0.59375, "answer_plan_current_total": 32, "answer_plan_selected_positive_coverage_at_5": 0.871795, "answer_plan_selected_positive_total": 78, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 32, "event5_positive_total": 78, "event_positive_coverage_at_5": 0.987179, "event_recall_at_1": 0.90625, "event_recall_at_5": 1.0, "event_recall_total": 32, "path3_positive_total": 96, "path_positive_coverage_at_3": 0.8125, "path_recall_at_3": 1.0, "path_recall_total": 32, "recall_event24_positive_total": 78, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 32, "samples": 32, "temporal_accuracy": 0.4, "temporal_total": 10}}, "temporal_accuracy": 0.4, "temporal_loss": 0.493367, "temporal_loss_count": 10, "temporal_total": 10, "token_role_loss": 0.0, "token_role_loss_count": 4, "training_weight_max": 1.35, "training_weight_mean": 1.35, "training_weight_min": 1.35}, "val_selection_enabled": true, "val_selection_score": 118.487224}, {"best_val_loss": 3.718748, "best_val_selection_key": [1.0, 1.0, 1.0, 0.428571, -3.718748], "best_val_selection_score": 118.424107, "epoch": 6, "global_step": 569, "is_best": false, "patience": 2, "total_steps": 704, "train": {"answer_plan_current_top1_accuracy": 0.375, "answer_plan_current_total": 80, "answer_plan_loss": 0.543159, "answer_plan_loss_count": 80, "answer_plan_selected_positive_coverage_at_5": 0.790244, "answer_plan_selected_positive_total": 205, "answer_plan_selected_recall_at_5": 0.9625, "answer_plan_selected_total": 80, "answer_refusal_loss": 0.0, "answer_refusal_loss_count": 80, "answer_type_loss": 4.808324, "answer_type_loss_count": 80, "answer_type_metrics": {"multi_evidence": {"answer_plan_current_top1_accuracy": 0.38961, "answer_plan_current_total": 77, "answer_plan_selected_positive_coverage_at_5": 0.798995, "answer_plan_selected_positive_total": 199, "answer_plan_selected_recall_at_5": 0.974026, "answer_plan_selected_total": 77, "event5_positive_total": 199, "event_positive_coverage_at_5": 0.98995, "event_recall_at_1": 0.87013, "event_recall_at_5": 1.0, "event_recall_total": 77, "path3_positive_total": 231, "path_positive_coverage_at_3": 0.874459, "path_recall_at_3": 0.961039, "path_recall_total": 77, "recall_event24_positive_total": 199, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 77, "samples": 77, "temporal_accuracy": 0.0, "temporal_total": 0}, "time": {"answer_plan_current_top1_accuracy": 0.0, "answer_plan_current_total": 3, "answer_plan_selected_positive_coverage_at_5": 0.5, "answer_plan_selected_positive_total": 6, "answer_plan_selected_recall_at_5": 0.666667, "answer_plan_selected_total": 3, "event5_positive_total": 6, "event_positive_coverage_at_5": 0.833333, "event_recall_at_1": 0.333333, "event_recall_at_5": 1.0, "event_recall_total": 3, "path3_positive_total": 9, "path_positive_coverage_at_3": 0.555555, "path_recall_at_3": 0.666667, "path_recall_total": 3, "recall_event24_positive_total": 6, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 3, "samples": 3, "temporal_accuracy": 0.0, "temporal_total": 3}}, "evaluation_limited": true, "event5_positive_total": 205, "event_distractor_loss": 0.661971, "event_distractor_loss_count": 80, "event_hard_negative_loss": 0.100501, "event_hard_negative_loss_count": 80, "event_loss": 0.37706, "event_loss_count": 80, "event_matrix_delta_loss": 0.130566, "event_matrix_delta_loss_count": 80, "event_positive_coverage_at_5": 0.985366, "event_recall_at_1": 0.85, "event_recall_at_5": 1.0, "event_recall_total": 80, "event_selection_loss": 0.013893, "event_selection_loss_count": 80, "event_tunnel_loss": 0.248382, "event_tunnel_loss_count": 80, "event_tunnel_selection_loss": 0.0, "event_tunnel_selection_loss_count": 80, "final_event_set_loss": 0.030997, "final_event_set_loss_count": 80, "graph_error_count": 0, "loss": 3.54849, "loss_group_count": 11, "memory_router_exact_match": 0.4, "memory_router_f1": 0.851339, "memory_router_loss": 1.375998, "memory_router_loss_count": 80, "memory_router_total": 80, "path3_positive_total": 240, "path_hard_negative_loss": 0.14603, "path_hard_negative_loss_count": 80, "path_loss": 0.419353, "path_loss_count": 80, "path_matrix_delta_loss": 0.530501, "path_matrix_delta_loss_count": 80, "path_positive_coverage_at_3": 0.8625, "path_recall_at_3": 0.95, "path_recall_total": 80, "path_selection_loss": 0.201806, "path_selection_loss_count": 80, "path_tunnel_delta_loss": 0.636599, "path_tunnel_delta_loss_count": 80, "path_tunnel_delta_positive_coverage_at_3": 0.4625, "path_tunnel_delta_positive_total": 240, "path_tunnel_delta_recall_at_3": 0.775, "path_tunnel_delta_recall_total": 80, "path_tunnel_loss": 0.368271, "path_tunnel_loss_count": 80, "path_tunnel_rescue025_recall_at_3": 0.95, "path_tunnel_rescue025_recall_total": 80, "path_tunnel_rescue050_recall_at_3": 0.925, "path_tunnel_rescue050_recall_total": 80, "path_tunnel_rescue100_recall_at_3": 0.9375, "path_tunnel_rescue100_recall_total": 80, "path_tunnel_selection_loss": 0.166694, "path_tunnel_selection_loss_count": 80, "path_tunnel_support_positive_coverage_at_3": 0.770833, "path_tunnel_support_positive_total": 240, "path_tunnel_support_recall_at_3": 0.9125, "path_tunnel_support_recall_total": 80, "question_understanding_loss": 5.215601, "question_understanding_loss_count": 80, "recall_event24_positive_total": 205, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 80, "recall_loss": 0.881145, "recall_loss_count": 80, "recall_selection_loss": 0.0, "recall_selection_loss_count": 80, "rows_evaluated": 80, "rows_total": 439, "samples": 80, "supervision_bucket_metrics": {"typed_tunnel_multi_positive": {"answer_plan_current_top1_accuracy": 0.375, "answer_plan_current_total": 80, "answer_plan_selected_positive_coverage_at_5": 0.790244, "answer_plan_selected_positive_total": 205, "answer_plan_selected_recall_at_5": 0.9625, "answer_plan_selected_total": 80, "event5_positive_total": 205, "event_positive_coverage_at_5": 0.985366, "event_recall_at_1": 0.85, "event_recall_at_5": 1.0, "event_recall_total": 80, "path3_positive_total": 240, "path_positive_coverage_at_3": 0.8625, "path_recall_at_3": 0.95, "path_recall_total": 80, "recall_event24_positive_total": 205, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 80, "samples": 80, "temporal_accuracy": 0.0, "temporal_total": 3}}, "temporal_accuracy": 0.0, "temporal_loss": 0.562548, "temporal_loss_count": 3, "temporal_total": 3, "token_role_loss": 0.0, "token_role_loss_count": 11, "training_weight_max": 1.35, "training_weight_mean": 1.35, "training_weight_min": 1.35}, "val": {"answer_plan_current_top1_accuracy": 0.4375, "answer_plan_current_total": 32, "answer_plan_loss": 0.532266, "answer_plan_loss_count": 32, "answer_plan_selected_positive_coverage_at_5": 0.822785, "answer_plan_selected_positive_total": 79, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 32, "answer_refusal_loss": 0.0, "answer_refusal_loss_count": 32, "answer_type_loss": 5.310994, "answer_type_loss_count": 32, "answer_type_metrics": {"multi_evidence": {"answer_plan_current_top1_accuracy": 0.5, "answer_plan_current_total": 22, "answer_plan_selected_positive_coverage_at_5": 0.81356, "answer_plan_selected_positive_total": 59, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 22, "event5_positive_total": 59, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 0.863636, "event_recall_at_5": 1.0, "event_recall_total": 22, "path3_positive_total": 66, "path_positive_coverage_at_3": 0.909091, "path_recall_at_3": 1.0, "path_recall_total": 22, "recall_event24_positive_total": 59, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 22, "samples": 22, "temporal_accuracy": 0.0, "temporal_total": 0}, "time": {"answer_plan_current_top1_accuracy": 0.3, "answer_plan_current_total": 10, "answer_plan_selected_positive_coverage_at_5": 0.85, "answer_plan_selected_positive_total": 20, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 10, "event5_positive_total": 20, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 0.8, "event_recall_at_5": 1.0, "event_recall_total": 10, "path3_positive_total": 30, "path_positive_coverage_at_3": 0.833333, "path_recall_at_3": 1.0, "path_recall_total": 10, "recall_event24_positive_total": 20, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 10, "samples": 10, "temporal_accuracy": 0.2, "temporal_total": 10}}, "evaluation_limited": true, "event5_positive_total": 79, "event_distractor_loss": 0.644329, "event_distractor_loss_count": 32, "event_hard_negative_loss": 0.071848, "event_hard_negative_loss_count": 32, "event_loss": 0.364922, "event_loss_count": 32, "event_matrix_delta_loss": 0.125276, "event_matrix_delta_loss_count": 32, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 0.84375, "event_recall_at_5": 1.0, "event_recall_total": 32, "event_selection_loss": 0.007421, "event_selection_loss_count": 32, "event_tunnel_loss": 0.238816, "event_tunnel_loss_count": 32, "event_tunnel_selection_loss": 0.0, "event_tunnel_selection_loss_count": 32, "final_event_set_loss": 0.012454, "final_event_set_loss_count": 32, "graph_error_count": 0, "loss": 3.330123, "loss_group_count": 4, "memory_router_exact_match": 0.375, "memory_router_f1": 0.921875, "memory_router_loss": 0.533256, "memory_router_loss_count": 32, "memory_router_total": 32, "path3_positive_total": 96, "path_hard_negative_loss": 0.245627, "path_hard_negative_loss_count": 32, "path_loss": 0.429462, "path_loss_count": 32, "path_matrix_delta_loss": 0.522375, "path_matrix_delta_loss_count": 32, "path_positive_coverage_at_3": 0.885417, "path_recall_at_3": 1.0, "path_recall_total": 32, "path_selection_loss": 0.182368, "path_selection_loss_count": 32, "path_tunnel_delta_loss": 0.671499, "path_tunnel_delta_loss_count": 32, "path_tunnel_delta_positive_coverage_at_3": 0.583333, "path_tunnel_delta_positive_total": 96, "path_tunnel_delta_recall_at_3": 0.8125, "path_tunnel_delta_recall_total": 32, "path_tunnel_loss": 0.368319, "path_tunnel_loss_count": 32, "path_tunnel_rescue025_recall_at_3": 1.0, "path_tunnel_rescue025_recall_total": 32, "path_tunnel_rescue050_recall_at_3": 1.0, "path_tunnel_rescue050_recall_total": 32, "path_tunnel_rescue100_recall_at_3": 1.0, "path_tunnel_rescue100_recall_total": 32, "path_tunnel_selection_loss": 0.164571, "path_tunnel_selection_loss_count": 32, "path_tunnel_support_positive_coverage_at_3": 0.791667, "path_tunnel_support_positive_total": 96, "path_tunnel_support_recall_at_3": 0.9375, "path_tunnel_support_recall_total": 32, "question_understanding_loss": 3.620285, "question_understanding_loss_count": 32, "recall_event24_positive_total": 79, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 32, "recall_loss": 0.86912, "recall_loss_count": 32, "recall_selection_loss": 0.0, "recall_selection_loss_count": 32, "rows_evaluated": 32, "rows_total": 55, "samples": 32, "supervision_bucket_metrics": {"typed_tunnel_multi_positive": {"answer_plan_current_top1_accuracy": 0.4375, "answer_plan_current_total": 32, "answer_plan_selected_positive_coverage_at_5": 0.822785, "answer_plan_selected_positive_total": 79, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 32, "event5_positive_total": 79, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 0.84375, "event_recall_at_5": 1.0, "event_recall_total": 32, "path3_positive_total": 96, "path_positive_coverage_at_3": 0.885417, "path_recall_at_3": 1.0, "path_recall_total": 32, "recall_event24_positive_total": 79, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 32, "samples": 32, "temporal_accuracy": 0.2, "temporal_total": 10}}, "temporal_accuracy": 0.2, "temporal_loss": 0.512047, "temporal_loss_count": 10, "temporal_total": 10, "token_role_loss": 0.0, "token_role_loss_count": 4, "training_weight_max": 1.35, "training_weight_mean": 1.35, "training_weight_min": 1.35}, "val_selection_enabled": true, "val_selection_score": 117.669877}, {"best_val_loss": 3.718748, "best_val_selection_key": [1.0, 1.0, 1.0, 0.428571, -3.718748], "best_val_selection_score": 118.424107, "epoch": 7, "global_step": 665, "is_best": false, "patience": 3, "total_steps": 704, "train": {"answer_plan_current_top1_accuracy": 0.379747, "answer_plan_current_total": 79, "answer_plan_loss": 0.549526, "answer_plan_loss_count": 80, "answer_plan_selected_positive_coverage_at_5": 0.821101, "answer_plan_selected_positive_total": 218, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 80, "answer_refusal_loss": 0.0, "answer_refusal_loss_count": 80, "answer_type_loss": 5.16927, "answer_type_loss_count": 80, "answer_type_metrics": {"multi_evidence": {"answer_plan_current_top1_accuracy": 0.376623, "answer_plan_current_total": 77, "answer_plan_selected_positive_coverage_at_5": 0.821596, "answer_plan_selected_positive_total": 213, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 78, "event5_positive_total": 213, "event_positive_coverage_at_5": 0.981221, "event_recall_at_1": 0.846154, "event_recall_at_5": 1.0, "event_recall_total": 78, "path3_positive_total": 234, "path_positive_coverage_at_3": 0.910256, "path_recall_at_3": 1.0, "path_recall_total": 78, "recall_event24_positive_total": 213, "recall_event_positive_coverage_at_24": 0.995305, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 78, "samples": 78, "temporal_accuracy": 0.0, "temporal_total": 0}, "time": {"answer_plan_current_top1_accuracy": 0.5, "answer_plan_current_total": 2, "answer_plan_selected_positive_coverage_at_5": 0.8, "answer_plan_selected_positive_total": 5, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 2, "event5_positive_total": 5, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 1.0, "event_recall_at_5": 1.0, "event_recall_total": 2, "path3_positive_total": 6, "path_positive_coverage_at_3": 1.0, "path_recall_at_3": 1.0, "path_recall_total": 2, "recall_event24_positive_total": 5, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 2, "samples": 2, "temporal_accuracy": 0.0, "temporal_total": 2}}, "evaluation_limited": true, "event5_positive_total": 218, "event_distractor_loss": 0.666098, "event_distractor_loss_count": 80, "event_hard_negative_loss": 0.078884, "event_hard_negative_loss_count": 80, "event_loss": 0.323909, "event_loss_count": 80, "event_matrix_delta_loss": 0.128518, "event_matrix_delta_loss_count": 80, "event_positive_coverage_at_5": 0.981651, "event_recall_at_1": 0.85, "event_recall_at_5": 1.0, "event_recall_total": 80, "event_selection_loss": 0.014046, "event_selection_loss_count": 80, "event_tunnel_loss": 0.213497, "event_tunnel_loss_count": 80, "event_tunnel_selection_loss": 0.0, "event_tunnel_selection_loss_count": 80, "final_event_set_loss": 0.013138, "final_event_set_loss_count": 80, "graph_error_count": 0, "loss": 3.466264, "loss_group_count": 11, "memory_router_exact_match": 0.375, "memory_router_f1": 0.845089, "memory_router_loss": 1.479962, "memory_router_loss_count": 80, "memory_router_total": 80, "path3_positive_total": 240, "path_hard_negative_loss": 0.117886, "path_hard_negative_loss_count": 80, "path_loss": 0.369553, "path_loss_count": 80, "path_matrix_delta_loss": 0.454712, "path_matrix_delta_loss_count": 80, "path_positive_coverage_at_3": 0.9125, "path_recall_at_3": 1.0, "path_recall_total": 80, "path_selection_loss": 0.12883, "path_selection_loss_count": 80, "path_tunnel_delta_loss": 0.60523, "path_tunnel_delta_loss_count": 80, "path_tunnel_delta_positive_coverage_at_3": 0.479167, "path_tunnel_delta_positive_total": 240, "path_tunnel_delta_recall_at_3": 0.75, "path_tunnel_delta_recall_total": 80, "path_tunnel_loss": 0.338907, "path_tunnel_loss_count": 80, "path_tunnel_rescue025_recall_at_3": 1.0, "path_tunnel_rescue025_recall_total": 80, "path_tunnel_rescue050_recall_at_3": 0.9875, "path_tunnel_rescue050_recall_total": 80, "path_tunnel_rescue100_recall_at_3": 0.9875, "path_tunnel_rescue100_recall_total": 80, "path_tunnel_selection_loss": 0.14157, "path_tunnel_selection_loss_count": 80, "path_tunnel_support_positive_coverage_at_3": 0.829167, "path_tunnel_support_positive_total": 240, "path_tunnel_support_recall_at_3": 0.9375, "path_tunnel_support_recall_total": 80, "question_understanding_loss": 5.293586, "question_understanding_loss_count": 80, "recall_event24_positive_total": 218, "recall_event_positive_coverage_at_24": 0.995413, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 80, "recall_loss": 0.820672, "recall_loss_count": 80, "recall_selection_loss": 0.0, "recall_selection_loss_count": 80, "rows_evaluated": 80, "rows_total": 439, "samples": 80, "supervision_bucket_metrics": {"typed_tunnel_multi_positive": {"answer_plan_current_top1_accuracy": 0.379747, "answer_plan_current_total": 79, "answer_plan_selected_positive_coverage_at_5": 0.821101, "answer_plan_selected_positive_total": 218, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 80, "event5_positive_total": 218, "event_positive_coverage_at_5": 0.981651, "event_recall_at_1": 0.85, "event_recall_at_5": 1.0, "event_recall_total": 80, "path3_positive_total": 240, "path_positive_coverage_at_3": 0.9125, "path_recall_at_3": 1.0, "path_recall_total": 80, "recall_event24_positive_total": 218, "recall_event_positive_coverage_at_24": 0.995413, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 80, "samples": 80, "temporal_accuracy": 0.0, "temporal_total": 2}}, "temporal_accuracy": 0.0, "temporal_loss": 0.513602, "temporal_loss_count": 2, "temporal_total": 2, "token_role_loss": 0.0, "token_role_loss_count": 11, "training_weight_max": 1.35, "training_weight_mean": 1.35, "training_weight_min": 1.35}, "val": {"answer_plan_current_top1_accuracy": 0.53125, "answer_plan_current_total": 32, "answer_plan_loss": 0.530247, "answer_plan_loss_count": 32, "answer_plan_selected_positive_coverage_at_5": 0.848101, "answer_plan_selected_positive_total": 79, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 32, "answer_refusal_loss": 0.0, "answer_refusal_loss_count": 32, "answer_type_loss": 6.441788, "answer_type_loss_count": 32, "answer_type_metrics": {"multi_evidence": {"answer_plan_current_top1_accuracy": 0.545455, "answer_plan_current_total": 22, "answer_plan_selected_positive_coverage_at_5": 0.847458, "answer_plan_selected_positive_total": 59, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 22, "event5_positive_total": 59, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 0.909091, "event_recall_at_5": 1.0, "event_recall_total": 22, "path3_positive_total": 66, "path_positive_coverage_at_3": 0.924242, "path_recall_at_3": 1.0, "path_recall_total": 22, "recall_event24_positive_total": 59, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 22, "samples": 22, "temporal_accuracy": 0.0, "temporal_total": 0}, "time": {"answer_plan_current_top1_accuracy": 0.5, "answer_plan_current_total": 10, "answer_plan_selected_positive_coverage_at_5": 0.85, "answer_plan_selected_positive_total": 20, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 10, "event5_positive_total": 20, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 0.7, "event_recall_at_5": 1.0, "event_recall_total": 10, "path3_positive_total": 30, "path_positive_coverage_at_3": 0.733334, "path_recall_at_3": 1.0, "path_recall_total": 10, "recall_event24_positive_total": 20, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 10, "samples": 10, "temporal_accuracy": 0.3, "temporal_total": 10}}, "evaluation_limited": true, "event5_positive_total": 79, "event_distractor_loss": 0.646066, "event_distractor_loss_count": 32, "event_hard_negative_loss": 0.062803, "event_hard_negative_loss_count": 32, "event_loss": 0.379233, "event_loss_count": 32, "event_matrix_delta_loss": 0.125441, "event_matrix_delta_loss_count": 32, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 0.84375, "event_recall_at_5": 1.0, "event_recall_total": 32, "event_selection_loss": 0.001734, "event_selection_loss_count": 32, "event_tunnel_loss": 0.248372, "event_tunnel_loss_count": 32, "event_tunnel_selection_loss": 0.0, "event_tunnel_selection_loss_count": 32, "final_event_set_loss": 0.012893, "final_event_set_loss_count": 32, "graph_error_count": 0, "loss": 3.604544, "loss_group_count": 4, "memory_router_exact_match": 0.34375, "memory_router_f1": 0.918403, "memory_router_loss": 0.575108, "memory_router_loss_count": 32, "memory_router_total": 32, "path3_positive_total": 96, "path_hard_negative_loss": 0.23831, "path_hard_negative_loss_count": 32, "path_loss": 0.422076, "path_loss_count": 32, "path_matrix_delta_loss": 0.564214, "path_matrix_delta_loss_count": 32, "path_positive_coverage_at_3": 0.864584, "path_recall_at_3": 1.0, "path_recall_total": 32, "path_selection_loss": 0.189845, "path_selection_loss_count": 32, "path_tunnel_delta_loss": 0.658501, "path_tunnel_delta_loss_count": 32, "path_tunnel_delta_positive_coverage_at_3": 0.614584, "path_tunnel_delta_positive_total": 96, "path_tunnel_delta_recall_at_3": 0.84375, "path_tunnel_delta_recall_total": 32, "path_tunnel_loss": 0.360756, "path_tunnel_loss_count": 32, "path_tunnel_rescue025_recall_at_3": 1.0, "path_tunnel_rescue025_recall_total": 32, "path_tunnel_rescue050_recall_at_3": 1.0, "path_tunnel_rescue050_recall_total": 32, "path_tunnel_rescue100_recall_at_3": 1.0, "path_tunnel_rescue100_recall_total": 32, "path_tunnel_selection_loss": 0.183911, "path_tunnel_selection_loss_count": 32, "path_tunnel_support_positive_coverage_at_3": 0.802083, "path_tunnel_support_positive_total": 96, "path_tunnel_support_recall_at_3": 0.96875, "path_tunnel_support_recall_total": 32, "question_understanding_loss": 3.751869, "question_understanding_loss_count": 32, "recall_event24_positive_total": 79, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 32, "recall_loss": 0.894676, "recall_loss_count": 32, "recall_selection_loss": 0.0, "recall_selection_loss_count": 32, "rows_evaluated": 32, "rows_total": 55, "samples": 32, "supervision_bucket_metrics": {"typed_tunnel_multi_positive": {"answer_plan_current_top1_accuracy": 0.53125, "answer_plan_current_total": 32, "answer_plan_selected_positive_coverage_at_5": 0.848101, "answer_plan_selected_positive_total": 79, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 32, "event5_positive_total": 79, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 0.84375, "event_recall_at_5": 1.0, "event_recall_total": 32, "path3_positive_total": 96, "path_positive_coverage_at_3": 0.864584, "path_recall_at_3": 1.0, "path_recall_total": 32, "recall_event24_positive_total": 79, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 32, "samples": 32, "temporal_accuracy": 0.3, "temporal_total": 10}}, "temporal_accuracy": 0.3, "temporal_loss": 0.50511, "temporal_loss_count": 10, "temporal_total": 10, "token_role_loss": 0.0, "token_role_loss_count": 4, "training_weight_max": 1.35, "training_weight_mean": 1.35, "training_weight_min": 1.35}, "val_selection_enabled": true, "val_selection_score": 117.895456}], "memory_router_variant": "question_pooled_multilabel_v1", "memory_tunnel_variant": "chain_depth_tunnel_support_delta_v1", "message_passing_variant": "support_to_event_bidirectional_shared_typed_v1", "metrics": {"test": {"answer_plan_current_top1_accuracy": 0.296296, "answer_plan_current_total": 54, "answer_plan_loss": 0.519994, "answer_plan_loss_count": 54, "answer_plan_selected_positive_coverage_at_5": 0.778761, "answer_plan_selected_positive_total": 113, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 54, "answer_refusal_loss": 0.0, "answer_refusal_loss_count": 54, "answer_type_loss": 3.65495, "answer_type_loss_count": 54, "answer_type_metrics": {"multi_evidence": {"answer_plan_current_top1_accuracy": 0.129032, "answer_plan_current_total": 31, "answer_plan_selected_positive_coverage_at_5": 0.776119, "answer_plan_selected_positive_total": 67, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 31, "event5_positive_total": 67, "event_positive_coverage_at_5": 0.940298, "event_recall_at_1": 0.903226, "event_recall_at_5": 1.0, "event_recall_total": 31, "path3_positive_total": 93, "path_positive_coverage_at_3": 0.849462, "path_recall_at_3": 1.0, "path_recall_total": 31, "recall_event24_positive_total": 67, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 31, "samples": 31, "temporal_accuracy": 0.0, "temporal_total": 0}, "time": {"answer_plan_current_top1_accuracy": 0.521739, "answer_plan_current_total": 23, "answer_plan_selected_positive_coverage_at_5": 0.782609, "answer_plan_selected_positive_total": 46, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 23, "event5_positive_total": 46, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 0.913044, "event_recall_at_5": 1.0, "event_recall_total": 23, "path3_positive_total": 69, "path_positive_coverage_at_3": 0.855072, "path_recall_at_3": 1.0, "path_recall_total": 23, "recall_event24_positive_total": 46, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 23, "samples": 23, "temporal_accuracy": 0.434783, "temporal_total": 23}}, "evaluation_limited": false, "event5_positive_total": 113, "event_distractor_loss": 0.652414, "event_distractor_loss_count": 54, "event_hard_negative_loss": 0.095313, "event_hard_negative_loss_count": 54, "event_loss": 0.388887, "event_loss_count": 54, "event_matrix_delta_loss": 0.121482, "event_matrix_delta_loss_count": 54, "event_positive_coverage_at_5": 0.964602, "event_recall_at_1": 0.907407, "event_recall_at_5": 1.0, "event_recall_total": 54, "event_selection_loss": 0.033558, "event_selection_loss_count": 54, "event_tunnel_loss": 0.257923, "event_tunnel_loss_count": 54, "event_tunnel_selection_loss": 0.0, "event_tunnel_selection_loss_count": 54, "final_event_set_loss": 0.006052, "final_event_set_loss_count": 54, "graph_error_count": 0, "loss": 3.097328, "loss_group_count": 7, "memory_router_exact_match": 0.851852, "memory_router_f1": 0.962963, "memory_router_loss": 0.407471, "memory_router_loss_count": 54, "memory_router_total": 54, "path3_positive_total": 162, "path_hard_negative_loss": 0.211284, "path_hard_negative_loss_count": 54, "path_loss": 0.429968, "path_loss_count": 54, "path_matrix_delta_loss": 0.494685, "path_matrix_delta_loss_count": 54, "path_positive_coverage_at_3": 0.851852, "path_recall_at_3": 1.0, "path_recall_total": 54, "path_selection_loss": 0.245353, "path_selection_loss_count": 54, "path_tunnel_delta_loss": 0.720266, "path_tunnel_delta_loss_count": 54, "path_tunnel_delta_positive_coverage_at_3": 0.487654, "path_tunnel_delta_positive_total": 162, "path_tunnel_delta_recall_at_3": 0.87037, "path_tunnel_delta_recall_total": 54, "path_tunnel_loss": 0.348827, "path_tunnel_loss_count": 54, "path_tunnel_rescue025_recall_at_3": 1.0, "path_tunnel_rescue025_recall_total": 54, "path_tunnel_rescue050_recall_at_3": 0.981481, "path_tunnel_rescue050_recall_total": 54, "path_tunnel_rescue100_recall_at_3": 0.981481, "path_tunnel_rescue100_recall_total": 54, "path_tunnel_selection_loss": 0.168474, "path_tunnel_selection_loss_count": 54, "path_tunnel_support_positive_coverage_at_3": 0.777778, "path_tunnel_support_positive_total": 162, "path_tunnel_support_recall_at_3": 0.981481, "path_tunnel_support_recall_total": 54, "question_understanding_loss": 3.784985, "question_understanding_loss_count": 54, "recall_event24_positive_total": 113, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 54, "recall_loss": 0.925862, "recall_loss_count": 54, "recall_selection_loss": 0.0, "recall_selection_loss_count": 54, "rows_evaluated": 54, "rows_total": 54, "samples": 54, "supervision_bucket_metrics": {"typed_tunnel_multi_positive": {"answer_plan_current_top1_accuracy": 0.296296, "answer_plan_current_total": 54, "answer_plan_selected_positive_coverage_at_5": 0.778761, "answer_plan_selected_positive_total": 113, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 54, "event5_positive_total": 113, "event_positive_coverage_at_5": 0.964602, "event_recall_at_1": 0.907407, "event_recall_at_5": 1.0, "event_recall_total": 54, "path3_positive_total": 162, "path_positive_coverage_at_3": 0.851852, "path_recall_at_3": 1.0, "path_recall_total": 54, "recall_event24_positive_total": 113, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 54, "samples": 54, "temporal_accuracy": 0.434783, "temporal_total": 23}}, "temporal_accuracy": 0.434783, "temporal_loss": 0.466158, "temporal_loss_count": 23, "temporal_total": 23, "token_role_loss": 0.0, "token_role_loss_count": 7, "training_weight_max": 1.35, "training_weight_mean": 1.35, "training_weight_min": 1.35}, "train": {"answer_plan_current_top1_accuracy": 0.418764, "answer_plan_current_total": 437, "answer_plan_loss": 0.551131, "answer_plan_loss_count": 439, "answer_plan_selected_positive_coverage_at_5": 0.794829, "answer_plan_selected_positive_total": 1199, "answer_plan_selected_recall_at_5": 0.970387, "answer_plan_selected_total": 439, "answer_refusal_loss": 0.0, "answer_refusal_loss_count": 439, "answer_type_loss": 5.186379, "answer_type_loss_count": 439, "answer_type_metrics": {"multi_evidence": {"answer_plan_current_top1_accuracy": 0.428571, "answer_plan_current_total": 420, "answer_plan_selected_positive_coverage_at_5": 0.796217, "answer_plan_selected_positive_total": 1163, "answer_plan_selected_recall_at_5": 0.971564, "answer_plan_selected_total": 422, "event5_positive_total": 1163, "event_positive_coverage_at_5": 0.986242, "event_recall_at_1": 0.845972, "event_recall_at_5": 0.99763, "event_recall_total": 422, "path3_positive_total": 1266, "path_positive_coverage_at_3": 0.888626, "path_recall_at_3": 0.990521, "path_recall_total": 422, "recall_event24_positive_total": 1163, "recall_event_positive_coverage_at_24": 0.993121, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 422, "samples": 422, "temporal_accuracy": 0.0, "temporal_total": 0}, "time": {"answer_plan_current_top1_accuracy": 0.176471, "answer_plan_current_total": 17, "answer_plan_selected_positive_coverage_at_5": 0.75, "answer_plan_selected_positive_total": 36, "answer_plan_selected_recall_at_5": 0.941176, "answer_plan_selected_total": 17, "event5_positive_total": 36, "event_positive_coverage_at_5": 0.944444, "event_recall_at_1": 0.647059, "event_recall_at_5": 1.0, "event_recall_total": 17, "path3_positive_total": 51, "path_positive_coverage_at_3": 0.784314, "path_recall_at_3": 0.941176, "path_recall_total": 17, "recall_event24_positive_total": 36, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 17, "samples": 17, "temporal_accuracy": 0.235294, "temporal_total": 17}}, "evaluation_limited": false, "event5_positive_total": 1199, "event_distractor_loss": 0.671777, "event_distractor_loss_count": 439, "event_hard_negative_loss": 0.090675, "event_hard_negative_loss_count": 439, "event_loss": 0.352816, "event_loss_count": 439, "event_matrix_delta_loss": 0.134491, "event_matrix_delta_loss_count": 439, "event_positive_coverage_at_5": 0.984987, "event_recall_at_1": 0.838269, "event_recall_at_5": 0.997722, "event_recall_total": 439, "event_selection_loss": 0.011378, "event_selection_loss_count": 439, "event_tunnel_loss": 0.235026, "event_tunnel_loss_count": 439, "event_tunnel_selection_loss": 0.0, "event_tunnel_selection_loss_count": 439, "final_event_set_loss": 0.023022, "final_event_set_loss_count": 439, "graph_error_count": 0, "loss": 3.572644, "loss_group_count": 60, "memory_router_exact_match": 0.366743, "memory_router_f1": 0.845193, "memory_router_loss": 1.455285, "memory_router_loss_count": 439, "memory_router_total": 439, "path3_positive_total": 1317, "path_hard_negative_loss": 0.128464, "path_hard_negative_loss_count": 439, "path_loss": 0.392987, "path_loss_count": 439, "path_matrix_delta_loss": 0.489592, "path_matrix_delta_loss_count": 439, "path_positive_coverage_at_3": 0.884586, "path_recall_at_3": 0.98861, "path_recall_total": 439, "path_selection_loss": 0.139808, "path_selection_loss_count": 439, "path_tunnel_delta_loss": 0.629687, "path_tunnel_delta_loss_count": 439, "path_tunnel_delta_positive_coverage_at_3": 0.458618, "path_tunnel_delta_positive_total": 1317, "path_tunnel_delta_recall_at_3": 0.788155, "path_tunnel_delta_recall_total": 439, "path_tunnel_loss": 0.352045, "path_tunnel_loss_count": 439, "path_tunnel_rescue025_recall_at_3": 0.98861, "path_tunnel_rescue025_recall_total": 439, "path_tunnel_rescue050_recall_at_3": 0.981777, "path_tunnel_rescue050_recall_total": 439, "path_tunnel_rescue100_recall_at_3": 0.977221, "path_tunnel_rescue100_recall_total": 439, "path_tunnel_selection_loss": 0.15888, "path_tunnel_selection_loss_count": 439, "path_tunnel_support_positive_coverage_at_3": 0.800304, "path_tunnel_support_positive_total": 1317, "path_tunnel_support_recall_at_3": 0.938497, "path_tunnel_support_recall_total": 439, "question_understanding_loss": 5.304419, "question_understanding_loss_count": 439, "recall_event24_positive_total": 1199, "recall_event_positive_coverage_at_24": 0.993328, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 439, "recall_loss": 0.878647, "recall_loss_count": 439, "recall_selection_loss": 0.0, "recall_selection_loss_count": 439, "rows_evaluated": 439, "rows_total": 439, "samples": 439, "supervision_bucket_metrics": {"typed_tunnel_multi_positive": {"answer_plan_current_top1_accuracy": 0.418764, "answer_plan_current_total": 437, "answer_plan_selected_positive_coverage_at_5": 0.794829, "answer_plan_selected_positive_total": 1199, "answer_plan_selected_recall_at_5": 0.970387, "answer_plan_selected_total": 439, "event5_positive_total": 1199, "event_positive_coverage_at_5": 0.984987, "event_recall_at_1": 0.838269, "event_recall_at_5": 0.997722, "event_recall_total": 439, "path3_positive_total": 1317, "path_positive_coverage_at_3": 0.884586, "path_recall_at_3": 0.98861, "path_recall_total": 439, "recall_event24_positive_total": 1199, "recall_event_positive_coverage_at_24": 0.993328, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 439, "samples": 439, "temporal_accuracy": 0.235294, "temporal_total": 17}}, "temporal_accuracy": 0.235294, "temporal_loss": 0.558015, "temporal_loss_count": 17, "temporal_total": 17, "token_role_loss": 0.0, "token_role_loss_count": 60, "training_weight_max": 1.35, "training_weight_mean": 1.35, "training_weight_min": 1.35}, "val": {"answer_plan_current_top1_accuracy": 0.490909, "answer_plan_current_total": 55, "answer_plan_loss": 0.528866, "answer_plan_loss_count": 55, "answer_plan_selected_positive_coverage_at_5": 0.843284, "answer_plan_selected_positive_total": 134, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 55, "answer_refusal_loss": 0.0, "answer_refusal_loss_count": 55, "answer_type_loss": 5.884438, "answer_type_loss_count": 55, "answer_type_metrics": {"multi_evidence": {"answer_plan_current_top1_accuracy": 0.486486, "answer_plan_current_total": 37, "answer_plan_selected_positive_coverage_at_5": 0.816327, "answer_plan_selected_positive_total": 98, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 37, "event5_positive_total": 98, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 0.918919, "event_recall_at_5": 1.0, "event_recall_total": 37, "path3_positive_total": 111, "path_positive_coverage_at_3": 0.900901, "path_recall_at_3": 1.0, "path_recall_total": 37, "recall_event24_positive_total": 98, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 37, "samples": 37, "temporal_accuracy": 0.0, "temporal_total": 0}, "time": {"answer_plan_current_top1_accuracy": 0.5, "answer_plan_current_total": 18, "answer_plan_selected_positive_coverage_at_5": 0.916666, "answer_plan_selected_positive_total": 36, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 18, "event5_positive_total": 36, "event_positive_coverage_at_5": 0.972222, "event_recall_at_1": 0.833333, "event_recall_at_5": 1.0, "event_recall_total": 18, "path3_positive_total": 54, "path_positive_coverage_at_3": 0.740741, "path_recall_at_3": 1.0, "path_recall_total": 18, "recall_event24_positive_total": 36, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 18, "samples": 18, "temporal_accuracy": 0.388889, "temporal_total": 18}}, "evaluation_limited": false, "event5_positive_total": 134, "event_distractor_loss": 0.642121, "event_distractor_loss_count": 55, "event_hard_negative_loss": 0.068247, "event_hard_negative_loss_count": 55, "event_loss": 0.364887, "event_loss_count": 55, "event_matrix_delta_loss": 0.122631, "event_matrix_delta_loss_count": 55, "event_positive_coverage_at_5": 0.992537, "event_recall_at_1": 0.890909, "event_recall_at_5": 1.0, "event_recall_total": 55, "event_selection_loss": 0.009935, "event_selection_loss_count": 55, "event_tunnel_loss": 0.240878, "event_tunnel_loss_count": 55, "event_tunnel_selection_loss": 0.0, "event_tunnel_selection_loss_count": 55, "final_event_set_loss": 0.014737, "final_event_set_loss_count": 55, "graph_error_count": 0, "loss": 3.474828, "loss_group_count": 7, "memory_router_exact_match": 0.381818, "memory_router_f1": 0.921212, "memory_router_loss": 0.550699, "memory_router_loss_count": 55, "memory_router_total": 55, "path3_positive_total": 165, "path_hard_negative_loss": 0.235719, "path_hard_negative_loss_count": 55, "path_loss": 0.424168, "path_loss_count": 55, "path_matrix_delta_loss": 0.554701, "path_matrix_delta_loss_count": 55, "path_positive_coverage_at_3": 0.848485, "path_recall_at_3": 1.0, "path_recall_total": 55, "path_selection_loss": 0.204065, "path_selection_loss_count": 55, "path_tunnel_delta_loss": 0.663551, "path_tunnel_delta_loss_count": 55, "path_tunnel_delta_positive_coverage_at_3": 0.630303, "path_tunnel_delta_positive_total": 165, "path_tunnel_delta_recall_at_3": 0.854546, "path_tunnel_delta_recall_total": 55, "path_tunnel_loss": 0.364049, "path_tunnel_loss_count": 55, "path_tunnel_rescue025_recall_at_3": 1.0, "path_tunnel_rescue025_recall_total": 55, "path_tunnel_rescue050_recall_at_3": 1.0, "path_tunnel_rescue050_recall_total": 55, "path_tunnel_rescue100_recall_at_3": 1.0, "path_tunnel_rescue100_recall_total": 55, "path_tunnel_selection_loss": 0.172115, "path_tunnel_selection_loss_count": 55, "path_tunnel_support_positive_coverage_at_3": 0.812121, "path_tunnel_support_positive_total": 165, "path_tunnel_support_recall_at_3": 0.981818, "path_tunnel_support_recall_total": 55, "question_understanding_loss": 3.861762, "question_understanding_loss_count": 55, "recall_event24_positive_total": 134, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 55, "recall_loss": 0.859693, "recall_loss_count": 55, "recall_selection_loss": 0.0, "recall_selection_loss_count": 55, "rows_evaluated": 55, "rows_total": 55, "samples": 55, "supervision_bucket_metrics": {"typed_tunnel_multi_positive": {"answer_plan_current_top1_accuracy": 0.490909, "answer_plan_current_total": 55, "answer_plan_selected_positive_coverage_at_5": 0.843284, "answer_plan_selected_positive_total": 134, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 55, "event5_positive_total": 134, "event_positive_coverage_at_5": 0.992537, "event_recall_at_1": 0.890909, "event_recall_at_5": 1.0, "event_recall_total": 55, "path3_positive_total": 165, "path_positive_coverage_at_3": 0.848485, "path_recall_at_3": 1.0, "path_recall_total": 55, "recall_event24_positive_total": 134, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 55, "samples": 55, "temporal_accuracy": 0.388889, "temporal_total": 18}}, "temporal_accuracy": 0.388889, "temporal_loss": 0.502006, "temporal_loss_count": 18, "temporal_total": 18, "token_role_loss": 0.0, "token_role_loss_count": 7, "training_weight_max": 1.35, "training_weight_mean": 1.35, "training_weight_min": 1.35}}, "model_arch_version": "node_memory_v11_tmcra_scale49m_trimaze", "node_encoder_variant": "hashed_tmcra_token_attn_pool_6l_w256", "node_max_tokens": 32, "pair_feature_adapter_variant": "residual_pair_feature_adapter_v1", "question_encoder_variant": "hashed_tmcra_token_attn_pool_6l_w256", "question_intent_variant": "text_pooled_multihead_v1", "question_max_tokens": 48, "resume": {"resume_checkpoint": "/models/tmcra_node_memory_v11_priorfusion_zeroload_20260517/node_scorer.pt", "resume_mode": "weights_only_forced", "resume_weights_only": true}, "resume_mode": "weights_only_forced", "status": "completed", "trainable_stage": {"frozen_parameter_count": 47963050, "trainable_name_prefixes": ["event_tunnel_head", "final_event_fusion_head", "final_path_fusion_head", "path_tunnel_head"], "trainable_parameter_count": 3831302, "trainable_stage": "tunnel_fusion_only"}, "training_sampling": {"answer_type_counts": {"multi_evidence": 422, "time": 17}, "base_row_count": 439, "effective_row_count": 697, "loss_weight_range_estimate": {"max": 1.549193, "min": 1.319091}, "mode": "source_aware_balanced", "repeat_histogram": {"1": 118, "2": 144}, "source_count": 1, "source_details": {"dataset": {"avg_example_weight_estimate": 1.540283, "loss_source_factor": 1.0, "row_count": 439, "row_share": 1.0, "sampled_row_count_estimate": 697, "sampled_row_share_estimate": 1.0, "sampling_source_factor": 1.0, "weighted_loss_mass": 676.184129}}, "source_row_counts": {"dataset": 439}, "supervision_bucket_counts": {"typed_tunnel_multi_positive": 439}, "weighted_answer_mass_estimate": {"multi_evidence": 653.759589, "time": 22.42454}, "weighted_supervision_bucket_mass_estimate": {"typed_tunnel_multi_positive": 680.095876}}}
\ No newline at end of file
diff --git a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/train_summary.partial.json b/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/train_summary.partial.json
deleted file mode 100644
index a300a64142f9906721e073f51d9dff4804633cc9..0000000000000000000000000000000000000000
--- a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/train_summary.partial.json
+++ /dev/null
@@ -1 +0,0 @@
-{"answer_calibration_variant": "competition_alignment_v3_trimaze", "answer_plan_variant": "candidate_event_answer_plan_v1", "artifacts": {"checkpoints_dir": "outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/checkpoints", "node_model_best_path": "outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer_best.pt", "node_model_last_path": "outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer_last.pt", "path_model_best_path": "outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer_best.pt", "path_model_last_path": "outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer_last.pt"}, "best_val_loss": 3.718748, "best_val_selection_score": 118.424107, "config": {"amp": true, "answer_calibration_variant": "competition_alignment_v3_trimaze", "answer_plan_current_old_margin": 0.2, "answer_plan_current_old_margin_loss_weight": 0.0, "answer_plan_loss_weight": 0.18, "answer_plan_selected_negative_margin": 0.3, "answer_plan_selected_negative_margin_loss_weight": 0.0, "answer_plan_selection_margin": 0.12, "answer_plan_selection_top_k": 5, "answer_plan_variant": "candidate_event_answer_plan_v1", "answer_refusal_loss_weight": 0.08, "answer_refusal_margin": 0.15, "answer_type_loss_weight": 0.2, "batch_prepare_lookahead_batches": 64, "batch_prepare_workers": 16, "batch_size": 8, "checkpoint_every_steps": 50, "early_stopping_patience": 3, "encoder_attention_heads": 8, "encoder_attention_layers": 6, "encoder_structural_bias_variant": "tmcra_token_role_relbias_v1", "epoch_train_eval_enabled": true, "epoch_train_eval_max_rows": 80, "epoch_val_eval_enabled": true, "epoch_val_eval_max_rows": 32, "epochs": 8, "event_distractor_loss_weight": 0.15, "event_distractor_variant": "candidate_distractor_delta_bce_v1", "event_hard_negative_loss_weight": 0.35, "event_hard_negative_margin": 0.35, "event_loss_weight": 1.1, "event_matrix_delta_loss_weight": 0.08, "event_matrix_delta_margin": 0.15, "event_selection_loss_weight": 0.35, "event_selection_margin": 0.2, "event_selection_positive_coverage_count": 3, "event_selection_top_k": 5, "event_subgraph_attention_heads": 8, "event_subgraph_attention_layers": 6, "event_subgraph_refiner_variant": "event_centered_subgraph_attn_6l_h448", "event_tunnel_loss_weight": 0.12, "event_tunnel_margin": 0.16, "event_tunnel_selection_loss_weight": 0.08, "final_event_set_loss_weight": 0.35, "final_event_set_margin": 0.08, "final_event_set_positive_coverage_count": 3, "final_event_set_support_path_k": 3, "final_event_set_top_k": 10, "grad_clip": 1.0, "graph_prefetch_lookahead_batches": 3, "l2sp_loss_weight": 0.02, "log_every_steps": 20, "loss_blend_uniform_ratio": 0.25, "loss_group_balancing_mode": "supervision_bucket", "loss_max_example_weight": 1.85, "loss_min_example_weight": 0.65, "loss_multi_evidence_boost": 2.0, "loss_source_alpha": 0.4, "loss_temporal_positive_boost": 1.2, "loss_time_boost": 1.45, "loss_weight_power": 0.5, "lr": 8e-05, "max_train_steps": 0, "memory_router_loss_weight": 0.18, "memory_router_variant": "question_pooled_multilabel_v1", "memory_tunnel_variant": "chain_depth_tunnel_support_delta_v1", "message_passing_variant": "support_to_event_bidirectional_shared_typed_v1", "model_arch_version": "node_memory_v11_tmcra_scale49m_trimaze", "multi_positive_coverage_fraction": 0.7, "multi_positive_event_coverage_count": 5, "multi_positive_final_event_set_coverage_count": 5, "multi_positive_path_coverage_count": 3, "multi_positive_recall_coverage_count": 6, "node_encoder_variant": "hashed_tmcra_token_attn_pool_6l_w256", "node_max_tokens": 32, "online_event_hard_negative_limit": 6, "online_path_hard_negative_limit": 8, "pair_feature_adapter_variant": "residual_pair_feature_adapter_v1", "path_hard_negative_loss_weight": 0.25, "path_hard_negative_margin": 0.28, "path_loss_weight": 0.55, "path_matrix_delta_loss_weight": 0.08, "path_matrix_delta_margin": 0.12, "path_selection_loss_weight": 0.3, "path_selection_margin": 0.15, "path_selection_positive_coverage_count": 3, "path_selection_top_k": 3, "path_tunnel_delta_loss_weight": 0.12, "path_tunnel_loss_weight": 0.06, "path_tunnel_margin": 0.16, "path_tunnel_selection_loss_weight": 0.04, "question_encoder_variant": "hashed_tmcra_token_attn_pool_6l_w256", "question_intent_variant": "text_pooled_multihead_v1", "question_max_tokens": 48, "question_understanding_loss_weight": 0.12, "recall_loss_weight": 0.6, "recall_selection_loss_weight": 0.3, "recall_selection_margin": 0.12, "recall_selection_positive_coverage_count": 3, "recall_selection_top_k": 24, "sampling_blend_uniform_ratio": 0.35, "sampling_max_conversation_multiplier": 2.0, "sampling_max_group_repeat": 2, "sampling_multi_evidence_boost": 1.45, "sampling_source_alpha": 0.35, "sampling_temporal_positive_boost": 1.15, "sampling_time_boost": 1.2, "temporal_loss_weight": 0.3, "token_role_loss_weight": 0.08, "train_sampling_mode": "source_aware_balanced", "trainable_stage": "tunnel_fusion_only", "warmup_ratio": 0.06, "weight_decay": 0.01}, "device": "cuda", "encoder_attention_heads": 8, "encoder_attention_layers": 6, "encoder_structural_bias_variant": "tmcra_token_role_relbias_v1", "epoch": 7, "event_distractor_variant": "candidate_distractor_delta_bce_v1", "event_subgraph_attention_heads": 8, "event_subgraph_attention_layers": 6, "event_subgraph_refiner_variant": "event_centered_subgraph_attn_6l_h448", "final_eval_progress": {"completed_splits": ["train", "val", "test"], "test_metrics": {"answer_plan_current_top1_accuracy": 0.296296, "answer_plan_current_total": 54, "answer_plan_loss": 0.519994, "answer_plan_loss_count": 54, "answer_plan_selected_positive_coverage_at_5": 0.778761, "answer_plan_selected_positive_total": 113, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 54, "answer_refusal_loss": 0.0, "answer_refusal_loss_count": 54, "answer_type_loss": 3.65495, "answer_type_loss_count": 54, "answer_type_metrics": {"multi_evidence": {"answer_plan_current_top1_accuracy": 0.129032, "answer_plan_current_total": 31, "answer_plan_selected_positive_coverage_at_5": 0.776119, "answer_plan_selected_positive_total": 67, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 31, "event5_positive_total": 67, "event_positive_coverage_at_5": 0.940298, "event_recall_at_1": 0.903226, "event_recall_at_5": 1.0, "event_recall_total": 31, "path3_positive_total": 93, "path_positive_coverage_at_3": 0.849462, "path_recall_at_3": 1.0, "path_recall_total": 31, "recall_event24_positive_total": 67, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 31, "samples": 31, "temporal_accuracy": 0.0, "temporal_total": 0}, "time": {"answer_plan_current_top1_accuracy": 0.521739, "answer_plan_current_total": 23, "answer_plan_selected_positive_coverage_at_5": 0.782609, "answer_plan_selected_positive_total": 46, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 23, "event5_positive_total": 46, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 0.913044, "event_recall_at_5": 1.0, "event_recall_total": 23, "path3_positive_total": 69, "path_positive_coverage_at_3": 0.855072, "path_recall_at_3": 1.0, "path_recall_total": 23, "recall_event24_positive_total": 46, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 23, "samples": 23, "temporal_accuracy": 0.434783, "temporal_total": 23}}, "evaluation_limited": false, "event5_positive_total": 113, "event_distractor_loss": 0.652414, "event_distractor_loss_count": 54, "event_hard_negative_loss": 0.095313, "event_hard_negative_loss_count": 54, "event_loss": 0.388887, "event_loss_count": 54, "event_matrix_delta_loss": 0.121482, "event_matrix_delta_loss_count": 54, "event_positive_coverage_at_5": 0.964602, "event_recall_at_1": 0.907407, "event_recall_at_5": 1.0, "event_recall_total": 54, "event_selection_loss": 0.033558, "event_selection_loss_count": 54, "event_tunnel_loss": 0.257923, "event_tunnel_loss_count": 54, "event_tunnel_selection_loss": 0.0, "event_tunnel_selection_loss_count": 54, "final_event_set_loss": 0.006052, "final_event_set_loss_count": 54, "graph_error_count": 0, "loss": 3.097328, "loss_group_count": 7, "memory_router_exact_match": 0.851852, "memory_router_f1": 0.962963, "memory_router_loss": 0.407471, "memory_router_loss_count": 54, "memory_router_total": 54, "path3_positive_total": 162, "path_hard_negative_loss": 0.211284, "path_hard_negative_loss_count": 54, "path_loss": 0.429968, "path_loss_count": 54, "path_matrix_delta_loss": 0.494685, "path_matrix_delta_loss_count": 54, "path_positive_coverage_at_3": 0.851852, "path_recall_at_3": 1.0, "path_recall_total": 54, "path_selection_loss": 0.245353, "path_selection_loss_count": 54, "path_tunnel_delta_loss": 0.720266, "path_tunnel_delta_loss_count": 54, "path_tunnel_delta_positive_coverage_at_3": 0.487654, "path_tunnel_delta_positive_total": 162, "path_tunnel_delta_recall_at_3": 0.87037, "path_tunnel_delta_recall_total": 54, "path_tunnel_loss": 0.348827, "path_tunnel_loss_count": 54, "path_tunnel_rescue025_recall_at_3": 1.0, "path_tunnel_rescue025_recall_total": 54, "path_tunnel_rescue050_recall_at_3": 0.981481, "path_tunnel_rescue050_recall_total": 54, "path_tunnel_rescue100_recall_at_3": 0.981481, "path_tunnel_rescue100_recall_total": 54, "path_tunnel_selection_loss": 0.168474, "path_tunnel_selection_loss_count": 54, "path_tunnel_support_positive_coverage_at_3": 0.777778, "path_tunnel_support_positive_total": 162, "path_tunnel_support_recall_at_3": 0.981481, "path_tunnel_support_recall_total": 54, "question_understanding_loss": 3.784985, "question_understanding_loss_count": 54, "recall_event24_positive_total": 113, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 54, "recall_loss": 0.925862, "recall_loss_count": 54, "recall_selection_loss": 0.0, "recall_selection_loss_count": 54, "rows_evaluated": 54, "rows_total": 54, "samples": 54, "supervision_bucket_metrics": {"typed_tunnel_multi_positive": {"answer_plan_current_top1_accuracy": 0.296296, "answer_plan_current_total": 54, "answer_plan_selected_positive_coverage_at_5": 0.778761, "answer_plan_selected_positive_total": 113, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 54, "event5_positive_total": 113, "event_positive_coverage_at_5": 0.964602, "event_recall_at_1": 0.907407, "event_recall_at_5": 1.0, "event_recall_total": 54, "path3_positive_total": 162, "path_positive_coverage_at_3": 0.851852, "path_recall_at_3": 1.0, "path_recall_total": 54, "recall_event24_positive_total": 113, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 54, "samples": 54, "temporal_accuracy": 0.434783, "temporal_total": 23}}, "temporal_accuracy": 0.434783, "temporal_loss": 0.466158, "temporal_loss_count": 23, "temporal_total": 23, "token_role_loss": 0.0, "token_role_loss_count": 7, "training_weight_max": 1.35, "training_weight_mean": 1.35, "training_weight_min": 1.35}, "train_metrics": {"answer_plan_current_top1_accuracy": 0.418764, "answer_plan_current_total": 437, "answer_plan_loss": 0.551131, "answer_plan_loss_count": 439, "answer_plan_selected_positive_coverage_at_5": 0.794829, "answer_plan_selected_positive_total": 1199, "answer_plan_selected_recall_at_5": 0.970387, "answer_plan_selected_total": 439, "answer_refusal_loss": 0.0, "answer_refusal_loss_count": 439, "answer_type_loss": 5.186379, "answer_type_loss_count": 439, "answer_type_metrics": {"multi_evidence": {"answer_plan_current_top1_accuracy": 0.428571, "answer_plan_current_total": 420, "answer_plan_selected_positive_coverage_at_5": 0.796217, "answer_plan_selected_positive_total": 1163, "answer_plan_selected_recall_at_5": 0.971564, "answer_plan_selected_total": 422, "event5_positive_total": 1163, "event_positive_coverage_at_5": 0.986242, "event_recall_at_1": 0.845972, "event_recall_at_5": 0.99763, "event_recall_total": 422, "path3_positive_total": 1266, "path_positive_coverage_at_3": 0.888626, "path_recall_at_3": 0.990521, "path_recall_total": 422, "recall_event24_positive_total": 1163, "recall_event_positive_coverage_at_24": 0.993121, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 422, "samples": 422, "temporal_accuracy": 0.0, "temporal_total": 0}, "time": {"answer_plan_current_top1_accuracy": 0.176471, "answer_plan_current_total": 17, "answer_plan_selected_positive_coverage_at_5": 0.75, "answer_plan_selected_positive_total": 36, "answer_plan_selected_recall_at_5": 0.941176, "answer_plan_selected_total": 17, "event5_positive_total": 36, "event_positive_coverage_at_5": 0.944444, "event_recall_at_1": 0.647059, "event_recall_at_5": 1.0, "event_recall_total": 17, "path3_positive_total": 51, "path_positive_coverage_at_3": 0.784314, "path_recall_at_3": 0.941176, "path_recall_total": 17, "recall_event24_positive_total": 36, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 17, "samples": 17, "temporal_accuracy": 0.235294, "temporal_total": 17}}, "evaluation_limited": false, "event5_positive_total": 1199, "event_distractor_loss": 0.671777, "event_distractor_loss_count": 439, "event_hard_negative_loss": 0.090675, "event_hard_negative_loss_count": 439, "event_loss": 0.352816, "event_loss_count": 439, "event_matrix_delta_loss": 0.134491, "event_matrix_delta_loss_count": 439, "event_positive_coverage_at_5": 0.984987, "event_recall_at_1": 0.838269, "event_recall_at_5": 0.997722, "event_recall_total": 439, "event_selection_loss": 0.011378, "event_selection_loss_count": 439, "event_tunnel_loss": 0.235026, "event_tunnel_loss_count": 439, "event_tunnel_selection_loss": 0.0, "event_tunnel_selection_loss_count": 439, "final_event_set_loss": 0.023022, "final_event_set_loss_count": 439, "graph_error_count": 0, "loss": 3.572644, "loss_group_count": 60, "memory_router_exact_match": 0.366743, "memory_router_f1": 0.845193, "memory_router_loss": 1.455285, "memory_router_loss_count": 439, "memory_router_total": 439, "path3_positive_total": 1317, "path_hard_negative_loss": 0.128464, "path_hard_negative_loss_count": 439, "path_loss": 0.392987, "path_loss_count": 439, "path_matrix_delta_loss": 0.489592, "path_matrix_delta_loss_count": 439, "path_positive_coverage_at_3": 0.884586, "path_recall_at_3": 0.98861, "path_recall_total": 439, "path_selection_loss": 0.139808, "path_selection_loss_count": 439, "path_tunnel_delta_loss": 0.629687, "path_tunnel_delta_loss_count": 439, "path_tunnel_delta_positive_coverage_at_3": 0.458618, "path_tunnel_delta_positive_total": 1317, "path_tunnel_delta_recall_at_3": 0.788155, "path_tunnel_delta_recall_total": 439, "path_tunnel_loss": 0.352045, "path_tunnel_loss_count": 439, "path_tunnel_rescue025_recall_at_3": 0.98861, "path_tunnel_rescue025_recall_total": 439, "path_tunnel_rescue050_recall_at_3": 0.981777, "path_tunnel_rescue050_recall_total": 439, "path_tunnel_rescue100_recall_at_3": 0.977221, "path_tunnel_rescue100_recall_total": 439, "path_tunnel_selection_loss": 0.15888, "path_tunnel_selection_loss_count": 439, "path_tunnel_support_positive_coverage_at_3": 0.800304, "path_tunnel_support_positive_total": 1317, "path_tunnel_support_recall_at_3": 0.938497, "path_tunnel_support_recall_total": 439, "question_understanding_loss": 5.304419, "question_understanding_loss_count": 439, "recall_event24_positive_total": 1199, "recall_event_positive_coverage_at_24": 0.993328, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 439, "recall_loss": 0.878647, "recall_loss_count": 439, "recall_selection_loss": 0.0, "recall_selection_loss_count": 439, "rows_evaluated": 439, "rows_total": 439, "samples": 439, "supervision_bucket_metrics": {"typed_tunnel_multi_positive": {"answer_plan_current_top1_accuracy": 0.418764, "answer_plan_current_total": 437, "answer_plan_selected_positive_coverage_at_5": 0.794829, "answer_plan_selected_positive_total": 1199, "answer_plan_selected_recall_at_5": 0.970387, "answer_plan_selected_total": 439, "event5_positive_total": 1199, "event_positive_coverage_at_5": 0.984987, "event_recall_at_1": 0.838269, "event_recall_at_5": 0.997722, "event_recall_total": 439, "path3_positive_total": 1317, "path_positive_coverage_at_3": 0.884586, "path_recall_at_3": 0.98861, "path_recall_total": 439, "recall_event24_positive_total": 1199, "recall_event_positive_coverage_at_24": 0.993328, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 439, "samples": 439, "temporal_accuracy": 0.235294, "temporal_total": 17}}, "temporal_accuracy": 0.235294, "temporal_loss": 0.558015, "temporal_loss_count": 17, "temporal_total": 17, "token_role_loss": 0.0, "token_role_loss_count": 60, "training_weight_max": 1.35, "training_weight_mean": 1.35, "training_weight_min": 1.35}, "val_metrics": {"answer_plan_current_top1_accuracy": 0.490909, "answer_plan_current_total": 55, "answer_plan_loss": 0.528866, "answer_plan_loss_count": 55, "answer_plan_selected_positive_coverage_at_5": 0.843284, "answer_plan_selected_positive_total": 134, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 55, "answer_refusal_loss": 0.0, "answer_refusal_loss_count": 55, "answer_type_loss": 5.884438, "answer_type_loss_count": 55, "answer_type_metrics": {"multi_evidence": {"answer_plan_current_top1_accuracy": 0.486486, "answer_plan_current_total": 37, "answer_plan_selected_positive_coverage_at_5": 0.816327, "answer_plan_selected_positive_total": 98, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 37, "event5_positive_total": 98, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 0.918919, "event_recall_at_5": 1.0, "event_recall_total": 37, "path3_positive_total": 111, "path_positive_coverage_at_3": 0.900901, "path_recall_at_3": 1.0, "path_recall_total": 37, "recall_event24_positive_total": 98, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 37, "samples": 37, "temporal_accuracy": 0.0, "temporal_total": 0}, "time": {"answer_plan_current_top1_accuracy": 0.5, "answer_plan_current_total": 18, "answer_plan_selected_positive_coverage_at_5": 0.916666, "answer_plan_selected_positive_total": 36, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 18, "event5_positive_total": 36, "event_positive_coverage_at_5": 0.972222, "event_recall_at_1": 0.833333, "event_recall_at_5": 1.0, "event_recall_total": 18, "path3_positive_total": 54, "path_positive_coverage_at_3": 0.740741, "path_recall_at_3": 1.0, "path_recall_total": 18, "recall_event24_positive_total": 36, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 18, "samples": 18, "temporal_accuracy": 0.388889, "temporal_total": 18}}, "evaluation_limited": false, "event5_positive_total": 134, "event_distractor_loss": 0.642121, "event_distractor_loss_count": 55, "event_hard_negative_loss": 0.068247, "event_hard_negative_loss_count": 55, "event_loss": 0.364887, "event_loss_count": 55, "event_matrix_delta_loss": 0.122631, "event_matrix_delta_loss_count": 55, "event_positive_coverage_at_5": 0.992537, "event_recall_at_1": 0.890909, "event_recall_at_5": 1.0, "event_recall_total": 55, "event_selection_loss": 0.009935, "event_selection_loss_count": 55, "event_tunnel_loss": 0.240878, "event_tunnel_loss_count": 55, "event_tunnel_selection_loss": 0.0, "event_tunnel_selection_loss_count": 55, "final_event_set_loss": 0.014737, "final_event_set_loss_count": 55, "graph_error_count": 0, "loss": 3.474828, "loss_group_count": 7, "memory_router_exact_match": 0.381818, "memory_router_f1": 0.921212, "memory_router_loss": 0.550699, "memory_router_loss_count": 55, "memory_router_total": 55, "path3_positive_total": 165, "path_hard_negative_loss": 0.235719, "path_hard_negative_loss_count": 55, "path_loss": 0.424168, "path_loss_count": 55, "path_matrix_delta_loss": 0.554701, "path_matrix_delta_loss_count": 55, "path_positive_coverage_at_3": 0.848485, "path_recall_at_3": 1.0, "path_recall_total": 55, "path_selection_loss": 0.204065, "path_selection_loss_count": 55, "path_tunnel_delta_loss": 0.663551, "path_tunnel_delta_loss_count": 55, "path_tunnel_delta_positive_coverage_at_3": 0.630303, "path_tunnel_delta_positive_total": 165, "path_tunnel_delta_recall_at_3": 0.854546, "path_tunnel_delta_recall_total": 55, "path_tunnel_loss": 0.364049, "path_tunnel_loss_count": 55, "path_tunnel_rescue025_recall_at_3": 1.0, "path_tunnel_rescue025_recall_total": 55, "path_tunnel_rescue050_recall_at_3": 1.0, "path_tunnel_rescue050_recall_total": 55, "path_tunnel_rescue100_recall_at_3": 1.0, "path_tunnel_rescue100_recall_total": 55, "path_tunnel_selection_loss": 0.172115, "path_tunnel_selection_loss_count": 55, "path_tunnel_support_positive_coverage_at_3": 0.812121, "path_tunnel_support_positive_total": 165, "path_tunnel_support_recall_at_3": 0.981818, "path_tunnel_support_recall_total": 55, "question_understanding_loss": 3.861762, "question_understanding_loss_count": 55, "recall_event24_positive_total": 134, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 55, "recall_loss": 0.859693, "recall_loss_count": 55, "recall_selection_loss": 0.0, "recall_selection_loss_count": 55, "rows_evaluated": 55, "rows_total": 55, "samples": 55, "supervision_bucket_metrics": {"typed_tunnel_multi_positive": {"answer_plan_current_top1_accuracy": 0.490909, "answer_plan_current_total": 55, "answer_plan_selected_positive_coverage_at_5": 0.843284, "answer_plan_selected_positive_total": 134, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 55, "event5_positive_total": 134, "event_positive_coverage_at_5": 0.992537, "event_recall_at_1": 0.890909, "event_recall_at_5": 1.0, "event_recall_total": 55, "path3_positive_total": 165, "path_positive_coverage_at_3": 0.848485, "path_recall_at_3": 1.0, "path_recall_total": 55, "recall_event24_positive_total": 134, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 55, "samples": 55, "temporal_accuracy": 0.388889, "temporal_total": 18}}, "temporal_accuracy": 0.388889, "temporal_loss": 0.502006, "temporal_loss_count": 18, "temporal_total": 18, "token_role_loss": 0.0, "token_role_loss_count": 7, "training_weight_max": 1.35, "training_weight_mean": 1.35, "training_weight_min": 1.35}}, "global_step": 665, "issue_counts": {"batch_prepare_pool_event": 1}, "issue_log": "outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/training_issues.jsonl", "last_epoch": {"best_val_loss": 3.718748, "best_val_selection_key": [1.0, 1.0, 1.0, 0.428571, -3.718748], "best_val_selection_score": 118.424107, "epoch": 7, "global_step": 665, "is_best": false, "patience": 3, "total_steps": 704, "train": {"answer_plan_current_top1_accuracy": 0.379747, "answer_plan_current_total": 79, "answer_plan_loss": 0.549526, "answer_plan_loss_count": 80, "answer_plan_selected_positive_coverage_at_5": 0.821101, "answer_plan_selected_positive_total": 218, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 80, "answer_refusal_loss": 0.0, "answer_refusal_loss_count": 80, "answer_type_loss": 5.16927, "answer_type_loss_count": 80, "answer_type_metrics": {"multi_evidence": {"answer_plan_current_top1_accuracy": 0.376623, "answer_plan_current_total": 77, "answer_plan_selected_positive_coverage_at_5": 0.821596, "answer_plan_selected_positive_total": 213, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 78, "event5_positive_total": 213, "event_positive_coverage_at_5": 0.981221, "event_recall_at_1": 0.846154, "event_recall_at_5": 1.0, "event_recall_total": 78, "path3_positive_total": 234, "path_positive_coverage_at_3": 0.910256, "path_recall_at_3": 1.0, "path_recall_total": 78, "recall_event24_positive_total": 213, "recall_event_positive_coverage_at_24": 0.995305, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 78, "samples": 78, "temporal_accuracy": 0.0, "temporal_total": 0}, "time": {"answer_plan_current_top1_accuracy": 0.5, "answer_plan_current_total": 2, "answer_plan_selected_positive_coverage_at_5": 0.8, "answer_plan_selected_positive_total": 5, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 2, "event5_positive_total": 5, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 1.0, "event_recall_at_5": 1.0, "event_recall_total": 2, "path3_positive_total": 6, "path_positive_coverage_at_3": 1.0, "path_recall_at_3": 1.0, "path_recall_total": 2, "recall_event24_positive_total": 5, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 2, "samples": 2, "temporal_accuracy": 0.0, "temporal_total": 2}}, "evaluation_limited": true, "event5_positive_total": 218, "event_distractor_loss": 0.666098, "event_distractor_loss_count": 80, "event_hard_negative_loss": 0.078884, "event_hard_negative_loss_count": 80, "event_loss": 0.323909, "event_loss_count": 80, "event_matrix_delta_loss": 0.128518, "event_matrix_delta_loss_count": 80, "event_positive_coverage_at_5": 0.981651, "event_recall_at_1": 0.85, "event_recall_at_5": 1.0, "event_recall_total": 80, "event_selection_loss": 0.014046, "event_selection_loss_count": 80, "event_tunnel_loss": 0.213497, "event_tunnel_loss_count": 80, "event_tunnel_selection_loss": 0.0, "event_tunnel_selection_loss_count": 80, "final_event_set_loss": 0.013138, "final_event_set_loss_count": 80, "graph_error_count": 0, "loss": 3.466264, "loss_group_count": 11, "memory_router_exact_match": 0.375, "memory_router_f1": 0.845089, "memory_router_loss": 1.479962, "memory_router_loss_count": 80, "memory_router_total": 80, "path3_positive_total": 240, "path_hard_negative_loss": 0.117886, "path_hard_negative_loss_count": 80, "path_loss": 0.369553, "path_loss_count": 80, "path_matrix_delta_loss": 0.454712, "path_matrix_delta_loss_count": 80, "path_positive_coverage_at_3": 0.9125, "path_recall_at_3": 1.0, "path_recall_total": 80, "path_selection_loss": 0.12883, "path_selection_loss_count": 80, "path_tunnel_delta_loss": 0.60523, "path_tunnel_delta_loss_count": 80, "path_tunnel_delta_positive_coverage_at_3": 0.479167, "path_tunnel_delta_positive_total": 240, "path_tunnel_delta_recall_at_3": 0.75, "path_tunnel_delta_recall_total": 80, "path_tunnel_loss": 0.338907, "path_tunnel_loss_count": 80, "path_tunnel_rescue025_recall_at_3": 1.0, "path_tunnel_rescue025_recall_total": 80, "path_tunnel_rescue050_recall_at_3": 0.9875, "path_tunnel_rescue050_recall_total": 80, "path_tunnel_rescue100_recall_at_3": 0.9875, "path_tunnel_rescue100_recall_total": 80, "path_tunnel_selection_loss": 0.14157, "path_tunnel_selection_loss_count": 80, "path_tunnel_support_positive_coverage_at_3": 0.829167, "path_tunnel_support_positive_total": 240, "path_tunnel_support_recall_at_3": 0.9375, "path_tunnel_support_recall_total": 80, "question_understanding_loss": 5.293586, "question_understanding_loss_count": 80, "recall_event24_positive_total": 218, "recall_event_positive_coverage_at_24": 0.995413, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 80, "recall_loss": 0.820672, "recall_loss_count": 80, "recall_selection_loss": 0.0, "recall_selection_loss_count": 80, "rows_evaluated": 80, "rows_total": 439, "samples": 80, "supervision_bucket_metrics": {"typed_tunnel_multi_positive": {"answer_plan_current_top1_accuracy": 0.379747, "answer_plan_current_total": 79, "answer_plan_selected_positive_coverage_at_5": 0.821101, "answer_plan_selected_positive_total": 218, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 80, "event5_positive_total": 218, "event_positive_coverage_at_5": 0.981651, "event_recall_at_1": 0.85, "event_recall_at_5": 1.0, "event_recall_total": 80, "path3_positive_total": 240, "path_positive_coverage_at_3": 0.9125, "path_recall_at_3": 1.0, "path_recall_total": 80, "recall_event24_positive_total": 218, "recall_event_positive_coverage_at_24": 0.995413, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 80, "samples": 80, "temporal_accuracy": 0.0, "temporal_total": 2}}, "temporal_accuracy": 0.0, "temporal_loss": 0.513602, "temporal_loss_count": 2, "temporal_total": 2, "token_role_loss": 0.0, "token_role_loss_count": 11, "training_weight_max": 1.35, "training_weight_mean": 1.35, "training_weight_min": 1.35}, "val": {"answer_plan_current_top1_accuracy": 0.53125, "answer_plan_current_total": 32, "answer_plan_loss": 0.530247, "answer_plan_loss_count": 32, "answer_plan_selected_positive_coverage_at_5": 0.848101, "answer_plan_selected_positive_total": 79, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 32, "answer_refusal_loss": 0.0, "answer_refusal_loss_count": 32, "answer_type_loss": 6.441788, "answer_type_loss_count": 32, "answer_type_metrics": {"multi_evidence": {"answer_plan_current_top1_accuracy": 0.545455, "answer_plan_current_total": 22, "answer_plan_selected_positive_coverage_at_5": 0.847458, "answer_plan_selected_positive_total": 59, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 22, "event5_positive_total": 59, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 0.909091, "event_recall_at_5": 1.0, "event_recall_total": 22, "path3_positive_total": 66, "path_positive_coverage_at_3": 0.924242, "path_recall_at_3": 1.0, "path_recall_total": 22, "recall_event24_positive_total": 59, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 22, "samples": 22, "temporal_accuracy": 0.0, "temporal_total": 0}, "time": {"answer_plan_current_top1_accuracy": 0.5, "answer_plan_current_total": 10, "answer_plan_selected_positive_coverage_at_5": 0.85, "answer_plan_selected_positive_total": 20, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 10, "event5_positive_total": 20, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 0.7, "event_recall_at_5": 1.0, "event_recall_total": 10, "path3_positive_total": 30, "path_positive_coverage_at_3": 0.733334, "path_recall_at_3": 1.0, "path_recall_total": 10, "recall_event24_positive_total": 20, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 10, "samples": 10, "temporal_accuracy": 0.3, "temporal_total": 10}}, "evaluation_limited": true, "event5_positive_total": 79, "event_distractor_loss": 0.646066, "event_distractor_loss_count": 32, "event_hard_negative_loss": 0.062803, "event_hard_negative_loss_count": 32, "event_loss": 0.379233, "event_loss_count": 32, "event_matrix_delta_loss": 0.125441, "event_matrix_delta_loss_count": 32, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 0.84375, "event_recall_at_5": 1.0, "event_recall_total": 32, "event_selection_loss": 0.001734, "event_selection_loss_count": 32, "event_tunnel_loss": 0.248372, "event_tunnel_loss_count": 32, "event_tunnel_selection_loss": 0.0, "event_tunnel_selection_loss_count": 32, "final_event_set_loss": 0.012893, "final_event_set_loss_count": 32, "graph_error_count": 0, "loss": 3.604544, "loss_group_count": 4, "memory_router_exact_match": 0.34375, "memory_router_f1": 0.918403, "memory_router_loss": 0.575108, "memory_router_loss_count": 32, "memory_router_total": 32, "path3_positive_total": 96, "path_hard_negative_loss": 0.23831, "path_hard_negative_loss_count": 32, "path_loss": 0.422076, "path_loss_count": 32, "path_matrix_delta_loss": 0.564214, "path_matrix_delta_loss_count": 32, "path_positive_coverage_at_3": 0.864584, "path_recall_at_3": 1.0, "path_recall_total": 32, "path_selection_loss": 0.189845, "path_selection_loss_count": 32, "path_tunnel_delta_loss": 0.658501, "path_tunnel_delta_loss_count": 32, "path_tunnel_delta_positive_coverage_at_3": 0.614584, "path_tunnel_delta_positive_total": 96, "path_tunnel_delta_recall_at_3": 0.84375, "path_tunnel_delta_recall_total": 32, "path_tunnel_loss": 0.360756, "path_tunnel_loss_count": 32, "path_tunnel_rescue025_recall_at_3": 1.0, "path_tunnel_rescue025_recall_total": 32, "path_tunnel_rescue050_recall_at_3": 1.0, "path_tunnel_rescue050_recall_total": 32, "path_tunnel_rescue100_recall_at_3": 1.0, "path_tunnel_rescue100_recall_total": 32, "path_tunnel_selection_loss": 0.183911, "path_tunnel_selection_loss_count": 32, "path_tunnel_support_positive_coverage_at_3": 0.802083, "path_tunnel_support_positive_total": 96, "path_tunnel_support_recall_at_3": 0.96875, "path_tunnel_support_recall_total": 32, "question_understanding_loss": 3.751869, "question_understanding_loss_count": 32, "recall_event24_positive_total": 79, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 32, "recall_loss": 0.894676, "recall_loss_count": 32, "recall_selection_loss": 0.0, "recall_selection_loss_count": 32, "rows_evaluated": 32, "rows_total": 55, "samples": 32, "supervision_bucket_metrics": {"typed_tunnel_multi_positive": {"answer_plan_current_top1_accuracy": 0.53125, "answer_plan_current_total": 32, "answer_plan_selected_positive_coverage_at_5": 0.848101, "answer_plan_selected_positive_total": 79, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 32, "event5_positive_total": 79, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 0.84375, "event_recall_at_5": 1.0, "event_recall_total": 32, "path3_positive_total": 96, "path_positive_coverage_at_3": 0.864584, "path_recall_at_3": 1.0, "path_recall_total": 32, "recall_event24_positive_total": 79, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 32, "samples": 32, "temporal_accuracy": 0.3, "temporal_total": 10}}, "temporal_accuracy": 0.3, "temporal_loss": 0.50511, "temporal_loss_count": 10, "temporal_total": 10, "token_role_loss": 0.0, "token_role_loss_count": 4, "training_weight_max": 1.35, "training_weight_mean": 1.35, "training_weight_min": 1.35}, "val_selection_enabled": true, "val_selection_score": 117.895456}, "last_step_checkpoint": {"epoch": 7, "epoch_step": 81, "global_step": 650, "node_model_last_path": "outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/node_scorer_last.pt", "path_model_last_path": "outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/path_scorer_last.pt", "total_steps": 704}, "latest_epoch": 7, "latest_global_step": 665, "memory_router_variant": "question_pooled_multilabel_v1", "memory_tunnel_variant": "chain_depth_tunnel_support_delta_v1", "message_passing_variant": "support_to_event_bidirectional_shared_typed_v1", "model_arch_version": "node_memory_v11_tmcra_scale49m_trimaze", "node_encoder_variant": "hashed_tmcra_token_attn_pool_6l_w256", "node_max_tokens": 32, "pair_feature_adapter_variant": "residual_pair_feature_adapter_v1", "prebuild_issue_log": "outputs/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/graph_cache_prebuild_errors.jsonl", "question_encoder_variant": "hashed_tmcra_token_attn_pool_6l_w256", "question_intent_variant": "text_pooled_multihead_v1", "question_max_tokens": 48, "status": "running", "train": {"answer_plan_current_top1_accuracy": 0.379747, "answer_plan_current_total": 79, "answer_plan_loss": 0.549526, "answer_plan_loss_count": 80, "answer_plan_selected_positive_coverage_at_5": 0.821101, "answer_plan_selected_positive_total": 218, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 80, "answer_refusal_loss": 0.0, "answer_refusal_loss_count": 80, "answer_type_loss": 5.16927, "answer_type_loss_count": 80, "answer_type_metrics": {"multi_evidence": {"answer_plan_current_top1_accuracy": 0.376623, "answer_plan_current_total": 77, "answer_plan_selected_positive_coverage_at_5": 0.821596, "answer_plan_selected_positive_total": 213, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 78, "event5_positive_total": 213, "event_positive_coverage_at_5": 0.981221, "event_recall_at_1": 0.846154, "event_recall_at_5": 1.0, "event_recall_total": 78, "path3_positive_total": 234, "path_positive_coverage_at_3": 0.910256, "path_recall_at_3": 1.0, "path_recall_total": 78, "recall_event24_positive_total": 213, "recall_event_positive_coverage_at_24": 0.995305, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 78, "samples": 78, "temporal_accuracy": 0.0, "temporal_total": 0}, "time": {"answer_plan_current_top1_accuracy": 0.5, "answer_plan_current_total": 2, "answer_plan_selected_positive_coverage_at_5": 0.8, "answer_plan_selected_positive_total": 5, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 2, "event5_positive_total": 5, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 1.0, "event_recall_at_5": 1.0, "event_recall_total": 2, "path3_positive_total": 6, "path_positive_coverage_at_3": 1.0, "path_recall_at_3": 1.0, "path_recall_total": 2, "recall_event24_positive_total": 5, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 2, "samples": 2, "temporal_accuracy": 0.0, "temporal_total": 2}}, "evaluation_limited": true, "event5_positive_total": 218, "event_distractor_loss": 0.666098, "event_distractor_loss_count": 80, "event_hard_negative_loss": 0.078884, "event_hard_negative_loss_count": 80, "event_loss": 0.323909, "event_loss_count": 80, "event_matrix_delta_loss": 0.128518, "event_matrix_delta_loss_count": 80, "event_positive_coverage_at_5": 0.981651, "event_recall_at_1": 0.85, "event_recall_at_5": 1.0, "event_recall_total": 80, "event_selection_loss": 0.014046, "event_selection_loss_count": 80, "event_tunnel_loss": 0.213497, "event_tunnel_loss_count": 80, "event_tunnel_selection_loss": 0.0, "event_tunnel_selection_loss_count": 80, "final_event_set_loss": 0.013138, "final_event_set_loss_count": 80, "graph_error_count": 0, "loss": 3.466264, "loss_group_count": 11, "memory_router_exact_match": 0.375, "memory_router_f1": 0.845089, "memory_router_loss": 1.479962, "memory_router_loss_count": 80, "memory_router_total": 80, "path3_positive_total": 240, "path_hard_negative_loss": 0.117886, "path_hard_negative_loss_count": 80, "path_loss": 0.369553, "path_loss_count": 80, "path_matrix_delta_loss": 0.454712, "path_matrix_delta_loss_count": 80, "path_positive_coverage_at_3": 0.9125, "path_recall_at_3": 1.0, "path_recall_total": 80, "path_selection_loss": 0.12883, "path_selection_loss_count": 80, "path_tunnel_delta_loss": 0.60523, "path_tunnel_delta_loss_count": 80, "path_tunnel_delta_positive_coverage_at_3": 0.479167, "path_tunnel_delta_positive_total": 240, "path_tunnel_delta_recall_at_3": 0.75, "path_tunnel_delta_recall_total": 80, "path_tunnel_loss": 0.338907, "path_tunnel_loss_count": 80, "path_tunnel_rescue025_recall_at_3": 1.0, "path_tunnel_rescue025_recall_total": 80, "path_tunnel_rescue050_recall_at_3": 0.9875, "path_tunnel_rescue050_recall_total": 80, "path_tunnel_rescue100_recall_at_3": 0.9875, "path_tunnel_rescue100_recall_total": 80, "path_tunnel_selection_loss": 0.14157, "path_tunnel_selection_loss_count": 80, "path_tunnel_support_positive_coverage_at_3": 0.829167, "path_tunnel_support_positive_total": 240, "path_tunnel_support_recall_at_3": 0.9375, "path_tunnel_support_recall_total": 80, "question_understanding_loss": 5.293586, "question_understanding_loss_count": 80, "recall_event24_positive_total": 218, "recall_event_positive_coverage_at_24": 0.995413, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 80, "recall_loss": 0.820672, "recall_loss_count": 80, "recall_selection_loss": 0.0, "recall_selection_loss_count": 80, "rows_evaluated": 80, "rows_total": 439, "samples": 80, "supervision_bucket_metrics": {"typed_tunnel_multi_positive": {"answer_plan_current_top1_accuracy": 0.379747, "answer_plan_current_total": 79, "answer_plan_selected_positive_coverage_at_5": 0.821101, "answer_plan_selected_positive_total": 218, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 80, "event5_positive_total": 218, "event_positive_coverage_at_5": 0.981651, "event_recall_at_1": 0.85, "event_recall_at_5": 1.0, "event_recall_total": 80, "path3_positive_total": 240, "path_positive_coverage_at_3": 0.9125, "path_recall_at_3": 1.0, "path_recall_total": 80, "recall_event24_positive_total": 218, "recall_event_positive_coverage_at_24": 0.995413, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 80, "samples": 80, "temporal_accuracy": 0.0, "temporal_total": 2}}, "temporal_accuracy": 0.0, "temporal_loss": 0.513602, "temporal_loss_count": 2, "temporal_total": 2, "token_role_loss": 0.0, "token_role_loss_count": 11, "training_weight_max": 1.35, "training_weight_mean": 1.35, "training_weight_min": 1.35}, "training_sampling": {"answer_type_counts": {"multi_evidence": 422, "time": 17}, "base_row_count": 439, "effective_row_count": 697, "loss_weight_range_estimate": {"max": 1.549193, "min": 1.319091}, "mode": "source_aware_balanced", "repeat_histogram": {"1": 118, "2": 144}, "source_count": 1, "source_details": {"dataset": {"avg_example_weight_estimate": 1.540283, "loss_source_factor": 1.0, "row_count": 439, "row_share": 1.0, "sampled_row_count_estimate": 697, "sampled_row_share_estimate": 1.0, "sampling_source_factor": 1.0, "weighted_loss_mass": 676.184129}}, "source_row_counts": {"dataset": 439}, "supervision_bucket_counts": {"typed_tunnel_multi_positive": 439}, "weighted_answer_mass_estimate": {"multi_evidence": 653.759589, "time": 22.42454}, "weighted_supervision_bucket_mass_estimate": {"typed_tunnel_multi_positive": 680.095876}}, "val": {"answer_plan_current_top1_accuracy": 0.53125, "answer_plan_current_total": 32, "answer_plan_loss": 0.530247, "answer_plan_loss_count": 32, "answer_plan_selected_positive_coverage_at_5": 0.848101, "answer_plan_selected_positive_total": 79, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 32, "answer_refusal_loss": 0.0, "answer_refusal_loss_count": 32, "answer_type_loss": 6.441788, "answer_type_loss_count": 32, "answer_type_metrics": {"multi_evidence": {"answer_plan_current_top1_accuracy": 0.545455, "answer_plan_current_total": 22, "answer_plan_selected_positive_coverage_at_5": 0.847458, "answer_plan_selected_positive_total": 59, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 22, "event5_positive_total": 59, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 0.909091, "event_recall_at_5": 1.0, "event_recall_total": 22, "path3_positive_total": 66, "path_positive_coverage_at_3": 0.924242, "path_recall_at_3": 1.0, "path_recall_total": 22, "recall_event24_positive_total": 59, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 22, "samples": 22, "temporal_accuracy": 0.0, "temporal_total": 0}, "time": {"answer_plan_current_top1_accuracy": 0.5, "answer_plan_current_total": 10, "answer_plan_selected_positive_coverage_at_5": 0.85, "answer_plan_selected_positive_total": 20, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 10, "event5_positive_total": 20, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 0.7, "event_recall_at_5": 1.0, "event_recall_total": 10, "path3_positive_total": 30, "path_positive_coverage_at_3": 0.733334, "path_recall_at_3": 1.0, "path_recall_total": 10, "recall_event24_positive_total": 20, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 10, "samples": 10, "temporal_accuracy": 0.3, "temporal_total": 10}}, "evaluation_limited": true, "event5_positive_total": 79, "event_distractor_loss": 0.646066, "event_distractor_loss_count": 32, "event_hard_negative_loss": 0.062803, "event_hard_negative_loss_count": 32, "event_loss": 0.379233, "event_loss_count": 32, "event_matrix_delta_loss": 0.125441, "event_matrix_delta_loss_count": 32, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 0.84375, "event_recall_at_5": 1.0, "event_recall_total": 32, "event_selection_loss": 0.001734, "event_selection_loss_count": 32, "event_tunnel_loss": 0.248372, "event_tunnel_loss_count": 32, "event_tunnel_selection_loss": 0.0, "event_tunnel_selection_loss_count": 32, "final_event_set_loss": 0.012893, "final_event_set_loss_count": 32, "graph_error_count": 0, "loss": 3.604544, "loss_group_count": 4, "memory_router_exact_match": 0.34375, "memory_router_f1": 0.918403, "memory_router_loss": 0.575108, "memory_router_loss_count": 32, "memory_router_total": 32, "path3_positive_total": 96, "path_hard_negative_loss": 0.23831, "path_hard_negative_loss_count": 32, "path_loss": 0.422076, "path_loss_count": 32, "path_matrix_delta_loss": 0.564214, "path_matrix_delta_loss_count": 32, "path_positive_coverage_at_3": 0.864584, "path_recall_at_3": 1.0, "path_recall_total": 32, "path_selection_loss": 0.189845, "path_selection_loss_count": 32, "path_tunnel_delta_loss": 0.658501, "path_tunnel_delta_loss_count": 32, "path_tunnel_delta_positive_coverage_at_3": 0.614584, "path_tunnel_delta_positive_total": 96, "path_tunnel_delta_recall_at_3": 0.84375, "path_tunnel_delta_recall_total": 32, "path_tunnel_loss": 0.360756, "path_tunnel_loss_count": 32, "path_tunnel_rescue025_recall_at_3": 1.0, "path_tunnel_rescue025_recall_total": 32, "path_tunnel_rescue050_recall_at_3": 1.0, "path_tunnel_rescue050_recall_total": 32, "path_tunnel_rescue100_recall_at_3": 1.0, "path_tunnel_rescue100_recall_total": 32, "path_tunnel_selection_loss": 0.183911, "path_tunnel_selection_loss_count": 32, "path_tunnel_support_positive_coverage_at_3": 0.802083, "path_tunnel_support_positive_total": 96, "path_tunnel_support_recall_at_3": 0.96875, "path_tunnel_support_recall_total": 32, "question_understanding_loss": 3.751869, "question_understanding_loss_count": 32, "recall_event24_positive_total": 79, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 32, "recall_loss": 0.894676, "recall_loss_count": 32, "recall_selection_loss": 0.0, "recall_selection_loss_count": 32, "rows_evaluated": 32, "rows_total": 55, "samples": 32, "supervision_bucket_metrics": {"typed_tunnel_multi_positive": {"answer_plan_current_top1_accuracy": 0.53125, "answer_plan_current_total": 32, "answer_plan_selected_positive_coverage_at_5": 0.848101, "answer_plan_selected_positive_total": 79, "answer_plan_selected_recall_at_5": 1.0, "answer_plan_selected_total": 32, "event5_positive_total": 79, "event_positive_coverage_at_5": 1.0, "event_recall_at_1": 0.84375, "event_recall_at_5": 1.0, "event_recall_total": 32, "path3_positive_total": 96, "path_positive_coverage_at_3": 0.864584, "path_recall_at_3": 1.0, "path_recall_total": 32, "recall_event24_positive_total": 79, "recall_event_positive_coverage_at_24": 1.0, "recall_event_recall_at_24": 1.0, "recall_event_recall_total": 32, "samples": 32, "temporal_accuracy": 0.3, "temporal_total": 10}}, "temporal_accuracy": 0.3, "temporal_loss": 0.50511, "temporal_loss_count": 10, "temporal_total": 10, "token_role_loss": 0.0, "token_role_loss_count": 4, "training_weight_max": 1.35, "training_weight_mean": 1.35, "training_weight_min": 1.35}}
\ No newline at end of file
diff --git a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/training_issues.jsonl b/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/training_issues.jsonl
deleted file mode 100644
index 640b12907f599becf1fcb1904121786017fa356d..0000000000000000000000000000000000000000
--- a/models/action_frame_tunnel_graph548_tunnel_fusion_train_20260524_042557/training_issues.jsonl
+++ /dev/null
@@ -1 +0,0 @@
-{"backend": "process", "event_type": "batch_prepare_process_pool_ready", "graph_error_stage": "train_batch_prepare", "issue_type": "batch_prepare_pool_event", "timestamp": "2026-05-24T04:26:31", "worker_count": 16}
diff --git a/models/tmcra_v4_longmemeval_s500_20260715/README.md b/models/tmcra_v4_longmemeval_s500_20260715/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..37ac067fd0b1255a2483f9224b95cb682a34b8b5
--- /dev/null
+++ b/models/tmcra_v4_longmemeval_s500_20260715/README.md
@@ -0,0 +1,25 @@
+# TMCRA V4 benchmark weights
+
+These are the TMCRA graph scorer and reranker weights used by the public
+LongMemEval-S500 reproduction pipeline.
+
+| File | Purpose | SHA-256 |
+| --- | --- | --- |
+| `node_scorer.pt` | graph node scoring | `d2318aafb07f9a15a1d95e6eac1b3e09afa07c570dd9d15d3f75544c9950f201` |
+| `path_scorer.pt` | graph path scoring | `ecdc2bca51b7646a8a56c7db9cea4aeec581e64c3b61e6e06f7e9fa2a93b3bf5` |
+| `tmcra_v3_reranker.pt` | learned retrieval reranking | `09a285b484ca857b24b53ad19d5998302f16d224ca9e4fd73a4eb5a52f022942` |
+
+The weights are released under Apache-2.0. Load them only through the
+repository runtime, which requires `torch.load(..., weights_only=True)` and
+validates the expected checkpoint structure. Do not disable these checks for
+untrusted files.
+
+The public checkpoints contain only their inference contracts and model state.
+Training state, training-machine paths, and run metadata are intentionally
+excluded. Their tensors are unchanged from the benchmark-bound checkpoints.
+
+The files are tracked with Git LFS. To pull only this release:
+
+```bash
+git lfs pull --include="models/tmcra_v4_longmemeval_s500_20260715/*.pt"
+```
diff --git a/models/tmcra_v4_longmemeval_s500_20260715/TMCRA_MODEL_MANIFEST.json b/models/tmcra_v4_longmemeval_s500_20260715/TMCRA_MODEL_MANIFEST.json
new file mode 100644
index 0000000000000000000000000000000000000000..1c7cbd684c1522b381a2b85b9301b665dd4c3103
--- /dev/null
+++ b/models/tmcra_v4_longmemeval_s500_20260715/TMCRA_MODEL_MANIFEST.json
@@ -0,0 +1,35 @@
+{
+ "schema_version": "tmcra.model-release.1",
+ "release": "tmcra_v4_longmemeval_s500_20260715",
+ "license": "Apache-2.0",
+ "safe_loading": {
+ "format": "pytorch_checkpoint",
+ "required_torch_load_argument": "weights_only=True",
+ "allow_pickle_fallback": false
+ },
+ "files": [
+ {
+ "name": "node_scorer.pt",
+ "bytes": 207356243,
+ "sha256": "d2318aafb07f9a15a1d95e6eac1b3e09afa07c570dd9d15d3f75544c9950f201",
+ "public_inference_keys": ["config", "state_dict"]
+ },
+ {
+ "name": "path_scorer.pt",
+ "bytes": 207356243,
+ "sha256": "ecdc2bca51b7646a8a56c7db9cea4aeec581e64c3b61e6e06f7e9fa2a93b3bf5",
+ "public_inference_keys": ["config", "state_dict"]
+ },
+ {
+ "name": "tmcra_v3_reranker.pt",
+ "bytes": 4232213,
+ "sha256": "09a285b484ca857b24b53ad19d5998302f16d224ca9e4fd73a4eb5a52f022942",
+ "public_inference_keys": [
+ "schema_version",
+ "channel_names",
+ "model_config",
+ "model_state"
+ ]
+ }
+ ]
+}
diff --git a/models/tmcra_v4_longmemeval_s500_20260715/node_scorer.pt b/models/tmcra_v4_longmemeval_s500_20260715/node_scorer.pt
new file mode 100644
index 0000000000000000000000000000000000000000..94826610490a5ce0b6ea9ef377440a6f6f78176c
--- /dev/null
+++ b/models/tmcra_v4_longmemeval_s500_20260715/node_scorer.pt
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d2318aafb07f9a15a1d95e6eac1b3e09afa07c570dd9d15d3f75544c9950f201
+size 207356243
diff --git a/models/tmcra_v4_longmemeval_s500_20260715/path_scorer.pt b/models/tmcra_v4_longmemeval_s500_20260715/path_scorer.pt
new file mode 100644
index 0000000000000000000000000000000000000000..a6e9bfbcbd652455921a0769b65c12c29260bf34
--- /dev/null
+++ b/models/tmcra_v4_longmemeval_s500_20260715/path_scorer.pt
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ecdc2bca51b7646a8a56c7db9cea4aeec581e64c3b61e6e06f7e9fa2a93b3bf5
+size 207356243
diff --git a/models/tmcra_v4_longmemeval_s500_20260715/tmcra_v3_reranker.pt b/models/tmcra_v4_longmemeval_s500_20260715/tmcra_v3_reranker.pt
new file mode 100644
index 0000000000000000000000000000000000000000..9cd042db9f697263a48b0f2fa4d55793f2c2f23f
--- /dev/null
+++ b/models/tmcra_v4_longmemeval_s500_20260715/tmcra_v3_reranker.pt
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:09a285b484ca857b24b53ad19d5998302f16d224ca9e4fd73a4eb5a52f022942
+size 4232213
diff --git a/results/README.md b/results/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..9a43561ce2d37229391c187391f8215f228d8db9
--- /dev/null
+++ b/results/README.md
@@ -0,0 +1,34 @@
+# Benchmark results / Benchmark 成绩说明
+
+## Current result / 当前成绩
+
+The current published TMCRA result is **411 / 500 = 82.2%** on LongMemEval S500. The machine-readable scorecard is [`latest_benchmark.json`](latest_benchmark.json), and the maintained reproduction pipeline is [`../benchmarks/longmemeval/`](../benchmarks/longmemeval/README.md).
+
+TMCRA 当前发布的 LongMemEval S500 成绩为 **411 / 500 = 82.2%**。机器可读成绩见 [`latest_benchmark.json`](latest_benchmark.json)，持续维护的复现链路见 [`../benchmarks/longmemeval/`](../benchmarks/longmemeval/README.zh-CN.md)。
+
+| Task / 任务 | Correct / Total | Accuracy |
+| --- | ---: | ---: |
+| Knowledge Update / 信息更新 | 71 / 78 | 91.0% |
+| Multi-session / 跨会话整合 | 90 / 133 | 67.7% |
+| Single-session Assistant / 单会话助手信息 | 55 / 56 | 98.2% |
+| Single-session Preference / 单会话偏好 | 27 / 30 | 90.0% |
+| Single-session User / 单会话用户信息 | 67 / 70 | 95.7% |
+| Temporal Reasoning / 时间推理 | 101 / 133 | 75.9% |
+| **Overall / 总成绩** | **411 / 500** | **82.2%** |
+
+## Retained historical baseline / 保留的历史基线
+
+The following files belong to the frozen 2026-05-25 baseline, which scored **310 / 500 = 62.0%**. They are retained for longitudinal comparison and must not be interpreted as the latest result:
+
+以下文件属于 2026-05-25 冻结的 **310 / 500 = 62.0%** 历史基线。它们仅用于纵向对比，不代表当前成绩：
+
+```text
+predictions.jsonl
+judge_gpt4o_alias_vectorengine.jsonl
+judge_gpt4o_alias_vectorengine.jsonl.summary.json
+lme_s500_frozen_baseline38_full10_20260525_results.tar.gz
+```
+
+The detailed historical record remains in [`../docs/BASELINE_S500_20260525.md`](../docs/BASELINE_S500_20260525.md).
+
+历史基线的详细记录仍保留在 [`../docs/BASELINE_S500_20260525.md`](../docs/BASELINE_S500_20260525.md)。
diff --git a/results/judge_gpt4o_alias_vectorengine.jsonl b/results/judge_gpt4o_alias_vectorengine.jsonl
deleted file mode 100644
index 5a7be22b5570268cb1c20fbcf5ce5afcfe32891d..0000000000000000000000000000000000000000
--- a/results/judge_gpt4o_alias_vectorengine.jsonl
+++ /dev/null
@@ -1,500 +0,0 @@
-{"question_id": "7e00a6cb", "hypothesis": "International Budget Hostel", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "1903aded", "hypothesis": "I can’t recover the 7th job from the provided memory evidence. The retrieved notes only show unrelated lists, like living room color schemes, dessert ideas, and sales tax questions, not the earlier work-from-home jobs for seniors list.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "ceb54acb", "hypothesis": "Sexual fixations, Problematic sexual behaviors, Sexual impulsivity, Compulsive sexuality", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "f523d9fe", "hypothesis": "I can’t tell from the available memory.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "0e5e2d1a", "hypothesis": "38 subjects", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "fea54f57", "hypothesis": "Evolution", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "cc539528", "hypothesis": "Ruby, Python, or PHP", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "dc439ea3", "hypothesis": "Hoop Dance", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "18dcd5a5", "hypothesis": "4", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "488d3006", "hypothesis": "GR-90", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "58470ed2", "hypothesis": "The Library is a sphere whose exact center is any one of its hexagons and whose circumference is inaccessible.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "8cf51dda", "hypothesis": "1. To identify molecular subtypes of endometrial cancer using a combination of genomic and transcriptomic approaches. 2. To investigate the clinical and biological significance of the identified molecular subtypes, including their association with patient outcomes. 3. To develop biomarkers for the early detection and prognosis of endometrial cancer based on the identified molecular subtypes.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "1d4da289", "hypothesis": "biometric authentication and one-time passwords (OTP)", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "8464fc84", "hypothesis": "Roscioli", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "8aef76bc", "hypothesis": "Mod Podge or another sealant", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "71a3fd6b", "hypothesis": "+49 (0) 62 32 / 14 23 - 0", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "2bf43736", "hypothesis": "Chapter 4 of Book 1, titled \"Vocal Prayer and Meditation.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "70b3e69b", "hypothesis": "Marina Rossell", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "8752c811", "hypothesis": "The 27th parameter was Perspective.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "3249768e", "hypothesis": "Absinthe", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "1b9b7252", "hypothesis": "Mindful.org", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "1568498a", "hypothesis": "29. Rd3", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "6222b6eb", "hypothesis": "6S", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "e8a79c70", "hypothesis": "2-3 eggs", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "d596882b", "hypothesis": "By Chloe", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "e3fc4d6e", "hypothesis": "Dr. Arati Prabhakar", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "51b23612", "hypothesis": "Nu, pogodi!", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "3e321797", "hypothesis": "10 minutes", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "e982271f", "hypothesis": "Revolution Hall", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "352ab8bd", "hypothesis": "approximately 20%", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "fca762bc", "hypothesis": "Memrise", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "7a8d0b71", "hypothesis": "In the visible remembered campaign-plan excerpt, I can confirm we discussed the DHL Wellness Retreats email marketing campaign, but the specific influencer marketing allocation amount is not shown in the evidence provided.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "a40e080f", "hypothesis": "Patagonia and Southwest Airlines", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "8b9d4367", "hypothesis": "Jaipur Rugs", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "5809eb10", "hypothesis": "2015", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "41275add", "hypothesis": "How to Sit Properly at a Desk to Avoid Back Pain (Mayo Clinic)", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "4388e9dd", "hypothesis": "an untidy, stained white shirt", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "4baee567", "hypothesis": "12", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "561fabcd", "hypothesis": "Contaminated Colossus", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "b759caee", "hypothesis": "@jessica_poole_jewellery", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "ac031881", "hypothesis": "squares", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "28bcfaac", "hypothesis": "MusicTheory.net", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "16c90bf4", "hypothesis": "A pilsner or lager.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "c8f1aeed", "hypothesis": "Pennsylvania", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "eaca4986", "hypothesis": "C D E F G A B A G F E D C", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "c7cf7dfd", "hypothesis": "Nostalgia", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "e48988bc", "hypothesis": "Patagonia", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "1de5cff2", "hypothesis": "Veja", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "65240037", "hypothesis": "1:10 ratio", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "778164c6", "hypothesis": "The evidence only shows that we discussed popular Caribbean dishes featuring snapper, but it does not include the actual Jamaican fruit-based dish name.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "e47becba", "hypothesis": "Business Administration", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "118b2229", "hypothesis": "45 minutes each way", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "51a45a95", "hypothesis": "The evidence does not say where you redeemed the $5 coffee creamer coupon.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "58bf7951", "hypothesis": "The Glass Menagerie", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "1e043500", "hypothesis": "Summer Vibes", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "c5e8278d", "hypothesis": "Johnson", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "6ade9755", "hypothesis": "Serenity Yoga", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "6f9b354f", "hypothesis": "a lighter shade of gray", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "58ef2f1c", "hypothesis": "back on Valentine's Day", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "f8c5f88b", "hypothesis": "The retrieved evidence does not say where you bought the new tennis racket from.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "5d3d2817", "hypothesis": "marketing specialist at a small startup", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "7527f7e2", "hypothesis": "$800", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "c960da58", "hypothesis": "20", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "3b6f954b", "hypothesis": "University of Melbourne", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "726462e0", "hypothesis": "10%", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "94f70d80", "hypothesis": "The retrieved evidence does not state any assembly time for an IKEA bookshelf. It only mentions plans to visit IKEA to look at coffee tables and read reviews, so the duration cannot be determined from this evidence.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "66f24dbb", "hypothesis": "No sister gift is shown in memory; the only recorded birthday gift purchase was a customized phone case with his favorite game's logo for your brother.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "ad7109d1", "hypothesis": "The retrieved evidence does not specify the speed of your new internet plan.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "af8d2e46", "hypothesis": "7", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "dccbc061", "hypothesis": "staunch atheist", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "c8c3f81d", "hypothesis": "Nike", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "8ebdbe50", "hypothesis": "Data Science certification", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "6b168ec8", "hypothesis": "3", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "75499fd8", "hypothesis": "Golden Retriever", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "21436231", "hypothesis": "Not stated", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "95bcc1c8", "hypothesis": "10", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "0862e8bf", "hypothesis": "Luna", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "853b0a1d", "hypothesis": "I can tell it was a gift from your grandma, but the evidence here doesn’t include your age when she gave you the silver necklace.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "a06e4cfe", "hypothesis": "3:1", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "37d43f65", "hypothesis": "16GB", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "b86304ba", "hypothesis": "Triple what you paid for it.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "d52b4f67", "hypothesis": "Grand Ballroom", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "25e5aa4f", "hypothesis": "UCLA", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "caf9ead2", "hypothesis": "around 5 hours", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "8550ddae", "hypothesis": "Lavender Dream cocktail", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "60d45044", "hypothesis": "Japanese short-grain rice", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "3f1e9474", "hypothesis": "Sarah", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "86b68151", "hypothesis": "IKEA", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "577d4d32", "hypothesis": "7 pm", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "ec81a493", "hypothesis": "500 copies", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "15745da0", "hypothesis": "about three months", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "e01b8e2f", "hypothesis": "Hawaii", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "bc8a6e93", "hypothesis": "lemon blueberry cake", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "ccb36322", "hypothesis": "There’s no evidence here of any music streaming service you’ve been using lately; the only services mentioned are photo album services like Shutterfly, Snapfish, and Mixbook.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "001be529", "hypothesis": "Your asylum application had been approved by May 28, 2023, and there was a waiting period, but the evidence does not state how long you waited for the decision.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "b320f3f8", "hypothesis": "a rare blue Snaggletooth action figure", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "19b5f2b3", "hypothesis": "The evidence only shows that you were in Japan/Kyoto and planning to travel elsewhere in Asia afterward, but it does not state how long you were in Japan.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "4fd1909e", "hypothesis": "Xfinity Center", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "545bd2b5", "hypothesis": "2 hours per day", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "8a137a7f", "hypothesis": "Philips LED bulb", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "76d63226", "hypothesis": "55-inch", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "86f00804", "hypothesis": "The Seven Husbands of Evelyn Hugo", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "8e9d538c", "hypothesis": "17 skeins", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "311778f1", "hypothesis": "around 10 hours", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "c19f7a0b", "hypothesis": "around 6:30 pm", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "4100d0a0", "hypothesis": "Irish and Italian", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "29f2956b", "hypothesis": "30 minutes daily", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "1faac195", "hypothesis": "Denver", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "faba32e5", "hypothesis": "24 hours", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "f4f1d8a4", "hypothesis": "my sister", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "c14c00dd", "hypothesis": "Trader Joe's", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "36580ce8", "hypothesis": "The evidence only says you’d been dealing with unspecified health issues for a few months; it does not identify any specific issue you initially thought was just a cold.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "3d86fd0a", "hypothesis": "Information unavailable", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "a82c026e", "hypothesis": "Dark Souls 3 DLC", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "0862e8bf_abs", "hypothesis": "I can’t determine your hamster’s name from the provided evidence because the memories only mention your cat and unrelated coding discussion, not any hamster.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "15745da0_abs", "hypothesis": "No evidence says how long you've been collecting vintage films.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "bc8a6e93_abs", "hypothesis": "I don't know", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "19b5f2b3_abs", "hypothesis": "The available evidence doesn’t mention a Korea trip or its duration, so I can’t determine how long you were in Korea.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "29f2956b_abs", "hypothesis": "I only have evidence that you practice guitar for 30 minutes daily; there’s no evidence here about a daily violin practice time.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "f4f1d8a4_abs", "hypothesis": "I only remember a new stand mixer as a birthday gift from your sister, not your dad.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "0a995998", "hypothesis": "2", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "6d550036", "hypothesis": "1", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "gpt4_59c863d7", "hypothesis": "4", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "b5ef892d", "hypothesis": "5 days", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "e831120c", "hypothesis": "3.5 weeks", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "3a704032", "hypothesis": "3", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "gpt4_d84a3211", "hypothesis": "I can identify bike-related activity this year, like getting bike lights installed, but there are no explicit dollar amounts for any bike-related purchases or services in the evidence, so I can’t compute a total spent.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "aae3761f", "hypothesis": "10 hours", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "gpt4_f2262a51", "hypothesis": "1", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "dd2973ad", "hypothesis": "I can only narrow it down indirectly: your doctor’s appointment was at 10 AM on Thursday 2023-05-18, so the day before was Wednesday 2023-05-17. The only bedtime evidence says you went to bed at 2 AM on last Wednesday relative to 2023-05-29, which refers to 2023-05-24, not 2023-05-17. So the closest bedtime mentioned is 2 AM, but it does not match the day before that appointment.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "c4a1ceb8", "hypothesis": "3", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "gpt4_a56e767c", "hypothesis": "3", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "6cb6f249", "hypothesis": "17 days", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "46a3abf7", "hypothesis": "3", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "36b9f61e", "hypothesis": "2000", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "28dc39ac", "hypothesis": "29 hours", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "gpt4_2f8be40d", "hypothesis": "2", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "2e6d26dc", "hypothesis": "3", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "gpt4_15e38248", "hypothesis": "2", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "88432d0a", "hypothesis": "4", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "80ec1f4f", "hypothesis": "1", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "d23cf73b", "hypothesis": "2", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "gpt4_7fce9456", "hypothesis": "2", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "d682f1a2", "hypothesis": "2", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "7024f17c", "hypothesis": "0.5 hours", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_5501fe77", "hypothesis": "TikTok", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "gpt4_2ba83207", "hypothesis": "Trader Joe's", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "2318644b", "hypothesis": "Tokyo is the only place with a numeric nightly accommodation cost in the evidence: about $30/night. For Hawaii, the evidence only says you had a luxurious resort stay in Maui, which suggests it was more expensive but does not give a dollar amount, so the exact per-night difference cannot be calculated from the evidence.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "2ce6a0f2", "hypothesis": "1", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "gpt4_d12ceb0e", "hypothesis": "Insufficient evidence to compute the average age of you, your parents, and your grandparents.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "00ca467f", "hypothesis": "2", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "b3c15d39", "hypothesis": "5 days", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_31ff4165", "hypothesis": "4", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "eeda8a6d", "hypothesis": "17", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "2788b940", "hypothesis": "2", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "60bf93ed", "hypothesis": "It arrived on 1/20, but I don’t see the purchase date in the evidence, so I can’t calculate the number of days exactly.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "9d25d4e0", "hypothesis": "2", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "129d1232", "hypothesis": "600 dollars", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "60472f9c", "hypothesis": "1", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "gpt4_194be4b3", "hypothesis": "3", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "a9f6b44c", "hypothesis": "2", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "d851d5ba", "hypothesis": "At least 2,250 USD", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "5a7937c8", "hypothesis": "2", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "gpt4_ab202e7f", "hypothesis": "2", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "gpt4_e05b82a6", "hypothesis": "7", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "gpt4_731e37d7", "hypothesis": "$200", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "edced276", "hypothesis": "8 days", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "10d9b85a", "hypothesis": "2", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "e3038f8c", "hypothesis": "87", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "2b8f3739", "hypothesis": "$495", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "1a8a66a6", "hypothesis": "2", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "c2ac3c61", "hypothesis": "5", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "bf659f65", "hypothesis": "1", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "gpt4_372c3eed", "hypothesis": "8 years", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "gpt4_2f91af09", "hypothesis": "6", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "81507db6", "hypothesis": "3", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "88432d0a_abs", "hypothesis": "0", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "80ec1f4f_abs", "hypothesis": "0", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "eeda8a6d_abs", "hypothesis": "not evidenced in the retrieved memory", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "60bf93ed_abs", "hypothesis": "70 days", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "edced276_abs", "hypothesis": "I can only confirm a Hawaii trip, and no number of days is given for it. There’s also no evidence here of a Seattle trip duration, so the total can’t be determined from the provided evidence.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "gpt4_372c3eed_abs", "hypothesis": "8 years documented; no evidence of a completed Master's degree", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "8a2466db", "hypothesis": "Sorry, I don’t know of any specific resources from memory.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "06878be2", "hypothesis": "A comfortable, durable Sony-compatible camera bag, a Godox V1 hard case or Think Tank Photo Flash Pouch, and a lens cleaning kit with a microfiber cloth, blower, and lens cleaning solution would complement your Sony A7R IV setup.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "75832dbd", "hypothesis": "Recent publications or conferences on the impact of mega-churches on Baptist church vitality, Baptist identity and mission, and practical collaboration models among smaller Baptist churches—especially case-study-focused work on regional or denominational cooperation.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "0edc2aef", "hypothesis": "Look for a Miami hotel with a balcony hot tub and a standout view, similar to your preferred room at The Edgewater Hotel.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "35a27287", "hypothesis": "French- or Spanish-focused language exchange meetups, conversation groups, or cultural events nearby would fit you best; check Meetup, Facebook Groups, local universities, or language schools this weekend.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "32260d93", "hypothesis": "Mike Birbiglia: My Girlfriend's Boyfriend", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "195a1a1b", "hypothesis": "Try a calm wind-down routine: read a non-stimulating book, do a guided sleep meditation or body scan, practice deep breathing, do gentle yoga or stretching, listen to soothing music or nature sounds, and take a 30-minute break from screens before bed.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "afdc33df", "hypothesis": "Regularly clean your granite countertop, use coasters or pads under pots and utensils, always use a cutting board, and check the faucet/aerator for moisture or buildup so the sink area stays cleaner.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "caf03d32", "hypothesis": "Try more plant-based slow cooker recipes like lentil curry, roasted vegetable stew, black bean chili, eggplant parmesan, vegan chili, or a quinoa and black bean bowl.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "54026fce", "hypothesis": "Try suggesting virtual coffee breaks in a collaborative way: ask open-ended questions about staying connected, share that you miss casual chats, propose a weekly virtual coffee break, and invite your team’s feedback before deciding.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "06f04340", "hypothesis": "Serve a hearty quinoa salad with your roasted homegrown vegetables over mixed greens, finished with vinaigrette; roast the vegetables first, then toss with the greens and quinoa.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "6b7dfb22", "hypothesis": "Study real flowers from life or photos, browse Instagram and art communities for ideas, experiment with new techniques like palette-knife texture, and take breaks while tracking small wins to help inspiration return.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "1a1907b4", "hypothesis": "Try a Pimm's Cup with a Twist, especially with Ruby Red Grapefruit simple syrup.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "09d032c9", "hypothesis": "A compact portable power bank could help as a backup.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "38146c39", "hypothesis": "Try adding warm spices, nuts like walnuts or pecans, a little citrus zest, or a touch of espresso or coffee for extra flavor.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "d24813b1", "hypothesis": "Bake chocolate chip cookies or a lemon poppyseed cake; both fit your recent baking interests, and the lemon poppyseed cake worked well for a colleague event before.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "57f827a0", "hypothesis": "No direct memory-backed evidence shows you were planning to rearrange your bedroom furniture this weekend.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "95228167", "hypothesis": "Try several guitars and focus on comfort, playability, build quality, tuning stability, and the sound you like best.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "505af2f5", "hypothesis": "Try a homemade vanilla-cinnamon creamer with unsweetened almond or oat milk, a little monk fruit or stevia instead of honey, and optional coconut or hazelnut extract.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "75f70248", "hypothesis": "Possibly yes—the prior memory suggests your living room had a dust issue and a shedding cat, so dust or pet hair there could be contributing to your sneezing.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "d6233ab6", "hypothesis": "Yes—if you’re feeling nostalgic and want to reconnect with old high school friends, attending your reunion sounds like a good idea.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "1da05512", "hypothesis": "Buy now if your storage issue is already a problem and a beginner-friendly 2-bay model fits your budget; wait only if you expect much larger future storage needs or still need to compare options.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "fca70973", "hypothesis": "Try Universal Studios Hollywood’s VIP Experience with the Gourmet Buffet and Behind-the-Scenes Tour as the top pick; Disneyland Resort Halloween dining or Knott’s Scary Farm Boo-fet would also fit your immersive, exclusive theme park weekend vibe.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "b6025781", "hypothesis": "Try herb-rosemary roasted sweet potatoes, quinoa and roasted vegetable bowls with salmon, tofu, or lentil bolognese, and a batch of homemade granola for snacks.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "a89d7624", "hypothesis": "Visit Red Rocks Park and Amphitheatre and check out other live music spots like The Ogden Theatre, The Bluebird Theater, The Fillmore Auditorium, and Larimer Lounge.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "b0479f84", "hypothesis": "Try Planet Earth, Blue Planet, Dynasties, Chasing Coral, and March of the Penguins—especially the nature and wildlife ones since that’s what you were most interested in.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "1d4e3b97", "hypothesis": "Yes — the recent upgrade to your bike’s chain and cassette could be why it’s performing better.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "07b6f563", "hypothesis": "An iPhone 13 Pro-compatible wallet case like the OtterBox wallet case would be a useful accessory; I’d skip suggesting a power bank since you already have one that’s working well.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "1c0ddc50", "hypothesis": "Listen to history and science podcasts during your 40-minute commute—try starter episodes from Hardcore History, Lore, The Dollop, StarTalk Radio, or Radiolab.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "0a34ad58", "hypothesis": "Since you already have a Suica card, use it for trains and subways to make getting around easier. If you’re heading out from Shinjuku, plan a little extra time, especially for early or busy destinations like Tsukiji, where it’s best to leave very early and expect crowds. Keep some yen/cash with you because some places, especially around markets, may not take cards, and if needed you can exchange money near Shinjuku.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "d3ab962e", "hypothesis": "8 miles", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "2311e44b", "hypothesis": "The Nightingale' is 440 pages long, but I don’t have a numeric page-progress value for how far you’ve already read, so the exact pages left can’t be calculated from the evidence.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "cc06de0d", "hypothesis": "I can’t compute the difference from the provided evidence because it gives your daily train fare as $6, but it doesn’t include the taxi ride cost.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "a11281a2", "hypothesis": "about 100 followers", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "4f54b7c9", "hypothesis": "5", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "85fa3a3f", "hypothesis": "50", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "9aaed6a3", "hypothesis": "$0.75", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "1f2b8d4f", "hypothesis": "I can’t determine an exact price difference from the evidence. The price of your luxury boots isn’t given, and no specific similar pair found at a budget store has a stated price. The closest budget-store-style prices mentioned were about $20-$50 for affordable ankle boots, but that still isn’t enough to calculate the difference.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "e6041065", "hypothesis": "40%", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "51c32626", "hypothesis": "No explicit submission date was given; it was only mentioned by 2023/05/22 as a past event.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "d905b33f", "hypothesis": "20%", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "7405e8b1", "hypothesis": "HelloFresh: 40% on your first order. For UberEats, the evidence only says you got 20% off an order last week, not that it was your first order.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "f35224e0", "hypothesis": "At least 12 episodes", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "6456829e", "hypothesis": "5 tomato plants and 3 cucumber plants", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "a4996e51", "hypothesis": "Up to 50 hours/week.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "3c1045c8", "hypothesis": "unavailable", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "60036106", "hypothesis": "12,000 people", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "681a1674", "hypothesis": "1", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "e25c3b8d", "hypothesis": "I can only tell that you paid $200 for the designer handbag at TK Maxx; the savings amount isn’t stated in the evidence, so it can’t be determined from this information alone.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "4adc0475", "hypothesis": "Insufficient evidence for an exact total.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "4bc144e2", "hypothesis": "Parking ticket: $50; car wash not found in the retrieved evidence.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "ef66a6e5", "hypothesis": "4", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "5025383b", "hypothesis": "cooking", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "a1cc6108", "hypothesis": "You were about 0 years old when Alex was born.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "9ee3ecd6", "hypothesis": "300 points", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "3fdac837", "hypothesis": "12 days", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "91b15a6e", "hypothesis": "$5,150", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "27016adc", "hypothesis": "The planned renovations are $20,000. I don’t see the countryside property’s purchase price in the provided evidence, so the exact percentage can’t be computed from this evidence alone.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "720133ac", "hypothesis": "$75", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "77eafa52", "hypothesis": "$300", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "8979f9ec", "hypothesis": "I can only confirm those were lunch meal-prep recipe ideas mentioned on 2023/05/26; there’s no evidence here giving a number of meals for the chicken fajitas or any lentil soup, so the total can’t be computed from the provided evidence.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "0100672e", "hypothesis": "$12 per coffee mug", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "a96c20ee", "hypothesis": "I presented a poster on my thesis research at my first research conference over the summer, but the university isn't stated in the evidence provided.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "92a0aa75", "hypothesis": "Senior Marketing Specialist at the company", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "3fe836c9", "hypothesis": "The final sale price was $325,000. The only other amount mentioned is $350,000, but in the evidence it appears as a generic mortgage example, not a clearly stated pre-approval amount, so the exact difference can’t be determined from the evidence.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "1c549ce4", "hypothesis": "The only purchase cost shown is the waterproof car cover at $120. I don’t see evidence that you purchased a detailing spray or what it cost, so I can’t add a total beyond the confirmed $120.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "6c49646a", "hypothesis": "1940 miles", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "1192316e", "hypothesis": "I can’t compute a total from the provided evidence because it only mentions meditation and breakfast timing, not how long you take to get ready or commute to work.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "0ea62687", "hypothesis": "I can’t determine the difference from the retrieved memory: a few months ago it was 30 mpg, but your current mpg isn’t stated.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "67e0d0f2", "hypothesis": "At least 8. You explicitly mentioned completing 8 edX courses, and you also said you had completed some Coursera courses, but without a number, so the exact total isn’t determined from the evidence.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "bb7c3b45", "hypothesis": "The evidence only gives the original retail price of the Jimmy Choo heels as $500. It does not say what you actually paid, so the amount saved cannot be determined from the provided evidence.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "ba358f49", "hypothesis": "I don't know—Rachel is expected to get married in 2023, but your age isn't provided.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "61f8c8f8", "hypothesis": "10 minutes faster", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "60159905", "hypothesis": "1", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "ef9cf60a", "hypothesis": "$100", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "73d42213", "hypothesis": "No explicit arrival time is stated for Monday. The only clinic-travel detail says it took two hours to get there last time, but that doesn’t give a reach time.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "bc149d6b", "hypothesis": "70 pounds", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "099778bb", "hypothesis": "Insufficient evidence to determine the percentage; only the total of 100 leadership positions is provided, not how many are held by women.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "09ba9854", "hypothesis": "About $50.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "d6062bb9", "hypothesis": "1998", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "157a136e", "hypothesis": "I don't know", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "c18a7dc8", "hypothesis": "7", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "a3332713", "hypothesis": "About $200 total", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "55241a1f", "hypothesis": "Your recent Facebook Live session had 12 comments, but I don’t have evidence here for the comment count of your most popular YouTube video, so I can’t compute the total from the available information.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "a08a253f", "hypothesis": "4", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "f0e564bc", "hypothesis": "The evidence shows $800 for the designer handbag, but no amount is provided for the high-end skincare products, so the total cannot be determined.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "078150f1", "hypothesis": "$50 more", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "8cf4d046", "hypothesis": "I can’t compute an average GPA from the evidence provided. It mentions your undergraduate result as a First-Class distinction from the University of Mumbai and that you completed a Master’s in Data Science at UIUC, but it does not give explicit GPA values for either degree.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "a346bb18", "hypothesis": "12 minutes", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "37f165cf", "hypothesis": "416 pages and 440 pages", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "8e91e7d9", "hypothesis": "1", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "87f22b4a", "hypothesis": "$120", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "e56a43b9", "hypothesis": "$5 discount", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "efc3f7c2", "hypothesis": "30 minutes earlier", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "21d02d0d", "hypothesis": "2", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "2311e44b_abs", "hypothesis": "You mentioned that you've been reading \"Sapiens\" at a pace of 10–20 pages a week, but there’s no evidence here stating the book’s total page count or how many pages you’ve already finished, so the exact number of pages left can’t be determined from the provided evidence.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "6456829e_abs", "hypothesis": "5", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "e5ba910e_abs", "hypothesis": "$378", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "a96c20ee_abs", "hypothesis": "Harvard University", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "ba358f49_abs", "hypothesis": "Insufficient evidence to determine Rachel's age when you get married.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "09ba9854_abs", "hypothesis": "The evidence doesn’t give enough to calculate the savings exactly. It does mention the Airport Limousine Bus costs ¥3,200 one way to Shinjuku Station, but it does not provide a full taxi fare from the airport to your hotel, only that a short taxi ride from Shinjuku Station to the hotel might be needed.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "gpt4_59149c77", "hypothesis": "7 days", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_f49edff3", "hypothesis": "helped my friend prepare a nursery, helped my cousin pick out stuff for her baby shower, ordered a customized phone case for my friend's birthday", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "71017276", "hypothesis": "4 weeks", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "b46e15ed", "hypothesis": "2 months", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_fa19884c", "hypothesis": "6 days", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "0bc8ad92", "hypothesis": "2 months", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "af082822", "hypothesis": "2 weeks", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_4929293a", "hypothesis": "Michael's engagement party", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_b5700ca9", "hypothesis": "4 days ago", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "9a707b81", "hypothesis": "21 days", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_1d4ab0c9", "hypothesis": "24 days", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_e072b769", "hypothesis": "about 3 weeks ago", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "0db4c65d", "hypothesis": "18 days", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_1d80365e", "hypothesis": "2 days", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_7f6b06db", "hypothesis": "Yosemite, Big Sur and Monterey, Yosemite National Park", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "gpt4_6dc9b45b", "hypothesis": "4 months ago", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_8279ba02", "hypothesis": "10 days ago", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_18c2b244", "hypothesis": "I used a Buy One Get One Free coupon on Luvs diapers at Walmart, then I redeemed $12 cashback for a $10 Amazon gift card from Ibotta, then I signed up for the rewards program at ShopRite.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_a1b77f9c", "hypothesis": "Insufficient evidence to calculate the total weeks spent.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "gpt4_1916e0ea", "hypothesis": "I can identify the Instacart online grocery shopping date as 2023/02/28, but the provided evidence does not include the date you cancelled your FarmFresh subscription, so the number of days passed cannot be determined from this evidence.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "gpt4_7a0daae1", "hypothesis": "I can tell you received your new tennis racket on 2023/03/17, but the buy date isn’t provided in the evidence, so the number of weeks between buying and receiving can’t be determined from the available information.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "gpt4_468eb063", "hypothesis": "9 days", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_7abb270c", "hypothesis": "Science Museum, Metropolitan Museum of Art, Museum of Contemporary Art, Modern Art Museum, and two missing museum visits", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "gpt4_1e4a8aeb", "hypothesis": "6 days", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_4fc4f797", "hypothesis": "38 days", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "4dfccbf7", "hypothesis": "24 days", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_61e13b3c", "hypothesis": "3 weeks", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "gpt4_45189cb4", "hypothesis": "College Football National Championship game (Georgia vs. Alabama), then Lakers game, then Kansas City Chiefs vs. Buffalo Bills Divisional Round game", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "2ebe6c90", "hypothesis": "21 days", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_e061b84f", "hypothesis": "volleyball league game, Spring Sprint Triathlon, company's annual charity soccer tournament", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "370a8ff4", "hypothesis": "11 weeks and 4 days", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "gpt4_d6585ce8", "hypothesis": "free outdoor concert series in the park, jazz night at a local bar", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "gpt4_4ef30696", "hypothesis": "1", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_ec93e27f", "hypothesis": "train", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "6e984301", "hypothesis": "about 9 weeks", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "8077ef71", "hypothesis": "26 days ago", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_f420262c", "hypothesis": "Delta, American Airlines", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "gpt4_8e165409", "hypothesis": "22", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "gpt4_74aed68e", "hypothesis": "29 days", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "bcbe585f", "hypothesis": "4 weeks and 2 days ago (about 4.3 weeks)", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_21adecb5", "hypothesis": "6 months", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "5e1b23de", "hypothesis": "3 months ago", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_98f46fc6", "hypothesis": "Insufficient evidence to determine", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "gpt4_af6db32f", "hypothesis": "17 days", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "eac54adc", "hypothesis": "19 days", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_7ddcf75f", "hypothesis": "3 days ago", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_a2d1d1f6", "hypothesis": "3 days ago", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_85da3956", "hypothesis": "3 weeks", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_b0863698", "hypothesis": "7 days ago", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_68e94287", "hypothesis": "The vegan chili recipe post", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_e414231e", "hypothesis": "4 days", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_7ca326fa", "hypothesis": "Rachel first, Alex second, Emma third", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "gpt4_7bc6cf22", "hypothesis": "12 days", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "2ebe6c92", "hypothesis": "The Song of Achilles", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "gpt4_e061b84g", "hypothesis": "the company's annual charity soccer tournament", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "71017277", "hypothesis": "my aunt", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "b46e15ee", "hypothesis": "Walk for Hunger", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_d6585ce9", "hypothesis": "my sister", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "gpt4_1e4a8aec", "hypothesis": "planted 12 new tomato saplings", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_f420262d", "hypothesis": "Delta", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "gpt4_59149c78", "hypothesis": "the Metropolitan Museum of Art", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_e414231f", "hypothesis": "mountain bike", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "gpt4_4929293b", "hypothesis": "your cousin's wedding", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_468eb064", "hypothesis": "Emma", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_fa19884d", "hypothesis": "Last Friday was 2023/03/31. The evidence from that day only shows artist recommendations, not a single explicit artist you definitely started listening to. The artists suggested then were Miles Davis, John Coltrane, Charlie Parker, and Thelonious Monk, plus bluegrass acts like Bill Monroe and the Blue Grass Boys, Earl Scruggs, Flatt and Scruggs, Alison Krauss and Union Station, and The Infamous Stringdusters.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "9a707b82", "hypothesis": "croissants", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "eac54add", "hypothesis": "I can’t identify any business milestone from the evidence. The only milestone-like detail mentioned around the retrieved time is that you had lost about 2 pounds after replacing your bathroom scale, which is a personal health update, not a business milestone.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "4dfccbf8", "hypothesis": "Ukulele lessons with Rachel", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "0bc8ad93", "hypothesis": "with a friend", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "6e984302", "hypothesis": "a set of sculpting tools: a modeling tool set, a wire cutter, and a sculpting mat", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_8279ba03", "hypothesis": "I only have evidence of you buying a Samsung Galaxy S21 for about $800 at Best Buy, not a kitchen appliance 10 days before 2023/03/25.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "gpt4_b5700ca0", "hypothesis": "the Episcopal Church", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_68e94288", "hypothesis": "#PlankChallenge", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_2655b836", "hypothesis": "No retrieved evidence mentions your new car’s first service or any issue you had after it, so the first post-service issue cannot be determined from the available evidence.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "gpt4_2487a7cb", "hypothesis": "The 'Data Analysis using Python' webinar", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_76048e76", "hypothesis": "bike", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "gpt4_2312f94c", "hypothesis": "Samsung Galaxy S22", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "0bb5a684", "hypothesis": "7 days", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "08f4fc43", "hypothesis": "30 days", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "2c63a862", "hypothesis": "14 days", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_385a5000", "hypothesis": "Tomatoes", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "2a1811e2", "hypothesis": "11", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "bbf86515", "hypothesis": "4 days", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "gpt4_5dcc0aab", "hypothesis": "white Adidas sneakers", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "gpt4_0b2f1d21", "hypothesis": "The stand mixer malfunction happened first.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "f0853d11", "hypothesis": "14 days", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "gpt4_6ed717ea", "hypothesis": "the training pads for Luna", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_70e84552", "hypothesis": "Fixing the fence", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "a3838d2b", "hypothesis": "2", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "gpt4_93159ced", "hypothesis": "I can’t compute the exact amount from the evidence: it shows you’ve been at NovaTech for about 4 years and 3 months, and that you started working in the field when you graduated, but it doesn’t give your graduation date or total career length before NovaTech.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "gpt4_2d58bcd6", "hypothesis": "The Hate U Give", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_65aabe59", "hypothesis": "Can't tell from the evidence.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "982b5123", "hypothesis": "3 months", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "b9cfe692", "hypothesis": "I have evidence only for one of the two books: \"The Seven Husbands of Evelyn Hugo\" took you two and a half weeks to finish. There’s no evidence here for how long \"The Nightingale\" took, so I can’t compute a combined total from the provided memory.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "gpt4_4edbafa2", "hypothesis": "2023-06-03", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "c8090214", "hypothesis": "7 days", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_483dd43c", "hypothesis": "Game of Thrones", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "e4e14d04", "hypothesis": "cannot be determined from the provided evidence", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "c9f37c46", "hypothesis": "unresolved", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "gpt4_2c50253f", "hypothesis": "I wake up at 6:30 AM on Tuesdays and Thursdays.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "dcfa8644", "hypothesis": "14 days", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_b4a80587", "hypothesis": "the arrival of the new prime lens", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "gpt4_9a159967", "hypothesis": "Southwest Airlines and American Airlines (tie)", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "cc6d1ec1", "hypothesis": "about two months", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_8c8961ae", "hypothesis": "the solo trip to Thailand", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "gpt4_d9af6064", "hypothesis": "new router", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_7de946e7", "hypothesis": "persistent cough", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "d01c6aa8", "hypothesis": "Cannot be determined from the retrieved evidence.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "993da5e2", "hypothesis": "I can only infer that you rearranged the furniture about three weeks before 2023/05/26; there’s no evidence here about when you started using the new area rug, so the duration before the rearrangement can’t be determined from the provided memory.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "a3045048", "hypothesis": "30 days", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "gpt4_d31cdae3", "hypothesis": "The family road trip across the American Southwest.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_cd90e484", "hypothesis": "Insufficient evidence to determine exactly", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "gpt4_88806d6e", "hypothesis": "Can't tell from the evidence.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "gpt4_4cd9eba1", "hypothesis": "1 week", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "gpt4_93f6379c", "hypothesis": "Page Turners", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "b29f3365", "hypothesis": "six weeks", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "gpt4_2f56ae70", "hypothesis": "There’s no evidence here of you starting any streaming service, so I can’t identify a most recent one from the provided memory.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "6613b389", "hypothesis": "2 months", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_78cf46a3", "hypothesis": "Receiving the new phone case happened first.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_0a05b494", "hypothesis": "the tourist from Australia", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "gpt4_1a1dc16d", "hypothesis": "The pride parade", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "gpt4_2f584639", "hypothesis": "The necklace for your sister.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "gpt4_213fd887", "hypothesis": "the volleyball league", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "gpt4_5438fa52", "hypothesis": "the start of your Spanish classes", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "gpt4_c27434e8", "hypothesis": "Japanese Zero fighter plane model", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_fe651585", "hypothesis": "Rachel", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "8c18457d", "hypothesis": "21 days", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "gpt4_70e84552_abs", "hypothesis": "Fixing the fence", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "gpt4_93159ced_abs", "hypothesis": "The evidence doesn’t support a current job at Google. On 2023/05/30, you were described as working at NovaTech, and while there’s a truncated note saying you’d been in software engineering before mentioning your current job, it doesn’t include the duration. So I can’t compute how long you’d been working before starting a job at Google from the provided evidence.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "982b5123_abs", "hypothesis": "There’s no evidence here of you booking an Airbnb in Sacramento. The only Airbnb mention is for San Francisco, tied to your stay in Haight-Ashbury on 2023/05/27.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "c8090214_abs", "hypothesis": "I attended the Holiday Market 7 days before Black Friday, but there’s no evidence here that you bought an iPad, so the day difference to an iPad purchase can’t be determined from the provided information.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_c27434e8_abs", "hypothesis": "Insufficient evidence to determine", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "gpt4_fe651585_abs", "hypothesis": "Alex", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "6a1eabeb", "hypothesis": "25:50", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "6aeb4375", "hypothesis": "2", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "830ce83f", "hypothesis": "Chicago", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "852ce960", "hypothesis": "$350,000", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "945e3d21", "hypothesis": "three times a week", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "d7c942c3", "hypothesis": "Yes — your mom is using the same grocery list app as you.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "71315a70", "hypothesis": "10-12 hours", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "89941a93", "hypothesis": "1", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "ce6d2d27", "hypothesis": "Friday", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "9ea5eabc", "hypothesis": "Paris", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "07741c44", "hypothesis": "shoe rack", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "a1eacc2a", "hypothesis": "7", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "184da446", "hypothesis": "220 pages", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "031748ae", "hypothesis": "4 when you started; 5 now", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "4d6b87c8", "hypothesis": "25", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "0f05491a", "hypothesis": "120 stars", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "08e075c7", "hypothesis": "I can’t determine that from the provided evidence. The memory snippets only mention your bedtime routine, reading, and using a massage roller, with no evidence about when you started using a Fitbit Charge 3.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "f9e8c073", "hypothesis": "5 sessions", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "41698283", "hypothesis": "70-200mm zoom lens", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "2698e78f", "hypothesis": "Not specified; the memory only says to review it regularly with Dr. Smith.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "b6019101", "hypothesis": "5", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "45dc21b6", "hypothesis": "2", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "5a4f22c0", "hypothesis": "TechCorp", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "6071bd76", "hypothesis": "less", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "e493bb7c", "hypothesis": "above my bed in my bedroom", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "618f13b2", "hypothesis": "6", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "72e3ee87", "hypothesis": "50", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "c4ea545c", "hypothesis": "Insufficient evidence", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "01493427", "hypothesis": "17", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "6a27ffc2", "hypothesis": "30", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "2133c1b5", "hypothesis": "3 months", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "18bc8abd", "hypothesis": "Kansas City Masterpiece BBQ sauce", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "db467c8c", "hypothesis": "about nine months", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "7a87bd0c", "hypothesis": "4 weeks", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "e61a7584", "hypothesis": "about 9 months", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "1cea1afa", "hypothesis": "600", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "ed4ddc30", "hypothesis": "20 dozen eggs", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "8fb83627", "hypothesis": "5", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "b01defab", "hypothesis": "Yes", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "22d2cb42", "hypothesis": "the music shop on Main St", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "0e4e4c46", "hypothesis": "132 points", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "4b24c848", "hypothesis": "5", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "7e974930", "hypothesis": "Your most recent Downtown Farmers Market visit was on 2023/09/30, but the exact earnings amount isn’t shown in the retrieved evidence.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "603deb26", "hypothesis": "10", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "59524333", "hypothesis": "6:00 pm", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "5831f84d", "hypothesis": "10", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "eace081b", "hypothesis": "Oahu", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "affe2881", "hypothesis": "32", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "50635ada", "hypothesis": "Premier Gold status", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "e66b632c", "hypothesis": "26 minutes and 30 seconds", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "0ddfec37", "hypothesis": "Cannot be determined from available evidence", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "f685340e", "hypothesis": "Previously, every other week. Now, not explicitly stated.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "cc5ded98", "hypothesis": "an hour each day", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "dfde3500", "hypothesis": "Wednesday", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "69fee5aa", "hypothesis": "37", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "7401057b", "hypothesis": "1 free night's stay", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "cf22b7bf", "hypothesis": "10 pounds", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "a2f3aa27", "hypothesis": "close to 1300 followers", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "c7dc5443", "hypothesis": "5-2", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "06db6396", "hypothesis": "5", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "3ba21379", "hypothesis": "Ford F-150 pickup truck model", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "9bbe84a2", "hypothesis": "level 100 before the end of the year", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "10e09553", "hypothesis": "7", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "dad224aa", "hypothesis": "7:30 am", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "yes"}}
-{"question_id": "ba61f0b9", "hypothesis": "6", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "42ec0761", "hypothesis": "Yes, you have a spare screwdriver.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "5c40ec5b", "hypothesis": "2", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "c6853660", "hypothesis": "increase", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "26bdc477", "hypothesis": "5", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "0977f2af", "hypothesis": "I can’t tell from the retrieved evidence.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "6aeb4375_abs", "hypothesis": "{\"answer\":0}", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "031748ae_abs", "hypothesis": "4", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "2698e78f_abs", "hypothesis": "No evidence provided states how often you see Dr. Johnson.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "2133c1b5_abs", "hypothesis": "About 7 months. The evidence says it had been 1 month in your new studio apartment in Harajuku on 2023/04/11, which makes it about 7 months by 2023/10/15. The memory mentions Harajuku, not Shinjuku.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "0ddfec37_abs", "hypothesis": "0", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "f685340e_abs", "hypothesis": "Every other week.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No"}}
-{"question_id": "89941a94", "hypothesis": "Yes — a road bike.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "07741c45", "hypothesis": "shoe rack", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "7161e7e2", "hypothesis": "8 am - 4 pm (Day Shift)", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
-{"question_id": "c4f10528", "hypothesis": "Miss Bee Providore", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "89527b6b", "hypothesis": "I can only recover that the dinosaur-book conversation was on 2023/05/23 and the visible excerpt includes a T-Rex with a green scaly body; the Plesiosaur image description and its body color are not present in the provided evidence.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "e9327a54", "hypothesis": "I don’t have the dessert shop’s name in the retrieved memory.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": false, "raw_response": "No."}}
-{"question_id": "4c36ccef", "hypothesis": "Roscioli", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes"}}
-{"question_id": "6ae235be", "hypothesis": "Atmospheric distillation, fluid catalytic cracking (FCC), alkylation, and hydrotreating.", "autoeval_label": {"model": "gpt-4o-2024-08-06", "label": true, "raw_response": "Yes."}}
diff --git a/results/judge_gpt4o_alias_vectorengine.jsonl.summary.json b/results/judge_gpt4o_alias_vectorengine.jsonl.summary.json
deleted file mode 100644
index 779910f69cdc56c7911ddd30ed8d80ba696e09d9..0000000000000000000000000000000000000000
--- a/results/judge_gpt4o_alias_vectorengine.jsonl.summary.json
+++ /dev/null
@@ -1,36 +0,0 @@
-{
- "metric_model_alias": "gpt-4o",
- "metric_model": "gpt-4o-2024-08-06",
- "base_url": "OpenAI-compatible judge endpoint",
- "hyp_file": "/runs/lme_s500_frozen_baseline38_gpt54_full10_20260525_085540/off/predictions.jsonl",
- "ref_file": "/data/longmemeval_s_cleaned.json",
- "result_file": "/runs/lme_s500_frozen_baseline38_gpt54_full10_20260525_085540/off/judge_gpt4o_alias_vectorengine.jsonl",
- "evaluated_count": 500,
- "accuracy": 0.62,
- "by_question_type": {
- "knowledge-update": {
- "accuracy": 0.7051,
- "count": 78
- },
- "multi-session": {
- "accuracy": 0.3985,
- "count": 133
- },
- "single-session-assistant": {
- "accuracy": 0.7857,
- "count": 56
- },
- "single-session-preference": {
- "accuracy": 0.5667,
- "count": 30
- },
- "single-session-user": {
- "accuracy": 0.8143,
- "count": 70
- },
- "temporal-reasoning": {
- "accuracy": 0.6316,
- "count": 133
- }
- }
-}
diff --git a/results/latest_benchmark.json b/results/latest_benchmark.json
new file mode 100644
index 0000000000000000000000000000000000000000..4856e1ca792379086c4b555ae1c8f83a2527263c
--- /dev/null
+++ b/results/latest_benchmark.json
@@ -0,0 +1,45 @@
+{
+ "schema_version": "tmcra.benchmark-result.2",
+ "status": "latest",
+ "reported_at": "2026-07-18",
+ "system": "TMCRA",
+ "benchmark": "LongMemEval S500",
+ "metric": "accuracy",
+ "correct": 411,
+ "total": 500,
+ "score_percent": 82.2,
+ "by_question_type": {
+ "knowledge-update": {
+ "correct": 71,
+ "total": 78,
+ "score_percent": 91.0
+ },
+ "multi-session": {
+ "correct": 90,
+ "total": 133,
+ "score_percent": 67.7
+ },
+ "single-session-assistant": {
+ "correct": 55,
+ "total": 56,
+ "score_percent": 98.2
+ },
+ "single-session-preference": {
+ "correct": 27,
+ "total": 30,
+ "score_percent": 90.0
+ },
+ "single-session-user": {
+ "correct": 67,
+ "total": 70,
+ "score_percent": 95.7
+ },
+ "temporal-reasoning": {
+ "correct": 101,
+ "total": 133,
+ "score_percent": 75.9
+ }
+ },
+ "scope_isolation": "one independent memory scope per LongMemEval QID",
+ "reproduction_entrypoint": "benchmarks/longmemeval"
+}
diff --git a/results/lme_s500_frozen_baseline38_full10_20260525_results.tar.gz b/results/lme_s500_frozen_baseline38_full10_20260525_results.tar.gz
deleted file mode 100644
index be4e6e78265d1125b5cf8e24caae1b8997e7c3fe..0000000000000000000000000000000000000000
--- a/results/lme_s500_frozen_baseline38_full10_20260525_results.tar.gz
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:e88380901df26ac2374524fb812f6405d994775d1163db228fd9aedf1e56acea
-size 79506510
diff --git a/results/predictions.jsonl b/results/predictions.jsonl
deleted file mode 100644
index 74b086391471ffa2f7c4f1b3dff776af376525b4..0000000000000000000000000000000000000000
--- a/results/predictions.jsonl
+++ /dev/null
@@ -1,500 +0,0 @@
-{"question_id": "7e00a6cb", "hypothesis": "International Budget Hostel"}
-{"question_id": "1903aded", "hypothesis": "I can’t recover the 7th job from the provided memory evidence. The retrieved notes only show unrelated lists, like living room color schemes, dessert ideas, and sales tax questions, not the earlier work-from-home jobs for seniors list."}
-{"question_id": "ceb54acb", "hypothesis": "Sexual fixations, Problematic sexual behaviors, Sexual impulsivity, Compulsive sexuality"}
-{"question_id": "f523d9fe", "hypothesis": "I can’t tell from the available memory."}
-{"question_id": "0e5e2d1a", "hypothesis": "38 subjects"}
-{"question_id": "fea54f57", "hypothesis": "Evolution"}
-{"question_id": "cc539528", "hypothesis": "Ruby, Python, or PHP"}
-{"question_id": "dc439ea3", "hypothesis": "Hoop Dance"}
-{"question_id": "18dcd5a5", "hypothesis": "4"}
-{"question_id": "488d3006", "hypothesis": "GR-90"}
-{"question_id": "58470ed2", "hypothesis": "The Library is a sphere whose exact center is any one of its hexagons and whose circumference is inaccessible."}
-{"question_id": "8cf51dda", "hypothesis": "1. To identify molecular subtypes of endometrial cancer using a combination of genomic and transcriptomic approaches. 2. To investigate the clinical and biological significance of the identified molecular subtypes, including their association with patient outcomes. 3. To develop biomarkers for the early detection and prognosis of endometrial cancer based on the identified molecular subtypes."}
-{"question_id": "1d4da289", "hypothesis": "biometric authentication and one-time passwords (OTP)"}
-{"question_id": "8464fc84", "hypothesis": "Roscioli"}
-{"question_id": "8aef76bc", "hypothesis": "Mod Podge or another sealant"}
-{"question_id": "71a3fd6b", "hypothesis": "+49 (0) 62 32 / 14 23 - 0"}
-{"question_id": "2bf43736", "hypothesis": "Chapter 4 of Book 1, titled \"Vocal Prayer and Meditation."}
-{"question_id": "70b3e69b", "hypothesis": "Marina Rossell"}
-{"question_id": "8752c811", "hypothesis": "The 27th parameter was Perspective."}
-{"question_id": "3249768e", "hypothesis": "Absinthe"}
-{"question_id": "1b9b7252", "hypothesis": "Mindful.org"}
-{"question_id": "1568498a", "hypothesis": "29. Rd3"}
-{"question_id": "6222b6eb", "hypothesis": "6S"}
-{"question_id": "e8a79c70", "hypothesis": "2-3 eggs"}
-{"question_id": "d596882b", "hypothesis": "By Chloe"}
-{"question_id": "e3fc4d6e", "hypothesis": "Dr. Arati Prabhakar"}
-{"question_id": "51b23612", "hypothesis": "Nu, pogodi!"}
-{"question_id": "3e321797", "hypothesis": "10 minutes"}
-{"question_id": "e982271f", "hypothesis": "Revolution Hall"}
-{"question_id": "352ab8bd", "hypothesis": "approximately 20%"}
-{"question_id": "fca762bc", "hypothesis": "Memrise"}
-{"question_id": "7a8d0b71", "hypothesis": "In the visible remembered campaign-plan excerpt, I can confirm we discussed the DHL Wellness Retreats email marketing campaign, but the specific influencer marketing allocation amount is not shown in the evidence provided."}
-{"question_id": "a40e080f", "hypothesis": "Patagonia and Southwest Airlines"}
-{"question_id": "8b9d4367", "hypothesis": "Jaipur Rugs"}
-{"question_id": "5809eb10", "hypothesis": "2015"}
-{"question_id": "41275add", "hypothesis": "How to Sit Properly at a Desk to Avoid Back Pain (Mayo Clinic)"}
-{"question_id": "4388e9dd", "hypothesis": "an untidy, stained white shirt"}
-{"question_id": "4baee567", "hypothesis": "12"}
-{"question_id": "561fabcd", "hypothesis": "Contaminated Colossus"}
-{"question_id": "b759caee", "hypothesis": "@jessica_poole_jewellery"}
-{"question_id": "ac031881", "hypothesis": "squares"}
-{"question_id": "28bcfaac", "hypothesis": "MusicTheory.net"}
-{"question_id": "16c90bf4", "hypothesis": "A pilsner or lager."}
-{"question_id": "c8f1aeed", "hypothesis": "Pennsylvania"}
-{"question_id": "eaca4986", "hypothesis": "C D E F G A B A G F E D C"}
-{"question_id": "c7cf7dfd", "hypothesis": "Nostalgia"}
-{"question_id": "e48988bc", "hypothesis": "Patagonia"}
-{"question_id": "1de5cff2", "hypothesis": "Veja"}
-{"question_id": "65240037", "hypothesis": "1:10 ratio"}
-{"question_id": "778164c6", "hypothesis": "The evidence only shows that we discussed popular Caribbean dishes featuring snapper, but it does not include the actual Jamaican fruit-based dish name."}
-{"question_id": "e47becba", "hypothesis": "Business Administration"}
-{"question_id": "118b2229", "hypothesis": "45 minutes each way"}
-{"question_id": "51a45a95", "hypothesis": "The evidence does not say where you redeemed the $5 coffee creamer coupon."}
-{"question_id": "58bf7951", "hypothesis": "The Glass Menagerie"}
-{"question_id": "1e043500", "hypothesis": "Summer Vibes"}
-{"question_id": "c5e8278d", "hypothesis": "Johnson"}
-{"question_id": "6ade9755", "hypothesis": "Serenity Yoga"}
-{"question_id": "6f9b354f", "hypothesis": "a lighter shade of gray"}
-{"question_id": "58ef2f1c", "hypothesis": "back on Valentine's Day"}
-{"question_id": "f8c5f88b", "hypothesis": "The retrieved evidence does not say where you bought the new tennis racket from."}
-{"question_id": "5d3d2817", "hypothesis": "marketing specialist at a small startup"}
-{"question_id": "7527f7e2", "hypothesis": "$800"}
-{"question_id": "c960da58", "hypothesis": "20"}
-{"question_id": "3b6f954b", "hypothesis": "University of Melbourne"}
-{"question_id": "726462e0", "hypothesis": "10%"}
-{"question_id": "94f70d80", "hypothesis": "The retrieved evidence does not state any assembly time for an IKEA bookshelf. It only mentions plans to visit IKEA to look at coffee tables and read reviews, so the duration cannot be determined from this evidence."}
-{"question_id": "66f24dbb", "hypothesis": "No sister gift is shown in memory; the only recorded birthday gift purchase was a customized phone case with his favorite game's logo for your brother."}
-{"question_id": "ad7109d1", "hypothesis": "The retrieved evidence does not specify the speed of your new internet plan."}
-{"question_id": "af8d2e46", "hypothesis": "7"}
-{"question_id": "dccbc061", "hypothesis": "staunch atheist"}
-{"question_id": "c8c3f81d", "hypothesis": "Nike"}
-{"question_id": "8ebdbe50", "hypothesis": "Data Science certification"}
-{"question_id": "6b168ec8", "hypothesis": "3"}
-{"question_id": "75499fd8", "hypothesis": "Golden Retriever"}
-{"question_id": "21436231", "hypothesis": "Not stated"}
-{"question_id": "95bcc1c8", "hypothesis": "10"}
-{"question_id": "0862e8bf", "hypothesis": "Luna"}
-{"question_id": "853b0a1d", "hypothesis": "I can tell it was a gift from your grandma, but the evidence here doesn’t include your age when she gave you the silver necklace."}
-{"question_id": "a06e4cfe", "hypothesis": "3:1"}
-{"question_id": "37d43f65", "hypothesis": "16GB"}
-{"question_id": "b86304ba", "hypothesis": "Triple what you paid for it."}
-{"question_id": "d52b4f67", "hypothesis": "Grand Ballroom"}
-{"question_id": "25e5aa4f", "hypothesis": "UCLA"}
-{"question_id": "caf9ead2", "hypothesis": "around 5 hours"}
-{"question_id": "8550ddae", "hypothesis": "Lavender Dream cocktail"}
-{"question_id": "60d45044", "hypothesis": "Japanese short-grain rice"}
-{"question_id": "3f1e9474", "hypothesis": "Sarah"}
-{"question_id": "86b68151", "hypothesis": "IKEA"}
-{"question_id": "577d4d32", "hypothesis": "7 pm"}
-{"question_id": "ec81a493", "hypothesis": "500 copies"}
-{"question_id": "15745da0", "hypothesis": "about three months"}
-{"question_id": "e01b8e2f", "hypothesis": "Hawaii"}
-{"question_id": "bc8a6e93", "hypothesis": "lemon blueberry cake"}
-{"question_id": "ccb36322", "hypothesis": "There’s no evidence here of any music streaming service you’ve been using lately; the only services mentioned are photo album services like Shutterfly, Snapfish, and Mixbook."}
-{"question_id": "001be529", "hypothesis": "Your asylum application had been approved by May 28, 2023, and there was a waiting period, but the evidence does not state how long you waited for the decision."}
-{"question_id": "b320f3f8", "hypothesis": "a rare blue Snaggletooth action figure"}
-{"question_id": "19b5f2b3", "hypothesis": "The evidence only shows that you were in Japan/Kyoto and planning to travel elsewhere in Asia afterward, but it does not state how long you were in Japan."}
-{"question_id": "4fd1909e", "hypothesis": "Xfinity Center"}
-{"question_id": "545bd2b5", "hypothesis": "2 hours per day"}
-{"question_id": "8a137a7f", "hypothesis": "Philips LED bulb"}
-{"question_id": "76d63226", "hypothesis": "55-inch"}
-{"question_id": "86f00804", "hypothesis": "The Seven Husbands of Evelyn Hugo"}
-{"question_id": "8e9d538c", "hypothesis": "17 skeins"}
-{"question_id": "311778f1", "hypothesis": "around 10 hours"}
-{"question_id": "c19f7a0b", "hypothesis": "around 6:30 pm"}
-{"question_id": "4100d0a0", "hypothesis": "Irish and Italian"}
-{"question_id": "29f2956b", "hypothesis": "30 minutes daily"}
-{"question_id": "1faac195", "hypothesis": "Denver"}
-{"question_id": "faba32e5", "hypothesis": "24 hours"}
-{"question_id": "f4f1d8a4", "hypothesis": "my sister"}
-{"question_id": "c14c00dd", "hypothesis": "Trader Joe's"}
-{"question_id": "36580ce8", "hypothesis": "The evidence only says you’d been dealing with unspecified health issues for a few months; it does not identify any specific issue you initially thought was just a cold."}
-{"question_id": "3d86fd0a", "hypothesis": "Information unavailable"}
-{"question_id": "a82c026e", "hypothesis": "Dark Souls 3 DLC"}
-{"question_id": "0862e8bf_abs", "hypothesis": "I can’t determine your hamster’s name from the provided evidence because the memories only mention your cat and unrelated coding discussion, not any hamster."}
-{"question_id": "15745da0_abs", "hypothesis": "No evidence says how long you've been collecting vintage films."}
-{"question_id": "bc8a6e93_abs", "hypothesis": "I don't know"}
-{"question_id": "19b5f2b3_abs", "hypothesis": "The available evidence doesn’t mention a Korea trip or its duration, so I can’t determine how long you were in Korea."}
-{"question_id": "29f2956b_abs", "hypothesis": "I only have evidence that you practice guitar for 30 minutes daily; there’s no evidence here about a daily violin practice time."}
-{"question_id": "f4f1d8a4_abs", "hypothesis": "I only remember a new stand mixer as a birthday gift from your sister, not your dad."}
-{"question_id": "0a995998", "hypothesis": "2"}
-{"question_id": "6d550036", "hypothesis": "1"}
-{"question_id": "gpt4_59c863d7", "hypothesis": "4"}
-{"question_id": "b5ef892d", "hypothesis": "5 days"}
-{"question_id": "e831120c", "hypothesis": "3.5 weeks"}
-{"question_id": "3a704032", "hypothesis": "3"}
-{"question_id": "gpt4_d84a3211", "hypothesis": "I can identify bike-related activity this year, like getting bike lights installed, but there are no explicit dollar amounts for any bike-related purchases or services in the evidence, so I can’t compute a total spent."}
-{"question_id": "aae3761f", "hypothesis": "10 hours"}
-{"question_id": "gpt4_f2262a51", "hypothesis": "1"}
-{"question_id": "dd2973ad", "hypothesis": "I can only narrow it down indirectly: your doctor’s appointment was at 10 AM on Thursday 2023-05-18, so the day before was Wednesday 2023-05-17. The only bedtime evidence says you went to bed at 2 AM on last Wednesday relative to 2023-05-29, which refers to 2023-05-24, not 2023-05-17. So the closest bedtime mentioned is 2 AM, but it does not match the day before that appointment."}
-{"question_id": "c4a1ceb8", "hypothesis": "3"}
-{"question_id": "gpt4_a56e767c", "hypothesis": "3"}
-{"question_id": "6cb6f249", "hypothesis": "17 days"}
-{"question_id": "46a3abf7", "hypothesis": "3"}
-{"question_id": "36b9f61e", "hypothesis": "2000"}
-{"question_id": "28dc39ac", "hypothesis": "29 hours"}
-{"question_id": "gpt4_2f8be40d", "hypothesis": "2"}
-{"question_id": "2e6d26dc", "hypothesis": "3"}
-{"question_id": "gpt4_15e38248", "hypothesis": "2"}
-{"question_id": "88432d0a", "hypothesis": "4"}
-{"question_id": "80ec1f4f", "hypothesis": "1"}
-{"question_id": "d23cf73b", "hypothesis": "2"}
-{"question_id": "gpt4_7fce9456", "hypothesis": "2"}
-{"question_id": "d682f1a2", "hypothesis": "2"}
-{"question_id": "7024f17c", "hypothesis": "0.5 hours"}
-{"question_id": "gpt4_5501fe77", "hypothesis": "TikTok"}
-{"question_id": "gpt4_2ba83207", "hypothesis": "Trader Joe's"}
-{"question_id": "2318644b", "hypothesis": "Tokyo is the only place with a numeric nightly accommodation cost in the evidence: about $30/night. For Hawaii, the evidence only says you had a luxurious resort stay in Maui, which suggests it was more expensive but does not give a dollar amount, so the exact per-night difference cannot be calculated from the evidence."}
-{"question_id": "2ce6a0f2", "hypothesis": "1"}
-{"question_id": "gpt4_d12ceb0e", "hypothesis": "Insufficient evidence to compute the average age of you, your parents, and your grandparents."}
-{"question_id": "00ca467f", "hypothesis": "2"}
-{"question_id": "b3c15d39", "hypothesis": "5 days"}
-{"question_id": "gpt4_31ff4165", "hypothesis": "4"}
-{"question_id": "eeda8a6d", "hypothesis": "17"}
-{"question_id": "2788b940", "hypothesis": "2"}
-{"question_id": "60bf93ed", "hypothesis": "It arrived on 1/20, but I don’t see the purchase date in the evidence, so I can’t calculate the number of days exactly."}
-{"question_id": "9d25d4e0", "hypothesis": "2"}
-{"question_id": "129d1232", "hypothesis": "600 dollars"}
-{"question_id": "60472f9c", "hypothesis": "1"}
-{"question_id": "gpt4_194be4b3", "hypothesis": "3"}
-{"question_id": "a9f6b44c", "hypothesis": "2"}
-{"question_id": "d851d5ba", "hypothesis": "At least 2,250 USD"}
-{"question_id": "5a7937c8", "hypothesis": "2"}
-{"question_id": "gpt4_ab202e7f", "hypothesis": "2"}
-{"question_id": "gpt4_e05b82a6", "hypothesis": "7"}
-{"question_id": "gpt4_731e37d7", "hypothesis": "$200"}
-{"question_id": "edced276", "hypothesis": "8 days"}
-{"question_id": "10d9b85a", "hypothesis": "2"}
-{"question_id": "e3038f8c", "hypothesis": "87"}
-{"question_id": "2b8f3739", "hypothesis": "$495"}
-{"question_id": "1a8a66a6", "hypothesis": "2"}
-{"question_id": "c2ac3c61", "hypothesis": "5"}
-{"question_id": "bf659f65", "hypothesis": "1"}
-{"question_id": "gpt4_372c3eed", "hypothesis": "8 years"}
-{"question_id": "gpt4_2f91af09", "hypothesis": "6"}
-{"question_id": "81507db6", "hypothesis": "3"}
-{"question_id": "88432d0a_abs", "hypothesis": "0"}
-{"question_id": "80ec1f4f_abs", "hypothesis": "0"}
-{"question_id": "eeda8a6d_abs", "hypothesis": "not evidenced in the retrieved memory"}
-{"question_id": "60bf93ed_abs", "hypothesis": "70 days"}
-{"question_id": "edced276_abs", "hypothesis": "I can only confirm a Hawaii trip, and no number of days is given for it. There’s also no evidence here of a Seattle trip duration, so the total can’t be determined from the provided evidence."}
-{"question_id": "gpt4_372c3eed_abs", "hypothesis": "8 years documented; no evidence of a completed Master's degree"}
-{"question_id": "8a2466db", "hypothesis": "Sorry, I don’t know of any specific resources from memory."}
-{"question_id": "06878be2", "hypothesis": "A comfortable, durable Sony-compatible camera bag, a Godox V1 hard case or Think Tank Photo Flash Pouch, and a lens cleaning kit with a microfiber cloth, blower, and lens cleaning solution would complement your Sony A7R IV setup."}
-{"question_id": "75832dbd", "hypothesis": "Recent publications or conferences on the impact of mega-churches on Baptist church vitality, Baptist identity and mission, and practical collaboration models among smaller Baptist churches—especially case-study-focused work on regional or denominational cooperation."}
-{"question_id": "0edc2aef", "hypothesis": "Look for a Miami hotel with a balcony hot tub and a standout view, similar to your preferred room at The Edgewater Hotel."}
-{"question_id": "35a27287", "hypothesis": "French- or Spanish-focused language exchange meetups, conversation groups, or cultural events nearby would fit you best; check Meetup, Facebook Groups, local universities, or language schools this weekend."}
-{"question_id": "32260d93", "hypothesis": "Mike Birbiglia: My Girlfriend's Boyfriend"}
-{"question_id": "195a1a1b", "hypothesis": "Try a calm wind-down routine: read a non-stimulating book, do a guided sleep meditation or body scan, practice deep breathing, do gentle yoga or stretching, listen to soothing music or nature sounds, and take a 30-minute break from screens before bed."}
-{"question_id": "afdc33df", "hypothesis": "Regularly clean your granite countertop, use coasters or pads under pots and utensils, always use a cutting board, and check the faucet/aerator for moisture or buildup so the sink area stays cleaner."}
-{"question_id": "caf03d32", "hypothesis": "Try more plant-based slow cooker recipes like lentil curry, roasted vegetable stew, black bean chili, eggplant parmesan, vegan chili, or a quinoa and black bean bowl."}
-{"question_id": "54026fce", "hypothesis": "Try suggesting virtual coffee breaks in a collaborative way: ask open-ended questions about staying connected, share that you miss casual chats, propose a weekly virtual coffee break, and invite your team’s feedback before deciding."}
-{"question_id": "06f04340", "hypothesis": "Serve a hearty quinoa salad with your roasted homegrown vegetables over mixed greens, finished with vinaigrette; roast the vegetables first, then toss with the greens and quinoa."}
-{"question_id": "6b7dfb22", "hypothesis": "Study real flowers from life or photos, browse Instagram and art communities for ideas, experiment with new techniques like palette-knife texture, and take breaks while tracking small wins to help inspiration return."}
-{"question_id": "1a1907b4", "hypothesis": "Try a Pimm's Cup with a Twist, especially with Ruby Red Grapefruit simple syrup."}
-{"question_id": "09d032c9", "hypothesis": "A compact portable power bank could help as a backup."}
-{"question_id": "38146c39", "hypothesis": "Try adding warm spices, nuts like walnuts or pecans, a little citrus zest, or a touch of espresso or coffee for extra flavor."}
-{"question_id": "d24813b1", "hypothesis": "Bake chocolate chip cookies or a lemon poppyseed cake; both fit your recent baking interests, and the lemon poppyseed cake worked well for a colleague event before."}
-{"question_id": "57f827a0", "hypothesis": "No direct memory-backed evidence shows you were planning to rearrange your bedroom furniture this weekend."}
-{"question_id": "95228167", "hypothesis": "Try several guitars and focus on comfort, playability, build quality, tuning stability, and the sound you like best."}
-{"question_id": "505af2f5", "hypothesis": "Try a homemade vanilla-cinnamon creamer with unsweetened almond or oat milk, a little monk fruit or stevia instead of honey, and optional coconut or hazelnut extract."}
-{"question_id": "75f70248", "hypothesis": "Possibly yes—the prior memory suggests your living room had a dust issue and a shedding cat, so dust or pet hair there could be contributing to your sneezing."}
-{"question_id": "d6233ab6", "hypothesis": "Yes—if you’re feeling nostalgic and want to reconnect with old high school friends, attending your reunion sounds like a good idea."}
-{"question_id": "1da05512", "hypothesis": "Buy now if your storage issue is already a problem and a beginner-friendly 2-bay model fits your budget; wait only if you expect much larger future storage needs or still need to compare options."}
-{"question_id": "fca70973", "hypothesis": "Try Universal Studios Hollywood’s VIP Experience with the Gourmet Buffet and Behind-the-Scenes Tour as the top pick; Disneyland Resort Halloween dining or Knott’s Scary Farm Boo-fet would also fit your immersive, exclusive theme park weekend vibe."}
-{"question_id": "b6025781", "hypothesis": "Try herb-rosemary roasted sweet potatoes, quinoa and roasted vegetable bowls with salmon, tofu, or lentil bolognese, and a batch of homemade granola for snacks."}
-{"question_id": "a89d7624", "hypothesis": "Visit Red Rocks Park and Amphitheatre and check out other live music spots like The Ogden Theatre, The Bluebird Theater, The Fillmore Auditorium, and Larimer Lounge."}
-{"question_id": "b0479f84", "hypothesis": "Try Planet Earth, Blue Planet, Dynasties, Chasing Coral, and March of the Penguins—especially the nature and wildlife ones since that’s what you were most interested in."}
-{"question_id": "1d4e3b97", "hypothesis": "Yes — the recent upgrade to your bike’s chain and cassette could be why it’s performing better."}
-{"question_id": "07b6f563", "hypothesis": "An iPhone 13 Pro-compatible wallet case like the OtterBox wallet case would be a useful accessory; I’d skip suggesting a power bank since you already have one that’s working well."}
-{"question_id": "1c0ddc50", "hypothesis": "Listen to history and science podcasts during your 40-minute commute—try starter episodes from Hardcore History, Lore, The Dollop, StarTalk Radio, or Radiolab."}
-{"question_id": "0a34ad58", "hypothesis": "Since you already have a Suica card, use it for trains and subways to make getting around easier. If you’re heading out from Shinjuku, plan a little extra time, especially for early or busy destinations like Tsukiji, where it’s best to leave very early and expect crowds. Keep some yen/cash with you because some places, especially around markets, may not take cards, and if needed you can exchange money near Shinjuku."}
-{"question_id": "d3ab962e", "hypothesis": "8 miles"}
-{"question_id": "2311e44b", "hypothesis": "The Nightingale' is 440 pages long, but I don’t have a numeric page-progress value for how far you’ve already read, so the exact pages left can’t be calculated from the evidence."}
-{"question_id": "cc06de0d", "hypothesis": "I can’t compute the difference from the provided evidence because it gives your daily train fare as $6, but it doesn’t include the taxi ride cost."}
-{"question_id": "a11281a2", "hypothesis": "about 100 followers"}
-{"question_id": "4f54b7c9", "hypothesis": "5"}
-{"question_id": "85fa3a3f", "hypothesis": "50"}
-{"question_id": "9aaed6a3", "hypothesis": "$0.75"}
-{"question_id": "1f2b8d4f", "hypothesis": "I can’t determine an exact price difference from the evidence. The price of your luxury boots isn’t given, and no specific similar pair found at a budget store has a stated price. The closest budget-store-style prices mentioned were about $20-$50 for affordable ankle boots, but that still isn’t enough to calculate the difference."}
-{"question_id": "e6041065", "hypothesis": "40%"}
-{"question_id": "51c32626", "hypothesis": "No explicit submission date was given; it was only mentioned by 2023/05/22 as a past event."}
-{"question_id": "d905b33f", "hypothesis": "20%"}
-{"question_id": "7405e8b1", "hypothesis": "HelloFresh: 40% on your first order. For UberEats, the evidence only says you got 20% off an order last week, not that it was your first order."}
-{"question_id": "f35224e0", "hypothesis": "At least 12 episodes"}
-{"question_id": "6456829e", "hypothesis": "5 tomato plants and 3 cucumber plants"}
-{"question_id": "a4996e51", "hypothesis": "Up to 50 hours/week."}
-{"question_id": "3c1045c8", "hypothesis": "unavailable"}
-{"question_id": "60036106", "hypothesis": "12,000 people"}
-{"question_id": "681a1674", "hypothesis": "1"}
-{"question_id": "e25c3b8d", "hypothesis": "I can only tell that you paid $200 for the designer handbag at TK Maxx; the savings amount isn’t stated in the evidence, so it can’t be determined from this information alone."}
-{"question_id": "4adc0475", "hypothesis": "Insufficient evidence for an exact total."}
-{"question_id": "4bc144e2", "hypothesis": "Parking ticket: $50; car wash not found in the retrieved evidence."}
-{"question_id": "ef66a6e5", "hypothesis": "4"}
-{"question_id": "5025383b", "hypothesis": "cooking"}
-{"question_id": "a1cc6108", "hypothesis": "You were about 0 years old when Alex was born."}
-{"question_id": "9ee3ecd6", "hypothesis": "300 points"}
-{"question_id": "3fdac837", "hypothesis": "12 days"}
-{"question_id": "91b15a6e", "hypothesis": "$5,150"}
-{"question_id": "27016adc", "hypothesis": "The planned renovations are $20,000. I don’t see the countryside property’s purchase price in the provided evidence, so the exact percentage can’t be computed from this evidence alone."}
-{"question_id": "720133ac", "hypothesis": "$75"}
-{"question_id": "77eafa52", "hypothesis": "$300"}
-{"question_id": "8979f9ec", "hypothesis": "I can only confirm those were lunch meal-prep recipe ideas mentioned on 2023/05/26; there’s no evidence here giving a number of meals for the chicken fajitas or any lentil soup, so the total can’t be computed from the provided evidence."}
-{"question_id": "0100672e", "hypothesis": "$12 per coffee mug"}
-{"question_id": "a96c20ee", "hypothesis": "I presented a poster on my thesis research at my first research conference over the summer, but the university isn't stated in the evidence provided."}
-{"question_id": "92a0aa75", "hypothesis": "Senior Marketing Specialist at the company"}
-{"question_id": "3fe836c9", "hypothesis": "The final sale price was $325,000. The only other amount mentioned is $350,000, but in the evidence it appears as a generic mortgage example, not a clearly stated pre-approval amount, so the exact difference can’t be determined from the evidence."}
-{"question_id": "1c549ce4", "hypothesis": "The only purchase cost shown is the waterproof car cover at $120. I don’t see evidence that you purchased a detailing spray or what it cost, so I can’t add a total beyond the confirmed $120."}
-{"question_id": "6c49646a", "hypothesis": "1940 miles"}
-{"question_id": "1192316e", "hypothesis": "I can’t compute a total from the provided evidence because it only mentions meditation and breakfast timing, not how long you take to get ready or commute to work."}
-{"question_id": "0ea62687", "hypothesis": "I can’t determine the difference from the retrieved memory: a few months ago it was 30 mpg, but your current mpg isn’t stated."}
-{"question_id": "67e0d0f2", "hypothesis": "At least 8. You explicitly mentioned completing 8 edX courses, and you also said you had completed some Coursera courses, but without a number, so the exact total isn’t determined from the evidence."}
-{"question_id": "bb7c3b45", "hypothesis": "The evidence only gives the original retail price of the Jimmy Choo heels as $500. It does not say what you actually paid, so the amount saved cannot be determined from the provided evidence."}
-{"question_id": "ba358f49", "hypothesis": "I don't know—Rachel is expected to get married in 2023, but your age isn't provided."}
-{"question_id": "61f8c8f8", "hypothesis": "10 minutes faster"}
-{"question_id": "60159905", "hypothesis": "1"}
-{"question_id": "ef9cf60a", "hypothesis": "$100"}
-{"question_id": "73d42213", "hypothesis": "No explicit arrival time is stated for Monday. The only clinic-travel detail says it took two hours to get there last time, but that doesn’t give a reach time."}
-{"question_id": "bc149d6b", "hypothesis": "70 pounds"}
-{"question_id": "099778bb", "hypothesis": "Insufficient evidence to determine the percentage; only the total of 100 leadership positions is provided, not how many are held by women."}
-{"question_id": "09ba9854", "hypothesis": "About $50."}
-{"question_id": "d6062bb9", "hypothesis": "1998"}
-{"question_id": "157a136e", "hypothesis": "I don't know"}
-{"question_id": "c18a7dc8", "hypothesis": "7"}
-{"question_id": "a3332713", "hypothesis": "About $200 total"}
-{"question_id": "55241a1f", "hypothesis": "Your recent Facebook Live session had 12 comments, but I don’t have evidence here for the comment count of your most popular YouTube video, so I can’t compute the total from the available information."}
-{"question_id": "a08a253f", "hypothesis": "4"}
-{"question_id": "f0e564bc", "hypothesis": "The evidence shows $800 for the designer handbag, but no amount is provided for the high-end skincare products, so the total cannot be determined."}
-{"question_id": "078150f1", "hypothesis": "$50 more"}
-{"question_id": "8cf4d046", "hypothesis": "I can’t compute an average GPA from the evidence provided. It mentions your undergraduate result as a First-Class distinction from the University of Mumbai and that you completed a Master’s in Data Science at UIUC, but it does not give explicit GPA values for either degree."}
-{"question_id": "a346bb18", "hypothesis": "12 minutes"}
-{"question_id": "37f165cf", "hypothesis": "416 pages and 440 pages"}
-{"question_id": "8e91e7d9", "hypothesis": "1"}
-{"question_id": "87f22b4a", "hypothesis": "$120"}
-{"question_id": "e56a43b9", "hypothesis": "$5 discount"}
-{"question_id": "efc3f7c2", "hypothesis": "30 minutes earlier"}
-{"question_id": "21d02d0d", "hypothesis": "2"}
-{"question_id": "2311e44b_abs", "hypothesis": "You mentioned that you've been reading \"Sapiens\" at a pace of 10–20 pages a week, but there’s no evidence here stating the book’s total page count or how many pages you’ve already finished, so the exact number of pages left can’t be determined from the provided evidence."}
-{"question_id": "6456829e_abs", "hypothesis": "5"}
-{"question_id": "e5ba910e_abs", "hypothesis": "$378"}
-{"question_id": "a96c20ee_abs", "hypothesis": "Harvard University"}
-{"question_id": "ba358f49_abs", "hypothesis": "Insufficient evidence to determine Rachel's age when you get married."}
-{"question_id": "09ba9854_abs", "hypothesis": "The evidence doesn’t give enough to calculate the savings exactly. It does mention the Airport Limousine Bus costs ¥3,200 one way to Shinjuku Station, but it does not provide a full taxi fare from the airport to your hotel, only that a short taxi ride from Shinjuku Station to the hotel might be needed."}
-{"question_id": "gpt4_59149c77", "hypothesis": "7 days"}
-{"question_id": "gpt4_f49edff3", "hypothesis": "helped my friend prepare a nursery, helped my cousin pick out stuff for her baby shower, ordered a customized phone case for my friend's birthday"}
-{"question_id": "71017276", "hypothesis": "4 weeks"}
-{"question_id": "b46e15ed", "hypothesis": "2 months"}
-{"question_id": "gpt4_fa19884c", "hypothesis": "6 days"}
-{"question_id": "0bc8ad92", "hypothesis": "2 months"}
-{"question_id": "af082822", "hypothesis": "2 weeks"}
-{"question_id": "gpt4_4929293a", "hypothesis": "Michael's engagement party"}
-{"question_id": "gpt4_b5700ca9", "hypothesis": "4 days ago"}
-{"question_id": "9a707b81", "hypothesis": "21 days"}
-{"question_id": "gpt4_1d4ab0c9", "hypothesis": "24 days"}
-{"question_id": "gpt4_e072b769", "hypothesis": "about 3 weeks ago"}
-{"question_id": "0db4c65d", "hypothesis": "18 days"}
-{"question_id": "gpt4_1d80365e", "hypothesis": "2 days"}
-{"question_id": "gpt4_7f6b06db", "hypothesis": "Yosemite, Big Sur and Monterey, Yosemite National Park"}
-{"question_id": "gpt4_6dc9b45b", "hypothesis": "4 months ago"}
-{"question_id": "gpt4_8279ba02", "hypothesis": "10 days ago"}
-{"question_id": "gpt4_18c2b244", "hypothesis": "I used a Buy One Get One Free coupon on Luvs diapers at Walmart, then I redeemed $12 cashback for a $10 Amazon gift card from Ibotta, then I signed up for the rewards program at ShopRite."}
-{"question_id": "gpt4_a1b77f9c", "hypothesis": "Insufficient evidence to calculate the total weeks spent."}
-{"question_id": "gpt4_1916e0ea", "hypothesis": "I can identify the Instacart online grocery shopping date as 2023/02/28, but the provided evidence does not include the date you cancelled your FarmFresh subscription, so the number of days passed cannot be determined from this evidence."}
-{"question_id": "gpt4_7a0daae1", "hypothesis": "I can tell you received your new tennis racket on 2023/03/17, but the buy date isn’t provided in the evidence, so the number of weeks between buying and receiving can’t be determined from the available information."}
-{"question_id": "gpt4_468eb063", "hypothesis": "9 days"}
-{"question_id": "gpt4_7abb270c", "hypothesis": "Science Museum, Metropolitan Museum of Art, Museum of Contemporary Art, Modern Art Museum, and two missing museum visits"}
-{"question_id": "gpt4_1e4a8aeb", "hypothesis": "6 days"}
-{"question_id": "gpt4_4fc4f797", "hypothesis": "38 days"}
-{"question_id": "4dfccbf7", "hypothesis": "24 days"}
-{"question_id": "gpt4_61e13b3c", "hypothesis": "3 weeks"}
-{"question_id": "gpt4_45189cb4", "hypothesis": "College Football National Championship game (Georgia vs. Alabama), then Lakers game, then Kansas City Chiefs vs. Buffalo Bills Divisional Round game"}
-{"question_id": "2ebe6c90", "hypothesis": "21 days"}
-{"question_id": "gpt4_e061b84f", "hypothesis": "volleyball league game, Spring Sprint Triathlon, company's annual charity soccer tournament"}
-{"question_id": "370a8ff4", "hypothesis": "11 weeks and 4 days"}
-{"question_id": "gpt4_d6585ce8", "hypothesis": "free outdoor concert series in the park, jazz night at a local bar"}
-{"question_id": "gpt4_4ef30696", "hypothesis": "1"}
-{"question_id": "gpt4_ec93e27f", "hypothesis": "train"}
-{"question_id": "6e984301", "hypothesis": "about 9 weeks"}
-{"question_id": "8077ef71", "hypothesis": "26 days ago"}
-{"question_id": "gpt4_f420262c", "hypothesis": "Delta, American Airlines"}
-{"question_id": "gpt4_8e165409", "hypothesis": "22"}
-{"question_id": "gpt4_74aed68e", "hypothesis": "29 days"}
-{"question_id": "bcbe585f", "hypothesis": "4 weeks and 2 days ago (about 4.3 weeks)"}
-{"question_id": "gpt4_21adecb5", "hypothesis": "6 months"}
-{"question_id": "5e1b23de", "hypothesis": "3 months ago"}
-{"question_id": "gpt4_98f46fc6", "hypothesis": "Insufficient evidence to determine"}
-{"question_id": "gpt4_af6db32f", "hypothesis": "17 days"}
-{"question_id": "eac54adc", "hypothesis": "19 days"}
-{"question_id": "gpt4_7ddcf75f", "hypothesis": "3 days ago"}
-{"question_id": "gpt4_a2d1d1f6", "hypothesis": "3 days ago"}
-{"question_id": "gpt4_85da3956", "hypothesis": "3 weeks"}
-{"question_id": "gpt4_b0863698", "hypothesis": "7 days ago"}
-{"question_id": "gpt4_68e94287", "hypothesis": "The vegan chili recipe post"}
-{"question_id": "gpt4_e414231e", "hypothesis": "4 days"}
-{"question_id": "gpt4_7ca326fa", "hypothesis": "Rachel first, Alex second, Emma third"}
-{"question_id": "gpt4_7bc6cf22", "hypothesis": "12 days"}
-{"question_id": "2ebe6c92", "hypothesis": "The Song of Achilles"}
-{"question_id": "gpt4_e061b84g", "hypothesis": "the company's annual charity soccer tournament"}
-{"question_id": "71017277", "hypothesis": "my aunt"}
-{"question_id": "b46e15ee", "hypothesis": "Walk for Hunger"}
-{"question_id": "gpt4_d6585ce9", "hypothesis": "my sister"}
-{"question_id": "gpt4_1e4a8aec", "hypothesis": "planted 12 new tomato saplings"}
-{"question_id": "gpt4_f420262d", "hypothesis": "Delta"}
-{"question_id": "gpt4_59149c78", "hypothesis": "the Metropolitan Museum of Art"}
-{"question_id": "gpt4_e414231f", "hypothesis": "mountain bike"}
-{"question_id": "gpt4_4929293b", "hypothesis": "your cousin's wedding"}
-{"question_id": "gpt4_468eb064", "hypothesis": "Emma"}
-{"question_id": "gpt4_fa19884d", "hypothesis": "Last Friday was 2023/03/31. The evidence from that day only shows artist recommendations, not a single explicit artist you definitely started listening to. The artists suggested then were Miles Davis, John Coltrane, Charlie Parker, and Thelonious Monk, plus bluegrass acts like Bill Monroe and the Blue Grass Boys, Earl Scruggs, Flatt and Scruggs, Alison Krauss and Union Station, and The Infamous Stringdusters."}
-{"question_id": "9a707b82", "hypothesis": "croissants"}
-{"question_id": "eac54add", "hypothesis": "I can’t identify any business milestone from the evidence. The only milestone-like detail mentioned around the retrieved time is that you had lost about 2 pounds after replacing your bathroom scale, which is a personal health update, not a business milestone."}
-{"question_id": "4dfccbf8", "hypothesis": "Ukulele lessons with Rachel"}
-{"question_id": "0bc8ad93", "hypothesis": "with a friend"}
-{"question_id": "6e984302", "hypothesis": "a set of sculpting tools: a modeling tool set, a wire cutter, and a sculpting mat"}
-{"question_id": "gpt4_8279ba03", "hypothesis": "I only have evidence of you buying a Samsung Galaxy S21 for about $800 at Best Buy, not a kitchen appliance 10 days before 2023/03/25."}
-{"question_id": "gpt4_b5700ca0", "hypothesis": "the Episcopal Church"}
-{"question_id": "gpt4_68e94288", "hypothesis": "#PlankChallenge"}
-{"question_id": "gpt4_2655b836", "hypothesis": "No retrieved evidence mentions your new car’s first service or any issue you had after it, so the first post-service issue cannot be determined from the available evidence."}
-{"question_id": "gpt4_2487a7cb", "hypothesis": "The 'Data Analysis using Python' webinar"}
-{"question_id": "gpt4_76048e76", "hypothesis": "bike"}
-{"question_id": "gpt4_2312f94c", "hypothesis": "Samsung Galaxy S22"}
-{"question_id": "0bb5a684", "hypothesis": "7 days"}
-{"question_id": "08f4fc43", "hypothesis": "30 days"}
-{"question_id": "2c63a862", "hypothesis": "14 days"}
-{"question_id": "gpt4_385a5000", "hypothesis": "Tomatoes"}
-{"question_id": "2a1811e2", "hypothesis": "11"}
-{"question_id": "bbf86515", "hypothesis": "4 days"}
-{"question_id": "gpt4_5dcc0aab", "hypothesis": "white Adidas sneakers"}
-{"question_id": "gpt4_0b2f1d21", "hypothesis": "The stand mixer malfunction happened first."}
-{"question_id": "f0853d11", "hypothesis": "14 days"}
-{"question_id": "gpt4_6ed717ea", "hypothesis": "the training pads for Luna"}
-{"question_id": "gpt4_70e84552", "hypothesis": "Fixing the fence"}
-{"question_id": "a3838d2b", "hypothesis": "2"}
-{"question_id": "gpt4_93159ced", "hypothesis": "I can’t compute the exact amount from the evidence: it shows you’ve been at NovaTech for about 4 years and 3 months, and that you started working in the field when you graduated, but it doesn’t give your graduation date or total career length before NovaTech."}
-{"question_id": "gpt4_2d58bcd6", "hypothesis": "The Hate U Give"}
-{"question_id": "gpt4_65aabe59", "hypothesis": "Can't tell from the evidence."}
-{"question_id": "982b5123", "hypothesis": "3 months"}
-{"question_id": "b9cfe692", "hypothesis": "I have evidence only for one of the two books: \"The Seven Husbands of Evelyn Hugo\" took you two and a half weeks to finish. There’s no evidence here for how long \"The Nightingale\" took, so I can’t compute a combined total from the provided memory."}
-{"question_id": "gpt4_4edbafa2", "hypothesis": "2023-06-03"}
-{"question_id": "c8090214", "hypothesis": "7 days"}
-{"question_id": "gpt4_483dd43c", "hypothesis": "Game of Thrones"}
-{"question_id": "e4e14d04", "hypothesis": "cannot be determined from the provided evidence"}
-{"question_id": "c9f37c46", "hypothesis": "unresolved"}
-{"question_id": "gpt4_2c50253f", "hypothesis": "I wake up at 6:30 AM on Tuesdays and Thursdays."}
-{"question_id": "dcfa8644", "hypothesis": "14 days"}
-{"question_id": "gpt4_b4a80587", "hypothesis": "the arrival of the new prime lens"}
-{"question_id": "gpt4_9a159967", "hypothesis": "Southwest Airlines and American Airlines (tie)"}
-{"question_id": "cc6d1ec1", "hypothesis": "about two months"}
-{"question_id": "gpt4_8c8961ae", "hypothesis": "the solo trip to Thailand"}
-{"question_id": "gpt4_d9af6064", "hypothesis": "new router"}
-{"question_id": "gpt4_7de946e7", "hypothesis": "persistent cough"}
-{"question_id": "d01c6aa8", "hypothesis": "Cannot be determined from the retrieved evidence."}
-{"question_id": "993da5e2", "hypothesis": "I can only infer that you rearranged the furniture about three weeks before 2023/05/26; there’s no evidence here about when you started using the new area rug, so the duration before the rearrangement can’t be determined from the provided memory."}
-{"question_id": "a3045048", "hypothesis": "30 days"}
-{"question_id": "gpt4_d31cdae3", "hypothesis": "The family road trip across the American Southwest."}
-{"question_id": "gpt4_cd90e484", "hypothesis": "Insufficient evidence to determine exactly"}
-{"question_id": "gpt4_88806d6e", "hypothesis": "Can't tell from the evidence."}
-{"question_id": "gpt4_4cd9eba1", "hypothesis": "1 week"}
-{"question_id": "gpt4_93f6379c", "hypothesis": "Page Turners"}
-{"question_id": "b29f3365", "hypothesis": "six weeks"}
-{"question_id": "gpt4_2f56ae70", "hypothesis": "There’s no evidence here of you starting any streaming service, so I can’t identify a most recent one from the provided memory."}
-{"question_id": "6613b389", "hypothesis": "2 months"}
-{"question_id": "gpt4_78cf46a3", "hypothesis": "Receiving the new phone case happened first."}
-{"question_id": "gpt4_0a05b494", "hypothesis": "the tourist from Australia"}
-{"question_id": "gpt4_1a1dc16d", "hypothesis": "The pride parade"}
-{"question_id": "gpt4_2f584639", "hypothesis": "The necklace for your sister."}
-{"question_id": "gpt4_213fd887", "hypothesis": "the volleyball league"}
-{"question_id": "gpt4_5438fa52", "hypothesis": "the start of your Spanish classes"}
-{"question_id": "gpt4_c27434e8", "hypothesis": "Japanese Zero fighter plane model"}
-{"question_id": "gpt4_fe651585", "hypothesis": "Rachel"}
-{"question_id": "8c18457d", "hypothesis": "21 days"}
-{"question_id": "gpt4_70e84552_abs", "hypothesis": "Fixing the fence"}
-{"question_id": "gpt4_93159ced_abs", "hypothesis": "The evidence doesn’t support a current job at Google. On 2023/05/30, you were described as working at NovaTech, and while there’s a truncated note saying you’d been in software engineering before mentioning your current job, it doesn’t include the duration. So I can’t compute how long you’d been working before starting a job at Google from the provided evidence."}
-{"question_id": "982b5123_abs", "hypothesis": "There’s no evidence here of you booking an Airbnb in Sacramento. The only Airbnb mention is for San Francisco, tied to your stay in Haight-Ashbury on 2023/05/27."}
-{"question_id": "c8090214_abs", "hypothesis": "I attended the Holiday Market 7 days before Black Friday, but there’s no evidence here that you bought an iPad, so the day difference to an iPad purchase can’t be determined from the provided information."}
-{"question_id": "gpt4_c27434e8_abs", "hypothesis": "Insufficient evidence to determine"}
-{"question_id": "gpt4_fe651585_abs", "hypothesis": "Alex"}
-{"question_id": "6a1eabeb", "hypothesis": "25:50"}
-{"question_id": "6aeb4375", "hypothesis": "2"}
-{"question_id": "830ce83f", "hypothesis": "Chicago"}
-{"question_id": "852ce960", "hypothesis": "$350,000"}
-{"question_id": "945e3d21", "hypothesis": "three times a week"}
-{"question_id": "d7c942c3", "hypothesis": "Yes — your mom is using the same grocery list app as you."}
-{"question_id": "71315a70", "hypothesis": "10-12 hours"}
-{"question_id": "89941a93", "hypothesis": "1"}
-{"question_id": "ce6d2d27", "hypothesis": "Friday"}
-{"question_id": "9ea5eabc", "hypothesis": "Paris"}
-{"question_id": "07741c44", "hypothesis": "shoe rack"}
-{"question_id": "a1eacc2a", "hypothesis": "7"}
-{"question_id": "184da446", "hypothesis": "220 pages"}
-{"question_id": "031748ae", "hypothesis": "4 when you started; 5 now"}
-{"question_id": "4d6b87c8", "hypothesis": "25"}
-{"question_id": "0f05491a", "hypothesis": "120 stars"}
-{"question_id": "08e075c7", "hypothesis": "I can’t determine that from the provided evidence. The memory snippets only mention your bedtime routine, reading, and using a massage roller, with no evidence about when you started using a Fitbit Charge 3."}
-{"question_id": "f9e8c073", "hypothesis": "5 sessions"}
-{"question_id": "41698283", "hypothesis": "70-200mm zoom lens"}
-{"question_id": "2698e78f", "hypothesis": "Not specified; the memory only says to review it regularly with Dr. Smith."}
-{"question_id": "b6019101", "hypothesis": "5"}
-{"question_id": "45dc21b6", "hypothesis": "2"}
-{"question_id": "5a4f22c0", "hypothesis": "TechCorp"}
-{"question_id": "6071bd76", "hypothesis": "less"}
-{"question_id": "e493bb7c", "hypothesis": "above my bed in my bedroom"}
-{"question_id": "618f13b2", "hypothesis": "6"}
-{"question_id": "72e3ee87", "hypothesis": "50"}
-{"question_id": "c4ea545c", "hypothesis": "Insufficient evidence"}
-{"question_id": "01493427", "hypothesis": "17"}
-{"question_id": "6a27ffc2", "hypothesis": "30"}
-{"question_id": "2133c1b5", "hypothesis": "3 months"}
-{"question_id": "18bc8abd", "hypothesis": "Kansas City Masterpiece BBQ sauce"}
-{"question_id": "db467c8c", "hypothesis": "about nine months"}
-{"question_id": "7a87bd0c", "hypothesis": "4 weeks"}
-{"question_id": "e61a7584", "hypothesis": "about 9 months"}
-{"question_id": "1cea1afa", "hypothesis": "600"}
-{"question_id": "ed4ddc30", "hypothesis": "20 dozen eggs"}
-{"question_id": "8fb83627", "hypothesis": "5"}
-{"question_id": "b01defab", "hypothesis": "Yes"}
-{"question_id": "22d2cb42", "hypothesis": "the music shop on Main St"}
-{"question_id": "0e4e4c46", "hypothesis": "132 points"}
-{"question_id": "4b24c848", "hypothesis": "5"}
-{"question_id": "7e974930", "hypothesis": "Your most recent Downtown Farmers Market visit was on 2023/09/30, but the exact earnings amount isn’t shown in the retrieved evidence."}
-{"question_id": "603deb26", "hypothesis": "10"}
-{"question_id": "59524333", "hypothesis": "6:00 pm"}
-{"question_id": "5831f84d", "hypothesis": "10"}
-{"question_id": "eace081b", "hypothesis": "Oahu"}
-{"question_id": "affe2881", "hypothesis": "32"}
-{"question_id": "50635ada", "hypothesis": "Premier Gold status"}
-{"question_id": "e66b632c", "hypothesis": "26 minutes and 30 seconds"}
-{"question_id": "0ddfec37", "hypothesis": "Cannot be determined from available evidence"}
-{"question_id": "f685340e", "hypothesis": "Previously, every other week. Now, not explicitly stated."}
-{"question_id": "cc5ded98", "hypothesis": "an hour each day"}
-{"question_id": "dfde3500", "hypothesis": "Wednesday"}
-{"question_id": "69fee5aa", "hypothesis": "37"}
-{"question_id": "7401057b", "hypothesis": "1 free night's stay"}
-{"question_id": "cf22b7bf", "hypothesis": "10 pounds"}
-{"question_id": "a2f3aa27", "hypothesis": "close to 1300 followers"}
-{"question_id": "c7dc5443", "hypothesis": "5-2"}
-{"question_id": "06db6396", "hypothesis": "5"}
-{"question_id": "3ba21379", "hypothesis": "Ford F-150 pickup truck model"}
-{"question_id": "9bbe84a2", "hypothesis": "level 100 before the end of the year"}
-{"question_id": "10e09553", "hypothesis": "7"}
-{"question_id": "dad224aa", "hypothesis": "7:30 am"}
-{"question_id": "ba61f0b9", "hypothesis": "6"}
-{"question_id": "42ec0761", "hypothesis": "Yes, you have a spare screwdriver."}
-{"question_id": "5c40ec5b", "hypothesis": "2"}
-{"question_id": "c6853660", "hypothesis": "increase"}
-{"question_id": "26bdc477", "hypothesis": "5"}
-{"question_id": "0977f2af", "hypothesis": "I can’t tell from the retrieved evidence."}
-{"question_id": "6aeb4375_abs", "hypothesis": "{\"answer\":0}"}
-{"question_id": "031748ae_abs", "hypothesis": "4"}
-{"question_id": "2698e78f_abs", "hypothesis": "No evidence provided states how often you see Dr. Johnson."}
-{"question_id": "2133c1b5_abs", "hypothesis": "About 7 months. The evidence says it had been 1 month in your new studio apartment in Harajuku on 2023/04/11, which makes it about 7 months by 2023/10/15. The memory mentions Harajuku, not Shinjuku."}
-{"question_id": "0ddfec37_abs", "hypothesis": "0"}
-{"question_id": "f685340e_abs", "hypothesis": "Every other week."}
-{"question_id": "89941a94", "hypothesis": "Yes — a road bike."}
-{"question_id": "07741c45", "hypothesis": "shoe rack"}
-{"question_id": "7161e7e2", "hypothesis": "8 am - 4 pm (Day Shift)"}
-{"question_id": "c4f10528", "hypothesis": "Miss Bee Providore"}
-{"question_id": "89527b6b", "hypothesis": "I can only recover that the dinosaur-book conversation was on 2023/05/23 and the visible excerpt includes a T-Rex with a green scaly body; the Plesiosaur image description and its body color are not present in the provided evidence."}
-{"question_id": "e9327a54", "hypothesis": "I don’t have the dessert shop’s name in the retrieved memory."}
-{"question_id": "4c36ccef", "hypothesis": "Roscioli"}
-{"question_id": "6ae235be", "hypothesis": "Atmospheric distillation, fluid catalytic cracking (FCC), alkylation, and hydrotreating."}