wandler67 commited on
Commit
2cc85fd
·
verified ·
1 Parent(s): 0f91d75

Publish Unified-3000 v2 Explorer

Browse files
README.md CHANGED
@@ -19,10 +19,7 @@ H²EPR-Bench Explorer is the interactive browsing layer for the Unified-3000
19
  release of `AgenticFinLab/H2EPR-Bench`. The existing Docker Space runs a
20
  Streamlit interface for searching all 3,000 catalog events, inspecting current
21
  Draft EPG summaries and stage timelines, and previewing or downloading the
22
- 2,876 public per-event Draft EPG files.
23
-
24
- The other 124 catalog events remain fully browsable and show a neutral local
25
- empty state because no public Draft EPG is available for them in this release.
26
 
27
  **Release boundary:** public Draft EPGs are sanitized FinMycelium construction
28
  artifacts. Official benchmark scoring uses expert-adjudicated reference EPGs in
@@ -34,38 +31,38 @@ including public Draft EPGs, remains CC BY-NC 4.0 under the Dataset release.
34
 
35
  ## Public data contract
36
 
37
- The Explorer loads five Parquet tables from one immutable dataset revision:
38
 
39
  | Table | Role |
40
  |---|---|
 
41
  | `event_catalog.parquet` | Stable event discovery metadata |
42
- | `event_instances.parquet` | Public access-state fields |
43
  | `finalcascade_summary.parquet` | Draft graph counts and event-level temporal summary |
44
- | `draft_availability.parquet` | Per-event Draft EPG availability and integrity metadata |
45
- | `event_stages.parquet` | Ordered stage rows for available drafts |
46
 
47
  Selected Draft EPGs are loaded directly from
48
  `draft_events/<H2EPR-ID>/draft_epg.json`. The path is derived only after the
49
- canonical event ID and availability state have been validated. The availability
50
- table's `draft_asset` value identifies the consolidated release asset and is
51
- not used as the selected-event path.
52
-
53
- Default dataset revision:
54
 
55
- ```text
56
- 1d01f3649ace0301ac3bbe9ee875eea660347a29
57
- ```
58
 
59
  ## Local development
60
 
61
- The app can use the frozen local Unified-3000 release candidate without any
62
- network access:
63
 
64
  ```bash
65
- export H2EPR_EXPLORER_LOCAL_DATASET_DIR=../../build/hf_unified3000_inplace_upgrade_rc_v3_dataset_card/H2EPR-Bench
66
  streamlit run app.py
67
  ```
68
 
69
- Without `H2EPR_EXPLORER_LOCAL_DATASET_DIR`, all public table and selected-event
70
- downloads use the pinned revision above. The application exposes no runtime
71
- revision override, so one process cannot mix dataset revisions.
 
 
19
  release of `AgenticFinLab/H2EPR-Bench`. The existing Docker Space runs a
20
  Streamlit interface for searching all 3,000 catalog events, inspecting current
21
  Draft EPG summaries and stage timelines, and previewing or downloading the
22
+ public per-event Draft EPG for every event.
 
 
 
23
 
24
  **Release boundary:** public Draft EPGs are sanitized FinMycelium construction
25
  artifacts. Official benchmark scoring uses expert-adjudicated reference EPGs in
 
31
 
32
  ## Public data contract
33
 
34
+ The Explorer validates five Parquet tables from one release identity:
35
 
36
  | Table | Role |
37
  |---|---|
38
+ | `event_gallery.parquet` | Lightweight event discovery metadata |
39
  | `event_catalog.parquet` | Stable event discovery metadata |
40
+ | `event_instances.parquet` | Public artifact-access fields |
41
  | `finalcascade_summary.parquet` | Draft graph counts and event-level temporal summary |
42
+ | `event_stages.parquet` | Ordered stage rows for every Draft EPG |
 
43
 
44
  Selected Draft EPGs are loaded directly from
45
  `draft_events/<H2EPR-ID>/draft_epg.json`. The path is derived only after the
46
+ canonical event ID has been validated. `manifests/draft_source_hashes.csv`
47
+ binds every direct file to its canonical source-payload digest and sanitized
48
+ record digest; the Explorer verifies both identities and the canonical JSON
49
+ digest before displaying a graph.
 
50
 
51
+ Remote reads and immutable per-file links are pinned to Dataset commit
52
+ `6156a6bb3b838143401cb3e5709f708e5d6e802c`. They fail closed if that fixed
53
+ release cannot be resolved and never fall back to a floating branch.
54
 
55
  ## Local development
56
 
57
+ The app can use the frozen local Unified-3000 release tree without any network
58
+ access:
59
 
60
  ```bash
61
+ export H2EPR_EXPLORER_LOCAL_DATASET_DIR=/path/to/H2EPR-Bench
62
  streamlit run app.py
63
  ```
64
 
65
+ For local verification, `H2EPR_EXPLORER_LOCAL_DATASET_DIR` is required. In the
66
+ published Space, the source constant fixes the immutable Dataset revision; the
67
+ application exposes no runtime revision override, so one process cannot mix
68
+ releases.
app.py CHANGED
@@ -14,13 +14,11 @@ import streamlit as st
14
 
15
  from h2epr_explorer.constants import (
16
  CATALOG_COLUMNS,
17
- DRAFT_UNAVAILABLE_MESSAGE,
18
  FINMYCELIUM_URL,
19
  GOLD_COMPANION_REPO,
20
  GOLD_COMPANION_URL,
21
  PROFILE_COLUMNS,
22
  PUBLIC_DATASET_REPO,
23
- PUBLIC_DATASET_REVISION,
24
  PUBLIC_DATASET_REVISION_URL,
25
  PROJECT_WEBSITE_URL,
26
  RELEASE_BOUNDARY_NOTICE,
@@ -30,7 +28,6 @@ from h2epr_explorer.data_loader import (
30
  DatasetTransportError,
31
  DraftAssetMissing,
32
  DraftIntegrityError,
33
- DraftUnavailable,
34
  ReleaseContractError,
35
  load_event_graph,
36
  load_release,
@@ -114,7 +111,7 @@ div[data-testid="stMetric"] {
114
  st.markdown('<div class="h2epr-kicker">H²EPR-Bench · public release explorer</div>', unsafe_allow_html=True)
115
  st.markdown('<div class="h2epr-title">Event-process graph browser</div>', unsafe_allow_html=True)
116
  st.markdown(
117
- '<div class="h2epr-subtitle">Browse 3,000 public event records, Draft EPG summaries, and stage timelines from one pinned Unified-3000 release.</div>',
118
  unsafe_allow_html=True,
119
  )
120
  st.info(RELEASE_BOUNDARY_NOTICE)
@@ -222,8 +219,7 @@ event_row = release.event_row(selected_event)
222
  event_record = event_row.to_dict()
223
  event_stages = release.stage_frame(selected_event)
224
  stage_records = _as_records(event_stages)
225
- draft_available = event_record["draft_status"] == "draft_available"
226
- event_links = build_event_links(selected_event, draft_available=draft_available)
227
 
228
  st.caption(filter_summary_text(len(filtered_rows), len(catalog_rows)))
229
 
@@ -255,12 +251,12 @@ with tabs[1]:
255
  width="stretch",
256
  )
257
 
258
- link_columns = st.columns(3 if draft_available else 2)
259
  link_columns[0].link_button("Open Public Dataset", event_links["public_dataset"], width="stretch")
260
  link_columns[1].link_button(
261
  "Reference EPGs (Gated)", event_links["reference_access"], width="stretch"
262
  )
263
- if draft_available:
264
  link_columns[2].link_button(
265
  "Open Draft EPG file", event_links["draft_epg"], width="stretch"
266
  )
@@ -282,41 +278,33 @@ with tabs[1]:
282
  with tabs[2]:
283
  figure = build_timeline_figure(stage_records, selected_event)
284
  if figure is None:
285
- st.info(DRAFT_UNAVAILABLE_MESSAGE)
286
  else:
287
  st.plotly_chart(figure, width="stretch")
288
 
289
  with tabs[3]:
290
- if event_stages.empty:
291
- st.info(DRAFT_UNAVAILABLE_MESSAGE)
292
- else:
293
- st.dataframe(event_stages, width="stretch", height=520)
294
 
295
  with tabs[4]:
296
- if not draft_available:
297
- st.info(DRAFT_UNAVAILABLE_MESSAGE)
 
 
 
 
 
 
298
  else:
299
- try:
300
- graph_result = load_event_graph(selected_event, release=release)
301
- except DatasetTransportError as exc:
302
- st.error(f"The selected Draft EPG could not be retrieved. Please retry. Details: {exc}")
303
- except DraftAssetMissing as exc:
304
- st.error(f"The selected event is marked available but its Draft EPG file is missing: {exc}")
305
- except DraftIntegrityError as exc:
306
- st.error(f"The selected Draft EPG failed integrity validation and was not shown: {exc}")
307
- else:
308
- if isinstance(graph_result, DraftUnavailable):
309
- st.info(graph_result.message)
310
- else:
311
- graph_json = json.dumps(graph_result, ensure_ascii=False, indent=2)
312
- st.download_button(
313
- "Download selected Draft EPG JSON",
314
- data=graph_json,
315
- file_name=f"{selected_event}_draft_epg.json",
316
- mime="application/json",
317
- )
318
- st.link_button("Open exact public file", event_links["draft_epg"])
319
- st.json(graph_result, expanded=False)
320
 
321
  with tabs[5]:
322
  st.markdown(
@@ -324,7 +312,7 @@ with tabs[5]:
324
  ### Release boundary
325
 
326
  - Project Website: [H²EPR-Bench]({PROJECT_WEBSITE_URL}).
327
- - Public Dataset: [`{PUBLIC_DATASET_REPO}`]({PUBLIC_DATASET_REVISION_URL}) at `{PUBLIC_DATASET_REVISION}`.
328
  - FinMycelium System: [multi-agent event reconstruction system]({FINMYCELIUM_URL}).
329
  - Reference EPGs (Gated): [`{GOLD_COMPANION_REPO}`]({GOLD_COMPANION_URL}).
330
  - Release Repository: [`AgenticFinLab/H2EPR-Bench`]({SOURCE_REPOSITORY_URL}).
 
14
 
15
  from h2epr_explorer.constants import (
16
  CATALOG_COLUMNS,
 
17
  FINMYCELIUM_URL,
18
  GOLD_COMPANION_REPO,
19
  GOLD_COMPANION_URL,
20
  PROFILE_COLUMNS,
21
  PUBLIC_DATASET_REPO,
 
22
  PUBLIC_DATASET_REVISION_URL,
23
  PROJECT_WEBSITE_URL,
24
  RELEASE_BOUNDARY_NOTICE,
 
28
  DatasetTransportError,
29
  DraftAssetMissing,
30
  DraftIntegrityError,
 
31
  ReleaseContractError,
32
  load_event_graph,
33
  load_release,
 
111
  st.markdown('<div class="h2epr-kicker">H²EPR-Bench · public release explorer</div>', unsafe_allow_html=True)
112
  st.markdown('<div class="h2epr-title">Event-process graph browser</div>', unsafe_allow_html=True)
113
  st.markdown(
114
+ '<div class="h2epr-subtitle">Browse 3,000 public event records, Draft EPG summaries, and stage timelines from one validated Unified-3000 release.</div>',
115
  unsafe_allow_html=True,
116
  )
117
  st.info(RELEASE_BOUNDARY_NOTICE)
 
219
  event_record = event_row.to_dict()
220
  event_stages = release.stage_frame(selected_event)
221
  stage_records = _as_records(event_stages)
222
+ event_links = build_event_links(selected_event)
 
223
 
224
  st.caption(filter_summary_text(len(filtered_rows), len(catalog_rows)))
225
 
 
251
  width="stretch",
252
  )
253
 
254
+ link_columns = st.columns(3 if "draft_epg" in event_links else 2)
255
  link_columns[0].link_button("Open Public Dataset", event_links["public_dataset"], width="stretch")
256
  link_columns[1].link_button(
257
  "Reference EPGs (Gated)", event_links["reference_access"], width="stretch"
258
  )
259
+ if "draft_epg" in event_links:
260
  link_columns[2].link_button(
261
  "Open Draft EPG file", event_links["draft_epg"], width="stretch"
262
  )
 
278
  with tabs[2]:
279
  figure = build_timeline_figure(stage_records, selected_event)
280
  if figure is None:
281
+ st.error("The validated stage rows could not be rendered as a timeline.")
282
  else:
283
  st.plotly_chart(figure, width="stretch")
284
 
285
  with tabs[3]:
286
+ st.dataframe(event_stages, width="stretch", height=520)
 
 
 
287
 
288
  with tabs[4]:
289
+ try:
290
+ graph_result = load_event_graph(selected_event, release=release)
291
+ except DatasetTransportError as exc:
292
+ st.error(f"The selected Draft EPG could not be retrieved. Please retry. Details: {exc}")
293
+ except DraftAssetMissing as exc:
294
+ st.error(f"The selected Draft EPG file is missing: {exc}")
295
+ except DraftIntegrityError as exc:
296
+ st.error(f"The selected Draft EPG failed integrity validation and was not shown: {exc}")
297
  else:
298
+ graph_json = json.dumps(graph_result, ensure_ascii=False, indent=2)
299
+ st.download_button(
300
+ "Download selected Draft EPG JSON",
301
+ data=graph_json,
302
+ file_name=f"{selected_event}_draft_epg.json",
303
+ mime="application/json",
304
+ )
305
+ if "draft_epg" in event_links:
306
+ st.link_button("Open exact public file", event_links["draft_epg"])
307
+ st.json(graph_result, expanded=False)
 
 
 
 
 
 
 
 
 
 
 
308
 
309
  with tabs[5]:
310
  st.markdown(
 
312
  ### Release boundary
313
 
314
  - Project Website: [H²EPR-Bench]({PROJECT_WEBSITE_URL}).
315
+ - Public Dataset: [`{PUBLIC_DATASET_REPO}`]({PUBLIC_DATASET_REVISION_URL}).
316
  - FinMycelium System: [multi-agent event reconstruction system]({FINMYCELIUM_URL}).
317
  - Reference EPGs (Gated): [`{GOLD_COMPANION_REPO}`]({GOLD_COMPANION_URL}).
318
  - Release Repository: [`AgenticFinLab/H2EPR-Bench`]({SOURCE_REPOSITORY_URL}).
src/h2epr_explorer/constants.py CHANGED
@@ -11,26 +11,47 @@ GOLD_COMPANION_URL = "https://huggingface.co/datasets/AgenticFinLab/H2EPR-Bench-
11
  SOURCE_REPOSITORY_URL = "https://github.com/AgenticFinLab/H2EPR-Bench"
12
  AGENTICFINLAB_URL = "https://agenticfinlab.github.io/"
13
 
14
- DEFAULT_PUBLIC_DATASET_REVISION = "1d01f3649ace0301ac3bbe9ee875eea660347a29"
15
  PUBLIC_DATASET_REVISION = DEFAULT_PUBLIC_DATASET_REVISION
16
- PUBLIC_DATASET_REVISION_URL = f"{PUBLIC_DATASET_URL}/tree/{PUBLIC_DATASET_REVISION}"
 
 
 
 
17
  LOCAL_DATASET_ENV = "H2EPR_EXPLORER_LOCAL_DATASET_DIR"
18
 
19
  EVENT_ID_PATTERN = r"^H2EPR-[0-9]{4}$"
20
  EVENT_ID_MIN = 1
21
  EVENT_ID_MAX = 3000
22
 
 
23
  CATALOG_PARQUET = "data/viewer_mirrors/event_catalog.parquet"
24
  EVENT_INSTANCES_PARQUET = "data/viewer_mirrors/event_instances.parquet"
25
  FINALCASCADE_SUMMARY_PARQUET = "data/viewer_mirrors/finalcascade_summary.parquet"
26
- DRAFT_AVAILABILITY_PARQUET = "data/viewer_mirrors/draft_availability.parquet"
27
  STAGES_PARQUET = "data/viewer_mirrors/event_stages.parquet"
 
28
  DRAFT_EPG_PATH_TEMPLATE = "draft_events/{event_id}/draft_epg.json"
29
 
 
 
 
 
 
 
 
 
 
30
  EXPECTED_EVENT_COUNT = 3000
31
- EXPECTED_AVAILABLE_DRAFT_COUNT = 2876
32
- EXPECTED_UNAVAILABLE_DRAFT_COUNT = 124
33
- EXPECTED_STAGE_ROW_COUNT = 8500
 
 
 
 
 
 
 
34
 
35
  CATALOG_SCHEMA = (
36
  "public_event_id",
@@ -41,11 +62,10 @@ CATALOG_SCHEMA = (
41
  "domain",
42
  "category",
43
  "keywords",
44
- "release_split",
45
- "version",
46
- "schema_version",
47
- "draft_status",
48
  "has_gold_reference",
 
 
 
49
  )
50
 
51
  EVENT_INSTANCES_SCHEMA = (
@@ -57,15 +77,11 @@ EVENT_INSTANCES_SCHEMA = (
57
  "domain",
58
  "category",
59
  "keywords",
60
- "release_split",
61
- "version",
62
- "schema_version",
63
- "has_finalcascade",
64
- "draft_status",
65
  "has_gold_reference",
66
  "finalcascade_access_level",
67
  "gold_reference_access_level",
68
  "evidence_context_access_level",
 
69
  )
70
 
71
  FINALCASCADE_SUMMARY_SCHEMA = (
@@ -74,7 +90,6 @@ FINALCASCADE_SUMMARY_SCHEMA = (
74
  "title",
75
  "domain",
76
  "category",
77
- "draft_status",
78
  "stage_count",
79
  "episode_count",
80
  "participant_count",
@@ -87,18 +102,14 @@ FINALCASCADE_SUMMARY_SCHEMA = (
87
  "known_action_time_anchor_count",
88
  "known_action_time_anchors",
89
  "relative_order_available",
 
90
  )
91
 
92
- DRAFT_AVAILABILITY_SCHEMA = (
93
  "public_event_id",
94
- "draft_status",
95
- "draft_source_kind",
96
- "draft_schema",
97
- "draft_asset",
98
- "draft_record_index",
99
- "draft_sha256",
100
  "source_payload_sha256",
101
- "has_reference_epg",
 
102
  )
103
 
104
  STAGES_SCHEMA = (
@@ -118,11 +129,21 @@ STAGES_SCHEMA = (
118
  "known_action_time_anchor_count",
119
  "known_action_time_anchors",
120
  "relative_order_available",
121
- "release_split",
122
- "version",
123
  "schema_version",
124
  )
125
 
 
 
 
 
 
 
 
 
 
 
 
 
126
  GRAPH_COUNT_COLUMNS = (
127
  "stage_count",
128
  "episode_count",
@@ -137,16 +158,13 @@ ARROW_INT64_COLUMNS = frozenset(
137
  *GRAPH_COUNT_COLUMNS,
138
  "stage_index",
139
  "known_action_time_anchor_count",
140
- "draft_record_index",
141
  }
142
  )
143
 
144
  ARROW_BOOL_COLUMNS = frozenset(
145
  {
146
  "has_gold_reference",
147
- "has_finalcascade",
148
  "relative_order_available",
149
- "has_reference_epg",
150
  }
151
  )
152
 
@@ -172,8 +190,6 @@ PROFILE_COLUMNS = (
172
  "finalcascade_access_level",
173
  )
174
 
175
- DRAFT_UNAVAILABLE_MESSAGE = "No public Draft EPG is available for this event in this release."
176
-
177
  RELEASE_BOUNDARY_NOTICE = (
178
  "Public Draft EPGs are FinMycelium construction artifacts. Official benchmark "
179
  "scoring uses reference EPGs in the manual-gated companion repository; this "
 
11
  SOURCE_REPOSITORY_URL = "https://github.com/AgenticFinLab/H2EPR-Bench"
12
  AGENTICFINLAB_URL = "https://agenticfinlab.github.io/"
13
 
14
+ DEFAULT_PUBLIC_DATASET_REVISION: str | None = "6156a6bb3b838143401cb3e5709f708e5d6e802c"
15
  PUBLIC_DATASET_REVISION = DEFAULT_PUBLIC_DATASET_REVISION
16
+ PUBLIC_DATASET_REVISION_URL = (
17
+ f"{PUBLIC_DATASET_URL}/tree/{PUBLIC_DATASET_REVISION}"
18
+ if PUBLIC_DATASET_REVISION
19
+ else PUBLIC_DATASET_URL
20
+ )
21
  LOCAL_DATASET_ENV = "H2EPR_EXPLORER_LOCAL_DATASET_DIR"
22
 
23
  EVENT_ID_PATTERN = r"^H2EPR-[0-9]{4}$"
24
  EVENT_ID_MIN = 1
25
  EVENT_ID_MAX = 3000
26
 
27
+ EVENT_GALLERY_PARQUET = "data/viewer_mirrors/event_gallery.parquet"
28
  CATALOG_PARQUET = "data/viewer_mirrors/event_catalog.parquet"
29
  EVENT_INSTANCES_PARQUET = "data/viewer_mirrors/event_instances.parquet"
30
  FINALCASCADE_SUMMARY_PARQUET = "data/viewer_mirrors/finalcascade_summary.parquet"
 
31
  STAGES_PARQUET = "data/viewer_mirrors/event_stages.parquet"
32
+ DRAFT_SOURCE_HASHES_CSV = "manifests/draft_source_hashes.csv"
33
  DRAFT_EPG_PATH_TEMPLATE = "draft_events/{event_id}/draft_epg.json"
34
 
35
+ RELEASE_ASSET_SHA256 = {
36
+ EVENT_GALLERY_PARQUET: "be68b57e42cfc0cde97c949b5dcfe14cc4ec80397d428f1f27d88b39e88a9b53",
37
+ CATALOG_PARQUET: "2a478a96aa2713b3b3894a222a511aaaadd06327c90498d03405ad1860a33ac0",
38
+ EVENT_INSTANCES_PARQUET: "ba258780091c10c46508684d90bebd5f34285a61cc4bc4c6600c75fa380817a8",
39
+ STAGES_PARQUET: "eeab17e56ceb14a99ffc6a64e8508f9b98bd5ea9d260b14cbf95a42374dc8db8",
40
+ FINALCASCADE_SUMMARY_PARQUET: "273fedfdc74aba8f00669b7e82d45ec4a312b16aaa98f7c28182a06d9c6f471f",
41
+ DRAFT_SOURCE_HASHES_CSV: "29f8c3af1641b4b7031cb0d21021177e5eda6816b57b82adbd3d4d7952f76d1d",
42
+ }
43
+
44
  EXPECTED_EVENT_COUNT = 3000
45
+ EXPECTED_STAGE_ROW_COUNT = 8843
46
+
47
+ EVENT_GALLERY_SCHEMA = (
48
+ "public_event_id",
49
+ "title",
50
+ "domain",
51
+ "category",
52
+ "event_descriptor",
53
+ "schema_version",
54
+ )
55
 
56
  CATALOG_SCHEMA = (
57
  "public_event_id",
 
62
  "domain",
63
  "category",
64
  "keywords",
 
 
 
 
65
  "has_gold_reference",
66
+ "stage_count",
67
+ "episode_count",
68
+ "schema_version",
69
  )
70
 
71
  EVENT_INSTANCES_SCHEMA = (
 
77
  "domain",
78
  "category",
79
  "keywords",
 
 
 
 
 
80
  "has_gold_reference",
81
  "finalcascade_access_level",
82
  "gold_reference_access_level",
83
  "evidence_context_access_level",
84
+ "schema_version",
85
  )
86
 
87
  FINALCASCADE_SUMMARY_SCHEMA = (
 
90
  "title",
91
  "domain",
92
  "category",
 
93
  "stage_count",
94
  "episode_count",
95
  "participant_count",
 
102
  "known_action_time_anchor_count",
103
  "known_action_time_anchors",
104
  "relative_order_available",
105
+ "schema_version",
106
  )
107
 
108
+ DRAFT_SOURCE_HASHES_SCHEMA = (
109
  "public_event_id",
 
 
 
 
 
 
110
  "source_payload_sha256",
111
+ "sanitized_record_sha256",
112
+ "draft_record_index",
113
  )
114
 
115
  STAGES_SCHEMA = (
 
129
  "known_action_time_anchor_count",
130
  "known_action_time_anchors",
131
  "relative_order_available",
 
 
132
  "schema_version",
133
  )
134
 
135
+ TABLE_SCHEMA_VERSIONS = {
136
+ "event_gallery": "h2epr-public-event-gallery-v3",
137
+ "event_catalog": "h2epr-public-event-catalog-v3",
138
+ "event_instances": "h2epr-public-event-instances-v3",
139
+ "event_stages": "h2epr-public-event-stages-v3",
140
+ "finalcascade_summary": "h2epr-public-finalcascade-summary-v3",
141
+ }
142
+
143
+ FINALCASCADE_ACCESS_LEVEL = "public_sanitized_full_graph"
144
+ GOLD_REFERENCE_ACCESS_LEVEL = "manual_gated_companion"
145
+ EVIDENCE_CONTEXT_ACCESS_LEVEL = "not_included_in_this_release"
146
+
147
  GRAPH_COUNT_COLUMNS = (
148
  "stage_count",
149
  "episode_count",
 
158
  *GRAPH_COUNT_COLUMNS,
159
  "stage_index",
160
  "known_action_time_anchor_count",
 
161
  }
162
  )
163
 
164
  ARROW_BOOL_COLUMNS = frozenset(
165
  {
166
  "has_gold_reference",
 
167
  "relative_order_available",
 
168
  }
169
  )
170
 
 
190
  "finalcascade_access_level",
191
  )
192
 
 
 
193
  RELEASE_BOUNDARY_NOTICE = (
194
  "Public Draft EPGs are FinMycelium construction artifacts. Official benchmark "
195
  "scoring uses reference EPGs in the manual-gated companion repository; this "
src/h2epr_explorer/data_loader.py CHANGED
@@ -1,5 +1,6 @@
1
  from __future__ import annotations
2
 
 
3
  from dataclasses import dataclass
4
  from functools import lru_cache
5
  import hashlib
@@ -19,27 +20,31 @@ from .constants import (
19
  ARROW_INT64_COLUMNS,
20
  CATALOG_PARQUET,
21
  CATALOG_SCHEMA,
22
- DRAFT_AVAILABILITY_PARQUET,
23
- DRAFT_AVAILABILITY_SCHEMA,
24
  DRAFT_EPG_PATH_TEMPLATE,
25
- DRAFT_UNAVAILABLE_MESSAGE,
 
 
 
26
  EVENT_ID_MAX,
27
  EVENT_ID_MIN,
28
  EVENT_ID_PATTERN,
29
  EVENT_INSTANCES_PARQUET,
30
  EVENT_INSTANCES_SCHEMA,
31
- EXPECTED_AVAILABLE_DRAFT_COUNT,
32
  EXPECTED_EVENT_COUNT,
33
  EXPECTED_STAGE_ROW_COUNT,
34
- EXPECTED_UNAVAILABLE_DRAFT_COUNT,
35
  FINALCASCADE_SUMMARY_PARQUET,
36
  FINALCASCADE_SUMMARY_SCHEMA,
 
37
  GRAPH_COUNT_COLUMNS,
38
  LOCAL_DATASET_ENV,
39
  PUBLIC_DATASET_REPO,
40
  PUBLIC_DATASET_REVISION,
 
41
  STAGES_PARQUET,
42
  STAGES_SCHEMA,
 
43
  )
44
 
45
 
@@ -51,12 +56,16 @@ class DatasetTransportError(ExplorerDataError):
51
  """A pinned public dataset asset could not be retrieved."""
52
 
53
 
 
 
 
 
54
  class ReleaseContractError(ExplorerDataError):
55
  """The loaded files do not form the expected Unified-3000 release."""
56
 
57
 
58
  class DraftAssetMissing(ExplorerDataError):
59
- """An available event is missing its required direct Draft EPG file."""
60
 
61
 
62
  class DraftIntegrityError(ExplorerDataError):
@@ -67,18 +76,12 @@ class InvalidEventId(ValueError):
67
  """An identifier is not a canonical Unified-3000 event ID."""
68
 
69
 
70
- @dataclass(frozen=True)
71
- class DraftUnavailable:
72
- event_id: str
73
- message: str = DRAFT_UNAVAILABLE_MESSAGE
74
-
75
-
76
  @dataclass
77
  class ExplorerRelease:
78
  events: pd.DataFrame
79
  stages: pd.DataFrame
80
  stages_by_event: dict[str, pd.DataFrame]
81
- revision: str = PUBLIC_DATASET_REVISION
82
 
83
  def event_row(self, event_id: str) -> pd.Series:
84
  validate_event_id(event_id)
@@ -93,7 +96,7 @@ class ExplorerRelease:
93
  self.event_row(event_id)
94
  frame = self.stages_by_event.get(event_id)
95
  if frame is None:
96
- return self.stages.iloc[0:0].copy()
97
  return frame.copy()
98
 
99
 
@@ -123,6 +126,13 @@ def resolve_dataset_file(filename: str, local_dataset_dir: Path | str | None = N
123
  raise FileNotFoundError(path)
124
  return path
125
 
 
 
 
 
 
 
 
126
  try:
127
  return Path(
128
  hf_hub_download(
@@ -141,23 +151,25 @@ def resolve_dataset_file(filename: str, local_dataset_dir: Path | str | None = N
141
  ) from exc
142
 
143
 
144
- def _read_required_parquet(
145
- filename: str,
146
- expected_schema: tuple[str, ...],
147
- local_dataset_dir: Path | str | None = None,
148
- ) -> pd.DataFrame:
149
- try:
150
- path = resolve_dataset_file(filename, local_dataset_dir=local_dataset_dir)
151
- arrow_schema = parquet.read_schema(path)
152
- frame = pd.read_parquet(path)
153
- except DatasetTransportError:
154
- raise
155
- except Exception as exc:
156
- raise ReleaseContractError(f"Unable to read required Parquet table: {filename}") from exc
 
 
 
157
 
158
- observed = tuple(frame.columns)
159
- observed_arrow = tuple((field.name, str(field.type)) for field in arrow_schema)
160
- expected_arrow = tuple(
161
  (
162
  column,
163
  "int64"
@@ -168,13 +180,67 @@ def _read_required_parquet(
168
  )
169
  for column in expected_schema
170
  )
171
- if observed != expected_schema or observed_arrow != expected_arrow:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
172
  raise ReleaseContractError(
173
  f"Schema mismatch for {filename}: expected {expected_arrow}, observed {observed_arrow}"
174
  )
175
  return frame
176
 
177
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
178
  def _require_row_count(frame: pd.DataFrame, expected: int, table_name: str) -> None:
179
  if len(frame) != expected:
180
  raise ReleaseContractError(
@@ -187,6 +253,23 @@ def _require_unique(frame: pd.DataFrame, columns: list[str], table_name: str) ->
187
  raise ReleaseContractError(f"{table_name} has duplicate identity rows for {columns}")
188
 
189
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
190
  def _require_equal_ids(frame: pd.DataFrame, table_name: str) -> None:
191
  if not frame["event_id"].equals(frame["public_event_id"]):
192
  raise ReleaseContractError(f"{table_name} event_id/public_event_id mismatch")
@@ -197,79 +280,159 @@ def _require_semantic_equality(
197
  other: pd.DataFrame,
198
  columns: tuple[str, ...],
199
  other_name: str,
 
 
 
200
  ) -> None:
201
- left = catalog.set_index("event_id").sort_index()
202
- right = other.set_index("event_id").sort_index()
203
  for column in columns:
204
  if not left[column].equals(right[column]):
205
  raise ReleaseContractError(f"{column} disagrees between catalog and {other_name}")
206
 
207
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
208
  def build_explorer_view(
 
209
  catalog: pd.DataFrame,
210
  instances: pd.DataFrame,
211
  summary: pd.DataFrame,
212
- availability: pd.DataFrame,
213
  ) -> pd.DataFrame:
214
- """Build the validated one-row-per-event Unified-3000 Explorer view."""
215
 
216
- for frame, schema, name in (
217
- (catalog, CATALOG_SCHEMA, "event_catalog"),
218
- (instances, EVENT_INSTANCES_SCHEMA, "event_instances"),
219
- (summary, FINALCASCADE_SUMMARY_SCHEMA, "finalcascade_summary"),
220
- (availability, DRAFT_AVAILABILITY_SCHEMA, "draft_availability"),
221
- ):
 
222
  if tuple(frame.columns) != schema:
223
  raise ReleaseContractError(f"Schema mismatch for {name}")
224
  _require_row_count(frame, EXPECTED_EVENT_COUNT, name)
 
 
 
 
 
 
 
 
 
 
 
 
225
 
226
- expected_ids = {f"H2EPR-{index:04d}" for index in range(EVENT_ID_MIN, EVENT_ID_MAX + 1)}
227
- _require_unique(catalog, ["public_event_id", "event_id"], "event_catalog")
228
- _require_unique(instances, ["public_event_id", "event_id"], "event_instances")
229
- _require_unique(summary, ["public_event_id", "event_id"], "finalcascade_summary")
230
- _require_unique(availability, ["public_event_id"], "draft_availability")
231
- _require_equal_ids(catalog, "event_catalog")
232
- _require_equal_ids(instances, "event_instances")
233
- _require_equal_ids(summary, "finalcascade_summary")
234
- if set(catalog["event_id"]) != expected_ids:
235
- raise ReleaseContractError("event_catalog does not contain the exact Unified-3000 ID set")
236
- if set(instances["event_id"]) != expected_ids or set(summary["event_id"]) != expected_ids:
237
- raise ReleaseContractError("instance/summary event identity does not match the catalog")
238
- if set(availability["public_event_id"]) != expected_ids:
239
- raise ReleaseContractError("availability identity does not match the catalog")
 
240
 
241
  _require_semantic_equality(
242
- catalog, instances, ("domain", "category", "draft_status"), "event_instances"
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
243
  )
244
  _require_semantic_equality(
245
- catalog, summary, ("domain", "category", "draft_status"), "finalcascade_summary"
 
 
 
246
  )
247
- catalog_status = catalog.set_index("public_event_id")["draft_status"].sort_index()
248
- availability_status = availability.set_index("public_event_id")["draft_status"].sort_index()
249
- if not catalog_status.equals(availability_status):
250
- raise ReleaseContractError("draft_status disagrees between catalog and availability")
251
-
252
- status_counts = availability["draft_status"].value_counts(dropna=False).to_dict()
253
- expected_status_counts = {
254
- "draft_available": EXPECTED_AVAILABLE_DRAFT_COUNT,
255
- "draft_unavailable": EXPECTED_UNAVAILABLE_DRAFT_COUNT,
256
  }
257
- if status_counts != expected_status_counts:
258
- raise ReleaseContractError(
259
- f"Draft availability mismatch: expected {expected_status_counts}, observed {status_counts}"
260
- )
261
 
262
- access_fields = [
263
- "public_event_id",
264
- "event_id",
265
- "has_finalcascade",
266
- "finalcascade_access_level",
267
- "gold_reference_access_level",
268
- "evidence_context_access_level",
269
- ]
270
  summary_fields = [
271
  "event_id",
272
- *GRAPH_COUNT_COLUMNS,
 
 
 
273
  "event_start_time",
274
  "event_end_time",
275
  "event_boundary_time_status",
@@ -277,106 +440,77 @@ def build_explorer_view(
277
  "known_action_time_anchors",
278
  "relative_order_available",
279
  ]
280
- availability_fields = [
281
- "public_event_id",
282
- "draft_source_kind",
283
- "draft_schema",
284
- "draft_asset",
285
- "draft_record_index",
286
- "draft_sha256",
287
- "source_payload_sha256",
288
- "has_reference_epg",
289
- ]
290
-
291
  try:
292
  joined = catalog.merge(
293
- instances[access_fields],
294
- on=["public_event_id", "event_id"],
295
- how="left",
296
- validate="one_to_one",
297
  )
298
  joined = joined.merge(
299
  summary[summary_fields], on="event_id", how="left", validate="one_to_one"
300
  )
301
  joined = joined.merge(
302
- availability[availability_fields],
303
- on="public_event_id",
304
- how="left",
305
- validate="one_to_one",
306
  )
307
  except Exception as exc:
308
  raise ReleaseContractError("Unified-3000 Explorer join multiplicity failure") from exc
309
 
310
  _require_row_count(joined, EXPECTED_EVENT_COUNT, "joined Explorer view")
 
311
  _require_unique(joined, ["public_event_id", "event_id"], "joined Explorer view")
312
- if joined[access_fields[2:] + availability_fields[1:]].isna().all(axis=1).any():
313
- raise ReleaseContractError("Joined Explorer view contains unmatched access/availability rows")
314
-
315
- available = joined["draft_status"].eq("draft_available")
316
- unavailable = joined["draft_status"].eq("draft_unavailable")
317
- if joined.loc[available, list(GRAPH_COUNT_COLUMNS)].isna().any().any():
318
- raise ReleaseContractError("Available drafts have null graph counts")
319
- if not joined.loc[unavailable, list(GRAPH_COUNT_COLUMNS)].isna().all().all():
320
- raise ReleaseContractError("Unavailable drafts contain observed graph counts")
321
- if not joined.loc[available, "has_finalcascade"].eq(True).all():
322
- raise ReleaseContractError("Available draft rows disagree with has_finalcascade")
323
- if not joined.loc[unavailable, "has_finalcascade"].eq(False).all():
324
- raise ReleaseContractError("Unavailable draft rows disagree with has_finalcascade")
325
- return joined.sort_values("event_id", kind="stable").reset_index(drop=True)
326
 
327
 
328
  def _validate_stages(stages: pd.DataFrame, events: pd.DataFrame) -> None:
329
  if tuple(stages.columns) != STAGES_SCHEMA:
330
  raise ReleaseContractError("Schema mismatch for event_stages")
331
  _require_row_count(stages, EXPECTED_STAGE_ROW_COUNT, "event_stages")
 
332
  _require_equal_ids(stages, "event_stages")
 
333
  if stages[["event_id", "stage_id"]].duplicated().any():
334
  raise ReleaseContractError("event_stages contains duplicate stage identity")
335
  if stages[["event_id", "stage_index"]].duplicated().any():
336
  raise ReleaseContractError("event_stages contains duplicate stage_index")
337
- invalid_stage_index = stages["stage_index"].map(
338
- lambda value: pd.isna(value) or int(value) != value or int(value) <= 0
 
 
 
 
339
  )
340
- if invalid_stage_index.any():
341
- raise ReleaseContractError("event_stages stage_index values must be positive integers")
342
- if not stages["event_id"].map(
343
- lambda value: isinstance(value, str) and bool(re.fullmatch(EVENT_ID_PATTERN, value))
344
- ).all():
345
- raise ReleaseContractError("event_stages contains a malformed event ID")
346
- available_ids = set(events.loc[events["draft_status"].eq("draft_available"), "event_id"])
347
- if set(stages["event_id"]) != available_ids:
348
- raise ReleaseContractError("Stage coverage does not equal the draft-available event set")
349
-
350
- available_summary = events.loc[
351
- events["draft_status"].eq("draft_available"),
352
- ["event_id", *GRAPH_COUNT_COLUMNS],
353
- ].set_index("event_id")
354
- for event_id in sorted(available_ids):
355
- event_stages = stages.loc[stages["event_id"].eq(event_id)]
356
- expected_stage_count = available_summary.at[event_id, "stage_count"]
357
- if (
358
- pd.isna(expected_stage_count)
359
- or int(expected_stage_count) != expected_stage_count
360
- or int(expected_stage_count) <= 0
361
- or len(event_stages) != int(expected_stage_count)
362
- ):
363
  raise ReleaseContractError(f"stage_count closure mismatch for {event_id}")
364
-
365
- observed_indices = sorted(int(value) for value in event_stages["stage_index"])
366
- expected_indices = list(range(1, int(expected_stage_count) + 1))
367
  if observed_indices != expected_indices:
368
  raise ReleaseContractError(f"Non-contiguous stage_index values for {event_id}")
369
-
370
  for column in GRAPH_COUNT_COLUMNS[1:]:
371
- expected_count = available_summary.at[event_id, column]
372
- observed_count = event_stages[column].sum(min_count=1)
373
- if pd.isna(expected_count) or pd.isna(observed_count) or observed_count != expected_count:
374
  raise ReleaseContractError(f"{column} closure mismatch for {event_id}")
 
 
 
 
 
375
 
376
 
377
  @lru_cache(maxsize=8)
378
  def _load_release_cached(local_root_value: str | None) -> ExplorerRelease:
379
  local_root = Path(local_root_value) if local_root_value else None
 
380
  catalog = _read_required_parquet(CATALOG_PARQUET, CATALOG_SCHEMA, local_root)
381
  instances = _read_required_parquet(
382
  EVENT_INSTANCES_PARQUET, EVENT_INSTANCES_SCHEMA, local_root
@@ -384,13 +518,10 @@ def _load_release_cached(local_root_value: str | None) -> ExplorerRelease:
384
  summary = _read_required_parquet(
385
  FINALCASCADE_SUMMARY_PARQUET, FINALCASCADE_SUMMARY_SCHEMA, local_root
386
  )
387
- availability = _read_required_parquet(
388
- DRAFT_AVAILABILITY_PARQUET, DRAFT_AVAILABILITY_SCHEMA, local_root
389
- )
390
  stages = _read_required_parquet(STAGES_PARQUET, STAGES_SCHEMA, local_root)
391
- events = build_explorer_view(catalog, instances, summary, availability)
 
392
  _validate_stages(stages, events)
393
- stages = stages.sort_values(["event_id", "stage_index", "stage_id"], kind="stable")
394
  stages_by_event = {
395
  event_id: group.reset_index(drop=True)
396
  for event_id, group in stages.groupby("event_id", sort=False)
@@ -434,8 +565,8 @@ def _validate_graph(payload: Any, event_id: str, event_row: pd.Series) -> dict[s
434
  raise DraftIntegrityError(f"Nested Draft EPG identity mismatch for {event_id}")
435
  if payload.get("source_payload_sha256") != event_row.get("source_payload_sha256"):
436
  raise DraftIntegrityError(f"Draft EPG source payload digest mismatch for {event_id}")
437
- if _canonical_graph_sha256(payload) != event_row.get("draft_sha256"):
438
- raise DraftIntegrityError(f"Draft EPG canonical digest mismatch for {event_id}")
439
  return payload
440
 
441
 
@@ -444,14 +575,14 @@ def _load_event_graph_cached(
444
  event_id: str,
445
  local_root_value: str | None,
446
  expected_source_sha256: str,
447
- expected_draft_sha256: str,
448
  ) -> dict[str, Any]:
449
  filename = DRAFT_EPG_PATH_TEMPLATE.format(event_id=event_id)
450
  local_root = Path(local_root_value) if local_root_value else None
451
  try:
452
  path = resolve_dataset_file(filename, local_dataset_dir=local_root)
453
  except FileNotFoundError as exc:
454
- raise DraftAssetMissing(f"Available Draft EPG file is missing: {filename}") from exc
455
  try:
456
  with path.open("r", encoding="utf-8") as handle:
457
  payload = json.load(handle)
@@ -460,7 +591,7 @@ def _load_event_graph_cached(
460
  expected = pd.Series(
461
  {
462
  "source_payload_sha256": expected_source_sha256,
463
- "draft_sha256": expected_draft_sha256,
464
  }
465
  )
466
  return _validate_graph(payload, event_id, expected)
@@ -471,21 +602,16 @@ def load_event_graph(
471
  *,
472
  release: ExplorerRelease | None = None,
473
  local_dataset_dir: Path | str | None = None,
474
- ) -> dict[str, Any] | DraftUnavailable:
475
- """Load one direct public Draft EPG after catalog and availability checks."""
476
 
477
  validate_event_id(event_id)
478
  selected_release = release or load_release(local_dataset_dir=local_dataset_dir)
479
  event_row = selected_release.event_row(event_id)
480
- if event_row["draft_status"] == "draft_unavailable":
481
- return DraftUnavailable(event_id)
482
- if event_row["draft_status"] != "draft_available":
483
- raise ReleaseContractError(f"Unknown draft_status for {event_id}")
484
-
485
  local_root = _as_local_root(local_dataset_dir)
486
  return _load_event_graph_cached(
487
  event_id,
488
  str(local_root) if local_root is not None else None,
489
  str(event_row["source_payload_sha256"]),
490
- str(event_row["draft_sha256"]),
491
  )
 
1
  from __future__ import annotations
2
 
3
+ import csv
4
  from dataclasses import dataclass
5
  from functools import lru_cache
6
  import hashlib
 
20
  ARROW_INT64_COLUMNS,
21
  CATALOG_PARQUET,
22
  CATALOG_SCHEMA,
 
 
23
  DRAFT_EPG_PATH_TEMPLATE,
24
+ DRAFT_SOURCE_HASHES_CSV,
25
+ DRAFT_SOURCE_HASHES_SCHEMA,
26
+ EVENT_GALLERY_PARQUET,
27
+ EVENT_GALLERY_SCHEMA,
28
  EVENT_ID_MAX,
29
  EVENT_ID_MIN,
30
  EVENT_ID_PATTERN,
31
  EVENT_INSTANCES_PARQUET,
32
  EVENT_INSTANCES_SCHEMA,
33
+ EVIDENCE_CONTEXT_ACCESS_LEVEL,
34
  EXPECTED_EVENT_COUNT,
35
  EXPECTED_STAGE_ROW_COUNT,
36
+ FINALCASCADE_ACCESS_LEVEL,
37
  FINALCASCADE_SUMMARY_PARQUET,
38
  FINALCASCADE_SUMMARY_SCHEMA,
39
+ GOLD_REFERENCE_ACCESS_LEVEL,
40
  GRAPH_COUNT_COLUMNS,
41
  LOCAL_DATASET_ENV,
42
  PUBLIC_DATASET_REPO,
43
  PUBLIC_DATASET_REVISION,
44
+ RELEASE_ASSET_SHA256,
45
  STAGES_PARQUET,
46
  STAGES_SCHEMA,
47
+ TABLE_SCHEMA_VERSIONS,
48
  )
49
 
50
 
 
56
  """A pinned public dataset asset could not be retrieved."""
57
 
58
 
59
+ class DatasetRevisionUnavailable(DatasetTransportError):
60
+ """Remote reads are disabled until an immutable release revision is set."""
61
+
62
+
63
  class ReleaseContractError(ExplorerDataError):
64
  """The loaded files do not form the expected Unified-3000 release."""
65
 
66
 
67
  class DraftAssetMissing(ExplorerDataError):
68
+ """A required per-event Draft EPG file is missing."""
69
 
70
 
71
  class DraftIntegrityError(ExplorerDataError):
 
76
  """An identifier is not a canonical Unified-3000 event ID."""
77
 
78
 
 
 
 
 
 
 
79
  @dataclass
80
  class ExplorerRelease:
81
  events: pd.DataFrame
82
  stages: pd.DataFrame
83
  stages_by_event: dict[str, pd.DataFrame]
84
+ revision: str | None = PUBLIC_DATASET_REVISION
85
 
86
  def event_row(self, event_id: str) -> pd.Series:
87
  validate_event_id(event_id)
 
96
  self.event_row(event_id)
97
  frame = self.stages_by_event.get(event_id)
98
  if frame is None:
99
+ raise ReleaseContractError(f"Validated release has no stage rows for {event_id}")
100
  return frame.copy()
101
 
102
 
 
126
  raise FileNotFoundError(path)
127
  return path
128
 
129
+ if PUBLIC_DATASET_REVISION is None:
130
+ raise DatasetRevisionUnavailable(
131
+ "Remote Explorer reads are disabled for this release candidate because no "
132
+ "immutable Hugging Face dataset revision has been assigned. Set "
133
+ f"{LOCAL_DATASET_ENV} to the validated local release root."
134
+ )
135
+
136
  try:
137
  return Path(
138
  hf_hub_download(
 
151
  ) from exc
152
 
153
 
154
+ def _file_sha256(path: Path) -> str:
155
+ digest = hashlib.sha256()
156
+ with path.open("rb") as handle:
157
+ for chunk in iter(lambda: handle.read(1024 * 1024), b""):
158
+ digest.update(chunk)
159
+ return digest.hexdigest()
160
+
161
+
162
+ def _require_release_asset_digest(path: Path, filename: str) -> None:
163
+ expected = RELEASE_ASSET_SHA256[filename]
164
+ observed = _file_sha256(path)
165
+ if observed != expected:
166
+ raise ReleaseContractError(
167
+ f"Release asset digest mismatch for {filename}: expected {expected}, observed {observed}"
168
+ )
169
+
170
 
171
+ def _expected_arrow_schema(expected_schema: tuple[str, ...]) -> tuple[tuple[str, str], ...]:
172
+ return tuple(
 
173
  (
174
  column,
175
  "int64"
 
180
  )
181
  for column in expected_schema
182
  )
183
+
184
+
185
+ def _read_required_parquet(
186
+ filename: str,
187
+ expected_schema: tuple[str, ...],
188
+ local_dataset_dir: Path | str | None = None,
189
+ ) -> pd.DataFrame:
190
+ try:
191
+ path = resolve_dataset_file(filename, local_dataset_dir=local_dataset_dir)
192
+ except (DatasetTransportError, FileNotFoundError):
193
+ raise
194
+ except Exception as exc:
195
+ raise ReleaseContractError(f"Unable to resolve required Parquet table: {filename}") from exc
196
+
197
+ _require_release_asset_digest(path, filename)
198
+ try:
199
+ table = parquet.read_table(path)
200
+ frame = table.to_pandas()
201
+ except Exception as exc:
202
+ raise ReleaseContractError(f"Unable to read required Parquet table: {filename}") from exc
203
+
204
+ observed_columns = tuple(frame.columns)
205
+ observed_arrow = tuple((field.name, str(field.type)) for field in table.schema)
206
+ expected_arrow = _expected_arrow_schema(expected_schema)
207
+ if observed_columns != expected_schema or observed_arrow != expected_arrow:
208
  raise ReleaseContractError(
209
  f"Schema mismatch for {filename}: expected {expected_arrow}, observed {observed_arrow}"
210
  )
211
  return frame
212
 
213
 
214
+ def _read_source_hash_registry(
215
+ local_dataset_dir: Path | str | None = None,
216
+ ) -> pd.DataFrame:
217
+ try:
218
+ path = resolve_dataset_file(DRAFT_SOURCE_HASHES_CSV, local_dataset_dir=local_dataset_dir)
219
+ except (DatasetTransportError, FileNotFoundError):
220
+ raise
221
+ except Exception as exc:
222
+ raise ReleaseContractError("Unable to resolve Draft EPG source-hash registry") from exc
223
+
224
+ _require_release_asset_digest(path, DRAFT_SOURCE_HASHES_CSV)
225
+ try:
226
+ with path.open("r", encoding="utf-8", newline="") as handle:
227
+ reader = csv.DictReader(handle)
228
+ if tuple(reader.fieldnames or ()) != DRAFT_SOURCE_HASHES_SCHEMA:
229
+ raise ReleaseContractError("Schema mismatch for draft_source_hashes")
230
+ rows = list(reader)
231
+ except ReleaseContractError:
232
+ raise
233
+ except Exception as exc:
234
+ raise ReleaseContractError("Unable to read Draft EPG source-hash registry") from exc
235
+
236
+ frame = pd.DataFrame(rows, columns=DRAFT_SOURCE_HASHES_SCHEMA)
237
+ try:
238
+ frame["draft_record_index"] = frame["draft_record_index"].map(int).astype("int64")
239
+ except (TypeError, ValueError) as exc:
240
+ raise ReleaseContractError("draft_record_index must contain decimal integers") from exc
241
+ return frame
242
+
243
+
244
  def _require_row_count(frame: pd.DataFrame, expected: int, table_name: str) -> None:
245
  if len(frame) != expected:
246
  raise ReleaseContractError(
 
253
  raise ReleaseContractError(f"{table_name} has duplicate identity rows for {columns}")
254
 
255
 
256
+ def _require_no_nulls(frame: pd.DataFrame, table_name: str) -> None:
257
+ columns = frame.columns[frame.isna().any()].tolist()
258
+ if columns:
259
+ raise ReleaseContractError(f"{table_name} contains null values in {columns}")
260
+
261
+
262
+ def _expected_event_ids() -> list[str]:
263
+ return [f"H2EPR-{index:04d}" for index in range(EVENT_ID_MIN, EVENT_ID_MAX + 1)]
264
+
265
+
266
+ def _require_exact_event_order(frame: pd.DataFrame, column: str, table_name: str) -> None:
267
+ if frame[column].tolist() != _expected_event_ids():
268
+ raise ReleaseContractError(
269
+ f"{table_name} does not contain the exact numeric Unified-3000 event order"
270
+ )
271
+
272
+
273
  def _require_equal_ids(frame: pd.DataFrame, table_name: str) -> None:
274
  if not frame["event_id"].equals(frame["public_event_id"]):
275
  raise ReleaseContractError(f"{table_name} event_id/public_event_id mismatch")
 
280
  other: pd.DataFrame,
281
  columns: tuple[str, ...],
282
  other_name: str,
283
+ *,
284
+ catalog_id: str = "event_id",
285
+ other_id: str = "event_id",
286
  ) -> None:
287
+ left = catalog.set_index(catalog_id)
288
+ right = other.set_index(other_id)
289
  for column in columns:
290
  if not left[column].equals(right[column]):
291
  raise ReleaseContractError(f"{column} disagrees between catalog and {other_name}")
292
 
293
 
294
+ def _require_schema_version(frame: pd.DataFrame, table_name: str) -> None:
295
+ expected = TABLE_SCHEMA_VERSIONS[table_name]
296
+ if not frame["schema_version"].eq(expected).all():
297
+ raise ReleaseContractError(f"{table_name} schema_version must be {expected}")
298
+
299
+
300
+ def _require_integer_range(
301
+ frame: pd.DataFrame,
302
+ columns: tuple[str, ...],
303
+ table_name: str,
304
+ *,
305
+ minimum: int,
306
+ ) -> None:
307
+ def invalid_integer(value: Any) -> bool:
308
+ try:
309
+ return bool(pd.isna(value)) or int(value) != value or int(value) < minimum
310
+ except (TypeError, ValueError):
311
+ return True
312
+
313
+ for column in columns:
314
+ invalid = frame[column].map(invalid_integer)
315
+ if invalid.any():
316
+ raise ReleaseContractError(
317
+ f"{table_name}.{column} must contain integers greater than or equal to {minimum}"
318
+ )
319
+
320
+
321
+ def _validate_source_hash_registry(source_hashes: pd.DataFrame) -> None:
322
+ if tuple(source_hashes.columns) != DRAFT_SOURCE_HASHES_SCHEMA:
323
+ raise ReleaseContractError("Schema mismatch for draft_source_hashes")
324
+ _require_row_count(source_hashes, EXPECTED_EVENT_COUNT, "draft_source_hashes")
325
+ _require_no_nulls(source_hashes, "draft_source_hashes")
326
+ _require_unique(source_hashes, ["public_event_id"], "draft_source_hashes")
327
+ _require_exact_event_order(source_hashes, "public_event_id", "draft_source_hashes")
328
+ if source_hashes["draft_record_index"].tolist() != list(
329
+ range(1, EXPECTED_EVENT_COUNT + 1)
330
+ ):
331
+ raise ReleaseContractError("draft_source_hashes has a non-canonical record index")
332
+ sha_pattern = re.compile(r"[0-9a-f]{64}")
333
+ for column in ("source_payload_sha256", "sanitized_record_sha256"):
334
+ if not source_hashes[column].map(
335
+ lambda value: isinstance(value, str) and bool(sha_pattern.fullmatch(value))
336
+ ).all():
337
+ raise ReleaseContractError(f"draft_source_hashes has malformed {column} values")
338
+ if source_hashes[column].duplicated().any():
339
+ raise ReleaseContractError(f"draft_source_hashes has duplicate {column} values")
340
+
341
+
342
  def build_explorer_view(
343
+ gallery: pd.DataFrame,
344
  catalog: pd.DataFrame,
345
  instances: pd.DataFrame,
346
  summary: pd.DataFrame,
347
+ source_hashes: pd.DataFrame,
348
  ) -> pd.DataFrame:
349
+ """Build the exact one-row-per-event Unified-3000 Explorer view."""
350
 
351
+ tables = (
352
+ (gallery, EVENT_GALLERY_SCHEMA, "event_gallery", "public_event_id"),
353
+ (catalog, CATALOG_SCHEMA, "event_catalog", "event_id"),
354
+ (instances, EVENT_INSTANCES_SCHEMA, "event_instances", "event_id"),
355
+ (summary, FINALCASCADE_SUMMARY_SCHEMA, "finalcascade_summary", "event_id"),
356
+ )
357
+ for frame, schema, name, order_column in tables:
358
  if tuple(frame.columns) != schema:
359
  raise ReleaseContractError(f"Schema mismatch for {name}")
360
  _require_row_count(frame, EXPECTED_EVENT_COUNT, name)
361
+ _require_no_nulls(frame, name)
362
+ _require_unique(frame, [order_column], name)
363
+ _require_exact_event_order(frame, order_column, name)
364
+ _require_schema_version(frame, name)
365
+
366
+ _validate_source_hash_registry(source_hashes)
367
+ for frame, name in (
368
+ (catalog, "event_catalog"),
369
+ (instances, "event_instances"),
370
+ (summary, "finalcascade_summary"),
371
+ ):
372
+ _require_equal_ids(frame, name)
373
 
374
+ _require_integer_range(
375
+ catalog,
376
+ ("stage_count", "episode_count"),
377
+ "event_catalog",
378
+ minimum=1,
379
+ )
380
+ _require_integer_range(summary, GRAPH_COUNT_COLUMNS, "finalcascade_summary", minimum=0)
381
+ if not summary["stage_count"].gt(0).all() or not summary["episode_count"].gt(0).all():
382
+ raise ReleaseContractError("Every Draft EPG must contain stages and episodes")
383
+ _require_integer_range(
384
+ summary,
385
+ ("known_action_time_anchor_count",),
386
+ "finalcascade_summary",
387
+ minimum=0,
388
+ )
389
 
390
  _require_semantic_equality(
391
+ catalog,
392
+ gallery,
393
+ ("title", "domain", "category", "event_descriptor"),
394
+ "event_gallery",
395
+ other_id="public_event_id",
396
+ )
397
+ _require_semantic_equality(
398
+ catalog,
399
+ instances,
400
+ (
401
+ "title",
402
+ "display_name",
403
+ "event_descriptor",
404
+ "domain",
405
+ "category",
406
+ "keywords",
407
+ "has_gold_reference",
408
+ ),
409
+ "event_instances",
410
  )
411
  _require_semantic_equality(
412
+ catalog,
413
+ summary,
414
+ ("title", "domain", "category", "stage_count", "episode_count"),
415
+ "finalcascade_summary",
416
  )
417
+
418
+ if not catalog["has_gold_reference"].eq(True).all():
419
+ raise ReleaseContractError("event_catalog has_gold_reference must be true for every event")
420
+ required_access = {
421
+ "finalcascade_access_level": FINALCASCADE_ACCESS_LEVEL,
422
+ "gold_reference_access_level": GOLD_REFERENCE_ACCESS_LEVEL,
423
+ "evidence_context_access_level": EVIDENCE_CONTEXT_ACCESS_LEVEL,
 
 
424
  }
425
+ for column, expected in required_access.items():
426
+ if not instances[column].eq(expected).all():
427
+ raise ReleaseContractError(f"event_instances.{column} must be {expected}")
 
428
 
429
+ access_fields = ["event_id", *required_access]
 
 
 
 
 
 
 
430
  summary_fields = [
431
  "event_id",
432
+ "participant_count",
433
+ "action_count",
434
+ "transaction_count",
435
+ "relation_count",
436
  "event_start_time",
437
  "event_end_time",
438
  "event_boundary_time_status",
 
440
  "known_action_time_anchors",
441
  "relative_order_available",
442
  ]
 
 
 
 
 
 
 
 
 
 
 
443
  try:
444
  joined = catalog.merge(
445
+ instances[access_fields], on="event_id", how="left", validate="one_to_one"
 
 
 
446
  )
447
  joined = joined.merge(
448
  summary[summary_fields], on="event_id", how="left", validate="one_to_one"
449
  )
450
  joined = joined.merge(
451
+ source_hashes, on="public_event_id", how="left", validate="one_to_one"
 
 
 
452
  )
453
  except Exception as exc:
454
  raise ReleaseContractError("Unified-3000 Explorer join multiplicity failure") from exc
455
 
456
  _require_row_count(joined, EXPECTED_EVENT_COUNT, "joined Explorer view")
457
+ _require_no_nulls(joined, "joined Explorer view")
458
  _require_unique(joined, ["public_event_id", "event_id"], "joined Explorer view")
459
+ _require_exact_event_order(joined, "event_id", "joined Explorer view")
460
+ return joined.reset_index(drop=True)
 
 
 
 
 
 
 
 
 
 
 
 
461
 
462
 
463
  def _validate_stages(stages: pd.DataFrame, events: pd.DataFrame) -> None:
464
  if tuple(stages.columns) != STAGES_SCHEMA:
465
  raise ReleaseContractError("Schema mismatch for event_stages")
466
  _require_row_count(stages, EXPECTED_STAGE_ROW_COUNT, "event_stages")
467
+ _require_no_nulls(stages, "event_stages")
468
  _require_equal_ids(stages, "event_stages")
469
+ _require_schema_version(stages, "event_stages")
470
  if stages[["event_id", "stage_id"]].duplicated().any():
471
  raise ReleaseContractError("event_stages contains duplicate stage identity")
472
  if stages[["event_id", "stage_index"]].duplicated().any():
473
  raise ReleaseContractError("event_stages contains duplicate stage_index")
474
+ _require_integer_range(stages, ("stage_index",), "event_stages", minimum=1)
475
+ _require_integer_range(
476
+ stages,
477
+ (*GRAPH_COUNT_COLUMNS[1:], "known_action_time_anchor_count"),
478
+ "event_stages",
479
+ minimum=0,
480
  )
481
+ if not stages["episode_count"].gt(0).all():
482
+ raise ReleaseContractError("Every stage must contain at least one episode")
483
+
484
+ expected_ids = _expected_event_ids()
485
+ observed_event_order = list(dict.fromkeys(stages["event_id"].tolist()))
486
+ if observed_event_order != expected_ids:
487
+ raise ReleaseContractError("event_stages does not cover all events in numeric order")
488
+
489
+ event_summary = events.set_index("event_id")
490
+ for event_id, event_stages in stages.groupby("event_id", sort=False):
491
+ expected_stage_count = int(event_summary.at[event_id, "stage_count"])
492
+ if len(event_stages) != expected_stage_count:
 
 
 
 
 
 
 
 
 
 
 
493
  raise ReleaseContractError(f"stage_count closure mismatch for {event_id}")
494
+ observed_indices = event_stages["stage_index"].astype(int).tolist()
495
+ expected_indices = list(range(1, expected_stage_count + 1))
 
496
  if observed_indices != expected_indices:
497
  raise ReleaseContractError(f"Non-contiguous stage_index values for {event_id}")
 
498
  for column in GRAPH_COUNT_COLUMNS[1:]:
499
+ observed_count = int(event_stages[column].sum())
500
+ expected_count = int(event_summary.at[event_id, column])
501
+ if observed_count != expected_count:
502
  raise ReleaseContractError(f"{column} closure mismatch for {event_id}")
503
+ expected_relative_order = bool(event_summary.at[event_id, "relative_order_available"])
504
+ if not event_stages["relative_order_available"].eq(expected_relative_order).all():
505
+ raise ReleaseContractError(
506
+ f"relative_order_available disagrees between event and stages for {event_id}"
507
+ )
508
 
509
 
510
  @lru_cache(maxsize=8)
511
  def _load_release_cached(local_root_value: str | None) -> ExplorerRelease:
512
  local_root = Path(local_root_value) if local_root_value else None
513
+ gallery = _read_required_parquet(EVENT_GALLERY_PARQUET, EVENT_GALLERY_SCHEMA, local_root)
514
  catalog = _read_required_parquet(CATALOG_PARQUET, CATALOG_SCHEMA, local_root)
515
  instances = _read_required_parquet(
516
  EVENT_INSTANCES_PARQUET, EVENT_INSTANCES_SCHEMA, local_root
 
518
  summary = _read_required_parquet(
519
  FINALCASCADE_SUMMARY_PARQUET, FINALCASCADE_SUMMARY_SCHEMA, local_root
520
  )
 
 
 
521
  stages = _read_required_parquet(STAGES_PARQUET, STAGES_SCHEMA, local_root)
522
+ source_hashes = _read_source_hash_registry(local_root)
523
+ events = build_explorer_view(gallery, catalog, instances, summary, source_hashes)
524
  _validate_stages(stages, events)
 
525
  stages_by_event = {
526
  event_id: group.reset_index(drop=True)
527
  for event_id, group in stages.groupby("event_id", sort=False)
 
565
  raise DraftIntegrityError(f"Nested Draft EPG identity mismatch for {event_id}")
566
  if payload.get("source_payload_sha256") != event_row.get("source_payload_sha256"):
567
  raise DraftIntegrityError(f"Draft EPG source payload digest mismatch for {event_id}")
568
+ if _canonical_graph_sha256(payload) != event_row.get("sanitized_record_sha256"):
569
+ raise DraftIntegrityError(f"Draft EPG sanitized record digest mismatch for {event_id}")
570
  return payload
571
 
572
 
 
575
  event_id: str,
576
  local_root_value: str | None,
577
  expected_source_sha256: str,
578
+ expected_sanitized_sha256: str,
579
  ) -> dict[str, Any]:
580
  filename = DRAFT_EPG_PATH_TEMPLATE.format(event_id=event_id)
581
  local_root = Path(local_root_value) if local_root_value else None
582
  try:
583
  path = resolve_dataset_file(filename, local_dataset_dir=local_root)
584
  except FileNotFoundError as exc:
585
+ raise DraftAssetMissing(f"Required Draft EPG file is missing: {filename}") from exc
586
  try:
587
  with path.open("r", encoding="utf-8") as handle:
588
  payload = json.load(handle)
 
591
  expected = pd.Series(
592
  {
593
  "source_payload_sha256": expected_source_sha256,
594
+ "sanitized_record_sha256": expected_sanitized_sha256,
595
  }
596
  )
597
  return _validate_graph(payload, event_id, expected)
 
602
  *,
603
  release: ExplorerRelease | None = None,
604
  local_dataset_dir: Path | str | None = None,
605
+ ) -> dict[str, Any]:
606
+ """Load one direct public Draft EPG after release and digest validation."""
607
 
608
  validate_event_id(event_id)
609
  selected_release = release or load_release(local_dataset_dir=local_dataset_dir)
610
  event_row = selected_release.event_row(event_id)
 
 
 
 
 
611
  local_root = _as_local_root(local_dataset_dir)
612
  return _load_event_graph_cached(
613
  event_id,
614
  str(local_root) if local_root is not None else None,
615
  str(event_row["source_payload_sha256"]),
616
+ str(event_row["sanitized_record_sha256"]),
617
  )
src/h2epr_explorer/navigation.py CHANGED
@@ -86,18 +86,36 @@ def resolve_selected_event_index(rows: list[dict[str, Any]], requested_event_id:
86
  return 0
87
 
88
 
89
- def build_event_links(event_id: str, *, draft_available: bool = False) -> dict[str, str]:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
90
  canonical = _canonical_id(event_id.strip())
91
  if not canonical:
92
  raise ValueError(f"Cannot build current-release links for event ID: {event_id!r}")
93
  links = {
94
  "explorer": f"{SPACE_URL}?event_id={canonical}",
95
- "public_dataset": f"{PUBLIC_DATASET_URL}/tree/{PUBLIC_DATASET_REVISION}",
96
  "reference_access": GOLD_COMPANION_URL,
97
  }
98
- if draft_available:
 
99
  draft_path = DRAFT_EPG_PATH_TEMPLATE.format(event_id=canonical)
100
- links["draft_epg"] = f"{PUBLIC_DATASET_URL}/blob/{PUBLIC_DATASET_REVISION}/{draft_path}"
101
  return links
102
 
103
 
 
86
  return 0
87
 
88
 
89
+ class ImmutableDatasetLinkUnavailable(RuntimeError):
90
+ """A content link cannot be built until the published revision is known."""
91
+
92
+
93
+ def build_immutable_dataset_link(path: str | None = None) -> str:
94
+ if not PUBLIC_DATASET_REVISION:
95
+ raise ImmutableDatasetLinkUnavailable(
96
+ "No immutable public dataset revision is assigned to this release candidate"
97
+ )
98
+ if path is None:
99
+ return f"{PUBLIC_DATASET_URL}/tree/{PUBLIC_DATASET_REVISION}"
100
+ normalized = path.strip("/")
101
+ if not normalized or ".." in normalized.split("/"):
102
+ raise ValueError(f"Invalid public dataset path: {path!r}")
103
+ return f"{PUBLIC_DATASET_URL}/blob/{PUBLIC_DATASET_REVISION}/{normalized}"
104
+
105
+
106
+ def build_event_links(event_id: str) -> dict[str, str]:
107
  canonical = _canonical_id(event_id.strip())
108
  if not canonical:
109
  raise ValueError(f"Cannot build current-release links for event ID: {event_id!r}")
110
  links = {
111
  "explorer": f"{SPACE_URL}?event_id={canonical}",
112
+ "public_dataset": PUBLIC_DATASET_URL,
113
  "reference_access": GOLD_COMPANION_URL,
114
  }
115
+ if PUBLIC_DATASET_REVISION:
116
+ links["public_dataset"] = build_immutable_dataset_link()
117
  draft_path = DRAFT_EPG_PATH_TEMPLATE.format(event_id=canonical)
118
+ links["draft_epg"] = build_immutable_dataset_link(draft_path)
119
  return links
120
 
121