AutoZYX commited on
Commit
d1430bc
·
verified ·
1 Parent(s): a6c9f19

Deploy PostgreSQL statistics 69f9770

Browse files
README.md CHANGED
@@ -25,9 +25,11 @@ short_description: 驭研科技航测自然驾驶数据 · PostgreSQL 自动统
25
  - 四张总览卡展示全库覆盖城市、采集批次、已知总时长和视频数;场景筛选使用代表帧统计。
26
  - 道路层级旭日图、标签柱图/环图、两维交叉分析及代表帧覆盖热力图。
27
  - 有效视频数、已知总时长和容量、按 GPS 行政区归属的城市分布。
 
28
  - 城市、采集日期、编码和时段筛选;采集时间、编码、分辨率、帧率、时长、入库与完整性分布。
29
  - 通过校验的自动标注结果:道路、路面、交通设施、动态目标、天气和光照等 27 个字段。
30
- - 场景示例图:按城市查看自动归组的采集点,每点一张、每页 12 个点;随当前 ODD 条件选取匹配代表帧,可展开完整标签并放大图片。
 
31
  - 标签观察状态总览、自选字段分布图、汇总 CSV;未知、未观察到与不适用单独统计。
32
  - 页面每 60 秒刷新,显示实际数据发布时间和同步存活检查时间。
33
 
@@ -39,6 +41,9 @@ short_description: 驭研科技航测自然驾驶数据 · PostgreSQL 自动统
39
 
40
  标签统计一批次一张代表帧,筛选按代表视频的城市、日期、编码和时段应用。
41
  单帧动态观察不代表关联视频全程;多选标签占比之和可能超过 100%。
 
 
 
42
  当前无需人工审核;纳入 `completed AND validation_passed`、代表视频有效且仍为
43
  批次成员的结果,排除明确退回和不符合完整字段契约的结果。结果明确标为自动标注。
44
 
@@ -69,9 +74,9 @@ published/frames/<thumbnail-key>.jpg
69
  私有 `NDS-Statistics-Data` 旧库保持原有权限,不再作为本站数据源。公开数据分支
70
  不改变入口仓库 `main` 的页面和域名,也不会触发每次数据更新都重建 GitHub Pages。
71
  数据未变化时只发布轻量存活状态,最多间隔 15 分钟,避免重复提交大文件。
72
- 批次更新采用 V2 公开结果,示例图关联采用 V3,采集点归组采用 V4;读取端继续兼容旧格式。
73
  发布器通过 `NDS_PUBLISH_BATCH_UPDATES=1`、`NDS_PUBLISH_REPRESENTATIVE_FRAMES=1`
74
- 和 `NDS_PUBLISH_LOCATION_GROUPS=1` 启用,
75
  先部署兼容读取端,再启用发布器。新增图片先以一次 Git 提交同步,随后发布引用它们的统计结果;
76
  图片同步失败不会推进本地发布状态,下次自动重试。未变化的原图使用私有缩略图缓存。
77
  Hugging Face 每 60 秒读取结果;这是分钟级自动同步,并非直连数据库的秒级订阅。
 
25
  - 四张总览卡展示全库覆盖城市、采集批次、已知总时长和视频数;场景筛选使用代表帧统计。
26
  - 道路层级旭日图、标签柱图/环图、两维交叉分析及代表帧覆盖热力图。
27
  - 有效视频数、已知总时长和容量、按 GPS 行政区归属的城市分布。
28
+ - 数据集城市统一显示英文名称和省份,例如 `Changchun (Jilin)`;直辖市显示 `Shanghai`,内部城市标识与筛选键保持不变。
29
  - 城市、采集日期、编码和时段筛选;采集时间、编码、分辨率、帧率、时长、入库与完整性分布。
30
  - 通过校验的自动标注结果:道路、路面、交通设施、动态目标、天气和光照等 27 个字段。
31
+ - 场景示例属于标签筛选结果:选择 ODD 标签或场景预设后,按城市查看自动归组采集点,每点一张、每页 12 个点;可展开完整标签并放大图片。
32
+ - 时间分布及涉及时段的交叉分析使用关联视频实际覆盖小时数;已知/未知时长和缺少覆盖的批次单独说明,不再用代表帧数量代替时长。
33
  - 标签观察状态总览、自选字段分布图、汇总 CSV;未知、未观察到与不适用单独统计。
34
  - 页面每 60 秒刷新,显示实际数据发布时间和同步存活检查时间。
35
 
 
41
 
42
  标签统计一批次一张代表帧,筛选按代表视频的城市、日期、编码和时段应用。
43
  单帧动态观察不代表关联视频全程;多选标签占比之和可能超过 100%。
44
+ 关联时长按匹配批次中未删除的视频实际 `duration_seconds` 累加,再除以 3600,
45
+ 不固定假定每帧半小时;成员视频也应用日期、编码和时段筛选,时段按每条视频采集时间归类。
46
+ 这些小时数表示关联视频覆盖,不代表天气或目标在全程持续存在。未知时长不估算,测得的零与未知分开处理。
47
  当前无需人工审核;纳入 `completed AND validation_passed`、代表视频有效且仍为
48
  批次成员的结果,排除明确退回和不符合完整字段契约的结果。结果明确标为自动标注。
49
 
 
74
  私有 `NDS-Statistics-Data` 旧库保持原有权限,不再作为本站数据源。公开数据分支
75
  不改变入口仓库 `main` 的页面和域名,也不会触发每次数据更新都重建 GitHub Pages。
76
  数据未变化时只发布轻量存活状态,最多间隔 15 分钟,避免重复提交大文件。
77
+ 批次更新采用 V2 公开结果,示例图关联采用 V3,采集点归组采用 V4,关联时长采用 V5;读取端继续兼容旧格式。
78
  发布器通过 `NDS_PUBLISH_BATCH_UPDATES=1`、`NDS_PUBLISH_REPRESENTATIVE_FRAMES=1`
79
+ 以及 `NDS_PUBLISH_LOCATION_GROUPS=1`、`NDS_PUBLISH_COVERAGE=1` 启用,
80
  先部署兼容读取端,再启用发布器。新增图片先以一次 Git 提交同步,随后发布引用它们的统计结果;
81
  图片同步失败不会推进本地发布状态,下次自动重试。未变化的原图使用私有缩略图缓存。
82
  Hugging Face 每 60 秒读取结果;这是分钟级自动同步,并非直连数据库的秒级订阅。
annotation_coverage.py ADDED
@@ -0,0 +1,164 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Public-safe associated video coverage for eligible representative batches.
2
+
3
+ Every active member contributes its full measured duration to the bucket of its
4
+ collection start. These durations describe associated video coverage, without
5
+ asserting that a representative frame's labels persist throughout the videos.
6
+ """
7
+ from __future__ import annotations
8
+
9
+ from collections.abc import Iterable
10
+ from dataclasses import dataclass
11
+ from datetime import date
12
+ from decimal import Decimal
13
+ import math
14
+ import re
15
+
16
+ import psycopg
17
+
18
+ from annotation_statistics import validate_public_labels
19
+ from live_inventory import _period_for_hour, _valid_number
20
+
21
+
22
+ _ELIGIBLE_BATCH_SQL = """
23
+ SELECT b.batch_id, b.labels
24
+ FROM aerial.annotation_batches b
25
+ JOIN aerial.videos representative
26
+ ON representative.sha256 = b.representative_video_sha256
27
+ WHERE b.processing_info->>'status' = 'completed'
28
+ AND b.validation_passed IS TRUE
29
+ AND b.review_status IS DISTINCT FROM 'rejected'
30
+ AND representative.deleted_at IS NULL
31
+ AND EXISTS (
32
+ SELECT 1 FROM aerial.annotation_batch_videos bv
33
+ WHERE bv.batch_id = b.batch_id
34
+ AND bv.video_sha256 = representative.sha256
35
+ )
36
+ ORDER BY b.batch_id
37
+ """
38
+ _MEMBER_SQL = """
39
+ SELECT bv.batch_id, v.sha256,
40
+ (v.collected_at AT TIME ZONE 'Asia/Shanghai')::date,
41
+ EXTRACT(HOUR FROM v.collected_at AT TIME ZONE 'Asia/Shanghai')::integer,
42
+ v.codec, v.duration_seconds
43
+ FROM aerial.annotation_batch_videos bv
44
+ JOIN aerial.videos v ON v.sha256 = bv.video_sha256
45
+ WHERE bv.batch_id = ANY(%s)
46
+ AND v.deleted_at IS NULL
47
+ ORDER BY bv.batch_id, v.sha256
48
+ """
49
+ _PUBLIC_CODE = re.compile(r'B-[0-9a-f]{16}')
50
+
51
+
52
+ @dataclass(frozen=True)
53
+ class CoverageSlice:
54
+ collected_on: date | None
55
+ period: str
56
+ codec: str
57
+ known_seconds: float
58
+ known_videos: int
59
+ unknown_videos: int
60
+
61
+
62
+ @dataclass(frozen=True)
63
+ class BatchCoverage:
64
+ batch_code: str
65
+ slices: tuple[CoverageSlice, ...]
66
+
67
+
68
+ @dataclass(frozen=True)
69
+ class CoverageCatalog:
70
+ records: tuple[BatchCoverage, ...]
71
+
72
+
73
+ def _slice_key(item: CoverageSlice) -> tuple[bool, date, str, str]:
74
+ return item.collected_on is None, item.collected_on or date.max, item.period, item.codec
75
+
76
+
77
+ def _validated_code(value: object) -> str:
78
+ if not isinstance(value, str) or not _PUBLIC_CODE.fullmatch(value):
79
+ raise ValueError('Invalid coverage batch code')
80
+ return value
81
+
82
+
83
+ def align_coverage_catalog(catalog: CoverageCatalog, batch_codes: Iterable[str]) -> CoverageCatalog:
84
+ """Match independent source coverage to the exact exported batch population.
85
+
86
+ Empty slices mark unavailable member coverage; a known zero remains a slice
87
+ with a positive known-video count. Reject duplicate source or export codes
88
+ rather than silently hiding an inconsistent publication population.
89
+ """
90
+ records = {}
91
+ for record in catalog.records:
92
+ code = _validated_code(record.batch_code)
93
+ if code in records:
94
+ raise ValueError('Duplicate coverage batch code')
95
+ records[code] = record
96
+ selected = set()
97
+ for value in batch_codes:
98
+ code = _validated_code(value)
99
+ if code in selected:
100
+ raise ValueError('Duplicate coverage batch code')
101
+ selected.add(code)
102
+ return CoverageCatalog(tuple(
103
+ BatchCoverage(code, tuple(sorted(records[code].slices, key=_slice_key)))
104
+ if code in records else BatchCoverage(code, ())
105
+ for code in sorted(selected)
106
+ ))
107
+
108
+
109
+ def fetch_coverage_catalog(dsn: str) -> CoverageCatalog:
110
+ """Read eligible labels once per batch, then only active member metadata.
111
+
112
+ Private identifiers are used locally for membership checks and discarded.
113
+ Duplicate membership rows inside a batch are counted once. Conflicting
114
+ active memberships across eligible batches fail with an identifier-free
115
+ error. Missing, negative and nonfinite durations count as unknown videos;
116
+ observed zero counts as a known duration. No duration is inferred.
117
+ """
118
+ # batch_updates imports the label validator too; keep a single safe B-code
119
+ # contract without creating an import-time dependency cycle.
120
+ from batch_updates import _public_code
121
+
122
+ with psycopg.connect(dsn, options='-c default_transaction_read_only=on') as connection:
123
+ with connection.cursor() as cursor:
124
+ cursor.execute(_ELIGIBLE_BATCH_SQL)
125
+ eligible = []
126
+ for batch_id, raw_labels in cursor.fetchall():
127
+ try:
128
+ validate_public_labels(raw_labels)
129
+ except ValueError:
130
+ continue
131
+ eligible.append(batch_id)
132
+ if not eligible:
133
+ return CoverageCatalog(())
134
+ cursor.execute(_MEMBER_SQL, (eligible,))
135
+ rows = cursor.fetchall()
136
+
137
+ buckets = {batch_id: {} for batch_id in eligible}
138
+ ownership = {}
139
+ for batch_id, video_id, collected_on, hour, codec, seconds in rows:
140
+ if video_id in ownership:
141
+ if ownership[video_id] != batch_id:
142
+ raise ValueError('Conflicting active video batch membership')
143
+ continue
144
+ ownership[video_id] = batch_id
145
+ key = (collected_on, _period_for_hour(hour),
146
+ codec.strip() if isinstance(codec, str) and codec.strip() else '未知')
147
+ bucket = buckets[batch_id].setdefault(key, [Decimal(0), 0, 0])
148
+ duration = _valid_number(seconds)
149
+ if duration is None:
150
+ bucket[2] += 1
151
+ else:
152
+ bucket[0] += duration
153
+ bucket[1] += 1
154
+
155
+ records = []
156
+ for batch_id, batch_buckets in buckets.items():
157
+ slices = []
158
+ for (collected_on, period, codec), (seconds, known, unknown) in batch_buckets.items():
159
+ total = float(seconds)
160
+ if not math.isfinite(total):
161
+ raise ValueError('Invalid coverage duration total')
162
+ slices.append(CoverageSlice(collected_on, period, codec, total, known, unknown))
163
+ records.append(BatchCoverage(_public_code(batch_id), tuple(sorted(slices, key=_slice_key))))
164
+ return CoverageCatalog(tuple(sorted(records, key=lambda record: record.batch_code)))
batch_updates_dashboard.py CHANGED
@@ -12,6 +12,7 @@ import plotly.graph_objects as go
12
  import streamlit as st
13
 
14
  from batch_updates import BatchUpdateCatalog, BatchUpdateRecord
 
15
 
16
  _BEIJING = ZoneInfo('Asia/Shanghai')
17
 
@@ -54,7 +55,7 @@ def batch_table(records: Iterable[BatchUpdateRecord], translate: Callable) -> pd
54
  columns = tuple(map(translate, ('批次编号','城市','活跃视频数','首次入库','最近入库',
55
  '源数据更新','标注完成','可用标签')))
56
  return pd.DataFrame([
57
- (record.code, record.city, record.videos, stamp(record.first_ingested_at),
58
  stamp(record.last_ingested_at), stamp(record.source_updated_at),
59
  stamp(record.annotation_completed_at), translate('有' if record.annotations_available else '暂无'))
60
  for record in records
@@ -122,10 +123,11 @@ def render_batch_updates(load_updates: Callable, translate: Callable) -> None:
122
  left, right = st.columns(2)
123
  query = left.text_input(tr('搜索批次编号或城市', 'Search batch code or city'), key='batch_updates_search').strip().casefold()
124
  cities = right.multiselect(tr('更新表城市', 'Batch table cities'), sorted({r.city for r in catalog.records}),
125
- key='batch_updates_cities')
126
  selected = tuple(record for record in catalog.records
127
  if (not cities or record.city in cities)
128
- and (not query or query in record.code.casefold() or query in record.city.casefold()))
 
129
  st.caption(tr(f'显示 {len(selected):,} / {len(catalog.records):,} 个批次;CSV 始终包含全部活跃批次。',
130
  f'Showing {len(selected):,} / {len(catalog.records):,} batches; CSV always includes all active batches.'))
131
  st.dataframe(batch_table(selected, table_translate), hide_index=True, width='stretch')
 
12
  import streamlit as st
13
 
14
  from batch_updates import BatchUpdateCatalog, BatchUpdateRecord
15
+ from city_labels import city_display_name
16
 
17
  _BEIJING = ZoneInfo('Asia/Shanghai')
18
 
 
55
  columns = tuple(map(translate, ('批次编号','城市','活跃视频数','首次入库','最近入库',
56
  '源数据更新','标注完成','可用标签')))
57
  return pd.DataFrame([
58
+ (record.code, city_display_name(record.city), record.videos, stamp(record.first_ingested_at),
59
  stamp(record.last_ingested_at), stamp(record.source_updated_at),
60
  stamp(record.annotation_completed_at), translate('有' if record.annotations_available else '暂无'))
61
  for record in records
 
123
  left, right = st.columns(2)
124
  query = left.text_input(tr('搜索批次编号或城市', 'Search batch code or city'), key='batch_updates_search').strip().casefold()
125
  cities = right.multiselect(tr('更新表城市', 'Batch table cities'), sorted({r.city for r in catalog.records}),
126
+ key='batch_updates_cities', format_func=city_display_name)
127
  selected = tuple(record for record in catalog.records
128
  if (not cities or record.city in cities)
129
+ and (not query or query in record.code.casefold() or query in record.city.casefold()
130
+ or query in city_display_name(record.city).casefold()))
131
  st.caption(tr(f'显示 {len(selected):,} / {len(catalog.records):,} 个批次;CSV 始终包含全部活跃批次。',
132
  f'Showing {len(selected):,} / {len(catalog.records):,} batches; CSV always includes all active batches.'))
133
  st.dataframe(batch_table(selected, table_translate), hide_index=True, width='stretch')
city_labels.py ADDED
@@ -0,0 +1,64 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """English city display labels derived only from the pinned boundary metadata.
2
+
3
+ Source records and filter keys retain their canonical city values. This module
4
+ reads feature properties once; it does not interpret geometry or resolve GPS.
5
+ """
6
+
7
+ from __future__ import annotations
8
+
9
+ from collections import defaultdict
10
+ from functools import lru_cache
11
+ import json
12
+ from pathlib import Path
13
+
14
+
15
+ _BOUNDARIES = Path(__file__).resolve().parent / 'data/prefectures-2023.geojson'
16
+ _PROVINCES = {
17
+ '11': 'Beijing', '12': 'Tianjin', '13': 'Hebei', '14': 'Shanxi',
18
+ '15': 'Inner Mongolia', '21': 'Liaoning', '22': 'Jilin', '23': 'Heilongjiang',
19
+ '31': 'Shanghai', '32': 'Jiangsu', '33': 'Zhejiang', '34': 'Anhui',
20
+ '35': 'Fujian', '36': 'Jiangxi', '37': 'Shandong', '41': 'Henan',
21
+ '42': 'Hubei', '43': 'Hunan', '44': 'Guangdong', '45': 'Guangxi',
22
+ '46': 'Hainan', '50': 'Chongqing', '51': 'Sichuan', '52': 'Guizhou',
23
+ '53': 'Yunnan', '54': 'Tibet', '61': 'Shaanxi', '62': 'Gansu',
24
+ '63': 'Qinghai', '64': 'Ningxia', '65': 'Xinjiang', '71': 'Taiwan',
25
+ '81': 'Hong Kong', '82': 'Macao',
26
+ }
27
+ _MUNICIPALITIES = frozenset(('11', '12', '31', '50'))
28
+ _CITY_STATUSES = {
29
+ '缺少坐标': 'Missing coordinates',
30
+ '边界外或坐标异常': 'Unassigned city',
31
+ }
32
+
33
+
34
+ @lru_cache(maxsize=1)
35
+ def _aliases() -> dict[str, str]:
36
+ aliases = {}
37
+ english_names = defaultdict(set)
38
+ features = json.loads(_BOUNDARIES.read_text(encoding='utf-8'))['features']
39
+ for feature in features:
40
+ properties = feature['properties']
41
+ name = properties['name']
42
+ prefix = str(properties['区划码'])[:2]
43
+ display = name if prefix in _MUNICIPALITIES else f'{name} ({_PROVINCES[prefix]})'
44
+ aliases[properties['地名'].casefold()] = display
45
+ aliases[display.casefold()] = display
46
+ english_names[name.casefold()].add(display)
47
+ # Bare romanizations are useful aliases only when they identify one place.
48
+ for name, displays in english_names.items():
49
+ if len(displays) == 1:
50
+ aliases[name] = next(iter(displays))
51
+ for canonical, display in _CITY_STATUSES.items():
52
+ aliases[canonical.casefold()] = display
53
+ aliases[display.casefold()] = display
54
+ return aliases
55
+
56
+
57
+ def city_display_name(city: str) -> str:
58
+ """Format a known city or alias, preserving unknown future source values."""
59
+ return _aliases().get(city.strip().casefold(), city)
60
+
61
+
62
+ def is_city_label(value: str) -> bool:
63
+ """Recognize city/status values without treating generic unknown as a city."""
64
+ return value.strip().casefold() in _aliases()
classic_dashboard.py CHANGED
@@ -17,6 +17,7 @@ import streamlit as st
17
 
18
  from live_inventory import summarize_records
19
  from postgres_dashboard import aggregate_rows, selected_records
 
20
 
21
  _NAVY = '#0f1923'
22
  _ORANGE = '#c85a3a'
@@ -120,11 +121,13 @@ def _reset_filters(bounds: tuple | None) -> None:
120
  def render_classic_dashboard(load_catalog: Callable, load_annotations: Callable,
121
  load_updates: Callable, clear_data: Callable,
122
  translate: Callable, render_source_notice: Callable | None = None, *,
123
- load_locations: Callable | None = None) -> None:
 
124
  """Render inventory and batch-frame statistics with optional point examples."""
125
  from classic_statistics import (
126
  SCENARIOS, coverage_table, cross_table, field_options, filter_odd_records,
127
  hierarchy_rows, scenario_counts, selection_label, tag_counts,
 
128
  )
129
 
130
  language = st.session_state.get('lang', 'zh')
@@ -158,13 +161,19 @@ def render_classic_dashboard(load_catalog: Callable, load_annotations: Callable,
158
  figure.update_layout(height=height)
159
  st.plotly_chart(figure, width='stretch', key=key, config={'displaylogo': False})
160
 
161
- def bar(counts: dict, title: str) -> None:
 
162
  items = sorted(counts.items(), key=lambda pair: (-pair[1], pair[0]))
163
- figure = go.Figure(go.Bar(x=[value_label(v) for v, _ in items], y=[n for _, n in items],
 
164
  text=[n for _, n in items], textposition='outside',
165
  marker=dict(color=[_PALETTE[i % len(_PALETTE)] for i in range(len(items))],
166
  line_width=0), opacity=.88))
167
- show_chart(figure, title)
 
 
 
 
168
 
169
  @st.fragment(run_every='60s')
170
  def current() -> None:
@@ -241,15 +250,15 @@ def render_classic_dashboard(load_catalog: Callable, load_annotations: Callable,
241
  st.markdown(f"<div class='kpi-card {style}'><div class='kpi-num'>{value}</div>"
242
  f"<div class='kpi-lbl'>{html.escape(label)}</div></div>", unsafe_allow_html=True)
243
  if detail == 'batches':
244
- st.markdown("<a class='example-entry' href='#nav-examples'>"
245
- + html.escape(tr('查看场景示例 →', 'Browse scene examples →'))
246
  + '</a>', unsafe_allow_html=True)
247
  with st.expander(tr('展开详情', 'View details')):
248
  if detail == 'batches':
249
  st.caption(tr('全部有效关联采集批次;不按城市、日期、编码、时段或代表帧 ODD 筛选缩减。',
250
  'All active linked collection batches; independent of city, date, codec, period and frame ODD filters.'))
251
  if batch_records is not None:
252
- st.dataframe(pd.DataFrame(sorted(Counter(r.city for r in batch_records).items()),
253
  columns=[tr('城市', 'City'), tr('批次数', 'Batches')]),
254
  hide_index=True, width='stretch')
255
  elif detail == 'duration':
@@ -259,6 +268,7 @@ def render_classic_dashboard(load_catalog: Callable, load_annotations: Callable,
259
  safe = aggregate_rows(inventory_records)
260
  if not safe.empty:
261
  grouped = safe.groupby('city', as_index=False)['videos'].sum()
 
262
  st.dataframe(grouped, hide_index=True, width='stretch')
263
  st.caption(tr(
264
  '总览统计当前有效视频库存;下方筛选用于代表帧和分布图。采集批次为全部有效关联批次。'
@@ -282,7 +292,7 @@ def render_classic_dashboard(load_catalog: Callable, load_annotations: Callable,
282
  if mapped:
283
  figure = go.Figure(go.Scattermap(
284
  lat=[centers[c].latitude for c in mapped], lon=[centers[c].longitude for c in mapped],
285
- mode='markers+text', text=[translate(c) for c in mapped], textposition='top center',
286
  textfont=dict(size=13, color=_NAVY),
287
  marker=dict(size=[14 + min(city_counts[c] ** .5, 24) for c in mapped],
288
  color=[_ORANGE if c in selected_cities else _BLUE for c in mapped], opacity=.85),
@@ -302,7 +312,7 @@ def render_classic_dashboard(load_catalog: Callable, load_annotations: Callable,
302
  st.caption(tr(f'另有 {missing:,} 条视频无法归属地图中的城市,未显示点位。',
303
  f'{missing:,} videos could not be mapped to an administrative city.'))
304
  city_options = sorted({r.city for r in catalog.records} | selected_cities)
305
- st.multiselect(tr('城市', 'City'), city_options, key='pg_cities', format_func=translate)
306
  with st.expander(tr('采集日期、编码与时段', 'Collection date, codec and period')):
307
  left, right = st.columns(2)
308
  with left:
@@ -355,19 +365,64 @@ def render_classic_dashboard(load_catalog: Callable, load_annotations: Callable,
355
  unsafe_allow_html=True)
356
  st.caption(tr('未知、未观察到、未标注与不适用分别筛选;“未观察到”不表示视频全程不存在该目标。每 60 秒刷新。',
357
  'Unknown, not observed, unlabeled and not applicable are distinct. Not observed in a frame does not imply absence throughout a video. Refreshes every 60 seconds.'))
358
- if annotation_available:
359
  from representative_gallery import render_representative_gallery
360
  locations, location_status = load_locations() if load_locations else (None, 'not_published')
361
  render_representative_gallery(matched, translate, locations=locations, location_status=location_status,
362
  language=language, field_label=field_label)
 
 
 
363
  if annotation_available and matched:
364
  heading('地域与时间分布', 'Geographic & temporal distribution',
365
- '当前匹配代表帧的城市与采集时段分布', 'Cities and collection periods of matching representative frames', 'nav-params')
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
366
  left, right = st.columns(2)
367
  with left:
368
- bar(dict(Counter(r.city for r in matched)), tr('各城市代表帧数量', 'Representative frames by city'))
369
  with right:
370
- bar(dict(Counter(r.period for r in matched)), tr('采集时段分布', 'Collection period distribution'))
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
371
  heading('道路类型分布', 'Road type distribution',
372
  '道路大类、子类与交叉类型;多选交叉类型按每帧总权重 1 分摊。',
373
  'Road category, subcategory and intersection; multiple values share a total weight of one per frame.')
@@ -404,10 +459,21 @@ def render_classic_dashboard(load_catalog: Callable, load_annotations: Callable,
404
  row = st.selectbox(tr('维度 1', 'Dimension 1'), dimensions, format_func=field_label, key='classic_cross_row')
405
  with right:
406
  col = st.selectbox(tr('维度 2', 'Dimension 2'), dimensions, index=1, format_func=field_label, key='classic_cross_column')
407
- table = cross_table(matched, row, col)
408
- table.index = [value_label(str(v)) for v in table.index]
409
- table.columns = [value_label(str(v)) for v in table.columns]
410
- st.dataframe(table, width='stretch')
 
 
 
 
 
 
 
 
 
 
 
411
  heading('ODD 覆盖度', 'ODD coverage',
412
  '道路子类 × 天气的代表帧观察密度。0 表示没有样本观察,不证明关联视频全程不存在该条件。',
413
  'Road subcategory × weather observations. Zero means no representative-frame observation, not proven absence throughout videos.')
 
17
 
18
  from live_inventory import summarize_records
19
  from postgres_dashboard import aggregate_rows, selected_records
20
+ from city_labels import city_display_name
21
 
22
  _NAVY = '#0f1923'
23
  _ORANGE = '#c85a3a'
 
121
  def render_classic_dashboard(load_catalog: Callable, load_annotations: Callable,
122
  load_updates: Callable, clear_data: Callable,
123
  translate: Callable, render_source_notice: Callable | None = None, *,
124
+ load_locations: Callable | None = None,
125
+ load_coverage: Callable | None = None) -> None:
126
  """Render inventory and batch-frame statistics with optional point examples."""
127
  from classic_statistics import (
128
  SCENARIOS, coverage_table, cross_table, field_options, filter_odd_records,
129
  hierarchy_rows, scenario_counts, selection_label, tag_counts,
130
+ coverage_cross_table, period_hours, selected_member_coverage,
131
  )
132
 
133
  language = st.session_state.get('lang', 'zh')
 
161
  figure.update_layout(height=height)
162
  st.plotly_chart(figure, width='stretch', key=key, config={'displaylogo': False})
163
 
164
+ def bar(counts: dict, title: str, *, dimension: str | None = None,
165
+ key: str | None = None) -> None:
166
  items = sorted(counts.items(), key=lambda pair: (-pair[1], pair[0]))
167
+ label = city_display_name if dimension == 'city' else value_label
168
+ figure = go.Figure(go.Bar(x=[label(v) for v, _ in items], y=[n for _, n in items],
169
  text=[n for _, n in items], textposition='outside',
170
  marker=dict(color=[_PALETTE[i % len(_PALETTE)] for i in range(len(items))],
171
  line_width=0), opacity=.88))
172
+ if key == 'classic_period_hours':
173
+ figure.update_traces(text=[f'{n:.2f}' for _, n in items],
174
+ hovertemplate='%{x}<br>%{y:.3f} ' + tr('小时', 'hours') + '<extra></extra>')
175
+ figure.update_yaxes(title_text=tr('小时', 'Hours'))
176
+ show_chart(figure, title, key=key)
177
 
178
  @st.fragment(run_every='60s')
179
  def current() -> None:
 
250
  st.markdown(f"<div class='kpi-card {style}'><div class='kpi-num'>{value}</div>"
251
  f"<div class='kpi-lbl'>{html.escape(label)}</div></div>", unsafe_allow_html=True)
252
  if detail == 'batches':
253
+ st.markdown("<a class='example-entry' href='#nav-odd'>"
254
+ + html.escape(tr('筛选标签查看场景示例 →', 'Filter tags to view scene examples →'))
255
  + '</a>', unsafe_allow_html=True)
256
  with st.expander(tr('展开详情', 'View details')):
257
  if detail == 'batches':
258
  st.caption(tr('全部有效关联采集批次;不按城市、日期、编码、时段或代表帧 ODD 筛选缩减。',
259
  'All active linked collection batches; independent of city, date, codec, period and frame ODD filters.'))
260
  if batch_records is not None:
261
+ st.dataframe(pd.DataFrame(sorted(Counter(city_display_name(r.city) for r in batch_records).items()),
262
  columns=[tr('城市', 'City'), tr('批次数', 'Batches')]),
263
  hide_index=True, width='stretch')
264
  elif detail == 'duration':
 
268
  safe = aggregate_rows(inventory_records)
269
  if not safe.empty:
270
  grouped = safe.groupby('city', as_index=False)['videos'].sum()
271
+ grouped['city'] = grouped['city'].map(city_display_name)
272
  st.dataframe(grouped, hide_index=True, width='stretch')
273
  st.caption(tr(
274
  '总览统计当前有效视频库存;下方筛选用于代表帧和分布图。采集批次为全部有效关联批次。'
 
292
  if mapped:
293
  figure = go.Figure(go.Scattermap(
294
  lat=[centers[c].latitude for c in mapped], lon=[centers[c].longitude for c in mapped],
295
+ mode='markers+text', text=[city_display_name(c) for c in mapped], textposition='top center',
296
  textfont=dict(size=13, color=_NAVY),
297
  marker=dict(size=[14 + min(city_counts[c] ** .5, 24) for c in mapped],
298
  color=[_ORANGE if c in selected_cities else _BLUE for c in mapped], opacity=.85),
 
312
  st.caption(tr(f'另有 {missing:,} 条视频无法归属地图中的城市,未显示点位。',
313
  f'{missing:,} videos could not be mapped to an administrative city.'))
314
  city_options = sorted({r.city for r in catalog.records} | selected_cities)
315
+ st.multiselect(tr('城市', 'City'), city_options, key='pg_cities', format_func=city_display_name)
316
  with st.expander(tr('采集日期、编码与时段', 'Collection date, codec and period')):
317
  left, right = st.columns(2)
318
  with left:
 
365
  unsafe_allow_html=True)
366
  st.caption(tr('未知、未观察到、未标注与不适用分别筛选;“未观察到”不表示视频全程不存在该目标。每 60 秒刷新。',
367
  'Unknown, not observed, unlabeled and not applicable are distinct. Not observed in a frame does not imply absence throughout a video. Refreshes every 60 seconds.'))
368
+ if annotation_available and any(selections.values()):
369
  from representative_gallery import render_representative_gallery
370
  locations, location_status = load_locations() if load_locations else (None, 'not_published')
371
  render_representative_gallery(matched, translate, locations=locations, location_status=location_status,
372
  language=language, field_label=field_label)
373
+ elif annotation_available:
374
+ st.caption(tr('选择上方至少一个 ODD 标签或场景预设,即可查看匹配的场景示例。',
375
+ 'Select an ODD tag or scenario preset above to view matching scene examples.'))
376
  if annotation_available and matched:
377
  heading('地域与时间分布', 'Geographic & temporal distribution',
378
+ '城市按代表帧计数;采集时段按关联有效视频的实际时长计量。',
379
+ 'Cities count representative frames; collection periods measure actual duration of associated active videos.', 'nav-params')
380
+ coverage_catalog, coverage_status = load_coverage() if load_coverage else (None, 'not_published')
381
+ member_coverage = selected_member_coverage(
382
+ matched, coverage_catalog, date_range=st.session_state.get('pg_date_range', ()),
383
+ include_unknown_dates=st.session_state['pg_include_unknown_dates'],
384
+ codecs=st.session_state['pg_codecs'], periods=st.session_state['pg_periods'],
385
+ ) if coverage_catalog is not None and coverage_status == 'ok' else None
386
+
387
+ def coverage_notice() -> None:
388
+ if coverage_status in ('not_published', 'not_configured'):
389
+ st.info(tr('关联视频时长尚未发布,暂无法显示时间统计。',
390
+ 'Associated video duration has not been published; temporal statistics are unavailable.'))
391
+ else:
392
+ st.warning(tr('关联视频时长暂不可用,请稍后刷新。',
393
+ 'Associated video duration is unavailable. Please refresh later.'))
394
+
395
  left, right = st.columns(2)
396
  with left:
397
+ bar(dict(Counter(r.city for r in matched)), tr('各城市代表帧数量', 'Representative frames by city'), dimension='city')
398
  with right:
399
+ if member_coverage is None:
400
+ coverage_notice()
401
+ elif not member_coverage.members and not member_coverage.missing_batch_codes:
402
+ st.info(tr('当前匹配批次中没有关联视频符合日期、编码及时段筛选。',
403
+ 'No associated videos in matching batches pass the date, codec and period filters.'))
404
+ elif member_coverage.known_hours is None:
405
+ st.info(tr('当前匹配的关联视频时长均未知或缺少覆盖,无法计算小时数。',
406
+ 'Associated video durations are all unknown or coverage is missing; hours cannot be calculated.'))
407
+ else:
408
+ hours_by_period = period_hours(member_coverage)
409
+ bar({period: hours for period, hours in hours_by_period.items() if hours is not None},
410
+ tr('关联视频覆盖(小时)', 'Associated video coverage (hours)'), key='classic_period_hours')
411
+ unknown_periods = [value_label(period) for period, hours in hours_by_period.items() if hours is None]
412
+ if unknown_periods:
413
+ st.caption(tr('以下时段的时长均未知:', 'Durations are all unknown for these periods: ')
414
+ + ', '.join(unknown_periods))
415
+ if member_coverage is not None:
416
+ st.caption(tr(
417
+ f'已知时长视频 {member_coverage.known_videos:,};未知时长视频 {member_coverage.unknown_videos:,};'
418
+ f'缺少覆盖的批次 {len(member_coverage.missing_batch_codes):,}。',
419
+ f'Known-duration videos: {member_coverage.known_videos:,}; unknown-duration videos: {member_coverage.unknown_videos:,}; '
420
+ f'batches missing coverage: {len(member_coverage.missing_batch_codes):,}.'))
421
+ st.caption(tr(
422
+ '小时数按全部匹配批次关联的有效视频实际时长求和,不按地点示例去重;未知时长不估算。'
423
+ '时段取每条视频的采集时间,不拆分视频;不证明天气或目标在关联视频中持续存在。',
424
+ 'Hours sum actual duration of active videos in every matching batch, before point-example deduplication; unknown durations are not estimated. '
425
+ 'Each video uses its collection timestamp period without splitting. Frame labels do not prove weather or object persistence through associated videos.'))
426
  heading('道路类型分布', 'Road type distribution',
427
  '道路大类、子类与交叉类型;多选交叉类型按每帧总权重 1 分摊。',
428
  'Road category, subcategory and intersection; multiple values share a total weight of one per frame.')
 
459
  row = st.selectbox(tr('维度 1', 'Dimension 1'), dimensions, format_func=field_label, key='classic_cross_row')
460
  with right:
461
  col = st.selectbox(tr('维度 2', 'Dimension 2'), dimensions, index=1, format_func=field_label, key='classic_cross_column')
462
+ if 'period' in (row, col):
463
+ table = coverage_cross_table(member_coverage, row, col) if member_coverage is not None else None
464
+ if table is None:
465
+ coverage_notice()
466
+ else:
467
+ st.caption(tr('单位:关联视频覆盖小时;多选标签均分时长。空值表示该组合时长均未知。',
468
+ 'Unit: associated video coverage hours. Multiple tag values share duration equally; blank cells have unknown duration.'))
469
+ else:
470
+ table = cross_table(matched, row, col)
471
+ st.caption(tr('单位:代表帧;多选维度按每帧总权重 1 分摊。',
472
+ 'Unit: representative frames; multiple values share a total weight of one per frame.'))
473
+ if table is not None:
474
+ table.index = pd.Index([city_display_name(str(v)) if row == 'city' else value_label(str(v)) for v in table.index], name=row)
475
+ table.columns = pd.Index([city_display_name(str(v)) if col == 'city' else value_label(str(v)) for v in table.columns], name=col)
476
+ st.dataframe(table, width='stretch')
477
  heading('ODD 覆盖度', 'ODD coverage',
478
  '道路子类 × 天气的代表帧观察密度。0 表示没有样本观察,不证明关联视频全程不存在该条件。',
479
  'Road subcategory × weather observations. Zero means no representative-frame observation, not proven absence throughout videos.')
classic_statistics.py CHANGED
@@ -9,12 +9,18 @@ from __future__ import annotations
9
 
10
  from collections import Counter
11
  from collections.abc import Iterable, Mapping, Sequence
 
 
12
  from itertools import product
 
13
 
14
  import pandas as pd
15
 
16
  from annotation_statistics import AnnotationRecord, definitions
17
 
 
 
 
18
 
19
  _FIELDS = {item["code"]: item for item in definitions()}
20
  _STATES = ("@unknown", "@not_observed", "@unlabeled", "@not_applicable")
@@ -28,7 +34,7 @@ SCENARIOS = [
28
  {"key": "structured", "zh": "结构化道路场景", "en": "Structured roads",
29
  "filters": {"top_road_subcategory": ["高速公路", "快速路"]}},
30
  {"key": "urban", "zh": "城市道路", "en": "Urban roads",
31
- "filters": {"top_road_subcategory": ["主干路", "次干路", "支路", "街巷"]}},
32
  {"key": "parking", "zh": "停车场区域", "en": "Parking areas",
33
  "filters": {"top_road_subcategory": ["室外停车场", "室内停车场", "路侧停车位"]}},
34
  {"key": "low_light", "zh": "低光场景", "en": "Low-light scenes",
@@ -145,6 +151,113 @@ def cross_table(
145
  return table
146
 
147
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
148
  def coverage_table(
149
  records: Iterable[AnnotationRecord], universe: Iterable[AnnotationRecord] | None = None,
150
  ) -> pd.DataFrame:
 
9
 
10
  from collections import Counter
11
  from collections.abc import Iterable, Mapping, Sequence
12
+ from dataclasses import dataclass
13
+ from datetime import date
14
  from itertools import product
15
+ from typing import TYPE_CHECKING
16
 
17
  import pandas as pd
18
 
19
  from annotation_statistics import AnnotationRecord, definitions
20
 
21
+ if TYPE_CHECKING:
22
+ from annotation_coverage import CoverageCatalog, CoverageSlice
23
+
24
 
25
  _FIELDS = {item["code"]: item for item in definitions()}
26
  _STATES = ("@unknown", "@not_observed", "@unlabeled", "@not_applicable")
 
34
  {"key": "structured", "zh": "结构化道路场景", "en": "Structured roads",
35
  "filters": {"top_road_subcategory": ["高速公路", "快速路"]}},
36
  {"key": "urban", "zh": "城市道路", "en": "Urban roads",
37
+ "filters": {"top_road_category": ["城市道路"]}},
38
  {"key": "parking", "zh": "停车场区域", "en": "Parking areas",
39
  "filters": {"top_road_subcategory": ["室外停车场", "室内停车场", "路侧停车位"]}},
40
  {"key": "low_light", "zh": "低光场景", "en": "Low-light scenes",
 
151
  return table
152
 
153
 
154
+ @dataclass(frozen=True)
155
+ class SelectedCoverage:
156
+ """Selected member buckets with the exact frame supplying their ODD labels."""
157
+
158
+ members: tuple[tuple[AnnotationRecord, CoverageSlice], ...]
159
+ missing_batch_codes: tuple[str | None, ...]
160
+
161
+ @property
162
+ def known_seconds(self) -> float:
163
+ return sum(bucket.known_seconds for _, bucket in self.members)
164
+
165
+ @property
166
+ def known_videos(self) -> int:
167
+ return sum(bucket.known_videos for _, bucket in self.members)
168
+
169
+ @property
170
+ def unknown_videos(self) -> int:
171
+ return sum(bucket.unknown_videos for _, bucket in self.members)
172
+
173
+ @property
174
+ def known_hours(self) -> float | None:
175
+ return self.known_seconds / 3600 if self.known_videos else None
176
+
177
+
178
+ def selected_member_coverage(
179
+ records: Iterable[AnnotationRecord], catalog: CoverageCatalog, *,
180
+ date_range: Sequence[date] | date = (), include_unknown_dates: bool = True,
181
+ codecs: Iterable[str] = (), periods: Iterable[str] = (),
182
+ ) -> SelectedCoverage:
183
+ """Join matched frame batches first, then filter their actual member metadata.
184
+
185
+ Empty source records remain missing coverage even if filters exclude all
186
+ buckets of another batch. Repeated frame inputs never multiply batch hours.
187
+ """
188
+ dates = (date_range, date_range) if isinstance(date_range, date) else date_range
189
+ start, end = (dates[0], dates[-1]) if dates else (None, None)
190
+ codecs, periods = set(codecs), set(periods)
191
+ batches = {batch.batch_code: batch for batch in catalog.records}
192
+ frames = {record.batch_code: record for record in records}
193
+ members, missing = [], []
194
+ for code, record in frames.items():
195
+ batch = batches.get(code)
196
+ if batch is None or not batch.slices:
197
+ missing.append(code)
198
+ continue
199
+ for bucket in batch.slices:
200
+ if codecs and bucket.codec not in codecs:
201
+ continue
202
+ if periods and bucket.period not in periods:
203
+ continue
204
+ if bucket.collected_on is None:
205
+ if not include_unknown_dates:
206
+ continue
207
+ elif ((start is not None and bucket.collected_on < start)
208
+ or (end is not None and bucket.collected_on > end)):
209
+ continue
210
+ members.append((record, bucket))
211
+ return SelectedCoverage(tuple(members), tuple(missing))
212
+
213
+
214
+ def period_hours(coverage: SelectedCoverage) -> dict[str, float | None]:
215
+ """Actual member hours by timestamp-derived period; None means all unknown."""
216
+ seconds, known = Counter(), Counter()
217
+ for _, bucket in coverage.members:
218
+ seconds[bucket.period] += bucket.known_seconds
219
+ known[bucket.period] += bucket.known_videos
220
+ return {period: value / 3600 if known[period] else None for period, value in seconds.items()}
221
+
222
+
223
+ def coverage_cross_table(
224
+ coverage: SelectedCoverage, row_dimension: str, column_dimension: str,
225
+ ) -> pd.DataFrame:
226
+ """Cross a member period with frame labels, conserving known member hours.
227
+
228
+ A multi-value frame label divides its batch-associated hours equally.
229
+ Unknown-only cells are NaN, while a measured zero remains zero. Period
230
+ crossed with itself has only diagonal mass and never invents co-occurrences.
231
+ """
232
+ _require_dimension(row_dimension)
233
+ _require_dimension(column_dimension)
234
+ if 'period' not in (row_dimension, column_dimension):
235
+ raise ValueError('Coverage cross tables require a period dimension')
236
+
237
+ def values(frame, bucket, dimension):
238
+ return (bucket.period,) if dimension == 'period' else _values(frame, dimension)
239
+
240
+ axes = [_ordered((value for frame, bucket in coverage.members
241
+ for value in values(frame, bucket, dimension)), dimension)
242
+ for dimension in (row_dimension, column_dimension)]
243
+ table = pd.DataFrame(0.0, index=pd.Index(axes[0], name=row_dimension),
244
+ columns=pd.Index(axes[1], name=column_dimension))
245
+ observed, unknown = set(), set()
246
+ for frame, bucket in coverage.members:
247
+ rows = values(frame, bucket, row_dimension)
248
+ columns = values(frame, bucket, column_dimension)
249
+ pairs = tuple(zip(rows, columns)) if row_dimension == column_dimension else tuple(product(rows, columns))
250
+ for pair in pairs:
251
+ if bucket.known_videos:
252
+ table.at[pair[0], pair[1]] += bucket.known_seconds / 3600 / len(pairs)
253
+ observed.add(pair)
254
+ elif bucket.unknown_videos:
255
+ unknown.add(pair)
256
+ for row, column in unknown - observed:
257
+ table.at[row, column] = float('nan')
258
+ return table
259
+
260
+
261
  def coverage_table(
262
  records: Iterable[AnnotationRecord], universe: Iterable[AnnotationRecord] | None = None,
263
  ) -> pd.DataFrame:
dashboard_st.py CHANGED
@@ -35,7 +35,7 @@ I18N_EN = {
35
  "⚠️ 本小时请求已达上限(20 次),请稍后再试": "⚠️ Hourly request limit reached (20). Please try again later.",
36
  "示例问题": "Example questions",
37
  "数据集一共覆盖了哪些城市?": "Which cities does the dataset cover?",
38
- "长春的数据为什么占比这么高?": "Why does Changchun make up such a large share?",
39
  "继续追问…": "Ask a follow-up…",
40
  "新对话": "New chat",
41
  "开始使用智能问答": "Get access to AI Q&A",
@@ -66,6 +66,9 @@ def T(s):
66
  Unknown keys pass through unchanged (safe degradation)."""
67
  if not isinstance(s, str):
68
  return s
 
 
 
69
  lang = st.session_state.get("lang", "zh") if hasattr(st, "session_state") else "zh"
70
  if lang == "en":
71
  return I18N_EN.get(s, s)
@@ -138,6 +141,7 @@ def _clear_catalog() -> None:
138
  _cached_annotations.clear()
139
  _cached_updates.clear()
140
  _cached_locations.clear()
 
141
  _cached_public_snapshot.clear()
142
 
143
 
@@ -206,6 +210,28 @@ def _load_locations():
206
  return None, "unavailable"
207
 
208
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
209
  def _render_publication_notice():
210
  if not _public_mode():
211
  return
@@ -238,7 +264,7 @@ def page_dashboard():
238
  from classic_dashboard import render_classic_dashboard
239
  render_classic_dashboard(_load_catalog, _load_annotations, _load_updates,
240
  _clear_catalog, T, render_source_notice=_render_publication_notice,
241
- load_locations=_load_locations)
242
  if not _public_mode():
243
  render_annotation_progress(_inventory_dsn(), T)
244
  _render_footer()
@@ -1625,6 +1651,7 @@ def _qa_dataset_facts(catalog) -> str:
1625
  from collections import Counter
1626
  from live_inventory import summarize_records
1627
  from postgres_dashboard import selected_records
 
1628
 
1629
  records = selected_records(catalog, st.session_state)
1630
  snapshot = summarize_records(records, catalog.latest_updated_at)
@@ -1635,6 +1662,16 @@ def _qa_dataset_facts(catalog) -> str:
1635
  capacity = (f"{snapshot.size_tb:.3f} TB (known-value total, {size_known}/{len(records)} videos)"
1636
  if size_known or not records else "unavailable (no known size values)")
1637
  counts = lambda values: dict(sorted(Counter(values).items()))
 
 
 
 
 
 
 
 
 
 
1638
  resolutions = counts(
1639
  f"{row.width}x{row.height}" if row.width and row.height else "unknown"
1640
  for row in records
@@ -1653,9 +1690,10 @@ def _qa_dataset_facts(catalog) -> str:
1653
  f"Full active inventory: {len(catalog.records)} videos.\n"
1654
  f"Active videos: {snapshot.videos} (current dashboard filters where available).\n"
1655
  f"Duration: {duration}; Size: {capacity}.\n"
1656
- f"City counts assigned from GPS: {counts(row.city for row in records)}\n"
1657
  f"Collection months: {dict((row.month, row.videos) for row in snapshot.month_totals)}\n"
1658
- f"Collection periods: {counts(row.period for row in records)}\n"
 
1659
  f"Codecs: {counts(row.codec for row in records)}\n"
1660
  f"Resolutions: {resolutions}\n"
1661
  f"Missing metadata counts: {missing}\n"
@@ -1673,6 +1711,7 @@ Every dataset fact below is derived solely from active PostgreSQL aerial_db.aeri
1673
  {dataset_facts}
1674
 
1675
  Answer concisely in the user's language. Use only the supplied aggregate facts for dataset numbers.
 
1676
  City attribution uses SRT-derived GPS and pinned 2023 administrative boundaries; boundary cases need review.
1677
  Do not infer city from folder names. Unknown metadata is missing information, not an observed zero.
1678
  If PostgreSQL is unavailable, say the current statistics are unavailable and do not estimate totals.
@@ -1761,7 +1800,7 @@ def page_qa():
1761
  if not history:
1762
  examples = [
1763
  T("数据集一共覆盖了哪些城市?"),
1764
- T("长春的数据为什么占比这么高?"),
1765
  T("哪些视频缺少经纬度或参数信息?"),
1766
  T("各城市的视频数量和总时长是多少?"),
1767
  ]
@@ -1885,7 +1924,6 @@ NAV_ITEMS = [
1885
  ("nav-scenario", "anchor", "场景速查", "Quick-pick"),
1886
  ("nav-city", "anchor", "城市分布", "Cities"),
1887
  ("nav-odd", "anchor", "ODD 维度浏览", "ODD"),
1888
- ("nav-examples", "anchor", "场景示例", "Examples"),
1889
  ("nav-params", "anchor", "参数分布", "Distributions"),
1890
  ("nav-updates", "anchor", "数据更新", "Data updates"),
1891
  ("annotation", "page", "标注进度", "Annotation progress"),
@@ -1993,7 +2031,7 @@ def main():
1993
  (function() {
1994
  try {
1995
  const W = window.parent, D = W.document;
1996
- const ANCHORS = ['nav-scenario', 'nav-city', 'nav-odd', 'nav-examples', 'nav-params', 'nav-updates'];
1997
  // Streamlit ≥1.40 uses [data-testid="stMain"] as the scrollable
1998
  // container (overflow-y: auto), NOT the window. Browser-default
1999
  // anchor scroll and IntersectionObserver (root=viewport) both
 
35
  "⚠️ 本小时请求已达上限(20 次),请稍后再试": "⚠️ Hourly request limit reached (20). Please try again later.",
36
  "示例问题": "Example questions",
37
  "数据集一共覆盖了哪些城市?": "Which cities does the dataset cover?",
38
+ "Changchun (Jilin) 的数据为什么占比这么高?": "Why does Changchun (Jilin) make up such a large share?",
39
  "继续追问…": "Ask a follow-up…",
40
  "新对话": "New chat",
41
  "开始使用智能问答": "Get access to AI Q&A",
 
66
  Unknown keys pass through unchanged (safe degradation)."""
67
  if not isinstance(s, str):
68
  return s
69
+ from city_labels import city_display_name, is_city_label
70
+ if is_city_label(s):
71
+ return city_display_name(s)
72
  lang = st.session_state.get("lang", "zh") if hasattr(st, "session_state") else "zh"
73
  if lang == "en":
74
  return I18N_EN.get(s, s)
 
141
  _cached_annotations.clear()
142
  _cached_updates.clear()
143
  _cached_locations.clear()
144
+ _cached_coverage.clear()
145
  _cached_public_snapshot.clear()
146
 
147
 
 
210
  return None, "unavailable"
211
 
212
 
213
+ @st.cache_data(ttl=60)
214
+ def _cached_coverage(dsn: str):
215
+ from annotation_coverage import fetch_coverage_catalog
216
+ return fetch_coverage_catalog(dsn)
217
+
218
+
219
+ def _load_coverage():
220
+ if _public_mode():
221
+ try:
222
+ coverage = getattr(_cached_public_snapshot(_snapshot_url()), 'coverage', None)
223
+ return (coverage, 'ok') if coverage is not None else (None, 'not_published')
224
+ except Exception:
225
+ return None, 'unavailable'
226
+ dsn = _inventory_dsn()
227
+ if not dsn:
228
+ return None, 'not_configured'
229
+ try:
230
+ return _cached_coverage(dsn), 'ok'
231
+ except Exception:
232
+ return None, 'unavailable'
233
+
234
+
235
  def _render_publication_notice():
236
  if not _public_mode():
237
  return
 
264
  from classic_dashboard import render_classic_dashboard
265
  render_classic_dashboard(_load_catalog, _load_annotations, _load_updates,
266
  _clear_catalog, T, render_source_notice=_render_publication_notice,
267
+ load_locations=_load_locations, load_coverage=_load_coverage)
268
  if not _public_mode():
269
  render_annotation_progress(_inventory_dsn(), T)
270
  _render_footer()
 
1651
  from collections import Counter
1652
  from live_inventory import summarize_records
1653
  from postgres_dashboard import selected_records
1654
+ from city_labels import city_display_name
1655
 
1656
  records = selected_records(catalog, st.session_state)
1657
  snapshot = summarize_records(records, catalog.latest_updated_at)
 
1662
  capacity = (f"{snapshot.size_tb:.3f} TB (known-value total, {size_known}/{len(records)} videos)"
1663
  if size_known or not records else "unavailable (no known size values)")
1664
  counts = lambda values: dict(sorted(Counter(values).items()))
1665
+ period_seconds, period_known, period_unknown = Counter(), Counter(), Counter()
1666
+ for record in records:
1667
+ period_seconds.setdefault(record.period, 0.0)
1668
+ if record.duration_seconds is None:
1669
+ period_unknown[record.period] += 1
1670
+ else:
1671
+ period_seconds[record.period] += record.duration_seconds
1672
+ period_known[record.period] += 1
1673
+ period_hours = {period: round(seconds / 3600, 6) if period_known[period] else None
1674
+ for period, seconds in sorted(period_seconds.items())}
1675
  resolutions = counts(
1676
  f"{row.width}x{row.height}" if row.width and row.height else "unknown"
1677
  for row in records
 
1690
  f"Full active inventory: {len(catalog.records)} videos.\n"
1691
  f"Active videos: {snapshot.videos} (current dashboard filters where available).\n"
1692
  f"Duration: {duration}; Size: {capacity}.\n"
1693
+ f"City counts assigned from GPS: {counts(city_display_name(row.city) for row in records)}\n"
1694
  f"Collection months: {dict((row.month, row.videos) for row in snapshot.month_totals)}\n"
1695
+ f"Collection period known duration (hours): {period_hours}\n"
1696
+ f"Unknown-duration videos by period: {dict(sorted(period_unknown.items()))}\n"
1697
  f"Codecs: {counts(row.codec for row in records)}\n"
1698
  f"Resolutions: {resolutions}\n"
1699
  f"Missing metadata counts: {missing}\n"
 
1711
  {dataset_facts}
1712
 
1713
  Answer concisely in the user's language. Use only the supplied aggregate facts for dataset numbers.
1714
+ Use the supplied English city/province labels verbatim, including when answering in Chinese.
1715
  City attribution uses SRT-derived GPS and pinned 2023 administrative boundaries; boundary cases need review.
1716
  Do not infer city from folder names. Unknown metadata is missing information, not an observed zero.
1717
  If PostgreSQL is unavailable, say the current statistics are unavailable and do not estimate totals.
 
1800
  if not history:
1801
  examples = [
1802
  T("数据集一共覆盖了哪些城市?"),
1803
+ T("Changchun (Jilin) 的数据为什么占比这么高?"),
1804
  T("哪些视频缺少经纬度或参数信息?"),
1805
  T("各城市的视频数量和总时长是多少?"),
1806
  ]
 
1924
  ("nav-scenario", "anchor", "场景速查", "Quick-pick"),
1925
  ("nav-city", "anchor", "城市分布", "Cities"),
1926
  ("nav-odd", "anchor", "ODD 维度浏览", "ODD"),
 
1927
  ("nav-params", "anchor", "参数分布", "Distributions"),
1928
  ("nav-updates", "anchor", "数据更新", "Data updates"),
1929
  ("annotation", "page", "标注进度", "Annotation progress"),
 
2031
  (function() {
2032
  try {
2033
  const W = window.parent, D = W.document;
2034
+ const ANCHORS = ['nav-scenario', 'nav-city', 'nav-odd', 'nav-params', 'nav-updates'];
2035
  // Streamlit ≥1.40 uses [data-testid="stMain"] as the scrollable
2036
  // container (overflow-y: auto), NOT the window. Browser-default
2037
  // anchor scroll and IntersectionObserver (root=viewport) both
postgres_dashboard.py CHANGED
@@ -12,6 +12,7 @@ import plotly.graph_objects as go
12
  import streamlit as st
13
 
14
  from live_inventory import VideoCatalog, VideoRecord, filter_records, summarize_records
 
15
 
16
 
17
  _UNKNOWN = "未知"
@@ -87,7 +88,7 @@ def aggregate_rows(records: Sequence[VideoRecord]) -> pd.DataFrame:
87
  totals[2] += float(record.size_bytes) / 1_000_000_000_000
88
  totals[4] += 1
89
  return pd.DataFrame(
90
- [(*key, totals[0], totals[1] if totals[3] else None,
91
  totals[2] if totals[4] else None, totals[3], totals[4])
92
  for key, totals in sorted(groups.items())],
93
  columns=_EXPORT_COLUMNS,
@@ -104,9 +105,11 @@ def render_dashboard(load_catalog: Callable, clear_catalog: Callable,
104
  return translated
105
 
106
  def category(value: str) -> str:
 
 
107
  labels = {
108
  _UNKNOWN: "Unknown", "缺少坐标": "Missing coordinates",
109
- "边界外或坐标异常": "Outside boundaries / invalid coordinates",
110
  "已归属城市": "Assigned city", "早高峰 (07-09)": "Morning peak (07-09)",
111
  "日常规": "Daytime", "晚高峰 (17-19)": "Evening peak (17-19)",
112
  "夜间 (19-07)": "Night (19-07)",
 
12
  import streamlit as st
13
 
14
  from live_inventory import VideoCatalog, VideoRecord, filter_records, summarize_records
15
+ from city_labels import city_display_name, is_city_label
16
 
17
 
18
  _UNKNOWN = "未知"
 
88
  totals[2] += float(record.size_bytes) / 1_000_000_000_000
89
  totals[4] += 1
90
  return pd.DataFrame(
91
+ [(city_display_name(key[0]), *key[1:], totals[0], totals[1] if totals[3] else None,
92
  totals[2] if totals[4] else None, totals[3], totals[4])
93
  for key, totals in sorted(groups.items())],
94
  columns=_EXPORT_COLUMNS,
 
105
  return translated
106
 
107
  def category(value: str) -> str:
108
+ if is_city_label(value):
109
+ return city_display_name(value)
110
  labels = {
111
  _UNKNOWN: "Unknown", "缺少坐标": "Missing coordinates",
112
+ "边界外或坐标异常": "Unassigned city",
113
  "已归属城市": "Assigned city", "早高峰 (07-09)": "Morning peak (07-09)",
114
  "日常规": "Daytime", "晚高峰 (17-19)": "Evening peak (17-19)",
115
  "夜间 (19-07)": "Night (19-07)",
public_snapshot.py CHANGED
@@ -32,6 +32,7 @@ from live_inventory import (
32
  _load_boundaries, _public_city_name,
33
  )
34
  from location_groups import LocationCatalog, LocationGroup
 
35
 
36
 
37
  PUBLIC_SNAPSHOT_URL = "https://raw.githubusercontent.com/AutoZYX-Labs/driveresearch-data-proxy/statistics-data/published/aerial-statistics-v1.json.gz"
@@ -63,6 +64,7 @@ class PublicSnapshot:
63
  updates: BatchUpdateCatalog | None = None
64
  includes_frames: bool = False
65
  locations: LocationCatalog | None = None
 
66
 
67
 
68
  def _object(value: object, keys: set[str]) -> dict:
@@ -244,14 +246,64 @@ def _parse_locations(value: object, annotations: list[AnnotationRecord]) -> Loca
244
  return LocationCatalog(tuple(groups), unresolved, 100)
245
 
246
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
247
  def _parse_document(document: object) -> PublicSnapshot:
248
  if not isinstance(document, dict):
249
  raise ValueError("Invalid snapshot object")
250
  version = document.get("schema_version")
251
  keys = {"schema_version", "source", "generated_at", "catalog", "annotations"}
252
- document = _object(document, keys | ({"updates"} if version in (2, 3, 4) else set())
253
- | ({"locations"} if version == 4 else set()))
254
- if type(version) is not int or version not in (1, 2, 3, 4) or document["source"] != SOURCE:
 
255
  raise ValueError("Unsupported snapshot source or schema")
256
  generated_at = _timestamp(document["generated_at"])
257
  catalog = _object(document["catalog"], {"records", "latest_updated_at"})
@@ -271,11 +323,11 @@ def _parse_document(document: object) -> PublicSnapshot:
271
  ))
272
  # Frame schemas keep an explicit updates key even when that catalog is absent.
273
  updates = _parse_updates(document["updates"]) if (
274
- version == 2 or (version in (3, 4) and document["updates"] is not None)) else None
275
  available_codes = {record.code for record in updates.records if record.annotations_available} if updates else None
276
  annotations, codes = [], set()
277
  for raw in _list(annotation_catalog["records"]):
278
- raw = _object(raw, _FRAME_ANNOTATION_KEYS if version in (3, 4) else _ANNOTATION_KEYS)
279
  compact = _list(raw["labels"])
280
  for label in compact:
281
  _object(label, {"field", "state", "values"})
@@ -284,7 +336,7 @@ def _parse_document(document: object) -> PublicSnapshot:
284
  if model_name not in PUBLIC_MODEL_NAMES:
285
  raise ValueError("Unknown public annotation model")
286
  references = {}
287
- if version in (3, 4):
288
  code = _text(raw["batch_code"], maximum=18)
289
  if (not re.fullmatch(r"B-[0-9a-f]{16}", code) or code in codes
290
  or (available_codes is not None and code not in available_codes)):
@@ -303,8 +355,10 @@ def _parse_document(document: object) -> PublicSnapshot:
303
  _integer(annotation_catalog["excluded_records"], optional=False)),
304
  generated_at=generated_at,
305
  updates=updates,
306
- includes_frames=version in (3, 4),
307
- locations=_parse_locations(document["locations"], annotations) if version == 4 else None,
 
 
308
  )
309
 
310
 
@@ -314,11 +368,13 @@ def _canonical_json(value: object) -> bytes:
314
 
315
  def _document(catalog: VideoCatalog, annotations: AnnotationCatalog, generated_at: datetime,
316
  updates: BatchUpdateCatalog | None = None, *, include_frames: bool = False,
317
- locations: LocationCatalog | None = None) -> dict:
318
  if type(include_frames) is not bool:
319
  raise ValueError("Invalid frame publication flag")
320
  if locations is not None and not include_frames:
321
  raise ValueError("Location publication requires exact batch/frame references")
 
 
322
  videos = []
323
  for record in catalog.records:
324
  # Explicit field selection also strips any accidental future dataclass additions.
@@ -367,20 +423,33 @@ def _document(catalog: VideoCatalog, annotations: AnnotationCatalog, generated_a
367
  key=lambda group: group["code"]),
368
  "unresolved_batch_codes": sorted(locations.unresolved_batch_codes),
369
  }
 
 
 
 
 
 
 
 
 
 
 
 
 
370
  _parse_document(document)
371
  return document
372
 
373
 
374
  def encode_snapshot(catalog: VideoCatalog, annotations: AnnotationCatalog, generated_at: datetime,
375
  updates: BatchUpdateCatalog | None = None, *, include_frames: bool = False,
376
- locations: LocationCatalog | None = None) -> bytes:
377
  """Encode V1/V2 by default, V3 frames, or V4 frames and location memberships.
378
 
379
  V3 requires safe unique batch codes; a missing frame is null. Its updates
380
  key is also explicit and nullable when no update catalog is supplied.
381
  """
382
  raw = _canonical_json(_document(catalog, annotations, generated_at, updates,
383
- include_frames=include_frames, locations=locations))
384
  if len(raw) > MAX_DECOMPRESSED_BYTES:
385
  raise ValueError("Public statistics snapshot is too large")
386
  encoded = gzip.compress(raw, mtime=0)
@@ -391,10 +460,10 @@ def encode_snapshot(catalog: VideoCatalog, annotations: AnnotationCatalog, gener
391
 
392
  def semantic_digest(catalog: VideoCatalog, annotations: AnnotationCatalog,
393
  updates: BatchUpdateCatalog | None = None, *, include_frames: bool = False,
394
- locations: LocationCatalog | None = None) -> str:
395
  """Hash facts, excluding row order, export time and discovery-only timestamps."""
396
  document = _document(catalog, annotations, datetime(2000, 1, 1, tzinfo=timezone.utc), updates,
397
- include_frames=include_frames, locations=locations)
398
  del document["generated_at"]
399
  del document["catalog"]["latest_updated_at"]
400
  del document["annotations"]["latest_updated_at"]
@@ -488,7 +557,8 @@ def _fetch_checked_at(url: str, snapshot: PublicSnapshot) -> datetime | None:
488
  or document["source"] != SOURCE
489
  or document["semantic_digest"] != semantic_digest(snapshot.catalog, snapshot.annotations, snapshot.updates,
490
  include_frames=snapshot.includes_frames,
491
- locations=snapshot.locations)):
 
492
  return None
493
  checked_at = _timestamp(document["checked_at"])
494
  return checked_at if checked_at >= snapshot.generated_at else None
 
32
  _load_boundaries, _public_city_name,
33
  )
34
  from location_groups import LocationCatalog, LocationGroup
35
+ from annotation_coverage import BatchCoverage, CoverageCatalog, CoverageSlice
36
 
37
 
38
  PUBLIC_SNAPSHOT_URL = "https://raw.githubusercontent.com/AutoZYX-Labs/driveresearch-data-proxy/statistics-data/published/aerial-statistics-v1.json.gz"
 
64
  updates: BatchUpdateCatalog | None = None
65
  includes_frames: bool = False
66
  locations: LocationCatalog | None = None
67
+ coverage: CoverageCatalog | None = None
68
 
69
 
70
  def _object(value: object, keys: set[str]) -> dict:
 
246
  return LocationCatalog(tuple(groups), unresolved, 100)
247
 
248
 
249
+ def _parse_coverage(value: object, annotations: list[AnnotationRecord], videos: list[VideoRecord]) -> CoverageCatalog:
250
+ value = _object(value, {'records'})
251
+ raw_records = _list(value['records'])
252
+ if len(raw_records) > MAX_BATCH_RECORDS:
253
+ raise ValueError('Too many coverage records')
254
+ population = {record.batch_code for record in annotations}
255
+ records, seen, slice_count = [], set(), 0
256
+ known_totals, known_videos, unknown_videos = [], 0, 0
257
+ for raw in raw_records:
258
+ raw = _object(raw, {'batch_code', 'slices'})
259
+ code = _text(raw['batch_code'], maximum=18)
260
+ if not re.fullmatch(r'B-[0-9a-f]{16}', code) or code not in population or code in seen:
261
+ raise ValueError('Invalid coverage batch code')
262
+ seen.add(code)
263
+ slices, keys = [], set()
264
+ for item in _list(raw['slices']):
265
+ slice_count += 1
266
+ if slice_count > MAX_BATCH_RECORDS:
267
+ raise ValueError('Too many coverage buckets')
268
+ item = _object(item, {'collected_on', 'period', 'codec', 'known_seconds', 'known_videos', 'unknown_videos'})
269
+ day, period, codec = _day(item['collected_on']), _text(item['period']), _text(item['codec'], maximum=64)
270
+ if period not in _PERIODS or codec not in PUBLIC_CODECS:
271
+ raise ValueError('Invalid coverage metadata')
272
+ key = day, period, codec
273
+ if key in keys:
274
+ raise ValueError('Duplicate coverage bucket')
275
+ keys.add(key)
276
+ seconds = _number(item['known_seconds'])
277
+ known = _integer(item['known_videos'], optional=False)
278
+ unknown = _integer(item['unknown_videos'], optional=False)
279
+ if seconds is None or known + unknown == 0 or (known == 0 and seconds != 0):
280
+ raise ValueError('Invalid coverage measurement')
281
+ slices.append(CoverageSlice(day, period, codec, seconds, known, unknown))
282
+ known_totals.append(seconds)
283
+ known_videos += known
284
+ unknown_videos += unknown
285
+ slices.sort(key=lambda item: (item.collected_on is None, item.collected_on or date.max, item.period, item.codec))
286
+ records.append(BatchCoverage(code, tuple(slices)))
287
+ if seen != population:
288
+ raise ValueError('Incomplete coverage population')
289
+ inventory_known = sum(record.duration_seconds is not None for record in videos)
290
+ inventory_seconds = math.fsum(record.duration_seconds for record in videos if record.duration_seconds is not None)
291
+ if (known_videos > inventory_known or unknown_videos > len(videos) - inventory_known
292
+ or known_videos + unknown_videos > len(videos)
293
+ or math.fsum(known_totals) > inventory_seconds + max(1e-6, inventory_seconds * 1e-12)):
294
+ raise ValueError('Coverage exceeds active video inventory')
295
+ return CoverageCatalog(tuple(sorted(records, key=lambda record: record.batch_code)))
296
+
297
+
298
  def _parse_document(document: object) -> PublicSnapshot:
299
  if not isinstance(document, dict):
300
  raise ValueError("Invalid snapshot object")
301
  version = document.get("schema_version")
302
  keys = {"schema_version", "source", "generated_at", "catalog", "annotations"}
303
+ document = _object(document, keys | ({"updates"} if version in (2, 3, 4, 5) else set())
304
+ | ({"locations"} if version in (4, 5) else set())
305
+ | ({"coverage"} if version == 5 else set()))
306
+ if type(version) is not int or version not in (1, 2, 3, 4, 5) or document["source"] != SOURCE:
307
  raise ValueError("Unsupported snapshot source or schema")
308
  generated_at = _timestamp(document["generated_at"])
309
  catalog = _object(document["catalog"], {"records", "latest_updated_at"})
 
323
  ))
324
  # Frame schemas keep an explicit updates key even when that catalog is absent.
325
  updates = _parse_updates(document["updates"]) if (
326
+ version == 2 or (version in (3, 4, 5) and document["updates"] is not None)) else None
327
  available_codes = {record.code for record in updates.records if record.annotations_available} if updates else None
328
  annotations, codes = [], set()
329
  for raw in _list(annotation_catalog["records"]):
330
+ raw = _object(raw, _FRAME_ANNOTATION_KEYS if version in (3, 4, 5) else _ANNOTATION_KEYS)
331
  compact = _list(raw["labels"])
332
  for label in compact:
333
  _object(label, {"field", "state", "values"})
 
336
  if model_name not in PUBLIC_MODEL_NAMES:
337
  raise ValueError("Unknown public annotation model")
338
  references = {}
339
+ if version in (3, 4, 5):
340
  code = _text(raw["batch_code"], maximum=18)
341
  if (not re.fullmatch(r"B-[0-9a-f]{16}", code) or code in codes
342
  or (available_codes is not None and code not in available_codes)):
 
355
  _integer(annotation_catalog["excluded_records"], optional=False)),
356
  generated_at=generated_at,
357
  updates=updates,
358
+ includes_frames=version in (3, 4, 5),
359
+ locations=_parse_locations(document["locations"], annotations) if (
360
+ version == 4 or (version == 5 and document['locations'] is not None)) else None,
361
+ coverage=_parse_coverage(document['coverage'], annotations, videos) if version == 5 else None,
362
  )
363
 
364
 
 
368
 
369
  def _document(catalog: VideoCatalog, annotations: AnnotationCatalog, generated_at: datetime,
370
  updates: BatchUpdateCatalog | None = None, *, include_frames: bool = False,
371
+ locations: LocationCatalog | None = None, coverage: CoverageCatalog | None = None) -> dict:
372
  if type(include_frames) is not bool:
373
  raise ValueError("Invalid frame publication flag")
374
  if locations is not None and not include_frames:
375
  raise ValueError("Location publication requires exact batch/frame references")
376
+ if coverage is not None and not include_frames:
377
+ raise ValueError('Coverage publication requires exact batch/frame references')
378
  videos = []
379
  for record in catalog.records:
380
  # Explicit field selection also strips any accidental future dataclass additions.
 
423
  key=lambda group: group["code"]),
424
  "unresolved_batch_codes": sorted(locations.unresolved_batch_codes),
425
  }
426
+ if coverage is not None:
427
+ document['schema_version'] = 5
428
+ document.setdefault('locations', None)
429
+ document['coverage'] = {'records': sorted([{
430
+ 'batch_code': record.batch_code,
431
+ 'slices': [{
432
+ 'collected_on': item.collected_on.isoformat() if item.collected_on else None,
433
+ 'period': item.period, 'codec': item.codec,
434
+ 'known_seconds': item.known_seconds,
435
+ 'known_videos': item.known_videos, 'unknown_videos': item.unknown_videos,
436
+ } for item in sorted(record.slices, key=lambda item: (
437
+ item.collected_on is None, item.collected_on or date.max, item.period, item.codec))],
438
+ } for record in coverage.records], key=lambda record: record['batch_code'])}
439
  _parse_document(document)
440
  return document
441
 
442
 
443
  def encode_snapshot(catalog: VideoCatalog, annotations: AnnotationCatalog, generated_at: datetime,
444
  updates: BatchUpdateCatalog | None = None, *, include_frames: bool = False,
445
+ locations: LocationCatalog | None = None, coverage: CoverageCatalog | None = None) -> bytes:
446
  """Encode V1/V2 by default, V3 frames, or V4 frames and location memberships.
447
 
448
  V3 requires safe unique batch codes; a missing frame is null. Its updates
449
  key is also explicit and nullable when no update catalog is supplied.
450
  """
451
  raw = _canonical_json(_document(catalog, annotations, generated_at, updates,
452
+ include_frames=include_frames, locations=locations, coverage=coverage))
453
  if len(raw) > MAX_DECOMPRESSED_BYTES:
454
  raise ValueError("Public statistics snapshot is too large")
455
  encoded = gzip.compress(raw, mtime=0)
 
460
 
461
  def semantic_digest(catalog: VideoCatalog, annotations: AnnotationCatalog,
462
  updates: BatchUpdateCatalog | None = None, *, include_frames: bool = False,
463
+ locations: LocationCatalog | None = None, coverage: CoverageCatalog | None = None) -> str:
464
  """Hash facts, excluding row order, export time and discovery-only timestamps."""
465
  document = _document(catalog, annotations, datetime(2000, 1, 1, tzinfo=timezone.utc), updates,
466
+ include_frames=include_frames, locations=locations, coverage=coverage)
467
  del document["generated_at"]
468
  del document["catalog"]["latest_updated_at"]
469
  del document["annotations"]["latest_updated_at"]
 
557
  or document["source"] != SOURCE
558
  or document["semantic_digest"] != semantic_digest(snapshot.catalog, snapshot.annotations, snapshot.updates,
559
  include_frames=snapshot.includes_frames,
560
+ locations=snapshot.locations,
561
+ coverage=snapshot.coverage)):
562
  return None
563
  checked_at = _timestamp(document["checked_at"])
564
  return checked_at if checked_at >= snapshot.generated_at else None
representative_gallery.py CHANGED
@@ -21,6 +21,7 @@ import streamlit as st
21
 
22
  from annotation_statistics import AnnotationRecord, definitions
23
  from classic_statistics import selection_label
 
24
 
25
  _PAGE_SIZE = 12
26
  _LOCATION_CODE = re.compile(r'L-[0-9a-f]{16}\Z')
@@ -201,7 +202,7 @@ def render_representative_gallery(
201
 
202
  st.markdown(_STYLE, unsafe_allow_html=True)
203
  st.markdown("<div id='nav-examples' class='dash-section'><h2 class='dash-section-title'>"
204
- + html.escape(tr('场景示例', 'Scene examples')) + '</h2></div>', unsafe_allow_html=True)
205
  st.caption(tr(
206
  '按城市浏览自动归组采集点(同城组内最大距离 100 米),每个地点展示一张匹配代表帧;'
207
  '这不是经人工确认的路口身份。示例随上方城市、日期、编码、时段及 ODD 筛选更新。使用图片右上角控件可放大。',
@@ -250,12 +251,12 @@ def render_representative_gallery(
250
  st.session_state[_CITY_KEY] = default_city
251
  st.session_state[_PAGE_KEY] = 1
252
  city = st.selectbox(tr('示例城市', 'Example city'), cities, key=_CITY_KEY,
253
- format_func=translate, on_change=_reset_gallery_page)
254
  city_examples = tuple(example for example in examples if example.city == city)
255
  _, selected_page, pages = gallery_page(city_examples, st.session_state[_PAGE_KEY])
256
  st.session_state[_PAGE_KEY] = selected_page
257
  st.markdown("<div class='example-city-heading'>"
258
- + html.escape(tr(f'{translate(city)} · 采集点示例', f'{translate(city)} · Collection-point examples'))
259
  + '</div>', unsafe_allow_html=True)
260
  st.selectbox(tr('示例页码', 'Example page'), list(range(1, pages + 1)), key=_PAGE_KEY,
261
  format_func=lambda page: tr(f'第 {page} / {pages} 页', f'Page {page} / {pages}'),
@@ -288,7 +289,7 @@ def render_representative_gallery(
288
  weather = label_value(labels.get('environment.weather'))
289
  st.markdown(
290
  "<div class='example-meta'>"
291
- f"<div class='example-city'>{html.escape(translate(record.city))}</div>"
292
  f"<div class='example-code'>{html.escape(example.code)}</div>"
293
  f"<div class='example-code'>{html.escape(tr('代表批次', 'Representative batch'))}: {html.escape(code)}</div>"
294
  f"<div class='example-date'>{html.escape(tr('匹配 / 全部批次', 'Matching / total batches'))}: {example.matching_batches:,} / {example.total_batches:,}</div>"
 
21
 
22
  from annotation_statistics import AnnotationRecord, definitions
23
  from classic_statistics import selection_label
24
+ from city_labels import city_display_name
25
 
26
  _PAGE_SIZE = 12
27
  _LOCATION_CODE = re.compile(r'L-[0-9a-f]{16}\Z')
 
202
 
203
  st.markdown(_STYLE, unsafe_allow_html=True)
204
  st.markdown("<div id='nav-examples' class='dash-section'><h2 class='dash-section-title'>"
205
+ + html.escape(tr('筛选结果 · 场景示例', 'Filter results · Scene examples')) + '</h2></div>', unsafe_allow_html=True)
206
  st.caption(tr(
207
  '按城市浏览自动归组采集点(同城组内最大距离 100 米),每个地点展示一张匹配代表帧;'
208
  '这不是经人工确认的路口身份。示例随上方城市、日期、编码、时段及 ODD 筛选更新。使用图片右上角控件可放大。',
 
251
  st.session_state[_CITY_KEY] = default_city
252
  st.session_state[_PAGE_KEY] = 1
253
  city = st.selectbox(tr('示例城市', 'Example city'), cities, key=_CITY_KEY,
254
+ format_func=city_display_name, on_change=_reset_gallery_page)
255
  city_examples = tuple(example for example in examples if example.city == city)
256
  _, selected_page, pages = gallery_page(city_examples, st.session_state[_PAGE_KEY])
257
  st.session_state[_PAGE_KEY] = selected_page
258
  st.markdown("<div class='example-city-heading'>"
259
+ + html.escape(tr(f'{city_display_name(city)} · 采集点示例', f'{city_display_name(city)} · Collection-point examples'))
260
  + '</div>', unsafe_allow_html=True)
261
  st.selectbox(tr('示例页码', 'Example page'), list(range(1, pages + 1)), key=_PAGE_KEY,
262
  format_func=lambda page: tr(f'第 {page} / {pages} 页', f'Page {page} / {pages}'),
 
289
  weather = label_value(labels.get('environment.weather'))
290
  st.markdown(
291
  "<div class='example-meta'>"
292
+ f"<div class='example-city'>{html.escape(city_display_name(record.city))}</div>"
293
  f"<div class='example-code'>{html.escape(example.code)}</div>"
294
  f"<div class='example-code'>{html.escape(tr('代表批次', 'Representative batch'))}: {html.escape(code)}</div>"
295
  f"<div class='example-date'>{html.escape(tr('匹配 / 全部批次', 'Matching / total batches'))}: {example.matching_batches:,} / {example.total_batches:,}</div>"