Spaces:
Running
Running
Deploy PostgreSQL statistics 69f9770
Browse files- README.md +8 -3
- annotation_coverage.py +164 -0
- batch_updates_dashboard.py +5 -3
- city_labels.py +64 -0
- classic_dashboard.py +83 -17
- classic_statistics.py +114 -1
- dashboard_st.py +45 -7
- postgres_dashboard.py +5 -2
- public_snapshot.py +84 -14
- representative_gallery.py +5 -4
README.md
CHANGED
|
@@ -25,9 +25,11 @@ short_description: 驭研科技航测自然驾驶数据 · PostgreSQL 自动统
|
|
| 25 |
- 四张总览卡展示全库覆盖城市、采集批次、已知总时长和视频数;场景筛选使用代表帧统计。
|
| 26 |
- 道路层级旭日图、标签柱图/环图、两维交叉分析及代表帧覆盖热力图。
|
| 27 |
- 有效视频数、已知总时长和容量、按 GPS 行政区归属的城市分布。
|
|
|
|
| 28 |
- 城市、采集日期、编码和时段筛选;采集时间、编码、分辨率、帧率、时长、入库与完整性分布。
|
| 29 |
- 通过校验的自动标注结果:道路、路面、交通设施、动态目标、天气和光照等 27 个字段。
|
| 30 |
-
- 场景示例
|
|
|
|
| 31 |
- 标签观察状态总览、自选字段分布图、汇总 CSV;未知、未观察到与不适用单独统计。
|
| 32 |
- 页面每 60 秒刷新,显示实际数据发布时间和同步存活检查时间。
|
| 33 |
|
|
@@ -39,6 +41,9 @@ short_description: 驭研科技航测自然驾驶数据 · PostgreSQL 自动统
|
|
| 39 |
|
| 40 |
标签统计一批次一张代表帧,筛选按代表视频的城市、日期、编码和时段应用。
|
| 41 |
单帧动态观察不代表关联视频全程;多选标签占比之和可能超过 100%。
|
|
|
|
|
|
|
|
|
|
| 42 |
当前无需人工审核;纳入 `completed AND validation_passed`、代表视频有效且仍为
|
| 43 |
批次成员的结果,排除明确退回和不符合完整字段契约的结果。结果明确标为自动标注。
|
| 44 |
|
|
@@ -69,9 +74,9 @@ published/frames/<thumbnail-key>.jpg
|
|
| 69 |
私有 `NDS-Statistics-Data` 旧库保持原有权限,不再作为本站数据源。公开数据分支
|
| 70 |
不改变入口仓库 `main` 的页面和域名,也不会触发每次数据更新都重建 GitHub Pages。
|
| 71 |
数据未变化时只发布轻量存活状态,最多间隔 15 分钟,避免重复提交大文件。
|
| 72 |
-
批次更新采用 V2 公开结果,示例图关联采用 V3,采集点归组采用 V4;读取端继续兼容旧格式。
|
| 73 |
发布器通过 `NDS_PUBLISH_BATCH_UPDATES=1`、`NDS_PUBLISH_REPRESENTATIVE_FRAMES=1`
|
| 74 |
-
|
| 75 |
先部署兼容读取端,再启用发布器。新增图片先以一次 Git 提交同步,随后发布引用它们的统计结果;
|
| 76 |
图片同步失败不会推进本地发布状态,下次自动重试。未变化的原图使用私有缩略图缓存。
|
| 77 |
Hugging Face 每 60 秒读取结果;这是分钟级自动同步,并非直连数据库的秒级订阅。
|
|
|
|
| 25 |
- 四张总览卡展示全库覆盖城市、采集批次、已知总时长和视频数;场景筛选使用代表帧统计。
|
| 26 |
- 道路层级旭日图、标签柱图/环图、两维交叉分析及代表帧覆盖热力图。
|
| 27 |
- 有效视频数、已知总时长和容量、按 GPS 行政区归属的城市分布。
|
| 28 |
+
- 数据集城市统一显示英文名称和省份,例如 `Changchun (Jilin)`;直辖市显示 `Shanghai`,内部城市标识与筛选键保持不变。
|
| 29 |
- 城市、采集日期、编码和时段筛选;采集时间、编码、分辨率、帧率、时长、入库与完整性分布。
|
| 30 |
- 通过校验的自动标注结果:道路、路面、交通设施、动态目标、天气和光照等 27 个字段。
|
| 31 |
+
- 场景示例属于标签筛选结果:选择 ODD 标签或场景预设后,按城市查看自动归组采集点,每点一张、每页 12 个点;可展开完整标签并放大图片。
|
| 32 |
+
- 时间分布及涉及时段的交叉分析使用关联视频实际覆盖小时数;已知/未知时长和缺少覆盖的批次单独说明,不再用代表帧数量代替时长。
|
| 33 |
- 标签观察状态总览、自选字段分布图、汇总 CSV;未知、未观察到与不适用单独统计。
|
| 34 |
- 页面每 60 秒刷新,显示实际数据发布时间和同步存活检查时间。
|
| 35 |
|
|
|
|
| 41 |
|
| 42 |
标签统计一批次一张代表帧,筛选按代表视频的城市、日期、编码和时段应用。
|
| 43 |
单帧动态观察不代表关联视频全程;多选标签占比之和可能超过 100%。
|
| 44 |
+
关联时长按匹配批次中未删除的视频实际 `duration_seconds` 累加,再除以 3600,
|
| 45 |
+
不固定假定每帧半小时;成员视频也应用日期、编码和时段筛选,时段按每条视频采集时间归类。
|
| 46 |
+
这些小时数表示关联视频覆盖,不代表天气或目标在全程持续存在。未知时长不估算,测得的零与未知分开处理。
|
| 47 |
当前无需人工审核;纳入 `completed AND validation_passed`、代表视频有效且仍为
|
| 48 |
批次成员的结果,排除明确退回和不符合完整字段契约的结果。结果明确标为自动标注。
|
| 49 |
|
|
|
|
| 74 |
私有 `NDS-Statistics-Data` 旧库保持原有权限,不再作为本站数据源。公开数据分支
|
| 75 |
不改变入口仓库 `main` 的页面和域名,也不会触发每次数据更新都重建 GitHub Pages。
|
| 76 |
数据未变化时只发布轻量存活状态,最多间隔 15 分钟,避免重复提交大文件。
|
| 77 |
+
批次更新采用 V2 公开结果,示例图关联采用 V3,采集点归组采用 V4,关联时长采用 V5;读取端继续兼容旧格式。
|
| 78 |
发布器通过 `NDS_PUBLISH_BATCH_UPDATES=1`、`NDS_PUBLISH_REPRESENTATIVE_FRAMES=1`
|
| 79 |
+
以及 `NDS_PUBLISH_LOCATION_GROUPS=1`、`NDS_PUBLISH_COVERAGE=1` 启用,
|
| 80 |
先部署兼容读取端,再启用发布器。新增图片先以一次 Git 提交同步,随后发布引用它们的统计结果;
|
| 81 |
图片同步失败不会推进本地发布状态,下次自动重试。未变化的原图使用私有缩略图缓存。
|
| 82 |
Hugging Face 每 60 秒读取结果;这是分钟级自动同步,并非直连数据库的秒级订阅。
|
annotation_coverage.py
ADDED
|
@@ -0,0 +1,164 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Public-safe associated video coverage for eligible representative batches.
|
| 2 |
+
|
| 3 |
+
Every active member contributes its full measured duration to the bucket of its
|
| 4 |
+
collection start. These durations describe associated video coverage, without
|
| 5 |
+
asserting that a representative frame's labels persist throughout the videos.
|
| 6 |
+
"""
|
| 7 |
+
from __future__ import annotations
|
| 8 |
+
|
| 9 |
+
from collections.abc import Iterable
|
| 10 |
+
from dataclasses import dataclass
|
| 11 |
+
from datetime import date
|
| 12 |
+
from decimal import Decimal
|
| 13 |
+
import math
|
| 14 |
+
import re
|
| 15 |
+
|
| 16 |
+
import psycopg
|
| 17 |
+
|
| 18 |
+
from annotation_statistics import validate_public_labels
|
| 19 |
+
from live_inventory import _period_for_hour, _valid_number
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
_ELIGIBLE_BATCH_SQL = """
|
| 23 |
+
SELECT b.batch_id, b.labels
|
| 24 |
+
FROM aerial.annotation_batches b
|
| 25 |
+
JOIN aerial.videos representative
|
| 26 |
+
ON representative.sha256 = b.representative_video_sha256
|
| 27 |
+
WHERE b.processing_info->>'status' = 'completed'
|
| 28 |
+
AND b.validation_passed IS TRUE
|
| 29 |
+
AND b.review_status IS DISTINCT FROM 'rejected'
|
| 30 |
+
AND representative.deleted_at IS NULL
|
| 31 |
+
AND EXISTS (
|
| 32 |
+
SELECT 1 FROM aerial.annotation_batch_videos bv
|
| 33 |
+
WHERE bv.batch_id = b.batch_id
|
| 34 |
+
AND bv.video_sha256 = representative.sha256
|
| 35 |
+
)
|
| 36 |
+
ORDER BY b.batch_id
|
| 37 |
+
"""
|
| 38 |
+
_MEMBER_SQL = """
|
| 39 |
+
SELECT bv.batch_id, v.sha256,
|
| 40 |
+
(v.collected_at AT TIME ZONE 'Asia/Shanghai')::date,
|
| 41 |
+
EXTRACT(HOUR FROM v.collected_at AT TIME ZONE 'Asia/Shanghai')::integer,
|
| 42 |
+
v.codec, v.duration_seconds
|
| 43 |
+
FROM aerial.annotation_batch_videos bv
|
| 44 |
+
JOIN aerial.videos v ON v.sha256 = bv.video_sha256
|
| 45 |
+
WHERE bv.batch_id = ANY(%s)
|
| 46 |
+
AND v.deleted_at IS NULL
|
| 47 |
+
ORDER BY bv.batch_id, v.sha256
|
| 48 |
+
"""
|
| 49 |
+
_PUBLIC_CODE = re.compile(r'B-[0-9a-f]{16}')
|
| 50 |
+
|
| 51 |
+
|
| 52 |
+
@dataclass(frozen=True)
|
| 53 |
+
class CoverageSlice:
|
| 54 |
+
collected_on: date | None
|
| 55 |
+
period: str
|
| 56 |
+
codec: str
|
| 57 |
+
known_seconds: float
|
| 58 |
+
known_videos: int
|
| 59 |
+
unknown_videos: int
|
| 60 |
+
|
| 61 |
+
|
| 62 |
+
@dataclass(frozen=True)
|
| 63 |
+
class BatchCoverage:
|
| 64 |
+
batch_code: str
|
| 65 |
+
slices: tuple[CoverageSlice, ...]
|
| 66 |
+
|
| 67 |
+
|
| 68 |
+
@dataclass(frozen=True)
|
| 69 |
+
class CoverageCatalog:
|
| 70 |
+
records: tuple[BatchCoverage, ...]
|
| 71 |
+
|
| 72 |
+
|
| 73 |
+
def _slice_key(item: CoverageSlice) -> tuple[bool, date, str, str]:
|
| 74 |
+
return item.collected_on is None, item.collected_on or date.max, item.period, item.codec
|
| 75 |
+
|
| 76 |
+
|
| 77 |
+
def _validated_code(value: object) -> str:
|
| 78 |
+
if not isinstance(value, str) or not _PUBLIC_CODE.fullmatch(value):
|
| 79 |
+
raise ValueError('Invalid coverage batch code')
|
| 80 |
+
return value
|
| 81 |
+
|
| 82 |
+
|
| 83 |
+
def align_coverage_catalog(catalog: CoverageCatalog, batch_codes: Iterable[str]) -> CoverageCatalog:
|
| 84 |
+
"""Match independent source coverage to the exact exported batch population.
|
| 85 |
+
|
| 86 |
+
Empty slices mark unavailable member coverage; a known zero remains a slice
|
| 87 |
+
with a positive known-video count. Reject duplicate source or export codes
|
| 88 |
+
rather than silently hiding an inconsistent publication population.
|
| 89 |
+
"""
|
| 90 |
+
records = {}
|
| 91 |
+
for record in catalog.records:
|
| 92 |
+
code = _validated_code(record.batch_code)
|
| 93 |
+
if code in records:
|
| 94 |
+
raise ValueError('Duplicate coverage batch code')
|
| 95 |
+
records[code] = record
|
| 96 |
+
selected = set()
|
| 97 |
+
for value in batch_codes:
|
| 98 |
+
code = _validated_code(value)
|
| 99 |
+
if code in selected:
|
| 100 |
+
raise ValueError('Duplicate coverage batch code')
|
| 101 |
+
selected.add(code)
|
| 102 |
+
return CoverageCatalog(tuple(
|
| 103 |
+
BatchCoverage(code, tuple(sorted(records[code].slices, key=_slice_key)))
|
| 104 |
+
if code in records else BatchCoverage(code, ())
|
| 105 |
+
for code in sorted(selected)
|
| 106 |
+
))
|
| 107 |
+
|
| 108 |
+
|
| 109 |
+
def fetch_coverage_catalog(dsn: str) -> CoverageCatalog:
|
| 110 |
+
"""Read eligible labels once per batch, then only active member metadata.
|
| 111 |
+
|
| 112 |
+
Private identifiers are used locally for membership checks and discarded.
|
| 113 |
+
Duplicate membership rows inside a batch are counted once. Conflicting
|
| 114 |
+
active memberships across eligible batches fail with an identifier-free
|
| 115 |
+
error. Missing, negative and nonfinite durations count as unknown videos;
|
| 116 |
+
observed zero counts as a known duration. No duration is inferred.
|
| 117 |
+
"""
|
| 118 |
+
# batch_updates imports the label validator too; keep a single safe B-code
|
| 119 |
+
# contract without creating an import-time dependency cycle.
|
| 120 |
+
from batch_updates import _public_code
|
| 121 |
+
|
| 122 |
+
with psycopg.connect(dsn, options='-c default_transaction_read_only=on') as connection:
|
| 123 |
+
with connection.cursor() as cursor:
|
| 124 |
+
cursor.execute(_ELIGIBLE_BATCH_SQL)
|
| 125 |
+
eligible = []
|
| 126 |
+
for batch_id, raw_labels in cursor.fetchall():
|
| 127 |
+
try:
|
| 128 |
+
validate_public_labels(raw_labels)
|
| 129 |
+
except ValueError:
|
| 130 |
+
continue
|
| 131 |
+
eligible.append(batch_id)
|
| 132 |
+
if not eligible:
|
| 133 |
+
return CoverageCatalog(())
|
| 134 |
+
cursor.execute(_MEMBER_SQL, (eligible,))
|
| 135 |
+
rows = cursor.fetchall()
|
| 136 |
+
|
| 137 |
+
buckets = {batch_id: {} for batch_id in eligible}
|
| 138 |
+
ownership = {}
|
| 139 |
+
for batch_id, video_id, collected_on, hour, codec, seconds in rows:
|
| 140 |
+
if video_id in ownership:
|
| 141 |
+
if ownership[video_id] != batch_id:
|
| 142 |
+
raise ValueError('Conflicting active video batch membership')
|
| 143 |
+
continue
|
| 144 |
+
ownership[video_id] = batch_id
|
| 145 |
+
key = (collected_on, _period_for_hour(hour),
|
| 146 |
+
codec.strip() if isinstance(codec, str) and codec.strip() else '未知')
|
| 147 |
+
bucket = buckets[batch_id].setdefault(key, [Decimal(0), 0, 0])
|
| 148 |
+
duration = _valid_number(seconds)
|
| 149 |
+
if duration is None:
|
| 150 |
+
bucket[2] += 1
|
| 151 |
+
else:
|
| 152 |
+
bucket[0] += duration
|
| 153 |
+
bucket[1] += 1
|
| 154 |
+
|
| 155 |
+
records = []
|
| 156 |
+
for batch_id, batch_buckets in buckets.items():
|
| 157 |
+
slices = []
|
| 158 |
+
for (collected_on, period, codec), (seconds, known, unknown) in batch_buckets.items():
|
| 159 |
+
total = float(seconds)
|
| 160 |
+
if not math.isfinite(total):
|
| 161 |
+
raise ValueError('Invalid coverage duration total')
|
| 162 |
+
slices.append(CoverageSlice(collected_on, period, codec, total, known, unknown))
|
| 163 |
+
records.append(BatchCoverage(_public_code(batch_id), tuple(sorted(slices, key=_slice_key))))
|
| 164 |
+
return CoverageCatalog(tuple(sorted(records, key=lambda record: record.batch_code)))
|
batch_updates_dashboard.py
CHANGED
|
@@ -12,6 +12,7 @@ import plotly.graph_objects as go
|
|
| 12 |
import streamlit as st
|
| 13 |
|
| 14 |
from batch_updates import BatchUpdateCatalog, BatchUpdateRecord
|
|
|
|
| 15 |
|
| 16 |
_BEIJING = ZoneInfo('Asia/Shanghai')
|
| 17 |
|
|
@@ -54,7 +55,7 @@ def batch_table(records: Iterable[BatchUpdateRecord], translate: Callable) -> pd
|
|
| 54 |
columns = tuple(map(translate, ('批次编号','城市','活跃视频数','首次入库','最近入库',
|
| 55 |
'源数据更新','标注完成','可用标签')))
|
| 56 |
return pd.DataFrame([
|
| 57 |
-
(record.code, record.city, record.videos, stamp(record.first_ingested_at),
|
| 58 |
stamp(record.last_ingested_at), stamp(record.source_updated_at),
|
| 59 |
stamp(record.annotation_completed_at), translate('有' if record.annotations_available else '暂无'))
|
| 60 |
for record in records
|
|
@@ -122,10 +123,11 @@ def render_batch_updates(load_updates: Callable, translate: Callable) -> None:
|
|
| 122 |
left, right = st.columns(2)
|
| 123 |
query = left.text_input(tr('搜索批次编号或城市', 'Search batch code or city'), key='batch_updates_search').strip().casefold()
|
| 124 |
cities = right.multiselect(tr('更新表城市', 'Batch table cities'), sorted({r.city for r in catalog.records}),
|
| 125 |
-
key='batch_updates_cities')
|
| 126 |
selected = tuple(record for record in catalog.records
|
| 127 |
if (not cities or record.city in cities)
|
| 128 |
-
and (not query or query in record.code.casefold() or query in record.city.casefold()
|
|
|
|
| 129 |
st.caption(tr(f'显示 {len(selected):,} / {len(catalog.records):,} 个批次;CSV 始终包含全部活跃批次。',
|
| 130 |
f'Showing {len(selected):,} / {len(catalog.records):,} batches; CSV always includes all active batches.'))
|
| 131 |
st.dataframe(batch_table(selected, table_translate), hide_index=True, width='stretch')
|
|
|
|
| 12 |
import streamlit as st
|
| 13 |
|
| 14 |
from batch_updates import BatchUpdateCatalog, BatchUpdateRecord
|
| 15 |
+
from city_labels import city_display_name
|
| 16 |
|
| 17 |
_BEIJING = ZoneInfo('Asia/Shanghai')
|
| 18 |
|
|
|
|
| 55 |
columns = tuple(map(translate, ('批次编号','城市','活跃视频数','首次入库','最近入库',
|
| 56 |
'源数据更新','标注完成','可用标签')))
|
| 57 |
return pd.DataFrame([
|
| 58 |
+
(record.code, city_display_name(record.city), record.videos, stamp(record.first_ingested_at),
|
| 59 |
stamp(record.last_ingested_at), stamp(record.source_updated_at),
|
| 60 |
stamp(record.annotation_completed_at), translate('有' if record.annotations_available else '暂无'))
|
| 61 |
for record in records
|
|
|
|
| 123 |
left, right = st.columns(2)
|
| 124 |
query = left.text_input(tr('搜索批次编号或城市', 'Search batch code or city'), key='batch_updates_search').strip().casefold()
|
| 125 |
cities = right.multiselect(tr('更新表城市', 'Batch table cities'), sorted({r.city for r in catalog.records}),
|
| 126 |
+
key='batch_updates_cities', format_func=city_display_name)
|
| 127 |
selected = tuple(record for record in catalog.records
|
| 128 |
if (not cities or record.city in cities)
|
| 129 |
+
and (not query or query in record.code.casefold() or query in record.city.casefold()
|
| 130 |
+
or query in city_display_name(record.city).casefold()))
|
| 131 |
st.caption(tr(f'显示 {len(selected):,} / {len(catalog.records):,} 个批次;CSV 始终包含全部活跃批次。',
|
| 132 |
f'Showing {len(selected):,} / {len(catalog.records):,} batches; CSV always includes all active batches.'))
|
| 133 |
st.dataframe(batch_table(selected, table_translate), hide_index=True, width='stretch')
|
city_labels.py
ADDED
|
@@ -0,0 +1,64 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""English city display labels derived only from the pinned boundary metadata.
|
| 2 |
+
|
| 3 |
+
Source records and filter keys retain their canonical city values. This module
|
| 4 |
+
reads feature properties once; it does not interpret geometry or resolve GPS.
|
| 5 |
+
"""
|
| 6 |
+
|
| 7 |
+
from __future__ import annotations
|
| 8 |
+
|
| 9 |
+
from collections import defaultdict
|
| 10 |
+
from functools import lru_cache
|
| 11 |
+
import json
|
| 12 |
+
from pathlib import Path
|
| 13 |
+
|
| 14 |
+
|
| 15 |
+
_BOUNDARIES = Path(__file__).resolve().parent / 'data/prefectures-2023.geojson'
|
| 16 |
+
_PROVINCES = {
|
| 17 |
+
'11': 'Beijing', '12': 'Tianjin', '13': 'Hebei', '14': 'Shanxi',
|
| 18 |
+
'15': 'Inner Mongolia', '21': 'Liaoning', '22': 'Jilin', '23': 'Heilongjiang',
|
| 19 |
+
'31': 'Shanghai', '32': 'Jiangsu', '33': 'Zhejiang', '34': 'Anhui',
|
| 20 |
+
'35': 'Fujian', '36': 'Jiangxi', '37': 'Shandong', '41': 'Henan',
|
| 21 |
+
'42': 'Hubei', '43': 'Hunan', '44': 'Guangdong', '45': 'Guangxi',
|
| 22 |
+
'46': 'Hainan', '50': 'Chongqing', '51': 'Sichuan', '52': 'Guizhou',
|
| 23 |
+
'53': 'Yunnan', '54': 'Tibet', '61': 'Shaanxi', '62': 'Gansu',
|
| 24 |
+
'63': 'Qinghai', '64': 'Ningxia', '65': 'Xinjiang', '71': 'Taiwan',
|
| 25 |
+
'81': 'Hong Kong', '82': 'Macao',
|
| 26 |
+
}
|
| 27 |
+
_MUNICIPALITIES = frozenset(('11', '12', '31', '50'))
|
| 28 |
+
_CITY_STATUSES = {
|
| 29 |
+
'缺少坐标': 'Missing coordinates',
|
| 30 |
+
'边界外或坐标异常': 'Unassigned city',
|
| 31 |
+
}
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
@lru_cache(maxsize=1)
|
| 35 |
+
def _aliases() -> dict[str, str]:
|
| 36 |
+
aliases = {}
|
| 37 |
+
english_names = defaultdict(set)
|
| 38 |
+
features = json.loads(_BOUNDARIES.read_text(encoding='utf-8'))['features']
|
| 39 |
+
for feature in features:
|
| 40 |
+
properties = feature['properties']
|
| 41 |
+
name = properties['name']
|
| 42 |
+
prefix = str(properties['区划码'])[:2]
|
| 43 |
+
display = name if prefix in _MUNICIPALITIES else f'{name} ({_PROVINCES[prefix]})'
|
| 44 |
+
aliases[properties['地名'].casefold()] = display
|
| 45 |
+
aliases[display.casefold()] = display
|
| 46 |
+
english_names[name.casefold()].add(display)
|
| 47 |
+
# Bare romanizations are useful aliases only when they identify one place.
|
| 48 |
+
for name, displays in english_names.items():
|
| 49 |
+
if len(displays) == 1:
|
| 50 |
+
aliases[name] = next(iter(displays))
|
| 51 |
+
for canonical, display in _CITY_STATUSES.items():
|
| 52 |
+
aliases[canonical.casefold()] = display
|
| 53 |
+
aliases[display.casefold()] = display
|
| 54 |
+
return aliases
|
| 55 |
+
|
| 56 |
+
|
| 57 |
+
def city_display_name(city: str) -> str:
|
| 58 |
+
"""Format a known city or alias, preserving unknown future source values."""
|
| 59 |
+
return _aliases().get(city.strip().casefold(), city)
|
| 60 |
+
|
| 61 |
+
|
| 62 |
+
def is_city_label(value: str) -> bool:
|
| 63 |
+
"""Recognize city/status values without treating generic unknown as a city."""
|
| 64 |
+
return value.strip().casefold() in _aliases()
|
classic_dashboard.py
CHANGED
|
@@ -17,6 +17,7 @@ import streamlit as st
|
|
| 17 |
|
| 18 |
from live_inventory import summarize_records
|
| 19 |
from postgres_dashboard import aggregate_rows, selected_records
|
|
|
|
| 20 |
|
| 21 |
_NAVY = '#0f1923'
|
| 22 |
_ORANGE = '#c85a3a'
|
|
@@ -120,11 +121,13 @@ def _reset_filters(bounds: tuple | None) -> None:
|
|
| 120 |
def render_classic_dashboard(load_catalog: Callable, load_annotations: Callable,
|
| 121 |
load_updates: Callable, clear_data: Callable,
|
| 122 |
translate: Callable, render_source_notice: Callable | None = None, *,
|
| 123 |
-
load_locations: Callable | None = None
|
|
|
|
| 124 |
"""Render inventory and batch-frame statistics with optional point examples."""
|
| 125 |
from classic_statistics import (
|
| 126 |
SCENARIOS, coverage_table, cross_table, field_options, filter_odd_records,
|
| 127 |
hierarchy_rows, scenario_counts, selection_label, tag_counts,
|
|
|
|
| 128 |
)
|
| 129 |
|
| 130 |
language = st.session_state.get('lang', 'zh')
|
|
@@ -158,13 +161,19 @@ def render_classic_dashboard(load_catalog: Callable, load_annotations: Callable,
|
|
| 158 |
figure.update_layout(height=height)
|
| 159 |
st.plotly_chart(figure, width='stretch', key=key, config={'displaylogo': False})
|
| 160 |
|
| 161 |
-
def bar(counts: dict, title: str
|
|
|
|
| 162 |
items = sorted(counts.items(), key=lambda pair: (-pair[1], pair[0]))
|
| 163 |
-
|
|
|
|
| 164 |
text=[n for _, n in items], textposition='outside',
|
| 165 |
marker=dict(color=[_PALETTE[i % len(_PALETTE)] for i in range(len(items))],
|
| 166 |
line_width=0), opacity=.88))
|
| 167 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 168 |
|
| 169 |
@st.fragment(run_every='60s')
|
| 170 |
def current() -> None:
|
|
@@ -241,15 +250,15 @@ def render_classic_dashboard(load_catalog: Callable, load_annotations: Callable,
|
|
| 241 |
st.markdown(f"<div class='kpi-card {style}'><div class='kpi-num'>{value}</div>"
|
| 242 |
f"<div class='kpi-lbl'>{html.escape(label)}</div></div>", unsafe_allow_html=True)
|
| 243 |
if detail == 'batches':
|
| 244 |
-
st.markdown("<a class='example-entry' href='#nav-
|
| 245 |
-
+ html.escape(tr('查看场景示例 →', '
|
| 246 |
+ '</a>', unsafe_allow_html=True)
|
| 247 |
with st.expander(tr('展开详情', 'View details')):
|
| 248 |
if detail == 'batches':
|
| 249 |
st.caption(tr('全部有效关联采集批次;不按城市、日期、编码、时段或代表帧 ODD 筛选缩减。',
|
| 250 |
'All active linked collection batches; independent of city, date, codec, period and frame ODD filters.'))
|
| 251 |
if batch_records is not None:
|
| 252 |
-
st.dataframe(pd.DataFrame(sorted(Counter(r.city for r in batch_records).items()),
|
| 253 |
columns=[tr('城市', 'City'), tr('批次数', 'Batches')]),
|
| 254 |
hide_index=True, width='stretch')
|
| 255 |
elif detail == 'duration':
|
|
@@ -259,6 +268,7 @@ def render_classic_dashboard(load_catalog: Callable, load_annotations: Callable,
|
|
| 259 |
safe = aggregate_rows(inventory_records)
|
| 260 |
if not safe.empty:
|
| 261 |
grouped = safe.groupby('city', as_index=False)['videos'].sum()
|
|
|
|
| 262 |
st.dataframe(grouped, hide_index=True, width='stretch')
|
| 263 |
st.caption(tr(
|
| 264 |
'总览统计当前有效视频库存;下方筛选用于代表帧和分布图。采集批次为全部有效关联批次。'
|
|
@@ -282,7 +292,7 @@ def render_classic_dashboard(load_catalog: Callable, load_annotations: Callable,
|
|
| 282 |
if mapped:
|
| 283 |
figure = go.Figure(go.Scattermap(
|
| 284 |
lat=[centers[c].latitude for c in mapped], lon=[centers[c].longitude for c in mapped],
|
| 285 |
-
mode='markers+text', text=[
|
| 286 |
textfont=dict(size=13, color=_NAVY),
|
| 287 |
marker=dict(size=[14 + min(city_counts[c] ** .5, 24) for c in mapped],
|
| 288 |
color=[_ORANGE if c in selected_cities else _BLUE for c in mapped], opacity=.85),
|
|
@@ -302,7 +312,7 @@ def render_classic_dashboard(load_catalog: Callable, load_annotations: Callable,
|
|
| 302 |
st.caption(tr(f'另有 {missing:,} 条视频无法归属地图中的城市,未显示点位。',
|
| 303 |
f'{missing:,} videos could not be mapped to an administrative city.'))
|
| 304 |
city_options = sorted({r.city for r in catalog.records} | selected_cities)
|
| 305 |
-
st.multiselect(tr('城市', 'City'), city_options, key='pg_cities', format_func=
|
| 306 |
with st.expander(tr('采集日期、编码与时段', 'Collection date, codec and period')):
|
| 307 |
left, right = st.columns(2)
|
| 308 |
with left:
|
|
@@ -355,19 +365,64 @@ def render_classic_dashboard(load_catalog: Callable, load_annotations: Callable,
|
|
| 355 |
unsafe_allow_html=True)
|
| 356 |
st.caption(tr('未知、未观察到、未标注与不适用分别筛选;“未观察到”不表示视频全程不存在该目标。每 60 秒刷新。',
|
| 357 |
'Unknown, not observed, unlabeled and not applicable are distinct. Not observed in a frame does not imply absence throughout a video. Refreshes every 60 seconds.'))
|
| 358 |
-
if annotation_available:
|
| 359 |
from representative_gallery import render_representative_gallery
|
| 360 |
locations, location_status = load_locations() if load_locations else (None, 'not_published')
|
| 361 |
render_representative_gallery(matched, translate, locations=locations, location_status=location_status,
|
| 362 |
language=language, field_label=field_label)
|
|
|
|
|
|
|
|
|
|
| 363 |
if annotation_available and matched:
|
| 364 |
heading('地域与时间分布', 'Geographic & temporal distribution',
|
| 365 |
-
'
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 366 |
left, right = st.columns(2)
|
| 367 |
with left:
|
| 368 |
-
bar(dict(Counter(r.city for r in matched)), tr('各城市代表帧数量', 'Representative frames by city'))
|
| 369 |
with right:
|
| 370 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 371 |
heading('道路类型分布', 'Road type distribution',
|
| 372 |
'道路大类、子类与交叉类型;多选交叉类型按每帧总权重 1 分摊。',
|
| 373 |
'Road category, subcategory and intersection; multiple values share a total weight of one per frame.')
|
|
@@ -404,10 +459,21 @@ def render_classic_dashboard(load_catalog: Callable, load_annotations: Callable,
|
|
| 404 |
row = st.selectbox(tr('维度 1', 'Dimension 1'), dimensions, format_func=field_label, key='classic_cross_row')
|
| 405 |
with right:
|
| 406 |
col = st.selectbox(tr('维度 2', 'Dimension 2'), dimensions, index=1, format_func=field_label, key='classic_cross_column')
|
| 407 |
-
|
| 408 |
-
|
| 409 |
-
|
| 410 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 411 |
heading('ODD 覆盖度', 'ODD coverage',
|
| 412 |
'道路子类 × 天气的代表帧观察密度。0 表示没有样本观察,不证明关联视频全程不存在该条件。',
|
| 413 |
'Road subcategory × weather observations. Zero means no representative-frame observation, not proven absence throughout videos.')
|
|
|
|
| 17 |
|
| 18 |
from live_inventory import summarize_records
|
| 19 |
from postgres_dashboard import aggregate_rows, selected_records
|
| 20 |
+
from city_labels import city_display_name
|
| 21 |
|
| 22 |
_NAVY = '#0f1923'
|
| 23 |
_ORANGE = '#c85a3a'
|
|
|
|
| 121 |
def render_classic_dashboard(load_catalog: Callable, load_annotations: Callable,
|
| 122 |
load_updates: Callable, clear_data: Callable,
|
| 123 |
translate: Callable, render_source_notice: Callable | None = None, *,
|
| 124 |
+
load_locations: Callable | None = None,
|
| 125 |
+
load_coverage: Callable | None = None) -> None:
|
| 126 |
"""Render inventory and batch-frame statistics with optional point examples."""
|
| 127 |
from classic_statistics import (
|
| 128 |
SCENARIOS, coverage_table, cross_table, field_options, filter_odd_records,
|
| 129 |
hierarchy_rows, scenario_counts, selection_label, tag_counts,
|
| 130 |
+
coverage_cross_table, period_hours, selected_member_coverage,
|
| 131 |
)
|
| 132 |
|
| 133 |
language = st.session_state.get('lang', 'zh')
|
|
|
|
| 161 |
figure.update_layout(height=height)
|
| 162 |
st.plotly_chart(figure, width='stretch', key=key, config={'displaylogo': False})
|
| 163 |
|
| 164 |
+
def bar(counts: dict, title: str, *, dimension: str | None = None,
|
| 165 |
+
key: str | None = None) -> None:
|
| 166 |
items = sorted(counts.items(), key=lambda pair: (-pair[1], pair[0]))
|
| 167 |
+
label = city_display_name if dimension == 'city' else value_label
|
| 168 |
+
figure = go.Figure(go.Bar(x=[label(v) for v, _ in items], y=[n for _, n in items],
|
| 169 |
text=[n for _, n in items], textposition='outside',
|
| 170 |
marker=dict(color=[_PALETTE[i % len(_PALETTE)] for i in range(len(items))],
|
| 171 |
line_width=0), opacity=.88))
|
| 172 |
+
if key == 'classic_period_hours':
|
| 173 |
+
figure.update_traces(text=[f'{n:.2f}' for _, n in items],
|
| 174 |
+
hovertemplate='%{x}<br>%{y:.3f} ' + tr('小时', 'hours') + '<extra></extra>')
|
| 175 |
+
figure.update_yaxes(title_text=tr('小时', 'Hours'))
|
| 176 |
+
show_chart(figure, title, key=key)
|
| 177 |
|
| 178 |
@st.fragment(run_every='60s')
|
| 179 |
def current() -> None:
|
|
|
|
| 250 |
st.markdown(f"<div class='kpi-card {style}'><div class='kpi-num'>{value}</div>"
|
| 251 |
f"<div class='kpi-lbl'>{html.escape(label)}</div></div>", unsafe_allow_html=True)
|
| 252 |
if detail == 'batches':
|
| 253 |
+
st.markdown("<a class='example-entry' href='#nav-odd'>"
|
| 254 |
+
+ html.escape(tr('筛选标签查看场景示例 →', 'Filter tags to view scene examples →'))
|
| 255 |
+ '</a>', unsafe_allow_html=True)
|
| 256 |
with st.expander(tr('展开详情', 'View details')):
|
| 257 |
if detail == 'batches':
|
| 258 |
st.caption(tr('全部有效关联采集批次;不按城市、日期、编码、时段或代表帧 ODD 筛选缩减。',
|
| 259 |
'All active linked collection batches; independent of city, date, codec, period and frame ODD filters.'))
|
| 260 |
if batch_records is not None:
|
| 261 |
+
st.dataframe(pd.DataFrame(sorted(Counter(city_display_name(r.city) for r in batch_records).items()),
|
| 262 |
columns=[tr('城市', 'City'), tr('批次数', 'Batches')]),
|
| 263 |
hide_index=True, width='stretch')
|
| 264 |
elif detail == 'duration':
|
|
|
|
| 268 |
safe = aggregate_rows(inventory_records)
|
| 269 |
if not safe.empty:
|
| 270 |
grouped = safe.groupby('city', as_index=False)['videos'].sum()
|
| 271 |
+
grouped['city'] = grouped['city'].map(city_display_name)
|
| 272 |
st.dataframe(grouped, hide_index=True, width='stretch')
|
| 273 |
st.caption(tr(
|
| 274 |
'总览统计当前有效视频库存;下方筛选用于代表帧和分布图。采集批次为全部有效关联批次。'
|
|
|
|
| 292 |
if mapped:
|
| 293 |
figure = go.Figure(go.Scattermap(
|
| 294 |
lat=[centers[c].latitude for c in mapped], lon=[centers[c].longitude for c in mapped],
|
| 295 |
+
mode='markers+text', text=[city_display_name(c) for c in mapped], textposition='top center',
|
| 296 |
textfont=dict(size=13, color=_NAVY),
|
| 297 |
marker=dict(size=[14 + min(city_counts[c] ** .5, 24) for c in mapped],
|
| 298 |
color=[_ORANGE if c in selected_cities else _BLUE for c in mapped], opacity=.85),
|
|
|
|
| 312 |
st.caption(tr(f'另有 {missing:,} 条视频无法归属地图中的城市,未显示点位。',
|
| 313 |
f'{missing:,} videos could not be mapped to an administrative city.'))
|
| 314 |
city_options = sorted({r.city for r in catalog.records} | selected_cities)
|
| 315 |
+
st.multiselect(tr('城市', 'City'), city_options, key='pg_cities', format_func=city_display_name)
|
| 316 |
with st.expander(tr('采集日期、编码与时段', 'Collection date, codec and period')):
|
| 317 |
left, right = st.columns(2)
|
| 318 |
with left:
|
|
|
|
| 365 |
unsafe_allow_html=True)
|
| 366 |
st.caption(tr('未知、未观察到、未标注与不适用分别筛选;“未观察到”不表示视频全程不存在该目标。每 60 秒刷新。',
|
| 367 |
'Unknown, not observed, unlabeled and not applicable are distinct. Not observed in a frame does not imply absence throughout a video. Refreshes every 60 seconds.'))
|
| 368 |
+
if annotation_available and any(selections.values()):
|
| 369 |
from representative_gallery import render_representative_gallery
|
| 370 |
locations, location_status = load_locations() if load_locations else (None, 'not_published')
|
| 371 |
render_representative_gallery(matched, translate, locations=locations, location_status=location_status,
|
| 372 |
language=language, field_label=field_label)
|
| 373 |
+
elif annotation_available:
|
| 374 |
+
st.caption(tr('选择上方至少一个 ODD 标签或场景预设,即可查看匹配的场景示例。',
|
| 375 |
+
'Select an ODD tag or scenario preset above to view matching scene examples.'))
|
| 376 |
if annotation_available and matched:
|
| 377 |
heading('地域与时间分布', 'Geographic & temporal distribution',
|
| 378 |
+
'城市按代表帧计数;采集时段按关联有效视频的实际时长计量。',
|
| 379 |
+
'Cities count representative frames; collection periods measure actual duration of associated active videos.', 'nav-params')
|
| 380 |
+
coverage_catalog, coverage_status = load_coverage() if load_coverage else (None, 'not_published')
|
| 381 |
+
member_coverage = selected_member_coverage(
|
| 382 |
+
matched, coverage_catalog, date_range=st.session_state.get('pg_date_range', ()),
|
| 383 |
+
include_unknown_dates=st.session_state['pg_include_unknown_dates'],
|
| 384 |
+
codecs=st.session_state['pg_codecs'], periods=st.session_state['pg_periods'],
|
| 385 |
+
) if coverage_catalog is not None and coverage_status == 'ok' else None
|
| 386 |
+
|
| 387 |
+
def coverage_notice() -> None:
|
| 388 |
+
if coverage_status in ('not_published', 'not_configured'):
|
| 389 |
+
st.info(tr('关联视频时长尚未发布,暂无法显示时间统计。',
|
| 390 |
+
'Associated video duration has not been published; temporal statistics are unavailable.'))
|
| 391 |
+
else:
|
| 392 |
+
st.warning(tr('关联视频时长暂不可用,请稍后刷新。',
|
| 393 |
+
'Associated video duration is unavailable. Please refresh later.'))
|
| 394 |
+
|
| 395 |
left, right = st.columns(2)
|
| 396 |
with left:
|
| 397 |
+
bar(dict(Counter(r.city for r in matched)), tr('各城市代表帧数量', 'Representative frames by city'), dimension='city')
|
| 398 |
with right:
|
| 399 |
+
if member_coverage is None:
|
| 400 |
+
coverage_notice()
|
| 401 |
+
elif not member_coverage.members and not member_coverage.missing_batch_codes:
|
| 402 |
+
st.info(tr('当前匹配批次中没有关联视频符合日期、编码及时段筛选。',
|
| 403 |
+
'No associated videos in matching batches pass the date, codec and period filters.'))
|
| 404 |
+
elif member_coverage.known_hours is None:
|
| 405 |
+
st.info(tr('当前匹配的关联视频时长均未知或缺少覆盖,无法计算小时数。',
|
| 406 |
+
'Associated video durations are all unknown or coverage is missing; hours cannot be calculated.'))
|
| 407 |
+
else:
|
| 408 |
+
hours_by_period = period_hours(member_coverage)
|
| 409 |
+
bar({period: hours for period, hours in hours_by_period.items() if hours is not None},
|
| 410 |
+
tr('关联视频覆盖(小时)', 'Associated video coverage (hours)'), key='classic_period_hours')
|
| 411 |
+
unknown_periods = [value_label(period) for period, hours in hours_by_period.items() if hours is None]
|
| 412 |
+
if unknown_periods:
|
| 413 |
+
st.caption(tr('以下时段的时长均未知:', 'Durations are all unknown for these periods: ')
|
| 414 |
+
+ ', '.join(unknown_periods))
|
| 415 |
+
if member_coverage is not None:
|
| 416 |
+
st.caption(tr(
|
| 417 |
+
f'已知时长视频 {member_coverage.known_videos:,};未知时长视频 {member_coverage.unknown_videos:,};'
|
| 418 |
+
f'缺少覆盖的批次 {len(member_coverage.missing_batch_codes):,}。',
|
| 419 |
+
f'Known-duration videos: {member_coverage.known_videos:,}; unknown-duration videos: {member_coverage.unknown_videos:,}; '
|
| 420 |
+
f'batches missing coverage: {len(member_coverage.missing_batch_codes):,}.'))
|
| 421 |
+
st.caption(tr(
|
| 422 |
+
'小时数按全部匹配批次关联的有效视频实际时长求和,不按地点示例去重;未知时长不估算。'
|
| 423 |
+
'时段取每条视频的采集时间,不拆分视频;不证明天气或目标在关联视频中持续存在。',
|
| 424 |
+
'Hours sum actual duration of active videos in every matching batch, before point-example deduplication; unknown durations are not estimated. '
|
| 425 |
+
'Each video uses its collection timestamp period without splitting. Frame labels do not prove weather or object persistence through associated videos.'))
|
| 426 |
heading('道路类型分布', 'Road type distribution',
|
| 427 |
'道路大类、子类与交叉类型;多选交叉类型按每帧总权重 1 分摊。',
|
| 428 |
'Road category, subcategory and intersection; multiple values share a total weight of one per frame.')
|
|
|
|
| 459 |
row = st.selectbox(tr('维度 1', 'Dimension 1'), dimensions, format_func=field_label, key='classic_cross_row')
|
| 460 |
with right:
|
| 461 |
col = st.selectbox(tr('维度 2', 'Dimension 2'), dimensions, index=1, format_func=field_label, key='classic_cross_column')
|
| 462 |
+
if 'period' in (row, col):
|
| 463 |
+
table = coverage_cross_table(member_coverage, row, col) if member_coverage is not None else None
|
| 464 |
+
if table is None:
|
| 465 |
+
coverage_notice()
|
| 466 |
+
else:
|
| 467 |
+
st.caption(tr('单位:关联视频覆盖小时;多选标签均分时长。空值表示该组合时长均未知。',
|
| 468 |
+
'Unit: associated video coverage hours. Multiple tag values share duration equally; blank cells have unknown duration.'))
|
| 469 |
+
else:
|
| 470 |
+
table = cross_table(matched, row, col)
|
| 471 |
+
st.caption(tr('单位:代表帧;多选维度按每帧总权重 1 分摊。',
|
| 472 |
+
'Unit: representative frames; multiple values share a total weight of one per frame.'))
|
| 473 |
+
if table is not None:
|
| 474 |
+
table.index = pd.Index([city_display_name(str(v)) if row == 'city' else value_label(str(v)) for v in table.index], name=row)
|
| 475 |
+
table.columns = pd.Index([city_display_name(str(v)) if col == 'city' else value_label(str(v)) for v in table.columns], name=col)
|
| 476 |
+
st.dataframe(table, width='stretch')
|
| 477 |
heading('ODD 覆盖度', 'ODD coverage',
|
| 478 |
'道路子类 × 天气的代表帧观察密度。0 表示没有样本观察,不证明关联视频全程不存在该条件。',
|
| 479 |
'Road subcategory × weather observations. Zero means no representative-frame observation, not proven absence throughout videos.')
|
classic_statistics.py
CHANGED
|
@@ -9,12 +9,18 @@ from __future__ import annotations
|
|
| 9 |
|
| 10 |
from collections import Counter
|
| 11 |
from collections.abc import Iterable, Mapping, Sequence
|
|
|
|
|
|
|
| 12 |
from itertools import product
|
|
|
|
| 13 |
|
| 14 |
import pandas as pd
|
| 15 |
|
| 16 |
from annotation_statistics import AnnotationRecord, definitions
|
| 17 |
|
|
|
|
|
|
|
|
|
|
| 18 |
|
| 19 |
_FIELDS = {item["code"]: item for item in definitions()}
|
| 20 |
_STATES = ("@unknown", "@not_observed", "@unlabeled", "@not_applicable")
|
|
@@ -28,7 +34,7 @@ SCENARIOS = [
|
|
| 28 |
{"key": "structured", "zh": "结构化道路场景", "en": "Structured roads",
|
| 29 |
"filters": {"top_road_subcategory": ["高速公路", "快速路"]}},
|
| 30 |
{"key": "urban", "zh": "城市道路", "en": "Urban roads",
|
| 31 |
-
"filters": {"
|
| 32 |
{"key": "parking", "zh": "停车场区域", "en": "Parking areas",
|
| 33 |
"filters": {"top_road_subcategory": ["室外停车场", "室内停车场", "路侧停车位"]}},
|
| 34 |
{"key": "low_light", "zh": "低光场景", "en": "Low-light scenes",
|
|
@@ -145,6 +151,113 @@ def cross_table(
|
|
| 145 |
return table
|
| 146 |
|
| 147 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 148 |
def coverage_table(
|
| 149 |
records: Iterable[AnnotationRecord], universe: Iterable[AnnotationRecord] | None = None,
|
| 150 |
) -> pd.DataFrame:
|
|
|
|
| 9 |
|
| 10 |
from collections import Counter
|
| 11 |
from collections.abc import Iterable, Mapping, Sequence
|
| 12 |
+
from dataclasses import dataclass
|
| 13 |
+
from datetime import date
|
| 14 |
from itertools import product
|
| 15 |
+
from typing import TYPE_CHECKING
|
| 16 |
|
| 17 |
import pandas as pd
|
| 18 |
|
| 19 |
from annotation_statistics import AnnotationRecord, definitions
|
| 20 |
|
| 21 |
+
if TYPE_CHECKING:
|
| 22 |
+
from annotation_coverage import CoverageCatalog, CoverageSlice
|
| 23 |
+
|
| 24 |
|
| 25 |
_FIELDS = {item["code"]: item for item in definitions()}
|
| 26 |
_STATES = ("@unknown", "@not_observed", "@unlabeled", "@not_applicable")
|
|
|
|
| 34 |
{"key": "structured", "zh": "结构化道路场景", "en": "Structured roads",
|
| 35 |
"filters": {"top_road_subcategory": ["高速公路", "快速路"]}},
|
| 36 |
{"key": "urban", "zh": "城市道路", "en": "Urban roads",
|
| 37 |
+
"filters": {"top_road_category": ["城市道路"]}},
|
| 38 |
{"key": "parking", "zh": "停车场区域", "en": "Parking areas",
|
| 39 |
"filters": {"top_road_subcategory": ["室外停车场", "室内停车场", "路侧停车位"]}},
|
| 40 |
{"key": "low_light", "zh": "低光场景", "en": "Low-light scenes",
|
|
|
|
| 151 |
return table
|
| 152 |
|
| 153 |
|
| 154 |
+
@dataclass(frozen=True)
|
| 155 |
+
class SelectedCoverage:
|
| 156 |
+
"""Selected member buckets with the exact frame supplying their ODD labels."""
|
| 157 |
+
|
| 158 |
+
members: tuple[tuple[AnnotationRecord, CoverageSlice], ...]
|
| 159 |
+
missing_batch_codes: tuple[str | None, ...]
|
| 160 |
+
|
| 161 |
+
@property
|
| 162 |
+
def known_seconds(self) -> float:
|
| 163 |
+
return sum(bucket.known_seconds for _, bucket in self.members)
|
| 164 |
+
|
| 165 |
+
@property
|
| 166 |
+
def known_videos(self) -> int:
|
| 167 |
+
return sum(bucket.known_videos for _, bucket in self.members)
|
| 168 |
+
|
| 169 |
+
@property
|
| 170 |
+
def unknown_videos(self) -> int:
|
| 171 |
+
return sum(bucket.unknown_videos for _, bucket in self.members)
|
| 172 |
+
|
| 173 |
+
@property
|
| 174 |
+
def known_hours(self) -> float | None:
|
| 175 |
+
return self.known_seconds / 3600 if self.known_videos else None
|
| 176 |
+
|
| 177 |
+
|
| 178 |
+
def selected_member_coverage(
|
| 179 |
+
records: Iterable[AnnotationRecord], catalog: CoverageCatalog, *,
|
| 180 |
+
date_range: Sequence[date] | date = (), include_unknown_dates: bool = True,
|
| 181 |
+
codecs: Iterable[str] = (), periods: Iterable[str] = (),
|
| 182 |
+
) -> SelectedCoverage:
|
| 183 |
+
"""Join matched frame batches first, then filter their actual member metadata.
|
| 184 |
+
|
| 185 |
+
Empty source records remain missing coverage even if filters exclude all
|
| 186 |
+
buckets of another batch. Repeated frame inputs never multiply batch hours.
|
| 187 |
+
"""
|
| 188 |
+
dates = (date_range, date_range) if isinstance(date_range, date) else date_range
|
| 189 |
+
start, end = (dates[0], dates[-1]) if dates else (None, None)
|
| 190 |
+
codecs, periods = set(codecs), set(periods)
|
| 191 |
+
batches = {batch.batch_code: batch for batch in catalog.records}
|
| 192 |
+
frames = {record.batch_code: record for record in records}
|
| 193 |
+
members, missing = [], []
|
| 194 |
+
for code, record in frames.items():
|
| 195 |
+
batch = batches.get(code)
|
| 196 |
+
if batch is None or not batch.slices:
|
| 197 |
+
missing.append(code)
|
| 198 |
+
continue
|
| 199 |
+
for bucket in batch.slices:
|
| 200 |
+
if codecs and bucket.codec not in codecs:
|
| 201 |
+
continue
|
| 202 |
+
if periods and bucket.period not in periods:
|
| 203 |
+
continue
|
| 204 |
+
if bucket.collected_on is None:
|
| 205 |
+
if not include_unknown_dates:
|
| 206 |
+
continue
|
| 207 |
+
elif ((start is not None and bucket.collected_on < start)
|
| 208 |
+
or (end is not None and bucket.collected_on > end)):
|
| 209 |
+
continue
|
| 210 |
+
members.append((record, bucket))
|
| 211 |
+
return SelectedCoverage(tuple(members), tuple(missing))
|
| 212 |
+
|
| 213 |
+
|
| 214 |
+
def period_hours(coverage: SelectedCoverage) -> dict[str, float | None]:
|
| 215 |
+
"""Actual member hours by timestamp-derived period; None means all unknown."""
|
| 216 |
+
seconds, known = Counter(), Counter()
|
| 217 |
+
for _, bucket in coverage.members:
|
| 218 |
+
seconds[bucket.period] += bucket.known_seconds
|
| 219 |
+
known[bucket.period] += bucket.known_videos
|
| 220 |
+
return {period: value / 3600 if known[period] else None for period, value in seconds.items()}
|
| 221 |
+
|
| 222 |
+
|
| 223 |
+
def coverage_cross_table(
|
| 224 |
+
coverage: SelectedCoverage, row_dimension: str, column_dimension: str,
|
| 225 |
+
) -> pd.DataFrame:
|
| 226 |
+
"""Cross a member period with frame labels, conserving known member hours.
|
| 227 |
+
|
| 228 |
+
A multi-value frame label divides its batch-associated hours equally.
|
| 229 |
+
Unknown-only cells are NaN, while a measured zero remains zero. Period
|
| 230 |
+
crossed with itself has only diagonal mass and never invents co-occurrences.
|
| 231 |
+
"""
|
| 232 |
+
_require_dimension(row_dimension)
|
| 233 |
+
_require_dimension(column_dimension)
|
| 234 |
+
if 'period' not in (row_dimension, column_dimension):
|
| 235 |
+
raise ValueError('Coverage cross tables require a period dimension')
|
| 236 |
+
|
| 237 |
+
def values(frame, bucket, dimension):
|
| 238 |
+
return (bucket.period,) if dimension == 'period' else _values(frame, dimension)
|
| 239 |
+
|
| 240 |
+
axes = [_ordered((value for frame, bucket in coverage.members
|
| 241 |
+
for value in values(frame, bucket, dimension)), dimension)
|
| 242 |
+
for dimension in (row_dimension, column_dimension)]
|
| 243 |
+
table = pd.DataFrame(0.0, index=pd.Index(axes[0], name=row_dimension),
|
| 244 |
+
columns=pd.Index(axes[1], name=column_dimension))
|
| 245 |
+
observed, unknown = set(), set()
|
| 246 |
+
for frame, bucket in coverage.members:
|
| 247 |
+
rows = values(frame, bucket, row_dimension)
|
| 248 |
+
columns = values(frame, bucket, column_dimension)
|
| 249 |
+
pairs = tuple(zip(rows, columns)) if row_dimension == column_dimension else tuple(product(rows, columns))
|
| 250 |
+
for pair in pairs:
|
| 251 |
+
if bucket.known_videos:
|
| 252 |
+
table.at[pair[0], pair[1]] += bucket.known_seconds / 3600 / len(pairs)
|
| 253 |
+
observed.add(pair)
|
| 254 |
+
elif bucket.unknown_videos:
|
| 255 |
+
unknown.add(pair)
|
| 256 |
+
for row, column in unknown - observed:
|
| 257 |
+
table.at[row, column] = float('nan')
|
| 258 |
+
return table
|
| 259 |
+
|
| 260 |
+
|
| 261 |
def coverage_table(
|
| 262 |
records: Iterable[AnnotationRecord], universe: Iterable[AnnotationRecord] | None = None,
|
| 263 |
) -> pd.DataFrame:
|
dashboard_st.py
CHANGED
|
@@ -35,7 +35,7 @@ I18N_EN = {
|
|
| 35 |
"⚠️ 本小时请求已达上限(20 次),请稍后再试": "⚠️ Hourly request limit reached (20). Please try again later.",
|
| 36 |
"示例问题": "Example questions",
|
| 37 |
"数据集一共覆盖了哪些城市?": "Which cities does the dataset cover?",
|
| 38 |
-
"
|
| 39 |
"继续追问…": "Ask a follow-up…",
|
| 40 |
"新对话": "New chat",
|
| 41 |
"开始使用智能问答": "Get access to AI Q&A",
|
|
@@ -66,6 +66,9 @@ def T(s):
|
|
| 66 |
Unknown keys pass through unchanged (safe degradation)."""
|
| 67 |
if not isinstance(s, str):
|
| 68 |
return s
|
|
|
|
|
|
|
|
|
|
| 69 |
lang = st.session_state.get("lang", "zh") if hasattr(st, "session_state") else "zh"
|
| 70 |
if lang == "en":
|
| 71 |
return I18N_EN.get(s, s)
|
|
@@ -138,6 +141,7 @@ def _clear_catalog() -> None:
|
|
| 138 |
_cached_annotations.clear()
|
| 139 |
_cached_updates.clear()
|
| 140 |
_cached_locations.clear()
|
|
|
|
| 141 |
_cached_public_snapshot.clear()
|
| 142 |
|
| 143 |
|
|
@@ -206,6 +210,28 @@ def _load_locations():
|
|
| 206 |
return None, "unavailable"
|
| 207 |
|
| 208 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 209 |
def _render_publication_notice():
|
| 210 |
if not _public_mode():
|
| 211 |
return
|
|
@@ -238,7 +264,7 @@ def page_dashboard():
|
|
| 238 |
from classic_dashboard import render_classic_dashboard
|
| 239 |
render_classic_dashboard(_load_catalog, _load_annotations, _load_updates,
|
| 240 |
_clear_catalog, T, render_source_notice=_render_publication_notice,
|
| 241 |
-
load_locations=_load_locations)
|
| 242 |
if not _public_mode():
|
| 243 |
render_annotation_progress(_inventory_dsn(), T)
|
| 244 |
_render_footer()
|
|
@@ -1625,6 +1651,7 @@ def _qa_dataset_facts(catalog) -> str:
|
|
| 1625 |
from collections import Counter
|
| 1626 |
from live_inventory import summarize_records
|
| 1627 |
from postgres_dashboard import selected_records
|
|
|
|
| 1628 |
|
| 1629 |
records = selected_records(catalog, st.session_state)
|
| 1630 |
snapshot = summarize_records(records, catalog.latest_updated_at)
|
|
@@ -1635,6 +1662,16 @@ def _qa_dataset_facts(catalog) -> str:
|
|
| 1635 |
capacity = (f"{snapshot.size_tb:.3f} TB (known-value total, {size_known}/{len(records)} videos)"
|
| 1636 |
if size_known or not records else "unavailable (no known size values)")
|
| 1637 |
counts = lambda values: dict(sorted(Counter(values).items()))
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1638 |
resolutions = counts(
|
| 1639 |
f"{row.width}x{row.height}" if row.width and row.height else "unknown"
|
| 1640 |
for row in records
|
|
@@ -1653,9 +1690,10 @@ def _qa_dataset_facts(catalog) -> str:
|
|
| 1653 |
f"Full active inventory: {len(catalog.records)} videos.\n"
|
| 1654 |
f"Active videos: {snapshot.videos} (current dashboard filters where available).\n"
|
| 1655 |
f"Duration: {duration}; Size: {capacity}.\n"
|
| 1656 |
-
f"City counts assigned from GPS: {counts(row.city for row in records)}\n"
|
| 1657 |
f"Collection months: {dict((row.month, row.videos) for row in snapshot.month_totals)}\n"
|
| 1658 |
-
f"Collection
|
|
|
|
| 1659 |
f"Codecs: {counts(row.codec for row in records)}\n"
|
| 1660 |
f"Resolutions: {resolutions}\n"
|
| 1661 |
f"Missing metadata counts: {missing}\n"
|
|
@@ -1673,6 +1711,7 @@ Every dataset fact below is derived solely from active PostgreSQL aerial_db.aeri
|
|
| 1673 |
{dataset_facts}
|
| 1674 |
|
| 1675 |
Answer concisely in the user's language. Use only the supplied aggregate facts for dataset numbers.
|
|
|
|
| 1676 |
City attribution uses SRT-derived GPS and pinned 2023 administrative boundaries; boundary cases need review.
|
| 1677 |
Do not infer city from folder names. Unknown metadata is missing information, not an observed zero.
|
| 1678 |
If PostgreSQL is unavailable, say the current statistics are unavailable and do not estimate totals.
|
|
@@ -1761,7 +1800,7 @@ def page_qa():
|
|
| 1761 |
if not history:
|
| 1762 |
examples = [
|
| 1763 |
T("数据集一共覆盖了哪些城市?"),
|
| 1764 |
-
T("
|
| 1765 |
T("哪些视频缺少经纬度或参数信息?"),
|
| 1766 |
T("各城市的视频数量和总时长是多少?"),
|
| 1767 |
]
|
|
@@ -1885,7 +1924,6 @@ NAV_ITEMS = [
|
|
| 1885 |
("nav-scenario", "anchor", "场景速查", "Quick-pick"),
|
| 1886 |
("nav-city", "anchor", "城市分布", "Cities"),
|
| 1887 |
("nav-odd", "anchor", "ODD 维度浏览", "ODD"),
|
| 1888 |
-
("nav-examples", "anchor", "场景示例", "Examples"),
|
| 1889 |
("nav-params", "anchor", "参数分布", "Distributions"),
|
| 1890 |
("nav-updates", "anchor", "数据更新", "Data updates"),
|
| 1891 |
("annotation", "page", "标注进度", "Annotation progress"),
|
|
@@ -1993,7 +2031,7 @@ def main():
|
|
| 1993 |
(function() {
|
| 1994 |
try {
|
| 1995 |
const W = window.parent, D = W.document;
|
| 1996 |
-
const ANCHORS = ['nav-scenario', 'nav-city', 'nav-odd', 'nav-
|
| 1997 |
// Streamlit ≥1.40 uses [data-testid="stMain"] as the scrollable
|
| 1998 |
// container (overflow-y: auto), NOT the window. Browser-default
|
| 1999 |
// anchor scroll and IntersectionObserver (root=viewport) both
|
|
|
|
| 35 |
"⚠️ 本小时请求已达上限(20 次),请稍后再试": "⚠️ Hourly request limit reached (20). Please try again later.",
|
| 36 |
"示例问题": "Example questions",
|
| 37 |
"数据集一共覆盖了哪些城市?": "Which cities does the dataset cover?",
|
| 38 |
+
"Changchun (Jilin) 的数据为什么占比这么高?": "Why does Changchun (Jilin) make up such a large share?",
|
| 39 |
"继续追问…": "Ask a follow-up…",
|
| 40 |
"新对话": "New chat",
|
| 41 |
"开始使用智能问答": "Get access to AI Q&A",
|
|
|
|
| 66 |
Unknown keys pass through unchanged (safe degradation)."""
|
| 67 |
if not isinstance(s, str):
|
| 68 |
return s
|
| 69 |
+
from city_labels import city_display_name, is_city_label
|
| 70 |
+
if is_city_label(s):
|
| 71 |
+
return city_display_name(s)
|
| 72 |
lang = st.session_state.get("lang", "zh") if hasattr(st, "session_state") else "zh"
|
| 73 |
if lang == "en":
|
| 74 |
return I18N_EN.get(s, s)
|
|
|
|
| 141 |
_cached_annotations.clear()
|
| 142 |
_cached_updates.clear()
|
| 143 |
_cached_locations.clear()
|
| 144 |
+
_cached_coverage.clear()
|
| 145 |
_cached_public_snapshot.clear()
|
| 146 |
|
| 147 |
|
|
|
|
| 210 |
return None, "unavailable"
|
| 211 |
|
| 212 |
|
| 213 |
+
@st.cache_data(ttl=60)
|
| 214 |
+
def _cached_coverage(dsn: str):
|
| 215 |
+
from annotation_coverage import fetch_coverage_catalog
|
| 216 |
+
return fetch_coverage_catalog(dsn)
|
| 217 |
+
|
| 218 |
+
|
| 219 |
+
def _load_coverage():
|
| 220 |
+
if _public_mode():
|
| 221 |
+
try:
|
| 222 |
+
coverage = getattr(_cached_public_snapshot(_snapshot_url()), 'coverage', None)
|
| 223 |
+
return (coverage, 'ok') if coverage is not None else (None, 'not_published')
|
| 224 |
+
except Exception:
|
| 225 |
+
return None, 'unavailable'
|
| 226 |
+
dsn = _inventory_dsn()
|
| 227 |
+
if not dsn:
|
| 228 |
+
return None, 'not_configured'
|
| 229 |
+
try:
|
| 230 |
+
return _cached_coverage(dsn), 'ok'
|
| 231 |
+
except Exception:
|
| 232 |
+
return None, 'unavailable'
|
| 233 |
+
|
| 234 |
+
|
| 235 |
def _render_publication_notice():
|
| 236 |
if not _public_mode():
|
| 237 |
return
|
|
|
|
| 264 |
from classic_dashboard import render_classic_dashboard
|
| 265 |
render_classic_dashboard(_load_catalog, _load_annotations, _load_updates,
|
| 266 |
_clear_catalog, T, render_source_notice=_render_publication_notice,
|
| 267 |
+
load_locations=_load_locations, load_coverage=_load_coverage)
|
| 268 |
if not _public_mode():
|
| 269 |
render_annotation_progress(_inventory_dsn(), T)
|
| 270 |
_render_footer()
|
|
|
|
| 1651 |
from collections import Counter
|
| 1652 |
from live_inventory import summarize_records
|
| 1653 |
from postgres_dashboard import selected_records
|
| 1654 |
+
from city_labels import city_display_name
|
| 1655 |
|
| 1656 |
records = selected_records(catalog, st.session_state)
|
| 1657 |
snapshot = summarize_records(records, catalog.latest_updated_at)
|
|
|
|
| 1662 |
capacity = (f"{snapshot.size_tb:.3f} TB (known-value total, {size_known}/{len(records)} videos)"
|
| 1663 |
if size_known or not records else "unavailable (no known size values)")
|
| 1664 |
counts = lambda values: dict(sorted(Counter(values).items()))
|
| 1665 |
+
period_seconds, period_known, period_unknown = Counter(), Counter(), Counter()
|
| 1666 |
+
for record in records:
|
| 1667 |
+
period_seconds.setdefault(record.period, 0.0)
|
| 1668 |
+
if record.duration_seconds is None:
|
| 1669 |
+
period_unknown[record.period] += 1
|
| 1670 |
+
else:
|
| 1671 |
+
period_seconds[record.period] += record.duration_seconds
|
| 1672 |
+
period_known[record.period] += 1
|
| 1673 |
+
period_hours = {period: round(seconds / 3600, 6) if period_known[period] else None
|
| 1674 |
+
for period, seconds in sorted(period_seconds.items())}
|
| 1675 |
resolutions = counts(
|
| 1676 |
f"{row.width}x{row.height}" if row.width and row.height else "unknown"
|
| 1677 |
for row in records
|
|
|
|
| 1690 |
f"Full active inventory: {len(catalog.records)} videos.\n"
|
| 1691 |
f"Active videos: {snapshot.videos} (current dashboard filters where available).\n"
|
| 1692 |
f"Duration: {duration}; Size: {capacity}.\n"
|
| 1693 |
+
f"City counts assigned from GPS: {counts(city_display_name(row.city) for row in records)}\n"
|
| 1694 |
f"Collection months: {dict((row.month, row.videos) for row in snapshot.month_totals)}\n"
|
| 1695 |
+
f"Collection period known duration (hours): {period_hours}\n"
|
| 1696 |
+
f"Unknown-duration videos by period: {dict(sorted(period_unknown.items()))}\n"
|
| 1697 |
f"Codecs: {counts(row.codec for row in records)}\n"
|
| 1698 |
f"Resolutions: {resolutions}\n"
|
| 1699 |
f"Missing metadata counts: {missing}\n"
|
|
|
|
| 1711 |
{dataset_facts}
|
| 1712 |
|
| 1713 |
Answer concisely in the user's language. Use only the supplied aggregate facts for dataset numbers.
|
| 1714 |
+
Use the supplied English city/province labels verbatim, including when answering in Chinese.
|
| 1715 |
City attribution uses SRT-derived GPS and pinned 2023 administrative boundaries; boundary cases need review.
|
| 1716 |
Do not infer city from folder names. Unknown metadata is missing information, not an observed zero.
|
| 1717 |
If PostgreSQL is unavailable, say the current statistics are unavailable and do not estimate totals.
|
|
|
|
| 1800 |
if not history:
|
| 1801 |
examples = [
|
| 1802 |
T("数据集一共覆盖了哪些城市?"),
|
| 1803 |
+
T("Changchun (Jilin) 的数据为什么占比这么高?"),
|
| 1804 |
T("哪些视频缺少经纬度或参数信息?"),
|
| 1805 |
T("各城市的视频数量和总时长是多少?"),
|
| 1806 |
]
|
|
|
|
| 1924 |
("nav-scenario", "anchor", "场景速查", "Quick-pick"),
|
| 1925 |
("nav-city", "anchor", "城市分布", "Cities"),
|
| 1926 |
("nav-odd", "anchor", "ODD 维度浏览", "ODD"),
|
|
|
|
| 1927 |
("nav-params", "anchor", "参数分布", "Distributions"),
|
| 1928 |
("nav-updates", "anchor", "数据更新", "Data updates"),
|
| 1929 |
("annotation", "page", "标注进度", "Annotation progress"),
|
|
|
|
| 2031 |
(function() {
|
| 2032 |
try {
|
| 2033 |
const W = window.parent, D = W.document;
|
| 2034 |
+
const ANCHORS = ['nav-scenario', 'nav-city', 'nav-odd', 'nav-params', 'nav-updates'];
|
| 2035 |
// Streamlit ≥1.40 uses [data-testid="stMain"] as the scrollable
|
| 2036 |
// container (overflow-y: auto), NOT the window. Browser-default
|
| 2037 |
// anchor scroll and IntersectionObserver (root=viewport) both
|
postgres_dashboard.py
CHANGED
|
@@ -12,6 +12,7 @@ import plotly.graph_objects as go
|
|
| 12 |
import streamlit as st
|
| 13 |
|
| 14 |
from live_inventory import VideoCatalog, VideoRecord, filter_records, summarize_records
|
|
|
|
| 15 |
|
| 16 |
|
| 17 |
_UNKNOWN = "未知"
|
|
@@ -87,7 +88,7 @@ def aggregate_rows(records: Sequence[VideoRecord]) -> pd.DataFrame:
|
|
| 87 |
totals[2] += float(record.size_bytes) / 1_000_000_000_000
|
| 88 |
totals[4] += 1
|
| 89 |
return pd.DataFrame(
|
| 90 |
-
[(*key, totals[0], totals[1] if totals[3] else None,
|
| 91 |
totals[2] if totals[4] else None, totals[3], totals[4])
|
| 92 |
for key, totals in sorted(groups.items())],
|
| 93 |
columns=_EXPORT_COLUMNS,
|
|
@@ -104,9 +105,11 @@ def render_dashboard(load_catalog: Callable, clear_catalog: Callable,
|
|
| 104 |
return translated
|
| 105 |
|
| 106 |
def category(value: str) -> str:
|
|
|
|
|
|
|
| 107 |
labels = {
|
| 108 |
_UNKNOWN: "Unknown", "缺少坐标": "Missing coordinates",
|
| 109 |
-
"边界外或坐标异常": "
|
| 110 |
"已归属城市": "Assigned city", "早高峰 (07-09)": "Morning peak (07-09)",
|
| 111 |
"日常规": "Daytime", "晚高峰 (17-19)": "Evening peak (17-19)",
|
| 112 |
"夜间 (19-07)": "Night (19-07)",
|
|
|
|
| 12 |
import streamlit as st
|
| 13 |
|
| 14 |
from live_inventory import VideoCatalog, VideoRecord, filter_records, summarize_records
|
| 15 |
+
from city_labels import city_display_name, is_city_label
|
| 16 |
|
| 17 |
|
| 18 |
_UNKNOWN = "未知"
|
|
|
|
| 88 |
totals[2] += float(record.size_bytes) / 1_000_000_000_000
|
| 89 |
totals[4] += 1
|
| 90 |
return pd.DataFrame(
|
| 91 |
+
[(city_display_name(key[0]), *key[1:], totals[0], totals[1] if totals[3] else None,
|
| 92 |
totals[2] if totals[4] else None, totals[3], totals[4])
|
| 93 |
for key, totals in sorted(groups.items())],
|
| 94 |
columns=_EXPORT_COLUMNS,
|
|
|
|
| 105 |
return translated
|
| 106 |
|
| 107 |
def category(value: str) -> str:
|
| 108 |
+
if is_city_label(value):
|
| 109 |
+
return city_display_name(value)
|
| 110 |
labels = {
|
| 111 |
_UNKNOWN: "Unknown", "缺少坐标": "Missing coordinates",
|
| 112 |
+
"边界外或坐标异常": "Unassigned city",
|
| 113 |
"已归属城市": "Assigned city", "早高峰 (07-09)": "Morning peak (07-09)",
|
| 114 |
"日常规": "Daytime", "晚高峰 (17-19)": "Evening peak (17-19)",
|
| 115 |
"夜间 (19-07)": "Night (19-07)",
|
public_snapshot.py
CHANGED
|
@@ -32,6 +32,7 @@ from live_inventory import (
|
|
| 32 |
_load_boundaries, _public_city_name,
|
| 33 |
)
|
| 34 |
from location_groups import LocationCatalog, LocationGroup
|
|
|
|
| 35 |
|
| 36 |
|
| 37 |
PUBLIC_SNAPSHOT_URL = "https://raw.githubusercontent.com/AutoZYX-Labs/driveresearch-data-proxy/statistics-data/published/aerial-statistics-v1.json.gz"
|
|
@@ -63,6 +64,7 @@ class PublicSnapshot:
|
|
| 63 |
updates: BatchUpdateCatalog | None = None
|
| 64 |
includes_frames: bool = False
|
| 65 |
locations: LocationCatalog | None = None
|
|
|
|
| 66 |
|
| 67 |
|
| 68 |
def _object(value: object, keys: set[str]) -> dict:
|
|
@@ -244,14 +246,64 @@ def _parse_locations(value: object, annotations: list[AnnotationRecord]) -> Loca
|
|
| 244 |
return LocationCatalog(tuple(groups), unresolved, 100)
|
| 245 |
|
| 246 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 247 |
def _parse_document(document: object) -> PublicSnapshot:
|
| 248 |
if not isinstance(document, dict):
|
| 249 |
raise ValueError("Invalid snapshot object")
|
| 250 |
version = document.get("schema_version")
|
| 251 |
keys = {"schema_version", "source", "generated_at", "catalog", "annotations"}
|
| 252 |
-
document = _object(document, keys | ({"updates"} if version in (2, 3, 4) else set())
|
| 253 |
-
| ({"locations"} if version
|
| 254 |
-
|
|
|
|
| 255 |
raise ValueError("Unsupported snapshot source or schema")
|
| 256 |
generated_at = _timestamp(document["generated_at"])
|
| 257 |
catalog = _object(document["catalog"], {"records", "latest_updated_at"})
|
|
@@ -271,11 +323,11 @@ def _parse_document(document: object) -> PublicSnapshot:
|
|
| 271 |
))
|
| 272 |
# Frame schemas keep an explicit updates key even when that catalog is absent.
|
| 273 |
updates = _parse_updates(document["updates"]) if (
|
| 274 |
-
version == 2 or (version in (3, 4) and document["updates"] is not None)) else None
|
| 275 |
available_codes = {record.code for record in updates.records if record.annotations_available} if updates else None
|
| 276 |
annotations, codes = [], set()
|
| 277 |
for raw in _list(annotation_catalog["records"]):
|
| 278 |
-
raw = _object(raw, _FRAME_ANNOTATION_KEYS if version in (3, 4) else _ANNOTATION_KEYS)
|
| 279 |
compact = _list(raw["labels"])
|
| 280 |
for label in compact:
|
| 281 |
_object(label, {"field", "state", "values"})
|
|
@@ -284,7 +336,7 @@ def _parse_document(document: object) -> PublicSnapshot:
|
|
| 284 |
if model_name not in PUBLIC_MODEL_NAMES:
|
| 285 |
raise ValueError("Unknown public annotation model")
|
| 286 |
references = {}
|
| 287 |
-
if version in (3, 4):
|
| 288 |
code = _text(raw["batch_code"], maximum=18)
|
| 289 |
if (not re.fullmatch(r"B-[0-9a-f]{16}", code) or code in codes
|
| 290 |
or (available_codes is not None and code not in available_codes)):
|
|
@@ -303,8 +355,10 @@ def _parse_document(document: object) -> PublicSnapshot:
|
|
| 303 |
_integer(annotation_catalog["excluded_records"], optional=False)),
|
| 304 |
generated_at=generated_at,
|
| 305 |
updates=updates,
|
| 306 |
-
includes_frames=version in (3, 4),
|
| 307 |
-
locations=_parse_locations(document["locations"], annotations) if
|
|
|
|
|
|
|
| 308 |
)
|
| 309 |
|
| 310 |
|
|
@@ -314,11 +368,13 @@ def _canonical_json(value: object) -> bytes:
|
|
| 314 |
|
| 315 |
def _document(catalog: VideoCatalog, annotations: AnnotationCatalog, generated_at: datetime,
|
| 316 |
updates: BatchUpdateCatalog | None = None, *, include_frames: bool = False,
|
| 317 |
-
locations: LocationCatalog | None = None) -> dict:
|
| 318 |
if type(include_frames) is not bool:
|
| 319 |
raise ValueError("Invalid frame publication flag")
|
| 320 |
if locations is not None and not include_frames:
|
| 321 |
raise ValueError("Location publication requires exact batch/frame references")
|
|
|
|
|
|
|
| 322 |
videos = []
|
| 323 |
for record in catalog.records:
|
| 324 |
# Explicit field selection also strips any accidental future dataclass additions.
|
|
@@ -367,20 +423,33 @@ def _document(catalog: VideoCatalog, annotations: AnnotationCatalog, generated_a
|
|
| 367 |
key=lambda group: group["code"]),
|
| 368 |
"unresolved_batch_codes": sorted(locations.unresolved_batch_codes),
|
| 369 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 370 |
_parse_document(document)
|
| 371 |
return document
|
| 372 |
|
| 373 |
|
| 374 |
def encode_snapshot(catalog: VideoCatalog, annotations: AnnotationCatalog, generated_at: datetime,
|
| 375 |
updates: BatchUpdateCatalog | None = None, *, include_frames: bool = False,
|
| 376 |
-
locations: LocationCatalog | None = None) -> bytes:
|
| 377 |
"""Encode V1/V2 by default, V3 frames, or V4 frames and location memberships.
|
| 378 |
|
| 379 |
V3 requires safe unique batch codes; a missing frame is null. Its updates
|
| 380 |
key is also explicit and nullable when no update catalog is supplied.
|
| 381 |
"""
|
| 382 |
raw = _canonical_json(_document(catalog, annotations, generated_at, updates,
|
| 383 |
-
include_frames=include_frames, locations=locations))
|
| 384 |
if len(raw) > MAX_DECOMPRESSED_BYTES:
|
| 385 |
raise ValueError("Public statistics snapshot is too large")
|
| 386 |
encoded = gzip.compress(raw, mtime=0)
|
|
@@ -391,10 +460,10 @@ def encode_snapshot(catalog: VideoCatalog, annotations: AnnotationCatalog, gener
|
|
| 391 |
|
| 392 |
def semantic_digest(catalog: VideoCatalog, annotations: AnnotationCatalog,
|
| 393 |
updates: BatchUpdateCatalog | None = None, *, include_frames: bool = False,
|
| 394 |
-
locations: LocationCatalog | None = None) -> str:
|
| 395 |
"""Hash facts, excluding row order, export time and discovery-only timestamps."""
|
| 396 |
document = _document(catalog, annotations, datetime(2000, 1, 1, tzinfo=timezone.utc), updates,
|
| 397 |
-
include_frames=include_frames, locations=locations)
|
| 398 |
del document["generated_at"]
|
| 399 |
del document["catalog"]["latest_updated_at"]
|
| 400 |
del document["annotations"]["latest_updated_at"]
|
|
@@ -488,7 +557,8 @@ def _fetch_checked_at(url: str, snapshot: PublicSnapshot) -> datetime | None:
|
|
| 488 |
or document["source"] != SOURCE
|
| 489 |
or document["semantic_digest"] != semantic_digest(snapshot.catalog, snapshot.annotations, snapshot.updates,
|
| 490 |
include_frames=snapshot.includes_frames,
|
| 491 |
-
locations=snapshot.locations
|
|
|
|
| 492 |
return None
|
| 493 |
checked_at = _timestamp(document["checked_at"])
|
| 494 |
return checked_at if checked_at >= snapshot.generated_at else None
|
|
|
|
| 32 |
_load_boundaries, _public_city_name,
|
| 33 |
)
|
| 34 |
from location_groups import LocationCatalog, LocationGroup
|
| 35 |
+
from annotation_coverage import BatchCoverage, CoverageCatalog, CoverageSlice
|
| 36 |
|
| 37 |
|
| 38 |
PUBLIC_SNAPSHOT_URL = "https://raw.githubusercontent.com/AutoZYX-Labs/driveresearch-data-proxy/statistics-data/published/aerial-statistics-v1.json.gz"
|
|
|
|
| 64 |
updates: BatchUpdateCatalog | None = None
|
| 65 |
includes_frames: bool = False
|
| 66 |
locations: LocationCatalog | None = None
|
| 67 |
+
coverage: CoverageCatalog | None = None
|
| 68 |
|
| 69 |
|
| 70 |
def _object(value: object, keys: set[str]) -> dict:
|
|
|
|
| 246 |
return LocationCatalog(tuple(groups), unresolved, 100)
|
| 247 |
|
| 248 |
|
| 249 |
+
def _parse_coverage(value: object, annotations: list[AnnotationRecord], videos: list[VideoRecord]) -> CoverageCatalog:
|
| 250 |
+
value = _object(value, {'records'})
|
| 251 |
+
raw_records = _list(value['records'])
|
| 252 |
+
if len(raw_records) > MAX_BATCH_RECORDS:
|
| 253 |
+
raise ValueError('Too many coverage records')
|
| 254 |
+
population = {record.batch_code for record in annotations}
|
| 255 |
+
records, seen, slice_count = [], set(), 0
|
| 256 |
+
known_totals, known_videos, unknown_videos = [], 0, 0
|
| 257 |
+
for raw in raw_records:
|
| 258 |
+
raw = _object(raw, {'batch_code', 'slices'})
|
| 259 |
+
code = _text(raw['batch_code'], maximum=18)
|
| 260 |
+
if not re.fullmatch(r'B-[0-9a-f]{16}', code) or code not in population or code in seen:
|
| 261 |
+
raise ValueError('Invalid coverage batch code')
|
| 262 |
+
seen.add(code)
|
| 263 |
+
slices, keys = [], set()
|
| 264 |
+
for item in _list(raw['slices']):
|
| 265 |
+
slice_count += 1
|
| 266 |
+
if slice_count > MAX_BATCH_RECORDS:
|
| 267 |
+
raise ValueError('Too many coverage buckets')
|
| 268 |
+
item = _object(item, {'collected_on', 'period', 'codec', 'known_seconds', 'known_videos', 'unknown_videos'})
|
| 269 |
+
day, period, codec = _day(item['collected_on']), _text(item['period']), _text(item['codec'], maximum=64)
|
| 270 |
+
if period not in _PERIODS or codec not in PUBLIC_CODECS:
|
| 271 |
+
raise ValueError('Invalid coverage metadata')
|
| 272 |
+
key = day, period, codec
|
| 273 |
+
if key in keys:
|
| 274 |
+
raise ValueError('Duplicate coverage bucket')
|
| 275 |
+
keys.add(key)
|
| 276 |
+
seconds = _number(item['known_seconds'])
|
| 277 |
+
known = _integer(item['known_videos'], optional=False)
|
| 278 |
+
unknown = _integer(item['unknown_videos'], optional=False)
|
| 279 |
+
if seconds is None or known + unknown == 0 or (known == 0 and seconds != 0):
|
| 280 |
+
raise ValueError('Invalid coverage measurement')
|
| 281 |
+
slices.append(CoverageSlice(day, period, codec, seconds, known, unknown))
|
| 282 |
+
known_totals.append(seconds)
|
| 283 |
+
known_videos += known
|
| 284 |
+
unknown_videos += unknown
|
| 285 |
+
slices.sort(key=lambda item: (item.collected_on is None, item.collected_on or date.max, item.period, item.codec))
|
| 286 |
+
records.append(BatchCoverage(code, tuple(slices)))
|
| 287 |
+
if seen != population:
|
| 288 |
+
raise ValueError('Incomplete coverage population')
|
| 289 |
+
inventory_known = sum(record.duration_seconds is not None for record in videos)
|
| 290 |
+
inventory_seconds = math.fsum(record.duration_seconds for record in videos if record.duration_seconds is not None)
|
| 291 |
+
if (known_videos > inventory_known or unknown_videos > len(videos) - inventory_known
|
| 292 |
+
or known_videos + unknown_videos > len(videos)
|
| 293 |
+
or math.fsum(known_totals) > inventory_seconds + max(1e-6, inventory_seconds * 1e-12)):
|
| 294 |
+
raise ValueError('Coverage exceeds active video inventory')
|
| 295 |
+
return CoverageCatalog(tuple(sorted(records, key=lambda record: record.batch_code)))
|
| 296 |
+
|
| 297 |
+
|
| 298 |
def _parse_document(document: object) -> PublicSnapshot:
|
| 299 |
if not isinstance(document, dict):
|
| 300 |
raise ValueError("Invalid snapshot object")
|
| 301 |
version = document.get("schema_version")
|
| 302 |
keys = {"schema_version", "source", "generated_at", "catalog", "annotations"}
|
| 303 |
+
document = _object(document, keys | ({"updates"} if version in (2, 3, 4, 5) else set())
|
| 304 |
+
| ({"locations"} if version in (4, 5) else set())
|
| 305 |
+
| ({"coverage"} if version == 5 else set()))
|
| 306 |
+
if type(version) is not int or version not in (1, 2, 3, 4, 5) or document["source"] != SOURCE:
|
| 307 |
raise ValueError("Unsupported snapshot source or schema")
|
| 308 |
generated_at = _timestamp(document["generated_at"])
|
| 309 |
catalog = _object(document["catalog"], {"records", "latest_updated_at"})
|
|
|
|
| 323 |
))
|
| 324 |
# Frame schemas keep an explicit updates key even when that catalog is absent.
|
| 325 |
updates = _parse_updates(document["updates"]) if (
|
| 326 |
+
version == 2 or (version in (3, 4, 5) and document["updates"] is not None)) else None
|
| 327 |
available_codes = {record.code for record in updates.records if record.annotations_available} if updates else None
|
| 328 |
annotations, codes = [], set()
|
| 329 |
for raw in _list(annotation_catalog["records"]):
|
| 330 |
+
raw = _object(raw, _FRAME_ANNOTATION_KEYS if version in (3, 4, 5) else _ANNOTATION_KEYS)
|
| 331 |
compact = _list(raw["labels"])
|
| 332 |
for label in compact:
|
| 333 |
_object(label, {"field", "state", "values"})
|
|
|
|
| 336 |
if model_name not in PUBLIC_MODEL_NAMES:
|
| 337 |
raise ValueError("Unknown public annotation model")
|
| 338 |
references = {}
|
| 339 |
+
if version in (3, 4, 5):
|
| 340 |
code = _text(raw["batch_code"], maximum=18)
|
| 341 |
if (not re.fullmatch(r"B-[0-9a-f]{16}", code) or code in codes
|
| 342 |
or (available_codes is not None and code not in available_codes)):
|
|
|
|
| 355 |
_integer(annotation_catalog["excluded_records"], optional=False)),
|
| 356 |
generated_at=generated_at,
|
| 357 |
updates=updates,
|
| 358 |
+
includes_frames=version in (3, 4, 5),
|
| 359 |
+
locations=_parse_locations(document["locations"], annotations) if (
|
| 360 |
+
version == 4 or (version == 5 and document['locations'] is not None)) else None,
|
| 361 |
+
coverage=_parse_coverage(document['coverage'], annotations, videos) if version == 5 else None,
|
| 362 |
)
|
| 363 |
|
| 364 |
|
|
|
|
| 368 |
|
| 369 |
def _document(catalog: VideoCatalog, annotations: AnnotationCatalog, generated_at: datetime,
|
| 370 |
updates: BatchUpdateCatalog | None = None, *, include_frames: bool = False,
|
| 371 |
+
locations: LocationCatalog | None = None, coverage: CoverageCatalog | None = None) -> dict:
|
| 372 |
if type(include_frames) is not bool:
|
| 373 |
raise ValueError("Invalid frame publication flag")
|
| 374 |
if locations is not None and not include_frames:
|
| 375 |
raise ValueError("Location publication requires exact batch/frame references")
|
| 376 |
+
if coverage is not None and not include_frames:
|
| 377 |
+
raise ValueError('Coverage publication requires exact batch/frame references')
|
| 378 |
videos = []
|
| 379 |
for record in catalog.records:
|
| 380 |
# Explicit field selection also strips any accidental future dataclass additions.
|
|
|
|
| 423 |
key=lambda group: group["code"]),
|
| 424 |
"unresolved_batch_codes": sorted(locations.unresolved_batch_codes),
|
| 425 |
}
|
| 426 |
+
if coverage is not None:
|
| 427 |
+
document['schema_version'] = 5
|
| 428 |
+
document.setdefault('locations', None)
|
| 429 |
+
document['coverage'] = {'records': sorted([{
|
| 430 |
+
'batch_code': record.batch_code,
|
| 431 |
+
'slices': [{
|
| 432 |
+
'collected_on': item.collected_on.isoformat() if item.collected_on else None,
|
| 433 |
+
'period': item.period, 'codec': item.codec,
|
| 434 |
+
'known_seconds': item.known_seconds,
|
| 435 |
+
'known_videos': item.known_videos, 'unknown_videos': item.unknown_videos,
|
| 436 |
+
} for item in sorted(record.slices, key=lambda item: (
|
| 437 |
+
item.collected_on is None, item.collected_on or date.max, item.period, item.codec))],
|
| 438 |
+
} for record in coverage.records], key=lambda record: record['batch_code'])}
|
| 439 |
_parse_document(document)
|
| 440 |
return document
|
| 441 |
|
| 442 |
|
| 443 |
def encode_snapshot(catalog: VideoCatalog, annotations: AnnotationCatalog, generated_at: datetime,
|
| 444 |
updates: BatchUpdateCatalog | None = None, *, include_frames: bool = False,
|
| 445 |
+
locations: LocationCatalog | None = None, coverage: CoverageCatalog | None = None) -> bytes:
|
| 446 |
"""Encode V1/V2 by default, V3 frames, or V4 frames and location memberships.
|
| 447 |
|
| 448 |
V3 requires safe unique batch codes; a missing frame is null. Its updates
|
| 449 |
key is also explicit and nullable when no update catalog is supplied.
|
| 450 |
"""
|
| 451 |
raw = _canonical_json(_document(catalog, annotations, generated_at, updates,
|
| 452 |
+
include_frames=include_frames, locations=locations, coverage=coverage))
|
| 453 |
if len(raw) > MAX_DECOMPRESSED_BYTES:
|
| 454 |
raise ValueError("Public statistics snapshot is too large")
|
| 455 |
encoded = gzip.compress(raw, mtime=0)
|
|
|
|
| 460 |
|
| 461 |
def semantic_digest(catalog: VideoCatalog, annotations: AnnotationCatalog,
|
| 462 |
updates: BatchUpdateCatalog | None = None, *, include_frames: bool = False,
|
| 463 |
+
locations: LocationCatalog | None = None, coverage: CoverageCatalog | None = None) -> str:
|
| 464 |
"""Hash facts, excluding row order, export time and discovery-only timestamps."""
|
| 465 |
document = _document(catalog, annotations, datetime(2000, 1, 1, tzinfo=timezone.utc), updates,
|
| 466 |
+
include_frames=include_frames, locations=locations, coverage=coverage)
|
| 467 |
del document["generated_at"]
|
| 468 |
del document["catalog"]["latest_updated_at"]
|
| 469 |
del document["annotations"]["latest_updated_at"]
|
|
|
|
| 557 |
or document["source"] != SOURCE
|
| 558 |
or document["semantic_digest"] != semantic_digest(snapshot.catalog, snapshot.annotations, snapshot.updates,
|
| 559 |
include_frames=snapshot.includes_frames,
|
| 560 |
+
locations=snapshot.locations,
|
| 561 |
+
coverage=snapshot.coverage)):
|
| 562 |
return None
|
| 563 |
checked_at = _timestamp(document["checked_at"])
|
| 564 |
return checked_at if checked_at >= snapshot.generated_at else None
|
representative_gallery.py
CHANGED
|
@@ -21,6 +21,7 @@ import streamlit as st
|
|
| 21 |
|
| 22 |
from annotation_statistics import AnnotationRecord, definitions
|
| 23 |
from classic_statistics import selection_label
|
|
|
|
| 24 |
|
| 25 |
_PAGE_SIZE = 12
|
| 26 |
_LOCATION_CODE = re.compile(r'L-[0-9a-f]{16}\Z')
|
|
@@ -201,7 +202,7 @@ def render_representative_gallery(
|
|
| 201 |
|
| 202 |
st.markdown(_STYLE, unsafe_allow_html=True)
|
| 203 |
st.markdown("<div id='nav-examples' class='dash-section'><h2 class='dash-section-title'>"
|
| 204 |
-
+ html.escape(tr('场景示例', 'Scene examples')) + '</h2></div>', unsafe_allow_html=True)
|
| 205 |
st.caption(tr(
|
| 206 |
'按城市浏览自动归组采集点(同城组内最大距离 100 米),每个地点展示一张匹配代表帧;'
|
| 207 |
'这不是经人工确认的路口身份。示例随上方城市、日期、编码、时段及 ODD 筛选更新。使用图片右上角控件可放大。',
|
|
@@ -250,12 +251,12 @@ def render_representative_gallery(
|
|
| 250 |
st.session_state[_CITY_KEY] = default_city
|
| 251 |
st.session_state[_PAGE_KEY] = 1
|
| 252 |
city = st.selectbox(tr('示例城市', 'Example city'), cities, key=_CITY_KEY,
|
| 253 |
-
format_func=
|
| 254 |
city_examples = tuple(example for example in examples if example.city == city)
|
| 255 |
_, selected_page, pages = gallery_page(city_examples, st.session_state[_PAGE_KEY])
|
| 256 |
st.session_state[_PAGE_KEY] = selected_page
|
| 257 |
st.markdown("<div class='example-city-heading'>"
|
| 258 |
-
+ html.escape(tr(f'{
|
| 259 |
+ '</div>', unsafe_allow_html=True)
|
| 260 |
st.selectbox(tr('示例页码', 'Example page'), list(range(1, pages + 1)), key=_PAGE_KEY,
|
| 261 |
format_func=lambda page: tr(f'第 {page} / {pages} 页', f'Page {page} / {pages}'),
|
|
@@ -288,7 +289,7 @@ def render_representative_gallery(
|
|
| 288 |
weather = label_value(labels.get('environment.weather'))
|
| 289 |
st.markdown(
|
| 290 |
"<div class='example-meta'>"
|
| 291 |
-
f"<div class='example-city'>{html.escape(
|
| 292 |
f"<div class='example-code'>{html.escape(example.code)}</div>"
|
| 293 |
f"<div class='example-code'>{html.escape(tr('代表批次', 'Representative batch'))}: {html.escape(code)}</div>"
|
| 294 |
f"<div class='example-date'>{html.escape(tr('匹配 / 全部批次', 'Matching / total batches'))}: {example.matching_batches:,} / {example.total_batches:,}</div>"
|
|
|
|
| 21 |
|
| 22 |
from annotation_statistics import AnnotationRecord, definitions
|
| 23 |
from classic_statistics import selection_label
|
| 24 |
+
from city_labels import city_display_name
|
| 25 |
|
| 26 |
_PAGE_SIZE = 12
|
| 27 |
_LOCATION_CODE = re.compile(r'L-[0-9a-f]{16}\Z')
|
|
|
|
| 202 |
|
| 203 |
st.markdown(_STYLE, unsafe_allow_html=True)
|
| 204 |
st.markdown("<div id='nav-examples' class='dash-section'><h2 class='dash-section-title'>"
|
| 205 |
+
+ html.escape(tr('筛选结果 · 场景示例', 'Filter results · Scene examples')) + '</h2></div>', unsafe_allow_html=True)
|
| 206 |
st.caption(tr(
|
| 207 |
'按城市浏览自动归组采集点(同城组内最大距离 100 米),每个地点展示一张匹配代表帧;'
|
| 208 |
'这不是经人工确认的路口身份。示例随上方城市、日期、编码、时段及 ODD 筛选更新。使用图片右上角控件可放大。',
|
|
|
|
| 251 |
st.session_state[_CITY_KEY] = default_city
|
| 252 |
st.session_state[_PAGE_KEY] = 1
|
| 253 |
city = st.selectbox(tr('示例城市', 'Example city'), cities, key=_CITY_KEY,
|
| 254 |
+
format_func=city_display_name, on_change=_reset_gallery_page)
|
| 255 |
city_examples = tuple(example for example in examples if example.city == city)
|
| 256 |
_, selected_page, pages = gallery_page(city_examples, st.session_state[_PAGE_KEY])
|
| 257 |
st.session_state[_PAGE_KEY] = selected_page
|
| 258 |
st.markdown("<div class='example-city-heading'>"
|
| 259 |
+
+ html.escape(tr(f'{city_display_name(city)} · 采集点示例', f'{city_display_name(city)} · Collection-point examples'))
|
| 260 |
+ '</div>', unsafe_allow_html=True)
|
| 261 |
st.selectbox(tr('示例页码', 'Example page'), list(range(1, pages + 1)), key=_PAGE_KEY,
|
| 262 |
format_func=lambda page: tr(f'第 {page} / {pages} 页', f'Page {page} / {pages}'),
|
|
|
|
| 289 |
weather = label_value(labels.get('environment.weather'))
|
| 290 |
st.markdown(
|
| 291 |
"<div class='example-meta'>"
|
| 292 |
+
f"<div class='example-city'>{html.escape(city_display_name(record.city))}</div>"
|
| 293 |
f"<div class='example-code'>{html.escape(example.code)}</div>"
|
| 294 |
f"<div class='example-code'>{html.escape(tr('代表批次', 'Representative batch'))}: {html.escape(code)}</div>"
|
| 295 |
f"<div class='example-date'>{html.escape(tr('匹配 / 全部批次', 'Matching / total batches'))}: {example.matching_batches:,} / {example.total_batches:,}</div>"
|