Add files using upload-large-folder tool
Browse files- .gitignore +27 -0
- HANDOVER.md +229 -0
- README.md +143 -0
- inference/attractive_leads_prompt.py +181 -0
- inference/eval.py +0 -0
- inference/fetch_youtube_online.py +253 -0
- inference/run_people_quality_inference.py +723 -0
- inference/video_dr_bridge.py +231 -0
- requirements.txt +204 -0
- run_people_online_tavily.sh +180 -0
- video_dr_gen/assemble_sft_dataset.py +197 -0
- video_dr_gen/config.py +334 -0
- video_dr_gen/prompts.py +541 -0
- video_dr_gen/utils.py +0 -0
.gitignore
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
.DS_Store
|
| 2 |
+
|
| 3 |
+
# Python caches
|
| 4 |
+
__pycache__/
|
| 5 |
+
*.py[cod]
|
| 6 |
+
.pytest_cache/
|
| 7 |
+
.mypy_cache/
|
| 8 |
+
.ruff_cache/
|
| 9 |
+
.ipynb_checkpoints/
|
| 10 |
+
|
| 11 |
+
# Local environments and machine-specific config
|
| 12 |
+
.env
|
| 13 |
+
.venv/
|
| 14 |
+
venv/
|
| 15 |
+
env/
|
| 16 |
+
|
| 17 |
+
# Secrets
|
| 18 |
+
secrets/
|
| 19 |
+
|
| 20 |
+
# Generated downloads, inference outputs, and local data
|
| 21 |
+
lead_filter_runs/
|
| 22 |
+
local_data/
|
| 23 |
+
inference/online_runs/
|
| 24 |
+
*.log
|
| 25 |
+
|
| 26 |
+
# Playwright browser binaries are installed per machine.
|
| 27 |
+
.playwright-browsers/
|
HANDOVER.md
ADDED
|
@@ -0,0 +1,229 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# 交接文档：Video Deep Research 视频优质度判别 Demo
|
| 2 |
+
|
| 3 |
+
> 负责人：高振坤（zhenkun / zhenkungao）
|
| 4 |
+
> 文档用途：说明本 demo 的功能、运行环境、运行方式、配置与密钥、排错与待办，便于他人接手 / 迁移到新服务器 / 后续网页化。
|
| 5 |
+
> 代码当前位置：`/home/tione/notebook/gaozhenkun/demo/video_deep_research_demo/`
|
| 6 |
+
> 本目录已**自包含全部源码**，可整目录拷贝/建 git 后迁移。
|
| 7 |
+
|
| 8 |
+
---
|
| 9 |
+
|
| 10 |
+
## 一、这个 Demo 做什么
|
| 11 |
+
|
| 12 |
+
给定一个 **YouTube 视频链接**，自动判别该视频是否为「优质素材」（当前业务口径：是否含可抽取的**高颜值男女（帅哥/美女）素材**），输出结构化判定（`approve` / `reject` / `undecided`）+ 推理轨迹。
|
| 13 |
+
|
| 14 |
+
核心方法是 **Video Deep Research**：让一个 VLM（视频多模态大模型）以「多轮工具调用」的方式自己去研究视频——抽帧、锁定关键帧、放大、反向图搜、网页文搜，最后给出判定。
|
| 15 |
+
|
| 16 |
+
判别逻辑**不是**简单分类，而是：模型在每一轮自行决定调用 5 个工具中的哪一个，最终在 `<answer>` 里输出 JSON 判定，由脚本解析（`parse_answer_json`），**本链路没有独立的 LLM-as-judge 打分环节**（judge/summarizer 只在 `image_search` 摘要里用到）。
|
| 17 |
+
|
| 18 |
+
### 5 个工具（Video Deep Research 工具集）
|
| 19 |
+
|
| 20 |
+
| 工具 | 作用 |
|
| 21 |
+
|------|------|
|
| 22 |
+
| `choose_frames` | 选定一个时间/帧区间，返回均匀采样的若干帧 |
|
| 23 |
+
| `find_frame` | 锁定单个具体帧，细看 |
|
| 24 |
+
| `zoom_in` | 放大锁定帧的某个 bbox 区域 |
|
| 25 |
+
| `image_search` | 对锁定帧的某个区域做反向图搜（识别未知实体） |
|
| 26 |
+
| `web_search` | 文本网页搜索（已知实体，查外部知识）。本 demo 走 **Tavily** 公网搜索 |
|
| 27 |
+
|
| 28 |
+
> 工具定义在 `inference/attractive_leads_prompt.py` 的 `_TOOLS_BLOCK`，与训练侧 `config/tool_config/tools_video_train.yaml` 的 5 个工具一一对应（名称/参数/required 一致）。bbox 文案一处是 0–1000、一处是 0.0–1.0，运行时自动识别坐标制式，无影响。
|
| 29 |
+
|
| 30 |
+
---
|
| 31 |
+
|
| 32 |
+
## 二、整体流程（脚本 `run_people_online_tavily.sh` 做了什么）
|
| 33 |
+
|
| 34 |
+
```
|
| 35 |
+
YouTube URL
|
| 36 |
+
│
|
| 37 |
+
├─[第1步 抓取] inference/fetch_youtube_online.py
|
| 38 |
+
│ yt-dlp 下载 mp4 + info.json(含评论) → 规范化 comments.json / metadata.json
|
| 39 |
+
│ 以 ---FETCH--- KEY=VALUE ---END--- 块把 VIDEO_ID/VIDEO_PATH/COMMENTS_FILE/... 回传 shell
|
| 40 |
+
│
|
| 41 |
+
├─[第2步 推理] inference/run_people_quality_inference.py (单条模式)
|
| 42 |
+
│ 调用 eval.py::evaluate_video_tool() 跑「5 工具多轮」推理
|
| 43 |
+
│ ├─ 每轮请求外部 VLM 服务 (MODEL_BASE_URL, OpenAI 兼容)
|
| 44 |
+
│ ├─ web_search → Tavily (公网)
|
| 45 |
+
│ └─ image_search → 内网 gateway/Serper + summarizer (可选)
|
| 46 |
+
│
|
| 47 |
+
└─ 产出：判定 JSON + HTML 轨迹 + 决策表 CSV + 帧图
|
| 48 |
+
```
|
| 49 |
+
|
| 50 |
+
- **demo 本机不需要 GPU**：它是「客户端」，模型在外部 `MODEL_BASE_URL` 上已部署。
|
| 51 |
+
- 两步之间用「临时文件 + KEY=VALUE 文本块」中转，不是函数直调（后续网页化时这里要改成 Python API）。
|
| 52 |
+
|
| 53 |
+
---
|
| 54 |
+
|
| 55 |
+
## 三、运行环境
|
| 56 |
+
|
| 57 |
+
### 1. 机器 / 硬件
|
| 58 |
+
- 无需 GPU（模型在远端服务上）。普通 CPU 机器即可。
|
| 59 |
+
- 需能访问：① 外部 VLM 推理服务；② 公网（yt-dlp 拉 YouTube、Tavily 搜索）；③（可选）内网图搜 gateway / MARS summarizer。
|
| 60 |
+
|
| 61 |
+
### 2. 系统依赖
|
| 62 |
+
- **ffmpeg**（必需，yt-dlp 合并 mp4 用，要在 PATH）。
|
| 63 |
+
- **yt-dlp**（pip 或系统包；脚本会在 `CONDA_ENV/bin`、`CONDA_BASE/bin`、PATH 里自动探测，也可用 `YT_DLP` 指定）。
|
| 64 |
+
- **playwright + chromium**：`eval.py` 顶层 `import playwright`，必须装浏览器内核，否则 import 直接失败。
|
| 65 |
+
|
| 66 |
+
### 3. Python 环境
|
| 67 |
+
当前本地环境：conda env `/home/tione/notebook/gaozhenkun/envs/sglang`。
|
| 68 |
+
|
| 69 |
+
新机器自建：
|
| 70 |
+
```bash
|
| 71 |
+
/home/tione/notebook/gaozhenkun/envs/sglang/bin/python -m pip install -r requirements.txt
|
| 72 |
+
PLAYWRIGHT_BROWSERS_PATH=$PWD/.playwright-browsers \
|
| 73 |
+
/home/tione/notebook/gaozhenkun/envs/sglang/bin/playwright install chromium # 必做
|
| 74 |
+
```
|
| 75 |
+
`requirements.txt` 关键依赖（版本锁自 sglang9，可放宽）：
|
| 76 |
+
`aiohttp / openai / pillow / playwright / requests / beautifulsoup4 / PyYAML / numpy / yt-dlp`。
|
| 77 |
+
`google-*`（Vertex/Gemini）仅 `--model-client vertex` 时需要，默认不用装。
|
| 78 |
+
|
| 79 |
+
### 4. 网络 / 外部依赖服务清单
|
| 80 |
+
| 依赖 | 用途 | 走代理? | 在哪配 |
|
| 81 |
+
|------|------|---------|--------|
|
| 82 |
+
| VLM 推理服务 `MODEL_BASE_URL` | 多轮推理主模型 | 否（内网直连，加入 no_proxy） | 脚本 `MODEL_BASE_URL`，默认 `http://100.102.37.219:8008` |
|
| 83 |
+
| Tavily | `web_search` 公网搜索 | 是 | `TAVILY_API_KEY_FILE` / `TAVILY_API_KEY` |
|
| 84 |
+
| yt-dlp → YouTube | 下载视频/评论 | 是 | `HTTP_PROXY_URL` |
|
| 85 |
+
| 内网 gateway 图搜 | `image_search`（可选） | 否（no_proxy） | `GATEWAY_URL` / `GATEWAY_TOKEN` 等环境变量 |
|
| 86 |
+
| MARS summarizer | 图搜结果摘要（可选） | 否（no_proxy） | `MARS_SUMMARIZER_ADDRESS` / `MARS_SUMMARIZER_MODEL` |
|
| 87 |
+
|
| 88 |
+
代理机制：脚本默认 `HTTP_PROXY_URL=http://127.0.0.1:17890`，公网流量走代理，内网服务进 `no_proxy` 直连。**新机器若直连公网，设 `HTTP_PROXY_URL=""` 关掉代理。**
|
| 89 |
+
|
| 90 |
+
---
|
| 91 |
+
|
| 92 |
+
## 四、如何运行
|
| 93 |
+
|
| 94 |
+
### 快速开始（当前本地环境）
|
| 95 |
+
```bash
|
| 96 |
+
cd /home/tione/notebook/gaozhenkun/demo/video_deep_research_demo
|
| 97 |
+
./run_people_online_tavily.sh https://www.youtube.com/shorts/ZHI5IkRfXAM
|
| 98 |
+
```
|
| 99 |
+
|
| 100 |
+
### 常用调用方式
|
| 101 |
+
```bash
|
| 102 |
+
# 用环境变量传 URL
|
| 103 |
+
VIDEO_URL=https://www.youtube.com/watch?v=XXXX ./run_people_online_tavily.sh
|
| 104 |
+
|
| 105 |
+
# 限制抓取评论数
|
| 106 |
+
MAX_COMMENTS=50 ./run_people_online_tavily.sh <url>
|
| 107 |
+
|
| 108 |
+
# `--` 之后的参数透传给推理脚本，例如改温度
|
| 109 |
+
./run_people_online_tavily.sh <url> --temperature 0.6
|
| 110 |
+
```
|
| 111 |
+
|
| 112 |
+
### 关键环境变量（迁移时覆盖默认值）
|
| 113 |
+
| 变量 | 作用 | 默认值 | 迁移必改 |
|
| 114 |
+
|------|------|--------|:--------:|
|
| 115 |
+
| `CONDA_ENV` | 要激活的 conda 环境路径/名 | `/home/tione/notebook/gaozhenkun/envs/sglang` | 视情况 |
|
| 116 |
+
| `CONDA_BASE` | conda 安装根（`conda shell.bash hook`） | `/root/miniconda3` | 视情况 |
|
| 117 |
+
| `MODEL_BASE_URL` | VLM 服务（OpenAI 兼容 `/v1`） | `http://100.102.37.219:8008` | ✅ |
|
| 118 |
+
| `MODEL_NAME` | 模型名 | `qwen3-vl-video-dr` | 视情况 |
|
| 119 |
+
| `TAVILY_API_KEY_FILE` | Tavily key 文件（每行一个 key） | `demo/secrets/tavily_api_keys.txt` | 视情况 |
|
| 120 |
+
| `HTTP_PROXY_URL` | 公网代理；直连设 `""` | `http://127.0.0.1:17890` | ✅ |
|
| 121 |
+
| `YT_DLP` | yt-dlp 路径 | 自动探测 | 视情况 |
|
| 122 |
+
| `MAX_COMMENTS` / `MAX_TURNS` / `MAX_CONCURRENT` | 评论数 / 最大轮数 / 并发 | 100 / 10 / 1 | 可选 |
|
| 123 |
+
| `WORK_DIR` | 复用已下载产物，避免重下 | 每次新建 | 可选 |
|
| 124 |
+
| `YT_COOKIES` / `YT_COOKIES_FROM_BROWSER` | 给 yt-dlp 传 cookie（应对 YouTube 风控） | 无 | 视情况 |
|
| 125 |
+
|
| 126 |
+
### 迁移示例（新机直连公网、自建 conda）
|
| 127 |
+
```bash
|
| 128 |
+
export CONDA_ENV=/home/tione/notebook/gaozhenkun/envs/sglang
|
| 129 |
+
export CONDA_BASE=/root/miniconda3
|
| 130 |
+
export MODEL_BASE_URL=http://<your-vlm-host>:8000
|
| 131 |
+
export MODEL_NAME=<your-model-name>
|
| 132 |
+
export TAVILY_API_KEY=tvly-xxxxxxxx
|
| 133 |
+
export HTTP_PROXY_URL="" # 直连，不走代理
|
| 134 |
+
./run_people_online_tavily.sh <youtube_url>
|
| 135 |
+
```
|
| 136 |
+
|
| 137 |
+
---
|
| 138 |
+
|
| 139 |
+
## 五、产物
|
| 140 |
+
|
| 141 |
+
- **抓取缓存** `<work_dir>/`（默认 `inference/online_runs/<id>_<ts>/`）：
|
| 142 |
+
`<id>.mp4`、`<id>.info.json`、`comments.json`、`metadata.json`。
|
| 143 |
+
- **推理结果** `<output_dir>/`（默认 `demo/lead_filter_runs/online_<id>_<ts>/`）：
|
| 144 |
+
- `results.jsonl` — 结构化结果（含每轮工具调用）
|
| 145 |
+
- `results.html` — **可视化推理轨迹**（逐轮工具/帧图，演示首选）
|
| 146 |
+
- `summary.json`、`decision_table.csv` — 判定汇总
|
| 147 |
+
- `images/`、`frame_cache/` — 抽帧与裁剪图
|
| 148 |
+
- **日志** `<output_dir>.log`。
|
| 149 |
+
|
| 150 |
+
---
|
| 151 |
+
|
| 152 |
+
## 六、配置与密钥 ⚠️（迁移 / 建 git 必读）
|
| 153 |
+
|
| 154 |
+
`video_dr_gen/config.py` 已改为**从环境变量读取凭证和内网地址**，默认不再写死旧机器路径或明文 secret：
|
| 155 |
+
|
| 156 |
+
| 位置 | 内容 |
|
| 157 |
+
|------|------|
|
| 158 |
+
| `config.py` | `SERPER_API_KEY`、`GATEWAY_URL/USERNAME/USERID/GATEWAY_TOKEN`、`MARS_RETRIEVAL_ADDRESS`、`MARS_SUMMARIZER_ADDRESS`、OSS 配置均读环境变量 |
|
| 159 |
+
| `run_people_online_tavily.sh` | `TAVILY_API_KEY_FILE` 默认指向 `demo/secrets/tavily_api_keys.txt`；`MODEL_BASE_URL` 仍需按实际模型服务配置 |
|
| 160 |
+
|
| 161 |
+
**因此：**
|
| 162 |
+
|
| 163 |
+
1. 不要把真实 `GATEWAY_TOKEN`、`SERPER_API_KEY`、Tavily key、OSS key 写回源码或提交。
|
| 164 |
+
2. `MARS_SUMMARIZER_ADDRESS` 默认空，自检会跳过；只有显式配置后才会探测。
|
| 165 |
+
3. 若需要 `image_search`，需要额外配置 gateway/Serper 与 OSS 上传相关环境变量，否则该工具会返回配置缺失错误。
|
| 166 |
+
|
| 167 |
+
---
|
| 168 |
+
|
| 169 |
+
## 七、运行前自检
|
| 170 |
+
```bash
|
| 171 |
+
# 1) 模型服务可达？
|
| 172 |
+
curl "$MODEL_BASE_URL/v1/models"
|
| 173 |
+
|
| 174 |
+
# 2) 源码闭包可导入？应输出 IMPORT_OK
|
| 175 |
+
cd demo/inference && python -c "import fetch_youtube_online, run_people_quality_inference; print('IMPORT_OK')"
|
| 176 |
+
|
| 177 |
+
# 3) ffmpeg / yt-dlp / chromium 就绪？
|
| 178 |
+
ffmpeg -version | head -1; yt-dlp --version; python -c "from playwright.sync_api import sync_playwright; print('pw ok')"
|
| 179 |
+
```
|
| 180 |
+
|
| 181 |
+
---
|
| 182 |
+
|
| 183 |
+
## 八、常见问题排错
|
| 184 |
+
|
| 185 |
+
| 现象 | 原因 / 处理 |
|
| 186 |
+
|------|------|
|
| 187 |
+
| import 阶段报 playwright 相关错 | 没装内核：`playwright install chromium` |
|
| 188 |
+
| yt-dlp 下载失败 / 被风控 / 403 | YouTube 对云 IP 风控。配 `HTTP_PROXY_URL`，或用 `YT_COOKIES` / `YT_COOKIES_FROM_BROWSER` 传 cookie；必要时降频 |
|
| 189 |
+
| 启动卡在自检 / judge 连不上 | `config.py` 的 `MARS_SUMMARIZER_ADDRESS` 内网不可达，按第六节设为 `""` |
|
| 190 |
+
| `web_search` 报缺 key | 提供 `TAVILY_API_KEY` / `TAVILY_API_KEYS` / `TAVILY_API_KEY_FILE` 三选一 |
|
| 191 |
+
| 模型请求超时 | 检查 `MODEL_BASE_URL` 可达、模型名 `MODEL_NAME` 是否匹配；`curl $MODEL_BASE_URL/v1/models` |
|
| 192 |
+
| 内网服务走了代理连不上 | 确认该地��在脚本 `no_proxy` 列表里（默认已含模型 host 与 MARS 地址） |
|
| 193 |
+
| ffmpeg 报错 / mp4 合并失败 | 装 ffmpeg 并加入 PATH |
|
| 194 |
+
|
| 195 |
+
---
|
| 196 |
+
|
| 197 |
+
## 九、目录结构速查
|
| 198 |
+
```
|
| 199 |
+
demo/
|
| 200 |
+
├── run_people_online_tavily.sh # 入口脚本（REPO_ROOT 自动取本目录）
|
| 201 |
+
├── requirements.txt # Python 依赖（锁自 sglang9）
|
| 202 |
+
├── README.md # 简版说明
|
| 203 |
+
├── HANDOVER.md # 本交接文档
|
| 204 |
+
├── inference/
|
| 205 |
+
│ ├── fetch_youtube_online.py # 第1步：yt-dlp 抓视频/评论
|
| 206 |
+
│ ├── run_people_quality_inference.py# 第2步：单条筛选推理入口（含 health_check）
|
| 207 |
+
│ ├── attractive_leads_prompt.py # ★工具定义 _TOOLS_BLOCK + 业务 system prompt
|
| 208 |
+
│ ├── eval.py # 核心引擎：5工具多轮 + Tavily/图搜 + HTML 轨迹
|
| 209 |
+
│ └── video_dr_bridge.py # 桥接层：自动定位 video_dr_gen（VIDEO_DR_SOURCE_ROOT 可覆盖）
|
| 210 |
+
└── video_dr_gen/ # 被 bridge 依赖的闭包（4 文件）
|
| 211 |
+
├── config.py # 本地默认配置，secret/内网地址从环境变量读取
|
| 212 |
+
├── utils.py # normalize_bbox / crop_frame / real_image_search ...
|
| 213 |
+
├── prompts.py
|
| 214 |
+
└── assemble_sft_dataset.py
|
| 215 |
+
```
|
| 216 |
+
> `inference/` 与 `video_dr_gen/` 必须保持现在的相对位置（同在 `demo/` 下）。
|
| 217 |
+
|
| 218 |
+
---
|
| 219 |
+
|
| 220 |
+
## 十、已知问题 / 后续待办
|
| 221 |
+
|
| 222 |
+
1. **网页化（GitHub / HuggingFace Space）需新增**：
|
| 223 |
+
- Web 层（Gradio/FastAPI）包住现有两步 CLI；
|
| 224 |
+
- 把 shell+文件中转改成可调用、可流式吐进度的 Python API（推理耗时长，需进度条）；
|
| 225 |
+
- 系统依赖打包（ffmpeg / playwright chromium，Dockerfile 或 HF `packages.txt`）；
|
| 226 |
+
- 增加「本地视频上传」备选输入，规避云端拉 YouTube 被风控；
|
| 227 |
+
- 并发队列 / 超时 / 临时目录清理 / 限流。
|
| 228 |
+
|
| 229 |
+
---
|
README.md
ADDED
|
@@ -0,0 +1,143 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# 在线人物筛选 Demo（YouTube + Tavily 文搜）
|
| 2 |
+
|
| 3 |
+
给定一个 YouTube 视频链接，自动：
|
| 4 |
+
|
| 5 |
+
1. 用 `yt-dlp` 抓取视频 + 顶层评论 + 元数据；
|
| 6 |
+
2. 跑“高颜值男女可抽取素材”筛选推理（多轮工具调用，`web_search` 走 Tavily 公网搜索）；
|
| 7 |
+
3. 产出结构化判定（approve / reject / undecided）、HTML 轨迹、决策表 CSV 等。
|
| 8 |
+
|
| 9 |
+
本目录已**自包含全部源码**，可整目录拷贝到其他服务器部署。
|
| 10 |
+
|
| 11 |
+
---
|
| 12 |
+
|
| 13 |
+
## 一、目录结构
|
| 14 |
+
|
| 15 |
+
```
|
| 16 |
+
demo/
|
| 17 |
+
├── run_people_online_tavily.sh # 入口脚本（可迁移版，REPO_ROOT 自动取本目录）
|
| 18 |
+
├── requirements.txt # Python 依赖（已清理为本地可安装格式）
|
| 19 |
+
├── README.md # 本文件
|
| 20 |
+
├── inference/
|
| 21 |
+
│ ├── fetch_youtube_online.py # 第 1 步：yt-dlp 抓视频/评论 -> work_dir
|
| 22 |
+
│ ├── run_people_quality_inference.py # 第 2 步：单条筛选推理入口
|
| 23 |
+
│ ├── attractive_leads_prompt.py # system / user prompt 构造
|
| 24 |
+
│ ├── eval.py # 核心：五工具多轮推理 + Tavily/图搜 + HTML
|
| 25 |
+
│ └── video_dr_bridge.py # 桥接层：复用 video_dr_gen 的帧处理/工具逻辑
|
| 26 |
+
└── video_dr_gen/ # VideoDR SFT 构造代码（帧抽取、bbox、图搜/web搜实现）
|
| 27 |
+
├── assemble_sft_dataset.py # NEW_SYSTEM_PROMPT 等
|
| 28 |
+
├── config.py # MARS / gateway / 默认帧参数等常量
|
| 29 |
+
├── prompts.py # PHASE1/PHASE2/BASE system prompts
|
| 30 |
+
└── utils.py # normalize_bbox / crop_frame / image_search / mars_web_search ...
|
| 31 |
+
```
|
| 32 |
+
|
| 33 |
+
> **重要**：`inference/` 与 `video_dr_gen/` 必须保持现在的相对位置（同在 `demo/` 下）。
|
| 34 |
+
> `video_dr_bridge.py` 按 `本文件父目录的父目录 / video_dr_gen` 定位依赖；也可用环境变量
|
| 35 |
+
> `VIDEO_DR_SOURCE_ROOT` 显式指定其它路径。
|
| 36 |
+
>
|
| 37 |
+
> 本 demo 已把用到的 4 个 `video_dr_gen` 文件打包进来并改成相对/可配置定位，
|
| 38 |
+
> 迁移后无需依赖旧机器共享盘路径。
|
| 39 |
+
|
| 40 |
+
---
|
| 41 |
+
|
| 42 |
+
## 二、运行流程（脚本做了什么）
|
| 43 |
+
|
| 44 |
+
`run_people_online_tavily.sh <youtube_url>`：
|
| 45 |
+
|
| 46 |
+
1. 设置出网代理 + `no_proxy`（内网模型直连、公网走代理），激活 conda 环境。
|
| 47 |
+
2. **抓取**：调用 `inference/fetch_youtube_online.py`
|
| 48 |
+
- `yt-dlp` 下载 mp4 + `info.json`（含评论），规范化后写出
|
| 49 |
+
`comments.json` / `metadata.json`；
|
| 50 |
+
- 以 `---FETCH--- KEY=VALUE ---END---` 块把 `VIDEO_ID/VIDEO_PATH/COMMENTS_FILE/
|
| 51 |
+
TITLE/CHANNEL/DURATION/VIEW_COUNT/VIDEO_URL` 回传给 shell。
|
| 52 |
+
3. **推理**：shell 解析上述 KEY=VALUE，拼成单条参数喂给
|
| 53 |
+
`inference/run_people_quality_inference.py`（`--web-search-backend tavily` + 一组
|
| 54 |
+
`--tavily-*`），复用 `eval.py::evaluate_video_tool()` 的五工具多轮推理逻辑。
|
| 55 |
+
4. 标准输出/错误 `tee` 到日志文件。
|
| 56 |
+
|
| 57 |
+
### 产物
|
| 58 |
+
- 抓取缓存：`<work_dir>/<id>.mp4`、`<id>.info.json`、`comments.json`、`metadata.json`
|
| 59 |
+
（默认 `inference/online_runs/<id>_<ts>/`，可用 `WORK_DIR` 复用避免重下）。
|
| 60 |
+
- 推理结果（默认 `demo/lead_filter_runs/online_<id>_<ts>/`）：
|
| 61 |
+
`results.jsonl`、`results.html`、`summary.json`、`decision_table.csv`、
|
| 62 |
+
`images/`、`frame_cache/`。
|
| 63 |
+
- 日志：`<output_dir>.log`。
|
| 64 |
+
|
| 65 |
+
---
|
| 66 |
+
|
| 67 |
+
## 三、用法
|
| 68 |
+
|
| 69 |
+
```bash
|
| 70 |
+
cd demo
|
| 71 |
+
./run_people_online_tavily.sh https://www.youtube.com/shorts/ZHI5IkRfXAM
|
| 72 |
+
# 或
|
| 73 |
+
VIDEO_URL=https://www.youtube.com/watch?v=XXXX ./run_people_online_tavily.sh
|
| 74 |
+
# 限制评论数
|
| 75 |
+
MAX_COMMENTS=50 ./run_people_online_tavily.sh <url>
|
| 76 |
+
```
|
| 77 |
+
|
| 78 |
+
`--` 之后透传给推理脚本，例如调温度：
|
| 79 |
+
```bash
|
| 80 |
+
./run_people_online_tavily.sh <url> --temperature 0.6
|
| 81 |
+
```
|
| 82 |
+
|
| 83 |
+
---
|
| 84 |
+
|
| 85 |
+
## 四、部署到新服务器
|
| 86 |
+
|
| 87 |
+
### 1. 系统依赖
|
| 88 |
+
- `ffmpeg`（yt-dlp 合并 mp4 必需，需在 PATH）。
|
| 89 |
+
- `yt-dlp`（pip 安装或系统包均可）。
|
| 90 |
+
|
| 91 |
+
### 2. Python 依赖
|
| 92 |
+
```bash
|
| 93 |
+
/home/tione/notebook/gaozhenkun/envs/sglang/bin/python -m pip install -r requirements.txt
|
| 94 |
+
# eval.py 顶层 import playwright，需要装浏览器内核：
|
| 95 |
+
PLAYWRIGHT_BROWSERS_PATH=$PWD/.playwright-browsers \
|
| 96 |
+
/home/tione/notebook/gaozhenkun/envs/sglang/bin/playwright install chromium
|
| 97 |
+
```
|
| 98 |
+
> 说明：当前默认使用本机 `/home/tione/notebook/gaozhenkun/envs/sglang` 环境。
|
| 99 |
+
> `google-*`（Vertex/Gemini）仅在 `--model-client vertex` 时才需要，默认不装也能跑。
|
| 100 |
+
|
| 101 |
+
### 3. 必填环境变量 / 配置
|
| 102 |
+
| 变量 | 作用 | 默认值 | 迁移时是否要改 |
|
| 103 |
+
|------|------|--------|----------------|
|
| 104 |
+
| `CONDA_ENV` | 要激活的 conda 环境路径/名 | `/home/tione/notebook/gaozhenkun/envs/sglang` | 视情况 |
|
| 105 |
+
| `CONDA_BASE` | conda 安装根（取 `conda shell.bash hook`） | `/root/miniconda3` | 视情况 |
|
| 106 |
+
| `MODEL_BASE_URL` | VLM 推理服务（OpenAI 兼容 `/v1`） | `http://100.102.37.219:8008` | **是** |
|
| 107 |
+
| `MODEL_NAME` | 模型名 | `qwen3-vl-video-dr` | 视情况 |
|
| 108 |
+
| `TAVILY_API_KEY_FILE` | Tavily key 列表文件（每行一个） | `demo/secrets/tavily_api_keys.txt` | 视情况 |
|
| 109 |
+
| `HTTP_PROXY_URL` | 公网代理（yt-dlp/Tavily 出网）；直连可设为空 `""` | `http://127.0.0.1:17890` | **是** |
|
| 110 |
+
| `YT_DLP` | yt-dlp 可执行路径 | 自动探测 | 视情况 |
|
| 111 |
+
| `VIDEO_DR_SOURCE_ROOT` | VideoDR 依赖目录 | 默认用打包的 `demo/video_dr_gen` | 一般不用改 |
|
| 112 |
+
| `VIDEO_DEEP_RESEARCH_DATA_ROOT` | 本地批量数据/评测数据根目录 | `demo/local_data` | 可选 |
|
| 113 |
+
| `VIDEO_DEEP_RESEARCH_SECRETS_DIR` | 本地 secret 文件目录 | `demo/secrets` | 可选 |
|
| 114 |
+
| `PLAYWRIGHT_BROWSERS_PATH` | Playwright 浏览器内核目录 | `demo/.playwright-browsers` | 可选 |
|
| 115 |
+
|
| 116 |
+
Tavily key 也可不用文件，直接 `export TAVILY_API_KEY=...` 或 `TAVILY_API_KEYS=k1,k2`。
|
| 117 |
+
|
| 118 |
+
示例（新机直连公网）：
|
| 119 |
+
```bash
|
| 120 |
+
export CONDA_ENV=/home/tione/notebook/gaozhenkun/envs/sglang
|
| 121 |
+
export CONDA_BASE=/root/miniconda3
|
| 122 |
+
export MODEL_BASE_URL=http://<your-vlm-host>:8000
|
| 123 |
+
export TAVILY_API_KEY=tvly-xxxxxxxx
|
| 124 |
+
export HTTP_PROXY_URL="" # 直连，不走代理
|
| 125 |
+
./run_people_online_tavily.sh <youtube_url>
|
| 126 |
+
```
|
| 127 |
+
|
| 128 |
+
### 4. 可选服务（默认不写死旧机器路径）
|
| 129 |
+
`web_search` 在本 demo 中走 Tavily。`MARS_RETRIEVAL_ADDRESS` / `MARS_SUMMARIZER_ADDRESS`
|
| 130 |
+
默认空字符串，只有显式 `export` 后才会探测内网 MARS/summarizer。
|
| 131 |
+
|
| 132 |
+
`image_search` 默认读取 `GATEWAY_URL` / `GATEWAY_TOKEN` / `SERPER_API_KEY` /
|
| 133 |
+
`OSS_ACCESS_KEY_ID` 等环境变量；未配置时工具会返回配置缺失错误，但不会依赖旧机器明文路径。
|
| 134 |
+
|
| 135 |
+
---
|
| 136 |
+
|
| 137 |
+
## 五、运行前自检
|
| 138 |
+
```bash
|
| 139 |
+
# 模型服务可达？
|
| 140 |
+
curl "$MODEL_BASE_URL/v1/models"
|
| 141 |
+
# 源码闭包可导入？（应输出 IMPORT_OK）
|
| 142 |
+
cd demo/inference && python -c "import fetch_youtube_online, run_people_quality_inference; print('IMPORT_OK')"
|
| 143 |
+
```
|
inference/attractive_leads_prompt.py
ADDED
|
@@ -0,0 +1,181 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
高颜值男女素材筛选 Prompt 模块。
|
| 3 |
+
|
| 4 |
+
输入:
|
| 5 |
+
- 单条视频元数据，至少包含视频链接、标题、频道、时长、播放量中的一部分。
|
| 6 |
+
- 可选评论列表；每条评论可带作者、点赞数、时间与正文。
|
| 7 |
+
|
| 8 |
+
处理:
|
| 9 |
+
- 提供一份**专门针对“识别帅哥美女并判断可抽取性”**的 system prompt（仿照 VideoDR 的
|
| 10 |
+
Role / Video Context / Tools / Workflow / Output Format / Stop Rules 六段式结构），
|
| 11 |
+
让模型通过 `choose_frames / find_frame / zoom_in / image_search / web_search`
|
| 12 |
+
五工具做多轮视频研究。
|
| 13 |
+
- 业务目标聚焦：扫描帧 → 锁定最清晰的男/女候选 → zoom_in 评估颜值与清晰度 →
|
| 14 |
+
image_search 认脸 → web_search 核身份与知名度 → 逐性别判定可抽取性 → 给出结论。
|
| 15 |
+
- 输出对齐 `run_people_quality_inference.parse_answer_json` 期望的富 JSON
|
| 16 |
+
（decision / male_candidate / female_candidate / ...），让结构化字段不再为空。
|
| 17 |
+
|
| 18 |
+
输出:
|
| 19 |
+
- 导出 `ATTRACTIVE_LEADS_SYSTEM_PROMPT` 与 `build_user_prompt()`。
|
| 20 |
+
- 输出的 user prompt 可直接喂给 `evaluate_video_tool()`。
|
| 21 |
+
"""
|
| 22 |
+
|
| 23 |
+
from __future__ import annotations
|
| 24 |
+
|
| 25 |
+
from typing import Iterable
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
# 五工具定义与 VideoDR 一致（harness 按 name 解析工具调用，签名必须保持原样）。
|
| 29 |
+
_TOOLS_BLOCK = """<tools>
|
| 30 |
+
{"type": "function", "function": {"name": "choose_frames", "description": "Select a specific time interval to investigate further. The system will return uniformly sampled frames from within this interval. Use this when the answer lies within a specific segment.", "parameters": {"type": "object", "properties": {"start_frame_index": {"type": "integer", "description": "The starting frame index of the interval."}, "end_frame_index": {"type": "integer", "description": "The ending frame index of the interval."}}, "required": ["start_frame_index", "end_frame_index"]}}}
|
| 31 |
+
{"type": "function", "function": {"name": "find_frame", "description": "Lock onto a specific single frame index to examine it closely. Returns the exact frame.", "parameters": {"type": "object", "properties": {"frame_index": {"type": "integer", "description": "The exact frame index to lock onto."}}, "required": ["frame_index"]}}}
|
| 32 |
+
{"type": "function", "function": {"name": "zoom_in", "description": "Magnify a specific region of the frame currently locked by find_frame for detailed visual analysis.", "parameters": {"type": "object", "properties": {"bbox": {"type": "array", "items": {"type": "integer"}, "minItems": 4, "maxItems": 4, "description": "Bounding box [x1, y1, x2, y2] in 0-1000 relative coordinates, where 0 <= x1 < x2 <= 1000 and 0 <= y1 < y2 <= 1000."}}, "required": ["bbox"]}}}
|
| 33 |
+
{"type": "function", "function": {"name": "image_search", "description": "Reverse image search a specific region of the locked frame to identify unknown entities. Use this if you DO NOT understand who the zoomed-in person is.", "parameters": {"type": "object", "properties": {"bbox": {"type": "array", "items": {"type": "integer"}, "minItems": 4, "maxItems": 4, "description": "Bounding box [x1, y1, x2, y2] in 0-1000 relative coordinates."}}, "required": ["bbox"]}}}
|
| 34 |
+
{"type": "function", "function": {"name": "web_search", "description": "Search the web using a text query. Use this if you ALREADY know who the person is and just need external facts such as their fame, profession, or notable works.", "parameters": {"type": "object", "properties": {"query": {"type": "string", "description": "Text query to search the web."}}, "required": ["query"]}}}
|
| 35 |
+
</tools>"""
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
ATTRACTIVE_LEADS_SYSTEM_PROMPT = """# Role
|
| 39 |
+
You are an advanced Video Talent-Scout agent for visual dataset curation.
|
| 40 |
+
Your job is to decide whether a video is suitable for extracting visually ATTRACTIVE male and/or female lead material ("帅哥 / 美女") for a curated dataset. You do this by navigating the video with tools, then verifying each candidate's attractiveness, clarity/extractability, identity, and fame before concluding. You are a scout for good-looking, cleanly-extractable people — NOT a video critic and NOT a generic entity-identification QA bot.
|
| 41 |
+
|
| 42 |
+
# Video Context
|
| 43 |
+
- The original video has been converted to 1 frame per second (1 fps).
|
| 44 |
+
- The original frame index (e.g., Frame 1, Frame 10) is watermarked in the top-left corner of every frame.
|
| 45 |
+
- You are initially provided with 64 uniformly sampled frames from the video.
|
| 46 |
+
|
| 47 |
+
# Tools
|
| 48 |
+
You may call one or more functions to assist with the task.
|
| 49 |
+
You are provided with function signatures within <tools></tools> XML tags:
|
| 50 |
+
""" + _TOOLS_BLOCK + """
|
| 51 |
+
|
| 52 |
+
# Tool Dependency & Workflow Rules (STRICT)
|
| 53 |
+
1. **Initial Action:** Skim the 64 sparse frames and spot the most visually attractive male and/or female person(s). If a strong candidate is already obvious, call `find_frame` directly; otherwise call `choose_frames` to narrow to the segment where that person appears clearest.
|
| 54 |
+
2. **Interval to Frame:** After `choose_frames` returns sub-frames, you MUST call `find_frame` to lock ONE specific frame before any detailed action.
|
| 55 |
+
3. **Detailing:** `zoom_in` and `image_search` may ONLY be used AFTER a successful `find_frame`. For each candidate you intend to mark usable: lock the clearest frame, then `zoom_in` on the face / upper body to judge attractiveness, face size, and sharpness. Skip `zoom_in` only if the locked frame is already large and sharp.
|
| 56 |
+
4. **Identity & Fame Verification:**
|
| 57 |
+
- If you see an attractive person but do not know who they are, use `image_search(bbox)` on their face.
|
| 58 |
+
- Only AFTER you have a plausible name from image_search (or you clearly recognize the person) may you use `web_search(query)` to confirm identity and gauge fame (actor / idol / influencer, follower scale, notable works).
|
| 59 |
+
- NEVER identify a person from the title, subtitles, watermark, channel name, or prior knowledge alone — it must come from the face via image_search / recognition.
|
| 60 |
+
5. **Per-gender Requirement:** You may mark a gender as "usable" ONLY if you have locked a clear frame of that person AND attempted `image_search` on their face. Approving BOTH genders is better but NOT required; one clearly usable gender is enough for an overall "good".
|
| 61 |
+
6. **Retry Mechanism:** If a candidate is unclear or a search fails, loop back to `choose_frames` / `find_frame` to try another segment or another person. You have a MAXIMUM OF 10 ATTEMPTS (loops). Mark a candidate "not_usable" if their best available face is tiny, heavily occluded, profile-only, too dark, blurry, over-filtered, covered by subtitles/stickers, or on screen too briefly to extract clean material.
|
| 62 |
+
|
| 63 |
+
# Attractiveness & Extractability Rubric
|
| 64 |
+
Judge each present candidate on:
|
| 65 |
+
- **Attractiveness:** facial aesthetics and overall visual appeal as a "帅哥" (handsome man) or "美女" (beautiful woman). This is the PRIMARY criterion.
|
| 66 |
+
- **Clarity & size:** face large, sharp, well-lit, frontal or near-frontal.
|
| 67 |
+
- **Stability & duration:** appears long enough or repeatedly; low motion blur; minimal occlusion / filters / text over the face.
|
| 68 |
+
- **Identity & fame (supporting):** a recognizable public figure raises value, but an attractive, cleanly-extractable unknown can still be "usable".
|
| 69 |
+
|
| 70 |
+
# Output Format (STRICT)
|
| 71 |
+
At each turn you must EITHER issue ONE precise tool call OR give the final answer. Every output MUST begin with a thought process in <think></think>.
|
| 72 |
+
|
| 73 |
+
1. If reasoning continues:
|
| 74 |
+
<think> ... </think>
|
| 75 |
+
<tool_call>{"name": "<function-name>", "arguments": <args-json-object>}</tool_call>
|
| 76 |
+
|
| 77 |
+
2. If ready to conclude (after locking + searching the candidate(s)):
|
| 78 |
+
<think> ... </think>
|
| 79 |
+
<answer>
|
| 80 |
+
{
|
| 81 |
+
"decision": "good | bad",
|
| 82 |
+
"decision_confidence": "high | medium | low",
|
| 83 |
+
"approved_genders": "male | female | both | none",
|
| 84 |
+
"comment_signal": "<short audience signal from comments, or 'unavailable'>",
|
| 85 |
+
"visual_signal": "<short overall visual / production quality note>",
|
| 86 |
+
"summary": "<one-sentence justification of the decision>",
|
| 87 |
+
"male_candidate": {
|
| 88 |
+
"status": "usable | not_usable | absent",
|
| 89 |
+
"actor_name": "<name if identified via image_search/web_search, else ''>",
|
| 90 |
+
"fame": "<celebrity / idol / influencer / unknown + brief basis, or ''>",
|
| 91 |
+
"extractability": "<face size, clarity, frontality, stability assessment>",
|
| 92 |
+
"evidence": "<frames inspected + image_search / web_search findings>"
|
| 93 |
+
},
|
| 94 |
+
"female_candidate": {
|
| 95 |
+
"status": "usable | not_usable | absent",
|
| 96 |
+
"actor_name": "<name if identified via image_search/web_search, else ''>",
|
| 97 |
+
"fame": "<celebrity / idol / influencer / unknown + brief basis, or ''>",
|
| 98 |
+
"extractability": "<face size, clarity, frontality, stability assessment>",
|
| 99 |
+
"evidence": "<frames inspected + image_search / web_search findings>"
|
| 100 |
+
}
|
| 101 |
+
}
|
| 102 |
+
</answer>
|
| 103 |
+
|
| 104 |
+
Final-answer rules:
|
| 105 |
+
- Output ONLY the JSON object inside <answer> (no prose outside <think>/<answer>, no markdown code fences).
|
| 106 |
+
- Set "decision" to "good" if at least one gender's candidate is "usable"; otherwise "bad".
|
| 107 |
+
- If evidence is weak, contradictory, or you never locked AND searched a clear candidate, default to "bad".
|
| 108 |
+
- If comments were not provided, set "comment_signal" to "unavailable" instead of inventing traction.
|
| 109 |
+
- For a gender that does not appear in the video, set its candidate "status" to "absent" and leave the other fields as "".
|
| 110 |
+
|
| 111 |
+
# Stop Searching Rules (STRICT)
|
| 112 |
+
- Once you have enough visual + identity evidence to judge every present gender, stop calling tools and emit <answer>.
|
| 113 |
+
- Do not repeat a near-duplicate web_search after it already returned the same fact.
|
| 114 |
+
- Near the attempt limit, decide with the evidence already collected; mention residual uncertainty only inside <think>, never as a third decision label."""
|
| 115 |
+
|
| 116 |
+
|
| 117 |
+
USER_QUESTION_TEMPLATE = """Evaluate whether this video is suitable for extracting visually attractive male / female lead material ("帅哥 / 美女") for dataset curation.
|
| 118 |
+
|
| 119 |
+
Use the tools to: scan the frames, lock the clearest male and/or female candidate, zoom in to judge attractiveness and clarity, image_search the face to identify the person, and web_search to check fame once you have a name. Then return the final decision in the JSON format defined by the system Output Format.
|
| 120 |
+
|
| 121 |
+
Video metadata:
|
| 122 |
+
- Title: {title}
|
| 123 |
+
- Channel: {channel}
|
| 124 |
+
- Duration (seconds): {duration}
|
| 125 |
+
- View count: {view_count}
|
| 126 |
+
- URL: {url}
|
| 127 |
+
|
| 128 |
+
Comments (treat as audience signal only if present; otherwise treat as unavailable):
|
| 129 |
+
{comments}
|
| 130 |
+
"""
|
| 131 |
+
|
| 132 |
+
|
| 133 |
+
def _format_comment(author: str, text: str, like_count: int | None, time_text: str) -> str:
|
| 134 |
+
safe_author = (author or "unknown").strip()
|
| 135 |
+
safe_text = " ".join((text or "").strip().split())
|
| 136 |
+
safe_text = safe_text[:400]
|
| 137 |
+
likes = 0 if like_count is None else int(like_count)
|
| 138 |
+
safe_time = (time_text or "").strip() or "unknown time"
|
| 139 |
+
return f"- [{likes} likes | {safe_time}] {safe_author}: {safe_text}"
|
| 140 |
+
|
| 141 |
+
|
| 142 |
+
def format_comments(comments: Iterable[dict], max_comments: int = 20) -> str:
|
| 143 |
+
normalized = []
|
| 144 |
+
for item in comments or []:
|
| 145 |
+
if not isinstance(item, dict):
|
| 146 |
+
continue
|
| 147 |
+
text = (item.get("text") or "").strip()
|
| 148 |
+
if not text:
|
| 149 |
+
continue
|
| 150 |
+
normalized.append({
|
| 151 |
+
"author": item.get("author", ""),
|
| 152 |
+
"text": text,
|
| 153 |
+
"like_count": item.get("like_count", 0) or 0,
|
| 154 |
+
"time_text": item.get("time_text", ""),
|
| 155 |
+
})
|
| 156 |
+
if not normalized:
|
| 157 |
+
return "(No comments available)"
|
| 158 |
+
|
| 159 |
+
normalized.sort(key=lambda x: (-int(x["like_count"]), x["text"]))
|
| 160 |
+
selected = normalized[:max_comments]
|
| 161 |
+
return "\n".join(
|
| 162 |
+
_format_comment(
|
| 163 |
+
author=item["author"],
|
| 164 |
+
text=item["text"],
|
| 165 |
+
like_count=item["like_count"],
|
| 166 |
+
time_text=item["time_text"],
|
| 167 |
+
)
|
| 168 |
+
for item in selected
|
| 169 |
+
)
|
| 170 |
+
|
| 171 |
+
|
| 172 |
+
def build_user_prompt(metadata: dict) -> str:
|
| 173 |
+
comments = format_comments(metadata.get("comments", []))
|
| 174 |
+
return USER_QUESTION_TEMPLATE.format(
|
| 175 |
+
title=(metadata.get("title") or "").strip() or "Unknown",
|
| 176 |
+
channel=(metadata.get("channel") or "").strip() or "Unknown",
|
| 177 |
+
duration=metadata.get("duration", "Unknown"),
|
| 178 |
+
view_count=metadata.get("view_count", "Unknown"),
|
| 179 |
+
url=(metadata.get("url") or "").strip() or "Unknown",
|
| 180 |
+
comments=comments,
|
| 181 |
+
)
|
inference/eval.py
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
inference/fetch_youtube_online.py
ADDED
|
@@ -0,0 +1,253 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
在线抓取一个 YouTube 视频 + 顶层评论，落地到 work_dir，方便单条推理使用。
|
| 3 |
+
|
| 4 |
+
输入:
|
| 5 |
+
- `--url`：视频链接（支持 `watch?v=`、`shorts/`、`live/`、`youtu.be/` 等）。
|
| 6 |
+
- `--work-dir`：抓取产物存放目录（默认 `inference/online_runs/<video_id>_<ts>`）。
|
| 7 |
+
- `--max-comments`：抓取评论条数上限，默认 100。
|
| 8 |
+
- `--yt-dlp`：可执行 yt-dlp 路径，默认走 `PATH`。
|
| 9 |
+
|
| 10 |
+
处理:
|
| 11 |
+
- 调用 yt-dlp CLI，跳过 storyboard，下载 mp4 视频，并把评论写入 info.json。
|
| 12 |
+
- 解析 info.json，规范化 metadata（title、channel、duration、view_count、url）。
|
| 13 |
+
- 把评论字段从 yt-dlp 的 `_time_text` 等改写成 `attractive_leads_prompt.format_comments`
|
| 14 |
+
期望的字段（author/text/like_count/time_text）。
|
| 15 |
+
|
| 16 |
+
输出:
|
| 17 |
+
- `<work_dir>/<video_id>.mp4`
|
| 18 |
+
- `<work_dir>/<video_id>.info.json`：yt-dlp 原始输出（含完整评论）。
|
| 19 |
+
- `<work_dir>/comments.json`：list[dict]，可直接喂给 `--comments-file`。
|
| 20 |
+
- `<work_dir>/metadata.json`：单条 metadata，对齐 people_cn 的 jsonl 字段。
|
| 21 |
+
- 同时把关键字段以 `KEY=VALUE` 的形式 echo 到 stdout，便于 shell 解析。
|
| 22 |
+
"""
|
| 23 |
+
|
| 24 |
+
from __future__ import annotations
|
| 25 |
+
|
| 26 |
+
import argparse
|
| 27 |
+
import json
|
| 28 |
+
import os
|
| 29 |
+
import shlex
|
| 30 |
+
import subprocess
|
| 31 |
+
import sys
|
| 32 |
+
import time
|
| 33 |
+
from pathlib import Path
|
| 34 |
+
from typing import Any
|
| 35 |
+
from urllib.parse import parse_qs, urlparse
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
def extract_video_id_from_url(video_url: str) -> str:
|
| 39 |
+
if not video_url:
|
| 40 |
+
return ""
|
| 41 |
+
parsed = urlparse(video_url)
|
| 42 |
+
host = (parsed.netloc or "").lower()
|
| 43 |
+
parts = [p for p in parsed.path.split("/") if p]
|
| 44 |
+
if "youtu.be" in host and parts:
|
| 45 |
+
return parts[0]
|
| 46 |
+
if "youtube.com" in host:
|
| 47 |
+
if parts[:1] == ["watch"]:
|
| 48 |
+
return parse_qs(parsed.query).get("v", [""])[0]
|
| 49 |
+
if parts[:1] in (["shorts"], ["live"], ["embed"]) and len(parts) >= 2:
|
| 50 |
+
return parts[1]
|
| 51 |
+
return ""
|
| 52 |
+
|
| 53 |
+
|
| 54 |
+
def run_yt_dlp(
|
| 55 |
+
yt_dlp: str,
|
| 56 |
+
url: str,
|
| 57 |
+
work_dir: Path,
|
| 58 |
+
max_comments: int,
|
| 59 |
+
cookies_from_browser: str = "",
|
| 60 |
+
cookies_file: str = "",
|
| 61 |
+
extra_args: list[str] | None = None,
|
| 62 |
+
) -> Path:
|
| 63 |
+
"""Download video + comments, return path to info.json."""
|
| 64 |
+
work_dir.mkdir(parents=True, exist_ok=True)
|
| 65 |
+
output_template = str(work_dir / "%(id)s.%(ext)s")
|
| 66 |
+
|
| 67 |
+
cmd: list[str] = [
|
| 68 |
+
yt_dlp,
|
| 69 |
+
"--no-playlist",
|
| 70 |
+
"--write-info-json",
|
| 71 |
+
"--write-comments",
|
| 72 |
+
"--extractor-args",
|
| 73 |
+
# all|N,reply N,reply max,replies per comment N,sort
|
| 74 |
+
f"youtube:max_comments={max_comments},all,{max_comments},all;comment_sort=top",
|
| 75 |
+
"--format",
|
| 76 |
+
"mp4/bestvideo*+bestaudio/best",
|
| 77 |
+
"--merge-output-format",
|
| 78 |
+
"mp4",
|
| 79 |
+
# 走代理时大媒体流容易 SSL EOF，分块下载 + 多重试 + 强制 IPv4 更稳。
|
| 80 |
+
"--http-chunk-size", os.environ.get("YT_DLP_HTTP_CHUNK_SIZE", "1M"),
|
| 81 |
+
"--retries", "20",
|
| 82 |
+
"--fragment-retries", "20",
|
| 83 |
+
"--socket-timeout", "30",
|
| 84 |
+
"--force-ipv4",
|
| 85 |
+
"-o",
|
| 86 |
+
output_template,
|
| 87 |
+
"--no-progress",
|
| 88 |
+
"--quiet",
|
| 89 |
+
url,
|
| 90 |
+
]
|
| 91 |
+
if cookies_from_browser:
|
| 92 |
+
cmd[1:1] = ["--cookies-from-browser", cookies_from_browser]
|
| 93 |
+
if cookies_file:
|
| 94 |
+
cmd[1:1] = ["--cookies", cookies_file]
|
| 95 |
+
if extra_args:
|
| 96 |
+
cmd.extend(extra_args)
|
| 97 |
+
|
| 98 |
+
print(f"[fetch] yt-dlp: {' '.join(shlex.quote(c) for c in cmd)}", flush=True)
|
| 99 |
+
proc = subprocess.run(cmd, capture_output=True, text=True)
|
| 100 |
+
if proc.stdout:
|
| 101 |
+
print(proc.stdout, end="")
|
| 102 |
+
if proc.stderr:
|
| 103 |
+
print(proc.stderr, end="", file=sys.stderr)
|
| 104 |
+
if proc.returncode != 0:
|
| 105 |
+
raise SystemExit(f"yt-dlp failed with exit code {proc.returncode}")
|
| 106 |
+
|
| 107 |
+
info_files = sorted(work_dir.glob("*.info.json"))
|
| 108 |
+
if not info_files:
|
| 109 |
+
raise SystemExit(f"yt-dlp did not produce an info.json under {work_dir}")
|
| 110 |
+
return info_files[-1]
|
| 111 |
+
|
| 112 |
+
|
| 113 |
+
def normalize_comment(raw: dict[str, Any]) -> dict[str, Any] | None:
|
| 114 |
+
text = (raw.get("text") or "").strip()
|
| 115 |
+
if not text:
|
| 116 |
+
return None
|
| 117 |
+
return {
|
| 118 |
+
"author": raw.get("author") or "",
|
| 119 |
+
"text": text,
|
| 120 |
+
"like_count": raw.get("like_count") or 0,
|
| 121 |
+
# yt-dlp uses `_time_text`; the prompt formatter expects `time_text`.
|
| 122 |
+
"time_text": raw.get("_time_text") or raw.get("time_text") or "",
|
| 123 |
+
"is_pinned": bool(raw.get("is_pinned")),
|
| 124 |
+
"author_is_uploader": bool(raw.get("author_is_uploader")),
|
| 125 |
+
"parent": raw.get("parent") or "root",
|
| 126 |
+
}
|
| 127 |
+
|
| 128 |
+
|
| 129 |
+
def build_metadata(info: dict[str, Any], url: str, max_comments: int) -> tuple[dict[str, Any], list[dict[str, Any]]]:
|
| 130 |
+
raw_comments = info.get("comments") or []
|
| 131 |
+
normalized: list[dict[str, Any]] = []
|
| 132 |
+
for raw in raw_comments:
|
| 133 |
+
if not isinstance(raw, dict):
|
| 134 |
+
continue
|
| 135 |
+
if (raw.get("parent") or "root") != "root":
|
| 136 |
+
continue
|
| 137 |
+
item = normalize_comment(raw)
|
| 138 |
+
if item is not None:
|
| 139 |
+
normalized.append(item)
|
| 140 |
+
normalized.sort(key=lambda x: (-int(x.get("like_count") or 0), x.get("text") or ""))
|
| 141 |
+
normalized = normalized[:max_comments]
|
| 142 |
+
|
| 143 |
+
video_id = info.get("id") or extract_video_id_from_url(url)
|
| 144 |
+
metadata = {
|
| 145 |
+
"video_id": video_id,
|
| 146 |
+
"title": info.get("title") or "",
|
| 147 |
+
"url": info.get("webpage_url") or url,
|
| 148 |
+
"duration": info.get("duration"),
|
| 149 |
+
"channel": info.get("channel") or info.get("uploader") or "",
|
| 150 |
+
"view_count": info.get("view_count"),
|
| 151 |
+
"comments": normalized,
|
| 152 |
+
"comment_count": info.get("comment_count"),
|
| 153 |
+
"upload_date": info.get("upload_date") or "",
|
| 154 |
+
"fetch_time": time.strftime("%Y-%m-%dT%H:%M:%S"),
|
| 155 |
+
}
|
| 156 |
+
return metadata, normalized
|
| 157 |
+
|
| 158 |
+
|
| 159 |
+
def parse_args() -> argparse.Namespace:
|
| 160 |
+
parser = argparse.ArgumentParser(description="使用 yt-dlp 在线抓取视频和评论，给单条推理用")
|
| 161 |
+
parser.add_argument("--url", required=True, help="YouTube 视频链接")
|
| 162 |
+
parser.add_argument("--work-dir", default="", help="抓取产物目录")
|
| 163 |
+
parser.add_argument("--max-comments", type=int, default=100, help="保留的顶层评论上限")
|
| 164 |
+
parser.add_argument("--yt-dlp", default="yt-dlp", help="yt-dlp 可执行路径")
|
| 165 |
+
parser.add_argument("--cookies-from-browser", default="", help="例如 chrome:Default")
|
| 166 |
+
parser.add_argument("--cookies", default="", help="cookies.txt 路径")
|
| 167 |
+
parser.add_argument("--force", action="store_true", help="即使已有 mp4/info.json 也强制重抓")
|
| 168 |
+
parser.add_argument(
|
| 169 |
+
"--extra-yt-dlp-arg",
|
| 170 |
+
action="append",
|
| 171 |
+
default=[],
|
| 172 |
+
help="透传给 yt-dlp 的额外参数，可重复",
|
| 173 |
+
)
|
| 174 |
+
return parser.parse_args()
|
| 175 |
+
|
| 176 |
+
|
| 177 |
+
def main() -> None:
|
| 178 |
+
args = parse_args()
|
| 179 |
+
video_id = extract_video_id_from_url(args.url)
|
| 180 |
+
if not video_id:
|
| 181 |
+
raise SystemExit(f"无法从 URL 解析 video_id: {args.url}")
|
| 182 |
+
|
| 183 |
+
if args.work_dir:
|
| 184 |
+
work_dir = Path(args.work_dir).expanduser().resolve()
|
| 185 |
+
else:
|
| 186 |
+
ts = time.strftime("%Y%m%d_%H%M%S")
|
| 187 |
+
work_dir = (
|
| 188 |
+
Path(__file__).resolve().parent
|
| 189 |
+
/ "online_runs"
|
| 190 |
+
/ f"{video_id}_{ts}"
|
| 191 |
+
)
|
| 192 |
+
work_dir.mkdir(parents=True, exist_ok=True)
|
| 193 |
+
|
| 194 |
+
info_path = work_dir / f"{video_id}.info.json"
|
| 195 |
+
video_path = work_dir / f"{video_id}.mp4"
|
| 196 |
+
|
| 197 |
+
need_fetch = args.force or not info_path.exists() or not video_path.exists()
|
| 198 |
+
if need_fetch:
|
| 199 |
+
run_yt_dlp(
|
| 200 |
+
yt_dlp=args.yt_dlp,
|
| 201 |
+
url=args.url,
|
| 202 |
+
work_dir=work_dir,
|
| 203 |
+
max_comments=args.max_comments,
|
| 204 |
+
cookies_from_browser=args.cookies_from_browser,
|
| 205 |
+
cookies_file=args.cookies,
|
| 206 |
+
extra_args=args.extra_yt_dlp_arg,
|
| 207 |
+
)
|
| 208 |
+
else:
|
| 209 |
+
print(f"[fetch] reusing existing files in {work_dir}", flush=True)
|
| 210 |
+
|
| 211 |
+
if not video_path.exists():
|
| 212 |
+
# yt-dlp may have produced .mkv/.webm if mp4 unavailable; pick whatever
|
| 213 |
+
# video file lives next to the info.json with the same id stem.
|
| 214 |
+
candidates = sorted(work_dir.glob(f"{video_id}.*"))
|
| 215 |
+
media_candidates = [c for c in candidates if c.suffix not in (".json", ".info.json")]
|
| 216 |
+
if not media_candidates:
|
| 217 |
+
raise SystemExit(f"未找到下载好的视频文件: {work_dir}")
|
| 218 |
+
video_path = media_candidates[0]
|
| 219 |
+
|
| 220 |
+
with open(info_path, "r", encoding="utf-8") as f:
|
| 221 |
+
info = json.load(f)
|
| 222 |
+
metadata, comments = build_metadata(info, args.url, args.max_comments)
|
| 223 |
+
|
| 224 |
+
comments_path = work_dir / "comments.json"
|
| 225 |
+
metadata_path = work_dir / "metadata.json"
|
| 226 |
+
with open(comments_path, "w", encoding="utf-8") as f:
|
| 227 |
+
json.dump(comments, f, ensure_ascii=False, indent=2)
|
| 228 |
+
with open(metadata_path, "w", encoding="utf-8") as f:
|
| 229 |
+
json.dump(metadata, f, ensure_ascii=False, indent=2)
|
| 230 |
+
|
| 231 |
+
print(f"[fetch] video: {video_path}", flush=True)
|
| 232 |
+
print(f"[fetch] comments: {len(comments)} -> {comments_path}", flush=True)
|
| 233 |
+
print(f"[fetch] metadata: {metadata_path}", flush=True)
|
| 234 |
+
|
| 235 |
+
# Easy-to-parse KEY=VALUE block for shell consumers.
|
| 236 |
+
print("---FETCH---", flush=True)
|
| 237 |
+
print(f"VIDEO_ID={video_id}")
|
| 238 |
+
print(f"VIDEO_PATH={video_path}")
|
| 239 |
+
print(f"COMMENTS_FILE={comments_path}")
|
| 240 |
+
print(f"METADATA_FILE={metadata_path}")
|
| 241 |
+
print(f"WORK_DIR={work_dir}")
|
| 242 |
+
print(f"TITLE={metadata.get('title') or ''}")
|
| 243 |
+
print(f"CHANNEL={metadata.get('channel') or ''}")
|
| 244 |
+
duration = metadata.get("duration")
|
| 245 |
+
print(f"DURATION={'' if duration is None else duration}")
|
| 246 |
+
view_count = metadata.get("view_count")
|
| 247 |
+
print(f"VIEW_COUNT={'' if view_count is None else view_count}")
|
| 248 |
+
print(f"VIDEO_URL={metadata.get('url') or args.url}")
|
| 249 |
+
print("---END---", flush=True)
|
| 250 |
+
|
| 251 |
+
|
| 252 |
+
if __name__ == "__main__":
|
| 253 |
+
main()
|
inference/run_people_quality_inference.py
ADDED
|
@@ -0,0 +1,723 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
批量或单条运行“高颜值男女可抽取视频”筛选推理。
|
| 3 |
+
|
| 4 |
+
输入:
|
| 5 |
+
- 批量模式默认读取 demo 本地 `local_data/celebrity/metadata.jsonl`，并在
|
| 6 |
+
`local_data/celebrity/<video_id>.mp4` 查找本地视频。
|
| 7 |
+
- 单条模式可直接传 `--video-url`，脚本会优先在 metadata 中匹配同一条记录，
|
| 8 |
+
再按 `video_root/<video_id>.mp4` 或显式 `--video-path` 找到本地视频文件。
|
| 9 |
+
- 模型服务地址（默认读取 `MODEL_BASE_URL`），以及现有 inference 逻辑依赖的
|
| 10 |
+
MARS / image_search 配置。
|
| 11 |
+
|
| 12 |
+
处理:
|
| 13 |
+
- 对每条视频构造“是否适合抽取高颜值男/女素材”的英文 user prompt。
|
| 14 |
+
- 复用 `inference/eval.py::evaluate_video_tool()` 的五工具推理逻辑执行逐条推理。
|
| 15 |
+
- 从模型 `<answer>` 中解析结构化 JSON，提取 `approve / reject / undecided`
|
| 16 |
+
决策、男女候选状态和可抽取性。
|
| 17 |
+
- 在保留完整输入、完整轨迹、工具调用和锁定帧的同时，额外导出一个便于汇报的
|
| 18 |
+
`decision_table.csv`。
|
| 19 |
+
|
| 20 |
+
输出:
|
| 21 |
+
- 在输出目录下写入：
|
| 22 |
+
- `results.jsonl`
|
| 23 |
+
- `results.html`
|
| 24 |
+
- `summary.json`
|
| 25 |
+
- `decision_table.csv`
|
| 26 |
+
- `images/`
|
| 27 |
+
- `frame_cache/`
|
| 28 |
+
- `results.jsonl` 保留原始模型输出；`decision_table.csv` 提供结构化业务判断。
|
| 29 |
+
"""
|
| 30 |
+
|
| 31 |
+
from __future__ import annotations
|
| 32 |
+
|
| 33 |
+
import argparse
|
| 34 |
+
import asyncio
|
| 35 |
+
import csv
|
| 36 |
+
import json
|
| 37 |
+
import os
|
| 38 |
+
import re
|
| 39 |
+
import sys
|
| 40 |
+
import time
|
| 41 |
+
from pathlib import Path
|
| 42 |
+
from typing import Any
|
| 43 |
+
from urllib.parse import parse_qs, urlparse
|
| 44 |
+
|
| 45 |
+
import aiohttp
|
| 46 |
+
|
| 47 |
+
CURRENT_DIR = Path(__file__).resolve().parent
|
| 48 |
+
DEMO_ROOT = CURRENT_DIR.parent
|
| 49 |
+
LOCAL_DATA_ROOT = Path(
|
| 50 |
+
os.environ.get("VIDEO_DEEP_RESEARCH_DATA_ROOT", str(DEMO_ROOT / "local_data"))
|
| 51 |
+
).expanduser()
|
| 52 |
+
if str(CURRENT_DIR) not in sys.path:
|
| 53 |
+
sys.path.insert(0, str(CURRENT_DIR))
|
| 54 |
+
|
| 55 |
+
try:
|
| 56 |
+
from .attractive_leads_prompt import ATTRACTIVE_LEADS_SYSTEM_PROMPT, build_user_prompt # type: ignore
|
| 57 |
+
from .eval import ( # type: ignore
|
| 58 |
+
ImageSearchCache,
|
| 59 |
+
TavilyApiKeyPool,
|
| 60 |
+
configure_local_service_no_proxy,
|
| 61 |
+
create_http_session,
|
| 62 |
+
evaluate_video_tool,
|
| 63 |
+
extract_answer_allow_no_tag,
|
| 64 |
+
generate_html,
|
| 65 |
+
load_tavily_api_keys,
|
| 66 |
+
messages_to_input_string,
|
| 67 |
+
resolve_web_search_backend,
|
| 68 |
+
)
|
| 69 |
+
from .video_dr_bridge import ( # type: ignore
|
| 70 |
+
DEFAULT_VIDEO_INITIAL_FRAMES,
|
| 71 |
+
DEFAULT_VIDEO_INTERVAL_SAMPLES,
|
| 72 |
+
DEFAULT_VIDEO_JPEG_QUALITY,
|
| 73 |
+
DEFAULT_VIDEO_MAX_RESOLUTION,
|
| 74 |
+
GATEWAY_URL as VIDEO_DR_GATEWAY_URL,
|
| 75 |
+
MARS_RETRIEVAL_ADDRESS,
|
| 76 |
+
MARS_SUMMARIZER_ADDRESS,
|
| 77 |
+
)
|
| 78 |
+
except ImportError:
|
| 79 |
+
from attractive_leads_prompt import ATTRACTIVE_LEADS_SYSTEM_PROMPT, build_user_prompt
|
| 80 |
+
from eval import (
|
| 81 |
+
ImageSearchCache,
|
| 82 |
+
TavilyApiKeyPool,
|
| 83 |
+
configure_local_service_no_proxy,
|
| 84 |
+
create_http_session,
|
| 85 |
+
evaluate_video_tool,
|
| 86 |
+
extract_answer_allow_no_tag,
|
| 87 |
+
generate_html,
|
| 88 |
+
load_tavily_api_keys,
|
| 89 |
+
messages_to_input_string,
|
| 90 |
+
resolve_web_search_backend,
|
| 91 |
+
)
|
| 92 |
+
from video_dr_bridge import (
|
| 93 |
+
DEFAULT_VIDEO_INITIAL_FRAMES,
|
| 94 |
+
DEFAULT_VIDEO_INTERVAL_SAMPLES,
|
| 95 |
+
DEFAULT_VIDEO_JPEG_QUALITY,
|
| 96 |
+
DEFAULT_VIDEO_MAX_RESOLUTION,
|
| 97 |
+
GATEWAY_URL as VIDEO_DR_GATEWAY_URL,
|
| 98 |
+
MARS_RETRIEVAL_ADDRESS,
|
| 99 |
+
MARS_SUMMARIZER_ADDRESS,
|
| 100 |
+
)
|
| 101 |
+
|
| 102 |
+
|
| 103 |
+
DEFAULT_METADATA_PATH = str(LOCAL_DATA_ROOT / "celebrity" / "metadata.jsonl")
|
| 104 |
+
DEFAULT_VIDEO_ROOT = str(LOCAL_DATA_ROOT / "celebrity")
|
| 105 |
+
DEFAULT_OUTPUT_ROOT = str(DEMO_ROOT / "lead_filter_runs")
|
| 106 |
+
|
| 107 |
+
|
| 108 |
+
def extract_video_id_from_url(video_url: str) -> str:
|
| 109 |
+
if not video_url:
|
| 110 |
+
return ""
|
| 111 |
+
parsed = urlparse(video_url)
|
| 112 |
+
host = (parsed.netloc or "").lower()
|
| 113 |
+
path_parts = [part for part in parsed.path.split("/") if part]
|
| 114 |
+
if "youtu.be" in host and path_parts:
|
| 115 |
+
return path_parts[0]
|
| 116 |
+
if "youtube.com" in host:
|
| 117 |
+
if path_parts[:1] == ["watch"]:
|
| 118 |
+
return parse_qs(parsed.query).get("v", [""])[0]
|
| 119 |
+
if path_parts[:1] in (["shorts"], ["live"]) and len(path_parts) >= 2:
|
| 120 |
+
return path_parts[1]
|
| 121 |
+
return ""
|
| 122 |
+
|
| 123 |
+
|
| 124 |
+
def read_metadata_records(metadata_path: str, allow_missing: bool = False) -> list[dict[str, Any]]:
|
| 125 |
+
if not metadata_path:
|
| 126 |
+
return []
|
| 127 |
+
if not os.path.exists(metadata_path):
|
| 128 |
+
if allow_missing:
|
| 129 |
+
return []
|
| 130 |
+
raise FileNotFoundError(f"metadata.jsonl 不存在: {metadata_path}")
|
| 131 |
+
|
| 132 |
+
records: list[dict[str, Any]] = []
|
| 133 |
+
with open(metadata_path, "r", encoding="utf-8") as f:
|
| 134 |
+
for line_idx, line in enumerate(f, start=1):
|
| 135 |
+
line = line.strip()
|
| 136 |
+
if not line:
|
| 137 |
+
continue
|
| 138 |
+
try:
|
| 139 |
+
records.append(json.loads(line))
|
| 140 |
+
except json.JSONDecodeError as exc:
|
| 141 |
+
raise ValueError(f"metadata.jsonl 第 {line_idx} 行不是合法 JSON: {exc}") from exc
|
| 142 |
+
return records
|
| 143 |
+
|
| 144 |
+
|
| 145 |
+
def load_comments_file(path: str) -> list[dict[str, Any]]:
|
| 146 |
+
if not path:
|
| 147 |
+
return []
|
| 148 |
+
with open(path, "r", encoding="utf-8") as f:
|
| 149 |
+
payload = json.load(f)
|
| 150 |
+
if not isinstance(payload, list):
|
| 151 |
+
raise ValueError(f"--comments-file 必须是 JSON list: {path}")
|
| 152 |
+
|
| 153 |
+
comments: list[dict[str, Any]] = []
|
| 154 |
+
for idx, item in enumerate(payload, start=1):
|
| 155 |
+
if isinstance(item, dict):
|
| 156 |
+
comments.append(item)
|
| 157 |
+
elif isinstance(item, str):
|
| 158 |
+
comments.append({"author": f"comment_{idx}", "text": item})
|
| 159 |
+
return comments
|
| 160 |
+
|
| 161 |
+
|
| 162 |
+
def resolve_video_path(video_root: str, video_id: str, explicit_video_path: str = "") -> str:
|
| 163 |
+
if explicit_video_path:
|
| 164 |
+
if not os.path.exists(explicit_video_path):
|
| 165 |
+
raise FileNotFoundError(f"视频不存在: {explicit_video_path}")
|
| 166 |
+
return explicit_video_path
|
| 167 |
+
if not video_id:
|
| 168 |
+
raise ValueError("缺少 video_id，无法从 --video-root 推断视频路径；请传 --video-path")
|
| 169 |
+
video_path = os.path.join(video_root, f"{video_id}.mp4")
|
| 170 |
+
if not os.path.exists(video_path):
|
| 171 |
+
raise FileNotFoundError(f"视频不存在: {video_path}")
|
| 172 |
+
return video_path
|
| 173 |
+
|
| 174 |
+
|
| 175 |
+
def build_sample(raw: dict[str, Any], video_root: str, explicit_video_path: str = "") -> dict[str, Any]:
|
| 176 |
+
metadata = dict(raw)
|
| 177 |
+
video_id = (metadata.get("video_id") or "").strip()
|
| 178 |
+
if not video_id:
|
| 179 |
+
video_id = extract_video_id_from_url((metadata.get("url") or "").strip())
|
| 180 |
+
if not video_id and explicit_video_path:
|
| 181 |
+
video_id = Path(explicit_video_path).stem
|
| 182 |
+
if not video_id:
|
| 183 |
+
raise ValueError("样本缺少 video_id，且无法从 URL 或 video_path 推断")
|
| 184 |
+
|
| 185 |
+
video_path = resolve_video_path(video_root, video_id, explicit_video_path)
|
| 186 |
+
metadata["video_id"] = video_id
|
| 187 |
+
metadata.setdefault("url", "")
|
| 188 |
+
metadata.setdefault("title", "")
|
| 189 |
+
metadata.setdefault("channel", "")
|
| 190 |
+
metadata.setdefault("duration", "")
|
| 191 |
+
metadata.setdefault("view_count", "")
|
| 192 |
+
metadata.setdefault("comments", [])
|
| 193 |
+
sample_id = (metadata.get("sample_id") or "").strip() or f"lead-filter-{video_id}"
|
| 194 |
+
return {
|
| 195 |
+
"id": sample_id,
|
| 196 |
+
"source_id": video_id,
|
| 197 |
+
"video_id": video_id,
|
| 198 |
+
"video_path": video_path,
|
| 199 |
+
"dataset": "attractive_leads_filter",
|
| 200 |
+
"task_kind": "video_dr",
|
| 201 |
+
"question": build_user_prompt(metadata),
|
| 202 |
+
"metadata": metadata,
|
| 203 |
+
}
|
| 204 |
+
|
| 205 |
+
|
| 206 |
+
def load_batch_samples(metadata_path: str, video_root: str) -> list[dict[str, Any]]:
|
| 207 |
+
return [build_sample(raw, video_root) for raw in read_metadata_records(metadata_path)]
|
| 208 |
+
|
| 209 |
+
|
| 210 |
+
def find_metadata_record(records: list[dict[str, Any]], video_url: str, video_id: str) -> dict[str, Any] | None:
|
| 211 |
+
for raw in records:
|
| 212 |
+
raw_video_id = (raw.get("video_id") or "").strip()
|
| 213 |
+
raw_url = (raw.get("url") or "").strip()
|
| 214 |
+
if video_url and raw_url and raw_url == video_url:
|
| 215 |
+
return dict(raw)
|
| 216 |
+
if video_id and raw_video_id == video_id:
|
| 217 |
+
return dict(raw)
|
| 218 |
+
return None
|
| 219 |
+
|
| 220 |
+
|
| 221 |
+
def build_single_sample(args: argparse.Namespace) -> dict[str, Any]:
|
| 222 |
+
metadata_records = read_metadata_records(args.metadata, allow_missing=True)
|
| 223 |
+
parsed_video_id = extract_video_id_from_url(args.video_url)
|
| 224 |
+
metadata = find_metadata_record(metadata_records, args.video_url, args.video_id or parsed_video_id) or {}
|
| 225 |
+
|
| 226 |
+
if args.comments_file:
|
| 227 |
+
metadata["comments"] = load_comments_file(args.comments_file)
|
| 228 |
+
for key, value in (
|
| 229 |
+
("video_id", args.video_id),
|
| 230 |
+
("url", args.video_url),
|
| 231 |
+
("title", args.title),
|
| 232 |
+
("channel", args.channel),
|
| 233 |
+
("sample_id", args.sample_id),
|
| 234 |
+
):
|
| 235 |
+
if value:
|
| 236 |
+
metadata[key] = value
|
| 237 |
+
if args.duration is not None:
|
| 238 |
+
metadata["duration"] = args.duration
|
| 239 |
+
if args.view_count is not None:
|
| 240 |
+
metadata["view_count"] = args.view_count
|
| 241 |
+
|
| 242 |
+
return build_sample(metadata, args.video_root, args.video_path)
|
| 243 |
+
|
| 244 |
+
|
| 245 |
+
def parse_answer_json(output: str) -> tuple[dict[str, Any] | None, str]:
|
| 246 |
+
answer_text = extract_answer_allow_no_tag(output).strip()
|
| 247 |
+
if answer_text.startswith("```"):
|
| 248 |
+
answer_text = re.sub(r"^```(?:json)?\s*", "", answer_text)
|
| 249 |
+
answer_text = re.sub(r"\s*```$", "", answer_text)
|
| 250 |
+
|
| 251 |
+
normalized = re.sub(r"\s+", " ", answer_text).strip().strip('"').strip("'").lower()
|
| 252 |
+
label_map = {
|
| 253 |
+
"good": "good",
|
| 254 |
+
"bad": "bad",
|
| 255 |
+
"approve": "good",
|
| 256 |
+
"approved": "good",
|
| 257 |
+
"reject": "bad",
|
| 258 |
+
"rejected": "bad",
|
| 259 |
+
"uncertain": "bad",
|
| 260 |
+
"undecided": "bad",
|
| 261 |
+
"unknown": "bad",
|
| 262 |
+
"好": "good",
|
| 263 |
+
"不好": "bad",
|
| 264 |
+
"不行": "bad",
|
| 265 |
+
"可以": "good",
|
| 266 |
+
}
|
| 267 |
+
if normalized in label_map:
|
| 268 |
+
return {"decision": label_map[normalized]}, answer_text
|
| 269 |
+
|
| 270 |
+
candidates = [answer_text]
|
| 271 |
+
brace_match = re.search(r"(\{.*\})", answer_text, re.DOTALL)
|
| 272 |
+
if brace_match and brace_match.group(1) not in candidates:
|
| 273 |
+
candidates.append(brace_match.group(1))
|
| 274 |
+
|
| 275 |
+
for candidate in candidates:
|
| 276 |
+
if not candidate:
|
| 277 |
+
continue
|
| 278 |
+
try:
|
| 279 |
+
parsed = json.loads(candidate)
|
| 280 |
+
except json.JSONDecodeError:
|
| 281 |
+
continue
|
| 282 |
+
if isinstance(parsed, dict):
|
| 283 |
+
return parsed, candidate
|
| 284 |
+
return None, answer_text
|
| 285 |
+
|
| 286 |
+
|
| 287 |
+
def _safe_text(value: Any) -> str:
|
| 288 |
+
if value is None:
|
| 289 |
+
return ""
|
| 290 |
+
if isinstance(value, list):
|
| 291 |
+
return ", ".join(item for item in (_safe_text(x) for x in value) if item)
|
| 292 |
+
if isinstance(value, dict):
|
| 293 |
+
return json.dumps(value, ensure_ascii=False, sort_keys=True)
|
| 294 |
+
return str(value).strip()
|
| 295 |
+
|
| 296 |
+
|
| 297 |
+
def enrich_row_with_answer(row: dict[str, Any]) -> dict[str, Any]:
|
| 298 |
+
parsed_answer, raw_answer_text = parse_answer_json(row.get("output", ""))
|
| 299 |
+
row["parsed_answer"] = parsed_answer
|
| 300 |
+
row["raw_answer_text"] = raw_answer_text
|
| 301 |
+
row["answer_parse_status"] = "ok" if parsed_answer else "invalid_or_missing_json"
|
| 302 |
+
|
| 303 |
+
if not parsed_answer:
|
| 304 |
+
row["final_decision"] = ""
|
| 305 |
+
row["decision_confidence"] = ""
|
| 306 |
+
row["approved_genders"] = ""
|
| 307 |
+
row["comment_signal"] = ""
|
| 308 |
+
row["visual_signal"] = ""
|
| 309 |
+
row["summary_text"] = ""
|
| 310 |
+
row["male_status"] = ""
|
| 311 |
+
row["male_actor_name"] = ""
|
| 312 |
+
row["male_fame"] = ""
|
| 313 |
+
row["male_extractability"] = ""
|
| 314 |
+
row["male_evidence"] = ""
|
| 315 |
+
row["female_status"] = ""
|
| 316 |
+
row["female_actor_name"] = ""
|
| 317 |
+
row["female_fame"] = ""
|
| 318 |
+
row["female_extractability"] = ""
|
| 319 |
+
row["female_evidence"] = ""
|
| 320 |
+
return row
|
| 321 |
+
|
| 322 |
+
male = parsed_answer.get("male_candidate", {}) if isinstance(parsed_answer.get("male_candidate"), dict) else {}
|
| 323 |
+
female = parsed_answer.get("female_candidate", {}) if isinstance(parsed_answer.get("female_candidate"), dict) else {}
|
| 324 |
+
|
| 325 |
+
row["final_decision"] = _safe_text(parsed_answer.get("decision"))
|
| 326 |
+
row["decision_confidence"] = _safe_text(parsed_answer.get("decision_confidence"))
|
| 327 |
+
row["approved_genders"] = _safe_text(parsed_answer.get("approved_genders"))
|
| 328 |
+
row["comment_signal"] = _safe_text(parsed_answer.get("comment_signal"))
|
| 329 |
+
row["visual_signal"] = _safe_text(parsed_answer.get("visual_signal"))
|
| 330 |
+
row["summary_text"] = _safe_text(parsed_answer.get("summary"))
|
| 331 |
+
row["male_status"] = _safe_text(male.get("status"))
|
| 332 |
+
row["male_actor_name"] = _safe_text(male.get("actor_name"))
|
| 333 |
+
row["male_fame"] = _safe_text(male.get("fame"))
|
| 334 |
+
row["male_extractability"] = _safe_text(male.get("extractability"))
|
| 335 |
+
row["male_evidence"] = _safe_text(male.get("evidence"))
|
| 336 |
+
row["female_status"] = _safe_text(female.get("status"))
|
| 337 |
+
row["female_actor_name"] = _safe_text(female.get("actor_name"))
|
| 338 |
+
row["female_fame"] = _safe_text(female.get("fame"))
|
| 339 |
+
row["female_extractability"] = _safe_text(female.get("extractability"))
|
| 340 |
+
row["female_evidence"] = _safe_text(female.get("evidence"))
|
| 341 |
+
return row
|
| 342 |
+
|
| 343 |
+
|
| 344 |
+
def write_decision_table(results: list[dict[str, Any]], output_path: str) -> None:
|
| 345 |
+
fieldnames = [
|
| 346 |
+
"index",
|
| 347 |
+
"sample_id",
|
| 348 |
+
"video_id",
|
| 349 |
+
"title",
|
| 350 |
+
"channel",
|
| 351 |
+
"url",
|
| 352 |
+
"view_count",
|
| 353 |
+
"duration",
|
| 354 |
+
"final_decision",
|
| 355 |
+
"decision_confidence",
|
| 356 |
+
"approved_genders",
|
| 357 |
+
"male_status",
|
| 358 |
+
"male_actor_name",
|
| 359 |
+
"male_fame",
|
| 360 |
+
"male_extractability",
|
| 361 |
+
"female_status",
|
| 362 |
+
"female_actor_name",
|
| 363 |
+
"female_fame",
|
| 364 |
+
"female_extractability",
|
| 365 |
+
"comment_signal",
|
| 366 |
+
"visual_signal",
|
| 367 |
+
"summary_text",
|
| 368 |
+
"finish_reason",
|
| 369 |
+
"answer_parse_status",
|
| 370 |
+
"error",
|
| 371 |
+
]
|
| 372 |
+
with open(output_path, "w", encoding="utf-8-sig", newline="") as f:
|
| 373 |
+
writer = csv.DictWriter(f, fieldnames=fieldnames)
|
| 374 |
+
writer.writeheader()
|
| 375 |
+
for idx, row in enumerate(results, start=1):
|
| 376 |
+
writer.writerow({
|
| 377 |
+
"index": idx,
|
| 378 |
+
"sample_id": row.get("sample_id", ""),
|
| 379 |
+
"video_id": row.get("video_id", ""),
|
| 380 |
+
"title": row.get("title", ""),
|
| 381 |
+
"channel": row.get("channel", ""),
|
| 382 |
+
"url": row.get("url", ""),
|
| 383 |
+
"view_count": row.get("view_count", ""),
|
| 384 |
+
"duration": row.get("duration", ""),
|
| 385 |
+
"final_decision": row.get("final_decision", ""),
|
| 386 |
+
"decision_confidence": row.get("decision_confidence", ""),
|
| 387 |
+
"approved_genders": row.get("approved_genders", ""),
|
| 388 |
+
"male_status": row.get("male_status", ""),
|
| 389 |
+
"male_actor_name": row.get("male_actor_name", ""),
|
| 390 |
+
"male_fame": row.get("male_fame", ""),
|
| 391 |
+
"male_extractability": row.get("male_extractability", ""),
|
| 392 |
+
"female_status": row.get("female_status", ""),
|
| 393 |
+
"female_actor_name": row.get("female_actor_name", ""),
|
| 394 |
+
"female_fame": row.get("female_fame", ""),
|
| 395 |
+
"female_extractability": row.get("female_extractability", ""),
|
| 396 |
+
"comment_signal": row.get("comment_signal", ""),
|
| 397 |
+
"visual_signal": row.get("visual_signal", ""),
|
| 398 |
+
"summary_text": row.get("summary_text", ""),
|
| 399 |
+
"finish_reason": row.get("finish_reason", ""),
|
| 400 |
+
"answer_parse_status": row.get("answer_parse_status", ""),
|
| 401 |
+
"error": row.get("error", ""),
|
| 402 |
+
})
|
| 403 |
+
|
| 404 |
+
|
| 405 |
+
def build_summary(results: list[dict[str, Any]], elapsed: float) -> dict[str, Any]:
|
| 406 |
+
finish_reason_counts: dict[str, int] = {}
|
| 407 |
+
decision_counts: dict[str, int] = {}
|
| 408 |
+
error_count = 0
|
| 409 |
+
total_rounds = 0
|
| 410 |
+
with_answer = 0
|
| 411 |
+
parsed_answer_count = 0
|
| 412 |
+
|
| 413 |
+
for row in results:
|
| 414 |
+
finish_reason = row.get("finish_reason", "unknown") or "unknown"
|
| 415 |
+
finish_reason_counts[finish_reason] = finish_reason_counts.get(finish_reason, 0) + 1
|
| 416 |
+
decision = row.get("final_decision", "") or "missing"
|
| 417 |
+
decision_counts[decision] = decision_counts.get(decision, 0) + 1
|
| 418 |
+
if row.get("error"):
|
| 419 |
+
error_count += 1
|
| 420 |
+
total_rounds += int(row.get("num_round", 0) or 0)
|
| 421 |
+
if "<answer>" in (row.get("output") or ""):
|
| 422 |
+
with_answer += 1
|
| 423 |
+
if row.get("parsed_answer"):
|
| 424 |
+
parsed_answer_count += 1
|
| 425 |
+
|
| 426 |
+
return {
|
| 427 |
+
"total_samples": len(results),
|
| 428 |
+
"error_count": error_count,
|
| 429 |
+
"finish_reason_counts": finish_reason_counts,
|
| 430 |
+
"decision_counts": decision_counts,
|
| 431 |
+
"answered_count": with_answer,
|
| 432 |
+
"parsed_answer_count": parsed_answer_count,
|
| 433 |
+
"avg_rounds": (total_rounds / len(results)) if results else 0.0,
|
| 434 |
+
"elapsed_seconds": elapsed,
|
| 435 |
+
"elapsed_minutes": elapsed / 60.0,
|
| 436 |
+
}
|
| 437 |
+
|
| 438 |
+
|
| 439 |
+
async def health_check(base_url: str) -> None:
|
| 440 |
+
timeout = aiohttp.ClientTimeout(total=10)
|
| 441 |
+
print("Running health checks...", flush=True)
|
| 442 |
+
print(f" Checking model server: {base_url}", flush=True)
|
| 443 |
+
async with create_http_session(timeout) as session:
|
| 444 |
+
async with session.get(f"{base_url.rstrip('/')}/v1/models") as resp:
|
| 445 |
+
if resp.status != 200:
|
| 446 |
+
raise RuntimeError(f"Model server returned HTTP {resp.status}")
|
| 447 |
+
print(" Model server OK", flush=True)
|
| 448 |
+
|
| 449 |
+
if MARS_SUMMARIZER_ADDRESS:
|
| 450 |
+
judge_url = f"http://{MARS_SUMMARIZER_ADDRESS.rstrip('/')}/v1/models"
|
| 451 |
+
print(f" Checking MARS judge/summarizer: {judge_url}", flush=True)
|
| 452 |
+
async with session.get(judge_url, proxy="") as resp:
|
| 453 |
+
if resp.status != 200:
|
| 454 |
+
raise RuntimeError(f"MARS summarizer returned HTTP {resp.status}")
|
| 455 |
+
print(" MARS judge/summarizer OK", flush=True)
|
| 456 |
+
|
| 457 |
+
print("Health checks passed!", flush=True)
|
| 458 |
+
|
| 459 |
+
|
| 460 |
+
async def run_batch(args: argparse.Namespace) -> list[dict[str, Any]]:
|
| 461 |
+
single_mode = bool(args.video_url or args.video_path or args.video_id or args.sample_id)
|
| 462 |
+
samples = [build_single_sample(args)] if single_mode else load_batch_samples(args.metadata, args.video_root)
|
| 463 |
+
if args.target_ids:
|
| 464 |
+
selected = set(args.target_ids)
|
| 465 |
+
before = len(samples)
|
| 466 |
+
samples = [
|
| 467 |
+
sample for sample in samples
|
| 468 |
+
if sample["id"] in selected or sample["source_id"] in selected
|
| 469 |
+
]
|
| 470 |
+
print(f"Filtered samples with --target-ids: {len(samples)}/{before}", flush=True)
|
| 471 |
+
if not samples:
|
| 472 |
+
raise SystemExit("--target-ids did not match any sample")
|
| 473 |
+
|
| 474 |
+
output_dir = args.output_dir
|
| 475 |
+
os.makedirs(output_dir, exist_ok=True)
|
| 476 |
+
images_dir = os.path.join(output_dir, "images")
|
| 477 |
+
frame_cache_dir = os.path.join(output_dir, "frame_cache")
|
| 478 |
+
os.makedirs(images_dir, exist_ok=True)
|
| 479 |
+
os.makedirs(frame_cache_dir, exist_ok=True)
|
| 480 |
+
|
| 481 |
+
results_file = os.path.join(output_dir, "results.jsonl")
|
| 482 |
+
completed_ids = set()
|
| 483 |
+
results_for_html: list[dict[str, Any]] = []
|
| 484 |
+
if os.path.exists(results_file):
|
| 485 |
+
with open(results_file, "r", encoding="utf-8") as f:
|
| 486 |
+
for line in f:
|
| 487 |
+
line = line.strip()
|
| 488 |
+
if not line:
|
| 489 |
+
continue
|
| 490 |
+
row = json.loads(line)
|
| 491 |
+
results_for_html.append(row)
|
| 492 |
+
completed_ids.add(row.get("sample_id"))
|
| 493 |
+
print(f"Resuming: loaded {len(results_for_html)} completed results", flush=True)
|
| 494 |
+
samples = [sample for sample in samples if sample["id"] not in completed_ids]
|
| 495 |
+
|
| 496 |
+
print(f"Loaded {len(samples)} samples to run", flush=True)
|
| 497 |
+
image_search_cache = ImageSearchCache(os.path.join(output_dir, "image_search_cache.json"))
|
| 498 |
+
|
| 499 |
+
tavily_api_keys = load_tavily_api_keys()
|
| 500 |
+
tavily_key_pool = TavilyApiKeyPool(tavily_api_keys, cooldown_seconds=args.tavily_key_cooldown_seconds)
|
| 501 |
+
resolved_web_search_backend = resolve_web_search_backend(args.web_search_backend, tavily_api_keys)
|
| 502 |
+
if resolved_web_search_backend == "tavily" and not tavily_api_keys:
|
| 503 |
+
raise SystemExit(
|
| 504 |
+
"TAVILY_API_KEY / TAVILY_API_KEYS / TAVILY_API_KEY_FILE required when --web-search-backend=tavily"
|
| 505 |
+
)
|
| 506 |
+
print(f"web_search backend: {resolved_web_search_backend}", flush=True)
|
| 507 |
+
if resolved_web_search_backend == "tavily":
|
| 508 |
+
print(f"Tavily API key pool: {len(tavily_api_keys)} key(s)", flush=True)
|
| 509 |
+
|
| 510 |
+
shared_timeout = aiohttp.ClientTimeout(total=3600)
|
| 511 |
+
semaphore = asyncio.Semaphore(args.max_concurrent)
|
| 512 |
+
started_at = time.time()
|
| 513 |
+
|
| 514 |
+
async def run_one(sample: dict[str, Any]) -> dict[str, Any]:
|
| 515 |
+
async with semaphore:
|
| 516 |
+
try:
|
| 517 |
+
result = await evaluate_video_tool(
|
| 518 |
+
client=args.model_client,
|
| 519 |
+
model=args.model,
|
| 520 |
+
base_url=args.base_url,
|
| 521 |
+
api_key=args.api_key,
|
| 522 |
+
question=sample["question"],
|
| 523 |
+
video_path=sample["video_path"],
|
| 524 |
+
tool_system_prompt=ATTRACTIVE_LEADS_SYSTEM_PROMPT,
|
| 525 |
+
max_turns=args.max_turns,
|
| 526 |
+
sample_id=sample["id"],
|
| 527 |
+
images_dir=images_dir,
|
| 528 |
+
shared_session=shared_session,
|
| 529 |
+
serper_api_key=args.serper_api_key,
|
| 530 |
+
image_search_cache=image_search_cache,
|
| 531 |
+
frame_cache_dir=frame_cache_dir,
|
| 532 |
+
max_tokens=args.max_tokens,
|
| 533 |
+
temperature=args.temperature,
|
| 534 |
+
top_p=args.top_p,
|
| 535 |
+
top_k=args.top_k,
|
| 536 |
+
presence_penalty=args.presence_penalty,
|
| 537 |
+
repetition_penalty=args.repetition_penalty,
|
| 538 |
+
seed=args.seed,
|
| 539 |
+
min_pixels=args.min_pixels,
|
| 540 |
+
max_pixels=args.max_pixels,
|
| 541 |
+
factor=args.factor,
|
| 542 |
+
qwen_vl_processing=args.qwen_vl_processing,
|
| 543 |
+
video_initial_frames=args.video_initial_frames,
|
| 544 |
+
video_interval_samples=args.video_interval_samples,
|
| 545 |
+
video_max_resolution=args.video_max_resolution,
|
| 546 |
+
video_jpeg_quality=args.video_jpeg_quality,
|
| 547 |
+
web_search_backend=args.web_search_backend,
|
| 548 |
+
tavily_api_key=tavily_api_keys[0] if tavily_api_keys else "",
|
| 549 |
+
tavily_api_key_pool=tavily_key_pool,
|
| 550 |
+
tavily_search_depth=args.tavily_search_depth,
|
| 551 |
+
tavily_max_results=args.tavily_max_results,
|
| 552 |
+
tavily_include_answer=args.tavily_include_answer,
|
| 553 |
+
tavily_include_raw_content=args.tavily_include_raw_content,
|
| 554 |
+
tavily_topic=args.tavily_topic,
|
| 555 |
+
tavily_timeout=args.tavily_timeout,
|
| 556 |
+
)
|
| 557 |
+
except Exception as e:
|
| 558 |
+
return enrich_row_with_answer({
|
| 559 |
+
"sample_id": sample["id"],
|
| 560 |
+
"dataset": sample["dataset"],
|
| 561 |
+
"video_id": sample["video_id"],
|
| 562 |
+
"title": sample["metadata"].get("title", ""),
|
| 563 |
+
"channel": sample["metadata"].get("channel", ""),
|
| 564 |
+
"url": sample["metadata"].get("url", ""),
|
| 565 |
+
"view_count": sample["metadata"].get("view_count", ""),
|
| 566 |
+
"duration": sample["metadata"].get("duration", ""),
|
| 567 |
+
"input": "",
|
| 568 |
+
"output": "",
|
| 569 |
+
"trajectory": "",
|
| 570 |
+
"finish_reason": "error",
|
| 571 |
+
"num_round": 0,
|
| 572 |
+
"tool_calls": [],
|
| 573 |
+
"error": str(e),
|
| 574 |
+
})
|
| 575 |
+
|
| 576 |
+
input_str = messages_to_input_string(result["input_messages"], result.get("saved_images", []))
|
| 577 |
+
trajectory_str = messages_to_input_string(
|
| 578 |
+
result.get("conversation_messages", result["input_messages"]),
|
| 579 |
+
result.get("saved_images", []),
|
| 580 |
+
append_assistant_prompt=False,
|
| 581 |
+
)
|
| 582 |
+
row = {
|
| 583 |
+
"sample_id": sample["id"],
|
| 584 |
+
"dataset": sample["dataset"],
|
| 585 |
+
"video_id": sample["video_id"],
|
| 586 |
+
"title": sample["metadata"].get("title", ""),
|
| 587 |
+
"channel": sample["metadata"].get("channel", ""),
|
| 588 |
+
"url": sample["metadata"].get("url", ""),
|
| 589 |
+
"view_count": sample["metadata"].get("view_count", ""),
|
| 590 |
+
"duration": sample["metadata"].get("duration", ""),
|
| 591 |
+
"input": input_str,
|
| 592 |
+
"output": result["output"],
|
| 593 |
+
"trajectory": trajectory_str,
|
| 594 |
+
"finish_reason": result["finish_reason"],
|
| 595 |
+
"num_round": result["num_round"],
|
| 596 |
+
"tool_calls": result.get("tool_calls", []),
|
| 597 |
+
"error": result.get("error"),
|
| 598 |
+
"saved_images": result.get("saved_images", []),
|
| 599 |
+
}
|
| 600 |
+
if result.get("locked_frame_path"):
|
| 601 |
+
row["locked_frame_path"] = result["locked_frame_path"]
|
| 602 |
+
return enrich_row_with_answer(row)
|
| 603 |
+
|
| 604 |
+
async with create_http_session(shared_timeout) as shared_session:
|
| 605 |
+
tasks = [run_one(sample) for sample in samples]
|
| 606 |
+
with open(results_file, "a", encoding="utf-8") as f:
|
| 607 |
+
for idx, task in enumerate(asyncio.as_completed(tasks), start=1):
|
| 608 |
+
row = await task
|
| 609 |
+
jsonl_row = {k: v for k, v in row.items() if k != "saved_images"}
|
| 610 |
+
f.write(json.dumps(jsonl_row, ensure_ascii=False) + "\n")
|
| 611 |
+
f.flush()
|
| 612 |
+
results_for_html.append(row)
|
| 613 |
+
|
| 614 |
+
elapsed = time.time() - started_at
|
| 615 |
+
rate = idx / elapsed if elapsed > 0 else 0.0
|
| 616 |
+
eta = (len(tasks) - idx) / rate if rate > 0 else 0.0
|
| 617 |
+
print(
|
| 618 |
+
f"Progress: {idx}/{len(tasks)} | Elapsed: {elapsed:.0f}s | "
|
| 619 |
+
f"{rate:.2f} samples/s | ETA: {eta:.0f}s | "
|
| 620 |
+
f"{row['sample_id']} | finish={row.get('finish_reason')} | "
|
| 621 |
+
f"decision={row.get('final_decision', '') or 'n/a'} | error={row.get('error')}",
|
| 622 |
+
flush=True,
|
| 623 |
+
)
|
| 624 |
+
|
| 625 |
+
image_search_cache.save()
|
| 626 |
+
generate_html(results_for_html, os.path.join(output_dir, "results.html"), images_subdir="images")
|
| 627 |
+
write_decision_table(results_for_html, os.path.join(output_dir, "decision_table.csv"))
|
| 628 |
+
summary = build_summary(results_for_html, time.time() - started_at)
|
| 629 |
+
with open(os.path.join(output_dir, "summary.json"), "w", encoding="utf-8") as f:
|
| 630 |
+
json.dump(summary, f, ensure_ascii=False, indent=2)
|
| 631 |
+
return results_for_html
|
| 632 |
+
|
| 633 |
+
|
| 634 |
+
def parse_args() -> argparse.Namespace:
|
| 635 |
+
timestamp = time.strftime("%Y%m%d_%H%M%S")
|
| 636 |
+
parser = argparse.ArgumentParser(description="高颜值男女可抽取视频筛选推理")
|
| 637 |
+
parser.add_argument("--metadata", default=DEFAULT_METADATA_PATH, help="metadata.jsonl 路径；单条模式下可选")
|
| 638 |
+
parser.add_argument("--video-root", default=DEFAULT_VIDEO_ROOT, help="视频目录")
|
| 639 |
+
parser.add_argument(
|
| 640 |
+
"--output-dir",
|
| 641 |
+
default=os.path.join(DEFAULT_OUTPUT_ROOT, timestamp),
|
| 642 |
+
help="输出目录",
|
| 643 |
+
)
|
| 644 |
+
parser.add_argument("--video-url", default="", help="单条模式的视频链接")
|
| 645 |
+
parser.add_argument("--video-id", default="", help="单条模式的视频 id；未传时尝试从 URL 解析")
|
| 646 |
+
parser.add_argument("--video-path", default="", help="单条模式的视频本地路径；未传时尝试从 video_root 推断")
|
| 647 |
+
parser.add_argument("--sample-id", default="", help="单条模式的样本 id")
|
| 648 |
+
parser.add_argument("--title", default="", help="单条模式的视频标题")
|
| 649 |
+
parser.add_argument("--channel", default="", help="单条模式的频道名")
|
| 650 |
+
parser.add_argument("--duration", type=float, default=None, help="单条模式的视频时长（秒）")
|
| 651 |
+
parser.add_argument("--view-count", type=int, default=None, help="单条模式的播放量")
|
| 652 |
+
parser.add_argument("--comments-file", default="", help="单条模式的评论 JSON 文件路径")
|
| 653 |
+
parser.add_argument("--target-ids", nargs="+", default=None, help="只跑指定 video_id 或 sample_id")
|
| 654 |
+
parser.add_argument("--model-client", default="openai", choices=["openai", "gemini", "azure"])
|
| 655 |
+
parser.add_argument("--model", default="qwen3-vl-video-dr", help="模型名")
|
| 656 |
+
parser.add_argument("--base-url", default=os.environ.get("MODEL_BASE_URL", ""), help="模型服务地址")
|
| 657 |
+
parser.add_argument("--api-key", default=os.environ.get("OPENAI_API_KEY", ""), help="模型 API Key")
|
| 658 |
+
parser.add_argument("--serper-api-key", default=os.environ.get("SERPER_API_KEY", ""), help="SERPER key")
|
| 659 |
+
parser.add_argument(
|
| 660 |
+
"--web-search-backend",
|
| 661 |
+
type=str,
|
| 662 |
+
choices=["serper_gateway", "gateway", "gateway_serper", "internal_serper", "company_serper", "mars", "tavily", "auto"],
|
| 663 |
+
default=os.environ.get("WEB_SEARCH_BACKEND", os.environ.get("VIDEO_DR_WEB_SEARCH_BACKEND", "serper_gateway")),
|
| 664 |
+
help="web_search 工具后端：serper_gateway / mars(本地 wiki) / tavily / auto",
|
| 665 |
+
)
|
| 666 |
+
parser.add_argument("--tavily-search-depth", type=str, choices=["basic", "advanced"],
|
| 667 |
+
default=os.environ.get("TAVILY_SEARCH_DEPTH", "advanced"),
|
| 668 |
+
help="Tavily search_depth（--web-search-backend=tavily 时生效）")
|
| 669 |
+
parser.add_argument("--tavily-max-results", type=int,
|
| 670 |
+
default=int(os.environ.get("TAVILY_MAX_RESULTS", "8")),
|
| 671 |
+
help="Tavily max_results，clamp 到 1-20")
|
| 672 |
+
parser.add_argument("--tavily-include-answer", type=str,
|
| 673 |
+
default=os.environ.get("TAVILY_INCLUDE_ANSWER", "advanced"),
|
| 674 |
+
help="Tavily include_answer：false / true(basic) / advanced")
|
| 675 |
+
parser.add_argument("--tavily-include-raw-content", type=str,
|
| 676 |
+
default=os.environ.get("TAVILY_INCLUDE_RAW_CONTENT", "false"),
|
| 677 |
+
help="Tavily include_raw_content：false / true(markdown) / text")
|
| 678 |
+
parser.add_argument("--tavily-topic", type=str, choices=["general", "news", "finance"],
|
| 679 |
+
default=os.environ.get("TAVILY_TOPIC", "general"),
|
| 680 |
+
help="Tavily topic")
|
| 681 |
+
parser.add_argument("--tavily-timeout", type=int,
|
| 682 |
+
default=int(os.environ.get("TAVILY_TIMEOUT", "60")),
|
| 683 |
+
help="Tavily 请求超时（秒）")
|
| 684 |
+
parser.add_argument("--tavily-key-cooldown-seconds", type=int,
|
| 685 |
+
default=int(os.environ.get("TAVILY_KEY_COOLDOWN_SECONDS", "60")),
|
| 686 |
+
help="Tavily key 触发 429/5xx 后的冷却时间（秒）")
|
| 687 |
+
parser.add_argument("--max-concurrent", type=int, default=1)
|
| 688 |
+
parser.add_argument("--max-turns", type=int, default=10)
|
| 689 |
+
parser.add_argument("--max-tokens", type=int, default=6144)
|
| 690 |
+
parser.add_argument("--temperature", type=float, default=0.8)
|
| 691 |
+
parser.add_argument("--top-p", type=float, default=0.95)
|
| 692 |
+
parser.add_argument("--top-k", type=int, default=20)
|
| 693 |
+
parser.add_argument("--presence-penalty", type=float, default=1.5)
|
| 694 |
+
parser.add_argument("--repetition-penalty", type=float, default=1.0)
|
| 695 |
+
parser.add_argument("--seed", type=int, default=3407)
|
| 696 |
+
parser.add_argument("--min-pixels", type=int, default=65536)
|
| 697 |
+
parser.add_argument("--max-pixels", type=int, default=8294400)
|
| 698 |
+
parser.add_argument("--factor", type=int, default=32)
|
| 699 |
+
parser.add_argument("--qwen-vl-processing", type=lambda x: x.lower() == "true", default=True)
|
| 700 |
+
parser.add_argument("--video-initial-frames", type=int, default=DEFAULT_VIDEO_INITIAL_FRAMES)
|
| 701 |
+
parser.add_argument("--video-interval-samples", type=int, default=DEFAULT_VIDEO_INTERVAL_SAMPLES)
|
| 702 |
+
parser.add_argument("--video-max-resolution", type=int, default=DEFAULT_VIDEO_MAX_RESOLUTION)
|
| 703 |
+
parser.add_argument("--video-jpeg-quality", type=int, default=DEFAULT_VIDEO_JPEG_QUALITY)
|
| 704 |
+
return parser.parse_args()
|
| 705 |
+
|
| 706 |
+
|
| 707 |
+
def main() -> None:
|
| 708 |
+
args = parse_args()
|
| 709 |
+
if not args.base_url:
|
| 710 |
+
raise SystemExit("缺少模型服务地址：请设置 MODEL_BASE_URL 或传 --base-url")
|
| 711 |
+
|
| 712 |
+
configure_local_service_no_proxy(args.base_url)
|
| 713 |
+
configure_local_service_no_proxy(MARS_RETRIEVAL_ADDRESS)
|
| 714 |
+
configure_local_service_no_proxy(MARS_SUMMARIZER_ADDRESS)
|
| 715 |
+
configure_local_service_no_proxy(VIDEO_DR_GATEWAY_URL)
|
| 716 |
+
|
| 717 |
+
asyncio.run(health_check(args.base_url))
|
| 718 |
+
asyncio.run(run_batch(args))
|
| 719 |
+
print(f"Done. Results saved to: {args.output_dir}", flush=True)
|
| 720 |
+
|
| 721 |
+
|
| 722 |
+
if __name__ == "__main__":
|
| 723 |
+
main()
|
inference/video_dr_bridge.py
ADDED
|
@@ -0,0 +1,231 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
VideoDR 评测桥接模块。
|
| 3 |
+
|
| 4 |
+
输入:
|
| 5 |
+
- VideoDR CSV 标注文件、原始视频目录，以及 demo 本地打包的 `video_dr_gen`
|
| 6 |
+
工具实现。
|
| 7 |
+
- 评测阶段生成的 bbox、检索 query、本地 MARS 服务地址与图片搜索配置。
|
| 8 |
+
|
| 9 |
+
处理:
|
| 10 |
+
- 复用 SFT 构造代码里的统一 system prompt、bbox 归一化、图片搜索与 MARS web search 实现。
|
| 11 |
+
- 为当前仓库补充 VideoDR CSV 解析、原始视频按 1fps 抽帧、区间采样与单帧定位能力。
|
| 12 |
+
- 将 VideoDR 的无表头 CSV 行正规化为评测样本，并把视频路径映射到 `video/<id>.mp4`。
|
| 13 |
+
|
| 14 |
+
输出:
|
| 15 |
+
- 导出 VideoDR 统一 prompt、CSV loader、1fps 抽帧函数，以及评测器需要调用的工具函数。
|
| 16 |
+
- 为 `inference/eval.py` 提供与 SFT 构造代码一致的工具后端。
|
| 17 |
+
"""
|
| 18 |
+
|
| 19 |
+
from __future__ import annotations
|
| 20 |
+
|
| 21 |
+
import csv
|
| 22 |
+
import os
|
| 23 |
+
import subprocess
|
| 24 |
+
import sys
|
| 25 |
+
from pathlib import Path
|
| 26 |
+
from typing import Dict, List, Tuple
|
| 27 |
+
|
| 28 |
+
import numpy as np
|
| 29 |
+
|
| 30 |
+
|
| 31 |
+
# VideoDR SFT 构造代码（assemble_sft_dataset / config / prompts / utils）的位置。
|
| 32 |
+
# 解析顺序：环境变量 VIDEO_DR_SOURCE_ROOT > 本目录同级的打包目录 demo/video_dr_gen。
|
| 33 |
+
def _resolve_video_dr_source_root() -> Path:
|
| 34 |
+
env_root = os.environ.get("VIDEO_DR_SOURCE_ROOT", "").strip()
|
| 35 |
+
candidates = []
|
| 36 |
+
if env_root:
|
| 37 |
+
candidates.append(Path(env_root))
|
| 38 |
+
# demo/inference/video_dr_bridge.py -> demo/video_dr_gen
|
| 39 |
+
candidates.append(Path(__file__).resolve().parent.parent / "video_dr_gen")
|
| 40 |
+
for cand in candidates:
|
| 41 |
+
if cand.exists():
|
| 42 |
+
return cand
|
| 43 |
+
raise RuntimeError(
|
| 44 |
+
"找不到 VideoDR SFT 构造代码目录（assemble_sft_dataset/config/prompts/utils）。"
|
| 45 |
+
f" 已尝试: {[str(c) for c in candidates]}。"
|
| 46 |
+
" 请设置环境变量 VIDEO_DR_SOURCE_ROOT，或把 video_dr_gen 放到 demo/ 下。"
|
| 47 |
+
)
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
VIDEO_DR_SOURCE_ROOT = _resolve_video_dr_source_root()
|
| 51 |
+
|
| 52 |
+
if str(VIDEO_DR_SOURCE_ROOT) not in sys.path:
|
| 53 |
+
sys.path.insert(0, str(VIDEO_DR_SOURCE_ROOT))
|
| 54 |
+
|
| 55 |
+
import assemble_sft_dataset as _video_dr_assemble # type: ignore # noqa: E402
|
| 56 |
+
import config as _video_dr_config # type: ignore # noqa: E402
|
| 57 |
+
import utils as _video_dr_utils # type: ignore # noqa: E402
|
| 58 |
+
|
| 59 |
+
|
| 60 |
+
VIDEO_DR_SYSTEM_PROMPT_STEP14 = _video_dr_assemble.NEW_SYSTEM_PROMPT.replace(
|
| 61 |
+
"You have a MAXIMUM OF 5 ATTEMPTS (loops) to find the answer.",
|
| 62 |
+
"You have a MAXIMUM OF 10 ATTEMPTS (loops) to find the answer.",
|
| 63 |
+
)
|
| 64 |
+
|
| 65 |
+
VIDEO_DR_STOP_SEARCH_RULES = """
|
| 66 |
+
|
| 67 |
+
# Stop Searching Rules (STRICT)
|
| 68 |
+
- If a search result already contains the requested fact or enough evidence to infer the answer, stop using tools and provide the final `<answer>`.
|
| 69 |
+
- Do not repeatedly refine the same web search query after it has already returned the same fact. A near-duplicate query is not useful evidence.
|
| 70 |
+
- Near the final attempt, you MUST use the evidence already collected and answer; do not call another tool just to confirm the same fact again.
|
| 71 |
+
- If evidence is incomplete but the attempt budget is nearly exhausted, give the best supported answer in `<answer>` and mention uncertainty only inside `<think>`.
|
| 72 |
+
"""
|
| 73 |
+
|
| 74 |
+
VIDEO_DR_SYSTEM_PROMPT_QWEN235B_REPAIR = VIDEO_DR_SYSTEM_PROMPT_STEP14 + VIDEO_DR_STOP_SEARCH_RULES
|
| 75 |
+
VIDEO_DR_SYSTEM_PROMPT = VIDEO_DR_SYSTEM_PROMPT_QWEN235B_REPAIR
|
| 76 |
+
|
| 77 |
+
|
| 78 |
+
def get_video_dr_system_prompt(profile: str = "current") -> str:
|
| 79 |
+
"""按评测兼容 profile 返回 VideoDR system prompt。"""
|
| 80 |
+
if profile == "step14_plus_tavily432":
|
| 81 |
+
return VIDEO_DR_SYSTEM_PROMPT_STEP14
|
| 82 |
+
return VIDEO_DR_SYSTEM_PROMPT_QWEN235B_REPAIR
|
| 83 |
+
|
| 84 |
+
MARS_RETRIEVAL_ADDRESS = getattr(_video_dr_config, "MARS_RETRIEVAL_ADDRESS", "")
|
| 85 |
+
MARS_SUMMARIZER_ADDRESS = getattr(_video_dr_config, "MARS_SUMMARIZER_ADDRESS", "")
|
| 86 |
+
MARS_SUMMARIZER_MODEL = getattr(_video_dr_config, "MARS_SUMMARIZER_MODEL", "")
|
| 87 |
+
IMAGE_SEARCH_MODE = getattr(_video_dr_config, "IMAGE_SEARCH_MODE", "")
|
| 88 |
+
GATEWAY_URL = getattr(_video_dr_config, "GATEWAY_URL", "")
|
| 89 |
+
GATEWAY_USERNAME = getattr(_video_dr_config, "GATEWAY_USERNAME", "")
|
| 90 |
+
GATEWAY_USERID = getattr(_video_dr_config, "GATEWAY_USERID", "")
|
| 91 |
+
GATEWAY_TOKEN = getattr(_video_dr_config, "GATEWAY_TOKEN", "")
|
| 92 |
+
|
| 93 |
+
DEFAULT_VIDEO_MAX_RESOLUTION = getattr(_video_dr_config, "DEFAULT_MAX_RESOLUTION", 768)
|
| 94 |
+
DEFAULT_VIDEO_JPEG_QUALITY = getattr(_video_dr_config, "DEFAULT_JPEG_QUALITY", 85)
|
| 95 |
+
DEFAULT_VIDEO_INITIAL_FRAMES = getattr(_video_dr_config, "DEFAULT_MAX_FRAMES", 64)
|
| 96 |
+
DEFAULT_VIDEO_INTERVAL_SAMPLES = getattr(_video_dr_config, "DEFAULT_INTERVAL_SAMPLES", 8)
|
| 97 |
+
|
| 98 |
+
normalize_bbox = _video_dr_utils.normalize_bbox
|
| 99 |
+
get_bbox_config = _video_dr_utils.get_bbox_config
|
| 100 |
+
add_search_padding = _video_dr_utils.add_search_padding
|
| 101 |
+
crop_frame = _video_dr_utils.crop_frame
|
| 102 |
+
real_image_search = _video_dr_utils.real_image_search
|
| 103 |
+
mars_web_search = _video_dr_utils.mars_web_search
|
| 104 |
+
ImageSearchCache = _video_dr_utils.ImageSearchCache
|
| 105 |
+
|
| 106 |
+
|
| 107 |
+
def extract_video_frames_1fps(
|
| 108 |
+
video_path: str,
|
| 109 |
+
output_dir: str,
|
| 110 |
+
max_resolution: int = DEFAULT_VIDEO_MAX_RESOLUTION,
|
| 111 |
+
jpeg_quality: int = DEFAULT_VIDEO_JPEG_QUALITY,
|
| 112 |
+
) -> Dict[int, str]:
|
| 113 |
+
"""按 1fps 抽取视频帧，返回 `{frame_index: path}`。"""
|
| 114 |
+
os.makedirs(output_dir, exist_ok=True)
|
| 115 |
+
|
| 116 |
+
existing = {}
|
| 117 |
+
for name in sorted(os.listdir(output_dir)):
|
| 118 |
+
if name.startswith("frame_") and name.endswith(".jpg"):
|
| 119 |
+
idx = int(name.replace("frame_", "").replace(".jpg", ""))
|
| 120 |
+
existing[idx] = os.path.join(output_dir, name)
|
| 121 |
+
if existing:
|
| 122 |
+
return existing
|
| 123 |
+
|
| 124 |
+
vf = (
|
| 125 |
+
f"fps=1,"
|
| 126 |
+
f"scale='min({max_resolution},iw)':'min({max_resolution},ih)'"
|
| 127 |
+
f":force_original_aspect_ratio=decrease"
|
| 128 |
+
)
|
| 129 |
+
q = max(2, min(31, round(2 + (100 - jpeg_quality) * 29 / 99)))
|
| 130 |
+
cmd = [
|
| 131 |
+
"ffmpeg",
|
| 132 |
+
"-y",
|
| 133 |
+
"-i",
|
| 134 |
+
video_path,
|
| 135 |
+
"-vf",
|
| 136 |
+
vf,
|
| 137 |
+
"-q:v",
|
| 138 |
+
str(q),
|
| 139 |
+
"-start_number",
|
| 140 |
+
"0",
|
| 141 |
+
os.path.join(output_dir, "frame_%06d.jpg"),
|
| 142 |
+
]
|
| 143 |
+
proc = subprocess.run(cmd, capture_output=True, text=True, timeout=3600)
|
| 144 |
+
if proc.returncode != 0:
|
| 145 |
+
raise RuntimeError(f"ffmpeg 1fps 抽帧失败: {proc.stderr[-500:]}")
|
| 146 |
+
|
| 147 |
+
frames = {}
|
| 148 |
+
for name in sorted(os.listdir(output_dir)):
|
| 149 |
+
if name.startswith("frame_") and name.endswith(".jpg"):
|
| 150 |
+
idx = int(name.replace("frame_", "").replace(".jpg", ""))
|
| 151 |
+
frames[idx] = os.path.join(output_dir, name)
|
| 152 |
+
if not frames:
|
| 153 |
+
raise RuntimeError(f"未从视频中抽取到任何帧: {video_path}")
|
| 154 |
+
return frames
|
| 155 |
+
|
| 156 |
+
|
| 157 |
+
def uniform_sample_indices(total_frames: int, num_samples: int) -> List[int]:
|
| 158 |
+
"""在 `[0, total_frames)` 上均匀采样。"""
|
| 159 |
+
if total_frames <= num_samples:
|
| 160 |
+
return list(range(total_frames))
|
| 161 |
+
return sorted(set(int(i) for i in np.linspace(0, total_frames - 1, num_samples)))
|
| 162 |
+
|
| 163 |
+
|
| 164 |
+
def sample_interval(
|
| 165 |
+
all_frames: Dict[int, str],
|
| 166 |
+
start: int,
|
| 167 |
+
end: int,
|
| 168 |
+
num_samples: int = DEFAULT_VIDEO_INTERVAL_SAMPLES,
|
| 169 |
+
) -> List[Tuple[int, str]]:
|
| 170 |
+
"""在区间 `[start, end]` 内均匀采样若干帧。"""
|
| 171 |
+
available = sorted(k for k in all_frames if start <= k <= end)
|
| 172 |
+
if not available:
|
| 173 |
+
return []
|
| 174 |
+
if len(available) <= num_samples:
|
| 175 |
+
return [(k, all_frames[k]) for k in available]
|
| 176 |
+
positions = np.linspace(0, len(available) - 1, num_samples, dtype=int)
|
| 177 |
+
selected = sorted(set(available[p] for p in positions))
|
| 178 |
+
return [(k, all_frames[k]) for k in selected]
|
| 179 |
+
|
| 180 |
+
|
| 181 |
+
def get_frame(all_frames: Dict[int, str], idx: int) -> Tuple[int, str]:
|
| 182 |
+
"""返回精确帧；若不存在则返回最近邻帧。"""
|
| 183 |
+
if idx in all_frames:
|
| 184 |
+
return idx, all_frames[idx]
|
| 185 |
+
nearest = min(all_frames.keys(), key=lambda key: abs(key - idx))
|
| 186 |
+
return nearest, all_frames[nearest]
|
| 187 |
+
|
| 188 |
+
|
| 189 |
+
def load_videodr_csv_samples(
|
| 190 |
+
annotation_path: str,
|
| 191 |
+
video_root: str,
|
| 192 |
+
dataset_name: str,
|
| 193 |
+
) -> List[dict]:
|
| 194 |
+
"""读取无表头的 VideoDR CSV，并映射到 `video/<id>.mp4`。"""
|
| 195 |
+
samples: List[dict] = []
|
| 196 |
+
with open(annotation_path, "r", encoding="utf-8-sig", newline="") as f:
|
| 197 |
+
reader = csv.reader(f)
|
| 198 |
+
for row_idx, row in enumerate(reader, start=1):
|
| 199 |
+
if not row:
|
| 200 |
+
continue
|
| 201 |
+
if len(row) < 5:
|
| 202 |
+
raise ValueError(
|
| 203 |
+
f"VideoDR CSV 第 {row_idx} 行字段不足 5 个: {row}"
|
| 204 |
+
)
|
| 205 |
+
|
| 206 |
+
sample_id, question, answer, category, difficulty = [item.strip() for item in row[:5]]
|
| 207 |
+
if not sample_id:
|
| 208 |
+
raise ValueError(f"VideoDR CSV 第 {row_idx} 行缺少样本 id。")
|
| 209 |
+
|
| 210 |
+
video_path = os.path.join(video_root, f"{sample_id}.mp4")
|
| 211 |
+
if not os.path.exists(video_path):
|
| 212 |
+
raise FileNotFoundError(
|
| 213 |
+
f"VideoDR 视频不存在: id={sample_id}, path={video_path}"
|
| 214 |
+
)
|
| 215 |
+
|
| 216 |
+
samples.append({
|
| 217 |
+
"id": f"{dataset_name}-{sample_id}",
|
| 218 |
+
"source_id": sample_id,
|
| 219 |
+
"question": question,
|
| 220 |
+
"answer": [answer],
|
| 221 |
+
"dataset": dataset_name,
|
| 222 |
+
"task_kind": "video_dr",
|
| 223 |
+
"video_path": video_path,
|
| 224 |
+
"video_root": video_root,
|
| 225 |
+
"image_path": "",
|
| 226 |
+
"judge_image_path": "",
|
| 227 |
+
"category": category,
|
| 228 |
+
"difficulty": difficulty,
|
| 229 |
+
})
|
| 230 |
+
|
| 231 |
+
return samples
|
requirements.txt
ADDED
|
@@ -0,0 +1,204 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
aiohappyeyeballs==2.6.1
|
| 2 |
+
aiohttp==3.13.3
|
| 3 |
+
aiosignal==1.4.0
|
| 4 |
+
airportsdata==20260304
|
| 5 |
+
aliyun-python-sdk-core==2.16.0
|
| 6 |
+
aliyun-python-sdk-kms==2.16.5
|
| 7 |
+
annotated-doc==0.0.4
|
| 8 |
+
annotated-types==0.7.0
|
| 9 |
+
anthropic==0.84.0
|
| 10 |
+
anyio==4.12.1
|
| 11 |
+
apache-tvm-ffi==0.1.9
|
| 12 |
+
asttokens==3.0.1
|
| 13 |
+
attrs==25.4.0
|
| 14 |
+
beautifulsoup4==4.14.3
|
| 15 |
+
blobfile==3.0.0
|
| 16 |
+
build==1.4.0
|
| 17 |
+
certifi==2026.2.25
|
| 18 |
+
cffi==2.0.0
|
| 19 |
+
charset-normalizer==3.4.5
|
| 20 |
+
click==8.3.1
|
| 21 |
+
cloudpickle==3.1.2
|
| 22 |
+
# Optional model-server/GPU package; the online demo client does not import it.
|
| 23 |
+
# compressed-tensors==0.14.0.1
|
| 24 |
+
crcmod==1.7
|
| 25 |
+
cryptography==46.0.7
|
| 26 |
+
# Optional CUDA/server packages; they conflict with the local torch CUDA pins.
|
| 27 |
+
# cuda-bindings==12.9.6
|
| 28 |
+
# cuda-pathfinder==1.4.2
|
| 29 |
+
# cuda-python==12.9.0
|
| 30 |
+
datasets==4.7.0
|
| 31 |
+
decorator==5.2.1
|
| 32 |
+
dill==0.4.0
|
| 33 |
+
diskcache==5.6.3
|
| 34 |
+
distro==1.9.0
|
| 35 |
+
docstring_parser==0.17.0
|
| 36 |
+
einops==0.8.2
|
| 37 |
+
executing==2.2.1
|
| 38 |
+
fastapi==0.135.1
|
| 39 |
+
filelock==3.25.2
|
| 40 |
+
# Optional model-server/GPU packages.
|
| 41 |
+
# flashinfer-cubin==0.6.4
|
| 42 |
+
# flashinfer-python==0.6.4
|
| 43 |
+
frozenlist==1.8.0
|
| 44 |
+
fsspec==2026.2.0
|
| 45 |
+
gguf==0.18.0
|
| 46 |
+
google-auth==2.52.0
|
| 47 |
+
greenlet==3.4.0
|
| 48 |
+
grpcio==1.78.0
|
| 49 |
+
grpcio-health-checking==1.78.0
|
| 50 |
+
grpcio-reflection==1.78.0
|
| 51 |
+
h11==0.16.0
|
| 52 |
+
hf-xet==1.4.3
|
| 53 |
+
hf_transfer==0.1.9
|
| 54 |
+
httpcore==1.0.9
|
| 55 |
+
httpx==0.28.1
|
| 56 |
+
huggingface_hub==0.36.0
|
| 57 |
+
idna==3.11
|
| 58 |
+
ImageIO==2.37.3
|
| 59 |
+
interegular==0.3.3
|
| 60 |
+
ipython==9.11.0
|
| 61 |
+
ipython_pygments_lexers==1.1.1
|
| 62 |
+
jedi==0.19.2
|
| 63 |
+
Jinja2==3.1.6
|
| 64 |
+
jiter==0.13.0
|
| 65 |
+
jmespath==0.10.0
|
| 66 |
+
jsonschema==4.26.0
|
| 67 |
+
jsonschema-specifications==2025.9.1
|
| 68 |
+
lark==1.3.1
|
| 69 |
+
# Optional model-server package.
|
| 70 |
+
# llguidance==0.7.30
|
| 71 |
+
loguru==0.7.3
|
| 72 |
+
lxml==6.0.2
|
| 73 |
+
markdown-it-py==4.0.0
|
| 74 |
+
MarkupSafe==3.0.3
|
| 75 |
+
matplotlib-inline==0.2.1
|
| 76 |
+
mdurl==0.1.2
|
| 77 |
+
modelscope==1.34.0
|
| 78 |
+
mpmath==1.3.0
|
| 79 |
+
msgspec==0.20.0
|
| 80 |
+
multidict==6.7.1
|
| 81 |
+
multiprocess==0.70.18
|
| 82 |
+
nest-asyncio==1.6.0
|
| 83 |
+
networkx==3.6.1
|
| 84 |
+
ninja==1.13.0
|
| 85 |
+
numpy==2.4.3
|
| 86 |
+
# Optional NVIDIA CUDA packages; torch manages its own compatible pins.
|
| 87 |
+
# nvidia-cublas-cu12==12.8.4.1
|
| 88 |
+
# nvidia-cuda-cupti-cu12==12.8.90
|
| 89 |
+
# nvidia-cuda-nvrtc-cu12==12.8.93
|
| 90 |
+
# nvidia-cuda-runtime-cu12==12.8.90
|
| 91 |
+
# nvidia-cudnn-cu12==9.16.0.29
|
| 92 |
+
# nvidia-cudnn-frontend==1.19.1
|
| 93 |
+
# nvidia-cufft-cu12==11.3.3.83
|
| 94 |
+
# nvidia-cufile-cu12==1.13.1.3
|
| 95 |
+
# nvidia-curand-cu12==10.3.9.90
|
| 96 |
+
# nvidia-cusolver-cu12==11.7.3.90
|
| 97 |
+
# nvidia-cusparse-cu12==12.5.8.93
|
| 98 |
+
# nvidia-cusparselt-cu12==0.7.1
|
| 99 |
+
# nvidia-cutlass-dsl==4.3.5
|
| 100 |
+
# nvidia-ml-py==13.590.48
|
| 101 |
+
# nvidia-nccl-cu12==2.27.5
|
| 102 |
+
# nvidia-nvjitlink-cu12==12.8.93
|
| 103 |
+
# nvidia-nvshmem-cu12==3.3.20
|
| 104 |
+
# nvidia-nvtx-cu12==12.8.90
|
| 105 |
+
openai==2.6.1
|
| 106 |
+
openai-harmony==0.0.4
|
| 107 |
+
orjson==3.11.7
|
| 108 |
+
oss2==2.19.1
|
| 109 |
+
outlines==0.1.11
|
| 110 |
+
outlines_core==0.1.26
|
| 111 |
+
packaging==26.0
|
| 112 |
+
pandas==3.0.1
|
| 113 |
+
parso==0.8.6
|
| 114 |
+
partial-json-parser==0.2.1.1.post7
|
| 115 |
+
pexpect==4.9.0
|
| 116 |
+
pillow==12.1.1
|
| 117 |
+
playwright==1.58.0
|
| 118 |
+
prometheus_client==0.24.1
|
| 119 |
+
prompt_toolkit==3.0.52
|
| 120 |
+
propcache==0.4.1
|
| 121 |
+
protobuf==6.33.5
|
| 122 |
+
psutil==7.2.2
|
| 123 |
+
ptyprocess==0.7.0
|
| 124 |
+
pure_eval==0.2.3
|
| 125 |
+
py-spy==0.4.1
|
| 126 |
+
pyarrow==23.0.1
|
| 127 |
+
pyasn1==0.6.3
|
| 128 |
+
pyasn1_modules==0.4.2
|
| 129 |
+
pybase64==1.4.3
|
| 130 |
+
pycountry==26.2.16
|
| 131 |
+
pycparser==3.0
|
| 132 |
+
pycryptodome==3.23.0
|
| 133 |
+
pycryptodomex==3.23.0
|
| 134 |
+
pydantic==2.12.5
|
| 135 |
+
pydantic_core==2.41.5
|
| 136 |
+
pyee==13.0.1
|
| 137 |
+
Pygments==2.19.2
|
| 138 |
+
pyproject_hooks==1.2.0
|
| 139 |
+
python-dateutil==2.9.0.post0
|
| 140 |
+
python-multipart==0.0.22
|
| 141 |
+
PyYAML==6.0.3
|
| 142 |
+
pyzmq==27.1.0
|
| 143 |
+
# Optional model-server/GPU package.
|
| 144 |
+
# quack-kernels==0.2.4
|
| 145 |
+
referencing==0.37.0
|
| 146 |
+
regex==2026.2.28
|
| 147 |
+
requests==2.32.5
|
| 148 |
+
rich==15.0.0
|
| 149 |
+
rpds-py==0.30.0
|
| 150 |
+
safetensors==0.7.0
|
| 151 |
+
scipy==1.17.1
|
| 152 |
+
sentencepiece==0.2.1
|
| 153 |
+
setproctitle==1.3.7
|
| 154 |
+
setuptools==80.10.2
|
| 155 |
+
# Optional model-server/GPU packages.
|
| 156 |
+
# sgl-fa4==4.0.5
|
| 157 |
+
# sgl-kernel==0.3.21
|
| 158 |
+
# Optional model-server package. This online demo is only a client and does not
|
| 159 |
+
# import sglang; the pinned git package conflicts with transformers==5.2.0.
|
| 160 |
+
# sglang @ git+https://github.com/sgl-project/sglang.git@197f80713435dee9320fe39b71995adb1c0a732d#subdirectory=python
|
| 161 |
+
shellingham==1.5.4
|
| 162 |
+
six==1.17.0
|
| 163 |
+
smg-grpc-proto==0.4.2
|
| 164 |
+
sniffio==1.3.1
|
| 165 |
+
soundfile==0.13.1
|
| 166 |
+
soupsieve==2.8.3
|
| 167 |
+
stack-data==0.6.3
|
| 168 |
+
starlette==0.52.1
|
| 169 |
+
sympy==1.14.0
|
| 170 |
+
tabulate==0.10.0
|
| 171 |
+
tiktoken==0.12.0
|
| 172 |
+
# Optional training/model package; pulls torch and is not used by this client.
|
| 173 |
+
# timm==1.0.16
|
| 174 |
+
# Optional training/server native package; no Python 3.12 pip wheel is published.
|
| 175 |
+
# The online demo client does not import it.
|
| 176 |
+
# tinycudann==2.0
|
| 177 |
+
tokenizers==0.22.2
|
| 178 |
+
# Optional torch/server packages; the online demo client does not import them.
|
| 179 |
+
# torch_c_dlpack_ext==0.1.5
|
| 180 |
+
# torch_memory_saver==0.0.9
|
| 181 |
+
# torchao==0.9.0
|
| 182 |
+
# torchaudio==2.9.1
|
| 183 |
+
# torchcodec==0.9.1
|
| 184 |
+
# torchvision==0.24.1
|
| 185 |
+
tqdm==4.67.3
|
| 186 |
+
traitlets==5.14.3
|
| 187 |
+
transformers==4.57.1
|
| 188 |
+
# Optional model-server/GPU package.
|
| 189 |
+
# triton==3.5.1
|
| 190 |
+
typer==0.24.1
|
| 191 |
+
typer-slim==0.24.0
|
| 192 |
+
typing-inspection==0.4.2
|
| 193 |
+
typing_extensions==4.15.0
|
| 194 |
+
urllib3==2.6.3
|
| 195 |
+
uvicorn==0.41.0
|
| 196 |
+
uvloop==0.22.1
|
| 197 |
+
watchfiles==1.1.1
|
| 198 |
+
wcwidth==0.6.0
|
| 199 |
+
wheel==0.46.3
|
| 200 |
+
# Optional model-server package.
|
| 201 |
+
# xgrammar==0.1.27
|
| 202 |
+
xxhash==3.6.0
|
| 203 |
+
yarl==1.23.0
|
| 204 |
+
yt-dlp==2026.3.17
|
run_people_online_tavily.sh
ADDED
|
@@ -0,0 +1,180 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/bin/bash
|
| 2 |
+
# ==============================================================================
|
| 3 |
+
# 在线 Demo（Tavily 文搜版）—— 可迁移自包含版本
|
| 4 |
+
#
|
| 5 |
+
# 传入一个 YouTube URL：先用 yt-dlp 抓视频 + 评论，再跑“高颜值人物筛选”推理；
|
| 6 |
+
# web_search 工具走 Tavily（公网搜索）。
|
| 7 |
+
#
|
| 8 |
+
# 本目录（demo/）已自包含全部源码：
|
| 9 |
+
# demo/
|
| 10 |
+
# ├── run_people_online_tavily.sh <- 本脚本（入口）
|
| 11 |
+
# ├── inference/ <- 推理与抓取脚本
|
| 12 |
+
# ├── video_dr_gen/ <- video_dr 工具/帧处理依赖包
|
| 13 |
+
# ├── requirements.txt <- Python 依赖
|
| 14 |
+
# └── README.md <- 部署说明
|
| 15 |
+
#
|
| 16 |
+
# 用法:
|
| 17 |
+
# ./run_people_online_tavily.sh https://www.youtube.com/shorts/ZHI5IkRfXAM
|
| 18 |
+
# VIDEO_URL=https://... ./run_people_online_tavily.sh
|
| 19 |
+
# MAX_COMMENTS=50 ./run_people_online_tavily.sh <url>
|
| 20 |
+
#
|
| 21 |
+
# 迁移到新服务器时，按需用环境变量覆盖以下默认值（详见 README.md）:
|
| 22 |
+
# CONDA_ENV / CONDA_BASE / MODEL_BASE_URL / MODEL_NAME / TAVILY_API_KEY_FILE
|
| 23 |
+
# HTTP_PROXY_URL / YT_DLP
|
| 24 |
+
# ==============================================================================
|
| 25 |
+
|
| 26 |
+
set -euo pipefail
|
| 27 |
+
|
| 28 |
+
# --- 自包含：REPO_ROOT = 本脚本所在目录（demo/）------------------------------------
|
| 29 |
+
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
| 30 |
+
export PLAYWRIGHT_BROWSERS_PATH="${PLAYWRIGHT_BROWSERS_PATH:-$REPO_ROOT/.playwright-browsers}"
|
| 31 |
+
|
| 32 |
+
# Conda 环境：默认使用当前机器的本地 sglang 环境；也可用 CONDA_ENV 覆盖。
|
| 33 |
+
CONDA_ENV="${CONDA_ENV:-/home/tione/notebook/gaozhenkun/envs/sglang}"
|
| 34 |
+
# conda 安装根目录（用于 `conda shell.bash hook`），迁移时用 CONDA_BASE 覆盖。
|
| 35 |
+
CONDA_BASE="${CONDA_BASE:-/root/miniconda3}"
|
| 36 |
+
|
| 37 |
+
VIDEO_URL_INPUT="${1:-${VIDEO_URL:-}}"
|
| 38 |
+
if [[ -z "$VIDEO_URL_INPUT" ]]; then
|
| 39 |
+
echo "用法: $0 <youtube_url> 或 VIDEO_URL=<url> $0" >&2
|
| 40 |
+
exit 2
|
| 41 |
+
fi
|
| 42 |
+
|
| 43 |
+
TIMESTAMP="$(date +%Y%m%d_%H%M%S)"
|
| 44 |
+
MODEL_BASE_URL="${MODEL_BASE_URL:-http://100.102.37.219:8008}"
|
| 45 |
+
MODEL_NAME="${MODEL_NAME:-qwen3-vl-video-dr}"
|
| 46 |
+
MAX_CONCURRENT="${MAX_CONCURRENT:-1}"
|
| 47 |
+
MAX_TURNS="${MAX_TURNS:-10}"
|
| 48 |
+
MAX_COMMENTS="${MAX_COMMENTS:-100}"
|
| 49 |
+
|
| 50 |
+
# yt-dlp 可执行：优先用环境变量 / conda 自带，找不到时回退 PATH。
|
| 51 |
+
if [[ -n "${YT_DLP:-}" ]]; then
|
| 52 |
+
:
|
| 53 |
+
elif [[ -x "${CONDA_ENV}/bin/yt-dlp" ]]; then
|
| 54 |
+
YT_DLP="${CONDA_ENV}/bin/yt-dlp"
|
| 55 |
+
elif [[ -x "${CONDA_BASE}/bin/yt-dlp" ]]; then
|
| 56 |
+
YT_DLP="${CONDA_BASE}/bin/yt-dlp"
|
| 57 |
+
else
|
| 58 |
+
YT_DLP="yt-dlp"
|
| 59 |
+
fi
|
| 60 |
+
|
| 61 |
+
# --- 出公网代理（yt-dlp + Tavily 经此出网）-----------------------------------------
|
| 62 |
+
# 内网模型/judge/MARS 走 no_proxy 直连，公网走代理。若新机器直连公网，可设
|
| 63 |
+
# HTTP_PROXY_URL="" 关闭代理。
|
| 64 |
+
HTTP_PROXY_URL="${HTTP_PROXY_URL:-http://127.0.0.1:17890}"
|
| 65 |
+
if [[ -n "$HTTP_PROXY_URL" ]]; then
|
| 66 |
+
export http_proxy="$HTTP_PROXY_URL" https_proxy="$HTTP_PROXY_URL"
|
| 67 |
+
export HTTP_PROXY="$HTTP_PROXY_URL" HTTPS_PROXY="$HTTP_PROXY_URL"
|
| 68 |
+
export all_proxy="$HTTP_PROXY_URL" ALL_PROXY="$HTTP_PROXY_URL"
|
| 69 |
+
fi
|
| 70 |
+
|
| 71 |
+
# 从 MODEL_BASE_URL 解析出模型 host，连同内网网段一起加入 no_proxy。
|
| 72 |
+
MODEL_HOST_PORT="${MODEL_BASE_URL#*://}"
|
| 73 |
+
MODEL_HOST_PORT="${MODEL_HOST_PORT%%/*}"
|
| 74 |
+
MODEL_HOST="${MODEL_HOST_PORT%%:*}"
|
| 75 |
+
export no_proxy="${no_proxy:+$no_proxy,}localhost,127.0.0.1,${MODEL_HOST},${MODEL_HOST_PORT},100.99.167.149,100.99.167.149:8123,100.102.37.215:8000,10.0.0.0/8,100.64.0.0/10,.pjlab.org.cn"
|
| 76 |
+
export NO_PROXY="$no_proxy"
|
| 77 |
+
|
| 78 |
+
# --- Tavily 文搜配置 ----------------------------------------------------------------
|
| 79 |
+
# 三选一提供 key：TAVILY_API_KEY_FILE（每行一个 key）/ TAVILY_API_KEY / TAVILY_API_KEYS。
|
| 80 |
+
export TAVILY_API_KEY_FILE="${TAVILY_API_KEY_FILE:-$REPO_ROOT/secrets/tavily_api_keys.txt}"
|
| 81 |
+
TAVILY_SEARCH_DEPTH="${TAVILY_SEARCH_DEPTH:-advanced}"
|
| 82 |
+
TAVILY_INCLUDE_ANSWER="${TAVILY_INCLUDE_ANSWER:-advanced}"
|
| 83 |
+
TAVILY_INCLUDE_RAW_CONTENT="${TAVILY_INCLUDE_RAW_CONTENT:-false}"
|
| 84 |
+
TAVILY_MAX_RESULTS="${TAVILY_MAX_RESULTS:-5}"
|
| 85 |
+
|
| 86 |
+
cd "$REPO_ROOT"
|
| 87 |
+
|
| 88 |
+
# --- 激活 conda 环境 ---------------------------------------------------------------
|
| 89 |
+
eval "$("${CONDA_BASE}/bin/conda" shell.bash hook)"
|
| 90 |
+
conda activate "$CONDA_ENV"
|
| 91 |
+
|
| 92 |
+
# --- 第一步：用 yt-dlp 抓视频 + 评论 -------------------------------------------------
|
| 93 |
+
FETCH_ARGS=(
|
| 94 |
+
--url "$VIDEO_URL_INPUT"
|
| 95 |
+
--max-comments "$MAX_COMMENTS"
|
| 96 |
+
--yt-dlp "$YT_DLP"
|
| 97 |
+
)
|
| 98 |
+
if [[ -n "${WORK_DIR:-}" ]]; then
|
| 99 |
+
FETCH_ARGS+=(--work-dir "$WORK_DIR")
|
| 100 |
+
fi
|
| 101 |
+
if [[ -n "${YT_COOKIES_FROM_BROWSER:-}" ]]; then
|
| 102 |
+
FETCH_ARGS+=(--cookies-from-browser "$YT_COOKIES_FROM_BROWSER")
|
| 103 |
+
fi
|
| 104 |
+
if [[ -n "${YT_COOKIES:-}" ]]; then
|
| 105 |
+
FETCH_ARGS+=(--cookies "$YT_COOKIES")
|
| 106 |
+
fi
|
| 107 |
+
|
| 108 |
+
FETCH_TMP="$(mktemp)"
|
| 109 |
+
trap 'rm -f "$FETCH_TMP"' EXIT
|
| 110 |
+
python inference/fetch_youtube_online.py "${FETCH_ARGS[@]}" | tee "$FETCH_TMP"
|
| 111 |
+
|
| 112 |
+
# 解析 fetch 阶段输出的 KEY=VALUE 块
|
| 113 |
+
PARSE_BLOCK="$(awk '/^---FETCH---$/{flag=1; next} /^---END---$/{flag=0} flag' "$FETCH_TMP")"
|
| 114 |
+
if [[ -z "$PARSE_BLOCK" ]]; then
|
| 115 |
+
echo "fetch 阶段没有产生 KEY=VALUE 块，��法继续" >&2
|
| 116 |
+
exit 3
|
| 117 |
+
fi
|
| 118 |
+
|
| 119 |
+
while IFS='=' read -r key value; do
|
| 120 |
+
case "$key" in
|
| 121 |
+
VIDEO_ID) FETCH_VIDEO_ID="$value" ;;
|
| 122 |
+
VIDEO_PATH) FETCH_VIDEO_PATH="$value" ;;
|
| 123 |
+
COMMENTS_FILE) FETCH_COMMENTS_FILE="$value" ;;
|
| 124 |
+
METADATA_FILE) FETCH_METADATA_FILE="$value" ;;
|
| 125 |
+
WORK_DIR) FETCH_WORK_DIR="$value" ;;
|
| 126 |
+
TITLE) FETCH_TITLE="$value" ;;
|
| 127 |
+
CHANNEL) FETCH_CHANNEL="$value" ;;
|
| 128 |
+
DURATION) FETCH_DURATION="$value" ;;
|
| 129 |
+
VIEW_COUNT) FETCH_VIEW_COUNT="$value" ;;
|
| 130 |
+
VIDEO_URL) FETCH_VIDEO_URL="$value" ;;
|
| 131 |
+
esac
|
| 132 |
+
done <<< "$PARSE_BLOCK"
|
| 133 |
+
|
| 134 |
+
: "${FETCH_VIDEO_PATH:?fetch 未返回 VIDEO_PATH}"
|
| 135 |
+
: "${FETCH_COMMENTS_FILE:?fetch 未返回 COMMENTS_FILE}"
|
| 136 |
+
: "${FETCH_VIDEO_ID:?fetch 未返回 VIDEO_ID}"
|
| 137 |
+
|
| 138 |
+
OUTPUT_DIR="${OUTPUT_DIR:-$REPO_ROOT/lead_filter_runs/online_${FETCH_VIDEO_ID}_${TIMESTAMP}}"
|
| 139 |
+
LOG_FILE="${LOG_FILE:-${OUTPUT_DIR}.log}"
|
| 140 |
+
mkdir -p "$(dirname "$LOG_FILE")"
|
| 141 |
+
|
| 142 |
+
# --- 第二步：单条推理 ---------------------------------------------------------------
|
| 143 |
+
INFER_ARGS=(
|
| 144 |
+
--output-dir "$OUTPUT_DIR"
|
| 145 |
+
--model "$MODEL_NAME"
|
| 146 |
+
--base-url "$MODEL_BASE_URL"
|
| 147 |
+
--max-concurrent "$MAX_CONCURRENT"
|
| 148 |
+
--max-turns "$MAX_TURNS"
|
| 149 |
+
--video-url "${FETCH_VIDEO_URL:-$VIDEO_URL_INPUT}"
|
| 150 |
+
--video-id "$FETCH_VIDEO_ID"
|
| 151 |
+
--video-path "$FETCH_VIDEO_PATH"
|
| 152 |
+
--comments-file "$FETCH_COMMENTS_FILE"
|
| 153 |
+
--sample-id "online-${FETCH_VIDEO_ID}-${TIMESTAMP}"
|
| 154 |
+
--web-search-backend tavily
|
| 155 |
+
--tavily-search-depth "$TAVILY_SEARCH_DEPTH"
|
| 156 |
+
--tavily-include-answer "$TAVILY_INCLUDE_ANSWER"
|
| 157 |
+
--tavily-include-raw-content "$TAVILY_INCLUDE_RAW_CONTENT"
|
| 158 |
+
--tavily-max-results "$TAVILY_MAX_RESULTS"
|
| 159 |
+
)
|
| 160 |
+
if [[ -n "${FETCH_TITLE:-}" ]]; then
|
| 161 |
+
INFER_ARGS+=(--title "$FETCH_TITLE")
|
| 162 |
+
fi
|
| 163 |
+
if [[ -n "${FETCH_CHANNEL:-}" ]]; then
|
| 164 |
+
INFER_ARGS+=(--channel "$FETCH_CHANNEL")
|
| 165 |
+
fi
|
| 166 |
+
if [[ -n "${FETCH_DURATION:-}" ]]; then
|
| 167 |
+
INFER_ARGS+=(--duration "$FETCH_DURATION")
|
| 168 |
+
fi
|
| 169 |
+
if [[ -n "${FETCH_VIEW_COUNT:-}" ]]; then
|
| 170 |
+
INFER_ARGS+=(--view-count "$FETCH_VIEW_COUNT")
|
| 171 |
+
fi
|
| 172 |
+
|
| 173 |
+
# 调用方可以再透传额外参数，例如 --temperature 等。
|
| 174 |
+
python inference/run_people_quality_inference.py "${INFER_ARGS[@]}" "${@:2}" 2>&1 | tee "$LOG_FILE"
|
| 175 |
+
|
| 176 |
+
echo
|
| 177 |
+
echo "Online 推理完成："
|
| 178 |
+
echo " 下载产物 (视频/评论/info.json): ${FETCH_WORK_DIR:-?}"
|
| 179 |
+
echo " 推理结果: $OUTPUT_DIR"
|
| 180 |
+
echo " 日志: $LOG_FILE"
|
video_dr_gen/assemble_sft_dataset.py
ADDED
|
@@ -0,0 +1,197 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""
|
| 3 |
+
Assemble final SFT dataset from filtered trajectories + LLM filter decisions.
|
| 4 |
+
|
| 5 |
+
Reads:
|
| 6 |
+
- trajectories_final_filtered.jsonl (conversations + images)
|
| 7 |
+
- llm_filter_results.jsonl (keep/drop decisions by uid)
|
| 8 |
+
|
| 9 |
+
Outputs:
|
| 10 |
+
- sft_dataset.jsonl with only {conversations, images} per record,
|
| 11 |
+
system prompt replaced with the new Video DeepResearch prompt.
|
| 12 |
+
"""
|
| 13 |
+
|
| 14 |
+
import argparse
|
| 15 |
+
import json
|
| 16 |
+
import os
|
| 17 |
+
import re
|
| 18 |
+
import uuid
|
| 19 |
+
from pathlib import Path
|
| 20 |
+
|
| 21 |
+
BASE_DIR = Path(
|
| 22 |
+
os.environ.get(
|
| 23 |
+
"VIDEO_DR_ASSEMBLE_BASE_DIR",
|
| 24 |
+
str(Path(__file__).resolve().parent / "local_sft_runs"),
|
| 25 |
+
)
|
| 26 |
+
)
|
| 27 |
+
|
| 28 |
+
NEW_SYSTEM_PROMPT = r"""# Role
|
| 29 |
+
You are an advanced Video DeepResearch reasoning assistant.
|
| 30 |
+
Given a user query that requires combining visual clues from a video with external knowledge, your task is to solve the problem step-by-step by deeply navigating the video and searching the web.
|
| 31 |
+
|
| 32 |
+
# Video Context
|
| 33 |
+
- The original video has been converted to 1 frame per second (1 fps).
|
| 34 |
+
- The original frame index (e.g., Frame 1, Frame 10) is watermarked in the top-left corner of every frame.
|
| 35 |
+
- You are initially provided with 64 uniformly sampled frames from the video.
|
| 36 |
+
|
| 37 |
+
# Tools
|
| 38 |
+
You may call one or more functions to assist with the user query.
|
| 39 |
+
You are provided with function signatures within <tools></tools> XML tags:
|
| 40 |
+
<tools>
|
| 41 |
+
{"type": "function", "function": {"name": "choose_frames", "description": "Select a specific time interval to investigate further. The system will return uniformly sampled frames from within this interval. Use this when the answer lies within a specific segment.", "parameters": {"type": "object", "properties": {"start_frame_index": {"type": "integer", "description": "The starting frame index of the interval."}, "end_frame_index": {"type": "integer", "description": "The ending frame index of the interval."}}, "required": ["start_frame_index", "end_frame_index"]}}}
|
| 42 |
+
{"type": "function", "function": {"name": "find_frame", "description": "Lock onto a specific single frame index to examine it closely. Returns the exact frame.", "parameters": {"type": "object", "properties": {"frame_index": {"type": "integer", "description": "The exact frame index to lock onto."}}, "required": ["frame_index"]}}}
|
| 43 |
+
{"type": "function", "function": {"name": "zoom_in", "description": "Magnify a specific region of the frame currently locked by find_frame for detailed visual analysis.", "parameters": {"type": "object", "properties": {"bbox": {"type": "array", "items": {"type": "integer"}, "minItems": 4, "maxItems": 4, "description": "Bounding box [x1, y1, x2, y2] in 0-1000 relative coordinates, where 0 <= x1 < x2 <= 1000 and 0 <= y1 < y2 <= 1000."}}, "required": ["bbox"]}}}
|
| 44 |
+
{"type": "function", "function": {"name": "image_search", "description": "Reverse image search a specific region of the locked frame to identify unknown entities. Use this if you DO NOT understand what the zoomed-in entity is.", "parameters": {"type": "object", "properties": {"bbox": {"type": "array", "items": {"type": "integer"}, "minItems": 4, "maxItems": 4, "description": "Bounding box [x1, y1, x2, y2] in 0-1000 relative coordinates."}}, "required": ["bbox"]}}}
|
| 45 |
+
{"type": "function", "function": {"name": "web_search", "description": "Search the web using a text query. Use this if you ALREADY UNDERSTAND what the entity is and just need to search for related external knowledge or facts.", "parameters": {"type": "object", "properties": {"query": {"type": "string", "description": "Text query to search the web."}}, "required": ["query"]}}}
|
| 46 |
+
</tools>
|
| 47 |
+
|
| 48 |
+
# Tool Dependency & Workflow Rules (STRICT)
|
| 49 |
+
1. **Initial Action:** You can directly call `find_frame` if the target is obvious in the initial 64 sparse frames. Otherwise, call `choose_frames` to narrow down the search interval.
|
| 50 |
+
2. **Interval to Frame:** After calling `choose_frames` and receiving the sub-frames, you MUST call `find_frame` to lock onto a specific single frame before performing any detailed actions.
|
| 51 |
+
3. **Detailing & Searching:** `zoom_in` and `image_search` MUST ONLY be used AFTER a successful `find_frame` call. After `find_frame`, you may call `zoom_in` first to inspect details, then decide whether to use `image_search` (if the entity remains unrecognized) or `web_search` (if recognized) based on what you see. You may also skip `zoom_in` and go directly to `image_search` or `web_search` if the locked frame is already clear enough.
|
| 52 |
+
4. **Search Strategy Selection:**
|
| 53 |
+
- If you visually locate an entity but do not know its name/identity, use `image_search(bbox)`.
|
| 54 |
+
- If you visually locate and recognize the entity, bypass image search and directly use `web_search(query)`.
|
| 55 |
+
5. **Retry Mechanism:** If your current search fails, yields incorrect info, or doesn't solve the query, you can loop back to `choose_frames` or `find_frame` to explore another segment/frame. You have a MAXIMUM OF 5 ATTEMPTS (loops) to find the answer. If all attempts are exhausted, provide the best answer you have with an explicit note about remaining uncertainty.
|
| 56 |
+
|
| 57 |
+
# Output Format (STRICT)
|
| 58 |
+
At each turn, you must either issue ONE precise tool call OR provide the final answer.
|
| 59 |
+
All outputs MUST begin with a thought process enclosed in <think></think> tags.
|
| 60 |
+
|
| 61 |
+
1. If reasoning continues:
|
| 62 |
+
<think> ... </think>
|
| 63 |
+
<tool_call>{"name": "<function-name>", "arguments": <args-json-object>}</tool_call>
|
| 64 |
+
|
| 65 |
+
2. If ready to conclude (after gathering sufficient information):
|
| 66 |
+
<think> ... </think>
|
| 67 |
+
<answer> Final answer to the user's query </answer>"""
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
TOOL_CALL_RE = re.compile(r"(<tool_call>)\s*(.*?)\s*(</tool_call>)", re.DOTALL)
|
| 71 |
+
|
| 72 |
+
# Pattern to strip internal MARS retrieval engine markers from web_search tool responses.
|
| 73 |
+
# Example: 'Web Search Results for "query" (MARS retrieve+summarize, 3 passages):'
|
| 74 |
+
# Cleaned: 'Web Search Results for "query":'
|
| 75 |
+
MARS_MARKER_RE = re.compile(r"\s*\(MARS retrieve\+summarize,\s*\d+\s*passages?\)")
|
| 76 |
+
|
| 77 |
+
|
| 78 |
+
def clean_tool_response_markers(text: str) -> str:
|
| 79 |
+
"""Remove internal retrieval-engine markers (e.g. MARS) from tool responses."""
|
| 80 |
+
return MARS_MARKER_RE.sub("", text)
|
| 81 |
+
|
| 82 |
+
|
| 83 |
+
def convert_bbox_yxyx_to_xyxy(bbox):
|
| 84 |
+
"""Convert Gemini [y1, x1, y2, x2] to Qwen3VL [x1, y1, x2, y2]."""
|
| 85 |
+
if len(bbox) == 4:
|
| 86 |
+
y1, x1, y2, x2 = bbox
|
| 87 |
+
return [x1, y1, x2, y2]
|
| 88 |
+
return bbox
|
| 89 |
+
|
| 90 |
+
|
| 91 |
+
def convert_tool_call_bboxes(text):
|
| 92 |
+
"""Find all <tool_call> blocks and convert bbox from yxyx to xyxy."""
|
| 93 |
+
def replacer(match):
|
| 94 |
+
prefix, body, suffix = match.group(1), match.group(2), match.group(3)
|
| 95 |
+
try:
|
| 96 |
+
call = json.loads(body)
|
| 97 |
+
args = call.get("arguments", {})
|
| 98 |
+
if "bbox" in args and isinstance(args["bbox"], list):
|
| 99 |
+
args["bbox"] = convert_bbox_yxyx_to_xyxy(args["bbox"])
|
| 100 |
+
call["arguments"] = args
|
| 101 |
+
return f"{prefix}\n{json.dumps(call)}\n{suffix}"
|
| 102 |
+
except (json.JSONDecodeError, TypeError):
|
| 103 |
+
return match.group(0)
|
| 104 |
+
return TOOL_CALL_RE.sub(replacer, text)
|
| 105 |
+
|
| 106 |
+
|
| 107 |
+
def has_consecutive_same_role(conversations):
|
| 108 |
+
"""Check if conversations have consecutive turns from the same role (e.g. gpt→gpt)."""
|
| 109 |
+
for i in range(1, len(conversations)):
|
| 110 |
+
if conversations[i].get("from") == conversations[i - 1].get("from"):
|
| 111 |
+
return True
|
| 112 |
+
return False
|
| 113 |
+
|
| 114 |
+
|
| 115 |
+
def main():
|
| 116 |
+
parser = argparse.ArgumentParser(description="Assemble SFT dataset")
|
| 117 |
+
parser.add_argument("--trajectories", type=Path,
|
| 118 |
+
default=BASE_DIR / "trajectories_final_filtered.jsonl")
|
| 119 |
+
parser.add_argument("--llm-filter", type=Path,
|
| 120 |
+
default=BASE_DIR / "llm_filter_results.jsonl")
|
| 121 |
+
parser.add_argument("--output", type=Path,
|
| 122 |
+
default=BASE_DIR / "sft_dataset.jsonl")
|
| 123 |
+
parser.add_argument("--rejected", type=Path, default=None,
|
| 124 |
+
help="Output file for rejected records (default: <output_dir>/sft_rejected.jsonl)")
|
| 125 |
+
parser.add_argument("--system-prompt-file", type=Path, default=None,
|
| 126 |
+
help="Optional file to read system prompt from (overrides built-in prompt)")
|
| 127 |
+
args = parser.parse_args()
|
| 128 |
+
|
| 129 |
+
system_prompt = NEW_SYSTEM_PROMPT
|
| 130 |
+
if args.system_prompt_file:
|
| 131 |
+
system_prompt = args.system_prompt_file.read_text(encoding="utf-8").strip()
|
| 132 |
+
|
| 133 |
+
rejected_path = args.rejected or args.output.parent / "sft_rejected.jsonl"
|
| 134 |
+
|
| 135 |
+
# Load LLM filter decisions: uid -> decision
|
| 136 |
+
keep_uids = set()
|
| 137 |
+
with open(args.llm_filter, "r", encoding="utf-8") as f:
|
| 138 |
+
for line in f:
|
| 139 |
+
rec = json.loads(line.strip())
|
| 140 |
+
if rec.get("decision") == "keep":
|
| 141 |
+
keep_uids.add(rec["uid"])
|
| 142 |
+
|
| 143 |
+
kept = skipped = rejected = 0
|
| 144 |
+
with open(args.output, "w", encoding="utf-8") as out, \
|
| 145 |
+
open(rejected_path, "w", encoding="utf-8") as rej_out:
|
| 146 |
+
with open(args.trajectories, "r", encoding="utf-8") as f:
|
| 147 |
+
for line in f:
|
| 148 |
+
rec = json.loads(line.strip())
|
| 149 |
+
uid = rec.get("uid")
|
| 150 |
+
if uid not in keep_uids:
|
| 151 |
+
skipped += 1
|
| 152 |
+
continue
|
| 153 |
+
|
| 154 |
+
conversations = rec["conversations"]
|
| 155 |
+
images = rec["images"]
|
| 156 |
+
|
| 157 |
+
# Replace system prompt
|
| 158 |
+
if conversations and conversations[0].get("from") == "system":
|
| 159 |
+
conversations[0]["value"] = system_prompt
|
| 160 |
+
|
| 161 |
+
# Clean internal retrieval-engine markers from tool responses
|
| 162 |
+
for turn in conversations:
|
| 163 |
+
if turn.get("from") == "human":
|
| 164 |
+
turn["value"] = clean_tool_response_markers(turn["value"])
|
| 165 |
+
|
| 166 |
+
# Convert bbox from Gemini yxyx to xyxy in gpt tool_calls
|
| 167 |
+
for turn in conversations:
|
| 168 |
+
if turn.get("from") == "gpt":
|
| 169 |
+
turn["value"] = convert_tool_call_bboxes(turn["value"])
|
| 170 |
+
|
| 171 |
+
sft_id = f"vdr-sft-{uuid.uuid4().hex[:12]}"
|
| 172 |
+
|
| 173 |
+
record = {
|
| 174 |
+
"id": sft_id,
|
| 175 |
+
"conversations": conversations,
|
| 176 |
+
"images": images,
|
| 177 |
+
}
|
| 178 |
+
|
| 179 |
+
# Reject records with broken turn structure (e.g. consecutive gpt turns
|
| 180 |
+
# caused by blocked tool calls). These have causal dependencies between
|
| 181 |
+
# the blocked turn and the reaction turn, so stripping individual turns
|
| 182 |
+
# would break logical coherence.
|
| 183 |
+
if has_consecutive_same_role(conversations):
|
| 184 |
+
rej_out.write(json.dumps(record, ensure_ascii=False) + "\n")
|
| 185 |
+
rejected += 1
|
| 186 |
+
continue
|
| 187 |
+
|
| 188 |
+
out.write(json.dumps(record, ensure_ascii=False) + "\n")
|
| 189 |
+
kept += 1
|
| 190 |
+
|
| 191 |
+
print(f"Done. Kept: {kept}, Skipped: {skipped}, Rejected: {rejected}")
|
| 192 |
+
print(f"Output: {args.output}")
|
| 193 |
+
print(f"Rejected: {rejected_path}")
|
| 194 |
+
|
| 195 |
+
|
| 196 |
+
if __name__ == "__main__":
|
| 197 |
+
main()
|
video_dr_gen/config.py
ADDED
|
@@ -0,0 +1,334 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""
|
| 3 |
+
Video DeepResearch 配置模块。
|
| 4 |
+
|
| 5 |
+
输入:
|
| 6 |
+
- 仓库内的静态配置常量。
|
| 7 |
+
- 可选的 Vertex 账号池文件 `vertex_account_pool.json` 或环境变量 `VERTEX_ACCOUNT_POOL_FILE`。
|
| 8 |
+
- 本地检索、总结、图片搜索等服务地址。
|
| 9 |
+
|
| 10 |
+
处理:
|
| 11 |
+
- 解析 API 模式并生成运行时 endpoint。
|
| 12 |
+
- 在 Vertex 模式下，优先从 JSON 池加载 `project_id + service account json` 组合，
|
| 13 |
+
将每个条目转成独立的认证与请求入口，供后续轮询。
|
| 14 |
+
- 为本地 IP:端口补充 `no_proxy` / `NO_PROXY`，避免访问内网服务时误走代理。
|
| 15 |
+
|
| 16 |
+
输出:
|
| 17 |
+
- 导出 `API_ENDPOINTS`、`VERTEX_CREDENTIALS_POOL`、默认模型参数与检索服务配置。
|
| 18 |
+
- 为其他模块提供统一的全局配置入口。
|
| 19 |
+
"""
|
| 20 |
+
|
| 21 |
+
import json
|
| 22 |
+
import os
|
| 23 |
+
from pathlib import Path
|
| 24 |
+
from typing import Dict, List
|
| 25 |
+
from urllib.parse import urlparse
|
| 26 |
+
|
| 27 |
+
BASE_DIR = Path(__file__).resolve().parent
|
| 28 |
+
DEMO_ROOT = BASE_DIR.parent
|
| 29 |
+
LOCAL_DATA_ROOT = Path(
|
| 30 |
+
os.environ.get("VIDEO_DEEP_RESEARCH_DATA_ROOT", str(DEMO_ROOT / "local_data"))
|
| 31 |
+
).expanduser()
|
| 32 |
+
LOCAL_SECRETS_DIR = Path(
|
| 33 |
+
os.environ.get("VIDEO_DEEP_RESEARCH_SECRETS_DIR", str(DEMO_ROOT / "secrets"))
|
| 34 |
+
).expanduser()
|
| 35 |
+
|
| 36 |
+
|
| 37 |
+
def _append_no_proxy_entry(raw_entry: str) -> None:
|
| 38 |
+
"""将本地服务地址追加到 no_proxy / NO_PROXY。"""
|
| 39 |
+
entry = (raw_entry or "").strip()
|
| 40 |
+
if not entry:
|
| 41 |
+
return
|
| 42 |
+
if "://" in entry:
|
| 43 |
+
parsed = urlparse(entry)
|
| 44 |
+
entry = parsed.netloc or parsed.path
|
| 45 |
+
entry = entry.strip().strip("/")
|
| 46 |
+
if not entry:
|
| 47 |
+
return
|
| 48 |
+
|
| 49 |
+
for env_name in ("no_proxy", "NO_PROXY"):
|
| 50 |
+
current = os.environ.get(env_name, "")
|
| 51 |
+
items = [item.strip() for item in current.split(",") if item.strip()]
|
| 52 |
+
if entry not in items:
|
| 53 |
+
items.append(entry)
|
| 54 |
+
os.environ[env_name] = ",".join(items)
|
| 55 |
+
|
| 56 |
+
|
| 57 |
+
def _load_vertex_account_pool(pool_file: str) -> List[Dict]:
|
| 58 |
+
"""读取 Vertex 账号池配置，支持 list 或 {accounts:[...]} 两种格式。"""
|
| 59 |
+
if not pool_file:
|
| 60 |
+
return []
|
| 61 |
+
|
| 62 |
+
path = Path(pool_file)
|
| 63 |
+
if not path.is_absolute():
|
| 64 |
+
path = BASE_DIR / path
|
| 65 |
+
if not path.exists():
|
| 66 |
+
return []
|
| 67 |
+
|
| 68 |
+
with path.open("r", encoding="utf-8") as f:
|
| 69 |
+
data = json.load(f)
|
| 70 |
+
|
| 71 |
+
if isinstance(data, list):
|
| 72 |
+
records = data
|
| 73 |
+
elif isinstance(data, dict) and isinstance(data.get("accounts"), list):
|
| 74 |
+
records = data["accounts"]
|
| 75 |
+
else:
|
| 76 |
+
raise ValueError(
|
| 77 |
+
f"Vertex 账号池文件格式不合法: {path}。"
|
| 78 |
+
"应为 JSON 数组，或对象格式 {'accounts': [...]}。"
|
| 79 |
+
)
|
| 80 |
+
|
| 81 |
+
normalized: List[Dict] = []
|
| 82 |
+
for idx, item in enumerate(records, start=1):
|
| 83 |
+
if not isinstance(item, dict):
|
| 84 |
+
raise ValueError(f"Vertex 账号池第 {idx} 项必须是对象。")
|
| 85 |
+
if item.get("enabled", True) is False:
|
| 86 |
+
continue
|
| 87 |
+
|
| 88 |
+
project_id = (item.get("project_id") or "").strip()
|
| 89 |
+
if not project_id:
|
| 90 |
+
raise ValueError(f"Vertex 账号池第 {idx} 项缺少 project_id。")
|
| 91 |
+
|
| 92 |
+
credential_path = (item.get("service_account_key") or item.get("json_path") or "").strip()
|
| 93 |
+
service_account_info = item.get("service_account_info")
|
| 94 |
+
if not credential_path and not isinstance(service_account_info, dict):
|
| 95 |
+
raise ValueError(
|
| 96 |
+
f"Vertex 账号池第 {idx} 项必须提供 service_account_key/json_path "
|
| 97 |
+
f"或 service_account_info。project_id={project_id}"
|
| 98 |
+
)
|
| 99 |
+
|
| 100 |
+
if credential_path:
|
| 101 |
+
key_path = Path(credential_path)
|
| 102 |
+
if not key_path.is_absolute():
|
| 103 |
+
key_path = BASE_DIR / key_path
|
| 104 |
+
credential_path = str(key_path.resolve())
|
| 105 |
+
|
| 106 |
+
normalized.append({
|
| 107 |
+
"account_name": item.get("account_name") or item.get("name") or f"vertex-{idx}",
|
| 108 |
+
"project_id": project_id,
|
| 109 |
+
"location": (item.get("location") or "global").strip(),
|
| 110 |
+
"service_account_key": credential_path,
|
| 111 |
+
"service_account_info": service_account_info if isinstance(service_account_info, dict) else None,
|
| 112 |
+
})
|
| 113 |
+
|
| 114 |
+
return normalized
|
| 115 |
+
|
| 116 |
+
# ════════════════════════════════════════════════════════════════════════
|
| 117 |
+
# LLM API 配置 — 切换 API_MODE 即可一键切换
|
| 118 |
+
# ════════════════════════════════════════════════════════════════════════
|
| 119 |
+
# "proxy" = OpenAI-compatible 中转站 → Gemini
|
| 120 |
+
# "vertex" = Google Vertex AI 原生 API (global 区域)
|
| 121 |
+
API_MODE = "vertex"
|
| 122 |
+
|
| 123 |
+
BBOX_CONFIGS = {
|
| 124 |
+
"gemini": {"order": "yxyx", "range": "permille"},
|
| 125 |
+
"default": {"order": "xyxy", "range": "norm"},
|
| 126 |
+
}
|
| 127 |
+
|
| 128 |
+
if API_MODE == "proxy":
|
| 129 |
+
API_ENDPOINTS = [
|
| 130 |
+
"http://35.220.164.252:3888/v1/chat/completions",
|
| 131 |
+
]
|
| 132 |
+
API_KEY = os.environ.get("VIDEO_DR_PROXY_API_KEY", os.environ.get("OPENAI_API_KEY", ""))
|
| 133 |
+
DEFAULT_MODEL = "gemini-2.5-pro"
|
| 134 |
+
|
| 135 |
+
elif API_MODE == "vertex":
|
| 136 |
+
# ── Vertex JSON 池 ──
|
| 137 |
+
# 一个条目就是一个可轮询的 project。
|
| 138 |
+
# 同一个 Vertex 账号下如果有多个 project，就放多条记录。
|
| 139 |
+
VERTEX_ACCOUNT_POOL_FILE = os.environ.get(
|
| 140 |
+
"VERTEX_ACCOUNT_POOL_FILE",
|
| 141 |
+
str(BASE_DIR / "vertex_account_pool.json"),
|
| 142 |
+
)
|
| 143 |
+
|
| 144 |
+
# 兼容旧配置：如果没有 JSON 池文件，仍然退回到单账号/单 project 逻辑。
|
| 145 |
+
GCP_PROJECT_POOL = [
|
| 146 |
+
"my-project-3-492910",
|
| 147 |
+
# "my-project-5-492910",
|
| 148 |
+
# "my-project-4-492910",
|
| 149 |
+
# "my-project-3-492910"
|
| 150 |
+
# "project-d1c7810d-d567-4008-bac",
|
| 151 |
+
# "stellar-psyche-492807-f5",
|
| 152 |
+
# "astral-pipe-492807-a1", # CONSUMER_SUSPENDED
|
| 153 |
+
# "my-project-1-492715", # My Project 1
|
| 154 |
+
# "my-project-2-492715", # My Project 2
|
| 155 |
+
# "my-project-3-492715", # My Project 3
|
| 156 |
+
# "my-project-4-492715", # My Project 4
|
| 157 |
+
# "my-project-5-492715", # My Project 5
|
| 158 |
+
# "jovial-lamp-492715-k4", # My Project 6
|
| 159 |
+
# "project-ff9a405b-c0ca-4387-bb8", # My First Project
|
| 160 |
+
# "steadfast-helix-492713-c6", # My Project
|
| 161 |
+
]
|
| 162 |
+
|
| 163 |
+
GCP_PROJECT_ID = GCP_PROJECT_POOL[0] # 默认使用第一个（兼容旧代码）
|
| 164 |
+
GCP_LOCATION = "global"
|
| 165 |
+
GCP_SERVICE_ACCOUNT_KEY = "" # 留空则使用 ADC
|
| 166 |
+
|
| 167 |
+
# global 区域必须使用原生 generateContent API
|
| 168 |
+
USE_VERTEX_NATIVE_API = True
|
| 169 |
+
|
| 170 |
+
def _make_vertex_url(project_id: str, location: str):
|
| 171 |
+
native = (
|
| 172 |
+
f"https://aiplatform.googleapis.com/v1/projects/{project_id}"
|
| 173 |
+
f"/locations/{location}/publishers/google/models"
|
| 174 |
+
)
|
| 175 |
+
openai = (
|
| 176 |
+
f"https://{location}-aiplatform.googleapis.com/v1beta1"
|
| 177 |
+
f"/projects/{project_id}/locations/{location}"
|
| 178 |
+
f"/endpoints/openapi/chat/completions"
|
| 179 |
+
)
|
| 180 |
+
return native if USE_VERTEX_NATIVE_API else openai
|
| 181 |
+
|
| 182 |
+
_vertex_pool_records = _load_vertex_account_pool(VERTEX_ACCOUNT_POOL_FILE)
|
| 183 |
+
VERTEX_CREDENTIALS_POOL: List[Dict] = []
|
| 184 |
+
|
| 185 |
+
if _vertex_pool_records:
|
| 186 |
+
for record in _vertex_pool_records:
|
| 187 |
+
api_url = _make_vertex_url(record["project_id"], record["location"])
|
| 188 |
+
VERTEX_CREDENTIALS_POOL.append({
|
| 189 |
+
**record,
|
| 190 |
+
"api_url": api_url,
|
| 191 |
+
})
|
| 192 |
+
API_ENDPOINTS = [item["api_url"] for item in VERTEX_CREDENTIALS_POOL]
|
| 193 |
+
GCP_PROJECT_ID = VERTEX_CREDENTIALS_POOL[0]["project_id"]
|
| 194 |
+
GCP_LOCATION = VERTEX_CREDENTIALS_POOL[0]["location"]
|
| 195 |
+
else:
|
| 196 |
+
VERTEX_CREDENTIALS_POOL = []
|
| 197 |
+
API_ENDPOINTS = [_make_vertex_url(pid, GCP_LOCATION) for pid in GCP_PROJECT_POOL]
|
| 198 |
+
|
| 199 |
+
API_KEY = "" # Vertex AI 认证由 LLMClient 中的 GCP OAuth2 自动处理
|
| 200 |
+
DEFAULT_MODEL = "gemini-3.1-pro-preview"
|
| 201 |
+
|
| 202 |
+
# ── Input Data ──
|
| 203 |
+
INPUT_JSONL = str(
|
| 204 |
+
LOCAL_DATA_ROOT / "inference_logs_image_v5-1.video_qa_results_v2.merged_gap2.topk_span_top3.jsonl"
|
| 205 |
+
)
|
| 206 |
+
|
| 207 |
+
# ════════════════════════════════════════════════════════════════════════
|
| 208 |
+
# Search Services
|
| 209 |
+
# ════════════════════════════════════════════════════════════════════════
|
| 210 |
+
|
| 211 |
+
MOCK_SEARCH = False # True = 无网络时用 mock 搜索
|
| 212 |
+
|
| 213 |
+
# ── Web Search ──
|
| 214 |
+
WEB_SEARCH_ADDRESS = os.environ.get("WEB_SEARCH_ADDRESS", "")
|
| 215 |
+
WEB_SEARCH_CONFIG = {
|
| 216 |
+
"llm_model": "qwen3-32b",
|
| 217 |
+
"prompt_type": "mmsearch_r1",
|
| 218 |
+
"use_jina": False,
|
| 219 |
+
"enable_thinking": False,
|
| 220 |
+
}
|
| 221 |
+
|
| 222 |
+
# ── MARS: Retrieval + Summarizer ──
|
| 223 |
+
MARS_RETRIEVAL_ADDRESS = os.environ.get("MARS_RETRIEVAL_ADDRESS", "")
|
| 224 |
+
MARS_SUMMARIZER_ADDRESS = os.environ.get("MARS_SUMMARIZER_ADDRESS", "")
|
| 225 |
+
MARS_RETRIEVAL_TOPK = 5
|
| 226 |
+
MARS_RETRIEVAL_TIMEOUT = 120 # retrieval 单次请求超时 (秒)
|
| 227 |
+
MARS_RETRIEVAL_CONCURRENCY = 1 # retrieval 并发锁 (0=不限制), GPU Faiss 不支持并发
|
| 228 |
+
MARS_SUMMARIZER_MODEL = os.environ.get(
|
| 229 |
+
"MARS_SUMMARIZER_MODEL",
|
| 230 |
+
str(LOCAL_DATA_ROOT / "models" / "Qwen-Qwen3.5-27B"),
|
| 231 |
+
)
|
| 232 |
+
MARS_WEB_SEARCH_MODE = "mars" # "mars" = retrieve+summarize; "serper" = legacy Serper
|
| 233 |
+
|
| 234 |
+
# ════════════════════════════════════════════════════════════════════════
|
| 235 |
+
# Image Search 配置 — 切换 IMAGE_SEARCH_MODE 即可
|
| 236 |
+
# ════════════════════════════════════════════════════════════════════════
|
| 237 |
+
# "serper" = 直连 Serper Google Lens
|
| 238 |
+
# "gateway" = 公司内部网关 → Serper Google Lens
|
| 239 |
+
IMAGE_SEARCH_MODE = os.environ.get("IMAGE_SEARCH_MODE", "gateway")
|
| 240 |
+
|
| 241 |
+
# Serper 直连 (IMAGE_SEARCH_MODE = "serper")
|
| 242 |
+
SERPER_API_KEY = os.environ.get("SERPER_API_KEY", "")
|
| 243 |
+
|
| 244 |
+
# 公司内部网关 (IMAGE_SEARCH_MODE = "gateway")
|
| 245 |
+
GATEWAY_URL = os.environ.get("GATEWAY_URL", "")
|
| 246 |
+
GATEWAY_USERNAME = os.environ.get("GATEWAY_USERNAME", "")
|
| 247 |
+
GATEWAY_USERID = os.environ.get("GATEWAY_USERID", "")
|
| 248 |
+
GATEWAY_TOKEN = os.environ.get("GATEWAY_TOKEN", "")
|
| 249 |
+
|
| 250 |
+
# ── Image Search Summarizer ──
|
| 251 |
+
IMAGE_SEARCH_SUMMARIZE_SERPER = True
|
| 252 |
+
IMAGE_SEARCH_SUMMARIZER_ADDRESS = MARS_SUMMARIZER_ADDRESS
|
| 253 |
+
IMAGE_SEARCH_SUMMARIZER_MODEL = MARS_SUMMARIZER_MODEL
|
| 254 |
+
IMAGE_SEARCH_SUMMARIZER_MAX_RESULTS = 7
|
| 255 |
+
IMAGE_SEARCH_SUMMARIZER_MAX_TOKENS = 1024
|
| 256 |
+
|
| 257 |
+
# ── Image Search Cache ──
|
| 258 |
+
IMAGE_SEARCH_CACHE_FILE = "image_search_cache.json"
|
| 259 |
+
|
| 260 |
+
# ── Image Search Fallback ──
|
| 261 |
+
# Serper Lens 对 data URI / base64 支持不稳定，默认禁用该回退路径。
|
| 262 |
+
IMAGE_SEARCH_ALLOW_BASE64_FALLBACK = False
|
| 263 |
+
|
| 264 |
+
# ════════════════════════════════════════════════════════════════════════
|
| 265 |
+
# OSS Image Upload (Serper Lens 不支持 base64, 需上传 OSS 获取公网 URL)
|
| 266 |
+
# ════════════════════════════════════════════════════════════════════════
|
| 267 |
+
OSS_ACCESS_KEY_ID = os.environ.get("OSS_ACCESS_KEY_ID", "")
|
| 268 |
+
OSS_ACCESS_KEY_SECRET = os.environ.get("OSS_ACCESS_KEY_SECRET", "")
|
| 269 |
+
OSS_ENDPOINT = os.environ.get("OSS_ENDPOINT", "oss-cn-shanghai.aliyuncs.com")
|
| 270 |
+
OSS_BUCKET_NAME = os.environ.get("OSS_BUCKET_NAME", "")
|
| 271 |
+
OSS_UPLOAD_PREFIX = "video_dr_search_crops"
|
| 272 |
+
|
| 273 |
+
SEARCH_CROP_MAX_SIZE = 500 # 搜索用裁剪图最大边长 (px)
|
| 274 |
+
SEARCH_CROP_JPEG_QUALITY = 80 # 搜索用裁剪图 JPEG 质量
|
| 275 |
+
|
| 276 |
+
# ════════════════════════════════════════════════════════════════════════
|
| 277 |
+
# LLM Parameters
|
| 278 |
+
# ════════════════════════════════════════════════════════════════════════
|
| 279 |
+
DEFAULT_TEMPERATURE = 0.8
|
| 280 |
+
DEFAULT_TOP_P = 0.95
|
| 281 |
+
DEFAULT_MAX_TOKENS = 6144
|
| 282 |
+
DEFAULT_TIMEOUT = 600
|
| 283 |
+
DEFAULT_MAX_RETRIES = 5
|
| 284 |
+
DEFAULT_CONCURRENCY = 1
|
| 285 |
+
|
| 286 |
+
# ── Vertex AI Rate Shaping ──
|
| 287 |
+
# 共享池更容易被秒级突发打出 429；这里默认做温和整形。
|
| 288 |
+
# 0 = 关闭节流。
|
| 289 |
+
VERTEX_MIN_REQUEST_INTERVAL_SECONDS = 12.0
|
| 290 |
+
VERTEX_RATE_LIMIT_COOLDOWN_SECONDS = 180.0
|
| 291 |
+
VERTEX_REQUEST_JITTER_SECONDS = 1.0
|
| 292 |
+
|
| 293 |
+
# ── Frame Extraction ──
|
| 294 |
+
DEFAULT_MAX_FRAMES = 64
|
| 295 |
+
DEFAULT_INTERVAL_SAMPLES = 8
|
| 296 |
+
DEFAULT_MAX_RESOLUTION = 768
|
| 297 |
+
DEFAULT_JPEG_QUALITY = 85
|
| 298 |
+
|
| 299 |
+
# ── Phase 2 Limits ──
|
| 300 |
+
PHASE2_MAX_IMAGE_SEARCH = 2
|
| 301 |
+
PHASE2_MAX_WEB_SEARCH = 2
|
| 302 |
+
PHASE2_MAX_TURNS = 6
|
| 303 |
+
|
| 304 |
+
# ════════════════════════════════════════════════════════════════════════
|
| 305 |
+
# Token Pricing (per 1M tokens, USD)
|
| 306 |
+
# ════════════════════════════════════════════════════════════════════════
|
| 307 |
+
TOKEN_PRICING = {
|
| 308 |
+
"gemini-2.5-flash": (0.15, 0.60),
|
| 309 |
+
"gemini-2.5-flash-lite": (0.0990, 0.3960),
|
| 310 |
+
"gemini-2.5-pro": (1.25, 10.00),
|
| 311 |
+
"gemini-2.0-flash": (0.10, 0.40),
|
| 312 |
+
"gemini-3.1-flash-lite-preview": (0.075, 0.30),
|
| 313 |
+
"gemini-3.1-pro-preview": (2, 12),
|
| 314 |
+
"gemini-2.5-flash-thinking": (0.2970, 2.4750),
|
| 315 |
+
"gpt-4o": (2.50, 10.00),
|
| 316 |
+
"gpt-4o-mini": (0.15, 0.60),
|
| 317 |
+
"Kimi-K2.5": (0.0, 0.0),
|
| 318 |
+
}
|
| 319 |
+
|
| 320 |
+
THINKING_TOKEN_PRICING = {
|
| 321 |
+
"gemini-2.5-flash": 3.50,
|
| 322 |
+
"gemini-2.5-pro": 10.00,
|
| 323 |
+
"gemini-3.1-pro-preview": 12,
|
| 324 |
+
}
|
| 325 |
+
|
| 326 |
+
DEFAULT_INPUT_PRICE_PER_1M = 0.10
|
| 327 |
+
DEFAULT_OUTPUT_PRICE_PER_1M = 0.40
|
| 328 |
+
|
| 329 |
+
for _local_service in [
|
| 330 |
+
WEB_SEARCH_ADDRESS,
|
| 331 |
+
MARS_RETRIEVAL_ADDRESS,
|
| 332 |
+
MARS_SUMMARIZER_ADDRESS,
|
| 333 |
+
]:
|
| 334 |
+
_append_no_proxy_entry(_local_service)
|
video_dr_gen/prompts.py
ADDED
|
@@ -0,0 +1,541 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""
|
| 3 |
+
Video DeepResearch — Prompt Templates (v2.2 — Anti-Distillation Safe, Lean Constraints)
|
| 4 |
+
===================================================================================
|
| 5 |
+
Phase 1: Candidate selection + think generation
|
| 6 |
+
Phase 2: Interactive tool-use prompts
|
| 7 |
+
|
| 8 |
+
Key updates:
|
| 9 |
+
- Removed all mentions of "generating training data" or "trajectory generation" to prevent triggering the LLM's internal anti-distillation safety filters.
|
| 10 |
+
- Changed persona to "advanced video reasoning AI".
|
| 11 |
+
- Softened strict sentence counts to avoid "structured CoT" refusals.
|
| 12 |
+
- Removed Hidden Ground Truth (GT) in Phase 2 to prevent Data Contamination / Leakage during trajectory generation.
|
| 13 |
+
"""
|
| 14 |
+
|
| 15 |
+
# ════════════════════════════════════════════════════════════════════════
|
| 16 |
+
# Base System Prompt (shared role + video context, NO tools)
|
| 17 |
+
# ════════════════════════════════════════════════════════════════════════
|
| 18 |
+
|
| 19 |
+
SYSTEM_PROMPT_BASE = """# Role
|
| 20 |
+
You are an advanced Video DeepResearch reasoning assistant.
|
| 21 |
+
Given a user query that requires combining visual clues from a video with external knowledge, your task is to solve the problem step-by-step by deeply navigating the video and searching the web.
|
| 22 |
+
|
| 23 |
+
# Video Context
|
| 24 |
+
- The original video has been converted to 1 frame per second (1 fps).
|
| 25 |
+
- The original frame index (e.g., Frame 1, Frame 10) is watermarked in the top-left corner of every frame.
|
| 26 |
+
- You are initially provided with 64 uniformly sampled frames from the video."""
|
| 27 |
+
|
| 28 |
+
|
| 29 |
+
# ════════════════════════════════════════════════════════════════════════
|
| 30 |
+
# Phase 1 System Prompt (only navigation tools)
|
| 31 |
+
# ════════════════════════════════════════════════════════════════════════
|
| 32 |
+
|
| 33 |
+
PHASE1_SYSTEM_PROMPT = SYSTEM_PROMPT_BASE + """
|
| 34 |
+
|
| 35 |
+
# Available Tools (Phase 1 — Frame Navigation ONLY)
|
| 36 |
+
You may call one tool per turn to navigate the video. Tool definitions are provided as JSON Schema within <tools></tools> tags:
|
| 37 |
+
<tools>
|
| 38 |
+
{"type":"function","function":{"name":"choose_frames","description":"Select a specific time interval to investigate further. The system will return uniformly sampled frames from within this interval. Use this when the answer lies within a specific segment.","parameters":{"type":"object","properties":{"start_frame_index":{"type":"integer","description":"The starting frame index of the interval."},"end_frame_index":{"type":"integer","description":"The ending frame index of the interval."}},"required":["start_frame_index","end_frame_index"]},"strict":false}}
|
| 39 |
+
{"type":"function","function":{"name":"find_frame","description":"Lock onto a specific single frame index to examine it closely. Returns the exact frame. You MUST call this after choose_frames before any detailed analysis.","parameters":{"type":"object","properties":{"frame_index":{"type":"integer","description":"The exact frame index to lock onto."}},"required":["frame_index"]},"strict":false}}
|
| 40 |
+
</tools>
|
| 41 |
+
|
| 42 |
+
# Tool Dependency & Workflow Rules (Phase 1)
|
| 43 |
+
1. You can call `find_frame` directly if the target is obvious in the initial 64 sparse frames.
|
| 44 |
+
2. Otherwise, call `choose_frames` first to narrow down the search interval, then call `find_frame` on a specific frame within that interval.
|
| 45 |
+
3. After `find_frame`, the frame is locked and you will proceed to Phase 2 (zoom/search tools).
|
| 46 |
+
|
| 47 |
+
# Thinking Requirements (STRICT)
|
| 48 |
+
Your <think> block at each turn MUST contain natural flowing prose (NOT bullet labels, NOT numbered lists). Describe:
|
| 49 |
+
- What you actually observe in the currently visible frames (specific visual details: colors, objects, text, scene changes).
|
| 50 |
+
- Why a particular time segment or frame stands out as relevant to the question.
|
| 51 |
+
- Your reasoning for choosing the next tool call.
|
| 52 |
+
|
| 53 |
+
# Output Format (STRICT)
|
| 54 |
+
At each turn, output MUST follow this exact structure:
|
| 55 |
+
<think>
|
| 56 |
+
Your reasoning in natural prose.
|
| 57 |
+
</think>
|
| 58 |
+
<tool_call>
|
| 59 |
+
{"name": "<function-name>", "arguments": <args-json-object>}
|
| 60 |
+
</tool_call>
|
| 61 |
+
|
| 62 |
+
# HARD CONSTRAINTS (must follow exactly):
|
| 63 |
+
1) Every turn MUST begin with exactly one non-empty `<think>...</think>` block.
|
| 64 |
+
2) The `<think>` text MUST be natural prose. Blank, placeholder, list-only, or garbled content is a violation.
|
| 65 |
+
3) After `<think>`, output exactly one `<tool_call>...</tool_call>` block.
|
| 66 |
+
4) NEVER output `<answer>` in Phase 1. NEVER output tools not listed above (zoom_in, image_search, web_search are NOT available yet).
|
| 67 |
+
5) NEVER output text outside of `<think>` and `<tool_call>` tags.
|
| 68 |
+
6) NEVER generate `<tool_response>` — tool responses come from the system, not from you.
|
| 69 |
+
7) Output EXACTLY ONE `<tool_call>` per turn. After your `</tool_call>`, STOP. Do not continue writing.
|
| 70 |
+
8) If you detect you are about to violate any constraint, self-correct immediately within the current output."""
|
| 71 |
+
|
| 72 |
+
|
| 73 |
+
# ════════════════════════════════════════════════════════════════════════
|
| 74 |
+
# Phase 2 System Prompt (only inspection/search tools)
|
| 75 |
+
# ════════════════════════════════════════════════════════════════════════
|
| 76 |
+
|
| 77 |
+
PHASE2_SYSTEM_PROMPT = SYSTEM_PROMPT_BASE + """
|
| 78 |
+
|
| 79 |
+
# Current State
|
| 80 |
+
You have already navigated through the video using choose_frames and find_frame. You are now examining the locked frame closely.
|
| 81 |
+
Continue your investigation using zoom_in, image_search, or web_search to answer the question.
|
| 82 |
+
|
| 83 |
+
# Available Tools (Phase 2 — Inspection & Search ONLY)
|
| 84 |
+
You may call one tool per turn to inspect or search. Tool definitions are provided as JSON Schema within <tools></tools> tags:
|
| 85 |
+
<tools>
|
| 86 |
+
{"type":"function","function":{"name":"zoom_in","description":"Magnify a specific region of the locked frame for detailed visual analysis. Use this to get a clearer view before searching.","parameters":{"type":"object","properties":{"bbox":{"type":"array","items":{"type":"number"},"minItems":4,"maxItems":4,"description":"Normalized bounding box [x1, y1, x2, y2] where 0.0 <= x1 < x2 <= 1.0 and 0.0 <= y1 < y2 <= 1.0."}},"required":["bbox"]},"strict":false}}
|
| 87 |
+
{"type":"function","function":{"name":"image_search","description":"Reverse image search a specific region of the locked frame to identify unknown entities. Use this when you DO NOT know what the entity is.","parameters":{"type":"object","properties":{"bbox":{"type":"array","items":{"type":"number"},"minItems":4,"maxItems":4,"description":"Normalized bounding box [x1, y1, x2, y2] to crop and search."}},"required":["bbox"]},"strict":false}}
|
| 88 |
+
{"type":"function","function":{"name":"web_search","description":"Search the web using a text query. Use this when you ALREADY RECOGNIZE the entity and need external facts or knowledge.","parameters":{"type":"object","properties":{"query":{"type":"string","description":"Text query to search the web."}},"required":["query"]},"strict":false}}
|
| 89 |
+
</tools>
|
| 90 |
+
|
| 91 |
+
# Tool Strategy Rules
|
| 92 |
+
1. If you visually locate an entity but do NOT know its name/identity → use `image_search(bbox)`.
|
| 93 |
+
2. If you visually locate and recognize the entity → bypass image search and directly use `web_search(query)`.
|
| 94 |
+
3. You may `zoom_in` first to get a clearer view before searching.
|
| 95 |
+
4. NEVER call `choose_frames` or `find_frame` — those are Phase 1 tools and are no longer available.
|
| 96 |
+
|
| 97 |
+
# Image Search Retry Strategy (CRITICAL)
|
| 98 |
+
- If `image_search` returns results that clearly do NOT match the entity in the frame (e.g., different people, unrelated objects), DO NOT immediately fall back to `web_search`.
|
| 99 |
+
- Instead, RETRY `image_search` with a DIFFERENT bbox:
|
| 100 |
+
* Try a TIGHTER crop focused more precisely on the entity (e.g., face only, logo only).
|
| 101 |
+
* Try a WIDER crop that includes more context (e.g., upper body + background, full sign).
|
| 102 |
+
* Try a SHIFTED crop if the entity was near the edge of the previous bbox.
|
| 103 |
+
- You should attempt at least 2 different image_search bboxes before giving up on image_search.
|
| 104 |
+
- NEVER use `web_search` with physical appearance descriptions (e.g., "man in black jacket with white stripes") to identify an unknown person — this strategy has an extremely low success rate and wastes your search budget.
|
| 105 |
+
- `web_search` should ONLY be used when you have a concrete name, term, or identifier to search for.
|
| 106 |
+
|
| 107 |
+
# Image Search Persistence Rule (CRITICAL)
|
| 108 |
+
- CRITICAL: Do NOT abandon image_search after just ONE attempt. You MUST try at least 2-3 different bbox strategies before switching to web_search:
|
| 109 |
+
* First try: region crop focused on the most distinctive feature.
|
| 110 |
+
* Second try: full-frame or near-full-frame search (sometimes broader context helps matching).
|
| 111 |
+
* Third try: a completely different visual element (e.g., if the building top failed, try the base or a different angle).
|
| 112 |
+
- After exhausting image_search retries, if you switch to web_search, your query MUST be built from OBJECTIVE visual features you observed — NOT from assumptions derived from logos, watermarks, or other metadata.
|
| 113 |
+
|
| 114 |
+
# Failed Search Protocol
|
| 115 |
+
When a search returns no useful results:
|
| 116 |
+
1. Do NOT output <answer> immediately — try at least one different approach first.
|
| 117 |
+
2. Reformulate your query with contextual keywords you ALREADY observed, switch between image_search and web_search, or zoom_in on a different region.
|
| 118 |
+
3. If all reasonable strategies are exhausted with no supporting evidence, answer with "UNABLE_TO_DETERMINE".
|
| 119 |
+
|
| 120 |
+
# Web Search Query Construction Rules (STRICT)
|
| 121 |
+
- Your web_search queries MUST be constructed from OBJECTIVE, VISUALLY-OBSERVED features:
|
| 122 |
+
* Architectural features: "concrete cylindrical tower flared observation deck"
|
| 123 |
+
* Material/construction: "reinforced concrete television tower 1960s construction"
|
| 124 |
+
* Scale/context: "tallest tower construction newsreel"
|
| 125 |
+
* Visible text: text you actually read from the frame (signs, banners)
|
| 126 |
+
|
| 127 |
+
# Thinking Requirements (STRICT)
|
| 128 |
+
Your <think> block at each turn MUST contain natural flowing prose (NOT bullet labels, NOT numbered lists). Cover three aspects organically woven into your narrative:
|
| 129 |
+
- **Evidence**: Describe what you actually observe in the current frame or zoomed region — colors, shapes, text, objects, scenes. Be specific and grounded in visual content.
|
| 130 |
+
- **Integration**: Connect this observation with information from previous turns (tool results, earlier observations). State how it narrows down or confirms your hypothesis.
|
| 131 |
+
- **Self-reflection**: Assess whether you have enough information to answer, or whether you need another tool call. If a search returned unhelpful results, note what to try differently.
|
| 132 |
+
|
| 133 |
+
# Anti-Redundancy Rule (STRICT)
|
| 134 |
+
- Do NOT re-describe visual details from previous turns. You have context memory — assume you already know what you observed before.
|
| 135 |
+
- On turn 2+, your <think> should START with what is NEW: the tool result you just received, or a new observation from a zoomed region. Do NOT repeat the base frame description.
|
| 136 |
+
- BAD example: "I see the frame shows a man in a black suit at a podium with logos behind him. [repeats entire previous turn's observation]..."
|
| 137 |
+
- GOOD example: "The zoomed region reveals the logo text reads 'ACME Corp'. Combined with the press conference setting, I should search for..."
|
| 138 |
+
- Each turn's <think> must advance the reasoning. If you find yourself writing something you already wrote in a previous turn, STOP and skip to the new information.
|
| 139 |
+
|
| 140 |
+
# Output Format (STRICT)
|
| 141 |
+
At each turn, output MUST follow one of these two structures:
|
| 142 |
+
|
| 143 |
+
**If reasoning continues (tool call):**
|
| 144 |
+
<think>
|
| 145 |
+
Your reasoning in natural prose.
|
| 146 |
+
</think>
|
| 147 |
+
<tool_call>
|
| 148 |
+
{"name": "<function-name>", "arguments": <args-json-object>}
|
| 149 |
+
</tool_call>
|
| 150 |
+
|
| 151 |
+
**If ready to conclude (final answer):**
|
| 152 |
+
<think>
|
| 153 |
+
Summarize all reasoning, visual clues, and search results that led to your answer.
|
| 154 |
+
</think>
|
| 155 |
+
<answer>
|
| 156 |
+
Short entity name or fact (NOT a full sentence).
|
| 157 |
+
</answer>
|
| 158 |
+
|
| 159 |
+
# HARD CONSTRAINTS (must follow exactly — every turn):
|
| 160 |
+
1) Every turn MUST begin with exactly one non-empty `<think>...</think>` block — no exceptions.
|
| 161 |
+
2) The `<think>` text MUST be natural prose. Blank, placeholder, list-only, or garbled content is a violation.
|
| 162 |
+
3) After `<think>`, output EXACTLY ONE of: `<tool_call>...</tool_call>` OR `<answer>...</answer>` — never both, never neither, never more than one.
|
| 163 |
+
4) NEVER output text outside of `<think>`, `<tool_call>`, and `<answer>` tags. No commentary, no "MODERATION:", no "---", no explanations between tags.
|
| 164 |
+
5) NEVER generate `<tool_response>` content — tool responses come from the system, not from you. Generating fake tool responses is strictly forbidden.
|
| 165 |
+
6) Output EXACTLY ONE action per turn. After your `</tool_call>` or `</answer>`, STOP IMMEDIATELY. Do not continue writing anything else.
|
| 166 |
+
7) Your final `<answer>` must be a SHORT entity name or fact, not a full sentence.
|
| 167 |
+
8) You must reason as if you are genuinely discovering the answer step by step. DO NOT jump to conclusions, guess, or state facts you haven't visually observed or retrieved via tools.
|
| 168 |
+
9) If you detect you are about to violate any constraint, self-correct immediately — do not start over.
|
| 169 |
+
|
| 170 |
+
# Anti-Hallucination Rule
|
| 171 |
+
You are in a multi-turn conversation. At each turn, you output your reasoning and ONE action, then STOP.
|
| 172 |
+
The system will execute your tool call and return the result. You MUST wait for the system's response — do NOT fabricate tool results.
|
| 173 |
+
Your visual perception is constant across turns — if you did not mention a detail in earlier turns, you cannot suddenly "discover" it after a failed search.
|
| 174 |
+
|
| 175 |
+
# Think Block Structure (STRICT)
|
| 176 |
+
In your <think> block, you MUST follow this order:
|
| 177 |
+
1. FIRST: Describe what you actually observe in the current frame or zoomed region (visual features, colors, objects, text). This is your PRIMARY evidence.
|
| 178 |
+
2. THEN: If relevant, naturally weave in clues from your earlier video navigation to guide your search strategy. Write as if these clues are part of your own earlier observations — because they are.
|
| 179 |
+
3. FINALLY: Decide your next action based on the combination of both.
|
| 180 |
+
You MUST NOT skip step 1 and jump directly to conclusions based solely on earlier observations. Current visual evidence always comes first.
|
| 181 |
+
|
| 182 |
+
# Example of a Good <think> Block
|
| 183 |
+
Here is an example of proper reasoning flow for a Phase 2 turn:
|
| 184 |
+
<think>
|
| 185 |
+
The locked frame shows a wide shot of a turquoise lake nestled among steep, snow-covered mountains under an overcast sky. In the lower portion of the frame, I can see evergreen trees and a text overlay that reads "Peyto Lake!" in large white font. During my earlier navigation, I noticed this video featured driving sequences on the Icefields Parkway through Banff National Park. The question asks about the outflow of this lake, which I cannot determine from the visual alone. Since I now know the lake's name from the on-screen text, I should search for information about its hydrology.
|
| 186 |
+
</think>"""
|
| 187 |
+
|
| 188 |
+
|
| 189 |
+
|
| 190 |
+
def get_phase2_system_prompt(bbox_config: dict = None) -> str:
|
| 191 |
+
"""根据 bbox 格式配置动态生成 Phase 2 system prompt。
|
| 192 |
+
|
| 193 |
+
如果 bbox_config 为 None 或为 xyxy_norm，返回原始 PHASE2_SYSTEM_PROMPT。
|
| 194 |
+
如果为 yxyx_permille（Gemini），替换 tool definition 中的 bbox 格式描述。
|
| 195 |
+
|
| 196 |
+
Args:
|
| 197 |
+
bbox_config: get_bbox_config() 返回的配置字典，
|
| 198 |
+
例如 {"order": "yxyx", "range": "permille"}
|
| 199 |
+
|
| 200 |
+
Returns:
|
| 201 |
+
完整的 Phase 2 system prompt 字符串
|
| 202 |
+
"""
|
| 203 |
+
prompt = PHASE2_SYSTEM_PROMPT
|
| 204 |
+
|
| 205 |
+
if bbox_config and bbox_config.get("order") == "yxyx":
|
| 206 |
+
# ── Gemini 风格: [y_min, x_min, y_max, x_max] in 0-1000 ──
|
| 207 |
+
|
| 208 |
+
# 替换 zoom_in 的 bbox 描述
|
| 209 |
+
prompt = prompt.replace(
|
| 210 |
+
'"Normalized bounding box [x1, y1, x2, y2] where 0.0 <= x1 < x2 <= 1.0 and 0.0 <= y1 < y2 <= 1.0."',
|
| 211 |
+
'"Bounding box [y_min, x_min, y_max, x_max] where coordinates are integers normalized to 0-1000 range."'
|
| 212 |
+
)
|
| 213 |
+
# 替换 image_search 的 bbox 描述
|
| 214 |
+
prompt = prompt.replace(
|
| 215 |
+
'"Normalized bounding box [x1, y1, x2, y2] to crop and search."',
|
| 216 |
+
'"Bounding box [y_min, x_min, y_max, x_max] (integers in 0-1000 range) to crop and search."'
|
| 217 |
+
)
|
| 218 |
+
# 替换 items type: Gemini 输出整数
|
| 219 |
+
prompt = prompt.replace(
|
| 220 |
+
'"items":{"type":"number"}',
|
| 221 |
+
'"items":{"type":"integer"}'
|
| 222 |
+
)
|
| 223 |
+
|
| 224 |
+
return prompt
|
| 225 |
+
|
| 226 |
+
|
| 227 |
+
# ════════════════════════════════════════════════════════════════════════
|
| 228 |
+
# Phase 1: Candidate Selection Prompt
|
| 229 |
+
# ════════════════════════════════════════════════════════════════════════
|
| 230 |
+
|
| 231 |
+
PHASE1_SELECTION_PROMPT = """You are an advanced Video DeepResearch reasoning assistant.
|
| 232 |
+
## Context
|
| 233 |
+
You are shown {num_frames} uniformly sampled frames from a 1fps video.
|
| 234 |
+
Each frame has a "Frame X" watermark in the top-left corner.
|
| 235 |
+
The user's question is: "{question}"
|
| 236 |
+
The ground truth answer is: "{answer}"
|
| 237 |
+
The entity to find is: "{entity}"
|
| 238 |
+
|
| 239 |
+
## Available Sparse Frame Indices
|
| 240 |
+
The frames you are currently viewing have the following indices (read the watermarks):
|
| 241 |
+
{sparse_frames_text}
|
| 242 |
+
|
| 243 |
+
## Your Task
|
| 244 |
+
Identify 1 or 2 short intervals in the video that are most relevant to answering the question.
|
| 245 |
+
You must define each interval by selecting a START and END frame from the sparse frame list above.
|
| 246 |
+
The interval must span at most 3 consecutive sparse frames (i.e., start and end can be at most 2 steps apart in the sparse frame list).
|
| 247 |
+
Do NOT try to pick a specific frame — you are only selecting which interval(s) to zoom into.
|
| 248 |
+
The specific frame will be chosen later after examining the interval at higher density.
|
| 249 |
+
|
| 250 |
+
### CRITICAL STRATEGY FOR "WHO IS" / IDENTITY QUESTIONS:
|
| 251 |
+
If the question asks you to identify a person or entity performing a specific action (e.g., "Who gets knocked down at the end?"), DO NOT simply select the interval where the action occurs. High-motion actions (fighting, running, falling) are usually blurry. Instead:
|
| 252 |
+
1. First, use ALL available clues in the question — visual attributes (clothing, build, hair), described actions, spatial relationships, and temporal references — to CONFIRM which person or entity in the video is the target across the sparse frames.
|
| 253 |
+
2. Then, scan ALL frames to find where that EXACT SAME TARGET's face or distinguishing features are SHARPEST and CLEAREST — often in close-ups, at the very beginning of the video, or during moments of rest.
|
| 254 |
+
3. Select the interval that provides the HIGHEST QUALITY visual features of THE TARGET for recognition or reverse image search, even if that interval is far away from where the action takes place.
|
| 255 |
+
WARNING: Do NOT select a frame just because it has a clear face or entity — make sure it is the CORRECT target matching the question's description.
|
| 256 |
+
|
| 257 |
+
**When to select 2 intervals:**
|
| 258 |
+
- When the question genuinely requires information from two different scenes (e.g., comparing events at different times, tracking changes across the video).
|
| 259 |
+
- When identifying a person/object involved in a fast-moving action: Select one interval to locate the action (to confirm WHO it is), and a SECOND interval where that same person/object is standing still, facing the camera, or clearly visible (to actually identify them).
|
| 260 |
+
|
| 261 |
+
## Output Format
|
| 262 |
+
Output ONLY a valid JSON object (no markdown, no explanation):
|
| 263 |
+
{{
|
| 264 |
+
"selections": [
|
| 265 |
+
{{
|
| 266 |
+
"start_frame": <a frame index from the sparse frame list>,
|
| 267 |
+
"end_frame": <a frame index from the sparse frame list, at most 2 steps after start_frame>,
|
| 268 |
+
"reasoning": "<briefly explain why this interval has the CLEAREST view of the entity>"
|
| 269 |
+
}}
|
| 270 |
+
]
|
| 271 |
+
}}"""
|
| 272 |
+
|
| 273 |
+
|
| 274 |
+
# ════════════════════════════════════════════════════════════════════════
|
| 275 |
+
# Phase 1: Think Generation Prompts
|
| 276 |
+
# ════════════════════════════════════════════════════════════════════════
|
| 277 |
+
|
| 278 |
+
THINK_VIDEO_OVERVIEW_PROMPT = """You are an advanced video reasoning AI.
|
| 279 |
+
You are shown {num_frames} uniformly sampled frames from a video (1 fps).
|
| 280 |
+
Each frame has a "Frame X" watermark in the top-left corner.
|
| 281 |
+
The user's question is: "{question}"
|
| 282 |
+
|
| 283 |
+
Provide a concise overview of the ENTIRE video based on what you see across all frames:
|
| 284 |
+
1. Describe the general setting and context (e.g., a press conference, a sports match, a cooking show, a documentary, etc.).
|
| 285 |
+
2. Actively extract and transcribe any LARGE, PROMINENT text visible in the frames — such as title cards, banners, marquee signs, opening/closing credits, large on-screen captions, channel logos, or overlay text. These are critical global clues.
|
| 286 |
+
3. Note key people, objects, logos, or other visual landmarks. Specifically, mention WHICH frames provide the clearest, highest-resolution close-ups of the main subjects' faces or distinctive clothing.
|
| 287 |
+
4. Mention any notable transitions, scene changes, or timeline shifts.
|
| 288 |
+
|
| 289 |
+
HARD CONSTRAINTS:
|
| 290 |
+
- Output ONLY natural prose text. No XML tags, no tool calls, no JSON, no bullet labels.
|
| 291 |
+
- Your output must be strictly 3 to 5 sentences of genuine visual analysis. Do NOT exceed 5 sentences.
|
| 292 |
+
- Reference ONLY the visual content you can actually see in the provided frames.
|
| 293 |
+
- Sound like you're naturally surveying the video before focusing in.
|
| 294 |
+
- Do NOT mention being told where to look or having ground truth information.
|
| 295 |
+
- Do NOT conclude which specific segment or frame range is "most relevant" or "best" for answering the question — that narrowing-down analysis will happen in a separate reasoning step immediately after this overview.
|
| 296 |
+
- Do NOT bind detailed descriptions of specific content to individual frame numbers (e.g., do NOT write "Frame 62 shows a castle with cylindrical towers"). Instead, describe what you see across general regions of the video (e.g., "the middle portion of the video features historical architecture").
|
| 297 |
+
- Do NOT leave the output blank or output placeholders.
|
| 298 |
+
- You MUST read and transcribe any large, clearly legible text (title cards, banners, prominent signs, credits). These are NOT "small text" — they are key contextual anchors.
|
| 299 |
+
- Do NOT attempt to read tiny, blurry, or low-resolution text (e.g., small print on badges, lanyards, jerseys, license plates). For those, simply note their presence. Detailed reading of small text will happen in a later zoom-in phase."""
|
| 300 |
+
|
| 301 |
+
|
| 302 |
+
|
| 303 |
+
THINK_INITIAL_PROMPT = """You are an advanced video reasoning AI.
|
| 304 |
+
You are shown {num_frames} uniformly sampled frames from a video (1 fps).
|
| 305 |
+
Each frame has a "Frame X" watermark in the top-left corner.
|
| 306 |
+
The user's question is: "{question}"
|
| 307 |
+
|
| 308 |
+
The visual clue needed to answer this question is located in the segment between Frame {interval_start} and Frame {interval_end}.
|
| 309 |
+
You are about to call choose_frames(start_frame_index={interval_start}, end_frame_index={interval_end}) to narrow down your search.
|
| 310 |
+
|
| 311 |
+
Write down your internal thought process as natural flowing prose explaining:
|
| 312 |
+
1. What you observe in the currently visible frames that are closest to this segment.
|
| 313 |
+
2. Why this general time segment seems worth investigating with a higher frame rate.
|
| 314 |
+
|
| 315 |
+
HARD CONSTRAINTS:
|
| 316 |
+
- Output ONLY natural prose text. No XML tags, no tool calls, no JSON, no bullet labels.
|
| 317 |
+
- Reference ONLY the visual content you can actually see in the provided sparse frames.
|
| 318 |
+
- Sound like you're narrowing down the area naturally through visual analysis.
|
| 319 |
+
- Do NOT mention being told where to look or having ground truth information.
|
| 320 |
+
- Do NOT leave the output blank or output placeholders.
|
| 321 |
+
- Do NOT provide a general overview or summary of the entire video — jump directly into analyzing the specific frames near the target segment.
|
| 322 |
+
- You MAY note and transcribe large, prominent text visible at this resolution (title cards, banners, large signs, credits). These are valuable global clues.
|
| 323 |
+
- Do NOT attempt to read tiny, blurry text (badges, lanyards, small print on jerseys, license plates). For those, simply note their presence. Detailed reading will happen after zooming in."""
|
| 324 |
+
|
| 325 |
+
|
| 326 |
+
THINK_INTERVAL_PROMPT = """You are continuing your video investigation.
|
| 327 |
+
You narrowed your search to frames [{start} - {end}] and now see {num_frames} frames from this interval.
|
| 328 |
+
The user's question is: "{question}"
|
| 329 |
+
|
| 330 |
+
You need to select ONE specific frame from this interval that is best for detailed examination (zoom_in / image_search).
|
| 331 |
+
|
| 332 |
+
## Frame Selection Criteria (BOTH matter equally):
|
| 333 |
+
1. **Entity Clarity**: The target entity should be clearly visible and reasonably sharp.
|
| 334 |
+
2. **Contextual Richness**: The frame should contain rich environmental clues — visible text (banners, signs, titles, credits), logos, distinctive backgrounds, or other identifiable landmarks.
|
| 335 |
+
|
| 336 |
+
A frame with a clear entity AND visible background text/logos is FAR MORE VALUABLE than a pure close-up with a blank or blurred background. Do NOT default to face close-ups if other frames offer richer context.
|
| 337 |
+
|
| 338 |
+
Generate your response in EXACTLY this format (no extra text):
|
| 339 |
+
BEST_FRAME: <frame number>
|
| 340 |
+
REASONING: <natural prose explaining what you observe in these interval frames and why the chosen frame is the best candidate for closer examination>
|
| 341 |
+
|
| 342 |
+
HARD CONSTRAINTS:
|
| 343 |
+
- You MUST choose from the frames you can actually see (check the "Frame X" watermarks).
|
| 344 |
+
- REASONING must be natural prose, NOT bullet points.
|
| 345 |
+
- Your reasoning should explain WHY this frame is the best CANDIDATE — considering BOTH entity visibility AND environmental context richness.
|
| 346 |
+
- Do NOT describe the chosen frame as if you have already examined it in fine detail or already identified the entities within it. You are making a SELECTION decision, not reporting findings from a completed investigation.
|
| 347 |
+
- Sound like genuine visual analysis at the interval-frame resolution level — you are choosing where to look next, not summarizing what you already found.
|
| 348 |
+
- Do NOT output XML tags, tool calls, or JSON.
|
| 349 |
+
- You MAY note the presence of large, prominent text (banners, title cards, signs) as a reason for choosing a frame — these are valuable contextual anchors.
|
| 350 |
+
- Do NOT attempt to read tiny, blurry text (badges, lanyards, small print). Simply note which frames show these elements most prominently for a future zoom-in."""
|
| 351 |
+
|
| 352 |
+
|
| 353 |
+
THINK_ITERATIVE_INTERVAL_PROMPT = """You are continuing your video investigation.
|
| 354 |
+
You narrowed your search to frames [{start} - {end}] and now see {num_frames} frames from this interval.
|
| 355 |
+
The user's question is: "{question}"
|
| 356 |
+
You have {remaining_iterations} more zoom iteration(s) remaining after this one.
|
| 357 |
+
|
| 358 |
+
You have two options:
|
| 359 |
+
1. **Lock onto a frame**: If you can already identify a specific frame that is clear enough for detailed examination (zoom_in / image_search), output BEST_FRAME.
|
| 360 |
+
2. **Zoom further**: If the target entity is visible but not yet clear enough, or you need to narrow down further, output NEED_ZOOM with a tighter sub-interval.
|
| 361 |
+
|
| 362 |
+
You may choose a BEST_FRAME from ANY frame you have seen in this or previous zoom rounds — not just the current set.
|
| 363 |
+
|
| 364 |
+
## Frame Selection Criteria (BOTH matter equally):
|
| 365 |
+
1. **Entity Clarity**: The target entity should be clearly visible and reasonably sharp.
|
| 366 |
+
2. **Contextual Richness**: The frame should contain rich environmental clues — visible text (banners, signs, titles, credits), logos, distinctive backgrounds, or other identifiable landmarks.
|
| 367 |
+
|
| 368 |
+
A frame with a clear entity AND visible background text/logos is FAR MORE VALUABLE than a pure close-up with a blank or blurred background. Do NOT default to face close-ups if other frames offer richer context.
|
| 369 |
+
|
| 370 |
+
## Available Sparse Frames for NEED_ZOOM
|
| 371 |
+
If you choose to zoom further, you must select a sub-interval from these sparse frame indices:
|
| 372 |
+
{available_sparse_text}
|
| 373 |
+
The sub-interval must span at most 3 consecutive sparse frames from this list.
|
| 374 |
+
|
| 375 |
+
Generate your response in EXACTLY one of these two formats (no extra text):
|
| 376 |
+
|
| 377 |
+
**Option 1 — Lock onto a frame:**
|
| 378 |
+
BEST_FRAME: <frame number>
|
| 379 |
+
REASONING: <natural prose explaining what you observe and why this frame is the best for closer examination>
|
| 380 |
+
|
| 381 |
+
**Option 2 — Zoom further:**
|
| 382 |
+
NEED_ZOOM: <start_frame> <end_frame>
|
| 383 |
+
REASONING: <natural prose explaining why you need to zoom further and what you expect to find in this sub-interval>
|
| 384 |
+
|
| 385 |
+
HARD CONSTRAINTS:
|
| 386 |
+
- You MUST choose from the frames you can actually see (check the "Frame X" watermarks).
|
| 387 |
+
- For BEST_FRAME, you may pick any frame you have seen across all zoom rounds.
|
| 388 |
+
- For NEED_ZOOM, start_frame and end_frame must be from the sparse frame list above, at most 2 steps apart.
|
| 389 |
+
- REASONING must be natural prose, NOT bullet points.
|
| 390 |
+
- Do NOT describe the chosen frame as if you have already examined it in fine detail. You are making a SELECTION decision, not reporting findings.
|
| 391 |
+
- Do NOT output XML tags, tool calls, or JSON.
|
| 392 |
+
- You MAY note large, prominent text (banners, title cards, signs) as a reason for choosing.
|
| 393 |
+
- Do NOT attempt to read tiny, blurry text (badges, lanyards, small print). Simply note which frames show these elements most prominently for a future zoom-in."""
|
| 394 |
+
|
| 395 |
+
|
| 396 |
+
THINK_DIRECT_FIND_PROMPT = """You are an advanced video reasoning AI.
|
| 397 |
+
You are shown {num_frames} uniformly sampled frames from a video (1 fps).
|
| 398 |
+
The user's question is: "{question}"
|
| 399 |
+
|
| 400 |
+
The visual clue is within a small segment [Frame {start} - Frame {end}] ({interval_len} frames).
|
| 401 |
+
You need to pick ONE frame from this segment to lock onto for detailed examination.
|
| 402 |
+
The segment is small enough that you should be able to identify the best frame directly.
|
| 403 |
+
|
| 404 |
+
Generate your response in EXACTLY this format (no extra text):
|
| 405 |
+
BEST_FRAME: <frame number from {start} to {end}>
|
| 406 |
+
REASONING: <natural prose explaining your observation and why this frame is best>
|
| 407 |
+
|
| 408 |
+
HARD CONSTRAINTS:
|
| 409 |
+
- Pick a frame within [{start}, {end}].
|
| 410 |
+
- REASONING must be natural prose, NOT bullet points.
|
| 411 |
+
- Reference actual visual content to justify your choice.
|
| 412 |
+
- Sound like genuine visual analysis.
|
| 413 |
+
- Do NOT output XML tags, tool calls, or JSON.
|
| 414 |
+
- You MAY note large, prominent text visible at sparse-frame resolution (banners, title cards, signs). These are valuable contextual anchors.
|
| 415 |
+
- Do NOT attempt to read tiny, blurry text (badges, lanyards, small print). Simply describe which frame shows the relevant element most clearly for a future zoom-in. The actual reading will happen in the next phase."""
|
| 416 |
+
|
| 417 |
+
THINK_LOCATE_INTENT_PROMPT = """You are an advanced video reasoning AI.
|
| 418 |
+
You are shown {num_frames} uniformly sampled frames from a video (1 fps).
|
| 419 |
+
Each frame has a "Frame X" watermark in the top-left corner.
|
| 420 |
+
The user's question is: "{question}"
|
| 421 |
+
|
| 422 |
+
You have decided to lock onto Frame {best_frame} for detailed examination.
|
| 423 |
+
You are about to call find_frame(frame_index={best_frame}).
|
| 424 |
+
|
| 425 |
+
Write down your internal thought process explaining:
|
| 426 |
+
1. What coarse visual cues in the nearby sparse frames led you to focus on this area of the video (e.g., you noticed a building silhouette, a person, a scene transition, relevant on-screen text, or a change in setting near this frame).
|
| 427 |
+
2. Why Frame {best_frame} specifically seems like the best candidate for closer inspection (e.g., it appears to be the clearest, most centered, or most relevant frame in this segment).
|
| 428 |
+
|
| 429 |
+
HARD CONSTRAINTS:
|
| 430 |
+
- Do NOT describe the detailed content of Frame {best_frame} as if you have already examined it in high detail — you are selecting it for FUTURE detailed examination.
|
| 431 |
+
- Your reasoning should be at the SPARSE-FRAME level: mention shapes, silhouettes, scene types, and general visual patterns — NOT fine-grained details like specific architectural features, facial features, text content, or object identities.
|
| 432 |
+
- Sound like you are making a hypothesis or narrowing your search, NOT reporting what you already found.
|
| 433 |
+
- For example, GOOD: "Around Frame {best_frame}, I can see what appears to be a historical structure against a town backdrop — this looks like it could be relevant to the question."
|
| 434 |
+
- For example, BAD: "This frame provides a clear aerial view of the distinctive medieval castle with its cylindrical towers, set against the backdrop of the town and mountains."
|
| 435 |
+
- Output ONLY natural prose text. No XML tags, no tool calls, no JSON, no bullet labels.
|
| 436 |
+
- Do NOT mention being told where to look or having ground truth information.
|
| 437 |
+
- Do NOT leave the output blank or output placeholders.
|
| 438 |
+
- Do NOT provide a general overview or summary of the entire video — focus only on the specific area around Frame {best_frame}. Do NOT repeat or paraphrase any video overview that was described earlier. Start directly with your reasoning about Frame {best_frame} and its surrounding context.
|
| 439 |
+
- You MAY note and transcribe large, prominent text visible at this resolution (banners, title cards, large signs). These are valuable clues.
|
| 440 |
+
- Do NOT attempt to read tiny, blurry text (badges, lanyards, small print). For those, simply note their presence and location. Detailed reading will happen after locking onto the frame."""
|
| 441 |
+
|
| 442 |
+
|
| 443 |
+
THINK_SECOND_INTERVAL_PROMPT = """You are continuing your video investigation.
|
| 444 |
+
You previously locked onto Frame {prev_frame} and examined it closely. Here is that frame again for reference.
|
| 445 |
+
The user's question is: "{question}"
|
| 446 |
+
|
| 447 |
+
Now you need to investigate a second segment between Frame {interval_start} and Frame {interval_end}.
|
| 448 |
+
You are about to call choose_frames(start_frame_index={interval_start}, end_frame_index={interval_end}) to examine this additional segment.
|
| 449 |
+
|
| 450 |
+
Write down your internal thought process explaining:
|
| 451 |
+
1. What you observed in Frame {prev_frame} — describe its actual visual content (objects, setting, text, etc.).
|
| 452 |
+
2. Why Frame {prev_frame} alone is insufficient to answer the question (what information is missing or unclear).
|
| 453 |
+
3. Why this second segment might provide the additional context you need.
|
| 454 |
+
|
| 455 |
+
HARD CONSTRAINTS:
|
| 456 |
+
- Output ONLY natural prose text. No XML tags, no tool calls, no JSON, no bullet labels.
|
| 457 |
+
- You MUST begin by describing what you actually see in the previous locked frame.
|
| 458 |
+
- Sound natural, building a logical chain: "I saw X in Frame {prev_frame}, but I still need Y, and this segment might show Z."
|
| 459 |
+
- Do NOT leave the output blank or output placeholders.
|
| 460 |
+
- You MAY note large, prominent text (banners, title cards, signs). Do NOT attempt to read tiny, blurry text. Simply note its presence."""
|
| 461 |
+
|
| 462 |
+
|
| 463 |
+
THINK_TRANSITION_FIND_PROMPT = """You are continuing your video investigation.
|
| 464 |
+
You previously locked onto Frame {prev_frame} and examined it closely. Here is that frame again for reference.
|
| 465 |
+
The user's question is: "{question}"
|
| 466 |
+
|
| 467 |
+
After examining Frame {prev_frame}, you now need to lock onto Frame {best_frame} for additional investigation.
|
| 468 |
+
|
| 469 |
+
Write down your internal thought process explaining:
|
| 470 |
+
1. What you observed in Frame {prev_frame} — describe its actual visual content.
|
| 471 |
+
2. Why Frame {prev_frame} is insufficient to fully answer the question.
|
| 472 |
+
3. Why Frame {best_frame} might provide better or complementary information based on what you see in the sparse frames nearby.
|
| 473 |
+
|
| 474 |
+
HARD CONSTRAINTS:
|
| 475 |
+
- Output ONLY natural prose text. No XML tags, no tool calls, no JSON, no bullet labels.
|
| 476 |
+
- You MUST begin by describing what you actually see in the previous locked frame.
|
| 477 |
+
- Sound like genuine visual analysis: "In the previous frame I can see X, however Y is unclear, so I want to examine Frame {best_frame} where Z appears..."
|
| 478 |
+
- Do NOT describe Frame {best_frame}'s detailed content as if you have already examined it — you are selecting it for FUTURE examination.
|
| 479 |
+
- Do NOT leave the output blank or output placeholders.
|
| 480 |
+
- You MAY note large, prominent text (banners, title cards, signs). Do NOT attempt to read tiny, blurry text. Simply note its presence."""
|
| 481 |
+
|
| 482 |
+
|
| 483 |
+
# ════════════════════════════════════════════════════════════════════════
|
| 484 |
+
# Phase 2: Initial User Prompt
|
| 485 |
+
# ════════════════════════════════════════════════════════════════════════
|
| 486 |
+
|
| 487 |
+
PHASE2_USER_PROMPT = """You have navigated through the video and locked onto Frame {frame_idx}.
|
| 488 |
+
Here is the frame:
|
| 489 |
+
|
| 490 |
+
{global_overview_block}Question: {question}
|
| 491 |
+
|
| 492 |
+
During your earlier navigation through the video, you identified this frame as the most relevant to the question. Now examine it in detail to find the answer.
|
| 493 |
+
|
| 494 |
+
Strategy:
|
| 495 |
+
1. First, describe what you observe in the frame. Identify regions of interest relevant to the question. Note any text, logos, or entities that may need closer inspection.
|
| 496 |
+
2. Use zoom_in to get a clearer view of a specific region — especially if there is text or a logo that was too small to read clearly during navigation.
|
| 497 |
+
3. After zooming in, if you can now read text or recognize an entity, decide your next step based on what you've learned.
|
| 498 |
+
4. If you see an entity but do NOT know its name/identity, use image_search on that region.
|
| 499 |
+
5. If you recognize the entity and need external facts, use web_search directly.
|
| 500 |
+
6. You have at most {max_image_searches} image_search attempts and {max_web_searches} web_search attempts.
|
| 501 |
+
|
| 502 |
+
## Cross-Dimensional Search Strategy (IMPORTANT)
|
| 503 |
+
- After describing the current frame's visual details, consider any relevant clues you recall from your earlier video navigation.
|
| 504 |
+
- If image_search returns irrelevant results (e.g., generic products, unrelated people), do NOT keep retrying image_search with minor bbox variations. Instead, SWITCH to web_search immediately.
|
| 505 |
+
- Your web_search query should combine: visual features from the current frame + specific nouns/names you observed during earlier navigation. For example: if you earlier noticed "magnet theatre" text in the video and the frame shows a woman in a classical costume, search "magnet theatre" or "magnet theatre production".
|
| 506 |
+
- Text clues observed during video navigation (theater names, show titles, production company logos) are often the fastest path to the answer.
|
| 507 |
+
|
| 508 |
+
REMINDER: Output exactly <think>...</think> then exactly ONE <tool_call>...</tool_call>.
|
| 509 |
+
Then STOP and wait for the system to return the tool result.
|
| 510 |
+
Do NOT generate the tool result yourself.
|
| 511 |
+
|
| 512 |
+
Use the <think> block to document your visual observations and logical deductions in natural prose. Begin your <think> by describing what you see in the frame, then naturally integrate any relevant clues from your earlier video navigation.
|
| 513 |
+
IMPORTANT: In your <think> block, do NOT mention search attempt counts, remaining searches, or search budget constraints (e.g., "I have used all my web searches", "I have one image search left"). Focus purely on your visual observations and reasoning."""
|
| 514 |
+
|
| 515 |
+
|
| 516 |
+
|
| 517 |
+
PHASE2_CONTEXT_REMINDER = """[Turn {current_turn}/{max_turns} | Image searches: {image_search_count}/{max_image_searches} | Web searches: {web_search_count}/{max_web_searches} | Tools used so far: {tools_used}]
|
| 518 |
+
|
| 519 |
+
Remember:
|
| 520 |
+
- Your <think> must be natural prose covering evidence, integration with prior findings, and self-reflection.
|
| 521 |
+
- START by summarizing what you learned from the previous tool result — this is the NEW information.
|
| 522 |
+
- Do NOT repeat observations from previous turns — build on them. Assume you already know the base frame content.
|
| 523 |
+
- If previous searches were unhelpful, try a different strategy (different bbox, different query, etc.).
|
| 524 |
+
- In your <think>, do NOT mention search counts, remaining attempts, or budget constraints. Focus on evidence and reasoning.
|
| 525 |
+
- Output EXACTLY <think>...</think> then ONE <tool_call> or <answer>. Then STOP."""
|
| 526 |
+
|
| 527 |
+
# ════════════════════════════════════════════════════════════════════════
|
| 528 |
+
# Phase 2: Force Answer Prompt (when search budget exhausted)
|
| 529 |
+
# ════════════════════════════════════════════════════════════════════════
|
| 530 |
+
|
| 531 |
+
PHASE2_FORCE_ANSWER = """You MUST provide your final answer NOW based on all information gathered so far.
|
| 532 |
+
|
| 533 |
+
CRITICAL: Your answer MUST be supported by concrete evidence from your tool results (image_search or web_search). If NONE of your searches returned information that directly identifies or confirms the answer, you MUST answer with "UNABLE_TO_DETERMINE" — do NOT guess based on visual appearance alone or intuition. An unsupported guess is worse than admitting uncertainty.
|
| 534 |
+
|
| 535 |
+
Output EXACTLY this structure and NOTHING else:
|
| 536 |
+
<think>
|
| 537 |
+
Summarize all evidence gathered across your investigation — what you observed visually, what searches revealed, and how these combine to answer the question. If no search provided useful evidence, state this honestly. Do NOT mention search attempt counts or budget limits.
|
| 538 |
+
</think>
|
| 539 |
+
<answer>
|
| 540 |
+
Your final short answer (entity name or fact, NOT a full sentence). If no search evidence supports an answer, output: UNABLE_TO_DETERMINE
|
| 541 |
+
</answer>"""
|
video_dr_gen/utils.py
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|