Add Farm04 Segment packaging guide
Browse filesDirect HF download, LoRA merge, official Docker build/test/save, and parity gate instructions for Tr006 V2.
pipelines/Tr006_V2/FARM04_PACKAGING_GUIDE.md
ADDED
|
@@ -0,0 +1,138 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Farm04:Tr006 V2 Segment Docker 打包指南
|
| 2 |
+
|
| 3 |
+
## 1. 拉取权重和 pipeline
|
| 4 |
+
|
| 5 |
+
```bash
|
| 6 |
+
HF_ROOT="$PWD/orena_tr006_v2"
|
| 7 |
+
|
| 8 |
+
hf download Gbone3176/interval \
|
| 9 |
+
--revision main \
|
| 10 |
+
--include 'weights/Tr006_V2/**' \
|
| 11 |
+
--include 'pipelines/Tr006_V2/**' \
|
| 12 |
+
--local-dir "$HF_ROOT" \
|
| 13 |
+
--max-workers 1
|
| 14 |
+
|
| 15 |
+
PIPELINE="$HF_ROOT/pipelines/Tr006_V2"
|
| 16 |
+
ADAPTER="$HF_ROOT/weights/Tr006_V2/checkpoint-6013"
|
| 17 |
+
```
|
| 18 |
+
|
| 19 |
+
检查关键文件:
|
| 20 |
+
|
| 21 |
+
```bash
|
| 22 |
+
test -f "$ADAPTER/adapter_model.safetensors"
|
| 23 |
+
test -f "$PIPELINE/project/scripts/merge_lora_model.py"
|
| 24 |
+
test -f "$PIPELINE/docker/segment-algorithm/inference.py"
|
| 25 |
+
test -f "$PIPELINE/manifest/pipeline_manifest.json"
|
| 26 |
+
```
|
| 27 |
+
|
| 28 |
+
## 2. 准备官方 Segment 目录和 base model
|
| 29 |
+
|
| 30 |
+
使用一份全新的官方 `segment-algorithm` 目录,不要复用含旧模型的构建目录。
|
| 31 |
+
|
| 32 |
+
```bash
|
| 33 |
+
OFFICIAL_SEGMENT_DIR=/absolute/path/to/official/segment-algorithm
|
| 34 |
+
BASE_MODEL=/absolute/path/to/Qwen3.5-9B
|
| 35 |
+
PYTHON_BIN=/absolute/path/to/python
|
| 36 |
+
|
| 37 |
+
test -f "$OFFICIAL_SEGMENT_DIR/do_build.sh"
|
| 38 |
+
test -f "$OFFICIAL_SEGMENT_DIR/do_test_run.sh"
|
| 39 |
+
test -f "$OFFICIAL_SEGMENT_DIR/do_save.sh"
|
| 40 |
+
test -f "$BASE_MODEL/config.json"
|
| 41 |
+
```
|
| 42 |
+
|
| 43 |
+
Merge 环境应与 Tr006 记录一致:`transformers==5.15.0`、`peft==0.19.1`、`safetensors==0.8.0`。
|
| 44 |
+
|
| 45 |
+
## 3. Merge checkpoint-6013
|
| 46 |
+
|
| 47 |
+
`resources/model` 应当不存在或为空;不要覆盖无法确认来源的旧模型。
|
| 48 |
+
|
| 49 |
+
```bash
|
| 50 |
+
"$PYTHON_BIN" "$PIPELINE/project/scripts/merge_lora_model.py" \
|
| 51 |
+
--base "$BASE_MODEL" \
|
| 52 |
+
--adapter "$ADAPTER" \
|
| 53 |
+
--output "$OFFICIAL_SEGMENT_DIR/resources/model" \
|
| 54 |
+
--max-shard-size 4GB
|
| 55 |
+
```
|
| 56 |
+
|
| 57 |
+
检查 merge:
|
| 58 |
+
|
| 59 |
+
```bash
|
| 60 |
+
"$PYTHON_BIN" - "$OFFICIAL_SEGMENT_DIR/resources/model/merge_report.json" <<'PY'
|
| 61 |
+
import json
|
| 62 |
+
import pathlib
|
| 63 |
+
import sys
|
| 64 |
+
|
| 65 |
+
report = json.loads(pathlib.Path(sys.argv[1]).read_text())
|
| 66 |
+
assert report["remaining_lora_modules_after_merge"] == 0
|
| 67 |
+
assert all(
|
| 68 |
+
item["abs_delta_sum"] > 0
|
| 69 |
+
for item in report["representative_weight_deltas"].values()
|
| 70 |
+
)
|
| 71 |
+
print("merge validation: PASS")
|
| 72 |
+
PY
|
| 73 |
+
|
| 74 |
+
test -f "$OFFICIAL_SEGMENT_DIR/resources/model/model.safetensors.index.json"
|
| 75 |
+
test -f "$OFFICIAL_SEGMENT_DIR/resources/model/processor_config.json"
|
| 76 |
+
```
|
| 77 |
+
|
| 78 |
+
## 4. 覆盖 Docker 推理文件
|
| 79 |
+
|
| 80 |
+
保留官方 `do_build.sh`、`do_test_run.sh`、`do_save.sh`,仅覆盖当前模型所需的三个文件:
|
| 81 |
+
|
| 82 |
+
```bash
|
| 83 |
+
cp "$PIPELINE/docker/segment-algorithm/inference.py" \
|
| 84 |
+
"$OFFICIAL_SEGMENT_DIR/inference.py"
|
| 85 |
+
cp "$PIPELINE/docker/segment-algorithm/requirements.txt" \
|
| 86 |
+
"$OFFICIAL_SEGMENT_DIR/requirements.txt"
|
| 87 |
+
cp "$PIPELINE/docker/segment-algorithm/Dockerfile" \
|
| 88 |
+
"$OFFICIAL_SEGMENT_DIR/Dockerfile"
|
| 89 |
+
```
|
| 90 |
+
|
| 91 |
+
## 5. 官方脚本构建和 GPU 测试
|
| 92 |
+
|
| 93 |
+
```bash
|
| 94 |
+
cd "$OFFICIAL_SEGMENT_DIR"
|
| 95 |
+
bash do_build.sh
|
| 96 |
+
docker image inspect segment-algorithm --format '{{.Id}} {{.Size}}'
|
| 97 |
+
nvidia-smi
|
| 98 |
+
bash do_test_run.sh
|
| 99 |
+
```
|
| 100 |
+
|
| 101 |
+
必须确认:
|
| 102 |
+
|
| 103 |
+
- 容器能看到 GPU,模型实际加载为 `Qwen3_5ForConditionalGeneration`。
|
| 104 |
+
- 一整个官方测试 batch 生成 `/output/answer.json`。
|
| 105 |
+
- 回答数量与请求数量相同,无异常退出、空答案或重复 ID。
|
| 106 |
+
- 日志显示每个问题的处理耗时满足官方预算。
|
| 107 |
+
|
| 108 |
+
## 6. 推理一致性门禁
|
| 109 |
+
|
| 110 |
+
当前 `docker/segment-algorithm/inference.py` 是低分线上提交的事实快照,不是已经修正后的 V2 对齐实现。重新提交前,必须在公开样本上对比:
|
| 111 |
+
|
| 112 |
+
1. V2 训练侧选择的 80 个绝对 source PTS。
|
| 113 |
+
2. 420x224 letterbox 后的帧内容。
|
| 114 |
+
3. 蓝屏区间是否被排除。
|
| 115 |
+
4. `video_metadata.frames_indices`、`video_grid_thw` 和输入 token。
|
| 116 |
+
5. 同一 checkpoint 在本地训练/评测链与 Docker Transformers 链中的最终答案。
|
| 117 |
+
|
| 118 |
+
若这些项目不一致,本次构建只能用于复现旧提交,不能视为修复版。
|
| 119 |
+
|
| 120 |
+
## 7. 保存官方 tar.gz
|
| 121 |
+
|
| 122 |
+
一致性测试通过后再执行:
|
| 123 |
+
|
| 124 |
+
```bash
|
| 125 |
+
cd "$OFFICIAL_SEGMENT_DIR"
|
| 126 |
+
bash do_save.sh
|
| 127 |
+
|
| 128 |
+
ARCHIVE=$(find "$OFFICIAL_SEGMENT_DIR" -maxdepth 1 \
|
| 129 |
+
-type f -name 'segment-algorithm_*.tar.gz' -printf '%T@ %p\n' \
|
| 130 |
+
| sort -nr | head -1 | cut -d' ' -f2-)
|
| 131 |
+
|
| 132 |
+
test -n "$ARCHIVE"
|
| 133 |
+
gzip -t "$ARCHIVE"
|
| 134 |
+
ls -lh "$ARCHIVE"
|
| 135 |
+
sha256sum "$ARCHIVE"
|
| 136 |
+
```
|
| 137 |
+
|
| 138 |
+
网页端提交 `do_save.sh` 生成并通过 `gzip -t` 的 `segment-algorithm_*.tar.gz`。
|
pipelines/Tr006_V2/README.md
CHANGED
|
@@ -4,6 +4,9 @@
|
|
| 4 |
`pipelines/Tr006_V2/`。权重位于同仓库的
|
| 5 |
`weights/Tr006_V2/checkpoint-6013/`。
|
| 6 |
|
|
|
|
|
|
|
|
|
|
| 7 |
## 范围
|
| 8 |
|
| 9 |
包含 Tr006 V2 实际使用的完整处理链:
|
|
|
|
| 4 |
`pipelines/Tr006_V2/`。权重位于同仓库的
|
| 5 |
`weights/Tr006_V2/checkpoint-6013/`。
|
| 6 |
|
| 7 |
+
Farm04 上只做拉取、merge、Docker 构建和 GPU 测试时,直接阅读
|
| 8 |
+
`FARM04_PACKAGING_GUIDE.md`。
|
| 9 |
+
|
| 10 |
## 范围
|
| 11 |
|
| 12 |
包含 Tr006 V2 实际使用的完整处理链:
|