Spaces:
Running
Running
sync processor from GitLab 2bbf9fe: UMI SLAM 动作模块、批次指纹与设备进度、导出/路由/前端更新
Browse files- processor/README.md +49 -47
- processor/app/api/worker.py +84 -26
- processor/app/batch_fingerprint.py +194 -0
- processor/app/device_progress.py +218 -0
- processor/app/lerobot_export.py +341 -25
- processor/app/localstore.py +269 -35
- processor/app/processing/modules/lerobot_export.py +20 -2
- processor/app/processing/modules/umi_slam_action.py +192 -0
- processor/app/project_dataset.py +203 -18
- processor/app/routes/devices.py +55 -8
- processor/app/routes/export.py +9 -2
- processor/app/routes/session.py +38 -0
- processor/app/routes/video.py +196 -34
- processor/app/umi_slam_action.py +269 -0
- processor/tests/test_batch_fingerprint.py +181 -0
- processor/tests/test_device_task_progress.py +181 -0
- processor/tests/test_lerobot_export.py +105 -0
- processor/tests/test_tactile_decode.py +146 -1
- processor/tests/test_umi_slam_action.py +369 -0
- processor/web/static/js/player.js +72 -98
- processor/web/templates/index.html +1 -1
processor/README.md
CHANGED
|
@@ -1,3 +1,5 @@
|
|
|
|
|
|
|
|
| 1 |
# processor
|
| 2 |
|
| 3 |
[English](README.md) | [中文](README.zh-CN.md)
|
|
@@ -159,12 +161,12 @@ the current Episode.
|
|
| 159 |
|
| 160 |
### Video sources
|
| 161 |
|
| 162 |
-
| Type
|
| 163 |
-
| --- | --- | --- |
|
| 164 |
-
| Mono RGB
|
| 165 |
-
| Metric depth | `head_depth`
|
| 166 |
-
| Stereo RGB
|
| 167 |
-
| Stereo depth | `stereo_depth`
|
| 168 |
|
| 169 |
RGB and Depth are always stored as independent streams. A depth-only stream is never treated as
|
| 170 |
RGB and is never duplicated alongside the RGB stream.
|
|
@@ -250,24 +252,24 @@ when old workflows are loaded; they are not shown in the new-workflow palette.
|
|
| 250 |
|
| 251 |
#### Capture modules (Input)
|
| 252 |
|
| 253 |
-
| Frontend module
|
| 254 |
-
| --- | --- | --- | --- |
|
| 255 |
-
| `RGB Camera`
|
| 256 |
-
| `RGB-D Camera`
|
| 257 |
-
| `Stereo RGB Camera`
|
| 258 |
-
| `Stereo RGB-D Camera` | None
|
| 259 |
-
| `Glove Sensor`
|
| 260 |
|
| 261 |
#### Video and keypoint processing modules (Process)
|
| 262 |
|
| 263 |
-
| Frontend module
|
| 264 |
-
| --- | --- | --- | --- |
|
| 265 |
-
| `Human Annotation`
|
| 266 |
-
| `AI Annotation`
|
| 267 |
-
| `RGB_TO_2D_BareHand`
|
| 268 |
-
| `RGB_TO_2D_BlackGlove` | `RGB Video`
|
| 269 |
-
| `RGB-D_3D_BareHand`
|
| 270 |
-
| `RGB-D_3D_BlackGlove`
|
| 271 |
|
| 272 |
`MediaPipe Hand` remains available for historical workflows and backend compatibility but is hidden
|
| 273 |
from the new-workflow palette. All nodes provide hover text describing their purpose and accepted
|
|
@@ -275,32 +277,32 @@ connection types.
|
|
| 275 |
|
| 276 |
#### Review modules (Review)
|
| 277 |
|
| 278 |
-
| Frontend module
|
| 279 |
-
| --- | --- | --- | --- |
|
| 280 |
-
| `Human Review`
|
| 281 |
| `AI Quality Review` | `Quality Review Target` | `Reviewed Data` | Checks video decoding, frame continuity, black screens, freezes, annotation coverage, and sensor data. |
|
| 282 |
|
| 283 |
#### Export modules (Export)
|
| 284 |
|
| 285 |
-
| Frontend module
|
| 286 |
-
| --- | --- | --- | --- |
|
| 287 |
| `LeRobot Export` | `Exportable Data` | `Dataset` | Exports LeRobot v2.1 or v3.0; the version is selected on the node. |
|
| 288 |
-
| `HDF5 Export`
|
| 289 |
|
| 290 |
#### Workflow Studio UI components
|
| 291 |
|
| 292 |
<details>
|
| 293 |
<summary>Click to expand: frontend component reference</summary>
|
| 294 |
|
| 295 |
-
| UI component
|
| 296 |
-
| --- | --- |
|
| 297 |
-
| `NodePalette`
|
| 298 |
-
| `WorkflowCanvas`
|
| 299 |
-
| `WorkflowNode`
|
| 300 |
-
| `WorkflowDrawer`
|
| 301 |
-
| `PipelineToolbar`
|
| 302 |
-
| `NodeSettingsModal` | Configures AI Annotation API vendor, model, endpoint, and key fields.
|
| 303 |
-
| `DeletableEdge`
|
| 304 |
|
| 305 |
Device selection is always scoped to the current project. A new empty workflow does not display a
|
| 306 |
device from another project. After data is uploaded, each card's selector shows only real sources
|
|
@@ -434,17 +436,17 @@ The build output is written to `web/static/workflow-studio/` and served by FastA
|
|
| 434 |
Configuration is stored in `processor/.env` or environment variables and must not be
|
| 435 |
committed:
|
| 436 |
|
| 437 |
-
| Setting
|
| 438 |
-
| --- | --- |
|
| 439 |
-
| `STORAGE_DIR`
|
| 440 |
-
| `STORAGE_BACKEND`
|
| 441 |
-
| `SFTP_*`
|
| 442 |
-
| `API_KEY`
|
| 443 |
-
| `WORKER_API_KEY`
|
| 444 |
-
| `JWT_SECRET`
|
| 445 |
-
| `UPLOAD_STAGING_DIR` | Local staging directory for upload archives
|
| 446 |
-
| `HOST` / `PORT`
|
| 447 |
-
| `PUBLIC_BASE_URL`
|
| 448 |
|
| 449 |
See [`.env.example`](.env.example) for the complete template. Production environments must use
|
| 450 |
random secrets and restrict `.env` permissions. Never put passwords, tokens, API keys, or database
|
|
|
|
| 1 |
+
|
| 2 |
+
|
| 3 |
# processor
|
| 4 |
|
| 5 |
[English](README.md) | [中文](README.zh-CN.md)
|
|
|
|
| 161 |
|
| 162 |
### Video sources
|
| 163 |
|
| 164 |
+
| Type | Example source name | Description |
|
| 165 |
+
| ------------ | ----------------------------------------- | ------------------------------ |
|
| 166 |
+
| Mono RGB | `head_rgb` | Regular RGB video |
|
| 167 |
+
| Metric depth | `head_depth` | Independent metric-depth video |
|
| 168 |
+
| Stereo RGB | `stereo_left_rgb`, `stereo_right_rgb` | Left and right RGB videos |
|
| 169 |
+
| Stereo depth | `stereo_depth` | Independent depth video |
|
| 170 |
|
| 171 |
RGB and Depth are always stored as independent streams. A depth-only stream is never treated as
|
| 172 |
RGB and is never duplicated alongside the RGB stream.
|
|
|
|
| 252 |
|
| 253 |
#### Capture modules (Input)
|
| 254 |
|
| 255 |
+
| Frontend module | Inputs | Outputs | Purpose and rules |
|
| 256 |
+
| ----------------------- | ------ | -------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
|
| 257 |
+
| `RGB Camera` | None | `RGB Video` | Mono RGB video. The device is selected from the current project's data; blank means auto-match. |
|
| 258 |
+
| `RGB-D Camera` | None | `RGB Video`, `Depth` | Emits RGB and real depth streams from the same device; Depth can connect only to a depth input. |
|
| 259 |
+
| `Stereo RGB Camera` | None | `Left RGB Video`, `Right RGB Video` | Stereo RGB input. Connecting one side creates the corresponding left/right relationship. |
|
| 260 |
+
| `Stereo RGB-D Camera` | None | `Left RGB Video`, `Right RGB Video`, `Depth` | Stereo RGB plus real depth; Depth is used by RGB-D 3D processing. |
|
| 261 |
+
| `Glove Sensor` | None | `Glove Sensor Data` | Pressure, joint, or glove sensor data; no video output. It can connect directly to quality review. |
|
| 262 |
|
| 263 |
#### Video and keypoint processing modules (Process)
|
| 264 |
|
| 265 |
+
| Frontend module | Inputs | Outputs | Purpose and rules |
|
| 266 |
+
| ------------------------ | ------------------------ | -------------- | -------------------------------------------------------------------------------------------------------------------- |
|
| 267 |
+
| `Human Annotation` | `RGB Video` | `Annotation` | Frame-level manual annotation on RGB video. |
|
| 268 |
+
| `AI Annotation` | `RGB Video` | `Annotation` | Video/segment annotation through a local or API VLM; API is recommended for deployment. |
|
| 269 |
+
| `RGB_TO_2D_BareHand` | `RGB Video` | `Hand 2D` | Bare-hand 2D keypoints.`Spatial` is display-only and exports no metric 3D. |
|
| 270 |
+
| `RGB_TO_2D_BlackGlove` | `RGB Video` | `Hand 2D` | Black-glove 2D keypoints. Without Depth, it does not calculate metric 3D. |
|
| 271 |
+
| `RGB-D_3D_BareHand` | `RGB Video`, `Depth` | `Hand 3D` | Metric 3D bare-hand keypoints from RGB and real Depth; a missing branch is skipped. |
|
| 272 |
+
| `RGB-D_3D_BlackGlove` | `RGB Video`, `Depth` | `Hand 3D` | Metric 3D black-glove keypoints from RGB and real Depth; it does not search the disk or fall back to RGB estimation. |
|
| 273 |
|
| 274 |
`MediaPipe Hand` remains available for historical workflows and backend compatibility but is hidden
|
| 275 |
from the new-workflow palette. All nodes provide hover text describing their purpose and accepted
|
|
|
|
| 277 |
|
| 278 |
#### Review modules (Review)
|
| 279 |
|
| 280 |
+
| Frontend module | Inputs | Outputs | Purpose and rules |
|
| 281 |
+
| --------------------- | ------------------------- | ----------------- | ------------------------------------------------------------------------------------------------------ |
|
| 282 |
+
| `Human Review` | `Review Target` | `Reviewed Data` | Human inspection of video, keypoints, or annotation results before downstream processing. |
|
| 283 |
| `AI Quality Review` | `Quality Review Target` | `Reviewed Data` | Checks video decoding, frame continuity, black screens, freezes, annotation coverage, and sensor data. |
|
| 284 |
|
| 285 |
#### Export modules (Export)
|
| 286 |
|
| 287 |
+
| Frontend module | Inputs | Outputs | Purpose and rules |
|
| 288 |
+
| ------------------ | ------------------- | ----------- | ------------------------------------------------------------------ |
|
| 289 |
| `LeRobot Export` | `Exportable Data` | `Dataset` | Exports LeRobot v2.1 or v3.0; the version is selected on the node. |
|
| 290 |
+
| `HDF5 Export` | `Exportable Data` | `Dataset` | Exports HDF5 with the configured compression parameters. |
|
| 291 |
|
| 292 |
#### Workflow Studio UI components
|
| 293 |
|
| 294 |
<details>
|
| 295 |
<summary>Click to expand: frontend component reference</summary>
|
| 296 |
|
| 297 |
+
| UI component | Function |
|
| 298 |
+
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
|
| 299 |
+
| `NodePalette` | Displays Input, Process, Review, and Export categories; supports search, category collapse, and hover descriptions. |
|
| 300 |
+
| `WorkflowCanvas` | Supports drag-and-drop node creation, typed-port connections, node movement, selection, zoom, grid snapping, and minimap navigation. |
|
| 301 |
+
| `WorkflowNode` | Renders the card title, ports, project device selector/input,`Spatial` button, and API settings button. |
|
| 302 |
+
| `WorkflowDrawer` | Lists workflows, creates new workflows, loads workflows, and refreshes project-specific input sources. |
|
| 303 |
+
| `PipelineToolbar` | Provides New, Save, Save As, workflow JSON Export, and Run actions; the yellow dot indicates unsaved changes. |
|
| 304 |
+
| `NodeSettingsModal` | Configures AI Annotation API vendor, model, endpoint, and key fields. |
|
| 305 |
+
| `DeletableEdge` | Renders data connections; a selected edge can be deleted. |
|
| 306 |
|
| 307 |
Device selection is always scoped to the current project. A new empty workflow does not display a
|
| 308 |
device from another project. After data is uploaded, each card's selector shows only real sources
|
|
|
|
| 436 |
Configuration is stored in `processor/.env` or environment variables and must not be
|
| 437 |
committed:
|
| 438 |
|
| 439 |
+
| Setting | Purpose |
|
| 440 |
+
| ---------------------- | ------------------------------------------------------ |
|
| 441 |
+
| `STORAGE_DIR` | Root directory for data and system state |
|
| 442 |
+
| `STORAGE_BACKEND` | `local` or `sftp` |
|
| 443 |
+
| `SFTP_*` | Remote data directory and SSH connection settings |
|
| 444 |
+
| `API_KEY` | API key for the capture client and protected endpoints |
|
| 445 |
+
| `WORKER_API_KEY` | API key used by the Worker to claim and return jobs |
|
| 446 |
+
| `JWT_SECRET` | Secret used to sign Web login sessions |
|
| 447 |
+
| `UPLOAD_STAGING_DIR` | Local staging directory for upload archives |
|
| 448 |
+
| `HOST` / `PORT` | Web/API listen address and port |
|
| 449 |
+
| `PUBLIC_BASE_URL` | Public service URL |
|
| 450 |
|
| 451 |
See [`.env.example`](.env.example) for the complete template. Production environments must use
|
| 452 |
random secrets and restrict `.env` permissions. Never put passwords, tokens, API keys, or database
|
processor/app/api/worker.py
CHANGED
|
@@ -22,7 +22,7 @@ from fastapi.responses import FileResponse, Response
|
|
| 22 |
|
| 23 |
from app.config import settings
|
| 24 |
from app.localstore import (
|
| 25 |
-
list_runs, get_run, save_run, get_episode,
|
| 26 |
set_episode_status, list_exceptions, delete_exception,
|
| 27 |
update_run_if_owned, save_annotations,
|
| 28 |
)
|
|
@@ -59,9 +59,11 @@ def _remove_later(path: str) -> None:
|
|
| 59 |
async def claim_job(body: dict, _: str = Depends(verify_worker_api_key)):
|
| 60 |
"""Worker 轮询领取任务:queued 或租约过期的 running。"""
|
| 61 |
now = _now()
|
| 62 |
-
|
|
|
|
|
|
|
| 63 |
candidate = None
|
| 64 |
-
for r in sorted(
|
| 65 |
status = r.get("status")
|
| 66 |
if status in ("queued", "pending"):
|
| 67 |
candidate = r
|
|
@@ -78,7 +80,9 @@ async def claim_job(body: dict, _: str = Depends(verify_worker_api_key)):
|
|
| 78 |
if candidate is None:
|
| 79 |
return Response(status_code=204)
|
| 80 |
|
| 81 |
-
|
|
|
|
|
|
|
| 82 |
run["status"] = "running"
|
| 83 |
run["worker_id"] = body.get("worker_id")
|
| 84 |
run["lease_token"] = uuid.uuid4().hex
|
|
@@ -182,6 +186,30 @@ def clear_input_zip_cache(episode_id: str) -> None:
|
|
| 182 |
pass
|
| 183 |
|
| 184 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 185 |
def prepare_episode_input_cache(episode_id: str, batch_dir: Path | None = None) -> Path:
|
| 186 |
"""Build a local immutable input archive for a workflow batch.
|
| 187 |
|
|
@@ -201,16 +229,47 @@ def prepare_episode_input_cache(episode_id: str, batch_dir: Path | None = None)
|
|
| 201 |
|
| 202 |
zip_path, stamp_path = _input_zip_cache_paths(str(episode_id))
|
| 203 |
with _INPUT_ZIP_LOCK:
|
| 204 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 205 |
return zip_path
|
| 206 |
|
| 207 |
-
|
| 208 |
-
|
| 209 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 210 |
)
|
| 211 |
-
|
| 212 |
-
|
| 213 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 214 |
canonical_episode = None
|
| 215 |
if (batch_dir / "meta" / "episodes").is_dir():
|
| 216 |
from app.project_dataset import episode_row, episode_files
|
|
@@ -252,21 +311,10 @@ def prepare_episode_input_cache(episode_id: str, batch_dir: Path | None = None)
|
|
| 252 |
else zipfile.ZIP_DEFLATED
|
| 253 |
)
|
| 254 |
archive.write(path, str(arcname), compress_type=compression)
|
| 255 |
-
|
| 256 |
-
|
| 257 |
-
|
| 258 |
-
|
| 259 |
-
try:
|
| 260 |
-
_input_zip_cache_dir().mkdir(parents=True, exist_ok=True)
|
| 261 |
-
shutil.move(tmp.name, str(zip_path))
|
| 262 |
-
stamp_path.write_text(
|
| 263 |
-
json.dumps(_input_zip_stamp(batch_dir)), encoding="utf-8"
|
| 264 |
-
)
|
| 265 |
-
_evict_input_zip_cache()
|
| 266 |
-
except Exception:
|
| 267 |
-
_remove_later(tmp.name)
|
| 268 |
-
raise
|
| 269 |
-
return zip_path
|
| 270 |
|
| 271 |
|
| 272 |
@router.get("/jobs/{run_id}/input")
|
|
@@ -410,6 +458,16 @@ async def complete_job(
|
|
| 410 |
finally:
|
| 411 |
shutil.rmtree(staging_dir, ignore_errors=True)
|
| 412 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 413 |
def mark_completed(current: dict) -> None:
|
| 414 |
current["status"] = "completed"
|
| 415 |
current["finished_at"] = finished_at
|
|
|
|
| 22 |
|
| 23 |
from app.config import settings
|
| 24 |
from app.localstore import (
|
| 25 |
+
list_runs, list_run_summaries, get_run, save_run, get_episode,
|
| 26 |
set_episode_status, list_exceptions, delete_exception,
|
| 27 |
update_run_if_owned, save_annotations,
|
| 28 |
)
|
|
|
|
| 59 |
async def claim_job(body: dict, _: str = Depends(verify_worker_api_key)):
|
| 60 |
"""Worker 轮询领取任务:queued 或租约过期的 running。"""
|
| 61 |
now = _now()
|
| 62 |
+
# Scan summaries, not full records: this endpoint is polled every few
|
| 63 |
+
# seconds and reading every historical run on each poll is what saturated
|
| 64 |
+
# the API process. Only the winner gets its full payload loaded.
|
| 65 |
candidate = None
|
| 66 |
+
for r in sorted(list_run_summaries(), key=lambda r: r.get("created_at") or ""):
|
| 67 |
status = r.get("status")
|
| 68 |
if status in ("queued", "pending"):
|
| 69 |
candidate = r
|
|
|
|
| 80 |
if candidate is None:
|
| 81 |
return Response(status_code=204)
|
| 82 |
|
| 83 |
+
# Reload the full record before mutating — save_run replaces the whole
|
| 84 |
+
# document, so writing the summary back would drop the run's payload.
|
| 85 |
+
run = get_run(str(candidate.get("id"))) or dict(candidate)
|
| 86 |
run["status"] = "running"
|
| 87 |
run["worker_id"] = body.get("worker_id")
|
| 88 |
run["lease_token"] = uuid.uuid4().hex
|
|
|
|
| 186 |
pass
|
| 187 |
|
| 188 |
|
| 189 |
+
def _zip_data_parquets_readable(archive_path: Path) -> bool:
|
| 190 |
+
"""包内 ``data/*.parquet`` 是否都能读出 footer。
|
| 191 |
+
|
| 192 |
+
NAS/SSHFS 偶尔返回被截断的内容,而 ``archive.write`` 会把它原样打进包。
|
| 193 |
+
这种包一旦缓存下来就会**永久**生效:worker 解压后读到的副本本身就是坏
|
| 194 |
+
的,重试也救不回来 —— 表现为
|
| 195 |
+
``ArrowInvalid: Parquet magic bytes not found in footer``。
|
| 196 |
+
|
| 197 |
+
只读 schema(footer)不做全量解码,代价很小。
|
| 198 |
+
"""
|
| 199 |
+
try:
|
| 200 |
+
import pyarrow.parquet as pq
|
| 201 |
+
with zipfile.ZipFile(archive_path) as archive:
|
| 202 |
+
for info in archive.infolist():
|
| 203 |
+
name = info.filename
|
| 204 |
+
if not (name.startswith("data/") and name.endswith(".parquet")):
|
| 205 |
+
continue
|
| 206 |
+
with archive.open(info) as handle:
|
| 207 |
+
pq.read_schema(handle)
|
| 208 |
+
return True
|
| 209 |
+
except Exception:
|
| 210 |
+
return False
|
| 211 |
+
|
| 212 |
+
|
| 213 |
def prepare_episode_input_cache(episode_id: str, batch_dir: Path | None = None) -> Path:
|
| 214 |
"""Build a local immutable input archive for a workflow batch.
|
| 215 |
|
|
|
|
| 229 |
|
| 230 |
zip_path, stamp_path = _input_zip_cache_paths(str(episode_id))
|
| 231 |
with _INPUT_ZIP_LOCK:
|
| 232 |
+
# 缓存命中也要校验:戳只看 meta/info.json,而合并只重写 data parquet
|
| 233 |
+
# —— 戳因此可能在内容已变时仍判为新鲜,把早先固化的坏包一直发出去。
|
| 234 |
+
# 读 footer 很便宜,换掉「坏包只能用一次才知道」的被动局面。
|
| 235 |
+
if (zip_path.is_file()
|
| 236 |
+
and _input_zip_stamp_matches(stamp_path, batch_dir)
|
| 237 |
+
and _zip_data_parquets_readable(zip_path)):
|
| 238 |
return zip_path
|
| 239 |
|
| 240 |
+
# 截断的网络读入不该被缓存:校验失败就重打,连续失败才报错。
|
| 241 |
+
# 打包本身很便宜(本地 zip),代价远低于把坏包固化进缓存。
|
| 242 |
+
packaged: str | None = None
|
| 243 |
+
for attempt in range(3):
|
| 244 |
+
packaged = _build_input_zip(episode_id, batch_dir)
|
| 245 |
+
if _zip_data_parquets_readable(Path(packaged)):
|
| 246 |
+
break
|
| 247 |
+
_remove_later(packaged)
|
| 248 |
+
packaged = None
|
| 249 |
+
print(f"[Worker] input archive unreadable, rebuilding "
|
| 250 |
+
f"({attempt + 1}/3): {episode_id}")
|
| 251 |
+
if packaged is None:
|
| 252 |
+
raise RuntimeError(
|
| 253 |
+
f"Input archive for {episode_id} stays unreadable after 3 attempts "
|
| 254 |
+
f"(source on the storage mount may be corrupt)")
|
| 255 |
+
_input_zip_cache_dir().mkdir(parents=True, exist_ok=True)
|
| 256 |
+
shutil.move(packaged, str(zip_path))
|
| 257 |
+
stamp_path.write_text(
|
| 258 |
+
json.dumps(_input_zip_stamp(batch_dir)), encoding="utf-8"
|
| 259 |
)
|
| 260 |
+
_evict_input_zip_cache()
|
| 261 |
+
return zip_path
|
| 262 |
+
|
| 263 |
+
|
| 264 |
+
def _build_input_zip(episode_id: str, batch_dir: Path) -> str:
|
| 265 |
+
"""把一集的 canonical 文件打成 zip,返回临时包路径。"""
|
| 266 |
+
tmp = tempfile.NamedTemporaryFile(
|
| 267 |
+
prefix=f"egodata-input-{episode_id}-", suffix=".zip",
|
| 268 |
+
dir=str(_worker_tmp_root()), delete=False,
|
| 269 |
+
)
|
| 270 |
+
tmp.close()
|
| 271 |
+
try:
|
| 272 |
+
with zipfile.ZipFile(tmp.name, "w", allowZip64=True) as archive:
|
| 273 |
canonical_episode = None
|
| 274 |
if (batch_dir / "meta" / "episodes").is_dir():
|
| 275 |
from app.project_dataset import episode_row, episode_files
|
|
|
|
| 311 |
else zipfile.ZIP_DEFLATED
|
| 312 |
)
|
| 313 |
archive.write(path, str(arcname), compress_type=compression)
|
| 314 |
+
except Exception:
|
| 315 |
+
_remove_later(tmp.name)
|
| 316 |
+
raise
|
| 317 |
+
return tmp.name
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 318 |
|
| 319 |
|
| 320 |
@router.get("/jobs/{run_id}/input")
|
|
|
|
| 458 |
finally:
|
| 459 |
shutil.rmtree(staging_dir, ignore_errors=True)
|
| 460 |
|
| 461 |
+
# 合并刚重写了该集的 data parquet,而 Worker 输入包的「新鲜度戳」只看
|
| 462 |
+
# meta/info.json —— 合并不碰那个文件,戳因此不变,坏包会被一直复用。
|
| 463 |
+
# 实测后果:包内 parquet 被截断(ArrowInvalid: Parquet magic bytes not
|
| 464 |
+
# found in footer),节点每次都以「列不可读」跳过,且重试无用(解压出来的
|
| 465 |
+
# 副本本身就是坏的)。这里主动清掉,下一次运行会重新打包。
|
| 466 |
+
try:
|
| 467 |
+
clear_input_zip_cache(str(run.get("episode_id") or ""))
|
| 468 |
+
except Exception:
|
| 469 |
+
pass # 清缓存失败不影响本次结果,下次打包仍会走戳校验
|
| 470 |
+
|
| 471 |
def mark_completed(current: dict) -> None:
|
| 472 |
current["status"] = "completed"
|
| 473 |
current["finished_at"] = finished_at
|
processor/app/batch_fingerprint.py
ADDED
|
@@ -0,0 +1,194 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""批次内容指纹 — 识别"换个包名重传同一份数据"。
|
| 2 |
+
|
| 3 |
+
采集端偶尔会把已经上传过的录制再推一遍,但**包名变了**(例如重装、
|
| 4 |
+
重开 App 后本地去重标记丢失,或包名规则从序号变成时间戳)。此时
|
| 5 |
+
``session.py`` 的 ``incoming_name in existing_ids`` 判不出重复,同一份
|
| 6 |
+
数据就会被当成新批次追加,审核页出现两条一模一样的数据,导出也会
|
| 7 |
+
重复计一份(实测:UMIGripper_AI 的 4 组批次视频字节级相同、
|
| 8 |
+
parquet 除 ``episode_index`` 外逐值相同)。
|
| 9 |
+
|
| 10 |
+
本模块给出一个与**包名无关**的内容身份:
|
| 11 |
+
|
| 12 |
+
- 每个视频文件的 md5(按 source_key 排序)
|
| 13 |
+
- canonical data parquet 的内容哈希(排除 ``episode_index`` ——
|
| 14 |
+
同一次录制被追加成不同 episode 时,只有这一列会变)
|
| 15 |
+
|
| 16 |
+
合成为一个短字符串 key,存到 ``state/batch_fingerprints.json``。
|
| 17 |
+
上传时先算新批次的 key,再按项目查表:命中且目标 episode 仍然存在,
|
| 18 |
+
就判定为重复上传。
|
| 19 |
+
|
| 20 |
+
**失败一律放行**:算不出指纹、索引读不到、目标 episode 已被删除 ——
|
| 21 |
+
任一情况都按"新批次"正常入库(即本模块不存在时的行为),绝不因为
|
| 22 |
+
指纹机制本身的问题丢数据。
|
| 23 |
+
"""
|
| 24 |
+
|
| 25 |
+
from __future__ import annotations
|
| 26 |
+
|
| 27 |
+
import hashlib
|
| 28 |
+
import json
|
| 29 |
+
import threading
|
| 30 |
+
from pathlib import Path
|
| 31 |
+
from typing import Any
|
| 32 |
+
|
| 33 |
+
from app.localstore import STATE_ROOT
|
| 34 |
+
|
| 35 |
+
INDEX_PATH = STATE_ROOT / "batch_fingerprints.json"
|
| 36 |
+
_LOCK = threading.Lock()
|
| 37 |
+
|
| 38 |
+
# 视频按块读;读取量固定,不随文件大小膨胀。
|
| 39 |
+
_CHUNK = 1 << 20
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
def _file_md5(path: Path) -> str | None:
|
| 43 |
+
digest = hashlib.md5()
|
| 44 |
+
try:
|
| 45 |
+
with open(path, "rb") as handle:
|
| 46 |
+
for block in iter(lambda: handle.read(_CHUNK), b""):
|
| 47 |
+
digest.update(block)
|
| 48 |
+
except OSError:
|
| 49 |
+
return None
|
| 50 |
+
return digest.hexdigest()
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
def _parquet_content_hash(path: Path) -> str | None:
|
| 54 |
+
"""canonical data parquet 的内容哈希,排除 ``episode_index``。
|
| 55 |
+
|
| 56 |
+
``episode_index`` 是同一次录制被追加成不同 episode 时唯一会变的列,
|
| 57 |
+
必须排除,否则重复上传算不出相同的 key。用 Arrow IPC 序列化取字节:
|
| 58 |
+
对嵌套 list 列(手套 250x250x3 力矩阵、手部关键点)比对 pandas 的
|
| 59 |
+
hash 更稳,且与行序无关的假设更少 —— 这里行序本身就是内容的一部分。
|
| 60 |
+
"""
|
| 61 |
+
try:
|
| 62 |
+
import pyarrow as pa
|
| 63 |
+
import pyarrow.parquet as pq
|
| 64 |
+
except ImportError:
|
| 65 |
+
return None
|
| 66 |
+
try:
|
| 67 |
+
table = pq.read_table(path)
|
| 68 |
+
if "episode_index" in table.column_names:
|
| 69 |
+
table = table.drop_columns(["episode_index"])
|
| 70 |
+
sink = pa.BufferOutputStream()
|
| 71 |
+
with pa.ipc.new_stream(sink, table.schema) as writer:
|
| 72 |
+
writer.write_table(table)
|
| 73 |
+
return hashlib.md5(sink.getvalue().to_pybytes()).hexdigest()
|
| 74 |
+
except Exception:
|
| 75 |
+
return None
|
| 76 |
+
|
| 77 |
+
|
| 78 |
+
def compute_fingerprint(root: Path) -> str | None:
|
| 79 |
+
"""Staging 目录(已 normalize)的内容指纹;算不出返回 None。"""
|
| 80 |
+
root = Path(root)
|
| 81 |
+
parts: list[str] = []
|
| 82 |
+
|
| 83 |
+
videos_root = root / "videos"
|
| 84 |
+
if videos_root.is_dir():
|
| 85 |
+
from app.lerobot_v21 import iter_video_streams
|
| 86 |
+
|
| 87 |
+
entries: list[tuple[str, str]] = []
|
| 88 |
+
for source, path in iter_video_streams(videos_root):
|
| 89 |
+
digest = _file_md5(Path(path))
|
| 90 |
+
if digest is None:
|
| 91 |
+
return None
|
| 92 |
+
entries.append((str(source), digest))
|
| 93 |
+
if not entries:
|
| 94 |
+
return None
|
| 95 |
+
parts.append("|".join(f"v:{s}={d}" for s, d in sorted(entries)))
|
| 96 |
+
|
| 97 |
+
data_files = sorted(
|
| 98 |
+
path for path in (root / "data").rglob("*.parquet")
|
| 99 |
+
if path.is_file() and "/meta/" not in path.as_posix().casefold()
|
| 100 |
+
)
|
| 101 |
+
if not data_files:
|
| 102 |
+
return None
|
| 103 |
+
digest = _parquet_content_hash(data_files[0])
|
| 104 |
+
if digest is None:
|
| 105 |
+
return None
|
| 106 |
+
parts.append(f"d:{digest}")
|
| 107 |
+
|
| 108 |
+
return ";".join(parts)
|
| 109 |
+
|
| 110 |
+
|
| 111 |
+
def _read_index() -> dict[str, dict[str, str]]:
|
| 112 |
+
try:
|
| 113 |
+
value = json.loads(INDEX_PATH.read_text(encoding="utf-8"))
|
| 114 |
+
except (OSError, ValueError):
|
| 115 |
+
return {}
|
| 116 |
+
return value if isinstance(value, dict) else {}
|
| 117 |
+
|
| 118 |
+
|
| 119 |
+
def _write_index(index: dict) -> None:
|
| 120 |
+
try:
|
| 121 |
+
INDEX_PATH.parent.mkdir(parents=True, exist_ok=True)
|
| 122 |
+
temporary = INDEX_PATH.with_suffix(".json.tmp")
|
| 123 |
+
temporary.write_text(
|
| 124 |
+
json.dumps(index, ensure_ascii=False, indent=2), encoding="utf-8")
|
| 125 |
+
temporary.replace(INDEX_PATH)
|
| 126 |
+
except OSError:
|
| 127 |
+
pass
|
| 128 |
+
|
| 129 |
+
|
| 130 |
+
def lookup_duplicate(project_folder: str, fingerprint: str | None,
|
| 131 |
+
live_ids: set[str]) -> str | None:
|
| 132 |
+
"""返回与本次上传内容相同的、且**仍然存在**的 episode_id。
|
| 133 |
+
|
| 134 |
+
``live_ids`` 来自当前项目数据集的实际扫描结果:索引里指向已删除
|
| 135 |
+
episode 的陈旧条目在这里被自然过滤掉,因此"删除后重新上传同一条
|
| 136 |
+
录制"仍会正常入库,不会被误判成重复。
|
| 137 |
+
"""
|
| 138 |
+
if not fingerprint:
|
| 139 |
+
return None
|
| 140 |
+
with _LOCK:
|
| 141 |
+
index = _read_index()
|
| 142 |
+
entries = index.get(str(project_folder))
|
| 143 |
+
if not isinstance(entries, dict):
|
| 144 |
+
return None
|
| 145 |
+
episode_id = entries.get(str(fingerprint))
|
| 146 |
+
if not episode_id:
|
| 147 |
+
return None
|
| 148 |
+
return str(episode_id) if str(episode_id) in live_ids else None
|
| 149 |
+
|
| 150 |
+
|
| 151 |
+
def record_fingerprint(project_folder: str, fingerprint: str | None,
|
| 152 |
+
episode_id: str) -> None:
|
| 153 |
+
"""记录已提交批次的内容指纹(失败静默:只是失去一次去重机会)。"""
|
| 154 |
+
if not fingerprint:
|
| 155 |
+
return
|
| 156 |
+
with _LOCK:
|
| 157 |
+
index = _read_index()
|
| 158 |
+
if not isinstance(index, dict):
|
| 159 |
+
index = {}
|
| 160 |
+
entries = index.get(str(project_folder))
|
| 161 |
+
if not isinstance(entries, dict):
|
| 162 |
+
entries = {}
|
| 163 |
+
entries[str(fingerprint)] = str(episode_id)
|
| 164 |
+
# 同一 episode 换内容(同名重传)时清掉它名下的旧指纹,避免
|
| 165 |
+
# 索引里留下永远指不到实际内容的条目。
|
| 166 |
+
for key in [k for k, v in entries.items()
|
| 167 |
+
if str(v) == str(episode_id) and k != str(fingerprint)]:
|
| 168 |
+
entries.pop(key, None)
|
| 169 |
+
index[str(project_folder)] = entries
|
| 170 |
+
_write_index(index)
|
| 171 |
+
|
| 172 |
+
|
| 173 |
+
def forget_fingerprint(episode_id: str) -> None:
|
| 174 |
+
"""批次被删除后清除其指纹条目。"""
|
| 175 |
+
target = str(episode_id)
|
| 176 |
+
with _LOCK:
|
| 177 |
+
index = _read_index()
|
| 178 |
+
changed = False
|
| 179 |
+
for entries in index.values():
|
| 180 |
+
if not isinstance(entries, dict):
|
| 181 |
+
continue
|
| 182 |
+
for key in [k for k, v in entries.items() if str(v) == target]:
|
| 183 |
+
entries.pop(key, None)
|
| 184 |
+
changed = True
|
| 185 |
+
if changed:
|
| 186 |
+
_write_index(index)
|
| 187 |
+
|
| 188 |
+
|
| 189 |
+
def describe(project_folder: str) -> dict[str, Any]:
|
| 190 |
+
"""诊断用:该项目已记录的内容指纹。"""
|
| 191 |
+
with _LOCK:
|
| 192 |
+
index = _read_index()
|
| 193 |
+
entries = index.get(str(project_folder))
|
| 194 |
+
return entries if isinstance(entries, dict) else {}
|
processor/app/device_progress.py
ADDED
|
@@ -0,0 +1,218 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""采集端任务进度上报账本 —— 本地 JSON 存储(无数据库)。
|
| 2 |
+
|
| 3 |
+
为什么要这个账本:``GET /device/tasks`` 原先把「项目目录下的 session 数」当
|
| 4 |
+
采集进度,而 session 是**上传**的产物。采集端本身在「录制完成」时就已经把
|
| 5 |
+
增量上报上来(见采集端 ``core/task_record.py`` 的分账模型),于是同一条数据
|
| 6 |
+
被算两次——录制一次、上传一次。2026-09-10 现场:UMIGripper_AI 本机录了 12 条、
|
| 7 |
+
服务器上 16 个 session,界面显示 28/20,两边的数都不可信。
|
| 8 |
+
|
| 9 |
+
现在的口径:
|
| 10 |
+
|
| 11 |
+
项目进度 = 各设备上报的录制完成数之和
|
| 12 |
+
|
| 13 |
+
只有**从未收到过上报**的项目才回退到 session 计数,这样升级前就存在的历史
|
| 14 |
+
项目不会突然归零;某个项目一旦有设备上报,它就固定按上报口径走(上传不再
|
| 15 |
+
改变它,采集端录一条 +1)。
|
| 16 |
+
|
| 17 |
+
幂等:采集端用 ``session_id = "{device}:{task_id}:{水位}"`` 标识一次增量,
|
| 18 |
+
同一 (task_id, device_name, session_id) 只计一次 —— 网络抖动重试、程序被杀
|
| 19 |
+
后重放同一个水位,都不会重复计数。
|
| 20 |
+
|
| 21 |
+
文件: ``data/state/device_task_progress.json``
|
| 22 |
+
|
| 23 |
+
{"tasks": {"<task_id>": {
|
| 24 |
+
"devices": {"<device_name>": {
|
| 25 |
+
"count": 12,
|
| 26 |
+
"sessions": ["EGO_001:<task_id>:0", ...], # 已计入的幂等键
|
| 27 |
+
"updated_at": "2026-09-10T02:11:39+00:00"}},
|
| 28 |
+
"updated_at": "..."}},
|
| 29 |
+
"updated_at": "..."}
|
| 30 |
+
"""
|
| 31 |
+
|
| 32 |
+
from __future__ import annotations
|
| 33 |
+
|
| 34 |
+
import json
|
| 35 |
+
import threading
|
| 36 |
+
from datetime import datetime, timezone
|
| 37 |
+
from pathlib import Path
|
| 38 |
+
from typing import Iterable
|
| 39 |
+
|
| 40 |
+
# 读-改-写必须串行:uvicorn 单进程多线程,两个设备同时上报时不能丢增量
|
| 41 |
+
_LOCK = threading.RLock()
|
| 42 |
+
|
| 43 |
+
PROGRESS_FILE_NAME = "device_task_progress.json"
|
| 44 |
+
|
| 45 |
+
# 单次上报上限:采集端 flush 的是「本机未同步增量」,正常是 1~几十;
|
| 46 |
+
# 超过这个数说明上报方数据脏了,直接拒绝而不是把进度顶飞。
|
| 47 |
+
MAX_INCREMENT = 100000
|
| 48 |
+
|
| 49 |
+
_MAX_FIELD = 200 # task_id / device_name / session_id 的长度上限
|
| 50 |
+
|
| 51 |
+
|
| 52 |
+
def _progress_file(path: Path | None = None) -> Path:
|
| 53 |
+
if path is not None:
|
| 54 |
+
return Path(path)
|
| 55 |
+
from app.localstore import STATE_ROOT
|
| 56 |
+
return STATE_ROOT / PROGRESS_FILE_NAME
|
| 57 |
+
|
| 58 |
+
|
| 59 |
+
def _now() -> str:
|
| 60 |
+
return datetime.now(timezone.utc).isoformat()
|
| 61 |
+
|
| 62 |
+
|
| 63 |
+
def _read(path: Path | None = None) -> dict:
|
| 64 |
+
try:
|
| 65 |
+
data = json.loads(_progress_file(path).read_text(encoding="utf-8"))
|
| 66 |
+
except Exception:
|
| 67 |
+
return {"tasks": {}}
|
| 68 |
+
if not isinstance(data, dict):
|
| 69 |
+
return {"tasks": {}}
|
| 70 |
+
tasks = data.get("tasks")
|
| 71 |
+
if not isinstance(tasks, dict):
|
| 72 |
+
data["tasks"] = {}
|
| 73 |
+
return data
|
| 74 |
+
|
| 75 |
+
|
| 76 |
+
def _write(data: dict, path: Path | None = None) -> None:
|
| 77 |
+
target = _progress_file(path)
|
| 78 |
+
target.parent.mkdir(parents=True, exist_ok=True)
|
| 79 |
+
tmp = target.with_suffix(target.suffix + ".tmp")
|
| 80 |
+
tmp.write_text(json.dumps(data, ensure_ascii=False, indent=1), encoding="utf-8")
|
| 81 |
+
tmp.replace(target)
|
| 82 |
+
|
| 83 |
+
|
| 84 |
+
def _clean(value, limit: int = _MAX_FIELD) -> str:
|
| 85 |
+
return str(value or "").strip()[:limit]
|
| 86 |
+
|
| 87 |
+
|
| 88 |
+
def reported_totals(path: Path | None = None) -> dict[str, int]:
|
| 89 |
+
"""各任务的上报聚合数。**只含收到过上报的任务** —— 没上报过的项目不在
|
| 90 |
+
这里,调用方据此回退到 session 计数。"""
|
| 91 |
+
out: dict[str, int] = {}
|
| 92 |
+
with _LOCK:
|
| 93 |
+
data = _read(path)
|
| 94 |
+
for task_id, entry in (data.get("tasks") or {}).items():
|
| 95 |
+
if not isinstance(entry, dict):
|
| 96 |
+
continue
|
| 97 |
+
total = 0
|
| 98 |
+
for dev in (entry.get("devices") or {}).values():
|
| 99 |
+
if isinstance(dev, dict):
|
| 100 |
+
total += int(dev.get("count") or 0)
|
| 101 |
+
out[str(task_id)] = total
|
| 102 |
+
return out
|
| 103 |
+
|
| 104 |
+
|
| 105 |
+
def reported_total(task_id: str, path: Path | None = None) -> int | None:
|
| 106 |
+
"""单个任务的上报聚合数;从未上报过返回 None。"""
|
| 107 |
+
return reported_totals(path).get(str(task_id))
|
| 108 |
+
|
| 109 |
+
|
| 110 |
+
def needs_session_fallback(projects: list[dict],
|
| 111 |
+
reported: dict[str, int]) -> bool:
|
| 112 |
+
"""是否还有项目需要 session 计数兜底。
|
| 113 |
+
|
| 114 |
+
全都有上报时调用方可跳过目录扫描(远程存储上的一次全量遍历)。
|
| 115 |
+
"""
|
| 116 |
+
return any(str(p.get("id") or "") not in reported for p in projects)
|
| 117 |
+
|
| 118 |
+
|
| 119 |
+
def resolve_count(project: dict, reported: dict[str, int],
|
| 120 |
+
sessions: int) -> tuple[int, str]:
|
| 121 |
+
"""单个项目的 (current_count, 口径)。
|
| 122 |
+
|
| 123 |
+
有上报 → 上报聚合数(录制口径,上传不再影响它);
|
| 124 |
+
没有 → session 数(升级前的历史项目不归零)。
|
| 125 |
+
"""
|
| 126 |
+
pid = str(project.get("id") or "")
|
| 127 |
+
if pid in reported:
|
| 128 |
+
return int(reported[pid]), "reported"
|
| 129 |
+
return int(sessions or 0), "sessions"
|
| 130 |
+
|
| 131 |
+
|
| 132 |
+
def apply_increment(task_id: str, device_name: str, session_id: str,
|
| 133 |
+
increment: int, path: Path | None = None) -> dict:
|
| 134 |
+
"""把一次上报落账,返回该任务的最新聚合数。
|
| 135 |
+
|
| 136 |
+
幂等:同一 (task_id, device_name, session_id) 第二次进来只读取不累加
|
| 137 |
+
(``applied`` 为 False),采集端重试因此安全。
|
| 138 |
+
"""
|
| 139 |
+
task_id = _clean(task_id)
|
| 140 |
+
device_name = _clean(device_name)
|
| 141 |
+
session_id = _clean(session_id)
|
| 142 |
+
with _LOCK:
|
| 143 |
+
data = _read(path)
|
| 144 |
+
tasks = data.setdefault("tasks", {})
|
| 145 |
+
entry = tasks.setdefault(task_id, {"devices": {}})
|
| 146 |
+
if not isinstance(entry.get("devices"), dict):
|
| 147 |
+
entry["devices"] = {}
|
| 148 |
+
dev = entry["devices"].setdefault(
|
| 149 |
+
device_name, {"count": 0, "sessions": []})
|
| 150 |
+
sessions = dev.get("sessions")
|
| 151 |
+
if not isinstance(sessions, list):
|
| 152 |
+
sessions = dev["sessions"] = []
|
| 153 |
+
applied = False
|
| 154 |
+
if session_id not in sessions:
|
| 155 |
+
dev["count"] = int(dev.get("count") or 0) + int(increment)
|
| 156 |
+
sessions.append(session_id)
|
| 157 |
+
dev["updated_at"] = _now()
|
| 158 |
+
entry["updated_at"] = dev["updated_at"]
|
| 159 |
+
data["updated_at"] = dev["updated_at"]
|
| 160 |
+
applied = True
|
| 161 |
+
total = sum(int(d.get("count") or 0)
|
| 162 |
+
for d in entry["devices"].values() if isinstance(d, dict))
|
| 163 |
+
if applied:
|
| 164 |
+
_write(data, path)
|
| 165 |
+
return {"total": total, "applied": applied,
|
| 166 |
+
"device_count": int(dev.get("count") or 0),
|
| 167 |
+
"device": device_name, "task_id": task_id}
|
| 168 |
+
|
| 169 |
+
|
| 170 |
+
def apply_report(body: dict, known_task_ids: Iterable[str] | None = None,
|
| 171 |
+
path: Path | None = None) -> dict:
|
| 172 |
+
"""校验并落账一次上报,返回响应体(路由直接回给采集端)。
|
| 173 |
+
|
| 174 |
+
Raises:
|
| 175 |
+
ValueError: 参数缺失/非法/任务不存在 —— 调用方转 HTTP 400。
|
| 176 |
+
|
| 177 |
+
刻意**不返回 404**:采集端把 404 当成「后端还没实现该端点」并永久降级
|
| 178 |
+
(``_progress_supported = False``),此后整机所有任务都不再上报。参数错
|
| 179 |
+
与未知任务都属于「这次请求有问题」,用 400 表达即可。
|
| 180 |
+
"""
|
| 181 |
+
if not isinstance(body, dict):
|
| 182 |
+
raise ValueError("body must be a JSON object")
|
| 183 |
+
task_id = _clean(body.get("task_id"))
|
| 184 |
+
device_name = _clean(body.get("device_name"))
|
| 185 |
+
session_id = _clean(body.get("session_id"))
|
| 186 |
+
if not task_id:
|
| 187 |
+
raise ValueError("task_id required")
|
| 188 |
+
if not device_name:
|
| 189 |
+
raise ValueError("device_name required")
|
| 190 |
+
if not session_id:
|
| 191 |
+
raise ValueError("session_id required (idempotency key)")
|
| 192 |
+
|
| 193 |
+
raw = body.get("increment", 0)
|
| 194 |
+
if isinstance(raw, bool):
|
| 195 |
+
raise ValueError("increment must be an integer")
|
| 196 |
+
try:
|
| 197 |
+
increment = int(raw)
|
| 198 |
+
except (TypeError, ValueError):
|
| 199 |
+
raise ValueError("increment must be an integer")
|
| 200 |
+
if increment < 0:
|
| 201 |
+
raise ValueError("increment must be >= 0")
|
| 202 |
+
if increment > MAX_INCREMENT:
|
| 203 |
+
raise ValueError(f"increment too large (max {MAX_INCREMENT})")
|
| 204 |
+
|
| 205 |
+
if known_task_ids is not None and task_id not in set(known_task_ids):
|
| 206 |
+
raise ValueError(f"unknown task_id: {task_id}")
|
| 207 |
+
|
| 208 |
+
result = apply_increment(task_id, device_name, session_id, increment, path)
|
| 209 |
+
return {
|
| 210 |
+
"ok": True,
|
| 211 |
+
"task_id": task_id,
|
| 212 |
+
"device_name": device_name,
|
| 213 |
+
"applied": result["applied"],
|
| 214 |
+
"increment": increment,
|
| 215 |
+
# 采集端读这个字段当后端权威数 —— 必须是 int(布尔会被 isinstance 放行)
|
| 216 |
+
"completed_count": int(result["total"]),
|
| 217 |
+
"updated_at": _now(),
|
| 218 |
+
}
|
processor/app/lerobot_export.py
CHANGED
|
@@ -129,9 +129,11 @@ def _write_stats(output_dir: Path,
|
|
| 129 |
if not np.isfinite(num).any():
|
| 130 |
continue
|
| 131 |
q = _quantiles(num)
|
|
|
|
|
|
|
|
|
|
|
|
|
| 132 |
stats[col] = {
|
| 133 |
-
"dtype": "float32" if np.issubdtype(s.dtype, np.floating) else "int64",
|
| 134 |
-
"shape": [1],
|
| 135 |
"mean": float(np.nanmean(num)),
|
| 136 |
"std": float(np.nanstd(num)),
|
| 137 |
"min": float(np.nanmin(num)),
|
|
@@ -268,7 +270,22 @@ def _write_stats(output_dir: Path,
|
|
| 268 |
}
|
| 269 |
continue
|
| 270 |
|
| 271 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 272 |
for vpath in videos:
|
| 273 |
cap = cv2.VideoCapture(str(vpath))
|
| 274 |
if not cap.isOpened():
|
|
@@ -283,25 +300,44 @@ def _write_stats(output_dir: Path,
|
|
| 283 |
if not ok:
|
| 284 |
break
|
| 285 |
if read_idx % step == 0 and frame.ndim == 3:
|
| 286 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 287 |
read_idx += 1
|
| 288 |
cap.release()
|
| 289 |
-
if not
|
| 290 |
continue
|
| 291 |
-
stack = np.stack(samples) # (T, H, W, C)
|
| 292 |
-
per_channel = stack.reshape(-1, stack.shape[-1])
|
| 293 |
def _nested(vals):
|
| 294 |
return [[[float(v)] for v in vals]]
|
| 295 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 296 |
stats[vkey] = {
|
| 297 |
-
"min": _nested(
|
| 298 |
-
"max": _nested(
|
| 299 |
-
"mean": _nested(
|
| 300 |
-
"std": _nested(
|
| 301 |
"q01": _nested(q[0]), "q10": _nested(q[1]),
|
| 302 |
"q50": _nested(q[2]), "q90": _nested(q[3]),
|
| 303 |
"q99": _nested(q[4]),
|
| 304 |
-
"count": [int(
|
| 305 |
}
|
| 306 |
except Exception:
|
| 307 |
pass
|
|
@@ -686,8 +722,14 @@ _TACTILE_COLUMN_MAP = {
|
|
| 686 |
|
| 687 |
|
| 688 |
def _read_sensor_rows(session_dir: Path,
|
| 689 |
-
episode_id: str | None = None
|
| 690 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 691 |
rows: dict[int, dict] = {}
|
| 692 |
try:
|
| 693 |
import pyarrow.parquet as pq
|
|
@@ -705,7 +747,13 @@ def _read_sensor_rows(session_dir: Path,
|
|
| 705 |
and path.name != f"episode_{episode_index:06d}.parquet"):
|
| 706 |
continue
|
| 707 |
try:
|
| 708 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 709 |
names = table.schema.names
|
| 710 |
if "frame_index" not in names:
|
| 711 |
continue
|
|
@@ -1059,16 +1107,221 @@ def _passthrough_dtype(key: str, sensor_rows: dict[int, dict]) -> str:
|
|
| 1059 |
return "float32"
|
| 1060 |
|
| 1061 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1062 |
def _pad_variable_columns(rows: list[dict],
|
| 1063 |
-
nan_fill: float = float("nan")) -> int:
|
| 1064 |
-
"""
|
| 1065 |
-
int64[1] 实测 cast 失败)。返回补齐后的每帧样本数 cap。
|
| 1066 |
|
| 1067 |
- observation.imu:每样本 6 维 → 展平为 cap*6(nan_fill 补尾)
|
| 1068 |
- observation.imu_ts_ns:补 0 到 cap
|
| 1069 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1070 |
"""
|
| 1071 |
import numpy as np
|
|
|
|
| 1072 |
imu_cap = 3
|
| 1073 |
for row in rows:
|
| 1074 |
for key in ("observation.imu", "observation.imu_ts_ns"):
|
|
@@ -1076,7 +1329,7 @@ def _pad_variable_columns(rows: list[dict],
|
|
| 1076 |
if isinstance(value, (list, tuple)):
|
| 1077 |
imu_cap = max(imu_cap, len(value))
|
| 1078 |
if not any("observation.imu" in row for row in rows):
|
| 1079 |
-
return imu_cap
|
| 1080 |
for row in rows:
|
| 1081 |
imu = row.get("observation.imu")
|
| 1082 |
if isinstance(imu, (list, tuple)):
|
|
@@ -1100,7 +1353,7 @@ def _pad_variable_columns(rows: list[dict],
|
|
| 1100 |
if isinstance(ts, (list, tuple)):
|
| 1101 |
row["observation.imu_ts_ns"] = (
|
| 1102 |
[int(v) for v in ts] + [0] * (imu_cap - len(ts)))[:imu_cap]
|
| 1103 |
-
return imu_cap
|
| 1104 |
|
| 1105 |
|
| 1106 |
def _propagated_hand_sets(source_row: dict) -> tuple[set[str], set[str], dict[str, set[str]]]:
|
|
@@ -1662,9 +1915,15 @@ def build_lerobot_dataset(dataset_name: str, episode_ids: list[str], output_dir:
|
|
| 1662 |
hand_3d_right_paths: list[str] | None = None,
|
| 1663 |
version: str = "v3.0",
|
| 1664 |
hand_3d_unit: str | None = None,
|
|
|
|
| 1665 |
progress_callback=None) -> Path:
|
| 1666 |
"""Build a LeRobot dataset (``version`` = "v2.1" | "v3.0").
|
| 1667 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1668 |
Depth sources are exported as their preview video (``is_depth_map``)
|
| 1669 |
plus the per-frame ``observation.depth.<source>.valid`` marker; the
|
| 1670 |
original 16-bit PNG sequence stays in the session directory and is not
|
|
@@ -1749,7 +2008,9 @@ def build_lerobot_dataset(dataset_name: str, episode_ids: list[str], output_dir:
|
|
| 1749 |
# 本集实际导出的每路视频(帧数, fps) → episodes parquet 的
|
| 1750 |
# videos/<key>/from_timestamp 映射(官方 v3 加载器按时间戳定位帧)
|
| 1751 |
episode_video_meta: dict[str, tuple[int, float]] = {}
|
| 1752 |
-
sensor_rows = _read_sensor_rows(
|
|
|
|
|
|
|
| 1753 |
annotation_map, annotation_defs, _skipped_candidates = _annotation_maps(episode_id)
|
| 1754 |
|
| 1755 |
description = str(episode.get("project") or "").strip()
|
|
@@ -1791,7 +2052,7 @@ def build_lerobot_dataset(dataset_name: str, episode_ids: list[str], output_dir:
|
|
| 1791 |
/ f"observation.images.{source_key}")
|
| 1792 |
output_video = destination / f"file-{out_episode_index:03d}.mp4"
|
| 1793 |
destination.mkdir(parents=True, exist_ok=True)
|
| 1794 |
-
|
| 1795 |
total_videos += 1
|
| 1796 |
probed_count, probed_fps, width, height = _probe_video(source)
|
| 1797 |
episode_video_meta[f"observation.images.{source_key}"] = (
|
|
@@ -2025,10 +2286,33 @@ def build_lerobot_dataset(dataset_name: str, episode_ids: list[str], output_dir:
|
|
| 2025 |
except Exception:
|
| 2026 |
pass # 进度回调失败不影响导出本身
|
| 2027 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2028 |
# 变长异步传感器列固定长度化:官方加载器不支持变长列(实测
|
| 2029 |
# list<int64> 对声明 int64[1] 直接 cast 失败)。取全数据集最大
|
| 2030 |
# 样本数补齐,声明 shape 同步为固定长度,变长语义保留在 note。
|
| 2031 |
-
imu_cap = _pad_variable_columns(
|
| 2032 |
rows, nan_fill=0.0 if is_v2 else float("nan"))
|
| 2033 |
|
| 2034 |
# 收敛:删恒定哨兵垃圾列(全程 NaN/-1/0/空串 → 数据从未写入)。
|
|
@@ -2301,6 +2585,11 @@ def build_lerobot_dataset(dataset_name: str, episode_ids: list[str], output_dir:
|
|
| 2301 |
import numpy as np
|
| 2302 |
arr = np.asarray(value)
|
| 2303 |
shape = list(arr.shape) if arr.ndim else [1]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2304 |
if key in ("observation.tactile.left", "observation.tactile.right"):
|
| 2305 |
features[key] = {
|
| 2306 |
"dtype": "float32", "shape": [256],
|
|
@@ -2341,6 +2630,33 @@ def build_lerobot_dataset(dataset_name: str, episode_ids: list[str], output_dir:
|
|
| 2341 |
"shape": shape}
|
| 2342 |
if junk_cols:
|
| 2343 |
features = {k: v for k, v in features.items() if k not in junk_cols}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2344 |
# 深度有效帧标记列(行循环写入,不声明会导致官方加载器读不完整)
|
| 2345 |
for depth_key in sorted(depth_source_keys):
|
| 2346 |
key = f"observation.depth.{depth_key}.valid"
|
|
|
|
| 129 |
if not np.isfinite(num).any():
|
| 130 |
continue
|
| 131 |
q = _quantiles(num)
|
| 132 |
+
# 只写数值统计量。不要再加 dtype/shape 之类的元数据字段:官方
|
| 133 |
+
# 加载器会把 stats 里的每一项都转成张量做归一化,字符串字段直接
|
| 134 |
+
# 崩在 torch.from_numpy(np.ndarray of type numpy.str_)。官方数据集
|
| 135 |
+
# 的 stats 也只有 min/max/mean/std/count。
|
| 136 |
stats[col] = {
|
|
|
|
|
|
|
| 137 |
"mean": float(np.nanmean(num)),
|
| 138 |
"std": float(np.nanstd(num)),
|
| 139 |
"min": float(np.nanmin(num)),
|
|
|
|
| 270 |
}
|
| 271 |
continue
|
| 272 |
|
| 273 |
+
# 增量累积:全部集的采样帧若一次性留在内存里会爆 —— 960×1280
|
| 274 |
+
# 的一帧 float32 是 14.75MB,34 集 × 约 200 采样帧 ≈ 100GB。
|
| 275 |
+
# 这里只保留跑动统计量(min/max/和/平方和/计数)与一个固定大小的
|
| 276 |
+
# 分位数蓄水池,内存与集数无关。
|
| 277 |
+
ch_sum = np.zeros(channels, dtype=np.float64)
|
| 278 |
+
ch_sq = np.zeros(channels, dtype=np.float64)
|
| 279 |
+
ch_min = np.full(channels, np.inf, dtype=np.float64)
|
| 280 |
+
ch_max = np.full(channels, -np.inf, dtype=np.float64)
|
| 281 |
+
pixel_count = 0
|
| 282 |
+
# 蓄水池上限与每帧贡献:分位数只需近似,但样本必须**跨帧、跨集**
|
| 283 |
+
# 均匀铺开 —— 只取前几帧会让分位数偏向最早的那一集。每帧按固定
|
| 284 |
+
# 步长抽约 1200 个像素,800 万像素上限可覆盖数千帧。
|
| 285 |
+
reservoir_cap = 8_000_000
|
| 286 |
+
per_frame_budget = 1200
|
| 287 |
+
taken = 0
|
| 288 |
+
reservoir: list = []
|
| 289 |
for vpath in videos:
|
| 290 |
cap = cv2.VideoCapture(str(vpath))
|
| 291 |
if not cap.isOpened():
|
|
|
|
| 300 |
if not ok:
|
| 301 |
break
|
| 302 |
if read_idx % step == 0 and frame.ndim == 3:
|
| 303 |
+
# 先按步长抽样再转 float:一帧 960×1280 有 120 万像素,
|
| 304 |
+
# 对整帧做 astype(float32)/255 是这一步的主要开销,
|
| 305 |
+
# 而统计只需要千级样本。抽样后再运算快约三个数量级。
|
| 306 |
+
flat_u8 = frame[..., :channels].reshape(-1, channels)
|
| 307 |
+
stride_u8 = max(1, flat_u8.shape[0] // per_frame_budget)
|
| 308 |
+
flat = (flat_u8[::stride_u8].astype(np.float32) / 255.0)
|
| 309 |
+
ch_sum += flat.sum(axis=0, dtype=np.float64)
|
| 310 |
+
ch_sq += np.square(flat, dtype=np.float64).sum(axis=0)
|
| 311 |
+
ch_min = np.minimum(ch_min, flat.min(axis=0))
|
| 312 |
+
ch_max = np.maximum(ch_max, flat.max(axis=0))
|
| 313 |
+
pixel_count += flat.shape[0]
|
| 314 |
+
if taken < reservoir_cap:
|
| 315 |
+
# flat 已是抽样结果,直接入池
|
| 316 |
+
take = min(reservoir_cap - taken, flat.shape[0])
|
| 317 |
+
reservoir.append(flat[:take].copy())
|
| 318 |
+
taken += take
|
| 319 |
read_idx += 1
|
| 320 |
cap.release()
|
| 321 |
+
if not pixel_count:
|
| 322 |
continue
|
|
|
|
|
|
|
| 323 |
def _nested(vals):
|
| 324 |
return [[[float(v)] for v in vals]]
|
| 325 |
+
mean = ch_sum / pixel_count
|
| 326 |
+
var = np.maximum(ch_sq / pixel_count - mean * mean, 0.0)
|
| 327 |
+
pooled = np.concatenate(reservoir, axis=0) if reservoir else None
|
| 328 |
+
if pooled is not None and pooled.shape[0]:
|
| 329 |
+
q = np.percentile(pooled, [1, 10, 50, 90, 99], axis=0)
|
| 330 |
+
else:
|
| 331 |
+
q = np.stack([mean] * 5)
|
| 332 |
stats[vkey] = {
|
| 333 |
+
"min": _nested(ch_min),
|
| 334 |
+
"max": _nested(ch_max),
|
| 335 |
+
"mean": _nested(mean),
|
| 336 |
+
"std": _nested(np.sqrt(var)),
|
| 337 |
"q01": _nested(q[0]), "q10": _nested(q[1]),
|
| 338 |
"q50": _nested(q[2]), "q90": _nested(q[3]),
|
| 339 |
"q99": _nested(q[4]),
|
| 340 |
+
"count": [int(pixel_count)],
|
| 341 |
}
|
| 342 |
except Exception:
|
| 343 |
pass
|
|
|
|
| 722 |
|
| 723 |
|
| 724 |
def _read_sensor_rows(session_dir: Path,
|
| 725 |
+
episode_id: str | None = None,
|
| 726 |
+
skip_columns: set[str] | None = None) -> dict[int, dict]:
|
| 727 |
+
"""Merge independently stored glove/action parquet rows by frame_index.
|
| 728 |
+
|
| 729 |
+
``skip_columns``: 调用方已排除的列。这些列在读取阶段就被跳过 ——
|
| 730 |
+
触觉力矩阵每帧 0.8MB,整集 14MB,在 SSHFS 上读完要十几秒;既然最终
|
| 731 |
+
不会写进数据集,就不该读。仅按列名裁剪,不改变任何保留列的值。
|
| 732 |
+
"""
|
| 733 |
rows: dict[int, dict] = {}
|
| 734 |
try:
|
| 735 |
import pyarrow.parquet as pq
|
|
|
|
| 747 |
and path.name != f"episode_{episode_index:06d}.parquet"):
|
| 748 |
continue
|
| 749 |
try:
|
| 750 |
+
if skip_columns:
|
| 751 |
+
# 先读 schema(只读 footer,便宜),再只取需要的列。
|
| 752 |
+
available = [str(n) for n in pq.read_schema(path).names]
|
| 753 |
+
wanted = [n for n in available if n not in skip_columns]
|
| 754 |
+
table = pq.read_table(path, columns=wanted)
|
| 755 |
+
else:
|
| 756 |
+
table = pq.read_table(path)
|
| 757 |
names = table.schema.names
|
| 758 |
if "frame_index" not in names:
|
| 759 |
continue
|
|
|
|
| 1107 |
return "float32"
|
| 1108 |
|
| 1109 |
|
| 1110 |
+
def _write_video_dense_keyframes(source: Path, destination: Path) -> None:
|
| 1111 |
+
"""把视频写入数据集,必要时重编码以加密关键帧。
|
| 1112 |
+
|
| 1113 |
+
采集端用 ffmpeg 默认设置编码(GOP=250),10 秒的集只有 2 个关键帧
|
| 1114 |
+
(0s 和 8.3s)。lerobot 按时间戳取帧时先 seek 到关键帧再向前解码,当
|
| 1115 |
+
查询点紧邻下一个关键帧时 seek 会落到**目标之后**,该帧永远取不到,
|
| 1116 |
+
训练直接抛 FrameTimestampError(实测 7 个采样点里坏 2 个)。
|
| 1117 |
+
|
| 1118 |
+
GOP=30(每秒一个关键帧)可让 seek 稳定落在目标之前,实测 7/7 全通过;
|
| 1119 |
+
代价是文件约大 2 倍(2.7MB -> 5.7MB/集)。
|
| 1120 |
+
|
| 1121 |
+
编码失败(缺 ffmpeg、编码器不可用)时退回原样复制:宁可保留稀疏关键帧
|
| 1122 |
+
也不要让整个导出失败 —— 取不到的帧会让那条样本报错,而不是产出错数据。
|
| 1123 |
+
"""
|
| 1124 |
+
import subprocess
|
| 1125 |
+
try:
|
| 1126 |
+
result = subprocess.run(
|
| 1127 |
+
["ffmpeg", "-y", "-v", "error", "-i", str(source),
|
| 1128 |
+
"-c:v", "libx264", "-g", "30", "-pix_fmt", "yuv420p",
|
| 1129 |
+
"-crf", "20", "-preset", "veryfast", str(destination)],
|
| 1130 |
+
capture_output=True, timeout=300)
|
| 1131 |
+
if result.returncode == 0 and destination.is_file() and destination.stat().st_size:
|
| 1132 |
+
return
|
| 1133 |
+
except (OSError, subprocess.SubprocessError):
|
| 1134 |
+
pass
|
| 1135 |
+
shutil.copy2(source, destination)
|
| 1136 |
+
|
| 1137 |
+
|
| 1138 |
+
def _ensure_observation_state(rows: list[dict]) -> str | None:
|
| 1139 |
+
"""确保每帧有 ``observation.state``;缺失时由 UMI 列派生。
|
| 1140 |
+
|
| 1141 |
+
ACT 的 VAE 编码器无条件读取 ``batch[observation.state]``
|
| 1142 |
+
(modeling_act.py 里那句 ``device=batch[OBS_STATE].device`` 没做
|
| 1143 |
+
robot_state_feature 判空),缺这列直接 KeyError。即便抛开这个上游
|
| 1144 |
+
问题,单帧观测的 ACT 也需要 proprioception 才知道夹爪在哪。
|
| 1145 |
+
|
| 1146 |
+
派生口径 ``[Δx, Δy, Δz, gripper]``:位置是本帧相对**本集第一帧**的
|
| 1147 |
+
偏移。SLAM 的世界原点是每次采集初始化时任意确定的,直接喂绝对坐标
|
| 1148 |
+
等于把无关的世界位置当特征。夹爪取开合度 0–1。
|
| 1149 |
+
|
| 1150 |
+
已有 observation.state 时不动(尊重上游的显式声明)。返回派生列名或
|
| 1151 |
+
None。
|
| 1152 |
+
"""
|
| 1153 |
+
import numpy as np
|
| 1154 |
+
|
| 1155 |
+
if any("observation.state" in row for row in rows):
|
| 1156 |
+
return None
|
| 1157 |
+
if not any("observation.slam_pose" in row for row in rows):
|
| 1158 |
+
return None
|
| 1159 |
+
|
| 1160 |
+
# 按 episode_index 分组:相对偏移必须在**各自集内**计算,跨集求差
|
| 1161 |
+
# 会把两集世界原点的差异当成运动。
|
| 1162 |
+
by_episode: dict[int, list[dict]] = {}
|
| 1163 |
+
for row in rows:
|
| 1164 |
+
try:
|
| 1165 |
+
episode = int(row.get("episode_index", 0))
|
| 1166 |
+
except (TypeError, ValueError):
|
| 1167 |
+
episode = 0
|
| 1168 |
+
by_episode.setdefault(episode, []).append(row)
|
| 1169 |
+
|
| 1170 |
+
derived = 0
|
| 1171 |
+
for episode_rows in by_episode.values():
|
| 1172 |
+
origin = None
|
| 1173 |
+
for row in episode_rows:
|
| 1174 |
+
pose = row.get("observation.slam_pose")
|
| 1175 |
+
try:
|
| 1176 |
+
pose_values = np.asarray(pose, dtype=float).reshape(-1)
|
| 1177 |
+
except (TypeError, ValueError):
|
| 1178 |
+
continue
|
| 1179 |
+
if pose_values.size < 3 or not np.isfinite(pose_values[:3]).all():
|
| 1180 |
+
continue
|
| 1181 |
+
if origin is None:
|
| 1182 |
+
origin = pose_values[:3].copy()
|
| 1183 |
+
gripper = 0.0
|
| 1184 |
+
raw = row.get("observation.gripper_state")
|
| 1185 |
+
try:
|
| 1186 |
+
gripper_values = np.asarray(raw, dtype=float).reshape(-1)
|
| 1187 |
+
if gripper_values.size:
|
| 1188 |
+
gripper = float(np.clip(gripper_values[0] / 100.0, 0.0, 1.0))
|
| 1189 |
+
except (TypeError, ValueError):
|
| 1190 |
+
pass
|
| 1191 |
+
row["observation.state"] = [
|
| 1192 |
+
float(value) for value in np.concatenate([
|
| 1193 |
+
pose_values[:3] - origin, [gripper]])
|
| 1194 |
+
]
|
| 1195 |
+
derived += 1
|
| 1196 |
+
if not derived:
|
| 1197 |
+
return None
|
| 1198 |
+
print(f"[lerobot_export] derived observation.state for {derived} frames")
|
| 1199 |
+
return "observation.state"
|
| 1200 |
+
|
| 1201 |
+
|
| 1202 |
+
def _ensure_slam_pose(rows: list[dict]) -> str | None:
|
| 1203 |
+
"""确保每帧有 ``observation.slam_pose``;缺失时由 ``slam_trajectory`` 重建。
|
| 1204 |
+
|
| 1205 |
+
采集端从 52 集起不再写 ``observation.slam_pose``,只写高频样本缓冲
|
| 1206 |
+
``observation.slam_trajectory``(旧批两列都有)。ACT 的输入特征里声明了
|
| 1207 |
+
slam_pose,整列缺失会让官方加载器与策略直接 KeyError —— 这批数据因此
|
| 1208 |
+
完全无法训练或评估。
|
| 1209 |
+
|
| 1210 |
+
重建口径见 ``app.umi_slam_action.poses_from_trajectory``(与派生 action
|
| 1211 |
+
用的是同一个函数)。拿旧集(有 slam_pose)做过对照:位置相关系数
|
| 1212 |
+
0.998~0.9999、逐帧位置差 std 仅 1~4mm(亚帧采样差),数值范围完全一致,
|
| 1213 |
+
四元数 |dot| 中位 1.0000 —— **同一个物理量、同一个世界系**,不是近似。
|
| 1214 |
+
重建版反而补上了采集端留的 28.5% 零填充行,质量更好。
|
| 1215 |
+
|
| 1216 |
+
起点偏移的口径与 ``_ensure_observation_state`` 一致,故两者可以串联:本
|
| 1217 |
+
函数先跑,缺 state 的集还能继续从重建出的 slam_pose 派生 state。
|
| 1218 |
+
|
| 1219 |
+
已有该列时不动(旧批重导行为完全不变)。返回写入的列名或 None。
|
| 1220 |
+
"""
|
| 1221 |
+
import numpy as np
|
| 1222 |
+
|
| 1223 |
+
if any("observation.slam_pose" in row for row in rows):
|
| 1224 |
+
return None
|
| 1225 |
+
if not any("observation.slam_trajectory" in row for row in rows):
|
| 1226 |
+
return None
|
| 1227 |
+
|
| 1228 |
+
from app.umi_slam_action import poses_from_trajectory
|
| 1229 |
+
|
| 1230 |
+
# 按 episode_index 分组:时间基映射是每集单独拟合的(各集 SLAM 会话
|
| 1231 |
+
# 独立),跨集混在一起会把不同集的时间基搅乱,重建出的位姿全错。
|
| 1232 |
+
by_episode: dict[int, list[dict]] = {}
|
| 1233 |
+
for row in rows:
|
| 1234 |
+
try:
|
| 1235 |
+
episode = int(row.get("episode_index", 0))
|
| 1236 |
+
except (TypeError, ValueError):
|
| 1237 |
+
episode = 0
|
| 1238 |
+
by_episode.setdefault(episode, []).append(row)
|
| 1239 |
+
|
| 1240 |
+
written = 0
|
| 1241 |
+
for episode_rows in by_episode.values():
|
| 1242 |
+
count = len(episode_rows)
|
| 1243 |
+
trajectories = [row.get("observation.slam_trajectory")
|
| 1244 |
+
for row in episode_rows]
|
| 1245 |
+
# 时间戳必须是稠密 float 数组:缺帧补 NaN(poses_from_trajectory 会
|
| 1246 |
+
# 把它们标为缺失再插值),混进 None 会让 np.asarray 推断出 object。
|
| 1247 |
+
stamps = np.full(count, np.nan, dtype=float)
|
| 1248 |
+
for index, row in enumerate(episode_rows):
|
| 1249 |
+
try:
|
| 1250 |
+
stamps[index] = float(row.get("timestamp"))
|
| 1251 |
+
except (TypeError, ValueError):
|
| 1252 |
+
continue
|
| 1253 |
+
poses = poses_from_trajectory(trajectories, stamps, count)
|
| 1254 |
+
if poses is None:
|
| 1255 |
+
continue
|
| 1256 |
+
for row, pose in zip(episode_rows, poses):
|
| 1257 |
+
row["observation.slam_pose"] = [float(value) for value in pose]
|
| 1258 |
+
written += 1
|
| 1259 |
+
if not written:
|
| 1260 |
+
return None
|
| 1261 |
+
print(f"[lerobot_export] derived observation.slam_pose for {written} frames")
|
| 1262 |
+
return "observation.slam_pose"
|
| 1263 |
+
|
| 1264 |
+
|
| 1265 |
+
def _pad_sparse_matrix_columns(rows: list[dict], nan_fill: float) -> dict[str, int]:
|
| 1266 |
+
"""定长矩阵/trajectory 列的空帧补齐到声明长度。
|
| 1267 |
+
|
| 1268 |
+
采集端在尚无数据的帧上写空数组(如触觉矩阵首帧、SLAM 丢帧时的
|
| 1269 |
+
trajectory),同一列因此长短不一。官方加载器按声明的 shape 转成定长
|
| 1270 |
+
Sequence,遇到空值直接 cast 失败:
|
| 1271 |
+
TypeError: Couldn't cast array of type list<element: int64>
|
| 1272 |
+
to List(Value('int64'), length=187500)
|
| 1273 |
+
|
| 1274 |
+
只补空值,不动有值帧 —— 缺失语义由列本身的值域表达。
|
| 1275 |
+
|
| 1276 |
+
返回 ``{列名: 补齐后的长度}``:features 声明必须用这个长度,而不是
|
| 1277 |
+
首帧样本的长度 —— 首帧常常正是空的那一帧,拿它声明会得到 shape [0],
|
| 1278 |
+
与实际写入的长度不符,加载器照样 cast 失败。
|
| 1279 |
+
"""
|
| 1280 |
+
import numpy as np
|
| 1281 |
+
targets: dict[str, int] = {}
|
| 1282 |
+
for key in {k for row in rows for k in row}:
|
| 1283 |
+
if not (key.endswith("_matrix") or key.endswith("_trajectory")):
|
| 1284 |
+
continue
|
| 1285 |
+
target = 0
|
| 1286 |
+
sample_value = None
|
| 1287 |
+
for row in rows:
|
| 1288 |
+
value = row.get(key)
|
| 1289 |
+
if isinstance(value, (list, tuple)) and len(value) > target:
|
| 1290 |
+
target = len(value)
|
| 1291 |
+
sample_value = value
|
| 1292 |
+
if not target or not sample_value:
|
| 1293 |
+
continue
|
| 1294 |
+
targets[key] = target
|
| 1295 |
+
# 补齐值的类型必须与该列已有元素**逐位一致**:触觉矩阵是 int16
|
| 1296 |
+
# (声明 int64),补浮点会让 pyarrow 把整列推断成 double,与声明的
|
| 1297 |
+
# dtype 冲突 → 加载器 cast 失败。int 列补 0,浮点列补 nan_fill。
|
| 1298 |
+
existing = next((v for v in sample_value if v is not None), None)
|
| 1299 |
+
if isinstance(existing, (int, np.integer)) and not isinstance(existing, bool):
|
| 1300 |
+
fill = 0
|
| 1301 |
+
else:
|
| 1302 |
+
fill = nan_fill
|
| 1303 |
+
for row in rows:
|
| 1304 |
+
value = row.get(key)
|
| 1305 |
+
if not isinstance(value, (list, tuple)) or len(value) == target:
|
| 1306 |
+
continue
|
| 1307 |
+
row[key] = list(value) + [fill] * (target - len(value))
|
| 1308 |
+
return targets
|
| 1309 |
+
|
| 1310 |
+
|
| 1311 |
def _pad_variable_columns(rows: list[dict],
|
| 1312 |
+
nan_fill: float = float("nan")) -> tuple[int, dict[str, int]]:
|
| 1313 |
+
"""变长/稀疏列固定长度化。返回 ``(imu_cap, {列名: 补齐长度})``。
|
|
|
|
| 1314 |
|
| 1315 |
- observation.imu:每样本 6 维 → 展平为 cap*6(nan_fill 补尾)
|
| 1316 |
- observation.imu_ts_ns:补 0 到 cap
|
| 1317 |
+
- 矩阵/trajectory 类的空帧:补齐到该列最大长度
|
| 1318 |
+
(见 _pad_sparse_matrix_columns)
|
| 1319 |
+
|
| 1320 |
+
第二个返回值供 features 声明使用 —— 声明必须等于**实际写入的长度**,
|
| 1321 |
+
否则官方加载器 cast 失败。
|
| 1322 |
"""
|
| 1323 |
import numpy as np
|
| 1324 |
+
matrix_targets = _pad_sparse_matrix_columns(rows, nan_fill)
|
| 1325 |
imu_cap = 3
|
| 1326 |
for row in rows:
|
| 1327 |
for key in ("observation.imu", "observation.imu_ts_ns"):
|
|
|
|
| 1329 |
if isinstance(value, (list, tuple)):
|
| 1330 |
imu_cap = max(imu_cap, len(value))
|
| 1331 |
if not any("observation.imu" in row for row in rows):
|
| 1332 |
+
return imu_cap, matrix_targets
|
| 1333 |
for row in rows:
|
| 1334 |
imu = row.get("observation.imu")
|
| 1335 |
if isinstance(imu, (list, tuple)):
|
|
|
|
| 1353 |
if isinstance(ts, (list, tuple)):
|
| 1354 |
row["observation.imu_ts_ns"] = (
|
| 1355 |
[int(v) for v in ts] + [0] * (imu_cap - len(ts)))[:imu_cap]
|
| 1356 |
+
return imu_cap, matrix_targets
|
| 1357 |
|
| 1358 |
|
| 1359 |
def _propagated_hand_sets(source_row: dict) -> tuple[set[str], set[str], dict[str, set[str]]]:
|
|
|
|
| 1915 |
hand_3d_right_paths: list[str] | None = None,
|
| 1916 |
version: str = "v3.0",
|
| 1917 |
hand_3d_unit: str | None = None,
|
| 1918 |
+
exclude_columns: list[str] | None = None,
|
| 1919 |
progress_callback=None) -> Path:
|
| 1920 |
"""Build a LeRobot dataset (``version`` = "v2.1" | "v3.0").
|
| 1921 |
|
| 1922 |
+
``exclude_columns``: 可选,精确列名列表,从数据集中整体剔除。
|
| 1923 |
+
用于丢掉训练用不到但体积/计算代价极高的列(如 250×250×3 的触觉
|
| 1924 |
+
力矩阵:每帧 0.8MB,且逐元素分位数统计是导出耗时的主要来源)。
|
| 1925 |
+
被剔除的列不会出现在 parquet、features 声明和 stats 里。
|
| 1926 |
+
|
| 1927 |
Depth sources are exported as their preview video (``is_depth_map``)
|
| 1928 |
plus the per-frame ``observation.depth.<source>.valid`` marker; the
|
| 1929 |
original 16-bit PNG sequence stays in the session directory and is not
|
|
|
|
| 2008 |
# 本集实际导出的每路视频(帧数, fps) → episodes parquet 的
|
| 2009 |
# videos/<key>/from_timestamp 映射(官方 v3 加载器按时间戳定位帧)
|
| 2010 |
episode_video_meta: dict[str, tuple[int, float]] = {}
|
| 2011 |
+
sensor_rows = _read_sensor_rows(
|
| 2012 |
+
session_dir, episode_id,
|
| 2013 |
+
skip_columns=set(str(name) for name in exclude_columns or []))
|
| 2014 |
annotation_map, annotation_defs, _skipped_candidates = _annotation_maps(episode_id)
|
| 2015 |
|
| 2016 |
description = str(episode.get("project") or "").strip()
|
|
|
|
| 2052 |
/ f"observation.images.{source_key}")
|
| 2053 |
output_video = destination / f"file-{out_episode_index:03d}.mp4"
|
| 2054 |
destination.mkdir(parents=True, exist_ok=True)
|
| 2055 |
+
_write_video_dense_keyframes(source, output_video)
|
| 2056 |
total_videos += 1
|
| 2057 |
probed_count, probed_fps, width, height = _probe_video(source)
|
| 2058 |
episode_video_meta[f"observation.images.{source_key}"] = (
|
|
|
|
| 2286 |
except Exception:
|
| 2287 |
pass # 进度回调失败不影响导出本身
|
| 2288 |
|
| 2289 |
+
# 调用方指定剔除的列(如触觉力矩阵):在补齐/统计/写盘之前删掉,
|
| 2290 |
+
# 这样 parquet、features 声明与 stats 三处天然一致,不需要各自过滤。
|
| 2291 |
+
if exclude_columns:
|
| 2292 |
+
dropped = set(str(name) for name in exclude_columns)
|
| 2293 |
+
if dropped:
|
| 2294 |
+
for row in rows:
|
| 2295 |
+
for key in dropped:
|
| 2296 |
+
row.pop(key, None)
|
| 2297 |
+
print(f"[lerobot_export] excluded columns: {sorted(dropped)}")
|
| 2298 |
+
|
| 2299 |
+
# SLAM 位姿列:采集端 52 集起只写 slam_trajectory,不再写 slam_pose。
|
| 2300 |
+
# 必须赶在 _pad_variable_columns 之前做 —— 补齐会把冲空帧填成 0,而
|
| 2301 |
+
# 0 在 slam_trajectory 里是一个"合法样本"(t=0 的位姿),重建时会被
|
| 2302 |
+
# 当成真实帧混进去。此处补出的 slam_pose 又是下面 observation.state
|
| 2303 |
+
# 的派生依据,故顺序不能颠倒。
|
| 2304 |
+
_ensure_slam_pose(rows)
|
| 2305 |
+
|
| 2306 |
+
# ACT 的观测需要 proprioception:单帧图像不足以让模型知道夹爪当前
|
| 2307 |
+
# 在哪。数据集若已有 observation.state 则原样保留;否则在 UMI 会话上
|
| 2308 |
+
# 由 SLAM 位姿 + 夹爪开合派生 [Δx, Δy, Δz, gripper](位置为相对本集
|
| 2309 |
+
# 起点的偏移 —— SLAM 世界原点是每集任意初始化的,绝对坐标是噪声)。
|
| 2310 |
+
_ensure_observation_state(rows)
|
| 2311 |
+
|
| 2312 |
# 变长异步传感器列固定长度化:官方加载器不支持变长列(实测
|
| 2313 |
# list<int64> 对声明 int64[1] 直接 cast 失败)。取全数据集最大
|
| 2314 |
# 样本数补齐,声明 shape 同步为固定长度,变长语义保留在 note。
|
| 2315 |
+
imu_cap, padded_lengths = _pad_variable_columns(
|
| 2316 |
rows, nan_fill=0.0 if is_v2 else float("nan"))
|
| 2317 |
|
| 2318 |
# 收敛:删恒定哨兵垃圾列(全程 NaN/-1/0/空串 → 数据从未写入)。
|
|
|
|
| 2585 |
import numpy as np
|
| 2586 |
arr = np.asarray(value)
|
| 2587 |
shape = list(arr.shape) if arr.ndim else [1]
|
| 2588 |
+
# 补齐过的列必须以实际写入的长度声明。首帧样本往往正是空的那
|
| 2589 |
+
# 一帧(触觉矩阵首帧、SLAM 丢帧),照它声明会得到 shape [0] 或
|
| 2590 |
+
# 偏小的长度,与实际写入的数据不符 → 官方加载器 cast 失败。
|
| 2591 |
+
if key in padded_lengths:
|
| 2592 |
+
shape = [padded_lengths[key]]
|
| 2593 |
if key in ("observation.tactile.left", "observation.tactile.right"):
|
| 2594 |
features[key] = {
|
| 2595 |
"dtype": "float32", "shape": [256],
|
|
|
|
| 2630 |
"shape": shape}
|
| 2631 |
if junk_cols:
|
| 2632 |
features = {k: v for k, v in features.items() if k not in junk_cols}
|
| 2633 |
+
# 调用方剔除的列同样从声明里去掉。features 由 sensor_rows 样本推导,
|
| 2634 |
+
# 而剔除只作用于 rows,不过滤这里会留下"声明了但数据里没有"的列,
|
| 2635 |
+
# 官方加载器读不出来会报错。
|
| 2636 |
+
if exclude_columns:
|
| 2637 |
+
excluded = set(str(name) for name in exclude_columns)
|
| 2638 |
+
features = {k: v for k, v in features.items() if k not in excluded}
|
| 2639 |
+
# 导出的行里有 observation.state 就必须声明,否则官方加载器读不完整。
|
| 2640 |
+
# 声明放在剔除之后:调用方显式排除的列不该被这里复活。
|
| 2641 |
+
# 补出来的 slam_pose 同样必须声明:features 是从**源 parquet** 采样的
|
| 2642 |
+
# (sensor_rows),而这列由 _ensure_slam_pose 在内存里写进 rows,采样
|
| 2643 |
+
# 看不到它 —— 不补声明就是"数据里有列、声明里没有",官方加载器读不完整。
|
| 2644 |
+
if rows and "observation.slam_pose" in rows[0] and "observation.slam_pose" not in features:
|
| 2645 |
+
features["observation.slam_pose"] = {
|
| 2646 |
+
"dtype": "float32", "shape": [7],
|
| 2647 |
+
"names": ["x", "y", "z", "qx", "qy", "qz", "qw"],
|
| 2648 |
+
"note": ("世界系绝对位姿(米 + 四元数 xyzw)。采集端 52 集起不再"
|
| 2649 |
+
"写入该列,导出时由 observation.slam_trajectory 的 50Hz"
|
| 2650 |
+
"样本重建(与采集端原始列实测位置相关 0.998+,同一世界系)。"),
|
| 2651 |
+
}
|
| 2652 |
+
if rows and "observation.state" in rows[0] and "observation.state" not in features:
|
| 2653 |
+
features["observation.state"] = {
|
| 2654 |
+
"dtype": "float32", "shape": [4],
|
| 2655 |
+
"names": ["delta_x", "delta_y", "delta_z", "gripper"],
|
| 2656 |
+
"note": ("proprioception:本帧位置相对**本集第一帧**的偏移(米)"
|
| 2657 |
+
"+ 夹爪开合(0–1)。SLAM 世界原点是每集任意初始化的,"
|
| 2658 |
+
"故用相对量而非绝对坐标。"),
|
| 2659 |
+
}
|
| 2660 |
# 深度有效帧标记列(行循环写入,不声明会导致官方加载器读不完整)
|
| 2661 |
for depth_key in sorted(depth_source_keys):
|
| 2662 |
key = f"observation.depth.{depth_key}.valid"
|
processor/app/localstore.py
CHANGED
|
@@ -37,6 +37,7 @@ _SESSIONS_CACHE_TTL = 300.0
|
|
| 37 |
# 全量遍历远程挂载很慢:只允许单一线程执行(_scan_lock),遍历本身不持有
|
| 38 |
# _lock —— 缓存过期触发的重扫不再把状态读写请求拖在锁后面排队。
|
| 39 |
_scan_lock = threading.Lock()
|
|
|
|
| 40 |
_scan_generation = 0 # invalidate_session_cache 递增;并发扫描据此丢弃过期结果
|
| 41 |
_state_write_generation = 0 # write_episode_state 递增;扫描期间的状态写入触发重扫
|
| 42 |
_video_probe_cache: dict[str, tuple[int, float]] = {}
|
|
@@ -48,6 +49,14 @@ _workflows_cache: list[dict] | None = None
|
|
| 48 |
_workflows_cache_at = 0.0
|
| 49 |
_runs_cache: list[dict] | None = None
|
| 50 |
_runs_cache_at = 0.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 51 |
# Runs are written by the API process and read by separate worker processes.
|
| 52 |
# A ten-minute process-local cache can hide a newly queued upload/reprocess
|
| 53 |
# from workers. Keep the cache short while still coalescing rapid UI scans.
|
|
@@ -129,6 +138,14 @@ def invalidate_session_cache() -> None:
|
|
| 129 |
_sessions_cache = None
|
| 130 |
_sessions_cache_at = 0.0
|
| 131 |
_video_probe_cache.clear()
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 132 |
|
| 133 |
|
| 134 |
def scan_sessions() -> list[dict]:
|
|
@@ -141,8 +158,53 @@ def scan_sessions() -> list[dict]:
|
|
| 141 |
_lock:缓存过期触发的重扫不再把状态读写等无关请求拖在锁后面排队。
|
| 142 |
遍历期间发生结构变化(invalidate)/状态写入时,按代际计数丢弃过期结果
|
| 143 |
重扫一次。
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 144 |
"""
|
| 145 |
global _sessions_cache, _sessions_cache_at, _task_sessions_cache
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 146 |
while True:
|
| 147 |
with _lock:
|
| 148 |
now = time.monotonic()
|
|
@@ -210,11 +272,19 @@ def _build_sessions_snapshot() -> list[dict]:
|
|
| 210 |
def _scan_project_dir(project_dir: Path, project: str) -> list[dict]:
|
| 211 |
"""Scan a project-level LeRobot tree and expose one row per episode."""
|
| 212 |
from app.lerobot_v21 import is_depth_source
|
| 213 |
-
from app.project_dataset import
|
|
|
|
|
|
|
|
|
|
|
|
|
| 214 |
|
| 215 |
info = _read_json(project_dir / "meta" / "info.json", {})
|
| 216 |
if not isinstance(info, dict):
|
| 217 |
info = {}
|
|
|
|
|
|
|
|
|
|
|
|
|
| 218 |
rows: list[dict] = []
|
| 219 |
for row in project_episode_rows(project_dir):
|
| 220 |
try:
|
|
@@ -223,7 +293,7 @@ def _scan_project_dir(project_dir: Path, project: str) -> list[dict]:
|
|
| 223 |
continue
|
| 224 |
episode_id = str(row.get("episode_id") or row.get("source_batch")
|
| 225 |
or f"{project}_{episode_index:06d}")
|
| 226 |
-
files =
|
| 227 |
state = read_episode_state(episode_id)
|
| 228 |
# Project-level info is an aggregate. Do not let it make a new
|
| 229 |
# one-camera episode display devices that belong to another episode.
|
|
@@ -683,6 +753,13 @@ def delete_episode(episode_id: str, permanent: bool = False,
|
|
| 683 |
_remove_json(EPISODE_STATES_DIR / f"{name}.json")
|
| 684 |
_remove_json(ANNOTATIONS_DIR / f"{name}.json")
|
| 685 |
_episode_state_cache.pop(name, None)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 686 |
global _deleted_episodes_cache, _deleted_episodes_cache_at
|
| 687 |
_deleted_episodes_cache = None
|
| 688 |
_deleted_episodes_cache_at = 0.0
|
|
@@ -1190,38 +1267,198 @@ def mutate_annotation_by_id(
|
|
| 1190 |
# ── 工作流运行队列 ───────────────────────────────────
|
| 1191 |
|
| 1192 |
RUNS_DIR = STATE_ROOT / "runs"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1193 |
|
|
|
|
|
|
|
|
|
|
| 1194 |
|
| 1195 |
-
|
| 1196 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1197 |
now = time.monotonic()
|
| 1198 |
-
|
| 1199 |
-
|
| 1200 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1201 |
if not RUNS_DIR.is_dir():
|
| 1202 |
-
|
| 1203 |
-
_runs_cache_at = now
|
| 1204 |
-
return []
|
| 1205 |
-
runs = []
|
| 1206 |
for f in sorted(RUNS_DIR.glob("*.json")):
|
| 1207 |
run = _read_json(f, None)
|
| 1208 |
-
if isinstance(run, dict):
|
| 1209 |
-
|
| 1210 |
-
|
| 1211 |
-
|
| 1212 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1213 |
|
| 1214 |
|
| 1215 |
def get_run(run_id: str) -> dict | None:
|
| 1216 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1217 |
|
| 1218 |
|
| 1219 |
def save_run(run: dict) -> None:
|
| 1220 |
-
|
| 1221 |
with _lock:
|
| 1222 |
-
|
| 1223 |
-
|
| 1224 |
-
_runs_cache_at = 0.0
|
| 1225 |
|
| 1226 |
|
| 1227 |
def update_run_if_owned(
|
|
@@ -1231,7 +1468,6 @@ def update_run_if_owned(
|
|
| 1231 |
mutator: Callable[[dict], None],
|
| 1232 |
) -> dict | None:
|
| 1233 |
"""Atomically mutate a running job only for its current lease owner."""
|
| 1234 |
-
global _runs_cache, _runs_cache_at
|
| 1235 |
with _lock:
|
| 1236 |
path = RUNS_DIR / f"{run_id}.json"
|
| 1237 |
run = _read_json(path, None)
|
|
@@ -1242,9 +1478,10 @@ def update_run_if_owned(
|
|
| 1242 |
or run.get("lease_token") != lease_token):
|
| 1243 |
return None
|
| 1244 |
mutator(run)
|
| 1245 |
-
|
| 1246 |
-
|
| 1247 |
-
|
|
|
|
| 1248 |
return run
|
| 1249 |
|
| 1250 |
|
|
@@ -1265,7 +1502,6 @@ def save_run_if_absent(
|
|
| 1265 |
``workflow_revision`` 时,queued/running 仍视为占用,completed 则
|
| 1266 |
允许第一次迁移到带版本的幂等记录。
|
| 1267 |
"""
|
| 1268 |
-
global _runs_cache, _runs_cache_at
|
| 1269 |
with _lock:
|
| 1270 |
revision = run.get("workflow_revision")
|
| 1271 |
for existing in list_runs():
|
|
@@ -1280,23 +1516,21 @@ def save_run_if_absent(
|
|
| 1280 |
existing["status"] = "superseded"
|
| 1281 |
existing["finished_at"] = utcnow_iso()
|
| 1282 |
existing["error_log"] = "Superseded by a newer upload or reprocess request"
|
| 1283 |
-
|
| 1284 |
-
|
| 1285 |
-
_runs_cache_at = 0.0
|
| 1286 |
continue
|
| 1287 |
if same_revision or legacy_active:
|
| 1288 |
if existing.get("status") in ("queued", "running"):
|
| 1289 |
return existing, False
|
| 1290 |
if existing.get("status") == "completed" and not allow_completed_rerun:
|
| 1291 |
return existing, False
|
| 1292 |
-
|
| 1293 |
-
|
| 1294 |
-
_runs_cache_at = 0.0
|
| 1295 |
return run, True
|
| 1296 |
|
| 1297 |
|
| 1298 |
def delete_run(run_id: str) -> None:
|
| 1299 |
-
|
| 1300 |
_remove_json(RUNS_DIR / f"{run_id}.json")
|
| 1301 |
-
|
| 1302 |
-
|
|
|
|
| 37 |
# 全量遍历远程挂载很慢:只允许单一线程执行(_scan_lock),遍历本身不持有
|
| 38 |
# _lock —— 缓存过期触发的重扫不再把状态读写请求拖在锁后面排队。
|
| 39 |
_scan_lock = threading.Lock()
|
| 40 |
+
_sessions_refresh_running = False # 后台 stale-while-revalidate 单飞标志
|
| 41 |
_scan_generation = 0 # invalidate_session_cache 递增;并发扫描据此丢弃过期结果
|
| 42 |
_state_write_generation = 0 # write_episode_state 递增;扫描期间的状态写入触发重扫
|
| 43 |
_video_probe_cache: dict[str, tuple[int, float]] = {}
|
|
|
|
| 49 |
_workflows_cache_at = 0.0
|
| 50 |
_runs_cache: list[dict] | None = None
|
| 51 |
_runs_cache_at = 0.0
|
| 52 |
+
_runs_cache_revision: tuple = ()
|
| 53 |
+
# list_runs(include_archived=False) keeps its own slot; the two variants are
|
| 54 |
+
# cached independently so neither call pattern can serve the other's result.
|
| 55 |
+
_runs_live_cache: list[dict] | None = None
|
| 56 |
+
_runs_live_cache_at = 0.0
|
| 57 |
+
_runs_live_cache_revision: tuple = ()
|
| 58 |
+
# 指纹变化时才重建快照;单飞,避免并发请求各读一遍 539 个 JSON。
|
| 59 |
+
_runs_scan_lock = threading.Lock()
|
| 60 |
# Runs are written by the API process and read by separate worker processes.
|
| 61 |
# A ten-minute process-local cache can hide a newly queued upload/reprocess
|
| 62 |
# from workers. Keep the cache short while still coalescing rapid UI scans.
|
|
|
|
| 138 |
_sessions_cache = None
|
| 139 |
_sessions_cache_at = 0.0
|
| 140 |
_video_probe_cache.clear()
|
| 141 |
+
# 项目索引缓存(project_dataset)按指纹自校验,但上传/删除/改名必须
|
| 142 |
+
# 立刻可见,不能等 TTL。懒导入避免模块级循环依赖。
|
| 143 |
+
try:
|
| 144 |
+
from app.project_dataset import invalidate_project_cache
|
| 145 |
+
|
| 146 |
+
invalidate_project_cache()
|
| 147 |
+
except Exception as exc:
|
| 148 |
+
print(f"[Cache] Project index invalidation skipped: {exc}")
|
| 149 |
|
| 150 |
|
| 151 |
def scan_sessions() -> list[dict]:
|
|
|
|
| 158 |
_lock:缓存过期触发的重扫不再把状态读写等无关请求拖在锁后面排队。
|
| 159 |
遍历期间发生结构变化(invalidate)/状态写入时,按代际计数丢弃过期结果
|
| 160 |
重扫一次。
|
| 161 |
+
|
| 162 |
+
TTL 过期但已有快照时走 stale-while-revalidate:立刻返回旧快照,后台单飞
|
| 163 |
+
重建。全量遍历要 5s+,让审核页的请求撞上它正是"打开一条数据等十几秒"
|
| 164 |
+
的一个来源;旧快照里缺的只是刚上传的那一集,而上传会走
|
| 165 |
+
invalidate_session_cache() 清空缓存,所以这里不会藏住新数据。
|
| 166 |
"""
|
| 167 |
global _sessions_cache, _sessions_cache_at, _task_sessions_cache
|
| 168 |
+
with _lock:
|
| 169 |
+
snapshot = (copy.deepcopy(_sessions_cache)
|
| 170 |
+
if _sessions_cache is not None else None)
|
| 171 |
+
fresh = (snapshot is not None
|
| 172 |
+
and time.monotonic() - _sessions_cache_at < _SESSIONS_CACHE_TTL)
|
| 173 |
+
if fresh:
|
| 174 |
+
return snapshot
|
| 175 |
+
if snapshot is not None:
|
| 176 |
+
_refresh_sessions_in_background()
|
| 177 |
+
return snapshot
|
| 178 |
+
# 冷启动或刚被 invalidate:必须同步扫出第一份快照。
|
| 179 |
+
return _scan_sessions_blocking()
|
| 180 |
+
|
| 181 |
+
|
| 182 |
+
def _refresh_sessions_in_background() -> None:
|
| 183 |
+
"""Single-flight background rebuild of the expired session snapshot."""
|
| 184 |
+
global _sessions_refresh_running
|
| 185 |
+
with _lock:
|
| 186 |
+
if _sessions_refresh_running:
|
| 187 |
+
return
|
| 188 |
+
_sessions_refresh_running = True
|
| 189 |
+
threading.Thread(
|
| 190 |
+
target=_run_session_refresh, name="session-refresh", daemon=True
|
| 191 |
+
).start()
|
| 192 |
+
|
| 193 |
+
|
| 194 |
+
def _run_session_refresh() -> None:
|
| 195 |
+
global _sessions_refresh_running
|
| 196 |
+
try:
|
| 197 |
+
_scan_sessions_blocking()
|
| 198 |
+
except Exception as exc:
|
| 199 |
+
print(f"[Cache] Background session refresh failed: {exc}")
|
| 200 |
+
finally:
|
| 201 |
+
with _lock:
|
| 202 |
+
_sessions_refresh_running = False
|
| 203 |
+
|
| 204 |
+
|
| 205 |
+
def _scan_sessions_blocking() -> list[dict]:
|
| 206 |
+
"""Full remote tree walk; callers block for the duration."""
|
| 207 |
+
global _sessions_cache, _sessions_cache_at, _task_sessions_cache
|
| 208 |
while True:
|
| 209 |
with _lock:
|
| 210 |
now = time.monotonic()
|
|
|
|
| 272 |
def _scan_project_dir(project_dir: Path, project: str) -> list[dict]:
|
| 273 |
"""Scan a project-level LeRobot tree and expose one row per episode."""
|
| 274 |
from app.lerobot_v21 import is_depth_source
|
| 275 |
+
from app.project_dataset import (
|
| 276 |
+
episode_files_from_index,
|
| 277 |
+
project_episode_file_index,
|
| 278 |
+
project_episode_rows,
|
| 279 |
+
)
|
| 280 |
|
| 281 |
info = _read_json(project_dir / "meta" / "info.json", {})
|
| 282 |
if not isinstance(info, dict):
|
| 283 |
info = {}
|
| 284 |
+
# One index for the whole project: asking ``episode_files`` per episode
|
| 285 |
+
# would re-walk data/, videos/ and every episode index file each time,
|
| 286 |
+
# costing (1 + episodes) x parquet reads on the storage mount.
|
| 287 |
+
file_index = project_episode_file_index(project_dir)
|
| 288 |
rows: list[dict] = []
|
| 289 |
for row in project_episode_rows(project_dir):
|
| 290 |
try:
|
|
|
|
| 293 |
continue
|
| 294 |
episode_id = str(row.get("episode_id") or row.get("source_batch")
|
| 295 |
or f"{project}_{episode_index:06d}")
|
| 296 |
+
files = episode_files_from_index(project_dir, episode_index, file_index)
|
| 297 |
state = read_episode_state(episode_id)
|
| 298 |
# Project-level info is an aggregate. Do not let it make a new
|
| 299 |
# one-camera episode display devices that belong to another episode.
|
|
|
|
| 753 |
_remove_json(EPISODE_STATES_DIR / f"{name}.json")
|
| 754 |
_remove_json(ANNOTATIONS_DIR / f"{name}.json")
|
| 755 |
_episode_state_cache.pop(name, None)
|
| 756 |
+
# 内容指纹指向已删除的批次:清掉,否则同一条录制重新上传时会被
|
| 757 |
+
# 误判成"重复"而试图覆盖一个已经不存在的 episode。
|
| 758 |
+
try:
|
| 759 |
+
from app.batch_fingerprint import forget_fingerprint
|
| 760 |
+
forget_fingerprint(name)
|
| 761 |
+
except Exception:
|
| 762 |
+
pass
|
| 763 |
global _deleted_episodes_cache, _deleted_episodes_cache_at
|
| 764 |
_deleted_episodes_cache = None
|
| 765 |
_deleted_episodes_cache_at = 0.0
|
|
|
|
| 1267 |
# ── 工作流运行队列 ───────────────────────────────────
|
| 1268 |
|
| 1269 |
RUNS_DIR = STATE_ROOT / "runs"
|
| 1270 |
+
# Finished runs keep their full record for the UI, but they are moved out of
|
| 1271 |
+
# the live queue directory: the worker polls /jobs/claim every few seconds and
|
| 1272 |
+
# used to re-read every historical run (hundreds of 26 KB JSON files on the
|
| 1273 |
+
# NAS mount) on each poll. Only unfinished work stays in RUNS_DIR.
|
| 1274 |
+
RUNS_ARCHIVE_DIR = RUNS_DIR / "archive"
|
| 1275 |
+
_TERMINAL_RUN_STATUSES = frozenset({"completed", "failed", "superseded"})
|
| 1276 |
+
|
| 1277 |
+
# claim_job only needs these to pick the next job; the full run payload is
|
| 1278 |
+
# loaded afterwards for the winner alone.
|
| 1279 |
+
_RUN_SUMMARY_FIELDS = ("id", "status", "created_at", "lease_until",
|
| 1280 |
+
"worker_id", "lease_token", "attempt")
|
| 1281 |
+
|
| 1282 |
+
|
| 1283 |
+
def list_runs(include_archived: bool = True) -> list[dict]:
|
| 1284 |
+
"""All runs, newest-finished records included by default.
|
| 1285 |
|
| 1286 |
+
``include_archived=False`` restricts the scan to the live queue and is
|
| 1287 |
+
only for callers that just need claimable work. Everything else keeps
|
| 1288 |
+
the historical view it had before archiving existed.
|
| 1289 |
|
| 1290 |
+
The two variants are cached separately — a single shared cache would
|
| 1291 |
+
serve whichever variant ran first to both callers.
|
| 1292 |
+
|
| 1293 |
+
TTL 过期后先比一次廉价指纹(目录 listing + stat,539 个 run 实测 4ms),
|
| 1294 |
+
内容没变就只刷新时间戳。历史快照要读 539 个 JSON,冷读 6s —— 那正是
|
| 1295 |
+
``/export/episode-format`` 每次点开数据都要付的代价;而 worker 领取
|
| 1296 |
+
走的是 :func:`list_run_summaries`,不经过这里,所以缓存不会耽误派活。
|
| 1297 |
+
"""
|
| 1298 |
+
global _runs_cache, _runs_cache_at, _runs_cache_revision
|
| 1299 |
+
global _runs_live_cache, _runs_live_cache_at, _runs_live_cache_revision
|
| 1300 |
now = time.monotonic()
|
| 1301 |
+
cache, cache_at = ((_runs_cache, _runs_cache_at) if include_archived
|
| 1302 |
+
else (_runs_live_cache, _runs_live_cache_at))
|
| 1303 |
+
if cache is not None and now - cache_at < _STATE_LIST_CACHE_TTL:
|
| 1304 |
+
return copy.deepcopy(cache)
|
| 1305 |
+
revision = _run_dirs_revision(include_archived)
|
| 1306 |
+
cached_revision = (_runs_cache_revision if include_archived
|
| 1307 |
+
else _runs_live_cache_revision)
|
| 1308 |
+
if cache is not None and cached_revision == revision:
|
| 1309 |
+
if include_archived:
|
| 1310 |
+
_runs_cache_at = now
|
| 1311 |
+
else:
|
| 1312 |
+
_runs_live_cache_at = now
|
| 1313 |
+
return copy.deepcopy(cache)
|
| 1314 |
+
# 指纹变化时才真读;单飞避免并发请求各读一遍 539 个文件。
|
| 1315 |
+
with _runs_scan_lock:
|
| 1316 |
+
now = time.monotonic()
|
| 1317 |
+
cache, cache_at = ((_runs_cache, _runs_cache_at) if include_archived
|
| 1318 |
+
else (_runs_live_cache, _runs_live_cache_at))
|
| 1319 |
+
cached_revision = (_runs_cache_revision if include_archived
|
| 1320 |
+
else _runs_live_cache_revision)
|
| 1321 |
+
if cache is not None and (cached_revision == revision
|
| 1322 |
+
or now - cache_at < _STATE_LIST_CACHE_TTL):
|
| 1323 |
+
return copy.deepcopy(cache)
|
| 1324 |
+
runs: list[dict] = []
|
| 1325 |
+
for directory in _run_dirs(include_archived):
|
| 1326 |
+
if not directory.is_dir():
|
| 1327 |
+
continue
|
| 1328 |
+
for f in sorted(directory.glob("*.json")):
|
| 1329 |
+
run = _read_json(f, None)
|
| 1330 |
+
if isinstance(run, dict):
|
| 1331 |
+
runs.append(run)
|
| 1332 |
+
revision = _run_dirs_revision(include_archived)
|
| 1333 |
+
if include_archived:
|
| 1334 |
+
_runs_cache = runs
|
| 1335 |
+
_runs_cache_at = time.monotonic()
|
| 1336 |
+
_runs_cache_revision = revision
|
| 1337 |
+
else:
|
| 1338 |
+
_runs_live_cache = runs
|
| 1339 |
+
_runs_live_cache_at = time.monotonic()
|
| 1340 |
+
_runs_live_cache_revision = revision
|
| 1341 |
+
return copy.deepcopy(runs)
|
| 1342 |
+
|
| 1343 |
+
|
| 1344 |
+
def _run_dirs_revision(include_archived: bool) -> tuple:
|
| 1345 |
+
"""Cheap fingerprint of the run files (name, size, mtime)."""
|
| 1346 |
+
parts: list[tuple] = []
|
| 1347 |
+
try:
|
| 1348 |
+
for directory in _run_dirs(include_archived):
|
| 1349 |
+
if not directory.is_dir():
|
| 1350 |
+
continue
|
| 1351 |
+
for f in sorted(directory.glob("*.json")):
|
| 1352 |
+
stat = f.stat()
|
| 1353 |
+
parts.append((f.name, stat.st_size, stat.st_mtime_ns))
|
| 1354 |
+
except OSError:
|
| 1355 |
+
return ()
|
| 1356 |
+
return tuple(parts)
|
| 1357 |
+
|
| 1358 |
+
|
| 1359 |
+
def _run_dirs(include_archived: bool = False) -> list[Path]:
|
| 1360 |
+
"""Live queue directory first; the archive only when explicitly asked."""
|
| 1361 |
+
dirs = [RUNS_DIR]
|
| 1362 |
+
if include_archived:
|
| 1363 |
+
dirs.append(RUNS_ARCHIVE_DIR)
|
| 1364 |
+
return dirs
|
| 1365 |
+
|
| 1366 |
+
|
| 1367 |
+
def _invalidate_runs_cache() -> None:
|
| 1368 |
+
"""Drop both run-list caches. Callers must hold ``_lock``."""
|
| 1369 |
+
global _runs_cache, _runs_cache_at, _runs_live_cache, _runs_live_cache_at
|
| 1370 |
+
_runs_cache = None
|
| 1371 |
+
_runs_cache_at = 0.0
|
| 1372 |
+
_runs_live_cache = None
|
| 1373 |
+
_runs_live_cache_at = 0.0
|
| 1374 |
+
|
| 1375 |
+
|
| 1376 |
+
def list_run_summaries() -> list[dict]:
|
| 1377 |
+
"""Live runs with only the fields the claim scan needs.
|
| 1378 |
+
|
| 1379 |
+
Parsing 26 KB of JSON per historical run is what made an idle worker burn
|
| 1380 |
+
the backend's CPU; the claim path only compares ``status`` /
|
| 1381 |
+
``created_at`` / ``lease_until``. Archived (terminal) runs are skipped
|
| 1382 |
+
entirely — nothing in the live queue can be claimed from them.
|
| 1383 |
+
"""
|
| 1384 |
+
summaries: list[dict] = []
|
| 1385 |
if not RUNS_DIR.is_dir():
|
| 1386 |
+
return summaries
|
|
|
|
|
|
|
|
|
|
| 1387 |
for f in sorted(RUNS_DIR.glob("*.json")):
|
| 1388 |
run = _read_json(f, None)
|
| 1389 |
+
if not isinstance(run, dict):
|
| 1390 |
+
continue
|
| 1391 |
+
summaries.append({key: run.get(key) for key in _RUN_SUMMARY_FIELDS})
|
| 1392 |
+
return summaries
|
| 1393 |
+
|
| 1394 |
+
|
| 1395 |
+
def archive_run_file(run_id: str) -> bool:
|
| 1396 |
+
"""Move a finished run out of the live queue directory.
|
| 1397 |
+
|
| 1398 |
+
Keeps the record (the UI can still load it by id) while taking it out of
|
| 1399 |
+
the per-poll directory listing. Safe to call repeatedly.
|
| 1400 |
+
"""
|
| 1401 |
+
source = RUNS_DIR / f"{run_id}.json"
|
| 1402 |
+
if not source.is_file():
|
| 1403 |
+
return False
|
| 1404 |
+
try:
|
| 1405 |
+
RUNS_ARCHIVE_DIR.mkdir(parents=True, exist_ok=True)
|
| 1406 |
+
destination = RUNS_ARCHIVE_DIR / source.name
|
| 1407 |
+
if destination.exists():
|
| 1408 |
+
destination.unlink()
|
| 1409 |
+
source.replace(destination)
|
| 1410 |
+
except OSError:
|
| 1411 |
+
return False
|
| 1412 |
+
with _lock:
|
| 1413 |
+
_invalidate_runs_cache()
|
| 1414 |
+
return True
|
| 1415 |
|
| 1416 |
|
| 1417 |
def get_run(run_id: str) -> dict | None:
|
| 1418 |
+
"""Load one run by id from the live queue, falling back to the archive."""
|
| 1419 |
+
run = _read_json(RUNS_DIR / f"{run_id}.json", None)
|
| 1420 |
+
if isinstance(run, dict):
|
| 1421 |
+
return run
|
| 1422 |
+
return _read_json(RUNS_ARCHIVE_DIR / f"{run_id}.json", None)
|
| 1423 |
+
|
| 1424 |
+
|
| 1425 |
+
def _write_run_record(run: dict) -> None:
|
| 1426 |
+
"""Persist a run, keeping finished ones out of the live queue directory.
|
| 1427 |
+
|
| 1428 |
+
Callers must hold ``_lock``. Terminal runs are archived so every
|
| 1429 |
+
completion path (worker complete/fail, dispatch, API status writes, the
|
| 1430 |
+
supersede branch) stops contributing to the claim scan without each site
|
| 1431 |
+
having to remember. A run that leaves a terminal state is pulled back
|
| 1432 |
+
into the live queue.
|
| 1433 |
+
"""
|
| 1434 |
+
run_id = str(run.get("id") or "")
|
| 1435 |
+
_write_json(RUNS_DIR / f"{run_id}.json", run)
|
| 1436 |
+
if not run_id:
|
| 1437 |
+
return
|
| 1438 |
+
if run.get("status") in _TERMINAL_RUN_STATUSES:
|
| 1439 |
+
try:
|
| 1440 |
+
RUNS_ARCHIVE_DIR.mkdir(parents=True, exist_ok=True)
|
| 1441 |
+
live = RUNS_DIR / f"{run_id}.json"
|
| 1442 |
+
if live.is_file():
|
| 1443 |
+
live.replace(RUNS_ARCHIVE_DIR / f"{run_id}.json")
|
| 1444 |
+
except OSError:
|
| 1445 |
+
pass
|
| 1446 |
+
else:
|
| 1447 |
+
# Re-queued after a re-run: drop the stale archived copy so the run
|
| 1448 |
+
# is not listed twice.
|
| 1449 |
+
stale = RUNS_ARCHIVE_DIR / f"{run_id}.json"
|
| 1450 |
+
try:
|
| 1451 |
+
if stale.is_file():
|
| 1452 |
+
stale.unlink()
|
| 1453 |
+
except OSError:
|
| 1454 |
+
pass
|
| 1455 |
|
| 1456 |
|
| 1457 |
def save_run(run: dict) -> None:
|
| 1458 |
+
"""Write a run, keeping finished runs out of the live queue directory."""
|
| 1459 |
with _lock:
|
| 1460 |
+
_write_run_record(run)
|
| 1461 |
+
_invalidate_runs_cache()
|
|
|
|
| 1462 |
|
| 1463 |
|
| 1464 |
def update_run_if_owned(
|
|
|
|
| 1468 |
mutator: Callable[[dict], None],
|
| 1469 |
) -> dict | None:
|
| 1470 |
"""Atomically mutate a running job only for its current lease owner."""
|
|
|
|
| 1471 |
with _lock:
|
| 1472 |
path = RUNS_DIR / f"{run_id}.json"
|
| 1473 |
run = _read_json(path, None)
|
|
|
|
| 1478 |
or run.get("lease_token") != lease_token):
|
| 1479 |
return None
|
| 1480 |
mutator(run)
|
| 1481 |
+
# The mutator may mark the run completed/failed; archiving happens in
|
| 1482 |
+
# the shared writer so this path stays in sync with save_run.
|
| 1483 |
+
_write_run_record(run)
|
| 1484 |
+
_invalidate_runs_cache()
|
| 1485 |
return run
|
| 1486 |
|
| 1487 |
|
|
|
|
| 1502 |
``workflow_revision`` 时,queued/running 仍视为占用,completed 则
|
| 1503 |
允许第一次迁移到带版本的幂等记录。
|
| 1504 |
"""
|
|
|
|
| 1505 |
with _lock:
|
| 1506 |
revision = run.get("workflow_revision")
|
| 1507 |
for existing in list_runs():
|
|
|
|
| 1516 |
existing["status"] = "superseded"
|
| 1517 |
existing["finished_at"] = utcnow_iso()
|
| 1518 |
existing["error_log"] = "Superseded by a newer upload or reprocess request"
|
| 1519 |
+
_write_run_record(existing)
|
| 1520 |
+
_invalidate_runs_cache()
|
|
|
|
| 1521 |
continue
|
| 1522 |
if same_revision or legacy_active:
|
| 1523 |
if existing.get("status") in ("queued", "running"):
|
| 1524 |
return existing, False
|
| 1525 |
if existing.get("status") == "completed" and not allow_completed_rerun:
|
| 1526 |
return existing, False
|
| 1527 |
+
_write_run_record(run)
|
| 1528 |
+
_invalidate_runs_cache()
|
|
|
|
| 1529 |
return run, True
|
| 1530 |
|
| 1531 |
|
| 1532 |
def delete_run(run_id: str) -> None:
|
| 1533 |
+
"""Remove a run from the live queue and from the archive."""
|
| 1534 |
_remove_json(RUNS_DIR / f"{run_id}.json")
|
| 1535 |
+
_remove_json(RUNS_ARCHIVE_DIR / f"{run_id}.json")
|
| 1536 |
+
_invalidate_runs_cache()
|
processor/app/processing/modules/lerobot_export.py
CHANGED
|
@@ -21,9 +21,20 @@ class LeRobotExportModule(ProcessingModule):
|
|
| 21 |
color = EXPORT_COLOR
|
| 22 |
inputs = ({"key": "data", "label": "Exportable Data"},)
|
| 23 |
outputs = ({"key": "dataset", "label": "Dataset"},)
|
| 24 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 25 |
config_schema = (
|
| 26 |
field("version", "select", "Version", "v3.0", options=["v2.1", "v3.0"]),
|
|
|
|
|
|
|
| 27 |
)
|
| 28 |
execution_target = "worker"
|
| 29 |
capabilities = ("dataset_export", "lerobot")
|
|
@@ -80,6 +91,12 @@ class LeRobotExportModule(ProcessingModule):
|
|
| 80 |
None)
|
| 81 |
# 数据集先输出到 worker 临时目录,API 完成阶段会把导出产品放到
|
| 82 |
# state/exports,不在项目根目录新增 processed/。
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 83 |
dataset_dir = build_lerobot_dataset(episode_id, [episode_id],
|
| 84 |
ctx.output_root, split_ratio,
|
| 85 |
include_video_keys=include_video_keys,
|
|
@@ -87,7 +104,8 @@ class LeRobotExportModule(ProcessingModule):
|
|
| 87 |
hand_3d_paths=hand_3d_paths,
|
| 88 |
hand_3d_right_paths=hand_3d_right_paths,
|
| 89 |
version=version,
|
| 90 |
-
hand_3d_unit=hand_3d_unit
|
|
|
|
| 91 |
info = dataset_dir / "meta" / "info.json"
|
| 92 |
if not info.exists():
|
| 93 |
ctx.skip("Dataset build produced no meta/info.json")
|
|
|
|
| 21 |
color = EXPORT_COLOR
|
| 22 |
inputs = ({"key": "data", "label": "Exportable Data"},)
|
| 23 |
outputs = ({"key": "dataset", "label": "Dataset"},)
|
| 24 |
+
# 默认排除 UMI 触觉力矩阵:每帧 250×250×3(187500 个整数),实测占
|
| 25 |
+
# canonical parquet 体积的九成以上,且 99.99% 是零 —— 训练用不到,
|
| 26 |
+
# 却让单集导出从 2.5 秒涨到 57 秒(写盘 + 逐元素统计 + 结果包上传),
|
| 27 |
+
# 结果 zip 也从几 MB 涨到几十 MB。要保留就把它从配置里删掉。
|
| 28 |
+
_DEFAULT_EXCLUDE = ",".join((
|
| 29 |
+
"observation.gripper_left_force_matrix",
|
| 30 |
+
"observation.gripper_right_force_matrix",
|
| 31 |
+
))
|
| 32 |
+
default_config = {"version": "v3.0", "split_ratio": 0.9,
|
| 33 |
+
"shard_size": 100000, "exclude_columns": _DEFAULT_EXCLUDE}
|
| 34 |
config_schema = (
|
| 35 |
field("version", "select", "Version", "v3.0", options=["v2.1", "v3.0"]),
|
| 36 |
+
field("exclude_columns", "string", "Exclude columns (comma separated)",
|
| 37 |
+
_DEFAULT_EXCLUDE),
|
| 38 |
)
|
| 39 |
execution_target = "worker"
|
| 40 |
capabilities = ("dataset_export", "lerobot")
|
|
|
|
| 91 |
None)
|
| 92 |
# 数据集先输出到 worker 临时目录,API 完成阶段会把导出产品放到
|
| 93 |
# state/exports,不在项目根目录新增 processed/。
|
| 94 |
+
# 排除列:配置里逗号分隔。留空 = 全量导出(旧行为)。
|
| 95 |
+
exclude_columns = [
|
| 96 |
+
name.strip() for name
|
| 97 |
+
in str(ctx.config.get("exclude_columns", "") or "").split(",")
|
| 98 |
+
if name.strip()
|
| 99 |
+
]
|
| 100 |
dataset_dir = build_lerobot_dataset(episode_id, [episode_id],
|
| 101 |
ctx.output_root, split_ratio,
|
| 102 |
include_video_keys=include_video_keys,
|
|
|
|
| 104 |
hand_3d_paths=hand_3d_paths,
|
| 105 |
hand_3d_right_paths=hand_3d_right_paths,
|
| 106 |
version=version,
|
| 107 |
+
hand_3d_unit=hand_3d_unit,
|
| 108 |
+
exclude_columns=exclude_columns)
|
| 109 |
info = dataset_dir / "meta" / "info.json"
|
| 110 |
if not info.exists():
|
| 111 |
ctx.skip("Dataset build produced no meta/info.json")
|
processor/app/processing/modules/umi_slam_action.py
ADDED
|
@@ -0,0 +1,192 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""UMI Slam Action — 从 SLAM 位姿派生 imitation learning 的 action 列。
|
| 2 |
+
|
| 3 |
+
UMI 采集端不录 action(写的是占位全零,导出时被恒定列过滤器剔除),
|
| 4 |
+
导致数据集无法训练。本模块读取 UMI 夹爪的 SLAM 位姿与夹爪开合,
|
| 5 |
+
按 UMI 官方口径反推出 7 维 action 并合并回 canonical:
|
| 6 |
+
|
| 7 |
+
Δp = R_tᵀ · (p_{t+1} − p_t) # 平移增量, 米
|
| 8 |
+
Δr = rotvec(R_tᵀ · R_{t+1}) # 旋转增量, 弧度
|
| 9 |
+
g = gripper_state[t+1, 0] / 100 # 夹爪开合, 0–1
|
| 10 |
+
|
| 11 |
+
纯计算在 app/umi_slam_action.py(不依赖框架,便于单测与跨格式复用)。
|
| 12 |
+
"""
|
| 13 |
+
|
| 14 |
+
from pathlib import Path
|
| 15 |
+
|
| 16 |
+
from app.processing import ProcessingModule, JobContext, ArtifactRef
|
| 17 |
+
from app.processing.registry import register
|
| 18 |
+
from app.processing.theme import HAND3D_COLOR
|
| 19 |
+
|
| 20 |
+
# 本模块产出的 artifact kind。合并白名单在 project_dataset 里单独维护,
|
| 21 |
+
# 新增 kind 必须同步加进那张表,否则列不会被并入 canonical。
|
| 22 |
+
ACTION_KIND = "umi_slam_action"
|
| 23 |
+
|
| 24 |
+
# 位姿只读 slam_trajectory:它是采集端的高频样本缓冲,丢帧时相邻格子里
|
| 25 |
+
# 仍有真实样本;而 slam_pose 在同样位置直接写零,只能插值硬补。
|
| 26 |
+
_TRAJECTORY_COLUMN = "observation.slam_trajectory"
|
| 27 |
+
_TIMESTAMP_COLUMN = "timestamp"
|
| 28 |
+
_GRIPPER_COLUMN = "observation.gripper_state"
|
| 29 |
+
|
| 30 |
+
|
| 31 |
+
@register
|
| 32 |
+
class UmiSlamActionModule(ProcessingModule):
|
| 33 |
+
slug = ACTION_KIND
|
| 34 |
+
version = "1.0"
|
| 35 |
+
category = "process"
|
| 36 |
+
label = "UMI Slam Action"
|
| 37 |
+
# 语义是"位姿 → 增量"的坐标变换,与同组做几何变换的手部模块一致;
|
| 38 |
+
# sync 也贴合"由轨迹反推动作"。必须取自 iconify-preload.js 的白名单
|
| 39 |
+
# (图标离线预载,不在表内的名字渲染为空白)。
|
| 40 |
+
icon = "ant-design:sync-outlined"
|
| 41 |
+
color = HAND3D_COLOR
|
| 42 |
+
# 端口 key 必须与上游 UMI Gripper 的输出 key 逐字一致 —— 前端对
|
| 43 |
+
# 类型化端口按 ``out.key === inp.key`` 校验,改名会导致连线被拒。
|
| 44 |
+
inputs = (
|
| 45 |
+
{"key": "slam_trajectory", "label": "SLAM Trajectory"},
|
| 46 |
+
{"key": "gripper_state", "label": "ESP Gripper State"},
|
| 47 |
+
)
|
| 48 |
+
outputs = ({"key": "action", "label": "Action"},)
|
| 49 |
+
default_config = {}
|
| 50 |
+
config_schema = ()
|
| 51 |
+
execution_target = "worker"
|
| 52 |
+
capabilities = ("umi_gripper", "action_derivation")
|
| 53 |
+
|
| 54 |
+
def run(self, ctx: JobContext) -> dict[str, ArtifactRef] | None:
|
| 55 |
+
parquet = self._find_source_parquet(ctx)
|
| 56 |
+
if parquet is None:
|
| 57 |
+
ctx.skip("No parquet with SLAM pose found in batch")
|
| 58 |
+
|
| 59 |
+
from app.umi_slam_action import derive_action
|
| 60 |
+
import time
|
| 61 |
+
import numpy as np
|
| 62 |
+
import pandas as pd
|
| 63 |
+
|
| 64 |
+
# 读取重试:采集端可能还在往同一项目目录写新集(每上传一集都会重写
|
| 65 |
+
# 项目数据树),worker 恰好读到写了一半的文件就会抛异常。实测
|
| 66 |
+
# ep52/ep71 都因此被误判为"列不可读"而跳过 —— 几秒后同一文件完全
|
| 67 |
+
# 正常。瞬时故障不该等同于永久失败,故重试若干次。
|
| 68 |
+
columns = [_TRAJECTORY_COLUMN, _TIMESTAMP_COLUMN, _GRIPPER_COLUMN]
|
| 69 |
+
frame = None
|
| 70 |
+
last_error: Exception | None = None
|
| 71 |
+
for attempt in range(5):
|
| 72 |
+
try:
|
| 73 |
+
frame = pd.read_parquet(parquet, columns=columns)
|
| 74 |
+
break
|
| 75 |
+
except Exception as exc:
|
| 76 |
+
last_error = exc
|
| 77 |
+
time.sleep(1.0 + attempt)
|
| 78 |
+
if frame is None:
|
| 79 |
+
# 带上路径与大小:同样的 ArrowInvalid 可能来自「文件被截断」或
|
| 80 |
+
# 「根本不是 parquet」,只看异常分不出来,必须知道读的是哪个文件。
|
| 81 |
+
try:
|
| 82 |
+
size = parquet.stat().st_size
|
| 83 |
+
head = parquet.open("rb").read(4)
|
| 84 |
+
detail = f"size={size}B head={head!r}"
|
| 85 |
+
except OSError as exc:
|
| 86 |
+
detail = f"stat failed: {exc}"
|
| 87 |
+
ctx.skip(f"SLAM columns unreadable after retries: "
|
| 88 |
+
f"{type(last_error).__name__}: {last_error} | "
|
| 89 |
+
f"file={parquet} ({detail})")
|
| 90 |
+
|
| 91 |
+
if _TRAJECTORY_COLUMN not in frame.columns:
|
| 92 |
+
# 非 UMI 批次(如 D435 手套、S80C)没有 SLAM 缓冲 —— 跳过,
|
| 93 |
+
# 不写入任何列,保证其他设备的导出行为完全不变。
|
| 94 |
+
ctx.skip("No SLAM trajectory column — not a UMI gripper batch")
|
| 95 |
+
|
| 96 |
+
total = len(frame)
|
| 97 |
+
stamps = (frame[_TIMESTAMP_COLUMN].to_numpy(dtype=float)
|
| 98 |
+
if _TIMESTAMP_COLUMN in frame.columns else np.arange(total) / 30.0)
|
| 99 |
+
derived = derive_action(
|
| 100 |
+
frame[_TRAJECTORY_COLUMN].tolist(),
|
| 101 |
+
stamps,
|
| 102 |
+
frame[_GRIPPER_COLUMN].tolist()
|
| 103 |
+
if _GRIPPER_COLUMN in frame.columns else [],
|
| 104 |
+
total,
|
| 105 |
+
)
|
| 106 |
+
if not derived:
|
| 107 |
+
ctx.skip("No usable SLAM samples (all frames dropped)")
|
| 108 |
+
|
| 109 |
+
# 输出按原 frame_index 对齐并保持升序:_merge_processing_frame 按
|
| 110 |
+
# frame_index 匹配,乱序会让列���位。
|
| 111 |
+
frames = sorted(derived)
|
| 112 |
+
ctx.progress(0.5)
|
| 113 |
+
output = ctx.output_dir / "umi_slam_action.parquet"
|
| 114 |
+
# 必须写 np.float32:pyarrow 对 Python float 推断 double,而 LeRobot
|
| 115 |
+
# 的 action 声明是 float32 —— 声明与实际 schema 不一致会让官方加载器
|
| 116 |
+
# 读出不匹配的类型。与导出端 _cast_float32 的做法保持一致。
|
| 117 |
+
pd.DataFrame({
|
| 118 |
+
"frame_index": [int(f) for f in frames],
|
| 119 |
+
"action": [[np.float32(value) for value in derived[f]["action"]]
|
| 120 |
+
for f in frames],
|
| 121 |
+
# ACT 的观测需要 proprioception:只有图像时模型不知道夹爪当前
|
| 122 |
+
# 在哪。这里的 observation.state 是相对本集起点的位置 + 夹爪开合。
|
| 123 |
+
"observation.state": [
|
| 124 |
+
[np.float32(value) for value in derived[f]["observation.state"]]
|
| 125 |
+
for f in frames],
|
| 126 |
+
}).to_parquet(output, index=False)
|
| 127 |
+
ctx.progress(1.0)
|
| 128 |
+
|
| 129 |
+
covered = len(frames)
|
| 130 |
+
return {
|
| 131 |
+
"action": ctx.ref(ACTION_KIND, output, metadata={
|
| 132 |
+
"column": "action",
|
| 133 |
+
"shape": [7],
|
| 134 |
+
"names": ["delta_x", "delta_y", "delta_z",
|
| 135 |
+
"rot_x", "rot_y", "rot_z", "gripper"],
|
| 136 |
+
"frames_covered": covered,
|
| 137 |
+
"frames_total": int(total),
|
| 138 |
+
"note": ("UMI 口径 action:当前夹爪局部系增量位姿 + 夹爪开合。"
|
| 139 |
+
"由 SLAM 位姿派生,零位姿丢帧已插值修复。"),
|
| 140 |
+
}),
|
| 141 |
+
}
|
| 142 |
+
|
| 143 |
+
@staticmethod
|
| 144 |
+
def _find_source_parquet(ctx: JobContext) -> Path | None:
|
| 145 |
+
"""定位含 SLAM 位姿的 canonical parquet。
|
| 146 |
+
|
| 147 |
+
优先用上游 ``slam_trajectory`` 端口连过来的产物(工作流连接驱动);
|
| 148 |
+
未连或产物不可用时回退到 ``ctx.find_parquet()`` 扫描批次,
|
| 149 |
+
保证"只连一根线也能跑"。
|
| 150 |
+
|
| 151 |
+
上游 ref **必须验证是 parquet**:worker 组装 incoming 时,若边的
|
| 152 |
+
sourceHandle 对不上上游任何输出端口,会把该节点的**全部输出平铺**
|
| 153 |
+
进来(runner `_incoming_artifacts` 规则 3),视频 ref 也会挂到
|
| 154 |
+
``slam_trajectory`` 这个键上。实测因此把 mp4 当 parquet 读,报
|
| 155 |
+
``magic bytes not found in footer`` —— 只看键名不验证就会中招。
|
| 156 |
+
|
| 157 |
+
回退扫描**必须按 episode_id 过滤**:批次目录里往往躺着几十集的
|
| 158 |
+
canonical parquet,直接取第一个会静默用错集的数据(实测表现为
|
| 159 |
+
每集算出的 action 完全相同)。worker 打包输入时通常只含一集,
|
| 160 |
+
但本地重跑/整目录传入时就不是了。
|
| 161 |
+
"""
|
| 162 |
+
import pyarrow.parquet as pq
|
| 163 |
+
|
| 164 |
+
def _is_slam_parquet(candidate: Path) -> bool:
|
| 165 |
+
if candidate.suffix.lower() != ".parquet" or not candidate.is_file():
|
| 166 |
+
return False
|
| 167 |
+
try:
|
| 168 |
+
return _TRAJECTORY_COLUMN in set(pq.read_schema(candidate).names)
|
| 169 |
+
except Exception:
|
| 170 |
+
return False
|
| 171 |
+
|
| 172 |
+
for key, ref in (ctx.incoming or {}).items():
|
| 173 |
+
if not str(key).startswith("slam_trajectory"):
|
| 174 |
+
continue
|
| 175 |
+
resolved = ctx.resolve(ref)
|
| 176 |
+
if resolved and _is_slam_parquet(resolved):
|
| 177 |
+
return resolved
|
| 178 |
+
|
| 179 |
+
episode_id = str((ctx.job or {}).get("episode_id") or "")
|
| 180 |
+
suffix = ""
|
| 181 |
+
if "_" in episode_id:
|
| 182 |
+
tail = episode_id.rsplit("_", 1)[-1]
|
| 183 |
+
if tail.isdigit():
|
| 184 |
+
suffix = f"episode_{int(tail):06d}.parquet"
|
| 185 |
+
candidates = ctx.find_parquet()
|
| 186 |
+
# 先只看文件名匹配本集的那一个
|
| 187 |
+
ordered = ([c for c in candidates if c.name == suffix] if suffix else [])
|
| 188 |
+
ordered += [c for c in candidates if c.name != suffix]
|
| 189 |
+
for candidate in ordered:
|
| 190 |
+
if _is_slam_parquet(candidate):
|
| 191 |
+
return candidate
|
| 192 |
+
return None
|
processor/app/project_dataset.py
CHANGED
|
@@ -12,10 +12,13 @@ migration command and the upload path so both produce the same layout.
|
|
| 12 |
|
| 13 |
from __future__ import annotations
|
| 14 |
|
|
|
|
| 15 |
import json
|
| 16 |
import os
|
| 17 |
import re
|
| 18 |
import shutil
|
|
|
|
|
|
|
| 19 |
from datetime import datetime, timezone
|
| 20 |
from pathlib import Path
|
| 21 |
from typing import Any, Iterable
|
|
@@ -442,7 +445,42 @@ def episode_index_from_name(name: str) -> int | None:
|
|
| 442 |
return int(match.group(1)) if match else None
|
| 443 |
|
| 444 |
|
| 445 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 446 |
rows: list[dict[str, Any]] = []
|
| 447 |
for path in _episode_parquet_files(project_root):
|
| 448 |
try:
|
|
@@ -455,6 +493,63 @@ def project_episode_rows(project_root: Path) -> list[dict[str, Any]]:
|
|
| 455 |
return rows
|
| 456 |
|
| 457 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 458 |
def _match_episode_row(rows: list[dict[str, Any]],
|
| 459 |
episode_id: str) -> dict[str, Any] | None:
|
| 460 |
"""Match an episode row by canonical id first, then trailing index.
|
|
@@ -519,31 +614,86 @@ def _chunk_number_from_path(path: Path) -> int:
|
|
| 519 |
return 0
|
| 520 |
|
| 521 |
|
| 522 |
-
def
|
| 523 |
-
|
|
|
|
|
|
|
| 524 |
root = Path(project_root)
|
| 525 |
-
|
| 526 |
-
|
| 527 |
-
|
| 528 |
-
|
| 529 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 530 |
for source, path in iter_video_streams(root / "videos"):
|
| 531 |
-
|
| 532 |
-
|
| 533 |
-
|
| 534 |
-
|
| 535 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 536 |
relative_meta: list[Path] = [Path("meta/info.json"), Path("meta/tasks.json")]
|
| 537 |
return {
|
| 538 |
"episode_index": episode_index,
|
| 539 |
-
"episode_id":
|
| 540 |
"row": row,
|
| 541 |
-
"data":
|
| 542 |
-
"videos": videos,
|
| 543 |
"meta": [path for path in relative_meta if (root / path).is_file()],
|
| 544 |
}
|
| 545 |
|
| 546 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 547 |
def _processing_token(value: Any, fallback: str = "result") -> str:
|
| 548 |
text = re.sub(r"[^A-Za-z0-9_.-]+", "_", str(value or "")).strip("._-")
|
| 549 |
return text or fallback
|
|
@@ -706,7 +856,8 @@ def publish_processing_result(
|
|
| 706 |
records = _processing_artifacts(outputs)
|
| 707 |
tabular_records = [item for item in records
|
| 708 |
if str(item[2].get("kind") or "") in
|
| 709 |
-
{"hand_keypoints", "hand_3d", "glove_sensor"
|
|
|
|
| 710 |
# A reprocess is authoritative for its hand-3D sources. Remove any
|
| 711 |
# source-qualified hand-3D columns left by an earlier partial publish
|
| 712 |
# (including legacy ``*_1`` collision columns) before merging the fresh
|
|
@@ -717,10 +868,26 @@ def publish_processing_result(
|
|
| 717 |
if str(name).startswith("processing.hand_3d.")]
|
| 718 |
if stale_hand3d:
|
| 719 |
data_frame = data_frame.drop(columns=stale_hand3d, errors="ignore")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 720 |
assigned_columns: set[str] = set(data_frame.columns)
|
| 721 |
merged_columns: list[str] = []
|
| 722 |
published = {"artifacts": {}}
|
| 723 |
merged_hand3d_sources: set[str] = set()
|
|
|
|
| 724 |
|
| 725 |
for node_id, handle, ref in records:
|
| 726 |
source_path = _processing_ref_path(Path(extracted_root), ref)
|
|
@@ -734,6 +901,24 @@ def publish_processing_result(
|
|
| 734 |
published_ref = dict(ref)
|
| 735 |
source = _processing_token(ref.get("source_key") or metadata.get("source_key")
|
| 736 |
or handle, kind)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 737 |
# Review/export nodes pass the node-36 hand-3D refs through again.
|
| 738 |
# They are the same source data, not a second measurement. Merge each
|
| 739 |
# source once and keep the passthrough artifact only as an audit link.
|
|
@@ -760,7 +945,7 @@ def publish_processing_result(
|
|
| 760 |
"storage": "browser_overlay"}
|
| 761 |
published["artifacts"].setdefault(node_id, {})[handle] = published_ref
|
| 762 |
continue
|
| 763 |
-
if kind in {"hand_keypoints", "hand_3d", "glove_sensor"}:
|
| 764 |
result_frame = pd.read_parquet(source_path)
|
| 765 |
raw_columns = [str(name) for name in result_frame.columns
|
| 766 |
if name not in {"frame_index", "episode_index"}]
|
|
|
|
| 12 |
|
| 13 |
from __future__ import annotations
|
| 14 |
|
| 15 |
+
import copy
|
| 16 |
import json
|
| 17 |
import os
|
| 18 |
import re
|
| 19 |
import shutil
|
| 20 |
+
import threading
|
| 21 |
+
import time
|
| 22 |
from datetime import datetime, timezone
|
| 23 |
from pathlib import Path
|
| 24 |
from typing import Any, Iterable
|
|
|
|
| 445 |
return int(match.group(1)) if match else None
|
| 446 |
|
| 447 |
|
| 448 |
+
# ── Project index cache ───────────────────────────────
|
| 449 |
+
# Every _canonical_episode_data_files() call reads the complete episode
|
| 450 |
+
# metadata parquet set (52 episodes = 52 remote reads) plus two project-wide
|
| 451 |
+
# directory walks — measured at 1.4s warm, and the review page issues ~11
|
| 452 |
+
# video requests per episode open, so the same index was rebuilt a dozen
|
| 453 |
+
# times per click. Cache the parsed rows and the file index against a cheap
|
| 454 |
+
# fingerprint of the episode index files.
|
| 455 |
+
_PROJECT_CACHE_LOCK = threading.RLock()
|
| 456 |
+
_PROJECT_CACHE: dict[str, dict[str, Any]] = {}
|
| 457 |
+
# Computing the fingerprint still stats every episode index file, so a short
|
| 458 |
+
# TTL skips even that inside one page load. Structural changes are announced
|
| 459 |
+
# explicitly through invalidate_project_cache().
|
| 460 |
+
_PROJECT_CACHE_TTL = 60.0
|
| 461 |
+
_PROJECT_SCAN_LOCK = threading.Lock()
|
| 462 |
+
|
| 463 |
+
|
| 464 |
+
def _project_source_revision(project_root: Path) -> tuple:
|
| 465 |
+
"""Fingerprint the episode index files (name, size, mtime).
|
| 466 |
+
|
| 467 |
+
The index files are written by whoever writes the data (upload, workflow
|
| 468 |
+
merge), so a change there means the project changed. ``data/`` and
|
| 469 |
+
``videos/`` are deliberately not walked: any addition or removal there is
|
| 470 |
+
accompanied by an index write.
|
| 471 |
+
"""
|
| 472 |
+
parts: list[tuple] = []
|
| 473 |
+
try:
|
| 474 |
+
for path in _episode_parquet_files(project_root):
|
| 475 |
+
stat = path.stat()
|
| 476 |
+
parts.append((path.name, stat.st_size, stat.st_mtime_ns))
|
| 477 |
+
except OSError:
|
| 478 |
+
return ()
|
| 479 |
+
return tuple(parts)
|
| 480 |
+
|
| 481 |
+
|
| 482 |
+
def _read_project_rows(project_root: Path) -> list[dict[str, Any]]:
|
| 483 |
+
"""Uncached: one parquet read per episode, sorted by episode index."""
|
| 484 |
rows: list[dict[str, Any]] = []
|
| 485 |
for path in _episode_parquet_files(project_root):
|
| 486 |
try:
|
|
|
|
| 493 |
return rows
|
| 494 |
|
| 495 |
|
| 496 |
+
def _project_cache_entry(project_root: Path) -> dict[str, Any]:
|
| 497 |
+
"""Return the cached ``{rows, index}`` for a project, rebuilding if stale."""
|
| 498 |
+
root = Path(project_root)
|
| 499 |
+
key = str(root)
|
| 500 |
+
now = time.monotonic()
|
| 501 |
+
with _PROJECT_CACHE_LOCK:
|
| 502 |
+
entry = _PROJECT_CACHE.get(key)
|
| 503 |
+
if entry is not None and now - entry["checked_at"] < _PROJECT_CACHE_TTL:
|
| 504 |
+
return entry
|
| 505 |
+
# The fingerprint touches the remote mount, so keep it outside the lock.
|
| 506 |
+
revision = _project_source_revision(root)
|
| 507 |
+
with _PROJECT_CACHE_LOCK:
|
| 508 |
+
entry = _PROJECT_CACHE.get(key)
|
| 509 |
+
if entry is not None and entry["revision"] == revision:
|
| 510 |
+
entry["checked_at"] = now
|
| 511 |
+
return entry
|
| 512 |
+
# Single-flight: a page load fires its requests in parallel, and they all
|
| 513 |
+
# miss at once. One rebuild serves the whole burst.
|
| 514 |
+
with _PROJECT_SCAN_LOCK:
|
| 515 |
+
now = time.monotonic()
|
| 516 |
+
with _PROJECT_CACHE_LOCK:
|
| 517 |
+
entry = _PROJECT_CACHE.get(key)
|
| 518 |
+
if entry is not None and (
|
| 519 |
+
entry["revision"] == revision
|
| 520 |
+
or now - entry["checked_at"] < _PROJECT_CACHE_TTL):
|
| 521 |
+
return entry
|
| 522 |
+
rows = _read_project_rows(root)
|
| 523 |
+
index = _build_project_file_index(root, rows)
|
| 524 |
+
entry = {
|
| 525 |
+
"revision": revision,
|
| 526 |
+
"checked_at": time.monotonic(),
|
| 527 |
+
"rows": rows,
|
| 528 |
+
"index": index,
|
| 529 |
+
}
|
| 530 |
+
with _PROJECT_CACHE_LOCK:
|
| 531 |
+
_PROJECT_CACHE[key] = entry
|
| 532 |
+
return entry
|
| 533 |
+
|
| 534 |
+
|
| 535 |
+
def invalidate_project_cache(project_root: Path | None = None) -> None:
|
| 536 |
+
"""Drop cached project indexes after a structural change.
|
| 537 |
+
|
| 538 |
+
Called from ``localstore.invalidate_session_cache`` so an upload, delete or
|
| 539 |
+
rename becomes visible on the next read instead of after the TTL.
|
| 540 |
+
"""
|
| 541 |
+
with _PROJECT_CACHE_LOCK:
|
| 542 |
+
if project_root is None:
|
| 543 |
+
_PROJECT_CACHE.clear()
|
| 544 |
+
else:
|
| 545 |
+
_PROJECT_CACHE.pop(str(Path(project_root)), None)
|
| 546 |
+
|
| 547 |
+
|
| 548 |
+
def project_episode_rows(project_root: Path) -> list[dict[str, Any]]:
|
| 549 |
+
# Callers may assume they own the rows, so never hand out the cache itself.
|
| 550 |
+
return copy.deepcopy(_project_cache_entry(project_root)["rows"])
|
| 551 |
+
|
| 552 |
+
|
| 553 |
def _match_episode_row(rows: list[dict[str, Any]],
|
| 554 |
episode_id: str) -> dict[str, Any] | None:
|
| 555 |
"""Match an episode row by canonical id first, then trailing index.
|
|
|
|
| 614 |
return 0
|
| 615 |
|
| 616 |
|
| 617 |
+
def _build_project_file_index(
|
| 618 |
+
project_root: Path, rows: list[dict[str, Any]] | None = None
|
| 619 |
+
) -> dict[int, dict[str, Any]]:
|
| 620 |
+
"""Uncached: one pass over ``data/``, ``videos/`` and the episode rows."""
|
| 621 |
root = Path(project_root)
|
| 622 |
+
index: dict[int, dict[str, Any]] = {}
|
| 623 |
+
data_root = root / "data"
|
| 624 |
+
if data_root.is_dir():
|
| 625 |
+
for path in data_root.rglob("*.parquet"):
|
| 626 |
+
if not path.is_file():
|
| 627 |
+
continue
|
| 628 |
+
number = _episode_number_from_path(path)
|
| 629 |
+
if number is None:
|
| 630 |
+
continue
|
| 631 |
+
index.setdefault(number, {"data": []})["data"].append(path)
|
| 632 |
for source, path in iter_video_streams(root / "videos"):
|
| 633 |
+
number = _episode_number_from_path(path)
|
| 634 |
+
if number is None:
|
| 635 |
+
continue
|
| 636 |
+
index.setdefault(number, {"data": []}).setdefault(
|
| 637 |
+
"videos", []).append((source, path))
|
| 638 |
+
if rows is None:
|
| 639 |
+
rows = _read_project_rows(root)
|
| 640 |
+
for row in rows:
|
| 641 |
+
number = _episode_index(row.get("episode_index"))
|
| 642 |
+
if number is None:
|
| 643 |
+
continue
|
| 644 |
+
# ``episode_files`` resolves a duplicate episode number to the first
|
| 645 |
+
# row in index order; ``setdefault`` keeps that same winner.
|
| 646 |
+
index.setdefault(number, {"data": []}).setdefault("row", row)
|
| 647 |
+
return index
|
| 648 |
+
|
| 649 |
+
|
| 650 |
+
def project_episode_file_index(project_root: Path) -> dict[int, dict[str, Any]]:
|
| 651 |
+
"""Index every episode's source files with a single pass over the project.
|
| 652 |
+
|
| 653 |
+
:func:`episode_files` re-walks ``data/``, ``videos/`` and the whole episode
|
| 654 |
+
index for each episode it is asked about, so scanning a project costs
|
| 655 |
+
``(1 + episodes) x index files`` parquet reads on the (possibly remote)
|
| 656 |
+
storage mount. This entry point keeps that contract while serving repeat
|
| 657 |
+
callers from the revision-checked project cache.
|
| 658 |
+
|
| 659 |
+
Keys are episode numbers; values carry the same ``data`` / ``videos`` /
|
| 660 |
+
``row`` payload :func:`episode_files` reports, and are ordered exactly as
|
| 661 |
+
a per-episode walk would have produced them.
|
| 662 |
+
"""
|
| 663 |
+
return copy.deepcopy(_project_cache_entry(Path(project_root))["index"])
|
| 664 |
+
|
| 665 |
+
|
| 666 |
+
def episode_files_from_index(project_root: Path, episode_index: int,
|
| 667 |
+
index: dict[int, dict[str, Any]]) -> dict[str, Any]:
|
| 668 |
+
"""Look one episode up in a prebuilt :func:`project_episode_file_index`.
|
| 669 |
+
|
| 670 |
+
Returns the same shape as :func:`episode_files`; the caller owns the index
|
| 671 |
+
lifetime and is responsible for rebuilding it when the project changes.
|
| 672 |
+
"""
|
| 673 |
+
root = Path(project_root)
|
| 674 |
+
entry = index.get(episode_index) or {}
|
| 675 |
+
row = entry.get("row") or {}
|
| 676 |
relative_meta: list[Path] = [Path("meta/info.json"), Path("meta/tasks.json")]
|
| 677 |
return {
|
| 678 |
"episode_index": episode_index,
|
| 679 |
+
"episode_id": _episode_id_for_row(row, root.name),
|
| 680 |
"row": row,
|
| 681 |
+
"data": list(entry.get("data") or []),
|
| 682 |
+
"videos": list(entry.get("videos") or []),
|
| 683 |
"meta": [path for path in relative_meta if (root / path).is_file()],
|
| 684 |
}
|
| 685 |
|
| 686 |
|
| 687 |
+
def episode_files(project_root: Path, episode_index: int) -> dict[str, Any]:
|
| 688 |
+
"""Return only one episode's source files from a project dataset.
|
| 689 |
+
|
| 690 |
+
Single-episode callers keep this entry point; it builds a one-shot index,
|
| 691 |
+
so it now walks the project once instead of three times.
|
| 692 |
+
"""
|
| 693 |
+
return episode_files_from_index(
|
| 694 |
+
project_root, episode_index, project_episode_file_index(project_root))
|
| 695 |
+
|
| 696 |
+
|
| 697 |
def _processing_token(value: Any, fallback: str = "result") -> str:
|
| 698 |
text = re.sub(r"[^A-Za-z0-9_.-]+", "_", str(value or "")).strip("._-")
|
| 699 |
return text or fallback
|
|
|
|
| 856 |
records = _processing_artifacts(outputs)
|
| 857 |
tabular_records = [item for item in records
|
| 858 |
if str(item[2].get("kind") or "") in
|
| 859 |
+
{"hand_keypoints", "hand_3d", "glove_sensor",
|
| 860 |
+
"umi_slam_action"}]
|
| 861 |
# A reprocess is authoritative for its hand-3D sources. Remove any
|
| 862 |
# source-qualified hand-3D columns left by an earlier partial publish
|
| 863 |
# (including legacy ``*_1`` collision columns) before merging the fresh
|
|
|
|
| 868 |
if str(name).startswith("processing.hand_3d.")]
|
| 869 |
if stale_hand3d:
|
| 870 |
data_frame = data_frame.drop(columns=stale_hand3d, errors="ignore")
|
| 871 |
+
# Derived action replaces the collector's placeholder column instead of
|
| 872 |
+
# being renamed to ``processing.<kind>.*.action``. The UMI collector
|
| 873 |
+
# writes an all-zero ``action`` that the LeRobot exporter correctly drops
|
| 874 |
+
# as a constant/zero-information column, so without this drop the fresh
|
| 875 |
+
# action would land under a name no exporter reads. Only drop it when a
|
| 876 |
+
# umi_slam_action result is actually being published for this episode.
|
| 877 |
+
if any(str(ref.get("kind") or "") == "umi_slam_action"
|
| 878 |
+
for _node, _handle, ref in records):
|
| 879 |
+
# 本模块产出的列全部按原名覆盖(重跑幂等):action 之外还有
|
| 880 |
+
# observation.state,它同样不该因为二次合并被改名成 processing.*。
|
| 881 |
+
stale_action = [str(name) for name in data_frame.columns
|
| 882 |
+
if str(name) in {"action", "observation.state"}
|
| 883 |
+
or str(name).startswith("processing.umi_slam_action.")]
|
| 884 |
+
if stale_action:
|
| 885 |
+
data_frame = data_frame.drop(columns=stale_action, errors="ignore")
|
| 886 |
assigned_columns: set[str] = set(data_frame.columns)
|
| 887 |
merged_columns: list[str] = []
|
| 888 |
published = {"artifacts": {}}
|
| 889 |
merged_hand3d_sources: set[str] = set()
|
| 890 |
+
merged_action_files: set[str] = set()
|
| 891 |
|
| 892 |
for node_id, handle, ref in records:
|
| 893 |
source_path = _processing_ref_path(Path(extracted_root), ref)
|
|
|
|
| 901 |
published_ref = dict(ref)
|
| 902 |
source = _processing_token(ref.get("source_key") or metadata.get("source_key")
|
| 903 |
or handle, kind)
|
| 904 |
+
# A review/review-export node forwards its upstream action ref, so the
|
| 905 |
+
# same derived action arrives twice (once from the producing node,
|
| 906 |
+
# once forwarded). Merging both would add a duplicate
|
| 907 |
+
# ``processing.<kind>.*.action`` column holding identical values.
|
| 908 |
+
# Deduplicate on the artifact file, not the node id: two genuinely
|
| 909 |
+
# separate umi_slam_action nodes produce different files and must both
|
| 910 |
+
# merge.
|
| 911 |
+
if kind == "umi_slam_action":
|
| 912 |
+
action_file = str(source_path.resolve())
|
| 913 |
+
if action_file in merged_action_files:
|
| 914 |
+
published_ref["path"] = str(data_path.relative_to(root)).replace("\\", "/")
|
| 915 |
+
published_ref["metadata"] = {
|
| 916 |
+
**metadata, "merged_into": published_ref["path"],
|
| 917 |
+
"deduplicated": True, "merged_columns": [],
|
| 918 |
+
}
|
| 919 |
+
published["artifacts"].setdefault(node_id, {})[handle] = published_ref
|
| 920 |
+
continue
|
| 921 |
+
merged_action_files.add(action_file)
|
| 922 |
# Review/export nodes pass the node-36 hand-3D refs through again.
|
| 923 |
# They are the same source data, not a second measurement. Merge each
|
| 924 |
# source once and keep the passthrough artifact only as an audit link.
|
|
|
|
| 945 |
"storage": "browser_overlay"}
|
| 946 |
published["artifacts"].setdefault(node_id, {})[handle] = published_ref
|
| 947 |
continue
|
| 948 |
+
if kind in {"hand_keypoints", "hand_3d", "glove_sensor", "umi_slam_action"}:
|
| 949 |
result_frame = pd.read_parquet(source_path)
|
| 950 |
raw_columns = [str(name) for name in result_frame.columns
|
| 951 |
if name not in {"frame_index", "episode_index"}]
|
processor/app/routes/devices.py
CHANGED
|
@@ -1,7 +1,8 @@
|
|
| 1 |
"""Device heartbeat & polling API — 本地 JSON 存储(无数据库)。
|
| 2 |
|
| 3 |
设备存 data/state/devices.json;采集端轮询 /device/tasks 获取
|
| 4 |
-
active 项目列表(任务概念已移除,项目即采集任务)
|
|
|
|
| 5 |
"""
|
| 6 |
|
| 7 |
from __future__ import annotations
|
|
@@ -18,6 +19,8 @@ from app.localstore import (
|
|
| 18 |
list_projects,
|
| 19 |
scan_sessions,
|
| 20 |
)
|
|
|
|
|
|
|
| 21 |
from app.security import verify_api_key
|
| 22 |
from app.device_naming import decorate_device_sources, display_names_for_sources
|
| 23 |
|
|
@@ -237,18 +240,31 @@ async def device_tasks(
|
|
| 237 |
device_name: str = Query(..., min_length=1),
|
| 238 |
_: str = Depends(verify_api_key),
|
| 239 |
):
|
| 240 |
-
"""采集端轮询此接口获取 active 项目(DAQ client 格式)。
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 241 |
project_rows = await asyncio.to_thread(list_projects)
|
| 242 |
projects = [p for p in project_rows if p.get("status", "active") == "active"]
|
| 243 |
-
|
| 244 |
-
#
|
| 245 |
-
|
| 246 |
-
|
| 247 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 248 |
|
| 249 |
tasks = []
|
| 250 |
for p in projects:
|
| 251 |
-
|
|
|
|
|
|
|
| 252 |
params = p.get("params") or {}
|
| 253 |
if not isinstance(params, dict):
|
| 254 |
params = {}
|
|
@@ -261,6 +277,37 @@ async def device_tasks(
|
|
| 261 |
"current_count": cur,
|
| 262 |
"assigned_at": p.get("created_at"),
|
| 263 |
"params": params,
|
|
|
|
|
|
|
|
|
|
| 264 |
})
|
| 265 |
|
| 266 |
return {"tasks": tasks, "updated_at": utcnow().isoformat()}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
"""Device heartbeat & polling API — 本地 JSON 存储(无数据库)。
|
| 2 |
|
| 3 |
设备存 data/state/devices.json;采集端轮询 /device/tasks 获取
|
| 4 |
+
active 项目列表(任务概念已移除,项目即采集任务);采集端「录制完成」的
|
| 5 |
+
增量上报存 data/state/device_task_progress.json(见 app/device_progress.py)。
|
| 6 |
"""
|
| 7 |
|
| 8 |
from __future__ import annotations
|
|
|
|
| 19 |
list_projects,
|
| 20 |
scan_sessions,
|
| 21 |
)
|
| 22 |
+
from app import device_progress
|
| 23 |
+
from app.device_progress import reported_totals
|
| 24 |
from app.security import verify_api_key
|
| 25 |
from app.device_naming import decorate_device_sources, display_names_for_sources
|
| 26 |
|
|
|
|
| 240 |
device_name: str = Query(..., min_length=1),
|
| 241 |
_: str = Depends(verify_api_key),
|
| 242 |
):
|
| 243 |
+
"""采集端轮询此接口获取 active 项目(DAQ client 格式)。
|
| 244 |
+
|
| 245 |
+
current_count 口径:各设备上报的录制完成数之和(见 app/device_progress.py);
|
| 246 |
+
只有从未收到过上报的项目才回退到「项目目录下的 session 数」,让升级前就
|
| 247 |
+
存在的历史项目不归零。session 是**上传**的产物,早先直接拿它当进度会让
|
| 248 |
+
同一条数据被算两次(录制一次、上传一次)。
|
| 249 |
+
"""
|
| 250 |
project_rows = await asyncio.to_thread(list_projects)
|
| 251 |
projects = [p for p in project_rows if p.get("status", "active") == "active"]
|
| 252 |
+
reported = await asyncio.to_thread(reported_totals)
|
| 253 |
+
# 只有需要 session 兜底时才扫目录(用了上报口径的项目不再触发远程遍历)
|
| 254 |
+
needs_sessions = device_progress.needs_session_fallback(projects, reported)
|
| 255 |
+
episodes: list = []
|
| 256 |
+
if needs_sessions:
|
| 257 |
+
# Prefer the last complete snapshot. A cache miss only happens during a
|
| 258 |
+
# cold start, and even that first remote scan is kept off the event loop.
|
| 259 |
+
episodes = cached_sessions_for_tasks()
|
| 260 |
+
if episodes is None:
|
| 261 |
+
episodes = await asyncio.to_thread(scan_sessions)
|
| 262 |
|
| 263 |
tasks = []
|
| 264 |
for p in projects:
|
| 265 |
+
sessions = (sum(1 for e in episodes if e.get("project") == p.get("name"))
|
| 266 |
+
if needs_sessions else None)
|
| 267 |
+
cur, source = device_progress.resolve_count(p, reported, sessions or 0)
|
| 268 |
params = p.get("params") or {}
|
| 269 |
if not isinstance(params, dict):
|
| 270 |
params = {}
|
|
|
|
| 277 |
"current_count": cur,
|
| 278 |
"assigned_at": p.get("created_at"),
|
| 279 |
"params": params,
|
| 280 |
+
# 供 Web UI/排查用:进度口径与该项目实际已上传的 session 数
|
| 281 |
+
"progress_source": source,
|
| 282 |
+
"session_count": sessions,
|
| 283 |
})
|
| 284 |
|
| 285 |
return {"tasks": tasks, "updated_at": utcnow().isoformat()}
|
| 286 |
+
|
| 287 |
+
|
| 288 |
+
@router.post("/device/tasks/progress")
|
| 289 |
+
async def device_task_progress(body: dict | None = None,
|
| 290 |
+
_: str = Depends(verify_api_key)):
|
| 291 |
+
"""采集端「录制完成」增量上报(幂等)。
|
| 292 |
+
|
| 293 |
+
body::
|
| 294 |
+
|
| 295 |
+
{"task_id": "<项目 id>", "device_name": "EGO_001",
|
| 296 |
+
"session_id": "EGO_001:<task_id>:<本地水位>", "increment": 1}
|
| 297 |
+
|
| 298 |
+
返回 ``{"completed_count": <该项目最新进度>}`` —— 采集端把这个数当后端
|
| 299 |
+
权威数落进自己的分账(backend_count)。同一 (device_name, session_id)
|
| 300 |
+
重放不重复计数:采集端崩溃/断网后重发同一个水位是安全的。
|
| 301 |
+
|
| 302 |
+
错误:参数缺失/非法、任务不存在 → 400。**刻意不用 404** —— 采集端把 404
|
| 303 |
+
当成「后端还没实现该端点」并永久降级(整机不再上报任何进度)。
|
| 304 |
+
"""
|
| 305 |
+
from fastapi import HTTPException
|
| 306 |
+
|
| 307 |
+
projects = await asyncio.to_thread(list_projects)
|
| 308 |
+
known = {str(p.get("id") or "") for p in projects}
|
| 309 |
+
try:
|
| 310 |
+
return await asyncio.to_thread(
|
| 311 |
+
device_progress.apply_report, body or {}, known)
|
| 312 |
+
except ValueError as exc:
|
| 313 |
+
raise HTTPException(status_code=400, detail=str(exc))
|
processor/app/routes/export.py
CHANGED
|
@@ -362,8 +362,15 @@ def _workflow_export_version(episode_id: str) -> str | None:
|
|
| 362 |
|
| 363 |
|
| 364 |
@router.get("/episode-format/{episode_id}")
|
| 365 |
-
|
| 366 |
-
"""前端按钮徽标:该批次下载时实际返回的格式。
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 367 |
if get_episode(episode_id) is None:
|
| 368 |
raise HTTPException(status_code=404, detail="Episode not found")
|
| 369 |
target = _episode_export_target(episode_id)
|
|
|
|
| 362 |
|
| 363 |
|
| 364 |
@router.get("/episode-format/{episode_id}")
|
| 365 |
+
def episode_export_format(episode_id: str):
|
| 366 |
+
"""前端按钮徽标:该批次下载时实际返回的格式。
|
| 367 |
+
|
| 368 |
+
同步端点(FastAPI 丢线程池执行)。以前写成 ``async def`` 却调
|
| 369 |
+
:func:`_episode_export_target` —— 那里读历史 run 快照 + 逐个 ``is_file()``
|
| 370 |
+
全是远程阻塞 I/O,在事件循环上跑会把同一次开页的其它请求一起冻住:
|
| 371 |
+
实测该接口冷调用 4.4s,期间 hand-3d / slam / tactile 全部排队,这正是
|
| 372 |
+
"打开一条数据时而 2 秒时而 7 秒"的来源。
|
| 373 |
+
"""
|
| 374 |
if get_episode(episode_id) is None:
|
| 375 |
raise HTTPException(status_code=404, detail="Episode not found")
|
| 376 |
target = _episode_export_target(episode_id)
|
processor/app/routes/session.py
CHANGED
|
@@ -489,6 +489,37 @@ async def _process_upload_job(
|
|
| 489 |
is_depth_source, iter_video_streams, normalize_extracted_dataset,
|
| 490 |
)
|
| 491 |
normalize_extracted_dataset(staging_dir, batch_name)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 492 |
info = _read_json(staging_dir / "meta" / "info.json")
|
| 493 |
if not isinstance(info, dict):
|
| 494 |
info = {}
|
|
@@ -573,6 +604,13 @@ async def _process_upload_job(
|
|
| 573 |
)
|
| 574 |
write_episode_state(batch_name, new_state)
|
| 575 |
save_annotations(batch_name, [])
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 576 |
# A new upload or a same-name re-upload changes files/cameras, not
|
| 577 |
# just review state. Force the next session scan to discover the new
|
| 578 |
# structure before a workflow or review request can use stale streams.
|
|
|
|
| 489 |
is_depth_source, iter_video_streams, normalize_extracted_dataset,
|
| 490 |
)
|
| 491 |
normalize_extracted_dataset(staging_dir, batch_name)
|
| 492 |
+
|
| 493 |
+
# ── 内容级重复检测(与包名无关)──
|
| 494 |
+
# 采集端可能把已上传过的录制换个包名再推一次(本地去重标记丢失 /
|
| 495 |
+
# 包名规则变化),此时上面的 incoming_name 判断不出来,同一份数据
|
| 496 |
+
# 会被追加成新批次。用规范化后的内容指纹兜底:命中已有批次就按
|
| 497 |
+
# 同名重传处理,覆盖原批次而不是新增一条重复数据。
|
| 498 |
+
# 算不出指纹时返回 None,一律按新批次正常入库。
|
| 499 |
+
try:
|
| 500 |
+
from app.batch_fingerprint import (
|
| 501 |
+
compute_fingerprint, lookup_duplicate,
|
| 502 |
+
)
|
| 503 |
+
content_fingerprint = compute_fingerprint(staging_dir)
|
| 504 |
+
duplicate_of = lookup_duplicate(
|
| 505 |
+
project_folder, content_fingerprint, existing_ids)
|
| 506 |
+
except Exception as fingerprint_err:
|
| 507 |
+
content_fingerprint = None
|
| 508 |
+
duplicate_of = None
|
| 509 |
+
print(f"[Upload] Fingerprint skipped: {fingerprint_err}")
|
| 510 |
+
if duplicate_of and duplicate_of != batch_name:
|
| 511 |
+
print(
|
| 512 |
+
f"[Upload] Duplicate content of {duplicate_of} "
|
| 513 |
+
f"(incoming name {incoming_name!r}) → replacing that episode"
|
| 514 |
+
)
|
| 515 |
+
batch_name = duplicate_of
|
| 516 |
+
is_reupload = True
|
| 517 |
+
batch_preexisted = True
|
| 518 |
+
# staging 目录名只是本地上传中转名,append_project_episode 以
|
| 519 |
+
# batch_name 参数为准,无需跟着改名。
|
| 520 |
+
old_state = read_episode_state(batch_name)
|
| 521 |
+
old_annotations = list_annotations(batch_name)
|
| 522 |
+
|
| 523 |
info = _read_json(staging_dir / "meta" / "info.json")
|
| 524 |
if not isinstance(info, dict):
|
| 525 |
info = {}
|
|
|
|
| 604 |
)
|
| 605 |
write_episode_state(batch_name, new_state)
|
| 606 |
save_annotations(batch_name, [])
|
| 607 |
+
# 指纹只在批次真正提交后记录:失败的上传不该占住一个内容身份,
|
| 608 |
+
# 否则重试时会被判成"重复"而覆盖掉自己。
|
| 609 |
+
try:
|
| 610 |
+
from app.batch_fingerprint import record_fingerprint
|
| 611 |
+
record_fingerprint(project_folder, content_fingerprint, batch_name)
|
| 612 |
+
except Exception as fingerprint_err:
|
| 613 |
+
print(f"[Upload] Fingerprint record skipped: {fingerprint_err}")
|
| 614 |
# A new upload or a same-name re-upload changes files/cameras, not
|
| 615 |
# just review state. Force the next session scan to discover the new
|
| 616 |
# structure before a workflow or review request can use stale streams.
|
processor/app/routes/video.py
CHANGED
|
@@ -57,6 +57,21 @@ _TACTILE_FORCE_COLUMNS = {
|
|
| 57 |
"left": "observation.gripper_left_force",
|
| 58 |
"right": "observation.gripper_right_force",
|
| 59 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 60 |
# Sequential decode is cheap, but decoding through more than this many
|
| 61 |
# frames to reach a scrub target is not; jump via a seek reader instead.
|
| 62 |
_DEPTH_WINDOW_SEEK_GAP = 6
|
|
@@ -276,7 +291,17 @@ def _read_gripper_state(value) -> list[float] | None:
|
|
| 276 |
|
| 277 |
|
| 278 |
def _read_slam_rows(path: Path) -> tuple[str, list[dict]]:
|
| 279 |
-
"""Read canonical
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 280 |
import numpy as np
|
| 281 |
import pandas as pd
|
| 282 |
|
|
@@ -290,23 +315,45 @@ def _read_slam_rows(path: Path) -> tuple[str, list[dict]]:
|
|
| 290 |
cached = _SLAM_DATA_CACHE.get(cache_key)
|
| 291 |
if cached is not None and cached[0] == revision:
|
| 292 |
return cached
|
| 293 |
-
|
| 294 |
-
has_gripper_state = False
|
| 295 |
try:
|
| 296 |
import pyarrow.parquet as pq
|
| 297 |
available_columns = set(pq.ParquetFile(path).schema_arrow.names)
|
| 298 |
-
has_gripper_state = "observation.gripper_state" in available_columns
|
| 299 |
except Exception:
|
| 300 |
-
|
| 301 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 302 |
columns.append("observation.gripper_state")
|
| 303 |
try:
|
| 304 |
df = pd.read_parquet(path, columns=columns, engine="pyarrow")
|
| 305 |
except Exception:
|
| 306 |
return revision, []
|
| 307 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 308 |
rows: list[dict] = []
|
| 309 |
-
for _, row in df.iterrows():
|
| 310 |
try:
|
| 311 |
frame = int(row.get("frame_index", len(rows)))
|
| 312 |
except (TypeError, ValueError):
|
|
@@ -316,11 +363,16 @@ def _read_slam_rows(path: Path) -> tuple[str, list[dict]]:
|
|
| 316 |
timestamp = float(timestamp) if np.isfinite(float(timestamp)) else None
|
| 317 |
except (TypeError, ValueError):
|
| 318 |
timestamp = None
|
| 319 |
-
|
| 320 |
-
|
| 321 |
-
|
| 322 |
-
|
| 323 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 324 |
|
| 325 |
# A missing native SLAM sample is represented by an all-zero pose in
|
| 326 |
# some recordings. It is not the same thing as a valid pose at the
|
|
@@ -337,7 +389,7 @@ def _read_slam_rows(path: Path) -> tuple[str, list[dict]]:
|
|
| 337 |
"p": [float(value) for value in values[:7]],
|
| 338 |
"t": timestamp,
|
| 339 |
"g": _read_gripper_state(row.get("observation.gripper_state"))
|
| 340 |
-
if
|
| 341 |
})
|
| 342 |
rows.sort(key=lambda item: item["f"])
|
| 343 |
|
|
@@ -424,6 +476,53 @@ def _read_tactile_force_series(path: Path) -> dict | None:
|
|
| 424 |
return None
|
| 425 |
|
| 426 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 427 |
def _decode_tactile_matrix(value):
|
| 428 |
"""Decode one canonical 250x250x3 row-diff force matrix to mN.
|
| 429 |
|
|
@@ -432,8 +531,9 @@ def _decode_tactile_matrix(value):
|
|
| 432 |
prefix sum must run over the full 750-value row before reshaping back to
|
| 433 |
HWC. Splitting H/W/C first and accumulating each channel independently
|
| 434 |
produces a different matrix and was the cause of the incorrect preview.
|
| 435 |
-
The int16 cast is intentional: the recorder's contract is modulo-2^16
|
| 436 |
-
|
|
|
|
| 437 |
"""
|
| 438 |
import numpy as np
|
| 439 |
|
|
@@ -466,6 +566,73 @@ def _decode_tactile_matrix(value):
|
|
| 466 |
return decoded
|
| 467 |
|
| 468 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 469 |
def _read_tactile_frame(path: Path, frame_index: int, side: str):
|
| 470 |
"""Read only the requested parquet row and side.
|
| 471 |
|
|
@@ -504,7 +671,7 @@ def _read_tactile_frame(path: Path, frame_index: int, side: str):
|
|
| 504 |
# making the UI show a broken image.
|
| 505 |
if decoded is None:
|
| 506 |
return np.zeros((3, *_TACTILE_SHAPE), dtype=np.float32)
|
| 507 |
-
return decoded
|
| 508 |
except Exception:
|
| 509 |
return None
|
| 510 |
return None
|
|
@@ -516,8 +683,8 @@ def _tactile_force_field(
|
|
| 516 |
"""Return one smoothed fz pressure field as float32 bytes.
|
| 517 |
|
| 518 |
The payload is numeric data, not a rendered image. The frontend applies
|
| 519 |
-
the color map in WebGL2. ``lo``/``hi``
|
| 520 |
-
|
| 521 |
"""
|
| 522 |
import cv2
|
| 523 |
import numpy as np
|
|
@@ -545,12 +712,7 @@ def _tactile_force_field(
|
|
| 545 |
# pressure visualization.
|
| 546 |
pressure = np.maximum(matrix[2], 0.0)
|
| 547 |
pressure = cv2.GaussianBlur(pressure, (3, 3), 0.8)
|
| 548 |
-
|
| 549 |
-
lo = 0.0
|
| 550 |
-
hi = 1.0
|
| 551 |
-
if valid.size > 50:
|
| 552 |
-
lo = float(np.percentile(valid, 10))
|
| 553 |
-
hi = max(float(np.percentile(valid, 98)) * 1.3, lo + 0.5)
|
| 554 |
field = np.ascontiguousarray(pressure, dtype="<f4")
|
| 555 |
result = (field.tobytes(), lo, hi)
|
| 556 |
with _TACTILE_PREVIEW_LOCK:
|
|
@@ -567,11 +729,12 @@ _TACTILE_RANGE_CACHE_MAX = 24
|
|
| 567 |
|
| 568 |
def _tactile_range_payload(path: Path, first: int, last: int,
|
| 569 |
side: str) -> bytes | None:
|
| 570 |
-
"""Encode a run of consecutive fz fields as R8 plus
|
| 571 |
|
| 572 |
Body layout: ``frames * 250 * 250`` bytes of R8 (one byte per cell, row
|
| 573 |
major) followed by ``frames * 2`` little-endian float32 values holding
|
| 574 |
-
each frame's ``lo``/``hi`` normalisation pair
|
|
|
|
| 575 |
replaces one 250 KB request per frame, which is what saturated the
|
| 576 |
browser's socket buffers (net::ERR_NO_BUFFER_SPACE) during playback.
|
| 577 |
"""
|
|
@@ -616,20 +779,19 @@ def _tactile_range_payload(path: Path, first: int, last: int,
|
|
| 616 |
pixels = _TACTILE_SHAPE[0] * _TACTILE_SHAPE[1]
|
| 617 |
blocks: list[bytes] = []
|
| 618 |
pairs: list[float] = []
|
|
|
|
|
|
|
|
|
|
|
|
|
| 619 |
for frame in range(int(first), int(last)):
|
| 620 |
-
matrix = _decode_tactile_matrix(rows.get(frame))
|
|
|
|
| 621 |
if matrix is None:
|
| 622 |
blocks.append(bytes(pixels))
|
| 623 |
-
pairs.extend((
|
| 624 |
continue
|
| 625 |
pressure = np.maximum(matrix[2], 0.0)
|
| 626 |
pressure = cv2.GaussianBlur(pressure, (3, 3), 0.8)
|
| 627 |
-
valid = pressure[pressure > 0.01]
|
| 628 |
-
lo = 0.0
|
| 629 |
-
hi = 1.0
|
| 630 |
-
if valid.size > 50:
|
| 631 |
-
lo = float(np.percentile(valid, 10))
|
| 632 |
-
hi = max(float(np.percentile(valid, 98)) * 1.3, lo + 0.5)
|
| 633 |
scaled = np.clip((pressure - lo) / (hi - lo + 1e-6), 0.0, 1.0)
|
| 634 |
blocks.append(np.rint(scaled * 255.0).astype(np.uint8).tobytes())
|
| 635 |
pairs.extend((lo, hi))
|
|
|
|
| 57 |
"left": "observation.gripper_left_force",
|
| 58 |
"right": "observation.gripper_right_force",
|
| 59 |
}
|
| 60 |
+
# Colour-scale anchors for the force heatmaps. The range is measured over the
|
| 61 |
+
# whole episode, never per frame: a frame with no contact holds nothing above
|
| 62 |
+
# the noise threshold but noise, so a per-frame range collapses onto that noise
|
| 63 |
+
# and stretches every blip to the top of the colour ramp - the tile lights up
|
| 64 |
+
# with red dots that read as real pressure. ``lo`` is the episode's median
|
| 65 |
+
# valid pressure (its noise floor; anything at or below renders as background)
|
| 66 |
+
# and ``hi`` its peak plus headroom.
|
| 67 |
+
_TACTILE_NOISE_FLOOR_MN = 0.01
|
| 68 |
+
_TACTILE_RANGE_BASELINE_PERCENTILE = 50.0
|
| 69 |
+
_TACTILE_RANGE_PEAK_PERCENTILE = 99.9
|
| 70 |
+
_TACTILE_RANGE_HEADROOM = 1.3
|
| 71 |
+
_TACTILE_RANGE_MIN_SPAN_MN = 0.5
|
| 72 |
+
_TACTILE_RANGE_SAMPLE_FRAMES = 256
|
| 73 |
+
_TACTILE_SCALE_CACHE: OrderedDict[tuple, tuple[float, float]] = OrderedDict()
|
| 74 |
+
_TACTILE_SCALE_CACHE_MAX = 8
|
| 75 |
# Sequential decode is cheap, but decoding through more than this many
|
| 76 |
# frames to reach a scrub target is not; jump via a seek reader instead.
|
| 77 |
_DEPTH_WINDOW_SEEK_GAP = 6
|
|
|
|
| 291 |
|
| 292 |
|
| 293 |
def _read_slam_rows(path: Path) -> tuple[str, list[dict]]:
|
| 294 |
+
"""Read canonical SLAM rows once per parquet revision.
|
| 295 |
+
|
| 296 |
+
位姿有两个来源,取决于采集端版本:
|
| 297 |
+
|
| 298 |
+
- 旧版写 ``observation.slam_pose``(每帧一个 7 维位姿);
|
| 299 |
+
- 新版只写 ``observation.slam_trajectory``(高频样本缓冲),没有
|
| 300 |
+
``slam_pose``。此时前端曾经整块空白 —— 因为这里硬编码读后者。
|
| 301 |
+
|
| 302 |
+
两条路径统一成每帧一个位姿:trajectory 交给 ``poses_from_trajectory``
|
| 303 |
+
重建(它已按时间戳取最近样本,丢帧时也能从相邻格子找回)。
|
| 304 |
+
"""
|
| 305 |
import numpy as np
|
| 306 |
import pandas as pd
|
| 307 |
|
|
|
|
| 315 |
cached = _SLAM_DATA_CACHE.get(cache_key)
|
| 316 |
if cached is not None and cached[0] == revision:
|
| 317 |
return cached
|
| 318 |
+
|
|
|
|
| 319 |
try:
|
| 320 |
import pyarrow.parquet as pq
|
| 321 |
available_columns = set(pq.ParquetFile(path).schema_arrow.names)
|
|
|
|
| 322 |
except Exception:
|
| 323 |
+
return revision, []
|
| 324 |
+
|
| 325 |
+
has_pose = "observation.slam_pose" in available_columns
|
| 326 |
+
has_trajectory = "observation.slam_trajectory" in available_columns
|
| 327 |
+
if not has_pose and not has_trajectory:
|
| 328 |
+
return revision, []
|
| 329 |
+
|
| 330 |
+
columns = ["frame_index", "timestamp"]
|
| 331 |
+
if has_pose:
|
| 332 |
+
columns.append("observation.slam_pose")
|
| 333 |
+
if has_trajectory:
|
| 334 |
+
columns.append("observation.slam_trajectory")
|
| 335 |
+
if "observation.gripper_state" in available_columns:
|
| 336 |
columns.append("observation.gripper_state")
|
| 337 |
try:
|
| 338 |
df = pd.read_parquet(path, columns=columns, engine="pyarrow")
|
| 339 |
except Exception:
|
| 340 |
return revision, []
|
| 341 |
|
| 342 |
+
# 新版只有 trajectory:重建出逐帧位姿,让下游按同一套逻辑消费。
|
| 343 |
+
rebuilt = None
|
| 344 |
+
if not has_pose:
|
| 345 |
+
from app.umi_slam_action import poses_from_trajectory
|
| 346 |
+
stamps = (df["timestamp"].to_numpy(dtype=float)
|
| 347 |
+
if "timestamp" in df.columns else None)
|
| 348 |
+
if stamps is None:
|
| 349 |
+
stamps = np.arange(len(df), dtype=float) / 30.0
|
| 350 |
+
rebuilt = poses_from_trajectory(
|
| 351 |
+
df["observation.slam_trajectory"].tolist(), stamps, len(df))
|
| 352 |
+
if rebuilt is None:
|
| 353 |
+
return revision, []
|
| 354 |
+
|
| 355 |
rows: list[dict] = []
|
| 356 |
+
for position, (_, row) in enumerate(df.iterrows()):
|
| 357 |
try:
|
| 358 |
frame = int(row.get("frame_index", len(rows)))
|
| 359 |
except (TypeError, ValueError):
|
|
|
|
| 363 |
timestamp = float(timestamp) if np.isfinite(float(timestamp)) else None
|
| 364 |
except (TypeError, ValueError):
|
| 365 |
timestamp = None
|
| 366 |
+
if has_pose:
|
| 367 |
+
pose = row.get("observation.slam_pose")
|
| 368 |
+
try:
|
| 369 |
+
values = np.asarray(pose, dtype=np.float64).reshape(-1)
|
| 370 |
+
except (TypeError, ValueError):
|
| 371 |
+
values = np.empty(0, dtype=np.float64)
|
| 372 |
+
else:
|
| 373 |
+
# 已从 trajectory 重建,按位置对齐取本帧位姿
|
| 374 |
+
values = (rebuilt[position] if position < len(rebuilt)
|
| 375 |
+
else np.empty(0, dtype=np.float64))
|
| 376 |
|
| 377 |
# A missing native SLAM sample is represented by an all-zero pose in
|
| 378 |
# some recordings. It is not the same thing as a valid pose at the
|
|
|
|
| 389 |
"p": [float(value) for value in values[:7]],
|
| 390 |
"t": timestamp,
|
| 391 |
"g": _read_gripper_state(row.get("observation.gripper_state"))
|
| 392 |
+
if "observation.gripper_state" in available_columns else None,
|
| 393 |
})
|
| 394 |
rows.sort(key=lambda item: item["f"])
|
| 395 |
|
|
|
|
| 476 |
return None
|
| 477 |
|
| 478 |
|
| 479 |
+
def _tactile_scale_factor(path: Path) -> float:
|
| 480 |
+
"""Read ``features.*.scale`` for the force matrix from dataset metadata.
|
| 481 |
+
|
| 482 |
+
Newer recorders store the matrix as quantised integers scaled by 100
|
| 483 |
+
(``"scale": 100`` alongside ``units: mN``), so the raw values must be
|
| 484 |
+
divided by that factor to recover mN. Older recordings omit the field and
|
| 485 |
+
already store plain mN, so the default is 1.
|
| 486 |
+
|
| 487 |
+
The factor is read from metadata rather than hard-coded because the
|
| 488 |
+
recordings genuinely differ; assuming 100 would divide old data by 100.
|
| 489 |
+
"""
|
| 490 |
+
try:
|
| 491 |
+
import json
|
| 492 |
+
# parquet 位于 <root>/data/chunk-000/,info.json 位于 <root>/meta/。
|
| 493 |
+
# 逐级上溯找 meta/info.json,避免依赖固定的目录深度。
|
| 494 |
+
root = Path(path).parent
|
| 495 |
+
for candidate in (root, *root.parents):
|
| 496 |
+
meta = candidate / "meta" / "info.json"
|
| 497 |
+
if meta.is_file():
|
| 498 |
+
info = json.loads(meta.read_text(encoding="utf-8"))
|
| 499 |
+
break
|
| 500 |
+
else:
|
| 501 |
+
return 1.0
|
| 502 |
+
except (OSError, ValueError, TypeError):
|
| 503 |
+
return 1.0
|
| 504 |
+
features = info.get("features") or {}
|
| 505 |
+
for key in _TACTILE_COLUMNS.values():
|
| 506 |
+
entry = features.get(key)
|
| 507 |
+
if not isinstance(entry, dict):
|
| 508 |
+
continue
|
| 509 |
+
try:
|
| 510 |
+
factor = float(entry.get("scale"))
|
| 511 |
+
except (TypeError, ValueError):
|
| 512 |
+
continue
|
| 513 |
+
# 0/负数会让整块力值失真,视为未声明。
|
| 514 |
+
if factor > 0:
|
| 515 |
+
return factor
|
| 516 |
+
return 1.0
|
| 517 |
+
|
| 518 |
+
|
| 519 |
+
def _apply_tactile_scale(matrix, factor: float):
|
| 520 |
+
"""把量化整数还原成物理单位(mN);factor<=1 时不做任何改动。"""
|
| 521 |
+
if matrix is None or not factor or factor <= 1:
|
| 522 |
+
return matrix
|
| 523 |
+
return matrix / factor
|
| 524 |
+
|
| 525 |
+
|
| 526 |
def _decode_tactile_matrix(value):
|
| 527 |
"""Decode one canonical 250x250x3 row-diff force matrix to mN.
|
| 528 |
|
|
|
|
| 531 |
prefix sum must run over the full 750-value row before reshaping back to
|
| 532 |
HWC. Splitting H/W/C first and accumulating each channel independently
|
| 533 |
produces a different matrix and was the cause of the incorrect preview.
|
| 534 |
+
The int16 cast is intentional: the recorder's contract is modulo-2^16.
|
| 535 |
+
Values are quantised integers in ``units: mN``; the physical scale is
|
| 536 |
+
applied by callers via ``_tactile_scale_factor``.
|
| 537 |
"""
|
| 538 |
import numpy as np
|
| 539 |
|
|
|
|
| 566 |
return decoded
|
| 567 |
|
| 568 |
|
| 569 |
+
def _compute_tactile_display_range(path: Path, column: str) -> tuple[float, float]:
|
| 570 |
+
"""Sample a whole episode and return its ``(lo, hi)`` colour range in mN.
|
| 571 |
+
|
| 572 |
+
Every stored frame shares one row group, so reading the column costs the
|
| 573 |
+
same as reading any slice of it; the sampling stride bounds the decode and
|
| 574 |
+
the pooled sample, not the I/O.
|
| 575 |
+
"""
|
| 576 |
+
import cv2
|
| 577 |
+
import numpy as np
|
| 578 |
+
import pyarrow as pa
|
| 579 |
+
import pyarrow.parquet as pq
|
| 580 |
+
|
| 581 |
+
try:
|
| 582 |
+
table = pq.read_table(path, columns=[column])
|
| 583 |
+
except Exception:
|
| 584 |
+
return 0.0, 1.0
|
| 585 |
+
column_data = table.column(column)
|
| 586 |
+
if isinstance(column_data, pa.ChunkedArray):
|
| 587 |
+
column_data = column_data.combine_chunks()
|
| 588 |
+
offsets = np.asarray(column_data.offsets, dtype=np.int64)
|
| 589 |
+
values = np.asarray(column_data.values, dtype=np.int16)
|
| 590 |
+
frames = len(offsets) - 1
|
| 591 |
+
if frames <= 0:
|
| 592 |
+
return 0.0, 1.0
|
| 593 |
+
stride = max(1, frames // _TACTILE_RANGE_SAMPLE_FRAMES)
|
| 594 |
+
scale_factor = _tactile_scale_factor(path)
|
| 595 |
+
samples: list[np.ndarray] = []
|
| 596 |
+
for index in range(0, frames, stride):
|
| 597 |
+
matrix = _apply_tactile_scale(
|
| 598 |
+
_decode_tactile_matrix(values[offsets[index]:offsets[index + 1]]),
|
| 599 |
+
scale_factor)
|
| 600 |
+
if matrix is None:
|
| 601 |
+
continue
|
| 602 |
+
pressure = cv2.GaussianBlur(np.maximum(matrix[2], 0.0), (3, 3), 0.8)
|
| 603 |
+
valid = pressure[pressure > _TACTILE_NOISE_FLOOR_MN]
|
| 604 |
+
if valid.size:
|
| 605 |
+
samples.append(valid)
|
| 606 |
+
if not samples:
|
| 607 |
+
return 0.0, 1.0
|
| 608 |
+
pooled = np.concatenate(samples)
|
| 609 |
+
lo = float(np.percentile(pooled, _TACTILE_RANGE_BASELINE_PERCENTILE))
|
| 610 |
+
hi = max(float(np.percentile(pooled, _TACTILE_RANGE_PEAK_PERCENTILE))
|
| 611 |
+
* _TACTILE_RANGE_HEADROOM, lo + _TACTILE_RANGE_MIN_SPAN_MN)
|
| 612 |
+
return lo, hi
|
| 613 |
+
|
| 614 |
+
|
| 615 |
+
def _tactile_display_range(path: Path, revision: str,
|
| 616 |
+
side: str) -> tuple[float, float]:
|
| 617 |
+
"""Return the cached episode-wide colour range ``(lo, hi)`` for one side."""
|
| 618 |
+
column = _TACTILE_COLUMNS.get(side)
|
| 619 |
+
if column is None:
|
| 620 |
+
return 0.0, 1.0
|
| 621 |
+
key = (str(path.resolve()), revision, side)
|
| 622 |
+
with _TACTILE_PREVIEW_LOCK:
|
| 623 |
+
cached = _TACTILE_SCALE_CACHE.get(key)
|
| 624 |
+
if cached is not None:
|
| 625 |
+
_TACTILE_SCALE_CACHE.move_to_end(key)
|
| 626 |
+
return cached
|
| 627 |
+
result = _compute_tactile_display_range(path, column)
|
| 628 |
+
with _TACTILE_PREVIEW_LOCK:
|
| 629 |
+
_TACTILE_SCALE_CACHE[key] = result
|
| 630 |
+
_TACTILE_SCALE_CACHE.move_to_end(key)
|
| 631 |
+
while len(_TACTILE_SCALE_CACHE) > _TACTILE_SCALE_CACHE_MAX:
|
| 632 |
+
_TACTILE_SCALE_CACHE.popitem(last=False)
|
| 633 |
+
return result
|
| 634 |
+
|
| 635 |
+
|
| 636 |
def _read_tactile_frame(path: Path, frame_index: int, side: str):
|
| 637 |
"""Read only the requested parquet row and side.
|
| 638 |
|
|
|
|
| 671 |
# making the UI show a broken image.
|
| 672 |
if decoded is None:
|
| 673 |
return np.zeros((3, *_TACTILE_SHAPE), dtype=np.float32)
|
| 674 |
+
return _apply_tactile_scale(decoded, _tactile_scale_factor(path))
|
| 675 |
except Exception:
|
| 676 |
return None
|
| 677 |
return None
|
|
|
|
| 683 |
"""Return one smoothed fz pressure field as float32 bytes.
|
| 684 |
|
| 685 |
The payload is numeric data, not a rendered image. The frontend applies
|
| 686 |
+
the color map in WebGL2. ``lo``/``hi`` are the episode-wide contrast
|
| 687 |
+
range, so the same mN value maps to the same colour in every frame.
|
| 688 |
"""
|
| 689 |
import cv2
|
| 690 |
import numpy as np
|
|
|
|
| 712 |
# pressure visualization.
|
| 713 |
pressure = np.maximum(matrix[2], 0.0)
|
| 714 |
pressure = cv2.GaussianBlur(pressure, (3, 3), 0.8)
|
| 715 |
+
lo, hi = _tactile_display_range(path, f"{revision[0]:x}-{revision[1]:x}", side)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 716 |
field = np.ascontiguousarray(pressure, dtype="<f4")
|
| 717 |
result = (field.tobytes(), lo, hi)
|
| 718 |
with _TACTILE_PREVIEW_LOCK:
|
|
|
|
| 729 |
|
| 730 |
def _tactile_range_payload(path: Path, first: int, last: int,
|
| 731 |
side: str) -> bytes | None:
|
| 732 |
+
"""Encode a run of consecutive fz fields as R8 plus the episode's min/max.
|
| 733 |
|
| 734 |
Body layout: ``frames * 250 * 250`` bytes of R8 (one byte per cell, row
|
| 735 |
major) followed by ``frames * 2`` little-endian float32 values holding
|
| 736 |
+
each frame's ``lo``/``hi`` normalisation pair - the episode-wide range,
|
| 737 |
+
repeated once per frame so the header count stays fixed. One request per window
|
| 738 |
replaces one 250 KB request per frame, which is what saturated the
|
| 739 |
browser's socket buffers (net::ERR_NO_BUFFER_SPACE) during playback.
|
| 740 |
"""
|
|
|
|
| 779 |
pixels = _TACTILE_SHAPE[0] * _TACTILE_SHAPE[1]
|
| 780 |
blocks: list[bytes] = []
|
| 781 |
pairs: list[float] = []
|
| 782 |
+
# 量化整数先还原成 mN;色阶取整集区间,不逐帧重算,否则没有接触的帧会
|
| 783 |
+
# 拿自身噪声当满量程,把噪点画成接触点。
|
| 784 |
+
scale_factor = _tactile_scale_factor(path)
|
| 785 |
+
lo, hi = _tactile_display_range(path, revision, side)
|
| 786 |
for frame in range(int(first), int(last)):
|
| 787 |
+
matrix = _apply_tactile_scale(_decode_tactile_matrix(rows.get(frame)),
|
| 788 |
+
scale_factor)
|
| 789 |
if matrix is None:
|
| 790 |
blocks.append(bytes(pixels))
|
| 791 |
+
pairs.extend((lo, hi))
|
| 792 |
continue
|
| 793 |
pressure = np.maximum(matrix[2], 0.0)
|
| 794 |
pressure = cv2.GaussianBlur(pressure, (3, 3), 0.8)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 795 |
scaled = np.clip((pressure - lo) / (hi - lo + 1e-6), 0.0, 1.0)
|
| 796 |
blocks.append(np.rint(scaled * 255.0).astype(np.uint8).tobytes())
|
| 797 |
pairs.extend((lo, hi))
|
processor/app/umi_slam_action.py
ADDED
|
@@ -0,0 +1,269 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""UMI Gripper 动作派生 —— 从 SLAM 位姿反推 imitation learning 的 action。
|
| 2 |
+
|
| 3 |
+
UMI 硬件不录 action:采集端写入的 ``action`` 列是占位全零,导出时会被
|
| 4 |
+
``lerobot_export._junk_columns`` 当零信息列剔除,导致数据集无法训练。
|
| 5 |
+
官方 UMI 的做法同样是从 SLAM 位姿反推 —— 本模块实现这套口径。
|
| 6 |
+
|
| 7 |
+
action(7 维,当前夹爪局部坐标系):
|
| 8 |
+
|
| 9 |
+
Δp = R_tᵀ · (p_{t+1} − p_t) # 平移增量, 米
|
| 10 |
+
Δr = rotvec(R_tᵀ · R_{t+1}) # 旋转增量, 弧度
|
| 11 |
+
g = gripper_state[t+1, 0] / 100 # 夹爪开合, 0–1
|
| 12 |
+
|
| 13 |
+
本模块是纯函数(不依赖 Web 框架与数据库),便于单测与跨格式复用。
|
| 14 |
+
"""
|
| 15 |
+
|
| 16 |
+
from __future__ import annotations
|
| 17 |
+
|
| 18 |
+
import numpy as np
|
| 19 |
+
|
| 20 |
+
# 动作维度:Δp(3) + Δr(3) + 夹爪(1)
|
| 21 |
+
ACTION_DIM = 7
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
def quat_to_matrix(q) -> np.ndarray | None:
|
| 25 |
+
"""四元数 ``(x, y, z, w)`` → 3×3 旋转矩阵。
|
| 26 |
+
|
| 27 |
+
范数过小(零四元数)说明该帧没有有效姿态,返回 ``None`` 而不是抛异常 ——
|
| 28 |
+
调用方据此跳过该帧。
|
| 29 |
+
"""
|
| 30 |
+
x, y, z, w = (float(v) for v in np.asarray(q, dtype=float).ravel()[:4])
|
| 31 |
+
norm = (x * x + y * y + z * z + w * w) ** 0.5
|
| 32 |
+
if norm < 1e-12:
|
| 33 |
+
return None
|
| 34 |
+
x, y, z, w = x / norm, y / norm, z / norm, w / norm
|
| 35 |
+
return np.array([
|
| 36 |
+
[1 - 2 * (y * y + z * z), 2 * (x * y - z * w), 2 * (x * z + y * w)],
|
| 37 |
+
[2 * (x * y + z * w), 1 - 2 * (x * x + z * z), 2 * (y * z - x * w)],
|
| 38 |
+
[2 * (x * z - y * w), 2 * (y * z + x * w), 1 - 2 * (x * x + y * y)],
|
| 39 |
+
])
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
def matrix_to_rotvec(rotation: np.ndarray) -> np.ndarray:
|
| 43 |
+
"""3×3 旋转矩阵 → 旋转向量(轴×角,弧度)。
|
| 44 |
+
|
| 45 |
+
近 180° 时 ``sin(角) → 0``,标准轴公式会除零并产生 NaN。手部大幅翻转
|
| 46 |
+
是真实存在的姿态,一旦产生 NaN 会污染整列 action,因此单独走
|
| 47 |
+
``(R + I)/2`` 的对角元求轴。
|
| 48 |
+
"""
|
| 49 |
+
trace = (float(rotation[0, 0]) + float(rotation[1, 1])
|
| 50 |
+
+ float(rotation[2, 2]) - 1.0) / 2.0
|
| 51 |
+
angle = float(np.arccos(max(-1.0, min(1.0, trace))))
|
| 52 |
+
|
| 53 |
+
if angle < 1e-9:
|
| 54 |
+
return np.zeros(3)
|
| 55 |
+
|
| 56 |
+
if np.pi - angle < 1e-6:
|
| 57 |
+
# 近 π:用 (R+I)/2 的列向量恢复转轴,再按对角元归一化。
|
| 58 |
+
symmetric = (rotation + np.eye(3)) / 2.0
|
| 59 |
+
diagonal = np.clip(np.diag(symmetric), 0.0, None)
|
| 60 |
+
index = int(np.argmax(diagonal))
|
| 61 |
+
if diagonal[index] < 1e-9:
|
| 62 |
+
return np.zeros(3)
|
| 63 |
+
axis = symmetric[:, index] / (diagonal[index] ** 0.5)
|
| 64 |
+
norm = float(np.linalg.norm(axis))
|
| 65 |
+
return axis / norm * angle if norm > 1e-9 else np.zeros(3)
|
| 66 |
+
|
| 67 |
+
axis = np.array([
|
| 68 |
+
float(rotation[2, 1]) - float(rotation[1, 2]),
|
| 69 |
+
float(rotation[0, 2]) - float(rotation[2, 0]),
|
| 70 |
+
float(rotation[1, 0]) - float(rotation[0, 1]),
|
| 71 |
+
]) / (2.0 * np.sin(angle))
|
| 72 |
+
return axis * angle
|
| 73 |
+
|
| 74 |
+
|
| 75 |
+
def poses_from_trajectory(trajectories, timestamps,
|
| 76 |
+
frame_count: int) -> np.ndarray | None:
|
| 77 |
+
"""从 ``observation.slam_trajectory`` 的样本缓冲重建逐帧位姿 ``(N, 7)``。
|
| 78 |
+
|
| 79 |
+
slam_trajectory 是采集端的高频(50Hz)样本缓冲,每格存
|
| 80 |
+
``[t, x, y, z, qx, qy, qz, qw]`` 的整数倍。数据帧是 30fps,所以两个
|
| 81 |
+
数据帧之间到达的样本会落在其中一格的格子里 —— 某一帧自己的格子为空,
|
| 82 |
+
不代表那个时刻没有位姿,**相邻格子里往往就有**。
|
| 83 |
+
|
| 84 |
+
因此正确做法是把所有格子的样本汇总,按时间戳逐帧取最近的:
|
| 85 |
+
实测 30% 的「丢帧」里有 97% 能被真实样本覆盖,且单帧位移上限从
|
| 86 |
+
113.9mm 降到 24.8mm(插值方案会横跨 SLAM 重定位跳变)。
|
| 87 |
+
|
| 88 |
+
时间基映射:slam 时间戳与数据时间戳之间是**线性关系**(实测斜率
|
| 89 |
+
1.02,采集端时钟略快)。用「每个非空格子的最后一组对应本帧时刻」
|
| 90 |
+
的锚点做最小二乘拟合,不依赖外部时钟。
|
| 91 |
+
|
| 92 |
+
返回 ``(frame_count, 7)``;样本不足返回 ``None``。
|
| 93 |
+
"""
|
| 94 |
+
import numpy as np
|
| 95 |
+
|
| 96 |
+
if frame_count <= 0:
|
| 97 |
+
return None
|
| 98 |
+
stamp = np.asarray(timestamps, dtype=float).reshape(-1)[:frame_count]
|
| 99 |
+
if stamp.size < frame_count:
|
| 100 |
+
stamp = np.concatenate([stamp,
|
| 101 |
+
np.full(frame_count - stamp.size, np.nan)])
|
| 102 |
+
|
| 103 |
+
# 1) 汇总样本,并收集时间基锚点(非空格子的最后一组 ≈ 本帧时刻)
|
| 104 |
+
samples: list[tuple[float, list[float]]] = []
|
| 105 |
+
anchor_slam: list[float] = []
|
| 106 |
+
anchor_data: list[float] = []
|
| 107 |
+
has_anchor = ~np.isnan(stamp)
|
| 108 |
+
for index in range(min(frame_count, len(trajectories))):
|
| 109 |
+
values = np.asarray(trajectories[index], dtype=float).reshape(-1)
|
| 110 |
+
groups = values.size // 8
|
| 111 |
+
if groups <= 0:
|
| 112 |
+
continue
|
| 113 |
+
for g in range(groups):
|
| 114 |
+
chunk = values[g * 8:g * 8 + 8]
|
| 115 |
+
if not np.isfinite(chunk).all():
|
| 116 |
+
continue
|
| 117 |
+
samples.append((float(chunk[0]), [float(v) for v in chunk[1:8]]))
|
| 118 |
+
if has_anchor[index]:
|
| 119 |
+
anchor_slam.append(float(values[(groups - 1) * 8]))
|
| 120 |
+
anchor_data.append(float(stamp[index]))
|
| 121 |
+
if len(samples) < 2 or len(anchor_slam) < 2:
|
| 122 |
+
return None
|
| 123 |
+
|
| 124 |
+
# 2) 线性映射 slam 时间 → 数据时间:data = slope*slam + intercept
|
| 125 |
+
slam_t = np.array(anchor_slam, dtype=float)
|
| 126 |
+
data_t = np.array(anchor_data, dtype=float)
|
| 127 |
+
slope, intercept = np.polyfit(slam_t, data_t, 1)
|
| 128 |
+
if not np.isfinite(slope) or slope <= 0:
|
| 129 |
+
return None
|
| 130 |
+
|
| 131 |
+
samples.sort(key=lambda item: item[0])
|
| 132 |
+
sample_t = (np.array([item[0] for item in samples], dtype=float)
|
| 133 |
+
* slope + intercept)
|
| 134 |
+
sample_p = np.array([item[1] for item in samples], dtype=float)
|
| 135 |
+
|
| 136 |
+
# 3) 逐帧取时间上最近的样本。超过约一帧半间隔就认为真的没有数据,
|
| 137 |
+
# 标为缺失留给插值,避免把远处的样本硬贴到当前帧。
|
| 138 |
+
poses = np.zeros((frame_count, 7), dtype=float)
|
| 139 |
+
missing = np.zeros(frame_count, dtype=bool)
|
| 140 |
+
dt = np.nanmedian(np.diff(stamp)) if frame_count > 1 else 1.0 / 30.0
|
| 141 |
+
if not np.isfinite(dt) or dt <= 0:
|
| 142 |
+
dt = 1.0 / 30.0
|
| 143 |
+
tolerance = dt * 1.5
|
| 144 |
+
for index in range(frame_count):
|
| 145 |
+
if np.isnan(stamp[index]):
|
| 146 |
+
missing[index] = True
|
| 147 |
+
continue
|
| 148 |
+
j = int(np.abs(sample_t - stamp[index]).argmin())
|
| 149 |
+
if abs(sample_t[j] - stamp[index]) > tolerance:
|
| 150 |
+
missing[index] = True
|
| 151 |
+
continue
|
| 152 |
+
poses[index] = sample_p[j]
|
| 153 |
+
|
| 154 |
+
# 4) 真实样本覆盖不到的少数帧(实测约 3%)才插值兜底
|
| 155 |
+
valid = np.where(~missing)[0]
|
| 156 |
+
if len(valid) < 2:
|
| 157 |
+
return None
|
| 158 |
+
for index in np.where(missing)[0]:
|
| 159 |
+
lo = valid[valid < index]
|
| 160 |
+
hi = valid[valid > index]
|
| 161 |
+
if len(lo) and len(hi):
|
| 162 |
+
a, b = int(lo[-1]), int(hi[0])
|
| 163 |
+
w = (index - a) / (b - a)
|
| 164 |
+
poses[index] = (1 - w) * poses[a] + w * poses[b]
|
| 165 |
+
else:
|
| 166 |
+
poses[index] = poses[int(lo[-1])] if len(lo) else poses[int(hi[0])]
|
| 167 |
+
|
| 168 |
+
norms = np.linalg.norm(poses[:, 3:7], axis=1, keepdims=True)
|
| 169 |
+
poses[:, 3:7] = poses[:, 3:7] / np.maximum(norms, 1e-12)
|
| 170 |
+
return poses
|
| 171 |
+
|
| 172 |
+
|
| 173 |
+
def _gripper_fraction(states, frame_count: int) -> np.ndarray:
|
| 174 |
+
"""取夹爪开合度 → 0–1。契约 ``[percent, gripped, raw]``,只用首维。
|
| 175 |
+
|
| 176 |
+
``gripped`` 恒为 0、``raw`` 是带噪编码器计数,都不参与 action。
|
| 177 |
+
"""
|
| 178 |
+
fraction = np.zeros(frame_count)
|
| 179 |
+
total = len(states) if states is not None else 0
|
| 180 |
+
for frame in range(frame_count):
|
| 181 |
+
raw = states[frame] if frame < total else None
|
| 182 |
+
if raw is None:
|
| 183 |
+
continue
|
| 184 |
+
values = np.asarray(raw, dtype=float).ravel()
|
| 185 |
+
if values.size:
|
| 186 |
+
fraction[frame] = float(values[0]) / 100.0
|
| 187 |
+
# 采集端开合度是 0–100;越界值(标定异常)夹到合法区间的边界,
|
| 188 |
+
# 避免动作维度出现 >1 的离群值把归一化统计带偏。
|
| 189 |
+
return np.clip(fraction, 0.0, 1.0)
|
| 190 |
+
|
| 191 |
+
|
| 192 |
+
def derive_action(trajectories, timestamps, gripper_states,
|
| 193 |
+
frame_count: int) -> dict[int, dict] | None:
|
| 194 |
+
"""派生 UMI 口径的 action 与 proprioception。
|
| 195 |
+
|
| 196 |
+
位姿**只从** ``observation.slam_trajectory`` 重建 —— 它是采集端的高频
|
| 197 |
+
样本缓冲,丢帧时相邻格子里仍有真实样本(实测可覆盖 97% 的丢帧);
|
| 198 |
+
而 ``observation.slam_pose`` 在同样位置直接写零,只能靠插值硬补,
|
| 199 |
+
还会横跨 SLAM 重定位跳变(实测产生过 113.9mm 的假位移)。
|
| 200 |
+
|
| 201 |
+
返回 ``{frame_index: {"action": [7], "observation.state": [4]}}``:
|
| 202 |
+
|
| 203 |
+
- ``action``(7 维):当前夹爪局部系的增量位姿 + 夹爪开合,见上文;
|
| 204 |
+
- ``observation.state``(4 维):``[Δx, Δy, Δz, gripper]``,位置为
|
| 205 |
+
**相对本集起点**的偏移。ACT 用单帧观测,没有 state 时模型无从得知
|
| 206 |
+
夹爪当前在哪(只能看图像猜);而 SLAM 的世界原点是每集任意初始化的,
|
| 207 |
+
给绝对坐标会把无关的世界位置当特征,故取相对量。
|
| 208 |
+
|
| 209 |
+
输入按 ``frame_index`` 顺序排列。无可用样本(非 UMI 数据、或全程丢帧)
|
| 210 |
+
返回 ``None``,调用方应跳过该集而非写入垃圾。
|
| 211 |
+
|
| 212 |
+
末帧没有 ``t+1``,沿用倒数第二帧的增量 —— 与 LeRobot 的
|
| 213 |
+
``next.done`` 语义一致(末帧只是终止标记,不承载新动作)。
|
| 214 |
+
"""
|
| 215 |
+
if frame_count <= 0:
|
| 216 |
+
return None
|
| 217 |
+
cleaned = poses_from_trajectory(trajectories, timestamps, frame_count)
|
| 218 |
+
if cleaned is None:
|
| 219 |
+
return None
|
| 220 |
+
if len(cleaned) != frame_count:
|
| 221 |
+
# 位姿列与帧数不一致时以较短者为准,避免错位写入。
|
| 222 |
+
frame_count = min(frame_count, len(cleaned))
|
| 223 |
+
|
| 224 |
+
fraction = _gripper_fraction(gripper_states, frame_count)
|
| 225 |
+
# observation.state 的位置分量取**相对本集起点**的偏移。SLAM 的世界
|
| 226 |
+
# 原点是每次采集初始化时任意确定的,直接给绝对坐标会让模型把无关的
|
| 227 |
+
# 世界坐标当特征;相对起点才是可迁移的"相对自己出发点多远"。
|
| 228 |
+
origin = cleaned[0, :3].copy()
|
| 229 |
+
result: dict[int, dict] = {}
|
| 230 |
+
|
| 231 |
+
for frame in range(frame_count):
|
| 232 |
+
if frame < frame_count - 1:
|
| 233 |
+
current = quat_to_matrix(cleaned[frame, 3:7])
|
| 234 |
+
following = quat_to_matrix(cleaned[frame + 1, 3:7])
|
| 235 |
+
index = frame + 1
|
| 236 |
+
else:
|
| 237 |
+
# 末帧沿用倒数第二帧的位姿对,只取夹爪当前值。
|
| 238 |
+
if frame_count < 2:
|
| 239 |
+
continue
|
| 240 |
+
current = quat_to_matrix(cleaned[frame - 1, 3:7])
|
| 241 |
+
following = quat_to_matrix(cleaned[frame, 3:7])
|
| 242 |
+
index = frame
|
| 243 |
+
if current is None or following is None:
|
| 244 |
+
continue
|
| 245 |
+
|
| 246 |
+
if frame < frame_count - 1:
|
| 247 |
+
delta_position = current.T @ (cleaned[frame + 1, :3]
|
| 248 |
+
- cleaned[frame, :3])
|
| 249 |
+
else:
|
| 250 |
+
delta_position = current.T @ (cleaned[frame, :3]
|
| 251 |
+
- cleaned[frame - 1, :3])
|
| 252 |
+
delta_rotation = matrix_to_rotvec(current.T @ following)
|
| 253 |
+
|
| 254 |
+
action = np.concatenate([
|
| 255 |
+
delta_position, delta_rotation, [fraction[min(index, frame_count - 1)]],
|
| 256 |
+
])
|
| 257 |
+
# 非有限值宁缺勿脏:该帧不写,由下游按缺失处理。
|
| 258 |
+
if not np.all(np.isfinite(action)):
|
| 259 |
+
continue
|
| 260 |
+
result[frame] = {
|
| 261 |
+
"action": [float(value) for value in action],
|
| 262 |
+
"observation.state": [
|
| 263 |
+
float(value) for value in np.concatenate([
|
| 264 |
+
cleaned[frame, :3] - origin,
|
| 265 |
+
[fraction[frame]],
|
| 266 |
+
])
|
| 267 |
+
],
|
| 268 |
+
}
|
| 269 |
+
return result or None
|
processor/tests/test_batch_fingerprint.py
ADDED
|
@@ -0,0 +1,181 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""内容指纹:识别"换个包名重传同一份数据"。
|
| 2 |
+
|
| 3 |
+
实测背景:采集端把已上传的录制重新推一遍时包名会变(本地去重标记丢失
|
| 4 |
+
或包名规则变化),``session.py`` 的 incoming_name 判断不出来,同一份数据
|
| 5 |
+
被追加成新 episode(UMIGripper_AI 出现 4 组字节级相同的批次)。这里的
|
| 6 |
+
指纹必须与包名无关,且对"同一次录制的两份拷贝"稳定命中。
|
| 7 |
+
"""
|
| 8 |
+
|
| 9 |
+
import json
|
| 10 |
+
import tempfile
|
| 11 |
+
import unittest
|
| 12 |
+
from pathlib import Path
|
| 13 |
+
from unittest import mock
|
| 14 |
+
|
| 15 |
+
import numpy as np
|
| 16 |
+
import pyarrow as pa
|
| 17 |
+
import pyarrow.parquet as pq
|
| 18 |
+
|
| 19 |
+
from app import batch_fingerprint
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
def _write_episode(root: Path, episode_index: int, video_bytes: bytes,
|
| 23 |
+
frames: int = 4) -> None:
|
| 24 |
+
"""写一个最小 canonical 布局:一个视频 + 一个 data parquet。"""
|
| 25 |
+
videos = root / "videos" / "observation.images.gripper_rgb" / "chunk-000"
|
| 26 |
+
videos.mkdir(parents=True, exist_ok=True)
|
| 27 |
+
(videos / f"episode_{episode_index:06d}.mp4").write_bytes(video_bytes)
|
| 28 |
+
|
| 29 |
+
data = root / "data" / "chunk-000"
|
| 30 |
+
data.mkdir(parents=True, exist_ok=True)
|
| 31 |
+
table = pa.table({
|
| 32 |
+
"episode_index": pa.array([episode_index] * frames, pa.int64()),
|
| 33 |
+
"frame_index": pa.array(list(range(frames)), pa.int64()),
|
| 34 |
+
"observation.value": pa.array(
|
| 35 |
+
[[float(i), float(i) + 1.0] for i in range(frames)],
|
| 36 |
+
pa.list_(pa.float32()),
|
| 37 |
+
),
|
| 38 |
+
})
|
| 39 |
+
pq.write_table(table, data / f"episode_{episode_index:06d}.parquet")
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
class FingerprintTests(unittest.TestCase):
|
| 43 |
+
def setUp(self):
|
| 44 |
+
self._tmp = tempfile.TemporaryDirectory()
|
| 45 |
+
self.root = Path(self._tmp.name)
|
| 46 |
+
# 索引落到临时目录,别碰真实 state/
|
| 47 |
+
self._index_patch = mock.patch.object(
|
| 48 |
+
batch_fingerprint, "INDEX_PATH", self.root / "fingerprints.json")
|
| 49 |
+
self._index_patch.start()
|
| 50 |
+
|
| 51 |
+
def tearDown(self):
|
| 52 |
+
self._index_patch.stop()
|
| 53 |
+
self._tmp.cleanup()
|
| 54 |
+
|
| 55 |
+
def test_same_recording_different_episode_index_matches(self):
|
| 56 |
+
"""同一次录制的两份拷贝,只有 episode_index 不同 → 指纹相同。"""
|
| 57 |
+
a = self.root / "a"
|
| 58 |
+
b = self.root / "b"
|
| 59 |
+
_write_episode(a, 3, b"IDENTICAL-VIDEO-BYTES")
|
| 60 |
+
_write_episode(b, 4, b"IDENTICAL-VIDEO-BYTES")
|
| 61 |
+
|
| 62 |
+
self.assertIsNotNone(batch_fingerprint.compute_fingerprint(a))
|
| 63 |
+
self.assertEqual(
|
| 64 |
+
batch_fingerprint.compute_fingerprint(a),
|
| 65 |
+
batch_fingerprint.compute_fingerprint(b),
|
| 66 |
+
)
|
| 67 |
+
|
| 68 |
+
def test_different_video_differs(self):
|
| 69 |
+
a = self.root / "a"
|
| 70 |
+
b = self.root / "b"
|
| 71 |
+
_write_episode(a, 0, b"VIDEO-ONE")
|
| 72 |
+
_write_episode(b, 0, b"VIDEO-TWO")
|
| 73 |
+
self.assertNotEqual(
|
| 74 |
+
batch_fingerprint.compute_fingerprint(a),
|
| 75 |
+
batch_fingerprint.compute_fingerprint(b),
|
| 76 |
+
)
|
| 77 |
+
|
| 78 |
+
def test_different_frame_data_differs(self):
|
| 79 |
+
"""视频相同但帧数据不同 → 不同指纹(不能只比视频)。"""
|
| 80 |
+
a = self.root / "a"
|
| 81 |
+
b = self.root / "b"
|
| 82 |
+
_write_episode(a, 0, b"SAME-VIDEO")
|
| 83 |
+
_write_episode(b, 0, b"SAME-VIDEO")
|
| 84 |
+
# 只改 b 的一帧数据
|
| 85 |
+
path = b / "data" / "chunk-000" / "episode_000000.parquet"
|
| 86 |
+
table = pq.read_table(path).to_pydict()
|
| 87 |
+
table["observation.value"] = [[9.0, 9.0] for _ in table["frame_index"]]
|
| 88 |
+
pq.write_table(pa.table(table), path)
|
| 89 |
+
self.assertNotEqual(
|
| 90 |
+
batch_fingerprint.compute_fingerprint(a),
|
| 91 |
+
batch_fingerprint.compute_fingerprint(b),
|
| 92 |
+
)
|
| 93 |
+
|
| 94 |
+
def test_missing_data_returns_none(self):
|
| 95 |
+
"""没有 parquet 就算不出指纹 → None(调用方按新批次放行)。"""
|
| 96 |
+
empty = self.root / "empty"
|
| 97 |
+
empty.mkdir()
|
| 98 |
+
self.assertIsNone(batch_fingerprint.compute_fingerprint(empty))
|
| 99 |
+
|
| 100 |
+
def test_record_and_lookup_roundtrip(self):
|
| 101 |
+
fingerprint = "v:test=abc;d:def"
|
| 102 |
+
batch_fingerprint.record_fingerprint("UMIGripper_AI", fingerprint, "ep_000003")
|
| 103 |
+
self.assertEqual(
|
| 104 |
+
batch_fingerprint.lookup_duplicate(
|
| 105 |
+
"UMIGripper_AI", fingerprint, {"ep_000003"}),
|
| 106 |
+
"ep_000003",
|
| 107 |
+
)
|
| 108 |
+
|
| 109 |
+
def test_lookup_ignores_deleted_episode(self):
|
| 110 |
+
"""索引指向的 episode 已删除 → 不判重(允许重新上传)。"""
|
| 111 |
+
fingerprint = "v:test=abc;d:def"
|
| 112 |
+
batch_fingerprint.record_fingerprint("UMIGripper_AI", fingerprint, "ep_000003")
|
| 113 |
+
self.assertIsNone(
|
| 114 |
+
batch_fingerprint.lookup_duplicate(
|
| 115 |
+
"UMIGripper_AI", fingerprint, {"ep_000004"}),
|
| 116 |
+
)
|
| 117 |
+
|
| 118 |
+
def test_lookup_none_fingerprint_is_safe(self):
|
| 119 |
+
self.assertIsNone(
|
| 120 |
+
batch_fingerprint.lookup_duplicate("p", None, {"a"}))
|
| 121 |
+
|
| 122 |
+
def test_record_drops_stale_entry_for_same_episode(self):
|
| 123 |
+
"""同名重传换了内容:旧指纹不该继续指向这个 episode。"""
|
| 124 |
+
batch_fingerprint.record_fingerprint("p", "old-content", "ep_1")
|
| 125 |
+
batch_fingerprint.record_fingerprint("p", "new-content", "ep_1")
|
| 126 |
+
self.assertIsNone(
|
| 127 |
+
batch_fingerprint.lookup_duplicate("p", "old-content", {"ep_1"}))
|
| 128 |
+
self.assertEqual(
|
| 129 |
+
batch_fingerprint.lookup_duplicate("p", "new-content", {"ep_1"}),
|
| 130 |
+
"ep_1",
|
| 131 |
+
)
|
| 132 |
+
|
| 133 |
+
def test_forget_fingerprint(self):
|
| 134 |
+
batch_fingerprint.record_fingerprint("p", "abc", "ep_1")
|
| 135 |
+
batch_fingerprint.forget_fingerprint("ep_1")
|
| 136 |
+
self.assertIsNone(
|
| 137 |
+
batch_fingerprint.lookup_duplicate("p", "abc", {"ep_1"}))
|
| 138 |
+
|
| 139 |
+
def test_corrupt_index_is_tolerated(self):
|
| 140 |
+
batch_fingerprint.INDEX_PATH.write_text("{not json", encoding="utf-8")
|
| 141 |
+
self.assertIsNone(
|
| 142 |
+
batch_fingerprint.lookup_duplicate("p", "abc", {"ep_1"}))
|
| 143 |
+
batch_fingerprint.record_fingerprint("p", "abc", "ep_1")
|
| 144 |
+
self.assertEqual(
|
| 145 |
+
batch_fingerprint.lookup_duplicate("p", "abc", {"ep_1"}), "ep_1")
|
| 146 |
+
|
| 147 |
+
def test_index_is_valid_json_with_episode_map(self):
|
| 148 |
+
batch_fingerprint.record_fingerprint("proj", "fp1", "ep_1")
|
| 149 |
+
data = json.loads(batch_fingerprint.INDEX_PATH.read_text(encoding="utf-8"))
|
| 150 |
+
self.assertEqual(data["proj"]["fp1"], "ep_1")
|
| 151 |
+
|
| 152 |
+
|
| 153 |
+
class ParquetHashTests(unittest.TestCase):
|
| 154 |
+
def test_episode_index_column_is_excluded(self):
|
| 155 |
+
with tempfile.TemporaryDirectory() as tmp:
|
| 156 |
+
root = Path(tmp)
|
| 157 |
+
table = pa.table({
|
| 158 |
+
"episode_index": pa.array([7], pa.int64()),
|
| 159 |
+
"frame_index": pa.array([0], pa.int64()),
|
| 160 |
+
"v": pa.array([[1.0, 2.0]], pa.list_(pa.float32())),
|
| 161 |
+
})
|
| 162 |
+
one = root / "one.parquet"
|
| 163 |
+
pq.write_table(table, one)
|
| 164 |
+
other = root / "other.parquet"
|
| 165 |
+
pq.write_table(
|
| 166 |
+
table.set_column(0, "episode_index",
|
| 167 |
+
pa.array([99], pa.int64())), other)
|
| 168 |
+
self.assertEqual(
|
| 169 |
+
batch_fingerprint._parquet_content_hash(one),
|
| 170 |
+
batch_fingerprint._parquet_content_hash(other),
|
| 171 |
+
)
|
| 172 |
+
|
| 173 |
+
def test_unreadable_parquet_returns_none(self):
|
| 174 |
+
with tempfile.TemporaryDirectory() as tmp:
|
| 175 |
+
bad = Path(tmp) / "bad.parquet"
|
| 176 |
+
bad.write_bytes(b"not a parquet file")
|
| 177 |
+
self.assertIsNone(batch_fingerprint._parquet_content_hash(bad))
|
| 178 |
+
|
| 179 |
+
|
| 180 |
+
if __name__ == "__main__":
|
| 181 |
+
unittest.main()
|
processor/tests/test_device_task_progress.py
ADDED
|
@@ -0,0 +1,181 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""采集端任务进度账本单测(app/device_progress.py)。
|
| 2 |
+
|
| 3 |
+
覆盖:幂等重放、多设备聚合、增量水位、参数校验、未知任务拒绝(400 而非
|
| 4 |
+
404 —— 采集端把 404 当"后端未实现"永久降级)、持久化与坏文件容错。
|
| 5 |
+
|
| 6 |
+
pytest tests/test_device_task_progress.py
|
| 7 |
+
python3 tests/test_device_task_progress.py # 无 pytest 也能跑
|
| 8 |
+
"""
|
| 9 |
+
import json
|
| 10 |
+
import sys
|
| 11 |
+
from pathlib import Path
|
| 12 |
+
|
| 13 |
+
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
| 14 |
+
|
| 15 |
+
from app import device_progress as dp # noqa: E402
|
| 16 |
+
|
| 17 |
+
TASK = "task-aaa"
|
| 18 |
+
OTHER = "task-bbb"
|
| 19 |
+
KNOWN = {TASK, OTHER}
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
def _report(tmp, increment, session_id, task=TASK, device="EGO_001"):
|
| 23 |
+
return dp.apply_report(
|
| 24 |
+
{"task_id": task, "device_name": device,
|
| 25 |
+
"session_id": session_id, "increment": increment},
|
| 26 |
+
KNOWN, path=tmp)
|
| 27 |
+
|
| 28 |
+
|
| 29 |
+
def test_first_report_sets_total(tmp_path):
|
| 30 |
+
f = tmp_path / "p.json"
|
| 31 |
+
out = _report(f, 1, "EGO_001:t:0")
|
| 32 |
+
assert out["applied"] is True
|
| 33 |
+
assert out["completed_count"] == 1
|
| 34 |
+
assert dp.reported_total(TASK, path=f) == 1
|
| 35 |
+
|
| 36 |
+
|
| 37 |
+
def test_replayed_watermark_is_idempotent(tmp_path):
|
| 38 |
+
"""采集端崩溃重启后会重发同一水位 —— 不能重复计数。"""
|
| 39 |
+
f = tmp_path / "p.json"
|
| 40 |
+
first = _report(f, 12, "EGO_001:t:0")
|
| 41 |
+
again = _report(f, 12, "EGO_001:t:0")
|
| 42 |
+
assert first["completed_count"] == 12
|
| 43 |
+
assert again["applied"] is False
|
| 44 |
+
assert again["completed_count"] == 12
|
| 45 |
+
# 重放仍要回当前聚合数(采集端拿它当后端权威数)
|
| 46 |
+
assert isinstance(again["completed_count"], int)
|
| 47 |
+
assert not isinstance(again["completed_count"], bool)
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
def test_incremental_watermarks_accumulate(tmp_path):
|
| 51 |
+
f = tmp_path / "p.json"
|
| 52 |
+
_report(f, 12, "EGO_001:t:0")
|
| 53 |
+
out = _report(f, 1, "EGO_001:t:12")
|
| 54 |
+
assert out["completed_count"] == 13
|
| 55 |
+
|
| 56 |
+
|
| 57 |
+
def test_multiple_devices_aggregate(tmp_path):
|
| 58 |
+
f = tmp_path / "p.json"
|
| 59 |
+
_report(f, 3, "EGO_001:t:0", device="EGO_001")
|
| 60 |
+
_report(f, 2, "EGO_002:t:0", device="EGO_002")
|
| 61 |
+
out = _report(f, 1, "EGO_003:t:0", device="EGO_003")
|
| 62 |
+
assert out["completed_count"] == 6
|
| 63 |
+
# 同一水位键跨设备不冲突
|
| 64 |
+
assert dp.reported_total(TASK, path=f) == 6
|
| 65 |
+
|
| 66 |
+
|
| 67 |
+
def test_zero_increment_is_recorded_without_growth(tmp_path):
|
| 68 |
+
f = tmp_path / "p.json"
|
| 69 |
+
_report(f, 5, "EGO_001:t:0")
|
| 70 |
+
out = _report(f, 0, "EGO_001:t:5")
|
| 71 |
+
assert out["applied"] is True and out["completed_count"] == 5
|
| 72 |
+
|
| 73 |
+
|
| 74 |
+
def test_totals_only_contain_reported_tasks(tmp_path):
|
| 75 |
+
"""没上报过的项目不能出现在这里 —— 调用方据此回退 session 计数。"""
|
| 76 |
+
f = tmp_path / "p.json"
|
| 77 |
+
_report(f, 1, "EGO_001:t:0")
|
| 78 |
+
totals = dp.reported_totals(path=f)
|
| 79 |
+
assert totals == {TASK: 1}
|
| 80 |
+
assert dp.reported_total(OTHER, path=f) is None
|
| 81 |
+
|
| 82 |
+
|
| 83 |
+
def test_validation_errors(tmp_path):
|
| 84 |
+
f = tmp_path / "p.json"
|
| 85 |
+
bad = [
|
| 86 |
+
({}, "task_id"), # 空 body
|
| 87 |
+
({"task_id": TASK, "session_id": "s"}, "device_name"),
|
| 88 |
+
({"task_id": TASK, "device_name": "d"}, "session_id"),
|
| 89 |
+
({"task_id": TASK, "device_name": "d", "session_id": "s",
|
| 90 |
+
"increment": -1}, "increment"),
|
| 91 |
+
({"task_id": TASK, "device_name": "d", "session_id": "s",
|
| 92 |
+
"increment": "x"}, "increment"),
|
| 93 |
+
({"task_id": TASK, "device_name": "d", "session_id": "s",
|
| 94 |
+
"increment": True}, "increment"),
|
| 95 |
+
({"task_id": TASK, "device_name": "d", "session_id": "s",
|
| 96 |
+
"increment": dp.MAX_INCREMENT + 1}, "increment"),
|
| 97 |
+
]
|
| 98 |
+
for body, needle in bad:
|
| 99 |
+
try:
|
| 100 |
+
dp.apply_report(body, KNOWN, path=f)
|
| 101 |
+
except ValueError as exc:
|
| 102 |
+
assert needle in str(exc), (body, str(exc))
|
| 103 |
+
else:
|
| 104 |
+
raise AssertionError(f"应当拒绝: {body}")
|
| 105 |
+
assert dp.reported_totals(path=f) == {} # 拒绝的请求不落账
|
| 106 |
+
|
| 107 |
+
|
| 108 |
+
def test_unknown_task_rejected_not_silently_accepted(tmp_path):
|
| 109 |
+
"""未知任务要报错(路由转 400)。绝不能 404:采集端会永久降级上报。"""
|
| 110 |
+
f = tmp_path / "p.json"
|
| 111 |
+
try:
|
| 112 |
+
_report(f, 1, "EGO_001:t:0", task="no-such-task")
|
| 113 |
+
except ValueError as exc:
|
| 114 |
+
assert "unknown task_id" in str(exc)
|
| 115 |
+
else:
|
| 116 |
+
raise AssertionError("未知任务应当被拒绝")
|
| 117 |
+
# known_task_ids=None(不校验)时照常落账 —— 路由必须传已知集合
|
| 118 |
+
out = dp.apply_report({"task_id": "no-such-task", "device_name": "d",
|
| 119 |
+
"session_id": "s", "increment": 1}, None, path=f)
|
| 120 |
+
assert out["completed_count"] == 1
|
| 121 |
+
|
| 122 |
+
|
| 123 |
+
def test_count_prefers_report_over_sessions(tmp_path):
|
| 124 |
+
"""核心口径:有上报就用录制数 —— 之后再上传多少条都不动它。"""
|
| 125 |
+
f = tmp_path / "p.json"
|
| 126 |
+
proj = {"id": TASK, "name": "UMIGripper_AI"}
|
| 127 |
+
_report(f, 12, "EGO_001:t:0")
|
| 128 |
+
reported = dp.reported_totals(path=f)
|
| 129 |
+
assert dp.resolve_count(proj, reported, sessions=16) == (12, "reported")
|
| 130 |
+
# 又上传了 3 条(项目下 session 变 19)→ 进度不变
|
| 131 |
+
assert dp.resolve_count(proj, reported, sessions=19) == (12, "reported")
|
| 132 |
+
# 录了一条 → 上报后 +1
|
| 133 |
+
_report(f, 1, "EGO_001:t:12")
|
| 134 |
+
assert dp.resolve_count(proj, dp.reported_totals(path=f),
|
| 135 |
+
sessions=19) == (13, "reported")
|
| 136 |
+
|
| 137 |
+
|
| 138 |
+
def test_count_falls_back_to_sessions_before_any_report(tmp_path):
|
| 139 |
+
"""升级前就有数据的历史项目不归零:没有上报时仍按 session 数。"""
|
| 140 |
+
proj = {"id": TASK, "name": "UMIGripper_AI"}
|
| 141 |
+
assert dp.resolve_count(proj, {}, sessions=16) == (16, "sessions")
|
| 142 |
+
assert dp.needs_session_fallback([proj], {}) is True
|
| 143 |
+
assert dp.needs_session_fallback([proj], {TASK: 12}) is False
|
| 144 |
+
|
| 145 |
+
|
| 146 |
+
def test_persisted_across_reload(tmp_path):
|
| 147 |
+
f = tmp_path / "p.json"
|
| 148 |
+
_report(f, 2, "EGO_001:t:0")
|
| 149 |
+
raw = json.loads(f.read_text(encoding="utf-8"))
|
| 150 |
+
assert raw["tasks"][TASK]["devices"]["EGO_001"]["count"] == 2
|
| 151 |
+
# 重新读文件(模拟进程重启)后水位仍生效
|
| 152 |
+
again = _report(f, 2, "EGO_001:t:0")
|
| 153 |
+
assert again["applied"] is False and again["completed_count"] == 2
|
| 154 |
+
|
| 155 |
+
|
| 156 |
+
def test_corrupt_file_degrades_to_empty(tmp_path):
|
| 157 |
+
f = tmp_path / "p.json"
|
| 158 |
+
f.write_text("{ not json", encoding="utf-8")
|
| 159 |
+
assert dp.reported_totals(path=f) == {}
|
| 160 |
+
out = _report(f, 1, "EGO_001:t:0")
|
| 161 |
+
assert out["completed_count"] == 1
|
| 162 |
+
|
| 163 |
+
|
| 164 |
+
if __name__ == "__main__":
|
| 165 |
+
import tempfile
|
| 166 |
+
import traceback
|
| 167 |
+
|
| 168 |
+
failed = 0
|
| 169 |
+
for name, fn in sorted(globals().items()):
|
| 170 |
+
if not name.startswith("test_") or not callable(fn):
|
| 171 |
+
continue
|
| 172 |
+
with tempfile.TemporaryDirectory() as d:
|
| 173 |
+
try:
|
| 174 |
+
fn(Path(d))
|
| 175 |
+
print(f" PASS: {name}")
|
| 176 |
+
except Exception:
|
| 177 |
+
failed += 1
|
| 178 |
+
print(f" FAIL: {name}")
|
| 179 |
+
traceback.print_exc()
|
| 180 |
+
print("FAIL" if failed else "PASS: device_task_progress 全部通过")
|
| 181 |
+
sys.exit(1 if failed else 0)
|
processor/tests/test_lerobot_export.py
ADDED
|
@@ -0,0 +1,105 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import unittest
|
| 2 |
+
|
| 3 |
+
import numpy as np
|
| 4 |
+
|
| 5 |
+
from app.lerobot_export import _ensure_slam_pose, _ensure_observation_state
|
| 6 |
+
from app.umi_slam_action import poses_from_trajectory
|
| 7 |
+
|
| 8 |
+
|
| 9 |
+
def _sample(t: float, x: float, z: float) -> list[float]:
|
| 10 |
+
"""一组 slam_trajectory 样本:[t, x, y, z, qx, qy, qz, qw]。"""
|
| 11 |
+
return [float(t), float(x), 0.0, float(z), 0.0, 0.0, 0.0, 1.0]
|
| 12 |
+
|
| 13 |
+
|
| 14 |
+
def _rows(episode: int, count: int, *, t0: float,
|
| 15 |
+
slam_offset: float = 0.0) -> list[dict]:
|
| 16 |
+
"""造一集 rows:每帧两个 slam 样本,位姿随帧号线性推进。"""
|
| 17 |
+
rows = []
|
| 18 |
+
for index in range(count):
|
| 19 |
+
stamp = t0 + index / 30.0
|
| 20 |
+
rows.append({
|
| 21 |
+
"episode_index": episode,
|
| 22 |
+
"timestamp": stamp,
|
| 23 |
+
"observation.slam_trajectory": [
|
| 24 |
+
_sample(stamp - slam_offset, index * 0.01, index * 0.002),
|
| 25 |
+
_sample(stamp - slam_offset + 0.01,
|
| 26 |
+
index * 0.01 + 0.001, index * 0.002),
|
| 27 |
+
],
|
| 28 |
+
})
|
| 29 |
+
return rows
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
class EnsureSlamPoseTests(unittest.TestCase):
|
| 33 |
+
def test_derives_column_and_matches_direct_reconstruction(self):
|
| 34 |
+
"""整列缺失时补出 slam_pose,且值与单独调用重建函数逐值一致。"""
|
| 35 |
+
rows = _rows(0, 40, t0=100.0)
|
| 36 |
+
self.assertEqual(_ensure_slam_pose(rows), "observation.slam_pose")
|
| 37 |
+
|
| 38 |
+
self.assertTrue(all(len(row["observation.slam_pose"]) == 7
|
| 39 |
+
for row in rows))
|
| 40 |
+
expected = poses_from_trajectory(
|
| 41 |
+
[row["observation.slam_trajectory"] for row in rows],
|
| 42 |
+
np.array([row["timestamp"] for row in rows]), len(rows))
|
| 43 |
+
actual = np.array([row["observation.slam_pose"] for row in rows])
|
| 44 |
+
self.assertTrue(np.allclose(actual, expected, atol=1e-12))
|
| 45 |
+
|
| 46 |
+
def test_episodes_are_fitted_independently(self):
|
| 47 |
+
"""时间基映射必须每集单独拟合,否则跨集求差会把原点差异当运动。
|
| 48 |
+
|
| 49 |
+
两集的 slam 时钟相差一个很大的偏移量;若混在一起拟合,时间轴被拉
|
| 50 |
+
成两段,重建值会偏离该集单独重建的结果。
|
| 51 |
+
"""
|
| 52 |
+
first = _rows(0, 30, t0=100.0, slam_offset=0.0)
|
| 53 |
+
second = _rows(1, 30, t0=500.0, slam_offset=400.0)
|
| 54 |
+
rows = first + second
|
| 55 |
+
self.assertEqual(_ensure_slam_pose(rows), "observation.slam_pose")
|
| 56 |
+
|
| 57 |
+
for group, offset in ((first, 0.0), (second, 400.0)):
|
| 58 |
+
expected = poses_from_trajectory(
|
| 59 |
+
[row["observation.slam_trajectory"] for row in group],
|
| 60 |
+
np.array([row["timestamp"] for row in group]), len(group),
|
| 61 |
+
)
|
| 62 |
+
actual = np.array([row["observation.slam_pose"] for row in group])
|
| 63 |
+
self.assertTrue(np.allclose(actual, expected, atol=1e-12))
|
| 64 |
+
|
| 65 |
+
def test_existing_column_is_left_untouched(self):
|
| 66 |
+
"""旧批两列都有 —— 重导时不得覆盖采集端原始 slam_pose。"""
|
| 67 |
+
rows = _rows(0, 10, t0=100.0)
|
| 68 |
+
original = [9.0] * 7
|
| 69 |
+
for row in rows:
|
| 70 |
+
row["observation.slam_pose"] = list(original)
|
| 71 |
+
self.assertIsNone(_ensure_slam_pose(rows))
|
| 72 |
+
self.assertTrue(all(row["observation.slam_pose"] == original
|
| 73 |
+
for row in rows))
|
| 74 |
+
|
| 75 |
+
def test_skips_batches_without_trajectory(self):
|
| 76 |
+
"""手套 / D435 批次没有 slam_trajectory —— 不得凭空写入该列。"""
|
| 77 |
+
rows = [{"episode_index": 0, "timestamp": 0.0,
|
| 78 |
+
"observation.tactile.left": [0.0] * 256}]
|
| 79 |
+
self.assertIsNone(_ensure_slam_pose(rows))
|
| 80 |
+
self.assertNotIn("observation.slam_pose", rows[0])
|
| 81 |
+
|
| 82 |
+
def test_missing_timestamp_does_not_break_reconstruction(self):
|
| 83 |
+
"""时间戳缺帧(None)不能把 np.asarray 推断成 object 而抛异常。"""
|
| 84 |
+
rows = _rows(0, 20, t0=200.0)
|
| 85 |
+
rows[5]["timestamp"] = None
|
| 86 |
+
self.assertEqual(_ensure_slam_pose(rows), "observation.slam_pose")
|
| 87 |
+
self.assertTrue(all(np.all(np.isfinite(row["observation.slam_pose"]))
|
| 88 |
+
for row in rows))
|
| 89 |
+
|
| 90 |
+
def test_derived_pose_feeds_observation_state(self):
|
| 91 |
+
"""串联顺序:补出的 slam_pose 要能继续供 state 派生使用。
|
| 92 |
+
|
| 93 |
+
新批里 ep72/ep73 两列都缺,只补 slam_pose 而不让 state 跟上,导出
|
| 94 |
+
产物仍然缺 observation.state,ACT 依旧 KeyError。
|
| 95 |
+
"""
|
| 96 |
+
rows = _rows(0, 25, t0=300.0)
|
| 97 |
+
_ensure_slam_pose(rows)
|
| 98 |
+
self.assertEqual(_ensure_observation_state(rows), "observation.state")
|
| 99 |
+
self.assertTrue(all(len(row["observation.state"]) == 4 for row in rows))
|
| 100 |
+
# 位置分量是本帧相对本集第一帧的偏移 —— 首帧必须为原点
|
| 101 |
+
self.assertTrue(np.allclose(rows[0]["observation.state"][:3], 0.0))
|
| 102 |
+
|
| 103 |
+
|
| 104 |
+
if __name__ == "__main__":
|
| 105 |
+
unittest.main()
|
processor/tests/test_tactile_decode.py
CHANGED
|
@@ -1,8 +1,16 @@
|
|
|
|
|
|
|
|
| 1 |
import unittest
|
|
|
|
| 2 |
|
| 3 |
import numpy as np
|
| 4 |
|
| 5 |
-
from app.routes.video import
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 6 |
|
| 7 |
|
| 8 |
class TactileDecodeTests(unittest.TestCase):
|
|
@@ -23,5 +31,142 @@ class TactileDecodeTests(unittest.TestCase):
|
|
| 23 |
np.testing.assert_array_equal(decoded.transpose(1, 2, 0), matrix)
|
| 24 |
|
| 25 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 26 |
if __name__ == "__main__":
|
| 27 |
unittest.main()
|
|
|
|
| 1 |
+
import json
|
| 2 |
+
import tempfile
|
| 3 |
import unittest
|
| 4 |
+
from pathlib import Path
|
| 5 |
|
| 6 |
import numpy as np
|
| 7 |
|
| 8 |
+
from app.routes.video import (
|
| 9 |
+
_apply_tactile_scale,
|
| 10 |
+
_compute_tactile_display_range,
|
| 11 |
+
_decode_tactile_matrix,
|
| 12 |
+
_tactile_scale_factor,
|
| 13 |
+
)
|
| 14 |
|
| 15 |
|
| 16 |
class TactileDecodeTests(unittest.TestCase):
|
|
|
|
| 31 |
np.testing.assert_array_equal(decoded.transpose(1, 2, 0), matrix)
|
| 32 |
|
| 33 |
|
| 34 |
+
class TactileScaleTests(unittest.TestCase):
|
| 35 |
+
"""量化整数 → mN 的比例因子必须取自元数据,不能硬编码。
|
| 36 |
+
|
| 37 |
+
新采集端按 ``scale: 100`` 存储(units 仍是 mN);旧数据无该字段且
|
| 38 |
+
本身就是 mN。硬编码 100 会把旧数据错误缩小 100 倍。
|
| 39 |
+
"""
|
| 40 |
+
|
| 41 |
+
def _dataset(self, root: Path, scale) -> Path:
|
| 42 |
+
(root / "meta").mkdir(parents=True)
|
| 43 |
+
(root / "data" / "chunk-000").mkdir(parents=True)
|
| 44 |
+
feature = {"dtype": "int16", "shape": [250, 250, 3],
|
| 45 |
+
"units": ["mN", "mN", "mN"]}
|
| 46 |
+
if scale is not None:
|
| 47 |
+
feature["scale"] = scale
|
| 48 |
+
(root / "meta" / "info.json").write_text(json.dumps({
|
| 49 |
+
"features": {"observation.gripper_left_force_matrix": feature},
|
| 50 |
+
}), encoding="utf-8")
|
| 51 |
+
return root / "data" / "chunk-000" / "episode_000000.parquet"
|
| 52 |
+
|
| 53 |
+
def test_reads_declared_scale(self):
|
| 54 |
+
with tempfile.TemporaryDirectory() as temp:
|
| 55 |
+
path = self._dataset(Path(temp), 100)
|
| 56 |
+
self.assertEqual(_tactile_scale_factor(path), 100.0)
|
| 57 |
+
|
| 58 |
+
def test_defaults_to_one_when_declared_or_missing(self):
|
| 59 |
+
with tempfile.TemporaryDirectory() as temp:
|
| 60 |
+
path = self._dataset(Path(temp), None)
|
| 61 |
+
self.assertEqual(_tactile_scale_factor(path), 1.0)
|
| 62 |
+
|
| 63 |
+
with tempfile.TemporaryDirectory() as temp:
|
| 64 |
+
path = self._dataset(Path(temp), 0)
|
| 65 |
+
self.assertEqual(_tactile_scale_factor(path), 1.0)
|
| 66 |
+
|
| 67 |
+
def test_missing_metadata_is_not_an_error(self):
|
| 68 |
+
with tempfile.TemporaryDirectory() as temp:
|
| 69 |
+
path = Path(temp) / "data" / "chunk-000" / "episode_000000.parquet"
|
| 70 |
+
path.parent.mkdir(parents=True)
|
| 71 |
+
self.assertEqual(_tactile_scale_factor(path), 1.0)
|
| 72 |
+
|
| 73 |
+
def test_old_data_is_left_untouched(self):
|
| 74 |
+
"""scale=1 时不得改变数值 —— 旧数据必须保持原样。"""
|
| 75 |
+
matrix = np.full((3, 250, 250), 5.0, dtype=np.float32)
|
| 76 |
+
scaled = _apply_tactile_scale(matrix, 1.0)
|
| 77 |
+
np.testing.assert_array_equal(scaled, matrix)
|
| 78 |
+
|
| 79 |
+
def test_new_data_is_divided(self):
|
| 80 |
+
matrix = np.full((3, 250, 250), 250.0, dtype=np.float32)
|
| 81 |
+
scaled = _apply_tactile_scale(matrix, 100.0)
|
| 82 |
+
np.testing.assert_allclose(scaled, 2.5, atol=1e-6)
|
| 83 |
+
|
| 84 |
+
def test_none_matrix_is_passed_through(self):
|
| 85 |
+
self.assertIsNone(_apply_tactile_scale(None, 100.0))
|
| 86 |
+
|
| 87 |
+
|
| 88 |
+
def _write_episode(path: Path, matrices) -> Path:
|
| 89 |
+
"""Write an episode whose row-diff column decodes back to ``matrices``."""
|
| 90 |
+
import pyarrow as pa
|
| 91 |
+
import pyarrow.parquet as pq
|
| 92 |
+
|
| 93 |
+
flattened = []
|
| 94 |
+
for matrix in matrices:
|
| 95 |
+
rows = np.asarray(matrix, dtype=np.int16).reshape(250, 750)
|
| 96 |
+
diff = np.empty_like(rows)
|
| 97 |
+
diff[:, 0] = rows[:, 0]
|
| 98 |
+
diff[:, 1:] = rows[:, 1:] - rows[:, :-1]
|
| 99 |
+
flattened.append(diff.reshape(-1))
|
| 100 |
+
table = pa.table({
|
| 101 |
+
"frame_index": pa.array(range(len(matrices)), type=pa.int64()),
|
| 102 |
+
"observation.gripper_left_force_matrix":
|
| 103 |
+
pa.array(flattened, type=pa.list_(pa.int16())),
|
| 104 |
+
})
|
| 105 |
+
path.parent.mkdir(parents=True, exist_ok=True)
|
| 106 |
+
pq.write_table(table, str(path))
|
| 107 |
+
return path
|
| 108 |
+
|
| 109 |
+
|
| 110 |
+
def _episode_frames(quiet=9, contact=80):
|
| 111 |
+
"""A quiet episode (noise only) with one frame carrying real contact."""
|
| 112 |
+
rng = np.random.default_rng(20260911)
|
| 113 |
+
matrices = []
|
| 114 |
+
for _ in range(quiet):
|
| 115 |
+
frame = np.zeros((250, 250, 3), dtype=np.int16)
|
| 116 |
+
frame[:, :, 2] = rng.integers(0, 3, (250, 250))
|
| 117 |
+
matrices.append(frame)
|
| 118 |
+
loaded = np.zeros((250, 250, 3), dtype=np.int16)
|
| 119 |
+
loaded[100:130, 100:130, 2] = contact
|
| 120 |
+
matrices.append(loaded)
|
| 121 |
+
return matrices
|
| 122 |
+
|
| 123 |
+
|
| 124 |
+
class TactileDisplayRangeTests(unittest.TestCase):
|
| 125 |
+
"""色阶必须取自整集,不能逐帧自比。
|
| 126 |
+
|
| 127 |
+
没有接触的帧里唯一超过噪声门限的就是噪声本身;逐帧归一化会把这份噪声
|
| 128 |
+
当成满量程,每个噪点都被拉成"接触点"。整集取色阶后安静帧保持背景色。
|
| 129 |
+
"""
|
| 130 |
+
|
| 131 |
+
def _episode(self, temp: str):
|
| 132 |
+
path = _write_episode(
|
| 133 |
+
Path(temp) / "data" / "chunk-000" / "episode_000000.parquet",
|
| 134 |
+
_episode_frames())
|
| 135 |
+
return path
|
| 136 |
+
|
| 137 |
+
def test_range_is_driven_by_the_contact_not_the_noise(self):
|
| 138 |
+
with tempfile.TemporaryDirectory() as temp:
|
| 139 |
+
path = self._episode(temp)
|
| 140 |
+
lo, hi = _compute_tactile_display_range(
|
| 141 |
+
path, "observation.gripper_left_force_matrix")
|
| 142 |
+
self.assertGreaterEqual(hi, 80.0)
|
| 143 |
+
self.assertLess(lo, hi)
|
| 144 |
+
|
| 145 |
+
def test_quiet_frame_stays_background(self):
|
| 146 |
+
"""整集色阶下,无接触的帧几乎全黑;旧逐帧逻辑在该帧给出 174 级亮度。"""
|
| 147 |
+
from app.routes.video import _tactile_range_payload
|
| 148 |
+
|
| 149 |
+
with tempfile.TemporaryDirectory() as temp:
|
| 150 |
+
path = self._episode(temp)
|
| 151 |
+
payload = _tactile_range_payload(path, 0, 10, "left")
|
| 152 |
+
self.assertIsNotNone(payload)
|
| 153 |
+
pixels = 250 * 250
|
| 154 |
+
blocks = np.frombuffer(payload[:10 * pixels], dtype=np.uint8)
|
| 155 |
+
quiet = blocks[:9 * pixels].reshape(9, pixels)
|
| 156 |
+
loud = blocks[9 * pixels:].reshape(pixels)
|
| 157 |
+
self.assertLess(float((quiet >= 128).mean()), 0.01)
|
| 158 |
+
self.assertGreater(int(loud.max()), 150)
|
| 159 |
+
|
| 160 |
+
def test_flat_episode_is_drawable(self):
|
| 161 |
+
with tempfile.TemporaryDirectory() as temp:
|
| 162 |
+
blank = np.zeros((250, 250, 3), dtype=np.int16)
|
| 163 |
+
path = _write_episode(
|
| 164 |
+
Path(temp) / "data" / "chunk-000" / "episode_000000.parquet",
|
| 165 |
+
[blank, blank])
|
| 166 |
+
lo, hi = _compute_tactile_display_range(
|
| 167 |
+
path, "observation.gripper_left_force_matrix")
|
| 168 |
+
self.assertLess(lo, hi)
|
| 169 |
+
|
| 170 |
+
|
| 171 |
if __name__ == "__main__":
|
| 172 |
unittest.main()
|
processor/tests/test_umi_slam_action.py
ADDED
|
@@ -0,0 +1,369 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import math
|
| 2 |
+
import tempfile
|
| 3 |
+
import unittest
|
| 4 |
+
from pathlib import Path
|
| 5 |
+
|
| 6 |
+
import numpy as np
|
| 7 |
+
import pyarrow as pa
|
| 8 |
+
import pyarrow.parquet as pq
|
| 9 |
+
|
| 10 |
+
from app.processing import JobContext, ModuleSkip
|
| 11 |
+
from app.processing.modules.umi_slam_action import UmiSlamActionModule
|
| 12 |
+
from app.umi_slam_action import (
|
| 13 |
+
derive_action,
|
| 14 |
+
matrix_to_rotvec,
|
| 15 |
+
quat_to_matrix,
|
| 16 |
+
)
|
| 17 |
+
|
| 18 |
+
|
| 19 |
+
def _yaw_quat(angle: float) -> list[float]:
|
| 20 |
+
"""绕 Z 轴 angle 弧度的四元数 (x, y, z, w)。"""
|
| 21 |
+
return [0.0, 0.0, math.sin(angle / 2.0), math.cos(angle / 2.0)]
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
def _pose(x, y, z, quat=None) -> list[float]:
|
| 25 |
+
return [float(x), float(y), float(z)] + (quat or [0.0, 0.0, 0.0, 1.0])
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
class RotationMathTests(unittest.TestCase):
|
| 29 |
+
def test_identity_and_quarter_turn(self):
|
| 30 |
+
# 单位四元数 → 单位阵 → 零旋转向量
|
| 31 |
+
self.assertTrue(np.allclose(matrix_to_rotvec(quat_to_matrix([0, 0, 0, 1])),
|
| 32 |
+
np.zeros(3), atol=1e-9))
|
| 33 |
+
|
| 34 |
+
# 绕 X 轴 90°
|
| 35 |
+
q = [math.sin(math.pi / 4), 0.0, 0.0, math.cos(math.pi / 4)]
|
| 36 |
+
self.assertTrue(np.allclose(matrix_to_rotvec(quat_to_matrix(q)),
|
| 37 |
+
[math.pi / 2, 0.0, 0.0], atol=1e-6))
|
| 38 |
+
|
| 39 |
+
def test_near_180_degrees_is_finite(self):
|
| 40 |
+
"""近 180° 时 sin(角)→0,标准轴公式会除零 —— 必须走退化分支。"""
|
| 41 |
+
for angle in (math.pi, math.pi - 1e-7, math.pi - 1e-4):
|
| 42 |
+
q = [0.0, 0.0, math.sin(angle / 2.0), math.cos(angle / 2.0)]
|
| 43 |
+
rotvec = matrix_to_rotvec(quat_to_matrix(q))
|
| 44 |
+
self.assertTrue(np.all(np.isfinite(rotvec)),
|
| 45 |
+
f"angle={angle} produced non-finite {rotvec}")
|
| 46 |
+
self.assertAlmostEqual(float(np.linalg.norm(rotvec)), angle, places=5)
|
| 47 |
+
|
| 48 |
+
def test_zero_quaternion_returns_none(self):
|
| 49 |
+
self.assertIsNone(quat_to_matrix([0.0, 0.0, 0.0, 0.0]))
|
| 50 |
+
|
| 51 |
+
|
| 52 |
+
def _traj(poses, dt=1/30, offset=1000.0):
|
| 53 |
+
"""把逐帧位姿包成 slam_trajectory 的格子。
|
| 54 |
+
|
| 55 |
+
每格是 ``[slam_t, x, y, z, qx, qy, qz, qw]`` 的整数倍;slam 时间戳与
|
| 56 |
+
数据时间戳差一个常数偏移(采集端时钟独立)。
|
| 57 |
+
"""
|
| 58 |
+
return [[ (i * dt + offset) ] + list(pose) for i, pose in enumerate(poses)]
|
| 59 |
+
|
| 60 |
+
|
| 61 |
+
def _stamps(n, dt=1/30):
|
| 62 |
+
return [i * dt for i in range(n)]
|
| 63 |
+
|
| 64 |
+
|
| 65 |
+
class DeriveActionTests(unittest.TestCase):
|
| 66 |
+
def test_pure_translation_matches_analytic_delta(self):
|
| 67 |
+
"""沿 +X 每帧 10mm、姿态恒为身份 → Δp 应为 [0.01, 0, 0]。"""
|
| 68 |
+
poses = [_pose(0.01 * i, 0, 0) for i in range(6)]
|
| 69 |
+
out = derive_action(_traj(poses), _stamps(6), [[0.0, 0, 0]] * 6, 6)
|
| 70 |
+
self.assertIsNotNone(out)
|
| 71 |
+
for frame in range(5):
|
| 72 |
+
self.assertTrue(np.allclose(out[frame]["action"][:3], [0.01, 0, 0],
|
| 73 |
+
atol=1e-9))
|
| 74 |
+
self.assertTrue(np.allclose(out[frame]["action"][3:6], [0, 0, 0],
|
| 75 |
+
atol=1e-9))
|
| 76 |
+
|
| 77 |
+
def test_local_frame_delta_follows_current_orientation(self):
|
| 78 |
+
"""世界系位移须换算到**当前**夹爪局部系,而不是原样透传。"""
|
| 79 |
+
step = 0.02
|
| 80 |
+
quat = [0.0, 0.0, math.sin(-math.pi / 4), math.cos(-math.pi / 4)]
|
| 81 |
+
poses = [_pose(0.0, -step * frame, 0.0, quat) for frame in range(5)]
|
| 82 |
+
out = derive_action(_traj(poses), _stamps(5), None, 5)
|
| 83 |
+
self.assertTrue(np.allclose(out[0]["action"][:3], [step, 0, 0],
|
| 84 |
+
atol=1e-6))
|
| 85 |
+
|
| 86 |
+
def test_world_delta_is_not_passed_through_verbatim(self):
|
| 87 |
+
step = 0.02
|
| 88 |
+
quat = [0.0, 0.0, math.sin(-math.pi / 4), math.cos(-math.pi / 4)]
|
| 89 |
+
poses = [_pose(0.0, -step * frame, 0.0, quat) for frame in range(3)]
|
| 90 |
+
out = derive_action(_traj(poses), _stamps(3), None, 3)
|
| 91 |
+
self.assertFalse(np.allclose(out[0]["action"][:3], [0, -step, 0],
|
| 92 |
+
atol=1e-6))
|
| 93 |
+
|
| 94 |
+
def test_pure_rotation_matches_analytic_delta(self):
|
| 95 |
+
angle = 0.1
|
| 96 |
+
poses = [_pose(0, 0, 0, _yaw_quat(angle * i)) for i in range(5)]
|
| 97 |
+
out = derive_action(_traj(poses), _stamps(5), None, 5)
|
| 98 |
+
self.assertTrue(np.allclose(out[0]["action"][3:6], [0, 0, angle],
|
| 99 |
+
atol=1e-6))
|
| 100 |
+
|
| 101 |
+
def test_gripper_dimension_tracks_open_fraction(self):
|
| 102 |
+
poses = [_pose(0.01 * i, 0, 0) for i in range(4)]
|
| 103 |
+
gripper = [[0.0, 0, 0], [50.0, 0, 0], [100.0, 0, 0], [100.0, 0, 0]]
|
| 104 |
+
out = derive_action(_traj(poses), _stamps(4), gripper, 4)
|
| 105 |
+
self.assertAlmostEqual(out[0]["action"][6], 0.5, places=6)
|
| 106 |
+
self.assertAlmostEqual(out[1]["action"][6], 1.0, places=6)
|
| 107 |
+
|
| 108 |
+
def test_gripper_is_clipped_to_unit_range(self):
|
| 109 |
+
poses = [_pose(0.01 * i, 0, 0) for i in range(3)]
|
| 110 |
+
out = derive_action(_traj(poses), _stamps(3), [[500.0, 0, 0]] * 3, 3)
|
| 111 |
+
self.assertLessEqual(out[0]["action"][6], 1.0)
|
| 112 |
+
self.assertGreaterEqual(out[0]["action"][6], 0.0)
|
| 113 |
+
|
| 114 |
+
def test_last_frame_reuses_previous_delta(self):
|
| 115 |
+
poses = [_pose(0.01 * i, 0, 0) for i in range(5)]
|
| 116 |
+
out = derive_action(_traj(poses), _stamps(5), None, 5)
|
| 117 |
+
self.assertIn(4, out)
|
| 118 |
+
self.assertTrue(np.allclose(out[4]["action"][:3], out[3]["action"][:3],
|
| 119 |
+
atol=1e-9))
|
| 120 |
+
|
| 121 |
+
def test_action_dimension_is_seven(self):
|
| 122 |
+
poses = [_pose(0.01 * i, 0, 0) for i in range(5)]
|
| 123 |
+
out = derive_action(_traj(poses), _stamps(5), None, 5)
|
| 124 |
+
self.assertEqual(len(out[0]["action"]), 7)
|
| 125 |
+
|
| 126 |
+
def test_all_finite_on_mixed_input(self):
|
| 127 |
+
poses = [_pose(0.01 * i, 0, 0) for i in range(6)]
|
| 128 |
+
traj = _traj(poses)
|
| 129 |
+
traj[3] = [] # 该格没有样本
|
| 130 |
+
out = derive_action(traj, _stamps(6), None, 6)
|
| 131 |
+
values = np.array([out[i]["action"] for i in sorted(out)])
|
| 132 |
+
self.assertTrue(np.all(np.isfinite(values)))
|
| 133 |
+
|
| 134 |
+
def test_empty_and_degenerate_inputs(self):
|
| 135 |
+
self.assertIsNone(derive_action([], [], None, 0))
|
| 136 |
+
self.assertIsNone(derive_action([[]] * 3, _stamps(3), None, 3))
|
| 137 |
+
self.assertIsNone(derive_action(_traj([_pose(0, 0, 0)]), _stamps(1),
|
| 138 |
+
None, 1))
|
| 139 |
+
|
| 140 |
+
def test_sample_from_neighbour_cell_recovers_empty_frame(self):
|
| 141 |
+
"""空帧若能从相邻格子的样本里找到位姿,就不该被当作丢帧。
|
| 142 |
+
|
| 143 |
+
这是换用 slam_trajectory 的核心收益:采集端 50Hz、数据帧 30fps,
|
| 144 |
+
两帧之间到达的样本落在其中一格,所以「本帧格子为空」不代表那一刻
|
| 145 |
+
没有位姿。
|
| 146 |
+
"""
|
| 147 |
+
# 真实形态:绝大多数帧的格子只装属于自己的那一个样本,个别格子会
|
| 148 |
+
# 多装一个「属于下一帧窗口」的样本。帧 5 自己的格子为空,但帧 4 的
|
| 149 |
+
# 格子里存着它 —— 这正是采集端 50Hz 快于数据 30fps 的结果。
|
| 150 |
+
dt = 1 / 30
|
| 151 |
+
n = 10
|
| 152 |
+
traj = []
|
| 153 |
+
for i in range(n):
|
| 154 |
+
if i == 5:
|
| 155 |
+
traj.append([]) # 本帧格子空
|
| 156 |
+
continue
|
| 157 |
+
cell = [1000.0 + i * dt] + _pose(0.01 * i, 0, 0)
|
| 158 |
+
if i == 4:
|
| 159 |
+
# 帧 5 的样本溢到了上一个格子里(它比本帧时刻晚一点)
|
| 160 |
+
cell += [1000.0 + 5 * dt] + _pose(0.05, 0, 0)
|
| 161 |
+
traj.append(cell)
|
| 162 |
+
|
| 163 |
+
out = derive_action(traj, _stamps(n), None, n)
|
| 164 |
+
self.assertIsNotNone(out)
|
| 165 |
+
self.assertIn(5, out, "空帧应能从相邻格子的样本恢复")
|
| 166 |
+
# 帧 5 用的应是那个真实样本(0.05),而不是帧 4/6 的插值中点
|
| 167 |
+
self.assertAlmostEqual(out[5]["observation.state"][0], 0.05, places=5)
|
| 168 |
+
|
| 169 |
+
def test_observation_state_is_relative_to_episode_start(self):
|
| 170 |
+
poses = [_pose(1.0 + 0.01 * i, 2.0, 3.0) for i in range(5)]
|
| 171 |
+
out = derive_action(_traj(poses), _stamps(5), [[50.0, 0, 0]] * 5, 5)
|
| 172 |
+
# 位置分量是相对本集首帧的偏移 → 首帧为 0
|
| 173 |
+
self.assertTrue(np.allclose(out[0]["observation.state"][:3],
|
| 174 |
+
[0.0, 0.0, 0.0], atol=1e-9))
|
| 175 |
+
self.assertAlmostEqual(out[3]["observation.state"][0], 0.03, places=6)
|
| 176 |
+
self.assertAlmostEqual(out[3]["observation.state"][3], 0.5, places=6)
|
| 177 |
+
|
| 178 |
+
|
| 179 |
+
class UmiSlamActionModuleTests(unittest.TestCase):
|
| 180 |
+
def _context(self, root: Path) -> JobContext:
|
| 181 |
+
return JobContext(
|
| 182 |
+
node_id="umi_slam_action",
|
| 183 |
+
node_type="umi_slam_action",
|
| 184 |
+
config={},
|
| 185 |
+
job={},
|
| 186 |
+
input_root=root,
|
| 187 |
+
output_root=root / "outputs",
|
| 188 |
+
incoming={},
|
| 189 |
+
progress=lambda _value: None,
|
| 190 |
+
node_states={},
|
| 191 |
+
)
|
| 192 |
+
|
| 193 |
+
def _write_episode(self, root: Path, poses, gripper=None,
|
| 194 |
+
extra_columns: dict | None = None) -> Path:
|
| 195 |
+
"""写入一份 canonical parquet。
|
| 196 |
+
|
| 197 |
+
位姿写成 ``observation.slam_trajectory`` 的格子格式(每格 8 的整数
|
| 198 |
+
倍)并配 ``timestamp`` 列 —— 模块只从这个缓冲重建位姿。
|
| 199 |
+
"""
|
| 200 |
+
(root / "data" / "chunk-000").mkdir(parents=True, exist_ok=True)
|
| 201 |
+
stamps = _stamps(len(poses))
|
| 202 |
+
columns = {
|
| 203 |
+
"frame_index": list(range(len(poses))),
|
| 204 |
+
"timestamp": stamps,
|
| 205 |
+
"observation.slam_trajectory": _traj(poses),
|
| 206 |
+
}
|
| 207 |
+
if gripper is not None:
|
| 208 |
+
columns["observation.gripper_state"] = gripper
|
| 209 |
+
for name, values in (extra_columns or {}).items():
|
| 210 |
+
columns[name] = values
|
| 211 |
+
path = root / "data" / "chunk-000" / "episode_000000.parquet"
|
| 212 |
+
pq.write_table(pa.table(columns), path)
|
| 213 |
+
return path
|
| 214 |
+
|
| 215 |
+
def test_derives_varying_action_and_reports_metadata(self):
|
| 216 |
+
with tempfile.TemporaryDirectory() as temp:
|
| 217 |
+
root = Path(temp)
|
| 218 |
+
# 用非等距轨迹:等距平移的每帧增量恒等,会被导出端的恒定列
|
| 219 |
+
# 过滤剔除,验证不出"动作确实在变化"这一点。
|
| 220 |
+
poses = [_pose(0.01 * i + 0.004 * (i % 3), 0, 0) for i in range(20)]
|
| 221 |
+
self._write_episode(root, poses, [[50.0, 0, 0]] * 20)
|
| 222 |
+
|
| 223 |
+
outputs = UmiSlamActionModule().run(self._context(root))
|
| 224 |
+
|
| 225 |
+
self.assertIn("action", outputs)
|
| 226 |
+
ref = outputs["action"]
|
| 227 |
+
self.assertEqual(ref.metadata["shape"], [7])
|
| 228 |
+
self.assertEqual(ref.metadata["frames_total"], 20)
|
| 229 |
+
self.assertEqual(len(ref.metadata["names"]), 7)
|
| 230 |
+
|
| 231 |
+
import pandas as pd
|
| 232 |
+
frame = pd.read_parquet(root / "outputs" / "umi_slam_action" / "umi_slam_action.parquet")
|
| 233 |
+
values = np.array(frame["action"].tolist())
|
| 234 |
+
self.assertEqual(values.shape[1], 7)
|
| 235 |
+
# 必须是真正变化的动作,否则导出的恒定列过滤会把它删掉
|
| 236 |
+
self.assertGreater(len(np.unique(values[:, 0])), 1)
|
| 237 |
+
|
| 238 |
+
def test_output_is_float32(self):
|
| 239 |
+
"""LeRobot 的 action 声明是 float32;写 double 会造成声明与 schema
|
| 240 |
+
不一致(官方加载器按声明 cast)。"""
|
| 241 |
+
import pyarrow.parquet as pq
|
| 242 |
+
with tempfile.TemporaryDirectory() as temp:
|
| 243 |
+
root = Path(temp)
|
| 244 |
+
poses = [_pose(0.01 * i, 0, 0) for i in range(10)]
|
| 245 |
+
self._write_episode(root, poses, [[50.0, 0, 0]] * 10)
|
| 246 |
+
UmiSlamActionModule().run(self._context(root))
|
| 247 |
+
|
| 248 |
+
path = root / "outputs" / "umi_slam_action" / "umi_slam_action.parquet"
|
| 249 |
+
field_type = pq.read_schema(path).field("action").type
|
| 250 |
+
self.assertIn("float", str(field_type))
|
| 251 |
+
self.assertNotIn("double", str(field_type))
|
| 252 |
+
|
| 253 |
+
def test_skips_batch_without_slam_pose(self):
|
| 254 |
+
"""非 UMI 批次(如 D435 手套)必须跳过,不能写坏数据。"""
|
| 255 |
+
with tempfile.TemporaryDirectory() as temp:
|
| 256 |
+
root = Path(temp)
|
| 257 |
+
self._write_episode(root, [_pose(0, 0, 0)], None,
|
| 258 |
+
extra_columns={"observation.glove": [[1.0, 2.0]]})
|
| 259 |
+
# 覆盖掉 slam 列,模拟没有 SLAM 的设备
|
| 260 |
+
(root / "data" / "chunk-000" / "episode_000000.parquet").unlink()
|
| 261 |
+
pq.write_table(pa.table({
|
| 262 |
+
"frame_index": [0, 1],
|
| 263 |
+
"observation.glove": [[1.0, 2.0], [3.0, 4.0]],
|
| 264 |
+
}), root / "data" / "chunk-000" / "episode_000000.parquet")
|
| 265 |
+
|
| 266 |
+
with self.assertRaises(ModuleSkip):
|
| 267 |
+
UmiSlamActionModule().run(self._context(root))
|
| 268 |
+
|
| 269 |
+
def test_retries_transient_read_failure(self):
|
| 270 |
+
"""上传/处理并发时 parquet 可能瞬时不可读,不能一次失败就放弃。
|
| 271 |
+
|
| 272 |
+
实测 ep52/ep71 都因此被误判成「列不可读」跳过,几秒后同一文件完全
|
| 273 |
+
正常。这里注入两次失败,验证第三次成功读到时模块照常产出。
|
| 274 |
+
"""
|
| 275 |
+
from unittest.mock import patch
|
| 276 |
+
|
| 277 |
+
import pandas as pd
|
| 278 |
+
|
| 279 |
+
with tempfile.TemporaryDirectory() as temp:
|
| 280 |
+
root = Path(temp)
|
| 281 |
+
poses = [_pose(0.01 * i + 0.004 * (i % 3), 0, 0) for i in range(12)]
|
| 282 |
+
self._write_episode(root, poses, [[50.0, 0, 0]] * 12)
|
| 283 |
+
|
| 284 |
+
original = pd.read_parquet
|
| 285 |
+
state = {"calls": 0}
|
| 286 |
+
|
| 287 |
+
def flaky(*args, **kwargs):
|
| 288 |
+
state["calls"] += 1
|
| 289 |
+
if state["calls"] <= 2:
|
| 290 |
+
raise OSError("file is being written")
|
| 291 |
+
return original(*args, **kwargs)
|
| 292 |
+
|
| 293 |
+
with patch.object(pd, "read_parquet", flaky), \
|
| 294 |
+
patch("time.sleep", lambda _s: None):
|
| 295 |
+
outputs = UmiSlamActionModule().run(self._context(root))
|
| 296 |
+
|
| 297 |
+
self.assertGreaterEqual(state["calls"], 3, "应当重试到成功")
|
| 298 |
+
self.assertIn("action", outputs)
|
| 299 |
+
|
| 300 |
+
def test_skip_message_carries_the_real_exception(self):
|
| 301 |
+
"""笼统提示无法区分竞态与设备类型不符,必须带上真实异常。"""
|
| 302 |
+
from unittest.mock import patch
|
| 303 |
+
|
| 304 |
+
import pandas as pd
|
| 305 |
+
|
| 306 |
+
with tempfile.TemporaryDirectory() as temp:
|
| 307 |
+
root = Path(temp)
|
| 308 |
+
self._write_episode(root, [_pose(0.01 * i, 0, 0) for i in range(8)],
|
| 309 |
+
[[50.0, 0, 0]] * 8)
|
| 310 |
+
|
| 311 |
+
def always_fail(*args, **kwargs):
|
| 312 |
+
raise OSError("mount is stale")
|
| 313 |
+
|
| 314 |
+
with patch.object(pd, "read_parquet", always_fail), \
|
| 315 |
+
patch("time.sleep", lambda _s: None):
|
| 316 |
+
with self.assertRaises(ModuleSkip) as caught:
|
| 317 |
+
UmiSlamActionModule().run(self._context(root))
|
| 318 |
+
|
| 319 |
+
message = str(caught.exception)
|
| 320 |
+
self.assertIn("OSError", message)
|
| 321 |
+
self.assertIn("stale", message)
|
| 322 |
+
|
| 323 |
+
def test_skips_when_all_poses_are_zero(self):
|
| 324 |
+
with tempfile.TemporaryDirectory() as temp:
|
| 325 |
+
root = Path(temp)
|
| 326 |
+
self._write_episode(root, [[0.0] * 7] * 10, [[0.0, 0, 0]] * 10)
|
| 327 |
+
with self.assertRaises(ModuleSkip):
|
| 328 |
+
UmiSlamActionModule().run(self._context(root))
|
| 329 |
+
|
| 330 |
+
def test_skips_when_no_parquet_present(self):
|
| 331 |
+
with tempfile.TemporaryDirectory() as temp:
|
| 332 |
+
root = Path(temp)
|
| 333 |
+
(root / "data").mkdir(parents=True, exist_ok=True)
|
| 334 |
+
with self.assertRaises(ModuleSkip):
|
| 335 |
+
UmiSlamActionModule().run(self._context(root))
|
| 336 |
+
|
| 337 |
+
def test_ignores_non_parquet_upstream_ref(self):
|
| 338 |
+
"""上游 ``slam_trajectory`` 端口可能挂着一个非 parquet 产物。
|
| 339 |
+
|
| 340 |
+
worker 组装 incoming 时,若边的 sourceHandle 对不上上游任何输出,
|
| 341 |
+
会把该节点的**全部输出平铺**进来 —— 视频 ref 也会落到这个键上。
|
| 342 |
+
实测因此把 mp4 当 parquet 读,报 magic bytes not found。模块必须
|
| 343 |
+
验证产物类型,并回退到扫描批次里的真 parquet。
|
| 344 |
+
"""
|
| 345 |
+
from app.processing import ArtifactRef
|
| 346 |
+
|
| 347 |
+
with tempfile.TemporaryDirectory() as temp:
|
| 348 |
+
root = Path(temp)
|
| 349 |
+
poses = [_pose(0.01 * i + 0.004 * (i % 3), 0, 0) for i in range(12)]
|
| 350 |
+
self._write_episode(root, poses, [[50.0, 0, 0]] * 12)
|
| 351 |
+
|
| 352 |
+
# 上游把视频文件挂到了 slam_trajectory 这个端口上
|
| 353 |
+
video = root / "videos" / "observation.images.gripper_rgb" / "chunk-000"
|
| 354 |
+
video.mkdir(parents=True, exist_ok=True)
|
| 355 |
+
clip = video / "episode_000000.mp4"
|
| 356 |
+
clip.write_bytes(b"\x00\x00\x00\x1cftypisom" + b"\x00" * 64)
|
| 357 |
+
|
| 358 |
+
ctx = self._context(root)
|
| 359 |
+
ctx.incoming = {"slam_trajectory": ArtifactRef(
|
| 360 |
+
kind="rgb_video", path=str(clip.relative_to(root)),
|
| 361 |
+
source_key="gripper_rgb")}
|
| 362 |
+
|
| 363 |
+
outputs = UmiSlamActionModule().run(ctx)
|
| 364 |
+
|
| 365 |
+
self.assertIn("action", outputs, "应回退到真正的 parquet")
|
| 366 |
+
|
| 367 |
+
|
| 368 |
+
if __name__ == "__main__":
|
| 369 |
+
unittest.main()
|
processor/web/static/js/player.js
CHANGED
|
@@ -1,5 +1,5 @@
|
|
| 1 |
/* Plyr video player management */
|
| 2 |
-
console.log('[player.js] build
|
| 3 |
|
| 4 |
const players = {}; // {camera_name: Plyr instance}
|
| 5 |
let currentEpisodeId = null;
|
|
@@ -1219,7 +1219,15 @@ function _syncWorkspacePlayers() {
|
|
| 1219 |
if (typeof startHeatmapSync === 'function') startHeatmapSync(active);
|
| 1220 |
if (masterPlaying) {
|
| 1221 |
Object.values(players).forEach(player => {
|
| 1222 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1223 |
});
|
| 1224 |
}
|
| 1225 |
}
|
|
@@ -1781,7 +1789,13 @@ function _createTactileWebGLRenderer(canvas) {
|
|
| 1781 |
canvas.width = width;
|
| 1782 |
canvas.height = height;
|
| 1783 |
}
|
| 1784 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1785 |
gl.clearColor(0.0039, 0.0039, 0.502, 1.0);
|
| 1786 |
gl.clear(gl.COLOR_BUFFER_BIT);
|
| 1787 |
if (!this.field) return;
|
|
@@ -1815,77 +1829,19 @@ function _createTactileWebGLRenderer(canvas) {
|
|
| 1815 |
};
|
| 1816 |
}
|
| 1817 |
|
| 1818 |
-
//
|
| 1819 |
-
//
|
| 1820 |
-
//
|
| 1821 |
-
//
|
| 1822 |
-
|
| 1823 |
-
|
| 1824 |
-
|
| 1825 |
-
|
| 1826 |
-
|
| 1827 |
-
|
| 1828 |
-
|
| 1829 |
-
|
| 1830 |
-
|
| 1831 |
-
const limit = Math.max(0, (1 - 1 / renderer.zoom) / 2);
|
| 1832 |
-
renderer.pan[0] = Math.max(-limit, Math.min(limit, renderer.pan[0]));
|
| 1833 |
-
renderer.pan[1] = Math.max(-limit, Math.min(limit, renderer.pan[1]));
|
| 1834 |
-
}
|
| 1835 |
-
|
| 1836 |
-
function initTactileView(entry) {
|
| 1837 |
-
const canvas = entry?.canvas;
|
| 1838 |
-
const renderer = entry?.renderer;
|
| 1839 |
-
if (!canvas || !renderer) return;
|
| 1840 |
-
const maxZoom = 12;
|
| 1841 |
-
canvas.addEventListener('wheel', event => {
|
| 1842 |
-
event.preventDefault();
|
| 1843 |
-
const [u, v] = _tactileScreenUv(canvas, event.clientX, event.clientY);
|
| 1844 |
-
const before = 1 / renderer.zoom;
|
| 1845 |
-
renderer.zoom = Math.max(1, Math.min(maxZoom,
|
| 1846 |
-
renderer.zoom * (event.deltaY < 0 ? 1.2 : 1 / 1.2)));
|
| 1847 |
-
const delta = before - 1 / renderer.zoom;
|
| 1848 |
-
renderer.pan[0] += (u - 0.5) * delta;
|
| 1849 |
-
renderer.pan[1] += (v - 0.5) * delta;
|
| 1850 |
-
_clampTactileView(renderer);
|
| 1851 |
-
canvas.style.cursor = renderer.zoom > 1 ? 'grab' : 'default';
|
| 1852 |
-
renderer.render();
|
| 1853 |
-
}, { passive: false });
|
| 1854 |
-
canvas.addEventListener('pointerdown', event => {
|
| 1855 |
-
if (event.button !== 0 || renderer.zoom <= 1) return;
|
| 1856 |
-
entry._panDrag = { x: event.clientX, y: event.clientY };
|
| 1857 |
-
canvas.style.cursor = 'grabbing';
|
| 1858 |
-
if (canvas.setPointerCapture) canvas.setPointerCapture(event.pointerId);
|
| 1859 |
-
event.preventDefault();
|
| 1860 |
-
});
|
| 1861 |
-
canvas.addEventListener('pointermove', event => {
|
| 1862 |
-
if (!entry._panDrag) return;
|
| 1863 |
-
const rect = canvas.getBoundingClientRect();
|
| 1864 |
-
renderer.pan[0] -= (event.clientX - entry._panDrag.x)
|
| 1865 |
-
/ Math.max(1, rect.width) / renderer.zoom;
|
| 1866 |
-
renderer.pan[1] += (event.clientY - entry._panDrag.y)
|
| 1867 |
-
/ Math.max(1, rect.height) / renderer.zoom;
|
| 1868 |
-
entry._panDrag = { x: event.clientX, y: event.clientY };
|
| 1869 |
-
_clampTactileView(renderer);
|
| 1870 |
-
renderer.render();
|
| 1871 |
-
});
|
| 1872 |
-
const endPan = event => {
|
| 1873 |
-
if (!entry._panDrag) return;
|
| 1874 |
-
entry._panDrag = null;
|
| 1875 |
-
canvas.style.cursor = renderer.zoom > 1 ? 'grab' : 'default';
|
| 1876 |
-
if (event && canvas.hasPointerCapture
|
| 1877 |
-
&& canvas.hasPointerCapture(event.pointerId)) {
|
| 1878 |
-
canvas.releasePointerCapture(event.pointerId);
|
| 1879 |
-
}
|
| 1880 |
-
};
|
| 1881 |
-
canvas.addEventListener('pointerup', endPan);
|
| 1882 |
-
canvas.addEventListener('pointercancel', endPan);
|
| 1883 |
-
canvas.addEventListener('dblclick', () => {
|
| 1884 |
-
renderer.zoom = 1;
|
| 1885 |
-
renderer.pan = [0, 0];
|
| 1886 |
-
canvas.style.cursor = 'default';
|
| 1887 |
-
renderer.render();
|
| 1888 |
-
});
|
| 1889 |
}
|
| 1890 |
|
| 1891 |
// One chunk = 96 frames of 250x250 R8 = 6 MB per request. The old code issued
|
|
@@ -1918,7 +1874,7 @@ function _fetchTactileChunk(entry, chunkStart) {
|
|
| 1918 |
const signal = getMediaLoadSignal();
|
| 1919 |
const url = `/api/v1/video/${currentEpisodeId}/tactile-matrix-range`
|
| 1920 |
+ `?side=${entry.source.side}&start=${chunkStart}`
|
| 1921 |
-
+ `&limit=${TACTILE_CHUNK_FRAMES}&v=
|
| 1922 |
const options = { credentials: 'same-origin', cache: 'force-cache' };
|
| 1923 |
if (signal) options.signal = signal;
|
| 1924 |
fetch(url, options).then(response => {
|
|
@@ -2057,8 +2013,8 @@ async function mountGroupedSource(source, tile) {
|
|
| 2057 |
curveCanvas, loadedFrame: -1, pendingFrame: null,
|
| 2058 |
chunks: new Map(), chunkInflight: new Set(),
|
| 2059 |
frameCount: Number(tactileMatrixData?.frame_count) || 0 };
|
| 2060 |
-
|
| 2061 |
-
canvas.title = '
|
| 2062 |
currentTactileTiles.push(entry);
|
| 2063 |
const frame = typeof currentFrameTarget === 'number' ? currentFrameTarget : 0;
|
| 2064 |
_updateTactileForceTile(entry, frame);
|
|
@@ -2625,6 +2581,12 @@ function renderGroupedWorkspace() {
|
|
| 2625 |
_restorePlayersToCurrentFrame();
|
| 2626 |
const active = getActivePlayer();
|
| 2627 |
if (!active) { hideVideoLoading(); return; }
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2628 |
_syncWorkspacePlayers();
|
| 2629 |
refreshPlayButton(false);
|
| 2630 |
try {
|
|
@@ -3324,6 +3286,10 @@ function playAll() {
|
|
| 3324 |
|
| 3325 |
|
| 3326 |
function pauseAll() {
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3327 |
Object.values(players).forEach(p => {
|
| 3328 |
try { p.pause(); } catch(e) {}
|
| 3329 |
});
|
|
@@ -4215,10 +4181,13 @@ function _slamSmoothRows() {
|
|
| 4215 |
// world frame: +Z is up (measured on real data - a physical turn of the
|
| 4216 |
// gripper rotates about world +Z, and the gripper's own up axis sits within
|
| 4217 |
// ~12 deg of it for the whole episode). The browser camera uses a right-handed
|
| 4218 |
-
// frame with Y up, so display
|
| 4219 |
-
//
|
|
|
|
|
|
|
|
|
|
| 4220 |
function _slamDisplayPoint(sourcePoint) {
|
| 4221 |
-
return [Number(sourcePoint[0]), Number(sourcePoint[2]), -Number(sourcePoint[1])];
|
| 4222 |
}
|
| 4223 |
|
| 4224 |
function _slamSourcePoint(row) {
|
|
@@ -4358,10 +4327,12 @@ function renderSlamTile(entry, frameOverride) {
|
|
| 4358 |
|
| 4359 |
// Small axis marker at the fitted area: the recorded world axes, drawn in
|
| 4360 |
// the display frame that _slamDisplayPoint produces (world +Z is up).
|
|
|
|
|
|
|
| 4361 |
const axisOrigin = [fit.center[0], fit.gridY, fit.center[2]];
|
| 4362 |
const axisTips = [
|
| 4363 |
// World axes: X and Y horizontal, Z up.
|
| 4364 |
-
[[axisOrigin[0]
|
| 4365 |
[[axisOrigin[0], axisOrigin[1], axisOrigin[2] - 0.2], 'rgb(80,220,130)', 'Y'],
|
| 4366 |
[[axisOrigin[0], axisOrigin[1] + 0.2, axisOrigin[2]], 'rgb(90,150,255)', 'Z'],
|
| 4367 |
];
|
|
@@ -4423,10 +4394,13 @@ function renderSlamTile(entry, frameOverride) {
|
|
| 4423 |
});
|
| 4424 |
|
| 4425 |
// Simplified UMI gripper model. The recorded pose frame is the S80M
|
| 4426 |
-
// camera frame; measured on
|
| 4427 |
-
//
|
| 4428 |
-
//
|
| 4429 |
-
//
|
|
|
|
|
|
|
|
|
|
| 4430 |
// plates separate along local X, and each plate is 0.20m long
|
| 4431 |
// (forward) by 0.05m tall (up) with a 0.12m maximum opening.
|
| 4432 |
const grip = _slamGripperState(currentRow);
|
|
@@ -4449,18 +4423,18 @@ function renderSlamTile(entry, frameOverride) {
|
|
| 4449 |
const jawFill = grip.gripped
|
| 4450 |
? 'rgba(248,90,90,0.16)' : 'rgba(88,168,232,0.12)';
|
| 4451 |
const relativeCorners = [
|
| 4452 |
-
[0, -
|
| 4453 |
-
[0,
|
| 4454 |
-
[0,
|
| 4455 |
-
[0,
|
| 4456 |
];
|
| 4457 |
|
| 4458 |
for (const sign of [1, -1]) {
|
| 4459 |
const centerX = sign * halfGap;
|
| 4460 |
const corners = relativeCorners.map(corner => project(toDisplay([
|
| 4461 |
centerX + corner[0],
|
| 4462 |
-
corner[1],
|
| 4463 |
-
|
| 4464 |
])));
|
| 4465 |
if (corners.every(item => Number.isFinite(item[0]) && Number.isFinite(item[1]))) {
|
| 4466 |
ctx.fillStyle = jawFill;
|
|
@@ -4474,21 +4448,21 @@ function renderSlamTile(entry, frameOverride) {
|
|
| 4474 |
ctx.stroke();
|
| 4475 |
}
|
| 4476 |
line(
|
| 4477 |
-
toDisplay([centerX,
|
| 4478 |
-
toDisplay([centerX,
|
| 4479 |
jawColor, 1);
|
| 4480 |
}
|
| 4481 |
|
| 4482 |
// Rear crossbar plus the two supports that tie each plate back to
|
| 4483 |
// the body, so the opening stays readable when it is small.
|
| 4484 |
-
line(toDisplay([-maxHalfGap,
|
| 4485 |
-
toDisplay([maxHalfGap,
|
| 4486 |
'rgba(148,163,184,0.8)', 1.5);
|
| 4487 |
for (const sign of [1, -1]) {
|
| 4488 |
-
for (const
|
| 4489 |
line(
|
| 4490 |
-
toDisplay([sign * halfGap,
|
| 4491 |
-
toDisplay([0,
|
| 4492 |
'rgba(148,163,184,0.8)', 1);
|
| 4493 |
}
|
| 4494 |
}
|
|
|
|
| 1 |
/* Plyr video player management */
|
| 2 |
+
console.log('[player.js] build 20260911-tactile-episode-color-scale');
|
| 3 |
|
| 4 |
const players = {}; // {camera_name: Plyr instance}
|
| 5 |
let currentEpisodeId = null;
|
|
|
|
| 1219 |
if (typeof startHeatmapSync === 'function') startHeatmapSync(active);
|
| 1220 |
if (masterPlaying) {
|
| 1221 |
Object.values(players).forEach(player => {
|
| 1222 |
+
// play() returns a promise: a try/catch cannot see its rejection,
|
| 1223 |
+
// and the load chain right after this call runs pauseAll(), which
|
| 1224 |
+
// aborts the pending play → uncaught AbortError in the console.
|
| 1225 |
+
try {
|
| 1226 |
+
const result = player.play();
|
| 1227 |
+
if (result && typeof result.catch === 'function') {
|
| 1228 |
+
result.catch(() => {});
|
| 1229 |
+
}
|
| 1230 |
+
} catch (e) {}
|
| 1231 |
});
|
| 1232 |
}
|
| 1233 |
}
|
|
|
|
| 1789 |
canvas.width = width;
|
| 1790 |
canvas.height = height;
|
| 1791 |
}
|
| 1792 |
+
// 传感器是 250x250 的方格阵列,必须等比显示 —— 铺满 canvas 会按
|
| 1793 |
+
// tile 的宽高比横向拉伸(实测宽窗约 2.9:1),格子就不再是正方形,
|
| 1794 |
+
// 接触位置的形状判断会失真。取短边做居中正方形 viewport,多出来的
|
| 1795 |
+
// 两侧留背景色(gl.clear 不受 viewport 限制,会整块清成底色)。
|
| 1796 |
+
const side = Math.min(width, height);
|
| 1797 |
+
gl.viewport(Math.round((width - side) / 2),
|
| 1798 |
+
Math.round((height - side) / 2), side, side);
|
| 1799 |
gl.clearColor(0.0039, 0.0039, 0.502, 1.0);
|
| 1800 |
gl.clear(gl.COLOR_BUFFER_BIT);
|
| 1801 |
if (!this.field) return;
|
|
|
|
| 1829 |
};
|
| 1830 |
}
|
| 1831 |
|
| 1832 |
+
// The tactile force matrix tile is display-only. Zoom/pan were removed on
|
| 1833 |
+
// purpose: the review page is read-only evidence, and a stray wheel or drag
|
| 1834 |
+
// could leave the matrix zoomed into a few cells and look like a sensor
|
| 1835 |
+
// defect. The whole 250x250 field stays fixed, one texture texel per cell.
|
| 1836 |
+
// The renderer keeps supporting zoom/pan (uniforms u_zoom/u_pan) so the
|
| 1837 |
+
// acquisition-side demo viewer and any future opt-in can reuse it; here the
|
| 1838 |
+
// view is pinned to identity.
|
| 1839 |
+
function _resetTactileView(renderer) {
|
| 1840 |
+
if (!renderer) return;
|
| 1841 |
+
renderer.zoom = 1;
|
| 1842 |
+
renderer.pan = [0, 0];
|
| 1843 |
+
if (renderer.canvas) renderer.canvas.style.cursor = 'default';
|
| 1844 |
+
renderer.render();
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1845 |
}
|
| 1846 |
|
| 1847 |
// One chunk = 96 frames of 250x250 R8 = 6 MB per request. The old code issued
|
|
|
|
| 1874 |
const signal = getMediaLoadSignal();
|
| 1875 |
const url = `/api/v1/video/${currentEpisodeId}/tactile-matrix-range`
|
| 1876 |
+ `?side=${entry.source.side}&start=${chunkStart}`
|
| 1877 |
+
+ `&limit=${TACTILE_CHUNK_FRAMES}&v=20260911-tactile-episode-color-scale`;
|
| 1878 |
const options = { credentials: 'same-origin', cache: 'force-cache' };
|
| 1879 |
if (signal) options.signal = signal;
|
| 1880 |
fetch(url, options).then(response => {
|
|
|
|
| 2013 |
curveCanvas, loadedFrame: -1, pendingFrame: null,
|
| 2014 |
chunks: new Map(), chunkInflight: new Set(),
|
| 2015 |
frameCount: Number(tactileMatrixData?.frame_count) || 0 };
|
| 2016 |
+
_resetTactileView(renderer);
|
| 2017 |
+
canvas.title = '接触力场(固定全幅,不可缩放)';
|
| 2018 |
currentTactileTiles.push(entry);
|
| 2019 |
const frame = typeof currentFrameTarget === 'number' ? currentFrameTarget : 0;
|
| 2020 |
_updateTactileForceTile(entry, frame);
|
|
|
|
| 2581 |
_restorePlayersToCurrentFrame();
|
| 2582 |
const active = getActivePlayer();
|
| 2583 |
if (!active) { hideVideoLoading(); return; }
|
| 2584 |
+
// A freshly mounted episode always starts paused. Clear the state and
|
| 2585 |
+
// sync before wiring the players: _syncWorkspacePlayers() re-issues
|
| 2586 |
+
// play() for a workspace still marked as playing, and the pauseAll()
|
| 2587 |
+
// further down then aborts it — a play/pause race that fires an
|
| 2588 |
+
// uncaught AbortError and leaves the transport button flapping.
|
| 2589 |
+
masterPlaying = false;
|
| 2590 |
_syncWorkspacePlayers();
|
| 2591 |
refreshPlayButton(false);
|
| 2592 |
try {
|
|
|
|
| 3286 |
|
| 3287 |
|
| 3288 |
function pauseAll() {
|
| 3289 |
+
// Keep the transport state consistent with reality: callers that pause
|
| 3290 |
+
// without flipping the flag left the button reading "Pause" while every
|
| 3291 |
+
// player was stopped, so the next click replayed instead of resuming.
|
| 3292 |
+
masterPlaying = false;
|
| 3293 |
Object.values(players).forEach(p => {
|
| 3294 |
try { p.pause(); } catch(e) {}
|
| 3295 |
});
|
|
|
|
| 4181 |
// world frame: +Z is up (measured on real data - a physical turn of the
|
| 4182 |
// gripper rotates about world +Z, and the gripper's own up axis sits within
|
| 4183 |
// ~12 deg of it for the whole episode). The browser camera uses a right-handed
|
| 4184 |
+
// frame with Y up, so the display mapping puts world up on screen up.
|
| 4185 |
+
//
|
| 4186 |
+
// 源 +X 指向设备的**左侧**(S80M R_corr 之后源系是"左、后、上"),
|
| 4187 |
+
// 直接透传会让整个空间视图相对真实动作左右镜像:设备往左绕圈,
|
| 4188 |
+
// 画面里往右走。取反 X 修正左右;Y/Z 不动,前后与上下仍与实测一致。
|
| 4189 |
function _slamDisplayPoint(sourcePoint) {
|
| 4190 |
+
return [-Number(sourcePoint[0]), Number(sourcePoint[2]), -Number(sourcePoint[1])];
|
| 4191 |
}
|
| 4192 |
|
| 4193 |
function _slamSourcePoint(row) {
|
|
|
|
| 4327 |
|
| 4328 |
// Small axis marker at the fitted area: the recorded world axes, drawn in
|
| 4329 |
// the display frame that _slamDisplayPoint produces (world +Z is up).
|
| 4330 |
+
// The X tip follows the same negation as _slamDisplayPoint: display +X is
|
| 4331 |
+
// source -X, so the recorded +X axis points toward display -X.
|
| 4332 |
const axisOrigin = [fit.center[0], fit.gridY, fit.center[2]];
|
| 4333 |
const axisTips = [
|
| 4334 |
// World axes: X and Y horizontal, Z up.
|
| 4335 |
+
[[axisOrigin[0] - 0.2, axisOrigin[1], axisOrigin[2]], 'rgb(248,90,90)', 'X'],
|
| 4336 |
[[axisOrigin[0], axisOrigin[1], axisOrigin[2] - 0.2], 'rgb(80,220,130)', 'Y'],
|
| 4337 |
[[axisOrigin[0], axisOrigin[1] + 0.2, axisOrigin[2]], 'rgb(90,150,255)', 'Z'],
|
| 4338 |
];
|
|
|
|
| 4394 |
});
|
| 4395 |
|
| 4396 |
// Simplified UMI gripper model. The recorded pose frame is the S80M
|
| 4397 |
+
// camera frame; measured on the uploaded UMIGripper_AI episodes (every
|
| 4398 |
+
// frame of every batch) the device axes sit at
|
| 4399 |
+
// +X = gripper right (jaw separation, horizontal),
|
| 4400 |
+
// +Y = gripper forward (the optical axis, horizontal) and
|
| 4401 |
+
// +Z = gripper up (vertical, within ~12 deg of world +Z — the world
|
| 4402 |
+
// frame is gravity-aligned, so this is a fixed sensor-frame fact, not
|
| 4403 |
+
// a per-episode pose). So the body extends along local +Y, the two
|
| 4404 |
// plates separate along local X, and each plate is 0.20m long
|
| 4405 |
// (forward) by 0.05m tall (up) with a 0.12m maximum opening.
|
| 4406 |
const grip = _slamGripperState(currentRow);
|
|
|
|
| 4423 |
const jawFill = grip.gripped
|
| 4424 |
? 'rgba(248,90,90,0.16)' : 'rgba(88,168,232,0.12)';
|
| 4425 |
const relativeCorners = [
|
| 4426 |
+
[0, -plateLength / 2, -plateHeight / 2],
|
| 4427 |
+
[0, plateLength / 2, -plateHeight / 2],
|
| 4428 |
+
[0, plateLength / 2, plateHeight / 2],
|
| 4429 |
+
[0, -plateLength / 2, plateHeight / 2],
|
| 4430 |
];
|
| 4431 |
|
| 4432 |
for (const sign of [1, -1]) {
|
| 4433 |
const centerX = sign * halfGap;
|
| 4434 |
const corners = relativeCorners.map(corner => project(toDisplay([
|
| 4435 |
centerX + corner[0],
|
| 4436 |
+
baseOffset + corner[1],
|
| 4437 |
+
corner[2],
|
| 4438 |
])));
|
| 4439 |
if (corners.every(item => Number.isFinite(item[0]) && Number.isFinite(item[1]))) {
|
| 4440 |
ctx.fillStyle = jawFill;
|
|
|
|
| 4448 |
ctx.stroke();
|
| 4449 |
}
|
| 4450 |
line(
|
| 4451 |
+
toDisplay([centerX, baseOffset - plateLength / 2, 0]),
|
| 4452 |
+
toDisplay([centerX, baseOffset + plateLength / 2, 0]),
|
| 4453 |
jawColor, 1);
|
| 4454 |
}
|
| 4455 |
|
| 4456 |
// Rear crossbar plus the two supports that tie each plate back to
|
| 4457 |
// the body, so the opening stays readable when it is small.
|
| 4458 |
+
line(toDisplay([-maxHalfGap, baseOffset, 0]),
|
| 4459 |
+
toDisplay([maxHalfGap, baseOffset, 0]),
|
| 4460 |
'rgba(148,163,184,0.8)', 1.5);
|
| 4461 |
for (const sign of [1, -1]) {
|
| 4462 |
+
for (const cornerY of [-plateLength / 2, plateLength / 2]) {
|
| 4463 |
line(
|
| 4464 |
+
toDisplay([sign * halfGap, baseOffset + cornerY, 0]),
|
| 4465 |
+
toDisplay([0, baseOffset + cornerY, 0]),
|
| 4466 |
'rgba(148,163,184,0.8)', 1);
|
| 4467 |
}
|
| 4468 |
}
|
processor/web/templates/index.html
CHANGED
|
@@ -564,7 +564,7 @@
|
|
| 564 |
|
| 565 |
<script src="/static/js/media-cache.js?v=202609081530"></script>
|
| 566 |
<script src="/static/js/depth-renderer.js?v=202609081530"></script>
|
| 567 |
-
<script src="/static/js/player.js?v=
|
| 568 |
<script src="/static/js/hand-overlay.js?v=202609081630"></script>
|
| 569 |
<script src="/static/js/heatmap.js?v=202609081530"></script>
|
| 570 |
<script src="/static/js/annotations.js?v=202609081530"></script>
|
|
|
|
| 564 |
|
| 565 |
<script src="/static/js/media-cache.js?v=202609081530"></script>
|
| 566 |
<script src="/static/js/depth-renderer.js?v=202609081530"></script>
|
| 567 |
+
<script src="/static/js/player.js?v=20260911-tactile-square-1to1"></script>
|
| 568 |
<script src="/static/js/hand-overlay.js?v=202609081630"></script>
|
| 569 |
<script src="/static/js/heatmap.js?v=202609081530"></script>
|
| 570 |
<script src="/static/js/annotations.js?v=202609081530"></script>
|