# Download, scene lookup, verification and recovery The release contains **103,288 Navtrainval scenes in 16 independent shards** and **204,831 SimScale scenes in 32 independent shards**. These are trajectory banks; images, logs, maps, evaluation caches and EPDMS-mined banks are not included. Each `.tar.gz` is a complete archive. Do not concatenate shards. The four trained models are directly usable `.ckpt` files under `weights/Checkpoints/` and require no extraction. ## Download Download the assets and their small support files together: ```bash hf download Rtwotwo/DriveSTA --local-dir DriveSTA cd DriveSTA ``` For a selected shard, retain `manifests/`, `scripts/` and `docs/release/` as well: ```bash hf download Rtwotwo/DriveSTA --local-dir DriveSTA \ --include 'manifests/*' 'scripts/*' 'docs/*' 'README.md' \ 'weights/Trajectories/Navtrainval/drivetraj_navtrainval_001-of-016.tar.gz' ``` The verification helpers require Python 3.9 or later and use only the standard library. Reading the numerical trajectory arrays additionally requires NumPy. ## Inventories | File | Purpose | | --- | --- | | `manifests/scene_to_shard.csv` | One row for every released scene: dataset, split, scene token, archive filename, extraction directory, file count and high-bank status marker | | `manifests/shard_summary.csv` | Compact 49-row archive table with scene/file counts, sizes, SHA-256 and file-manifest paths | | `manifests/index.json` | Dataset totals, per-shard scene/file counts, archive sizes and SHA-256, and checksum-manifest locations and SHA-256 | | `manifests/shards/*.files.jsonl.gz` | Every original extracted file's relative path, byte size and SHA-256, exported from the checksums embedded in its archive | | `docs/release/archives.json` | Full inventory of the 48 trajectory archives and metadata archive | | `docs/release/SHA256SUMS` | Checksums of downloadable assets, inventories and verification helpers; run from the download root | The scene identity is `(dataset, split, token)`; always retain the dataset and split when joining with other indexes. `COMPLETE.json` and `SHORTFALL.json` in the CSV describe the original mining result, not download health. A shortfall is an intentional source-data state and does not indicate a broken archive. To find the shard containing a scene, replace `SCENE_TOKEN` below: ```bash python scripts/verify_trajectories.py --root . \ --dataset navtrainval --token SCENE_TOKEN --locate ``` Use `--dataset simscale` for synthetic scenes. The CSV is also readable directly with Python's `csv.DictReader` or spreadsheet applications. When importing into a spreadsheet, set the `token` column to text before import to preserve leading zeros and long numeric identifiers. `csv.DictReader` and the provided lookup helper preserve tokens as strings automatically. ## Verify downloads and extract For a full download, check all listed checksums before extraction: ```bash sha256sum -c docs/release/SHA256SUMS python scripts/extract_release.py weights/Trajectories/*/*.tar.gz \ --destination . --workers 4 ``` For a partial download, verify or extract only the selected archive: ```bash python scripts/extract_release.py \ weights/Trajectories/Navtrainval/drivetraj_navtrainval_001-of-016.tar.gz \ --verify-only # Remove --verify-only and add --destination . to extract it. ``` The extractor checks each archive's complete SHA-256 before writing. It refuses to overwrite existing files. Use an empty destination for first extraction, or the repair command below when recovering an interrupted extraction. Retain the archives until the extracted files have passed verification. ## Check extracted files After a full extraction: ```bash python scripts/verify_trajectories.py --root . --workers 4 python scripts/verify_trajectories.py --root . --shard drivetraj_metadata.tar.gz ``` After a partial extraction, select the corresponding shard or scene: ```bash python scripts/verify_trajectories.py --root . \ --shard drivetraj_navtrainval_001-of-016.tar.gz python scripts/verify_trajectories.py --root . \ --dataset simscale --token SCENE_TOKEN ``` Checks include file presence, byte size and SHA-256. Exit status is `0` only when all selected files pass. `trajectory_integrity_report.json` lists the precise broken/missing files and their shards; use `--report PATH` to keep separate reports. `--quick` checks only existence and size, so it cannot detect corruption that preserves file size and must not replace the full final check. ## Repair only the affected files ```bash python scripts/verify_trajectories.py --root . \ --shard drivetraj_navtrainval_001-of-016.tar.gz --repair # Alternatively target one scene: python scripts/verify_trajectories.py --root . \ --dataset simscale --token SCENE_TOKEN --repair ``` Repair verifies the entire local archive first, then replaces only missing or incorrect files, using temporary files and atomic replacement. Every restored file is checked against its original SHA-256 and read back. Files that already pass verification are retained. This also recovers an interrupted extraction without extracting the remaining 47 shards again. If the archive itself is missing or fails its checksum, redownload that archive alone and repeat the repair command: ```bash hf download Rtwotwo/DriveSTA \ weights/Trajectories/Navtrainval/drivetraj_navtrainval_001-of-016.tar.gz \ --local-dir . --force-download ``` For a damaged inventory, redownload `manifests/` and `docs/release/SHA256SUMS` before checking again. Keep helpers and inventories from the same repository revision; `hf download --revision COMMIT_ID ...` pins a release revision. ## 中文速查 - `manifests/scene_to_shard.csv` 列出全部 308,119 个场景与分片的对应关系。 - 用 Excel 等表格软件导入时,将 `token` 列设为文本,保留前导零和长编号;核对脚本会自动保留原始字符串。 - 用 `--token 场景TOKEN --locate` 查分片;用 `--shard 分片名` 检查解压文件。 - 默认逐文件检查 SHA-256,能发现同等大小的内容损坏;`--quick` 仅作快速检查。 - 添加 `--repair` 只恢复缺失或损坏的文件,并保留已经正确的文件。 - 压缩包自身损坏时,只需重新下载报告指出的分片,再运行 `--repair`。 - 初次解压用 `extract_release.py`;中断后继续恢复用 `verify_trajectories.py --repair`。