|
Download docs/DOWNLOAD_VERIFY.md from Rtwotwo/DriveSTA: direct link, hf CLI and curl.
- Browser
- Download file 6.41 kB
-
https://huggingface.co/Rtwotwo/DriveSTA/resolve/main/docs/DOWNLOAD_VERIFY.md
- Command line
-
hf download hf://Rtwotwo/DriveSTA/docs/DOWNLOAD_VERIFY.md
-
curl -L -o DOWNLOAD_VERIFY.md https://huggingface.co/Rtwotwo/DriveSTA/resolve/main/docs/DOWNLOAD_VERIFY.md
6.41 kB
| # Download, scene lookup, verification and recovery | |
| The release contains **103,288 Navtrainval scenes in 16 independent shards** and | |
| **204,831 SimScale scenes in 32 independent shards**. These are trajectory banks; | |
| images, logs, maps, evaluation caches and EPDMS-mined banks are not included. | |
| Each `.tar.gz` is a complete archive. Do not concatenate shards. | |
| The four trained models are directly usable `.ckpt` files under | |
| `weights/Checkpoints/` and require no extraction. | |
| ## Download | |
| Download the assets and their small support files together: | |
| ```bash | |
| hf download Rtwotwo/DriveSTA --local-dir DriveSTA | |
| cd DriveSTA | |
| ``` | |
| For a selected shard, retain `manifests/`, `scripts/` and `docs/release/` as well: | |
| ```bash | |
| hf download Rtwotwo/DriveSTA --local-dir DriveSTA \ | |
| --include 'manifests/*' 'scripts/*' 'docs/*' 'README.md' \ | |
| 'weights/Trajectories/Navtrainval/drivetraj_navtrainval_001-of-016.tar.gz' | |
| ``` | |
| The verification helpers require Python 3.9 or later and use only the standard | |
| library. Reading the numerical trajectory arrays additionally requires NumPy. | |
| ## Inventories | |
| | File | Purpose | | |
| | --- | --- | | |
| | `manifests/scene_to_shard.csv` | One row for every released scene: dataset, split, scene token, archive filename, extraction directory, file count and high-bank status marker | | |
| | `manifests/shard_summary.csv` | Compact 49-row archive table with scene/file counts, sizes, SHA-256 and file-manifest paths | | |
| | `manifests/index.json` | Dataset totals, per-shard scene/file counts, archive sizes and SHA-256, and checksum-manifest locations and SHA-256 | | |
| | `manifests/shards/*.files.jsonl.gz` | Every original extracted file's relative path, byte size and SHA-256, exported from the checksums embedded in its archive | | |
| | `docs/release/archives.json` | Full inventory of the 48 trajectory archives and metadata archive | | |
| | `docs/release/SHA256SUMS` | Checksums of downloadable assets, inventories and verification helpers; run from the download root | | |
| The scene identity is `(dataset, split, token)`; always retain the dataset and | |
| split when joining with other indexes. `COMPLETE.json` and `SHORTFALL.json` in the | |
| CSV describe the original mining result, not download health. A shortfall is an | |
| intentional source-data state and does not indicate a broken archive. | |
| To find the shard containing a scene, replace `SCENE_TOKEN` below: | |
| ```bash | |
| python scripts/verify_trajectories.py --root . \ | |
| --dataset navtrainval --token SCENE_TOKEN --locate | |
| ``` | |
| Use `--dataset simscale` for synthetic scenes. The CSV is also readable directly | |
| with Python's `csv.DictReader` or spreadsheet applications. | |
| When importing into a spreadsheet, set the `token` column to text before import | |
| to preserve leading zeros and long numeric identifiers. `csv.DictReader` and the | |
| provided lookup helper preserve tokens as strings automatically. | |
| ## Verify downloads and extract | |
| For a full download, check all listed checksums before extraction: | |
| ```bash | |
| sha256sum -c docs/release/SHA256SUMS | |
| python scripts/extract_release.py weights/Trajectories/*/*.tar.gz \ | |
| --destination . --workers 4 | |
| ``` | |
| For a partial download, verify or extract only the selected archive: | |
| ```bash | |
| python scripts/extract_release.py \ | |
| weights/Trajectories/Navtrainval/drivetraj_navtrainval_001-of-016.tar.gz \ | |
| --verify-only | |
| # Remove --verify-only and add --destination . to extract it. | |
| ``` | |
| The extractor checks each archive's complete SHA-256 before writing. It refuses | |
| to overwrite existing files. Use an empty destination for first extraction, or | |
| the repair command below when recovering an interrupted extraction. Retain the | |
| archives until the extracted files have passed verification. | |
| ## Check extracted files | |
| After a full extraction: | |
| ```bash | |
| python scripts/verify_trajectories.py --root . --workers 4 | |
| python scripts/verify_trajectories.py --root . --shard drivetraj_metadata.tar.gz | |
| ``` | |
| After a partial extraction, select the corresponding shard or scene: | |
| ```bash | |
| python scripts/verify_trajectories.py --root . \ | |
| --shard drivetraj_navtrainval_001-of-016.tar.gz | |
| python scripts/verify_trajectories.py --root . \ | |
| --dataset simscale --token SCENE_TOKEN | |
| ``` | |
| Checks include file presence, byte size and SHA-256. Exit status is `0` only when | |
| all selected files pass. `trajectory_integrity_report.json` lists the precise | |
| broken/missing files and their shards; use `--report PATH` to keep separate | |
| reports. `--quick` checks only existence and size, so it cannot detect corruption | |
| that preserves file size and must not replace the full final check. | |
| ## Repair only the affected files | |
| ```bash | |
| python scripts/verify_trajectories.py --root . \ | |
| --shard drivetraj_navtrainval_001-of-016.tar.gz --repair | |
| # Alternatively target one scene: | |
| python scripts/verify_trajectories.py --root . \ | |
| --dataset simscale --token SCENE_TOKEN --repair | |
| ``` | |
| Repair verifies the entire local archive first, then replaces only missing or | |
| incorrect files, using temporary files and atomic replacement. Every restored | |
| file is checked against its original SHA-256 and read back. Files that already | |
| pass verification are retained. This also recovers an interrupted extraction | |
| without extracting the remaining 47 shards again. | |
| If the archive itself is missing or fails its checksum, redownload that archive | |
| alone and repeat the repair command: | |
| ```bash | |
| hf download Rtwotwo/DriveSTA \ | |
| weights/Trajectories/Navtrainval/drivetraj_navtrainval_001-of-016.tar.gz \ | |
| --local-dir . --force-download | |
| ``` | |
| For a damaged inventory, redownload `manifests/` and `docs/release/SHA256SUMS` | |
| before checking again. Keep helpers and inventories from the same repository | |
| revision; `hf download --revision COMMIT_ID ...` pins a release revision. | |
| ## 中文速查 | |
| - `manifests/scene_to_shard.csv` 列出全部 308,119 个场景与分片的对应关系。 | |
| - 用 Excel 等表格软件导入时,将 `token` 列设为文本,保留前导零和长编号;核对脚本会自动保留原始字符串。 | |
| - 用 `--token 场景TOKEN --locate` 查分片;用 `--shard 分片名` 检查解压文件。 | |
| - 默认逐文件检查 SHA-256,能发现同等大小的内容损坏;`--quick` 仅作快速检查。 | |
| - 添加 `--repair` 只恢复缺失或损坏的文件,并保留已经正确的文件。 | |
| - 压缩包自身损坏时,只需重新下载报告指出的分片,再运行 `--repair`。 | |
| - 初次解压用 `extract_release.py`;中断后继续恢复用 `verify_trajectories.py --repair`。 | |