Hugging Face's logo

nmarshall
/
urt-rs

Model card Files
xet
 Community
urt-rs / docs /STORAGE.md
nmarshall's picture
nmarshall
Extract every PK3 variant and keep originals cache-only
2811ca0
preview code
|
Raw History Blame Contribute Delete
8.16 kB

Ordinary Git code and Xet assets in one repository

One Hugging Face repository stores this project: nmarshall/urt-rs, SSH remote hf at git@hf.co:nmarshall/urt-rs. Code, scripts, modern WGSL sources, documentation and readable research metadata are ordinary Git text. Non-empty files below assets/ are canonical Git LFS pointers; payload uploads/downloads use the locally registered Git Xet transfer agent. A single commit identifies code and derived assets. No separate dataset or nested asset repository is used.

Layout 

urban-terror-rs/
  .gitattributes              # Required text filter; assets/** uses LFS
  .githooks/                  # Commit/index/history guards and LFS upload hook
  scripts/                    # Setup, extraction and storage-policy code
  docs/                       # Findings and per-occurrence metadata in ordinary Git
  assets/
    README.md                 # Uses LFS
    extracted/                # All PK3 variants; archive/member-index isolation
    converted/                # Future Bevy/Solari runtime content
    captures/                 # Future reference media and traces
  .cache/UrbanTerror43/        # The original installation; ignored and unchanged

The 39 original installation files belong in .cache/ only. Their copied assets/original/ tree is removed and ignored. Both the original directory path and LFS pointers with any of the 39 installation-file SHA-256 values are rejected by the storage guard, preventing renamed original files from returning to published assets. The immutable source identity remains recorded in docs/inventory/snapshot.json.

assets/extracted/ contains every PK3 member occurrence, as described in the extraction findings. None of its payload hashes equals an original installation-file hash. Standalone binaries, libraries, license documents and installation configs remain available for research in .cache/; extraction does not discard them or claim their reverse engineering is complete. The retired unpublished nested checkout metadata also remains under ignored .cache/retired-assets-checkout/.

Filters and guards 

* text=auto eol=lf filter=urt-code
assets/** filter=lfs diff=lfs merge=lfs -text !eol

The required urt-code clean filter passes through valid UTF-8 source/documentation. It rejects known asset/binary extensions, asset/build/cache directories, NUL bytes, invalid encoding, LFS pointers outside assets/ and ordinary Git files above 2 MiB. Readable inventory/extraction manifests are metadata, not game payloads. New asset formats must extend the policy before ingestion.

Every asset path has LFS attributes, including small configs and original shader text extracted from PK3s. Git LFS writes a SHA-256 pointer for non-empty files. Its specification defines an empty file as its own empty pointer; the three zero-byte skins therefore contain zero bytes in Git and need no remote payload object. No non-empty raw asset data is accepted in ordinary Git. See the Git LFS pointer specification.

Pre-commit validates staged content, cached routing attributes and local filter/Xet configuration. Pre-push repeats those checks and validates every newly reachable commit, including payloads deleted later. A bounded git cat-file --batch stream handles thousands of pointers without launching Git for each file. Guards reject symlinks, unresolved entries, nested repositories, cached original-file identities and non-pointer payloads. The push hook uploads through git lfs pre-push before Git refs advance.

Git Xet is registered as the custom LFS transfer agent; filter=lfs is the supported Git filter. See Hugging Face's Git/Xet integration. Setup changes local repository settings only. Clones do not install hooks automatically; deliberate disabling of hooks/configuration can bypass local checks. Future CI/server protection should run the same content policy.

Configure or clone

Requirements: Git, Python 3, Git LFS, Git Xet and Hugging Face SSH access.

GIT_LFS_SKIP_SMUDGE=1 git clone git@hf.co:nmarshall/urt-rs
cd urt-rs
python3 scripts/setup-repository.py
git lfs pull
git lfs fsck
python3 scripts/storage_policy.py check --require-setup

The clone names its remote origin; this checkout names it hf. Substitute the configured name in push commands. For further extraction, provide the verified original installation under .cache/UrbanTerror43/; Git intentionally does not distribute it. Setup is repeatable and refuses to overwrite an unrelated hooks directory.

Work and publish 

python3 scripts/test_storage_policy.py
python3 scripts/test_unpack_pk3.py
python3 scripts/unpack_pk3.py extract
python3 scripts/unpack_pk3.py verify
git add .
python3 scripts/storage_policy.py check --require-setup
git commit -m "Describe the change"
git lfs fsck
GIT_LFS_FORCE_PROGRESS=1 git push hf main

For large uploads, enable progress and optionally set GIT_LFS_PROGRESS to an absolute ignored log path. If a Xet worker stops advancing with no live transfer connections, stop only that push process group and retry; already completed objects remain uploaded. A retry with git -c lfs.concurrenttransfers=1 push hf main successfully completed the earlier initial installation upload. It is a per-command setting, rather than a change to repository routing.

A non-empty asset's git show HEAD:assets/path must show a small LFS pointer; the working file contains the payload. Source files must show their source text directly. If a derived asset was staged incorrectly, restore setup/attributes, run git add --renormalize assets/extracted, and validate before committing. Remove copied installation files instead of renormalizing them.

The first extraction upload exposed another distinct failure: the Hub rejected LFS batch calls after the account reached 1,000 API requests per five minutes. The attempt did not advance Git refs. Completed object uploads were reused on retry. Thousands of small files can exhaust request quotas even when bandwidth and disk usage are modest. Allow the rate-limit window to reset before retrying, reduce per-command transfer concurrency (for example -c lfs.concurrenttransfers=2), and retain an ignored diagnostic log. Do not repeatedly retry immediately or infer success from progress reaching “done”; check the Git command's exit status and remote ref. See the Hub rate-limit documentation.

Original payloads and remote history cleanup

Deleting files in a normal commit does not remove their earlier pointers or stored LFS/Xet objects. Hugging Face supports targeted permanent LFS object deletion with history rewriting; object identity is the SHA-256, and deletion affects every reference to that object. This needs authenticated Hub access in addition to SSH Git access. The repository's Settings → List Large Files also exposes this operation.

The cleanup selects exactly the 39 source installation hashes from the snapshot, preserves the repository's asset README and code/docs, and checks that no extracted object shares those hashes. It avoids squashing unrelated source/documentation history. After server-side rewriting, fetch the remote's rewritten main and base the extraction commit on it so an old local parent does not reintroduce the removed pointers. Clones retaining the previous history must likewise resynchronize before pushing.

Execution evidence and final cleanup status are recorded in the extraction research ledger. Provider quota accounting can lag object cleanup; the Hub storage documentation describes delays of up to 36 hours. Removing objects from this repository does not establish physical erasure of shared provider-level Xet chunks.