Blame for docs/extraction/README.md

# PK3 extraction: first completed research step

Verified on **2026-10-11**, using only `.cache/UrbanTerror43/`. No original executable or QVM was run. All 39 cached installation files still match the recorded [source snapshot](../inventory/snapshot.json). Original installation copies are removed from `assets/`; extracted member payloads are the durable asset inputs.

## Reproduce and verify

```sh
python3 scripts/setup-repository.py
python3 scripts/test_unpack_pk3.py
python3 scripts/test_storage_policy.py
python3 scripts/unpack_pk3.py extract
python3 scripts/unpack_pk3.py verify
```

The Python 3 standard library is sufficient for extraction. Git LFS and Git Xet are needed for repository storage. `extract` verifies existing outputs before reusing them, refuses altered/unexpected files, and never overwrites a changed payload. `verify` reconstructs member hashes and ZIP metadata from the original archives and compares the entire output/report set. Missing or changed reports fail verification. Custom source/output/report/snapshot roots and explicit decompression limits are available via `--help`; the three roots must be disjoint. The default snapshot requires exactly the supplied 23 PK3s with their recorded sizes/hashes.

## Layout and identities

```text
assets/extracted/<archive filename>/members/<six-digit entry index>/<original path>
docs/extraction/archives/<archive filename>.jsonl
```

For example, `sound/weapons/FRF1/frf1-inside.wav` from `zUrT43_010.pk3` is stored at `assets/extracted/zUrT43_010.pk3/members/000174/sound/weapons/FRF1/frf1-inside.wav`. Its lowercase-name counterpart in `zUrT43_021.pk3` is stored separately under member `000063`. The index is zero-based in central-directory order and includes directory records, so gaps between file indices are intentional.

Each member is identified by the full archive SHA-256, entry index, exact logical path, same-name occurrence number and uncompressed SHA-256. Archive order is only deterministic enumeration, **not an established game loading order**. No winner is selected and no override is flattened. Duplicate ZIP filenames also have separate indexed directories, although none occurs within these 23 archives. Exact duplicate bytes remain separate working-tree files with separate provenance; LFS/Xet may deduplicate their storage.

Git control names are escaped (`.gitattributes` becomes `%2Egitattributes`) to stop game payloads changing repository filters. Platform punctuation/percent signs are percent-encoded; unusually long components use a hash label. The manifest retains the exact decoded original path. Each isolated member directory avoids case-folding and Unicode-normalization collisions. The supplied archives need no Git-control-name escaping. ZIP names use their original flags/CP437 decoding; flags, extras and comments are retained for further format research.

The **743 empty directory entries are preserved as metadata**, because Git does not store empty directories. The **three empty skin files are physically present** and remain file entries, distinguished from directory records. Empty files use the Git LFS specification's empty-pointer form, which is zero bytes; all non-empty assets must be canonical SHA-256 LFS pointers. See the [Git LFS specification](https://github.com/git-lfs/git-lfs/blob/main/docs/spec.md).

## Verified findings

| Measurement | Result |
| --- | ---: |
| PK3 archives | 23 |
| Central-directory entries | 10,578 |
| File occurrences extracted and verified | 9,835 |
| Directory records | 743 |
| Uncompressed file bytes | 3,125,455,778 |
| Distinct exact logical paths | 9,721 |
| Distinct normalized/case-folded logical paths | 9,718 |
| Repeated exact-path groups/events | 114 / 114 |
| Repeated-path groups with differing bytes | 92 |
| Repeated-path groups with identical bytes | 22 |
| Case-collision groups | 3 |
| Distinct payload SHA-256 values | 9,069 |
| Identical-payload groups across any paths | 509 |
| Empty file occurrences | 3 |

All members were decompressed, checked by Python's ZIP CRC verification, hashed and then compared against the extracted files in a separate full verification pass. All source archive hashes match the earlier inventory. Every central-directory record uses flag bits `0` and ZIP creator system `0` (DOS metadata); this records archive metadata and does not establish the host on which the game was built. Compression is ZIP stored (`0`) or DEFLATE (`8`); no encrypted entries, symbolic links, duplicate filenames within an archive, or non-empty directory payloads were found.

The three empty files occur in `zUrT43_008.pk3`: `models/weapons2/negev/negev_hold_female_b.skin`, `negev_hold_female_w.skin`, and `negev_hold_male_b.skin`. Their presence is proven; their runtime role remains to be established.

For FRF1 sounds, `frf1-inside.wav` and `frf1-outside.wav` differ between the uppercase directory in archive `010` and the lowercase directory in archive `021`. `frf1-shelldrop.wav` is byte-identical in the two archives. All six occurrences are retained. Case-insensitive filesystems must not choose a variant for the project; the original engine's VFS behavior remains a research task.

None of the extracted payload hashes equals any of the 39 original installation-file hashes. The original LFS objects can therefore be removed without deleting an object required by these extracted members. This comparison is by SHA-256, rather than guessed filenames.

## Evidence files

- [summary.json](summary.json): counts, layout, extraction bounds, archive identities and extension census.
- `archives/*.jsonl`: an archive header followed by every file/directory record. Records retain original name, occurrence/index, output path, size, compressed size, CRC, SHA-256, compression/flags, date tuple, creator/extractor versions, external/internal attributes, local-header offset, volume, comments and extras.
- [path-variants.jsonl](path-variants.jsonl): every repeated logical path and all its versions; no runtime winner.
- [case-collisions.jsonl](case-collisions.jsonl): all case/normalization collisions and their retained paths/hashes.
- [identical-payloads.jsonl](identical-payloads.jsonl): every shared content hash and all occurrences, including empty skins.

The largest per-archive metadata report is below 1 MiB. Reports are ordinary UTF-8 Git documentation; extracted payloads use the `assets/**` LFS/Xet policy. These are descriptive metadata records, not decoded game logic or copied media inside documentation.

## Safety, validation and remaining work

The extractor preflights archive identities, paths, methods, encryption flags, file types and decompression bounds before producing outputs. It rejects absolute/drive paths, parent traversal, malformed path separators, embedded NUL names, symlink/special entries, filesystem symlink ancestors, root overlap and unexpected output files. Streaming decompression checks length and CRC; per-archive staging is promoted atomically and removed on failure. Defaults allow 1 GiB per member, 8 GiB total expansion, 100,000 entries and a 2,000:1 expansion ratio. These bounds accommodate this verified snapshot and can be explicitly changed for a separately reviewed input.

Synthetic tests cover archive overrides, same-archive duplicate entries, case and Unicode collisions, zero bytes, empty directories, Git control names, path escapes, links, decompression bounds, CRC corruption, changed outputs/reports, input snapshot drift and deterministic reruns. Storage tests also cover the empty-pointer form, original-file reintroduction, bypassed filters and payloads deleted later in history. Batch Git-object reading keeps guards practical for thousands of staged assets while preserving bounded reads.

This step completes provenance and extraction work in AP-003/AP-004/AP-005/AP-008. It establishes the layout and extraction-side safety portions of AP-006/AP-007; metadata-only/selected-member modes and future runtime package readers remain pending. Format parsing, VFS precedence, shaders/material semantics, BSP geometry/collision, MD3 animation, QVM instructions/syscalls, dependency closure, reference gameplay and Solari conversion are not inferred from successful unpacking.

Next research steps: inspect every QVM and native binary/library, validate BSP/MD3/AAS structures, parse original shader scripts with dependency records, then determine original VFS precedence through source evidence and controlled reference runs. Use the preserved member identities in every subsequent finding.