Public image archive and retrieval database
This archive spans two public Hugging Face model repositories:
The data/*.bin objects contain image frames. The final SQLite database maps original image URLs and member paths to their repository, bundle, byte range, nonce, and SHA-256. Encrypted bundle indexes are in state/. All image decryption material is public: public-release/master.key is the master decryption key, not an account credential. Per-bundle data and index keys are deterministically derived from it; there are no additional image keys to obtain.
The complete database, key, release checksums, verification report, and usage scripts are in test-b/public-release. The release manifest is the authoritative inventory. Transfer totals: 942,675 source archives, 120,416,625 unique image entries, and 5,487 published data bundles. About 19.7 TB of encrypted bundles are split between the two repositories. Overlapping source archives share verified canonical entries.
Restore the database
Install Python 3.10 or newer and these packages:
python -m pip install huggingface_hub requests zstandard blake3 cryptography
hf download mashablestacenti/test-b public-release/restore_database.py public-release/retrieve.py --local-dir archive-tools
python archive-tools/public-release/restore_database.py --output migration.sqlite3
Allow at least 250 GB of disk space for the restored database and compressed download cache. Downloads are resumable through the Hugging Face cache. Re-running the restoration reuses downloaded parts, but you must first remove an incomplete migration.sqlite3.part or choose another --output. The script verifies each compressed part, the complete restored database SHA-256, and the public key checksum. It prints the downloaded key's local path. It never replaces an existing output database.
The numbered .zst.partNNNN files form one zstd stream in manifest order, not separate zstd archives. Do not extract them individually. Keep the database as a single file; WAL/SHM files are not needed for this finalized release.
Retrieve an image
Supply an exact original image URL and the key path printed by the restore script:
python archive-tools/public-release/retrieve.py --db migration.sqlite3 --key /path/to/master.key --url 'https://xcomic.me/EXACT/ORIGINAL/IMAGE.webp' --output image.webp
You can instead use --target 'urls/xcomic.me/...' with an exact entries.target_path. The script fetches only the requested HTTP byte range, decrypts with AES-256-GCM, authenticates the frame header, and checks the original URL identity, size, and SHA-256. It refuses servers that ignore Range and refuses to overwrite an existing output file. No account token is required to read this public release.
For example, find a real URL using SQLite:
SELECT url, target_path FROM entries LIMIT 1;
SELECT e.url, b.repo, b.remote_path, e.frame_offset, e.frame_length
FROM entries e JOIN bundles b ON b.id=e.bundle_id
WHERE e.url_hash = 'BLAKE3 HEX OF EXACT ORIGINAL URL';
The retrieval script computes the URL hash automatically and uses its database index. Avoid searching entries.url directly on this very large database; use url_hash or target_path for indexed lookups.
Database and file format
entries: one canonical record per URL/target, with source metadata, frame offsets, nonce, plaintext SHA-256, and bundle ID.bundles: repository, remote path, ciphertext size/SHA-256, encrypted index identity, and publication verification metadata.sources: source archive completion and original byte counts.source_members: dependencies for archives that overlap other archives; these explain shared canonical image records.
Bundle headers are 64 bytes (MHF1). Image frame headers are 128 bytes (FRM1), followed by AES-256-GCM ciphertext including its 16-byte authentication tag. Integers are little endian. The complete 128-byte frame header is authenticated additional data.
For bundle ID N, derive its 32-byte data key as BLAKE3(key=master_key, input=b'data' + uint64_le(N)). The frame nonce is stored in both the SQLite entry and header bytes 48–59. The plaintext SHA-256 is at header bytes 76–107.
Legacy boundary compatibility: some frames at the beginning of a bundle were encoded in a batch before that batch rolled over from bundle N-1. These frames use the data key derived for N-1. The supplied reader tries N, then N-1, and accepts output only after successful AES-GCM authentication and plaintext SHA-256 validation. Both keys derive from the same public master key. The verification report records the actual key bundle ID for each sampled frame. This compatibility rule avoids rewriting the published archive; future assembler code re-encodes boundary frames with the destination key.
Encrypted indexes use the same derivation with domain b'index', a 64-byte IDX1 authenticated header, and its nonce at bytes 32–43. Their decrypted payload is a JSON array of entry metadata. The final SQLite database already contains these mappings, so index decryption is optional.
Check public-release/verification.json for the actual checks performed. Transfer completion and sampled authenticated retrieval do not mean every image was downloaded and decrypted a second time. This publication includes no Hugging Face write tokens, MEGA login credentials, SSH credentials, or unrelated application secrets.
Complete source metadata and missing-image records
The full original and processed source catalogs, series/chapter metadata and page ordering are preserved in public-release/metadata. Read its README for extraction, schema, ordering and URL-to-image reconstruction. All 34,776,104 incomplete downloader records and historical 404 mappings are in public-release/supplement. The public master.key and supplied retrieval scripts are all that is needed to decrypt images.