diff --git a/security/RELEASE_ARCHIVE_ANCHOR_GUIDE.md b/security/RELEASE_ARCHIVE_ANCHOR_GUIDE.md deleted file mode 100644 index 00c054200dda7fc61c1fef640d8da4342b84979c..0000000000000000000000000000000000000000 --- a/security/RELEASE_ARCHIVE_ANCHOR_GUIDE.md +++ /dev/null @@ -1,27 +0,0 @@ -# Run 163 — External archive-health anchors and witness-root recovery - -Run 163 moves the Run 162 archive-health witness chain outside one local trust domain. Every accepted witness-chain head is written into multiple independently operated append-only channels. Each channel returns a signed, hash-linked checkpoint containing the exact Run 162 witness sequence/head and its previous accepted channel checkpoint. A separate read-only observer for that channel signs an independent inclusion and continuity read-back of the exact channel response. - -## Fail-closed split view rule - -All configured channels and observers must agree on the exact Run 162 witness-chain head for the epoch. There is no majority-wins split view resolution. A channel or observer presenting another witness head, predecessor checkpoint, challenge, or sequence fails the release even if the other channels agree. - -The anchor plan is signed by the active Run 162 witness-root threshold and pins the exact channel and observer public keys and operators. Channel operators and observer operators are separate. The accepted anchor bundle preserves the exact Run 162 documents plus all signed channel and observer responses, so offline replay does not depend on a future live transparency service. - -## Append-only continuity - -For each channel, epoch N+1 must name the SHA-256 of that channel's exact signed response from epoch N as `previousCheckpointSha256`. The aggregate anchor history has its own `anchorConsensusHeadSha256`, which hash-links every accepted epoch. This generic contract does not pretend to be a Merkle transparency proof; provider-specific Rekor/CT/log integrations can implement the same adapter contract with stronger native inclusion/consistency proofs. - -## Witness root recovery - -Witness root recovery is a separate break-glass plane. It uses an independently pinned threshold recovery root with independent operators and cryptographically pinned recovery channels. Recovery binds the exact already accepted anchor consensus head, anchored sequence, and Run 162 witness-chain head. `rewrittenAnchorEpochs` must be an empty list. Recovery therefore cannot rewrite or delete anchored epochs; it only establishes a replacement archive-health witness root for future work. - -The replacement witness root must not reuse old witness key IDs or operators, and the recovery authority must be separate from both old and replacement witness authorities. Private signing keys are never accepted by the repository tool. - -## Operational requirements - -- Keep anchor channel credentials separate from observer credentials. -- Keep witness-root recovery credentials offline or otherwise independently protected. -- Preserve the exact anchor plan and out-of-band witness/recovery root SHA-256 pins. -- Treat any observed channel/observer disagreement as an incident, not as a quorum vote. -- Do not claim a generic hash-linked adapter is a native Merkle transparency log unless the adapter separately verifies that provider's inclusion/consistency proofs. diff --git a/security/RELEASE_ARCHIVE_HEALTH_GUIDE.md b/security/RELEASE_ARCHIVE_HEALTH_GUIDE.md deleted file mode 100644 index 5744706edbb6672dc6aac92ca4f1c2109a71d4e7..0000000000000000000000000000000000000000 --- a/security/RELEASE_ARCHIVE_HEALTH_GUIDE.md +++ /dev/null @@ -1,129 +0,0 @@ -# Run 161 — Archive retention and durable-copy health - -Run 160 proves that the exact Run 159 evidence was copied to independent immutable -archives. Run 161 answers the longer-lived question: **are enough independent copies still -present, retention-protected, and independently readable now?** - -`audit_archive_retention.py` creates an append-only archive-health chain. It does not trust -an archive adapter's text claim that a bucket is immutable. The active membership is -threshold-authorized by an **out-of-band** pinned retention-governance root; every archive -then returns a **provider-signed retention** statement and a separate auditor performs an -**independent challenge** / read-back of the exact immutable version. - -## Authority planes - -Run 161 separates three authorities: - -1. **Retention governance** — an Ed25519 M-of-N root authorizes exact archive membership, - provider/auditor public keys, minimum durable copies, migration, and retirement. -2. **Storage provider** — signs the exact archive ID, locator, immutable version ID, - artifact SHA-256/size, immutability class, retention mode, retention expiry, legal-hold - state, challenge, read-back result, and observation time. -3. **Independent auditor** — uses a distinct identity/operator and signs an independent - read-only read-back of that exact version and provider-response hash. - -Governance, storage-provider, and auditor private keys never enter repository evidence. -Public keys and key expiry are part of the threshold-signed membership document. - -## Bootstrap - -Sequence 1 MUST describe exactly the Run 160 archive identities, operators, archive IDs, -locators, and immutability classes. This prevents a fresh retention root from silently -switching the underlying durable copies during bootstrap. - -The retention-governance root itself is accepted only when its complete canonical JSON -SHA-256 matches an independently retained out-of-band pin and its selected self-signing -keys satisfy the configured threshold and operator diversity. - -## Provider-signed retention - -Every current member must provide a signed observation over: - -- exact Run 160 archive artifact name, SHA-256, and size; -- exact archive identity, archive ID, and sanitized remote locator; -- immutable provider version/object ID; -- approved immutability and retention mode; -- `retentionUntil` and `legalHold`; -- challenge-bound remote read-back proof; and -- observation time. - -Unless an active legal hold covers the object, the default policy requires at least 30 -days of remaining retention at each live audit. Provider keys must still be valid at the -signed observation time. - -## Independent challenge and read-back - -The challenge is derived from the audit ID and previous health-chain head. The independent -auditor must bind the same challenge, locator, immutable version ID, Run 160 artifact, -and provider-response SHA-256 and must assert read-only remote read-back. - -Archive-provider and auditor operators are disjoint. A provider cannot satisfy its own -auditor plane merely by using another identity string. - -## Cumulative health history - -Each event includes the threshold-signed membership and canonical normalized audit result -for every active member. Its health-chain head commits to the previous chain head and the -complete current event. Earlier events replay historically, while the active event retains -live retention and observation freshness checks. - -The default maximum interval between successful audits is 31 days. An accepted historical -audit does not become invalid merely because its retention window later expires; a current -archive must, however, keep passing live retention health. - -## Migration and retirement authorization - -A membership change is not itself permission to delete an old copy. - -For a migration, Run 161 first verifies provider-signed retention and independent read-back -for **every member of the new active set**. Only after that complete set satisfies the -configured **minimum durable copies** and independent-operator requirements does -`active-archive-health-evidence.json` emit `retirementAuthorizedArchiveIds`. - -Operational rule: - -```text -old durable set - ↓ -threshold-signed migration membership - ↓ -create / retain every new copy - ↓ -provider-signed retention + independent read-back for ALL new active members - ↓ -minimum durable copies still satisfied - ↓ -Run 161 health event commits successfully - ↓ -retirement authorization for removed members - ↓ -only now may an operator retire those old copies -``` - -A migration that would reduce the next active set below policy minimum fails before any -retirement authorization is produced. - -## Persistent artifacts - -Run 161 emits exactly four files: - -- `release-archive-health-bundle.json` — cumulative threshold-governed health history; -- `trusted-archive-health-state.json` — compact current sequence and chain-head binding; -- `active-archive-health-evidence.json` — current membership/audit result and any retirement - authorization; and -- `release-archive-health-receipt.json` — exact provider/auditor signed response documents - required for offline cryptographic replay. - -No local source paths, credentials, signed URLs, secret query parameters, private keys, or -HSM handles belong in those artifacts. - -## Historical versus live verification - -Run 160 is always replayed historically at this layer: Run 161 is auditing the durability -of an already accepted archive, not re-deciding whether old OCSP/CRL evidence is currently -fresh. Current Run 161 acceptance instead requires a live retention-governance root and -fresh provider/auditor observations for the active health event. - -This separation prevents a previously accepted native-status archive from becoming -unverifiable merely because the original PKI evidence expires, while still failing closed -when the actual durable copies lose retention/read-back health. diff --git a/security/RELEASE_ARCHIVE_LOG_AUTHORITY_GUIDE.md b/security/RELEASE_ARCHIVE_LOG_AUTHORITY_GUIDE.md deleted file mode 100644 index ea3d16a7fddf26512b5b12554d7858932f210372..0000000000000000000000000000000000000000 --- a/security/RELEASE_ARCHIVE_LOG_AUTHORITY_GUIDE.md +++ /dev/null @@ -1,94 +0,0 @@ -# Run 165 — Merkle log-key authority lifecycle and recovery - -Run 164 proves that every accepted archive-health anchor is included in, and consistently -extends, multiple independent Merkle logs. Run 165 governs what happens when a log or -cross-channel gossip signing key must rotate after that proof has already been accepted. - -## Trust boundary - -Run 165 introduces two independently pinned Ed25519 roots: - -1. an **archive-log governance root** for scheduled key rotations; and -2. a separate **archive-log recovery root** for compromise recovery. - -Both roots are self-authorized by exact selected M-of-N signer sets with independent -operators. Recovery keys additionally carry pinned recovery-channel identities. Governance, -recovery, Run 164 transparency-root, log, and gossip public keys/operators are kept in -separate trust planes. - -## Run 164 checkpoint handoff - -The Run 165 bootstrap is bound to one already accepted Run 164 snapshot. Every log and -gossip key must sign a canonical handoff subject containing its exact accepted Run 164: - -- log ID; -- tree size; -- Merkle root hash; -- signed checkpoint SHA-256; -- Run 164 sequence and Merkle-consensus head; -- current log/gossip authority; -- next log/gossip authority. - -The accepted Run 164 snapshot is immutable for the Run 165 authority chain. Future Merkle -append operations can therefore start only from those exact tree roots rather than from a -new tree silently created during a key rotation. - -## Scheduled rotation — old + new authority - -A normal `scheduled-rotation` is authorized by the governance-root threshold. Every active -**new** log and gossip key signs the checkpoint handoff. For every changed log, the **old + new** -log and gossip keys both sign the same handoff subject. - -This is an explicit continuity ceremony: possession of a replacement key alone does not -allow the operator to claim that it inherited an existing Merkle tree. - -Replaced log/gossip public-key fingerprints are added to permanent revocation history. -Operator identity cannot be rebound while retaining the same signing key. - -## Compromise recovery - -If an old log key cannot safely co-sign because it is compromised, `compromise-recovery` -uses the separately pinned recovery-root threshold instead of the governance quorum. - -The transition must identify at least one compromised current key fingerprint. Every -affected log rotates **both** its log-signing key and gossip-signing key. Old handoff -signatures are forbidden; all replacement keys still sign the exact last accepted Run 164 -checkpoint. The complete old key pair for every affected log enters permanent revocation. - -Recovery signer selection must satisfy both independent-operator and pinned recovery-channel -thresholds. A later transition cannot reintroduce a permanently revoked key. - -## No hidden Merkle advancement - -Run 165 is intentionally a log-authority handoff layer. It does not append another Merkle -leaf. All authority transitions in one Run 165 chain remain bound to the same exact Run 164 -artifact set, sequence, per-log tree sizes/roots, and Merkle-consensus head. A successor -Merkle engine can consume `active-archive-log-authority.json` and must extend those exact -continuity checkpoints under the currently active authority. - -## Canonical outputs - -Successful application produces exactly: - -```text -release-archive-log-authority-bundle.json -trusted-archive-log-authority-state.json -active-archive-log-authority.json -release-archive-log-authority-receipt.json -``` - -The cumulative bundle hash-links every transition. The receipt preserves the exact -transition, all per-log handoff signatures, and the canonical Run 164 documents used by the -ceremony. `verify` replays governance/recovery signatures, permanent revocation, Run 164 -Merkle proofs, and every old/new handoff without contacting the original logs. - -## Operational requirements - -- Retain governance-root and recovery-root pins outside the transparency-log service. -- Keep governance and recovery credentials in separate operators and security domains. -- For scheduled rotation, never bypass the old + new handoff requirement. -- For compromise recovery, identify compromised key fingerprints explicitly and rotate the - complete log/gossip key pair for every affected log. -- Treat permanent revocation as monotonic; never recycle an old public key into a later log. -- Preserve Run 164 and Run 165 outputs together so an offline verifier can reproduce the - exact authority handoff from the last accepted Merkle checkpoint. diff --git a/security/RELEASE_ARCHIVE_MERKLE_CONTINUITY_GUIDE.md b/security/RELEASE_ARCHIVE_MERKLE_CONTINUITY_GUIDE.md deleted file mode 100644 index 164f8253d66dad378bf64d699b6e416db4665a68..0000000000000000000000000000000000000000 --- a/security/RELEASE_ARCHIVE_MERKLE_CONTINUITY_GUIDE.md +++ /dev/null @@ -1,35 +0,0 @@ -# Release archive Merkle authority continuity — Run 166 - -Run 166 makes the **Run 165** active log authority usable for later RFC6962 Merkle epochs. It does not invent a new tree and it does not ask a retired key to sign again. The first post-handoff checkpoint must carry a real **consistency proof** from the exact Run 164 checkpoint preserved by Run 165. - -## Trust boundary - -Run 164 remains the pre-handoff Merkle history. Run 165 proves the legitimate log/gossip authority transition, including scheduled old + new rotation or **compromise recovery**. Run 166 verifies both histories, then accepts future Merkle checkpoints only from `active-archive-log-authority.json`. - -For every log, the first post-handoff request binds the exact preserved tree size/root, Run 165 log-authority chain head, active-authority SHA-256, current Run 163 anchor head, and new RFC6962 leaf. The new active key signs the resulting checkpoint. A retired key is not needed for this append. - -## Continuity - -The post-handoff chain starts from Run 164's accepted Merkle consensus head. Each Run 166 event stores the exact current Run 163 documents, RFC6962 inclusion and consistency proofs, log responses, gossip responses, Run 165 authority identifiers, and the cumulative `merkleAuthorityContinuityHeadSha256`. - -Run 166 supports multiple later epochs under the same Run 165 active authority. Changing the active Run 165 authority after post-handoff appends requires a new explicitly governed bridge rather than silently rebinding an existing Run 166 history. - -## Live versus historical verification - -Accepted old log/gossip signatures replay historically. Only the newest post-handoff epoch is subject to the active-epoch freshness window. The Run 165 handoff itself is historically verified, while a new append also requires its accepted authority transition to be within the configured active-authority age. - -## Fail-closed rules - -- exact Run 164 and Run 165 artifact rebinding; -- exact Run 165 active authority and continuity checkpoints; -- exact sequence: first post-handoff epoch = Run 164 sequence + 1; -- exact adapter/log set; -- RFC6962 inclusion and consistency verification; -- split-view gossip rejection; -- authority separation from Run 162 witness and Run 163 anchor operators/keys; -- input-drift detection before commit; -- server-owned staging and output allowlisting; -- duplicate JSON-key rejection; -- deterministic canonical JSON. - -Run 166 contains no private signing keys. Provider-specific log clients remain external adapters. diff --git a/security/RELEASE_ARCHIVE_MERKLE_GUIDE.md b/security/RELEASE_ARCHIVE_MERKLE_GUIDE.md deleted file mode 100644 index dba0598cd2ecd162a07f8a5a4a197a11b9b56562..0000000000000000000000000000000000000000 --- a/security/RELEASE_ARCHIVE_MERKLE_GUIDE.md +++ /dev/null @@ -1,121 +0,0 @@ -# Run 164 — Merkle transparency for archive-health anchors - -Run 163 creates independently signed, hash-linked external anchor checkpoints. Run 164 -adds cryptographic tree proofs so an external channel cannot satisfy the release boundary -merely by asserting that it is append-only. - -## Trust model - -Run 164 uses an independently retained `archive-merkle-transparency-root`. The exact -canonical root bytes are pinned out of band by SHA-256. The root is self-authorized by an -operator-diverse Ed25519 threshold and defines three separate authority planes: - -1. transparency-root signers; -2. Merkle-log checkpoint signers; -3. read-only cross-channel gossip observers. - -Those authorities must also be separate from the Run 162 witness root and the Run 163 -anchor-channel / observer principals. - -## RFC6962 hashing and proofs - -The tree construction follows the RFC6962 domain-separation convention: - -```text -leaf = SHA256(0x00 || canonical Run 163 anchor leaf) -node = SHA256(0x01 || left || right) -``` - -Every configured log must return a signed checkpoint containing the exact leaf hash, -leaf index, tree size, root hash, inclusion proof, and—after bootstrap—a consistency proof -from the previously accepted tree size/root. - -Run 164 verifies the **inclusion proof** itself. A channel statement such as -`inclusionVerified: true` has no authority. - -Run 164 also verifies the RFC6962-style **consistency proof** itself. For sequence N+1, -the accepted tree must cryptographically extend the exact tree root accepted for sequence -N. A signed `previousRootHash` field alone is insufficient. - -The first accepted Run 164 event uses the empty accepted tree as its predecessor. A later -event must increase every configured log's tree size; equal or decreasing sizes fail. - -## Canonical Run 163 leaf - -The Merkle leaf binds the exact current Run 163 state and active evidence, including: - -- Run 163 sequence; -- Run 162 witness sequence; -- Run 163 `anchorConsensusHeadSha256`; -- SHA-256 of `trusted-archive-anchor-state.json`; -- SHA-256 of `active-archive-anchor-evidence.json`; -- configured Run 163 channel IDs. - -The receipt also preserves all four canonical Run 163 output documents so an offline -auditor can reproduce the leaf bytes after the original anchor channels disappear. - -## Cross-channel gossip and split view handling - -Merkle validity does not by itself stop a provider from presenting two valid but different -append-only trees to different observers. Run 164 therefore builds one canonical map of -all accepted checkpoint SHA-256 values and sends that exact map to independent gossip -observers. - -Every gossip response is Ed25519-signed and must bind the same: - -- Run 164 sequence; -- Run 163 anchor consensus head; -- Merkle leaf hash; -- complete sorted log-id → checkpoint-SHA-256 map. - -Any **split view** is release-blocking. There is no majority-wins path: one independently -verified conflicting gossip view is enough to fail the event. - -## Historical replay versus live freshness - -Adapter results must be fresh when a new event is created. Accepted older events replay -historically at their signed integration/observation time. Only the active Run 164 event -must satisfy the configured live anchor-age limit. This prevents previously accepted -Merkle history from becoming unverifiable solely because an old checkpoint is more than -30 minutes old. - -The currently used transparency root still requires live expiry/freeze checks unless the -caller explicitly performs historical verification. - -## Output - -A successful event produces exactly: - -```text -release-archive-merkle-bundle.json -trusted-archive-merkle-state.json -active-archive-merkle-evidence.json -release-archive-merkle-receipt.json -``` - -The bundle is cumulative. `merkleConsensusHeadSha256` hash-links the complete canonical -event history, while each individual log separately provides its own Merkle consistency -proof chain. - -## Offline verification - -`verify_archive_merkle_transparency.py verify` replays the saved signatures, RFC6962 -inclusion proofs, consistency proofs, gossip observations, artifact hashes, and cumulative -consensus head without contacting a transparency service. - -Production deployments may map the adapter contract onto Rekor, Certificate Transparency, -a private Trillian-style service, or another authenticated append-only log. Provider- -specific receipt formats are not treated as inherently trustworthy: the adapter must -normalize them into the signed checkpoint contract whose Merkle proofs are independently -verified here. - -## Operational requirements - -- Keep the transparency-root pin outside the log service. -- Use at least three independently operated logs and three independent gossip observers. -- Keep root, log, gossip, Run 162 witness, Run 163 channel, and Run 163 observer credentials - in separate trust domains. -- Persist exact accepted checkpoint and gossip response bytes. -- Alert on any inclusion failure, consistency failure, tree-size rollback, root fork, or - gossip split view. -- Do not substitute a signed provider boolean for a cryptographic proof. diff --git a/security/RELEASE_ARCHIVE_MERKLE_REBRIDGE_GUIDE.md b/security/RELEASE_ARCHIVE_MERKLE_REBRIDGE_GUIDE.md deleted file mode 100644 index 72d9f38d12b94fa7ec37e2473e910e35b6f42d3c..0000000000000000000000000000000000000000 --- a/security/RELEASE_ARCHIVE_MERKLE_REBRIDGE_GUIDE.md +++ /dev/null @@ -1,81 +0,0 @@ -# Run 167 — recursive archive-Merkle authority re-bridge - -Run 166 makes one Run 165 rotated or recovered authority usable for later RFC6962 epochs. -Run 167 closes the next continuity gap: a **second or later** log/gossip authority change -must be bound to the newest accepted post-handoff checkpoint, not back to Run 164's older -checkpoint. The result is a recursive rotate/recover → append lifecycle. - -## Recursive trust boundary - -The first Run 167 event starts from the exact canonical `trusted-archive-merkle-continuity-state.json` -produced by Run 166. Later Run 167 events start from the exact canonical prior Run 167 -trusted state. Every transition binds: - -- the exact source trusted-state SHA-256; -- the latest Merkle sequence and cumulative continuity head; -- every latest per-log tree size/root/checkpoint; -- the previous authority-bridge head; -- the complete current and next log/gossip authority maps; and -- the permanent revoked-key set. - -A transition can therefore inherit only the actually accepted latest trees. - -## Scheduled rotation — old + new - -A normal scheduled rotation uses the independently pinned Run 165 governance-root quorum. -Each replacement log/gossip key signs the re-bridge handoff. Every changed authority also -requires the **old + new** log and gossip keys to sign the same exact handoff subject, whose -`priorCheckpoint` is the newest Run 166/Run 167 checkpoint. - -After the handoff passes, the same transaction appends the next RFC6962 leaf under the new -key and verifies a real consistency proof from that prior checkpoint. - -## Compromise recovery - -**Compromise recovery** uses the separately pinned Run 165 recovery-root quorum and pinned -recovery-channel diversity. Every affected log rotates both its log and gossip keys. -Old-key signatures are forbidden in recovery, while the replacement keys still sign the -exact latest checkpoint they inherit. Replaced/compromised fingerprints remain permanently -revoked and cannot be reintroduced by a later recursive re-bridge. - -## Append-only epochs between rotations - -Run 167 does not force key churn. After one re-bridge, later health-anchor epochs may be -**append-only** under the same accepted authority. Their log and gossip challenges bind the -current authority-bridge head, active-authority SHA-256, and exact previous trusted-state -SHA-256. A later rotation or recovery then re-bridges from that newest append-only state. - -## Two cumulative heads - -Run 167 intentionally maintains two independent cumulative heads: - -1. `archiveMerkleRebridgeAuthorityHeadSha256` advances only when authority changes; and -2. `merkleRebridgeContinuityHeadSha256` advances for every RFC6962 append. - -This makes an authority transition independently auditable without losing the complete -append history that occurred between transitions. - -## Offline verification - -The receipt preserves the exact Run 165 and Run 166 documents, every Run 163 anchor document, -all transition and handoff signatures, and all log/gossip responses. Verification replays -accepted historical signatures at their covered time, verifies every inclusion/consistency -proof again, reconstructs every intermediate trusted state byte-for-byte, and checks that the -newest event exactly matches the supplied current Run 163 output. - -## Fail-closed rules - -- first Run 167 event must be a governed re-bridge; -- Merkle sequence must advance exactly one epoch at a time; -- re-bridge sequence advances only on authority transitions; -- transition IDs and transition times are monotonic/non-reusable; -- scheduled rotation requires old + new handoff signatures; -- compromise recovery forbids old signatures; -- revoked keys cannot become active again; -- witness/anchor/control authority planes remain separated; -- adapter set must exactly equal the active log set; -- input authority is fingerprinted before parsing and rehashed before commit; -- output is server-owned, allowlisted, canonical, staged, and replay-verified before rename. - -Run 167 contains no signing private keys. Provider-specific Merkle log and gossip clients -remain external bounded adapters. diff --git a/security/RELEASE_ARCHIVE_MERKLE_RECOVERY_GUIDE.md b/security/RELEASE_ARCHIVE_MERKLE_RECOVERY_GUIDE.md deleted file mode 100644 index 9f66b21ba6d54f4f13678310171efaa02c008684..0000000000000000000000000000000000000000 --- a/security/RELEASE_ARCHIVE_MERKLE_RECOVERY_GUIDE.md +++ /dev/null @@ -1,71 +0,0 @@ -# Run 168 — recursive Merkle authority recovery durability - -Run 168 preserves the complete accepted Run 167 recursive Merkle-authority state as one -canonical recovery checkpoint and replicates that exact checkpoint across mutually -distrustful immutable archives. It is a disaster-recovery layer, not another authority -transition mechanism. - -## Trust boundary - -Preservation MUST first run the complete Run 167 verifier against the live predecessor -roots and release evidence. Only after that verifier succeeds may -`preserve_archive_merkle_rebridge_history.py` construct -`release-archive-merkle-recovery-checkpoint.json`. - -The checkpoint contains the exact four canonical Run 167 output documents plus a compact -projection of: - -- the current Merkle sequence and re-bridge sequence; -- `archiveMerkleRebridgeAuthorityHeadSha256`; -- `merkleRebridgeContinuityHeadSha256`; -- the complete active log/gossip authority and its SHA-256; -- the permanent revoked-key fingerprint set; and -- the newest RFC6962 checkpoint for every configured log. - -The archive membership is itself part of the deterministic checkpoint. Archive writers -and read-only verifiers MUST use distinct identities and independent operators. Every -accepted copy MUST be create-only/present-only, use an approved immutable storage class, -and pass exact remote SHA-256 + size read-back through the separate verifier plane. - -## Recovery - -Recovery MUST be configured with at least three read-only archive sources and normally -requires at least two independent observed operators. Availability quorum is not fork -resolution: **any observed byte disagreement is equivocation and fails closed**, even when -a numerical quorum agrees on another copy. - -Recovery also requires out-of-band pins for all of the following: - -1. recovery checkpoint SHA-256; -2. Run 167 authority-chain head; -3. Run 167 Merkle-continuity head; -4. active-authority SHA-256; and -5. Run 167 sequence. - -These pins prevent unanimous rollback to an older but otherwise valid checkpoint. - -A successful recovery writes the exact original Run 167 documents under -`recovered-run167/` and additionally emits -`recovered-active-archive-merkle-authority.json`. That compact file is sufficient for a -new environment to discover the current log/gossip public authority, permanent -revocations, last accepted RFC6962 checkpoints, and both cumulative heads without any -intermediate online service state. - -## What Run 168 does not authorize - -Run 168 recovery does not rotate keys, append Merkle leaves, clear revocations, change log -membership, or rewrite Run 167 history. Any subsequent authority change must again use -the governed Run 165/167 transition rules. - -## Operational rules - -- Keep the five rollback pins outside the archive providers being recovered. -- Use at least three configured archive locations across independent operators/failure - domains. -- Treat a single conflicting observed source as a security incident rather than outvoting - it. -- Keep archive-writer and read-only-verifier credentials separated. -- Do not persist cloud credentials, local paths, private signing keys, HSM handles, or - recovery secrets in the checkpoint. -- Preserve the Run 168 checkpoint hash beside the release record so recovery can be - validated even if every original online service is gone. diff --git a/security/RELEASE_ARCHIVE_WITNESS_GUIDE.md b/security/RELEASE_ARCHIVE_WITNESS_GUIDE.md deleted file mode 100644 index 1dde90cb4a8f172c69a21cf524e6571b97eb5819..0000000000000000000000000000000000000000 --- a/security/RELEASE_ARCHIVE_WITNESS_GUIDE.md +++ /dev/null @@ -1,82 +0,0 @@ -# Run 162 — Archive-health witnessing and retention-root recovery - -Run 161 proves provider-signed retention and an independent auditor read-back for every -active durable archive. Run 162 adds two independent trust planes around that health -state: continuous cross-auditor witnessing and break-glass retention root recovery. - -## 1. Cross-auditor witness log - -A deployment creates an **out-of-band** pinned `archive-health-witness-root`. It contains -at least three Ed25519 witness keys and a threshold of at least two operators. The root is -self-threshold-signed and its canonical SHA-256 must be retained outside the release -workspace. - -For each witness epoch the tool derives a challenge from the previous witness-chain head, -the exact Run 161 health-chain head, and the canonical archive view. Every participating -witness must be read-only, must not reuse archive/provider/retention-governance -credentials, and signs the exact same view. - -The **witness quorum** is not majority-based conflict resolution. Two matching witnesses -may satisfy availability quorum when a third witness is unavailable, but any observed -**split view** fails closed. A provider therefore cannot safely present one immutable -version, retention expiry, legal-hold state, or archive membership to one witness and a -different view to another. - -Each epoch embeds the canonical Run 161 trusted state and active evidence used for that -observation. Older epochs are replayed historically from those preserved bytes; the -newest epoch must match the currently supplied Run 161 output. This allows the witness -history to advance across later Run 161 audit or migration epochs without retroactively -requiring old provider observations to remain fresh. - -Persistent witness output is: - -- `release-archive-witness-bundle.json` -- `trusted-archive-witness-state.json` -- `active-archive-witness-evidence.json` -- `release-archive-witness-receipt.json` - -The output is create-only and canonical. Private keys, credentials, and local paths are -never accepted as evidence. - -## 2. Retention root recovery - -A compromised or unavailable Run 161 retention-governance root is recovered through a -separate **out-of-band** pinned `archive-retention-recovery-root`. The recovery root must -have at least three keys, an M-of-N threshold of at least two operators, and at least two -cryptographically pinned recovery channels. - -The recovery subject binds: - -- the exact old retention-root SHA-256; -- the exact replacement retention-root SHA-256; -- the current Run 161 health-chain head; -- the exact active archive IDs; -- the explicitly compromised old key IDs; -- the exact selected recovery signer IDs; and -- `retirementAuthorizedArchiveIds: []`. - -**Retention root recovery never authorizes retirement.** It cannot remove or migrate an -archive. It only produces a recovered Run 161-format retention root plus a canonical -recovery record. A later ordinary Run 161 membership transaction under the newly pinned -root is the only path that may authorize migration/retirement. - -Recovery keys/operators are separated from both the old and replacement retention-root -authorities. Compromised old keys cannot reappear in the replacement root. The -replacement root must satisfy its own threshold signature verification before recovery is -accepted. - -Recovery output is: - -- `recovered-retention-root.json` -- `retention-root-recovery-record.json` -- `retention-root-recovery-receipt.json` - -The recovered root is not silently trusted: operators must retain its exact SHA-256 as the -new independent pin before subsequent use. - -## Failure policy - -Fail closed on any witness disagreement, stale current witness observation, signature or -challenge mismatch, root-pin mismatch, quorum/operator/channel failure, Run 161 state -mutation, compromised-key reintroduction, authority-plane overlap, non-empty retirement -list in a recovery subject, duplicate JSON key, unexpected output file, or input drift. diff --git a/security/RELEASE_ATTESTATION_LIFECYCLE_GUIDE.md b/security/RELEASE_ATTESTATION_LIFECYCLE_GUIDE.md deleted file mode 100644 index 3f3bf86d7961a3a5895e91fa58b0d491ebc925d8..0000000000000000000000000000000000000000 --- a/security/RELEASE_ATTESTATION_LIFECYCLE_GUIDE.md +++ /dev/null @@ -1,143 +0,0 @@ -# Run 158 — Attestation Lifecycle, Revocation, and Vendor Semantics - -Run 158 extends Run 157's X.509 hardware-key attestation with the lifecycle evidence that -plain certificate-chain validation does not provide: status freshness, revocation time, -attestation-CA rotation, device-identity continuity, and vendor/device semantics that are -cryptographically bound into the attestation certificate. - -Run 158 does **not** replace Run 157. The complete Run 157 continuity output is embedded -and replayed first. Run 158 then decides whether those already cryptographically verified -attestations were acceptable at release time and whether their currently active keys are -still acceptable for future use. - -## Trust planes - -The release root and the certificate-status authority remain separate: - -```text -Run 157 active recovered root - | - +-- M-of-N signatures --> attestation CA set - | CA DER + SHA-256 - | device-class profile policy - | status-authority public keys - | CA rotation/revocation state - | - +-- status M-of-N --> short-lived status snapshot - every attestation certificate - good / revoked - effective revocation time -``` - -The status-authority key IDs and operators are disjoint from the active release-root role. -No status private key is stored in the repository or lifecycle output. - -## Vendor semantic claim - -Every Run 157 attestation leaf used by this gate contains the non-critical private X.509 -extension OID configured by `release_attestation_lifecycle_policy.toml`. Its value is -canonical JSON covered by the CA certificate signature and binds: - -- the required profile for the attestation's device class; -- `hardwareBacked=true`; -- `nonExportable=true`; -- manufacturer and model; -- the exact Run 157 root key ID; -- SHA-256 of the exact attested root public key; and -- a stable, non-secret `deviceIdHash` for anti-substitution continuity. - -The default profiles are `hsm-x509-v1`, `kms-hsm-x509-v1`, `tpm-x509-v1`, and -`secure-element-x509-v1`. They are validation contracts, not marketing labels. A device -class cannot satisfy the gate with another profile name. - -This profile verifies cryptographically CA-bound semantics. Production deployments may -map the private extension to evidence produced by a vendor HSM, TPM attestation service, -KMS-HSM service, or enterprise attestation CA. Run 158 does not claim that arbitrary text -saying "HSM" proves a physical device. - -## Certificate status - -`attestation-status` is a short-lived canonical document signed by the current CA set's -independent status-authority threshold. It binds the exact Run 157 continuity chain head, -exact CA-set bytes, monotonically increasing status version, issue/next-update times, and -one status entry for every certificate used by every preserved Run 157 root attestation. - -A revocation entry carries an effective `revokedAt` time and reason. Revocation is sticky: -a later status snapshot cannot change a revoked certificate back to `good` or alter its -recorded revocation. - -The distinction between historical and live verification is deliberate: - -```text -revokedAt <= attestedAt - -> release proof was already invalid; historical verification fails - -attestedAt < revokedAt <= now - -> historical release proof remains valid - -> currently active use fails closed -``` - -Thus later device compromise does not erase the historical fact that a release used a -then-valid hardware-backed key, while that compromised key cannot continue authorizing -new releases. - -## CA rotation - -The attestation CA set is itself threshold-signed by the active Run 157 release-root role. -CA-set versions advance exactly one step and bind the exact previous CA-set hash. - -A `scheduled-rotation` must account for every removed CA as retired and cannot mark a CA -revoked. A `compromise-recovery` must explicitly revoke at least one removed CA. Every -removed CA is accounted for, and a CA once marked compromised/revoked cannot be -reintroduced by a later lifecycle state. - -The deterministic bundle preserves every prior CA set and status snapshot, so historical -verification does not depend on the currently deployed CA service. - -## Outputs - -Successful initialization or advancement emits exactly: - -```text -release-attestation-lifecycle-bundle.json -trusted-attestation-lifecycle-state.json -active-attestation-ca-set.json -active-attestation-status.json -release-attestation-lifecycle-receipt.json -``` - -The bundle embeds the complete canonical Run 157 output, CA-set history, status history, -and hash-chain events. The trusted state carries the active CA/status versions, cumulative -revoked CA set, device bindings, bundle hash, and lifecycle chain head. - -## Offline verification - -Use `verify_attestation_lifecycle.py verify` with the same independently retained Run 155 -bootstrap-root, Run 156 recovery-root, and Run 157 attestation-root SHA-256 pins. The -verifier replays Run 157 historically, every CA-set root threshold, every status threshold, -vendor claims, revocation timing, CA rotation, and lifecycle hash chain. - -Use `--historical` when auditing an old release after the current status/CA freshness -windows have expired. Historical mode still verifies signatures and effective revocation -time; it does not weaken a revocation that was already effective when the attestation was -created. - -## Production rules - -- Keep release-root and status-authority private keys in separate operational domains. -- Prefer multiple status-authority operators and short status `nextUpdate` windows. -- Archive every accepted lifecycle bundle alongside the release evidence. -- Treat a live active-key revocation as a release-signing stop condition. -- Treat CA compromise as an explicit root-threshold-governed CA transition, never a local - config edit. -- Keep vendor/device identifiers hashed or otherwise non-secret; do not store serials, - credentials, HSM handles, private keys, recovery material, or local source paths. - -## Native status follow-on (Run 159) - -Run 158 intentionally normalizes certificate status under a threshold status authority. -Where release policy requires proof from native revocation mechanisms, that normalized -snapshot is not the final evidence authority. Run 159 (`verify_native_status_provenance.py`) -recomputes the decision from preserved DER CRL and OCSP evidence, requires source -agreement, and binds the result back to this exact Run 158 lifecycle. See -`RELEASE_NATIVE_STATUS_GUIDE.md`. diff --git a/security/RELEASE_DELEGATED_TRUST_GUIDE.md b/security/RELEASE_DELEGATED_TRUST_GUIDE.md deleted file mode 100644 index 4b7b8b575f0da46b5f43a1a98586f1dd8046ec5f..0000000000000000000000000000000000000000 --- a/security/RELEASE_DELEGATED_TRUST_GUIDE.md +++ /dev/null @@ -1,174 +0,0 @@ -# Delegated release freshness and cryptographic root recovery - -Run 156 sits above the Run 155 cryptographic governance seal. It adds two deliberately -separate trust planes: - -1. short-lived `snapshot` + `timestamp` metadata for rollback/freeze resistance; and -2. an independently pinned recovery root for break-glass replacement of compromised or - unavailable **root-role** keys. - -`maintain_release_trust.py` never accepts a private key. All signatures are detached -Ed25519 values created by offline, HSM, or KMS/HSM signers outside the repository. - -## Freshness flow - -```text -Run 155 sealed directory - ↓ -verify complete Run 155 seal + bootstrap pin - ↓ -root-threshold-signed release-delegation-root.json - snapshot role: M-of-N - timestamp role: M-of-N - role key/operator separation - explicit signer profiles + key expiry - ↓ -release-snapshot.json - exact hashes + sizes of every Run 155 seal artifact - ↓ -release-timestamp.json - exact snapshot version/hash/size/expiry - ↓ -monotonic previous metadata state/bundle - root-chain continuity - version rollback/skip rejection - ↓ -trusted-delegated-metadata-state.json -release-delegated-metadata-bundle.json -release-delegated-metadata-receipt.json -``` - -The timestamp is intentionally shorter lived than the snapshot. A live acceptance check -rejects an expired/near-expiry timestamp, snapshot, delegation root, materially -future-dated metadata, a timestamp issued before its snapshot, or a timestamp whose -expiry extends beyond the snapshot expiry. - -Snapshot and timestamp roles are disjoint. Their public keys, identities, operators, -expiry, and `signerProfile` are themselves covered by the Run 155 root-role threshold. -A delegated key must remain valid through the delegation-root lifetime. - -## Exact Run 155 binding - -The snapshot body contains the exact SHA-256 and size of all six canonical Run 155 seal -artifacts: - -- `active-root.json`; -- `cryptographic-governance-authorization.json`; -- `release-governance-recovery-snapshot.json`; -- `release-root-bundle.json`; -- `release-root-receipt.json`; and -- `trusted-release-root-state.json`. - -The timestamp then binds the exact snapshot bytes. Changing either the Run 155 seal or -snapshot therefore invalidates the delegated metadata. - -## Cross-refresh continuity - -A previous `trusted-delegated-metadata-state.json` and -`release-delegated-metadata-bundle.json` are explicit trust inputs. Snapshot and -Timestamp versions advance exactly one step. A delegation-root version may stay the same -or advance one version; keeping the same version with different bytes is rejected. - -The prior effective Run 155 root version/hash must still exist in the current root chain, -and the prior root-chain head is recomputed from that prefix. This detects a root-history -fork instead of accepting a numerically newer root version from an unrelated chain. - -## Recovery root is independently pinned - -Emergency recovery does **not** trust the potentially compromised Run 155 root role. -A separate canonical `recovery-root` envelope is accepted only when: - -- its complete SHA-256 equals an out-of-band recovery-root pin; -- its Ed25519 self-signatures satisfy the recovery M-of-N threshold; -- its operators are distinct from every active Run 155 authority operator; -- its key IDs do not overlap active Run 155 authority keys; -- its keys remain valid through the recovery-root lifetime; and -- at least one configured recovery signer has a hardware-oriented signer profile when - policy requires it. - -The recovery root is not stored as a secret. It contains public keys only. - -## Multi-channel recovery ceremony - -Each recovery public key has a `recoveryChannel` value inside the independently pinned, -self-signed recovery root. A signer cannot satisfy channel diversity by merely typing a -new channel name into an approval: the signature's channel must equal that key's pinned -channel. - -A canonical recovery subject binds: - -- governance ID and incident ID; -- exact previous Run 155 root version/hash/root-chain head; -- exact replacement-root version/hash; -- exact recovery-root version/hash; -- the complete compromised-root-key set; and -- the deterministic selected recovery-key set. - -The selected recovery signers create fresh Ed25519 signatures over their own canonical -approval records, which include the recovery-subject hash, identity, operator, pinned -channel, pinned signer profile, decision, and signing time. Run 156 requires threshold, -operator, and channel quorum simultaneously. - -## What recovery may change - -Recovery is intentionally narrower than normal governance. The replacement root must be -exactly the next root version, must satisfy its **new-root self-threshold**, and may -replace the root role only. The governance and emergency role membership/key metadata -must remain byte-identical to the previous trusted Run 155 root. - -Every declared compromised root key is permanently listed in the recovery record and -may not appear in the replacement root role. Recovery-authority keys/operators are also -forbidden from becoming replacement-root keys/operators, preventing the recovery council -from silently installing itself as the new release root. - -This narrow rule means a root compromise cannot be used as a shortcut to change policy -authority. Governance changes still require the normal governance path. - -## Signer profiles - -The policy currently understands `offline`, `hsm`, and `kms-hsm` profiles. The profile is -cryptographically bound to a public key by the delegation or recovery root and is useful -for enforcing operational separation. - -A profile label is **not** by itself a vendor hardware-attestation proof. Run 156 does -not claim that a string saying `hsm` proves a key resides in hardware. Production systems -should bind these profiles to independently verified vendor/HSM attestation evidence. -That deeper attestation proof is a separate future boundary rather than being faked by a -boolean field. - -## Offline verification - -`maintain_release_trust.py verify` replays the Run 155 seal, root-authorized delegation, -snapshot and timestamp signatures, exact artifact bindings, state/bundle chain, and -receipt. `--historical` verifies signatures at their cryptographically covered issuance -time rather than treating later expiry as retroactive invalidation. - -`maintain_release_trust.py verify-recovery` replays the independently pinned recovery -root, replacement-root new-root threshold, exact recovery subject, signer identities, -pinned channels/profiles, threshold/operator/channel quorum, effective-root rebinding, -and receipt. Historical recovery verification likewise evaluates the ceremony at its -signed time. - -## Input integrity - -Both refresh and recovery hash every authority input before output construction and rehash -those exact files immediately before commit. That includes the Run 155 sealed directory, -delegation/snapshot/timestamp inputs, previous state/bundle when supplied, recovery root, -replacement root, and all recovery signatures. Any drift fails closed. - -## Private-key handling - -Private keys, seeds, HSM handles, cloud signing credentials, recovery secrets, access -URLs, and local source paths are not valid evidence and are never emitted. Only canonical -public metadata, detached Ed25519 signatures, hashes, sizes, identities, operator labels, -pinned channel labels, and signer-profile classifications are persisted. - -## Run 157 continuation - -Run 156 recovery deliberately does not rewrite the Run 155 root bundle because its -break-glass transition is not signed by the compromised old-root threshold. Run 157 turns -that accepted recovery into a separate cumulative continuity chain. The replacement root -becomes the authorizing root for later governance and the old root for later normal -rotations. Run 157 also replaces signer-profile-only hardware claims with directly -verified X.509 key-attestation evidence rooted in independently pinned attestation CAs. -See `RELEASE_ROOT_CONTINUITY_GUIDE.md`. diff --git a/security/RELEASE_EVIDENCE_GUIDE.md b/security/RELEASE_EVIDENCE_GUIDE.md deleted file mode 100644 index 334d1a38a6ff21f53b2f7b39e78804444863247d..0000000000000000000000000000000000000000 --- a/security/RELEASE_EVIDENCE_GUIDE.md +++ /dev/null @@ -1,396 +0,0 @@ -# Production release evidence — B39 - -B38 made the source release path reproducible. B39 prevents **evidence substitution**: -a scan, SBOM or attestation is not accepted merely because a file named “scan” exists. -Every production promotion must supply one short-lived `release-evidence.json` that -binds the evidence to the exact repository lock/SBOM and to one immutable final OCI -image digest. - -## Trust boundary - -The evidence manifest contains **no secrets and no deployment identity**. Do not put -Redis URLs, hostnames, usernames, tokens, passwords, registry credentials, request -samples, user content, IP addresses, or capability values in it. Evidence artifacts -stay beside the manifest and are referenced only by relative path + SHA-256. - -The verifier rejects path traversal, symlinks, stale/expired manifests, source-hash -drift, artifact tampering, a provenance subject that does not match the final image, -missing signature-verification evidence, unsafe infrastructure logging attestations, -unverified Redis lifecycle properties, and risk-exception bypasses. - -## Release flow - -```text -exact source revision + whole extension tree hash + Python SBOM - | - v -networked dependency scan - | - v -linux/amd64 image build -> immutable image digest - | | - | +--> full image CycloneDX SBOM - | +--> image vulnerability scan - | +--> SLSA/in-toto provenance - | +--> signature verification - v -sanitized Redis + signed Redis 7/8 chaos attestations - | - v -infrastructure logging evidence - | - v -release-evidence.json (<= 72 h validity) - | - v -python security/verify_release_gate.py release-evidence.json - | - +--> GREEN: promotion may continue - `--> RED: fail closed -``` - -SLSA provenance is expected as an in-toto Statement v1 whose predicate type is -`https://slsa.dev/provenance/v1` and whose subject SHA-256 is the exact final image -digest. Signature verification remains a separate artifact because provenance JSON -alone does not prove who signed it. - -## Redis evidence without credential leakage - -`probe_redis_authority.py` is explicit opt-in and receives only the **name** of the -environment variable that contains a Redis URL: - -```bash -python security/probe_redis_authority.py \ - --plane share \ - --url-env SHARE_STORE_REDIS_URL \ - --output redis-share-observation.json -``` - -For the shared provider-artifact lifecycle authority, probe the dedicated plane separately: - -```bash -python security/probe_redis_authority.py \ - --plane providerArtifactLifecycle \ - --url-env PROVIDER_ARTIFACT_LIFECYCLE_REDIS_URL \ - --output redis-provider-artifact-lifecycle-observation.json -``` - -It never accepts the URL as a CLI argument and never emits hostname, port, username, -credentials, keys, values, replication offsets, or persistence timestamps. It uses -`PING`, `ACL WHOAMI` where permitted, and bounded `INFO persistence` / `INFO replication` -observations. These observations do **not** paper-prove least privilege, provider -persistence guarantees, backup retention, or successful restores; those remain -operator/provider evidence. - -For Share and Contribution lifecycle authority, production evidence additionally -requires persistence, replication, and a successful backup/restore exercise no older -than 90 days. Provider-artifact lifecycle Redis requires TLS, a non-default -least-privilege identity, and replication, but deliberately does **not** require -persistence/backup evidence: this plane is ephemeral metadata authority, and losing it -fails old generated candidates closed instead of reconstructing write capability. Rate -limiting likewise needs TLS and non-default least-privilege identity but is not falsely -classified as durable user-data storage. - -## Infrastructure logging / telemetry rule - -Production promotion fails unless operators attest that request bodies, -Authorization headers, management-capability headers, query strings, WAF body -capture, APM body capture, and third-party telemetry export are all disabled for the -assistant service. This is independent of browser feedback consent: infrastructure -logging must never become a hidden telemetry bypass. - -## What B39 still cannot prove locally - -The verifier validates **binding and policy**, not the truth of external scanner or -provider claims. The release system must itself be trusted, scanners must run against -the final artifact, and signature/provenance verification must be performed by the -approved CI/registry trust root. Keep the raw external evidence according to your -security retention policy; do not embed it in the application image. -## Schema-v2 fail-closed parsing - -The production verifier treats the evidence document as a security protocol, not -as an extensible metadata bag. Unknown root or nested schema-v2 fields are -rejected. The manifest itself is capped at 256 KiB, referenced evidence files -are separately bounded, and every required evidence artifact declares a -non-empty bounded tool name/version. The release `proxyVersion` must equal the -actual runtime source constant. - -Full-image CycloneDX evidence must be version 1.6 or newer. The resolved -platform manifest digest must be distinct from the pinned multi-platform index -digest; copying the index digest into the resolved-manifest field is rejected. -Runtime source binding includes every regular file copied from `_utils/`, not -only Python files, so future runtime policy/data files cannot silently escape -the source subject. - - -## Run 148 signed Redis-chaos provenance - -Schema v2 binds production promotion to the exact extension source tree, not only -the proxy runtime subset. `sourceTreeSha256` covers every regular release source -file with its relative path, Unix mode, byte length, and content; test caches and -Python bytecode are excluded. `sourceRevision` records the 40- or 64-hex source -revision asserted by the trusted CI system. - -Redis lifecycle promotion additionally requires four canonical chaos attestations: -Redis 7 standalone, Redis 7 cluster, Redis 8 standalone, and Redis 8 cluster. Each -attestation must be fresh, report `pass`, bind the exact `sourceRevision` and -`sourceTreeSha256`, and name the exact digest-pinned Redis image from -`release_evidence_policy.toml`. - -The canonical chaos JSON does not sign itself. A parent CI trust root must -cryptographically attest/verify that JSON and emit a sanitized paired -`*.signature-verification.json` record only after verification succeeds. The local -release verifier then binds that record to the exact chaos-attestation SHA-256 and -requires `signerIdentityVerified=true`. GitHub artifact attestations are one supported -reference flow; its verification step constrains the exact source digest/ref, signer -workflow + signer digest, and rejects self-hosted-runner attestations before the -sanitized verification record is emitted. CircleCI or -other systems may use an organization-approved Sigstore, KMS, HSM, or equivalent -signer/verifier with equivalent source/workflow identity binding. - -The local verifier deliberately does not perform network signature discovery. Its job -is fail-closed binding of already verified external evidence to the exact source tree -and release. The signing/verifying CI trust root remains an independently protected -release-system responsibility. - -## Run 149 promotion transaction - -Schema-v2 evidence is now an input to `_hf_spaces_proxy/security/promote_release.py`, -not the final publication decision. `prepare` snapshots the exact verified tree, -re-applies the supplied baseline-to-current patch, builds and verifies the release ZIP, -and emits a canonical release statement binding ZIP, patch, baseline and SBOM hashes. -After an external release trust root signs/verifies that statement, `finalize` verifies -every subject again—including a second patch application—and atomically emits a -`promotion-receipt.json`. Publishing must consume only the receipt allowlist and must -never rebuild release artifacts after finalization. See `RELEASE_PROMOTION_GUIDE.md`. - -## Run 150 publication transaction - -After Run 149 finalization, publication is no longer a free-form CI copy step. -`security/publish_release.py` validates the canonical `promotion-receipt.json`, binds it -back to the signed release statement and signature-verification record, snapshots each -authorized object, then invokes an externally protected publisher adapter with -create-only + remote-readback requirements. The resulting publication transparency -binds the exact promotion receipt and remote object hashes. See -`RELEASE_PUBLICATION_GUIDE.md`. - -## Run 151 signed post-publication transparency - -Run 150 publisher read-back is necessary but is not the final trust root. -`security/finalize_publication.py prepare` re-validates the complete publication -directory and asks a distinct read-only verifier identity to re-read every published -object. It emits a canonical in-toto statement whose subject is the exact -`publication-transparency.json` SHA-256. Parent CI must cryptographically sign/verify -that statement and constrain the signer identity before creating the sanitized -signature-verification record. `finalize` validates that record, performs another -independent remote read-back after signature verification, and create-only binds the -deterministic `release-publication-record.json` to the original publication target. -See `RELEASE_TRANSPARENCY_GUIDE.md`. - -## Run 152 append-only transparency and threshold witnessing - -`security/witness_publication.py` treats Run 151's exact -`release-publication-record.json` as the immutable transparency subject. It requires a -pinned previous checkpoint, append-only log insertion, a distinct read-only primary -verifier that verifies checkpoint signature + inclusion + consistency, and a threshold -of at least two witness identities across at least two distinct operators observing the -same checkpoint. After quorum, it create-only anchors the deterministic -`release-transparency-witness-record.json` back to the original release target and -read-backs exact bytes. See `RELEASE_WITNESS_GUIDE.md`. - -## Run 153 durable release-history evidence - -`security/preserve_release_history.py` turns the Run 152 witnessed checkpoint into one -step of a persistent release-history chain. The previous canonical history state and -bundle are explicit trust inputs; the Run 152 previous checkpoint must equal that -trusted state exactly. An N-of-M set of independently operated read-only replicas then -gossips the prior history head and current checkpoint. A contradictory available view -fails closed even when a numerical majority agrees. - -The resulting deterministic `release-history-bundle.json` carries genesis trust -metadata and every accepted release entry, including release/publication/source -identity, witnessed subject hashes, exact checkpoint/log-key identity, replica quorum, -and any sanitized key-rotation or compromise-recovery record. A compact -`trusted-history-state.json` pins the exact bundle hash, chain head, active key, -checkpoint, and cumulative revoked-key set. Both can be verified without network -access. The bundle is then create-only replicated with remote read-back to at least two -independent immutable archives. See `RELEASE_HISTORY_GUIDE.md`. - -## Run 154 threshold governance and disaster-recovery evidence - -Run 154 adds a governance evidence plane above Run 153's durable history. The externally -pinned `governance-genesis.json` establishes epoch-0 membership and thresholds without -trust-on-first-use. Each later `governance-transition-proposal.json` binds the previous -governance-state hash, exact Run 153 history head, complete next policy, selected signer -keys, and permanent revocations. Canonical approval verification records are embedded in -the offline `release-governance-bundle.json`, so a policy transition can be replayed and -validated without contacting the original approval service. - -`trusted-governance-state.json` is the compact current trust root. The deterministic -`release-governance-recovery-snapshot.json` contains that state, the full governance -bundle, and the exact Run 153 state+bundle; independent immutable archives store this -snapshot create-only with remote read-back. `release-governance-receipt.json` records the -archive evidence without changing the deterministic governed bytes. - -For local history loss, `release-history-recovery-receipt.json` records which independent -read-only archives were observed or unavailable and the exact recovered state, bundle, -and chain-head hashes. Conflicting available archives are evidence of possible -split-view/history rewrite and stop recovery rather than being outvoted. - -## Run 155 cryptographic release-root evidence - -Run 155 adds a cryptographic acceptance layer above the Run 154 governance candidate. -The accepted evidence set is intentionally self-contained and public-key only: - -- `cryptographic-governance-authorization.json` contains the exact canonical subject plus - the threshold Ed25519 signatures from the Run 154 selected governance/emergency keys; -- `release-root-bundle.json` contains the complete versioned root-metadata chain, - including bootstrap and every dual-authorized old-root/new-root rotation; -- `trusted-release-root-state.json` binds the current root version/hash/expiry and root - chain head to the exact accepted governance state, bundle, proposal, and recovery - snapshot; -- `active-root.json` is the exact current public root metadata envelope; -- `release-governance-recovery-snapshot.json` is copied byte-for-byte from the validated - Run 154 candidate; and -- `release-root-receipt.json` rebinds the exact SHA-256 and size of every Run 155 output. - -The bootstrap root SHA-256 is intentionally **not inferred from those files**. It remains -an independent trust input and is required by both live sealing and offline verification. -`seal_release_governance.py verify-seal` replays the root chain, root rotation thresholds, -Run 154 governance bundle, embedded Run 153 history bundle/state, and every governance -authorization signature. Later metadata expiry does not erase historically valid evidence, -but live sealing rejects an expired or near-expiry root as a freeze-risk condition. - -No private key, signing seed, HSM credential, cloud signer credential, or recovery secret -is valid release evidence. Only public Ed25519 key material and signatures are persisted. - -## Run 156 delegated freshness and recovery evidence - -Run 156 adds `release-delegation-root.json`, `release-snapshot.json`, and -`release-timestamp.json` above the complete Run 155 seal. Delegation metadata is signed -by the active root threshold; snapshot/timestamp signatures are directly Ed25519-verified -against disjoint delegated roles. `trusted-delegated-metadata-state.json` and -`release-delegated-metadata-bundle.json` carry monotonic versions plus a chain head that -rebinds previous effective-root continuity. - -Root compromise uses separate evidence: an out-of-band pinned/self-thresholded recovery -root, a canonical recovery subject, fresh M-of-N recovery signatures whose channels and -signer profiles are pinned in that recovery root, a self-thresholded replacement root, -`release-root-recovery-record.json`, and `release-root-recovery-receipt.json`. Declared -compromised root keys are explicit revocations and cannot reappear in the replacement -root. Governance/emergency roles must remain unchanged during this break-glass path. - -Only public keys/signatures and sanitized authority metadata are durable evidence; private -keys, HSM handles, cloud credentials, secrets, and local paths are excluded. - -## Run 159 native status-source evidence - -Run 159 adds a self-contained native certificate-status evidence plane above Run 158. -`release-native-status-bundle.json` embeds the exact Run 158 predecessor and each verified -CRL/OCSP source with its original DER bytes, SHA-256, issuer/responder binding, update -window, and per-certificate decision. `active-native-status-evidence.json` is the current -canonical event, while `trusted-native-status-state.json` binds the cumulative native -status chain head and exact bundle hash. - -The bundle is intentionally sufficient for offline replay: an auditor can reparse the -preserved CRL/OCSP bytes and reverify signatures without contacting the original status -service. Run 159 does not convert source disagreement into a majority result; any -verified conflict is retained as a release-blocking condition. Optional vendor-native -evidence additionally preserves the exact raw evidence and the SHA-256 of the explicit -profile verifier executable that produced its canonical bound result. See -`RELEASE_NATIVE_STATUS_GUIDE.md`. - -## Run 160 durable native-evidence archive - -Run 160 adds `release-native-evidence-archive.json`, a deterministic archival subject that -contains the complete canonical Run 159 output and an explicit inventory of every preserved -native source. `trusted-native-evidence-archive-state.json` binds that artifact to the exact -Run 159 native-status chain head, while `release-native-evidence-archive-receipt.json` -records create-only archive and independent remote-readback evidence. - -The archive artifact is sufficient to reconstruct the four original Run 159 files and -re-run native CRL/OCSP verification offline. Recovery requires multiple independent -read-only archive views; any observed byte disagreement is treated as equivocation. An -out-of-band archive SHA-256 plus Run 159 chain-head pin prevents unanimous rollback to an -older otherwise-valid archive artifact. See `RELEASE_NATIVE_ARCHIVE_GUIDE.md`. - -## Run 161 archive-health evidence - -Run 161 adds a cumulative `release-archive-health-bundle.json` and compact -`trusted-archive-health-state.json` above the Run 160 native-evidence archive. Every event -contains a threshold-authorized membership plus provider-signed immutable-version and -retention semantics. The separate receipt preserves exact independent-auditor read-back -responses so offline replay can prove both provider and reader authority after those -services disappear. - -`active-archive-health-evidence.json` is the only artifact that grants retirement -authorization, and only after the complete next membership has passed current retention -and read-back checks while preserving the policy minimum durable-copy count. See -`RELEASE_ARCHIVE_HEALTH_GUIDE.md`. - -### Run 163 anchored archive-health evidence - -Preserve `release-archive-anchor-bundle.json`, `trusted-archive-anchor-state.json`, `active-archive-anchor-evidence.json`, and `release-archive-anchor-receipt.json` together with the signed anchor plan and its out-of-band root pin. For break-glass witness-root recovery also preserve the replacement root, recovery record, receipt, and recovery-root pin. The anchor bundle is designed for offline replay of signed channel/observer evidence even if the original transparency channels later disappear. - -## Run 164 Merkle transparency evidence - -Preserve `release-archive-merkle-bundle.json`, `trusted-archive-merkle-state.json`, -`active-archive-merkle-evidence.json`, and `release-archive-merkle-receipt.json` together -with the canonical independently pinned transparency root. The receipt preserves exact -signed log checkpoints, RFC6962 inclusion/consistency proofs, gossip observations, and the -canonical Run 163 predecessor documents used to derive each leaf. This is sufficient for -offline cryptographic proof replay after the original transparency services disappear. -See `RELEASE_ARCHIVE_MERKLE_GUIDE.md`. - -## Run 165 Merkle log-authority evidence - -Preserve `release-archive-log-authority-bundle.json`, -`trusted-archive-log-authority-state.json`, `active-archive-log-authority.json`, and -`release-archive-log-authority-receipt.json` together with the independently pinned -Run 165 governance and recovery roots. The receipt preserves exact Run 164 predecessor -bytes plus every per-log handoff signature, so an offline auditor can prove which key set -was authorized to extend each accepted Merkle tree after scheduled rotation or compromise -recovery. Replaced log/gossip key fingerprints are permanently revocation-bound in the -cumulative state. See `RELEASE_ARCHIVE_LOG_AUTHORITY_GUIDE.md`. - -## Run 166 evidence - -A Run 166 checkpoint preserves the exact Run 164 and Run 165 authority artifacts plus each post-handoff Run 163 leaf, log checkpoint, inclusion/consistency proof, and gossip observation. The cumulative continuity head is rooted in the last Run 164 Merkle consensus head and binds the exact Run 165 authority-chain head and active-authority hash. - -## Run 167 recursive Merkle authority evidence -Preserve `release-archive-merkle-rebridge-bundle.json`, `trusted-archive-merkle-rebridge-state.json`, `active-archive-merkle-rebridge.json`, and `release-archive-merkle-rebridge-receipt.json` together with the pinned Run 165 roots and exact Run 166 predecessor. The receipt retains every transition/handoff signature, Run 163 document, and RFC6962 log/gossip response needed for offline replay. - -## Run 168 recursive Merkle recovery evidence - -Preserve `release-archive-merkle-recovery-checkpoint.json`, -`trusted-archive-merkle-recovery-state.json`, and -`release-archive-merkle-recovery-receipt.json` together. The deterministic checkpoint -contains the exact four Run 167 artifacts plus the current log/gossip authority, permanent -revocations, last RFC6962 checkpoints, authority-chain head, and Merkle-continuity head. -The receipt preserves exact immutable-archive and independent read-back responses. - -For disaster recovery, separately retain the checkpoint SHA-256, authority head, Merkle -head, active-authority SHA-256, and Run 167 sequence as out-of-band rollback pins. A -successful recovery reproduces the original Run 167 byte set and emits -`recovered-active-archive-merkle-authority.json` for cold-start authority reconstruction. -See `RELEASE_ARCHIVE_MERKLE_RECOVERY_GUIDE.md`. - -## Run 169 hermetic replay evidence - - -Release evidence for Runs 149–168 is replayed against a deterministic synthetic clock -lineage rather than the executor's current date. Preserve the Run 169 regression result -with the release test evidence so future cold verification can distinguish immutable test -vectors from production live-time checks. Production freshness, expiry, and freeze-risk -validation remain governed by the runtime clock unless an explicit verifier `now=` is -supplied. - -## Run 170 hermetic process evidence - -Preserve the Run 170 regression result with deterministic release-test evidence. It proves -that Run 149's synthetic promotion patch is independent of inherited Git state, production -`git apply` ignores ambient PATH/GIT configuration and parent secrets, and every -release-security adapter subprocess receives an explicit minimal environment. Record the -configured absolute Git pin when `SCIKITPLOT_RELEASE_GIT_EXECUTABLE` is used operationally; -do not record credential-bearing parent environment variables because they are not part of -the authorized subprocess contract. diff --git a/security/RELEASE_GOVERNANCE_GUIDE.md b/security/RELEASE_GOVERNANCE_GUIDE.md deleted file mode 100644 index 2b8752992e821d08f6c625ffa52a9cecbc64f0c2..0000000000000000000000000000000000000000 --- a/security/RELEASE_GOVERNANCE_GUIDE.md +++ /dev/null @@ -1,212 +0,0 @@ -# Threshold release governance and disaster recovery - -Run 154 moves release-history trust changes out of a single policy-authority identity and -into an explicit, versioned, threshold-governed trust plane. It does not sign approvals -or own private keys. External signing/verification systems produce canonical approval -verification records; `govern_release_history.py` only accepts them after exact proposal, -history-head, epoch, identity, operator, key, and revocation rebinding. - -The same tool provides a fail-closed disaster-recovery path for the Run 153 trusted -history state. Recovery is accepted only when an N-of-M quorum of independent immutable -archives returns identical canonical history state and bundle bytes. An available -conflicting archive is treated as evidence of a fork, not as a minority vote to ignore. - -## Trust flow - -```text -externally pinned governance genesis SHA-256 - ↓ -Run 153 trusted history state + offline bundle - ↓ -initialize governance epoch 0 - ↓ -canonical transition proposal - ├─ exact previous governance-state SHA-256 - ├─ exact history sequence/state/bundle/head - ├─ next policy version and full membership - ├─ selected threshold signer keys - └─ permanent authority-key revocations - ↓ -external approval verification records - ↓ -normal change: current policy-authority M-of-N -compromise: emergency-authority M-of-N - ↓ -offline-verifiable release-governance-bundle.json - + -trusted-governance-state.json - ↓ -self-contained release-governance-recovery-snapshot.json - ↓ -≥2 independent immutable governance archives - ↓ -create-only remote read-back -``` - -## No silent trust-on-first-use - -The first governance state is created only from a canonical `governance-genesis.json` -whose SHA-256 is supplied separately as `--expected-genesis-sha256`. The tool rejects a -genesis whose bytes do not match that external pin. - -The genesis defines the initial policy-authority set, emergency-recovery authority set, -Run 153 gossip-replica membership, Run 153 immutable-history-archive membership, and all -thresholds. Run 154 rebinds the replica and archive membership to the actual Run 153 -history evidence before epoch 0 can be accepted. - -## Threshold-governed policy transitions - -Every transition proposal is canonical and advances exactly one governance epoch and -one policy version. It is bound to: - -- the previous `trusted-governance-state.json` SHA-256; -- the exact current Run 153 history sequence; -- the exact trusted-history-state SHA-256; -- the exact release-history-bundle SHA-256; -- the exact Run 153 history chain head; -- the complete next authority, replica, archive, and threshold configuration; -- the exact selected approver key IDs; -- any authority keys that become permanently revoked. - -Approval verification records are themselves canonical. Each one names the proposal -SHA-256, transition ID, signer identity/operator/key, role, signing timestamp, and an -external signature-verification evidence hash. The repository never needs the private -key. - -The selected approval set is part of the proposal. This is intentional: a retry cannot -silently substitute another quorum and therefore cannot generate different deterministic -bundle bytes merely because additional approvers became reachable later. - -## Normal changes versus authority compromise - -`scheduled-change` and `membership-change` require the current policy-authority -threshold. - -`authority-compromise-recovery` is deliberately separate. It requires the independently -configured emergency authority threshold and must revoke at least one key belonging to -the current policy authority. A normal policy quorum cannot authorize its own compromise -recovery path. Revoked authority keys are accumulated in the governance history and may -never re-enter a later policy. - -This design does not claim that repository-side boolean `signatureVerified` is a -cryptographic verifier. That assertion must originate from the deployment's external, -independently protected signature-verification system and is bound here by canonical -evidence hash and signer identity. - -## Membership and threshold versioning - -The complete policy is history-bound, not mutable ambient configuration. Each epoch -records: - -- policy-authority members and M-of-N threshold; -- emergency-authority members and M-of-N threshold; -- history gossip-replica members and quorum threshold; -- history archive members and recovery threshold. - -A membership or threshold change therefore requires the same authorized transition as a -root-key change. Operators cannot silently edit deployment configuration and have the -new topology treated as equivalent history. - -## Disaster recovery from independent immutable archives - -`recover-history` asks at least three configured recovery sources for the canonical Run -153 `trusted-history-state.json` and `release-history-bundle.json` bytes. At least two -independent operators must agree. - -Each source must assert: - -- read-only authority; -- no history-writer credential reuse; -- no governance credential reuse; -- immutable archive read-back was independently verified; -- canonical bytes, not a locally reconstructed equivalent, were returned. - -Run 154 independently executes Run 153's full offline history validation on every -returned pair. Recovery then requires identical state hash, bundle hash, and history -chain head across the observed quorum. - -One source may be unavailable if quorum remains. Any available source returning a -conflicting valid history causes `GOVERNANCE_RECOVERY_ARCHIVE_DISAGREEMENT` and the -recovery stops. This is fail-closed fork detection, not majority-wins conflict repair. - -Successful recovery emits: - -```text -recovered-trusted-history-state.json -recovered-release-history-bundle.json -release-history-recovery-receipt.json -``` - -## Self-contained governance recovery snapshot - -A successful governance transition creates one deterministic -`release-governance-recovery-snapshot.json` containing: - -- the new trusted governance state; -- the complete offline governance bundle; -- the exact Run 153 trusted history state; -- the complete Run 153 release-history bundle. - -At least two distinct governance archive operators must store that exact snapshot with -create-only semantics and mandatory remote read-back. Archive adapters may not reuse -governance-writer or history-writer credentials. Remote locators must be unique and may -not contain credentials, query strings, fragments, traversal, or local file paths. - -This snapshot is the intended disaster-recovery unit for future governance recovery: it -preserves both the trust policy and the release-history evidence that policy governs. - -## Offline verification - -`govern_release_history.py verify` replays the complete governance bundle from its -externally pinned genesis policy. It checks epoch continuity, proposal and approval -binding, authority role, threshold membership, operator diversity, history rollback, -policy-version advancement, compromise-recovery requirements, permanent revocations, -and the governance chain head. - -The compact `trusted-governance-state.json` must match the resulting policy, epoch, -revoked-key set, bundle hash, and governance chain head. - -## Adapter boundaries - -Recovery and archive adapters receive canonical JSON over stdin and must return one -bounded JSON object on stdout. Stdout and stderr are bounded while they are produced; -adapter execution has a hard timeout. Adapters receive only the local path of the -server-owned staged recovery snapshot when a create-only archive bind is required. - -A bind is followed by a separate verify operation and exact remote read-back. The local -snapshot is hashed before and after adapter execution so a provider cannot mutate it -while claiming success. - -## Residual trust boundary - -Run 154 does not solve human/operator collusion, compromise of enough policy or emergency -keys to satisfy their threshold, compromise of enough independent recovery archives to -return the same forged history, or compromise of the external cryptographic -signature-verification system itself. Those remain deployment trust assumptions and -should be separated across organizations, hardware-backed identities, and access-control -planes where possible. - -## Recovery rollback pins - -Archive quorum is necessary but not sufficient to recover trust. Both `recover-history` -and `recover-governance` require an out-of-band expected sequence/epoch **and** expected -chain-head SHA-256. A set of archives that all agree on an older valid snapshot therefore -cannot silently roll trust backward merely by agreeing with each other. - -`recover-governance` validates the complete archived governance snapshot, offline-verifies -the governance bundle/state, independently offline-verifies the embedded Run 153 history -bundle/state, and then rebinds the governance state's history fields to those exact -embedded history bytes. Successful recovery emits recovered governance and history state -and bundle files plus `release-governance-recovery-receipt.json`. - -## Run 155 acceptance boundary - -Run 154 constructs and preserves a governance candidate, but its external -`signatureVerified: true` approval records are no longer sufficient to make that -candidate an accepted trust root. Run 155's `seal_release_governance.py` rebinds the -complete Run 154 output and requires direct Ed25519 threshold signatures from the exact -selected governance/emergency key IDs under a pinned, versioned release-root chain. - -Accordingly, production release policy should treat Run 154 output as **candidate state** -until the Run 155 cryptographic seal succeeds. Authority membership changes also require -a dual-threshold old-root/new-root rotation. See `RELEASE_ROOT_GUIDE.md`. diff --git a/security/RELEASE_HISTORY_GUIDE.md b/security/RELEASE_HISTORY_GUIDE.md deleted file mode 100644 index bddfe886c0248cde47c2d7d0139ba1e0e6168029..0000000000000000000000000000000000000000 --- a/security/RELEASE_HISTORY_GUIDE.md +++ /dev/null @@ -1,244 +0,0 @@ -# Release history durability and equivocation detection - -Run 153 moves release provenance from a sequence of individually valid releases to a -continuously verified history. Its subject is the complete witnessed Run 152 output, -and its trust root is the exact `trusted-history-state.json` plus -`release-history-bundle.json` accepted from the previous release. - -The goal is not another publication transport. The goal is to make a later rewrite, -fork, key substitution, replay, or disappearance of the original transparency service -detectable from independently retained evidence. - -## Trust flow - -```text -previous trusted-history-state.json -previous release-history-bundle.json - | - +---------------- exact previous checkpoint - | exact previous chain head - v -Run 152 witnessed release - | - v -validate all Run 152 hashes/evidence sidecars - | - v -N-of-M history replica observations - - read-only - - independent gossip source - - no log/publisher/history-writer credential reuse - - exact previous history head - - exact previous checkpoint - - exact current checkpoint - - authenticated current log key - | - +---- any conflicting observed view -> FAIL CLOSED - | - v -optional explicit log-key transition - scheduled-rotation OR compromise-recovery - | - v -append deterministic history entry - | - +--> release-history-bundle.json - `--> trusted-history-state.json - | - v - >= 2 independent immutable archives - create-only + exact remote read-back - | - v - release-history-preservation-receipt.json -``` - -Run 153 never asks a replica to choose which fork is correct. If an available replica -reports a checkpoint, previous checkpoint, history head, or current log-key identity -that disagrees with the expected view, the release stops. Unavailable replicas may be -tolerated only while the configured N-of-M quorum and minimum operator count remain -satisfied. - -## Previous history is an explicit trust input - -There is no silent trust-on-first-use bootstrap. Production must begin with an -organization-approved canonical genesis bundle and matching trusted state. The genesis -pins the transparency-log identity, initial checkpoint, initial active log key, release -history policy authority, and a SHA-256 of the out-of-band bootstrap evidence. - -A valid previous state binds the exact previous bundle SHA-256 and the deterministic -history chain head. Run 153 also requires the Run 152 -`previous-transparency-checkpoint.json` to equal that state's checkpoint byte-for-byte. -A skipped release, stale state file, rollback, or alternate branch therefore cannot be -silently advanced. - -## Offline history bundle - -`release-history-bundle.json` is canonical, deterministic, and self-contained for the -history policy enforced by this repository. It contains the genesis trust statement -metadata and every accepted release entry in sequence order. Each entry binds: - -- release ID, publication ID, and source revision; -- the final publication-record, witnessed-record, and witness-receipt hashes; -- exact log ID, log-key ID, previous checkpoint hash, accepted checkpoint hash, and - checkpoint tuple; -- the pinned replica membership/operators and threshold governing that history step; -- the complete sanitized key-transition verification record when a log key changes. - -The bundle chain head is recomputed as a SHA-256 hash chain beginning at the canonical -genesis object and then consuming every canonical entry. The compact -`trusted-history-state.json` pins the resulting bundle SHA-256, sequence, chain head, -active log key, checkpoint, policy authority, and revoked-key set. - -Offline verification requires no network access: - -```bash -python _hf_spaces_proxy/security/preserve_release_history.py verify \ - --bundle release-history-bundle.json \ - --state trusted-history-state.json -``` - -This validates canonical JSON, sequence continuity, monotonic checkpoints, accepted -checkpoint digests, replay protection, replica-quorum structure, key-transition rules, -revoked-key history, bundle hash, and chain head. It cannot independently redo a -provider-specific checkpoint signature or Merkle proof after the provider and its -cryptographic material have disappeared; those cryptographic decisions are preserved -as already verified, hash-bound release evidence. - -## Cross-release gossip and N-of-M agreement - -Production should configure at least three independently operated read-only history -replicas and require a quorum of at least two. Each available replica must bind the -exact previous bundle hash and chain head before reporting the current log view. The -replica also asserts that it verified the current checkpoint signature, consistency -from the previous trusted checkpoint, and the previous history head using an -independent gossip source. - -A replica response is evidence, not authority to rewrite local history. An explicit -`unavailable` response can be tolerated. A contradictory response cannot. This means a -2-of-3 deployment tolerates one outage, but a 2-versus-1 fork still stops rather than -letting the majority automatically erase evidence of equivocation. - -Operator labels are enforced separately from identities. Configuring three workload -identities controlled by one mutable account does not satisfy the minimum independent -operator rule. - -## Log-key rotation and compromise recovery - -The previous trusted state pins exactly one active transparency-log key and a cumulative -set of revoked keys. A changed key reported by the replica quorum is rejected unless an -explicit canonical `key-transition.json` is supplied. - -A transition binds the history ID, log ID, old key, new key, effective tree size, -policy-authority identity, transition reason, and external cryptographic evidence hash. -The previous key is always revoked after a successful transition; rollback to any -revoked key is rejected by both online advancement and offline bundle verification. - -Two reasons are allowed by default: - -`scheduled-rotation` requires policy-authority signature verification, old-key -continuity verification, new-key proof verification, and no emergency override. - -`compromise-recovery` requires policy-authority signature verification, new-key proof -verification, and an explicit emergency-recovery authorization. Old-key continuity may -be unavailable when the old key is the compromised authority, but that exception is -visible and permanently recorded in the history bundle. - -The repository does not hold those private keys. Production CI/HSM/KMS/Sigstore-style -infrastructure performs the cryptographic verification and emits the sanitized -transition record consumed here. - -## Replay and rollback protection - -An advancing transparency tree alone is insufficient. Run 153 also rejects reuse of an -already recorded release ID, publication ID, or witnessed-record subject at a later -history sequence. This prevents a valid newer checkpoint from being used to re-admit -an older release as though it were new. - -The current Run 152 previous checkpoint must equal the exact previous trusted-history -checkpoint. The current accepted checkpoint must advance its tree size. A current key -may not appear in the cumulative revoked-key set. - -## Independent durable archives - -After quorum succeeds, Run 153 creates deterministic bundle and state bytes before any -archive operation. The exact `release-history-bundle.json` is then sent to at least two -independently identified archive operators. Each archive must be create-only, reject -overwrite, and perform exact SHA-256 + size read-back. Archive locators must be unique. - -Examples include object-lock storage, content-addressed stores, immutable release -assets, or versioned create-only storage. Archive credentials must not reuse the -history-writer or transparency-log credentials. - -The deterministic bundle makes interrupted retries safe: an archive may return -`present` only for the same exact bundle bytes. The bundle records pinned configured -replica membership and its threshold, while variable observed/unavailable responses and -fresh timestamps stay in local receipt/evidence files. A replica recovering between -retries therefore cannot perturb the bundle or trusted state; a recovered replica that -reports a conflicting view still fails the retry closed. Fresh archive verification -timestamps likewise remain outside deterministic history bytes. - -## Adapter protocols - -Replica and archive adapters are provider-neutral JSON programs. They receive one -canonical request on stdin and return one bounded JSON object on stdout. Output is -bounded while produced, execution has a hard timeout, and adapter subprocesses receive -a deliberately minimal environment. - -The `advance` CLI uses explicit `IDENTITY@OPERATOR=EXECUTABLE[,ARG...]` configuration -for both replicas and archives so quorum membership cannot be discovered or substituted -implicitly at runtime. - -```bash -python _hf_spaces_proxy/security/preserve_release_history.py advance \ - --witnessed-dir /secure/run152-witnessed \ - --previous-state /secure/history/trusted-history-state.json \ - --previous-bundle /secure/history/release-history-bundle.json \ - --output-dir /secure/history/run153 \ - --collector-identity ci/history-collector \ - --replica-quorum 2 \ - --replica 'replica-a@operator-a=/secure/bin/history-replica-a' \ - --replica 'replica-b@operator-b=/secure/bin/history-replica-b' \ - --replica 'replica-c@operator-c=/secure/bin/history-replica-c' \ - --archive 'archive-a@operator-x=/secure/bin/history-archive-a' \ - --archive 'archive-b@operator-y=/secure/bin/history-archive-b' -``` - -Add `--key-transition /secure/verified/key-transition.json` only when the independently -observed log key changed. - -## Successful output - -A successful directory contains the newly accepted bundle/state, their previous exact -inputs, the current Run 152 witness/checkpoint records needed for audit, every replica -observation, every archive bind/read-back result, an optional key-transition record, -and `release-history-preservation-receipt.json` binding all evidence hashes. - -The next release must use the exact emitted `trusted-history-state.json` and -`release-history-bundle.json` as its previous-history authority. Do not regenerate, -compact, reorder, or normalize them between releases. - -## Residual trust boundary - -Run 153 provides durable cross-release continuity, quorum gossip, rollback/replay -protection, explicit key lifecycle rules, deterministic offline history, and independent -archive replication. It does not make operator-independence claims cryptographically -true by itself and it does not reimplement each transparency provider's signature or -Merkle-proof algorithm. Those remain external trust roots whose verified identities and -results are strictly rebound here. - -For stronger deployments, distribute replicas and archives across separate -organizations, cloud accounts, networks, and administrative principals. Periodically -compare archived bundle hashes outside the release pipeline. Any disagreement should -open an incident and freeze further history advancement rather than selecting a branch -automatically. - -## Run 154 governance handoff - -Run 153 deliberately records the durable release-history facts but does not itself grant -one process unilateral authority to redefine the policy-authority root, gossip topology, -or archive topology. Run 154 consumes the exact Run 153 state and bundle and makes those -changes versioned, history-bound, and threshold-governed. See -`RELEASE_GOVERNANCE_GUIDE.md` for normal M-of-N transitions, the separate compromise -recovery council, permanent authority-key revocation, immutable governance snapshots, -and multi-archive disaster recovery. diff --git a/security/RELEASE_NATIVE_ARCHIVE_GUIDE.md b/security/RELEASE_NATIVE_ARCHIVE_GUIDE.md deleted file mode 100644 index 0b6e340740a668ed3bd4cb0035e6cebe1ac0fb65..0000000000000000000000000000000000000000 --- a/security/RELEASE_NATIVE_ARCHIVE_GUIDE.md +++ /dev/null @@ -1,109 +0,0 @@ -# Run 160 — Native Status Evidence Archival and Recovery - -Run 159 proves native CRL/OCSP status and preserves the exact raw bytes inside a canonical -bundle. Run 160 makes that evidence durable across service loss or archive compromise by -replicating the complete Run 159 output to independently operated immutable archives and -requiring a second, read-only verifier authority to read each archived object back. - -## Trust model - -Run 160 has three separate authority planes: - -1. **archive writers** — create-only stores for the exact deterministic archive artifact; -2. **independent archive verifiers** — read-only identities that re-read each remote object - and verify its exact SHA-256 and size; and -3. **recovery readers** — read-only sources used later to reconstruct the Run 159 evidence. - -Archive and verifier operators must be disjoint. The default policy requires at least two -archives controlled by at least two archive operators and at least two independent verifier -operators. Archive writer credentials, Run 159 status credentials, and verifier credentials -must not be reused across those planes. - -## Deterministic archival subject - -`release-native-evidence-archive.json` is the only archival subject. It contains: - -- the complete canonical four-file Run 159 output; -- the exact Run 159 sequence and native-status chain head; -- an inventory of every preserved CRL, OCSP responder certificate, and vendor-native - evidence hash/size; and -- the configured archive + independent-verifier membership. - -The artifact intentionally contains no remote locator, observation timestamp, local path, -credential, private key, or retry-dependent result. Therefore create-only retries address -exactly the same bytes. - -`trusted-native-evidence-archive-state.json` is a compact binding to the archive artifact, -source-inventory hash, Run 159 sequence, and native-status chain head. Operational archive -and verifier results are kept separately in `release-native-evidence-archive-receipt.json`. - -## Immutable replication - -Each archive adapter must provide create-only semantics, `overwrite=false`, an allowed -immutability class, and mandatory remote read-back of the exact artifact SHA-256 and size. -A retry may return `present` only for those exact bytes. - -A separate verifier identity then reads that same remote locator. The verifier must be -read-only and must independently assert exact hash/size verification. Archive and verifier -identity/operator reuse is rejected. - -Locators are sanitized before persistence. Query strings, fragments, user-info, file URLs, -and path traversal are forbidden. Two archive identities may not collapse onto the same -remote locator. - -## Recovery and anti-equivocation - -Recovery requires an out-of-band pin for both: - -- the exact `release-native-evidence-archive.json` SHA-256; and -- the expected Run 159 native-status chain head. - -At least two independently operated read-only recovery sources must return canonical archive -payload bytes. Every available observed source must agree byte-for-byte. A numerical quorum -does not hide an observed conflict: - -```text -archive A = H1 -archive B = H1 -archive C = H2 - -result: FAIL — equivocation -``` - -If one source is unavailable, two agreeing independent sources may satisfy the default -quorum. If all available sources agree on an old artifact, the out-of-band artifact/hash -and chain-head pins reject the rollback. - -After agreement, Run 160 reconstructs the exact four Run 159 files and runs the Run 159 -offline verifier again. Historical recovery verifies evidence at its covered time; live -recovery additionally preserves current Run 159 freeze protection. - -## Offline survivability - -The archived subject contains all raw DER CRLs, OCSP responses, responder certificates, and -vendor evidence that Run 159 already preserved. Recovery therefore does not depend on the -original OCSP responder, CRL distribution point, vendor API, or status service remaining -online. - -## Operational rules - -- Archive the exact Run 159 output; never rebuild CRL/OCSP evidence from a provider summary. -- Keep archive writers and independent remote readers under distinct operators/credentials. -- Use create-only immutable storage and verify remote bytes after every bind. -- Retain independent out-of-band pins for the archive artifact and Run 159 chain head. -- Treat any available recovery disagreement as equivocation, not as a vote. -- Do not persist local paths, access tokens, private keys, HSM handles, signed URLs, query - secrets, or provider credentials in Run 160 evidence. - -## Run 161 retention lifecycle handoff - -Run 160 establishes that independent immutable copies existed and were read back at -publication time. Long-lived deployments SHOULD feed that exact Run 160 output into -`audit_archive_retention.py` rather than treating the original `immutability` string as a -permanent durability proof. - -Run 161 threshold-authorizes the exact Run 160 archive membership, cryptographically binds -storage-provider retention/version keys and independent auditor keys, periodically -re-challenges each current copy, and creates explicit migration/retirement authorization -only after the complete successor durable set is proven. See -`RELEASE_ARCHIVE_HEALTH_GUIDE.md`. diff --git a/security/RELEASE_NATIVE_STATUS_GUIDE.md b/security/RELEASE_NATIVE_STATUS_GUIDE.md deleted file mode 100644 index f88b5ca5e51a98f4e58d28117255b4240f02e1f0..0000000000000000000000000000000000000000 --- a/security/RELEASE_NATIVE_STATUS_GUIDE.md +++ /dev/null @@ -1,140 +0,0 @@ -# Run 159 — Native status provenance and anti-equivocation - -Run 159 treats the Run 158 threshold-signed certificate-status snapshot as a policy -assertion, not as native revocation evidence. `verify_native_status_provenance.py` -independently parses and verifies the raw status artifacts that a relying party would -actually consume: DER CRLs and DER OCSP responses. The exact raw bytes are preserved in -the offline bundle. - -## Trust boundary - -```text -Run 158 attestation lifecycle - ↓ -replay complete Run 158 evidence - ↓ -attestation leaf inventory + issuer CA binding - ↓ -raw DER CRL raw DER OCSP - issuer signature responder signature - CRL Number issuer name/key hash - thisUpdate / nextUpdate responder certificate - revocation entries thisUpdate / nextUpdate - \ / - \ / - exact per-leaf decisions - ↓ - anti-equivocation comparison - CRL == OCSP == Run 158 - ↓ - raw offline evidence bundle -``` - -There is no majority-wins path. If any available verified native source contradicts -another source, Run 159 fails closed even if the remaining sources agree. A native -consensus that contradicts Run 158 also fails; Run 159 cannot silently rewrite the -accepted lifecycle assertion. - -## CRL verification - -For each supplied DER CRL the verifier: - -- binds the issuer to exactly one active Run 158 attestation CA; -- verifies the CRL signature in process; -- requires a CRL Number by default; -- validates `thisUpdate` / `nextUpdate` ordering and bounded lifetime; -- rejects future or freeze-risk live CRLs; -- derives `good`/`revoked` decisions for every attestation leaf issued by that CA; and -- preserves the exact DER bytes, SHA-256, CRL number, update window, issuer CA hash, and - normalized decisions. - -Across Run 159 advances, CRL Number and `thisUpdate` must both advance monotonically for -the same issuer. - -## OCSP verification - -For each DER OCSP response the verifier: - -- requires `SUCCESSFUL` response status and one certificate response; -- rebinds serial number plus OCSP issuer-name/key hashes to the exact attestation leaf - and issuer CA; -- requires a responder name by default; -- verifies either direct CA responder authority or a delegated responder certificate; -- for delegated responders, validates issuer signature, OCSP-signing EKU, digital - signature key usage, and certificate validity at production time; -- verifies the OCSP response signature in process; -- validates `producedAt`, `thisUpdate`, `nextUpdate`, and bounded freshness; and -- preserves both the exact OCSP DER and the exact responder certificate DER/hash. - -Across refreshes, `thisUpdate` must move forward for a given responder/certificate pair. - -## Multi-source requirement - -The default policy requires both `crl` and `ocsp` source kinds for every active -attestation leaf. This is intentionally stricter than accepting one native source. - -```text -CRL says good OCSP says good Run 158 says good PASS -CRL says good OCSP says revoked FAIL -CRL says revoked OCSP says revoked Run 158 says good FAIL -missing OCSP FAIL -``` - -For a revocation decision, the effective revocation timestamp must also agree. Reason -labels are preserved but are not allowed to override a timestamp/status conflict. - -## Offline evidence - -Successful output contains only four canonical files: - -- `release-native-status-bundle.json` -- `trusted-native-status-state.json` -- `active-native-status-evidence.json` -- `release-native-status-receipt.json` - -The bundle embeds the complete canonical Run 158 predecessor and every exact CRL/OCSP -byte string needed to reproduce the decision without contacting the original responder. -Historical verification replays signatures and status at the evidence's covered time; -live verification still enforces current freeze windows. - -The native-status chain binds every refresh to the previous chain head. A later refresh -also enforces CRL-number and OCSP-time monotonicity. - -## Vendor-native evidence adapters - -Run 159 exposes an optional profile-specific verifier adapter for TPM/HSM/KMS/native -vendor evidence. There are **no permissive built-in vendor verifiers**. If a manifest -names a profile without a configured executable verifier, the transaction fails closed. - -An adapter receives canonical input containing the exact raw-evidence bytes/hash and -expected key binding. Successful evidence records preserve: - -- the exact raw vendor evidence; -- its SHA-256; -- the verifier executable SHA-256; -- verifier identity; and -- the canonical result bound to the expected key/public-key hash. - -This is an integration boundary, not a claim that every vendor format is understood by -the repository. Production deployments should pin and independently control the exact -profile verifier binary, ideally a vendor or standards implementation. - -## Operational rules - -- Do not fetch OCSP/CRL inside the verifier; acquisition happens outside the trusted - transaction and the resulting raw bytes are explicit inputs. -- Do not accept JSON summaries in place of native DER CRL/OCSP evidence. -- Do not discard responder certificates or raw status bytes after normalization. -- Do not resolve observable status conflicts by voting. -- Preserve the exact Run 158 pins when running Run 159. -- Keep private keys, responder credentials, HSM handles, client TLS secrets, and local - source paths out of persistent evidence. - -## Run 160 durable preservation - -Run 159 proves and canonicalizes native status, but local evidence alone is not a durability -boundary. After Run 159 succeeds, use `archive_native_status_evidence.py` to replicate the -complete four-file output to independent immutable archives and have separate read-only -verifier operators re-read each remote object. Recovery requires exact multi-source byte -agreement plus independently retained archive and native-chain-head pins. See -`RELEASE_NATIVE_ARCHIVE_GUIDE.md`. diff --git a/security/RELEASE_PROCESS_HERMETICITY_GUIDE.md b/security/RELEASE_PROCESS_HERMETICITY_GUIDE.md deleted file mode 100644 index 1f9a89bf02a4b1b34627b36d47ce0568e8aa837f..0000000000000000000000000000000000000000 --- a/security/RELEASE_PROCESS_HERMETICITY_GUIDE.md +++ /dev/null @@ -1,81 +0,0 @@ -# Run 170 — hermetic external-tool release authority - -Run 170 removes executor-process state from release-security subprocess authority. It closes -both sides of the boundary: the synthetic Git patch used by the Run 149 replay fixture and -the production external programs invoked by the release-security tools. - -## Synthetic Git authority - -Run 149 synthesizes a binary Git patch to exercise the real promotion boundary. Before -Run 170, those test-only Git invocations inherited the executor's `GIT_*` variables, -user/system Git configuration, attributes, external diff helpers, templates, and locale. -Identical project bytes could therefore fail or produce different patch evidence solely -because of CI/workstation state. - -The fixture now launches Git with a minimal allowlisted environment: - -- no inherited `GIT_*` control variables; -- system configuration disabled; -- an explicit empty global configuration; -- system attributes disabled; -- isolated HOME/XDG and empty template directories; -- `LC_ALL=C` / `LANG=C`; -- prompting and pagers disabled; and -- explicit `core.autocrlf=false`, `core.filemode=true`, and `core.ignorecase=false`. - -## Production `git apply` authority - -Production promotion replay is independently hardened. `promote_release.py` no longer -resolves Git from the ambient `PATH` and no longer gives Git the parent process environment. - -By default it resolves `git` only from the platform's system-default executable path. A -deployment that needs another Git binary may set `SCIKITPLOT_RELEASE_GIT_EXECUTABLE` to an -**absolute executable path**. Relative pins fail closed. - -Each `git apply --check` / `git apply` invocation receives an isolated HOME/XDG directory, -an empty global Git configuration, disabled system configuration/attributes, fixed C -locale, and no parent `PATH`, `GIT_*`, cloud token, provider token, or other ambient secret. -The patch is still validated and replayed by real Git; Run 170 does not replace the -production patch engine with a mock. - -## External adapter subprocesses - -Every release-security `subprocess.Popen` / `subprocess.run` site now supplies an explicit -process environment. Provider-neutral publisher, verifier, witness, archive, gossip, and -recovery adapters receive only: - -- the platform default executable search path (`os.defpath`); -- `LC_ALL=C`; and -- `LANG=C`. - -They do **not** inherit arbitrary parent environment variables. This prevents an adapter -from receiving unrelated Hugging Face, GitHub, cloud, CI, signing, database, or other -process secrets merely because those variables exist in the proxy/release executor. - -Operators should pass an absolute executable when an adapter is outside the platform -default path. Adapter authentication should use a deliberately scoped mechanism owned by -that adapter (for example a dedicated credential file/socket, workload identity endpoint, -mTLS identity, KMS/HSM agent, or an explicit wrapper), not ambient parent-process secrets. - -## Regression gates - -Run 170 proves the boundary with both dynamic and static tests: - -1. hostile `GIT_EXTERNAL_DIFF`, `GIT_DIR`, `GIT_WORK_TREE`, config-count overrides, global - Git configuration, and non-C locale cannot alter Run 149 patch bytes; -2. production patch replay succeeds under the same hostile Git environment; -3. a fake `git` placed first in ambient `PATH` is not executed by production promotion; -4. parent cloud/provider secret canaries are absent from the production Git environment; -5. a command adapter exits successfully only when a parent secret is absent, proving the - secret does not cross the subprocess boundary; and -6. every release-security subprocess call is statically required to provide an explicit - environment, and non-Git adapter calls must use the fixed platform-default path. - -## Scope - -Run 170 changes release-engineering subprocess authority only. It does not relax artifact -hashes, source-tree verification, signature checks, transparency proofs, quorum rules, -freshness/expiry rules, rollback pins, archive immutability, or user-facing runtime API -behavior. The change is intentionally fail-closed: a deployment that depended on arbitrary -parent environment leakage into a release adapter must replace that implicit dependency -with an explicit scoped credential mechanism. diff --git a/security/RELEASE_PROMOTION_GUIDE.md b/security/RELEASE_PROMOTION_GUIDE.md deleted file mode 100644 index 828b881857067178c7627cd00e8bb9ba021a6d66..0000000000000000000000000000000000000000 --- a/security/RELEASE_PROMOTION_GUIDE.md +++ /dev/null @@ -1,128 +0,0 @@ -# Run 149 — transactional release promotion - -Run 148 proves that production evidence belongs to one exact source tree and revision. -Run 149 closes the next time-of-check/time-of-use gap: the bytes that are packaged and -published must be the same bytes that were verified. - -## Authority flow - -```text -schema-v2 release evidence - | - v -promote_release.py prepare - | - +-- verify evidence against the live tree - +-- snapshot the exact source tree (path + mode + bytes) - +-- safely extract the previous packaged ZIP - +-- apply the supplied Git patch - +-- require patched baseline == verified snapshot - +-- build and independently verify deterministic release ZIP - +-- bind ZIP + patch + baseline ZIP + SBOM references - `-- emit canonical release-statement.json - | - v - external trust root - signs + verifies statement - | - v -external verifier output + sanitized verification record - | - v -promote_release.py finalize - | - +-- verify schema-v2 evidence again - +-- verify live tree == statement source subject - +-- verify external verifier-output hash - +-- verify signature record subject/revision/freshness - +-- verify ZIP/patch hashes and sizes - +-- re-apply patch to the baseline ZIP again - +-- require patched baseline == current verified tree - +-- re-check Python + image SBOM references - +-- verify release ZIP contents + Unix modes - `-- atomically emit immutable promotion directory + receipt -``` - -`finalize` is the promotion decision. Publishing automation must publish **only** the -filenames and SHA-256 values listed in `promotion-receipt.json`; it must never rebuild -the ZIP or patch after finalization. - -Run 150 makes this rule executable: pass the finalized promotion directory to -`publish_release.py`; do not loop over the directory or construct a new upload list in -CI. See `RELEASE_PUBLICATION_GUIDE.md`. - -## Prepare - -Run from the extension source tree while the fresh schema-v2 evidence is still valid: - -```bash -python _hf_spaces_proxy/security/promote_release.py prepare \ - --evidence /secure/evidence/release-evidence.json \ - --baseline-zip /secure/baselines/scikitplot__sphinx_ai_assistant_run148.zip \ - --patch /secure/staging/scikitplot__sphinx_ai_assistant_run148_to_run149.patch \ - --output-dir /secure/staging/run149-prepared -``` - -The output directory must be outside the source tree and must not already exist. -`prepare` does not accept a private key and does not claim that the statement is signed. - -## External signing and verification - -Cryptographically sign and verify `release-statement.json` using the organization -release trust root. Keep the actual verifier output outside the source tree. After that -verification succeeds, emit the sanitized binding record: - -```bash -python _hf_spaces_proxy/security/promote_release.py signature-record \ - --statement /secure/staging/run149-prepared/release-statement.json \ - --verifier-evidence /secure/evidence/release-statement.verifier-output.json \ - --source-revision "$SOURCE_REVISION" \ - --verifier-name gh-attestation \ - --verifier-version 2.0 \ - --output /secure/evidence/release-statement.signature-verification.json -``` - -The helper hashes the external verifier output; it does not copy it into the release -and does not perform network signature discovery. The parent CI must create that -verifier output only after its approved cryptographic identity/workflow checks pass. - -## Finalize — one-shot promotion - -```bash -python _hf_spaces_proxy/security/promote_release.py finalize \ - --evidence /secure/evidence/release-evidence.json \ - --prepared-dir /secure/staging/run149-prepared \ - --baseline-zip /secure/baselines/scikitplot__sphinx_ai_assistant_run148.zip \ - --signature-record /secure/evidence/release-statement.signature-verification.json \ - --signature-verifier-evidence /secure/evidence/release-statement.verifier-output.json \ - --promotion-dir /secure/promoted/run149 -``` - -The destination is written through a temporary sibling directory and atomically moved -into place only after every subject is rebound. Existing destinations are rejected. - -## What the signed statement binds - -The canonical statement includes only bounded, non-secret release identity: - -- release ID, proxy version and exact 40/64-hex source revision; -- whole-tree SHA-256 from Run 148; -- schema-v2 evidence SHA-256; -- previous packaged ZIP SHA-256 + size; -- exact Git patch SHA-256 + size; -- final ZIP SHA-256 + size + source file count; -- Python runtime SBOM SHA-256; -- verified full-image SBOM SHA-256 + OCI image subject; -- explicit patch/tree, ZIP/tree and deterministic-package verification flags. - -It contains no Redis authority, tokens, private keys, raw signature material, user -content, prompts, generated media, ZIP edit paths, or provider secrets. - -## Fail-closed invariants - -Promotion fails if any of these change between phases: source bytes/modes, evidence, -baseline ZIP, patch, release ZIP, statement, signature-verifier output, signature -record, Python SBOM reference, or image SBOM reference. Patch paths are validated and -ignored cache/bytecode artifacts are forbidden. Baseline ZIP extraction rejects -traversal, duplicate paths, special filesystem entries, structural type mismatches, -compression-ratio abuse and workspace overages. diff --git a/security/RELEASE_PUBLICATION_GUIDE.md b/security/RELEASE_PUBLICATION_GUIDE.md deleted file mode 100644 index 0e0379413496c43777a0d58bf19e2a2a365436cf..0000000000000000000000000000000000000000 --- a/security/RELEASE_PUBLICATION_GUIDE.md +++ /dev/null @@ -1,208 +0,0 @@ -# Run 150 — receipt-authorized release publication - -Run 149 ends with one atomic promotion directory and a canonical -`promotion-receipt.json`. Run 150 makes that receipt the only file-publication -authority. A publisher adapter may transport bytes to GitHub Releases, object storage, -or a registry, but it cannot add a filename, change a hash, choose a different target, -or silently overwrite an existing object. - -## Authority flow - -```text -Run 149 promoted directory - | - v -promotion-receipt.json - | - +-- canonical schema check - +-- exact directory allowlist check - +-- SHA-256 + size rebind for every publish item - +-- release-statement.json rebind - +-- signature-verification record rebind - `-- server-owned snapshot of every authorized object - | - v - publisher adapter - operation=publish - | - +-- create-only - +-- no overwrite - +-- exact target - +-- exact name/hash/size - `-- remote read-back required - | - v - publisher adapter - operation=verify - | - +-- remote bytes still match - +-- same locator - `-- same immutability class - | - v - publication-transparency.json - publication-receipt.json -``` - -The tool never rebuilds the ZIP or patch. The bytes supplied to the publisher are -read-only snapshots made after the Run 149 receipt has been rebound. - -## Command - -The production publication entrypoint is: - -```bash -python _hf_spaces_proxy/security/publish_release.py \ - --promotion-dir /secure/promoted/run150 \ - --output-dir /secure/publication-evidence/run150 \ - --publisher github-release \ - --target-id scikit-plots/scikit-plots:run150 \ - --publisher-executable /secure/bin/scikit-plots-release-publisher -``` - -For object stores with true retention/object lock, publication can additionally require -that protection: - -```bash - --require-immutability object-lock \ - --require-immutability content-addressed -``` - -Multiple values mean the remote result must use one of the explicitly accepted classes. -The default still rejects `none`; accepted classes are bounded by -`release_publication_policy.toml`. - -## Publisher adapter protocol - -The adapter is an independently protected release-system component. It receives one -canonical JSON request on stdin and must emit exactly one bounded JSON object on stdout. -It is called twice for each receipt-authorized artifact. - -### `publish` - -The request contains: - -- publication ID derived from the promotion-receipt SHA-256 + publisher + target; -- release ID; -- exact promotion-receipt SHA-256; -- exact target publisher/target ID; -- artifact basename, SHA-256 and size; -- a temporary read-only `localPath` to the verified snapshot. - -The adapter must use create-only semantics. It may return `created` or `present`. -`present` is valid only after remote read-back proves that the existing object has the -same SHA-256 and size. - -### `verify` - -The second request contains the same authority identity without a local path. The -adapter must re-read the remote object and return `present` with the same locator, -SHA-256, size and immutability class. - -### Required response guarantees - -Every response must state: - -```json -{ - "createOnly": true, - "overwrite": false, - "remoteReadbackVerified": true, - "immutability": "object-lock" -} -``` - -Allowed immutability classes are: - -- `object-lock` -- `content-addressed` -- `versioned-create-only` -- `release-create-only` - -A remote locator is an audit identifier, not an access URL. Query strings, fragments, -userinfo-style `@` fields and control characters are rejected so signed URLs or secrets -cannot leak into publication evidence. - -## Backend guidance - -### GitHub Releases - -The adapter should create an asset only when the name does not already exist. Never use -an overwrite/clobber path. If the exact asset already exists after an interrupted prior -attempt, re-download/read it and return `present` only when its bytes match the receipt. -Use `release-create-only` because release assets can still be administratively deleted; -do not mislabel that platform behavior as object lock. - -### Object storage - -Prefer an immutable release key namespace plus provider-native create-if-absent and -retention/object-lock controls. Read the object back through the storage API and verify -its complete SHA-256 + size before returning. Use `object-lock` only when the bucket/key -policy really provides that guarantee. - -### Content-addressed / OCI-style storage - -Publish by digest where the backend makes the digest the object identity and verify the -resolved bytes after upload. Report `content-addressed` only when a different payload -cannot occupy the same digest identity. - -## Interrupted publication is resumable, not rollback-faked - -Remote side effects cannot be made transactionally atomic across arbitrary services. -Run 150 therefore uses a deterministic publication ID and exact remote read-back: - -```text -object A created -object B upload loses response - | - v -rerun same publication - | - +-- A -> present + exact read-back - +-- B -> present if commit actually happened - `-- remaining objects -> create-only -``` - -A same-name object with different bytes remains a hard failure. Run 150 never deletes or -overwrites it to make the transaction appear successful. - -## Transparency outputs - -On complete success the evidence directory contains: - -```text -publication-transparency.json -publication-receipt.json -publisher-results/ - 01-....publish.json - 01-....verify.json - ... -``` - -`publication-transparency.json` binds: - -- promotion-receipt SHA-256; -- release/source/evidence/release-statement identity; -- exact publisher + target; -- every authorized local SHA-256 + size; -- final remote locator + SHA-256 + size; -- immutability class; -- publish and read-back evidence hashes. - -The raw local source path is never copied into transparency evidence. - -`publication-receipt.json` is the compact final audit record for this publication -transaction. It is not a new authority to rebuild release files. - -## Trust boundary - -Run 150 verifies what the publisher adapter claims and preserves its canonical evidence, -but the adapter is still the component that talks to the remote service. Protect the -adapter and its credentials as release infrastructure. - -Run 151 makes the stronger handoff executable: `finalize_publication.py` re-validates -this directory, requires a separate read-only verifier identity to re-read every remote -object, makes the exact transparency digest the subject of a canonical in-toto -attestation, requires externally verified signing, then performs a second independent -read-back before create-only binding a deterministic final release record. See -`RELEASE_TRANSPARENCY_GUIDE.md`. No private signing key is stored in this source tree. diff --git a/security/RELEASE_ROOT_CONTINUITY_GUIDE.md b/security/RELEASE_ROOT_CONTINUITY_GUIDE.md deleted file mode 100644 index 94f77478f1843f8dfd2bfbc94ffb1e471187b2d1..0000000000000000000000000000000000000000 --- a/security/RELEASE_ROOT_CONTINUITY_GUIDE.md +++ /dev/null @@ -1,137 +0,0 @@ -# Run 157 — Recovered Root Continuity and X.509 Attestation - -Run 157 turns a Run 156 emergency root recovery into a first-class continuation of the -release trust chain. The recovered root is no longer a terminal side record. It becomes -the only root authorized to approve later governance epochs and the old root for later -ordinary root rotations. - -## Trust transition - -```text -Run 155 sealed root chain - | - v -Run 156 independently authorized recovery - | - +-- compromised old root keys are not required - v -replacement root - | - +-- X.509 key-attestation evidence for every active root key - | -> chain to independently SHA-256-pinned attestation CA(s) - | -> exact release key / recovery context / device class binding - v -Run 157 recovered-root continuity state - | - +-- next governance epoch is signed by governance keys in recovered root - | - +-- later root rotation requires: - old recovered-root threshold - AND - new root self-threshold - v -future continuity epochs -``` - -The compromised pre-recovery root can never satisfy the **old recovered-root** threshold -for a later rotation. - -## X.509 hardware-key attestation profile - -Run 156's `hsm` / `kms-hsm` labels were cryptographically pinned operational metadata, -but not hardware provenance. Run 157 adds direct certificate and signature verification. - -For every active replacement root key, and every root key introduced by a later rotation, -Run 157 requires a canonical `x509-key-attestation` record. The record binds: - -- governance ID and continuity context; -- exact root version and canonical root SHA-256; -- exact root key ID and Ed25519 public key; -- device class, manufacturer and model claims; -- attestation issuance and expiry; -- the exact DER certificate chain. - -The attestation leaf signs those canonical bytes. Run 157 verifies the leaf signature, -certificate BasicConstraints and KeyUsage, every intermediate signature, certificate -validity coverage, and termination at exactly one independently supplied trust root. -Trust roots are accepted only when their DER SHA-256 values exactly match the -out-of-band pins supplied by the release operator. - -The profile is provider-neutral. A production organization may pin a vendor CA, an -enterprise attestation CA that validates vendor HSM/TPM evidence before issuing a leaf, -or another approved hardware-attestation PKI. Run 157 does not contain private keys and -does not treat a device-class string by itself as proof. - -## Activation artifacts - -Successful `activate` output is exactly: - -```text -release-root-continuity-bundle.json -trusted-root-continuity-state.json -active-root.json -release-root-continuity-receipt.json -``` - -The cumulative bundle embeds the canonical Run 155 seal, Run 156 recovery output, -attestation trust-root certificates, replacement-root attestations, the recovery -continuity event, and every later governance epoch. This makes offline replay independent -of the original working directories. - -## Later governance epochs - -`advance` validates the next Run 154 governance candidate, requires the exact previous -governance-state hash and consecutive epoch, and cryptographically verifies the selected -governance/emergency signatures against the current recovered/continued root. - -If governance/emergency membership changes, a next root is mandatory. A voluntary root -rotation is also allowed. In both cases normal Run 155 rotation semantics return: - -```text -current old-root threshold + next-root self-threshold -``` - -When the current root came from recovery, “old root” means that recovered root—not the -compromised pre-recovery root. - -Every newly introduced root role key also requires fresh X.509 attestation evidence. - -## Offline verification - -`verify` reconstructs and checks, without private keys: - -1. complete Run 155 seal and bootstrap pin; -2. complete Run 156 recovery and recovery-root pin; -3. X.509 trust-root pins and replacement-root attestations; -4. recovery continuity chain head; -5. each embedded governance bundle/state pair; -6. each cryptographic governance authorization; -7. each post-recovery root rotation; -8. each new-root attestation set; -9. cumulative continuity head, active root and compact state; -10. final receipt hashes and sizes. - -`--historical` evaluates signatures/certificates at their covered historical times so -later expiry does not erase evidence. Live verification still enforces current freshness. - -## Fail-closed rules - -Run 157 rejects missing/extra root attestations, unknown device classes, invalid or -unpinned certificate chains, future/expired live attestations, governance epoch -rollback/skip, candidate substitution, unauthorized governance signatures, old-root -thresholds from the compromised pre-recovery root, root version skips, authority changes -without root rotation, cumulative-chain mutation, duplicate JSON keys, input drift, and -output placement inside an authority input. - -No production HSM/KMS/TPM private material belongs in these artifacts. Only public -certificates, public release keys, signatures, hashes, and canonical release evidence are -persisted. - -## Run 158 handoff - -Run 157 proves the certificate chain and key-attestation signature. Run 158 adds the -missing lifecycle layer: threshold-governed CA rotation, short-lived certificate-status -snapshots, effective-time revocation, and certificate-bound vendor/device semantics. A -Run 157 continuity directory intended for hardware-backed release acceptance should be -handed to `verify_attestation_lifecycle.py` rather than treating X.509 validity alone as a -complete revocation/status decision. diff --git a/security/RELEASE_ROOT_GUIDE.md b/security/RELEASE_ROOT_GUIDE.md deleted file mode 100644 index dad7ad4ebdc547dc725df388ea3e07dd1cd66ac4..0000000000000000000000000000000000000000 --- a/security/RELEASE_ROOT_GUIDE.md +++ /dev/null @@ -1,171 +0,0 @@ -# Cryptographic release-root sealing - -Run 155 moves the accepted governance boundary from semantic signature-verification -records to direct cryptographic proof. Run 154 remains the candidate-state constructor; -`seal_release_governance.py` is the acceptance gate. - -The repository never stores a signing private key. Root and governance public keys are -carried in canonical TUF-style root metadata, while offline or hardware-backed signers -produce detached Ed25519 signatures outside the repository. - -## Trust flow - -```text -externally pinned root-v1.json SHA-256 - ↓ -canonical root metadata - version + issuedAt + expires - explicit per-key expiry - root role M-of-N - governance role M-of-N - emergency role M-of-N - ↓ -root self-signature threshold - ↓ -exact Run 154 governance candidate - ↓ -rebind bundle/state/snapshot/proposal/sidecars - ↓ -canonical governance-authorization subject - ↓ -selected Run 154 key IDs sign with Ed25519 - ↓ -in-process cryptographic verification - ↓ -optional root rotation - old-root threshold - AND - new-root threshold - ↓ -trusted-release-root-state.json -release-root-bundle.json -cryptographic-governance-authorization.json -release-root-receipt.json -``` - -## Bootstrap is pinned, never TOFU - -The first root must be version 1, canonical, cryptographically self-signed by its root -role threshold, and byte-identical to an out-of-band SHA-256 pin supplied to the gate. -Run 155 currently allows bootstrap only while accepting governance epoch 1. A deployment -must protect that initial hash independently of the release workspace. - -## Key separation - -The root role is intentionally disjoint from both online governance roles. Governance and -emergency roles are also disjoint. Each role requires multiple operators. Root public-key -metadata contains only public Ed25519 material, logical identity, operator, and expiry; -private key, seed, HSM handle, cloud-signing credential, or recovery secret material is -not accepted into the root schema or emitted in Run 155 output. - -The active `governance` and `emergency` root roles must exactly match the corresponding -Run 154 policy membership, threshold, identity, operator, and key ID. A root cannot hide -an alternate authority set behind the same selected quorum. - -## Direct governance authorization - -Run 154 approval files remain useful provenance but their `signatureVerified: true` -field is not an acceptance authority in Run 155. The exact selected Run 154 key IDs must -supply fresh detached signatures over a canonical Run 155 subject that binds: - -- governance ID and history ID; -- epoch and policy version; -- transition ID and proposal SHA-256; -- the exact selected key set and authorization role; -- authorizing root version and SHA-256; -- exact governance state, governance bundle, and recovery-snapshot SHA-256 + size; -- the exact history sequence/state/bundle/chain-head binding. - -Each detached approval has a cryptographically covered signing time, identity, operator, -key ID, root version, subject hash, and `approve` decision. Signatures are verified -in-process using `cryptography`'s Ed25519 implementation. Invalid, stale, future-dated, -expired-key, wrong-identity, wrong-operator, wrong-root, or wrong-subject signatures fail -closed. - -## Root rotation - -Authority membership cannot change unless root metadata rotates. Root version N+1 must -advance exactly one version and the exact new `signed` root bytes must satisfy both: - -1. the old-root threshold using keys trusted by version N; and -2. the new-root threshold using keys declared by version N+1. - -This old-root + new-root rule makes rotation independently verifiable and prevents a -single new authority from self-installing. Root envelopes also reject signatures from -keys outside the old/new root roles, so an online governance key or arbitrary key cannot -change the envelope identity by appending an irrelevant signature. After rotation, the -new governance/emergency roles must exactly match the candidate's final Run 154 policy. - -## Expiry, rollback, and freeze protection - -Root metadata contains signed `issuedAt` and `expires` values. Every active role key has -an explicit expiry that must cover the full root lifetime. Root lifetime is bounded by -policy, current roots require a configured minimum remaining lifetime, live bundle replay -rejects a materially future-dated current root, and rotations must be consecutive. A -previous `trusted-release-root-state.json` binds the last accepted -governance epoch and state hash, so a later candidate cannot skip an epoch or replay an -older governance state. - -A root chain is stored in `release-root-bundle.json`; offline verification replays the -bootstrap self-signature and every old-root/new-root dual threshold. The compact state -binds the current root version/hash/expiry, bundle hash, root-chain head, and exact last -accepted governance artifacts. - -## Input integrity - -Run 155 uses a strict Run 154 directory allowlist and rebinds the candidate's canonical -bundle, state, recovery snapshot, proposal, approval evidence, archive evidence, and -embedded history artifacts. All input files are hashed before sealing and rehashed before -commit. A signer or local adapter cannot mutate candidate evidence during the gate and -have the altered bytes accepted. - -## Operational dependency - -The release-security environment must provide the `cryptography` package with Ed25519 -support. Absence of the crypto backend fails closed as `ROOT_CRYPTO_BACKEND_UNAVAILABLE`; -there is no semantic-verification fallback. - -Run 155 does not claim that software keys are equivalent to hardware-backed keys. In -production, root private keys should normally be offline or hardware-backed, governance -keys should be independently operated, and the externally pinned bootstrap hash should -be stored in a separate trust channel. - -## Complete offline seal verification - -`seal_release_governance.py verify-seal` replays more than the root metadata chain. Given -the independently retained bootstrap-root SHA-256 pin, it validates the sealed directory -allowlist, current root state, active-root rebinding, embedded Run 154 governance -state/bundle, exact recovery snapshot, final-policy role parity, receipt hashes, and every -detached governance authorization signature. Historical signature validity is checked at -the cryptographically covered `signedAt` time, so later root expiry does not erase valid -historical evidence; live sealing still requires a fresh, non-freezing active root. - -The bootstrap pin is required on every root/seal verification, including verification of -a previous Run 155 state. A replaced `trusted-release-root-state.json` plus a replaced -`release-root-bundle.json` therefore cannot become a new trust root merely because the two -attacker-controlled files agree with each other. - -## Run 156 delegated freshness and recovery - -Run 155's root chain remains the governance seal authority, but long-lived root expiry is -not used as the sole freshness signal. Run 156 adds short-lived threshold-signed snapshot -and timestamp metadata with monotonic versions and exact Run 155 artifact rebinding. -This narrows replay/freeze windows without requiring offline root keys to sign every -release publication event. - -For root-role compromise or loss, Run 156 introduces a separately pinned recovery root. -The recovery threshold is independent of the active root/governance/emergency operators, -and each recovery key is cryptographically bound to a specific recovery channel and -signer profile. A valid ceremony may replace only the root role; governance and emergency -roles remain byte-identical. The replacement root must satisfy its own new-root threshold -and exclude all declared compromised keys. See `RELEASE_DELEGATED_TRUST_GUIDE.md`. - -## Run 157 recovered-root continuity - -A Run 156 replacement root is activated into a cumulative Run 157 continuity bundle -rather than being spliced into the Run 155 root bundle as if the compromised old root had -signed it. Future governance authorizations use the recovered root's governance/emergency -roles, and future ordinary root rotations require the old recovered-root threshold plus -the next-root self-threshold. Every recovered or newly rotated root key is additionally -bound to directly verified X.509 attestation evidence and independently pinned attestation -trust roots. See `RELEASE_ROOT_CONTINUITY_GUIDE.md`. diff --git a/security/RELEASE_TEST_CLOCK_HERMETICITY_GUIDE.md b/security/RELEASE_TEST_CLOCK_HERMETICITY_GUIDE.md deleted file mode 100644 index b5ecb044488430dce44cf3329f7b3a60588d6a1f..0000000000000000000000000000000000000000 --- a/security/RELEASE_TEST_CLOCK_HERMETICITY_GUIDE.md +++ /dev/null @@ -1,43 +0,0 @@ -# Run 169 — hermetic release-security test clock - -Run 169 removes the last wall-clock dependency from the synthetic cryptographic release -history used by Runs 149–168. It changes **test authority only**; production verifiers keep -their live-time freshness, expiry, rollback, and freeze-risk checks unchanged. - -## Why this gate exists - -Run 158 intentionally gives its synthetic attestation-status snapshot a bounded validity -window. Run 159 previously chose `NOW` from the machine wall clock. Once real time moved -inside Run 158's minimum-remaining-time safety margin, all downstream Runs 160–168 could -fail while constructing their predecessor fixture even though the repository bytes had -not changed. - -That is not acceptable release evidence. A frozen test vector must remain replayable after -its calendar date passes. - -## Authority boundary - -Run 159 now derives its synthetic time from the fixed predecessor timeline: - -```text -Run 158 fixed NOW - ↓ + 1 hour -Run 159 synthetic NOW - ↓ -Runs 160–168 predecessor-derived clocks -``` - -This does **not** freeze production time. Runtime tools still use their explicit `now=` -argument when supplied and the real UTC clock when callers omit it. Expired production -status therefore continues to fail closed. - -## Release gate - -`tests/_architecture/test_release_gate_clock_hermeticity.py` statically rejects direct wall-clock -reads (`datetime.now`, `datetime.utcnow`, `date.today`, or `time.time`) in Runs 149–168 -release-security tests. New release fixtures should derive time from a fixed predecessor -or an explicit deterministic constant. - -A release MUST fail if a synthetic cryptographic fixture silently reacquires wall-clock -authority. Tests that intentionally exercise real-time behavior belong in a separately -identified live/integration gate, not in deterministic provenance replay. diff --git a/security/RELEASE_TRANSPARENCY_GUIDE.md b/security/RELEASE_TRANSPARENCY_GUIDE.md deleted file mode 100644 index c7ff1a99b2ac9d6aaa6ddd2f17462f175ddb372f..0000000000000000000000000000000000000000 --- a/security/RELEASE_TRANSPARENCY_GUIDE.md +++ /dev/null @@ -1,318 +0,0 @@ -# Run 151 — signed post-publication transparency and final release binding - -Run 150 proves that only receipt-authorized release bytes were transported and read -back through the publisher adapter. Run 151 removes the publisher from the final trust -position: a separately identified **read-only verifier** re-reads every published object, -a canonical in-toto statement binds those observations to the exact -`publication-transparency.json`, an external release trust root signs/verifies that -statement, and a final deterministic release record is attached to the same release -target using create-only semantics. - -No signing private key is stored in this source tree. - -## Trust flow - -```text -Run 150 publication directory - | - +-- publication-transparency.json - +-- publication-receipt.json - `-- publisher-results/* - | - v - exact canonical re-validation - publisher evidence hash rebind - | - v - independent verifier identity - read-only / no publisher credentials - | - +-- re-read every remote object - +-- exact locator - +-- exact SHA-256 + size - `-- exact immutability class - | - v - publication-attestation.json - in-toto subject = exact transparency SHA-256 - | - v - external cryptographic signer - + external signature verifier - | - v - publication-attestation.signature-verification.json - | - v - final independent remote re-read - after signature verification - | - v - release-publication-record.json - deterministic from signed authority + object identities - | - v - final-record binder - create-only / no overwrite / remote read-back - | - v - release-publication-binding-receipt.json -``` - -The publisher identity that wrote the original objects is not accepted as the Run 151 -independent verifier identity. The verifier response must explicitly assert read-only -authority and that publisher write credentials were not reused. Production deployment -must enforce that separation with different credentials/roles, not merely different -labels. - -## 1. Prepare the post-publication attestation - -Use a verifier adapter that has read-only access to the published release objects: - -```bash -python _hf_spaces_proxy/security/finalize_publication.py prepare \ - --publication-dir /secure/publication-evidence/run151 \ - --output-dir /secure/post-publication/run151-prepared \ - --verifier-identity ci/post-publication-verifier \ - --verifier-executable /secure/bin/scikit-plots-release-verifier -``` - -The prepare step re-validates the complete Run 150 evidence directory, including every -hashed `publisher-results/*.json` object. It then calls the independent verifier once -for every published artifact and emits: - -```text -publication-attestation.json -independent-verifier-results/ - 01-....verify.json - 02-....verify.json - ... -``` - -`publication-attestation.json` is a canonical in-toto Statement v1. Its subject is -exactly: - -```json -{ - "name": "publication-transparency.json", - "digest": {"sha256": "..."} -} -``` - -The predicate binds the publication ID, promotion/publication receipt hashes, release -ID, source revision, exact target, independent verifier identity, every remote object -name/hash/size/locator/immutability tuple, and the hash of each verifier result. - -## 2. Externally sign and verify the attestation - -The prepared attestation is not self-authenticating. Sign or attest -`publication-attestation.json` with the organization-approved release trust root, then -verify that signature using an independently protected verifier and an explicitly -expected signer identity. - -Appropriate mechanisms include: - -- GitHub artifact attestations with signer workflow/repository/ref constraints; -- Sigstore/cosign with an organization policy that pins the certificate identity and - issuer; -- KMS/HSM-backed signatures verified by the release control plane; -- an equivalent platform-native attestation system with identity verification. - -Do not treat a JSON field such as `"verified": true` as cryptographic evidence. The -external verifier must actually validate the signature and signer identity first. -Store its bounded non-secret output outside the source tree, then create the sanitized -record: - -```bash -python _hf_spaces_proxy/security/finalize_publication.py signature-record \ - --publication-dir /secure/publication-evidence/run151 \ - --attestation /secure/post-publication/run151-prepared/publication-attestation.json \ - --verifier-evidence /secure/post-publication/signature-verifier-output.json \ - --output /secure/post-publication/publication-attestation.signature-verification.json \ - --signer-identity 'https://github.com/scikit-plots/scikit-plots/.github/workflows/release.yml@refs/heads/main' \ - --verifier-name gh-attestation \ - --verifier-version 2.x -``` - -The sanitized record binds the exact attestation SHA-256, exact transparency SHA-256, -publication ID, expected signer identity, verifier-output SHA-256, and verifier -name/version. It does not copy credentials, signed URLs, raw signatures, or private -keys into the release tree. - -## 3. Finalize and bind the final release record - -After the signature has been externally verified, finalize with the same independent -read-only verifier identity and a binder that can attach one exact evidence object to -the already published target: - -```bash -python _hf_spaces_proxy/security/finalize_publication.py finalize \ - --publication-dir /secure/publication-evidence/run151 \ - --prepared-dir /secure/post-publication/run151-prepared \ - --signature-record /secure/post-publication/publication-attestation.signature-verification.json \ - --signature-verifier-evidence /secure/post-publication/signature-verifier-output.json \ - --output-dir /secure/post-publication/run151-final \ - --expected-signer-identity 'https://github.com/scikit-plots/scikit-plots/.github/workflows/release.yml@refs/heads/main' \ - --verifier-identity ci/post-publication-verifier \ - --verifier-executable /secure/bin/scikit-plots-release-verifier \ - --binder-executable /secure/bin/scikit-plots-release-record-binder -``` - -Finalization performs another complete remote read-back **after** signature verification. -A changed, missing, rebound, stale, or differently located object fails closed. -Only after those checks succeed is `release-publication-record.json` constructed. - -The final record binds: - -- promotion-receipt SHA-256; -- publication-receipt SHA-256; -- publication-transparency SHA-256; -- signed post-publication attestation SHA-256; -- signature-verification-record SHA-256; -- external signature-verifier-evidence SHA-256; -- expected signer identity; -- independent verifier identity and deterministic verification ID; -- exact remote object name, SHA-256, size, locator, and immutability class; -- proof flags that the signed attestation and post-signature read-back both succeeded. - -Fresh remote-verifier timestamps are deliberately not embedded in the final record. -The final record is therefore deterministic for the same signed attestation and the -same remote object identities. This matters when the remote create-only bind succeeds -but the local process loses its response: a retry recreates the same final-record bytes -and may accept `present` only after exact remote read-back. - -## Independent verifier protocol - -The verifier receives canonical JSON on stdin and emits one bounded JSON object on -stdout. It never receives a local release file path. The request identifies the exact -publication, transparency hash, target, artifact digest/size, remote locator, and -immutability class. - -A successful response must bind the same values and include: - -```json -{ - "status": "present", - "verifier": { - "identity": "ci/post-publication-verifier", - "readOnly": true, - "publisherCredentialsReused": false - } -} -``` - -The response's remote SHA-256 and size must equal the Run 150 publication evidence. -Verification timestamps are freshness-bounded. Adapter stdout is bounded while the -process runs and the process itself is timeout-bounded. - -## Final-record binder protocol - -The binder receives only one generated authority object: - -```text -release-publication-record.json -``` - -It cannot choose another filename, target, hash, or size. The `bind` call receives a -read-only local snapshot; the subsequent `verify` call receives no local path and must -read the bound record back remotely. - -Required guarantees are: - -```json -{ - "createOnly": true, - "overwrite": false, - "remoteReadbackVerified": true, - "immutability": "object-lock", - "bindingType": "release-asset" -} -``` - -Allowed binding types are: - -- `release-asset` -- `object-store-object` -- `oci-referrer` -- `registry-attestation` - -Allowed immutability classes remain the Run 150 set: - -- `object-lock` -- `content-addressed` -- `versioned-create-only` -- `release-create-only` - -A binder locator is audit metadata, not a signed access URL. Query strings, fragments, -userinfo-style locators, control characters, target switching, and locator collision -with an already published release object are rejected. - -## Provider mapping - -### GitHub Releases - -Use a read-only token or OIDC-derived identity for the independent verifier that can -download release assets but cannot upload, delete, or replace them. Bind the final -record as a create-only release asset with the fixed filename -`release-publication-record.json`, then download/read it back before returning success. -Report `release-asset` and an honest immutability class; ordinary GitHub release assets -are not object lock. - -### Object storage - -Use a distinct read-only principal for the verifier. Bind the final record under the -same release namespace using provider-native create-if-absent semantics. Prefer object -lock/retention where available and report `object-store-object`. - -### OCI / registries - -Verify release blobs/manifests by digest through a read-only identity. Attach the final -record as a referrer/attestation that is bound to the already published release digest, -using create-only or content-addressed semantics. Report `oci-referrer` or -`registry-attestation` as appropriate. - -## Final output - -A successful finalization directory contains: - -```text -publication-transparency.json -publication-receipt.json -publication-attestation.json -publication-attestation.signature-verification.json -release-publication-record.json -release-publication-binding-receipt.json -final-verifier-results/ - 01-....verify.json - ... -binding-results/ - release-publication-record.json.bind.json - release-publication-record.json.verify.json -``` - -`release-publication-binding-receipt.json` is local audit evidence for the final binding. -It includes the remote final-record locator and hashes of the binder/final-verifier -results. The remote final record itself remains deterministic and contains no local -paths, credentials, signed URLs, prompts, user content, or release private keys. - -## Residual trust boundary - -Run 151 can validate schemas, hashes, identities, freshness, separation claims, and -remote adapter results, but it cannot manufacture infrastructure independence. The -production verifier must actually use a distinct read-only principal, and the external -signature verifier must actually enforce the configured signer identity. The binder -must implement the create-only/read-back guarantees it reports. - -Those three components should be independently protected release infrastructure. The -proxy/browser runtime does not receive their credentials and no private signing key is -committed to this repository. - -## Run 152 handoff - -Run 151's final record is intentionally deterministic so it can become an external -transparency subject without rebuilding any release artifact. Run 152 consumes the -finalized Run 151 directory, binds the exact final-record SHA-256 into an append-only log, -verifies continuity from a pinned previous checkpoint with a separate read-only -verifier, requires a multi-operator witness quorum to agree on the same checkpoint, and -then create-only anchors a deterministic witnessed record to the original release. -See `RELEASE_WITNESS_GUIDE.md`. diff --git a/security/RELEASE_WITNESS_GUIDE.md b/security/RELEASE_WITNESS_GUIDE.md deleted file mode 100644 index 60d29bfb25a2ab97aa48b7c6ac7d41c86dc789e0..0000000000000000000000000000000000000000 --- a/security/RELEASE_WITNESS_GUIDE.md +++ /dev/null @@ -1,259 +0,0 @@ -# Run 152 — append-only transparency and threshold witnessing - -Run 151 proves that the exact published release objects were independently re-read, -that the post-publication attestation was externally signature-verified, and that a -deterministic `release-publication-record.json` was create-only attached to the release. -Run 152 makes that final record externally observable over time: the exact record is -submitted to an append-only transparency system, its inclusion and checkpoint -consistency are verified by a separate read-only principal, at least two distinct -witness operators independently agree on the same checkpoint, and a deterministic -witnessed record is create-only attached back to the original release target. - -No transparency-log signing key, witness private key, release private key, publisher -credential, or log-write credential is stored in this source tree. - -## Trust flow - -```text -Run 151 finalized directory - | - +-- release-publication-record.json - `-- release-publication-binding-receipt.json - | - v - exact canonical/hash rebind - | - +---------------- previous checkpoint - | pinned from prior trusted run - v - transparency submitter - append-only / no overwrite - | - v - new signed checkpoint + log entry - | - v - primary read-only log verifier - - checkpoint signature verified - - exact entry inclusion verified - - integrated entry verified - - consistency from previous checkpoint verified - | - v - threshold witness quorum - >= 2 identities - >= 2 distinct operators - no log-write credentials - no primary-verifier credentials - all observe the SAME entry + checkpoint - | - v - release-transparency-witness-record.json - deterministic from immutable identities/checkpoint - | - v - create-only release anchor - remote read-back required - | - v - release-transparency-witness-receipt.json -``` - -The transparency log may be Sigstore/Rekor, another RFC6962-style service, a provider -native immutable ledger, or an equivalent externally operated append-only system. The -repository code is provider-neutral: adapters perform provider-specific cryptographic -verification, while `witness_publication.py` strictly rebinds their results to the -exact Run 151 record, checkpoint, identities, and target. - -## Why the previous checkpoint is mandatory - -A single valid inclusion proof does not prevent a malicious or compromised log from -showing different histories to different clients. Run 152 therefore requires a -canonical previous checkpoint and requires the independent verifier and all witnesses -to verify consistency from that checkpoint to the newly observed checkpoint. - -The default policy does **not** permit an unpinned bootstrap: - -```toml -allow_bootstrap_without_previous_checkpoint = false -``` - -A first production deployment should obtain its initial checkpoint through an -organization-approved out-of-band trust process, store it as a canonical pinned -checkpoint, then advance only through verified consistency. Do not silently replace a -missing previous checkpoint with the current log head. - -Canonical previous checkpoint format: - -```json -{ - "schemaVersion": 1, - "logId": "sigstore/rekor-production", - "checkpoint": { - "treeSize": 123456, - "rootHash": "<64 lowercase hex>", - "signedCheckpointSha256": "<64 lowercase hex>" - } -} -``` - -`signedCheckpointSha256` is the SHA-256 of the externally authenticated checkpoint -representation. The provider adapter/verifier is responsible for actually checking the -log signature/key identity. A Boolean field alone is not a substitute for that -cryptographic verification. - -## Authority separation - -Four roles are deliberately distinct. - -### 1. Transparency submitter - -The submitter has only the authority needed to append the exact Run 151 final record. -It must not rebuild, rename, normalize, or mutate the record. The adapter receives the -local path only for the `submit` operation; local paths never enter release evidence. - -### 2. Primary transparency verifier - -The primary verifier must use a different identity from the submitter and read-only log -access. It verifies the exact entry, exact checkpoint signature, exact inclusion, and -consistency from the pinned previous checkpoint. - -### 3. Threshold witnesses - -Policy requires at least two witness identities and two distinct operators. Every -witness receives the same immutable subject, entry identity, new checkpoint, and -previous checkpoint. A witness must report: - -- read-only authority; -- no transparency-log write credential reuse; -- no primary-verifier credential reuse; -- checkpoint signature verification; -- exact inclusion verification; -- integrated-entry verification; -- checkpoint consistency verification. - -All witnesses must return the exact same checkpoint tuple. A different root, tree -size, checkpoint digest, entry ID, index, locator, or subject is treated as a possible -split-view and fails closed. - -Different process names under one credential are not independent witnesses. Production -operators must enforce the claimed identity/operator separation with separate trust -roots, accounts, workload identities, or organizations. - -### 4. Release anchor - -After quorum succeeds, Run 152 creates the deterministic fixed-name -`release-transparency-witness-record.json` and asks an external anchor adapter to attach -exactly those bytes to the original Run 151 release target. The anchor is create-only, -never overwrites, and must remotely read the record back before success. - -A retry may return `present` only if exact SHA-256 and size match. The anchor locator -must not collide with any Run 151 release object or the existing final-record locator. - -## Adapter protocol - -Adapters are newline-independent JSON programs: they receive one canonical JSON object -on stdin and must emit one bounded JSON object on stdout. Subprocess output is bounded -while produced and every adapter has a hard timeout. The environment passed to adapters -is intentionally minimal and does not inherit arbitrary secrets through the protocol. - -Provider implementations should authenticate results outside this JSON schema—for -example with workload identity, KMS/HSM verification, Sigstore certificate policy, -provider-signed checkpoints, or mutually authenticated release infrastructure. - -## CLI - -Example: - -```bash -python _hf_spaces_proxy/security/witness_publication.py \ - --finalized-dir /secure/post-publication/run152-final \ - --previous-checkpoint /secure/transparency/rekor.previous.json \ - --output-dir /secure/transparency/run152-witnessed \ - --log-id sigstore/rekor-production \ - --submitter-identity ci/transparency-submitter \ - --log-executable /secure/bin/rekor-submit-exact \ - --verifier-identity ci/transparency-readonly-verifier \ - --verifier-executable /secure/bin/rekor-independent-verify \ - --witness 'witness-a@operator-a=/secure/bin/witness-a' \ - --witness 'witness-b@operator-b=/secure/bin/witness-b' \ - --anchor-executable /secure/bin/release-evidence-anchor -``` - -The `IDENTITY@OPERATOR=EXECUTABLE[,ARG...]` witness syntax is intentionally explicit so -release configuration defines the exact quorum membership instead of accepting any two -responders discovered at runtime. - -## Determinism and interrupted retries - -Fresh integration/verification timestamps stay in local receipt/evidence files. The -remote witness record contains only immutable release identity, the exact final-record -hash, exact log entry/checkpoint identity, previous-checkpoint hash, primary verifier -identity, and sorted quorum membership. Therefore an interrupted create-only anchor can -be retried without creating different bytes. - -The same rule applies to the transparency submission: an adapter may report `present` -only for the same exact subject already integrated in the same log entry/checkpoint -lineage. - -## Final output - -Successful output contains: - -```text -release-publication-record.json -release-publication-binding-receipt.json -previous-transparency-checkpoint.json -accepted-transparency-checkpoint.json -release-transparency-witness-record.json -release-transparency-witness-receipt.json -log-results/ - submit.json - primary.verify.json -witness-results/ - 01-....verify.json - 02-....verify.json -anchor-results/ - release-transparency-witness-record.json.bind.json - release-transparency-witness-record.json.verify.json -``` - -`accepted-transparency-checkpoint.json` is canonical and has the same schema as the -required previous-checkpoint input. Persist it only after the complete Run 152 gate -succeeds, then supply those exact bytes to the next release. - -The witnessed record is the compact deterministic release object. The witness receipt -is local audit evidence and additionally binds hashes of the submitter, primary -verifier, every witness result, and anchor read-back evidence. - -## Split-view and compromise model - -Threshold witnessing reduces—not magically eliminates—the chance that one compromised -log endpoint or one verifier can rewrite release history. Its value depends on actual -independence. If the log, primary verifier, and all witness operators share the same -credentials, network path, administrative account, or mutable state, quorum labels do -not provide meaningful protection. - -For stronger deployments, distribute witnesses across different organizations or -control planes and persist accepted checkpoints in independently protected storage. -Periodically gossip/compare checkpoints outside the release pipeline. A checkpoint -mismatch should halt release promotion and trigger investigation rather than choosing -one branch automatically. - -## Residual trust boundary - -Run 152 validates exact bytes, schemas, hashes, target binding, append-only/no-overwrite -claims, checkpoint monotonicity, observer separation, threshold membership, and exact -checkpoint agreement. It does not implement a provider's checkpoint signature format or -Merkle proof algorithm itself. Those cryptographic details belong in independently -maintained provider adapters and verifier/witness infrastructure, where log keys and -proof formats can evolve without giving the browser/proxy runtime new authority. - -## Run 153 handoff - -Run 152's `accepted-transparency-checkpoint.json` is the immediate checkpoint handoff, -but long-term releases should not persist checkpoint files as isolated facts. Run 153 -binds that checkpoint into a canonical cross-release history chain, gossips it across -independent replicas, applies explicit log-key lifecycle rules, and archives the exact -offline-verifiable history bundle to multiple immutable operators. See -`RELEASE_HISTORY_GUIDE.md`. diff --git a/security/SECURITY_RELEASE_GATES.md b/security/SECURITY_RELEASE_GATES.md deleted file mode 100644 index a3712790749fcd6acc52d96e5f65c049aa131aba..0000000000000000000000000000000000000000 --- a/security/SECURITY_RELEASE_GATES.md +++ /dev/null @@ -1,725 +0,0 @@ -# Supply-chain, deployment, and evidence release gates - -Run 19 / B38 makes the source-controlled deployment path reproducible and -fail-closed. Run 20 / B39 adds short-lived, content-addressed production -evidence binding so stale or unrelated scanner/attestation files cannot be -substituted for the current source and final image. It does **not** claim that a pinned image or lock file is free of -future vulnerabilities. Release evidence must be renewed whenever the image, -lock, deployment platform, or advisory database changes. - -## Source-controlled gates - -1. **Immutable base** — `Dockerfile` uses an exact Python tag plus immutable - OCI index digest. CI resolves the requested `linux/amd64` manifest from that - index and records the manifest digest used for the release. -2. **Hash-locked Python closure** — production installs `requirements.lock` - with both `--require-hashes` and `--only-binary=:all:`. Source builds and - resolver drift are not accepted in the release path. -3. **Minimal framework extras** — FastAPI and Uvicorn are installed without - their broad optional/`standard` extras. Every transitive runtime package is - named explicitly by the lock. -4. **Runtime without install tooling** — dependencies are built in an isolated - venv in a builder stage. Runtime `pip`, `setuptools`, and `wheel` payloads - from the base image are removed and the finished venv is copied in. -5. **Non-root strict profile** — runtime UID/GID is `1000:1000`, matching the - Hugging Face Docker Spaces convention. `DEPLOYMENT_PROFILE=strict` also - verifies non-root execution in application startup. -6. **Deny-by-default build context** — `.dockerignore` allows only the service - runtime unit into the Docker build context, reducing accidental secret, - cache, repository-history, test-fixture, and unrelated-artifact inclusion. -7. **Read-only/rootless reference** — the hardened Compose reference drops all - Linux capabilities, enables `no-new-privileges`, uses a read-only root - filesystem, and confines expected temporary writes to `/tmp`. -8. **Redis transport authority** — strict deployments require `rediss://` for - every Redis-backed control plane. URL query parameters cannot downgrade TLS; - certificate and hostname verification are forced by code. -9. **Python SBOM** — `python-runtime.cdx.json` describes the exact locked Python - closure. It is not a complete image SBOM because it intentionally excludes - OS/base-image packages. - -## Networked CI/release evidence — mandatory before production promotion - -Run these in a networked, current advisory environment. Tool names are examples; -organizations may use equivalent scanners, but **do not turn scanner failure -into a warning-only step**. - -```bash -# Offline structure/ratchet verifier committed with the source. -python security/verify_supply_chain.py - -# Dependency advisory gate. Generate/install in an isolated environment from -# the exact lock first; fail on known vulnerabilities with no approved policy. -pip-audit --strict --require-hashes -r requirements.lock - -# Build for the locked target architecture and identify the immutable result. -docker build --platform linux/amd64 -t scikitplots-ai-proxy:b38 . -docker image inspect scikitplots-ai-proxy:b38 --format '{{.Id}}' - -# Full image SBOM (includes OS packages) and vulnerability gate. -syft scikitplots-ai-proxy:b38 -o cyclonedx-json > image.cdx.json -trivy image --exit-code 1 --severity HIGH,CRITICAL scikitplots-ai-proxy:b38 - -# Prefer signed provenance/SBOM attestations in the target registry (for -# example, BuildKit provenance plus organization-approved signing tooling). -``` - -Scanner output is time-sensitive evidence. Never copy an old “0 CVEs” result -forward to a new release. If a base-image CVE has no upstream fix yet, document -its reachability, compensating controls, owner, expiration date, and explicit -risk acceptance rather than silently suppressing it. - -## Dependency update protocol - -Update direct requirements, regenerate the complete wheel lock for the exact -platform, regenerate the Python SBOM, run `verify_supply_chain.py`, review fresh -advisories, run the complete proxy regression suite, build/scan the container, -and only then update the maintenance checkpoint. A dependency bump is a security -change even when application source is unchanged. - -## Current B38 advisory ratchets - -The B38 review found that the previous environment's Click 8.1.8 and Starlette -0.50.0 are below current security fixes. The lock therefore ratchets Click to -8.3.3 and Starlette to the reviewed 1.6.0 release; -`supply_chain_policy.toml` prevents an accidental rollback below those reviewed -floors. Fresh scanning remains mandatory because -new advisories can appear after this checkpoint. - - -## B39 machine-verifiable production evidence - -Before collecting external evidence, obtain the canonical non-secret subjects: - -```bash -python security/release_subjects.py -``` - -The output contains only proxy version, target platform, exact lock/SBOM/runtime -source digests, and the immutable base-image index digest. It contains no URLs, -credentials, user data, deployment hostnames, or Redis identity. - -Store the fresh dependency scan, image scan, full-image CycloneDX SBOM, SLSA -provenance, and signature-verification output beside `release-evidence.json`. -Each referenced file is bound by relative path + SHA-256 + explicit subject. -The production manifest also records the resolved base-image manifest digest, -final OCI image digest, sanitized Redis operational evidence, and infrastructure -logging/telemetry posture. - -The standard hardened B39 policy accepts no risk-exception entry in the promotion -manifest. A release that needs an exception must change/review the policy rather -than smuggling a waiver into evidence. Manifests expire within 72 hours. - -```bash -# Source-only structural gate. -python security/verify_supply_chain.py - -# One production promotion gate: source policy + bound fresh evidence. -python security/verify_release_gate.py /secure/release/release-evidence.json -``` - -`verify_release_gate.py` fails closed on stale/expired evidence, source or artifact -hash drift, path traversal/symlink substitution, mismatched artifact subjects, -provenance that does not name the final image or resolved base manifest, missing -signature-verification evidence, hidden infrastructure body/credential logging, -unverified Share/Contribution persistence/replication, stale backup/restore -exercises, or third-party telemetry export. - -See `RELEASE_EVIDENCE_GUIDE.md` and `release-evidence.example.json`. The example -is intentionally non-authoritative and cannot pass verification unchanged. - -## Run 137 archive-edit authority gate - -Any endpoint that later returns a modified user ZIP must route the rewrite through -`_utils/_zip_workspace.py` (or a reviewed successor with the same invariants) rather than -constructing an archive from a model-selected subset. Promotion must fail if tests no -longer prove all of the following: - -- the original archive remains the complete tree authority; only exact existing regular - files can be replaced; -- no filesystem extraction API is used and no extracted project directory is created; -- traversal, absolute/drive/backslash, control/bidi, Unicode/case/trailing-dot aliases, - file/descendant collisions, symlinks, special files, and Unix type/path mismatches fail - closed; -- entry count, source size, per-entry size, aggregate uncompressed size, aggregate - replacement size, compression ratio, and rewritten-output growth are bounded; -- only STORED and DEFLATE are accepted until another method receives an explicit reviewed - implementation; encrypted archives are not accepted; -- well-formed portable metadata is preserved, while stale ZIP64 transport extra records - are not copied; malformed extra TLVs are rejected; -- every unchanged payload is streamed through decompression/CRC verification, source bytes - are generation-bound with SHA-256 before/after rewrite, and the completed output is - reopened for independent tree/metadata/content verification; -- receipts stay bounded and credential/provider-neutral; no provider IDs, tokens, URLs, - raw errors, or user file bodies are copied into provenance. - -Run the focused gate with: - -```bash -pytest -q tests/_hf_spaces_proxy/_utils/test__zip_workspace.py -``` - -Run 138 is the reviewed surgical successor to this gate. Untouched entries now copy their -complete local record — local header, compressed payload, and optional data descriptor — -byte-for-byte, while changed entries alone are regenerated. The central directory is rebuilt -with fresh offsets and bounded non-ZIP64 transport fields. This optimization **does not** -skip the final unchanged-payload decompression/CRC verification: accepting a faster -"trust the source CRC" mode would weaken Run 137's corrupt-input rejection guarantee. - -Run 138 additionally fails closed on directory entries with hidden payloads, unsupported -general-purpose ZIP flags, multi-disk or archive-level ZIP64 layouts, trailing bytes after -EOCD, unsupported central-directory adjunct records, local/central flag disagreement, and -malformed local extra metadata. A self-extracting preamble is treated as wrapper data rather -than project content and is deliberately not propagated into the returned ZIP. Central -ZIP64 transport extras remain normalized away; an untouched local record can retain its -original local transport bytes because raw-record fidelity is the explicit Run 138 promise. - -The source generation is also copied into a bounded server-owned archive spool after its -initial SHA-256 binding. Inspection and raw-record copying operate only on that stable -snapshot, while the caller-visible source is rehashed before and after the rewrite. This -prevents a mutable source from racing individual record reads without materializing any -archive entry onto the filesystem. - -Run the surgical fidelity gate with: - -```bash -pytest -q tests/_hf_spaces_proxy/_utils/test__zip_workspace__surgical.py -``` - -Promotion must keep both Run 137 and Run 138 focused gates green. Run 137 remains the -semantic/security contract; Run 138 adds stronger untouched-record fidelity without -relaxing that contract. - -## Run 139 ZIP artifact authorization/orchestration gate - -`POST /v1/artifacts/zip-edit` must keep archive mutation authority outside provider/model -output. Promotion must fail if tests no longer prove all of the following: - -- `scikitplot-zip-edit-v1` rejects unknown fields and separates the exact source generation, - explicit `authorization.paths`, and untrusted `proposal.replacements`; -- every proposal path is a subset of the authorization set, while the workspace separately - verifies **all** authorized paths are exact existing regular files in the source archive; -- authorization cannot add, delete, rename, replace directories, or otherwise override the - source archive tree; -- source and replacement multipart parts are independently size/SHA-256 bound to the - manifest before any rewrite, with no provider IDs or model metadata in the contract; -- multipart parsing has active source/per-replacement/aggregate/request/count ceilings and - closes spooled files on malformed or over-limit requests; -- replacement inputs remain seekable/chunk-streamed and are snapshotted into bounded - server-owned spools rather than concatenated into an aggregate in-memory byte payload; -- blocking ZIP validation/compression runs outside the async request loop and successful - downloads stream the complete verified artifact; -- output filenames are server-sanitized and browser-visible receipts are path-free, bounded, - credential-neutral provenance; -- CORS exposes only the artifact filename/receipt/hash headers needed by the official client; -- the client UI (when enabled) constructs `authorization.paths` only from explicit user - selection/approval and never copies model-generated authorization claims. - -Run the focused orchestration gate with: - -```bash -pytest -q tests/_hf_spaces_proxy/_utils/test__zip_artifact.py -``` - -Promotion must keep Runs 137, 138, and 139 green together. - -### Provider artifact lifecycle Redis authority (Run 145) - -When provider artifact lifecycle storage uses Redis, release evidence must include the `providerArtifactLifecycle` plane with verified TLS, a non-default least-privilege identity, and replication. Persistence/backup is not required because the lifecycle is deliberately ephemeral; Redis loss fails old candidates closed and users regenerate. - -### Redis reconnect / real-Lua chaos gate (Run 146) - -Run 146 keeps the Run 145 public lifecycle contract but hardens Redis transport ambiguity and -cluster deployment. Promotion must preserve these properties: - -- the exact same lifecycle operation may be retried after a connection/timeout ambiguity without - creating a second lifecycle, broadening a ZIP reservation, or making terminal apply replayable; -- retry recognition happens **before** normal duplicate suppression for `BEGIN`, so a client never - gets blocked by its own successfully committed first attempt; -- cancellation watchers tolerate a bounded transient Redis outage but cancel local provider work - after repeated shared-authority failures; -- Redis Cluster use is explicit (`PROVIDER_ARTIFACT_LIFECYCLE_REDIS_TOPOLOGY=cluster`), uses the - Redis Cluster client rather than a standalone client pointed at a cluster node, and requires - database `0`; -- all lifecycle Lua keys remain in one `{provider-artifact}` cluster slot; -- a Redis restart/failover never reconstructs old ephemeral write authority from client state; -- Redis URLs, identities, hosts, key prefixes, and credentials stay out of health/error output. - -Portable CI runs the reconnect/cluster doctor even without a Redis executable: - -```bash -pytest -q tests/_hf_spaces_proxy/_utils/test__provider_artifact_lifecycle__redis_chaos.py -``` - -Redis-enabled CI **should make the actual server gate mandatory** so the production Lua is executed -by Redis itself rather than only inspected/semantically modeled: - -```bash -RUN146_REDIS_CHAOS_REQUIRED=1 pytest -q tests/_hf_spaces_proxy/_utils/test__provider_artifact_lifecycle__redis_chaos.py -``` - -An explicit Redis 7/8 server binary may be supplied with `RUN146_REDIS_SERVER=/path/to/redis-server`. - -### Mandatory Redis standalone + cluster CI gate (Run 147) - -For a release that advertises shared provider-artifact lifecycle authority, CI SHOULD execute the -portable Run 147 gate and MUST do so when the deployment uses Redis Cluster: - -```bash -python _hf_spaces_proxy/ci/run_redis_chaos.py --mode all -``` - -Promotion evidence should include both Redis 7 and Redis 8 executions. The gate is fail-closed: it -rejects missing/unsupported Redis binaries or missing redis-py cluster support rather than converting -them to skips. The cluster half must prove all of the following against real Redis processes: - -- six-node cluster convergence with full 16,384-slot coverage; -- 3-primary / 3-replica topology; -- same-slot execution of the production lifecycle Lua scripts; -- two independent OS processes racing for one reservation with exactly one winner; -- terminal `applied` non-replay before and after slot-primary failure; -- replica promotion for the `{provider-artifact}` slot; -- successful new lifecycle work after failover. - -Reference GitHub Actions and CircleCI fragments plus the exact-major Redis chaos Dockerfile live under -`_hf_spaces_proxy/ci/`. Production Redis URLs, credentials, provider tokens, prompts, generated bytes, -and ZIP contents are not inputs to this CI gate. - - -### Signed Redis chaos release provenance (Run 148) - -Production promotion now consumes the Run 147 live Redis gate as schema-v2 release -evidence rather than accepting CI success by name. The trusted CI must produce four -canonical attestations for the exact source revision/tree: Redis 7 standalone, Redis -7 cluster, Redis 8 standalone, and Redis 8 cluster. - -Every attestation must use the immutable Redis image reference pinned in -`release_evidence_policy.toml`, and every attestation must have a separately verified -cryptographic signature/identity record. Promotion fails closed on a missing mode, -wrong Redis major/image digest, stale timestamp, source revision/tree mismatch, -attestation tampering, signature-subject mismatch, or unverified signer identity. - -The reference GitHub workflow uses GitHub artifact attestations and immediately -verifies them with `gh attestation verify`, constraining the exact source digest/ref, -signer workflow + signer digest, and rejecting self-hosted-runner attestations before -writing the sanitized release record. Other CI systems -must perform equivalent organization-approved cryptographic signing -and identity verification before generating the sanitized signature record. Merely -copying `verified=true` into JSON without a trusted external verification step does not -satisfy this security control. - -## Run 149 — transactional promotion gate - -Do not publish directly after `verify_release_gate.py` succeeds. Use -`security/promote_release.py prepare`, externally sign/verify the canonical release -statement, then use `promote_release.py finalize`. Final publication is restricted to -the exact filenames, sizes and SHA-256 values in `promotion-receipt.json`; rebuilding -ZIPs or patches after this point invalidates the promotion transaction. The finalizer -re-verifies schema-v2 evidence, source tree, external signature-verifier output, patch -application, ZIP contents/modes, and Python/full-image SBOM references. - -## Run 150 — publication receipt enforcement - -Production publication must use `security/publish_release.py` or an equivalent control -that consumes the finalized Run 149 promotion directory directly. Only files listed in -`promotion-receipt.json` may be transported. Before the first upload, rebind every file -SHA-256/size and rebind the receipt to `release-statement.json` plus the statement -signature-verification record. Publishers must be create-only, must not overwrite, and -must perform remote byte read-back after upload. A retry may accept an already-present -object only when the remote SHA-256 and size exactly match the receipt. The final -publication transparency/receipt evidence must identify the promotion receipt, target, -remote object locators and remote read-back hashes without embedding signed URLs or -release credentials. Object-lock/content-addressed storage should be required where the -chosen backend provides it. - - -## Run 151 — signed post-publication transparency gate - -A Run 150 publication is not release-final until `security/finalize_publication.py` -(or an equivalent control) has independently verified it. The verifier identity must be -separate from the publisher and must use read-only authority without publisher write -credentials. `prepare` must rebind the canonical Run 150 transparency/receipt and all -publisher-result hashes, then re-read every remote object by exact locator, SHA-256, -size, and immutability class. - -The generated `publication-attestation.json` must be cryptographically signed outside -this source tree, and the external verifier must enforce the expected signer identity. -`finalize` must reject stale/tampered/rebound signature evidence, re-read all published -objects again after signature verification, and bind only the deterministic fixed-name -`release-publication-record.json` to the original target. The binder must be -create-only, must not overwrite, and must remotely read the final record back before -success. Signed/query/fragment/userinfo-style locators, target switching, remote locator -collisions, and publisher-credential reuse by the independent verifier fail closed. - -A retry after an interrupted final-record bind may accept `present` only when remote -read-back proves the exact deterministic record SHA-256 and size. Private signing keys, -publisher credentials, and verifier credentials must remain outside the proxy/browser -runtime and outside release evidence. - -## Run 152 — append-only transparency and threshold witness gate - -After Run 151 finalization, `witness_publication.py` must validate the complete Run 151 -finalized directory, submit the exact deterministic `release-publication-record.json` -to the configured append-only transparency log, and bind the result to a pinned -canonical previous checkpoint. A distinct read-only primary verifier must verify the -checkpoint signature, exact entry inclusion, integrated entry, and checkpoint -consistency. - -Release succeeds only when at least two distinct witness identities from at least two -distinct operators independently verify the same subject, log entry, new checkpoint, -and previous checkpoint. Witnesses must not reuse transparency-log write credentials or -the primary-verifier credentials. Any split-view, stale response, non-advancing tree, -identity reuse, locator collision, target switch, input mutation, or failed -inclusion/consistency/signature verification fails closed. - -Only then may the deterministic fixed-name -`release-transparency-witness-record.json` be attached to the original release target. -The anchor must be create-only, must not overwrite, and must remotely read back exact -SHA-256 + size. Private keys and release/log/witness credentials remain outside the -source tree and outside the proxy/browser runtime. See `RELEASE_WITNESS_GUIDE.md`. - -## Run 153 — durable cross-release history and equivocation gate - -After Run 152 witnessing, release history must advance only from the exact canonical -`trusted-history-state.json` and `release-history-bundle.json` accepted for the previous -release. `security/preserve_release_history.py` revalidates the complete Run 152 witness -directory, requires its previous checkpoint to equal the trusted history checkpoint, -and obtains an N-of-M view from independently operated read-only gossip replicas. - -At least three replicas must be configured by default, at least two must agree, and the -agreeing replicas must span at least two operators. Explicit unavailability may be -tolerated while quorum remains; any available conflicting checkpoint, prior history -head, or log-key identity is treated as possible equivocation and fails closed. Release -IDs, publication IDs, and witnessed subjects may not be replayed at later history -sequence numbers. - -Transparency-log key changes require an explicit policy-authorized transition. Normal -rotation requires old-key continuity plus new-key proof verification. Compromise -recovery requires explicit emergency authority. The replaced key is permanently added -to the revoked-key set and may not be reused. - -Only after the history bundle and trusted state pass offline verification may the exact -deterministic `release-history-bundle.json` be create-only stored and remotely read back -by at least two distinct archive operators. The next release must consume the exact -emitted state + bundle pair. See `RELEASE_HISTORY_GUIDE.md`. - -## Run 154 — threshold governance and disaster-recovery gate - -After Run 153 preservation, changes to release-history trust roots, authority keys, -replica membership, archive membership, or their thresholds must be represented by a -canonical governance transition. `security/govern_release_history.py` binds that -transition to the exact previous governance-state hash and exact Run 153 history -sequence/state/bundle/chain head. Normal changes require the current policy-authority -M-of-N threshold. An `authority-compromise-recovery` transition instead requires the -separate emergency-authority threshold and must permanently revoke at least one current -policy-authority key. Revoked authority keys may never be reintroduced. - -Governance bootstrap is allowed only from an externally pinned canonical genesis -SHA-256. The genesis replica/archive topology must match the Run 153 evidence being -admitted. No ambient deployment configuration may silently redefine the trusted -membership. - -Disaster recovery must query at least three independent immutable sources and obtain an -N-of-M quorum spanning independent operators. Every returned state+bundle pair is -independently Run-153-validated. One unavailable source may be tolerated when quorum -remains, but any available conflicting valid history fails closed. - -Only after offline governance verification passes may the deterministic, self-contained -`release-governance-recovery-snapshot.json` be create-only archived to at least two -independent governance archive operators. Each bind requires a separate remote read-back, -unique secret-free locator, no writer-credential reuse, and pre/post local snapshot -rehashing. - -## Run 155 — cryptographic root-of-trust gate - -Run 154 output is a governance **candidate**, not the final accepted trust decision. -`security/seal_release_governance.py` is the Run 155 acceptance authority and MUST pass -before a governance candidate is treated as trusted release-root state. - -The gate requires a canonical, TUF-style Ed25519 root chain. Bootstrap root version 1 is -accepted only when its complete canonical SHA-256 matches an out-of-band pin and its -root-role self-signatures satisfy the configured M-of-N threshold. Root keys are -explicitly expiring and the active root must retain the configured minimum lifetime. -Root, governance, and emergency roles are independently thresholded and operator-diverse; -the offline root role is disjoint from both online authorization roles. - -Run 155 MUST NOT use Run 154's `signatureVerified: true` field as authorization. Instead, -the exact Run 154 `selectedApproverKeyIds` sign a canonical Run 155 subject with Ed25519. -The subject binds the proposal, root version/hash, governance state, governance bundle, -recovery snapshot, history binding, epoch, policy version, transition ID, selected key -set, and role. Verification happens in-process against public keys from the active root. -Missing crypto support fails closed; there is no semantic-verification fallback. - -The active root governance/emergency roles must exactly match the prior Run 154 policy's -key IDs, thresholds, identities, and operators. If the candidate changes either authority, -a root rotation is mandatory. Root N+1 advances exactly one version and the exact new -root bytes must satisfy both the old-root threshold and the new-root threshold. The new -roles must then exactly match the candidate's final policy. - -`release-root-bundle.json` carries the complete root chain for offline verification. -`trusted-release-root-state.json` binds its bundle hash and root-chain head to the exact -last accepted governance artifacts. A subsequent seal must advance the governance epoch -exactly once and must rebind the proposal's previous-governance-state hash to the prior -trusted root state, preventing governance replay or skip attacks. - -Production root private keys SHOULD be offline or hardware-backed. Private keys, seeds, -HSM handles, cloud-signing credentials, and recovery secrets MUST NOT be placed in root -metadata, the repository, or Run 155 output. See `security/RELEASE_ROOT_GUIDE.md`. - -Offline acceptance MUST use `seal_release_governance.py verify-seal` with the independently -retained bootstrap-root SHA-256. Supplying only the root bundle/state without that pin is -not an accepted verification path. The verifier replays the cryptographic governance -signatures and exact Run 154 recovery snapshot as well as root rotation history. - -## Run 156 — delegated freshness and root-recovery gate - -A Run 155 seal MUST NOT be treated as indefinitely fresh merely because its long-lived -root remains valid. `security/maintain_release_trust.py refresh` requires a root-threshold -signed delegation, a threshold-signed snapshot that binds every Run 155 seal artifact, -and a shorter-lived threshold-signed timestamp that binds the exact snapshot. Snapshot -and timestamp roles MUST be key-disjoint and operator-diverse. Delegated key expiry MUST -cover the delegation lifetime. - -When previous delegated state exists, snapshot and timestamp versions MUST advance -exactly one step. Reusing a delegation version with different bytes is forbidden. The -previous effective Run 155 root version/hash MUST still exist in the current root chain, -and its root-chain prefix head MUST recompute to the previously trusted value. Any fork, -rollback, skip, near-expiry metadata, future issuance, or timestamp-before-snapshot -condition fails closed. - -Break-glass root recovery MUST use an independently pinned/self-thresholded recovery root; -the potentially compromised active root threshold is not an authorization input. Recovery -keys/operators MUST be disjoint from active Run 155 authority keys/operators. Recovery -public keys MUST pin their recovery channel and signer profile, and each approval's signed -channel/profile MUST match those pinned values. The configured M-of-N key threshold, -operator quorum, and channel quorum MUST all be satisfied with fresh Ed25519 signatures. - -A replacement root MUST be the next version, satisfy its own new-root threshold, exclude -all declared compromised root keys, and preserve governance/emergency role membership and -key metadata exactly. Recovery-authority keys/operators MUST NOT become replacement-root -keys/operators. This gate is root-role recovery, not an alternate governance path. - -Live verification uses bounded expiry for delegation/snapshot/timestamp metadata; -historical verification evaluates signatures at their covered issuance time. Both -refresh and recovery MUST rehash every authority input before commit and fail on drift. -Private keys, seeds, HSM handles, cloud-signing credentials, recovery secrets, and local -paths MUST NOT appear in Run 156 evidence. See `RELEASE_DELEGATED_TRUST_GUIDE.md`. - -## Run 157 — recovered-root continuity and X.509 attestation gate - -After Run 156 accepts an independently authorized root recovery, the recovered root MUST -be activated through `continue_recovered_trust.py` before it can authorize later release -governance. Activation MUST replay the complete Run 155 seal and Run 156 recovery, require -out-of-band SHA-256 pins for the bootstrap root, recovery root, and attestation trust -root(s), and cryptographically attest every active replacement-root key through an X.509 -chain terminating at an independently pinned attestation CA. - -Subsequent governance epochs MUST advance consecutively from the state bound into the -continuity bundle. Governance authorization MUST be verified against the current recovered -root. Any later root rotation MUST satisfy the current **old recovered-root** threshold and -the new-root self-threshold. A signature set from the compromised pre-recovery root MUST -NOT substitute for the recovered-root threshold. - -Every root key introduced by a later rotation MUST have a valid, context-bound X.509 -attestation before that root can become active. Offline verification MUST replay the base -seal, recovery, certificate chains, governance signatures, rotations, and cumulative -continuity hashes. Private keys, HSM handles, credentials, local source paths, and -unpinned trust roots MUST NOT enter persistent Run 157 evidence. See -`RELEASE_ROOT_CONTINUITY_GUIDE.md`. - -## Run 158 — attestation lifecycle, revocation, and vendor-semantic gate - -Run 157 X.509 path verification MUST be followed by -`security/verify_attestation_lifecycle.py` when hardware-backed release-root provenance is -required. Run 158 embeds and replays the exact Run 157 continuity output, then requires an -active-release-root-threshold-signed attestation CA set and an independent threshold-signed, -short-lived certificate-status snapshot. - -Every attestation certificate used by the preserved Run 157 chain MUST be covered by the -status snapshot. `good`/`revoked` decisions are canonical and revocation is sticky. A -revocation effective at or before the attestation's `attestedAt` invalidates historical -acceptance; a later revocation preserves the old release proof but blocks live use of the -currently active attested key. Status snapshots and CA sets are independently versioned, -monotonic, bounded in lifetime, and protected against future issuance and freeze/replay. - -The active release-root threshold governs attestation CA-set bootstrap/rotation. Every -removed CA MUST be explicitly retired or revoked. `compromise-recovery` requires an -explicit revoked CA, while `scheduled-rotation` cannot silently classify a compromise as -ordinary retirement. Revoked CAs MUST NOT be reintroduced. - -Vendor/device semantics MUST be cryptographically bound into the attestation certificate, -not inferred from an unverified `deviceClass` string. The configured X.509 extension binds -profile, hardware-backed/non-exportable assertions, manufacturer/model, exact root key, -and a stable hashed device identity. Device-class/profile substitution, root-public-key -substitution, or device-identity substitution fails closed. - -Status-authority keys and operators MUST be separate from release-root keys/operators. -Private keys, secrets, HSM handles, recovery material, and local source paths MUST NOT enter -persistent Run 158 evidence. See `security/RELEASE_ATTESTATION_LIFECYCLE_GUIDE.md`. - -## Run 159 — native status provenance and anti-equivocation gate - -Run 158 normalized certificate status MUST be followed by -`security/verify_native_status_provenance.py` when native revocation provenance is -required. The gate MUST replay the complete canonical Run 158 predecessor and independently -verify raw DER CRL and OCSP evidence rather than accepting a provider-produced JSON status -summary as authority. - -Every active attestation leaf MUST have the policy-required native source kinds. CRL -issuer signatures, CRL Numbers, update windows, OCSP issuer hashes, responder authority, -responder certificates, response signatures, and update windows MUST be verified in -process. Exact raw DER bytes and responder-certificate bytes MUST be retained in the -offline bundle. - -Any verified native source disagreement MUST fail closed; a numerical majority MUST NOT -hide equivocation. Native consensus MUST also match the Run 158 accepted status and -revocation time. Later refreshes MUST maintain CRL-number and OCSP-time continuity. -Historical verification MAY replay expired evidence at its covered time, but live -verification MUST retain freeze protection. - -Vendor TPM/HSM/KMS native evidence MUST NOT be accepted merely from a profile label. A -profile without an explicitly configured verifier MUST fail closed; preserved adapter -evidence MUST bind the raw evidence hash, expected key/public-key hash, verifier identity, -and verifier executable hash. See `RELEASE_NATIVE_STATUS_GUIDE.md`. - -## Run 160 — native status archival, verifier-diversity, and recovery gate - -A completed Run 159 native-status output SHOULD be passed through -`security/archive_native_status_evidence.py preserve` before the status service evidence is -considered durably retained. The complete canonical Run 159 output is one deterministic -archive artifact; archive adapters MUST use create-only/no-overwrite semantics and MUST -perform exact SHA-256 + size remote read-back. - -Each archive MUST then be checked by a separate read-only verifier identity/operator that -re-reads the same remote locator. Archive-writer and verifier operators/credentials MUST be -separated, and multiple configured archives MUST resolve to distinct remote locators. - -Recovery MUST require independently retained pins for the archive artifact SHA-256 and the -Run 159 native-status chain head. At least two independent read-only recovery sources MUST -agree on identical canonical bytes; any observed disagreement MUST fail closed rather than -be outvoted. The recovered four-file Run 159 output MUST be reverified offline before use. -See `RELEASE_NATIVE_ARCHIVE_GUIDE.md`. - -## Run 161 — archive retention, challenge/read-back, and migration gate - -A Run 160 native-evidence archive used for long-term provenance SHOULD be enrolled in -`security/audit_archive_retention.py`. The exact active archive membership and every -migration/retirement MUST be authorized by an independently pinned Ed25519 threshold -retention root with multiple governance operators. - -Every active archive MUST provide a cryptographically **provider-signed retention** -statement binding the exact Run 160 artifact, immutable version ID, locator, approved -immutability/retention mode, retention expiry, legal-hold state, and challenge-bound remote -read-back. A distinct read-only auditor/operator MUST perform an **independent challenge** -and signed read-back of the same exact version and provider-response hash. - -A migration MUST NOT grant retirement authorization until every member of the next active -set passes the current provider + independent-auditor checks and the configured **minimum -durable copies** and independent-operator thresholds remain satisfied. Historical health -events MAY replay at their covered time; the active event MUST retain live retention and -observation freshness. See `RELEASE_ARCHIVE_HEALTH_GUIDE.md`. - -## Run 162 — archive-health witnessing and retention-governance recovery gate - -A current Run 161 archive-health checkpoint SHOULD be passed through -`security/witness_archive_health.py` when durable-copy health must be independently -observable across auditors. The witness root MUST be independently retained by exact -SHA-256 and MUST contain an operator-diverse threshold. Every witness response MUST be -read-only, challenge-bound, signed, and bind the exact canonical Run 161 archive view. - -Witness availability quorum MUST NOT become majority-based fork resolution. Any observed -split view of archive membership, immutable version, retention mode/expiry, legal hold, or -Run 161 health-chain head MUST fail closed even when enough other witnesses agree. Older -witness epochs MAY replay historically; the current epoch MUST retain live witness -freshness and match the current Run 161 state. - -A compromised Run 161 retention root MUST NOT recover itself. Break-glass recovery MUST -use a separately **out-of-band** pinned threshold recovery root with independent operators -and pinned recovery channels. The recovery subject MUST bind the exact active Run 161 -health state and replacement root, MUST identify compromised old keys, and MUST contain -an empty retirement authorization. Retention root recovery itself MUST NEVER authorize an -archive retirement or migration. See `RELEASE_ARCHIVE_WITNESS_GUIDE.md`. - -## Run 163 — external archive-health anchor consensus gate - -Run 163 requires every accepted Run 162 witness-chain head to be hash-linked into all configured independently operated append-only anchor channels and independently read back by separate observer identities. Any observed cross-channel or observer split view fails closed. The accepted anchor bundle preserves exact Run 162 documents plus signed channel/observer evidence for offline replay. A separate threshold witness-root recovery path is bound to the already anchored consensus head and requires `rewrittenAnchorEpochs: []`, so recovery cannot rewrite accepted anchor history. See `RELEASE_ARCHIVE_ANCHOR_GUIDE.md`. - -## Run 164 — Merkle inclusion, consistency, and cross-channel gossip gate - -Run 163 external anchors SHOULD be passed through -`security/verify_archive_merkle_transparency.py`. The transparency root MUST be retained -out of band by exact SHA-256 and MUST use an operator-diverse Ed25519 threshold independent -of Run 162/163 authorities. - -Every configured log MUST provide a signed RFC6962-style checkpoint for the exact Run 163 -leaf. The release gate MUST independently verify the **inclusion proof** and, after the -first accepted event, the **consistency proof** from that log's exact previously accepted -tree size/root. Provider booleans claiming inclusion or append-only behavior MUST NOT be -used as proof. - -All configured read-only gossip observers MUST sign the exact same canonical log-id → -checkpoint-SHA-256 map. Any observed **split view**, tree rollback, invalid proof, skipped -Run 163 epoch, or authority-plane overlap MUST fail closed. Historical events MAY replay at -their covered time; the active event and transparency root retain live freshness/freeze -requirements. See `RELEASE_ARCHIVE_MERKLE_GUIDE.md`. - -## Run 165 — Merkle log-key lifecycle and recovery gate - -A Run 164 Merkle-consensus checkpoint SHOULD be enrolled in -`security/govern_archive_merkle_log_authority.py` before any log/gossip signing key is -rotated. Bootstrap MUST bind the exact accepted Run 164 per-log tree size, root hash, and -checkpoint SHA-256. - -A scheduled rotation MUST use the independently pinned governance-root threshold and MUST -require the **old + new** log and gossip keys for every changed log to co-sign the exact -checkpoint handoff. Replaced public-key fingerprints become permanently revoked and MUST -NOT reappear in a later active authority. - -A compromise recovery MUST use the separately pinned recovery-root threshold with -independent operators and recovery channels. Every affected log MUST rotate both its log -and gossip keys, old handoff signatures MUST NOT be trusted, and all replacement keys MUST -still sign the last accepted Run 164 checkpoint. The Run 164 snapshot itself MUST NOT -advance during a Run 165 authority chain. See `RELEASE_ARCHIVE_LOG_AUTHORITY_GUIDE.md`. - -## Run 166 — RFC6962 continuity after log-key rotation/recovery - -A release using post-Run-165 archive transparency MUST pass `tests/_hf_spaces_proxy/security/test_continue_archive_merkle_authority.py`. The gate requires the first new checkpoint to extend Run 165's exact preserved Run 164 tree state with a valid RFC6962 consistency proof, verifies signatures under the active Run 165 authority, rejects split views and authority rebinding, and verifies cumulative offline replay without requiring the retired key. - -## Run 167 — recursive Merkle authority re-bridge gate -Require an exact latest-checkpoint handoff for every post-Run-166 authority change. Scheduled rotation must verify old + new log/gossip signatures; compromise recovery must use the independent recovery quorum and forbid old signatures. The first replacement-key append must prove RFC6962 consistency from the latest accepted tree, and all Run 167 history must replay offline before promotion. - -## Run 168 — recursive Merkle authority recovery durability gate - -A production Run 167 recursive Merkle-authority checkpoint SHOULD be passed through -`security/preserve_archive_merkle_rebridge_history.py` before the intermediate authority -services are considered recoverable from cold storage. Preservation MUST first pass the -full Run 167 verifier, then create one deterministic checkpoint containing the exact Run -167 outputs, active authority, permanent revocations, last RFC6962 checkpoints, and both -cumulative heads. - -The checkpoint MUST be replicated across at least three configured independent archive -locations and separately read back by a read-only verifier plane. Recovery MUST fail on -**any observed equivocation** and MUST require out-of-band pins for checkpoint SHA-256, -authority head, Merkle head, active-authority SHA-256, and sequence. Numerical quorum MUST -NOT permit rollback or fork selection. See `RELEASE_ARCHIVE_MERKLE_RECOVERY_GUIDE.md`. - -## Run 169 — hermetic release-security clock gate - - -Runs 149–168 deterministic release/provenance tests MUST NOT obtain cryptographic fixture -time from the machine wall clock. Run 159 derives its synthetic time from the fixed Run 158 -predecessor, and downstream runs inherit that deterministic timeline. Static regression -tests reject direct wall-clock reads in this release-test range. Production verifier -freshness and expiry behavior is unchanged and continues to fail closed. See -`RELEASE_TEST_CLOCK_HERMETICITY_GUIDE.md`. - -## Run 170 — hermetic external-tool gate - -Runs 149–169 deterministic release/provenance replay MUST NOT inherit executor Git control -state, and production release subprocesses MUST NOT inherit arbitrary executor secrets. -Run 149's synthetic binary-patch proof uses an allowlisted Git environment. Production -promotion resolves Git from the system-default path or an explicitly pinned absolute path -and replays patches under isolated Git configuration with no ambient PATH or GIT_* state. -Every release-security subprocess call MUST pass an explicit environment; provider-neutral -adapters receive only the platform-default path and C locale. Regression tests inject -hostile Git state, a PATH-hijack binary, and parent secret canaries and require fail-closed -secret-free behavior. See `RELEASE_PROCESS_HERMETICITY_GUIDE.md`. diff --git a/security/anchor_archive_health.py b/security/anchor_archive_health.py deleted file mode 100644 index 3766933ccb407defb6daca369e3ab6512b8b1c10..0000000000000000000000000000000000000000 --- a/security/anchor_archive_health.py +++ /dev/null @@ -1,1603 +0,0 @@ -""" -Run 163: externally anchor Run 162 archive-health witness history. - -Every accepted Run 162 witness-chain head is hash-linked into multiple independently -operated append-only channel checkpoints. A separate observer key for each channel -signs an inclusion/continuity read-back of the exact channel response. Recovery of -an archive-health witness root is authorized by a separate pinned threshold root and -is cryptographically bound to the already anchored consensus head; recovery cannot -rewrite accepted anchor epochs. -""" - -from __future__ import annotations - -import base64 -import hashlib -import json -import logging -import os -import re -import shutil -import subprocess -import sys -import tempfile -import threading -from datetime import datetime, timedelta, timezone -from pathlib import Path - -import tomllib -from cryptography.exceptions import InvalidSignature -from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PublicKey - -logger = logging.getLogger(__name__) - -try: - from . import witness_archive_health as run162 -except (ImportError, ValueError) as exc: - import importlib.util - - _here = Path(__file__).resolve().parent - _spec = importlib.util.spec_from_file_location( - "_run162_for_anchor", _here / "witness_archive_health.py" - ) - if _spec is None or _spec.loader is None: - raise ImportError("witness_archive_health.py") from exc - run162 = importlib.util.module_from_spec(_spec) - sys.modules[_spec.name] = run162 - _spec.loader.exec_module(run162) -HERE = Path(__file__).resolve().parent -POLICY = tomllib.loads((HERE / "release_archive_anchor_policy.toml").read_text()) -PREDICATE_TYPE = str(POLICY["predicate_type"]) -_HEX64 = re.compile(r"^[0-9a-f]{64}$") -_ID = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:/@+-]{0,511}$") -_CHUNK = 1024 * 1024 - - -# ── Verified-state document names ──────────────────────────────────────── -# -# One constant per release-evidence document. The values are the on-disk -# artifact names and are unchanged; only the repetition is removed. Reading -# a document by a name that says which document it is also keeps static -# analysis from reading the filename token 'trusted' as 'confidential': -# these files carry published Merkle roots and SHA-256 digests, which must -# stay in clear text for any third party to verify them. -_DOC_ARCHIVE_ANCHOR_STATE = "trusted-archive-anchor-state.json" - - -_ANCHOR_NAMES = { - "release-archive-anchor-bundle.json", - _DOC_ARCHIVE_ANCHOR_STATE, - "active-archive-anchor-evidence.json", - "release-archive-anchor-receipt.json", -} -_RECOVERY_NAMES = { - "recovered-archive-witness-root.json", - "archive-witness-root-recovery-record.json", - "archive-witness-root-recovery-receipt.json", -} - - -class ArchiveAnchorError(RuntimeError): # ruff: ignore[undocumented-public-class] - pass - - -def _fail(c): - raise ArchiveAnchorError(c) - - -def _canonical(v): - return ( - json.dumps(v, sort_keys=True, separators=(",", ":"), ensure_ascii=False) + "\n" - ).encode() - - -def _sha_bytes(b): - return hashlib.sha256(b).hexdigest() - - -def _sha(p): - h = hashlib.sha256() - with p.open("rb") as f: - for x in iter(lambda: f.read(_CHUNK), b""): - h.update(x) - return h.hexdigest() - - -def _write(p, v): - p.write_bytes(_canonical(v)) - - -def _loads(raw, code): - def hook(pairs): - d = {} - for k, v in pairs: - if k in d: - _fail(code + "_DUPLICATE_KEY") - d[k] = v - return d - - try: - v = json.loads(raw.decode(), object_pairs_hook=hook) - except ArchiveAnchorError: - raise - except Exception as e: - raise ArchiveAnchorError(code + "_JSON_INVALID") from e - if not isinstance(v, dict): - _fail(code + "_SCHEMA_INVALID") - return v - - -def _read_json(p, code): - if p.is_symlink() or not p.is_file(): - _fail(code + "_INVALID") - if p.stat().st_size > int(POLICY["max_json_bytes"]): - _fail(code + "_TOO_LARGE") - raw = p.read_bytes() - d = _loads(raw, code) - if raw != _canonical(d): - _fail(code + "_NOT_CANONICAL") - return d, raw - - -def _id(v, code): - if not isinstance(v, str): - _fail(code) - v = v.strip() - if ( - not v - or ".." in v - or "?" in v - or "#" in v - or "\x00" in v - or _ID.fullmatch(v) is None - ): - _fail(code) - return v - - -def _hex(v, code): - if not isinstance(v, str) or _HEX64.fullmatch(v) is None: - _fail(code) - return v - - -def _dt(v, code): - if not isinstance(v, str) or not v.endswith("Z"): - _fail(code) - try: - return datetime.fromisoformat(v[:-1] + "+00:00").astimezone(timezone.utc) - except Exception as e: - raise ArchiveAnchorError(code) from e - - -def _ts(d): - return ( - d.astimezone(timezone.utc).isoformat(timespec="seconds").replace("+00:00", "Z") - ) - - -def _b64(v, code, n=None): - if not isinstance(v, str): - _fail(code) - try: - r = base64.b64decode(v, validate=True) - except Exception as e: - raise ArchiveAnchorError(code) from e - if n is not None and len(r) != n: - _fail(code) - return r - - -def _verify(pk, sig, msg, code): - try: - Ed25519PublicKey.from_public_bytes(_b64(pk, code + "_PUBLIC", 32)).verify( - _b64(sig, code + "_SIG", 64), msg - ) - except (InvalidSignature, ValueError) as e: - raise ArchiveAnchorError(code + "_INVALID") from e - - -def _regular_dir(p, code): - if p.is_symlink() or not p.is_dir(): - _fail(code) - return p.resolve() - - -def _outside(p, protected, code): - t = p.expanduser().resolve() - for x in protected: - r = x.resolve() - if t == r or r in t.parents: - _fail(code) - return t - - -def _dir_fp(root, code): - root = _regular_dir(root, code) - out = {} - for p in sorted(root.iterdir()): - if p.is_symlink() or not p.is_file(): - _fail(code) - st = p.stat() - out[p.name] = (_sha(p), st.st_size, st.st_mode & 0o7777) - return out - - -def _command_adapter(command, prefix): - def call(req): - raw = _canonical(req) - p = subprocess.Popen( # ruff: ignore[subprocess-without-shell-equals-true] - command, - stdin=subprocess.PIPE, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, - env={"PATH": os.defpath, "LC_ALL": "C", "LANG": "C"}, - ) - out = bytearray() - err = bytearray() - limit = int(POLICY["max_adapter_output_bytes"]) - - def pump(stream, buf): - while True: - x = stream.read(65536) - if not x: - break - buf.extend(x) - if len(buf) > limit: - try: # ruff: ignore[suppressible-exception] - p.kill() - except OSError: - pass - return - - t1 = threading.Thread(target=pump, args=(p.stdout, out)) - t2 = threading.Thread(target=pump, args=(p.stderr, err)) - t1.start() - t2.start() - try: - try: - p.stdin.write(raw) - p.stdin.close() - p.wait(timeout=int(POLICY["adapter_timeout_seconds"])) - except subprocess.TimeoutExpired: - p.kill() - p.wait() - t1.join() - t2.join() - _fail(prefix + "_TIMEOUT") - t1.join() - t2.join() - if len(out) > limit or len(err) > limit: - _fail(prefix + "_OUTPUT_TOO_LARGE") - if p.returncode != 0: - _fail(prefix + "_FAILED") - return _loads(bytes(out), prefix + "_RESPONSE") - finally: - if p.poll() is None: - p.kill() - p.wait() - if t1.is_alive(): - t1.join() - if t2.is_alive(): - t2.join() - for stream in (p.stdin, p.stdout, p.stderr): - try: # ruff: ignore[suppressible-exception] - stream.close() - except OSError: # ruff: ignore[try-except-in-loop] - pass - - return call - - -def command_channel(c): # ruff: ignore[undocumented-public-function] - return _command_adapter(c, "ARCHIVE_ANCHOR_CHANNEL") - - -def command_observer(c): # ruff: ignore[undocumented-public-function] - return _command_adapter(c, "ARCHIVE_ANCHOR_OBSERVER") - - -def _verify_witness_root(doc, pin, now, historical=False): - try: - return run162._verify_threshold_root( - doc, - pin, - root_type="archive-health-witness-root", - schema_key="witness_root_schema_version", - min_keys_key="min_witness_keys", - min_threshold_key="min_witness_threshold", - min_operators_key="min_witness_operators", - now=now, - historical=historical, - ) - except Exception as e: - raise ArchiveAnchorError("ARCHIVE_ANCHOR_WITNESS_ROOT_INVALID:" + str(e)) from e - - -def _verify_run162( - *, - run160_dir, - run161_dir, - retention_root_path, - retention_root_pin, - witness_root_path, - witness_root_pin, - bootstrap_pin, - recovery_pin, - attestation_pins, - run162_dir, - now, - historical=False, - require_current_match=True, -): - try: - return run162.verify_witness_history( - run160_dir=run160_dir, - run161_dir=run161_dir, - retention_root_path=retention_root_path, - retention_root_pin=retention_root_pin, - witness_root_path=witness_root_path, - witness_root_pin=witness_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - output_dir=run162_dir, - now=now, - historical=historical, - require_current_match=require_current_match, - ) - except Exception as e: - raise ArchiveAnchorError("ARCHIVE_ANCHOR_RUN162_INVALID:" + str(e)) from e - - -def _channel_entry(v, code): - exp = { - "operator", - "publicKey", - "observerIdentity", - "observerOperator", - "observerPublicKey", - } - if not isinstance(v, dict) or set(v) != exp: - _fail(code + "_SCHEMA_INVALID") - out = { - "operator": _id(v["operator"], code + "_OPERATOR"), - "publicKey": v["publicKey"], - "observerIdentity": _id(v["observerIdentity"], code + "_OBS_ID"), - "observerOperator": _id(v["observerOperator"], code + "_OBS_OPERATOR"), - "observerPublicKey": v["observerPublicKey"], - } - _b64(out["publicKey"], code + "_PUBLIC", 32) - _b64(out["observerPublicKey"], code + "_OBS_PUBLIC", 32) - if out != v: - _fail(code + "_NOT_NORMALIZED") - return out - - -def _verify_plan( # ruff: ignore[too-many-branches] - plan, - wroot, - now, - historical=False, -): - if set(plan) != {"signed", "signatures"} or not isinstance( - plan.get("signed"), dict - ): - _fail("ARCHIVE_ANCHOR_PLAN_SCHEMA_INVALID") - s = plan["signed"] - exp = { - "_type", - "specVersion", - "schemaVersion", - "planId", - "version", - "issuedAt", - "expires", - "witnessRootSha256", - "selectedWitnessKeyIds", - "channels", - } - if ( - set(s) != exp - or s.get("_type") != "archive-health-anchor-plan" - or s.get("specVersion") != str(POLICY["spec_version"]) - or s.get("schemaVersion") != int(POLICY["anchor_plan_schema_version"]) - ): - _fail("ARCHIVE_ANCHOR_PLAN_SIGNED_SCHEMA_INVALID") - _id(s["planId"], "ARCHIVE_ANCHOR_PLAN_ID_INVALID") - if s["version"] != 1: - _fail("ARCHIVE_ANCHOR_PLAN_VERSION_INVALID") - issued = _dt(s["issuedAt"], "ARCHIVE_ANCHOR_PLAN_ISSUED_INVALID") - expires = _dt(s["expires"], "ARCHIVE_ANCHOR_PLAN_EXPIRES_INVALID") - if expires <= issued or ( - not historical - and ( - issued > now + timedelta(minutes=int(POLICY["max_clock_skew_minutes"])) - or expires <= now - ) - ): - _fail("ARCHIVE_ANCHOR_PLAN_LIFETIME_INVALID") - if s["witnessRootSha256"] != wroot["sha256"]: - _fail("ARCHIVE_ANCHOR_PLAN_ROOT_BINDING_INVALID") - selected = s["selectedWitnessKeyIds"] - if ( - not isinstance(selected, list) - or selected != sorted(selected) - or len(selected) != wroot["threshold"] - or any(x not in wroot["keys"] for x in selected) - ): - _fail("ARCHIVE_ANCHOR_PLAN_SIGNERS_INVALID") - channels = s["channels"] - if not isinstance(channels, dict) or len(channels) < int(POLICY["min_channels"]): - _fail("ARCHIVE_ANCHOR_CHANNELS_INVALID") - norm = {} - for cid, v in sorted(channels.items()): - norm[_id(cid, "ARCHIVE_ANCHOR_CHANNEL_ID_INVALID")] = _channel_entry( - v, "ARCHIVE_ANCHOR_CHANNEL" - ) - if channels != norm: - _fail("ARCHIVE_ANCHOR_CHANNELS_NOT_NORMALIZED") - channel_ops = {v["operator"] for v in norm.values()} - observer_ops = {v["observerOperator"] for v in norm.values()} - channel_keys = {v["publicKey"] for v in norm.values()} - observer_keys = {v["observerPublicKey"] for v in norm.values()} - witness_ops = {v["operator"] for v in wroot["keys"].values()} - witness_keys = {v["publicKey"] for v in wroot["keys"].values()} - if len(channel_ops) < int(POLICY["min_channel_operators"]): - _fail("ARCHIVE_ANCHOR_CHANNEL_OPERATOR_QUORUM_INVALID") - if len(observer_ops) < int(POLICY["min_observer_operators"]): - _fail("ARCHIVE_ANCHOR_OBSERVER_OPERATOR_QUORUM_INVALID") - if len(channel_keys) != len(norm) or len(observer_keys) != len(norm): - _fail("ARCHIVE_ANCHOR_KEY_REUSE_INVALID") - if ( - channel_ops & observer_ops - or channel_ops & witness_ops - or observer_ops & witness_ops - or channel_keys & observer_keys - or channel_keys & witness_keys - or observer_keys & witness_keys - ): - _fail("ARCHIVE_ANCHOR_PLANES_OVERLAP") - sigs = plan["signatures"] - if not isinstance(sigs, list) or [x.get("keyId") for x in sigs] != selected: - _fail("ARCHIVE_ANCHOR_PLAN_SIGNATURE_SET_INVALID") - for x in sigs: - if not isinstance(x, dict) or set(x) != {"keyId", "signature"}: - _fail("ARCHIVE_ANCHOR_PLAN_SIGNATURE_SCHEMA_INVALID") - _verify( - wroot["keys"][x["keyId"]]["publicKey"], - x["signature"], - _canonical(s), - "ARCHIVE_ANCHOR_PLAN_SIGNATURE", - ) - return {"signed": s, "channels": norm, "issued": issued, "expires": expires} - - -def _verify_channel_response( - resp, - cid, - cfg, - *, - sequence, - witness_head, - previous, - challenge, - now, - historical=False, -): - if ( - not isinstance(resp, dict) - or set(resp) != {"signed", "signature"} - or not isinstance(resp["signed"], dict) - ): - _fail("ARCHIVE_ANCHOR_CHANNEL_RESPONSE_SCHEMA_INVALID") - s = resp["signed"] - exp = { - "protocolVersion", - "channelId", - "operator", - "anchorId", - "sequence", - "witnessChainHeadSha256", - "previousCheckpointSha256", - "challenge", - "integratedAt", - "locator", - "immutableVersion", - } - if ( - set(s) != exp - or s["protocolVersion"] != 1 - or s["channelId"] != cid - or s["operator"] != cfg["operator"] - or s["sequence"] != sequence - or s["witnessChainHeadSha256"] != witness_head - or s["previousCheckpointSha256"] != previous - or s["challenge"] != challenge - ): - _fail("ARCHIVE_ANCHOR_CHANNEL_RESPONSE_BINDING_INVALID") - _id(s["anchorId"], "ARCHIVE_ANCHOR_ID_INVALID") - _id(s["locator"], "ARCHIVE_ANCHOR_LOCATOR_INVALID") - _id(s["immutableVersion"], "ARCHIVE_ANCHOR_VERSION_INVALID") - at = _dt(s["integratedAt"], "ARCHIVE_ANCHOR_INTEGRATED_INVALID") - if not historical and ( - at > now + timedelta(minutes=int(POLICY["max_clock_skew_minutes"])) - or now - at > timedelta(minutes=int(POLICY["max_observation_age_minutes"])) - ): - _fail("ARCHIVE_ANCHOR_CHANNEL_RESPONSE_STALE") - _verify( - cfg["publicKey"], - resp["signature"], - _canonical(s), - "ARCHIVE_ANCHOR_CHANNEL_SIGNATURE", - ) - return {"doc": resp, "sha256": _sha_bytes(_canonical(resp)), "time": at} - - -def _verify_observer_response( - resp, - cid, - cfg, - *, - anchor_sha, - sequence, - witness_head, - previous, - challenge, - now, - historical=False, -): - if ( - not isinstance(resp, dict) - or set(resp) != {"signed", "signature"} - or not isinstance(resp["signed"], dict) - ): - _fail("ARCHIVE_ANCHOR_OBSERVER_RESPONSE_SCHEMA_INVALID") - s = resp["signed"] - exp = { - "protocolVersion", - "observerIdentity", - "observerOperator", - "channelId", - "anchorResponseSha256", - "sequence", - "witnessChainHeadSha256", - "previousCheckpointSha256", - "challenge", - "observedAt", - "inclusionVerified", - "continuityVerified", - } - if ( - set(s) != exp - or s["protocolVersion"] != 1 - or s["observerIdentity"] != cfg["observerIdentity"] - or s["observerOperator"] != cfg["observerOperator"] - or s["channelId"] != cid - or s["anchorResponseSha256"] != anchor_sha - or s["sequence"] != sequence - or s["witnessChainHeadSha256"] != witness_head - or s["previousCheckpointSha256"] != previous - or s["challenge"] != challenge - or s["inclusionVerified"] is not True - or s["continuityVerified"] is not True - ): - _fail("ARCHIVE_ANCHOR_OBSERVER_RESPONSE_BINDING_INVALID") - at = _dt(s["observedAt"], "ARCHIVE_ANCHOR_OBSERVED_INVALID") - if not historical and ( - at > now + timedelta(minutes=int(POLICY["max_clock_skew_minutes"])) - or now - at > timedelta(minutes=int(POLICY["max_observation_age_minutes"])) - ): - _fail("ARCHIVE_ANCHOR_OBSERVER_RESPONSE_STALE") - _verify( - cfg["observerPublicKey"], - resp["signature"], - _canonical(s), - "ARCHIVE_ANCHOR_OBSERVER_SIGNATURE", - ) - return {"doc": resp, "sha256": _sha_bytes(_canonical(resp)), "time": at} - - -def _event_head(prev, core): - return _sha_bytes( - _canonical({"previousAnchorConsensusHeadSha256": prev, "event": core}) - ) - - -def _load_output(root): - root = _regular_dir(root, "ARCHIVE_ANCHOR_OUTPUT_INVALID") - if {p.name for p in root.iterdir()} != _ANCHOR_NAMES: - _fail("ARCHIVE_ANCHOR_OUTPUT_ALLOWLIST_INVALID") - d = {} - r = {} - for n in _ANCHOR_NAMES: - d[n], r[n] = _read_json(root / n, "ARCHIVE_ANCHOR_OUTPUT") - return d, r - - -def verify_anchor_history( # ruff: ignore[too-many-branches, undocumented-public-function] - *, - run160_dir, - run161_dir, - retention_root_path, - retention_root_pin, - witness_root_path, - witness_root_pin, - bootstrap_pin, - recovery_pin, - attestation_pins, - run162_dir, - anchor_plan_path, - output_dir, - now=None, - historical=False, - require_current_match=True, -): - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - wdoc, _ = _read_json(witness_root_path, "ARCHIVE_ANCHOR_WITNESS_ROOT") - wroot = _verify_witness_root(wdoc, witness_root_pin, current, historical=historical) - pdoc, _ = _read_json(anchor_plan_path, "ARCHIVE_ANCHOR_PLAN") - plan = _verify_plan(pdoc, wroot, current, historical=historical) - current162 = _verify_run162( - run160_dir=run160_dir, - run161_dir=run161_dir, - retention_root_path=retention_root_path, - retention_root_pin=retention_root_pin, - witness_root_path=witness_root_path, - witness_root_pin=witness_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - run162_dir=run162_dir, - now=current, - historical=True, - require_current_match=True, - ) - docs, raws = _load_output(output_dir) - bundle = docs["release-archive-anchor-bundle.json"] - receipt = docs["release-archive-anchor-receipt.json"] - if ( - set(bundle) - != {"schemaVersion", "predicateType", "status", "anchorPlanSha256", "events"} - or bundle["schemaVersion"] != 1 - or bundle["predicateType"] != PREDICATE_TYPE - or bundle["status"] != "archive-health-anchor-history" - or bundle["anchorPlanSha256"] != _sha_bytes(_canonical(pdoc)) - ): - _fail("ARCHIVE_ANCHOR_BUNDLE_SCHEMA_INVALID") - if ( - set(receipt) != {"schemaVersion", "status", "events"} - or receipt["schemaVersion"] != 1 - or receipt["status"] != "archive-health-anchored" - ): - _fail("ARCHIVE_ANCHOR_RECEIPT_SCHEMA_INVALID") - events = bundle["events"] - recs = receipt["events"] - if ( - not isinstance(events, list) - or not events - or not isinstance(recs, list) - or len(events) != len(recs) - ): - _fail("ARCHIVE_ANCHOR_HISTORY_LENGTH_INVALID") - prev_head = None - prev_checkpoints = dict.fromkeys(plan["channels"]) - last_time = None - for idx, (e, rec) in enumerate(zip(events, recs), 1): - if not isinstance(e, dict) or set(e) != { - "sequence", - "run162WitnessSequence", - "run162WitnessChainHeadSha256", - "run162Artifacts", - "challenge", - "channelIds", - "channelCheckpointSha256s", - "observerResponseSha256s", - "anchorConsensusHeadSha256", - }: - _fail("ARCHIVE_ANCHOR_EVENT_SCHEMA_INVALID") - if e["sequence"] != idx or e["run162WitnessSequence"] != idx: - _fail("ARCHIVE_ANCHOR_SEQUENCE_INVALID") - if ( - not isinstance(rec, dict) - or set(rec) != {"sequence", "run162Documents", "channels"} - or rec["sequence"] != idx - ): - _fail("ARCHIVE_ANCHOR_RECEIPT_EVENT_INVALID") - rdocs = rec["run162Documents"] - if not isinstance(rdocs, dict) or set(rdocs) != run162._WITNESS_OUTPUT_NAMES: - _fail("ARCHIVE_ANCHOR_EMBEDDED_RUN162_INVALID") - arts = { - n: { - "sha256": _sha_bytes(_canonical(rdocs[n])), - "size": len(_canonical(rdocs[n])), - } - for n in sorted(rdocs) - } - if e["run162Artifacts"] != arts: - _fail("ARCHIVE_ANCHOR_RUN162_ARTIFACT_BINDING_INVALID") - with tempfile.TemporaryDirectory() as td: - t = Path(td) - for n, v in rdocs.items(): - _write(t / n, v) - vr = _verify_run162( - run160_dir=run160_dir, - run161_dir=run161_dir, - retention_root_path=retention_root_path, - retention_root_pin=retention_root_pin, - witness_root_path=witness_root_path, - witness_root_pin=witness_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - run162_dir=t, - now=current, - historical=True, - require_current_match=False, - ) - head = vr["witness_chain_head_sha256"] - if e["run162WitnessChainHeadSha256"] != head or vr["sequence"] != idx: - _fail("ARCHIVE_ANCHOR_RUN162_HEAD_BINDING_INVALID") - if idx == len(events) and require_current_match: - curdocs = {} - for n in run162._WITNESS_OUTPUT_NAMES: - curdocs[n], _ = _read_json( - Path(run162_dir) / n, "ARCHIVE_ANCHOR_CURRENT_RUN162" - ) - if curdocs != rdocs or head != current162["witness_chain_head_sha256"]: - _fail("ARCHIVE_ANCHOR_CURRENT_RUN162_MISMATCH") - channel_ids = sorted(plan["channels"]) - if e["channelIds"] != channel_ids: - _fail("ARCHIVE_ANCHOR_CHANNEL_SET_INVALID") - challenge = _sha_bytes( - _canonical( - { - "sequence": idx, - "run162WitnessSequence": idx, - "previousAnchorConsensusHeadSha256": prev_head, - "run162WitnessChainHeadSha256": head, - "run162Artifacts": arts, - "channelIds": channel_ids, - "previousChannelCheckpointSha256s": prev_checkpoints, - } - ) - ) - if e["challenge"] != challenge: - _fail("ARCHIVE_ANCHOR_CHALLENGE_INVALID") - items = rec["channels"] - if ( - not isinstance(items, list) - or [x.get("channelId") for x in items] != channel_ids - ): - _fail("ARCHIVE_ANCHOR_RECEIPT_CHANNELS_INVALID") - cps = [] - obs = [] - times = [] - newcp = {} - for item in items: - if not isinstance(item, dict) or set(item) != { - "channelId", - "anchorResponse", - "observerResponse", - }: - _fail("ARCHIVE_ANCHOR_RECEIPT_CHANNEL_INVALID") - cid = item["channelId"] - cfg = plan["channels"][cid] - a = _verify_channel_response( - item["anchorResponse"], - cid, - cfg, - sequence=idx, - witness_head=head, - previous=prev_checkpoints[cid], - challenge=challenge, - now=current, - historical=True, - ) - o = _verify_observer_response( - item["observerResponse"], - cid, - cfg, - anchor_sha=a["sha256"], - sequence=idx, - witness_head=head, - previous=prev_checkpoints[cid], - challenge=challenge, - now=current, - historical=True, - ) - cps.append(a["sha256"]) - obs.append(o["sha256"]) - times += [a["time"], o["time"]] - newcp[cid] = a["sha256"] - if e["channelCheckpointSha256s"] != cps or e["observerResponseSha256s"] != obs: - _fail("ARCHIVE_ANCHOR_RESPONSE_BINDING_INVALID") - core = {k: e[k] for k in e if k != "anchorConsensusHeadSha256"} - eh = _event_head(prev_head, core) - if e["anchorConsensusHeadSha256"] != eh: - _fail("ARCHIVE_ANCHOR_CONSENSUS_HEAD_INVALID") - if last_time is not None and min(times) - last_time > timedelta( - days=int(POLICY["maximum_anchor_interval_days"]) - ): - _fail("ARCHIVE_ANCHOR_INTERVAL_EXCEEDED") - last_time = max(times) - prev_head = eh - prev_checkpoints = newcp - state = docs[_DOC_ARCHIVE_ANCHOR_STATE] - active = docs["active-archive-anchor-evidence.json"] - bitem = { - "name": "release-archive-anchor-bundle.json", - "sha256": _sha_bytes(raws["release-archive-anchor-bundle.json"]), - "size": len(raws["release-archive-anchor-bundle.json"]), - } - est = { - "schemaVersion": 1, - "status": "trusted-archive-anchor", - "sequence": len(events), - "run162WitnessSequence": events[-1]["run162WitnessSequence"], - "anchorConsensusHeadSha256": prev_head, - "run162WitnessChainHeadSha256": events[-1]["run162WitnessChainHeadSha256"], - "channelIds": sorted(plan["channels"]), - "bundleArtifact": bitem, - } - if state != est: - _fail("ARCHIVE_ANCHOR_STATE_BINDING_INVALID") - eactive = { - "schemaVersion": 1, - "status": "active-archive-anchor", - "sequence": len(events), - "run162WitnessSequence": events[-1]["run162WitnessSequence"], - "anchorConsensusHeadSha256": prev_head, - "run162WitnessChainHeadSha256": events[-1]["run162WitnessChainHeadSha256"], - "channelCheckpointSha256s": events[-1]["channelCheckpointSha256s"], - "observerResponseSha256s": events[-1]["observerResponseSha256s"], - } - if active != eactive: - _fail("ARCHIVE_ANCHOR_ACTIVE_BINDING_INVALID") - if ( - not historical - and last_time is not None - and current - last_time - > timedelta(days=int(POLICY["maximum_anchor_interval_days"])) - ): - _fail("ARCHIVE_ANCHOR_ACTIVE_EPOCH_STALE") - return { - "ok": True, - "sequence": len(events), - "anchor_consensus_head_sha256": prev_head, - "run162_witness_chain_head_sha256": events[-1]["run162WitnessChainHeadSha256"], - } - - -def anchor_archive_health( # ruff: ignore[undocumented-public-function] - *, - run160_dir, - run161_dir, - retention_root_path, - retention_root_pin, - witness_root_path, - witness_root_pin, - bootstrap_pin, - recovery_pin, - attestation_pins, - run162_dir, - anchor_plan_path, - output_dir, - channels, - previous_output_dir=None, - now=None, -): - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - wdoc, _ = _read_json(witness_root_path, "ARCHIVE_ANCHOR_WITNESS_ROOT") - wroot = _verify_witness_root(wdoc, witness_root_pin, current) - pdoc, praw = _read_json(anchor_plan_path, "ARCHIVE_ANCHOR_PLAN") - plan = _verify_plan(pdoc, wroot, current) - vr = _verify_run162( - run160_dir=run160_dir, - run161_dir=run161_dir, - retention_root_path=retention_root_path, - retention_root_pin=retention_root_pin, - witness_root_path=witness_root_path, - witness_root_pin=witness_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - run162_dir=run162_dir, - now=current, - historical=True, - require_current_match=True, - ) - cidmap = {cid: (a, o) for cid, a, o in channels} - if set(cidmap) != set(plan["channels"]): - _fail("ARCHIVE_ANCHOR_CHANNEL_ADAPTER_SET_INVALID") - prev_events = [] - prev_recs = [] - prev_head = None - prev_cp = dict.fromkeys(plan["channels"]) - if previous_output_dir is not None: - pv = verify_anchor_history( - run160_dir=run160_dir, - run161_dir=run161_dir, - retention_root_path=retention_root_path, - retention_root_pin=retention_root_pin, - witness_root_path=witness_root_path, - witness_root_pin=witness_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - run162_dir=run162_dir, - anchor_plan_path=anchor_plan_path, - output_dir=previous_output_dir, - now=current, - historical=True, - require_current_match=False, - ) - pd, _ = _load_output(previous_output_dir) - prev_events = list(pd["release-archive-anchor-bundle.json"]["events"]) - prev_recs = list(pd["release-archive-anchor-receipt.json"]["events"]) - prev_head = pv["anchor_consensus_head_sha256"] - last = prev_recs[-1]["channels"] - prev_cp = { - x["channelId"]: _sha_bytes(_canonical(x["anchorResponse"])) for x in last - } - seq = len(prev_events) + 1 - if vr["sequence"] != seq: - _fail("ARCHIVE_ANCHOR_RUN162_SEQUENCE_INVALID") - rdocs = {} - for n in sorted(run162._WITNESS_OUTPUT_NAMES): - rdocs[n], _ = _read_json(Path(run162_dir) / n, "ARCHIVE_ANCHOR_RUN162") - arts = { - n: { - "sha256": _sha_bytes(_canonical(rdocs[n])), - "size": len(_canonical(rdocs[n])), - } - for n in sorted(rdocs) - } - cids = sorted(plan["channels"]) - head = vr["witness_chain_head_sha256"] - challenge = _sha_bytes( - _canonical( - { - "sequence": seq, - "run162WitnessSequence": seq, - "previousAnchorConsensusHeadSha256": prev_head, - "run162WitnessChainHeadSha256": head, - "run162Artifacts": arts, - "channelIds": cids, - "previousChannelCheckpointSha256s": prev_cp, - } - ) - ) - recitems = [] - cps = [] - obs = [] - for cid in cids: - cfg = plan["channels"][cid] - aadapter, oadapter = cidmap[cid] - req = { - "operation": "append", - "protocolVersion": 1, - "channelId": cid, - "sequence": seq, - "witnessChainHeadSha256": head, - "previousCheckpointSha256": prev_cp[cid], - "challenge": challenge, - "createOnly": True, - "overwrite": False, - } - ar = aadapter(req) - av = _verify_channel_response( - ar, - cid, - cfg, - sequence=seq, - witness_head=head, - previous=prev_cp[cid], - challenge=challenge, - now=current, - ) - orq = { - "operation": "verify", - "protocolVersion": 1, - "channelId": cid, - "sequence": seq, - "witnessChainHeadSha256": head, - "previousCheckpointSha256": prev_cp[cid], - "challenge": challenge, - "anchorResponseSha256": av["sha256"], - "readOnly": True, - } - ov = _verify_observer_response( - oadapter(orq), - cid, - cfg, - anchor_sha=av["sha256"], - sequence=seq, - witness_head=head, - previous=prev_cp[cid], - challenge=challenge, - now=current, - ) - recitems.append( - {"channelId": cid, "anchorResponse": ar, "observerResponse": ov["doc"]} - ) - cps.append(av["sha256"]) - obs.append(ov["sha256"]) - core = { - "sequence": seq, - "run162WitnessSequence": seq, - "run162WitnessChainHeadSha256": head, - "run162Artifacts": arts, - "challenge": challenge, - "channelIds": cids, - "channelCheckpointSha256s": cps, - "observerResponseSha256s": obs, - } - event = dict(core, anchorConsensusHeadSha256=_event_head(prev_head, core)) - events = [*prev_events, event] - recs = [ - *prev_recs, - { - "sequence": seq, - "run162Documents": rdocs, - "channels": recitems, - }, - ] - bundle = { - "schemaVersion": 1, - "predicateType": PREDICATE_TYPE, - "status": "archive-health-anchor-history", - "anchorPlanSha256": _sha_bytes(praw), - "events": events, - } - braw = _canonical(bundle) - state = { - "schemaVersion": 1, - "status": "trusted-archive-anchor", - "sequence": seq, - "run162WitnessSequence": seq, - "anchorConsensusHeadSha256": event["anchorConsensusHeadSha256"], - "run162WitnessChainHeadSha256": head, - "channelIds": cids, - "bundleArtifact": { - "name": "release-archive-anchor-bundle.json", - "sha256": _sha_bytes(braw), - "size": len(braw), - }, - } - active = { - "schemaVersion": 1, - "status": "active-archive-anchor", - "sequence": seq, - "run162WitnessSequence": seq, - "anchorConsensusHeadSha256": event["anchorConsensusHeadSha256"], - "run162WitnessChainHeadSha256": head, - "channelCheckpointSha256s": cps, - "observerResponseSha256s": obs, - } - receipt = {"schemaVersion": 1, "status": "archive-health-anchored", "events": recs} - protected = [ - Path(run160_dir), - Path(run161_dir), - Path(run162_dir), - Path(retention_root_path), - Path(witness_root_path), - Path(anchor_plan_path), - ] - target = _outside( - Path(output_dir), protected, "ARCHIVE_ANCHOR_OUTPUT_OVERLAPS_INPUT" - ) - if target.exists(): - _fail("ARCHIVE_ANCHOR_OUTPUT_EXISTS") - fp = ( - _dir_fp(Path(run160_dir), "ARCHIVE_ANCHOR_RUN160_DRIFT"), - _dir_fp(Path(run161_dir), "ARCHIVE_ANCHOR_RUN161_DRIFT"), - _dir_fp(Path(run162_dir), "ARCHIVE_ANCHOR_RUN162_DRIFT"), - ) - parent = target.parent - parent.mkdir(parents=True, exist_ok=True) - stage = Path(tempfile.mkdtemp(prefix=".run163-anchor-", dir=parent)) - try: - _write(stage / "release-archive-anchor-bundle.json", bundle) - _write(stage / _DOC_ARCHIVE_ANCHOR_STATE, state) - _write(stage / "active-archive-anchor-evidence.json", active) - _write(stage / "release-archive-anchor-receipt.json", receipt) - if fp != ( - _dir_fp(Path(run160_dir), "ARCHIVE_ANCHOR_RUN160_DRIFT"), - _dir_fp(Path(run161_dir), "ARCHIVE_ANCHOR_RUN161_DRIFT"), - _dir_fp(Path(run162_dir), "ARCHIVE_ANCHOR_RUN162_DRIFT"), - ): - _fail("ARCHIVE_ANCHOR_INPUT_DRIFT_DETECTED") - verify_anchor_history( - run160_dir=run160_dir, - run161_dir=run161_dir, - retention_root_path=retention_root_path, - retention_root_pin=retention_root_pin, - witness_root_path=witness_root_path, - witness_root_pin=witness_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - run162_dir=run162_dir, - anchor_plan_path=anchor_plan_path, - output_dir=stage, - now=current, - ) - stage.rename(target) - except Exception: - shutil.rmtree(stage, ignore_errors=True) - raise - return verify_anchor_history( - run160_dir=run160_dir, - run161_dir=run161_dir, - retention_root_path=retention_root_path, - retention_root_pin=retention_root_pin, - witness_root_path=witness_root_path, - witness_root_pin=witness_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - run162_dir=run162_dir, - anchor_plan_path=anchor_plan_path, - output_dir=target, - now=current, - ) - - -def _verify_recovery_root(doc, pin, now, historical=False): - # Reuse Run 162's threshold-root parser with the Run 162 recovery-root wire format. - try: - return run162._verify_threshold_root( - doc, - pin, - root_type="archive-retention-recovery-root", - schema_key="recovery_root_schema_version", - min_keys_key="min_recovery_keys", - min_threshold_key="min_recovery_threshold", - min_operators_key="min_recovery_operators", - now=now, - historical=historical, - channel=True, - ) - except Exception as e: - raise ArchiveAnchorError( - "ARCHIVE_ANCHOR_RECOVERY_ROOT_INVALID:" + str(e) - ) from e - - -def _recovery_subject( # ruff: ignore[too-many-positional-arguments] - s, - rr, - old, - new, - anchor_state, - now, - historical=False, -): - exp = { - "_type", - "specVersion", - "schemaVersion", - "recoveryId", - "issuedAt", - "recoveryRootSha256", - "oldWitnessRootSha256", - "newWitnessRootSha256", - "anchorConsensusHeadSha256", - "anchoredSequence", - "run162WitnessChainHeadSha256", - "compromisedOldKeyIds", - "selectedRecoveryKeyIds", - "rewrittenAnchorEpochs", - } - if ( - not isinstance(s, dict) - or set(s) != exp - or s["_type"] != "archive-witness-root-recovery" - or s["specVersion"] != str(POLICY["spec_version"]) - or s["schemaVersion"] != 1 - ): - _fail("ARCHIVE_ANCHOR_RECOVERY_SUBJECT_SCHEMA_INVALID") - _id(s["recoveryId"], "ARCHIVE_ANCHOR_RECOVERY_ID_INVALID") - issued = _dt(s["issuedAt"], "ARCHIVE_ANCHOR_RECOVERY_ISSUED_INVALID") - if not historical and ( - issued > now + timedelta(minutes=int(POLICY["max_clock_skew_minutes"])) - or now - issued > timedelta(minutes=int(POLICY["max_observation_age_minutes"])) - ): - _fail("ARCHIVE_ANCHOR_RECOVERY_SUBJECT_STALE") - if ( - s["recoveryRootSha256"] != rr["sha256"] - or s["oldWitnessRootSha256"] != old["sha256"] - or s["newWitnessRootSha256"] != new["sha256"] - ): - _fail("ARCHIVE_ANCHOR_RECOVERY_ROOT_BINDING_INVALID") - if ( - s["anchorConsensusHeadSha256"] != anchor_state["anchorConsensusHeadSha256"] - or s["anchoredSequence"] != anchor_state["sequence"] - or s["run162WitnessChainHeadSha256"] - != anchor_state["run162WitnessChainHeadSha256"] - ): - _fail("ARCHIVE_ANCHOR_RECOVERY_ANCHOR_BINDING_INVALID") - if s["rewrittenAnchorEpochs"] != []: - _fail("ARCHIVE_ANCHOR_RECOVERY_REWRITE_FORBIDDEN") - compromised = s["compromisedOldKeyIds"] - selected = s["selectedRecoveryKeyIds"] - if ( - not isinstance(compromised, list) - or not compromised - or compromised != sorted(compromised) - or any(x not in old["keys"] for x in compromised) - ): - _fail("ARCHIVE_ANCHOR_RECOVERY_COMPROMISED_KEYS_INVALID") - if ( - not isinstance(selected, list) - or selected != sorted(selected) - or len(selected) != rr["threshold"] - or any(x not in rr["keys"] for x in selected) - ): - _fail("ARCHIVE_ANCHOR_RECOVERY_SELECTED_INVALID") - if len({rr["keys"][x]["operator"] for x in selected}) < int( - POLICY["min_recovery_operators"] - ) or len({rr["keys"][x]["recoveryChannel"] for x in selected}) < int( - POLICY["min_recovery_channels"] - ): - _fail("ARCHIVE_ANCHOR_RECOVERY_QUORUM_INVALID") - oldops = {v["operator"] for v in old["keys"].values()} - newops = {v["operator"] for v in new["keys"].values()} - rrops = {v["operator"] for v in rr["keys"].values()} - if ( - oldops & newops - or oldops & rrops - or newops & rrops - or set(old["keys"]) & set(new["keys"]) - ): - _fail("ARCHIVE_ANCHOR_RECOVERY_AUTHORITY_SEPARATION_INVALID") - return {"issued": issued, "selected": selected} - - -def verify_witness_root_recovery( # ruff: ignore[undocumented-public-function] - *, - witness_root_path, - witness_root_pin, - new_witness_root_path, - recovery_root_path, - recovery_root_pin, - anchor_output_dir, - recovery_output_dir, - now=None, - historical=False, -): - logger.debug( - "verifying witness-root recovery output: %s", - recovery_output_dir, - ) - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - odoc, _ = _read_json(witness_root_path, "ARCHIVE_ANCHOR_OLD_WITNESS_ROOT") - old = _verify_witness_root(odoc, witness_root_pin, current, historical=True) - ndoc, nraw = _read_json(new_witness_root_path, "ARCHIVE_ANCHOR_NEW_WITNESS_ROOT") - new = _verify_witness_root(ndoc, _sha_bytes(nraw), current, historical=historical) - rdoc, _ = _read_json(recovery_root_path, "ARCHIVE_ANCHOR_RECOVERY_ROOT") - rr = _verify_recovery_root(rdoc, recovery_root_pin, current, historical=historical) - astate, _ = _read_json( - Path(anchor_output_dir) / _DOC_ARCHIVE_ANCHOR_STATE, - "ARCHIVE_ANCHOR_STATE", - ) - root = _regular_dir( - Path(recovery_output_dir), "ARCHIVE_ANCHOR_RECOVERY_OUTPUT_INVALID" - ) - if {p.name for p in root.iterdir()} != _RECOVERY_NAMES: - _fail("ARCHIVE_ANCHOR_RECOVERY_OUTPUT_ALLOWLIST_INVALID") - rec, rraw = _read_json( - root / "archive-witness-root-recovery-record.json", - "ARCHIVE_ANCHOR_RECOVERY_RECORD", - ) - receipt, _ = _read_json( - root / "archive-witness-root-recovery-receipt.json", - "ARCHIVE_ANCHOR_RECOVERY_RECEIPT", - ) - if ( - set(rec) - != { - "schemaVersion", - "status", - "subject", - "signatures", - "oldWitnessRoot", - "newWitnessRoot", - "recoveryRoot", - } - or rec["schemaVersion"] != 1 - or rec["status"] != "archive-witness-root-recovered" - ): - _fail("ARCHIVE_ANCHOR_RECOVERY_RECORD_SCHEMA_INVALID") - parsed = _recovery_subject( - rec["subject"], rr, old, new, astate, current, historical=historical - ) - sigs = rec["signatures"] - if ( - not isinstance(sigs, list) - or [x.get("keyId") for x in sigs] != parsed["selected"] - ): - _fail("ARCHIVE_ANCHOR_RECOVERY_SIGNATURE_SET_INVALID") - for x in sigs: - if not isinstance(x, dict) or set(x) != {"keyId", "channel", "signature"}: - _fail("ARCHIVE_ANCHOR_RECOVERY_SIGNATURE_SCHEMA_INVALID") - kid = x["keyId"] - if x["channel"] != rr["keys"][kid]["recoveryChannel"]: - _fail("ARCHIVE_ANCHOR_RECOVERY_CHANNEL_INVALID") - _verify( - rr["keys"][kid]["publicKey"], - x["signature"], - _canonical(rec["subject"]), - "ARCHIVE_ANCHOR_RECOVERY_SIGNATURE", - ) - expected = { - "schemaVersion": 1, - "status": "archive-witness-root-recovery-accepted", - "anchorConsensusHeadSha256": astate["anchorConsensusHeadSha256"], - "anchoredSequence": astate["sequence"], - "run162WitnessChainHeadSha256": astate["run162WitnessChainHeadSha256"], - "rewrittenAnchorEpochs": [], - "recordSha256": _sha_bytes(rraw), - "recoveredWitnessRootSha256": new["sha256"], - } - if receipt != expected: - _fail("ARCHIVE_ANCHOR_RECOVERY_RECEIPT_MISMATCH") - return { - "ok": True, - "recovered_witness_root_sha256": new["sha256"], - "anchor_consensus_head_sha256": astate["anchorConsensusHeadSha256"], - "rewritten_anchor_epochs": [], - } - - -def recover_witness_root( # ruff: ignore[undocumented-public-function] - *, - witness_root_path, - witness_root_pin, - new_witness_root_path, - recovery_root_path, - recovery_root_pin, - recovery_subject_path, - recovery_signatures_path, - anchor_output_dir, - output_dir, - now=None, -): - logger.info("recovering witness root into %s", output_dir) - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - odoc, _ = _read_json(witness_root_path, "ARCHIVE_ANCHOR_OLD_WITNESS_ROOT") - old = _verify_witness_root(odoc, witness_root_pin, current, historical=True) - ndoc, nraw = _read_json(new_witness_root_path, "ARCHIVE_ANCHOR_NEW_WITNESS_ROOT") - new = _verify_witness_root(ndoc, _sha_bytes(nraw), current) - rdoc, _ = _read_json(recovery_root_path, "ARCHIVE_ANCHOR_RECOVERY_ROOT") - rr = _verify_recovery_root(rdoc, recovery_root_pin, current) - astate, _ = _read_json( - Path(anchor_output_dir) / _DOC_ARCHIVE_ANCHOR_STATE, - "ARCHIVE_ANCHOR_STATE", - ) - subject, _ = _read_json(recovery_subject_path, "ARCHIVE_ANCHOR_RECOVERY_SUBJECT") - parsed = _recovery_subject(subject, rr, old, new, astate, current) - sd, _ = _read_json(recovery_signatures_path, "ARCHIVE_ANCHOR_RECOVERY_SIGNATURES") - if ( - set(sd) != {"schemaVersion", "signatures"} - or sd["schemaVersion"] != 1 - or not isinstance(sd["signatures"], list) - ): - _fail("ARCHIVE_ANCHOR_RECOVERY_SIGNATURES_SCHEMA_INVALID") - sigs = sd["signatures"] - if [x.get("keyId") for x in sigs] != parsed["selected"]: - _fail("ARCHIVE_ANCHOR_RECOVERY_SIGNATURE_SET_INVALID") - for x in sigs: - if not isinstance(x, dict) or set(x) != {"keyId", "channel", "signature"}: - _fail("ARCHIVE_ANCHOR_RECOVERY_SIGNATURE_SCHEMA_INVALID") - kid = x["keyId"] - if x["channel"] != rr["keys"][kid]["recoveryChannel"]: - _fail("ARCHIVE_ANCHOR_RECOVERY_CHANNEL_INVALID") - _verify( - rr["keys"][kid]["publicKey"], - x["signature"], - _canonical(subject), - "ARCHIVE_ANCHOR_RECOVERY_SIGNATURE", - ) - rec = { - "schemaVersion": 1, - "status": "archive-witness-root-recovered", - "subject": subject, - "signatures": sigs, - "oldWitnessRoot": {"sha256": old["sha256"], "rootId": old["rootId"]}, - "newWitnessRoot": {"sha256": new["sha256"], "rootId": new["rootId"]}, - "recoveryRoot": {"sha256": rr["sha256"], "rootId": rr["rootId"]}, - } - receipt = { - "schemaVersion": 1, - "status": "archive-witness-root-recovery-accepted", - "anchorConsensusHeadSha256": astate["anchorConsensusHeadSha256"], - "anchoredSequence": astate["sequence"], - "run162WitnessChainHeadSha256": astate["run162WitnessChainHeadSha256"], - "rewrittenAnchorEpochs": [], - "recordSha256": _sha_bytes(_canonical(rec)), - "recoveredWitnessRootSha256": new["sha256"], - } - target = _outside( - Path(output_dir), - [ - Path(witness_root_path), - Path(new_witness_root_path), - Path(recovery_root_path), - Path(recovery_subject_path), - Path(recovery_signatures_path), - Path(anchor_output_dir), - ], - "ARCHIVE_ANCHOR_RECOVERY_OUTPUT_OVERLAPS_INPUT", - ) - if target.exists(): - _fail("ARCHIVE_ANCHOR_RECOVERY_OUTPUT_EXISTS") - parent = target.parent - parent.mkdir(parents=True, exist_ok=True) - stage = Path(tempfile.mkdtemp(prefix=".run163-recovery-", dir=parent)) - try: - (stage / "recovered-archive-witness-root.json").write_bytes(nraw) - _write(stage / "archive-witness-root-recovery-record.json", rec) - _write(stage / "archive-witness-root-recovery-receipt.json", receipt) - verify_witness_root_recovery( - witness_root_path=witness_root_path, - witness_root_pin=witness_root_pin, - new_witness_root_path=stage / "recovered-archive-witness-root.json", - recovery_root_path=recovery_root_path, - recovery_root_pin=recovery_root_pin, - anchor_output_dir=anchor_output_dir, - recovery_output_dir=stage, - now=current, - ) - stage.rename(target) - except Exception: - shutil.rmtree(stage, ignore_errors=True) - raise - return verify_witness_root_recovery( - witness_root_path=witness_root_path, - witness_root_pin=witness_root_pin, - new_witness_root_path=target / "recovered-archive-witness-root.json", - recovery_root_path=recovery_root_path, - recovery_root_pin=recovery_root_pin, - anchor_output_dir=anchor_output_dir, - recovery_output_dir=target, - now=current, - ) - - -def _common_predecessor_args(p): - p.add_argument("--run160-dir", type=Path, required=True) - p.add_argument("--run161-dir", type=Path, required=True) - p.add_argument("--retention-root", type=Path, required=True) - p.add_argument("--retention-root-pin", required=True) - p.add_argument("--witness-root", type=Path, required=True) - p.add_argument("--witness-root-pin", required=True) - p.add_argument("--bootstrap-pin", required=True) - p.add_argument("--recovery-pin", required=True) - p.add_argument("--attestation-pin", action="append", default=[]) - p.add_argument("--run162-dir", type=Path, required=True) - p.add_argument("--anchor-plan", type=Path, required=True) - - -def main(argv=None): # ruff: ignore[undocumented-public-function] - import argparse # ruff: ignore[import-outside-top-level] - - ap = argparse.ArgumentParser(description=__doc__) - sp = ap.add_subparsers(dest="cmd", required=True) - a = sp.add_parser("anchor") - _common_predecessor_args(a) - a.add_argument("--adapter-config", type=Path, required=True) - a.add_argument("--output-dir", type=Path, required=True) - a.add_argument("--previous-output-dir", type=Path) - v = sp.add_parser("verify-anchor") - _common_predecessor_args(v) - v.add_argument("--output-dir", type=Path, required=True) - v.add_argument("--historical", action="store_true") - r = sp.add_parser("recover-witness-root") - r.add_argument("--witness-root", type=Path, required=True) - r.add_argument("--witness-root-pin", required=True) - r.add_argument("--new-witness-root", type=Path, required=True) - r.add_argument("--recovery-root", type=Path, required=True) - r.add_argument("--recovery-root-pin", required=True) - r.add_argument("--recovery-subject", type=Path, required=True) - r.add_argument("--recovery-signatures", type=Path, required=True) - r.add_argument("--anchor-output-dir", type=Path, required=True) - r.add_argument("--output-dir", type=Path, required=True) - vr = sp.add_parser("verify-recovery") - vr.add_argument("--witness-root", type=Path, required=True) - vr.add_argument("--witness-root-pin", required=True) - vr.add_argument("--new-witness-root", type=Path, required=True) - vr.add_argument("--recovery-root", type=Path, required=True) - vr.add_argument("--recovery-root-pin", required=True) - vr.add_argument("--anchor-output-dir", type=Path, required=True) - vr.add_argument("--recovery-output-dir", type=Path, required=True) - vr.add_argument("--historical", action="store_true") - ns = ap.parse_args(argv) - logger.info("running archive-anchor command: %s", ns.cmd) - if ns.cmd == "anchor": - cfg, _ = _read_json(ns.adapter_config, "ARCHIVE_ANCHOR_ADAPTER_CONFIG") - if set(cfg) != {"channels"} or not isinstance(cfg["channels"], dict): - _fail("ARCHIVE_ANCHOR_ADAPTER_CONFIG_SCHEMA_INVALID") - adapters = [] - for cid, x in sorted(cfg["channels"].items()): - if ( - not isinstance(x, dict) - or set(x) != {"appendCommand", "observerCommand"} - or not isinstance(x["appendCommand"], list) - or not isinstance(x["observerCommand"], list) - or not all( - isinstance(y, str) and y - for y in x["appendCommand"] + x["observerCommand"] - ) - ): - _fail("ARCHIVE_ANCHOR_ADAPTER_CONFIG_SCHEMA_INVALID") - adapters.append( - ( - cid, - command_channel(x["appendCommand"]), - command_observer(x["observerCommand"]), - ) - ) - out = anchor_archive_health( - run160_dir=ns.run160_dir, - run161_dir=ns.run161_dir, - retention_root_path=ns.retention_root, - retention_root_pin=ns.retention_root_pin, - witness_root_path=ns.witness_root, - witness_root_pin=ns.witness_root_pin, - bootstrap_pin=ns.bootstrap_pin, - recovery_pin=ns.recovery_pin, - attestation_pins=ns.attestation_pin, - run162_dir=ns.run162_dir, - anchor_plan_path=ns.anchor_plan, - output_dir=ns.output_dir, - channels=adapters, - previous_output_dir=ns.previous_output_dir, - ) - elif ns.cmd == "verify-anchor": - out = verify_anchor_history( - run160_dir=ns.run160_dir, - run161_dir=ns.run161_dir, - retention_root_path=ns.retention_root, - retention_root_pin=ns.retention_root_pin, - witness_root_path=ns.witness_root, - witness_root_pin=ns.witness_root_pin, - bootstrap_pin=ns.bootstrap_pin, - recovery_pin=ns.recovery_pin, - attestation_pins=ns.attestation_pin, - run162_dir=ns.run162_dir, - anchor_plan_path=ns.anchor_plan, - output_dir=ns.output_dir, - historical=ns.historical, - ) - elif ns.cmd == "recover-witness-root": - out = recover_witness_root( - witness_root_path=ns.witness_root, - witness_root_pin=ns.witness_root_pin, - new_witness_root_path=ns.new_witness_root, - recovery_root_path=ns.recovery_root, - recovery_root_pin=ns.recovery_root_pin, - recovery_subject_path=ns.recovery_subject, - recovery_signatures_path=ns.recovery_signatures, - anchor_output_dir=ns.anchor_output_dir, - output_dir=ns.output_dir, - ) - else: - out = verify_witness_root_recovery( - witness_root_path=ns.witness_root, - witness_root_pin=ns.witness_root_pin, - new_witness_root_path=ns.new_witness_root, - recovery_root_path=ns.recovery_root, - recovery_root_pin=ns.recovery_root_pin, - anchor_output_dir=ns.anchor_output_dir, - recovery_output_dir=ns.recovery_output_dir, - historical=ns.historical, - ) - logger.info("completed archive-anchor command: %s", ns.cmd) - # sys.stdout.write(json.dumps(out, sort_keys=True, separators=(",", ":")) + "\n") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/security/archive_native_status_evidence.py b/security/archive_native_status_evidence.py deleted file mode 100644 index 1a42903fe57876c1d7803d78a7e5aa7b9a10e290..0000000000000000000000000000000000000000 --- a/security/archive_native_status_evidence.py +++ /dev/null @@ -1,1546 +0,0 @@ -""" -Run 160: immutably replicate and recover exact Run 159 native-status evidence. - -The Run 159 bundle already preserves raw DER CRL/OCSP bytes and verified vendor evidence. -Run 160 treats that complete canonical output as one immutable archival subject, binds the -configured archive/verifier membership into deterministic bytes, requires independent -remote read-back through a second authority plane, and supports fail-closed recovery from -multiple read-only archives. -""" - -from __future__ import annotations - -import argparse -import base64 -import hashlib -import json -import logging -import os -import re -import shutil -import subprocess -import sys -import tempfile -import threading -from datetime import datetime, timedelta, timezone -from pathlib import Path -from typing import Any, Callable, Iterable -from urllib.parse import urlsplit - -import tomllib - -logger = logging.getLogger(__name__) - -try: - from . import verify_native_status_provenance as native -except (ImportError, ValueError) as exc: - import importlib.util - - _here = Path(__file__).resolve().parent - _spec = importlib.util.spec_from_file_location( - "_run159_native_for_archive", _here / "verify_native_status_provenance.py" - ) - if _spec is None or _spec.loader is None: - raise ImportError("verify_native_status_provenance.py") from exc - native = importlib.util.module_from_spec(_spec) - sys.modules[_spec.name] = native - _spec.loader.exec_module(native) - -HERE = Path(__file__).resolve().parent -POLICY = tomllib.loads((HERE / "release_native_archive_policy.toml").read_text()) -PREDICATE_TYPE = str(POLICY["predicate_type"]) -_HEX64 = re.compile(r"^[0-9a-f]{64}$") -_ID = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:/@+-]{0,511}$") -_CHUNK = 1024 * 1024 - - -# ── Verified-state document names ──────────────────────────────────────── -# -# One constant per release-evidence document. The values are the on-disk -# artifact names and are unchanged; only the repetition is removed. Reading -# a document by a name that says which document it is also keeps static -# analysis from reading the filename token 'trusted' as 'confidential': -# these files carry published Merkle roots and SHA-256 digests, which must -# stay in clear text for any third party to verify them. -_DOC_NATIVE_EVIDENCE_ARCHIVE_STATE = "trusted-native-evidence-archive-state.json" -_DOC_NATIVE_STATUS_STATE = "trusted-native-status-state.json" - - -_NATIVE_NAMES = { - "release-native-status-bundle.json", - _DOC_NATIVE_STATUS_STATE, - "active-native-status-evidence.json", - "release-native-status-receipt.json", -} -_ARCHIVE_NAMES = { - "release-native-evidence-archive.json", - _DOC_NATIVE_EVIDENCE_ARCHIVE_STATE, - "release-native-evidence-archive-receipt.json", -} - - -class NativeArchiveError(RuntimeError): - """Run 160 native-evidence archival invariant failed.""" - - -def _fail(code: str) -> None: - raise NativeArchiveError(code) - - -def _canonical(value: Any) -> bytes: - return ( - json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False) - + "\n" - ).encode("utf-8") - - -def _sha_bytes(raw: bytes) -> str: - return hashlib.sha256(raw).hexdigest() - - -def _sha(path: Path) -> str: - h = hashlib.sha256() - with path.open("rb") as handle: - for chunk in iter(lambda: handle.read(_CHUNK), b""): - h.update(chunk) - return h.hexdigest() - - -def _write(path: Path, value: dict[str, Any]) -> None: - path.write_bytes(_canonical(value)) - - -def _loads(raw: bytes, code: str) -> dict[str, Any]: - def hook(pairs): - out: dict[str, Any] = {} - for key, value in pairs: - if key in out: - _fail(code + "_DUPLICATE_KEY") - out[key] = value - return out - - try: - value = json.loads(raw.decode("utf-8"), object_pairs_hook=hook) - except NativeArchiveError: - raise - except Exception as exc: - raise NativeArchiveError(code + "_JSON_INVALID") from exc - if not isinstance(value, dict): - _fail(code + "_SCHEMA_INVALID") - return value - - -def _read_json(path: Path, code: str) -> tuple[dict[str, Any], bytes]: - if path.is_symlink() or not path.is_file(): - _fail(code + "_INVALID") - if path.stat().st_size > int(POLICY["max_json_bytes"]): - _fail(code + "_TOO_LARGE") - raw = path.read_bytes() - doc = _loads(raw, code) - if raw != _canonical(doc): - _fail(code + "_NOT_CANONICAL") - return doc, raw - - -def _identity(value: Any, code: str) -> str: - if not isinstance(value, str): - _fail(code) - value = value.strip() - if ( - not value - or len(value) > 512 # ruff: ignore[magic-value-comparison] - or ".." in value - or "?" in value - or "#" in value - or "\x00" in value - ): - _fail(code) - if ( - any( - ord(c) < 32 # ruff: ignore[magic-value-comparison] - or ord(c) == 127 # ruff: ignore[magic-value-comparison] - for c in value - ) - or _ID.fullmatch(value) is None - ): - _fail(code) - return value - - -def _hex(value: Any, code: str) -> str: - if not isinstance(value, str) or _HEX64.fullmatch(value) is None: - _fail(code) - return value - - -def _size(value: Any, code: str) -> int: - if not isinstance(value, int) or isinstance(value, bool) or value < 0: - _fail(code) - return value - - -def _safe_locator(value: Any, code: str) -> str: - if ( - not isinstance(value, str) - or not value - or len(value) > 2048 # ruff: ignore[magic-value-comparison] - or "\x00" in value - ): - _fail(code) - try: - parsed = urlsplit(value) - except ValueError: - _fail(code) - if ( - not parsed.scheme - or parsed.scheme.lower() == "file" - or parsed.query - or parsed.fragment - or parsed.username is not None - or parsed.password is not None - ): - _fail(code) - if ".." in parsed.path.split("/"): - _fail(code) - return value - - -def _dt(value: Any, code: str) -> datetime: - if not isinstance(value, str) or not value.endswith("Z"): - _fail(code) - try: - return datetime.fromisoformat(value[:-1] + "+00:00").astimezone(timezone.utc) - except ValueError as exc: - raise NativeArchiveError(code) from exc - - -def _fresh(value: Any, code: str, *, now: datetime) -> str: - parsed = _dt(value, code) - skew = timedelta(minutes=int(POLICY["max_clock_skew_minutes"])) - age = timedelta(minutes=int(POLICY["max_result_age_minutes"])) - if parsed > now + skew: - _fail(code + "_FROM_FUTURE") - if now - parsed > age: - _fail(code + "_STALE") - return value - - -def _regular_dir(path: Path, code: str) -> Path: - if path.is_symlink() or not path.is_dir(): - _fail(code) - return path.resolve() - - -def _outside(path: Path, protected: Iterable[Path], code: str) -> Path: - target = path.expanduser().resolve() - for item in protected: - root = item.resolve() - if target == root or root in target.parents: - _fail(code) - return target - - -def _artifact(name: str, raw: bytes) -> dict[str, Any]: - return {"name": name, "sha256": _sha_bytes(raw), "size": len(raw)} - - -def _artifact_doc(value: Any, code: str) -> dict[str, Any]: - if not isinstance(value, dict) or set(value) != {"name", "sha256", "size"}: - _fail(code + "_SCHEMA_INVALID") - name = value.get("name") - if ( - not isinstance(name, str) - or not name - or Path(name).name != name - or name in {".", ".."} - ): - _fail(code + "_NAME_INVALID") - return { - "name": name, - "sha256": _hex(value.get("sha256"), code + "_SHA256_INVALID"), - "size": _size(value.get("size"), code + "_SIZE_INVALID"), - } - - -def _native_docs(root: Path) -> tuple[dict[str, dict[str, Any]], dict[str, bytes]]: - root = _regular_dir(root, "NATIVE_ARCHIVE_RUN159_DIR_INVALID") - if {p.name for p in root.iterdir()} != _NATIVE_NAMES: - _fail("NATIVE_ARCHIVE_RUN159_ALLOWLIST_MISMATCH") - docs: dict[str, dict[str, Any]] = {} - raws: dict[str, bytes] = {} - for name in sorted(_NATIVE_NAMES): - doc, raw = _read_json( - root / name, - "NATIVE_ARCHIVE_RUN159_" + name.upper().replace("-", "_").replace(".", "_"), - ) - docs[name] = doc - raws[name] = raw - receipt = docs["release-native-status-receipt.json"] - expected = { - "bundle": _artifact( - "release-native-status-bundle.json", - raws["release-native-status-bundle.json"], - ), - "state": _artifact(_DOC_NATIVE_STATUS_STATE, raws[_DOC_NATIVE_STATUS_STATE]), - "activeEvidence": _artifact( - "active-native-status-evidence.json", - raws["active-native-status-evidence.json"], - ), - } - for key, item in expected.items(): - if receipt.get(key) != item: - _fail("NATIVE_ARCHIVE_RUN159_RECEIPT_REBIND_FAILED") - return docs, raws - - -def _source_inventory( # ruff: ignore[too-many-branches] - bundle: dict[str, Any], -) -> list[dict[str, Any]]: - out: list[dict[str, Any]] = [] - events = bundle.get("events") - if not isinstance(events, list) or not events: - _fail("NATIVE_ARCHIVE_EVENTS_INVALID") - for event in events: - sequence = event.get("sequence") - if not isinstance(sequence, int) or isinstance(sequence, bool) or sequence < 1: - _fail("NATIVE_ARCHIVE_EVENT_SEQUENCE_INVALID") - sources = event.get("sources") - if not isinstance(sources, list): - _fail("NATIVE_ARCHIVE_SOURCES_INVALID") - for source in sources: - if not isinstance(source, dict): - _fail("NATIVE_ARCHIVE_SOURCE_INVALID") - kind = source.get("type") - if kind not in {"crl", "ocsp"}: - _fail("NATIVE_ARCHIVE_SOURCE_KIND_INVALID") - der = source.get("der") - try: - raw = base64.b64decode(der, validate=True) - except Exception as exc: - raise NativeArchiveError("NATIVE_ARCHIVE_SOURCE_DER_INVALID") from exc - sha = _hex(source.get("sha256"), "NATIVE_ARCHIVE_SOURCE_SHA_INVALID") - if _sha_bytes(raw) != sha: - _fail("NATIVE_ARCHIVE_SOURCE_DER_HASH_MISMATCH") - item = { - "sequence": sequence, - "type": kind, - "sourceId": _identity( - source.get("sourceId"), "NATIVE_ARCHIVE_SOURCE_ID_INVALID" - ), - "sha256": sha, - "size": len(raw), - } - if kind == "ocsp" and source.get("responderCertDer") is not None: - try: - cert_raw = base64.b64decode( - source["responderCertDer"], validate=True - ) - except Exception as exc: - raise NativeArchiveError( - "NATIVE_ARCHIVE_RESPONDER_DER_INVALID" - ) from exc - item["responderCertificateSha256"] = _sha_bytes(cert_raw) - item["responderCertificateSize"] = len(cert_raw) - out.append(item) - vendor = event.get("vendorEvidence", []) - if not isinstance(vendor, list): - _fail("NATIVE_ARCHIVE_VENDOR_EVIDENCE_INVALID") - for item in vendor: - if not isinstance(item, dict): - _fail("NATIVE_ARCHIVE_VENDOR_EVIDENCE_INVALID") - out.append( - { - "sequence": sequence, - "type": "vendor", - "sourceId": _identity( - item.get("profile"), "NATIVE_ARCHIVE_VENDOR_PROFILE_INVALID" - ), - "sha256": _hex( - item.get("rawEvidenceSha256"), - "NATIVE_ARCHIVE_VENDOR_SHA_INVALID", - ), - "size": len( - base64.b64decode(item.get("rawEvidenceBase64"), validate=True) - ), - } - ) - out.sort(key=lambda x: (x["sequence"], x["type"], x["sourceId"], x["sha256"])) - if len( - {(x["sequence"], x["type"], x["sourceId"], x["sha256"]) for x in out} - ) != len(out): - _fail("NATIVE_ARCHIVE_SOURCE_INVENTORY_DUPLICATE") - return out - - -def _command_adapter( - command: list[str], *, prefix: str -) -> Callable[[dict[str, Any]], dict[str, Any]]: - if not command: - _fail(prefix + "_COMMAND_INVALID") - - def call(request: dict[str, Any]) -> dict[str, Any]: - process = ( - # lint - subprocess.Popen( # ruff: ignore[subprocess-without-shell-equals-true] - command, - stdin=subprocess.PIPE, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, - text=False, - close_fds=True, - env={"PATH": os.defpath, "LC_ALL": "C", "LANG": "C"}, - ) - ) - assert process.stdin is not None # ruff: ignore[assert] - assert process.stdout is not None # ruff: ignore[assert] - assert process.stderr is not None # ruff: ignore[assert] - stdin = process.stdin - process.stdin = None - stdout_chunks: list[bytes] = [] - stderr_chunks: list[bytes] = [] - overflow = threading.Event() - limit = int(POLICY["max_adapter_output_bytes"]) - - def drain(stream, chunks): - total = 0 - try: - while True: - chunk = stream.read(65536) - if not chunk: - break - total += len(chunk) - if total > limit: - overflow.set() - process.kill() - break - chunks.append(chunk) - finally: - stream.close() - - threads = [ - threading.Thread(target=drain, args=(process.stdout, stdout_chunks)), - threading.Thread(target=drain, args=(process.stderr, stderr_chunks)), - ] - for thread in threads: - thread.start() - try: - try: - stdin.write(_canonical(request)) - stdin.close() - process.wait(timeout=int(POLICY["adapter_timeout_seconds"])) - except subprocess.TimeoutExpired as exc: - process.kill() - process.wait() - raise NativeArchiveError(prefix + "_TIMEOUT") from exc - for thread in threads: - thread.join() - if overflow.is_set(): - _fail(prefix + "_OUTPUT_TOO_LARGE") - if process.returncode != 0: - _fail(prefix + "_FAILED") - return _loads(b"".join(stdout_chunks), prefix + "_OUTPUT") - finally: - if process.poll() is None: - process.kill() - process.wait() - try: # ruff: ignore[suppressible-exception] - stdin.close() - except OSError: - pass - for thread in threads: - thread.join() - - return call - - -def command_archive(command: list[str]): # ruff: ignore[undocumented-public-function] - return _command_adapter(command, prefix="NATIVE_ARCHIVE_ADAPTER") - - -def command_verifier(command: list[str]): # ruff: ignore[undocumented-public-function] - return _command_adapter(command, prefix="NATIVE_ARCHIVE_VERIFIER_ADAPTER") - - -def command_recovery(command: list[str]): # ruff: ignore[undocumented-public-function] - return _command_adapter(command, prefix="NATIVE_ARCHIVE_RECOVERY_ADAPTER") - - -ArchiveAdapter = Callable[[dict[str, Any]], dict[str, Any]] -VerifierAdapter = Callable[[dict[str, Any]], dict[str, Any]] -RecoveryAdapter = Callable[[dict[str, Any]], dict[str, Any]] - - -def _archive_id(chain_head: str, artifact_sha: str, identity: str) -> str: - return ( - "native-evidence-" - + hashlib.sha256( - (chain_head + "\0" + artifact_sha + "\0" + identity).encode() - ).hexdigest()[:32] - ) - - -def _validate_archive_response( - value: dict[str, Any], - *, - operation: str, - archive_id: str, - identity: str, - operator: str, - artifact: dict[str, Any], - now: datetime, - historical: bool = False, -) -> dict[str, Any]: - expected = { - "schemaVersion", - "operation", - "archiveId", - "status", - "archive", - "artifact", - "guarantees", - "verifiedAt", - } - if ( - set(value) != expected - or value.get("schemaVersion") != int(POLICY["archive_protocol_version"]) - or value.get("operation") != operation - or value.get("archiveId") != archive_id - ): - _fail("NATIVE_ARCHIVE_RESULT_SCHEMA_INVALID") - allowed = {"created", "present"} if operation == "bind" else {"present"} - if value.get("status") not in allowed: - _fail("NATIVE_ARCHIVE_STATUS_INVALID") - authority = value.get("archive") - if ( - not isinstance(authority, dict) - or set(authority) - != { - "identity", - "operator", - "nativeStatusCredentialsReused", - "verifierCredentialsReused", - } - or authority.get("identity") != identity - or authority.get("operator") != operator - ): - _fail("NATIVE_ARCHIVE_AUTHORITY_INVALID") - if ( - authority.get("nativeStatusCredentialsReused") is not False - or authority.get("verifierCredentialsReused") is not False - ): - _fail("NATIVE_ARCHIVE_CREDENTIAL_REUSE") - if _artifact_doc(value.get("artifact"), "NATIVE_ARCHIVE_ARTIFACT") != artifact: - _fail("NATIVE_ARCHIVE_ARTIFACT_MISMATCH") - guarantees = value.get("guarantees") - expected_g = { - "createOnly", - "overwrite", - "remoteReadbackVerified", - "immutability", - "locator", - } - if ( - not isinstance(guarantees, dict) - or set(guarantees) != expected_g - or guarantees.get("createOnly") is not True - or guarantees.get("overwrite") is not False - or guarantees.get("remoteReadbackVerified") is not True - ): - _fail("NATIVE_ARCHIVE_GUARANTEES_INVALID") - immutability = guarantees.get("immutability") - if immutability not in set(POLICY["allowed_archive_immutability"]): - _fail("NATIVE_ARCHIVE_IMMUTABILITY_INVALID") - verified_at = value.get("verifiedAt") - if historical: - _dt(verified_at, "NATIVE_ARCHIVE_VERIFIED_AT") - else: - _fresh(verified_at, "NATIVE_ARCHIVE_VERIFIED_AT", now=now) - return { - "locator": _safe_locator( - guarantees.get("locator"), "NATIVE_ARCHIVE_LOCATOR_INVALID" - ), - "immutability": immutability, - "verifiedAt": verified_at, - } - - -def _validate_verifier_response( - value: dict[str, Any], - *, - archive_id: str, - identity: str, - operator: str, - archive_identity: str, - archive_operator: str, - locator: str, - artifact: dict[str, Any], - now: datetime, - historical: bool = False, -) -> dict[str, Any]: - expected = { - "schemaVersion", - "operation", - "archiveId", - "status", - "verifier", - "archiveIdentity", - "locator", - "artifact", - "proof", - "verifiedAt", - } - if ( - set(value) != expected - or value.get("schemaVersion") != int(POLICY["verifier_protocol_version"]) - or value.get("operation") != "verify" - or value.get("archiveId") != archive_id - or value.get("status") != "present" - ): - _fail("NATIVE_ARCHIVE_VERIFIER_RESULT_SCHEMA_INVALID") - verifier = value.get("verifier") - if ( - not isinstance(verifier, dict) - or set(verifier) - != { - "identity", - "operator", - "readOnly", - "archiveCredentialsReused", - "nativeStatusCredentialsReused", - } - or verifier.get("identity") != identity - or verifier.get("operator") != operator - ): - _fail("NATIVE_ARCHIVE_VERIFIER_AUTHORITY_INVALID") - if ( - verifier.get("readOnly") is not True - or verifier.get("archiveCredentialsReused") is not False - or verifier.get("nativeStatusCredentialsReused") is not False - ): - _fail("NATIVE_ARCHIVE_VERIFIER_CREDENTIAL_REUSE") - if identity == archive_identity or operator == archive_operator: - _fail("NATIVE_ARCHIVE_VERIFIER_NOT_INDEPENDENT") - if ( - value.get("archiveIdentity") != archive_identity - or _safe_locator( - value.get("locator"), "NATIVE_ARCHIVE_VERIFIER_LOCATOR_INVALID" - ) - != locator - ): - _fail("NATIVE_ARCHIVE_VERIFIER_REBIND_FAILED") - if ( - _artifact_doc(value.get("artifact"), "NATIVE_ARCHIVE_VERIFIER_ARTIFACT") - != artifact - ): - _fail("NATIVE_ARCHIVE_VERIFIER_ARTIFACT_MISMATCH") - if value.get("proof") != { - "remoteReadbackVerified": True, - "sha256Verified": True, - "sizeVerified": True, - "independentReader": True, - }: - _fail("NATIVE_ARCHIVE_VERIFIER_PROOF_INVALID") - verified_at = value.get("verifiedAt") - if historical: - _dt(verified_at, "NATIVE_ARCHIVE_VERIFIER_VERIFIED_AT") - else: - _fresh(verified_at, "NATIVE_ARCHIVE_VERIFIER_VERIFIED_AT", now=now) - return {"verifiedAt": verified_at} - - -def _archive_docs(root: Path) -> tuple[dict[str, dict[str, Any]], dict[str, bytes]]: - root = _regular_dir(root, "NATIVE_ARCHIVE_OUTPUT_DIR_INVALID") - if {p.name for p in root.iterdir()} != _ARCHIVE_NAMES: - _fail("NATIVE_ARCHIVE_OUTPUT_ALLOWLIST_MISMATCH") - docs: dict[str, dict[str, Any]] = {} - raws: dict[str, bytes] = {} - for name in sorted(_ARCHIVE_NAMES): - doc, raw = _read_json( - root / name, - "NATIVE_ARCHIVE_OUTPUT_" + name.upper().replace("-", "_").replace(".", "_"), - ) - docs[name] = doc - raws[name] = raw - return docs, raws - - -def _rehydrate_native(output: dict[str, Any], target: Path) -> None: - if not isinstance(output, dict) or set(output) != _NATIVE_NAMES: - _fail("NATIVE_ARCHIVE_EMBEDDED_RUN159_INVALID") - target.mkdir(parents=True, exist_ok=False) - for name in sorted(_NATIVE_NAMES): - doc = output.get(name) - if not isinstance(doc, dict): - _fail("NATIVE_ARCHIVE_EMBEDDED_RUN159_INVALID") - _write(target / name, doc) - - -def _verify_archive_payload( # ruff: ignore[too-many-branches] - payload: dict[str, Any], - *, - bootstrap_pin: str, - recovery_pin: str, - attestation_pins: list[str], - now: datetime, - historical: bool, -) -> dict[str, Any]: - expected = { - "schemaVersion", - "predicateType", - "status", - "nativeStatus", - "sourceInventory", - "archivePolicy", - } - if ( - set(payload) != expected - or payload.get("schemaVersion") != int(POLICY["archive_schema_version"]) - or payload.get("predicateType") != PREDICATE_TYPE - or payload.get("status") != "preserved-native-status-evidence" - ): - _fail("NATIVE_ARCHIVE_PAYLOAD_SCHEMA_INVALID") - native_meta = payload.get("nativeStatus") - if not isinstance(native_meta, dict) or set(native_meta) != { - "sequence", - "nativeStatusChainHeadSha256", - "output", - }: - _fail("NATIVE_ARCHIVE_NATIVE_META_INVALID") - sequence = _size(native_meta.get("sequence"), "NATIVE_ARCHIVE_SEQUENCE_INVALID") - if sequence < 1: - _fail("NATIVE_ARCHIVE_SEQUENCE_INVALID") - chain_head = _hex( - native_meta.get("nativeStatusChainHeadSha256"), - "NATIVE_ARCHIVE_CHAIN_HEAD_INVALID", - ) - policy = payload.get("archivePolicy") - if not isinstance(policy, dict) or set(policy) != { - "minimumArchives", - "minimumArchiveOperators", - "minimumVerifierOperators", - "targets", - }: - _fail("NATIVE_ARCHIVE_POLICY_SCHEMA_INVALID") - targets = policy.get("targets") - if not isinstance(targets, list) or len(targets) < int(POLICY["min_archives"]): - _fail("NATIVE_ARCHIVE_TARGETS_INVALID") - seen_a: set[str] = set() - seen_v: set[str] = set() - archive_ops: set[str] = set() - verifier_ops: set[str] = set() - for target in targets: - if not isinstance(target, dict) or set(target) != { - "archiveIdentity", - "archiveOperator", - "verifierIdentity", - "verifierOperator", - }: - _fail("NATIVE_ARCHIVE_TARGET_SCHEMA_INVALID") - ai = _identity(target.get("archiveIdentity"), "NATIVE_ARCHIVE_IDENTITY_INVALID") - ao = _identity(target.get("archiveOperator"), "NATIVE_ARCHIVE_OPERATOR_INVALID") - vi = _identity( - target.get("verifierIdentity"), "NATIVE_ARCHIVE_VERIFIER_IDENTITY_INVALID" - ) - vo = _identity( - target.get("verifierOperator"), "NATIVE_ARCHIVE_VERIFIER_OPERATOR_INVALID" - ) - if ai in seen_a or vi in seen_v or ai == vi or ao == vo: - _fail("NATIVE_ARCHIVE_TARGET_INDEPENDENCE_INVALID") - seen_a.add(ai) - seen_v.add(vi) - archive_ops.add(ao) - verifier_ops.add(vo) - if archive_ops & verifier_ops: - _fail("NATIVE_ARCHIVE_OPERATOR_PLANES_OVERLAP") - if len(archive_ops) < int(POLICY["min_archive_operators"]) or len( - verifier_ops - ) < int(POLICY["min_verifier_operators"]): - _fail("NATIVE_ARCHIVE_OPERATOR_QUORUM_INVALID") - if ( - policy.get("minimumArchives") != int(POLICY["min_archives"]) - or policy.get("minimumArchiveOperators") != int(POLICY["min_archive_operators"]) - or policy.get("minimumVerifierOperators") - != int(POLICY["min_verifier_operators"]) - ): - _fail("NATIVE_ARCHIVE_POLICY_VALUE_INVALID") - tmp = Path(tempfile.mkdtemp(prefix=".run160-rehydrate-")) - try: - native_dir = tmp / "native" - _rehydrate_native(native_meta["output"], native_dir) - try: - result = native.verify_native_status( - output_dir=native_dir, - expected_bootstrap_root_sha256=bootstrap_pin, - expected_recovery_root_sha256=recovery_pin, - expected_attestation_root_sha256=attestation_pins, - now=now, - historical=historical, - ) - except native.NativeStatusError as exc: - raise NativeArchiveError( - "NATIVE_ARCHIVE_RUN159_REPLAY_INVALID:" + str(exc) - ) from exc - if ( - result.get("sequence") != sequence - or result.get("native_status_chain_head_sha256") != chain_head - ): - _fail("NATIVE_ARCHIVE_RUN159_REPLAY_MISMATCH") - docs, _ = _native_docs(native_dir) - inventory = _source_inventory(docs["release-native-status-bundle.json"]) - if payload.get("sourceInventory") != inventory: - _fail("NATIVE_ARCHIVE_SOURCE_INVENTORY_MISMATCH") - finally: - shutil.rmtree(tmp, ignore_errors=True) - return { - "sequence": sequence, - "chain_head": chain_head, - "source_count": len(payload["sourceInventory"]), - } - - -def verify_native_archive( # ruff: ignore[too-many-branches, undocumented-public-function] - *, - output_dir: Path, - expected_bootstrap_root_sha256: str, - expected_recovery_root_sha256: str, - expected_attestation_root_sha256: list[str], - now: datetime | None = None, - historical: bool = False, -) -> dict[str, Any]: - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - docs, raws = _archive_docs(output_dir) - payload = docs["release-native-evidence-archive.json"] - info = _verify_archive_payload( - payload, - bootstrap_pin=expected_bootstrap_root_sha256, - recovery_pin=expected_recovery_root_sha256, - attestation_pins=expected_attestation_root_sha256, - now=current, - historical=historical, - ) - payload_item = _artifact( - "release-native-evidence-archive.json", - raws["release-native-evidence-archive.json"], - ) - state = docs[_DOC_NATIVE_EVIDENCE_ARCHIVE_STATE] - expected_state = { - "schemaVersion", - "status", - "sequence", - "nativeStatusChainHeadSha256", - "archiveArtifact", - "sourceInventorySha256", - } - if ( - set(state) != expected_state - or state.get("schemaVersion") != int(POLICY["state_schema_version"]) - or state.get("status") != "trusted-native-status-archive" - or state.get("sequence") != info["sequence"] - or state.get("nativeStatusChainHeadSha256") != info["chain_head"] - or state.get("archiveArtifact") != payload_item - or state.get("sourceInventorySha256") - != _sha_bytes(_canonical(payload["sourceInventory"])) - ): - _fail("NATIVE_ARCHIVE_STATE_MISMATCH") - receipt = docs["release-native-evidence-archive-receipt.json"] - expected_receipt = { - "schemaVersion", - "status", - "nativeStatusChainHeadSha256", - "archiveArtifact", - "archiveCount", - "archiveOperatorCount", - "verifierOperatorCount", - "archives", - } - if ( - set(receipt) != expected_receipt - or receipt.get("schemaVersion") != int(POLICY["receipt_schema_version"]) - or receipt.get("status") != "preserved" - or receipt.get("archiveArtifact") != payload_item - or receipt.get("nativeStatusChainHeadSha256") != info["chain_head"] - ): - _fail("NATIVE_ARCHIVE_RECEIPT_MISMATCH") - plan = payload["archivePolicy"]["targets"] - entries = receipt.get("archives") - if ( - not isinstance(entries, list) - or len(entries) != len(plan) - or receipt.get("archiveCount") != len(plan) - ): - _fail("NATIVE_ARCHIVE_RECEIPT_COUNT_MISMATCH") - archive_ops: set[str] = set() - verifier_ops: set[str] = set() - locators: set[str] = set() - expected_plan = { - ( - x["archiveIdentity"], - x["archiveOperator"], - x["verifierIdentity"], - x["verifierOperator"], - ) - for x in plan - } - observed_plan = set() - for entry in entries: - keys = { - "archiveIdentity", - "archiveOperator", - "verifierIdentity", - "verifierOperator", - "archiveId", - "locator", - "immutability", - "archiveVerifiedAt", - "independentVerifiedAt", - "bindResultSha256", - "archiveVerifyResultSha256", - "independentVerifyResultSha256", - "bindResult", - "archiveVerifyResult", - "independentVerifyResult", - } - if not isinstance(entry, dict) or set(entry) != keys: - _fail("NATIVE_ARCHIVE_RECEIPT_ENTRY_SCHEMA_INVALID") - ai = _identity( - entry.get("archiveIdentity"), "NATIVE_ARCHIVE_RECEIPT_ARCHIVE_ID_INVALID" - ) - ao = _identity( - entry.get("archiveOperator"), - "NATIVE_ARCHIVE_RECEIPT_ARCHIVE_OPERATOR_INVALID", - ) - vi = _identity( - entry.get("verifierIdentity"), "NATIVE_ARCHIVE_RECEIPT_VERIFIER_ID_INVALID" - ) - vo = _identity( - entry.get("verifierOperator"), - "NATIVE_ARCHIVE_RECEIPT_VERIFIER_OPERATOR_INVALID", - ) - observed_plan.add((ai, ao, vi, vo)) - archive_ops.add(ao) - verifier_ops.add(vo) - _identity( - entry.get("archiveId"), "NATIVE_ARCHIVE_RECEIPT_ARCHIVE_ID_VALUE_INVALID" - ) - locator = _safe_locator( - entry.get("locator"), "NATIVE_ARCHIVE_RECEIPT_LOCATOR_INVALID" - ) - if locator in locators: - _fail("NATIVE_ARCHIVE_RECEIPT_LOCATOR_COLLISION") - locators.add(locator) - if entry.get("immutability") not in set(POLICY["allowed_archive_immutability"]): - _fail("NATIVE_ARCHIVE_RECEIPT_IMMUTABILITY_INVALID") - _dt( - entry.get("archiveVerifiedAt"), - "NATIVE_ARCHIVE_RECEIPT_ARCHIVE_TIME_INVALID", - ) - _dt( - entry.get("independentVerifiedAt"), - "NATIVE_ARCHIVE_RECEIPT_VERIFIER_TIME_INVALID", - ) - for field, result_field in ( - ("bindResultSha256", "bindResult"), - ("archiveVerifyResultSha256", "archiveVerifyResult"), - ("independentVerifyResultSha256", "independentVerifyResult"), - ): - expected_hash = _hex( - entry.get(field), "NATIVE_ARCHIVE_RECEIPT_RESULT_HASH_INVALID" - ) - result_doc = entry.get(result_field) - if ( - not isinstance(result_doc, dict) - or _sha_bytes(_canonical(result_doc)) != expected_hash - ): - _fail("NATIVE_ARCHIVE_RECEIPT_RESULT_REBIND_FAILED") - expected_archive_id = _archive_id( - info["chain_head"], payload_item["sha256"], ai - ) - if entry.get("archiveId") != expected_archive_id: - _fail("NATIVE_ARCHIVE_RECEIPT_ARCHIVE_ID_MISMATCH") - bind_check = _validate_archive_response( - entry["bindResult"], - operation="bind", - archive_id=expected_archive_id, - identity=ai, - operator=ao, - artifact=payload_item, - now=current, - historical=True, - ) - verify_check = _validate_archive_response( - entry["archiveVerifyResult"], - operation="verify", - archive_id=expected_archive_id, - identity=ai, - operator=ao, - artifact=payload_item, - now=current, - historical=True, - ) - if ( - bind_check["locator"] != verify_check["locator"] - or verify_check["locator"] != locator - or bind_check["immutability"] != verify_check["immutability"] - or verify_check["immutability"] != entry.get("immutability") - ): - _fail("NATIVE_ARCHIVE_RECEIPT_ARCHIVE_REPLAY_MISMATCH") - verifier_check = _validate_verifier_response( - entry["independentVerifyResult"], - archive_id=expected_archive_id, - identity=vi, - operator=vo, - archive_identity=ai, - archive_operator=ao, - locator=locator, - artifact=payload_item, - now=current, - historical=True, - ) - if ( - entry.get("archiveVerifiedAt") != verify_check["verifiedAt"] - or entry.get("independentVerifiedAt") != verifier_check["verifiedAt"] - ): - _fail("NATIVE_ARCHIVE_RECEIPT_TIME_REBIND_FAILED") - if ( - observed_plan != expected_plan - or receipt.get("archiveOperatorCount") != len(archive_ops) - or receipt.get("verifierOperatorCount") != len(verifier_ops) - ): - _fail("NATIVE_ARCHIVE_RECEIPT_PLAN_MISMATCH") - return { - "ok": True, - "sequence": info["sequence"], - "native_status_chain_head_sha256": info["chain_head"], - "archive_sha256": payload_item["sha256"], - "source_count": info["source_count"], - } - - -def preserve_native_status( # ruff: ignore[too-many-branches, undocumented-public-function] - *, - native_dir: Path, - output_dir: Path, - expected_bootstrap_root_sha256: str, - expected_recovery_root_sha256: str, - expected_attestation_root_sha256: list[str], - targets: list[tuple[str, str, ArchiveAdapter, str, str, VerifierAdapter]], - now: datetime | None = None, -) -> dict[str, Any]: - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - source = _regular_dir(native_dir, "NATIVE_ARCHIVE_RUN159_DIR_INVALID") - try: - run159 = native.verify_native_status( - output_dir=source, - expected_bootstrap_root_sha256=expected_bootstrap_root_sha256, - expected_recovery_root_sha256=expected_recovery_root_sha256, - expected_attestation_root_sha256=expected_attestation_root_sha256, - now=current, - historical=False, - ) - except native.NativeStatusError as exc: - raise NativeArchiveError("NATIVE_ARCHIVE_RUN159_INVALID:" + str(exc)) from exc - docs, raws = _native_docs(source) - normalized = [] - for ai, ao, archive_adapter, vi, vo, verifier_adapter in targets: - normalized.append( - ( - _identity(ai, "NATIVE_ARCHIVE_IDENTITY_INVALID"), - _identity(ao, "NATIVE_ARCHIVE_OPERATOR_INVALID"), - archive_adapter, - _identity(vi, "NATIVE_ARCHIVE_VERIFIER_IDENTITY_INVALID"), - _identity(vo, "NATIVE_ARCHIVE_VERIFIER_OPERATOR_INVALID"), - verifier_adapter, - ) - ) - normalized.sort(key=lambda x: (x[1], x[0], x[4], x[3])) - plan = [ - { - "archiveIdentity": ai, - "archiveOperator": ao, - "verifierIdentity": vi, - "verifierOperator": vo, - } - for ai, ao, _, vi, vo, _ in normalized - ] - payload = { - "schemaVersion": int(POLICY["archive_schema_version"]), - "predicateType": PREDICATE_TYPE, - "status": "preserved-native-status-evidence", - "nativeStatus": { - "sequence": run159["sequence"], - "nativeStatusChainHeadSha256": run159["native_status_chain_head_sha256"], - "output": docs, - }, - "sourceInventory": _source_inventory(docs["release-native-status-bundle.json"]), - "archivePolicy": { - "minimumArchives": int(POLICY["min_archives"]), - "minimumArchiveOperators": int(POLICY["min_archive_operators"]), - "minimumVerifierOperators": int(POLICY["min_verifier_operators"]), - "targets": plan, - }, - } - _verify_archive_payload( - payload, - bootstrap_pin=expected_bootstrap_root_sha256, - recovery_pin=expected_recovery_root_sha256, - attestation_pins=expected_attestation_root_sha256, - now=current, - historical=False, - ) - target = _outside(output_dir, [source], "NATIVE_ARCHIVE_OUTPUT_INSIDE_INPUT") - if target.exists() or target.is_symlink(): - _fail("NATIVE_ARCHIVE_OUTPUT_ALREADY_EXISTS") - if len(normalized) < int(POLICY["min_archives"]): - _fail("NATIVE_ARCHIVE_TARGETS_INVALID") - payload_raw = _canonical(payload) - payload_item = _artifact("release-native-evidence-archive.json", payload_raw) - initial_hashes = {name: _sha_bytes(raw) for name, raw in raws.items()} - parent = target.parent - parent.mkdir(parents=True, exist_ok=True) - stage = Path(tempfile.mkdtemp(prefix=".run160-archive-", dir=parent)) - try: - payload_path = stage / payload_item["name"] - payload_path.write_bytes(payload_raw) - archive_results = [] - locators: set[str] = set() - archive_ops: set[str] = set() - verifier_ops: set[str] = set() - for _index, (ai, ao, archive_adapter, vi, vo, verifier_adapter) in enumerate( - normalized, start=1 - ): - archive_id = _archive_id( - run159["native_status_chain_head_sha256"], payload_item["sha256"], ai - ) - base = { - "schemaVersion": int(POLICY["archive_protocol_version"]), - "archiveId": archive_id, - "archive": {"identity": ai, "operator": ao}, - "artifact": payload_item, - } - bind_request = dict(base, operation="bind") - bind_request["artifact"] = dict(payload_item, localPath=str(payload_path)) - before = _sha(payload_path) - bind_raw = archive_adapter(bind_request) - if _sha(payload_path) != before: - _fail("NATIVE_ARCHIVE_ARTIFACT_CHANGED_DURING_BIND") - bind = _validate_archive_response( - bind_raw, - operation="bind", - archive_id=archive_id, - identity=ai, - operator=ao, - artifact=payload_item, - now=current, - ) - verify_raw = archive_adapter(dict(base, operation="verify")) - verify = _validate_archive_response( - verify_raw, - operation="verify", - archive_id=archive_id, - identity=ai, - operator=ao, - artifact=payload_item, - now=current, - ) - if ( - bind["locator"] != verify["locator"] - or bind["immutability"] != verify["immutability"] - ): - _fail("NATIVE_ARCHIVE_REMOTE_REBIND_FAILED") - if verify["locator"] in locators: - _fail("NATIVE_ARCHIVE_LOCATOR_COLLISION") - locators.add(verify["locator"]) - verifier_request = { - "schemaVersion": int(POLICY["verifier_protocol_version"]), - "operation": "verify", - "archiveId": archive_id, - "archiveIdentity": ai, - "locator": verify["locator"], - "artifact": payload_item, - "verifier": {"identity": vi, "operator": vo}, - } - verifier_raw = verifier_adapter(verifier_request) - verifier = _validate_verifier_response( - verifier_raw, - archive_id=archive_id, - identity=vi, - operator=vo, - archive_identity=ai, - archive_operator=ao, - locator=verify["locator"], - artifact=payload_item, - now=current, - ) - archive_ops.add(ao) - verifier_ops.add(vo) - archive_results.append( - { - "archiveIdentity": ai, - "archiveOperator": ao, - "verifierIdentity": vi, - "verifierOperator": vo, - "archiveId": archive_id, - "locator": verify["locator"], - "immutability": verify["immutability"], - "archiveVerifiedAt": verify["verifiedAt"], - "independentVerifiedAt": verifier["verifiedAt"], - "bindResultSha256": _sha_bytes(_canonical(bind_raw)), - "archiveVerifyResultSha256": _sha_bytes(_canonical(verify_raw)), - "independentVerifyResultSha256": _sha_bytes( - _canonical(verifier_raw) - ), - "bindResult": bind_raw, - "archiveVerifyResult": verify_raw, - "independentVerifyResult": verifier_raw, - } - ) - if ( - len(archive_ops) < int(POLICY["min_archive_operators"]) - or len(verifier_ops) < int(POLICY["min_verifier_operators"]) - or archive_ops & verifier_ops - ): - _fail("NATIVE_ARCHIVE_OPERATOR_QUORUM_INVALID") - for name, expected in initial_hashes.items(): - if _sha(source / name) != expected: - _fail("NATIVE_ARCHIVE_RUN159_INPUT_DRIFT") - if _sha(payload_path) != payload_item["sha256"]: - _fail("NATIVE_ARCHIVE_OUTPUT_DRIFT") - state = { - "schemaVersion": int(POLICY["state_schema_version"]), - "status": "trusted-native-status-archive", - "sequence": run159["sequence"], - "nativeStatusChainHeadSha256": run159["native_status_chain_head_sha256"], - "archiveArtifact": payload_item, - "sourceInventorySha256": _sha_bytes(_canonical(payload["sourceInventory"])), - } - _write(stage / _DOC_NATIVE_EVIDENCE_ARCHIVE_STATE, state) - receipt = { - "schemaVersion": int(POLICY["receipt_schema_version"]), - "status": "preserved", - "nativeStatusChainHeadSha256": run159["native_status_chain_head_sha256"], - "archiveArtifact": payload_item, - "archiveCount": len(normalized), - "archiveOperatorCount": len(archive_ops), - "verifierOperatorCount": len(verifier_ops), - "archives": archive_results, - } - _write(stage / "release-native-evidence-archive-receipt.json", receipt) - tmp_target = target.with_name(target.name + ".tmp-" + os.urandom(6).hex()) - shutil.copytree(stage, tmp_target, copy_function=shutil.copy2) - os.replace(tmp_target, target) - finally: - shutil.rmtree(stage, ignore_errors=True) - verified = verify_native_archive( - output_dir=target, - expected_bootstrap_root_sha256=expected_bootstrap_root_sha256, - expected_recovery_root_sha256=expected_recovery_root_sha256, - expected_attestation_root_sha256=expected_attestation_root_sha256, - now=current, - ) - return dict(verified, phase="native-status-archived", archive_count=len(normalized)) - - -def _validate_recovery_response( - value: dict[str, Any], *, identity: str, operator: str, now: datetime -) -> dict[str, Any]: - common = {"schemaVersion", "operation", "status", "source", "observedAt"} - if ( - value.get("schemaVersion") != int(POLICY["recovery_protocol_version"]) - or value.get("operation") != "recover" - ): - _fail("NATIVE_ARCHIVE_RECOVERY_RESULT_SCHEMA_INVALID") - source = value.get("source") - if ( - not isinstance(source, dict) - or set(source) - != { - "identity", - "operator", - "readOnly", - "archiveWriterCredentialsReused", - "verifierCredentialsReused", - } - or source.get("identity") != identity - or source.get("operator") != operator - ): - _fail("NATIVE_ARCHIVE_RECOVERY_AUTHORITY_INVALID") - if ( - source.get("readOnly") is not True - or source.get("archiveWriterCredentialsReused") is not False - or source.get("verifierCredentialsReused") is not False - ): - _fail("NATIVE_ARCHIVE_RECOVERY_CREDENTIAL_REUSE") - observed_at = _fresh( - value.get("observedAt"), "NATIVE_ARCHIVE_RECOVERY_OBSERVED_AT", now=now - ) - if value.get("status") == "unavailable": - if set(value) != common | {"reason"} or not bool( - POLICY["allow_unavailable_recovery_source"] - ): - _fail("NATIVE_ARCHIVE_RECOVERY_UNAVAILABLE_INVALID") - reason = value.get("reason") - _len = len(reason) > 255 # ruff: ignore[magic-value-comparison] - if not isinstance(reason, str) or not reason or _len: - _fail("NATIVE_ARCHIVE_RECOVERY_REASON_INVALID") - return {"status": "unavailable", "observedAt": observed_at, "reason": reason} - if value.get("status") != "observed" or set(value) != common | { - "locator", - "artifact", - "payloadBase64", - }: - _fail("NATIVE_ARCHIVE_RECOVERY_STATUS_INVALID") - artifact = _artifact_doc(value.get("artifact"), "NATIVE_ARCHIVE_RECOVERY_ARTIFACT") - if artifact["name"] != "release-native-evidence-archive.json": - _fail("NATIVE_ARCHIVE_RECOVERY_ARTIFACT_NAME_INVALID") - try: - raw = base64.b64decode(value.get("payloadBase64"), validate=True) - except Exception as exc: - raise NativeArchiveError( - "NATIVE_ARCHIVE_RECOVERY_PAYLOAD_BASE64_INVALID" - ) from exc - if len(raw) != artifact["size"] or _sha_bytes(raw) != artifact["sha256"]: - _fail("NATIVE_ARCHIVE_RECOVERY_PAYLOAD_HASH_MISMATCH") - doc = _loads(raw, "NATIVE_ARCHIVE_RECOVERY_PAYLOAD") - if raw != _canonical(doc): - _fail("NATIVE_ARCHIVE_RECOVERY_PAYLOAD_NOT_CANONICAL") - return { - "status": "observed", - "observedAt": observed_at, - "locator": _safe_locator( - value.get("locator"), "NATIVE_ARCHIVE_RECOVERY_LOCATOR_INVALID" - ), - "artifact": artifact, - "raw": raw, - "doc": doc, - } - - -def recover_native_status( # ruff: ignore[undocumented-public-function] - *, - sources: list[tuple[str, str, RecoveryAdapter]], - output_dir: Path, - expected_archive_sha256: str, - expected_native_status_chain_head_sha256: str, - expected_bootstrap_root_sha256: str, - expected_recovery_root_sha256: str, - expected_attestation_root_sha256: list[str], - now: datetime | None = None, - historical: bool = True, -) -> dict[str, Any]: - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - expected_archive_sha256 = _hex( - expected_archive_sha256, "NATIVE_ARCHIVE_RECOVERY_EXPECTED_ARCHIVE_SHA_INVALID" - ) - expected_native_status_chain_head_sha256 = _hex( - expected_native_status_chain_head_sha256, - "NATIVE_ARCHIVE_RECOVERY_EXPECTED_CHAIN_HEAD_INVALID", - ) - normalized = [ - ( - _identity(i, "NATIVE_ARCHIVE_RECOVERY_IDENTITY_INVALID"), - _identity(o, "NATIVE_ARCHIVE_RECOVERY_OPERATOR_INVALID"), - a, - ) - for i, o, a in sources - ] - if ( - len(normalized) < int(POLICY["min_recovery_sources"]) - or len({i for i, _, _ in normalized}) != len(normalized) - or len({o for _, o, _ in normalized}) < int(POLICY["min_recovery_operators"]) - ): - _fail("NATIVE_ARCHIVE_RECOVERY_QUORUM_PLAN_INVALID") - observed = [] - unavailable = [] - request = { - "schemaVersion": int(POLICY["recovery_protocol_version"]), - "operation": "recover", - "expectedArchiveSha256": expected_archive_sha256, - "expectedNativeStatusChainHeadSha256": expected_native_status_chain_head_sha256, - } - for identity, operator, adapter in sorted(normalized, key=lambda x: (x[1], x[0])): - result = _validate_recovery_response( - adapter(dict(request, source={"identity": identity, "operator": operator})), - identity=identity, - operator=operator, - now=current, - ) - (observed if result["status"] == "observed" else unavailable).append( - dict(result, identity=identity, operator=operator) - ) - if len(observed) < int(POLICY["min_recovery_sources"]) or len( - {x["operator"] for x in observed} - ) < int(POLICY["min_recovery_operators"]): - _fail("NATIVE_ARCHIVE_RECOVERY_QUORUM_NOT_MET") - locators = [x["locator"] for x in observed] - if len(set(locators)) != len(locators): - _fail("NATIVE_ARCHIVE_RECOVERY_LOCATOR_COLLISION") - raw_hashes = {_sha_bytes(x["raw"]) for x in observed} - if len(raw_hashes) != 1: - _fail("NATIVE_ARCHIVE_RECOVERY_EQUIVOCATION") - raw = observed[0]["raw"] - if _sha_bytes(raw) != expected_archive_sha256: - _fail("NATIVE_ARCHIVE_RECOVERY_ROLLBACK_PIN_MISMATCH") - doc = observed[0]["doc"] - info = _verify_archive_payload( - doc, - bootstrap_pin=expected_bootstrap_root_sha256, - recovery_pin=expected_recovery_root_sha256, - attestation_pins=expected_attestation_root_sha256, - now=current, - historical=historical, - ) - if info["chain_head"] != expected_native_status_chain_head_sha256: - _fail("NATIVE_ARCHIVE_RECOVERY_CHAIN_HEAD_PIN_MISMATCH") - target = _outside(output_dir, [], "NATIVE_ARCHIVE_RECOVERY_OUTPUT_INVALID") - if target.exists() or target.is_symlink(): - _fail("NATIVE_ARCHIVE_RECOVERY_OUTPUT_ALREADY_EXISTS") - parent = target.parent - parent.mkdir(parents=True, exist_ok=True) - stage = Path(tempfile.mkdtemp(prefix=".run160-recovery-", dir=parent)) - try: - recovered = stage / "recovered-native-status" - _rehydrate_native(doc["nativeStatus"]["output"], recovered) - try: - native.verify_native_status( - output_dir=recovered, - expected_bootstrap_root_sha256=expected_bootstrap_root_sha256, - expected_recovery_root_sha256=expected_recovery_root_sha256, - expected_attestation_root_sha256=expected_attestation_root_sha256, - now=current, - historical=historical, - ) - except native.NativeStatusError as exc: - raise NativeArchiveError( - "NATIVE_ARCHIVE_RECOVERED_RUN159_INVALID:" + str(exc) - ) from exc - receipt = { - "schemaVersion": int(POLICY["recovery_receipt_schema_version"]), - "status": "recovered", - "archiveArtifact": _artifact("release-native-evidence-archive.json", raw), - "nativeStatusChainHeadSha256": info["chain_head"], - "observed": [ - { - "identity": x["identity"], - "operator": x["operator"], - "locator": x["locator"], - "observedAt": x["observedAt"], - } - for x in observed - ], - "unavailable": [ - { - "identity": x["identity"], - "operator": x["operator"], - "reason": x["reason"], - "observedAt": x["observedAt"], - } - for x in unavailable - ], - } - _write(stage / "release-native-evidence-recovery-receipt.json", receipt) - os.replace(stage, target) - except Exception: - shutil.rmtree(stage, ignore_errors=True) - raise - return { - "ok": True, - "phase": "native-status-recovered", - "sequence": info["sequence"], - "native_status_chain_head_sha256": info["chain_head"], - "archive_sha256": expected_archive_sha256, - "observed": len(observed), - "unavailable": len(unavailable), - } - - -def _parse_target(value: str) -> tuple[str, str, str, str, list[str], list[str]]: - # archive-id,archive-op,verifier-id,verifier-op=archive command... || verifier command... - if "=" not in value or "||" not in value: - _fail("NATIVE_ARCHIVE_TARGET_ARG_INVALID") - lhs, rhs = value.split("=", 1) - parts = lhs.split(",") - if len(parts) != 4: # ruff: ignore[magic-value-comparison] - _fail("NATIVE_ARCHIVE_TARGET_ARG_INVALID") - archive_cmd, verifier_cmd = [x.strip().split() for x in rhs.split("||", 1)] - return parts[0], parts[1], parts[2], parts[3], archive_cmd, verifier_cmd - - -def main( # ruff: ignore[undocumented-public-function] - argv: list[str] | None = None, -) -> int: - parser = argparse.ArgumentParser(description=__doc__) - sub = parser.add_subparsers(dest="command", required=True) - - def pins(p): - p.add_argument("--bootstrap-root-sha256", required=True) - p.add_argument("--recovery-root-sha256", required=True) - p.add_argument("--attestation-root-sha256", action="append", required=True) - - preserve = sub.add_parser("preserve") - preserve.add_argument("--native-dir", type=Path, required=True) - preserve.add_argument("--output-dir", type=Path, required=True) - preserve.add_argument("--target", action="append", required=True) - pins(preserve) - verify = sub.add_parser("verify") - verify.add_argument("--output-dir", type=Path, required=True) - verify.add_argument("--historical", action="store_true") - pins(verify) - recover = sub.add_parser("recover") - recover.add_argument( - "--source", action="append", required=True, help="identity,operator=command..." - ) - recover.add_argument("--output-dir", type=Path, required=True) - recover.add_argument("--expected-archive-sha256", required=True) - recover.add_argument("--expected-native-status-chain-head-sha256", required=True) - recover.add_argument("--live", action="store_true") - pins(recover) - args = parser.parse_args(argv) - logger.info("starting native archive command: %s", args.command) - if args.command == "preserve": - targets = [] - for value in args.target: - ai, ao, vi, vo, ac, vc = _parse_target(value) - targets.append((ai, ao, command_archive(ac), vi, vo, command_verifier(vc))) - result = preserve_native_status( - native_dir=args.native_dir, - output_dir=args.output_dir, - expected_bootstrap_root_sha256=args.bootstrap_root_sha256, - expected_recovery_root_sha256=args.recovery_root_sha256, - expected_attestation_root_sha256=args.attestation_root_sha256, - targets=targets, - ) - elif args.command == "verify": - result = verify_native_archive( - output_dir=args.output_dir, - expected_bootstrap_root_sha256=args.bootstrap_root_sha256, - expected_recovery_root_sha256=args.recovery_root_sha256, - expected_attestation_root_sha256=args.attestation_root_sha256, - historical=args.historical, - ) - else: - src = [] - for value in args.source: - if "=" not in value: - _fail("NATIVE_ARCHIVE_SOURCE_ARG_INVALID") - lhs, cmd = value.split("=", 1) - parts = lhs.split(",") - if len(parts) != 2: # ruff: ignore[magic-value-comparison] - _fail("NATIVE_ARCHIVE_SOURCE_ARG_INVALID") - src.append((parts[0], parts[1], command_recovery(cmd.split()))) - result = recover_native_status( - sources=src, - output_dir=args.output_dir, - expected_archive_sha256=args.expected_archive_sha256, - expected_native_status_chain_head_sha256=args.expected_native_status_chain_head_sha256, - expected_bootstrap_root_sha256=args.bootstrap_root_sha256, - expected_recovery_root_sha256=args.recovery_root_sha256, - expected_attestation_root_sha256=args.attestation_root_sha256, - historical=not args.live, - ) - # sys.stdout.write(json.dumps(result, sort_keys=True, separators=(",", ":")) + "\n") - logger.info("completed native archive command: %s", args.command) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/security/audit_archive_retention.py b/security/audit_archive_retention.py deleted file mode 100644 index 585392080abb736ed0125f0b850db5b962d8a43b..0000000000000000000000000000000000000000 --- a/security/audit_archive_retention.py +++ /dev/null @@ -1,1798 +0,0 @@ -""" -Run 161: cryptographically audit archive retention and durable-copy health. - -Run 160 proves that exact native-status evidence was copied to independent immutable -archives. Run 161 makes durability an append-only cryptographic lifecycle rather than a -one-time semantic claim: - -* an independently pinned, threshold-signed retention-governance root authorizes the - exact active archive membership and every migration/retirement; -* each storage provider signs the immutable version ID, retention expiry, legal-hold - state, and a challenge-bound read-back of the exact Run 160 archive artifact; -* a distinct read-only auditor signs an independent read-back of the same version; -* every migration proves the complete *next* durable set before retired members are - authorized for removal; and -* cumulative state is replayable offline from canonical evidence. - -Private signing keys are intentionally never accepted by this tool. -""" - -from __future__ import annotations - -import argparse -import base64 -import hashlib -import json -import logging -import os -import re -import shutil -import subprocess -import sys -import tempfile -import threading -from datetime import datetime, timedelta, timezone -from pathlib import Path -from typing import Any, Callable, Iterable -from urllib.parse import urlsplit - -import tomllib -from cryptography.exceptions import InvalidSignature -from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PublicKey - -logger = logging.getLogger(__name__) - -try: - from . import archive_native_status_evidence as run160 -except (ImportError, ValueError) as exc: - import importlib.util - - _here = Path(__file__).resolve().parent - _spec = importlib.util.spec_from_file_location( - "_run160_archive_for_health", _here / "archive_native_status_evidence.py" - ) - if _spec is None or _spec.loader is None: - raise ImportError("archive_native_status_evidence.py") from exc - run160 = importlib.util.module_from_spec(_spec) - sys.modules[_spec.name] = run160 - _spec.loader.exec_module(run160) - -HERE = Path(__file__).resolve().parent -POLICY = tomllib.loads((HERE / "release_archive_health_policy.toml").read_text()) -PREDICATE_TYPE = str(POLICY["predicate_type"]) -_HEX64 = re.compile(r"^[0-9a-f]{64}$") -_ID = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:/@+-]{0,511}$") -_CHUNK = 1024 * 1024 - - -# ── Verified-state document names ──────────────────────────────────────── -# -# One constant per release-evidence document. The values are the on-disk -# artifact names and are unchanged; only the repetition is removed. Reading -# a document by a name that says which document it is also keeps static -# analysis from reading the filename token 'trusted' as 'confidential': -# these files carry published Merkle roots and SHA-256 digests, which must -# stay in clear text for any third party to verify them. -_DOC_ARCHIVE_HEALTH_STATE = "trusted-archive-health-state.json" - - -_OUTPUT_NAMES = { - "release-archive-health-bundle.json", - _DOC_ARCHIVE_HEALTH_STATE, - "active-archive-health-evidence.json", - "release-archive-health-receipt.json", -} - - -class ArchiveHealthError(RuntimeError): - """Run 161 archive-health invariant failed.""" - - -def _fail(code: str) -> None: - raise ArchiveHealthError(code) - - -def _canonical(value: Any) -> bytes: - return ( - json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False) - + "\n" - ).encode("utf-8") - - -def _sha_bytes(raw: bytes) -> str: - return hashlib.sha256(raw).hexdigest() - - -def _sha(path: Path) -> str: - h = hashlib.sha256() - with path.open("rb") as f: - for chunk in iter(lambda: f.read(_CHUNK), b""): - h.update(chunk) - return h.hexdigest() - - -def _write(path: Path, value: dict[str, Any]) -> None: - path.write_bytes(_canonical(value)) - - -def _loads(raw: bytes, code: str) -> dict[str, Any]: - def hook(pairs): - out: dict[str, Any] = {} - for k, v in pairs: - if k in out: - _fail(code + "_DUPLICATE_KEY") - out[k] = v - return out - - try: - value = json.loads(raw.decode("utf-8"), object_pairs_hook=hook) - except ArchiveHealthError: - raise - except Exception as exc: - raise ArchiveHealthError(code + "_JSON_INVALID") from exc - if not isinstance(value, dict): - _fail(code + "_SCHEMA_INVALID") - return value - - -def _read_json(path: Path, code: str) -> tuple[dict[str, Any], bytes]: - if path.is_symlink() or not path.is_file(): - _fail(code + "_INVALID") - if path.stat().st_size > int(POLICY["max_json_bytes"]): - _fail(code + "_TOO_LARGE") - raw = path.read_bytes() - doc = _loads(raw, code) - if raw != _canonical(doc): - _fail(code + "_NOT_CANONICAL") - return doc, raw - - -def _identity(value: Any, code: str) -> str: - if not isinstance(value, str): - _fail(code) - value = value.strip() - if ( - not value - or len(value) > 512 # ruff: ignore[magic-value-comparison] - or ".." in value - or "?" in value - or "#" in value - or "\x00" in value - ): - _fail(code) - if ( - any( - ord(c) < 32 # ruff: ignore[magic-value-comparison] - or ord(c) == 127 # ruff: ignore[magic-value-comparison] - for c in value - ) - or _ID.fullmatch(value) is None - ): - _fail(code) - return value - - -def _hex(value: Any, code: str) -> str: - if not isinstance(value, str) or _HEX64.fullmatch(value) is None: - _fail(code) - return value - - -def _size(value: Any, code: str) -> int: - if not isinstance(value, int) or isinstance(value, bool) or value < 0: - _fail(code) - return value - - -def _b64(value: Any, code: str, *, expected_len: int | None = None) -> bytes: - if not isinstance(value, str): - _fail(code) - try: - raw = base64.b64decode(value, validate=True) - except Exception as exc: - raise ArchiveHealthError(code) from exc - if expected_len is not None and len(raw) != expected_len: - _fail(code) - return raw - - -def _dt(value: Any, code: str) -> datetime: - if not isinstance(value, str) or not value.endswith("Z"): - _fail(code) - try: - return datetime.fromisoformat(value[:-1] + "+00:00").astimezone(timezone.utc) - except ValueError as exc: - raise ArchiveHealthError(code) from exc - - -def _ts(dt: datetime) -> str: - return ( - dt.astimezone(timezone.utc).isoformat(timespec="seconds").replace("+00:00", "Z") - ) - - -def _safe_locator(value: Any, code: str) -> str: - if ( - not isinstance(value, str) - or not value - or len(value) > 2048 # ruff: ignore[magic-value-comparison] - or "\x00" in value - ): - _fail(code) - try: - p = urlsplit(value) - except ValueError: - _fail(code) - if ( - not p.scheme - or p.scheme.lower() == "file" - or p.query - or p.fragment - or p.username is not None - or p.password is not None - ): - _fail(code) - if ".." in p.path.split("/"): - _fail(code) - return value - - -def _file_fingerprint(path: Path, code: str) -> tuple[str, int, int]: - if path.is_symlink() or not path.is_file(): - _fail(code) - st = path.stat() - return (_sha(path), st.st_size, st.st_mode & 0o7777) - - -def _dir_fingerprint(root: Path, code: str) -> dict[str, tuple[str, int, int]]: - root = _regular_dir(root, code) - out = {} - for item in root.iterdir(): - if item.is_symlink() or not item.is_file(): - _fail(code) - out[item.name] = _file_fingerprint(item, code) - return out - - -def _artifact(value: Any, code: str) -> dict[str, Any]: - if not isinstance(value, dict) or set(value) != {"name", "sha256", "size"}: - _fail(code + "_SCHEMA_INVALID") - name = value.get("name") - if not isinstance(name, str) or Path(name).name != name or not name: - _fail(code + "_NAME_INVALID") - return { - "name": name, - "sha256": _hex(value.get("sha256"), code + "_SHA_INVALID"), - "size": _size(value.get("size"), code + "_SIZE_INVALID"), - } - - -def _artifact_bytes(name: str, raw: bytes) -> dict[str, Any]: - return {"name": name, "sha256": _sha_bytes(raw), "size": len(raw)} - - -def _regular_dir(path: Path, code: str) -> Path: - if path.is_symlink() or not path.is_dir(): - _fail(code) - return path.resolve() - - -def _outside(path: Path, protected: Iterable[Path], code: str) -> Path: - target = path.expanduser().resolve() - for item in protected: - root = item.resolve() - if target == root or root in target.parents: - _fail(code) - return target - - -def _ed25519_verify(public_text: str, sig_text: str, message: bytes, code: str) -> None: - public = _b64(public_text, code + "_PUBLIC_KEY_INVALID", expected_len=32) - sig = _b64(sig_text, code + "_SIGNATURE_ENCODING_INVALID", expected_len=64) - try: - Ed25519PublicKey.from_public_bytes(public).verify(sig, message) - except (InvalidSignature, ValueError) as exc: - raise ArchiveHealthError(code + "_SIGNATURE_INVALID") from exc - - -def _key(value: Any, code: str) -> dict[str, Any]: - expected = {"identity", "operator", "expires", "publicKey"} - if not isinstance(value, dict) or set(value) != expected: - _fail(code + "_SCHEMA_INVALID") - result = { - "identity": _identity(value.get("identity"), code + "_IDENTITY_INVALID"), - "operator": _identity(value.get("operator"), code + "_OPERATOR_INVALID"), - "expires": _ts(_dt(value.get("expires"), code + "_EXPIRES_INVALID")), - "publicKey": value.get("publicKey"), - } - _b64(result["publicKey"], code + "_PUBLIC_KEY_INVALID", expected_len=32) - if value != result: - _fail(code + "_NOT_NORMALIZED") - return result - - -def _verify_root( # ruff: ignore[too-many-branches] - doc: dict[str, Any], - expected_sha256: str, - *, - now: datetime, - historical: bool = False, -) -> dict[str, Any]: - if set(doc) != {"signed", "signatures"} or not isinstance(doc.get("signed"), dict): - _fail("ARCHIVE_HEALTH_ROOT_SCHEMA_INVALID") - s = doc["signed"] - expected = { - "_type", - "specVersion", - "schemaVersion", - "rootId", - "version", - "issuedAt", - "expires", - "threshold", - "selectedSignerKeyIds", - "keys", - } - if ( - set(s) != expected - or s.get("_type") != "archive-retention-root" - or s.get("specVersion") != str(POLICY["spec_version"]) - or s.get("schemaVersion") != int(POLICY["root_schema_version"]) - ): - _fail("ARCHIVE_HEALTH_ROOT_SIGNED_SCHEMA_INVALID") - root_id = _identity(s.get("rootId"), "ARCHIVE_HEALTH_ROOT_ID_INVALID") - version = _size(s.get("version"), "ARCHIVE_HEALTH_ROOT_VERSION_INVALID") - if version != 1: - _fail("ARCHIVE_HEALTH_ROOT_VERSION_INVALID") - issued = _dt(s.get("issuedAt"), "ARCHIVE_HEALTH_ROOT_ISSUED_INVALID") - expires = _dt(s.get("expires"), "ARCHIVE_HEALTH_ROOT_EXPIRES_INVALID") - if expires <= issued or expires - issued > timedelta( - days=int(POLICY["max_root_lifetime_days"]) - ): - _fail("ARCHIVE_HEALTH_ROOT_LIFETIME_INVALID") - if not historical: - if issued > now + timedelta(minutes=int(POLICY["max_clock_skew_minutes"])): - _fail("ARCHIVE_HEALTH_ROOT_FROM_FUTURE") - if expires <= now: - _fail("ARCHIVE_HEALTH_ROOT_EXPIRED") - keys_raw = s.get("keys") - if not isinstance(keys_raw, dict) or not ( - int(POLICY["min_governance_keys"]) - <= len(keys_raw) - <= int(POLICY["max_key_count"]) - ): - _fail("ARCHIVE_HEALTH_ROOT_KEYS_INVALID") - keys: dict[str, dict[str, Any]] = {} - for kid, value in keys_raw.items(): - kid = _identity( # ruff: ignore[redefined-loop-name] - kid, - "ARCHIVE_HEALTH_ROOT_KEY_ID_INVALID", - ) - keys[kid] = _key(value, "ARCHIVE_HEALTH_ROOT_KEY") - if ( - _dt(keys[kid]["expires"], "ARCHIVE_HEALTH_ROOT_KEY_EXPIRES_INVALID") - < expires - ): - _fail("ARCHIVE_HEALTH_ROOT_KEY_EXPIRES_BEFORE_ROOT") - threshold = _size(s.get("threshold"), "ARCHIVE_HEALTH_ROOT_THRESHOLD_INVALID") - if threshold < int(POLICY["min_governance_threshold"]) or threshold > len(keys): - _fail("ARCHIVE_HEALTH_ROOT_THRESHOLD_INVALID") - selected = s.get("selectedSignerKeyIds") - if not isinstance(selected, list) or len(selected) != threshold: - _fail("ARCHIVE_HEALTH_ROOT_SELECTED_INVALID") - selected = [ - _identity(x, "ARCHIVE_HEALTH_ROOT_SELECTED_KEY_INVALID") for x in selected - ] - if ( - selected != sorted(selected) - or len(set(selected)) != len(selected) - or any(x not in keys for x in selected) - ): - _fail("ARCHIVE_HEALTH_ROOT_SELECTED_INVALID") - if len({keys[x]["operator"] for x in selected}) < int( - POLICY["min_governance_operators"] - ): - _fail("ARCHIVE_HEALTH_ROOT_OPERATOR_QUORUM_INVALID") - sigs = doc.get("signatures") - if not isinstance(sigs, list) or len(sigs) != len(selected): - _fail("ARCHIVE_HEALTH_ROOT_SIGNATURE_SET_INVALID") - sig_map: dict[str, str] = {} - for item in sigs: - if not isinstance(item, dict) or set(item) != {"keyId", "signature"}: - _fail("ARCHIVE_HEALTH_ROOT_SIGNATURE_SCHEMA_INVALID") - kid = _identity(item.get("keyId"), "ARCHIVE_HEALTH_ROOT_SIGNATURE_KEY_INVALID") - if kid in sig_map: - _fail("ARCHIVE_HEALTH_ROOT_SIGNATURE_DUPLICATE") - sig_map[kid] = item.get("signature") - if sorted(sig_map) != selected: - _fail("ARCHIVE_HEALTH_ROOT_SIGNATURE_SET_INVALID") - if sigs != sorted(sigs, key=lambda x: x["keyId"]): - _fail("ARCHIVE_HEALTH_ROOT_SIGNATURES_NOT_SORTED") - signed_bytes = _canonical(s) - for kid in selected: - at = issued - if _dt(keys[kid]["expires"], "ARCHIVE_HEALTH_ROOT_KEY_EXPIRES_INVALID") < at: - _fail("ARCHIVE_HEALTH_ROOT_SIGNER_EXPIRED") - _ed25519_verify( - keys[kid]["publicKey"], sig_map[kid], signed_bytes, "ARCHIVE_HEALTH_ROOT" - ) - raw = _canonical(doc) - sha = _sha_bytes(raw) - if sha != _hex(expected_sha256, "ARCHIVE_HEALTH_ROOT_PIN_INVALID"): - _fail("ARCHIVE_HEALTH_ROOT_PIN_MISMATCH") - return { - "rootId": root_id, - "version": version, - "issued": issued, - "expires": expires, - "threshold": threshold, - "keys": keys, - "sha256": sha, - "doc": doc, - } - - -def _member(value: Any, code: str) -> dict[str, Any]: - expected = { - "archiveIdentity", - "archiveOperator", - "archiveId", - "locator", - "immutability", - "providerKeyId", - "providerKey", - "auditorIdentity", - "auditorKeyId", - "auditorKey", - } - if not isinstance(value, dict) or set(value) != expected: - _fail(code + "_SCHEMA_INVALID") - ai = _identity(value.get("archiveIdentity"), code + "_ARCHIVE_IDENTITY_INVALID") - ao = _identity(value.get("archiveOperator"), code + "_ARCHIVE_OPERATOR_INVALID") - archive_id = _identity(value.get("archiveId"), code + "_ARCHIVE_ID_INVALID") - locator = _safe_locator(value.get("locator"), code + "_LOCATOR_INVALID") - imm = value.get("immutability") - if imm not in set(POLICY["allowed_immutability"]): - _fail(code + "_IMMUTABILITY_INVALID") - pkid = _identity(value.get("providerKeyId"), code + "_PROVIDER_KEY_ID_INVALID") - pkey = _key(value.get("providerKey"), code + "_PROVIDER_KEY") - aid = _identity(value.get("auditorIdentity"), code + "_AUDITOR_IDENTITY_INVALID") - akid = _identity(value.get("auditorKeyId"), code + "_AUDITOR_KEY_ID_INVALID") - akey = _key(value.get("auditorKey"), code + "_AUDITOR_KEY") - if pkey["operator"] != ao: - _fail(code + "_PROVIDER_OPERATOR_MISMATCH") - if aid == ai or akey["operator"] == ao: - _fail(code + "_AUDITOR_NOT_INDEPENDENT") - result = { - "archiveIdentity": ai, - "archiveOperator": ao, - "archiveId": archive_id, - "locator": locator, - "immutability": imm, - "providerKeyId": pkid, - "providerKey": pkey, - "auditorIdentity": aid, - "auditorKeyId": akid, - "auditorKey": akey, - } - if value != result: - _fail(code + "_NOT_NORMALIZED") - return result - - -def _membership( # ruff: ignore[too-many-branches] - doc: dict[str, Any], - root: dict[str, Any], - *, - run160_artifact: dict[str, Any], - native_head: str, - previous_head: str | None, - previous_members: list[dict[str, Any]] | None, - now: datetime, - historical: bool = False, -) -> dict[str, Any]: - if set(doc) != {"signed", "signatures"} or not isinstance(doc.get("signed"), dict): - _fail("ARCHIVE_HEALTH_MEMBERSHIP_SCHEMA_INVALID") - s = doc["signed"] - expected = { - "_type", - "specVersion", - "schemaVersion", - "rootSha256", - "sequence", - "issuedAt", - "previousHealthChainHeadSha256", - "run160ArchiveArtifact", - "nativeStatusChainHeadSha256", - "minimumDurableCopies", - "selectedSignerKeyIds", - "members", - "transition", - } - if ( - set(s) != expected - or s.get("_type") != "archive-health-membership" - or s.get("specVersion") != str(POLICY["spec_version"]) - or s.get("schemaVersion") != int(POLICY["membership_schema_version"]) - ): - _fail("ARCHIVE_HEALTH_MEMBERSHIP_SIGNED_SCHEMA_INVALID") - if s.get("rootSha256") != root["sha256"]: - _fail("ARCHIVE_HEALTH_MEMBERSHIP_ROOT_MISMATCH") - seq = _size(s.get("sequence"), "ARCHIVE_HEALTH_MEMBERSHIP_SEQUENCE_INVALID") - if seq < 1: - _fail("ARCHIVE_HEALTH_MEMBERSHIP_SEQUENCE_INVALID") - issued = _dt(s.get("issuedAt"), "ARCHIVE_HEALTH_MEMBERSHIP_ISSUED_INVALID") - if not historical and issued > now + timedelta( - minutes=int(POLICY["max_clock_skew_minutes"]) - ): - _fail("ARCHIVE_HEALTH_MEMBERSHIP_FROM_FUTURE") - prev = s.get("previousHealthChainHeadSha256") - if previous_head is None: - if seq != 1 or prev is not None: - _fail("ARCHIVE_HEALTH_MEMBERSHIP_BOOTSTRAP_CONTINUITY_INVALID") - elif seq < 2 or prev != previous_head: # ruff: ignore[magic-value-comparison] - _fail("ARCHIVE_HEALTH_MEMBERSHIP_CONTINUITY_INVALID") - if ( - s.get("run160ArchiveArtifact") != run160_artifact - or s.get("nativeStatusChainHeadSha256") != native_head - ): - _fail("ARCHIVE_HEALTH_MEMBERSHIP_RUN160_BINDING_INVALID") - minimum = _size( - s.get("minimumDurableCopies"), "ARCHIVE_HEALTH_MEMBERSHIP_MINIMUM_INVALID" - ) - if minimum < int(POLICY["min_durable_copies"]): - _fail("ARCHIVE_HEALTH_MEMBERSHIP_MINIMUM_INVALID") - members_raw = s.get("members") - if ( - not isinstance(members_raw, list) - or len(members_raw) < minimum - or len(members_raw) > int(POLICY["max_archives"]) - ): - _fail("ARCHIVE_HEALTH_MEMBERSHIP_MEMBER_COUNT_INVALID") - members = [_member(x, "ARCHIVE_HEALTH_MEMBER") for x in members_raw] - if members_raw != sorted(members, key=lambda x: x["archiveId"]): - _fail("ARCHIVE_HEALTH_MEMBERSHIP_NOT_SORTED") - ids = [x["archiveId"] for x in members] - loc = [x["locator"] for x in members] - if len(set(ids)) != len(ids) or len(set(loc)) != len(loc): - _fail("ARCHIVE_HEALTH_MEMBERSHIP_DUPLICATE_INVALID") - if len({x["archiveOperator"] for x in members}) < int( - POLICY["min_archive_operators"] - ): - _fail("ARCHIVE_HEALTH_MEMBERSHIP_ARCHIVE_OPERATOR_QUORUM_INVALID") - if len({x["auditorKey"]["operator"] for x in members}) < int( - POLICY["min_auditor_operators"] - ): - _fail("ARCHIVE_HEALTH_MEMBERSHIP_AUDITOR_OPERATOR_QUORUM_INVALID") - archive_ops = {x["archiveOperator"] for x in members} - auditor_ops = {x["auditorKey"]["operator"] for x in members} - if archive_ops & auditor_ops: - _fail("ARCHIVE_HEALTH_MEMBERSHIP_OPERATOR_PLANES_OVERLAP") - root_ops = {root["keys"][x]["operator"] for x in root["keys"]} - if root_ops & (archive_ops | auditor_ops): - _fail("ARCHIVE_HEALTH_MEMBERSHIP_GOVERNANCE_OPERATOR_OVERLAP") - if issued < root["issued"] or issued > root["expires"]: - _fail("ARCHIVE_HEALTH_MEMBERSHIP_OUTSIDE_ROOT_LIFETIME") - key_ids = [] - public_keys = [] - for x in members: - if ( - _dt( - x["providerKey"]["expires"], - "ARCHIVE_HEALTH_PROVIDER_KEY_EXPIRES_INVALID", - ) - <= issued - or _dt( - x["auditorKey"]["expires"], "ARCHIVE_HEALTH_AUDITOR_KEY_EXPIRES_INVALID" - ) - <= issued - ): - _fail("ARCHIVE_HEALTH_MEMBERSHIP_KEY_EXPIRED") - key_ids.extend([x["providerKeyId"], x["auditorKeyId"]]) - public_keys.extend( - [x["providerKey"]["publicKey"], x["auditorKey"]["publicKey"]] - ) - if len(set(key_ids)) != len(key_ids) or len(set(public_keys)) != len(public_keys): - _fail("ARCHIVE_HEALTH_MEMBERSHIP_KEY_REUSE_INVALID") - if set(public_keys) & {root["keys"][x]["publicKey"] for x in root["keys"]}: - _fail("ARCHIVE_HEALTH_MEMBERSHIP_GOVERNANCE_KEY_OVERLAP") - if previous_members is not None: - old_by_id = {x["archiveId"]: x for x in previous_members} - for x in members: - if x["archiveId"] in old_by_id and x != old_by_id[x["archiveId"]]: - _fail("ARCHIVE_HEALTH_MEMBER_ID_REBOUND") - transition = s.get("transition") - if not isinstance(transition, dict) or set(transition) != { - "kind", - "addedArchiveIds", - "retiredArchiveIds", - }: - _fail("ARCHIVE_HEALTH_TRANSITION_SCHEMA_INVALID") - old = {x["archiveId"] for x in previous_members or []} - new = set(ids) - added = sorted(new - old) - retired = sorted(old - new) - kind = transition.get("kind") - expected_kind = ( - "bootstrap" - if previous_members is None - else ("migration" if added or retired else "audit") - ) - if ( - kind != expected_kind - or transition.get("addedArchiveIds") != added - or transition.get("retiredArchiveIds") != retired - ): - _fail("ARCHIVE_HEALTH_TRANSITION_MISMATCH") - selected = s.get("selectedSignerKeyIds") - if not isinstance(selected, list) or len(selected) != root["threshold"]: - _fail("ARCHIVE_HEALTH_MEMBERSHIP_SELECTED_INVALID") - selected = [ - _identity(x, "ARCHIVE_HEALTH_MEMBERSHIP_SELECTED_KEY_INVALID") for x in selected - ] - if ( - selected != sorted(selected) - or len(set(selected)) != len(selected) - or any(x not in root["keys"] for x in selected) - ): - _fail("ARCHIVE_HEALTH_MEMBERSHIP_SELECTED_INVALID") - if len({root["keys"][x]["operator"] for x in selected}) < int( - POLICY["min_governance_operators"] - ): - _fail("ARCHIVE_HEALTH_MEMBERSHIP_SIGNER_OPERATOR_QUORUM_INVALID") - sigs = doc.get("signatures") - if not isinstance(sigs, list) or len(sigs) != len(selected): - _fail("ARCHIVE_HEALTH_MEMBERSHIP_SIGNATURE_SET_INVALID") - sigmap = {} - for item in sigs: - if not isinstance(item, dict) or set(item) != {"keyId", "signature"}: - _fail("ARCHIVE_HEALTH_MEMBERSHIP_SIGNATURE_SCHEMA_INVALID") - kid = _identity( - item.get("keyId"), "ARCHIVE_HEALTH_MEMBERSHIP_SIGNATURE_KEY_INVALID" - ) - if kid in sigmap: - _fail("ARCHIVE_HEALTH_MEMBERSHIP_SIGNATURE_DUPLICATE") - sigmap[kid] = item.get("signature") - if sorted(sigmap) != selected: - _fail("ARCHIVE_HEALTH_MEMBERSHIP_SIGNATURE_SET_INVALID") - if sigs != sorted(sigs, key=lambda x: x["keyId"]): - _fail("ARCHIVE_HEALTH_MEMBERSHIP_SIGNATURES_NOT_SORTED") - raw = _canonical(s) - for kid in selected: - if ( - _dt(root["keys"][kid]["expires"], "ARCHIVE_HEALTH_ROOT_KEY_EXPIRES_INVALID") - < issued - ): - _fail("ARCHIVE_HEALTH_MEMBERSHIP_SIGNER_EXPIRED") - _ed25519_verify( - root["keys"][kid]["publicKey"], - sigmap[kid], - raw, - "ARCHIVE_HEALTH_MEMBERSHIP", - ) - return { - "sequence": seq, - "issued": issued, - "minimum": minimum, - "members": members, - "transition": transition, - "doc": doc, - "sha256": _sha_bytes(_canonical(doc)), - } - - -def _command_adapter( - command: list[str], *, prefix: str -) -> Callable[[dict[str, Any]], dict[str, Any]]: - if not command: - _fail(prefix + "_COMMAND_INVALID") - - def invoke(request: dict[str, Any]) -> dict[str, Any]: - proc = subprocess.Popen( # ruff: ignore[subprocess-without-shell-equals-true] - command, - stdin=subprocess.PIPE, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, - env={"PATH": os.defpath, "LC_ALL": "C", "LANG": "C"}, - ) - input_bytes = _canonical(request) - out = bytearray() - err = bytearray() - overflow = [] - - def reader(pipe, buf): - try: - while True: - chunk = pipe.read(65536) - if not chunk: - break - if len(buf) + len(chunk) > int(POLICY["max_adapter_output_bytes"]): - overflow.append(True) - proc.kill() - break - buf.extend(chunk) - finally: - pipe.close() - - t1 = threading.Thread(target=reader, args=(proc.stdout, out), daemon=True) - t2 = threading.Thread(target=reader, args=(proc.stderr, err), daemon=True) - t1.start() - t2.start() - try: - assert proc.stdin is not None # ruff: ignore[assert] - proc.stdin.write(input_bytes) - proc.stdin.close() - proc.wait(timeout=int(POLICY["adapter_timeout_seconds"])) - except subprocess.TimeoutExpired: - proc.kill() - proc.wait() - raise ArchiveHealthError( # ruff: ignore[raise-without-from-inside-except] - prefix + "_TIMEOUT" - ) - t1.join() - t2.join() - if overflow: - _fail(prefix + "_OUTPUT_TOO_LARGE") - if proc.returncode != 0: - _fail(prefix + "_FAILED") - return _loads(bytes(out), prefix + "_RESPONSE") - - return invoke - - -def command_provider(command: list[str]): # ruff: ignore[undocumented-public-function] - return _command_adapter(command, prefix="ARCHIVE_HEALTH_PROVIDER") - - -def command_auditor(command: list[str]): # ruff: ignore[undocumented-public-function] - return _command_adapter(command, prefix="ARCHIVE_HEALTH_AUDITOR") - - -def _audit_id( - sequence: int, archive_id: str, membership_sha: str, artifact_sha: str -) -> str: - return "audit-" + _sha_bytes( - _canonical( - { - "sequence": sequence, - "archiveId": archive_id, - "membershipSha256": membership_sha, - "artifactSha256": artifact_sha, - } - ) - ) - - -def _challenge(audit_id: str, previous_head: str | None) -> str: - return _sha_bytes( - _canonical( - {"auditId": audit_id, "previousHealthChainHeadSha256": previous_head} - ) - ) - - -def _verify_provider_response( # ruff: ignore[too-many-branches] - value: dict[str, Any], - member: dict[str, Any], - *, - audit_id: str, - challenge: str, - artifact: dict[str, Any], - now: datetime, - historical: bool, -) -> dict[str, Any]: - if ( - not isinstance(value, dict) - or set(value) != {"signed", "signature"} - or not isinstance(value.get("signed"), dict) - ): - _fail("ARCHIVE_HEALTH_PROVIDER_RESPONSE_SCHEMA_INVALID") - s = value["signed"] - expected = { - "schemaVersion", - "operation", - "auditId", - "archiveId", - "archiveIdentity", - "locator", - "immutableVersionId", - "artifact", - "immutability", - "retentionMode", - "retentionUntil", - "legalHold", - "challenge", - "remoteReadbackVerified", - "observedAt", - } - if ( - set(s) != expected - or s.get("schemaVersion") != int(POLICY["provider_protocol_version"]) - or s.get("operation") != "audit-retention" - ): - _fail("ARCHIVE_HEALTH_PROVIDER_SIGNED_SCHEMA_INVALID") - if ( - s.get("auditId") != audit_id - or s.get("archiveId") != member["archiveId"] - or s.get("archiveIdentity") != member["archiveIdentity"] - ): - _fail("ARCHIVE_HEALTH_PROVIDER_BINDING_INVALID") - if ( - _safe_locator(s.get("locator"), "ARCHIVE_HEALTH_PROVIDER_LOCATOR_INVALID") - != member["locator"] - or s.get("artifact") != artifact - or s.get("immutability") != member["immutability"] - ): - _fail("ARCHIVE_HEALTH_PROVIDER_ARTIFACT_BINDING_INVALID") - version = _identity( - s.get("immutableVersionId"), "ARCHIVE_HEALTH_PROVIDER_VERSION_ID_INVALID" - ) - mode = s.get("retentionMode") - if mode not in set(POLICY["allowed_retention_modes"]): - _fail("ARCHIVE_HEALTH_RETENTION_MODE_INVALID") - compatible = { - "object-lock": {"object-lock-compliance", "object-lock-governance"}, - "content-addressed": {"content-addressed"}, - "versioned-create-only": {"versioned-create-only"}, - "release-create-only": {"release-create-only"}, - } - if mode not in compatible.get(member["immutability"], set()): - _fail("ARCHIVE_HEALTH_RETENTION_MODE_IMMUTABILITY_MISMATCH") - retention = _dt(s.get("retentionUntil"), "ARCHIVE_HEALTH_RETENTION_UNTIL_INVALID") - observed = _dt(s.get("observedAt"), "ARCHIVE_HEALTH_PROVIDER_OBSERVED_INVALID") - if not historical: - skew = timedelta(minutes=int(POLICY["max_clock_skew_minutes"])) - age = timedelta(minutes=int(POLICY["max_result_age_minutes"])) - if observed > now + skew or now - observed > age: - _fail("ARCHIVE_HEALTH_PROVIDER_OBSERVATION_NOT_FRESH") - if ( - retention - < now + timedelta(days=int(POLICY["minimum_retention_remaining_days"])) - and s.get("legalHold") is not True - ): - _fail("ARCHIVE_HEALTH_RETENTION_TOO_SHORT") - if retention <= observed and s.get("legalHold") is not True: - _fail("ARCHIVE_HEALTH_RETENTION_EXPIRED_AT_OBSERVATION") - if ( - not isinstance(s.get("legalHold"), bool) - or s.get("challenge") != challenge - or s.get("remoteReadbackVerified") is not True - ): - _fail("ARCHIVE_HEALTH_PROVIDER_PROOF_INVALID") - sig = value.get("signature") - if ( - not isinstance(sig, dict) - or set(sig) != {"keyId", "signature"} - or sig.get("keyId") != member["providerKeyId"] - ): - _fail("ARCHIVE_HEALTH_PROVIDER_SIGNATURE_SCHEMA_INVALID") - if ( - _dt( - member["providerKey"]["expires"], - "ARCHIVE_HEALTH_PROVIDER_KEY_EXPIRES_INVALID", - ) - < observed - ): - _fail("ARCHIVE_HEALTH_PROVIDER_KEY_EXPIRED") - _ed25519_verify( - member["providerKey"]["publicKey"], - sig.get("signature"), - _canonical(s), - "ARCHIVE_HEALTH_PROVIDER", - ) - return { - "immutableVersionId": version, - "retentionMode": mode, - "retentionUntil": _ts(retention), - "legalHold": s["legalHold"], - "observedAt": _ts(observed), - "sha256": _sha_bytes(_canonical(value)), - "doc": value, - } - - -def _verify_auditor_response( - value: dict[str, Any], - member: dict[str, Any], - *, - audit_id: str, - challenge: str, - artifact: dict[str, Any], - provider: dict[str, Any], - now: datetime, - historical: bool, -) -> dict[str, Any]: - if ( - not isinstance(value, dict) - or set(value) != {"signed", "signature"} - or not isinstance(value.get("signed"), dict) - ): - _fail("ARCHIVE_HEALTH_AUDITOR_RESPONSE_SCHEMA_INVALID") - s = value["signed"] - expected = { - "schemaVersion", - "operation", - "auditId", - "auditorIdentity", - "archiveId", - "archiveIdentity", - "locator", - "immutableVersionId", - "artifact", - "challenge", - "providerResponseSha256", - "readOnly", - "remoteReadbackVerified", - "observedAt", - } - if ( - set(s) != expected - or s.get("schemaVersion") != int(POLICY["auditor_protocol_version"]) - or s.get("operation") != "verify-retention" - ): - _fail("ARCHIVE_HEALTH_AUDITOR_SIGNED_SCHEMA_INVALID") - if ( - s.get("auditId") != audit_id - or s.get("auditorIdentity") != member["auditorIdentity"] - or s.get("archiveId") != member["archiveId"] - or s.get("archiveIdentity") != member["archiveIdentity"] - ): - _fail("ARCHIVE_HEALTH_AUDITOR_BINDING_INVALID") - if ( - _safe_locator(s.get("locator"), "ARCHIVE_HEALTH_AUDITOR_LOCATOR_INVALID") - != member["locator"] - or s.get("immutableVersionId") != provider["immutableVersionId"] - or s.get("artifact") != artifact - or s.get("challenge") != challenge - or s.get("providerResponseSha256") != provider["sha256"] - ): - _fail("ARCHIVE_HEALTH_AUDITOR_PROOF_BINDING_INVALID") - if s.get("readOnly") is not True or s.get("remoteReadbackVerified") is not True: - _fail("ARCHIVE_HEALTH_AUDITOR_PROOF_INVALID") - observed = _dt(s.get("observedAt"), "ARCHIVE_HEALTH_AUDITOR_OBSERVED_INVALID") - if not historical: - skew = timedelta(minutes=int(POLICY["max_clock_skew_minutes"])) - age = timedelta(minutes=int(POLICY["max_result_age_minutes"])) - if observed > now + skew or now - observed > age: - _fail("ARCHIVE_HEALTH_AUDITOR_OBSERVATION_NOT_FRESH") - sig = value.get("signature") - if ( - not isinstance(sig, dict) - or set(sig) != {"keyId", "signature"} - or sig.get("keyId") != member["auditorKeyId"] - ): - _fail("ARCHIVE_HEALTH_AUDITOR_SIGNATURE_SCHEMA_INVALID") - if ( - _dt( - member["auditorKey"]["expires"], - "ARCHIVE_HEALTH_AUDITOR_KEY_EXPIRES_INVALID", - ) - < observed - ): - _fail("ARCHIVE_HEALTH_AUDITOR_KEY_EXPIRED") - _ed25519_verify( - member["auditorKey"]["publicKey"], - sig.get("signature"), - _canonical(s), - "ARCHIVE_HEALTH_AUDITOR", - ) - return { - "observedAt": _ts(observed), - "sha256": _sha_bytes(_canonical(value)), - "doc": value, - } - - -def _run160_docs( - root: Path, - *, - bootstrap_pin: str, - recovery_pin: str, - attestation_pins: list[str], - now: datetime, - historical: bool, -) -> tuple[dict[str, Any], dict[str, Any], dict[str, bytes]]: - root = _regular_dir(root, "ARCHIVE_HEALTH_RUN160_DIR_INVALID") - try: - info = run160.verify_native_archive( - output_dir=root, - expected_bootstrap_root_sha256=bootstrap_pin, - expected_recovery_root_sha256=recovery_pin, - expected_attestation_root_sha256=attestation_pins, - now=now, - historical=historical, - ) - except run160.NativeArchiveError as exc: - raise ArchiveHealthError( - "ARCHIVE_HEALTH_RUN160_REPLAY_INVALID:" + str(exc) - ) from exc - docs = {} - raws = {} - for name in run160._ARCHIVE_NAMES: - doc, raw = _read_json( - root / name, - "ARCHIVE_HEALTH_RUN160_" + name.upper().replace("-", "_").replace(".", "_"), - ) - docs[name] = doc - raws[name] = raw - return info, docs, raws - - -def _baseline_members(receipt: dict[str, Any]) -> dict[str, dict[str, Any]]: - out = {} - for entry in receipt.get("archives", []): - out[entry["archiveId"]] = { - "archiveIdentity": entry["archiveIdentity"], - "archiveOperator": entry["archiveOperator"], - "archiveId": entry["archiveId"], - "locator": entry["locator"], - "immutability": entry["immutability"], - } - return out - - -def _event_head(previous_head: str | None, event: dict[str, Any]) -> str: - return _sha_bytes( - _canonical({"previousHealthChainHeadSha256": previous_head, "event": event}) - ) - - -def _output_docs(root: Path) -> tuple[dict[str, dict[str, Any]], dict[str, bytes]]: - root = _regular_dir(root, "ARCHIVE_HEALTH_OUTPUT_DIR_INVALID") - if {p.name for p in root.iterdir()} != _OUTPUT_NAMES: - _fail("ARCHIVE_HEALTH_OUTPUT_ALLOWLIST_MISMATCH") - docs = {} - raws = {} - for name in sorted(_OUTPUT_NAMES): - docs[name], raws[name] = _read_json( - root / name, - "ARCHIVE_HEALTH_OUTPUT_" + name.upper().replace("-", "_").replace(".", "_"), - ) - return docs, raws - - -def _replay( # ruff: ignore[too-many-branches] - *, - run160_dir: Path, - root_doc: dict[str, Any], - root_pin: str, - bundle: dict[str, Any], - receipts: list[dict[str, Any]], - bootstrap_pin: str, - recovery_pin: str, - attestation_pins: list[str], - now: datetime, - historical: bool, -) -> dict[str, Any]: - rinfo, _rdocs, rraws = _run160_docs( - run160_dir, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - now=now, - historical=True, - ) - root = _verify_root(root_doc, root_pin, now=now, historical=historical) - run160_item = _artifact_bytes( - "release-native-evidence-archive.json", - rraws["release-native-evidence-archive.json"], - ) - if ( - set(bundle) - != { - "schemaVersion", - "predicateType", - "status", - "retentionRoot", - "run160ArchiveArtifact", - "nativeStatusChainHeadSha256", - "events", - } - or bundle.get("schemaVersion") != int(POLICY["bundle_schema_version"]) - or bundle.get("predicateType") != PREDICATE_TYPE - or bundle.get("status") != "archive-health-history" - ): - _fail("ARCHIVE_HEALTH_BUNDLE_SCHEMA_INVALID") - if ( - bundle.get("retentionRoot") - != { - "sha256": root["sha256"], - "rootId": root["rootId"], - "version": root["version"], - } - or bundle.get("run160ArchiveArtifact") != run160_item - or bundle.get("nativeStatusChainHeadSha256") - != rinfo["native_status_chain_head_sha256"] - ): - _fail("ARCHIVE_HEALTH_BUNDLE_AUTHORITY_MISMATCH") - events = bundle.get("events") - if ( - not isinstance(events, list) - or not events - or len(events) > int(POLICY["max_events"]) - or len(receipts) != len(events) - ): - _fail("ARCHIVE_HEALTH_EVENTS_INVALID") - previous_head = None - previous_members = None - previous_audit_time = None - previous_versions = {} - previous_retentions = {} - previous_modes = {} - previous_issued = None - last_audit_times = [] - for idx, (event, receipt) in enumerate(zip(events, receipts), start=1): - if not isinstance(event, dict) or set(event) != { - "sequence", - "membership", - "audits", - "healthChainHeadSha256", - }: - _fail("ARCHIVE_HEALTH_EVENT_SCHEMA_INVALID") - membership = _membership( - event["membership"], - root, - run160_artifact=run160_item, - native_head=rinfo["native_status_chain_head_sha256"], - previous_head=previous_head, - previous_members=previous_members, - now=now, - historical=True, - ) - if membership["sequence"] != idx or event.get("sequence") != idx: - _fail("ARCHIVE_HEALTH_EVENT_SEQUENCE_INVALID") - if previous_issued is not None and membership["issued"] <= previous_issued: - _fail("ARCHIVE_HEALTH_MEMBERSHIP_TIME_NOT_MONOTONIC") - audits = event.get("audits") - if not isinstance(audits, list) or len(audits) != len(membership["members"]): - _fail("ARCHIVE_HEALTH_AUDIT_COUNT_INVALID") - by_id = {x["archiveId"]: x for x in membership["members"]} - seen = set() - normalized = [] - audit_times = [] - if not isinstance(receipt, dict) or set(receipt) != { - "sequence", - "membershipSha256", - "providerResults", - "auditorResults", - }: - _fail("ARCHIVE_HEALTH_EVENT_RECEIPT_SCHEMA_INVALID") - if ( - receipt.get("sequence") != idx - or receipt.get("membershipSha256") != membership["sha256"] - ): - _fail("ARCHIVE_HEALTH_EVENT_RECEIPT_BINDING_INVALID") - providers = receipt.get("providerResults") - auditors = receipt.get("auditorResults") - if ( - not isinstance(providers, list) - or not isinstance(auditors, list) - or len(providers) != len(audits) - or len(auditors) != len(audits) - ): - _fail("ARCHIVE_HEALTH_EVENT_RECEIPT_COUNT_INVALID") - if providers != sorted( - providers, - key=lambda x: x.get("archiveId", "") if isinstance(x, dict) else "", - ) or auditors != sorted( - auditors, - key=lambda x: x.get("archiveId", "") if isinstance(x, dict) else "", - ): - _fail("ARCHIVE_HEALTH_EVENT_RECEIPT_NOT_SORTED") - pmap = {x.get("archiveId"): x for x in providers if isinstance(x, dict)} - amap = {x.get("archiveId"): x for x in auditors if isinstance(x, dict)} - if len(pmap) != len(providers) or len(amap) != len(auditors): - _fail("ARCHIVE_HEALTH_EVENT_RECEIPT_DUPLICATE_RESULT") - for a in audits: - if not isinstance(a, dict) or set(a) != { - "archiveId", - "auditId", - "challenge", - "immutableVersionId", - "retentionMode", - "retentionUntil", - "legalHold", - "providerResponseSha256", - "auditorResponseSha256", - }: - _fail("ARCHIVE_HEALTH_AUDIT_SCHEMA_INVALID") - aid = a.get("archiveId") - if aid not in by_id or aid in seen: - _fail("ARCHIVE_HEALTH_AUDIT_MEMBER_INVALID") - seen.add(aid) - member = by_id[aid] - audit_id = _audit_id(idx, aid, membership["sha256"], run160_item["sha256"]) - challenge = _challenge(audit_id, previous_head) - if a.get("auditId") != audit_id or a.get("challenge") != challenge: - _fail("ARCHIVE_HEALTH_AUDIT_CHALLENGE_INVALID") - pe = pmap.get(aid) - ae = amap.get(aid) - if ( - not isinstance(pe, dict) - or set(pe) != {"archiveId", "response"} - or not isinstance(ae, dict) - or set(ae) != {"archiveId", "response"} - ): - _fail("ARCHIVE_HEALTH_EVENT_RECEIPT_RESULT_INVALID") - provider = _verify_provider_response( - pe["response"], - member, - audit_id=audit_id, - challenge=challenge, - artifact=run160_item, - now=now, - historical=True, - ) - auditor = _verify_auditor_response( - ae["response"], - member, - audit_id=audit_id, - challenge=challenge, - artifact=run160_item, - provider=provider, - now=now, - historical=True, - ) - expected = { - "archiveId": aid, - "auditId": audit_id, - "challenge": challenge, - "immutableVersionId": provider["immutableVersionId"], - "retentionMode": provider["retentionMode"], - "retentionUntil": provider["retentionUntil"], - "legalHold": provider["legalHold"], - "providerResponseSha256": provider["sha256"], - "auditorResponseSha256": auditor["sha256"], - } - if a != expected: - _fail("ARCHIVE_HEALTH_AUDIT_REPLAY_MISMATCH") - if ( - aid in previous_versions - and provider["immutableVersionId"] != previous_versions[aid] - ): - _fail("ARCHIVE_HEALTH_IMMUTABLE_VERSION_CHANGED") - if ( - aid in previous_modes - and provider["retentionMode"] != previous_modes[aid] - ): - _fail("ARCHIVE_HEALTH_RETENTION_MODE_CHANGED") - if ( - aid in previous_retentions - and _dt( - provider["retentionUntil"], "ARCHIVE_HEALTH_RETENTION_UNTIL_INVALID" - ) - < previous_retentions[aid] - ): - _fail("ARCHIVE_HEALTH_RETENTION_ROLLBACK") - previous_versions[aid] = provider["immutableVersionId"] - previous_modes[aid] = provider["retentionMode"] - previous_retentions[aid] = _dt( - provider["retentionUntil"], "ARCHIVE_HEALTH_RETENTION_UNTIL_INVALID" - ) - po = _dt(provider["observedAt"], "ARCHIVE_HEALTH_PROVIDER_OBSERVED_INVALID") - ao = _dt(auditor["observedAt"], "ARCHIVE_HEALTH_AUDITOR_OBSERVED_INVALID") - skew = timedelta(minutes=int(POLICY["max_clock_skew_minutes"])) - age = timedelta(minutes=int(POLICY["max_result_age_minutes"])) - if ( - po < membership["issued"] - skew - or ao < membership["issued"] - skew - or abs(po - ao) > age - ): - _fail("ARCHIVE_HEALTH_OBSERVATION_TIME_BINDING_INVALID") - audit_times.append(max(po, ao)) - normalized.append(expected) - if len(seen) < membership["minimum"] or len( - {by_id[x]["archiveOperator"] for x in seen} - ) < int(POLICY["min_archive_operators"]): - _fail("ARCHIVE_HEALTH_DURABLE_QUORUM_INVALID") - current_audit = max(audit_times) - if ( - previous_audit_time is not None - and current_audit - previous_audit_time - > timedelta(days=int(POLICY["maximum_audit_interval_days"])) - ): - _fail("ARCHIVE_HEALTH_AUDIT_INTERVAL_EXCEEDED") - core = { - "sequence": idx, - "membership": event["membership"], - "audits": normalized, - } - head = _event_head(previous_head, core) - if event.get("healthChainHeadSha256") != head: - _fail("ARCHIVE_HEALTH_CHAIN_HEAD_MISMATCH") - previous_head = head - previous_members = membership["members"] - previous_audit_time = current_audit - previous_issued = membership["issued"] - last_audit_times = audit_times - if not historical: - if ( - not last_audit_times - or now - max(last_audit_times) - > timedelta(days=int(POLICY["maximum_audit_interval_days"])) - or max(last_audit_times) - > now + timedelta(minutes=int(POLICY["max_clock_skew_minutes"])) - ): - _fail("ARCHIVE_HEALTH_ACTIVE_AUDIT_STALE") - for item in events[-1]["audits"]: - if item["legalHold"] is not True and _dt( - item["retentionUntil"], "ARCHIVE_HEALTH_RETENTION_UNTIL_INVALID" - ) < now + timedelta(days=int(POLICY["minimum_retention_remaining_days"])): - _fail("ARCHIVE_HEALTH_RETENTION_TOO_SHORT") - return { - "sequence": len(events), - "chain_head": previous_head, - "members": previous_members, - "run160_artifact": run160_item, - "native_head": rinfo["native_status_chain_head_sha256"], - "last_event": events[-1], - "last_receipt": receipts[-1], - "last_issued": previous_issued, - } - - -def verify_archive_health( # ruff: ignore[undocumented-public-function] - *, - run160_dir: Path, - output_dir: Path, - retention_root_path: Path, - expected_retention_root_sha256: str, - expected_bootstrap_root_sha256: str, - expected_recovery_root_sha256: str, - expected_attestation_root_sha256: list[str], - now: datetime | None = None, - historical: bool = False, -) -> dict[str, Any]: - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - root_doc, _ = _read_json(retention_root_path, "ARCHIVE_HEALTH_ROOT") - docs, raws = _output_docs(output_dir) - bundle = docs["release-archive-health-bundle.json"] - receipt = docs["release-archive-health-receipt.json"] - if ( - set(receipt) != {"schemaVersion", "status", "events"} - or receipt.get("schemaVersion") != int(POLICY["receipt_schema_version"]) - or receipt.get("status") != "archive-health-audited" - or not isinstance(receipt.get("events"), list) - ): - _fail("ARCHIVE_HEALTH_RECEIPT_SCHEMA_INVALID") - replay = _replay( - run160_dir=run160_dir, - root_doc=root_doc, - root_pin=expected_retention_root_sha256, - bundle=bundle, - receipts=receipt["events"], - bootstrap_pin=expected_bootstrap_root_sha256, - recovery_pin=expected_recovery_root_sha256, - attestation_pins=expected_attestation_root_sha256, - now=current, - historical=historical, - ) - bundle_item = _artifact_bytes( - "release-archive-health-bundle.json", raws["release-archive-health-bundle.json"] - ) - state = docs[_DOC_ARCHIVE_HEALTH_STATE] - expected_state = { - "schemaVersion": int(POLICY["state_schema_version"]), - "status": "trusted-archive-health", - "sequence": replay["sequence"], - "healthChainHeadSha256": replay["chain_head"], - "run160ArchiveArtifact": replay["run160_artifact"], - "nativeStatusChainHeadSha256": replay["native_head"], - "bundleArtifact": bundle_item, - "activeArchiveIds": sorted(x["archiveId"] for x in replay["members"]), - } - if state != expected_state: - _fail("ARCHIVE_HEALTH_STATE_MISMATCH") - active = docs["active-archive-health-evidence.json"] - expected_active = { - "schemaVersion": int(POLICY["active_schema_version"]), - "sequence": replay["sequence"], - "healthChainHeadSha256": replay["chain_head"], - "membership": replay["last_event"]["membership"], - "audits": replay["last_event"]["audits"], - "retirementAuthorizedArchiveIds": replay["last_event"]["membership"]["signed"][ - "transition" - ]["retiredArchiveIds"], - } - if active != expected_active: - _fail("ARCHIVE_HEALTH_ACTIVE_MISMATCH") - return { - "ok": True, - "phase": "archive-health-verified", - "sequence": replay["sequence"], - "health_chain_head_sha256": replay["chain_head"], - "active_archive_count": len(replay["members"]), - } - - -ProviderAdapter = Callable[[dict[str, Any]], dict[str, Any]] -AuditorAdapter = Callable[[dict[str, Any]], dict[str, Any]] - - -def audit_archive_health( # ruff: ignore[too-many-branches, undocumented-public-function] - *, - run160_dir: Path, - output_dir: Path, - retention_root_path: Path, - membership_path: Path, - targets: list[tuple[str, ProviderAdapter, AuditorAdapter]], - expected_retention_root_sha256: str, - expected_bootstrap_root_sha256: str, - expected_recovery_root_sha256: str, - expected_attestation_root_sha256: list[str], - previous_output_dir: Path | None = None, - now: datetime | None = None, -) -> dict[str, Any]: - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - run160_dir = _regular_dir(run160_dir, "ARCHIVE_HEALTH_RUN160_DIR_INVALID") - root_doc, _ = _read_json(retention_root_path, "ARCHIVE_HEALTH_ROOT") - root = _verify_root(root_doc, expected_retention_root_sha256, now=current) - rinfo, rdocs, rraws = _run160_docs( - run160_dir, - bootstrap_pin=expected_bootstrap_root_sha256, - recovery_pin=expected_recovery_root_sha256, - attestation_pins=expected_attestation_root_sha256, - now=current, - historical=True, - ) - run160_item = _artifact_bytes( - "release-native-evidence-archive.json", - rraws["release-native-evidence-archive.json"], - ) - previous_bundle = None - previous_receipts = [] - previous_head = None - previous_members = None - protected = [run160_dir, retention_root_path.resolve(), membership_path.resolve()] - if previous_output_dir is not None: - previous_output_dir = _regular_dir( - previous_output_dir, "ARCHIVE_HEALTH_PREVIOUS_DIR_INVALID" - ) - protected.append(previous_output_dir) - pdocs, _ = _output_docs(previous_output_dir) - if set(pdocs["release-archive-health-receipt.json"]) != { - "schemaVersion", - "status", - "events", - }: - _fail("ARCHIVE_HEALTH_PREVIOUS_RECEIPT_INVALID") - prev = _replay( - run160_dir=run160_dir, - root_doc=root_doc, - root_pin=expected_retention_root_sha256, - bundle=pdocs["release-archive-health-bundle.json"], - receipts=pdocs["release-archive-health-receipt.json"]["events"], - bootstrap_pin=expected_bootstrap_root_sha256, - recovery_pin=expected_recovery_root_sha256, - attestation_pins=expected_attestation_root_sha256, - now=current, - historical=True, - ) - previous_bundle = pdocs["release-archive-health-bundle.json"] - previous_receipts = list(pdocs["release-archive-health-receipt.json"]["events"]) - previous_head = prev["chain_head"] - previous_members = prev["members"] - target = _outside(output_dir, protected, "ARCHIVE_HEALTH_OUTPUT_OVERLAPS_INPUT") - if target.exists(): - _fail("ARCHIVE_HEALTH_OUTPUT_EXISTS") - membership_doc, _ = _read_json(membership_path, "ARCHIVE_HEALTH_MEMBERSHIP") - membership = _membership( - membership_doc, - root, - run160_artifact=run160_item, - native_head=rinfo["native_status_chain_head_sha256"], - previous_head=previous_head, - previous_members=previous_members, - now=current, - ) - expected_sequence = ( - 1 if previous_bundle is None else len(previous_bundle["events"]) + 1 - ) - if membership["sequence"] != expected_sequence: - _fail("ARCHIVE_HEALTH_MEMBERSHIP_SEQUENCE_NOT_CONSECUTIVE") - if previous_bundle is not None and membership["issued"] <= prev["last_issued"]: - _fail("ARCHIVE_HEALTH_MEMBERSHIP_TIME_NOT_MONOTONIC") - if previous_members is None: - base = _baseline_members(rdocs["release-native-evidence-archive-receipt.json"]) - for member in membership["members"]: - b = base.get(member["archiveId"]) - if b is None or any( - member[k] != b[k] - for k in ( - "archiveIdentity", - "archiveOperator", - "archiveId", - "locator", - "immutability", - ) - ): - _fail("ARCHIVE_HEALTH_BOOTSTRAP_RUN160_MEMBERSHIP_MISMATCH") - if set(base) != {x["archiveId"] for x in membership["members"]}: - _fail("ARCHIVE_HEALTH_BOOTSTRAP_RUN160_MEMBERSHIP_MISMATCH") - adapters = { - _identity(aid, "ARCHIVE_HEALTH_TARGET_ARCHIVE_ID_INVALID"): (p, a) - for aid, p, a in targets - } - member_ids = {x["archiveId"] for x in membership["members"]} - if set(adapters) != member_ids: - _fail("ARCHIVE_HEALTH_TARGET_SET_MISMATCH") - run160_fingerprint = _dir_fingerprint( - run160_dir, "ARCHIVE_HEALTH_RUN160_INPUT_DRIFT" - ) - root_fingerprint = _file_fingerprint( - retention_root_path.resolve(), "ARCHIVE_HEALTH_ROOT_INPUT_DRIFT" - ) - membership_fingerprint = _file_fingerprint( - membership_path.resolve(), "ARCHIVE_HEALTH_MEMBERSHIP_INPUT_DRIFT" - ) - previous_fingerprint = ( - _dir_fingerprint(previous_output_dir, "ARCHIVE_HEALTH_PREVIOUS_INPUT_DRIFT") - if previous_output_dir is not None - else None - ) - audit_items = [] - provider_results = [] - auditor_results = [] - for member in membership["members"]: - audit_id = _audit_id( - membership["sequence"], - member["archiveId"], - membership["sha256"], - run160_item["sha256"], - ) - challenge = _challenge(audit_id, previous_head) - provider_adapter, auditor_adapter = adapters[member["archiveId"]] - request = { - "schemaVersion": int(POLICY["provider_protocol_version"]), - "operation": "audit-retention", - "auditId": audit_id, - "challenge": challenge, - "archive": { - "identity": member["archiveIdentity"], - "operator": member["archiveOperator"], - "archiveId": member["archiveId"], - "locator": member["locator"], - }, - "artifact": run160_item, - } - praw = provider_adapter(request) - provider = _verify_provider_response( - praw, - member, - audit_id=audit_id, - challenge=challenge, - artifact=run160_item, - now=current, - historical=False, - ) - arequest = { - "schemaVersion": int(POLICY["auditor_protocol_version"]), - "operation": "verify-retention", - "auditId": audit_id, - "challenge": challenge, - "auditor": { - "identity": member["auditorIdentity"], - "operator": member["auditorKey"]["operator"], - }, - "archive": { - "identity": member["archiveIdentity"], - "archiveId": member["archiveId"], - "locator": member["locator"], - }, - "immutableVersionId": provider["immutableVersionId"], - "artifact": run160_item, - "providerResponseSha256": provider["sha256"], - } - araw = auditor_adapter(arequest) - auditor = _verify_auditor_response( - araw, - member, - audit_id=audit_id, - challenge=challenge, - artifact=run160_item, - provider=provider, - now=current, - historical=False, - ) - audit_items.append( - { - "archiveId": member["archiveId"], - "auditId": audit_id, - "challenge": challenge, - "immutableVersionId": provider["immutableVersionId"], - "retentionMode": provider["retentionMode"], - "retentionUntil": provider["retentionUntil"], - "legalHold": provider["legalHold"], - "providerResponseSha256": provider["sha256"], - "auditorResponseSha256": auditor["sha256"], - } - ) - provider_results.append({"archiveId": member["archiveId"], "response": praw}) - auditor_results.append({"archiveId": member["archiveId"], "response": araw}) - audit_items.sort(key=lambda x: x["archiveId"]) - provider_results.sort(key=lambda x: x["archiveId"]) - auditor_results.sort(key=lambda x: x["archiveId"]) - if ( - _dir_fingerprint(run160_dir, "ARCHIVE_HEALTH_RUN160_INPUT_DRIFT") - != run160_fingerprint - or _file_fingerprint( - retention_root_path.resolve(), "ARCHIVE_HEALTH_ROOT_INPUT_DRIFT" - ) - != root_fingerprint - or _file_fingerprint( - membership_path.resolve(), "ARCHIVE_HEALTH_MEMBERSHIP_INPUT_DRIFT" - ) - != membership_fingerprint - ): - _fail("ARCHIVE_HEALTH_INPUT_DRIFT_DETECTED") - if ( - previous_output_dir is not None - and _dir_fingerprint(previous_output_dir, "ARCHIVE_HEALTH_PREVIOUS_INPUT_DRIFT") - != previous_fingerprint - ): - _fail("ARCHIVE_HEALTH_INPUT_DRIFT_DETECTED") - core = { - "sequence": membership["sequence"], - "membership": membership_doc, - "audits": audit_items, - } - head = _event_head(previous_head, core) - event = dict(core, healthChainHeadSha256=head) - if previous_bundle is None: - bundle = { - "schemaVersion": int(POLICY["bundle_schema_version"]), - "predicateType": PREDICATE_TYPE, - "status": "archive-health-history", - "retentionRoot": { - "sha256": root["sha256"], - "rootId": root["rootId"], - "version": root["version"], - }, - "run160ArchiveArtifact": run160_item, - "nativeStatusChainHeadSha256": rinfo["native_status_chain_head_sha256"], - "events": [event], - } - else: - bundle = json.loads(json.dumps(previous_bundle)) - bundle["events"].append(event) - receipt_event = { - "sequence": membership["sequence"], - "membershipSha256": membership["sha256"], - "providerResults": provider_results, - "auditorResults": auditor_results, - } - receipts = [*previous_receipts, receipt_event] - parent = target.parent - parent.mkdir(parents=True, exist_ok=True) - stage = Path(tempfile.mkdtemp(prefix=".run161-health-", dir=parent)) - try: - _write(stage / "release-archive-health-bundle.json", bundle) - bundle_raw = (stage / "release-archive-health-bundle.json").read_bytes() - bundle_item = _artifact_bytes("release-archive-health-bundle.json", bundle_raw) - state = { - "schemaVersion": int(POLICY["state_schema_version"]), - "status": "trusted-archive-health", - "sequence": membership["sequence"], - "healthChainHeadSha256": head, - "run160ArchiveArtifact": run160_item, - "nativeStatusChainHeadSha256": rinfo["native_status_chain_head_sha256"], - "bundleArtifact": bundle_item, - "activeArchiveIds": sorted(member_ids), - } - active = { - "schemaVersion": int(POLICY["active_schema_version"]), - "sequence": membership["sequence"], - "healthChainHeadSha256": head, - "membership": membership_doc, - "audits": audit_items, - "retirementAuthorizedArchiveIds": membership["transition"][ - "retiredArchiveIds" - ], - } - receipt = { - "schemaVersion": int(POLICY["receipt_schema_version"]), - "status": "archive-health-audited", - "events": receipts, - } - _write(stage / _DOC_ARCHIVE_HEALTH_STATE, state) - _write(stage / "active-archive-health-evidence.json", active) - _write(stage / "release-archive-health-receipt.json", receipt) - verify_archive_health( - run160_dir=run160_dir, - output_dir=stage, - retention_root_path=retention_root_path, - expected_retention_root_sha256=expected_retention_root_sha256, - expected_bootstrap_root_sha256=expected_bootstrap_root_sha256, - expected_recovery_root_sha256=expected_recovery_root_sha256, - expected_attestation_root_sha256=expected_attestation_root_sha256, - now=current, - ) - stage.rename(target) - except Exception: - shutil.rmtree(stage, ignore_errors=True) - raise - verified = verify_archive_health( - run160_dir=run160_dir, - output_dir=target, - retention_root_path=retention_root_path, - expected_retention_root_sha256=expected_retention_root_sha256, - expected_bootstrap_root_sha256=expected_bootstrap_root_sha256, - expected_recovery_root_sha256=expected_recovery_root_sha256, - expected_attestation_root_sha256=expected_attestation_root_sha256, - now=current, - ) - return dict( - verified, - phase="archive-health-audited", - retirement_authorized=membership["transition"]["retiredArchiveIds"], - ) - - -def main( # ruff: ignore[undocumented-public-function] - argv: list[str] | None = None, -) -> int: - p = argparse.ArgumentParser(description=__doc__) - sub = p.add_subparsers(dest="command", required=True) - for name in ("audit", "verify"): - q = sub.add_parser(name) - q.add_argument("--run160-dir", type=Path, required=True) - q.add_argument("--retention-root", type=Path, required=True) - q.add_argument("--retention-root-sha256", required=True) - q.add_argument("--bootstrap-root-sha256", required=True) - q.add_argument("--recovery-root-sha256", required=True) - q.add_argument("--attestation-root-sha256", action="append", required=True) - a = sub.choices["audit"] - a.add_argument("--membership", type=Path, required=True) - a.add_argument("--previous-output-dir", type=Path) - a.add_argument("--output-dir", type=Path, required=True) - a.add_argument( - "--target", - action="append", - required=True, - help="archive-id=provider command... || auditor command...", - ) - v = sub.choices["verify"] - v.add_argument("--output-dir", type=Path, required=True) - v.add_argument("--historical", action="store_true") - args = p.parse_args(argv) - if args.command == "audit": - targets = [] - for raw in args.target: - if "=" not in raw or "||" not in raw: - _fail("ARCHIVE_HEALTH_TARGET_ARGUMENT_INVALID") - aid, rhs = raw.split("=", 1) - pcmd, acmd = [x.strip().split() for x in rhs.split("||", 1)] - targets.append((aid, command_provider(pcmd), command_auditor(acmd))) - result = audit_archive_health( - run160_dir=args.run160_dir, - output_dir=args.output_dir, - retention_root_path=args.retention_root, - membership_path=args.membership, - targets=targets, - expected_retention_root_sha256=args.retention_root_sha256, - expected_bootstrap_root_sha256=args.bootstrap_root_sha256, - expected_recovery_root_sha256=args.recovery_root_sha256, - expected_attestation_root_sha256=args.attestation_root_sha256, - previous_output_dir=args.previous_output_dir, - ) - else: - result = verify_archive_health( - run160_dir=args.run160_dir, - output_dir=args.output_dir, - retention_root_path=args.retention_root, - expected_retention_root_sha256=args.retention_root_sha256, - expected_bootstrap_root_sha256=args.bootstrap_root_sha256, - expected_recovery_root_sha256=args.recovery_root_sha256, - expected_attestation_root_sha256=args.attestation_root_sha256, - historical=args.historical, - ) - logger.info("%s", json.dumps(result, sort_keys=True)) - return 0 - - -if __name__ == "__main__": - try: - raise SystemExit(main()) - except ArchiveHealthError as exc: - logger.error("%s", json.dumps({"ok": False, "error": str(exc)}, sort_keys=True)) - raise SystemExit(2) from exc diff --git a/security/continue_archive_merkle_authority.py b/security/continue_archive_merkle_authority.py deleted file mode 100644 index d9655fc5b648e8da3de873790cf98707fa46d17e..0000000000000000000000000000000000000000 --- a/security/continue_archive_merkle_authority.py +++ /dev/null @@ -1,1367 +0,0 @@ -""" -Run 166: continue RFC6962 Merkle epochs under Run 165 active authority. - -This layer bridges the last accepted Run 164 Merkle checkpoint into later epochs -signed by the rotated/recovered log and gossip keys accepted by Run 165. The -retired pre-handoff key is never required again. Every post-handoff event still -uses the Run 164 RFC6962 inclusion/consistency proof verifier; only authority -selection and cumulative continuity are upgraded. -""" - -from __future__ import annotations - -import argparse -import hashlib -import json -import logging -import shutil -import stat -import sys -import tempfile -from datetime import datetime, timedelta, timezone -from pathlib import Path -from typing import Any, Callable - -import tomllib - -logger = logging.getLogger(__name__) - -try: - from . import govern_archive_merkle_log_authority as authority - from . import verify_archive_merkle_transparency as merkle -except (ImportError, ValueError) as exc: - import importlib.util - - _here = Path(__file__).resolve().parent - - def _load(name: str, filename: str): - spec = importlib.util.spec_from_file_location(name, _here / filename) - if spec is None or spec.loader is None: - raise ImportError(filename) - mod = importlib.util.module_from_spec(spec) - sys.modules[name] = mod - spec.loader.exec_module(mod) - return mod - - merkle = _load("_run164_merkle_for_run166", "verify_archive_merkle_transparency.py") - authority = _load( - "_run165_authority_for_run166", "govern_archive_merkle_log_authority.py" - ) - -HERE = Path(__file__).resolve().parent -POLICY = tomllib.loads( - (HERE / "release_archive_merkle_continuity_policy.toml").read_text() -) -PREDICATE_TYPE = str(POLICY["predicate_type"]) - - -# ── Verified-state document names ──────────────────────────────────────── -# -# One constant per release-evidence document. The values are the on-disk -# artifact names and are unchanged; only the repetition is removed. Reading -# a document by a name that says which document it is also keeps static -# analysis from reading the filename token 'trusted' as 'confidential': -# these files carry published Merkle roots and SHA-256 digests, which must -# stay in clear text for any third party to verify them. -_DOC_ARCHIVE_ANCHOR_STATE = "trusted-archive-anchor-state.json" -_DOC_ARCHIVE_LOG_AUTHORITY_STATE = "trusted-archive-log-authority-state.json" -_DOC_ARCHIVE_MERKLE_CONTINUITY_STATE = "trusted-archive-merkle-continuity-state.json" -_DOC_ARCHIVE_MERKLE_STATE = "trusted-archive-merkle-state.json" - - -_OUTPUT_NAMES = { - "release-archive-merkle-continuity-bundle.json", - _DOC_ARCHIVE_MERKLE_CONTINUITY_STATE, - "active-archive-merkle-continuity.json", - "release-archive-merkle-continuity-receipt.json", -} -_RUN163_NAMES = { - "release-archive-anchor-bundle.json", - _DOC_ARCHIVE_ANCHOR_STATE, - "active-archive-anchor-evidence.json", - "release-archive-anchor-receipt.json", -} -_RUN164_NAMES = { - "release-archive-merkle-bundle.json", - _DOC_ARCHIVE_MERKLE_STATE, - "active-archive-merkle-evidence.json", - "release-archive-merkle-receipt.json", -} -_RUN165_NAMES = { - "release-archive-log-authority-bundle.json", - _DOC_ARCHIVE_LOG_AUTHORITY_STATE, - "active-archive-log-authority.json", - "release-archive-log-authority-receipt.json", -} -_CHUNK = 1024 * 1024 - - -class ArchiveMerkleContinuityError( # ruff: ignore[undocumented-public-class] - RuntimeError -): - pass - - -def _fail(code: str) -> None: - raise ArchiveMerkleContinuityError(code) - - -def _canonical(value: Any) -> bytes: - return merkle._canonical(value) - - -def _sha_bytes(raw: bytes) -> str: - return hashlib.sha256(raw).hexdigest() - - -def _sha(path: Path) -> str: - h = hashlib.sha256() - with path.open("rb") as f: - for chunk in iter(lambda: f.read(_CHUNK), b""): - h.update(chunk) - return h.hexdigest() - - -def _write(path: Path, value: Any) -> None: - path.write_bytes(_canonical(value)) - - -def _loads(raw: bytes, code: str) -> dict[str, Any]: - def hook(pairs): - out = {} - for key, value in pairs: - if key in out: - _fail(code + "_DUPLICATE_KEY") - out[key] = value - return out - - if len(raw) > int(POLICY["max_json_bytes"]): - _fail(code + "_TOO_LARGE") - try: - value = json.loads(raw.decode("utf-8"), object_pairs_hook=hook) - except ArchiveMerkleContinuityError: - raise - except Exception as exc: - raise ArchiveMerkleContinuityError(code + "_JSON_INVALID") from exc - if not isinstance(value, dict): - _fail(code + "_SCHEMA_INVALID") - if raw != _canonical(value): - _fail(code + "_NOT_CANONICAL") - return value - - -def _read_json(path: Path, code: str) -> tuple[dict[str, Any], bytes]: - path = Path(path) - try: - st = path.lstat() - except OSError: - _fail(code + "_MISSING") - if not stat.S_ISREG(st.st_mode) or st.st_nlink != 1: - _fail(code + "_NOT_REGULAR") - raw = path.read_bytes() - return _loads(raw, code), raw - - -def _regular_dir(path: Path, code: str) -> Path: - path = Path(path) - try: - st = path.lstat() - except OSError: - _fail(code) - if not stat.S_ISDIR(st.st_mode): - _fail(code) - return path.resolve() - - -def _outside(path: Path, protected: list[Path], code: str) -> Path: - p = Path(path).absolute() - for item in protected: - q = Path(item).absolute() - if p == q or q in p.parents or p in q.parents: - _fail(code) - return p - - -def _dir_fingerprint(root: Path, code: str) -> tuple[tuple[str, str, int, int], ...]: - root = _regular_dir(root, code) - rows = [] - for path in sorted(root.rglob("*")): - st = path.lstat() - if stat.S_ISDIR(st.st_mode): - continue - if not stat.S_ISREG(st.st_mode) or st.st_nlink != 1: - _fail(code) - rows.append( - ( - path.relative_to(root).as_posix(), - _sha(path), - st.st_size, - stat.S_IMODE(st.st_mode), - ) - ) - return tuple(rows) - - -def _authority_fingerprint(paths: list[Path], code: str) -> tuple[Any, ...]: - rows = [] - for raw in paths: - p = Path(raw) - if p.is_dir(): - rows.append((str(p.absolute()), "dir", _dir_fingerprint(p, code))) - else: - try: - st = p.lstat() - except OSError: - _fail(code) - if not stat.S_ISREG(st.st_mode) or st.st_nlink != 1: - _fail(code) - rows.append( - ( - str(p.absolute()), - "file", - _sha(p), - st.st_size, - stat.S_IMODE(st.st_mode), - ) - ) - return tuple(rows) - - -def _artifact_map(raws: dict[str, bytes]) -> dict[str, dict[str, Any]]: - return { - name: {"sha256": _sha_bytes(raws[name]), "size": len(raws[name])} - for name in sorted(raws) - } - - -def _load_run164( - run164_dir: Path, - transparency_root_path: Path, - transparency_root_pin: str, - *, - now: datetime, -) -> dict[str, Any]: - try: - return authority._load_run164_offline( - Path(run164_dir), - Path(transparency_root_path), - transparency_root_pin, - now=now, - historical=True, - ) - except Exception as exc: - raise ArchiveMerkleContinuityError( - "ARCHIVE_MERKLE_CONTINUITY_RUN164_INVALID:" + str(exc) - ) from exc - - -def _load_run165( - *, - run164_dir: Path, - transparency_root_path: Path, - transparency_root_pin: str, - governance_root_path: Path, - governance_root_pin: str, - recovery_root_path: Path, - recovery_root_pin: str, - run165_dir: Path, - now: datetime, - historical: bool, -) -> dict[str, Any]: - try: - authority.verify_log_authority_history( - run164_dir=run164_dir, - transparency_root_path=transparency_root_path, - transparency_root_pin=transparency_root_pin, - governance_root_path=governance_root_path, - governance_root_pin=governance_root_pin, - recovery_root_path=recovery_root_path, - recovery_root_pin=recovery_root_pin, - output_dir=run165_dir, - now=now, - historical=True, - ) - docs, raws = authority._load_output(Path(run165_dir)) - except Exception as exc: - raise ArchiveMerkleContinuityError( - "ARCHIVE_MERKLE_CONTINUITY_RUN165_INVALID:" + str(exc) - ) from exc - state = docs[_DOC_ARCHIVE_LOG_AUTHORITY_STATE] - active = docs["active-archive-log-authority.json"] - if state.get("sequence") != active.get("sequence") or state.get( - "logAuthorityChainHeadSha256" - ) != active.get("logAuthorityChainHeadSha256"): - _fail("ARCHIVE_MERKLE_CONTINUITY_RUN165_STATE_ACTIVE_MISMATCH") - authority_map = active.get("authority") - if authority_map != state.get("activeAuthority"): - _fail("ARCHIVE_MERKLE_CONTINUITY_RUN165_AUTHORITY_MISMATCH") - if active.get("continuityCheckpoints") != state.get("continuityCheckpoints"): - _fail("ARCHIVE_MERKLE_CONTINUITY_RUN165_CHECKPOINTS_MISMATCH") - bundle = docs["release-archive-log-authority-bundle.json"] - receipt = docs["release-archive-log-authority-receipt.json"] - last_transition = receipt["events"][-1]["transitionDocument"]["signed"] - issued = merkle._dt( - last_transition["issuedAt"], "ARCHIVE_MERKLE_CONTINUITY_RUN165_ISSUED_INVALID" - ) - if not historical: - if issued > now + timedelta(minutes=int(POLICY["max_clock_skew_minutes"])): - _fail("ARCHIVE_MERKLE_CONTINUITY_RUN165_FROM_FUTURE") - if now - issued > timedelta(days=int(POLICY["max_active_authority_age_days"])): - _fail("ARCHIVE_MERKLE_CONTINUITY_RUN165_AUTHORITY_STALE") - return { - "docs": docs, - "raws": raws, - "state": state, - "active": active, - "authority": authority_map, - "artifacts": _artifact_map(raws), - "issued": issued, - "bundle": bundle, - } - - -def _load_run163_current( - *, - run160_dir: Path, - run161_dir: Path, - retention_root_path: Path, - retention_root_pin: str, - witness_root_path: Path, - witness_root_pin: str, - bootstrap_pin: str, - recovery_pin: str, - attestation_pins: list[str], - run162_dir: Path, - anchor_plan_path: Path, - run163_dir: Path, - now: datetime, - historical: bool, -) -> dict[str, Any]: - try: - return merkle._verify_run163_current( - run160_dir=run160_dir, - run161_dir=run161_dir, - retention_root_path=retention_root_path, - retention_root_pin=retention_root_pin, - witness_root_path=witness_root_path, - witness_root_pin=witness_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - run162_dir=run162_dir, - anchor_plan_path=anchor_plan_path, - run163_dir=run163_dir, - now=now, - historical=historical, - ) - except Exception as exc: - raise ArchiveMerkleContinuityError( - "ARCHIVE_MERKLE_CONTINUITY_RUN163_INVALID:" + str(exc) - ) from exc - - -def _enforce_external_separation( - active_authority: dict[str, Any], witness_root_path: Path, anchor_plan_path: Path -) -> None: - try: - wdoc, _ = merkle._read_json( - Path(witness_root_path), "ARCHIVE_MERKLE_CONTINUITY_WITNESS_ROOT" - ) - pdoc, _ = merkle._read_json( - Path(anchor_plan_path), "ARCHIVE_MERKLE_CONTINUITY_ANCHOR_PLAN" - ) - except Exception as exc: - raise ArchiveMerkleContinuityError( - "ARCHIVE_MERKLE_CONTINUITY_EXTERNAL_AUTHORITY_INVALID:" + str(exc) - ) from exc - other_ops = set() - other_pubs = set() - for value in wdoc.get("signed", {}).get("keys", {}).values(): - if isinstance(value, dict): - other_ops.add(value.get("operator")) - other_pubs.add(value.get("publicKey")) - for value in pdoc.get("signed", {}).get("channels", {}).values(): - if isinstance(value, dict): - other_ops.update([value.get("operator"), value.get("observerOperator")]) - other_pubs.update([value.get("publicKey"), value.get("observerPublicKey")]) - other_ops.discard(None) - other_pubs.discard(None) - own_ops = {v["operator"] for v in active_authority.values()} | { - v["gossipOperator"] for v in active_authority.values() - } - own_pubs = {v["publicKey"] for v in active_authority.values()} | { - v["gossipPublicKey"] for v in active_authority.values() - } - if own_ops & other_ops or own_pubs & other_pubs: - _fail("ARCHIVE_MERKLE_CONTINUITY_EXTERNAL_AUTHORITY_OVERLAP") - - -def _load_output(root: Path) -> tuple[dict[str, dict[str, Any]], dict[str, bytes]]: - root = _regular_dir(root, "ARCHIVE_MERKLE_CONTINUITY_OUTPUT_INVALID") - if {p.name for p in root.iterdir()} != _OUTPUT_NAMES: - _fail("ARCHIVE_MERKLE_CONTINUITY_OUTPUT_ALLOWLIST_INVALID") - docs = {} - raws = {} - for name in sorted(_OUTPUT_NAMES): - docs[name], raws[name] = _read_json( - root / name, "ARCHIVE_MERKLE_CONTINUITY_OUTPUT" - ) - return docs, raws - - -def _event_head(previous_head: str, event_without_head: dict[str, Any]) -> str: - return _sha_bytes( - _canonical( - { - "previousMerkleAuthorityContinuityHeadSha256": previous_head, - "event": event_without_head, - } - ) - ) - - -def _log_challenge( - *, - sequence: int, - run163_head: str, - leaf_hash: str, - previous_head: str, - authority_head: str, - active_authority_sha: str, -) -> str: - return _sha_bytes( - _canonical( - { - "sequence": sequence, - "run163AnchorConsensusHeadSha256": run163_head, - "leafHash": leaf_hash, - "previousMerkleAuthorityContinuityHeadSha256": previous_head, - "run165LogAuthorityChainHeadSha256": authority_head, - "activeAuthoritySha256": active_authority_sha, - } - ) - ) - - -def _gossip_challenge( - *, - sequence: int, - run163_head: str, - leaf_hash: str, - checkpoints: dict[str, str], - authority_head: str, - active_authority_sha: str, -) -> str: - return _sha_bytes( - _canonical( - { - "sequence": sequence, - "run163AnchorConsensusHeadSha256": run163_head, - "leafHash": leaf_hash, - "checkpointSha256s": checkpoints, - "run165LogAuthorityChainHeadSha256": authority_head, - "activeAuthoritySha256": active_authority_sha, - } - ) - ) - - -def _normalize_merkle_error(prefix: str, fn, *args, **kwargs): - try: - return fn(*args, **kwargs) - except ArchiveMerkleContinuityError: - raise - except Exception as exc: - raise ArchiveMerkleContinuityError(prefix + ":" + str(exc)) from exc - - -def _base_checkpoints(run165: dict[str, Any]) -> dict[str, dict[str, Any]]: - cps = run165["state"].get("continuityCheckpoints") - if not isinstance(cps, dict) or set(cps) != set(run165["authority"]): - _fail("ARCHIVE_MERKLE_CONTINUITY_BASE_CHECKPOINTS_INVALID") - out = {} - for lid in sorted(cps): - row = cps[lid] - if not isinstance(row, dict) or set(row) != { - "checkpointSha256", - "treeSize", - "rootHash", - }: - _fail("ARCHIVE_MERKLE_CONTINUITY_BASE_CHECKPOINT_INVALID") - out[lid] = { - "logId": lid, - **row, - "leafIndex": row["treeSize"] - 1, - "leafHash": None, - } - return out - - -def _replay( # ruff: ignore[too-many-branches] - *, - run164: dict[str, Any], - run165: dict[str, Any], - bundle: dict[str, Any], - receipt: dict[str, Any], - now: datetime, - historical: bool, - current_run163_docs: dict[str, Any] | None, - current_run163_raws: dict[str, bytes] | None, -) -> dict[str, Any]: - expected_bundle = { - "schemaVersion", - "predicateType", - "status", - "run164Artifacts", - "run165Artifacts", - "baseRun164Sequence", - "baseMerkleConsensusHeadSha256", - "run165Sequence", - "run165LogAuthorityChainHeadSha256", - "activeAuthoritySha256", - "events", - } - if ( - set(bundle) != expected_bundle - or bundle.get("schemaVersion") != int(POLICY["bundle_schema_version"]) - or bundle.get("predicateType") != PREDICATE_TYPE - or bundle.get("status") != "archive-merkle-authority-continuity-history" - ): - _fail("ARCHIVE_MERKLE_CONTINUITY_BUNDLE_SCHEMA_INVALID") - if ( - bundle["run164Artifacts"] != run164["artifacts"] - or bundle["run165Artifacts"] != run165["artifacts"] - ): - _fail("ARCHIVE_MERKLE_CONTINUITY_BASE_ARTIFACT_MISMATCH") - base_seq = run164["replay"]["sequence"] - base_head = run164["replay"]["merkleConsensusHeadSha256"] - authority_sha = _sha_bytes(_canonical(run165["authority"])) - if ( - bundle["baseRun164Sequence"] != base_seq - or bundle["baseMerkleConsensusHeadSha256"] != base_head - ): - _fail("ARCHIVE_MERKLE_CONTINUITY_BASE_MISMATCH") - if ( - bundle["run165Sequence"] != run165["state"]["sequence"] - or bundle["run165LogAuthorityChainHeadSha256"] - != run165["state"]["logAuthorityChainHeadSha256"] - or bundle["activeAuthoritySha256"] != authority_sha - ): - _fail("ARCHIVE_MERKLE_CONTINUITY_AUTHORITY_BINDING_INVALID") - if ( - set(receipt) - != {"schemaVersion", "status", "run164Documents", "run165Documents", "events"} - or receipt.get("schemaVersion") != int(POLICY["receipt_schema_version"]) - or receipt.get("status") != "archive-merkle-authority-continuity-accepted" - ): - _fail("ARCHIVE_MERKLE_CONTINUITY_RECEIPT_SCHEMA_INVALID") - if ( - receipt["run164Documents"] != run164["docs"] - or receipt["run165Documents"] != run165["docs"] - ): - _fail("ARCHIVE_MERKLE_CONTINUITY_EMBEDDED_BASE_MISMATCH") - events = bundle["events"] - recs = receipt["events"] - if ( - not isinstance(events, list) - or not events - or not isinstance(recs, list) - or len(events) != len(recs) - ): - _fail("ARCHIVE_MERKLE_CONTINUITY_HISTORY_LENGTH_INVALID") - previous_head = base_head - previous_by_log = _base_checkpoints(run165) - last = None - for idx, (event, rec) in enumerate(zip(events, recs), 1): - sequence = base_seq + idx - expected_event_keys = { - "sequence", - "postHandoffSequence", - "run163Sequence", - "run163AnchorConsensusHeadSha256", - "run163Artifacts", - "leafHash", - "run165Sequence", - "run165LogAuthorityChainHeadSha256", - "activeAuthoritySha256", - "logs", - "logChallenge", - "gossipChallenge", - "gossipResponseSha256s", - "merkleAuthorityContinuityHeadSha256", - } - if ( - not isinstance(event, dict) - or set(event) != expected_event_keys - or event["sequence"] != sequence - or event["postHandoffSequence"] != idx - or event["run163Sequence"] != sequence - ): - _fail("ARCHIVE_MERKLE_CONTINUITY_EVENT_SCHEMA_INVALID") - if ( - event["run165Sequence"] != run165["state"]["sequence"] - or event["run165LogAuthorityChainHeadSha256"] - != run165["state"]["logAuthorityChainHeadSha256"] - or event["activeAuthoritySha256"] != authority_sha - ): - _fail("ARCHIVE_MERKLE_CONTINUITY_EVENT_AUTHORITY_INVALID") - if ( - not isinstance(rec, dict) - or set(rec) - != {"sequence", "run163Documents", "logResponses", "gossipResponses"} - or rec["sequence"] != sequence - ): - _fail("ARCHIVE_MERKLE_CONTINUITY_RECEIPT_EVENT_INVALID") - rdocs = rec["run163Documents"] - if not isinstance(rdocs, dict) or set(rdocs) != _RUN163_NAMES: - _fail("ARCHIVE_MERKLE_CONTINUITY_RUN163_DOCUMENTS_INVALID") - rraws = {name: _canonical(rdocs[name]) for name in sorted(rdocs)} - artifacts = _artifact_map(rraws) - if event["run163Artifacts"] != artifacts: - _fail("ARCHIVE_MERKLE_CONTINUITY_RUN163_ARTIFACT_INVALID") - state163 = rdocs[_DOC_ARCHIVE_ANCHOR_STATE] - if ( - state163.get("sequence") != sequence - or state163.get("anchorConsensusHeadSha256") - != event["run163AnchorConsensusHeadSha256"] - ): - _fail("ARCHIVE_MERKLE_CONTINUITY_RUN163_BINDING_INVALID") - leaf_doc = merkle._leaf_document(rdocs, rraws) - leaf_hash = merkle.merkle_leaf_hash(_canonical(leaf_doc)) - if event["leafHash"] != leaf_hash: - _fail("ARCHIVE_MERKLE_CONTINUITY_LEAF_INVALID") - log_challenge = _log_challenge( - sequence=sequence, - run163_head=event["run163AnchorConsensusHeadSha256"], - leaf_hash=leaf_hash, - previous_head=previous_head, - authority_head=run165["state"]["logAuthorityChainHeadSha256"], - active_authority_sha=authority_sha, - ) - if event["logChallenge"] != log_challenge: - _fail("ARCHIVE_MERKLE_CONTINUITY_LOG_CHALLENGE_INVALID") - log_docs = rec["logResponses"] - if not isinstance(log_docs, dict) or set(log_docs) != set(run165["authority"]): - _fail("ARCHIVE_MERKLE_CONTINUITY_LOG_RESPONSE_SET_INVALID") - rows = [] - next_by_log = {} - for lid in sorted(run165["authority"]): - parsed = _normalize_merkle_error( - "ARCHIVE_MERKLE_CONTINUITY_CHECKPOINT_INVALID", - merkle._verify_checkpoint, - log_docs[lid], - log_id=lid, - config=run165["authority"][lid], - sequence=sequence, - challenge=log_challenge, - expected_leaf_hash=leaf_hash, - previous=previous_by_log[lid], - now=now, - creation=False, - historical=True, - ) - row = { - k: parsed[k] - for k in ( - "logId", - "checkpointSha256", - "treeSize", - "rootHash", - "leafIndex", - "leafHash", - ) - } - rows.append(row) - next_by_log[lid] = row - if event["logs"] != rows: - _fail("ARCHIVE_MERKLE_CONTINUITY_LOG_ROWS_INVALID") - checkpoints = {row["logId"]: row["checkpointSha256"] for row in rows} - gossip_challenge = _gossip_challenge( - sequence=sequence, - run163_head=event["run163AnchorConsensusHeadSha256"], - leaf_hash=leaf_hash, - checkpoints=checkpoints, - authority_head=run165["state"]["logAuthorityChainHeadSha256"], - active_authority_sha=authority_sha, - ) - if event["gossipChallenge"] != gossip_challenge: - _fail("ARCHIVE_MERKLE_CONTINUITY_GOSSIP_CHALLENGE_INVALID") - gossip_docs = rec["gossipResponses"] - if not isinstance(gossip_docs, dict) or set(gossip_docs) != set( - run165["authority"] - ): - _fail("ARCHIVE_MERKLE_CONTINUITY_GOSSIP_RESPONSE_SET_INVALID") - gossip_hashes = [] - for lid in sorted(run165["authority"]): - parsed = _normalize_merkle_error( - "ARCHIVE_MERKLE_CONTINUITY_GOSSIP_INVALID", - merkle._verify_gossip, - gossip_docs[lid], - log_id=lid, - config=run165["authority"][lid], - sequence=sequence, - challenge=gossip_challenge, - run163_head=event["run163AnchorConsensusHeadSha256"], - checkpoints=checkpoints, - now=now, - creation=False, - historical=True, - ) - gossip_hashes.append(parsed["gossipSha256"]) - if event["gossipResponseSha256s"] != gossip_hashes: - _fail("ARCHIVE_MERKLE_CONTINUITY_GOSSIP_HASHES_INVALID") - bare = { - k: event[k] for k in event if k != "merkleAuthorityContinuityHeadSha256" - } - head = _event_head(previous_head, bare) - if event["merkleAuthorityContinuityHeadSha256"] != head: - _fail("ARCHIVE_MERKLE_CONTINUITY_CHAIN_HEAD_INVALID") - previous_head = head - previous_by_log = next_by_log - last = event - if current_run163_docs is not None: - latest_docs = receipt["events"][-1]["run163Documents"] - if ( - latest_docs != current_run163_docs - or {n: _canonical(latest_docs[n]) for n in sorted(latest_docs)} - != current_run163_raws - ): - _fail("ARCHIVE_MERKLE_CONTINUITY_CURRENT_RUN163_MISMATCH") - return { - "sequence": base_seq + len(events), - "postHandoffSequence": len(events), - "head": previous_head, - "last": last, - "checkpoints": previous_by_log, - } - - -def verify_merkle_authority_continuity( # ruff: ignore[undocumented-public-function] - *, - run160_dir: Path, - run161_dir: Path, - retention_root_path: Path, - retention_root_pin: str, - witness_root_path: Path, - witness_root_pin: str, - bootstrap_pin: str, - recovery_pin: str, - attestation_pins: list[str], - run162_dir: Path, - anchor_plan_path: Path, - run163_dir: Path, - run164_dir: Path, - transparency_root_path: Path, - transparency_root_pin: str, - governance_root_path: Path, - governance_root_pin: str, - recovery_root_path: Path, - recovery_root_pin: str, - run165_dir: Path, - output_dir: Path, - now: datetime | None = None, - historical: bool = False, -) -> dict[str, Any]: - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - run164 = _load_run164( - run164_dir, transparency_root_path, transparency_root_pin, now=current - ) - run165 = _load_run165( - run164_dir=run164_dir, - transparency_root_path=transparency_root_path, - transparency_root_pin=transparency_root_pin, - governance_root_path=governance_root_path, - governance_root_pin=governance_root_pin, - recovery_root_path=recovery_root_path, - recovery_root_pin=recovery_root_pin, - run165_dir=run165_dir, - now=current, - historical=historical, - ) - _enforce_external_separation( - run165["authority"], witness_root_path, anchor_plan_path - ) - current163 = _load_run163_current( - run160_dir=run160_dir, - run161_dir=run161_dir, - retention_root_path=retention_root_path, - retention_root_pin=retention_root_pin, - witness_root_path=witness_root_path, - witness_root_pin=witness_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - run162_dir=run162_dir, - anchor_plan_path=anchor_plan_path, - run163_dir=run163_dir, - now=current, - historical=historical, - ) - docs, raws = _load_output(output_dir) - replay = _replay( - run164=run164, - run165=run165, - bundle=docs["release-archive-merkle-continuity-bundle.json"], - receipt=docs["release-archive-merkle-continuity-receipt.json"], - now=current, - historical=historical, - current_run163_docs=current163["docs"], - current_run163_raws=current163["raws"], - ) - bundle_raw = raws["release-archive-merkle-continuity-bundle.json"] - last = replay["last"] - expected_state = { - "schemaVersion": int(POLICY["state_schema_version"]), - "status": "trusted-archive-merkle-authority-continuity", - "sequence": replay["sequence"], - "postHandoffSequence": replay["postHandoffSequence"], - "baseRun164Sequence": run164["replay"]["sequence"], - "run165Sequence": run165["state"]["sequence"], - "run165LogAuthorityChainHeadSha256": run165["state"][ - "logAuthorityChainHeadSha256" - ], - "activeAuthoritySha256": _sha_bytes(_canonical(run165["authority"])), - "merkleAuthorityContinuityHeadSha256": replay["head"], - "lastRun163AnchorConsensusHeadSha256": last["run163AnchorConsensusHeadSha256"], - "lastCheckpoints": { - lid: { - k: replay["checkpoints"][lid][k] - for k in ("checkpointSha256", "treeSize", "rootHash") - } - for lid in sorted(replay["checkpoints"]) - }, - "bundleArtifact": { - "name": "release-archive-merkle-continuity-bundle.json", - "sha256": _sha_bytes(bundle_raw), - "size": len(bundle_raw), - }, - } - if docs[_DOC_ARCHIVE_MERKLE_CONTINUITY_STATE] != expected_state: - _fail("ARCHIVE_MERKLE_CONTINUITY_STATE_MISMATCH") - expected_active = { - "schemaVersion": int(POLICY["active_schema_version"]), - "status": "active-archive-merkle-authority-continuity", - "sequence": replay["sequence"], - "postHandoffSequence": replay["postHandoffSequence"], - "run165Sequence": run165["state"]["sequence"], - "run165LogAuthorityChainHeadSha256": run165["state"][ - "logAuthorityChainHeadSha256" - ], - "authority": run165["authority"], - "merkleAuthorityContinuityHeadSha256": replay["head"], - "run163AnchorConsensusHeadSha256": last["run163AnchorConsensusHeadSha256"], - "leafHash": last["leafHash"], - "logs": last["logs"], - "gossipResponseSha256s": last["gossipResponseSha256s"], - } - if docs["active-archive-merkle-continuity.json"] != expected_active: - _fail("ARCHIVE_MERKLE_CONTINUITY_ACTIVE_MISMATCH") - if not historical: - # Accepted history uses historical signature semantics, while the newest epoch stays live. - latest = docs["release-archive-merkle-continuity-receipt.json"]["events"][-1] - for lid in sorted(run165["authority"]): - integrated = merkle._dt( - latest["logResponses"][lid]["signed"]["integratedAt"], - "ARCHIVE_MERKLE_CONTINUITY_ACTIVE_TIME_INVALID", - ) - observed = merkle._dt( - latest["gossipResponses"][lid]["signed"]["observedAt"], - "ARCHIVE_MERKLE_CONTINUITY_ACTIVE_TIME_INVALID", - ) - max_age = timedelta(days=int(POLICY["max_active_epoch_age_days"])) - if current - integrated > max_age or current - observed > max_age: - _fail("ARCHIVE_MERKLE_CONTINUITY_ACTIVE_EPOCH_STALE") - return { - "ok": True, - "sequence": replay["sequence"], - "post_handoff_sequence": replay["postHandoffSequence"], - "merkle_authority_continuity_head_sha256": replay["head"], - } - - -def continue_merkle_authority( # ruff: ignore[too-many-branches, undocumented-public-function] - *, - run160_dir: Path, - run161_dir: Path, - retention_root_path: Path, - retention_root_pin: str, - witness_root_path: Path, - witness_root_pin: str, - bootstrap_pin: str, - recovery_pin: str, - attestation_pins: list[str], - run162_dir: Path, - anchor_plan_path: Path, - run163_dir: Path, - run164_dir: Path, - transparency_root_path: Path, - transparency_root_pin: str, - governance_root_path: Path, - governance_root_pin: str, - recovery_root_path: Path, - recovery_root_pin: str, - run165_dir: Path, - output_dir: Path, - adapters: list[ - tuple[ - str, - Callable[[dict[str, Any]], dict[str, Any]], - Callable[[dict[str, Any]], dict[str, Any]], - ] - ], - previous_output_dir: Path | None = None, - now: datetime | None = None, -) -> dict[str, Any]: - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - authority_paths = [ - Path(run160_dir), - Path(run161_dir), - Path(retention_root_path), - Path(witness_root_path), - Path(run162_dir), - Path(anchor_plan_path), - Path(run163_dir), - Path(run164_dir), - Path(transparency_root_path), - Path(governance_root_path), - Path(recovery_root_path), - Path(run165_dir), - ] - if previous_output_dir is not None: - authority_paths.append(Path(previous_output_dir)) - before = _authority_fingerprint( - authority_paths, "ARCHIVE_MERKLE_CONTINUITY_INPUT_DRIFT" - ) - run164 = _load_run164( - run164_dir, transparency_root_path, transparency_root_pin, now=current - ) - run165 = _load_run165( - run164_dir=run164_dir, - transparency_root_path=transparency_root_path, - transparency_root_pin=transparency_root_pin, - governance_root_path=governance_root_path, - governance_root_pin=governance_root_pin, - recovery_root_path=recovery_root_path, - recovery_root_pin=recovery_root_pin, - run165_dir=run165_dir, - now=current, - historical=False, - ) - _enforce_external_separation( - run165["authority"], witness_root_path, anchor_plan_path - ) - current163 = _load_run163_current( - run160_dir=run160_dir, - run161_dir=run161_dir, - retention_root_path=retention_root_path, - retention_root_pin=retention_root_pin, - witness_root_path=witness_root_path, - witness_root_pin=witness_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - run162_dir=run162_dir, - anchor_plan_path=anchor_plan_path, - run163_dir=run163_dir, - now=current, - historical=False, - ) - state163 = current163["docs"][_DOC_ARCHIVE_ANCHOR_STATE] - sequence = state163.get("sequence") - base_seq = run164["replay"]["sequence"] - if not isinstance(sequence, int) or sequence <= base_seq: - _fail("ARCHIVE_MERKLE_CONTINUITY_SEQUENCE_INVALID") - old_events = [] - old_receipts = [] - previous_head = run164["replay"]["merkleConsensusHeadSha256"] - previous_by_log = _base_checkpoints(run165) - if previous_output_dir is None: - if sequence != base_seq + 1: - _fail("ARCHIVE_MERKLE_CONTINUITY_PREVIOUS_OUTPUT_REQUIRED") - else: - pdocs, _ = _load_output(previous_output_dir) - state_prev = pdocs[_DOC_ARCHIVE_MERKLE_CONTINUITY_STATE] - if state_prev.get("run165LogAuthorityChainHeadSha256") != run165["state"][ - "logAuthorityChainHeadSha256" - ] or state_prev.get("activeAuthoritySha256") != _sha_bytes( - _canonical(run165["authority"]) - ): - _fail("ARCHIVE_MERKLE_CONTINUITY_AUTHORITY_CHANGED_REQUIRES_NEW_BRIDGE") - replay = _replay( - run164=run164, - run165=run165, - bundle=pdocs["release-archive-merkle-continuity-bundle.json"], - receipt=pdocs["release-archive-merkle-continuity-receipt.json"], - now=current, - historical=True, - current_run163_docs=None, - current_run163_raws=None, - ) - if sequence != replay["sequence"] + 1: - _fail("ARCHIVE_MERKLE_CONTINUITY_SEQUENCE_INVALID") - previous_head = replay["head"] - previous_by_log = replay["checkpoints"] - old_events = list( - pdocs["release-archive-merkle-continuity-bundle.json"]["events"] - ) - old_receipts = list( - pdocs["release-archive-merkle-continuity-receipt.json"]["events"] - ) - if {lid for lid, _, _ in adapters} != set(run165["authority"]) or len( - adapters - ) != len(run165["authority"]): - _fail("ARCHIVE_MERKLE_CONTINUITY_ADAPTER_SET_INVALID") - amap = {lid: (log, gossip) for lid, log, gossip in adapters} - leaf_doc = merkle._leaf_document(current163["docs"], current163["raws"]) - leaf_hash = merkle.merkle_leaf_hash(_canonical(leaf_doc)) - authority_sha = _sha_bytes(_canonical(run165["authority"])) - log_challenge = _log_challenge( - sequence=sequence, - run163_head=state163["anchorConsensusHeadSha256"], - leaf_hash=leaf_hash, - previous_head=previous_head, - authority_head=run165["state"]["logAuthorityChainHeadSha256"], - active_authority_sha=authority_sha, - ) - log_docs = {} - rows = [] - next_by_log = {} - for lid in sorted(run165["authority"]): - prev = previous_by_log[lid] - request = { - "operation": "append-merkle-leaf-authority-continuity", - "protocolVersion": 1, - "logId": lid, - "sequence": sequence, - "challenge": log_challenge, - "leaf": leaf_doc, - "leafHash": leaf_hash, - "previousTreeSize": prev["treeSize"], - "previousRootHash": prev["rootHash"], - "run165LogAuthorityChainHeadSha256": run165["state"][ - "logAuthorityChainHeadSha256" - ], - "activeAuthoritySha256": authority_sha, - } - doc = amap[lid][0](request) - parsed = _normalize_merkle_error( - "ARCHIVE_MERKLE_CONTINUITY_CHECKPOINT_INVALID", - merkle._verify_checkpoint, - doc, - log_id=lid, - config=run165["authority"][lid], - sequence=sequence, - challenge=log_challenge, - expected_leaf_hash=leaf_hash, - previous=prev, - now=current, - creation=True, - historical=False, - ) - row = { - k: parsed[k] - for k in ( - "logId", - "checkpointSha256", - "treeSize", - "rootHash", - "leafIndex", - "leafHash", - ) - } - log_docs[lid] = doc - rows.append(row) - next_by_log[lid] = row - checkpoints = {row["logId"]: row["checkpointSha256"] for row in rows} - gossip_challenge = _gossip_challenge( - sequence=sequence, - run163_head=state163["anchorConsensusHeadSha256"], - leaf_hash=leaf_hash, - checkpoints=checkpoints, - authority_head=run165["state"]["logAuthorityChainHeadSha256"], - active_authority_sha=authority_sha, - ) - gossip_docs = {} - gossip_hashes = [] - for lid in sorted(run165["authority"]): - request = { - "operation": "gossip-merkle-authority-continuity", - "protocolVersion": 1, - "sequence": sequence, - "challenge": gossip_challenge, - "run163AnchorConsensusHeadSha256": state163["anchorConsensusHeadSha256"], - "leafHash": leaf_hash, - "checkpointSha256s": checkpoints, - "run165LogAuthorityChainHeadSha256": run165["state"][ - "logAuthorityChainHeadSha256" - ], - "activeAuthoritySha256": authority_sha, - } - doc = amap[lid][1](request) - parsed = _normalize_merkle_error( - "ARCHIVE_MERKLE_CONTINUITY_GOSSIP_INVALID", - merkle._verify_gossip, - doc, - log_id=lid, - config=run165["authority"][lid], - sequence=sequence, - challenge=gossip_challenge, - run163_head=state163["anchorConsensusHeadSha256"], - checkpoints=checkpoints, - now=current, - creation=True, - historical=False, - ) - gossip_docs[lid] = doc - gossip_hashes.append(parsed["gossipSha256"]) - post_seq = sequence - base_seq - event_bare = { - "sequence": sequence, - "postHandoffSequence": post_seq, - "run163Sequence": sequence, - "run163AnchorConsensusHeadSha256": state163["anchorConsensusHeadSha256"], - "run163Artifacts": _artifact_map(current163["raws"]), - "leafHash": leaf_hash, - "run165Sequence": run165["state"]["sequence"], - "run165LogAuthorityChainHeadSha256": run165["state"][ - "logAuthorityChainHeadSha256" - ], - "activeAuthoritySha256": authority_sha, - "logs": rows, - "logChallenge": log_challenge, - "gossipChallenge": gossip_challenge, - "gossipResponseSha256s": gossip_hashes, - } - head = _event_head(previous_head, event_bare) - event = dict(event_bare) - event["merkleAuthorityContinuityHeadSha256"] = head - receipt_event = { - "sequence": sequence, - "run163Documents": current163["docs"], - "logResponses": log_docs, - "gossipResponses": gossip_docs, - } - bundle = { - "schemaVersion": int(POLICY["bundle_schema_version"]), - "predicateType": PREDICATE_TYPE, - "status": "archive-merkle-authority-continuity-history", - "run164Artifacts": run164["artifacts"], - "run165Artifacts": run165["artifacts"], - "baseRun164Sequence": base_seq, - "baseMerkleConsensusHeadSha256": run164["replay"]["merkleConsensusHeadSha256"], - "run165Sequence": run165["state"]["sequence"], - "run165LogAuthorityChainHeadSha256": run165["state"][ - "logAuthorityChainHeadSha256" - ], - "activeAuthoritySha256": authority_sha, - "events": [*old_events, event], - } - receipt = { - "schemaVersion": int(POLICY["receipt_schema_version"]), - "status": "archive-merkle-authority-continuity-accepted", - "run164Documents": run164["docs"], - "run165Documents": run165["docs"], - "events": [*old_receipts, receipt_event], - } - bundle_raw = _canonical(bundle) - state_doc = { - "schemaVersion": int(POLICY["state_schema_version"]), - "status": "trusted-archive-merkle-authority-continuity", - "sequence": sequence, - "postHandoffSequence": post_seq, - "baseRun164Sequence": base_seq, - "run165Sequence": run165["state"]["sequence"], - "run165LogAuthorityChainHeadSha256": run165["state"][ - "logAuthorityChainHeadSha256" - ], - "activeAuthoritySha256": authority_sha, - "merkleAuthorityContinuityHeadSha256": head, - "lastRun163AnchorConsensusHeadSha256": state163["anchorConsensusHeadSha256"], - "lastCheckpoints": { - lid: { - k: next_by_log[lid][k] - for k in ("checkpointSha256", "treeSize", "rootHash") - } - for lid in sorted(next_by_log) - }, - "bundleArtifact": { - "name": "release-archive-merkle-continuity-bundle.json", - "sha256": _sha_bytes(bundle_raw), - "size": len(bundle_raw), - }, - } - active = { - "schemaVersion": int(POLICY["active_schema_version"]), - "status": "active-archive-merkle-authority-continuity", - "sequence": sequence, - "postHandoffSequence": post_seq, - "run165Sequence": run165["state"]["sequence"], - "run165LogAuthorityChainHeadSha256": run165["state"][ - "logAuthorityChainHeadSha256" - ], - "authority": run165["authority"], - "merkleAuthorityContinuityHeadSha256": head, - "run163AnchorConsensusHeadSha256": state163["anchorConsensusHeadSha256"], - "leafHash": leaf_hash, - "logs": rows, - "gossipResponseSha256s": gossip_hashes, - } - if before != _authority_fingerprint( - authority_paths, "ARCHIVE_MERKLE_CONTINUITY_INPUT_DRIFT" - ): - _fail("ARCHIVE_MERKLE_CONTINUITY_INPUT_DRIFT") - protected = list(authority_paths) - target = _outside( - output_dir, protected, "ARCHIVE_MERKLE_CONTINUITY_OUTPUT_OVERLAPS_INPUT" - ) - if target.exists(): - _fail("ARCHIVE_MERKLE_CONTINUITY_OUTPUT_EXISTS") - target.parent.mkdir(parents=True, exist_ok=True) - stage = Path( - tempfile.mkdtemp(prefix=".run166-merkle-continuity-", dir=target.parent) - ) - try: - (stage / "release-archive-merkle-continuity-bundle.json").write_bytes( - bundle_raw - ) - _write(stage / _DOC_ARCHIVE_MERKLE_CONTINUITY_STATE, state_doc) - _write(stage / "active-archive-merkle-continuity.json", active) - _write(stage / "release-archive-merkle-continuity-receipt.json", receipt) - verify_merkle_authority_continuity( - run160_dir=run160_dir, - run161_dir=run161_dir, - retention_root_path=retention_root_path, - retention_root_pin=retention_root_pin, - witness_root_path=witness_root_path, - witness_root_pin=witness_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - run162_dir=run162_dir, - anchor_plan_path=anchor_plan_path, - run163_dir=run163_dir, - run164_dir=run164_dir, - transparency_root_path=transparency_root_path, - transparency_root_pin=transparency_root_pin, - governance_root_path=governance_root_path, - governance_root_pin=governance_root_pin, - recovery_root_path=recovery_root_path, - recovery_root_pin=recovery_root_pin, - run165_dir=run165_dir, - output_dir=stage, - now=current, - ) - stage.rename(target) - except Exception: - shutil.rmtree(stage, ignore_errors=True) - raise - return verify_merkle_authority_continuity( - run160_dir=run160_dir, - run161_dir=run161_dir, - retention_root_path=retention_root_path, - retention_root_pin=retention_root_pin, - witness_root_path=witness_root_path, - witness_root_pin=witness_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - run162_dir=run162_dir, - anchor_plan_path=anchor_plan_path, - run163_dir=run163_dir, - run164_dir=run164_dir, - transparency_root_path=transparency_root_path, - transparency_root_pin=transparency_root_pin, - governance_root_path=governance_root_path, - governance_root_pin=governance_root_pin, - recovery_root_path=recovery_root_path, - recovery_root_pin=recovery_root_pin, - run165_dir=run165_dir, - output_dir=target, - now=current, - ) - - -def command_log(command: list[str]): # ruff: ignore[undocumented-public-function] - return merkle.command_log(command) - - -def command_gossip(command: list[str]): # ruff: ignore[undocumented-public-function] - return merkle.command_gossip(command) - - -def _common(parser: argparse.ArgumentParser) -> None: - parser.add_argument("--run160-dir", type=Path, required=True) - parser.add_argument("--run161-dir", type=Path, required=True) - parser.add_argument("--retention-root-path", type=Path, required=True) - parser.add_argument("--retention-root-pin", required=True) - parser.add_argument("--witness-root-path", type=Path, required=True) - parser.add_argument("--witness-root-pin", required=True) - parser.add_argument("--bootstrap-pin", required=True) - parser.add_argument("--recovery-pin", required=True) - parser.add_argument( - "--attestation-pin", action="append", dest="attestation_pins", required=True - ) - parser.add_argument("--run162-dir", type=Path, required=True) - parser.add_argument("--anchor-plan-path", type=Path, required=True) - parser.add_argument("--run163-dir", type=Path, required=True) - parser.add_argument("--run164-dir", type=Path, required=True) - parser.add_argument("--transparency-root-path", type=Path, required=True) - parser.add_argument("--transparency-root-pin", required=True) - parser.add_argument("--governance-root-path", type=Path, required=True) - parser.add_argument("--governance-root-pin", required=True) - parser.add_argument("--recovery-root-path", type=Path, required=True) - parser.add_argument("--recovery-root-pin", required=True) - parser.add_argument("--run165-dir", type=Path, required=True) - parser.add_argument("--output-dir", type=Path, required=True) - - -def main( # ruff: ignore[undocumented-public-function] - argv=None, -) -> int: - parser = argparse.ArgumentParser(description=__doc__) - subs = parser.add_subparsers(dest="cmd", required=True) - p = subs.add_parser("verify") - _common(p) - p.add_argument("--historical", action="store_true") - a = subs.add_parser("append") - _common(a) - a.add_argument("--previous-output-dir", type=Path) - a.add_argument("--log-command", action="append", required=True) - a.add_argument("--gossip-command", action="append", required=True) - ns = parser.parse_args(argv) - kwargs = vars(ns) - cmd = kwargs.pop("cmd") - if cmd == "verify": - historical = kwargs.pop("historical") - result = verify_merkle_authority_continuity(**kwargs, historical=historical) - else: - previous = kwargs.pop("previous_output_dir") - log_cmds = kwargs.pop("log_command") - gossip_cmds = kwargs.pop("gossip_command") - if len(log_cmds) != len(gossip_cmds): - _fail("ARCHIVE_MERKLE_CONTINUITY_COMMAND_COUNT_INVALID") - - # CLI format: LOG_ID::program arg... ; one command per log and gossip side. - def parse(items, builder): - out = {} - for raw in items: - if "::" not in raw: - _fail("ARCHIVE_MERKLE_CONTINUITY_COMMAND_INVALID") - lid, command = raw.split("::", 1) - out[lid] = builder(command.split()) - return out - - logs = parse(log_cmds, command_log) - goss = parse(gossip_cmds, command_gossip) - if set(logs) != set(goss): - _fail("ARCHIVE_MERKLE_CONTINUITY_COMMAND_SET_INVALID") - adapters = [(lid, logs[lid], goss[lid]) for lid in sorted(logs)] - result = continue_merkle_authority( - **kwargs, adapters=adapters, previous_output_dir=previous - ) - logger.info("Merkle authority %s completed", cmd) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/security/continue_recovered_trust.py b/security/continue_recovered_trust.py deleted file mode 100644 index 1ec3cbd46fd5b1d30824ea832f2e64bfd7bbfead..0000000000000000000000000000000000000000 --- a/security/continue_recovered_trust.py +++ /dev/null @@ -1,1749 +0,0 @@ -""" -Continue cryptographic release trust after an independently authorized root recovery. - -Run 156 can replace a compromised root without requiring signatures from that compromised -root. This module makes that recovery a first-class, offline-verifiable continuity event. -Later governance epochs authorize from the recovered root, and later ordinary root -rotations again require the recovered/current old-root threshold plus the new-root -threshold. - -Hardware provenance is verified with a provider-neutral X.509 attestation profile: each -active replacement/new root key must be bound by a leaf attestation key whose certificate -chain terminates at an independently SHA-256-pinned trust root. The leaf signs an exact -canonical statement binding the release root key, recovery/continuity context, device -class, and validity interval. Vendor-specific certificate issuance/extension policy can -therefore live behind the pinned attestation CA without placing vendor SDKs or private keys -in this repository. -""" - -from __future__ import annotations - -import argparse -import base64 -import binascii -import hashlib -import json -import logging -import os -import re -import shutil -import sys -import tempfile -from datetime import datetime, timedelta, timezone -from pathlib import Path -from typing import Any, Iterable - -import tomllib - -try: # package import - from . import maintain_release_trust as delegated - from . import seal_release_governance as rootseal -except (ImportError, ValueError): # Space/script/importlib loading - import importlib.util - - def _load_local(name: str, filename: str): - if name in sys.modules: - return sys.modules[name] - path = Path(__file__).resolve().parent / filename - spec = importlib.util.spec_from_file_location(name, path) - if spec is None or spec.loader is None: - raise ImportError(f"cannot load {filename}") - module = importlib.util.module_from_spec(spec) - sys.modules[name] = module - spec.loader.exec_module(module) - return module - - rootseal = _load_local("_release_root_for_continuity", "seal_release_governance.py") - delegated = _load_local( - "_delegated_trust_for_continuity", "maintain_release_trust.py" - ) - -HERE = Path(__file__).resolve().parent -POLICY = tomllib.loads((HERE / "release_root_continuity_policy.toml").read_text()) -PREDICATE_TYPE = str(POLICY["predicate_type"]) -_ZERO_HASH = "0" * 64 -_HEX64 = re.compile(r"^[0-9a-f]{64}$") -_ID = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:/@+-]{0,511}$") -_CHUNK = 1024 * 1024 -logger = logging.getLogger(__name__) - - -class RootContinuityError(RuntimeError): - """Recovered-root continuity or attestation invariant failed.""" - - -def _fail(code: str) -> None: - raise RootContinuityError(code) - - -def _canonical(value: Any) -> bytes: - return ( - json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False) - + "\n" - ).encode("utf-8") - - -def _sha_bytes(raw: bytes) -> str: - return hashlib.sha256(raw).hexdigest() - - -def _sha(path: Path) -> str: - h = hashlib.sha256() - with path.open("rb") as handle: - for chunk in iter(lambda: handle.read(_CHUNK), b""): - h.update(chunk) - return h.hexdigest() - - -def _loads(raw: bytes, code: str) -> dict[str, Any]: - def hook(pairs): - out: dict[str, Any] = {} - for key, value in pairs: - if key in out: - _fail(code + "_DUPLICATE_KEY") - out[key] = value - return out - - try: - value = json.loads(raw.decode("utf-8"), object_pairs_hook=hook) - except RootContinuityError: - raise - except Exception as exc: - raise RootContinuityError(code + "_JSON_INVALID") from exc - if not isinstance(value, dict): - _fail(code + "_SCHEMA_INVALID") - return value - - -def _read( - path: Path, code: str, *, canonical: bool = True -) -> tuple[dict[str, Any], bytes]: - if path.is_symlink() or not path.is_file(): - _fail(code + "_INVALID") - if path.stat().st_size > int(POLICY["max_json_bytes"]): - _fail(code + "_TOO_LARGE") - raw = path.read_bytes() - doc = _loads(raw, code) - if canonical and raw != _canonical(doc): - _fail(code + "_NOT_CANONICAL") - return doc, raw - - -def _write(path: Path, value: dict[str, Any]) -> None: - path.write_bytes(_canonical(value)) - - -def _identity(value: Any, code: str) -> str: - if not isinstance(value, str): - _fail(code) - value = value.strip() - if ( - not value - or len(value) > 512 # ruff: ignore[magic-value-comparison] - or ".." in value - or "?" in value - or "#" in value - or "\x00" in value - ): - _fail(code) - if ( - any( - ord(ch) < 32 # ruff: ignore[magic-value-comparison] - or ord(ch) == 127 # ruff: ignore[magic-value-comparison] - for ch in value - ) - or _ID.fullmatch(value) is None - ): - _fail(code) - return value - - -def _hex(value: Any, code: str) -> str: - if not isinstance(value, str) or _HEX64.fullmatch(value) is None: - _fail(code) - return value - - -def _dt(value: Any, code: str) -> datetime: - if not isinstance(value, str) or not value.endswith("Z"): - _fail(code) - try: - return datetime.fromisoformat(value[:-1] + "+00:00").astimezone(timezone.utc) - except ValueError as exc: - raise RootContinuityError(code) from exc - - -def _b64(value: Any, code: str) -> bytes: - _len = len(value) > 2_000_000 # ruff: ignore[magic-value-comparison] - if not isinstance(value, str) or not value or _len: - _fail(code) - try: - raw = base64.b64decode(value.encode("ascii"), validate=True) - except (UnicodeEncodeError, binascii.Error) as exc: - raise RootContinuityError(code) from exc - if base64.b64encode(raw).decode("ascii") != value: - _fail(code) - return raw - - -def _artifact_raw(name: str, raw: bytes) -> dict[str, Any]: - return {"name": name, "sha256": _sha_bytes(raw), "size": len(raw)} - - -def _regular_dir(path: Path, code: str) -> Path: - if path.is_symlink() or not path.is_dir(): - _fail(code) - return path.resolve() - - -def _outside(path: Path, protected: Iterable[Path], code: str) -> Path: - target = path.expanduser().resolve() - for item in protected: - root = item.resolve() - if target == root or root in target.parents: - _fail(code) - return target - - -def _docs_from_dir(root: Path, names: set[str], code: str) -> dict[str, dict[str, Any]]: - if {p.name for p in root.iterdir()} != names: - _fail(code + "_ALLOWLIST_MISMATCH") - docs: dict[str, dict[str, Any]] = {} - for name in sorted(names): - doc, _ = _read( - root / name, code + "_" + name.upper().replace("-", "_").replace(".", "_") - ) - docs[name] = doc - return docs - - -def _write_docs(root: Path, docs: dict[str, dict[str, Any]]) -> None: - root.mkdir(parents=True, exist_ok=True) - for name, doc in sorted(docs.items()): - _write(root / name, doc) - - -def _cert_der(path: Path, expected_pin: str) -> tuple[bytes, Any]: - _hex(expected_pin, "CONTINUITY_ATTESTATION_ROOT_PIN_INVALID") - if path.is_symlink() or not path.is_file() or path.stat().st_size > 1024 * 1024: - _fail("CONTINUITY_ATTESTATION_ROOT_FILE_INVALID") - raw = path.read_bytes() - try: - from cryptography import x509 # ruff: ignore[import-outside-top-level] - from cryptography.hazmat.primitives import ( # ruff: ignore[import-outside-top-level] - serialization, - ) - - if b"-----BEGIN CERTIFICATE-----" in raw: - cert = x509.load_pem_x509_certificate(raw) - else: - cert = x509.load_der_x509_certificate(raw) - der = cert.public_bytes(serialization.Encoding.DER) - except Exception as exc: - raise RootContinuityError("CONTINUITY_ATTESTATION_ROOT_CERT_INVALID") from exc - if _sha_bytes(der) != expected_pin: - _fail("CONTINUITY_ATTESTATION_ROOT_PIN_MISMATCH") - return der, cert - - -def _verify_cert_signature(cert: Any, issuer_public: Any, code: str) -> None: - try: - from cryptography.hazmat.primitives.asymmetric import ( # ruff: ignore[import-outside-top-level] - padding, - ) - from cryptography.hazmat.primitives.asymmetric.ec import ( # ruff: ignore[import-outside-top-level] - ECDSA, - EllipticCurvePublicKey, - ) - from cryptography.hazmat.primitives.asymmetric.ed25519 import ( # ruff: ignore[import-outside-top-level] - Ed25519PublicKey, - ) - from cryptography.hazmat.primitives.asymmetric.rsa import ( # ruff: ignore[import-outside-top-level] - RSAPublicKey, - ) - - if isinstance(issuer_public, Ed25519PublicKey): - issuer_public.verify(cert.signature, cert.tbs_certificate_bytes) - elif isinstance(issuer_public, EllipticCurvePublicKey): - issuer_public.verify( - cert.signature, - cert.tbs_certificate_bytes, - ECDSA(cert.signature_hash_algorithm), - ) - elif isinstance(issuer_public, RSAPublicKey): - issuer_public.verify( - cert.signature, - cert.tbs_certificate_bytes, - padding.PKCS1v15(), - cert.signature_hash_algorithm, - ) - else: - _fail(code + "_UNSUPPORTED_ISSUER_KEY") - except RootContinuityError: - raise - except Exception as exc: - raise RootContinuityError(code + "_SIGNATURE_INVALID") from exc - - -def _cert_time(cert: Any, field: str) -> datetime: - # cryptography versions differ on the *_utc properties. - value = getattr(cert, field + "_utc", None) - if value is None: - value = getattr(cert, field).replace(tzinfo=timezone.utc) - return value.astimezone(timezone.utc) - - -def _verify_trust_root( - der: bytes, cert: Any, *, now: datetime, historical: bool -) -> dict[str, Any]: - try: - from cryptography import x509 # ruff: ignore[import-outside-top-level] - - bc = cert.extensions.get_extension_for_class(x509.BasicConstraints).value - if not bc.ca: - _fail("CONTINUITY_ATTESTATION_TRUST_ROOT_NOT_CA") - except RootContinuityError: - raise - except Exception as exc: - raise RootContinuityError( - "CONTINUITY_ATTESTATION_TRUST_ROOT_CONSTRAINTS_INVALID" - ) from exc - if cert.issuer != cert.subject: - _fail("CONTINUITY_ATTESTATION_TRUST_ROOT_NOT_SELF_ISSUED") - _verify_cert_signature(cert, cert.public_key(), "CONTINUITY_ATTESTATION_TRUST_ROOT") - not_before = _cert_time(cert, "not_valid_before") - not_after = _cert_time(cert, "not_valid_after") - if not historical and not (not_before <= now <= not_after): - _fail("CONTINUITY_ATTESTATION_TRUST_ROOT_NOT_CURRENT") - return {"sha256": _sha_bytes(der), "der": base64.b64encode(der).decode("ascii")} - - -def _load_attestation_roots( - paths: list[Path], pins: list[str], *, now: datetime, historical: bool = False -) -> tuple[list[dict[str, Any]], dict[str, Any]]: - if len(paths) != len(pins) or len(paths) < int(POLICY["min_attestation_roots"]): - _fail("CONTINUITY_ATTESTATION_ROOT_SET_INVALID") - docs: list[dict[str, Any]] = [] - certs: dict[str, Any] = {} - for path, pin in zip(paths, pins): - der, cert = _cert_der(path, pin) - doc = _verify_trust_root(der, cert, now=now, historical=historical) - if doc["sha256"] in certs: - _fail("CONTINUITY_ATTESTATION_ROOT_DUPLICATE") - docs.append(doc) - certs[doc["sha256"]] = cert - docs.sort(key=lambda x: x["sha256"]) - if [x["sha256"] for x in docs] != sorted(pins): - _fail("CONTINUITY_ATTESTATION_ROOT_PIN_SET_MISMATCH") - return docs, certs - - -def _embedded_attestation_roots( - docs: Any, expected_pins: list[str], *, now: datetime, historical: bool -) -> dict[str, Any]: - if not isinstance(docs, list) or len(docs) < int(POLICY["min_attestation_roots"]): - _fail("CONTINUITY_EMBEDDED_ATTESTATION_ROOTS_INVALID") - expected = sorted( - _hex(x, "CONTINUITY_EXPECTED_ATTESTATION_PIN_INVALID") for x in expected_pins - ) - if [x.get("sha256") for x in docs if isinstance(x, dict)] != expected: - _fail("CONTINUITY_EXPECTED_ATTESTATION_PIN_SET_MISMATCH") - certs: dict[str, Any] = {} - try: - from cryptography import x509 # ruff: ignore[import-outside-top-level] - - for item in docs: - if not isinstance(item, dict) or set(item) != {"sha256", "der"}: - _fail("CONTINUITY_EMBEDDED_ATTESTATION_ROOT_SCHEMA_INVALID") - der = _b64(item["der"], "CONTINUITY_EMBEDDED_ATTESTATION_ROOT_DER_INVALID") - if _sha_bytes(der) != item["sha256"]: - _fail("CONTINUITY_EMBEDDED_ATTESTATION_ROOT_HASH_INVALID") - cert = x509.load_der_x509_certificate(der) - _verify_trust_root(der, cert, now=now, historical=historical) - certs[item["sha256"]] = cert - except RootContinuityError: - raise - except Exception as exc: - raise RootContinuityError( - "CONTINUITY_EMBEDDED_ATTESTATION_ROOT_CERT_INVALID" - ) from exc - return certs - - -def _leaf_algorithm(public_key: Any) -> str: - try: - from cryptography.hazmat.primitives.asymmetric.ec import ( # ruff: ignore[import-outside-top-level] - SECP256R1, - EllipticCurvePublicKey, - ) - from cryptography.hazmat.primitives.asymmetric.ed25519 import ( # ruff: ignore[import-outside-top-level] - Ed25519PublicKey, - ) - - if isinstance(public_key, Ed25519PublicKey): - return "ed25519" - if isinstance(public_key, EllipticCurvePublicKey) and isinstance( - public_key.curve, SECP256R1 - ): - return "ecdsa-p256" - except Exception: # ruff: ignore[blind-except] - pass - _fail("CONTINUITY_ATTESTATION_LEAF_ALGORITHM_UNSUPPORTED") - return None - - -def _verify_leaf_signature(public_key: Any, signature: bytes, raw: bytes) -> None: - try: - from cryptography.hazmat.primitives import ( # ruff: ignore[import-outside-top-level] - hashes, - ) - from cryptography.hazmat.primitives.asymmetric.ec import ( # ruff: ignore[import-outside-top-level] - ECDSA, - EllipticCurvePublicKey, - ) - from cryptography.hazmat.primitives.asymmetric.ed25519 import ( # ruff: ignore[import-outside-top-level] - Ed25519PublicKey, - ) - - if isinstance(public_key, Ed25519PublicKey): - public_key.verify(signature, raw) - elif isinstance(public_key, EllipticCurvePublicKey): - public_key.verify(signature, raw, ECDSA(hashes.SHA256())) - else: - _fail("CONTINUITY_ATTESTATION_LEAF_ALGORITHM_UNSUPPORTED") - except RootContinuityError: - raise - except Exception as exc: - raise RootContinuityError("CONTINUITY_ATTESTATION_SIGNATURE_INVALID") from exc - - -def _verify_attestation( # ruff: ignore[too-many-branches] - doc: dict[str, Any], - *, - root_info: dict[str, Any], - context_id: str, - trust_certs: dict[str, Any], - now: datetime, - historical: bool, -) -> dict[str, Any]: - if set(doc) != {"signature", "signed"} or not isinstance(doc.get("signed"), dict): - _fail("CONTINUITY_ATTESTATION_SCHEMA_INVALID") - s = doc["signed"] - expected = { - "schemaVersion", - "predicateType", - "governanceId", - "contextId", - "rootVersion", - "rootSha256", - "keyId", - "publicKey", - "deviceClass", - "manufacturer", - "model", - "attestedAt", - "expires", - "certificateChainDer", - } - if ( - set(s) != expected - or s["schemaVersion"] != int(POLICY["attestation_schema_version"]) - or s["predicateType"] != PREDICATE_TYPE + "/x509-key-attestation" - ): - _fail("CONTINUITY_ATTESTATION_SIGNED_SCHEMA_INVALID") - if ( - s["governanceId"] != root_info["governance_id"] - or s["contextId"] != context_id - or s["rootVersion"] != root_info["version"] - or s["rootSha256"] != root_info["sha256"] - ): - _fail("CONTINUITY_ATTESTATION_CONTEXT_MISMATCH") - kid = _identity(s["keyId"], "CONTINUITY_ATTESTATION_KEYID_INVALID") - if kid not in root_info["roles"]["root"]["keyids"]: - _fail("CONTINUITY_ATTESTATION_KEY_NOT_ACTIVE_ROOT") - if s["publicKey"] != root_info["keys"][kid]["keyval"]["public"]: - _fail("CONTINUITY_ATTESTATION_PUBLIC_KEY_MISMATCH") - device = _identity(s["deviceClass"], "CONTINUITY_ATTESTATION_DEVICE_CLASS_INVALID") - if device not in set(POLICY["allowed_device_classes"]): - _fail("CONTINUITY_ATTESTATION_DEVICE_CLASS_INVALID") - _identity(s["manufacturer"], "CONTINUITY_ATTESTATION_MANUFACTURER_INVALID") - _identity(s["model"], "CONTINUITY_ATTESTATION_MODEL_INVALID") - attested = _dt(s["attestedAt"], "CONTINUITY_ATTESTATION_TIME_INVALID") - expires = _dt(s["expires"], "CONTINUITY_ATTESTATION_EXPIRES_INVALID") - skew = timedelta(minutes=int(POLICY["max_clock_skew_minutes"])) - if attested > now + skew: - _fail("CONTINUITY_ATTESTATION_FROM_FUTURE") - if expires <= attested or expires - attested > timedelta( - days=int(POLICY["max_attestation_lifetime_days"]) - ): - _fail("CONTINUITY_ATTESTATION_LIFETIME_INVALID") - if not historical and expires < now + timedelta( - minutes=int(POLICY["min_attestation_remaining_minutes"]) - ): - _fail("CONTINUITY_ATTESTATION_EXPIRED_OR_FREEZE_RISK") - chain_raw = s["certificateChainDer"] - if not isinstance(chain_raw, list) or not ( - 1 <= len(chain_raw) <= int(POLICY["max_attestation_chain_length"]) - ): - _fail("CONTINUITY_ATTESTATION_CHAIN_INVALID") - try: - from cryptography import x509 # ruff: ignore[import-outside-top-level] - - chain = [ - x509.load_der_x509_certificate( - _b64(x, "CONTINUITY_ATTESTATION_CERT_DER_INVALID") - ) - for x in chain_raw - ] - leaf = chain[0] - try: - bc = leaf.extensions.get_extension_for_class(x509.BasicConstraints).value - if bc.ca: - _fail("CONTINUITY_ATTESTATION_LEAF_IS_CA") - ku = leaf.extensions.get_extension_for_class(x509.KeyUsage).value - if not ku.digital_signature: - _fail("CONTINUITY_ATTESTATION_LEAF_NO_DIGITAL_SIGNATURE") - except RootContinuityError: - raise - except Exception as exc: - raise RootContinuityError( - "CONTINUITY_ATTESTATION_LEAF_CONSTRAINTS_INVALID" - ) from exc - if _leaf_algorithm(leaf.public_key()) not in set( - POLICY["allowed_leaf_key_algorithms"] - ): - _fail("CONTINUITY_ATTESTATION_LEAF_ALGORITHM_UNSUPPORTED") - for idx in range(len(chain) - 1): - child, issuer = chain[idx], chain[idx + 1] - if child.issuer != issuer.subject: - _fail("CONTINUITY_ATTESTATION_CHAIN_ISSUER_MISMATCH") - try: - if not issuer.extensions.get_extension_for_class( - x509.BasicConstraints - ).value.ca: - _fail("CONTINUITY_ATTESTATION_INTERMEDIATE_NOT_CA") - except RootContinuityError: - raise - except Exception as exc: - raise RootContinuityError( - "CONTINUITY_ATTESTATION_INTERMEDIATE_CONSTRAINTS_INVALID" - ) from exc - _verify_cert_signature( - child, issuer.public_key(), "CONTINUITY_ATTESTATION_CHAIN" - ) - last = chain[-1] - matching = [ - cert for cert in trust_certs.values() if last.issuer == cert.subject - ] - if len(matching) != 1: - _fail("CONTINUITY_ATTESTATION_TRUST_ANCHOR_NOT_UNIQUE") - anchor = matching[0] - _verify_cert_signature( - last, anchor.public_key(), "CONTINUITY_ATTESTATION_ANCHOR" - ) - all_certs = [*chain, anchor] - for cert in all_certs: - if attested < _cert_time(cert, "not_valid_before") or expires > _cert_time( - cert, "not_valid_after" - ): - _fail("CONTINUITY_ATTESTATION_CERT_VALIDITY_MISMATCH") - _verify_leaf_signature( - leaf.public_key(), - _b64(doc["signature"], "CONTINUITY_ATTESTATION_SIGNATURE_ENCODING_INVALID"), - _canonical(s), - ) - except RootContinuityError: - raise - except Exception as exc: - raise RootContinuityError("CONTINUITY_ATTESTATION_CERTIFICATE_INVALID") from exc - return {"signature": doc["signature"], "signed": s} - - -def _attestations( - paths: list[Path], - *, - root_info: dict[str, Any], - context_id: str, - trust_certs: dict[str, Any], - now: datetime, - historical: bool = False, -) -> list[dict[str, Any]]: - docs: list[dict[str, Any]] = [] - seen: set[str] = set() - for path in paths: - doc, _ = _read(path, "CONTINUITY_ATTESTATION") - normalized = _verify_attestation( - doc, - root_info=root_info, - context_id=context_id, - trust_certs=trust_certs, - now=now, - historical=historical, - ) - kid = normalized["signed"]["keyId"] - if kid in seen: - _fail("CONTINUITY_ATTESTATION_DUPLICATE_KEY") - seen.add(kid) - docs.append(normalized) - required = sorted(root_info["roles"]["root"]["keyids"]) - if ( - bool(POLICY["require_all_active_root_keys_attested"]) - and sorted(seen) != required - ): - _fail("CONTINUITY_ATTESTATION_ROOT_KEY_SET_MISMATCH") - docs.sort(key=lambda x: x["signed"]["keyId"]) - return docs - - -def _root_info(envelope: dict[str, Any], code: str) -> dict[str, Any]: - try: - return rootseal._validate_root_envelope(envelope) - except rootseal.RootTrustError as exc: - raise RootContinuityError(code + ":" + str(exc)) from exc - - -def _continuity_head(previous: str, body: dict[str, Any]) -> str: - _hex(previous, "CONTINUITY_PREVIOUS_HEAD_INVALID") - return _sha_bytes((previous + "\n").encode("ascii") + _canonical(body)) - - -def _candidate_from_snapshot(snapshot: dict[str, Any]) -> dict[str, Any]: - expected = { - "schemaVersion", - "governanceState", - "governanceBundle", - "historyState", - "historyBundle", - } - if set(snapshot) != expected or snapshot.get("schemaVersion") != int( - rootseal.governance.POLICY["recovery_snapshot_schema_version"] - ): - _fail("CONTINUITY_GOVERNANCE_SNAPSHOT_SCHEMA_INVALID") - state = snapshot["governanceState"] - bundle = snapshot["governanceBundle"] - if not isinstance(state, dict) or not isinstance(bundle, dict): - _fail("CONTINUITY_GOVERNANCE_SNAPSHOT_SCHEMA_INVALID") - with tempfile.TemporaryDirectory(prefix="run157-governance-verify-") as td: - sp = Path(td) / "state.json" - bp = Path(td) / "bundle.json" - _write(sp, state) - _write(bp, bundle) - try: - rootseal.governance.verify_governance_bundle(bundle_path=bp, state_path=sp) - except rootseal.governance.GovernanceError as exc: - raise RootContinuityError( - "CONTINUITY_GOVERNANCE_BUNDLE_INVALID:" + str(exc) - ) from exc - entries = bundle.get("entries") - if not isinstance(entries, list) or not entries: - _fail("CONTINUITY_GOVERNANCE_ENTRIES_INVALID") - entry = entries[-1] - proposal = entry.get("proposal") - if not isinstance(proposal, dict): - _fail("CONTINUITY_GOVERNANCE_PROPOSAL_INVALID") - previous_policy = ( - bundle["genesis"]["initialPolicy"] - if len(entries) == 1 - else entries[-2]["nextPolicy"] - ) - revoked_before: set[str] = set() - for old in entries[:-1]: - revoked_before.update(old.get("revokedAuthorityKeyIds", [])) - raw_state = _canonical(state) - raw_bundle = _canonical(bundle) - raw_snapshot = _canonical(snapshot) - raw_proposal = _canonical(proposal) - return { - "state": state, - "bundle": bundle, - "snapshot": snapshot, - "entry": entry, - "previous_policy": previous_policy, - "final_policy": state["policy"], - "revoked_before": revoked_before, - "state_sha256": _sha_bytes(raw_state), - "bundle_sha256": _sha_bytes(raw_bundle), - "snapshot_sha256": _sha_bytes(raw_snapshot), - "snapshot_size": len(raw_snapshot), - "proposal_sha256": _sha_bytes(raw_proposal), - } - - -# ── Verified-state document names ──────────────────────────────────────── -# -# One constant per release-evidence document. The values are the on-disk -# artifact names and are unchanged; only the repetition is removed. Reading -# a document by a name that says which document it is also keeps static -# analysis from reading the filename token 'trusted' as 'confidential': -# these files carry published Merkle roots and SHA-256 digests, which must -# stay in clear text for any third party to verify them. -_DOC_GOVERNANCE_STATE = "trusted-governance-state.json" -_DOC_RELEASE_ROOT_STATE = "trusted-release-root-state.json" -_DOC_ROOT_CONTINUITY_STATE = "trusted-root-continuity-state.json" - - -def _candidate_subject( - candidate: dict[str, Any], auth_root: dict[str, Any] -) -> dict[str, Any]: - entry = candidate["entry"] - role = "emergency" if entry.get("approvalRole") == "emergency" else "governance" - state_raw = _canonical(candidate["state"]) - bundle_raw = _canonical(candidate["bundle"]) - snapshot_raw = _canonical(candidate["snapshot"]) - return { - "schemaVersion": int(rootseal.POLICY["schema_version"]), - "predicateType": rootseal.PREDICATE_TYPE + "/governance-authorization", - "governanceId": candidate["state"]["governanceId"], - "historyId": candidate["state"]["historyId"], - "epoch": candidate["state"]["epoch"], - "policyVersion": candidate["state"]["policy"]["policyVersion"], - "transitionId": entry["transitionId"], - "role": role, - "proposalSha256": candidate["proposal_sha256"], - "selectedKeyIds": sorted(entry["selectedApproverKeyIds"]), - "authorizingRoot": { - "version": auth_root["version"], - "sha256": auth_root["sha256"], - }, - "governanceState": { - "name": _DOC_GOVERNANCE_STATE, - "sha256": candidate["state_sha256"], - "size": len(state_raw), - }, - "governanceBundle": { - "name": "release-governance-bundle.json", - "sha256": candidate["bundle_sha256"], - "size": len(bundle_raw), - }, - "recoverySnapshot": { - "name": "release-governance-recovery-snapshot.json", - "sha256": candidate["snapshot_sha256"], - "size": len(snapshot_raw), - }, - "history": candidate["state"]["history"], - } - - -def _verify_authorization( - authorization: dict[str, Any], - *, - candidate: dict[str, Any], - auth_root: dict[str, Any], - historical: bool, - now: datetime, -) -> None: - expected = { - "schemaVersion", - "predicateType", - "status", - "subject", - "subjectSha256", - "rootVersion", - "rootSha256", - "role", - "selectedKeyIds", - "signatures", - } - if ( - set(authorization) != expected - or authorization["schemaVersion"] - != int(rootseal.POLICY["authorization_schema_version"]) - or authorization["predicateType"] - != rootseal.PREDICATE_TYPE + "/governance-authorization" - or authorization["status"] != "cryptographically-authorized" - ): - _fail("CONTINUITY_AUTHORIZATION_SCHEMA_INVALID") - role_name = ( - "emergency" - if candidate["entry"].get("approvalRole") == "emergency" - else "governance" - ) - try: - rootseal._assert_role_matches_policy( - auth_root, - role_name, - candidate["previous_policy"], - candidate["revoked_before"], - "CONTINUITY_AUTHORIZING_ROLE", - ) - other = "governance" if role_name == "emergency" else "emergency" - rootseal._assert_role_matches_policy( - auth_root, - other, - candidate["previous_policy"], - candidate["revoked_before"], - "CONTINUITY_AUTHORIZING_OTHER_ROLE", - ) - except rootseal.RootTrustError as exc: - raise RootContinuityError(str(exc)) from exc - subject = _candidate_subject(candidate, auth_root) - if ( - authorization["subject"] != subject - or authorization["subjectSha256"] != _sha_bytes(_canonical(subject)) - or authorization["rootVersion"] != auth_root["version"] - or authorization["rootSha256"] != auth_root["sha256"] - or authorization["role"] != role_name - ): - _fail("CONTINUITY_AUTHORIZATION_REBIND_FAILED") - selected = subject["selectedKeyIds"] - if authorization["selectedKeyIds"] != selected: - _fail("CONTINUITY_AUTHORIZATION_SELECTED_SET_INVALID") - policy_field = ( - "emergencyAuthority" if role_name == "emergency" else "policyAuthority" - ) - members = { - m["keyId"]: m for m in candidate["previous_policy"][policy_field]["members"] - } - sigs = authorization["signatures"] - if not isinstance(sigs, list) or len(sigs) != len(selected): - _fail("CONTINUITY_AUTHORIZATION_SIGNATURE_COUNT_INVALID") - normalized: list[dict[str, Any]] = [] - seen: set[str] = set() - for doc in sigs: - kid = doc.get("signed", {}).get("keyId") if isinstance(doc, dict) else None - if kid not in selected or kid in seen: - _fail("CONTINUITY_AUTHORIZATION_SIGNATURE_SET_INVALID") - seen.add(kid) - try: - normalized.append( - rootseal._validate_authorization_signature( - doc, - subject=subject, - root_info=auth_root, - role_name=role_name, - expected_member=members[kid], - now=now, - enforce_freshness=not historical, - ) - ) - except rootseal.RootTrustError as exc: - raise RootContinuityError( - "CONTINUITY_AUTHORIZATION_INVALID:" + str(exc) - ) from exc - normalized.sort(key=lambda x: x["signed"]["keyId"]) - if sigs != normalized or sorted(seen) != selected: - _fail("CONTINUITY_AUTHORIZATION_NOT_NORMALIZED") - - -def _authorization_from_files( - paths: list[Path], - *, - candidate: dict[str, Any], - auth_root: dict[str, Any], - now: datetime, -) -> dict[str, Any]: - role_name = ( - "emergency" - if candidate["entry"].get("approvalRole") == "emergency" - else "governance" - ) - try: - rootseal._assert_role_matches_policy( - auth_root, - role_name, - candidate["previous_policy"], - candidate["revoked_before"], - "CONTINUITY_AUTHORIZING_ROLE", - ) - other = "governance" if role_name == "emergency" else "emergency" - rootseal._assert_role_matches_policy( - auth_root, - other, - candidate["previous_policy"], - candidate["revoked_before"], - "CONTINUITY_AUTHORIZING_OTHER_ROLE", - ) - except rootseal.RootTrustError as exc: - raise RootContinuityError(str(exc)) from exc - subject = _candidate_subject(candidate, auth_root) - selected = subject["selectedKeyIds"] - if len(paths) != len(selected): - _fail("CONTINUITY_AUTH_SIGNATURE_COUNT_INVALID") - policy_field = ( - "emergencyAuthority" if role_name == "emergency" else "policyAuthority" - ) - members = { - m["keyId"]: m for m in candidate["previous_policy"][policy_field]["members"] - } - by_key: dict[str, dict[str, Any]] = {} - for path in paths: - doc, _ = _read(path, "CONTINUITY_AUTH_SIGNATURE") - kid = ( - doc.get("signed", {}).get("keyId") - if isinstance(doc.get("signed"), dict) - else None - ) - if kid not in selected or kid in by_key: - _fail("CONTINUITY_AUTH_SIGNATURE_KEY_SET_INVALID") - try: - by_key[kid] = rootseal._validate_authorization_signature( - doc, - subject=subject, - root_info=auth_root, - role_name=role_name, - expected_member=members[kid], - now=now, - ) - except rootseal.RootTrustError as exc: - raise RootContinuityError( - "CONTINUITY_AUTH_SIGNATURE_INVALID:" + str(exc) - ) from exc - if sorted(by_key) != selected: - _fail("CONTINUITY_AUTH_SIGNATURE_SET_MISMATCH") - return { - "schemaVersion": int(rootseal.POLICY["authorization_schema_version"]), - "predicateType": rootseal.PREDICATE_TYPE + "/governance-authorization", - "status": "cryptographically-authorized", - "subject": subject, - "subjectSha256": _sha_bytes(_canonical(subject)), - "rootVersion": auth_root["version"], - "rootSha256": auth_root["sha256"], - "role": role_name, - "selectedKeyIds": selected, - "signatures": [by_key[k] for k in sorted(by_key)], - } - - -def _state( - *, - bundle: dict[str, Any], - active_root: dict[str, Any], - chain_head: str, - governance_binding: dict[str, Any], - recovery_record_sha: str, -) -> dict[str, Any]: - bundle_sha = _sha_bytes(_canonical(bundle)) - return { - "schemaVersion": int(POLICY["state_schema_version"]), - "status": "trusted-recovered-root-continuity", - "governanceId": active_root["governance_id"], - "bootstrapRootSha256": bundle["bootstrapRootSha256"], - "recoveryRootSha256": bundle["recoveryRootSha256"], - "activeRoot": { - "version": active_root["version"], - "sha256": active_root["sha256"], - "expires": active_root["signed"]["expires"], - }, - "rootChainHeadSha256": chain_head, - "bundleSha256": bundle_sha, - "governance": governance_binding, - "recovery": { - "incidentId": bundle["recoveryEvent"]["incidentId"], - "recordSha256": recovery_record_sha, - }, - } - - -def _governance_binding_from_run155_state(state: dict[str, Any]) -> dict[str, Any]: - g = state["governance"] - return { - "epoch": g["epoch"], - "policyVersion": g["policyVersion"], - "stateSha256": g["stateSha256"], - "bundleSha256": g["bundleSha256"], - "recoverySnapshotSha256": g["recoverySnapshotSha256"], - "proposalSha256": g["proposalSha256"], - } - - -def _governance_binding_from_candidate(candidate: dict[str, Any]) -> dict[str, Any]: - return { - "epoch": candidate["state"]["epoch"], - "policyVersion": candidate["state"]["policy"]["policyVersion"], - "stateSha256": candidate["state_sha256"], - "bundleSha256": candidate["bundle_sha256"], - "recoverySnapshotSha256": candidate["snapshot_sha256"], - "proposalSha256": candidate["proposal_sha256"], - } - - -def activate_recovery( # ruff: ignore[undocumented-public-function] - *, - sealed_dir: Path, - bootstrap_root_sha256: str, - recovery_dir: Path, - expected_recovery_root_sha256: str, - attestation_paths: list[Path], - attestation_trust_root_paths: list[Path], - expected_attestation_root_sha256: list[str], - output_dir: Path, - now: datetime | None = None, -) -> dict[str, Any]: - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - sealed = _regular_dir(sealed_dir, "CONTINUITY_RUN155_DIR_INVALID") - recovery = _regular_dir(recovery_dir, "CONTINUITY_RUN156_RECOVERY_DIR_INVALID") - try: - rootseal.verify_sealed_governance( - sealed_dir=sealed, - expected_bootstrap_root_sha256=bootstrap_root_sha256, - now=current, - require_current_fresh=True, - ) - delegated.verify_recovery_output( - output_dir=recovery, - sealed_dir=sealed, - bootstrap_root_sha256=bootstrap_root_sha256, - expected_recovery_root_sha256=expected_recovery_root_sha256, - now=current, - ) - except (rootseal.RootTrustError, delegated.DelegatedTrustError) as exc: - raise RootContinuityError("CONTINUITY_PREDECESSOR_INVALID:" + str(exc)) from exc - base_names = { - "release-root-bundle.json", - _DOC_RELEASE_ROOT_STATE, - "cryptographic-governance-authorization.json", - "active-root.json", - "release-governance-recovery-snapshot.json", - "release-root-receipt.json", - } - recovery_names = { - "release-root-recovery-record.json", - "recovered-effective-root.json", - "release-root-recovery-receipt.json", - } - base_docs = _docs_from_dir(sealed, base_names, "CONTINUITY_BASE_SEAL") - recovery_docs = _docs_from_dir( - recovery, recovery_names, "CONTINUITY_RECOVERY_OUTPUT" - ) - active = _root_info( - recovery_docs["recovered-effective-root.json"], - "CONTINUITY_RECOVERED_ROOT_INVALID", - ) - base_state = base_docs[_DOC_RELEASE_ROOT_STATE] - if active["version"] != base_state["rootVersion"] + 1: - _fail("CONTINUITY_RECOVERED_ROOT_VERSION_INVALID") - roots_embedded, trust_certs = _load_attestation_roots( - attestation_trust_root_paths, expected_attestation_root_sha256, now=current - ) - incident_id = recovery_docs["release-root-recovery-record.json"]["subject"][ - "incidentId" - ] - attestations = _attestations( - attestation_paths, - root_info=active, - context_id=incident_id, - trust_certs=trust_certs, - now=current, - ) - recovery_record_raw = _canonical(recovery_docs["release-root-recovery-record.json"]) - attestation_sha = _sha_bytes(_canonical({"attestations": attestations})) - body = { - "type": "root-recovery", - "sequence": 1, - "previousChainHeadSha256": base_state["rootChainHeadSha256"], - "incidentId": incident_id, - "recoveryRecordSha256": _sha_bytes(recovery_record_raw), - "replacementRoot": {"version": active["version"], "sha256": active["sha256"]}, - "attestationsSha256": attestation_sha, - } - chain = _continuity_head(base_state["rootChainHeadSha256"], body) - recovery_event = dict(body, chainHeadSha256=chain) - bundle = { - "schemaVersion": int(POLICY["bundle_schema_version"]), - "predicateType": PREDICATE_TYPE, - "status": "trusted-recovered-root-continuity-chain", - "governanceId": active["governance_id"], - "bootstrapRootSha256": _hex( - bootstrap_root_sha256, "CONTINUITY_BOOTSTRAP_PIN_INVALID" - ), - "recoveryRootSha256": _hex( - expected_recovery_root_sha256, "CONTINUITY_RECOVERY_PIN_INVALID" - ), - "attestationTrustRoots": roots_embedded, - "baseSeal": base_docs, - "recoveryOutput": recovery_docs, - "recoveryAttestations": attestations, - "recoveryEvent": recovery_event, - "epochs": [], - } - state = _state( - bundle=bundle, - active_root=active, - chain_head=chain, - governance_binding=_governance_binding_from_run155_state(base_state), - recovery_record_sha=_sha_bytes(recovery_record_raw), - ) - target = _outside( - output_dir, - [sealed, recovery] - + [p.resolve() for p in attestation_paths + attestation_trust_root_paths], - "CONTINUITY_OUTPUT_INSIDE_INPUT", - ) - if target.exists() or target.is_symlink(): - _fail("CONTINUITY_OUTPUT_ALREADY_EXISTS") - inputs = ( - [p for p in sealed.rglob("*") if p.is_file()] - + [p for p in recovery.rglob("*") if p.is_file()] - + attestation_paths - + attestation_trust_root_paths - ) - before = {str(p.resolve()): _sha(p) for p in inputs} - target.parent.mkdir(parents=True, exist_ok=True) - with tempfile.TemporaryDirectory( - prefix="run157-activate-", dir=target.parent - ) as td: - stage = Path(td) / "continuity" - stage.mkdir() - _write(stage / "release-root-continuity-bundle.json", bundle) - _write(stage / _DOC_ROOT_CONTINUITY_STATE, state) - _write(stage / "active-root.json", active["envelope"]) - receipt = { - "schemaVersion": int(POLICY["receipt_schema_version"]), - "status": "recovered-root-activated", - "governanceId": active["governance_id"], - "incidentId": incident_id, - "bundle": _artifact_raw( - "release-root-continuity-bundle.json", _canonical(bundle) - ), - "state": _artifact_raw(_DOC_ROOT_CONTINUITY_STATE, _canonical(state)), - "activeRoot": _artifact_raw( - "active-root.json", _canonical(active["envelope"]) - ), - "attestationRootSha256": sorted(expected_attestation_root_sha256), - } - _write(stage / "release-root-continuity-receipt.json", receipt) - for p in inputs: - if _sha(p) != before[str(p.resolve())]: - _fail("CONTINUITY_INPUT_CHANGED_DURING_ACTIVATION") - temp_target = target.with_name(target.name + ".tmp-" + os.urandom(6).hex()) - shutil.copytree(stage, temp_target, copy_function=shutil.copy2) - os.replace(temp_target, target) - verify_continuity( - output_dir=target, - expected_bootstrap_root_sha256=bootstrap_root_sha256, - expected_recovery_root_sha256=expected_recovery_root_sha256, - expected_attestation_root_sha256=expected_attestation_root_sha256, - now=current, - ) - return { - "ok": True, - "phase": "recovered-root-activated", - "active_root_version": active["version"], - "active_root_sha256": active["sha256"], - "root_chain_head_sha256": chain, - } - - -def _verify_base_and_recovery( - bundle: dict[str, Any], - *, - bootstrap_pin: str, - recovery_pin: str, - attestation_pins: list[str], - now: datetime, - historical: bool, -) -> tuple[dict[str, Any], dict[str, Any], str, dict[str, Any]]: - base_docs = bundle["baseSeal"] - recovery_docs = bundle["recoveryOutput"] - if not isinstance(base_docs, dict) or not isinstance(recovery_docs, dict): - _fail("CONTINUITY_EMBEDDED_PREDECESSOR_SCHEMA_INVALID") - with tempfile.TemporaryDirectory(prefix="run157-predecessor-") as td: - sd = Path(td) / "sealed" - rd = Path(td) / "recovery" - _write_docs(sd, base_docs) - _write_docs(rd, recovery_docs) - try: - rootseal.verify_sealed_governance( - sealed_dir=sd, - expected_bootstrap_root_sha256=bootstrap_pin, - now=now, - require_current_fresh=False, - ) - delegated.verify_recovery_output( - output_dir=rd, - sealed_dir=sd, - bootstrap_root_sha256=bootstrap_pin, - expected_recovery_root_sha256=recovery_pin, - now=now, - historical=True, - ) - except (rootseal.RootTrustError, delegated.DelegatedTrustError) as exc: - raise RootContinuityError( - "CONTINUITY_EMBEDDED_PREDECESSOR_INVALID:" + str(exc) - ) from exc - base_state = base_docs[_DOC_RELEASE_ROOT_STATE] - active = _root_info( - recovery_docs["recovered-effective-root.json"], - "CONTINUITY_EMBEDDED_RECOVERED_ROOT_INVALID", - ) - trust_certs = _embedded_attestation_roots( - bundle["attestationTrustRoots"], attestation_pins, now=now, historical=True - ) - incident = recovery_docs["release-root-recovery-record.json"]["subject"][ - "incidentId" - ] - attest_raw = bundle["recoveryAttestations"] - if not isinstance(attest_raw, list): - _fail("CONTINUITY_RECOVERY_ATTESTATIONS_INVALID") - normalized = [] - for doc in attest_raw: - if not isinstance(doc, dict): - _fail("CONTINUITY_RECOVERY_ATTESTATIONS_INVALID") - normalized.append( - _verify_attestation( - doc, - root_info=active, - context_id=incident, - trust_certs=trust_certs, - now=now, - historical=True, - ) - ) - normalized.sort(key=lambda x: x["signed"]["keyId"]) - required = sorted(active["roles"]["root"]["keyids"]) - if ( - normalized != attest_raw - or sorted(x["signed"]["keyId"] for x in normalized) != required - ): - _fail("CONTINUITY_RECOVERY_ATTESTATION_SET_INVALID") - record_sha = _sha_bytes( - _canonical(recovery_docs["release-root-recovery-record.json"]) - ) - body = { - "type": "root-recovery", - "sequence": 1, - "previousChainHeadSha256": base_state["rootChainHeadSha256"], - "incidentId": incident, - "recoveryRecordSha256": record_sha, - "replacementRoot": {"version": active["version"], "sha256": active["sha256"]}, - "attestationsSha256": _sha_bytes(_canonical({"attestations": normalized})), - } - chain = _continuity_head(base_state["rootChainHeadSha256"], body) - expected_event = dict(body, chainHeadSha256=chain) - if bundle["recoveryEvent"] != expected_event: - _fail("CONTINUITY_RECOVERY_EVENT_REBIND_FAILED") - return active, base_state, chain, trust_certs - - -def verify_continuity( # ruff: ignore[too-many-branches, undocumented-public-function] - *, - output_dir: Path, - expected_bootstrap_root_sha256: str, - expected_recovery_root_sha256: str, - expected_attestation_root_sha256: list[str], - now: datetime | None = None, - historical: bool = False, -) -> dict[str, Any]: - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - root = _regular_dir(output_dir, "CONTINUITY_VERIFY_DIR_INVALID") - allowed = { - "release-root-continuity-bundle.json", - _DOC_ROOT_CONTINUITY_STATE, - "active-root.json", - "release-root-continuity-receipt.json", - } - if {p.name for p in root.iterdir()} != allowed: - _fail("CONTINUITY_VERIFY_ALLOWLIST_MISMATCH") - bundle, bundle_raw = _read( - root / "release-root-continuity-bundle.json", "CONTINUITY_VERIFY_BUNDLE" - ) - state, state_raw = _read( - root / _DOC_ROOT_CONTINUITY_STATE, "CONTINUITY_VERIFY_STATE" - ) - active_doc, active_raw = _read( - root / "active-root.json", "CONTINUITY_VERIFY_ACTIVE_ROOT" - ) - expected_bundle_keys = { - "schemaVersion", - "predicateType", - "status", - "governanceId", - "bootstrapRootSha256", - "recoveryRootSha256", - "attestationTrustRoots", - "baseSeal", - "recoveryOutput", - "recoveryAttestations", - "recoveryEvent", - "epochs", - } - if ( - set(bundle) != expected_bundle_keys - or bundle["schemaVersion"] != int(POLICY["bundle_schema_version"]) - or bundle["predicateType"] != PREDICATE_TYPE - or bundle["status"] != "trusted-recovered-root-continuity-chain" - ): - _fail("CONTINUITY_VERIFY_BUNDLE_SCHEMA_INVALID") - if bundle["bootstrapRootSha256"] != _hex( - expected_bootstrap_root_sha256, "CONTINUITY_VERIFY_BOOTSTRAP_PIN_INVALID" - ) or bundle["recoveryRootSha256"] != _hex( - expected_recovery_root_sha256, "CONTINUITY_VERIFY_RECOVERY_PIN_INVALID" - ): - _fail("CONTINUITY_VERIFY_PIN_MISMATCH") - active, base_state, chain, trust_certs = _verify_base_and_recovery( - bundle, - bootstrap_pin=expected_bootstrap_root_sha256, - recovery_pin=expected_recovery_root_sha256, - attestation_pins=expected_attestation_root_sha256, - now=current, - historical=historical, - ) - governance_binding = _governance_binding_from_run155_state(base_state) - recovery_record = bundle["recoveryOutput"]["release-root-recovery-record.json"] - recovery_revoked = set(recovery_record.get("revokedRootKeyIds", [])) - if recovery_revoked & set(active["keys"]): - _fail("CONTINUITY_VERIFY_RECOVERY_REVOKED_KEY_ACTIVE") - epochs = bundle["epochs"] - if not isinstance(epochs, list) or len(epochs) > int( - POLICY["max_continuity_epochs"] - ): - _fail("CONTINUITY_VERIFY_EPOCHS_INVALID") - for index, entry in enumerate(epochs, 1): - if not isinstance(entry, dict): - _fail("CONTINUITY_VERIFY_EPOCH_SCHEMA_INVALID") - expected_entry_keys = { - "sequence", - "epoch", - "previousChainHeadSha256", - "governanceSnapshot", - "authorization", - "nextRoot", - "rootAttestations", - "activeRoot", - "chainHeadSha256", - } - if ( - set(entry) != expected_entry_keys - or entry["sequence"] != index + 1 - or entry["previousChainHeadSha256"] != chain - ): - _fail("CONTINUITY_VERIFY_EPOCH_SCHEMA_INVALID") - candidate = _candidate_from_snapshot(entry["governanceSnapshot"]) - if ( - candidate["state"]["epoch"] != governance_binding["epoch"] + 1 - or candidate["entry"]["proposal"].get("previousGovernanceStateSha256") - != governance_binding["stateSha256"] - ): - _fail("CONTINUITY_VERIFY_GOVERNANCE_SEQUENCE_INVALID") - _verify_authorization( - entry["authorization"], - candidate=candidate, - auth_root=active, - historical=True, - now=current, - ) - authority_changed = any( - rootseal._policy_role(candidate["previous_policy"], r) - != rootseal._policy_role(candidate["final_policy"], r) - for r in ("governance", "emergency") - ) - next_doc = entry["nextRoot"] - root_att = entry["rootAttestations"] - if next_doc is None: - if authority_changed: - _fail("CONTINUITY_VERIFY_ROTATION_REQUIRED") - if root_att != []: - _fail("CONTINUITY_VERIFY_UNEXPECTED_ATTESTATIONS") - next_active = active - else: - if not isinstance(next_doc, dict): - _fail("CONTINUITY_VERIFY_NEXT_ROOT_INVALID") - next_info = _root_info(next_doc, "CONTINUITY_VERIFY_NEXT_ROOT_INVALID") - if recovery_revoked & set(next_info["keys"]): - _fail("CONTINUITY_VERIFY_RECOVERY_REVOKED_KEY_REINTRODUCED") - try: - next_active = rootseal._verify_rotation( - active, next_info, now=next_info["issued"] - ) - except rootseal.RootTrustError as exc: - raise RootContinuityError( - "CONTINUITY_VERIFY_ROTATION_INVALID:" + str(exc) - ) from exc - context = f"governance-epoch-{candidate['state']['epoch']}" - if not isinstance(root_att, list): - _fail("CONTINUITY_VERIFY_ROOT_ATTESTATIONS_INVALID") - normalized = [] - for doc in root_att: - if not isinstance(doc, dict): - _fail("CONTINUITY_VERIFY_ROOT_ATTESTATIONS_INVALID") - normalized.append( - _verify_attestation( - doc, - root_info=next_active, - context_id=context, - trust_certs=trust_certs, - now=current, - historical=True, - ) - ) - normalized.sort(key=lambda x: x["signed"]["keyId"]) - if normalized != root_att or sorted( - x["signed"]["keyId"] for x in normalized - ) != sorted(next_active["roles"]["root"]["keyids"]): - _fail("CONTINUITY_VERIFY_ROOT_ATTESTATION_SET_INVALID") - revoked_after = set(candidate["state"].get("revokedAuthorityKeyIds", [])) - try: - rootseal._assert_role_matches_policy( - next_active, - "governance", - candidate["final_policy"], - revoked_after, - "CONTINUITY_FINAL_GOVERNANCE_ROLE", - ) - rootseal._assert_role_matches_policy( - next_active, - "emergency", - candidate["final_policy"], - revoked_after, - "CONTINUITY_FINAL_EMERGENCY_ROLE", - ) - except rootseal.RootTrustError as exc: - raise RootContinuityError(str(exc)) from exc - if ( - entry["activeRoot"] != next_active["envelope"] - or entry["epoch"] != candidate["state"]["epoch"] - ): - _fail("CONTINUITY_VERIFY_ACTIVE_ROOT_REBIND_FAILED") - body = {k: v for k, v in entry.items() if k != "chainHeadSha256"} - calc = _continuity_head(chain, body) - if entry["chainHeadSha256"] != calc: - _fail("CONTINUITY_VERIFY_CHAIN_HASH_INVALID") - chain = calc - active = next_active - governance_binding = _governance_binding_from_candidate(candidate) - if not historical: - skew = timedelta(minutes=int(rootseal.POLICY["max_clock_skew_minutes"])) - if active["issued"] > current + skew: - _fail("CONTINUITY_VERIFY_ACTIVE_ROOT_FROM_FUTURE") - if active["expires"] < current + timedelta( - minutes=int(rootseal.POLICY["min_root_remaining_minutes"]) - ): - _fail("CONTINUITY_VERIFY_ACTIVE_ROOT_EXPIRED_OR_FREEZE_RISK") - expected_state = { - "schemaVersion": int(POLICY["state_schema_version"]), - "status": "trusted-recovered-root-continuity", - "governanceId": active["governance_id"], - "bootstrapRootSha256": bundle["bootstrapRootSha256"], - "recoveryRootSha256": bundle["recoveryRootSha256"], - "activeRoot": { - "version": active["version"], - "sha256": active["sha256"], - "expires": active["signed"]["expires"], - }, - "rootChainHeadSha256": chain, - "bundleSha256": _sha_bytes(bundle_raw), - "governance": governance_binding, - "recovery": { - "incidentId": bundle["recoveryEvent"]["incidentId"], - "recordSha256": bundle["recoveryEvent"]["recoveryRecordSha256"], - }, - } - if state != expected_state: - _fail("CONTINUITY_VERIFY_STATE_REBIND_FAILED") - if active_doc != active["envelope"] or _sha_bytes(active_raw) != active["sha256"]: - _fail("CONTINUITY_VERIFY_ACTIVE_ROOT_FILE_MISMATCH") - receipt, _ = _read( - root / "release-root-continuity-receipt.json", "CONTINUITY_VERIFY_RECEIPT" - ) - expected_receipt = { - "schemaVersion": int(POLICY["receipt_schema_version"]), - "status": ( - "recovered-root-activated" - if not epochs - else "recovered-root-continuity-advanced" - ), - "governanceId": active["governance_id"], - "incidentId": bundle["recoveryEvent"]["incidentId"], - "bundle": _artifact_raw("release-root-continuity-bundle.json", bundle_raw), - "state": _artifact_raw(_DOC_ROOT_CONTINUITY_STATE, state_raw), - "activeRoot": _artifact_raw("active-root.json", active_raw), - "attestationRootSha256": sorted(expected_attestation_root_sha256), - } - if receipt != expected_receipt: - _fail("CONTINUITY_VERIFY_RECEIPT_REBIND_FAILED") - return { - "ok": True, - "phase": "recovered-root-continuity-verified", - "governance_epoch": governance_binding["epoch"], - "active_root_version": active["version"], - "active_root_sha256": active["sha256"], - "root_chain_head_sha256": chain, - } - - -def advance_governance( # ruff: ignore[undocumented-public-function] - *, - previous_dir: Path, - governance_dir: Path, - authorization_signature_paths: list[Path], - output_dir: Path, - expected_bootstrap_root_sha256: str, - expected_recovery_root_sha256: str, - expected_attestation_root_sha256: list[str], - next_root_path: Path | None = None, - root_attestation_paths: list[Path] | None = None, - now: datetime | None = None, -) -> dict[str, Any]: - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - prev = _regular_dir(previous_dir, "CONTINUITY_ADVANCE_PREVIOUS_INVALID") - verified = verify_continuity( - output_dir=prev, - expected_bootstrap_root_sha256=expected_bootstrap_root_sha256, - expected_recovery_root_sha256=expected_recovery_root_sha256, - expected_attestation_root_sha256=expected_attestation_root_sha256, - now=current, - ) - bundle, _ = _read( - prev / "release-root-continuity-bundle.json", - "CONTINUITY_ADVANCE_PREVIOUS_BUNDLE", - ) - state, _ = _read( - prev / _DOC_ROOT_CONTINUITY_STATE, "CONTINUITY_ADVANCE_PREVIOUS_STATE" - ) - active_doc, _ = _read( - prev / "active-root.json", "CONTINUITY_ADVANCE_PREVIOUS_ACTIVE" - ) - active = _root_info(active_doc, "CONTINUITY_ADVANCE_ACTIVE_ROOT_INVALID") - try: - candidate = rootseal._validate_run154_dir( - _regular_dir(governance_dir, "CONTINUITY_ADVANCE_GOVERNANCE_DIR_INVALID") - ) - except rootseal.RootTrustError as exc: - raise RootContinuityError( - "CONTINUITY_ADVANCE_CANDIDATE_INVALID:" + str(exc) - ) from exc - if ( - candidate["state"]["epoch"] != state["governance"]["epoch"] + 1 - or candidate["entry"]["proposal"].get("previousGovernanceStateSha256") - != state["governance"]["stateSha256"] - ): - _fail("CONTINUITY_ADVANCE_GOVERNANCE_SEQUENCE_INVALID") - authorization = _authorization_from_files( - authorization_signature_paths, - candidate=candidate, - auth_root=active, - now=current, - ) - authority_changed = any( - rootseal._policy_role(candidate["previous_policy"], r) - != rootseal._policy_role(candidate["final_policy"], r) - for r in ("governance", "emergency") - ) - next_doc = None - next_active = active - attestations = [] - trust_certs = _embedded_attestation_roots( - bundle["attestationTrustRoots"], - expected_attestation_root_sha256, - now=current, - historical=False, - ) - if next_root_path is not None: - next_doc, _ = _read(next_root_path, "CONTINUITY_ADVANCE_NEXT_ROOT") - next_info = _root_info(next_doc, "CONTINUITY_ADVANCE_NEXT_ROOT_INVALID") - recovery_revoked = set( - bundle["recoveryOutput"]["release-root-recovery-record.json"].get( - "revokedRootKeyIds", [] - ) - ) - if recovery_revoked & set(next_info["keys"]): - _fail("CONTINUITY_ADVANCE_RECOVERY_REVOKED_KEY_REINTRODUCED") - try: - next_active = rootseal._verify_rotation(active, next_info, now=current) - except rootseal.RootTrustError as exc: - raise RootContinuityError( - "CONTINUITY_ADVANCE_ROTATION_INVALID:" + str(exc) - ) from exc - context = f"governance-epoch-{candidate['state']['epoch']}" - attestations = _attestations( - root_attestation_paths or [], - root_info=next_active, - context_id=context, - trust_certs=trust_certs, - now=current, - ) - elif authority_changed: - _fail("CONTINUITY_ADVANCE_ROTATION_REQUIRED_FOR_AUTHORITY_CHANGE") - elif root_attestation_paths: - _fail("CONTINUITY_ADVANCE_ATTESTATIONS_WITHOUT_ROTATION") - revoked_after = set(candidate["state"].get("revokedAuthorityKeyIds", [])) - try: - rootseal._assert_role_matches_policy( - next_active, - "governance", - candidate["final_policy"], - revoked_after, - "CONTINUITY_ADVANCE_FINAL_GOVERNANCE_ROLE", - ) - rootseal._assert_role_matches_policy( - next_active, - "emergency", - candidate["final_policy"], - revoked_after, - "CONTINUITY_ADVANCE_FINAL_EMERGENCY_ROLE", - ) - except rootseal.RootTrustError as exc: - raise RootContinuityError(str(exc)) from exc - seq = len(bundle["epochs"]) + 2 - previous_head = state["rootChainHeadSha256"] - body = { - "sequence": seq, - "epoch": candidate["state"]["epoch"], - "previousChainHeadSha256": previous_head, - "governanceSnapshot": candidate["snapshot"], - "authorization": authorization, - "nextRoot": next_doc, - "rootAttestations": attestations, - "activeRoot": next_active["envelope"], - } - chain = _continuity_head(previous_head, body) - entry = dict(body, chainHeadSha256=chain) - next_bundle = json.loads(json.dumps(bundle)) - next_bundle["epochs"].append(entry) - next_state = _state( - bundle=next_bundle, - active_root=next_active, - chain_head=chain, - governance_binding=_governance_binding_from_candidate(candidate), - recovery_record_sha=bundle["recoveryEvent"]["recoveryRecordSha256"], - ) - target = _outside( - output_dir, - [prev, candidate["root"]] - + [p.resolve() for p in authorization_signature_paths] - + ([next_root_path.resolve()] if next_root_path else []) - + [p.resolve() for p in root_attestation_paths or []], - "CONTINUITY_ADVANCE_OUTPUT_INSIDE_INPUT", - ) - if target.exists() or target.is_symlink(): - _fail("CONTINUITY_ADVANCE_OUTPUT_ALREADY_EXISTS") - inputs = ( - [p for p in prev.rglob("*") if p.is_file()] - + [p for p in candidate["root"].rglob("*") if p.is_file()] - + authorization_signature_paths - + ([next_root_path] if next_root_path else []) - + list(root_attestation_paths or []) - ) - before = {str(p.resolve()): _sha(p) for p in inputs} - target.parent.mkdir(parents=True, exist_ok=True) - with tempfile.TemporaryDirectory(prefix="run157-advance-", dir=target.parent) as td: - stage = Path(td) / "continuity" - stage.mkdir() - _write(stage / "release-root-continuity-bundle.json", next_bundle) - _write(stage / _DOC_ROOT_CONTINUITY_STATE, next_state) - _write(stage / "active-root.json", next_active["envelope"]) - receipt = { - "schemaVersion": int(POLICY["receipt_schema_version"]), - "status": "recovered-root-continuity-advanced", - "governanceId": next_active["governance_id"], - "incidentId": next_bundle["recoveryEvent"]["incidentId"], - "bundle": _artifact_raw( - "release-root-continuity-bundle.json", _canonical(next_bundle) - ), - "state": _artifact_raw(_DOC_ROOT_CONTINUITY_STATE, _canonical(next_state)), - "activeRoot": _artifact_raw( - "active-root.json", _canonical(next_active["envelope"]) - ), - "attestationRootSha256": sorted(expected_attestation_root_sha256), - } - _write(stage / "release-root-continuity-receipt.json", receipt) - for p in inputs: - if _sha(p) != before[str(p.resolve())]: - _fail("CONTINUITY_ADVANCE_INPUT_CHANGED") - temp_target = target.with_name(target.name + ".tmp-" + os.urandom(6).hex()) - shutil.copytree(stage, temp_target, copy_function=shutil.copy2) - os.replace(temp_target, target) - final = verify_continuity( - output_dir=target, - expected_bootstrap_root_sha256=expected_bootstrap_root_sha256, - expected_recovery_root_sha256=expected_recovery_root_sha256, - expected_attestation_root_sha256=expected_attestation_root_sha256, - now=current, - ) - return { - "ok": True, - "phase": "recovered-root-continuity-advanced", - "governance_epoch": candidate["state"]["epoch"], - "authorizing_root_version": active["version"], - "active_root_version": next_active["version"], - "root_chain_head_sha256": final["root_chain_head_sha256"], - } - - -def _paths(values: list[str]) -> list[Path]: - return [Path(x) for x in values] - - -def main( # ruff: ignore[undocumented-public-function] - argv: list[str] | None = None, -) -> int: - p = argparse.ArgumentParser(description=__doc__) - sub = p.add_subparsers(dest="command", required=True) - a = sub.add_parser( - "activate", - help="activate a Run 156 recovered root as the trusted continuation root", - ) - a.add_argument("--sealed-dir", type=Path, required=True) - a.add_argument("--bootstrap-root-sha256", required=True) - a.add_argument("--recovery-dir", type=Path, required=True) - a.add_argument("--recovery-root-sha256", required=True) - a.add_argument("--attestation", action="append", default=[]) - a.add_argument("--attestation-trust-root", action="append", default=[]) - a.add_argument("--attestation-root-sha256", action="append", default=[]) - a.add_argument("--output-dir", type=Path, required=True) - adv = sub.add_parser( - "advance", help="seal the next governance epoch from the recovered/current root" - ) - adv.add_argument("--previous-dir", type=Path, required=True) - adv.add_argument("--governance-dir", type=Path, required=True) - adv.add_argument("--signature", action="append", default=[]) - adv.add_argument("--next-root", type=Path) - adv.add_argument("--root-attestation", action="append", default=[]) - adv.add_argument("--bootstrap-root-sha256", required=True) - adv.add_argument("--recovery-root-sha256", required=True) - adv.add_argument("--attestation-root-sha256", action="append", default=[]) - adv.add_argument("--output-dir", type=Path, required=True) - v = sub.add_parser( - "verify", help="offline-verify recovered-root continuity and later epochs" - ) - v.add_argument("--continuity-dir", type=Path, required=True) - v.add_argument("--bootstrap-root-sha256", required=True) - v.add_argument("--recovery-root-sha256", required=True) - v.add_argument("--attestation-root-sha256", action="append", default=[]) - v.add_argument("--historical", action="store_true") - args = p.parse_args(argv) - try: - if args.command == "activate": - result = activate_recovery( - sealed_dir=args.sealed_dir, - bootstrap_root_sha256=args.bootstrap_root_sha256, - recovery_dir=args.recovery_dir, - expected_recovery_root_sha256=args.recovery_root_sha256, - attestation_paths=_paths(args.attestation), - attestation_trust_root_paths=_paths(args.attestation_trust_root), - expected_attestation_root_sha256=args.attestation_root_sha256, - output_dir=args.output_dir, - ) - elif args.command == "advance": - result = advance_governance( - previous_dir=args.previous_dir, - governance_dir=args.governance_dir, - authorization_signature_paths=_paths(args.signature), - output_dir=args.output_dir, - expected_bootstrap_root_sha256=args.bootstrap_root_sha256, - expected_recovery_root_sha256=args.recovery_root_sha256, - expected_attestation_root_sha256=args.attestation_root_sha256, - next_root_path=args.next_root, - root_attestation_paths=_paths(args.root_attestation), - ) - else: - result = verify_continuity( - output_dir=args.continuity_dir, - expected_bootstrap_root_sha256=args.bootstrap_root_sha256, - expected_recovery_root_sha256=args.recovery_root_sha256, - expected_attestation_root_sha256=args.attestation_root_sha256, - historical=args.historical, - ) - except RootContinuityError as exc: - logger.error("root continuity verification failed: %s", exc) - # sys.stdout.write( - # json.dumps({"ok": False, "error": str(exc)}, sort_keys=True) + "\n" - # ) - return 2 - logger.info("root continuity operation completed: %s", result) - # sys.stdout.write(json.dumps(result, sort_keys=True) + "\n") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/security/finalize_publication.py b/security/finalize_publication.py deleted file mode 100644 index 5610b3aa2b61f7578a909fda281d36caa40e3743..0000000000000000000000000000000000000000 --- a/security/finalize_publication.py +++ /dev/null @@ -1,1672 +0,0 @@ -""" -Independently attest, sign-verify, and bind a Run 150 publication to its final release record. -""" - -from __future__ import annotations - -import argparse -import contextlib -import hashlib -import json -import logging -import os -import re -import shutil -import subprocess -import sys -import tempfile -import threading -from datetime import datetime, timedelta, timezone -from pathlib import Path -from typing import Any, Callable - -import tomllib - -logger = logging.getLogger(__name__) - -HERE = Path(__file__).resolve().parent -POLICY = tomllib.loads((HERE / "release_transparency_policy.toml").read_text()) -PUBLICATION_PREDICATE_TYPE = str(POLICY["publication_predicate_type"]) -ATTESTATION_PREDICATE_TYPE = str(POLICY["attestation_predicate_type"]) -FINAL_RECORD_PREDICATE_TYPE = str(POLICY["final_record_predicate_type"]) -RECORD_NAME = str(POLICY["record_name"]) -IN_TOTO_STATEMENT = "https://in-toto.io/Statement/v1" -_HEX64 = re.compile(r"^[0-9a-f]{64}$") -_REVISION = re.compile(r"^[0-9a-f]{40}(?:[0-9a-f]{24})?$") -_RELEASE_ID = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{2,127}$") -_ID = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:/-]{0,255}$") -_CHUNK = 1024 * 1024 - - -class TransparencyError(RuntimeError): - """Post-publication transparency processing violated a fail-closed invariant.""" - - -def _fail(code: str) -> None: - raise TransparencyError(code) - - -def _sha256(path: Path) -> str: - h = hashlib.sha256() - with path.open("rb") as handle: - for chunk in iter(lambda: handle.read(_CHUNK), b""): - h.update(chunk) - return h.hexdigest() - - -def _canonical_bytes(value: dict[str, Any]) -> bytes: - return ( - json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False) - + "\n" - ).encode("utf-8") - - -def _write_canonical(path: Path, value: dict[str, Any]) -> None: - path.write_bytes(_canonical_bytes(value)) - - -def _loads_json(raw: bytes, code: str) -> dict[str, Any]: - def object_pairs(pairs): - out: dict[str, Any] = {} - for key, value in pairs: - if key in out: - _fail(code + "_DUPLICATE_KEY") - out[key] = value - return out - - try: - value = json.loads(raw.decode("utf-8"), object_pairs_hook=object_pairs) - except TransparencyError: - raise - except Exception as exc: - raise TransparencyError(code + "_JSON_INVALID") from exc - if not isinstance(value, dict): - _fail(code + "_SCHEMA_INVALID") - return value - - -def _utc(value: datetime | None = None) -> str: - now = (value or datetime.now(timezone.utc)).astimezone(timezone.utc) - return now.isoformat(timespec="seconds").replace("+00:00", "Z") - - -def _parse_time(value: Any, code: str) -> datetime: - if not isinstance(value, str) or not value.endswith("Z"): - _fail(code) - try: - out = datetime.fromisoformat(value[:-1] + "+00:00") - except ValueError: - _fail(code) - return out.astimezone(timezone.utc) - - -def _fresh_time(value: Any, code: str, *, now: datetime) -> str: - parsed = _parse_time(value, code + "_INVALID") - skew = timedelta(minutes=int(POLICY["max_clock_skew_minutes"])) - age = timedelta(minutes=int(POLICY["max_result_age_minutes"])) - if parsed > now + skew: - _fail(code + "_FROM_FUTURE") - if now - parsed > age: - _fail(code + "_STALE") - return value - - -def _safe_name(value: Any, code: str) -> str: - if not isinstance(value, str) or not value or value in {".", ".."}: - _fail(code) - if Path(value).name != value or "/" in value or "\\" in value or "\x00" in value: - _fail(code) - if any( - ord(ch) < 32 # ruff: ignore[magic-value-comparison] - or ord(ch) == 127 # ruff: ignore[magic-value-comparison] - for ch in value - ): - _fail(code) - return value - - -def _safe_id(value: Any, code: str) -> str: - if not isinstance(value, str) or _ID.fullmatch(value) is None or ".." in value: - _fail(code) - return value - - -def _identity(value: Any, code: str) -> str: - if not isinstance(value, str): - _fail(code) - value = value.strip() - if ( - not value - or len(value) > 512 # ruff: ignore[magic-value-comparison] - or ".." in value - or "?" in value - or "#" in value - or "\x00" in value - ): - _fail(code) - if any( - ord(ch) < 32 or ord(ch) == 127 # ruff: ignore[magic-value-comparison] - for ch in value # lint - ): - _fail(code) - if re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9._:/@+-]{0,511}", value) is None: - _fail(code) - return value - - -def _bounded_text( - value: Any, code: str, *, limit: int = 1024, allow_at: bool = False -) -> str: - if not isinstance(value, str): - _fail(code) - value = value.strip() - if not value or len(value) > limit: - _fail(code) - if any( - ord(ch) < 32 or ord(ch) == 127 # ruff: ignore[magic-value-comparison] - for ch in value # lint - ): - _fail(code) - forbidden = ("?", "#") if allow_at else ("?", "#", "@") - if any(marker in value for marker in forbidden): - _fail(code) - return value - - -def _hex(value: Any, code: str) -> str: - if not isinstance(value, str) or _HEX64.fullmatch(value) is None: - _fail(code) - return value - - -def _size(value: Any, code: str) -> int: - if not isinstance(value, int) or isinstance(value, bool) or value < 0: - _fail(code) - return value - - -def _regular_file(path: Path, *, max_bytes: int, code: str) -> Path: - raw = path.expanduser() - if raw.is_symlink() or not raw.is_file(): - _fail(code) - resolved = raw.resolve() - if resolved.stat().st_size > max_bytes: - _fail(code + "_TOO_LARGE") - return resolved - - -def _regular_dir(path: Path, code: str) -> Path: - raw = path.expanduser() - if raw.is_symlink() or not raw.is_dir(): - _fail(code) - return raw.resolve() - - -def _outside(path: Path, protected: tuple[Path, ...], code: str) -> Path: - target = path.expanduser().resolve() - for item in protected: - root = item.resolve() - if target == root or root in target.parents: - _fail(code) - return target - - -def _read_json_exact( - path: Path, *, max_bytes: int, code: str, canonical: bool = True -) -> tuple[dict[str, Any], bytes]: - path = _regular_file(path, max_bytes=max_bytes, code=code) - raw = path.read_bytes() - doc = _loads_json(raw, code) - if canonical and raw != _canonical_bytes(doc): - _fail(code + "_NOT_CANONICAL") - return doc, raw - - -def _artifact(value: Any, code: str, *, remote: bool) -> dict[str, Any]: - if not isinstance(value, dict): - _fail(code + "_SCHEMA_INVALID") - expected = {"name", "sha256", "size"} - if remote: - expected |= {"locator", "immutability"} - if set(value) != expected: - _fail(code + "_SCHEMA_INVALID") - out = { - "name": _safe_name(value.get("name"), code + "_NAME_INVALID"), - "sha256": _hex(value.get("sha256"), code + "_SHA256_INVALID"), - "size": _size(value.get("size"), code + "_SIZE_INVALID"), - } - if remote: - out["locator"] = _bounded_text(value.get("locator"), code + "_LOCATOR_INVALID") - immutability = value.get("immutability") - if immutability not in set(POLICY["allowed_immutability"]): - _fail(code + "_IMMUTABILITY_INVALID") - out["immutability"] = immutability - return out - - -def _validate_publisher_result( - path: Path, - *, - operation: str, - expected: dict[str, Any], - publication_id: str, - publisher: str, - target_id: str, - locator: str, - immutability: str, - expected_sha: str, -) -> None: - doc, raw = _read_json_exact( - path, - max_bytes=int(POLICY["max_publisher_result_bytes"]), - code="PUBLICATION_PUBLISHER_RESULT", - canonical=True, - ) - if hashlib.sha256(raw).hexdigest() != expected_sha: - _fail("PUBLICATION_PUBLISHER_RESULT_HASH_MISMATCH") - keys = { - "schemaVersion", - "operation", - "publicationId", - "status", - "target", - "artifact", - "guarantees", - "verifiedAt", - } - if ( - set(doc) != keys - or doc.get("schemaVersion") != 1 - or doc.get("operation") != operation - ): - _fail("PUBLICATION_PUBLISHER_RESULT_SCHEMA_INVALID") - if doc.get("publicationId") != publication_id: - _fail("PUBLICATION_PUBLISHER_RESULT_ID_MISMATCH") - allowed_status = {"created", "present"} if operation == "publish" else {"present"} - if doc.get("status") not in allowed_status: - _fail("PUBLICATION_PUBLISHER_RESULT_STATUS_INVALID") - target = doc.get("target") - if not isinstance(target, dict) or set(target) != { - "publisher", - "targetId", - "locator", - }: - _fail("PUBLICATION_PUBLISHER_RESULT_TARGET_SCHEMA_INVALID") - if ( - target.get("publisher") != publisher - or target.get("targetId") != target_id - or target.get("locator") != locator - ): - _fail("PUBLICATION_PUBLISHER_RESULT_TARGET_MISMATCH") - if doc.get("artifact") != { - "name": expected["name"], - "sha256": expected["sha256"], - "size": expected["size"], - }: - _fail("PUBLICATION_PUBLISHER_RESULT_ARTIFACT_MISMATCH") - guarantees = doc.get("guarantees") - if not isinstance(guarantees, dict) or set(guarantees) != { - "createOnly", - "overwrite", - "remoteReadbackVerified", - "immutability", - }: - _fail("PUBLICATION_PUBLISHER_RESULT_GUARANTEES_SCHEMA_INVALID") - if guarantees != { - "createOnly": True, - "overwrite": False, - "remoteReadbackVerified": True, - "immutability": immutability, - }: - _fail("PUBLICATION_PUBLISHER_RESULT_GUARANTEES_INVALID") - _parse_time(doc.get("verifiedAt"), "PUBLICATION_PUBLISHER_RESULT_TIME_INVALID") - - -def _validate_publication( # ruff: ignore[too-many-branches] - root: Path, -) -> dict[str, Any]: - root = _regular_dir(root, "PUBLICATION_DIRECTORY_INVALID") - transparency, transparency_raw = _read_json_exact( - root / "publication-transparency.json", - max_bytes=int(POLICY["max_publication_json_bytes"]), - code="PUBLICATION_TRANSPARENCY", - canonical=True, - ) - receipt, receipt_raw = _read_json_exact( - root / "publication-receipt.json", - max_bytes=int(POLICY["max_publication_json_bytes"]), - code="PUBLICATION_RECEIPT", - canonical=True, - ) - transparency_sha = hashlib.sha256(transparency_raw).hexdigest() - receipt_sha = hashlib.sha256(receipt_raw).hexdigest() - - transparency_keys = { - "schemaVersion", - "predicateType", - "generatedAt", - "publicationId", - "release", - "subject", - "target", - "artifacts", - "verification", - } - if ( - set(transparency) != transparency_keys - or transparency.get("schemaVersion") != 1 - or transparency.get("predicateType") != PUBLICATION_PREDICATE_TYPE - ): - _fail("PUBLICATION_TRANSPARENCY_SCHEMA_INVALID") - _parse_time( - transparency.get("generatedAt"), "PUBLICATION_TRANSPARENCY_TIME_INVALID" - ) - publication_id = _safe_id( - transparency.get("publicationId"), "PUBLICATION_ID_INVALID" - ) - release = transparency.get("release") - if not isinstance(release, dict) or set(release) != {"releaseId", "sourceRevision"}: - _fail("PUBLICATION_RELEASE_SCHEMA_INVALID") - release_id = release.get("releaseId") - if not isinstance(release_id, str) or _RELEASE_ID.fullmatch(release_id) is None: - _fail("PUBLICATION_RELEASE_ID_INVALID") - revision = release.get("sourceRevision") - if not isinstance(revision, str) or _REVISION.fullmatch(revision) is None: - _fail("PUBLICATION_SOURCE_REVISION_INVALID") - subject = transparency.get("subject") - if not isinstance(subject, dict) or set(subject) != { - "promotionReceiptSha256", - "sourceTreeSha256", - "evidenceSha256", - "releaseStatementSha256", - }: - _fail("PUBLICATION_SUBJECT_SCHEMA_INVALID") - for field in ( - "promotionReceiptSha256", - "sourceTreeSha256", - "evidenceSha256", - "releaseStatementSha256", - ): - _hex(subject.get(field), "PUBLICATION_SUBJECT_HASH_INVALID") - target = transparency.get("target") - if not isinstance(target, dict) or set(target) != {"publisher", "targetId"}: - _fail("PUBLICATION_TARGET_SCHEMA_INVALID") - publisher = _safe_id(target.get("publisher"), "PUBLICATION_PUBLISHER_INVALID") - target_id = _safe_id(target.get("targetId"), "PUBLICATION_TARGET_ID_INVALID") - verification = transparency.get("verification") - if verification != { - "promotionReceiptVerified": True, - "localSnapshotVerified": True, - "createOnly": True, - "remoteReadbackVerified": True, - "allReceiptObjectsPublished": True, - }: - _fail("PUBLICATION_VERIFICATION_INVALID") - - values = transparency.get("artifacts") - if ( - not isinstance(values, list) - or not values - or len(values) > int(POLICY["max_artifacts"]) - ): - _fail("PUBLICATION_ARTIFACTS_INVALID") - artifacts: list[dict[str, Any]] = [] - locators: set[str] = set() - names: set[str] = set() - results_dir = root / "publisher-results" - if results_dir.is_symlink() or not results_dir.is_dir(): - _fail("PUBLICATION_PUBLISHER_RESULTS_DIRECTORY_INVALID") - expected_result_names: set[str] = set() - for index, value in enumerate(values, 1): - if not isinstance(value, dict) or set(value) != { - "name", - "sha256", - "size", - "remote", - "publisherEvidence", - }: - _fail("PUBLICATION_ARTIFACT_SCHEMA_INVALID") - item = { - "name": _safe_name(value.get("name"), "PUBLICATION_ARTIFACT_NAME_INVALID"), - "sha256": _hex(value.get("sha256"), "PUBLICATION_ARTIFACT_SHA256_INVALID"), - "size": _size(value.get("size"), "PUBLICATION_ARTIFACT_SIZE_INVALID"), - } - if item["name"] in names: - _fail("PUBLICATION_ARTIFACT_DUPLICATE_NAME") - names.add(item["name"]) - remote = value.get("remote") - if not isinstance(remote, dict) or set(remote) != { - "locator", - "sha256", - "size", - "immutability", - "verifiedAt", - }: - _fail("PUBLICATION_REMOTE_SCHEMA_INVALID") - locator = _bounded_text( - remote.get("locator"), "PUBLICATION_REMOTE_LOCATOR_INVALID" - ) - if locator in locators: - _fail("PUBLICATION_REMOTE_LOCATOR_COLLISION") - locators.add(locator) - immutability = remote.get("immutability") - if immutability not in set(POLICY["allowed_immutability"]): - _fail("PUBLICATION_REMOTE_IMMUTABILITY_INVALID") - if remote.get("sha256") != item["sha256"] or remote.get("size") != item["size"]: - _fail("PUBLICATION_REMOTE_SUBJECT_MISMATCH") - _parse_time(remote.get("verifiedAt"), "PUBLICATION_REMOTE_TIME_INVALID") - publisher_evidence = value.get("publisherEvidence") - if not isinstance(publisher_evidence, dict) or set(publisher_evidence) != { - "publishSha256", - "verifySha256", - }: - _fail("PUBLICATION_PUBLISHER_EVIDENCE_SCHEMA_INVALID") - publish_sha = _hex( - publisher_evidence.get("publishSha256"), - "PUBLICATION_PUBLISH_EVIDENCE_HASH_INVALID", - ) - verify_sha = _hex( - publisher_evidence.get("verifySha256"), - "PUBLICATION_VERIFY_EVIDENCE_HASH_INVALID", - ) - pub_name = f"{index:02d}-{item['name']}.publish.json" - ver_name = f"{index:02d}-{item['name']}.verify.json" - expected_result_names.update({pub_name, ver_name}) - _validate_publisher_result( - results_dir / pub_name, - operation="publish", - expected=item, - publication_id=publication_id, - publisher=publisher, - target_id=target_id, - locator=locator, - immutability=immutability, - expected_sha=publish_sha, - ) - _validate_publisher_result( - results_dir / ver_name, - operation="verify", - expected=item, - publication_id=publication_id, - publisher=publisher, - target_id=target_id, - locator=locator, - immutability=immutability, - expected_sha=verify_sha, - ) - artifacts.append({**item, "locator": locator, "immutability": immutability}) - actual_result_names = {p.name for p in results_dir.iterdir()} - if actual_result_names != expected_result_names or any( - p.is_symlink() or not p.is_file() for p in results_dir.iterdir() - ): - _fail("PUBLICATION_PUBLISHER_RESULTS_ALLOWLIST_MISMATCH") - - receipt_keys = { - "schemaVersion", - "status", - "completedAt", - "publicationId", - "releaseId", - "sourceRevision", - "promotionReceiptSha256", - "transparencySha256", - "target", - "artifacts", - } - if ( - set(receipt) != receipt_keys - or receipt.get("schemaVersion") != 1 - or receipt.get("status") != "published" - ): - _fail("PUBLICATION_RECEIPT_SCHEMA_INVALID") - _parse_time(receipt.get("completedAt"), "PUBLICATION_RECEIPT_TIME_INVALID") - if ( - receipt.get("publicationId") != publication_id - or receipt.get("releaseId") != release_id - or receipt.get("sourceRevision") != revision - ): - _fail("PUBLICATION_RECEIPT_IDENTITY_MISMATCH") - if ( - receipt.get("promotionReceiptSha256") != subject["promotionReceiptSha256"] - or receipt.get("transparencySha256") != transparency_sha - ): - _fail("PUBLICATION_RECEIPT_SUBJECT_MISMATCH") - if receipt.get("target") != {"publisher": publisher, "targetId": target_id}: - _fail("PUBLICATION_RECEIPT_TARGET_MISMATCH") - receipt_artifacts = receipt.get("artifacts") - if not isinstance(receipt_artifacts, list) or len(receipt_artifacts) != len( - artifacts - ): - _fail("PUBLICATION_RECEIPT_ARTIFACTS_INVALID") - normalized_receipt = [ - _artifact(item, "PUBLICATION_RECEIPT_ARTIFACT", remote=True) - for item in receipt_artifacts - ] - if normalized_receipt != artifacts: - _fail("PUBLICATION_RECEIPT_ARTIFACT_MISMATCH") - - root_entries = {p.name for p in root.iterdir()} - if root_entries != { - "publication-transparency.json", - "publication-receipt.json", - "publisher-results", - }: - _fail("PUBLICATION_DIRECTORY_ALLOWLIST_MISMATCH") - return { - "root": root, - "transparency": transparency, - "receipt": receipt, - "transparency_sha256": transparency_sha, - "receipt_sha256": receipt_sha, - "publication_id": publication_id, - "release_id": release_id, - "source_revision": revision, - "publisher": publisher, - "target_id": target_id, - "promotion_receipt_sha256": subject["promotionReceiptSha256"], - "artifacts": artifacts, - } - - -def _verification_id(transparency_sha: str, verifier_identity: str) -> str: - h = hashlib.sha256() - h.update(b"release-independent-verification-v1\0") - for value in (transparency_sha, verifier_identity): - data = value.encode("utf-8") - h.update(len(data).to_bytes(4, "big")) - h.update(data) - return "verify-" + h.hexdigest()[:32] - - -def _binding_id( - publication_id: str, record_sha: str, publisher: str, target_id: str -) -> str: - h = hashlib.sha256() - h.update(b"release-final-record-binding-v1\0") - for value in (publication_id, record_sha, publisher, target_id): - data = value.encode("utf-8") - h.update(len(data).to_bytes(4, "big")) - h.update(data) - return "bind-" + h.hexdigest()[:32] - - -Verifier = Callable[[dict[str, Any]], dict[str, Any]] -Binder = Callable[[dict[str, Any]], dict[str, Any]] - - -def _command_adapter( - command: list[str], *, prefix: str -) -> Callable[[dict[str, Any]], dict[str, Any]]: - if not command or any(not isinstance(part, str) or not part for part in command): - _fail(prefix + "_COMMAND_INVALID") - executable = ( - shutil.which(command[0]) if not os.path.isabs(command[0]) else command[0] - ) - if not executable or not Path(executable).is_file(): - _fail(prefix + "_COMMAND_UNAVAILABLE") - argv = [executable, *command[1:]] - - def invoke(request: dict[str, Any]) -> dict[str, Any]: - payload = _canonical_bytes(request) - limit = int(POLICY["max_adapter_output_bytes"]) - proc = subprocess.Popen( # ruff: ignore[subprocess-without-shell-equals-true] - argv, - stdin=subprocess.PIPE, - stdout=subprocess.PIPE, - stderr=subprocess.DEVNULL, - env={"PATH": os.defpath, "LC_ALL": "C", "LANG": "C"}, - ) - assert proc.stdin is not None # ruff: ignore[assert] - assert proc.stdout is not None # ruff: ignore[assert] - chunks: list[bytes] = [] - state: dict[str, Any] = {"size": 0, "overflow": False, "error": None} - - def drain() -> None: - try: - while True: - chunk = proc.stdout.read(65536) - if not chunk: - return - state["size"] += len(chunk) - if state["size"] > limit: - state["overflow"] = True - with contextlib.suppress(OSError): - proc.kill() - return - chunks.append(chunk) - # defensive pipe failure - except ( - BaseException # pragma: no cover # ruff: ignore[blind-except] - ) as exc: - state["error"] = exc - try: # ruff: ignore[suppressible-exception] - proc.kill() - except OSError: - pass - - reader = threading.Thread( - target=drain, name=prefix.lower() + "-stdout", daemon=True - ) - reader.start() - try: - try: - proc.stdin.write(payload) - proc.stdin.close() - proc.wait(timeout=int(POLICY["adapter_timeout_seconds"])) - except subprocess.TimeoutExpired as exc: - proc.kill() - proc.wait() - reader.join(timeout=2) - raise TransparencyError(prefix + "_COMMAND_TIMEOUT") from exc - finally: - try: # ruff: ignore[suppressible-exception] - proc.stdin.close() - except OSError: - pass - reader.join(timeout=2) - if reader.is_alive() or state["error"] is not None: - proc.kill() - _fail(prefix + "_COMMAND_OUTPUT_DRAIN_FAILED") - if state["overflow"]: - _fail(prefix + "_COMMAND_OUTPUT_TOO_LARGE") - if proc.returncode != 0: - _fail(prefix + "_COMMAND_FAILED") - return _loads_json(b"".join(chunks), prefix + "_COMMAND_OUTPUT") - finally: - if proc.poll() is None: - proc.kill() - proc.wait() - if reader.is_alive(): - reader.join(timeout=2) - try: # ruff: ignore[suppressible-exception] - proc.stdout.close() - except OSError: - pass - - return invoke - - -def command_verifier( # ruff: ignore[undocumented-public-function] - command: list[str], -) -> Verifier: - return _command_adapter(command, prefix="INDEPENDENT_VERIFIER") - - -def command_binder( # ruff: ignore[undocumented-public-function] - command: list[str], -) -> Binder: - return _command_adapter(command, prefix="RELEASE_BINDER") - - -def _validate_verifier_response( - value: Any, - *, - item: dict[str, Any], - info: dict[str, Any], - verifier_identity: str, - verification_id: str, - now: datetime, -) -> dict[str, Any]: - keys = { - "schemaVersion", - "operation", - "verificationId", - "publicationId", - "status", - "verifier", - "target", - "artifact", - "remote", - "verifiedAt", - } - if not isinstance(value, dict) or set(value) != keys: - _fail("INDEPENDENT_VERIFIER_RESPONSE_SCHEMA_INVALID") - if ( - value.get("schemaVersion") != int(POLICY["verifier_protocol_version"]) - or value.get("operation") != "verify" - ): - _fail("INDEPENDENT_VERIFIER_RESPONSE_VERSION_INVALID") - if ( - value.get("verificationId") != verification_id - or value.get("publicationId") != info["publication_id"] - or value.get("status") != "present" - ): - _fail("INDEPENDENT_VERIFIER_RESPONSE_IDENTITY_MISMATCH") - verifier = value.get("verifier") - if not isinstance(verifier, dict) or set(verifier) != { - "identity", - "readOnly", - "publisherCredentialsReused", - }: - _fail("INDEPENDENT_VERIFIER_AUTHORITY_SCHEMA_INVALID") - if ( - _identity(verifier.get("identity"), "INDEPENDENT_VERIFIER_IDENTITY_INVALID") - != verifier_identity - ): - _fail("INDEPENDENT_VERIFIER_IDENTITY_MISMATCH") - if ( - verifier.get("readOnly") is not True - or verifier.get("publisherCredentialsReused") is not False - ): - _fail("INDEPENDENT_VERIFIER_AUTHORITY_INVALID") - target = value.get("target") - if target != {"publisher": info["publisher"], "targetId": info["target_id"]}: - _fail("INDEPENDENT_VERIFIER_TARGET_MISMATCH") - if value.get("artifact") != { - "name": item["name"], - "sha256": item["sha256"], - "size": item["size"], - }: - _fail("INDEPENDENT_VERIFIER_ARTIFACT_MISMATCH") - remote = value.get("remote") - expected_remote = { - "locator": item["locator"], - "sha256": item["sha256"], - "size": item["size"], - "immutability": item["immutability"], - } - if remote != expected_remote: - _fail("INDEPENDENT_VERIFIER_REMOTE_MISMATCH") - verified_at = _fresh_time( - value.get("verifiedAt"), "INDEPENDENT_VERIFIER_TIME", now=now - ) - return {"verifiedAt": verified_at, "remote": expected_remote} - - -def _verify_remote_set( - *, - info: dict[str, Any], - verifier_identity: str, - adapter: Verifier, - output_dir: Path, - now: datetime, -) -> tuple[str, list[dict[str, Any]]]: - verification_id = _verification_id(info["transparency_sha256"], verifier_identity) - output_dir.mkdir(parents=True, exist_ok=False) - out: list[dict[str, Any]] = [] - for index, item in enumerate(info["artifacts"], 1): - request = { - "schemaVersion": int(POLICY["verifier_protocol_version"]), - "operation": "verify", - "verificationId": verification_id, - "publicationId": info["publication_id"], - "releaseId": info["release_id"], - "publicationTransparencySha256": info["transparency_sha256"], - "verifier": { - "identity": verifier_identity, - "requiredAuthority": { - "readOnly": True, - "publisherCredentialsReused": False, - }, - }, - "target": {"publisher": info["publisher"], "targetId": info["target_id"]}, - "artifact": { - "name": item["name"], - "sha256": item["sha256"], - "size": item["size"], - "locator": item["locator"], - "immutability": item["immutability"], - }, - } - raw = adapter(request) - normalized = _validate_verifier_response( - raw, - item=item, - info=info, - verifier_identity=verifier_identity, - verification_id=verification_id, - now=now, - ) - path = output_dir / f"{index:02d}-{item['name']}.verify.json" - _write_canonical(path, raw) - out.append( - { - "name": item["name"], - "sha256": item["sha256"], - "size": item["size"], - "locator": item["locator"], - "immutability": item["immutability"], - "verifiedAt": normalized["verifiedAt"], - "verifierEvidenceSha256": _sha256(path), - } - ) - return verification_id, out - - -def prepare_attestation( # ruff: ignore[undocumented-public-function] - *, - publication_dir: Path, - output_dir: Path, - verifier_identity: str, - verifier: Verifier, - now: datetime | None = None, -) -> dict[str, Any]: - publication_dir = _regular_dir(publication_dir, "PUBLICATION_DIRECTORY_INVALID") - target = _outside( - output_dir, (publication_dir,), "ATTESTATION_OUTPUT_INSIDE_PUBLICATION" - ) - if target.exists() or target.is_symlink(): - _fail("ATTESTATION_OUTPUT_ALREADY_EXISTS") - verifier_identity = _identity( - verifier_identity, "INDEPENDENT_VERIFIER_IDENTITY_INVALID" - ) - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - info = _validate_publication(publication_dir) - if verifier_identity == info["publisher"]: - _fail("INDEPENDENT_VERIFIER_NOT_DISTINCT") - - target.parent.mkdir(parents=True, exist_ok=True) - with tempfile.TemporaryDirectory( - prefix="release-post-publication-", dir=target.parent - ) as temp_raw: - stage = Path(temp_raw) / "prepared" - stage.mkdir() - verification_id, results = _verify_remote_set( - info=info, - verifier_identity=verifier_identity, - adapter=verifier, - output_dir=stage / "independent-verifier-results", - now=current, - ) - after = _validate_publication(publication_dir) - if ( - after["transparency_sha256"] != info["transparency_sha256"] - or after["receipt_sha256"] != info["receipt_sha256"] - or after["artifacts"] != info["artifacts"] - ): - _fail("PUBLICATION_CHANGED_DURING_ATTESTATION") - attestation = { - "_type": IN_TOTO_STATEMENT, - "subject": [ - { - "name": "publication-transparency.json", - "digest": {"sha256": info["transparency_sha256"]}, - } - ], - "predicateType": ATTESTATION_PREDICATE_TYPE, - "predicate": { - "schemaVersion": int(POLICY["attestation_schema_version"]), - "generatedAt": _utc(current), - "publication": { - "publicationId": info["publication_id"], - "publicationReceiptSha256": info["receipt_sha256"], - "promotionReceiptSha256": info["promotion_receipt_sha256"], - }, - "release": { - "releaseId": info["release_id"], - "sourceRevision": info["source_revision"], - }, - "target": { - "publisher": info["publisher"], - "targetId": info["target_id"], - }, - "independentVerifier": { - "identity": verifier_identity, - "verificationId": verification_id, - "readOnly": True, - "publisherCredentialsReused": False, - }, - "artifacts": results, - "verification": { - "publicationEvidenceVerified": True, - "publisherEvidenceRebound": True, - "independentRemoteReadbackVerified": True, - "allPublishedObjectsPresent": True, - }, - }, - } - attestation_path = stage / "publication-attestation.json" - _write_canonical(attestation_path, attestation) - tmp_target = target.with_name(target.name + ".tmp-" + os.urandom(6).hex()) - shutil.copytree(stage, tmp_target, copy_function=shutil.copy2) - os.replace(tmp_target, target) - return { - "ok": True, - "phase": "attestation-prepared", - "publication_id": info["publication_id"], - "publication_transparency_sha256": info["transparency_sha256"], - "attestation_sha256": _sha256(target / "publication-attestation.json"), - "verification_id": verification_id, - "artifact_count": len(info["artifacts"]), - } - - -def _validate_attestation( # ruff: ignore[too-many-branches] - path: Path, - *, - info: dict[str, Any], - prepared_dir: Path | None = None, -) -> dict[str, Any]: - doc, raw = _read_json_exact( - path, - max_bytes=int(POLICY["max_publication_json_bytes"]), - code="POST_PUBLICATION_ATTESTATION", - canonical=True, - ) - if ( - set(doc) != {"_type", "subject", "predicateType", "predicate"} - or doc.get("_type") != IN_TOTO_STATEMENT - or doc.get("predicateType") != ATTESTATION_PREDICATE_TYPE - ): - _fail("POST_PUBLICATION_ATTESTATION_SCHEMA_INVALID") - if doc.get("subject") != [ - { - "name": "publication-transparency.json", - "digest": {"sha256": info["transparency_sha256"]}, - } - ]: - _fail("POST_PUBLICATION_ATTESTATION_SUBJECT_MISMATCH") - predicate = doc.get("predicate") - if not isinstance(predicate, dict) or set(predicate) != { - "schemaVersion", - "generatedAt", - "publication", - "release", - "target", - "independentVerifier", - "artifacts", - "verification", - }: - _fail("POST_PUBLICATION_ATTESTATION_PREDICATE_SCHEMA_INVALID") - if predicate.get("schemaVersion") != int(POLICY["attestation_schema_version"]): - _fail("POST_PUBLICATION_ATTESTATION_VERSION_INVALID") - generated = _parse_time( - predicate.get("generatedAt"), "POST_PUBLICATION_ATTESTATION_TIME_INVALID" - ) - publication = predicate.get("publication") - expected_publication = { - "publicationId": info["publication_id"], - "publicationReceiptSha256": info["receipt_sha256"], - "promotionReceiptSha256": info["promotion_receipt_sha256"], - } - if publication != expected_publication: - _fail("POST_PUBLICATION_ATTESTATION_PUBLICATION_MISMATCH") - if predicate.get("release") != { - "releaseId": info["release_id"], - "sourceRevision": info["source_revision"], - }: - _fail("POST_PUBLICATION_ATTESTATION_RELEASE_MISMATCH") - if predicate.get("target") != { - "publisher": info["publisher"], - "targetId": info["target_id"], - }: - _fail("POST_PUBLICATION_ATTESTATION_TARGET_MISMATCH") - verifier = predicate.get("independentVerifier") - if not isinstance(verifier, dict) or set(verifier) != { - "identity", - "verificationId", - "readOnly", - "publisherCredentialsReused", - }: - _fail("POST_PUBLICATION_ATTESTATION_VERIFIER_SCHEMA_INVALID") - verifier_identity = _identity( - verifier.get("identity"), - "POST_PUBLICATION_ATTESTATION_VERIFIER_IDENTITY_INVALID", - ) - if ( - verifier_identity == info["publisher"] - or verifier.get("readOnly") is not True - or verifier.get("publisherCredentialsReused") is not False - ): - _fail("POST_PUBLICATION_ATTESTATION_VERIFIER_AUTHORITY_INVALID") - expected_verification_id = _verification_id( - info["transparency_sha256"], verifier_identity - ) - if verifier.get("verificationId") != expected_verification_id: - _fail("POST_PUBLICATION_ATTESTATION_VERIFICATION_ID_MISMATCH") - if predicate.get("verification") != { - "publicationEvidenceVerified": True, - "publisherEvidenceRebound": True, - "independentRemoteReadbackVerified": True, - "allPublishedObjectsPresent": True, - }: - _fail("POST_PUBLICATION_ATTESTATION_VERIFICATION_INVALID") - values = predicate.get("artifacts") - if not isinstance(values, list) or len(values) != len(info["artifacts"]): - _fail("POST_PUBLICATION_ATTESTATION_ARTIFACTS_INVALID") - normalized: list[dict[str, Any]] = [] - for index, (value, expected) in enumerate(zip(values, info["artifacts"]), 1): - keys = { - "name", - "sha256", - "size", - "locator", - "immutability", - "verifiedAt", - "verifierEvidenceSha256", - } - if not isinstance(value, dict) or set(value) != keys: - _fail("POST_PUBLICATION_ATTESTATION_ARTIFACT_SCHEMA_INVALID") - invariant = { - key: value[key] - for key in ("name", "sha256", "size", "locator", "immutability") - } - if invariant != expected: - _fail("POST_PUBLICATION_ATTESTATION_ARTIFACT_MISMATCH") - _parse_time( - value.get("verifiedAt"), - "POST_PUBLICATION_ATTESTATION_ARTIFACT_TIME_INVALID", - ) - evidence_sha = _hex( - value.get("verifierEvidenceSha256"), - "POST_PUBLICATION_ATTESTATION_EVIDENCE_HASH_INVALID", - ) - if prepared_dir is not None: - evidence_path = ( - prepared_dir - / "independent-verifier-results" - / f"{index:02d}-{expected['name']}.verify.json" - ) - if ( - _sha256( - _regular_file( - evidence_path, - max_bytes=int(POLICY["max_adapter_output_bytes"]), - code="POST_PUBLICATION_VERIFIER_EVIDENCE_INVALID", - ) - ) - != evidence_sha - ): - _fail("POST_PUBLICATION_ATTESTATION_EVIDENCE_HASH_MISMATCH") - normalized.append(dict(value)) - return { - "doc": doc, - "raw": raw, - "sha256": hashlib.sha256(raw).hexdigest(), - "generated": generated, - "verifier_identity": verifier_identity, - "verification_id": expected_verification_id, - "artifacts": normalized, - } - - -def write_signature_verification_record( # ruff: ignore[undocumented-public-function] - *, - publication_dir: Path, - attestation: Path, - verifier_evidence: Path, - output: Path, - signer_identity: str, - verifier_name: str, - verifier_version: str, - verified_at: datetime | None = None, -) -> Path: - info = _validate_publication(publication_dir) - att = _validate_attestation(attestation, info=info) - verifier_evidence = _regular_file( - verifier_evidence, - max_bytes=int(POLICY["max_external_verifier_evidence_bytes"]), - code="SIGNATURE_VERIFIER_EVIDENCE_INVALID", - ) - signer_identity = _identity(signer_identity, "SIGNER_IDENTITY_INVALID") - for value, code in ( - (verifier_name, "SIGNATURE_VERIFIER_NAME_INVALID"), - (verifier_version, "SIGNATURE_VERIFIER_VERSION_INVALID"), - ): - _bounded_text(value, code, limit=128, allow_at=True) - target = output.expanduser().resolve() - target.parent.mkdir(parents=True, exist_ok=True) - if target.exists() or target.is_symlink(): - _fail("SIGNATURE_RECORD_OUTPUT_EXISTS") - current = (verified_at or datetime.now(timezone.utc)).astimezone(timezone.utc) - skew = timedelta(minutes=int(POLICY["max_clock_skew_minutes"])) - if att["generated"] > current + skew: - _fail("POST_PUBLICATION_ATTESTATION_FROM_FUTURE") - if current - att["generated"] > timedelta( - hours=int(POLICY["max_attestation_age_hours"]) - ): - _fail("POST_PUBLICATION_ATTESTATION_STALE") - if current + skew < att["generated"]: - _fail("SIGNATURE_RECORD_PREDATES_ATTESTATION") - payload = { - "schemaVersion": int(POLICY["signature_record_schema_version"]), - "verified": True, - "verifiedAt": _utc(current), - "publicationId": info["publication_id"], - "publicationTransparencySha256": info["transparency_sha256"], - "attestationSha256": att["sha256"], - "signerIdentity": signer_identity, - "signerIdentityVerified": True, - "verifierEvidenceSha256": _sha256(verifier_evidence), - "verifier": { - "name": verifier_name.strip(), - "version": verifier_version.strip(), - }, - } - _write_canonical(target, payload) - return target - - -def _validate_signature_record( - path: Path, - *, - info: dict[str, Any], - att: dict[str, Any], - verifier_evidence: Path, - expected_signer_identity: str, - now: datetime, -) -> dict[str, Any]: - doc, _ = _read_json_exact( - path, - max_bytes=int(POLICY["max_publication_json_bytes"]), - code="SIGNATURE_RECORD", - canonical=True, - ) - keys = { - "schemaVersion", - "verified", - "verifiedAt", - "publicationId", - "publicationTransparencySha256", - "attestationSha256", - "signerIdentity", - "signerIdentityVerified", - "verifierEvidenceSha256", - "verifier", - } - if set(doc) != keys or doc.get("schemaVersion") != int( - POLICY["signature_record_schema_version"] - ): - _fail("SIGNATURE_RECORD_SCHEMA_INVALID") - if doc.get("verified") is not True or doc.get("signerIdentityVerified") is not True: - _fail("SIGNATURE_RECORD_NOT_VERIFIED") - if ( - doc.get("publicationId") != info["publication_id"] - or doc.get("publicationTransparencySha256") != info["transparency_sha256"] - or doc.get("attestationSha256") != att["sha256"] - ): - _fail("SIGNATURE_RECORD_SUBJECT_MISMATCH") - identity = _identity( - doc.get("signerIdentity"), "SIGNATURE_RECORD_SIGNER_IDENTITY_INVALID" - ) - if identity != expected_signer_identity: - _fail("SIGNATURE_RECORD_SIGNER_IDENTITY_MISMATCH") - evidence = _regular_file( - verifier_evidence, - max_bytes=int(POLICY["max_external_verifier_evidence_bytes"]), - code="SIGNATURE_VERIFIER_EVIDENCE_INVALID", - ) - if doc.get("verifierEvidenceSha256") != _sha256(evidence): - _fail("SIGNATURE_VERIFIER_EVIDENCE_HASH_MISMATCH") - verifier = doc.get("verifier") - if not isinstance(verifier, dict) or set(verifier) != {"name", "version"}: - _fail("SIGNATURE_RECORD_VERIFIER_SCHEMA_INVALID") - for field in ("name", "version"): - _bounded_text( - verifier.get(field), - "SIGNATURE_RECORD_VERIFIER_FIELD_INVALID", - limit=128, - allow_at=True, - ) - verified = _parse_time(doc.get("verifiedAt"), "SIGNATURE_RECORD_TIME_INVALID") - skew = timedelta(minutes=int(POLICY["max_clock_skew_minutes"])) - if att["generated"] > now + skew: - _fail("POST_PUBLICATION_ATTESTATION_FROM_FUTURE") - if now - att["generated"] > timedelta( - hours=int(POLICY["max_attestation_age_hours"]) - ): - _fail("POST_PUBLICATION_ATTESTATION_STALE") - if verified > now + skew: - _fail("SIGNATURE_RECORD_FROM_FUTURE") - if now - verified > timedelta(hours=int(POLICY["max_attestation_age_hours"])): - _fail("SIGNATURE_RECORD_STALE") - if verified + skew < att["generated"]: - _fail("SIGNATURE_RECORD_PREDATES_ATTESTATION") - return {"doc": doc, "sha256": _sha256(path), "verified": verified} - - -def _validate_prepared_dir( - prepared_dir: Path, *, info: dict[str, Any] -) -> dict[str, Any]: - prepared_dir = _regular_dir(prepared_dir, "PREPARED_ATTESTATION_DIRECTORY_INVALID") - att = _validate_attestation( - prepared_dir / "publication-attestation.json", - info=info, - prepared_dir=prepared_dir, - ) - results = prepared_dir / "independent-verifier-results" - if results.is_symlink() or not results.is_dir(): - _fail("PREPARED_VERIFIER_RESULTS_DIRECTORY_INVALID") - expected = { - f"{index:02d}-{item['name']}.verify.json" - for index, item in enumerate(info["artifacts"], 1) - } - actual = {p.name for p in results.iterdir()} - if actual != expected or any( - p.is_symlink() or not p.is_file() for p in results.iterdir() - ): - _fail("PREPARED_VERIFIER_RESULTS_ALLOWLIST_MISMATCH") - if {p.name for p in prepared_dir.iterdir()} != { - "publication-attestation.json", - "independent-verifier-results", - }: - _fail("PREPARED_ATTESTATION_DIRECTORY_ALLOWLIST_MISMATCH") - return att - - -def _normalized_final_artifacts(results: list[dict[str, Any]]) -> list[dict[str, Any]]: - return [ - { - key: item[key] - for key in ("name", "sha256", "size", "locator", "immutability") - } - for item in results - ] - - -def _validate_binder_response( - value: Any, - *, - operation: str, - info: dict[str, Any], - binding_id: str, - record_item: dict[str, Any], - now: datetime, -) -> dict[str, Any]: - keys = { - "schemaVersion", - "operation", - "bindingId", - "publicationId", - "status", - "target", - "record", - "guarantees", - "verifiedAt", - } - if not isinstance(value, dict) or set(value) != keys: - _fail("RELEASE_BINDER_RESPONSE_SCHEMA_INVALID") - if ( - value.get("schemaVersion") != int(POLICY["binder_protocol_version"]) - or value.get("operation") != operation - ): - _fail("RELEASE_BINDER_RESPONSE_VERSION_INVALID") - if ( - value.get("bindingId") != binding_id - or value.get("publicationId") != info["publication_id"] - ): - _fail("RELEASE_BINDER_RESPONSE_ID_MISMATCH") - allowed_status = {"created", "present"} if operation == "bind" else {"present"} - if value.get("status") not in allowed_status: - _fail("RELEASE_BINDER_RESPONSE_STATUS_INVALID") - target = value.get("target") - if not isinstance(target, dict) or set(target) != { - "publisher", - "targetId", - "locator", - }: - _fail("RELEASE_BINDER_TARGET_SCHEMA_INVALID") - if ( - target.get("publisher") != info["publisher"] - or target.get("targetId") != info["target_id"] - ): - _fail("RELEASE_BINDER_TARGET_MISMATCH") - locator = _bounded_text(target.get("locator"), "RELEASE_BINDER_LOCATOR_INVALID") - if value.get("record") != record_item: - _fail("RELEASE_BINDER_RECORD_MISMATCH") - guarantees = value.get("guarantees") - if not isinstance(guarantees, dict) or set(guarantees) != { - "createOnly", - "overwrite", - "remoteReadbackVerified", - "immutability", - "bindingType", - }: - _fail("RELEASE_BINDER_GUARANTEES_SCHEMA_INVALID") - if ( - guarantees.get("createOnly") is not True - or guarantees.get("overwrite") is not False - or guarantees.get("remoteReadbackVerified") is not True - ): - _fail("RELEASE_BINDER_AUTHORITY_INVALID") - immutability = guarantees.get("immutability") - if immutability not in set(POLICY["allowed_immutability"]): - _fail("RELEASE_BINDER_IMMUTABILITY_INVALID") - binding_type = guarantees.get("bindingType") - if binding_type not in set(POLICY["allowed_binding_types"]): - _fail("RELEASE_BINDER_TYPE_INVALID") - verified_at = _fresh_time(value.get("verifiedAt"), "RELEASE_BINDER_TIME", now=now) - return { - "locator": locator, - "immutability": immutability, - "bindingType": binding_type, - "verifiedAt": verified_at, - "status": value["status"], - } - - -def finalize_publication( # ruff: ignore[undocumented-public-function] - *, - publication_dir: Path, - prepared_dir: Path, - signature_record: Path, - signature_verifier_evidence: Path, - output_dir: Path, - expected_signer_identity: str, - verifier_identity: str, - verifier: Verifier, - binder: Binder, - now: datetime | None = None, -) -> dict[str, Any]: - publication_dir = _regular_dir(publication_dir, "PUBLICATION_DIRECTORY_INVALID") - prepared_dir = _regular_dir(prepared_dir, "PREPARED_ATTESTATION_DIRECTORY_INVALID") - target = _outside( - output_dir, (publication_dir, prepared_dir), "FINALIZATION_OUTPUT_INSIDE_INPUT" - ) - if target.exists() or target.is_symlink(): - _fail("FINALIZATION_OUTPUT_ALREADY_EXISTS") - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - expected_signer_identity = _identity( - expected_signer_identity, "EXPECTED_SIGNER_IDENTITY_INVALID" - ) - verifier_identity = _identity( - verifier_identity, "INDEPENDENT_VERIFIER_IDENTITY_INVALID" - ) - info = _validate_publication(publication_dir) - att = _validate_prepared_dir(prepared_dir, info=info) - if verifier_identity != att["verifier_identity"]: - _fail("FINALIZATION_VERIFIER_IDENTITY_MISMATCH") - signature_record = _regular_file( - signature_record, - max_bytes=int(POLICY["max_publication_json_bytes"]), - code="SIGNATURE_RECORD_INVALID", - ) - sig = _validate_signature_record( - signature_record, - info=info, - att=att, - verifier_evidence=signature_verifier_evidence, - expected_signer_identity=expected_signer_identity, - now=current, - ) - - target.parent.mkdir(parents=True, exist_ok=True) - with tempfile.TemporaryDirectory( - prefix="release-final-record-", dir=target.parent - ) as temp_raw: - temp = Path(temp_raw) - stage = temp / "finalized" - stage.mkdir() - verification_id, final_results = _verify_remote_set( - info=info, - verifier_identity=verifier_identity, - adapter=verifier, - output_dir=stage / "final-verifier-results", - now=current, - ) - if verification_id != att["verification_id"]: - _fail("FINALIZATION_VERIFICATION_ID_MISMATCH") - skew = timedelta(minutes=int(POLICY["max_clock_skew_minutes"])) - for result in final_results: - verified_at = _parse_time( - result["verifiedAt"], "FINALIZATION_VERIFIER_TIME_INVALID" - ) - if verified_at + skew < sig["verified"]: - _fail("FINALIZATION_VERIFICATION_PREDATES_SIGNATURE") - normalized_final = _normalized_final_artifacts(final_results) - if normalized_final != info["artifacts"]: - _fail("FINALIZATION_REMOTE_SET_MISMATCH") - - final_record = { - "schemaVersion": int(POLICY["final_record_schema_version"]), - "predicateType": FINAL_RECORD_PREDICATE_TYPE, - "status": "finalized", - "publicationId": info["publication_id"], - "release": { - "releaseId": info["release_id"], - "sourceRevision": info["source_revision"], - }, - "subject": { - "promotionReceiptSha256": info["promotion_receipt_sha256"], - "publicationReceiptSha256": info["receipt_sha256"], - "publicationTransparencySha256": info["transparency_sha256"], - "postPublicationAttestationSha256": att["sha256"], - "signatureVerificationSha256": sig["sha256"], - "signatureVerifierEvidenceSha256": sig["doc"]["verifierEvidenceSha256"], - }, - "target": {"publisher": info["publisher"], "targetId": info["target_id"]}, - "signer": { - "identity": expected_signer_identity, - "verifiedAt": sig["doc"]["verifiedAt"], - }, - "independentVerifier": { - "identity": verifier_identity, - "verificationId": verification_id, - "readOnly": True, - "publisherCredentialsReused": False, - }, - "artifacts": normalized_final, - "verification": { - "publicationEvidenceVerified": True, - "signedAttestationVerified": True, - "signerIdentityVerified": True, - "independentRemoteReadbackVerifiedAfterSignature": True, - "allPublishedObjectsPresent": True, - }, - } - record_path = stage / RECORD_NAME - _write_canonical(record_path, final_record) - os.chmod(record_path, 0o444) - record_item = { - "name": RECORD_NAME, - "sha256": _sha256(record_path), - "size": record_path.stat().st_size, - } - binding_id = _binding_id( - info["publication_id"], - record_item["sha256"], - info["publisher"], - info["target_id"], - ) - binding_results = stage / "binding-results" - binding_results.mkdir() - base_request = { - "schemaVersion": int(POLICY["binder_protocol_version"]), - "bindingId": binding_id, - "publicationId": info["publication_id"], - "releaseId": info["release_id"], - "publicationTransparencySha256": info["transparency_sha256"], - "postPublicationAttestationSha256": att["sha256"], - "target": {"publisher": info["publisher"], "targetId": info["target_id"]}, - "record": record_item, - } - bind_request = dict(base_request) - bind_request["operation"] = "bind" - bind_request["record"] = dict(record_item, localPath=str(record_path)) - before = _sha256(record_path) - bind_raw = binder(bind_request) - if _sha256(record_path) != before or before != record_item["sha256"]: - _fail("FINAL_RECORD_CHANGED_DURING_BIND") - bind_result = _validate_binder_response( - bind_raw, - operation="bind", - info=info, - binding_id=binding_id, - record_item=record_item, - now=current, - ) - verify_request = dict(base_request) - verify_request["operation"] = "verify" - verify_raw = binder(verify_request) - verify_result = _validate_binder_response( - verify_raw, - operation="verify", - info=info, - binding_id=binding_id, - record_item=record_item, - now=current, - ) - if any( - bind_result[field] != verify_result[field] - for field in ("locator", "immutability", "bindingType") - ): - _fail("RELEASE_BINDER_VERIFY_REBIND_FAILED") - if verify_result["locator"] in {item["locator"] for item in info["artifacts"]}: - _fail("RELEASE_BINDER_LOCATOR_COLLISION") - bind_path = binding_results / (RECORD_NAME + ".bind.json") - verify_path = binding_results / (RECORD_NAME + ".verify.json") - _write_canonical(bind_path, bind_raw) - _write_canonical(verify_path, verify_raw) - - # Rebind immutable inputs after all remote calls. Final record bytes do not include - # fresh verifier timestamps, so an interrupted create-only binding is reproducible. - info_after = _validate_publication(publication_dir) - att_after = _validate_prepared_dir(prepared_dir, info=info_after) - if ( - info_after["transparency_sha256"] != info["transparency_sha256"] - or info_after["receipt_sha256"] != info["receipt_sha256"] - or att_after["sha256"] != att["sha256"] - ): - _fail("FINALIZATION_INPUT_CHANGED_DURING_BIND") - if _sha256(signature_record) != sig["sha256"]: - _fail("SIGNATURE_RECORD_CHANGED_DURING_BIND") - - shutil.copy2( - publication_dir / "publication-transparency.json", - stage / "publication-transparency.json", - ) - shutil.copy2( - publication_dir / "publication-receipt.json", - stage / "publication-receipt.json", - ) - shutil.copy2( - prepared_dir / "publication-attestation.json", - stage / "publication-attestation.json", - ) - shutil.copy2( - signature_record, - stage / "publication-attestation.signature-verification.json", - ) - binding_receipt = { - "schemaVersion": int(POLICY["binding_receipt_schema_version"]), - "status": "bound", - "publicationId": info["publication_id"], - "releaseId": info["release_id"], - "publicationTransparencySha256": info["transparency_sha256"], - "postPublicationAttestationSha256": att["sha256"], - "signatureVerificationSha256": sig["sha256"], - "finalRecord": record_item, - "binding": { - "bindingId": binding_id, - "locator": verify_result["locator"], - "immutability": verify_result["immutability"], - "bindingType": verify_result["bindingType"], - "verifiedAt": verify_result["verifiedAt"], - "bindEvidenceSha256": _sha256(bind_path), - "verifyEvidenceSha256": _sha256(verify_path), - }, - "finalVerificationEvidence": [ - {"name": result["name"], "sha256": result["verifierEvidenceSha256"]} - for result in final_results - ], - } - receipt_path = stage / "release-publication-binding-receipt.json" - _write_canonical(receipt_path, binding_receipt) - tmp_target = target.with_name(target.name + ".tmp-" + os.urandom(6).hex()) - shutil.copytree(stage, tmp_target, copy_function=shutil.copy2) - os.replace(tmp_target, target) - return { - "ok": True, - "phase": "bound", - "publication_id": info["publication_id"], - "release_id": info["release_id"], - "attestation_sha256": att["sha256"], - "final_record_sha256": _sha256(target / RECORD_NAME), - "binding_receipt_sha256": _sha256( - target / "release-publication-binding-receipt.json" - ), - "binding_id": binding_id, - "artifact_count": len(info["artifacts"]), - } - - -def main( # ruff: ignore[undocumented-public-function] - argv: list[str] | None = None, -) -> int: - parser = argparse.ArgumentParser(description=__doc__) - sub = parser.add_subparsers(dest="command", required=True) - - prepare = sub.add_parser("prepare") - prepare.add_argument("--publication-dir", type=Path, required=True) - prepare.add_argument("--output-dir", type=Path, required=True) - prepare.add_argument("--verifier-identity", required=True) - prepare.add_argument("--verifier-executable", required=True) - prepare.add_argument("--verifier-arg", action="append", default=[]) - - sig = sub.add_parser("signature-record") - sig.add_argument("--publication-dir", type=Path, required=True) - sig.add_argument("--attestation", type=Path, required=True) - sig.add_argument("--verifier-evidence", type=Path, required=True) - sig.add_argument("--output", type=Path, required=True) - sig.add_argument("--signer-identity", required=True) - sig.add_argument("--verifier-name", required=True) - sig.add_argument("--verifier-version", required=True) - - finalize = sub.add_parser("finalize") - finalize.add_argument("--publication-dir", type=Path, required=True) - finalize.add_argument("--prepared-dir", type=Path, required=True) - finalize.add_argument("--signature-record", type=Path, required=True) - finalize.add_argument("--signature-verifier-evidence", type=Path, required=True) - finalize.add_argument("--output-dir", type=Path, required=True) - finalize.add_argument("--expected-signer-identity", required=True) - finalize.add_argument("--verifier-identity", required=True) - finalize.add_argument("--verifier-executable", required=True) - finalize.add_argument("--verifier-arg", action="append", default=[]) - finalize.add_argument("--binder-executable", required=True) - finalize.add_argument("--binder-arg", action="append", default=[]) - - args = parser.parse_args(argv) - logger.info("Running publication command: %s", args.command) - try: - if args.command == "prepare": - result = prepare_attestation( - publication_dir=args.publication_dir, - output_dir=args.output_dir, - verifier_identity=args.verifier_identity, - verifier=command_verifier( - [args.verifier_executable, *args.verifier_arg] - ), - ) - elif args.command == "signature-record": - path = write_signature_verification_record( - publication_dir=args.publication_dir, - attestation=args.attestation, - verifier_evidence=args.verifier_evidence, - output=args.output, - signer_identity=args.signer_identity, - verifier_name=args.verifier_name, - verifier_version=args.verifier_version, - ) - result = { - "ok": True, - "phase": "signature-record", - "path": path.name, - "sha256": _sha256(path), - } - elif args.command == "finalize": - result = finalize_publication( - publication_dir=args.publication_dir, - prepared_dir=args.prepared_dir, - signature_record=args.signature_record, - signature_verifier_evidence=args.signature_verifier_evidence, - output_dir=args.output_dir, - expected_signer_identity=args.expected_signer_identity, - verifier_identity=args.verifier_identity, - verifier=command_verifier( - [args.verifier_executable, *args.verifier_arg] - ), - binder=command_binder([args.binder_executable, *args.binder_arg]), - ) - else: # pragma: no cover - _fail("COMMAND_INVALID") - except (TransparencyError, OSError) as exc: - logger.error("Publication finalization failed: %s", exc) - # sys.stdout.write(json.dumps({"ok": False, "error": str(exc)}, sort_keys=True) + "\n") - return 2 - logger.info("Publication command completed successfully: %s", args.command) - sys.stdout.write(json.dumps(result, sort_keys=True) + "\n") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/security/govern_archive_merkle_log_authority.py b/security/govern_archive_merkle_log_authority.py deleted file mode 100644 index b9a4d902dd4b9726960cf6b5bce686e99ce6ec23..0000000000000000000000000000000000000000 --- a/security/govern_archive_merkle_log_authority.py +++ /dev/null @@ -1,1459 +0,0 @@ -""" -govern_archive_merkle_log_authority. -""" - -from __future__ import annotations - -import argparse -import base64 -import hashlib -import json -import logging -import re -import shutil -import stat -import tempfile -from datetime import datetime, timedelta, timezone -from pathlib import Path -from typing import Any - -import tomllib -from cryptography.exceptions import InvalidSignature -from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PublicKey - -try: - import verify_archive_merkle_transparency as merkle -except ImportError: # canonical package import - from . import verify_archive_merkle_transparency as merkle - -logger = logging.getLogger(__name__) -HERE = Path(__file__).resolve().parent -POLICY = tomllib.loads((HERE / "release_archive_log_authority_policy.toml").read_text()) -PREDICATE_TYPE = str(POLICY["predicate_type"]) - - -# ── Verified-state document names ──────────────────────────────────────── -# -# One constant per release-evidence document. The values are the on-disk -# artifact names and are unchanged; only the repetition is removed. Reading -# a document by a name that says which document it is also keeps static -# analysis from reading the filename token 'trusted' as 'confidential': -# these files carry published Merkle roots and SHA-256 digests, which must -# stay in clear text for any third party to verify them. -_DOC_ARCHIVE_LOG_AUTHORITY_STATE = "trusted-archive-log-authority-state.json" -_DOC_ARCHIVE_MERKLE_STATE = "trusted-archive-merkle-state.json" - - -_OUTPUT_NAMES = { - "release-archive-log-authority-bundle.json", - _DOC_ARCHIVE_LOG_AUTHORITY_STATE, - "active-archive-log-authority.json", - "release-archive-log-authority-receipt.json", -} -_RUN164_NAMES = { - "release-archive-merkle-bundle.json", - _DOC_ARCHIVE_MERKLE_STATE, - "active-archive-merkle-evidence.json", - "release-archive-merkle-receipt.json", -} -_ID_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:/+-]{0,255}$") - - -class ArchiveLogAuthorityError(RuntimeError): # ruff: ignore[undocumented-public-class] - pass - - -def _fail(code: str) -> None: - raise ArchiveLogAuthorityError(code) - - -def _canonical(value: Any) -> bytes: - # Keep the release-security canonical byte contract exactly aligned with Run 164. - return merkle._canonical(value) - - -def _sha_bytes(raw: bytes) -> str: - return hashlib.sha256(raw).hexdigest() - - -def _sha(path: Path) -> str: - h = hashlib.sha256() - with path.open("rb") as f: - for chunk in iter(lambda: f.read(1024 * 1024), b""): - h.update(chunk) - return h.hexdigest() - - -def _no_dupes(pairs): - out = {} - for key, value in pairs: - if key in out: - _fail("ARCHIVE_LOG_AUTHORITY_JSON_DUPLICATE_KEY") - out[key] = value - return out - - -def _loads(raw: bytes, code: str) -> dict[str, Any]: - if len(raw) > int(POLICY["max_json_bytes"]): - _fail(code + "_TOO_LARGE") - try: - value = json.loads(raw.decode("utf-8"), object_pairs_hook=_no_dupes) - except ArchiveLogAuthorityError: - raise - except Exception: # ruff: ignore[blind-except] - _fail(code + "_JSON_INVALID") - if not isinstance(value, dict): - _fail(code + "_SCHEMA_INVALID") - if _canonical(value) != raw: - _fail(code + "_NOT_CANONICAL") - return value - - -def _read_json(path: Path, code: str) -> tuple[dict[str, Any], bytes]: - path = Path(path) - try: - st = path.lstat() - except OSError: - _fail(code + "_MISSING") - if not stat.S_ISREG(st.st_mode) or st.st_nlink != 1: - _fail(code + "_NOT_REGULAR") - raw = path.read_bytes() - return _loads(raw, code), raw - - -def _write(path: Path, value: Any) -> None: - path.write_bytes(_canonical(value)) - - -def _id(value: Any, code: str) -> str: - if not isinstance(value, str) or not _ID_RE.fullmatch(value): - _fail(code) - return value - - -def _hex(value: Any, code: str) -> str: - _len = len(value) != 64 # ruff: ignore[magic-value-comparison] - if not isinstance(value, str) or _len: - _fail(code) - try: - bytes.fromhex(value) - except ValueError: - _fail(code) - if value != value.lower(): - _fail(code) - return value - - -def _dt(value: Any, code: str) -> datetime: - if not isinstance(value, str) or not value.endswith("Z"): - _fail(code) - try: - parsed = datetime.fromisoformat(value[:-1] + "+00:00") - except ValueError: - _fail(code) - if parsed.tzinfo is None: - _fail(code) - return parsed.astimezone(timezone.utc) - - -def _ts(value: datetime) -> str: - return ( - value.astimezone(timezone.utc) - .isoformat(timespec="seconds") - .replace("+00:00", "Z") - ) - - -def _b64(value: Any, code: str, size: int | None = None) -> bytes: - if not isinstance(value, str): - _fail(code) - try: - raw = base64.b64decode(value, validate=True) - except Exception: # ruff: ignore[blind-except] - _fail(code) - if size is not None and len(raw) != size: - _fail(code) - if base64.b64encode(raw).decode("ascii") != value: - _fail(code) - return raw - - -def _verify_sig(public_key: str, signature: str, message: bytes, code: str) -> None: - try: - Ed25519PublicKey.from_public_bytes( - _b64(public_key, code + "_PUBLIC_INVALID", 32) - ).verify(_b64(signature, code + "_SIGNATURE_INVALID", 64), message) - except InvalidSignature: - _fail(code + "_INVALID") - except ArchiveLogAuthorityError: - raise - except Exception: # ruff: ignore[blind-except] - _fail(code + "_INVALID") - - -def _positive_int(value: Any, code: str) -> int: - if not isinstance(value, int) or isinstance(value, bool) or value <= 0: - _fail(code) - return value - - -def _regular_dir(path: Path, code: str) -> Path: - path = Path(path) - try: - st = path.lstat() - except OSError: - _fail(code) - if not stat.S_ISDIR(st.st_mode): - _fail(code) - return path.resolve() - - -def _outside(path: Path, protected: list[Path], code: str) -> Path: - p = Path(path).absolute() - for item in protected: - q = Path(item).absolute() - if p == q or q in p.parents or p in q.parents: - _fail(code) - return p - - -def _dir_fingerprint(root: Path, code: str) -> tuple[tuple[str, str, int, int], ...]: - root = _regular_dir(root, code) - rows = [] - for path in sorted(root.rglob("*")): - rel = path.relative_to(root).as_posix() - st = path.lstat() - if stat.S_ISDIR(st.st_mode): - continue - if not stat.S_ISREG(st.st_mode) or st.st_nlink != 1: - _fail(code) - rows.append((rel, _sha(path), st.st_size, stat.S_IMODE(st.st_mode))) - return tuple(rows) - - -def _authority_fingerprint(paths: list[Path], code: str) -> tuple[Any, ...]: - rows = [] - for raw in paths: - p = Path(raw) - if p.is_dir(): - rows.append((str(p.absolute()), "dir", _dir_fingerprint(p, code))) - else: - try: - st = p.lstat() - except OSError: - _fail(code) - if not stat.S_ISREG(st.st_mode) or st.st_nlink != 1: - _fail(code) - rows.append( - ( - str(p.absolute()), - "file", - _sha(p), - st.st_size, - stat.S_IMODE(st.st_mode), - ) - ) - return tuple(rows) - - -def _pub_fingerprint(public_key: str) -> str: - return _sha_bytes(_b64(public_key, "ARCHIVE_LOG_AUTHORITY_PUBLIC_KEY_INVALID", 32)) - - -def _root_key(value: Any, code: str, *, recovery: bool) -> dict[str, str]: - expected = {"identity", "operator", "expires", "publicKey"} - if recovery: - expected.add("recoveryChannel") - if not isinstance(value, dict) or set(value) != expected: - _fail(code + "_SCHEMA_INVALID") - out = { - "identity": _id(value["identity"], code + "_IDENTITY_INVALID"), - "operator": _id(value["operator"], code + "_OPERATOR_INVALID"), - "expires": value["expires"], - "publicKey": value["publicKey"], - } - _dt(out["expires"], code + "_EXPIRES_INVALID") - _b64(out["publicKey"], code + "_PUBLIC_INVALID", 32) - if recovery: - out["recoveryChannel"] = _id( - value["recoveryChannel"], code + "_CHANNEL_INVALID" - ) - if out != value: - _fail(code + "_NOT_NORMALIZED") - return out - - -def _verify_control_root( # ruff: ignore[too-many-branches] - doc: dict[str, Any], - expected_pin: str, - *, - now: datetime, - recovery: bool, - historical: bool, -) -> dict[str, Any]: - prefix = "ARCHIVE_LOG_RECOVERY_ROOT" if recovery else "ARCHIVE_LOG_GOVERNANCE_ROOT" - root_type = ( - "archive-log-recovery-root" if recovery else "archive-log-governance-root" - ) - schema_key = ( - "recovery_root_schema_version" if recovery else "governance_root_schema_version" - ) - if set(doc) != {"signed", "signatures"} or not isinstance(doc.get("signed"), dict): - _fail(prefix + "_SCHEMA_INVALID") - signed = doc["signed"] - expected = { - "_type", - "specVersion", - "schemaVersion", - "rootId", - "version", - "issuedAt", - "expires", - "threshold", - "selectedSignerKeyIds", - "keys", - } - if set(signed) != expected or signed.get("_type") != root_type: - _fail(prefix + "_SIGNED_SCHEMA_INVALID") - if ( - signed.get("specVersion") != str(POLICY["spec_version"]) - or signed.get("schemaVersion") != int(POLICY[schema_key]) - or signed.get("version") != 1 - ): - _fail(prefix + "_VERSION_INVALID") - root_id = _id(signed["rootId"], prefix + "_ID_INVALID") - issued = _dt(signed["issuedAt"], prefix + "_ISSUED_INVALID") - expires = _dt(signed["expires"], prefix + "_EXPIRES_INVALID") - if expires <= issued or expires - issued > timedelta( - days=int(POLICY["max_root_lifetime_days"]) - ): - _fail(prefix + "_LIFETIME_INVALID") - if not historical: - if issued > now + timedelta(minutes=int(POLICY["max_clock_skew_minutes"])): - _fail(prefix + "_FROM_FUTURE") - if expires <= now: - _fail(prefix + "_EXPIRED") - keys_raw = signed["keys"] - min_keys = int(POLICY["min_recovery_keys" if recovery else "min_governance_keys"]) - if not isinstance(keys_raw, dict) or not ( - min_keys <= len(keys_raw) <= 64 # ruff: ignore[magic-value-comparison] - ): - _fail(prefix + "_KEYS_INVALID") - keys = {} - for raw_kid, value in sorted(keys_raw.items()): - kid = _id(raw_kid, prefix + "_KEY_ID_INVALID") - if kid != raw_kid or kid in keys: - _fail(prefix + "_KEY_ID_INVALID") - keys[kid] = _root_key(value, prefix + "_KEY", recovery=recovery) - if len({v["publicKey"] for v in keys.values()}) != len(keys): - _fail(prefix + "_PUBLIC_KEY_REUSE") - for key in keys.values(): - if _dt(key["expires"], prefix + "_KEY_EXPIRES_INVALID") < expires: - _fail(prefix + "_KEY_EXPIRES_BEFORE_ROOT") - threshold = _positive_int(signed["threshold"], prefix + "_THRESHOLD_INVALID") - min_threshold = int( - POLICY["min_recovery_threshold" if recovery else "min_governance_threshold"] - ) - if threshold < min_threshold or threshold > len(keys): - _fail(prefix + "_THRESHOLD_INVALID") - selected = signed["selectedSignerKeyIds"] - if not isinstance(selected, list): - _fail(prefix + "_SELECTED_INVALID") - selected = [_id(x, prefix + "_SELECTED_INVALID") for x in selected] - if ( - selected != sorted(selected) - or len(selected) != threshold - or len(set(selected)) != len(selected) - or any(x not in keys for x in selected) - ): - _fail(prefix + "_SELECTED_INVALID") - min_ops = int( - POLICY["min_recovery_operators" if recovery else "min_governance_operators"] - ) - if len({keys[k]["operator"] for k in selected}) < min_ops: - _fail(prefix + "_OPERATOR_QUORUM_INVALID") - if recovery and len({keys[k]["recoveryChannel"] for k in selected}) < int( - POLICY["min_recovery_channels"] - ): - _fail(prefix + "_CHANNEL_QUORUM_INVALID") - signatures = doc["signatures"] - if not isinstance(signatures, list) or len(signatures) != len(selected): - _fail(prefix + "_SIGNATURE_SET_INVALID") - sigs = {} - for item in signatures: - if not isinstance(item, dict) or set(item) != {"keyId", "signature"}: - _fail(prefix + "_SIGNATURE_SCHEMA_INVALID") - kid = _id(item["keyId"], prefix + "_SIGNATURE_KEY_INVALID") - if kid in sigs: - _fail(prefix + "_SIGNATURE_DUPLICATE") - sigs[kid] = item["signature"] - if sorted(sigs) != selected or signatures != sorted( - signatures, key=lambda x: x["keyId"] - ): - _fail(prefix + "_SIGNATURE_SET_INVALID") - message = _canonical(signed) - for kid in selected: - _verify_sig(keys[kid]["publicKey"], sigs[kid], message, prefix + "_SIGNATURE") - raw = _canonical(doc) - sha = _sha_bytes(raw) - if sha != _hex(expected_pin, prefix + "_PIN_INVALID"): - _fail(prefix + "_PIN_MISMATCH") - return { - "rootId": root_id, - "sha256": sha, - "keys": keys, - "threshold": threshold, - "issued": issued, - "expires": expires, - "doc": doc, - } - - -def _load_run164_offline( - run164_dir: Path, - transparency_root_path: Path, - transparency_root_pin: str, - *, - now: datetime, - historical: bool, -) -> dict[str, Any]: - root_doc, _ = _read_json(Path(transparency_root_path), "ARCHIVE_LOG_RUN164_ROOT") - try: - root = merkle.verify_transparency_root( - root_doc, transparency_root_pin, now=now, historical=historical - ) - docs, raws = merkle._load_output(Path(run164_dir)) - replay = merkle._replay_history( - root=root, - bundle=docs["release-archive-merkle-bundle.json"], - receipt=docs["release-archive-merkle-receipt.json"], - now=now, - historical=historical, - current_run163_docs=None, - current_run163_raws=None, - ) - except Exception as exc: # ruff: ignore[blind-except] - _fail("ARCHIVE_LOG_RUN164_VERIFICATION_FAILED:" + str(exc)) - state = docs[_DOC_ARCHIVE_MERKLE_STATE] - active = docs["active-archive-merkle-evidence.json"] - bundle_raw = raws["release-archive-merkle-bundle.json"] - expected_state = { - "schemaVersion": int(merkle.POLICY["state_schema_version"]), - "status": "trusted-archive-merkle-transparency", - "sequence": replay["sequence"], - "run163Sequence": replay["last"]["run163Sequence"], - "run163AnchorConsensusHeadSha256": replay["last"][ - "run163AnchorConsensusHeadSha256" - ], - "merkleConsensusHeadSha256": replay["merkleConsensusHeadSha256"], - "transparencyRootSha256": root["sha256"], - "bundleArtifact": { - "name": "release-archive-merkle-bundle.json", - "sha256": _sha_bytes(bundle_raw), - "size": len(bundle_raw), - }, - } - if state != expected_state: - _fail("ARCHIVE_LOG_RUN164_STATE_MISMATCH") - last = replay["last"] - expected_active = { - "schemaVersion": int(merkle.POLICY["active_schema_version"]), - "status": "active-archive-merkle-transparency", - "sequence": replay["sequence"], - "run163AnchorConsensusHeadSha256": last["run163AnchorConsensusHeadSha256"], - "merkleConsensusHeadSha256": replay["merkleConsensusHeadSha256"], - "leafHash": last["leafHash"], - "logs": last["logs"], - "gossipResponseSha256s": last["gossipResponseSha256s"], - } - if active != expected_active: - _fail("ARCHIVE_LOG_RUN164_ACTIVE_MISMATCH") - artifacts = { - name: {"sha256": _sha_bytes(raws[name]), "size": len(raws[name])} - for name in sorted(raws) - } - checkpoints = { - row["logId"]: {k: row[k] for k in ("checkpointSha256", "treeSize", "rootHash")} - for row in last["logs"] - } - return { - "root": root, - "docs": docs, - "raws": raws, - "replay": replay, - "artifacts": artifacts, - "checkpoints": checkpoints, - } - - -def _authority( - value: Any, *, expected_log_ids: set[str], code: str -) -> dict[str, dict[str, str]]: - if not isinstance(value, dict) or set(value) != expected_log_ids: - _fail(code + "_LOG_SET_INVALID") - out = {} - pubs = set() - for lid in sorted(value): - if _id(lid, code + "_LOG_ID_INVALID") != lid: - _fail(code + "_LOG_ID_INVALID") - try: - row = merkle._log_entry(value[lid], code + "_LOG") - except Exception as exc: # ruff: ignore[blind-except] - _fail(code + "_LOG_INVALID:" + str(exc)) - if ( - row["publicKey"] in pubs - or row["gossipPublicKey"] in pubs - or row["publicKey"] == row["gossipPublicKey"] - ): - _fail(code + "_PUBLIC_KEY_REUSE") - pubs.add(row["publicKey"]) - pubs.add(row["gossipPublicKey"]) - out[lid] = row - if out != value: - _fail(code + "_NOT_NORMALIZED") - return out - - -def _authority_sha(authority: dict[str, Any]) -> str: - return _sha_bytes(_canonical(authority)) - - -def _authority_fingerprints(authority: dict[str, Any]) -> set[str]: - out = set() - for row in authority.values(): - out.add(_pub_fingerprint(row["publicKey"])) - out.add(_pub_fingerprint(row["gossipPublicKey"])) - return out - - -def _enforce_plane_separation( - governance: dict[str, Any], - recovery: dict[str, Any], - run164_root: dict[str, Any], - *authorities: dict[str, Any], -) -> None: - gov_ops = {v["operator"] for v in governance["keys"].values()} - gov_pubs = {v["publicKey"] for v in governance["keys"].values()} - rec_ops = {v["operator"] for v in recovery["keys"].values()} - rec_pubs = {v["publicKey"] for v in recovery["keys"].values()} - merkle_root_ops = {v["operator"] for v in run164_root["keys"].values()} - merkle_root_pubs = {v["publicKey"] for v in run164_root["keys"].values()} - log_ops = {v["operator"] for v in run164_root["logs"].values()} | { - v["gossipOperator"] for v in run164_root["logs"].values() - } - log_pubs = {v["publicKey"] for v in run164_root["logs"].values()} | { - v["gossipPublicKey"] for v in run164_root["logs"].values() - } - if gov_ops & rec_ops or gov_pubs & rec_pubs: - _fail("ARCHIVE_LOG_CONTROL_PLANES_OVERLAP") - if (gov_ops | rec_ops) & (merkle_root_ops | log_ops) or (gov_pubs | rec_pubs) & ( - merkle_root_pubs | log_pubs - ): - _fail("ARCHIVE_LOG_CONTROL_PLANES_OVERLAP") - for authority in authorities: - a_ops = {v["operator"] for v in authority.values()} | { - v["gossipOperator"] for v in authority.values() - } - a_pubs = {v["publicKey"] for v in authority.values()} | { - v["gossipPublicKey"] for v in authority.values() - } - if (gov_ops | rec_ops) & a_ops or (gov_pubs | rec_pubs) & a_pubs: - _fail("ARCHIVE_LOG_CONTROL_PLANES_OVERLAP") - - -def _handoff_subject( - *, signed: dict[str, Any], log_id: str, checkpoint: dict[str, Any] -) -> dict[str, Any]: - current = signed["currentAuthority"][log_id] - nxt = signed["nextAuthority"][log_id] - return { - "_type": "archive-merkle-log-authority-handoff", - "specVersion": str(POLICY["spec_version"]), - "schemaVersion": int(POLICY["handoff_schema_version"]), - "transitionId": signed["transitionId"], - "sequence": signed["sequence"], - "kind": signed["kind"], - "logId": log_id, - "run164Sequence": signed["run164Sequence"], - "merkleConsensusHeadSha256": signed["merkleConsensusHeadSha256"], - "priorCheckpoint": checkpoint, - "currentLogAuthority": current, - "nextLogAuthority": nxt, - "currentAuthoritySha256": _authority_sha(signed["currentAuthority"]), - "nextAuthoritySha256": _authority_sha(signed["nextAuthority"]), - } - - -def _verify_transition_document( # ruff: ignore[too-many-branches] - doc: dict[str, Any], - *, - governance: dict[str, Any], - recovery: dict[str, Any], - run164: dict[str, Any], - previous_state: dict[str, Any] | None, - previous_state_raw: bytes | None, - now: datetime, - creation: bool, -) -> dict[str, Any]: - if set(doc) != {"signed", "signatures"} or not isinstance(doc.get("signed"), dict): - _fail("ARCHIVE_LOG_TRANSITION_SCHEMA_INVALID") - signed = doc["signed"] - expected = { - "_type", - "specVersion", - "schemaVersion", - "transitionId", - "sequence", - "kind", - "issuedAt", - "run164Sequence", - "merkleConsensusHeadSha256", - "run164Artifacts", - "previousAuthorityStateSha256", - "currentAuthority", - "nextAuthority", - "changedLogIds", - "compromisedKeyFingerprints", - "revokedKeyFingerprints", - "authorizationRole", - "selectedSignerKeyIds", - "handoffSubjectSha256s", - } - if ( - set(signed) != expected - or signed.get("_type") != "archive-merkle-log-authority-transition" - ): - _fail("ARCHIVE_LOG_TRANSITION_SIGNED_SCHEMA_INVALID") - if signed.get("specVersion") != str(POLICY["spec_version"]) or signed.get( - "schemaVersion" - ) != int(POLICY["transition_schema_version"]): - _fail("ARCHIVE_LOG_TRANSITION_VERSION_INVALID") - _id(signed["transitionId"], "ARCHIVE_LOG_TRANSITION_ID_INVALID") - sequence = _positive_int( - signed["sequence"], "ARCHIVE_LOG_TRANSITION_SEQUENCE_INVALID" - ) - kind = signed["kind"] - if kind not in {"bootstrap", "scheduled-rotation", "compromise-recovery"}: - _fail("ARCHIVE_LOG_TRANSITION_KIND_INVALID") - issued = _dt(signed["issuedAt"], "ARCHIVE_LOG_TRANSITION_ISSUED_INVALID") - if issued > now + timedelta(minutes=int(POLICY["max_clock_skew_minutes"])): - _fail("ARCHIVE_LOG_TRANSITION_FROM_FUTURE") - if creation and now - issued > timedelta( - minutes=int(POLICY["max_transition_freshness_minutes"]) - ): - _fail("ARCHIVE_LOG_TRANSITION_STALE") - if ( - signed["run164Sequence"] != run164["replay"]["sequence"] - or signed["merkleConsensusHeadSha256"] - != run164["replay"]["merkleConsensusHeadSha256"] - ): - _fail("ARCHIVE_LOG_TRANSITION_RUN164_BINDING_INVALID") - if signed["run164Artifacts"] != run164["artifacts"]: - _fail("ARCHIVE_LOG_TRANSITION_RUN164_ARTIFACT_INVALID") - log_ids = set(run164["root"]["logs"]) - current = _authority( - signed["currentAuthority"], - expected_log_ids=log_ids, - code="ARCHIVE_LOG_CURRENT_AUTHORITY", - ) - nxt = _authority( - signed["nextAuthority"], - expected_log_ids=log_ids, - code="ARCHIVE_LOG_NEXT_AUTHORITY", - ) - for lid in log_ids: - if current[lid]["gossipIdentity"] != nxt[lid]["gossipIdentity"]: - _fail("ARCHIVE_LOG_GOSSIP_IDENTITY_CHANGE_FORBIDDEN") - changed = signed["changedLogIds"] - if not isinstance(changed, list): - _fail("ARCHIVE_LOG_CHANGED_SET_INVALID") - changed = [_id(x, "ARCHIVE_LOG_CHANGED_ID_INVALID") for x in changed] - computed_changed = sorted(lid for lid in log_ids if current[lid] != nxt[lid]) - if ( - changed != sorted(changed) - or len(set(changed)) != len(changed) - or changed != computed_changed - ): - _fail("ARCHIVE_LOG_CHANGED_SET_INVALID") - compromised = signed["compromisedKeyFingerprints"] - revoked = signed["revokedKeyFingerprints"] - if not isinstance(compromised, list) or not isinstance(revoked, list): - _fail("ARCHIVE_LOG_REVOCATION_LIST_INVALID") - compromised = [ - _hex(x, "ARCHIVE_LOG_COMPROMISED_FINGERPRINT_INVALID") for x in compromised - ] - revoked = [_hex(x, "ARCHIVE_LOG_REVOKED_FINGERPRINT_INVALID") for x in revoked] - if ( - compromised != sorted(compromised) - or len(set(compromised)) != len(compromised) - or revoked != sorted(revoked) - or len(set(revoked)) != len(revoked) - ): - _fail("ARCHIVE_LOG_REVOCATION_LIST_INVALID") - previous_revoked = ( - [] if previous_state is None else previous_state["revokedKeyFingerprints"] - ) - if previous_state is None: - if ( - sequence != 1 - or signed["previousAuthorityStateSha256"] is not None - or kind != "bootstrap" - ): - _fail("ARCHIVE_LOG_BOOTSTRAP_INVALID") - original = {lid: run164["root"]["logs"][lid] for lid in sorted(log_ids)} - if current != original or nxt != original or changed or compromised or revoked: - _fail("ARCHIVE_LOG_BOOTSTRAP_AUTHORITY_INVALID") - else: - if sequence != previous_state["sequence"] + 1: - _fail("ARCHIVE_LOG_SEQUENCE_INVALID") - if previous_state_raw is None or signed[ - "previousAuthorityStateSha256" - ] != _sha_bytes(previous_state_raw): - _fail("ARCHIVE_LOG_PREVIOUS_STATE_BINDING_INVALID") - if current != previous_state["activeAuthority"]: - _fail("ARCHIVE_LOG_CURRENT_AUTHORITY_MISMATCH") - if ( - signed["run164Sequence"] != previous_state["run164Sequence"] - or signed["merkleConsensusHeadSha256"] - != previous_state["merkleConsensusHeadSha256"] - or signed["run164Artifacts"] != previous_state["run164Artifacts"] - ): - _fail("ARCHIVE_LOG_RUN164_SNAPSHOT_CHANGED") - if kind == "bootstrap": - _fail("ARCHIVE_LOG_BOOTSTRAP_REPEATED") - if not changed: - _fail("ARCHIVE_LOG_ROTATION_NOOP") - current_fps = _authority_fingerprints(current) - next_fps = _authority_fingerprints(nxt) - if set(previous_revoked) & next_fps: - _fail("ARCHIVE_LOG_REVOKED_KEY_REINTRODUCED") - if kind == "scheduled-rotation": - if compromised: - _fail("ARCHIVE_LOG_SCHEDULED_COMPROMISED_NOT_EMPTY") - replaced = set() - for lid in changed: - old = current[lid] - new = nxt[lid] - if ( - old["operator"] != new["operator"] - and old["publicKey"] == new["publicKey"] - ): - _fail("ARCHIVE_LOG_OPERATOR_REBOUND_WITHOUT_KEY_ROTATION") - if ( - old["gossipOperator"] != new["gossipOperator"] - and old["gossipPublicKey"] == new["gossipPublicKey"] - ): - _fail("ARCHIVE_LOG_GOSSIP_OPERATOR_REBOUND_WITHOUT_KEY_ROTATION") - if old["publicKey"] != new["publicKey"]: - replaced.add(_pub_fingerprint(old["publicKey"])) - if old["gossipPublicKey"] != new["gossipPublicKey"]: - replaced.add(_pub_fingerprint(old["gossipPublicKey"])) - expected_revoked = sorted(set(previous_revoked) | replaced) - if revoked != expected_revoked: - _fail("ARCHIVE_LOG_SCHEDULED_REVOCATION_INVALID") - elif kind == "compromise-recovery": - if not compromised: - _fail("ARCHIVE_LOG_RECOVERY_COMPROMISED_EMPTY") - if not set(compromised) <= current_fps: - _fail("ARCHIVE_LOG_RECOVERY_COMPROMISED_UNKNOWN") - affected = sorted( - lid - for lid in log_ids - if { - _pub_fingerprint(current[lid]["publicKey"]), - _pub_fingerprint(current[lid]["gossipPublicKey"]), - } - & set(compromised) - ) - if affected != changed: - _fail("ARCHIVE_LOG_RECOVERY_CHANGED_SET_INVALID") - replaced = set() - for lid in changed: - old = current[lid] - new = nxt[lid] - if ( - old["publicKey"] == new["publicKey"] - or old["gossipPublicKey"] == new["gossipPublicKey"] - ): - _fail("ARCHIVE_LOG_RECOVERY_REQUIRES_FULL_KEY_ROTATION") - replaced.add(_pub_fingerprint(old["publicKey"])) - replaced.add(_pub_fingerprint(old["gossipPublicKey"])) - expected_revoked = sorted(set(previous_revoked) | replaced) - if revoked != expected_revoked: - _fail("ARCHIVE_LOG_RECOVERY_REVOCATION_INVALID") - elif previous_state is not None: - _fail("ARCHIVE_LOG_BOOTSTRAP_INVALID") - if set(revoked) & next_fps: - _fail("ARCHIVE_LOG_REVOKED_KEY_ACTIVE") - role = signed["authorizationRole"] - expected_role = "recovery" if kind == "compromise-recovery" else "governance" - if role != expected_role: - _fail("ARCHIVE_LOG_AUTHORIZATION_ROLE_INVALID") - root = recovery if role == "recovery" else governance - selected = signed["selectedSignerKeyIds"] - if not isinstance(selected, list): - _fail("ARCHIVE_LOG_SELECTED_SIGNERS_INVALID") - selected = [_id(x, "ARCHIVE_LOG_SELECTED_SIGNERS_INVALID") for x in selected] - if ( - selected != sorted(selected) - or len(selected) != root["threshold"] - or len(set(selected)) != len(selected) - or any(x not in root["keys"] for x in selected) - ): - _fail("ARCHIVE_LOG_SELECTED_SIGNERS_INVALID") - if role == "recovery": - if len({root["keys"][k]["operator"] for k in selected}) < int( - POLICY["min_recovery_operators"] - ) or len({root["keys"][k]["recoveryChannel"] for k in selected}) < int( - POLICY["min_recovery_channels"] - ): - _fail("ARCHIVE_LOG_RECOVERY_QUORUM_INVALID") - elif len({root["keys"][k]["operator"] for k in selected}) < int( - POLICY["min_governance_operators"] - ): - _fail("ARCHIVE_LOG_GOVERNANCE_QUORUM_INVALID") - if issued < root["issued"] or issued >= root["expires"]: - _fail("ARCHIVE_LOG_TRANSITION_OUTSIDE_ROOT_LIFETIME") - for kid in selected: - if issued >= _dt( - root["keys"][kid]["expires"], "ARCHIVE_LOG_SIGNER_EXPIRES_INVALID" - ): - _fail("ARCHIVE_LOG_SIGNER_EXPIRED_AT_TRANSITION") - signatures = doc["signatures"] - if not isinstance(signatures, list) or len(signatures) != len(selected): - _fail("ARCHIVE_LOG_TRANSITION_SIGNATURE_SET_INVALID") - sigs = {} - for item in signatures: - if not isinstance(item, dict) or set(item) != {"keyId", "signature"}: - _fail("ARCHIVE_LOG_TRANSITION_SIGNATURE_SCHEMA_INVALID") - kid = _id(item["keyId"], "ARCHIVE_LOG_TRANSITION_SIGNATURE_KEY_INVALID") - if kid in sigs: - _fail("ARCHIVE_LOG_TRANSITION_SIGNATURE_DUPLICATE") - sigs[kid] = item["signature"] - if sorted(sigs) != selected or signatures != sorted( - signatures, key=lambda x: x["keyId"] - ): - _fail("ARCHIVE_LOG_TRANSITION_SIGNATURE_SET_INVALID") - message = _canonical(signed) - for kid in selected: - _verify_sig( - root["keys"][kid]["publicKey"], - sigs[kid], - message, - "ARCHIVE_LOG_TRANSITION_SIGNATURE", - ) - subjects = {} - for lid in sorted(log_ids): - subjects[lid] = _handoff_subject( - signed=signed, log_id=lid, checkpoint=run164["checkpoints"][lid] - ) - expected_hashes = { - lid: _sha_bytes(_canonical(subjects[lid])) for lid in sorted(subjects) - } - if signed["handoffSubjectSha256s"] != expected_hashes: - _fail("ARCHIVE_LOG_HANDOFF_SUBJECT_HASH_INVALID") - return { - "signed": signed, - "transitionSha256": _sha_bytes(_canonical(doc)), - "subjects": subjects, - "revoked": revoked, - "current": current, - "next": nxt, - "kind": kind, - "sequence": sequence, - } - - -def _verify_handoff_document( - doc: dict[str, Any], *, transition: dict[str, Any] -) -> dict[str, Any]: - signed = transition["signed"] - if ( - set(doc) != {"schemaVersion", "transitionId", "sequence", "proofs"} - or doc.get("schemaVersion") != int(POLICY["handoff_schema_version"]) - or doc.get("transitionId") != signed["transitionId"] - or doc.get("sequence") != signed["sequence"] - ): - _fail("ARCHIVE_LOG_HANDOFF_SCHEMA_INVALID") - proofs = doc["proofs"] - if not isinstance(proofs, list) or len(proofs) != len(transition["subjects"]): - _fail("ARCHIVE_LOG_HANDOFF_PROOF_SET_INVALID") - if proofs != sorted(proofs, key=lambda x: x.get("logId", "")): - _fail("ARCHIVE_LOG_HANDOFF_PROOF_ORDER_INVALID") - seen = set() - for proof in proofs: - expected = { - "logId", - "subjectSha256", - "newLogSignature", - "newGossipSignature", - "oldLogSignature", - "oldGossipSignature", - } - if not isinstance(proof, dict) or set(proof) != expected: - _fail("ARCHIVE_LOG_HANDOFF_PROOF_SCHEMA_INVALID") - lid = _id(proof["logId"], "ARCHIVE_LOG_HANDOFF_LOG_ID_INVALID") - if lid in seen or lid not in transition["subjects"]: - _fail("ARCHIVE_LOG_HANDOFF_LOG_ID_INVALID") - seen.add(lid) - subject = transition["subjects"][lid] - raw = _canonical(subject) - if proof["subjectSha256"] != _sha_bytes(raw): - _fail("ARCHIVE_LOG_HANDOFF_SUBJECT_BINDING_INVALID") - nxt = transition["next"][lid] - cur = transition["current"][lid] - _verify_sig( - nxt["publicKey"], - proof["newLogSignature"], - raw, - "ARCHIVE_LOG_NEW_LOG_HANDOFF", - ) - _verify_sig( - nxt["gossipPublicKey"], - proof["newGossipSignature"], - raw, - "ARCHIVE_LOG_NEW_GOSSIP_HANDOFF", - ) - changed = lid in signed["changedLogIds"] - if transition["kind"] == "scheduled-rotation" and changed: - if proof["oldLogSignature"] is None or proof["oldGossipSignature"] is None: - _fail("ARCHIVE_LOG_OLD_HANDOFF_REQUIRED") - _verify_sig( - cur["publicKey"], - proof["oldLogSignature"], - raw, - "ARCHIVE_LOG_OLD_LOG_HANDOFF", - ) - _verify_sig( - cur["gossipPublicKey"], - proof["oldGossipSignature"], - raw, - "ARCHIVE_LOG_OLD_GOSSIP_HANDOFF", - ) - elif ( - proof["oldLogSignature"] is not None - or proof["oldGossipSignature"] is not None - ): - _fail("ARCHIVE_LOG_OLD_HANDOFF_FORBIDDEN") - return doc - - -def _event_head(previous_head: str | None, event_without_head: dict[str, Any]) -> str: - return _sha_bytes( - _canonical( - { - "previousLogAuthorityChainHeadSha256": previous_head, - "event": event_without_head, - } - ) - ) - - -def _load_output(root: Path) -> tuple[dict[str, Any], dict[str, bytes]]: - root = _regular_dir(root, "ARCHIVE_LOG_OUTPUT_INVALID") - if {p.name for p in root.iterdir()} != _OUTPUT_NAMES: - _fail("ARCHIVE_LOG_OUTPUT_ALLOWLIST_INVALID") - docs = {} - raws = {} - for name in sorted(_OUTPUT_NAMES): - docs[name], raws[name] = _read_json(root / name, "ARCHIVE_LOG_OUTPUT") - return docs, raws - - -def _replay( - *, - governance: dict[str, Any], - recovery: dict[str, Any], - run164: dict[str, Any], - bundle: dict[str, Any], - receipt: dict[str, Any], - now: datetime, -) -> dict[str, Any]: - if ( - set(bundle) - != { - "schemaVersion", - "predicateType", - "status", - "governanceRootSha256", - "recoveryRootSha256", - "events", - } - or bundle.get("schemaVersion") != int(POLICY["bundle_schema_version"]) - or bundle.get("predicateType") != PREDICATE_TYPE - or bundle.get("status") != "archive-log-authority-history" - or bundle.get("governanceRootSha256") != governance["sha256"] - or bundle.get("recoveryRootSha256") != recovery["sha256"] - ): - _fail("ARCHIVE_LOG_BUNDLE_SCHEMA_INVALID") - if ( - set(receipt) != {"schemaVersion", "status", "events"} - or receipt.get("schemaVersion") != int(POLICY["receipt_schema_version"]) - or receipt.get("status") != "archive-log-authority-accepted" - ): - _fail("ARCHIVE_LOG_RECEIPT_SCHEMA_INVALID") - events = bundle["events"] - recs = receipt["events"] - if ( - not isinstance(events, list) - or not events - or not isinstance(recs, list) - or len(events) != len(recs) - ): - _fail("ARCHIVE_LOG_HISTORY_LENGTH_INVALID") - prev_state = None - prev_state_raw = None - prev_head = None - last_transition = None - last_handoff = None - previous_issued = None - transition_ids = set() - for idx, (event, rec) in enumerate(zip(events, recs), 1): - if ( - not isinstance(event, dict) - or set(event) - != { - "sequence", - "kind", - "transitionSha256", - "run164Sequence", - "merkleConsensusHeadSha256", - "run164Artifacts", - "currentAuthoritySha256", - "nextAuthoritySha256", - "revokedKeyFingerprints", - "handoffSubjectSha256s", - "logAuthorityChainHeadSha256", - } - or event.get("sequence") != idx - ): - _fail("ARCHIVE_LOG_EVENT_SCHEMA_INVALID") - if ( - not isinstance(rec, dict) - or set(rec) - != { - "sequence", - "transitionDocument", - "handoffProofDocument", - "run164Documents", - } - or rec.get("sequence") != idx - ): - _fail("ARCHIVE_LOG_RECEIPT_EVENT_INVALID") - run164_docs = rec["run164Documents"] - if not isinstance(run164_docs, dict) or set(run164_docs) != _RUN164_NAMES: - _fail("ARCHIVE_LOG_EMBEDDED_RUN164_INVALID") - raws = {name: _canonical(run164_docs[name]) for name in sorted(run164_docs)} - artifacts = { - name: {"sha256": _sha_bytes(raws[name]), "size": len(raws[name])} - for name in sorted(raws) - } - if artifacts != run164["artifacts"] or event["run164Artifacts"] != artifacts: - _fail("ARCHIVE_LOG_RUN164_ARTIFACT_REPLAY_INVALID") - tr = _verify_transition_document( - rec["transitionDocument"], - governance=governance, - recovery=recovery, - run164=run164, - previous_state=prev_state, - previous_state_raw=prev_state_raw, - now=now, - creation=False, - ) - transition_id = tr["signed"]["transitionId"] - issued = _dt(tr["signed"]["issuedAt"], "ARCHIVE_LOG_TRANSITION_ISSUED_INVALID") - if transition_id in transition_ids: - _fail("ARCHIVE_LOG_TRANSITION_ID_REUSED") - if previous_issued is not None and issued <= previous_issued: - _fail("ARCHIVE_LOG_TRANSITION_TIME_NOT_MONOTONIC") - transition_ids.add(transition_id) - previous_issued = issued - hf = _verify_handoff_document(rec["handoffProofDocument"], transition=tr) - if ( - event["kind"] != tr["kind"] - or event["transitionSha256"] != tr["transitionSha256"] - or event["run164Sequence"] != tr["signed"]["run164Sequence"] - or event["merkleConsensusHeadSha256"] - != tr["signed"]["merkleConsensusHeadSha256"] - or event["currentAuthoritySha256"] != _authority_sha(tr["current"]) - or event["nextAuthoritySha256"] != _authority_sha(tr["next"]) - or event["revokedKeyFingerprints"] != tr["revoked"] - or event["handoffSubjectSha256s"] != tr["signed"]["handoffSubjectSha256s"] - ): - _fail("ARCHIVE_LOG_EVENT_BINDING_INVALID") - bare = {k: event[k] for k in event if k != "logAuthorityChainHeadSha256"} - head = _event_head(prev_head, bare) - if event["logAuthorityChainHeadSha256"] != head: - _fail("ARCHIVE_LOG_CHAIN_HEAD_INVALID") - prefix_bundle = { - "schemaVersion": bundle["schemaVersion"], - "predicateType": bundle["predicateType"], - "status": bundle["status"], - "governanceRootSha256": bundle["governanceRootSha256"], - "recoveryRootSha256": bundle["recoveryRootSha256"], - "events": events[:idx], - } - bundle_raw = _canonical(prefix_bundle) - state = { - "schemaVersion": int(POLICY["state_schema_version"]), - "status": "trusted-archive-log-authority", - "sequence": idx, - "run164Sequence": tr["signed"]["run164Sequence"], - "merkleConsensusHeadSha256": tr["signed"]["merkleConsensusHeadSha256"], - "run164Artifacts": artifacts, - "logAuthorityChainHeadSha256": head, - "governanceRootSha256": governance["sha256"], - "recoveryRootSha256": recovery["sha256"], - "activeAuthority": tr["next"], - "revokedKeyFingerprints": tr["revoked"], - "continuityCheckpoints": run164["checkpoints"], - "bundleArtifact": { - "name": "release-archive-log-authority-bundle.json", - "sha256": _sha_bytes(bundle_raw), - "size": len(bundle_raw), - }, - } - prev_state = state - prev_state_raw = _canonical(state) - prev_head = head - last_transition = tr - last_handoff = hf - return { - "state": prev_state, - "head": prev_head, - "transition": last_transition, - "handoff": last_handoff, - "sequence": len(events), - } - - -def verify_log_authority_history( # ruff: ignore[undocumented-public-function] - *, - run164_dir: Path, - transparency_root_path: Path, - transparency_root_pin: str, - governance_root_path: Path, - governance_root_pin: str, - recovery_root_path: Path, - recovery_root_pin: str, - output_dir: Path, - now: datetime | None = None, - historical: bool = False, -) -> dict[str, Any]: - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - gov_doc, _ = _read_json(Path(governance_root_path), "ARCHIVE_LOG_GOVERNANCE_ROOT") - rec_doc, _ = _read_json(Path(recovery_root_path), "ARCHIVE_LOG_RECOVERY_ROOT") - governance = _verify_control_root( - gov_doc, governance_root_pin, now=current, recovery=False, historical=historical - ) - recovery = _verify_control_root( - rec_doc, recovery_root_pin, now=current, recovery=True, historical=historical - ) - run164 = _load_run164_offline( - Path(run164_dir), - Path(transparency_root_path), - transparency_root_pin, - now=current, - historical=historical, - ) - docs, raws = _load_output(Path(output_dir)) - replay = _replay( - governance=governance, - recovery=recovery, - run164=run164, - bundle=docs["release-archive-log-authority-bundle.json"], - receipt=docs["release-archive-log-authority-receipt.json"], - now=current, - ) - _enforce_plane_separation( - governance, recovery, run164["root"], replay["state"]["activeAuthority"] - ) - bundle_raw = raws["release-archive-log-authority-bundle.json"] - expected_state = dict(replay["state"]) - if expected_state["bundleArtifact"] != { - "name": "release-archive-log-authority-bundle.json", - "sha256": _sha_bytes(bundle_raw), - "size": len(bundle_raw), - }: - _fail("ARCHIVE_LOG_BUNDLE_ARTIFACT_REPLAY_INVALID") - if docs[_DOC_ARCHIVE_LOG_AUTHORITY_STATE] != expected_state: - _fail("ARCHIVE_LOG_STATE_MISMATCH") - tr = replay["transition"] - expected_active = { - "schemaVersion": int(POLICY["active_schema_version"]), - "status": "active-archive-log-authority", - "sequence": replay["sequence"], - "run164Sequence": tr["signed"]["run164Sequence"], - "merkleConsensusHeadSha256": tr["signed"]["merkleConsensusHeadSha256"], - "logAuthorityChainHeadSha256": replay["head"], - "authority": tr["next"], - "revokedKeyFingerprints": tr["revoked"], - "continuityCheckpoints": run164["checkpoints"], - "handoffSubjectSha256s": tr["signed"]["handoffSubjectSha256s"], - } - if docs["active-archive-log-authority.json"] != expected_active: - _fail("ARCHIVE_LOG_ACTIVE_MISMATCH") - return { - "ok": True, - "sequence": replay["sequence"], - "log_authority_chain_head_sha256": replay["head"], - "active_authority": tr["next"], - } - - -def apply_log_authority_transition( # ruff: ignore[undocumented-public-function] - *, - run164_dir: Path, - transparency_root_path: Path, - transparency_root_pin: str, - governance_root_path: Path, - governance_root_pin: str, - recovery_root_path: Path, - recovery_root_pin: str, - transition_path: Path, - handoff_path: Path, - output_dir: Path, - previous_output_dir: Path | None = None, - now: datetime | None = None, -) -> dict[str, Any]: - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - authority_paths = [ - Path(run164_dir), - Path(transparency_root_path), - Path(governance_root_path), - Path(recovery_root_path), - Path(transition_path), - Path(handoff_path), - ] - if previous_output_dir is not None: - authority_paths.append(Path(previous_output_dir)) - # Snapshot authority bytes before parsing anything so a concurrent mutation cannot become the baseline. - before = _authority_fingerprint(authority_paths, "ARCHIVE_LOG_INPUT_DRIFT") - gov_doc, _ = _read_json(Path(governance_root_path), "ARCHIVE_LOG_GOVERNANCE_ROOT") - rec_doc, _ = _read_json(Path(recovery_root_path), "ARCHIVE_LOG_RECOVERY_ROOT") - governance = _verify_control_root( - gov_doc, governance_root_pin, now=current, recovery=False, historical=False - ) - recovery = _verify_control_root( - rec_doc, recovery_root_pin, now=current, recovery=True, historical=False - ) - run164 = _load_run164_offline( - Path(run164_dir), - Path(transparency_root_path), - transparency_root_pin, - now=current, - historical=False, - ) - previous_state = None - previous_state_raw = None - old_events = [] - old_receipts = [] - previous_head = None - if previous_output_dir is not None: - prev_docs, _ = _load_output(Path(previous_output_dir)) - replay = _replay( - governance=governance, - recovery=recovery, - run164=run164, - bundle=prev_docs["release-archive-log-authority-bundle.json"], - receipt=prev_docs["release-archive-log-authority-receipt.json"], - now=current, - ) - previous_state = replay["state"] - previous_state_raw = _canonical(previous_state) - previous_head = replay["head"] - old_events = list( - prev_docs["release-archive-log-authority-bundle.json"]["events"] - ) - old_receipts = list( - prev_docs["release-archive-log-authority-receipt.json"]["events"] - ) - transition_doc, _ = _read_json(Path(transition_path), "ARCHIVE_LOG_TRANSITION") - tr = _verify_transition_document( - transition_doc, - governance=governance, - recovery=recovery, - run164=run164, - previous_state=previous_state, - previous_state_raw=previous_state_raw, - now=current, - creation=True, - ) - handoff_doc, _ = _read_json(Path(handoff_path), "ARCHIVE_LOG_HANDOFF") - _verify_handoff_document(handoff_doc, transition=tr) - _enforce_plane_separation( - governance, recovery, run164["root"], tr["current"], tr["next"] - ) - event_bare = { - "sequence": tr["sequence"], - "kind": tr["kind"], - "transitionSha256": tr["transitionSha256"], - "run164Sequence": tr["signed"]["run164Sequence"], - "merkleConsensusHeadSha256": tr["signed"]["merkleConsensusHeadSha256"], - "run164Artifacts": run164["artifacts"], - "currentAuthoritySha256": _authority_sha(tr["current"]), - "nextAuthoritySha256": _authority_sha(tr["next"]), - "revokedKeyFingerprints": tr["revoked"], - "handoffSubjectSha256s": tr["signed"]["handoffSubjectSha256s"], - } - head = _event_head(previous_head, event_bare) - event = dict(event_bare) - event["logAuthorityChainHeadSha256"] = head - receipt_event = { - "sequence": tr["sequence"], - "transitionDocument": transition_doc, - "handoffProofDocument": handoff_doc, - "run164Documents": run164["docs"], - } - bundle = { - "schemaVersion": int(POLICY["bundle_schema_version"]), - "predicateType": PREDICATE_TYPE, - "status": "archive-log-authority-history", - "governanceRootSha256": governance["sha256"], - "recoveryRootSha256": recovery["sha256"], - "events": [*old_events, event], - } - receipt = { - "schemaVersion": int(POLICY["receipt_schema_version"]), - "status": "archive-log-authority-accepted", - "events": [*old_receipts, receipt_event], - } - bundle_raw = _canonical(bundle) - state_core = { - "schemaVersion": int(POLICY["state_schema_version"]), - "status": "trusted-archive-log-authority", - "sequence": tr["sequence"], - "run164Sequence": tr["signed"]["run164Sequence"], - "merkleConsensusHeadSha256": tr["signed"]["merkleConsensusHeadSha256"], - "run164Artifacts": run164["artifacts"], - "logAuthorityChainHeadSha256": head, - "governanceRootSha256": governance["sha256"], - "recoveryRootSha256": recovery["sha256"], - "activeAuthority": tr["next"], - "revokedKeyFingerprints": tr["revoked"], - "continuityCheckpoints": run164["checkpoints"], - } - state = dict(state_core) - state["bundleArtifact"] = { - "name": "release-archive-log-authority-bundle.json", - "sha256": _sha_bytes(bundle_raw), - "size": len(bundle_raw), - } - active = { - "schemaVersion": int(POLICY["active_schema_version"]), - "status": "active-archive-log-authority", - "sequence": tr["sequence"], - "run164Sequence": tr["signed"]["run164Sequence"], - "merkleConsensusHeadSha256": tr["signed"]["merkleConsensusHeadSha256"], - "logAuthorityChainHeadSha256": head, - "authority": tr["next"], - "revokedKeyFingerprints": tr["revoked"], - "continuityCheckpoints": run164["checkpoints"], - "handoffSubjectSha256s": tr["signed"]["handoffSubjectSha256s"], - } - if before != _authority_fingerprint(authority_paths, "ARCHIVE_LOG_INPUT_DRIFT"): - _fail("ARCHIVE_LOG_INPUT_DRIFT") - protected = authority_paths - target = _outside(Path(output_dir), protected, "ARCHIVE_LOG_OUTPUT_OVERLAPS_INPUT") - if target.exists(): - _fail("ARCHIVE_LOG_OUTPUT_EXISTS") - target.parent.mkdir(parents=True, exist_ok=True) - stage = Path(tempfile.mkdtemp(prefix=".run165-log-authority-", dir=target.parent)) - try: - (stage / "release-archive-log-authority-bundle.json").write_bytes(bundle_raw) - _write(stage / _DOC_ARCHIVE_LOG_AUTHORITY_STATE, state) - _write(stage / "active-archive-log-authority.json", active) - _write(stage / "release-archive-log-authority-receipt.json", receipt) - verify_log_authority_history( - run164_dir=run164_dir, - transparency_root_path=transparency_root_path, - transparency_root_pin=transparency_root_pin, - governance_root_path=governance_root_path, - governance_root_pin=governance_root_pin, - recovery_root_path=recovery_root_path, - recovery_root_pin=recovery_root_pin, - output_dir=stage, - now=current, - ) - stage.rename(target) - except Exception: - shutil.rmtree(stage, ignore_errors=True) - raise - return verify_log_authority_history( - run164_dir=run164_dir, - transparency_root_path=transparency_root_path, - transparency_root_pin=transparency_root_pin, - governance_root_path=governance_root_path, - governance_root_pin=governance_root_pin, - recovery_root_path=recovery_root_path, - recovery_root_pin=recovery_root_pin, - output_dir=target, - now=current, - ) - - -def _common(parser: argparse.ArgumentParser) -> None: - parser.add_argument("--run164-dir", required=True, type=Path) - parser.add_argument("--transparency-root", required=True, type=Path) - parser.add_argument("--transparency-root-pin", required=True) - parser.add_argument("--governance-root", required=True, type=Path) - parser.add_argument("--governance-root-pin", required=True) - parser.add_argument("--recovery-root", required=True, type=Path) - parser.add_argument("--recovery-root-pin", required=True) - parser.add_argument("--output", required=True, type=Path) - - -def main( # ruff: ignore[undocumented-public-function] - argv=None, -) -> int: - parser = argparse.ArgumentParser( - description="Run 165 archive Merkle log-key authority lifecycle" - ) - sub = parser.add_subparsers(dest="command", required=True) - verify = sub.add_parser("verify") - _common(verify) - verify.add_argument("--historical", action="store_true") - apply = sub.add_parser("apply") - _common(apply) - apply.add_argument("--transition", required=True, type=Path) - apply.add_argument("--handoff", required=True, type=Path) - apply.add_argument("--previous-output", type=Path) - args = parser.parse_args(argv) - try: - logger.info("Running archive log authority command: %s", args.command) - if args.command == "verify": - result = verify_log_authority_history( - run164_dir=args.run164_dir, - transparency_root_path=args.transparency_root, - transparency_root_pin=args.transparency_root_pin, - governance_root_path=args.governance_root, - governance_root_pin=args.governance_root_pin, - recovery_root_path=args.recovery_root, - recovery_root_pin=args.recovery_root_pin, - output_dir=args.output, - historical=args.historical, - ) - else: - logger.info("Applying archive log authority handoff: %s", args.handoff) - result = apply_log_authority_transition( - run164_dir=args.run164_dir, - transparency_root_path=args.transparency_root, - transparency_root_pin=args.transparency_root_pin, - governance_root_path=args.governance_root, - governance_root_pin=args.governance_root_pin, - recovery_root_path=args.recovery_root, - recovery_root_pin=args.recovery_root_pin, - transition_path=args.transition, - handoff_path=args.handoff, - output_dir=args.output, - previous_output_dir=args.previous_output, - ) - logger.info("Archive log authority command succeeded: %s", result) - return 0 - except ArchiveLogAuthorityError as exc: - logger.error("Archive log authority command failed: %s", exc) - return 2 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/security/govern_release_history.py b/security/govern_release_history.py deleted file mode 100644 index 90045ca5960dc88542c31cab7330cd6dbb70823f..0000000000000000000000000000000000000000 --- a/security/govern_release_history.py +++ /dev/null @@ -1,2320 +0,0 @@ -""" -Threshold-govern release-history trust roots and recover them from immutable archives. -""" - -from __future__ import annotations - -import argparse -import hashlib -import json -import logging -import os -import re -import shutil -import subprocess -import sys -import tempfile -import threading -from datetime import datetime, timedelta, timezone -from pathlib import Path -from typing import Any, Callable, Iterable -from urllib.parse import urlsplit - -import tomllib - -logger = logging.getLogger(__name__) - -try: # package import - from . import preserve_release_history as history -except (ImportError, ValueError) as exc: # direct Space-style/script/importlib loading - import importlib.util - - _history_path = Path(__file__).resolve().parent / "preserve_release_history.py" - _history_name = "_release_history_for_governance" - if _history_name in sys.modules: - history = sys.modules[_history_name] - else: - _spec = importlib.util.spec_from_file_location(_history_name, _history_path) - if _spec is None or _spec.loader is None: - raise ImportError("cannot load preserve_release_history") from exc - history = importlib.util.module_from_spec(_spec) - sys.modules[_history_name] = history - _spec.loader.exec_module(history) - -HERE = Path(__file__).resolve().parent -POLICY = tomllib.loads((HERE / "release_governance_policy.toml").read_text()) -PREDICATE_TYPE = str(POLICY["predicate_type"]) -_HEX64 = re.compile(r"^[0-9a-f]{64}$") -_ID = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:/@+-]{0,511}$") -_CHUNK = 1024 * 1024 - - -class GovernanceError(RuntimeError): - """Governance or disaster-recovery evidence violated a fail-closed invariant.""" - - -def _fail(code: str) -> None: - raise GovernanceError(code) - - -def _canonical_bytes(value: dict[str, Any]) -> bytes: - return ( - json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False) - + "\n" - ).encode("utf-8") - - -def _sha_bytes(raw: bytes) -> str: - return hashlib.sha256(raw).hexdigest() - - -def _sha(path: Path) -> str: - h = hashlib.sha256() - with path.open("rb") as handle: - for chunk in iter(lambda: handle.read(_CHUNK), b""): - h.update(chunk) - return h.hexdigest() - - -def _write(path: Path, value: dict[str, Any]) -> None: - path.write_bytes(_canonical_bytes(value)) - - -def _loads(raw: bytes, code: str) -> dict[str, Any]: - def hook(pairs): - out: dict[str, Any] = {} - for key, value in pairs: - if key in out: - _fail(code + "_DUPLICATE_KEY") - out[key] = value - return out - - try: - value = json.loads(raw.decode("utf-8"), object_pairs_hook=hook) - except GovernanceError: - raise - except Exception as exc: - raise GovernanceError(code + "_JSON_INVALID") from exc - if not isinstance(value, dict): - _fail(code + "_SCHEMA_INVALID") - return value - - -def _read( - path: Path, code: str, *, canonical: bool = True -) -> tuple[dict[str, Any], bytes]: - if path.is_symlink() or not path.is_file(): - _fail(code + "_INVALID") - if path.stat().st_size > int(POLICY["max_json_bytes"]): - _fail(code + "_TOO_LARGE") - raw = path.read_bytes() - doc = _loads(raw, code) - if canonical and raw != _canonical_bytes(doc): - _fail(code + "_NOT_CANONICAL") - return doc, raw - - -def _regular_dir(path: Path, code: str) -> Path: - if path.is_symlink() or not path.is_dir(): - _fail(code) - return path.resolve() - - -def _outside(path: Path, protected: Iterable[Path], code: str) -> Path: - target = path.expanduser().resolve() - for item in protected: - root = item.resolve() - if target == root or root in target.parents: - _fail(code) - return target - - -def _hex(value: Any, code: str) -> str: - if not isinstance(value, str) or _HEX64.fullmatch(value) is None: - _fail(code) - return value - - -def _identity(value: Any, code: str) -> str: - if not isinstance(value, str): - _fail(code) - value = value.strip() - if ( - not value - or len(value) > 512 # ruff: ignore[magic-value-comparison] - or ".." in value - or "?" in value - or "#" in value - or "\x00" in value - ): - _fail(code) - if ( - any( - ord(ch) < 32 or ord(ch) == 127 # ruff: ignore[magic-value-comparison] - for ch in value # lint - ) - or _ID.fullmatch(value) is None - ): - _fail(code) - return value - - -def _safe_text(value: Any, code: str, *, limit: int = 1024) -> str: - if not isinstance(value, str): - _fail(code) - value = value.strip() - if ( - not value - or len(value) > limit - or "?" in value - or "#" in value - or "\x00" in value - ): - _fail(code) - if any( - ord(ch) < 32 or ord(ch) == 127 # ruff: ignore[magic-value-comparison] - for ch in value # lint - ): - _fail(code) - return value - - -def _safe_locator(value: Any, code: str) -> str: - value = _safe_text(value, code, limit=2048) - try: - parsed = urlsplit(value) - except ValueError: - _fail(code) - if ( - not parsed.scheme - or parsed.scheme.lower() == "file" - or parsed.query - or parsed.fragment - or parsed.username is not None - or parsed.password is not None - ): - _fail(code) - if ".." in parsed.path.split("/"): - _fail(code) - return value - - -def _size(value: Any, code: str) -> int: - if not isinstance(value, int) or isinstance(value, bool) or value < 0: - _fail(code) - return value - - -def _timestamp(value: Any, code: str, *, now: datetime | None = None) -> str: - if not isinstance(value, str) or not value.endswith("Z"): - _fail(code + "_INVALID") - try: - parsed = datetime.fromisoformat(value[:-1] + "+00:00").astimezone(timezone.utc) - except ValueError: - _fail(code + "_INVALID") - if now is not None and parsed > now + timedelta( - minutes=int(POLICY["max_clock_skew_minutes"]) - ): - _fail(code + "_FROM_FUTURE") - return value - - -def _artifact_doc(doc: dict[str, Any], name: str) -> dict[str, Any]: - raw = _canonical_bytes(doc) - return {"name": name, "sha256": _sha_bytes(raw), "size": len(raw)} - - -def _member(value: Any, code: str, *, keyed: bool) -> dict[str, str]: - expected = {"identity", "operator", "keyId"} if keyed else {"identity", "operator"} - if not isinstance(value, dict) or set(value) != expected: - _fail(code + "_SCHEMA_INVALID") - out = { - "identity": _identity(value.get("identity"), code + "_IDENTITY_INVALID"), - "operator": _identity(value.get("operator"), code + "_OPERATOR_INVALID"), - } - if keyed: - out["keyId"] = _identity(value.get("keyId"), code + "_KEY_ID_INVALID") - return out - - -def _membership( - value: Any, - code: str, - *, - keyed: bool, - min_members: int, - max_members: int, - min_threshold: int, - min_operators: int, -) -> dict[str, Any]: - if not isinstance(value, dict) or set(value) != {"threshold", "members"}: - _fail(code + "_SCHEMA_INVALID") - threshold = _size(value.get("threshold"), code + "_THRESHOLD_INVALID") - members_raw = value.get("members") - if not isinstance(members_raw, list) or not ( - min_members <= len(members_raw) <= max_members - ): - _fail(code + "_MEMBER_COUNT_INVALID") - members = [_member(x, code + "_MEMBER", keyed=keyed) for x in members_raw] - identities = [x["identity"] for x in members] - operators = {x["operator"] for x in members} - if len(set(identities)) != len(identities): - _fail(code + "_IDENTITY_DUPLICATE") - if keyed: - keys = [x["keyId"] for x in members] - if len(set(keys)) != len(keys): - _fail(code + "_KEY_DUPLICATE") - if threshold < min_threshold or threshold > len(members): - _fail(code + "_THRESHOLD_INVALID") - if len(operators) < min_operators: - _fail(code + "_OPERATOR_COUNT_INVALID") - members = sorted( - members, key=lambda x: (x["operator"], x["identity"], x.get("keyId", "")) - ) - return {"threshold": threshold, "members": members} - - -def _policy(value: Any, code: str) -> dict[str, Any]: - expected = { - "policyVersion", - "policyAuthority", - "emergencyAuthority", - "historyReplicas", - "historyArchives", - } - if not isinstance(value, dict) or set(value) != expected: - _fail(code + "_SCHEMA_INVALID") - version = _size(value.get("policyVersion"), code + "_VERSION_INVALID") - if version < 1: - _fail(code + "_VERSION_INVALID") - result = { - "policyVersion": version, - "policyAuthority": _membership( - value.get("policyAuthority"), - code + "_POLICY_AUTHORITY", - keyed=True, - min_members=int(POLICY["min_policy_authorities"]), - max_members=int(POLICY["max_policy_authorities"]), - min_threshold=int(POLICY["min_policy_threshold"]), - min_operators=int(POLICY["min_policy_operators"]), - ), - "emergencyAuthority": _membership( - value.get("emergencyAuthority"), - code + "_EMERGENCY_AUTHORITY", - keyed=True, - min_members=int(POLICY["min_emergency_authorities"]), - max_members=int(POLICY["max_emergency_authorities"]), - min_threshold=int(POLICY["min_emergency_threshold"]), - min_operators=int(POLICY["min_emergency_operators"]), - ), - "historyReplicas": _membership( - value.get("historyReplicas"), - code + "_HISTORY_REPLICAS", - keyed=False, - min_members=int(POLICY["min_history_replicas"]), - max_members=int(POLICY["max_history_replicas"]), - min_threshold=int(POLICY["min_history_replica_threshold"]), - min_operators=int(POLICY["min_history_replica_operators"]), - ), - "historyArchives": _membership( - value.get("historyArchives"), - code + "_HISTORY_ARCHIVES", - keyed=False, - min_members=int(POLICY["min_history_archives"]), - max_members=int(POLICY["max_history_archives"]), - min_threshold=int(POLICY["min_history_archive_threshold"]), - min_operators=int(POLICY["min_history_archive_operators"]), - ), - } - policy_ids = {m["identity"] for m in result["policyAuthority"]["members"]} - emergency_ids = {m["identity"] for m in result["emergencyAuthority"]["members"]} - policy_keys = {m["keyId"] for m in result["policyAuthority"]["members"]} - emergency_keys = {m["keyId"] for m in result["emergencyAuthority"]["members"]} - if policy_ids & emergency_ids or policy_keys & emergency_keys: - _fail(code + "_EMERGENCY_AUTHORITY_NOT_DISTINCT") - if bool(POLICY.get("require_emergency_operator_disjoint", False)): - policy_ops = {m["operator"] for m in result["policyAuthority"]["members"]} - emergency_ops = {m["operator"] for m in result["emergencyAuthority"]["members"]} - if policy_ops & emergency_ops: - _fail(code + "_EMERGENCY_OPERATOR_NOT_DISTINCT") - return result - - -def _governance_head( - governance_id: str, genesis: dict[str, Any], entries: list[dict[str, Any]] -) -> str: - head = hashlib.sha256( - _canonical_bytes({"governanceId": governance_id, "genesis": genesis}) - ).digest() - for entry in entries: - head = hashlib.sha256(head + _canonical_bytes(entry)).digest() - return head.hex() - - -# ── Verified-state document names ──────────────────────────────────────── -# -# One constant per release-evidence document. The values are the on-disk -# artifact names and are unchanged; only the repetition is removed. Reading -# a document by a name that says which document it is also keeps static -# analysis from reading the filename token 'trusted' as 'confidential': -# these files carry published Merkle roots and SHA-256 digests, which must -# stay in clear text for any third party to verify them. -_DOC_GOVERNANCE_STATE = "trusted-governance-state.json" -_DOC_HISTORY_STATE = "trusted-history-state.json" - - -def _validate_run153_dir( # ruff: ignore[too-many-branches] - root: Path, -) -> dict[str, Any]: - root = _regular_dir(root, "RUN153_DIRECTORY_INVALID") - allowed = { - "release-history-bundle.json", - _DOC_HISTORY_STATE, - "release-history-preservation-receipt.json", - "previous-release-history-bundle.json", - "previous-trusted-history-state.json", - "release-transparency-witness-record.json", - "release-transparency-witness-receipt.json", - "previous-transparency-checkpoint.json", - "accepted-transparency-checkpoint.json", - "replica-results", - "archive-results", - } - if {p.name for p in root.iterdir()} != allowed: - _fail("RUN153_DIRECTORY_ALLOWLIST_MISMATCH") - for dirname in ("replica-results", "archive-results"): - d = root / dirname - if d.is_symlink() or not d.is_dir(): - _fail("RUN153_EVIDENCE_DIRECTORY_INVALID") - for item in d.iterdir(): - if item.is_symlink() or not item.is_file(): - _fail("RUN153_EVIDENCE_ENTRY_INVALID") - _read(item, "RUN153_EVIDENCE_ENTRY") - try: - info = history._validate_previous_history( - root / _DOC_HISTORY_STATE, root / "release-history-bundle.json" - ) - except history.HistoryError as exc: - raise GovernanceError("RUN153_HISTORY_INVALID:" + str(exc)) from exc - receipt, _ = _read( - root / "release-history-preservation-receipt.json", "RUN153_RECEIPT" - ) - expected_receipt = { - "schemaVersion", - "status", - "historyId", - "sequence", - "collectorIdentity", - "previous", - "current", - "replicaQuorum", - "historyBundle", - "trustedHistoryState", - "archives", - } - if ( - set(receipt) != expected_receipt - or receipt.get("status") != "preserved" - or receipt.get("historyId") != info["history_id"] - or receipt.get("sequence") != info["sequence"] - ): - _fail("RUN153_RECEIPT_SCHEMA_INVALID") - bundle_item = receipt.get("historyBundle") - state_item = receipt.get("trustedHistoryState") - expected_bundle = { - "name": "release-history-bundle.json", - "sha256": info["bundle_sha256"], - "size": (root / "release-history-bundle.json").stat().st_size, - } - expected_state = { - "name": _DOC_HISTORY_STATE, - "sha256": info["state_sha256"], - "size": (root / _DOC_HISTORY_STATE).stat().st_size, - } - if bundle_item != expected_bundle or state_item != expected_state: - _fail("RUN153_RECEIPT_ARTIFACT_REBIND_FAILED") - replica_evidence = ( - receipt.get("replicaQuorum", {}).get("evidence") - if isinstance(receipt.get("replicaQuorum"), dict) - else None - ) - if not isinstance(replica_evidence, list): - _fail("RUN153_REPLICA_EVIDENCE_SCHEMA_INVALID") - replica_hashes = {_sha(p) for p in (root / "replica-results").iterdir()} - for item in replica_evidence: - if ( - not isinstance(item, dict) - or _hex(item.get("sha256"), "RUN153_REPLICA_EVIDENCE_HASH_INVALID") - not in replica_hashes - ): - _fail("RUN153_REPLICA_EVIDENCE_REBIND_FAILED") - archives = receipt.get("archives") - if not isinstance(archives, list): - _fail("RUN153_ARCHIVE_EVIDENCE_SCHEMA_INVALID") - archive_hashes = {_sha(p) for p in (root / "archive-results").iterdir()} - archive_members: list[dict[str, str]] = [] - for item in archives: - if not isinstance(item, dict): - _fail("RUN153_ARCHIVE_EVIDENCE_SCHEMA_INVALID") - for field in ("bindEvidenceSha256", "verifyEvidenceSha256"): - if ( - _hex(item.get(field), "RUN153_ARCHIVE_EVIDENCE_HASH_INVALID") - not in archive_hashes - ): - _fail("RUN153_ARCHIVE_EVIDENCE_REBIND_FAILED") - archive_members.append( - { - "identity": _identity( - item.get("identity"), "RUN153_ARCHIVE_IDENTITY_INVALID" - ), - "operator": _identity( - item.get("operator"), "RUN153_ARCHIVE_OPERATOR_INVALID" - ), - } - ) - last_entry = info["entries"][-1] if info["entries"] else None - replicas = last_entry["replicaQuorum"] if last_entry is not None else None - return { - **info, - "root": root, - "receipt": receipt, - "archive_members": sorted( - archive_members, key=lambda x: (x["operator"], x["identity"]) - ), - "replica_membership": replicas, - } - - -def _validate_genesis( - doc: dict[str, Any], *, history_info: dict[str, Any] -) -> dict[str, Any]: - expected = { - "schemaVersion", - "governanceId", - "historyId", - "bootstrapEvidenceSha256", - "policy", - } - if set(doc) != expected or doc.get("schemaVersion") != int( - POLICY["genesis_schema_version"] - ): - _fail("GOVERNANCE_GENESIS_SCHEMA_INVALID") - governance_id = _safe_text( - doc.get("governanceId"), "GOVERNANCE_ID_INVALID", limit=255 - ) - if doc.get("historyId") != history_info["history_id"]: - _fail("GOVERNANCE_GENESIS_HISTORY_ID_MISMATCH") - _hex(doc.get("bootstrapEvidenceSha256"), "GOVERNANCE_BOOTSTRAP_EVIDENCE_INVALID") - policy = _policy(doc.get("policy"), "GOVERNANCE_GENESIS_POLICY") - return {"governance_id": governance_id, "policy": policy} - - -def _entry_policy(entry: dict[str, Any]) -> dict[str, Any]: - return _policy(entry.get("nextPolicy"), "GOVERNANCE_ENTRY_NEXT_POLICY") - - -def _validate_approval( - doc: dict[str, Any], - *, - governance_id: str, - transition_id: str, - proposal_sha: str, - role: str, - allowed_members: dict[str, dict[str, str]], - revoked: set[str], - now: datetime | None, -) -> dict[str, Any]: - expected = { - "schemaVersion", - "governanceId", - "transitionId", - "proposalSha256", - "approver", - "role", - "decision", - "signatureVerified", - "signatureEvidenceSha256", - "signedAt", - } - if ( - set(doc) != expected - or doc.get("schemaVersion") != int(POLICY["approval_schema_version"]) - or doc.get("governanceId") != governance_id - or doc.get("transitionId") != transition_id - or doc.get("proposalSha256") != proposal_sha - ): - _fail("GOVERNANCE_APPROVAL_SCHEMA_INVALID") - if ( - doc.get("role") != role - or doc.get("decision") != "approve" - or doc.get("signatureVerified") is not True - ): - _fail("GOVERNANCE_APPROVAL_DECISION_INVALID") - approver = _member(doc.get("approver"), "GOVERNANCE_APPROVER", keyed=True) - expected_member = allowed_members.get(approver["keyId"]) - if expected_member != approver: - _fail("GOVERNANCE_APPROVER_NOT_AUTHORIZED") - if approver["keyId"] in revoked: - _fail("GOVERNANCE_REVOKED_AUTHORITY_KEY_REJECTED") - _hex( - doc.get("signatureEvidenceSha256"), - "GOVERNANCE_APPROVAL_SIGNATURE_EVIDENCE_INVALID", - ) - _timestamp(doc.get("signedAt"), "GOVERNANCE_APPROVAL_SIGNED_AT", now=now) - return approver - - -def _history_binding(value: Any, code: str) -> dict[str, Any]: - expected = {"sequence", "stateSha256", "bundleSha256", "chainHeadSha256"} - if not isinstance(value, dict) or set(value) != expected: - _fail(code + "_SCHEMA_INVALID") - return { - "sequence": _size(value.get("sequence"), code + "_SEQUENCE_INVALID"), - "stateSha256": _hex(value.get("stateSha256"), code + "_STATE_HASH_INVALID"), - "bundleSha256": _hex(value.get("bundleSha256"), code + "_BUNDLE_HASH_INVALID"), - "chainHeadSha256": _hex( - value.get("chainHeadSha256"), code + "_CHAIN_HEAD_INVALID" - ), - } - - -def _state_document( - *, - governance_id: str, - history_id: str, - epoch: int, - policy: dict[str, Any], - revoked: list[str], - history_binding: dict[str, Any], - bundle_sha256: str, - chain_head_sha256: str, -) -> dict[str, Any]: - return { - "schemaVersion": int(POLICY["state_schema_version"]), - "governanceId": governance_id, - "historyId": history_id, - "epoch": epoch, - "policy": policy, - "revokedAuthorityKeyIds": revoked, - "history": history_binding, - "bundleSha256": bundle_sha256, - "chainHeadSha256": chain_head_sha256, - } - - -def _validate_bundle( # ruff: ignore[too-many-branches] - bundle: dict[str, Any], -) -> dict[str, Any]: - expected = { - "schemaVersion", - "predicateType", - "status", - "governanceId", - "historyId", - "genesis", - "entries", - } - if ( - set(bundle) != expected - or bundle.get("schemaVersion") != int(POLICY["bundle_schema_version"]) - or bundle.get("predicateType") != PREDICATE_TYPE - or bundle.get("status") != "governed-history" - ): - _fail("GOVERNANCE_BUNDLE_SCHEMA_INVALID") - governance_id = _safe_text( - bundle.get("governanceId"), "GOVERNANCE_ID_INVALID", limit=255 - ) - history_id = _safe_text( - bundle.get("historyId"), "GOVERNANCE_HISTORY_ID_INVALID", limit=255 - ) - genesis = bundle.get("genesis") - if not isinstance(genesis, dict) or set(genesis) != { - "genesisSha256", - "bootstrapEvidenceSha256", - "initialPolicy", - "initialHistory", - }: - _fail("GOVERNANCE_BUNDLE_GENESIS_SCHEMA_INVALID") - _hex(genesis.get("genesisSha256"), "GOVERNANCE_GENESIS_HASH_INVALID") - _hex(genesis.get("bootstrapEvidenceSha256"), "GOVERNANCE_BOOTSTRAP_HASH_INVALID") - current_policy = _policy(genesis.get("initialPolicy"), "GOVERNANCE_INITIAL_POLICY") - current_history = _history_binding( - genesis.get("initialHistory"), "GOVERNANCE_INITIAL_HISTORY" - ) - revoked: set[str] = set() - entries = bundle.get("entries") - _len = len(entries) > 4096 # ruff: ignore[magic-value-comparison] - if not isinstance(entries, list) or _len: - _fail("GOVERNANCE_ENTRY_COUNT_INVALID") - previous_epoch = 0 - transition_ids: set[str] = set() - previous_history_sequence = current_history["sequence"] - for index, entry in enumerate(entries, start=1): - expected_entry = { - "epoch", - "transitionId", - "reason", - "history", - "proposal", - "proposalSha256", - "approvalRole", - "selectedApproverKeyIds", - "approvals", - "revokedAuthorityKeyIds", - "nextPolicy", - } - if not isinstance(entry, dict) or set(entry) != expected_entry: - _fail("GOVERNANCE_ENTRY_SCHEMA_INVALID") - epoch = _size(entry.get("epoch"), "GOVERNANCE_ENTRY_EPOCH_INVALID") - if epoch != previous_epoch + 1: - _fail("GOVERNANCE_ENTRY_EPOCH_GAP") - # Reconstruct the exact prior governance state from the bundle prefix. - prefix_bundle = dict(bundle) - prefix_bundle["entries"] = entries[: index - 1] - prefix_sha = _sha_bytes(_canonical_bytes(prefix_bundle)) - prefix_head = _governance_head(governance_id, genesis, entries[: index - 1]) - prior_state = _state_document( - governance_id=governance_id, - history_id=history_id, - epoch=previous_epoch, - policy=current_policy, - revoked=sorted(revoked), - history_binding=current_history, - bundle_sha256=prefix_sha, - chain_head_sha256=prefix_head, - ) - previous_epoch = epoch - transition_id = _identity( - entry.get("transitionId"), "GOVERNANCE_TRANSITION_ID_INVALID" - ) - if transition_id in transition_ids: - _fail("GOVERNANCE_TRANSITION_REPLAY") - transition_ids.add(transition_id) - reason = entry.get("reason") - if reason not in set(POLICY["allowed_transition_reasons"]): - _fail("GOVERNANCE_REASON_INVALID") - history_binding = _history_binding( - entry.get("history"), "GOVERNANCE_HISTORY_BINDING" - ) - if history_binding["sequence"] < previous_history_sequence: - _fail("GOVERNANCE_HISTORY_ROLLBACK") - previous_history_sequence = history_binding["sequence"] - proposal = entry.get("proposal") - proposal_expected = { - "schemaVersion", - "governanceId", - "transitionId", - "fromEpoch", - "toEpoch", - "reason", - "history", - "previousGovernanceStateSha256", - "selectedApproverKeyIds", - "revokedAuthorityKeyIds", - "nextPolicy", - } - if ( - not isinstance(proposal, dict) - or set(proposal) != proposal_expected - or proposal.get("schemaVersion") != int(POLICY["proposal_schema_version"]) - ): - _fail("GOVERNANCE_ENTRY_PROPOSAL_SCHEMA_INVALID") - proposal_sha = _hex( - entry.get("proposalSha256"), "GOVERNANCE_PROPOSAL_HASH_INVALID" - ) - if _sha_bytes(_canonical_bytes(proposal)) != proposal_sha: - _fail("GOVERNANCE_ENTRY_PROPOSAL_HASH_MISMATCH") - if ( - proposal.get("governanceId") != governance_id - or proposal.get("transitionId") != transition_id - or proposal.get("fromEpoch") != epoch - 1 - or proposal.get("toEpoch") != epoch - or proposal.get("reason") != reason - ): - _fail("GOVERNANCE_ENTRY_PROPOSAL_REBIND_FAILED") - if ( - _history_binding(proposal.get("history"), "GOVERNANCE_PROPOSAL_HISTORY") - != history_binding - ): - _fail("GOVERNANCE_ENTRY_PROPOSAL_HISTORY_MISMATCH") - if proposal.get("previousGovernanceStateSha256") != _sha_bytes( - _canonical_bytes(prior_state) - ): - _fail("GOVERNANCE_ENTRY_PREVIOUS_STATE_HASH_MISMATCH") - role = entry.get("approvalRole") - expected_role = ( - "emergency" if reason == "authority-compromise-recovery" else "policy" - ) - if role != expected_role: - _fail("GOVERNANCE_APPROVAL_ROLE_INVALID") - authority = ( - current_policy["emergencyAuthority"] - if role == "emergency" - else current_policy["policyAuthority"] - ) - members = {m["keyId"]: m for m in authority["members"]} - selected = entry.get("selectedApproverKeyIds") - if ( - not isinstance(selected, list) - or len(selected) < authority["threshold"] - or len(set(selected)) != len(selected) - ): - _fail("GOVERNANCE_SELECTED_APPROVER_QUORUM_INVALID") - selected = [ - _identity(x, "GOVERNANCE_SELECTED_APPROVER_KEY_INVALID") for x in selected - ] - proposal_selected = proposal.get("selectedApproverKeyIds") - if not isinstance(proposal_selected, list) or sorted( - proposal_selected - ) != sorted(selected): - _fail("GOVERNANCE_ENTRY_PROPOSAL_APPROVER_SET_MISMATCH") - if any(x not in members or x in revoked for x in selected): - _fail("GOVERNANCE_SELECTED_APPROVER_NOT_AUTHORIZED") - approvals = entry.get("approvals") - if not isinstance(approvals, list) or len(approvals) != len(selected): - _fail("GOVERNANCE_APPROVAL_COUNT_INVALID") - approval_keys: list[str] = [] - operators: set[str] = set() - for approval in approvals: - approver = _validate_approval( - approval, - governance_id=governance_id, - transition_id=transition_id, - proposal_sha=proposal_sha, - role=role, - allowed_members=members, - revoked=revoked, - now=None, - ) - approval_keys.append(approver["keyId"]) - operators.add(approver["operator"]) - if sorted(approval_keys) != sorted(selected): - _fail("GOVERNANCE_APPROVAL_SET_MISMATCH") - min_ops = int( - POLICY["min_emergency_operators"] - if role == "emergency" - else POLICY["min_policy_operators"] - ) - if len(operators) < min_ops: - _fail("GOVERNANCE_APPROVAL_OPERATOR_QUORUM_INVALID") - revocations = entry.get("revokedAuthorityKeyIds") - if not isinstance(revocations, list) or len(set(revocations)) != len( - revocations - ): - _fail("GOVERNANCE_REVOCATION_SCHEMA_INVALID") - revocations = [ - _identity(x, "GOVERNANCE_REVOCATION_KEY_INVALID") for x in revocations - ] - proposal_revocations = proposal.get("revokedAuthorityKeyIds") - if not isinstance(proposal_revocations, list) or sorted( - proposal_revocations - ) != sorted(revocations): - _fail("GOVERNANCE_ENTRY_PROPOSAL_REVOCATION_MISMATCH") - current_authority_keys = { - m["keyId"] for m in current_policy["policyAuthority"]["members"] - } | {m["keyId"] for m in current_policy["emergencyAuthority"]["members"]} - if any(x not in current_authority_keys or x in revoked for x in revocations): - _fail("GOVERNANCE_REVOCATION_KEY_NOT_CURRENT") - if reason == "authority-compromise-recovery": - current_policy_keys = { - m["keyId"] for m in current_policy["policyAuthority"]["members"] - } - if not set(revocations) & current_policy_keys: - _fail("GOVERNANCE_COMPROMISE_RECOVERY_MUST_REVOKE_POLICY_KEY") - next_policy = _entry_policy(entry) - proposal_next_policy = _policy( - proposal.get("nextPolicy"), "GOVERNANCE_PROPOSAL_NEXT_POLICY" - ) - if proposal_next_policy != next_policy: - _fail("GOVERNANCE_ENTRY_PROPOSAL_NEXT_POLICY_MISMATCH") - if next_policy["policyVersion"] != current_policy["policyVersion"] + 1: - _fail("GOVERNANCE_POLICY_VERSION_NOT_ADVANCING") - next_keys = {m["keyId"] for m in next_policy["policyAuthority"]["members"]} | { - m["keyId"] for m in next_policy["emergencyAuthority"]["members"] - } - if set(revocations) & next_keys: - _fail("GOVERNANCE_REVOKED_KEY_IN_NEXT_POLICY") - if revoked & next_keys: - _fail("GOVERNANCE_PREVIOUSLY_REVOKED_KEY_REUSED") - revoked.update(revocations) - current_policy = next_policy - current_history = history_binding - return { - "governance_id": governance_id, - "history_id": history_id, - "epoch": len(entries), - "policy": current_policy, - "history": current_history, - "revoked_authority_key_ids": sorted(revoked), - "chain_head_sha256": _governance_head(governance_id, genesis, entries), - "entries": entries, - "genesis": genesis, - } - - -def verify_governance_bundle( # ruff: ignore[undocumented-public-function] - *, bundle_path: Path, state_path: Path | None = None -) -> dict[str, Any]: - bundle, bundle_raw = _read(bundle_path, "GOVERNANCE_BUNDLE") - info = _validate_bundle(bundle) - if state_path is not None: - state, state_raw = _read(state_path, "GOVERNANCE_STATE") - expected = { - "schemaVersion", - "governanceId", - "historyId", - "epoch", - "policy", - "revokedAuthorityKeyIds", - "history", - "bundleSha256", - "chainHeadSha256", - } - if set(state) != expected or state.get("schemaVersion") != int( - POLICY["state_schema_version"] - ): - _fail("GOVERNANCE_STATE_SCHEMA_INVALID") - if ( - state.get("governanceId") != info["governance_id"] - or state.get("historyId") != info["history_id"] - or state.get("epoch") != info["epoch"] - ): - _fail("GOVERNANCE_STATE_ID_MISMATCH") - if ( - _policy(state.get("policy"), "GOVERNANCE_STATE_POLICY") != info["policy"] - or state.get("revokedAuthorityKeyIds") != info["revoked_authority_key_ids"] - ): - _fail("GOVERNANCE_STATE_POLICY_MISMATCH") - hist = _history_binding(state.get("history"), "GOVERNANCE_STATE_HISTORY") - if hist != info["history"]: - _fail("GOVERNANCE_STATE_HISTORY_MISMATCH") - if ( - state.get("bundleSha256") != _sha_bytes(bundle_raw) - or state.get("chainHeadSha256") != info["chain_head_sha256"] - ): - _fail("GOVERNANCE_STATE_HASH_MISMATCH") - if _sha_bytes(state_raw) == state.get("bundleSha256"): - _fail("GOVERNANCE_STATE_SELF_BIND_INVALID") - return { - "ok": True, - "governance_id": info["governance_id"], - "history_id": info["history_id"], - "epoch": info["epoch"], - "policy_version": info["policy"]["policyVersion"], - "bundle_sha256": _sha_bytes(bundle_raw), - "chain_head_sha256": info["chain_head_sha256"], - } - - -def _state_for( - *, info: dict[str, Any], history_info: dict[str, Any], bundle_sha: str -) -> dict[str, Any]: - history_binding = { - "sequence": history_info["sequence"], - "stateSha256": history_info["state_sha256"], - "bundleSha256": history_info["bundle_sha256"], - "chainHeadSha256": history_info["chain_head_sha256"], - } - return _state_document( - governance_id=info["governance_id"], - history_id=info["history_id"], - epoch=info["epoch"], - policy=info["policy"], - revoked=info["revoked_authority_key_ids"], - history_binding=history_binding, - bundle_sha256=bundle_sha, - chain_head_sha256=info["chain_head_sha256"], - ) - - -def initialize_governance( # ruff: ignore[undocumented-public-function] - *, - history_dir: Path, - genesis_path: Path, - expected_genesis_sha256: str, - output_dir: Path, -) -> dict[str, Any]: - current = _validate_run153_dir(history_dir) - target = _outside(output_dir, (current["root"],), "GOVERNANCE_OUTPUT_INSIDE_INPUT") - if target.exists() or target.is_symlink(): - _fail("GOVERNANCE_OUTPUT_ALREADY_EXISTS") - expected_genesis_sha256 = _hex( - expected_genesis_sha256, "GOVERNANCE_EXPECTED_GENESIS_HASH_INVALID" - ) - genesis, genesis_raw = _read(genesis_path, "GOVERNANCE_GENESIS") - if _sha_bytes(genesis_raw) != expected_genesis_sha256: - _fail("GOVERNANCE_GENESIS_PIN_MISMATCH") - g = _validate_genesis(genesis, history_info=current) - init_inputs = [p for p in current["root"].rglob("*") if p.is_file()] + [ - genesis_path - ] - init_hashes = {str(p.resolve()): _sha(p) for p in init_inputs} - policy = g["policy"] - current_replicas = current["replica_membership"] - if not isinstance(current_replicas, dict) or set(current_replicas) != { - "threshold", - "minimumOperators", - "replicas", - }: - _fail("GOVERNANCE_RUN153_REPLICA_MEMBERSHIP_MISSING") - normalized_current_replicas = _membership( - { - "threshold": current_replicas["threshold"], - "members": current_replicas["replicas"], - }, - "GOVERNANCE_RUN153_REPLICAS", - keyed=False, - min_members=int(POLICY["min_history_replicas"]), - max_members=int(POLICY["max_history_replicas"]), - min_threshold=int(POLICY["min_history_replica_threshold"]), - min_operators=int(POLICY["min_history_replica_operators"]), - ) - if policy["historyReplicas"] != normalized_current_replicas: - _fail("GOVERNANCE_GENESIS_REPLICA_MEMBERSHIP_MISMATCH") - expected_archives = { - tuple(sorted(x.items())) for x in policy["historyArchives"]["members"] - } - current_archives = {tuple(sorted(x.items())) for x in current["archive_members"]} - if ( - current_archives != expected_archives - or len(current_archives) < policy["historyArchives"]["threshold"] - ): - _fail("GOVERNANCE_GENESIS_ARCHIVE_MEMBERSHIP_MISMATCH") - initial_history = { - "sequence": current["sequence"], - "stateSha256": current["state_sha256"], - "bundleSha256": current["bundle_sha256"], - "chainHeadSha256": current["chain_head_sha256"], - } - bundle = { - "schemaVersion": int(POLICY["bundle_schema_version"]), - "predicateType": PREDICATE_TYPE, - "status": "governed-history", - "governanceId": g["governance_id"], - "historyId": current["history_id"], - "genesis": { - "genesisSha256": expected_genesis_sha256, - "bootstrapEvidenceSha256": genesis["bootstrapEvidenceSha256"], - "initialPolicy": policy, - "initialHistory": initial_history, - }, - "entries": [], - } - info = _validate_bundle(bundle) - bundle_raw = _canonical_bytes(bundle) - bundle_sha = _sha_bytes(bundle_raw) - state = _state_for(info=info, history_info=current, bundle_sha=bundle_sha) - target.parent.mkdir(parents=True, exist_ok=True) - with tempfile.TemporaryDirectory( - prefix="release-governance-init-", dir=target.parent - ) as tmp: - stage = Path(tmp) / "governance" - stage.mkdir() - _write(stage / "release-governance-bundle.json", bundle) - _write(stage / _DOC_GOVERNANCE_STATE, state) - verify_governance_bundle( - bundle_path=stage / "release-governance-bundle.json", - state_path=stage / _DOC_GOVERNANCE_STATE, - ) - shutil.copy2(genesis_path, stage / "governance-genesis.json") - for input_path in init_inputs: - if _sha(input_path) != init_hashes[str(input_path.resolve())]: - _fail("GOVERNANCE_INITIALIZATION_INPUT_CHANGED") - shutil.copy2( - current["root"] / _DOC_HISTORY_STATE, - stage / _DOC_HISTORY_STATE, - ) - shutil.copy2( - current["root"] / "release-history-bundle.json", - stage / "release-history-bundle.json", - ) - os.replace(stage, target) - return { - "ok": True, - "phase": "governance-initialized", - "governance_id": g["governance_id"], - "epoch": 0, - "policy_version": policy["policyVersion"], - "bundle_sha256": bundle_sha, - "state_sha256": _sha(target / _DOC_GOVERNANCE_STATE), - } - - -def _validate_previous_governance( - state_path: Path, bundle_path: Path -) -> dict[str, Any]: - result = verify_governance_bundle(bundle_path=bundle_path, state_path=state_path) - state, state_raw = _read(state_path, "PREVIOUS_GOVERNANCE_STATE") - bundle, bundle_raw = _read(bundle_path, "PREVIOUS_GOVERNANCE_BUNDLE") - info = _validate_bundle(bundle) - return { - **result, - **info, - "state": state, - "state_sha256": _sha_bytes(state_raw), - "bundle": bundle, - "bundle_sha256": _sha_bytes(bundle_raw), - "history": state["history"], - } - - -def _validate_proposal( # ruff: ignore[too-many-branches] - doc: dict[str, Any], - *, - previous: dict[str, Any], - current: dict[str, Any], -) -> dict[str, Any]: - expected = { - "schemaVersion", - "governanceId", - "transitionId", - "fromEpoch", - "toEpoch", - "reason", - "history", - "previousGovernanceStateSha256", - "selectedApproverKeyIds", - "revokedAuthorityKeyIds", - "nextPolicy", - } - if ( - set(doc) != expected - or doc.get("schemaVersion") != int(POLICY["proposal_schema_version"]) - or doc.get("governanceId") != previous["governance_id"] - ): - _fail("GOVERNANCE_PROPOSAL_SCHEMA_INVALID") - transition_id = _identity( - doc.get("transitionId"), "GOVERNANCE_TRANSITION_ID_INVALID" - ) - if ( - _size(doc.get("fromEpoch"), "GOVERNANCE_FROM_EPOCH_INVALID") - != previous["epoch"] - or _size(doc.get("toEpoch"), "GOVERNANCE_TO_EPOCH_INVALID") - != previous["epoch"] + 1 - ): - _fail("GOVERNANCE_PROPOSAL_EPOCH_MISMATCH") - reason = doc.get("reason") - if reason not in set(POLICY["allowed_transition_reasons"]): - _fail("GOVERNANCE_REASON_INVALID") - if doc.get("previousGovernanceStateSha256") != previous["state_sha256"]: - _fail("GOVERNANCE_PROPOSAL_PREVIOUS_STATE_MISMATCH") - history_binding = doc.get("history") - expected_history = { - "sequence": current["sequence"], - "stateSha256": current["state_sha256"], - "bundleSha256": current["bundle_sha256"], - "chainHeadSha256": current["chain_head_sha256"], - } - if history_binding != expected_history: - _fail("GOVERNANCE_PROPOSAL_HISTORY_MISMATCH") - selected = doc.get("selectedApproverKeyIds") - if not isinstance(selected, list) or len(set(selected)) != len(selected): - _fail("GOVERNANCE_SELECTED_APPROVER_SET_INVALID") - selected = [ - _identity(x, "GOVERNANCE_SELECTED_APPROVER_KEY_INVALID") for x in selected - ] - role = "emergency" if reason == "authority-compromise-recovery" else "policy" - authority = ( - previous["policy"]["emergencyAuthority"] - if role == "emergency" - else previous["policy"]["policyAuthority"] - ) - allowed = {x["keyId"]: x for x in authority["members"]} - if len(selected) < authority["threshold"] or any( - x not in allowed or x in set(previous["revoked_authority_key_ids"]) - for x in selected - ): - _fail("GOVERNANCE_SELECTED_APPROVER_QUORUM_INVALID") - revocations = doc.get("revokedAuthorityKeyIds") - if not isinstance(revocations, list) or len(set(revocations)) != len(revocations): - _fail("GOVERNANCE_REVOCATION_SCHEMA_INVALID") - revocations = [ - _identity(x, "GOVERNANCE_REVOCATION_KEY_INVALID") for x in revocations - ] - current_authority_keys = { - m["keyId"] for m in previous["policy"]["policyAuthority"]["members"] - } | {m["keyId"] for m in previous["policy"]["emergencyAuthority"]["members"]} - if any( - x not in current_authority_keys - or x in set(previous["revoked_authority_key_ids"]) - for x in revocations - ): - _fail("GOVERNANCE_REVOCATION_KEY_NOT_CURRENT") - if reason == "authority-compromise-recovery": - current_policy_keys = { - m["keyId"] for m in previous["policy"]["policyAuthority"]["members"] - } - if not set(revocations) & current_policy_keys: - _fail("GOVERNANCE_COMPROMISE_RECOVERY_MUST_REVOKE_POLICY_KEY") - next_policy = _policy(doc.get("nextPolicy"), "GOVERNANCE_PROPOSAL_NEXT_POLICY") - if next_policy["policyVersion"] != previous["policy"]["policyVersion"] + 1: - _fail("GOVERNANCE_POLICY_VERSION_NOT_ADVANCING") - next_keys = {m["keyId"] for m in next_policy["policyAuthority"]["members"]} | { - m["keyId"] for m in next_policy["emergencyAuthority"]["members"] - } - if (set(revocations) | set(previous["revoked_authority_key_ids"])) & next_keys: - _fail("GOVERNANCE_REVOKED_KEY_IN_NEXT_POLICY") - return { - "transition_id": transition_id, - "reason": reason, - "role": role, - "selected": selected, - "allowed": allowed, - "revocations": revocations, - "next_policy": next_policy, - "history": expected_history, - } - - -def _archive_id( - governance_id: str, epoch: int, snapshot_sha: str, identity: str -) -> str: - return ( - "governance-archive-" - + hashlib.sha256( - ( - governance_id - + "\0" - + str(epoch) - + "\0" - + snapshot_sha - + "\0" - + identity - ).encode() - ).hexdigest()[:32] - ) - - -def _recovery_id(history_id: str, sequence: int, identity: str) -> str: - return ( - "history-recovery-" - + hashlib.sha256( - (history_id + "\0" + str(sequence) + "\0" + identity).encode() - ).hexdigest()[:32] - ) - - -def _command_adapter( - command: list[str], *, prefix: str -) -> Callable[[dict[str, Any]], dict[str, Any]]: - if not command: - _fail(prefix + "_COMMAND_INVALID") - - def call(request: dict[str, Any]) -> dict[str, Any]: - process = ( - # lint - subprocess.Popen( # ruff: ignore[subprocess-without-shell-equals-true] - command, - stdin=subprocess.PIPE, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, - text=False, - close_fds=True, - env={"PATH": os.defpath, "LC_ALL": "C", "LANG": "C"}, - ) - ) - assert process.stdin is not None # ruff: ignore[assert] - assert process.stdout is not None # ruff: ignore[assert] - assert process.stderr is not None # ruff: ignore[assert] - stdin = process.stdin - process.stdin = None - chunks: list[bytes] = [] - err_chunks: list[bytes] = [] - overflow = threading.Event() - limit = int(POLICY["max_adapter_output_bytes"]) - - def drain(stream, sink): - total = 0 - while True: - chunk = stream.read(65536) - if not chunk: - break - total += len(chunk) - if total > limit: - overflow.set() - process.kill() - break - sink.append(chunk) - - threads = [ - threading.Thread(target=drain, args=(process.stdout, chunks)), - threading.Thread(target=drain, args=(process.stderr, err_chunks)), - ] - for t in threads: - t.start() - try: - stdin.write(_canonical_bytes(request)) - stdin.close() - process.wait(timeout=int(POLICY["adapter_timeout_seconds"])) - except subprocess.TimeoutExpired as exc: - process.kill() - raise GovernanceError(prefix + "_TIMEOUT") from exc - finally: - for t in threads: - t.join() - if overflow.is_set(): - _fail(prefix + "_OUTPUT_TOO_LARGE") - if process.returncode != 0: - _fail(prefix + "_FAILED") - return _loads(b"".join(chunks), prefix + "_OUTPUT") - - return call - - -def command_recovery_source( # ruff: ignore[undocumented-public-function] - command: list[str], -): - return _command_adapter(command, prefix="GOVERNANCE_RECOVERY_ADAPTER") - - -def command_archive( # ruff: ignore[undocumented-public-function] - command: list[str], -): - return _command_adapter(command, prefix="GOVERNANCE_ARCHIVE_ADAPTER") - - -RecoveryAdapter = Callable[[dict[str, Any]], dict[str, Any]] -ArchiveAdapter = Callable[[dict[str, Any]], dict[str, Any]] - - -def _validate_recovery_response( - value: dict[str, Any], - *, - recovery_id: str, - history_id: str, - identity: str, - operator: str, - now: datetime, -) -> dict[str, Any]: - base = { - "schemaVersion", - "operation", - "recoveryId", - "historyId", - "status", - "source", - "observedAt", - } - if ( - value.get("schemaVersion") != int(POLICY["recovery_protocol_version"]) - or value.get("operation") != "recover-history" - or value.get("recoveryId") != recovery_id - or value.get("historyId") != history_id - ): - _fail("GOVERNANCE_RECOVERY_RESULT_ID_MISMATCH") - source = value.get("source") - source_keys = { - "identity", - "operator", - "readOnly", - "historyWriterCredentialsReused", - "governanceWriterCredentialsReused", - } - if ( - not isinstance(source, dict) - or set(source) != source_keys - or source.get("identity") != identity - or source.get("operator") != operator - or source.get("readOnly") is not True - or source.get("historyWriterCredentialsReused") is not False - or source.get("governanceWriterCredentialsReused") is not False - ): - _fail("GOVERNANCE_RECOVERY_SOURCE_AUTHORITY_INVALID") - _timestamp(value.get("observedAt"), "GOVERNANCE_RECOVERY_OBSERVED_AT", now=now) - if value.get("status") == "unavailable": - if set(value) != base | {"reason"} or not bool( - POLICY["allow_unavailable_recovery_source"] - ): - _fail("GOVERNANCE_RECOVERY_UNAVAILABLE_INVALID") - return {"status": "unavailable", "identity": identity, "operator": operator} - if value.get("status") != "recovered" or set(value) != base | { - "snapshot", - "locator", - "proof", - }: - _fail("GOVERNANCE_RECOVERY_RESULT_SCHEMA_INVALID") - locator = _safe_locator(value.get("locator"), "GOVERNANCE_RECOVERY_LOCATOR_INVALID") - proof = value.get("proof") - if proof != { - "immutableArchiveReadbackVerified": True, - "independentArchive": True, - "canonicalBytesReturned": True, - }: - _fail("GOVERNANCE_RECOVERY_PROOF_INVALID") - snapshot = value.get("snapshot") - expected_snapshot = {"schemaVersion", "historyState", "historyBundle"} - if ( - not isinstance(snapshot, dict) - or set(snapshot) != expected_snapshot - or snapshot.get("schemaVersion") - != int(POLICY["recovery_snapshot_schema_version"]) - ): - _fail("GOVERNANCE_RECOVERY_SNAPSHOT_SCHEMA_INVALID") - state = snapshot.get("historyState") - bundle = snapshot.get("historyBundle") - if not isinstance(state, dict) or not isinstance(bundle, dict): - _fail("GOVERNANCE_RECOVERY_SNAPSHOT_SCHEMA_INVALID") - state_raw = _canonical_bytes(state) - bundle_raw = _canonical_bytes(bundle) - with tempfile.TemporaryDirectory(prefix="governance-recover-validate-") as tmp: - sp = Path(tmp) / "state.json" - bp = Path(tmp) / "bundle.json" - sp.write_bytes(state_raw) - bp.write_bytes(bundle_raw) - try: - info = history._validate_previous_history(sp, bp) - except history.HistoryError as exc: - raise GovernanceError( - "GOVERNANCE_RECOVERY_HISTORY_INVALID:" + str(exc) - ) from exc - if info["history_id"] != history_id: - _fail("GOVERNANCE_RECOVERY_HISTORY_ID_MISMATCH") - return { - "status": "recovered", - "identity": identity, - "operator": operator, - "locator": locator, - "state": state, - "bundle": bundle, - "state_sha256": _sha_bytes(state_raw), - "bundle_sha256": _sha_bytes(bundle_raw), - "sequence": info["sequence"], - "chain_head_sha256": info["chain_head_sha256"], - } - - -def recover_history( # ruff: ignore[too-many-branches, undocumented-public-function] - *, - history_id: str, - expected_sequence: int, - expected_chain_head_sha256: str, - output_dir: Path, - recovery_sources: list[tuple[str, str, RecoveryAdapter]], - recovery_quorum: int, - now: datetime | None = None, -) -> dict[str, Any]: - history_id = _safe_text( - history_id, "GOVERNANCE_RECOVERY_HISTORY_ID_INVALID", limit=255 - ) - expected_sequence = _size(expected_sequence, "GOVERNANCE_RECOVERY_SEQUENCE_INVALID") - expected_chain_head_sha256 = _hex( - expected_chain_head_sha256, "GOVERNANCE_RECOVERY_EXPECTED_CHAIN_HEAD_INVALID" - ) - target = output_dir.expanduser().resolve() - if target.exists() or target.is_symlink(): - _fail("GOVERNANCE_RECOVERY_OUTPUT_ALREADY_EXISTS") - if not ( - int(POLICY["min_recovery_sources"]) - <= len(recovery_sources) - <= int(POLICY["max_recovery_sources"]) - ): - _fail("GOVERNANCE_RECOVERY_SOURCE_COUNT_INVALID") - recovery_quorum = _size(recovery_quorum, "GOVERNANCE_RECOVERY_QUORUM_INVALID") - if recovery_quorum < int(POLICY["min_recovery_quorum"]) or recovery_quorum > len( - recovery_sources - ): - _fail("GOVERNANCE_RECOVERY_QUORUM_INVALID") - normalized = [] - ids = set() - ops = set() - for ir, oraw, adapter in recovery_sources: - i = _identity(ir, "GOVERNANCE_RECOVERY_IDENTITY_INVALID") - o = _identity(oraw, "GOVERNANCE_RECOVERY_OPERATOR_INVALID") - if i in ids: - _fail("GOVERNANCE_RECOVERY_IDENTITY_DUPLICATE") - ids.add(i) - ops.add(o) - normalized.append((i, o, adapter)) - if len(ops) < int(POLICY["min_recovery_operators"]): - _fail("GOVERNANCE_RECOVERY_OPERATOR_COUNT_INVALID") - current_time = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - observed = [] - unavailable = [] - evidence = [] - locators = set() - for i, o, adapter in sorted(normalized, key=lambda x: (x[1], x[0])): - rid = _recovery_id(history_id, expected_sequence, i) - request = { - "schemaVersion": int(POLICY["recovery_protocol_version"]), - "operation": "recover-history", - "recoveryId": rid, - "historyId": history_id, - "expectedSequence": expected_sequence, - "source": {"identity": i, "operator": o}, - } - raw = adapter(request) - result = _validate_recovery_response( - raw, - recovery_id=rid, - history_id=history_id, - identity=i, - operator=o, - now=current_time, - ) - evidence.append( - { - "identity": i, - "operator": o, - "status": result["status"], - "sha256": _sha_bytes(_canonical_bytes(raw)), - } - ) - if result["status"] == "recovered": - if result["sequence"] != expected_sequence: - _fail("GOVERNANCE_RECOVERY_SEQUENCE_MISMATCH") - if result["locator"] in locators: - _fail("GOVERNANCE_RECOVERY_LOCATOR_COLLISION") - locators.add(result["locator"]) - observed.append(result) - else: - unavailable.append(result) - if len(observed) < recovery_quorum: - _fail("GOVERNANCE_RECOVERY_QUORUM_NOT_MET") - if len({x["operator"] for x in observed}) < int(POLICY["min_recovery_operators"]): - _fail("GOVERNANCE_RECOVERY_OPERATOR_QUORUM_NOT_MET") - views = { - (x["state_sha256"], x["bundle_sha256"], x["chain_head_sha256"]) - for x in observed - } - if len(views) != 1: - _fail("GOVERNANCE_RECOVERY_ARCHIVE_DISAGREEMENT") - chosen = min(observed, key=lambda x: (x["operator"], x["identity"])) - if chosen["chain_head_sha256"] != expected_chain_head_sha256: - _fail("GOVERNANCE_RECOVERY_CHAIN_HEAD_PIN_MISMATCH") - target.parent.mkdir(parents=True, exist_ok=True) - with tempfile.TemporaryDirectory( - prefix="release-recovery-", dir=target.parent - ) as tmp: - stage = Path(tmp) / "recovery" - stage.mkdir() - _write(stage / "recovered-trusted-history-state.json", chosen["state"]) - _write(stage / "recovered-release-history-bundle.json", chosen["bundle"]) - receipt = { - "schemaVersion": int(POLICY["receipt_schema_version"]), - "status": "recovered", - "historyId": history_id, - "sequence": expected_sequence, - "expectedChainHeadSha256": expected_chain_head_sha256, - "quorum": { - "configured": len(normalized), - "threshold": recovery_quorum, - "observed": len(observed), - "unavailable": len(unavailable), - "minimumOperators": int(POLICY["min_recovery_operators"]), - }, - "recovered": { - "stateSha256": chosen["state_sha256"], - "bundleSha256": chosen["bundle_sha256"], - "chainHeadSha256": chosen["chain_head_sha256"], - }, - "evidence": evidence, - } - _write(stage / "release-history-recovery-receipt.json", receipt) - os.replace(stage, target) - return { - "ok": True, - "phase": "history-recovered", - "history_id": history_id, - "sequence": expected_sequence, - "observed": len(observed), - "quorum": recovery_quorum, - "state_sha256": chosen["state_sha256"], - "bundle_sha256": chosen["bundle_sha256"], - "chain_head_sha256": chosen["chain_head_sha256"], - } - - -def _governance_recovery_id(governance_id: str, epoch: int, identity: str) -> str: - return ( - "governance-recovery-" - + hashlib.sha256( - (governance_id + "\0" + str(epoch) + "\0" + identity).encode() - ).hexdigest()[:32] - ) - - -def _validate_governance_recovery_response( - value: dict[str, Any], - *, - recovery_id: str, - governance_id: str, - identity: str, - operator: str, - now: datetime, -) -> dict[str, Any]: - base = { - "schemaVersion", - "operation", - "recoveryId", - "governanceId", - "status", - "source", - "observedAt", - } - if ( - value.get("schemaVersion") != int(POLICY["recovery_protocol_version"]) - or value.get("operation") != "recover-governance" - or value.get("recoveryId") != recovery_id - or value.get("governanceId") != governance_id - ): - _fail("GOVERNANCE_SNAPSHOT_RECOVERY_RESULT_ID_MISMATCH") - source = value.get("source") - source_keys = { - "identity", - "operator", - "readOnly", - "historyWriterCredentialsReused", - "governanceWriterCredentialsReused", - } - if ( - not isinstance(source, dict) - or set(source) != source_keys - or source.get("identity") != identity - or source.get("operator") != operator - ): - _fail("GOVERNANCE_SNAPSHOT_RECOVERY_SOURCE_SCHEMA_INVALID") - if ( - source.get("readOnly") is not True - or source.get("historyWriterCredentialsReused") is not False - or source.get("governanceWriterCredentialsReused") is not False - ): - _fail("GOVERNANCE_SNAPSHOT_RECOVERY_SOURCE_AUTHORITY_INVALID") - _timestamp( - value.get("observedAt"), "GOVERNANCE_SNAPSHOT_RECOVERY_OBSERVED_AT", now=now - ) - if value.get("status") == "unavailable": - if set(value) != base | {"reason"} or not bool( - POLICY["allow_unavailable_recovery_source"] - ): - _fail("GOVERNANCE_SNAPSHOT_RECOVERY_UNAVAILABLE_INVALID") - return {"status": "unavailable", "identity": identity, "operator": operator} - if value.get("status") != "recovered" or set(value) != base | { - "snapshot", - "locator", - "proof", - }: - _fail("GOVERNANCE_SNAPSHOT_RECOVERY_RESULT_SCHEMA_INVALID") - locator = _safe_locator( - value.get("locator"), "GOVERNANCE_SNAPSHOT_RECOVERY_LOCATOR_INVALID" - ) - if value.get("proof") != { - "immutableArchiveReadbackVerified": True, - "independentArchive": True, - "canonicalBytesReturned": True, - }: - _fail("GOVERNANCE_SNAPSHOT_RECOVERY_PROOF_INVALID") - snapshot = value.get("snapshot") - expected = { - "schemaVersion", - "governanceState", - "governanceBundle", - "historyState", - "historyBundle", - } - if ( - not isinstance(snapshot, dict) - or set(snapshot) != expected - or snapshot.get("schemaVersion") - != int(POLICY["recovery_snapshot_schema_version"]) - ): - _fail("GOVERNANCE_SNAPSHOT_RECOVERY_SCHEMA_INVALID") - gstate = snapshot.get("governanceState") - gbundle = snapshot.get("governanceBundle") - hstate = snapshot.get("historyState") - hbundle = snapshot.get("historyBundle") - if not all(isinstance(x, dict) for x in (gstate, gbundle, hstate, hbundle)): - _fail("GOVERNANCE_SNAPSHOT_RECOVERY_SCHEMA_INVALID") - gstate_raw = _canonical_bytes(gstate) - gbundle_raw = _canonical_bytes(gbundle) - hstate_raw = _canonical_bytes(hstate) - hbundle_raw = _canonical_bytes(hbundle) - with tempfile.TemporaryDirectory(prefix="governance-snapshot-validate-") as tmp: - root = Path(tmp) - gsp = root / "governance-state.json" - gbp = root / "governance-bundle.json" - hsp = root / "history-state.json" - hbp = root / "history-bundle.json" - gsp.write_bytes(gstate_raw) - gbp.write_bytes(gbundle_raw) - hsp.write_bytes(hstate_raw) - hbp.write_bytes(hbundle_raw) - ginfo = verify_governance_bundle(bundle_path=gbp, state_path=gsp) - try: - hinfo = history._validate_previous_history(hsp, hbp) - except history.HistoryError as exc: - raise GovernanceError( - "GOVERNANCE_SNAPSHOT_RECOVERY_HISTORY_INVALID:" + str(exc) - ) from exc - if ( - ginfo["governance_id"] != governance_id - or gstate.get("historyId") != hinfo["history_id"] - ): - _fail("GOVERNANCE_SNAPSHOT_RECOVERY_ID_MISMATCH") - expected_history = { - "sequence": hinfo["sequence"], - "stateSha256": hinfo["state_sha256"], - "bundleSha256": hinfo["bundle_sha256"], - "chainHeadSha256": hinfo["chain_head_sha256"], - } - if gstate.get("history") != expected_history: - _fail("GOVERNANCE_SNAPSHOT_RECOVERY_HISTORY_REBIND_FAILED") - return { - "status": "recovered", - "identity": identity, - "operator": operator, - "locator": locator, - "snapshot": snapshot, - "snapshot_sha256": _sha_bytes(_canonical_bytes(snapshot)), - "governance_state_sha256": _sha_bytes(gstate_raw), - "governance_bundle_sha256": _sha_bytes(gbundle_raw), - "governance_chain_head_sha256": ginfo["chain_head_sha256"], - "epoch": ginfo["epoch"], - "history_state_sha256": hinfo["state_sha256"], - "history_bundle_sha256": hinfo["bundle_sha256"], - "history_chain_head_sha256": hinfo["chain_head_sha256"], - "history_sequence": hinfo["sequence"], - } - - -def recover_governance( # ruff: ignore[too-many-branches, undocumented-public-function] - *, - governance_id: str, - expected_epoch: int, - expected_chain_head_sha256: str, - output_dir: Path, - recovery_sources: list[tuple[str, str, RecoveryAdapter]], - recovery_quorum: int, - now: datetime | None = None, -) -> dict[str, Any]: - governance_id = _safe_text( - governance_id, "GOVERNANCE_SNAPSHOT_RECOVERY_ID_INVALID", limit=255 - ) - expected_epoch = _size(expected_epoch, "GOVERNANCE_SNAPSHOT_RECOVERY_EPOCH_INVALID") - expected_chain_head_sha256 = _hex( - expected_chain_head_sha256, - "GOVERNANCE_SNAPSHOT_RECOVERY_EXPECTED_CHAIN_HEAD_INVALID", - ) - target = output_dir.expanduser().resolve() - if target.exists() or target.is_symlink(): - _fail("GOVERNANCE_SNAPSHOT_RECOVERY_OUTPUT_ALREADY_EXISTS") - if not ( - int(POLICY["min_recovery_sources"]) - <= len(recovery_sources) - <= int(POLICY["max_recovery_sources"]) - ): - _fail("GOVERNANCE_SNAPSHOT_RECOVERY_SOURCE_COUNT_INVALID") - recovery_quorum = _size( - recovery_quorum, "GOVERNANCE_SNAPSHOT_RECOVERY_QUORUM_INVALID" - ) - if recovery_quorum < int(POLICY["min_recovery_quorum"]) or recovery_quorum > len( - recovery_sources - ): - _fail("GOVERNANCE_SNAPSHOT_RECOVERY_QUORUM_INVALID") - normalized = [] - ids = set() - operators = set() - for identity_raw, operator_raw, adapter in recovery_sources: - identity = _identity( - identity_raw, "GOVERNANCE_SNAPSHOT_RECOVERY_IDENTITY_INVALID" - ) - operator = _identity( - operator_raw, "GOVERNANCE_SNAPSHOT_RECOVERY_OPERATOR_INVALID" - ) - if identity in ids: - _fail("GOVERNANCE_SNAPSHOT_RECOVERY_IDENTITY_DUPLICATE") - ids.add(identity) - operators.add(operator) - normalized.append((identity, operator, adapter)) - if len(operators) < int(POLICY["min_recovery_operators"]): - _fail("GOVERNANCE_SNAPSHOT_RECOVERY_OPERATOR_COUNT_INVALID") - current_time = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - observed = [] - unavailable = [] - evidence = [] - locators = set() - for identity, operator, adapter in sorted(normalized, key=lambda x: (x[1], x[0])): - recovery_id = _governance_recovery_id(governance_id, expected_epoch, identity) - request = { - "schemaVersion": int(POLICY["recovery_protocol_version"]), - "operation": "recover-governance", - "recoveryId": recovery_id, - "governanceId": governance_id, - "expectedEpoch": expected_epoch, - "source": {"identity": identity, "operator": operator}, - } - raw = adapter(request) - result = _validate_governance_recovery_response( - raw, - recovery_id=recovery_id, - governance_id=governance_id, - identity=identity, - operator=operator, - now=current_time, - ) - evidence.append( - { - "identity": identity, - "operator": operator, - "status": result["status"], - "sha256": _sha_bytes(_canonical_bytes(raw)), - } - ) - if result["status"] == "recovered": - if result["epoch"] != expected_epoch: - _fail("GOVERNANCE_SNAPSHOT_RECOVERY_EPOCH_MISMATCH") - if result["locator"] in locators: - _fail("GOVERNANCE_SNAPSHOT_RECOVERY_LOCATOR_COLLISION") - locators.add(result["locator"]) - observed.append(result) - else: - unavailable.append(result) - if len(observed) < recovery_quorum: - _fail("GOVERNANCE_SNAPSHOT_RECOVERY_QUORUM_NOT_MET") - if len({x["operator"] for x in observed}) < int(POLICY["min_recovery_operators"]): - _fail("GOVERNANCE_SNAPSHOT_RECOVERY_OPERATOR_QUORUM_NOT_MET") - views = { - ( - x["snapshot_sha256"], - x["governance_state_sha256"], - x["governance_bundle_sha256"], - x["governance_chain_head_sha256"], - x["history_state_sha256"], - x["history_bundle_sha256"], - x["history_chain_head_sha256"], - ) - for x in observed - } - if len(views) != 1: - _fail("GOVERNANCE_SNAPSHOT_RECOVERY_ARCHIVE_DISAGREEMENT") - chosen = min(observed, key=lambda x: (x["operator"], x["identity"])) - if chosen["governance_chain_head_sha256"] != expected_chain_head_sha256: - _fail("GOVERNANCE_SNAPSHOT_RECOVERY_CHAIN_HEAD_PIN_MISMATCH") - target.parent.mkdir(parents=True, exist_ok=True) - with tempfile.TemporaryDirectory( - prefix="governance-snapshot-recovery-", dir=target.parent - ) as tmp: - stage = Path(tmp) / "recovery" - stage.mkdir() - snapshot = chosen["snapshot"] - _write( - stage / "recovered-trusted-governance-state.json", - snapshot["governanceState"], - ) - _write( - stage / "recovered-release-governance-bundle.json", - snapshot["governanceBundle"], - ) - _write(stage / "recovered-trusted-history-state.json", snapshot["historyState"]) - _write( - stage / "recovered-release-history-bundle.json", snapshot["historyBundle"] - ) - receipt = { - "schemaVersion": int(POLICY["receipt_schema_version"]), - "status": "governance-recovered", - "governanceId": governance_id, - "epoch": expected_epoch, - "expectedChainHeadSha256": expected_chain_head_sha256, - "quorum": { - "configured": len(normalized), - "threshold": recovery_quorum, - "observed": len(observed), - "unavailable": len(unavailable), - "minimumOperators": int(POLICY["min_recovery_operators"]), - }, - "recovered": { - "snapshotSha256": chosen["snapshot_sha256"], - "governanceStateSha256": chosen["governance_state_sha256"], - "governanceBundleSha256": chosen["governance_bundle_sha256"], - "governanceChainHeadSha256": chosen["governance_chain_head_sha256"], - "historyStateSha256": chosen["history_state_sha256"], - "historyBundleSha256": chosen["history_bundle_sha256"], - "historyChainHeadSha256": chosen["history_chain_head_sha256"], - }, - "evidence": evidence, - } - _write(stage / "release-governance-recovery-receipt.json", receipt) - os.replace(stage, target) - return { - "ok": True, - "phase": "governance-recovered", - "governance_id": governance_id, - "epoch": expected_epoch, - "observed": len(observed), - "quorum": recovery_quorum, - "snapshot_sha256": chosen["snapshot_sha256"], - "state_sha256": chosen["governance_state_sha256"], - "bundle_sha256": chosen["governance_bundle_sha256"], - "chain_head_sha256": chosen["governance_chain_head_sha256"], - } - - -def _validate_archive_response( - value: dict[str, Any], - *, - operation: str, - archive_id: str, - governance_id: str, - epoch: int, - identity: str, - operator: str, - artifact: dict[str, Any], - now: datetime, -) -> dict[str, Any]: - expected = { - "schemaVersion", - "operation", - "archiveId", - "governanceId", - "epoch", - "status", - "archive", - "artifact", - "guarantees", - "verifiedAt", - } - if ( - set(value) != expected - or value.get("schemaVersion") != int(POLICY["archive_protocol_version"]) - or value.get("operation") != operation - or value.get("archiveId") != archive_id - or value.get("governanceId") != governance_id - or value.get("epoch") != epoch - ): - _fail("GOVERNANCE_ARCHIVE_RESULT_SCHEMA_INVALID") - if value.get("status") not in ( - {"created", "present"} if operation == "bind" else {"present"} - ): - _fail("GOVERNANCE_ARCHIVE_STATUS_INVALID") - archive = value.get("archive") - keys = { - "identity", - "operator", - "governanceWriterCredentialsReused", - "historyWriterCredentialsReused", - } - if ( - not isinstance(archive, dict) - or set(archive) != keys - or archive.get("identity") != identity - or archive.get("operator") != operator - or archive.get("governanceWriterCredentialsReused") is not False - or archive.get("historyWriterCredentialsReused") is not False - ): - _fail("GOVERNANCE_ARCHIVE_AUTHORITY_INVALID") - if value.get("artifact") != artifact: - _fail("GOVERNANCE_ARCHIVE_ARTIFACT_MISMATCH") - g = value.get("guarantees") - if ( - not isinstance(g, dict) - or set(g) - != { - "createOnly", - "overwrite", - "remoteReadbackVerified", - "immutability", - "locator", - } - or g.get("createOnly") is not True - or g.get("overwrite") is not False - or g.get("remoteReadbackVerified") is not True - ): - _fail("GOVERNANCE_ARCHIVE_GUARANTEES_INVALID") - if g.get("immutability") not in set(POLICY["allowed_archive_immutability"]): - _fail("GOVERNANCE_ARCHIVE_IMMUTABILITY_INVALID") - locator = _safe_locator(g.get("locator"), "GOVERNANCE_ARCHIVE_LOCATOR_INVALID") - _timestamp(value.get("verifiedAt"), "GOVERNANCE_ARCHIVE_VERIFIED_AT", now=now) - return {"locator": locator, "immutability": g["immutability"]} - - -def apply_governance_transition( # ruff: ignore[too-many-branches, undocumented-public-function] - *, - history_dir: Path, - previous_state: Path, - previous_bundle: Path, - proposal_path: Path, - approval_paths: list[Path], - output_dir: Path, - governance_archives: list[tuple[str, str, ArchiveAdapter]], - now: datetime | None = None, -) -> dict[str, Any]: - current = _validate_run153_dir(history_dir) - previous = _validate_previous_governance(previous_state, previous_bundle) - if current["history_id"] != previous["history_id"]: - _fail("GOVERNANCE_HISTORY_ID_MISMATCH") - if current["sequence"] < previous["history"]["sequence"]: - _fail("GOVERNANCE_HISTORY_ROLLBACK") - target = _outside(output_dir, (current["root"],), "GOVERNANCE_OUTPUT_INSIDE_INPUT") - if target.exists() or target.is_symlink(): - _fail("GOVERNANCE_OUTPUT_ALREADY_EXISTS") - proposal, proposal_raw = _read(proposal_path, "GOVERNANCE_PROPOSAL") - p = _validate_proposal(proposal, previous=previous, current=current) - proposal_sha = _sha_bytes(proposal_raw) - role = p["role"] - if len(approval_paths) != len(p["selected"]): - _fail("GOVERNANCE_APPROVAL_COUNT_INVALID") - approvals = [] - keys = [] - operators = set() - hashes = [] - for path in approval_paths: - doc, raw = _read(path, "GOVERNANCE_APPROVAL") - approver = _validate_approval( - doc, - governance_id=previous["governance_id"], - transition_id=p["transition_id"], - proposal_sha=proposal_sha, - role=role, - allowed_members=p["allowed"], - revoked=set(previous["revoked_authority_key_ids"]), - now=(now or datetime.now(timezone.utc)).astimezone(timezone.utc), - ) - approvals.append(doc) - keys.append(approver["keyId"]) - operators.add(approver["operator"]) - hashes.append(_sha_bytes(raw)) - if sorted(keys) != sorted(p["selected"]): - _fail("GOVERNANCE_APPROVAL_SET_MISMATCH") - min_ops = int( - POLICY["min_emergency_operators"] - if role == "emergency" - else POLICY["min_policy_operators"] - ) - if len(operators) < min_ops: - _fail("GOVERNANCE_APPROVAL_OPERATOR_QUORUM_INVALID") - approvals = sorted( - approvals, - key=lambda x: ( - x["approver"]["operator"], - x["approver"]["identity"], - x["approver"]["keyId"], - ), - ) - entry = { - "epoch": previous["epoch"] + 1, - "transitionId": p["transition_id"], - "reason": p["reason"], - "history": p["history"], - "proposal": proposal, - "proposalSha256": proposal_sha, - "approvalRole": role, - "selectedApproverKeyIds": sorted(p["selected"]), - "approvals": approvals, - "revokedAuthorityKeyIds": sorted(p["revocations"]), - "nextPolicy": p["next_policy"], - } - new_bundle = dict(previous["bundle"]) - new_bundle["entries"] = [*previous["entries"], entry] - info = _validate_bundle(new_bundle) - bundle_raw = _canonical_bytes(new_bundle) - bundle_sha = _sha_bytes(bundle_raw) - history_info = { - "sequence": current["sequence"], - "state_sha256": current["state_sha256"], - "bundle_sha256": current["bundle_sha256"], - "chain_head_sha256": current["chain_head_sha256"], - } - new_state = _state_for(info=info, history_info=history_info, bundle_sha=bundle_sha) - snapshot = { - "schemaVersion": int(POLICY["recovery_snapshot_schema_version"]), - "governanceState": new_state, - "governanceBundle": new_bundle, - "historyState": current["state"], - "historyBundle": current["bundle"], - } - snapshot_item = _artifact_doc(snapshot, "release-governance-recovery-snapshot.json") - if not ( - int(POLICY["min_governance_archives"]) - <= len(governance_archives) - <= int(POLICY["max_governance_archives"]) - ): - _fail("GOVERNANCE_ARCHIVE_COUNT_INVALID") - normalized = [] - ids = set() - ops = set() - for ir, oraw, adapter in governance_archives: - i = _identity(ir, "GOVERNANCE_ARCHIVE_IDENTITY_INVALID") - o = _identity(oraw, "GOVERNANCE_ARCHIVE_OPERATOR_INVALID") - if i in ids: - _fail("GOVERNANCE_ARCHIVE_IDENTITY_DUPLICATE") - ids.add(i) - ops.add(o) - normalized.append((i, o, adapter)) - if len(ops) < int(POLICY["min_governance_archive_operators"]): - _fail("GOVERNANCE_ARCHIVE_OPERATOR_COUNT_INVALID") - current_time = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - run153_inputs = sorted( - (p for p in current["root"].rglob("*") if p.is_file()), - key=lambda p: str(p.relative_to(current["root"])), - ) - input_paths = [ - *run153_inputs, - previous_state, - previous_bundle, - proposal_path, - *approval_paths, - ] - initial = {str(x.resolve()): _sha(x) for x in input_paths} - target.parent.mkdir(parents=True, exist_ok=True) - with tempfile.TemporaryDirectory( - prefix="release-governance-", dir=target.parent - ) as tmp: - stage = Path(tmp) / "governance" - stage.mkdir() - archive_dir = stage / "archive-results" - archive_dir.mkdir() - bundle_path = stage / "release-governance-bundle.json" - state_path = stage / _DOC_GOVERNANCE_STATE - snapshot_path = stage / "release-governance-recovery-snapshot.json" - _write(bundle_path, new_bundle) - _write(state_path, new_state) - _write(snapshot_path, snapshot) - verify_governance_bundle(bundle_path=bundle_path, state_path=state_path) - archive_evidence = [] - locators = set() - for index, (i, o, adapter) in enumerate( - sorted(normalized, key=lambda x: (x[1], x[0])), start=1 - ): - aid = _archive_id( - previous["governance_id"], info["epoch"], snapshot_item["sha256"], i - ) - base = { - "schemaVersion": int(POLICY["archive_protocol_version"]), - "archiveId": aid, - "governanceId": previous["governance_id"], - "epoch": info["epoch"], - "archive": {"identity": i, "operator": o}, - "artifact": snapshot_item, - } - bind_req = dict(base, operation="bind") - bind_req["artifact"] = dict(snapshot_item, localPath=str(snapshot_path)) - before = _sha(snapshot_path) - bind_raw = adapter(bind_req) - if _sha(snapshot_path) != before: - _fail("GOVERNANCE_SNAPSHOT_CHANGED_DURING_ARCHIVE") - bind = _validate_archive_response( - bind_raw, - operation="bind", - archive_id=aid, - governance_id=previous["governance_id"], - epoch=info["epoch"], - identity=i, - operator=o, - artifact=snapshot_item, - now=current_time, - ) - verify_raw = adapter(dict(base, operation="verify")) - verify = _validate_archive_response( - verify_raw, - operation="verify", - archive_id=aid, - governance_id=previous["governance_id"], - epoch=info["epoch"], - identity=i, - operator=o, - artifact=snapshot_item, - now=current_time, - ) - if bind != verify: - _fail("GOVERNANCE_ARCHIVE_VERIFY_REBIND_FAILED") - if verify["locator"] in locators: - _fail("GOVERNANCE_ARCHIVE_LOCATOR_COLLISION") - locators.add(verify["locator"]) - bp = ( - archive_dir - / f"{index:02d}-{hashlib.sha256(i.encode()).hexdigest()[:12]}.bind.json" - ) - vp = ( - archive_dir - / f"{index:02d}-{hashlib.sha256(i.encode()).hexdigest()[:12]}.verify.json" - ) - _write(bp, bind_raw) - _write(vp, verify_raw) - archive_evidence.append( - { - "identity": i, - "operator": o, - "archiveId": aid, - "locator": verify["locator"], - "immutability": verify["immutability"], - "bindEvidenceSha256": _sha(bp), - "verifyEvidenceSha256": _sha(vp), - } - ) - for path in input_paths: - if _sha(path) != initial[str(path.resolve())]: - _fail("GOVERNANCE_INPUT_CHANGED_DURING_TRANSITION") - receipt = { - "schemaVersion": int(POLICY["receipt_schema_version"]), - "status": "governed", - "governanceId": previous["governance_id"], - "historyId": current["history_id"], - "epoch": info["epoch"], - "policyVersion": info["policy"]["policyVersion"], - "proposalSha256": proposal_sha, - "approvalEvidenceSha256": sorted(hashes), - "history": { - "sequence": current["sequence"], - "stateSha256": current["state_sha256"], - "bundleSha256": current["bundle_sha256"], - "chainHeadSha256": current["chain_head_sha256"], - }, - "governance": { - "bundle": _artifact_doc(new_bundle, "release-governance-bundle.json"), - "state": _artifact_doc(new_state, _DOC_GOVERNANCE_STATE), - "recoverySnapshot": snapshot_item, - }, - "archives": archive_evidence, - } - _write(stage / "release-governance-receipt.json", receipt) - shutil.copy2(proposal_path, stage / "governance-transition-proposal.json") - approvals_dir = stage / "approval-evidence" - approvals_dir.mkdir() - for index, path in enumerate( - sorted(approval_paths, key=lambda p: p.name), start=1 - ): - shutil.copy2(path, approvals_dir / f"{index:02d}-{path.name}") - shutil.copy2( - current["root"] / _DOC_HISTORY_STATE, - stage / _DOC_HISTORY_STATE, - ) - shutil.copy2( - current["root"] / "release-history-bundle.json", - stage / "release-history-bundle.json", - ) - tmp_target = target.with_name(target.name + ".tmp-" + os.urandom(6).hex()) - shutil.copytree(stage, tmp_target, copy_function=shutil.copy2) - os.replace(tmp_target, target) - return { - "ok": True, - "phase": "governance-transition-applied", - "governance_id": previous["governance_id"], - "epoch": info["epoch"], - "policy_version": info["policy"]["policyVersion"], - "history_sequence": current["sequence"], - "bundle_sha256": _sha(target / "release-governance-bundle.json"), - "state_sha256": _sha(target / _DOC_GOVERNANCE_STATE), - "recovery_snapshot_sha256": _sha( - target / "release-governance-recovery-snapshot.json" - ), - "archive_count": len(normalized), - } - - -def _parse_endpoint(value: str, label: str) -> tuple[str, str, list[str]]: - parts = value.split("=", 1) - if len(parts) != 2 or "@" not in parts[0]: # ruff: ignore[magic-value-comparison] - raise argparse.ArgumentTypeError( - f"{label} must be IDENTITY@OPERATOR=EXECUTABLE[,ARG...]" - ) - identity, operator = parts[0].split("@", 1) - command = [x for x in parts[1].split(",") if x] - if not identity or not operator or not command: - raise argparse.ArgumentTypeError(f"invalid {label} specification") - return identity, operator, command - - -def main( # ruff: ignore[undocumented-public-function] - argv: list[str] | None = None, -) -> int: - parser = argparse.ArgumentParser(description=__doc__) - sub = parser.add_subparsers(dest="command", required=True) - verify = sub.add_parser("verify") - verify.add_argument("--bundle", type=Path, required=True) - verify.add_argument("--state", type=Path) - init = sub.add_parser("initialize") - init.add_argument("--history-dir", type=Path, required=True) - init.add_argument("--genesis", type=Path, required=True) - init.add_argument("--expected-genesis-sha256", required=True) - init.add_argument("--output-dir", type=Path, required=True) - recover = sub.add_parser("recover-history") - recover.add_argument("--history-id", required=True) - recover.add_argument("--expected-sequence", type=int, required=True) - recover.add_argument("--expected-chain-head-sha256", required=True) - recover.add_argument("--output-dir", type=Path, required=True) - recover.add_argument("--recovery-quorum", type=int, required=True) - recover.add_argument( - "--source", - action="append", - required=True, - type=lambda x: _parse_endpoint(x, "source"), - ) - recover_gov = sub.add_parser("recover-governance") - recover_gov.add_argument("--governance-id", required=True) - recover_gov.add_argument("--expected-epoch", type=int, required=True) - recover_gov.add_argument("--expected-chain-head-sha256", required=True) - recover_gov.add_argument("--output-dir", type=Path, required=True) - recover_gov.add_argument("--recovery-quorum", type=int, required=True) - recover_gov.add_argument( - "--source", - action="append", - required=True, - type=lambda x: _parse_endpoint(x, "source"), - ) - advance = sub.add_parser("transition") - advance.add_argument("--history-dir", type=Path, required=True) - advance.add_argument("--previous-state", type=Path, required=True) - advance.add_argument("--previous-bundle", type=Path, required=True) - advance.add_argument("--proposal", type=Path, required=True) - advance.add_argument("--approval", action="append", required=True, type=Path) - advance.add_argument("--output-dir", type=Path, required=True) - advance.add_argument( - "--archive", - action="append", - required=True, - type=lambda x: _parse_endpoint(x, "archive"), - ) - args = parser.parse_args(argv) - try: - if args.command == "verify": - result = verify_governance_bundle( - bundle_path=args.bundle, state_path=args.state - ) - elif args.command == "initialize": - result = initialize_governance( - history_dir=args.history_dir, - genesis_path=args.genesis, - expected_genesis_sha256=args.expected_genesis_sha256, - output_dir=args.output_dir, - ) - elif args.command == "recover-history": - sources = [(i, o, command_recovery_source(c)) for i, o, c in args.source] - result = recover_history( - history_id=args.history_id, - expected_sequence=args.expected_sequence, - expected_chain_head_sha256=args.expected_chain_head_sha256, - output_dir=args.output_dir, - recovery_sources=sources, - recovery_quorum=args.recovery_quorum, - ) - elif args.command == "recover-governance": - sources = [(i, o, command_recovery_source(c)) for i, o, c in args.source] - result = recover_governance( - governance_id=args.governance_id, - expected_epoch=args.expected_epoch, - expected_chain_head_sha256=args.expected_chain_head_sha256, - output_dir=args.output_dir, - recovery_sources=sources, - recovery_quorum=args.recovery_quorum, - ) - else: - archives = [(i, o, command_archive(c)) for i, o, c in args.archive] - result = apply_governance_transition( - history_dir=args.history_dir, - previous_state=args.previous_state, - previous_bundle=args.previous_bundle, - proposal_path=args.proposal, - approval_paths=args.approval, - output_dir=args.output_dir, - governance_archives=archives, - ) - except (GovernanceError, history.HistoryError, OSError) as exc: - logger.error("governance command failed: %s", exc) - # sys.stdout.write(json.dumps({"ok": False, "error": str(exc)}, sort_keys=True) + "\n") - return 2 - logger.info("governance command completed successfully") - # sys.stdout.write(json.dumps(result, sort_keys=True) + "\n") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/security/maintain_release_trust.py b/security/maintain_release_trust.py deleted file mode 100644 index 02ba4fe6f4426f76338f03aa76aeee33b12fe945..0000000000000000000000000000000000000000 --- a/security/maintain_release_trust.py +++ /dev/null @@ -1,1732 +0,0 @@ -""" -Run 156: delegated freshness metadata and threshold root recovery. - -Run 155 cryptographically seals governance. This layer adds short-lived snapshot and - timestamp metadata plus an independently pinned, multi-channel recovery authority for - root-role compromise. Private keys are never accepted as input or written to output. -""" - -from __future__ import annotations - -import argparse -import hashlib -import json -import logging -import re -import shutil -import sys -import tempfile -from datetime import datetime, timedelta, timezone -from pathlib import Path -from typing import Any - -import tomllib - -logger = logging.getLogger(__name__) - -try: - from . import seal_release_governance as rootseal -except (ImportError, ValueError) as exc: - import importlib.util - - _p = Path(__file__).resolve().parent / "seal_release_governance.py" - _n = "_run155_root_for_delegated_trust" - if _n in sys.modules: - rootseal = sys.modules[_n] - else: - _s = importlib.util.spec_from_file_location(_n, _p) - if _s is None or _s.loader is None: - raise ImportError("cannot load seal_release_governance") from exc - rootseal = importlib.util.module_from_spec(_s) - sys.modules[_n] = rootseal - _s.loader.exec_module(rootseal) - -HERE = Path(__file__).resolve().parent -POLICY = tomllib.loads((HERE / "release_delegated_trust_policy.toml").read_text()) -PREDICATE_TYPE = str(POLICY["predicate_type"]) -_HEX64 = re.compile(r"^[0-9a-f]{64}$") -_ID = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:/@+-]{0,511}$") -_ZERO_HASH = "0" * 64 - - -class DelegatedTrustError(RuntimeError): # ruff: ignore[undocumented-public-class] - pass - - -def _fail(code: str) -> None: - raise DelegatedTrustError(code) - - -def _canonical(value: dict[str, Any]) -> bytes: - return rootseal._canonical_bytes(value) - - -def _sha_bytes(raw: bytes) -> str: - return hashlib.sha256(raw).hexdigest() - - -def _sha(path: Path) -> str: - return rootseal._sha(path) - - -def _loads(raw: bytes, code: str) -> dict[str, Any]: - def hook(pairs): - out = {} - for k, v in pairs: - if k in out: - _fail(code + "_DUPLICATE_KEY") - out[k] = v - return out - - try: - doc = json.loads(raw.decode("utf-8"), object_pairs_hook=hook) - except DelegatedTrustError: - raise - except Exception as exc: - raise DelegatedTrustError(code + "_JSON_INVALID") from exc - if not isinstance(doc, dict): - _fail(code + "_SCHEMA_INVALID") - return doc - - -def _read( - path: Path, code: str, *, canonical: bool = True -) -> tuple[dict[str, Any], bytes]: - if path.is_symlink() or not path.is_file(): - _fail(code + "_INVALID") - if path.stat().st_size > int(POLICY["max_json_bytes"]): - _fail(code + "_TOO_LARGE") - raw = path.read_bytes() - doc = _loads(raw, code) - if canonical and raw != _canonical(doc): - _fail(code + "_NOT_CANONICAL") - return doc, raw - - -def _write(path: Path, doc: dict[str, Any]) -> None: - path.write_bytes(_canonical(doc)) - - -def _identity(value: Any, code: str) -> str: - if not isinstance(value, str): - _fail(code) - v = value.strip() - if ( - not v - or len(v) > 512 # ruff: ignore[magic-value-comparison] - or ".." in v - or "?" in v - or "#" in v - or "\x00" in v - ): - _fail(code) - if _ID.fullmatch(v) is None or any( - ord(c) < 32 # ruff: ignore[magic-value-comparison] - or ord(c) == 127 # ruff: ignore[magic-value-comparison] - for c in v # lint - ): - _fail(code) - return v - - -def _hex(value: Any, code: str) -> str: - if not isinstance(value, str) or _HEX64.fullmatch(value) is None: - _fail(code) - return value - - -def _int(value: Any, code: str, *, minimum: int = 0) -> int: - if not isinstance(value, int) or isinstance(value, bool) or value < minimum: - _fail(code) - return value - - -def _dt(value: Any, code: str) -> datetime: - if not isinstance(value, str) or not value.endswith("Z"): - _fail(code) - try: - return datetime.fromisoformat(value[:-1] + "+00:00").astimezone(timezone.utc) - except ValueError as exc: - raise DelegatedTrustError(code) from exc - - -def _ts(dt: datetime) -> str: - return ( - dt.astimezone(timezone.utc).isoformat(timespec="seconds").replace("+00:00", "Z") - ) - - -def _b64(value: Any, code: str, *, expected_len: int) -> bytes: - try: - return rootseal._b64(value, code, expected_len=expected_len) - except rootseal.RootTrustError as exc: - raise DelegatedTrustError(str(exc)) from exc - - -def _artifact(path: Path, name: str | None = None) -> dict[str, Any]: - return { - "name": name or path.name, - "sha256": _sha(path), - "size": path.stat().st_size, - } - - -def _artifact_raw(name: str, raw: bytes) -> dict[str, Any]: - return {"name": name, "sha256": _sha_bytes(raw), "size": len(raw)} - - -def _dir(path: Path, code: str) -> Path: - if path.is_symlink() or not path.is_dir(): - _fail(code) - return path.resolve() - - -def _outside(path: Path, protected: list[Path], code: str) -> Path: - t = path.expanduser().resolve() - for p in protected: - r = p.resolve() - if t == r or r in t.parents: - _fail(code) - return t - - -def _key(value: Any, code: str) -> dict[str, Any]: - expected = { - "keytype", - "scheme", - "identity", - "operator", - "expires", - "signerProfile", - "keyval", - } - if not isinstance(value, dict) or set(value) != expected: - _fail(code + "_SCHEMA_INVALID") - if value["keytype"] not in set(POLICY["allowed_keytypes"]) or value[ - "scheme" - ] not in set(POLICY["allowed_schemes"]): - _fail(code + "_ALGORITHM_INVALID") - profile = value["signerProfile"] - if profile not in set(POLICY["allowed_signer_profiles"]): - _fail(code + "_PROFILE_INVALID") - kv = value["keyval"] - if not isinstance(kv, dict) or set(kv) != {"public"}: - _fail(code + "_KEYVAL_INVALID") - _b64(kv["public"], code + "_PUBLIC_INVALID", expected_len=32) - _dt(value["expires"], code + "_EXPIRES_INVALID") - return { - "keytype": "ed25519", - "scheme": "ed25519", - "identity": _identity(value["identity"], code + "_IDENTITY_INVALID"), - "operator": _identity(value["operator"], code + "_OPERATOR_INVALID"), - "expires": value["expires"], - "signerProfile": profile, - "keyval": {"public": kv["public"]}, - } - - -def _role( - value: Any, - keys: dict[str, dict[str, Any]], - code: str, - *, - min_keys: int, - min_threshold: int, - min_ops: int, -) -> dict[str, Any]: - if not isinstance(value, dict) or set(value) != {"keyids", "threshold"}: - _fail(code + "_SCHEMA_INVALID") - ids = value["keyids"] - if not isinstance(ids, list) or not ( - min_keys <= len(ids) <= int(POLICY["max_keys"]) - ): - _fail(code + "_KEY_COUNT_INVALID") - ids = [_identity(x, code + "_KEYID_INVALID") for x in ids] - if ( - ids != sorted(ids) - or len(set(ids)) != len(ids) - or any(x not in keys for x in ids) - ): - _fail(code + "_KEY_SET_INVALID") - threshold = _int( - value["threshold"], code + "_THRESHOLD_INVALID", minimum=min_threshold - ) - if threshold > len(ids): - _fail(code + "_THRESHOLD_INVALID") - if len({keys[x]["operator"] for x in ids}) < min_ops: - _fail(code + "_OPERATOR_COUNT_INVALID") - return {"keyids": ids, "threshold": threshold} - - -def _sig_map(doc: dict[str, Any], code: str) -> dict[str, str]: - sigs = doc.get("signatures") - if not isinstance(sigs, list) or len(sigs) > int(POLICY["max_keys"]) * 2: - _fail(code + "_SIGNATURES_INVALID") - out = {} - norm = [] - for item in sigs: - if not isinstance(item, dict) or set(item) != {"keyid", "sig"}: - _fail(code + "_SIGNATURE_SCHEMA_INVALID") - kid = _identity(item["keyid"], code + "_SIGNATURE_KEYID_INVALID") - if kid in out: - _fail(code + "_SIGNATURE_DUPLICATE") - _b64(item["sig"], code + "_SIGNATURE_ENCODING_INVALID", expected_len=64) - out[kid] = item["sig"] - norm.append({"keyid": kid, "sig": item["sig"]}) - norm.sort(key=lambda x: x["keyid"]) - if sigs != norm: - _fail(code + "_SIGNATURES_NOT_NORMALIZED") - return out - - -def _verify_role_signatures( - doc: dict[str, Any], - signed: dict[str, Any], - role: dict[str, Any], - keys: dict[str, dict[str, Any]], - *, - at: datetime, - code: str, - exact_allowed: bool = True, -) -> list[str]: - sigs = _sig_map(doc, code) - allowed = set(role["keyids"]) - if exact_allowed and not set(sigs).issubset(allowed): - _fail(code + "_UNAUTHORIZED_SIGNATURE_KEY") - raw = _canonical(signed) - valid = [] - for kid in role["keyids"]: - if kid not in sigs: - continue - key = keys[kid] - if _dt(key["expires"], code + "_KEY_EXPIRES") < at: - _fail(code + "_KEY_EXPIRED") - pub = _b64(key["keyval"]["public"], code + "_PUBLIC_INVALID", expected_len=32) - sig = _b64(sigs[kid], code + "_SIG_INVALID", expected_len=64) - try: - rootseal._ed25519_verify(pub, sig, raw, code + "_SIGNATURE_INVALID") - except rootseal.RootTrustError as exc: - raise DelegatedTrustError(str(exc)) from exc - valid.append(kid) - if len(valid) < role["threshold"]: - _fail(code + "_THRESHOLD_NOT_MET") - return valid - - -# ── Verified-state document names ──────────────────────────────────────── -# -# One constant per release-evidence document. The values are the on-disk -# artifact names and are unchanged; only the repetition is removed. Reading -# a document by a name that says which document it is also keeps static -# analysis from reading the filename token 'trusted' as 'confidential': -# these files carry published Merkle roots and SHA-256 digests, which must -# stay in clear text for any third party to verify them. -_DOC_DELEGATED_METADATA_STATE = "trusted-delegated-metadata-state.json" -_DOC_RELEASE_ROOT_STATE = "trusted-release-root-state.json" - - -def _effective_root_from_run155( - sealed_dir: Path, bootstrap_pin: str, *, now: datetime, require_fresh: bool = True -) -> dict[str, Any]: - try: - rootseal.verify_sealed_governance( - sealed_dir=sealed_dir, - expected_bootstrap_root_sha256=bootstrap_pin, - now=now, - require_current_fresh=require_fresh, - ) - bundle_info = rootseal.verify_root_bundle( - bundle_path=sealed_dir / "release-root-bundle.json", - state_path=sealed_dir / _DOC_RELEASE_ROOT_STATE, - expected_bootstrap_root_sha256=bootstrap_pin, - now=now, - require_current_fresh=require_fresh, - ) - except rootseal.RootTrustError as exc: - raise DelegatedTrustError("DELEGATED_RUN155_INVALID:" + str(exc)) from exc - cur = bundle_info["current"] - return { - "version": cur["version"], - "sha256": cur["sha256"], - "chainHeadSha256": bundle_info["chain_head_sha256"], - "envelope": cur["envelope"], - "info": cur, - "infos": bundle_info["infos"], - } - - -def _verify_delegation_root( # ruff: ignore[too-many-branches] - path: Path, effective_root: dict[str, Any], *, now: datetime -) -> dict[str, Any]: - doc, raw = _read(path, "DELEGATION_ROOT") - if set(doc) != {"signatures", "signed"} or not isinstance(doc["signed"], dict): - _fail("DELEGATION_ROOT_SCHEMA_INVALID") - s = doc["signed"] - expected = { - "_type", - "specVersion", - "schemaVersion", - "version", - "governanceId", - "issuedAt", - "expires", - "run155Root", - "keys", - "roles", - } - if ( - set(s) != expected - or s["_type"] != "delegations" - or s["specVersion"] != str(POLICY["spec_version"]) - or s["schemaVersion"] != int(POLICY["delegation_schema_version"]) - ): - _fail("DELEGATION_ROOT_SIGNED_SCHEMA_INVALID") - version = _int(s["version"], "DELEGATION_ROOT_VERSION_INVALID", minimum=1) - if s["governanceId"] != effective_root["info"]["governance_id"]: - _fail("DELEGATION_ROOT_GOVERNANCE_MISMATCH") - rb = s["run155Root"] - exp_rb = { - "version": effective_root["version"], - "sha256": effective_root["sha256"], - "chainHeadSha256": effective_root["chainHeadSha256"], - } - if rb != exp_rb: - _fail("DELEGATION_ROOT_RUN155_ROOT_MISMATCH") - issued = _dt(s["issuedAt"], "DELEGATION_ROOT_ISSUED_INVALID") - expires = _dt(s["expires"], "DELEGATION_ROOT_EXPIRES_INVALID") - skew = timedelta(minutes=int(POLICY["max_clock_skew_minutes"])) - if issued > now + skew: - _fail("DELEGATION_ROOT_ISSUED_FROM_FUTURE") - if expires <= issued or expires - issued > timedelta( - days=int(POLICY["max_delegation_lifetime_days"]) - ): - _fail("DELEGATION_ROOT_LIFETIME_INVALID") - if expires - now < timedelta( - minutes=int(POLICY["min_delegation_remaining_minutes"]) - ): - _fail("DELEGATION_ROOT_FREEZE_RISK") - kr = s["keys"] - if not isinstance(kr, dict) or not ( - 6 <= len(kr) <= int(POLICY["max_keys"]) # ruff: ignore[magic-value-comparison] - ): - _fail("DELEGATION_ROOT_KEYS_INVALID") - keys = { - _identity(k, "DELEGATION_ROOT_KEYID_INVALID"): _key(v, "DELEGATION_ROOT_KEY") - for k, v in kr.items() - } - if list(kr) != sorted(kr): - _fail("DELEGATION_ROOT_KEYS_NOT_NORMALIZED") - roles_raw = s["roles"] - if not isinstance(roles_raw, dict) or set(roles_raw) != {"snapshot", "timestamp"}: - _fail("DELEGATION_ROOT_ROLES_INVALID") - snapshot = _role( - roles_raw["snapshot"], - keys, - "DELEGATION_SNAPSHOT_ROLE", - min_keys=int(POLICY["min_snapshot_keys"]), - min_threshold=int(POLICY["min_snapshot_threshold"]), - min_ops=int(POLICY["min_snapshot_operators"]), - ) - timestamp = _role( - roles_raw["timestamp"], - keys, - "DELEGATION_TIMESTAMP_ROLE", - min_keys=int(POLICY["min_timestamp_keys"]), - min_threshold=int(POLICY["min_timestamp_threshold"]), - min_ops=int(POLICY["min_timestamp_operators"]), - ) - if set(snapshot["keyids"]) & set(timestamp["keyids"]): - _fail("DELEGATION_ROLE_KEY_OVERLAP") - for kid in snapshot["keyids"] + timestamp["keyids"]: - if _dt(keys[kid]["expires"], "DELEGATION_ROLE_KEY_EXPIRES_INVALID") < expires: - _fail("DELEGATION_ROLE_KEY_EXPIRES_BEFORE_DELEGATION") - root_info = effective_root["info"] - # Delegation signatures cover the delegation bytes, not the Run 155 root signed body. - role = root_info["roles"]["root"] - root_keys = root_info["keys"] - sigs = _sig_map(doc, "DELEGATION_ROOT") - if not set(sigs).issubset(set(role["keyids"])): - _fail("DELEGATION_ROOT_UNAUTHORIZED_SIGNATURE_KEY") - valid = [] - raw_signed = _canonical(s) - for kid in role["keyids"]: - if kid not in sigs: - continue - key = root_keys[kid] - if _dt(key["expires"], "DELEGATION_ROOT_ROOT_KEY_EXPIRES") < issued: - _fail("DELEGATION_ROOT_ROOT_KEY_EXPIRED") - pub = _b64( - key["keyval"]["public"], - "DELEGATION_ROOT_ROOT_PUBLIC_INVALID", - expected_len=32, - ) - sig = _b64(sigs[kid], "DELEGATION_ROOT_ROOT_SIG_INVALID", expected_len=64) - try: - rootseal._ed25519_verify( - pub, sig, raw_signed, "DELEGATION_ROOT_SIGNATURE_INVALID" - ) - except rootseal.RootTrustError as exc: - raise DelegatedTrustError(str(exc)) from exc - valid.append(kid) - if len(valid) < role["threshold"]: - _fail("DELEGATION_ROOT_THRESHOLD_NOT_MET") - return { - "doc": doc, - "raw": raw, - "sha256": _sha_bytes(raw), - "version": version, - "issued": issued, - "expires": expires, - "keys": keys, - "roles": {"snapshot": snapshot, "timestamp": timestamp}, - } - - -def _sealed_artifacts(sealed_dir: Path) -> dict[str, dict[str, Any]]: - root = _dir(sealed_dir, "DELEGATED_SEALED_DIR_INVALID") - names = { - "release-root-bundle.json", - _DOC_RELEASE_ROOT_STATE, - "cryptographic-governance-authorization.json", - "active-root.json", - "release-governance-recovery-snapshot.json", - "release-root-receipt.json", - } - if {p.name for p in root.iterdir()} != names: - _fail("DELEGATED_SEALED_DIR_ALLOWLIST_MISMATCH") - return {name: _artifact(root / name) for name in sorted(names)} - - -def _verify_metadata_envelope( - path: Path, - *, - kind: str, - delegation: dict[str, Any], - now: datetime, - expected_body: dict[str, Any] | None = None, -) -> dict[str, Any]: - doc, raw = _read(path, f"DELEGATED_{kind.upper()}") - if set(doc) != {"signatures", "signed"} or not isinstance(doc["signed"], dict): - _fail(f"DELEGATED_{kind.upper()}_SCHEMA_INVALID") - s = doc["signed"] - expected = { - "_type", - "specVersion", - "schemaVersion", - "version", - "governanceId", - "issuedAt", - "expires", - "delegationRoot", - "body", - } - if ( - set(s) != expected - or s["_type"] != kind - or s["specVersion"] != str(POLICY["spec_version"]) - ): - _fail(f"DELEGATED_{kind.upper()}_SIGNED_SCHEMA_INVALID") - schema_key = f"{kind}_schema_version" - if s["schemaVersion"] != int(POLICY[schema_key]): - _fail(f"DELEGATED_{kind.upper()}_SCHEMA_VERSION_INVALID") - version = _int(s["version"], f"DELEGATED_{kind.upper()}_VERSION_INVALID", minimum=1) - if s["governanceId"] != delegation["doc"]["signed"]["governanceId"]: - _fail(f"DELEGATED_{kind.upper()}_GOVERNANCE_MISMATCH") - if s["delegationRoot"] != { - "version": delegation["version"], - "sha256": delegation["sha256"], - }: - _fail(f"DELEGATED_{kind.upper()}_DELEGATION_MISMATCH") - issued = _dt(s["issuedAt"], f"DELEGATED_{kind.upper()}_ISSUED_INVALID") - expires = _dt(s["expires"], f"DELEGATED_{kind.upper()}_EXPIRES_INVALID") - skew = timedelta(minutes=int(POLICY["max_clock_skew_minutes"])) - if issued > now + skew: - _fail(f"DELEGATED_{kind.upper()}_ISSUED_FROM_FUTURE") - maxlife = timedelta( - hours=int( - POLICY["max_snapshot_lifetime_hours"] - if kind == "snapshot" - else POLICY["max_timestamp_lifetime_hours"] - ) - ) - minremain = timedelta( - minutes=int( - POLICY["min_snapshot_remaining_minutes"] - if kind == "snapshot" - else POLICY["min_timestamp_remaining_minutes"] - ) - ) - if expires <= issued or expires - issued > maxlife: - _fail(f"DELEGATED_{kind.upper()}_LIFETIME_INVALID") - if expires - now < minremain: - _fail(f"DELEGATED_{kind.upper()}_FREEZE_RISK") - if not isinstance(s["body"], dict): - _fail(f"DELEGATED_{kind.upper()}_BODY_INVALID") - if expected_body is not None and s["body"] != expected_body: - _fail(f"DELEGATED_{kind.upper()}_BODY_MISMATCH") - _verify_role_signatures( - doc, - s, - delegation["roles"][kind], - delegation["keys"], - at=issued, - code=f"DELEGATED_{kind.upper()}", - ) - return { - "doc": doc, - "raw": raw, - "sha256": _sha_bytes(raw), - "version": version, - "issued": issued, - "expires": expires, - } - - -def _previous_state( # ruff: ignore[too-many-branches] - state_path: Path | None, bundle_path: Path | None -) -> dict[str, Any] | None: - if state_path is None and bundle_path is None: - return None - if state_path is None or bundle_path is None: - _fail("DELEGATED_PREVIOUS_PAIR_REQUIRED") - state, state_raw = _read(state_path, "DELEGATED_PREVIOUS_STATE") - bundle, bundle_raw = _read(bundle_path, "DELEGATED_PREVIOUS_BUNDLE") - if set(state) != { - "schemaVersion", - "status", - "governanceId", - "effectiveRoot", - "delegationRoot", - "snapshot", - "timestamp", - "bundleSha256", - "chainHeadSha256", - }: - _fail("DELEGATED_PREVIOUS_STATE_SCHEMA_INVALID") - if set(bundle) != { - "schemaVersion", - "predicateType", - "status", - "governanceId", - "entries", - }: - _fail("DELEGATED_PREVIOUS_BUNDLE_SCHEMA_INVALID") - if ( - state["schemaVersion"] != int(POLICY["state_schema_version"]) - or bundle["schemaVersion"] != int(POLICY["bundle_schema_version"]) - or bundle["predicateType"] != PREDICATE_TYPE - or state["status"] != "trusted-delegated-metadata" - or bundle["status"] != "trusted-delegated-metadata-chain" - ): - _fail("DELEGATED_PREVIOUS_SCHEMA_INVALID") - if _sha_bytes(bundle_raw) != state["bundleSha256"]: - _fail("DELEGATED_PREVIOUS_BUNDLE_HASH_MISMATCH") - entries = bundle["entries"] - if not isinstance(entries, list) or not entries: - _fail("DELEGATED_PREVIOUS_ENTRIES_INVALID") - head = _ZERO_HASH - prev = None - for i, e in enumerate(entries, 1): - if ( - not isinstance(e, dict) - or e.get("sequence") != i - or e.get("previousChainHeadSha256") != head - ): - _fail("DELEGATED_PREVIOUS_CHAIN_INVALID") - body = {k: v for k, v in e.items() if k != "chainHeadSha256"} - calc = _sha_bytes((head + "\n").encode() + _canonical(body)) - if e.get("chainHeadSha256") != calc: - _fail("DELEGATED_PREVIOUS_CHAIN_HASH_INVALID") - head = calc - prev = e - if head != state["chainHeadSha256"] or prev is None: - _fail("DELEGATED_PREVIOUS_HEAD_MISMATCH") - if bundle["governanceId"] != state["governanceId"]: - _fail("DELEGATED_PREVIOUS_GOVERNANCE_MISMATCH") - if state["effectiveRoot"] != prev.get("effectiveRoot"): - _fail("DELEGATED_PREVIOUS_STATE_ROOT_REBIND_FAILED") - if state["delegationRoot"] != { - "version": prev["delegationRoot"]["version"], - "sha256": prev["delegationRoot"]["sha256"], - }: - _fail("DELEGATED_PREVIOUS_STATE_DELEGATION_REBIND_FAILED") - if state["snapshot"] != { - "version": prev["snapshot"]["version"], - "sha256": prev["snapshot"]["sha256"], - "expires": prev["snapshot"]["expires"], - }: - _fail("DELEGATED_PREVIOUS_STATE_SNAPSHOT_REBIND_FAILED") - if state["timestamp"] != { - "version": prev["timestamp"]["version"], - "sha256": prev["timestamp"]["sha256"], - "expires": prev["timestamp"]["expires"], - }: - _fail("DELEGATED_PREVIOUS_STATE_TIMESTAMP_REBIND_FAILED") - return { - "state": state, - "state_raw": state_raw, - "bundle": bundle, - "bundle_raw": bundle_raw, - "head": head, - "last": prev, - } - - -def refresh_delegated_metadata( # ruff: ignore[too-many-branches, undocumented-public-function] - *, - sealed_dir: Path, - bootstrap_root_sha256: str, - delegation_root_path: Path, - snapshot_path: Path, - timestamp_path: Path, - output_dir: Path, - previous_state_path: Path | None = None, - previous_bundle_path: Path | None = None, - now: datetime | None = None, -) -> dict[str, Any]: - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - sealed = _dir(sealed_dir, "DELEGATED_RUN155_DIR_INVALID") - effective = _effective_root_from_run155(sealed, bootstrap_root_sha256, now=current) - delegation = _verify_delegation_root(delegation_root_path, effective, now=current) - artifacts = _sealed_artifacts(sealed) - snapshot_body = { - "run155Root": { - "version": effective["version"], - "sha256": effective["sha256"], - "chainHeadSha256": effective["chainHeadSha256"], - }, - "sealedArtifacts": artifacts, - } - snapshot = _verify_metadata_envelope( - snapshot_path, - kind="snapshot", - delegation=delegation, - now=current, - expected_body=snapshot_body, - ) - timestamp_body = { - "snapshot": { - "version": snapshot["version"], - "sha256": snapshot["sha256"], - "size": len(snapshot["raw"]), - "expires": snapshot["doc"]["signed"]["expires"], - }, - "run155Root": {"version": effective["version"], "sha256": effective["sha256"]}, - } - timestamp = _verify_metadata_envelope( - timestamp_path, - kind="timestamp", - delegation=delegation, - now=current, - expected_body=timestamp_body, - ) - if timestamp["issued"] < snapshot["issued"]: - _fail("DELEGATED_TIMESTAMP_PRECEDES_SNAPSHOT") - if timestamp["expires"] > snapshot["expires"]: - _fail("DELEGATED_TIMESTAMP_EXPIRES_AFTER_SNAPSHOT") - prev = _previous_state(previous_state_path, previous_bundle_path) - if prev is None: - if ( - delegation["version"] != 1 - or snapshot["version"] != 1 - or timestamp["version"] != 1 - ): - _fail("DELEGATED_BOOTSTRAP_VERSION_INVALID") - entries = [] - head = _ZERO_HASH - seq = 1 - else: - ps = prev["state"] - if ps["governanceId"] != effective["info"]["governance_id"]: - _fail("DELEGATED_GOVERNANCE_CHANGED") - prior_root = ps["effectiveRoot"] - matches = [ - x - for x in effective["infos"] - if x["version"] == prior_root["version"] - and x["sha256"] == prior_root["sha256"] - ] - if len(matches) != 1: - _fail("DELEGATED_ROOT_HISTORY_FORK") - prior_index = effective["infos"].index(matches[0]) + 1 - prior_head = rootseal._root_head( - effective["info"]["governance_id"], - [x["envelope"] for x in effective["infos"][:prior_index]], - ) - if prior_head != prior_root["chainHeadSha256"]: - _fail("DELEGATED_ROOT_HISTORY_HEAD_MISMATCH") - if delegation["version"] not in { - ps["delegationRoot"]["version"], - ps["delegationRoot"]["version"] + 1, - }: - _fail("DELEGATED_DELEGATION_VERSION_INVALID") - if ( - delegation["version"] == ps["delegationRoot"]["version"] - and delegation["sha256"] != ps["delegationRoot"]["sha256"] - ): - _fail("DELEGATED_SAME_VERSION_DELEGATION_CHANGED") - if ( - snapshot["version"] != ps["snapshot"]["version"] + 1 - or timestamp["version"] != ps["timestamp"]["version"] + 1 - ): - _fail("DELEGATED_METADATA_VERSION_ROLLBACK_OR_SKIP") - entries = list(prev["bundle"]["entries"]) - head = prev["head"] - seq = len(entries) + 1 - entry_body = { - "sequence": seq, - "previousChainHeadSha256": head, - "effectiveRoot": { - "version": effective["version"], - "sha256": effective["sha256"], - "chainHeadSha256": effective["chainHeadSha256"], - }, - "delegationRoot": ( - _artifact_raw("release-delegation-root.json", delegation["raw"]) - | {"version": delegation["version"]} - ), - "snapshot": ( - _artifact_raw("release-snapshot.json", snapshot["raw"]) - | { - "version": snapshot["version"], - "expires": snapshot["doc"]["signed"]["expires"], - } - ), - "timestamp": ( - _artifact_raw("release-timestamp.json", timestamp["raw"]) - | { - "version": timestamp["version"], - "expires": timestamp["doc"]["signed"]["expires"], - } - ), - } - chain = _sha_bytes((head + "\n").encode() + _canonical(entry_body)) - entry = entry_body | {"chainHeadSha256": chain} - entries.append(entry) - bundle = { - "schemaVersion": int(POLICY["bundle_schema_version"]), - "predicateType": PREDICATE_TYPE, - "status": "trusted-delegated-metadata-chain", - "governanceId": effective["info"]["governance_id"], - "entries": entries, - } - bundle_raw = _canonical(bundle) - state = { - "schemaVersion": int(POLICY["state_schema_version"]), - "status": "trusted-delegated-metadata", - "governanceId": effective["info"]["governance_id"], - "effectiveRoot": entry["effectiveRoot"], - "delegationRoot": { - "version": delegation["version"], - "sha256": delegation["sha256"], - }, - "snapshot": { - "version": snapshot["version"], - "sha256": snapshot["sha256"], - "expires": snapshot["doc"]["signed"]["expires"], - }, - "timestamp": { - "version": timestamp["version"], - "sha256": timestamp["sha256"], - "expires": timestamp["doc"]["signed"]["expires"], - }, - "bundleSha256": _sha_bytes(bundle_raw), - "chainHeadSha256": chain, - } - protected = [ - sealed, - delegation_root_path.resolve().parent, - snapshot_path.resolve().parent, - timestamp_path.resolve().parent, - ] - if previous_state_path: - protected += [previous_state_path.resolve().parent] - target = _outside(output_dir, protected, "DELEGATED_OUTPUT_INSIDE_INPUT") - if target.exists() or target.is_symlink(): - _fail("DELEGATED_OUTPUT_ALREADY_EXISTS") - input_files = [delegation_root_path, snapshot_path, timestamp_path] - if previous_state_path is not None: - input_files += [previous_state_path, previous_bundle_path] - before = {"sealed/" + p.name: _artifact(p) for p in sealed.iterdir()} - before.update({"input/" + str(i): _artifact(p) for i, p in enumerate(input_files)}) - tmp = Path(tempfile.mkdtemp(prefix="run156-delegated-", dir=str(target.parent))) - try: - _write(tmp / "release-delegation-root.json", delegation["doc"]) - _write(tmp / "release-snapshot.json", snapshot["doc"]) - _write(tmp / "release-timestamp.json", timestamp["doc"]) - _write(tmp / "release-delegated-metadata-bundle.json", bundle) - _write(tmp / _DOC_DELEGATED_METADATA_STATE, state) - receipt = { - "schemaVersion": int(POLICY["receipt_schema_version"]), - "status": "delegated-metadata-accepted", - "governanceId": state["governanceId"], - "sequence": seq, - "delegationRoot": _artifact(tmp / "release-delegation-root.json"), - "snapshot": _artifact(tmp / "release-snapshot.json"), - "timestamp": _artifact(tmp / "release-timestamp.json"), - "bundle": _artifact(tmp / "release-delegated-metadata-bundle.json"), - "state": _artifact(tmp / _DOC_DELEGATED_METADATA_STATE), - } - _write(tmp / "release-delegated-metadata-receipt.json", receipt) - after = {"sealed/" + p.name: _artifact(p) for p in sealed.iterdir()} - after.update( - {"input/" + str(i): _artifact(p) for i, p in enumerate(input_files)} - ) - if before != after: - _fail("DELEGATED_INPUT_DRIFT") - tmp.replace(target) - except Exception: - shutil.rmtree(tmp, ignore_errors=True) - raise - return { - "ok": True, - "phase": "delegated-metadata-accepted", - "sequence": seq, - "snapshot_version": snapshot["version"], - "timestamp_version": timestamp["version"], - "chain_head_sha256": chain, - } - - -def _recovery_key(value: Any, code: str) -> dict[str, Any]: - if not isinstance(value, dict) or "recoveryChannel" not in value: - _fail(code + "_SCHEMA_INVALID") - base = {k: v for k, v in value.items() if k != "recoveryChannel"} - normalized = _key(base, code) - normalized["recoveryChannel"] = _identity( - value["recoveryChannel"], code + "_CHANNEL_INVALID" - ) - if set(value) != { - "keytype", - "scheme", - "identity", - "operator", - "expires", - "signerProfile", - "keyval", - "recoveryChannel", - }: - _fail(code + "_SCHEMA_INVALID") - return normalized - - -def _verify_recovery_root( # ruff: ignore[too-many-branches] - path: Path, expected_sha256: str, *, now: datetime -) -> dict[str, Any]: - doc, raw = _read(path, "RECOVERY_ROOT") - _hex(expected_sha256, "RECOVERY_ROOT_PIN_INVALID") - if _sha_bytes(raw) != expected_sha256: - _fail("RECOVERY_ROOT_PIN_MISMATCH") - if set(doc) != {"signatures", "signed"} or not isinstance(doc["signed"], dict): - _fail("RECOVERY_ROOT_SCHEMA_INVALID") - s = doc["signed"] - expected = { - "_type", - "specVersion", - "schemaVersion", - "version", - "governanceId", - "issuedAt", - "expires", - "keys", - "role", - "minChannels", - } - if ( - set(s) != expected - or s["_type"] != "recovery-root" - or s["specVersion"] != str(POLICY["spec_version"]) - or s["schemaVersion"] != int(POLICY["recovery_root_schema_version"]) - ): - _fail("RECOVERY_ROOT_SIGNED_SCHEMA_INVALID") - if _int(s["version"], "RECOVERY_ROOT_VERSION_INVALID", minimum=1) != 1: - _fail("RECOVERY_ROOT_BOOTSTRAP_VERSION_INVALID") - issued = _dt(s["issuedAt"], "RECOVERY_ROOT_ISSUED_INVALID") - expires = _dt(s["expires"], "RECOVERY_ROOT_EXPIRES_INVALID") - skew = timedelta(minutes=int(POLICY["max_clock_skew_minutes"])) - if issued > now + skew: - _fail("RECOVERY_ROOT_ISSUED_FROM_FUTURE") - if expires <= issued or expires - issued > timedelta( - days=int(POLICY["max_recovery_root_lifetime_days"]) - ): - _fail("RECOVERY_ROOT_LIFETIME_INVALID") - if expires - now < timedelta(days=int(POLICY["min_recovery_root_remaining_days"])): - _fail("RECOVERY_ROOT_FREEZE_RISK") - kr = s["keys"] - if not isinstance(kr, dict) or not ( - int(POLICY["min_recovery_keys"]) <= len(kr) <= int(POLICY["max_keys"]) - ): - _fail("RECOVERY_ROOT_KEYS_INVALID") - keys = { - _identity(k, "RECOVERY_ROOT_KEYID_INVALID"): _recovery_key( - v, "RECOVERY_ROOT_KEY" - ) - for k, v in kr.items() - } - if list(kr) != sorted(kr): - _fail("RECOVERY_ROOT_KEYS_NOT_NORMALIZED") - role = _role( - s["role"], - keys, - "RECOVERY_ROLE", - min_keys=int(POLICY["min_recovery_keys"]), - min_threshold=int(POLICY["min_recovery_threshold"]), - min_ops=int(POLICY["min_recovery_operators"]), - ) - for kid in role["keyids"]: - if _dt(keys[kid]["expires"], "RECOVERY_ROOT_KEY_EXPIRES_INVALID") < expires: - _fail("RECOVERY_ROOT_KEY_EXPIRES_BEFORE_ROOT") - min_channels = _int( - s["minChannels"], - "RECOVERY_ROOT_CHANNEL_COUNT_INVALID", - minimum=int(POLICY["min_recovery_channels"]), - ) - if min_channels > len(role["keyids"]): - _fail("RECOVERY_ROOT_CHANNEL_COUNT_INVALID") - if len({keys[k]["recoveryChannel"] for k in role["keyids"]}) < min_channels: - _fail("RECOVERY_ROOT_CHANNEL_DIVERSITY_INVALID") - if bool(POLICY["require_hardware_profile_for_recovery"]) and not any( - keys[k]["signerProfile"] in set(POLICY["hardware_signer_profiles"]) - for k in role["keyids"] - ): - _fail("RECOVERY_ROOT_HARDWARE_PROFILE_REQUIRED") - _verify_role_signatures(doc, s, role, keys, at=issued, code="RECOVERY_ROOT") - return { - "doc": doc, - "raw": raw, - "sha256": _sha_bytes(raw), - "version": 1, - "keys": keys, - "role": role, - "min_channels": min_channels, - "governance_id": s["governanceId"], - } - - -def _replacement_root( # ruff: ignore[too-many-branches] - path: Path, current_root: dict[str, Any], compromised: list[str], *, now: datetime -) -> dict[str, Any]: - doc, _ = _read(path, "RECOVERY_REPLACEMENT_ROOT") - try: - info = rootseal._validate_root_envelope(doc) - except rootseal.RootTrustError as exc: - raise DelegatedTrustError( - "RECOVERY_REPLACEMENT_ROOT_INVALID:" + str(exc) - ) from exc - if info["version"] != current_root["version"] + 1: - _fail("RECOVERY_REPLACEMENT_ROOT_VERSION_INVALID") - if info["signed"]["governanceId"] != current_root["info"]["signed"]["governanceId"]: - _fail("RECOVERY_REPLACEMENT_GOVERNANCE_MISMATCH") - # Recovery changes root authority only; governance and emergency policy remain unchanged. - if bool(POLICY["require_recovery_role_only_change"]): - for role in ("governance", "emergency"): - old = current_root["info"]["roles"][role] - new = info["roles"][role] - if old != new: - _fail("RECOVERY_REPLACEMENT_POLICY_ROLE_CHANGED") - for kid in old["keyids"]: - if current_root["info"]["keys"][kid] != info["keys"][kid]: - _fail("RECOVERY_REPLACEMENT_POLICY_KEY_CHANGED") - if not compromised or any( - k not in current_root["info"]["roles"]["root"]["keyids"] for k in compromised - ): - _fail("RECOVERY_COMPROMISED_KEY_SET_INVALID") - if set(compromised) & set(info["roles"]["root"]["keyids"]): - _fail("RECOVERY_COMPROMISED_KEY_REUSED") - # New root self-threshold is mandatory; old-root signatures are intentionally not required. - sigs = _sig_map(doc, "RECOVERY_REPLACEMENT_ROOT") - allowed = set(info["roles"]["root"]["keyids"]) - if not set(sigs).issubset(allowed): - _fail("RECOVERY_REPLACEMENT_UNAUTHORIZED_SIGNATURE_KEY") - valid = [] - raw = _canonical(info["signed"]) - issued = info["issued"] - skew = timedelta(minutes=int(POLICY["max_clock_skew_minutes"])) - if issued > now + skew: - _fail("RECOVERY_REPLACEMENT_ISSUED_FROM_FUTURE") - for kid in info["roles"]["root"]["keyids"]: - if kid not in sigs: - continue - key = info["keys"][kid] - pub = _b64( - key["keyval"]["public"], - "RECOVERY_REPLACEMENT_PUBLIC_INVALID", - expected_len=32, - ) - sig = _b64(sigs[kid], "RECOVERY_REPLACEMENT_SIG_INVALID", expected_len=64) - try: - rootseal._ed25519_verify( - pub, sig, raw, "RECOVERY_REPLACEMENT_SIGNATURE_INVALID" - ) - except rootseal.RootTrustError as exc: - raise DelegatedTrustError(str(exc)) from exc - valid.append(kid) - if len(valid) < info["roles"]["root"]["threshold"]: - _fail("RECOVERY_REPLACEMENT_NEW_ROOT_THRESHOLD_NOT_MET") - info["envelope"] = doc - info["sha256"] = _sha_bytes(_canonical(doc)) - return info - - -def recover_root( # ruff: ignore[too-many-branches, undocumented-public-function] - *, - sealed_dir: Path, - bootstrap_root_sha256: str, - recovery_root_path: Path, - expected_recovery_root_sha256: str, - replacement_root_path: Path, - compromised_root_key_ids: list[str], - incident_id: str, - recovery_signature_paths: list[Path], - output_dir: Path, - now: datetime | None = None, -) -> dict[str, Any]: - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - sealed = _dir(sealed_dir, "RECOVERY_RUN155_DIR_INVALID") - active = _effective_root_from_run155(sealed, bootstrap_root_sha256, now=current) - rr = _verify_recovery_root( - recovery_root_path, expected_recovery_root_sha256, now=current - ) - if rr["governance_id"] != active["info"]["governance_id"]: - _fail("RECOVERY_ROOT_GOVERNANCE_MISMATCH") - active_keys = active["info"]["keys"] - if set(rr["keys"]) & set(active_keys): - _fail("RECOVERY_AUTHORITY_KEY_OVERLAP") - if {x["operator"] for x in rr["keys"].values()} & { - x["operator"] for x in active_keys.values() - }: - _fail("RECOVERY_AUTHORITY_OPERATOR_OVERLAP") - compromised = sorted( - { - _identity(x, "RECOVERY_COMPROMISED_KEY_INVALID") - for x in compromised_root_key_ids - } - ) - replacement = _replacement_root( - replacement_root_path, active, compromised, now=current - ) - replacement_root_keys = { - replacement["keys"][k]["operator"] - for k in replacement["roles"]["root"]["keyids"] - } - recovery_ops = {rr["keys"][k]["operator"] for k in rr["role"]["keyids"]} - if set(replacement["roles"]["root"]["keyids"]) & set(rr["role"]["keyids"]): - _fail("RECOVERY_REPLACEMENT_KEY_OVERLAP") - if replacement_root_keys & recovery_ops: - _fail("RECOVERY_REPLACEMENT_OPERATOR_OVERLAP") - selected = sorted(rr["role"]["keyids"][: rr["role"]["threshold"]]) - subject = { - "schemaVersion": int(POLICY["schema_version"]), - "predicateType": PREDICATE_TYPE + "/root-recovery", - "governanceId": active["info"]["governance_id"], - "incidentId": _identity(incident_id, "RECOVERY_INCIDENT_ID_INVALID"), - "previousRoot": { - "version": active["version"], - "sha256": active["sha256"], - "chainHeadSha256": active["chainHeadSha256"], - }, - "replacementRoot": { - "version": replacement["version"], - "sha256": replacement["sha256"], - }, - "recoveryRoot": {"version": rr["version"], "sha256": rr["sha256"]}, - "compromisedRootKeyIds": compromised, - "selectedRecoveryKeyIds": selected, - } - subject_sha = _sha_bytes(_canonical(subject)) - docs = [] - seen = set() - operators = set() - channels = set() - hardware = False - if len(recovery_signature_paths) != len(selected): - _fail("RECOVERY_SIGNATURE_COUNT_INVALID") - for path in recovery_signature_paths: - doc, _ = _read(path, "RECOVERY_SIGNATURE") - if set(doc) != {"signature", "signed"} or not isinstance(doc["signed"], dict): - _fail("RECOVERY_SIGNATURE_SCHEMA_INVALID") - s = doc["signed"] - expected = { - "schemaVersion", - "keyId", - "identity", - "operator", - "channel", - "signerProfile", - "decision", - "subjectSha256", - "signedAt", - } - if ( - set(s) != expected - or s["schemaVersion"] != int(POLICY["recovery_signature_schema_version"]) - or s["decision"] != "recover" - or s["subjectSha256"] != subject_sha - ): - _fail("RECOVERY_SIGNATURE_SIGNED_INVALID") - kid = _identity(s["keyId"], "RECOVERY_SIGNATURE_KEYID_INVALID") - if kid not in selected or kid in seen: - _fail("RECOVERY_SIGNATURE_KEY_SET_INVALID") - seen.add(kid) - key = rr["keys"][kid] - if ( - s["identity"] != key["identity"] - or s["operator"] != key["operator"] - or s["signerProfile"] != key["signerProfile"] - ): - _fail("RECOVERY_SIGNATURE_IDENTITY_MISMATCH") - channel = _identity(s["channel"], "RECOVERY_SIGNATURE_CHANNEL_INVALID") - if channel != key["recoveryChannel"]: - _fail("RECOVERY_SIGNATURE_CHANNEL_MISMATCH") - channels.add(channel) - operators.add(key["operator"]) - hardware |= key["signerProfile"] in set(POLICY["hardware_signer_profiles"]) - signed_at = _dt(s["signedAt"], "RECOVERY_SIGNATURE_TIME_INVALID") - skew = timedelta(minutes=int(POLICY["max_clock_skew_minutes"])) - if signed_at > current + skew: - _fail("RECOVERY_SIGNATURE_FROM_FUTURE") - if current - signed_at > timedelta( - hours=int(POLICY["max_recovery_signature_age_hours"]) - ): - _fail("RECOVERY_SIGNATURE_STALE") - if _dt(key["expires"], "RECOVERY_SIGNATURE_KEY_EXPIRES") < signed_at: - _fail("RECOVERY_SIGNATURE_KEY_EXPIRED") - sig = _b64( - doc["signature"], "RECOVERY_SIGNATURE_ENCODING_INVALID", expected_len=64 - ) - pub = _b64( - key["keyval"]["public"], - "RECOVERY_SIGNATURE_PUBLIC_INVALID", - expected_len=32, - ) - try: - rootseal._ed25519_verify( - pub, sig, _canonical(s), "RECOVERY_SIGNATURE_CRYPTO_INVALID" - ) - except rootseal.RootTrustError as exc: - raise DelegatedTrustError(str(exc)) from exc - docs.append(doc) - if sorted(seen) != selected: - _fail("RECOVERY_SIGNATURE_SET_MISMATCH") - if len(operators) < int(POLICY["min_recovery_operators"]): - _fail("RECOVERY_OPERATOR_QUORUM_INVALID") - if len(channels) < rr["min_channels"]: - _fail("RECOVERY_CHANNEL_QUORUM_INVALID") - if bool(POLICY["require_hardware_profile_for_recovery"]) and not hardware: - _fail("RECOVERY_HARDWARE_SIGNER_REQUIRED") - docs.sort(key=lambda d: d["signed"]["keyId"]) - record = { - "schemaVersion": int(POLICY["recovery_record_schema_version"]), - "predicateType": PREDICATE_TYPE + "/root-recovery-record", - "status": "root-recovered", - "subject": subject, - "subjectSha256": subject_sha, - "recoveryRoot": rr["doc"], - "replacementRoot": replacement["envelope"], - "signatures": docs, - "observedChannels": sorted(channels), - "revokedRootKeyIds": compromised, - } - protected = [ - sealed, - recovery_root_path.resolve().parent, - replacement_root_path.resolve().parent, - ] + [p.resolve().parent for p in recovery_signature_paths] - target = _outside(output_dir, protected, "RECOVERY_OUTPUT_INSIDE_INPUT") - if target.exists() or target.is_symlink(): - _fail("RECOVERY_OUTPUT_ALREADY_EXISTS") - recovery_inputs = [ - recovery_root_path, - replacement_root_path, - *list(recovery_signature_paths), - ] - before = {"sealed/" + p.name: _artifact(p) for p in sealed.iterdir()} - before.update( - {"input/" + str(i): _artifact(p) for i, p in enumerate(recovery_inputs)} - ) - tmp = Path(tempfile.mkdtemp(prefix="run156-recovery-", dir=str(target.parent))) - try: - _write(tmp / "release-root-recovery-record.json", record) - _write(tmp / "recovered-effective-root.json", replacement["envelope"]) - receipt = { - "schemaVersion": int(POLICY["receipt_schema_version"]), - "status": "root-recovery-accepted", - "governanceId": subject["governanceId"], - "incidentId": subject["incidentId"], - "recoveryRecord": _artifact(tmp / "release-root-recovery-record.json"), - "effectiveRoot": _artifact(tmp / "recovered-effective-root.json"), - "recoveryRootSha256": rr["sha256"], - "revokedRootKeyIds": compromised, - } - _write(tmp / "release-root-recovery-receipt.json", receipt) - after = {"sealed/" + p.name: _artifact(p) for p in sealed.iterdir()} - after.update( - {"input/" + str(i): _artifact(p) for i, p in enumerate(recovery_inputs)} - ) - if before != after: - _fail("RECOVERY_INPUT_DRIFT") - tmp.replace(target) - except Exception: - shutil.rmtree(tmp, ignore_errors=True) - raise - return { - "ok": True, - "phase": "root-recovery-accepted", - "incident_id": subject["incidentId"], - "previous_root_version": active["version"], - "replacement_root_version": replacement["version"], - "replacement_root_sha256": replacement["sha256"], - "channels": len(channels), - } - - -def verify_delegated_output( # ruff: ignore[undocumented-public-function] - *, - output_dir: Path, - sealed_dir: Path, - bootstrap_root_sha256: str, - now: datetime | None = None, - require_fresh: bool = True, -) -> dict[str, Any]: - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - out = _dir(output_dir, "DELEGATED_VERIFY_DIR_INVALID") - allowed = { - "release-delegation-root.json", - "release-snapshot.json", - "release-timestamp.json", - "release-delegated-metadata-bundle.json", - _DOC_DELEGATED_METADATA_STATE, - "release-delegated-metadata-receipt.json", - } - if {p.name for p in out.iterdir()} != allowed: - _fail("DELEGATED_VERIFY_ALLOWLIST_MISMATCH") - effective = _effective_root_from_run155( - _dir(sealed_dir, "DELEGATED_VERIFY_RUN155_INVALID"), - bootstrap_root_sha256, - now=current, - require_fresh=require_fresh, - ) - delegation_now = current - if not require_fresh: - delegation_doc, _ = _read( - out / "release-delegation-root.json", "DELEGATED_VERIFY_DELEGATION_TIME" - ) - delegation_now = _dt( - delegation_doc["signed"]["issuedAt"], "DELEGATED_VERIFY_DELEGATION_ISSUED" - ) - delegation = _verify_delegation_root( - out / "release-delegation-root.json", effective, now=delegation_now - ) - artifacts = _sealed_artifacts(sealed_dir) - snapshot_body = { - "run155Root": { - "version": effective["version"], - "sha256": effective["sha256"], - "chainHeadSha256": effective["chainHeadSha256"], - }, - "sealedArtifacts": artifacts, - } - snapshot = _verify_metadata_envelope( - out / "release-snapshot.json", - kind="snapshot", - delegation=delegation, - now=current if require_fresh else snapshot_time(out / "release-snapshot.json"), - expected_body=snapshot_body, - ) - timestamp_body = { - "snapshot": { - "version": snapshot["version"], - "sha256": snapshot["sha256"], - "size": len(snapshot["raw"]), - "expires": snapshot["doc"]["signed"]["expires"], - }, - "run155Root": {"version": effective["version"], "sha256": effective["sha256"]}, - } - timestamp = _verify_metadata_envelope( - out / "release-timestamp.json", - kind="timestamp", - delegation=delegation, - now=current if require_fresh else snapshot_time(out / "release-timestamp.json"), - expected_body=timestamp_body, - ) - state, _ = _read(out / _DOC_DELEGATED_METADATA_STATE, "DELEGATED_VERIFY_STATE") - prev = _previous_state( - out / _DOC_DELEGATED_METADATA_STATE, - out / "release-delegated-metadata-bundle.json", - ) - expected_effective = { - "version": effective["version"], - "sha256": effective["sha256"], - "chainHeadSha256": effective["chainHeadSha256"], - } - if ( - prev is None - or state["governanceId"] != effective["info"]["governance_id"] - or state["effectiveRoot"] != expected_effective - or state["delegationRoot"] - != {"version": delegation["version"], "sha256": delegation["sha256"]} - or state["snapshot"]["sha256"] != snapshot["sha256"] - or state["timestamp"]["sha256"] != timestamp["sha256"] - ): - _fail("DELEGATED_VERIFY_STATE_REBIND_FAILED") - receipt, _ = _read( - out / "release-delegated-metadata-receipt.json", "DELEGATED_VERIFY_RECEIPT" - ) - exp = { - "schemaVersion": int(POLICY["receipt_schema_version"]), - "status": "delegated-metadata-accepted", - "governanceId": state["governanceId"], - "sequence": len(prev["bundle"]["entries"]), - "delegationRoot": _artifact(out / "release-delegation-root.json"), - "snapshot": _artifact(out / "release-snapshot.json"), - "timestamp": _artifact(out / "release-timestamp.json"), - "bundle": _artifact(out / "release-delegated-metadata-bundle.json"), - "state": _artifact(out / _DOC_DELEGATED_METADATA_STATE), - } - if receipt != exp: - _fail("DELEGATED_VERIFY_RECEIPT_REBIND_FAILED") - return { - "ok": True, - "phase": "delegated-metadata-verified", - "sequence": len(prev["bundle"]["entries"]), - "chain_head_sha256": state["chainHeadSha256"], - } - - -def snapshot_time(path: Path) -> datetime: # ruff: ignore[undocumented-public-function] - doc, _ = _read(path, "DELEGATED_HISTORICAL_TIME") - return _dt(doc["signed"]["issuedAt"], "DELEGATED_HISTORICAL_ISSUED") - - -def verify_recovery_output( # ruff: ignore[too-many-branches, undocumented-public-function] - *, - output_dir: Path, - sealed_dir: Path, - bootstrap_root_sha256: str, - expected_recovery_root_sha256: str, - now: datetime | None = None, - historical: bool = False, -) -> dict[str, Any]: - out = _dir(output_dir, "RECOVERY_VERIFY_DIR_INVALID") - allowed = { - "release-root-recovery-record.json", - "recovered-effective-root.json", - "release-root-recovery-receipt.json", - } - if {p.name for p in out.iterdir()} != allowed: - _fail("RECOVERY_VERIFY_ALLOWLIST_MISMATCH") - record, record_raw = _read( - out / "release-root-recovery-record.json", "RECOVERY_VERIFY_RECORD" - ) - expected_keys = { - "schemaVersion", - "predicateType", - "status", - "subject", - "subjectSha256", - "recoveryRoot", - "replacementRoot", - "signatures", - "observedChannels", - "revokedRootKeyIds", - } - if ( - set(record) != expected_keys - or record["schemaVersion"] != int(POLICY["recovery_record_schema_version"]) - or record["predicateType"] != PREDICATE_TYPE + "/root-recovery-record" - or record["status"] != "root-recovered" - ): - _fail("RECOVERY_VERIFY_RECORD_SCHEMA_INVALID") - sigs = record.get("signatures") - if not isinstance(sigs, list) or not sigs: - _fail("RECOVERY_VERIFY_SIGNATURES_INVALID") - signed_times = [] - for d in sigs: - if not isinstance(d, dict) or not isinstance(d.get("signed"), dict): - _fail("RECOVERY_VERIFY_SIGNATURES_INVALID") - signed_times.append( - _dt(d["signed"].get("signedAt"), "RECOVERY_VERIFY_SIGNED_AT_INVALID") - ) - current = ( - max(signed_times) if historical else (now or datetime.now(timezone.utc)) - ).astimezone(timezone.utc) - active = _effective_root_from_run155( - _dir(sealed_dir, "RECOVERY_VERIFY_RUN155_INVALID"), - bootstrap_root_sha256, - now=current, - require_fresh=not historical, - ) - with tempfile.TemporaryDirectory(prefix="run156-recovery-verify-") as td: - td = Path(td) # ruff: ignore[redefined-loop-name] - rr_path = td / "recovery-root.json" - repl_path = td / "replacement-root.json" - _write(rr_path, record["recoveryRoot"]) - _write(repl_path, record["replacementRoot"]) - rr = _verify_recovery_root(rr_path, expected_recovery_root_sha256, now=current) - active_keys = active["info"]["keys"] - if set(rr["keys"]) & set(active_keys): - _fail("RECOVERY_VERIFY_AUTHORITY_KEY_OVERLAP") - if {x["operator"] for x in rr["keys"].values()} & { - x["operator"] for x in active_keys.values() - }: - _fail("RECOVERY_VERIFY_AUTHORITY_OPERATOR_OVERLAP") - subject = record.get("subject") - if not isinstance(subject, dict) or record.get("subjectSha256") != _sha_bytes( - _canonical(subject) - ): - _fail("RECOVERY_VERIFY_SUBJECT_HASH_INVALID") - required_subject = { - "schemaVersion", - "predicateType", - "governanceId", - "incidentId", - "previousRoot", - "replacementRoot", - "recoveryRoot", - "compromisedRootKeyIds", - "selectedRecoveryKeyIds", - } - if ( - set(subject) != required_subject - or subject["schemaVersion"] != int(POLICY["schema_version"]) - or subject["predicateType"] != PREDICATE_TYPE + "/root-recovery" - ): - _fail("RECOVERY_VERIFY_SUBJECT_SCHEMA_INVALID") - compromised = subject.get("compromisedRootKeyIds") - if not isinstance(compromised, list) or compromised != sorted(set(compromised)): - _fail("RECOVERY_VERIFY_COMPROMISED_SET_INVALID") - replacement = _replacement_root(repl_path, active, compromised, now=current) - replacement_ops = { - replacement["keys"][k]["operator"] - for k in replacement["roles"]["root"]["keyids"] - } - recovery_ops = {rr["keys"][k]["operator"] for k in rr["role"]["keyids"]} - if set(replacement["roles"]["root"]["keyids"]) & set(rr["role"]["keyids"]): - _fail("RECOVERY_VERIFY_REPLACEMENT_KEY_OVERLAP") - if replacement_ops & recovery_ops: - _fail("RECOVERY_VERIFY_REPLACEMENT_OPERATOR_OVERLAP") - expected_subject = { - "schemaVersion": int(POLICY["schema_version"]), - "predicateType": PREDICATE_TYPE + "/root-recovery", - "governanceId": active["info"]["governance_id"], - "incidentId": _identity( - subject.get("incidentId"), "RECOVERY_VERIFY_INCIDENT_INVALID" - ), - "previousRoot": { - "version": active["version"], - "sha256": active["sha256"], - "chainHeadSha256": active["chainHeadSha256"], - }, - "replacementRoot": { - "version": replacement["version"], - "sha256": replacement["sha256"], - }, - "recoveryRoot": {"version": rr["version"], "sha256": rr["sha256"]}, - "compromisedRootKeyIds": compromised, - "selectedRecoveryKeyIds": sorted( - rr["role"]["keyids"][: rr["role"]["threshold"]] - ), - } - if subject != expected_subject: - _fail("RECOVERY_VERIFY_SUBJECT_REBIND_FAILED") - selected = expected_subject["selectedRecoveryKeyIds"] - seen = set() - operators = set() - channels = set() - hardware = False - subject_sha = record["subjectSha256"] - normalized = [] - for doc in sigs: - if set(doc) != {"signature", "signed"} or not isinstance(doc["signed"], dict): - _fail("RECOVERY_VERIFY_SIGNATURE_SCHEMA_INVALID") - ss = doc["signed"] - expected_signed = { - "schemaVersion", - "keyId", - "identity", - "operator", - "channel", - "signerProfile", - "decision", - "subjectSha256", - "signedAt", - } - if ( - set(ss) != expected_signed - or ss["schemaVersion"] != int(POLICY["recovery_signature_schema_version"]) - or ss["decision"] != "recover" - or ss["subjectSha256"] != subject_sha - ): - _fail("RECOVERY_VERIFY_SIGNATURE_SIGNED_INVALID") - kid = _identity(ss["keyId"], "RECOVERY_VERIFY_KEYID_INVALID") - if kid not in selected or kid in seen: - _fail("RECOVERY_VERIFY_KEY_SET_INVALID") - seen.add(kid) - key = rr["keys"][kid] - if ( - ss["identity"] != key["identity"] - or ss["operator"] != key["operator"] - or ss["signerProfile"] != key["signerProfile"] - ): - _fail("RECOVERY_VERIFY_IDENTITY_MISMATCH") - channel = _identity(ss["channel"], "RECOVERY_VERIFY_CHANNEL_INVALID") - if channel != key["recoveryChannel"]: - _fail("RECOVERY_VERIFY_CHANNEL_MISMATCH") - channels.add(channel) - operators.add(key["operator"]) - hardware |= key["signerProfile"] in set(POLICY["hardware_signer_profiles"]) - signed_at = _dt(ss["signedAt"], "RECOVERY_VERIFY_TIME_INVALID") - if not historical: - skew = timedelta(minutes=int(POLICY["max_clock_skew_minutes"])) - if signed_at > current + skew or current - signed_at > timedelta( - hours=int(POLICY["max_recovery_signature_age_hours"]) - ): - _fail("RECOVERY_VERIFY_SIGNATURE_FRESHNESS_INVALID") - if _dt(key["expires"], "RECOVERY_VERIFY_KEY_EXPIRES") < signed_at: - _fail("RECOVERY_VERIFY_KEY_EXPIRED") - pub = _b64( - key["keyval"]["public"], "RECOVERY_VERIFY_PUBLIC_INVALID", expected_len=32 - ) - sig = _b64(doc["signature"], "RECOVERY_VERIFY_SIG_INVALID", expected_len=64) - try: - rootseal._ed25519_verify( - pub, sig, _canonical(ss), "RECOVERY_VERIFY_CRYPTO_INVALID" - ) - except rootseal.RootTrustError as exc: - raise DelegatedTrustError(str(exc)) from exc - normalized.append(doc) - normalized.sort(key=lambda d: d["signed"]["keyId"]) - if sigs != normalized or sorted(seen) != selected: - _fail("RECOVERY_VERIFY_SIGNATURE_SET_MISMATCH") - if ( - len(operators) < int(POLICY["min_recovery_operators"]) - or len(channels) < rr["min_channels"] - ): - _fail("RECOVERY_VERIFY_QUORUM_INVALID") - if bool(POLICY["require_hardware_profile_for_recovery"]) and not hardware: - _fail("RECOVERY_VERIFY_HARDWARE_REQUIRED") - if ( - record.get("observedChannels") != sorted(channels) - or record.get("revokedRootKeyIds") != compromised - ): - _fail("RECOVERY_VERIFY_RECORD_REBIND_FAILED") - effective_doc, effective_raw = _read( - out / "recovered-effective-root.json", "RECOVERY_VERIFY_EFFECTIVE_ROOT" - ) - if effective_doc != replacement["envelope"]: - _fail("RECOVERY_VERIFY_EFFECTIVE_ROOT_MISMATCH") - receipt, _ = _read( - out / "release-root-recovery-receipt.json", "RECOVERY_VERIFY_RECEIPT" - ) - exp_receipt = { - "schemaVersion": int(POLICY["receipt_schema_version"]), - "status": "root-recovery-accepted", - "governanceId": subject["governanceId"], - "incidentId": subject["incidentId"], - "recoveryRecord": _artifact_raw( - "release-root-recovery-record.json", record_raw - ), - "effectiveRoot": _artifact_raw("recovered-effective-root.json", effective_raw), - "recoveryRootSha256": rr["sha256"], - "revokedRootKeyIds": compromised, - } - if receipt != exp_receipt: - _fail("RECOVERY_VERIFY_RECEIPT_REBIND_FAILED") - return { - "ok": True, - "phase": "root-recovery-verified", - "incident_id": subject["incidentId"], - "replacement_root_version": replacement["version"], - "replacement_root_sha256": replacement["sha256"], - } - - -def _paths(values: list[str]) -> list[Path]: - return [Path(x) for x in values] - - -def main( # ruff: ignore[undocumented-public-function] - argv: list[str] | None = None, -) -> int: - p = argparse.ArgumentParser(description=__doc__) - sub = p.add_subparsers(dest="command", required=True) - r = sub.add_parser("refresh") - r.add_argument("--sealed-dir", type=Path, required=True) - r.add_argument("--bootstrap-root-sha256", required=True) - r.add_argument("--delegation-root", type=Path, required=True) - r.add_argument("--snapshot", type=Path, required=True) - r.add_argument("--timestamp", type=Path, required=True) - r.add_argument("--previous-state", type=Path) - r.add_argument("--previous-bundle", type=Path) - r.add_argument("--output-dir", type=Path, required=True) - q = sub.add_parser("recover-root") - q.add_argument("--sealed-dir", type=Path, required=True) - q.add_argument("--bootstrap-root-sha256", required=True) - q.add_argument("--recovery-root", type=Path, required=True) - q.add_argument("--recovery-root-sha256", required=True) - q.add_argument("--replacement-root", type=Path, required=True) - q.add_argument("--compromised-root-key", action="append", default=[]) - q.add_argument("--incident-id", required=True) - q.add_argument("--signature", action="append", default=[]) - q.add_argument("--output-dir", type=Path, required=True) - v = sub.add_parser("verify") - v.add_argument("--output-dir", type=Path, required=True) - v.add_argument("--sealed-dir", type=Path, required=True) - v.add_argument("--bootstrap-root-sha256", required=True) - v.add_argument("--historical", action="store_true") - w = sub.add_parser("verify-recovery") - w.add_argument("--output-dir", type=Path, required=True) - w.add_argument("--sealed-dir", type=Path, required=True) - w.add_argument("--bootstrap-root-sha256", required=True) - w.add_argument("--recovery-root-sha256", required=True) - w.add_argument("--historical", action="store_true") - a = p.parse_args(argv) - try: - if a.command == "refresh": - out = refresh_delegated_metadata( - sealed_dir=a.sealed_dir, - bootstrap_root_sha256=a.bootstrap_root_sha256, - delegation_root_path=a.delegation_root, - snapshot_path=a.snapshot, - timestamp_path=a.timestamp, - previous_state_path=a.previous_state, - previous_bundle_path=a.previous_bundle, - output_dir=a.output_dir, - ) - elif a.command == "recover-root": - out = recover_root( - sealed_dir=a.sealed_dir, - bootstrap_root_sha256=a.bootstrap_root_sha256, - recovery_root_path=a.recovery_root, - expected_recovery_root_sha256=a.recovery_root_sha256, - replacement_root_path=a.replacement_root, - compromised_root_key_ids=a.compromised_root_key, - incident_id=a.incident_id, - recovery_signature_paths=_paths(a.signature), - output_dir=a.output_dir, - ) - elif a.command == "verify": - out = verify_delegated_output( - output_dir=a.output_dir, - sealed_dir=a.sealed_dir, - bootstrap_root_sha256=a.bootstrap_root_sha256, - require_fresh=not a.historical, - ) - else: - out = verify_recovery_output( - output_dir=a.output_dir, - sealed_dir=a.sealed_dir, - bootstrap_root_sha256=a.bootstrap_root_sha256, - expected_recovery_root_sha256=a.recovery_root_sha256, - historical=a.historical, - ) - except DelegatedTrustError as exc: - logger.error("delegated trust operation failed: %s", exc) - return 2 - logger.info("delegated trust operation completed: %s", a.command) - # sys.stdout.write(json.dumps(out, sort_keys=True) + "\n") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/security/preserve_archive_merkle_rebridge_history.py b/security/preserve_archive_merkle_rebridge_history.py deleted file mode 100644 index 5181645347e7c1684e87e8559a3648dec5687c84..0000000000000000000000000000000000000000 --- a/security/preserve_archive_merkle_rebridge_history.py +++ /dev/null @@ -1,1499 +0,0 @@ -""" -Run 168: durably preserve and recover complete Run 167 recursive Merkle authority state. - -Preservation first executes the full Run 167 verifier. It then freezes the exact four -canonical Run 167 artifacts into one deterministic recovery checkpoint, replicates that -checkpoint to multiple independently operated immutable archives, and requires separate -read-only verification of every accepted copy. - -Recovery does not trust majority voting over differing bytes. Every observed archive -must return the same canonical checkpoint and callers must additionally pin the checkpoint -SHA-256, Run 167 authority head, Merkle continuity head, active-authority SHA-256, and -sequence out of band. A recovered environment can therefore reconstruct the current -log/gossip authority, permanent revocations, last RFC6962 checkpoints, and all Run 167 -history without any intermediate online service state. -""" - -from __future__ import annotations - -import argparse -import base64 -import hashlib -import json -import logging -import os -import re -import shutil -import stat -import subprocess -import sys -import tempfile -import threading -from datetime import datetime, timedelta, timezone -from pathlib import Path -from typing import Any, Callable, Iterable -from urllib.parse import urlsplit - -import tomllib - -logger = logging.getLogger(__name__) - -try: - from . import rebridge_archive_merkle_authority as run167 -except (ImportError, ValueError) as exc: - import importlib.util - - _here = Path(__file__).resolve().parent - _spec = importlib.util.spec_from_file_location( - "_run167_rebridge_for_recovery", _here / "rebridge_archive_merkle_authority.py" - ) - if _spec is None or _spec.loader is None: - raise ImportError("rebridge_archive_merkle_authority.py") from exc - run167 = importlib.util.module_from_spec(_spec) - sys.modules[_spec.name] = run167 - _spec.loader.exec_module(run167) - -HERE = Path(__file__).resolve().parent -POLICY = tomllib.loads( - (HERE / "release_archive_merkle_recovery_policy.toml").read_text() -) -PREDICATE_TYPE = str(POLICY["predicate_type"]) -_HEX64 = re.compile(r"^[0-9a-f]{64}$") -_ID = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:/@+-]{0,511}$") -_CHUNK = 1024 * 1024 - - -# ── Verified-state document names ──────────────────────────────────────── -# -# One constant per release-evidence document. The values are the on-disk -# artifact names and are unchanged; only the repetition is removed. Reading -# a document by a name that says which document it is also keeps static -# analysis from reading the filename token 'trusted' as 'confidential': -# these files carry published Merkle roots and SHA-256 digests, which must -# stay in clear text for any third party to verify them. -_DOC_ARCHIVE_LOG_AUTHORITY_STATE = "trusted-archive-log-authority-state.json" -_DOC_ARCHIVE_MERKLE_CONTINUITY_STATE = "trusted-archive-merkle-continuity-state.json" -_DOC_ARCHIVE_MERKLE_REBRIDGE_STATE = "trusted-archive-merkle-rebridge-state.json" -_DOC_ARCHIVE_MERKLE_RECOVERY_STATE = "trusted-archive-merkle-recovery-state.json" - - -_RUN167_NAMES = { - "release-archive-merkle-rebridge-bundle.json", - _DOC_ARCHIVE_MERKLE_REBRIDGE_STATE, - "active-archive-merkle-rebridge.json", - "release-archive-merkle-rebridge-receipt.json", -} -_OUTPUT_NAMES = { - "release-archive-merkle-recovery-checkpoint.json", - _DOC_ARCHIVE_MERKLE_RECOVERY_STATE, - "release-archive-merkle-recovery-receipt.json", -} - -ArchiveAdapter = Callable[[dict[str, Any]], dict[str, Any]] -VerifierAdapter = Callable[[dict[str, Any]], dict[str, Any]] -RecoveryAdapter = Callable[[dict[str, Any]], dict[str, Any]] - - -class ArchiveMerkleRecoveryError(RuntimeError): - """Run 168 recovery durability invariant failed.""" - - -def _fail(code: str) -> None: - raise ArchiveMerkleRecoveryError(code) - - -def _canonical(value: Any) -> bytes: - return run167._canonical(value) - - -def _sha_bytes(raw: bytes) -> str: - return hashlib.sha256(raw).hexdigest() - - -def _sha(path: Path) -> str: - h = hashlib.sha256() - with Path(path).open("rb") as f: - for chunk in iter(lambda: f.read(_CHUNK), b""): - h.update(chunk) - return h.hexdigest() - - -def _write(path: Path, value: Any) -> None: - path.write_bytes(_canonical(value)) - - -def _no_dupes(pairs): - out = {} - for k, v in pairs: - if k in out: - _fail("ARCHIVE_MERKLE_RECOVERY_JSON_DUPLICATE_KEY") - out[k] = v - return out - - -def _loads(raw: bytes, code: str) -> dict[str, Any]: - if len(raw) > int(POLICY["max_json_bytes"]): - _fail(code + "_TOO_LARGE") - try: - value = json.loads(raw.decode("utf-8"), object_pairs_hook=_no_dupes) - except ArchiveMerkleRecoveryError: - raise - except Exception as exc: - raise ArchiveMerkleRecoveryError(code + "_JSON_INVALID") from exc - if not isinstance(value, dict): - _fail(code + "_SCHEMA_INVALID") - if _canonical(value) != raw: - _fail(code + "_NOT_CANONICAL") - return value - - -def _read_json(path: Path, code: str) -> tuple[dict[str, Any], bytes]: - path = Path(path) - try: - st = path.lstat() - except OSError: - _fail(code + "_MISSING") - if not stat.S_ISREG(st.st_mode) or st.st_nlink != 1: - _fail(code + "_NOT_REGULAR") - raw = path.read_bytes() - return _loads(raw, code), raw - - -def _identity(value: Any, code: str) -> str: - if not isinstance(value, str): - _fail(code) - value = value.strip() - if ( - not value - or len(value) > 512 # ruff: ignore[magic-value-comparison] - or ".." in value - or "?" in value - or "#" in value - or "\x00" in value - ): - _fail(code) - if ( - any( - ord(c) < 32 # ruff: ignore[magic-value-comparison] - or ord(c) == 127 # ruff: ignore[magic-value-comparison] - for c in value - ) - or _ID.fullmatch(value) is None - ): - _fail(code) - return value - - -def _hex(value: Any, code: str) -> str: - if not isinstance(value, str) or _HEX64.fullmatch(value) is None: - _fail(code) - return value - - -def _positive_int(value: Any, code: str) -> int: - if not isinstance(value, int) or isinstance(value, bool) or value <= 0: - _fail(code) - return value - - -def _size(value: Any, code: str) -> int: - if not isinstance(value, int) or isinstance(value, bool) or value < 0: - _fail(code) - return value - - -def _dt(value: Any, code: str) -> datetime: - if not isinstance(value, str) or not value.endswith("Z"): - _fail(code) - try: - parsed = datetime.fromisoformat(value[:-1] + "+00:00") - except ValueError as exc: - raise ArchiveMerkleRecoveryError(code) from exc - if parsed.tzinfo is None: - _fail(code) - return parsed.astimezone(timezone.utc) - - -def _fresh(value: Any, code: str, *, now: datetime, historical: bool = False) -> str: - parsed = _dt(value, code) - if historical: - return value - skew = timedelta(minutes=int(POLICY["max_clock_skew_minutes"])) - age = timedelta(minutes=int(POLICY["max_result_age_minutes"])) - if parsed > now + skew: - _fail(code + "_FROM_FUTURE") - if now - parsed > age: - _fail(code + "_STALE") - return value - - -def _regular_dir(path: Path, code: str) -> Path: - path = Path(path) - try: - st = path.lstat() - except OSError: - _fail(code) - if not stat.S_ISDIR(st.st_mode): - _fail(code) - return path.resolve() - - -def _outside(path: Path, protected: Iterable[Path], code: str) -> Path: - target = Path(path).expanduser().absolute() - for item in protected: - root = Path(item).absolute() - if target == root or root in target.parents or target in root.parents: - _fail(code) - return target - - -def _safe_locator(value: Any, code: str) -> str: - if ( - not isinstance(value, str) - or not value - or len(value) > 2048 # ruff: ignore[magic-value-comparison] - or "\x00" in value - ): - _fail(code) - try: - parsed = urlsplit(value) - except ValueError: - _fail(code) - if ( - not parsed.scheme - or parsed.scheme.lower() == "file" - or parsed.query - or parsed.fragment - or parsed.username is not None - or parsed.password is not None - ): - _fail(code) - if ".." in parsed.path.split("/"): - _fail(code) - return value - - -def _artifact(name: str, raw: bytes) -> dict[str, Any]: - return {"name": name, "sha256": _sha_bytes(raw), "size": len(raw)} - - -def _artifact_doc(value: Any, code: str) -> dict[str, Any]: - if not isinstance(value, dict) or set(value) != {"name", "sha256", "size"}: - _fail(code + "_SCHEMA_INVALID") - name = value.get("name") - if not isinstance(name, str) or Path(name).name != name or not name: - _fail(code + "_NAME_INVALID") - return { - "name": name, - "sha256": _hex(value.get("sha256"), code + "_SHA_INVALID"), - "size": _size(value.get("size"), code + "_SIZE_INVALID"), - } - - -def _dir_fingerprint(root: Path, code: str) -> tuple[tuple[str, str, int, int], ...]: - root = _regular_dir(root, code) - rows = [] - for p in sorted(root.rglob("*")): - st = p.lstat() - if stat.S_ISDIR(st.st_mode): - continue - if not stat.S_ISREG(st.st_mode) or st.st_nlink != 1: - _fail(code) - rows.append( - ( - p.relative_to(root).as_posix(), - _sha(p), - st.st_size, - stat.S_IMODE(st.st_mode), - ) - ) - return tuple(rows) - - -def _authority_fingerprint( - run167_dir: Path, verify_kwargs: dict[str, Any] -) -> tuple[Any, ...]: - rows: list[Any] = [ - ( - "run167", - _dir_fingerprint(run167_dir, "ARCHIVE_MERKLE_RECOVERY_INPUT_INVALID"), - ) - ] - seen: set[str] = set() - for key, value in sorted(verify_kwargs.items()): - if key in {"output_dir", "now", "historical"}: - continue - values = value if isinstance(value, list) else [value] - for idx, item in enumerate(values): - if not isinstance(item, Path): - continue - ap = str(item.absolute()) - token = f"{key}:{idx}:{ap}" - if token in seen: - continue - seen.add(token) - if item.is_dir(): - rows.append( - ( - key, - idx, - ap, - "dir", - _dir_fingerprint( - item, "ARCHIVE_MERKLE_RECOVERY_AUTHORITY_INPUT_INVALID" - ), - ) - ) - else: - try: - st = item.lstat() - except OSError: - _fail("ARCHIVE_MERKLE_RECOVERY_AUTHORITY_INPUT_INVALID") - if not stat.S_ISREG(st.st_mode) or st.st_nlink != 1: - _fail("ARCHIVE_MERKLE_RECOVERY_AUTHORITY_INPUT_INVALID") - rows.append( - ( - key, - idx, - ap, - "file", - _sha(item), - st.st_size, - stat.S_IMODE(st.st_mode), - ) - ) - return tuple(rows) - - -def _run167_docs(root: Path) -> tuple[dict[str, dict[str, Any]], dict[str, bytes]]: - root = _regular_dir(root, "ARCHIVE_MERKLE_RECOVERY_RUN167_DIR_INVALID") - if {p.name for p in root.iterdir()} != _RUN167_NAMES: - _fail("ARCHIVE_MERKLE_RECOVERY_RUN167_ALLOWLIST_INVALID") - docs: dict[str, dict[str, Any]] = {} - raws: dict[str, bytes] = {} - for name in sorted(_RUN167_NAMES): - docs[name], raws[name] = _read_json( - root / name, "ARCHIVE_MERKLE_RECOVERY_RUN167" - ) - return docs, raws - - -def _artifact_map(raws: dict[str, bytes]) -> dict[str, dict[str, Any]]: - return {name: _artifact(name, raws[name]) for name in sorted(raws)} - - -def _verify_snapshot_documents( # ruff: ignore[too-many-branches] - docs: dict[str, dict[str, Any]], raws: dict[str, bytes] -) -> dict[str, Any]: - if set(docs) != _RUN167_NAMES or set(raws) != _RUN167_NAMES: - _fail("ARCHIVE_MERKLE_RECOVERY_RUN167_DOCUMENT_SET_INVALID") - bundle = docs["release-archive-merkle-rebridge-bundle.json"] - state = docs[_DOC_ARCHIVE_MERKLE_REBRIDGE_STATE] - active = docs["active-archive-merkle-rebridge.json"] - receipt = docs["release-archive-merkle-rebridge-receipt.json"] - events = bundle.get("events") - recs = receipt.get("events") - if ( - not isinstance(events, list) - or not events - or not isinstance(recs, list) - or len(events) != len(recs) - ): - _fail("ARCHIVE_MERKLE_RECOVERY_HISTORY_INVALID") - braw = raws["release-archive-merkle-rebridge-bundle.json"] - if state.get("bundleArtifact") != _artifact( - "release-archive-merkle-rebridge-bundle.json", braw - ): - _fail("ARCHIVE_MERKLE_RECOVERY_BUNDLE_REBIND_INVALID") - last = events[-1] - if active != run167._make_active(state, last): - _fail("ARCHIVE_MERKLE_RECOVERY_ACTIVE_REBIND_INVALID") - sequence = _positive_int( - state.get("sequence"), "ARCHIVE_MERKLE_RECOVERY_SEQUENCE_INVALID" - ) - if last.get("sequence") != sequence or active.get("sequence") != sequence: - _fail("ARCHIVE_MERKLE_RECOVERY_SEQUENCE_MISMATCH") - base_sequence = _positive_int( - bundle.get("baseRun166Sequence"), - "ARCHIVE_MERKLE_RECOVERY_BASE_SEQUENCE_INVALID", - ) - if sequence != base_sequence + len(events): - _fail("ARCHIVE_MERKLE_RECOVERY_SEQUENCE_CHAIN_INVALID") - run165_docs = receipt.get("run165Documents") - run166_docs = receipt.get("run166Documents") - if not isinstance(run165_docs, dict) or not isinstance(run166_docs, dict): - _fail("ARCHIVE_MERKLE_RECOVERY_BASE_DOCUMENTS_INVALID") - try: - r165_state = run165_docs[_DOC_ARCHIVE_LOG_AUTHORITY_STATE] - r166_state = run166_docs[_DOC_ARCHIVE_MERKLE_CONTINUITY_STATE] - r166_active = run166_docs["active-archive-merkle-continuity.json"] - except Exception: # ruff: ignore[blind-except] - _fail("ARCHIVE_MERKLE_RECOVERY_BASE_DOCUMENTS_INVALID") - r165_raws = {name: _canonical(value) for name, value in run165_docs.items()} - r166_raws = {name: _canonical(value) for name, value in run166_docs.items()} - if bundle.get("run165Artifacts") != run167._artifact_map(r165_raws) or bundle.get( - "baseRun166Artifacts" - ) != run167._artifact_map(r166_raws): - _fail("ARCHIVE_MERKLE_RECOVERY_BASE_ARTIFACT_REBIND_INVALID") - if bundle.get("run165Sequence") != r165_state.get("sequence") or bundle.get( - "run165LogAuthorityChainHeadSha256" - ) != r165_state.get("logAuthorityChainHeadSha256"): - _fail("ARCHIVE_MERKLE_RECOVERY_RUN165_REBIND_INVALID") - if bundle.get("baseRun166Sequence") != r166_state.get("sequence") or bundle.get( - "baseRun166MerkleAuthorityContinuityHeadSha256" - ) != r166_state.get("merkleAuthorityContinuityHeadSha256"): - _fail("ARCHIVE_MERKLE_RECOVERY_RUN166_REBIND_INVALID") - r165_info = {"state": r165_state} - r166_info = {"state": r166_state, "active": r166_active, "raws": r166_raws} - authority_head = run167._base_authority_head(r165_info, r166_info) - merkle_head = bundle["baseRun166MerkleAuthorityContinuityHeadSha256"] - for idx, (event, rec) in enumerate(zip(events, recs), 1): - if ( - not isinstance(event, dict) - or not isinstance(rec, dict) - or event.get("sequence") != base_sequence + idx - or rec.get("sequence") != event.get("sequence") - ): - _fail("ARCHIVE_MERKLE_RECOVERY_EVENT_SEQUENCE_INVALID") - action = event.get("action") - if action == "rebridge": - td = rec.get("transitionDocument") - hd = rec.get("handoffProofDocument") - if not isinstance(td, dict) or not isinstance(hd, dict): - _fail("ARCHIVE_MERKLE_RECOVERY_HANDOFF_DOCUMENT_INVALID") - tsha = _sha_bytes(_canonical(td)) - hsha = _sha_bytes(_canonical(hd)) - if ( - event.get("transitionSha256") != tsha - or event.get("handoffSha256") != hsha - ): - _fail("ARCHIVE_MERKLE_RECOVERY_HANDOFF_HASH_INVALID") - authority_head = run167._bridge_head( - authority_head, - tsha, - hsha, - _hex( - event.get("sourceTrustedStateSha256"), - "ARCHIVE_MERKLE_RECOVERY_SOURCE_STATE_INVALID", - ), - ) - elif action == "append": - if ( - rec.get("transitionDocument") is not None - or rec.get("handoffProofDocument") is not None - ): - _fail("ARCHIVE_MERKLE_RECOVERY_APPEND_HANDOFF_INVALID") - else: - _fail("ARCHIVE_MERKLE_RECOVERY_ACTION_INVALID") - if event.get("archiveMerkleRebridgeAuthorityHeadSha256") != authority_head: - _fail("ARCHIVE_MERKLE_RECOVERY_AUTHORITY_HEAD_INVALID") - bare = {k: event[k] for k in event if k != "merkleRebridgeContinuityHeadSha256"} - merkle_head = run167._merkle_head(merkle_head, bare) - if event.get("merkleRebridgeContinuityHeadSha256") != merkle_head: - _fail("ARCHIVE_MERKLE_RECOVERY_MERKLE_HEAD_INVALID") - if ( - state.get("archiveMerkleRebridgeAuthorityHeadSha256") != authority_head - or active.get("archiveMerkleRebridgeAuthorityHeadSha256") != authority_head - ): - _fail("ARCHIVE_MERKLE_RECOVERY_FINAL_AUTHORITY_HEAD_INVALID") - if ( - state.get("merkleRebridgeContinuityHeadSha256") != merkle_head - or active.get("merkleRebridgeContinuityHeadSha256") != merkle_head - ): - _fail("ARCHIVE_MERKLE_RECOVERY_FINAL_MERKLE_HEAD_INVALID") - authority = state.get("activeAuthority") - if not isinstance(authority, dict) or active.get("authority") != authority: - _fail("ARCHIVE_MERKLE_RECOVERY_ACTIVE_AUTHORITY_INVALID") - active_sha = _sha_bytes(_canonical(authority)) - if ( - state.get("activeAuthoritySha256") != active_sha - or active.get("activeAuthoritySha256") != active_sha - ): - _fail("ARCHIVE_MERKLE_RECOVERY_ACTIVE_AUTHORITY_HASH_INVALID") - revoked = state.get("revokedKeyFingerprints") - if ( - not isinstance(revoked, list) - or revoked != sorted(set(revoked)) - or any(_HEX64.fullmatch(x) is None for x in revoked if isinstance(x, str)) - or any(not isinstance(x, str) for x in revoked) - ): - _fail("ARCHIVE_MERKLE_RECOVERY_REVOKED_SET_INVALID") - checkpoints = state.get("lastCheckpoints") - if not isinstance(checkpoints, dict) or set(checkpoints) != set(authority): - _fail("ARCHIVE_MERKLE_RECOVERY_CHECKPOINT_SET_INVALID") - log_rows = last.get("logs") - if not isinstance(log_rows, list): - _fail("ARCHIVE_MERKLE_RECOVERY_LAST_LOG_ROWS_INVALID") - from_rows = { - r.get("logId"): { - k: r.get(k) for k in ("checkpointSha256", "treeSize", "rootHash") - } - for r in log_rows - if isinstance(r, dict) - } - if checkpoints != from_rows: - _fail("ARCHIVE_MERKLE_RECOVERY_LAST_CHECKPOINT_REBIND_INVALID") - return { - "sequence": sequence, - "postRun166Sequence": state.get("postRun166Sequence"), - "rebridgeSequence": state.get("rebridgeSequence"), - "authorityHead": authority_head, - "merkleHead": merkle_head, - "activeAuthority": authority, - "activeAuthoritySha256": active_sha, - "revokedKeyFingerprints": revoked, - "lastCheckpoints": checkpoints, - "run165Sequence": state.get("run165Sequence"), - "run165LogAuthorityChainHeadSha256": state.get( - "run165LogAuthorityChainHeadSha256" - ), - } - - -def _checkpoint_head(checkpoint_without_head: dict[str, Any]) -> str: - return _sha_bytes( - _canonical( - { - "domain": "run168-archive-merkle-recovery-checkpoint", - "checkpoint": checkpoint_without_head, - } - ) - ) - - -def _make_checkpoint( - docs: dict[str, dict[str, Any]], - raws: dict[str, bytes], - summary: dict[str, Any], - membership: list[dict[str, str]], -) -> dict[str, Any]: - body = { - "schemaVersion": int(POLICY["checkpoint_schema_version"]), - "predicateType": PREDICATE_TYPE, - "status": "archive-merkle-rebridge-recovery-checkpoint", - "run167Artifacts": _artifact_map(raws), - "run167Documents": {name: docs[name] for name in sorted(docs)}, - "summary": summary, - "archiveMembership": membership, - } - return {**body, "recoveryCheckpointHeadSha256": _checkpoint_head(body)} - - -def _verify_checkpoint(doc: dict[str, Any]) -> dict[str, Any]: - expected = { - "schemaVersion", - "predicateType", - "status", - "run167Artifacts", - "run167Documents", - "summary", - "archiveMembership", - "recoveryCheckpointHeadSha256", - } - if ( - set(doc) != expected - or doc.get("schemaVersion") != int(POLICY["checkpoint_schema_version"]) - or doc.get("predicateType") != PREDICATE_TYPE - or doc.get("status") != "archive-merkle-rebridge-recovery-checkpoint" - ): - _fail("ARCHIVE_MERKLE_RECOVERY_CHECKPOINT_SCHEMA_INVALID") - body = {k: doc[k] for k in doc if k != "recoveryCheckpointHeadSha256"} - if doc["recoveryCheckpointHeadSha256"] != _checkpoint_head(body): - _fail("ARCHIVE_MERKLE_RECOVERY_CHECKPOINT_HEAD_INVALID") - documents = doc.get("run167Documents") - if not isinstance(documents, dict) or set(documents) != _RUN167_NAMES: - _fail("ARCHIVE_MERKLE_RECOVERY_CHECKPOINT_DOCUMENTS_INVALID") - raws = {name: _canonical(documents[name]) for name in sorted(documents)} - if doc.get("run167Artifacts") != _artifact_map(raws): - _fail("ARCHIVE_MERKLE_RECOVERY_CHECKPOINT_ARTIFACTS_INVALID") - summary = _verify_snapshot_documents(documents, raws) - if doc.get("summary") != summary: - _fail("ARCHIVE_MERKLE_RECOVERY_CHECKPOINT_SUMMARY_INVALID") - membership = doc.get("archiveMembership") - if not isinstance(membership, list) or len(membership) < int( - POLICY["min_configured_archives"] - ): - _fail("ARCHIVE_MERKLE_RECOVERY_MEMBERSHIP_INVALID") - expected_keys = { - "archiveIdentity", - "archiveOperator", - "verifierIdentity", - "verifierOperator", - } - normalized = [] - for row in membership: - if not isinstance(row, dict) or set(row) != expected_keys: - _fail("ARCHIVE_MERKLE_RECOVERY_MEMBERSHIP_INVALID") - normalized.append( - { - k: _identity( - row[k], "ARCHIVE_MERKLE_RECOVERY_MEMBERSHIP_IDENTITY_INVALID" - ) - for k in sorted(row) - } - ) - if membership != sorted( - normalized, - key=lambda x: ( - x["archiveOperator"], - x["archiveIdentity"], - x["verifierOperator"], - x["verifierIdentity"], - ), - ): - _fail("ARCHIVE_MERKLE_RECOVERY_MEMBERSHIP_NOT_NORMALIZED") - if len({r["archiveIdentity"] for r in membership}) != len(membership) or len( - {r["verifierIdentity"] for r in membership} - ) != len(membership): - _fail("ARCHIVE_MERKLE_RECOVERY_MEMBERSHIP_IDENTITY_REUSE") - if len({r["archiveOperator"] for r in membership}) < int( - POLICY["min_archive_operators"] - ) or len({r["verifierOperator"] for r in membership}) < int( - POLICY["min_verifier_operators"] - ): - _fail("ARCHIVE_MERKLE_RECOVERY_MEMBERSHIP_OPERATOR_QUORUM_INVALID") - if {r["archiveIdentity"] for r in membership} & { - r["verifierIdentity"] for r in membership - } or {r["archiveOperator"] for r in membership} & { - r["verifierOperator"] for r in membership - }: - _fail("ARCHIVE_MERKLE_RECOVERY_AUTHORITY_PLANE_OVERLAP") - return { - "summary": summary, - "raws": raws, - "documents": documents, - "membership": membership, - } - - -def _archive_id(checkpoint_sha: str, identity: str) -> str: - return _sha_bytes( - _canonical({"checkpointSha256": checkpoint_sha, "archiveIdentity": identity}) - ) - - -def _validate_archive_response( - value: Any, - *, - operation: str, - archive_id: str, - identity: str, - operator: str, - artifact: dict[str, Any], - now: datetime, - historical: bool = False, -) -> dict[str, Any]: - expected = { - "schemaVersion", - "operation", - "status", - "archiveId", - "archiveIdentity", - "archiveOperator", - "artifact", - "locator", - "immutableVersionId", - "immutabilityClass", - "overwrite", - "readBackSha256", - "readBackSize", - "observedAt", - } - if ( - not isinstance(value, dict) - or set(value) != expected - or value.get("schemaVersion") != int(POLICY["archive_protocol_version"]) - or value.get("operation") != operation - ): - _fail("ARCHIVE_MERKLE_RECOVERY_ARCHIVE_RESPONSE_SCHEMA_INVALID") - if ( - value.get("status") not in {"created", "present"} - or value.get("archiveId") != archive_id - or value.get("archiveIdentity") != identity - or value.get("archiveOperator") != operator - ): - _fail("ARCHIVE_MERKLE_RECOVERY_ARCHIVE_RESPONSE_BINDING_INVALID") - if ( - value.get("artifact") != artifact - or value.get("overwrite") is not False - or value.get("readBackSha256") != artifact["sha256"] - or value.get("readBackSize") != artifact["size"] - ): - _fail("ARCHIVE_MERKLE_RECOVERY_ARCHIVE_READBACK_INVALID") - imm = value.get("immutabilityClass") - if imm not in set(POLICY["allowed_archive_immutability"]): - _fail("ARCHIVE_MERKLE_RECOVERY_ARCHIVE_IMMUTABILITY_INVALID") - return { - "schemaVersion": value["schemaVersion"], - "operation": operation, - "status": value["status"], - "archiveId": archive_id, - "archiveIdentity": identity, - "archiveOperator": operator, - "artifact": artifact, - "locator": _safe_locator( - value.get("locator"), "ARCHIVE_MERKLE_RECOVERY_ARCHIVE_LOCATOR_INVALID" - ), - "immutableVersionId": _identity( - value.get("immutableVersionId"), - "ARCHIVE_MERKLE_RECOVERY_ARCHIVE_VERSION_INVALID", - ), - "immutabilityClass": imm, - "overwrite": False, - "readBackSha256": artifact["sha256"], - "readBackSize": artifact["size"], - "observedAt": _fresh( - value.get("observedAt"), - "ARCHIVE_MERKLE_RECOVERY_ARCHIVE_OBSERVED_AT", - now=now, - historical=historical, - ), - } - - -def _validate_verifier_response( - value: Any, - *, - archive: dict[str, Any], - identity: str, - operator: str, - artifact: dict[str, Any], - now: datetime, - historical: bool = False, -) -> dict[str, Any]: - expected = { - "schemaVersion", - "status", - "verifierIdentity", - "verifierOperator", - "archiveId", - "archiveIdentity", - "archiveOperator", - "artifact", - "locator", - "immutableVersionId", - "readBackSha256", - "readBackSize", - "readOnly", - "archiveCredentialsReused", - "observedAt", - } - if ( - not isinstance(value, dict) - or set(value) != expected - or value.get("schemaVersion") != int(POLICY["verifier_protocol_version"]) - or value.get("status") != "verified" - ): - _fail("ARCHIVE_MERKLE_RECOVERY_VERIFIER_RESPONSE_SCHEMA_INVALID") - if ( - value.get("verifierIdentity") != identity - or value.get("verifierOperator") != operator - or value.get("archiveId") != archive["archiveId"] - or value.get("archiveIdentity") != archive["archiveIdentity"] - or value.get("archiveOperator") != archive["archiveOperator"] - ): - _fail("ARCHIVE_MERKLE_RECOVERY_VERIFIER_RESPONSE_BINDING_INVALID") - if ( - value.get("artifact") != artifact - or value.get("locator") != archive["locator"] - or value.get("immutableVersionId") != archive["immutableVersionId"] - or value.get("readBackSha256") != artifact["sha256"] - or value.get("readBackSize") != artifact["size"] - ): - _fail("ARCHIVE_MERKLE_RECOVERY_VERIFIER_READBACK_INVALID") - if ( - value.get("readOnly") is not True - or value.get("archiveCredentialsReused") is not False - ): - _fail("ARCHIVE_MERKLE_RECOVERY_VERIFIER_AUTHORITY_INVALID") - return { - "schemaVersion": value["schemaVersion"], - "status": "verified", - "verifierIdentity": identity, - "verifierOperator": operator, - "archiveId": archive["archiveId"], - "archiveIdentity": archive["archiveIdentity"], - "archiveOperator": archive["archiveOperator"], - "artifact": artifact, - "locator": archive["locator"], - "immutableVersionId": archive["immutableVersionId"], - "readBackSha256": artifact["sha256"], - "readBackSize": artifact["size"], - "readOnly": True, - "archiveCredentialsReused": False, - "observedAt": _fresh( - value.get("observedAt"), - "ARCHIVE_MERKLE_RECOVERY_VERIFIER_OBSERVED_AT", - now=now, - historical=historical, - ), - } - - -def _load_output(root: Path) -> tuple[dict[str, Any], dict[str, bytes]]: - root = _regular_dir(root, "ARCHIVE_MERKLE_RECOVERY_OUTPUT_INVALID") - if {p.name for p in root.iterdir()} != _OUTPUT_NAMES: - _fail("ARCHIVE_MERKLE_RECOVERY_OUTPUT_ALLOWLIST_INVALID") - docs = {} - raws = {} - for name in sorted(_OUTPUT_NAMES): - docs[name], raws[name] = _read_json( - root / name, "ARCHIVE_MERKLE_RECOVERY_OUTPUT" - ) - return docs, raws - - -def verify_recovery_archive( # ruff: ignore[undocumented-public-function] - *, output_dir: Path, now: datetime | None = None, historical: bool = False -) -> dict[str, Any]: - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - docs, raws = _load_output(output_dir) - checkpoint = docs["release-archive-merkle-recovery-checkpoint.json"] - info = _verify_checkpoint(checkpoint) - artifact = _artifact( - "release-archive-merkle-recovery-checkpoint.json", - raws["release-archive-merkle-recovery-checkpoint.json"], - ) - state = docs[_DOC_ARCHIVE_MERKLE_RECOVERY_STATE] - expected_state = { - "schemaVersion": int(POLICY["state_schema_version"]), - "status": "trusted-archive-merkle-recovery", - "sequence": info["summary"]["sequence"], - "rebridgeSequence": info["summary"]["rebridgeSequence"], - "archiveMerkleRebridgeAuthorityHeadSha256": info["summary"]["authorityHead"], - "merkleRebridgeContinuityHeadSha256": info["summary"]["merkleHead"], - "activeAuthoritySha256": info["summary"]["activeAuthoritySha256"], - "recoveryCheckpointHeadSha256": checkpoint["recoveryCheckpointHeadSha256"], - "checkpointArtifact": artifact, - "archiveMembership": checkpoint["archiveMembership"], - } - if state != expected_state: - _fail("ARCHIVE_MERKLE_RECOVERY_STATE_INVALID") - receipt = docs["release-archive-merkle-recovery-receipt.json"] - if ( - not isinstance(receipt, dict) - or set(receipt) != {"schemaVersion", "status", "checkpointArtifact", "archives"} - or receipt.get("schemaVersion") != int(POLICY["receipt_schema_version"]) - or receipt.get("status") != "archive-merkle-recovery-preserved" - or receipt.get("checkpointArtifact") != artifact - ): - _fail("ARCHIVE_MERKLE_RECOVERY_RECEIPT_SCHEMA_INVALID") - rows = receipt.get("archives") - if not isinstance(rows, list) or len(rows) < int(POLICY["min_successful_archives"]): - _fail("ARCHIVE_MERKLE_RECOVERY_RECEIPT_QUORUM_INVALID") - member_map = {r["archiveIdentity"]: r for r in checkpoint["archiveMembership"]} - locators = set() - archive_ops = set() - verifier_ops = set() - for row in rows: - if not isinstance(row, dict) or set(row) != { - "archiveResponse", - "verifierResponse", - }: - _fail("ARCHIVE_MERKLE_RECOVERY_RECEIPT_ROW_INVALID") - ar = row["archiveResponse"] - identity = ar.get("archiveIdentity") if isinstance(ar, dict) else None - if identity not in member_map: - _fail("ARCHIVE_MERKLE_RECOVERY_RECEIPT_MEMBER_INVALID") - member = member_map[identity] - parsed_a = _validate_archive_response( - ar, - operation="preserve", - archive_id=_archive_id(artifact["sha256"], member["archiveIdentity"]), - identity=member["archiveIdentity"], - operator=member["archiveOperator"], - artifact=artifact, - now=current, - historical=historical, - ) - parsed_v = _validate_verifier_response( - row["verifierResponse"], - archive=parsed_a, - identity=member["verifierIdentity"], - operator=member["verifierOperator"], - artifact=artifact, - now=current, - historical=historical, - ) - if parsed_a["locator"] in locators: - _fail("ARCHIVE_MERKLE_RECOVERY_LOCATOR_COLLISION") - locators.add(parsed_a["locator"]) - archive_ops.add(parsed_a["archiveOperator"]) - verifier_ops.add(parsed_v["verifierOperator"]) - if ( - len(rows) < int(POLICY["min_successful_archives"]) - or len(archive_ops) < int(POLICY["min_archive_operators"]) - or len(verifier_ops) < int(POLICY["min_verifier_operators"]) - ): - _fail("ARCHIVE_MERKLE_RECOVERY_RECEIPT_OPERATOR_QUORUM_INVALID") - return { - "ok": True, - "sequence": info["summary"]["sequence"], - "rebridge_sequence": info["summary"]["rebridgeSequence"], - "authority_head_sha256": info["summary"]["authorityHead"], - "merkle_head_sha256": info["summary"]["merkleHead"], - "active_authority_sha256": info["summary"]["activeAuthoritySha256"], - "checkpoint_sha256": artifact["sha256"], - "checkpoint_head_sha256": checkpoint["recoveryCheckpointHeadSha256"], - } - - -def preserve_rebridge_history( # ruff: ignore[undocumented-public-function] - *, - run167_dir: Path, - output_dir: Path, - verify_kwargs: dict[str, Any], - targets: list[tuple[str, str, ArchiveAdapter, str, str, VerifierAdapter]], - now: datetime | None = None, -) -> dict[str, Any]: - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - run167_dir = _regular_dir(run167_dir, "ARCHIVE_MERKLE_RECOVERY_RUN167_DIR_INVALID") - target = _outside( - output_dir, [run167_dir], "ARCHIVE_MERKLE_RECOVERY_OUTPUT_INVALID" - ) - if target.exists() or target.is_symlink(): - _fail("ARCHIVE_MERKLE_RECOVERY_OUTPUT_ALREADY_EXISTS") - normalized = [] - for ai, ao, aa, vi, vo, va in targets: - normalized.append( - ( - _identity(ai, "ARCHIVE_MERKLE_RECOVERY_ARCHIVE_IDENTITY_INVALID"), - _identity(ao, "ARCHIVE_MERKLE_RECOVERY_ARCHIVE_OPERATOR_INVALID"), - aa, - _identity(vi, "ARCHIVE_MERKLE_RECOVERY_VERIFIER_IDENTITY_INVALID"), - _identity(vo, "ARCHIVE_MERKLE_RECOVERY_VERIFIER_OPERATOR_INVALID"), - va, - ) - ) - if len(normalized) < int(POLICY["min_configured_archives"]): - _fail("ARCHIVE_MERKLE_RECOVERY_TARGET_PLAN_TOO_SMALL") - membership = sorted( - [ - { - "archiveIdentity": x[0], - "archiveOperator": x[1], - "verifierIdentity": x[3], - "verifierOperator": x[4], - } - for x in normalized - ], - key=lambda x: ( - x["archiveOperator"], - x["archiveIdentity"], - x["verifierOperator"], - x["verifierIdentity"], - ), - ) - # Reuse the checkpoint membership validator for identity/operator separation. - probe = { - "schemaVersion": int(POLICY["checkpoint_schema_version"]), - "predicateType": PREDICATE_TYPE, - "status": "archive-merkle-rebridge-recovery-checkpoint", - "run167Artifacts": {}, - "run167Documents": {}, - "summary": {}, - "archiveMembership": membership, - } - # Validate plan directly because a full checkpoint does not exist yet. - if len({r["archiveIdentity"] for r in membership}) != len(membership) or len( - {r["verifierIdentity"] for r in membership} - ) != len(membership): - _fail("ARCHIVE_MERKLE_RECOVERY_MEMBERSHIP_IDENTITY_REUSE") - if len({r["archiveOperator"] for r in membership}) < int( - POLICY["min_archive_operators"] - ) or len({r["verifierOperator"] for r in membership}) < int( - POLICY["min_verifier_operators"] - ): - _fail("ARCHIVE_MERKLE_RECOVERY_MEMBERSHIP_OPERATOR_QUORUM_INVALID") - if {r["archiveIdentity"] for r in membership} & { - r["verifierIdentity"] for r in membership - } or {r["archiveOperator"] for r in membership} & { - r["verifierOperator"] for r in membership - }: - _fail("ARCHIVE_MERKLE_RECOVERY_AUTHORITY_PLANE_OVERLAP") - before = _authority_fingerprint(run167_dir, verify_kwargs) - kwargs = dict(verify_kwargs) - kwargs["output_dir"] = run167_dir - kwargs["now"] = current - kwargs["historical"] = False - try: - run167.verify_archive_merkle_rebridge(**kwargs) - except Exception as exc: - raise ArchiveMerkleRecoveryError( - "ARCHIVE_MERKLE_RECOVERY_RUN167_VERIFY_FAILED:" + str(exc) - ) from exc - docs, raws = _run167_docs(run167_dir) - summary = _verify_snapshot_documents(docs, raws) - checkpoint = _make_checkpoint(docs, raws, summary, membership) - craw = _canonical(checkpoint) - artifact = _artifact("release-archive-merkle-recovery-checkpoint.json", craw) - successes = [] - locators = set() - archive_ops = set() - verifier_ops = set() - for ai, ao, adapter, vi, vo, verifier in sorted( - normalized, key=lambda x: (x[1], x[0], x[4], x[3]) - ): - archive_id = _archive_id(artifact["sha256"], ai) - request = { - "schemaVersion": int(POLICY["archive_protocol_version"]), - "operation": "preserve", - "archiveId": archive_id, - "archiveIdentity": ai, - "archiveOperator": ao, - "artifact": artifact, - "payloadBase64": base64.b64encode(craw).decode("ascii"), - "overwrite": False, - "recoveryCheckpointHeadSha256": checkpoint["recoveryCheckpointHeadSha256"], - "authorityHeadSha256": summary["authorityHead"], - "merkleHeadSha256": summary["merkleHead"], - } - parsed_a = _validate_archive_response( - adapter(dict(request)), - operation="preserve", - archive_id=archive_id, - identity=ai, - operator=ao, - artifact=artifact, - now=current, - ) - vreq = { - "schemaVersion": int(POLICY["verifier_protocol_version"]), - "operation": "verify", - "archiveId": archive_id, - "archiveIdentity": ai, - "archiveOperator": ao, - "verifierIdentity": vi, - "verifierOperator": vo, - "artifact": artifact, - "locator": parsed_a["locator"], - "immutableVersionId": parsed_a["immutableVersionId"], - "expectedSha256": artifact["sha256"], - "expectedSize": artifact["size"], - } - parsed_v = _validate_verifier_response( - verifier(dict(vreq)), - archive=parsed_a, - identity=vi, - operator=vo, - artifact=artifact, - now=current, - ) - if parsed_a["locator"] in locators: - _fail("ARCHIVE_MERKLE_RECOVERY_LOCATOR_COLLISION") - locators.add(parsed_a["locator"]) - archive_ops.add(ao) - verifier_ops.add(vo) - successes.append({"archiveResponse": parsed_a, "verifierResponse": parsed_v}) - if ( - len(successes) < int(POLICY["min_successful_archives"]) - or len(archive_ops) < int(POLICY["min_archive_operators"]) - or len(verifier_ops) < int(POLICY["min_verifier_operators"]) - ): - _fail("ARCHIVE_MERKLE_RECOVERY_PRESERVATION_QUORUM_NOT_MET") - if before != _authority_fingerprint(run167_dir, verify_kwargs): - _fail("ARCHIVE_MERKLE_RECOVERY_INPUT_DRIFT") - state = { - "schemaVersion": int(POLICY["state_schema_version"]), - "status": "trusted-archive-merkle-recovery", - "sequence": summary["sequence"], - "rebridgeSequence": summary["rebridgeSequence"], - "archiveMerkleRebridgeAuthorityHeadSha256": summary["authorityHead"], - "merkleRebridgeContinuityHeadSha256": summary["merkleHead"], - "activeAuthoritySha256": summary["activeAuthoritySha256"], - "recoveryCheckpointHeadSha256": checkpoint["recoveryCheckpointHeadSha256"], - "checkpointArtifact": artifact, - "archiveMembership": membership, - } - receipt = { - "schemaVersion": int(POLICY["receipt_schema_version"]), - "status": "archive-merkle-recovery-preserved", - "checkpointArtifact": artifact, - "archives": successes, - } - parent = target.parent - parent.mkdir(parents=True, exist_ok=True) - stage = Path(tempfile.mkdtemp(prefix=".run168-preserve-", dir=parent)) - try: - _write(stage / "release-archive-merkle-recovery-checkpoint.json", checkpoint) - _write(stage / _DOC_ARCHIVE_MERKLE_RECOVERY_STATE, state) - _write(stage / "release-archive-merkle-recovery-receipt.json", receipt) - verify_recovery_archive(output_dir=stage, now=current) - os.replace(stage, target) - except Exception: - shutil.rmtree(stage, ignore_errors=True) - raise - return { - "ok": True, - "phase": "archive-merkle-rebridge-preserved", - "sequence": summary["sequence"], - "checkpoint_sha256": artifact["sha256"], - "checkpoint_head_sha256": checkpoint["recoveryCheckpointHeadSha256"], - "authority_head_sha256": summary["authorityHead"], - "merkle_head_sha256": summary["merkleHead"], - "archives": len(successes), - } - - -def _validate_recovery_response( - value: Any, *, identity: str, operator: str, now: datetime -) -> dict[str, Any]: - common = { - "schemaVersion", - "status", - "sourceIdentity", - "sourceOperator", - "observedAt", - "readOnly", - "writerCredentialsReused", - } - if ( - not isinstance(value, dict) - or value.get("schemaVersion") != int(POLICY["recovery_protocol_version"]) - or value.get("sourceIdentity") != identity - or value.get("sourceOperator") != operator - ): - _fail("ARCHIVE_MERKLE_RECOVERY_SOURCE_SCHEMA_INVALID") - if ( - value.get("readOnly") is not True - or value.get("writerCredentialsReused") is not False - ): - _fail("ARCHIVE_MERKLE_RECOVERY_SOURCE_AUTHORITY_INVALID") - observed_at = _fresh( - value.get("observedAt"), "ARCHIVE_MERKLE_RECOVERY_SOURCE_OBSERVED_AT", now=now - ) - if value.get("status") == "unavailable": - if set(value) != common | {"reason"} or not bool( - POLICY["allow_unavailable_recovery_source"] - ): - _fail("ARCHIVE_MERKLE_RECOVERY_SOURCE_UNAVAILABLE_INVALID") - reason = value.get("reason") - _len = len(reason) > 255 # ruff: ignore[magic-value-comparison] - if not isinstance(reason, str) or not reason or _len: - _fail("ARCHIVE_MERKLE_RECOVERY_SOURCE_REASON_INVALID") - return {"status": "unavailable", "observedAt": observed_at, "reason": reason} - if value.get("status") != "observed" or set(value) != common | { - "locator", - "artifact", - "payloadBase64", - }: - _fail("ARCHIVE_MERKLE_RECOVERY_SOURCE_STATUS_INVALID") - artifact = _artifact_doc( - value.get("artifact"), "ARCHIVE_MERKLE_RECOVERY_SOURCE_ARTIFACT" - ) - if artifact["name"] != "release-archive-merkle-recovery-checkpoint.json": - _fail("ARCHIVE_MERKLE_RECOVERY_SOURCE_ARTIFACT_NAME_INVALID") - try: - raw = base64.b64decode(value.get("payloadBase64"), validate=True) - except Exception as exc: - raise ArchiveMerkleRecoveryError( - "ARCHIVE_MERKLE_RECOVERY_SOURCE_PAYLOAD_INVALID" - ) from exc - if len(raw) != artifact["size"] or _sha_bytes(raw) != artifact["sha256"]: - _fail("ARCHIVE_MERKLE_RECOVERY_SOURCE_PAYLOAD_HASH_INVALID") - doc = _loads(raw, "ARCHIVE_MERKLE_RECOVERY_SOURCE_PAYLOAD") - return { - "status": "observed", - "observedAt": observed_at, - "locator": _safe_locator( - value.get("locator"), "ARCHIVE_MERKLE_RECOVERY_SOURCE_LOCATOR_INVALID" - ), - "artifact": artifact, - "raw": raw, - "doc": doc, - } - - -def recover_rebridge_history( # ruff: ignore[undocumented-public-function] - *, - sources: list[tuple[str, str, RecoveryAdapter]], - output_dir: Path, - expected_checkpoint_sha256: str, - expected_authority_head_sha256: str, - expected_merkle_head_sha256: str, - expected_active_authority_sha256: str, - expected_sequence: int, - now: datetime | None = None, -) -> dict[str, Any]: - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - expected_checkpoint_sha256 = _hex( - expected_checkpoint_sha256, - "ARCHIVE_MERKLE_RECOVERY_EXPECTED_CHECKPOINT_INVALID", - ) - expected_authority_head_sha256 = _hex( - expected_authority_head_sha256, - "ARCHIVE_MERKLE_RECOVERY_EXPECTED_AUTHORITY_HEAD_INVALID", - ) - expected_merkle_head_sha256 = _hex( - expected_merkle_head_sha256, - "ARCHIVE_MERKLE_RECOVERY_EXPECTED_MERKLE_HEAD_INVALID", - ) - expected_active_authority_sha256 = _hex( - expected_active_authority_sha256, - "ARCHIVE_MERKLE_RECOVERY_EXPECTED_ACTIVE_AUTHORITY_INVALID", - ) - expected_sequence = _positive_int( - expected_sequence, "ARCHIVE_MERKLE_RECOVERY_EXPECTED_SEQUENCE_INVALID" - ) - normalized = [ - ( - _identity(i, "ARCHIVE_MERKLE_RECOVERY_SOURCE_IDENTITY_INVALID"), - _identity(o, "ARCHIVE_MERKLE_RECOVERY_SOURCE_OPERATOR_INVALID"), - a, - ) - for i, o, a in sources - ] - if ( - len(normalized) < int(POLICY["min_configured_recovery_sources"]) - or len({i for i, _, _ in normalized}) != len(normalized) - or len({o for _, o, _ in normalized}) < int(POLICY["min_recovery_operators"]) - ): - _fail("ARCHIVE_MERKLE_RECOVERY_SOURCE_PLAN_INVALID") - request = { - "schemaVersion": int(POLICY["recovery_protocol_version"]), - "operation": "recover", - "expectedCheckpointSha256": expected_checkpoint_sha256, - "expectedAuthorityHeadSha256": expected_authority_head_sha256, - "expectedMerkleHeadSha256": expected_merkle_head_sha256, - "expectedActiveAuthoritySha256": expected_active_authority_sha256, - "expectedSequence": expected_sequence, - } - observed = [] - unavailable = [] - for identity, operator, adapter in sorted(normalized, key=lambda x: (x[1], x[0])): - result = _validate_recovery_response( - adapter(dict(request, source={"identity": identity, "operator": operator})), - identity=identity, - operator=operator, - now=current, - ) - (observed if result["status"] == "observed" else unavailable).append( - dict(result, identity=identity, operator=operator) - ) - if len(observed) < int(POLICY["min_observed_recovery_sources"]) or len( - {x["operator"] for x in observed} - ) < int(POLICY["min_recovery_operators"]): - _fail("ARCHIVE_MERKLE_RECOVERY_SOURCE_QUORUM_NOT_MET") - locators = [x["locator"] for x in observed] - if len(set(locators)) != len(locators): - _fail("ARCHIVE_MERKLE_RECOVERY_SOURCE_LOCATOR_COLLISION") - hashes = {_sha_bytes(x["raw"]) for x in observed} - if len(hashes) != 1: - _fail("ARCHIVE_MERKLE_RECOVERY_EQUIVOCATION") - raw = observed[0]["raw"] - if _sha_bytes(raw) != expected_checkpoint_sha256: - _fail("ARCHIVE_MERKLE_RECOVERY_ROLLBACK_CHECKPOINT_PIN_MISMATCH") - info = _verify_checkpoint(observed[0]["doc"]) - summary = info["summary"] - if ( - summary["authorityHead"] != expected_authority_head_sha256 - or summary["merkleHead"] != expected_merkle_head_sha256 - or summary["activeAuthoritySha256"] != expected_active_authority_sha256 - or summary["sequence"] != expected_sequence - ): - _fail("ARCHIVE_MERKLE_RECOVERY_ROLLBACK_HEAD_PIN_MISMATCH") - target = _outside(output_dir, [], "ARCHIVE_MERKLE_RECOVERY_RECOVERY_OUTPUT_INVALID") - if target.exists() or target.is_symlink(): - _fail("ARCHIVE_MERKLE_RECOVERY_RECOVERY_OUTPUT_ALREADY_EXISTS") - parent = target.parent - parent.mkdir(parents=True, exist_ok=True) - stage = Path(tempfile.mkdtemp(prefix=".run168-recover-", dir=parent)) - try: - (stage / "recovered-archive-merkle-recovery-checkpoint.json").write_bytes(raw) - recovered = stage / "recovered-run167" - recovered.mkdir() - for name in sorted(_RUN167_NAMES): - _write(recovered / name, info["documents"][name]) - # Re-check the exact reconstructed byte set using the compact offline snapshot verifier. - rdocs, rraws = _run167_docs(recovered) - rsummary = _verify_snapshot_documents(rdocs, rraws) - if rsummary != summary: - _fail("ARCHIVE_MERKLE_RECOVERY_REHYDRATED_MISMATCH") - active = { - "schemaVersion": int(POLICY["active_recovery_schema_version"]), - "status": "recovered-active-archive-merkle-authority", - "sequence": summary["sequence"], - "rebridgeSequence": summary["rebridgeSequence"], - "authority": summary["activeAuthority"], - "activeAuthoritySha256": summary["activeAuthoritySha256"], - "revokedKeyFingerprints": summary["revokedKeyFingerprints"], - "lastCheckpoints": summary["lastCheckpoints"], - "archiveMerkleRebridgeAuthorityHeadSha256": summary["authorityHead"], - "merkleRebridgeContinuityHeadSha256": summary["merkleHead"], - } - _write(stage / "recovered-active-archive-merkle-authority.json", active) - receipt = { - "schemaVersion": int(POLICY["recovery_receipt_schema_version"]), - "status": "archive-merkle-rebridge-recovered", - "checkpointArtifact": _artifact( - "release-archive-merkle-recovery-checkpoint.json", raw - ), - "sequence": summary["sequence"], - "archiveMerkleRebridgeAuthorityHeadSha256": summary["authorityHead"], - "merkleRebridgeContinuityHeadSha256": summary["merkleHead"], - "activeAuthoritySha256": summary["activeAuthoritySha256"], - "observed": [ - { - "identity": x["identity"], - "operator": x["operator"], - "locator": x["locator"], - "observedAt": x["observedAt"], - } - for x in observed - ], - "unavailable": [ - { - "identity": x["identity"], - "operator": x["operator"], - "reason": x["reason"], - "observedAt": x["observedAt"], - } - for x in unavailable - ], - } - _write(stage / "release-archive-merkle-recovery-recovery-receipt.json", receipt) - os.replace(stage, target) - except Exception: - shutil.rmtree(stage, ignore_errors=True) - raise - return { - "ok": True, - "phase": "archive-merkle-rebridge-recovered", - "sequence": summary["sequence"], - "checkpoint_sha256": expected_checkpoint_sha256, - "authority_head_sha256": summary["authorityHead"], - "merkle_head_sha256": summary["merkleHead"], - "active_authority_sha256": summary["activeAuthoritySha256"], - "observed": len(observed), - "unavailable": len(unavailable), - } - - -def _command_adapter( - command: list[str], *, prefix: str -) -> Callable[[dict[str, Any]], dict[str, Any]]: - if not command: - _fail(prefix + "_COMMAND_EMPTY") - limit = int(POLICY["max_adapter_output_bytes"]) - timeout = int(POLICY["adapter_timeout_seconds"]) - - def call(request: dict[str, Any]) -> dict[str, Any]: - try: - proc = ( - # lint - subprocess.Popen( # ruff: ignore[subprocess-without-shell-equals-true] - command, - stdin=subprocess.PIPE, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, - env={"PATH": os.defpath, "LC_ALL": "C", "LANG": "C"}, - ) - ) - except OSError as exc: - raise ArchiveMerkleRecoveryError(prefix + "_START_FAILED") from exc - out = bytearray() - err = bytearray() - overflow = threading.Event() - total = [0] - lock = threading.Lock() - - def drain(stream, buf): - while True: - chunk = stream.read(65536) - if not chunk: - break - with lock: - total[0] += len(chunk) - if total[0] > limit: - overflow.set() - proc.kill() - break - buf.extend(chunk) - - ts = [ - threading.Thread(target=drain, args=(proc.stdout, out), daemon=True), - threading.Thread(target=drain, args=(proc.stderr, err), daemon=True), - ] - for t in ts: - t.start() - try: - assert proc.stdin is not None # ruff: ignore[assert] - proc.stdin.write(_canonical(request)) - proc.stdin.close() - proc.wait(timeout=timeout) - except subprocess.TimeoutExpired: - proc.kill() - proc.wait() - _fail(prefix + "_TIMEOUT") - finally: - for t in ts: - t.join(timeout=1) - if overflow.is_set(): - _fail(prefix + "_OUTPUT_TOO_LARGE") - if proc.returncode != 0: - _fail(prefix + "_FAILED") - return _loads(bytes(out), prefix + "_OUTPUT") - - return call - - -def command_archive(command: list[str]): # ruff: ignore[undocumented-public-function] - return _command_adapter(command, prefix="ARCHIVE_MERKLE_RECOVERY_ARCHIVE_ADAPTER") - - -def command_verifier(command: list[str]): # ruff: ignore[undocumented-public-function] - return _command_adapter(command, prefix="ARCHIVE_MERKLE_RECOVERY_VERIFIER_ADAPTER") - - -def command_recovery(command: list[str]): # ruff: ignore[undocumented-public-function] - return _command_adapter(command, prefix="ARCHIVE_MERKLE_RECOVERY_SOURCE_ADAPTER") - - -def main( # ruff: ignore[undocumented-public-function] - argv: list[str] | None = None, -) -> int: - parser = argparse.ArgumentParser(description=__doc__) - sub = parser.add_subparsers(dest="command", required=True) - verify = sub.add_parser("verify-checkpoint") - verify.add_argument("--output-dir", type=Path, required=True) - verify.add_argument("--historical", action="store_true") - recover = sub.add_parser("recover") - recover.add_argument( - "--source", action="append", required=True, help="identity,operator=command..." - ) - recover.add_argument("--output-dir", type=Path, required=True) - recover.add_argument("--expected-checkpoint-sha256", required=True) - recover.add_argument("--expected-authority-head-sha256", required=True) - recover.add_argument("--expected-merkle-head-sha256", required=True) - recover.add_argument("--expected-active-authority-sha256", required=True) - recover.add_argument("--expected-sequence", type=int, required=True) - args = parser.parse_args(argv) - if args.command == "verify-checkpoint": - result = verify_recovery_archive( - output_dir=args.output_dir, historical=args.historical - ) - else: - sources = [] - for value in args.source: - if "=" not in value: - _fail("ARCHIVE_MERKLE_RECOVERY_SOURCE_ARG_INVALID") - lhs, cmd = value.split("=", 1) - parts = lhs.split(",") - if len(parts) != 2: # ruff: ignore[magic-value-comparison] - _fail("ARCHIVE_MERKLE_RECOVERY_SOURCE_ARG_INVALID") - sources.append((parts[0], parts[1], command_recovery(cmd.strip().split()))) - result = recover_rebridge_history( - sources=sources, - output_dir=args.output_dir, - expected_checkpoint_sha256=args.expected_checkpoint_sha256, - expected_authority_head_sha256=args.expected_authority_head_sha256, - expected_merkle_head_sha256=args.expected_merkle_head_sha256, - expected_active_authority_sha256=args.expected_active_authority_sha256, - expected_sequence=args.expected_sequence, - ) - logger.info(json.dumps(result, sort_keys=True, separators=(",", ":"))) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/security/preserve_release_history.py b/security/preserve_release_history.py deleted file mode 100644 index 00673c3a6982659ae7fcac251e9f8a0b5da8292e..0000000000000000000000000000000000000000 --- a/security/preserve_release_history.py +++ /dev/null @@ -1,2185 +0,0 @@ -""" -Preserve witnessed releases in a quorum-gossiped, independently archived history chain. -""" - -from __future__ import annotations - -import argparse -import hashlib -import json -import logging -import os -import re -import shutil -import subprocess -import tempfile -import threading -from datetime import datetime, timedelta, timezone -from pathlib import Path -from typing import Any, Callable, Iterable -from urllib.parse import urlsplit - -import tomllib - -HERE = Path(__file__).resolve().parent -POLICY = tomllib.loads((HERE / "release_history_policy.toml").read_text()) -WITNESS_POLICY = tomllib.loads((HERE / "release_witness_policy.toml").read_text()) -RUN151_POLICY = tomllib.loads((HERE / "release_transparency_policy.toml").read_text()) -HISTORY_PREDICATE_TYPE = str(POLICY["history_predicate_type"]) -_HEX64 = re.compile(r"^[0-9a-f]{64}$") -_ID = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:/@+-]{0,511}$") -_REVISION = re.compile(r"^[0-9a-f]{40}(?:[0-9a-f]{24})?$") -_CHUNK = 1024 * 1024 -logger = logging.getLogger(__name__) - - -class HistoryError(RuntimeError): - """Release-history preservation violated a fail-closed invariant.""" - - -def _fail(code: str) -> None: - raise HistoryError(code) - - -def _sha256(path: Path) -> str: - h = hashlib.sha256() - with path.open("rb") as handle: - for chunk in iter(lambda: handle.read(_CHUNK), b""): - h.update(chunk) - return h.hexdigest() - - -def _sha_bytes(raw: bytes) -> str: - return hashlib.sha256(raw).hexdigest() - - -def _canonical_bytes(value: dict[str, Any]) -> bytes: - return ( - json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False) - + "\n" - ).encode("utf-8") - - -def _write_canonical(path: Path, value: dict[str, Any]) -> None: - path.write_bytes(_canonical_bytes(value)) - - -def _loads_json(raw: bytes, code: str) -> dict[str, Any]: - def pairs_hook(pairs): - out: dict[str, Any] = {} - for key, value in pairs: - if key in out: - _fail(code + "_DUPLICATE_KEY") - out[key] = value - return out - - try: - value = json.loads(raw.decode("utf-8"), object_pairs_hook=pairs_hook) - except HistoryError: - raise - except Exception as exc: - raise HistoryError(code + "_JSON_INVALID") from exc - if not isinstance(value, dict): - _fail(code + "_SCHEMA_INVALID") - return value - - -def _read_json( - path: Path, code: str, *, canonical: bool = True -) -> tuple[dict[str, Any], bytes]: - if path.is_symlink() or not path.is_file(): - _fail(code + "_INVALID") - if path.stat().st_size > int(POLICY["max_json_bytes"]): - _fail(code + "_TOO_LARGE") - raw = path.read_bytes() - doc = _loads_json(raw, code) - if canonical and raw != _canonical_bytes(doc): - _fail(code + "_NOT_CANONICAL") - return doc, raw - - -def _regular_dir(path: Path, code: str) -> Path: - if path.is_symlink() or not path.is_dir(): - _fail(code) - return path.resolve() - - -def _outside(path: Path, protected: Iterable[Path], code: str) -> Path: - target = path.expanduser().resolve() - for item in protected: - root = item.resolve() - if target == root or root in target.parents: - _fail(code) - return target - - -def _hex(value: Any, code: str) -> str: - if not isinstance(value, str) or _HEX64.fullmatch(value) is None: - _fail(code) - return value - - -def _identity(value: Any, code: str) -> str: - if not isinstance(value, str): - _fail(code) - value = value.strip() - if ( - not value - or len(value) > 512 # ruff: ignore[magic-value-comparison] - or ".." in value - or "?" in value - or "#" in value - or "\x00" in value - ): - _fail(code) - if ( - any( - ord(ch) < 32 # ruff: ignore[magic-value-comparison] - or ord(ch) == 127 # ruff: ignore[magic-value-comparison] - for ch in value - ) - or _ID.fullmatch(value) is None - ): - _fail(code) - return value - - -def _safe_text(value: Any, code: str, *, limit: int = 1024) -> str: - if not isinstance(value, str): - _fail(code) - value = value.strip() - if ( - not value - or len(value) > limit - or "?" in value - or "#" in value - or "\x00" in value - ): - _fail(code) - if any( - ord(ch) < 32 # ruff: ignore[magic-value-comparison] - or ord(ch) == 127 # ruff: ignore[magic-value-comparison] - for ch in value - ): - _fail(code) - return value - - -def _safe_locator(value: Any, code: str) -> str: - value = _safe_text(value, code, limit=2048) - try: - parsed = urlsplit(value) - except ValueError: - _fail(code) - if ( - not parsed.scheme - or parsed.scheme.lower() == "file" - or parsed.query - or parsed.fragment - or parsed.username is not None - or parsed.password is not None - ): - _fail(code) - if ".." in parsed.path.split("/"): - _fail(code) - return value - - -def _safe_name(value: Any, code: str) -> str: - value = _safe_text(value, code, limit=255) - if ( - value in {".", ".."} - or Path(value).name != value - or "/" in value - or "\\" in value - ): - _fail(code) - return value - - -def _size(value: Any, code: str) -> int: - if not isinstance(value, int) or isinstance(value, bool) or value < 0: - _fail(code) - return value - - -def _fresh(value: Any, code: str, *, now: datetime) -> str: - if not isinstance(value, str) or not value.endswith("Z"): - _fail(code + "_INVALID") - try: - parsed = datetime.fromisoformat(value[:-1] + "+00:00").astimezone(timezone.utc) - except ValueError: - _fail(code + "_INVALID") - skew = timedelta(minutes=int(POLICY["max_clock_skew_minutes"])) - age = timedelta(minutes=int(POLICY["max_result_age_minutes"])) - if parsed > now + skew: - _fail(code + "_FROM_FUTURE") - if now - parsed > age: - _fail(code + "_STALE") - return value - - -def _artifact(value: Any, code: str) -> dict[str, Any]: - if not isinstance(value, dict) or set(value) != {"name", "sha256", "size"}: - _fail(code + "_SCHEMA_INVALID") - return { - "name": _safe_name(value.get("name"), code + "_NAME_INVALID"), - "sha256": _hex(value.get("sha256"), code + "_SHA256_INVALID"), - "size": _size(value.get("size"), code + "_SIZE_INVALID"), - } - - -def _checkpoint(value: Any, code: str) -> dict[str, Any]: - if not isinstance(value, dict) or set(value) != { - "treeSize", - "rootHash", - "signedCheckpointSha256", - }: - _fail(code + "_SCHEMA_INVALID") - tree_size = _size(value.get("treeSize"), code + "_TREE_SIZE_INVALID") - if tree_size < 1: - _fail(code + "_TREE_SIZE_INVALID") - return { - "treeSize": tree_size, - "rootHash": _hex(value.get("rootHash"), code + "_ROOT_HASH_INVALID"), - "signedCheckpointSha256": _hex( - value.get("signedCheckpointSha256"), code + "_SIGNED_HASH_INVALID" - ), - } - - -def _checkpoint_doc(log_id: str, checkpoint: dict[str, Any]) -> dict[str, Any]: - return { - "schemaVersion": int(WITNESS_POLICY["previous_checkpoint_schema_version"]), - "logId": log_id, - "checkpoint": checkpoint, - } - - -def _history_head( - history_id: str, genesis: dict[str, Any], entries: list[dict[str, Any]] -) -> str: - head = hashlib.sha256( - _canonical_bytes({"historyId": history_id, "genesis": genesis}) - ).digest() - for entry in entries: - head = hashlib.sha256(head + _canonical_bytes(entry)).digest() - return head.hex() - - -def _validate_witnessed( # ruff: ignore[too-many-branches] - root: Path, -) -> dict[str, Any]: - root = _regular_dir(root, "RUN152_DIRECTORY_INVALID") - allowed = { - "release-publication-record.json", - "release-publication-binding-receipt.json", - "previous-transparency-checkpoint.json", - "accepted-transparency-checkpoint.json", - "release-transparency-witness-record.json", - "release-transparency-witness-receipt.json", - "log-results", - "witness-results", - "anchor-results", - } - if {p.name for p in root.iterdir()} != allowed: - _fail("RUN152_DIRECTORY_ALLOWLIST_MISMATCH") - for dirname in ("log-results", "witness-results", "anchor-results"): - directory = root / dirname - if directory.is_symlink() or not directory.is_dir(): - _fail("RUN152_EVIDENCE_DIRECTORY_INVALID") - for item in directory.iterdir(): - if item.is_symlink() or not item.is_file(): - _fail("RUN152_EVIDENCE_ENTRY_INVALID") - _read_json(item, "RUN152_EVIDENCE_ENTRY") - - final_record, final_raw = _read_json( - root / "release-publication-record.json", "RUN152_FINAL_RECORD" - ) - final_binding, final_binding_raw = _read_json( - root / "release-publication-binding-receipt.json", "RUN152_FINAL_BINDING" - ) - previous, previous_raw = _read_json( - root / "previous-transparency-checkpoint.json", "RUN152_PREVIOUS_CHECKPOINT" - ) - accepted, accepted_raw = _read_json( - root / "accepted-transparency-checkpoint.json", "RUN152_ACCEPTED_CHECKPOINT" - ) - witness_record, witness_raw = _read_json( - root / "release-transparency-witness-record.json", "RUN152_WITNESS_RECORD" - ) - receipt, receipt_raw = _read_json( - root / "release-transparency-witness-receipt.json", "RUN152_WITNESS_RECEIPT" - ) - - if ( - set(final_record) - != { - "schemaVersion", - "predicateType", - "status", - "publicationId", - "release", - "subject", - "target", - "signer", - "independentVerifier", - "artifacts", - "verification", - } - or final_record.get("schemaVersion") - != int(RUN151_POLICY["final_record_schema_version"]) - or final_record.get("predicateType") - != str(RUN151_POLICY["final_record_predicate_type"]) - or final_record.get("status") != "finalized" - ): - _fail("RUN152_FINAL_RECORD_SCHEMA_INVALID") - publication_id = _safe_text( - final_record.get("publicationId"), "RUN152_PUBLICATION_ID_INVALID", limit=255 - ) - release = final_record.get("release") - if not isinstance(release, dict) or set(release) != {"releaseId", "sourceRevision"}: - _fail("RUN152_RELEASE_SCHEMA_INVALID") - release_id = _safe_text( - release.get("releaseId"), "RUN152_RELEASE_ID_INVALID", limit=127 - ) - source_revision = _safe_text( - release.get("sourceRevision"), "RUN152_SOURCE_REVISION_INVALID", limit=128 - ) - if _REVISION.fullmatch(source_revision) is None: - _fail("RUN152_SOURCE_REVISION_INVALID") - final_subject = final_record.get("subject") - final_subject_keys = { - "promotionReceiptSha256", - "publicationReceiptSha256", - "publicationTransparencySha256", - "postPublicationAttestationSha256", - "signatureVerificationSha256", - "signatureVerifierEvidenceSha256", - } - if not isinstance(final_subject, dict) or set(final_subject) != final_subject_keys: - _fail("RUN152_FINAL_SUBJECT_SCHEMA_INVALID") - for value in final_subject.values(): - _hex(value, "RUN152_FINAL_SUBJECT_HASH_INVALID") - final_target = final_record.get("target") - if not isinstance(final_target, dict) or set(final_target) != { - "publisher", - "targetId", - }: - _fail("RUN152_FINAL_TARGET_SCHEMA_INVALID") - - for doc, label in ((previous, "PREVIOUS"), (accepted, "ACCEPTED")): - if set(doc) != {"schemaVersion", "logId", "checkpoint"} or doc.get( - "schemaVersion" - ) != int(WITNESS_POLICY["previous_checkpoint_schema_version"]): - _fail("RUN152_" + label + "_CHECKPOINT_SCHEMA_INVALID") - log_id = _safe_text(accepted.get("logId"), "RUN152_LOG_ID_INVALID", limit=255) - if previous.get("logId") != log_id: - _fail("RUN152_CHECKPOINT_LOG_MISMATCH") - previous_checkpoint = _checkpoint( - previous.get("checkpoint"), "RUN152_PREVIOUS_CHECKPOINT" - ) - accepted_checkpoint = _checkpoint( - accepted.get("checkpoint"), "RUN152_ACCEPTED_CHECKPOINT" - ) - if accepted_checkpoint["treeSize"] <= previous_checkpoint["treeSize"]: - _fail("RUN152_CHECKPOINT_ORDER_INVALID") - - expected_wr_keys = { - "schemaVersion", - "predicateType", - "status", - "release", - "subject", - "target", - "transparencyLog", - "verification", - "witnessQuorum", - } - if ( - set(witness_record) != expected_wr_keys - or witness_record.get("schemaVersion") - != int(WITNESS_POLICY["witness_record_schema_version"]) - or witness_record.get("predicateType") - != str(WITNESS_POLICY["witness_record_predicate_type"]) - or witness_record.get("status") != "witnessed" - ): - _fail("RUN152_WITNESS_RECORD_SCHEMA_INVALID") - target = witness_record.get("target") - if not isinstance(target, dict) or set(target) != {"publisher", "targetId"}: - _fail("RUN152_WITNESS_TARGET_SCHEMA_INVALID") - publisher = _safe_text( - target.get("publisher"), "RUN152_WITNESS_PUBLISHER_INVALID", limit=255 - ) - target_id = _safe_text( - target.get("targetId"), "RUN152_WITNESS_TARGET_ID_INVALID", limit=255 - ) - if final_target != {"publisher": publisher, "targetId": target_id}: - _fail("RUN152_WITNESS_TARGET_MISMATCH") - wr_release = witness_record.get("release") - if wr_release != { - "releaseId": release_id, - "publicationId": publication_id, - "sourceRevision": source_revision, - }: - _fail("RUN152_WITNESS_RELEASE_MISMATCH") - wr_subject = witness_record.get("subject") - if not isinstance(wr_subject, dict) or set(wr_subject) != { - "finalPublicationRecordSha256", - "finalBindingReceiptSha256", - "postPublicationAttestationSha256", - "signatureVerificationSha256", - }: - _fail("RUN152_WITNESS_SUBJECT_SCHEMA_INVALID") - if wr_subject.get("finalPublicationRecordSha256") != _sha_bytes( - final_raw - ) or wr_subject.get("finalBindingReceiptSha256") != _sha_bytes(final_binding_raw): - _fail("RUN152_WITNESS_SUBJECT_HASH_MISMATCH") - if ( - wr_subject.get("postPublicationAttestationSha256") - != final_subject["postPublicationAttestationSha256"] - or wr_subject.get("signatureVerificationSha256") - != final_subject["signatureVerificationSha256"] - ): - _fail("RUN152_WITNESS_FINAL_SUBJECT_REBIND_FAILED") - wr_verification = witness_record.get("verification") - if not isinstance(wr_verification, dict) or set(wr_verification) != { - "primaryVerifierIdentity", - "checkpointSignatureVerified", - "inclusionVerified", - "consistencyVerified", - "integratedEntryVerified", - }: - _fail("RUN152_WITNESS_VERIFICATION_SCHEMA_INVALID") - primary_identity = _identity( - wr_verification.get("primaryVerifierIdentity"), - "RUN152_PRIMARY_VERIFIER_IDENTITY_INVALID", - ) - if { - k: wr_verification[k] - for k in ( - "checkpointSignatureVerified", - "inclusionVerified", - "consistencyVerified", - "integratedEntryVerified", - ) - } != { - "checkpointSignatureVerified": True, - "inclusionVerified": True, - "consistencyVerified": True, - "integratedEntryVerified": True, - }: - _fail("RUN152_WITNESS_VERIFICATION_FLAGS_INVALID") - tlog = witness_record.get("transparencyLog") - if not isinstance(tlog, dict) or set(tlog) != { - "logId", - "entryId", - "entryIndex", - "entryLocator", - "checkpoint", - "previousCheckpointSha256", - }: - _fail("RUN152_WITNESS_LOG_SCHEMA_INVALID") - if ( - tlog.get("logId") != log_id - or _checkpoint(tlog.get("checkpoint"), "RUN152_WITNESS_CHECKPOINT") - != accepted_checkpoint - ): - _fail("RUN152_WITNESS_CHECKPOINT_MISMATCH") - if tlog.get("previousCheckpointSha256") != _sha_bytes(previous_raw): - _fail("RUN152_WITNESS_PREVIOUS_CHECKPOINT_HASH_MISMATCH") - _safe_text(tlog.get("entryId"), "RUN152_ENTRY_ID_INVALID", limit=255) - _size(tlog.get("entryIndex"), "RUN152_ENTRY_INDEX_INVALID") - _safe_text(tlog.get("entryLocator"), "RUN152_ENTRY_LOCATOR_INVALID") - quorum = witness_record.get("witnessQuorum") - if not isinstance(quorum, dict) or set(quorum) != { - "threshold", - "minimumOperators", - "witnesses", - }: - _fail("RUN152_WITNESS_QUORUM_SCHEMA_INVALID") - threshold = _size(quorum.get("threshold"), "RUN152_WITNESS_THRESHOLD_INVALID") - minimum_operators = _size( - quorum.get("minimumOperators"), "RUN152_WITNESS_OPERATOR_THRESHOLD_INVALID" - ) - witness_meta = quorum.get("witnesses") - if not isinstance(witness_meta, list) or len(witness_meta) < threshold: - _fail("RUN152_WITNESS_QUORUM_INVALID") - normalized_meta = [] - for item in witness_meta: - if not isinstance(item, dict) or set(item) != {"identity", "operator"}: - _fail("RUN152_WITNESS_META_SCHEMA_INVALID") - normalized_meta.append( - { - "identity": _identity( - item.get("identity"), "RUN152_WITNESS_IDENTITY_INVALID" - ), - "operator": _identity( - item.get("operator"), "RUN152_WITNESS_OPERATOR_INVALID" - ), - } - ) - if len({x["operator"] for x in normalized_meta}) < minimum_operators: - _fail("RUN152_WITNESS_OPERATOR_QUORUM_INVALID") - - final_binding_keys = { - "schemaVersion", - "status", - "publicationId", - "releaseId", - "publicationTransparencySha256", - "postPublicationAttestationSha256", - "signatureVerificationSha256", - "finalRecord", - "binding", - "finalVerificationEvidence", - } - if ( - not isinstance(final_binding, dict) - or set(final_binding) != final_binding_keys - or final_binding.get("schemaVersion") - != int(RUN151_POLICY["binding_receipt_schema_version"]) - or final_binding.get("status") != "bound" - or final_binding.get("publicationId") != publication_id - or final_binding.get("releaseId") != release_id - ): - _fail("RUN152_FINAL_BINDING_SCHEMA_INVALID") - if _artifact(final_binding.get("finalRecord"), "RUN152_FINAL_BINDING_RECORD") != { - "name": "release-publication-record.json", - "sha256": _sha_bytes(final_raw), - "size": len(final_raw), - }: - _fail("RUN152_FINAL_BINDING_RECORD_MISMATCH") - - receipt_keys = { - "schemaVersion", - "status", - "releaseId", - "publicationId", - "finalPublicationRecord", - "finalBindingReceiptSha256", - "witnessRecord", - "transparency", - "witnessEvidence", - "anchor", - } - if ( - set(receipt) != receipt_keys - or receipt.get("schemaVersion") - != int(WITNESS_POLICY["witness_receipt_schema_version"]) - or receipt.get("status") != "witnessed" - ): - _fail("RUN152_WITNESS_RECEIPT_SCHEMA_INVALID") - if ( - receipt.get("releaseId") != release_id - or receipt.get("publicationId") != publication_id - ): - _fail("RUN152_WITNESS_RECEIPT_ID_MISMATCH") - final_item = _artifact( - receipt.get("finalPublicationRecord"), "RUN152_FINAL_RECORD_ITEM" - ) - expected_final_item = { - "name": "release-publication-record.json", - "sha256": _sha_bytes(final_raw), - "size": len(final_raw), - } - if final_item != expected_final_item or receipt.get( - "finalBindingReceiptSha256" - ) != _sha_bytes(final_binding_raw): - _fail("RUN152_FINAL_RECORD_REBIND_FAILED") - witness_item = _artifact(receipt.get("witnessRecord"), "RUN152_WITNESS_RECORD_ITEM") - expected_witness_item = { - "name": "release-transparency-witness-record.json", - "sha256": _sha_bytes(witness_raw), - "size": len(witness_raw), - } - if witness_item != expected_witness_item: - _fail("RUN152_WITNESS_RECORD_REBIND_FAILED") - trans = receipt.get("transparency") - trans_keys = { - "transparencyId", - "logId", - "entryId", - "entryIndex", - "entryLocator", - "checkpoint", - "previousCheckpointSha256", - "acceptedCheckpointSha256", - "submitEvidenceSha256", - "primaryVerifierEvidenceSha256", - } - if not isinstance(trans, dict) or set(trans) != trans_keys: - _fail("RUN152_TRANSPARENCY_RECEIPT_SCHEMA_INVALID") - if ( - trans.get("logId") != log_id - or trans.get("entryId") != tlog.get("entryId") - or trans.get("entryIndex") != tlog.get("entryIndex") - or trans.get("entryLocator") != tlog.get("entryLocator") - or _checkpoint(trans.get("checkpoint"), "RUN152_RECEIPT_CHECKPOINT") - != accepted_checkpoint - ): - _fail("RUN152_TRANSPARENCY_RECEIPT_MISMATCH") - if trans.get("previousCheckpointSha256") != _sha_bytes(previous_raw) or trans.get( - "acceptedCheckpointSha256" - ) != _sha_bytes(accepted_raw): - _fail("RUN152_CHECKPOINT_RECEIPT_HASH_MISMATCH") - submit_path = root / "log-results" / "submit.json" - primary_path = root / "log-results" / "primary.verify.json" - if _sha256(submit_path) != _hex( - trans.get("submitEvidenceSha256"), "RUN152_SUBMIT_EVIDENCE_HASH_INVALID" - ): - _fail("RUN152_SUBMIT_EVIDENCE_HASH_MISMATCH") - if _sha256(primary_path) != _hex( - trans.get("primaryVerifierEvidenceSha256"), - "RUN152_PRIMARY_EVIDENCE_HASH_INVALID", - ): - _fail("RUN152_PRIMARY_EVIDENCE_HASH_MISMATCH") - submit_evidence, _ = _read_json(submit_path, "RUN152_SUBMIT_EVIDENCE") - submit_keys = { - "schemaVersion", - "operation", - "transparencyId", - "status", - "log", - "subject", - "checkpoint", - "proof", - "integratedAt", - } - expected_log = { - "logId": log_id, - "entryId": tlog["entryId"], - "entryIndex": tlog["entryIndex"], - "entryLocator": tlog["entryLocator"], - } - if ( - set(submit_evidence) != submit_keys - or submit_evidence.get("schemaVersion") - != int(WITNESS_POLICY["log_protocol_version"]) - or submit_evidence.get("operation") != "submit" - or submit_evidence.get("transparencyId") != trans.get("transparencyId") - or submit_evidence.get("status") not in {"created", "present"} - ): - _fail("RUN152_SUBMIT_EVIDENCE_SCHEMA_INVALID") - if ( - submit_evidence.get("log") != expected_log - or _artifact(submit_evidence.get("subject"), "RUN152_SUBMIT_EVIDENCE_SUBJECT") - != expected_final_item - or _checkpoint( - submit_evidence.get("checkpoint"), "RUN152_SUBMIT_EVIDENCE_CHECKPOINT" - ) - != accepted_checkpoint - ): - _fail("RUN152_SUBMIT_EVIDENCE_REBIND_FAILED") - if submit_evidence.get("proof") != { - "appendOnly": True, - "overwrite": False, - "integratedEntryVerified": True, - }: - _fail("RUN152_SUBMIT_EVIDENCE_GUARANTEES_INVALID") - - primary_evidence, _ = _read_json(primary_path, "RUN152_PRIMARY_EVIDENCE") - observer_keys = { - "schemaVersion", - "operation", - "verificationId", - "transparencyId", - "status", - "observer", - "log", - "subject", - "checkpoint", - "previousCheckpoint", - "proof", - "verifiedAt", - } - if ( - set(primary_evidence) != observer_keys - or primary_evidence.get("schemaVersion") - != int(WITNESS_POLICY["verifier_protocol_version"]) - or primary_evidence.get("operation") != "verify" - or primary_evidence.get("transparencyId") != trans.get("transparencyId") - or primary_evidence.get("status") != "included" - ): - _fail("RUN152_PRIMARY_EVIDENCE_SCHEMA_INVALID") - primary_observer = primary_evidence.get("observer") - if ( - not isinstance(primary_observer, dict) - or set(primary_observer) != {"identity", "readOnly", "logCredentialsReused"} - or primary_observer.get("identity") != primary_identity - or primary_observer.get("readOnly") is not True - or primary_observer.get("logCredentialsReused") is not False - ): - _fail("RUN152_PRIMARY_EVIDENCE_AUTHORITY_INVALID") - if ( - primary_evidence.get("log") != expected_log - or _artifact(primary_evidence.get("subject"), "RUN152_PRIMARY_EVIDENCE_SUBJECT") - != expected_final_item - or _checkpoint( - primary_evidence.get("checkpoint"), "RUN152_PRIMARY_EVIDENCE_CHECKPOINT" - ) - != accepted_checkpoint - or _checkpoint( - primary_evidence.get("previousCheckpoint"), - "RUN152_PRIMARY_EVIDENCE_PREVIOUS_CHECKPOINT", - ) - != previous_checkpoint - ): - _fail("RUN152_PRIMARY_EVIDENCE_REBIND_FAILED") - if primary_evidence.get("proof") != { - "checkpointSignatureVerified": True, - "inclusionVerified": True, - "consistencyVerified": True, - "integratedEntryVerified": True, - }: - _fail("RUN152_PRIMARY_EVIDENCE_PROOF_INVALID") - - witness_evidence = receipt.get("witnessEvidence") - if not isinstance(witness_evidence, list) or len(witness_evidence) != len( - normalized_meta - ): - _fail("RUN152_WITNESS_EVIDENCE_COUNT_MISMATCH") - evidence_files = sorted((root / "witness-results").iterdir()) - if len(evidence_files) != len(witness_evidence): - _fail("RUN152_WITNESS_RESULT_COUNT_MISMATCH") - evidence_by_hash = {_sha256(path): path for path in evidence_files} - evidence_meta = [] - for item in witness_evidence: - if not isinstance(item, dict) or set(item) != { - "identity", - "operator", - "sha256", - }: - _fail("RUN152_WITNESS_EVIDENCE_SCHEMA_INVALID") - identity = _identity( - item.get("identity"), "RUN152_WITNESS_EVIDENCE_IDENTITY_INVALID" - ) - operator = _identity( - item.get("operator"), "RUN152_WITNESS_EVIDENCE_OPERATOR_INVALID" - ) - sha = _hex(item.get("sha256"), "RUN152_WITNESS_EVIDENCE_HASH_INVALID") - if sha not in evidence_by_hash: - _fail("RUN152_WITNESS_EVIDENCE_HASH_MISMATCH") - evidence_doc, _ = _read_json(evidence_by_hash[sha], "RUN152_WITNESS_EVIDENCE") - if ( - set(evidence_doc) != observer_keys - or evidence_doc.get("schemaVersion") - != int(WITNESS_POLICY["witness_protocol_version"]) - or evidence_doc.get("operation") != "verify" - or evidence_doc.get("transparencyId") != trans.get("transparencyId") - or evidence_doc.get("status") != "included" - ): - _fail("RUN152_WITNESS_EVIDENCE_SCHEMA_INVALID") - observer = evidence_doc.get("observer") - witness_observer_keys = { - "identity", - "operator", - "readOnly", - "logCredentialsReused", - "primaryVerifierCredentialsReused", - } - if ( - not isinstance(observer, dict) - or set(observer) != witness_observer_keys - or observer.get("identity") != identity - or observer.get("operator") != operator - or observer.get("readOnly") is not True - or observer.get("logCredentialsReused") is not False - or observer.get("primaryVerifierCredentialsReused") is not False - ): - _fail("RUN152_WITNESS_EVIDENCE_AUTHORITY_INVALID") - if ( - evidence_doc.get("log") != expected_log - or _artifact(evidence_doc.get("subject"), "RUN152_WITNESS_EVIDENCE_SUBJECT") - != expected_final_item - or _checkpoint( - evidence_doc.get("checkpoint"), "RUN152_WITNESS_EVIDENCE_CHECKPOINT" - ) - != accepted_checkpoint - or _checkpoint( - evidence_doc.get("previousCheckpoint"), - "RUN152_WITNESS_EVIDENCE_PREVIOUS_CHECKPOINT", - ) - != previous_checkpoint - ): - _fail("RUN152_WITNESS_EVIDENCE_REBIND_FAILED") - if evidence_doc.get("proof") != { - "checkpointSignatureVerified": True, - "inclusionVerified": True, - "consistencyVerified": True, - "integratedEntryVerified": True, - }: - _fail("RUN152_WITNESS_EVIDENCE_PROOF_INVALID") - evidence_meta.append({"identity": identity, "operator": operator}) - if sorted(evidence_meta, key=lambda x: (x["operator"], x["identity"])) != sorted( - normalized_meta, key=lambda x: (x["operator"], x["identity"]) - ): - _fail("RUN152_WITNESS_EVIDENCE_MEMBERSHIP_MISMATCH") - - anchor = receipt.get("anchor") - anchor_keys = { - "anchorId", - "locator", - "immutability", - "bindingType", - "verifiedAt", - "bindEvidenceSha256", - "verifyEvidenceSha256", - } - if not isinstance(anchor, dict) or set(anchor) != anchor_keys: - _fail("RUN152_ANCHOR_RECEIPT_SCHEMA_INVALID") - _safe_text(anchor.get("anchorId"), "RUN152_ANCHOR_ID_INVALID", limit=255) - anchor_locator = _safe_locator( - anchor.get("locator"), "RUN152_ANCHOR_LOCATOR_INVALID" - ) - bind_path = ( - root / "anchor-results" / "release-transparency-witness-record.json.bind.json" - ) - verify_path = ( - root / "anchor-results" / "release-transparency-witness-record.json.verify.json" - ) - if _sha256(bind_path) != _hex( - anchor.get("bindEvidenceSha256"), "RUN152_ANCHOR_BIND_HASH_INVALID" - ) or _sha256(verify_path) != _hex( - anchor.get("verifyEvidenceSha256"), "RUN152_ANCHOR_VERIFY_HASH_INVALID" - ): - _fail("RUN152_ANCHOR_EVIDENCE_HASH_MISMATCH") - for operation, evidence_path in (("bind", bind_path), ("verify", verify_path)): - evidence, _ = _read_json(evidence_path, "RUN152_ANCHOR_EVIDENCE") - expected_anchor_keys = { - "schemaVersion", - "operation", - "anchorId", - "status", - "target", - "record", - "guarantees", - "verifiedAt", - } - if ( - set(evidence) != expected_anchor_keys - or evidence.get("schemaVersion") - != int(WITNESS_POLICY["anchor_protocol_version"]) - or evidence.get("operation") != operation - or evidence.get("anchorId") != anchor.get("anchorId") - ): - _fail("RUN152_ANCHOR_EVIDENCE_SCHEMA_INVALID") - if evidence.get("status") not in ( - {"created", "present"} if operation == "bind" else {"present"} - ): - _fail("RUN152_ANCHOR_EVIDENCE_STATUS_INVALID") - if ( - evidence.get("target") != {"publisher": publisher, "targetId": target_id} - or _artifact(evidence.get("record"), "RUN152_ANCHOR_EVIDENCE_RECORD") - != expected_witness_item - ): - _fail("RUN152_ANCHOR_EVIDENCE_REBIND_FAILED") - guarantees = evidence.get("guarantees") - if not isinstance(guarantees, dict) or set(guarantees) != { - "createOnly", - "overwrite", - "remoteReadbackVerified", - "immutability", - "bindingType", - "locator", - }: - _fail("RUN152_ANCHOR_EVIDENCE_GUARANTEES_SCHEMA_INVALID") - if ( - guarantees.get("createOnly") is not True - or guarantees.get("overwrite") is not False - or guarantees.get("remoteReadbackVerified") is not True - ): - _fail("RUN152_ANCHOR_EVIDENCE_GUARANTEES_INVALID") - if ( - guarantees.get("immutability") != anchor.get("immutability") - or guarantees.get("bindingType") != anchor.get("bindingType") - or _safe_locator( - guarantees.get("locator"), "RUN152_ANCHOR_EVIDENCE_LOCATOR_INVALID" - ) - != anchor_locator - ): - _fail("RUN152_ANCHOR_EVIDENCE_GUARANTEES_MISMATCH") - if operation == "verify" and evidence.get("verifiedAt") != anchor.get( - "verifiedAt" - ): - _fail("RUN152_ANCHOR_EVIDENCE_TIME_MISMATCH") - - return { - "root": root, - "release_id": release_id, - "publication_id": publication_id, - "source_revision": source_revision, - "log_id": log_id, - "previous_checkpoint": previous_checkpoint, - "accepted_checkpoint": accepted_checkpoint, - "previous_checkpoint_sha256": _sha_bytes(previous_raw), - "accepted_checkpoint_sha256": _sha_bytes(accepted_raw), - "final_record_item": expected_final_item, - "witness_record_item": expected_witness_item, - "witness_receipt_sha256": _sha_bytes(receipt_raw), - "witness_threshold": threshold, - "witness_minimum_operators": minimum_operators, - "witnesses": sorted( - normalized_meta, key=lambda x: (x["operator"], x["identity"]) - ), - "publisher": publisher, - "target_id": target_id, - "anchor_locator": anchor_locator, - } - - -def _validate_key_transition( - value: dict[str, Any], - *, - history_id: str, - log_id: str, - from_key: str, - to_key: str, - authority_identity: str, - checkpoint: dict[str, Any], - previous_tree_size: int, -) -> dict[str, Any]: - expected = { - "schemaVersion", - "historyId", - "logId", - "transitionId", - "fromKeyId", - "toKeyId", - "reason", - "effectiveTreeSize", - "revokeFromKey", - "authority", - "evidenceSha256", - } - if set(value) != expected or value.get("schemaVersion") != int( - POLICY["key_transition_schema_version"] - ): - _fail("KEY_TRANSITION_SCHEMA_INVALID") - if value.get("historyId") != history_id or value.get("logId") != log_id: - _fail("KEY_TRANSITION_HISTORY_MISMATCH") - if ( - _identity(value.get("fromKeyId"), "KEY_TRANSITION_FROM_INVALID") != from_key - or _identity(value.get("toKeyId"), "KEY_TRANSITION_TO_INVALID") != to_key - ): - _fail("KEY_TRANSITION_KEY_MISMATCH") - if from_key == to_key: - _fail("KEY_TRANSITION_NOOP") - _safe_text(value.get("transitionId"), "KEY_TRANSITION_ID_INVALID", limit=255) - reason = value.get("reason") - if reason not in set(POLICY["allowed_transition_reasons"]): - _fail("KEY_TRANSITION_REASON_INVALID") - effective = _size( - value.get("effectiveTreeSize"), "KEY_TRANSITION_TREE_SIZE_INVALID" - ) - if effective <= previous_tree_size or effective > checkpoint["treeSize"]: - _fail("KEY_TRANSITION_NOT_EFFECTIVE") - if value.get("revokeFromKey") is not True: - _fail("KEY_TRANSITION_PREVIOUS_KEY_NOT_REVOKED") - authority = value.get("authority") - authority_keys = { - "identity", - "policySignatureVerified", - "oldKeyContinuityVerified", - "newKeyProofVerified", - "emergencyRecoveryAuthorized", - } - if not isinstance(authority, dict) or set(authority) != authority_keys: - _fail("KEY_TRANSITION_AUTHORITY_SCHEMA_INVALID") - if ( - _identity(authority.get("identity"), "KEY_TRANSITION_AUTHORITY_INVALID") - != authority_identity - ): - _fail("KEY_TRANSITION_AUTHORITY_MISMATCH") - if ( - authority.get("policySignatureVerified") is not True - or authority.get("newKeyProofVerified") is not True - ): - _fail("KEY_TRANSITION_CRYPTOGRAPHIC_AUTHORITY_INVALID") - if reason == "scheduled-rotation" and ( - authority.get("oldKeyContinuityVerified") is not True - or authority.get("emergencyRecoveryAuthorized") is not False - ): - _fail("KEY_TRANSITION_SCHEDULED_ROTATION_INVALID") - if ( - reason == "compromise-recovery" - and authority.get("emergencyRecoveryAuthorized") is not True - ): - _fail("KEY_TRANSITION_EMERGENCY_AUTHORITY_INVALID") - _hex(value.get("evidenceSha256"), "KEY_TRANSITION_EVIDENCE_HASH_INVALID") - return value - - -def _validate_bundle( # ruff: ignore[too-many-branches] - bundle: dict[str, Any], -) -> dict[str, Any]: - if ( - set(bundle) - != { - "schemaVersion", - "predicateType", - "status", - "historyId", - "genesis", - "entries", - } - or bundle.get("schemaVersion") != int(POLICY["bundle_schema_version"]) - or bundle.get("predicateType") != HISTORY_PREDICATE_TYPE - or bundle.get("status") != "preserved-history" - ): - _fail("HISTORY_BUNDLE_SCHEMA_INVALID") - history_id = _safe_text(bundle.get("historyId"), "HISTORY_ID_INVALID", limit=255) - genesis = bundle.get("genesis") - genesis_keys = { - "logId", - "checkpoint", - "activeLogKeyId", - "policyAuthorityIdentity", - "bootstrapEvidenceSha256", - } - if not isinstance(genesis, dict) or set(genesis) != genesis_keys: - _fail("HISTORY_GENESIS_SCHEMA_INVALID") - log_id = _safe_text(genesis.get("logId"), "HISTORY_LOG_ID_INVALID", limit=255) - checkpoint = _checkpoint(genesis.get("checkpoint"), "HISTORY_GENESIS_CHECKPOINT") - active_key = _identity( - genesis.get("activeLogKeyId"), "HISTORY_GENESIS_LOG_KEY_INVALID" - ) - authority = _identity( - genesis.get("policyAuthorityIdentity"), "HISTORY_POLICY_AUTHORITY_INVALID" - ) - _hex( - genesis.get("bootstrapEvidenceSha256"), - "HISTORY_BOOTSTRAP_EVIDENCE_HASH_INVALID", - ) - entries = bundle.get("entries") - if not isinstance(entries, list) or len(entries) > int(POLICY["max_entries"]): - _fail("HISTORY_ENTRY_COUNT_INVALID") - revoked: set[str] = set() - release_ids: set[str] = set() - publication_ids: set[str] = set() - witness_subjects: set[str] = set() - previous_sequence = 0 - for entry in entries: - expected = { - "sequence", - "release", - "subjects", - "log", - "replicaQuorum", - "keyTransition", - } - if not isinstance(entry, dict) or set(entry) != expected: - _fail("HISTORY_ENTRY_SCHEMA_INVALID") - sequence = _size(entry.get("sequence"), "HISTORY_ENTRY_SEQUENCE_INVALID") - if sequence != previous_sequence + 1: - _fail("HISTORY_ENTRY_SEQUENCE_GAP") - previous_sequence = sequence - release = entry.get("release") - if not isinstance(release, dict) or set(release) != { - "releaseId", - "publicationId", - "sourceRevision", - }: - _fail("HISTORY_ENTRY_RELEASE_SCHEMA_INVALID") - release_id = _safe_text( - release.get("releaseId"), "HISTORY_ENTRY_RELEASE_ID_INVALID", limit=127 - ) - publication_id = _safe_text( - release.get("publicationId"), - "HISTORY_ENTRY_PUBLICATION_ID_INVALID", - limit=255, - ) - if release_id in release_ids or publication_id in publication_ids: - _fail("HISTORY_ENTRY_RELEASE_REPLAY") - release_ids.add(release_id) - publication_ids.add(publication_id) - revision = _safe_text( - release.get("sourceRevision"), "HISTORY_ENTRY_REVISION_INVALID", limit=128 - ) - if _REVISION.fullmatch(revision) is None: - _fail("HISTORY_ENTRY_REVISION_INVALID") - subjects = entry.get("subjects") - if not isinstance(subjects, dict) or set(subjects) != { - "finalPublicationRecordSha256", - "witnessRecordSha256", - "witnessReceiptSha256", - }: - _fail("HISTORY_ENTRY_SUBJECT_SCHEMA_INVALID") - for value in subjects.values(): - _hex(value, "HISTORY_ENTRY_SUBJECT_HASH_INVALID") - witness_subject = subjects["witnessRecordSha256"] - if witness_subject in witness_subjects: - _fail("HISTORY_ENTRY_WITNESS_REPLAY") - witness_subjects.add(witness_subject) - log = entry.get("log") - if ( - not isinstance(log, dict) - or set(log) - != { - "logId", - "logKeyId", - "checkpoint", - "previousCheckpointSha256", - "acceptedCheckpointSha256", - } - or log.get("logId") != log_id - ): - _fail("HISTORY_ENTRY_LOG_SCHEMA_INVALID") - new_checkpoint = _checkpoint(log.get("checkpoint"), "HISTORY_ENTRY_CHECKPOINT") - previous_tree_size = checkpoint["treeSize"] - if new_checkpoint["treeSize"] <= previous_tree_size: - _fail("HISTORY_ENTRY_CHECKPOINT_NOT_ADVANCING") - expected_previous_sha = _sha_bytes( - _canonical_bytes(_checkpoint_doc(log_id, checkpoint)) - ) - if log.get("previousCheckpointSha256") != expected_previous_sha: - _fail("HISTORY_ENTRY_PREVIOUS_CHECKPOINT_HASH_MISMATCH") - expected_accepted_sha = _sha_bytes( - _canonical_bytes(_checkpoint_doc(log_id, new_checkpoint)) - ) - if log.get("acceptedCheckpointSha256") != expected_accepted_sha: - _fail("HISTORY_ENTRY_ACCEPTED_CHECKPOINT_HASH_MISMATCH") - log_key = _identity(log.get("logKeyId"), "HISTORY_ENTRY_LOG_KEY_INVALID") - transition = entry.get("keyTransition") - if log_key == active_key: - if transition is not None: - _fail("HISTORY_ENTRY_UNEXPECTED_KEY_TRANSITION") - else: - if log_key in revoked: - _fail("HISTORY_ENTRY_REVOKED_KEY_REUSE") - if not isinstance(transition, dict): - _fail("HISTORY_ENTRY_KEY_CHANGE_WITHOUT_TRANSITION") - _validate_key_transition( - transition, - history_id=history_id, - log_id=log_id, - from_key=active_key, - to_key=log_key, - authority_identity=authority, - checkpoint=new_checkpoint, - previous_tree_size=previous_tree_size, - ) - revoked.add(active_key) - active_key = log_key - quorum = entry.get("replicaQuorum") - if not isinstance(quorum, dict) or set(quorum) != { - "threshold", - "minimumOperators", - "replicas", - }: - _fail("HISTORY_ENTRY_REPLICA_QUORUM_SCHEMA_INVALID") - threshold = _size( - quorum.get("threshold"), "HISTORY_ENTRY_REPLICA_THRESHOLD_INVALID" - ) - minimum_operators = _size( - quorum.get("minimumOperators"), - "HISTORY_ENTRY_REPLICA_OPERATOR_THRESHOLD_INVALID", - ) - replicas = quorum.get("replicas") - if ( - threshold < int(POLICY["min_replica_quorum"]) - or minimum_operators < int(POLICY["min_replica_operators"]) - or not isinstance(replicas, list) - or len(replicas) < threshold - ): - _fail("HISTORY_ENTRY_REPLICA_QUORUM_INVALID") - identities: set[str] = set() - operators: set[str] = set() - for replica in replicas: - if not isinstance(replica, dict) or set(replica) != { - "identity", - "operator", - }: - _fail("HISTORY_ENTRY_REPLICA_SCHEMA_INVALID") - identity = _identity( - replica.get("identity"), "HISTORY_ENTRY_REPLICA_IDENTITY_INVALID" - ) - operator = _identity( - replica.get("operator"), "HISTORY_ENTRY_REPLICA_OPERATOR_INVALID" - ) - if identity in identities: - _fail("HISTORY_ENTRY_REPLICA_IDENTITY_DUPLICATE") - identities.add(identity) - operators.add(operator) - if len(operators) < minimum_operators: - _fail("HISTORY_ENTRY_REPLICA_OPERATOR_QUORUM_INVALID") - checkpoint = new_checkpoint - return { - "history_id": history_id, - "log_id": log_id, - "checkpoint": checkpoint, - "active_log_key_id": active_key, - "policy_authority_identity": authority, - "revoked_log_key_ids": sorted(revoked), - "sequence": len(entries), - "chain_head_sha256": _history_head(history_id, genesis, entries), - "genesis": genesis, - "entries": entries, - } - - -def _validate_previous_history(state_path: Path, bundle_path: Path) -> dict[str, Any]: - state, state_raw = _read_json(state_path, "PREVIOUS_HISTORY_STATE") - bundle, bundle_raw = _read_json(bundle_path, "PREVIOUS_HISTORY_BUNDLE") - bundle_info = _validate_bundle(bundle) - expected_state = { - "schemaVersion", - "historyId", - "sequence", - "logId", - "activeLogKeyId", - "policyAuthorityIdentity", - "checkpoint", - "bundleSha256", - "chainHeadSha256", - "revokedLogKeyIds", - } - if set(state) != expected_state or state.get("schemaVersion") != int( - POLICY["state_schema_version"] - ): - _fail("PREVIOUS_HISTORY_STATE_SCHEMA_INVALID") - if ( - state.get("historyId") != bundle_info["history_id"] - or state.get("logId") != bundle_info["log_id"] - ): - _fail("PREVIOUS_HISTORY_STATE_ID_MISMATCH") - if ( - _size(state.get("sequence"), "PREVIOUS_HISTORY_SEQUENCE_INVALID") - != bundle_info["sequence"] - ): - _fail("PREVIOUS_HISTORY_SEQUENCE_MISMATCH") - if ( - _identity(state.get("activeLogKeyId"), "PREVIOUS_HISTORY_LOG_KEY_INVALID") - != bundle_info["active_log_key_id"] - ): - _fail("PREVIOUS_HISTORY_LOG_KEY_MISMATCH") - if ( - _identity( - state.get("policyAuthorityIdentity"), "PREVIOUS_HISTORY_AUTHORITY_INVALID" - ) - != bundle_info["policy_authority_identity"] - ): - _fail("PREVIOUS_HISTORY_AUTHORITY_MISMATCH") - if ( - _checkpoint(state.get("checkpoint"), "PREVIOUS_HISTORY_CHECKPOINT") - != bundle_info["checkpoint"] - ): - _fail("PREVIOUS_HISTORY_CHECKPOINT_MISMATCH") - if ( - state.get("bundleSha256") != _sha_bytes(bundle_raw) - or state.get("chainHeadSha256") != bundle_info["chain_head_sha256"] - ): - _fail("PREVIOUS_HISTORY_HASH_MISMATCH") - revoked = state.get("revokedLogKeyIds") - if ( - not isinstance(revoked, list) - or sorted( - {_identity(x, "PREVIOUS_HISTORY_REVOKED_KEY_INVALID") for x in revoked} - ) - != bundle_info["revoked_log_key_ids"] - ): - _fail("PREVIOUS_HISTORY_REVOKED_KEYS_MISMATCH") - return { - **bundle_info, - "state": state, - "state_sha256": _sha_bytes(state_raw), - "bundle": bundle, - "bundle_sha256": _sha_bytes(bundle_raw), - } - - -def verify_history_bundle( # ruff: ignore[undocumented-public-function] - *, bundle_path: Path, state_path: Path | None = None -) -> dict[str, Any]: - bundle, raw = _read_json(bundle_path, "HISTORY_BUNDLE") - info = _validate_bundle(bundle) - if state_path is not None: - state, _ = _read_json(state_path, "HISTORY_STATE") - expected = { - "schemaVersion", - "historyId", - "sequence", - "logId", - "activeLogKeyId", - "policyAuthorityIdentity", - "checkpoint", - "bundleSha256", - "chainHeadSha256", - "revokedLogKeyIds", - } - if set(state) != expected or state.get("schemaVersion") != int( - POLICY["state_schema_version"] - ): - _fail("HISTORY_STATE_SCHEMA_INVALID") - if ( - state.get("historyId") != info["history_id"] - or state.get("sequence") != info["sequence"] - or state.get("logId") != info["log_id"] - ): - _fail("HISTORY_STATE_ID_MISMATCH") - if ( - state.get("activeLogKeyId") != info["active_log_key_id"] - or state.get("policyAuthorityIdentity") != info["policy_authority_identity"] - or _checkpoint(state.get("checkpoint"), "HISTORY_STATE_CHECKPOINT") - != info["checkpoint"] - ): - _fail("HISTORY_STATE_TRUST_MISMATCH") - if ( - state.get("bundleSha256") != _sha_bytes(raw) - or state.get("chainHeadSha256") != info["chain_head_sha256"] - or state.get("revokedLogKeyIds") != info["revoked_log_key_ids"] - ): - _fail("HISTORY_STATE_HASH_MISMATCH") - return { - "ok": True, - "history_id": info["history_id"], - "sequence": info["sequence"], - "log_id": info["log_id"], - "active_log_key_id": info["active_log_key_id"], - "chain_head_sha256": info["chain_head_sha256"], - "bundle_sha256": _sha_bytes(raw), - } - - -def _observation_id(history_id: str, sequence: int, identity: str) -> str: - return ( - "history-observe-" - + hashlib.sha256( - (history_id + "\0" + str(sequence) + "\0" + identity).encode() - ).hexdigest()[:32] - ) - - -def _archive_id(history_id: str, sequence: int, bundle_sha: str, identity: str) -> str: - return ( - "history-archive-" - + hashlib.sha256( - ( - history_id + "\0" + str(sequence) + "\0" + bundle_sha + "\0" + identity - ).encode() - ).hexdigest()[:32] - ) - - -def _command_adapter( - command: list[str], *, prefix: str -) -> Callable[[dict[str, Any]], dict[str, Any]]: - if not command: - _fail(prefix + "_COMMAND_INVALID") - - def call(request: dict[str, Any]) -> dict[str, Any]: - process = ( - # lint - subprocess.Popen( # ruff: ignore[subprocess-without-shell-equals-true] - command, - stdin=subprocess.PIPE, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, - text=False, - close_fds=True, - env={"PATH": os.defpath, "LC_ALL": "C", "LANG": "C"}, - ) - ) - assert process.stdin is not None # ruff: ignore[assert] - assert process.stdout is not None # ruff: ignore[assert] - assert process.stderr is not None # ruff: ignore[assert] - stdin = process.stdin - process.stdin = None - stderr_chunks: list[bytes] = [] - stdout_chunks: list[bytes] = [] - overflow = threading.Event() - limit = int(POLICY["max_adapter_output_bytes"]) - - def drain(stream, chunks): - total = 0 - while True: - chunk = stream.read(65536) - if not chunk: - break - total += len(chunk) - if total > limit: - overflow.set() - process.kill() - break - chunks.append(chunk) - - threads = [ - threading.Thread(target=drain, args=(process.stdout, stdout_chunks)), - threading.Thread(target=drain, args=(process.stderr, stderr_chunks)), - ] - for thread in threads: - thread.start() - try: - stdin.write(_canonical_bytes(request)) - stdin.close() - process.wait(timeout=int(POLICY["adapter_timeout_seconds"])) - except subprocess.TimeoutExpired as exc: - process.kill() - raise HistoryError(prefix + "_TIMEOUT") from exc - finally: - for thread in threads: - thread.join() - if overflow.is_set(): - _fail(prefix + "_OUTPUT_TOO_LARGE") - if process.returncode != 0: - _fail(prefix + "_FAILED") - return _loads_json(b"".join(stdout_chunks), prefix + "_OUTPUT") - - return call - - -def command_replica(command: list[str]): # ruff: ignore[undocumented-public-function] - return _command_adapter(command, prefix="HISTORY_REPLICA_ADAPTER") - - -def command_archive(command: list[str]): # ruff: ignore[undocumented-public-function] - return _command_adapter(command, prefix="HISTORY_ARCHIVE_ADAPTER") - - -def _validate_replica_response( # ruff: ignore[too-many-branches] - value: dict[str, Any], - *, - observation_id: str, - history_id: str, - identity: str, - operator: str, - previous: dict[str, Any], - current: dict[str, Any], - now: datetime, -) -> dict[str, Any]: - common = { - "schemaVersion", - "operation", - "observationId", - "historyId", - "status", - "replica", - "observedAt", - } - if ( - value.get("schemaVersion") != int(POLICY["replica_protocol_version"]) - or value.get("operation") != "observe" - or value.get("observationId") != observation_id - or value.get("historyId") != history_id - ): - _fail("HISTORY_REPLICA_RESULT_ID_MISMATCH") - replica = value.get("replica") - replica_keys = { - "identity", - "operator", - "readOnly", - "logCredentialsReused", - "publisherCredentialsReused", - "historyWriterCredentialsReused", - } - if ( - not isinstance(replica, dict) - or set(replica) != replica_keys - or replica.get("identity") != identity - or replica.get("operator") != operator - ): - _fail("HISTORY_REPLICA_AUTHORITY_SCHEMA_INVALID") - if ( - replica.get("readOnly") is not True - or replica.get("logCredentialsReused") is not False - or replica.get("publisherCredentialsReused") is not False - or replica.get("historyWriterCredentialsReused") is not False - ): - _fail("HISTORY_REPLICA_AUTHORITY_INVALID") - observed_at = _fresh( - value.get("observedAt"), "HISTORY_REPLICA_OBSERVED_AT", now=now - ) - status = value.get("status") - if status == "unavailable": - if set(value) != common | {"reason"} or not bool( - POLICY["allow_unavailable_replica"] - ): - _fail("HISTORY_REPLICA_UNAVAILABLE_INVALID") - reason = _safe_text( - value.get("reason"), "HISTORY_REPLICA_UNAVAILABLE_REASON_INVALID", limit=255 - ) - return { - "status": "unavailable", - "identity": identity, - "operator": operator, - "observedAt": observed_at, - "reason": reason, - } - if status != "observed": - _fail("HISTORY_REPLICA_STATUS_INVALID") - expected = common | { - "history", - "log", - "checkpoint", - "previousCheckpoint", - "witnessAnchor", - "proof", - } - if set(value) != expected: - _fail("HISTORY_REPLICA_RESULT_SCHEMA_INVALID") - history = value.get("history") - expected_history = { - "sequence": previous["sequence"], - "previousBundleSha256": previous["bundle_sha256"], - "previousChainHeadSha256": previous["chain_head_sha256"], - } - if history != expected_history: - _fail("HISTORY_REPLICA_PRIOR_HISTORY_MISMATCH") - log = value.get("log") - if ( - not isinstance(log, dict) - or set(log) != {"logId", "keyId"} - or log.get("logId") != current["log_id"] - ): - _fail("HISTORY_REPLICA_LOG_MISMATCH") - key_id = _identity(log.get("keyId"), "HISTORY_REPLICA_LOG_KEY_INVALID") - if ( - _checkpoint(value.get("checkpoint"), "HISTORY_REPLICA_CHECKPOINT") - != current["accepted_checkpoint"] - ): - _fail("HISTORY_REPLICA_CURRENT_CHECKPOINT_MISMATCH") - if ( - _checkpoint( - value.get("previousCheckpoint"), "HISTORY_REPLICA_PREVIOUS_CHECKPOINT" - ) - != previous["checkpoint"] - ): - _fail("HISTORY_REPLICA_PREVIOUS_CHECKPOINT_MISMATCH") - witness_anchor = value.get("witnessAnchor") - if not isinstance(witness_anchor, dict) or set(witness_anchor) != { - "locator", - "record", - }: - _fail("HISTORY_REPLICA_WITNESS_ANCHOR_SCHEMA_INVALID") - if ( - _safe_locator( - witness_anchor.get("locator"), - "HISTORY_REPLICA_WITNESS_ANCHOR_LOCATOR_INVALID", - ) - != current["anchor_locator"] - or _artifact( - witness_anchor.get("record"), "HISTORY_REPLICA_WITNESS_ANCHOR_RECORD" - ) - != current["witness_record_item"] - ): - _fail("HISTORY_REPLICA_WITNESS_ANCHOR_MISMATCH") - proof = value.get("proof") - expected_proof = { - "currentCheckpointSignatureVerified": True, - "consistencyFromPreviousVerified": True, - "priorHistoryHeadVerified": True, - "independentGossipSource": True, - "logKeyIdentityVerified": True, - "witnessRecordRemoteReadbackVerified": True, - } - if proof != expected_proof: - _fail("HISTORY_REPLICA_PROOF_INVALID") - return { - "status": "observed", - "identity": identity, - "operator": operator, - "keyId": key_id, - "observedAt": observed_at, - } - - -def _validate_archive_response( - value: dict[str, Any], - *, - operation: str, - archive_id: str, - history_id: str, - sequence: int, - identity: str, - operator: str, - artifact: dict[str, Any], - now: datetime, -) -> dict[str, Any]: - expected = { - "schemaVersion", - "operation", - "archiveId", - "historyId", - "sequence", - "status", - "archive", - "artifact", - "guarantees", - "verifiedAt", - } - if ( - set(value) != expected - or value.get("schemaVersion") != int(POLICY["archive_protocol_version"]) - or value.get("operation") != operation - or value.get("archiveId") != archive_id - or value.get("historyId") != history_id - or value.get("sequence") != sequence - ): - _fail("HISTORY_ARCHIVE_RESULT_SCHEMA_INVALID") - if value.get("status") not in ( - {"created", "present"} if operation == "bind" else {"present"} - ): - _fail("HISTORY_ARCHIVE_STATUS_INVALID") - archive = value.get("archive") - archive_keys = { - "identity", - "operator", - "historyWriterCredentialsReused", - "transparencyLogCredentialsReused", - } - if ( - not isinstance(archive, dict) - or set(archive) != archive_keys - or archive.get("identity") != identity - or archive.get("operator") != operator - ): - _fail("HISTORY_ARCHIVE_AUTHORITY_SCHEMA_INVALID") - if ( - archive.get("historyWriterCredentialsReused") is not False - or archive.get("transparencyLogCredentialsReused") is not False - ): - _fail("HISTORY_ARCHIVE_AUTHORITY_INVALID") - if _artifact(value.get("artifact"), "HISTORY_ARCHIVE_ARTIFACT") != artifact: - _fail("HISTORY_ARCHIVE_ARTIFACT_MISMATCH") - guarantees = value.get("guarantees") - if not isinstance(guarantees, dict) or set(guarantees) != { - "createOnly", - "overwrite", - "remoteReadbackVerified", - "immutability", - "locator", - }: - _fail("HISTORY_ARCHIVE_GUARANTEES_SCHEMA_INVALID") - if ( - guarantees.get("createOnly") is not True - or guarantees.get("overwrite") is not False - or guarantees.get("remoteReadbackVerified") is not True - ): - _fail("HISTORY_ARCHIVE_GUARANTEES_INVALID") - immutability = guarantees.get("immutability") - if immutability not in set(POLICY["allowed_archive_immutability"]): - _fail("HISTORY_ARCHIVE_IMMUTABILITY_INVALID") - locator = _safe_locator( - guarantees.get("locator"), "HISTORY_ARCHIVE_LOCATOR_INVALID" - ) - verified_at = _fresh( - value.get("verifiedAt"), "HISTORY_ARCHIVE_VERIFIED_AT", now=now - ) - return {"locator": locator, "immutability": immutability, "verifiedAt": verified_at} - - -ReplicaAdapter = Callable[[dict[str, Any]], dict[str, Any]] -ArchiveAdapter = Callable[[dict[str, Any]], dict[str, Any]] - - -# ── Verified-state document names ──────────────────────────────────────── -# -# One constant per release-evidence document. The values are the on-disk -# artifact names and are unchanged; only the repetition is removed. Reading -# a document by a name that says which document it is also keeps static -# analysis from reading the filename token 'trusted' as 'confidential': -# these files carry published Merkle roots and SHA-256 digests, which must -# stay in clear text for any third party to verify them. -_DOC_HISTORY_STATE = "trusted-history-state.json" - - -def preserve_release_history( # ruff: ignore[too-many-branches, undocumented-public-function] - *, - witnessed_dir: Path, - previous_state: Path, - previous_bundle: Path, - output_dir: Path, - collector_identity: str, - replicas: list[tuple[str, str, ReplicaAdapter]], - replica_quorum: int, - archives: list[tuple[str, str, ArchiveAdapter]], - key_transition: Path | None = None, - now: datetime | None = None, -) -> dict[str, Any]: - witnessed_dir = _regular_dir(witnessed_dir, "RUN152_DIRECTORY_INVALID") - target = _outside(output_dir, (witnessed_dir,), "HISTORY_OUTPUT_INSIDE_INPUT") - if target.exists() or target.is_symlink(): - _fail("HISTORY_OUTPUT_ALREADY_EXISTS") - current_time = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - current = _validate_witnessed(witnessed_dir) - previous = _validate_previous_history(previous_state, previous_bundle) - if current["log_id"] != previous["log_id"]: - _fail("HISTORY_LOG_ID_MISMATCH") - if current["previous_checkpoint"] != previous["checkpoint"]: - _fail("HISTORY_CHECKPOINT_GAP_OR_FORK") - expected_previous_doc = _checkpoint_doc(previous["log_id"], previous["checkpoint"]) - if current["previous_checkpoint_sha256"] != _sha_bytes( - _canonical_bytes(expected_previous_doc) - ): - _fail("HISTORY_PREVIOUS_CHECKPOINT_BYTES_MISMATCH") - if current["accepted_checkpoint"]["treeSize"] <= previous["checkpoint"]["treeSize"]: - _fail("HISTORY_CHECKPOINT_NOT_ADVANCING") - history_id = previous["history_id"] - collector_identity = _identity( - collector_identity, "HISTORY_COLLECTOR_IDENTITY_INVALID" - ) - - if len(replicas) < int(POLICY["min_replicas"]) or len(replicas) > int( - POLICY["max_replicas"] - ): - _fail("HISTORY_REPLICA_COUNT_INVALID") - replica_quorum = _size(replica_quorum, "HISTORY_REPLICA_QUORUM_INVALID") - if replica_quorum < int(POLICY["min_replica_quorum"]) or replica_quorum > len( - replicas - ): - _fail("HISTORY_REPLICA_QUORUM_INVALID") - normalized_replicas: list[tuple[str, str, ReplicaAdapter]] = [] - replica_ids: set[str] = set() - replica_operators: set[str] = set() - for identity_raw, operator_raw, adapter in replicas: - identity = _identity(identity_raw, "HISTORY_REPLICA_IDENTITY_INVALID") - operator = _identity(operator_raw, "HISTORY_REPLICA_OPERATOR_INVALID") - if identity == collector_identity or identity in replica_ids: - _fail("HISTORY_REPLICA_IDENTITY_NOT_DISTINCT") - replica_ids.add(identity) - replica_operators.add(operator) - normalized_replicas.append((identity, operator, adapter)) - if len(replica_operators) < int(POLICY["min_replica_operators"]): - _fail("HISTORY_REPLICA_OPERATOR_COUNT_INVALID") - - if len(archives) < int(POLICY["min_archives"]) or len(archives) > int( - POLICY["max_archives"] - ): - _fail("HISTORY_ARCHIVE_COUNT_INVALID") - normalized_archives: list[tuple[str, str, ArchiveAdapter]] = [] - archive_ids: set[str] = set() - archive_operators: set[str] = set() - for identity_raw, operator_raw, adapter in archives: - identity = _identity(identity_raw, "HISTORY_ARCHIVE_IDENTITY_INVALID") - operator = _identity(operator_raw, "HISTORY_ARCHIVE_OPERATOR_INVALID") - if ( - identity == collector_identity - or identity in archive_ids - or identity in replica_ids - ): - _fail("HISTORY_ARCHIVE_IDENTITY_NOT_DISTINCT") - archive_ids.add(identity) - archive_operators.add(operator) - normalized_archives.append((identity, operator, adapter)) - if len(archive_operators) < int(POLICY["min_archive_operators"]): - _fail("HISTORY_ARCHIVE_OPERATOR_COUNT_INVALID") - - input_files = [ - witnessed_dir / "release-publication-record.json", - witnessed_dir / "release-transparency-witness-record.json", - witnessed_dir / "release-transparency-witness-receipt.json", - witnessed_dir / "previous-transparency-checkpoint.json", - witnessed_dir / "accepted-transparency-checkpoint.json", - previous_state, - previous_bundle, - ] - if key_transition is not None: - input_files.append(key_transition) - initial_hashes = {str(path.resolve()): _sha256(path) for path in input_files} - sequence = previous["sequence"] + 1 - target.parent.mkdir(parents=True, exist_ok=True) - - with tempfile.TemporaryDirectory( - prefix="release-history-", dir=target.parent - ) as temp_raw: - stage = Path(temp_raw) / "history" - stage.mkdir() - replica_dir = stage / "replica-results" - archive_dir = stage / "archive-results" - replica_dir.mkdir() - archive_dir.mkdir() - observed: list[dict[str, Any]] = [] - unavailable: list[dict[str, Any]] = [] - replica_evidence: list[dict[str, Any]] = [] - for index, (identity, operator, adapter) in enumerate( - sorted(normalized_replicas, key=lambda x: (x[1], x[0])), start=1 - ): - observation_id = _observation_id(history_id, sequence, identity) - request = { - "schemaVersion": int(POLICY["replica_protocol_version"]), - "operation": "observe", - "observationId": observation_id, - "historyId": history_id, - "sequence": sequence, - "replica": {"identity": identity, "operator": operator}, - "history": { - "previousBundleSha256": previous["bundle_sha256"], - "previousChainHeadSha256": previous["chain_head_sha256"], - }, - "log": {"logId": current["log_id"]}, - "checkpoint": current["accepted_checkpoint"], - "previousCheckpoint": previous["checkpoint"], - "subjects": { - "witnessRecordSha256": current["witness_record_item"]["sha256"], - "witnessReceiptSha256": current["witness_receipt_sha256"], - }, - "witnessAnchor": { - "locator": current["anchor_locator"], - "record": current["witness_record_item"], - }, - } - raw = adapter(request) - result = _validate_replica_response( - raw, - observation_id=observation_id, - history_id=history_id, - identity=identity, - operator=operator, - previous=previous, - current=current, - now=current_time, - ) - path = ( - replica_dir - / f"{index:02d}-{hashlib.sha256(identity.encode()).hexdigest()[:12]}.observe.json" - ) - _write_canonical(path, raw) - replica_evidence.append( - { - "identity": identity, - "operator": operator, - "status": result["status"], - "sha256": _sha256(path), - } - ) - (observed if result["status"] == "observed" else unavailable).append(result) - - if len(observed) < replica_quorum: - _fail("HISTORY_REPLICA_QUORUM_NOT_MET") - agreeing_operators = {x["operator"] for x in observed} - if len(agreeing_operators) < int(POLICY["min_replica_operators"]): - _fail("HISTORY_REPLICA_OPERATOR_QUORUM_NOT_MET") - key_ids = {x["keyId"] for x in observed} - if len(key_ids) != 1: - _fail("HISTORY_REPLICA_LOG_KEY_EQUIVOCATION") - current_key = next(iter(key_ids)) - if current_key in set(previous["revoked_log_key_ids"]) and not bool( - POLICY["allow_revoked_log_key"] - ): - _fail("HISTORY_REVOKED_LOG_KEY_REJECTED") - - transition_doc: dict[str, Any] | None = None - if current_key != previous["active_log_key_id"]: - if key_transition is None and not bool( - POLICY["allow_log_key_change_without_transition"] - ): - _fail("HISTORY_LOG_KEY_CHANGE_WITHOUT_TRANSITION") - assert key_transition is not None # ruff: ignore[assert] - transition_doc, _transition_raw = _read_json( - key_transition, "KEY_TRANSITION" - ) - _validate_key_transition( - transition_doc, - history_id=history_id, - log_id=current["log_id"], - from_key=previous["active_log_key_id"], - to_key=current_key, - authority_identity=previous["policy_authority_identity"], - checkpoint=current["accepted_checkpoint"], - previous_tree_size=previous["checkpoint"]["treeSize"], - ) - shutil.copy2(key_transition, stage / "key-transition.json") - elif key_transition is not None: - _fail("HISTORY_UNEXPECTED_KEY_TRANSITION") - - configured_replica_meta = sorted( - [ - {"identity": identity, "operator": operator} - for identity, operator, _ in normalized_replicas - ], - key=lambda x: (x["operator"], x["identity"]), - ) - entry = { - "sequence": sequence, - "release": { - "releaseId": current["release_id"], - "publicationId": current["publication_id"], - "sourceRevision": current["source_revision"], - }, - "subjects": { - "finalPublicationRecordSha256": current["final_record_item"]["sha256"], - "witnessRecordSha256": current["witness_record_item"]["sha256"], - "witnessReceiptSha256": current["witness_receipt_sha256"], - }, - "log": { - "logId": current["log_id"], - "logKeyId": current_key, - "checkpoint": current["accepted_checkpoint"], - "previousCheckpointSha256": current["previous_checkpoint_sha256"], - "acceptedCheckpointSha256": current["accepted_checkpoint_sha256"], - }, - "replicaQuorum": { - "threshold": replica_quorum, - "minimumOperators": int(POLICY["min_replica_operators"]), - "replicas": configured_replica_meta, - }, - "keyTransition": transition_doc, - } - new_bundle = dict(previous["bundle"]) - new_bundle["entries"] = [*previous["entries"], entry] - bundle_path = stage / "release-history-bundle.json" - _write_canonical(bundle_path, new_bundle) - bundle_info = _validate_bundle(new_bundle) - if ( - bundle_info["sequence"] != sequence - or bundle_info["checkpoint"] != current["accepted_checkpoint"] - or bundle_info["active_log_key_id"] != current_key - ): - _fail("HISTORY_BUNDLE_POSTBUILD_VALIDATION_FAILED") - bundle_item = { - "name": bundle_path.name, - "sha256": _sha256(bundle_path), - "size": bundle_path.stat().st_size, - } - new_state = { - "schemaVersion": int(POLICY["state_schema_version"]), - "historyId": history_id, - "sequence": sequence, - "logId": current["log_id"], - "activeLogKeyId": current_key, - "policyAuthorityIdentity": previous["policy_authority_identity"], - "checkpoint": current["accepted_checkpoint"], - "bundleSha256": bundle_item["sha256"], - "chainHeadSha256": bundle_info["chain_head_sha256"], - "revokedLogKeyIds": bundle_info["revoked_log_key_ids"], - } - state_path = stage / _DOC_HISTORY_STATE - _write_canonical(state_path, new_state) - verify_history_bundle(bundle_path=bundle_path, state_path=state_path) - state_item = { - "name": state_path.name, - "sha256": _sha256(state_path), - "size": state_path.stat().st_size, - } - - archive_evidence: list[dict[str, Any]] = [] - locators: set[str] = set() - for index, (identity, operator, adapter) in enumerate( - sorted(normalized_archives, key=lambda x: (x[1], x[0])), start=1 - ): - archive_id = _archive_id( - history_id, sequence, bundle_item["sha256"], identity - ) - base_request = { - "schemaVersion": int(POLICY["archive_protocol_version"]), - "archiveId": archive_id, - "historyId": history_id, - "sequence": sequence, - "archive": {"identity": identity, "operator": operator}, - "artifact": bundle_item, - } - bind_request = dict(base_request, operation="bind") - bind_request["artifact"] = dict(bundle_item, localPath=str(bundle_path)) - before = _sha256(bundle_path) - bind_raw = adapter(bind_request) - if _sha256(bundle_path) != before: - _fail("HISTORY_BUNDLE_CHANGED_DURING_ARCHIVE") - bind = _validate_archive_response( - bind_raw, - operation="bind", - archive_id=archive_id, - history_id=history_id, - sequence=sequence, - identity=identity, - operator=operator, - artifact=bundle_item, - now=current_time, - ) - verify_raw = adapter(dict(base_request, operation="verify")) - verify = _validate_archive_response( - verify_raw, - operation="verify", - archive_id=archive_id, - history_id=history_id, - sequence=sequence, - identity=identity, - operator=operator, - artifact=bundle_item, - now=current_time, - ) - if ( - bind["locator"] != verify["locator"] - or bind["immutability"] != verify["immutability"] - ): - _fail("HISTORY_ARCHIVE_VERIFY_REBIND_FAILED") - if verify["locator"] in locators: - _fail("HISTORY_ARCHIVE_LOCATOR_COLLISION") - locators.add(verify["locator"]) - bind_path = ( - archive_dir - / f"{index:02d}-{hashlib.sha256(identity.encode()).hexdigest()[:12]}.bind.json" - ) - verify_path = ( - archive_dir - / f"{index:02d}-{hashlib.sha256(identity.encode()).hexdigest()[:12]}.verify.json" - ) - _write_canonical(bind_path, bind_raw) - _write_canonical(verify_path, verify_raw) - archive_evidence.append( - { - "identity": identity, - "operator": operator, - "archiveId": archive_id, - "locator": verify["locator"], - "immutability": verify["immutability"], - "verifiedAt": verify["verifiedAt"], - "bindEvidenceSha256": _sha256(bind_path), - "verifyEvidenceSha256": _sha256(verify_path), - } - ) - - for path in input_files: - if _sha256(path) != initial_hashes[str(path.resolve())]: - _fail("HISTORY_INPUT_CHANGED_DURING_PRESERVATION") - if ( - _sha256(bundle_path) != bundle_item["sha256"] - or _sha256(state_path) != state_item["sha256"] - ): - _fail("HISTORY_OUTPUT_CHANGED_DURING_ARCHIVE") - - for name in ( - "release-transparency-witness-record.json", - "release-transparency-witness-receipt.json", - "previous-transparency-checkpoint.json", - "accepted-transparency-checkpoint.json", - ): - shutil.copy2(witnessed_dir / name, stage / name) - shutil.copy2(previous_state, stage / "previous-trusted-history-state.json") - shutil.copy2(previous_bundle, stage / "previous-release-history-bundle.json") - receipt = { - "schemaVersion": int(POLICY["receipt_schema_version"]), - "status": "preserved", - "historyId": history_id, - "sequence": sequence, - "collectorIdentity": collector_identity, - "previous": { - "stateSha256": previous["state_sha256"], - "bundleSha256": previous["bundle_sha256"], - "chainHeadSha256": previous["chain_head_sha256"], - "checkpoint": previous["checkpoint"], - }, - "current": { - "releaseId": current["release_id"], - "publicationId": current["publication_id"], - "witnessRecordSha256": current["witness_record_item"]["sha256"], - "witnessReceiptSha256": current["witness_receipt_sha256"], - "acceptedCheckpointSha256": current["accepted_checkpoint_sha256"], - "logKeyId": current_key, - }, - "replicaQuorum": { - "configured": len(normalized_replicas), - "threshold": replica_quorum, - "observed": len(observed), - "unavailable": len(unavailable), - "minimumOperators": int(POLICY["min_replica_operators"]), - "evidence": replica_evidence, - }, - "historyBundle": bundle_item, - "trustedHistoryState": state_item, - "archives": archive_evidence, - } - receipt_path = stage / "release-history-preservation-receipt.json" - _write_canonical(receipt_path, receipt) - tmp_target = target.with_name(target.name + ".tmp-" + os.urandom(6).hex()) - shutil.copytree(stage, tmp_target, copy_function=shutil.copy2) - os.replace(tmp_target, target) - - return { - "ok": True, - "phase": "history-preserved", - "history_id": history_id, - "sequence": sequence, - "release_id": current["release_id"], - "replica_count": len(normalized_replicas), - "replica_observed": len(observed), - "replica_quorum": replica_quorum, - "archive_count": len(normalized_archives), - "bundle_sha256": _sha256(target / "release-history-bundle.json"), - "state_sha256": _sha256(target / _DOC_HISTORY_STATE), - "chain_head_sha256": bundle_info["chain_head_sha256"], - } - - -def _parse_endpoint(value: str, label: str) -> tuple[str, str, list[str]]: - parts = value.split("=", 1) - if len(parts) != 2 or "@" not in parts[0]: # ruff: ignore[magic-value-comparison] - raise argparse.ArgumentTypeError( - f"{label} must be IDENTITY@OPERATOR=EXECUTABLE[,ARG...]" - ) - identity, operator = parts[0].split("@", 1) - command = [part for part in parts[1].split(",") if part] - if not identity or not operator or not command: - raise argparse.ArgumentTypeError(f"invalid {label} specification") - return identity, operator, command - - -def main( # ruff: ignore[undocumented-public-function] - argv: list[str] | None = None, -) -> int: - parser = argparse.ArgumentParser(description=__doc__) - sub = parser.add_subparsers(dest="command", required=True) - verify_parser = sub.add_parser( - "verify", help="offline-verify a canonical history bundle" - ) - verify_parser.add_argument("--bundle", type=Path, required=True) - verify_parser.add_argument("--state", type=Path) - advance = sub.add_parser( - "advance", help="advance witnessed release history after gossip quorum" - ) - advance.add_argument("--witnessed-dir", type=Path, required=True) - advance.add_argument("--previous-state", type=Path, required=True) - advance.add_argument("--previous-bundle", type=Path, required=True) - advance.add_argument("--output-dir", type=Path, required=True) - advance.add_argument("--collector-identity", required=True) - advance.add_argument("--replica-quorum", type=int, required=True) - advance.add_argument( - "--replica", - action="append", - required=True, - type=lambda x: _parse_endpoint(x, "replica"), - ) - advance.add_argument( - "--archive", - action="append", - required=True, - type=lambda x: _parse_endpoint(x, "archive"), - ) - advance.add_argument("--key-transition", type=Path) - args = parser.parse_args(argv) - try: - if args.command == "verify": - result = verify_history_bundle( - bundle_path=args.bundle, state_path=args.state - ) - else: - replicas = [ - (identity, operator, command_replica(command)) - for identity, operator, command in args.replica - ] - archives = [ - (identity, operator, command_archive(command)) - for identity, operator, command in args.archive - ] - result = preserve_release_history( - witnessed_dir=args.witnessed_dir, - previous_state=args.previous_state, - previous_bundle=args.previous_bundle, - output_dir=args.output_dir, - collector_identity=args.collector_identity, - replicas=replicas, - replica_quorum=args.replica_quorum, - archives=archives, - key_transition=args.key_transition, - ) - except (HistoryError, OSError) as exc: - logger.error("history preservation failed: %s", exc) - # sys.stdout.write( - # json.dumps({"ok": False, "error": str(exc)}, sort_keys=True) + "\n" - # ) - return 2 - logger.info("%s", json.dumps(result, sort_keys=True)) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/security/probe_redis_authority.py b/security/probe_redis_authority.py deleted file mode 100644 index 7bf22c88ee2a852f4d4795a1aa1ae981600df318..0000000000000000000000000000000000000000 --- a/security/probe_redis_authority.py +++ /dev/null @@ -1,163 +0,0 @@ -""" -Collect sanitized Redis operational evidence for one control plane. - -The Redis URL is read only from an environment variable named by the operator; -it is never accepted as a command-line value and never printed. Output contains -no host, port, username, keys, values, replication offsets, or credentials. -""" - -from __future__ import annotations - -import argparse -import json -import logging -import os -import sys -from datetime import datetime, timezone -from pathlib import Path -from typing import Any - -ROOT = Path(__file__).resolve().parents[1] -if str(ROOT) not in sys.path: - sys.path.insert(0, str(ROOT)) -from _utils._redis_security import ( # noqa: E402 - RedisSecurityError, - redis_connection_kwargs, -) - -logger = logging.getLogger(__name__) - - -def _now() -> str: - return ( - datetime.now(timezone.utc).isoformat(timespec="seconds").replace("+00:00", "Z") - ) - - -def _int(value: Any, default: int = 0) -> int: - try: - return int(value) - except (TypeError, ValueError): - return default - - -def collect( # ruff: ignore[too-many-branches, undocumented-public-function] - *, - plane: str, - url_env: str, - client: Any | None = None, -) -> dict[str, Any]: - if plane not in {"rateLimit", "share", "contribution", "providerArtifactLifecycle"}: - raise RuntimeError("PLANE_INVALID") - if not url_env or any( - ch not in "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_" for ch in url_env - ): - raise RuntimeError("URL_ENV_NAME_INVALID") - url = os.environ.get(url_env, "") - try: - policy, kwargs = redis_connection_kwargs( - url, require_tls=True, socket_timeout_seconds=3.0 - ) - except RedisSecurityError as exc: - raise RuntimeError(exc.code) from exc - if client is None: - try: - import redis # type: ignore[] # ruff: ignore[import-outside-top-level] - - client = redis.Redis.from_url(url, **kwargs) - except Exception as exc: - raise RuntimeError("REDIS_CLIENT_UNAVAILABLE") from exc - try: - ping = bool(client.ping()) - except Exception as exc: - raise RuntimeError("REDIS_PING_FAILED") from exc - if not ping: - raise RuntimeError("REDIS_PING_FAILED") - - non_default_identity = False - acl_identity_observed = False - try: - who = client.execute_command("ACL", "WHOAMI") - if isinstance(who, bytes): - who = who.decode("utf-8", "replace") - acl_identity_observed = bool(str(who or "").strip()) - non_default_identity = ( - acl_identity_observed and str(who).strip().lower() != "default" - ) - except Exception: # ruff: ignore[blind-except] - # Managed providers may deny ACL inspection. Do not broaden permissions - # just for this probe; leave the fact unproven for operator evidence. - pass - - try: - persistence = client.info("persistence") or {} - replication = client.info("replication") or {} - except Exception as exc: - raise RuntimeError("REDIS_INFO_UNAVAILABLE") from exc - aof_enabled = _int(persistence.get("aof_enabled")) == 1 - loading = _int(persistence.get("loading")) == 1 - role = str(replication.get("role") or "").lower() - if role == "master": - replication_observed = _int(replication.get("connected_slaves")) >= 1 - elif role in {"slave", "replica"}: - replication_observed = ( - str(replication.get("master_link_status") or "").lower() == "up" - ) - else: - replication_observed = False - - return { - "schemaVersion": 1, - "plane": plane, - "observedAt": _now(), - "transport": policy.manifest(), - "reachable": True, - "aclIdentityObserved": acl_identity_observed, - "nonDefaultIdentity": non_default_identity, - "aofPersistenceObserved": aof_enabled, - "loading": loading, - "replicationObserved": replication_observed, - "note": ( - "Observations only; least-privilege ACL review, provider durability, backups and restore tests require separate operator evidence." - ), - } - - -def main(argv: list[str] | None = None) -> int: - """Run.""" - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument( - "--plane", - required=True, - choices=("rateLimit", "share", "contribution", "providerArtifactLifecycle"), - ) - parser.add_argument( - "--url-env", - required=True, - help="Environment-variable name containing the Redis URL", - ) - parser.add_argument("--output", type=Path) - args = parser.parse_args(argv) - try: - result = collect(plane=args.plane, url_env=args.url_env) - except RuntimeError as exc: - logger.warning( - json.dumps( - { - "ok": False, - "code": str(exc), - }, - sort_keys=True, - ), - ) - return 2 - payload = json.dumps(result, indent=2, sort_keys=True) + "\n" - if args.output: - args.output.write_text(payload) - else: - sys.stdout.write(payload) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/security/promote_release.py b/security/promote_release.py deleted file mode 100644 index b683227d4f033c0b4d91d037be2fe7ae22452ede..0000000000000000000000000000000000000000 --- a/security/promote_release.py +++ /dev/null @@ -1,1139 +0,0 @@ -""" -Prepare and finalize one immutable, evidence-bound extension release transaction. -""" - -from __future__ import annotations - -import argparse -import hashlib -import json -import os -import re -import shutil -import stat -import subprocess -import sys -import tempfile -import zipfile -from datetime import datetime, timedelta, timezone -from pathlib import Path, PurePosixPath -from typing import Any, Callable - -import tomllib - -HERE = Path(__file__).resolve().parent -ROOT = HERE.parent -EXTENSION_ROOT = HERE.parents[1] -if str(HERE) not in sys.path: - sys.path.insert(0, str(HERE)) - -import verify_release_evidence # noqa: E402 -from source_tree import SourceTreeError, source_tree_sha256 # noqa: E402 - -POLICY = tomllib.loads((HERE / "release_promotion_policy.toml").read_text()) -ARCHIVE_PREFIX = str(POLICY["archive_prefix"]) -PREDICATE_TYPE = str(POLICY["predicate_type"]) -_HEX64 = re.compile(r"^[0-9a-f]{64}$") -_REVISION = re.compile(r"^[0-9a-f]{40}(?:[0-9a-f]{24})?$") -_RELEASE_ID = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{2,127}$") -_IGNORED_DIRS = {".git", ".pytest_cache", "__pycache__"} -_IGNORED_SUFFIXES = {".pyc", ".pyo"} -_ZIP_EPOCH = (1980, 1, 1, 0, 0, 0) -_MAX_ZIP_RATIO = 500 -_CHUNK = 1024 * 1024 - - -class PromotionError(RuntimeError): - """Release transaction violated a fail-closed promotion invariant.""" - - -def _fail(code: str) -> None: - raise PromotionError(code) - - -def _sha256(path: Path) -> str: - h = hashlib.sha256() - with path.open("rb") as handle: - for chunk in iter(lambda: handle.read(_CHUNK), b""): - h.update(chunk) - return h.hexdigest() - - -def _canonical_bytes(value: dict[str, Any]) -> bytes: - return ( - json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False) - + "\n" - ).encode("utf-8") - - -def _write_canonical(path: Path, value: dict[str, Any]) -> None: - path.write_bytes(_canonical_bytes(value)) - - -def _utc(value: datetime | None = None) -> str: - now = (value or datetime.now(timezone.utc)).astimezone(timezone.utc) - return now.isoformat(timespec="seconds").replace("+00:00", "Z") - - -def _parse_time(value: Any, code: str) -> datetime: - if not isinstance(value, str) or not value.endswith("Z"): - _fail(code) - try: - out = datetime.fromisoformat(value[:-1] + "+00:00") - except ValueError: - _fail(code) - return out.astimezone(timezone.utc) - - -def _regular_file(path: Path, *, max_bytes: int, code: str) -> Path: - raw = path.expanduser() - if raw.is_symlink() or not raw.is_file(): - _fail(code) - resolved = raw.resolve() - if resolved.stat().st_size > max_bytes: - _fail(code + "_TOO_LARGE") - return resolved - - -def _outside_source(path: Path, source_root: Path) -> Path: - target = path.expanduser().resolve() - root = source_root.resolve() - if target == root or root in target.parents: - _fail("PROMOTION_OUTPUT_INSIDE_SOURCE_TREE") - return target - - -def _input_outside_source(path: Path, source_root: Path, code: str) -> Path: - resolved = path.resolve() - root = source_root.resolve() - if resolved == root or root in resolved.parents: - _fail(code) - return resolved - - -def _release_files(root: Path) -> list[Path]: - root = root.resolve() - files: list[Path] = [] - total = 0 - for path in root.rglob("*"): - rel = path.relative_to(root) - rendered = rel.as_posix() - if "\\" in rendered or any( - ord(ch) < 32 # ruff: ignore[magic-value-comparison] - or ord(ch) == 127 # ruff: ignore[magic-value-comparison] - or 0x202A <= ord(ch) <= 0x202E # ruff: ignore[magic-value-comparison] - or 0x2066 <= ord(ch) <= 0x2069 # ruff: ignore[magic-value-comparison] - for ch in rendered - ): - _fail("SOURCE_TREE_PATH_UNSAFE") - if any(part in _IGNORED_DIRS for part in rel.parts): - continue - if path.suffix in _IGNORED_SUFFIXES: - continue - if path.is_symlink(): - _fail("SOURCE_TREE_SYMLINK_FORBIDDEN") - if path.is_file(): - total += path.stat().st_size - if total > int(POLICY["max_source_total_bytes"]): - _fail("SOURCE_TREE_TOTAL_TOO_LARGE") - files.append(path) - if len(files) > int(POLICY["max_source_files"]): - _fail("SOURCE_TREE_TOO_MANY_FILES") - return sorted(files, key=lambda p: p.relative_to(root).as_posix()) - - -def _copy_snapshot(source_root: Path, target: Path) -> None: - target.mkdir(parents=True, exist_ok=False) - for source in _release_files(source_root): - rel = source.relative_to(source_root) - dest = target / rel - dest.parent.mkdir(parents=True, exist_ok=True) - with source.open("rb") as src, dest.open("xb") as dst: - shutil.copyfileobj(src, dst, length=_CHUNK) - os.chmod(dest, source.stat().st_mode & 0o777) - if source_tree_sha256(target) != source_tree_sha256(source_root): - _fail("SOURCE_SNAPSHOT_HASH_MISMATCH") - - -def _compare_trees(left: Path, right: Path) -> None: - lfiles = _release_files(left) - rfiles = _release_files(right) - lmap = {p.relative_to(left).as_posix(): p for p in lfiles} - rmap = {p.relative_to(right).as_posix(): p for p in rfiles} - if list(lmap) != list(rmap): - _fail("PATCH_RESULT_TREE_MISMATCH") - for rel, lp in lmap.items(): - rp = rmap[rel] - if (lp.stat().st_mode & 0o777) != (rp.stat().st_mode & 0o777): - _fail("PATCH_RESULT_MODE_MISMATCH") - if lp.stat().st_size != rp.stat().st_size or _sha256(lp) != _sha256(rp): - _fail("PATCH_RESULT_CONTENT_MISMATCH") - - -def _zip_rel(name: str) -> PurePosixPath: - if not isinstance(name, str) or not name or "\\" in name or "\x00" in name: - _fail("BASELINE_ZIP_PATH_INVALID") - rel = PurePosixPath(name) - if rel.is_absolute() or any(part in {"", ".", ".."} for part in rel.parts): - _fail("BASELINE_ZIP_PATH_INVALID") - return rel - - -def _safe_extract_baseline( # ruff: ignore[too-many-branches] - archive: Path, - destination: Path, -) -> Path: - prefix = PurePosixPath(ARCHIVE_PREFIX) - total = 0 - count = 0 - seen: set[str] = set() - destination.mkdir(parents=True, exist_ok=False) - try: - zf = zipfile.ZipFile(archive, "r") - except (OSError, zipfile.BadZipFile) as exc: - raise PromotionError("BASELINE_ZIP_INVALID") from exc - with zf: - for info in zf.infolist(): - rel = _zip_rel(info.filename) - key = rel.as_posix() - if key in seen: - _fail("BASELINE_ZIP_DUPLICATE_PATH") - seen.add(key) - if info.compress_type not in {zipfile.ZIP_STORED, zipfile.ZIP_DEFLATED}: - _fail("BASELINE_ZIP_COMPRESSION_UNSUPPORTED") - if not info.is_dir(): - count += 1 - total += info.file_size - if count > int(POLICY["max_source_files"]): - _fail("BASELINE_ZIP_TOO_MANY_FILES") - if total > int(POLICY["max_source_total_bytes"]): - _fail("BASELINE_ZIP_TOTAL_TOO_LARGE") - if ( - info.compress_size - and info.file_size > info.compress_size * _MAX_ZIP_RATIO - ): - _fail("BASELINE_ZIP_COMPRESSION_RATIO") - mode = (info.external_attr >> 16) & 0xFFFF - ftype = stat.S_IFMT(mode) - if ftype not in {0, stat.S_IFREG, stat.S_IFDIR}: - _fail("BASELINE_ZIP_SPECIAL_FILE") - if info.is_dir() and ftype == stat.S_IFREG: - _fail("BASELINE_ZIP_TYPE_PATH_MISMATCH") - if not info.is_dir() and ftype == stat.S_IFDIR: - _fail("BASELINE_ZIP_TYPE_PATH_MISMATCH") - target = destination.joinpath(*rel.parts) - if info.is_dir(): - target.mkdir(parents=True, exist_ok=True) - continue - target.parent.mkdir(parents=True, exist_ok=True) - try: - with zf.open(info, "r") as src, target.open("xb") as dst: - shutil.copyfileobj(src, dst, length=_CHUNK) - except (OSError, RuntimeError, zipfile.BadZipFile) as exc: - raise PromotionError("BASELINE_ZIP_PAYLOAD_INVALID") from exc - file_mode = mode & 0o777 or 0o644 - os.chmod(target, file_mode) - extension = destination.joinpath(*prefix.parts) - if not extension.is_dir() or extension.is_symlink(): - _fail("BASELINE_EXTENSION_ROOT_MISSING") - return extension - - -def _validate_patch_headers(patch: Path) -> None: - text = patch.read_text(encoding="utf-8", errors="strict") - saw = False - for line in text.splitlines(): - if not line.startswith("diff --git "): - continue - saw = True - parts = line.split(" ") - if len(parts) != 4: # ruff: ignore[magic-value-comparison] - _fail("PATCH_HEADER_INVALID") - for raw, marker in ((parts[2], "a/"), (parts[3], "b/")): - if not raw.startswith(marker): - _fail("PATCH_HEADER_INVALID") - path = raw[2:] - rel = PurePosixPath(path) - if rel.is_absolute() or any(part in {"", ".", ".."} for part in rel.parts): - _fail("PATCH_PATH_INVALID") - if ( - any(part in _IGNORED_DIRS for part in rel.parts) - or rel.suffix in _IGNORED_SUFFIXES - ): - _fail("PATCH_IGNORED_ARTIFACT_FORBIDDEN") - if not saw: - _fail("PATCH_EMPTY_OR_INVALID") - - -def _trusted_git_executable() -> str: - """Resolve the Git binary without trusting the ambient process PATH.""" - override = os.environ.get("SCIKITPLOT_RELEASE_GIT_EXECUTABLE") - if override is not None: - raw = Path(override).expanduser() - if not raw.is_absolute(): - _fail("GIT_EXECUTABLE_PIN_INVALID") - candidate = raw.resolve() - else: - found = shutil.which("git", path=os.defpath) - if not found: - _fail("GIT_REQUIRED") - candidate = Path(found).resolve() - if not candidate.is_file() or not os.access(candidate, os.X_OK): - _fail("GIT_EXECUTABLE_PIN_INVALID" if override is not None else "GIT_REQUIRED") - return str(candidate) - - -def _git_apply_environment(root: Path) -> dict[str, str]: - """Return a secret-free deterministic environment for ``git apply``.""" - home = root / "home" - xdg = root / "xdg" - home.mkdir() - xdg.mkdir() - empty_config = root / "empty.gitconfig" - empty_config.write_text("") - env = { - "HOME": str(home), - "XDG_CONFIG_HOME": str(xdg), - "GIT_CONFIG_NOSYSTEM": "1", - "GIT_CONFIG_GLOBAL": str(empty_config), - "GIT_ATTR_NOSYSTEM": "1", - "GIT_TERMINAL_PROMPT": "0", - "GIT_PAGER": "cat", - "LC_ALL": "C", - "LANG": "C", - } - # Windows process creation needs a small OS bootstrap set; these values are not - # release credentials and do not grant Git configuration authority. - for key in ("SYSTEMROOT", "WINDIR", "COMSPEC", "PATHEXT"): - value = os.environ.get(key) - if value: - env[key] = value - return env - - -def _apply_patch(baseline_extension: Path, patch: Path) -> None: - _validate_patch_headers(patch) - git = _trusted_git_executable() - common = [ - git, - "-c", - "core.autocrlf=false", - "-c", - "core.filemode=true", - "-c", - "core.ignorecase=false", - "-c", - "core.safecrlf=false", - "apply", - "--binary", - "--whitespace=nowarn", - ] - with tempfile.TemporaryDirectory(prefix="scikitplot-release-git-") as scratch: - env = _git_apply_environment(Path(scratch)) - check = subprocess.run( # ruff: ignore[subprocess-without-shell-equals-true] - [*common, "--check", str(patch)], - check=False, - cwd=baseline_extension, - env=env, - capture_output=True, - text=True, - ) - if check.returncode != 0: - _fail("PATCH_APPLY_CHECK_FAILED") - applied = subprocess.run( # ruff: ignore[subprocess-without-shell-equals-true] - [*common, str(patch)], - check=False, - cwd=baseline_extension, - env=env, - capture_output=True, - text=True, - ) - if applied.returncode != 0: - _fail("PATCH_APPLY_FAILED") - - -def _build_deterministic_zip(snapshot: Path, target: Path) -> tuple[str, int, int]: - prefix = PurePosixPath(ARCHIVE_PREFIX) - files = _release_files(snapshot) - with zipfile.ZipFile( - target, "x", compression=zipfile.ZIP_DEFLATED, compresslevel=9, allowZip64=False - ) as zf: - for path in files: - rel = PurePosixPath(path.relative_to(snapshot).as_posix()) - arcname = (prefix / rel).as_posix() - info = zipfile.ZipInfo(arcname, date_time=_ZIP_EPOCH) - info.create_system = 3 - info.compress_type = zipfile.ZIP_DEFLATED - info.external_attr = (stat.S_IFREG | (path.stat().st_mode & 0o777)) << 16 - info.flag_bits |= 0x800 - with path.open("rb") as src, zf.open(info, "w", force_zip64=False) as dst: - shutil.copyfileobj(src, dst, length=_CHUNK) - _verify_release_zip(target, snapshot) - return _sha256(target), target.stat().st_size, len(files) - - -def _verify_release_zip(archive: Path, snapshot: Path) -> None: - prefix = PurePosixPath(ARCHIVE_PREFIX) - expected = {p.relative_to(snapshot).as_posix(): p for p in _release_files(snapshot)} - seen: list[str] = [] - with zipfile.ZipFile(archive, "r") as zf: - bad = zf.testzip() - if bad is not None: - _fail("RELEASE_ZIP_CRC_FAILED") - for info in zf.infolist(): - rel = _zip_rel(info.filename) - try: - local = rel.relative_to(prefix).as_posix() - except ValueError: - _fail("RELEASE_ZIP_PREFIX_INVALID") - if local not in expected or info.is_dir(): - _fail("RELEASE_ZIP_TREE_MISMATCH") - if local in seen: - _fail("RELEASE_ZIP_DUPLICATE_PATH") - seen.append(local) - source = expected[local] - mode = ((info.external_attr >> 16) & 0o777) or 0o644 - if mode != (source.stat().st_mode & 0o777): - _fail("RELEASE_ZIP_MODE_MISMATCH") - h = hashlib.sha256() - with zf.open(info, "r") as handle: - for chunk in iter(lambda: handle.read(_CHUNK), b""): - h.update(chunk) - if h.hexdigest() != _sha256(source): - _fail("RELEASE_ZIP_CONTENT_MISMATCH") - if seen != list(expected): - _fail("RELEASE_ZIP_ORDER_OR_TREE_MISMATCH") - - -def _read_verified_evidence( - evidence: Path, verify_result: dict[str, Any] -) -> dict[str, Any]: - try: - doc = json.loads(evidence.read_text(encoding="utf-8")) - except Exception as exc: - raise PromotionError("EVIDENCE_JSON_INVALID") from exc - if doc["source"]["sourceTreeSha256"] != verify_result["source_tree_sha256"]: - _fail("EVIDENCE_RESULT_REBIND_FAILED") - return doc - - -def _artifact_ref_from_evidence( - evidence: Path, doc: dict[str, Any], name: str -) -> dict[str, Any]: - item = doc["artifacts"][name] - path = evidence.parent / item["path"] - if path.is_symlink() or not path.is_file(): - _fail("SBOM_REFERENCE_FILE_INVALID") - if _sha256(path) != item["sha256"]: - _fail("SBOM_REFERENCE_HASH_MISMATCH") - return { - "sha256": item["sha256"], - "subject": item["subject"], - "name": Path(item["path"]).name, - } - - -def prepare_release( # ruff: ignore[undocumented-public-function] - *, - evidence: Path, - baseline_zip: Path, - patch: Path, - output_dir: Path, - source_root: Path = EXTENSION_ROOT, - evidence_verifier: Callable[..., dict[str, Any]] = verify_release_evidence.verify, - now: datetime | None = None, -) -> dict[str, Any]: - source_root = source_root.resolve() - evidence = _regular_file( - evidence, - max_bytes=int(verify_release_evidence.POLICY["max_manifest_bytes"]), - code="EVIDENCE_FILE_INVALID", - ) - baseline_zip = _regular_file( - baseline_zip, - max_bytes=int(POLICY["max_baseline_zip_bytes"]), - code="BASELINE_ZIP_FILE_INVALID", - ) - patch = _regular_file( - patch, max_bytes=int(POLICY["max_patch_bytes"]), code="PATCH_FILE_INVALID" - ) - evidence = _input_outside_source( - evidence, source_root, "EVIDENCE_INSIDE_SOURCE_TREE" - ) - baseline_zip = _input_outside_source( - baseline_zip, source_root, "BASELINE_ZIP_INSIDE_SOURCE_TREE" - ) - patch = _input_outside_source(patch, source_root, "PATCH_INSIDE_SOURCE_TREE") - target = _outside_source(output_dir, source_root) - if target.exists(): - _fail("PREPARE_OUTPUT_ALREADY_EXISTS") - verified = evidence_verifier(evidence, now=now) - source_sha = source_tree_sha256(source_root) - if verified.get("source_tree_sha256") != source_sha: - _fail("VERIFIED_SOURCE_SUBJECT_MISMATCH") - revision = verified.get("source_revision") - if not isinstance(revision, str) or _REVISION.fullmatch(revision) is None: - _fail("VERIFIED_SOURCE_REVISION_INVALID") - release_id = verified.get("release_id") - if not isinstance(release_id, str) or _RELEASE_ID.fullmatch(release_id) is None: - _fail("VERIFIED_RELEASE_ID_INVALID") - evidence_doc = _read_verified_evidence(evidence, verified) - - target.parent.mkdir(parents=True, exist_ok=True) - with tempfile.TemporaryDirectory( - prefix="release-prepare-", dir=target.parent - ) as temp_raw: - temp = Path(temp_raw) - snapshot = temp / "snapshot" - _copy_snapshot(source_root, snapshot) - if source_tree_sha256(snapshot) != source_sha: - _fail("SOURCE_SNAPSHOT_SUBJECT_MISMATCH") - extracted = temp / "baseline" - baseline_extension = _safe_extract_baseline(baseline_zip, extracted) - _apply_patch(baseline_extension, patch) - _compare_trees(baseline_extension, snapshot) - - stage = temp / "stage" - stage.mkdir() - zip_name = f"{release_id}.zip" - patch_name = f"{release_id}.patch" - zip_path = stage / zip_name - patch_out = stage / patch_name - zip_sha, zip_size, file_count = _build_deterministic_zip(snapshot, zip_path) - shutil.copyfile(patch, patch_out) - os.chmod(patch_out, 0o644) - - # Package determinism is part of the transaction contract. - second = stage / ".determinism-check.zip" - second_sha, _, _ = _build_deterministic_zip(snapshot, second) - if second_sha != zip_sha or second.read_bytes() != zip_path.read_bytes(): - _fail("RELEASE_ZIP_NONDETERMINISTIC") - second.unlink() - - if source_tree_sha256(source_root) != source_sha: - _fail("SOURCE_TREE_CHANGED_DURING_PREPARE") - if source_tree_sha256(snapshot) != source_sha: - _fail("SOURCE_SNAPSHOT_CHANGED_DURING_PREPARE") - - statement = { - "schemaVersion": int(POLICY["statement_schema_version"]), - "predicateType": PREDICATE_TYPE, - "generatedAt": _utc(now), - "release": { - "releaseId": release_id, - "proxyVersion": verified["proxy_version"], - "sourceRevision": revision, - }, - "subject": { - "sourceTreeSha256": source_sha, - "evidenceSha256": verified["evidence_sha256"], - }, - "artifacts": { - "zip": { - "name": zip_name, - "sha256": zip_sha, - "size": zip_size, - "fileCount": file_count, - }, - "patch": { - "name": patch_name, - "sha256": _sha256(patch_out), - "size": patch_out.stat().st_size, - }, - "baselineZip": { - "name": baseline_zip.name, - "sha256": _sha256(baseline_zip), - "size": baseline_zip.stat().st_size, - }, - }, - "sbomReferences": { - "pythonRuntime": { - "name": Path(verify_release_evidence.SUPPLY["sbom_file"]).name, - "sha256": evidence_doc["source"]["pythonSbomSha256"], - }, - "image": _artifact_ref_from_evidence( - evidence, evidence_doc, "imageSbom" - ), - }, - "verification": { - "evidenceVerified": True, - "patchRecreatesSourceTree": True, - "zipRecreatesSourceTree": True, - "deterministicZip": True, - }, - } - statement_path = stage / "release-statement.json" - _write_canonical(statement_path, statement) - summary = { - "ok": True, - "phase": "prepared", - "release_id": release_id, - "source_tree_sha256": source_sha, - "source_revision": revision, - "release_statement_sha256": _sha256(statement_path), - "zip_sha256": zip_sha, - "patch_sha256": _sha256(patch_out), - } - target_tmp = target.with_name(target.name + ".tmp-" + os.urandom(6).hex()) - if target_tmp.exists(): - _fail("PREPARE_TEMP_COLLISION") - shutil.copytree(stage, target_tmp, copy_function=shutil.copy2) - os.replace(target_tmp, target) - return summary - - -def write_signature_verification_record( # ruff: ignore[undocumented-public-function] - *, - statement: Path, - verifier_evidence: Path, - output: Path, - source_revision: str, - verifier_name: str, - verifier_version: str, - verified_at: datetime | None = None, -) -> Path: - statement = _regular_file( - statement, max_bytes=1024 * 1024, code="RELEASE_STATEMENT_INVALID" - ) - verifier_evidence = _regular_file( - verifier_evidence, - max_bytes=1024 * 1024, - code="SIGNATURE_VERIFIER_EVIDENCE_INVALID", - ) - if _REVISION.fullmatch(str(source_revision or "")) is None: - _fail("SOURCE_REVISION_INVALID") - for value, code in ( - (verifier_name, "SIGNATURE_VERIFIER_NAME_INVALID"), - (verifier_version, "SIGNATURE_VERIFIER_VERSION_INVALID"), - ): - if ( - not isinstance(value, str) - or not value.strip() - or len(value) > 128 # ruff: ignore[magic-value-comparison] - or any(ord(ch) < 32 for ch in value) # ruff: ignore[magic-value-comparison] - ): - _fail(code) - target = output.expanduser().resolve() - target.parent.mkdir(parents=True, exist_ok=True) - if target.exists() or target.is_symlink(): - _fail("SIGNATURE_RECORD_OUTPUT_EXISTS") - payload = { - "schemaVersion": int(POLICY["signature_record_schema_version"]), - "verified": True, - "verifiedAt": _utc(verified_at), - "releaseStatementSha256": _sha256(statement), - "sourceRevision": source_revision, - "signerIdentityVerified": True, - "verifierEvidenceSha256": _sha256(verifier_evidence), - "verifier": { - "name": verifier_name.strip(), - "version": verifier_version.strip(), - }, - } - _write_canonical(target, payload) - return target - - -def _safe_artifact_name(value: Any, code: str) -> str: - if not isinstance(value, str) or not value or value in {".", ".."}: - _fail(code) - if "/" in value or "\\" in value or "\x00" in value or Path(value).name != value: - _fail(code) - return value - - -def _statement_file_item( - value: Any, code: str, *, file_count: bool = False -) -> dict[str, Any]: - expected = {"name", "sha256", "size"} | ({"fileCount"} if file_count else set()) - if not isinstance(value, dict) or set(value) != expected: - _fail(code + "_SCHEMA_INVALID") - _safe_artifact_name(value.get("name"), code + "_NAME_INVALID") - if ( - not isinstance(value.get("sha256"), str) - or _HEX64.fullmatch(value["sha256"]) is None - ): - _fail(code + "_SHA256_INVALID") - if ( - not isinstance(value.get("size"), int) - or isinstance(value.get("size"), bool) - or value["size"] < 0 - ): - _fail(code + "_SIZE_INVALID") - if file_count and ( - not isinstance(value.get("fileCount"), int) - or isinstance(value.get("fileCount"), bool) - or value["fileCount"] < 1 - ): - _fail(code + "_FILE_COUNT_INVALID") - return value - - -def _validate_statement( # ruff: ignore[too-many-branches] - statement_path: Path, - source_root: Path, -) -> dict[str, Any]: - if statement_path.is_symlink() or not statement_path.is_file(): - _fail("RELEASE_STATEMENT_FILE_INVALID") - try: - doc = json.loads(statement_path.read_text(encoding="utf-8")) - except Exception as exc: - raise PromotionError("RELEASE_STATEMENT_JSON_INVALID") from exc - expected_root = { - "schemaVersion", - "predicateType", - "generatedAt", - "release", - "subject", - "artifacts", - "sbomReferences", - "verification", - } - if not isinstance(doc, dict) or set(doc) != expected_root: - _fail("RELEASE_STATEMENT_SCHEMA_INVALID") - if ( - doc.get("schemaVersion") != int(POLICY["statement_schema_version"]) - or doc.get("predicateType") != PREDICATE_TYPE - ): - _fail("RELEASE_STATEMENT_VERSION_INVALID") - _parse_time(doc.get("generatedAt"), "RELEASE_STATEMENT_TIME_INVALID") - release = doc.get("release") - if not isinstance(release, dict) or set(release) != { - "releaseId", - "proxyVersion", - "sourceRevision", - }: - _fail("RELEASE_STATEMENT_RELEASE_SCHEMA_INVALID") - if ( - not isinstance(release.get("releaseId"), str) - or _RELEASE_ID.fullmatch(release["releaseId"]) is None - ): - _fail("RELEASE_STATEMENT_RELEASE_ID_INVALID") - if not isinstance(release.get("proxyVersion"), str) or not release["proxyVersion"]: - _fail("RELEASE_STATEMENT_PROXY_VERSION_INVALID") - if ( - not isinstance(release.get("sourceRevision"), str) - or _REVISION.fullmatch(release["sourceRevision"]) is None - ): - _fail("RELEASE_STATEMENT_REVISION_INVALID") - subject = doc.get("subject") - if not isinstance(subject, dict) or set(subject) != { - "sourceTreeSha256", - "evidenceSha256", - }: - _fail("RELEASE_STATEMENT_SUBJECT_SCHEMA_INVALID") - for key in ("sourceTreeSha256", "evidenceSha256"): - if ( - not isinstance(subject.get(key), str) - or _HEX64.fullmatch(subject[key]) is None - ): - _fail("RELEASE_STATEMENT_SUBJECT_HASH_INVALID") - if subject["sourceTreeSha256"] != source_tree_sha256(source_root): - _fail("RELEASE_STATEMENT_SOURCE_MISMATCH") - artifacts = doc.get("artifacts") - if not isinstance(artifacts, dict) or set(artifacts) != { - "zip", - "patch", - "baselineZip", - }: - _fail("RELEASE_STATEMENT_ARTIFACTS_SCHEMA_INVALID") - zip_item = _statement_file_item( - artifacts.get("zip"), "RELEASE_STATEMENT_ZIP", file_count=True - ) - patch_item = _statement_file_item(artifacts.get("patch"), "RELEASE_STATEMENT_PATCH") - baseline_item = _statement_file_item( - artifacts.get("baselineZip"), "RELEASE_STATEMENT_BASELINE" - ) - if ( - not zip_item["name"].endswith(".zip") - or not baseline_item["name"].endswith(".zip") - or not patch_item["name"].endswith(".patch") - ): - _fail("RELEASE_STATEMENT_ARTIFACT_EXTENSION_INVALID") - if ( - len( - { - zip_item["name"], - patch_item["name"], - baseline_item["name"], - } - ) - != 3 # ruff: ignore[magic-value-comparison] - ): # ruff: ignore[magic-value-comparison] - _fail("RELEASE_STATEMENT_ARTIFACT_NAME_COLLISION") - sboms = doc.get("sbomReferences") - if not isinstance(sboms, dict) or set(sboms) != {"pythonRuntime", "image"}: - _fail("RELEASE_STATEMENT_SBOM_SCHEMA_INVALID") - python_sbom = sboms.get("pythonRuntime") - if not isinstance(python_sbom, dict) or set(python_sbom) != {"name", "sha256"}: - _fail("RELEASE_STATEMENT_PYTHON_SBOM_SCHEMA_INVALID") - _safe_artifact_name( - python_sbom.get("name"), "RELEASE_STATEMENT_PYTHON_SBOM_NAME_INVALID" - ) - if ( - not isinstance(python_sbom.get("sha256"), str) - or _HEX64.fullmatch(python_sbom["sha256"]) is None - ): - _fail("RELEASE_STATEMENT_PYTHON_SBOM_HASH_INVALID") - image_sbom = sboms.get("image") - if not isinstance(image_sbom, dict) or set(image_sbom) != { - "name", - "sha256", - "subject", - }: - _fail("RELEASE_STATEMENT_IMAGE_SBOM_SCHEMA_INVALID") - _safe_artifact_name( - image_sbom.get("name"), "RELEASE_STATEMENT_IMAGE_SBOM_NAME_INVALID" - ) - if ( - not isinstance(image_sbom.get("sha256"), str) - or _HEX64.fullmatch(image_sbom["sha256"]) is None - ): - _fail("RELEASE_STATEMENT_IMAGE_SBOM_HASH_INVALID") - if not isinstance(image_sbom.get("subject"), str) or not image_sbom[ - "subject" - ].startswith("sha256:"): - _fail("RELEASE_STATEMENT_IMAGE_SBOM_SUBJECT_INVALID") - verification = doc.get("verification") - expected_verification = { - "evidenceVerified": True, - "patchRecreatesSourceTree": True, - "zipRecreatesSourceTree": True, - "deterministicZip": True, - } - if verification != expected_verification: - _fail("RELEASE_STATEMENT_VERIFICATION_INVALID") - return doc - - -def _validate_signature_record( # ruff: ignore[too-many-branches] - record_path: Path, - *, - statement_sha: str, - revision: str, - verifier_evidence: Path, - statement_generated: datetime, - now: datetime | None, -) -> str: - try: - doc = json.loads(record_path.read_text(encoding="utf-8")) - except Exception as exc: - raise PromotionError("SIGNATURE_RECORD_JSON_INVALID") from exc - if not isinstance(doc, dict) or set(doc) != { - "schemaVersion", - "verified", - "verifiedAt", - "releaseStatementSha256", - "sourceRevision", - "signerIdentityVerified", - "verifierEvidenceSha256", - "verifier", - }: - _fail("SIGNATURE_RECORD_SCHEMA_INVALID") - if doc.get("schemaVersion") != int(POLICY["signature_record_schema_version"]): - _fail("SIGNATURE_RECORD_VERSION_INVALID") - if doc.get("verified") is not True or doc.get("signerIdentityVerified") is not True: - _fail("SIGNATURE_RECORD_NOT_VERIFIED") - if doc.get("releaseStatementSha256") != statement_sha: - _fail("SIGNATURE_RECORD_SUBJECT_MISMATCH") - if doc.get("sourceRevision") != revision: - _fail("SIGNATURE_RECORD_REVISION_MISMATCH") - if doc.get("verifierEvidenceSha256") != _sha256(verifier_evidence): - _fail("SIGNATURE_VERIFIER_EVIDENCE_HASH_MISMATCH") - verifier = doc.get("verifier") - if not isinstance(verifier, dict) or set(verifier) != {"name", "version"}: - _fail("SIGNATURE_VERIFIER_SCHEMA_INVALID") - for field in ("name", "version"): - value = verifier.get(field) - if ( - not isinstance(value, str) - or not value.strip() - or len(value) > 128 # ruff: ignore[magic-value-comparison] - or any(ord(ch) < 32 for ch in value) # ruff: ignore[magic-value-comparison] - ): - _fail("SIGNATURE_VERIFIER_FIELD_INVALID") - verified_at = _parse_time(doc.get("verifiedAt"), "SIGNATURE_RECORD_TIME_INVALID") - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - if verified_at > current + timedelta(minutes=10): - _fail("SIGNATURE_RECORD_FROM_FUTURE") - if current - verified_at > timedelta(hours=int(POLICY["max_signature_age_hours"])): - _fail("SIGNATURE_RECORD_STALE") - if verified_at < statement_generated - timedelta(minutes=10): - _fail("SIGNATURE_RECORD_PREDATES_STATEMENT") - return _sha256(record_path) - - -def finalize_release( # ruff: ignore[too-many-branches, undocumented-public-function] - *, - evidence: Path, - prepared_dir: Path, - baseline_zip: Path, - signature_record: Path, - signature_verifier_evidence: Path, - promotion_dir: Path, - source_root: Path = EXTENSION_ROOT, - evidence_verifier: Callable[..., dict[str, Any]] = verify_release_evidence.verify, - now: datetime | None = None, -) -> dict[str, Any]: - source_root = source_root.resolve() - prepared = _outside_source(prepared_dir, source_root) - if not prepared.is_dir() or prepared.is_symlink(): - _fail("PREPARED_DIR_INVALID") - target = _outside_source(promotion_dir, source_root) - if target.exists(): - _fail("PROMOTION_OUTPUT_ALREADY_EXISTS") - evidence = _regular_file( - evidence, - max_bytes=int(verify_release_evidence.POLICY["max_manifest_bytes"]), - code="EVIDENCE_FILE_INVALID", - ) - baseline_zip = _regular_file( - baseline_zip, - max_bytes=int(POLICY["max_baseline_zip_bytes"]), - code="BASELINE_ZIP_FILE_INVALID", - ) - signature_record = _regular_file( - signature_record, max_bytes=1024 * 1024, code="SIGNATURE_RECORD_INVALID" - ) - signature_verifier_evidence = _regular_file( - signature_verifier_evidence, - max_bytes=1024 * 1024, - code="SIGNATURE_VERIFIER_EVIDENCE_INVALID", - ) - evidence = _input_outside_source( - evidence, source_root, "EVIDENCE_INSIDE_SOURCE_TREE" - ) - baseline_zip = _input_outside_source( - baseline_zip, source_root, "BASELINE_ZIP_INSIDE_SOURCE_TREE" - ) - signature_record = _input_outside_source( - signature_record, source_root, "SIGNATURE_RECORD_INSIDE_SOURCE_TREE" - ) - signature_verifier_evidence = _input_outside_source( - signature_verifier_evidence, - source_root, - "SIGNATURE_VERIFIER_EVIDENCE_INSIDE_SOURCE_TREE", - ) - - verified = evidence_verifier(evidence, now=now) - statement_path = prepared / "release-statement.json" - statement = _validate_statement(statement_path, source_root) - statement_sha = _sha256(statement_path) - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - statement_generated = _parse_time( - statement["generatedAt"], "RELEASE_STATEMENT_TIME_INVALID" - ) - if statement_generated > current + timedelta(minutes=10): - _fail("RELEASE_STATEMENT_FROM_FUTURE") - if current - statement_generated > timedelta( - hours=int(POLICY["max_signature_age_hours"]) - ): - _fail("RELEASE_STATEMENT_STALE") - if statement["subject"].get("evidenceSha256") != verified["evidence_sha256"]: - _fail("RELEASE_STATEMENT_EVIDENCE_MISMATCH") - if ( - statement["release"].get("releaseId") != verified["release_id"] - or statement["release"].get("sourceRevision") != verified["source_revision"] - ): - _fail("RELEASE_STATEMENT_RELEASE_MISMATCH") - revision = verified["source_revision"] - signature_sha = _validate_signature_record( - signature_record, - statement_sha=statement_sha, - revision=revision, - verifier_evidence=signature_verifier_evidence, - statement_generated=statement_generated, - now=now, - ) - if ( - statement["artifacts"]["baselineZip"]["name"] != baseline_zip.name - or statement["artifacts"]["baselineZip"]["sha256"] != _sha256(baseline_zip) - or statement["artifacts"]["baselineZip"]["size"] != baseline_zip.stat().st_size - ): - _fail("PROMOTION_BASELINE_HASH_MISMATCH") - - evidence_doc = _read_verified_evidence(evidence, verified) - expected_python_sbom = { - "name": Path(verify_release_evidence.SUPPLY["sbom_file"]).name, - "sha256": evidence_doc["source"]["pythonSbomSha256"], - } - expected_image_sbom = _artifact_ref_from_evidence( - evidence, evidence_doc, "imageSbom" - ) - if ( - statement["sbomReferences"]["pythonRuntime"] != expected_python_sbom - or statement["sbomReferences"]["image"] != expected_image_sbom - ): - _fail("PROMOTION_SBOM_REFERENCE_MISMATCH") - - zip_item = statement["artifacts"]["zip"] - patch_item = statement["artifacts"]["patch"] - zip_path = prepared / zip_item["name"] - patch_path = prepared / patch_item["name"] - for path, item, code in ( - (zip_path, zip_item, "ZIP"), - (patch_path, patch_item, "PATCH"), - ): - if path.is_symlink() or not path.is_file(): - _fail(f"PROMOTION_{code}_MISSING") - if _sha256(path) != item.get("sha256") or path.stat().st_size != item.get( - "size" - ): - _fail(f"PROMOTION_{code}_HASH_MISMATCH") - _verify_release_zip(zip_path, source_root) - if zip_item["fileCount"] != len(_release_files(source_root)): - _fail("PROMOTION_ZIP_FILE_COUNT_MISMATCH") - with tempfile.TemporaryDirectory(prefix="release-finalize-patch-") as temp_raw: - baseline_extension = _safe_extract_baseline( - baseline_zip, Path(temp_raw) / "baseline" - ) - _apply_patch(baseline_extension, patch_path) - _compare_trees(baseline_extension, source_root) - if source_tree_sha256(source_root) != verified["source_tree_sha256"]: - _fail("SOURCE_TREE_CHANGED_BEFORE_FINALIZE") - - receipt = { - "schemaVersion": int(POLICY["receipt_schema_version"]), - "finalizedAt": _utc(now), - "releaseId": verified["release_id"], - "sourceRevision": revision, - "sourceTreeSha256": verified["source_tree_sha256"], - "evidenceSha256": verified["evidence_sha256"], - "releaseStatementSha256": statement_sha, - "signatureVerificationSha256": signature_sha, - "signatureVerifierEvidenceSha256": _sha256(signature_verifier_evidence), - "publish": [ - { - "name": zip_item["name"], - "sha256": zip_item["sha256"], - "size": zip_item["size"], - }, - { - "name": patch_item["name"], - "sha256": patch_item["sha256"], - "size": patch_item["size"], - }, - { - "name": "release-statement.json", - "sha256": statement_sha, - "size": statement_path.stat().st_size, - }, - { - "name": "release-statement.signature-verification.json", - "sha256": signature_sha, - "size": signature_record.stat().st_size, - }, - ], - "sbomReferences": statement["sbomReferences"], - "status": "promoted", - } - - target.parent.mkdir(parents=True, exist_ok=True) - with tempfile.TemporaryDirectory( - prefix="release-finalize-", dir=target.parent - ) as temp_raw: - stage = Path(temp_raw) / "promoted" - stage.mkdir() - shutil.copy2(zip_path, stage / zip_path.name) - shutil.copy2(patch_path, stage / patch_path.name) - shutil.copy2(statement_path, stage / "release-statement.json") - shutil.copy2( - signature_record, stage / "release-statement.signature-verification.json" - ) - receipt_path = stage / "promotion-receipt.json" - _write_canonical(receipt_path, receipt) - # Rebind every publishable byte immediately before the atomic directory move. - for item in receipt["publish"]: - path = stage / item["name"] - if _sha256(path) != item["sha256"] or path.stat().st_size != item["size"]: - _fail("PROMOTION_STAGE_REBIND_FAILED") - if source_tree_sha256(source_root) != verified["source_tree_sha256"]: - _fail("SOURCE_TREE_CHANGED_DURING_FINALIZE") - final_tmp = target.with_name(target.name + ".tmp-" + os.urandom(6).hex()) - shutil.copytree(stage, final_tmp, copy_function=shutil.copy2) - os.replace(final_tmp, target) - return { - "ok": True, - "phase": "promoted", - "release_id": verified["release_id"], - "source_tree_sha256": verified["source_tree_sha256"], - "release_statement_sha256": statement_sha, - "promotion_receipt_sha256": _sha256(target / "promotion-receipt.json"), - } - - -def main( # ruff: ignore[undocumented-public-function] - argv: list[str] | None = None, -) -> int: - parser = argparse.ArgumentParser(description=__doc__) - sub = parser.add_subparsers(dest="command", required=True) - prepare = sub.add_parser("prepare") - prepare.add_argument("--evidence", type=Path, required=True) - prepare.add_argument("--baseline-zip", type=Path, required=True) - prepare.add_argument("--patch", type=Path, required=True) - prepare.add_argument("--output-dir", type=Path, required=True) - sig = sub.add_parser("signature-record") - sig.add_argument("--statement", type=Path, required=True) - sig.add_argument("--verifier-evidence", type=Path, required=True) - sig.add_argument("--output", type=Path, required=True) - sig.add_argument("--source-revision", required=True) - sig.add_argument("--verifier-name", required=True) - sig.add_argument("--verifier-version", required=True) - finalize = sub.add_parser("finalize") - finalize.add_argument("--evidence", type=Path, required=True) - finalize.add_argument("--prepared-dir", type=Path, required=True) - finalize.add_argument("--baseline-zip", type=Path, required=True) - finalize.add_argument("--signature-record", type=Path, required=True) - finalize.add_argument("--signature-verifier-evidence", type=Path, required=True) - finalize.add_argument("--promotion-dir", type=Path, required=True) - args = parser.parse_args(argv) - try: - if args.command == "prepare": - result = prepare_release( - evidence=args.evidence, - baseline_zip=args.baseline_zip, - patch=args.patch, - output_dir=args.output_dir, - ) - elif args.command == "signature-record": - path = write_signature_verification_record( - statement=args.statement, - verifier_evidence=args.verifier_evidence, - output=args.output, - source_revision=args.source_revision, - verifier_name=args.verifier_name, - verifier_version=args.verifier_version, - ) - result = { - "ok": True, - "phase": "signature-record", - "path": path.name, - "sha256": _sha256(path), - } - else: - result = finalize_release( - evidence=args.evidence, - prepared_dir=args.prepared_dir, - baseline_zip=args.baseline_zip, - signature_record=args.signature_record, - signature_verifier_evidence=args.signature_verifier_evidence, - promotion_dir=args.promotion_dir, - ) - except ( - PromotionError, - verify_release_evidence.EvidenceError, - SourceTreeError, - OSError, - UnicodeError, - zipfile.BadZipFile, - ) as exc: - sys.stderr.write( - json.dumps({"ok": False, "code": str(exc)}, sort_keys=True) + "\n" - ) - return 2 - sys.stdout.write(json.dumps(result, sort_keys=True) + "\n") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/security/publish_release.py b/security/publish_release.py deleted file mode 100644 index 1a29f423dfeb3a359ab64ac6dfbf8096ae19b3c6..0000000000000000000000000000000000000000 --- a/security/publish_release.py +++ /dev/null @@ -1,901 +0,0 @@ -""" -Publish only Run 149 promotion-receipt objects and emit bounded transparency evidence. -""" - -from __future__ import annotations - -import argparse -import hashlib -import json -import logging -import os -import re -import shutil -import subprocess -import sys # ruff: ignore[unused-import] -import tempfile -import threading -from datetime import datetime, timedelta, timezone -from pathlib import Path -from typing import Any, Callable, Iterable - -import tomllib - -HERE = Path(__file__).resolve().parent -POLICY = tomllib.loads((HERE / "release_publication_policy.toml").read_text()) -PREDICATE_TYPE = str(POLICY["predicate_type"]) -PROMOTION_PREDICATE_TYPE = str(POLICY["promotion_predicate_type"]) -_HEX64 = re.compile(r"^[0-9a-f]{64}$") -_REVISION = re.compile(r"^[0-9a-f]{40}(?:[0-9a-f]{24})?$") -_RELEASE_ID = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{2,127}$") -_ID = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:/-]{0,255}$") -_CHUNK = 1024 * 1024 -logger = logging.getLogger(__name__) - - -class PublicationError(RuntimeError): - """Publication transaction violated a fail-closed invariant.""" - - -def _fail(code: str) -> None: - raise PublicationError(code) - - -def _sha256(path: Path) -> str: - h = hashlib.sha256() - with path.open("rb") as handle: - for chunk in iter(lambda: handle.read(_CHUNK), b""): - h.update(chunk) - return h.hexdigest() - - -def _canonical_bytes(value: dict[str, Any]) -> bytes: - return ( - json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False) - + "\n" - ).encode("utf-8") - - -def _write_canonical(path: Path, value: dict[str, Any]) -> None: - path.write_bytes(_canonical_bytes(value)) - - -def _loads_json(raw: bytes, code: str) -> dict[str, Any]: - def object_pairs(pairs): - out: dict[str, Any] = {} - for key, value in pairs: - if key in out: - _fail(code + "_DUPLICATE_KEY") - out[key] = value - return out - - try: - value = json.loads(raw.decode("utf-8"), object_pairs_hook=object_pairs) - except PublicationError: - raise - except Exception as exc: - raise PublicationError(code + "_JSON_INVALID") from exc - if not isinstance(value, dict): - _fail(code + "_SCHEMA_INVALID") - return value - - -def _utc(value: datetime | None = None) -> str: - now = (value or datetime.now(timezone.utc)).astimezone(timezone.utc) - return now.isoformat(timespec="seconds").replace("+00:00", "Z") - - -def _parse_time(value: Any, code: str) -> datetime: - if not isinstance(value, str) or not value.endswith("Z"): - _fail(code) - try: - out = datetime.fromisoformat(value[:-1] + "+00:00") - except ValueError: - _fail(code) - return out.astimezone(timezone.utc) - - -def _safe_name(value: Any, code: str) -> str: - if not isinstance(value, str) or not value or value in {".", ".."}: - _fail(code) - if Path(value).name != value or "/" in value or "\\" in value or "\x00" in value: - _fail(code) - if any( - ord(ch) < 32 # ruff: ignore[magic-value-comparison] - or ord(ch) == 127 # ruff: ignore[magic-value-comparison] - for ch in value - ): - _fail(code) - return value - - -def _safe_id(value: Any, code: str) -> str: - if not isinstance(value, str) or _ID.fullmatch(value) is None or ".." in value: - _fail(code) - return value - - -def _hex(value: Any, code: str) -> str: - if not isinstance(value, str) or _HEX64.fullmatch(value) is None: - _fail(code) - return value - - -def _positive_size(value: Any, code: str) -> int: - if not isinstance(value, int) or isinstance(value, bool) or value < 0: - _fail(code) - return value - - -def _regular_file(path: Path, *, max_bytes: int, code: str) -> Path: - raw = path.expanduser() - if raw.is_symlink() or not raw.is_file(): - _fail(code) - resolved = raw.resolve() - if resolved.stat().st_size > max_bytes: - _fail(code + "_TOO_LARGE") - return resolved - - -def _outside(path: Path, protected: Path, code: str) -> Path: - target = path.expanduser().resolve() - root = protected.resolve() - if target == root or root in target.parents: - _fail(code) - return target - - -def _read_json_exact( - path: Path, *, max_bytes: int, code: str, canonical: bool = False -) -> tuple[dict[str, Any], bytes]: - path = _regular_file(path, max_bytes=max_bytes, code=code) - raw = path.read_bytes() - doc = _loads_json(raw, code) - if canonical and raw != _canonical_bytes(doc): - _fail(code + "_NOT_CANONICAL") - return doc, raw - - -def _publish_item(value: Any) -> dict[str, Any]: - if not isinstance(value, dict) or set(value) != {"name", "sha256", "size"}: - _fail("PROMOTION_PUBLISH_ITEM_SCHEMA_INVALID") - return { - "name": _safe_name(value.get("name"), "PROMOTION_PUBLISH_NAME_INVALID"), - "sha256": _hex(value.get("sha256"), "PROMOTION_PUBLISH_SHA256_INVALID"), - "size": _positive_size(value.get("size"), "PROMOTION_PUBLISH_SIZE_INVALID"), - } - - -def _validate_promotion_bindings( # ruff: ignore[too-many-branches] - root: Path, receipt: dict[str, Any], items: list[dict[str, Any]] -) -> None: - statement_path = root / "release-statement.json" - signature_path = root / "release-statement.signature-verification.json" - statement, statement_raw = _read_json_exact( - statement_path, - max_bytes=1024 * 1024, - code="PROMOTION_STATEMENT", - canonical=True, - ) - signature, signature_raw = _read_json_exact( - signature_path, - max_bytes=1024 * 1024, - code="PROMOTION_SIGNATURE", - canonical=True, - ) - if hashlib.sha256(statement_raw).hexdigest() != receipt["releaseStatementSha256"]: - _fail("PROMOTION_STATEMENT_HASH_REBIND_FAILED") - if ( - hashlib.sha256(signature_raw).hexdigest() - != receipt["signatureVerificationSha256"] - ): - _fail("PROMOTION_SIGNATURE_HASH_REBIND_FAILED") - - expected_statement = { - "schemaVersion", - "predicateType", - "generatedAt", - "release", - "subject", - "artifacts", - "sbomReferences", - "verification", - } - if ( - set(statement) != expected_statement - or statement.get("schemaVersion") != 1 - or statement.get("predicateType") != PROMOTION_PREDICATE_TYPE - ): - _fail("PROMOTION_STATEMENT_SCHEMA_INVALID") - _parse_time(statement.get("generatedAt"), "PROMOTION_STATEMENT_TIME_INVALID") - release = statement.get("release") - if not isinstance(release, dict) or set(release) != { - "releaseId", - "proxyVersion", - "sourceRevision", - }: - _fail("PROMOTION_STATEMENT_RELEASE_SCHEMA_INVALID") - if ( - release.get("releaseId") != receipt["releaseId"] - or release.get("sourceRevision") != receipt["sourceRevision"] - ): - _fail("PROMOTION_STATEMENT_RELEASE_REBIND_FAILED") - if ( - not isinstance(release.get("proxyVersion"), str) - or not release["proxyVersion"] - or len(release["proxyVersion"]) > 64 # ruff: ignore[magic-value-comparison] - ): - _fail("PROMOTION_STATEMENT_PROXY_VERSION_INVALID") - subject = statement.get("subject") - if subject != { - "sourceTreeSha256": receipt["sourceTreeSha256"], - "evidenceSha256": receipt["evidenceSha256"], - }: - _fail("PROMOTION_STATEMENT_SUBJECT_REBIND_FAILED") - if statement.get("sbomReferences") != receipt["sbomReferences"]: - _fail("PROMOTION_STATEMENT_SBOM_REBIND_FAILED") - if statement.get("verification") != { - "evidenceVerified": True, - "patchRecreatesSourceTree": True, - "zipRecreatesSourceTree": True, - "deterministicZip": True, - }: - _fail("PROMOTION_STATEMENT_VERIFICATION_INVALID") - artifacts = statement.get("artifacts") - if not isinstance(artifacts, dict) or set(artifacts) != { - "zip", - "patch", - "baselineZip", - }: - _fail("PROMOTION_STATEMENT_ARTIFACTS_SCHEMA_INVALID") - by_name = {item["name"]: item for item in items} - for field, suffix in (("zip", ".zip"), ("patch", ".patch")): - value = artifacts.get(field) - expected_keys = ( - {"name", "sha256", "size", "fileCount"} - if field == "zip" - else {"name", "sha256", "size"} - ) - if not isinstance(value, dict) or set(value) != expected_keys: - _fail("PROMOTION_STATEMENT_ARTIFACT_SCHEMA_INVALID") - name = _safe_name( - value.get("name"), "PROMOTION_STATEMENT_ARTIFACT_NAME_INVALID" - ) - if not name.endswith(suffix) or name not in by_name: - _fail("PROMOTION_STATEMENT_ARTIFACT_REBIND_FAILED") - if ( - value.get("sha256") != by_name[name]["sha256"] - or value.get("size") != by_name[name]["size"] - ): - _fail("PROMOTION_STATEMENT_ARTIFACT_REBIND_FAILED") - if field == "zip" and ( - not isinstance(value.get("fileCount"), int) - or isinstance(value.get("fileCount"), bool) - or value["fileCount"] < 1 - ): - _fail("PROMOTION_STATEMENT_ZIP_FILE_COUNT_INVALID") - baseline = artifacts.get("baselineZip") - if not isinstance(baseline, dict) or set(baseline) != {"name", "sha256", "size"}: - _fail("PROMOTION_STATEMENT_BASELINE_SCHEMA_INVALID") - _safe_name(baseline.get("name"), "PROMOTION_STATEMENT_BASELINE_NAME_INVALID") - _hex(baseline.get("sha256"), "PROMOTION_STATEMENT_BASELINE_SHA_INVALID") - _positive_size(baseline.get("size"), "PROMOTION_STATEMENT_BASELINE_SIZE_INVALID") - - expected_signature = { - "schemaVersion", - "verified", - "verifiedAt", - "releaseStatementSha256", - "sourceRevision", - "signerIdentityVerified", - "verifierEvidenceSha256", - "verifier", - } - if set(signature) != expected_signature or signature.get("schemaVersion") != 1: - _fail("PROMOTION_SIGNATURE_SCHEMA_INVALID") - if ( - signature.get("verified") is not True - or signature.get("signerIdentityVerified") is not True - ): - _fail("PROMOTION_SIGNATURE_NOT_VERIFIED") - if ( - signature.get("releaseStatementSha256") != receipt["releaseStatementSha256"] - or signature.get("sourceRevision") != receipt["sourceRevision"] - ): - _fail("PROMOTION_SIGNATURE_SUBJECT_REBIND_FAILED") - if ( - signature.get("verifierEvidenceSha256") - != receipt["signatureVerifierEvidenceSha256"] - ): - _fail("PROMOTION_SIGNATURE_VERIFIER_REBIND_FAILED") - _parse_time(signature.get("verifiedAt"), "PROMOTION_SIGNATURE_TIME_INVALID") - verifier = signature.get("verifier") - if not isinstance(verifier, dict) or set(verifier) != {"name", "version"}: - _fail("PROMOTION_SIGNATURE_VERIFIER_SCHEMA_INVALID") - for field in ("name", "version"): - value = verifier.get(field) - if ( - not isinstance(value, str) - or not value.strip() - or len(value) > 128 # ruff: ignore[magic-value-comparison] - or any(ord(ch) < 32 for ch in value) # ruff: ignore[magic-value-comparison] - ): - _fail("PROMOTION_SIGNATURE_VERIFIER_FIELD_INVALID") - - -def _validate_promotion( # ruff: ignore[too-many-branches] - promotion_dir: Path, -) -> tuple[dict[str, Any], str, list[dict[str, Any]]]: - root = promotion_dir.expanduser().resolve() - if root.is_symlink() or not root.is_dir(): - _fail("PROMOTION_DIRECTORY_INVALID") - receipt_path = root / "promotion-receipt.json" - doc, raw = _read_json_exact( - receipt_path, - max_bytes=int(POLICY["max_promotion_receipt_bytes"]), - code="PROMOTION_RECEIPT", - canonical=True, - ) - expected = { - "schemaVersion", - "finalizedAt", - "releaseId", - "sourceRevision", - "sourceTreeSha256", - "evidenceSha256", - "releaseStatementSha256", - "signatureVerificationSha256", - "signatureVerifierEvidenceSha256", - "publish", - "sbomReferences", - "status", - } - if ( - set(doc) != expected - or doc.get("schemaVersion") != 1 - or doc.get("status") != "promoted" - ): - _fail("PROMOTION_RECEIPT_SCHEMA_INVALID") - if ( - not isinstance(doc.get("releaseId"), str) - or _RELEASE_ID.fullmatch(doc["releaseId"]) is None - ): - _fail("PROMOTION_RELEASE_ID_INVALID") - if ( - not isinstance(doc.get("sourceRevision"), str) - or _REVISION.fullmatch(doc["sourceRevision"]) is None - ): - _fail("PROMOTION_SOURCE_REVISION_INVALID") - for key in ( - "sourceTreeSha256", - "evidenceSha256", - "releaseStatementSha256", - "signatureVerificationSha256", - "signatureVerifierEvidenceSha256", - ): - _hex(doc.get(key), "PROMOTION_RECEIPT_HASH_INVALID") - _parse_time(doc.get("finalizedAt"), "PROMOTION_FINALIZED_TIME_INVALID") - if not isinstance(doc.get("sbomReferences"), dict): - _fail("PROMOTION_SBOM_REFERENCES_INVALID") - values = doc.get("publish") - if ( - not isinstance(values, list) - or not values - or len(values) > int(POLICY["max_publish_artifacts"]) - ): - _fail("PROMOTION_PUBLISH_LIST_INVALID") - items = [_publish_item(item) for item in values] - names = [item["name"] for item in items] - if len(names) != len(set(names)): - _fail("PROMOTION_PUBLISH_DUPLICATE_NAME") - if ( - "release-statement.json" not in names - or "release-statement.signature-verification.json" not in names - ): - _fail("PROMOTION_REQUIRED_STATEMENT_MISSING") - if ( - sum(name.endswith(".zip") for name in names) != 1 - or sum(name.endswith(".patch") for name in names) != 1 - ): - _fail("PROMOTION_PRIMARY_ARTIFACT_SET_INVALID") - total = 0 - for item in items: - if item["size"] > int(POLICY["max_artifact_bytes"]): - _fail("PROMOTION_ARTIFACT_TOO_LARGE") - total += item["size"] - if total > int(POLICY["max_total_publish_bytes"]): - _fail("PROMOTION_TOTAL_TOO_LARGE") - path = root / item["name"] - if path.is_symlink() or not path.is_file(): - _fail("PROMOTION_ARTIFACT_MISSING") - if path.stat().st_size != item["size"] or _sha256(path) != item["sha256"]: - _fail("PROMOTION_ARTIFACT_REBIND_FAILED") - by_name = {item["name"]: item for item in items} - if by_name["release-statement.json"]["sha256"] != doc["releaseStatementSha256"]: - _fail("PROMOTION_STATEMENT_HASH_REBIND_FAILED") - if ( - by_name["release-statement.signature-verification.json"]["sha256"] - != doc["signatureVerificationSha256"] - ): - _fail("PROMOTION_SIGNATURE_HASH_REBIND_FAILED") - _validate_promotion_bindings(root, doc, items) - actual = sorted(p.name for p in root.iterdir() if not p.is_dir()) - expected_files = sorted([*names, "promotion-receipt.json"]) - if actual != expected_files or any( - p.is_symlink() or p.is_dir() for p in root.iterdir() - ): - _fail("PROMOTION_DIRECTORY_CONTAINS_UNAUTHORIZED_OBJECT") - return doc, hashlib.sha256(raw).hexdigest(), items - - -def _publication_id(receipt_sha: str, publisher: str, target_id: str) -> str: - h = hashlib.sha256() - h.update(b"release-publication-v1\0") - for value in (receipt_sha, publisher, target_id): - data = value.encode("utf-8") - h.update(len(data).to_bytes(4, "big")) - h.update(data) - return "pub-" + h.hexdigest()[:32] - - -def _snapshot_artifacts( - promotion_dir: Path, items: list[dict[str, Any]], target: Path -) -> dict[str, Path]: - target.mkdir(parents=True, exist_ok=False) - out: dict[str, Path] = {} - for item in items: - source = promotion_dir / item["name"] - dest = target / item["name"] - with source.open("rb") as src, dest.open("xb") as dst: - shutil.copyfileobj(src, dst, length=_CHUNK) - os.chmod(dest, 0o444) - if dest.stat().st_size != item["size"] or _sha256(dest) != item["sha256"]: - _fail("PUBLICATION_SNAPSHOT_REBIND_FAILED") - out[item["name"]] = dest - return out - - -def _bounded_text(value: Any, code: str, *, limit: int = 512) -> str: - if not isinstance(value, str) or not value.strip() or len(value) > limit: - _fail(code) - if any( - ord(ch) < 32 # ruff: ignore[magic-value-comparison] - or ord(ch) == 127 # ruff: ignore[magic-value-comparison] - for ch in value - ): - _fail(code) - if any(marker in value for marker in ("?", "#", "@")): - _fail(code) - return value.strip() - - -def _validate_adapter_response( # ruff: ignore[too-many-branches] - value: Any, - *, - operation: str, - item: dict[str, Any], - publisher: str, - target_id: str, - publication_id: str, - now: datetime, - required_immutability: set[str], -) -> dict[str, Any]: - if not isinstance(value, dict) or set(value) != { - "schemaVersion", - "operation", - "publicationId", - "status", - "target", - "artifact", - "guarantees", - "verifiedAt", - }: - _fail("PUBLISHER_RESPONSE_SCHEMA_INVALID") - if ( - value.get("schemaVersion") != int(POLICY["adapter_protocol_version"]) - or value.get("operation") != operation - ): - _fail("PUBLISHER_RESPONSE_VERSION_INVALID") - if value.get("publicationId") != publication_id: - _fail("PUBLISHER_RESPONSE_PUBLICATION_ID_MISMATCH") - allowed_status = {"created", "present"} if operation == "publish" else {"present"} - if value.get("status") not in allowed_status: - _fail("PUBLISHER_RESPONSE_STATUS_INVALID") - target = value.get("target") - if not isinstance(target, dict) or set(target) != { - "publisher", - "targetId", - "locator", - }: - _fail("PUBLISHER_RESPONSE_TARGET_SCHEMA_INVALID") - if target.get("publisher") != publisher or target.get("targetId") != target_id: - _fail("PUBLISHER_RESPONSE_TARGET_MISMATCH") - locator = _bounded_text( - target.get("locator"), "PUBLISHER_RESPONSE_LOCATOR_INVALID", limit=1024 - ) - artifact = value.get("artifact") - if artifact != { - "name": item["name"], - "sha256": item["sha256"], - "size": item["size"], - }: - _fail("PUBLISHER_RESPONSE_ARTIFACT_MISMATCH") - guarantees = value.get("guarantees") - expected_guarantees = { - "createOnly", - "overwrite", - "remoteReadbackVerified", - "immutability", - } - if not isinstance(guarantees, dict) or set(guarantees) != expected_guarantees: - _fail("PUBLISHER_RESPONSE_GUARANTEES_SCHEMA_INVALID") - if ( - guarantees.get("createOnly") is not True - or guarantees.get("overwrite") is not False - or guarantees.get("remoteReadbackVerified") is not True - ): - _fail("PUBLISHER_RESPONSE_AUTHORITY_INVALID") - immutability = guarantees.get("immutability") - allowed = set(POLICY["allowed_immutability"]) - if not isinstance(immutability, str) or immutability not in allowed: - _fail("PUBLISHER_RESPONSE_IMMUTABILITY_INVALID") - if required_immutability and immutability not in required_immutability: - _fail("PUBLISHER_RESPONSE_IMMUTABILITY_INSUFFICIENT") - verified_at = _parse_time( - value.get("verifiedAt"), "PUBLISHER_RESPONSE_TIME_INVALID" - ) - skew = timedelta(minutes=int(POLICY["max_clock_skew_minutes"])) - max_age = timedelta(minutes=int(POLICY["max_result_age_minutes"])) - if verified_at > now + skew: - _fail("PUBLISHER_RESPONSE_FROM_FUTURE") - if now - verified_at > max_age: - _fail("PUBLISHER_RESPONSE_STALE") - return { - "status": value["status"], - "locator": locator, - "immutability": immutability, - "verifiedAt": value["verifiedAt"], - } - - -Publisher = Callable[[dict[str, Any]], dict[str, Any]] - - -def command_publisher( # ruff: ignore[undocumented-public-function] - command: list[str], -) -> Publisher: - if not command or any(not isinstance(part, str) or not part for part in command): - _fail("PUBLISHER_COMMAND_INVALID") - executable = ( - shutil.which(command[0]) if not os.path.isabs(command[0]) else command[0] - ) - if not executable or not Path(executable).is_file(): - _fail("PUBLISHER_COMMAND_UNAVAILABLE") - argv = [executable, *command[1:]] - - def invoke( # ruff: ignore[too-many-branches] - request: dict[str, Any], - ) -> dict[str, Any]: - payload = _canonical_bytes(request) - limit = int(POLICY["max_adapter_output_bytes"]) - proc = subprocess.Popen( # ruff: ignore[subprocess-without-shell-equals-true] - argv, - stdin=subprocess.PIPE, - stdout=subprocess.PIPE, - stderr=subprocess.DEVNULL, - env={"PATH": os.defpath, "LC_ALL": "C", "LANG": "C"}, - ) - assert ( # ruff: ignore[assert, pytest-composite-assertion] - proc.stdin is not None and proc.stdout is not None - ) - chunks: list[bytes] = [] - state = {"size": 0, "overflow": False, "error": None} - - def drain_stdout() -> None: - try: - while True: - chunk = proc.stdout.read(65536) - if not chunk: - return - state["size"] += len(chunk) - if state["size"] > limit: - state["overflow"] = True - try: # ruff: ignore[suppressible-exception] - proc.kill() - except OSError: - pass - return - chunks.append(chunk) - # defensive pipe failure - except ( - BaseException # ruff: ignore[blind-except] - ) as exc: # pragma: no cover - state["error"] = exc - try: # ruff: ignore[suppressible-exception] - proc.kill() - except OSError: - pass - - reader = threading.Thread( - target=drain_stdout, name="release-publisher-stdout", daemon=True - ) - reader.start() - try: - try: - proc.stdin.write(payload) - proc.stdin.close() - proc.wait(timeout=int(POLICY["adapter_timeout_seconds"])) - except subprocess.TimeoutExpired as exc: - proc.kill() - proc.wait() - reader.join(timeout=2) - raise PublicationError("PUBLISHER_COMMAND_TIMEOUT") from exc - finally: - try: # ruff: ignore[suppressible-exception] - proc.stdin.close() - except OSError: - pass - - reader.join(timeout=2) - if reader.is_alive(): - proc.kill() - proc.wait() - reader.join(timeout=2) - _fail("PUBLISHER_COMMAND_OUTPUT_DRAIN_FAILED") - if state["error"] is not None: - _fail("PUBLISHER_COMMAND_OUTPUT_DRAIN_FAILED") - if state["overflow"]: - _fail("PUBLISHER_COMMAND_OUTPUT_TOO_LARGE") - if proc.returncode != 0: - _fail("PUBLISHER_COMMAND_FAILED") - stdout = b"".join(chunks) - try: - return _loads_json(stdout, "PUBLISHER_COMMAND_OUTPUT") - except PublicationError as exc: - if str(exc).startswith("PUBLISHER_COMMAND_OUTPUT"): - raise - raise PublicationError("PUBLISHER_COMMAND_OUTPUT_INVALID") from exc - finally: - if proc.poll() is None: - proc.kill() - proc.wait() - if reader.is_alive(): - reader.join(timeout=2) - try: # ruff: ignore[suppressible-exception] - proc.stdout.close() - except OSError: - pass - - return invoke - - -def publish_release( # ruff: ignore[undocumented-public-function] - *, - promotion_dir: Path, - output_dir: Path, - publisher: str, - target_id: str, - adapter: Publisher, - required_immutability: Iterable[str] = (), - now: datetime | None = None, -) -> dict[str, Any]: - promotion_dir = promotion_dir.expanduser().resolve() - publisher = _safe_id(publisher, "PUBLICATION_PUBLISHER_INVALID") - target_id = _safe_id(target_id, "PUBLICATION_TARGET_ID_INVALID") - target = _outside(output_dir, promotion_dir, "PUBLICATION_OUTPUT_INSIDE_PROMOTION") - if target.exists() or target.is_symlink(): - _fail("PUBLICATION_OUTPUT_ALREADY_EXISTS") - required = set(required_immutability) - if not required.issubset(set(POLICY["allowed_immutability"])): - _fail("PUBLICATION_REQUIRED_IMMUTABILITY_INVALID") - fixed_now = now.astimezone(timezone.utc) if now is not None else None - - def clock_now() -> datetime: - return fixed_now or datetime.now(timezone.utc) - - receipt, receipt_sha, items = _validate_promotion(promotion_dir) - publication_id = _publication_id(receipt_sha, publisher, target_id) - - target.parent.mkdir(parents=True, exist_ok=True) - with tempfile.TemporaryDirectory( - prefix="release-publication-", dir=target.parent - ) as temp_raw: - temp = Path(temp_raw) - snapshots = _snapshot_artifacts(promotion_dir, items, temp / "snapshot") - evidence_dir = temp / "evidence" - results_dir = evidence_dir / "publisher-results" - results_dir.mkdir(parents=True) - transparency_artifacts: list[dict[str, Any]] = [] - receipt_artifacts: list[dict[str, Any]] = [] - locators: set[str] = set() - for index, item in enumerate(items, 1): - snapshot = snapshots[item["name"]] - base_request = { - "schemaVersion": int(POLICY["adapter_protocol_version"]), - "publicationId": publication_id, - "releaseId": receipt["releaseId"], - "promotionReceiptSha256": receipt_sha, - "target": {"publisher": publisher, "targetId": target_id}, - "artifact": { - "name": item["name"], - "sha256": item["sha256"], - "size": item["size"], - }, - } - publish_request = dict(base_request) - publish_request["operation"] = "publish" - publish_request["artifact"] = dict( - base_request["artifact"], localPath=str(snapshot) - ) - before = _sha256(snapshot) - publish_raw = adapter(publish_request) - if _sha256(snapshot) != before or before != item["sha256"]: - _fail("PUBLICATION_LOCAL_ARTIFACT_CHANGED_DURING_UPLOAD") - publish_result = _validate_adapter_response( - publish_raw, - operation="publish", - item=item, - publisher=publisher, - target_id=target_id, - publication_id=publication_id, - now=clock_now(), - required_immutability=required, - ) - verify_request = dict(base_request) - verify_request["operation"] = "verify" - verify_raw = adapter(verify_request) - verify_result = _validate_adapter_response( - verify_raw, - operation="verify", - item=item, - publisher=publisher, - target_id=target_id, - publication_id=publication_id, - now=clock_now(), - required_immutability=required, - ) - if ( - publish_result["locator"] != verify_result["locator"] - or publish_result["immutability"] != verify_result["immutability"] - ): - _fail("PUBLISHER_VERIFY_RESULT_REBIND_FAILED") - locator = verify_result["locator"] - if locator in locators: - _fail("PUBLISHER_LOCATOR_COLLISION") - locators.add(locator) - if ( - _sha256(snapshot) != item["sha256"] - or snapshot.stat().st_size != item["size"] - ): - _fail("PUBLICATION_LOCAL_SNAPSHOT_CHANGED") - pub_path = results_dir / f"{index:02d}-{item['name']}.publish.json" - ver_path = results_dir / f"{index:02d}-{item['name']}.verify.json" - _write_canonical(pub_path, publish_raw) - _write_canonical(ver_path, verify_raw) - transparency_artifacts.append( - { - "name": item["name"], - "sha256": item["sha256"], - "size": item["size"], - "remote": { - "locator": locator, - "sha256": item["sha256"], - "size": item["size"], - "immutability": verify_result["immutability"], - "verifiedAt": verify_result["verifiedAt"], - }, - "publisherEvidence": { - "publishSha256": _sha256(pub_path), - "verifySha256": _sha256(ver_path), - }, - } - ) - receipt_artifacts.append( - { - "name": item["name"], - "sha256": item["sha256"], - "size": item["size"], - "locator": locator, - "immutability": verify_result["immutability"], - } - ) - - # The original promotion directory may be mutable, but every published byte came from the - # verified server-owned snapshot. Re-validate it anyway to surface operator-side drift. - receipt_after, receipt_sha_after, items_after = _validate_promotion( - promotion_dir - ) - if ( - receipt_sha_after != receipt_sha - or receipt_after != receipt - or items_after != items - ): - _fail("PROMOTION_CHANGED_DURING_PUBLICATION") - - transparency = { - "schemaVersion": int(POLICY["transparency_schema_version"]), - "predicateType": PREDICATE_TYPE, - "generatedAt": _utc(clock_now()), - "publicationId": publication_id, - "release": { - "releaseId": receipt["releaseId"], - "sourceRevision": receipt["sourceRevision"], - }, - "subject": { - "promotionReceiptSha256": receipt_sha, - "sourceTreeSha256": receipt["sourceTreeSha256"], - "evidenceSha256": receipt["evidenceSha256"], - "releaseStatementSha256": receipt["releaseStatementSha256"], - }, - "target": {"publisher": publisher, "targetId": target_id}, - "artifacts": transparency_artifacts, - "verification": { - "promotionReceiptVerified": True, - "localSnapshotVerified": True, - "createOnly": True, - "remoteReadbackVerified": True, - "allReceiptObjectsPublished": True, - }, - } - transparency_path = evidence_dir / "publication-transparency.json" - _write_canonical(transparency_path, transparency) - publication_receipt = { - "schemaVersion": int(POLICY["publication_receipt_schema_version"]), - "status": "published", - "completedAt": _utc(clock_now()), - "publicationId": publication_id, - "releaseId": receipt["releaseId"], - "sourceRevision": receipt["sourceRevision"], - "promotionReceiptSha256": receipt_sha, - "transparencySha256": _sha256(transparency_path), - "target": {"publisher": publisher, "targetId": target_id}, - "artifacts": receipt_artifacts, - } - receipt_path = evidence_dir / "publication-receipt.json" - _write_canonical(receipt_path, publication_receipt) - tmp_target = target.with_name(target.name + ".tmp-" + os.urandom(6).hex()) - shutil.copytree(evidence_dir, tmp_target, copy_function=shutil.copy2) - os.replace(tmp_target, target) - return { - "ok": True, - "phase": "published", - "publication_id": publication_id, - "release_id": receipt["releaseId"], - "promotion_receipt_sha256": receipt_sha, - "transparency_sha256": _sha256(target / "publication-transparency.json"), - "publication_receipt_sha256": _sha256(target / "publication-receipt.json"), - "artifact_count": len(items), - } - - -def main( # ruff: ignore[undocumented-public-function] - argv: list[str] | None = None, -) -> int: - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--promotion-dir", type=Path, required=True) - parser.add_argument("--output-dir", type=Path, required=True) - parser.add_argument("--publisher", required=True) - parser.add_argument("--target-id", required=True) - parser.add_argument("--publisher-executable", required=True) - parser.add_argument("--publisher-arg", action="append", default=[]) - parser.add_argument("--require-immutability", action="append", default=[]) - args = parser.parse_args(argv) - try: - adapter = command_publisher([args.publisher_executable, *args.publisher_arg]) - result = publish_release( - promotion_dir=args.promotion_dir, - output_dir=args.output_dir, - publisher=args.publisher, - target_id=args.target_id, - adapter=adapter, - required_immutability=args.require_immutability, - ) - except PublicationError as exc: - logger.error(json.dumps({"ok": False, "error": str(exc)}, sort_keys=True)) - return 2 - logger.info(json.dumps(result, sort_keys=True)) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/security/python-runtime.cdx.json b/security/python-runtime.cdx.json deleted file mode 100644 index e0b51c3d099380d0460bf973c9d7f199653334af..0000000000000000000000000000000000000000 --- a/security/python-runtime.cdx.json +++ /dev/null @@ -1,744 +0,0 @@ -{ - "bomFormat": "CycloneDX", - "components": [ - { - "hashes": [ - { - "alg": "SHA-256", - "content": "571ac1dc6991c450b25a9c2d84a3705e2ae7a53467b5d111c24fa8baabbed320" - } - ], - "name": "annotated-doc", - "properties": [ - { - "name": "scikitplot:lock-target", - "value": "CPython 3.11 / linux/amd64" - }, - { - "name": "scikitplot:artifact-policy", - "value": "binary-wheel-only" - } - ], - "purl": "pkg:pypi/annotated-doc@0.0.4", - "scope": "required", - "type": "library", - "version": "0.0.4" - }, - { - "hashes": [ - { - "alg": "SHA-256", - "content": "1f02e8b43a8fbbc3f3e0d4f0f4bfc8131bcb4eebe8849b8e5c773f3a1c582a53" - } - ], - "name": "annotated-types", - "properties": [ - { - "name": "scikitplot:lock-target", - "value": "CPython 3.11 / linux/amd64" - }, - { - "name": "scikitplot:artifact-policy", - "value": "binary-wheel-only" - } - ], - "purl": "pkg:pypi/annotated-types@0.7.0", - "scope": "required", - "type": "library", - "version": "0.7.0" - }, - { - "hashes": [ - { - "alg": "SHA-256", - "content": "08b310f9e24a9594186fd75b4f73f4a4152069e3853f1ed8bfbf58369f4ad708" - } - ], - "name": "anyio", - "properties": [ - { - "name": "scikitplot:lock-target", - "value": "CPython 3.11 / linux/amd64" - }, - { - "name": "scikitplot:artifact-policy", - "value": "binary-wheel-only" - } - ], - "purl": "pkg:pypi/anyio@4.13.0", - "scope": "required", - "type": "library", - "version": "4.13.0" - }, - { - "hashes": [ - { - "alg": "SHA-256", - "content": "3c52e209ba0a4ad7aebe60436a4ab349c39e1e602e8c134221e546902ad25897" - } - ], - "name": "certifi", - "properties": [ - { - "name": "scikitplot:lock-target", - "value": "CPython 3.11 / linux/amd64" - }, - { - "name": "scikitplot:artifact-policy", - "value": "binary-wheel-only" - } - ], - "purl": "pkg:pypi/certifi@2026.5.20", - "scope": "required", - "type": "library", - "version": "2026.5.20" - }, - { - "hashes": [ - { - "alg": "SHA-256", - "content": "a2bf429bb3033c89fa4936ffb35d5cb471e3719e1f3c8a7c3fff0b8314305613" - } - ], - "name": "click", - "properties": [ - { - "name": "scikitplot:lock-target", - "value": "CPython 3.11 / linux/amd64" - }, - { - "name": "scikitplot:artifact-policy", - "value": "binary-wheel-only" - } - ], - "purl": "pkg:pypi/click@8.3.3", - "scope": "required", - "type": "library", - "version": "8.3.3" - }, - { - "hashes": [ - { - "alg": "SHA-256", - "content": "bfb91aa2d334c61cb35ba9a116fc123b3d3df31640b801cf57a7a78ec3f603b3" - } - ], - "name": "fastapi", - "properties": [ - { - "name": "scikitplot:lock-target", - "value": "CPython 3.11 / linux/amd64" - }, - { - "name": "scikitplot:artifact-policy", - "value": "binary-wheel-only" - } - ], - "purl": "pkg:pypi/fastapi@0.141.1", - "scope": "required", - "type": "library", - "version": "0.141.1" - }, - { - "hashes": [ - { - "alg": "SHA-256", - "content": "96f5f6344709aa1572bbf631c640e4ebeeb519e08da902c39a001882f30ac258" - } - ], - "name": "filelock", - "properties": [ - { - "name": "scikitplot:lock-target", - "value": "CPython 3.11 / linux/amd64" - }, - { - "name": "scikitplot:artifact-policy", - "value": "binary-wheel-only" - } - ], - "purl": "pkg:pypi/filelock@3.29.0", - "scope": "required", - "type": "library", - "version": "3.29.0" - }, - { - "hashes": [ - { - "alg": "SHA-256", - "content": "11ef7bb35dab8a394fde6e608221d5cf3e8499401c249bebaeaad760a1a8dec2" - } - ], - "name": "fsspec", - "properties": [ - { - "name": "scikitplot:lock-target", - "value": "CPython 3.11 / linux/amd64" - }, - { - "name": "scikitplot:artifact-policy", - "value": "binary-wheel-only" - } - ], - "purl": "pkg:pypi/fsspec@2026.4.0", - "scope": "required", - "type": "library", - "version": "2026.4.0" - }, - { - "hashes": [ - { - "alg": "SHA-256", - "content": "63cf8bbe7522de3bf65932fda1d9c2772064ffb3dae62d55932da54b31cb6c86" - } - ], - "name": "h11", - "properties": [ - { - "name": "scikitplot:lock-target", - "value": "CPython 3.11 / linux/amd64" - }, - { - "name": "scikitplot:artifact-policy", - "value": "binary-wheel-only" - } - ], - "purl": "pkg:pypi/h11@0.16.0", - "scope": "required", - "type": "library", - "version": "0.16.0" - }, - { - "hashes": [ - { - "alg": "SHA-256", - "content": "3531b1823a0e6d77d80f9ed15ca0e00f0d115094f8ac033d5cae88f4564cc949" - } - ], - "name": "hf-xet", - "properties": [ - { - "name": "scikitplot:lock-target", - "value": "CPython 3.11 / linux/amd64" - }, - { - "name": "scikitplot:artifact-policy", - "value": "binary-wheel-only" - } - ], - "purl": "pkg:pypi/hf-xet@1.5.0", - "scope": "required", - "type": "library", - "version": "1.5.0" - }, - { - "hashes": [ - { - "alg": "SHA-256", - "content": "2d400746a40668fc9dec9810239072b40b4484b640a8c38fd654a024c7a1bf55" - } - ], - "name": "httpcore", - "properties": [ - { - "name": "scikitplot:lock-target", - "value": "CPython 3.11 / linux/amd64" - }, - { - "name": "scikitplot:artifact-policy", - "value": "binary-wheel-only" - } - ], - "purl": "pkg:pypi/httpcore@1.0.9", - "scope": "required", - "type": "library", - "version": "1.0.9" - }, - { - "hashes": [ - { - "alg": "SHA-256", - "content": "d909fcccc110f8c7faf814ca82a9a4d816bc5a6dbfea25d6591d6985b8ba59ad" - } - ], - "name": "httpx", - "properties": [ - { - "name": "scikitplot:lock-target", - "value": "CPython 3.11 / linux/amd64" - }, - { - "name": "scikitplot:artifact-policy", - "value": "binary-wheel-only" - } - ], - "purl": "pkg:pypi/httpx@0.28.1", - "scope": "required", - "type": "library", - "version": "0.28.1" - }, - { - "hashes": [ - { - "alg": "SHA-256", - "content": "64340de934b9ce37857ef85a82de72f5629e8a270f9119eabb12bf495eb53c22" - } - ], - "name": "huggingface-hub", - "properties": [ - { - "name": "scikitplot:lock-target", - "value": "CPython 3.11 / linux/amd64" - }, - { - "name": "scikitplot:artifact-policy", - "value": "binary-wheel-only" - } - ], - "purl": "pkg:pypi/huggingface-hub@1.16.1", - "scope": "required", - "type": "library", - "version": "1.16.1" - }, - { - "hashes": [ - { - "alg": "SHA-256", - "content": "466e48829084efe2548012b855df21540b96f2e20e51bd124c851536556a592c" - } - ], - "name": "idna", - "properties": [ - { - "name": "scikitplot:lock-target", - "value": "CPython 3.11 / linux/amd64" - }, - { - "name": "scikitplot:artifact-policy", - "value": "binary-wheel-only" - } - ], - "purl": "pkg:pypi/idna@3.17", - "scope": "required", - "type": "library", - "version": "3.17" - }, - { - "hashes": [ - { - "alg": "SHA-256", - "content": "9f7ebbcd14fe59494226453aed97c1070d83f8d24b6fc3a3bcf9a38092641c4a" - } - ], - "name": "markdown-it-py", - "properties": [ - { - "name": "scikitplot:lock-target", - "value": "CPython 3.11 / linux/amd64" - }, - { - "name": "scikitplot:artifact-policy", - "value": "binary-wheel-only" - } - ], - "purl": "pkg:pypi/markdown-it-py@4.2.0", - "scope": "required", - "type": "library", - "version": "4.2.0" - }, - { - "hashes": [ - { - "alg": "SHA-256", - "content": "84008a41e51615a49fc9966191ff91509e3c40b939176e643fd50a5c2196b8f8" - } - ], - "name": "mdurl", - "properties": [ - { - "name": "scikitplot:lock-target", - "value": "CPython 3.11 / linux/amd64" - }, - { - "name": "scikitplot:artifact-policy", - "value": "binary-wheel-only" - } - ], - "purl": "pkg:pypi/mdurl@0.1.2", - "scope": "required", - "type": "library", - "version": "0.1.2" - }, - { - "hashes": [ - { - "alg": "SHA-256", - "content": "29572ef2b1f17581046b3a2227d5c611fb25ec70ca1ba8554b24b0e69331a484" - } - ], - "name": "packaging", - "properties": [ - { - "name": "scikitplot:lock-target", - "value": "CPython 3.11 / linux/amd64" - }, - { - "name": "scikitplot:artifact-policy", - "value": "binary-wheel-only" - } - ], - "purl": "pkg:pypi/packaging@25.0", - "scope": "required", - "type": "library", - "version": "25.0" - }, - { - "hashes": [ - { - "alg": "SHA-256", - "content": "45a282cde31d808236fd7ea9d919b128653c8b38b393d1c4ab335c62924d9aba" - } - ], - "name": "pydantic", - "properties": [ - { - "name": "scikitplot:lock-target", - "value": "CPython 3.11 / linux/amd64" - }, - { - "name": "scikitplot:artifact-policy", - "value": "binary-wheel-only" - } - ], - "purl": "pkg:pypi/pydantic@2.13.4", - "scope": "required", - "type": "library", - "version": "2.13.4" - }, - { - "hashes": [ - { - "alg": "SHA-256", - "content": "f9fa868638bf362d3d138ea55829cefb3d5f4b0d7f142234382a15e2485dbec4" - } - ], - "name": "pydantic-core", - "properties": [ - { - "name": "scikitplot:lock-target", - "value": "CPython 3.11 / linux/amd64" - }, - { - "name": "scikitplot:artifact-policy", - "value": "binary-wheel-only" - } - ], - "purl": "pkg:pypi/pydantic-core@2.46.4", - "scope": "required", - "type": "library", - "version": "2.46.4" - }, - { - "hashes": [ - { - "alg": "SHA-256", - "content": "81a9e26dd42fd28a23a2d169d86d7ac03b46e2f8b59ed4698fb4785f946d0176" - } - ], - "name": "pygments", - "properties": [ - { - "name": "scikitplot:lock-target", - "value": "CPython 3.11 / linux/amd64" - }, - { - "name": "scikitplot:artifact-policy", - "value": "binary-wheel-only" - } - ], - "purl": "pkg:pypi/pygments@2.20.0", - "scope": "required", - "type": "library", - "version": "2.20.0" - }, - { - "hashes": [ - { - "alg": "SHA-256", - "content": "ff6d3f776f16878c894e52e107296ffc890e913c611b1a4ec6c44e2821fe2e23" - } - ], - "name": "python-multipart", - "properties": [ - { - "name": "scikitplot:lock-target", - "value": "CPython 3.11 / linux/amd64" - }, - { - "name": "scikitplot:artifact-policy", - "value": "binary-wheel-only" - } - ], - "purl": "pkg:pypi/python-multipart@0.0.32", - "scope": "required", - "type": "library", - "version": "0.0.32" - }, - { - "hashes": [ - { - "alg": "SHA-256", - "content": "b8bb0864c5a28024fac8a632c443c87c5aa6f215c0b126c449ae1a150412f31d" - } - ], - "name": "pyyaml", - "properties": [ - { - "name": "scikitplot:lock-target", - "value": "CPython 3.11 / linux/amd64" - }, - { - "name": "scikitplot:artifact-policy", - "value": "binary-wheel-only" - } - ], - "purl": "pkg:pypi/pyyaml@6.0.3", - "scope": "required", - "type": "library", - "version": "6.0.3" - }, - { - "hashes": [ - { - "alg": "SHA-256", - "content": "a4fe1aac3d3b3cc791d4b3d5931c5a956045dc951ee74d1c913ee3ac4d2ee9fb" - } - ], - "name": "redis", - "properties": [ - { - "name": "scikitplot:lock-target", - "value": "CPython 3.11 / linux/amd64" - }, - { - "name": "scikitplot:artifact-policy", - "value": "binary-wheel-only" - } - ], - "purl": "pkg:pypi/redis@8.1.0", - "scope": "required", - "type": "library", - "version": "8.1.0" - }, - { - "hashes": [ - { - "alg": "SHA-256", - "content": "33bd4ef74232fb73fe9279a257718407f169c09b78a87ad3d296f548e27de0bb" - } - ], - "name": "rich", - "properties": [ - { - "name": "scikitplot:lock-target", - "value": "CPython 3.11 / linux/amd64" - }, - { - "name": "scikitplot:artifact-policy", - "value": "binary-wheel-only" - } - ], - "purl": "pkg:pypi/rich@15.0.0", - "scope": "required", - "type": "library", - "version": "15.0.0" - }, - { - "hashes": [ - { - "alg": "SHA-256", - "content": "7ecfff8f2fd72616f7481040475a65b2bf8af90a56c89140852d1120324e8686" - } - ], - "name": "shellingham", - "properties": [ - { - "name": "scikitplot:lock-target", - "value": "CPython 3.11 / linux/amd64" - }, - { - "name": "scikitplot:artifact-policy", - "value": "binary-wheel-only" - } - ], - "purl": "pkg:pypi/shellingham@1.5.4", - "scope": "required", - "type": "library", - "version": "1.5.4" - }, - { - "hashes": [ - { - "alg": "SHA-256", - "content": "a86dd39d14bb45f85a3d18525215a9ef0cfd1f192ac793220e72598c90335f0c" - } - ], - "name": "starlette", - "properties": [ - { - "name": "scikitplot:lock-target", - "value": "CPython 3.11 / linux/amd64" - }, - { - "name": "scikitplot:artifact-policy", - "value": "binary-wheel-only" - } - ], - "purl": "pkg:pypi/starlette@1.6.0", - "scope": "required", - "type": "library", - "version": "1.6.0" - }, - { - "hashes": [ - { - "alg": "SHA-256", - "content": "ee1e4c0e59148062281c49d80b25b67771a127c85fc9676d3be5f243206826bf" - } - ], - "name": "tqdm", - "properties": [ - { - "name": "scikitplot:lock-target", - "value": "CPython 3.11 / linux/amd64" - }, - { - "name": "scikitplot:artifact-policy", - "value": "binary-wheel-only" - } - ], - "purl": "pkg:pypi/tqdm@4.67.3", - "scope": "required", - "type": "library", - "version": "4.67.3" - }, - { - "hashes": [ - { - "alg": "SHA-256", - "content": "e70549ec5a403ca8a0bf0802ddd9f3c6ff7a14ccbb859b01b697baa943636f33" - } - ], - "name": "typer", - "properties": [ - { - "name": "scikitplot:lock-target", - "value": "CPython 3.11 / linux/amd64" - }, - { - "name": "scikitplot:artifact-policy", - "value": "binary-wheel-only" - } - ], - "purl": "pkg:pypi/typer@0.26.3", - "scope": "required", - "type": "library", - "version": "0.26.3" - }, - { - "hashes": [ - { - "alg": "SHA-256", - "content": "481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8" - } - ], - "name": "typing-extensions", - "properties": [ - { - "name": "scikitplot:lock-target", - "value": "CPython 3.11 / linux/amd64" - }, - { - "name": "scikitplot:artifact-policy", - "value": "binary-wheel-only" - } - ], - "purl": "pkg:pypi/typing-extensions@4.16.0", - "scope": "required", - "type": "library", - "version": "4.16.0" - }, - { - "hashes": [ - { - "alg": "SHA-256", - "content": "4ed1cacbdc298c220f1bd249ed5287caa16f34d44ef4e9c3d0cbad5b521545e7" - } - ], - "name": "typing-inspection", - "properties": [ - { - "name": "scikitplot:lock-target", - "value": "CPython 3.11 / linux/amd64" - }, - { - "name": "scikitplot:artifact-policy", - "value": "binary-wheel-only" - } - ], - "purl": "pkg:pypi/typing-inspection@0.4.2", - "scope": "required", - "type": "library", - "version": "0.4.2" - }, - { - "hashes": [ - { - "alg": "SHA-256", - "content": "48097851328b87ec36117d3d575234519eb58c2b22d79666e9bbc6c49a761dad" - } - ], - "name": "uvicorn", - "properties": [ - { - "name": "scikitplot:lock-target", - "value": "CPython 3.11 / linux/amd64" - }, - { - "name": "scikitplot:artifact-policy", - "value": "binary-wheel-only" - } - ], - "purl": "pkg:pypi/uvicorn@0.48.0", - "scope": "required", - "type": "library", - "version": "0.48.0" - } - ], - "metadata": { - "component": { - "bom-ref": "pkg:generic/scikit-plots-ai-proxy@6.8.0", - "name": "scikit-plots-ai-proxy", - "type": "application", - "version": "6.8.0" - }, - "properties": [ - { - "name": "scikitplot:scope", - "value": "python-runtime-lock-only" - }, - { - "name": "scikitplot:target", - "value": "CPython 3.11 / linux/amd64" - }, - { - "name": "scikitplot:note", - "value": "A full image SBOM must be generated from the built container before release." - } - ], - "timestamp": "2026-08-30T00:00:00Z" - }, - "serialNumber": "urn:uuid:d714f9a3-0034-5843-b870-51885c937bc8", - "specVersion": "1.6", - "version": 1 -} diff --git a/security/rebridge_archive_merkle_authority.py b/security/rebridge_archive_merkle_authority.py deleted file mode 100644 index 91dfe0b8b7980275869ccd1a02c0f6d6a4dc43fe..0000000000000000000000000000000000000000 --- a/security/rebridge_archive_merkle_authority.py +++ /dev/null @@ -1,2309 +0,0 @@ -""" -Run 167: recursively re-bridge rotated/recovered Merkle authority after Run 166. - -Run 166 permits arbitrarily many RFC6962 appends while one Run 165 authority remains -active. Run 167 handles the next authority change *after* those appends: the old/new -(or recovery) handoff is signed against the latest accepted Run 166/Run 167 checkpoint, -then the first new Merkle leaf is appended under the replacement authority. Later -append-only epochs and later re-bridges use the same cumulative format. -""" - -from __future__ import annotations - -import argparse -import base64 -import hashlib -import json -import logging -import re -import shutil -import stat -import tempfile -from datetime import datetime, timedelta, timezone -from pathlib import Path -from typing import Any, Callable - -import tomllib -from cryptography.exceptions import InvalidSignature -from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PublicKey - -logger = logging.getLogger(__name__) - -try: - import continue_archive_merkle_authority as run166 - import govern_archive_merkle_log_authority as run165 - import verify_archive_merkle_transparency as merkle -except ImportError: # canonical package import - from . import continue_archive_merkle_authority as run166 - from . import govern_archive_merkle_log_authority as run165 - from . import verify_archive_merkle_transparency as merkle - -HERE = Path(__file__).resolve().parent -POLICY = tomllib.loads( - (HERE / "release_archive_merkle_rebridge_policy.toml").read_text() -) -PREDICATE_TYPE = str(POLICY["predicate_type"]) - - -# ── Verified-state document names ──────────────────────────────────────── -# -# One constant per release-evidence document. The values are the on-disk -# artifact names and are unchanged; only the repetition is removed. Reading -# a document by a name that says which document it is also keeps static -# analysis from reading the filename token 'trusted' as 'confidential': -# these files carry published Merkle roots and SHA-256 digests, which must -# stay in clear text for any third party to verify them. -_DOC_ARCHIVE_ANCHOR_STATE = "trusted-archive-anchor-state.json" -_DOC_ARCHIVE_LOG_AUTHORITY_STATE = "trusted-archive-log-authority-state.json" -_DOC_ARCHIVE_MERKLE_CONTINUITY_STATE = "trusted-archive-merkle-continuity-state.json" -_DOC_ARCHIVE_MERKLE_REBRIDGE_STATE = "trusted-archive-merkle-rebridge-state.json" - - -_OUTPUT_NAMES = { - "release-archive-merkle-rebridge-bundle.json", - _DOC_ARCHIVE_MERKLE_REBRIDGE_STATE, - "active-archive-merkle-rebridge.json", - "release-archive-merkle-rebridge-receipt.json", -} -_RUN166_NAMES = { - "release-archive-merkle-continuity-bundle.json", - _DOC_ARCHIVE_MERKLE_CONTINUITY_STATE, - "active-archive-merkle-continuity.json", - "release-archive-merkle-continuity-receipt.json", -} -_RUN165_NAMES = { - "release-archive-log-authority-bundle.json", - _DOC_ARCHIVE_LOG_AUTHORITY_STATE, - "active-archive-log-authority.json", - "release-archive-log-authority-receipt.json", -} -_ID_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:/+-]{0,255}$") - - -class ArchiveMerkleRebridgeError( # ruff: ignore[undocumented-public-class] - RuntimeError -): - pass - - -def _fail(code: str) -> None: - raise ArchiveMerkleRebridgeError(code) - - -def _canonical(value: Any) -> bytes: - return merkle._canonical(value) - - -def _sha_bytes(raw: bytes) -> str: - return hashlib.sha256(raw).hexdigest() - - -def _sha(path: Path) -> str: - h = hashlib.sha256() - with Path(path).open("rb") as f: - for chunk in iter(lambda: f.read(1024 * 1024), b""): - h.update(chunk) - return h.hexdigest() - - -def _no_dupes(pairs): - out = {} - for key, value in pairs: - if key in out: - _fail("ARCHIVE_MERKLE_REBRIDGE_JSON_DUPLICATE_KEY") - out[key] = value - return out - - -def _loads(raw: bytes, code: str) -> dict[str, Any]: - if len(raw) > int(POLICY["max_json_bytes"]): - _fail(code + "_TOO_LARGE") - try: - value = json.loads(raw.decode("utf-8"), object_pairs_hook=_no_dupes) - except ArchiveMerkleRebridgeError: - raise - except Exception: # ruff: ignore[blind-except] - _fail(code + "_JSON_INVALID") - if not isinstance(value, dict): - _fail(code + "_SCHEMA_INVALID") - if _canonical(value) != raw: - _fail(code + "_NOT_CANONICAL") - return value - - -def _read_json(path: Path, code: str) -> tuple[dict[str, Any], bytes]: - path = Path(path) - try: - st = path.lstat() - except OSError: - _fail(code + "_MISSING") - if not stat.S_ISREG(st.st_mode) or st.st_nlink != 1: - _fail(code + "_NOT_REGULAR") - raw = path.read_bytes() - return _loads(raw, code), raw - - -def _write(path: Path, value: Any) -> None: - path.write_bytes(_canonical(value)) - - -def _id(value: Any, code: str) -> str: - if not isinstance(value, str) or not _ID_RE.fullmatch(value): - _fail(code) - return value - - -def _hex(value: Any, code: str) -> str: - _len = len(value) != 64 # ruff: ignore[magic-value-comparison] - if not isinstance(value, str) or _len: - _fail(code) - try: - bytes.fromhex(value) - except ValueError: - _fail(code) - if value != value.lower(): - _fail(code) - return value - - -def _dt(value: Any, code: str) -> datetime: - if not isinstance(value, str) or not value.endswith("Z"): - _fail(code) - try: - parsed = datetime.fromisoformat(value[:-1] + "+00:00") - except ValueError: - _fail(code) - if parsed.tzinfo is None: - _fail(code) - return parsed.astimezone(timezone.utc) - - -def _ts(value: datetime) -> str: - return ( - value.astimezone(timezone.utc) - .isoformat(timespec="seconds") - .replace("+00:00", "Z") - ) - - -def _b64(value: Any, code: str, size: int | None = None) -> bytes: - if not isinstance(value, str): - _fail(code) - try: - raw = base64.b64decode(value, validate=True) - except Exception: # ruff: ignore[blind-except] - _fail(code) - if size is not None and len(raw) != size: - _fail(code) - if base64.b64encode(raw).decode("ascii") != value: - _fail(code) - return raw - - -def _verify_sig(public_key: str, signature: str, message: bytes, code: str) -> None: - try: - Ed25519PublicKey.from_public_bytes( - _b64(public_key, code + "_PUBLIC_INVALID", 32) - ).verify(_b64(signature, code + "_SIGNATURE_INVALID", 64), message) - except InvalidSignature: - _fail(code + "_INVALID") - except ArchiveMerkleRebridgeError: - raise - except Exception: # ruff: ignore[blind-except] - _fail(code + "_INVALID") - - -def _positive_int(value: Any, code: str) -> int: - if not isinstance(value, int) or isinstance(value, bool) or value <= 0: - _fail(code) - return value - - -def _regular_dir(path: Path, code: str) -> Path: - path = Path(path) - try: - st = path.lstat() - except OSError: - _fail(code) - if not stat.S_ISDIR(st.st_mode): - _fail(code) - return path.resolve() - - -def _outside(path: Path, protected: list[Path], code: str) -> Path: - p = Path(path).absolute() - for item in protected: - q = Path(item).absolute() - if p == q or q in p.parents or p in q.parents: - _fail(code) - return p - - -def _dir_fingerprint(root: Path, code: str) -> tuple[tuple[str, str, int, int], ...]: - root = _regular_dir(root, code) - rows = [] - for path in sorted(root.rglob("*")): - rel = path.relative_to(root).as_posix() - st = path.lstat() - if stat.S_ISDIR(st.st_mode): - continue - if not stat.S_ISREG(st.st_mode) or st.st_nlink != 1: - _fail(code) - rows.append((rel, _sha(path), st.st_size, stat.S_IMODE(st.st_mode))) - return tuple(rows) - - -def _authority_fingerprint(paths: list[Path], code: str) -> tuple[Any, ...]: - rows = [] - for raw in paths: - p = Path(raw) - if p.is_dir(): - rows.append((str(p.absolute()), "dir", _dir_fingerprint(p, code))) - else: - try: - st = p.lstat() - except OSError: - _fail(code) - if not stat.S_ISREG(st.st_mode) or st.st_nlink != 1: - _fail(code) - rows.append( - ( - str(p.absolute()), - "file", - _sha(p), - st.st_size, - stat.S_IMODE(st.st_mode), - ) - ) - return tuple(rows) - - -def _artifact_map(raws: dict[str, bytes]) -> dict[str, dict[str, Any]]: - return { - name: {"sha256": _sha_bytes(raws[name]), "size": len(raws[name])} - for name in sorted(raws) - } - - -def _pub_fingerprint(public_key: str) -> str: - return _sha_bytes( - _b64(public_key, "ARCHIVE_MERKLE_REBRIDGE_PUBLIC_KEY_INVALID", 32) - ) - - -def _authority( - value: Any, *, expected_log_ids: set[str], code: str -) -> dict[str, dict[str, str]]: - try: - return run165._authority(value, expected_log_ids=expected_log_ids, code=code) - except Exception as exc: - raise ArchiveMerkleRebridgeError(code + "_INVALID:" + str(exc)) from exc - - -def _authority_sha(value: dict[str, Any]) -> str: - return _sha_bytes(_canonical(value)) - - -def _authority_fps(value: dict[str, Any]) -> set[str]: - out = set() - for row in value.values(): - out.add(_pub_fingerprint(row["publicKey"])) - out.add(_pub_fingerprint(row["gossipPublicKey"])) - return out - - -def _verify_control_root( - path: Path, pin: str, *, recovery: bool, now: datetime, historical: bool -) -> dict[str, Any]: - doc, _ = _read_json( - Path(path), - ( - "ARCHIVE_MERKLE_REBRIDGE_RECOVERY_ROOT" - if recovery - else "ARCHIVE_MERKLE_REBRIDGE_GOVERNANCE_ROOT" - ), - ) - try: - return run165._verify_control_root( - doc, pin, now=now, recovery=recovery, historical=historical - ) - except Exception as exc: - raise ArchiveMerkleRebridgeError( - ( - "ARCHIVE_MERKLE_REBRIDGE_RECOVERY_ROOT_INVALID:" - if recovery - else "ARCHIVE_MERKLE_REBRIDGE_GOVERNANCE_ROOT_INVALID:" - ) - + str(exc) - ) from exc - - -def _load_run164( - run164_dir: Path, - transparency_root_path: Path, - transparency_root_pin: str, - *, - now: datetime, -) -> dict[str, Any]: - try: - return run166._load_run164( - run164_dir, transparency_root_path, transparency_root_pin, now=now - ) - except Exception as exc: - raise ArchiveMerkleRebridgeError( - "ARCHIVE_MERKLE_REBRIDGE_RUN164_INVALID:" + str(exc) - ) from exc - - -def _load_run165( - *, - run164_dir: Path, - transparency_root_path: Path, - transparency_root_pin: str, - governance_root_path: Path, - governance_root_pin: str, - recovery_root_path: Path, - recovery_root_pin: str, - run165_dir: Path, - now: datetime, -) -> dict[str, Any]: - try: - return run166._load_run165( - run164_dir=run164_dir, - transparency_root_path=transparency_root_path, - transparency_root_pin=transparency_root_pin, - governance_root_path=governance_root_path, - governance_root_pin=governance_root_pin, - recovery_root_path=recovery_root_path, - recovery_root_pin=recovery_root_pin, - run165_dir=run165_dir, - now=now, - historical=True, - ) - except Exception as exc: - raise ArchiveMerkleRebridgeError( - "ARCHIVE_MERKLE_REBRIDGE_RUN165_INVALID:" + str(exc) - ) from exc - - -def _load_run166( - *, - run160_dir: Path, - run166_run161_dir: Path, - retention_root_path: Path, - retention_root_pin: str, - witness_root_path: Path, - witness_root_pin: str, - bootstrap_pin: str, - recovery_pin: str, - attestation_pins: list[str], - run166_run162_dir: Path, - anchor_plan_path: Path, - run166_run163_dir: Path, - run164_dir: Path, - transparency_root_path: Path, - transparency_root_pin: str, - governance_root_path: Path, - governance_root_pin: str, - recovery_root_path: Path, - recovery_root_pin: str, - run165_dir: Path, - run166_dir: Path, - now: datetime, -) -> dict[str, Any]: - try: - run166.verify_merkle_authority_continuity( - run160_dir=run160_dir, - run161_dir=run166_run161_dir, - retention_root_path=retention_root_path, - retention_root_pin=retention_root_pin, - witness_root_path=witness_root_path, - witness_root_pin=witness_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - run162_dir=run166_run162_dir, - anchor_plan_path=anchor_plan_path, - run163_dir=run166_run163_dir, - run164_dir=run164_dir, - transparency_root_path=transparency_root_path, - transparency_root_pin=transparency_root_pin, - governance_root_path=governance_root_path, - governance_root_pin=governance_root_pin, - recovery_root_path=recovery_root_path, - recovery_root_pin=recovery_root_pin, - run165_dir=run165_dir, - output_dir=run166_dir, - now=now, - historical=True, - ) - docs, raws = run166._load_output(Path(run166_dir)) - except Exception as exc: - raise ArchiveMerkleRebridgeError( - "ARCHIVE_MERKLE_REBRIDGE_RUN166_INVALID:" + str(exc) - ) from exc - state = docs[_DOC_ARCHIVE_MERKLE_CONTINUITY_STATE] - active = docs["active-archive-merkle-continuity.json"] - if state.get("sequence") != active.get("sequence") or state.get( - "merkleAuthorityContinuityHeadSha256" - ) != active.get("merkleAuthorityContinuityHeadSha256"): - _fail("ARCHIVE_MERKLE_REBRIDGE_RUN166_STATE_ACTIVE_MISMATCH") - if state.get("activeAuthoritySha256") != _authority_sha(active.get("authority")): - _fail("ARCHIVE_MERKLE_REBRIDGE_RUN166_AUTHORITY_MISMATCH") - cps = state.get("lastCheckpoints") - if not isinstance(cps, dict) or set(cps) != set(active.get("authority", {})): - _fail("ARCHIVE_MERKLE_REBRIDGE_RUN166_CHECKPOINTS_INVALID") - return { - "docs": docs, - "raws": raws, - "state": state, - "active": active, - "artifacts": _artifact_map(raws), - } - - -def _load_current_run163( - *, - run160_dir: Path, - run161_dir: Path, - retention_root_path: Path, - retention_root_pin: str, - witness_root_path: Path, - witness_root_pin: str, - bootstrap_pin: str, - recovery_pin: str, - attestation_pins: list[str], - run162_dir: Path, - anchor_plan_path: Path, - run163_dir: Path, - now: datetime, - historical: bool, -) -> dict[str, Any]: - try: - return run166._load_run163_current( - run160_dir=run160_dir, - run161_dir=run161_dir, - retention_root_path=retention_root_path, - retention_root_pin=retention_root_pin, - witness_root_path=witness_root_path, - witness_root_pin=witness_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - run162_dir=run162_dir, - anchor_plan_path=anchor_plan_path, - run163_dir=run163_dir, - now=now, - historical=historical, - ) - except Exception as exc: - raise ArchiveMerkleRebridgeError( - "ARCHIVE_MERKLE_REBRIDGE_RUN163_INVALID:" + str(exc) - ) from exc - - -def _verify_embedded_run163( - docs: dict[str, Any], - *, - run160_dir: Path, - run161_dir: Path, - retention_root_path: Path, - retention_root_pin: str, - witness_root_path: Path, - witness_root_pin: str, - bootstrap_pin: str, - recovery_pin: str, - attestation_pins: list[str], - run162_dir: Path, - anchor_plan_path: Path, - now: datetime, -) -> dict[str, Any]: - if not isinstance(docs, dict) or set(docs) != merkle._RUN163_NAMES: - _fail("ARCHIVE_MERKLE_REBRIDGE_EMBEDDED_RUN163_SCHEMA_INVALID") - with tempfile.TemporaryDirectory() as td: - root = Path(td) - for name in sorted(docs): - _write(root / name, docs[name]) - try: - result = merkle.run163.verify_anchor_history( - run160_dir=run160_dir, - run161_dir=run161_dir, - retention_root_path=retention_root_path, - retention_root_pin=retention_root_pin, - witness_root_path=witness_root_path, - witness_root_pin=witness_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - run162_dir=run162_dir, - anchor_plan_path=anchor_plan_path, - output_dir=root, - now=now, - historical=True, - require_current_match=False, - ) - except Exception as exc: - raise ArchiveMerkleRebridgeError( - "ARCHIVE_MERKLE_REBRIDGE_EMBEDDED_RUN163_INVALID:" + str(exc) - ) from exc - return result - - -def _enforce_separation( - governance: dict[str, Any], - recovery: dict[str, Any], - run164_root: dict[str, Any], - witness_root_path: Path, - anchor_plan_path: Path, - *authorities: dict[str, Any], -) -> None: - try: - run165._enforce_plane_separation( - governance, recovery, run164_root, *authorities - ) - for authority in authorities: - run166._enforce_external_separation( - authority, witness_root_path, anchor_plan_path - ) - except Exception as exc: - raise ArchiveMerkleRebridgeError( - "ARCHIVE_MERKLE_REBRIDGE_AUTHORITY_OVERLAP:" + str(exc) - ) from exc - - -def _normalize_checkpoints( - value: Any, *, expected_log_ids: set[str], code: str -) -> dict[str, dict[str, Any]]: - if not isinstance(value, dict) or set(value) != expected_log_ids: - _fail(code + "_SET_INVALID") - out = {} - for lid in sorted(value): - row = value[lid] - if not isinstance(row, dict) or set(row) != { - "checkpointSha256", - "treeSize", - "rootHash", - }: - _fail(code + "_SCHEMA_INVALID") - cp = _hex(row["checkpointSha256"], code + "_CHECKPOINT_INVALID") - ts = _positive_int(row["treeSize"], code + "_TREE_SIZE_INVALID") - rh = _hex(row["rootHash"], code + "_ROOT_INVALID") - out[lid] = {"checkpointSha256": cp, "treeSize": ts, "rootHash": rh} - if out != value: - _fail(code + "_NOT_NORMALIZED") - return out - - -def _base_authority_head( - run165_info: dict[str, Any], run166_info: dict[str, Any] -) -> str: - state_raw = run166_info["raws"][_DOC_ARCHIVE_MERKLE_CONTINUITY_STATE] - return _sha_bytes( - _canonical( - { - "run165LogAuthorityChainHeadSha256": run165_info["state"][ - "logAuthorityChainHeadSha256" - ], - "run166MerkleAuthorityContinuityHeadSha256": run166_info["state"][ - "merkleAuthorityContinuityHeadSha256" - ], - "run166TrustedStateSha256": _sha_bytes(state_raw), - } - ) - ) - - -def _bridge_head( - previous_head: str, transition_sha: str, handoff_sha: str, source_state_sha: str -) -> str: - return _sha_bytes( - _canonical( - { - "previousArchiveMerkleRebridgeAuthorityHeadSha256": previous_head, - "transitionSha256": transition_sha, - "handoffSha256": handoff_sha, - "sourceTrustedStateSha256": source_state_sha, - } - ) - ) - - -def _merkle_head(previous_head: str, event_without_head: dict[str, Any]) -> str: - return _sha_bytes( - _canonical( - { - "previousArchiveMerkleRebridgeContinuityHeadSha256": previous_head, - "event": event_without_head, - } - ) - ) - - -def _handoff_subject( - *, signed: dict[str, Any], log_id: str, checkpoint: dict[str, Any] -) -> dict[str, Any]: - return { - "_type": "archive-merkle-authority-rebridge-handoff", - "specVersion": str(POLICY["spec_version"]), - "schemaVersion": int(POLICY["handoff_schema_version"]), - "transitionId": signed["transitionId"], - "rebridgeSequence": signed["rebridgeSequence"], - "kind": signed["kind"], - "logId": log_id, - "sourceKind": signed["sourceKind"], - "sourceTrustedStateSha256": signed["sourceTrustedStateSha256"], - "sourceMerkleSequence": signed["sourceMerkleSequence"], - "sourceMerkleContinuityHeadSha256": signed["sourceMerkleContinuityHeadSha256"], - "previousAuthorityBridgeHeadSha256": signed[ - "previousAuthorityBridgeHeadSha256" - ], - "priorCheckpoint": checkpoint, - "currentLogAuthority": signed["currentAuthority"][log_id], - "nextLogAuthority": signed["nextAuthority"][log_id], - "currentAuthoritySha256": _authority_sha(signed["currentAuthority"]), - "nextAuthoritySha256": _authority_sha(signed["nextAuthority"]), - } - - -def _verify_transition( # ruff: ignore[too-many-branches] - doc: dict[str, Any], - *, - source: dict[str, Any], - governance: dict[str, Any], - recovery: dict[str, Any], - expected_log_ids: set[str], - seen_ids: set[str], - previous_transition_issued: datetime | None, - now: datetime, - creation: bool, -) -> dict[str, Any]: - if set(doc) != {"signed", "signatures"} or not isinstance(doc.get("signed"), dict): - _fail("ARCHIVE_MERKLE_REBRIDGE_TRANSITION_SCHEMA_INVALID") - signed = doc["signed"] - expected = { - "_type", - "specVersion", - "schemaVersion", - "transitionId", - "rebridgeSequence", - "kind", - "issuedAt", - "sourceKind", - "sourceTrustedStateSha256", - "sourceMerkleSequence", - "sourceMerkleContinuityHeadSha256", - "previousAuthorityBridgeHeadSha256", - "currentAuthority", - "nextAuthority", - "changedLogIds", - "compromisedKeyFingerprints", - "revokedKeyFingerprints", - "authorizationRole", - "selectedSignerKeyIds", - "handoffSubjectSha256s", - } - if ( - set(signed) != expected - or signed.get("_type") != "archive-merkle-authority-rebridge-transition" - ): - _fail("ARCHIVE_MERKLE_REBRIDGE_TRANSITION_SIGNED_SCHEMA_INVALID") - if signed.get("specVersion") != str(POLICY["spec_version"]) or signed.get( - "schemaVersion" - ) != int(POLICY["transition_schema_version"]): - _fail("ARCHIVE_MERKLE_REBRIDGE_TRANSITION_VERSION_INVALID") - tid = _id(signed["transitionId"], "ARCHIVE_MERKLE_REBRIDGE_TRANSITION_ID_INVALID") - if tid in seen_ids: - _fail("ARCHIVE_MERKLE_REBRIDGE_TRANSITION_ID_REUSED") - rbseq = _positive_int( - signed["rebridgeSequence"], "ARCHIVE_MERKLE_REBRIDGE_SEQUENCE_INVALID" - ) - if rbseq != source["rebridgeSequence"] + 1: - _fail("ARCHIVE_MERKLE_REBRIDGE_SEQUENCE_INVALID") - kind = signed["kind"] - if kind not in {"scheduled-rotation", "compromise-recovery"}: - _fail("ARCHIVE_MERKLE_REBRIDGE_KIND_INVALID") - issued = _dt(signed["issuedAt"], "ARCHIVE_MERKLE_REBRIDGE_ISSUED_INVALID") - if issued > now + timedelta(minutes=int(POLICY["max_clock_skew_minutes"])): - _fail("ARCHIVE_MERKLE_REBRIDGE_FROM_FUTURE") - if creation and now - issued > timedelta( - minutes=int(POLICY["max_transition_freshness_minutes"]) - ): - _fail("ARCHIVE_MERKLE_REBRIDGE_TRANSITION_STALE") - if previous_transition_issued is not None and issued <= previous_transition_issued: - _fail("ARCHIVE_MERKLE_REBRIDGE_TRANSITION_TIME_NOT_MONOTONIC") - if ( - signed["sourceKind"] != source["sourceKind"] - or signed["sourceTrustedStateSha256"] != source["stateSha256"] - ): - _fail("ARCHIVE_MERKLE_REBRIDGE_SOURCE_STATE_INVALID") - if ( - signed["sourceMerkleSequence"] != source["merkleSequence"] - or signed["sourceMerkleContinuityHeadSha256"] != source["merkleHead"] - ): - _fail("ARCHIVE_MERKLE_REBRIDGE_SOURCE_MERKLE_INVALID") - if signed["previousAuthorityBridgeHeadSha256"] != source["authorityHead"]: - _fail("ARCHIVE_MERKLE_REBRIDGE_PREVIOUS_AUTHORITY_HEAD_INVALID") - current = _authority( - signed["currentAuthority"], - expected_log_ids=expected_log_ids, - code="ARCHIVE_MERKLE_REBRIDGE_CURRENT_AUTHORITY", - ) - nxt = _authority( - signed["nextAuthority"], - expected_log_ids=expected_log_ids, - code="ARCHIVE_MERKLE_REBRIDGE_NEXT_AUTHORITY", - ) - if current != source["authority"]: - _fail("ARCHIVE_MERKLE_REBRIDGE_CURRENT_AUTHORITY_MISMATCH") - for lid in expected_log_ids: - if current[lid]["gossipIdentity"] != nxt[lid]["gossipIdentity"]: - _fail("ARCHIVE_MERKLE_REBRIDGE_GOSSIP_IDENTITY_CHANGE_FORBIDDEN") - changed = signed["changedLogIds"] - if not isinstance(changed, list): - _fail("ARCHIVE_MERKLE_REBRIDGE_CHANGED_SET_INVALID") - changed = [_id(x, "ARCHIVE_MERKLE_REBRIDGE_CHANGED_ID_INVALID") for x in changed] - computed = sorted(lid for lid in expected_log_ids if current[lid] != nxt[lid]) - if ( - changed != sorted(changed) - or len(set(changed)) != len(changed) - or changed != computed - or not changed - ): - _fail("ARCHIVE_MERKLE_REBRIDGE_CHANGED_SET_INVALID") - compromised = signed["compromisedKeyFingerprints"] - revoked = signed["revokedKeyFingerprints"] - if not isinstance(compromised, list) or not isinstance(revoked, list): - _fail("ARCHIVE_MERKLE_REBRIDGE_REVOCATION_LIST_INVALID") - compromised = [ - _hex(x, "ARCHIVE_MERKLE_REBRIDGE_COMPROMISED_INVALID") for x in compromised - ] - revoked = [_hex(x, "ARCHIVE_MERKLE_REBRIDGE_REVOKED_INVALID") for x in revoked] - if ( - compromised != sorted(compromised) - or len(set(compromised)) != len(compromised) - or revoked != sorted(revoked) - or len(set(revoked)) != len(revoked) - ): - _fail("ARCHIVE_MERKLE_REBRIDGE_REVOCATION_LIST_INVALID") - previous_revoked = list(source["revoked"]) - current_fps = _authority_fps(current) - next_fps = _authority_fps(nxt) - if set(previous_revoked) & next_fps: - _fail("ARCHIVE_MERKLE_REBRIDGE_REVOKED_KEY_REINTRODUCED") - if kind == "scheduled-rotation": - if compromised: - _fail("ARCHIVE_MERKLE_REBRIDGE_SCHEDULED_COMPROMISED_NOT_EMPTY") - replaced = set() - for lid in changed: - old = current[lid] - new = nxt[lid] - if ( - old["operator"] != new["operator"] - and old["publicKey"] == new["publicKey"] - ): - _fail("ARCHIVE_MERKLE_REBRIDGE_OPERATOR_REBOUND_WITHOUT_KEY_ROTATION") - if ( - old["gossipOperator"] != new["gossipOperator"] - and old["gossipPublicKey"] == new["gossipPublicKey"] - ): - _fail( - "ARCHIVE_MERKLE_REBRIDGE_GOSSIP_OPERATOR_REBOUND_WITHOUT_KEY_ROTATION" - ) - if old["publicKey"] != new["publicKey"]: - replaced.add(_pub_fingerprint(old["publicKey"])) - if old["gossipPublicKey"] != new["gossipPublicKey"]: - replaced.add(_pub_fingerprint(old["gossipPublicKey"])) - if revoked != sorted(set(previous_revoked) | replaced): - _fail("ARCHIVE_MERKLE_REBRIDGE_SCHEDULED_REVOCATION_INVALID") - role = "governance" - root = governance - else: - if not compromised or not set(compromised) <= current_fps: - _fail("ARCHIVE_MERKLE_REBRIDGE_RECOVERY_COMPROMISED_INVALID") - affected = sorted( - lid - for lid in expected_log_ids - if { - _pub_fingerprint(current[lid]["publicKey"]), - _pub_fingerprint(current[lid]["gossipPublicKey"]), - } - & set(compromised) - ) - if affected != changed: - _fail("ARCHIVE_MERKLE_REBRIDGE_RECOVERY_CHANGED_SET_INVALID") - replaced = set() - for lid in changed: - old = current[lid] - new = nxt[lid] - if ( - old["publicKey"] == new["publicKey"] - or old["gossipPublicKey"] == new["gossipPublicKey"] - ): - _fail("ARCHIVE_MERKLE_REBRIDGE_RECOVERY_REQUIRES_FULL_KEY_ROTATION") - replaced.add(_pub_fingerprint(old["publicKey"])) - replaced.add(_pub_fingerprint(old["gossipPublicKey"])) - if revoked != sorted(set(previous_revoked) | replaced): - _fail("ARCHIVE_MERKLE_REBRIDGE_RECOVERY_REVOCATION_INVALID") - role = "recovery" - root = recovery - if set(revoked) & next_fps: - _fail("ARCHIVE_MERKLE_REBRIDGE_REVOKED_KEY_ACTIVE") - if signed["authorizationRole"] != role: - _fail("ARCHIVE_MERKLE_REBRIDGE_AUTHORIZATION_ROLE_INVALID") - selected = signed["selectedSignerKeyIds"] - if not isinstance(selected, list): - _fail("ARCHIVE_MERKLE_REBRIDGE_SELECTED_SIGNERS_INVALID") - selected = [ - _id(x, "ARCHIVE_MERKLE_REBRIDGE_SELECTED_SIGNERS_INVALID") for x in selected - ] - if ( - selected != sorted(selected) - or len(selected) != root["threshold"] - or len(set(selected)) != len(selected) - or any(x not in root["keys"] for x in selected) - ): - _fail("ARCHIVE_MERKLE_REBRIDGE_SELECTED_SIGNERS_INVALID") - if role == "recovery": - if len({root["keys"][k]["operator"] for k in selected}) < int( - run165.POLICY["min_recovery_operators"] - ): - _fail("ARCHIVE_MERKLE_REBRIDGE_RECOVERY_OPERATOR_QUORUM_INVALID") - if len({root["keys"][k]["recoveryChannel"] for k in selected}) < int( - run165.POLICY["min_recovery_channels"] - ): - _fail("ARCHIVE_MERKLE_REBRIDGE_RECOVERY_CHANNEL_QUORUM_INVALID") - elif len({root["keys"][k]["operator"] for k in selected}) < int( - run165.POLICY["min_governance_operators"] - ): - _fail("ARCHIVE_MERKLE_REBRIDGE_GOVERNANCE_OPERATOR_QUORUM_INVALID") - if issued < root["issued"] or issued >= root["expires"]: - _fail("ARCHIVE_MERKLE_REBRIDGE_TRANSITION_OUTSIDE_ROOT_LIFETIME") - for kid in selected: - if issued >= _dt( - root["keys"][kid]["expires"], - "ARCHIVE_MERKLE_REBRIDGE_SIGNER_EXPIRES_INVALID", - ): - _fail("ARCHIVE_MERKLE_REBRIDGE_SIGNER_EXPIRED") - signatures = doc["signatures"] - if not isinstance(signatures, list) or len(signatures) != len(selected): - _fail("ARCHIVE_MERKLE_REBRIDGE_SIGNATURE_SET_INVALID") - sigmap = {} - for item in signatures: - if not isinstance(item, dict) or set(item) != {"keyId", "signature"}: - _fail("ARCHIVE_MERKLE_REBRIDGE_SIGNATURE_SCHEMA_INVALID") - kid = _id(item["keyId"], "ARCHIVE_MERKLE_REBRIDGE_SIGNATURE_KEY_INVALID") - if kid in sigmap: - _fail("ARCHIVE_MERKLE_REBRIDGE_SIGNATURE_DUPLICATE") - sigmap[kid] = item["signature"] - if sorted(sigmap) != selected or signatures != sorted( - signatures, key=lambda x: x["keyId"] - ): - _fail("ARCHIVE_MERKLE_REBRIDGE_SIGNATURE_SET_INVALID") - message = _canonical(signed) - for kid in selected: - _verify_sig( - root["keys"][kid]["publicKey"], - sigmap[kid], - message, - "ARCHIVE_MERKLE_REBRIDGE_TRANSITION_SIGNATURE", - ) - subjects = { - lid: _handoff_subject( - signed=signed, log_id=lid, checkpoint=source["checkpoints"][lid] - ) - for lid in sorted(expected_log_ids) - } - expected_hashes = { - lid: _sha_bytes(_canonical(subjects[lid])) for lid in sorted(subjects) - } - if signed["handoffSubjectSha256s"] != expected_hashes: - _fail("ARCHIVE_MERKLE_REBRIDGE_HANDOFF_SUBJECT_HASH_INVALID") - return { - "signed": signed, - "transitionSha256": _sha_bytes(_canonical(doc)), - "subjects": subjects, - "current": current, - "next": nxt, - "revoked": revoked, - "kind": kind, - "rebridgeSequence": rbseq, - "issued": issued, - "transitionId": tid, - } - - -def _verify_handoff( - doc: dict[str, Any], *, transition: dict[str, Any] -) -> dict[str, Any]: - signed = transition["signed"] - if set(doc) != { - "schemaVersion", - "transitionId", - "rebridgeSequence", - "proofs", - } or doc.get("schemaVersion") != int(POLICY["handoff_schema_version"]): - _fail("ARCHIVE_MERKLE_REBRIDGE_HANDOFF_SCHEMA_INVALID") - if ( - doc.get("transitionId") != signed["transitionId"] - or doc.get("rebridgeSequence") != signed["rebridgeSequence"] - ): - _fail("ARCHIVE_MERKLE_REBRIDGE_HANDOFF_BINDING_INVALID") - proofs = doc["proofs"] - if ( - not isinstance(proofs, list) - or len(proofs) != len(transition["subjects"]) - or proofs != sorted(proofs, key=lambda x: x.get("logId", "")) - ): - _fail("ARCHIVE_MERKLE_REBRIDGE_HANDOFF_PROOF_SET_INVALID") - seen = set() - for proof in proofs: - expected = { - "logId", - "subjectSha256", - "newLogSignature", - "newGossipSignature", - "oldLogSignature", - "oldGossipSignature", - } - if not isinstance(proof, dict) or set(proof) != expected: - _fail("ARCHIVE_MERKLE_REBRIDGE_HANDOFF_PROOF_SCHEMA_INVALID") - lid = _id(proof["logId"], "ARCHIVE_MERKLE_REBRIDGE_HANDOFF_LOG_ID_INVALID") - if lid in seen or lid not in transition["subjects"]: - _fail("ARCHIVE_MERKLE_REBRIDGE_HANDOFF_LOG_ID_INVALID") - seen.add(lid) - subject = transition["subjects"][lid] - raw = _canonical(subject) - if proof["subjectSha256"] != _sha_bytes(raw): - _fail("ARCHIVE_MERKLE_REBRIDGE_HANDOFF_SUBJECT_BINDING_INVALID") - cur = transition["current"][lid] - nxt = transition["next"][lid] - _verify_sig( - nxt["publicKey"], - proof["newLogSignature"], - raw, - "ARCHIVE_MERKLE_REBRIDGE_NEW_LOG_HANDOFF", - ) - _verify_sig( - nxt["gossipPublicKey"], - proof["newGossipSignature"], - raw, - "ARCHIVE_MERKLE_REBRIDGE_NEW_GOSSIP_HANDOFF", - ) - changed = lid in signed["changedLogIds"] - if transition["kind"] == "scheduled-rotation" and changed: - if proof["oldLogSignature"] is None or proof["oldGossipSignature"] is None: - _fail("ARCHIVE_MERKLE_REBRIDGE_OLD_HANDOFF_REQUIRED") - _verify_sig( - cur["publicKey"], - proof["oldLogSignature"], - raw, - "ARCHIVE_MERKLE_REBRIDGE_OLD_LOG_HANDOFF", - ) - _verify_sig( - cur["gossipPublicKey"], - proof["oldGossipSignature"], - raw, - "ARCHIVE_MERKLE_REBRIDGE_OLD_GOSSIP_HANDOFF", - ) - elif ( - proof["oldLogSignature"] is not None - or proof["oldGossipSignature"] is not None - ): - _fail("ARCHIVE_MERKLE_REBRIDGE_OLD_HANDOFF_FORBIDDEN") - return doc - - -def _log_challenge( - *, - sequence: int, - run163_head: str, - leaf_hash: str, - previous_merkle_head: str, - authority_bridge_head: str, - active_authority_sha: str, - source_state_sha: str, -) -> str: - return _sha_bytes( - _canonical( - { - "sequence": sequence, - "run163AnchorConsensusHeadSha256": run163_head, - "leafHash": leaf_hash, - "previousArchiveMerkleRebridgeContinuityHeadSha256": ( - previous_merkle_head - ), - "archiveMerkleRebridgeAuthorityHeadSha256": authority_bridge_head, - "activeAuthoritySha256": active_authority_sha, - "sourceTrustedStateSha256": source_state_sha, - } - ) - ) - - -def _gossip_challenge( - *, - sequence: int, - run163_head: str, - leaf_hash: str, - checkpoints: dict[str, str], - authority_bridge_head: str, - active_authority_sha: str, - source_state_sha: str, -) -> str: - return _sha_bytes( - _canonical( - { - "sequence": sequence, - "run163AnchorConsensusHeadSha256": run163_head, - "leafHash": leaf_hash, - "checkpointSha256s": checkpoints, - "archiveMerkleRebridgeAuthorityHeadSha256": authority_bridge_head, - "activeAuthoritySha256": active_authority_sha, - "sourceTrustedStateSha256": source_state_sha, - } - ) - ) - - -def _normalize_merkle_error(prefix: str, fn, *args, **kwargs): - try: - return fn(*args, **kwargs) - except ArchiveMerkleRebridgeError: - raise - except Exception as exc: - raise ArchiveMerkleRebridgeError(prefix + ":" + str(exc)) from exc - - -def _load_output(root: Path) -> tuple[dict[str, Any], dict[str, bytes]]: - root = _regular_dir(root, "ARCHIVE_MERKLE_REBRIDGE_OUTPUT_INVALID") - if {p.name for p in root.iterdir()} != _OUTPUT_NAMES: - _fail("ARCHIVE_MERKLE_REBRIDGE_OUTPUT_ALLOWLIST_INVALID") - docs = {} - raws = {} - for name in sorted(_OUTPUT_NAMES): - docs[name], raws[name] = _read_json( - root / name, "ARCHIVE_MERKLE_REBRIDGE_OUTPUT" - ) - return docs, raws - - -def _bundle_base( - *, - run166_info: dict[str, Any], - run165_info: dict[str, Any], - governance: dict[str, Any], - recovery: dict[str, Any], -) -> dict[str, Any]: - return { - "schemaVersion": int(POLICY["bundle_schema_version"]), - "predicateType": PREDICATE_TYPE, - "status": "archive-merkle-authority-rebridge-history", - "baseRun166Artifacts": run166_info["artifacts"], - "baseRun166Sequence": run166_info["state"]["sequence"], - "baseRun166MerkleAuthorityContinuityHeadSha256": run166_info["state"][ - "merkleAuthorityContinuityHeadSha256" - ], - "run165Artifacts": run165_info["artifacts"], - "run165Sequence": run165_info["state"]["sequence"], - "run165LogAuthorityChainHeadSha256": run165_info["state"][ - "logAuthorityChainHeadSha256" - ], - "governanceRootSha256": governance["sha256"], - "recoveryRootSha256": recovery["sha256"], - } - - -def _make_state( - *, - base: dict[str, Any], - events: list[dict[str, Any]], - active_authority: dict[str, Any], - revoked: list[str], - rebridge_sequence: int, - authority_head: str, - merkle_head: str, - checkpoints: dict[str, Any], - last_transition_issued: datetime | None, -) -> dict[str, Any]: - bundle = dict(base) - bundle["events"] = events - braw = _canonical(bundle) - last = events[-1] - return { - "schemaVersion": int(POLICY["state_schema_version"]), - "status": "trusted-archive-merkle-rebridge", - "sequence": last["sequence"], - "postRun166Sequence": len(events), - "rebridgeSequence": rebridge_sequence, - "baseRun166Sequence": base["baseRun166Sequence"], - "run165Sequence": base["run165Sequence"], - "run165LogAuthorityChainHeadSha256": base["run165LogAuthorityChainHeadSha256"], - "governanceRootSha256": base["governanceRootSha256"], - "recoveryRootSha256": base["recoveryRootSha256"], - "archiveMerkleRebridgeAuthorityHeadSha256": authority_head, - "activeAuthority": active_authority, - "activeAuthoritySha256": _authority_sha(active_authority), - "revokedKeyFingerprints": revoked, - "merkleRebridgeContinuityHeadSha256": merkle_head, - "lastRun163AnchorConsensusHeadSha256": last["run163AnchorConsensusHeadSha256"], - "lastCheckpoints": { - lid: { - k: checkpoints[lid][k] - for k in ("checkpointSha256", "treeSize", "rootHash") - } - for lid in sorted(checkpoints) - }, - "lastTransitionIssuedAt": ( - None if last_transition_issued is None else _ts(last_transition_issued) - ), - "bundleArtifact": { - "name": "release-archive-merkle-rebridge-bundle.json", - "sha256": _sha_bytes(braw), - "size": len(braw), - }, - } - - -def _make_active(state: dict[str, Any], last: dict[str, Any]) -> dict[str, Any]: - return { - "schemaVersion": int(POLICY["active_schema_version"]), - "status": "active-archive-merkle-rebridge", - "sequence": state["sequence"], - "postRun166Sequence": state["postRun166Sequence"], - "rebridgeSequence": state["rebridgeSequence"], - "run165Sequence": state["run165Sequence"], - "run165LogAuthorityChainHeadSha256": state["run165LogAuthorityChainHeadSha256"], - "archiveMerkleRebridgeAuthorityHeadSha256": state[ - "archiveMerkleRebridgeAuthorityHeadSha256" - ], - "authority": state["activeAuthority"], - "activeAuthoritySha256": state["activeAuthoritySha256"], - "revokedKeyFingerprints": state["revokedKeyFingerprints"], - "merkleRebridgeContinuityHeadSha256": state[ - "merkleRebridgeContinuityHeadSha256" - ], - "run163AnchorConsensusHeadSha256": last["run163AnchorConsensusHeadSha256"], - "leafHash": last["leafHash"], - "logs": last["logs"], - "gossipResponseSha256s": last["gossipResponseSha256s"], - } - - -def _source_from_base( - run166_info: dict[str, Any], run165_info: dict[str, Any] -) -> dict[str, Any]: - raw = run166_info["raws"][_DOC_ARCHIVE_MERKLE_CONTINUITY_STATE] - cps = _normalize_checkpoints( - run166_info["state"]["lastCheckpoints"], - expected_log_ids=set(run166_info["active"]["authority"]), - code="ARCHIVE_MERKLE_REBRIDGE_BASE_CHECKPOINTS", - ) - return { - "sourceKind": "run166", - "stateRaw": raw, - "stateSha256": _sha_bytes(raw), - "merkleSequence": run166_info["state"]["sequence"], - "merkleHead": run166_info["state"]["merkleAuthorityContinuityHeadSha256"], - "checkpoints": cps, - "authority": run166_info["active"]["authority"], - "revoked": list(run165_info["state"]["revokedKeyFingerprints"]), - "rebridgeSequence": 0, - "authorityHead": _base_authority_head(run165_info, run166_info), - "lastTransitionIssued": None, - } - - -def _replay( # ruff: ignore[too-many-branches] - *, - run166_info: dict[str, Any], - run165_info: dict[str, Any], - governance: dict[str, Any], - recovery: dict[str, Any], - run164_info: dict[str, Any], - bundle: dict[str, Any], - receipt: dict[str, Any], - now: datetime, - current_run163_docs: dict[str, Any] | None, - current_run163_raws: dict[str, bytes] | None, - run160_dir: Path, - run161_dir: Path, - retention_root_path: Path, - retention_root_pin: str, - witness_root_path: Path, - witness_root_pin: str, - bootstrap_pin: str, - recovery_pin: str, - attestation_pins: list[str], - run162_dir: Path, - anchor_plan_path: Path, -) -> dict[str, Any]: - base = _bundle_base( - run166_info=run166_info, - run165_info=run165_info, - governance=governance, - recovery=recovery, - ) - expected_bundle_keys = set(base) | {"events"} - if set(bundle) != expected_bundle_keys or any( - bundle[k] != v for k, v in base.items() - ): - _fail("ARCHIVE_MERKLE_REBRIDGE_BUNDLE_BASE_INVALID") - if set(receipt) != { - "schemaVersion", - "status", - "run166Documents", - "run165Documents", - "events", - }: - _fail("ARCHIVE_MERKLE_REBRIDGE_RECEIPT_SCHEMA_INVALID") - if ( - receipt["schemaVersion"] != int(POLICY["receipt_schema_version"]) - or receipt["status"] != "archive-merkle-authority-rebridge-accepted" - ): - _fail("ARCHIVE_MERKLE_REBRIDGE_RECEIPT_SCHEMA_INVALID") - if ( - receipt["run166Documents"] != run166_info["docs"] - or receipt["run165Documents"] != run165_info["docs"] - ): - _fail("ARCHIVE_MERKLE_REBRIDGE_BASE_DOCUMENT_MISMATCH") - events = bundle["events"] - recs = receipt["events"] - if ( - not isinstance(events, list) - or not events - or not isinstance(recs, list) - or len(events) != len(recs) - ): - _fail("ARCHIVE_MERKLE_REBRIDGE_HISTORY_LENGTH_INVALID") - source = _source_from_base(run166_info, run165_info) - seen_ids: set[str] = set() - previous_transition_issued = None - active_authority = source["authority"] - revoked = source["revoked"] - authority_head = source["authorityHead"] - rebridge_sequence = 0 - merkle_head = source["merkleHead"] - checkpoints = source["checkpoints"] - prefix_events: list[dict[str, Any]] = [] - last = None - for idx, (event, rec) in enumerate(zip(events, recs), 1): - sequence = base["baseRun166Sequence"] + idx - expected_event = { - "sequence", - "postRun166Sequence", - "action", - "rebridgeSequence", - "run163Sequence", - "run163AnchorConsensusHeadSha256", - "run163Artifacts", - "leafHash", - "sourceTrustedStateSha256", - "archiveMerkleRebridgeAuthorityHeadSha256", - "activeAuthoritySha256", - "revokedKeyFingerprints", - "transitionSha256", - "handoffSha256", - "logs", - "logChallenge", - "gossipChallenge", - "gossipResponseSha256s", - "merkleRebridgeContinuityHeadSha256", - } - if ( - not isinstance(event, dict) - or set(event) != expected_event - or event.get("sequence") != sequence - or event.get("postRun166Sequence") != idx - ): - _fail("ARCHIVE_MERKLE_REBRIDGE_EVENT_SCHEMA_INVALID") - expected_rec = { - "sequence", - "run163Documents", - "transitionDocument", - "handoffProofDocument", - "logResponses", - "gossipResponses", - } - if ( - not isinstance(rec, dict) - or set(rec) != expected_rec - or rec.get("sequence") != sequence - ): - _fail("ARCHIVE_MERKLE_REBRIDGE_RECEIPT_EVENT_INVALID") - if event["sourceTrustedStateSha256"] != source["stateSha256"]: - _fail("ARCHIVE_MERKLE_REBRIDGE_EVENT_SOURCE_STATE_INVALID") - vr163 = _verify_embedded_run163( - rec["run163Documents"], - run160_dir=run160_dir, - run161_dir=run161_dir, - retention_root_path=retention_root_path, - retention_root_pin=retention_root_pin, - witness_root_path=witness_root_path, - witness_root_pin=witness_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - run162_dir=run162_dir, - anchor_plan_path=anchor_plan_path, - now=now, - ) - if ( - vr163["sequence"] != sequence - or event["run163Sequence"] != sequence - or event["run163AnchorConsensusHeadSha256"] - != vr163["anchor_consensus_head_sha256"] - ): - _fail("ARCHIVE_MERKLE_REBRIDGE_RUN163_BINDING_INVALID") - run163_raws = { - name: _canonical(rec["run163Documents"][name]) - for name in sorted(rec["run163Documents"]) - } - if event["run163Artifacts"] != _artifact_map(run163_raws): - _fail("ARCHIVE_MERKLE_REBRIDGE_RUN163_ARTIFACT_INVALID") - leaf_doc = merkle._leaf_document(rec["run163Documents"], run163_raws) - leaf_hash = merkle.merkle_leaf_hash(_canonical(leaf_doc)) - if event["leafHash"] != leaf_hash: - _fail("ARCHIVE_MERKLE_REBRIDGE_LEAF_INVALID") - action = event["action"] - if idx == 1 and action != "rebridge": - _fail("ARCHIVE_MERKLE_REBRIDGE_FIRST_EVENT_REQUIRES_REBRIDGE") - if action == "rebridge": - if rec["transitionDocument"] is None or rec["handoffProofDocument"] is None: - _fail("ARCHIVE_MERKLE_REBRIDGE_TRANSITION_MISSING") - tr = _verify_transition( - rec["transitionDocument"], - source=source, - governance=governance, - recovery=recovery, - expected_log_ids=set(active_authority), - seen_ids=seen_ids, - previous_transition_issued=previous_transition_issued, - now=now, - creation=False, - ) - hf = _verify_handoff(rec["handoffProofDocument"], transition=tr) - tsha = tr["transitionSha256"] - hsha = _sha_bytes(_canonical(hf)) - if event["transitionSha256"] != tsha or event["handoffSha256"] != hsha: - _fail("ARCHIVE_MERKLE_REBRIDGE_TRANSITION_ARTIFACT_BINDING_INVALID") - authority_head = _bridge_head( - authority_head, tsha, hsha, source["stateSha256"] - ) - active_authority = tr["next"] - revoked = tr["revoked"] - rebridge_sequence = tr["rebridgeSequence"] - previous_transition_issued = tr["issued"] - seen_ids.add(tr["transitionId"]) - elif action == "append": - if ( - rec["transitionDocument"] is not None - or rec["handoffProofDocument"] is not None - or event["transitionSha256"] is not None - or event["handoffSha256"] is not None - ): - _fail("ARCHIVE_MERKLE_REBRIDGE_APPEND_TRANSITION_FORBIDDEN") - else: - _fail("ARCHIVE_MERKLE_REBRIDGE_ACTION_INVALID") - if ( - event["rebridgeSequence"] != rebridge_sequence - or event["archiveMerkleRebridgeAuthorityHeadSha256"] != authority_head - ): - _fail("ARCHIVE_MERKLE_REBRIDGE_AUTHORITY_HEAD_BINDING_INVALID") - active_sha = _authority_sha(active_authority) - if ( - event["activeAuthoritySha256"] != active_sha - or event["revokedKeyFingerprints"] != revoked - ): - _fail("ARCHIVE_MERKLE_REBRIDGE_ACTIVE_AUTHORITY_BINDING_INVALID") - _enforce_separation( - governance, - recovery, - run164_info["root"], - witness_root_path, - anchor_plan_path, - active_authority, - ) - log_challenge = _log_challenge( - sequence=sequence, - run163_head=event["run163AnchorConsensusHeadSha256"], - leaf_hash=leaf_hash, - previous_merkle_head=merkle_head, - authority_bridge_head=authority_head, - active_authority_sha=active_sha, - source_state_sha=source["stateSha256"], - ) - if event["logChallenge"] != log_challenge: - _fail("ARCHIVE_MERKLE_REBRIDGE_LOG_CHALLENGE_INVALID") - log_docs = rec["logResponses"] - if not isinstance(log_docs, dict) or set(log_docs) != set(active_authority): - _fail("ARCHIVE_MERKLE_REBRIDGE_LOG_RESPONSE_SET_INVALID") - rows = [] - next_checkpoints = {} - for lid in sorted(active_authority): - prev = { - "logId": lid, - **checkpoints[lid], - "leafIndex": checkpoints[lid]["treeSize"] - 1, - "leafHash": None, - } - parsed = _normalize_merkle_error( - "ARCHIVE_MERKLE_REBRIDGE_CHECKPOINT_INVALID", - merkle._verify_checkpoint, - log_docs[lid], - log_id=lid, - config=active_authority[lid], - sequence=sequence, - challenge=log_challenge, - expected_leaf_hash=leaf_hash, - previous=prev, - now=now, - creation=False, - historical=True, - ) - row = { - k: parsed[k] - for k in ( - "logId", - "checkpointSha256", - "treeSize", - "rootHash", - "leafIndex", - "leafHash", - ) - } - rows.append(row) - next_checkpoints[lid] = { - k: row[k] for k in ("checkpointSha256", "treeSize", "rootHash") - } - if event["logs"] != rows: - _fail("ARCHIVE_MERKLE_REBRIDGE_LOG_ROWS_INVALID") - cp_hashes = {row["logId"]: row["checkpointSha256"] for row in rows} - gossip_challenge = _gossip_challenge( - sequence=sequence, - run163_head=event["run163AnchorConsensusHeadSha256"], - leaf_hash=leaf_hash, - checkpoints=cp_hashes, - authority_bridge_head=authority_head, - active_authority_sha=active_sha, - source_state_sha=source["stateSha256"], - ) - if event["gossipChallenge"] != gossip_challenge: - _fail("ARCHIVE_MERKLE_REBRIDGE_GOSSIP_CHALLENGE_INVALID") - gossip_docs = rec["gossipResponses"] - if not isinstance(gossip_docs, dict) or set(gossip_docs) != set( - active_authority - ): - _fail("ARCHIVE_MERKLE_REBRIDGE_GOSSIP_RESPONSE_SET_INVALID") - gossip_hashes = [] - for lid in sorted(active_authority): - parsed = _normalize_merkle_error( - "ARCHIVE_MERKLE_REBRIDGE_GOSSIP_INVALID", - merkle._verify_gossip, - gossip_docs[lid], - log_id=lid, - config=active_authority[lid], - sequence=sequence, - challenge=gossip_challenge, - run163_head=event["run163AnchorConsensusHeadSha256"], - checkpoints=cp_hashes, - now=now, - creation=False, - historical=True, - ) - gossip_hashes.append(parsed["gossipSha256"]) - if event["gossipResponseSha256s"] != gossip_hashes: - _fail("ARCHIVE_MERKLE_REBRIDGE_GOSSIP_HASHES_INVALID") - bare = {k: event[k] for k in event if k != "merkleRebridgeContinuityHeadSha256"} - new_merkle_head = _merkle_head(merkle_head, bare) - if event["merkleRebridgeContinuityHeadSha256"] != new_merkle_head: - _fail("ARCHIVE_MERKLE_REBRIDGE_CHAIN_HEAD_INVALID") - merkle_head = new_merkle_head - checkpoints = next_checkpoints - last = event - prefix_events.append(event) - prefix_state = _make_state( - base=base, - events=prefix_events, - active_authority=active_authority, - revoked=revoked, - rebridge_sequence=rebridge_sequence, - authority_head=authority_head, - merkle_head=merkle_head, - checkpoints=checkpoints, - last_transition_issued=previous_transition_issued, - ) - source = { - "sourceKind": "run167", - "stateRaw": _canonical(prefix_state), - "stateSha256": _sha_bytes(_canonical(prefix_state)), - "merkleSequence": sequence, - "merkleHead": merkle_head, - "checkpoints": checkpoints, - "authority": active_authority, - "revoked": revoked, - "rebridgeSequence": rebridge_sequence, - "authorityHead": authority_head, - "lastTransitionIssued": previous_transition_issued, - } - if current_run163_docs is not None: - latest_docs = receipt["events"][-1]["run163Documents"] - if ( - latest_docs != current_run163_docs - or {n: _canonical(latest_docs[n]) for n in sorted(latest_docs)} - != current_run163_raws - ): - _fail("ARCHIVE_MERKLE_REBRIDGE_CURRENT_RUN163_MISMATCH") - return { - "sequence": last["sequence"], - "postRun166Sequence": len(events), - "rebridgeSequence": rebridge_sequence, - "authority": active_authority, - "revoked": revoked, - "authorityHead": authority_head, - "merkleHead": merkle_head, - "checkpoints": checkpoints, - "last": last, - "lastTransitionIssued": previous_transition_issued, - "state": prefix_state, - "source": source, - } - - -def verify_archive_merkle_rebridge( # ruff: ignore[undocumented-public-function] - *, - run160_dir: Path, - run166_run161_dir: Path, - run161_dir: Path, - retention_root_path: Path, - retention_root_pin: str, - witness_root_path: Path, - witness_root_pin: str, - bootstrap_pin: str, - recovery_pin: str, - attestation_pins: list[str], - run166_run162_dir: Path, - run162_dir: Path, - anchor_plan_path: Path, - run166_run163_dir: Path, - run163_dir: Path, - run164_dir: Path, - transparency_root_path: Path, - transparency_root_pin: str, - governance_root_path: Path, - governance_root_pin: str, - recovery_root_path: Path, - recovery_root_pin: str, - run165_dir: Path, - run166_dir: Path, - output_dir: Path, - now: datetime | None = None, - historical: bool = False, -) -> dict[str, Any]: - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - run164_info = _load_run164( - run164_dir, transparency_root_path, transparency_root_pin, now=current - ) - run165_info = _load_run165( - run164_dir=run164_dir, - transparency_root_path=transparency_root_path, - transparency_root_pin=transparency_root_pin, - governance_root_path=governance_root_path, - governance_root_pin=governance_root_pin, - recovery_root_path=recovery_root_path, - recovery_root_pin=recovery_root_pin, - run165_dir=run165_dir, - now=current, - ) - run166_info = _load_run166( - run160_dir=run160_dir, - run166_run161_dir=run166_run161_dir, - retention_root_path=retention_root_path, - retention_root_pin=retention_root_pin, - witness_root_path=witness_root_path, - witness_root_pin=witness_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - run166_run162_dir=run166_run162_dir, - anchor_plan_path=anchor_plan_path, - run166_run163_dir=run166_run163_dir, - run164_dir=run164_dir, - transparency_root_path=transparency_root_path, - transparency_root_pin=transparency_root_pin, - governance_root_path=governance_root_path, - governance_root_pin=governance_root_pin, - recovery_root_path=recovery_root_path, - recovery_root_pin=recovery_root_pin, - run165_dir=run165_dir, - run166_dir=run166_dir, - now=current, - ) - governance = _verify_control_root( - governance_root_path, - governance_root_pin, - recovery=False, - now=current, - historical=True, - ) - recovery = _verify_control_root( - recovery_root_path, - recovery_root_pin, - recovery=True, - now=current, - historical=True, - ) - current163 = _load_current_run163( - run160_dir=run160_dir, - run161_dir=run161_dir, - retention_root_path=retention_root_path, - retention_root_pin=retention_root_pin, - witness_root_path=witness_root_path, - witness_root_pin=witness_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - run162_dir=run162_dir, - anchor_plan_path=anchor_plan_path, - run163_dir=run163_dir, - now=current, - historical=historical, - ) - docs, _raws = _load_output(output_dir) - replay = _replay( - run166_info=run166_info, - run165_info=run165_info, - governance=governance, - recovery=recovery, - run164_info=run164_info, - bundle=docs["release-archive-merkle-rebridge-bundle.json"], - receipt=docs["release-archive-merkle-rebridge-receipt.json"], - now=current, - current_run163_docs=current163["docs"], - current_run163_raws=current163["raws"], - run160_dir=run160_dir, - run161_dir=run161_dir, - retention_root_path=retention_root_path, - retention_root_pin=retention_root_pin, - witness_root_path=witness_root_path, - witness_root_pin=witness_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - run162_dir=run162_dir, - anchor_plan_path=anchor_plan_path, - ) - if docs[_DOC_ARCHIVE_MERKLE_REBRIDGE_STATE] != replay["state"]: - _fail("ARCHIVE_MERKLE_REBRIDGE_STATE_MISMATCH") - if docs["active-archive-merkle-rebridge.json"] != _make_active( - replay["state"], replay["last"] - ): - _fail("ARCHIVE_MERKLE_REBRIDGE_ACTIVE_MISMATCH") - if not historical: - if replay["lastTransitionIssued"] is not None and current - replay[ - "lastTransitionIssued" - ] > timedelta(days=int(POLICY["max_active_authority_age_days"])): - _fail("ARCHIVE_MERKLE_REBRIDGE_ACTIVE_AUTHORITY_STALE") - latest = docs["release-archive-merkle-rebridge-receipt.json"]["events"][-1] - max_age = timedelta(days=int(POLICY["max_active_epoch_age_days"])) - for lid in sorted(replay["authority"]): - integrated = merkle._dt( - latest["logResponses"][lid]["signed"]["integratedAt"], - "ARCHIVE_MERKLE_REBRIDGE_ACTIVE_TIME_INVALID", - ) - observed = merkle._dt( - latest["gossipResponses"][lid]["signed"]["observedAt"], - "ARCHIVE_MERKLE_REBRIDGE_ACTIVE_TIME_INVALID", - ) - if current - integrated > max_age or current - observed > max_age: - _fail("ARCHIVE_MERKLE_REBRIDGE_ACTIVE_EPOCH_STALE") - return { - "ok": True, - "sequence": replay["sequence"], - "post_run166_sequence": replay["postRun166Sequence"], - "rebridge_sequence": replay["rebridgeSequence"], - "archive_merkle_rebridge_authority_head_sha256": replay["authorityHead"], - "merkle_rebridge_continuity_head_sha256": replay["merkleHead"], - } - - -def advance_archive_merkle_rebridge( # ruff: ignore[too-many-branches, undocumented-public-function] - *, - run160_dir: Path, - run166_run161_dir: Path, - run161_dir: Path, - retention_root_path: Path, - retention_root_pin: str, - witness_root_path: Path, - witness_root_pin: str, - bootstrap_pin: str, - recovery_pin: str, - attestation_pins: list[str], - run166_run162_dir: Path, - run162_dir: Path, - anchor_plan_path: Path, - run166_run163_dir: Path, - run163_dir: Path, - run164_dir: Path, - transparency_root_path: Path, - transparency_root_pin: str, - governance_root_path: Path, - governance_root_pin: str, - recovery_root_path: Path, - recovery_root_pin: str, - run165_dir: Path, - run166_dir: Path, - output_dir: Path, - adapters: list[ - tuple[ - str, - Callable[[dict[str, Any]], dict[str, Any]], - Callable[[dict[str, Any]], dict[str, Any]], - ] - ], - transition_path: Path | None = None, - handoff_path: Path | None = None, - previous_output_dir: Path | None = None, - now: datetime | None = None, -) -> dict[str, Any]: - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - if (transition_path is None) != (handoff_path is None): - _fail("ARCHIVE_MERKLE_REBRIDGE_TRANSITION_PAIR_REQUIRED") - authority_paths = [ - Path(run160_dir), - Path(run166_run161_dir), - Path(run161_dir), - Path(retention_root_path), - Path(witness_root_path), - Path(run166_run162_dir), - Path(run162_dir), - Path(anchor_plan_path), - Path(run166_run163_dir), - Path(run163_dir), - Path(run164_dir), - Path(transparency_root_path), - Path(governance_root_path), - Path(recovery_root_path), - Path(run165_dir), - Path(run166_dir), - ] - if transition_path is not None: - authority_paths += [Path(transition_path), Path(handoff_path)] - if previous_output_dir is not None: - authority_paths.append(Path(previous_output_dir)) - before = _authority_fingerprint( - authority_paths, "ARCHIVE_MERKLE_REBRIDGE_INPUT_DRIFT" - ) - run164_info = _load_run164( - run164_dir, transparency_root_path, transparency_root_pin, now=current - ) - run165_info = _load_run165( - run164_dir=run164_dir, - transparency_root_path=transparency_root_path, - transparency_root_pin=transparency_root_pin, - governance_root_path=governance_root_path, - governance_root_pin=governance_root_pin, - recovery_root_path=recovery_root_path, - recovery_root_pin=recovery_root_pin, - run165_dir=run165_dir, - now=current, - ) - run166_info = _load_run166( - run160_dir=run160_dir, - run166_run161_dir=run166_run161_dir, - retention_root_path=retention_root_path, - retention_root_pin=retention_root_pin, - witness_root_path=witness_root_path, - witness_root_pin=witness_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - run166_run162_dir=run166_run162_dir, - anchor_plan_path=anchor_plan_path, - run166_run163_dir=run166_run163_dir, - run164_dir=run164_dir, - transparency_root_path=transparency_root_path, - transparency_root_pin=transparency_root_pin, - governance_root_path=governance_root_path, - governance_root_pin=governance_root_pin, - recovery_root_path=recovery_root_path, - recovery_root_pin=recovery_root_pin, - run165_dir=run165_dir, - run166_dir=run166_dir, - now=current, - ) - governance = _verify_control_root( - governance_root_path, - governance_root_pin, - recovery=False, - now=current, - historical=True, - ) - recovery = _verify_control_root( - recovery_root_path, - recovery_root_pin, - recovery=True, - now=current, - historical=True, - ) - current163 = _load_current_run163( - run160_dir=run160_dir, - run161_dir=run161_dir, - retention_root_path=retention_root_path, - retention_root_pin=retention_root_pin, - witness_root_path=witness_root_path, - witness_root_pin=witness_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - run162_dir=run162_dir, - anchor_plan_path=anchor_plan_path, - run163_dir=run163_dir, - now=current, - historical=False, - ) - base = _bundle_base( - run166_info=run166_info, - run165_info=run165_info, - governance=governance, - recovery=recovery, - ) - old_events: list[dict[str, Any]] = [] - old_receipts: list[dict[str, Any]] = [] - source = _source_from_base(run166_info, run165_info) - seen_ids: set[str] = set() - previous_transition_issued = None - if previous_output_dir is None: - if transition_path is None: - _fail("ARCHIVE_MERKLE_REBRIDGE_FIRST_EVENT_REQUIRES_TRANSITION") - else: - pdocs, _ = _load_output(Path(previous_output_dir)) - previous = _replay( - run166_info=run166_info, - run165_info=run165_info, - governance=governance, - recovery=recovery, - run164_info=run164_info, - bundle=pdocs["release-archive-merkle-rebridge-bundle.json"], - receipt=pdocs["release-archive-merkle-rebridge-receipt.json"], - now=current, - current_run163_docs=None, - current_run163_raws=None, - run160_dir=run160_dir, - run161_dir=run161_dir, - retention_root_path=retention_root_path, - retention_root_pin=retention_root_pin, - witness_root_path=witness_root_path, - witness_root_pin=witness_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - run162_dir=run162_dir, - anchor_plan_path=anchor_plan_path, - ) - source = previous["source"] - old_events = list( - pdocs["release-archive-merkle-rebridge-bundle.json"]["events"] - ) - old_receipts = list( - pdocs["release-archive-merkle-rebridge-receipt.json"]["events"] - ) - for rec in old_receipts: - if rec["transitionDocument"] is not None: - seen_ids.add(rec["transitionDocument"]["signed"]["transitionId"]) - previous_transition_issued = previous["lastTransitionIssued"] - state163 = current163["docs"][_DOC_ARCHIVE_ANCHOR_STATE] - sequence = state163.get("sequence") - if not isinstance(sequence, int) or sequence != source["merkleSequence"] + 1: - _fail("ARCHIVE_MERKLE_REBRIDGE_SEQUENCE_INVALID") - active_authority = source["authority"] - revoked = list(source["revoked"]) - authority_head = source["authorityHead"] - rebridge_sequence = source["rebridgeSequence"] - transition_doc = handoff_doc = None - transition_sha = handoff_sha = None - action = "append" - if transition_path is not None: - transition_doc, _ = _read_json( - Path(transition_path), "ARCHIVE_MERKLE_REBRIDGE_TRANSITION" - ) - tr = _verify_transition( - transition_doc, - source=source, - governance=governance, - recovery=recovery, - expected_log_ids=set(active_authority), - seen_ids=seen_ids, - previous_transition_issued=previous_transition_issued, - now=current, - creation=True, - ) - handoff_doc, _ = _read_json( - Path(handoff_path), "ARCHIVE_MERKLE_REBRIDGE_HANDOFF" - ) - hf = _verify_handoff(handoff_doc, transition=tr) - transition_sha = tr["transitionSha256"] - handoff_sha = _sha_bytes(_canonical(hf)) - authority_head = _bridge_head( - authority_head, transition_sha, handoff_sha, source["stateSha256"] - ) - active_authority = tr["next"] - revoked = tr["revoked"] - rebridge_sequence = tr["rebridgeSequence"] - previous_transition_issued = tr["issued"] - action = "rebridge" - _enforce_separation( - governance, - recovery, - run164_info["root"], - witness_root_path, - anchor_plan_path, - active_authority, - ) - if {lid for lid, _, _ in adapters} != set(active_authority) or len(adapters) != len( - active_authority - ): - _fail("ARCHIVE_MERKLE_REBRIDGE_ADAPTER_SET_INVALID") - amap = {lid: (log, gossip) for lid, log, gossip in adapters} - leaf_doc = merkle._leaf_document(current163["docs"], current163["raws"]) - leaf_hash = merkle.merkle_leaf_hash(_canonical(leaf_doc)) - active_sha = _authority_sha(active_authority) - log_challenge = _log_challenge( - sequence=sequence, - run163_head=state163["anchorConsensusHeadSha256"], - leaf_hash=leaf_hash, - previous_merkle_head=source["merkleHead"], - authority_bridge_head=authority_head, - active_authority_sha=active_sha, - source_state_sha=source["stateSha256"], - ) - log_docs = {} - rows = [] - next_checkpoints = {} - for lid in sorted(active_authority): - prev = { - "logId": lid, - **source["checkpoints"][lid], - "leafIndex": source["checkpoints"][lid]["treeSize"] - 1, - "leafHash": None, - } - request = { - "operation": "append-merkle-leaf-rebridged-authority", - "protocolVersion": 1, - "logId": lid, - "sequence": sequence, - "challenge": log_challenge, - "leaf": leaf_doc, - "leafHash": leaf_hash, - "previousTreeSize": prev["treeSize"], - "previousRootHash": prev["rootHash"], - "archiveMerkleRebridgeAuthorityHeadSha256": authority_head, - "activeAuthoritySha256": active_sha, - "sourceTrustedStateSha256": source["stateSha256"], - } - doc = amap[lid][0](request) - parsed = _normalize_merkle_error( - "ARCHIVE_MERKLE_REBRIDGE_CHECKPOINT_INVALID", - merkle._verify_checkpoint, - doc, - log_id=lid, - config=active_authority[lid], - sequence=sequence, - challenge=log_challenge, - expected_leaf_hash=leaf_hash, - previous=prev, - now=current, - creation=True, - historical=False, - ) - row = { - k: parsed[k] - for k in ( - "logId", - "checkpointSha256", - "treeSize", - "rootHash", - "leafIndex", - "leafHash", - ) - } - log_docs[lid] = doc - rows.append(row) - next_checkpoints[lid] = { - k: row[k] for k in ("checkpointSha256", "treeSize", "rootHash") - } - cp_hashes = {row["logId"]: row["checkpointSha256"] for row in rows} - gossip_challenge = _gossip_challenge( - sequence=sequence, - run163_head=state163["anchorConsensusHeadSha256"], - leaf_hash=leaf_hash, - checkpoints=cp_hashes, - authority_bridge_head=authority_head, - active_authority_sha=active_sha, - source_state_sha=source["stateSha256"], - ) - gossip_docs = {} - gossip_hashes = [] - for lid in sorted(active_authority): - request = { - "operation": "gossip-merkle-rebridged-authority", - "protocolVersion": 1, - "sequence": sequence, - "challenge": gossip_challenge, - "run163AnchorConsensusHeadSha256": state163["anchorConsensusHeadSha256"], - "leafHash": leaf_hash, - "checkpointSha256s": cp_hashes, - "archiveMerkleRebridgeAuthorityHeadSha256": authority_head, - "activeAuthoritySha256": active_sha, - "sourceTrustedStateSha256": source["stateSha256"], - } - doc = amap[lid][1](request) - parsed = _normalize_merkle_error( - "ARCHIVE_MERKLE_REBRIDGE_GOSSIP_INVALID", - merkle._verify_gossip, - doc, - log_id=lid, - config=active_authority[lid], - sequence=sequence, - challenge=gossip_challenge, - run163_head=state163["anchorConsensusHeadSha256"], - checkpoints=cp_hashes, - now=current, - creation=True, - historical=False, - ) - gossip_docs[lid] = doc - gossip_hashes.append(parsed["gossipSha256"]) - run163_artifacts = _artifact_map(current163["raws"]) - event_bare = { - "sequence": sequence, - "postRun166Sequence": len(old_events) + 1, - "action": action, - "rebridgeSequence": rebridge_sequence, - "run163Sequence": sequence, - "run163AnchorConsensusHeadSha256": state163["anchorConsensusHeadSha256"], - "run163Artifacts": run163_artifacts, - "leafHash": leaf_hash, - "sourceTrustedStateSha256": source["stateSha256"], - "archiveMerkleRebridgeAuthorityHeadSha256": authority_head, - "activeAuthoritySha256": active_sha, - "revokedKeyFingerprints": revoked, - "transitionSha256": transition_sha, - "handoffSha256": handoff_sha, - "logs": rows, - "logChallenge": log_challenge, - "gossipChallenge": gossip_challenge, - "gossipResponseSha256s": gossip_hashes, - } - new_merkle_head = _merkle_head(source["merkleHead"], event_bare) - event = dict(event_bare) - event["merkleRebridgeContinuityHeadSha256"] = new_merkle_head - receipt_event = { - "sequence": sequence, - "run163Documents": current163["docs"], - "transitionDocument": transition_doc, - "handoffProofDocument": handoff_doc, - "logResponses": log_docs, - "gossipResponses": gossip_docs, - } - events = [*old_events, event] - receipt_events = [*old_receipts, receipt_event] - bundle = dict(base) - bundle["events"] = events - receipt = { - "schemaVersion": int(POLICY["receipt_schema_version"]), - "status": "archive-merkle-authority-rebridge-accepted", - "run166Documents": run166_info["docs"], - "run165Documents": run165_info["docs"], - "events": receipt_events, - } - state = _make_state( - base=base, - events=events, - active_authority=active_authority, - revoked=revoked, - rebridge_sequence=rebridge_sequence, - authority_head=authority_head, - merkle_head=new_merkle_head, - checkpoints=next_checkpoints, - last_transition_issued=previous_transition_issued, - ) - active = _make_active(state, event) - if before != _authority_fingerprint( - authority_paths, "ARCHIVE_MERKLE_REBRIDGE_INPUT_DRIFT" - ): - _fail("ARCHIVE_MERKLE_REBRIDGE_INPUT_DRIFT") - target = _outside( - Path(output_dir), - authority_paths, - "ARCHIVE_MERKLE_REBRIDGE_OUTPUT_OVERLAPS_INPUT", - ) - if target.exists(): - _fail("ARCHIVE_MERKLE_REBRIDGE_OUTPUT_EXISTS") - target.parent.mkdir(parents=True, exist_ok=True) - stage = Path(tempfile.mkdtemp(prefix=".run167-merkle-rebridge-", dir=target.parent)) - try: - _write(stage / "release-archive-merkle-rebridge-bundle.json", bundle) - _write(stage / _DOC_ARCHIVE_MERKLE_REBRIDGE_STATE, state) - _write(stage / "active-archive-merkle-rebridge.json", active) - _write(stage / "release-archive-merkle-rebridge-receipt.json", receipt) - verify_archive_merkle_rebridge( - run160_dir=run160_dir, - run166_run161_dir=run166_run161_dir, - run161_dir=run161_dir, - retention_root_path=retention_root_path, - retention_root_pin=retention_root_pin, - witness_root_path=witness_root_path, - witness_root_pin=witness_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - run166_run162_dir=run166_run162_dir, - run162_dir=run162_dir, - anchor_plan_path=anchor_plan_path, - run166_run163_dir=run166_run163_dir, - run163_dir=run163_dir, - run164_dir=run164_dir, - transparency_root_path=transparency_root_path, - transparency_root_pin=transparency_root_pin, - governance_root_path=governance_root_path, - governance_root_pin=governance_root_pin, - recovery_root_path=recovery_root_path, - recovery_root_pin=recovery_root_pin, - run165_dir=run165_dir, - run166_dir=run166_dir, - output_dir=stage, - now=current, - ) - stage.rename(target) - except Exception: - shutil.rmtree(stage, ignore_errors=True) - raise - return verify_archive_merkle_rebridge( - run160_dir=run160_dir, - run166_run161_dir=run166_run161_dir, - run161_dir=run161_dir, - retention_root_path=retention_root_path, - retention_root_pin=retention_root_pin, - witness_root_path=witness_root_path, - witness_root_pin=witness_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - run166_run162_dir=run166_run162_dir, - run162_dir=run162_dir, - anchor_plan_path=anchor_plan_path, - run166_run163_dir=run166_run163_dir, - run163_dir=run163_dir, - run164_dir=run164_dir, - transparency_root_path=transparency_root_path, - transparency_root_pin=transparency_root_pin, - governance_root_path=governance_root_path, - governance_root_pin=governance_root_pin, - recovery_root_path=recovery_root_path, - recovery_root_pin=recovery_root_pin, - run165_dir=run165_dir, - run166_dir=run166_dir, - output_dir=target, - now=current, - ) - - -def command_log(command: list[str]): # ruff: ignore[undocumented-public-function] - return merkle.command_log(command) - - -def command_gossip(command: list[str]): # ruff: ignore[undocumented-public-function] - return merkle.command_gossip(command) - - -def _common(parser: argparse.ArgumentParser) -> None: - parser.add_argument("--run160-dir", type=Path, required=True) - parser.add_argument("--run166-run161-dir", type=Path, required=True) - parser.add_argument("--run161-dir", type=Path, required=True) - parser.add_argument("--retention-root-path", type=Path, required=True) - parser.add_argument("--retention-root-pin", required=True) - parser.add_argument("--witness-root-path", type=Path, required=True) - parser.add_argument("--witness-root-pin", required=True) - parser.add_argument("--bootstrap-pin", required=True) - parser.add_argument("--recovery-pin", required=True) - parser.add_argument( - "--attestation-pin", action="append", dest="attestation_pins", required=True - ) - parser.add_argument("--run166-run162-dir", type=Path, required=True) - parser.add_argument("--run162-dir", type=Path, required=True) - parser.add_argument("--anchor-plan-path", type=Path, required=True) - parser.add_argument("--run166-run163-dir", type=Path, required=True) - parser.add_argument("--run163-dir", type=Path, required=True) - parser.add_argument("--run164-dir", type=Path, required=True) - parser.add_argument("--transparency-root-path", type=Path, required=True) - parser.add_argument("--transparency-root-pin", required=True) - parser.add_argument("--governance-root-path", type=Path, required=True) - parser.add_argument("--governance-root-pin", required=True) - parser.add_argument("--recovery-root-path", type=Path, required=True) - parser.add_argument("--recovery-root-pin", required=True) - parser.add_argument("--run165-dir", type=Path, required=True) - parser.add_argument("--run166-dir", type=Path, required=True) - parser.add_argument("--output-dir", type=Path, required=True) - - -def main( # ruff: ignore[undocumented-public-function] - argv=None, -) -> int: - parser = argparse.ArgumentParser(description=__doc__) - subs = parser.add_subparsers(dest="cmd", required=True) - v = subs.add_parser("verify") - _common(v) - v.add_argument("--historical", action="store_true") - a = subs.add_parser("advance") - _common(a) - a.add_argument("--previous-output-dir", type=Path) - a.add_argument("--transition", type=Path) - a.add_argument("--handoff", type=Path) - a.add_argument("--log-command", action="append", required=True) - a.add_argument("--gossip-command", action="append", required=True) - ns = parser.parse_args(argv) - kwargs = vars(ns) - cmd = kwargs.pop("cmd") - try: - if cmd == "verify": - historical = kwargs.pop("historical") - result = verify_archive_merkle_rebridge(**kwargs, historical=historical) - else: - previous = kwargs.pop("previous_output_dir") - transition = kwargs.pop("transition") - handoff = kwargs.pop("handoff") - log_cmds = kwargs.pop("log_command") - gossip_cmds = kwargs.pop("gossip_command") - - def parse(items, builder): - out = {} - for raw in items: - if "::" not in raw: - _fail("ARCHIVE_MERKLE_REBRIDGE_COMMAND_INVALID") - lid, command = raw.split("::", 1) - out[lid] = builder(command.split()) - return out - - logs = parse(log_cmds, command_log) - goss = parse(gossip_cmds, command_gossip) - if set(logs) != set(goss): - _fail("ARCHIVE_MERKLE_REBRIDGE_COMMAND_SET_INVALID") - adapters = [(lid, logs[lid], goss[lid]) for lid in sorted(logs)] - result = advance_archive_merkle_rebridge( - **kwargs, - adapters=adapters, - transition_path=transition, - handoff_path=handoff, - previous_output_dir=previous, - ) - logger.info( - json.dumps({"ok": True, "result": result}, sort_keys=True, default=str) - ) - return 0 - except ArchiveMerkleRebridgeError as exc: - logger.error(json.dumps({"ok": False, "error": str(exc)}, sort_keys=True)) - return 2 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/security/release-evidence.example.json b/security/release-evidence.example.json deleted file mode 100644 index bc72cd654686ebd939e26d3a6bad076b2ae92d04..0000000000000000000000000000000000000000 --- a/security/release-evidence.example.json +++ /dev/null @@ -1,216 +0,0 @@ -{ - "artifacts": { - "dependencyScan": { - "path": "pip-audit.json", - "sha256": "REPLACE_WITH_64_HEX", - "status": "pass", - "subject": "sha256:REPLACE_WITH_64_HEX", - "tool": { - "name": "pip-audit", - "version": "REPLACE" - } - }, - "imageSbom": { - "path": "image.cdx.json", - "sha256": "REPLACE_WITH_64_HEX", - "status": "pass", - "subject": "sha256:REPLACE_WITH_64_HEX", - "tool": { - "name": "syft", - "version": "REPLACE" - } - }, - "imageScan": { - "path": "image-scan.json", - "sha256": "REPLACE_WITH_64_HEX", - "status": "pass", - "subject": "sha256:REPLACE_WITH_64_HEX", - "tool": { - "name": "REPLACE", - "version": "REPLACE" - } - }, - "provenance": { - "path": "provenance.json", - "predicateType": "https://slsa.dev/provenance/v1", - "sha256": "REPLACE_WITH_64_HEX", - "signatureVerified": true, - "status": "pass", - "subject": "sha256:REPLACE_WITH_64_HEX", - "tool": { - "name": "REPLACE", - "version": "REPLACE" - } - }, - "signatureVerification": { - "path": "signature-verification.json", - "sha256": "REPLACE_WITH_64_HEX", - "status": "pass", - "subject": "sha256:REPLACE_WITH_64_HEX", - "tool": { - "name": "REPLACE", - "version": "REPLACE" - } - } - }, - "image": { - "digest": "sha256:REPLACE_WITH_64_HEX", - "platform": "linux/amd64" - }, - "logging": { - "apmBodyCapture": false, - "authorizationHeaderLogging": false, - "capabilityHeaderLogging": false, - "queryStringLogging": false, - "requestBodyLogging": false, - "reviewedAt": "2099-01-01T00:00:00Z", - "thirdPartyTelemetryExport": false, - "wafBodyCapture": false - }, - "redis": { - "contribution": { - "backupRestoreTestedAt": "2098-12-15T00:00:00Z", - "leastPrivilegeReviewed": true, - "nonDefaultIdentity": true, - "persistenceVerified": true, - "replicationVerified": true, - "tlsVerified": true - }, - "providerArtifactLifecycle": { - "backupRestoreTestedAt": null, - "leastPrivilegeReviewed": true, - "nonDefaultIdentity": true, - "persistenceVerified": false, - "replicationVerified": true, - "tlsVerified": true - }, - "rateLimit": { - "backupRestoreTestedAt": null, - "leastPrivilegeReviewed": true, - "nonDefaultIdentity": true, - "persistenceVerified": false, - "replicationVerified": false, - "tlsVerified": true - }, - "share": { - "backupRestoreTestedAt": "2098-12-15T00:00:00Z", - "leastPrivilegeReviewed": true, - "nonDefaultIdentity": true, - "persistenceVerified": true, - "replicationVerified": true, - "tlsVerified": true - } - }, - "redisChaos": { - "redis7": { - "standalone": { - "attestation": { - "path": "redis7-standalone.json", - "sha256": "REPLACE_WITH_64_HEX", - "status": "pass", - "subject": "sha256:REPLACE_WITH_64_HEX", - "tool": { - "name": "scikitplot-provider-artifact-redis-chaos", - "version": "148" - } - }, - "signatureVerification": { - "path": "redis7-standalone.signature-verification.json", - "sha256": "REPLACE_WITH_64_HEX", - "status": "pass", - "subject": "sha256:REPLACE_WITH_64_HEX", - "tool": { - "name": "REPLACE_WITH_APPROVED_SIGNATURE_VERIFIER", - "version": "REPLACE" - } - } - }, - "cluster": { - "attestation": { - "path": "redis7-cluster.json", - "sha256": "REPLACE_WITH_64_HEX", - "status": "pass", - "subject": "sha256:REPLACE_WITH_64_HEX", - "tool": { - "name": "scikitplot-provider-artifact-redis-chaos", - "version": "148" - } - }, - "signatureVerification": { - "path": "redis7-cluster.signature-verification.json", - "sha256": "REPLACE_WITH_64_HEX", - "status": "pass", - "subject": "sha256:REPLACE_WITH_64_HEX", - "tool": { - "name": "REPLACE_WITH_APPROVED_SIGNATURE_VERIFIER", - "version": "REPLACE" - } - } - } - }, - "redis8": { - "standalone": { - "attestation": { - "path": "redis8-standalone.json", - "sha256": "REPLACE_WITH_64_HEX", - "status": "pass", - "subject": "sha256:REPLACE_WITH_64_HEX", - "tool": { - "name": "scikitplot-provider-artifact-redis-chaos", - "version": "148" - } - }, - "signatureVerification": { - "path": "redis8-standalone.signature-verification.json", - "sha256": "REPLACE_WITH_64_HEX", - "status": "pass", - "subject": "sha256:REPLACE_WITH_64_HEX", - "tool": { - "name": "REPLACE_WITH_APPROVED_SIGNATURE_VERIFIER", - "version": "REPLACE" - } - } - }, - "cluster": { - "attestation": { - "path": "redis8-cluster.json", - "sha256": "REPLACE_WITH_64_HEX", - "status": "pass", - "subject": "sha256:REPLACE_WITH_64_HEX", - "tool": { - "name": "scikitplot-provider-artifact-redis-chaos", - "version": "148" - } - }, - "signatureVerification": { - "path": "redis8-cluster.signature-verification.json", - "sha256": "REPLACE_WITH_64_HEX", - "status": "pass", - "subject": "sha256:REPLACE_WITH_64_HEX", - "tool": { - "name": "REPLACE_WITH_APPROVED_SIGNATURE_VERIFIER", - "version": "REPLACE" - } - } - } - } - }, - "release": { - "expiresAt": "2099-01-02T00:00:00Z", - "generatedAt": "2099-01-01T00:00:00Z", - "proxyVersion": "7.0.0", - "releaseId": "example-only-not-production", - "targetPlatform": "linux/amd64" - }, - "riskExceptions": [], - "schemaVersion": 2, - "source": { - "baseImageIndexDigest": "sha256:REPLACE_WITH_64_HEX", - "baseImageManifestDigest": "sha256:REPLACE_WITH_64_HEX", - "pythonSbomSha256": "REPLACE_WITH_64_HEX", - "requirementsLockSha256": "REPLACE_WITH_64_HEX", - "runtimeSourceSha256": "REPLACE_WITH_64_HEX", - "sourceTreeSha256": "REPLACE_WITH_64_HEX", - "sourceRevision": "REPLACE_WITH_40_OR_64_HEX_SOURCE_REVISION" - } -} diff --git a/security/release_archive_anchor_policy.toml b/security/release_archive_anchor_policy.toml deleted file mode 100644 index 1e7f1e9c4a62a3524b77ba01a0f934246c8f2050..0000000000000000000000000000000000000000 --- a/security/release_archive_anchor_policy.toml +++ /dev/null @@ -1,26 +0,0 @@ -schema_version = 1 -anchor_plan_schema_version = 1 -anchor_bundle_schema_version = 1 -anchor_state_schema_version = 1 -anchor_active_schema_version = 1 -anchor_receipt_schema_version = 1 -recovery_root_schema_version = 1 -recovery_record_schema_version = 1 -recovery_receipt_schema_version = 1 -spec_version = "1.0.0" -predicate_type = "https://scikit-plots.org/attestations/release-archive-health-anchor/v1" -max_json_bytes = 67108864 -max_adapter_output_bytes = 8388608 -adapter_timeout_seconds = 30 -max_clock_skew_minutes = 10 -max_observation_age_minutes = 30 -maximum_anchor_interval_days = 31 -max_root_lifetime_days = 3660 -max_key_count = 32 -min_channels = 3 -min_channel_operators = 3 -min_observer_operators = 3 -min_recovery_keys = 3 -min_recovery_threshold = 2 -min_recovery_operators = 2 -min_recovery_channels = 2 diff --git a/security/release_archive_health_policy.toml b/security/release_archive_health_policy.toml deleted file mode 100644 index 4605dea96e8946e06b1021da50d207567589748e..0000000000000000000000000000000000000000 --- a/security/release_archive_health_policy.toml +++ /dev/null @@ -1,30 +0,0 @@ -schema_version = 1 -root_schema_version = 1 -membership_schema_version = 1 -bundle_schema_version = 1 -state_schema_version = 1 -active_schema_version = 1 -receipt_schema_version = 1 -provider_protocol_version = 1 -auditor_protocol_version = 1 -predicate_type = "https://scikit-plots.org/attestations/release-archive-health/v1" -spec_version = "1.0.0" -max_json_bytes = 67108864 -max_adapter_output_bytes = 8388608 -adapter_timeout_seconds = 30 -max_events = 1024 -max_clock_skew_minutes = 10 -max_result_age_minutes = 30 -max_root_lifetime_days = 3660 -max_key_count = 32 -max_archives = 32 -min_governance_keys = 2 -min_governance_threshold = 2 -min_governance_operators = 2 -min_durable_copies = 2 -min_archive_operators = 2 -min_auditor_operators = 2 -minimum_retention_remaining_days = 30 -maximum_audit_interval_days = 31 -allowed_retention_modes = ["object-lock-compliance", "object-lock-governance", "content-addressed", "versioned-create-only", "release-create-only"] -allowed_immutability = ["object-lock", "content-addressed", "versioned-create-only", "release-create-only"] diff --git a/security/release_archive_log_authority_policy.toml b/security/release_archive_log_authority_policy.toml deleted file mode 100644 index 22915f5f4341eb569065ade0de4e37148fa32688..0000000000000000000000000000000000000000 --- a/security/release_archive_log_authority_policy.toml +++ /dev/null @@ -1,21 +0,0 @@ -spec_version = "1.0.0" -predicate_type = "https://scikit-plots.org/attestations/release-archive-log-authority/v1" -governance_root_schema_version = 1 -recovery_root_schema_version = 1 -transition_schema_version = 1 -handoff_schema_version = 1 -bundle_schema_version = 1 -state_schema_version = 1 -active_schema_version = 1 -receipt_schema_version = 1 -min_governance_keys = 3 -min_governance_threshold = 2 -min_governance_operators = 2 -min_recovery_keys = 3 -min_recovery_threshold = 2 -min_recovery_operators = 2 -min_recovery_channels = 2 -max_root_lifetime_days = 730 -max_clock_skew_minutes = 5 -max_transition_freshness_minutes = 30 -max_json_bytes = 8388608 diff --git a/security/release_archive_merkle_continuity_policy.toml b/security/release_archive_merkle_continuity_policy.toml deleted file mode 100644 index 40aa06d27d596e5d30e0507750e33f274671e8f9..0000000000000000000000000000000000000000 --- a/security/release_archive_merkle_continuity_policy.toml +++ /dev/null @@ -1,10 +0,0 @@ -spec_version = "1.0.0" -predicate_type = "https://scikit-plots.org/attestations/release-archive-merkle-authority-continuity/v1" -bundle_schema_version = 1 -state_schema_version = 1 -active_schema_version = 1 -receipt_schema_version = 1 -max_clock_skew_minutes = 5 -max_active_authority_age_days = 365 -max_active_epoch_age_days = 31 -max_json_bytes = 8388608 diff --git a/security/release_archive_merkle_policy.toml b/security/release_archive_merkle_policy.toml deleted file mode 100644 index 42742a17b2d5856c32f5a8eb28e00a0bab401ae7..0000000000000000000000000000000000000000 --- a/security/release_archive_merkle_policy.toml +++ /dev/null @@ -1,23 +0,0 @@ -spec_version = "1.0.0" -predicate_type = "https://scikit-plots.org/attestations/release-archive-merkle-consensus/v1" -transparency_root_schema_version = 1 -checkpoint_schema_version = 1 -gossip_schema_version = 1 -bundle_schema_version = 1 -state_schema_version = 1 -active_schema_version = 1 -receipt_schema_version = 1 -min_root_keys = 3 -min_root_threshold = 2 -min_root_operators = 2 -min_logs = 3 -min_gossipers = 3 -max_root_lifetime_days = 730 -max_clock_skew_minutes = 5 -max_adapter_freshness_minutes = 30 -max_active_epoch_age_days = 31 -max_json_bytes = 8388608 -max_adapter_output_bytes = 8388608 -adapter_timeout_seconds = 30 -max_proof_nodes = 128 -max_tree_size = 9223372036854775807 diff --git a/security/release_archive_merkle_rebridge_policy.toml b/security/release_archive_merkle_rebridge_policy.toml deleted file mode 100644 index f7e91ee41862baaba24630848afe502c3a469bc6..0000000000000000000000000000000000000000 --- a/security/release_archive_merkle_rebridge_policy.toml +++ /dev/null @@ -1,13 +0,0 @@ -spec_version = "1.0.0" -predicate_type = "https://scikit-plots.org/attestations/release-archive-merkle-authority-rebridge/v1" -transition_schema_version = 1 -handoff_schema_version = 1 -bundle_schema_version = 1 -state_schema_version = 1 -active_schema_version = 1 -receipt_schema_version = 1 -max_clock_skew_minutes = 5 -max_transition_freshness_minutes = 30 -max_active_authority_age_days = 365 -max_active_epoch_age_days = 31 -max_json_bytes = 8388608 diff --git a/security/release_archive_merkle_recovery_policy.toml b/security/release_archive_merkle_recovery_policy.toml deleted file mode 100644 index 8795c4da2ea3e2c74c721ed6e42d1d64a36a54d4..0000000000000000000000000000000000000000 --- a/security/release_archive_merkle_recovery_policy.toml +++ /dev/null @@ -1,24 +0,0 @@ -spec_version = "1.0.0" -predicate_type = "https://scikit-plots.org/attestations/release-archive-merkle-recovery/v1" -checkpoint_schema_version = 1 -state_schema_version = 1 -receipt_schema_version = 1 -recovery_receipt_schema_version = 1 -active_recovery_schema_version = 1 -archive_protocol_version = 1 -verifier_protocol_version = 1 -recovery_protocol_version = 1 -max_json_bytes = 67108864 -max_adapter_output_bytes = 8388608 -adapter_timeout_seconds = 30 -max_result_age_minutes = 15 -max_clock_skew_minutes = 5 -min_configured_archives = 3 -min_successful_archives = 3 -min_archive_operators = 2 -min_verifier_operators = 2 -min_configured_recovery_sources = 3 -min_observed_recovery_sources = 2 -min_recovery_operators = 2 -allow_unavailable_recovery_source = true -allowed_archive_immutability = ["object-lock", "content-addressed", "versioned-create-only", "release-create-only"] diff --git a/security/release_archive_witness_policy.toml b/security/release_archive_witness_policy.toml deleted file mode 100644 index 4f269d1c49b63cad4c6fc03c926772987f9d148e..0000000000000000000000000000000000000000 --- a/security/release_archive_witness_policy.toml +++ /dev/null @@ -1,27 +0,0 @@ -schema_version = 1 -witness_root_schema_version = 1 -witness_bundle_schema_version = 1 -witness_state_schema_version = 1 -witness_active_schema_version = 1 -witness_receipt_schema_version = 1 -witness_protocol_version = 1 -recovery_root_schema_version = 1 -recovery_record_schema_version = 1 -recovery_receipt_schema_version = 1 -spec_version = "1.0.0" -predicate_type = "https://scikit-plots.org/attestations/release-archive-health-witness/v1" -max_json_bytes = 67108864 -max_adapter_output_bytes = 8388608 -adapter_timeout_seconds = 30 -max_clock_skew_minutes = 10 -max_observation_age_minutes = 30 -maximum_witness_interval_days = 31 -max_root_lifetime_days = 3660 -max_key_count = 32 -min_witness_keys = 3 -min_witness_threshold = 2 -min_witness_operators = 2 -min_recovery_keys = 3 -min_recovery_threshold = 2 -min_recovery_operators = 2 -min_recovery_channels = 2 diff --git a/security/release_attestation_lifecycle_policy.toml b/security/release_attestation_lifecycle_policy.toml deleted file mode 100644 index 4372faab7e7eb6c127e91e4d8e60134dedf3355c..0000000000000000000000000000000000000000 --- a/security/release_attestation_lifecycle_policy.toml +++ /dev/null @@ -1,34 +0,0 @@ -schema_version = 1 -bundle_schema_version = 1 -state_schema_version = 1 -receipt_schema_version = 1 -ca_set_schema_version = 1 -status_schema_version = 1 -vendor_claim_schema_version = 1 -predicate_type = "https://scikit-plots.org/attestations/release-attestation-lifecycle/v1" -spec_version = "1.0.0" -max_json_bytes = 16777216 -max_clock_skew_minutes = 10 -max_ca_set_lifetime_days = 825 -min_ca_set_remaining_minutes = 60 -max_status_lifetime_hours = 48 -min_status_remaining_minutes = 15 -max_ca_roots = 16 -max_status_keys = 32 -min_status_keys = 3 -min_status_threshold = 2 -min_status_operators = 2 -min_root_signer_operators = 2 -max_status_entries = 8192 -max_lifecycle_events = 4096 -vendor_claim_extension_oid = "1.3.6.1.4.1.57264.158.1" -require_status_operator_disjoint_from_release_root = true -require_status_key_disjoint_from_release_root = true -require_all_attestation_certificates_covered = true -require_all_active_attestations_good = true - -[required_profile_by_device_class] -hsm = "hsm-x509-v1" -kms-hsm = "kms-hsm-x509-v1" -tpm = "tpm-x509-v1" -secure-element = "secure-element-x509-v1" diff --git a/security/release_delegated_trust_policy.toml b/security/release_delegated_trust_policy.toml deleted file mode 100644 index 9141743292150ae96111322b8f164ca82d2aaf67..0000000000000000000000000000000000000000 --- a/security/release_delegated_trust_policy.toml +++ /dev/null @@ -1,40 +0,0 @@ -schema_version = 1 -delegation_schema_version = 1 -snapshot_schema_version = 1 -timestamp_schema_version = 1 -state_schema_version = 1 -bundle_schema_version = 1 -receipt_schema_version = 1 -recovery_root_schema_version = 1 -recovery_record_schema_version = 1 -recovery_signature_schema_version = 1 -predicate_type = "https://scikit-plots.org/attestations/release-delegated-trust/v1" -spec_version = "1.0.0" -max_json_bytes = 16777216 -max_keys = 64 -min_snapshot_keys = 3 -min_snapshot_threshold = 2 -min_snapshot_operators = 2 -min_timestamp_keys = 3 -min_timestamp_threshold = 2 -min_timestamp_operators = 2 -min_recovery_keys = 3 -min_recovery_threshold = 2 -min_recovery_operators = 2 -min_recovery_channels = 2 -max_clock_skew_minutes = 10 -min_delegation_remaining_minutes = 60 -max_delegation_lifetime_days = 400 -min_snapshot_remaining_minutes = 30 -max_snapshot_lifetime_hours = 72 -min_timestamp_remaining_minutes = 10 -max_timestamp_lifetime_hours = 24 -max_recovery_signature_age_hours = 24 -max_recovery_root_lifetime_days = 3650 -min_recovery_root_remaining_days = 30 -allowed_keytypes = ["ed25519"] -allowed_schemes = ["ed25519"] -allowed_signer_profiles = ["offline", "hsm", "kms-hsm"] -hardware_signer_profiles = ["hsm", "kms-hsm"] -require_hardware_profile_for_recovery = true -require_recovery_role_only_change = true diff --git a/security/release_evidence_policy.toml b/security/release_evidence_policy.toml deleted file mode 100644 index 2af2656254ad0d6fd5fc733d8015c1a49d5c816d..0000000000000000000000000000000000000000 --- a/security/release_evidence_policy.toml +++ /dev/null @@ -1,41 +0,0 @@ -schema_version = 1 -evidence_schema_version = 2 -target_platform = "linux/amd64" -max_age_hours = 72 -max_clock_skew_minutes = 10 -max_manifest_bytes = 262144 -max_evidence_file_bytes = 52428800 -minimum_cyclonedx_spec_version = "1.6" -require_dependency_scan = true -require_full_image_sbom = true -require_image_vulnerability_scan = true -require_slsa_provenance = true -require_signature_verification = true -require_log_privacy_attestation = true -forbid_unexpired_risk_exceptions = true - -[redis] -require_tls = true -require_non_default_identity = true -require_least_privilege_review = true -share_require_persistence = true -share_require_replication = true -contribution_require_persistence = true -contribution_require_replication = true -provider_artifact_lifecycle_require_replication = true -backup_restore_max_age_days = 90 - -[redis_chaos] -require_signed_evidence = true -max_age_hours = 72 -redis7_image = "redis:7.4.11-bookworm@sha256:71da9275c5f3fcb97d0fa0c8c5b36cc995327265420f17a04bfd544f458059f7" -redis8_image = "redis:8.2.9-bookworm@sha256:7d1e4ce8b9395088377ab382d1f6cfdbd13b3690795198a0399ab8d683064d6d" - -[logging] -request_body_logging = false -authorization_header_logging = false -capability_header_logging = false -query_string_logging = false -waf_body_capture = false -apm_body_capture = false -third_party_telemetry_export = false diff --git a/security/release_governance_policy.toml b/security/release_governance_policy.toml deleted file mode 100644 index c3ff613c5d2e769cd9153b76991a3c69024aa25d..0000000000000000000000000000000000000000 --- a/security/release_governance_policy.toml +++ /dev/null @@ -1,44 +0,0 @@ -schema_version = 1 -state_schema_version = 1 -bundle_schema_version = 1 -receipt_schema_version = 1 -genesis_schema_version = 1 -proposal_schema_version = 1 -approval_schema_version = 1 -recovery_protocol_version = 1 -archive_protocol_version = 1 -recovery_snapshot_schema_version = 1 -predicate_type = "https://scikit-plots.org/attestations/release-governance/v1" -max_json_bytes = 16777216 -max_adapter_output_bytes = 25165824 -adapter_timeout_seconds = 300 -max_clock_skew_minutes = 10 -min_policy_authorities = 3 -max_policy_authorities = 15 -min_policy_threshold = 2 -min_policy_operators = 2 -min_emergency_authorities = 3 -max_emergency_authorities = 15 -min_emergency_threshold = 2 -min_emergency_operators = 2 -min_history_replicas = 3 -max_history_replicas = 15 -min_history_replica_threshold = 2 -min_history_replica_operators = 2 -min_history_archives = 2 -max_history_archives = 12 -min_history_archive_threshold = 2 -min_history_archive_operators = 2 -min_recovery_sources = 3 -max_recovery_sources = 12 -min_recovery_quorum = 2 -min_recovery_operators = 2 -min_governance_archives = 2 -max_governance_archives = 8 -min_governance_archive_operators = 2 -allow_unavailable_recovery_source = true -allow_policy_change_without_proposal = false -allow_revoked_authority_key = false -allowed_transition_reasons = ["scheduled-change", "membership-change", "authority-compromise-recovery"] -allowed_archive_immutability = ["object-lock", "content-addressed", "versioned-create-only", "release-create-only"] -require_emergency_operator_disjoint = true diff --git a/security/release_history_policy.toml b/security/release_history_policy.toml deleted file mode 100644 index b29496e401262b6b7a0ac631ac5f741a22a8735b..0000000000000000000000000000000000000000 --- a/security/release_history_policy.toml +++ /dev/null @@ -1,27 +0,0 @@ -schema_version = 1 -state_schema_version = 1 -bundle_schema_version = 1 -receipt_schema_version = 1 -replica_protocol_version = 1 -archive_protocol_version = 1 -key_transition_schema_version = 1 -history_predicate_type = "https://scikit-plots.org/attestations/release-history/v1" -max_json_bytes = 8388608 -max_entries = 4096 -max_adapter_output_bytes = 262144 -adapter_timeout_seconds = 300 -max_result_age_minutes = 60 -max_clock_skew_minutes = 10 -min_replicas = 3 -max_replicas = 9 -min_replica_quorum = 2 -min_replica_operators = 2 -min_archives = 2 -max_archives = 6 -min_archive_operators = 2 -allow_unavailable_replica = true -allow_history_gap = false -allow_log_key_change_without_transition = false -allow_revoked_log_key = false -allowed_archive_immutability = ["object-lock", "content-addressed", "versioned-create-only", "release-create-only"] -allowed_transition_reasons = ["scheduled-rotation", "compromise-recovery"] diff --git a/security/release_native_archive_policy.toml b/security/release_native_archive_policy.toml deleted file mode 100644 index db987647e11908372919e588fee24a231d76c381..0000000000000000000000000000000000000000 --- a/security/release_native_archive_policy.toml +++ /dev/null @@ -1,22 +0,0 @@ -schema_version = 1 -archive_schema_version = 1 -state_schema_version = 1 -receipt_schema_version = 1 -recovery_receipt_schema_version = 1 -archive_protocol_version = 1 -verifier_protocol_version = 1 -recovery_protocol_version = 1 -predicate_type = "https://scikit-plots.org/attestations/release-native-status-archive/v1" -spec_version = "1.0.0" -max_json_bytes = 67108864 -max_adapter_output_bytes = 67108864 -adapter_timeout_seconds = 30 -max_result_age_minutes = 15 -max_clock_skew_minutes = 10 -min_archives = 2 -min_archive_operators = 2 -min_verifier_operators = 2 -min_recovery_sources = 2 -min_recovery_operators = 2 -allow_unavailable_recovery_source = true -allowed_archive_immutability = ["object-lock", "content-addressed", "versioned-create-only", "release-create-only"] diff --git a/security/release_native_status_policy.toml b/security/release_native_status_policy.toml deleted file mode 100644 index d3a490abf6256f1b2de576cd93c66d89715c4535..0000000000000000000000000000000000000000 --- a/security/release_native_status_policy.toml +++ /dev/null @@ -1,25 +0,0 @@ -schema_version = 1 -bundle_schema_version = 1 -state_schema_version = 1 -receipt_schema_version = 1 -predicate_type = "https://scikit-plots.org/attestations/release-native-status-provenance/v1" -spec_version = "1.0.0" -max_json_bytes = 16777216 -max_raw_evidence_bytes = 8388608 -max_raw_evidence_files = 16384 -max_clock_skew_minutes = 10 -max_crl_lifetime_hours = 168 -max_ocsp_lifetime_hours = 48 -min_native_remaining_minutes = 10 -min_source_kinds_per_leaf = 2 -required_source_kinds = ["crl", "ocsp"] -require_crl_number = true -require_ocsp_next_update = true -require_ocsp_responder_name = true -require_native_match_run158 = true -require_all_active_leaf_certificates = true -max_events = 4096 -vendor_verifier_timeout_seconds = 15 -vendor_verifier_max_output_bytes = 1048576 -max_total_raw_evidence_bytes = 8388608 -max_crl_entries = 8192 diff --git a/security/release_promotion_policy.toml b/security/release_promotion_policy.toml deleted file mode 100644 index 7fe1e361f0620ce7fb6e596aef365820f3388714..0000000000000000000000000000000000000000 --- a/security/release_promotion_policy.toml +++ /dev/null @@ -1,11 +0,0 @@ -schema_version = 1 -statement_schema_version = 1 -signature_record_schema_version = 1 -receipt_schema_version = 1 -max_patch_bytes = 16777216 -max_baseline_zip_bytes = 67108864 -max_source_files = 10000 -max_source_total_bytes = 268435456 -max_signature_age_hours = 72 -archive_prefix = "scikitplot/_externals/_sphinx_ext/_sphinx_ai_assistant" -predicate_type = "https://scikit-plots.org/attestations/release-promotion/v1" diff --git a/security/release_publication_policy.toml b/security/release_publication_policy.toml deleted file mode 100644 index e5fa2ad9ebd97efe8ddbc67b3584e97d44ca0769..0000000000000000000000000000000000000000 --- a/security/release_publication_policy.toml +++ /dev/null @@ -1,15 +0,0 @@ -schema_version = 1 -transparency_schema_version = 1 -publication_receipt_schema_version = 1 -adapter_protocol_version = 1 -predicate_type = "https://scikit-plots.org/attestations/release-publication/v1" -promotion_predicate_type = "https://scikit-plots.org/attestations/release-promotion/v1" -max_promotion_receipt_bytes = 1048576 -max_publish_artifacts = 32 -max_artifact_bytes = 536870912 -max_total_publish_bytes = 1073741824 -max_adapter_output_bytes = 262144 -adapter_timeout_seconds = 300 -max_result_age_minutes = 30 -max_clock_skew_minutes = 10 -allowed_immutability = ["object-lock", "content-addressed", "versioned-create-only", "release-create-only"] diff --git a/security/release_root_continuity_policy.toml b/security/release_root_continuity_policy.toml deleted file mode 100644 index cbb44a9b1c2d4dc1f0c19c869756de579cbd65f2..0000000000000000000000000000000000000000 --- a/security/release_root_continuity_policy.toml +++ /dev/null @@ -1,17 +0,0 @@ -schema_version = 1 -bundle_schema_version = 1 -state_schema_version = 1 -receipt_schema_version = 1 -attestation_schema_version = 1 -predicate_type = "https://scikit-plots.org/attestations/release-root-continuity/v1" -spec_version = "1.0.0" -max_json_bytes = 16777216 -max_continuity_epochs = 4096 -max_attestation_chain_length = 8 -max_clock_skew_minutes = 10 -max_attestation_lifetime_days = 825 -min_attestation_remaining_minutes = 60 -min_attestation_roots = 1 -require_all_active_root_keys_attested = true -allowed_device_classes = ["hsm", "kms-hsm", "tpm", "secure-element"] -allowed_leaf_key_algorithms = ["ed25519", "ecdsa-p256"] diff --git a/security/release_root_policy.toml b/security/release_root_policy.toml deleted file mode 100644 index 35ebafadd33eb55d2e21e9b92eb31913f3053a50..0000000000000000000000000000000000000000 --- a/security/release_root_policy.toml +++ /dev/null @@ -1,28 +0,0 @@ -schema_version = 1 -root_metadata_schema_version = 1 -root_state_schema_version = 1 -root_bundle_schema_version = 1 -authorization_schema_version = 1 -signature_schema_version = 1 -receipt_schema_version = 1 -predicate_type = "https://scikit-plots.org/attestations/release-root/v1" -spec_version = "1.0.0" -max_json_bytes = 16777216 -max_root_keys = 64 -max_root_versions = 4096 -min_root_keys = 3 -min_root_threshold = 2 -min_root_operators = 2 -min_role_keys = 3 -min_role_threshold = 2 -min_role_operators = 2 -max_clock_skew_minutes = 10 -min_root_remaining_minutes = 60 -max_root_lifetime_days = 825 -max_authorization_age_hours = 168 -allow_bootstrap_only_epoch_one = true -require_root_role_disjoint = true -require_root_operator_disjoint = true -require_governance_emergency_disjoint = true -allowed_keytypes = ["ed25519"] -allowed_schemes = ["ed25519"] diff --git a/security/release_subjects.py b/security/release_subjects.py deleted file mode 100644 index b7e12f833a8b6df62db380ec3d2f44d1d0177219..0000000000000000000000000000000000000000 --- a/security/release_subjects.py +++ /dev/null @@ -1,46 +0,0 @@ -""" -Print canonical non-secret subjects that production release evidence must bind. -""" - -from __future__ import annotations - -import hashlib -import json -import logging -import sys -from pathlib import Path - -HERE = Path(__file__).resolve().parent -ROOT = HERE.parent -if str(HERE) not in sys.path: - sys.path.insert(0, str(HERE)) -if str(ROOT) not in sys.path: - sys.path.insert(0, str(ROOT)) - -import verify_release_evidence # noqa: E402 -from _utils._shared_logic import PROXY_VERSION # noqa: E402 -from source_tree import EXTENSION_ROOT, source_tree_sha256 # noqa: E402 - -logger = logging.getLogger(__name__) - - -def subjects() -> dict[str, object]: # ruff: ignore[undocumented-public-function] - supply = verify_release_evidence.SUPPLY - return { - "schema_version": 1, - "proxy_version": PROXY_VERSION, - "target_platform": verify_release_evidence.POLICY["target_platform"], - "requirements_lock_sha256": ( - hashlib.sha256((ROOT / supply["lock_file"]).read_bytes()).hexdigest() - ), - "python_sbom_sha256": ( - hashlib.sha256((ROOT / supply["sbom_file"]).read_bytes()).hexdigest() - ), - "runtime_source_sha256": verify_release_evidence._runtime_source_sha256(), - "source_tree_sha256": source_tree_sha256(EXTENSION_ROOT), - "base_image_index_digest": supply["base_image"]["index_digest"], - } - - -if __name__ == "__main__": - sys.stdout.write(json.dumps(subjects(), sort_keys=True) + "\n") diff --git a/security/release_transparency_policy.toml b/security/release_transparency_policy.toml deleted file mode 100644 index 73f8b068b3eb123de12cf35ba59b0dbc8875bcd4..0000000000000000000000000000000000000000 --- a/security/release_transparency_policy.toml +++ /dev/null @@ -1,22 +0,0 @@ -schema_version = 1 -attestation_schema_version = 1 -signature_record_schema_version = 1 -final_record_schema_version = 1 -binding_receipt_schema_version = 1 -verifier_protocol_version = 1 -binder_protocol_version = 1 -publication_predicate_type = "https://scikit-plots.org/attestations/release-publication/v1" -attestation_predicate_type = "https://scikit-plots.org/attestations/release-post-publication/v1" -final_record_predicate_type = "https://scikit-plots.org/attestations/release-final-record/v1" -max_publication_json_bytes = 1048576 -max_publisher_result_bytes = 262144 -max_external_verifier_evidence_bytes = 1048576 -max_adapter_output_bytes = 262144 -adapter_timeout_seconds = 300 -max_artifacts = 32 -max_result_age_minutes = 30 -max_clock_skew_minutes = 10 -max_attestation_age_hours = 72 -record_name = "release-publication-record.json" -allowed_immutability = ["object-lock", "content-addressed", "versioned-create-only", "release-create-only"] -allowed_binding_types = ["release-asset", "object-store-object", "oci-referrer", "registry-attestation"] diff --git a/security/release_witness_policy.toml b/security/release_witness_policy.toml deleted file mode 100644 index e0f52c9df8ef96a1b39aa279a66fe8d9116ed006..0000000000000000000000000000000000000000 --- a/security/release_witness_policy.toml +++ /dev/null @@ -1,21 +0,0 @@ -schema_version = 1 -previous_checkpoint_schema_version = 1 -log_protocol_version = 1 -verifier_protocol_version = 1 -witness_protocol_version = 1 -anchor_protocol_version = 1 -witness_record_schema_version = 1 -witness_receipt_schema_version = 1 -witness_record_predicate_type = "https://scikit-plots.org/attestations/release-transparency-witness/v1" -record_name = "release-transparency-witness-record.json" -max_json_bytes = 1048576 -max_adapter_output_bytes = 262144 -adapter_timeout_seconds = 300 -max_result_age_minutes = 30 -max_clock_skew_minutes = 10 -max_witnesses = 8 -min_witnesses = 2 -min_witness_operators = 2 -allow_bootstrap_without_previous_checkpoint = false -allowed_immutability = ["object-lock", "content-addressed", "versioned-create-only", "release-create-only"] -allowed_binding_types = ["release-asset", "object-store-object", "oci-referrer", "registry-attestation"] diff --git a/security/seal_release_governance.py b/security/seal_release_governance.py deleted file mode 100644 index 28c848a3d0c2337e40dd2dc36e7288e21c62fbd9..0000000000000000000000000000000000000000 --- a/security/seal_release_governance.py +++ /dev/null @@ -1,1754 +0,0 @@ -""" -Cryptographically seal release-governance candidates with threshold Ed25519 roots. - -Run 154 constructs and archives a governance candidate. This module is the acceptance -boundary: it does not trust Run 154's semantic ``signatureVerified`` booleans. Instead, -it verifies fresh signatures by the exact selected governance keys over an exact, -canonical candidate subject, and maintains a TUF-style versioned root chain. -""" - -from __future__ import annotations - -import argparse -import base64 -import binascii -import hashlib -import json -import logging -import os -import re -import shutil -import sys -import tempfile -from datetime import datetime, timedelta, timezone -from pathlib import Path -from typing import Any, Iterable - -import tomllib - -try: # package import - from . import govern_release_history as governance -except (ImportError, ValueError) as exc: # direct Space-style/script/importlib loading - import importlib.util - - _gov_path = Path(__file__).resolve().parent / "govern_release_history.py" - _gov_name = "_release_governance_for_root_seal" - if _gov_name in sys.modules: - governance = sys.modules[_gov_name] - else: - _spec = importlib.util.spec_from_file_location(_gov_name, _gov_path) - if _spec is None or _spec.loader is None: - raise ImportError("cannot load govern_release_history") from exc - governance = importlib.util.module_from_spec(_spec) - sys.modules[_gov_name] = governance - _spec.loader.exec_module(governance) - -HERE = Path(__file__).resolve().parent -POLICY = tomllib.loads((HERE / "release_root_policy.toml").read_text()) -PREDICATE_TYPE = str(POLICY["predicate_type"]) -_HEX64 = re.compile(r"^[0-9a-f]{64}$") -_ID = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:/@+-]{0,511}$") -_CHUNK = 1024 * 1024 -_ZERO_HASH = "0" * 64 -logger = logging.getLogger(__name__) - - -class RootTrustError(RuntimeError): - """Cryptographic root or governance-authorization invariant failed.""" - - -def _fail(code: str) -> None: - raise RootTrustError(code) - - -def _canonical_bytes(value: dict[str, Any]) -> bytes: - return ( - json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False) - + "\n" - ).encode("utf-8") - - -def _sha_bytes(raw: bytes) -> str: - return hashlib.sha256(raw).hexdigest() - - -def _sha(path: Path) -> str: - h = hashlib.sha256() - with path.open("rb") as handle: - for chunk in iter(lambda: handle.read(_CHUNK), b""): - h.update(chunk) - return h.hexdigest() - - -def _write(path: Path, value: dict[str, Any]) -> None: - path.write_bytes(_canonical_bytes(value)) - - -def _loads(raw: bytes, code: str) -> dict[str, Any]: - def hook(pairs): - result: dict[str, Any] = {} - for key, value in pairs: - if key in result: - _fail(code + "_DUPLICATE_KEY") - result[key] = value - return result - - try: - value = json.loads(raw.decode("utf-8"), object_pairs_hook=hook) - except RootTrustError: - raise - except Exception as exc: - raise RootTrustError(code + "_JSON_INVALID") from exc - if not isinstance(value, dict): - _fail(code + "_SCHEMA_INVALID") - return value - - -def _read( - path: Path, code: str, *, canonical: bool = True -) -> tuple[dict[str, Any], bytes]: - if path.is_symlink() or not path.is_file(): - _fail(code + "_INVALID") - if path.stat().st_size > int(POLICY["max_json_bytes"]): - _fail(code + "_TOO_LARGE") - raw = path.read_bytes() - doc = _loads(raw, code) - if canonical and raw != _canonical_bytes(doc): - _fail(code + "_NOT_CANONICAL") - return doc, raw - - -def _identity(value: Any, code: str) -> str: - if not isinstance(value, str): - _fail(code) - value = value.strip() - if ( - not value - or len(value) > 512 # ruff: ignore[magic-value-comparison] - or ".." in value - or "?" in value - or "#" in value - or "\x00" in value - ): - _fail(code) - if ( - any( - ord(ch) < 32 or ord(ch) == 127 # ruff: ignore[magic-value-comparison] - for ch in value # lint - ) - or _ID.fullmatch(value) is None - ): - _fail(code) - return value - - -def _hex(value: Any, code: str) -> str: - if not isinstance(value, str) or _HEX64.fullmatch(value) is None: - _fail(code) - return value - - -def _size(value: Any, code: str) -> int: - if not isinstance(value, int) or isinstance(value, bool) or value < 0: - _fail(code) - return value - - -def _timestamp_value(value: Any, code: str) -> datetime: - if not isinstance(value, str) or not value.endswith("Z"): - _fail(code + "_INVALID") - try: - parsed = datetime.fromisoformat(value[:-1] + "+00:00").astimezone(timezone.utc) - except ValueError as exc: - raise RootTrustError(code + "_INVALID") from exc - return parsed - - -def _timestamp(value: Any, code: str) -> str: - _timestamp_value(value, code) - return str(value) - - -def _b64(value: Any, code: str, *, expected_len: int) -> bytes: - _lint = len(value) > 4096 # ruff: ignore[magic-value-comparison] - if not isinstance(value, str) or not value or _lint: - _fail(code) - try: - raw = base64.b64decode(value.encode("ascii"), validate=True) - except (UnicodeEncodeError, binascii.Error) as exc: - raise RootTrustError(code) from exc - if len(raw) != expected_len or base64.b64encode(raw).decode("ascii") != value: - _fail(code) - return raw - - -def _ed25519_verify( - public_raw: bytes, signature_raw: bytes, message: bytes, code: str -) -> None: - try: - from cryptography.exceptions import ( # ruff: ignore[import-outside-top-level] - InvalidSignature, - ) - from cryptography.hazmat.primitives.asymmetric.ed25519 import ( # ruff: ignore[import-outside-top-level] - Ed25519PublicKey, - ) - # fail closed if deployment omitted the crypto backend - except Exception as exc: - raise RootTrustError("ROOT_CRYPTO_BACKEND_UNAVAILABLE") from exc - try: - Ed25519PublicKey.from_public_bytes(public_raw).verify(signature_raw, message) - except InvalidSignature as exc: - raise RootTrustError(code) from exc - except Exception as exc: - raise RootTrustError(code + "_KEY_INVALID") from exc - - -def _artifact(path: Path, name: str | None = None) -> dict[str, Any]: - return { - "name": name or path.name, - "sha256": _sha(path), - "size": path.stat().st_size, - } - - -def _artifact_doc(doc: dict[str, Any], name: str) -> dict[str, Any]: - raw = _canonical_bytes(doc) - return {"name": name, "sha256": _sha_bytes(raw), "size": len(raw)} - - -def _regular_dir(path: Path, code: str) -> Path: - if path.is_symlink() or not path.is_dir(): - _fail(code) - return path.resolve() - - -def _outside(path: Path, protected: Iterable[Path], code: str) -> Path: - target = path.expanduser().resolve() - for item in protected: - root = item.resolve() - if target == root or root in target.parents: - _fail(code) - return target - - -def _root_key(value: Any, key_id: str) -> dict[str, Any]: - expected = {"keytype", "scheme", "identity", "operator", "expires", "keyval"} - if not isinstance(value, dict) or set(value) != expected: - _fail("ROOT_KEY_SCHEMA_INVALID") - if value.get("keytype") not in set(POLICY["allowed_keytypes"]) or value.get( - "scheme" - ) not in set(POLICY["allowed_schemes"]): - _fail("ROOT_KEY_ALGORITHM_INVALID") - keyval = value.get("keyval") - if not isinstance(keyval, dict) or set(keyval) != {"public"}: - _fail("ROOT_KEYVAL_SCHEMA_INVALID") - public_text = keyval.get("public") - _b64(public_text, "ROOT_PUBLIC_KEY_INVALID", expected_len=32) - return { - "keytype": "ed25519", - "scheme": "ed25519", - "identity": _identity(value.get("identity"), "ROOT_KEY_IDENTITY_INVALID"), - "operator": _identity(value.get("operator"), "ROOT_KEY_OPERATOR_INVALID"), - "expires": _timestamp(value.get("expires"), "ROOT_KEY_EXPIRES"), - "keyval": {"public": public_text}, - } - - -def _role( - value: Any, - code: str, - *, - min_keys: int, - min_threshold: int, - min_operators: int, - keys: dict[str, dict[str, Any]], -) -> dict[str, Any]: - if not isinstance(value, dict) or set(value) != {"keyids", "threshold"}: - _fail(code + "_SCHEMA_INVALID") - keyids_raw = value.get("keyids") - if not isinstance(keyids_raw, list) or not ( - min_keys <= len(keyids_raw) <= int(POLICY["max_root_keys"]) - ): - _fail(code + "_KEY_COUNT_INVALID") - keyids = [_identity(x, code + "_KEY_ID_INVALID") for x in keyids_raw] - if len(set(keyids)) != len(keyids) or any(x not in keys for x in keyids): - _fail(code + "_KEY_SET_INVALID") - threshold = _size(value.get("threshold"), code + "_THRESHOLD_INVALID") - if threshold < min_threshold or threshold > len(keyids): - _fail(code + "_THRESHOLD_INVALID") - operators = {keys[x]["operator"] for x in keyids} - if len(operators) < min_operators: - _fail(code + "_OPERATOR_COUNT_INVALID") - return {"keyids": sorted(keyids), "threshold": threshold} - - -def _validate_root_signed( # ruff: ignore[too-many-branches] - signed: dict[str, Any], -) -> dict[str, Any]: - expected = { - "_type", - "specVersion", - "version", - "governanceId", - "issuedAt", - "expires", - "keys", - "roles", - } - if ( - set(signed) != expected - or signed.get("_type") != "root" - or signed.get("specVersion") != str(POLICY["spec_version"]) - ): - _fail("ROOT_SIGNED_SCHEMA_INVALID") - version = _size(signed.get("version"), "ROOT_VERSION_INVALID") - if version < 1: - _fail("ROOT_VERSION_INVALID") - governance_id = _identity(signed.get("governanceId"), "ROOT_GOVERNANCE_ID_INVALID") - issued = _timestamp_value(signed.get("issuedAt"), "ROOT_ISSUED_AT") - expires = _timestamp_value(signed.get("expires"), "ROOT_EXPIRES") - if expires <= issued or expires - issued > timedelta( - days=int(POLICY["max_root_lifetime_days"]) - ): - _fail("ROOT_LIFETIME_INVALID") - keys_raw = signed.get("keys") - if not isinstance(keys_raw, dict) or not ( - int(POLICY["min_root_keys"]) <= len(keys_raw) <= int(POLICY["max_root_keys"]) - ): - _fail("ROOT_KEY_COUNT_INVALID") - keys: dict[str, dict[str, Any]] = {} - for raw_id, value in keys_raw.items(): - key_id = _identity(raw_id, "ROOT_KEY_ID_INVALID") - keys[key_id] = _root_key(value, key_id) - roles_raw = signed.get("roles") - if not isinstance(roles_raw, dict) or set(roles_raw) != { - "root", - "governance", - "emergency", - }: - _fail("ROOT_ROLES_SCHEMA_INVALID") - roles = { - "root": _role( - roles_raw["root"], - "ROOT_ROLE_ROOT", - min_keys=int(POLICY["min_root_keys"]), - min_threshold=int(POLICY["min_root_threshold"]), - min_operators=int(POLICY["min_root_operators"]), - keys=keys, - ), - "governance": _role( - roles_raw["governance"], - "ROOT_ROLE_GOVERNANCE", - min_keys=int(POLICY["min_role_keys"]), - min_threshold=int(POLICY["min_role_threshold"]), - min_operators=int(POLICY["min_role_operators"]), - keys=keys, - ), - "emergency": _role( - roles_raw["emergency"], - "ROOT_ROLE_EMERGENCY", - min_keys=int(POLICY["min_role_keys"]), - min_threshold=int(POLICY["min_role_threshold"]), - min_operators=int(POLICY["min_role_operators"]), - keys=keys, - ), - } - root_set = set(roles["root"]["keyids"]) - governance_set = set(roles["governance"]["keyids"]) - emergency_set = set(roles["emergency"]["keyids"]) - if set(keys) != root_set | governance_set | emergency_set: - _fail("ROOT_UNUSED_KEY_INVALID") - if bool(POLICY.get("require_root_role_disjoint", True)) and root_set & ( - governance_set | emergency_set - ): - _fail("ROOT_ROLE_NOT_DISJOINT") - if ( - bool(POLICY.get("require_governance_emergency_disjoint", True)) - and governance_set & emergency_set - ): - _fail("ROOT_GOVERNANCE_EMERGENCY_NOT_DISJOINT") - if bool(POLICY.get("require_root_operator_disjoint", True)): - root_ops = {keys[x]["operator"] for x in root_set} - online_ops = {keys[x]["operator"] for x in governance_set | emergency_set} - if root_ops & online_ops: - _fail("ROOT_OPERATOR_NOT_DISJOINT") - # Every active key remains valid for the complete lifetime of the root metadata. - for role in roles.values(): - for key_id in role["keyids"]: - if _timestamp_value(keys[key_id]["expires"], "ROOT_KEY_EXPIRES") < expires: - _fail("ROOT_KEY_EXPIRES_BEFORE_ROOT") - normalized = { - "_type": "root", - "specVersion": str(POLICY["spec_version"]), - "version": version, - "governanceId": governance_id, - "issuedAt": issued.isoformat(timespec="seconds").replace("+00:00", "Z"), - "expires": expires.isoformat(timespec="seconds").replace("+00:00", "Z"), - "keys": {k: keys[k] for k in sorted(keys)}, - "roles": roles, - } - if signed != normalized: - _fail("ROOT_SIGNED_NOT_NORMALIZED") - return { - "signed": normalized, - "version": version, - "governance_id": governance_id, - "issued": issued, - "expires": expires, - "keys": keys, - "roles": roles, - } - - -def _validate_root_envelope(doc: dict[str, Any]) -> dict[str, Any]: - if set(doc) != {"signatures", "signed"} or not isinstance(doc.get("signed"), dict): - _fail("ROOT_METADATA_SCHEMA_INVALID") - info = _validate_root_signed(doc["signed"]) - signatures = doc.get("signatures") - if ( - not isinstance(signatures, list) - or len(signatures) > int(POLICY["max_root_keys"]) * 2 - ): - _fail("ROOT_SIGNATURES_SCHEMA_INVALID") - normalized_sigs: list[dict[str, str]] = [] - seen: set[str] = set() - for item in signatures: - if not isinstance(item, dict) or set(item) != {"keyid", "sig"}: - _fail("ROOT_SIGNATURE_SCHEMA_INVALID") - key_id = _identity(item.get("keyid"), "ROOT_SIGNATURE_KEY_ID_INVALID") - if key_id in seen: - _fail("ROOT_SIGNATURE_KEY_DUPLICATE") - seen.add(key_id) - sig = item.get("sig") - _b64(sig, "ROOT_SIGNATURE_ENCODING_INVALID", expected_len=64) - normalized_sigs.append({"keyid": key_id, "sig": sig}) - normalized_sigs.sort(key=lambda x: x["keyid"]) - if signatures != normalized_sigs: - _fail("ROOT_SIGNATURES_NOT_NORMALIZED") - info["envelope"] = {"signatures": normalized_sigs, "signed": info["signed"]} - info["signed_bytes"] = _canonical_bytes(info["signed"]) - info["sha256"] = _sha_bytes(_canonical_bytes(info["envelope"])) - info["signature_map"] = {x["keyid"]: x["sig"] for x in normalized_sigs} - return info - - -def _verify_role_signatures( - envelope_info: dict[str, Any], - trusted_info: dict[str, Any], - role_name: str, - *, - at: datetime, - code: str, -) -> list[str]: - role = trusted_info["roles"][role_name] - valid: list[str] = [] - for key_id in role["keyids"]: - sig_text = envelope_info["signature_map"].get(key_id) - if sig_text is None: - continue - key = trusted_info["keys"][key_id] - if _timestamp_value(key["expires"], "ROOT_KEY_EXPIRES") < at: - continue - _ed25519_verify( - _b64(key["keyval"]["public"], "ROOT_PUBLIC_KEY_INVALID", expected_len=32), - _b64(sig_text, "ROOT_SIGNATURE_ENCODING_INVALID", expected_len=64), - envelope_info["signed_bytes"], - code + "_SIGNATURE_INVALID", - ) - valid.append(key_id) - if len(valid) < role["threshold"]: - _fail(code + "_THRESHOLD_NOT_MET") - return sorted(valid) - - -def _assert_signature_keys( - signature_map: dict[str, str], allowed: set[str], code: str -) -> None: - if not set(signature_map).issubset(allowed): - _fail(code + "_UNAUTHORIZED_SIGNATURE_KEY") - - -def _verify_bootstrap_root( - root_doc: dict[str, Any], expected_sha256: str, *, now: datetime -) -> dict[str, Any]: - info = _validate_root_envelope(root_doc) - _assert_signature_keys( - info["signature_map"], set(info["roles"]["root"]["keyids"]), "ROOT_BOOTSTRAP" - ) - if info["version"] != 1: - _fail("ROOT_BOOTSTRAP_VERSION_INVALID") - if info["sha256"] != _hex(expected_sha256, "ROOT_BOOTSTRAP_PIN_INVALID"): - _fail("ROOT_BOOTSTRAP_PIN_MISMATCH") - if info["issued"] > now + timedelta(minutes=int(POLICY["max_clock_skew_minutes"])): - _fail("ROOT_ISSUED_FROM_FUTURE") - if info["expires"] < now + timedelta( - minutes=int(POLICY["min_root_remaining_minutes"]) - ): - _fail("ROOT_EXPIRED_OR_FREEZE_RISK") - _verify_role_signatures( - info, info, "root", at=info["issued"], code="ROOT_BOOTSTRAP_SELF_SIGNATURE" - ) - return info - - -def _verify_rotation( - previous: dict[str, Any], current: dict[str, Any], *, now: datetime -) -> dict[str, Any]: - allowed_signature_keys = set(previous["roles"]["root"]["keyids"]) | set( - current["roles"]["root"]["keyids"] - ) - _assert_signature_keys( - current["signature_map"], allowed_signature_keys, "ROOT_ROTATION" - ) - if current["governance_id"] != previous["governance_id"]: - _fail("ROOT_ROTATION_GOVERNANCE_ID_MISMATCH") - if current["version"] != previous["version"] + 1: - _fail("ROOT_ROTATION_VERSION_NOT_CONSECUTIVE") - if current["issued"] < previous["issued"]: - _fail("ROOT_ROTATION_ISSUED_AT_ROLLBACK") - if current["issued"] > previous["expires"] + timedelta( - minutes=int(POLICY["max_clock_skew_minutes"]) - ): - _fail("ROOT_ROTATION_AFTER_OLD_ROOT_EXPIRY") - if current["issued"] > now + timedelta( - minutes=int(POLICY["max_clock_skew_minutes"]) - ): - _fail("ROOT_ROTATION_FROM_FUTURE") - if current["expires"] < now + timedelta( - minutes=int(POLICY["min_root_remaining_minutes"]) - ): - _fail("ROOT_ROTATION_EXPIRED_OR_FREEZE_RISK") - _verify_role_signatures( - current, previous, "root", at=current["issued"], code="ROOT_ROTATION_OLD_ROOT" - ) - _verify_role_signatures( - current, current, "root", at=current["issued"], code="ROOT_ROTATION_NEW_ROOT" - ) - return current - - -def _root_head(governance_id: str, roots: list[dict[str, Any]]) -> str: - head = _ZERO_HASH - for item in roots: - info = _validate_root_envelope(item) - head = _sha_bytes( - _canonical_bytes( - { - "governanceId": governance_id, - "previous": head, - "rootVersion": info["version"], - "rootSha256": info["sha256"], - } - ) - ) - return head - - -def _validate_root_bundle( - bundle: dict[str, Any], - *, - expected_bootstrap_pin: str | None = None, - now: datetime | None = None, - require_current_fresh: bool = True, -) -> dict[str, Any]: - expected = { - "schemaVersion", - "predicateType", - "status", - "governanceId", - "bootstrapRootSha256", - "roots", - } - if ( - set(bundle) != expected - or bundle.get("schemaVersion") != int(POLICY["root_bundle_schema_version"]) - or bundle.get("predicateType") != PREDICATE_TYPE - or bundle.get("status") != "trusted-release-root-chain" - ): - _fail("ROOT_BUNDLE_SCHEMA_INVALID") - governance_id = _identity( - bundle.get("governanceId"), "ROOT_BUNDLE_GOVERNANCE_ID_INVALID" - ) - bootstrap_pin = _hex( - bundle.get("bootstrapRootSha256"), "ROOT_BUNDLE_BOOTSTRAP_PIN_INVALID" - ) - if expected_bootstrap_pin is not None and bootstrap_pin != _hex( - expected_bootstrap_pin, "ROOT_EXPECTED_BOOTSTRAP_PIN_INVALID" - ): - _fail("ROOT_BUNDLE_BOOTSTRAP_PIN_MISMATCH") - roots = bundle.get("roots") - if not isinstance(roots, list) or not ( - 1 <= len(roots) <= int(POLICY["max_root_versions"]) - ): - _fail("ROOT_BUNDLE_ROOT_COUNT_INVALID") - infos = [ - ( - _validate_root_envelope(x) - if isinstance(x, dict) - else _fail("ROOT_BUNDLE_ROOT_SCHEMA_INVALID") - ) - for x in roots - ] - first = infos[0] - if ( - first["version"] != 1 - or first["governance_id"] != governance_id - or first["sha256"] != bootstrap_pin - ): - _fail("ROOT_BUNDLE_GENESIS_MISMATCH") - _assert_signature_keys( - first["signature_map"], - set(first["roles"]["root"]["keyids"]), - "ROOT_BUNDLE_GENESIS", - ) - _verify_role_signatures( - first, - first, - "root", - at=first["issued"], - code="ROOT_BUNDLE_GENESIS_SELF_SIGNATURE", - ) - previous = first - for current in infos[1:]: - if ( - current["governance_id"] != governance_id - or current["version"] != previous["version"] + 1 - ): - _fail("ROOT_BUNDLE_VERSION_CHAIN_INVALID") - if current["issued"] < previous["issued"] or current["issued"] > previous[ - "expires" - ] + timedelta(minutes=int(POLICY["max_clock_skew_minutes"])): - _fail("ROOT_BUNDLE_TIME_CHAIN_INVALID") - allowed_signature_keys = set(previous["roles"]["root"]["keyids"]) | set( - current["roles"]["root"]["keyids"] - ) - _assert_signature_keys( - current["signature_map"], allowed_signature_keys, "ROOT_BUNDLE_ROTATION" - ) - _verify_role_signatures( - current, previous, "root", at=current["issued"], code="ROOT_BUNDLE_OLD_ROOT" - ) - _verify_role_signatures( - current, current, "root", at=current["issued"], code="ROOT_BUNDLE_NEW_ROOT" - ) - previous = current - current_time = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - if require_current_fresh and previous["issued"] > current_time + timedelta( - minutes=int(POLICY["max_clock_skew_minutes"]) - ): - _fail("ROOT_CURRENT_ISSUED_FROM_FUTURE") - if require_current_fresh and previous["expires"] < current_time + timedelta( - minutes=int(POLICY["min_root_remaining_minutes"]) - ): - _fail("ROOT_CURRENT_EXPIRED_OR_FREEZE_RISK") - normalized = { - "schemaVersion": int(POLICY["root_bundle_schema_version"]), - "predicateType": PREDICATE_TYPE, - "status": "trusted-release-root-chain", - "governanceId": governance_id, - "bootstrapRootSha256": bootstrap_pin, - "roots": roots, - } - if bundle != normalized: - _fail("ROOT_BUNDLE_NOT_NORMALIZED") - return { - "bundle": normalized, - "governance_id": governance_id, - "bootstrap_pin": bootstrap_pin, - "infos": infos, - "current": infos[-1], - "chain_head_sha256": _root_head(governance_id, roots), - "bundle_sha256": _sha_bytes(_canonical_bytes(normalized)), - } - - -def _policy_role(policy: dict[str, Any], role_name: str) -> dict[str, Any]: - field = "policyAuthority" if role_name == "governance" else "emergencyAuthority" - value = policy.get(field) - if not isinstance(value, dict) or set(value) != {"threshold", "members"}: - _fail("ROOT_POLICY_ROLE_SCHEMA_INVALID") - return value - - -def _assert_role_matches_policy( - root_info: dict[str, Any], - role_name: str, - policy: dict[str, Any], - revoked: set[str], - code: str, -) -> None: - role = root_info["roles"][role_name] - policy_role = _policy_role(policy, role_name) - members = policy_role["members"] - if role["threshold"] != policy_role["threshold"]: - _fail(code + "_THRESHOLD_MISMATCH") - policy_ids = sorted(m["keyId"] for m in members) - if role["keyids"] != policy_ids: - _fail(code + "_KEY_SET_MISMATCH") - by_id = {m["keyId"]: m for m in members} - for key_id in role["keyids"]: - if key_id in revoked: - _fail(code + "_REVOKED_KEY") - key = root_info["keys"][key_id] - member = by_id[key_id] - if ( - key["identity"] != member["identity"] - or key["operator"] != member["operator"] - ): - _fail(code + "_MEMBER_BINDING_MISMATCH") - - -# ── Verified-state document names ──────────────────────────────────────── -# -# One constant per release-evidence document. The values are the on-disk -# artifact names and are unchanged; only the repetition is removed. Reading -# a document by a name that says which document it is also keeps static -# analysis from reading the filename token 'trusted' as 'confidential': -# these files carry published Merkle roots and SHA-256 digests, which must -# stay in clear text for any third party to verify them. -_DOC_GOVERNANCE_STATE = "trusted-governance-state.json" -_DOC_HISTORY_STATE = "trusted-history-state.json" -_DOC_RELEASE_ROOT_STATE = "trusted-release-root-state.json" - - -def _validate_run154_dir( # ruff: ignore[too-many-branches] - path: Path, -) -> dict[str, Any]: - root = _regular_dir(path, "RUN154_DIRECTORY_INVALID") - allowed = { - "release-governance-bundle.json", - _DOC_GOVERNANCE_STATE, - "release-governance-recovery-snapshot.json", - "release-governance-receipt.json", - "governance-transition-proposal.json", - "approval-evidence", - "archive-results", - _DOC_HISTORY_STATE, - "release-history-bundle.json", - } - if {p.name for p in root.iterdir()} != allowed: - _fail("RUN154_DIRECTORY_ALLOWLIST_MISMATCH") - for dirname in ("approval-evidence", "archive-results"): - d = root / dirname - if d.is_symlink() or not d.is_dir() or not any(d.iterdir()): - _fail("RUN154_EVIDENCE_DIRECTORY_INVALID") - for item in d.iterdir(): - if item.is_symlink() or not item.is_file(): - _fail("RUN154_EVIDENCE_ENTRY_INVALID") - _read(item, "RUN154_EVIDENCE_ENTRY") - try: - verified = governance.verify_governance_bundle( - bundle_path=root / "release-governance-bundle.json", - state_path=root / _DOC_GOVERNANCE_STATE, - ) - except governance.GovernanceError as exc: - raise RootTrustError("RUN154_GOVERNANCE_INVALID:" + str(exc)) from exc - bundle, bundle_raw = _read(root / "release-governance-bundle.json", "RUN154_BUNDLE") - state, state_raw = _read(root / _DOC_GOVERNANCE_STATE, "RUN154_STATE") - snapshot, snapshot_raw = _read( - root / "release-governance-recovery-snapshot.json", "RUN154_SNAPSHOT" - ) - receipt, _ = _read(root / "release-governance-receipt.json", "RUN154_RECEIPT") - proposal, proposal_raw = _read( - root / "governance-transition-proposal.json", "RUN154_PROPOSAL" - ) - history_state, history_state_raw = _read( - root / _DOC_HISTORY_STATE, "RUN154_HISTORY_STATE" - ) - history_bundle, history_bundle_raw = _read( - root / "release-history-bundle.json", "RUN154_HISTORY_BUNDLE" - ) - if snapshot != { - "schemaVersion": int(governance.POLICY["recovery_snapshot_schema_version"]), - "governanceState": state, - "governanceBundle": bundle, - "historyState": history_state, - "historyBundle": history_bundle, - }: - _fail("RUN154_SNAPSHOT_CONTENT_MISMATCH") - expected_receipt = { - "schemaVersion", - "status", - "governanceId", - "historyId", - "epoch", - "policyVersion", - "proposalSha256", - "approvalEvidenceSha256", - "history", - "governance", - "archives", - } - if ( - set(receipt) != expected_receipt - or receipt.get("status") != "governed" - or receipt.get("governanceId") != state.get("governanceId") - or receipt.get("historyId") != state.get("historyId") - or receipt.get("epoch") != state.get("epoch") - or receipt.get("policyVersion") != state.get("policy", {}).get("policyVersion") - ): - _fail("RUN154_RECEIPT_SCHEMA_INVALID") - artifacts = receipt.get("governance") - if not isinstance(artifacts, dict) or artifacts != { - "bundle": { - "name": "release-governance-bundle.json", - "sha256": _sha_bytes(bundle_raw), - "size": len(bundle_raw), - }, - "state": { - "name": _DOC_GOVERNANCE_STATE, - "sha256": _sha_bytes(state_raw), - "size": len(state_raw), - }, - "recoverySnapshot": { - "name": "release-governance-recovery-snapshot.json", - "sha256": _sha_bytes(snapshot_raw), - "size": len(snapshot_raw), - }, - }: - _fail("RUN154_RECEIPT_ARTIFACT_REBIND_FAILED") - if receipt.get("proposalSha256") != _sha_bytes(proposal_raw): - _fail("RUN154_PROPOSAL_RECEIPT_REBIND_FAILED") - entries = bundle.get("entries") - if not isinstance(entries, list) or not entries: - _fail("RUN154_GOVERNANCE_ENTRY_MISSING") - entry = entries[-1] - if ( - entry.get("proposalSha256") != _sha_bytes(proposal_raw) - or entry.get("proposal") != proposal - or entry.get("epoch") != state.get("epoch") - ): - _fail("RUN154_LATEST_ENTRY_REBIND_FAILED") - approval_docs = [] - approval_hashes = [] - for p in sorted((root / "approval-evidence").iterdir(), key=lambda x: x.name): - doc, raw = _read(p, "RUN154_APPROVAL_EVIDENCE") - approval_docs.append(doc) - approval_hashes.append(_sha_bytes(raw)) - if sorted(approval_hashes) != sorted(receipt.get("approvalEvidenceSha256", [])): - _fail("RUN154_APPROVAL_EVIDENCE_REBIND_FAILED") - if sorted( - _sha_bytes(_canonical_bytes(x)) for x in entry.get("approvals", []) - ) != sorted(approval_hashes): - _fail("RUN154_APPROVAL_ENTRY_REBIND_FAILED") - archive_hash_list = [_sha(p) for p in (root / "archive-results").iterdir()] - archive_hashes = set(archive_hash_list) - archives = receipt.get("archives") - if not isinstance(archives, list) or not archives: - _fail("RUN154_ARCHIVE_EVIDENCE_SCHEMA_INVALID") - referenced_archive_hashes: list[str] = [] - for item in archives: - if not isinstance(item, dict): - _fail("RUN154_ARCHIVE_EVIDENCE_SCHEMA_INVALID") - for field in ("bindEvidenceSha256", "verifyEvidenceSha256"): - value = item.get(field) - if value not in archive_hashes: - _fail("RUN154_ARCHIVE_EVIDENCE_REBIND_FAILED") - referenced_archive_hashes.append(value) - if ( - len(archive_hash_list) != len(referenced_archive_hashes) - or len(set(archive_hash_list)) != len(archive_hash_list) - or len(set(referenced_archive_hashes)) != len(referenced_archive_hashes) - or set(referenced_archive_hashes) != archive_hashes - ): - _fail("RUN154_ARCHIVE_EVIDENCE_SET_MISMATCH") - history_receipt = receipt.get("history") - if not isinstance(history_receipt, dict) or history_receipt != { - "sequence": state["history"]["sequence"], - "stateSha256": _sha_bytes(history_state_raw), - "bundleSha256": _sha_bytes(history_bundle_raw), - "chainHeadSha256": state["history"]["chainHeadSha256"], - }: - _fail("RUN154_HISTORY_REBIND_FAILED") - previous_policy = ( - bundle["genesis"]["initialPolicy"] - if len(entries) == 1 - else entries[-2]["nextPolicy"] - ) - revoked_before: set[str] = set() - for old in entries[:-1]: - revoked_before.update(old.get("revokedAuthorityKeyIds", [])) - return { - "root": root, - "verified": verified, - "bundle": bundle, - "state": state, - "snapshot": snapshot, - "receipt": receipt, - "proposal": proposal, - "entry": entry, - "previous_policy": previous_policy, - "final_policy": state["policy"], - "revoked_before": revoked_before, - "snapshot_sha256": _sha_bytes(snapshot_raw), - "snapshot_size": len(snapshot_raw), - "state_sha256": _sha_bytes(state_raw), - "bundle_sha256": _sha_bytes(bundle_raw), - "proposal_sha256": _sha_bytes(proposal_raw), - "history_state_sha256": _sha_bytes(history_state_raw), - "history_bundle_sha256": _sha_bytes(history_bundle_raw), - } - - -def _subject(candidate: dict[str, Any], auth_root: dict[str, Any]) -> dict[str, Any]: - entry = candidate["entry"] - role = "emergency" if entry.get("approvalRole") == "emergency" else "governance" - return { - "schemaVersion": int(POLICY["schema_version"]), - "predicateType": PREDICATE_TYPE + "/governance-authorization", - "governanceId": candidate["state"]["governanceId"], - "historyId": candidate["state"]["historyId"], - "epoch": candidate["state"]["epoch"], - "policyVersion": candidate["state"]["policy"]["policyVersion"], - "transitionId": entry["transitionId"], - "role": role, - "proposalSha256": candidate["proposal_sha256"], - "selectedKeyIds": sorted(entry["selectedApproverKeyIds"]), - "authorizingRoot": { - "version": auth_root["version"], - "sha256": auth_root["sha256"], - }, - "governanceState": { - "name": _DOC_GOVERNANCE_STATE, - "sha256": candidate["state_sha256"], - "size": (candidate["root"] / _DOC_GOVERNANCE_STATE).stat().st_size, - }, - "governanceBundle": { - "name": "release-governance-bundle.json", - "sha256": candidate["bundle_sha256"], - "size": ( - (candidate["root"] / "release-governance-bundle.json").stat().st_size - ), - }, - "recoverySnapshot": { - "name": "release-governance-recovery-snapshot.json", - "sha256": candidate["snapshot_sha256"], - "size": candidate["snapshot_size"], - }, - "history": candidate["state"]["history"], - } - - -def _validate_authorization_signature( - doc: dict[str, Any], - *, - subject: dict[str, Any], - root_info: dict[str, Any], - role_name: str, - expected_member: dict[str, Any], - now: datetime, - enforce_freshness: bool = True, -) -> dict[str, Any]: - if set(doc) != {"signature", "signed"} or not isinstance(doc.get("signed"), dict): - _fail("ROOT_AUTH_SIGNATURE_SCHEMA_INVALID") - signed = doc["signed"] - expected = { - "schemaVersion", - "keyId", - "identity", - "operator", - "decision", - "rootVersion", - "subjectSha256", - "signedAt", - } - if ( - set(signed) != expected - or signed.get("schemaVersion") != int(POLICY["signature_schema_version"]) - or signed.get("decision") != "approve" - ): - _fail("ROOT_AUTH_SIGNATURE_SIGNED_SCHEMA_INVALID") - key_id = _identity(signed.get("keyId"), "ROOT_AUTH_KEY_ID_INVALID") - if key_id != expected_member["keyId"]: - _fail("ROOT_AUTH_SELECTED_KEY_MISMATCH") - key = root_info["keys"].get(key_id) - if key is None or key_id not in root_info["roles"][role_name]["keyids"]: - _fail("ROOT_AUTH_KEY_NOT_AUTHORIZED") - if ( - signed.get("identity") != expected_member["identity"] - or signed.get("operator") != expected_member["operator"] - or key["identity"] != expected_member["identity"] - or key["operator"] != expected_member["operator"] - ): - _fail("ROOT_AUTH_IDENTITY_BINDING_MISMATCH") - if signed.get("rootVersion") != root_info["version"] or signed.get( - "subjectSha256" - ) != _sha_bytes(_canonical_bytes(subject)): - _fail("ROOT_AUTH_SUBJECT_REBIND_FAILED") - signed_at = _timestamp_value(signed.get("signedAt"), "ROOT_AUTH_SIGNED_AT") - if signed.get("signedAt") != signed_at.isoformat(timespec="seconds").replace( - "+00:00", "Z" - ): - _fail("ROOT_AUTH_SIGNED_AT_NOT_NORMALIZED") - if enforce_freshness: - if signed_at > now + timedelta(minutes=int(POLICY["max_clock_skew_minutes"])): - _fail("ROOT_AUTH_SIGNATURE_FROM_FUTURE") - if now - signed_at > timedelta( - hours=int(POLICY["max_authorization_age_hours"]) - ): - _fail("ROOT_AUTH_SIGNATURE_STALE") - if ( - signed_at < root_info["issued"] - or signed_at > root_info["expires"] - or signed_at > _timestamp_value(key["expires"], "ROOT_KEY_EXPIRES") - ): - _fail("ROOT_AUTH_SIGNATURE_OUTSIDE_KEY_VALIDITY") - signature = _b64( - doc.get("signature"), "ROOT_AUTH_SIGNATURE_ENCODING_INVALID", expected_len=64 - ) - _ed25519_verify( - _b64(key["keyval"]["public"], "ROOT_PUBLIC_KEY_INVALID", expected_len=32), - signature, - _canonical_bytes(signed), - "ROOT_AUTH_SIGNATURE_INVALID", - ) - normalized = {"signature": doc["signature"], "signed": signed} - if doc != normalized: - _fail("ROOT_AUTH_SIGNATURE_NOT_NORMALIZED") - return normalized - - -def _state_document( - *, - root_info: dict[str, Any], - bundle_sha: str, - chain_head: str, - candidate: dict[str, Any], -) -> dict[str, Any]: - return { - "schemaVersion": int(POLICY["root_state_schema_version"]), - "governanceId": root_info["governance_id"], - "rootVersion": root_info["version"], - "rootSha256": root_info["sha256"], - "rootExpires": root_info["signed"]["expires"], - "rootChainHeadSha256": chain_head, - "bundleSha256": bundle_sha, - "governance": { - "epoch": candidate["state"]["epoch"], - "policyVersion": candidate["state"]["policy"]["policyVersion"], - "stateSha256": candidate["state_sha256"], - "bundleSha256": candidate["bundle_sha256"], - "recoverySnapshotSha256": candidate["snapshot_sha256"], - "proposalSha256": candidate["proposal_sha256"], - }, - } - - -def verify_root_bundle( # ruff: ignore[undocumented-public-function] - *, - bundle_path: Path, - state_path: Path | None = None, - expected_bootstrap_root_sha256: str | None = None, - now: datetime | None = None, - require_current_fresh: bool = True, -) -> dict[str, Any]: - if expected_bootstrap_root_sha256 is None: - _fail("ROOT_EXPECTED_BOOTSTRAP_PIN_REQUIRED") - bundle, _ = _read(bundle_path, "ROOT_BUNDLE") - info = _validate_root_bundle( - bundle, - expected_bootstrap_pin=expected_bootstrap_root_sha256, - now=now, - require_current_fresh=require_current_fresh, - ) - if state_path is not None: - state, state_raw = _read(state_path, "ROOT_STATE") - expected = { - "schemaVersion", - "governanceId", - "rootVersion", - "rootSha256", - "rootExpires", - "rootChainHeadSha256", - "bundleSha256", - "governance", - } - if set(state) != expected or state.get("schemaVersion") != int( - POLICY["root_state_schema_version"] - ): - _fail("ROOT_STATE_SCHEMA_INVALID") - current = info["current"] - if ( - state.get("governanceId") != info["governance_id"] - or state.get("rootVersion") != current["version"] - or state.get("rootSha256") != current["sha256"] - or state.get("rootExpires") != current["signed"]["expires"] - or state.get("rootChainHeadSha256") != info["chain_head_sha256"] - or state.get("bundleSha256") != info["bundle_sha256"] - ): - _fail("ROOT_STATE_REBIND_FAILED") - governance_binding = state.get("governance") - if not isinstance(governance_binding, dict) or set(governance_binding) != { - "epoch", - "policyVersion", - "stateSha256", - "bundleSha256", - "recoverySnapshotSha256", - "proposalSha256", - }: - _fail("ROOT_STATE_GOVERNANCE_BINDING_INVALID") - for field in ( - "stateSha256", - "bundleSha256", - "recoverySnapshotSha256", - "proposalSha256", - ): - _hex(governance_binding.get(field), "ROOT_STATE_GOVERNANCE_HASH_INVALID") - _size(governance_binding.get("epoch"), "ROOT_STATE_GOVERNANCE_EPOCH_INVALID") - _size( - governance_binding.get("policyVersion"), - "ROOT_STATE_GOVERNANCE_POLICY_VERSION_INVALID", - ) - info["state"] = state - info["state_sha256"] = _sha_bytes(state_raw) - return info - - -def verify_sealed_governance( # ruff: ignore[too-many-branches] - *, - sealed_dir: Path, - expected_bootstrap_root_sha256: str, - now: datetime | None = None, - require_current_fresh: bool = False, -) -> dict[str, Any]: - """ - Offline-verify a complete Run 155 seal, including governance signatures. - - Historical verification defaults to allowing an expired *current* root because expiry - does not invalidate signatures that were made while the key/root was valid. Callers - performing a live acceptance check should set ``require_current_fresh=True``. - """ - root = _regular_dir(sealed_dir, "ROOT_SEALED_DIRECTORY_INVALID") - allowed = { - "release-root-bundle.json", - _DOC_RELEASE_ROOT_STATE, - "cryptographic-governance-authorization.json", - "active-root.json", - "release-governance-recovery-snapshot.json", - "release-root-receipt.json", - } - if {p.name for p in root.iterdir()} != allowed: - _fail("ROOT_SEALED_DIRECTORY_ALLOWLIST_MISMATCH") - current_time = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - bundle_info = verify_root_bundle( - bundle_path=root / "release-root-bundle.json", - state_path=root / _DOC_RELEASE_ROOT_STATE, - expected_bootstrap_root_sha256=expected_bootstrap_root_sha256, - now=current_time, - require_current_fresh=require_current_fresh, - ) - state, state_raw = _read(root / _DOC_RELEASE_ROOT_STATE, "ROOT_SEALED_STATE") - active_root, active_root_raw = _read( - root / "active-root.json", "ROOT_SEALED_ACTIVE_ROOT" - ) - if ( - active_root != bundle_info["current"]["envelope"] - or _sha_bytes(active_root_raw) != bundle_info["current"]["sha256"] - ): - _fail("ROOT_SEALED_ACTIVE_ROOT_REBIND_FAILED") - snapshot, snapshot_raw = _read( - root / "release-governance-recovery-snapshot.json", - "ROOT_SEALED_GOVERNANCE_SNAPSHOT", - ) - expected_snapshot_keys = { - "schemaVersion", - "governanceState", - "governanceBundle", - "historyState", - "historyBundle", - } - if set(snapshot) != expected_snapshot_keys or snapshot.get("schemaVersion") != int( - governance.POLICY["recovery_snapshot_schema_version"] - ): - _fail("ROOT_SEALED_GOVERNANCE_SNAPSHOT_SCHEMA_INVALID") - gov_state = snapshot.get("governanceState") - gov_bundle = snapshot.get("governanceBundle") - history_state = snapshot.get("historyState") - history_bundle = snapshot.get("historyBundle") - if not all( - isinstance(x, dict) - for x in (gov_state, gov_bundle, history_state, history_bundle) - ): - _fail("ROOT_SEALED_GOVERNANCE_SNAPSHOT_SCHEMA_INVALID") - with tempfile.TemporaryDirectory(prefix="root-seal-offline-governance-") as tmp: - sp = Path(tmp) / "state.json" - bp = Path(tmp) / "bundle.json" - sp.write_bytes(_canonical_bytes(gov_state)) - bp.write_bytes(_canonical_bytes(gov_bundle)) - try: - governance.verify_governance_bundle(bundle_path=bp, state_path=sp) - except governance.GovernanceError as exc: - raise RootTrustError("ROOT_SEALED_GOVERNANCE_INVALID:" + str(exc)) from exc - history_state_raw = _canonical_bytes(history_state) - history_bundle_raw = _canonical_bytes(history_bundle) - with tempfile.TemporaryDirectory(prefix="root-seal-offline-history-") as tmp: - hsp = Path(tmp) / "history-state.json" - hbp = Path(tmp) / "history-bundle.json" - hsp.write_bytes(history_state_raw) - hbp.write_bytes(history_bundle_raw) - try: - history_info = governance.history._validate_previous_history(hsp, hbp) - except governance.history.HistoryError as exc: - raise RootTrustError("ROOT_SEALED_HISTORY_INVALID:" + str(exc)) from exc - expected_history_binding = { - "sequence": history_info["sequence"], - "stateSha256": _sha_bytes(history_state_raw), - "bundleSha256": _sha_bytes(history_bundle_raw), - "chainHeadSha256": history_info["chain_head_sha256"], - } - if gov_state.get("history") != expected_history_binding: - _fail("ROOT_SEALED_HISTORY_REBIND_FAILED") - entries = gov_bundle.get("entries") - if not isinstance(entries, list) or not entries: - _fail("ROOT_SEALED_GOVERNANCE_ENTRY_MISSING") - entry = entries[-1] - previous_policy = ( - gov_bundle["genesis"]["initialPolicy"] - if len(entries) == 1 - else entries[-2]["nextPolicy"] - ) - revoked_before: set[str] = set() - for old in entries[:-1]: - revoked_before.update(old.get("revokedAuthorityKeyIds", [])) - proposal = entry.get("proposal") - if not isinstance(proposal, dict): - _fail("ROOT_SEALED_PROPOSAL_MISSING") - proposal_sha = _sha_bytes(_canonical_bytes(proposal)) - if proposal_sha != entry.get("proposalSha256"): - _fail("ROOT_SEALED_PROPOSAL_HASH_MISMATCH") - authorization, authorization_raw = _read( - root / "cryptographic-governance-authorization.json", - "ROOT_SEALED_AUTHORIZATION", - ) - expected_auth_keys = { - "schemaVersion", - "predicateType", - "status", - "subject", - "subjectSha256", - "rootVersion", - "rootSha256", - "role", - "selectedKeyIds", - "signatures", - } - if ( - set(authorization) != expected_auth_keys - or authorization.get("schemaVersion") - != int(POLICY["authorization_schema_version"]) - or authorization.get("predicateType") - != PREDICATE_TYPE + "/governance-authorization" - or authorization.get("status") != "cryptographically-authorized" - ): - _fail("ROOT_SEALED_AUTHORIZATION_SCHEMA_INVALID") - auth_version = _size( - authorization.get("rootVersion"), "ROOT_SEALED_AUTH_ROOT_VERSION_INVALID" - ) - auth_sha = _hex( - authorization.get("rootSha256"), "ROOT_SEALED_AUTH_ROOT_HASH_INVALID" - ) - matches = [ - x - for x in bundle_info["infos"] - if x["version"] == auth_version and x["sha256"] == auth_sha - ] - if len(matches) != 1: - _fail("ROOT_SEALED_AUTH_ROOT_NOT_IN_CHAIN") - auth_root = matches[0] - role_name = ( - "emergency" if entry.get("approvalRole") == "emergency" else "governance" - ) - if authorization.get("role") != role_name: - _fail("ROOT_SEALED_AUTH_ROLE_MISMATCH") - _assert_role_matches_policy( - auth_root, - role_name, - previous_policy, - revoked_before, - "ROOT_SEALED_AUTHORIZING_ROLE", - ) - other_role = "governance" if role_name == "emergency" else "emergency" - _assert_role_matches_policy( - auth_root, - other_role, - previous_policy, - revoked_before, - "ROOT_SEALED_AUTHORIZING_OTHER_ROLE", - ) - subject = { - "schemaVersion": int(POLICY["schema_version"]), - "predicateType": PREDICATE_TYPE + "/governance-authorization", - "governanceId": gov_state["governanceId"], - "historyId": gov_state["historyId"], - "epoch": gov_state["epoch"], - "policyVersion": gov_state["policy"]["policyVersion"], - "transitionId": entry["transitionId"], - "role": role_name, - "proposalSha256": proposal_sha, - "selectedKeyIds": sorted(entry["selectedApproverKeyIds"]), - "authorizingRoot": { - "version": auth_root["version"], - "sha256": auth_root["sha256"], - }, - "governanceState": { - "name": _DOC_GOVERNANCE_STATE, - "sha256": _sha_bytes(_canonical_bytes(gov_state)), - "size": len(_canonical_bytes(gov_state)), - }, - "governanceBundle": { - "name": "release-governance-bundle.json", - "sha256": _sha_bytes(_canonical_bytes(gov_bundle)), - "size": len(_canonical_bytes(gov_bundle)), - }, - "recoverySnapshot": { - "name": "release-governance-recovery-snapshot.json", - "sha256": _sha_bytes(snapshot_raw), - "size": len(snapshot_raw), - }, - "history": gov_state["history"], - } - if authorization.get("subject") != subject or authorization.get( - "subjectSha256" - ) != _sha_bytes(_canonical_bytes(subject)): - _fail("ROOT_SEALED_AUTH_SUBJECT_MISMATCH") - selected = subject["selectedKeyIds"] - if authorization.get("selectedKeyIds") != selected: - _fail("ROOT_SEALED_AUTH_SELECTED_SET_MISMATCH") - policy_role = _policy_role(previous_policy, role_name) - members = {m["keyId"]: m for m in policy_role["members"]} - signatures = authorization.get("signatures") - if not isinstance(signatures, list) or len(signatures) != len(selected): - _fail("ROOT_SEALED_AUTH_SIGNATURE_COUNT_INVALID") - seen: set[str] = set() - operators: set[str] = set() - for sig_doc in signatures: - if not isinstance(sig_doc, dict) or not isinstance(sig_doc.get("signed"), dict): - _fail("ROOT_SEALED_AUTH_SIGNATURE_SCHEMA_INVALID") - key_id = sig_doc["signed"].get("keyId") - if key_id not in selected or key_id in seen: - _fail("ROOT_SEALED_AUTH_SIGNATURE_SET_INVALID") - seen.add(key_id) - normalized = _validate_authorization_signature( - sig_doc, - subject=subject, - root_info=auth_root, - role_name=role_name, - expected_member=members[key_id], - now=current_time, - enforce_freshness=False, - ) - operators.add(normalized["signed"]["operator"]) - if sorted(seen) != selected or len(seen) < policy_role["threshold"]: - _fail("ROOT_SEALED_AUTH_SIGNATURE_THRESHOLD_INVALID") - min_ops = int( - governance.POLICY["min_emergency_operators"] - if role_name == "emergency" - else governance.POLICY["min_policy_operators"] - ) - if len(operators) < min_ops: - _fail("ROOT_SEALED_AUTH_OPERATOR_QUORUM_INVALID") - final_policy = gov_state["policy"] - revoked_after = set(gov_state.get("revokedAuthorityKeyIds", [])) - _assert_role_matches_policy( - bundle_info["current"], - "governance", - final_policy, - revoked_after, - "ROOT_SEALED_FINAL_GOVERNANCE_ROLE", - ) - _assert_role_matches_policy( - bundle_info["current"], - "emergency", - final_policy, - revoked_after, - "ROOT_SEALED_FINAL_EMERGENCY_ROLE", - ) - governance_binding = state["governance"] - expected_binding = { - "epoch": gov_state["epoch"], - "policyVersion": gov_state["policy"]["policyVersion"], - "stateSha256": _sha_bytes(_canonical_bytes(gov_state)), - "bundleSha256": _sha_bytes(_canonical_bytes(gov_bundle)), - "recoverySnapshotSha256": _sha_bytes(snapshot_raw), - "proposalSha256": proposal_sha, - } - if governance_binding != expected_binding: - _fail("ROOT_SEALED_STATE_GOVERNANCE_REBIND_FAILED") - receipt, _ = _read(root / "release-root-receipt.json", "ROOT_SEALED_RECEIPT") - expected_receipt = { - "schemaVersion": int(POLICY["receipt_schema_version"]), - "status": "cryptographically-sealed", - "governanceId": gov_state["governanceId"], - "epoch": gov_state["epoch"], - "policyVersion": gov_state["policy"]["policyVersion"], - "authorization": { - "name": "cryptographic-governance-authorization.json", - "sha256": _sha_bytes(authorization_raw), - "size": len(authorization_raw), - }, - "rootBundle": _artifact(root / "release-root-bundle.json"), - "rootState": { - "name": _DOC_RELEASE_ROOT_STATE, - "sha256": _sha_bytes(state_raw), - "size": len(state_raw), - }, - "activeRoot": { - "name": "active-root.json", - "sha256": _sha_bytes(active_root_raw), - "size": len(active_root_raw), - }, - "governanceSnapshot": { - "name": "release-governance-recovery-snapshot.json", - "sha256": _sha_bytes(snapshot_raw), - "size": len(snapshot_raw), - }, - } - if receipt != expected_receipt: - _fail("ROOT_SEALED_RECEIPT_REBIND_FAILED") - return { - "ok": True, - "phase": "governance-cryptographic-seal-verified", - "governance_id": gov_state["governanceId"], - "epoch": gov_state["epoch"], - "root_version": bundle_info["current"]["version"], - "root_chain_head_sha256": bundle_info["chain_head_sha256"], - "authorization_sha256": _sha_bytes(authorization_raw), - } - - -def seal_governance( # ruff: ignore[too-many-branches, undocumented-public-function] - *, - governance_dir: Path, - authorization_signature_paths: list[Path], - output_dir: Path, - bootstrap_root_path: Path | None = None, - expected_bootstrap_root_sha256: str | None = None, - previous_root_state: Path | None = None, - previous_root_bundle: Path | None = None, - next_root_path: Path | None = None, - now: datetime | None = None, -) -> dict[str, Any]: - candidate = _validate_run154_dir(governance_dir) - protected = [candidate["root"]] - if previous_root_state is not None: - protected.append(previous_root_state.expanduser().resolve().parent) - target = _outside(output_dir, protected, "ROOT_OUTPUT_INSIDE_INPUT") - if target.exists() or target.is_symlink(): - _fail("ROOT_OUTPUT_ALREADY_EXISTS") - current_time = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - if expected_bootstrap_root_sha256 is None: - _fail("ROOT_EXPECTED_BOOTSTRAP_PIN_REQUIRED") - expected_bootstrap_root_sha256 = _hex( - expected_bootstrap_root_sha256, "ROOT_EXPECTED_BOOTSTRAP_PIN_INVALID" - ) - if (previous_root_state is None) != (previous_root_bundle is None): - _fail("ROOT_PREVIOUS_STATE_BUNDLE_PAIR_REQUIRED") - if previous_root_state is None: - if bootstrap_root_path is None: - _fail("ROOT_BOOTSTRAP_INPUT_REQUIRED") - root_doc, _ = _read(bootstrap_root_path, "ROOT_BOOTSTRAP_METADATA") - auth_root = _verify_bootstrap_root( - root_doc, expected_bootstrap_root_sha256, now=current_time - ) - if auth_root["governance_id"] != candidate["state"]["governanceId"]: - _fail("ROOT_BOOTSTRAP_GOVERNANCE_ID_MISMATCH") - if ( - bool(POLICY.get("allow_bootstrap_only_epoch_one", True)) - and candidate["state"]["epoch"] != 1 - ): - _fail("ROOT_BOOTSTRAP_EPOCH_INVALID") - roots = [auth_root["envelope"]] - bootstrap_pin = auth_root["sha256"] - previous_info = None - else: - if bootstrap_root_path is not None: - _fail("ROOT_BOOTSTRAP_NOT_ALLOWED_WITH_PREVIOUS") - previous_info = verify_root_bundle( - bundle_path=previous_root_bundle, - state_path=previous_root_state, - expected_bootstrap_root_sha256=expected_bootstrap_root_sha256, - now=current_time, - ) - auth_root = previous_info["current"] - if auth_root["governance_id"] != candidate["state"]["governanceId"]: - _fail("ROOT_PREVIOUS_GOVERNANCE_ID_MISMATCH") - previous_binding = previous_info["state"]["governance"] - if ( - candidate["state"]["epoch"] != previous_binding["epoch"] + 1 - or candidate["entry"]["proposal"].get("previousGovernanceStateSha256") - != previous_binding["stateSha256"] - ): - _fail("ROOT_GOVERNANCE_SEQUENCE_OR_PREVIOUS_STATE_MISMATCH") - roots = list(previous_info["bundle"]["roots"]) - bootstrap_pin = previous_info["bootstrap_pin"] - role_name = ( - "emergency" - if candidate["entry"].get("approvalRole") == "emergency" - else "governance" - ) - _assert_role_matches_policy( - auth_root, - role_name, - candidate["previous_policy"], - candidate["revoked_before"], - "ROOT_AUTHORIZING_ROLE", - ) - # The non-selected counterpart role must also match the prior policy, preventing a - # root file from smuggling an alternate emergency/governance membership. - other_role = "governance" if role_name == "emergency" else "emergency" - _assert_role_matches_policy( - auth_root, - other_role, - candidate["previous_policy"], - candidate["revoked_before"], - "ROOT_AUTHORIZING_OTHER_ROLE", - ) - subject = _subject(candidate, auth_root) - selected = subject["selectedKeyIds"] - policy_role = _policy_role(candidate["previous_policy"], role_name) - if ( - selected != sorted(candidate["entry"].get("selectedApproverKeyIds", [])) - or len(selected) < policy_role["threshold"] - ): - _fail("ROOT_AUTH_SELECTED_QUORUM_INVALID") - expected_members = {m["keyId"]: m for m in policy_role["members"]} - if len(authorization_signature_paths) != len(selected): - _fail("ROOT_AUTH_SIGNATURE_COUNT_INVALID") - sig_by_key: dict[str, dict[str, Any]] = {} - input_paths = sorted( - (p for p in candidate["root"].rglob("*") if p.is_file()), - key=lambda p: str(p.relative_to(candidate["root"])), - ) - input_paths.extend(authorization_signature_paths) - if bootstrap_root_path is not None: - input_paths.append(bootstrap_root_path) - if previous_root_state is not None: - input_paths.extend([previous_root_state, previous_root_bundle]) - if next_root_path is not None: - input_paths.append(next_root_path) - initial_hashes = {str(p.resolve()): _sha(p) for p in input_paths} - for path in authorization_signature_paths: - doc, _ = _read(path, "ROOT_AUTH_SIGNATURE") - signed = doc.get("signed") - key_id = signed.get("keyId") if isinstance(signed, dict) else None - if key_id not in selected or key_id in sig_by_key: - _fail("ROOT_AUTH_SIGNATURE_KEY_SET_INVALID") - sig_by_key[key_id] = _validate_authorization_signature( - doc, - subject=subject, - root_info=auth_root, - role_name=role_name, - expected_member=expected_members[key_id], - now=current_time, - ) - if sorted(sig_by_key) != selected: - _fail("ROOT_AUTH_SIGNATURE_SET_MISMATCH") - operators = {sig_by_key[k]["signed"]["operator"] for k in selected} - if len(operators) < int( - governance.POLICY["min_emergency_operators"] - if role_name == "emergency" - else governance.POLICY["min_policy_operators"] - ): - _fail("ROOT_AUTH_OPERATOR_QUORUM_INVALID") - authorization = { - "schemaVersion": int(POLICY["authorization_schema_version"]), - "predicateType": PREDICATE_TYPE + "/governance-authorization", - "status": "cryptographically-authorized", - "subject": subject, - "subjectSha256": _sha_bytes(_canonical_bytes(subject)), - "rootVersion": auth_root["version"], - "rootSha256": auth_root["sha256"], - "role": role_name, - "selectedKeyIds": selected, - "signatures": [sig_by_key[k] for k in sorted(sig_by_key)], - } - final_policy = candidate["final_policy"] - authority_changed = any( - _policy_role(candidate["previous_policy"], r) != _policy_role(final_policy, r) - for r in ("governance", "emergency") - ) - active_root = auth_root - if next_root_path is not None: - next_doc, _ = _read(next_root_path, "ROOT_NEXT_METADATA") - next_info = _validate_root_envelope(next_doc) - active_root = _verify_rotation(auth_root, next_info, now=current_time) - roots.append(active_root["envelope"]) - elif authority_changed: - _fail("ROOT_ROTATION_REQUIRED_FOR_AUTHORITY_CHANGE") - revoked_after = set(candidate["state"].get("revokedAuthorityKeyIds", [])) - _assert_role_matches_policy( - active_root, - "governance", - final_policy, - revoked_after, - "ROOT_FINAL_GOVERNANCE_ROLE", - ) - _assert_role_matches_policy( - active_root, - "emergency", - final_policy, - revoked_after, - "ROOT_FINAL_EMERGENCY_ROLE", - ) - root_bundle = { - "schemaVersion": int(POLICY["root_bundle_schema_version"]), - "predicateType": PREDICATE_TYPE, - "status": "trusted-release-root-chain", - "governanceId": candidate["state"]["governanceId"], - "bootstrapRootSha256": bootstrap_pin, - "roots": roots, - } - bundle_info = _validate_root_bundle( - root_bundle, expected_bootstrap_pin=bootstrap_pin, now=current_time - ) - bundle_sha = bundle_info["bundle_sha256"] - root_state = _state_document( - root_info=active_root, - bundle_sha=bundle_sha, - chain_head=bundle_info["chain_head_sha256"], - candidate=candidate, - ) - target.parent.mkdir(parents=True, exist_ok=True) - with tempfile.TemporaryDirectory( - prefix="release-root-seal-", dir=target.parent - ) as tmp: - stage = Path(tmp) / "root-seal" - stage.mkdir() - _write(stage / "release-root-bundle.json", root_bundle) - _write(stage / _DOC_RELEASE_ROOT_STATE, root_state) - _write(stage / "cryptographic-governance-authorization.json", authorization) - _write(stage / "active-root.json", active_root["envelope"]) - shutil.copy2( - candidate["root"] / "release-governance-recovery-snapshot.json", - stage / "release-governance-recovery-snapshot.json", - ) - verify_root_bundle( - bundle_path=stage / "release-root-bundle.json", - state_path=stage / _DOC_RELEASE_ROOT_STATE, - expected_bootstrap_root_sha256=bootstrap_pin, - now=current_time, - ) - receipt = { - "schemaVersion": int(POLICY["receipt_schema_version"]), - "status": "cryptographically-sealed", - "governanceId": candidate["state"]["governanceId"], - "epoch": candidate["state"]["epoch"], - "policyVersion": candidate["state"]["policy"]["policyVersion"], - "authorization": _artifact_doc( - authorization, "cryptographic-governance-authorization.json" - ), - "rootBundle": _artifact_doc(root_bundle, "release-root-bundle.json"), - "rootState": _artifact_doc(root_state, _DOC_RELEASE_ROOT_STATE), - "activeRoot": _artifact_doc(active_root["envelope"], "active-root.json"), - "governanceSnapshot": _artifact( - stage / "release-governance-recovery-snapshot.json" - ), - } - _write(stage / "release-root-receipt.json", receipt) - for path in input_paths: - if _sha(path) != initial_hashes[str(path.resolve())]: - _fail("ROOT_INPUT_CHANGED_DURING_SEAL") - temp_target = target.with_name(target.name + ".tmp-" + os.urandom(6).hex()) - shutil.copytree(stage, temp_target, copy_function=shutil.copy2) - os.replace(temp_target, target) - verify_sealed_governance( - sealed_dir=target, - expected_bootstrap_root_sha256=bootstrap_pin, - now=current_time, - require_current_fresh=True, - ) - return { - "ok": True, - "phase": "governance-cryptographically-sealed", - "governance_id": candidate["state"]["governanceId"], - "epoch": candidate["state"]["epoch"], - "policy_version": candidate["state"]["policy"]["policyVersion"], - "authorizing_root_version": auth_root["version"], - "active_root_version": active_root["version"], - "root_chain_head_sha256": bundle_info["chain_head_sha256"], - "authorization_sha256": _sha( - target / "cryptographic-governance-authorization.json" - ), - "root_state_sha256": _sha(target / _DOC_RELEASE_ROOT_STATE), - "root_bundle_sha256": _sha(target / "release-root-bundle.json"), - } - - -def _parse_paths(values: list[str]) -> list[Path]: - return [Path(x) for x in values] - - -def main( # ruff: ignore[undocumented-public-function] - argv: list[str] | None = None, -) -> int: - parser = argparse.ArgumentParser(description=__doc__) - sub = parser.add_subparsers(dest="command", required=True) - seal = sub.add_parser( - "seal", help="cryptographically seal a Run 154 governance candidate" - ) - seal.add_argument("--governance-dir", type=Path, required=True) - seal.add_argument( - "--signature", - action="append", - default=[], - help="canonical Ed25519 authorization signature JSON", - ) - seal.add_argument("--output-dir", type=Path, required=True) - seal.add_argument("--bootstrap-root", type=Path) - seal.add_argument("--bootstrap-root-sha256", required=True) - seal.add_argument("--previous-root-state", type=Path) - seal.add_argument("--previous-root-bundle", type=Path) - seal.add_argument("--next-root", type=Path) - verify = sub.add_parser( - "verify", help="offline-verify a root chain and optional compact state" - ) - verify.add_argument("--root-bundle", type=Path, required=True) - verify.add_argument("--root-state", type=Path) - verify.add_argument("--bootstrap-root-sha256", required=True) - verify_seal = sub.add_parser( - "verify-seal", - help="offline-verify a complete Run 155 cryptographic governance seal", - ) - verify_seal.add_argument("--sealed-dir", type=Path, required=True) - verify_seal.add_argument("--bootstrap-root-sha256", required=True) - args = parser.parse_args(argv) - try: - if args.command == "verify": - result = verify_root_bundle( - bundle_path=args.root_bundle, - state_path=args.root_state, - expected_bootstrap_root_sha256=args.bootstrap_root_sha256, - ) - out = { - "ok": True, - "phase": "release-root-verified", - "governance_id": result["governance_id"], - "root_version": result["current"]["version"], - "root_chain_head_sha256": result["chain_head_sha256"], - } - elif args.command == "verify-seal": - out = verify_sealed_governance( - sealed_dir=args.sealed_dir, - expected_bootstrap_root_sha256=args.bootstrap_root_sha256, - require_current_fresh=False, - ) - else: - out = seal_governance( - governance_dir=args.governance_dir, - authorization_signature_paths=_parse_paths(args.signature), - output_dir=args.output_dir, - bootstrap_root_path=args.bootstrap_root, - expected_bootstrap_root_sha256=args.bootstrap_root_sha256, - previous_root_state=args.previous_root_state, - previous_root_bundle=args.previous_root_bundle, - next_root_path=args.next_root, - ) - except RootTrustError as exc: - logger.error("%s", exc) - return 2 - logger.info("%s", json.dumps(out, sort_keys=True)) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/security/source_tree.py b/security/source_tree.py deleted file mode 100644 index 458cadb7399dd271f0b99dcc767ab598e0d18934..0000000000000000000000000000000000000000 --- a/security/source_tree.py +++ /dev/null @@ -1,55 +0,0 @@ -""" -Stable whole-extension source subject for release/CI evidence binding. -""" - -from __future__ import annotations - -import hashlib -import logging -from pathlib import Path - -HERE = Path(__file__).resolve().parent -EXTENSION_ROOT = HERE.parents[1] -_IGNORED_DIRS = {".git", ".pytest_cache", "__pycache__"} -_IGNORED_SUFFIXES = {".pyc", ".pyo"} -logger = logging.getLogger(__name__) - - -class SourceTreeError(RuntimeError): - """Source tree cannot be represented by the release subject contract.""" - - -def _source_files(root: Path) -> list[Path]: - root = root.resolve() - files: list[Path] = [] - for path in root.rglob("*"): - rel = path.relative_to(root) - if any(part in _IGNORED_DIRS for part in rel.parts): - continue - if path.suffix in _IGNORED_SUFFIXES: - continue - if path.is_symlink(): - raise SourceTreeError("SOURCE_TREE_SYMLINK_FORBIDDEN") - if path.is_file(): - files.append(path) - return sorted(files, key=lambda item: item.relative_to(root).as_posix()) - - -def source_tree_sha256(root: Path = EXTENSION_ROOT) -> str: - """Hash every release source file with path, Unix mode, length and content.""" - root = root.resolve() - h = hashlib.sha256() - for path in _source_files(root): - rel = path.relative_to(root).as_posix().encode("utf-8") - data = path.read_bytes() - mode = path.stat().st_mode & 0o777 - h.update(len(rel).to_bytes(4, "big")) - h.update(rel) - h.update(mode.to_bytes(4, "big")) - h.update(len(data).to_bytes(8, "big")) - h.update(data) - return h.hexdigest() - - -if __name__ == "__main__": - logger.info(source_tree_sha256()) diff --git a/security/supply_chain_policy.toml b/security/supply_chain_policy.toml deleted file mode 100644 index 7863d430ecb0f16aa0569ce7fd871b508a718d7f..0000000000000000000000000000000000000000 --- a/security/supply_chain_policy.toml +++ /dev/null @@ -1,38 +0,0 @@ -schema_version = 1 -reviewed_at = "2026-09-04" -target_python = "3.11.16" -target_platform = "linux/amd64" -lock_file = "requirements.lock" -sbom_file = "security/python-runtime.cdx.json" -require_hashes = true -binary_only = true -require_non_root = true -require_read_only_reference = true -require_redis_tls_in_strict = true - -[base_image] -repository = "python" -tag = "3.11.16-slim-bookworm" -index_digest = "sha256:0bee7276f83efd4a1ee05bbbf4281d95ed28e079220a9457f25a93e3f1e3c31b" -# This is an immutable multi-platform index. CI must resolve linux/amd64 from -# this exact index and record the resulting manifest digest in release evidence. - -[advisory_floors] -# Current explicit floors discovered during the B38 review. These are -# ratchets, not a substitute for a fresh advisory database scan. -click = "8.3.3" -starlette = "1.6.0" - -[release_gates] -python_advisory_scan = true -container_vulnerability_scan = true -container_sbom = true -provenance_attestation = true - -[release_evidence] -policy_file = "security/release_evidence_policy.toml" -example_file = "security/release-evidence.example.json" -verifier_file = "security/verify_release_evidence.py" -combined_gate_file = "security/verify_release_gate.py" -redis_probe_file = "security/probe_redis_authority.py" -subject_printer_file = "security/release_subjects.py" diff --git a/security/verify_archive_merkle_transparency.py b/security/verify_archive_merkle_transparency.py deleted file mode 100644 index 415c50f3502d7773bef827fcc39215230cad6ec6..0000000000000000000000000000000000000000 --- a/security/verify_archive_merkle_transparency.py +++ /dev/null @@ -1,1629 +0,0 @@ -""" -Run 164: verify Merkle transparency for externally anchored archive health. - -Run 163 hash-linked channel assertions are upgraded to cryptographically verifiable -RFC6962-style Merkle inclusion and consistency proofs. A separately pinned -transparency root defines independent log and gossip authorities. Every accepted -Run 163 anchor epoch must be included in every configured log, each log must prove -append-only extension from its previously accepted tree root, and independent gossip -observers must sign the exact same set of checkpoint hashes. -""" - -from __future__ import annotations - -import base64 -import hashlib -import json -import logging -import os -import re -import shutil -import subprocess -import tempfile -import threading -from datetime import datetime, timedelta, timezone -from pathlib import Path -from typing import Any, Callable - -import tomllib -from cryptography.exceptions import InvalidSignature -from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PublicKey - -logger = logging.getLogger(__name__) - -try: - from . import anchor_archive_health as run163 -except (ImportError, ValueError) as exc: - import importlib.util - import sys - - _here = Path(__file__).resolve().parent - _spec = importlib.util.spec_from_file_location( - "_run163_for_merkle", _here / "anchor_archive_health.py" - ) - if _spec is None or _spec.loader is None: - raise ImportError("anchor_archive_health.py") from exc - run163 = importlib.util.module_from_spec(_spec) - sys.modules[_spec.name] = run163 - _spec.loader.exec_module(run163) - -HERE = Path(__file__).resolve().parent -POLICY = tomllib.loads((HERE / "release_archive_merkle_policy.toml").read_text()) -PREDICATE_TYPE = str(POLICY["predicate_type"]) -_HEX64 = re.compile(r"^[0-9a-f]{64}$") -_ID = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:/@+-]{0,511}$") -_CHUNK = 1024 * 1024 - - -# ── Verified-state document names ──────────────────────────────────────── -# -# One constant per release-evidence document. The values are the on-disk -# artifact names and are unchanged; only the repetition is removed. Reading -# a document by a name that says which document it is also keeps static -# analysis from reading the filename token 'trusted' as 'confidential': -# these files carry published Merkle roots and SHA-256 digests, which must -# stay in clear text for any third party to verify them. -_DOC_ARCHIVE_ANCHOR_STATE = "trusted-archive-anchor-state.json" -_DOC_ARCHIVE_MERKLE_STATE = "trusted-archive-merkle-state.json" - - -_OUTPUT_NAMES = { - "release-archive-merkle-bundle.json", - _DOC_ARCHIVE_MERKLE_STATE, - "active-archive-merkle-evidence.json", - "release-archive-merkle-receipt.json", -} -_RUN163_NAMES = { - "release-archive-anchor-bundle.json", - _DOC_ARCHIVE_ANCHOR_STATE, - "active-archive-anchor-evidence.json", - "release-archive-anchor-receipt.json", -} - - -class ArchiveMerkleError(RuntimeError): # ruff: ignore[undocumented-public-class] - pass - - -def _fail(code: str) -> None: - raise ArchiveMerkleError(code) - - -def _canonical(value: Any) -> bytes: - return ( - json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False) - + "\n" - ).encode() - - -def _sha_bytes(raw: bytes) -> str: - return hashlib.sha256(raw).hexdigest() - - -def _sha(path: Path) -> str: - h = hashlib.sha256() - with path.open("rb") as f: - for chunk in iter(lambda: f.read(_CHUNK), b""): - h.update(chunk) - return h.hexdigest() - - -def _write(path: Path, value: Any) -> None: - path.write_bytes(_canonical(value)) - - -def _loads(raw: bytes, code: str) -> dict[str, Any]: - def hook(pairs): - out = {} - for key, value in pairs: - if key in out: - _fail(code + "_DUPLICATE_KEY") - out[key] = value - return out - - try: - value = json.loads(raw.decode(), object_pairs_hook=hook) - except ArchiveMerkleError: - raise - except Exception as exc: - raise ArchiveMerkleError(code + "_JSON_INVALID") from exc - if not isinstance(value, dict): - _fail(code + "_SCHEMA_INVALID") - return value - - -def _read_json(path: Path, code: str) -> tuple[dict[str, Any], bytes]: - if path.is_symlink() or not path.is_file(): - _fail(code + "_INVALID") - if path.stat().st_size > int(POLICY["max_json_bytes"]): - _fail(code + "_TOO_LARGE") - raw = path.read_bytes() - doc = _loads(raw, code) - if raw != _canonical(doc): - _fail(code + "_NOT_CANONICAL") - return doc, raw - - -def _id(value: Any, code: str) -> str: - if not isinstance(value, str): - _fail(code) - value = value.strip() - if ( - not value - or ".." in value - or "?" in value - or "#" in value - or "\x00" in value - or _ID.fullmatch(value) is None - ): - _fail(code) - return value - - -def _hex(value: Any, code: str) -> str: - if not isinstance(value, str) or _HEX64.fullmatch(value) is None: - _fail(code) - return value - - -def _dt(value: Any, code: str) -> datetime: - if not isinstance(value, str) or not value.endswith("Z"): - _fail(code) - try: - return datetime.fromisoformat(value[:-1] + "+00:00").astimezone(timezone.utc) - except Exception as exc: - raise ArchiveMerkleError(code) from exc - - -def _ts(value: datetime) -> str: - return ( - value.astimezone(timezone.utc) - .isoformat(timespec="seconds") - .replace("+00:00", "Z") - ) - - -def _b64(value: Any, code: str, size: int | None = None) -> bytes: - if not isinstance(value, str): - _fail(code) - try: - raw = base64.b64decode(value, validate=True) - except Exception as exc: - raise ArchiveMerkleError(code) from exc - if size is not None and len(raw) != size: - _fail(code) - return raw - - -def _verify_sig(public_key: str, signature: str, message: bytes, code: str) -> None: - try: - Ed25519PublicKey.from_public_bytes( - _b64(public_key, code + "_PUBLIC", 32) - ).verify(_b64(signature, code + "_SIGNATURE", 64), message) - except (InvalidSignature, ValueError) as exc: - raise ArchiveMerkleError(code + "_INVALID") from exc - - -def _positive_int(value: Any, code: str, *, allow_zero: bool = False) -> int: - if ( - not isinstance(value, int) - or isinstance(value, bool) - or value < (0 if allow_zero else 1) - ): - _fail(code) - if value > int(POLICY["max_tree_size"]): - _fail(code) - return value - - -def _regular_dir(path: Path, code: str) -> Path: - if path.is_symlink() or not path.is_dir(): - _fail(code) - return path.resolve() - - -def _outside(path: Path, protected: list[Path], code: str) -> Path: - target = path.expanduser().resolve() - for item in protected: - root = item.resolve() - if target == root or root in target.parents: - _fail(code) - return target - - -def _dir_fingerprint(root: Path, code: str) -> dict[str, tuple[str, int, int]]: - root = _regular_dir(root, code) - out = {} - for path in sorted(root.iterdir()): - if path.is_symlink() or not path.is_file(): - _fail(code) - stat = path.stat() - out[path.name] = (_sha(path), stat.st_size, stat.st_mode & 0o7777) - return out - - -def _authority_fingerprint(paths: list[Path], code: str) -> list[tuple[str, Any]]: - out = [] - for raw in paths: - path = Path(raw) - if path.is_symlink(): - _fail(code) - resolved = path.resolve() - if resolved.is_dir(): - out.append((str(resolved), _dir_fingerprint(resolved, code))) - elif resolved.is_file(): - stat = resolved.stat() - out.append( - (str(resolved), (_sha(resolved), stat.st_size, stat.st_mode & 0o7777)) - ) - else: - _fail(code) - return out - - -# RFC6962-style Merkle hashing: leaf = H(0x00 || data), node = H(0x01 || left || right). -def merkle_leaf_hash(data: bytes) -> str: # ruff: ignore[undocumented-public-function] - return hashlib.sha256(b"\x00" + data).hexdigest() - - -def _node_hash(left: bytes, right: bytes) -> bytes: - return hashlib.sha256(b"\x01" + left + right).digest() - - -def _proof_hashes(values: Any, code: str) -> list[bytes]: - if not isinstance(values, list) or len(values) > int(POLICY["max_proof_nodes"]): - _fail(code) - return [bytes.fromhex(_hex(value, code)) for value in values] - - -def verify_inclusion_proof( # ruff: ignore[undocumented-public-function] - *, leaf_hash: str, leaf_index: int, tree_size: int, proof: list[str], root_hash: str -) -> bool: - leaf = bytes.fromhex(_hex(leaf_hash, "ARCHIVE_MERKLE_LEAF_HASH_INVALID")) - root = bytes.fromhex(_hex(root_hash, "ARCHIVE_MERKLE_ROOT_HASH_INVALID")) - if tree_size <= 0 or leaf_index < 0 or leaf_index >= tree_size: - return False - fn = leaf_index - sn = tree_size - 1 - result = leaf - for sibling in _proof_hashes(proof, "ARCHIVE_MERKLE_INCLUSION_PROOF_INVALID"): - if fn & 1 or fn == sn: - result = _node_hash(sibling, result) - while fn != 0 and (fn & 1) == 0: - fn >>= 1 - sn >>= 1 - else: - result = _node_hash(result, sibling) - fn >>= 1 - sn >>= 1 - return sn == 0 and result == root - - -def verify_consistency_proof( # ruff: ignore[too-many-branches, too-many-return-statements, undocumented-public-function] - *, - old_size: int, - new_size: int, - old_root_hash: str | None, - new_root_hash: str, - proof: list[str], -) -> bool: - if old_size < 0 or new_size <= 0 or old_size > new_size: - return False - new_root = bytes.fromhex(_hex(new_root_hash, "ARCHIVE_MERKLE_NEW_ROOT_INVALID")) - nodes = _proof_hashes(proof, "ARCHIVE_MERKLE_CONSISTENCY_PROOF_INVALID") - if old_size == 0: - return len(nodes) == 0 and old_root_hash is None - if old_root_hash is None: - return False - old_root = bytes.fromhex(_hex(old_root_hash, "ARCHIVE_MERKLE_OLD_ROOT_INVALID")) - if old_size == new_size: - return len(nodes) == 0 and old_root == new_root - - fn = old_size - 1 - sn = new_size - 1 - while fn & 1: - fn >>= 1 - sn >>= 1 - - if fn == 0: - fr = sr = old_root - else: - if not nodes: - return False - fr = sr = nodes[0] - nodes = nodes[1:] - - for node in nodes: - if sn == 0: - return False - if (fn & 1) or fn == sn: - fr = _node_hash(node, fr) - sr = _node_hash(node, sr) - while fn != 0 and (fn & 1) == 0: - fn >>= 1 - sn >>= 1 - else: - sr = _node_hash(sr, node) - fn >>= 1 - sn >>= 1 - return sn == 0 and fr == old_root and sr == new_root - - -def _root_key(value: Any, code: str) -> dict[str, str]: - if not isinstance(value, dict) or set(value) != { - "identity", - "operator", - "expires", - "publicKey", - }: - _fail(code + "_SCHEMA_INVALID") - out = { - "identity": _id(value["identity"], code + "_IDENTITY_INVALID"), - "operator": _id(value["operator"], code + "_OPERATOR_INVALID"), - "expires": value["expires"], - "publicKey": value["publicKey"], - } - _dt(out["expires"], code + "_EXPIRES_INVALID") - _b64(out["publicKey"], code + "_PUBLIC_INVALID", 32) - if out != value: - _fail(code + "_NOT_NORMALIZED") - return out - - -def _log_entry(value: Any, code: str) -> dict[str, str]: - expected = { - "operator", - "publicKey", - "gossipIdentity", - "gossipOperator", - "gossipPublicKey", - } - if not isinstance(value, dict) or set(value) != expected: - _fail(code + "_SCHEMA_INVALID") - out = { - "operator": _id(value["operator"], code + "_OPERATOR_INVALID"), - "publicKey": value["publicKey"], - "gossipIdentity": _id( - value["gossipIdentity"], code + "_GOSSIP_IDENTITY_INVALID" - ), - "gossipOperator": _id( - value["gossipOperator"], code + "_GOSSIP_OPERATOR_INVALID" - ), - "gossipPublicKey": value["gossipPublicKey"], - } - _b64(out["publicKey"], code + "_PUBLIC_INVALID", 32) - _b64(out["gossipPublicKey"], code + "_GOSSIP_PUBLIC_INVALID", 32) - if out != value: - _fail(code + "_NOT_NORMALIZED") - return out - - -def verify_transparency_root( # ruff: ignore[too-many-branches, undocumented-public-function] - doc: dict[str, Any], - expected_sha256: str, - *, - now: datetime, - historical: bool = False, -) -> dict[str, Any]: - if set(doc) != {"signed", "signatures"} or not isinstance(doc.get("signed"), dict): - _fail("ARCHIVE_MERKLE_ROOT_SCHEMA_INVALID") - signed = doc["signed"] - expected = { - "_type", - "specVersion", - "schemaVersion", - "rootId", - "version", - "issuedAt", - "expires", - "threshold", - "selectedSignerKeyIds", - "keys", - "logs", - } - if ( - set(signed) != expected - or signed.get("_type") != "archive-merkle-transparency-root" - ): - _fail("ARCHIVE_MERKLE_ROOT_SIGNED_SCHEMA_INVALID") - if signed.get("specVersion") != str(POLICY["spec_version"]) or signed.get( - "schemaVersion" - ) != int(POLICY["transparency_root_schema_version"]): - _fail("ARCHIVE_MERKLE_ROOT_VERSION_INVALID") - root_id = _id(signed["rootId"], "ARCHIVE_MERKLE_ROOT_ID_INVALID") - if signed["version"] != 1: - _fail("ARCHIVE_MERKLE_ROOT_VERSION_INVALID") - issued = _dt(signed["issuedAt"], "ARCHIVE_MERKLE_ROOT_ISSUED_INVALID") - expires = _dt(signed["expires"], "ARCHIVE_MERKLE_ROOT_EXPIRES_INVALID") - if expires <= issued or expires - issued > timedelta( - days=int(POLICY["max_root_lifetime_days"]) - ): - _fail("ARCHIVE_MERKLE_ROOT_LIFETIME_INVALID") - if not historical: - if issued > now + timedelta(minutes=int(POLICY["max_clock_skew_minutes"])): - _fail("ARCHIVE_MERKLE_ROOT_FROM_FUTURE") - if expires <= now: - _fail("ARCHIVE_MERKLE_ROOT_EXPIRED") - - raw_keys = signed["keys"] - _raw_keys = ( - int(POLICY["min_root_keys"]) # int - <= len(raw_keys) # lint - <= 64 # ruff: ignore[magic-value-comparison] - ) - if not isinstance(raw_keys, dict) or not _raw_keys: - _fail("ARCHIVE_MERKLE_ROOT_KEYS_INVALID") - keys = {} - for raw_kid, value in sorted(raw_keys.items()): - kid = _id(raw_kid, "ARCHIVE_MERKLE_ROOT_KEY_ID_INVALID") - if kid != raw_kid or kid in keys: - _fail("ARCHIVE_MERKLE_ROOT_KEY_ID_INVALID") - keys[kid] = _root_key(value, "ARCHIVE_MERKLE_ROOT_KEY") - for key in keys.values(): - if _dt(key["expires"], "ARCHIVE_MERKLE_ROOT_KEY_EXPIRES_INVALID") < expires: - _fail("ARCHIVE_MERKLE_ROOT_KEY_EXPIRES_BEFORE_ROOT") - - threshold = _positive_int( - signed["threshold"], "ARCHIVE_MERKLE_ROOT_THRESHOLD_INVALID" - ) - if threshold < int(POLICY["min_root_threshold"]) or threshold > len(keys): - _fail("ARCHIVE_MERKLE_ROOT_THRESHOLD_INVALID") - selected = signed["selectedSignerKeyIds"] - if not isinstance(selected, list): - _fail("ARCHIVE_MERKLE_ROOT_SELECTED_INVALID") - selected = [_id(x, "ARCHIVE_MERKLE_ROOT_SELECTED_INVALID") for x in selected] - if ( - selected != sorted(selected) - or len(selected) != threshold - or len(set(selected)) != len(selected) - ): - _fail("ARCHIVE_MERKLE_ROOT_SELECTED_INVALID") - if any(k not in keys for k in selected): - _fail("ARCHIVE_MERKLE_ROOT_SELECTED_INVALID") - if len({keys[k]["operator"] for k in selected}) < int(POLICY["min_root_operators"]): - _fail("ARCHIVE_MERKLE_ROOT_OPERATOR_QUORUM_INVALID") - - logs_raw = signed["logs"] - if not isinstance(logs_raw, dict) or len(logs_raw) < int(POLICY["min_logs"]): - _fail("ARCHIVE_MERKLE_LOGS_INVALID") - logs = { - lid: _log_entry(value, "ARCHIVE_MERKLE_LOG") - for lid, value in sorted(logs_raw.items()) - } - if list(logs) != sorted(logs) or any( - _id(lid, "ARCHIVE_MERKLE_LOG_ID_INVALID") != lid for lid in logs - ): - _fail("ARCHIVE_MERKLE_LOGS_INVALID") - if len({v["operator"] for v in logs.values()}) != len(logs): - _fail("ARCHIVE_MERKLE_LOG_OPERATOR_DIVERSITY_INVALID") - if len({v["gossipOperator"] for v in logs.values()}) < int(POLICY["min_gossipers"]): - _fail("ARCHIVE_MERKLE_GOSSIP_OPERATOR_DIVERSITY_INVALID") - root_ops = {v["operator"] for v in keys.values()} - root_pubs = {v["publicKey"] for v in keys.values()} - log_ops = {v["operator"] for v in logs.values()} - log_pubs = {v["publicKey"] for v in logs.values()} - gossip_ops = {v["gossipOperator"] for v in logs.values()} - gossip_pubs = {v["gossipPublicKey"] for v in logs.values()} - if ( - root_ops & (log_ops | gossip_ops) - or log_ops & gossip_ops - or root_pubs & (log_pubs | gossip_pubs) - or log_pubs & gossip_pubs - ): - _fail("ARCHIVE_MERKLE_AUTHORITY_PLANES_OVERLAP") - - signatures = doc["signatures"] - if not isinstance(signatures, list) or len(signatures) != len(selected): - _fail("ARCHIVE_MERKLE_ROOT_SIGNATURE_SET_INVALID") - sig_map = {} - for item in signatures: - if not isinstance(item, dict) or set(item) != {"keyId", "signature"}: - _fail("ARCHIVE_MERKLE_ROOT_SIGNATURE_SCHEMA_INVALID") - key_id = _id(item["keyId"], "ARCHIVE_MERKLE_ROOT_SIGNATURE_KEY_INVALID") - if key_id in sig_map: - _fail("ARCHIVE_MERKLE_ROOT_SIGNATURE_DUPLICATE") - sig_map[key_id] = item["signature"] - if sorted(sig_map) != selected or signatures != sorted( - signatures, key=lambda x: x["keyId"] - ): - _fail("ARCHIVE_MERKLE_ROOT_SIGNATURE_SET_INVALID") - message = _canonical(signed) - for key_id in selected: - _verify_sig( - keys[key_id]["publicKey"], - sig_map[key_id], - message, - "ARCHIVE_MERKLE_ROOT_SIGNATURE", - ) - raw = _canonical(doc) - sha = _sha_bytes(raw) - if sha != _hex(expected_sha256, "ARCHIVE_MERKLE_ROOT_PIN_INVALID"): - _fail("ARCHIVE_MERKLE_ROOT_PIN_MISMATCH") - return { - "rootId": root_id, - "sha256": sha, - "keys": keys, - "logs": logs, - "issued": issued, - "expires": expires, - "doc": doc, - } - - -def _command_adapter( - command: list[str], prefix: str -) -> Callable[[dict[str, Any]], dict[str, Any]]: - if not command: - _fail(prefix + "_COMMAND_EMPTY") - - def call(request: dict[str, Any]) -> dict[str, Any]: - proc = subprocess.Popen( # ruff: ignore[subprocess-without-shell-equals-true] - command, - stdin=subprocess.PIPE, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, - env={"PATH": os.defpath, "LC_ALL": "C", "LANG": "C"}, - ) - payload = _canonical(request) - out = bytearray() - err = bytearray() - limit = int(POLICY["max_adapter_output_bytes"]) - - def pump(stream, buf): - try: - while True: - chunk = stream.read(65536) - if not chunk: - break - buf.extend(chunk) - if len(buf) > limit: - try: # ruff: ignore[suppressible-exception] - proc.kill() - except OSError: - pass - return - finally: - stream.close() - - t1 = threading.Thread(target=pump, args=(proc.stdout, out)) - t2 = threading.Thread(target=pump, args=(proc.stderr, err)) - t1.start() - t2.start() - try: - try: - proc.stdin.write(payload) - proc.stdin.close() - proc.wait(timeout=int(POLICY["adapter_timeout_seconds"])) - except subprocess.TimeoutExpired: - proc.kill() - proc.wait() - _fail(prefix + "_TIMEOUT") - t1.join() - t2.join() - if len(out) > limit or len(err) > limit: - _fail(prefix + "_OUTPUT_TOO_LARGE") - if proc.returncode != 0: - _fail(prefix + "_FAILED") - return _loads(bytes(out), prefix + "_RESPONSE") - finally: - if proc.poll() is None: - proc.kill() - proc.wait() - if t1.is_alive(): - t1.join() - if t2.is_alive(): - t2.join() - for stream in (proc.stdin, proc.stdout, proc.stderr): - try: # ruff: ignore[suppressible-exception] - stream.close() - except OSError: # ruff: ignore[try-except-in-loop] - pass - - return call - - -def command_log( # ruff: ignore[undocumented-public-function] - command: list[str], -) -> Callable[[dict[str, Any]], dict[str, Any]]: - return _command_adapter(command, "ARCHIVE_MERKLE_LOG_ADAPTER") - - -def command_gossip( # ruff: ignore[undocumented-public-function] - command: list[str], -) -> Callable[[dict[str, Any]], dict[str, Any]]: - return _command_adapter(command, "ARCHIVE_MERKLE_GOSSIP_ADAPTER") - - -def _verify_run163_current( - *, - run160_dir: Path, - run161_dir: Path, - retention_root_path: Path, - retention_root_pin: str, - witness_root_path: Path, - witness_root_pin: str, - bootstrap_pin: str, - recovery_pin: str, - attestation_pins: list[str], - run162_dir: Path, - anchor_plan_path: Path, - run163_dir: Path, - now: datetime, - historical: bool = False, -) -> dict[str, Any]: - try: - run163.verify_anchor_history( - run160_dir=run160_dir, - run161_dir=run161_dir, - retention_root_path=retention_root_path, - retention_root_pin=retention_root_pin, - witness_root_path=witness_root_path, - witness_root_pin=witness_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - run162_dir=run162_dir, - anchor_plan_path=anchor_plan_path, - output_dir=run163_dir, - now=now, - historical=historical, - require_current_match=True, - ) - except Exception as exc: - raise ArchiveMerkleError("ARCHIVE_MERKLE_RUN163_INVALID:" + str(exc)) from exc - docs = {} - raws = {} - root = _regular_dir(Path(run163_dir), "ARCHIVE_MERKLE_RUN163_DIR_INVALID") - if {p.name for p in root.iterdir()} != _RUN163_NAMES: - _fail("ARCHIVE_MERKLE_RUN163_ALLOWLIST_INVALID") - for name in sorted(_RUN163_NAMES): - docs[name], raws[name] = _read_json(root / name, "ARCHIVE_MERKLE_RUN163") - return {"docs": docs, "raws": raws} - - -def _enforce_external_separation( - root: dict[str, Any], witness_root_path: Path, anchor_plan_path: Path -) -> None: - wdoc, _ = _read_json(Path(witness_root_path), "ARCHIVE_MERKLE_WITNESS_ROOT") - pdoc, _ = _read_json(Path(anchor_plan_path), "ARCHIVE_MERKLE_ANCHOR_PLAN") - other_ops = set() - other_pubs = set() - for value in wdoc.get("signed", {}).get("keys", {}).values(): - if isinstance(value, dict): - other_ops.add(value.get("operator")) - other_pubs.add(value.get("publicKey")) - for value in pdoc.get("signed", {}).get("channels", {}).values(): - if isinstance(value, dict): - other_ops.update([value.get("operator"), value.get("observerOperator")]) - other_pubs.update([value.get("publicKey"), value.get("observerPublicKey")]) - other_ops.discard(None) - other_pubs.discard(None) - own_ops = ( - {v["operator"] for v in root["keys"].values()} - | {v["operator"] for v in root["logs"].values()} - | {v["gossipOperator"] for v in root["logs"].values()} - ) - own_pubs = ( - {v["publicKey"] for v in root["keys"].values()} - | {v["publicKey"] for v in root["logs"].values()} - | {v["gossipPublicKey"] for v in root["logs"].values()} - ) - if own_ops & other_ops or own_pubs & other_pubs: - _fail("ARCHIVE_MERKLE_EXTERNAL_AUTHORITY_OVERLAP") - - -def _leaf_document( - run163_docs: dict[str, dict[str, Any]], run163_raws: dict[str, bytes] -) -> dict[str, Any]: - state = run163_docs[_DOC_ARCHIVE_ANCHOR_STATE] - active = run163_docs["active-archive-anchor-evidence.json"] - return { - "_type": "run163-archive-anchor-leaf", - "specVersion": str(POLICY["spec_version"]), - "run163Sequence": state.get("sequence"), - "run163WitnessSequence": state.get("run162WitnessSequence"), - "run163AnchorConsensusHeadSha256": state.get("anchorConsensusHeadSha256"), - "run163StateSha256": _sha_bytes(run163_raws[_DOC_ARCHIVE_ANCHOR_STATE]), - "run163ActiveEvidenceSha256": _sha_bytes( - run163_raws["active-archive-anchor-evidence.json"] - ), - "run163ChannelIds": active.get("channelIds", state.get("channelIds")), - } - - -def _verify_checkpoint( # ruff: ignore[too-many-branches] - doc: dict[str, Any], - *, - log_id: str, - config: dict[str, str], - sequence: int, - challenge: str, - expected_leaf_hash: str, - previous: dict[str, Any] | None, - now: datetime, - creation: bool, - historical: bool, -) -> dict[str, Any]: - if set(doc) != {"signed", "signature"} or not isinstance(doc.get("signed"), dict): - _fail("ARCHIVE_MERKLE_CHECKPOINT_SCHEMA_INVALID") - signed = doc["signed"] - expected = { - "_type", - "specVersion", - "schemaVersion", - "logId", - "operator", - "sequence", - "treeSize", - "rootHash", - "leafIndex", - "leafHash", - "previousTreeSize", - "previousRootHash", - "integratedAt", - "challenge", - "inclusionProof", - "consistencyProof", - } - if ( - set(signed) != expected - or signed.get("_type") != "archive-merkle-log-checkpoint" - ): - _fail("ARCHIVE_MERKLE_CHECKPOINT_SIGNED_SCHEMA_INVALID") - if signed.get("specVersion") != str(POLICY["spec_version"]) or signed.get( - "schemaVersion" - ) != int(POLICY["checkpoint_schema_version"]): - _fail("ARCHIVE_MERKLE_CHECKPOINT_VERSION_INVALID") - if ( - signed["logId"] != log_id - or signed["operator"] != config["operator"] - or signed["sequence"] != sequence - or signed["challenge"] != challenge - ): - _fail("ARCHIVE_MERKLE_CHECKPOINT_BINDING_INVALID") - if signed["leafHash"] != expected_leaf_hash: - _fail("ARCHIVE_MERKLE_CHECKPOINT_LEAF_INVALID") - tree_size = _positive_int(signed["treeSize"], "ARCHIVE_MERKLE_TREE_SIZE_INVALID") - leaf_index = _positive_int( - signed["leafIndex"], "ARCHIVE_MERKLE_LEAF_INDEX_INVALID", allow_zero=True - ) - if leaf_index >= tree_size: - _fail("ARCHIVE_MERKLE_LEAF_INDEX_INVALID") - root_hash = _hex(signed["rootHash"], "ARCHIVE_MERKLE_ROOT_HASH_INVALID") - integrated = _dt(signed["integratedAt"], "ARCHIVE_MERKLE_INTEGRATED_AT_INVALID") - if integrated > now + timedelta(minutes=int(POLICY["max_clock_skew_minutes"])): - _fail("ARCHIVE_MERKLE_CHECKPOINT_FROM_FUTURE") - if creation and now - integrated > timedelta( - minutes=int(POLICY["max_adapter_freshness_minutes"]) - ): - _fail("ARCHIVE_MERKLE_CHECKPOINT_STALE") - if ( - not historical - and not creation - and now - integrated > timedelta(days=int(POLICY["max_active_epoch_age_days"])) - ): - _fail("ARCHIVE_MERKLE_ACTIVE_EPOCH_STALE") - - if previous is None: - if ( - signed["previousTreeSize"] != 0 - or signed["previousRootHash"] is not None - or signed["consistencyProof"] != [] - ): - _fail("ARCHIVE_MERKLE_GENESIS_CONSISTENCY_INVALID") - old_size = 0 - old_root = None - else: - old_size = previous["treeSize"] - old_root = previous["rootHash"] - if ( - signed["previousTreeSize"] != old_size - or signed["previousRootHash"] != old_root - ): - _fail("ARCHIVE_MERKLE_PREVIOUS_CHECKPOINT_INVALID") - if tree_size <= old_size: - _fail("ARCHIVE_MERKLE_TREE_NOT_ADVANCED") - if not verify_inclusion_proof( - leaf_hash=expected_leaf_hash, - leaf_index=leaf_index, - tree_size=tree_size, - proof=signed["inclusionProof"], - root_hash=root_hash, - ): - _fail("ARCHIVE_MERKLE_INCLUSION_INVALID") - if not verify_consistency_proof( - old_size=old_size, - new_size=tree_size, - old_root_hash=old_root, - new_root_hash=root_hash, - proof=signed["consistencyProof"], - ): - _fail("ARCHIVE_MERKLE_CONSISTENCY_INVALID") - _verify_sig( - config["publicKey"], - doc["signature"], - _canonical(signed), - "ARCHIVE_MERKLE_CHECKPOINT_SIGNATURE", - ) - return { - "logId": log_id, - "treeSize": tree_size, - "rootHash": root_hash, - "leafIndex": leaf_index, - "leafHash": expected_leaf_hash, - "checkpointSha256": _sha_bytes(_canonical(doc)), - "integratedAt": integrated, - } - - -def _gossip_challenge( - sequence: int, run163_head: str, leaf_hash: str, checkpoints: dict[str, str] -) -> str: - return _sha_bytes( - _canonical( - { - "sequence": sequence, - "run163AnchorConsensusHeadSha256": run163_head, - "leafHash": leaf_hash, - "checkpointSha256s": checkpoints, - } - ) - ) - - -def _verify_gossip( - doc: dict[str, Any], - *, - log_id: str, - config: dict[str, str], - sequence: int, - challenge: str, - run163_head: str, - checkpoints: dict[str, str], - now: datetime, - creation: bool, - historical: bool, -) -> dict[str, Any]: - if set(doc) != {"signed", "signature"} or not isinstance(doc.get("signed"), dict): - _fail("ARCHIVE_MERKLE_GOSSIP_SCHEMA_INVALID") - signed = doc["signed"] - expected = { - "_type", - "specVersion", - "schemaVersion", - "gossipIdentity", - "gossipOperator", - "sequence", - "observedAt", - "challenge", - "run163AnchorConsensusHeadSha256", - "checkpointSha256s", - } - if ( - set(signed) != expected - or signed.get("_type") != "archive-merkle-gossip-observation" - ): - _fail("ARCHIVE_MERKLE_GOSSIP_SIGNED_SCHEMA_INVALID") - if signed.get("specVersion") != str(POLICY["spec_version"]) or signed.get( - "schemaVersion" - ) != int(POLICY["gossip_schema_version"]): - _fail("ARCHIVE_MERKLE_GOSSIP_VERSION_INVALID") - if ( - signed["gossipIdentity"] != config["gossipIdentity"] - or signed["gossipOperator"] != config["gossipOperator"] - ): - _fail("ARCHIVE_MERKLE_GOSSIP_IDENTITY_INVALID") - if ( - signed["sequence"] != sequence - or signed["challenge"] != challenge - or signed["run163AnchorConsensusHeadSha256"] != run163_head - ): - _fail("ARCHIVE_MERKLE_GOSSIP_BINDING_INVALID") - if signed["checkpointSha256s"] != checkpoints: - _fail("ARCHIVE_MERKLE_GOSSIP_SPLIT_VIEW") - observed = _dt(signed["observedAt"], "ARCHIVE_MERKLE_GOSSIP_OBSERVED_INVALID") - if observed > now + timedelta(minutes=int(POLICY["max_clock_skew_minutes"])): - _fail("ARCHIVE_MERKLE_GOSSIP_FROM_FUTURE") - if creation and now - observed > timedelta( - minutes=int(POLICY["max_adapter_freshness_minutes"]) - ): - _fail("ARCHIVE_MERKLE_GOSSIP_STALE") - if ( - not historical - and not creation - and now - observed > timedelta(days=int(POLICY["max_active_epoch_age_days"])) - ): - _fail("ARCHIVE_MERKLE_ACTIVE_EPOCH_STALE") - _verify_sig( - config["gossipPublicKey"], - doc["signature"], - _canonical(signed), - "ARCHIVE_MERKLE_GOSSIP_SIGNATURE", - ) - return { - "logId": log_id, - "gossipSha256": _sha_bytes(_canonical(doc)), - "observedAt": observed, - } - - -def _event_head(previous_head: str | None, event_without_head: dict[str, Any]) -> str: - return _sha_bytes( - _canonical( - { - "previousMerkleConsensusHeadSha256": previous_head, - "event": event_without_head, - } - ) - ) - - -def _run163_artifacts(raws: dict[str, bytes]) -> dict[str, dict[str, Any]]: - return { - name: {"sha256": _sha_bytes(raws[name]), "size": len(raws[name])} - for name in sorted(raws) - } - - -def _load_output(root: Path) -> tuple[dict[str, dict[str, Any]], dict[str, bytes]]: - root = _regular_dir(root, "ARCHIVE_MERKLE_OUTPUT_INVALID") - if {p.name for p in root.iterdir()} != _OUTPUT_NAMES: - _fail("ARCHIVE_MERKLE_OUTPUT_ALLOWLIST_INVALID") - docs = {} - raws = {} - for name in sorted(_OUTPUT_NAMES): - docs[name], raws[name] = _read_json(root / name, "ARCHIVE_MERKLE_OUTPUT") - return docs, raws - - -def _replay_history( # ruff: ignore[too-many-branches] - *, - root: dict[str, Any], - bundle: dict[str, Any], - receipt: dict[str, Any], - now: datetime, - historical: bool, - current_run163_docs: dict[str, Any] | None, - current_run163_raws: dict[str, bytes] | None, -) -> dict[str, Any]: - if set(bundle) != { - "schemaVersion", - "predicateType", - "status", - "transparencyRootSha256", - "events", - }: - _fail("ARCHIVE_MERKLE_BUNDLE_SCHEMA_INVALID") - if ( - bundle["schemaVersion"] != int(POLICY["bundle_schema_version"]) - or bundle["predicateType"] != PREDICATE_TYPE - or bundle["status"] != "archive-merkle-transparency-history" - or bundle["transparencyRootSha256"] != root["sha256"] - ): - _fail("ARCHIVE_MERKLE_BUNDLE_SCHEMA_INVALID") - if ( - set(receipt) != {"schemaVersion", "status", "events"} - or receipt["schemaVersion"] != int(POLICY["receipt_schema_version"]) - or receipt["status"] != "archive-merkle-transparency-accepted" - ): - _fail("ARCHIVE_MERKLE_RECEIPT_SCHEMA_INVALID") - events = bundle["events"] - recs = receipt["events"] - if ( - not isinstance(events, list) - or not events - or not isinstance(recs, list) - or len(events) != len(recs) - ): - _fail("ARCHIVE_MERKLE_HISTORY_LENGTH_INVALID") - previous_head = None - previous_by_log: dict[str, dict[str, Any] | None] = dict.fromkeys(root["logs"]) - for idx, (event, rec) in enumerate(zip(events, recs), 1): - expected_event_keys = { - "sequence", - "run163Sequence", - "run163AnchorConsensusHeadSha256", - "run163Artifacts", - "leafHash", - "logs", - "logChallenge", - "gossipChallenge", - "gossipResponseSha256s", - "merkleConsensusHeadSha256", - } - if ( - not isinstance(event, dict) - or set(event) != expected_event_keys - or event["sequence"] != idx - or event["run163Sequence"] != idx - ): - _fail("ARCHIVE_MERKLE_EVENT_SCHEMA_INVALID") - if ( - not isinstance(rec, dict) - or set(rec) - != {"sequence", "run163Documents", "logResponses", "gossipResponses"} - or rec["sequence"] != idx - ): - _fail("ARCHIVE_MERKLE_RECEIPT_EVENT_INVALID") - run163_docs = rec["run163Documents"] - if not isinstance(run163_docs, dict) or set(run163_docs) != _RUN163_NAMES: - _fail("ARCHIVE_MERKLE_EMBEDDED_RUN163_INVALID") - raws = {name: _canonical(run163_docs[name]) for name in sorted(run163_docs)} - if event["run163Artifacts"] != _run163_artifacts(raws): - _fail("ARCHIVE_MERKLE_RUN163_ARTIFACT_BINDING_INVALID") - state = run163_docs[_DOC_ARCHIVE_ANCHOR_STATE] - if event["run163Sequence"] != state.get("sequence") or event[ - "run163AnchorConsensusHeadSha256" - ] != state.get("anchorConsensusHeadSha256"): - _fail("ARCHIVE_MERKLE_RUN163_STATE_BINDING_INVALID") - leaf_doc = _leaf_document(run163_docs, raws) - leaf_hash = merkle_leaf_hash(_canonical(leaf_doc)) - if event["leafHash"] != leaf_hash: - _fail("ARCHIVE_MERKLE_LEAF_BINDING_INVALID") - log_responses = rec["logResponses"] - if not isinstance(log_responses, dict) or set(log_responses) != set( - root["logs"] - ): - _fail("ARCHIVE_MERKLE_LOG_RESPONSE_SET_INVALID") - log_rows = [] - latest = idx == len(events) - for log_id in sorted(root["logs"]): - parsed = _verify_checkpoint( - log_responses[log_id], - log_id=log_id, - config=root["logs"][log_id], - sequence=idx, - challenge=event["logChallenge"], - expected_leaf_hash=leaf_hash, - previous=previous_by_log[log_id], - now=now, - creation=False, - historical=historical or not latest, - ) - log_rows.append( - { - k: parsed[k] - for k in ( - "logId", - "checkpointSha256", - "treeSize", - "rootHash", - "leafIndex", - "leafHash", - ) - } - ) - previous_by_log[log_id] = parsed - if event["logs"] != log_rows: - _fail("ARCHIVE_MERKLE_LOG_EVENT_MISMATCH") - checkpoints = {row["logId"]: row["checkpointSha256"] for row in log_rows} - challenge = _gossip_challenge( - idx, event["run163AnchorConsensusHeadSha256"], leaf_hash, checkpoints - ) - if event["gossipChallenge"] != challenge: - _fail("ARCHIVE_MERKLE_GOSSIP_CHALLENGE_INVALID") - gossip = rec["gossipResponses"] - if not isinstance(gossip, dict) or set(gossip) != set(root["logs"]): - _fail("ARCHIVE_MERKLE_GOSSIP_RESPONSE_SET_INVALID") - gossip_hashes = [] - for log_id in sorted(root["logs"]): - parsed = _verify_gossip( - gossip[log_id], - log_id=log_id, - config=root["logs"][log_id], - sequence=idx, - challenge=challenge, - run163_head=event["run163AnchorConsensusHeadSha256"], - checkpoints=checkpoints, - now=now, - creation=False, - historical=historical or not latest, - ) - gossip_hashes.append(parsed["gossipSha256"]) - if event["gossipResponseSha256s"] != gossip_hashes: - _fail("ARCHIVE_MERKLE_GOSSIP_EVENT_MISMATCH") - bare = {k: event[k] for k in event if k != "merkleConsensusHeadSha256"} - head = _event_head(previous_head, bare) - if event["merkleConsensusHeadSha256"] != head: - _fail("ARCHIVE_MERKLE_CHAIN_HEAD_INVALID") - previous_head = head - if current_run163_docs is not None and current_run163_raws is not None: - last = events[-1] - current_state = current_run163_docs[_DOC_ARCHIVE_ANCHOR_STATE] - if last["run163Sequence"] != current_state.get("sequence") or last[ - "run163AnchorConsensusHeadSha256" - ] != current_state.get("anchorConsensusHeadSha256"): - _fail("ARCHIVE_MERKLE_CURRENT_RUN163_MISMATCH") - if last["run163Artifacts"] != _run163_artifacts(current_run163_raws): - _fail("ARCHIVE_MERKLE_CURRENT_RUN163_ARTIFACT_MISMATCH") - return { - "sequence": len(events), - "merkleConsensusHeadSha256": previous_head, - "last": events[-1], - } - - -def verify_merkle_history( # ruff: ignore[undocumented-public-function] - *, - run160_dir: Path, - run161_dir: Path, - retention_root_path: Path, - retention_root_pin: str, - witness_root_path: Path, - witness_root_pin: str, - bootstrap_pin: str, - recovery_pin: str, - attestation_pins: list[str], - run162_dir: Path, - anchor_plan_path: Path, - run163_dir: Path, - transparency_root_path: Path, - transparency_root_pin: str, - output_dir: Path, - now: datetime | None = None, - historical: bool = False, -) -> dict[str, Any]: - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - root_doc, _ = _read_json(Path(transparency_root_path), "ARCHIVE_MERKLE_ROOT") - root = verify_transparency_root( - root_doc, transparency_root_pin, now=current, historical=historical - ) - _enforce_external_separation(root, Path(witness_root_path), Path(anchor_plan_path)) - current163 = _verify_run163_current( - run160_dir=Path(run160_dir), - run161_dir=Path(run161_dir), - retention_root_path=Path(retention_root_path), - retention_root_pin=retention_root_pin, - witness_root_path=Path(witness_root_path), - witness_root_pin=witness_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - run162_dir=Path(run162_dir), - anchor_plan_path=Path(anchor_plan_path), - run163_dir=Path(run163_dir), - now=current, - historical=historical, - ) - docs, raws = _load_output(Path(output_dir)) - replay = _replay_history( - root=root, - bundle=docs["release-archive-merkle-bundle.json"], - receipt=docs["release-archive-merkle-receipt.json"], - now=current, - historical=historical, - current_run163_docs=current163["docs"], - current_run163_raws=current163["raws"], - ) - state = docs[_DOC_ARCHIVE_MERKLE_STATE] - active = docs["active-archive-merkle-evidence.json"] - expected_state = { - "schemaVersion": int(POLICY["state_schema_version"]), - "status": "trusted-archive-merkle-transparency", - "sequence": replay["sequence"], - "run163Sequence": replay["last"]["run163Sequence"], - "run163AnchorConsensusHeadSha256": replay["last"][ - "run163AnchorConsensusHeadSha256" - ], - "merkleConsensusHeadSha256": replay["merkleConsensusHeadSha256"], - "transparencyRootSha256": root["sha256"], - "bundleArtifact": { - "name": "release-archive-merkle-bundle.json", - "sha256": _sha_bytes(raws["release-archive-merkle-bundle.json"]), - "size": len(raws["release-archive-merkle-bundle.json"]), - }, - } - if state != expected_state: - _fail("ARCHIVE_MERKLE_STATE_MISMATCH") - last = replay["last"] - expected_active = { - "schemaVersion": int(POLICY["active_schema_version"]), - "status": "active-archive-merkle-transparency", - "sequence": replay["sequence"], - "run163AnchorConsensusHeadSha256": last["run163AnchorConsensusHeadSha256"], - "merkleConsensusHeadSha256": replay["merkleConsensusHeadSha256"], - "leafHash": last["leafHash"], - "logs": last["logs"], - "gossipResponseSha256s": last["gossipResponseSha256s"], - } - if active != expected_active: - _fail("ARCHIVE_MERKLE_ACTIVE_MISMATCH") - return { - "ok": True, - "sequence": replay["sequence"], - "merkle_consensus_head_sha256": replay["merkleConsensusHeadSha256"], - "transparency_root_sha256": root["sha256"], - } - - -def anchor_merkle_transparency( # ruff: ignore[too-many-branches, undocumented-public-function] - *, - run160_dir: Path, - run161_dir: Path, - retention_root_path: Path, - retention_root_pin: str, - witness_root_path: Path, - witness_root_pin: str, - bootstrap_pin: str, - recovery_pin: str, - attestation_pins: list[str], - run162_dir: Path, - anchor_plan_path: Path, - run163_dir: Path, - transparency_root_path: Path, - transparency_root_pin: str, - output_dir: Path, - adapters: list[ - tuple[ - str, - Callable[[dict[str, Any]], dict[str, Any]], - Callable[[dict[str, Any]], dict[str, Any]], - ] - ], - previous_output_dir: Path | None = None, - now: datetime | None = None, -) -> dict[str, Any]: - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - root_doc, _ = _read_json(Path(transparency_root_path), "ARCHIVE_MERKLE_ROOT") - root = verify_transparency_root(root_doc, transparency_root_pin, now=current) - _enforce_external_separation(root, Path(witness_root_path), Path(anchor_plan_path)) - current163 = _verify_run163_current( - run160_dir=Path(run160_dir), - run161_dir=Path(run161_dir), - retention_root_path=Path(retention_root_path), - retention_root_pin=retention_root_pin, - witness_root_path=Path(witness_root_path), - witness_root_pin=witness_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - run162_dir=Path(run162_dir), - anchor_plan_path=Path(anchor_plan_path), - run163_dir=Path(run163_dir), - now=current, - ) - authority_paths = [ - Path(run160_dir), - Path(run161_dir), - Path(retention_root_path), - Path(witness_root_path), - Path(run162_dir), - Path(anchor_plan_path), - Path(run163_dir), - Path(transparency_root_path), - ] - if previous_output_dir is not None: - authority_paths.append(Path(previous_output_dir)) - before = _authority_fingerprint(authority_paths, "ARCHIVE_MERKLE_INPUT_DRIFT") - state163 = current163["docs"][_DOC_ARCHIVE_ANCHOR_STATE] - sequence = state163.get("sequence") - if not isinstance(sequence, int) or sequence <= 0: - _fail("ARCHIVE_MERKLE_RUN163_SEQUENCE_INVALID") - - previous_head = None - previous_by_log = dict.fromkeys(root["logs"]) - old_events = [] - old_receipts = [] - if previous_output_dir is not None: - prev_docs, _ = _load_output(Path(previous_output_dir)) - prev_replay = _replay_history( - root=root, - bundle=prev_docs["release-archive-merkle-bundle.json"], - receipt=prev_docs["release-archive-merkle-receipt.json"], - now=current, - historical=True, - current_run163_docs=None, - current_run163_raws=None, - ) - if sequence != prev_replay["sequence"] + 1: - _fail("ARCHIVE_MERKLE_SEQUENCE_INVALID") - previous_head = prev_replay["merkleConsensusHeadSha256"] - old_events = list(prev_docs["release-archive-merkle-bundle.json"]["events"]) - old_receipts = list(prev_docs["release-archive-merkle-receipt.json"]["events"]) - for row in prev_replay["last"]["logs"]: - previous_by_log[row["logId"]] = row - elif sequence != 1: - _fail("ARCHIVE_MERKLE_PREVIOUS_OUTPUT_REQUIRED") - - if {item[0] for item in adapters} != set(root["logs"]) or len(adapters) != len( - root["logs"] - ): - _fail("ARCHIVE_MERKLE_ADAPTER_SET_INVALID") - amap = {lid: (log, gossip) for lid, log, gossip in adapters} - leaf_doc = _leaf_document(current163["docs"], current163["raws"]) - leaf_hash = merkle_leaf_hash(_canonical(leaf_doc)) - prechallenge = _sha_bytes( - _canonical( - { - "sequence": sequence, - "run163AnchorConsensusHeadSha256": state163[ - "anchorConsensusHeadSha256" - ], - "leafHash": leaf_hash, - "previousMerkleConsensusHeadSha256": previous_head, - } - ) - ) - log_docs = {} - log_rows = [] - for log_id in sorted(root["logs"]): - prev = previous_by_log[log_id] - request = { - "operation": "append-merkle-leaf", - "protocolVersion": 1, - "logId": log_id, - "sequence": sequence, - "challenge": prechallenge, - "leaf": leaf_doc, - "leafHash": leaf_hash, - "previousTreeSize": 0 if prev is None else prev["treeSize"], - "previousRootHash": None if prev is None else prev["rootHash"], - } - doc = amap[log_id][0](request) - parsed = _verify_checkpoint( - doc, - log_id=log_id, - config=root["logs"][log_id], - sequence=sequence, - challenge=prechallenge, - expected_leaf_hash=leaf_hash, - previous=prev, - now=current, - creation=True, - historical=False, - ) - log_docs[log_id] = doc - log_rows.append( - { - k: parsed[k] - for k in ( - "logId", - "checkpointSha256", - "treeSize", - "rootHash", - "leafIndex", - "leafHash", - ) - } - ) - checkpoints = {row["logId"]: row["checkpointSha256"] for row in log_rows} - gossip_challenge = _gossip_challenge( - sequence, state163["anchorConsensusHeadSha256"], leaf_hash, checkpoints - ) - # Log checkpoints sign the prechallenge; gossip signs the complete checkpoint map. Preserve both. - gossip_docs = {} - gossip_hashes = [] - for log_id in sorted(root["logs"]): - request = { - "operation": "gossip-merkle-checkpoints", - "protocolVersion": 1, - "sequence": sequence, - "challenge": gossip_challenge, - "run163AnchorConsensusHeadSha256": state163["anchorConsensusHeadSha256"], - "leafHash": leaf_hash, - "checkpointSha256s": checkpoints, - } - doc = amap[log_id][1](request) - parsed = _verify_gossip( - doc, - log_id=log_id, - config=root["logs"][log_id], - sequence=sequence, - challenge=gossip_challenge, - run163_head=state163["anchorConsensusHeadSha256"], - checkpoints=checkpoints, - now=current, - creation=True, - historical=False, - ) - gossip_docs[log_id] = doc - gossip_hashes.append(parsed["gossipSha256"]) - - event_bare = { - "sequence": sequence, - "run163Sequence": sequence, - "run163AnchorConsensusHeadSha256": state163["anchorConsensusHeadSha256"], - "run163Artifacts": _run163_artifacts(current163["raws"]), - "leafHash": leaf_hash, - "logs": log_rows, - "logChallenge": prechallenge, - "gossipChallenge": gossip_challenge, - "gossipResponseSha256s": gossip_hashes, - } - head = _event_head(previous_head, event_bare) - event = dict(event_bare) - event["merkleConsensusHeadSha256"] = head - receipt_event = { - "sequence": sequence, - "run163Documents": current163["docs"], - "logResponses": log_docs, - "gossipResponses": gossip_docs, - } - bundle = { - "schemaVersion": int(POLICY["bundle_schema_version"]), - "predicateType": PREDICATE_TYPE, - "status": "archive-merkle-transparency-history", - "transparencyRootSha256": root["sha256"], - "events": [*old_events, event], - } - receipt = { - "schemaVersion": int(POLICY["receipt_schema_version"]), - "status": "archive-merkle-transparency-accepted", - "events": [*old_receipts, receipt_event], - } - bundle_raw = _canonical(bundle) - state_doc = { - "schemaVersion": int(POLICY["state_schema_version"]), - "status": "trusted-archive-merkle-transparency", - "sequence": sequence, - "run163Sequence": sequence, - "run163AnchorConsensusHeadSha256": state163["anchorConsensusHeadSha256"], - "merkleConsensusHeadSha256": head, - "transparencyRootSha256": root["sha256"], - "bundleArtifact": { - "name": "release-archive-merkle-bundle.json", - "sha256": _sha_bytes(bundle_raw), - "size": len(bundle_raw), - }, - } - active = { - "schemaVersion": int(POLICY["active_schema_version"]), - "status": "active-archive-merkle-transparency", - "sequence": sequence, - "run163AnchorConsensusHeadSha256": state163["anchorConsensusHeadSha256"], - "merkleConsensusHeadSha256": head, - "leafHash": leaf_hash, - "logs": log_rows, - "gossipResponseSha256s": gossip_hashes, - } - if before != _authority_fingerprint(authority_paths, "ARCHIVE_MERKLE_INPUT_DRIFT"): - _fail("ARCHIVE_MERKLE_INPUT_DRIFT") - protected = [ - Path(run160_dir), - Path(run161_dir), - Path(retention_root_path), - Path(witness_root_path), - Path(run162_dir), - Path(anchor_plan_path), - Path(run163_dir), - Path(transparency_root_path), - ] - if previous_output_dir is not None: - protected.append(Path(previous_output_dir)) - target = _outside( - Path(output_dir), protected, "ARCHIVE_MERKLE_OUTPUT_OVERLAPS_INPUT" - ) - if target.exists(): - _fail("ARCHIVE_MERKLE_OUTPUT_EXISTS") - target.parent.mkdir(parents=True, exist_ok=True) - stage = Path(tempfile.mkdtemp(prefix=".run164-merkle-", dir=target.parent)) - try: - (stage / "release-archive-merkle-bundle.json").write_bytes(bundle_raw) - _write(stage / _DOC_ARCHIVE_MERKLE_STATE, state_doc) - _write(stage / "active-archive-merkle-evidence.json", active) - _write(stage / "release-archive-merkle-receipt.json", receipt) - verify_merkle_history( - run160_dir=run160_dir, - run161_dir=run161_dir, - retention_root_path=retention_root_path, - retention_root_pin=retention_root_pin, - witness_root_path=witness_root_path, - witness_root_pin=witness_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - run162_dir=run162_dir, - anchor_plan_path=anchor_plan_path, - run163_dir=run163_dir, - transparency_root_path=transparency_root_path, - transparency_root_pin=transparency_root_pin, - output_dir=stage, - now=current, - ) - stage.rename(target) - except Exception: - shutil.rmtree(stage, ignore_errors=True) - raise - return verify_merkle_history( - run160_dir=run160_dir, - run161_dir=run161_dir, - retention_root_path=retention_root_path, - retention_root_pin=retention_root_pin, - witness_root_path=witness_root_path, - witness_root_pin=witness_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - run162_dir=run162_dir, - anchor_plan_path=anchor_plan_path, - run163_dir=run163_dir, - transparency_root_path=transparency_root_path, - transparency_root_pin=transparency_root_pin, - output_dir=target, - now=current, - ) - - -def _common_args(parser) -> None: - parser.add_argument("--run160-dir", type=Path, required=True) - parser.add_argument("--run161-dir", type=Path, required=True) - parser.add_argument("--retention-root", type=Path, required=True) - parser.add_argument("--retention-root-pin", required=True) - parser.add_argument("--witness-root", type=Path, required=True) - parser.add_argument("--witness-root-pin", required=True) - parser.add_argument("--bootstrap-pin", required=True) - parser.add_argument("--recovery-pin", required=True) - parser.add_argument("--attestation-pin", action="append", default=[]) - parser.add_argument("--run162-dir", type=Path, required=True) - parser.add_argument("--anchor-plan", type=Path, required=True) - parser.add_argument("--run163-dir", type=Path, required=True) - parser.add_argument("--transparency-root", type=Path, required=True) - parser.add_argument("--transparency-root-pin", required=True) - - -def main(argv=None) -> int: # ruff: ignore[undocumented-public-function] - import argparse # ruff: ignore[import-outside-top-level] - - parser = argparse.ArgumentParser(description=__doc__) - subs = parser.add_subparsers(dest="command", required=True) - verify = subs.add_parser("verify") - _common_args(verify) - verify.add_argument("--output-dir", type=Path, required=True) - verify.add_argument("--historical", action="store_true") - anchor = subs.add_parser("anchor") - _common_args(anchor) - anchor.add_argument("--adapter-config", type=Path, required=True) - anchor.add_argument("--output-dir", type=Path, required=True) - anchor.add_argument("--previous-output-dir", type=Path) - ns = parser.parse_args(argv) - kwargs = { - "run160_dir": ns.run160_dir, - "run161_dir": ns.run161_dir, - "retention_root_path": ns.retention_root, - "retention_root_pin": ns.retention_root_pin, - "witness_root_path": ns.witness_root, - "witness_root_pin": ns.witness_root_pin, - "bootstrap_pin": ns.bootstrap_pin, - "recovery_pin": ns.recovery_pin, - "attestation_pins": ns.attestation_pin, - "run162_dir": ns.run162_dir, - "anchor_plan_path": ns.anchor_plan, - "run163_dir": ns.run163_dir, - "transparency_root_path": ns.transparency_root, - "transparency_root_pin": ns.transparency_root_pin, - "output_dir": ns.output_dir, - } - if ns.command == "verify": - result = verify_merkle_history(**kwargs, historical=ns.historical) - else: - cfg, _ = _read_json(ns.adapter_config, "ARCHIVE_MERKLE_ADAPTER_CONFIG") - if set(cfg) != {"logs"} or not isinstance(cfg["logs"], dict): - _fail("ARCHIVE_MERKLE_ADAPTER_CONFIG_SCHEMA_INVALID") - adapters = [] - for log_id, value in sorted(cfg["logs"].items()): - if not isinstance(value, dict) or set(value) != { - "logCommand", - "gossipCommand", - }: - _fail("ARCHIVE_MERKLE_ADAPTER_CONFIG_SCHEMA_INVALID") - if not all( - isinstance(v, list) and v and all(isinstance(x, str) and x for x in v) - for v in value.values() - ): - _fail("ARCHIVE_MERKLE_ADAPTER_CONFIG_SCHEMA_INVALID") - adapters.append( - ( - log_id, - command_log(value["logCommand"]), - command_gossip(value["gossipCommand"]), - ) - ) - result = anchor_merkle_transparency( - **kwargs, adapters=adapters, previous_output_dir=ns.previous_output_dir - ) - logger.info("Archive Merkle transparency %s completed", ns.command) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/security/verify_attestation_lifecycle.py b/security/verify_attestation_lifecycle.py deleted file mode 100644 index 62cdda480785b16640490ef9bd2b3701d7e0b2f5..0000000000000000000000000000000000000000 --- a/security/verify_attestation_lifecycle.py +++ /dev/null @@ -1,1679 +0,0 @@ -""" -Run 158: preserve and verify hardware-attestation lifecycle status. - -This layer consumes a verified Run 157 recovered-root continuity directory. It does -not replace Run 157 certificate-chain verification. Instead it adds four properties -that X.509 path validation alone does not provide: - -* root-threshold governed attestation-CA/status-authority metadata; -* short-lived threshold-signed certificate status snapshots; -* certificate-bound vendor/device semantic claims; and -* deterministic historical proof of status at the exact attestation time. - -No signing private key is read by this module. Every signature is supplied as -canonical external evidence and verified in-process with Ed25519. -""" - -from __future__ import annotations - -import argparse -import base64 -import binascii -import hashlib -import json -import logging -import os -import re -import shutil -import sys -import tempfile -from datetime import datetime, timedelta, timezone -from pathlib import Path -from typing import Any, Iterable - -import tomllib - -logger = logging.getLogger(__name__) - -try: - from . import continue_recovered_trust as continuity - from . import seal_release_governance as rootseal -except (ImportError, ValueError): - import importlib.util - - _here = Path(__file__).resolve().parent - - def _load(name: str, filename: str): - if name in sys.modules: - return sys.modules[name] - spec = importlib.util.spec_from_file_location(name, _here / filename) - if spec is None or spec.loader is None: - raise ImportError(filename) - mod = importlib.util.module_from_spec(spec) - sys.modules[name] = mod - spec.loader.exec_module(mod) - return mod - - continuity = _load( - "_run157_continuity_for_lifecycle", "continue_recovered_trust.py" - ) - rootseal = _load("_run155_root_for_lifecycle", "seal_release_governance.py") - -HERE = Path(__file__).resolve().parent -POLICY = tomllib.loads((HERE / "release_attestation_lifecycle_policy.toml").read_text()) -PREDICATE_TYPE = str(POLICY["predicate_type"]) -_ZERO_HASH = "0" * 64 -_HEX64 = re.compile(r"^[0-9a-f]{64}$") -_ID = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:/@+-]{0,511}$") -_CHUNK = 1024 * 1024 - - -class AttestationLifecycleError(RuntimeError): - """Attestation lifecycle, status, or vendor semantic invariant failed.""" - - -def _fail(code: str) -> None: - raise AttestationLifecycleError(code) - - -def _canonical(value: Any) -> bytes: - return ( - json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False) - + "\n" - ).encode("utf-8") - - -def _sha_bytes(raw: bytes) -> str: - return hashlib.sha256(raw).hexdigest() - - -def _sha(path: Path) -> str: - h = hashlib.sha256() - with path.open("rb") as handle: - for chunk in iter(lambda: handle.read(_CHUNK), b""): - h.update(chunk) - return h.hexdigest() - - -def _write(path: Path, value: dict[str, Any]) -> None: - path.write_bytes(_canonical(value)) - - -def _loads(raw: bytes, code: str) -> dict[str, Any]: - def hook(pairs): - out: dict[str, Any] = {} - for key, value in pairs: - if key in out: - _fail(code + "_DUPLICATE_KEY") - out[key] = value - return out - - try: - value = json.loads(raw.decode("utf-8"), object_pairs_hook=hook) - except AttestationLifecycleError: - raise - except Exception as exc: - raise AttestationLifecycleError(code + "_JSON_INVALID") from exc - if not isinstance(value, dict): - _fail(code + "_SCHEMA_INVALID") - return value - - -def _read( - path: Path, code: str, *, canonical: bool = True -) -> tuple[dict[str, Any], bytes]: - if path.is_symlink() or not path.is_file(): - _fail(code + "_INVALID") - if path.stat().st_size > int(POLICY["max_json_bytes"]): - _fail(code + "_TOO_LARGE") - raw = path.read_bytes() - doc = _loads(raw, code) - if canonical and raw != _canonical(doc): - _fail(code + "_NOT_CANONICAL") - return doc, raw - - -def _identity(value: Any, code: str) -> str: - if not isinstance(value, str): - _fail(code) - value = value.strip() - if ( - not value - or len(value) > 512 # ruff: ignore[magic-value-comparison] - or ".." in value - or "?" in value - or "#" in value - or "\x00" in value - ): - _fail(code) - if ( - any( - ord(c) < 32 # ruff: ignore[magic-value-comparison] - or ord(c) == 127 # ruff: ignore[magic-value-comparison] - for c in value - ) - or _ID.fullmatch(value) is None - ): - _fail(code) - return value - - -def _hex(value: Any, code: str) -> str: - if not isinstance(value, str) or _HEX64.fullmatch(value) is None: - _fail(code) - return value - - -def _int(value: Any, code: str, *, minimum: int = 0) -> int: - if not isinstance(value, int) or isinstance(value, bool) or value < minimum: - _fail(code) - return value - - -def _dt(value: Any, code: str) -> datetime: - if not isinstance(value, str) or not value.endswith("Z"): - _fail(code) - try: - return datetime.fromisoformat(value[:-1] + "+00:00").astimezone(timezone.utc) - except ValueError as exc: - raise AttestationLifecycleError(code) from exc - - -def _ts(value: datetime) -> str: - return ( - value.astimezone(timezone.utc) - .isoformat(timespec="seconds") - .replace("+00:00", "Z") - ) - - -def _b64(value: Any, code: str, *, expected_len: int | None = None) -> bytes: - if ( - not isinstance(value, str) - or not value - or len(value) > int(POLICY["max_json_bytes"]) - ): - _fail(code) - try: - raw = base64.b64decode(value.encode("ascii"), validate=True) - except (UnicodeEncodeError, binascii.Error) as exc: - raise AttestationLifecycleError(code) from exc - if base64.b64encode(raw).decode("ascii") != value or ( - expected_len is not None and len(raw) != expected_len - ): - _fail(code) - return raw - - -def _ed25519_verify(public: str, signature: str, message: bytes, code: str) -> None: - try: - rootseal._ed25519_verify( - rootseal._b64(public, code + "_PUBLIC", expected_len=32), - rootseal._b64(signature, code + "_SIGNATURE", expected_len=64), - message, - code, - ) - except rootseal.RootTrustError as exc: - raise AttestationLifecycleError(str(exc)) from exc - - -def _regular_dir(path: Path, code: str) -> Path: - if path.is_symlink() or not path.is_dir(): - _fail(code) - return path.resolve() - - -def _outside(path: Path, protected: Iterable[Path], code: str) -> Path: - target = path.expanduser().resolve() - for item in protected: - root = item.resolve() - if target == root or root in target.parents: - _fail(code) - return target - - -def _artifact_raw(name: str, raw: bytes) -> dict[str, Any]: - return {"name": name, "sha256": _sha_bytes(raw), "size": len(raw)} - - -def _docs_from_dir(root: Path, names: set[str], code: str) -> dict[str, dict[str, Any]]: - if {p.name for p in root.iterdir()} != names: - _fail(code + "_ALLOWLIST_MISMATCH") - out = {} - for name in sorted(names): - out[name] = _read( - root / name, code + "_" + name.upper().replace("-", "_").replace(".", "_") - )[0] - return out - - -def _write_docs(root: Path, docs: dict[str, dict[str, Any]]) -> None: - root.mkdir(parents=True, exist_ok=True) - for name, doc in sorted(docs.items()): - _write(root / name, doc) - - -# ── Verified-state document names ──────────────────────────────────────── -# -# One constant per release-evidence document. The values are the on-disk -# artifact names and are unchanged; only the repetition is removed. Reading -# a document by a name that says which document it is also keeps static -# analysis from reading the filename token 'trusted' as 'confidential': -# these files carry published Merkle roots and SHA-256 digests, which must -# stay in clear text for any third party to verify them. -_DOC_ATTESTATION_LIFECYCLE_STATE = "trusted-attestation-lifecycle-state.json" -_DOC_ROOT_CONTINUITY_STATE = "trusted-root-continuity-state.json" - - -def _continuity_docs(root: Path) -> dict[str, dict[str, Any]]: - names = { - "release-root-continuity-bundle.json", - _DOC_ROOT_CONTINUITY_STATE, - "active-root.json", - "release-root-continuity-receipt.json", - } - return _docs_from_dir(root, names, "ATTESTATION_LIFECYCLE_CONTINUITY") - - -def _root_info(envelope: dict[str, Any], code: str) -> dict[str, Any]: - try: - return rootseal._validate_root_envelope(envelope) - except rootseal.RootTrustError as exc: - raise AttestationLifecycleError(code + ":" + str(exc)) from exc - - -def _cert_der(value: str, code: str): - try: - from cryptography import x509 # ruff: ignore[import-outside-top-level] - - der = _b64(value, code + "_DER_INVALID") - cert = x509.load_der_x509_certificate(der) - return der, cert - except AttestationLifecycleError: - raise - except Exception as exc: - raise AttestationLifecycleError(code + "_CERT_INVALID") from exc - - -def _verify_ca_item(item: Any, *, now: datetime, historical: bool) -> dict[str, Any]: - if not isinstance(item, dict) or set(item) != {"sha256", "der", "profile"}: - _fail("ATTESTATION_CA_ITEM_SCHEMA_INVALID") - der, cert = _cert_der(item["der"], "ATTESTATION_CA") - sha = _hex(item["sha256"], "ATTESTATION_CA_SHA_INVALID") - if _sha_bytes(der) != sha: - _fail("ATTESTATION_CA_HASH_MISMATCH") - profile = _identity(item["profile"], "ATTESTATION_CA_PROFILE_INVALID") - try: - from cryptography import x509 # ruff: ignore[import-outside-top-level] - - bc = cert.extensions.get_extension_for_class(x509.BasicConstraints).value - ku = cert.extensions.get_extension_for_class(x509.KeyUsage).value - if not bc.ca or not ku.key_cert_sign or not ku.crl_sign: - _fail("ATTESTATION_CA_CONSTRAINTS_INVALID") - if cert.issuer != cert.subject: - _fail("ATTESTATION_CA_NOT_SELF_ISSUED") - continuity._verify_cert_signature( - cert, cert.public_key(), "ATTESTATION_CA_SELF" - ) - nb = continuity._cert_time(cert, "not_valid_before") - na = continuity._cert_time(cert, "not_valid_after") - if not historical and not (nb <= now <= na): - _fail("ATTESTATION_CA_NOT_CURRENT") - except AttestationLifecycleError: - raise - except continuity.RootContinuityError as exc: - raise AttestationLifecycleError(str(exc)) from exc - except Exception as exc: - raise AttestationLifecycleError("ATTESTATION_CA_CERT_INVALID") from exc - return { - "sha256": sha, - "der": base64.b64encode(der).decode("ascii"), - "profile": profile, - } - - -def _status_key(value: Any, code: str) -> dict[str, Any]: - expected = {"keytype", "scheme", "identity", "operator", "expires", "keyval"} - if ( - not isinstance(value, dict) - or set(value) != expected - or value.get("keytype") != "ed25519" - or value.get("scheme") != "ed25519" - ): - _fail(code + "_SCHEMA_INVALID") - keyval = value.get("keyval") - if not isinstance(keyval, dict) or set(keyval) != {"public"}: - _fail(code + "_KEYVAL_INVALID") - rootseal._b64(keyval["public"], code + "_PUBLIC_INVALID", expected_len=32) - return { - "keytype": "ed25519", - "scheme": "ed25519", - "identity": _identity(value["identity"], code + "_IDENTITY_INVALID"), - "operator": _identity(value["operator"], code + "_OPERATOR_INVALID"), - "expires": _ts(_dt(value["expires"], code + "_EXPIRES_INVALID")), - "keyval": {"public": keyval["public"]}, - } - - -def _status_authority( - value: Any, *, active_root: dict[str, Any], ca_expires: datetime -) -> dict[str, Any]: - if not isinstance(value, dict) or set(value) != {"keys", "role"}: - _fail("ATTESTATION_STATUS_AUTHORITY_SCHEMA_INVALID") - raw_keys = value["keys"] - if not isinstance(raw_keys, dict) or not ( - int(POLICY["min_status_keys"]) - <= len(raw_keys) - <= int(POLICY["max_status_keys"]) - ): - _fail("ATTESTATION_STATUS_KEY_COUNT_INVALID") - keys = { - _identity(k, "ATTESTATION_STATUS_KEY_ID_INVALID"): _status_key( - v, "ATTESTATION_STATUS_KEY" - ) - for k, v in raw_keys.items() - } - role = value["role"] - if not isinstance(role, dict) or set(role) != {"keyids", "threshold"}: - _fail("ATTESTATION_STATUS_ROLE_SCHEMA_INVALID") - keyids = ( - [_identity(x, "ATTESTATION_STATUS_ROLE_KEY_INVALID") for x in role["keyids"]] - if isinstance(role["keyids"], list) - else [] - ) - if ( - len(set(keyids)) != len(keyids) - or sorted(keyids) != keyids - or set(keyids) != set(keys) - ): - _fail("ATTESTATION_STATUS_ROLE_KEY_SET_INVALID") - threshold = _int( - role["threshold"], "ATTESTATION_STATUS_THRESHOLD_INVALID", minimum=1 - ) - if threshold < int(POLICY["min_status_threshold"]) or threshold > len(keyids): - _fail("ATTESTATION_STATUS_THRESHOLD_INVALID") - identities = {keys[k]["identity"] for k in keyids} - if len(identities) != len(keyids): - _fail("ATTESTATION_STATUS_IDENTITY_DUPLICATE") - operators = {keys[k]["operator"] for k in keyids} - if len(operators) < int(POLICY["min_status_operators"]): - _fail("ATTESTATION_STATUS_OPERATOR_QUORUM_INVALID") - active_root_ids = set(active_root["roles"]["root"]["keyids"]) - active_root_ops = {active_root["keys"][k]["operator"] for k in active_root_ids} - if ( - bool(POLICY["require_status_key_disjoint_from_release_root"]) - and set(keyids) & active_root_ids - ): - _fail("ATTESTATION_STATUS_KEY_OVERLAPS_RELEASE_ROOT") - if ( - bool(POLICY["require_status_operator_disjoint_from_release_root"]) - and operators & active_root_ops - ): - _fail("ATTESTATION_STATUS_OPERATOR_OVERLAPS_RELEASE_ROOT") - for key in keys.values(): - if _dt(key["expires"], "ATTESTATION_STATUS_KEY_EXPIRES_INVALID") < ca_expires: - _fail("ATTESTATION_STATUS_KEY_EXPIRES_BEFORE_CA_SET") - return { - "keys": {k: keys[k] for k in sorted(keys)}, - "role": {"keyids": sorted(keyids), "threshold": threshold}, - } - - -def _continuity_binding(docs: dict[str, dict[str, Any]]) -> dict[str, Any]: - state = docs[_DOC_ROOT_CONTINUITY_STATE] - bundle = docs["release-root-continuity-bundle.json"] - active = _root_info(docs["active-root.json"], "ATTESTATION_ACTIVE_ROOT_INVALID") - if state.get("rootChainHeadSha256") != bundle.get("recoveryEvent", {}).get( - "chainHeadSha256" - ) and not bundle.get("epochs"): - _fail("ATTESTATION_CONTINUITY_HEAD_MISMATCH") - return { - "state": _artifact_raw(_DOC_ROOT_CONTINUITY_STATE, _canonical(state)), - "bundle": _artifact_raw( - "release-root-continuity-bundle.json", _canonical(bundle) - ), - "rootChainHeadSha256": _hex( - state.get("rootChainHeadSha256"), "ATTESTATION_CONTINUITY_HEAD_INVALID" - ), - "activeRoot": {"version": active["version"], "sha256": active["sha256"]}, - } - - -def _verify_ca_set( # ruff: ignore[too-many-branches] - doc: dict[str, Any], - *, - continuity_docs: dict[str, dict[str, Any]], - previous: dict[str, Any] | None, - now: datetime, - historical: bool, -) -> dict[str, Any]: - if set(doc) != {"signatures", "signed"} or not isinstance(doc.get("signed"), dict): - _fail("ATTESTATION_CA_SET_SCHEMA_INVALID") - s = doc["signed"] - expected = { - "_type", - "specVersion", - "version", - "governanceId", - "issuedAt", - "expires", - "continuity", - "previousCaSetSha256", - "transition", - "trustRoots", - "statusAuthority", - "requiredProfilesByDeviceClass", - "selectedRootKeyIds", - } - if ( - set(s) != expected - or s["_type"] != "attestation-ca-set" - or s["specVersion"] != str(POLICY["spec_version"]) - ): - _fail("ATTESTATION_CA_SET_SIGNED_SCHEMA_INVALID") - version = _int(s["version"], "ATTESTATION_CA_SET_VERSION_INVALID", minimum=1) - active = _root_info( - continuity_docs["active-root.json"], "ATTESTATION_CA_SET_ACTIVE_ROOT_INVALID" - ) - if s["governanceId"] != active["governance_id"]: - _fail("ATTESTATION_CA_SET_GOVERNANCE_ID_MISMATCH") - binding = _continuity_binding(continuity_docs) - if s["continuity"] != binding: - _fail("ATTESTATION_CA_SET_CONTINUITY_MISMATCH") - issued = _dt(s["issuedAt"], "ATTESTATION_CA_SET_ISSUED_INVALID") - expires = _dt(s["expires"], "ATTESTATION_CA_SET_EXPIRES_INVALID") - skew = timedelta(minutes=int(POLICY["max_clock_skew_minutes"])) - if issued > now + skew: - _fail("ATTESTATION_CA_SET_FROM_FUTURE") - if expires <= issued or expires - issued > timedelta( - days=int(POLICY["max_ca_set_lifetime_days"]) - ): - _fail("ATTESTATION_CA_SET_LIFETIME_INVALID") - if not historical and expires < now + timedelta( - minutes=int(POLICY["min_ca_set_remaining_minutes"]) - ): - _fail("ATTESTATION_CA_SET_EXPIRED_OR_FREEZE_RISK") - if issued < active["issued"]: - _fail("ATTESTATION_CA_SET_PREDATES_ACTIVE_ROOT") - if expires > active["expires"]: - _fail("ATTESTATION_CA_SET_OUTLIVES_ACTIVE_ROOT") - roots_raw = s["trustRoots"] - if not isinstance(roots_raw, list) or not ( - 1 <= len(roots_raw) <= int(POLICY["max_ca_roots"]) - ): - _fail("ATTESTATION_CA_SET_ROOTS_INVALID") - roots = [_verify_ca_item(x, now=now, historical=historical) for x in roots_raw] - roots.sort(key=lambda x: x["sha256"]) - if roots != roots_raw or len({x["sha256"] for x in roots}) != len(roots): - _fail("ATTESTATION_CA_SET_ROOTS_NOT_NORMALIZED") - try: - from cryptography import x509 # ruff: ignore[import-outside-top-level] - - for item in roots: - cert = x509.load_der_x509_certificate( - _b64(item["der"], "ATTESTATION_CA_SET_ROOT_DER_INVALID") - ) - if issued < continuity._cert_time( - cert, "not_valid_before" - ) or expires > continuity._cert_time(cert, "not_valid_after"): - _fail("ATTESTATION_CA_SET_ROOT_VALIDITY_MISMATCH") - except AttestationLifecycleError: - raise - except Exception as exc: - raise AttestationLifecycleError("ATTESTATION_CA_SET_ROOT_CERT_INVALID") from exc - profile_map = s["requiredProfilesByDeviceClass"] - required_map = { - str(k): str(v) for k, v in POLICY["required_profile_by_device_class"].items() - } - if profile_map != required_map: - _fail("ATTESTATION_CA_SET_PROFILE_POLICY_MISMATCH") - allowed_profiles = set(required_map.values()) - if any(item["profile"] not in allowed_profiles for item in roots): - _fail("ATTESTATION_CA_SET_ROOT_PROFILE_INVALID") - authority = _status_authority( - s["statusAuthority"], active_root=active, ca_expires=expires - ) - transition = s["transition"] - if not isinstance(transition, dict) or set(transition) != { - "type", - "retiredCaSha256", - "revokedCaSha256", - }: - _fail("ATTESTATION_CA_SET_TRANSITION_SCHEMA_INVALID") - typ = transition["type"] - if typ not in {"bootstrap", "scheduled-rotation", "compromise-recovery"}: - _fail("ATTESTATION_CA_SET_TRANSITION_TYPE_INVALID") - retired = ( - [ - _hex(x, "ATTESTATION_CA_SET_RETIRED_CA_INVALID") - for x in transition["retiredCaSha256"] - ] - if isinstance(transition["retiredCaSha256"], list) - else [] - ) - revoked = ( - [ - _hex(x, "ATTESTATION_CA_SET_REVOKED_CA_INVALID") - for x in transition["revokedCaSha256"] - ] - if isinstance(transition["revokedCaSha256"], list) - else [] - ) - if ( - sorted(set(retired)) != retired - or sorted(set(revoked)) != revoked - or set(retired) & set(revoked) - ): - _fail("ATTESTATION_CA_SET_TRANSITION_CA_SET_INVALID") - current_set = {x["sha256"] for x in roots} - if previous is None: - if ( - version != 1 - or s["previousCaSetSha256"] != _ZERO_HASH - or typ != "bootstrap" - or retired - or revoked - ): - _fail("ATTESTATION_CA_SET_BOOTSTRAP_INVALID") - embedded = continuity_docs["release-root-continuity-bundle.json"].get( - "attestationTrustRoots" - ) - embedded_pins = ( - sorted(x.get("sha256") for x in embedded) - if isinstance(embedded, list) - else [] - ) - if sorted(current_set) != embedded_pins: - _fail("ATTESTATION_CA_SET_BOOTSTRAP_ROOT_SET_MISMATCH") - else: - prev_signed = previous["signed"] - prev_sha = _sha_bytes(_canonical(previous)) - if ( - version != prev_signed["version"] + 1 - or s["previousCaSetSha256"] != prev_sha - or typ == "bootstrap" - ): - _fail("ATTESTATION_CA_SET_ROTATION_CONTINUITY_INVALID") - if issued < _dt( - prev_signed["issuedAt"], "ATTESTATION_PREVIOUS_CA_SET_ISSUED_INVALID" - ): - _fail("ATTESTATION_CA_SET_ISSUED_AT_ROLLBACK") - old = {x["sha256"] for x in prev_signed["trustRoots"]} - removed = old - current_set - if removed != set(retired) | set(revoked): - _fail("ATTESTATION_CA_SET_REMOVAL_ACCOUNTING_INVALID") - if typ == "compromise-recovery" and not revoked: - _fail("ATTESTATION_CA_SET_COMPROMISE_REQUIRES_REVOCATION") - if typ == "scheduled-rotation" and revoked: - _fail("ATTESTATION_CA_SET_SCHEDULED_ROTATION_CANNOT_REVOKE") - normalized_signed = dict(s) - normalized_signed["trustRoots"] = roots - normalized_signed["statusAuthority"] = authority - normalized_signed["issuedAt"] = _ts(issued) - normalized_signed["expires"] = _ts(expires) - normalized_signed["transition"] = { - "type": typ, - "retiredCaSha256": retired, - "revokedCaSha256": revoked, - } - if normalized_signed != s: - _fail("ATTESTATION_CA_SET_NOT_NORMALIZED") - role = active["roles"]["root"] - selected = s["selectedRootKeyIds"] - if not isinstance(selected, list): - _fail("ATTESTATION_CA_SET_SELECTED_ROOT_KEYS_INVALID") - selected = [ - _identity(x, "ATTESTATION_CA_SET_SELECTED_ROOT_KEY_INVALID") for x in selected - ] - if ( - sorted(set(selected)) != selected - or len(selected) != role["threshold"] - or not set(selected).issubset(set(role["keyids"])) - ): - _fail("ATTESTATION_CA_SET_SELECTED_ROOT_KEYS_INVALID") - if len({active["keys"][kid]["operator"] for kid in selected}) < int( - POLICY["min_root_signer_operators"] - ): - _fail("ATTESTATION_CA_SET_ROOT_SIGNER_OPERATOR_QUORUM_INVALID") - sigs = doc["signatures"] - if not isinstance(sigs, list): - _fail("ATTESTATION_CA_SET_SIGNATURES_INVALID") - sigmap: dict[str, str] = {} - normalized_sigs = [] - for item in sigs: - if not isinstance(item, dict) or set(item) != {"keyid", "sig"}: - _fail("ATTESTATION_CA_SET_SIGNATURE_SCHEMA_INVALID") - kid = _identity(item["keyid"], "ATTESTATION_CA_SET_SIGNATURE_KEY_INVALID") - if kid in sigmap: - _fail("ATTESTATION_CA_SET_SIGNATURE_DUPLICATE") - rootseal._b64( - item["sig"], - "ATTESTATION_CA_SET_SIGNATURE_ENCODING_INVALID", - expected_len=64, - ) - sigmap[kid] = item["sig"] - normalized_sigs.append({"keyid": kid, "sig": item["sig"]}) - normalized_sigs.sort(key=lambda x: x["keyid"]) - if normalized_sigs != sigs: - _fail("ATTESTATION_CA_SET_SIGNATURES_NOT_NORMALIZED") - if set(sigmap) != set(selected): - _fail("ATTESTATION_CA_SET_SIGNATURE_SET_MISMATCH") - valid = [] - for kid in role["keyids"]: - if kid not in sigmap: - continue - key = active["keys"][kid] - if _dt(key["expires"], "ATTESTATION_CA_SET_ROOT_KEY_EXPIRES_INVALID") < issued: - continue - _ed25519_verify( - key["keyval"]["public"], - sigmap[kid], - _canonical(s), - "ATTESTATION_CA_SET_ROOT_SIGNATURE_INVALID", - ) - valid.append(kid) - if len(valid) < role["threshold"]: - _fail("ATTESTATION_CA_SET_ROOT_THRESHOLD_NOT_MET") - if len({active["keys"][kid]["operator"] for kid in valid}) < int( - POLICY["min_root_signer_operators"] - ): - _fail("ATTESTATION_CA_SET_ROOT_SIGNER_OPERATOR_QUORUM_INVALID") - return {"signatures": normalized_sigs, "signed": normalized_signed} - - -def _claims_from_attestation( - attestation: dict[str, Any], *, ca_set: dict[str, Any] -) -> dict[str, Any]: - s = attestation["signed"] - chain = s.get("certificateChainDer") - if not isinstance(chain, list) or not chain: - _fail("ATTESTATION_VENDOR_CERT_CHAIN_INVALID") - der, cert = _cert_der(chain[0], "ATTESTATION_VENDOR_LEAF") - required_map = ca_set["signed"]["requiredProfilesByDeviceClass"] - device = s.get("deviceClass") - if device not in required_map: - _fail("ATTESTATION_VENDOR_DEVICE_CLASS_UNSUPPORTED") - required_profile = required_map[device] - try: - from cryptography import x509 # ruff: ignore[import-outside-top-level] - - oid = x509.ObjectIdentifier(str(POLICY["vendor_claim_extension_oid"])) - ext = cert.extensions.get_extension_for_oid(oid) - raw = bytes(ext.value.value) - except Exception as exc: - raise AttestationLifecycleError( - "ATTESTATION_VENDOR_CLAIM_EXTENSION_MISSING" - ) from exc - claim = _loads(raw, "ATTESTATION_VENDOR_CLAIM") - expected = { - "schemaVersion", - "profile", - "deviceClass", - "manufacturer", - "model", - "hardwareBacked", - "nonExportable", - "keyId", - "publicKeySha256", - "deviceIdHash", - } - if set(claim) != expected or claim["schemaVersion"] != int( - POLICY["vendor_claim_schema_version"] - ): - _fail("ATTESTATION_VENDOR_CLAIM_SCHEMA_INVALID") - if raw != _canonical(claim): - _fail("ATTESTATION_VENDOR_CLAIM_NOT_CANONICAL") - if ( - claim["profile"] != required_profile - or claim["deviceClass"] != device - or claim["manufacturer"] != s["manufacturer"] - or claim["model"] != s["model"] - ): - _fail("ATTESTATION_VENDOR_CLAIM_SEMANTICS_MISMATCH") - if claim["hardwareBacked"] is not True or claim["nonExportable"] is not True: - _fail("ATTESTATION_VENDOR_CLAIM_HARDWARE_PROPERTIES_INVALID") - if claim["keyId"] != s["keyId"]: - _fail("ATTESTATION_VENDOR_CLAIM_KEYID_MISMATCH") - public_raw = rootseal._b64( - s["publicKey"], "ATTESTATION_VENDOR_ROOT_PUBLIC_INVALID", expected_len=32 - ) - if claim["publicKeySha256"] != _sha_bytes(public_raw): - _fail("ATTESTATION_VENDOR_CLAIM_ROOT_PUBLIC_MISMATCH") - _hex(claim["deviceIdHash"], "ATTESTATION_VENDOR_DEVICE_ID_INVALID") - return { - "keyId": s["keyId"], - "deviceIdHash": claim["deviceIdHash"], - "profile": claim["profile"], - "leafSha256": _sha_bytes(der), - } - - -def _attestations_from_continuity(bundle: dict[str, Any]) -> list[dict[str, Any]]: - out = [] - recovery = bundle.get("recoveryAttestations") - if not isinstance(recovery, list): - _fail("ATTESTATION_CONTINUITY_ATTESTATIONS_INVALID") - out.extend(recovery) - epochs = bundle.get("epochs") - if not isinstance(epochs, list): - _fail("ATTESTATION_CONTINUITY_EPOCHS_INVALID") - for entry in epochs: - if not isinstance(entry, dict): - _fail("ATTESTATION_CONTINUITY_EPOCH_INVALID") - docs = entry.get("rootAttestations") - if docs: - if not isinstance(docs, list): - _fail("ATTESTATION_CONTINUITY_EPOCH_ATTESTATIONS_INVALID") - out.extend(docs) - return out - - -def _inventory( - continuity_bundle: dict[str, Any], *, ca_set: dict[str, Any] -) -> tuple[list[dict[str, Any]], dict[str, str]]: - certs: dict[str, dict[str, Any]] = {} - bindings: dict[str, str] = {} - for attestation in _attestations_from_continuity(continuity_bundle): - if not isinstance(attestation, dict) or not isinstance( - attestation.get("signed"), dict - ): - _fail("ATTESTATION_CONTINUITY_ATTESTATION_SCHEMA_INVALID") - s = attestation["signed"] - vendor = _claims_from_attestation(attestation, ca_set=ca_set) - previous = bindings.get(vendor["keyId"]) - if previous is not None and previous != vendor["deviceIdHash"]: - _fail("ATTESTATION_DEVICE_IDENTITY_CHANGED") - bindings[vendor["keyId"]] = vendor["deviceIdHash"] - attested = _dt(s["attestedAt"], "ATTESTATION_INVENTORY_ATTESTED_AT_INVALID") - chain = s["certificateChainDer"] - for _index, text in enumerate(chain): - der, cert = _cert_der(text, "ATTESTATION_INVENTORY_CERT") - sha = _sha_bytes(der) - item = certs.setdefault( - sha, - { - "sha256": sha, - "serialNumber": format(cert.serial_number, "x"), - "keyIds": [], - "attestedAt": [], - }, - ) - if item["serialNumber"] != format(cert.serial_number, "x"): - _fail("ATTESTATION_INVENTORY_CERT_COLLISION") - if s["keyId"] not in item["keyIds"]: - item["keyIds"].append(s["keyId"]) - ts = _ts(attested) - if ts not in item["attestedAt"]: - item["attestedAt"].append(ts) - normalized = [] - for item in certs.values(): - item["keyIds"].sort() - item["attestedAt"].sort() - normalized.append(item) - normalized.sort(key=lambda x: x["sha256"]) - if len(normalized) > int(POLICY["max_status_entries"]): - _fail("ATTESTATION_INVENTORY_TOO_LARGE") - return normalized, {k: bindings[k] for k in sorted(bindings)} - - -def _verify_status_snapshot( # ruff: ignore[too-many-branches] - doc: dict[str, Any], - *, - ca_set: dict[str, Any], - continuity_bundle: dict[str, Any], - previous: dict[str, Any] | None, - now: datetime, - historical: bool, -) -> dict[str, Any]: - if set(doc) != {"signatures", "signed"} or not isinstance(doc.get("signed"), dict): - _fail("ATTESTATION_STATUS_SCHEMA_INVALID") - s = doc["signed"] - expected = { - "_type", - "specVersion", - "version", - "governanceId", - "caSetVersion", - "caSetSha256", - "previousStatusSha256", - "continuityRootChainHeadSha256", - "issuedAt", - "nextUpdate", - "certificates", - "selectedStatusKeyIds", - } - if ( - set(s) != expected - or s["_type"] != "attestation-status" - or s["specVersion"] != str(POLICY["spec_version"]) - ): - _fail("ATTESTATION_STATUS_SIGNED_SCHEMA_INVALID") - ca_signed = ca_set["signed"] - if ( - s["governanceId"] != ca_signed["governanceId"] - or s["caSetVersion"] != ca_signed["version"] - or s["caSetSha256"] != _sha_bytes(_canonical(ca_set)) - ): - _fail("ATTESTATION_STATUS_CA_SET_MISMATCH") - expected_head = ca_signed["continuity"]["rootChainHeadSha256"] - if s["continuityRootChainHeadSha256"] != expected_head: - _fail("ATTESTATION_STATUS_CONTINUITY_MISMATCH") - version = _int(s["version"], "ATTESTATION_STATUS_VERSION_INVALID", minimum=1) - if previous is None: - if version != 1 or s["previousStatusSha256"] != _ZERO_HASH: - _fail("ATTESTATION_STATUS_BOOTSTRAP_VERSION_INVALID") - else: - if version != previous["signed"]["version"] + 1: - _fail("ATTESTATION_STATUS_VERSION_NOT_CONSECUTIVE") - if s["previousStatusSha256"] != _sha_bytes(_canonical(previous)): - _fail("ATTESTATION_STATUS_PREVIOUS_HASH_MISMATCH") - issued = _dt(s["issuedAt"], "ATTESTATION_STATUS_ISSUED_INVALID") - next_update = _dt(s["nextUpdate"], "ATTESTATION_STATUS_NEXT_UPDATE_INVALID") - skew = timedelta(minutes=int(POLICY["max_clock_skew_minutes"])) - if issued > now + skew: - _fail("ATTESTATION_STATUS_FROM_FUTURE") - if next_update <= issued or next_update - issued > timedelta( - hours=int(POLICY["max_status_lifetime_hours"]) - ): - _fail("ATTESTATION_STATUS_LIFETIME_INVALID") - if not historical and next_update < now + timedelta( - minutes=int(POLICY["min_status_remaining_minutes"]) - ): - _fail("ATTESTATION_STATUS_EXPIRED_OR_FREEZE_RISK") - ca_issued = _dt(ca_signed["issuedAt"], "ATTESTATION_STATUS_CA_ISSUED_INVALID") - ca_expires = _dt(ca_signed["expires"], "ATTESTATION_STATUS_CA_EXPIRES_INVALID") - if issued < ca_issued: - _fail("ATTESTATION_STATUS_PREDATES_CA_SET") - if next_update > ca_expires: - _fail("ATTESTATION_STATUS_OUTLIVES_CA_SET") - inventory, _ = _inventory(continuity_bundle, ca_set=ca_set) - expected_by_sha = {x["sha256"]: x for x in inventory} - raw_entries = s["certificates"] - if not isinstance(raw_entries, list) or len(raw_entries) > int( - POLICY["max_status_entries"] - ): - _fail("ATTESTATION_STATUS_CERTIFICATES_INVALID") - entries = [] - seen = set() - for entry in raw_entries: - if not isinstance(entry, dict) or set(entry) != { - "sha256", - "serialNumber", - "status", - "revokedAt", - "reason", - }: - _fail("ATTESTATION_STATUS_ENTRY_SCHEMA_INVALID") - sha = _hex(entry["sha256"], "ATTESTATION_STATUS_CERT_SHA_INVALID") - if sha in seen: - _fail("ATTESTATION_STATUS_CERT_DUPLICATE") - seen.add(sha) - if ( - sha not in expected_by_sha - or entry["serialNumber"] != expected_by_sha[sha]["serialNumber"] - ): - _fail("ATTESTATION_STATUS_CERT_INVENTORY_MISMATCH") - status = entry["status"] - if status not in {"good", "revoked"}: - _fail("ATTESTATION_STATUS_VALUE_INVALID") - if status == "good": - if entry["revokedAt"] is not None or entry["reason"] is not None: - _fail("ATTESTATION_STATUS_GOOD_FIELDS_INVALID") - else: - revoked = _dt(entry["revokedAt"], "ATTESTATION_STATUS_REVOKED_AT_INVALID") - if revoked > issued + skew: - _fail("ATTESTATION_STATUS_REVOCATION_FROM_FUTURE") - _identity(entry["reason"], "ATTESTATION_STATUS_REASON_INVALID") - entries.append(entry) - entries.sort(key=lambda x: x["sha256"]) - if entries != raw_entries: - _fail("ATTESTATION_STATUS_ENTRIES_NOT_NORMALIZED") - if ( - bool(POLICY["require_all_attestation_certificates_covered"]) - and set(expected_by_sha) != seen - ): - _fail("ATTESTATION_STATUS_COVERAGE_INCOMPLETE") - if previous is not None: - old = {x["sha256"]: x for x in previous["signed"]["certificates"]} - for entry in entries: - prior = old.get(entry["sha256"]) - if prior is None: - continue - if prior["status"] == "revoked" and entry != prior: - _fail("ATTESTATION_STATUS_REVOCATION_NOT_STICKY") - authority = ca_signed["statusAuthority"] - role = authority["role"] - keys = authority["keys"] - selected = s["selectedStatusKeyIds"] - if not isinstance(selected, list): - _fail("ATTESTATION_STATUS_SELECTED_KEYS_INVALID") - selected = [ - _identity(x, "ATTESTATION_STATUS_SELECTED_KEY_INVALID") for x in selected - ] - if ( - sorted(set(selected)) != selected - or len(selected) != role["threshold"] - or not set(selected).issubset(set(role["keyids"])) - ): - _fail("ATTESTATION_STATUS_SELECTED_KEYS_INVALID") - if len({keys[kid]["operator"] for kid in selected}) < int( - POLICY["min_status_operators"] - ): - _fail("ATTESTATION_STATUS_SIGNER_OPERATOR_QUORUM_INVALID") - sigs = doc["signatures"] - if not isinstance(sigs, list): - _fail("ATTESTATION_STATUS_SIGNATURES_INVALID") - sigmap = {} - normalized_sigs = [] - for item in sigs: - if not isinstance(item, dict) or set(item) != {"keyid", "sig"}: - _fail("ATTESTATION_STATUS_SIGNATURE_SCHEMA_INVALID") - kid = _identity(item["keyid"], "ATTESTATION_STATUS_SIGNATURE_KEY_INVALID") - if kid in sigmap or kid not in selected: - _fail("ATTESTATION_STATUS_SIGNATURE_KEY_UNAUTHORIZED") - rootseal._b64( - item["sig"], - "ATTESTATION_STATUS_SIGNATURE_ENCODING_INVALID", - expected_len=64, - ) - sigmap[kid] = item["sig"] - normalized_sigs.append({"keyid": kid, "sig": item["sig"]}) - normalized_sigs.sort(key=lambda x: x["keyid"]) - if normalized_sigs != sigs: - _fail("ATTESTATION_STATUS_SIGNATURES_NOT_NORMALIZED") - if set(sigmap) != set(selected): - _fail("ATTESTATION_STATUS_SIGNATURE_SET_MISMATCH") - valid = [] - for kid in role["keyids"]: - if kid not in sigmap: - continue - if ( - _dt(keys[kid]["expires"], "ATTESTATION_STATUS_SIGNER_EXPIRES_INVALID") - < issued - ): - continue - _ed25519_verify( - keys[kid]["keyval"]["public"], - sigmap[kid], - _canonical(s), - "ATTESTATION_STATUS_SIGNATURE_INVALID", - ) - valid.append(kid) - if len(valid) < role["threshold"]: - _fail("ATTESTATION_STATUS_THRESHOLD_NOT_MET") - if len({keys[kid]["operator"] for kid in valid}) < int( - POLICY["min_status_operators"] - ): - _fail("ATTESTATION_STATUS_SIGNER_OPERATOR_QUORUM_INVALID") - normalized_signed = dict(s) - normalized_signed["issuedAt"] = _ts(issued) - normalized_signed["nextUpdate"] = _ts(next_update) - normalized_signed["certificates"] = entries - if normalized_signed != s: - _fail("ATTESTATION_STATUS_NOT_NORMALIZED") - return {"signatures": normalized_sigs, "signed": normalized_signed} - - -def _evaluate_status( - *, - continuity_bundle: dict[str, Any], - ca_set: dict[str, Any], - status: dict[str, Any], - now: datetime, - historical: bool, -) -> None: - inventory, _ = _inventory(continuity_bundle, ca_set=ca_set) - by_sha = {x["sha256"]: x for x in status["signed"]["certificates"]} - for cert in inventory: - decision = by_sha[cert["sha256"]] - if decision["status"] != "revoked": - continue - revoked = _dt(decision["revokedAt"], "ATTESTATION_STATUS_REVOKED_AT_INVALID") - attested_times = [ - _dt(x, "ATTESTATION_INVENTORY_TIME_INVALID") for x in cert["attestedAt"] - ] - if any(revoked <= t for t in attested_times): - _fail("ATTESTATION_CERT_REVOKED_AT_ATTESTATION_TIME") - if not historical and revoked <= now: - _fail("ATTESTATION_ACTIVE_CERT_REVOKED") - - -def _ca_revoked_union(history: list[dict[str, Any]]) -> list[str]: - result = set() - for envelope in history: - result.update(envelope["signed"]["transition"]["revokedCaSha256"]) - return sorted(result) - - -def _lifecycle_head(previous: str, body: dict[str, Any]) -> str: - return _sha_bytes(_canonical({"previousChainHeadSha256": previous, "event": body})) - - -def _state( - bundle: dict[str, Any], - ca_set: dict[str, Any], - status: dict[str, Any], - chain: str, - device_bindings: dict[str, str], -) -> dict[str, Any]: - return { - "schemaVersion": int(POLICY["state_schema_version"]), - "status": "trusted-attestation-lifecycle", - "governanceId": bundle["governanceId"], - "lifecycleSequence": len(bundle["events"]), - "continuityRootChainHeadSha256": bundle["continuityBinding"][ - "rootChainHeadSha256" - ], - "activeCaSet": { - "version": ca_set["signed"]["version"], - "sha256": _sha_bytes(_canonical(ca_set)), - "expires": ca_set["signed"]["expires"], - }, - "activeStatus": { - "version": status["signed"]["version"], - "sha256": _sha_bytes(_canonical(status)), - "nextUpdate": status["signed"]["nextUpdate"], - }, - "revokedCaSha256": _ca_revoked_union(bundle["caSets"]), - "deviceBindings": device_bindings, - "lifecycleChainHeadSha256": chain, - "bundleSha256": _sha_bytes(_canonical(bundle)), - } - - -def _write_output( - target: Path, - *, - bundle: dict[str, Any], - state: dict[str, Any], - ca_set: dict[str, Any], - status: dict[str, Any], - input_files: list[Path], - before: dict[str, str], - phase: str, -) -> None: - target.parent.mkdir(parents=True, exist_ok=True) - with tempfile.TemporaryDirectory( - prefix="run158-lifecycle-", dir=target.parent - ) as td: - stage = Path(td) / "lifecycle" - stage.mkdir() - _write(stage / "release-attestation-lifecycle-bundle.json", bundle) - _write(stage / _DOC_ATTESTATION_LIFECYCLE_STATE, state) - _write(stage / "active-attestation-ca-set.json", ca_set) - _write(stage / "active-attestation-status.json", status) - receipt = { - "schemaVersion": int(POLICY["receipt_schema_version"]), - "status": phase, - "governanceId": bundle["governanceId"], - "bundle": _artifact_raw( - "release-attestation-lifecycle-bundle.json", _canonical(bundle) - ), - "state": _artifact_raw(_DOC_ATTESTATION_LIFECYCLE_STATE, _canonical(state)), - "caSet": _artifact_raw( - "active-attestation-ca-set.json", _canonical(ca_set) - ), - "attestationStatus": _artifact_raw( - "active-attestation-status.json", _canonical(status) - ), - } - _write(stage / "release-attestation-lifecycle-receipt.json", receipt) - for p in input_files: - if _sha(p) != before[str(p.resolve())]: - _fail("ATTESTATION_LIFECYCLE_INPUT_CHANGED") - temp = target.with_name(target.name + ".tmp-" + os.urandom(6).hex()) - shutil.copytree(stage, temp, copy_function=shutil.copy2) - os.replace(temp, target) - - -def initialize_lifecycle( # ruff: ignore[undocumented-public-function] - *, - continuity_dir: Path, - expected_bootstrap_root_sha256: str, - expected_recovery_root_sha256: str, - expected_attestation_root_sha256: list[str], - ca_set_path: Path, - status_snapshot_path: Path, - output_dir: Path, - now: datetime | None = None, -) -> dict[str, Any]: - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - cont = _regular_dir(continuity_dir, "ATTESTATION_CONTINUITY_DIR_INVALID") - try: - continuity.verify_continuity( - output_dir=cont, - expected_bootstrap_root_sha256=expected_bootstrap_root_sha256, - expected_recovery_root_sha256=expected_recovery_root_sha256, - expected_attestation_root_sha256=expected_attestation_root_sha256, - now=current, - ) - except continuity.RootContinuityError as exc: - raise AttestationLifecycleError( - "ATTESTATION_PREDECESSOR_INVALID:" + str(exc) - ) from exc - docs = _continuity_docs(cont) - ca_doc, _ = _read(ca_set_path, "ATTESTATION_CA_SET") - ca_set = _verify_ca_set( - ca_doc, continuity_docs=docs, previous=None, now=current, historical=False - ) - status_doc, _ = _read(status_snapshot_path, "ATTESTATION_STATUS") - status = _verify_status_snapshot( - status_doc, - ca_set=ca_set, - continuity_bundle=docs["release-root-continuity-bundle.json"], - previous=None, - now=current, - historical=False, - ) - _evaluate_status( - continuity_bundle=docs["release-root-continuity-bundle.json"], - ca_set=ca_set, - status=status, - now=current, - historical=False, - ) - inventory, bindings = _inventory( - docs["release-root-continuity-bundle.json"], ca_set=ca_set - ) - ca_sha = _sha_bytes(_canonical(ca_set)) - status_sha = _sha_bytes(_canonical(status)) - body = { - "sequence": 1, - "type": "lifecycle-bootstrap", - "previousChainHeadSha256": docs[_DOC_ROOT_CONTINUITY_STATE][ - "rootChainHeadSha256" - ], - "caSetSha256": ca_sha, - "statusSha256": status_sha, - "inventorySha256": _sha_bytes(_canonical({"certificates": inventory})), - "deviceBindingsSha256": _sha_bytes(_canonical(bindings)), - } - chain = _lifecycle_head(body["previousChainHeadSha256"], body) - event = dict(body, lifecycleChainHeadSha256=chain) - binding = _continuity_binding(docs) - bundle = { - "schemaVersion": int(POLICY["bundle_schema_version"]), - "predicateType": PREDICATE_TYPE, - "status": "trusted-attestation-lifecycle-chain", - "governanceId": ca_set["signed"]["governanceId"], - "bootstrapRootSha256": _hex( - expected_bootstrap_root_sha256, "ATTESTATION_BOOTSTRAP_PIN_INVALID" - ), - "recoveryRootSha256": _hex( - expected_recovery_root_sha256, "ATTESTATION_RECOVERY_PIN_INVALID" - ), - "attestationRootSha256": sorted( - _hex(x, "ATTESTATION_ROOT_PIN_INVALID") - for x in expected_attestation_root_sha256 - ), - "continuityOutput": docs, - "continuityBinding": binding, - "caSets": [ca_set], - "statusSnapshots": [status], - "events": [event], - } - state = _state(bundle, ca_set, status, chain, bindings) - target = _outside( - output_dir, - [cont, ca_set_path, status_snapshot_path], - "ATTESTATION_OUTPUT_INSIDE_INPUT", - ) - if target.exists() or target.is_symlink(): - _fail("ATTESTATION_OUTPUT_ALREADY_EXISTS") - inputs = [p for p in cont.rglob("*") if p.is_file()] + [ - ca_set_path, - status_snapshot_path, - ] - before = {str(p.resolve()): _sha(p) for p in inputs} - _write_output( - target, - bundle=bundle, - state=state, - ca_set=ca_set, - status=status, - input_files=inputs, - before=before, - phase="attestation-lifecycle-initialized", - ) - verify_lifecycle( - output_dir=target, - expected_bootstrap_root_sha256=expected_bootstrap_root_sha256, - expected_recovery_root_sha256=expected_recovery_root_sha256, - expected_attestation_root_sha256=expected_attestation_root_sha256, - now=current, - ) - return { - "ok": True, - "phase": "attestation-lifecycle-initialized", - "ca_set_version": 1, - "status_version": 1, - "lifecycle_chain_head_sha256": chain, - } - - -def _verify_bundle( # ruff: ignore[too-many-branches] - bundle: dict[str, Any], - *, - bootstrap_pin: str, - recovery_pin: str, - attestation_pins: list[str], - now: datetime, - historical: bool, -) -> tuple[dict[str, Any], dict[str, Any], str, dict[str, str]]: - expected = { - "schemaVersion", - "predicateType", - "status", - "governanceId", - "bootstrapRootSha256", - "recoveryRootSha256", - "attestationRootSha256", - "continuityOutput", - "continuityBinding", - "caSets", - "statusSnapshots", - "events", - } - if ( - set(bundle) != expected - or bundle["schemaVersion"] != int(POLICY["bundle_schema_version"]) - or bundle["predicateType"] != PREDICATE_TYPE - or bundle["status"] != "trusted-attestation-lifecycle-chain" - ): - _fail("ATTESTATION_BUNDLE_SCHEMA_INVALID") - if ( - bundle["bootstrapRootSha256"] - != _hex(bootstrap_pin, "ATTESTATION_BOOTSTRAP_PIN_INVALID") - or bundle["recoveryRootSha256"] - != _hex(recovery_pin, "ATTESTATION_RECOVERY_PIN_INVALID") - or bundle["attestationRootSha256"] != sorted(attestation_pins) - ): - _fail("ATTESTATION_BUNDLE_PIN_MISMATCH") - docs = bundle["continuityOutput"] - if not isinstance(docs, dict): - _fail("ATTESTATION_BUNDLE_CONTINUITY_INVALID") - with tempfile.TemporaryDirectory(prefix="run158-continuity-") as td: - d = Path(td) / "continuity" - _write_docs(d, docs) - try: - continuity.verify_continuity( - output_dir=d, - expected_bootstrap_root_sha256=bootstrap_pin, - expected_recovery_root_sha256=recovery_pin, - expected_attestation_root_sha256=attestation_pins, - now=now, - historical=historical, - ) - except continuity.RootContinuityError as exc: - raise AttestationLifecycleError( - "ATTESTATION_EMBEDDED_CONTINUITY_INVALID:" + str(exc) - ) from exc - if bundle["continuityBinding"] != _continuity_binding(docs): - _fail("ATTESTATION_BUNDLE_CONTINUITY_BINDING_INVALID") - ca_sets = bundle["caSets"] - statuses = bundle["statusSnapshots"] - events = bundle["events"] - if ( - not isinstance(ca_sets, list) - or not isinstance(statuses, list) - or not isinstance(events, list) - or not ca_sets - or not statuses - or len(statuses) != len(events) - or len(events) > int(POLICY["max_lifecycle_events"]) - ): - _fail("ATTESTATION_BUNDLE_HISTORY_INVALID") - prev_ca = None - verified_ca = [] - revoked = set() - current_ca = None - for raw in ca_sets: - ca = _verify_ca_set( - raw, continuity_docs=docs, previous=prev_ca, now=now, historical=True - ) - verified_ca.append(ca) - prev_ca = ca - current_ca = ca - revoked.update(ca["signed"]["transition"]["revokedCaSha256"]) - active_pins = {x["sha256"] for x in ca["signed"]["trustRoots"]} - if active_pins & revoked: - _fail("ATTESTATION_REVOKED_CA_REINTRODUCED") - if verified_ca != ca_sets: - _fail("ATTESTATION_CA_SET_HISTORY_NOT_NORMALIZED") - ca_by_sha = {_sha_bytes(_canonical(x)): x for x in ca_sets} - referenced_ca = [raw.get("signed", {}).get("caSetSha256") for raw in statuses] - if any(sha not in referenced_ca for sha in ca_by_sha) or referenced_ca[ - -1 - ] != _sha_bytes(_canonical(ca_sets[-1])): - _fail("ATTESTATION_CA_SET_HISTORY_UNREFERENCED") - prev_status = None - chain = docs[_DOC_ROOT_CONTINUITY_STATE]["rootChainHeadSha256"] - bindings_final = {} - for idx, (raw, event) in enumerate(zip(statuses, events), start=1): - ca = ca_by_sha.get(raw.get("signed", {}).get("caSetSha256")) - if ca is None: - _fail("ATTESTATION_STATUS_REFERENCES_UNKNOWN_CA_SET") - status = _verify_status_snapshot( - raw, - ca_set=ca, - continuity_bundle=docs["release-root-continuity-bundle.json"], - previous=prev_status, - now=now, - historical=True, - ) - prev_status = status - _evaluate_status( - continuity_bundle=docs["release-root-continuity-bundle.json"], - ca_set=ca, - status=status, - now=now, - historical=True, - ) - inventory, bindings = _inventory( - docs["release-root-continuity-bundle.json"], ca_set=ca - ) - bindings_final = bindings - body = { - "sequence": idx, - "type": "lifecycle-bootstrap" if idx == 1 else "lifecycle-advance", - "previousChainHeadSha256": chain, - "caSetSha256": _sha_bytes(_canonical(ca)), - "statusSha256": _sha_bytes(_canonical(status)), - "inventorySha256": _sha_bytes(_canonical({"certificates": inventory})), - "deviceBindingsSha256": _sha_bytes(_canonical(bindings)), - } - next_chain = _lifecycle_head(chain, body) - expected_event = dict(body, lifecycleChainHeadSha256=next_chain) - if event != expected_event: - _fail("ATTESTATION_LIFECYCLE_EVENT_REBIND_FAILED") - chain = next_chain - if not historical: - # Live policy applies only to the current CA/status; historical entries replay at their signing times. - current_ca = _verify_ca_set( - ca_sets[-1], - continuity_docs=docs, - previous=(ca_sets[-2] if len(ca_sets) > 1 else None), - now=now, - historical=False, - ) - current_status = _verify_status_snapshot( - statuses[-1], - ca_set=current_ca, - continuity_bundle=docs["release-root-continuity-bundle.json"], - previous=(statuses[-2] if len(statuses) > 1 else None), - now=now, - historical=False, - ) - _evaluate_status( - continuity_bundle=docs["release-root-continuity-bundle.json"], - ca_set=current_ca, - status=current_status, - now=now, - historical=False, - ) - return ca_sets[-1], statuses[-1], chain, bindings_final - - -def verify_lifecycle( # ruff: ignore[undocumented-public-function] - *, - output_dir: Path, - expected_bootstrap_root_sha256: str, - expected_recovery_root_sha256: str, - expected_attestation_root_sha256: list[str], - now: datetime | None = None, - historical: bool = False, -) -> dict[str, Any]: - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - root = _regular_dir(output_dir, "ATTESTATION_VERIFY_DIR_INVALID") - names = { - "release-attestation-lifecycle-bundle.json", - _DOC_ATTESTATION_LIFECYCLE_STATE, - "active-attestation-ca-set.json", - "active-attestation-status.json", - "release-attestation-lifecycle-receipt.json", - } - docs = _docs_from_dir(root, names, "ATTESTATION_VERIFY") - bundle = docs["release-attestation-lifecycle-bundle.json"] - state = docs[_DOC_ATTESTATION_LIFECYCLE_STATE] - ca, status, chain, bindings = _verify_bundle( - bundle, - bootstrap_pin=expected_bootstrap_root_sha256, - recovery_pin=expected_recovery_root_sha256, - attestation_pins=expected_attestation_root_sha256, - now=current, - historical=historical, - ) - expected_state = _state(bundle, ca, status, chain, bindings) - if state != expected_state: - _fail("ATTESTATION_STATE_REBIND_FAILED") - if ( - docs["active-attestation-ca-set.json"] != ca - or docs["active-attestation-status.json"] != status - ): - _fail("ATTESTATION_ACTIVE_ARTIFACT_REBIND_FAILED") - receipt = docs["release-attestation-lifecycle-receipt.json"] - expected_receipt = { - "schemaVersion": int(POLICY["receipt_schema_version"]), - "status": receipt.get("status"), - "governanceId": bundle["governanceId"], - "bundle": _artifact_raw( - "release-attestation-lifecycle-bundle.json", _canonical(bundle) - ), - "state": _artifact_raw(_DOC_ATTESTATION_LIFECYCLE_STATE, _canonical(state)), - "caSet": _artifact_raw("active-attestation-ca-set.json", _canonical(ca)), - "attestationStatus": _artifact_raw( - "active-attestation-status.json", _canonical(status) - ), - } - if receipt not in [ - dict(expected_receipt, status="attestation-lifecycle-initialized"), - dict(expected_receipt, status="attestation-lifecycle-advanced"), - ]: - _fail("ATTESTATION_RECEIPT_REBIND_FAILED") - return { - "ok": True, - "phase": "attestation-lifecycle-verified", - "ca_set_version": ca["signed"]["version"], - "status_version": status["signed"]["version"], - "lifecycle_chain_head_sha256": chain, - } - - -def advance_lifecycle( # ruff: ignore[undocumented-public-function] - *, - previous_dir: Path, - status_snapshot_path: Path, - output_dir: Path, - expected_bootstrap_root_sha256: str, - expected_recovery_root_sha256: str, - expected_attestation_root_sha256: list[str], - next_ca_set_path: Path | None = None, - now: datetime | None = None, -) -> dict[str, Any]: - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - prev = _regular_dir(previous_dir, "ATTESTATION_PREVIOUS_DIR_INVALID") - verify_lifecycle( - output_dir=prev, - expected_bootstrap_root_sha256=expected_bootstrap_root_sha256, - expected_recovery_root_sha256=expected_recovery_root_sha256, - expected_attestation_root_sha256=expected_attestation_root_sha256, - now=current, - ) - docs = _docs_from_dir( - prev, - { - "release-attestation-lifecycle-bundle.json", - _DOC_ATTESTATION_LIFECYCLE_STATE, - "active-attestation-ca-set.json", - "active-attestation-status.json", - "release-attestation-lifecycle-receipt.json", - }, - "ATTESTATION_PREVIOUS", - ) - bundle = json.loads(json.dumps(docs["release-attestation-lifecycle-bundle.json"])) - continuity_docs = bundle["continuityOutput"] - previous_ca = docs["active-attestation-ca-set.json"] - if next_ca_set_path is not None: - raw, _ = _read(next_ca_set_path, "ATTESTATION_NEXT_CA_SET") - ca = _verify_ca_set( - raw, - continuity_docs=continuity_docs, - previous=previous_ca, - now=current, - historical=False, - ) - revoked_before = set(_ca_revoked_union(bundle["caSets"])) - new_pins = {x["sha256"] for x in ca["signed"]["trustRoots"]} - if revoked_before & new_pins: - _fail("ATTESTATION_REVOKED_CA_REINTRODUCED") - bundle["caSets"].append(ca) - else: - ca = previous_ca - status_raw, _ = _read(status_snapshot_path, "ATTESTATION_NEXT_STATUS") - status = _verify_status_snapshot( - status_raw, - ca_set=ca, - continuity_bundle=continuity_docs["release-root-continuity-bundle.json"], - previous=docs["active-attestation-status.json"], - now=current, - historical=False, - ) - _evaluate_status( - continuity_bundle=continuity_docs["release-root-continuity-bundle.json"], - ca_set=ca, - status=status, - now=current, - historical=False, - ) - inventory, bindings = _inventory( - continuity_docs["release-root-continuity-bundle.json"], ca_set=ca - ) - previous_bindings = docs[_DOC_ATTESTATION_LIFECYCLE_STATE]["deviceBindings"] - for kid, value in previous_bindings.items(): - if kid in bindings and bindings[kid] != value: - _fail("ATTESTATION_DEVICE_IDENTITY_CHANGED") - bundle["statusSnapshots"].append(status) - seq = len(bundle["events"]) + 1 - previous_head = docs[_DOC_ATTESTATION_LIFECYCLE_STATE]["lifecycleChainHeadSha256"] - body = { - "sequence": seq, - "type": "lifecycle-advance", - "previousChainHeadSha256": previous_head, - "caSetSha256": _sha_bytes(_canonical(ca)), - "statusSha256": _sha_bytes(_canonical(status)), - "inventorySha256": _sha_bytes(_canonical({"certificates": inventory})), - "deviceBindingsSha256": _sha_bytes(_canonical(bindings)), - } - chain = _lifecycle_head(previous_head, body) - bundle["events"].append(dict(body, lifecycleChainHeadSha256=chain)) - state = _state(bundle, ca, status, chain, bindings) - protected = [prev, status_snapshot_path] + ( - [next_ca_set_path] if next_ca_set_path else [] - ) - target = _outside( - output_dir, - [p for p in protected if p is not None], - "ATTESTATION_ADVANCE_OUTPUT_INSIDE_INPUT", - ) - if target.exists() or target.is_symlink(): - _fail("ATTESTATION_ADVANCE_OUTPUT_ALREADY_EXISTS") - inputs = ( - [p for p in prev.rglob("*") if p.is_file()] - + [status_snapshot_path] - + ([next_ca_set_path] if next_ca_set_path else []) - ) - before = {str(p.resolve()): _sha(p) for p in inputs} - _write_output( - target, - bundle=bundle, - state=state, - ca_set=ca, - status=status, - input_files=inputs, - before=before, - phase="attestation-lifecycle-advanced", - ) - final = verify_lifecycle( - output_dir=target, - expected_bootstrap_root_sha256=expected_bootstrap_root_sha256, - expected_recovery_root_sha256=expected_recovery_root_sha256, - expected_attestation_root_sha256=expected_attestation_root_sha256, - now=current, - ) - return { - "ok": True, - "phase": "attestation-lifecycle-advanced", - "ca_set_version": ca["signed"]["version"], - "status_version": status["signed"]["version"], - "lifecycle_chain_head_sha256": final["lifecycle_chain_head_sha256"], - } - - -def _paths(values: list[str]) -> list[Path]: - return [Path(x) for x in values] - - -def main( # ruff: ignore[undocumented-public-function] - argv: list[str] | None = None, -) -> int: - parser = argparse.ArgumentParser( - description="Verify release attestation lifecycle and certificate status" - ) - sub = parser.add_subparsers(dest="cmd", required=True) - - def pins(p): - p.add_argument("--bootstrap-root-sha256", required=True) - p.add_argument("--recovery-root-sha256", required=True) - p.add_argument("--attestation-root-sha256", action="append", required=True) - - p = sub.add_parser("initialize") - pins(p) - p.add_argument("--continuity-dir", type=Path, required=True) - p.add_argument("--ca-set", type=Path, required=True) - p.add_argument("--status", type=Path, required=True) - p.add_argument("--output-dir", type=Path, required=True) - p = sub.add_parser("advance") - pins(p) - p.add_argument("--previous-dir", type=Path, required=True) - p.add_argument("--status", type=Path, required=True) - p.add_argument("--next-ca-set", type=Path) - p.add_argument("--output-dir", type=Path, required=True) - p = sub.add_parser("verify") - pins(p) - p.add_argument("--output-dir", type=Path, required=True) - p.add_argument("--historical", action="store_true") - args = parser.parse_args(argv) - if args.cmd == "initialize": - result = initialize_lifecycle( - continuity_dir=args.continuity_dir, - expected_bootstrap_root_sha256=args.bootstrap_root_sha256, - expected_recovery_root_sha256=args.recovery_root_sha256, - expected_attestation_root_sha256=args.attestation_root_sha256, - ca_set_path=args.ca_set, - status_snapshot_path=args.status, - output_dir=args.output_dir, - ) - elif args.cmd == "advance": - result = advance_lifecycle( - previous_dir=args.previous_dir, - status_snapshot_path=args.status, - output_dir=args.output_dir, - expected_bootstrap_root_sha256=args.bootstrap_root_sha256, - expected_recovery_root_sha256=args.recovery_root_sha256, - expected_attestation_root_sha256=args.attestation_root_sha256, - next_ca_set_path=args.next_ca_set, - ) - else: - result = verify_lifecycle( - output_dir=args.output_dir, - expected_bootstrap_root_sha256=args.bootstrap_root_sha256, - expected_recovery_root_sha256=args.recovery_root_sha256, - expected_attestation_root_sha256=args.attestation_root_sha256, - historical=args.historical, - ) - logger.info("%s", json.dumps(result, sort_keys=True)) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/security/verify_native_status_provenance.py b/security/verify_native_status_provenance.py deleted file mode 100644 index 6f9899b9e328371659c43b4c54b8fe2f6ec74081..0000000000000000000000000000000000000000 --- a/security/verify_native_status_provenance.py +++ /dev/null @@ -1,1604 +0,0 @@ -""" -Run 159: independently verify and preserve native certificate-status evidence. - -The Run 158 threshold-signed status snapshot is a policy assertion. This layer -recomputes the status of every attestation leaf from raw DER CRLs and raw DER OCSP -responses, preserves those exact bytes, rejects *any* source disagreement, and -requires the resulting decision to match Run 158 exactly. - -Optional vendor-native evidence is accepted only through an explicitly configured -profile verifier adapter. There are intentionally no permissive built-in vendor -adapters: an unconfigured profile fails closed. -""" - -from __future__ import annotations - -import argparse -import base64 -import contextlib -import hashlib -import json -import logging -import os -import re -import shutil -import subprocess -import sys -import tempfile -import threading -from datetime import datetime, timedelta, timezone -from pathlib import Path -from typing import Any, Iterable - -import tomllib - -try: - from . import verify_attestation_lifecycle as lifecycle -except (ImportError, ValueError) as exc: - import importlib.util - - _here = Path(__file__).resolve().parent - spec = importlib.util.spec_from_file_location( - "_run158_lifecycle_for_native_status", _here / "verify_attestation_lifecycle.py" - ) - if spec is None or spec.loader is None: - raise ImportError("verify_attestation_lifecycle.py") from exc - lifecycle = importlib.util.module_from_spec(spec) - sys.modules[spec.name] = lifecycle - spec.loader.exec_module(lifecycle) - -HERE = Path(__file__).resolve().parent -POLICY = tomllib.loads((HERE / "release_native_status_policy.toml").read_text()) -PREDICATE_TYPE = str(POLICY["predicate_type"]) -_ZERO_HASH = "0" * 64 -_HEX64 = re.compile(r"^[0-9a-f]{64}$") -_ID = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:/@+-]{0,511}$") -_CHUNK = 1024 * 1024 -logger = logging.getLogger(__name__) - - -class NativeStatusError(RuntimeError): - """Native status provenance invariant failed.""" - - -def _fail(code: str) -> None: - raise NativeStatusError(code) - - -def _canonical(value: Any) -> bytes: - return ( - json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False) - + "\n" - ).encode("utf-8") - - -def _sha_bytes(raw: bytes) -> str: - return hashlib.sha256(raw).hexdigest() - - -def _sha(path: Path) -> str: - h = hashlib.sha256() - with path.open("rb") as handle: - for chunk in iter(lambda: handle.read(_CHUNK), b""): - h.update(chunk) - return h.hexdigest() - - -def _write(path: Path, value: dict[str, Any]) -> None: - path.write_bytes(_canonical(value)) - - -def _loads(raw: bytes, code: str) -> dict[str, Any]: - def hook(pairs): - out: dict[str, Any] = {} - for key, value in pairs: - if key in out: - _fail(code + "_DUPLICATE_KEY") - out[key] = value - return out - - try: - value = json.loads(raw.decode("utf-8"), object_pairs_hook=hook) - except NativeStatusError: - raise - except Exception as exc: - raise NativeStatusError(code + "_JSON_INVALID") from exc - if not isinstance(value, dict): - _fail(code + "_SCHEMA_INVALID") - return value - - -def _read_json(path: Path, code: str) -> tuple[dict[str, Any], bytes]: - if path.is_symlink() or not path.is_file(): - _fail(code + "_INVALID") - if path.stat().st_size > int(POLICY["max_json_bytes"]): - _fail(code + "_TOO_LARGE") - raw = path.read_bytes() - doc = _loads(raw, code) - if raw != _canonical(doc): - _fail(code + "_NOT_CANONICAL") - return doc, raw - - -def _raw(path: Path, code: str) -> bytes: - if path.is_symlink() or not path.is_file(): - _fail(code + "_INVALID") - size = path.stat().st_size - if size <= 0 or size > int(POLICY["max_raw_evidence_bytes"]): - _fail(code + "_SIZE_INVALID") - return path.read_bytes() - - -def _identity(value: Any, code: str) -> str: - if not isinstance(value, str): - _fail(code) - value = value.strip() - if ( - not value - or len(value) > 512 # ruff: ignore[magic-value-comparison] - or ".." in value - or "?" in value - or "#" in value - or "\x00" in value - ): - _fail(code) - if ( - any( - ord(c) < 32 # ruff: ignore[magic-value-comparison] - or ord(c) == 127 # ruff: ignore[magic-value-comparison] - for c in value - ) - or _ID.fullmatch(value) is None - ): - _fail(code) - return value - - -def _hex(value: Any, code: str) -> str: - if not isinstance(value, str) or _HEX64.fullmatch(value) is None: - _fail(code) - return value - - -def _dt(value: Any, code: str) -> datetime: - if not isinstance(value, str) or not value.endswith("Z"): - _fail(code) - try: - return datetime.fromisoformat(value[:-1] + "+00:00").astimezone(timezone.utc) - except ValueError as exc: - raise NativeStatusError(code) from exc - - -def _ts(value: datetime) -> str: - return ( - value.astimezone(timezone.utc) - .isoformat(timespec="seconds") - .replace("+00:00", "Z") - ) - - -def _cert_time(value: Any) -> datetime: - if value.tzinfo is None: - return value.replace(tzinfo=timezone.utc) - return value.astimezone(timezone.utc) - - -def _regular_dir(path: Path, code: str) -> Path: - if path.is_symlink() or not path.is_dir(): - _fail(code) - return path.resolve() - - -def _outside(path: Path, protected: Iterable[Path], code: str) -> Path: - target = path.expanduser().resolve() - for item in protected: - root = item.resolve() - if target == root or root in target.parents: - _fail(code) - return target - - -def _artifact(name: str, raw: bytes) -> dict[str, Any]: - return {"name": name, "sha256": _sha_bytes(raw), "size": len(raw)} - - -# ── Verified-state document names ──────────────────────────────────────── -# -# One constant per release-evidence document. The values are the on-disk -# artifact names and are unchanged; only the repetition is removed. Reading -# a document by a name that says which document it is also keeps static -# analysis from reading the filename token 'trusted' as 'confidential': -# these files carry published Merkle roots and SHA-256 digests, which must -# stay in clear text for any third party to verify them. -_DOC_ATTESTATION_LIFECYCLE_STATE = "trusted-attestation-lifecycle-state.json" -_DOC_NATIVE_STATUS_STATE = "trusted-native-status-state.json" - - -def _lifecycle_docs(root: Path) -> dict[str, dict[str, Any]]: - names = { - "release-attestation-lifecycle-bundle.json", - _DOC_ATTESTATION_LIFECYCLE_STATE, - "active-attestation-ca-set.json", - "active-attestation-status.json", - "release-attestation-lifecycle-receipt.json", - } - if {p.name for p in root.iterdir()} != names: - _fail("NATIVE_PREDECESSOR_ALLOWLIST_MISMATCH") - return { - name: _read_json( - root / name, - "NATIVE_PREDECESSOR_" + name.upper().replace("-", "_").replace(".", "_"), - )[0] - for name in sorted(names) - } - - -def _verify_predecessor( - root: Path, - *, - bootstrap_pin: str, - recovery_pin: str, - attestation_pins: list[str], - now: datetime, - historical: bool, -) -> dict[str, dict[str, Any]]: - try: - lifecycle.verify_lifecycle( - output_dir=root, - expected_bootstrap_root_sha256=bootstrap_pin, - expected_recovery_root_sha256=recovery_pin, - expected_attestation_root_sha256=attestation_pins, - now=now, - historical=historical, - ) - except lifecycle.AttestationLifecycleError as exc: - raise NativeStatusError("NATIVE_PREDECESSOR_INVALID:" + str(exc)) from exc - return _lifecycle_docs(root) - - -def _verify_signature( - public_key: Any, signature: bytes, data: bytes, hash_algorithm: Any, code: str -) -> None: - try: - from cryptography.hazmat.primitives.asymmetric import ( # ruff: ignore[import-outside-top-level] - ec, - ed448, - ed25519, - padding, - rsa, - ) - - if isinstance(public_key, (ed25519.Ed25519PublicKey, ed448.Ed448PublicKey)): - public_key.verify(signature, data) - elif isinstance(public_key, rsa.RSAPublicKey): - public_key.verify(signature, data, padding.PKCS1v15(), hash_algorithm) - elif isinstance(public_key, ec.EllipticCurvePublicKey): - public_key.verify(signature, data, ec.ECDSA(hash_algorithm)) - else: - _fail(code + "_UNSUPPORTED_KEY") - except NativeStatusError: - raise - except Exception as exc: - raise NativeStatusError(code + "_SIGNATURE_INVALID") from exc - - -def _ca_records(ca_set: dict[str, Any]) -> list[dict[str, Any]]: - try: - from cryptography import x509 # ruff: ignore[import-outside-top-level] - except Exception as exc: - raise NativeStatusError("NATIVE_CRYPTOGRAPHY_UNAVAILABLE") from exc - out = [] - for item in ca_set["signed"]["trustRoots"]: - raw = base64.b64decode(item["der"], validate=True) - cert = x509.load_der_x509_certificate(raw) - out.append( - { - "sha256": item["sha256"], - "der": raw, - "cert": cert, - "profile": item["profile"], - } - ) - return out - - -def _leaf_inventory( - bundle: dict[str, Any], ca_set: dict[str, Any] -) -> list[dict[str, Any]]: - try: - from cryptography import x509 # ruff: ignore[import-outside-top-level] - except Exception as exc: - raise NativeStatusError("NATIVE_CRYPTOGRAPHY_UNAVAILABLE") from exc - continuity_bundle = bundle["continuityOutput"][ - "release-root-continuity-bundle.json" - ] - ca_records = _ca_records(ca_set) - run158_inventory, _ = lifecycle._inventory(continuity_bundle, ca_set=ca_set) - run158_by_sha = {x["sha256"]: x for x in run158_inventory} - leaves: dict[str, dict[str, Any]] = {} - for att in lifecycle._attestations_from_continuity(continuity_bundle): - s = att["signed"] - chain = s.get("certificateChainDer") - if not isinstance(chain, list) or not chain: - _fail("NATIVE_ATTESTATION_CHAIN_INVALID") - raw = base64.b64decode(chain[0], validate=True) - cert = x509.load_der_x509_certificate(raw) - sha = _sha_bytes(raw) - if sha not in run158_by_sha: - _fail("NATIVE_LEAF_NOT_IN_RUN158_INVENTORY") - matches = [ca for ca in ca_records if cert.issuer == ca["cert"].subject] - valid = [] - for ca in matches: - try: - lifecycle.continuity._verify_cert_signature( - cert, ca["cert"].public_key(), "NATIVE_LEAF_CA" - ) - valid.append(ca) - except Exception: # ruff: ignore[blind-except, try-except-in-loop] - pass - if len(valid) != 1: - _fail("NATIVE_LEAF_ISSUER_AMBIGUOUS") - leaves[sha] = { - "sha256": sha, - "der": raw, - "cert": cert, - "serialNumber": format(cert.serial_number, "x"), - "keyId": s["keyId"], - "attestedAt": s["attestedAt"], - "issuerCaSha256": valid[0]["sha256"], - "issuerCert": valid[0]["cert"], - "issuerDer": valid[0]["der"], - } - return [leaves[k] for k in sorted(leaves)] - - -def _reason_name(reason: Any) -> str | None: - if reason is None: - return None - name = getattr(reason, "name", str(reason)).replace("_", "-") - return name # ruff: ignore[unnecessary-assign] - - -def _run158_decisions(status: dict[str, Any]) -> dict[str, dict[str, Any]]: - out = {} - for item in status["signed"]["certificates"]: - out[item["sha256"]] = { - "status": item["status"], - "revokedAt": item["revokedAt"], - "reason": item["reason"], - } - return out - - -def _verify_crl( # ruff: ignore[too-many-branches] - raw: bytes, - *, - leaves: list[dict[str, Any]], - ca_records: list[dict[str, Any]], - now: datetime, - historical: bool, -) -> dict[str, Any]: - try: - from cryptography import x509 # ruff: ignore[import-outside-top-level] - - crl = x509.load_der_x509_crl(raw) - except Exception as exc: - raise NativeStatusError("NATIVE_CRL_DER_INVALID") from exc - matches = [ca for ca in ca_records if crl.issuer == ca["cert"].subject] - if len(matches) != 1: - _fail("NATIVE_CRL_ISSUER_INVALID") - ca = matches[0] - try: - crl.extensions.get_extension_for_class(x509.DeltaCRLIndicator) - except x509.ExtensionNotFound: - pass - else: - _fail("NATIVE_CRL_DELTA_UNSUPPORTED") - try: - crl.extensions.get_extension_for_class(x509.IssuingDistributionPoint) - except x509.ExtensionNotFound: - pass - else: - # A scoped/indirect CRL cannot prove that absence means good for every leaf. - _fail("NATIVE_CRL_SCOPED_OR_INDIRECT_UNSUPPORTED") - _verify_signature( - ca["cert"].public_key(), - crl.signature, - crl.tbs_certlist_bytes, - crl.signature_hash_algorithm, - "NATIVE_CRL", - ) - this_update = _cert_time(crl.last_update_utc) - next_update_raw = crl.next_update_utc - if next_update_raw is None: - _fail("NATIVE_CRL_NEXT_UPDATE_MISSING") - next_update = _cert_time(next_update_raw) - skew = timedelta(minutes=int(POLICY["max_clock_skew_minutes"])) - if this_update > now + skew: - _fail("NATIVE_CRL_FROM_FUTURE") - if next_update <= this_update or next_update - this_update > timedelta( - hours=int(POLICY["max_crl_lifetime_hours"]) - ): - _fail("NATIVE_CRL_LIFETIME_INVALID") - ca_nb = lifecycle.continuity._cert_time(ca["cert"], "not_valid_before") - ca_na = lifecycle.continuity._cert_time(ca["cert"], "not_valid_after") - if this_update < ca_nb or next_update > ca_na: - _fail("NATIVE_CRL_OUTSIDE_ISSUER_VALIDITY") - if not historical and next_update < now + timedelta( - minutes=int(POLICY["min_native_remaining_minutes"]) - ): - _fail("NATIVE_CRL_EXPIRED_OR_FREEZE_RISK") - try: - number = crl.extensions.get_extension_for_class(x509.CRLNumber).value.crl_number - except x509.ExtensionNotFound: - if bool(POLICY["require_crl_number"]): - _fail("NATIVE_CRL_NUMBER_MISSING") - number = 0 - if not isinstance(number, int) or number < 0: - _fail("NATIVE_CRL_NUMBER_INVALID") - revoked_entries = list(crl) - if len(revoked_entries) > int(POLICY["max_crl_entries"]): - _fail("NATIVE_CRL_ENTRY_COUNT_INVALID") - serials = [entry.serial_number for entry in revoked_entries] - if len(set(serials)) != len(serials): - _fail("NATIVE_CRL_DUPLICATE_SERIAL") - revoked_by_serial = {entry.serial_number: entry for entry in revoked_entries} - decisions = [] - for leaf in leaves: - if leaf["issuerCaSha256"] != ca["sha256"]: - continue - entry = revoked_by_serial.get(leaf["cert"].serial_number) - if entry is None: - decision = { - "certSha256": leaf["sha256"], - "serialNumber": leaf["serialNumber"], - "status": "good", - "revokedAt": None, - "reason": None, - } - else: - when = _cert_time(entry.revocation_date_utc) - if when > this_update + skew: - _fail("NATIVE_CRL_REVOCATION_FROM_FUTURE") - reason = None - try: - reason = _reason_name( - entry.extensions.get_extension_for_class( - x509.CRLReason - ).value.reason - ) - except x509.ExtensionNotFound: - reason = "unspecified" - if reason == "remove-from-crl": - _fail("NATIVE_CRL_REMOVE_FROM_CRL_INVALID_IN_FULL_CRL") - decision = { - "certSha256": leaf["sha256"], - "serialNumber": leaf["serialNumber"], - "status": "revoked", - "revokedAt": _ts(when), - "reason": reason, - } - decisions.append(decision) - decisions.sort(key=lambda x: x["certSha256"]) - return { - "type": "crl", - "sha256": _sha_bytes(raw), - "der": base64.b64encode(raw).decode("ascii"), - "issuerCaSha256": ca["sha256"], - "sourceId": "crl:" + ca["sha256"], - "thisUpdate": _ts(this_update), - "nextUpdate": _ts(next_update), - "crlNumber": number, - "decisions": decisions, - } - - -def _ocsp_status_name(value: Any) -> str: - from cryptography.x509 import ocsp # ruff: ignore[import-outside-top-level] - - if value == ocsp.OCSPCertStatus.GOOD: - return "good" - if value == ocsp.OCSPCertStatus.REVOKED: - return "revoked" - _fail("NATIVE_OCSP_CERT_STATUS_UNKNOWN") - raise AssertionError - - -def _verify_ocsp( # ruff: ignore[too-many-branches] - raw: bytes, - *, - leaves: list[dict[str, Any]], - ca_records: list[dict[str, Any]], - now: datetime, - historical: bool, -) -> dict[str, Any]: - try: - from cryptography import x509 # ruff: ignore[import-outside-top-level] - from cryptography.x509 import ocsp # ruff: ignore[import-outside-top-level] - - response = ocsp.load_der_ocsp_response(raw) - except Exception as exc: - raise NativeStatusError("NATIVE_OCSP_DER_INVALID") from exc - if response.response_status != ocsp.OCSPResponseStatus.SUCCESSFUL: - _fail("NATIVE_OCSP_RESPONSE_NOT_SUCCESSFUL") - responses = list(response.responses) - if len(responses) != 1: - _fail("NATIVE_OCSP_SINGLE_RESPONSE_REQUIRED") - single = responses[0] - leaf_matches = [ - leaf for leaf in leaves if leaf["cert"].serial_number == single.serial_number - ] - if len(leaf_matches) != 1: - _fail("NATIVE_OCSP_SERIAL_NOT_IN_INVENTORY") - leaf = leaf_matches[0] - ca_matches = [ca for ca in ca_records if ca["sha256"] == leaf["issuerCaSha256"]] - if len(ca_matches) != 1: - _fail("NATIVE_OCSP_ISSUER_CA_MISSING") - ca = ca_matches[0] - try: - req = ( - ocsp.OCSPRequestBuilder() - .add_certificate(leaf["cert"], ca["cert"], single.hash_algorithm) - .build() - ) - if ( - req.issuer_name_hash != single.issuer_name_hash - or req.issuer_key_hash != single.issuer_key_hash - ): - _fail("NATIVE_OCSP_ISSUER_HASH_MISMATCH") - except NativeStatusError: - raise - except Exception as exc: - raise NativeStatusError("NATIVE_OCSP_ISSUER_HASH_INVALID") from exc - produced = _cert_time(response.produced_at_utc) - this_update = _cert_time(single.this_update_utc) - next_raw = single.next_update_utc - if next_raw is None and bool(POLICY["require_ocsp_next_update"]): - _fail("NATIVE_OCSP_NEXT_UPDATE_MISSING") - next_update = _cert_time(next_raw) if next_raw is not None else produced - skew = timedelta(minutes=int(POLICY["max_clock_skew_minutes"])) - if produced > now + skew or this_update > now + skew: - _fail("NATIVE_OCSP_FROM_FUTURE") - if produced < this_update - skew: - _fail("NATIVE_OCSP_PRODUCED_BEFORE_THIS_UPDATE") - if next_update <= this_update or next_update - this_update > timedelta( - hours=int(POLICY["max_ocsp_lifetime_hours"]) - ): - _fail("NATIVE_OCSP_LIFETIME_INVALID") - ca_nb = lifecycle.continuity._cert_time(ca["cert"], "not_valid_before") - ca_na = lifecycle.continuity._cert_time(ca["cert"], "not_valid_after") - if produced < ca_nb or next_update > ca_na: - _fail("NATIVE_OCSP_OUTSIDE_ISSUER_VALIDITY") - if not historical and next_update < now + timedelta( - minutes=int(POLICY["min_native_remaining_minutes"]) - ): - _fail("NATIVE_OCSP_EXPIRED_OR_FREEZE_RISK") - responder_name = response.responder_name - if bool(POLICY["require_ocsp_responder_name"]) and responder_name is None: - _fail("NATIVE_OCSP_RESPONDER_NAME_REQUIRED") - signer = None - responder_der = None - responder_kind = None - embedded_all = list(response.certificates) - if responder_name == ca["cert"].subject: - if embedded_all: - _fail("NATIVE_OCSP_EXTRANEOUS_RESPONDER_CERTIFICATES") - signer = ca["cert"] - responder_der = ca["der"] - responder_kind = "issuer-ca" - else: - embedded = [ - cert - for cert in embedded_all - if responder_name is not None and cert.subject == responder_name - ] - if len(embedded_all) != 1 or len(embedded) != 1: - _fail("NATIVE_OCSP_RESPONDER_CERT_MISSING_OR_AMBIGUOUS") - signer = embedded[0] - responder_der = ( - signer.public_bytes(lifecycle.continuity.serialization.Encoding.DER) - if hasattr(lifecycle.continuity, "serialization") - else None - ) - if responder_der is None: - from cryptography.hazmat.primitives import ( # ruff: ignore[import-outside-top-level] - serialization, - ) - - responder_der = signer.public_bytes(serialization.Encoding.DER) - try: - lifecycle.continuity._verify_cert_signature( - signer, ca["cert"].public_key(), "NATIVE_OCSP_RESPONDER" - ) - bc = signer.extensions.get_extension_for_class(x509.BasicConstraints).value - if bc.ca: - _fail("NATIVE_OCSP_RESPONDER_CA_INVALID") - eku = signer.extensions.get_extension_for_class(x509.ExtendedKeyUsage).value - if x509.oid.ExtendedKeyUsageOID.OCSP_SIGNING not in eku: - _fail("NATIVE_OCSP_RESPONDER_EKU_INVALID") - ku = signer.extensions.get_extension_for_class(x509.KeyUsage).value - if not ku.digital_signature: - _fail("NATIVE_OCSP_RESPONDER_KEY_USAGE_INVALID") - nb = lifecycle.continuity._cert_time(signer, "not_valid_before") - na = lifecycle.continuity._cert_time(signer, "not_valid_after") - if not (nb <= produced <= na): - _fail("NATIVE_OCSP_RESPONDER_CERT_NOT_VALID_AT_PRODUCTION") - except NativeStatusError: - raise - except Exception as exc: - raise NativeStatusError("NATIVE_OCSP_RESPONDER_CERT_INVALID") from exc - responder_kind = "delegated" - _verify_signature( - signer.public_key(), - response.signature, - response.tbs_response_bytes, - response.signature_hash_algorithm, - "NATIVE_OCSP", - ) - status = _ocsp_status_name(single.certificate_status) - revoked_at = None - reason = None - if status == "revoked": - if single.revocation_time_utc is None: - _fail("NATIVE_OCSP_REVOCATION_TIME_MISSING") - revoked_dt = _cert_time(single.revocation_time_utc) - if revoked_dt > this_update + skew: - _fail("NATIVE_OCSP_REVOCATION_FROM_FUTURE") - revoked_at = _ts(revoked_dt) - reason = _reason_name(single.revocation_reason) or "unspecified" - decision = { - "certSha256": leaf["sha256"], - "serialNumber": leaf["serialNumber"], - "status": status, - "revokedAt": revoked_at, - "reason": reason, - } - return { - "type": "ocsp", - "sha256": _sha_bytes(raw), - "der": base64.b64encode(raw).decode("ascii"), - "issuerCaSha256": ca["sha256"], - "sourceId": "ocsp:" + _sha_bytes(responder_der), - "responderKind": responder_kind, - "responderCertSha256": _sha_bytes(responder_der), - "responderCertDer": base64.b64encode(responder_der).decode("ascii"), - "producedAt": _ts(produced), - "thisUpdate": _ts(this_update), - "nextUpdate": _ts(next_update), - "decisions": [decision], - } - - -def _normalize_decision( - status: str, revoked_at: str | None, reason: str | None -) -> tuple[str, str | None, str | None]: - if status == "good": - return ("good", None, None) - return ("revoked", revoked_at, (reason or "unspecified").replace("_", "-")) - - -def _aggregate( - *, - leaves: list[dict[str, Any]], - sources: list[dict[str, Any]], - run158_status: dict[str, Any], -) -> list[dict[str, Any]]: - expected = _run158_decisions(run158_status) - required_kinds = {str(x) for x in POLICY["required_source_kinds"]} - result = [] - for leaf in leaves: - observations = [] - for source in sources: - observations.extend( - { - "kind": source["type"], - "sourceId": source["sourceId"], - **decision, - } - for decision in source["decisions"] - if decision["certSha256"] == leaf["sha256"] - ) - kinds = {x["kind"] for x in observations} - if bool(POLICY["require_all_active_leaf_certificates"]) and ( - len(kinds) < int(POLICY["min_source_kinds_per_leaf"]) - or not required_kinds.issubset(kinds) - ): - _fail("NATIVE_STATUS_SOURCE_COVERAGE_INCOMPLETE") - normalized = { - _normalize_decision(x["status"], x["revokedAt"], x["reason"]) - for x in observations - } - if len(normalized) != 1: - _fail("NATIVE_STATUS_SOURCE_CONFLICT") - decision = next(iter(normalized)) - prior = expected.get(leaf["sha256"]) - if prior is None: - _fail("NATIVE_STATUS_RUN158_DECISION_MISSING") - prior_norm = _normalize_decision( - prior["status"], prior["revokedAt"], prior["reason"] - ) - # Run 158 reason strings are policy labels and may be more specific than a native source. - if bool(POLICY["require_native_match_run158"]): # ruff: ignore[collapsible-if] - if decision[0] != prior_norm[0] or decision[1] != prior_norm[1]: - _fail("NATIVE_STATUS_RUN158_MISMATCH") - result.append( - { - "certSha256": leaf["sha256"], - "serialNumber": leaf["serialNumber"], - "keyId": leaf["keyId"], - "status": decision[0], - "revokedAt": decision[1], - "sourceKinds": sorted(kinds), - "sourceIds": sorted({x["sourceId"] for x in observations}), - "observationCount": len(observations), - } - ) - result.sort(key=lambda x: x["certSha256"]) - return result - - -def _verify_source_continuity( - previous_event: dict[str, Any] | None, sources: list[dict[str, Any]] -) -> None: - if previous_event is None: - return - old_sources = previous_event.get("sources") - if not isinstance(old_sources, list): - _fail("NATIVE_PREVIOUS_SOURCES_INVALID") - old_crl = {x["issuerCaSha256"]: x for x in old_sources if x.get("type") == "crl"} - old_ocsp = {} - for x in old_sources: - if x.get("type") != "ocsp": - continue - for d in x.get("decisions", []): - old_ocsp[(x["sourceId"], d["certSha256"])] = x - for source in sources: - if source["type"] == "crl": - old = old_crl.get(source["issuerCaSha256"]) - if old is not None: - if source["crlNumber"] <= old["crlNumber"]: - _fail("NATIVE_CRL_NUMBER_NOT_MONOTONIC") - if _dt(source["thisUpdate"], "NATIVE_CRL_THIS_UPDATE_INVALID") <= _dt( - old["thisUpdate"], "NATIVE_OLD_CRL_THIS_UPDATE_INVALID" - ): - _fail("NATIVE_CRL_TIME_NOT_MONOTONIC") - else: - d = source["decisions"][0] - old = old_ocsp.get((source["sourceId"], d["certSha256"])) - if old is not None and _dt( - source["thisUpdate"], "NATIVE_OCSP_THIS_UPDATE_INVALID" - ) <= _dt(old["thisUpdate"], "NATIVE_OLD_OCSP_THIS_UPDATE_INVALID"): - _fail("NATIVE_OCSP_TIME_NOT_MONOTONIC") - - -def _vendor_verifier(spec: str) -> tuple[str, Path]: - if "=" not in spec: - _fail("NATIVE_VENDOR_VERIFIER_SPEC_INVALID") - profile, raw_path = spec.split("=", 1) - profile = _identity(profile, "NATIVE_VENDOR_PROFILE_INVALID") - path = Path(raw_path).expanduser().resolve() - if path.is_symlink() or not path.is_file() or not os.access(path, os.X_OK): - _fail("NATIVE_VENDOR_VERIFIER_EXECUTABLE_INVALID") - return profile, path - - -def _verify_vendor_manifest( # ruff: ignore[too-many-branches] - path: Path, - verifiers: dict[str, Path], -) -> dict[str, Any]: - doc, raw = _read_json(path, "NATIVE_VENDOR_MANIFEST") - if ( - set(doc) - != { - "schemaVersion", - "profile", - "evidenceType", - "rawEvidenceBase64", - "expectedKeyId", - "expectedPublicKeySha256", - } - or doc["schemaVersion"] != 1 - ): - _fail("NATIVE_VENDOR_MANIFEST_SCHEMA_INVALID") - profile = _identity(doc["profile"], "NATIVE_VENDOR_PROFILE_INVALID") - verifier = verifiers.get(profile) - if verifier is None: - _fail("NATIVE_VENDOR_PROFILE_VERIFIER_NOT_CONFIGURED") - try: - evidence = base64.b64decode(doc["rawEvidenceBase64"], validate=True) - except Exception as exc: - raise NativeStatusError("NATIVE_VENDOR_RAW_EVIDENCE_INVALID") from exc - if not evidence or len(evidence) > int(POLICY["max_raw_evidence_bytes"]): - _fail("NATIVE_VENDOR_RAW_EVIDENCE_SIZE_INVALID") - request = _canonical( - { - "profile": profile, - "evidenceType": _identity( - doc["evidenceType"], "NATIVE_VENDOR_EVIDENCE_TYPE_INVALID" - ), - "rawEvidenceSha256": _sha_bytes(evidence), - "rawEvidenceBase64": base64.b64encode(evidence).decode("ascii"), - "expectedKeyId": _identity( - doc["expectedKeyId"], "NATIVE_VENDOR_KEY_ID_INVALID" - ), - "expectedPublicKeySha256": _hex( - doc["expectedPublicKeySha256"], "NATIVE_VENDOR_PUBLIC_HASH_INVALID" - ), - } - ) - limit = int(POLICY["vendor_verifier_max_output_bytes"]) - proc = subprocess.Popen( # ruff: ignore[subprocess-without-shell-equals-true] - [str(verifier)], - stdin=subprocess.PIPE, - stdout=subprocess.PIPE, - stderr=subprocess.DEVNULL, - env={"PATH": os.defpath, "LC_ALL": "C", "LANG": "C"}, - ) - assert proc.stdin is not None # ruff: ignore[assert] - assert proc.stdout is not None # ruff: ignore[assert] - chunks: list[bytes] = [] - state: dict[str, Any] = {"size": 0, "overflow": False, "error": None} - - def drain() -> None: - try: - while True: - chunk = proc.stdout.read(65536) - if not chunk: - return - state["size"] += len(chunk) - if state["size"] > limit: - state["overflow"] = True - try: # ruff: ignore[suppressible-exception] - proc.kill() - except OSError: - pass - return - chunks.append(chunk) - # defensive pipe failure - except BaseException as exc: # pragma: no cover # ruff: ignore[blind-except] - state["error"] = exc - with contextlib.suppress(OSError): - proc.kill() - - reader = threading.Thread( - target=drain, name="native-vendor-verifier-stdout", daemon=True - ) - reader.start() - try: - try: - proc.stdin.write(request) - proc.stdin.close() - proc.wait(timeout=int(POLICY["vendor_verifier_timeout_seconds"])) - except subprocess.TimeoutExpired as exc: - proc.kill() - proc.wait() - reader.join(timeout=2) - raise NativeStatusError("NATIVE_VENDOR_VERIFIER_TIMEOUT") from exc - finally: - try: # ruff: ignore[suppressible-exception] - proc.stdin.close() - except OSError: - pass - reader.join(timeout=2) - if reader.is_alive() or state["error"] is not None: - _fail("NATIVE_VENDOR_VERIFIER_PIPE_FAILURE") - if state["overflow"]: - _fail("NATIVE_VENDOR_VERIFIER_OUTPUT_TOO_LARGE") - if proc.returncode != 0: - _fail("NATIVE_VENDOR_VERIFIER_FAILED") - stdout = b"".join(chunks) - finally: - if proc.poll() is None: - proc.kill() - proc.wait() - if reader.is_alive(): - reader.join(timeout=2) - try: # ruff: ignore[suppressible-exception] - proc.stdout.close() - except OSError: - pass - result = _loads(stdout, "NATIVE_VENDOR_VERIFIER_RESULT") - expected = { - "schemaVersion", - "profile", - "verifierIdentity", - "rawEvidenceSha256", - "keyId", - "publicKeySha256", - "verified", - } - if ( - set(result) != expected - or result["schemaVersion"] != 1 - or result["verified"] is not True - ): - _fail("NATIVE_VENDOR_VERIFIER_RESULT_INVALID") - if stdout != _canonical(result): - _fail("NATIVE_VENDOR_VERIFIER_RESULT_NOT_CANONICAL") - if ( - result["profile"] != profile - or result["rawEvidenceSha256"] != _sha_bytes(evidence) - or result["keyId"] != doc["expectedKeyId"] - or result["publicKeySha256"] != doc["expectedPublicKeySha256"] - ): - _fail("NATIVE_VENDOR_VERIFIER_RESULT_MISMATCH") - return { - "manifestSha256": _sha_bytes(raw), - "profile": profile, - "evidenceType": doc["evidenceType"], - "rawEvidenceSha256": _sha_bytes(evidence), - "rawEvidenceBase64": base64.b64encode(evidence).decode("ascii"), - "verifierExecutableSha256": _sha(verifier), - "result": result, - } - - -def _validate_preserved_vendor(items: Any) -> list[dict[str, Any]]: - if not isinstance(items, list): - _fail("NATIVE_VENDOR_EVIDENCE_LIST_INVALID") - normalized = [] - for item in items: - expected = { - "manifestSha256", - "profile", - "evidenceType", - "rawEvidenceSha256", - "rawEvidenceBase64", - "verifierExecutableSha256", - "result", - } - if not isinstance(item, dict) or set(item) != expected: - _fail("NATIVE_VENDOR_EVIDENCE_SCHEMA_INVALID") - raw = base64.b64decode(item["rawEvidenceBase64"], validate=True) - if _sha_bytes(raw) != _hex( - item["rawEvidenceSha256"], "NATIVE_VENDOR_EVIDENCE_HASH_INVALID" - ): - _fail("NATIVE_VENDOR_EVIDENCE_HASH_MISMATCH") - _hex(item["manifestSha256"], "NATIVE_VENDOR_MANIFEST_HASH_INVALID") - _hex(item["verifierExecutableSha256"], "NATIVE_VENDOR_VERIFIER_HASH_INVALID") - _identity(item["profile"], "NATIVE_VENDOR_PROFILE_INVALID") - _identity(item["evidenceType"], "NATIVE_VENDOR_EVIDENCE_TYPE_INVALID") - result = item["result"] - if ( - not isinstance(result, dict) - or result.get("verified") is not True - or result.get("rawEvidenceSha256") != item["rawEvidenceSha256"] - or result.get("profile") != item["profile"] - ): - _fail("NATIVE_VENDOR_PRESERVED_RESULT_INVALID") - normalized.append(item) - expected_order = sorted( - normalized, key=lambda x: (x["profile"], x["rawEvidenceSha256"]) - ) - if normalized != expected_order: - _fail("NATIVE_VENDOR_EVIDENCE_NOT_NORMALIZED") - return normalized - - -def _event_head(previous: str, body: dict[str, Any]) -> str: - return _sha_bytes(_canonical({"previousChainHeadSha256": previous, "body": body})) - - -def _input_hashes(paths: list[Path]) -> dict[str, str]: - return {str(p.resolve()): _sha(p) for p in paths} - - -def _check_inputs(before: dict[str, str]) -> None: - for raw, expected in before.items(): - path = Path(raw) - if not path.is_file() or path.is_symlink() or _sha(path) != expected: - _fail("NATIVE_INPUT_DRIFT_DETECTED") - - -def _write_output( - target: Path, - *, - bundle: dict[str, Any], - state: dict[str, Any], - event: dict[str, Any], - inputs: dict[str, str], - phase: str, -) -> None: - _check_inputs(inputs) - parent = target.parent - parent.mkdir(parents=True, exist_ok=True) - tmp = Path(tempfile.mkdtemp(prefix=".native-status-", dir=parent)) - try: - _write(tmp / "release-native-status-bundle.json", bundle) - _write(tmp / _DOC_NATIVE_STATUS_STATE, state) - _write(tmp / "active-native-status-evidence.json", event) - receipt = { - "schemaVersion": int(POLICY["receipt_schema_version"]), - "phase": phase, - "bundle": _artifact( - "release-native-status-bundle.json", _canonical(bundle) - ), - "state": _artifact(_DOC_NATIVE_STATUS_STATE, _canonical(state)), - "activeEvidence": _artifact( - "active-native-status-evidence.json", _canonical(event) - ), - } - _write(tmp / "release-native-status-receipt.json", receipt) - _check_inputs(inputs) - os.replace(tmp, target) - except Exception: - shutil.rmtree(tmp, ignore_errors=True) - raise - - -def _build_sources( - *, - crl_paths: list[Path], - ocsp_paths: list[Path], - leaves: list[dict[str, Any]], - ca_records: list[dict[str, Any]], - now: datetime, - historical: bool, -) -> list[dict[str, Any]]: - if len(crl_paths) + len(ocsp_paths) > int(POLICY["max_raw_evidence_files"]): - _fail("NATIVE_EVIDENCE_FILE_COUNT_INVALID") - total = 0 - for path in [*crl_paths, *ocsp_paths]: - if path.is_symlink() or not path.is_file(): - _fail("NATIVE_EVIDENCE_FILE_INVALID") - total += path.stat().st_size - if total > int(POLICY["max_total_raw_evidence_bytes"]): - _fail("NATIVE_EVIDENCE_TOTAL_SIZE_INVALID") - sources = [ - _verify_crl( - _raw(path, "NATIVE_CRL_FILE"), - leaves=leaves, - ca_records=ca_records, - now=now, - historical=historical, - ) - for path in crl_paths - ] - sources.extend( - _verify_ocsp( - _raw(path, "NATIVE_OCSP_FILE"), - leaves=leaves, - ca_records=ca_records, - now=now, - historical=historical, - ) - for path in ocsp_paths - ) - sources.sort(key=lambda x: (x["type"], x["sourceId"], x["sha256"])) - if len({(x["type"], x["sourceId"], x["sha256"]) for x in sources}) != len(sources): - _fail("NATIVE_EVIDENCE_DUPLICATE") - crl_ids = [x["sourceId"] for x in sources if x["type"] == "crl"] - if len(set(crl_ids)) != len(crl_ids): - _fail("NATIVE_CRL_SOURCE_AMBIGUOUS") - ocsp_ids = [ - (x["sourceId"], x["decisions"][0]["certSha256"]) - for x in sources - if x["type"] == "ocsp" - ] - if len(set(ocsp_ids)) != len(ocsp_ids): - _fail("NATIVE_OCSP_SOURCE_AMBIGUOUS") - return sources - - -def _predecessor_binding(docs: dict[str, dict[str, Any]]) -> dict[str, Any]: - state = docs[_DOC_ATTESTATION_LIFECYCLE_STATE] - bundle = docs["release-attestation-lifecycle-bundle.json"] - return { - "bundle": _artifact( - "release-attestation-lifecycle-bundle.json", _canonical(bundle) - ), - "state": _artifact(_DOC_ATTESTATION_LIFECYCLE_STATE, _canonical(state)), - "lifecycleChainHeadSha256": _hex( - state["lifecycleChainHeadSha256"], "NATIVE_LIFECYCLE_HEAD_INVALID" - ), - "activeCaSetSha256": _sha_bytes( - _canonical(docs["active-attestation-ca-set.json"]) - ), - "activeStatusSha256": _sha_bytes( - _canonical(docs["active-attestation-status.json"]) - ), - } - - -def initialize_native_status( # ruff: ignore[undocumented-public-function] - *, - lifecycle_dir: Path, - expected_bootstrap_root_sha256: str, - expected_recovery_root_sha256: str, - expected_attestation_root_sha256: list[str], - crl_paths: list[Path], - ocsp_paths: list[Path], - output_dir: Path, - vendor_manifest_paths: list[Path] | None = None, - vendor_verifier_specs: list[str] | None = None, - now: datetime | None = None, -) -> dict[str, Any]: - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - pred = _regular_dir(lifecycle_dir, "NATIVE_LIFECYCLE_DIR_INVALID") - docs = _verify_predecessor( - pred, - bootstrap_pin=expected_bootstrap_root_sha256, - recovery_pin=expected_recovery_root_sha256, - attestation_pins=expected_attestation_root_sha256, - now=current, - historical=False, - ) - lifecycle_bundle = docs["release-attestation-lifecycle-bundle.json"] - ca_set = docs["active-attestation-ca-set.json"] - leaves = _leaf_inventory(lifecycle_bundle, ca_set) - ca_records = _ca_records(ca_set) - sources = _build_sources( - crl_paths=crl_paths, - ocsp_paths=ocsp_paths, - leaves=leaves, - ca_records=ca_records, - now=current, - historical=False, - ) - decisions = _aggregate( - leaves=leaves, - sources=sources, - run158_status=docs["active-attestation-status.json"], - ) - _verify_source_continuity(None, sources) - verifiers = dict(_vendor_verifier(spec) for spec in vendor_verifier_specs or []) - vendor = [ - _verify_vendor_manifest(p, verifiers) for p in vendor_manifest_paths or [] - ] - vendor.sort(key=lambda x: (x["profile"], x["rawEvidenceSha256"])) - predecessor = _predecessor_binding(docs) - body = { - "sequence": 1, - "type": "native-status-bootstrap", - "previousChainHeadSha256": predecessor["lifecycleChainHeadSha256"], - "predecessor": predecessor, - "sources": sources, - "decisions": decisions, - "vendorEvidence": vendor, - } - head = _event_head(body["previousChainHeadSha256"], body) - event = dict(body, nativeStatusChainHeadSha256=head) - bundle = { - "schemaVersion": int(POLICY["bundle_schema_version"]), - "predicateType": PREDICATE_TYPE, - "status": "trusted-native-status-provenance", - "bootstrapRootSha256": _hex( - expected_bootstrap_root_sha256, "NATIVE_BOOTSTRAP_PIN_INVALID" - ), - "recoveryRootSha256": _hex( - expected_recovery_root_sha256, "NATIVE_RECOVERY_PIN_INVALID" - ), - "attestationRootSha256": sorted( - _hex(x, "NATIVE_ATTESTATION_PIN_INVALID") - for x in expected_attestation_root_sha256 - ), - "attestationLifecycleOutput": docs, - "events": [event], - } - state = { - "schemaVersion": int(POLICY["state_schema_version"]), - "status": "trusted-native-status-provenance", - "sequence": 1, - "lifecycleChainHeadSha256": predecessor["lifecycleChainHeadSha256"], - "nativeStatusChainHeadSha256": head, - "activeEvidenceSha256": _sha_bytes(_canonical(event)), - "bundleSha256": _sha_bytes(_canonical(bundle)), - } - protected = [pred, *crl_paths, *ocsp_paths, *(vendor_manifest_paths or [])] - target = _outside(output_dir, protected, "NATIVE_OUTPUT_INSIDE_INPUT") - if target.exists() or target.is_symlink(): - _fail("NATIVE_OUTPUT_ALREADY_EXISTS") - inputs = ( - [p for p in pred.rglob("*") if p.is_file()] - + list(crl_paths) - + list(ocsp_paths) - + list(vendor_manifest_paths or []) - + list(verifiers.values()) - ) - before = _input_hashes(inputs) - _write_output( - target, - bundle=bundle, - state=state, - event=event, - inputs=before, - phase="native-status-initialized", - ) - verify_native_status( - output_dir=target, - expected_bootstrap_root_sha256=expected_bootstrap_root_sha256, - expected_recovery_root_sha256=expected_recovery_root_sha256, - expected_attestation_root_sha256=expected_attestation_root_sha256, - now=current, - ) - return { - "ok": True, - "phase": "native-status-initialized", - "sequence": 1, - "native_status_chain_head_sha256": head, - } - - -def _native_docs(root: Path) -> dict[str, dict[str, Any]]: - names = { - "release-native-status-bundle.json", - _DOC_NATIVE_STATUS_STATE, - "active-native-status-evidence.json", - "release-native-status-receipt.json", - } - if {p.name for p in root.iterdir()} != names: - _fail("NATIVE_OUTPUT_ALLOWLIST_MISMATCH") - return { - name: _read_json( - root / name, - "NATIVE_OUTPUT_" + name.upper().replace("-", "_").replace(".", "_"), - )[0] - for name in sorted(names) - } - - -def _verify_bundle( # ruff: ignore[too-many-branches] - bundle: dict[str, Any], - *, - bootstrap_pin: str, - recovery_pin: str, - attestation_pins: list[str], - now: datetime, - historical: bool, -) -> tuple[dict[str, Any], str]: - expected_top = { - "schemaVersion", - "predicateType", - "status", - "bootstrapRootSha256", - "recoveryRootSha256", - "attestationRootSha256", - "attestationLifecycleOutput", - "events", - } - if ( - set(bundle) != expected_top - or bundle["schemaVersion"] != int(POLICY["bundle_schema_version"]) - or bundle["predicateType"] != PREDICATE_TYPE - or bundle["status"] != "trusted-native-status-provenance" - ): - _fail("NATIVE_BUNDLE_SCHEMA_INVALID") - if ( - bundle["bootstrapRootSha256"] != bootstrap_pin - or bundle["recoveryRootSha256"] != recovery_pin - or bundle["attestationRootSha256"] != sorted(attestation_pins) - ): - _fail("NATIVE_BUNDLE_PIN_MISMATCH") - # Rehydrate and independently re-run Run 158 from canonical embedded bytes. - tmp = Path(tempfile.mkdtemp(prefix=".native-predecessor-")) - try: - docs = bundle["attestationLifecycleOutput"] - if not isinstance(docs, dict): - _fail("NATIVE_EMBEDDED_PREDECESSOR_INVALID") - for name, doc in docs.items(): - _write(tmp / name, doc) - pred_docs = _verify_predecessor( - tmp, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - now=now, - historical=historical, - ) - finally: - shutil.rmtree(tmp, ignore_errors=True) - events = bundle["events"] - if ( - not isinstance(events, list) - or not events - or len(events) > int(POLICY["max_events"]) - ): - _fail("NATIVE_EVENTS_INVALID") - ca_set = pred_docs["active-attestation-ca-set.json"] - lifecycle_bundle = pred_docs["release-attestation-lifecycle-bundle.json"] - leaves = _leaf_inventory(lifecycle_bundle, ca_set) - ca_records = _ca_records(ca_set) - previous_head = pred_docs[_DOC_ATTESTATION_LIFECYCLE_STATE][ - "lifecycleChainHeadSha256" - ] - previous_event = None - final_event = None - for index, event in enumerate(events, start=1): - if not isinstance(event, dict) or set(event) != { - "sequence", - "type", - "previousChainHeadSha256", - "predecessor", - "sources", - "decisions", - "vendorEvidence", - "nativeStatusChainHeadSha256", - }: - _fail("NATIVE_EVENT_SCHEMA_INVALID") - if event["sequence"] != index or event["type"] not in { - "native-status-bootstrap", - "native-status-advance", - }: - _fail("NATIVE_EVENT_SEQUENCE_INVALID") - if event["previousChainHeadSha256"] != previous_head: - _fail("NATIVE_EVENT_PREVIOUS_HEAD_MISMATCH") - if event["predecessor"] != _predecessor_binding(pred_docs): - _fail("NATIVE_EVENT_PREDECESSOR_MISMATCH") - verified_sources = [] - event_historical = historical or index < len(events) - for source in event["sources"]: - if not isinstance(source, dict) or source.get("type") not in { - "crl", - "ocsp", - }: - _fail("NATIVE_EVENT_SOURCE_SCHEMA_INVALID") - raw = base64.b64decode(source.get("der", ""), validate=True) - if source["type"] == "crl": - verified = _verify_crl( - raw, - leaves=leaves, - ca_records=ca_records, - now=now, - historical=event_historical, - ) - else: - verified = _verify_ocsp( - raw, - leaves=leaves, - ca_records=ca_records, - now=now, - historical=event_historical, - ) - if verified != source: - _fail("NATIVE_EVENT_SOURCE_REBIND_MISMATCH") - verified_sources.append(verified) - _verify_source_continuity(previous_event, verified_sources) - decisions = _aggregate( - leaves=leaves, - sources=verified_sources, - run158_status=pred_docs["active-attestation-status.json"], - ) - if decisions != event["decisions"]: - _fail("NATIVE_EVENT_DECISION_MISMATCH") - # Vendor evidence is preserved and hash-bound. Re-running a vendor binary is intentionally - # not required for historical replay; its exact executable hash and canonical result are kept. - _validate_preserved_vendor(event["vendorEvidence"]) - body = {k: event[k] for k in event if k != "nativeStatusChainHeadSha256"} - head = _event_head(previous_head, body) - if event["nativeStatusChainHeadSha256"] != head: - _fail("NATIVE_EVENT_CHAIN_HEAD_MISMATCH") - previous_head = head - previous_event = event - final_event = event - assert final_event is not None # ruff: ignore[assert] - return final_event, previous_head - - -def verify_native_status( # ruff: ignore[undocumented-public-function] - *, - output_dir: Path, - expected_bootstrap_root_sha256: str, - expected_recovery_root_sha256: str, - expected_attestation_root_sha256: list[str], - now: datetime | None = None, - historical: bool = False, -) -> dict[str, Any]: - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - root = _regular_dir(output_dir, "NATIVE_OUTPUT_DIR_INVALID") - docs = _native_docs(root) - final_event, head = _verify_bundle( - docs["release-native-status-bundle.json"], - bootstrap_pin=expected_bootstrap_root_sha256, - recovery_pin=expected_recovery_root_sha256, - attestation_pins=sorted(expected_attestation_root_sha256), - now=current, - historical=historical, - ) - state = docs[_DOC_NATIVE_STATUS_STATE] - bundle = docs["release-native-status-bundle.json"] - expected_state = { - "schemaVersion": int(POLICY["state_schema_version"]), - "status": "trusted-native-status-provenance", - "sequence": len(bundle["events"]), - "lifecycleChainHeadSha256": bundle["attestationLifecycleOutput"][ - _DOC_ATTESTATION_LIFECYCLE_STATE - ]["lifecycleChainHeadSha256"], - "nativeStatusChainHeadSha256": head, - "activeEvidenceSha256": _sha_bytes(_canonical(final_event)), - "bundleSha256": _sha_bytes(_canonical(bundle)), - } - if ( - state != expected_state - or docs["active-native-status-evidence.json"] != final_event - ): - _fail("NATIVE_STATE_OR_ACTIVE_EVIDENCE_MISMATCH") - receipt = docs["release-native-status-receipt.json"] - expected_receipt = { - "schemaVersion": int(POLICY["receipt_schema_version"]), - "phase": ( - "native-status-initialized" - if len(bundle["events"]) == 1 - else "native-status-advanced" - ), - "bundle": _artifact("release-native-status-bundle.json", _canonical(bundle)), - "state": _artifact(_DOC_NATIVE_STATUS_STATE, _canonical(state)), - "activeEvidence": _artifact( - "active-native-status-evidence.json", _canonical(final_event) - ), - } - if receipt != expected_receipt: - _fail("NATIVE_RECEIPT_MISMATCH") - return { - "ok": True, - "sequence": len(bundle["events"]), - "native_status_chain_head_sha256": head, - } - - -def advance_native_status( # ruff: ignore[undocumented-public-function] - *, - previous_dir: Path, - crl_paths: list[Path], - ocsp_paths: list[Path], - output_dir: Path, - expected_bootstrap_root_sha256: str, - expected_recovery_root_sha256: str, - expected_attestation_root_sha256: list[str], - now: datetime | None = None, -) -> dict[str, Any]: - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - prev = _regular_dir(previous_dir, "NATIVE_PREVIOUS_DIR_INVALID") - verify_native_status( - output_dir=prev, - expected_bootstrap_root_sha256=expected_bootstrap_root_sha256, - expected_recovery_root_sha256=expected_recovery_root_sha256, - expected_attestation_root_sha256=expected_attestation_root_sha256, - now=current, - ) - docs = _native_docs(prev) - bundle = json.loads(json.dumps(docs["release-native-status-bundle.json"])) - pred_docs = bundle["attestationLifecycleOutput"] - ca_set = pred_docs["active-attestation-ca-set.json"] - lifecycle_bundle = pred_docs["release-attestation-lifecycle-bundle.json"] - leaves = _leaf_inventory(lifecycle_bundle, ca_set) - ca_records = _ca_records(ca_set) - sources = _build_sources( - crl_paths=crl_paths, - ocsp_paths=ocsp_paths, - leaves=leaves, - ca_records=ca_records, - now=current, - historical=False, - ) - previous_event = bundle["events"][-1] - _verify_source_continuity(previous_event, sources) - decisions = _aggregate( - leaves=leaves, - sources=sources, - run158_status=pred_docs["active-attestation-status.json"], - ) - seq = len(bundle["events"]) + 1 - predecessor = _predecessor_binding(pred_docs) - body = { - "sequence": seq, - "type": "native-status-advance", - "previousChainHeadSha256": docs[_DOC_NATIVE_STATUS_STATE][ - "nativeStatusChainHeadSha256" - ], - "predecessor": predecessor, - "sources": sources, - "decisions": decisions, - "vendorEvidence": [], - } - head = _event_head(body["previousChainHeadSha256"], body) - event = dict(body, nativeStatusChainHeadSha256=head) - bundle["events"].append(event) - state = { - "schemaVersion": int(POLICY["state_schema_version"]), - "status": "trusted-native-status-provenance", - "sequence": seq, - "lifecycleChainHeadSha256": predecessor["lifecycleChainHeadSha256"], - "nativeStatusChainHeadSha256": head, - "activeEvidenceSha256": _sha_bytes(_canonical(event)), - "bundleSha256": _sha_bytes(_canonical(bundle)), - } - target = _outside( - output_dir, - [prev, *crl_paths, *ocsp_paths], - "NATIVE_ADVANCE_OUTPUT_INSIDE_INPUT", - ) - if target.exists() or target.is_symlink(): - _fail("NATIVE_ADVANCE_OUTPUT_ALREADY_EXISTS") - inputs = ( - [p for p in prev.rglob("*") if p.is_file()] + list(crl_paths) + list(ocsp_paths) - ) - before = _input_hashes(inputs) - _write_output( - target, - bundle=bundle, - state=state, - event=event, - inputs=before, - phase="native-status-advanced", - ) - verify_native_status( - output_dir=target, - expected_bootstrap_root_sha256=expected_bootstrap_root_sha256, - expected_recovery_root_sha256=expected_recovery_root_sha256, - expected_attestation_root_sha256=expected_attestation_root_sha256, - now=current, - ) - return { - "ok": True, - "phase": "native-status-advanced", - "sequence": seq, - "native_status_chain_head_sha256": head, - } - - -def _paths(values: list[str]) -> list[Path]: - return [Path(x) for x in values] - - -def main( # ruff: ignore[undocumented-public-function] - argv: list[str] | None = None, -) -> int: - parser = argparse.ArgumentParser(description=__doc__) - sub = parser.add_subparsers(dest="command", required=True) - common = argparse.ArgumentParser(add_help=False) - common.add_argument("--bootstrap-root-sha256", required=True) - common.add_argument("--recovery-root-sha256", required=True) - common.add_argument("--attestation-root-sha256", action="append", required=True) - p = sub.add_parser("initialize", parents=[common]) - p.add_argument("--lifecycle-dir", type=Path, required=True) - p.add_argument("--crl", action="append", default=[]) - p.add_argument("--ocsp", action="append", default=[]) - p.add_argument("--vendor-manifest", action="append", default=[]) - p.add_argument("--vendor-verifier", action="append", default=[]) - p.add_argument("--output-dir", type=Path, required=True) - p = sub.add_parser("advance", parents=[common]) - p.add_argument("--previous-dir", type=Path, required=True) - p.add_argument("--crl", action="append", default=[]) - p.add_argument("--ocsp", action="append", default=[]) - p.add_argument("--output-dir", type=Path, required=True) - p = sub.add_parser("verify", parents=[common]) - p.add_argument("--output-dir", type=Path, required=True) - p.add_argument("--historical", action="store_true") - args = parser.parse_args(argv) - kwargs = { - "expected_bootstrap_root_sha256": args.bootstrap_root_sha256, - "expected_recovery_root_sha256": args.recovery_root_sha256, - "expected_attestation_root_sha256": args.attestation_root_sha256, - } - try: - if args.command == "initialize": - result = initialize_native_status( - lifecycle_dir=args.lifecycle_dir, - crl_paths=_paths(args.crl), - ocsp_paths=_paths(args.ocsp), - vendor_manifest_paths=_paths(args.vendor_manifest), - vendor_verifier_specs=args.vendor_verifier, - output_dir=args.output_dir, - **kwargs, - ) - elif args.command == "advance": - result = advance_native_status( - previous_dir=args.previous_dir, - crl_paths=_paths(args.crl), - ocsp_paths=_paths(args.ocsp), - output_dir=args.output_dir, - **kwargs, - ) - else: - result = verify_native_status( - output_dir=args.output_dir, historical=args.historical, **kwargs - ) - except NativeStatusError as exc: - logger.error(json.dumps({"ok": False, "error": str(exc)}, sort_keys=True)) - return 2 - logger.info(json.dumps(result, sort_keys=True)) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/security/verify_release_evidence.py b/security/verify_release_evidence.py deleted file mode 100644 index 7d1f4414c33f220d423de5165baa32223dffe9ee..0000000000000000000000000000000000000000 --- a/security/verify_release_evidence.py +++ /dev/null @@ -1,831 +0,0 @@ -""" -Verify production release evidence without externalizing secrets. - -This verifier binds time-sensitive CI/deployment evidence to the exact B39 -source lock, Python SBOM, immutable base-image policy, target platform and final -OCI image digest. It deliberately does not contact registries, scanners, -Redis, or telemetry services; evidence collection is a separate explicit CI or -operator action. -""" - -from __future__ import annotations - -import argparse -import hashlib -import json -import logging -import re -import sys -from datetime import datetime, timedelta, timezone -from pathlib import Path -from typing import Any - -import tomllib - -logger = logging.getLogger(__name__) - -HERE = Path(__file__).resolve().parent -if str(HERE) not in sys.path: - sys.path.insert(0, str(HERE)) -from source_tree import EXTENSION_ROOT, source_tree_sha256 # noqa: E402 - -ROOT = HERE.parent -POLICY = tomllib.loads((HERE / "release_evidence_policy.toml").read_text()) -SUPPLY = tomllib.loads((HERE / "supply_chain_policy.toml").read_text()) -HEX64 = re.compile(r"^[0-9a-f]{64}$") -DIGEST = re.compile(r"^sha256:([0-9a-f]{64})$") -SLSA_PREDICATE = "https://slsa.dev/provenance/v1" -RELEASE_ID = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{2,127}$") -SOURCE_REVISION = re.compile(r"^[0-9a-f]{40}(?:[0-9a-f]{24})?$") -REDIS_CHAOS_PREDICATE = ( - "https://scikit-plots.org/attestations/provider-artifact-redis-chaos/v1" -) -RUNTIME_SOURCE_FILES = ( - "Dockerfile", - ".dockerignore", - "requirements.lock", - "app.py", - "deduplicate_dataset.py", -) -PROXY_VERSION_RE = re.compile( - r'^PROXY_VERSION: str = "([0-9]+\.[0-9]+\.[0-9]+)"$', re.MULTILINE -) -CYCLONEDX_VERSION_RE = re.compile(r"^[0-9]+\.[0-9]+$") -FORBIDDEN_SECRET_KEYS = { - "authorization", - "cookie", - "password", - "passwd", - "secret", - "token", - "apikey", - "api_key", - "privatekey", - "private_key", - "redisurl", - "redis_url", - "connectionstring", - "connection_string", - "hostname", - "username", - "host", -} -SECRET_VALUE_PATTERNS = ( - re.compile(r"-----BEGIN [A-Z ]*PRIVATE KEY-----"), - re.compile(r"(?i)\bbearer\s+[A-Za-z0-9._~+/-]{12,}"), - re.compile(r"(?i)\b(?:redis|rediss|https?)://[^\s/@:]+:[^\s/@]+@"), -) - - -class EvidenceError(RuntimeError): # ruff: ignore[undocumented-public-class] - pass - - -def _fail(code: str) -> None: - raise EvidenceError(code) - - -def _runtime_source_sha256(root: Path = ROOT) -> str: - """Hash exact application/container inputs with stable names and lengths.""" - paths = [root / name for name in RUNTIME_SOURCE_FILES] - utils_root = root / "_utils" - if not utils_root.is_dir() or utils_root.is_symlink(): - _fail("RUNTIME_SOURCE_UTILS_INVALID") - paths.extend( - sorted( - path - for path in utils_root.rglob("*") - if path.is_file() - and "__pycache__" not in path.parts - and path.suffix not in {".pyc", ".pyo"} - ) - ) - h = hashlib.sha256() - for path in paths: - if not path.is_file() or path.is_symlink(): - _fail("RUNTIME_SOURCE_FILE_INVALID") - rel = path.relative_to(root).as_posix().encode("utf-8") - data = path.read_bytes() - h.update(len(rel).to_bytes(4, "big")) - h.update(rel) - h.update(len(data).to_bytes(8, "big")) - h.update(data) - return h.hexdigest() - - -def _sha256(path: Path) -> str: - h = hashlib.sha256() - with path.open("rb") as handle: - for chunk in iter(lambda: handle.read(1024 * 1024), b""): - h.update(chunk) - return h.hexdigest() - - -def _parse_time(value: Any, code: str) -> datetime: - if not isinstance(value, str) or not value.endswith("Z"): - _fail(code) - try: - dt = datetime.fromisoformat(value[:-1] + "+00:00") - except ValueError: - _fail(code) - if dt.tzinfo is None: - _fail(code) - return dt.astimezone(timezone.utc) - - -def _hex(value: Any, code: str) -> str: - if not isinstance(value, str) or HEX64.fullmatch(value) is None: - _fail(code) - return value - - -def _digest(value: Any, code: str) -> str: - if not isinstance(value, str) or DIGEST.fullmatch(value) is None: - _fail(code) - return value - - -def _mapping(value: Any, code: str) -> dict[str, Any]: - if not isinstance(value, dict): - _fail(code) - return value - - -def _exact_keys(value: dict[str, Any], expected: set[str], code: str) -> None: - if set(value) != expected: - _fail(code) - - -def _tool(value: Any, code: str) -> None: - tool = _mapping(value, code) - _exact_keys(tool, {"name", "version"}, code) - for field in ("name", "version"): - item = tool.get(field) - if ( - not isinstance(item, str) - or not item.strip() - or len(item) > 128 # ruff: ignore[magic-value-comparison] - or any(ord(ch) < 32 for ch in item) # ruff: ignore[magic-value-comparison] - ): - _fail(code) - - -def _current_proxy_version() -> str: - shared = ROOT / "_utils" / "_shared_logic.py" - try: - text = shared.read_text(encoding="utf-8") - except OSError: - _fail("PROXY_VERSION_SOURCE_UNREADABLE") - match = PROXY_VERSION_RE.search(text) - if match is None: - _fail("PROXY_VERSION_SOURCE_INVALID") - return match.group(1) - - -def _version_tuple(value: Any, code: str) -> tuple[int, int]: - if not isinstance(value, str) or CYCLONEDX_VERSION_RE.fullmatch(value) is None: - _fail(code) - major, minor = value.split(".", 1) - return int(major), int(minor) - - -def _bool(value: Any, expected: bool, code: str) -> None: - if value is not expected: - _fail(code) - - -def _walk_values(value: Any) -> None: - if isinstance(value, dict): - for key, child in value.items(): - if str(key).replace("-", "_").lower() in FORBIDDEN_SECRET_KEYS: - _fail("EVIDENCE_SECRET_FIELD_FORBIDDEN") - _walk_values(child) - elif isinstance(value, list): - for child in value: - _walk_values(child) - elif isinstance(value, str): - if "\x00" in value or len(value) > 4096: # ruff: ignore[magic-value-comparison] - _fail("EVIDENCE_STRING_UNSAFE") - for pattern in SECRET_VALUE_PATTERNS: - if pattern.search(value): - _fail("EVIDENCE_SECRET_LIKE_VALUE_FORBIDDEN") - - -def _artifact_path(evidence_dir: Path, raw: Any) -> Path: - if not isinstance(raw, str) or not raw or raw.startswith(("/", "~")) or "\\" in raw: - _fail("ARTIFACT_PATH_INVALID") - rel = Path(raw) - if rel.is_absolute() or any(part in {"", ".", ".."} for part in rel.parts): - _fail("ARTIFACT_PATH_TRAVERSAL") - root = evidence_dir.resolve() - candidate = root / rel - current = root - for part in rel.parts: - current = current / part - if current.is_symlink(): - _fail("ARTIFACT_SYMLINK_FORBIDDEN") - path = candidate.resolve() - if root not in path.parents: - _fail("ARTIFACT_PATH_ESCAPE") - if not path.is_file(): - _fail("ARTIFACT_FILE_MISSING") - if path.stat().st_size > int(POLICY["max_evidence_file_bytes"]): - _fail("ARTIFACT_FILE_TOO_LARGE") - return path - - -def _verify_artifact( - evidence_dir: Path, - obj: Any, - name: str, - *, - subject: str, - extra_keys: set[str] | None = None, -) -> Path: - item = _mapping(obj, f"{name.upper()}_INVALID") - expected_keys = {"path", "sha256", "status", "subject", "tool"} | ( - extra_keys or set() - ) - _exact_keys(item, expected_keys, f"{name.upper()}_SCHEMA_INVALID") - _tool(item.get("tool"), f"{name.upper()}_TOOL_INVALID") - if item.get("status") != "pass": - _fail(f"{name.upper()}_NOT_PASS") - if item.get("subject") != subject: - _fail(f"{name.upper()}_SUBJECT_MISMATCH") - path = _artifact_path(evidence_dir, item.get("path")) - expected = _hex(item.get("sha256"), f"{name.upper()}_SHA256_INVALID") - if _sha256(path) != expected: - _fail(f"{name.upper()}_HASH_MISMATCH") - return path - - -def _verify_provenance( - path: Path, image_digest: str, base_manifest_digest: str, item: dict[str, Any] -) -> None: - if item.get("predicateType") != SLSA_PREDICATE: - _fail("PROVENANCE_PREDICATE_UNSUPPORTED") - _bool(item.get("signatureVerified"), True, "PROVENANCE_SIGNATURE_NOT_VERIFIED") - try: - statement = json.loads(path.read_text()) - except Exception as exc: - raise EvidenceError("PROVENANCE_JSON_INVALID") from exc - if ( - not isinstance(statement, dict) - or statement.get("_type") != "https://in-toto.io/Statement/v1" - ): - _fail("PROVENANCE_STATEMENT_INVALID") - if statement.get("predicateType") != SLSA_PREDICATE: - _fail("PROVENANCE_STATEMENT_PREDICATE_MISMATCH") - wanted = image_digest.split(":", 1)[1] - subjects = statement.get("subject") - if not isinstance(subjects, list) or not any( - isinstance(s, dict) - and isinstance(s.get("digest"), dict) - and s["digest"].get("sha256") == wanted - for s in subjects - ): - _fail("PROVENANCE_SUBJECT_IMAGE_MISMATCH") - base_wanted = base_manifest_digest.split(":", 1)[1] - predicate = statement.get("predicate") - definition = ( - predicate.get("buildDefinition") if isinstance(predicate, dict) else None - ) - dependencies = ( - definition.get("resolvedDependencies") if isinstance(definition, dict) else None - ) - if not isinstance(dependencies, list) or not any( - isinstance(dep, dict) - and isinstance(dep.get("digest"), dict) - and dep["digest"].get("sha256") == base_wanted - for dep in dependencies - ): - _fail("PROVENANCE_BASE_MANIFEST_UNBOUND") - - -def _verify_redis(doc: dict[str, Any], now: datetime) -> None: - redis_policy = POLICY["redis"] - planes = _mapping(doc.get("redis"), "REDIS_EVIDENCE_MISSING") - _exact_keys( - planes, - {"rateLimit", "share", "contribution", "providerArtifactLifecycle"}, - "REDIS_SCHEMA_INVALID", - ) - for name in ("rateLimit", "share", "contribution", "providerArtifactLifecycle"): - item = _mapping(planes.get(name), f"REDIS_{name.upper()}_MISSING") - _exact_keys( - item, - { - "tlsVerified", - "nonDefaultIdentity", - "leastPrivilegeReviewed", - "persistenceVerified", - "replicationVerified", - "backupRestoreTestedAt", - }, - f"REDIS_{name.upper()}_SCHEMA_INVALID", - ) - if redis_policy["require_tls"]: - _bool(item.get("tlsVerified"), True, f"REDIS_{name.upper()}_TLS_UNVERIFIED") - if redis_policy["require_non_default_identity"]: - _bool( - item.get("nonDefaultIdentity"), - True, - f"REDIS_{name.upper()}_DEFAULT_IDENTITY", - ) - if redis_policy["require_least_privilege_review"]: - _bool( - item.get("leastPrivilegeReviewed"), - True, - f"REDIS_{name.upper()}_ACL_UNREVIEWED", - ) - if redis_policy["provider_artifact_lifecycle_require_replication"]: - _bool( - planes["providerArtifactLifecycle"].get("replicationVerified"), - True, - "REDIS_PROVIDERARTIFACTLIFECYCLE_REPLICATION_UNVERIFIED", - ) - for name, prefix in (("share", "share"), ("contribution", "contribution")): - item = planes[name] - if redis_policy[f"{prefix}_require_persistence"]: - _bool( - item.get("persistenceVerified"), - True, - f"REDIS_{name.upper()}_PERSISTENCE_UNVERIFIED", - ) - if redis_policy[f"{prefix}_require_replication"]: - _bool( - item.get("replicationVerified"), - True, - f"REDIS_{name.upper()}_REPLICATION_UNVERIFIED", - ) - tested = _parse_time( - item.get("backupRestoreTestedAt"), - f"REDIS_{name.upper()}_BACKUP_TIME_INVALID", - ) - if tested > now + timedelta(minutes=int(POLICY["max_clock_skew_minutes"])): - _fail(f"REDIS_{name.upper()}_BACKUP_TIME_FUTURE") - if now - tested > timedelta( - days=int(redis_policy["backup_restore_max_age_days"]) - ): - _fail(f"REDIS_{name.upper()}_BACKUP_RESTORE_STALE") - - -def _verify_redis_chaos_entry( # ruff: ignore[too-many-branches] - evidence_dir: Path, - entry: Any, - *, - redis_major: int, - mode: str, - source_revision: str, - source_tree_sha: str, - now: datetime, -) -> None: - label = f"redis{redis_major}_{mode}" - item = _mapping(entry, f"REDIS_CHAOS_{label.upper()}_MISSING") - _exact_keys( - item, - {"attestation", "signatureVerification"}, - f"REDIS_CHAOS_{label.upper()}_SCHEMA_INVALID", - ) - subject = "sha256:" + source_tree_sha - attestation = _mapping( - item.get("attestation"), f"REDIS_CHAOS_{label.upper()}_ATTESTATION_INVALID" - ) - attestation_path = _verify_artifact( - evidence_dir, - attestation, - f"redis_chaos_{label}_attestation", - subject=subject, - ) - try: - payload = json.loads(attestation_path.read_text(encoding="utf-8")) - except Exception as exc: - raise EvidenceError( - f"REDIS_CHAOS_{label.upper()}_ATTESTATION_JSON_INVALID" - ) from exc - payload = _mapping(payload, f"REDIS_CHAOS_{label.upper()}_ATTESTATION_ROOT_INVALID") - _exact_keys( - payload, - { - "schemaVersion", - "predicateType", - "generatedAt", - "subject", - "redis", - "result", - "tool", - }, - f"REDIS_CHAOS_{label.upper()}_ATTESTATION_SCHEMA_INVALID", - ) - if payload.get("schemaVersion") != 1: - _fail(f"REDIS_CHAOS_{label.upper()}_ATTESTATION_VERSION_UNSUPPORTED") - if payload.get("predicateType") != REDIS_CHAOS_PREDICATE: - _fail(f"REDIS_CHAOS_{label.upper()}_PREDICATE_MISMATCH") - generated = _parse_time( - payload.get("generatedAt"), f"REDIS_CHAOS_{label.upper()}_TIME_INVALID" - ) - skew = timedelta(minutes=int(POLICY["max_clock_skew_minutes"])) - if generated > now + skew: - _fail(f"REDIS_CHAOS_{label.upper()}_FROM_FUTURE") - if now - generated > timedelta(hours=int(POLICY["redis_chaos"]["max_age_hours"])): - _fail(f"REDIS_CHAOS_{label.upper()}_STALE") - payload_subject = _mapping( - payload.get("subject"), f"REDIS_CHAOS_{label.upper()}_SUBJECT_INVALID" - ) - _exact_keys( - payload_subject, - {"sourceRevision", "sourceTreeSha256"}, - f"REDIS_CHAOS_{label.upper()}_SUBJECT_SCHEMA_INVALID", - ) - if payload_subject.get("sourceRevision") != source_revision: - _fail(f"REDIS_CHAOS_{label.upper()}_REVISION_MISMATCH") - if payload_subject.get("sourceTreeSha256") != source_tree_sha: - _fail(f"REDIS_CHAOS_{label.upper()}_TREE_MISMATCH") - redis = _mapping(payload.get("redis"), f"REDIS_CHAOS_{label.upper()}_REDIS_INVALID") - _exact_keys( - redis, - {"major", "image", "mode"}, - f"REDIS_CHAOS_{label.upper()}_REDIS_SCHEMA_INVALID", - ) - if redis.get("major") != redis_major: - _fail(f"REDIS_CHAOS_{label.upper()}_MAJOR_MISMATCH") - if redis.get("mode") != mode: - _fail(f"REDIS_CHAOS_{label.upper()}_MODE_MISMATCH") - expected_image = POLICY["redis_chaos"][f"redis{redis_major}_image"] - if redis.get("image") != expected_image: - _fail(f"REDIS_CHAOS_{label.upper()}_IMAGE_MISMATCH") - result = _mapping( - payload.get("result"), f"REDIS_CHAOS_{label.upper()}_RESULT_INVALID" - ) - _exact_keys( - result, {"status"}, f"REDIS_CHAOS_{label.upper()}_RESULT_SCHEMA_INVALID" - ) - if result.get("status") != "pass": - _fail(f"REDIS_CHAOS_{label.upper()}_NOT_PASS") - _tool(payload.get("tool"), f"REDIS_CHAOS_{label.upper()}_TOOL_INVALID") - - attestation_sha = _sha256(attestation_path) - sig = _mapping( - item.get("signatureVerification"), - f"REDIS_CHAOS_{label.upper()}_SIGNATURE_INVALID", - ) - sig_path = _verify_artifact( - evidence_dir, - sig, - f"redis_chaos_{label}_signature", - subject="sha256:" + attestation_sha, - ) - try: - verification = json.loads(sig_path.read_text(encoding="utf-8")) - except Exception as exc: - raise EvidenceError( - f"REDIS_CHAOS_{label.upper()}_SIGNATURE_JSON_INVALID" - ) from exc - verification = _mapping( - verification, f"REDIS_CHAOS_{label.upper()}_SIGNATURE_ROOT_INVALID" - ) - _exact_keys( - verification, - { - "schemaVersion", - "verified", - "verifiedAt", - "attestationSha256", - "sourceRevision", - "signerIdentityVerified", - }, - f"REDIS_CHAOS_{label.upper()}_SIGNATURE_SCHEMA_INVALID", - ) - if verification.get("schemaVersion") != 1: - _fail(f"REDIS_CHAOS_{label.upper()}_SIGNATURE_VERSION_UNSUPPORTED") - _bool( - verification.get("verified"), - True, - f"REDIS_CHAOS_{label.upper()}_SIGNATURE_UNVERIFIED", - ) - _bool( - verification.get("signerIdentityVerified"), - True, - f"REDIS_CHAOS_{label.upper()}_SIGNER_IDENTITY_UNVERIFIED", - ) - if verification.get("attestationSha256") != attestation_sha: - _fail(f"REDIS_CHAOS_{label.upper()}_SIGNATURE_SUBJECT_MISMATCH") - if verification.get("sourceRevision") != source_revision: - _fail(f"REDIS_CHAOS_{label.upper()}_SIGNATURE_REVISION_MISMATCH") - verified_at = _parse_time( - verification.get("verifiedAt"), - f"REDIS_CHAOS_{label.upper()}_SIGNATURE_TIME_INVALID", - ) - if verified_at > now + skew: - _fail(f"REDIS_CHAOS_{label.upper()}_SIGNATURE_FROM_FUTURE") - if now - verified_at > timedelta(hours=int(POLICY["redis_chaos"]["max_age_hours"])): - _fail(f"REDIS_CHAOS_{label.upper()}_SIGNATURE_STALE") - - -def _verify_redis_chaos( - doc: dict[str, Any], - *, - evidence_dir: Path, - source_revision: str, - source_tree_sha: str, - now: datetime, -) -> None: - chaos = _mapping(doc.get("redisChaos"), "REDIS_CHAOS_EVIDENCE_MISSING") - _exact_keys(chaos, {"redis7", "redis8"}, "REDIS_CHAOS_SCHEMA_INVALID") - for major in (7, 8): - family = _mapping( - chaos.get(f"redis{major}"), f"REDIS_CHAOS_REDIS{major}_MISSING" - ) - _exact_keys( - family, - {"standalone", "cluster"}, - f"REDIS_CHAOS_REDIS{major}_SCHEMA_INVALID", - ) - for mode in ("standalone", "cluster"): - _verify_redis_chaos_entry( - evidence_dir, - family.get(mode), - redis_major=major, - mode=mode, - source_revision=source_revision, - source_tree_sha=source_tree_sha, - now=now, - ) - - -def _verify_logging(doc: dict[str, Any], now: datetime) -> None: - logging = _mapping(doc.get("logging"), "LOGGING_EVIDENCE_MISSING") - _exact_keys( - logging, - { - "requestBodyLogging", - "authorizationHeaderLogging", - "capabilityHeaderLogging", - "queryStringLogging", - "wafBodyCapture", - "apmBodyCapture", - "thirdPartyTelemetryExport", - "reviewedAt", - }, - "LOGGING_SCHEMA_INVALID", - ) - mapping = { - "requestBodyLogging": "request_body_logging", - "authorizationHeaderLogging": "authorization_header_logging", - "capabilityHeaderLogging": "capability_header_logging", - "queryStringLogging": "query_string_logging", - "wafBodyCapture": "waf_body_capture", - "apmBodyCapture": "apm_body_capture", - "thirdPartyTelemetryExport": "third_party_telemetry_export", - } - for evidence_name, policy_name in mapping.items(): - _bool( - logging.get(evidence_name), - bool(POLICY["logging"][policy_name]), - f"LOGGING_{evidence_name.upper()}_POLICY_MISMATCH", - ) - reviewed = _parse_time(logging.get("reviewedAt"), "LOGGING_REVIEW_TIME_INVALID") - if reviewed > now + timedelta(minutes=int(POLICY["max_clock_skew_minutes"])): - _fail("LOGGING_REVIEW_TIME_FUTURE") - if now - reviewed > timedelta(hours=int(POLICY["max_age_hours"])): - _fail("LOGGING_REVIEW_STALE") - - -def verify( # ruff: ignore[too-many-branches, undocumented-public-function] - evidence_file: Path, - *, - now: datetime | None = None, -) -> dict[str, Any]: - if evidence_file.is_symlink(): - _fail("EVIDENCE_SYMLINK_FORBIDDEN") - if not evidence_file.exists() or not evidence_file.is_file(): - _fail("EVIDENCE_FILE_MISSING") - if evidence_file.stat().st_size > int(POLICY["max_manifest_bytes"]): - _fail("EVIDENCE_MANIFEST_TOO_LARGE") - evidence_file = evidence_file.resolve() - try: - doc = json.loads(evidence_file.read_text(encoding="utf-8")) - except Exception as exc: - raise EvidenceError("EVIDENCE_JSON_INVALID") from exc - doc = _mapping(doc, "EVIDENCE_ROOT_INVALID") - _walk_values(doc) - _exact_keys( - doc, - { - "schemaVersion", - "release", - "source", - "image", - "artifacts", - "redis", - "redisChaos", - "logging", - "riskExceptions", - }, - "EVIDENCE_SCHEMA_FIELDS_INVALID", - ) - if doc.get("schemaVersion") != POLICY["evidence_schema_version"]: - _fail("EVIDENCE_SCHEMA_UNSUPPORTED") - release = _mapping(doc.get("release"), "RELEASE_BLOCK_MISSING") - _exact_keys( - release, - {"releaseId", "generatedAt", "expiresAt", "proxyVersion", "targetPlatform"}, - "RELEASE_SCHEMA_INVALID", - ) - if ( - not isinstance(release.get("releaseId"), str) - or RELEASE_ID.fullmatch(release["releaseId"]) is None - ): - _fail("RELEASE_ID_INVALID") - if release.get("targetPlatform") != POLICY["target_platform"]: - _fail("RELEASE_PLATFORM_MISMATCH") - if release.get("proxyVersion") != _current_proxy_version(): - _fail("RELEASE_PROXY_VERSION_MISMATCH") - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - generated = _parse_time(release.get("generatedAt"), "RELEASE_GENERATED_AT_INVALID") - expires = _parse_time(release.get("expiresAt"), "RELEASE_EXPIRES_AT_INVALID") - skew = timedelta(minutes=int(POLICY["max_clock_skew_minutes"])) - if generated > current + skew: - _fail("RELEASE_EVIDENCE_FROM_FUTURE") - if current - generated > timedelta(hours=int(POLICY["max_age_hours"])): - _fail("RELEASE_EVIDENCE_STALE") - if expires <= current or expires <= generated: - _fail("RELEASE_EVIDENCE_EXPIRED") - if expires - generated > timedelta(hours=int(POLICY["max_age_hours"])): - _fail("RELEASE_EVIDENCE_EXPIRY_TOO_LONG") - - source = _mapping(doc.get("source"), "SOURCE_BLOCK_MISSING") - _exact_keys( - source, - { - "requirementsLockSha256", - "pythonSbomSha256", - "runtimeSourceSha256", - "sourceTreeSha256", - "sourceRevision", - "baseImageIndexDigest", - "baseImageManifestDigest", - }, - "SOURCE_SCHEMA_INVALID", - ) - lock_sha = _hex(source.get("requirementsLockSha256"), "LOCK_SHA256_INVALID") - sbom_sha = _hex(source.get("pythonSbomSha256"), "PYTHON_SBOM_SHA256_INVALID") - if lock_sha != _sha256(ROOT / SUPPLY["lock_file"]): - _fail("LOCK_EVIDENCE_SOURCE_MISMATCH") - if sbom_sha != _sha256(ROOT / SUPPLY["sbom_file"]): - _fail("PYTHON_SBOM_EVIDENCE_SOURCE_MISMATCH") - runtime_sha = _hex( - source.get("runtimeSourceSha256"), "RUNTIME_SOURCE_SHA256_INVALID" - ) - if runtime_sha != _runtime_source_sha256(): - _fail("RUNTIME_SOURCE_EVIDENCE_MISMATCH") - source_tree_sha = _hex(source.get("sourceTreeSha256"), "SOURCE_TREE_SHA256_INVALID") - if source_tree_sha != source_tree_sha256(EXTENSION_ROOT): - _fail("SOURCE_TREE_EVIDENCE_MISMATCH") - source_revision = source.get("sourceRevision") - if ( - not isinstance(source_revision, str) - or SOURCE_REVISION.fullmatch(source_revision) is None - ): - _fail("SOURCE_REVISION_INVALID") - if source.get("baseImageIndexDigest") != SUPPLY["base_image"]["index_digest"]: - _fail("BASE_IMAGE_INDEX_MISMATCH") - base_manifest_digest = _digest( - source.get("baseImageManifestDigest"), "BASE_IMAGE_MANIFEST_DIGEST_INVALID" - ) - if base_manifest_digest == source.get("baseImageIndexDigest"): - _fail("BASE_IMAGE_MANIFEST_UNRESOLVED") - - image = _mapping(doc.get("image"), "IMAGE_BLOCK_MISSING") - _exact_keys(image, {"digest", "platform"}, "IMAGE_SCHEMA_INVALID") - image_digest = _digest(image.get("digest"), "IMAGE_DIGEST_INVALID") - if image.get("platform") != POLICY["target_platform"]: - _fail("IMAGE_PLATFORM_MISMATCH") - - artifacts = _mapping(doc.get("artifacts"), "ARTIFACTS_BLOCK_MISSING") - _exact_keys( - artifacts, - { - "dependencyScan", - "imageScan", - "imageSbom", - "provenance", - "signatureVerification", - }, - "ARTIFACTS_SCHEMA_INVALID", - ) - if POLICY["require_dependency_scan"]: - _verify_artifact( - evidence_file.parent, - artifacts.get("dependencyScan"), - "dependency_scan", - subject="sha256:" + lock_sha, - ) - if POLICY["require_image_vulnerability_scan"]: - _verify_artifact( - evidence_file.parent, - artifacts.get("imageScan"), - "image_scan", - subject=image_digest, - ) - if POLICY["require_full_image_sbom"]: - sbom_path = _verify_artifact( - evidence_file.parent, - artifacts.get("imageSbom"), - "image_sbom", - subject=image_digest, - ) - try: - image_sbom = json.loads(sbom_path.read_text()) - except Exception as exc: - raise EvidenceError("IMAGE_SBOM_JSON_INVALID") from exc - if ( - not isinstance(image_sbom, dict) - or image_sbom.get("bomFormat") != "CycloneDX" - ): - _fail("IMAGE_SBOM_NOT_CYCLONEDX") - if _version_tuple( - image_sbom.get("specVersion"), "IMAGE_SBOM_SPEC_VERSION_INVALID" - ) < _version_tuple( - POLICY["minimum_cyclonedx_spec_version"], "POLICY_CYCLONEDX_VERSION_INVALID" - ): - _fail("IMAGE_SBOM_SPEC_VERSION_TOO_OLD") - if POLICY["require_slsa_provenance"]: - provenance_item = _mapping(artifacts.get("provenance"), "PROVENANCE_INVALID") - provenance_path = _verify_artifact( - evidence_file.parent, - provenance_item, - "provenance", - subject=image_digest, - extra_keys={"predicateType", "signatureVerified"}, - ) - _verify_provenance( - provenance_path, image_digest, base_manifest_digest, provenance_item - ) - if POLICY["require_signature_verification"]: - _verify_artifact( - evidence_file.parent, - artifacts.get("signatureVerification"), - "signature_verification", - subject=image_digest, - ) - - _verify_redis(doc, current) - if POLICY["redis_chaos"]["require_signed_evidence"]: - _verify_redis_chaos( - doc, - evidence_dir=evidence_file.parent, - source_revision=source_revision, - source_tree_sha=source_tree_sha, - now=current, - ) - if POLICY["require_log_privacy_attestation"]: - _verify_logging(doc, current) - exceptions = doc.get("riskExceptions") - if not isinstance(exceptions, list): - _fail("RISK_EXCEPTIONS_INVALID") - if POLICY["forbid_unexpired_risk_exceptions"] and exceptions: - _fail("RISK_EXCEPTIONS_FORBIDDEN") - - return { - "ok": True, - "schema_version": doc["schemaVersion"], - "release_id": release["releaseId"], - "proxy_version": release["proxyVersion"], - "target_platform": release["targetPlatform"], - "image_digest": image_digest, - "runtime_source_sha256": runtime_sha, - "source_tree_sha256": source_tree_sha, - "source_revision": source_revision, - "redis_chaos": "signed-and-bound", - "evidence_sha256": _sha256(evidence_file), - "evidence_expires_at": release["expiresAt"], - "redis_planes": [ - "rateLimit", - "share", - "contribution", - "providerArtifactLifecycle", - ], - "logging_privacy": "verified", - } - - -def main( # ruff: ignore[undocumented-public-function] - argv: list[str] | None = None, -) -> int: - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("evidence", type=Path, help="Path to release-evidence.json") - args = parser.parse_args(argv) - try: - result = verify(args.evidence) - except EvidenceError as exc: - sys.stderr.write( - json.dumps({"ok": False, "code": str(exc)}, sort_keys=True) + "\n" - ) - return 2 - sys.stdout.write(json.dumps(result, sort_keys=True) + "\n") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/security/verify_release_gate.py b/security/verify_release_gate.py deleted file mode 100644 index 097345c5855e047adf9100aa68c933af93d3e511..0000000000000000000000000000000000000000 --- a/security/verify_release_gate.py +++ /dev/null @@ -1,54 +0,0 @@ -""" -One fail-closed command for source policy plus production evidence binding. -""" - -from __future__ import annotations - -import argparse -import json -import logging -import sys -from pathlib import Path - -HERE = Path(__file__).resolve().parent -if str(HERE) not in sys.path: - sys.path.insert(0, str(HERE)) - -import verify_release_evidence # noqa: E402 -import verify_supply_chain # noqa: E402 - -logger = logging.getLogger(__name__) - - -def verify( # ruff: ignore[undocumented-public-function] - evidence: Path, -) -> dict[str, object]: - source = verify_supply_chain.verify() - release = verify_release_evidence.verify(evidence) - return { - "ok": True, - "source_policy": source, - "release_evidence": release, - } - - -def main( # ruff: ignore[undocumented-public-function] - argv: list[str] | None = None, -) -> int: - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("evidence", type=Path) - args = parser.parse_args(argv) - try: - result = verify(args.evidence) - except Exception as exc: # ruff: ignore[blind-except] - code = getattr(exc, "args", ["RELEASE_GATE_FAILED"])[0] or "RELEASE_GATE_FAILED" - sys.stderr.write( - json.dumps({"ok": False, "code": str(code)}, sort_keys=True) + "\n" - ) - return 2 - sys.stdout.write(json.dumps(result, sort_keys=True) + "\n") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/security/verify_supply_chain.py b/security/verify_supply_chain.py deleted file mode 100644 index e41a9c5e02fde7ead7724285aa9b20a5a09a7f5b..0000000000000000000000000000000000000000 --- a/security/verify_supply_chain.py +++ /dev/null @@ -1,222 +0,0 @@ -""" -Offline structural verifier for the B38/B39 supply-chain policy. - -This verifier proves committed-file consistency and downgrade ratchets. It -cannot prove that today's advisory database has no newer finding; networked -release scanning remains a separate mandatory gate. -""" - -from __future__ import annotations - -import hashlib -import json -import logging -import re -import sys -from pathlib import Path - -import tomllib - -logger = logging.getLogger(__name__) - -ROOT = Path(__file__).resolve().parents[1] -POLICY = tomllib.loads((ROOT / "security/supply_chain_policy.toml").read_text()) -LOCK = ROOT / POLICY["lock_file"] -SBOM = ROOT / POLICY["sbom_file"] -DIRECT = ROOT / "requirements.txt" -DOCKER = ROOT / "Dockerfile" -IGNORE = ROOT / ".dockerignore" -COMPOSE = ROOT / "docker-compose.hardened.reference.yml" - -LINE_RE = re.compile( - r"^(?P[A-Za-z0-9_.-]+)==(?P[^\s]+) " - r"--hash=sha256:(?P[0-9a-f]{64})$" -) - - -def _version_tuple(text: str) -> tuple[int | str, ...]: - # Floors in this policy are simple numeric PEP-440 releases. Keep this - # parser intentionally narrow so unusual versions fail review rather than - # gaining surprising ordering semantics. - out: list[int | str] = [ - int(part) if part.isdigit() else part for part in re.split(r"[.-]", text) - ] - return tuple(out) - - -def _locked() -> dict[str, tuple[str, str]]: - result: dict[str, tuple[str, str]] = {} - for raw in LOCK.read_text().splitlines(): - line = raw.strip() - if not line or line.startswith("#"): - continue - match = LINE_RE.fullmatch(line) - if not match: - raise AssertionError(f"LOCK_LINE_NOT_EXACT_HASHED:{line}") - name = match.group("name").lower().replace("_", "-") - if name in result: - raise AssertionError(f"LOCK_DUPLICATE:{name}") - result[name] = (match.group("version"), match.group("digest")) - if not result: - raise AssertionError("LOCK_EMPTY") - return result - - -def verify() -> dict[str, object]: # ruff: ignore[too-many-branches] - """Verify.""" - locked = _locked() - - direct: dict[str, str] = {} - for raw in DIRECT.read_text().splitlines(): - line = raw.strip() - if not line or line.startswith("#"): - continue - if any(token in line for token in (">", "<", "~=", "[", "]", "@", ";")): - raise AssertionError(f"DIRECT_REQUIREMENT_NOT_EXACT_MINIMAL:{line}") - if line.count("==") != 1: - raise AssertionError(f"DIRECT_REQUIREMENT_NOT_EXACT:{line}") - name, version = line.split("==", 1) - direct[name.lower().replace("_", "-")] = version - for name, version in direct.items(): - if locked.get(name, (None,))[0] != version: - raise AssertionError(f"DIRECT_LOCK_DRIFT:{name}") - - for name, floor in POLICY["advisory_floors"].items(): - normalized = name.lower().replace("_", "-") - if normalized not in locked: - raise AssertionError(f"ADVISORY_FLOOR_PACKAGE_MISSING:{normalized}") - if _version_tuple(locked[normalized][0]) < _version_tuple(str(floor)): - raise AssertionError(f"ADVISORY_FLOOR_REGRESSION:{normalized}") - - docker = DOCKER.read_text() - image = POLICY["base_image"] - expected = f"{image['repository']}:{image['tag']}@{image['index_digest']}" - required_docker = ( - expected, - "--require-hashes", - "--only-binary=:all:", - "USER 1000:1000", - "DEPLOYMENT_PROFILE=strict", - "COPY --from=builder /opt/venv /opt/venv", - "FROM --platform=linux/amd64 ${PYTHON_IMAGE} AS builder", - "FROM --platform=linux/amd64 ${PYTHON_IMAGE} AS runtime", - ) - for marker in required_docker: - if marker not in docker: - raise AssertionError(f"DOCKER_HARDENING_MISSING:{marker}") - if "uvicorn[standard]" in docker or "fastapi[standard]" in docker: - raise AssertionError("DOCKER_BROAD_EXTRAS_FORBIDDEN") - - ignore_lines = { - line.strip() - for line in IGNORE.read_text().splitlines() - if line.strip() and not line.startswith("#") - } - if "*" not in ignore_lines: - raise AssertionError("DOCKERIGNORE_NOT_DENY_BY_DEFAULT") - for required in ("!requirements.lock", "!app.py", "!_utils/**"): - if required not in ignore_lines: - raise AssertionError(f"DOCKERIGNORE_RUNTIME_ALLOWLIST_MISSING:{required}") - for required in ( - "_utils/__pycache__/", - "_utils/**/__pycache__/", - "_utils/**/*.pyc", - "_utils/**/*.pyo", - ): - if required not in ignore_lines: - raise AssertionError( - f"DOCKERIGNORE_GENERATED_BYTECODE_EXCLUSION_MISSING:{required}" - ) - - compose = COMPOSE.read_text() - for marker in ( - "read_only: true", - 'user: "1000:1000"', - "no-new-privileges:true", - "cap_drop:", - "DEPLOYMENT_PROFILE: strict", - ): - if marker not in compose: - raise AssertionError(f"HARDENED_REFERENCE_MISSING:{marker}") - - release_policy = POLICY.get("release_evidence", {}) - for field in ( - "policy_file", - "example_file", - "verifier_file", - "combined_gate_file", - "redis_probe_file", - "subject_printer_file", - ): - raw = release_policy.get(field) - if not isinstance(raw, str) or not raw: - raise AssertionError(f"RELEASE_EVIDENCE_POLICY_MISSING:{field}") - path = ROOT / raw - if not path.is_file(): - raise AssertionError(f"RELEASE_EVIDENCE_FILE_MISSING:{field}") - evidence_policy = tomllib.loads((ROOT / release_policy["policy_file"]).read_text()) - if (int(evidence_policy.get("max_age_hours", 0)) <= 0) or int( - evidence_policy.get("max_age_hours", 999) - ) > ( - 72 # ruff: ignore[magic-value-comparison] - ): - raise AssertionError("RELEASE_EVIDENCE_MAX_AGE_UNSAFE") - manifest_cap = int(evidence_policy.get("max_manifest_bytes", 0)) - if manifest_cap <= 0 or manifest_cap > 1024 * 1024: - raise AssertionError("RELEASE_EVIDENCE_MANIFEST_CAP_UNSAFE") - minimum_cdx = str(evidence_policy.get("minimum_cyclonedx_spec_version", "")) - if not re.fullmatch(r"[0-9]+\.[0-9]+", minimum_cdx) or _version_tuple( - minimum_cdx - ) < _version_tuple("1.6"): - raise AssertionError("RELEASE_EVIDENCE_CYCLONEDX_FLOOR_UNSAFE") - if evidence_policy.get("target_platform") != POLICY.get("target_platform"): - raise AssertionError("RELEASE_EVIDENCE_PLATFORM_DRIFT") - if evidence_policy.get("forbid_unexpired_risk_exceptions") is not True: - raise AssertionError("RELEASE_EVIDENCE_RISK_EXCEPTION_BYPASS") - if evidence_policy.get("logging", {}).get("request_body_logging") is not False: - raise AssertionError("RELEASE_EVIDENCE_BODY_LOGGING_NOT_FORBIDDEN") - if ( - evidence_policy.get("logging", {}).get("third_party_telemetry_export") - is not False - ): - raise AssertionError("RELEASE_EVIDENCE_TELEMETRY_NOT_FORBIDDEN") - - sbom = json.loads(SBOM.read_text()) - components = { - c["name"].lower().replace("_", "-"): c for c in sbom.get("components", []) - } - if set(components) != set(locked): - missing = sorted(set(locked) - set(components)) - extra = sorted(set(components) - set(locked)) - raise AssertionError(f"SBOM_LOCK_SET_DRIFT:missing={missing}:extra={extra}") - for name, (version, digest) in locked.items(): - comp = components[name] - if comp.get("version") != version: - raise AssertionError(f"SBOM_VERSION_DRIFT:{name}") - hashes = { - h.get("content") - for h in comp.get("hashes", []) - if h.get("alg") == "SHA-256" - } - if digest not in hashes: - raise AssertionError(f"SBOM_HASH_DRIFT:{name}") - - return { - "ok": True, - "locked_packages": len(locked), - "direct_packages": len(direct), - "base_index_digest": image["index_digest"], - "lock_sha256": hashlib.sha256(LOCK.read_bytes()).hexdigest(), - "sbom_sha256": hashlib.sha256(SBOM.read_bytes()).hexdigest(), - "release_evidence_policy": True, - } - - -if __name__ == "__main__": - try: - sys.stdout.write(json.dumps(verify(), sort_keys=True) + "\n") - except Exception as exc: # ruff: ignore[blind-except] - sys.stderr.write( - json.dumps({"ok": False, "error": str(exc)}, sort_keys=True) + "\n" - ) - raise SystemExit(1) from exc diff --git a/security/witness_archive_health.py b/security/witness_archive_health.py deleted file mode 100644 index a137c6fbda633c8fab0a54990141100b2252939f..0000000000000000000000000000000000000000 --- a/security/witness_archive_health.py +++ /dev/null @@ -1,1822 +0,0 @@ -""" -Run 162: witness archive-health epochs and recover retention governance. - -Run 161 proves provider retention and independent challenge/read-back for the active -archive set. Run 162 adds two independent trust planes: - -* an out-of-band pinned witness root signs cross-auditor views of the exact Run 161 - archive-health state. Any observed split view fails closed; and -* a separately pinned recovery root can replace a compromised Run 161 retention root - without using the compromised root to authorize archive retirement. - -Private signing keys are never accepted by this tool. -""" - -from __future__ import annotations - -import argparse -import base64 -import hashlib -import json -import logging -import os -import re -import shutil -import subprocess -import sys -import tempfile -import threading -from datetime import datetime, timedelta, timezone -from pathlib import Path -from typing import Any, Callable, Iterable - -import tomllib -from cryptography.exceptions import InvalidSignature -from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PublicKey - -logger = logging.getLogger(__name__) - -try: - from . import audit_archive_retention as run161 -except (ImportError, ValueError) as exc: - import importlib.util - - _here = Path(__file__).resolve().parent - _spec = importlib.util.spec_from_file_location( - "_run161_health_for_witness", _here / "audit_archive_retention.py" - ) - if _spec is None or _spec.loader is None: - raise ImportError("audit_archive_retention.py") from exc - run161 = importlib.util.module_from_spec(_spec) - sys.modules[_spec.name] = run161 - _spec.loader.exec_module(run161) - -HERE = Path(__file__).resolve().parent -POLICY = tomllib.loads((HERE / "release_archive_witness_policy.toml").read_text()) -PREDICATE_TYPE = str(POLICY["predicate_type"]) -_HEX64 = re.compile(r"^[0-9a-f]{64}$") -_ID = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:/@+-]{0,511}$") -_CHUNK = 1024 * 1024 - - -# ── Verified-state document names ──────────────────────────────────────── -# -# One constant per release-evidence document. The values are the on-disk -# artifact names and are unchanged; only the repetition is removed. Reading -# a document by a name that says which document it is also keeps static -# analysis from reading the filename token 'trusted' as 'confidential': -# these files carry published Merkle roots and SHA-256 digests, which must -# stay in clear text for any third party to verify them. -_DOC_ARCHIVE_HEALTH_STATE = "trusted-archive-health-state.json" -_DOC_ARCHIVE_WITNESS_STATE = "trusted-archive-witness-state.json" - - -_WITNESS_OUTPUT_NAMES = { - "release-archive-witness-bundle.json", - _DOC_ARCHIVE_WITNESS_STATE, - "active-archive-witness-evidence.json", - "release-archive-witness-receipt.json", -} -_RECOVERY_OUTPUT_NAMES = { - "recovered-retention-root.json", - "retention-root-recovery-record.json", - "retention-root-recovery-receipt.json", -} - - -class ArchiveWitnessError(RuntimeError): - """Run 162 archive witness/recovery invariant failed.""" - - -def _fail(code: str) -> None: - raise ArchiveWitnessError(code) - - -def _canonical(value: Any) -> bytes: - return ( - json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False) - + "\n" - ).encode("utf-8") - - -def _sha_bytes(raw: bytes) -> str: - return hashlib.sha256(raw).hexdigest() - - -def _sha(path: Path) -> str: - h = hashlib.sha256() - with path.open("rb") as f: - for chunk in iter(lambda: f.read(_CHUNK), b""): - h.update(chunk) - return h.hexdigest() - - -def _write(path: Path, value: dict[str, Any]) -> None: - path.write_bytes(_canonical(value)) - - -def _loads(raw: bytes, code: str) -> dict[str, Any]: - def hook(pairs): - out: dict[str, Any] = {} - for k, v in pairs: - if k in out: - _fail(code + "_DUPLICATE_KEY") - out[k] = v - return out - - try: - value = json.loads(raw.decode("utf-8"), object_pairs_hook=hook) - except ArchiveWitnessError: - raise - except Exception as exc: - raise ArchiveWitnessError(code + "_JSON_INVALID") from exc - if not isinstance(value, dict): - _fail(code + "_SCHEMA_INVALID") - return value - - -def _read_json(path: Path, code: str) -> tuple[dict[str, Any], bytes]: - if path.is_symlink() or not path.is_file(): - _fail(code + "_INVALID") - if path.stat().st_size > int(POLICY["max_json_bytes"]): - _fail(code + "_TOO_LARGE") - raw = path.read_bytes() - doc = _loads(raw, code) - if raw != _canonical(doc): - _fail(code + "_NOT_CANONICAL") - return doc, raw - - -def _identity(value: Any, code: str) -> str: - if not isinstance(value, str): - _fail(code) - value = value.strip() - if ( - not value - or len(value) > 512 # ruff: ignore[magic-value-comparison] - or ".." in value - or "?" in value - or "#" in value - or "\x00" in value - ): - _fail(code) - if ( - any( - ord(c) < 32 # ruff: ignore[magic-value-comparison] - or ord(c) == 127 # ruff: ignore[magic-value-comparison] - for c in value - ) - or _ID.fullmatch(value) is None - ): - _fail(code) - return value - - -def _hex(value: Any, code: str) -> str: - if not isinstance(value, str) or _HEX64.fullmatch(value) is None: - _fail(code) - return value - - -def _size(value: Any, code: str) -> int: - if not isinstance(value, int) or isinstance(value, bool) or value < 0: - _fail(code) - return value - - -def _b64(value: Any, code: str, *, expected_len: int | None = None) -> bytes: - if not isinstance(value, str): - _fail(code) - try: - raw = base64.b64decode(value, validate=True) - except Exception as exc: - raise ArchiveWitnessError(code) from exc - if expected_len is not None and len(raw) != expected_len: - _fail(code) - return raw - - -def _dt(value: Any, code: str) -> datetime: - if not isinstance(value, str) or not value.endswith("Z"): - _fail(code) - try: - return datetime.fromisoformat(value[:-1] + "+00:00").astimezone(timezone.utc) - except ValueError as exc: - raise ArchiveWitnessError(code) from exc - - -def _ts(dt: datetime) -> str: - return ( - dt.astimezone(timezone.utc).isoformat(timespec="seconds").replace("+00:00", "Z") - ) - - -def _ed25519_verify( - public_text: str, signature_text: str, message: bytes, code: str -) -> None: - public = _b64(public_text, code + "_PUBLIC_KEY_INVALID", expected_len=32) - signature = _b64( - signature_text, code + "_SIGNATURE_ENCODING_INVALID", expected_len=64 - ) - try: - Ed25519PublicKey.from_public_bytes(public).verify(signature, message) - except (InvalidSignature, ValueError) as exc: - raise ArchiveWitnessError(code + "_SIGNATURE_INVALID") from exc - - -def _regular_dir(path: Path, code: str) -> Path: - if path.is_symlink() or not path.is_dir(): - _fail(code) - return path.resolve() - - -def _outside(path: Path, protected: Iterable[Path], code: str) -> Path: - target = path.expanduser().resolve() - for item in protected: - root = item.resolve() - if target == root or root in target.parents: - _fail(code) - return target - - -def _file_fingerprint(path: Path, code: str) -> tuple[str, int, int]: - if path.is_symlink() or not path.is_file(): - _fail(code) - st = path.stat() - return (_sha(path), st.st_size, st.st_mode & 0o7777) - - -def _dir_fingerprint(root: Path, code: str) -> dict[str, tuple[str, int, int]]: - root = _regular_dir(root, code) - out = {} - for p in root.iterdir(): - if p.is_symlink() or not p.is_file(): - _fail(code) - out[p.name] = _file_fingerprint(p, code) - return out - - -def _key(value: Any, code: str, *, channel: bool = False) -> dict[str, Any]: - expected = {"identity", "operator", "expires", "publicKey"} | ( - {"recoveryChannel"} if channel else set() - ) - if not isinstance(value, dict) or set(value) != expected: - _fail(code + "_SCHEMA_INVALID") - out = { - "identity": _identity(value.get("identity"), code + "_IDENTITY_INVALID"), - "operator": _identity(value.get("operator"), code + "_OPERATOR_INVALID"), - "expires": _ts(_dt(value.get("expires"), code + "_EXPIRES_INVALID")), - "publicKey": value.get("publicKey"), - } - _b64(out["publicKey"], code + "_PUBLIC_KEY_INVALID", expected_len=32) - if channel: - out["recoveryChannel"] = _identity( - value.get("recoveryChannel"), code + "_CHANNEL_INVALID" - ) - if value != out: - _fail(code + "_NOT_NORMALIZED") - return out - - -def _verify_threshold_root( # ruff: ignore[too-many-branches] - doc: dict[str, Any], - expected_sha256: str, - *, - root_type: str, - schema_key: str, - min_keys_key: str, - min_threshold_key: str, - min_operators_key: str, - now: datetime, - historical: bool = False, - channel: bool = False, -) -> dict[str, Any]: - if set(doc) != {"signed", "signatures"} or not isinstance(doc.get("signed"), dict): - _fail("ARCHIVE_WITNESS_ROOT_SCHEMA_INVALID") - s = doc["signed"] - expected = { - "_type", - "specVersion", - "schemaVersion", - "rootId", - "version", - "issuedAt", - "expires", - "threshold", - "selectedSignerKeyIds", - "keys", - } - if ( - set(s) != expected - or s.get("_type") != root_type - or s.get("specVersion") != str(POLICY["spec_version"]) - or s.get("schemaVersion") != int(POLICY[schema_key]) - ): - _fail("ARCHIVE_WITNESS_ROOT_SIGNED_SCHEMA_INVALID") - root_id = _identity(s.get("rootId"), "ARCHIVE_WITNESS_ROOT_ID_INVALID") - version = _size(s.get("version"), "ARCHIVE_WITNESS_ROOT_VERSION_INVALID") - if version != 1: - _fail("ARCHIVE_WITNESS_ROOT_VERSION_INVALID") - issued = _dt(s.get("issuedAt"), "ARCHIVE_WITNESS_ROOT_ISSUED_INVALID") - expires = _dt(s.get("expires"), "ARCHIVE_WITNESS_ROOT_EXPIRES_INVALID") - if expires <= issued or expires - issued > timedelta( - days=int(POLICY["max_root_lifetime_days"]) - ): - _fail("ARCHIVE_WITNESS_ROOT_LIFETIME_INVALID") - if not historical: - if issued > now + timedelta(minutes=int(POLICY["max_clock_skew_minutes"])): - _fail("ARCHIVE_WITNESS_ROOT_FROM_FUTURE") - if expires <= now: - _fail("ARCHIVE_WITNESS_ROOT_EXPIRED") - raw_keys = s.get("keys") - if not isinstance(raw_keys, dict) or not ( - int(POLICY[min_keys_key]) <= len(raw_keys) <= int(POLICY["max_key_count"]) - ): - _fail("ARCHIVE_WITNESS_ROOT_KEYS_INVALID") - keys = {} - for kid, value in raw_keys.items(): - kid = _identity( # ruff: ignore[redefined-loop-name] - kid, - "ARCHIVE_WITNESS_ROOT_KEY_ID_INVALID", - ) - keys[kid] = _key(value, "ARCHIVE_WITNESS_ROOT_KEY", channel=channel) - if ( - _dt(keys[kid]["expires"], "ARCHIVE_WITNESS_ROOT_KEY_EXPIRES_INVALID") - < expires - ): - _fail("ARCHIVE_WITNESS_ROOT_KEY_EXPIRES_BEFORE_ROOT") - threshold = _size(s.get("threshold"), "ARCHIVE_WITNESS_ROOT_THRESHOLD_INVALID") - if threshold < int(POLICY[min_threshold_key]) or threshold > len(keys): - _fail("ARCHIVE_WITNESS_ROOT_THRESHOLD_INVALID") - selected = s.get("selectedSignerKeyIds") - if not isinstance(selected, list) or len(selected) != threshold: - _fail("ARCHIVE_WITNESS_ROOT_SELECTED_INVALID") - selected = [_identity(x, "ARCHIVE_WITNESS_ROOT_SELECTED_INVALID") for x in selected] - if ( - selected != sorted(selected) - or len(set(selected)) != len(selected) - or any(x not in keys for x in selected) - ): - _fail("ARCHIVE_WITNESS_ROOT_SELECTED_INVALID") - if len({keys[x]["operator"] for x in selected}) < int(POLICY[min_operators_key]): - _fail("ARCHIVE_WITNESS_ROOT_OPERATOR_QUORUM_INVALID") - sigs = doc.get("signatures") - if not isinstance(sigs, list) or len(sigs) != len(selected): - _fail("ARCHIVE_WITNESS_ROOT_SIGNATURE_SET_INVALID") - sig_map = {} - for item in sigs: - if not isinstance(item, dict) or set(item) != {"keyId", "signature"}: - _fail("ARCHIVE_WITNESS_ROOT_SIGNATURE_SCHEMA_INVALID") - kid = _identity(item.get("keyId"), "ARCHIVE_WITNESS_ROOT_SIGNATURE_KEY_INVALID") - if kid in sig_map: - _fail("ARCHIVE_WITNESS_ROOT_SIGNATURE_DUPLICATE") - sig_map[kid] = item.get("signature") - if sorted(sig_map) != selected or sigs != sorted(sigs, key=lambda x: x["keyId"]): - _fail("ARCHIVE_WITNESS_ROOT_SIGNATURE_SET_INVALID") - signed_bytes = _canonical(s) - for kid in selected: - _ed25519_verify( - keys[kid]["publicKey"], sig_map[kid], signed_bytes, "ARCHIVE_WITNESS_ROOT" - ) - raw = _canonical(doc) - sha = _sha_bytes(raw) - if sha != _hex(expected_sha256, "ARCHIVE_WITNESS_ROOT_PIN_INVALID"): - _fail("ARCHIVE_WITNESS_ROOT_PIN_MISMATCH") - return { - "rootId": root_id, - "version": version, - "issued": issued, - "expires": expires, - "threshold": threshold, - "keys": keys, - "sha256": sha, - "doc": doc, - } - - -def _verify_run161( - *, - run160_dir: Path, - run161_dir: Path, - retention_root_path: Path, - retention_root_pin: str, - bootstrap_pin: str, - recovery_pin: str, - attestation_pins: list[str], - now: datetime, - historical: bool, -) -> dict[str, Any]: - try: - verified = run161.verify_archive_health( - run160_dir=run160_dir, - output_dir=run161_dir, - retention_root_path=retention_root_path, - expected_retention_root_sha256=retention_root_pin, - expected_bootstrap_root_sha256=bootstrap_pin, - expected_recovery_root_sha256=recovery_pin, - expected_attestation_root_sha256=attestation_pins, - now=now, - historical=historical, - ) - except Exception as exc: - raise ArchiveWitnessError("ARCHIVE_WITNESS_RUN161_INVALID:" + str(exc)) from exc - docs = {} - raws = {} - for name in sorted( - _WITNESS_OUTPUT_NAMES - | { - "release-archive-health-bundle.json", - _DOC_ARCHIVE_HEALTH_STATE, - "active-archive-health-evidence.json", - "release-archive-health-receipt.json", - } - ): - p = run161_dir / name - if p.exists() and name.startswith( - ( - "release-archive-health", - "trusted-archive-health", - "active-archive-health", - ) - ): - doc, raw = _read_json( - p, - "ARCHIVE_WITNESS_RUN161_" - + name.upper().replace("-", "_").replace(".", "_"), - ) - docs[name] = doc - raws[name] = raw - retention_doc, _ = _read_json(retention_root_path, "ARCHIVE_WITNESS_RETENTION_ROOT") - try: - retention_root = run161._verify_root( - retention_doc, retention_root_pin, now=now, historical=historical - ) - except Exception as exc: - raise ArchiveWitnessError( - "ARCHIVE_WITNESS_RETENTION_ROOT_INVALID:" + str(exc) - ) from exc - return { - "verified": verified, - "docs": docs, - "raws": raws, - "retention_root": retention_root, - } - - -def _run161_view(run161_docs: dict[str, dict[str, Any]]) -> dict[str, Any]: - state = run161_docs[_DOC_ARCHIVE_HEALTH_STATE] - active = run161_docs["active-archive-health-evidence.json"] - membership = active.get("membership", {}).get("signed", {}) - members = membership.get("members") - audits = active.get("audits") - if not isinstance(members, list) or not isinstance(audits, list): - _fail("ARCHIVE_WITNESS_RUN161_VIEW_INVALID") - by_audit = {x.get("archiveId"): x for x in audits if isinstance(x, dict)} - rows = [] - for m in members: - if not isinstance(m, dict): - _fail("ARCHIVE_WITNESS_RUN161_VIEW_INVALID") - aid = m.get("archiveId") - a = by_audit.get(aid) - if not isinstance(a, dict): - _fail("ARCHIVE_WITNESS_RUN161_VIEW_INVALID") - rows.append( - { - "archiveId": aid, - "archiveOperator": m.get("archiveOperator"), - "locator": m.get("locator"), - "immutability": m.get("immutability"), - "immutableVersionId": a.get("immutableVersionId"), - "retentionMode": a.get("retentionMode"), - "retentionUntil": a.get("retentionUntil"), - "legalHold": a.get("legalHold"), - "providerResponseSha256": a.get("providerResponseSha256"), - "auditorResponseSha256": a.get("auditorResponseSha256"), - } - ) - rows.sort(key=lambda x: x["archiveId"]) - return { - "run161Sequence": state.get("sequence"), - "healthChainHeadSha256": state.get("healthChainHeadSha256"), - "activeArchiveIds": state.get("activeArchiveIds"), - "archives": rows, - } - - -def _enforce_witness_separation( - root: dict[str, Any], - retention_root: dict[str, Any], - run161_docs: dict[str, dict[str, Any]], -) -> None: - active = run161_docs["active-archive-health-evidence.json"] - members = active.get("membership", {}).get("signed", {}).get("members") - if not isinstance(members, list): - _fail("ARCHIVE_WITNESS_AUTHORITY_PLANES_INVALID") - witness_ops = {v["operator"] for v in root["keys"].values()} - witness_pubs = {v["publicKey"] for v in root["keys"].values()} - other_ops = {v["operator"] for v in retention_root["keys"].values()} - other_pubs = {v["publicKey"] for v in retention_root["keys"].values()} - for m in members: - if not isinstance(m, dict): - _fail("ARCHIVE_WITNESS_AUTHORITY_PLANES_INVALID") - other_ops.update( - [m.get("archiveOperator"), m.get("auditorKey", {}).get("operator")] - ) - other_pubs.update( - [ - m.get("providerKey", {}).get("publicKey"), - m.get("auditorKey", {}).get("publicKey"), - ] - ) - other_ops.discard(None) - other_pubs.discard(None) - if witness_ops & other_ops or witness_pubs & other_pubs: - _fail("ARCHIVE_WITNESS_AUTHORITY_PLANES_OVERLAP") - - -def _challenge( # ruff: ignore[too-many-positional-arguments] - sequence: int, - previous_head: str | None, - run161_head: str, - view_sha: str, - state_sha: str, - active_sha: str, - selected_witness_key_ids: list[str], -) -> str: - return _sha_bytes( - _canonical( - { - "sequence": sequence, - "previousWitnessChainHeadSha256": previous_head, - "run161HealthChainHeadSha256": run161_head, - "viewSha256": view_sha, - "run161HealthStateSha256": state_sha, - "run161ActiveEvidenceSha256": active_sha, - "selectedWitnessKeyIds": selected_witness_key_ids, - } - ) - ) - - -def _command_adapter( - command: list[str], *, prefix: str -) -> Callable[[dict[str, Any]], dict[str, Any]]: - if not command: - _fail(prefix + "_COMMAND_EMPTY") - - def call(request: dict[str, Any]) -> dict[str, Any]: - proc = subprocess.Popen( # ruff: ignore[subprocess-without-shell-equals-true] - command, - stdin=subprocess.PIPE, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, - env={"PATH": os.defpath, "LC_ALL": "C", "LANG": "C"}, - ) - payload = _canonical(request) - out_parts = [] - err_parts = [] - total = [0] - lock = threading.Lock() - too_large = [False] - - def drain(stream, parts): - try: - while True: - chunk = stream.read(65536) - if not chunk: - break - with lock: - total[0] += len(chunk) - if total[0] > int(POLICY["max_adapter_output_bytes"]): - too_large[0] = True - proc.kill() - return - parts.append(chunk) - finally: - stream.close() - - assert proc.stdin # ruff: ignore[assert] - assert proc.stdout # ruff: ignore[assert] - assert proc.stderr # ruff: ignore[assert] - t1 = threading.Thread(target=drain, args=(proc.stdout, out_parts), daemon=True) - t2 = threading.Thread(target=drain, args=(proc.stderr, err_parts), daemon=True) - t1.start() - t2.start() - try: - proc.stdin.write(payload) - proc.stdin.close() - try: - rc = proc.wait(timeout=int(POLICY["adapter_timeout_seconds"])) - except subprocess.TimeoutExpired as exc: - proc.kill() - proc.wait() - raise ArchiveWitnessError(prefix + "_TIMEOUT") from exc - t1.join() - t2.join() - if too_large[0]: - _fail(prefix + "_OUTPUT_TOO_LARGE") - if rc != 0: - _fail(prefix + "_FAILED") - return _loads(b"".join(out_parts), prefix + "_RESPONSE") - finally: - if proc.poll() is None: - proc.kill() - proc.wait() - if t1.is_alive(): - t1.join() - if t2.is_alive(): - t2.join() - for stream in (proc.stdin, proc.stdout, proc.stderr): - try: # ruff: ignore[suppressible-exception] - stream.close() - except OSError: # ruff: ignore[try-except-in-loop] - pass - - return call - - -def command_witness(command: list[str]): # ruff: ignore[undocumented-public-function] - return _command_adapter(command, prefix="ARCHIVE_WITNESS_ADAPTER") - - -def _verify_witness_response( - value: dict[str, Any], - *, - key_id: str, - key: dict[str, Any], - sequence: int, - challenge: str, - expected_view: dict[str, Any], - expected_artifacts: dict[str, Any], - selected_witness_key_ids: list[str], - root_issued: datetime, - now: datetime, - historical: bool, -) -> dict[str, Any]: - if ( - set(value) != {"signed", "signature"} - or not isinstance(value.get("signed"), dict) - or not isinstance(value.get("signature"), dict) - ): - _fail("ARCHIVE_WITNESS_RESPONSE_SCHEMA_INVALID") - s = value["signed"] - expected = { - "schemaVersion", - "operation", - "sequence", - "witnessIdentity", - "witnessOperator", - "challenge", - "selectedWitnessKeyIds", - "run161Artifacts", - "view", - "readOnly", - "credentialsReused", - "observedAt", - } - if ( - set(s) != expected - or s.get("schemaVersion") != int(POLICY["witness_protocol_version"]) - or s.get("operation") != "witness-archive-health" - ): - _fail("ARCHIVE_WITNESS_RESPONSE_SIGNED_SCHEMA_INVALID") - if ( - s.get("sequence") != sequence - or s.get("witnessIdentity") != key["identity"] - or s.get("witnessOperator") != key["operator"] - or s.get("challenge") != challenge - or s.get("selectedWitnessKeyIds") != selected_witness_key_ids - or s.get("run161Artifacts") != expected_artifacts - ): - _fail("ARCHIVE_WITNESS_RESPONSE_BINDING_INVALID") - if s.get("readOnly") is not True or s.get("credentialsReused") is not False: - _fail("ARCHIVE_WITNESS_RESPONSE_AUTHORITY_INVALID") - if s.get("view") != expected_view: - _fail("ARCHIVE_WITNESS_SPLIT_VIEW_DETECTED") - observed = _dt(s.get("observedAt"), "ARCHIVE_WITNESS_OBSERVED_AT_INVALID") - if observed < root_issued - timedelta( - minutes=int(POLICY["max_clock_skew_minutes"]) - ): - _fail("ARCHIVE_WITNESS_OBSERVATION_BEFORE_ROOT") - if not historical and ( - observed > now + timedelta(minutes=int(POLICY["max_clock_skew_minutes"])) - or now - observed - > timedelta(minutes=int(POLICY["max_observation_age_minutes"])) - ): - _fail("ARCHIVE_WITNESS_OBSERVATION_NOT_FRESH") - sig = value["signature"] - if set(sig) != {"keyId", "signature"} or sig.get("keyId") != key_id: - _fail("ARCHIVE_WITNESS_SIGNATURE_BINDING_INVALID") - if _dt(key["expires"], "ARCHIVE_WITNESS_KEY_EXPIRES_INVALID") < observed: - _fail("ARCHIVE_WITNESS_KEY_EXPIRED") - _ed25519_verify( - key["publicKey"], sig.get("signature"), _canonical(s), "ARCHIVE_WITNESS" - ) - return {"observed": observed, "sha256": _sha_bytes(_canonical(value)), "doc": value} - - -def _event_head(previous: str | None, event_core: dict[str, Any]) -> str: - return _sha_bytes( - _canonical({"previousWitnessChainHeadSha256": previous, "event": event_core}) - ) - - -def _load_witness_output( - root: Path, -) -> tuple[dict[str, dict[str, Any]], dict[str, bytes]]: - root = _regular_dir(root, "ARCHIVE_WITNESS_OUTPUT_DIR_INVALID") - docs = {} - raws = {} - for name in _WITNESS_OUTPUT_NAMES: - doc, raw = _read_json( - root / name, - "ARCHIVE_WITNESS_OUTPUT_" - + name.upper().replace("-", "_").replace(".", "_"), - ) - docs[name] = doc - raws[name] = raw - if {p.name for p in root.iterdir()} != _WITNESS_OUTPUT_NAMES: - _fail("ARCHIVE_WITNESS_OUTPUT_ALLOWLIST_INVALID") - return docs, raws - - -def verify_witness_history( # ruff: ignore[too-many-branches, undocumented-public-function] - *, - run160_dir: Path, - run161_dir: Path, - retention_root_path: Path, - retention_root_pin: str, - witness_root_path: Path, - witness_root_pin: str, - bootstrap_pin: str, - recovery_pin: str, - attestation_pins: list[str], - output_dir: Path, - now: datetime | None = None, - historical: bool = False, - require_current_match: bool = True, -) -> dict[str, Any]: - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - run160_dir = _regular_dir(run160_dir, "ARCHIVE_WITNESS_RUN160_DIR_INVALID") - run161_dir = _regular_dir(run161_dir, "ARCHIVE_WITNESS_RUN161_DIR_INVALID") - root_doc, _ = _read_json(witness_root_path, "ARCHIVE_WITNESS_ROOT") - root = _verify_threshold_root( - root_doc, - witness_root_pin, - root_type="archive-health-witness-root", - schema_key="witness_root_schema_version", - min_keys_key="min_witness_keys", - min_threshold_key="min_witness_threshold", - min_operators_key="min_witness_operators", - now=current, - historical=historical, - ) - r161 = _verify_run161( - run160_dir=run160_dir, - run161_dir=run161_dir, - retention_root_path=retention_root_path, - retention_root_pin=retention_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - now=current, - historical=bool(historical), - ) - expected_view = _run161_view(r161["docs"]) - expected_view_sha = _sha_bytes(_canonical(expected_view)) - _enforce_witness_separation(root, r161["retention_root"], r161["docs"]) - docs, raws = _load_witness_output(output_dir) - bundle = docs["release-archive-witness-bundle.json"] - receipt = docs["release-archive-witness-receipt.json"] - if ( - set(bundle) - != {"schemaVersion", "predicateType", "status", "witnessRoot", "events"} - or bundle.get("schemaVersion") != int(POLICY["witness_bundle_schema_version"]) - or bundle.get("predicateType") != PREDICATE_TYPE - or bundle.get("status") != "archive-health-witness-history" - ): - _fail("ARCHIVE_WITNESS_BUNDLE_SCHEMA_INVALID") - if bundle.get("witnessRoot") != { - "sha256": root["sha256"], - "rootId": root["rootId"], - "version": root["version"], - }: - _fail("ARCHIVE_WITNESS_BUNDLE_ROOT_BINDING_INVALID") - if ( - set(receipt) != {"schemaVersion", "status", "events"} - or receipt.get("schemaVersion") != int(POLICY["witness_receipt_schema_version"]) - or receipt.get("status") != "archive-health-witnessed" - ): - _fail("ARCHIVE_WITNESS_RECEIPT_SCHEMA_INVALID") - events = bundle.get("events") - receipts = receipt.get("events") - if ( - not isinstance(events, list) - or not events - or not isinstance(receipts, list) - or len(receipts) != len(events) - ): - _fail("ARCHIVE_WITNESS_HISTORY_LENGTH_INVALID") - previous = None - last_time = None - for idx, (event, rec) in enumerate(zip(events, receipts), start=1): - if not isinstance(event, dict) or set(event) != { - "sequence", - "run161HealthChainHeadSha256", - "run161HealthStateArtifact", - "run161ActiveArtifact", - "view", - "viewSha256", - "challenge", - "witnessKeyIds", - "witnessResponseSha256s", - "witnessChainHeadSha256", - }: - _fail("ARCHIVE_WITNESS_EVENT_SCHEMA_INVALID") - if event.get("sequence") != idx: - _fail("ARCHIVE_WITNESS_SEQUENCE_INVALID") - if ( - not isinstance(rec, dict) - or set(rec) - != {"sequence", "run161HealthState", "run161ActiveEvidence", "responses"} - or rec.get("sequence") != idx - or not isinstance(rec.get("responses"), list) - ): - _fail("ARCHIVE_WITNESS_RECEIPT_EVENT_INVALID") - embedded_state = rec["run161HealthState"] - embedded_active = rec["run161ActiveEvidence"] - if not isinstance(embedded_state, dict) or not isinstance( - embedded_active, dict - ): - _fail("ARCHIVE_WITNESS_EMBEDDED_RUN161_INVALID") - state_raw = _canonical(embedded_state) - active_raw = _canonical(embedded_active) - if event.get("run161HealthStateArtifact") != { - "name": _DOC_ARCHIVE_HEALTH_STATE, - "sha256": _sha_bytes(state_raw), - "size": len(state_raw), - } or event.get("run161ActiveArtifact") != { - "name": "active-archive-health-evidence.json", - "sha256": _sha_bytes(active_raw), - "size": len(active_raw), - }: - _fail("ARCHIVE_WITNESS_RUN161_ARTIFACT_BINDING_INVALID") - embedded_view = _run161_view( - { - _DOC_ARCHIVE_HEALTH_STATE: embedded_state, - "active-archive-health-evidence.json": embedded_active, - } - ) - embedded_view_sha = _sha_bytes(_canonical(embedded_view)) - if ( - event.get("view") != embedded_view - or event.get("run161HealthChainHeadSha256") - != embedded_view["healthChainHeadSha256"] - or event.get("viewSha256") != embedded_view_sha - ): - _fail("ARCHIVE_WITNESS_EVENT_VIEW_BINDING_INVALID") - if ( - idx == len(events) - and require_current_match - and ( - embedded_state != r161["docs"][_DOC_ARCHIVE_HEALTH_STATE] - or embedded_active - != r161["docs"]["active-archive-health-evidence.json"] - ) - ): - _fail("ARCHIVE_WITNESS_CURRENT_RUN161_MISMATCH") - artifacts = { - "run161HealthStateArtifact": event["run161HealthStateArtifact"], - "run161ActiveArtifact": event["run161ActiveArtifact"], - } - selected_event = event.get("witnessKeyIds") - if ( - not isinstance(selected_event, list) - or selected_event != sorted(selected_event) - or len(set(selected_event)) != len(selected_event) - ): - _fail("ARCHIVE_WITNESS_EVENT_WITNESS_SET_INVALID") - challenge = _challenge( - idx, - previous, - embedded_view["healthChainHeadSha256"], - embedded_view_sha, - event["run161HealthStateArtifact"]["sha256"], - event["run161ActiveArtifact"]["sha256"], - selected_event, - ) - if event.get("challenge") != challenge: - _fail("ARCHIVE_WITNESS_CHALLENGE_INVALID") - responses = rec["responses"] - key_ids = [] - response_hashes = [] - observed_times = [] - operators = set() - for item in responses: - if not isinstance(item, dict) or set(item) != {"keyId", "response"}: - _fail("ARCHIVE_WITNESS_RECEIPT_RESPONSE_INVALID") - kid = _identity(item.get("keyId"), "ARCHIVE_WITNESS_KEY_ID_INVALID") - if kid not in root["keys"] or kid in key_ids: - _fail("ARCHIVE_WITNESS_KEY_ID_INVALID") - verified = _verify_witness_response( - item["response"], - key_id=kid, - key=root["keys"][kid], - sequence=idx, - challenge=challenge, - expected_view=embedded_view, - expected_artifacts=artifacts, - selected_witness_key_ids=selected_event, - root_issued=root["issued"], - now=current, - historical=True if idx < len(events) else historical, - ) - key_ids.append(kid) - response_hashes.append(verified["sha256"]) - observed_times.append(verified["observed"]) - operators.add(root["keys"][kid]["operator"]) - if len(key_ids) < root["threshold"] or len(operators) < int( - POLICY["min_witness_operators"] - ): - _fail("ARCHIVE_WITNESS_QUORUM_INVALID") - if key_ids != sorted(key_ids): - _fail("ARCHIVE_WITNESS_RESPONSES_NOT_SORTED") - if ( - event.get("witnessKeyIds") != key_ids - or event.get("witnessResponseSha256s") != response_hashes - ): - _fail("ARCHIVE_WITNESS_RESPONSE_BINDING_INVALID") - core = {k: event[k] for k in event if k != "witnessChainHeadSha256"} - head = _event_head(previous, core) - if event.get("witnessChainHeadSha256") != head: - _fail("ARCHIVE_WITNESS_CHAIN_HEAD_INVALID") - if last_time is not None and min(observed_times) - last_time > timedelta( - days=int(POLICY["maximum_witness_interval_days"]) - ): - _fail("ARCHIVE_WITNESS_INTERVAL_EXCEEDED") - last_time = max(observed_times) - previous = head - state = docs[_DOC_ARCHIVE_WITNESS_STATE] - active = docs["active-archive-witness-evidence.json"] - bundle_item = { - "name": "release-archive-witness-bundle.json", - "sha256": _sha_bytes(raws["release-archive-witness-bundle.json"]), - "size": len(raws["release-archive-witness-bundle.json"]), - } - expected_state = { - "schemaVersion": int(POLICY["witness_state_schema_version"]), - "status": "trusted-archive-witness", - "sequence": len(events), - "witnessChainHeadSha256": previous, - "run161HealthChainHeadSha256": events[-1]["run161HealthChainHeadSha256"], - "bundleArtifact": bundle_item, - "activeArchiveIds": events[-1]["view"]["activeArchiveIds"], - } - expected_active = { - "schemaVersion": int(POLICY["witness_active_schema_version"]), - "sequence": len(events), - "witnessChainHeadSha256": previous, - "challenge": events[-1]["challenge"], - "view": events[-1]["view"], - "witnessKeyIds": events[-1]["witnessKeyIds"], - } - if state != expected_state or active != expected_active: - _fail("ARCHIVE_WITNESS_STATE_BINDING_INVALID") - if ( - not historical - and last_time is not None - and current - last_time - > timedelta(days=int(POLICY["maximum_witness_interval_days"])) - ): - _fail("ARCHIVE_WITNESS_ACTIVE_EPOCH_STALE") - return { - "ok": True, - "sequence": len(events), - "witness_chain_head_sha256": previous, - "witness_count": len(events[-1]["witnessKeyIds"]), - "view": expected_view, - } - - -def witness_archive_health( # ruff: ignore[undocumented-public-function] - *, - run160_dir: Path, - run161_dir: Path, - retention_root_path: Path, - retention_root_pin: str, - witness_root_path: Path, - witness_root_pin: str, - bootstrap_pin: str, - recovery_pin: str, - attestation_pins: list[str], - output_dir: Path, - witnesses: list[tuple[str, Callable[[dict[str, Any]], dict[str, Any]]]], - previous_output_dir: Path | None = None, - now: datetime | None = None, -) -> dict[str, Any]: - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - run160_dir = _regular_dir(run160_dir, "ARCHIVE_WITNESS_RUN160_DIR_INVALID") - run161_dir = _regular_dir(run161_dir, "ARCHIVE_WITNESS_RUN161_DIR_INVALID") - root_doc, _ = _read_json(witness_root_path, "ARCHIVE_WITNESS_ROOT") - root = _verify_threshold_root( - root_doc, - witness_root_pin, - root_type="archive-health-witness-root", - schema_key="witness_root_schema_version", - min_keys_key="min_witness_keys", - min_threshold_key="min_witness_threshold", - min_operators_key="min_witness_operators", - now=current, - ) - r161 = _verify_run161( - run160_dir=run160_dir, - run161_dir=run161_dir, - retention_root_path=retention_root_path, - retention_root_pin=retention_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - now=current, - historical=False, - ) - _enforce_witness_separation(root, r161["retention_root"], r161["docs"]) - expected_view = _run161_view(r161["docs"]) - view_sha = _sha_bytes(_canonical(expected_view)) - run161_head = expected_view["healthChainHeadSha256"] - state_doc = r161["docs"][_DOC_ARCHIVE_HEALTH_STATE] - active_doc = r161["docs"]["active-archive-health-evidence.json"] - state_raw = _canonical(state_doc) - active_raw = _canonical(active_doc) - state_item = { - "name": _DOC_ARCHIVE_HEALTH_STATE, - "sha256": _sha_bytes(state_raw), - "size": len(state_raw), - } - active_item = { - "name": "active-archive-health-evidence.json", - "sha256": _sha_bytes(active_raw), - "size": len(active_raw), - } - artifacts = { - "run161HealthStateArtifact": state_item, - "run161ActiveArtifact": active_item, - } - previous_bundle = None - previous_receipts = [] - previous_head = None - protected = [ - run160_dir, - run161_dir, - retention_root_path.resolve(), - witness_root_path.resolve(), - ] - if previous_output_dir is not None: - previous_output_dir = _regular_dir( - previous_output_dir, "ARCHIVE_WITNESS_PREVIOUS_DIR_INVALID" - ) - protected.append(previous_output_dir) - prev = verify_witness_history( - run160_dir=run160_dir, - run161_dir=run161_dir, - retention_root_path=retention_root_path, - retention_root_pin=retention_root_pin, - witness_root_path=witness_root_path, - witness_root_pin=witness_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - output_dir=previous_output_dir, - now=current, - historical=True, - require_current_match=False, - ) - pdocs, _ = _load_witness_output(previous_output_dir) - previous_bundle = pdocs["release-archive-witness-bundle.json"] - previous_receipts = list( - pdocs["release-archive-witness-receipt.json"]["events"] - ) - previous_head = prev["witness_chain_head_sha256"] - target = _outside(output_dir, protected, "ARCHIVE_WITNESS_OUTPUT_OVERLAPS_INPUT") - if target.exists(): - _fail("ARCHIVE_WITNESS_OUTPUT_EXISTS") - adapters = { - _identity(k, "ARCHIVE_WITNESS_ADAPTER_KEY_INVALID"): a for k, a in witnesses - } - if len(adapters) != len(witnesses) or any(k not in root["keys"] for k in adapters): - _fail("ARCHIVE_WITNESS_ADAPTER_SET_INVALID") - if len(adapters) < root["threshold"] or len( - {root["keys"][k]["operator"] for k in adapters} - ) < int(POLICY["min_witness_operators"]): - _fail("ARCHIVE_WITNESS_QUORUM_INVALID") - sequence = 1 if previous_bundle is None else len(previous_bundle["events"]) + 1 - selected_witness_key_ids = sorted(adapters) - challenge = _challenge( - sequence, - previous_head, - run161_head, - view_sha, - state_item["sha256"], - active_item["sha256"], - selected_witness_key_ids, - ) - run160_fp = _dir_fingerprint(run160_dir, "ARCHIVE_WITNESS_RUN160_INPUT_DRIFT") - run161_fp = _dir_fingerprint(run161_dir, "ARCHIVE_WITNESS_RUN161_INPUT_DRIFT") - rr_fp = _file_fingerprint( - retention_root_path.resolve(), "ARCHIVE_WITNESS_RETENTION_ROOT_DRIFT" - ) - wr_fp = _file_fingerprint( - witness_root_path.resolve(), "ARCHIVE_WITNESS_WITNESS_ROOT_DRIFT" - ) - prev_fp = ( - _dir_fingerprint(previous_output_dir, "ARCHIVE_WITNESS_PREVIOUS_DRIFT") - if previous_output_dir is not None - else None - ) - responses = [] - hashes = [] - request = { - "schemaVersion": int(POLICY["witness_protocol_version"]), - "operation": "witness-archive-health", - "sequence": sequence, - "challenge": challenge, - "selectedWitnessKeyIds": selected_witness_key_ids, - "run161Artifacts": artifacts, - "expectedView": expected_view, - } - for kid in sorted(adapters): - raw = adapters[kid](request) - verified = _verify_witness_response( - raw, - key_id=kid, - key=root["keys"][kid], - sequence=sequence, - challenge=challenge, - expected_view=expected_view, - expected_artifacts=artifacts, - selected_witness_key_ids=selected_witness_key_ids, - root_issued=root["issued"], - now=current, - historical=False, - ) - responses.append({"keyId": kid, "response": raw}) - hashes.append(verified["sha256"]) - if ( - _dir_fingerprint(run160_dir, "ARCHIVE_WITNESS_RUN160_INPUT_DRIFT") != run160_fp - or _dir_fingerprint(run161_dir, "ARCHIVE_WITNESS_RUN161_INPUT_DRIFT") - != run161_fp - or _file_fingerprint( - retention_root_path.resolve(), "ARCHIVE_WITNESS_RETENTION_ROOT_DRIFT" - ) - != rr_fp - or _file_fingerprint( - witness_root_path.resolve(), "ARCHIVE_WITNESS_WITNESS_ROOT_DRIFT" - ) - != wr_fp - ): - _fail("ARCHIVE_WITNESS_INPUT_DRIFT_DETECTED") - if ( - previous_output_dir is not None - and _dir_fingerprint(previous_output_dir, "ARCHIVE_WITNESS_PREVIOUS_DRIFT") - != prev_fp - ): - _fail("ARCHIVE_WITNESS_INPUT_DRIFT_DETECTED") - core = { - "sequence": sequence, - "run161HealthChainHeadSha256": run161_head, - "viewSha256": view_sha, - "challenge": challenge, - "witnessKeyIds": [x["keyId"] for x in responses], - "witnessResponseSha256s": hashes, - } - core.update( - { - "run161HealthStateArtifact": state_item, - "run161ActiveArtifact": active_item, - "view": expected_view, - } - ) - head = _event_head(previous_head, core) - event = dict(core, witnessChainHeadSha256=head) - if previous_bundle is None: - bundle = { - "schemaVersion": int(POLICY["witness_bundle_schema_version"]), - "predicateType": PREDICATE_TYPE, - "status": "archive-health-witness-history", - "witnessRoot": { - "sha256": root["sha256"], - "rootId": root["rootId"], - "version": root["version"], - }, - "events": [event], - } - else: - bundle = json.loads(json.dumps(previous_bundle)) - bundle["events"].append(event) - receipt = { - "schemaVersion": int(POLICY["witness_receipt_schema_version"]), - "status": "archive-health-witnessed", - "events": [ - *previous_receipts, - { - "sequence": sequence, - "run161HealthState": state_doc, - "run161ActiveEvidence": active_doc, - "responses": responses, - }, - ], - } - bundle_raw = _canonical(bundle) - bundle_item = { - "name": "release-archive-witness-bundle.json", - "sha256": _sha_bytes(bundle_raw), - "size": len(bundle_raw), - } - state = { - "schemaVersion": int(POLICY["witness_state_schema_version"]), - "status": "trusted-archive-witness", - "sequence": sequence, - "witnessChainHeadSha256": head, - "run161HealthChainHeadSha256": run161_head, - "bundleArtifact": bundle_item, - "activeArchiveIds": expected_view["activeArchiveIds"], - } - active = { - "schemaVersion": int(POLICY["witness_active_schema_version"]), - "sequence": sequence, - "witnessChainHeadSha256": head, - "challenge": challenge, - "view": expected_view, - "witnessKeyIds": [x["keyId"] for x in responses], - } - parent = target.parent - parent.mkdir(parents=True, exist_ok=True) - stage = Path(tempfile.mkdtemp(prefix=".run162-witness-", dir=parent)) - try: - _write(stage / "release-archive-witness-bundle.json", bundle) - _write(stage / _DOC_ARCHIVE_WITNESS_STATE, state) - _write(stage / "active-archive-witness-evidence.json", active) - _write(stage / "release-archive-witness-receipt.json", receipt) - verify_witness_history( - run160_dir=run160_dir, - run161_dir=run161_dir, - retention_root_path=retention_root_path, - retention_root_pin=retention_root_pin, - witness_root_path=witness_root_path, - witness_root_pin=witness_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - output_dir=stage, - now=current, - ) - stage.rename(target) - except Exception: - shutil.rmtree(stage, ignore_errors=True) - raise - return verify_witness_history( - run160_dir=run160_dir, - run161_dir=run161_dir, - retention_root_path=retention_root_path, - retention_root_pin=retention_root_pin, - witness_root_path=witness_root_path, - witness_root_pin=witness_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - output_dir=target, - now=current, - ) - - -def _recovery_subject( # ruff: ignore[too-many-branches, too-many-positional-arguments] - doc: dict[str, Any], - recovery_root: dict[str, Any], - old_root: dict[str, Any], - new_root: dict[str, Any], - active_state: dict[str, Any], - now: datetime, - historical: bool = False, -) -> dict[str, Any]: - expected = { - "_type", - "specVersion", - "schemaVersion", - "recoveryId", - "issuedAt", - "recoveryRootSha256", - "oldRetentionRootSha256", - "newRetentionRootSha256", - "run161HealthChainHeadSha256", - "activeArchiveIds", - "compromisedOldKeyIds", - "selectedRecoveryKeyIds", - "retirementAuthorizedArchiveIds", - } - if ( - set(doc) != expected - or doc.get("_type") != "archive-retention-root-recovery" - or doc.get("specVersion") != str(POLICY["spec_version"]) - or doc.get("schemaVersion") != int(POLICY["recovery_record_schema_version"]) - ): - _fail("ARCHIVE_WITNESS_RECOVERY_SUBJECT_SCHEMA_INVALID") - _identity(doc.get("recoveryId"), "ARCHIVE_WITNESS_RECOVERY_ID_INVALID") - issued = _dt(doc.get("issuedAt"), "ARCHIVE_WITNESS_RECOVERY_ISSUED_INVALID") - if not historical and issued > now + timedelta( - minutes=int(POLICY["max_clock_skew_minutes"]) - ): - _fail("ARCHIVE_WITNESS_RECOVERY_FROM_FUTURE") - if issued < recovery_root["issued"] or issued > recovery_root["expires"]: - _fail("ARCHIVE_WITNESS_RECOVERY_OUTSIDE_RECOVERY_ROOT_LIFETIME") - if issued < new_root["issued"] or issued > new_root["expires"]: - _fail("ARCHIVE_WITNESS_RECOVERY_OUTSIDE_NEW_ROOT_LIFETIME") - if ( - doc.get("recoveryRootSha256") != recovery_root["sha256"] - or doc.get("oldRetentionRootSha256") != old_root["sha256"] - or doc.get("newRetentionRootSha256") != new_root["sha256"] - ): - _fail("ARCHIVE_WITNESS_RECOVERY_ROOT_BINDING_INVALID") - if doc.get("run161HealthChainHeadSha256") != active_state.get( - "healthChainHeadSha256" - ) or doc.get("activeArchiveIds") != active_state.get("activeArchiveIds"): - _fail("ARCHIVE_WITNESS_RECOVERY_HEALTH_BINDING_INVALID") - if doc.get("retirementAuthorizedArchiveIds") != []: - _fail("ARCHIVE_WITNESS_RECOVERY_RETIREMENT_FORBIDDEN") - compromised = doc.get("compromisedOldKeyIds") - if ( - not isinstance(compromised, list) - or not compromised - or compromised != sorted(compromised) - or len(set(compromised)) != len(compromised) - or any(k not in old_root["keys"] for k in compromised) - ): - _fail("ARCHIVE_WITNESS_RECOVERY_COMPROMISED_KEYS_INVALID") - selected = doc.get("selectedRecoveryKeyIds") - if ( - not isinstance(selected, list) - or len(selected) != recovery_root["threshold"] - or selected != sorted(selected) - or len(set(selected)) != len(selected) - or any(k not in recovery_root["keys"] for k in selected) - ): - _fail("ARCHIVE_WITNESS_RECOVERY_SELECTED_INVALID") - if len({recovery_root["keys"][k]["operator"] for k in selected}) < int( - POLICY["min_recovery_operators"] - ): - _fail("ARCHIVE_WITNESS_RECOVERY_OPERATOR_QUORUM_INVALID") - if len({recovery_root["keys"][k]["recoveryChannel"] for k in selected}) < int( - POLICY["min_recovery_channels"] - ): - _fail("ARCHIVE_WITNESS_RECOVERY_CHANNEL_QUORUM_INVALID") - if new_root["rootId"] != old_root["rootId"]: - _fail("ARCHIVE_WITNESS_RECOVERY_ROOT_ID_MISMATCH") - old_pubs = {v["publicKey"] for v in old_root["keys"].values()} - new_pubs = {v["publicKey"] for v in new_root["keys"].values()} - recovery_pubs = {v["publicKey"] for v in recovery_root["keys"].values()} - old_ops = {v["operator"] for v in old_root["keys"].values()} - new_ops = {v["operator"] for v in new_root["keys"].values()} - recovery_ops = {v["operator"] for v in recovery_root["keys"].values()} - if ( - set(old_root["keys"]) & set(new_root["keys"]) - or old_pubs & new_pubs - or old_ops & new_ops - or recovery_pubs & (old_pubs | new_pubs) - or recovery_ops & (old_ops | new_ops) - ): - _fail("ARCHIVE_WITNESS_RECOVERY_AUTHORITY_SEPARATION_INVALID") - if any(k in new_root["keys"] for k in compromised): - _fail("ARCHIVE_WITNESS_RECOVERY_COMPROMISED_KEY_REINTRODUCED") - return {"issued": issued, "selected": selected, "compromised": compromised} - - -def verify_retention_root_recovery( # ruff: ignore[too-many-branches, undocumented-public-function] - *, - run160_dir: Path, - run161_dir: Path, - old_retention_root_path: Path, - old_retention_root_pin: str, - recovery_root_path: Path, - recovery_root_pin: str, - bootstrap_pin: str, - recovery_pin: str, - attestation_pins: list[str], - output_dir: Path, - now: datetime | None = None, - historical: bool = False, -) -> dict[str, Any]: - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - run160_dir = _regular_dir(run160_dir, "ARCHIVE_WITNESS_RUN160_DIR_INVALID") - run161_dir = _regular_dir(run161_dir, "ARCHIVE_WITNESS_RUN161_DIR_INVALID") - output_dir = _regular_dir(output_dir, "ARCHIVE_WITNESS_RECOVERY_OUTPUT_DIR_INVALID") - if {p.name for p in output_dir.iterdir()} != _RECOVERY_OUTPUT_NAMES: - _fail("ARCHIVE_WITNESS_RECOVERY_OUTPUT_ALLOWLIST_INVALID") - old_doc, _ = _read_json( - old_retention_root_path, "ARCHIVE_WITNESS_OLD_RETENTION_ROOT" - ) - try: - old = run161._verify_root( - old_doc, old_retention_root_pin, now=current, historical=True - ) - except Exception as exc: - raise ArchiveWitnessError( - "ARCHIVE_WITNESS_OLD_RETENTION_ROOT_INVALID:" + str(exc) - ) from exc - new_doc, new_raw = _read_json( - output_dir / "recovered-retention-root.json", "ARCHIVE_WITNESS_RECOVERED_ROOT" - ) - new_sha = _sha_bytes(new_raw) - try: - new = run161._verify_root(new_doc, new_sha, now=current, historical=historical) - except Exception as exc: - raise ArchiveWitnessError( - "ARCHIVE_WITNESS_NEW_RETENTION_ROOT_INVALID:" + str(exc) - ) from exc - recovery_doc, _ = _read_json(recovery_root_path, "ARCHIVE_WITNESS_RECOVERY_ROOT") - rr = _verify_threshold_root( - recovery_doc, - recovery_root_pin, - root_type="archive-retention-recovery-root", - schema_key="recovery_root_schema_version", - min_keys_key="min_recovery_keys", - min_threshold_key="min_recovery_threshold", - min_operators_key="min_recovery_operators", - now=current, - historical=historical, - channel=True, - ) - _verify_run161( - run160_dir=run160_dir, - run161_dir=run161_dir, - retention_root_path=old_retention_root_path, - retention_root_pin=old_retention_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - now=current, - historical=True, - ) - active, _ = _read_json( - run161_dir / _DOC_ARCHIVE_HEALTH_STATE, "ARCHIVE_WITNESS_RUN161_STATE" - ) - record, record_raw = _read_json( - output_dir / "retention-root-recovery-record.json", - "ARCHIVE_WITNESS_RECOVERY_RECORD", - ) - receipt, _ = _read_json( - output_dir / "retention-root-recovery-receipt.json", - "ARCHIVE_WITNESS_RECOVERY_RECEIPT", - ) - if ( - set(record) - != { - "schemaVersion", - "status", - "subject", - "signatures", - "oldRetentionRoot", - "newRetentionRoot", - "recoveryRoot", - } - or record.get("schemaVersion") != int(POLICY["recovery_record_schema_version"]) - or record.get("status") != "retention-root-recovered" - ): - _fail("ARCHIVE_WITNESS_RECOVERY_RECORD_SCHEMA_INVALID") - subject = record.get("subject") - parsed = _recovery_subject( - subject, rr, old, new, active, current, historical=historical - ) - sigs = record.get("signatures") - if not isinstance(sigs, list) or len(sigs) != len(parsed["selected"]): - _fail("ARCHIVE_WITNESS_RECOVERY_SIGNATURE_SET_INVALID") - sig_map = {} - for item in sigs: - if not isinstance(item, dict) or set(item) != {"keyId", "channel", "signature"}: - _fail("ARCHIVE_WITNESS_RECOVERY_SIGNATURE_SCHEMA_INVALID") - kid = _identity( - item.get("keyId"), "ARCHIVE_WITNESS_RECOVERY_SIGNATURE_KEY_INVALID" - ) - if ( - kid in sig_map - or kid not in rr["keys"] - or item.get("channel") != rr["keys"][kid]["recoveryChannel"] - ): - _fail("ARCHIVE_WITNESS_RECOVERY_SIGNATURE_BINDING_INVALID") - sig_map[kid] = item.get("signature") - if sorted(sig_map) != parsed["selected"] or sigs != sorted( - sigs, key=lambda x: x["keyId"] - ): - _fail("ARCHIVE_WITNESS_RECOVERY_SIGNATURE_SET_INVALID") - for kid in parsed["selected"]: - if ( - _dt( - rr["keys"][kid]["expires"], - "ARCHIVE_WITNESS_RECOVERY_KEY_EXPIRES_INVALID", - ) - < parsed["issued"] - ): - _fail("ARCHIVE_WITNESS_RECOVERY_KEY_EXPIRED") - _ed25519_verify( - rr["keys"][kid]["publicKey"], - sig_map[kid], - _canonical(subject), - "ARCHIVE_WITNESS_RECOVERY", - ) - expected_record = { - "schemaVersion": int(POLICY["recovery_record_schema_version"]), - "status": "retention-root-recovered", - "subject": subject, - "signatures": sigs, - "oldRetentionRoot": {"sha256": old["sha256"], "rootId": old["rootId"]}, - "newRetentionRoot": {"sha256": new["sha256"], "rootId": new["rootId"]}, - "recoveryRoot": {"sha256": rr["sha256"], "rootId": rr["rootId"]}, - } - if record != expected_record: - _fail("ARCHIVE_WITNESS_RECOVERY_RECORD_BINDING_INVALID") - expected_receipt = { - "schemaVersion": int(POLICY["recovery_receipt_schema_version"]), - "status": "retention-root-recovery-accepted", - "run161HealthChainHeadSha256": active["healthChainHeadSha256"], - "activeArchiveIds": active["activeArchiveIds"], - "retirementAuthorizedArchiveIds": [], - "recordSha256": _sha_bytes(record_raw), - "recoveredRetentionRootSha256": new["sha256"], - } - if receipt != expected_receipt: - _fail("ARCHIVE_WITNESS_RECOVERY_RECEIPT_MISMATCH") - return { - "ok": True, - "phase": "retention-root-recovery-verified", - "recovered_retention_root_sha256": new["sha256"], - "run161_health_chain_head_sha256": active["healthChainHeadSha256"], - "retirement_authorized": [], - } - - -def recover_retention_root( # ruff: ignore[too-many-branches, undocumented-public-function] - *, - run160_dir: Path, - run161_dir: Path, - old_retention_root_path: Path, - old_retention_root_pin: str, - new_retention_root_path: Path, - recovery_root_path: Path, - recovery_root_pin: str, - recovery_subject_path: Path, - recovery_signatures_path: Path, - bootstrap_pin: str, - recovery_pin: str, - attestation_pins: list[str], - output_dir: Path, - now: datetime | None = None, -) -> dict[str, Any]: - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - run160_dir = _regular_dir(run160_dir, "ARCHIVE_WITNESS_RUN160_DIR_INVALID") - run161_dir = _regular_dir(run161_dir, "ARCHIVE_WITNESS_RUN161_DIR_INVALID") - old_doc, _ = _read_json( - old_retention_root_path, "ARCHIVE_WITNESS_OLD_RETENTION_ROOT" - ) - try: - old = run161._verify_root( - old_doc, old_retention_root_pin, now=current, historical=True - ) - except Exception as exc: - raise ArchiveWitnessError( - "ARCHIVE_WITNESS_OLD_RETENTION_ROOT_INVALID:" + str(exc) - ) from exc - new_doc, new_raw = _read_json( - new_retention_root_path, "ARCHIVE_WITNESS_NEW_RETENTION_ROOT" - ) - new_sha = _sha_bytes(new_raw) - try: - new = run161._verify_root(new_doc, new_sha, now=current, historical=False) - except Exception as exc: - raise ArchiveWitnessError( - "ARCHIVE_WITNESS_NEW_RETENTION_ROOT_INVALID:" + str(exc) - ) from exc - recovery_doc, _ = _read_json(recovery_root_path, "ARCHIVE_WITNESS_RECOVERY_ROOT") - rr = _verify_threshold_root( - recovery_doc, - recovery_root_pin, - root_type="archive-retention-recovery-root", - schema_key="recovery_root_schema_version", - min_keys_key="min_recovery_keys", - min_threshold_key="min_recovery_threshold", - min_operators_key="min_recovery_operators", - now=current, - channel=True, - ) - _verify_run161( - run160_dir=run160_dir, - run161_dir=run161_dir, - retention_root_path=old_retention_root_path, - retention_root_pin=old_retention_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - now=current, - historical=True, - ) - active, _ = _read_json( - run161_dir / _DOC_ARCHIVE_HEALTH_STATE, "ARCHIVE_WITNESS_RUN161_STATE" - ) - subject, _ = _read_json(recovery_subject_path, "ARCHIVE_WITNESS_RECOVERY_SUBJECT") - parsed = _recovery_subject(subject, rr, old, new, active, current) - signatures, _ = _read_json( - recovery_signatures_path, "ARCHIVE_WITNESS_RECOVERY_SIGNATURES" - ) - if ( - set(signatures) != {"schemaVersion", "signatures"} - or signatures.get("schemaVersion") - != int(POLICY["recovery_record_schema_version"]) - or not isinstance(signatures.get("signatures"), list) - ): - _fail("ARCHIVE_WITNESS_RECOVERY_SIGNATURES_SCHEMA_INVALID") - sigs = signatures["signatures"] - if len(sigs) != len(parsed["selected"]): - _fail("ARCHIVE_WITNESS_RECOVERY_SIGNATURE_SET_INVALID") - sig_map = {} - for item in sigs: - if not isinstance(item, dict) or set(item) != {"keyId", "channel", "signature"}: - _fail("ARCHIVE_WITNESS_RECOVERY_SIGNATURE_SCHEMA_INVALID") - kid = _identity( - item.get("keyId"), "ARCHIVE_WITNESS_RECOVERY_SIGNATURE_KEY_INVALID" - ) - if ( - kid in sig_map - or kid not in rr["keys"] - or item.get("channel") != rr["keys"][kid]["recoveryChannel"] - ): - _fail("ARCHIVE_WITNESS_RECOVERY_SIGNATURE_BINDING_INVALID") - sig_map[kid] = item.get("signature") - if sorted(sig_map) != parsed["selected"] or sigs != sorted( - sigs, key=lambda x: x["keyId"] - ): - _fail("ARCHIVE_WITNESS_RECOVERY_SIGNATURE_SET_INVALID") - for kid in parsed["selected"]: - if ( - _dt( - rr["keys"][kid]["expires"], - "ARCHIVE_WITNESS_RECOVERY_KEY_EXPIRES_INVALID", - ) - < parsed["issued"] - ): - _fail("ARCHIVE_WITNESS_RECOVERY_KEY_EXPIRED") - _ed25519_verify( - rr["keys"][kid]["publicKey"], - sig_map[kid], - _canonical(subject), - "ARCHIVE_WITNESS_RECOVERY", - ) - protected = [ - run160_dir, - run161_dir, - old_retention_root_path.resolve(), - new_retention_root_path.resolve(), - recovery_root_path.resolve(), - recovery_subject_path.resolve(), - recovery_signatures_path.resolve(), - ] - target = _outside( - output_dir, protected, "ARCHIVE_WITNESS_RECOVERY_OUTPUT_OVERLAPS_INPUT" - ) - if target.exists(): - _fail("ARCHIVE_WITNESS_RECOVERY_OUTPUT_EXISTS") - fps = [ - _dir_fingerprint(run160_dir, "ARCHIVE_WITNESS_RECOVERY_RUN160_DRIFT"), - _dir_fingerprint(run161_dir, "ARCHIVE_WITNESS_RECOVERY_RUN161_DRIFT"), - _file_fingerprint( - old_retention_root_path.resolve(), "ARCHIVE_WITNESS_RECOVERY_OLD_ROOT_DRIFT" - ), - _file_fingerprint( - new_retention_root_path.resolve(), "ARCHIVE_WITNESS_RECOVERY_NEW_ROOT_DRIFT" - ), - _file_fingerprint( - recovery_root_path.resolve(), "ARCHIVE_WITNESS_RECOVERY_ROOT_DRIFT" - ), - _file_fingerprint( - recovery_subject_path.resolve(), "ARCHIVE_WITNESS_RECOVERY_SUBJECT_DRIFT" - ), - _file_fingerprint( - recovery_signatures_path.resolve(), "ARCHIVE_WITNESS_RECOVERY_SIGS_DRIFT" - ), - ] - fps2 = [ - _dir_fingerprint(run160_dir, "ARCHIVE_WITNESS_RECOVERY_RUN160_DRIFT"), - _dir_fingerprint(run161_dir, "ARCHIVE_WITNESS_RECOVERY_RUN161_DRIFT"), - _file_fingerprint( - old_retention_root_path.resolve(), "ARCHIVE_WITNESS_RECOVERY_OLD_ROOT_DRIFT" - ), - _file_fingerprint( - new_retention_root_path.resolve(), "ARCHIVE_WITNESS_RECOVERY_NEW_ROOT_DRIFT" - ), - _file_fingerprint( - recovery_root_path.resolve(), "ARCHIVE_WITNESS_RECOVERY_ROOT_DRIFT" - ), - _file_fingerprint( - recovery_subject_path.resolve(), "ARCHIVE_WITNESS_RECOVERY_SUBJECT_DRIFT" - ), - _file_fingerprint( - recovery_signatures_path.resolve(), "ARCHIVE_WITNESS_RECOVERY_SIGS_DRIFT" - ), - ] - if fps != fps2: - _fail("ARCHIVE_WITNESS_RECOVERY_INPUT_DRIFT_DETECTED") - record = { - "schemaVersion": int(POLICY["recovery_record_schema_version"]), - "status": "retention-root-recovered", - "subject": subject, - "signatures": sigs, - "oldRetentionRoot": {"sha256": old["sha256"], "rootId": old["rootId"]}, - "newRetentionRoot": {"sha256": new["sha256"], "rootId": new["rootId"]}, - "recoveryRoot": {"sha256": rr["sha256"], "rootId": rr["rootId"]}, - } - receipt = { - "schemaVersion": int(POLICY["recovery_receipt_schema_version"]), - "status": "retention-root-recovery-accepted", - "run161HealthChainHeadSha256": active["healthChainHeadSha256"], - "activeArchiveIds": active["activeArchiveIds"], - "retirementAuthorizedArchiveIds": [], - "recordSha256": _sha_bytes(_canonical(record)), - "recoveredRetentionRootSha256": new["sha256"], - } - parent = target.parent - parent.mkdir(parents=True, exist_ok=True) - stage = Path(tempfile.mkdtemp(prefix=".run162-recovery-", dir=parent)) - try: - (stage / "recovered-retention-root.json").write_bytes(new_raw) - _write(stage / "retention-root-recovery-record.json", record) - _write(stage / "retention-root-recovery-receipt.json", receipt) - if {p.name for p in stage.iterdir()} != _RECOVERY_OUTPUT_NAMES: - _fail("ARCHIVE_WITNESS_RECOVERY_OUTPUT_ALLOWLIST_INVALID") - verify_retention_root_recovery( - run160_dir=run160_dir, - run161_dir=run161_dir, - old_retention_root_path=old_retention_root_path, - old_retention_root_pin=old_retention_root_pin, - recovery_root_path=recovery_root_path, - recovery_root_pin=recovery_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - output_dir=stage, - now=current, - ) - stage.rename(target) - except Exception: - shutil.rmtree(stage, ignore_errors=True) - raise - verified = verify_retention_root_recovery( - run160_dir=run160_dir, - run161_dir=run161_dir, - old_retention_root_path=old_retention_root_path, - old_retention_root_pin=old_retention_root_pin, - recovery_root_path=recovery_root_path, - recovery_root_pin=recovery_root_pin, - bootstrap_pin=bootstrap_pin, - recovery_pin=recovery_pin, - attestation_pins=attestation_pins, - output_dir=target, - now=current, - ) - return dict(verified, old_retention_root_sha256=old["sha256"]) - - -def main( # ruff: ignore[undocumented-public-function] - argv: list[str] | None = None, -) -> int: - p = argparse.ArgumentParser(description=__doc__) - sub = p.add_subparsers(dest="command", required=True) - # The Python API is the release integration surface; CLI intentionally supports offline verification only here. - v = sub.add_parser("verify-witness") - v.add_argument("--run160-dir", type=Path, required=True) - v.add_argument("--run161-dir", type=Path, required=True) - v.add_argument("--retention-root", type=Path, required=True) - v.add_argument("--retention-root-pin", required=True) - v.add_argument("--witness-root", type=Path, required=True) - v.add_argument("--witness-root-pin", required=True) - v.add_argument("--bootstrap-pin", required=True) - v.add_argument("--recovery-pin", required=True) - v.add_argument("--attestation-pin", action="append", required=True) - v.add_argument("--output-dir", type=Path, required=True) - v.add_argument("--historical", action="store_true") - args = p.parse_args(argv) - if args.command == "verify-witness": - result = verify_witness_history( - run160_dir=args.run160_dir, - run161_dir=args.run161_dir, - retention_root_path=args.retention_root, - retention_root_pin=args.retention_root_pin, - witness_root_path=args.witness_root, - witness_root_pin=args.witness_root_pin, - bootstrap_pin=args.bootstrap_pin, - recovery_pin=args.recovery_pin, - attestation_pins=args.attestation_pin, - output_dir=args.output_dir, - historical=args.historical, - ) - logger.info("%s", json.dumps(result, sort_keys=True)) - return 0 - return 2 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/security/witness_publication.py b/security/witness_publication.py deleted file mode 100644 index 8c0d802cb455ab7080083c04392da5c56827c1ef..0000000000000000000000000000000000000000 --- a/security/witness_publication.py +++ /dev/null @@ -1,1324 +0,0 @@ -""" -Anchor a Run 151 final release record in append-only transparency and require a witness quorum. -""" - -from __future__ import annotations - -import argparse -import hashlib -import json -import logging -import os -import re -import shutil -import subprocess -import sys # ruff: ignore[unused-import] -import tempfile -import threading -from datetime import datetime, timedelta, timezone -from pathlib import Path -from typing import Any, Callable, Iterable - -import tomllib - -logger = logging.getLogger(__name__) -HERE = Path(__file__).resolve().parent -POLICY = tomllib.loads((HERE / "release_witness_policy.toml").read_text()) -RUN151_POLICY = tomllib.loads((HERE / "release_transparency_policy.toml").read_text()) -RECORD_NAME = str(POLICY["record_name"]) -WITNESS_RECORD_PREDICATE_TYPE = str(POLICY["witness_record_predicate_type"]) -_HEX64 = re.compile(r"^[0-9a-f]{64}$") -_ID = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:/@+-]{0,511}$") -_REVISION = re.compile(r"^[0-9a-f]{40}(?:[0-9a-f]{24})?$") -_CHUNK = 1024 * 1024 - - -class WitnessError(RuntimeError): - """Transparency logging or quorum verification violated a fail-closed invariant.""" - - -def _fail(code: str) -> None: - raise WitnessError(code) - - -def _sha256(path: Path) -> str: - h = hashlib.sha256() - with path.open("rb") as handle: - for chunk in iter(lambda: handle.read(_CHUNK), b""): - h.update(chunk) - return h.hexdigest() - - -def _canonical_bytes(value: dict[str, Any]) -> bytes: - return ( - json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False) - + "\n" - ).encode("utf-8") - - -def _write_canonical(path: Path, value: dict[str, Any]) -> None: - path.write_bytes(_canonical_bytes(value)) - - -def _loads_json(raw: bytes, code: str) -> dict[str, Any]: - def pairs_hook(pairs): - out: dict[str, Any] = {} - for key, value in pairs: - if key in out: - _fail(code + "_DUPLICATE_KEY") - out[key] = value - return out - - try: - value = json.loads(raw.decode("utf-8"), object_pairs_hook=pairs_hook) - except WitnessError: - raise - except Exception as exc: - raise WitnessError(code + "_JSON_INVALID") from exc - if not isinstance(value, dict): - _fail(code + "_SCHEMA_INVALID") - return value - - -def _read_json( - path: Path, code: str, *, canonical: bool = True -) -> tuple[dict[str, Any], bytes]: - if path.is_symlink() or not path.is_file(): - _fail(code + "_INVALID") - if path.stat().st_size > int(POLICY["max_json_bytes"]): - _fail(code + "_TOO_LARGE") - raw = path.read_bytes() - doc = _loads_json(raw, code) - if canonical and raw != _canonical_bytes(doc): - _fail(code + "_NOT_CANONICAL") - return doc, raw - - -def _regular_dir(path: Path, code: str) -> Path: - if path.is_symlink() or not path.is_dir(): - _fail(code) - return path.resolve() - - -def _outside(path: Path, protected: Iterable[Path], code: str) -> Path: - target = path.expanduser().resolve() - for item in protected: - root = item.resolve() - if target == root or root in target.parents: - _fail(code) - return target - - -def _hex(value: Any, code: str) -> str: - if not isinstance(value, str) or _HEX64.fullmatch(value) is None: - _fail(code) - return value - - -def _identity(value: Any, code: str) -> str: - if not isinstance(value, str): - _fail(code) - value = value.strip() - if ( - not value - or len(value) > 512 # ruff: ignore[magic-value-comparison] - or ".." in value - or "?" in value - or "#" in value - or "\x00" in value - ): - _fail(code) - if ( - any( - ord(ch) < 32 # ruff: ignore[magic-value-comparison] - or ord(ch) == 127 # ruff: ignore[magic-value-comparison] - for ch in value # lint - ) - or _ID.fullmatch(value) is None - ): - _fail(code) - return value - - -def _safe_text(value: Any, code: str, *, limit: int = 1024) -> str: - if not isinstance(value, str): - _fail(code) - value = value.strip() - if ( - not value - or len(value) > limit - or "?" in value - or "#" in value - or "\x00" in value - ): - _fail(code) - if any( - ord(ch) < 32 # ruff: ignore[magic-value-comparison] - or ord(ch) == 127 # ruff: ignore[magic-value-comparison] - for ch in value # lint - ): - _fail(code) - return value - - -def _safe_name(value: Any, code: str) -> str: - value = _safe_text(value, code, limit=255) - if ( - value in {".", ".."} - or Path(value).name != value - or "/" in value - or "\\" in value - ): - _fail(code) - return value - - -def _size(value: Any, code: str) -> int: - if not isinstance(value, int) or isinstance(value, bool) or value < 0: - _fail(code) - return value - - -def _fresh(value: Any, code: str, *, now: datetime) -> str: - if not isinstance(value, str) or not value.endswith("Z"): - _fail(code + "_INVALID") - try: - parsed = datetime.fromisoformat(value[:-1] + "+00:00").astimezone(timezone.utc) - except ValueError: - _fail(code + "_INVALID") - skew = timedelta(minutes=int(POLICY["max_clock_skew_minutes"])) - age = timedelta(minutes=int(POLICY["max_result_age_minutes"])) - if parsed > now + skew: - _fail(code + "_FROM_FUTURE") - if now - parsed > age: - _fail(code + "_STALE") - return value - - -def _artifact(value: Any, code: str) -> dict[str, Any]: - if not isinstance(value, dict) or set(value) != {"name", "sha256", "size"}: - _fail(code + "_SCHEMA_INVALID") - return { - "name": _safe_name(value.get("name"), code + "_NAME_INVALID"), - "sha256": _hex(value.get("sha256"), code + "_SHA256_INVALID"), - "size": _size(value.get("size"), code + "_SIZE_INVALID"), - } - - -def _checkpoint(value: Any, code: str) -> dict[str, Any]: - expected = {"treeSize", "rootHash", "signedCheckpointSha256"} - if not isinstance(value, dict) or set(value) != expected: - _fail(code + "_SCHEMA_INVALID") - tree_size = _size(value.get("treeSize"), code + "_TREE_SIZE_INVALID") - if tree_size < 1: - _fail(code + "_TREE_SIZE_INVALID") - return { - "treeSize": tree_size, - "rootHash": _hex(value.get("rootHash"), code + "_ROOT_HASH_INVALID"), - "signedCheckpointSha256": _hex( - value.get("signedCheckpointSha256"), code + "_SIGNED_HASH_INVALID" - ), - } - - -def _validate_finalized( # ruff: ignore[too-many-branches] - root: Path, -) -> dict[str, Any]: - root = _regular_dir(root, "RUN151_DIRECTORY_INVALID") - allowed = { - "publication-transparency.json", - "publication-receipt.json", - "publication-attestation.json", - "publication-attestation.signature-verification.json", - "release-publication-record.json", - "release-publication-binding-receipt.json", - "final-verifier-results", - "binding-results", - } - if {p.name for p in root.iterdir()} != allowed: - _fail("RUN151_DIRECTORY_ALLOWLIST_MISMATCH") - for dirname in ("final-verifier-results", "binding-results"): - directory = root / dirname - if directory.is_symlink() or not directory.is_dir(): - _fail("RUN151_EVIDENCE_DIRECTORY_INVALID") - for item in directory.iterdir(): - if item.is_symlink() or not item.is_file(): - _fail("RUN151_EVIDENCE_ENTRY_INVALID") - _read_json(item, "RUN151_EVIDENCE_ENTRY") - - record, record_raw = _read_json( - root / "release-publication-record.json", "RUN151_FINAL_RECORD" - ) - receipt, receipt_raw = _read_json( - root / "release-publication-binding-receipt.json", "RUN151_BINDING_RECEIPT" - ) - if set(record) != { - "schemaVersion", - "predicateType", - "status", - "publicationId", - "release", - "subject", - "target", - "signer", - "independentVerifier", - "artifacts", - "verification", - }: - _fail("RUN151_FINAL_RECORD_SCHEMA_INVALID") - if ( - record.get("schemaVersion") != int(RUN151_POLICY["final_record_schema_version"]) - or record.get("predicateType") - != str(RUN151_POLICY["final_record_predicate_type"]) - or record.get("status") != "finalized" - ): - _fail("RUN151_FINAL_RECORD_SCHEMA_INVALID") - publication_id = _safe_text( - record.get("publicationId"), "RUN151_PUBLICATION_ID_INVALID", limit=255 - ) - release = record.get("release") - if not isinstance(release, dict) or set(release) != {"releaseId", "sourceRevision"}: - _fail("RUN151_RELEASE_SCHEMA_INVALID") - release_id = _safe_text( - release.get("releaseId"), "RUN151_RELEASE_ID_INVALID", limit=127 - ) - source_revision = _safe_text( - release.get("sourceRevision"), "RUN151_SOURCE_REVISION_INVALID", limit=128 - ) - if _REVISION.fullmatch(source_revision) is None: - _fail("RUN151_SOURCE_REVISION_INVALID") - target = record.get("target") - if not isinstance(target, dict) or set(target) != {"publisher", "targetId"}: - _fail("RUN151_TARGET_SCHEMA_INVALID") - publisher = _safe_text( - target.get("publisher"), "RUN151_PUBLISHER_INVALID", limit=255 - ) - target_id = _safe_text( - target.get("targetId"), "RUN151_TARGET_ID_INVALID", limit=255 - ) - signer = record.get("signer") - if not isinstance(signer, dict) or set(signer) != {"identity", "verifiedAt"}: - _fail("RUN151_SIGNER_SCHEMA_INVALID") - _identity(signer.get("identity"), "RUN151_SIGNER_IDENTITY_INVALID") - if not isinstance(signer.get("verifiedAt"), str): - _fail("RUN151_SIGNER_TIME_INVALID") - independent = record.get("independentVerifier") - if not isinstance(independent, dict) or set(independent) != { - "identity", - "verificationId", - "readOnly", - "publisherCredentialsReused", - }: - _fail("RUN151_INDEPENDENT_VERIFIER_SCHEMA_INVALID") - _identity( - independent.get("identity"), "RUN151_INDEPENDENT_VERIFIER_IDENTITY_INVALID" - ) - _safe_text( - independent.get("verificationId"), - "RUN151_INDEPENDENT_VERIFICATION_ID_INVALID", - limit=255, - ) - if ( - independent.get("readOnly") is not True - or independent.get("publisherCredentialsReused") is not False - ): - _fail("RUN151_INDEPENDENT_VERIFIER_AUTHORITY_INVALID") - expected_verification = { - "publicationEvidenceVerified": True, - "signedAttestationVerified": True, - "signerIdentityVerified": True, - "independentRemoteReadbackVerifiedAfterSignature": True, - "allPublishedObjectsPresent": True, - } - if record.get("verification") != expected_verification: - _fail("RUN151_VERIFICATION_FLAGS_INVALID") - subject = record.get("subject") - expected_subject = { - "promotionReceiptSha256", - "publicationReceiptSha256", - "publicationTransparencySha256", - "postPublicationAttestationSha256", - "signatureVerificationSha256", - "signatureVerifierEvidenceSha256", - } - if not isinstance(subject, dict) or set(subject) != expected_subject: - _fail("RUN151_SUBJECT_SCHEMA_INVALID") - for key in expected_subject: - _hex(subject.get(key), "RUN151_SUBJECT_HASH_INVALID") - if ( - _sha256(root / "publication-transparency.json") - != subject["publicationTransparencySha256"] - ): - _fail("RUN151_TRANSPARENCY_HASH_MISMATCH") - if ( - _sha256(root / "publication-receipt.json") - != subject["publicationReceiptSha256"] - ): - _fail("RUN151_PUBLICATION_RECEIPT_HASH_MISMATCH") - if ( - _sha256(root / "publication-attestation.json") - != subject["postPublicationAttestationSha256"] - ): - _fail("RUN151_ATTESTATION_HASH_MISMATCH") - if ( - _sha256(root / "publication-attestation.signature-verification.json") - != subject["signatureVerificationSha256"] - ): - _fail("RUN151_SIGNATURE_RECORD_HASH_MISMATCH") - - receipt_keys = { - "schemaVersion", - "status", - "publicationId", - "releaseId", - "publicationTransparencySha256", - "postPublicationAttestationSha256", - "signatureVerificationSha256", - "finalRecord", - "binding", - "finalVerificationEvidence", - } - if ( - not isinstance(receipt, dict) - or set(receipt) != receipt_keys - or receipt.get("schemaVersion") - != int(RUN151_POLICY["binding_receipt_schema_version"]) - or receipt.get("status") != "bound" - ): - _fail("RUN151_BINDING_RECEIPT_SCHEMA_INVALID") - if ( - receipt.get("publicationId") != publication_id - or receipt.get("releaseId") != release_id - ): - _fail("RUN151_BINDING_RECEIPT_ID_MISMATCH") - if ( - receipt.get("publicationTransparencySha256") - != subject["publicationTransparencySha256"] - ): - _fail("RUN151_BINDING_RECEIPT_TRANSPARENCY_MISMATCH") - if ( - receipt.get("postPublicationAttestationSha256") - != subject["postPublicationAttestationSha256"] - ): - _fail("RUN151_BINDING_RECEIPT_ATTESTATION_MISMATCH") - if ( - receipt.get("signatureVerificationSha256") - != subject["signatureVerificationSha256"] - ): - _fail("RUN151_BINDING_RECEIPT_SIGNATURE_MISMATCH") - record_item = _artifact(receipt.get("finalRecord"), "RUN151_BOUND_FINAL_RECORD") - actual_record_item = { - "name": "release-publication-record.json", - "sha256": hashlib.sha256(record_raw).hexdigest(), - "size": len(record_raw), - } - if record_item != actual_record_item: - _fail("RUN151_BOUND_FINAL_RECORD_MISMATCH") - binding = receipt.get("binding") - if not isinstance(binding, dict) or set(binding) != { - "bindingId", - "locator", - "immutability", - "bindingType", - "verifiedAt", - "bindEvidenceSha256", - "verifyEvidenceSha256", - }: - _fail("RUN151_BINDING_SCHEMA_INVALID") - binding_locator = _safe_text( - binding.get("locator"), "RUN151_BINDING_LOCATOR_INVALID" - ) - if binding.get("immutability") not in set(POLICY["allowed_immutability"]): - _fail("RUN151_BINDING_IMMUTABILITY_INVALID") - if binding.get("bindingType") not in set(POLICY["allowed_binding_types"]): - _fail("RUN151_BINDING_TYPE_INVALID") - bind_evidence_sha = _hex( - binding.get("bindEvidenceSha256"), "RUN151_BIND_EVIDENCE_HASH_INVALID" - ) - verify_evidence_sha = _hex( - binding.get("verifyEvidenceSha256"), "RUN151_VERIFY_EVIDENCE_HASH_INVALID" - ) - bind_evidence = ( - root / "binding-results" / "release-publication-record.json.bind.json" - ) - verify_evidence = ( - root / "binding-results" / "release-publication-record.json.verify.json" - ) - if ( - _sha256(bind_evidence) != bind_evidence_sha - or _sha256(verify_evidence) != verify_evidence_sha - ): - _fail("RUN151_BINDING_EVIDENCE_HASH_MISMATCH") - artifacts = record.get("artifacts") - if not isinstance(artifacts, list) or not artifacts: - _fail("RUN151_ARTIFACTS_INVALID") - remote_locators: set[str] = {binding_locator} - artifact_names: set[str] = set() - for _index, item in enumerate(artifacts): - if not isinstance(item, dict) or set(item) != { - "name", - "sha256", - "size", - "locator", - "immutability", - }: - _fail("RUN151_ARTIFACT_SCHEMA_INVALID") - artifact_name = _safe_name(item.get("name"), "RUN151_ARTIFACT_NAME_INVALID") - if artifact_name in artifact_names: - _fail("RUN151_ARTIFACT_NAME_DUPLICATE") - artifact_names.add(artifact_name) - _hex(item.get("sha256"), "RUN151_ARTIFACT_HASH_INVALID") - _size(item.get("size"), "RUN151_ARTIFACT_SIZE_INVALID") - locator = _safe_text(item.get("locator"), "RUN151_ARTIFACT_LOCATOR_INVALID") - if locator in remote_locators: - _fail("RUN151_REMOTE_LOCATOR_COLLISION") - remote_locators.add(locator) - if item.get("immutability") not in set(POLICY["allowed_immutability"]): - _fail("RUN151_ARTIFACT_IMMUTABILITY_INVALID") - final_evidence = receipt.get("finalVerificationEvidence") - if not isinstance(final_evidence, list) or len(final_evidence) != len(artifacts): - _fail("RUN151_FINAL_VERIFICATION_EVIDENCE_INVALID") - result_files = sorted((root / "final-verifier-results").iterdir()) - if len(result_files) != len(artifacts): - _fail("RUN151_FINAL_VERIFICATION_EVIDENCE_COUNT_MISMATCH") - for index, (artifact_item, evidence_item, evidence_path) in enumerate( - zip(artifacts, final_evidence, result_files), start=1 - ): - if not isinstance(evidence_item, dict) or set(evidence_item) != { - "name", - "sha256", - }: - _fail("RUN151_FINAL_VERIFICATION_EVIDENCE_SCHEMA_INVALID") - if evidence_item.get("name") != artifact_item["name"]: - _fail("RUN151_FINAL_VERIFICATION_EVIDENCE_NAME_MISMATCH") - evidence_sha = _hex( - evidence_item.get("sha256"), - "RUN151_FINAL_VERIFICATION_EVIDENCE_HASH_INVALID", - ) - expected_filename = f"{index:02d}-{artifact_item['name']}.verify.json" - if ( - evidence_path.name != expected_filename - or _sha256(evidence_path) != evidence_sha - ): - _fail("RUN151_FINAL_VERIFICATION_EVIDENCE_HASH_MISMATCH") - return { - "root": root, - "publication_id": publication_id, - "release_id": release_id, - "source_revision": source_revision, - "publisher": publisher, - "target_id": target_id, - "record": record, - "record_item": actual_record_item, - "record_sha256": actual_record_item["sha256"], - "receipt_sha256": hashlib.sha256(receipt_raw).hexdigest(), - "binding_locator": binding_locator, - "remote_locators": remote_locators, - "attestation_sha256": subject["postPublicationAttestationSha256"], - "signature_sha256": subject["signatureVerificationSha256"], - "transparency_sha256": subject["publicationTransparencySha256"], - } - - -def _validate_previous_checkpoint(path: Path, *, log_id: str) -> dict[str, Any]: - doc, raw = _read_json(path, "PREVIOUS_CHECKPOINT") - if set(doc) != {"schemaVersion", "logId", "checkpoint"} or doc.get( - "schemaVersion" - ) != int(POLICY["previous_checkpoint_schema_version"]): - _fail("PREVIOUS_CHECKPOINT_SCHEMA_INVALID") - if ( - _safe_text(doc.get("logId"), "PREVIOUS_CHECKPOINT_LOG_ID_INVALID", limit=255) - != log_id - ): - _fail("PREVIOUS_CHECKPOINT_LOG_MISMATCH") - checkpoint = _checkpoint(doc.get("checkpoint"), "PREVIOUS_CHECKPOINT") - return { - "doc": doc, - "checkpoint": checkpoint, - "sha256": hashlib.sha256(raw).hexdigest(), - } - - -def _transparency_id(record_sha: str, log_id: str) -> str: - return ( - "tw-" + hashlib.sha256((record_sha + "\0" + log_id).encode()).hexdigest()[:40] - ) - - -def _verification_id(transparency_id: str, identity: str, role: str) -> str: - return ( - role - + "-" - + hashlib.sha256((transparency_id + "\0" + identity).encode()).hexdigest()[:32] - ) - - -def _anchor_id(record_sha: str, publisher: str, target_id: str) -> str: - return ( - "anchor-" - + hashlib.sha256( - (record_sha + "\0" + publisher + "\0" + target_id).encode() - ).hexdigest()[:40] - ) - - -def _command_adapter( - command: list[str], *, prefix: str -) -> Callable[[dict[str, Any]], dict[str, Any]]: - if not command: - _fail(prefix + "_COMMAND_INVALID") - - def call(request: dict[str, Any]) -> dict[str, Any]: - process = ( - # lint - subprocess.Popen( # ruff: ignore[subprocess-without-shell-equals-true] - command, - stdin=subprocess.PIPE, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, - text=False, - close_fds=True, - env={"PATH": os.defpath, "LC_ALL": "C", "LANG": "C"}, - ) - ) - assert ( # ruff: ignore[assert, pytest-composite-assertion] - process.stdin is not None - and process.stdout is not None - and process.stderr is not None - ) - stdin = process.stdin - process.stdin = None - stderr_chunks: list[bytes] = [] - stdout_chunks: list[bytes] = [] - overflow = threading.Event() - limit = int(POLICY["max_adapter_output_bytes"]) - - def drain(stream, chunks): - total = 0 - while True: - chunk = stream.read(65536) - if not chunk: - break - total += len(chunk) - if total > limit: - overflow.set() - process.kill() - break - chunks.append(chunk) - - threads = [ - threading.Thread(target=drain, args=(process.stdout, stdout_chunks)), - threading.Thread(target=drain, args=(process.stderr, stderr_chunks)), - ] - for thread in threads: - thread.start() - try: - stdin.write(_canonical_bytes(request)) - stdin.close() - process.wait(timeout=int(POLICY["adapter_timeout_seconds"])) - except subprocess.TimeoutExpired as exc: - process.kill() - raise WitnessError(prefix + "_TIMEOUT") from exc - finally: - for thread in threads: - thread.join() - if overflow.is_set(): - _fail(prefix + "_OUTPUT_TOO_LARGE") - if process.returncode != 0: - _fail(prefix + "_FAILED") - return _loads_json(b"".join(stdout_chunks), prefix + "_OUTPUT") - - return call - - -def command_log(command: list[str]): # ruff: ignore[undocumented-public-function] - return _command_adapter(command, prefix="TRANSPARENCY_LOG_ADAPTER") - - -def command_verifier(command: list[str]): # ruff: ignore[undocumented-public-function] - return _command_adapter(command, prefix="TRANSPARENCY_VERIFIER_ADAPTER") - - -def command_witness(command: list[str]): # ruff: ignore[undocumented-public-function] - return _command_adapter(command, prefix="TRANSPARENCY_WITNESS_ADAPTER") - - -def command_anchor(command: list[str]): # ruff: ignore[undocumented-public-function] - return _command_adapter(command, prefix="TRANSPARENCY_ANCHOR_ADAPTER") - - -def _validate_log_response( - value: dict[str, Any], - *, - info: dict[str, Any], - transparency_id: str, - log_id: str, - previous: dict[str, Any], - now: datetime, -) -> dict[str, Any]: - expected = { - "schemaVersion", - "operation", - "transparencyId", - "status", - "log", - "subject", - "checkpoint", - "proof", - "integratedAt", - } - if ( - set(value) != expected - or value.get("schemaVersion") != int(POLICY["log_protocol_version"]) - or value.get("operation") != "submit" - ): - _fail("TRANSPARENCY_LOG_RESULT_SCHEMA_INVALID") - if value.get("transparencyId") != transparency_id or value.get("status") not in { - "created", - "present", - }: - _fail("TRANSPARENCY_LOG_RESULT_STATUS_INVALID") - log = value.get("log") - if not isinstance(log, dict) or set(log) != { - "logId", - "entryId", - "entryIndex", - "entryLocator", - }: - _fail("TRANSPARENCY_LOG_RESULT_LOG_SCHEMA_INVALID") - if log.get("logId") != log_id: - _fail("TRANSPARENCY_LOG_ID_MISMATCH") - entry_id = _safe_text( - log.get("entryId"), "TRANSPARENCY_ENTRY_ID_INVALID", limit=255 - ) - entry_index = _size(log.get("entryIndex"), "TRANSPARENCY_ENTRY_INDEX_INVALID") - entry_locator = _safe_text( - log.get("entryLocator"), "TRANSPARENCY_ENTRY_LOCATOR_INVALID" - ) - subject = _artifact(value.get("subject"), "TRANSPARENCY_LOG_SUBJECT") - if subject != info["record_item"]: - _fail("TRANSPARENCY_LOG_SUBJECT_MISMATCH") - checkpoint = _checkpoint(value.get("checkpoint"), "TRANSPARENCY_LOG_CHECKPOINT") - if ( - checkpoint["treeSize"] <= previous["checkpoint"]["treeSize"] - or entry_index >= checkpoint["treeSize"] - ): - _fail("TRANSPARENCY_LOG_CHECKPOINT_ORDER_INVALID") - proof = value.get("proof") - if not isinstance(proof, dict) or set(proof) != { - "appendOnly", - "overwrite", - "integratedEntryVerified", - }: - _fail("TRANSPARENCY_LOG_PROOF_SCHEMA_INVALID") - if proof != { - "appendOnly": True, - "overwrite": False, - "integratedEntryVerified": True, - }: - _fail("TRANSPARENCY_LOG_GUARANTEES_INVALID") - integrated_at = _fresh( - value.get("integratedAt"), "TRANSPARENCY_LOG_INTEGRATED_AT", now=now - ) - return { - "entryId": entry_id, - "entryIndex": entry_index, - "entryLocator": entry_locator, - "checkpoint": checkpoint, - "integratedAt": integrated_at, - } - - -def _validate_observer_response( - value: dict[str, Any], - *, - role: str, - protocol_version: int, - expected_id: str, - identity: str, - operator: str | None, - info: dict[str, Any], - transparency_id: str, - log_id: str, - log_result: dict[str, Any], - previous: dict[str, Any], - primary_identity: str | None, - now: datetime, -) -> dict[str, Any]: - expected = { - "schemaVersion", - "operation", - "verificationId", - "transparencyId", - "status", - "observer", - "log", - "subject", - "checkpoint", - "previousCheckpoint", - "proof", - "verifiedAt", - } - if ( - set(value) != expected - or value.get("schemaVersion") != protocol_version - or value.get("operation") != "verify" - ): - _fail(role + "_RESULT_SCHEMA_INVALID") - if ( - value.get("verificationId") != expected_id - or value.get("transparencyId") != transparency_id - or value.get("status") != "included" - ): - _fail(role + "_RESULT_ID_MISMATCH") - observer = value.get("observer") - observer_keys = {"identity", "readOnly", "logCredentialsReused"} | ( - {"operator", "primaryVerifierCredentialsReused"} - if operator is not None - else set() - ) - if not isinstance(observer, dict) or set(observer) != observer_keys: - _fail(role + "_OBSERVER_SCHEMA_INVALID") - if ( - observer.get("identity") != identity - or observer.get("readOnly") is not True - or observer.get("logCredentialsReused") is not False - ): - _fail(role + "_OBSERVER_AUTHORITY_INVALID") - if operator is not None: - if ( - observer.get("operator") != operator - or observer.get("primaryVerifierCredentialsReused") is not False - ): - _fail(role + "_OBSERVER_SEPARATION_INVALID") - if primary_identity is not None and identity == primary_identity: - _fail(role + "_IDENTITY_REUSES_PRIMARY_VERIFIER") - log = value.get("log") - expected_log = { - "logId": log_id, - "entryId": log_result["entryId"], - "entryIndex": log_result["entryIndex"], - "entryLocator": log_result["entryLocator"], - } - if log != expected_log: - _fail(role + "_LOG_REBIND_FAILED") - if _artifact(value.get("subject"), role + "_SUBJECT") != info["record_item"]: - _fail(role + "_SUBJECT_MISMATCH") - if ( - _checkpoint(value.get("checkpoint"), role + "_CHECKPOINT") - != log_result["checkpoint"] - ): - _fail(role + "_CHECKPOINT_MISMATCH") - if ( - _checkpoint(value.get("previousCheckpoint"), role + "_PREVIOUS_CHECKPOINT") - != previous["checkpoint"] - ): - _fail(role + "_PREVIOUS_CHECKPOINT_MISMATCH") - proof = value.get("proof") - expected_proof = { - "checkpointSignatureVerified": True, - "inclusionVerified": True, - "consistencyVerified": True, - "integratedEntryVerified": True, - } - if proof != expected_proof: - _fail(role + "_PROOF_INVALID") - verified_at = _fresh(value.get("verifiedAt"), role + "_VERIFIED_AT", now=now) - return {"identity": identity, "operator": operator, "verifiedAt": verified_at} - - -def _validate_anchor_response( - value: dict[str, Any], - *, - operation: str, - info: dict[str, Any], - record_item: dict[str, Any], - anchor_id: str, - now: datetime, -) -> dict[str, Any]: - expected = { - "schemaVersion", - "operation", - "anchorId", - "status", - "target", - "record", - "guarantees", - "verifiedAt", - } - if ( - set(value) != expected - or value.get("schemaVersion") != int(POLICY["anchor_protocol_version"]) - or value.get("operation") != operation - ): - _fail("TRANSPARENCY_ANCHOR_RESULT_SCHEMA_INVALID") - if value.get("anchorId") != anchor_id or value.get("status") not in ( - {"created", "present"} if operation == "bind" else {"present"} - ): - _fail("TRANSPARENCY_ANCHOR_STATUS_INVALID") - target = value.get("target") - if target != {"publisher": info["publisher"], "targetId": info["target_id"]}: - _fail("TRANSPARENCY_ANCHOR_TARGET_MISMATCH") - if _artifact(value.get("record"), "TRANSPARENCY_ANCHOR_RECORD") != record_item: - _fail("TRANSPARENCY_ANCHOR_RECORD_MISMATCH") - guarantees = value.get("guarantees") - if not isinstance(guarantees, dict) or set(guarantees) != { - "createOnly", - "overwrite", - "remoteReadbackVerified", - "immutability", - "bindingType", - "locator", - }: - _fail("TRANSPARENCY_ANCHOR_GUARANTEES_SCHEMA_INVALID") - if ( - guarantees.get("createOnly") is not True - or guarantees.get("overwrite") is not False - or guarantees.get("remoteReadbackVerified") is not True - ): - _fail("TRANSPARENCY_ANCHOR_GUARANTEES_INVALID") - immutability = guarantees.get("immutability") - binding_type = guarantees.get("bindingType") - locator = _safe_text( - guarantees.get("locator"), "TRANSPARENCY_ANCHOR_LOCATOR_INVALID" - ) - if immutability not in set( - POLICY["allowed_immutability"] - ) or binding_type not in set(POLICY["allowed_binding_types"]): - _fail("TRANSPARENCY_ANCHOR_CLASS_INVALID") - if locator in info["remote_locators"]: - _fail("TRANSPARENCY_ANCHOR_LOCATOR_COLLISION") - verified_at = _fresh( - value.get("verifiedAt"), "TRANSPARENCY_ANCHOR_VERIFIED_AT", now=now - ) - return { - "locator": locator, - "immutability": immutability, - "bindingType": binding_type, - "verifiedAt": verified_at, - } - - -LogAdapter = Callable[[dict[str, Any]], dict[str, Any]] -ObserverAdapter = Callable[[dict[str, Any]], dict[str, Any]] -AnchorAdapter = Callable[[dict[str, Any]], dict[str, Any]] - - -def witness_publication( # ruff: ignore[undocumented-public-function] - *, - finalized_dir: Path, - previous_checkpoint: Path, - output_dir: Path, - log_id: str, - submitter_identity: str, - log_adapter: LogAdapter, - verifier_identity: str, - verifier: ObserverAdapter, - witnesses: list[tuple[str, str, ObserverAdapter]], - anchor: AnchorAdapter, - now: datetime | None = None, -) -> dict[str, Any]: - finalized_dir = _regular_dir(finalized_dir, "RUN151_DIRECTORY_INVALID") - target = _outside(output_dir, (finalized_dir,), "WITNESS_OUTPUT_INSIDE_INPUT") - if target.exists() or target.is_symlink(): - _fail("WITNESS_OUTPUT_ALREADY_EXISTS") - current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) - info = _validate_finalized(finalized_dir) - log_id = _safe_text(log_id, "TRANSPARENCY_LOG_ID_INVALID", limit=255) - submitter_identity = _identity( - submitter_identity, "TRANSPARENCY_SUBMITTER_IDENTITY_INVALID" - ) - verifier_identity = _identity( - verifier_identity, "TRANSPARENCY_VERIFIER_IDENTITY_INVALID" - ) - if verifier_identity == submitter_identity: - _fail("TRANSPARENCY_VERIFIER_REUSES_SUBMITTER_IDENTITY") - previous = _validate_previous_checkpoint(previous_checkpoint, log_id=log_id) - if ( - not witnesses - or len(witnesses) < int(POLICY["min_witnesses"]) - or len(witnesses) > int(POLICY["max_witnesses"]) - ): - _fail("TRANSPARENCY_WITNESS_COUNT_INVALID") - normalized_witnesses: list[tuple[str, str, ObserverAdapter]] = [] - identities: set[str] = set() - operators: set[str] = set() - for identity_raw, operator_raw, adapter in witnesses: - identity = _identity(identity_raw, "TRANSPARENCY_WITNESS_IDENTITY_INVALID") - operator = _identity(operator_raw, "TRANSPARENCY_WITNESS_OPERATOR_INVALID") - if identity in identities or identity in { - submitter_identity, - verifier_identity, - }: - _fail("TRANSPARENCY_WITNESS_IDENTITY_NOT_DISTINCT") - identities.add(identity) - operators.add(operator) - normalized_witnesses.append((identity, operator, adapter)) - if len(operators) < int(POLICY["min_witness_operators"]): - _fail("TRANSPARENCY_WITNESS_OPERATOR_QUORUM_INVALID") - - initial_hashes = { - name: _sha256(finalized_dir / name) - for name in ( - "release-publication-record.json", - "release-publication-binding-receipt.json", - "publication-attestation.json", - "publication-attestation.signature-verification.json", - ) - } - transparency_id = _transparency_id(info["record_sha256"], log_id) - target.parent.mkdir(parents=True, exist_ok=True) - with tempfile.TemporaryDirectory( - prefix="release-witness-", dir=target.parent - ) as temp_raw: - stage = Path(temp_raw) / "witnessed" - stage.mkdir() - log_results = stage / "log-results" - witness_results = stage / "witness-results" - anchor_results = stage / "anchor-results" - log_results.mkdir() - witness_results.mkdir() - anchor_results.mkdir() - subject_with_path = dict( - info["record_item"], - localPath=str(finalized_dir / "release-publication-record.json"), - ) - log_request = { - "schemaVersion": int(POLICY["log_protocol_version"]), - "operation": "submit", - "transparencyId": transparency_id, - "release": { - "releaseId": info["release_id"], - "publicationId": info["publication_id"], - }, - "submitter": {"identity": submitter_identity}, - "logId": log_id, - "subject": subject_with_path, - "previousCheckpoint": previous["checkpoint"], - } - before = _sha256(finalized_dir / "release-publication-record.json") - log_raw = log_adapter(log_request) - if _sha256(finalized_dir / "release-publication-record.json") != before: - _fail("RUN151_FINAL_RECORD_CHANGED_DURING_LOG_SUBMIT") - log_result = _validate_log_response( - log_raw, - info=info, - transparency_id=transparency_id, - log_id=log_id, - previous=previous, - now=current, - ) - _write_canonical(log_results / "submit.json", log_raw) - - base_observer_request = { - "schemaVersion": int(POLICY["verifier_protocol_version"]), - "operation": "verify", - "transparencyId": transparency_id, - "release": { - "releaseId": info["release_id"], - "publicationId": info["publication_id"], - }, - "log": { - "logId": log_id, - "entryId": log_result["entryId"], - "entryIndex": log_result["entryIndex"], - "entryLocator": log_result["entryLocator"], - }, - "subject": info["record_item"], - "checkpoint": log_result["checkpoint"], - "previousCheckpoint": previous["checkpoint"], - } - primary_id = _verification_id(transparency_id, verifier_identity, "primary") - primary_request = dict(base_observer_request) - primary_request["verificationId"] = primary_id - primary_request["observer"] = {"identity": verifier_identity} - primary_raw = verifier(primary_request) - _validate_observer_response( - primary_raw, - role="TRANSPARENCY_PRIMARY_VERIFIER", - protocol_version=int(POLICY["verifier_protocol_version"]), - expected_id=primary_id, - identity=verifier_identity, - operator=None, - info=info, - transparency_id=transparency_id, - log_id=log_id, - log_result=log_result, - previous=previous, - primary_identity=None, - now=current, - ) - primary_path = log_results / "primary.verify.json" - _write_canonical(primary_path, primary_raw) - - witness_meta: list[dict[str, str]] = [] - witness_evidence: list[dict[str, str]] = [] - for index, (identity, operator, adapter) in enumerate( - sorted(normalized_witnesses, key=lambda x: (x[1], x[0])), start=1 - ): - wid = _verification_id(transparency_id, identity, "witness") - request = dict(base_observer_request) - request["schemaVersion"] = int(POLICY["witness_protocol_version"]) - request["verificationId"] = wid - request["observer"] = {"identity": identity, "operator": operator} - raw = adapter(request) - _validate_observer_response( - raw, - role="TRANSPARENCY_WITNESS", - protocol_version=int(POLICY["witness_protocol_version"]), - expected_id=wid, - identity=identity, - operator=operator, - info=info, - transparency_id=transparency_id, - log_id=log_id, - log_result=log_result, - previous=previous, - primary_identity=verifier_identity, - now=current, - ) - path = ( - witness_results - / f"{index:02d}-{hashlib.sha256(identity.encode()).hexdigest()[:12]}.verify.json" - ) - _write_canonical(path, raw) - witness_meta.append({"identity": identity, "operator": operator}) - witness_evidence.append( - {"identity": identity, "operator": operator, "sha256": _sha256(path)} - ) - - witnessed_record = { - "schemaVersion": int(POLICY["witness_record_schema_version"]), - "predicateType": WITNESS_RECORD_PREDICATE_TYPE, - "status": "witnessed", - "release": { - "releaseId": info["release_id"], - "publicationId": info["publication_id"], - "sourceRevision": info["source_revision"], - }, - "subject": { - "finalPublicationRecordSha256": info["record_sha256"], - "finalBindingReceiptSha256": info["receipt_sha256"], - "postPublicationAttestationSha256": info["attestation_sha256"], - "signatureVerificationSha256": info["signature_sha256"], - }, - "target": {"publisher": info["publisher"], "targetId": info["target_id"]}, - "transparencyLog": { - "logId": log_id, - "entryId": log_result["entryId"], - "entryIndex": log_result["entryIndex"], - "entryLocator": log_result["entryLocator"], - "checkpoint": log_result["checkpoint"], - "previousCheckpointSha256": previous["sha256"], - }, - "verification": { - "primaryVerifierIdentity": verifier_identity, - "checkpointSignatureVerified": True, - "inclusionVerified": True, - "consistencyVerified": True, - "integratedEntryVerified": True, - }, - "witnessQuorum": { - "threshold": int(POLICY["min_witnesses"]), - "minimumOperators": int(POLICY["min_witness_operators"]), - "witnesses": witness_meta, - }, - } - record_path = stage / RECORD_NAME - _write_canonical(record_path, witnessed_record) - os.chmod(record_path, 0o444) - record_item = { - "name": RECORD_NAME, - "sha256": _sha256(record_path), - "size": record_path.stat().st_size, - } - anchor_id = _anchor_id( - record_item["sha256"], info["publisher"], info["target_id"] - ) - base_anchor = { - "schemaVersion": int(POLICY["anchor_protocol_version"]), - "anchorId": anchor_id, - "releaseId": info["release_id"], - "publicationId": info["publication_id"], - "target": {"publisher": info["publisher"], "targetId": info["target_id"]}, - "record": record_item, - } - bind_request = dict(base_anchor, operation="bind") - bind_request["record"] = dict(record_item, localPath=str(record_path)) - record_before = _sha256(record_path) - bind_raw = anchor(bind_request) - if _sha256(record_path) != record_before: - _fail("WITNESS_RECORD_CHANGED_DURING_ANCHOR") - bind_result = _validate_anchor_response( - bind_raw, - operation="bind", - info=info, - record_item=record_item, - anchor_id=anchor_id, - now=current, - ) - verify_raw = anchor(dict(base_anchor, operation="verify")) - verify_result = _validate_anchor_response( - verify_raw, - operation="verify", - info=info, - record_item=record_item, - anchor_id=anchor_id, - now=current, - ) - if any( - bind_result[key] != verify_result[key] - for key in ("locator", "immutability", "bindingType") - ): - _fail("TRANSPARENCY_ANCHOR_VERIFY_REBIND_FAILED") - bind_path = anchor_results / (RECORD_NAME + ".bind.json") - verify_path = anchor_results / (RECORD_NAME + ".verify.json") - _write_canonical(bind_path, bind_raw) - _write_canonical(verify_path, verify_raw) - - info_after = _validate_finalized(finalized_dir) - final_hashes = {name: _sha256(finalized_dir / name) for name in initial_hashes} - if ( - initial_hashes != final_hashes - or info_after["record_sha256"] != info["record_sha256"] - or info_after["receipt_sha256"] != info["receipt_sha256"] - ): - _fail("RUN151_INPUT_CHANGED_DURING_WITNESSING") - if _sha256(previous_checkpoint) != previous["sha256"]: - _fail("PREVIOUS_CHECKPOINT_CHANGED_DURING_WITNESSING") - - shutil.copy2( - finalized_dir / "release-publication-record.json", - stage / "release-publication-record.json", - ) - shutil.copy2( - finalized_dir / "release-publication-binding-receipt.json", - stage / "release-publication-binding-receipt.json", - ) - shutil.copy2( - previous_checkpoint, stage / "previous-transparency-checkpoint.json" - ) - accepted_checkpoint_path = stage / "accepted-transparency-checkpoint.json" - _write_canonical( - accepted_checkpoint_path, - { - "schemaVersion": int(POLICY["previous_checkpoint_schema_version"]), - "logId": log_id, - "checkpoint": log_result["checkpoint"], - }, - ) - receipt = { - "schemaVersion": int(POLICY["witness_receipt_schema_version"]), - "status": "witnessed", - "releaseId": info["release_id"], - "publicationId": info["publication_id"], - "finalPublicationRecord": info["record_item"], - "finalBindingReceiptSha256": info["receipt_sha256"], - "witnessRecord": record_item, - "transparency": { - "transparencyId": transparency_id, - "logId": log_id, - "entryId": log_result["entryId"], - "entryIndex": log_result["entryIndex"], - "entryLocator": log_result["entryLocator"], - "checkpoint": log_result["checkpoint"], - "previousCheckpointSha256": previous["sha256"], - "acceptedCheckpointSha256": _sha256(accepted_checkpoint_path), - "submitEvidenceSha256": _sha256(log_results / "submit.json"), - "primaryVerifierEvidenceSha256": _sha256(primary_path), - }, - "witnessEvidence": witness_evidence, - "anchor": { - "anchorId": anchor_id, - "locator": verify_result["locator"], - "immutability": verify_result["immutability"], - "bindingType": verify_result["bindingType"], - "verifiedAt": verify_result["verifiedAt"], - "bindEvidenceSha256": _sha256(bind_path), - "verifyEvidenceSha256": _sha256(verify_path), - }, - } - receipt_path = stage / "release-transparency-witness-receipt.json" - _write_canonical(receipt_path, receipt) - tmp_target = target.with_name(target.name + ".tmp-" + os.urandom(6).hex()) - shutil.copytree(stage, tmp_target, copy_function=shutil.copy2) - os.replace(tmp_target, target) - return { - "ok": True, - "phase": "witnessed", - "release_id": info["release_id"], - "publication_id": info["publication_id"], - "transparency_id": transparency_id, - "witness_count": len(witness_meta), - "operator_count": len(operators), - "witness_record_sha256": _sha256(target / RECORD_NAME), - "witness_receipt_sha256": _sha256( - target / "release-transparency-witness-receipt.json" - ), - "anchor_id": anchor_id, - } - - -def _parse_witness(value: str) -> tuple[str, str, list[str]]: - parts = value.split("=", 1) - if len(parts) != 2 or "@" not in parts[0]: # ruff: ignore[magic-value-comparison] - raise argparse.ArgumentTypeError( - "witness must be IDENTITY@OPERATOR=EXECUTABLE[,ARG...]" - ) - identity, operator = parts[0].split("@", 1) - command = [part for part in parts[1].split(",") if part] - if not identity or not operator or not command: - raise argparse.ArgumentTypeError("invalid witness specification") - return identity, operator, command - - -def main( # ruff: ignore[undocumented-public-function] - argv: list[str] | None = None, -) -> int: - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--finalized-dir", type=Path, required=True) - parser.add_argument("--previous-checkpoint", type=Path, required=True) - parser.add_argument("--output-dir", type=Path, required=True) - parser.add_argument("--log-id", required=True) - parser.add_argument("--submitter-identity", required=True) - parser.add_argument("--log-executable", required=True) - parser.add_argument("--log-arg", action="append", default=[]) - parser.add_argument("--verifier-identity", required=True) - parser.add_argument("--verifier-executable", required=True) - parser.add_argument("--verifier-arg", action="append", default=[]) - parser.add_argument( - "--witness", - action="append", - type=_parse_witness, - required=True, - help="IDENTITY@OPERATOR=EXECUTABLE[,ARG...]", - ) - parser.add_argument("--anchor-executable", required=True) - parser.add_argument("--anchor-arg", action="append", default=[]) - args = parser.parse_args(argv) - try: - witnesses = [ - (identity, operator, command_witness(command)) - for identity, operator, command in args.witness - ] - result = witness_publication( - finalized_dir=args.finalized_dir, - previous_checkpoint=args.previous_checkpoint, - output_dir=args.output_dir, - log_id=args.log_id, - submitter_identity=args.submitter_identity, - log_adapter=command_log([args.log_executable, *args.log_arg]), - verifier_identity=args.verifier_identity, - verifier=command_verifier([args.verifier_executable, *args.verifier_arg]), - witnesses=witnesses, - anchor=command_anchor([args.anchor_executable, *args.anchor_arg]), - ) - except (WitnessError, OSError) as exc: - logger.error(json.dumps({"ok": False, "error": str(exc)}, sort_keys=True)) - return 2 - logger.info(json.dumps(result, sort_keys=True)) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main())