Download cloud-forge/docs/CLOUD_ARCHITECTURE.md from Snapkitty/tlm-jxcl-forge: direct link, hf CLI and curl.
- Browser
- Download file 67 kB
-
https://huggingface.co/Snapkitty/tlm-jxcl-forge/resolve/main/cloud-forge/docs/CLOUD_ARCHITECTURE.md
- Command line
-
hf download hf://Snapkitty/tlm-jxcl-forge/cloud-forge/docs/CLOUD_ARCHITECTURE.md
-
curl -L -o CLOUD_ARCHITECTURE.md https://huggingface.co/Snapkitty/tlm-jxcl-forge/resolve/main/cloud-forge/docs/CLOUD_ARCHITECTURE.md
cloud-forge architecture
The non-negotiable rule
Do not create one crate per AWS service. Build the primitives once, then compose services from those primitives.
cloud-forge is a from-first-principles attempt at the substrate
underneath an AWS-shaped cloud platform: resource identity, lifecycle,
ownership, policy, tagging, events, and quota — the handful of concepts
that every cloud service (compute, storage, database, messaging, …)
reuses rather than reinvents. No crate in this workspace is named after
an AWS product, and none will be until it is a thin composition of
already-real primitive crates, not a container for logic that belongs
in one of them.
This mandate carries the same weight as the root workspace's "100
crates, none of them fake" rule (see the repository root
docs/CRATE_ARCHITECTURE.md) and
verification-forge's hard invariants: a crate exists here only when
it owns a real, independently-testable invariant.
Why a third workspace
tlm-jxcl-forge already separates two independent Cargo workspaces:
the root workspace (the TLM JXCL ISA, post-quantum crypto/storage, one
reference service) and verification-forge (a from-scratch proof
kernel). Neither is a natural home for cloud-resource modeling:
- The root workspace's
docs/crates.tomlregistry is machine-checked at exactly 100 crates with its own dependency-DAG rules (tools/check_workspace.py); folding an open-ended, many-phase cloud platform into it would either blow past that validated count or force cloud crates to misrepresent themselves as ISA/crypto crates. verification-forgeis a general-purpose proof kernel with no cloud-specific concept in it at all — it will eventually verify invariants aboutcloud-forge(see "Relationship to verification-forge" below), which means it must stay independent of it, not host it.
So cloud-forge/ is a third, independent top-level Cargo workspace,
with its own Cargo.toml, crates/, and quality gates — exactly the
same shape as verification-forge/.
Layering
flowchart TB
subgraph existing["Already real, in the root workspace"]
isa["jxcl* — ISA, execution engine"]
pq["pq* — PQ crypto, storage, ZK proofs"]
end
subgraph kernel["cloud-forge Phase 1: the primitive kernel (this pass)"]
types["cloud-types<br/>ResourceId, Arn, AccountId, RegionId"]
errors["cloud-errors<br/>shared CloudError"]
tags["cloud-tags"]
lifecycle["cloud-lifecycle<br/>state machine"]
resource["cloud-resource<br/>Resource<T> wrapper"]
region["cloud-region"]
account["cloud-account"]
identity["cloud-identity<br/>Principal (minimal)"]
policy["cloud-policy<br/>Allow/Deny, deny dominates"]
events["cloud-events"]
quota["cloud-quota"]
core["cloud-core<br/>facade"]
types --> errors
errors --> tags
errors --> lifecycle
types --> resource
tags --> resource
lifecycle --> resource
types --> region
types --> account
types --> identity
identity --> policy
types --> policy
errors --> policy
types --> events
types --> quota
resource --> core
region --> core
account --> core
identity --> core
policy --> core
events --> core
quota --> core
end
subgraph future["Planned, later phases — not built yet"]
control["Phase 2: control plane<br/>(provisioning, reconciliation, scheduler)"]
services["Phase 4+: service layer<br/>(compute, storage, database, …)"]
end
existing -.underlies.-> kernel
kernel --> future
Every arrow above is a real [dependencies] edge in a crate's own
Cargo.toml — there is no crate in this phase that depends on
something not yet built, and no crate that exists only to be depended
on later without doing anything itself today.
The resource model (Phase 0 / Phase 1)
Every cloud resource, regardless of which future service creates it, carries the same fields — this is the one model every service composes from rather than redefining:
| Field | Type | Owning crate |
|---|---|---|
id |
ResourceId |
cloud-types |
resource_type |
ResourceType |
cloud-types |
region |
RegionId |
cloud-types / cloud-region |
account |
AccountId |
cloud-types / cloud-account |
owner |
AccountId |
cloud-types |
lifecycle |
Lifecycle |
cloud-lifecycle |
version |
u64 |
cloud-resource |
tags |
Tags |
cloud-tags |
created_at / updated_at |
u64 (unix millis) |
cloud-resource |
owner here means account ownership (who a resource bills to and
belongs to) — distinct from cloud-identity::Principal, which
cloud-policy evaluates access for. A resource-based policy attaching
a Principal to a specific resource is a Phase 13 (IAM) concept, not
pulled forward into this phase's Resource<T>.
cloud-resource::Resource<T> combines all of these plus a
service-specific payload T, so a future Instance or Bucket type
is Resource<InstanceSpec> / Resource<BucketSpec>, not a
hand-rolled struct that has to remember to include tags and
lifecycle correctly on its own.
The internal ARN format cloud-types::Arn implements is exactly the
one specified in the roadmap:
jxcl:cloud:<partition>:<service>:<region>:<account>:<resource>
What Phase 1 deliberately does not include
cloud-metadatais not a separate crate. Its proposed fields (created_at/updated_at/tags/version) are exactlycloud-resourceandcloud-tags's fields; a separate crate would own nothing a caller couldn't already get from those two.cloud-runtimeis deferred past Phase 2 too, now to Phase 4 (compute primitives). Phase 2 does add a scheduler, but what it schedules is placement (whichAzIda resource is assigned to) — a decisioncloud-schedulerowns directly usingcloud-types's existingAzId, with no new wrapper type needed. A "runtime" only becomes a real, distinct primitive once something is actually executing (a VM, a container, a function invocation) — Phase 4's concern, not Phase 2's.cloud-identityis intentionally minimal in this phase: aPrincipalenum (User/Role/Service, each carrying an id) — enough forcloud-policyto evaluate against. The full IAM surface (credentials, sessions, federation, policy documents-as-JSON) is Phase 13 in the roadmap and is not pulled forward.- No AWS-named crate exists anywhere in this phase.
services/ec2,services/s3, etc. cannot be started honestly until the primitives they'd compose (compute, storage, network primitives — later phases) are themselves real.
Phase 2: the control plane
Phase 2 is where Phase 1's primitives stop being independent data types and start composing into an actual pipeline: a request to create a resource gets authorized, checked against quota, placed somewhere, constructed, and recorded — with the whole operation succeeding atomically or rolling back cleanly. This is also where the roadmap's deterministic state machine
REQUEST -> AUTHENTICATE -> AUTHORIZE -> VALIDATE -> PLAN -> APPLY -> VERIFY -> COMMIT -> AUDIT
becomes a real, tested code path rather than a diagram. (AUTHENTICATE
is not separately modeled here: this phase takes an already-resolved
Principal as input, since credential/session verification is Phase
13's IAM surface, not Phase 2's control-plane concern.)
What's real in Phase 2
| Crate | Owns |
|---|---|
cloud-scheduler |
Placement: picks the least-loaded AzId in a region, from cloud-region's registry and a caller-supplied usage count per AZ |
cloud-service-registry |
A registry of named control-plane services and their health (Healthy/Unhealthy/Unknown) |
cloud-reconciler |
Given a current and a desired Lifecycle, computes the shortest valid transition path through cloud-lifecycle's state graph (via BFS), or reports the desired state as unreachable |
cloud-provisioner |
The AUTHORIZE -> VALIDATE -> PLAN -> APPLY -> VERIFY -> COMMIT -> AUDIT pipeline itself, composing cloud-policy + cloud-quota + cloud-scheduler + cloud-resource + cloud-events, with each stage's failure distinguishable and every reservation made by an earlier stage rolled back if a later stage fails |
cloud-control-plane |
ControlPlane<T>: holds the account/region registries, policy, quota tracker, event log, and a resource store; exposes create (via cloud-provisioner), get, list, delete |
What Phase 2 deliberately skips or defers
resource-manager,lifecycle-manager,quota-manager,policy-engine,account-manager,region-managerare not separate crates. Each would be a thin wrapper doing nothing but forwarding tocloud-resource,cloud-lifecycle,cloud-quota,cloud-policy,cloud-account, orcloud-region— exactly the one-crate-per-name anti-pattern this workspace's non-negotiable rule exists to prevent.cloud-control-planeholds these Phase 1 primitives directly instead of through an extra indirection layer that owns nothing new.control-api(a REST/gRPC layer) is deferred. The roadmap's own execution order puts the API layer (Phase 30) well after the control plane it exposes; building an HTTP surface before the control plane it would serve is stable is building a service, not a primitive.cloud-runtimeis deferred to Phase 4 — see above.
The rollback invariant
cloud-provisioner's pipeline makes a real claim worth stating
explicitly: if AUTHORIZE succeeds but a later stage (VALIDATE,
PLAN, APPLY, or COMMIT) fails, every resource reservation an
earlier stage made (currently: cloud-quota usage) is released before
the pipeline returns its error. This is tested directly — not inferred
from "the individual stages are each atomic" — since a multi-stage
pipeline built from atomic parts is not itself atomic unless someone
wires the rollback path and verifies it.
Phase 3: AWS-like resource identification
The roadmap names four crates for this phase: cloud-arn,
cloud-resource-id, cloud-resource-parser, cloud-resource-registry.
Three of the four already exist — just earlier than the roadmap's own
ordering expected, because the resource model itself needed them from
day one:
| Roadmap name | Where it actually lives | Since |
|---|---|---|
cloud-arn |
cloud_types::Arn — parses and formats jxcl:cloud:<partition>:<service>:<region>:<account>:<resource> |
Phase 1 |
cloud-resource-id |
cloud_types::ResourceId |
Phase 1 |
cloud-resource-parser |
Arn's own FromStr impl — a separate parser crate would parse into the exact same Arn struct cloud-types already owns, with nothing new to own |
Phase 1 (not a separate crate — see below) |
cloud-resource-registry |
New this phase | Phase 3 |
Resource<T> (Phase 1) has an id, a type, a region, and an owning
account, but nothing before Phase 3 ever assembles those into the
canonical, fully-qualified name for a provisioned resource, or lets
anyone look a resource up by that name rather than by its bare
ResourceId. That's what this phase adds:
cloud-resource-registryowns exactly one thing: anArn↔ResourceIdmapping. It rejects a second registration of an already-used ARN, and (a stricter, deliberate choice beyond what the roadmap specifies) rejects registering the same resource under a second ARN too — a provisioned resource's canonical name is chosen once, at creation, and does not change.cloud-control-planenow knows its ownpartitionandservice(constructor arguments), and computes each resource'sArnfrompartition+service+ the resource's own region/account/id atcreate()time, registering it in an internally-ownedResourceRegistry— with the same rollback discipline as every othercreate()failure mode: if ARN construction or registration fails, the quota reservation and placementcreate()already made throughcloud-provisionerare released before the error returns, exactly like anAPPLYfailure.resolve_arn()andarn_of()expose the new lookup direction.
cloud-provisioner itself is untouched by this phase: ARN identity is
a naming concern that belongs to whatever owns a partition/
service (the control plane), not to the provisioning pipeline, which
stays deliberately ignorant of naming schemes.
Phase 4: compute primitives
Phase 1 deferred cloud-runtime twice (first past Phase 1, then again
past Phase 2) on the same stated grounds: "a runtime only becomes a
real, distinct primitive once something is actually executing." This
phase is where that becomes true, and where the two other primitives a
compute-shaped resource needs alongside a runtime -- a supply of
capacity to run on, and something to boot from -- are built:
| Crate | Owns |
|---|---|
cloud-runtime |
RuntimeState: the execution-state machine (Pending/Running/Stopping/Stopped/Terminating/Terminated) for something actually running, kept deliberately separate from cloud-lifecycle::Lifecycle (see below) |
cloud-capacity |
Per-AZ vCPU/memory-MiB capacity: reservations fail closed against an AZ's registered total |
cloud-image |
A registry of machine images to launch from: validated id, name, size, architecture; immutable once registered |
cloud-runtime vs. cloud-lifecycle
These are not the same state machine wearing two names. Lifecycle
answers "does this resource record exist, and in what stage of
being provisioned/updated/torn down?" -- a question every resource in
this workspace has needed an answer to since Phase 1, whether or not
anything about it ever executes. RuntimeState answers "is the thing
that record represents currently running?" -- a question that is
meaningless for a bucket or a policy, and that varies independently
of Lifecycle for the resources it does apply to: a compute instance
can sit Lifecycle::Active for its entire existence while cycling
between RuntimeState::Running and RuntimeState::Stopped many times
over. Collapsing the two into one enum would either force every
non-compute resource to carry meaningless Running/Stopped states,
or force compute resources to smuggle their execution state through
Lifecycle::Updating -- both of which misrepresent what actually
changed. Two small, single-purpose state machines is the primitive
answer; one overloaded one is the shortcut this workspace's rule
exists to reject.
Why capacity is a different shape from quota
cloud-quota (Phase 1) and cloud-capacity (this phase) look similar
-- both track a limit and a running usage -- but encode an opposite
default on purpose. cloud-quota is an account cap: a resource
category nobody has configured a limit for is unlimited, because the
absence of a quota policy should not itself block anything. cloud-capacity
is a physical(-ish) pool: an availability zone nobody has told its
own size has nothing to schedule onto, so CapacityTracker::try_reserve
fails closed with NotFound against an unregistered AZ rather than
inventing an unlimited pool that doesn't exist. Same shape of code,
opposite failure mode, because they model opposite kinds of limit.
The one real composition this phase adds
cloud-scheduler (Phase 2) gains place_least_loaded_with_capacity,
which filters cloud-region's candidate AZs down to the ones
cloud-capacity reports enough spare room in, picks the least-loaded
survivor exactly as place_least_loaded already did, and reserves
that capacity on the winner in the same call -- all-or-nothing, per
cloud-capacity::CapacityTracker::try_reserve's own atomicity, and
touching neither usage nor capacity for any AZ if none qualifies.
This is a deliberately generic extension (any resource type can ask
for capacity-aware placement, not just compute), which is why it lives
in cloud-scheduler rather than in a new crate: capacity-aware
placement is a placement-policy concern cloud-scheduler already
owns, not a compute-specific one.
What Phase 4 deliberately does not include
- No
cloud-compute(or similarly named) crate exists yet. Wiringcloud-runtime+cloud-capacity+cloud-imageinto an actual "launch an instance" operation -- a compute-specificProvisionRequestpayload, start/stop/terminate operations drivingRuntimeStatetransitions -- is the first genuinely compute-shaped service this workspace would build, and per the non-negotiable rule, that composition doesn't get built (or named) until it is one: three standalone, independently real primitives today, not a shell crate waiting for them. cloud-provisionerandcloud-control-planeare untouched by this phase. Reserving vCPU/memory at provisioning time is a compute-specific requirement, not a property every resourcecloud-provisioner's generic pipeline creates has (a bucket or a policy has no vCPU count) -- baking it into the generic pipeline would be exactly the kind of one-off special case this workspace's primitives are supposed to prevent. The capacity-aware scheduler function above is available to whatever later composes a compute service on top of the existing pipeline.- Image deregistration is not implemented. Whether an image that's
still referenced by a launched (but not yet modeled) instance can be
safely removed is a question
cloud-imagealone cannot answer -- it would need to see resource-to-image references that don't exist in this workspace until compute resources themselves do.
Phase 5: storage primitives
Compute (Phase 4) needed something to execute, a pool to run it in, and something to boot it from. Storage needs a different set of concepts entirely -- nothing about vCPUs or images applies -- but the same discipline: build what every storage-shaped resource (an object, a volume, a database's write-ahead log) would otherwise redefine for itself, and nothing that presupposes a specific service shape:
| Crate | Owns |
|---|---|
cloud-checksum |
Checksum: a from-scratch CRC-32/ISO-HDLC content-integrity checksum, streaming or one-shot |
cloud-redundancy |
RedundancyScheme: replication or erasure coding, shard counts, reconstruction threshold, max tolerable loss |
cloud-attachment |
AttachmentState: the attach/detach state machine for a volume-like resource |
Why cloud-checksum has zero dependencies and isn't a wrapper around a crates.io hash crate
Every other crate in this workspace either has zero dependencies or
depends only on other cloud-forge crates -- there has never been an
external (crates.io) dependency anywhere in this workspace, and this
phase doesn't start now. CRC-32/ISO-HDLC is implemented directly,
bit-by-bit rather than via a precomputed table, favoring an
obviously-correct implementation over a fast one. Its correctness is
anchored to the algorithm's own published standard check value
(crc32(b"123456789") == 0xCBF4_3926, the same value every real
implementation -- zlib, gzip, PNG -- uses to confirm itself), not just
internal self-consistency between of() and a hand-rolled
alternative. This is a deliberate application of this workspace's
anti-hallucination discipline to code, not just prose: a checksum
algorithm invented for this crate, with no external value to check it
against, would be exactly the kind of unverifiable claim this project
avoids elsewhere.
Why redundancy and attachment are their own primitives
RedundancyScheme models an arithmetic property (how many of N shards
must survive to reconstruct the original), not an encoder -- this
crate never touches actual bytes, which is why it composes with any
future storage service regardless of whether that service replicates
whole objects or erasure-codes them. AttachmentState is a third
state machine alongside cloud-lifecycle::Lifecycle and
cloud-runtime::RuntimeState (Phase 4), and is worth contrasting with
both explicitly: Lifecycle tracks whether a resource record
exists, RuntimeState tracks whether a compute resource is
executing (meaningless for a volume), and AttachmentState tracks
whether this volume is currently connected to something -- three
questions that vary independently, not one overloaded enum wearing
three names. Unlike the other two, AttachmentState has no terminal
state: every state has a path back to Detached, because attachment
cycles for the volume's whole life; ending that life is Lifecycle's
job, not this crate's. This is tested directly (every_state_has_a_path_back_to_detached),
not just asserted from the transition table's shape.
What Phase 5 deliberately does not include
- No
cloud-storage-capacitycrate.cloud-capacity(Phase 4) tracks vCPU/memory, a two-dimensional quantity meaningful to compute placement; a byte-capacity pool for storage nodes is a real, different primitive, but building it now with no concrete allocator or storage control-plane phase to feed would produce exactly the kind of crate that exists only to be depended on later without doing anything itself today -- the thing this workspace's layering rule explicitly prohibits (see "Layering" above). It's deferred until a storage provisioning phase actually needs it. - No object/blob storage crate and no
cloud-volumecrate. Composingcloud-checksum+cloud-redundancy+cloud-attachment(andcloud-provisioner/cloud-control-plane) into an actual storage service is the same kind of composition Phase 4 deferred for compute (cloud-compute): it doesn't get built, or named, until it's a real composition of already-real primitives. - No erasure-coding encoder/decoder.
cloud-redundancymodels the arithmetic a real erasure code (e.g. Reed-Solomon) guarantees --data_shardsofdata_shards + parity_shardsshards reconstruct the original -- without implementing the encoding itself, which is a substantial cryptographic/mathematical undertaking out of scope for a placement/durability-accounting primitive.
Phase 6: database primitives
Compute (Phase 4) needed execution state, capacity, and images. Storage (Phase 5) needed integrity, durability, and attachment state. Database-shaped resources need a third, disjoint set of concepts -- none of the prior nine crates from those two phases apply here:
| Crate | Owns |
|---|---|
cloud-consistency |
ConsistencyLevel (Eventual/BoundedStaleness/Strong): a total order plus satisfies() against a caller's minimum requirement |
cloud-migration |
MigrationLedger: enforces that schema migrations apply strictly sequentially and without gaps |
cloud-retention |
RetentionPolicy: which backups/snapshots are eligible for deletion, by age, with a floor protecting the most recent N regardless of age |
Why the ordering is derived, not hand-rolled
ConsistencyLevel's three variants are declared weakest-to-strongest
specifically so #[derive(PartialOrd, Ord)] produces the correct
total order for free -- Eventual < BoundedStaleness < Strong falls
out of the derive macro's own documented rule (variants compare by
declaration position) rather than a hand-written match that could
silently drift out of sync with the enum if a variant were ever
reordered. satisfies() is then just self >= required: the derive
does the real work, the method exists only to name the operation.
Why cloud-migration and cloud-retention don't touch each other
Both crates could plausibly want to know about each other -- a
retention policy might seem like it should care whether a migration
happened -- but they answer genuinely unrelated questions:
MigrationLedger is about schema shape over time (has version N's
change been applied), RetentionPolicy is about data lifetime (is
this snapshot's age past a threshold, and is it protected by the
floor). Coupling them would mean forcing every caller of one to drag
in a concept the other owns. RetentionPolicy depends on cloud-types
(for ResourceId and Timestamp) precisely because those are the
workspace's own shared value types, not because it needs anything
migration- or consistency-specific.
What Phase 6 deliberately does not include
- No
cloud-database(or similarly named) crate, and nocloud-table/cloud-index/cloud-querycrates. Composingcloud-consistency+cloud-migration+cloud-retention(and earlier phases'cloud-provisioner/cloud-control-plane) into an actual database service is the same kind of composition Phases 4 and 5 deferred for compute and storage: it doesn't get built, or named, until it's a real composition of already-real primitives. - No query language, execution engine, or storage engine. This phase's crates answer "what guarantee is this database offering," "is the schema in a consistent state," and "what data may be discarded" -- not "how is data stored or queried," which is a vastly larger undertaking out of scope for a primitives phase.
cloud-migrationdoes not run migrations. It has no concept of SQL, DDL, or any specific transformation -- only the version-ordering invariant. Actually executing a migration's contents is a concern of whatever future service calls this ledger, exactly ascloud-reconciler(Phase 2) computes a lifecycle transition path without executing whatever makes a resource actually reach the next state.
Phase 7: messaging primitives
Compute, storage, and database (Phases 4-6) each needed their own disjoint set of concepts. Messaging-shaped resources (queues, topics, pub/sub) need a fourth, equally disjoint set -- the last of the four service categories this repository's own README names before AWS-shaped service composition begins:
| Crate | Owns |
|---|---|
cloud-delivery |
DeliverySemantics (AtMostOnce/AtLeastOnce/ExactlyOnce): a genuine partial order, not a total one |
cloud-visibility |
MessageLease: the visibility-timeout mechanism that actually implements at-least-once delivery, plus a dead-letter threshold on receive count |
cloud-fanout |
FanoutRegistry: the topic-to-subscriber pub/sub topology |
Why cloud-delivery doesn't derive Ord, unlike cloud-consistency
Phase 6's cloud-consistency::ConsistencyLevel is a genuine total
order: every pair of levels is comparable, and #[derive(Ord)]'s
declaration-order rule gives the correct answer for free. Delivery
semantics are different in kind, not just in domain: they vary along
two independent axes -- whether loss is tolerated, and whether
duplication is tolerated. AtMostOnce (loss allowed, duplication
forbidden) and AtLeastOnce (loss forbidden, duplication allowed) are
incomparable: neither is "stronger" than the other, because each
permits something the other categorically forbids. A derived (or
hand-rolled linear) ordering would have to put one before the other
arbitrarily, and satisfies() would then give a wrong answer for that
pair. cloud-delivery instead checks both axes explicitly, which
correctly reports AtMostOnce.satisfies(AtLeastOnce) and its reverse
as both false, while ExactlyOnce (which forbids both) satisfies
every requirement. This is worth stating explicitly because it would
have been easy, and wrong, to copy Phase 6's pattern here without
checking whether the underlying relationship was actually total.
Why cloud-visibility is a separate crate from cloud-delivery
DeliverySemantics declares a guarantee; MessageLease is the
actual mechanism (a hide-until-deadline lease plus a receive counter)
that makes AtLeastOnce true in practice -- a message that isn't
acknowledged in time becomes visible again automatically, which is how
"never loses, may duplicate" is achieved mechanically rather than
merely claimed. Keeping the declaration and the mechanism in separate
crates mirrors cloud-lifecycle (Phase 1, a state machine) staying
separate from cloud-provisioner (Phase 2, what actually drives state
transitions) -- a semantics type and the engine that upholds it are
different kinds of thing even when tightly related.
What Phase 7 deliberately does not include
- No
cloud-queue/cloud-topic(or similarly named) crate. Composingcloud-delivery+cloud-visibility+cloud-fanout(and earlier phases'cloud-provisioner/cloud-control-plane) into an actual messaging service is the same composition every prior phase deferred for its own service category: it doesn't get built, or named, until it's real. cloud-fanoutdoes not deliver anything. It owns the topic-to-subscriber mapping only -- no idea what a message is, no retry, no per-subscriberMessageLease. Actually fanning a published message out (and tracking each subscriber's own lease) is a future service's job, exactly as the Phase 3Arnregistry owns a name-to-resource mapping without touching provisioning.- No message ordering (FIFO) primitive. Ordering guarantees interact with partitioning and fanout in ways that need a concrete queue/topic shape to be meaningful; modeling it against primitives alone risked producing an untestable abstraction.
Phase 8: cloud-compute, the first real composed service
Phases 4-7 built sixteen primitive crates across four service
categories (compute, storage, database, messaging) and, at every
single one, explicitly deferred composing them into anything -- "no
cloud-compute (or similarly named) crate exists yet," repeated
almost verbatim in each phase's own section above. Phase 8 is where
that deferral ends for compute, the category with the most complete
primitive set: cloud-runtime and cloud-capacity and cloud-image
(Phase 4), plus cloud-resource-registry (Phase 3) and the Phase 1/2
resource model, are enough to build a real "launch an instance"
operation without inventing a single new primitive.
| Crate | Owns |
|---|---|
cloud-compute |
ComputeService: launch/transition_runtime/terminate, composing eleven existing crates |
Why this doesn't reuse cloud-provisioner/cloud-control-plane
Phase 2's pipeline is deliberately generic: it works for any resource
type and places with cloud_scheduler::place_least_loaded, which
takes no capacity into account because most resources (a policy, a
bucket in later phases) have none to check. ComputeService needs
place_least_loaded_with_capacity (Phase 4) instead, since an
instance genuinely cannot be placed somewhere without room for it.
Two ways existed to get there: teach the generic pipeline about
vCPU/memory (a one-off special case baked into a crate every other
resource type also goes through -- exactly the anti-pattern the
non-negotiable rule exists to prevent), or have cloud-compute run
its own pipeline directly against the same underlying primitives.
cloud-compute does the latter: it reimplements the shape of
AUTHORIZE -> VALIDATE -> PLAN -> APPLY (policy check, quota
reservation, capacity-aware placement, resource construction plus ARN
registration) with the same rollback discipline cloud-provisioner
established, but calling cloud-capacity/cloud-image/cloud-runtime
directly rather than through a generic layer that doesn't know they
exist.
The rollback chain launch upholds
launch reserves, in order: a quota unit, then placement capacity
(which place_least_loaded_with_capacity reserves atomically with
its own AZ-usage increment), then an ARN registration. A failure at
any later stage releases every earlier reservation before returning --
tested directly for the capacity-exhaustion case (QuotaExceeded from
PLAN rolls back the VALIDATE-stage quota unit) exactly as
cloud-provisioner's own tests do for its stages.
RuntimeState lives in its own store, deliberately
ComputeService keeps RuntimeState in a separate map from
Resource<InstanceSpec>, not as a field inside the resource or its
payload. This is not an implementation shortcut: cloud-runtime's own
docs (Phase 4) describe execution state and the resource record as two
things that vary independently, and storing them together would
re-couple what Phase 4 deliberately kept apart. transition_runtime
touches only the runtime store; a resource's own version/updated_at
never changes just because the instance was started or stopped.
What Phase 8 deliberately does not include
- No image deregistration, no instance resize, no attached
volumes.
terminatereleases exactly whatlaunchreserved (quota, capacity); nothing else about an instance's lifecycle beyond launch/run/stop/terminate is modeled yet. cloud-storage,cloud-database,cloud-messaging(or similarly named) services do not exist yet. Phase 8 is compute's turn only -- the other three service categories built in Phases 5-7 remain primitives without a composed service, for exactly the reason this phase just finished demonstrating takes real, deliberate design work rather than a copy-paste ofcloud-compute's shape.- No HTTP/gRPC API surface.
control-apiremains deferred from Phase 2's own reasoning: an API layer belongs after the service it exposes is itself real, andcloud-computeis the first phase where that precondition starts to hold for any one category.
Phase 9: cloud-storage, the second composed service
Phase 8's closing note named exactly this: "the other three primitive
categories built in Phases 5-7 remain uncomposed." Phase 9 composes
the second -- storage -- following cloud-compute's own shape
deliberately rather than inventing a new one: a service composes
already-real primitives, adds no new primitive of its own, and keeps
the same AUTHORIZE -> VALIDATE -> APPLY-shaped pipeline with full
rollback.
| Crate | Owns |
|---|---|
cloud-storage |
StorageService: create_volume/transition_attachment/delete_volume, composing nine existing crates |
What's different from cloud-compute, and why
- No AZ placement.
cloud-computeplaces instances viacloud_scheduler::place_least_loaded_with_capacitybecausecloud-capacity(Phase 4) gives it a real per-AZ vCPU/memory pool to place against. Phase 5 deliberately never built acloud-storage-capacityequivalent -- "no concrete allocator to feed" -- and Phase 9 doesn't retroactively invent one just to mirrorcloud-compute's shape. A volume's size is tracked the same waycloud-computetracks instance count: an account-levelcloud-quotareservation ("storage_bytes"), real and tested, but not AZ-scoped. - A real, new business rule:
delete_volumerefuses to proceed unless the volume'sAttachmentStateisDetached, checked before any lifecycle reconciliation or quota release happens. This is the first place in the workspace a service-level operation is gated by a different primitive's current state (attachment) rather than just its own resource's lifecycle -- a real cross-primitive invariant, not a restatement of somethingcloud-attachmentorcloud-lifecyclealready enforces on its own. transition_attachmentmirrorscloud-compute::transition_runtimeexactly: a thin, single-step wrapper around the primitive's owntransition_to, leaving the higher-level operation (delete_volume, parallel toterminate) to own the multi-step, side-effecting case.
What Phase 9 deliberately does not include
- No attachment target validation.
transition_attachmentmoves a volume's own state machine; it does not check that some real compute instance exists to attach to, or call intocloud-computeto verify one. Two service crates directly depending on each other's internal stores would be a tighter coupling than either has needed so far -- reconciling "does the thing I'm attaching to actually exist" across two independent services is an orchestration concern for whatever future layer calls both, exactly ascloud-fanout(Phase 7) doesn't verify a subscriber id names a real resource either. - No
cloud-database/cloud-messaging(or similarly named) services yet. Phase 9 is storage's turn only. - No snapshots, no
cloud-checksum/cloud-retentionintegration. A volume in this phase has no content-integrity check and no backup lifecycle; wiring those Phase 5/6 primitives in is future work once a snapshot concept exists to attach them to.
Phase 10: cloud-database, the third composed service
The third of the four service categories (Phases 5-7) gets composed,
following exactly the shape cloud-compute and cloud-storage
established.
| Crate | Owns |
|---|---|
cloud-database |
DatabaseService: create_database/apply_migration/create_snapshot/expire_snapshots/delete_database, composing ten existing crates |
cloud-retention finally gets a stateful caller
Phase 6 built RetentionPolicy::eligible_for_deletion as a pure
function: given "now" and a list of (id, created_at) pairs, it
computes which ids a policy would allow deleting. Nothing before this
phase actually kept such a list anywhere. cloud-database is the
first thing that does: create_snapshot appends to a real,
per-database snapshot list, and expire_snapshots calls
eligible_for_deletion against it and then actually removes what
comes back -- the stateful half of a primitive Phase 6 deliberately
left as pure computation. This also resolves Phase 9's own deferral
note on cloud-storage ("wiring [retention] in is future work once a
snapshot concept exists to attach them to"): the snapshot concept now
exists, here, in the service where it was always going to make more
sense (a database's point-in-time backups, not a raw block volume).
The delete-time gate, and how it differs from cloud-storage's
cloud-storage::delete_volume refuses based on another primitive's
current state (AttachmentState != Detached). cloud-database::delete_database
refuses based on this service's own recorded state (a non-empty
snapshot list) rather than a different primitive's state machine --
a different kind of cross-cutting check, but the same underlying
discipline: a service-level delete operation must look past its own
resource's Lifecycle before deciding it's safe to proceed.
Why apply_migration is a thin pass-through
Exactly like cloud-compute::transition_runtime and
cloud-storage::transition_attachment, apply_migration does nothing
beyond delegating to the primitive's own method
(MigrationLedger::apply) and keeping the per-database map in sync.
The pattern across all three services is now well-established: a
primitive's own state-changing method is never reimplemented at the
service layer, only looked up, called, and stored back.
What Phase 10 deliberately does not include
- No query execution, no schema DDL.
apply_migrationrecords that a version was applied; it has no idea what SQL or transformation that version represents, exactly ascloud-migrationitself (Phase 6) does not run migrations. - No
cloud-messaging(or similarly named) service yet. Phase 10 is database's turn only; messaging (Phase 7's primitives) remains the one service category not yet composed. create_snapshotdoes not usecloud-checksum. A snapshot here is just an id and a creation time -- verifying a snapshot's actual content integrity would need the snapshot to contain real bytes, which this phase's model doesn't have.
Phase 11: cloud-messaging, the fourth and last composed service
The fourth and final service category (Phases 5-7) gets composed,
completing the set cloud-compute, cloud-storage, and
cloud-database began. Every service category a primitive phase left
deferred is now real.
| Crate | Owns |
|---|---|
cloud-messaging |
MessagingService: create_queue/create_topic/subscribe/publish/enqueue/receive/delete_message/delete_queue/delete_topic, composing eleven existing crates |
Two resource kinds, not one
cloud-compute, cloud-storage, and cloud-database each had exactly
one top-level resource (an instance, a volume, a database).
cloud-messaging has two: a queue, which holds messages directly, and
a topic, which holds none of its own -- only a fan-out list of
subscriber queues via cloud-fanout::FanoutRegistry. This is precisely
the composition cloud-fanout's own Phase 7 doc comment named as
deferred: "actually fanning a published message out to each subscriber
... is a concern for whatever future service composes all three
[cloud-delivery, cloud-visibility, cloud-fanout]." publish
is that composition: it fans a message out to every subscriber queue,
inserting a fresh cloud_visibility::MessageLease into each one, so
receive inherits real at-least-once mechanics (in-flight messages
can't be redelivered until their visibility timeout elapses, and a
message received too many times is dead-lettered) whether the message
arrived by direct enqueue or by fan-out from a topic.
subscribe's gate: a new operation, an established gate shape
cloud-storage::delete_volume (Phase 9) refused based on another
primitive's current state (a volume's own AttachmentState).
cloud-messaging::subscribe reuses that exact shape on a different
operation: it refuses to link a queue to a topic unless the queue's own
cloud_delivery::DeliverySemantics satisfies the topic's required
semantics (DeliverySemantics::satisfies, Phase 7) -- an AtMostOnce
queue cannot subscribe to a topic that requires ExactlyOnce. The
cross-primitive check that Phase 9 first applied at delete time now
applies at subscribe time instead.
delete_queue/delete_topic's gate: the cloud-database shape, twice
cloud-database::delete_database (Phase 10) refused based on the
service's own recorded state (a non-empty snapshot list) rather than
another primitive's state machine. Both of cloud-messaging's delete
operations reuse that shape: delete_queue refuses while its inbox is
non-empty, and delete_topic refuses while it still has subscribers --
two independent applications of the same "look past your own
Lifecycle at your own recorded state before deleting" discipline,
one per resource kind.
publish validates every subscriber before delivering to any
A duplicate message id colliding with one subscriber's existing inbox
entry, discovered partway through a fan-out, must not leave some
subscribers holding the message and others without it. publish checks
every subscriber has room for the new id in a first pass, and only
inserts into any of them in a second pass once every check has
succeeded -- the same validate-before-apply discipline every pipeline
in this workspace follows, applied here to a fan-out instead of a
single resource's creation.
What Phase 11 deliberately does not include
- No message body. A message is just a
ResourceId, exactly as acloud-databasesnapshot (Phase 10) was just an id and a creation time -- there is no payload to store, checksum, or serialize. - No ordering guarantee.
BTreeMap's key order is used for determinism in tests and listings, not as a delivery-order promise; nothing in this phase models FIFO queues. - No per-account message-count quota. Exactly as
cloud-databasedid not quota snapshots (only"databases"itself), this phase quotas"queues"and"topics"at creation time but not the messages flowing through them. - Every service category originally identified back in Phases 4-7 is now composed. A future phase's "what's still deferred" is no longer a service category; it is whatever composing these four services together (e.g. a compute instance's logs delivered through a queue) would require, which is deliberately out of scope here.
Phase 12: cloud-orchestration, the first cross-service composition
Phases 8-11 each composed primitives within one service category.
Phase 12 is a different kind of composition: it composes two already-real
services together, directly resolving a deferral cloud-storage
(Phase 9) stated in its own words.
| Crate | Owns |
|---|---|
cloud-orchestration |
attach_volume/detach_volume: free functions coordinating ComputeService and StorageService, composing two existing services |
Resolving cloud-storage's own deferral, verbatim
Phase 9's doc comment said this explicitly: "cloud-storage doesn't
validate attachment targets -- attaching a volume moves only its own
AttachmentState; whether the id it's attached to names a real
cloud-compute instance is an orchestration concern for whatever
future layer calls both services." cloud-orchestration is that layer.
attach_volume takes a &ComputeService and a &mut StorageService
together, checks the instance exists and is not Terminating/
Terminated, and only then drives the volume's attachment transitions.
An unknown or terminated instance leaves the volume's AttachmentState
completely untouched -- validate before mutate, the same discipline
every pipeline in this workspace already follows, just applied across
a service boundary instead of within one service's own stages.
Free functions, not a third stateful service
Unlike cloud-compute/cloud-storage/cloud-database/cloud-messaging,
cloud-orchestration owns no resource store, no quota, no policy, no
event log -- it has nothing to persist between calls. Its two functions
take &ComputeService/&mut StorageService directly and return,
exactly the shape cloud-provisioner (Phase 2) already established for
a pipeline with nothing of its own to hold onto. A caller keeps owning
both services; cloud-orchestration only ever borrows them for the
duration of one call.
Why there is no rollback to write
Every earlier multi-stage pipeline in this workspace (cloud-provisioner,
cloud-compute::launch, cloud-database::create_database, ...) needed
rollback because a later stage could fail after an earlier stage had
already reserved something real (quota, capacity, a name). attach_volume
has no such stage: once the instance check passes, Attaching -> Attached
is unconditionally valid in cloud-attachment's own transition table
(Phase 5) from the exact state attach_volume itself just placed the
volume into. There is no failure mode between those two calls to roll
back from, and the doc comment says so explicitly rather than leaving a
reader to wonder why this phase, alone among the pipeline-shaped crates,
has no rollback path.
Why detach_volume doesn't check compute at all
Detaching is presumed always safe, regardless of the instance's state -- including an instance that has since been terminated, or was deleted entirely. This mirrors real block-storage services, which support a force-detach specifically for the case where the thing a volume was attached to is already gone: the operation exists to free the volume, not to protect an instance that no longer needs protecting.
What Phase 12 deliberately does not include
- No orchestration for
cloud-databaseorcloud-messaging. This phase resolves exactly the one deferralcloud-storagestated; wiring a database or a queue to a compute instance is a different cross-service question with no equivalent stated deferral yet. - No compute-instance-to-queue log delivery, the example Phase 11's own closing note raised as one of many possible future cross-service compositions. This phase picks the one with an existing, explicit deferral behind it rather than inventing a new composition with no prior art in this workspace to resolve.
- No new state machine.
cloud-orchestrationreadscloud-runtime::RuntimeStateand drivescloud-attachment::AttachmentState-- both already exist; this phase adds no new states and no new transitions to either.
Phase 13: the IAM surface -- credentials, sessions, policy documents
cloud-identity's own Phase 1 doc comment named this explicitly: "the
full IAM surface (credentials, sessions, federation, policy
documents-as-JSON) is Phase 13 in the roadmap and is not pulled
forward." Phase 13 is that surface -- three new primitive crates, the
same shape Phases 4-7 each used (multiple disjoint primitives, no
service composition yet), rather than a service. A future IAM service
composing them is deferred exactly as cloud-compute was deferred
until Phase 8, well after Phase 4 built its primitives.
| Crate | Owns |
|---|---|
cloud-credentials |
CredentialStore: at most two Active credentials per principal, the real IAM rule AWS itself enforces |
cloud-session |
SessionStore: assumed-role and federated sessions, valid only while un-revoked and unexpired |
cloud-policy-document |
to_json/from_json: a cloud-policy::Policy serialized to and parsed from a JSON-shaped document, via a hand-rolled parser restricted to exactly this schema |
Why no real secret material
cloud-credentials models a credential's lifecycle -- its id, the
principal it belongs to, and whether it's Active or Inactive -- and
nothing about its actual secret bytes. Generating a cryptographically
secure secret needs a real source of randomness, which this
zero-external-dependency workspace deliberately doesn't pull in (every
crate here depends only on other cloud-* crates). This mirrors
cloud-lifecycle modeling a resource's lifecycle without knowing what
the resource actually does: the state machine is real; the payload it
governs is out of scope for the crate that owns the state machine.
The real IAM rule: at most two active credentials
Rather than invent a business rule, cloud-credentials implements one
AWS itself enforces on its own IAM users: a principal may hold at most
MAX_ACTIVE_CREDENTIALS_PER_PRINCIPAL (2) Active credentials at
once. Inactive credentials don't count against the cap, so a
principal can rotate safely -- create a new key, deactivate the old
one, delete it later -- without ever exceeding the limit at any point
in between. Reactivating a credential re-checks the same cap create
does, since reactivation can push a principal back over the limit just
as creation can.
cloud-session's two independent invalidity paths
A [Session] is valid only while it is both un-revoked and
unexpired -- deliberately the same two-independent-axis shape
cloud_visibility::MessageLease (Phase 7) already established for
at-least-once redelivery, where a message became undeliverable either
by being in flight or by exceeding its receive count. Here, a session
becomes invalid either by outliving its expires_at or by an explicit
SessionStore::revoke call ahead of that natural expiry; neither
condition implies the other, and Session::is_valid checks both.
SessionSource captures the two ways Phase 13's own scope named a
session can come to exist -- AssumedRole and Federated -- covering
both "sessions" and "federation" from cloud-identity's original
deferral in one type, rather than a separate cloud-federation crate
for what is, underneath, still just a session with a different origin.
cloud-policy-document's hand-rolled, deliberately narrow parser
Serializing a Policy to JSON and back needs something that
understands JSON syntax, and pulling in a general-purpose JSON crate
would break this workspace's zero-external-dependency posture --
exactly the same posture cloud-checksum (Phase 5) upheld by
implementing CRC-32 from scratch, and cloud-types::Arn (Phase 1)
upheld with its own hand-rolled Display/FromStr pair instead of a
generic serialization framework. cloud-policy-document::from_json is
a parser restricted to exactly the fixed schema a policy document
needs -- strings, arrays, and objects, no numbers, booleans, or null
-- not a general JSON value type. It also deliberately does not
replicate AWS's own polymorphism, where Principal/Action/Resource
may each be either a bare string or an array interchangeably: here
Action/Resource are always arrays, and Principal is either the
literal string "*" or an array, never a bare non-"*" string. This
is the same kind of deliberate scope-narrowing cloud-policy's own
matches_pattern already applies (a single trailing-wildcard position,
not a general glob engine) -- one shape per field is all this phase
needs.
Policy::statements, a small accessor added to an earlier crate
cloud-policy-document needs to read a Policy's statements to
serialize them, and Policy (Phase 1) never exposed a way to do that
from outside its own crate. Rather than duplicate that data somewhere
else, cloud-policy gained one narrow, backward-compatible accessor,
statements(&self) -> &[Statement] -- the same precedent
cloud-scheduler (Phase 2) already set when it gained
place_least_loaded_with_capacity in Phase 4: a later phase's
composition need is satisfied by adding to the crate that already owns
the data, not by duplicating it or routing around it.
What Phase 13 deliberately does not include
- No IAM service composing these three crates. Exactly as Phases
4-7 each built primitives with service composition deferred to
Phases 8-11, Phase 13 is primitives only; a service that creates
credentials/sessions/policy documents together, with quota, events,
and an
Arn-resolvable identity, is a future phase's concern. - No real cryptographic secret material, signing, or verification
for
cloud-credentials-- see above. - No
cloud-federationcrate. Federation is modeled as oneSessionSourcevariant incloud-session, not a fourth crate, since a federated identity's whole lifecycle is a session with a different origin. cloud-policy-documentdoes not support AWS's bare-string-or-array polymorphism forAction/Resource, and supports only the literal"*"(not an arbitrary bare string) forPrincipal-- see above.- No resource-based policies (a policy document attached to a
specific resource rather than a principal) --
cloud-resource::Resource<T>'s own Phase 1 doc comment already named this as a Phase 13 concept not pulled forward intoResource<T>itself, and this phase's policy documents remain principal-facing, the same shapecloud-policyalready evaluates against.
Phase 14: cloud-iam, the fifth composed service
Phase 13 built the IAM surface as three disjoint primitives, deliberately
without a service, the same shape Phases 4-7 used before their own
service phases arrived later. Phase 14 is that service: it composes
cloud-credentials, cloud-session, and cloud-policy-document
together with cloud-policy and cloud-events, following exactly the
"compose already-real primitives, introduce none" discipline
cloud-compute/cloud-storage/cloud-database/cloud-messaging
(Phases 8-11) each established.
| Crate | Owns |
|---|---|
cloud-iam |
IamService: create_credential/activate_credential/deactivate_credential/delete_credential/assume_role/federate/revoke_session/authorize/load_policy_document/export_policy_document, composing five existing crates |
The one real business rule: policy gates session creation
Every composed service so far has called Policy::authorize before
mutating -- but always to gate another service's own resource
operation (launch, create_volume, create_database, create_queue).
IamService::assume_role and IamService::federate are the first
operations in this workspace where cloud-policy (Phase 1) governs
something within the IAM surface itself: a principal now needs
permission to assume a role or federate in at all, checked before any
session is created, so a denied attempt leaves cloud-session's store
completely untouched. The mechanism is unchanged from every earlier
phase; what's new is that IAM's own operations are finally subject to
the same authorization discipline as everyone else's.
load_policy_document/export_policy_document: the live policy, round-trippable
IamService holds exactly one active Policy, and load_policy_document
replaces it immediately -- every subsequent authorize/assume_role/
federate call is governed by the newly loaded policy, not just parsed
data sitting inertly. export_policy_document is the inverse, so a
policy can be extracted, edited as JSON, and reloaded. A malformed
document leaves the previous policy in place: self.policy = ...from_json(json)?
only assigns once the parse has already succeeded, so a failed
load_policy_document call is a pure no-op on the active policy,
consistent with every other fallible mutation in this workspace leaving
state untouched on error.
No accounts, regions, quota, or Arn -- deliberately
Every prior composed service validated an AccountId/RegionId and
reserved a cloud-quota unit before creating anything, and registered
a resolvable Arn for what it created. IamService does none of
these: real IAM systems (AWS's own included) are inherently global,
not region-scoped, and neither a credential nor a session is
provisioned infrastructure with a finite regional capacity or a
canonical external name -- mirroring real IAM's own inconsistent
resource model, where a role or user has an ARN but an access key or
an STS session token does not. Omitting all four here is a deliberate
design choice mirroring what IAM actually is, not an oversight or a
gap to fill later.
What Phase 14 deliberately does not include
- No resource-based policy attachment.
IamServiceholds one policy governing every principal; attaching a distinct policy to a specific resource remains the Phase 13 (IAM) conceptcloud-resource's own Phase 1 doc comment named as not pulled forward intoResource<T>. - No credential-to-session linkage. A
Credentialand aSessionare tracked independently;IamServicedoes not model long-term credentials being exchanged for a short-term session the way real STSGetSessionTokendoes, sincecloud-credentialshas no secret material for such an exchange to authenticate against in the first place (Phase 13's own deferral). - No policy versioning or multiple named policies.
load_policy_documentreplaces the single active policy outright; there is no history of prior policies, no policy identifiers, and no way to attach different policies to different principals within oneIamService.
Phase 15: cloud-orchestration gets a second composition, and its first rollback
Phase 12 gave cloud-orchestration its first cross-service
composition (cloud-compute + cloud-storage, resolving
cloud-storage's own Phase 9 deferral). Phase 15 gives it a second,
in the same crate: cloud-compute + cloud-messaging, resolving a
deferral cloud-messaging itself named in its own Phase 11 closing
note -- "what remains deferred is composing these services together
(e.g. a compute instance's logs delivered through a queue)."
| Crate | Owns |
|---|---|
cloud-orchestration |
Adds transition_runtime_and_notify/terminate_and_notify alongside Phase 12's attach_volume/detach_volume |
Why this composition needs a rollback and Phase 12's didn't
Phase 12's doc comment explained, at some length, why attach_volume
needed no rollback: once its instance check passed, cloud-attachment's
own transition table made the second step (Attaching -> Attached)
unconditionally valid, so there was no failure mode between the two
steps to roll back from. transition_runtime_and_notify and
terminate_and_notify are different: a RuntimeState transition
genuinely can fail (an invalid jump, an unknown instance), and unlike
cloud-attachment's state machine, RuntimeState transitions are not
generally reversible -- there is no transition that undoes an arbitrary
runtime-state change the way Detaching -> Detached undoes
Attaching. That asymmetry decides the ordering: both functions
enqueue the notification message into cloud-messaging first, then
attempt the compute mutation, then -- only if that mutation fails --
delete the message they just enqueued via MessagingService::delete_message.
A failed transition or termination therefore never leaves a stray
notification sitting in a queue, and this is the first rollback
anywhere in this workspace that spans two independent services rather
than undoing steps within one.
What this composition still doesn't do
- No message body describing the event. Exactly as
cloud-messaging(Phase 11) itself deferred, a message is just aResourceId; the caller chooses amessage_idthat means whatever it needs to mean (an event id, a correlation id), andcloud-orchestrationdoes not interpret it. - No notification on
launchor on volume attach/detach. This phase covers exactly the two operations its motivating deferral named (an instance's own lifecycle events); extending notification to other operations or other services is a separate question with no equivalent stated deferral yet. - No fan-out to multiple queues. Both functions take exactly one
queue_id; delivering the same notification to several queues would needcloud-messaging's own topic/subscriber model (publish), not a directenqueue, and composing that is left for whenever a concrete need for it exists.
Phase 16: cloud-orchestration gets a third composition, with database operations
Phase 15 composed cloud-compute with cloud-messaging. Phase 16
adds a third composition in the same crate: cloud-database +
cloud-messaging. Both [apply_migration_and_notify] and
[delete_database_and_notify] follow the same enqueue-first,
rollback-on-failure discipline, since database operations like schema
migrations can fail (an invalid version jump, incompatible changes) and
are typically not reversible (schema changes are one-way; deletions are
final).
| Crate | Owns |
|---|---|
cloud-orchestration |
Adds apply_migration_and_notify/delete_database_and_notify alongside Phase 12's attach/detach and Phase 15's compute+messaging |
Why database operations need the same rollback pattern
Unlike [attach_volume]'s second step (which cloud-attachment's own
transition table made unconditionally valid once the first succeeded),
and matching Phase 15's reasoning for RuntimeState transitions:
database mutations can fail independently. Applying a migration version
requires the prior version to exist; deleting a database requires all
snapshots to have expired. These operations are also not generally
reversible -- there is no unapply_migration that undoes an arbitrary
schema change, and deletion is final. So both functions enqueue the
notification message first, then attempt the database mutation, then
delete the message if the mutation fails. This mirrors Phase 15 exactly,
and is the second (and second-and-third) composition in this workspace
spanning two independent services.
What this composition still doesn't do
- No message body describing the event. Exactly as Phase 15 deferred
this, a message is just a
ResourceId. - No notification on
create_databaseor other database operations. This phase covers exactly the two operations with clear, stated use cases (migration and deletion); extending to other operations or other services is a separate question. - No fan-out to multiple queues. Both functions take exactly one
queue_id, matching Phase 15's scope limitation.
Definition of done, per crate
Reusing the roadmap's own maturity levels, scoped to what a primitive crate (not a service) can claim:
- L1 (data model) — the types exist, are validated at construction,
and round-trip through their own
Display/FromStr(or equivalent). - L2 (deterministic local implementation) — the crate's actual logic (a state-machine transition, a policy evaluation, a quota check) is implemented and produces the same result for the same input every time, with no hidden clock/randomness/iteration-order dependency.
- L8 (tests) — every invariant claimed above has a passing test that would fail if the invariant were violated (not just a compiles-and-returns-something test).
Every crate in this phase targets L1+L2+L8. Persistence (L5),
distribution (L4), and formal verification (L9/L10, eventually via
verification-forge) are explicitly out of scope until the control
plane (Phase 2) gives them something real to attach to.
Relationship to verification-forge
Per the roadmap: verification-forge should eventually verify
cloud-forge's invariants (resource-id uniqueness, deny-dominates
policy evaluation, lifecycle transitions being total functions, quota
violations never committing), not become a runtime dependency of it.
That work is not started in this phase — it requires cloud-forge's
own resource/lifecycle/policy model to exist and stabilize first, which
is exactly what this phase builds.
Quality gates
Same discipline as the other two workspaces in this repository: no
crate is considered done until cargo test -p <crate> --release
passes, cargo clippy -p <crate> --all-targets -- -D warnings is
clean, and cargo fmt -p <crate> -- --check is clean — checked after
every crate, not batched at the end. #![forbid(unsafe_code)] in every
crate. No unwrap()/expect() on anything derived from a caller's
input -- a malformed resource id or an invalid policy input is exactly
the kind of thing this substrate has to reject cleanly (a CloudError)
rather than panic on. The only expect() calls in this phase
(cloud-region's AZ-name construction, cloud-types::AzId::region())
are on values built from a already-validated piece of the same type's
own state -- provably unreachable to fail, and each carries a comment
saying so -- never on anything a caller supplied that hasn't already
gone through that type's constructor.