Add files using upload-large-folder tool

13c5606 verified about 1 month ago

32.1 kB

SpecForge Architecture Plan (Reconciled)

Status: Draft — reconciles the original "SpecForge Redesign Plan" (a from-scratch, torchrun-native, HTTP-only target architecture) with the landed DataFlow runtime/ spine (control/data/inference/training planes, SampleRef + FeatureStore + Mooncake).

Supersedes the previous redesign draft. That draft's detailed component sketches (CheckpointManager, Evaluator, Pydantic config, MLA draft, SGLang export, train-with-decode) remain valid as the domain-layer target and are preserved verbatim in docs/redesign-draft-legacy.md; this document re-frames where they sit now that the DataFlow runtime exists, and corrects the one bet the original draft got wrong for our actual workloads (see §6).

0. TL;DR — the decision

There were two architecture efforts in flight for the same goal (organize SpecForge so new draft models / training modes / disaggregation compose cleanly without a train_XXX.py per method):

The redesign draft (this doc's predecessor): clean domain decomposition (models/drafts registry, TargetEngine, HiddenStateStream, DraftTrainStrategy, Trainer, config/eval/export/cli) — but explicitly torchrun-native with no Ray/Mooncake; disaggregation was "the target is an HTTP service" (RemoteStream).
The DataFlow runtime/ spine (landed: the dataflow-up-1..20 stack, M1–M7/O1): a metadata-only control plane + a tensor-carrying data plane (SampleRef + FeatureStore incl. Mooncake + StreamingRefChannel + durable metadata + backpressure
- resume), with disaggregated producer/consumer and an interleaved online loop.

The fact that settles it: there is a real multi-node / >100 GB/s / isolated-pool requirement (separate rollout and trainer pools, no shared FS / no viable HTTP path for the hidden-state volume). That breaks the original draft's central bet ("HTTP/gRPC to SGLang is sufficient; Mooncake only matters >100 GB/s which we never hit"). So the runtime's Mooncake control/data plane is necessary, not premature.

Decision: do not pick one — layer them.

Canonical substrate = the DataFlow runtime/ control + data plane. SampleRef (metadata on the control plane) + FeatureStore (tensors, Local/SharedDir/Mooncake) + FeatureDataLoader → TrainBatch is the canonical data path. online / offline / disaggregated converge at SampleRef.
Front-end = the redesign draft's domain layer. TargetEngine / DraftTrainStrategy / Trainer / CheckpointManager / Evaluator / config / eval / export / cli become the user-facing surface. The canonical "stream" is FeatureDataLoader over SampleRef + FeatureStore — we do not re-introduce a HiddenStateStream as a separate source of truth (the loader already plays that role). The online/offline/disaggregated variation lives in the ref source (OfflineManifestReader / RolloutWorker / StreamingRefChannel) + the transport (FeatureStore: Local for colocated, Mooncake/SharedDir for isolated pools), and is shielded from training by FeatureDataLoader → TrainBatch. (RemoteStream-over-HTTP is just one more ref source / engine — the light cross-node option, not the base abstraction.)

This is, in the predecessor's own terms, pulling feature #32 ("Mooncake streaming backend") forward from Phase 7 to now because the requirement demands it — and implementing it with the already-built runtime/ work, behind the clean domain interfaces.

Two scope decisions that bound this plan (consolidating with the online-disaggregation roadmap, #618):

Frozen target — no weight sync. "Train-with-decode" means a frozen target streams hidden states; the draft is never in the generation loop. Weight sync / hot draft-update / weight-version registry / on-policy are out of scope; draft_weight_version is provenance only. (This removes the predecessor's W4 weight-lifecycle workload.)
Ray is an open decision, not a non-goal. It is a candidate for the O2 scale-out orchestrator (multi-node N-producer/M-trainer) — likely necessary, but not committed. Until the decision gate fires we keep the home-grown metadata-only control plane and add nothing.

Per-phase target/implementation detail lives in docs/roadmap/ (domain, online-disaggregation [folds in #618], eval & breadth); this document is the architecture; the roadmap is the build order.

1. Why each side is right (and where each is incomplete)

Two orthogonal slicings. The runtime is sliced by plane (how bytes/metadata move); the draft is sliced by domain object (draft / target / stream / strategy). They compose; they do not conflict.

Layer	Verdict	Rationale
control plane (`runtime/control_plane`)	Keep — canonical. No predecessor equivalent.	Metadata-only (every entrypoint runs `assert_no_tensors`), lease/ack/dedup/reconcile/backpressure behind a `MetadataStore` seam. This is exactly the cross-pool machinery the isolated-pool requirement needs, and the HTTP design has nothing like it.
data plane (`runtime/data_plane`)	Keep — canonical; it is the stream.	`SampleRef` (metadata) ÷ `FeatureStore` (tensors) ÷ `FeatureDataLoader` (materialize) is a finer decomposition than the predecessor's coarse `HiddenStateStream`, and it's the better substrate for backpressure / lease-ack / resume / Mooncake. `FeatureDataLoader` over `SampleRef`+`FeatureStore` already plays the role plan.md gave `HiddenStateStream`, so no separate `HiddenStateStream` source-of-truth is introduced — `seek()`/prefetch land on the loader/ref-source; `FeatureStore` is the transport swap point.
inference plane (`runtime/inference`)	Converge to the predecessor's `TargetEngine`.	Today it conflates "wrap a target engine" + "capture" inside `SGLangAdapter`, and is bound to `generate_eagle3_data` / EAGLE3 names. The `FeatureSource` Protocol already exists (good); the gap is a real `TargetEngine` abstraction (hf / sglang / sglang_server / custom) + de-EAGLE3-ifying. This is the layer that most borrows the draft.
training plane (`runtime/training`)	Keep the seam; fill the managers.	`DraftTrainStrategy` / `TrainerCore` / `TrainingBackend` (+ the `StepContext` added for Domino) is already the predecessor's `training/strategies` shape — keep it. What's missing are the managers, not the seam: `no_sync()` accumulation, full optimizer/scheduler/RNG resume, `CheckpointManager` (rotation/best), `Evaluator`.

Landed (merged — the spine):

runtime/ planes (M1–M7/O1): control/data/inference/training, SampleRef, FeatureStore (Local/SharedDir/Mooncake), StreamingRefChannel, durable MetadataStore, disagg producer/consumer + interleaved online loop.

In the stacked PRs #627/#628/#629 (validated, in review — not yet merged):

Composable launch (StrategySpec registry + parameterized launch.py): adding a model is a spec entry, not a build_*_runtime family. eagle3 / dflash / domino all train end-to-end through one strategy-parameterized path (validated: 197 tests/test_runtime OK on H200). Domino added StepContext{global_step, total_steps} threaded through forward_loss — the one deliberate contract extension for schedule-dependent loss.

Explicitly not yet implemented in runtime/ (flagged in-source — contracts.py, trainer.py, controller.py, both DESIGN.mds, runtime/README.md): live frozen-target online capture from a real SGLang server (O1.3), full optimizer/scheduler resume, and no_sync() accumulation. These are the §3 gaps. (The in-source weight-publication NOTEs — WeightVersion / WeightPublisher / update_draft_weights — are descoped: the target is frozen, see §8.)

2. Target architecture

2.1 Canonical data path (unchanged from the runtime, this is the spine)

ref source            ─┐
  OfflineManifestReader │  (offline: re-iterable refs)
  RolloutWorker         │  (online: produced into a SampleRefQueue)
  StreamingRefChannel   │  (disaggregated: cross-process/pool stream)
                        ▼
DataFlowController  ── metadata only (SampleRef), assert_no_tensors
                        │
FeatureStore  ── tensors only: LocalFeatureStore (mem://, colocated)
                              SharedDirFeatureStore (shared mount)
                              MooncakeFeatureStore (RDMA, isolated pools)
                        ▼
FeatureDataLoader (per_sample_transform + collate) ─► TrainBatch
                        ▼
DraftTrainStrategy.forward_loss(batch, ctx)  ─► TrainerCore / TrainerController ─► FSDP

2.2 Domain layer on top (user-facing abstractions over the substrate)

The substrate's source of truth is SampleRef + FeatureStore + FeatureDataLoader (§2.1). The domain layer does not re-introduce a HiddenStateStream as a parallel source of truth — FeatureDataLoader already is the stream. The domain layer is what user/training code sees:

PRODUCE (inference):
  TargetEngine        ── the engine rollout wraps to produce features; the abstraction that
                         replaces the EAGLE3-bound SGLangAdapter.
     HFTargetEngine / SGLangTargetEngine / CustomTargetEngine (in-process)
     SGLangServerEngine (frozen target as a live SGLang server, cross-node). ONE engine, two
       feature transports (the engine is identical; only WHERE features land differs):
         · capture transport — engine-side hook writes hidden states INTO a FeatureStore
           (Mooncake/SharedDir). This is W3 / online O1.3.
         · inline-HTTP transport — features serialized in the HTTP response, no shared store
           (RemoteStream-style). This is the light W3′ path, the one case features do NOT live
           in a FeatureStore.
  RolloutWorker       ── drives a TargetEngine → writes tensors to FeatureStore → commits
                         SampleRef to the control plane. Stays at the domain↔substrate seam.

CONSUME (training):
  Trainer             ── owns the lifecycle: loop / eval / checkpoint. WRAPS the
                         TrainerCore/TrainerController seam; does NOT replace it. (No weight-sync —
                         the target is frozen, §8.)
  DraftTrainStrategy  ── per-algorithm forward+loss (eagle3 TTT / dflash block / domino). The
                         per-step seam; lives in training/strategies (relocated from
                         runtime/training in E0) — already plan-shaped.
  CheckpointManager / Evaluator / lr_scheduler / fsdp seam  ── the managers (G1).

COMPOSE:
  modeling/draft      ── DRAFT_REGISTRY (@register_draft) for draft *architecture* classes
                         (llama / deepseek-MLA / dflash-qwen3). Separate axis from strategy.
  StrategyRegistry    ── per-algorithm spec (today's StrategySpec, converged here).
  config / cli / export ── first-class run surface (predecessor §4.3–4.8, carried forward).

The online / offline / disaggregated distinction is not visible to Trainer/strategy — it is fully absorbed by (ref source + FeatureStore) behind FeatureDataLoader → TrainBatch.

2.3 Module layout (runtime spine + domain layer)

specforge/
├── runtime/                         # SUBSTRATE ONLY — the canonical DataFlow spine (keep as-is)
│   ├── contracts.py                 # SampleRef, TrainBatch, PromptTask, *Strategy literal
│   ├── control_plane/               # metadata-only: controller, metadata_store, backpressure
│   └── data_plane/                  # FeatureStore (Local/SharedDir/Mooncake), FeatureDataLoader,
│                                    #   SampleRefQueue, StreamingRefChannel, offline/disagg readers
│
├── inference/                       # TOP-LEVEL — the single home for all rollout/capture execution
│   ├── rollout_worker.py            #   (from runtime/inference/rollout_worker.py)
│   ├── capture.py                   #   CaptureConfig (from runtime/inference/capture.py)
│   ├── feature_source.py            #   the FeatureSource Protocol (the worker's only contract)
│   ├── target_engine/               #   TargetEngine, EXTRACTED from modeling/target/*TargetModel
│   │   ├── base.py  factory.py       #     (adapters then wrap an engine, §G2)
│   │   ├── hf.py  sglang.py  sglang_server.py  custom.py   # per-backend generic engines
│   │   └── sglang_capture_backend.py                       # sglang version-pinned glue
│   └── adapters/
│       └── eagle3.py  dflash.py     #   (from runtime/inference/{sglang,dflash}_adapter.py)
│
├── training/                        # TOP-LEVEL — the single home for all training execution (NO facade)
│   ├── trainer.py                   # domain Trainer: owns loop/eval/checkpoint (B3)
│   ├── controller.py                # TrainerCore + TrainerController seam (from runtime/training/trainer.py)
│   ├── backend.py                   # FSDPTrainingBackend + no_sync (from runtime/training/backend.py)
│   ├── checkpoint.py  resume.py  evaluator.py   # NEW managers (§3) — born here
│   └── strategies/                  # StrategySpec registry + per-algorithm strategies (converges HERE)
│       ├── base.py                   #   DraftTrainStrategy ABC + StepContext (from runtime/training/strategy.py)
│       ├── registry.py               #   StrategySpec / resolve_strategy (from runtime/training/registry.py)
│       └── eagle3.py  dflash.py  domino.py
│
├── modeling/                        # MODEL DEFINITIONS ONLY — no orchestration, no capture factory
│   ├── draft/                       # DRAFT_REGISTRY + @register_draft (NEW registry.py — predecessor §4.2)
│   │   └── base.py  registry.py  llama3_eagle.py  deepseek_eagle3.py(MLA)  dflash.py  flex_attention.py
│   └── target/                      # target model nn.Modules + model-specific glue ONLY
│       └── target_head.py  custom_backend/  sglang_backend/   # (capture orchestration moved to inference/)
│
│                                    # (NO separate data/streams package — FeatureDataLoader over
│                                    #  SampleRef+FeatureStore IS the stream. Ref sources
│                                    #  (offline/rollout/streaming) live in runtime/data_plane;
│                                    #  live frozen-target capture is just another ref source)
│
├── launch.py                        # TOP-LEVEL — topology assembly only (spec-driven builders; from runtime/launch.py)
├── eval/  export/  config/  cli.py  # NEW — carried forward from predecessor §4.4–4.8
└── core/  optimizer.py  tracker.py  distributed.py  lr_scheduler.py  # kept verbatim (predecessor §1)

One implementation home per concern; runtime/ is substrate-only. The two structural moves (both gated by the byte-identical suite, executed as the move-only step E0 — see docs/roadmap/domain-refactor.md): (1) extract TargetEngine out of the EAGLE3-bound modeling/target/*TargetModel into top-level inference/target_engine/ (the adapters/ then wrap an engine); (2) lift the training seam (TrainerCore/TrainerController/TrainingBackend/strategy+registry) up into top-level training/. runtime/ shrinks to the substrate (control + data plane + contracts) — it is not a home for domain/algorithm code, and there is no facade package. The control + data planes stay exactly where they are; the online/offline/disaggregated distinction stays absorbed by (ref source + FeatureStore) behind FeatureDataLoader, invisible to training/.

3. Gaps to close (prioritized) — what the domain layer adds on top of the spine

These are the items the landed runtime/ does not have, drawn from the predecessor and from the in-source NOTEs. Each lands behind the canonical spine without re-plumbing it.

G1 — Training managers (highest leverage; the spine works but is bare)

no_sync() accumulation. runtime/training has zero no_sync() — FSDP all-reduces every micro-step, defeating accumulation_steps. (Predecessor §4.2 #10; verify with one profiler check: one all-reduce per optimizer.step().)
Full resume. save_checkpoint persists training state only; no optimizer/scheduler/RNG restore, no rotation, no best-tracking. Add CheckpointManager (predecessor §4.3) + a seek()-equivalent on the colocated streams.
Evaluator + EvalCache. No simulated_acc_len / per-position-acc / best-checkpoint. (Predecessor §4.4 — per-position acc aggregated across batches before the geometric sum.)
lr WSD + per-strategy fsdp_wrap_policy() (predecessor §4.6, §4.2).

G2 — `TargetEngine` abstraction (inference convergence)

Introduce a TargetEngine ABC extracted from the modeling/target/*TargetModel classes (Eagle3TargetModel, DFlashTargetModel); the runtime/inference adapters (SGLangAdapter/DFlashAdapter) then wrap a TargetEngine rather than being the engine, and stop binding to generate_eagle3_data / EAGLE3 names. Keep the existing FeatureSource Protocol. Add SGLangServerEngine (live SGLang server) as the cross-node engine — see §2.2 for its two feature transports (capture-into-FeatureStore for W3/O1.3, inline-HTTP for the light W3′).

G3 — Live online capture (frozen target; no weight sync)

Replace the in-process generator with live SGLang-server hidden-state capture: a frozen target streams aux+final hidden states into MooncakeFeatureStore; the producer commits SampleRefs. The cross-process control plane + async loop are in-review; live capture (the gating spike) is next; scale-out (Ray = open) and hardening follow.
Weight sync / hot draft-update / on-policy are explicitly out of scope. The target is frozen, so the streamed data is independent of draft weights — there is nothing to re-sync and no staleness. draft_weight_version is kept only as provenance.
Detailed phases (O1–O3): docs/roadmap/online-disaggregation.md.

G4 — Composition + models (predecessor Phase 1)

modeling/draft DRAFT_REGISTRY (@register_draft) for draft architectures (separate axis from strategy); MLA Eagle3 draft (deepseek/Kimi); converge StrategySpec → training/strategies registry. Note: draft-arch registry and strategy registry are two registries, not one.

G5 — Run surface (predecessor Phase 3)

Pydantic config/ + specforge CLI; export/to_sglang (+ the documented MLA weight-name map) and export/to_hf with vocab pruning.

4. Topologies & the colocated lightweight path

The workloads differ only in which ref source + FeatureStore + TargetEngine compose. All consume one FeatureDataLoader → TrainBatch iterator; the trainer/strategy/backend are identical regardless (no per-topology stream class). In every case the target is frozen — the draft is never in the generation loop, so the streamed features do not depend on draft training progress (no staleness, nothing to re-sync).

#	Workload	Ref source (→ FeatureDataLoader)	FeatureStore	Control plane
W1	Offline (precomputed)	`OfflineManifestReader` (refs)	Local (`file://`/`mem://`)	no-op
W2	In-process online (frozen target)	`RolloutWorker` → `SampleRefQueue`	Local (`mem://`)	no-op
W3	Disaggregated online (frozen target; isolated pools, high BW)	`RolloutWorker` (producer pool) → `StreamingRefChannel`, `SGLangServerEngine` capture transport	Mooncake	active (lease/ack/reconcile/backpressure)
W3′	Disaggregated online (light/cross-node)	`SGLangServerEngine` inline-HTTP transport (RemoteStream-style source)	n/a — features inline over HTTP, no shared store	minimal

"Train-with-decode" = live frozen-target generation — i.e. W2 (colocated) or W3 (disaggregated), not a separate workload. The predecessor's dual-purpose serve-and-push-weights "W4" is out of scope: a frozen target means there is nothing to push. See docs/roadmap/online-disaggregation.md.

Colocated lightweight path (W1/W2): keep it, but as a no-op control plane, not a fork. The control-plane machinery (lease/ack/reconcile/backpressure/durable metadata/cross-process transport) only earns its keep when producer and consumer are decoupled. For W1/W2 (same process) it is pure overhead. Resolution:

Single canonical path through SampleRef + FeatureStore, so colocated and disaggregated produce byte-identical batches for free (today's property — "disaggregation changes where features live, not their values"). We do not fork a second trainer/launch path.
"Lightweight" = for colocated, the controller's lease/ack/metadata-store are opt-in / no-op, FeatureStore is LocalFeatureStore(mem://), and there is no SQLite, no cross-process, no backpressure. The colocated ref sources (OfflineManifestReader / in-process RolloutWorker) flow through the spine but skip the heavy bits.
Cost to own honestly: this preserves the free byte-identical guarantee only if we keep one path. A numerical-equivalence gate (§7) locks "colocated stream output == disagg stream output" so the no-op path can never silently diverge.

5. What changed from the predecessor draft (explicit deltas)

"No Mooncake in Phase 1-6" → revised. The Mooncake control/data plane is now the canonical disaggregation backend, because the isolated-pool / >100 GB/s requirement is real and HTTP cannot serve it. The predecessor's RemoteStream-over-HTTP remains as the light second backend (W3′). Mooncake is used as a transport; the scale-out orchestration layer (Ray vs. home-grown) is an open decision (see §6, §8 and docs/roadmap/online-disaggregation.md §O2), no longer a flat non-goal.
Primary abstraction / HiddenStateStream dropped. Predecessor: a coarse HiddenStateStream produces TrainBatch and is the source of truth. Reconciled: SampleRef + FeatureStore + FeatureDataLoader is canonical, and FeatureDataLoader already is the stream — so HiddenStateStream is not introduced as a parallel abstraction. Topology variation lives in (ref source + FeatureStore); seek()/prefetch land on the loader/ref-source. This is strictly finer and is what makes lease/ack/backpressure/Mooncake clean.
runtime/ is not "thrown away." The predecessor's "what we throw away" never mentioned runtime/ (it predated it). The DataFlow spine is the foundation, not debt.
Module layout gains a top-level runtime/ (the spine) under the domain layer; models/ (drafts + targets), training/ (Trainer + managers), eval/, export/, config/, cli.py are the layer on top. No separate data/streams/ package — the loader is the stream.
W4 weight lifecycle → dropped. The predecessor's serve-and-push-weights workload + weight registry are out of scope. "Train-with-decode" is frozen-target live generation (W2/W3); weight sync / hot draft-update / staleness gate / on-policy are explicitly cut, aligning with #618. draft_weight_version survives only as provenance metadata.
Ray reframed from non-goal → open. It is a candidate for the O2 scale-out orchestrator (multi-node N-producer/M-trainer), not committed and not forbidden; the decision gate lives in the roadmap.

Everything else in the predecessor (workloads W1–W3, the domain abstractions, the testing discipline, the MLA/export/eval/config detail) carries forward unchanged.

6. Tradeoffs (updated)

Mooncake transport vs HTTP/gRPC — transport bet reversed

The predecessor bet "HTTP is sufficient, Mooncake is gated behind profiling." For the data transport that bet is now wrong (isolated-pool / >100 GB/s requirement) — Mooncake is in, canonically. The RemoteStream-over-HTTP path stays as the light cross-node backend (W3′).

Scale-out orchestration: Ray vs. home-grown — OPEN

Today's metadata-only control plane handles a single producer-pool ↔ trainer scope. O2 (multi-node N-producer/M-trainer scale-out) needs an orchestration layer, and whether that is Ray or a home-grown scheduler is undecided — likely necessary, but not committed. It is not a non-goal anymore; the decision gate (when ≥1 of: >1 producer pool, cross-node failover, multi-job pool sharing) lives in docs/roadmap/online-disaggregation.md §O2. Until then we keep the home-grown control plane and add nothing.

One canonical path vs a light colocated fork

Chosen: one path (spine everywhere) + control-plane-as-no-op for colocated. Keeps byte-identical free and avoids two implementations; costs an equivalence gate. (See §4.)

Two registries, not one

modeling/draft DRAFT_REGISTRY (architecture) and training/strategies (algorithm) are separate axes — an algorithm (eagle3) runs on multiple draft architectures. Don't merge them; the current StrategySpec is the strategy registry and should converge there, not into modeling/draft.

(The predecessor's other tradeoffs — Pydantic vs OmegaConf, MLA cache compressed vs expanded, registry vs HF AutoModel — are unchanged and carry forward.)

7. Migration path (incremental, gated)

The spine is already landed, so migration is "grow the domain layer on top + close the gaps," not a rewrite. Every phase that touches the training path passes the numerical-equivalence gate (below) against the prior commit. Per-phase target/implementation/tests/done-when detail lives in docs/roadmap/ — the phases below are the index; the online track there also folds in the former online-disaggregation roadmap (#618).

Phase A — composable launch (in review). StrategySpec registry + parameterized launch.py; eagle3/dflash/domino end-to-end in the stacked PRs #627/#628/#629 (validated, in review). 197 tests/test_runtime OK.
Phase B — domain abstractions (no behavior change). Introduce TargetEngine (wrap the existing adapters; de-EAGLE3 the names) and the domain Trainer wrapping the kept runtime/training core. No HiddenStateStream — FeatureDataLoader stays the stream. Gate: byte-identical batches/loss vs the pre-refactor path.
Phase C — colocated no-op control plane + equivalence gate. Make lease/ack/metadata opt-in; add the colocated≡disagg equivalence test.
Phase D — training managers (G1). no_sync(), full optimizer/scheduler/RNG resume, CheckpointManager, Evaluator. Gate: loss/eval parity at fixed steps.
Phase E0 — layout consolidation (move-only). Relocate the training seam + target engine into the top-level training/ / inference/ homes; runtime/ shrinks to substrate. Pure git mv + import shims; gate: suite + B byte-identical gate unchanged (no functional diff).
Phase E — modeling/draft registry + MLA draft + config/CLI/export (G4/G5). Adding a draft arch = one decorated file; one validated YAML per run; export-loop test.
Online track (parallel; G3) — live frozen-target capture. O1.1 shared control plane + O1.2 async loop (in review) → O1.3 live SGLang-server hidden-state capture (next; gated by a throughput spike) → O2 scale-out (orchestrator: Ray = open) → O3 hardening. No weight sync. Detail: docs/roadmap/online-disaggregation.md.
Eval track (parallel) — E1 acceptance-length eval harness → E2 algorithm breadth (new algo = a StrategySpec + loss). Detail: docs/roadmap/eval-and-breadth.md.

Doc debt to fix alongside Phase B: revise the predecessor's "No Mooncake / HTTP is sufficient" statements (now §5/§6) so the code and the plan stop contradicting each other.

8. Non-goals (updated)

No weight sync / hot draft-update / on-policy training. The target is frozen; the draft is never in the generation loop. No WeightPublisher, no weight-version registry, no staleness gate. draft_weight_version is provenance metadata only.
No two-stack fork for colocated. One canonical data path; colocated is the spine with the control plane as a no-op, not a parallel implementation.
No vLLM target backend (possible via TargetEngine, not prioritized).
No multi-engine load balancing / multi-job inference-pool sharing for now — gated behind ≥5 concurrent jobs sharing one target.

Open decisions (NOT non-goals):

Ray (or a home-grown scheduler) for O2 scale-out — likely necessary for multi-node N-producer/M-trainer; undecided. Decision gate in §6 / the online roadmap.

(Reversed from the predecessor: "No Mooncake" → Mooncake transport is now canonical for W3; "No Ray" → Ray is now an open scale-out decision, not a flat non-goal.)

9. Success criteria

Area	Criterion
Composable launch (in review)	eagle3/dflash/domino train via one strategy-parameterized path; full `test_runtime` green.
Abstractions (B)	`TargetEngine` (wrapping existing adapters) + domain `Trainer` produce byte-identical batches/loss vs the direct spine path.
Colocated (C)	W1/W2 run with control plane as no-op; colocated≡disagg equivalence gate passes.
Managers (D)	resume reproduces the no-resume loss curve; one all-reduce per optimizer step; best-checkpoint tracked.
Drafts/MLA (E)	MLA Eagle3 trains + loads in SGLang; new draft arch = one `@register_draft` file.
Online (O1.3)	a live frozen-target SGLang server feeds training with zero precomputed features; loss/eval matches the offline baseline on the same prompts+seed. (Scale-out O2 / Ray = open.)

10. Testing strategy (gating)

Carries the predecessor's gates, plus the reconciliation-specific one.

10.1 Numerical-equivalence gate (per PR touching core/ / training/ / runtime/data_plane/ / modeling/draft): fixed seed, 3×4 batches; per-step loss atol/rtol=1e-4 at steps 0/1/100/500/1000; per-position eval acc + simulated_acc_len atol=1e-3.
10.2 Colocated ≡ disaggregated gate (NEW): same data through the colocated (no-op control plane, Local store) and the disaggregated (control plane, SharedDir/Mooncake) paths must yield identical TrainBatches and identical loss. This is what licenses the lightweight path.
10.3 Smoke tests: each draft arch trains 20 steps under TP=1 / TP=2 / TP=2+SP=2; checkpoint save+resume matches no-resume (validates seek); eval cache miss==hit.
10.4 Distributed correctness: MLA + Yunchang USP with asymmetric head dims; gradient accumulation = one all-reduce per optimizer.step().
10.5 Export-loop test: train MLA draft 100 steps → export → load in SGLang → 32 gens, acceptance > 0 (catches weight-name-map regressions).
10.6 Online-capture parity gate (O1.3): a live frozen-target capture run produces features + loss matching the offline-precomputed baseline on the same prompts/seed (the target is frozen, so this must hold exactly up to nondeterminism tolerance). No weight-sync gate — weight sync is out of scope.