docs/architecture.md

# Architecture

InferScale-Sim separates **serving-system logic**, **analytical latency estimation**, and **research methodology**.

## Main Python modules

1. `workloads.py` creates deterministic synthetic workloads or exact trace-replay requests.
2. `simulator.py` implements the colocated serving loop.
3. `disaggregated.py` implements separate prefill/decode worker pools plus KV transfer.
4. `kv_cache.py` handles memory/admission and shared-prefix allocation.
5. `latency.py` predicts reference prefill/decode operation durations and exposes sensitivity scales.
6. `metrics.py` derives TTFT, TPOT, E2E, queueing, throughput, goodput, and SLO attainment.
7. `diagnostics.py` converts simulated telemetry into explicit heuristic bottleneck labels.
8. `optimizer.py` implements capacity search, scheduler comparison, topology/cache comparison, and Pareto sweeps.
9. `research.py` implements paired common-seed A/B studies, bootstrap intervals, and analytical-model sensitivity analysis.
10. `agentic.py` models multi-turn programs, tool gaps, session routing, KV retention/TTL eviction, host tiering, and online tool-gap prediction.
11. `execution.py` models online agent-role transition learning, calibration, multi-step forecast planning, and bounded static-prefix prefetch under a shifting workflow distribution.
12. `consolidation.py` repeats execution-policy studies across matched seeds, computes bootstrap uncertainty and Pareto stability, and evaluates regret to a bounded full-trace oracle family.
13. `measurements.py` normalizes external vLLM/SGLang-style benchmark artifacts and fits simple train-only timing-scale calibration.
14. `validation.py` compares baseline or calibrated simulator predictions against externally supplied measured cases.
15. `reports.py` exports the consolidated study and optional empirical calibration as Markdown.
16. `api.py` exposes JSON-like actions to local Python and Pyodide.

## Colocated path

```text
arrival -> waiting -> prefill -> active decode batch -> complete
```

## P/D path

```text
arrival
  -> prefill queue
  -> prefill worker batch
  -> KV-transfer link
  -> decode-ready queue
  -> continuous decode worker
  -> complete
```

The P/D path is a genuine discrete-event loop: prefill workers, decode workers, and the transfer link can overlap in virtual time.


## Stateful agent-session path

```text
session arrival
  -> turn ready
  -> route to replica
  -> [KV hit: append prefill | KV miss: full-history prefill]
  -> decode
  -> retain / TTL / evict KV
  -> tool gap
  -> next turn ready
  -> ...
  -> session complete
```

Each replica is intentionally a serial service station in this mode. This isolates state residency, routing locality, and tool-gap effects from the dynamic-batching questions already covered by the request-level simulators.

## Research path

```text
base configuration
      |
      +--> paired A/B study --> shared seeds --> paired deltas --> bootstrap CI
      |
      +--> sensitivity study --> shared latency perturbations --> ranking/SLO stability
      |
      +--> repeated-seed policy study
      |       |
      |       +--> TTFT win rate / bootstrap CI / worst seed
      |       +--> Pareto stability
      |       `--> bounded full-trace oracle --> policy regret
      |
      +--> external serving artifacts
      |       |
      |       +--> normalize cases
      |       +--> train-only timing-scale fit
      |       `--> held-out residuals / MAPE
      |
      `--> Markdown research report
```

The simulator and statistical layer are separate on purpose: research conclusions are derived from repeated simulations rather than from one displayed run. The bounded oracle is exhaustive only over a declared family of deployable and clairvoyant future-set plans; it is not a proof of globally optimal cache scheduling.

## Browser execution

Canonical source lives in `src/inferscale`. `scripts/sync_web_python.py` mirrors it into `py/inferscale`. A module Web Worker loads Pyodide, writes those files into the virtual filesystem, imports `inferscale.api`, and exchanges JSON messages with the UI.

The simulator therefore runs away from the browser main thread and uses the same Python implementation as the local tests.

## Extension boundary

`AnalyticalLatencyModel` can later be replaced by an empirical profile interpolator exposing:

- `prefill_seconds(token_counts)`
- `decode_step_seconds(context_lengths)`
- `model_weight_gb`
- `kv_bytes_per_token()`

without rewriting workload generation, scheduling, cache logic, P/D orchestration, SLO metrics, or research protocols.

## Agent memory tiering

Stateful Stateful Sessions has an additional memory path that is independent from the stateless/P-D simulator:

```text
turn completes
   |
   +-- retain HBM --------------------------+
   |                                        |
   +-- TTL -> expire / pressure evict       | next turn
   |                                        |
   +-- host offload -> host KV -> restore --+
   |                                        |
   +-- evict -> history recomputation ------+
```

A global host tier models capacity, residency, offload/restore volume, and transfer latency. A bounded-affinity router can trade cached-replica locality against estimated queue imbalance.


## Execution-learning path

```text
current agent role
  -> predict next role from observed transition history
  -> confidence gate
  -> [optional host -> HBM static-prefix prefetch]
  -> tool gap
  -> next role becomes observable
  -> update transition model
  -> run next step with prefix hit/miss
```

This path is intentionally separate from session-KV retention. It studies cross-workflow reuse of static agent prefixes, transition-model adaptation, prefetch precision/coverage, cache pollution, and transfer waste. The default workflow generator changes its transition matrix partway through the trace so cumulative and forgetting-based learners can be compared under non-stationarity.