README.md

---
title: InferScale-Sim
colorFrom: indigo
colorTo: blue
sdk: static
app_file: index.html
pinned: false
license: mit
short_description: Interactive LLM serving simulator and SLO planner
---

# InferScale-Sim

A browser-based research workbench for studying LLM serving systems without provisioning a GPU.

The simulator is written in Python. Hugging Face serves a static site; Pyodide runs the same `src/inferscale` package inside a Web Worker on the visitor's CPU. There is no backend service, model download, API key, or server-side accelerator.

> **Timing scope.** The bundled L4/A10G/A100 profiles are analytical references, not measured hardware benchmarks. Queueing, scheduling, KV-cache, transfer, SLO, and policy behavior is simulated live. Hardware claims require imported measurements and held-out validation.

## Research question

InferScale asks how serving-policy choices interact with workload shape and resource constraints:

- when continuous batching or chunked prefill changes tail latency;
- when prefill/decode disaggregation helps enough to justify extra accelerators and KV transfer;
- when reusable prefixes improve goodput versus consume scarce cache capacity;
- how stateful agent sessions change KV-retention and routing decisions;
- whether online prediction improves serving outcomes, not merely prediction accuracy;
- which conclusions remain stable across matched workload seeds and timing uncertainty.

The motivation is practical. Microsoft's [Vidur](https://arxiv.org/abs/2405.05465) reported an example LLaMA2-70B configuration search that took roughly one CPU-hour through simulation versus an estimated 42,000 GPU-hours (~$218K) using deployment-based exploration.

## What is implemented

### Stateless serving

- open-loop constant, Poisson, bursty, and exact trace-replay workloads;
- static and continuous batching;
- FCFS, shortest-job-first, least-slack/SLO-aware scheduling;
- chunked prefill;
- paged KV-cache accounting and memory admission;
- controlled shared-prefix reuse;
- colocated and prefill/decode-disaggregated topologies;
- independent P/D worker counts, accelerator profiles, and serialized KV-transfer cost;
- TTFT, TPOT, E2E, queueing, throughput, goodput, SLO attainment, and KV telemetry.

### Stateful sessions

- multi-turn session dependencies separated by sampled tool gaps;
- HBM retention, TTL expiry, host-memory offload/restore, and pressure eviction;
- least-load, strict-affinity, and bounded-affinity routing;
- cross-turn cache hits, history recomputation, HBM/host GB-seconds, and session SLOs;
- online global/per-tool EWMA tool-gap prediction under a controlled distribution shift.

### Execution learning

- online first-order role-transition learning with no future-trace lookahead;
- cumulative and decayed transition models;
- confidence-gated top-1 prefetch;
- multi-step transition rollout and top-k prefetch;
- utility-aware prefetch using transfer and forecast-weighted eviction costs;
- Brier score, log loss, ECE, reliability diagrams, future-role recall@K, prefetch utilization, and speculative-transfer waste;
- confidence, forgetting-rate, forecast-horizon, and cache-budget studies.

### Research consolidation

- common-random-number A/B experiments;
- paired bootstrap intervals;
- timing-model sensitivity analysis;
- repeated-seed policy ranking;
- Pareto stability across latency, HBM residency, and unused speculative transfer;
- a bounded full-trace serving oracle over a declared policy/action-plan family;
- per-policy regret to that bounded reference;
- vLLM/SGLang-style measurement import, calibration, and held-out validation hooks;
- Markdown/JSON/CSV/PNG research-artifact export.

## Reference result

The repository includes a deterministic 12-seed analytical-reference consolidation study in [`reports/reference_consolidation.md`](reports/reference_consolidation.md).

| Policy | Median p95 TTFT | TTFT wins | Pareto stability | Median oracle regret |
|---|---:|---:|---:|---:|
| Top-1 decayed | 6,474.9 ms | 41.7% | 75.0% | 59.3 ms |
| No prefetch | 6,490.7 ms | 41.7% | **91.7%** | **34.7 ms** |
| Utility-aware multi-step | 6,507.5 ms | 8.3% | 58.3% | 133.1 ms |
| Multi-step top-k | 6,640.3 ms | 8.3% | 83.3% | 215.0 ms |

`Top-1 decayed` is the robust TTFT winner under the configured ranking protocol, while `No prefetch` is Pareto-stable on more seeds and has lower median regret to the bounded oracle. The point is not to collapse the study to one score: prediction quality, latency, speculative transfer, and HBM pressure can prefer different policies.

The bounded oracle searches 19 candidate plans per seed and has median p95 TTFT **6,408.6 ms** in this reference study. It is an information upper bound over that declared family, not a proof of globally optimal cache scheduling.

## Interface map

The public Space is organized as a research tool rather than a product dashboard:

| Section | Purpose |
|---|---|
| **Serving** | Inspect one workload and request-level behavior. |
| **Schedulers** | Compare colocated schedulers on an identical trace. |
| **Capacity** | Find the highest repeated SLO-compliant offered load. |
| **P/D + Cache** | Compare colocated/P-D serving with and without prefix reuse. |
| **Design Space** | Sweep bounded configurations and inspect performance/efficiency Pareto fronts. |
| **A/B Studies** | Run paired bootstrap and timing-sensitivity experiments. |
| **Stateful Sessions** | Study KV retention, host offload, affinity, and tool-gap prediction. |
| **Execution Model** | Study transition learning, multi-step prefetch, calibration, and cache pressure. |
| **Evidence** | Run repeated-seed consolidation and import external measurements. |
| **Methods** | Read the simulator assumptions and research lineage. |

## Architecture

```text
Hugging Face Static Space
        |
        | serves HTML / CSS / JS / Python sources
        v
Browser
  |
  +-- UI + Chart.js
  |
  +-- Web Worker
        |
        +-- Pyodide
              |
              +-- src/inferscale
                    |
                    +-- discrete-event simulation
                    +-- research protocols
                    +-- calibration / reporting
```

The browser mirror under `py/inferscale/` is generated from `src/inferscale/`; `scripts/release_check.py` fails if the copies diverge.

## Run locally

```bash
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -U pip
pip install -e '.[dev]'

pytest -q
python scripts/release_check.py
```

For the static UI, serve the repository root with any local HTTP server:

```bash
python -m http.server 8000
```

The deployed Space needs no secrets.

## Command-line examples

Run a single simulation:

```bash
python scripts/run_simulation.py examples/balanced.json
```

Import a serving-benchmark artifact:

```bash
python scripts/import_measurements.py path/to/benchmark.json --format auto
```

Calibrate analytical timing scales and evaluate held-out cases:

```bash
python scripts/calibrate_profiles.py measurements.json --holdout 0.25
```

Generate a research report:

```bash
python scripts/generate_report.py robust-policy-study.json \
  --calibration calibration_result.json
```

## Repository layout

```text
src/inferscale/         canonical Python simulator and research code
py/inferscale/          browser mirror loaded by Pyodide
index.html              static workbench structure
styles.css              project-specific design system
app.js                  UI, charts, exports, worker orchestration
worker.mjs              Pyodide bootstrap and Python action bridge
scripts/                release, import, calibration, report utilities
tests/                  deterministic unit/integration coverage
docs/                   architecture, methodology, validation, research notes
examples/                workload and validation schemas
reports/                 reference analytical study
```

## Validation

The release suite checks both simulation behavior and deployment invariants, including:

- stateless and P/D smoke tests;
- trace replay;
- prefix reuse and design-space Pareto logic;
- stateful-session, host-tier, HBM-pressure, and affinity experiments;
- predictive tiering and execution-learning studies;
- paired bootstrap, timing sensitivity, repeated-seed consolidation, oracle regret;
- measurement import/calibration/report generation;
- source/browser Python parity;
- HF metadata and public provenance guardrails;
- DOM-reference integrity and chart-export controls.

Run:

```bash
pytest -q
python scripts/release_check.py
python -m compileall -q src scripts
node --check app.js
node --check worker.mjs
```

## Design and implementation choices

The interface deliberately avoids a product/SaaS visual language. It uses one restrained accent, flat surfaces, square geometry, system typography, and editorial hierarchy instead of gradients, floating cards, badge-heavy status UI, or decorative marketing sections. The rationale and reusable tokens are documented in [`docs/design.md`](docs/design.md).

The simulator core is dependency-light Python rather than a simulation framework. Events, requests, schedulers, caches, predictors, and studies remain inspectable from source and testable outside the browser.

## Limitations

1. Bundled device timings are analytical reference profiles.
2. The agent-session simulator intentionally isolates state/routing effects from the full dynamic-batching model used in stateless serving.
3. Transfer models are simplified serialized bandwidth + base-latency abstractions, not packet/NCCL/NIXL simulators.
4. Oracle policies are bounded information references over declared action families.
5. Calibration can correct global timing bias; it does not establish fidelity on unseen models, hardware, schedulers, or workload regimes.

These limitations are surfaced in the UI and exported reports rather than hidden.

## Research lineage

InferScale is informed by work on serving simulation, disaggregation, stateful agent workloads, and predictive cache management. See [`docs/research.md`](docs/research.md) for the annotated list and the precise distinction between implemented abstractions and cited systems.

Key starting points include:

- [Vidur](https://arxiv.org/abs/2405.05465) — predictive profiling and workload-aware serving simulation.
- [SGLang / RadixAttention](https://arxiv.org/abs/2312.07104) — structured prefix reuse and scheduling motivation.
- Recent 2026 work discussed in `docs/research.md` on P/D disaggregation, GPU-free emulation, agent-session serving, KV tiering, routing locality, and predictive prefetch.

## License

MIT.