| # Final analysis - `experiments/analysis/` |
|
|
| Generated by `scripts/analyze_final.py` (Python 3.12, stdlib only + numpy + matplotlib). |
|
|
| ## Methodology |
|
|
| ### Sources |
|
|
| - `experiments/runs/<run>/results.jsonl` - one OK record per line (schema in the |
| inventory). `config.json` normalizes the run: family (`final/curves/ksweep`), |
| target (`{qwen|gemma}-{q4|q5|q8}` from the model path), drafter from |
| `spec_type` + `p_min` (dspark) + draft quantization (dflash-f16/q4/q8). |
| - `server.log` from `curves-*`/`ksweep-*` runs - `draft acceptance` and |
| `acc per pos` lines (only place with per-position curves; the `final-*` runs |
| rotated logs and their original segment has no acceptance lines). |
|
|
| ### Exclusions (timing sentinels) |
|
|
| llama-server reports `tok_per_s = 1,000,000` / `predicted_ms = 0` in 271 records |
| from Gemma runs (timing quirk, not real speedup). **Rule applied in every |
| calculation**: exclude any record with `tok_per_s >= 1e5` or `predicted_ms <= 0` |
| (together with `alpha/tau/draft_n = None`). Qwen has no sentinels. The Gemma |
| `metrics.json` files are contaminated by this (mean 12,291 tok/s) and are **not** |
| used for tok/s; only `duration_s` and `errors` are used. Counts per run in |
| `summary.json` -> `meta.exclusions_per_run` (q4: 18/run, q5: 25/run, q8: 20/run |
| in final; 3-4/run in gemma curves/ksweep). |
|
|
| ### Log <-> record mapping (curves/ksweep) |
|
|
| The order of the `draft acceptance` lines in `server.log` == order of records |
| with non-None `alpha` in `results.jsonl` (verified in 41/41 runs, max |diff| = |
| 5e-5, rounding only). Gemma sentinel records (alpha=None) have no line and are |
| skipped. This assigns the domain to each line. No mismatches in any run. |
|
|
| ### Speedup vs solo |
|
|
| Per-prompt ratio `tps_draft / tps_solo` matched by `id` against the `solo` run |
| of the same family+quant; mean and median of the ratio are reported, plus the |
| ratio of means (`agg`). The 6 solo runs (qwen q4/q5/q8, gemma q4/q5/q8) exist |
| and are the baseline. |
|
|
| ### Break-even alpha_be (paper #32, Bielik et al.) |
| |
| Per `ksweep-*` run and domain (and aggregated): OLS of `TPS = a + beta*alpha` |
| over the run records (alpha and per-prompt tok/s, clean records with alpha). |
| `alpha_be = (TPS_base - a) / beta` where `TPS_base` = mean tok/s of the |
| corresponding `solo` run (same family+quant+domain, final run). CI95 by delta |
| method over the OLS covariance of (a, beta). alpha_be > 1 = no reachable |
| break-even; < 0 = always above baseline. Paper #32 reports k=2: 38.0-52.8% and |
| k=4: 77.7-90.1% (Table 5); the compact 40-77% range covers k=2..4. |
| |
| ## Files |
| |
| - `summary.json` - consolidated numbers (per config x domain and breakeven block). |
| - `tables/t2_speedup.md` ... `tables/t6_breakeven.md` - markdown tables for the paper. |
| - `curves/acc_by_pos_<run>.csv` - per-position acceptance per domain (41 runs). |
| - `checks.md` - result of the checks vs session 08. |
| - `manuscript/figures/F1..F4_*.png` - figures (300 dpi; local paper output, `manuscript/` is not versioned). |
|
|
| ## Execution |
|
|
| ``` |
| .venv/bin/python scripts/analyze_final.py \ |
| --runs-dir experiments/runs --out-dir experiments/analysis \ |
| --figs-dir manuscript/figures |
| ``` |
|
|
