| # `bankML/metrics.rs` — bankML's own measurements of its answers |
|
|
| ## Summary |
|
|
| `metrics.rs` (0.3.7) keeps one record per completion, taken inside the engine (`Native::complete_with`, which |
| `Native::complete` wraps since 0.3.8, so streamed, non-streamed and logprobs answers are all counted), in a ring of the |
| last 256, and serves them at `GET /bankml/metrics`. The definitions are the conventional ones, so the numbers compare with |
| llama.cpp's: time to first token (TTFT), prompt processing in tokens per second (llama-bench's `pp`), generation in tokens |
| per second (`tg`), and — when the CPU package's RAPL counter is readable — the energy a completion took and joules per |
| generated token. Nothing is estimated: a field that was not measured is `null`. The console's charts and the bankML |
| persona's SELF block read from here. |
|
|
| ## Technical usage |
|
|
| ```rust |
| pub const KEEP: usize = 256; |
| pub struct Record { pub at: f64, pub model: String, pub prompt_tokens: usize, pub cached_tokens: usize, |
| pub completion_tokens: usize, pub prompt_ms: f64, pub ttft_ms: Option<f64>, pub eval_ms: f64, |
| pub finish: String, pub grammar_ms: f64, pub resampled: usize, pub energy_j: Option<f64> } |
| impl Record { pub fn prompt_tps(&self) -> Option<f64>; pub fn eval_tps(&self) -> Option<f64>; |
| pub fn joules_per_token(&self) -> Option<f64>; pub fn json(&self) -> String } |
| pub fn push(r: Record); pub fn records() -> Vec<Record>; pub fn json() -> String |
| ``` |
|
|
| - `model` is the GGUF's file stem; `at` the end of the completion in Unix seconds. |
| - `prompt_tps` counts only the prompt tokens actually computed, over `prompt_ms`. `cached_tokens` is the prefix found |
| in the slot, or brought back from the host prompt cache ([prompt_cache.md](prompt_cache.md)); it is the `cache_n` |
| of the answer's `timings`. |
| - `energy_j` is the difference of two `sys::energy_uj()` readings around the completion (the counter's wrap handled); it |
| includes everything the package drew in that time, an integrated GPU too. |
| - `Record::json` adds the derived `prompt_tps`, `eval_tps` and `joules_per_token` to the stored fields. |
| - `GET /bankml/metrics` → `{"source", "kept", "keep", "prompt_tokens", "completion_tokens", "energy_j", |
| "joules_per_token", "records": […]}`. The token totals cover every kept record; `energy_j` and `joules_per_token` |
| only the records whose energy was measured (`null` when none was). |
|
|
| ```sh |
| curl -s 127.0.0.1:18093/bankml/metrics | python3 -m json.tool | head |
| ``` |
|
|
| ## How it is verified |
|
|
| Unit tests: `rates_and_energy_are_measured_or_null` (the rates, J/token, and `null` when not measured) and |
| `the_ring_keeps_the_last_ones`. Live (0.3.7, Bonsai-1.7B): a non-streamed answer's receipt carries `ttft_ms` (it was |
| `null` before), and the record matches the answer's `timings`. |
|
|
| ## Advantages and efficiency |
|
|
| - **Measured where the work is done.** The engine times its own prompt computation, first piece and generation, so the |
| numbers do not include HTTP or the client. |
| - **Bounded and cheap.** A fixed ring of 256 records behind one mutex, one RAPL read before and after a completion. |
| - **Honest by construction.** `Option` fields render `null`; no field is ever derived from a guess. |
|
|
| ## Limitations |
|
|
| - Energy needs `./install.sh power` (the counter is root-only by default) and covers the whole CPU package, not bankML |
| alone; a discrete GPU's power is not included (no driver-neutral sensor). |
| - The ring is in memory: a restart empties it (the console keeps its own log of exchanges). |
|
|
| ## See also |
|
|
| [sys.md](sys.md) · [native.md](native.md) · [serve.md](serve.md) · [prompt_cache.md](prompt_cache.md) · [console.md](console.md) · [../usage.md](../usage.md) §13 |
|
|
