docs/modules/metrics.md

bankML/metrics.rs — bankML's own measurements of its answers

Summary

metrics.rs (0.3.7) keeps one record per completion, taken inside the engine (Native::complete_with, which Native::complete wraps since 0.3.8, so streamed, non-streamed and logprobs answers are all counted), in a ring of the last 256, and serves them at GET /bankml/metrics. The definitions are the conventional ones, so the numbers compare with llama.cpp's: time to first token (TTFT), prompt processing in tokens per second (llama-bench's pp), generation in tokens per second (tg), and — when the CPU package's RAPL counter is readable — the energy a completion took and joules per generated token. Nothing is estimated: a field that was not measured is null. The console's charts and the bankML persona's SELF block read from here.

Technical usage

pub const KEEP: usize = 256;
pub struct Record { pub at: f64, pub model: String, pub prompt_tokens: usize, pub cached_tokens: usize,
                    pub completion_tokens: usize, pub prompt_ms: f64, pub ttft_ms: Option<f64>, pub eval_ms: f64,
                    pub finish: String, pub grammar_ms: f64, pub resampled: usize, pub energy_j: Option<f64> }
impl Record { pub fn prompt_tps(&self) -> Option<f64>; pub fn eval_tps(&self) -> Option<f64>;
              pub fn joules_per_token(&self) -> Option<f64>; pub fn json(&self) -> String }
pub fn push(r: Record); pub fn records() -> Vec<Record>; pub fn json() -> String

• model is the GGUF's file stem; at the end of the completion in Unix seconds.
• prompt_tps counts only the prompt tokens actually computed, over prompt_ms. cached_tokens is the prefix found in the slot, or brought back from the host prompt cache (prompt_cache.md); it is the cache_n of the answer's timings.
• energy_j is the difference of two sys::energy_uj() readings around the completion (the counter's wrap handled); it includes everything the package drew in that time, an integrated GPU too.
• Record::json adds the derived prompt_tps, eval_tps and joules_per_token to the stored fields.
• GET /bankml/metrics → {"source", "kept", "keep", "prompt_tokens", "completion_tokens", "energy_j", "joules_per_token", "records": […]}. The token totals cover every kept record; energy_j and joules_per_token only the records whose energy was measured (null when none was).

curl -s 127.0.0.1:18093/bankml/metrics | python3 -m json.tool | head

How it is verified

Unit tests: rates_and_energy_are_measured_or_null (the rates, J/token, and null when not measured) and the_ring_keeps_the_last_ones. Live (0.3.7, Bonsai-1.7B): a non-streamed answer's receipt carries ttft_ms (it was null before), and the record matches the answer's timings.

Advantages and efficiency

• Measured where the work is done. The engine times its own prompt computation, first piece and generation, so the numbers do not include HTTP or the client.
• Bounded and cheap. A fixed ring of 256 records behind one mutex, one RAPL read before and after a completion.
• Honest by construction. Option fields render null; no field is ever derived from a guess.

Limitations

• Energy needs ./install.sh power (the counter is root-only by default) and covers the whole CPU package, not bankML alone; a discrete GPU's power is not included (no driver-neutral sensor).
• The ring is in memory: a restart empties it (the console keeps its own log of exchanges).

See also

sys.md · native.md · serve.md · prompt_cache.md · console.md · ../usage.md §13