evaluation/README.md

# Evaluation protocol

`metrics.json` contains retained measurements for the released GRU controller.
This release changes packaging and documentation, without a new quality
evaluation. Scores are offline diagnostics, not a live voice-agent benchmark.
Serialization checks are a separate engineering validation.

## Data and coverage

| Record | Source and split | Coverage |
| --- | --- | --- |
| `open_yap_test_frames` | Open Yap 1K v1.0, test | 23,936 windows from 100 conversations |
| `open_yap_test_events` | Open Yap test, first 240 seconds per conversation, both channel assignments | 1,268 yield and 1,077 backchannel onsets |
| `endpoints.open_yap_dev` | Open Yap development | 600 turn + 600 pause probes |
| `endpoints.open_yap_test` | Open Yap test | 1,000 turn + 1,000 pause probes |
| `endpoints.fdb_turn_taking` | FDB v1.0, `candor_turn_taking` | 119 turn probes |
| `endpoints.fdb_pause_handling` | FDB v1.0, `candor_pause_handling` | 277 pause intervals; several can share a source clip |
| `endpoints.internal` | Non-public internal evaluation | 242 turn + 539 pause probes |
| `open_yap_dev_filler_proxy` | Open Yap development | 12,647 candidate turn ends |

Open Yap has 1,395 training, 107 development and 100 test conversations. Both
channel assignments stay in the same split. Conversation-disjoint splits are
not a claim of speaker-disjointness. See the
[Open Yap page](https://theagenticdatacompany.com/open-yap-1k) for access and citation.

The FDB source is [Full-Duplex-Bench-Data](https://huggingface.co/datasets/Ssshangfu/Full-Duplex-Bench-Data),
revision `a3579cb9ebbe164204b196e33d4d0e91860eb90e`. Only the two v1.0 CANDOR
subsets are used. These results do not cover synthetic interruption,
backchannel or FDB-v3 tool-use tasks. Internal evaluation data are non-public;
the package provides aggregates without source identifiers or data access.

## Operating point and calibration

Offline thresholds are endpoint **0.91**, yield **0.93**, backchannel **0.80**,
with two-frame debounce. Endpoint replay includes six warmup frames, voice
activity above -50 dB, at least 350 ms accumulated voiced audio, and optional
1.2-second silence fallback. Backchannel replay has a 15-frame cooldown.
A frame is 80 ms. `evaluation_policy` records the settings in JSON.

Thresholds were selected on development data and frozen before candidate test
scoring. Endpoint calibration minimized replay latency subject to the reference
system's development pause-cutin ceiling; yield matched its false-event rate,
and backchannel matched its acknowledgment rate. Settings remained frozen
across evaluation domains.

The release defaults use **0.50 / 0.40** for yield/backchannel. No new event
measurements exist for those lower defaults. Offline replay and the streaming
wrapper are separate implementations with their own state/reset timing; these
records do not establish complete live equivalence.

## Metric definitions

- **State macro-F1:** mean F1 over LISTEN, HOLD, SPEAK and BACKCHANNEL.
  Confusion-matrix rows are ground truth, columns predictions, in that order.
  State/yield use teacher-forced previous state; endpoint/backchannel force LISTEN.
- **ROC-AUC / average precision:** frame ranking metrics over task-specific
  valid masks. Positive and masked frame counts are retained. These are not
  event recall or probability-calibration guarantees.
- **Endpoint latency:** first endpoint at or after annotated turn end minus
  that turn-end time. Mean/median/p90 are conditional on firing. Replay resets
  after early endpoints and continues; early cut-ins are counted separately.
- **Early cut-in share:** fraction of turn probes with any endpoint before
  turn end. A probe can have both an early cut-in and a later fast endpoint.
- **No endpoint share:** no final endpoint before clip end. **Miss within 1.2 s**
  also includes late endpoints, on clips long enough to observe the deadline.
  Eligible/censored counts are explicit; all reported turns are eligible.
- **Head/fallback shares:** fractions of all turn probes ending through the
  head or fallback. Together with no-endpoint share they sum to one.
  `head_only` disables fallback; `fallback_on` enables it.
- **Pause false endpoints:** one binary false-cut outcome per pause probe,
  divided by reference duration, rather than every repeated firing in a pause.
- **Yield detection / backchannel recall:** fraction of labeled onsets matched
  by the frozen event matcher. Yield matches a firing in the first eight
  80 ms frames from onset (the interval is inclusive at onset and exclusive
  at +640 ms). Backchannel matches within [-640, +320) ms of onset, excluding
  the upper bound and allowing anticipatory acknowledgments. Latency uses matches only. Prefix event
  scoring differs from full-test frame scoring.
- **Acknowledgment rate:** all backchannel firings per LISTEN hour, not recall.
  Negative latency means firing before the labeled onset.
- **Filler proxy:** a dense development-set scan of floor-taking in an internal
  user pause after a speech run of at most 600 ms, reported alongside turn-end
  recall. The 390 filler floor-takings
  over 57.3844 user-floor hours give 6.80/h. The separate dense false-endpoint
  rate is 1,004.51/h. Neither rate is the mined pause-probe metric.

For Open Yap/internal probes, `floor_h` is user-floor time in source
conversations, including both assignments. FDB uses unique source-clip time:
approximately 0.46361 h for turn-taking and 0.70996 h for pauses. Multiple pause
probes from one clip do not duplicate its denominator. Hourly rates are not
directly comparable across these sources. Endpoint outcomes are control
proxies, not official FDB speech-output Takeover Rate (TOR).

## Compute and interpretation

Controller compute uses RTX 4090, batch 1, 64 prefill frames, then three repeats
of 200 one-frame updates. All five heads are included, FP32 with TF32 off.
Median/p90 wall and CUDA event time are separate. Weight bytes, peak allocated
memory and recurrent-state bytes measure different things. Nemotron, ASR,
audio waiting, LLM, TTS and playback are excluded.

Tables and the endpoint figure show point estimates. The source research used
conversation-grouped paired bootstraps for model comparisons; those are not
reinterpreted as absolute confidence intervals here. One training seed and
reused diagnostic sets limit generalization. Controller timings do not support
comparisons against another paper's full-agent response latency.

No new ASR WER, production concurrency or live conversation evaluation is added.
The model card provides deployment notes, links and citations.