Text Classification
Transformers
ONNX
Safetensors
bert
barge-in
backchannel
multilingual
experimental
text-embeddings-inference
Instructions to use HashNuke/barge-in-detector with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use HashNuke/barge-in-detector with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="HashNuke/barge-in-detector")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSequenceClassification tokenizer = AutoTokenizer.from_pretrained("HashNuke/barge-in-detector") model = AutoModelForSequenceClassification.from_pretrained("HashNuke/barge-in-detector", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Akash Manohar commited on
Commit ·
9aa77e5
1
Parent(s): dcbadfd
Add standalone training, evaluation and model instructions
Browse filesPin the separate dataset revision in release metadata; include scoped agent instructions, model milestones, labnotes helper, CPU benchmarks, tests and recorded diagnostic results.
- .gitattributes +1 -1
- .gitignore +11 -0
- .python-version +1 -0
- AGENTS.md +95 -0
- README.md +8 -4
- barge_in_detector_model/__init__.py +1 -0
- barge_in_detector_model/data.py +39 -0
- barge_in_detector_model/evaluate.py +111 -0
- barge_in_detector_model/infer.py +27 -0
- barge_in_detector_model/metrics.py +46 -0
- barge_in_detector_model/train.py +140 -0
- bin/create-labnotes +59 -0
- experiments/benchmark_cpu.py +140 -0
- experiments/benchmark_cpu_cold.py +80 -0
- experiments/evaluate_english_challenge.py +132 -0
- labnotes/20261115-1332-repository-split.md +24 -0
- labnotes/issues/.gitkeep +0 -0
- labnotes/milestones/experimental-checkpoint-release.md +30 -0
- labnotes/milestones/index.md +7 -0
- labnotes/milestones/short-interruption-recall.md +28 -0
- pyproject.toml +25 -0
- release.json +34 -0
- reports/challenge-predictions.csv +99 -0
- reports/challenge-report.json +438 -0
- reports/cpu-benchmark.json +94 -0
- reports/cpu-cold-benchmark.json +33 -0
- reports/cpu-thread-sweep.json +194 -0
- reports/eval-test.json +81 -0
- reports/eval-validation.json +81 -0
- tests/test_data.py +43 -0
- tests/test_metrics.py +29 -0
- uv.lock +0 -0
.gitattributes
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
| 2 |
/tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
| 3 |
-
|
|
|
|
| 1 |
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
| 2 |
/tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
| 3 |
+
*.csv whitespace=cr-at-eol
|
.gitignore
ADDED
|
@@ -0,0 +1,11 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
.venv/
|
| 2 |
+
.venv-cpu/
|
| 3 |
+
__pycache__/
|
| 4 |
+
*.pyc
|
| 5 |
+
.pytest_cache/
|
| 6 |
+
.ruff_cache/
|
| 7 |
+
/data/
|
| 8 |
+
/models/
|
| 9 |
+
/checkpoints/
|
| 10 |
+
.env
|
| 11 |
+
.env.*
|
.python-version
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
3.14
|
AGENTS.md
ADDED
|
@@ -0,0 +1,95 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Barge-in detector model
|
| 2 |
+
|
| 3 |
+
## Ownership
|
| 4 |
+
|
| 5 |
+
This Hugging Face model repository (`HashNuke/barge-in-detector`) owns weights,
|
| 6 |
+
tokenizer, training, inference, evaluation, model experiments and CPU benchmarks.
|
| 7 |
+
The dataset repository (`HashNuke/barge-in-detector-data`) owns generation,
|
| 8 |
+
cleaning, source utterances, splits, review UI and historical project labnotes.
|
| 9 |
+
Do not copy training data into this repo. Fetch a dataset revision or pass local
|
| 10 |
+
splits. Record the resolved dataset commit, base model, seed, hyperparameters,
|
| 11 |
+
source commit and metrics for every release. Never train on challenge examples.
|
| 12 |
+
|
| 13 |
+
## Contract and quality gates
|
| 14 |
+
|
| 15 |
+
- Caller utterance spoken over an ongoing voice-agent turn; cleaned lowercase
|
| 16 |
+
text without punctuation. No audio, agent line or language hint is required.
|
| 17 |
+
- Exactly two labels: `backchannel` (continue speaking) and `barge_in` (yield).
|
| 18 |
+
Repetitions, acknowledgments, surprise, empathy and laughter are backchannels
|
| 19 |
+
unless the text develops into a correction, question, request or floor claim.
|
| 20 |
+
Context and an ambiguous/needs-context class are unavailable.
|
| 21 |
+
- At most 64 tokens; keep the first 64. Evaluate partial transcripts and
|
| 22 |
+
transitions from backchannel to interruption. No universal ignored-word list.
|
| 23 |
+
- Output `p_barge_in` in [0,1]. Current experimental probabilities are not yet
|
| 24 |
+
calibrated; thresholds belong to the consumer. Use validation for tuning.
|
| 25 |
+
- CPU inference only. Warmed batch-one model-path p99 <=20 ms and whole path
|
| 26 |
+
including Vxpipe boundary <=50 ms on the designated reference CPU. Report
|
| 27 |
+
p50/p95/p99, hardware, threads, memory, warm-up and token lengths. Local VM
|
| 28 |
+
measurements do not satisfy reference-CPU acceptance gates.
|
| 29 |
+
- Fourteen languages: English (US/India/Singapore), Hindi, Kannada, Tamil,
|
| 30 |
+
Telugu, Malayalam, Mandarin, Japanese, Indonesian, Filipino, French, German,
|
| 31 |
+
Spanish and Italian. Roll out English/Hindi first, then Kannada/Tamil.
|
| 32 |
+
Native scripts, optionally mixed with English; no romanization/pinyin.
|
| 33 |
+
Language support requires evaluation evidence, not tokenizer coverage.
|
| 34 |
+
- Tier A owner verification: English, Hindi, Kannada, Tamil. Other languages
|
| 35 |
+
remain synthetic/unverified unless native-speaker checks are recorded.
|
| 36 |
+
- Report false interruptions and missed interruptions, per-language results,
|
| 37 |
+
short-input slices, prefix behaviour and calibration. Keep synthetic and
|
| 38 |
+
owner-reviewed evidence separate; a perfect synthetic score is insufficient.
|
| 39 |
+
- Prefer permissive base licences. Preserve the MIT attribution for the
|
| 40 |
+
intfloat/multilingual-e5-small base model.
|
| 41 |
+
|
| 42 |
+
## Tools and remote training
|
| 43 |
+
|
| 44 |
+
- Use uv; add dependencies with `uv add` and keep uv.lock with dependency edits.
|
| 45 |
+
Always source ~/.zshrc or use zsh -lc for preconfigured tools.
|
| 46 |
+
- Always rent exactly one RTX 5090 (32 GB), in a datacenter, below $0.65/hour,
|
| 47 |
+
with good internet and the Vast.ai PyTorch image. Do not substitute another
|
| 48 |
+
GPU or rent multiple instances. Inspect existing instances before renting.
|
| 49 |
+
- All Vast.ai setup and training commands run inside tmux session `gpubox`.
|
| 50 |
+
Save logs/checkpoints to disk. Copy and verify artifacts before destroying
|
| 51 |
+
instances; follow the owner's explicit cleanup instructions.
|
| 52 |
+
- CPU evaluation and benchmarks run locally. Do not rent or restart a GPU for
|
| 53 |
+
work that the local CPU can perform. No Playwright; use agent-browser if
|
| 54 |
+
browser work is needed. Do not disable browser security to work around failures.
|
| 55 |
+
- Never print or commit tokens, authorization headers, .env, PII, caches or
|
| 56 |
+
virtualenvs. Hugging Face credentials come from existing login/environment.
|
| 57 |
+
- Root model.safetensors and tokenizer.json are versioned large artifacts;
|
| 58 |
+
use Hugging Face large-file storage. Keep experiment checkpoints under
|
| 59 |
+
ignored models/ or checkpoints/. Publish selected reports with the release.
|
| 60 |
+
|
| 61 |
+
## Labnotes and milestones
|
| 62 |
+
|
| 63 |
+
- Start at labnotes/milestones/index.md; its checklist owns implementation order.
|
| 64 |
+
Read linked prerequisites and sources before starting a checkpoint.
|
| 65 |
+
- For implementation or research, use `bin/create-labnotes two-to-four-words`.
|
| 66 |
+
Write factual progress, failures, workarounds, decisions and evidence into
|
| 67 |
+
the generated top-level labnotes file. Link it from the relevant milestone.
|
| 68 |
+
Do not create labnotes for simple status/read-only checks or running an
|
| 69 |
+
existing command. Labnotes accompany the work in its commit.
|
| 70 |
+
- Keep milestones under labnotes/milestones/, deferred issues under
|
| 71 |
+
labnotes/issues/. Use descriptive unnumbered filenames and Markdown task
|
| 72 |
+
lists. Each milestone needs a runnable outcome, prerequisites, contracts,
|
| 73 |
+
checkpoint tasks and acceptance gates. A spec is not an implementation.
|
| 74 |
+
Leave index entries unchecked until all gates pass; record blockers honestly.
|
| 75 |
+
- Historical generation/training labnotes remain in the dataset repo; link
|
| 76 |
+
them rather than copying that history here. New model work gets model labnotes.
|
| 77 |
+
- Record substantial architecture decisions and rejected alternatives under docs/.
|
| 78 |
+
|
| 79 |
+
## Checkpoints and Git
|
| 80 |
+
|
| 81 |
+
Use small coherent commits of code, focused tests, docs, labnotes and lockfile.
|
| 82 |
+
For behaviour changes, write a focused failing test before implementation.
|
| 83 |
+
Experiments/prompts/benchmarks use recorded evidence; deterministic parsers,
|
| 84 |
+
metrics, tokenization and serving get focused tests. Default tests exclude
|
| 85 |
+
network integration tests. Before finishing code changes run:
|
| 86 |
+
|
| 87 |
+
```shell
|
| 88 |
+
uv run ruff format --check
|
| 89 |
+
uv run ruff check
|
| 90 |
+
uv run pytest
|
| 91 |
+
```
|
| 92 |
+
|
| 93 |
+
Inspect status and staged diffs, preserve unrelated work, stage exact paths.
|
| 94 |
+
Do not commit or publish unless requested. Do not amend, rebase, reset,
|
| 95 |
+
force-push, delete branches or undo commits without explicit instruction.
|
README.md
CHANGED
|
@@ -51,8 +51,10 @@ has not been established.
|
|
| 51 |
## Infer on CPU
|
| 52 |
|
| 53 |
```shell
|
| 54 |
-
uv
|
| 55 |
-
uv
|
|
|
|
|
|
|
| 56 |
```
|
| 57 |
|
| 58 |
Input must already follow the dataset cleaner: lowercase, punctuation removed,
|
|
@@ -109,7 +111,9 @@ uv run python -m experiments.benchmark_cpu_cold --threads 4
|
|
| 109 |
uv run python -m experiments.evaluate_english_challenge --dataset-dir /path/to/dataset
|
| 110 |
```
|
| 111 |
|
| 112 |
-
|
| 113 |
-
|
|
|
|
|
|
|
| 114 |
locally; rent only a single datacenter RTX 5090 below $0.65/hour for training,
|
| 115 |
using the Vast.ai PyTorch image and tmux session gpubox.
|
|
|
|
| 51 |
## Infer on CPU
|
| 52 |
|
| 53 |
```shell
|
| 54 |
+
uv venv --python 3.14
|
| 55 |
+
uv pip install torch==2.15.0 --index-url https://download.pytorch.org/whl/cpu
|
| 56 |
+
uv pip install transformers==5.23.0 huggingface-hub==2.4.0
|
| 57 |
+
uv run --no-sync python -m barge_in_detector_model.infer "please stop for a moment"
|
| 58 |
```
|
| 59 |
|
| 60 |
Input must already follow the dataset cleaner: lowercase, punctuation removed,
|
|
|
|
| 111 |
uv run python -m experiments.evaluate_english_challenge --dataset-dir /path/to/dataset
|
| 112 |
```
|
| 113 |
|
| 114 |
+
The CPU setup above intentionally uses the matching CPU wheel; use
|
| 115 |
+
`uv run --no-sync` with that environment to retain it. For CPU development,
|
| 116 |
+
install `pytest==9.2.0 ruff==0.17.5` with `uv pip install`. Standard `uv sync`
|
| 117 |
+
uses the lockfile and includes GPU training dependencies. Do CPU tests
|
| 118 |
locally; rent only a single datacenter RTX 5090 below $0.65/hour for training,
|
| 119 |
using the Vast.ai PyTorch image and tmux session gpubox.
|
barge_in_detector_model/__init__.py
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
"""Training, inference and evaluation for separately versioned barge-in data."""
|
barge_in_detector_model/data.py
ADDED
|
@@ -0,0 +1,39 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Read versioned dataset splits without owning their generation or assignment."""
|
| 2 |
+
|
| 3 |
+
import csv
|
| 4 |
+
from pathlib import Path
|
| 5 |
+
|
| 6 |
+
DEFAULT_DATASET = "HashNuke/barge-in-detector-data"
|
| 7 |
+
|
| 8 |
+
|
| 9 |
+
def read_csv(path):
|
| 10 |
+
with Path(path).open(encoding="utf-8", newline="") as f:
|
| 11 |
+
return list(csv.DictReader(f))
|
| 12 |
+
|
| 13 |
+
|
| 14 |
+
def resolve_splits(splits, *, repo_id=DEFAULT_DATASET, revision=None):
|
| 15 |
+
if splits is not None:
|
| 16 |
+
directory = Path(splits)
|
| 17 |
+
resolved_revision = None
|
| 18 |
+
else:
|
| 19 |
+
from huggingface_hub import HfApi, snapshot_download
|
| 20 |
+
|
| 21 |
+
resolved_revision = HfApi().dataset_info(repo_id, revision=revision).sha
|
| 22 |
+
root = Path(
|
| 23 |
+
snapshot_download(
|
| 24 |
+
repo_id=repo_id,
|
| 25 |
+
repo_type="dataset",
|
| 26 |
+
revision=resolved_revision,
|
| 27 |
+
allow_patterns=[
|
| 28 |
+
"data/splits/*.csv",
|
| 29 |
+
"data/evaluation/*.csv",
|
| 30 |
+
"data/evaluation/reviews/*.json",
|
| 31 |
+
"generation/factors/languages.yaml",
|
| 32 |
+
],
|
| 33 |
+
)
|
| 34 |
+
)
|
| 35 |
+
directory = root / "data/splits"
|
| 36 |
+
for split in ("train", "validation", "test"):
|
| 37 |
+
if not (directory / f"{split}.csv").is_file():
|
| 38 |
+
raise ValueError(f"Dataset split missing: {directory / split}.csv")
|
| 39 |
+
return directory, resolved_revision
|
barge_in_detector_model/evaluate.py
ADDED
|
@@ -0,0 +1,111 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Evaluate a trained model on one split (README.md).
|
| 2 |
+
|
| 3 |
+
uv run python -m barge_in_detector_model.evaluate --model models/run1 --split test
|
| 4 |
+
|
| 5 |
+
Reports accuracy, macro F1, false and missed interruption rates, and results
|
| 6 |
+
per language; writes them to <model>/eval-<split>.json and the mistakes to
|
| 7 |
+
<model>/errors-<split>.csv.
|
| 8 |
+
"""
|
| 9 |
+
|
| 10 |
+
import argparse
|
| 11 |
+
import csv
|
| 12 |
+
import json
|
| 13 |
+
from collections.abc import Sequence
|
| 14 |
+
from pathlib import Path
|
| 15 |
+
from typing import Any
|
| 16 |
+
|
| 17 |
+
import torch
|
| 18 |
+
from transformers import AutoModelForSequenceClassification, AutoTokenizer
|
| 19 |
+
|
| 20 |
+
from barge_in_detector_model.data import read_csv, resolve_splits
|
| 21 |
+
from barge_in_detector_model.metrics import LABELS, compute
|
| 22 |
+
|
| 23 |
+
BARGE_IN = LABELS.index("barge_in")
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
def device() -> torch.device:
|
| 27 |
+
return torch.device("cuda" if torch.cuda.is_available() else "cpu")
|
| 28 |
+
|
| 29 |
+
|
| 30 |
+
@torch.no_grad()
|
| 31 |
+
def predict(
|
| 32 |
+
model: Any,
|
| 33 |
+
tokenizer: Any,
|
| 34 |
+
rows: Sequence[dict[str, str]],
|
| 35 |
+
*,
|
| 36 |
+
batch_size: int = 128,
|
| 37 |
+
max_length: int = 64,
|
| 38 |
+
) -> list[float]:
|
| 39 |
+
"""P(barge_in) for each row's cleaned text."""
|
| 40 |
+
model.eval()
|
| 41 |
+
dev = next(model.parameters()).device
|
| 42 |
+
out: list[float] = []
|
| 43 |
+
for i in range(0, len(rows), batch_size):
|
| 44 |
+
texts = [r["text_clean"] for r in rows[i : i + batch_size]]
|
| 45 |
+
enc = tokenizer(
|
| 46 |
+
texts, truncation=True, max_length=max_length, padding=True, return_tensors="pt"
|
| 47 |
+
).to(dev)
|
| 48 |
+
logits = model(**enc).logits.float()
|
| 49 |
+
out += torch.softmax(logits, dim=-1)[:, BARGE_IN].tolist()
|
| 50 |
+
return out
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
def evaluate(
|
| 54 |
+
model: Any,
|
| 55 |
+
tokenizer: Any,
|
| 56 |
+
rows: Sequence[dict[str, str]],
|
| 57 |
+
*,
|
| 58 |
+
threshold: float = 0.5,
|
| 59 |
+
max_length: int = 64,
|
| 60 |
+
) -> tuple[dict[str, Any], list[float]]:
|
| 61 |
+
probs = predict(model, tokenizer, rows, max_length=max_length)
|
| 62 |
+
preds = ["barge_in" if p >= threshold else "backchannel" for p in probs]
|
| 63 |
+
metrics = compute([r["label"] for r in rows], preds, [r["language"] for r in rows])
|
| 64 |
+
return metrics, probs
|
| 65 |
+
|
| 66 |
+
|
| 67 |
+
def main(argv: Sequence[str] | None = None) -> int:
|
| 68 |
+
parser = argparse.ArgumentParser(description=__doc__)
|
| 69 |
+
parser.add_argument("--model", type=Path, required=True, help="a directory written by train")
|
| 70 |
+
parser.add_argument("--split", default="test", choices=["train", "validation", "test"])
|
| 71 |
+
parser.add_argument("--splits", type=Path, default=None)
|
| 72 |
+
parser.add_argument("--threshold", type=float, default=0.5, help="P(barge_in) cut-off")
|
| 73 |
+
parser.add_argument("--max-length", type=int, default=64, help="token budget")
|
| 74 |
+
parser.add_argument("--dataset-repo", default="HashNuke/barge-in-detector-data")
|
| 75 |
+
parser.add_argument(
|
| 76 |
+
"--dataset-revision", help="Exact dataset commit; resolves current main when omitted"
|
| 77 |
+
)
|
| 78 |
+
args = parser.parse_args(argv)
|
| 79 |
+
args.splits, args.dataset_revision = resolve_splits(
|
| 80 |
+
args.splits, repo_id=args.dataset_repo, revision=args.dataset_revision
|
| 81 |
+
)
|
| 82 |
+
|
| 83 |
+
tokenizer = AutoTokenizer.from_pretrained(args.model)
|
| 84 |
+
model = AutoModelForSequenceClassification.from_pretrained(args.model).to(device())
|
| 85 |
+
rows = read_csv(args.splits / f"{args.split}.csv")
|
| 86 |
+
metrics, probs = evaluate(
|
| 87 |
+
model, tokenizer, rows, threshold=args.threshold, max_length=args.max_length
|
| 88 |
+
)
|
| 89 |
+
metrics.update(split=args.split, threshold=args.threshold)
|
| 90 |
+
(args.model / f"eval-{args.split}.json").write_text(json.dumps(metrics, indent=2))
|
| 91 |
+
with (args.model / f"errors-{args.split}.csv").open("w", encoding="utf-8", newline="") as f:
|
| 92 |
+
writer = csv.writer(f)
|
| 93 |
+
writer.writerow(["label", "p_barge_in", "language", "caller_intent", "text"])
|
| 94 |
+
for r, p in zip(rows, probs, strict=True):
|
| 95 |
+
if (p >= args.threshold) != (r["label"] == "barge_in"):
|
| 96 |
+
writer.writerow(
|
| 97 |
+
[r["label"], f"{p:.3f}", r["language"], r["caller_intent"], r["text"]]
|
| 98 |
+
)
|
| 99 |
+
print(
|
| 100 |
+
f"{args.split}: n={metrics['count']} accuracy={metrics['accuracy']:.3f} "
|
| 101 |
+
f"macro_f1={metrics['macro_f1']:.3f} "
|
| 102 |
+
f"false_interruptions={metrics['false_interruption_rate']:.3f} "
|
| 103 |
+
f"missed_interruptions={metrics['missed_interruption_rate']:.3f}"
|
| 104 |
+
)
|
| 105 |
+
for lang, m in metrics["per_language"].items():
|
| 106 |
+
print(f" {lang:>4} n={m['count']:<4} accuracy={m['accuracy']:.3f}")
|
| 107 |
+
return 0
|
| 108 |
+
|
| 109 |
+
|
| 110 |
+
if __name__ == "__main__":
|
| 111 |
+
raise SystemExit(main())
|
barge_in_detector_model/infer.py
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Predict a binary label for an already-cleaned overlapping caller utterance."""
|
| 2 |
+
|
| 3 |
+
import argparse
|
| 4 |
+
import json
|
| 5 |
+
from pathlib import Path
|
| 6 |
+
|
| 7 |
+
import torch
|
| 8 |
+
from transformers import AutoModelForSequenceClassification, AutoTokenizer
|
| 9 |
+
|
| 10 |
+
|
| 11 |
+
def main():
|
| 12 |
+
parser = argparse.ArgumentParser(description=__doc__)
|
| 13 |
+
parser.add_argument("text")
|
| 14 |
+
parser.add_argument("--model", type=Path, default=Path("."))
|
| 15 |
+
parser.add_argument("--threads", type=int, default=4)
|
| 16 |
+
args = parser.parse_args()
|
| 17 |
+
torch.set_num_threads(args.threads)
|
| 18 |
+
tokenizer = AutoTokenizer.from_pretrained(args.model)
|
| 19 |
+
model = AutoModelForSequenceClassification.from_pretrained(args.model).cpu().eval()
|
| 20 |
+
with torch.inference_mode():
|
| 21 |
+
enc = tokenizer(args.text, truncation=True, max_length=64, return_tensors="pt")
|
| 22 |
+
p = torch.softmax(model(**enc).logits, dim=-1)[0, model.config.label2id["barge_in"]].item()
|
| 23 |
+
print(json.dumps({"p_barge_in": p, "label": "barge_in" if p >= 0.5 else "backchannel"}))
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
if __name__ == "__main__":
|
| 27 |
+
main()
|
barge_in_detector_model/metrics.py
ADDED
|
@@ -0,0 +1,46 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Metrics for the two labels. Pure Python, so they are tested without torch."""
|
| 2 |
+
|
| 3 |
+
from collections import defaultdict
|
| 4 |
+
from collections.abc import Sequence
|
| 5 |
+
from typing import Any
|
| 6 |
+
|
| 7 |
+
LABELS = ("backchannel", "barge_in")
|
| 8 |
+
|
| 9 |
+
|
| 10 |
+
def _prf(labels: Sequence[str], preds: Sequence[str], cls: str) -> dict[str, float]:
|
| 11 |
+
tp = sum(1 for y, p in zip(labels, preds, strict=True) if y == cls and p == cls)
|
| 12 |
+
predicted = sum(1 for p in preds if p == cls)
|
| 13 |
+
actual = sum(1 for y in labels if y == cls)
|
| 14 |
+
precision = tp / predicted if predicted else 0.0
|
| 15 |
+
recall = tp / actual if actual else 0.0
|
| 16 |
+
f1 = 2 * precision * recall / (precision + recall) if precision + recall else 0.0
|
| 17 |
+
return {"precision": precision, "recall": recall, "f1": f1, "count": actual}
|
| 18 |
+
|
| 19 |
+
|
| 20 |
+
def compute(
|
| 21 |
+
labels: Sequence[str], preds: Sequence[str], languages: Sequence[str]
|
| 22 |
+
) -> dict[str, Any]:
|
| 23 |
+
per_class = {cls: _prf(labels, preds, cls) for cls in LABELS}
|
| 24 |
+
backchannel = [p for y, p in zip(labels, preds, strict=True) if y == "backchannel"]
|
| 25 |
+
barge_in = [p for y, p in zip(labels, preds, strict=True) if y == "barge_in"]
|
| 26 |
+
by_language: dict[str, list[bool]] = defaultdict(list)
|
| 27 |
+
for y, p, lang in zip(labels, preds, languages, strict=True):
|
| 28 |
+
by_language[lang].append(y == p)
|
| 29 |
+
return {
|
| 30 |
+
"count": len(labels),
|
| 31 |
+
"accuracy": sum(y == p for y, p in zip(labels, preds, strict=True)) / len(labels),
|
| 32 |
+
"macro_f1": sum(c["f1"] for c in per_class.values()) / len(LABELS),
|
| 33 |
+
# A backchannel predicted as barge-in stops the agent for nothing.
|
| 34 |
+
"false_interruption_rate": (
|
| 35 |
+
sum(p == "barge_in" for p in backchannel) / len(backchannel) if backchannel else 0.0
|
| 36 |
+
),
|
| 37 |
+
# A barge-in predicted as backchannel lets the agent talk over the caller.
|
| 38 |
+
"missed_interruption_rate": (
|
| 39 |
+
sum(p == "backchannel" for p in barge_in) / len(barge_in) if barge_in else 0.0
|
| 40 |
+
),
|
| 41 |
+
"per_class": per_class,
|
| 42 |
+
"per_language": {
|
| 43 |
+
lang: {"accuracy": sum(ok) / len(ok), "count": len(ok)}
|
| 44 |
+
for lang, ok in sorted(by_language.items())
|
| 45 |
+
},
|
| 46 |
+
}
|
barge_in_detector_model/train.py
ADDED
|
@@ -0,0 +1,140 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Fine-tune a multilingual encoder on the utterance splits (README.md).
|
| 2 |
+
|
| 3 |
+
uv sync --group train
|
| 4 |
+
uv run python -m barge_in_detector_model.train --out models/run1
|
| 5 |
+
|
| 6 |
+
Trains on train.csv's cleaned text, evaluates on validation.csv after every
|
| 7 |
+
epoch, and keeps the epoch with the best validation macro F1 in --out (model,
|
| 8 |
+
tokenizer, metrics.json). Backchannel is the smaller class, so the loss
|
| 9 |
+
weights each class by its inverse frequency. Uses the GPU when there is one.
|
| 10 |
+
"""
|
| 11 |
+
|
| 12 |
+
import argparse
|
| 13 |
+
import json
|
| 14 |
+
import random
|
| 15 |
+
import time
|
| 16 |
+
from collections import Counter
|
| 17 |
+
from collections.abc import Sequence
|
| 18 |
+
from pathlib import Path
|
| 19 |
+
|
| 20 |
+
import torch
|
| 21 |
+
from transformers import (
|
| 22 |
+
AutoModelForSequenceClassification,
|
| 23 |
+
AutoTokenizer,
|
| 24 |
+
get_linear_schedule_with_warmup,
|
| 25 |
+
)
|
| 26 |
+
|
| 27 |
+
from barge_in_detector_model.data import read_csv, resolve_splits
|
| 28 |
+
from barge_in_detector_model.evaluate import device, evaluate
|
| 29 |
+
from barge_in_detector_model.metrics import LABELS
|
| 30 |
+
|
| 31 |
+
# MIT licence, 118M parameters, about 100 languages (dataset repo docs/utterance-first-decision.md).
|
| 32 |
+
DEFAULT_MODEL = "intfloat/multilingual-e5-small"
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
def main(argv: Sequence[str] | None = None) -> int:
|
| 36 |
+
parser = argparse.ArgumentParser(description=__doc__)
|
| 37 |
+
parser.add_argument("--out", type=Path, required=True)
|
| 38 |
+
parser.add_argument("--model", default=DEFAULT_MODEL, help="Hugging Face model id or path")
|
| 39 |
+
parser.add_argument("--splits", type=Path, default=None)
|
| 40 |
+
parser.add_argument("--epochs", type=int, default=5)
|
| 41 |
+
parser.add_argument("--batch-size", type=int, default=32)
|
| 42 |
+
parser.add_argument("--lr", type=float, default=3e-5)
|
| 43 |
+
parser.add_argument("--warmup", type=float, default=0.1, help="share of steps")
|
| 44 |
+
parser.add_argument("--max-length", type=int, default=64, help="token budget")
|
| 45 |
+
parser.add_argument("--seed", type=int, default=20261115)
|
| 46 |
+
parser.add_argument("--dataset-repo", default="HashNuke/barge-in-detector-data")
|
| 47 |
+
parser.add_argument(
|
| 48 |
+
"--dataset-revision", help="Exact dataset commit; resolves current main when omitted"
|
| 49 |
+
)
|
| 50 |
+
args = parser.parse_args(argv)
|
| 51 |
+
args.splits, args.dataset_revision = resolve_splits(
|
| 52 |
+
args.splits, repo_id=args.dataset_repo, revision=args.dataset_revision
|
| 53 |
+
)
|
| 54 |
+
|
| 55 |
+
random.seed(args.seed)
|
| 56 |
+
torch.manual_seed(args.seed)
|
| 57 |
+
dev = device()
|
| 58 |
+
train_rows = read_csv(args.splits / "train.csv")
|
| 59 |
+
validation_rows = read_csv(args.splits / "validation.csv")
|
| 60 |
+
label_id = {label: i for i, label in enumerate(LABELS)}
|
| 61 |
+
|
| 62 |
+
tokenizer = AutoTokenizer.from_pretrained(args.model)
|
| 63 |
+
model = AutoModelForSequenceClassification.from_pretrained(
|
| 64 |
+
args.model,
|
| 65 |
+
num_labels=len(LABELS),
|
| 66 |
+
id2label=dict(enumerate(LABELS)),
|
| 67 |
+
label2id=label_id,
|
| 68 |
+
).to(dev)
|
| 69 |
+
|
| 70 |
+
counts = Counter(r["label"] for r in train_rows)
|
| 71 |
+
weights = torch.tensor(
|
| 72 |
+
[len(train_rows) / (len(LABELS) * counts[label]) for label in LABELS], device=dev
|
| 73 |
+
)
|
| 74 |
+
loss_fn = torch.nn.CrossEntropyLoss(weight=weights)
|
| 75 |
+
steps = args.epochs * -(-len(train_rows) // args.batch_size)
|
| 76 |
+
optimizer = torch.optim.AdamW(model.parameters(), lr=args.lr, weight_decay=0.01)
|
| 77 |
+
scheduler = get_linear_schedule_with_warmup(optimizer, int(args.warmup * steps), steps)
|
| 78 |
+
use_bf16 = dev.type == "cuda" and torch.cuda.is_bf16_supported()
|
| 79 |
+
|
| 80 |
+
print(
|
| 81 |
+
f"device={dev} bf16={use_bf16} model={args.model} train={len(train_rows)} "
|
| 82 |
+
f"validation={len(validation_rows)} class_weights={weights.tolist()}"
|
| 83 |
+
)
|
| 84 |
+
args.out.mkdir(parents=True, exist_ok=True)
|
| 85 |
+
history, best = [], None
|
| 86 |
+
for epoch in range(1, args.epochs + 1):
|
| 87 |
+
model.train()
|
| 88 |
+
order = list(range(len(train_rows)))
|
| 89 |
+
random.shuffle(order)
|
| 90 |
+
started, total = time.time(), 0.0
|
| 91 |
+
for i in range(0, len(order), args.batch_size):
|
| 92 |
+
batch = [train_rows[j] for j in order[i : i + args.batch_size]]
|
| 93 |
+
enc = tokenizer(
|
| 94 |
+
[r["text_clean"] for r in batch],
|
| 95 |
+
truncation=True,
|
| 96 |
+
max_length=args.max_length,
|
| 97 |
+
padding=True,
|
| 98 |
+
return_tensors="pt",
|
| 99 |
+
).to(dev)
|
| 100 |
+
labels = torch.tensor([label_id[r["label"]] for r in batch], device=dev)
|
| 101 |
+
with torch.autocast(device_type=dev.type, dtype=torch.bfloat16, enabled=use_bf16):
|
| 102 |
+
logits = model(**enc).logits
|
| 103 |
+
loss = loss_fn(logits.float(), labels)
|
| 104 |
+
loss.backward()
|
| 105 |
+
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
|
| 106 |
+
optimizer.step()
|
| 107 |
+
scheduler.step()
|
| 108 |
+
optimizer.zero_grad()
|
| 109 |
+
total += loss.item() * len(batch)
|
| 110 |
+
metrics, _ = evaluate(model, tokenizer, validation_rows, max_length=args.max_length)
|
| 111 |
+
record = {
|
| 112 |
+
"epoch": epoch,
|
| 113 |
+
"train_loss": total / len(train_rows),
|
| 114 |
+
"seconds": round(time.time() - started, 1),
|
| 115 |
+
"validation": metrics,
|
| 116 |
+
}
|
| 117 |
+
history.append(record)
|
| 118 |
+
print(
|
| 119 |
+
f"epoch {epoch}: loss={record['train_loss']:.4f} "
|
| 120 |
+
f"validation macro_f1={metrics['macro_f1']:.3f} accuracy={metrics['accuracy']:.3f} "
|
| 121 |
+
f"false_interruptions={metrics['false_interruption_rate']:.3f} "
|
| 122 |
+
f"missed_interruptions={metrics['missed_interruption_rate']:.3f}"
|
| 123 |
+
)
|
| 124 |
+
if best is None or metrics["macro_f1"] > best["validation"]["macro_f1"]:
|
| 125 |
+
best = record
|
| 126 |
+
model.save_pretrained(args.out)
|
| 127 |
+
tokenizer.save_pretrained(args.out)
|
| 128 |
+
(args.out / "metrics.json").write_text(
|
| 129 |
+
json.dumps(
|
| 130 |
+
{"args": {k: str(v) for k, v in vars(args).items()}, "best": best, "history": history},
|
| 131 |
+
indent=2,
|
| 132 |
+
)
|
| 133 |
+
)
|
| 134 |
+
print(f"best epoch {best['epoch']}: validation macro_f1={best['validation']['macro_f1']:.3f}")
|
| 135 |
+
print(f"saved to {args.out}")
|
| 136 |
+
return 0
|
| 137 |
+
|
| 138 |
+
|
| 139 |
+
if __name__ == "__main__":
|
| 140 |
+
raise SystemExit(main())
|
bin/create-labnotes
ADDED
|
@@ -0,0 +1,59 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env bash
|
| 2 |
+
set -euo pipefail
|
| 3 |
+
|
| 4 |
+
usage() {
|
| 5 |
+
cat <<'USAGE'
|
| 6 |
+
Usage:
|
| 7 |
+
bin/create-labnotes 2-4-word-hyphenated-task-name
|
| 8 |
+
|
| 9 |
+
Creates an empty markdown labnotes file in labnotes/.
|
| 10 |
+
|
| 11 |
+
The input must be a 2-4 word hyphenated string, for example:
|
| 12 |
+
bin/create-labnotes test-crawl-resume
|
| 13 |
+
|
| 14 |
+
If spaces are provided, they will be converted to hyphens:
|
| 15 |
+
bin/create-labnotes "test crawl resume"
|
| 16 |
+
USAGE
|
| 17 |
+
}
|
| 18 |
+
|
| 19 |
+
if [[ $# -eq 0 ]]; then
|
| 20 |
+
usage
|
| 21 |
+
exit 0
|
| 22 |
+
fi
|
| 23 |
+
|
| 24 |
+
input="$*"
|
| 25 |
+
slug="$(printf '%s' "$input" \
|
| 26 |
+
| tr '[:upper:]' '[:lower:]' \
|
| 27 |
+
| sed -E 's/[[:space:]]+/-/g; s/[^a-z0-9-]+/-/g; s/-+/-/g; s/^-//; s/-$//')"
|
| 28 |
+
|
| 29 |
+
if [[ -z "$slug" ]]; then
|
| 30 |
+
echo "Error: task name must contain letters or numbers." >&2
|
| 31 |
+
echo >&2
|
| 32 |
+
usage >&2
|
| 33 |
+
exit 1
|
| 34 |
+
fi
|
| 35 |
+
|
| 36 |
+
word_count="$(awk -F- '{ print NF }' <<<"$slug")"
|
| 37 |
+
if (( word_count < 2 || word_count > 4 )); then
|
| 38 |
+
echo "Error: input must be a 2-4 word hyphenated string." >&2
|
| 39 |
+
echo >&2
|
| 40 |
+
usage >&2
|
| 41 |
+
exit 1
|
| 42 |
+
fi
|
| 43 |
+
|
| 44 |
+
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
| 45 |
+
labnotes_dir="$repo_root/labnotes"
|
| 46 |
+
mkdir -p "$labnotes_dir"
|
| 47 |
+
|
| 48 |
+
timestamp="$(date '+%Y%m%d-%H%M')"
|
| 49 |
+
filename="$timestamp-$slug.md"
|
| 50 |
+
path="$labnotes_dir/$filename"
|
| 51 |
+
|
| 52 |
+
if [[ -e "$path" ]]; then
|
| 53 |
+
echo "Error: labnotes/$filename already exists." >&2
|
| 54 |
+
exit 1
|
| 55 |
+
fi
|
| 56 |
+
|
| 57 |
+
touch "$path"
|
| 58 |
+
|
| 59 |
+
echo "Created labnotes/$filename"
|
experiments/benchmark_cpu.py
ADDED
|
@@ -0,0 +1,140 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Measure warmed batch-one tokenization through label output on local CPU.
|
| 2 |
+
|
| 3 |
+
Experimental timing harness: run from the repo root with a CPU PyTorch environment.
|
| 4 |
+
It measures no runtime boundary, queue, HTTP call or cold model load.
|
| 5 |
+
"""
|
| 6 |
+
|
| 7 |
+
import argparse
|
| 8 |
+
import hashlib
|
| 9 |
+
import json
|
| 10 |
+
import platform
|
| 11 |
+
import random
|
| 12 |
+
import resource
|
| 13 |
+
import statistics
|
| 14 |
+
import time
|
| 15 |
+
from pathlib import Path
|
| 16 |
+
|
| 17 |
+
import torch
|
| 18 |
+
import transformers
|
| 19 |
+
from transformers import AutoModelForSequenceClassification, AutoTokenizer
|
| 20 |
+
|
| 21 |
+
from barge_in_detector_model.data import read_csv, resolve_splits
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
def percentile(values, quantile):
|
| 25 |
+
ordered = sorted(values)
|
| 26 |
+
index = (len(ordered) - 1) * quantile
|
| 27 |
+
low = int(index)
|
| 28 |
+
high = min(low + 1, len(ordered) - 1)
|
| 29 |
+
return ordered[low] + (ordered[high] - ordered[low]) * (index - low)
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
def summary(values):
|
| 33 |
+
return {
|
| 34 |
+
"p50_ms": percentile(values, 0.5),
|
| 35 |
+
"p95_ms": percentile(values, 0.95),
|
| 36 |
+
"p99_ms": percentile(values, 0.99),
|
| 37 |
+
"mean_ms": statistics.mean(values),
|
| 38 |
+
"max_ms": max(values),
|
| 39 |
+
}
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
@torch.inference_mode()
|
| 43 |
+
def main():
|
| 44 |
+
parser = argparse.ArgumentParser(description=__doc__)
|
| 45 |
+
parser.add_argument("--iterations", type=int, default=1000)
|
| 46 |
+
parser.add_argument("--warmup", type=int, default=100)
|
| 47 |
+
parser.add_argument("--model", type=Path, default=Path("."))
|
| 48 |
+
parser.add_argument("--threads", type=int, nargs="+", default=[1, 2, 4])
|
| 49 |
+
parser.add_argument("--splits", type=Path, help="Local dataset split directory")
|
| 50 |
+
parser.add_argument("--dataset-revision")
|
| 51 |
+
args = parser.parse_args()
|
| 52 |
+
split_dir, dataset_revision = resolve_splits(args.splits, revision=args.dataset_revision)
|
| 53 |
+
if args.iterations < 100 or args.warmup < 1 or min(args.threads) < 1:
|
| 54 |
+
parser.error("Use >=100 measured iterations, positive warmup and threads")
|
| 55 |
+
torch.set_num_interop_threads(1)
|
| 56 |
+
torch.set_num_threads(1)
|
| 57 |
+
started = time.perf_counter()
|
| 58 |
+
tokenizer = AutoTokenizer.from_pretrained(args.model, local_files_only=True)
|
| 59 |
+
model = (
|
| 60 |
+
AutoModelForSequenceClassification.from_pretrained(args.model, local_files_only=True)
|
| 61 |
+
.cpu()
|
| 62 |
+
.eval()
|
| 63 |
+
)
|
| 64 |
+
load_seconds = time.perf_counter() - started
|
| 65 |
+
rows = read_csv(split_dir / "test.csv")
|
| 66 |
+
texts = [r["text_clean"] for r in rows]
|
| 67 |
+
random.Random(20261115).shuffle(texts)
|
| 68 |
+
stress = (
|
| 69 |
+
"please stop for a moment because i need to correct the information you are reading " * 12
|
| 70 |
+
)
|
| 71 |
+
assert len(tokenizer(stress, truncation=True, max_length=64)["input_ids"]) == 64
|
| 72 |
+
report = {
|
| 73 |
+
"dataset_revision": dataset_revision,
|
| 74 |
+
"model": str(args.model),
|
| 75 |
+
"runtime": "PyTorch eager fp32, CPU only",
|
| 76 |
+
"model_sha256": hashlib.sha256((args.model / "model.safetensors").read_bytes()).hexdigest(),
|
| 77 |
+
"python": platform.python_version(),
|
| 78 |
+
"torch": torch.__version__,
|
| 79 |
+
"transformers": transformers.__version__,
|
| 80 |
+
"platform": platform.platform(),
|
| 81 |
+
"cpuinfo": next(
|
| 82 |
+
line.split(":", 1)[1].strip()
|
| 83 |
+
for line in Path("/proc/cpuinfo").read_text().splitlines()
|
| 84 |
+
if line.startswith("model name")
|
| 85 |
+
),
|
| 86 |
+
"load_seconds_excluded": load_seconds,
|
| 87 |
+
"iterations": args.iterations,
|
| 88 |
+
"warmup_per_scenario": args.warmup,
|
| 89 |
+
"batch_size": 1,
|
| 90 |
+
"max_length": 64,
|
| 91 |
+
"interop_threads": 1,
|
| 92 |
+
"parameter_bytes": sum(p.numel() * p.element_size() for p in model.parameters()),
|
| 93 |
+
"corpus_sha256": hashlib.sha256((split_dir / "test.csv").read_bytes()).hexdigest(),
|
| 94 |
+
"corpus_rows": len(rows),
|
| 95 |
+
"languages": sorted({r["language"] for r in rows}),
|
| 96 |
+
"measurements": [],
|
| 97 |
+
"scope": "Local virtual CPU; reference CPU unspecified. Excludes Vxpipe boundary and queueing; no whole-path acceptance claim.",
|
| 98 |
+
}
|
| 99 |
+
lengths = [len(tokenizer(t, truncation=True, max_length=64)["input_ids"]) for t in texts]
|
| 100 |
+
report["corpus_token_lengths"] = {
|
| 101 |
+
"p50": percentile(lengths, 0.5),
|
| 102 |
+
"p95": percentile(lengths, 0.95),
|
| 103 |
+
"max": max(lengths),
|
| 104 |
+
}
|
| 105 |
+
for threads in args.threads:
|
| 106 |
+
torch.set_num_threads(threads)
|
| 107 |
+
for name, inputs in [("multilingual_test", texts), ("full_64_tokens", [stress])]:
|
| 108 |
+
samples, token_times, forward_times = [], [], []
|
| 109 |
+
for index in range(args.warmup + args.iterations):
|
| 110 |
+
text = inputs[index % len(inputs)]
|
| 111 |
+
start = time.perf_counter_ns()
|
| 112 |
+
enc = tokenizer(text, truncation=True, max_length=64, return_tensors="pt")
|
| 113 |
+
tokenized = time.perf_counter_ns()
|
| 114 |
+
logits = model(**enc).logits
|
| 115 |
+
probability = torch.softmax(logits, dim=-1)[
|
| 116 |
+
0, model.config.label2id["barge_in"]
|
| 117 |
+
].item()
|
| 118 |
+
label = "barge_in" if probability >= 0.5 else "backchannel"
|
| 119 |
+
finish = time.perf_counter_ns()
|
| 120 |
+
assert 0 <= probability <= 1
|
| 121 |
+
assert label in model.config.label2id
|
| 122 |
+
if index >= args.warmup:
|
| 123 |
+
samples.append((finish - start) / 1e6)
|
| 124 |
+
token_times.append((tokenized - start) / 1e6)
|
| 125 |
+
forward_times.append((finish - tokenized) / 1e6)
|
| 126 |
+
result = {
|
| 127 |
+
"threads": threads,
|
| 128 |
+
"scenario": name,
|
| 129 |
+
"model_path": summary(samples),
|
| 130 |
+
"tokenization": summary(token_times),
|
| 131 |
+
"forward_and_probability": summary(forward_times),
|
| 132 |
+
}
|
| 133 |
+
report["measurements"].append(result)
|
| 134 |
+
print(json.dumps(result), flush=True)
|
| 135 |
+
report["peak_process_rss_mib"] = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024
|
| 136 |
+
(args.model / "cpu-benchmark.json").write_text(json.dumps(report, indent=2))
|
| 137 |
+
|
| 138 |
+
|
| 139 |
+
if __name__ == "__main__":
|
| 140 |
+
main()
|
experiments/benchmark_cpu_cold.py
ADDED
|
@@ -0,0 +1,80 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Time a new Python process through its first label from local model files."""
|
| 2 |
+
|
| 3 |
+
import argparse
|
| 4 |
+
import json
|
| 5 |
+
import os
|
| 6 |
+
import statistics
|
| 7 |
+
import subprocess
|
| 8 |
+
import sys
|
| 9 |
+
import time
|
| 10 |
+
from pathlib import Path
|
| 11 |
+
|
| 12 |
+
CHILD = """
|
| 13 |
+
import json
|
| 14 |
+
import time
|
| 15 |
+
import torch
|
| 16 |
+
from transformers import AutoModelForSequenceClassification, AutoTokenizer
|
| 17 |
+
import sys
|
| 18 |
+
model_path, threads = sys.argv[1], int(sys.argv[2])
|
| 19 |
+
torch.set_num_threads(threads)
|
| 20 |
+
torch.set_num_interop_threads(1)
|
| 21 |
+
tokenizer = AutoTokenizer.from_pretrained(model_path, local_files_only=True)
|
| 22 |
+
model = AutoModelForSequenceClassification.from_pretrained(model_path, local_files_only=True).cpu().eval()
|
| 23 |
+
with torch.inference_mode():
|
| 24 |
+
loaded_start = time.perf_counter_ns()
|
| 25 |
+
enc = tokenizer('please stop for a moment', truncation=True, max_length=64, return_tensors='pt')
|
| 26 |
+
p = torch.softmax(model(**enc).logits, dim=-1)[0, model.config.label2id['barge_in']].item()
|
| 27 |
+
label = 'barge_in' if p >= 0.5 else 'backchannel'
|
| 28 |
+
loaded_ms = (time.perf_counter_ns() - loaded_start) / 1e6
|
| 29 |
+
print(json.dumps({'label': label, 'p_barge_in': p, 'loaded_to_first_label_ms': loaded_ms}), flush=True)
|
| 30 |
+
"""
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
def main():
|
| 34 |
+
parser = argparse.ArgumentParser(description=__doc__)
|
| 35 |
+
parser.add_argument("--model", type=Path, default=Path("."))
|
| 36 |
+
parser.add_argument("--runs", type=int, default=10)
|
| 37 |
+
parser.add_argument("--threads", type=int, default=1)
|
| 38 |
+
args = parser.parse_args()
|
| 39 |
+
samples = []
|
| 40 |
+
loaded_first = []
|
| 41 |
+
env = {
|
| 42 |
+
**os.environ,
|
| 43 |
+
"OMP_NUM_THREADS": str(args.threads),
|
| 44 |
+
"MKL_NUM_THREADS": str(args.threads),
|
| 45 |
+
"TOKENIZERS_PARALLELISM": "false",
|
| 46 |
+
}
|
| 47 |
+
for run in range(args.runs):
|
| 48 |
+
start = time.perf_counter_ns()
|
| 49 |
+
with subprocess.Popen(
|
| 50 |
+
[sys.executable, "-c", CHILD, str(args.model), str(args.threads)],
|
| 51 |
+
stdout=subprocess.PIPE,
|
| 52 |
+
stderr=subprocess.DEVNULL,
|
| 53 |
+
text=True,
|
| 54 |
+
env=env,
|
| 55 |
+
) as process:
|
| 56 |
+
line = process.stdout.readline()
|
| 57 |
+
elapsed = (time.perf_counter_ns() - start) / 1e9
|
| 58 |
+
result = json.loads(line)
|
| 59 |
+
if process.wait() != 0:
|
| 60 |
+
raise RuntimeError("Cold-start child failed")
|
| 61 |
+
samples.append(elapsed)
|
| 62 |
+
loaded_first.append(result["loaded_to_first_label_ms"])
|
| 63 |
+
print(f"cold start {run + 1}: {elapsed:.3f}s {result}", flush=True)
|
| 64 |
+
report = {
|
| 65 |
+
"cold_process_seconds": samples,
|
| 66 |
+
"loaded_to_first_label_ms": loaded_first,
|
| 67 |
+
"loaded_to_first_label_median_ms": statistics.median(loaded_first),
|
| 68 |
+
"median_seconds": statistics.median(samples),
|
| 69 |
+
"min_seconds": min(samples),
|
| 70 |
+
"max_seconds": max(samples),
|
| 71 |
+
"runs": args.runs,
|
| 72 |
+
"threads": args.threads,
|
| 73 |
+
"scope": "Python process creation through reading first flushed label. Includes imports, tokenizer/model load, first inference. Local files only; OS file cache may be warm. Excludes process shutdown, downloads and HTTP/network boundary.",
|
| 74 |
+
}
|
| 75 |
+
(args.model / "cpu-cold-benchmark.json").write_text(json.dumps(report, indent=2))
|
| 76 |
+
print(json.dumps(report), flush=True)
|
| 77 |
+
|
| 78 |
+
|
| 79 |
+
if __name__ == "__main__":
|
| 80 |
+
main()
|
experiments/evaluate_english_challenge.py
ADDED
|
@@ -0,0 +1,132 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Frozen-checkpoint diagnostic experiment; labels are authored before inference.
|
| 2 |
+
|
| 3 |
+
Run from the repo root with uv run --group train python -m experiments.evaluate_english_challenge.
|
| 4 |
+
Evidence is saved under the model directory, outside the training splits.
|
| 5 |
+
"""
|
| 6 |
+
|
| 7 |
+
import argparse
|
| 8 |
+
import csv
|
| 9 |
+
import hashlib
|
| 10 |
+
import json
|
| 11 |
+
from collections import defaultdict
|
| 12 |
+
from pathlib import Path
|
| 13 |
+
|
| 14 |
+
import torch
|
| 15 |
+
import transformers
|
| 16 |
+
from transformers import AutoModelForSequenceClassification, AutoTokenizer
|
| 17 |
+
|
| 18 |
+
from barge_in_detector_model.data import read_csv, resolve_splits
|
| 19 |
+
from barge_in_detector_model.evaluate import device, predict
|
| 20 |
+
from barge_in_detector_model.metrics import compute
|
| 21 |
+
|
| 22 |
+
MODEL = Path(".")
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
def score(rows, probs):
|
| 26 |
+
return compute(
|
| 27 |
+
[r["label"] for r in rows],
|
| 28 |
+
["barge_in" if p >= 0.5 else "backchannel" for p in probs],
|
| 29 |
+
[r["language"] for r in rows],
|
| 30 |
+
)
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
def main():
|
| 34 |
+
parser = argparse.ArgumentParser(description=__doc__)
|
| 35 |
+
parser.add_argument("--dataset-dir", type=Path, help="Local dataset repository root")
|
| 36 |
+
parser.add_argument("--dataset-revision")
|
| 37 |
+
args = parser.parse_args()
|
| 38 |
+
if args.dataset_dir is None:
|
| 39 |
+
split_dir, revision = resolve_splits(None, revision=args.dataset_revision)
|
| 40 |
+
dataset_dir = split_dir.parents[1]
|
| 41 |
+
else:
|
| 42 |
+
dataset_dir = args.dataset_dir
|
| 43 |
+
revision = None
|
| 44 |
+
challenge = dataset_dir / "data/evaluation/english-challenge-v1.csv"
|
| 45 |
+
rows = read_csv(challenge)
|
| 46 |
+
splits = {
|
| 47 |
+
s: read_csv(dataset_dir / f"data/splits/{s}.csv") for s in ("train", "validation", "test")
|
| 48 |
+
}
|
| 49 |
+
texts = {s: {r["text_clean"] for r in rs} for s, rs in splits.items()}
|
| 50 |
+
all_seen = set.union(*texts.values())
|
| 51 |
+
tokenizer = AutoTokenizer.from_pretrained(MODEL)
|
| 52 |
+
model = AutoModelForSequenceClassification.from_pretrained(MODEL).to(device())
|
| 53 |
+
probs = predict(model, tokenizer, rows, max_length=64)
|
| 54 |
+
single_probs = predict(model, tokenizer, rows, batch_size=1, max_length=64)
|
| 55 |
+
report = {
|
| 56 |
+
"dataset_revision": revision,
|
| 57 |
+
"challenge_sha256": hashlib.sha256(challenge.read_bytes()).hexdigest(),
|
| 58 |
+
"model_sha256": hashlib.sha256((MODEL / "model.safetensors").read_bytes()).hexdigest(),
|
| 59 |
+
"threshold": 0.5,
|
| 60 |
+
"runtime": {
|
| 61 |
+
"device": str(next(model.parameters()).device),
|
| 62 |
+
"torch": torch.__version__,
|
| 63 |
+
"transformers": transformers.__version__,
|
| 64 |
+
"threads": torch.get_num_threads(),
|
| 65 |
+
},
|
| 66 |
+
"batch_vs_single_max_probability_difference": max(
|
| 67 |
+
abs(a - b) for a, b in zip(probs, single_probs, strict=True)
|
| 68 |
+
),
|
| 69 |
+
"batch_vs_single_label_disagreements": sum(
|
| 70 |
+
(a >= 0.5) != (b >= 0.5) for a, b in zip(probs, single_probs, strict=True)
|
| 71 |
+
),
|
| 72 |
+
"review_status": "Owner-reviewed v1 labels match authored expectations after authorized reconciliation. Diagnostic development set; not independent representative accuracy evidence.",
|
| 73 |
+
"all": score(rows, probs),
|
| 74 |
+
"by_category": {},
|
| 75 |
+
"overlap_counts": {s: sum(r["text_clean"] in ts for r in rows) for s, ts in texts.items()},
|
| 76 |
+
"cross_split_exact_overlap": {
|
| 77 |
+
f"{a}/{b}": len(texts[a] & texts[b])
|
| 78 |
+
for a, b in (("train", "validation"), ("train", "test"), ("validation", "test"))
|
| 79 |
+
},
|
| 80 |
+
}
|
| 81 |
+
for category in sorted({r["category"] for r in rows}):
|
| 82 |
+
pairs = [(r, p) for r, p in zip(rows, probs, strict=True) if r["category"] == category]
|
| 83 |
+
report["by_category"][category] = score(*zip(*pairs, strict=True))
|
| 84 |
+
unseen = [(r, p) for r, p in zip(rows, probs, strict=True) if r["text_clean"] not in all_seen]
|
| 85 |
+
report["unseen"] = score(*zip(*unseen, strict=True))
|
| 86 |
+
groups = defaultdict(list)
|
| 87 |
+
for r, p in zip(rows, probs, strict=True):
|
| 88 |
+
if r["group"]:
|
| 89 |
+
groups[r["group"]].append(
|
| 90 |
+
{"text": r["text_clean"], "label": r["label"], "p_barge_in": p}
|
| 91 |
+
)
|
| 92 |
+
report["prefix_groups"] = dict(groups)
|
| 93 |
+
report["prefix_groups_all_labels_correct"] = sum(
|
| 94 |
+
all((r["p_barge_in"] >= 0.5) == (r["label"] == "barge_in") for r in group)
|
| 95 |
+
for group in groups.values()
|
| 96 |
+
)
|
| 97 |
+
# A diagnostic baseline: select its word-count cutoff using validation only.
|
| 98 |
+
english = {s: [r for r in rs if r["language"] == "en"] for s, rs in splits.items()}
|
| 99 |
+
|
| 100 |
+
def length_probs(rs, cutoff):
|
| 101 |
+
return [float(len(r["text_clean"].split()) >= cutoff) for r in rs]
|
| 102 |
+
|
| 103 |
+
cutoff = max(
|
| 104 |
+
range(1, 25),
|
| 105 |
+
key=lambda n: score(english["validation"], length_probs(english["validation"], n))[
|
| 106 |
+
"macro_f1"
|
| 107 |
+
],
|
| 108 |
+
)
|
| 109 |
+
report["english_length_baseline"] = {
|
| 110 |
+
"cutoff_words": cutoff,
|
| 111 |
+
"validation": score(english["validation"], length_probs(english["validation"], cutoff)),
|
| 112 |
+
"test": score(english["test"], length_probs(english["test"], cutoff)),
|
| 113 |
+
"challenge": score(rows, length_probs(rows, cutoff)),
|
| 114 |
+
}
|
| 115 |
+
with (MODEL / "challenge-predictions.csv").open("w", newline="") as f:
|
| 116 |
+
writer = csv.DictWriter(f, fieldnames=[*rows[0], "p_barge_in", "prediction", "seen_in"])
|
| 117 |
+
writer.writeheader()
|
| 118 |
+
for r, p in zip(rows, probs, strict=True):
|
| 119 |
+
writer.writerow(
|
| 120 |
+
{
|
| 121 |
+
**r,
|
| 122 |
+
"p_barge_in": p,
|
| 123 |
+
"prediction": "barge_in" if p >= 0.5 else "backchannel",
|
| 124 |
+
"seen_in": ";".join(s for s, ts in texts.items() if r["text_clean"] in ts),
|
| 125 |
+
}
|
| 126 |
+
)
|
| 127 |
+
(MODEL / "challenge-report.json").write_text(json.dumps(report, indent=2))
|
| 128 |
+
print(json.dumps({k: v for k, v in report.items() if k != "prefix_groups"}, indent=2))
|
| 129 |
+
|
| 130 |
+
|
| 131 |
+
if __name__ == "__main__":
|
| 132 |
+
main()
|
labnotes/20261115-1332-repository-split.md
ADDED
|
@@ -0,0 +1,24 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Model repository split
|
| 2 |
+
|
| 3 |
+
Owner moved model artifacts, training, inference, evaluation and benchmarks into
|
| 4 |
+
this standalone Hugging Face model repo. Dataset generation, splitting, review
|
| 5 |
+
and all earlier labnotes remain in HashNuke/barge-in-detector-data. This repo has
|
| 6 |
+
its own scoped AGENTS.md, bin/create-labnotes and milestone index.
|
| 7 |
+
|
| 8 |
+
- New package barge_in_detector_model isolates training/metrics/data-loading
|
| 9 |
+
code from the dataset generator. Model code downloads a resolved dataset
|
| 10 |
+
commit, or uses explicit local splits; it does not assign splits.
|
| 11 |
+
- Four unit tests cover metrics, local CSVs, missing split detection and pinned
|
| 12 |
+
Hub downloads. Tiny CPU smoke trained one epoch, saved, reloaded and evaluated.
|
| 13 |
+
- Actual unchanged checkpoint predicts stop as backchannel, p=0.0134437,
|
| 14 |
+
reproducing the known failure. No GPU rental or retraining of release weights.
|
| 15 |
+
- Token budget remains 64; two labels; predictions are not calibrated.
|
| 16 |
+
- Root weights and tokenizer match the original artifact hashes. Original source
|
| 17 |
+
manifest and sanitized training metrics retain run provenance. Model card
|
| 18 |
+
reports synthetic and reviewed diagnostics and local CPU timing limitations.
|
| 19 |
+
- AGENTS.md derives CPU serving gates, fourteen languages, single RTX 5090,
|
| 20 |
+
under-$0.65/hour datacenter/PyTorch policy, gpubox, tests, and labnotes/milestone
|
| 21 |
+
usage from the dataset instructions. Existing history is linked, not duplicated.
|
| 22 |
+
|
| 23 |
+
Historical run and latency evidence is in the dataset repo labnotes. Follow
|
| 24 |
+
labnotes/milestones/index.md for new model work. Publication verification follows.
|
labnotes/issues/.gitkeep
ADDED
|
File without changes
|
labnotes/milestones/experimental-checkpoint-release.md
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Experimental checkpoint release
|
| 2 |
+
|
| 3 |
+
## Runnable outcome
|
| 4 |
+
|
| 5 |
+
A self-contained Hugging Face model repo that loads the first checkpoint on CPU,
|
| 6 |
+
trains/evaluates with a separate versioned dataset, and records honest limitations.
|
| 7 |
+
Prerequisites: dataset HashNuke/barge-in-detector-data, original generation/training
|
| 8 |
+
history in that dataset repo, root README and AGENTS.md.
|
| 9 |
+
|
| 10 |
+
## Contracts and checkpoints
|
| 11 |
+
|
| 12 |
+
- [x] Copy and verify selected epoch-5 weights and tokenizer without retraining.
|
| 13 |
+
- [x] Move training/evaluation/metrics/benchmarks into independent model modules.
|
| 14 |
+
- [x] Resolve Hub dataset revisions to immutable commits; support local splits.
|
| 15 |
+
- [x] Copy labnotes helper; derive model AGENTS.md and establish milestone usage.
|
| 16 |
+
- [x] Tiny CPU train/save/reload/evaluate smoke and actual checkpoint inference.
|
| 17 |
+
- [ ] Publish artifacts, instructions and code to the model repo.
|
| 18 |
+
- [ ] Record dataset commit, source snapshot and artifact checksums in release.json.
|
| 19 |
+
|
| 20 |
+
## Acceptance gates
|
| 21 |
+
|
| 22 |
+
- [ ] Four model unit tests, lint and formatting pass.
|
| 23 |
+
- [ ] Model weights and tokenizer resolve as large-file artifacts on the Hub.
|
| 24 |
+
- [ ] Published files match local release hashes; CPU inference works from files.
|
| 25 |
+
- [ ] Model card separates synthetic and reviewed diagnostics and records failures.
|
| 26 |
+
|
| 27 |
+
This milestone does not claim production readiness, calibration or reference-CPU
|
| 28 |
+
latency acceptance. See repository-split labnotes for evidence.
|
| 29 |
+
|
| 30 |
+
Checkpoint evidence: [repository split labnotes](../20261115-1332-repository-split.md).
|
labnotes/milestones/index.md
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Model milestones
|
| 2 |
+
|
| 3 |
+
The checklist owns implementation order. Read AGENTS.md and linked sources;
|
| 4 |
+
leave entries unchecked until every acceptance gate passes.
|
| 5 |
+
|
| 6 |
+
- [ ] [Publish the experimental checkpoint](experimental-checkpoint-release.md)
|
| 7 |
+
- [ ] [Improve short-interruption recall](short-interruption-recall.md)
|
labnotes/milestones/short-interruption-recall.md
ADDED
|
@@ -0,0 +1,28 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Improve short-interruption recall
|
| 2 |
+
|
| 3 |
+
## Runnable outcome and prerequisites
|
| 4 |
+
|
| 5 |
+
Improve the current checkpoint's 5/30 short-interruption recall on reviewed
|
| 6 |
+
English development examples, then establish performance on fresh owner-reviewed
|
| 7 |
+
English/Hindi evaluation data. Prerequisite: experimental checkpoint release;
|
| 8 |
+
coordinate data changes with the dataset repo coverage-audit milestone.
|
| 9 |
+
|
| 10 |
+
## Contract and checkpoint tasks
|
| 11 |
+
|
| 12 |
+
- [ ] Agree acceptance floors for false and missed interruptions before tuning.
|
| 13 |
+
- [ ] Compare a character n-gram baseline and encoder on exactly the same splits.
|
| 14 |
+
- [ ] Train against the revised dataset distribution; never train on challenge rows.
|
| 15 |
+
- [ ] Evaluate short interruptions, longer backchannels, prefix transitions and
|
| 16 |
+
per-language errors. Preserve old checkpoint and use validation for selection.
|
| 17 |
+
- [ ] Record immutable dataset/source revisions, parameters and costs.
|
| 18 |
+
- [ ] Calibrate on realistic validation priors.
|
| 19 |
+
- [ ] Measure ONNX/int8 parity and reference-CPU latency/memory; measure Vxpipe
|
| 20 |
+
serialization/queue/boundary overhead for the whole-path budget.
|
| 21 |
+
|
| 22 |
+
## Acceptance gates
|
| 23 |
+
|
| 24 |
+
- [ ] Fresh human-reviewed English/Hindi results meet agreed floors and beat baseline.
|
| 25 |
+
- [ ] Model-path p99 <=20 ms; whole path p99 <=50 ms on reference CPU, warmed batch one.
|
| 26 |
+
- [ ] Prefix behaviour and Vxpipe overlapping-speech scenarios pass with evidence.
|
| 27 |
+
|
| 28 |
+
Use `bin/create-labnotes` and link each checkpoint's factual evidence here.
|
pyproject.toml
ADDED
|
@@ -0,0 +1,25 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[project]
|
| 2 |
+
name = "barge-in-detector-model"
|
| 3 |
+
version = "0.1.0"
|
| 4 |
+
description = "Training, evaluation and CPU inference for the Vxpipe barge-in classifier"
|
| 5 |
+
requires-python = ">=3.14"
|
| 6 |
+
dependencies = ["torch>=2.15.0", "transformers>=5.23.0", "huggingface-hub>=2.4.0"]
|
| 7 |
+
|
| 8 |
+
[dependency-groups]
|
| 9 |
+
dev = ["pytest>=9.2.0", "ruff>=0.17.5"]
|
| 10 |
+
|
| 11 |
+
[tool.uv]
|
| 12 |
+
package = false
|
| 13 |
+
|
| 14 |
+
[tool.pytest.ini_options]
|
| 15 |
+
pythonpath = ["."]
|
| 16 |
+
testpaths = ["tests"]
|
| 17 |
+
addopts = "-m 'not integration'"
|
| 18 |
+
markers = ["integration: calls networks; excluded by default"]
|
| 19 |
+
|
| 20 |
+
[tool.ruff]
|
| 21 |
+
line-length = 100
|
| 22 |
+
target-version = "py314"
|
| 23 |
+
[tool.ruff.lint]
|
| 24 |
+
select = ["E", "F", "I", "B", "UP", "SIM"]
|
| 25 |
+
ignore = ["E501"]
|
release.json
ADDED
|
@@ -0,0 +1,34 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model_id": "HashNuke/barge-in-detector",
|
| 3 |
+
"status": "experimental, uncalibrated, short-interruption failures",
|
| 4 |
+
"base_model": "intfloat/multilingual-e5-small",
|
| 5 |
+
"dataset_repo": "HashNuke/barge-in-detector-data",
|
| 6 |
+
"dataset_revision": "dda7115ace1fe7b07c49139860131f39910521ad",
|
| 7 |
+
"dataset_files_sha256": {
|
| 8 |
+
"data/export/utterances.csv": "556a2820e4a24ac13bab44321fb32b464c5fb225750d41a5a10ebcde7cdbab8a",
|
| 9 |
+
"data/splits/train.csv": "aab224bd0526eee915295e6f5c52076e29bb751cd0d664eefb4321843ad22533",
|
| 10 |
+
"data/splits/validation.csv": "c6b4c526fb4b1df96754119e409cedc8c7e85c77659e1fc1c3829da70084edc1",
|
| 11 |
+
"data/splits/test.csv": "8b630782362b05249b3be94b0213850258422be5ddeb62fcf703817b0bf9e11a",
|
| 12 |
+
"data/evaluation/english-challenge-v1.csv": "3d46b9e024f4d3bedd69f07603cde934e85ec2272988e90d2acebc30d5e99519",
|
| 13 |
+
"data/evaluation/reviews/english-challenge-v1.json": "ee6e07c3218ab0412d179e73b8e7457d550447a7ae08db2a005f37d794b855fa"
|
| 14 |
+
},
|
| 15 |
+
"artifacts_sha256": {
|
| 16 |
+
"model.safetensors": "bddb65c08f6260cfc9d320e65b4229229ec9f3ca95204aed98ab5ae45ff758d3",
|
| 17 |
+
"config.json": "e37f926b37b69e1ae223add18a9b0e993d6000cfe87c44f2dd42ce157654d93b",
|
| 18 |
+
"tokenizer.json": "255f5e32cb320b4dc2b31e0f203832fecafe87ca4cc87c3c41eb8ccae73f441a",
|
| 19 |
+
"tokenizer_config.json": "33b75086cc783ee5d78cab4dae0762d9cab1c40a8e0feb6e65d60c8d69a6afc1"
|
| 20 |
+
},
|
| 21 |
+
"original_training_source": {
|
| 22 |
+
"git_base_commit": "5ce0ed1283401f8c73bafcf5a7dbd2101c4bbee6",
|
| 23 |
+
"overlay_manifest": "reports/source-manifest.json"
|
| 24 |
+
},
|
| 25 |
+
"migration_source_git_commit": "1f92917a59908e320c3a3489c3e5de413a3e4672",
|
| 26 |
+
"labels": {
|
| 27 |
+
"0": "backchannel",
|
| 28 |
+
"1": "barge_in"
|
| 29 |
+
},
|
| 30 |
+
"max_length": 64,
|
| 31 |
+
"default_diagnostic_threshold": 0.5,
|
| 32 |
+
"selected_epoch": 5,
|
| 33 |
+
"training_metrics": "reports/training-metrics.json"
|
| 34 |
+
}
|
reports/challenge-predictions.csv
ADDED
|
@@ -0,0 +1,99 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
case_id,category,group,language,label,text,text_clean,review_status,p_barge_in,prediction,seen_in
|
| 2 |
+
en-001,short_interruption,,en,barge_in,Stop,stop,pending,0.013443702831864357,backchannel,
|
| 3 |
+
en-002,short_interruption,,en,barge_in,Wait,wait,pending,0.013491915538907051,backchannel,
|
| 4 |
+
en-003,short_interruption,,en,barge_in,Pause,pause,pending,0.013399876654148102,backchannel,
|
| 5 |
+
en-004,short_interruption,,en,barge_in,Cancel,cancel,pending,0.0172202717512846,backchannel,
|
| 6 |
+
en-005,short_interruption,,en,barge_in,Wrong,wrong,pending,0.015773363411426544,backchannel,
|
| 7 |
+
en-006,short_interruption,,en,barge_in,Repeat,repeat,pending,0.01339809875935316,backchannel,
|
| 8 |
+
en-007,short_interruption,,en,barge_in,Slower,slower,pending,0.014412087388336658,backchannel,
|
| 9 |
+
en-008,short_interruption,,en,barge_in,Restart,restart,pending,0.014540360309183598,backchannel,
|
| 10 |
+
en-009,short_interruption,,en,barge_in,Enough,enough,pending,0.013300837948918343,backchannel,
|
| 11 |
+
en-010,short_interruption,,en,barge_in,Stop talking,stop talking,pending,0.01404176652431488,backchannel,
|
| 12 |
+
en-011,short_interruption,,en,barge_in,Wait please,wait please,pending,0.013346969150006771,backchannel,
|
| 13 |
+
en-012,short_interruption,,en,barge_in,Not that,not that,pending,0.015475512482225895,backchannel,
|
| 14 |
+
en-013,short_interruption,,en,barge_in,Wrong date,wrong date,pending,0.9878827333450317,barge_in,
|
| 15 |
+
en-014,short_interruption,,en,barge_in,Wrong person,wrong person,pending,0.8659051060676575,barge_in,
|
| 16 |
+
en-015,short_interruption,,en,barge_in,Go back,go back,pending,0.013336537405848503,backchannel,
|
| 17 |
+
en-016,short_interruption,,en,barge_in,Say again,say again,pending,0.013324333354830742,backchannel,
|
| 18 |
+
en-017,short_interruption,,en,barge_in,Speak slower,speak slower,pending,0.013362826779484749,backchannel,
|
| 19 |
+
en-018,short_interruption,,en,barge_in,Start over,start over,pending,0.01339060626924038,backchannel,
|
| 20 |
+
en-019,short_interruption,,en,barge_in,Let me finish,let me finish,pending,0.013296245597302914,backchannel,
|
| 21 |
+
en-020,short_interruption,,en,barge_in,You misunderstood,you misunderstood,pending,0.013550461269915104,backchannel,
|
| 22 |
+
en-021,short_interruption,,en,barge_in,Cancel it,cancel it,pending,0.015159462578594685,backchannel,
|
| 23 |
+
en-022,short_interruption,,en,barge_in,Change it,change it,pending,0.013769036158919334,backchannel,
|
| 24 |
+
en-023,short_interruption,,en,barge_in,Which one,which one,pending,0.017033280804753304,backchannel,
|
| 25 |
+
en-024,short_interruption,,en,barge_in,Why not,why not,pending,0.014318976551294327,backchannel,
|
| 26 |
+
en-025,short_interruption,,en,barge_in,How much,how much,pending,0.016760120168328285,backchannel,
|
| 27 |
+
en-026,short_interruption,,en,barge_in,What time,what time,pending,0.9822450876235962,barge_in,
|
| 28 |
+
en-027,short_interruption,,en,barge_in,What address,what address,pending,0.9860103130340576,barge_in,
|
| 29 |
+
en-028,short_interruption,,en,barge_in,Tuesday instead,tuesday instead,pending,0.7567923069000244,barge_in,
|
| 30 |
+
en-029,short_interruption,,en,barge_in,The second one,the second one,pending,0.02150067687034607,backchannel,
|
| 31 |
+
en-030,short_interruption,,en,barge_in,That is wrong,that is wrong,pending,0.1904229372739792,backchannel,
|
| 32 |
+
en-031,long_backchannel,,en,backchannel,Yes yes yes yes yes,yes yes yes yes yes,pending,0.013356473296880722,backchannel,
|
| 33 |
+
en-032,long_backchannel,,en,backchannel,Okay okay okay okay okay,okay okay okay okay okay,pending,0.013345411978662014,backchannel,
|
| 34 |
+
en-033,long_backchannel,,en,backchannel,Right right right I see,right right right i see,pending,0.013284565880894661,backchannel,
|
| 35 |
+
en-034,long_backchannel,,en,backchannel,Mm hmm yes mm hmm yes,mm hmm yes mm hmm yes,pending,0.013306752778589725,backchannel,
|
| 36 |
+
en-035,long_backchannel,,en,backchannel,Oh wow oh wow yeah,oh wow oh wow yeah,pending,0.013319982215762138,backchannel,
|
| 37 |
+
en-036,long_backchannel,,en,backchannel,Yeah I see I see,yeah i see i see,pending,0.013288560323417187,backchannel,
|
| 38 |
+
en-037,long_backchannel,,en,backchannel,Yes I understand what you mean,yes i understand what you mean,pending,0.0892898291349411,backchannel,
|
| 39 |
+
en-038,long_backchannel,,en,backchannel,Right that makes sense to me,right that makes sense to me,pending,0.013280664570629597,backchannel,
|
| 40 |
+
en-039,long_backchannel,,en,backchannel,Okay I am following you so far,okay i am following you so far,pending,0.28858309984207153,backchannel,
|
| 41 |
+
en-040,long_backchannel,,en,backchannel,Yes yes that sounds right,yes yes that sounds right,pending,0.013292225077748299,backchannel,
|
| 42 |
+
en-041,long_backchannel,,en,backchannel,Ah I see what you mean now,ah i see what you mean now,pending,0.32564374804496765,backchannel,
|
| 43 |
+
en-042,long_backchannel,,en,backchannel,Mm hmm I understand yes,mm hmm i understand yes,pending,0.0132778724655509,backchannel,
|
| 44 |
+
en-043,long_backchannel,,en,backchannel,Oh dear that sounds really difficult,oh dear that sounds really difficult,pending,0.047465287148952484,backchannel,
|
| 45 |
+
en-044,long_backchannel,,en,backchannel,Wow that must have been exciting,wow that must have been exciting,pending,0.013364902697503567,backchannel,
|
| 46 |
+
en-045,long_backchannel,,en,backchannel,That is interesting yes I see,that is interesting yes i see,pending,0.013278285041451454,backchannel,
|
| 47 |
+
en-046,long_backchannel,,en,backchannel,Fair enough I understand that,fair enough i understand that,pending,0.013284215703606606,backchannel,
|
| 48 |
+
en-047,long_backchannel,,en,backchannel,Yes absolutely I agree with that,yes absolutely i agree with that,pending,0.013294750824570656,backchannel,
|
| 49 |
+
en-048,long_backchannel,,en,backchannel,Right okay yes that makes sense,right okay yes that makes sense,pending,0.013285371474921703,backchannel,
|
| 50 |
+
en-049,long_backchannel,,en,backchannel,Oh no that is really sad,oh no that is really sad,pending,0.01329362578690052,backchannel,
|
| 51 |
+
en-050,long_backchannel,,en,backchannel,Ha ha that is funny,ha ha that is funny,pending,0.013284615240991116,backchannel,
|
| 52 |
+
en-051,long_backchannel,,en,backchannel,Mmm yes okay I hear you,mmm yes okay i hear you,pending,0.013274511322379112,backchannel,
|
| 53 |
+
en-052,long_backchannel,,en,backchannel,Sure sure I am with you,sure sure i am with you,pending,0.013305414468050003,backchannel,
|
| 54 |
+
en-053,long_backchannel,,en,backchannel,Got it yes understood,got it yes understood,pending,0.013286358676850796,backchannel,
|
| 55 |
+
en-054,long_backchannel,,en,backchannel,Uh huh uh huh right,uh huh uh huh right,pending,0.013300181366503239,backchannel,
|
| 56 |
+
en-055,ordinary_backchannel,,en,backchannel,Okay,okay,pending,0.013304519467055798,backchannel,
|
| 57 |
+
en-056,ordinary_backchannel,,en,backchannel,Yes,yes,pending,0.013361463323235512,backchannel,
|
| 58 |
+
en-057,ordinary_backchannel,,en,backchannel,Yeah,yeah,pending,0.013312126509845257,backchannel,
|
| 59 |
+
en-058,ordinary_backchannel,,en,backchannel,Mm hmm,mm hmm,pending,0.013299750164151192,backchannel,validation
|
| 60 |
+
en-059,ordinary_backchannel,,en,backchannel,Uh huh,uh huh,pending,0.013296402990818024,backchannel,train
|
| 61 |
+
en-060,ordinary_backchannel,,en,backchannel,Right,right,pending,0.013306460343301296,backchannel,
|
| 62 |
+
en-061,ordinary_backchannel,,en,backchannel,I see,i see,pending,0.013295376673340797,backchannel,
|
| 63 |
+
en-062,ordinary_backchannel,,en,backchannel,Got it,got it,pending,0.01331440731883049,backchannel,
|
| 64 |
+
en-063,ordinary_backchannel,,en,backchannel,Oh wow,oh wow,pending,0.013293537311255932,backchannel,train
|
| 65 |
+
en-064,ordinary_backchannel,,en,backchannel,I understand,i understand,pending,0.013287884183228016,backchannel,
|
| 66 |
+
en-065,ordinary_interruption,,en,barge_in,I need to change my delivery address,i need to change my delivery address,pending,0.9881365299224854,barge_in,
|
| 67 |
+
en-066,ordinary_interruption,,en,barge_in,You have the wrong customer on the line,you have the wrong customer on the line,pending,0.9880920648574829,barge_in,
|
| 68 |
+
en-067,ordinary_interruption,,en,barge_in,Can you explain what that fee is for,can you explain what that fee is for,pending,0.9879797697067261,barge_in,
|
| 69 |
+
en-068,ordinary_interruption,,en,barge_in,Please stop reading and let me speak,please stop reading and let me speak,pending,0.9880907535552979,barge_in,
|
| 70 |
+
en-069,ordinary_interruption,,en,barge_in,Actually I already paid that invoice,actually i already paid that invoice,pending,0.9881226420402527,barge_in,
|
| 71 |
+
en-070,ordinary_interruption,,en,barge_in,I want to speak to someone else,i want to speak to someone else,pending,0.9879534244537354,barge_in,
|
| 72 |
+
en-071,ordinary_interruption,,en,barge_in,No I said Thursday and not Tuesday,no i said thursday and not tuesday,pending,0.9881582856178284,barge_in,
|
| 73 |
+
en-072,ordinary_interruption,,en,barge_in,Before you continue I have a question,before you continue i have a question,pending,0.9879493713378906,barge_in,
|
| 74 |
+
en-073,ordinary_interruption,,en,barge_in,That is not what I was asking about,that is not what i was asking about,pending,0.9879181981086731,barge_in,
|
| 75 |
+
en-074,ordinary_interruption,,en,barge_in,I cannot hear you please say that again,i cannot hear you please say that again,pending,0.9881183505058289,barge_in,
|
| 76 |
+
en-075,prefix,prefix-1,en,backchannel,Yes,yes,pending,0.013361463323235512,backchannel,
|
| 77 |
+
en-076,prefix,prefix-1,en,backchannel,Yes yes,yes yes,pending,0.013298917561769485,backchannel,
|
| 78 |
+
en-077,prefix,prefix-1,en,barge_in,Yes yes but change the time,yes yes but change the time,pending,0.9880699515342712,barge_in,
|
| 79 |
+
en-078,prefix,prefix-2,en,backchannel,Okay,okay,pending,0.013304519467055798,backchannel,
|
| 80 |
+
en-079,prefix,prefix-2,en,backchannel,Okay okay,okay okay,pending,0.01330505684018135,backchannel,
|
| 81 |
+
en-080,prefix,prefix-2,en,barge_in,Okay okay wait that is the wrong address,okay okay wait that is the wrong address,pending,0.988145112991333,barge_in,
|
| 82 |
+
en-081,prefix,prefix-3,en,backchannel,Right,right,pending,0.013306460343301296,backchannel,
|
| 83 |
+
en-082,prefix,prefix-3,en,backchannel,Right right,right right,pending,0.01329006627202034,backchannel,train
|
| 84 |
+
en-083,prefix,prefix-3,en,barge_in,Right right but I meant the other account,right right but i meant the other account,pending,0.988115668296814,barge_in,
|
| 85 |
+
en-084,prefix,prefix-4,en,backchannel,Mm hmm,mm hmm,pending,0.013299750164151192,backchannel,validation
|
| 86 |
+
en-085,prefix,prefix-4,en,backchannel,Mm hmm yes,mm hmm yes,pending,0.013292054645717144,backchannel,
|
| 87 |
+
en-086,prefix,prefix-4,en,barge_in,Mm hmm yes can you repeat the price,mm hmm yes can you repeat the price,pending,0.9881290793418884,barge_in,
|
| 88 |
+
en-087,prefix,prefix-5,en,backchannel,I see,i see,pending,0.013295376673340797,backchannel,
|
| 89 |
+
en-088,prefix,prefix-5,en,backchannel,I see yes,i see yes,pending,0.013287578709423542,backchannel,
|
| 90 |
+
en-089,prefix,prefix-5,en,barge_in,I see yes but I need to cancel,i see yes but i need to cancel,pending,0.9879875183105469,barge_in,
|
| 91 |
+
en-090,prefix,prefix-6,en,backchannel,Sure,sure,pending,0.013307417742908001,backchannel,
|
| 92 |
+
en-091,prefix,prefix-6,en,backchannel,Sure sure,sure sure,pending,0.013290559872984886,backchannel,
|
| 93 |
+
en-092,prefix,prefix-6,en,barge_in,Sure sure stop there please,sure sure stop there please,pending,0.013294538483023643,backchannel,
|
| 94 |
+
en-093,prefix,prefix-7,en,backchannel,Got it,got it,pending,0.01331440731883049,backchannel,
|
| 95 |
+
en-094,prefix,prefix-7,en,backchannel,Got it okay,got it okay,pending,0.013302822597324848,backchannel,
|
| 96 |
+
en-095,prefix,prefix-7,en,barge_in,Got it okay now let me explain,got it okay now let me explain,pending,0.6444560885429382,barge_in,
|
| 97 |
+
en-096,prefix,prefix-8,en,backchannel,Yeah,yeah,pending,0.013312126509845257,backchannel,
|
| 98 |
+
en-097,prefix,prefix-8,en,backchannel,Yeah yeah,yeah yeah,pending,0.013376967050135136,backchannel,
|
| 99 |
+
en-098,prefix,prefix-8,en,barge_in,Yeah yeah no that will not work for me,yeah yeah no that will not work for me,pending,0.9878116250038147,barge_in,
|
reports/challenge-report.json
ADDED
|
@@ -0,0 +1,438 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"challenge_sha256": "3d46b9e024f4d3bedd69f07603cde934e85ec2272988e90d2acebc30d5e99519",
|
| 3 |
+
"model_sha256": "bddb65c08f6260cfc9d320e65b4229229ec9f3ca95204aed98ab5ae45ff758d3",
|
| 4 |
+
"threshold": 0.5,
|
| 5 |
+
"runtime": {
|
| 6 |
+
"device": "cpu",
|
| 7 |
+
"torch": "2.15.0+cpu",
|
| 8 |
+
"transformers": "5.23.0",
|
| 9 |
+
"threads": 4
|
| 10 |
+
},
|
| 11 |
+
"batch_vs_single_max_probability_difference": 8.285045623779297e-06,
|
| 12 |
+
"batch_vs_single_label_disagreements": 0,
|
| 13 |
+
"review_status": "Owner reviewed all 98 cases; 13 subsequent revisions explicitly authorized; final binary labels match authored v1 labels. Diagnostic development set, not independent representative evaluation.",
|
| 14 |
+
"all": {
|
| 15 |
+
"count": 98,
|
| 16 |
+
"accuracy": 0.7346938775510204,
|
| 17 |
+
"macro_f1": 0.7111111111111111,
|
| 18 |
+
"false_interruption_rate": 0.0,
|
| 19 |
+
"missed_interruption_rate": 0.5416666666666666,
|
| 20 |
+
"per_class": {
|
| 21 |
+
"backchannel": {
|
| 22 |
+
"precision": 0.6578947368421053,
|
| 23 |
+
"recall": 1.0,
|
| 24 |
+
"f1": 0.7936507936507937,
|
| 25 |
+
"count": 50
|
| 26 |
+
},
|
| 27 |
+
"barge_in": {
|
| 28 |
+
"precision": 1.0,
|
| 29 |
+
"recall": 0.4583333333333333,
|
| 30 |
+
"f1": 0.6285714285714286,
|
| 31 |
+
"count": 48
|
| 32 |
+
}
|
| 33 |
+
},
|
| 34 |
+
"per_language": {
|
| 35 |
+
"en": {
|
| 36 |
+
"accuracy": 0.7346938775510204,
|
| 37 |
+
"count": 98
|
| 38 |
+
}
|
| 39 |
+
}
|
| 40 |
+
},
|
| 41 |
+
"by_category": {
|
| 42 |
+
"long_backchannel": {
|
| 43 |
+
"count": 24,
|
| 44 |
+
"accuracy": 1.0,
|
| 45 |
+
"macro_f1": 0.5,
|
| 46 |
+
"false_interruption_rate": 0.0,
|
| 47 |
+
"missed_interruption_rate": 0.0,
|
| 48 |
+
"per_class": {
|
| 49 |
+
"backchannel": {
|
| 50 |
+
"precision": 1.0,
|
| 51 |
+
"recall": 1.0,
|
| 52 |
+
"f1": 1.0,
|
| 53 |
+
"count": 24
|
| 54 |
+
},
|
| 55 |
+
"barge_in": {
|
| 56 |
+
"precision": 0.0,
|
| 57 |
+
"recall": 0.0,
|
| 58 |
+
"f1": 0.0,
|
| 59 |
+
"count": 0
|
| 60 |
+
}
|
| 61 |
+
},
|
| 62 |
+
"per_language": {
|
| 63 |
+
"en": {
|
| 64 |
+
"accuracy": 1.0,
|
| 65 |
+
"count": 24
|
| 66 |
+
}
|
| 67 |
+
}
|
| 68 |
+
},
|
| 69 |
+
"ordinary_backchannel": {
|
| 70 |
+
"count": 10,
|
| 71 |
+
"accuracy": 1.0,
|
| 72 |
+
"macro_f1": 0.5,
|
| 73 |
+
"false_interruption_rate": 0.0,
|
| 74 |
+
"missed_interruption_rate": 0.0,
|
| 75 |
+
"per_class": {
|
| 76 |
+
"backchannel": {
|
| 77 |
+
"precision": 1.0,
|
| 78 |
+
"recall": 1.0,
|
| 79 |
+
"f1": 1.0,
|
| 80 |
+
"count": 10
|
| 81 |
+
},
|
| 82 |
+
"barge_in": {
|
| 83 |
+
"precision": 0.0,
|
| 84 |
+
"recall": 0.0,
|
| 85 |
+
"f1": 0.0,
|
| 86 |
+
"count": 0
|
| 87 |
+
}
|
| 88 |
+
},
|
| 89 |
+
"per_language": {
|
| 90 |
+
"en": {
|
| 91 |
+
"accuracy": 1.0,
|
| 92 |
+
"count": 10
|
| 93 |
+
}
|
| 94 |
+
}
|
| 95 |
+
},
|
| 96 |
+
"ordinary_interruption": {
|
| 97 |
+
"count": 10,
|
| 98 |
+
"accuracy": 1.0,
|
| 99 |
+
"macro_f1": 0.5,
|
| 100 |
+
"false_interruption_rate": 0.0,
|
| 101 |
+
"missed_interruption_rate": 0.0,
|
| 102 |
+
"per_class": {
|
| 103 |
+
"backchannel": {
|
| 104 |
+
"precision": 0.0,
|
| 105 |
+
"recall": 0.0,
|
| 106 |
+
"f1": 0.0,
|
| 107 |
+
"count": 0
|
| 108 |
+
},
|
| 109 |
+
"barge_in": {
|
| 110 |
+
"precision": 1.0,
|
| 111 |
+
"recall": 1.0,
|
| 112 |
+
"f1": 1.0,
|
| 113 |
+
"count": 10
|
| 114 |
+
}
|
| 115 |
+
},
|
| 116 |
+
"per_language": {
|
| 117 |
+
"en": {
|
| 118 |
+
"accuracy": 1.0,
|
| 119 |
+
"count": 10
|
| 120 |
+
}
|
| 121 |
+
}
|
| 122 |
+
},
|
| 123 |
+
"prefix": {
|
| 124 |
+
"count": 24,
|
| 125 |
+
"accuracy": 0.9583333333333334,
|
| 126 |
+
"macro_f1": 0.9515151515151515,
|
| 127 |
+
"false_interruption_rate": 0.0,
|
| 128 |
+
"missed_interruption_rate": 0.125,
|
| 129 |
+
"per_class": {
|
| 130 |
+
"backchannel": {
|
| 131 |
+
"precision": 0.9411764705882353,
|
| 132 |
+
"recall": 1.0,
|
| 133 |
+
"f1": 0.9696969696969697,
|
| 134 |
+
"count": 16
|
| 135 |
+
},
|
| 136 |
+
"barge_in": {
|
| 137 |
+
"precision": 1.0,
|
| 138 |
+
"recall": 0.875,
|
| 139 |
+
"f1": 0.9333333333333333,
|
| 140 |
+
"count": 8
|
| 141 |
+
}
|
| 142 |
+
},
|
| 143 |
+
"per_language": {
|
| 144 |
+
"en": {
|
| 145 |
+
"accuracy": 0.9583333333333334,
|
| 146 |
+
"count": 24
|
| 147 |
+
}
|
| 148 |
+
}
|
| 149 |
+
},
|
| 150 |
+
"short_interruption": {
|
| 151 |
+
"count": 30,
|
| 152 |
+
"accuracy": 0.16666666666666666,
|
| 153 |
+
"macro_f1": 0.14285714285714285,
|
| 154 |
+
"false_interruption_rate": 0.0,
|
| 155 |
+
"missed_interruption_rate": 0.8333333333333334,
|
| 156 |
+
"per_class": {
|
| 157 |
+
"backchannel": {
|
| 158 |
+
"precision": 0.0,
|
| 159 |
+
"recall": 0.0,
|
| 160 |
+
"f1": 0.0,
|
| 161 |
+
"count": 0
|
| 162 |
+
},
|
| 163 |
+
"barge_in": {
|
| 164 |
+
"precision": 1.0,
|
| 165 |
+
"recall": 0.16666666666666666,
|
| 166 |
+
"f1": 0.2857142857142857,
|
| 167 |
+
"count": 30
|
| 168 |
+
}
|
| 169 |
+
},
|
| 170 |
+
"per_language": {
|
| 171 |
+
"en": {
|
| 172 |
+
"accuracy": 0.16666666666666666,
|
| 173 |
+
"count": 30
|
| 174 |
+
}
|
| 175 |
+
}
|
| 176 |
+
}
|
| 177 |
+
},
|
| 178 |
+
"overlap_counts": {
|
| 179 |
+
"train": 3,
|
| 180 |
+
"validation": 2,
|
| 181 |
+
"test": 0
|
| 182 |
+
},
|
| 183 |
+
"cross_split_exact_overlap": {
|
| 184 |
+
"train/validation": 0,
|
| 185 |
+
"train/test": 0,
|
| 186 |
+
"validation/test": 0
|
| 187 |
+
},
|
| 188 |
+
"unseen": {
|
| 189 |
+
"count": 93,
|
| 190 |
+
"accuracy": 0.7204301075268817,
|
| 191 |
+
"macro_f1": 0.702216748768473,
|
| 192 |
+
"false_interruption_rate": 0.0,
|
| 193 |
+
"missed_interruption_rate": 0.5416666666666666,
|
| 194 |
+
"per_class": {
|
| 195 |
+
"backchannel": {
|
| 196 |
+
"precision": 0.6338028169014085,
|
| 197 |
+
"recall": 1.0,
|
| 198 |
+
"f1": 0.7758620689655172,
|
| 199 |
+
"count": 45
|
| 200 |
+
},
|
| 201 |
+
"barge_in": {
|
| 202 |
+
"precision": 1.0,
|
| 203 |
+
"recall": 0.4583333333333333,
|
| 204 |
+
"f1": 0.6285714285714286,
|
| 205 |
+
"count": 48
|
| 206 |
+
}
|
| 207 |
+
},
|
| 208 |
+
"per_language": {
|
| 209 |
+
"en": {
|
| 210 |
+
"accuracy": 0.7204301075268817,
|
| 211 |
+
"count": 93
|
| 212 |
+
}
|
| 213 |
+
}
|
| 214 |
+
},
|
| 215 |
+
"prefix_groups": {
|
| 216 |
+
"prefix-1": [
|
| 217 |
+
{
|
| 218 |
+
"text": "yes",
|
| 219 |
+
"label": "backchannel",
|
| 220 |
+
"p_barge_in": 0.013361463323235512
|
| 221 |
+
},
|
| 222 |
+
{
|
| 223 |
+
"text": "yes yes",
|
| 224 |
+
"label": "backchannel",
|
| 225 |
+
"p_barge_in": 0.013298917561769485
|
| 226 |
+
},
|
| 227 |
+
{
|
| 228 |
+
"text": "yes yes but change the time",
|
| 229 |
+
"label": "barge_in",
|
| 230 |
+
"p_barge_in": 0.9880699515342712
|
| 231 |
+
}
|
| 232 |
+
],
|
| 233 |
+
"prefix-2": [
|
| 234 |
+
{
|
| 235 |
+
"text": "okay",
|
| 236 |
+
"label": "backchannel",
|
| 237 |
+
"p_barge_in": 0.013304519467055798
|
| 238 |
+
},
|
| 239 |
+
{
|
| 240 |
+
"text": "okay okay",
|
| 241 |
+
"label": "backchannel",
|
| 242 |
+
"p_barge_in": 0.01330505684018135
|
| 243 |
+
},
|
| 244 |
+
{
|
| 245 |
+
"text": "okay okay wait that is the wrong address",
|
| 246 |
+
"label": "barge_in",
|
| 247 |
+
"p_barge_in": 0.988145112991333
|
| 248 |
+
}
|
| 249 |
+
],
|
| 250 |
+
"prefix-3": [
|
| 251 |
+
{
|
| 252 |
+
"text": "right",
|
| 253 |
+
"label": "backchannel",
|
| 254 |
+
"p_barge_in": 0.013306460343301296
|
| 255 |
+
},
|
| 256 |
+
{
|
| 257 |
+
"text": "right right",
|
| 258 |
+
"label": "backchannel",
|
| 259 |
+
"p_barge_in": 0.01329006627202034
|
| 260 |
+
},
|
| 261 |
+
{
|
| 262 |
+
"text": "right right but i meant the other account",
|
| 263 |
+
"label": "barge_in",
|
| 264 |
+
"p_barge_in": 0.988115668296814
|
| 265 |
+
}
|
| 266 |
+
],
|
| 267 |
+
"prefix-4": [
|
| 268 |
+
{
|
| 269 |
+
"text": "mm hmm",
|
| 270 |
+
"label": "backchannel",
|
| 271 |
+
"p_barge_in": 0.013299750164151192
|
| 272 |
+
},
|
| 273 |
+
{
|
| 274 |
+
"text": "mm hmm yes",
|
| 275 |
+
"label": "backchannel",
|
| 276 |
+
"p_barge_in": 0.013292054645717144
|
| 277 |
+
},
|
| 278 |
+
{
|
| 279 |
+
"text": "mm hmm yes can you repeat the price",
|
| 280 |
+
"label": "barge_in",
|
| 281 |
+
"p_barge_in": 0.9881290793418884
|
| 282 |
+
}
|
| 283 |
+
],
|
| 284 |
+
"prefix-5": [
|
| 285 |
+
{
|
| 286 |
+
"text": "i see",
|
| 287 |
+
"label": "backchannel",
|
| 288 |
+
"p_barge_in": 0.013295376673340797
|
| 289 |
+
},
|
| 290 |
+
{
|
| 291 |
+
"text": "i see yes",
|
| 292 |
+
"label": "backchannel",
|
| 293 |
+
"p_barge_in": 0.013287578709423542
|
| 294 |
+
},
|
| 295 |
+
{
|
| 296 |
+
"text": "i see yes but i need to cancel",
|
| 297 |
+
"label": "barge_in",
|
| 298 |
+
"p_barge_in": 0.9879875183105469
|
| 299 |
+
}
|
| 300 |
+
],
|
| 301 |
+
"prefix-6": [
|
| 302 |
+
{
|
| 303 |
+
"text": "sure",
|
| 304 |
+
"label": "backchannel",
|
| 305 |
+
"p_barge_in": 0.013307417742908001
|
| 306 |
+
},
|
| 307 |
+
{
|
| 308 |
+
"text": "sure sure",
|
| 309 |
+
"label": "backchannel",
|
| 310 |
+
"p_barge_in": 0.013290559872984886
|
| 311 |
+
},
|
| 312 |
+
{
|
| 313 |
+
"text": "sure sure stop there please",
|
| 314 |
+
"label": "barge_in",
|
| 315 |
+
"p_barge_in": 0.013294538483023643
|
| 316 |
+
}
|
| 317 |
+
],
|
| 318 |
+
"prefix-7": [
|
| 319 |
+
{
|
| 320 |
+
"text": "got it",
|
| 321 |
+
"label": "backchannel",
|
| 322 |
+
"p_barge_in": 0.01331440731883049
|
| 323 |
+
},
|
| 324 |
+
{
|
| 325 |
+
"text": "got it okay",
|
| 326 |
+
"label": "backchannel",
|
| 327 |
+
"p_barge_in": 0.013302822597324848
|
| 328 |
+
},
|
| 329 |
+
{
|
| 330 |
+
"text": "got it okay now let me explain",
|
| 331 |
+
"label": "barge_in",
|
| 332 |
+
"p_barge_in": 0.6444560885429382
|
| 333 |
+
}
|
| 334 |
+
],
|
| 335 |
+
"prefix-8": [
|
| 336 |
+
{
|
| 337 |
+
"text": "yeah",
|
| 338 |
+
"label": "backchannel",
|
| 339 |
+
"p_barge_in": 0.013312126509845257
|
| 340 |
+
},
|
| 341 |
+
{
|
| 342 |
+
"text": "yeah yeah",
|
| 343 |
+
"label": "backchannel",
|
| 344 |
+
"p_barge_in": 0.013376967050135136
|
| 345 |
+
},
|
| 346 |
+
{
|
| 347 |
+
"text": "yeah yeah no that will not work for me",
|
| 348 |
+
"label": "barge_in",
|
| 349 |
+
"p_barge_in": 0.9878116250038147
|
| 350 |
+
}
|
| 351 |
+
]
|
| 352 |
+
},
|
| 353 |
+
"prefix_groups_all_labels_correct": 7,
|
| 354 |
+
"english_length_baseline": {
|
| 355 |
+
"cutoff_words": 6,
|
| 356 |
+
"validation": {
|
| 357 |
+
"count": 81,
|
| 358 |
+
"accuracy": 0.9753086419753086,
|
| 359 |
+
"macro_f1": 0.9716783216783216,
|
| 360 |
+
"false_interruption_rate": 0.07407407407407407,
|
| 361 |
+
"missed_interruption_rate": 0.0,
|
| 362 |
+
"per_class": {
|
| 363 |
+
"backchannel": {
|
| 364 |
+
"precision": 1.0,
|
| 365 |
+
"recall": 0.9259259259259259,
|
| 366 |
+
"f1": 0.9615384615384615,
|
| 367 |
+
"count": 27
|
| 368 |
+
},
|
| 369 |
+
"barge_in": {
|
| 370 |
+
"precision": 0.9642857142857143,
|
| 371 |
+
"recall": 1.0,
|
| 372 |
+
"f1": 0.9818181818181818,
|
| 373 |
+
"count": 54
|
| 374 |
+
}
|
| 375 |
+
},
|
| 376 |
+
"per_language": {
|
| 377 |
+
"en": {
|
| 378 |
+
"accuracy": 0.9753086419753086,
|
| 379 |
+
"count": 81
|
| 380 |
+
}
|
| 381 |
+
}
|
| 382 |
+
},
|
| 383 |
+
"test": {
|
| 384 |
+
"count": 81,
|
| 385 |
+
"accuracy": 0.9382716049382716,
|
| 386 |
+
"macro_f1": 0.9268556980314249,
|
| 387 |
+
"false_interruption_rate": 0.18518518518518517,
|
| 388 |
+
"missed_interruption_rate": 0.0,
|
| 389 |
+
"per_class": {
|
| 390 |
+
"backchannel": {
|
| 391 |
+
"precision": 1.0,
|
| 392 |
+
"recall": 0.8148148148148148,
|
| 393 |
+
"f1": 0.8979591836734693,
|
| 394 |
+
"count": 27
|
| 395 |
+
},
|
| 396 |
+
"barge_in": {
|
| 397 |
+
"precision": 0.9152542372881356,
|
| 398 |
+
"recall": 1.0,
|
| 399 |
+
"f1": 0.9557522123893806,
|
| 400 |
+
"count": 54
|
| 401 |
+
}
|
| 402 |
+
},
|
| 403 |
+
"per_language": {
|
| 404 |
+
"en": {
|
| 405 |
+
"accuracy": 0.9382716049382716,
|
| 406 |
+
"count": 81
|
| 407 |
+
}
|
| 408 |
+
}
|
| 409 |
+
},
|
| 410 |
+
"challenge": {
|
| 411 |
+
"count": 98,
|
| 412 |
+
"accuracy": 0.5510204081632653,
|
| 413 |
+
"macro_f1": 0.5315080399826162,
|
| 414 |
+
"false_interruption_rate": 0.26,
|
| 415 |
+
"missed_interruption_rate": 0.6458333333333334,
|
| 416 |
+
"per_class": {
|
| 417 |
+
"backchannel": {
|
| 418 |
+
"precision": 0.5441176470588235,
|
| 419 |
+
"recall": 0.74,
|
| 420 |
+
"f1": 0.6271186440677965,
|
| 421 |
+
"count": 50
|
| 422 |
+
},
|
| 423 |
+
"barge_in": {
|
| 424 |
+
"precision": 0.5666666666666667,
|
| 425 |
+
"recall": 0.3541666666666667,
|
| 426 |
+
"f1": 0.4358974358974359,
|
| 427 |
+
"count": 48
|
| 428 |
+
}
|
| 429 |
+
},
|
| 430 |
+
"per_language": {
|
| 431 |
+
"en": {
|
| 432 |
+
"accuracy": 0.5510204081632653,
|
| 433 |
+
"count": 98
|
| 434 |
+
}
|
| 435 |
+
}
|
| 436 |
+
}
|
| 437 |
+
}
|
| 438 |
+
}
|
reports/cpu-benchmark.json
ADDED
|
@@ -0,0 +1,94 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model": "models/e5-small-run1",
|
| 3 |
+
"runtime": "PyTorch eager fp32, CPU only",
|
| 4 |
+
"model_sha256": "bddb65c08f6260cfc9d320e65b4229229ec9f3ca95204aed98ab5ae45ff758d3",
|
| 5 |
+
"python": "3.14.4",
|
| 6 |
+
"torch": "2.15.0+cpu",
|
| 7 |
+
"transformers": "5.23.0",
|
| 8 |
+
"platform": "Linux-7.0.0-43-generic-x86_64-with-glibc2.43",
|
| 9 |
+
"cpuinfo": "AMD EPYC-Genoa Processor",
|
| 10 |
+
"load_seconds_excluded": 2.8975897090276703,
|
| 11 |
+
"iterations": 1000,
|
| 12 |
+
"warmup_per_scenario": 100,
|
| 13 |
+
"batch_size": 1,
|
| 14 |
+
"max_length": 64,
|
| 15 |
+
"interop_threads": 1,
|
| 16 |
+
"parameter_bytes": 470618120,
|
| 17 |
+
"corpus_sha256": "8b630782362b05249b3be94b0213850258422be5ddeb62fcf703817b0bf9e11a",
|
| 18 |
+
"corpus_rows": 331,
|
| 19 |
+
"languages": [
|
| 20 |
+
"de",
|
| 21 |
+
"en",
|
| 22 |
+
"es",
|
| 23 |
+
"fil",
|
| 24 |
+
"fr",
|
| 25 |
+
"hi",
|
| 26 |
+
"id",
|
| 27 |
+
"it",
|
| 28 |
+
"ja",
|
| 29 |
+
"kn",
|
| 30 |
+
"ml",
|
| 31 |
+
"ta",
|
| 32 |
+
"te",
|
| 33 |
+
"zh"
|
| 34 |
+
],
|
| 35 |
+
"measurements": [
|
| 36 |
+
{
|
| 37 |
+
"threads": 4,
|
| 38 |
+
"scenario": "multilingual_test",
|
| 39 |
+
"model_path": {
|
| 40 |
+
"p50_ms": 13.088757,
|
| 41 |
+
"p95_ms": 19.419374299999998,
|
| 42 |
+
"p99_ms": 25.316384159999995,
|
| 43 |
+
"mean_ms": 13.625100593,
|
| 44 |
+
"max_ms": 83.599
|
| 45 |
+
},
|
| 46 |
+
"tokenization": {
|
| 47 |
+
"p50_ms": 0.222889,
|
| 48 |
+
"p95_ms": 0.30127505,
|
| 49 |
+
"p99_ms": 0.45513418999999933,
|
| 50 |
+
"mean_ms": 0.23647671,
|
| 51 |
+
"max_ms": 2.407625
|
| 52 |
+
},
|
| 53 |
+
"forward_and_probability": {
|
| 54 |
+
"p50_ms": 12.8741705,
|
| 55 |
+
"p95_ms": 19.1815334,
|
| 56 |
+
"p99_ms": 25.09679618,
|
| 57 |
+
"mean_ms": 13.388623883,
|
| 58 |
+
"max_ms": 83.391099
|
| 59 |
+
}
|
| 60 |
+
},
|
| 61 |
+
{
|
| 62 |
+
"threads": 4,
|
| 63 |
+
"scenario": "full_64_tokens",
|
| 64 |
+
"model_path": {
|
| 65 |
+
"p50_ms": 26.794403000000003,
|
| 66 |
+
"p95_ms": 33.269715649999995,
|
| 67 |
+
"p99_ms": 37.862168669999996,
|
| 68 |
+
"mean_ms": 27.779209678,
|
| 69 |
+
"max_ms": 93.899777
|
| 70 |
+
},
|
| 71 |
+
"tokenization": {
|
| 72 |
+
"p50_ms": 0.6677489999999999,
|
| 73 |
+
"p95_ms": 0.8626962999999997,
|
| 74 |
+
"p99_ms": 1.7089376799999993,
|
| 75 |
+
"mean_ms": 0.704945851,
|
| 76 |
+
"max_ms": 7.957467
|
| 77 |
+
},
|
| 78 |
+
"forward_and_probability": {
|
| 79 |
+
"p50_ms": 26.103834499999998,
|
| 80 |
+
"p95_ms": 32.575319,
|
| 81 |
+
"p99_ms": 37.10665824,
|
| 82 |
+
"mean_ms": 27.074263827,
|
| 83 |
+
"max_ms": 93.297535
|
| 84 |
+
}
|
| 85 |
+
}
|
| 86 |
+
],
|
| 87 |
+
"scope": "Local virtual CPU; reference CPU unspecified. Excludes Vxpipe boundary and queueing; no whole-path acceptance claim.",
|
| 88 |
+
"corpus_token_lengths": {
|
| 89 |
+
"p50": 13.0,
|
| 90 |
+
"p95": 19.0,
|
| 91 |
+
"max": 26
|
| 92 |
+
},
|
| 93 |
+
"peak_process_rss_mib": 1163.6171875
|
| 94 |
+
}
|
reports/cpu-cold-benchmark.json
ADDED
|
@@ -0,0 +1,33 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"cold_process_seconds": [
|
| 3 |
+
4.768154836,
|
| 4 |
+
4.810785561,
|
| 5 |
+
4.136632718,
|
| 6 |
+
4.255589102,
|
| 7 |
+
3.847660745,
|
| 8 |
+
4.045793715,
|
| 9 |
+
3.973606716,
|
| 10 |
+
4.190341404,
|
| 11 |
+
4.387973674,
|
| 12 |
+
4.28136222
|
| 13 |
+
],
|
| 14 |
+
"loaded_to_first_label_ms": [
|
| 15 |
+
26.506046,
|
| 16 |
+
40.193806,
|
| 17 |
+
18.425675,
|
| 18 |
+
22.50301,
|
| 19 |
+
23.081508,
|
| 20 |
+
40.155708,
|
| 21 |
+
16.969254,
|
| 22 |
+
23.919822,
|
| 23 |
+
23.060476,
|
| 24 |
+
24.334184
|
| 25 |
+
],
|
| 26 |
+
"loaded_to_first_label_median_ms": 23.500664999999998,
|
| 27 |
+
"median_seconds": 4.222965253,
|
| 28 |
+
"min_seconds": 3.847660745,
|
| 29 |
+
"max_seconds": 4.810785561,
|
| 30 |
+
"runs": 10,
|
| 31 |
+
"threads": 4,
|
| 32 |
+
"scope": "Python process creation through reading first flushed label. Includes imports, tokenizer/model load, first inference. Local files only; OS file cache may be warm. Excludes process shutdown, downloads and HTTP/network boundary."
|
| 33 |
+
}
|
reports/cpu-thread-sweep.json
ADDED
|
@@ -0,0 +1,194 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model": "models/e5-small-run1",
|
| 3 |
+
"runtime": "PyTorch eager fp32, CPU only",
|
| 4 |
+
"model_sha256": "bddb65c08f6260cfc9d320e65b4229229ec9f3ca95204aed98ab5ae45ff758d3",
|
| 5 |
+
"python": "3.14.4",
|
| 6 |
+
"torch": "2.15.0+cpu",
|
| 7 |
+
"transformers": "5.23.0",
|
| 8 |
+
"platform": "Linux-7.0.0-43-generic-x86_64-with-glibc2.43",
|
| 9 |
+
"cpuinfo": "AMD EPYC-Genoa Processor",
|
| 10 |
+
"load_seconds_excluded": 3.1383294460829347,
|
| 11 |
+
"iterations": 1000,
|
| 12 |
+
"warmup_per_scenario": 100,
|
| 13 |
+
"batch_size": 1,
|
| 14 |
+
"max_length": 64,
|
| 15 |
+
"interop_threads": 1,
|
| 16 |
+
"parameter_bytes": 470618120,
|
| 17 |
+
"corpus_sha256": "8b630782362b05249b3be94b0213850258422be5ddeb62fcf703817b0bf9e11a",
|
| 18 |
+
"corpus_rows": 331,
|
| 19 |
+
"languages": [
|
| 20 |
+
"de",
|
| 21 |
+
"en",
|
| 22 |
+
"es",
|
| 23 |
+
"fil",
|
| 24 |
+
"fr",
|
| 25 |
+
"hi",
|
| 26 |
+
"id",
|
| 27 |
+
"it",
|
| 28 |
+
"ja",
|
| 29 |
+
"kn",
|
| 30 |
+
"ml",
|
| 31 |
+
"ta",
|
| 32 |
+
"te",
|
| 33 |
+
"zh"
|
| 34 |
+
],
|
| 35 |
+
"corpus_token_lengths": {
|
| 36 |
+
"p50": 13.0,
|
| 37 |
+
"p95": 19.0,
|
| 38 |
+
"max": 26
|
| 39 |
+
},
|
| 40 |
+
"measurements": [
|
| 41 |
+
{
|
| 42 |
+
"threads": 1,
|
| 43 |
+
"scenario": "multilingual_test",
|
| 44 |
+
"model_path": {
|
| 45 |
+
"p50_ms": 20.395149500000002,
|
| 46 |
+
"p95_ms": 25.477123599999995,
|
| 47 |
+
"p99_ms": 28.2604041,
|
| 48 |
+
"mean_ms": 20.432651312,
|
| 49 |
+
"max_ms": 46.969913
|
| 50 |
+
},
|
| 51 |
+
"tokenization": {
|
| 52 |
+
"p50_ms": 0.23074050000000002,
|
| 53 |
+
"p95_ms": 0.29089349999999997,
|
| 54 |
+
"p99_ms": 0.33883417,
|
| 55 |
+
"mean_ms": 0.234400798,
|
| 56 |
+
"max_ms": 0.640549
|
| 57 |
+
},
|
| 58 |
+
"forward_and_probability": {
|
| 59 |
+
"p50_ms": 20.162170500000002,
|
| 60 |
+
"p95_ms": 25.207171749999997,
|
| 61 |
+
"p99_ms": 28.02515483,
|
| 62 |
+
"mean_ms": 20.198250514,
|
| 63 |
+
"max_ms": 46.709714
|
| 64 |
+
}
|
| 65 |
+
},
|
| 66 |
+
{
|
| 67 |
+
"threads": 1,
|
| 68 |
+
"scenario": "full_64_tokens",
|
| 69 |
+
"model_path": {
|
| 70 |
+
"p50_ms": 44.6955575,
|
| 71 |
+
"p95_ms": 50.545173899999995,
|
| 72 |
+
"p99_ms": 53.76246217,
|
| 73 |
+
"mean_ms": 45.379929543,
|
| 74 |
+
"max_ms": 57.243489
|
| 75 |
+
},
|
| 76 |
+
"tokenization": {
|
| 77 |
+
"p50_ms": 0.6161775,
|
| 78 |
+
"p95_ms": 0.7613426,
|
| 79 |
+
"p99_ms": 0.84228681,
|
| 80 |
+
"mean_ms": 0.631765294,
|
| 81 |
+
"max_ms": 0.891216
|
| 82 |
+
},
|
| 83 |
+
"forward_and_probability": {
|
| 84 |
+
"p50_ms": 44.084312499999996,
|
| 85 |
+
"p95_ms": 49.9083266,
|
| 86 |
+
"p99_ms": 53.07674325,
|
| 87 |
+
"mean_ms": 44.748164249,
|
| 88 |
+
"max_ms": 56.405244
|
| 89 |
+
}
|
| 90 |
+
},
|
| 91 |
+
{
|
| 92 |
+
"threads": 2,
|
| 93 |
+
"scenario": "multilingual_test",
|
| 94 |
+
"model_path": {
|
| 95 |
+
"p50_ms": 19.406584000000002,
|
| 96 |
+
"p95_ms": 23.1596329,
|
| 97 |
+
"p99_ms": 25.6503671,
|
| 98 |
+
"mean_ms": 19.536068109,
|
| 99 |
+
"max_ms": 28.630568
|
| 100 |
+
},
|
| 101 |
+
"tokenization": {
|
| 102 |
+
"p50_ms": 0.23920799999999998,
|
| 103 |
+
"p95_ms": 0.31088855,
|
| 104 |
+
"p99_ms": 0.36521733999999995,
|
| 105 |
+
"mean_ms": 0.244807045,
|
| 106 |
+
"max_ms": 0.475011
|
| 107 |
+
},
|
| 108 |
+
"forward_and_probability": {
|
| 109 |
+
"p50_ms": 19.154210499999998,
|
| 110 |
+
"p95_ms": 22.828605549999995,
|
| 111 |
+
"p99_ms": 25.33145532,
|
| 112 |
+
"mean_ms": 19.291261064,
|
| 113 |
+
"max_ms": 28.31147
|
| 114 |
+
}
|
| 115 |
+
},
|
| 116 |
+
{
|
| 117 |
+
"threads": 2,
|
| 118 |
+
"scenario": "full_64_tokens",
|
| 119 |
+
"model_path": {
|
| 120 |
+
"p50_ms": 38.023134999999996,
|
| 121 |
+
"p95_ms": 45.51507395,
|
| 122 |
+
"p99_ms": 53.98291254,
|
| 123 |
+
"mean_ms": 38.847798787,
|
| 124 |
+
"max_ms": 62.944328
|
| 125 |
+
},
|
| 126 |
+
"tokenization": {
|
| 127 |
+
"p50_ms": 0.636223,
|
| 128 |
+
"p95_ms": 0.8138672499999998,
|
| 129 |
+
"p99_ms": 0.9455087999999999,
|
| 130 |
+
"mean_ms": 0.659222932,
|
| 131 |
+
"max_ms": 1.44213
|
| 132 |
+
},
|
| 133 |
+
"forward_and_probability": {
|
| 134 |
+
"p50_ms": 37.383997,
|
| 135 |
+
"p95_ms": 44.81712879999999,
|
| 136 |
+
"p99_ms": 53.05854066,
|
| 137 |
+
"mean_ms": 38.188575855,
|
| 138 |
+
"max_ms": 61.995607
|
| 139 |
+
}
|
| 140 |
+
},
|
| 141 |
+
{
|
| 142 |
+
"threads": 4,
|
| 143 |
+
"scenario": "multilingual_test",
|
| 144 |
+
"model_path": {
|
| 145 |
+
"p50_ms": 13.1820115,
|
| 146 |
+
"p95_ms": 19.21180095,
|
| 147 |
+
"p99_ms": 24.81463212999999,
|
| 148 |
+
"mean_ms": 13.731945525,
|
| 149 |
+
"max_ms": 89.107149
|
| 150 |
+
},
|
| 151 |
+
"tokenization": {
|
| 152 |
+
"p50_ms": 0.2276755,
|
| 153 |
+
"p95_ms": 0.30882,
|
| 154 |
+
"p99_ms": 0.3798859499999996,
|
| 155 |
+
"mean_ms": 0.23842783,
|
| 156 |
+
"max_ms": 4.009645
|
| 157 |
+
},
|
| 158 |
+
"forward_and_probability": {
|
| 159 |
+
"p50_ms": 12.937595,
|
| 160 |
+
"p95_ms": 18.841387699999995,
|
| 161 |
+
"p99_ms": 24.56872725999999,
|
| 162 |
+
"mean_ms": 13.493517695,
|
| 163 |
+
"max_ms": 88.911056
|
| 164 |
+
}
|
| 165 |
+
},
|
| 166 |
+
{
|
| 167 |
+
"threads": 4,
|
| 168 |
+
"scenario": "full_64_tokens",
|
| 169 |
+
"model_path": {
|
| 170 |
+
"p50_ms": 26.392652,
|
| 171 |
+
"p95_ms": 32.489633899999994,
|
| 172 |
+
"p99_ms": 41.04232582999999,
|
| 173 |
+
"mean_ms": 27.24307507,
|
| 174 |
+
"max_ms": 93.673078
|
| 175 |
+
},
|
| 176 |
+
"tokenization": {
|
| 177 |
+
"p50_ms": 0.614555,
|
| 178 |
+
"p95_ms": 0.7339711499999999,
|
| 179 |
+
"p99_ms": 0.9166563899999999,
|
| 180 |
+
"mean_ms": 0.624646764,
|
| 181 |
+
"max_ms": 2.092334
|
| 182 |
+
},
|
| 183 |
+
"forward_and_probability": {
|
| 184 |
+
"p50_ms": 25.7561225,
|
| 185 |
+
"p95_ms": 31.824941300000003,
|
| 186 |
+
"p99_ms": 40.46172110999999,
|
| 187 |
+
"mean_ms": 26.618428306,
|
| 188 |
+
"max_ms": 93.090937
|
| 189 |
+
}
|
| 190 |
+
}
|
| 191 |
+
],
|
| 192 |
+
"scope": "Local virtual CPU; reference CPU unspecified. Excludes Vxpipe boundary and queueing; no whole-path acceptance claim.",
|
| 193 |
+
"peak_process_rss_mib": 1168.8125
|
| 194 |
+
}
|
reports/eval-test.json
ADDED
|
@@ -0,0 +1,81 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"count": 331,
|
| 3 |
+
"accuracy": 1.0,
|
| 4 |
+
"macro_f1": 1.0,
|
| 5 |
+
"false_interruption_rate": 0.0,
|
| 6 |
+
"missed_interruption_rate": 0.0,
|
| 7 |
+
"per_class": {
|
| 8 |
+
"backchannel": {
|
| 9 |
+
"precision": 1.0,
|
| 10 |
+
"recall": 1.0,
|
| 11 |
+
"f1": 1.0,
|
| 12 |
+
"count": 109
|
| 13 |
+
},
|
| 14 |
+
"barge_in": {
|
| 15 |
+
"precision": 1.0,
|
| 16 |
+
"recall": 1.0,
|
| 17 |
+
"f1": 1.0,
|
| 18 |
+
"count": 222
|
| 19 |
+
}
|
| 20 |
+
},
|
| 21 |
+
"per_language": {
|
| 22 |
+
"de": {
|
| 23 |
+
"accuracy": 1.0,
|
| 24 |
+
"count": 17
|
| 25 |
+
},
|
| 26 |
+
"en": {
|
| 27 |
+
"accuracy": 1.0,
|
| 28 |
+
"count": 81
|
| 29 |
+
},
|
| 30 |
+
"es": {
|
| 31 |
+
"accuracy": 1.0,
|
| 32 |
+
"count": 36
|
| 33 |
+
},
|
| 34 |
+
"fil": {
|
| 35 |
+
"accuracy": 1.0,
|
| 36 |
+
"count": 18
|
| 37 |
+
},
|
| 38 |
+
"fr": {
|
| 39 |
+
"accuracy": 1.0,
|
| 40 |
+
"count": 18
|
| 41 |
+
},
|
| 42 |
+
"hi": {
|
| 43 |
+
"accuracy": 1.0,
|
| 44 |
+
"count": 18
|
| 45 |
+
},
|
| 46 |
+
"id": {
|
| 47 |
+
"accuracy": 1.0,
|
| 48 |
+
"count": 17
|
| 49 |
+
},
|
| 50 |
+
"it": {
|
| 51 |
+
"accuracy": 1.0,
|
| 52 |
+
"count": 18
|
| 53 |
+
},
|
| 54 |
+
"ja": {
|
| 55 |
+
"accuracy": 1.0,
|
| 56 |
+
"count": 18
|
| 57 |
+
},
|
| 58 |
+
"kn": {
|
| 59 |
+
"accuracy": 1.0,
|
| 60 |
+
"count": 18
|
| 61 |
+
},
|
| 62 |
+
"ml": {
|
| 63 |
+
"accuracy": 1.0,
|
| 64 |
+
"count": 18
|
| 65 |
+
},
|
| 66 |
+
"ta": {
|
| 67 |
+
"accuracy": 1.0,
|
| 68 |
+
"count": 18
|
| 69 |
+
},
|
| 70 |
+
"te": {
|
| 71 |
+
"accuracy": 1.0,
|
| 72 |
+
"count": 18
|
| 73 |
+
},
|
| 74 |
+
"zh": {
|
| 75 |
+
"accuracy": 1.0,
|
| 76 |
+
"count": 18
|
| 77 |
+
}
|
| 78 |
+
},
|
| 79 |
+
"split": "test",
|
| 80 |
+
"threshold": 0.5
|
| 81 |
+
}
|
reports/eval-validation.json
ADDED
|
@@ -0,0 +1,81 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"count": 333,
|
| 3 |
+
"accuracy": 0.993993993993994,
|
| 4 |
+
"macro_f1": 0.9932432432432432,
|
| 5 |
+
"false_interruption_rate": 0.009009009009009009,
|
| 6 |
+
"missed_interruption_rate": 0.0045045045045045045,
|
| 7 |
+
"per_class": {
|
| 8 |
+
"backchannel": {
|
| 9 |
+
"precision": 0.990990990990991,
|
| 10 |
+
"recall": 0.990990990990991,
|
| 11 |
+
"f1": 0.990990990990991,
|
| 12 |
+
"count": 111
|
| 13 |
+
},
|
| 14 |
+
"barge_in": {
|
| 15 |
+
"precision": 0.9954954954954955,
|
| 16 |
+
"recall": 0.9954954954954955,
|
| 17 |
+
"f1": 0.9954954954954955,
|
| 18 |
+
"count": 222
|
| 19 |
+
}
|
| 20 |
+
},
|
| 21 |
+
"per_language": {
|
| 22 |
+
"de": {
|
| 23 |
+
"accuracy": 1.0,
|
| 24 |
+
"count": 18
|
| 25 |
+
},
|
| 26 |
+
"en": {
|
| 27 |
+
"accuracy": 1.0,
|
| 28 |
+
"count": 81
|
| 29 |
+
},
|
| 30 |
+
"es": {
|
| 31 |
+
"accuracy": 1.0,
|
| 32 |
+
"count": 36
|
| 33 |
+
},
|
| 34 |
+
"fil": {
|
| 35 |
+
"accuracy": 1.0,
|
| 36 |
+
"count": 18
|
| 37 |
+
},
|
| 38 |
+
"fr": {
|
| 39 |
+
"accuracy": 1.0,
|
| 40 |
+
"count": 18
|
| 41 |
+
},
|
| 42 |
+
"hi": {
|
| 43 |
+
"accuracy": 1.0,
|
| 44 |
+
"count": 18
|
| 45 |
+
},
|
| 46 |
+
"id": {
|
| 47 |
+
"accuracy": 1.0,
|
| 48 |
+
"count": 18
|
| 49 |
+
},
|
| 50 |
+
"it": {
|
| 51 |
+
"accuracy": 1.0,
|
| 52 |
+
"count": 18
|
| 53 |
+
},
|
| 54 |
+
"ja": {
|
| 55 |
+
"accuracy": 1.0,
|
| 56 |
+
"count": 18
|
| 57 |
+
},
|
| 58 |
+
"kn": {
|
| 59 |
+
"accuracy": 1.0,
|
| 60 |
+
"count": 18
|
| 61 |
+
},
|
| 62 |
+
"ml": {
|
| 63 |
+
"accuracy": 1.0,
|
| 64 |
+
"count": 18
|
| 65 |
+
},
|
| 66 |
+
"ta": {
|
| 67 |
+
"accuracy": 0.9444444444444444,
|
| 68 |
+
"count": 18
|
| 69 |
+
},
|
| 70 |
+
"te": {
|
| 71 |
+
"accuracy": 1.0,
|
| 72 |
+
"count": 18
|
| 73 |
+
},
|
| 74 |
+
"zh": {
|
| 75 |
+
"accuracy": 0.9444444444444444,
|
| 76 |
+
"count": 18
|
| 77 |
+
}
|
| 78 |
+
},
|
| 79 |
+
"split": "validation",
|
| 80 |
+
"threshold": 0.5
|
| 81 |
+
}
|
tests/test_data.py
ADDED
|
@@ -0,0 +1,43 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import sys
|
| 2 |
+
from types import SimpleNamespace
|
| 3 |
+
|
| 4 |
+
import pytest
|
| 5 |
+
|
| 6 |
+
from barge_in_detector_model.data import read_csv, resolve_splits
|
| 7 |
+
|
| 8 |
+
|
| 9 |
+
def make_splits(path):
|
| 10 |
+
path.mkdir(parents=True)
|
| 11 |
+
for name in ["train", "validation", "test"]:
|
| 12 |
+
(path / f"{name}.csv").write_text("text_clean,label,language\nstop,barge_in,en\n")
|
| 13 |
+
|
| 14 |
+
|
| 15 |
+
def test_local_splits_do_not_require_hub(tmp_path):
|
| 16 |
+
make_splits(tmp_path / "splits")
|
| 17 |
+
directory, revision = resolve_splits(tmp_path / "splits")
|
| 18 |
+
assert revision is None
|
| 19 |
+
assert read_csv(directory / "test.csv")[0]["label"] == "barge_in"
|
| 20 |
+
with pytest.raises(ValueError, match="missing"):
|
| 21 |
+
resolve_splits(tmp_path / "absent")
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
def test_hub_download_uses_resolved_immutable_revision(tmp_path, monkeypatch):
|
| 25 |
+
make_splits(tmp_path / "data/splits")
|
| 26 |
+
seen = {}
|
| 27 |
+
|
| 28 |
+
def download(**kwargs):
|
| 29 |
+
seen.update(kwargs)
|
| 30 |
+
return str(tmp_path)
|
| 31 |
+
|
| 32 |
+
fake = SimpleNamespace(
|
| 33 |
+
HfApi=lambda: SimpleNamespace(
|
| 34 |
+
dataset_info=lambda *a, **k: SimpleNamespace(sha="fixed-commit")
|
| 35 |
+
),
|
| 36 |
+
snapshot_download=download,
|
| 37 |
+
)
|
| 38 |
+
monkeypatch.setitem(sys.modules, "huggingface_hub", fake)
|
| 39 |
+
directory, revision = resolve_splits(None, repo_id="owner/data", revision="v1")
|
| 40 |
+
assert revision == "fixed-commit"
|
| 41 |
+
assert seen["revision"] == "fixed-commit"
|
| 42 |
+
assert seen["repo_type"] == "dataset"
|
| 43 |
+
assert directory == tmp_path / "data/splits"
|
tests/test_metrics.py
ADDED
|
@@ -0,0 +1,29 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Evaluation metrics for the two labels."""
|
| 2 |
+
|
| 3 |
+
import pytest
|
| 4 |
+
|
| 5 |
+
from barge_in_detector_model.metrics import compute
|
| 6 |
+
|
| 7 |
+
|
| 8 |
+
def test_metrics_report_false_and_missed_interruptions():
|
| 9 |
+
labels = ["backchannel", "backchannel", "barge_in", "barge_in", "barge_in"]
|
| 10 |
+
preds = ["backchannel", "barge_in", "barge_in", "barge_in", "backchannel"]
|
| 11 |
+
languages = ["en", "en", "en", "hi", "hi"]
|
| 12 |
+
m = compute(labels, preds, languages)
|
| 13 |
+
assert m["accuracy"] == pytest.approx(3 / 5)
|
| 14 |
+
# A backchannel predicted as barge-in cuts the agent off for nothing.
|
| 15 |
+
assert m["false_interruption_rate"] == pytest.approx(1 / 2)
|
| 16 |
+
# A barge-in predicted as backchannel lets the agent talk over the caller.
|
| 17 |
+
assert m["missed_interruption_rate"] == pytest.approx(1 / 3)
|
| 18 |
+
assert m["per_class"]["barge_in"]["precision"] == pytest.approx(2 / 3)
|
| 19 |
+
assert m["per_class"]["barge_in"]["recall"] == pytest.approx(2 / 3)
|
| 20 |
+
assert m["per_language"]["en"]["accuracy"] == pytest.approx(2 / 3)
|
| 21 |
+
assert m["per_language"]["hi"]["count"] == 2
|
| 22 |
+
assert 0 < m["macro_f1"] < 1
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
def test_perfect_predictions():
|
| 26 |
+
labels = ["backchannel", "barge_in"]
|
| 27 |
+
m = compute(labels, labels, ["en", "en"])
|
| 28 |
+
assert m["accuracy"] == m["macro_f1"] == 1.0
|
| 29 |
+
assert m["false_interruption_rate"] == m["missed_interruption_rate"] == 0.0
|
uv.lock
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|