Akash Manohar commited on
Commit
9aa77e5
·
1 Parent(s): dcbadfd

Add standalone training, evaluation and model instructions

Browse files

Pin the separate dataset revision in release metadata; include scoped agent instructions, model milestones, labnotes helper, CPU benchmarks, tests and recorded diagnostic results.

.gitattributes CHANGED
@@ -1,3 +1,3 @@
1
  *.safetensors filter=lfs diff=lfs merge=lfs -text
2
  /tokenizer.json filter=lfs diff=lfs merge=lfs -text
3
- tokenizer.json filter=lfs diff=lfs merge=lfs -text
 
1
  *.safetensors filter=lfs diff=lfs merge=lfs -text
2
  /tokenizer.json filter=lfs diff=lfs merge=lfs -text
3
+ *.csv whitespace=cr-at-eol
.gitignore ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ .venv/
2
+ .venv-cpu/
3
+ __pycache__/
4
+ *.pyc
5
+ .pytest_cache/
6
+ .ruff_cache/
7
+ /data/
8
+ /models/
9
+ /checkpoints/
10
+ .env
11
+ .env.*
.python-version ADDED
@@ -0,0 +1 @@
 
 
1
+ 3.14
AGENTS.md ADDED
@@ -0,0 +1,95 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Barge-in detector model
2
+
3
+ ## Ownership
4
+
5
+ This Hugging Face model repository (`HashNuke/barge-in-detector`) owns weights,
6
+ tokenizer, training, inference, evaluation, model experiments and CPU benchmarks.
7
+ The dataset repository (`HashNuke/barge-in-detector-data`) owns generation,
8
+ cleaning, source utterances, splits, review UI and historical project labnotes.
9
+ Do not copy training data into this repo. Fetch a dataset revision or pass local
10
+ splits. Record the resolved dataset commit, base model, seed, hyperparameters,
11
+ source commit and metrics for every release. Never train on challenge examples.
12
+
13
+ ## Contract and quality gates
14
+
15
+ - Caller utterance spoken over an ongoing voice-agent turn; cleaned lowercase
16
+ text without punctuation. No audio, agent line or language hint is required.
17
+ - Exactly two labels: `backchannel` (continue speaking) and `barge_in` (yield).
18
+ Repetitions, acknowledgments, surprise, empathy and laughter are backchannels
19
+ unless the text develops into a correction, question, request or floor claim.
20
+ Context and an ambiguous/needs-context class are unavailable.
21
+ - At most 64 tokens; keep the first 64. Evaluate partial transcripts and
22
+ transitions from backchannel to interruption. No universal ignored-word list.
23
+ - Output `p_barge_in` in [0,1]. Current experimental probabilities are not yet
24
+ calibrated; thresholds belong to the consumer. Use validation for tuning.
25
+ - CPU inference only. Warmed batch-one model-path p99 <=20 ms and whole path
26
+ including Vxpipe boundary <=50 ms on the designated reference CPU. Report
27
+ p50/p95/p99, hardware, threads, memory, warm-up and token lengths. Local VM
28
+ measurements do not satisfy reference-CPU acceptance gates.
29
+ - Fourteen languages: English (US/India/Singapore), Hindi, Kannada, Tamil,
30
+ Telugu, Malayalam, Mandarin, Japanese, Indonesian, Filipino, French, German,
31
+ Spanish and Italian. Roll out English/Hindi first, then Kannada/Tamil.
32
+ Native scripts, optionally mixed with English; no romanization/pinyin.
33
+ Language support requires evaluation evidence, not tokenizer coverage.
34
+ - Tier A owner verification: English, Hindi, Kannada, Tamil. Other languages
35
+ remain synthetic/unverified unless native-speaker checks are recorded.
36
+ - Report false interruptions and missed interruptions, per-language results,
37
+ short-input slices, prefix behaviour and calibration. Keep synthetic and
38
+ owner-reviewed evidence separate; a perfect synthetic score is insufficient.
39
+ - Prefer permissive base licences. Preserve the MIT attribution for the
40
+ intfloat/multilingual-e5-small base model.
41
+
42
+ ## Tools and remote training
43
+
44
+ - Use uv; add dependencies with `uv add` and keep uv.lock with dependency edits.
45
+ Always source ~/.zshrc or use zsh -lc for preconfigured tools.
46
+ - Always rent exactly one RTX 5090 (32 GB), in a datacenter, below $0.65/hour,
47
+ with good internet and the Vast.ai PyTorch image. Do not substitute another
48
+ GPU or rent multiple instances. Inspect existing instances before renting.
49
+ - All Vast.ai setup and training commands run inside tmux session `gpubox`.
50
+ Save logs/checkpoints to disk. Copy and verify artifacts before destroying
51
+ instances; follow the owner's explicit cleanup instructions.
52
+ - CPU evaluation and benchmarks run locally. Do not rent or restart a GPU for
53
+ work that the local CPU can perform. No Playwright; use agent-browser if
54
+ browser work is needed. Do not disable browser security to work around failures.
55
+ - Never print or commit tokens, authorization headers, .env, PII, caches or
56
+ virtualenvs. Hugging Face credentials come from existing login/environment.
57
+ - Root model.safetensors and tokenizer.json are versioned large artifacts;
58
+ use Hugging Face large-file storage. Keep experiment checkpoints under
59
+ ignored models/ or checkpoints/. Publish selected reports with the release.
60
+
61
+ ## Labnotes and milestones
62
+
63
+ - Start at labnotes/milestones/index.md; its checklist owns implementation order.
64
+ Read linked prerequisites and sources before starting a checkpoint.
65
+ - For implementation or research, use `bin/create-labnotes two-to-four-words`.
66
+ Write factual progress, failures, workarounds, decisions and evidence into
67
+ the generated top-level labnotes file. Link it from the relevant milestone.
68
+ Do not create labnotes for simple status/read-only checks or running an
69
+ existing command. Labnotes accompany the work in its commit.
70
+ - Keep milestones under labnotes/milestones/, deferred issues under
71
+ labnotes/issues/. Use descriptive unnumbered filenames and Markdown task
72
+ lists. Each milestone needs a runnable outcome, prerequisites, contracts,
73
+ checkpoint tasks and acceptance gates. A spec is not an implementation.
74
+ Leave index entries unchecked until all gates pass; record blockers honestly.
75
+ - Historical generation/training labnotes remain in the dataset repo; link
76
+ them rather than copying that history here. New model work gets model labnotes.
77
+ - Record substantial architecture decisions and rejected alternatives under docs/.
78
+
79
+ ## Checkpoints and Git
80
+
81
+ Use small coherent commits of code, focused tests, docs, labnotes and lockfile.
82
+ For behaviour changes, write a focused failing test before implementation.
83
+ Experiments/prompts/benchmarks use recorded evidence; deterministic parsers,
84
+ metrics, tokenization and serving get focused tests. Default tests exclude
85
+ network integration tests. Before finishing code changes run:
86
+
87
+ ```shell
88
+ uv run ruff format --check
89
+ uv run ruff check
90
+ uv run pytest
91
+ ```
92
+
93
+ Inspect status and staged diffs, preserve unrelated work, stage exact paths.
94
+ Do not commit or publish unless requested. Do not amend, rebase, reset,
95
+ force-push, delete branches or undo commits without explicit instruction.
README.md CHANGED
@@ -51,8 +51,10 @@ has not been established.
51
  ## Infer on CPU
52
 
53
  ```shell
54
- uv sync
55
- uv run python -m barge_in_detector_model.infer "please stop for a moment"
 
 
56
  ```
57
 
58
  Input must already follow the dataset cleaner: lowercase, punctuation removed,
@@ -109,7 +111,9 @@ uv run python -m experiments.benchmark_cpu_cold --threads 4
109
  uv run python -m experiments.evaluate_english_challenge --dataset-dir /path/to/dataset
110
  ```
111
 
112
- For CPU-only use, install the matching CPU PyTorch wheel from the official CPU
113
- index. The standard lock also covers GPU training dependencies. Do CPU tests
 
 
114
  locally; rent only a single datacenter RTX 5090 below $0.65/hour for training,
115
  using the Vast.ai PyTorch image and tmux session gpubox.
 
51
  ## Infer on CPU
52
 
53
  ```shell
54
+ uv venv --python 3.14
55
+ uv pip install torch==2.15.0 --index-url https://download.pytorch.org/whl/cpu
56
+ uv pip install transformers==5.23.0 huggingface-hub==2.4.0
57
+ uv run --no-sync python -m barge_in_detector_model.infer "please stop for a moment"
58
  ```
59
 
60
  Input must already follow the dataset cleaner: lowercase, punctuation removed,
 
111
  uv run python -m experiments.evaluate_english_challenge --dataset-dir /path/to/dataset
112
  ```
113
 
114
+ The CPU setup above intentionally uses the matching CPU wheel; use
115
+ `uv run --no-sync` with that environment to retain it. For CPU development,
116
+ install `pytest==9.2.0 ruff==0.17.5` with `uv pip install`. Standard `uv sync`
117
+ uses the lockfile and includes GPU training dependencies. Do CPU tests
118
  locally; rent only a single datacenter RTX 5090 below $0.65/hour for training,
119
  using the Vast.ai PyTorch image and tmux session gpubox.
barge_in_detector_model/__init__.py ADDED
@@ -0,0 +1 @@
 
 
1
+ """Training, inference and evaluation for separately versioned barge-in data."""
barge_in_detector_model/data.py ADDED
@@ -0,0 +1,39 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Read versioned dataset splits without owning their generation or assignment."""
2
+
3
+ import csv
4
+ from pathlib import Path
5
+
6
+ DEFAULT_DATASET = "HashNuke/barge-in-detector-data"
7
+
8
+
9
+ def read_csv(path):
10
+ with Path(path).open(encoding="utf-8", newline="") as f:
11
+ return list(csv.DictReader(f))
12
+
13
+
14
+ def resolve_splits(splits, *, repo_id=DEFAULT_DATASET, revision=None):
15
+ if splits is not None:
16
+ directory = Path(splits)
17
+ resolved_revision = None
18
+ else:
19
+ from huggingface_hub import HfApi, snapshot_download
20
+
21
+ resolved_revision = HfApi().dataset_info(repo_id, revision=revision).sha
22
+ root = Path(
23
+ snapshot_download(
24
+ repo_id=repo_id,
25
+ repo_type="dataset",
26
+ revision=resolved_revision,
27
+ allow_patterns=[
28
+ "data/splits/*.csv",
29
+ "data/evaluation/*.csv",
30
+ "data/evaluation/reviews/*.json",
31
+ "generation/factors/languages.yaml",
32
+ ],
33
+ )
34
+ )
35
+ directory = root / "data/splits"
36
+ for split in ("train", "validation", "test"):
37
+ if not (directory / f"{split}.csv").is_file():
38
+ raise ValueError(f"Dataset split missing: {directory / split}.csv")
39
+ return directory, resolved_revision
barge_in_detector_model/evaluate.py ADDED
@@ -0,0 +1,111 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Evaluate a trained model on one split (README.md).
2
+
3
+ uv run python -m barge_in_detector_model.evaluate --model models/run1 --split test
4
+
5
+ Reports accuracy, macro F1, false and missed interruption rates, and results
6
+ per language; writes them to <model>/eval-<split>.json and the mistakes to
7
+ <model>/errors-<split>.csv.
8
+ """
9
+
10
+ import argparse
11
+ import csv
12
+ import json
13
+ from collections.abc import Sequence
14
+ from pathlib import Path
15
+ from typing import Any
16
+
17
+ import torch
18
+ from transformers import AutoModelForSequenceClassification, AutoTokenizer
19
+
20
+ from barge_in_detector_model.data import read_csv, resolve_splits
21
+ from barge_in_detector_model.metrics import LABELS, compute
22
+
23
+ BARGE_IN = LABELS.index("barge_in")
24
+
25
+
26
+ def device() -> torch.device:
27
+ return torch.device("cuda" if torch.cuda.is_available() else "cpu")
28
+
29
+
30
+ @torch.no_grad()
31
+ def predict(
32
+ model: Any,
33
+ tokenizer: Any,
34
+ rows: Sequence[dict[str, str]],
35
+ *,
36
+ batch_size: int = 128,
37
+ max_length: int = 64,
38
+ ) -> list[float]:
39
+ """P(barge_in) for each row's cleaned text."""
40
+ model.eval()
41
+ dev = next(model.parameters()).device
42
+ out: list[float] = []
43
+ for i in range(0, len(rows), batch_size):
44
+ texts = [r["text_clean"] for r in rows[i : i + batch_size]]
45
+ enc = tokenizer(
46
+ texts, truncation=True, max_length=max_length, padding=True, return_tensors="pt"
47
+ ).to(dev)
48
+ logits = model(**enc).logits.float()
49
+ out += torch.softmax(logits, dim=-1)[:, BARGE_IN].tolist()
50
+ return out
51
+
52
+
53
+ def evaluate(
54
+ model: Any,
55
+ tokenizer: Any,
56
+ rows: Sequence[dict[str, str]],
57
+ *,
58
+ threshold: float = 0.5,
59
+ max_length: int = 64,
60
+ ) -> tuple[dict[str, Any], list[float]]:
61
+ probs = predict(model, tokenizer, rows, max_length=max_length)
62
+ preds = ["barge_in" if p >= threshold else "backchannel" for p in probs]
63
+ metrics = compute([r["label"] for r in rows], preds, [r["language"] for r in rows])
64
+ return metrics, probs
65
+
66
+
67
+ def main(argv: Sequence[str] | None = None) -> int:
68
+ parser = argparse.ArgumentParser(description=__doc__)
69
+ parser.add_argument("--model", type=Path, required=True, help="a directory written by train")
70
+ parser.add_argument("--split", default="test", choices=["train", "validation", "test"])
71
+ parser.add_argument("--splits", type=Path, default=None)
72
+ parser.add_argument("--threshold", type=float, default=0.5, help="P(barge_in) cut-off")
73
+ parser.add_argument("--max-length", type=int, default=64, help="token budget")
74
+ parser.add_argument("--dataset-repo", default="HashNuke/barge-in-detector-data")
75
+ parser.add_argument(
76
+ "--dataset-revision", help="Exact dataset commit; resolves current main when omitted"
77
+ )
78
+ args = parser.parse_args(argv)
79
+ args.splits, args.dataset_revision = resolve_splits(
80
+ args.splits, repo_id=args.dataset_repo, revision=args.dataset_revision
81
+ )
82
+
83
+ tokenizer = AutoTokenizer.from_pretrained(args.model)
84
+ model = AutoModelForSequenceClassification.from_pretrained(args.model).to(device())
85
+ rows = read_csv(args.splits / f"{args.split}.csv")
86
+ metrics, probs = evaluate(
87
+ model, tokenizer, rows, threshold=args.threshold, max_length=args.max_length
88
+ )
89
+ metrics.update(split=args.split, threshold=args.threshold)
90
+ (args.model / f"eval-{args.split}.json").write_text(json.dumps(metrics, indent=2))
91
+ with (args.model / f"errors-{args.split}.csv").open("w", encoding="utf-8", newline="") as f:
92
+ writer = csv.writer(f)
93
+ writer.writerow(["label", "p_barge_in", "language", "caller_intent", "text"])
94
+ for r, p in zip(rows, probs, strict=True):
95
+ if (p >= args.threshold) != (r["label"] == "barge_in"):
96
+ writer.writerow(
97
+ [r["label"], f"{p:.3f}", r["language"], r["caller_intent"], r["text"]]
98
+ )
99
+ print(
100
+ f"{args.split}: n={metrics['count']} accuracy={metrics['accuracy']:.3f} "
101
+ f"macro_f1={metrics['macro_f1']:.3f} "
102
+ f"false_interruptions={metrics['false_interruption_rate']:.3f} "
103
+ f"missed_interruptions={metrics['missed_interruption_rate']:.3f}"
104
+ )
105
+ for lang, m in metrics["per_language"].items():
106
+ print(f" {lang:>4} n={m['count']:<4} accuracy={m['accuracy']:.3f}")
107
+ return 0
108
+
109
+
110
+ if __name__ == "__main__":
111
+ raise SystemExit(main())
barge_in_detector_model/infer.py ADDED
@@ -0,0 +1,27 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Predict a binary label for an already-cleaned overlapping caller utterance."""
2
+
3
+ import argparse
4
+ import json
5
+ from pathlib import Path
6
+
7
+ import torch
8
+ from transformers import AutoModelForSequenceClassification, AutoTokenizer
9
+
10
+
11
+ def main():
12
+ parser = argparse.ArgumentParser(description=__doc__)
13
+ parser.add_argument("text")
14
+ parser.add_argument("--model", type=Path, default=Path("."))
15
+ parser.add_argument("--threads", type=int, default=4)
16
+ args = parser.parse_args()
17
+ torch.set_num_threads(args.threads)
18
+ tokenizer = AutoTokenizer.from_pretrained(args.model)
19
+ model = AutoModelForSequenceClassification.from_pretrained(args.model).cpu().eval()
20
+ with torch.inference_mode():
21
+ enc = tokenizer(args.text, truncation=True, max_length=64, return_tensors="pt")
22
+ p = torch.softmax(model(**enc).logits, dim=-1)[0, model.config.label2id["barge_in"]].item()
23
+ print(json.dumps({"p_barge_in": p, "label": "barge_in" if p >= 0.5 else "backchannel"}))
24
+
25
+
26
+ if __name__ == "__main__":
27
+ main()
barge_in_detector_model/metrics.py ADDED
@@ -0,0 +1,46 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Metrics for the two labels. Pure Python, so they are tested without torch."""
2
+
3
+ from collections import defaultdict
4
+ from collections.abc import Sequence
5
+ from typing import Any
6
+
7
+ LABELS = ("backchannel", "barge_in")
8
+
9
+
10
+ def _prf(labels: Sequence[str], preds: Sequence[str], cls: str) -> dict[str, float]:
11
+ tp = sum(1 for y, p in zip(labels, preds, strict=True) if y == cls and p == cls)
12
+ predicted = sum(1 for p in preds if p == cls)
13
+ actual = sum(1 for y in labels if y == cls)
14
+ precision = tp / predicted if predicted else 0.0
15
+ recall = tp / actual if actual else 0.0
16
+ f1 = 2 * precision * recall / (precision + recall) if precision + recall else 0.0
17
+ return {"precision": precision, "recall": recall, "f1": f1, "count": actual}
18
+
19
+
20
+ def compute(
21
+ labels: Sequence[str], preds: Sequence[str], languages: Sequence[str]
22
+ ) -> dict[str, Any]:
23
+ per_class = {cls: _prf(labels, preds, cls) for cls in LABELS}
24
+ backchannel = [p for y, p in zip(labels, preds, strict=True) if y == "backchannel"]
25
+ barge_in = [p for y, p in zip(labels, preds, strict=True) if y == "barge_in"]
26
+ by_language: dict[str, list[bool]] = defaultdict(list)
27
+ for y, p, lang in zip(labels, preds, languages, strict=True):
28
+ by_language[lang].append(y == p)
29
+ return {
30
+ "count": len(labels),
31
+ "accuracy": sum(y == p for y, p in zip(labels, preds, strict=True)) / len(labels),
32
+ "macro_f1": sum(c["f1"] for c in per_class.values()) / len(LABELS),
33
+ # A backchannel predicted as barge-in stops the agent for nothing.
34
+ "false_interruption_rate": (
35
+ sum(p == "barge_in" for p in backchannel) / len(backchannel) if backchannel else 0.0
36
+ ),
37
+ # A barge-in predicted as backchannel lets the agent talk over the caller.
38
+ "missed_interruption_rate": (
39
+ sum(p == "backchannel" for p in barge_in) / len(barge_in) if barge_in else 0.0
40
+ ),
41
+ "per_class": per_class,
42
+ "per_language": {
43
+ lang: {"accuracy": sum(ok) / len(ok), "count": len(ok)}
44
+ for lang, ok in sorted(by_language.items())
45
+ },
46
+ }
barge_in_detector_model/train.py ADDED
@@ -0,0 +1,140 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Fine-tune a multilingual encoder on the utterance splits (README.md).
2
+
3
+ uv sync --group train
4
+ uv run python -m barge_in_detector_model.train --out models/run1
5
+
6
+ Trains on train.csv's cleaned text, evaluates on validation.csv after every
7
+ epoch, and keeps the epoch with the best validation macro F1 in --out (model,
8
+ tokenizer, metrics.json). Backchannel is the smaller class, so the loss
9
+ weights each class by its inverse frequency. Uses the GPU when there is one.
10
+ """
11
+
12
+ import argparse
13
+ import json
14
+ import random
15
+ import time
16
+ from collections import Counter
17
+ from collections.abc import Sequence
18
+ from pathlib import Path
19
+
20
+ import torch
21
+ from transformers import (
22
+ AutoModelForSequenceClassification,
23
+ AutoTokenizer,
24
+ get_linear_schedule_with_warmup,
25
+ )
26
+
27
+ from barge_in_detector_model.data import read_csv, resolve_splits
28
+ from barge_in_detector_model.evaluate import device, evaluate
29
+ from barge_in_detector_model.metrics import LABELS
30
+
31
+ # MIT licence, 118M parameters, about 100 languages (dataset repo docs/utterance-first-decision.md).
32
+ DEFAULT_MODEL = "intfloat/multilingual-e5-small"
33
+
34
+
35
+ def main(argv: Sequence[str] | None = None) -> int:
36
+ parser = argparse.ArgumentParser(description=__doc__)
37
+ parser.add_argument("--out", type=Path, required=True)
38
+ parser.add_argument("--model", default=DEFAULT_MODEL, help="Hugging Face model id or path")
39
+ parser.add_argument("--splits", type=Path, default=None)
40
+ parser.add_argument("--epochs", type=int, default=5)
41
+ parser.add_argument("--batch-size", type=int, default=32)
42
+ parser.add_argument("--lr", type=float, default=3e-5)
43
+ parser.add_argument("--warmup", type=float, default=0.1, help="share of steps")
44
+ parser.add_argument("--max-length", type=int, default=64, help="token budget")
45
+ parser.add_argument("--seed", type=int, default=20261115)
46
+ parser.add_argument("--dataset-repo", default="HashNuke/barge-in-detector-data")
47
+ parser.add_argument(
48
+ "--dataset-revision", help="Exact dataset commit; resolves current main when omitted"
49
+ )
50
+ args = parser.parse_args(argv)
51
+ args.splits, args.dataset_revision = resolve_splits(
52
+ args.splits, repo_id=args.dataset_repo, revision=args.dataset_revision
53
+ )
54
+
55
+ random.seed(args.seed)
56
+ torch.manual_seed(args.seed)
57
+ dev = device()
58
+ train_rows = read_csv(args.splits / "train.csv")
59
+ validation_rows = read_csv(args.splits / "validation.csv")
60
+ label_id = {label: i for i, label in enumerate(LABELS)}
61
+
62
+ tokenizer = AutoTokenizer.from_pretrained(args.model)
63
+ model = AutoModelForSequenceClassification.from_pretrained(
64
+ args.model,
65
+ num_labels=len(LABELS),
66
+ id2label=dict(enumerate(LABELS)),
67
+ label2id=label_id,
68
+ ).to(dev)
69
+
70
+ counts = Counter(r["label"] for r in train_rows)
71
+ weights = torch.tensor(
72
+ [len(train_rows) / (len(LABELS) * counts[label]) for label in LABELS], device=dev
73
+ )
74
+ loss_fn = torch.nn.CrossEntropyLoss(weight=weights)
75
+ steps = args.epochs * -(-len(train_rows) // args.batch_size)
76
+ optimizer = torch.optim.AdamW(model.parameters(), lr=args.lr, weight_decay=0.01)
77
+ scheduler = get_linear_schedule_with_warmup(optimizer, int(args.warmup * steps), steps)
78
+ use_bf16 = dev.type == "cuda" and torch.cuda.is_bf16_supported()
79
+
80
+ print(
81
+ f"device={dev} bf16={use_bf16} model={args.model} train={len(train_rows)} "
82
+ f"validation={len(validation_rows)} class_weights={weights.tolist()}"
83
+ )
84
+ args.out.mkdir(parents=True, exist_ok=True)
85
+ history, best = [], None
86
+ for epoch in range(1, args.epochs + 1):
87
+ model.train()
88
+ order = list(range(len(train_rows)))
89
+ random.shuffle(order)
90
+ started, total = time.time(), 0.0
91
+ for i in range(0, len(order), args.batch_size):
92
+ batch = [train_rows[j] for j in order[i : i + args.batch_size]]
93
+ enc = tokenizer(
94
+ [r["text_clean"] for r in batch],
95
+ truncation=True,
96
+ max_length=args.max_length,
97
+ padding=True,
98
+ return_tensors="pt",
99
+ ).to(dev)
100
+ labels = torch.tensor([label_id[r["label"]] for r in batch], device=dev)
101
+ with torch.autocast(device_type=dev.type, dtype=torch.bfloat16, enabled=use_bf16):
102
+ logits = model(**enc).logits
103
+ loss = loss_fn(logits.float(), labels)
104
+ loss.backward()
105
+ torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
106
+ optimizer.step()
107
+ scheduler.step()
108
+ optimizer.zero_grad()
109
+ total += loss.item() * len(batch)
110
+ metrics, _ = evaluate(model, tokenizer, validation_rows, max_length=args.max_length)
111
+ record = {
112
+ "epoch": epoch,
113
+ "train_loss": total / len(train_rows),
114
+ "seconds": round(time.time() - started, 1),
115
+ "validation": metrics,
116
+ }
117
+ history.append(record)
118
+ print(
119
+ f"epoch {epoch}: loss={record['train_loss']:.4f} "
120
+ f"validation macro_f1={metrics['macro_f1']:.3f} accuracy={metrics['accuracy']:.3f} "
121
+ f"false_interruptions={metrics['false_interruption_rate']:.3f} "
122
+ f"missed_interruptions={metrics['missed_interruption_rate']:.3f}"
123
+ )
124
+ if best is None or metrics["macro_f1"] > best["validation"]["macro_f1"]:
125
+ best = record
126
+ model.save_pretrained(args.out)
127
+ tokenizer.save_pretrained(args.out)
128
+ (args.out / "metrics.json").write_text(
129
+ json.dumps(
130
+ {"args": {k: str(v) for k, v in vars(args).items()}, "best": best, "history": history},
131
+ indent=2,
132
+ )
133
+ )
134
+ print(f"best epoch {best['epoch']}: validation macro_f1={best['validation']['macro_f1']:.3f}")
135
+ print(f"saved to {args.out}")
136
+ return 0
137
+
138
+
139
+ if __name__ == "__main__":
140
+ raise SystemExit(main())
bin/create-labnotes ADDED
@@ -0,0 +1,59 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env bash
2
+ set -euo pipefail
3
+
4
+ usage() {
5
+ cat <<'USAGE'
6
+ Usage:
7
+ bin/create-labnotes 2-4-word-hyphenated-task-name
8
+
9
+ Creates an empty markdown labnotes file in labnotes/.
10
+
11
+ The input must be a 2-4 word hyphenated string, for example:
12
+ bin/create-labnotes test-crawl-resume
13
+
14
+ If spaces are provided, they will be converted to hyphens:
15
+ bin/create-labnotes "test crawl resume"
16
+ USAGE
17
+ }
18
+
19
+ if [[ $# -eq 0 ]]; then
20
+ usage
21
+ exit 0
22
+ fi
23
+
24
+ input="$*"
25
+ slug="$(printf '%s' "$input" \
26
+ | tr '[:upper:]' '[:lower:]' \
27
+ | sed -E 's/[[:space:]]+/-/g; s/[^a-z0-9-]+/-/g; s/-+/-/g; s/^-//; s/-$//')"
28
+
29
+ if [[ -z "$slug" ]]; then
30
+ echo "Error: task name must contain letters or numbers." >&2
31
+ echo >&2
32
+ usage >&2
33
+ exit 1
34
+ fi
35
+
36
+ word_count="$(awk -F- '{ print NF }' <<<"$slug")"
37
+ if (( word_count < 2 || word_count > 4 )); then
38
+ echo "Error: input must be a 2-4 word hyphenated string." >&2
39
+ echo >&2
40
+ usage >&2
41
+ exit 1
42
+ fi
43
+
44
+ repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
45
+ labnotes_dir="$repo_root/labnotes"
46
+ mkdir -p "$labnotes_dir"
47
+
48
+ timestamp="$(date '+%Y%m%d-%H%M')"
49
+ filename="$timestamp-$slug.md"
50
+ path="$labnotes_dir/$filename"
51
+
52
+ if [[ -e "$path" ]]; then
53
+ echo "Error: labnotes/$filename already exists." >&2
54
+ exit 1
55
+ fi
56
+
57
+ touch "$path"
58
+
59
+ echo "Created labnotes/$filename"
experiments/benchmark_cpu.py ADDED
@@ -0,0 +1,140 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Measure warmed batch-one tokenization through label output on local CPU.
2
+
3
+ Experimental timing harness: run from the repo root with a CPU PyTorch environment.
4
+ It measures no runtime boundary, queue, HTTP call or cold model load.
5
+ """
6
+
7
+ import argparse
8
+ import hashlib
9
+ import json
10
+ import platform
11
+ import random
12
+ import resource
13
+ import statistics
14
+ import time
15
+ from pathlib import Path
16
+
17
+ import torch
18
+ import transformers
19
+ from transformers import AutoModelForSequenceClassification, AutoTokenizer
20
+
21
+ from barge_in_detector_model.data import read_csv, resolve_splits
22
+
23
+
24
+ def percentile(values, quantile):
25
+ ordered = sorted(values)
26
+ index = (len(ordered) - 1) * quantile
27
+ low = int(index)
28
+ high = min(low + 1, len(ordered) - 1)
29
+ return ordered[low] + (ordered[high] - ordered[low]) * (index - low)
30
+
31
+
32
+ def summary(values):
33
+ return {
34
+ "p50_ms": percentile(values, 0.5),
35
+ "p95_ms": percentile(values, 0.95),
36
+ "p99_ms": percentile(values, 0.99),
37
+ "mean_ms": statistics.mean(values),
38
+ "max_ms": max(values),
39
+ }
40
+
41
+
42
+ @torch.inference_mode()
43
+ def main():
44
+ parser = argparse.ArgumentParser(description=__doc__)
45
+ parser.add_argument("--iterations", type=int, default=1000)
46
+ parser.add_argument("--warmup", type=int, default=100)
47
+ parser.add_argument("--model", type=Path, default=Path("."))
48
+ parser.add_argument("--threads", type=int, nargs="+", default=[1, 2, 4])
49
+ parser.add_argument("--splits", type=Path, help="Local dataset split directory")
50
+ parser.add_argument("--dataset-revision")
51
+ args = parser.parse_args()
52
+ split_dir, dataset_revision = resolve_splits(args.splits, revision=args.dataset_revision)
53
+ if args.iterations < 100 or args.warmup < 1 or min(args.threads) < 1:
54
+ parser.error("Use >=100 measured iterations, positive warmup and threads")
55
+ torch.set_num_interop_threads(1)
56
+ torch.set_num_threads(1)
57
+ started = time.perf_counter()
58
+ tokenizer = AutoTokenizer.from_pretrained(args.model, local_files_only=True)
59
+ model = (
60
+ AutoModelForSequenceClassification.from_pretrained(args.model, local_files_only=True)
61
+ .cpu()
62
+ .eval()
63
+ )
64
+ load_seconds = time.perf_counter() - started
65
+ rows = read_csv(split_dir / "test.csv")
66
+ texts = [r["text_clean"] for r in rows]
67
+ random.Random(20261115).shuffle(texts)
68
+ stress = (
69
+ "please stop for a moment because i need to correct the information you are reading " * 12
70
+ )
71
+ assert len(tokenizer(stress, truncation=True, max_length=64)["input_ids"]) == 64
72
+ report = {
73
+ "dataset_revision": dataset_revision,
74
+ "model": str(args.model),
75
+ "runtime": "PyTorch eager fp32, CPU only",
76
+ "model_sha256": hashlib.sha256((args.model / "model.safetensors").read_bytes()).hexdigest(),
77
+ "python": platform.python_version(),
78
+ "torch": torch.__version__,
79
+ "transformers": transformers.__version__,
80
+ "platform": platform.platform(),
81
+ "cpuinfo": next(
82
+ line.split(":", 1)[1].strip()
83
+ for line in Path("/proc/cpuinfo").read_text().splitlines()
84
+ if line.startswith("model name")
85
+ ),
86
+ "load_seconds_excluded": load_seconds,
87
+ "iterations": args.iterations,
88
+ "warmup_per_scenario": args.warmup,
89
+ "batch_size": 1,
90
+ "max_length": 64,
91
+ "interop_threads": 1,
92
+ "parameter_bytes": sum(p.numel() * p.element_size() for p in model.parameters()),
93
+ "corpus_sha256": hashlib.sha256((split_dir / "test.csv").read_bytes()).hexdigest(),
94
+ "corpus_rows": len(rows),
95
+ "languages": sorted({r["language"] for r in rows}),
96
+ "measurements": [],
97
+ "scope": "Local virtual CPU; reference CPU unspecified. Excludes Vxpipe boundary and queueing; no whole-path acceptance claim.",
98
+ }
99
+ lengths = [len(tokenizer(t, truncation=True, max_length=64)["input_ids"]) for t in texts]
100
+ report["corpus_token_lengths"] = {
101
+ "p50": percentile(lengths, 0.5),
102
+ "p95": percentile(lengths, 0.95),
103
+ "max": max(lengths),
104
+ }
105
+ for threads in args.threads:
106
+ torch.set_num_threads(threads)
107
+ for name, inputs in [("multilingual_test", texts), ("full_64_tokens", [stress])]:
108
+ samples, token_times, forward_times = [], [], []
109
+ for index in range(args.warmup + args.iterations):
110
+ text = inputs[index % len(inputs)]
111
+ start = time.perf_counter_ns()
112
+ enc = tokenizer(text, truncation=True, max_length=64, return_tensors="pt")
113
+ tokenized = time.perf_counter_ns()
114
+ logits = model(**enc).logits
115
+ probability = torch.softmax(logits, dim=-1)[
116
+ 0, model.config.label2id["barge_in"]
117
+ ].item()
118
+ label = "barge_in" if probability >= 0.5 else "backchannel"
119
+ finish = time.perf_counter_ns()
120
+ assert 0 <= probability <= 1
121
+ assert label in model.config.label2id
122
+ if index >= args.warmup:
123
+ samples.append((finish - start) / 1e6)
124
+ token_times.append((tokenized - start) / 1e6)
125
+ forward_times.append((finish - tokenized) / 1e6)
126
+ result = {
127
+ "threads": threads,
128
+ "scenario": name,
129
+ "model_path": summary(samples),
130
+ "tokenization": summary(token_times),
131
+ "forward_and_probability": summary(forward_times),
132
+ }
133
+ report["measurements"].append(result)
134
+ print(json.dumps(result), flush=True)
135
+ report["peak_process_rss_mib"] = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024
136
+ (args.model / "cpu-benchmark.json").write_text(json.dumps(report, indent=2))
137
+
138
+
139
+ if __name__ == "__main__":
140
+ main()
experiments/benchmark_cpu_cold.py ADDED
@@ -0,0 +1,80 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Time a new Python process through its first label from local model files."""
2
+
3
+ import argparse
4
+ import json
5
+ import os
6
+ import statistics
7
+ import subprocess
8
+ import sys
9
+ import time
10
+ from pathlib import Path
11
+
12
+ CHILD = """
13
+ import json
14
+ import time
15
+ import torch
16
+ from transformers import AutoModelForSequenceClassification, AutoTokenizer
17
+ import sys
18
+ model_path, threads = sys.argv[1], int(sys.argv[2])
19
+ torch.set_num_threads(threads)
20
+ torch.set_num_interop_threads(1)
21
+ tokenizer = AutoTokenizer.from_pretrained(model_path, local_files_only=True)
22
+ model = AutoModelForSequenceClassification.from_pretrained(model_path, local_files_only=True).cpu().eval()
23
+ with torch.inference_mode():
24
+ loaded_start = time.perf_counter_ns()
25
+ enc = tokenizer('please stop for a moment', truncation=True, max_length=64, return_tensors='pt')
26
+ p = torch.softmax(model(**enc).logits, dim=-1)[0, model.config.label2id['barge_in']].item()
27
+ label = 'barge_in' if p >= 0.5 else 'backchannel'
28
+ loaded_ms = (time.perf_counter_ns() - loaded_start) / 1e6
29
+ print(json.dumps({'label': label, 'p_barge_in': p, 'loaded_to_first_label_ms': loaded_ms}), flush=True)
30
+ """
31
+
32
+
33
+ def main():
34
+ parser = argparse.ArgumentParser(description=__doc__)
35
+ parser.add_argument("--model", type=Path, default=Path("."))
36
+ parser.add_argument("--runs", type=int, default=10)
37
+ parser.add_argument("--threads", type=int, default=1)
38
+ args = parser.parse_args()
39
+ samples = []
40
+ loaded_first = []
41
+ env = {
42
+ **os.environ,
43
+ "OMP_NUM_THREADS": str(args.threads),
44
+ "MKL_NUM_THREADS": str(args.threads),
45
+ "TOKENIZERS_PARALLELISM": "false",
46
+ }
47
+ for run in range(args.runs):
48
+ start = time.perf_counter_ns()
49
+ with subprocess.Popen(
50
+ [sys.executable, "-c", CHILD, str(args.model), str(args.threads)],
51
+ stdout=subprocess.PIPE,
52
+ stderr=subprocess.DEVNULL,
53
+ text=True,
54
+ env=env,
55
+ ) as process:
56
+ line = process.stdout.readline()
57
+ elapsed = (time.perf_counter_ns() - start) / 1e9
58
+ result = json.loads(line)
59
+ if process.wait() != 0:
60
+ raise RuntimeError("Cold-start child failed")
61
+ samples.append(elapsed)
62
+ loaded_first.append(result["loaded_to_first_label_ms"])
63
+ print(f"cold start {run + 1}: {elapsed:.3f}s {result}", flush=True)
64
+ report = {
65
+ "cold_process_seconds": samples,
66
+ "loaded_to_first_label_ms": loaded_first,
67
+ "loaded_to_first_label_median_ms": statistics.median(loaded_first),
68
+ "median_seconds": statistics.median(samples),
69
+ "min_seconds": min(samples),
70
+ "max_seconds": max(samples),
71
+ "runs": args.runs,
72
+ "threads": args.threads,
73
+ "scope": "Python process creation through reading first flushed label. Includes imports, tokenizer/model load, first inference. Local files only; OS file cache may be warm. Excludes process shutdown, downloads and HTTP/network boundary.",
74
+ }
75
+ (args.model / "cpu-cold-benchmark.json").write_text(json.dumps(report, indent=2))
76
+ print(json.dumps(report), flush=True)
77
+
78
+
79
+ if __name__ == "__main__":
80
+ main()
experiments/evaluate_english_challenge.py ADDED
@@ -0,0 +1,132 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Frozen-checkpoint diagnostic experiment; labels are authored before inference.
2
+
3
+ Run from the repo root with uv run --group train python -m experiments.evaluate_english_challenge.
4
+ Evidence is saved under the model directory, outside the training splits.
5
+ """
6
+
7
+ import argparse
8
+ import csv
9
+ import hashlib
10
+ import json
11
+ from collections import defaultdict
12
+ from pathlib import Path
13
+
14
+ import torch
15
+ import transformers
16
+ from transformers import AutoModelForSequenceClassification, AutoTokenizer
17
+
18
+ from barge_in_detector_model.data import read_csv, resolve_splits
19
+ from barge_in_detector_model.evaluate import device, predict
20
+ from barge_in_detector_model.metrics import compute
21
+
22
+ MODEL = Path(".")
23
+
24
+
25
+ def score(rows, probs):
26
+ return compute(
27
+ [r["label"] for r in rows],
28
+ ["barge_in" if p >= 0.5 else "backchannel" for p in probs],
29
+ [r["language"] for r in rows],
30
+ )
31
+
32
+
33
+ def main():
34
+ parser = argparse.ArgumentParser(description=__doc__)
35
+ parser.add_argument("--dataset-dir", type=Path, help="Local dataset repository root")
36
+ parser.add_argument("--dataset-revision")
37
+ args = parser.parse_args()
38
+ if args.dataset_dir is None:
39
+ split_dir, revision = resolve_splits(None, revision=args.dataset_revision)
40
+ dataset_dir = split_dir.parents[1]
41
+ else:
42
+ dataset_dir = args.dataset_dir
43
+ revision = None
44
+ challenge = dataset_dir / "data/evaluation/english-challenge-v1.csv"
45
+ rows = read_csv(challenge)
46
+ splits = {
47
+ s: read_csv(dataset_dir / f"data/splits/{s}.csv") for s in ("train", "validation", "test")
48
+ }
49
+ texts = {s: {r["text_clean"] for r in rs} for s, rs in splits.items()}
50
+ all_seen = set.union(*texts.values())
51
+ tokenizer = AutoTokenizer.from_pretrained(MODEL)
52
+ model = AutoModelForSequenceClassification.from_pretrained(MODEL).to(device())
53
+ probs = predict(model, tokenizer, rows, max_length=64)
54
+ single_probs = predict(model, tokenizer, rows, batch_size=1, max_length=64)
55
+ report = {
56
+ "dataset_revision": revision,
57
+ "challenge_sha256": hashlib.sha256(challenge.read_bytes()).hexdigest(),
58
+ "model_sha256": hashlib.sha256((MODEL / "model.safetensors").read_bytes()).hexdigest(),
59
+ "threshold": 0.5,
60
+ "runtime": {
61
+ "device": str(next(model.parameters()).device),
62
+ "torch": torch.__version__,
63
+ "transformers": transformers.__version__,
64
+ "threads": torch.get_num_threads(),
65
+ },
66
+ "batch_vs_single_max_probability_difference": max(
67
+ abs(a - b) for a, b in zip(probs, single_probs, strict=True)
68
+ ),
69
+ "batch_vs_single_label_disagreements": sum(
70
+ (a >= 0.5) != (b >= 0.5) for a, b in zip(probs, single_probs, strict=True)
71
+ ),
72
+ "review_status": "Owner-reviewed v1 labels match authored expectations after authorized reconciliation. Diagnostic development set; not independent representative accuracy evidence.",
73
+ "all": score(rows, probs),
74
+ "by_category": {},
75
+ "overlap_counts": {s: sum(r["text_clean"] in ts for r in rows) for s, ts in texts.items()},
76
+ "cross_split_exact_overlap": {
77
+ f"{a}/{b}": len(texts[a] & texts[b])
78
+ for a, b in (("train", "validation"), ("train", "test"), ("validation", "test"))
79
+ },
80
+ }
81
+ for category in sorted({r["category"] for r in rows}):
82
+ pairs = [(r, p) for r, p in zip(rows, probs, strict=True) if r["category"] == category]
83
+ report["by_category"][category] = score(*zip(*pairs, strict=True))
84
+ unseen = [(r, p) for r, p in zip(rows, probs, strict=True) if r["text_clean"] not in all_seen]
85
+ report["unseen"] = score(*zip(*unseen, strict=True))
86
+ groups = defaultdict(list)
87
+ for r, p in zip(rows, probs, strict=True):
88
+ if r["group"]:
89
+ groups[r["group"]].append(
90
+ {"text": r["text_clean"], "label": r["label"], "p_barge_in": p}
91
+ )
92
+ report["prefix_groups"] = dict(groups)
93
+ report["prefix_groups_all_labels_correct"] = sum(
94
+ all((r["p_barge_in"] >= 0.5) == (r["label"] == "barge_in") for r in group)
95
+ for group in groups.values()
96
+ )
97
+ # A diagnostic baseline: select its word-count cutoff using validation only.
98
+ english = {s: [r for r in rs if r["language"] == "en"] for s, rs in splits.items()}
99
+
100
+ def length_probs(rs, cutoff):
101
+ return [float(len(r["text_clean"].split()) >= cutoff) for r in rs]
102
+
103
+ cutoff = max(
104
+ range(1, 25),
105
+ key=lambda n: score(english["validation"], length_probs(english["validation"], n))[
106
+ "macro_f1"
107
+ ],
108
+ )
109
+ report["english_length_baseline"] = {
110
+ "cutoff_words": cutoff,
111
+ "validation": score(english["validation"], length_probs(english["validation"], cutoff)),
112
+ "test": score(english["test"], length_probs(english["test"], cutoff)),
113
+ "challenge": score(rows, length_probs(rows, cutoff)),
114
+ }
115
+ with (MODEL / "challenge-predictions.csv").open("w", newline="") as f:
116
+ writer = csv.DictWriter(f, fieldnames=[*rows[0], "p_barge_in", "prediction", "seen_in"])
117
+ writer.writeheader()
118
+ for r, p in zip(rows, probs, strict=True):
119
+ writer.writerow(
120
+ {
121
+ **r,
122
+ "p_barge_in": p,
123
+ "prediction": "barge_in" if p >= 0.5 else "backchannel",
124
+ "seen_in": ";".join(s for s, ts in texts.items() if r["text_clean"] in ts),
125
+ }
126
+ )
127
+ (MODEL / "challenge-report.json").write_text(json.dumps(report, indent=2))
128
+ print(json.dumps({k: v for k, v in report.items() if k != "prefix_groups"}, indent=2))
129
+
130
+
131
+ if __name__ == "__main__":
132
+ main()
labnotes/20261115-1332-repository-split.md ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Model repository split
2
+
3
+ Owner moved model artifacts, training, inference, evaluation and benchmarks into
4
+ this standalone Hugging Face model repo. Dataset generation, splitting, review
5
+ and all earlier labnotes remain in HashNuke/barge-in-detector-data. This repo has
6
+ its own scoped AGENTS.md, bin/create-labnotes and milestone index.
7
+
8
+ - New package barge_in_detector_model isolates training/metrics/data-loading
9
+ code from the dataset generator. Model code downloads a resolved dataset
10
+ commit, or uses explicit local splits; it does not assign splits.
11
+ - Four unit tests cover metrics, local CSVs, missing split detection and pinned
12
+ Hub downloads. Tiny CPU smoke trained one epoch, saved, reloaded and evaluated.
13
+ - Actual unchanged checkpoint predicts stop as backchannel, p=0.0134437,
14
+ reproducing the known failure. No GPU rental or retraining of release weights.
15
+ - Token budget remains 64; two labels; predictions are not calibrated.
16
+ - Root weights and tokenizer match the original artifact hashes. Original source
17
+ manifest and sanitized training metrics retain run provenance. Model card
18
+ reports synthetic and reviewed diagnostics and local CPU timing limitations.
19
+ - AGENTS.md derives CPU serving gates, fourteen languages, single RTX 5090,
20
+ under-$0.65/hour datacenter/PyTorch policy, gpubox, tests, and labnotes/milestone
21
+ usage from the dataset instructions. Existing history is linked, not duplicated.
22
+
23
+ Historical run and latency evidence is in the dataset repo labnotes. Follow
24
+ labnotes/milestones/index.md for new model work. Publication verification follows.
labnotes/issues/.gitkeep ADDED
File without changes
labnotes/milestones/experimental-checkpoint-release.md ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Experimental checkpoint release
2
+
3
+ ## Runnable outcome
4
+
5
+ A self-contained Hugging Face model repo that loads the first checkpoint on CPU,
6
+ trains/evaluates with a separate versioned dataset, and records honest limitations.
7
+ Prerequisites: dataset HashNuke/barge-in-detector-data, original generation/training
8
+ history in that dataset repo, root README and AGENTS.md.
9
+
10
+ ## Contracts and checkpoints
11
+
12
+ - [x] Copy and verify selected epoch-5 weights and tokenizer without retraining.
13
+ - [x] Move training/evaluation/metrics/benchmarks into independent model modules.
14
+ - [x] Resolve Hub dataset revisions to immutable commits; support local splits.
15
+ - [x] Copy labnotes helper; derive model AGENTS.md and establish milestone usage.
16
+ - [x] Tiny CPU train/save/reload/evaluate smoke and actual checkpoint inference.
17
+ - [ ] Publish artifacts, instructions and code to the model repo.
18
+ - [ ] Record dataset commit, source snapshot and artifact checksums in release.json.
19
+
20
+ ## Acceptance gates
21
+
22
+ - [ ] Four model unit tests, lint and formatting pass.
23
+ - [ ] Model weights and tokenizer resolve as large-file artifacts on the Hub.
24
+ - [ ] Published files match local release hashes; CPU inference works from files.
25
+ - [ ] Model card separates synthetic and reviewed diagnostics and records failures.
26
+
27
+ This milestone does not claim production readiness, calibration or reference-CPU
28
+ latency acceptance. See repository-split labnotes for evidence.
29
+
30
+ Checkpoint evidence: [repository split labnotes](../20261115-1332-repository-split.md).
labnotes/milestones/index.md ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ # Model milestones
2
+
3
+ The checklist owns implementation order. Read AGENTS.md and linked sources;
4
+ leave entries unchecked until every acceptance gate passes.
5
+
6
+ - [ ] [Publish the experimental checkpoint](experimental-checkpoint-release.md)
7
+ - [ ] [Improve short-interruption recall](short-interruption-recall.md)
labnotes/milestones/short-interruption-recall.md ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Improve short-interruption recall
2
+
3
+ ## Runnable outcome and prerequisites
4
+
5
+ Improve the current checkpoint's 5/30 short-interruption recall on reviewed
6
+ English development examples, then establish performance on fresh owner-reviewed
7
+ English/Hindi evaluation data. Prerequisite: experimental checkpoint release;
8
+ coordinate data changes with the dataset repo coverage-audit milestone.
9
+
10
+ ## Contract and checkpoint tasks
11
+
12
+ - [ ] Agree acceptance floors for false and missed interruptions before tuning.
13
+ - [ ] Compare a character n-gram baseline and encoder on exactly the same splits.
14
+ - [ ] Train against the revised dataset distribution; never train on challenge rows.
15
+ - [ ] Evaluate short interruptions, longer backchannels, prefix transitions and
16
+ per-language errors. Preserve old checkpoint and use validation for selection.
17
+ - [ ] Record immutable dataset/source revisions, parameters and costs.
18
+ - [ ] Calibrate on realistic validation priors.
19
+ - [ ] Measure ONNX/int8 parity and reference-CPU latency/memory; measure Vxpipe
20
+ serialization/queue/boundary overhead for the whole-path budget.
21
+
22
+ ## Acceptance gates
23
+
24
+ - [ ] Fresh human-reviewed English/Hindi results meet agreed floors and beat baseline.
25
+ - [ ] Model-path p99 <=20 ms; whole path p99 <=50 ms on reference CPU, warmed batch one.
26
+ - [ ] Prefix behaviour and Vxpipe overlapping-speech scenarios pass with evidence.
27
+
28
+ Use `bin/create-labnotes` and link each checkpoint's factual evidence here.
pyproject.toml ADDED
@@ -0,0 +1,25 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [project]
2
+ name = "barge-in-detector-model"
3
+ version = "0.1.0"
4
+ description = "Training, evaluation and CPU inference for the Vxpipe barge-in classifier"
5
+ requires-python = ">=3.14"
6
+ dependencies = ["torch>=2.15.0", "transformers>=5.23.0", "huggingface-hub>=2.4.0"]
7
+
8
+ [dependency-groups]
9
+ dev = ["pytest>=9.2.0", "ruff>=0.17.5"]
10
+
11
+ [tool.uv]
12
+ package = false
13
+
14
+ [tool.pytest.ini_options]
15
+ pythonpath = ["."]
16
+ testpaths = ["tests"]
17
+ addopts = "-m 'not integration'"
18
+ markers = ["integration: calls networks; excluded by default"]
19
+
20
+ [tool.ruff]
21
+ line-length = 100
22
+ target-version = "py314"
23
+ [tool.ruff.lint]
24
+ select = ["E", "F", "I", "B", "UP", "SIM"]
25
+ ignore = ["E501"]
release.json ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model_id": "HashNuke/barge-in-detector",
3
+ "status": "experimental, uncalibrated, short-interruption failures",
4
+ "base_model": "intfloat/multilingual-e5-small",
5
+ "dataset_repo": "HashNuke/barge-in-detector-data",
6
+ "dataset_revision": "dda7115ace1fe7b07c49139860131f39910521ad",
7
+ "dataset_files_sha256": {
8
+ "data/export/utterances.csv": "556a2820e4a24ac13bab44321fb32b464c5fb225750d41a5a10ebcde7cdbab8a",
9
+ "data/splits/train.csv": "aab224bd0526eee915295e6f5c52076e29bb751cd0d664eefb4321843ad22533",
10
+ "data/splits/validation.csv": "c6b4c526fb4b1df96754119e409cedc8c7e85c77659e1fc1c3829da70084edc1",
11
+ "data/splits/test.csv": "8b630782362b05249b3be94b0213850258422be5ddeb62fcf703817b0bf9e11a",
12
+ "data/evaluation/english-challenge-v1.csv": "3d46b9e024f4d3bedd69f07603cde934e85ec2272988e90d2acebc30d5e99519",
13
+ "data/evaluation/reviews/english-challenge-v1.json": "ee6e07c3218ab0412d179e73b8e7457d550447a7ae08db2a005f37d794b855fa"
14
+ },
15
+ "artifacts_sha256": {
16
+ "model.safetensors": "bddb65c08f6260cfc9d320e65b4229229ec9f3ca95204aed98ab5ae45ff758d3",
17
+ "config.json": "e37f926b37b69e1ae223add18a9b0e993d6000cfe87c44f2dd42ce157654d93b",
18
+ "tokenizer.json": "255f5e32cb320b4dc2b31e0f203832fecafe87ca4cc87c3c41eb8ccae73f441a",
19
+ "tokenizer_config.json": "33b75086cc783ee5d78cab4dae0762d9cab1c40a8e0feb6e65d60c8d69a6afc1"
20
+ },
21
+ "original_training_source": {
22
+ "git_base_commit": "5ce0ed1283401f8c73bafcf5a7dbd2101c4bbee6",
23
+ "overlay_manifest": "reports/source-manifest.json"
24
+ },
25
+ "migration_source_git_commit": "1f92917a59908e320c3a3489c3e5de413a3e4672",
26
+ "labels": {
27
+ "0": "backchannel",
28
+ "1": "barge_in"
29
+ },
30
+ "max_length": 64,
31
+ "default_diagnostic_threshold": 0.5,
32
+ "selected_epoch": 5,
33
+ "training_metrics": "reports/training-metrics.json"
34
+ }
reports/challenge-predictions.csv ADDED
@@ -0,0 +1,99 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ case_id,category,group,language,label,text,text_clean,review_status,p_barge_in,prediction,seen_in
2
+ en-001,short_interruption,,en,barge_in,Stop,stop,pending,0.013443702831864357,backchannel,
3
+ en-002,short_interruption,,en,barge_in,Wait,wait,pending,0.013491915538907051,backchannel,
4
+ en-003,short_interruption,,en,barge_in,Pause,pause,pending,0.013399876654148102,backchannel,
5
+ en-004,short_interruption,,en,barge_in,Cancel,cancel,pending,0.0172202717512846,backchannel,
6
+ en-005,short_interruption,,en,barge_in,Wrong,wrong,pending,0.015773363411426544,backchannel,
7
+ en-006,short_interruption,,en,barge_in,Repeat,repeat,pending,0.01339809875935316,backchannel,
8
+ en-007,short_interruption,,en,barge_in,Slower,slower,pending,0.014412087388336658,backchannel,
9
+ en-008,short_interruption,,en,barge_in,Restart,restart,pending,0.014540360309183598,backchannel,
10
+ en-009,short_interruption,,en,barge_in,Enough,enough,pending,0.013300837948918343,backchannel,
11
+ en-010,short_interruption,,en,barge_in,Stop talking,stop talking,pending,0.01404176652431488,backchannel,
12
+ en-011,short_interruption,,en,barge_in,Wait please,wait please,pending,0.013346969150006771,backchannel,
13
+ en-012,short_interruption,,en,barge_in,Not that,not that,pending,0.015475512482225895,backchannel,
14
+ en-013,short_interruption,,en,barge_in,Wrong date,wrong date,pending,0.9878827333450317,barge_in,
15
+ en-014,short_interruption,,en,barge_in,Wrong person,wrong person,pending,0.8659051060676575,barge_in,
16
+ en-015,short_interruption,,en,barge_in,Go back,go back,pending,0.013336537405848503,backchannel,
17
+ en-016,short_interruption,,en,barge_in,Say again,say again,pending,0.013324333354830742,backchannel,
18
+ en-017,short_interruption,,en,barge_in,Speak slower,speak slower,pending,0.013362826779484749,backchannel,
19
+ en-018,short_interruption,,en,barge_in,Start over,start over,pending,0.01339060626924038,backchannel,
20
+ en-019,short_interruption,,en,barge_in,Let me finish,let me finish,pending,0.013296245597302914,backchannel,
21
+ en-020,short_interruption,,en,barge_in,You misunderstood,you misunderstood,pending,0.013550461269915104,backchannel,
22
+ en-021,short_interruption,,en,barge_in,Cancel it,cancel it,pending,0.015159462578594685,backchannel,
23
+ en-022,short_interruption,,en,barge_in,Change it,change it,pending,0.013769036158919334,backchannel,
24
+ en-023,short_interruption,,en,barge_in,Which one,which one,pending,0.017033280804753304,backchannel,
25
+ en-024,short_interruption,,en,barge_in,Why not,why not,pending,0.014318976551294327,backchannel,
26
+ en-025,short_interruption,,en,barge_in,How much,how much,pending,0.016760120168328285,backchannel,
27
+ en-026,short_interruption,,en,barge_in,What time,what time,pending,0.9822450876235962,barge_in,
28
+ en-027,short_interruption,,en,barge_in,What address,what address,pending,0.9860103130340576,barge_in,
29
+ en-028,short_interruption,,en,barge_in,Tuesday instead,tuesday instead,pending,0.7567923069000244,barge_in,
30
+ en-029,short_interruption,,en,barge_in,The second one,the second one,pending,0.02150067687034607,backchannel,
31
+ en-030,short_interruption,,en,barge_in,That is wrong,that is wrong,pending,0.1904229372739792,backchannel,
32
+ en-031,long_backchannel,,en,backchannel,Yes yes yes yes yes,yes yes yes yes yes,pending,0.013356473296880722,backchannel,
33
+ en-032,long_backchannel,,en,backchannel,Okay okay okay okay okay,okay okay okay okay okay,pending,0.013345411978662014,backchannel,
34
+ en-033,long_backchannel,,en,backchannel,Right right right I see,right right right i see,pending,0.013284565880894661,backchannel,
35
+ en-034,long_backchannel,,en,backchannel,Mm hmm yes mm hmm yes,mm hmm yes mm hmm yes,pending,0.013306752778589725,backchannel,
36
+ en-035,long_backchannel,,en,backchannel,Oh wow oh wow yeah,oh wow oh wow yeah,pending,0.013319982215762138,backchannel,
37
+ en-036,long_backchannel,,en,backchannel,Yeah I see I see,yeah i see i see,pending,0.013288560323417187,backchannel,
38
+ en-037,long_backchannel,,en,backchannel,Yes I understand what you mean,yes i understand what you mean,pending,0.0892898291349411,backchannel,
39
+ en-038,long_backchannel,,en,backchannel,Right that makes sense to me,right that makes sense to me,pending,0.013280664570629597,backchannel,
40
+ en-039,long_backchannel,,en,backchannel,Okay I am following you so far,okay i am following you so far,pending,0.28858309984207153,backchannel,
41
+ en-040,long_backchannel,,en,backchannel,Yes yes that sounds right,yes yes that sounds right,pending,0.013292225077748299,backchannel,
42
+ en-041,long_backchannel,,en,backchannel,Ah I see what you mean now,ah i see what you mean now,pending,0.32564374804496765,backchannel,
43
+ en-042,long_backchannel,,en,backchannel,Mm hmm I understand yes,mm hmm i understand yes,pending,0.0132778724655509,backchannel,
44
+ en-043,long_backchannel,,en,backchannel,Oh dear that sounds really difficult,oh dear that sounds really difficult,pending,0.047465287148952484,backchannel,
45
+ en-044,long_backchannel,,en,backchannel,Wow that must have been exciting,wow that must have been exciting,pending,0.013364902697503567,backchannel,
46
+ en-045,long_backchannel,,en,backchannel,That is interesting yes I see,that is interesting yes i see,pending,0.013278285041451454,backchannel,
47
+ en-046,long_backchannel,,en,backchannel,Fair enough I understand that,fair enough i understand that,pending,0.013284215703606606,backchannel,
48
+ en-047,long_backchannel,,en,backchannel,Yes absolutely I agree with that,yes absolutely i agree with that,pending,0.013294750824570656,backchannel,
49
+ en-048,long_backchannel,,en,backchannel,Right okay yes that makes sense,right okay yes that makes sense,pending,0.013285371474921703,backchannel,
50
+ en-049,long_backchannel,,en,backchannel,Oh no that is really sad,oh no that is really sad,pending,0.01329362578690052,backchannel,
51
+ en-050,long_backchannel,,en,backchannel,Ha ha that is funny,ha ha that is funny,pending,0.013284615240991116,backchannel,
52
+ en-051,long_backchannel,,en,backchannel,Mmm yes okay I hear you,mmm yes okay i hear you,pending,0.013274511322379112,backchannel,
53
+ en-052,long_backchannel,,en,backchannel,Sure sure I am with you,sure sure i am with you,pending,0.013305414468050003,backchannel,
54
+ en-053,long_backchannel,,en,backchannel,Got it yes understood,got it yes understood,pending,0.013286358676850796,backchannel,
55
+ en-054,long_backchannel,,en,backchannel,Uh huh uh huh right,uh huh uh huh right,pending,0.013300181366503239,backchannel,
56
+ en-055,ordinary_backchannel,,en,backchannel,Okay,okay,pending,0.013304519467055798,backchannel,
57
+ en-056,ordinary_backchannel,,en,backchannel,Yes,yes,pending,0.013361463323235512,backchannel,
58
+ en-057,ordinary_backchannel,,en,backchannel,Yeah,yeah,pending,0.013312126509845257,backchannel,
59
+ en-058,ordinary_backchannel,,en,backchannel,Mm hmm,mm hmm,pending,0.013299750164151192,backchannel,validation
60
+ en-059,ordinary_backchannel,,en,backchannel,Uh huh,uh huh,pending,0.013296402990818024,backchannel,train
61
+ en-060,ordinary_backchannel,,en,backchannel,Right,right,pending,0.013306460343301296,backchannel,
62
+ en-061,ordinary_backchannel,,en,backchannel,I see,i see,pending,0.013295376673340797,backchannel,
63
+ en-062,ordinary_backchannel,,en,backchannel,Got it,got it,pending,0.01331440731883049,backchannel,
64
+ en-063,ordinary_backchannel,,en,backchannel,Oh wow,oh wow,pending,0.013293537311255932,backchannel,train
65
+ en-064,ordinary_backchannel,,en,backchannel,I understand,i understand,pending,0.013287884183228016,backchannel,
66
+ en-065,ordinary_interruption,,en,barge_in,I need to change my delivery address,i need to change my delivery address,pending,0.9881365299224854,barge_in,
67
+ en-066,ordinary_interruption,,en,barge_in,You have the wrong customer on the line,you have the wrong customer on the line,pending,0.9880920648574829,barge_in,
68
+ en-067,ordinary_interruption,,en,barge_in,Can you explain what that fee is for,can you explain what that fee is for,pending,0.9879797697067261,barge_in,
69
+ en-068,ordinary_interruption,,en,barge_in,Please stop reading and let me speak,please stop reading and let me speak,pending,0.9880907535552979,barge_in,
70
+ en-069,ordinary_interruption,,en,barge_in,Actually I already paid that invoice,actually i already paid that invoice,pending,0.9881226420402527,barge_in,
71
+ en-070,ordinary_interruption,,en,barge_in,I want to speak to someone else,i want to speak to someone else,pending,0.9879534244537354,barge_in,
72
+ en-071,ordinary_interruption,,en,barge_in,No I said Thursday and not Tuesday,no i said thursday and not tuesday,pending,0.9881582856178284,barge_in,
73
+ en-072,ordinary_interruption,,en,barge_in,Before you continue I have a question,before you continue i have a question,pending,0.9879493713378906,barge_in,
74
+ en-073,ordinary_interruption,,en,barge_in,That is not what I was asking about,that is not what i was asking about,pending,0.9879181981086731,barge_in,
75
+ en-074,ordinary_interruption,,en,barge_in,I cannot hear you please say that again,i cannot hear you please say that again,pending,0.9881183505058289,barge_in,
76
+ en-075,prefix,prefix-1,en,backchannel,Yes,yes,pending,0.013361463323235512,backchannel,
77
+ en-076,prefix,prefix-1,en,backchannel,Yes yes,yes yes,pending,0.013298917561769485,backchannel,
78
+ en-077,prefix,prefix-1,en,barge_in,Yes yes but change the time,yes yes but change the time,pending,0.9880699515342712,barge_in,
79
+ en-078,prefix,prefix-2,en,backchannel,Okay,okay,pending,0.013304519467055798,backchannel,
80
+ en-079,prefix,prefix-2,en,backchannel,Okay okay,okay okay,pending,0.01330505684018135,backchannel,
81
+ en-080,prefix,prefix-2,en,barge_in,Okay okay wait that is the wrong address,okay okay wait that is the wrong address,pending,0.988145112991333,barge_in,
82
+ en-081,prefix,prefix-3,en,backchannel,Right,right,pending,0.013306460343301296,backchannel,
83
+ en-082,prefix,prefix-3,en,backchannel,Right right,right right,pending,0.01329006627202034,backchannel,train
84
+ en-083,prefix,prefix-3,en,barge_in,Right right but I meant the other account,right right but i meant the other account,pending,0.988115668296814,barge_in,
85
+ en-084,prefix,prefix-4,en,backchannel,Mm hmm,mm hmm,pending,0.013299750164151192,backchannel,validation
86
+ en-085,prefix,prefix-4,en,backchannel,Mm hmm yes,mm hmm yes,pending,0.013292054645717144,backchannel,
87
+ en-086,prefix,prefix-4,en,barge_in,Mm hmm yes can you repeat the price,mm hmm yes can you repeat the price,pending,0.9881290793418884,barge_in,
88
+ en-087,prefix,prefix-5,en,backchannel,I see,i see,pending,0.013295376673340797,backchannel,
89
+ en-088,prefix,prefix-5,en,backchannel,I see yes,i see yes,pending,0.013287578709423542,backchannel,
90
+ en-089,prefix,prefix-5,en,barge_in,I see yes but I need to cancel,i see yes but i need to cancel,pending,0.9879875183105469,barge_in,
91
+ en-090,prefix,prefix-6,en,backchannel,Sure,sure,pending,0.013307417742908001,backchannel,
92
+ en-091,prefix,prefix-6,en,backchannel,Sure sure,sure sure,pending,0.013290559872984886,backchannel,
93
+ en-092,prefix,prefix-6,en,barge_in,Sure sure stop there please,sure sure stop there please,pending,0.013294538483023643,backchannel,
94
+ en-093,prefix,prefix-7,en,backchannel,Got it,got it,pending,0.01331440731883049,backchannel,
95
+ en-094,prefix,prefix-7,en,backchannel,Got it okay,got it okay,pending,0.013302822597324848,backchannel,
96
+ en-095,prefix,prefix-7,en,barge_in,Got it okay now let me explain,got it okay now let me explain,pending,0.6444560885429382,barge_in,
97
+ en-096,prefix,prefix-8,en,backchannel,Yeah,yeah,pending,0.013312126509845257,backchannel,
98
+ en-097,prefix,prefix-8,en,backchannel,Yeah yeah,yeah yeah,pending,0.013376967050135136,backchannel,
99
+ en-098,prefix,prefix-8,en,barge_in,Yeah yeah no that will not work for me,yeah yeah no that will not work for me,pending,0.9878116250038147,barge_in,
reports/challenge-report.json ADDED
@@ -0,0 +1,438 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "challenge_sha256": "3d46b9e024f4d3bedd69f07603cde934e85ec2272988e90d2acebc30d5e99519",
3
+ "model_sha256": "bddb65c08f6260cfc9d320e65b4229229ec9f3ca95204aed98ab5ae45ff758d3",
4
+ "threshold": 0.5,
5
+ "runtime": {
6
+ "device": "cpu",
7
+ "torch": "2.15.0+cpu",
8
+ "transformers": "5.23.0",
9
+ "threads": 4
10
+ },
11
+ "batch_vs_single_max_probability_difference": 8.285045623779297e-06,
12
+ "batch_vs_single_label_disagreements": 0,
13
+ "review_status": "Owner reviewed all 98 cases; 13 subsequent revisions explicitly authorized; final binary labels match authored v1 labels. Diagnostic development set, not independent representative evaluation.",
14
+ "all": {
15
+ "count": 98,
16
+ "accuracy": 0.7346938775510204,
17
+ "macro_f1": 0.7111111111111111,
18
+ "false_interruption_rate": 0.0,
19
+ "missed_interruption_rate": 0.5416666666666666,
20
+ "per_class": {
21
+ "backchannel": {
22
+ "precision": 0.6578947368421053,
23
+ "recall": 1.0,
24
+ "f1": 0.7936507936507937,
25
+ "count": 50
26
+ },
27
+ "barge_in": {
28
+ "precision": 1.0,
29
+ "recall": 0.4583333333333333,
30
+ "f1": 0.6285714285714286,
31
+ "count": 48
32
+ }
33
+ },
34
+ "per_language": {
35
+ "en": {
36
+ "accuracy": 0.7346938775510204,
37
+ "count": 98
38
+ }
39
+ }
40
+ },
41
+ "by_category": {
42
+ "long_backchannel": {
43
+ "count": 24,
44
+ "accuracy": 1.0,
45
+ "macro_f1": 0.5,
46
+ "false_interruption_rate": 0.0,
47
+ "missed_interruption_rate": 0.0,
48
+ "per_class": {
49
+ "backchannel": {
50
+ "precision": 1.0,
51
+ "recall": 1.0,
52
+ "f1": 1.0,
53
+ "count": 24
54
+ },
55
+ "barge_in": {
56
+ "precision": 0.0,
57
+ "recall": 0.0,
58
+ "f1": 0.0,
59
+ "count": 0
60
+ }
61
+ },
62
+ "per_language": {
63
+ "en": {
64
+ "accuracy": 1.0,
65
+ "count": 24
66
+ }
67
+ }
68
+ },
69
+ "ordinary_backchannel": {
70
+ "count": 10,
71
+ "accuracy": 1.0,
72
+ "macro_f1": 0.5,
73
+ "false_interruption_rate": 0.0,
74
+ "missed_interruption_rate": 0.0,
75
+ "per_class": {
76
+ "backchannel": {
77
+ "precision": 1.0,
78
+ "recall": 1.0,
79
+ "f1": 1.0,
80
+ "count": 10
81
+ },
82
+ "barge_in": {
83
+ "precision": 0.0,
84
+ "recall": 0.0,
85
+ "f1": 0.0,
86
+ "count": 0
87
+ }
88
+ },
89
+ "per_language": {
90
+ "en": {
91
+ "accuracy": 1.0,
92
+ "count": 10
93
+ }
94
+ }
95
+ },
96
+ "ordinary_interruption": {
97
+ "count": 10,
98
+ "accuracy": 1.0,
99
+ "macro_f1": 0.5,
100
+ "false_interruption_rate": 0.0,
101
+ "missed_interruption_rate": 0.0,
102
+ "per_class": {
103
+ "backchannel": {
104
+ "precision": 0.0,
105
+ "recall": 0.0,
106
+ "f1": 0.0,
107
+ "count": 0
108
+ },
109
+ "barge_in": {
110
+ "precision": 1.0,
111
+ "recall": 1.0,
112
+ "f1": 1.0,
113
+ "count": 10
114
+ }
115
+ },
116
+ "per_language": {
117
+ "en": {
118
+ "accuracy": 1.0,
119
+ "count": 10
120
+ }
121
+ }
122
+ },
123
+ "prefix": {
124
+ "count": 24,
125
+ "accuracy": 0.9583333333333334,
126
+ "macro_f1": 0.9515151515151515,
127
+ "false_interruption_rate": 0.0,
128
+ "missed_interruption_rate": 0.125,
129
+ "per_class": {
130
+ "backchannel": {
131
+ "precision": 0.9411764705882353,
132
+ "recall": 1.0,
133
+ "f1": 0.9696969696969697,
134
+ "count": 16
135
+ },
136
+ "barge_in": {
137
+ "precision": 1.0,
138
+ "recall": 0.875,
139
+ "f1": 0.9333333333333333,
140
+ "count": 8
141
+ }
142
+ },
143
+ "per_language": {
144
+ "en": {
145
+ "accuracy": 0.9583333333333334,
146
+ "count": 24
147
+ }
148
+ }
149
+ },
150
+ "short_interruption": {
151
+ "count": 30,
152
+ "accuracy": 0.16666666666666666,
153
+ "macro_f1": 0.14285714285714285,
154
+ "false_interruption_rate": 0.0,
155
+ "missed_interruption_rate": 0.8333333333333334,
156
+ "per_class": {
157
+ "backchannel": {
158
+ "precision": 0.0,
159
+ "recall": 0.0,
160
+ "f1": 0.0,
161
+ "count": 0
162
+ },
163
+ "barge_in": {
164
+ "precision": 1.0,
165
+ "recall": 0.16666666666666666,
166
+ "f1": 0.2857142857142857,
167
+ "count": 30
168
+ }
169
+ },
170
+ "per_language": {
171
+ "en": {
172
+ "accuracy": 0.16666666666666666,
173
+ "count": 30
174
+ }
175
+ }
176
+ }
177
+ },
178
+ "overlap_counts": {
179
+ "train": 3,
180
+ "validation": 2,
181
+ "test": 0
182
+ },
183
+ "cross_split_exact_overlap": {
184
+ "train/validation": 0,
185
+ "train/test": 0,
186
+ "validation/test": 0
187
+ },
188
+ "unseen": {
189
+ "count": 93,
190
+ "accuracy": 0.7204301075268817,
191
+ "macro_f1": 0.702216748768473,
192
+ "false_interruption_rate": 0.0,
193
+ "missed_interruption_rate": 0.5416666666666666,
194
+ "per_class": {
195
+ "backchannel": {
196
+ "precision": 0.6338028169014085,
197
+ "recall": 1.0,
198
+ "f1": 0.7758620689655172,
199
+ "count": 45
200
+ },
201
+ "barge_in": {
202
+ "precision": 1.0,
203
+ "recall": 0.4583333333333333,
204
+ "f1": 0.6285714285714286,
205
+ "count": 48
206
+ }
207
+ },
208
+ "per_language": {
209
+ "en": {
210
+ "accuracy": 0.7204301075268817,
211
+ "count": 93
212
+ }
213
+ }
214
+ },
215
+ "prefix_groups": {
216
+ "prefix-1": [
217
+ {
218
+ "text": "yes",
219
+ "label": "backchannel",
220
+ "p_barge_in": 0.013361463323235512
221
+ },
222
+ {
223
+ "text": "yes yes",
224
+ "label": "backchannel",
225
+ "p_barge_in": 0.013298917561769485
226
+ },
227
+ {
228
+ "text": "yes yes but change the time",
229
+ "label": "barge_in",
230
+ "p_barge_in": 0.9880699515342712
231
+ }
232
+ ],
233
+ "prefix-2": [
234
+ {
235
+ "text": "okay",
236
+ "label": "backchannel",
237
+ "p_barge_in": 0.013304519467055798
238
+ },
239
+ {
240
+ "text": "okay okay",
241
+ "label": "backchannel",
242
+ "p_barge_in": 0.01330505684018135
243
+ },
244
+ {
245
+ "text": "okay okay wait that is the wrong address",
246
+ "label": "barge_in",
247
+ "p_barge_in": 0.988145112991333
248
+ }
249
+ ],
250
+ "prefix-3": [
251
+ {
252
+ "text": "right",
253
+ "label": "backchannel",
254
+ "p_barge_in": 0.013306460343301296
255
+ },
256
+ {
257
+ "text": "right right",
258
+ "label": "backchannel",
259
+ "p_barge_in": 0.01329006627202034
260
+ },
261
+ {
262
+ "text": "right right but i meant the other account",
263
+ "label": "barge_in",
264
+ "p_barge_in": 0.988115668296814
265
+ }
266
+ ],
267
+ "prefix-4": [
268
+ {
269
+ "text": "mm hmm",
270
+ "label": "backchannel",
271
+ "p_barge_in": 0.013299750164151192
272
+ },
273
+ {
274
+ "text": "mm hmm yes",
275
+ "label": "backchannel",
276
+ "p_barge_in": 0.013292054645717144
277
+ },
278
+ {
279
+ "text": "mm hmm yes can you repeat the price",
280
+ "label": "barge_in",
281
+ "p_barge_in": 0.9881290793418884
282
+ }
283
+ ],
284
+ "prefix-5": [
285
+ {
286
+ "text": "i see",
287
+ "label": "backchannel",
288
+ "p_barge_in": 0.013295376673340797
289
+ },
290
+ {
291
+ "text": "i see yes",
292
+ "label": "backchannel",
293
+ "p_barge_in": 0.013287578709423542
294
+ },
295
+ {
296
+ "text": "i see yes but i need to cancel",
297
+ "label": "barge_in",
298
+ "p_barge_in": 0.9879875183105469
299
+ }
300
+ ],
301
+ "prefix-6": [
302
+ {
303
+ "text": "sure",
304
+ "label": "backchannel",
305
+ "p_barge_in": 0.013307417742908001
306
+ },
307
+ {
308
+ "text": "sure sure",
309
+ "label": "backchannel",
310
+ "p_barge_in": 0.013290559872984886
311
+ },
312
+ {
313
+ "text": "sure sure stop there please",
314
+ "label": "barge_in",
315
+ "p_barge_in": 0.013294538483023643
316
+ }
317
+ ],
318
+ "prefix-7": [
319
+ {
320
+ "text": "got it",
321
+ "label": "backchannel",
322
+ "p_barge_in": 0.01331440731883049
323
+ },
324
+ {
325
+ "text": "got it okay",
326
+ "label": "backchannel",
327
+ "p_barge_in": 0.013302822597324848
328
+ },
329
+ {
330
+ "text": "got it okay now let me explain",
331
+ "label": "barge_in",
332
+ "p_barge_in": 0.6444560885429382
333
+ }
334
+ ],
335
+ "prefix-8": [
336
+ {
337
+ "text": "yeah",
338
+ "label": "backchannel",
339
+ "p_barge_in": 0.013312126509845257
340
+ },
341
+ {
342
+ "text": "yeah yeah",
343
+ "label": "backchannel",
344
+ "p_barge_in": 0.013376967050135136
345
+ },
346
+ {
347
+ "text": "yeah yeah no that will not work for me",
348
+ "label": "barge_in",
349
+ "p_barge_in": 0.9878116250038147
350
+ }
351
+ ]
352
+ },
353
+ "prefix_groups_all_labels_correct": 7,
354
+ "english_length_baseline": {
355
+ "cutoff_words": 6,
356
+ "validation": {
357
+ "count": 81,
358
+ "accuracy": 0.9753086419753086,
359
+ "macro_f1": 0.9716783216783216,
360
+ "false_interruption_rate": 0.07407407407407407,
361
+ "missed_interruption_rate": 0.0,
362
+ "per_class": {
363
+ "backchannel": {
364
+ "precision": 1.0,
365
+ "recall": 0.9259259259259259,
366
+ "f1": 0.9615384615384615,
367
+ "count": 27
368
+ },
369
+ "barge_in": {
370
+ "precision": 0.9642857142857143,
371
+ "recall": 1.0,
372
+ "f1": 0.9818181818181818,
373
+ "count": 54
374
+ }
375
+ },
376
+ "per_language": {
377
+ "en": {
378
+ "accuracy": 0.9753086419753086,
379
+ "count": 81
380
+ }
381
+ }
382
+ },
383
+ "test": {
384
+ "count": 81,
385
+ "accuracy": 0.9382716049382716,
386
+ "macro_f1": 0.9268556980314249,
387
+ "false_interruption_rate": 0.18518518518518517,
388
+ "missed_interruption_rate": 0.0,
389
+ "per_class": {
390
+ "backchannel": {
391
+ "precision": 1.0,
392
+ "recall": 0.8148148148148148,
393
+ "f1": 0.8979591836734693,
394
+ "count": 27
395
+ },
396
+ "barge_in": {
397
+ "precision": 0.9152542372881356,
398
+ "recall": 1.0,
399
+ "f1": 0.9557522123893806,
400
+ "count": 54
401
+ }
402
+ },
403
+ "per_language": {
404
+ "en": {
405
+ "accuracy": 0.9382716049382716,
406
+ "count": 81
407
+ }
408
+ }
409
+ },
410
+ "challenge": {
411
+ "count": 98,
412
+ "accuracy": 0.5510204081632653,
413
+ "macro_f1": 0.5315080399826162,
414
+ "false_interruption_rate": 0.26,
415
+ "missed_interruption_rate": 0.6458333333333334,
416
+ "per_class": {
417
+ "backchannel": {
418
+ "precision": 0.5441176470588235,
419
+ "recall": 0.74,
420
+ "f1": 0.6271186440677965,
421
+ "count": 50
422
+ },
423
+ "barge_in": {
424
+ "precision": 0.5666666666666667,
425
+ "recall": 0.3541666666666667,
426
+ "f1": 0.4358974358974359,
427
+ "count": 48
428
+ }
429
+ },
430
+ "per_language": {
431
+ "en": {
432
+ "accuracy": 0.5510204081632653,
433
+ "count": 98
434
+ }
435
+ }
436
+ }
437
+ }
438
+ }
reports/cpu-benchmark.json ADDED
@@ -0,0 +1,94 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model": "models/e5-small-run1",
3
+ "runtime": "PyTorch eager fp32, CPU only",
4
+ "model_sha256": "bddb65c08f6260cfc9d320e65b4229229ec9f3ca95204aed98ab5ae45ff758d3",
5
+ "python": "3.14.4",
6
+ "torch": "2.15.0+cpu",
7
+ "transformers": "5.23.0",
8
+ "platform": "Linux-7.0.0-43-generic-x86_64-with-glibc2.43",
9
+ "cpuinfo": "AMD EPYC-Genoa Processor",
10
+ "load_seconds_excluded": 2.8975897090276703,
11
+ "iterations": 1000,
12
+ "warmup_per_scenario": 100,
13
+ "batch_size": 1,
14
+ "max_length": 64,
15
+ "interop_threads": 1,
16
+ "parameter_bytes": 470618120,
17
+ "corpus_sha256": "8b630782362b05249b3be94b0213850258422be5ddeb62fcf703817b0bf9e11a",
18
+ "corpus_rows": 331,
19
+ "languages": [
20
+ "de",
21
+ "en",
22
+ "es",
23
+ "fil",
24
+ "fr",
25
+ "hi",
26
+ "id",
27
+ "it",
28
+ "ja",
29
+ "kn",
30
+ "ml",
31
+ "ta",
32
+ "te",
33
+ "zh"
34
+ ],
35
+ "measurements": [
36
+ {
37
+ "threads": 4,
38
+ "scenario": "multilingual_test",
39
+ "model_path": {
40
+ "p50_ms": 13.088757,
41
+ "p95_ms": 19.419374299999998,
42
+ "p99_ms": 25.316384159999995,
43
+ "mean_ms": 13.625100593,
44
+ "max_ms": 83.599
45
+ },
46
+ "tokenization": {
47
+ "p50_ms": 0.222889,
48
+ "p95_ms": 0.30127505,
49
+ "p99_ms": 0.45513418999999933,
50
+ "mean_ms": 0.23647671,
51
+ "max_ms": 2.407625
52
+ },
53
+ "forward_and_probability": {
54
+ "p50_ms": 12.8741705,
55
+ "p95_ms": 19.1815334,
56
+ "p99_ms": 25.09679618,
57
+ "mean_ms": 13.388623883,
58
+ "max_ms": 83.391099
59
+ }
60
+ },
61
+ {
62
+ "threads": 4,
63
+ "scenario": "full_64_tokens",
64
+ "model_path": {
65
+ "p50_ms": 26.794403000000003,
66
+ "p95_ms": 33.269715649999995,
67
+ "p99_ms": 37.862168669999996,
68
+ "mean_ms": 27.779209678,
69
+ "max_ms": 93.899777
70
+ },
71
+ "tokenization": {
72
+ "p50_ms": 0.6677489999999999,
73
+ "p95_ms": 0.8626962999999997,
74
+ "p99_ms": 1.7089376799999993,
75
+ "mean_ms": 0.704945851,
76
+ "max_ms": 7.957467
77
+ },
78
+ "forward_and_probability": {
79
+ "p50_ms": 26.103834499999998,
80
+ "p95_ms": 32.575319,
81
+ "p99_ms": 37.10665824,
82
+ "mean_ms": 27.074263827,
83
+ "max_ms": 93.297535
84
+ }
85
+ }
86
+ ],
87
+ "scope": "Local virtual CPU; reference CPU unspecified. Excludes Vxpipe boundary and queueing; no whole-path acceptance claim.",
88
+ "corpus_token_lengths": {
89
+ "p50": 13.0,
90
+ "p95": 19.0,
91
+ "max": 26
92
+ },
93
+ "peak_process_rss_mib": 1163.6171875
94
+ }
reports/cpu-cold-benchmark.json ADDED
@@ -0,0 +1,33 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "cold_process_seconds": [
3
+ 4.768154836,
4
+ 4.810785561,
5
+ 4.136632718,
6
+ 4.255589102,
7
+ 3.847660745,
8
+ 4.045793715,
9
+ 3.973606716,
10
+ 4.190341404,
11
+ 4.387973674,
12
+ 4.28136222
13
+ ],
14
+ "loaded_to_first_label_ms": [
15
+ 26.506046,
16
+ 40.193806,
17
+ 18.425675,
18
+ 22.50301,
19
+ 23.081508,
20
+ 40.155708,
21
+ 16.969254,
22
+ 23.919822,
23
+ 23.060476,
24
+ 24.334184
25
+ ],
26
+ "loaded_to_first_label_median_ms": 23.500664999999998,
27
+ "median_seconds": 4.222965253,
28
+ "min_seconds": 3.847660745,
29
+ "max_seconds": 4.810785561,
30
+ "runs": 10,
31
+ "threads": 4,
32
+ "scope": "Python process creation through reading first flushed label. Includes imports, tokenizer/model load, first inference. Local files only; OS file cache may be warm. Excludes process shutdown, downloads and HTTP/network boundary."
33
+ }
reports/cpu-thread-sweep.json ADDED
@@ -0,0 +1,194 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model": "models/e5-small-run1",
3
+ "runtime": "PyTorch eager fp32, CPU only",
4
+ "model_sha256": "bddb65c08f6260cfc9d320e65b4229229ec9f3ca95204aed98ab5ae45ff758d3",
5
+ "python": "3.14.4",
6
+ "torch": "2.15.0+cpu",
7
+ "transformers": "5.23.0",
8
+ "platform": "Linux-7.0.0-43-generic-x86_64-with-glibc2.43",
9
+ "cpuinfo": "AMD EPYC-Genoa Processor",
10
+ "load_seconds_excluded": 3.1383294460829347,
11
+ "iterations": 1000,
12
+ "warmup_per_scenario": 100,
13
+ "batch_size": 1,
14
+ "max_length": 64,
15
+ "interop_threads": 1,
16
+ "parameter_bytes": 470618120,
17
+ "corpus_sha256": "8b630782362b05249b3be94b0213850258422be5ddeb62fcf703817b0bf9e11a",
18
+ "corpus_rows": 331,
19
+ "languages": [
20
+ "de",
21
+ "en",
22
+ "es",
23
+ "fil",
24
+ "fr",
25
+ "hi",
26
+ "id",
27
+ "it",
28
+ "ja",
29
+ "kn",
30
+ "ml",
31
+ "ta",
32
+ "te",
33
+ "zh"
34
+ ],
35
+ "corpus_token_lengths": {
36
+ "p50": 13.0,
37
+ "p95": 19.0,
38
+ "max": 26
39
+ },
40
+ "measurements": [
41
+ {
42
+ "threads": 1,
43
+ "scenario": "multilingual_test",
44
+ "model_path": {
45
+ "p50_ms": 20.395149500000002,
46
+ "p95_ms": 25.477123599999995,
47
+ "p99_ms": 28.2604041,
48
+ "mean_ms": 20.432651312,
49
+ "max_ms": 46.969913
50
+ },
51
+ "tokenization": {
52
+ "p50_ms": 0.23074050000000002,
53
+ "p95_ms": 0.29089349999999997,
54
+ "p99_ms": 0.33883417,
55
+ "mean_ms": 0.234400798,
56
+ "max_ms": 0.640549
57
+ },
58
+ "forward_and_probability": {
59
+ "p50_ms": 20.162170500000002,
60
+ "p95_ms": 25.207171749999997,
61
+ "p99_ms": 28.02515483,
62
+ "mean_ms": 20.198250514,
63
+ "max_ms": 46.709714
64
+ }
65
+ },
66
+ {
67
+ "threads": 1,
68
+ "scenario": "full_64_tokens",
69
+ "model_path": {
70
+ "p50_ms": 44.6955575,
71
+ "p95_ms": 50.545173899999995,
72
+ "p99_ms": 53.76246217,
73
+ "mean_ms": 45.379929543,
74
+ "max_ms": 57.243489
75
+ },
76
+ "tokenization": {
77
+ "p50_ms": 0.6161775,
78
+ "p95_ms": 0.7613426,
79
+ "p99_ms": 0.84228681,
80
+ "mean_ms": 0.631765294,
81
+ "max_ms": 0.891216
82
+ },
83
+ "forward_and_probability": {
84
+ "p50_ms": 44.084312499999996,
85
+ "p95_ms": 49.9083266,
86
+ "p99_ms": 53.07674325,
87
+ "mean_ms": 44.748164249,
88
+ "max_ms": 56.405244
89
+ }
90
+ },
91
+ {
92
+ "threads": 2,
93
+ "scenario": "multilingual_test",
94
+ "model_path": {
95
+ "p50_ms": 19.406584000000002,
96
+ "p95_ms": 23.1596329,
97
+ "p99_ms": 25.6503671,
98
+ "mean_ms": 19.536068109,
99
+ "max_ms": 28.630568
100
+ },
101
+ "tokenization": {
102
+ "p50_ms": 0.23920799999999998,
103
+ "p95_ms": 0.31088855,
104
+ "p99_ms": 0.36521733999999995,
105
+ "mean_ms": 0.244807045,
106
+ "max_ms": 0.475011
107
+ },
108
+ "forward_and_probability": {
109
+ "p50_ms": 19.154210499999998,
110
+ "p95_ms": 22.828605549999995,
111
+ "p99_ms": 25.33145532,
112
+ "mean_ms": 19.291261064,
113
+ "max_ms": 28.31147
114
+ }
115
+ },
116
+ {
117
+ "threads": 2,
118
+ "scenario": "full_64_tokens",
119
+ "model_path": {
120
+ "p50_ms": 38.023134999999996,
121
+ "p95_ms": 45.51507395,
122
+ "p99_ms": 53.98291254,
123
+ "mean_ms": 38.847798787,
124
+ "max_ms": 62.944328
125
+ },
126
+ "tokenization": {
127
+ "p50_ms": 0.636223,
128
+ "p95_ms": 0.8138672499999998,
129
+ "p99_ms": 0.9455087999999999,
130
+ "mean_ms": 0.659222932,
131
+ "max_ms": 1.44213
132
+ },
133
+ "forward_and_probability": {
134
+ "p50_ms": 37.383997,
135
+ "p95_ms": 44.81712879999999,
136
+ "p99_ms": 53.05854066,
137
+ "mean_ms": 38.188575855,
138
+ "max_ms": 61.995607
139
+ }
140
+ },
141
+ {
142
+ "threads": 4,
143
+ "scenario": "multilingual_test",
144
+ "model_path": {
145
+ "p50_ms": 13.1820115,
146
+ "p95_ms": 19.21180095,
147
+ "p99_ms": 24.81463212999999,
148
+ "mean_ms": 13.731945525,
149
+ "max_ms": 89.107149
150
+ },
151
+ "tokenization": {
152
+ "p50_ms": 0.2276755,
153
+ "p95_ms": 0.30882,
154
+ "p99_ms": 0.3798859499999996,
155
+ "mean_ms": 0.23842783,
156
+ "max_ms": 4.009645
157
+ },
158
+ "forward_and_probability": {
159
+ "p50_ms": 12.937595,
160
+ "p95_ms": 18.841387699999995,
161
+ "p99_ms": 24.56872725999999,
162
+ "mean_ms": 13.493517695,
163
+ "max_ms": 88.911056
164
+ }
165
+ },
166
+ {
167
+ "threads": 4,
168
+ "scenario": "full_64_tokens",
169
+ "model_path": {
170
+ "p50_ms": 26.392652,
171
+ "p95_ms": 32.489633899999994,
172
+ "p99_ms": 41.04232582999999,
173
+ "mean_ms": 27.24307507,
174
+ "max_ms": 93.673078
175
+ },
176
+ "tokenization": {
177
+ "p50_ms": 0.614555,
178
+ "p95_ms": 0.7339711499999999,
179
+ "p99_ms": 0.9166563899999999,
180
+ "mean_ms": 0.624646764,
181
+ "max_ms": 2.092334
182
+ },
183
+ "forward_and_probability": {
184
+ "p50_ms": 25.7561225,
185
+ "p95_ms": 31.824941300000003,
186
+ "p99_ms": 40.46172110999999,
187
+ "mean_ms": 26.618428306,
188
+ "max_ms": 93.090937
189
+ }
190
+ }
191
+ ],
192
+ "scope": "Local virtual CPU; reference CPU unspecified. Excludes Vxpipe boundary and queueing; no whole-path acceptance claim.",
193
+ "peak_process_rss_mib": 1168.8125
194
+ }
reports/eval-test.json ADDED
@@ -0,0 +1,81 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "count": 331,
3
+ "accuracy": 1.0,
4
+ "macro_f1": 1.0,
5
+ "false_interruption_rate": 0.0,
6
+ "missed_interruption_rate": 0.0,
7
+ "per_class": {
8
+ "backchannel": {
9
+ "precision": 1.0,
10
+ "recall": 1.0,
11
+ "f1": 1.0,
12
+ "count": 109
13
+ },
14
+ "barge_in": {
15
+ "precision": 1.0,
16
+ "recall": 1.0,
17
+ "f1": 1.0,
18
+ "count": 222
19
+ }
20
+ },
21
+ "per_language": {
22
+ "de": {
23
+ "accuracy": 1.0,
24
+ "count": 17
25
+ },
26
+ "en": {
27
+ "accuracy": 1.0,
28
+ "count": 81
29
+ },
30
+ "es": {
31
+ "accuracy": 1.0,
32
+ "count": 36
33
+ },
34
+ "fil": {
35
+ "accuracy": 1.0,
36
+ "count": 18
37
+ },
38
+ "fr": {
39
+ "accuracy": 1.0,
40
+ "count": 18
41
+ },
42
+ "hi": {
43
+ "accuracy": 1.0,
44
+ "count": 18
45
+ },
46
+ "id": {
47
+ "accuracy": 1.0,
48
+ "count": 17
49
+ },
50
+ "it": {
51
+ "accuracy": 1.0,
52
+ "count": 18
53
+ },
54
+ "ja": {
55
+ "accuracy": 1.0,
56
+ "count": 18
57
+ },
58
+ "kn": {
59
+ "accuracy": 1.0,
60
+ "count": 18
61
+ },
62
+ "ml": {
63
+ "accuracy": 1.0,
64
+ "count": 18
65
+ },
66
+ "ta": {
67
+ "accuracy": 1.0,
68
+ "count": 18
69
+ },
70
+ "te": {
71
+ "accuracy": 1.0,
72
+ "count": 18
73
+ },
74
+ "zh": {
75
+ "accuracy": 1.0,
76
+ "count": 18
77
+ }
78
+ },
79
+ "split": "test",
80
+ "threshold": 0.5
81
+ }
reports/eval-validation.json ADDED
@@ -0,0 +1,81 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "count": 333,
3
+ "accuracy": 0.993993993993994,
4
+ "macro_f1": 0.9932432432432432,
5
+ "false_interruption_rate": 0.009009009009009009,
6
+ "missed_interruption_rate": 0.0045045045045045045,
7
+ "per_class": {
8
+ "backchannel": {
9
+ "precision": 0.990990990990991,
10
+ "recall": 0.990990990990991,
11
+ "f1": 0.990990990990991,
12
+ "count": 111
13
+ },
14
+ "barge_in": {
15
+ "precision": 0.9954954954954955,
16
+ "recall": 0.9954954954954955,
17
+ "f1": 0.9954954954954955,
18
+ "count": 222
19
+ }
20
+ },
21
+ "per_language": {
22
+ "de": {
23
+ "accuracy": 1.0,
24
+ "count": 18
25
+ },
26
+ "en": {
27
+ "accuracy": 1.0,
28
+ "count": 81
29
+ },
30
+ "es": {
31
+ "accuracy": 1.0,
32
+ "count": 36
33
+ },
34
+ "fil": {
35
+ "accuracy": 1.0,
36
+ "count": 18
37
+ },
38
+ "fr": {
39
+ "accuracy": 1.0,
40
+ "count": 18
41
+ },
42
+ "hi": {
43
+ "accuracy": 1.0,
44
+ "count": 18
45
+ },
46
+ "id": {
47
+ "accuracy": 1.0,
48
+ "count": 18
49
+ },
50
+ "it": {
51
+ "accuracy": 1.0,
52
+ "count": 18
53
+ },
54
+ "ja": {
55
+ "accuracy": 1.0,
56
+ "count": 18
57
+ },
58
+ "kn": {
59
+ "accuracy": 1.0,
60
+ "count": 18
61
+ },
62
+ "ml": {
63
+ "accuracy": 1.0,
64
+ "count": 18
65
+ },
66
+ "ta": {
67
+ "accuracy": 0.9444444444444444,
68
+ "count": 18
69
+ },
70
+ "te": {
71
+ "accuracy": 1.0,
72
+ "count": 18
73
+ },
74
+ "zh": {
75
+ "accuracy": 0.9444444444444444,
76
+ "count": 18
77
+ }
78
+ },
79
+ "split": "validation",
80
+ "threshold": 0.5
81
+ }
tests/test_data.py ADDED
@@ -0,0 +1,43 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import sys
2
+ from types import SimpleNamespace
3
+
4
+ import pytest
5
+
6
+ from barge_in_detector_model.data import read_csv, resolve_splits
7
+
8
+
9
+ def make_splits(path):
10
+ path.mkdir(parents=True)
11
+ for name in ["train", "validation", "test"]:
12
+ (path / f"{name}.csv").write_text("text_clean,label,language\nstop,barge_in,en\n")
13
+
14
+
15
+ def test_local_splits_do_not_require_hub(tmp_path):
16
+ make_splits(tmp_path / "splits")
17
+ directory, revision = resolve_splits(tmp_path / "splits")
18
+ assert revision is None
19
+ assert read_csv(directory / "test.csv")[0]["label"] == "barge_in"
20
+ with pytest.raises(ValueError, match="missing"):
21
+ resolve_splits(tmp_path / "absent")
22
+
23
+
24
+ def test_hub_download_uses_resolved_immutable_revision(tmp_path, monkeypatch):
25
+ make_splits(tmp_path / "data/splits")
26
+ seen = {}
27
+
28
+ def download(**kwargs):
29
+ seen.update(kwargs)
30
+ return str(tmp_path)
31
+
32
+ fake = SimpleNamespace(
33
+ HfApi=lambda: SimpleNamespace(
34
+ dataset_info=lambda *a, **k: SimpleNamespace(sha="fixed-commit")
35
+ ),
36
+ snapshot_download=download,
37
+ )
38
+ monkeypatch.setitem(sys.modules, "huggingface_hub", fake)
39
+ directory, revision = resolve_splits(None, repo_id="owner/data", revision="v1")
40
+ assert revision == "fixed-commit"
41
+ assert seen["revision"] == "fixed-commit"
42
+ assert seen["repo_type"] == "dataset"
43
+ assert directory == tmp_path / "data/splits"
tests/test_metrics.py ADDED
@@ -0,0 +1,29 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Evaluation metrics for the two labels."""
2
+
3
+ import pytest
4
+
5
+ from barge_in_detector_model.metrics import compute
6
+
7
+
8
+ def test_metrics_report_false_and_missed_interruptions():
9
+ labels = ["backchannel", "backchannel", "barge_in", "barge_in", "barge_in"]
10
+ preds = ["backchannel", "barge_in", "barge_in", "barge_in", "backchannel"]
11
+ languages = ["en", "en", "en", "hi", "hi"]
12
+ m = compute(labels, preds, languages)
13
+ assert m["accuracy"] == pytest.approx(3 / 5)
14
+ # A backchannel predicted as barge-in cuts the agent off for nothing.
15
+ assert m["false_interruption_rate"] == pytest.approx(1 / 2)
16
+ # A barge-in predicted as backchannel lets the agent talk over the caller.
17
+ assert m["missed_interruption_rate"] == pytest.approx(1 / 3)
18
+ assert m["per_class"]["barge_in"]["precision"] == pytest.approx(2 / 3)
19
+ assert m["per_class"]["barge_in"]["recall"] == pytest.approx(2 / 3)
20
+ assert m["per_language"]["en"]["accuracy"] == pytest.approx(2 / 3)
21
+ assert m["per_language"]["hi"]["count"] == 2
22
+ assert 0 < m["macro_f1"] < 1
23
+
24
+
25
+ def test_perfect_predictions():
26
+ labels = ["backchannel", "barge_in"]
27
+ m = compute(labels, labels, ["en", "en"])
28
+ assert m["accuracy"] == m["macro_f1"] == 1.0
29
+ assert m["false_interruption_rate"] == m["missed_interruption_rate"] == 0.0
uv.lock ADDED
The diff for this file is too large to render. See raw diff