Akash Manohar commited on
Commit
e5336e8
·
1 Parent(s): a12cfe5

Keep only model release artifacts and provenance

Browse files

Move training, inference, evaluation, benchmarks, tests and labnotes to GitHub. Preserve weights/tokenizer hashes and pin the release-only dataset and canonical source revisions.

.python-version DELETED
@@ -1 +0,0 @@
1
- 3.14
 
 
AGENTS.md CHANGED
@@ -1,95 +1,19 @@
1
- # Barge-in detector model
2
-
3
- ## Ownership
4
-
5
- This Hugging Face model repository (`HashNuke/barge-in-detector`) owns weights,
6
- tokenizer, training, inference, evaluation, model experiments and CPU benchmarks.
7
- The dataset repository (`HashNuke/barge-in-detector-data`) owns generation,
8
- cleaning, source utterances, splits, review UI and historical project labnotes.
9
- Do not copy training data into this repo. Fetch a dataset revision or pass local
10
- splits. Record the resolved dataset commit, base model, seed, hyperparameters,
11
- source commit and metrics for every release. Never train on challenge examples.
12
-
13
- ## Contract and quality gates
14
-
15
- - Caller utterance spoken over an ongoing voice-agent turn; cleaned lowercase
16
- text without punctuation. No audio, agent line or language hint is required.
17
- - Exactly two labels: `backchannel` (continue speaking) and `barge_in` (yield).
18
- Repetitions, acknowledgments, surprise, empathy and laughter are backchannels
19
- unless the text develops into a correction, question, request or floor claim.
20
- Context and an ambiguous/needs-context class are unavailable.
21
- - At most 64 tokens; keep the first 64. Evaluate partial transcripts and
22
- transitions from backchannel to interruption. No universal ignored-word list.
23
- - Output `p_barge_in` in [0,1]. Current experimental probabilities are not yet
24
- calibrated; thresholds belong to the consumer. Use validation for tuning.
25
- - CPU inference only. Warmed batch-one model-path p99 <=20 ms and whole path
26
- including Vxpipe boundary <=50 ms on the designated reference CPU. Report
27
- p50/p95/p99, hardware, threads, memory, warm-up and token lengths. Local VM
28
- measurements do not satisfy reference-CPU acceptance gates.
29
- - Fourteen languages: English (US/India/Singapore), Hindi, Kannada, Tamil,
30
- Telugu, Malayalam, Mandarin, Japanese, Indonesian, Filipino, French, German,
31
- Spanish and Italian. Roll out English/Hindi first, then Kannada/Tamil.
32
- Native scripts, optionally mixed with English; no romanization/pinyin.
33
- Language support requires evaluation evidence, not tokenizer coverage.
34
- - Tier A owner verification: English, Hindi, Kannada, Tamil. Other languages
35
- remain synthetic/unverified unless native-speaker checks are recorded.
36
- - Report false interruptions and missed interruptions, per-language results,
37
- short-input slices, prefix behaviour and calibration. Keep synthetic and
38
- owner-reviewed evidence separate; a perfect synthetic score is insufficient.
39
- - Prefer permissive base licences. Preserve the MIT attribution for the
40
- intfloat/multilingual-e5-small base model.
41
-
42
- ## Tools and remote training
43
-
44
- - Use uv; add dependencies with `uv add` and keep uv.lock with dependency edits.
45
- Always source ~/.zshrc or use zsh -lc for preconfigured tools.
46
- - Always rent exactly one RTX 5090 (32 GB), in a datacenter, below $0.65/hour,
47
- with good internet and the Vast.ai PyTorch image. Do not substitute another
48
- GPU or rent multiple instances. Inspect existing instances before renting.
49
- - All Vast.ai setup and training commands run inside tmux session `gpubox`.
50
- Save logs/checkpoints to disk. Copy and verify artifacts before destroying
51
- instances; follow the owner's explicit cleanup instructions.
52
- - CPU evaluation and benchmarks run locally. Do not rent or restart a GPU for
53
- work that the local CPU can perform. No Playwright; use agent-browser if
54
- browser work is needed. Do not disable browser security to work around failures.
55
- - Never print or commit tokens, authorization headers, .env, PII, caches or
56
- virtualenvs. Hugging Face credentials come from existing login/environment.
57
- - Root model.safetensors and tokenizer.json are versioned large artifacts;
58
- use Hugging Face large-file storage. Keep experiment checkpoints under
59
- ignored models/ or checkpoints/. Publish selected reports with the release.
60
-
61
- ## Labnotes and milestones
62
-
63
- - Start at labnotes/milestones/index.md; its checklist owns implementation order.
64
- Read linked prerequisites and sources before starting a checkpoint.
65
- - For implementation or research, use `bin/create-labnotes two-to-four-words`.
66
- Write factual progress, failures, workarounds, decisions and evidence into
67
- the generated top-level labnotes file. Link it from the relevant milestone.
68
- Do not create labnotes for simple status/read-only checks or running an
69
- existing command. Labnotes accompany the work in its commit.
70
- - Keep milestones under labnotes/milestones/, deferred issues under
71
- labnotes/issues/. Use descriptive unnumbered filenames and Markdown task
72
- lists. Each milestone needs a runnable outcome, prerequisites, contracts,
73
- checkpoint tasks and acceptance gates. A spec is not an implementation.
74
- Leave index entries unchecked until all gates pass; record blockers honestly.
75
- - Historical generation/training labnotes remain in the dataset repo; link
76
- them rather than copying that history here. New model work gets model labnotes.
77
- - Record substantial architecture decisions and rejected alternatives under docs/.
78
-
79
- ## Checkpoints and Git
80
-
81
- Use small coherent commits of code, focused tests, docs, labnotes and lockfile.
82
- For behaviour changes, write a focused failing test before implementation.
83
- Experiments/prompts/benchmarks use recorded evidence; deterministic parsers,
84
- metrics, tokenization and serving get focused tests. Default tests exclude
85
- network integration tests. Before finishing code changes run:
86
-
87
- ```shell
88
- uv run ruff format --check
89
- uv run ruff check
90
- uv run pytest
91
- ```
92
-
93
- Inspect status and staged diffs, preserve unrelated work, stage exact paths.
94
- Do not commit or publish unless requested. Do not amend, rebase, reset,
95
- force-push, delete branches or undo commits without explicit instruction.
 
1
+ # Model releases only
2
+
3
+ Canonical development source: https://github.com/vxpipe/barge-in-detector.
4
+ Keep this repo limited to model weights, tokenizer/configuration, model card,
5
+ licence/attribution and provenance. Dataset releases belong in
6
+ HashNuke/barge-in-detector-data. Training data, executable scripts, tests,
7
+ benchmarks, labnotes and milestones belong in GitHub.
8
+
9
+ Follow the GitHub AGENTS.md for model contracts, CPU latency/quality gates,
10
+ the single RTX 5090 training policy, gpubox and verification. Create checkpoint
11
+ labnotes there using bin/create-labnotes and use labnotes/milestones/index.md;
12
+ keep the helper and development records in the source repo.
13
+
14
+ Publish artifacts from a committed source revision. release.json must pin the
15
+ exact dataset and code commits and artifact hashes. Verify unchanged hashes,
16
+ public model loading and accurate limitations in the card. Do not publish
17
+ credentials, local environments or training checkpoints outside a selected release.
18
+ Use Hub large-file storage for weights/tokenizer. Use ordinary release commits;
19
+ do not rewrite history, reset or force-push.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
README.md CHANGED
@@ -7,60 +7,27 @@ base_model: intfloat/multilingual-e5-small
7
  datasets: [HashNuke/barge-in-detector-data]
8
  tags: [barge-in, backchannel, multilingual, experimental]
9
  ---
10
- # Barge-in detector: experimental checkpoint
11
 
12
- Classifies a caller's cleaned utterance spoken **while a voice agent is talking**.
13
- `backchannel` lets it continue; `barge_in` yields the floor. This repo owns
14
- weights, tokenizer, training, inference, evaluation and benchmarks.
15
- [The dataset repo](https://huggingface.co/datasets/HashNuke/barge-in-detector-data)
16
- owns generation, cleaning, splits, review and historical labnotes.
17
 
18
- **Not production-ready:** it misses 25/30 short English challenge interruptions,
19
- including “stop” and “wait”. Probabilities are not calibrated. The runtime owns
20
- thresholds and handles answers after the agent finishes speaking.
 
 
 
21
 
22
- ## Checkpoint and evidence
 
 
23
 
24
- Root model.safetensors, config.json, tokenizer.json and tokenizer_config.json
25
- are the selected epoch-5 fp32 checkpoint. Base: intfloat/multilingual-e5-small
26
- (MIT); see LICENSE and NOTICE. `release.json` identifies dataset/artifact hashes
27
- and the published dataset revision. `reports/source-manifest.json` captures the
28
- original source snapshot, which predates the repository separation.
29
 
30
- Training: 2,640 synthetic rows; 333 validation; 331 test; five epochs, batch 32,
31
- weighted cross-entropy, AdamW, learning rate 3e-5, 10% warm-up, seed 20261115.
32
- BF16 on one RTX 5090. Epoch selected using validation macro F1 (0.99324).
33
- Full training history and breakdowns are in reports/training-metrics.json.
34
-
35
- | Evaluation | Accuracy | Missed interruptions | False interruptions |
36
- | --- | --- | --- | --- |
37
- | Synthetic validation (333) | 99.4% | 0.45% | 0.90% |
38
- | Synthetic test (331) | 100% | 0% | 0% |
39
- | Reviewed English challenge (98) | 73.5% | 54.2% | 0% |
40
- | Challenge unseen in any synthetic split (93) | 72.0% | 54.2% | 0% |
41
-
42
- The challenge was authored before inference, owner-reviewed and reconciled with
43
- authorization. Final binary review labels match the frozen authored labels.
44
- It is deliberately adversarial, includes repeated prefixes, and is now a
45
- regression/development set. Synthetic splits share generation criteria; exact
46
- cleaned-text leakage is absent, but template and semantic leakage remain unaudited.
47
- These scores do not establish representative real-world accuracy. All 14 listed
48
- languages have synthetic results; human-verified accuracy across all languages
49
- has not been established.
50
-
51
- ## Infer on CPU
52
-
53
- ```shell
54
- uv venv --python 3.14
55
- uv pip install torch==2.15.0 --index-url https://download.pytorch.org/whl/cpu
56
- uv pip install transformers==5.23.0 huggingface-hub==2.4.0
57
- uv run --no-sync python -m barge_in_detector_model.infer "please stop for a moment"
58
- ```
59
-
60
- Input must already follow the dataset cleaner: lowercase, punctuation removed,
61
- native scripts optionally mixed with English. No agent context, audio or language
62
- hint is needed. Keep the first 64 tokenizer tokens. Labels are 0=backchannel,
63
- 1=barge_in. The command emits `p_barge_in` and a label at the 0.5 threshold.
64
 
65
  ```python
66
  import torch
@@ -70,28 +37,43 @@ repo = "HashNuke/barge-in-detector"
70
  tokenizer = AutoTokenizer.from_pretrained(repo)
71
  model = AutoModelForSequenceClassification.from_pretrained(repo).cpu().eval()
72
  with torch.inference_mode():
73
- enc = tokenizer("please stop for a moment", return_tensors="pt", truncation=True, max_length=64)
74
- p_barge_in = model(**enc).logits.softmax(-1)[0, 1].item()
 
 
75
  ```
76
 
77
- ## Train and evaluate
 
 
 
78
 
79
- ```shell
80
- uv sync --group dev
81
- uv run python -m barge_in_detector_model.train --out models/run2 --dataset-revision DATASET_COMMIT
82
- uv run python -m barge_in_detector_model.evaluate --model models/run2 --split test --dataset-revision DATASET_COMMIT
83
- ```
84
 
85
- Data is fetched from the separate dataset repository at a resolved immutable
86
- revision. `--splits /path/to/dataset/data/splits` supports an existing checkout;
87
- retain its revision and file hashes with the run. Tune settings and thresholds
88
- on validation, report test without tuning, and never train on challenge rows.
89
- Migration does not retrain this checkpoint. Read AGENTS.md and the model
90
- milestone index before starting work. Use `bin/create-labnotes` for checkpoints.
91
 
92
- ## CPU latency
 
 
 
 
 
93
 
94
- Local four-vCPU AMD EPYC Genoa VM; eager PyTorch 2.15.0+cpu fp32; four threads,
 
 
 
 
 
 
 
 
 
 
95
  batch one, 100 warm-ups and 1,000 measured calls per scenario:
96
 
97
  | Warm text through label | p50 | p95 | p99 |
@@ -100,20 +82,7 @@ batch one, 100 warm-ups and 1,000 measured calls per scenario:
100
  | Full 64-token budget | 26.8 ms | 33.3 ms | 37.9 ms |
101
 
102
  Cold Python process through first label: median 4.22 seconds, range 3.85–4.81
103
- seconds across ten runs. Includes imports and local model/tokenizer load; OS file
104
- cache may be warm. Peak warm process RSS approximately 1,164 MiB. Network, HTTP,
105
- queueing and Vxpipe boundary are excluded. Reference-CPU gates are unestablished;
106
- ONNX/int8 is unmeasured. Reports retain the original benchmark provenance.
107
-
108
- ```shell
109
- uv run python -m experiments.benchmark_cpu --threads 4 --splits /path/to/dataset/data/splits
110
- uv run python -m experiments.benchmark_cpu_cold --threads 4
111
- uv run python -m experiments.evaluate_english_challenge --dataset-dir /path/to/dataset
112
- ```
113
-
114
- The CPU setup above intentionally uses the matching CPU wheel; use
115
- `uv run --no-sync` with that environment to retain it. For CPU development,
116
- install `pytest==9.2.0 ruff==0.17.5` with `uv pip install`. Standard `uv sync`
117
- uses the lockfile and includes GPU training dependencies. Do CPU tests
118
- locally; rent only a single datacenter RTX 5090 below $0.65/hour for training,
119
- using the Vast.ai PyTorch image and tmux session gpubox.
 
7
  datasets: [HashNuke/barge-in-detector-data]
8
  tags: [barge-in, backchannel, multilingual, experimental]
9
  ---
10
+ # Barge-in detector: experimental model release
11
 
12
+ Classifies a cleaned caller utterance spoken while a voice agent is talking:
13
+ backchannel (continue) or barge_in (yield). Exactly two labels; no audio,
14
+ preceding agent context or language hint is required. Answers after the agent
15
+ finishes are handled by the runtime. Keep the first 64 tokenizer tokens.
 
16
 
17
+ This repo contains **model release artifacts only**. All training, inference,
18
+ evaluation and benchmark code, tests, reports and labnotes live in the
19
+ [GitHub source](https://github.com/vxpipe/barge-in-detector).
20
+ [Dataset releases](https://huggingface.co/datasets/HashNuke/barge-in-detector-data)
21
+ are separate. release.json pins the exact code and dataset revisions and records
22
+ artifact hashes and original training provenance.
23
 
24
+ **Experimental, not production-ready.** The model misses 25/30 short English
25
+ challenge interruptions, including “stop” and “wait”. Probabilities are not
26
+ calibrated; the runtime owns thresholds. The 0.5 cutoff is diagnostic.
27
 
28
+ ## Use
 
 
 
 
29
 
30
+ Install compatible PyTorch and transformers, then:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
31
 
32
  ```python
33
  import torch
 
37
  tokenizer = AutoTokenizer.from_pretrained(repo)
38
  model = AutoModelForSequenceClassification.from_pretrained(repo).cpu().eval()
39
  with torch.inference_mode():
40
+ inputs = tokenizer(
41
+ "please stop for a moment", return_tensors="pt", truncation=True, max_length=64
42
+ )
43
+ p_barge_in = model(**inputs).logits.softmax(-1)[0, 1].item()
44
  ```
45
 
46
+ Input must already follow the source cleaner: lowercase, punctuation removed;
47
+ native scripts optionally mixed with English. Labels are 0=backchannel,
48
+ 1=barge_in. Pin the model revision when deploying. Current files are the original
49
+ epoch-5 fp32 checkpoint; changing repository ownership does not retrain it.
50
 
51
+ ## Training and evaluation
 
 
 
 
52
 
53
+ Base intfloat/multilingual-e5-small (MIT); see LICENSE and NOTICE. First run:
54
+ 2,640 synthetic train / 333 validation / 331 test; five epochs, batch 32,
55
+ weighted cross-entropy, AdamW, 3e-5 learning rate, 10% warm-up, seed 20261115,
56
+ bf16 on one RTX 5090. Epoch 5 selected on validation macro F1 (0.99324).
57
+ Training scripts and complete reports are in the pinned GitHub source.
 
58
 
59
+ | Evaluation | Accuracy | Missed interruptions | False interruptions |
60
+ | --- | --- | --- | --- |
61
+ | Synthetic validation (333) | 99.4% | 0.45% | 0.90% |
62
+ | Synthetic test (331) | 100% | 0% | 0% |
63
+ | Reviewed English challenge (98) | 73.5% | 54.2% | 0% |
64
+ | Challenge unseen in any synthetic split (93) | 72.0% | 54.2% | 0% |
65
 
66
+ The challenge was authored before inference, owner-reviewed and reconciled with
67
+ authorization. Final binary labels match the frozen expectations. It is now a
68
+ regression/development set with intentional repeated prefixes. Criteria overlap
69
+ between synthetic splits. Exact cleaned-text leakage is absent; template/semantic
70
+ leakage remains unaudited. These results do not establish real-world performance.
71
+ All 14 listed languages have synthetic results; human-verified accuracy across
72
+ all languages has not been established.
73
+
74
+ ## CPU timing
75
+
76
+ Four virtual AMD EPYC Genoa cores, PyTorch 2.15.0+cpu eager fp32, four threads,
77
  batch one, 100 warm-ups and 1,000 measured calls per scenario:
78
 
79
  | Warm text through label | p50 | p95 | p99 |
 
82
  | Full 64-token budget | 26.8 ms | 33.3 ms | 37.9 ms |
83
 
84
  Cold Python process through first label: median 4.22 seconds, range 3.85–4.81
85
+ seconds over ten runs. Includes imports and local loading; OS file cache may
86
+ be warm. Peak warm process RSS approximately 1,164 MiB. Network, HTTP, queueing
87
+ and Vxpipe boundary are excluded. Reference-CPU gates are unestablished;
88
+ ONNX/int8 is unmeasured. See the source labnotes and reports for recorded evidence.
 
 
 
 
 
 
 
 
 
 
 
 
 
barge_in_detector_model/__init__.py DELETED
@@ -1 +0,0 @@
1
- """Training, inference and evaluation for separately versioned barge-in data."""
 
 
barge_in_detector_model/data.py DELETED
@@ -1,39 +0,0 @@
1
- """Read versioned dataset splits without owning their generation or assignment."""
2
-
3
- import csv
4
- from pathlib import Path
5
-
6
- DEFAULT_DATASET = "HashNuke/barge-in-detector-data"
7
-
8
-
9
- def read_csv(path):
10
- with Path(path).open(encoding="utf-8", newline="") as f:
11
- return list(csv.DictReader(f))
12
-
13
-
14
- def resolve_splits(splits, *, repo_id=DEFAULT_DATASET, revision=None):
15
- if splits is not None:
16
- directory = Path(splits)
17
- resolved_revision = None
18
- else:
19
- from huggingface_hub import HfApi, snapshot_download
20
-
21
- resolved_revision = HfApi().dataset_info(repo_id, revision=revision).sha
22
- root = Path(
23
- snapshot_download(
24
- repo_id=repo_id,
25
- repo_type="dataset",
26
- revision=resolved_revision,
27
- allow_patterns=[
28
- "data/splits/*.csv",
29
- "data/evaluation/*.csv",
30
- "data/evaluation/reviews/*.json",
31
- "generation/factors/languages.yaml",
32
- ],
33
- )
34
- )
35
- directory = root / "data/splits"
36
- for split in ("train", "validation", "test"):
37
- if not (directory / f"{split}.csv").is_file():
38
- raise ValueError(f"Dataset split missing: {directory / split}.csv")
39
- return directory, resolved_revision
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
barge_in_detector_model/evaluate.py DELETED
@@ -1,111 +0,0 @@
1
- """Evaluate a trained model on one split (README.md).
2
-
3
- uv run python -m barge_in_detector_model.evaluate --model models/run1 --split test
4
-
5
- Reports accuracy, macro F1, false and missed interruption rates, and results
6
- per language; writes them to <model>/eval-<split>.json and the mistakes to
7
- <model>/errors-<split>.csv.
8
- """
9
-
10
- import argparse
11
- import csv
12
- import json
13
- from collections.abc import Sequence
14
- from pathlib import Path
15
- from typing import Any
16
-
17
- import torch
18
- from transformers import AutoModelForSequenceClassification, AutoTokenizer
19
-
20
- from barge_in_detector_model.data import read_csv, resolve_splits
21
- from barge_in_detector_model.metrics import LABELS, compute
22
-
23
- BARGE_IN = LABELS.index("barge_in")
24
-
25
-
26
- def device() -> torch.device:
27
- return torch.device("cuda" if torch.cuda.is_available() else "cpu")
28
-
29
-
30
- @torch.no_grad()
31
- def predict(
32
- model: Any,
33
- tokenizer: Any,
34
- rows: Sequence[dict[str, str]],
35
- *,
36
- batch_size: int = 128,
37
- max_length: int = 64,
38
- ) -> list[float]:
39
- """P(barge_in) for each row's cleaned text."""
40
- model.eval()
41
- dev = next(model.parameters()).device
42
- out: list[float] = []
43
- for i in range(0, len(rows), batch_size):
44
- texts = [r["text_clean"] for r in rows[i : i + batch_size]]
45
- enc = tokenizer(
46
- texts, truncation=True, max_length=max_length, padding=True, return_tensors="pt"
47
- ).to(dev)
48
- logits = model(**enc).logits.float()
49
- out += torch.softmax(logits, dim=-1)[:, BARGE_IN].tolist()
50
- return out
51
-
52
-
53
- def evaluate(
54
- model: Any,
55
- tokenizer: Any,
56
- rows: Sequence[dict[str, str]],
57
- *,
58
- threshold: float = 0.5,
59
- max_length: int = 64,
60
- ) -> tuple[dict[str, Any], list[float]]:
61
- probs = predict(model, tokenizer, rows, max_length=max_length)
62
- preds = ["barge_in" if p >= threshold else "backchannel" for p in probs]
63
- metrics = compute([r["label"] for r in rows], preds, [r["language"] for r in rows])
64
- return metrics, probs
65
-
66
-
67
- def main(argv: Sequence[str] | None = None) -> int:
68
- parser = argparse.ArgumentParser(description=__doc__)
69
- parser.add_argument("--model", type=Path, required=True, help="a directory written by train")
70
- parser.add_argument("--split", default="test", choices=["train", "validation", "test"])
71
- parser.add_argument("--splits", type=Path, default=None)
72
- parser.add_argument("--threshold", type=float, default=0.5, help="P(barge_in) cut-off")
73
- parser.add_argument("--max-length", type=int, default=64, help="token budget")
74
- parser.add_argument("--dataset-repo", default="HashNuke/barge-in-detector-data")
75
- parser.add_argument(
76
- "--dataset-revision", help="Exact dataset commit; resolves current main when omitted"
77
- )
78
- args = parser.parse_args(argv)
79
- args.splits, args.dataset_revision = resolve_splits(
80
- args.splits, repo_id=args.dataset_repo, revision=args.dataset_revision
81
- )
82
-
83
- tokenizer = AutoTokenizer.from_pretrained(args.model)
84
- model = AutoModelForSequenceClassification.from_pretrained(args.model).to(device())
85
- rows = read_csv(args.splits / f"{args.split}.csv")
86
- metrics, probs = evaluate(
87
- model, tokenizer, rows, threshold=args.threshold, max_length=args.max_length
88
- )
89
- metrics.update(split=args.split, threshold=args.threshold)
90
- (args.model / f"eval-{args.split}.json").write_text(json.dumps(metrics, indent=2))
91
- with (args.model / f"errors-{args.split}.csv").open("w", encoding="utf-8", newline="") as f:
92
- writer = csv.writer(f)
93
- writer.writerow(["label", "p_barge_in", "language", "caller_intent", "text"])
94
- for r, p in zip(rows, probs, strict=True):
95
- if (p >= args.threshold) != (r["label"] == "barge_in"):
96
- writer.writerow(
97
- [r["label"], f"{p:.3f}", r["language"], r["caller_intent"], r["text"]]
98
- )
99
- print(
100
- f"{args.split}: n={metrics['count']} accuracy={metrics['accuracy']:.3f} "
101
- f"macro_f1={metrics['macro_f1']:.3f} "
102
- f"false_interruptions={metrics['false_interruption_rate']:.3f} "
103
- f"missed_interruptions={metrics['missed_interruption_rate']:.3f}"
104
- )
105
- for lang, m in metrics["per_language"].items():
106
- print(f" {lang:>4} n={m['count']:<4} accuracy={m['accuracy']:.3f}")
107
- return 0
108
-
109
-
110
- if __name__ == "__main__":
111
- raise SystemExit(main())
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
barge_in_detector_model/infer.py DELETED
@@ -1,27 +0,0 @@
1
- """Predict a binary label for an already-cleaned overlapping caller utterance."""
2
-
3
- import argparse
4
- import json
5
- from pathlib import Path
6
-
7
- import torch
8
- from transformers import AutoModelForSequenceClassification, AutoTokenizer
9
-
10
-
11
- def main():
12
- parser = argparse.ArgumentParser(description=__doc__)
13
- parser.add_argument("text")
14
- parser.add_argument("--model", type=Path, default=Path("."))
15
- parser.add_argument("--threads", type=int, default=4)
16
- args = parser.parse_args()
17
- torch.set_num_threads(args.threads)
18
- tokenizer = AutoTokenizer.from_pretrained(args.model)
19
- model = AutoModelForSequenceClassification.from_pretrained(args.model).cpu().eval()
20
- with torch.inference_mode():
21
- enc = tokenizer(args.text, truncation=True, max_length=64, return_tensors="pt")
22
- p = torch.softmax(model(**enc).logits, dim=-1)[0, model.config.label2id["barge_in"]].item()
23
- print(json.dumps({"p_barge_in": p, "label": "barge_in" if p >= 0.5 else "backchannel"}))
24
-
25
-
26
- if __name__ == "__main__":
27
- main()
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
barge_in_detector_model/metrics.py DELETED
@@ -1,46 +0,0 @@
1
- """Metrics for the two labels. Pure Python, so they are tested without torch."""
2
-
3
- from collections import defaultdict
4
- from collections.abc import Sequence
5
- from typing import Any
6
-
7
- LABELS = ("backchannel", "barge_in")
8
-
9
-
10
- def _prf(labels: Sequence[str], preds: Sequence[str], cls: str) -> dict[str, float]:
11
- tp = sum(1 for y, p in zip(labels, preds, strict=True) if y == cls and p == cls)
12
- predicted = sum(1 for p in preds if p == cls)
13
- actual = sum(1 for y in labels if y == cls)
14
- precision = tp / predicted if predicted else 0.0
15
- recall = tp / actual if actual else 0.0
16
- f1 = 2 * precision * recall / (precision + recall) if precision + recall else 0.0
17
- return {"precision": precision, "recall": recall, "f1": f1, "count": actual}
18
-
19
-
20
- def compute(
21
- labels: Sequence[str], preds: Sequence[str], languages: Sequence[str]
22
- ) -> dict[str, Any]:
23
- per_class = {cls: _prf(labels, preds, cls) for cls in LABELS}
24
- backchannel = [p for y, p in zip(labels, preds, strict=True) if y == "backchannel"]
25
- barge_in = [p for y, p in zip(labels, preds, strict=True) if y == "barge_in"]
26
- by_language: dict[str, list[bool]] = defaultdict(list)
27
- for y, p, lang in zip(labels, preds, languages, strict=True):
28
- by_language[lang].append(y == p)
29
- return {
30
- "count": len(labels),
31
- "accuracy": sum(y == p for y, p in zip(labels, preds, strict=True)) / len(labels),
32
- "macro_f1": sum(c["f1"] for c in per_class.values()) / len(LABELS),
33
- # A backchannel predicted as barge-in stops the agent for nothing.
34
- "false_interruption_rate": (
35
- sum(p == "barge_in" for p in backchannel) / len(backchannel) if backchannel else 0.0
36
- ),
37
- # A barge-in predicted as backchannel lets the agent talk over the caller.
38
- "missed_interruption_rate": (
39
- sum(p == "backchannel" for p in barge_in) / len(barge_in) if barge_in else 0.0
40
- ),
41
- "per_class": per_class,
42
- "per_language": {
43
- lang: {"accuracy": sum(ok) / len(ok), "count": len(ok)}
44
- for lang, ok in sorted(by_language.items())
45
- },
46
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
barge_in_detector_model/train.py DELETED
@@ -1,140 +0,0 @@
1
- """Fine-tune a multilingual encoder on the utterance splits (README.md).
2
-
3
- uv sync
4
- uv run python -m barge_in_detector_model.train --out models/run1
5
-
6
- Trains on train.csv's cleaned text, evaluates on validation.csv after every
7
- epoch, and keeps the epoch with the best validation macro F1 in --out (model,
8
- tokenizer, metrics.json). Backchannel is the smaller class, so the loss
9
- weights each class by its inverse frequency. Uses the GPU when there is one.
10
- """
11
-
12
- import argparse
13
- import json
14
- import random
15
- import time
16
- from collections import Counter
17
- from collections.abc import Sequence
18
- from pathlib import Path
19
-
20
- import torch
21
- from transformers import (
22
- AutoModelForSequenceClassification,
23
- AutoTokenizer,
24
- get_linear_schedule_with_warmup,
25
- )
26
-
27
- from barge_in_detector_model.data import read_csv, resolve_splits
28
- from barge_in_detector_model.evaluate import device, evaluate
29
- from barge_in_detector_model.metrics import LABELS
30
-
31
- # MIT licence, 118M parameters, about 100 languages (dataset repo docs/utterance-first-decision.md).
32
- DEFAULT_MODEL = "intfloat/multilingual-e5-small"
33
-
34
-
35
- def main(argv: Sequence[str] | None = None) -> int:
36
- parser = argparse.ArgumentParser(description=__doc__)
37
- parser.add_argument("--out", type=Path, required=True)
38
- parser.add_argument("--model", default=DEFAULT_MODEL, help="Hugging Face model id or path")
39
- parser.add_argument("--splits", type=Path, default=None)
40
- parser.add_argument("--epochs", type=int, default=5)
41
- parser.add_argument("--batch-size", type=int, default=32)
42
- parser.add_argument("--lr", type=float, default=3e-5)
43
- parser.add_argument("--warmup", type=float, default=0.1, help="share of steps")
44
- parser.add_argument("--max-length", type=int, default=64, help="token budget")
45
- parser.add_argument("--seed", type=int, default=20261115)
46
- parser.add_argument("--dataset-repo", default="HashNuke/barge-in-detector-data")
47
- parser.add_argument(
48
- "--dataset-revision", help="Exact dataset commit; resolves current main when omitted"
49
- )
50
- args = parser.parse_args(argv)
51
- args.splits, args.dataset_revision = resolve_splits(
52
- args.splits, repo_id=args.dataset_repo, revision=args.dataset_revision
53
- )
54
-
55
- random.seed(args.seed)
56
- torch.manual_seed(args.seed)
57
- dev = device()
58
- train_rows = read_csv(args.splits / "train.csv")
59
- validation_rows = read_csv(args.splits / "validation.csv")
60
- label_id = {label: i for i, label in enumerate(LABELS)}
61
-
62
- tokenizer = AutoTokenizer.from_pretrained(args.model)
63
- model = AutoModelForSequenceClassification.from_pretrained(
64
- args.model,
65
- num_labels=len(LABELS),
66
- id2label=dict(enumerate(LABELS)),
67
- label2id=label_id,
68
- ).to(dev)
69
-
70
- counts = Counter(r["label"] for r in train_rows)
71
- weights = torch.tensor(
72
- [len(train_rows) / (len(LABELS) * counts[label]) for label in LABELS], device=dev
73
- )
74
- loss_fn = torch.nn.CrossEntropyLoss(weight=weights)
75
- steps = args.epochs * -(-len(train_rows) // args.batch_size)
76
- optimizer = torch.optim.AdamW(model.parameters(), lr=args.lr, weight_decay=0.01)
77
- scheduler = get_linear_schedule_with_warmup(optimizer, int(args.warmup * steps), steps)
78
- use_bf16 = dev.type == "cuda" and torch.cuda.is_bf16_supported()
79
-
80
- print(
81
- f"device={dev} bf16={use_bf16} model={args.model} train={len(train_rows)} "
82
- f"validation={len(validation_rows)} class_weights={weights.tolist()}"
83
- )
84
- args.out.mkdir(parents=True, exist_ok=True)
85
- history, best = [], None
86
- for epoch in range(1, args.epochs + 1):
87
- model.train()
88
- order = list(range(len(train_rows)))
89
- random.shuffle(order)
90
- started, total = time.time(), 0.0
91
- for i in range(0, len(order), args.batch_size):
92
- batch = [train_rows[j] for j in order[i : i + args.batch_size]]
93
- enc = tokenizer(
94
- [r["text_clean"] for r in batch],
95
- truncation=True,
96
- max_length=args.max_length,
97
- padding=True,
98
- return_tensors="pt",
99
- ).to(dev)
100
- labels = torch.tensor([label_id[r["label"]] for r in batch], device=dev)
101
- with torch.autocast(device_type=dev.type, dtype=torch.bfloat16, enabled=use_bf16):
102
- logits = model(**enc).logits
103
- loss = loss_fn(logits.float(), labels)
104
- loss.backward()
105
- torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
106
- optimizer.step()
107
- scheduler.step()
108
- optimizer.zero_grad()
109
- total += loss.item() * len(batch)
110
- metrics, _ = evaluate(model, tokenizer, validation_rows, max_length=args.max_length)
111
- record = {
112
- "epoch": epoch,
113
- "train_loss": total / len(train_rows),
114
- "seconds": round(time.time() - started, 1),
115
- "validation": metrics,
116
- }
117
- history.append(record)
118
- print(
119
- f"epoch {epoch}: loss={record['train_loss']:.4f} "
120
- f"validation macro_f1={metrics['macro_f1']:.3f} accuracy={metrics['accuracy']:.3f} "
121
- f"false_interruptions={metrics['false_interruption_rate']:.3f} "
122
- f"missed_interruptions={metrics['missed_interruption_rate']:.3f}"
123
- )
124
- if best is None or metrics["macro_f1"] > best["validation"]["macro_f1"]:
125
- best = record
126
- model.save_pretrained(args.out)
127
- tokenizer.save_pretrained(args.out)
128
- (args.out / "metrics.json").write_text(
129
- json.dumps(
130
- {"args": {k: str(v) for k, v in vars(args).items()}, "best": best, "history": history},
131
- indent=2,
132
- )
133
- )
134
- print(f"best epoch {best['epoch']}: validation macro_f1={best['validation']['macro_f1']:.3f}")
135
- print(f"saved to {args.out}")
136
- return 0
137
-
138
-
139
- if __name__ == "__main__":
140
- raise SystemExit(main())
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
bin/create-labnotes DELETED
@@ -1,59 +0,0 @@
1
- #!/usr/bin/env bash
2
- set -euo pipefail
3
-
4
- usage() {
5
- cat <<'USAGE'
6
- Usage:
7
- bin/create-labnotes 2-4-word-hyphenated-task-name
8
-
9
- Creates an empty markdown labnotes file in labnotes/.
10
-
11
- The input must be a 2-4 word hyphenated string, for example:
12
- bin/create-labnotes test-crawl-resume
13
-
14
- If spaces are provided, they will be converted to hyphens:
15
- bin/create-labnotes "test crawl resume"
16
- USAGE
17
- }
18
-
19
- if [[ $# -eq 0 ]]; then
20
- usage
21
- exit 0
22
- fi
23
-
24
- input="$*"
25
- slug="$(printf '%s' "$input" \
26
- | tr '[:upper:]' '[:lower:]' \
27
- | sed -E 's/[[:space:]]+/-/g; s/[^a-z0-9-]+/-/g; s/-+/-/g; s/^-//; s/-$//')"
28
-
29
- if [[ -z "$slug" ]]; then
30
- echo "Error: task name must contain letters or numbers." >&2
31
- echo >&2
32
- usage >&2
33
- exit 1
34
- fi
35
-
36
- word_count="$(awk -F- '{ print NF }' <<<"$slug")"
37
- if (( word_count < 2 || word_count > 4 )); then
38
- echo "Error: input must be a 2-4 word hyphenated string." >&2
39
- echo >&2
40
- usage >&2
41
- exit 1
42
- fi
43
-
44
- repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
45
- labnotes_dir="$repo_root/labnotes"
46
- mkdir -p "$labnotes_dir"
47
-
48
- timestamp="$(date '+%Y%m%d-%H%M')"
49
- filename="$timestamp-$slug.md"
50
- path="$labnotes_dir/$filename"
51
-
52
- if [[ -e "$path" ]]; then
53
- echo "Error: labnotes/$filename already exists." >&2
54
- exit 1
55
- fi
56
-
57
- touch "$path"
58
-
59
- echo "Created labnotes/$filename"
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
experiments/benchmark_cpu.py DELETED
@@ -1,140 +0,0 @@
1
- """Measure warmed batch-one tokenization through label output on local CPU.
2
-
3
- Experimental timing harness: run from the repo root with a CPU PyTorch environment.
4
- It measures no runtime boundary, queue, HTTP call or cold model load.
5
- """
6
-
7
- import argparse
8
- import hashlib
9
- import json
10
- import platform
11
- import random
12
- import resource
13
- import statistics
14
- import time
15
- from pathlib import Path
16
-
17
- import torch
18
- import transformers
19
- from transformers import AutoModelForSequenceClassification, AutoTokenizer
20
-
21
- from barge_in_detector_model.data import read_csv, resolve_splits
22
-
23
-
24
- def percentile(values, quantile):
25
- ordered = sorted(values)
26
- index = (len(ordered) - 1) * quantile
27
- low = int(index)
28
- high = min(low + 1, len(ordered) - 1)
29
- return ordered[low] + (ordered[high] - ordered[low]) * (index - low)
30
-
31
-
32
- def summary(values):
33
- return {
34
- "p50_ms": percentile(values, 0.5),
35
- "p95_ms": percentile(values, 0.95),
36
- "p99_ms": percentile(values, 0.99),
37
- "mean_ms": statistics.mean(values),
38
- "max_ms": max(values),
39
- }
40
-
41
-
42
- @torch.inference_mode()
43
- def main():
44
- parser = argparse.ArgumentParser(description=__doc__)
45
- parser.add_argument("--iterations", type=int, default=1000)
46
- parser.add_argument("--warmup", type=int, default=100)
47
- parser.add_argument("--model", type=Path, default=Path("."))
48
- parser.add_argument("--threads", type=int, nargs="+", default=[1, 2, 4])
49
- parser.add_argument("--splits", type=Path, help="Local dataset split directory")
50
- parser.add_argument("--dataset-revision")
51
- args = parser.parse_args()
52
- split_dir, dataset_revision = resolve_splits(args.splits, revision=args.dataset_revision)
53
- if args.iterations < 100 or args.warmup < 1 or min(args.threads) < 1:
54
- parser.error("Use >=100 measured iterations, positive warmup and threads")
55
- torch.set_num_interop_threads(1)
56
- torch.set_num_threads(1)
57
- started = time.perf_counter()
58
- tokenizer = AutoTokenizer.from_pretrained(args.model, local_files_only=True)
59
- model = (
60
- AutoModelForSequenceClassification.from_pretrained(args.model, local_files_only=True)
61
- .cpu()
62
- .eval()
63
- )
64
- load_seconds = time.perf_counter() - started
65
- rows = read_csv(split_dir / "test.csv")
66
- texts = [r["text_clean"] for r in rows]
67
- random.Random(20261115).shuffle(texts)
68
- stress = (
69
- "please stop for a moment because i need to correct the information you are reading " * 12
70
- )
71
- assert len(tokenizer(stress, truncation=True, max_length=64)["input_ids"]) == 64
72
- report = {
73
- "dataset_revision": dataset_revision,
74
- "model": str(args.model),
75
- "runtime": "PyTorch eager fp32, CPU only",
76
- "model_sha256": hashlib.sha256((args.model / "model.safetensors").read_bytes()).hexdigest(),
77
- "python": platform.python_version(),
78
- "torch": torch.__version__,
79
- "transformers": transformers.__version__,
80
- "platform": platform.platform(),
81
- "cpuinfo": next(
82
- line.split(":", 1)[1].strip()
83
- for line in Path("/proc/cpuinfo").read_text().splitlines()
84
- if line.startswith("model name")
85
- ),
86
- "load_seconds_excluded": load_seconds,
87
- "iterations": args.iterations,
88
- "warmup_per_scenario": args.warmup,
89
- "batch_size": 1,
90
- "max_length": 64,
91
- "interop_threads": 1,
92
- "parameter_bytes": sum(p.numel() * p.element_size() for p in model.parameters()),
93
- "corpus_sha256": hashlib.sha256((split_dir / "test.csv").read_bytes()).hexdigest(),
94
- "corpus_rows": len(rows),
95
- "languages": sorted({r["language"] for r in rows}),
96
- "measurements": [],
97
- "scope": "Local virtual CPU; reference CPU unspecified. Excludes Vxpipe boundary and queueing; no whole-path acceptance claim.",
98
- }
99
- lengths = [len(tokenizer(t, truncation=True, max_length=64)["input_ids"]) for t in texts]
100
- report["corpus_token_lengths"] = {
101
- "p50": percentile(lengths, 0.5),
102
- "p95": percentile(lengths, 0.95),
103
- "max": max(lengths),
104
- }
105
- for threads in args.threads:
106
- torch.set_num_threads(threads)
107
- for name, inputs in [("multilingual_test", texts), ("full_64_tokens", [stress])]:
108
- samples, token_times, forward_times = [], [], []
109
- for index in range(args.warmup + args.iterations):
110
- text = inputs[index % len(inputs)]
111
- start = time.perf_counter_ns()
112
- enc = tokenizer(text, truncation=True, max_length=64, return_tensors="pt")
113
- tokenized = time.perf_counter_ns()
114
- logits = model(**enc).logits
115
- probability = torch.softmax(logits, dim=-1)[
116
- 0, model.config.label2id["barge_in"]
117
- ].item()
118
- label = "barge_in" if probability >= 0.5 else "backchannel"
119
- finish = time.perf_counter_ns()
120
- assert 0 <= probability <= 1
121
- assert label in model.config.label2id
122
- if index >= args.warmup:
123
- samples.append((finish - start) / 1e6)
124
- token_times.append((tokenized - start) / 1e6)
125
- forward_times.append((finish - tokenized) / 1e6)
126
- result = {
127
- "threads": threads,
128
- "scenario": name,
129
- "model_path": summary(samples),
130
- "tokenization": summary(token_times),
131
- "forward_and_probability": summary(forward_times),
132
- }
133
- report["measurements"].append(result)
134
- print(json.dumps(result), flush=True)
135
- report["peak_process_rss_mib"] = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024
136
- (args.model / "cpu-benchmark.json").write_text(json.dumps(report, indent=2))
137
-
138
-
139
- if __name__ == "__main__":
140
- main()
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
experiments/benchmark_cpu_cold.py DELETED
@@ -1,80 +0,0 @@
1
- """Time a new Python process through its first label from local model files."""
2
-
3
- import argparse
4
- import json
5
- import os
6
- import statistics
7
- import subprocess
8
- import sys
9
- import time
10
- from pathlib import Path
11
-
12
- CHILD = """
13
- import json
14
- import time
15
- import torch
16
- from transformers import AutoModelForSequenceClassification, AutoTokenizer
17
- import sys
18
- model_path, threads = sys.argv[1], int(sys.argv[2])
19
- torch.set_num_threads(threads)
20
- torch.set_num_interop_threads(1)
21
- tokenizer = AutoTokenizer.from_pretrained(model_path, local_files_only=True)
22
- model = AutoModelForSequenceClassification.from_pretrained(model_path, local_files_only=True).cpu().eval()
23
- with torch.inference_mode():
24
- loaded_start = time.perf_counter_ns()
25
- enc = tokenizer('please stop for a moment', truncation=True, max_length=64, return_tensors='pt')
26
- p = torch.softmax(model(**enc).logits, dim=-1)[0, model.config.label2id['barge_in']].item()
27
- label = 'barge_in' if p >= 0.5 else 'backchannel'
28
- loaded_ms = (time.perf_counter_ns() - loaded_start) / 1e6
29
- print(json.dumps({'label': label, 'p_barge_in': p, 'loaded_to_first_label_ms': loaded_ms}), flush=True)
30
- """
31
-
32
-
33
- def main():
34
- parser = argparse.ArgumentParser(description=__doc__)
35
- parser.add_argument("--model", type=Path, default=Path("."))
36
- parser.add_argument("--runs", type=int, default=10)
37
- parser.add_argument("--threads", type=int, default=1)
38
- args = parser.parse_args()
39
- samples = []
40
- loaded_first = []
41
- env = {
42
- **os.environ,
43
- "OMP_NUM_THREADS": str(args.threads),
44
- "MKL_NUM_THREADS": str(args.threads),
45
- "TOKENIZERS_PARALLELISM": "false",
46
- }
47
- for run in range(args.runs):
48
- start = time.perf_counter_ns()
49
- with subprocess.Popen(
50
- [sys.executable, "-c", CHILD, str(args.model), str(args.threads)],
51
- stdout=subprocess.PIPE,
52
- stderr=subprocess.DEVNULL,
53
- text=True,
54
- env=env,
55
- ) as process:
56
- line = process.stdout.readline()
57
- elapsed = (time.perf_counter_ns() - start) / 1e9
58
- result = json.loads(line)
59
- if process.wait() != 0:
60
- raise RuntimeError("Cold-start child failed")
61
- samples.append(elapsed)
62
- loaded_first.append(result["loaded_to_first_label_ms"])
63
- print(f"cold start {run + 1}: {elapsed:.3f}s {result}", flush=True)
64
- report = {
65
- "cold_process_seconds": samples,
66
- "loaded_to_first_label_ms": loaded_first,
67
- "loaded_to_first_label_median_ms": statistics.median(loaded_first),
68
- "median_seconds": statistics.median(samples),
69
- "min_seconds": min(samples),
70
- "max_seconds": max(samples),
71
- "runs": args.runs,
72
- "threads": args.threads,
73
- "scope": "Python process creation through reading first flushed label. Includes imports, tokenizer/model load, first inference. Local files only; OS file cache may be warm. Excludes process shutdown, downloads and HTTP/network boundary.",
74
- }
75
- (args.model / "cpu-cold-benchmark.json").write_text(json.dumps(report, indent=2))
76
- print(json.dumps(report), flush=True)
77
-
78
-
79
- if __name__ == "__main__":
80
- main()
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
experiments/evaluate_english_challenge.py DELETED
@@ -1,132 +0,0 @@
1
- """Frozen-checkpoint diagnostic experiment; labels are authored before inference.
2
-
3
- Run from the repo root with uv run --group train python -m experiments.evaluate_english_challenge.
4
- Evidence is saved under the model directory, outside the training splits.
5
- """
6
-
7
- import argparse
8
- import csv
9
- import hashlib
10
- import json
11
- from collections import defaultdict
12
- from pathlib import Path
13
-
14
- import torch
15
- import transformers
16
- from transformers import AutoModelForSequenceClassification, AutoTokenizer
17
-
18
- from barge_in_detector_model.data import read_csv, resolve_splits
19
- from barge_in_detector_model.evaluate import device, predict
20
- from barge_in_detector_model.metrics import compute
21
-
22
- MODEL = Path(".")
23
-
24
-
25
- def score(rows, probs):
26
- return compute(
27
- [r["label"] for r in rows],
28
- ["barge_in" if p >= 0.5 else "backchannel" for p in probs],
29
- [r["language"] for r in rows],
30
- )
31
-
32
-
33
- def main():
34
- parser = argparse.ArgumentParser(description=__doc__)
35
- parser.add_argument("--dataset-dir", type=Path, help="Local dataset repository root")
36
- parser.add_argument("--dataset-revision")
37
- args = parser.parse_args()
38
- if args.dataset_dir is None:
39
- split_dir, revision = resolve_splits(None, revision=args.dataset_revision)
40
- dataset_dir = split_dir.parents[1]
41
- else:
42
- dataset_dir = args.dataset_dir
43
- revision = None
44
- challenge = dataset_dir / "data/evaluation/english-challenge-v1.csv"
45
- rows = read_csv(challenge)
46
- splits = {
47
- s: read_csv(dataset_dir / f"data/splits/{s}.csv") for s in ("train", "validation", "test")
48
- }
49
- texts = {s: {r["text_clean"] for r in rs} for s, rs in splits.items()}
50
- all_seen = set.union(*texts.values())
51
- tokenizer = AutoTokenizer.from_pretrained(MODEL)
52
- model = AutoModelForSequenceClassification.from_pretrained(MODEL).to(device())
53
- probs = predict(model, tokenizer, rows, max_length=64)
54
- single_probs = predict(model, tokenizer, rows, batch_size=1, max_length=64)
55
- report = {
56
- "dataset_revision": revision,
57
- "challenge_sha256": hashlib.sha256(challenge.read_bytes()).hexdigest(),
58
- "model_sha256": hashlib.sha256((MODEL / "model.safetensors").read_bytes()).hexdigest(),
59
- "threshold": 0.5,
60
- "runtime": {
61
- "device": str(next(model.parameters()).device),
62
- "torch": torch.__version__,
63
- "transformers": transformers.__version__,
64
- "threads": torch.get_num_threads(),
65
- },
66
- "batch_vs_single_max_probability_difference": max(
67
- abs(a - b) for a, b in zip(probs, single_probs, strict=True)
68
- ),
69
- "batch_vs_single_label_disagreements": sum(
70
- (a >= 0.5) != (b >= 0.5) for a, b in zip(probs, single_probs, strict=True)
71
- ),
72
- "review_status": "Owner-reviewed v1 labels match authored expectations after authorized reconciliation. Diagnostic development set; not independent representative accuracy evidence.",
73
- "all": score(rows, probs),
74
- "by_category": {},
75
- "overlap_counts": {s: sum(r["text_clean"] in ts for r in rows) for s, ts in texts.items()},
76
- "cross_split_exact_overlap": {
77
- f"{a}/{b}": len(texts[a] & texts[b])
78
- for a, b in (("train", "validation"), ("train", "test"), ("validation", "test"))
79
- },
80
- }
81
- for category in sorted({r["category"] for r in rows}):
82
- pairs = [(r, p) for r, p in zip(rows, probs, strict=True) if r["category"] == category]
83
- report["by_category"][category] = score(*zip(*pairs, strict=True))
84
- unseen = [(r, p) for r, p in zip(rows, probs, strict=True) if r["text_clean"] not in all_seen]
85
- report["unseen"] = score(*zip(*unseen, strict=True))
86
- groups = defaultdict(list)
87
- for r, p in zip(rows, probs, strict=True):
88
- if r["group"]:
89
- groups[r["group"]].append(
90
- {"text": r["text_clean"], "label": r["label"], "p_barge_in": p}
91
- )
92
- report["prefix_groups"] = dict(groups)
93
- report["prefix_groups_all_labels_correct"] = sum(
94
- all((r["p_barge_in"] >= 0.5) == (r["label"] == "barge_in") for r in group)
95
- for group in groups.values()
96
- )
97
- # A diagnostic baseline: select its word-count cutoff using validation only.
98
- english = {s: [r for r in rs if r["language"] == "en"] for s, rs in splits.items()}
99
-
100
- def length_probs(rs, cutoff):
101
- return [float(len(r["text_clean"].split()) >= cutoff) for r in rs]
102
-
103
- cutoff = max(
104
- range(1, 25),
105
- key=lambda n: score(english["validation"], length_probs(english["validation"], n))[
106
- "macro_f1"
107
- ],
108
- )
109
- report["english_length_baseline"] = {
110
- "cutoff_words": cutoff,
111
- "validation": score(english["validation"], length_probs(english["validation"], cutoff)),
112
- "test": score(english["test"], length_probs(english["test"], cutoff)),
113
- "challenge": score(rows, length_probs(rows, cutoff)),
114
- }
115
- with (MODEL / "challenge-predictions.csv").open("w", newline="") as f:
116
- writer = csv.DictWriter(f, fieldnames=[*rows[0], "p_barge_in", "prediction", "seen_in"])
117
- writer.writeheader()
118
- for r, p in zip(rows, probs, strict=True):
119
- writer.writerow(
120
- {
121
- **r,
122
- "p_barge_in": p,
123
- "prediction": "barge_in" if p >= 0.5 else "backchannel",
124
- "seen_in": ";".join(s for s, ts in texts.items() if r["text_clean"] in ts),
125
- }
126
- )
127
- (MODEL / "challenge-report.json").write_text(json.dumps(report, indent=2))
128
- print(json.dumps({k: v for k, v in report.items() if k != "prefix_groups"}, indent=2))
129
-
130
-
131
- if __name__ == "__main__":
132
- main()
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
labnotes/20261115-1332-repository-split.md DELETED
@@ -1,41 +0,0 @@
1
- # Model repository split
2
-
3
- Owner moved model artifacts, training, inference, evaluation and benchmarks into
4
- this standalone Hugging Face model repo. Dataset generation, splitting, review
5
- and all earlier labnotes remain in HashNuke/barge-in-detector-data. This repo has
6
- its own scoped AGENTS.md, bin/create-labnotes and milestone index.
7
-
8
- - New package barge_in_detector_model isolates training/metrics/data-loading
9
- code from the dataset generator. Model code downloads a resolved dataset
10
- commit, or uses explicit local splits; it does not assign splits.
11
- - Four unit tests cover metrics, local CSVs, missing split detection and pinned
12
- Hub downloads. Tiny CPU smoke trained one epoch, saved, reloaded and evaluated.
13
- - Actual unchanged checkpoint predicts stop as backchannel, p=0.0134437,
14
- reproducing the known failure. No GPU rental or retraining of release weights.
15
- - Token budget remains 64; two labels; predictions are not calibrated.
16
- - Root weights and tokenizer match the original artifact hashes. Original source
17
- manifest and sanitized training metrics retain run provenance. Model card
18
- reports synthetic and reviewed diagnostics and local CPU timing limitations.
19
- - AGENTS.md derives CPU serving gates, fourteen languages, single RTX 5090,
20
- under-$0.65/hour datacenter/PyTorch policy, gpubox, tests, and labnotes/milestone
21
- usage from the dataset instructions. Existing history is linked, not duplicated.
22
-
23
- Historical run and latency evidence is in the dataset repo labnotes. Follow
24
- labnotes/milestones/index.md for new model work. Publication verification follows.
25
-
26
- ## Publication verified
27
-
28
- - Initial weights/tokenizer commit dcbadfd7572e08ef3ccd19e35576e85faffe00f8;
29
- standalone code commit 9aa77e5ff33f479ec5559f3f65bcfc092e5f86ab.
30
- - release.json pins dataset dda7115ace1fe7b07c49139860131f39910521ad.
31
- - Downloaded that dataset through the model loader: 2,640/333/331 rows and
32
- exact file hashes match. Standard datasets.load_dataset also loads those
33
- splits and the separate 98-example challenge configuration correctly.
34
- - Anonymous Hub model download loaded on local CPU and reproduced stop
35
- probability 0.0134437. Remote large-file checksums match local model/tokenizer.
36
- - Four model unit tests, Ruff format and lint passed. Tiny local train/save/
37
- reload/evaluate smoke passed before original scripts were removed.
38
- - bin/create-labnotes is executable in the Git tree. Each repo has its own
39
- AGENTS.md and milestone index. New model work stays here; history stays in data.
40
- - No model or tokenizer weights were committed to the original GitHub repo.
41
- No GPU was rented or restarted during this publication work.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
labnotes/issues/.gitkeep DELETED
File without changes
labnotes/milestones/experimental-checkpoint-release.md DELETED
@@ -1,30 +0,0 @@
1
- # Experimental checkpoint release
2
-
3
- ## Runnable outcome
4
-
5
- A self-contained Hugging Face model repo that loads the first checkpoint on CPU,
6
- trains/evaluates with a separate versioned dataset, and records honest limitations.
7
- Prerequisites: dataset HashNuke/barge-in-detector-data, original generation/training
8
- history in that dataset repo, root README and AGENTS.md.
9
-
10
- ## Contracts and checkpoints
11
-
12
- - [x] Copy and verify selected epoch-5 weights and tokenizer without retraining.
13
- - [x] Move training/evaluation/metrics/benchmarks into independent model modules.
14
- - [x] Resolve Hub dataset revisions to immutable commits; support local splits.
15
- - [x] Copy labnotes helper; derive model AGENTS.md and establish milestone usage.
16
- - [x] Tiny CPU train/save/reload/evaluate smoke and actual checkpoint inference.
17
- - [x] Publish artifacts, instructions and code to the model repo.
18
- - [x] Record dataset commit, source snapshot and artifact checksums in release.json.
19
-
20
- ## Acceptance gates
21
-
22
- - [x] Four model unit tests, lint and formatting pass.
23
- - [x] Model weights and tokenizer resolve as large-file artifacts on the Hub.
24
- - [x] Published files match local release hashes; CPU inference works from files.
25
- - [x] Model card separates synthetic and reviewed diagnostics and records failures.
26
-
27
- This milestone does not claim production readiness, calibration or reference-CPU
28
- latency acceptance. See repository-split labnotes for evidence.
29
-
30
- Checkpoint evidence: [repository split labnotes](../20261115-1332-repository-split.md).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
labnotes/milestones/index.md DELETED
@@ -1,7 +0,0 @@
1
- # Model milestones
2
-
3
- The checklist owns implementation order. Read AGENTS.md and linked sources;
4
- leave entries unchecked until every acceptance gate passes.
5
-
6
- - [x] [Publish the experimental checkpoint](experimental-checkpoint-release.md)
7
- - [ ] [Improve short-interruption recall](short-interruption-recall.md)
 
 
 
 
 
 
 
 
labnotes/milestones/short-interruption-recall.md DELETED
@@ -1,28 +0,0 @@
1
- # Improve short-interruption recall
2
-
3
- ## Runnable outcome and prerequisites
4
-
5
- Improve the current checkpoint's 5/30 short-interruption recall on reviewed
6
- English development examples, then establish performance on fresh owner-reviewed
7
- English/Hindi evaluation data. Prerequisite: experimental checkpoint release;
8
- coordinate data changes with the dataset repo coverage-audit milestone.
9
-
10
- ## Contract and checkpoint tasks
11
-
12
- - [ ] Agree acceptance floors for false and missed interruptions before tuning.
13
- - [ ] Compare a character n-gram baseline and encoder on exactly the same splits.
14
- - [ ] Train against the revised dataset distribution; never train on challenge rows.
15
- - [ ] Evaluate short interruptions, longer backchannels, prefix transitions and
16
- per-language errors. Preserve old checkpoint and use validation for selection.
17
- - [ ] Record immutable dataset/source revisions, parameters and costs.
18
- - [ ] Calibrate on realistic validation priors.
19
- - [ ] Measure ONNX/int8 parity and reference-CPU latency/memory; measure Vxpipe
20
- serialization/queue/boundary overhead for the whole-path budget.
21
-
22
- ## Acceptance gates
23
-
24
- - [ ] Fresh human-reviewed English/Hindi results meet agreed floors and beat baseline.
25
- - [ ] Model-path p99 <=20 ms; whole path p99 <=50 ms on reference CPU, warmed batch one.
26
- - [ ] Prefix behaviour and Vxpipe overlapping-speech scenarios pass with evidence.
27
-
28
- Use `bin/create-labnotes` and link each checkpoint's factual evidence here.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
pyproject.toml DELETED
@@ -1,25 +0,0 @@
1
- [project]
2
- name = "barge-in-detector-model"
3
- version = "0.1.0"
4
- description = "Training, evaluation and CPU inference for the Vxpipe barge-in classifier"
5
- requires-python = ">=3.14"
6
- dependencies = ["torch>=2.15.0", "transformers>=5.23.0", "huggingface-hub>=2.4.0"]
7
-
8
- [dependency-groups]
9
- dev = ["pytest>=9.2.0", "ruff>=0.17.5"]
10
-
11
- [tool.uv]
12
- package = false
13
-
14
- [tool.pytest.ini_options]
15
- pythonpath = ["."]
16
- testpaths = ["tests"]
17
- addopts = "-m 'not integration'"
18
- markers = ["integration: calls networks; excluded by default"]
19
-
20
- [tool.ruff]
21
- line-length = 100
22
- target-version = "py314"
23
- [tool.ruff.lint]
24
- select = ["E", "F", "I", "B", "UP", "SIM"]
25
- ignore = ["E501"]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
release.json CHANGED
@@ -3,7 +3,7 @@
3
  "status": "experimental, uncalibrated, short-interruption failures",
4
  "base_model": "intfloat/multilingual-e5-small",
5
  "dataset_repo": "HashNuke/barge-in-detector-data",
6
- "dataset_revision": "dda7115ace1fe7b07c49139860131f39910521ad",
7
  "dataset_files_sha256": {
8
  "data/export/utterances.csv": "556a2820e4a24ac13bab44321fb32b464c5fb225750d41a5a10ebcde7cdbab8a",
9
  "data/splits/train.csv": "aab224bd0526eee915295e6f5c52076e29bb751cd0d664eefb4321843ad22533",
@@ -20,9 +20,8 @@
20
  },
21
  "original_training_source": {
22
  "git_base_commit": "5ce0ed1283401f8c73bafcf5a7dbd2101c4bbee6",
23
- "overlay_manifest": "reports/source-manifest.json"
24
  },
25
- "migration_source_git_commit": "1f92917a59908e320c3a3489c3e5de413a3e4672",
26
  "labels": {
27
  "0": "backchannel",
28
  "1": "barge_in"
@@ -30,5 +29,9 @@
30
  "max_length": 64,
31
  "default_diagnostic_threshold": 0.5,
32
  "selected_epoch": 5,
33
- "training_metrics": "reports/training-metrics.json"
 
 
 
 
34
  }
 
3
  "status": "experimental, uncalibrated, short-interruption failures",
4
  "base_model": "intfloat/multilingual-e5-small",
5
  "dataset_repo": "HashNuke/barge-in-detector-data",
6
+ "dataset_revision": "9e914ce8b50aa12ebd3fe212bb183e09a1dee370",
7
  "dataset_files_sha256": {
8
  "data/export/utterances.csv": "556a2820e4a24ac13bab44321fb32b464c5fb225750d41a5a10ebcde7cdbab8a",
9
  "data/splits/train.csv": "aab224bd0526eee915295e6f5c52076e29bb751cd0d664eefb4321843ad22533",
 
20
  },
21
  "original_training_source": {
22
  "git_base_commit": "5ce0ed1283401f8c73bafcf5a7dbd2101c4bbee6",
23
+ "overlay_manifest": "https://github.com/vxpipe/barge-in-detector/blob/836de6924c3da68cd871e6e08da34291e9778432/reports/e5-small-run1/source-manifest.json"
24
  },
 
25
  "labels": {
26
  "0": "backchannel",
27
  "1": "barge_in"
 
29
  "max_length": 64,
30
  "default_diagnostic_threshold": 0.5,
31
  "selected_epoch": 5,
32
+ "training_metrics": "https://github.com/vxpipe/barge-in-detector/blob/836de6924c3da68cd871e6e08da34291e9778432/reports/e5-small-run1/training-metrics.json",
33
+ "source_repo": "https://github.com/vxpipe/barge-in-detector",
34
+ "code_revision": "836de6924c3da68cd871e6e08da34291e9778432",
35
+ "first_publication_source_commit": "1f92917a59908e320c3a3489c3e5de413a3e4672",
36
+ "ownership": "Release artifacts only. All development code, tests, reports and labnotes are in the GitHub source repository."
37
  }
reports/challenge-predictions.csv DELETED
@@ -1,99 +0,0 @@
1
- case_id,category,group,language,label,text,text_clean,review_status,p_barge_in,prediction,seen_in
2
- en-001,short_interruption,,en,barge_in,Stop,stop,pending,0.013443702831864357,backchannel,
3
- en-002,short_interruption,,en,barge_in,Wait,wait,pending,0.013491915538907051,backchannel,
4
- en-003,short_interruption,,en,barge_in,Pause,pause,pending,0.013399876654148102,backchannel,
5
- en-004,short_interruption,,en,barge_in,Cancel,cancel,pending,0.0172202717512846,backchannel,
6
- en-005,short_interruption,,en,barge_in,Wrong,wrong,pending,0.015773363411426544,backchannel,
7
- en-006,short_interruption,,en,barge_in,Repeat,repeat,pending,0.01339809875935316,backchannel,
8
- en-007,short_interruption,,en,barge_in,Slower,slower,pending,0.014412087388336658,backchannel,
9
- en-008,short_interruption,,en,barge_in,Restart,restart,pending,0.014540360309183598,backchannel,
10
- en-009,short_interruption,,en,barge_in,Enough,enough,pending,0.013300837948918343,backchannel,
11
- en-010,short_interruption,,en,barge_in,Stop talking,stop talking,pending,0.01404176652431488,backchannel,
12
- en-011,short_interruption,,en,barge_in,Wait please,wait please,pending,0.013346969150006771,backchannel,
13
- en-012,short_interruption,,en,barge_in,Not that,not that,pending,0.015475512482225895,backchannel,
14
- en-013,short_interruption,,en,barge_in,Wrong date,wrong date,pending,0.9878827333450317,barge_in,
15
- en-014,short_interruption,,en,barge_in,Wrong person,wrong person,pending,0.8659051060676575,barge_in,
16
- en-015,short_interruption,,en,barge_in,Go back,go back,pending,0.013336537405848503,backchannel,
17
- en-016,short_interruption,,en,barge_in,Say again,say again,pending,0.013324333354830742,backchannel,
18
- en-017,short_interruption,,en,barge_in,Speak slower,speak slower,pending,0.013362826779484749,backchannel,
19
- en-018,short_interruption,,en,barge_in,Start over,start over,pending,0.01339060626924038,backchannel,
20
- en-019,short_interruption,,en,barge_in,Let me finish,let me finish,pending,0.013296245597302914,backchannel,
21
- en-020,short_interruption,,en,barge_in,You misunderstood,you misunderstood,pending,0.013550461269915104,backchannel,
22
- en-021,short_interruption,,en,barge_in,Cancel it,cancel it,pending,0.015159462578594685,backchannel,
23
- en-022,short_interruption,,en,barge_in,Change it,change it,pending,0.013769036158919334,backchannel,
24
- en-023,short_interruption,,en,barge_in,Which one,which one,pending,0.017033280804753304,backchannel,
25
- en-024,short_interruption,,en,barge_in,Why not,why not,pending,0.014318976551294327,backchannel,
26
- en-025,short_interruption,,en,barge_in,How much,how much,pending,0.016760120168328285,backchannel,
27
- en-026,short_interruption,,en,barge_in,What time,what time,pending,0.9822450876235962,barge_in,
28
- en-027,short_interruption,,en,barge_in,What address,what address,pending,0.9860103130340576,barge_in,
29
- en-028,short_interruption,,en,barge_in,Tuesday instead,tuesday instead,pending,0.7567923069000244,barge_in,
30
- en-029,short_interruption,,en,barge_in,The second one,the second one,pending,0.02150067687034607,backchannel,
31
- en-030,short_interruption,,en,barge_in,That is wrong,that is wrong,pending,0.1904229372739792,backchannel,
32
- en-031,long_backchannel,,en,backchannel,Yes yes yes yes yes,yes yes yes yes yes,pending,0.013356473296880722,backchannel,
33
- en-032,long_backchannel,,en,backchannel,Okay okay okay okay okay,okay okay okay okay okay,pending,0.013345411978662014,backchannel,
34
- en-033,long_backchannel,,en,backchannel,Right right right I see,right right right i see,pending,0.013284565880894661,backchannel,
35
- en-034,long_backchannel,,en,backchannel,Mm hmm yes mm hmm yes,mm hmm yes mm hmm yes,pending,0.013306752778589725,backchannel,
36
- en-035,long_backchannel,,en,backchannel,Oh wow oh wow yeah,oh wow oh wow yeah,pending,0.013319982215762138,backchannel,
37
- en-036,long_backchannel,,en,backchannel,Yeah I see I see,yeah i see i see,pending,0.013288560323417187,backchannel,
38
- en-037,long_backchannel,,en,backchannel,Yes I understand what you mean,yes i understand what you mean,pending,0.0892898291349411,backchannel,
39
- en-038,long_backchannel,,en,backchannel,Right that makes sense to me,right that makes sense to me,pending,0.013280664570629597,backchannel,
40
- en-039,long_backchannel,,en,backchannel,Okay I am following you so far,okay i am following you so far,pending,0.28858309984207153,backchannel,
41
- en-040,long_backchannel,,en,backchannel,Yes yes that sounds right,yes yes that sounds right,pending,0.013292225077748299,backchannel,
42
- en-041,long_backchannel,,en,backchannel,Ah I see what you mean now,ah i see what you mean now,pending,0.32564374804496765,backchannel,
43
- en-042,long_backchannel,,en,backchannel,Mm hmm I understand yes,mm hmm i understand yes,pending,0.0132778724655509,backchannel,
44
- en-043,long_backchannel,,en,backchannel,Oh dear that sounds really difficult,oh dear that sounds really difficult,pending,0.047465287148952484,backchannel,
45
- en-044,long_backchannel,,en,backchannel,Wow that must have been exciting,wow that must have been exciting,pending,0.013364902697503567,backchannel,
46
- en-045,long_backchannel,,en,backchannel,That is interesting yes I see,that is interesting yes i see,pending,0.013278285041451454,backchannel,
47
- en-046,long_backchannel,,en,backchannel,Fair enough I understand that,fair enough i understand that,pending,0.013284215703606606,backchannel,
48
- en-047,long_backchannel,,en,backchannel,Yes absolutely I agree with that,yes absolutely i agree with that,pending,0.013294750824570656,backchannel,
49
- en-048,long_backchannel,,en,backchannel,Right okay yes that makes sense,right okay yes that makes sense,pending,0.013285371474921703,backchannel,
50
- en-049,long_backchannel,,en,backchannel,Oh no that is really sad,oh no that is really sad,pending,0.01329362578690052,backchannel,
51
- en-050,long_backchannel,,en,backchannel,Ha ha that is funny,ha ha that is funny,pending,0.013284615240991116,backchannel,
52
- en-051,long_backchannel,,en,backchannel,Mmm yes okay I hear you,mmm yes okay i hear you,pending,0.013274511322379112,backchannel,
53
- en-052,long_backchannel,,en,backchannel,Sure sure I am with you,sure sure i am with you,pending,0.013305414468050003,backchannel,
54
- en-053,long_backchannel,,en,backchannel,Got it yes understood,got it yes understood,pending,0.013286358676850796,backchannel,
55
- en-054,long_backchannel,,en,backchannel,Uh huh uh huh right,uh huh uh huh right,pending,0.013300181366503239,backchannel,
56
- en-055,ordinary_backchannel,,en,backchannel,Okay,okay,pending,0.013304519467055798,backchannel,
57
- en-056,ordinary_backchannel,,en,backchannel,Yes,yes,pending,0.013361463323235512,backchannel,
58
- en-057,ordinary_backchannel,,en,backchannel,Yeah,yeah,pending,0.013312126509845257,backchannel,
59
- en-058,ordinary_backchannel,,en,backchannel,Mm hmm,mm hmm,pending,0.013299750164151192,backchannel,validation
60
- en-059,ordinary_backchannel,,en,backchannel,Uh huh,uh huh,pending,0.013296402990818024,backchannel,train
61
- en-060,ordinary_backchannel,,en,backchannel,Right,right,pending,0.013306460343301296,backchannel,
62
- en-061,ordinary_backchannel,,en,backchannel,I see,i see,pending,0.013295376673340797,backchannel,
63
- en-062,ordinary_backchannel,,en,backchannel,Got it,got it,pending,0.01331440731883049,backchannel,
64
- en-063,ordinary_backchannel,,en,backchannel,Oh wow,oh wow,pending,0.013293537311255932,backchannel,train
65
- en-064,ordinary_backchannel,,en,backchannel,I understand,i understand,pending,0.013287884183228016,backchannel,
66
- en-065,ordinary_interruption,,en,barge_in,I need to change my delivery address,i need to change my delivery address,pending,0.9881365299224854,barge_in,
67
- en-066,ordinary_interruption,,en,barge_in,You have the wrong customer on the line,you have the wrong customer on the line,pending,0.9880920648574829,barge_in,
68
- en-067,ordinary_interruption,,en,barge_in,Can you explain what that fee is for,can you explain what that fee is for,pending,0.9879797697067261,barge_in,
69
- en-068,ordinary_interruption,,en,barge_in,Please stop reading and let me speak,please stop reading and let me speak,pending,0.9880907535552979,barge_in,
70
- en-069,ordinary_interruption,,en,barge_in,Actually I already paid that invoice,actually i already paid that invoice,pending,0.9881226420402527,barge_in,
71
- en-070,ordinary_interruption,,en,barge_in,I want to speak to someone else,i want to speak to someone else,pending,0.9879534244537354,barge_in,
72
- en-071,ordinary_interruption,,en,barge_in,No I said Thursday and not Tuesday,no i said thursday and not tuesday,pending,0.9881582856178284,barge_in,
73
- en-072,ordinary_interruption,,en,barge_in,Before you continue I have a question,before you continue i have a question,pending,0.9879493713378906,barge_in,
74
- en-073,ordinary_interruption,,en,barge_in,That is not what I was asking about,that is not what i was asking about,pending,0.9879181981086731,barge_in,
75
- en-074,ordinary_interruption,,en,barge_in,I cannot hear you please say that again,i cannot hear you please say that again,pending,0.9881183505058289,barge_in,
76
- en-075,prefix,prefix-1,en,backchannel,Yes,yes,pending,0.013361463323235512,backchannel,
77
- en-076,prefix,prefix-1,en,backchannel,Yes yes,yes yes,pending,0.013298917561769485,backchannel,
78
- en-077,prefix,prefix-1,en,barge_in,Yes yes but change the time,yes yes but change the time,pending,0.9880699515342712,barge_in,
79
- en-078,prefix,prefix-2,en,backchannel,Okay,okay,pending,0.013304519467055798,backchannel,
80
- en-079,prefix,prefix-2,en,backchannel,Okay okay,okay okay,pending,0.01330505684018135,backchannel,
81
- en-080,prefix,prefix-2,en,barge_in,Okay okay wait that is the wrong address,okay okay wait that is the wrong address,pending,0.988145112991333,barge_in,
82
- en-081,prefix,prefix-3,en,backchannel,Right,right,pending,0.013306460343301296,backchannel,
83
- en-082,prefix,prefix-3,en,backchannel,Right right,right right,pending,0.01329006627202034,backchannel,train
84
- en-083,prefix,prefix-3,en,barge_in,Right right but I meant the other account,right right but i meant the other account,pending,0.988115668296814,barge_in,
85
- en-084,prefix,prefix-4,en,backchannel,Mm hmm,mm hmm,pending,0.013299750164151192,backchannel,validation
86
- en-085,prefix,prefix-4,en,backchannel,Mm hmm yes,mm hmm yes,pending,0.013292054645717144,backchannel,
87
- en-086,prefix,prefix-4,en,barge_in,Mm hmm yes can you repeat the price,mm hmm yes can you repeat the price,pending,0.9881290793418884,barge_in,
88
- en-087,prefix,prefix-5,en,backchannel,I see,i see,pending,0.013295376673340797,backchannel,
89
- en-088,prefix,prefix-5,en,backchannel,I see yes,i see yes,pending,0.013287578709423542,backchannel,
90
- en-089,prefix,prefix-5,en,barge_in,I see yes but I need to cancel,i see yes but i need to cancel,pending,0.9879875183105469,barge_in,
91
- en-090,prefix,prefix-6,en,backchannel,Sure,sure,pending,0.013307417742908001,backchannel,
92
- en-091,prefix,prefix-6,en,backchannel,Sure sure,sure sure,pending,0.013290559872984886,backchannel,
93
- en-092,prefix,prefix-6,en,barge_in,Sure sure stop there please,sure sure stop there please,pending,0.013294538483023643,backchannel,
94
- en-093,prefix,prefix-7,en,backchannel,Got it,got it,pending,0.01331440731883049,backchannel,
95
- en-094,prefix,prefix-7,en,backchannel,Got it okay,got it okay,pending,0.013302822597324848,backchannel,
96
- en-095,prefix,prefix-7,en,barge_in,Got it okay now let me explain,got it okay now let me explain,pending,0.6444560885429382,barge_in,
97
- en-096,prefix,prefix-8,en,backchannel,Yeah,yeah,pending,0.013312126509845257,backchannel,
98
- en-097,prefix,prefix-8,en,backchannel,Yeah yeah,yeah yeah,pending,0.013376967050135136,backchannel,
99
- en-098,prefix,prefix-8,en,barge_in,Yeah yeah no that will not work for me,yeah yeah no that will not work for me,pending,0.9878116250038147,barge_in,
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
reports/challenge-report.json DELETED
@@ -1,438 +0,0 @@
1
- {
2
- "challenge_sha256": "3d46b9e024f4d3bedd69f07603cde934e85ec2272988e90d2acebc30d5e99519",
3
- "model_sha256": "bddb65c08f6260cfc9d320e65b4229229ec9f3ca95204aed98ab5ae45ff758d3",
4
- "threshold": 0.5,
5
- "runtime": {
6
- "device": "cpu",
7
- "torch": "2.15.0+cpu",
8
- "transformers": "5.23.0",
9
- "threads": 4
10
- },
11
- "batch_vs_single_max_probability_difference": 8.285045623779297e-06,
12
- "batch_vs_single_label_disagreements": 0,
13
- "review_status": "Owner reviewed all 98 cases; 13 subsequent revisions explicitly authorized; final binary labels match authored v1 labels. Diagnostic development set, not independent representative evaluation.",
14
- "all": {
15
- "count": 98,
16
- "accuracy": 0.7346938775510204,
17
- "macro_f1": 0.7111111111111111,
18
- "false_interruption_rate": 0.0,
19
- "missed_interruption_rate": 0.5416666666666666,
20
- "per_class": {
21
- "backchannel": {
22
- "precision": 0.6578947368421053,
23
- "recall": 1.0,
24
- "f1": 0.7936507936507937,
25
- "count": 50
26
- },
27
- "barge_in": {
28
- "precision": 1.0,
29
- "recall": 0.4583333333333333,
30
- "f1": 0.6285714285714286,
31
- "count": 48
32
- }
33
- },
34
- "per_language": {
35
- "en": {
36
- "accuracy": 0.7346938775510204,
37
- "count": 98
38
- }
39
- }
40
- },
41
- "by_category": {
42
- "long_backchannel": {
43
- "count": 24,
44
- "accuracy": 1.0,
45
- "macro_f1": 0.5,
46
- "false_interruption_rate": 0.0,
47
- "missed_interruption_rate": 0.0,
48
- "per_class": {
49
- "backchannel": {
50
- "precision": 1.0,
51
- "recall": 1.0,
52
- "f1": 1.0,
53
- "count": 24
54
- },
55
- "barge_in": {
56
- "precision": 0.0,
57
- "recall": 0.0,
58
- "f1": 0.0,
59
- "count": 0
60
- }
61
- },
62
- "per_language": {
63
- "en": {
64
- "accuracy": 1.0,
65
- "count": 24
66
- }
67
- }
68
- },
69
- "ordinary_backchannel": {
70
- "count": 10,
71
- "accuracy": 1.0,
72
- "macro_f1": 0.5,
73
- "false_interruption_rate": 0.0,
74
- "missed_interruption_rate": 0.0,
75
- "per_class": {
76
- "backchannel": {
77
- "precision": 1.0,
78
- "recall": 1.0,
79
- "f1": 1.0,
80
- "count": 10
81
- },
82
- "barge_in": {
83
- "precision": 0.0,
84
- "recall": 0.0,
85
- "f1": 0.0,
86
- "count": 0
87
- }
88
- },
89
- "per_language": {
90
- "en": {
91
- "accuracy": 1.0,
92
- "count": 10
93
- }
94
- }
95
- },
96
- "ordinary_interruption": {
97
- "count": 10,
98
- "accuracy": 1.0,
99
- "macro_f1": 0.5,
100
- "false_interruption_rate": 0.0,
101
- "missed_interruption_rate": 0.0,
102
- "per_class": {
103
- "backchannel": {
104
- "precision": 0.0,
105
- "recall": 0.0,
106
- "f1": 0.0,
107
- "count": 0
108
- },
109
- "barge_in": {
110
- "precision": 1.0,
111
- "recall": 1.0,
112
- "f1": 1.0,
113
- "count": 10
114
- }
115
- },
116
- "per_language": {
117
- "en": {
118
- "accuracy": 1.0,
119
- "count": 10
120
- }
121
- }
122
- },
123
- "prefix": {
124
- "count": 24,
125
- "accuracy": 0.9583333333333334,
126
- "macro_f1": 0.9515151515151515,
127
- "false_interruption_rate": 0.0,
128
- "missed_interruption_rate": 0.125,
129
- "per_class": {
130
- "backchannel": {
131
- "precision": 0.9411764705882353,
132
- "recall": 1.0,
133
- "f1": 0.9696969696969697,
134
- "count": 16
135
- },
136
- "barge_in": {
137
- "precision": 1.0,
138
- "recall": 0.875,
139
- "f1": 0.9333333333333333,
140
- "count": 8
141
- }
142
- },
143
- "per_language": {
144
- "en": {
145
- "accuracy": 0.9583333333333334,
146
- "count": 24
147
- }
148
- }
149
- },
150
- "short_interruption": {
151
- "count": 30,
152
- "accuracy": 0.16666666666666666,
153
- "macro_f1": 0.14285714285714285,
154
- "false_interruption_rate": 0.0,
155
- "missed_interruption_rate": 0.8333333333333334,
156
- "per_class": {
157
- "backchannel": {
158
- "precision": 0.0,
159
- "recall": 0.0,
160
- "f1": 0.0,
161
- "count": 0
162
- },
163
- "barge_in": {
164
- "precision": 1.0,
165
- "recall": 0.16666666666666666,
166
- "f1": 0.2857142857142857,
167
- "count": 30
168
- }
169
- },
170
- "per_language": {
171
- "en": {
172
- "accuracy": 0.16666666666666666,
173
- "count": 30
174
- }
175
- }
176
- }
177
- },
178
- "overlap_counts": {
179
- "train": 3,
180
- "validation": 2,
181
- "test": 0
182
- },
183
- "cross_split_exact_overlap": {
184
- "train/validation": 0,
185
- "train/test": 0,
186
- "validation/test": 0
187
- },
188
- "unseen": {
189
- "count": 93,
190
- "accuracy": 0.7204301075268817,
191
- "macro_f1": 0.702216748768473,
192
- "false_interruption_rate": 0.0,
193
- "missed_interruption_rate": 0.5416666666666666,
194
- "per_class": {
195
- "backchannel": {
196
- "precision": 0.6338028169014085,
197
- "recall": 1.0,
198
- "f1": 0.7758620689655172,
199
- "count": 45
200
- },
201
- "barge_in": {
202
- "precision": 1.0,
203
- "recall": 0.4583333333333333,
204
- "f1": 0.6285714285714286,
205
- "count": 48
206
- }
207
- },
208
- "per_language": {
209
- "en": {
210
- "accuracy": 0.7204301075268817,
211
- "count": 93
212
- }
213
- }
214
- },
215
- "prefix_groups": {
216
- "prefix-1": [
217
- {
218
- "text": "yes",
219
- "label": "backchannel",
220
- "p_barge_in": 0.013361463323235512
221
- },
222
- {
223
- "text": "yes yes",
224
- "label": "backchannel",
225
- "p_barge_in": 0.013298917561769485
226
- },
227
- {
228
- "text": "yes yes but change the time",
229
- "label": "barge_in",
230
- "p_barge_in": 0.9880699515342712
231
- }
232
- ],
233
- "prefix-2": [
234
- {
235
- "text": "okay",
236
- "label": "backchannel",
237
- "p_barge_in": 0.013304519467055798
238
- },
239
- {
240
- "text": "okay okay",
241
- "label": "backchannel",
242
- "p_barge_in": 0.01330505684018135
243
- },
244
- {
245
- "text": "okay okay wait that is the wrong address",
246
- "label": "barge_in",
247
- "p_barge_in": 0.988145112991333
248
- }
249
- ],
250
- "prefix-3": [
251
- {
252
- "text": "right",
253
- "label": "backchannel",
254
- "p_barge_in": 0.013306460343301296
255
- },
256
- {
257
- "text": "right right",
258
- "label": "backchannel",
259
- "p_barge_in": 0.01329006627202034
260
- },
261
- {
262
- "text": "right right but i meant the other account",
263
- "label": "barge_in",
264
- "p_barge_in": 0.988115668296814
265
- }
266
- ],
267
- "prefix-4": [
268
- {
269
- "text": "mm hmm",
270
- "label": "backchannel",
271
- "p_barge_in": 0.013299750164151192
272
- },
273
- {
274
- "text": "mm hmm yes",
275
- "label": "backchannel",
276
- "p_barge_in": 0.013292054645717144
277
- },
278
- {
279
- "text": "mm hmm yes can you repeat the price",
280
- "label": "barge_in",
281
- "p_barge_in": 0.9881290793418884
282
- }
283
- ],
284
- "prefix-5": [
285
- {
286
- "text": "i see",
287
- "label": "backchannel",
288
- "p_barge_in": 0.013295376673340797
289
- },
290
- {
291
- "text": "i see yes",
292
- "label": "backchannel",
293
- "p_barge_in": 0.013287578709423542
294
- },
295
- {
296
- "text": "i see yes but i need to cancel",
297
- "label": "barge_in",
298
- "p_barge_in": 0.9879875183105469
299
- }
300
- ],
301
- "prefix-6": [
302
- {
303
- "text": "sure",
304
- "label": "backchannel",
305
- "p_barge_in": 0.013307417742908001
306
- },
307
- {
308
- "text": "sure sure",
309
- "label": "backchannel",
310
- "p_barge_in": 0.013290559872984886
311
- },
312
- {
313
- "text": "sure sure stop there please",
314
- "label": "barge_in",
315
- "p_barge_in": 0.013294538483023643
316
- }
317
- ],
318
- "prefix-7": [
319
- {
320
- "text": "got it",
321
- "label": "backchannel",
322
- "p_barge_in": 0.01331440731883049
323
- },
324
- {
325
- "text": "got it okay",
326
- "label": "backchannel",
327
- "p_barge_in": 0.013302822597324848
328
- },
329
- {
330
- "text": "got it okay now let me explain",
331
- "label": "barge_in",
332
- "p_barge_in": 0.6444560885429382
333
- }
334
- ],
335
- "prefix-8": [
336
- {
337
- "text": "yeah",
338
- "label": "backchannel",
339
- "p_barge_in": 0.013312126509845257
340
- },
341
- {
342
- "text": "yeah yeah",
343
- "label": "backchannel",
344
- "p_barge_in": 0.013376967050135136
345
- },
346
- {
347
- "text": "yeah yeah no that will not work for me",
348
- "label": "barge_in",
349
- "p_barge_in": 0.9878116250038147
350
- }
351
- ]
352
- },
353
- "prefix_groups_all_labels_correct": 7,
354
- "english_length_baseline": {
355
- "cutoff_words": 6,
356
- "validation": {
357
- "count": 81,
358
- "accuracy": 0.9753086419753086,
359
- "macro_f1": 0.9716783216783216,
360
- "false_interruption_rate": 0.07407407407407407,
361
- "missed_interruption_rate": 0.0,
362
- "per_class": {
363
- "backchannel": {
364
- "precision": 1.0,
365
- "recall": 0.9259259259259259,
366
- "f1": 0.9615384615384615,
367
- "count": 27
368
- },
369
- "barge_in": {
370
- "precision": 0.9642857142857143,
371
- "recall": 1.0,
372
- "f1": 0.9818181818181818,
373
- "count": 54
374
- }
375
- },
376
- "per_language": {
377
- "en": {
378
- "accuracy": 0.9753086419753086,
379
- "count": 81
380
- }
381
- }
382
- },
383
- "test": {
384
- "count": 81,
385
- "accuracy": 0.9382716049382716,
386
- "macro_f1": 0.9268556980314249,
387
- "false_interruption_rate": 0.18518518518518517,
388
- "missed_interruption_rate": 0.0,
389
- "per_class": {
390
- "backchannel": {
391
- "precision": 1.0,
392
- "recall": 0.8148148148148148,
393
- "f1": 0.8979591836734693,
394
- "count": 27
395
- },
396
- "barge_in": {
397
- "precision": 0.9152542372881356,
398
- "recall": 1.0,
399
- "f1": 0.9557522123893806,
400
- "count": 54
401
- }
402
- },
403
- "per_language": {
404
- "en": {
405
- "accuracy": 0.9382716049382716,
406
- "count": 81
407
- }
408
- }
409
- },
410
- "challenge": {
411
- "count": 98,
412
- "accuracy": 0.5510204081632653,
413
- "macro_f1": 0.5315080399826162,
414
- "false_interruption_rate": 0.26,
415
- "missed_interruption_rate": 0.6458333333333334,
416
- "per_class": {
417
- "backchannel": {
418
- "precision": 0.5441176470588235,
419
- "recall": 0.74,
420
- "f1": 0.6271186440677965,
421
- "count": 50
422
- },
423
- "barge_in": {
424
- "precision": 0.5666666666666667,
425
- "recall": 0.3541666666666667,
426
- "f1": 0.4358974358974359,
427
- "count": 48
428
- }
429
- },
430
- "per_language": {
431
- "en": {
432
- "accuracy": 0.5510204081632653,
433
- "count": 98
434
- }
435
- }
436
- }
437
- }
438
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
reports/cpu-benchmark.json DELETED
@@ -1,94 +0,0 @@
1
- {
2
- "model": "models/e5-small-run1",
3
- "runtime": "PyTorch eager fp32, CPU only",
4
- "model_sha256": "bddb65c08f6260cfc9d320e65b4229229ec9f3ca95204aed98ab5ae45ff758d3",
5
- "python": "3.14.4",
6
- "torch": "2.15.0+cpu",
7
- "transformers": "5.23.0",
8
- "platform": "Linux-7.0.0-43-generic-x86_64-with-glibc2.43",
9
- "cpuinfo": "AMD EPYC-Genoa Processor",
10
- "load_seconds_excluded": 2.8975897090276703,
11
- "iterations": 1000,
12
- "warmup_per_scenario": 100,
13
- "batch_size": 1,
14
- "max_length": 64,
15
- "interop_threads": 1,
16
- "parameter_bytes": 470618120,
17
- "corpus_sha256": "8b630782362b05249b3be94b0213850258422be5ddeb62fcf703817b0bf9e11a",
18
- "corpus_rows": 331,
19
- "languages": [
20
- "de",
21
- "en",
22
- "es",
23
- "fil",
24
- "fr",
25
- "hi",
26
- "id",
27
- "it",
28
- "ja",
29
- "kn",
30
- "ml",
31
- "ta",
32
- "te",
33
- "zh"
34
- ],
35
- "measurements": [
36
- {
37
- "threads": 4,
38
- "scenario": "multilingual_test",
39
- "model_path": {
40
- "p50_ms": 13.088757,
41
- "p95_ms": 19.419374299999998,
42
- "p99_ms": 25.316384159999995,
43
- "mean_ms": 13.625100593,
44
- "max_ms": 83.599
45
- },
46
- "tokenization": {
47
- "p50_ms": 0.222889,
48
- "p95_ms": 0.30127505,
49
- "p99_ms": 0.45513418999999933,
50
- "mean_ms": 0.23647671,
51
- "max_ms": 2.407625
52
- },
53
- "forward_and_probability": {
54
- "p50_ms": 12.8741705,
55
- "p95_ms": 19.1815334,
56
- "p99_ms": 25.09679618,
57
- "mean_ms": 13.388623883,
58
- "max_ms": 83.391099
59
- }
60
- },
61
- {
62
- "threads": 4,
63
- "scenario": "full_64_tokens",
64
- "model_path": {
65
- "p50_ms": 26.794403000000003,
66
- "p95_ms": 33.269715649999995,
67
- "p99_ms": 37.862168669999996,
68
- "mean_ms": 27.779209678,
69
- "max_ms": 93.899777
70
- },
71
- "tokenization": {
72
- "p50_ms": 0.6677489999999999,
73
- "p95_ms": 0.8626962999999997,
74
- "p99_ms": 1.7089376799999993,
75
- "mean_ms": 0.704945851,
76
- "max_ms": 7.957467
77
- },
78
- "forward_and_probability": {
79
- "p50_ms": 26.103834499999998,
80
- "p95_ms": 32.575319,
81
- "p99_ms": 37.10665824,
82
- "mean_ms": 27.074263827,
83
- "max_ms": 93.297535
84
- }
85
- }
86
- ],
87
- "scope": "Local virtual CPU; reference CPU unspecified. Excludes Vxpipe boundary and queueing; no whole-path acceptance claim.",
88
- "corpus_token_lengths": {
89
- "p50": 13.0,
90
- "p95": 19.0,
91
- "max": 26
92
- },
93
- "peak_process_rss_mib": 1163.6171875
94
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
reports/cpu-cold-benchmark.json DELETED
@@ -1,33 +0,0 @@
1
- {
2
- "cold_process_seconds": [
3
- 4.768154836,
4
- 4.810785561,
5
- 4.136632718,
6
- 4.255589102,
7
- 3.847660745,
8
- 4.045793715,
9
- 3.973606716,
10
- 4.190341404,
11
- 4.387973674,
12
- 4.28136222
13
- ],
14
- "loaded_to_first_label_ms": [
15
- 26.506046,
16
- 40.193806,
17
- 18.425675,
18
- 22.50301,
19
- 23.081508,
20
- 40.155708,
21
- 16.969254,
22
- 23.919822,
23
- 23.060476,
24
- 24.334184
25
- ],
26
- "loaded_to_first_label_median_ms": 23.500664999999998,
27
- "median_seconds": 4.222965253,
28
- "min_seconds": 3.847660745,
29
- "max_seconds": 4.810785561,
30
- "runs": 10,
31
- "threads": 4,
32
- "scope": "Python process creation through reading first flushed label. Includes imports, tokenizer/model load, first inference. Local files only; OS file cache may be warm. Excludes process shutdown, downloads and HTTP/network boundary."
33
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
reports/cpu-thread-sweep.json DELETED
@@ -1,194 +0,0 @@
1
- {
2
- "model": "models/e5-small-run1",
3
- "runtime": "PyTorch eager fp32, CPU only",
4
- "model_sha256": "bddb65c08f6260cfc9d320e65b4229229ec9f3ca95204aed98ab5ae45ff758d3",
5
- "python": "3.14.4",
6
- "torch": "2.15.0+cpu",
7
- "transformers": "5.23.0",
8
- "platform": "Linux-7.0.0-43-generic-x86_64-with-glibc2.43",
9
- "cpuinfo": "AMD EPYC-Genoa Processor",
10
- "load_seconds_excluded": 3.1383294460829347,
11
- "iterations": 1000,
12
- "warmup_per_scenario": 100,
13
- "batch_size": 1,
14
- "max_length": 64,
15
- "interop_threads": 1,
16
- "parameter_bytes": 470618120,
17
- "corpus_sha256": "8b630782362b05249b3be94b0213850258422be5ddeb62fcf703817b0bf9e11a",
18
- "corpus_rows": 331,
19
- "languages": [
20
- "de",
21
- "en",
22
- "es",
23
- "fil",
24
- "fr",
25
- "hi",
26
- "id",
27
- "it",
28
- "ja",
29
- "kn",
30
- "ml",
31
- "ta",
32
- "te",
33
- "zh"
34
- ],
35
- "corpus_token_lengths": {
36
- "p50": 13.0,
37
- "p95": 19.0,
38
- "max": 26
39
- },
40
- "measurements": [
41
- {
42
- "threads": 1,
43
- "scenario": "multilingual_test",
44
- "model_path": {
45
- "p50_ms": 20.395149500000002,
46
- "p95_ms": 25.477123599999995,
47
- "p99_ms": 28.2604041,
48
- "mean_ms": 20.432651312,
49
- "max_ms": 46.969913
50
- },
51
- "tokenization": {
52
- "p50_ms": 0.23074050000000002,
53
- "p95_ms": 0.29089349999999997,
54
- "p99_ms": 0.33883417,
55
- "mean_ms": 0.234400798,
56
- "max_ms": 0.640549
57
- },
58
- "forward_and_probability": {
59
- "p50_ms": 20.162170500000002,
60
- "p95_ms": 25.207171749999997,
61
- "p99_ms": 28.02515483,
62
- "mean_ms": 20.198250514,
63
- "max_ms": 46.709714
64
- }
65
- },
66
- {
67
- "threads": 1,
68
- "scenario": "full_64_tokens",
69
- "model_path": {
70
- "p50_ms": 44.6955575,
71
- "p95_ms": 50.545173899999995,
72
- "p99_ms": 53.76246217,
73
- "mean_ms": 45.379929543,
74
- "max_ms": 57.243489
75
- },
76
- "tokenization": {
77
- "p50_ms": 0.6161775,
78
- "p95_ms": 0.7613426,
79
- "p99_ms": 0.84228681,
80
- "mean_ms": 0.631765294,
81
- "max_ms": 0.891216
82
- },
83
- "forward_and_probability": {
84
- "p50_ms": 44.084312499999996,
85
- "p95_ms": 49.9083266,
86
- "p99_ms": 53.07674325,
87
- "mean_ms": 44.748164249,
88
- "max_ms": 56.405244
89
- }
90
- },
91
- {
92
- "threads": 2,
93
- "scenario": "multilingual_test",
94
- "model_path": {
95
- "p50_ms": 19.406584000000002,
96
- "p95_ms": 23.1596329,
97
- "p99_ms": 25.6503671,
98
- "mean_ms": 19.536068109,
99
- "max_ms": 28.630568
100
- },
101
- "tokenization": {
102
- "p50_ms": 0.23920799999999998,
103
- "p95_ms": 0.31088855,
104
- "p99_ms": 0.36521733999999995,
105
- "mean_ms": 0.244807045,
106
- "max_ms": 0.475011
107
- },
108
- "forward_and_probability": {
109
- "p50_ms": 19.154210499999998,
110
- "p95_ms": 22.828605549999995,
111
- "p99_ms": 25.33145532,
112
- "mean_ms": 19.291261064,
113
- "max_ms": 28.31147
114
- }
115
- },
116
- {
117
- "threads": 2,
118
- "scenario": "full_64_tokens",
119
- "model_path": {
120
- "p50_ms": 38.023134999999996,
121
- "p95_ms": 45.51507395,
122
- "p99_ms": 53.98291254,
123
- "mean_ms": 38.847798787,
124
- "max_ms": 62.944328
125
- },
126
- "tokenization": {
127
- "p50_ms": 0.636223,
128
- "p95_ms": 0.8138672499999998,
129
- "p99_ms": 0.9455087999999999,
130
- "mean_ms": 0.659222932,
131
- "max_ms": 1.44213
132
- },
133
- "forward_and_probability": {
134
- "p50_ms": 37.383997,
135
- "p95_ms": 44.81712879999999,
136
- "p99_ms": 53.05854066,
137
- "mean_ms": 38.188575855,
138
- "max_ms": 61.995607
139
- }
140
- },
141
- {
142
- "threads": 4,
143
- "scenario": "multilingual_test",
144
- "model_path": {
145
- "p50_ms": 13.1820115,
146
- "p95_ms": 19.21180095,
147
- "p99_ms": 24.81463212999999,
148
- "mean_ms": 13.731945525,
149
- "max_ms": 89.107149
150
- },
151
- "tokenization": {
152
- "p50_ms": 0.2276755,
153
- "p95_ms": 0.30882,
154
- "p99_ms": 0.3798859499999996,
155
- "mean_ms": 0.23842783,
156
- "max_ms": 4.009645
157
- },
158
- "forward_and_probability": {
159
- "p50_ms": 12.937595,
160
- "p95_ms": 18.841387699999995,
161
- "p99_ms": 24.56872725999999,
162
- "mean_ms": 13.493517695,
163
- "max_ms": 88.911056
164
- }
165
- },
166
- {
167
- "threads": 4,
168
- "scenario": "full_64_tokens",
169
- "model_path": {
170
- "p50_ms": 26.392652,
171
- "p95_ms": 32.489633899999994,
172
- "p99_ms": 41.04232582999999,
173
- "mean_ms": 27.24307507,
174
- "max_ms": 93.673078
175
- },
176
- "tokenization": {
177
- "p50_ms": 0.614555,
178
- "p95_ms": 0.7339711499999999,
179
- "p99_ms": 0.9166563899999999,
180
- "mean_ms": 0.624646764,
181
- "max_ms": 2.092334
182
- },
183
- "forward_and_probability": {
184
- "p50_ms": 25.7561225,
185
- "p95_ms": 31.824941300000003,
186
- "p99_ms": 40.46172110999999,
187
- "mean_ms": 26.618428306,
188
- "max_ms": 93.090937
189
- }
190
- }
191
- ],
192
- "scope": "Local virtual CPU; reference CPU unspecified. Excludes Vxpipe boundary and queueing; no whole-path acceptance claim.",
193
- "peak_process_rss_mib": 1168.8125
194
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
reports/eval-test.json DELETED
@@ -1,81 +0,0 @@
1
- {
2
- "count": 331,
3
- "accuracy": 1.0,
4
- "macro_f1": 1.0,
5
- "false_interruption_rate": 0.0,
6
- "missed_interruption_rate": 0.0,
7
- "per_class": {
8
- "backchannel": {
9
- "precision": 1.0,
10
- "recall": 1.0,
11
- "f1": 1.0,
12
- "count": 109
13
- },
14
- "barge_in": {
15
- "precision": 1.0,
16
- "recall": 1.0,
17
- "f1": 1.0,
18
- "count": 222
19
- }
20
- },
21
- "per_language": {
22
- "de": {
23
- "accuracy": 1.0,
24
- "count": 17
25
- },
26
- "en": {
27
- "accuracy": 1.0,
28
- "count": 81
29
- },
30
- "es": {
31
- "accuracy": 1.0,
32
- "count": 36
33
- },
34
- "fil": {
35
- "accuracy": 1.0,
36
- "count": 18
37
- },
38
- "fr": {
39
- "accuracy": 1.0,
40
- "count": 18
41
- },
42
- "hi": {
43
- "accuracy": 1.0,
44
- "count": 18
45
- },
46
- "id": {
47
- "accuracy": 1.0,
48
- "count": 17
49
- },
50
- "it": {
51
- "accuracy": 1.0,
52
- "count": 18
53
- },
54
- "ja": {
55
- "accuracy": 1.0,
56
- "count": 18
57
- },
58
- "kn": {
59
- "accuracy": 1.0,
60
- "count": 18
61
- },
62
- "ml": {
63
- "accuracy": 1.0,
64
- "count": 18
65
- },
66
- "ta": {
67
- "accuracy": 1.0,
68
- "count": 18
69
- },
70
- "te": {
71
- "accuracy": 1.0,
72
- "count": 18
73
- },
74
- "zh": {
75
- "accuracy": 1.0,
76
- "count": 18
77
- }
78
- },
79
- "split": "test",
80
- "threshold": 0.5
81
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
reports/eval-validation.json DELETED
@@ -1,81 +0,0 @@
1
- {
2
- "count": 333,
3
- "accuracy": 0.993993993993994,
4
- "macro_f1": 0.9932432432432432,
5
- "false_interruption_rate": 0.009009009009009009,
6
- "missed_interruption_rate": 0.0045045045045045045,
7
- "per_class": {
8
- "backchannel": {
9
- "precision": 0.990990990990991,
10
- "recall": 0.990990990990991,
11
- "f1": 0.990990990990991,
12
- "count": 111
13
- },
14
- "barge_in": {
15
- "precision": 0.9954954954954955,
16
- "recall": 0.9954954954954955,
17
- "f1": 0.9954954954954955,
18
- "count": 222
19
- }
20
- },
21
- "per_language": {
22
- "de": {
23
- "accuracy": 1.0,
24
- "count": 18
25
- },
26
- "en": {
27
- "accuracy": 1.0,
28
- "count": 81
29
- },
30
- "es": {
31
- "accuracy": 1.0,
32
- "count": 36
33
- },
34
- "fil": {
35
- "accuracy": 1.0,
36
- "count": 18
37
- },
38
- "fr": {
39
- "accuracy": 1.0,
40
- "count": 18
41
- },
42
- "hi": {
43
- "accuracy": 1.0,
44
- "count": 18
45
- },
46
- "id": {
47
- "accuracy": 1.0,
48
- "count": 18
49
- },
50
- "it": {
51
- "accuracy": 1.0,
52
- "count": 18
53
- },
54
- "ja": {
55
- "accuracy": 1.0,
56
- "count": 18
57
- },
58
- "kn": {
59
- "accuracy": 1.0,
60
- "count": 18
61
- },
62
- "ml": {
63
- "accuracy": 1.0,
64
- "count": 18
65
- },
66
- "ta": {
67
- "accuracy": 0.9444444444444444,
68
- "count": 18
69
- },
70
- "te": {
71
- "accuracy": 1.0,
72
- "count": 18
73
- },
74
- "zh": {
75
- "accuracy": 0.9444444444444444,
76
- "count": 18
77
- }
78
- },
79
- "split": "validation",
80
- "threshold": 0.5
81
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
reports/source-manifest.json DELETED
@@ -1,73 +0,0 @@
1
- {
2
- ".gitattributes": "d76d90c6cfbe7c1e793b13dc70485f1a71c80b19c72e7f3a2829786ac9eef810",
3
- ".gitignore": "01e2dc4c57a1279bf408ae19a88c62b2e29e92ded5bdc337ca4cadff47dd92fc",
4
- ".python-version": "a876e0b10411037a012498b9fe18d9bc1df32ed8b722a13564dc944ddcfd9135",
5
- "AGENTS.md": "e9e62c3227ce52f8ba930c3f5119816c168a7f1477f102eb7fd781598c68b1f1",
6
- "barge_in_detector/__init__.py": "28de6fe1d126dec7b9cd85273f805831b24fb7c443ed2707c47552caede29072",
7
- "barge_in_detector/__main__.py": "8efaf592230270b84d1ba8513bdb244b104aba73a517f0734197ccbfa18afb8a",
8
- "barge_in_detector/catalogue.py": "8b59351941bce2a92420ea0b13989a4c002a52d23c6279c10331645fac9796e2",
9
- "barge_in_detector/cleaner.py": "58bcb1c3f21f66aecb534cbc20e07d5350c3d8d4c92e2084c4943ca9493c3a22",
10
- "barge_in_detector/cli.py": "a68d6d7348ab34520f0d8ba88731367da6af0fb574548a290e43f45bd8941517",
11
- "barge_in_detector/generation/__init__.py": "b81aff66375501799977abd59786a5ad9da99f6998febc5fac64a016a6cea3fd",
12
- "barge_in_detector/generation/config.py": "844d5945b93c44ff04a35f6f161a1ab29a29d39ffacc7af1b268276818d20662",
13
- "barge_in_detector/generation/llm.py": "972ee568321255f6ceeee3f34c323eedac5b5ec83062d67cc662b53eee86b7a9",
14
- "barge_in_detector/generation/prompts.py": "04c3b61b4ef0a7cbd5edcb0438f544b564c602e0d213291dafbcff9fe1f086ce",
15
- "barge_in_detector/generation/responses.py": "cb5849ddbc90e800bd8c20a17896c0f7cb406b6eacf7126ddc2a1d6468c1ef5d",
16
- "barge_in_detector/generation/utterance_criteria.py": "78b938e758fa5f8d8d6dedbc48096747a8521eca6812822ef8cccfd70ff62c72",
17
- "barge_in_detector/generation/utterance_runner.py": "51d383bee865e79242997a6ef1a3ac2add896fb86969689cdbd7a8d26cf734a4",
18
- "barge_in_detector/generation/utterances.py": "535f7c54e9dff1ed61903713b8c825e98b453932b753da61b46d41479374cd42",
19
- "barge_in_detector/ids.py": "7a5abdf7ebfb64e5fb1588f887fd951a0df345d71c90d59b445ef2a8c1bf214e",
20
- "barge_in_detector/storage/__init__.py": "4e44293470af3982b64c05c2d17a949185c10b4bd53bf72b5ba6229f51507db4",
21
- "barge_in_detector/storage/schema.py": "0ccc76ec546a0fe1c414fed1f9f866b0103a55319f04e6d5d3bed12e83862e03",
22
- "barge_in_detector/storage/store.py": "b3b32e195dc06653702e307d7199cc380b4a74dd687ab860836d491c8166311f",
23
- "barge_in_detector/training/__init__.py": "230e5ab5c3a00f056980f02d06acc65a0efe31b7738a18caf9c7c6de1b33c1d1",
24
- "barge_in_detector/training/evaluate.py": "4762ae1a3df0b942049b1eb4dd9faad7e37b5c77bb8200f2041e9e3f30ee1e5e",
25
- "barge_in_detector/training/metrics.py": "6909e1cf3e4ac6b763c97ea9a8c26b5ee751a01fffc86c308604d5e437bbfad6",
26
- "barge_in_detector/training/split.py": "44e05118111617a5649d94d09e82228864f2f2f5405a15dde974db78d9829a29",
27
- "barge_in_detector/training/train.py": "a643f92b5c99d7c318e460dc2fdb5ae0bbb37003a8dae28fc3be60f8d37d1bd2",
28
- "bin/create-labnotes": "5306e747935b105f878fa5f144c83595debb3f32f87c7d097720975bb1729884",
29
- "data/export/utterances.csv": "556a2820e4a24ac13bab44321fb32b464c5fb225750d41a5a10ebcde7cdbab8a",
30
- "data/splits/test.csv": "8b630782362b05249b3be94b0213850258422be5ddeb62fcf703817b0bf9e11a",
31
- "data/splits/train.csv": "aab224bd0526eee915295e6f5c52076e29bb751cd0d664eefb4321843ad22533",
32
- "data/splits/validation.csv": "c6b4c526fb4b1df96754119e409cedc8c7e85c77659e1fc1c3829da70084edc1",
33
- "docs/data-format.md": "5d14d3aac1e432157c03bdf10cabf87e7366164629c70fb5904d632f472254ff",
34
- "docs/factor-catalogue.md": "651e27021557eb7ce08c3ae5af0e341dee1aba665c1b6f72ad19625768167ec5",
35
- "docs/generation.md": "1aae32dc0cb9c35904f8b2c5721ce7910494a27384a97b5f40fe0373c2d70b33",
36
- "docs/training.md": "a2a96447aa065e4cb9f1e8a4b9ec97041d41ee32736cb68115706638b562d68f",
37
- "docs/transcript-format.md": "9927370e6fb906d1dd1fabd98888d7e95d8ba2f373e7a1248e430f260c57aa32",
38
- "docs/utterance-first-decision.md": "6c9ab89b7d579d83769e09eb3b77bf1cfa70434fe0cc45e24aaa959385a064e5",
39
- "generation/config.yaml": "f4e707d4efc610b882289f29f170cd763c876c4e366f6853d1225cbb5965c88b",
40
- "generation/factors/catalogue.yaml": "1135f3a3bf6000eb1f9d41b1957a348c760e78f96fbe61a8829aa818be0de5b5",
41
- "generation/factors/constraints.yaml": "422318e1010483e4c5f7eb0de99170eb906b8d21927183a5d6581f13c21ac59c",
42
- "generation/factors/domains.yaml": "19605dbdd0bb4044c481d74ce5c798534d61dd6c7c46d4edcb2f800c4aa53fc5",
43
- "generation/factors/intents.yaml": "a3d343addabde711a2395e1ed598f432dd00e547afa962255ae246e17874ae63",
44
- "generation/factors/languages.yaml": "c5f0a5b753c588782c536108184b12615a54c9b8f4b4cbf0073c0d8aec1a376e",
45
- "generation/factors/utterances.yaml": "a045546c59d59588a261b03e1f1dc9dc147146f93d3381a813c0920825c435c4",
46
- "generation/plans/english.yaml": "f1e7e17e2249bb736cd5cb75b1af0f0babbe3b2320e12b2b812346ff5c12d725",
47
- "generation/plans/utterance-all.yaml": "20349fb83cd17d067736e4d26d746be7a77fa7cd7a8d41569fc44335cefb0a5d",
48
- "generation/plans/utterance-sample.yaml": "7ab683e2c38af4f0105c957971efa2caaeca17c7bc2e11397e362591d0a31276",
49
- "generation/prompts/generator_utterances_v5.md": "0a2e3eeeb7b7c7ff1ad2e2de8973af5469fcb8c369d9092d251c7590477b1266",
50
- "labnotes/20261114-0907-project-brief-setup.md": "ab069c92129480d35758375cf5c57cbce96a643f007ed1d7d5941a6718bf0c51",
51
- "labnotes/20261114-1023-factor-catalogue-draft.md": "6e3c2261c5bdb325548f59763ababe178d1f442f6440497c47d53c2773b513ee",
52
- "labnotes/20261114-1541-synthetic-data-generator.md": "1b5f4bb84e3c224164d4e295b7639aad3a27d82d01948220b0c177fd1e93ea2b",
53
- "labnotes/20261115-0100-utterance-first-generation.md": "5e81c46e2799a970b783e660b06ed16fdf4ea7bc43859887a4ae2d42fe5f6d0f",
54
- "labnotes/20261115-1211-vastai-model-training.md": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
55
- "labnotes/issues/.gitkeep": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
56
- "labnotes/milestones/index.md": "0f3b3889fbbb7ea869c3c4e264d1da7cbd4535b28890113f35edcb8cf39b7463",
57
- "labnotes/milestones/multilingual-text-barge-in-classifier.md": "c292dbefd18e5056cf219f8bd5c66403bcf291eab0cb9589840beb198ce5a74d",
58
- "labnotes/milestones/utterance-first-backchannel-classifier.md": "e3bf3bd0ce94ac80d5b3ac4d697535cdb95fda30d7d4d1cad282b2d7e40c0d16",
59
- "pyproject.toml": "4aef3a80e879d58c4c559b1f276e4b51c57006583cce3fd7a302e91336d9bbef",
60
- "tests/fake_llm.py": "2c604f09ff23619225e0c62a57466f300191fd8391777ac7cf05e5556dee4f33",
61
- "tests/fixtures/cleaner_golden.yaml": "3e013d3775a619a08432fe6c0ced5558934e4dbcf98a4afa651a0bd4bfc1c1db",
62
- "tests/test_catalogue.py": "9fbfc7bc848e5db38d34ccb08f6bf10bd0908b21cbcc73d70d337c1e0dbae2ca",
63
- "tests/test_cleaner.py": "048942dafd6e91772d7b17515606cb6c7f44c2225b876542d8598d19116a3d47",
64
- "tests/test_gemini_live.py": "64f4ba3a80503ad7e0a43bfbaf7e43fbecbbc5821a41b189c08da5eff390e884",
65
- "tests/test_ids.py": "3802dc4bd8b75f8764c1d4eafbc882b614096d5d871ec4f5d001990bb3f87d89",
66
- "tests/test_llm.py": "eb8255bc88a0d9b8dde268ae67d700f5e2406c87b82cbda27dbba920ba9ad19a",
67
- "tests/test_metrics.py": "072656cb74e8750f1a9a52f5d6dadb2ac3250e5f1cd538d6e64f3bd5a46d9567",
68
- "tests/test_split.py": "d4556a43db6f706e6bac282bb5749792ddc66f12ba439d0b7fa1ad6cf666ff63",
69
- "tests/test_store.py": "f23ab069512838929f7f0f91f8439ceaac4ba38484a6b235750fd7adf672614d",
70
- "tests/test_utterance_criteria.py": "bb076041b038a6b707a6df61ee708408bec89922c011fef8dce0824991691a53",
71
- "tests/test_utterances.py": "3a48f590d83cddc4db2f57099f5f90bff3b965195f0aeb1adcd663a27bc1a387",
72
- "uv.lock": "3992e539e9e18aeb3182978ef9ef0b6ae8c0e15d649416e2de11b9ba98317ba8"
73
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
reports/training-metrics.json DELETED
@@ -1,519 +0,0 @@
1
- {
2
- "args": {
3
- "out": "models/e5-small-run1",
4
- "model": "intfloat/multilingual-e5-small",
5
- "splits": "data/splits",
6
- "epochs": "5",
7
- "batch_size": "32",
8
- "lr": "3e-05",
9
- "warmup": "0.1",
10
- "max_length": "64",
11
- "seed": "20261115"
12
- },
13
- "best": {
14
- "epoch": 5,
15
- "train_loss": 0.02041708185817256,
16
- "seconds": 1.7,
17
- "validation": {
18
- "count": 333,
19
- "accuracy": 0.993993993993994,
20
- "macro_f1": 0.9932432432432432,
21
- "false_interruption_rate": 0.009009009009009009,
22
- "missed_interruption_rate": 0.0045045045045045045,
23
- "per_class": {
24
- "backchannel": {
25
- "precision": 0.990990990990991,
26
- "recall": 0.990990990990991,
27
- "f1": 0.990990990990991,
28
- "count": 111
29
- },
30
- "barge_in": {
31
- "precision": 0.9954954954954955,
32
- "recall": 0.9954954954954955,
33
- "f1": 0.9954954954954955,
34
- "count": 222
35
- }
36
- },
37
- "per_language": {
38
- "de": {
39
- "accuracy": 1.0,
40
- "count": 18
41
- },
42
- "en": {
43
- "accuracy": 1.0,
44
- "count": 81
45
- },
46
- "es": {
47
- "accuracy": 1.0,
48
- "count": 36
49
- },
50
- "fil": {
51
- "accuracy": 1.0,
52
- "count": 18
53
- },
54
- "fr": {
55
- "accuracy": 1.0,
56
- "count": 18
57
- },
58
- "hi": {
59
- "accuracy": 1.0,
60
- "count": 18
61
- },
62
- "id": {
63
- "accuracy": 1.0,
64
- "count": 18
65
- },
66
- "it": {
67
- "accuracy": 1.0,
68
- "count": 18
69
- },
70
- "ja": {
71
- "accuracy": 1.0,
72
- "count": 18
73
- },
74
- "kn": {
75
- "accuracy": 1.0,
76
- "count": 18
77
- },
78
- "ml": {
79
- "accuracy": 1.0,
80
- "count": 18
81
- },
82
- "ta": {
83
- "accuracy": 0.9444444444444444,
84
- "count": 18
85
- },
86
- "te": {
87
- "accuracy": 1.0,
88
- "count": 18
89
- },
90
- "zh": {
91
- "accuracy": 0.9444444444444444,
92
- "count": 18
93
- }
94
- }
95
- }
96
- },
97
- "history": [
98
- {
99
- "epoch": 1,
100
- "train_loss": 0.48989867342240884,
101
- "seconds": 2.3,
102
- "validation": {
103
- "count": 333,
104
- "accuracy": 0.984984984984985,
105
- "macro_f1": 0.98306980527734,
106
- "false_interruption_rate": 0.02702702702702703,
107
- "missed_interruption_rate": 0.009009009009009009,
108
- "per_class": {
109
- "backchannel": {
110
- "precision": 0.9818181818181818,
111
- "recall": 0.972972972972973,
112
- "f1": 0.9773755656108598,
113
- "count": 111
114
- },
115
- "barge_in": {
116
- "precision": 0.9865470852017937,
117
- "recall": 0.990990990990991,
118
- "f1": 0.9887640449438202,
119
- "count": 222
120
- }
121
- },
122
- "per_language": {
123
- "de": {
124
- "accuracy": 0.9444444444444444,
125
- "count": 18
126
- },
127
- "en": {
128
- "accuracy": 1.0,
129
- "count": 81
130
- },
131
- "es": {
132
- "accuracy": 1.0,
133
- "count": 36
134
- },
135
- "fil": {
136
- "accuracy": 1.0,
137
- "count": 18
138
- },
139
- "fr": {
140
- "accuracy": 1.0,
141
- "count": 18
142
- },
143
- "hi": {
144
- "accuracy": 1.0,
145
- "count": 18
146
- },
147
- "id": {
148
- "accuracy": 1.0,
149
- "count": 18
150
- },
151
- "it": {
152
- "accuracy": 1.0,
153
- "count": 18
154
- },
155
- "ja": {
156
- "accuracy": 1.0,
157
- "count": 18
158
- },
159
- "kn": {
160
- "accuracy": 0.9444444444444444,
161
- "count": 18
162
- },
163
- "ml": {
164
- "accuracy": 0.9444444444444444,
165
- "count": 18
166
- },
167
- "ta": {
168
- "accuracy": 0.9444444444444444,
169
- "count": 18
170
- },
171
- "te": {
172
- "accuracy": 1.0,
173
- "count": 18
174
- },
175
- "zh": {
176
- "accuracy": 0.9444444444444444,
177
- "count": 18
178
- }
179
- }
180
- }
181
- },
182
- {
183
- "epoch": 2,
184
- "train_loss": 0.08939109126275235,
185
- "seconds": 1.7,
186
- "validation": {
187
- "count": 333,
188
- "accuracy": 0.990990990990991,
189
- "macro_f1": 0.9898875380862241,
190
- "false_interruption_rate": 0.009009009009009009,
191
- "missed_interruption_rate": 0.009009009009009009,
192
- "per_class": {
193
- "backchannel": {
194
- "precision": 0.9821428571428571,
195
- "recall": 0.990990990990991,
196
- "f1": 0.9865470852017937,
197
- "count": 111
198
- },
199
- "barge_in": {
200
- "precision": 0.995475113122172,
201
- "recall": 0.990990990990991,
202
- "f1": 0.9932279909706545,
203
- "count": 222
204
- }
205
- },
206
- "per_language": {
207
- "de": {
208
- "accuracy": 1.0,
209
- "count": 18
210
- },
211
- "en": {
212
- "accuracy": 1.0,
213
- "count": 81
214
- },
215
- "es": {
216
- "accuracy": 1.0,
217
- "count": 36
218
- },
219
- "fil": {
220
- "accuracy": 1.0,
221
- "count": 18
222
- },
223
- "fr": {
224
- "accuracy": 1.0,
225
- "count": 18
226
- },
227
- "hi": {
228
- "accuracy": 1.0,
229
- "count": 18
230
- },
231
- "id": {
232
- "accuracy": 1.0,
233
- "count": 18
234
- },
235
- "it": {
236
- "accuracy": 1.0,
237
- "count": 18
238
- },
239
- "ja": {
240
- "accuracy": 1.0,
241
- "count": 18
242
- },
243
- "kn": {
244
- "accuracy": 1.0,
245
- "count": 18
246
- },
247
- "ml": {
248
- "accuracy": 0.9444444444444444,
249
- "count": 18
250
- },
251
- "ta": {
252
- "accuracy": 0.9444444444444444,
253
- "count": 18
254
- },
255
- "te": {
256
- "accuracy": 1.0,
257
- "count": 18
258
- },
259
- "zh": {
260
- "accuracy": 0.9444444444444444,
261
- "count": 18
262
- }
263
- }
264
- }
265
- },
266
- {
267
- "epoch": 3,
268
- "train_loss": 0.03671385896476832,
269
- "seconds": 1.7,
270
- "validation": {
271
- "count": 333,
272
- "accuracy": 0.987987987987988,
273
- "macro_f1": 0.9865465416936006,
274
- "false_interruption_rate": 0.009009009009009009,
275
- "missed_interruption_rate": 0.013513513513513514,
276
- "per_class": {
277
- "backchannel": {
278
- "precision": 0.9734513274336283,
279
- "recall": 0.990990990990991,
280
- "f1": 0.9821428571428571,
281
- "count": 111
282
- },
283
- "barge_in": {
284
- "precision": 0.9954545454545455,
285
- "recall": 0.9864864864864865,
286
- "f1": 0.9909502262443439,
287
- "count": 222
288
- }
289
- },
290
- "per_language": {
291
- "de": {
292
- "accuracy": 1.0,
293
- "count": 18
294
- },
295
- "en": {
296
- "accuracy": 1.0,
297
- "count": 81
298
- },
299
- "es": {
300
- "accuracy": 1.0,
301
- "count": 36
302
- },
303
- "fil": {
304
- "accuracy": 1.0,
305
- "count": 18
306
- },
307
- "fr": {
308
- "accuracy": 1.0,
309
- "count": 18
310
- },
311
- "hi": {
312
- "accuracy": 1.0,
313
- "count": 18
314
- },
315
- "id": {
316
- "accuracy": 1.0,
317
- "count": 18
318
- },
319
- "it": {
320
- "accuracy": 1.0,
321
- "count": 18
322
- },
323
- "ja": {
324
- "accuracy": 1.0,
325
- "count": 18
326
- },
327
- "kn": {
328
- "accuracy": 1.0,
329
- "count": 18
330
- },
331
- "ml": {
332
- "accuracy": 1.0,
333
- "count": 18
334
- },
335
- "ta": {
336
- "accuracy": 0.9444444444444444,
337
- "count": 18
338
- },
339
- "te": {
340
- "accuracy": 1.0,
341
- "count": 18
342
- },
343
- "zh": {
344
- "accuracy": 0.8333333333333334,
345
- "count": 18
346
- }
347
- }
348
- }
349
- },
350
- {
351
- "epoch": 4,
352
- "train_loss": 0.025003968517888674,
353
- "seconds": 1.7,
354
- "validation": {
355
- "count": 333,
356
- "accuracy": 0.987987987987988,
357
- "macro_f1": 0.9865465416936006,
358
- "false_interruption_rate": 0.009009009009009009,
359
- "missed_interruption_rate": 0.013513513513513514,
360
- "per_class": {
361
- "backchannel": {
362
- "precision": 0.9734513274336283,
363
- "recall": 0.990990990990991,
364
- "f1": 0.9821428571428571,
365
- "count": 111
366
- },
367
- "barge_in": {
368
- "precision": 0.9954545454545455,
369
- "recall": 0.9864864864864865,
370
- "f1": 0.9909502262443439,
371
- "count": 222
372
- }
373
- },
374
- "per_language": {
375
- "de": {
376
- "accuracy": 1.0,
377
- "count": 18
378
- },
379
- "en": {
380
- "accuracy": 1.0,
381
- "count": 81
382
- },
383
- "es": {
384
- "accuracy": 1.0,
385
- "count": 36
386
- },
387
- "fil": {
388
- "accuracy": 1.0,
389
- "count": 18
390
- },
391
- "fr": {
392
- "accuracy": 1.0,
393
- "count": 18
394
- },
395
- "hi": {
396
- "accuracy": 1.0,
397
- "count": 18
398
- },
399
- "id": {
400
- "accuracy": 1.0,
401
- "count": 18
402
- },
403
- "it": {
404
- "accuracy": 1.0,
405
- "count": 18
406
- },
407
- "ja": {
408
- "accuracy": 1.0,
409
- "count": 18
410
- },
411
- "kn": {
412
- "accuracy": 1.0,
413
- "count": 18
414
- },
415
- "ml": {
416
- "accuracy": 1.0,
417
- "count": 18
418
- },
419
- "ta": {
420
- "accuracy": 0.9444444444444444,
421
- "count": 18
422
- },
423
- "te": {
424
- "accuracy": 1.0,
425
- "count": 18
426
- },
427
- "zh": {
428
- "accuracy": 0.8333333333333334,
429
- "count": 18
430
- }
431
- }
432
- }
433
- },
434
- {
435
- "epoch": 5,
436
- "train_loss": 0.02041708185817256,
437
- "seconds": 1.7,
438
- "validation": {
439
- "count": 333,
440
- "accuracy": 0.993993993993994,
441
- "macro_f1": 0.9932432432432432,
442
- "false_interruption_rate": 0.009009009009009009,
443
- "missed_interruption_rate": 0.0045045045045045045,
444
- "per_class": {
445
- "backchannel": {
446
- "precision": 0.990990990990991,
447
- "recall": 0.990990990990991,
448
- "f1": 0.990990990990991,
449
- "count": 111
450
- },
451
- "barge_in": {
452
- "precision": 0.9954954954954955,
453
- "recall": 0.9954954954954955,
454
- "f1": 0.9954954954954955,
455
- "count": 222
456
- }
457
- },
458
- "per_language": {
459
- "de": {
460
- "accuracy": 1.0,
461
- "count": 18
462
- },
463
- "en": {
464
- "accuracy": 1.0,
465
- "count": 81
466
- },
467
- "es": {
468
- "accuracy": 1.0,
469
- "count": 36
470
- },
471
- "fil": {
472
- "accuracy": 1.0,
473
- "count": 18
474
- },
475
- "fr": {
476
- "accuracy": 1.0,
477
- "count": 18
478
- },
479
- "hi": {
480
- "accuracy": 1.0,
481
- "count": 18
482
- },
483
- "id": {
484
- "accuracy": 1.0,
485
- "count": 18
486
- },
487
- "it": {
488
- "accuracy": 1.0,
489
- "count": 18
490
- },
491
- "ja": {
492
- "accuracy": 1.0,
493
- "count": 18
494
- },
495
- "kn": {
496
- "accuracy": 1.0,
497
- "count": 18
498
- },
499
- "ml": {
500
- "accuracy": 1.0,
501
- "count": 18
502
- },
503
- "ta": {
504
- "accuracy": 0.9444444444444444,
505
- "count": 18
506
- },
507
- "te": {
508
- "accuracy": 1.0,
509
- "count": 18
510
- },
511
- "zh": {
512
- "accuracy": 0.9444444444444444,
513
- "count": 18
514
- }
515
- }
516
- }
517
- }
518
- ]
519
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
tests/test_data.py DELETED
@@ -1,43 +0,0 @@
1
- import sys
2
- from types import SimpleNamespace
3
-
4
- import pytest
5
-
6
- from barge_in_detector_model.data import read_csv, resolve_splits
7
-
8
-
9
- def make_splits(path):
10
- path.mkdir(parents=True)
11
- for name in ["train", "validation", "test"]:
12
- (path / f"{name}.csv").write_text("text_clean,label,language\nstop,barge_in,en\n")
13
-
14
-
15
- def test_local_splits_do_not_require_hub(tmp_path):
16
- make_splits(tmp_path / "splits")
17
- directory, revision = resolve_splits(tmp_path / "splits")
18
- assert revision is None
19
- assert read_csv(directory / "test.csv")[0]["label"] == "barge_in"
20
- with pytest.raises(ValueError, match="missing"):
21
- resolve_splits(tmp_path / "absent")
22
-
23
-
24
- def test_hub_download_uses_resolved_immutable_revision(tmp_path, monkeypatch):
25
- make_splits(tmp_path / "data/splits")
26
- seen = {}
27
-
28
- def download(**kwargs):
29
- seen.update(kwargs)
30
- return str(tmp_path)
31
-
32
- fake = SimpleNamespace(
33
- HfApi=lambda: SimpleNamespace(
34
- dataset_info=lambda *a, **k: SimpleNamespace(sha="fixed-commit")
35
- ),
36
- snapshot_download=download,
37
- )
38
- monkeypatch.setitem(sys.modules, "huggingface_hub", fake)
39
- directory, revision = resolve_splits(None, repo_id="owner/data", revision="v1")
40
- assert revision == "fixed-commit"
41
- assert seen["revision"] == "fixed-commit"
42
- assert seen["repo_type"] == "dataset"
43
- assert directory == tmp_path / "data/splits"
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
tests/test_metrics.py DELETED
@@ -1,29 +0,0 @@
1
- """Evaluation metrics for the two labels."""
2
-
3
- import pytest
4
-
5
- from barge_in_detector_model.metrics import compute
6
-
7
-
8
- def test_metrics_report_false_and_missed_interruptions():
9
- labels = ["backchannel", "backchannel", "barge_in", "barge_in", "barge_in"]
10
- preds = ["backchannel", "barge_in", "barge_in", "barge_in", "backchannel"]
11
- languages = ["en", "en", "en", "hi", "hi"]
12
- m = compute(labels, preds, languages)
13
- assert m["accuracy"] == pytest.approx(3 / 5)
14
- # A backchannel predicted as barge-in cuts the agent off for nothing.
15
- assert m["false_interruption_rate"] == pytest.approx(1 / 2)
16
- # A barge-in predicted as backchannel lets the agent talk over the caller.
17
- assert m["missed_interruption_rate"] == pytest.approx(1 / 3)
18
- assert m["per_class"]["barge_in"]["precision"] == pytest.approx(2 / 3)
19
- assert m["per_class"]["barge_in"]["recall"] == pytest.approx(2 / 3)
20
- assert m["per_language"]["en"]["accuracy"] == pytest.approx(2 / 3)
21
- assert m["per_language"]["hi"]["count"] == 2
22
- assert 0 < m["macro_f1"] < 1
23
-
24
-
25
- def test_perfect_predictions():
26
- labels = ["backchannel", "barge_in"]
27
- m = compute(labels, labels, ["en", "en"])
28
- assert m["accuracy"] == m["macro_f1"] == 1.0
29
- assert m["false_interruption_rate"] == m["missed_interruption_rate"] == 0.0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
uv.lock DELETED
The diff for this file is too large to render. See raw diff