Text Classification
Transformers
ONNX
Safetensors
bert
barge-in
backchannel
multilingual
experimental
text-embeddings-inference
Instructions to use HashNuke/barge-in-detector with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use HashNuke/barge-in-detector with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="HashNuke/barge-in-detector")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSequenceClassification tokenizer = AutoTokenizer.from_pretrained("HashNuke/barge-in-detector") model = AutoModelForSequenceClassification.from_pretrained("HashNuke/barge-in-detector", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Akash Manohar commited on
Commit ·
e5336e8
1
Parent(s): a12cfe5
Keep only model release artifacts and provenance
Browse filesMove training, inference, evaluation, benchmarks, tests and labnotes to GitHub. Preserve weights/tokenizer hashes and pin the release-only dataset and canonical source revisions.
- .python-version +0 -1
- AGENTS.md +19 -95
- README.md +51 -82
- barge_in_detector_model/__init__.py +0 -1
- barge_in_detector_model/data.py +0 -39
- barge_in_detector_model/evaluate.py +0 -111
- barge_in_detector_model/infer.py +0 -27
- barge_in_detector_model/metrics.py +0 -46
- barge_in_detector_model/train.py +0 -140
- bin/create-labnotes +0 -59
- experiments/benchmark_cpu.py +0 -140
- experiments/benchmark_cpu_cold.py +0 -80
- experiments/evaluate_english_challenge.py +0 -132
- labnotes/20261115-1332-repository-split.md +0 -41
- labnotes/issues/.gitkeep +0 -0
- labnotes/milestones/experimental-checkpoint-release.md +0 -30
- labnotes/milestones/index.md +0 -7
- labnotes/milestones/short-interruption-recall.md +0 -28
- pyproject.toml +0 -25
- release.json +7 -4
- reports/challenge-predictions.csv +0 -99
- reports/challenge-report.json +0 -438
- reports/cpu-benchmark.json +0 -94
- reports/cpu-cold-benchmark.json +0 -33
- reports/cpu-thread-sweep.json +0 -194
- reports/eval-test.json +0 -81
- reports/eval-validation.json +0 -81
- reports/source-manifest.json +0 -73
- reports/training-metrics.json +0 -519
- tests/test_data.py +0 -43
- tests/test_metrics.py +0 -29
- uv.lock +0 -0
.python-version
DELETED
|
@@ -1 +0,0 @@
|
|
| 1 |
-
3.14
|
|
|
|
|
|
AGENTS.md
CHANGED
|
@@ -1,95 +1,19 @@
|
|
| 1 |
-
#
|
| 2 |
-
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
Context and an ambiguous/needs-context class are unavailable.
|
| 21 |
-
- At most 64 tokens; keep the first 64. Evaluate partial transcripts and
|
| 22 |
-
transitions from backchannel to interruption. No universal ignored-word list.
|
| 23 |
-
- Output `p_barge_in` in [0,1]. Current experimental probabilities are not yet
|
| 24 |
-
calibrated; thresholds belong to the consumer. Use validation for tuning.
|
| 25 |
-
- CPU inference only. Warmed batch-one model-path p99 <=20 ms and whole path
|
| 26 |
-
including Vxpipe boundary <=50 ms on the designated reference CPU. Report
|
| 27 |
-
p50/p95/p99, hardware, threads, memory, warm-up and token lengths. Local VM
|
| 28 |
-
measurements do not satisfy reference-CPU acceptance gates.
|
| 29 |
-
- Fourteen languages: English (US/India/Singapore), Hindi, Kannada, Tamil,
|
| 30 |
-
Telugu, Malayalam, Mandarin, Japanese, Indonesian, Filipino, French, German,
|
| 31 |
-
Spanish and Italian. Roll out English/Hindi first, then Kannada/Tamil.
|
| 32 |
-
Native scripts, optionally mixed with English; no romanization/pinyin.
|
| 33 |
-
Language support requires evaluation evidence, not tokenizer coverage.
|
| 34 |
-
- Tier A owner verification: English, Hindi, Kannada, Tamil. Other languages
|
| 35 |
-
remain synthetic/unverified unless native-speaker checks are recorded.
|
| 36 |
-
- Report false interruptions and missed interruptions, per-language results,
|
| 37 |
-
short-input slices, prefix behaviour and calibration. Keep synthetic and
|
| 38 |
-
owner-reviewed evidence separate; a perfect synthetic score is insufficient.
|
| 39 |
-
- Prefer permissive base licences. Preserve the MIT attribution for the
|
| 40 |
-
intfloat/multilingual-e5-small base model.
|
| 41 |
-
|
| 42 |
-
## Tools and remote training
|
| 43 |
-
|
| 44 |
-
- Use uv; add dependencies with `uv add` and keep uv.lock with dependency edits.
|
| 45 |
-
Always source ~/.zshrc or use zsh -lc for preconfigured tools.
|
| 46 |
-
- Always rent exactly one RTX 5090 (32 GB), in a datacenter, below $0.65/hour,
|
| 47 |
-
with good internet and the Vast.ai PyTorch image. Do not substitute another
|
| 48 |
-
GPU or rent multiple instances. Inspect existing instances before renting.
|
| 49 |
-
- All Vast.ai setup and training commands run inside tmux session `gpubox`.
|
| 50 |
-
Save logs/checkpoints to disk. Copy and verify artifacts before destroying
|
| 51 |
-
instances; follow the owner's explicit cleanup instructions.
|
| 52 |
-
- CPU evaluation and benchmarks run locally. Do not rent or restart a GPU for
|
| 53 |
-
work that the local CPU can perform. No Playwright; use agent-browser if
|
| 54 |
-
browser work is needed. Do not disable browser security to work around failures.
|
| 55 |
-
- Never print or commit tokens, authorization headers, .env, PII, caches or
|
| 56 |
-
virtualenvs. Hugging Face credentials come from existing login/environment.
|
| 57 |
-
- Root model.safetensors and tokenizer.json are versioned large artifacts;
|
| 58 |
-
use Hugging Face large-file storage. Keep experiment checkpoints under
|
| 59 |
-
ignored models/ or checkpoints/. Publish selected reports with the release.
|
| 60 |
-
|
| 61 |
-
## Labnotes and milestones
|
| 62 |
-
|
| 63 |
-
- Start at labnotes/milestones/index.md; its checklist owns implementation order.
|
| 64 |
-
Read linked prerequisites and sources before starting a checkpoint.
|
| 65 |
-
- For implementation or research, use `bin/create-labnotes two-to-four-words`.
|
| 66 |
-
Write factual progress, failures, workarounds, decisions and evidence into
|
| 67 |
-
the generated top-level labnotes file. Link it from the relevant milestone.
|
| 68 |
-
Do not create labnotes for simple status/read-only checks or running an
|
| 69 |
-
existing command. Labnotes accompany the work in its commit.
|
| 70 |
-
- Keep milestones under labnotes/milestones/, deferred issues under
|
| 71 |
-
labnotes/issues/. Use descriptive unnumbered filenames and Markdown task
|
| 72 |
-
lists. Each milestone needs a runnable outcome, prerequisites, contracts,
|
| 73 |
-
checkpoint tasks and acceptance gates. A spec is not an implementation.
|
| 74 |
-
Leave index entries unchecked until all gates pass; record blockers honestly.
|
| 75 |
-
- Historical generation/training labnotes remain in the dataset repo; link
|
| 76 |
-
them rather than copying that history here. New model work gets model labnotes.
|
| 77 |
-
- Record substantial architecture decisions and rejected alternatives under docs/.
|
| 78 |
-
|
| 79 |
-
## Checkpoints and Git
|
| 80 |
-
|
| 81 |
-
Use small coherent commits of code, focused tests, docs, labnotes and lockfile.
|
| 82 |
-
For behaviour changes, write a focused failing test before implementation.
|
| 83 |
-
Experiments/prompts/benchmarks use recorded evidence; deterministic parsers,
|
| 84 |
-
metrics, tokenization and serving get focused tests. Default tests exclude
|
| 85 |
-
network integration tests. Before finishing code changes run:
|
| 86 |
-
|
| 87 |
-
```shell
|
| 88 |
-
uv run ruff format --check
|
| 89 |
-
uv run ruff check
|
| 90 |
-
uv run pytest
|
| 91 |
-
```
|
| 92 |
-
|
| 93 |
-
Inspect status and staged diffs, preserve unrelated work, stage exact paths.
|
| 94 |
-
Do not commit or publish unless requested. Do not amend, rebase, reset,
|
| 95 |
-
force-push, delete branches or undo commits without explicit instruction.
|
|
|
|
| 1 |
+
# Model releases only
|
| 2 |
+
|
| 3 |
+
Canonical development source: https://github.com/vxpipe/barge-in-detector.
|
| 4 |
+
Keep this repo limited to model weights, tokenizer/configuration, model card,
|
| 5 |
+
licence/attribution and provenance. Dataset releases belong in
|
| 6 |
+
HashNuke/barge-in-detector-data. Training data, executable scripts, tests,
|
| 7 |
+
benchmarks, labnotes and milestones belong in GitHub.
|
| 8 |
+
|
| 9 |
+
Follow the GitHub AGENTS.md for model contracts, CPU latency/quality gates,
|
| 10 |
+
the single RTX 5090 training policy, gpubox and verification. Create checkpoint
|
| 11 |
+
labnotes there using bin/create-labnotes and use labnotes/milestones/index.md;
|
| 12 |
+
keep the helper and development records in the source repo.
|
| 13 |
+
|
| 14 |
+
Publish artifacts from a committed source revision. release.json must pin the
|
| 15 |
+
exact dataset and code commits and artifact hashes. Verify unchanged hashes,
|
| 16 |
+
public model loading and accurate limitations in the card. Do not publish
|
| 17 |
+
credentials, local environments or training checkpoints outside a selected release.
|
| 18 |
+
Use Hub large-file storage for weights/tokenizer. Use ordinary release commits;
|
| 19 |
+
do not rewrite history, reset or force-push.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
README.md
CHANGED
|
@@ -7,60 +7,27 @@ base_model: intfloat/multilingual-e5-small
|
|
| 7 |
datasets: [HashNuke/barge-in-detector-data]
|
| 8 |
tags: [barge-in, backchannel, multilingual, experimental]
|
| 9 |
---
|
| 10 |
-
# Barge-in detector: experimental
|
| 11 |
|
| 12 |
-
Classifies a
|
| 13 |
-
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
owns generation, cleaning, splits, review and historical labnotes.
|
| 17 |
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
|
|
|
|
|
|
|
|
|
| 21 |
|
| 22 |
-
|
|
|
|
|
|
|
| 23 |
|
| 24 |
-
|
| 25 |
-
are the selected epoch-5 fp32 checkpoint. Base: intfloat/multilingual-e5-small
|
| 26 |
-
(MIT); see LICENSE and NOTICE. `release.json` identifies dataset/artifact hashes
|
| 27 |
-
and the published dataset revision. `reports/source-manifest.json` captures the
|
| 28 |
-
original source snapshot, which predates the repository separation.
|
| 29 |
|
| 30 |
-
|
| 31 |
-
weighted cross-entropy, AdamW, learning rate 3e-5, 10% warm-up, seed 20261115.
|
| 32 |
-
BF16 on one RTX 5090. Epoch selected using validation macro F1 (0.99324).
|
| 33 |
-
Full training history and breakdowns are in reports/training-metrics.json.
|
| 34 |
-
|
| 35 |
-
| Evaluation | Accuracy | Missed interruptions | False interruptions |
|
| 36 |
-
| --- | --- | --- | --- |
|
| 37 |
-
| Synthetic validation (333) | 99.4% | 0.45% | 0.90% |
|
| 38 |
-
| Synthetic test (331) | 100% | 0% | 0% |
|
| 39 |
-
| Reviewed English challenge (98) | 73.5% | 54.2% | 0% |
|
| 40 |
-
| Challenge unseen in any synthetic split (93) | 72.0% | 54.2% | 0% |
|
| 41 |
-
|
| 42 |
-
The challenge was authored before inference, owner-reviewed and reconciled with
|
| 43 |
-
authorization. Final binary review labels match the frozen authored labels.
|
| 44 |
-
It is deliberately adversarial, includes repeated prefixes, and is now a
|
| 45 |
-
regression/development set. Synthetic splits share generation criteria; exact
|
| 46 |
-
cleaned-text leakage is absent, but template and semantic leakage remain unaudited.
|
| 47 |
-
These scores do not establish representative real-world accuracy. All 14 listed
|
| 48 |
-
languages have synthetic results; human-verified accuracy across all languages
|
| 49 |
-
has not been established.
|
| 50 |
-
|
| 51 |
-
## Infer on CPU
|
| 52 |
-
|
| 53 |
-
```shell
|
| 54 |
-
uv venv --python 3.14
|
| 55 |
-
uv pip install torch==2.15.0 --index-url https://download.pytorch.org/whl/cpu
|
| 56 |
-
uv pip install transformers==5.23.0 huggingface-hub==2.4.0
|
| 57 |
-
uv run --no-sync python -m barge_in_detector_model.infer "please stop for a moment"
|
| 58 |
-
```
|
| 59 |
-
|
| 60 |
-
Input must already follow the dataset cleaner: lowercase, punctuation removed,
|
| 61 |
-
native scripts optionally mixed with English. No agent context, audio or language
|
| 62 |
-
hint is needed. Keep the first 64 tokenizer tokens. Labels are 0=backchannel,
|
| 63 |
-
1=barge_in. The command emits `p_barge_in` and a label at the 0.5 threshold.
|
| 64 |
|
| 65 |
```python
|
| 66 |
import torch
|
|
@@ -70,28 +37,43 @@ repo = "HashNuke/barge-in-detector"
|
|
| 70 |
tokenizer = AutoTokenizer.from_pretrained(repo)
|
| 71 |
model = AutoModelForSequenceClassification.from_pretrained(repo).cpu().eval()
|
| 72 |
with torch.inference_mode():
|
| 73 |
-
|
| 74 |
-
|
|
|
|
|
|
|
| 75 |
```
|
| 76 |
|
| 77 |
-
|
|
|
|
|
|
|
|
|
|
| 78 |
|
| 79 |
-
|
| 80 |
-
uv sync --group dev
|
| 81 |
-
uv run python -m barge_in_detector_model.train --out models/run2 --dataset-revision DATASET_COMMIT
|
| 82 |
-
uv run python -m barge_in_detector_model.evaluate --model models/run2 --split test --dataset-revision DATASET_COMMIT
|
| 83 |
-
```
|
| 84 |
|
| 85 |
-
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
on
|
| 89 |
-
|
| 90 |
-
milestone index before starting work. Use `bin/create-labnotes` for checkpoints.
|
| 91 |
|
| 92 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 93 |
|
| 94 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 95 |
batch one, 100 warm-ups and 1,000 measured calls per scenario:
|
| 96 |
|
| 97 |
| Warm text through label | p50 | p95 | p99 |
|
|
@@ -100,20 +82,7 @@ batch one, 100 warm-ups and 1,000 measured calls per scenario:
|
|
| 100 |
| Full 64-token budget | 26.8 ms | 33.3 ms | 37.9 ms |
|
| 101 |
|
| 102 |
Cold Python process through first label: median 4.22 seconds, range 3.85–4.81
|
| 103 |
-
seconds
|
| 104 |
-
|
| 105 |
-
|
| 106 |
-
ONNX/int8 is unmeasured.
|
| 107 |
-
|
| 108 |
-
```shell
|
| 109 |
-
uv run python -m experiments.benchmark_cpu --threads 4 --splits /path/to/dataset/data/splits
|
| 110 |
-
uv run python -m experiments.benchmark_cpu_cold --threads 4
|
| 111 |
-
uv run python -m experiments.evaluate_english_challenge --dataset-dir /path/to/dataset
|
| 112 |
-
```
|
| 113 |
-
|
| 114 |
-
The CPU setup above intentionally uses the matching CPU wheel; use
|
| 115 |
-
`uv run --no-sync` with that environment to retain it. For CPU development,
|
| 116 |
-
install `pytest==9.2.0 ruff==0.17.5` with `uv pip install`. Standard `uv sync`
|
| 117 |
-
uses the lockfile and includes GPU training dependencies. Do CPU tests
|
| 118 |
-
locally; rent only a single datacenter RTX 5090 below $0.65/hour for training,
|
| 119 |
-
using the Vast.ai PyTorch image and tmux session gpubox.
|
|
|
|
| 7 |
datasets: [HashNuke/barge-in-detector-data]
|
| 8 |
tags: [barge-in, backchannel, multilingual, experimental]
|
| 9 |
---
|
| 10 |
+
# Barge-in detector: experimental model release
|
| 11 |
|
| 12 |
+
Classifies a cleaned caller utterance spoken while a voice agent is talking:
|
| 13 |
+
backchannel (continue) or barge_in (yield). Exactly two labels; no audio,
|
| 14 |
+
preceding agent context or language hint is required. Answers after the agent
|
| 15 |
+
finishes are handled by the runtime. Keep the first 64 tokenizer tokens.
|
|
|
|
| 16 |
|
| 17 |
+
This repo contains **model release artifacts only**. All training, inference,
|
| 18 |
+
evaluation and benchmark code, tests, reports and labnotes live in the
|
| 19 |
+
[GitHub source](https://github.com/vxpipe/barge-in-detector).
|
| 20 |
+
[Dataset releases](https://huggingface.co/datasets/HashNuke/barge-in-detector-data)
|
| 21 |
+
are separate. release.json pins the exact code and dataset revisions and records
|
| 22 |
+
artifact hashes and original training provenance.
|
| 23 |
|
| 24 |
+
**Experimental, not production-ready.** The model misses 25/30 short English
|
| 25 |
+
challenge interruptions, including “stop” and “wait”. Probabilities are not
|
| 26 |
+
calibrated; the runtime owns thresholds. The 0.5 cutoff is diagnostic.
|
| 27 |
|
| 28 |
+
## Use
|
|
|
|
|
|
|
|
|
|
|
|
|
| 29 |
|
| 30 |
+
Install compatible PyTorch and transformers, then:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 31 |
|
| 32 |
```python
|
| 33 |
import torch
|
|
|
|
| 37 |
tokenizer = AutoTokenizer.from_pretrained(repo)
|
| 38 |
model = AutoModelForSequenceClassification.from_pretrained(repo).cpu().eval()
|
| 39 |
with torch.inference_mode():
|
| 40 |
+
inputs = tokenizer(
|
| 41 |
+
"please stop for a moment", return_tensors="pt", truncation=True, max_length=64
|
| 42 |
+
)
|
| 43 |
+
p_barge_in = model(**inputs).logits.softmax(-1)[0, 1].item()
|
| 44 |
```
|
| 45 |
|
| 46 |
+
Input must already follow the source cleaner: lowercase, punctuation removed;
|
| 47 |
+
native scripts optionally mixed with English. Labels are 0=backchannel,
|
| 48 |
+
1=barge_in. Pin the model revision when deploying. Current files are the original
|
| 49 |
+
epoch-5 fp32 checkpoint; changing repository ownership does not retrain it.
|
| 50 |
|
| 51 |
+
## Training and evaluation
|
|
|
|
|
|
|
|
|
|
|
|
|
| 52 |
|
| 53 |
+
Base intfloat/multilingual-e5-small (MIT); see LICENSE and NOTICE. First run:
|
| 54 |
+
2,640 synthetic train / 333 validation / 331 test; five epochs, batch 32,
|
| 55 |
+
weighted cross-entropy, AdamW, 3e-5 learning rate, 10% warm-up, seed 20261115,
|
| 56 |
+
bf16 on one RTX 5090. Epoch 5 selected on validation macro F1 (0.99324).
|
| 57 |
+
Training scripts and complete reports are in the pinned GitHub source.
|
|
|
|
| 58 |
|
| 59 |
+
| Evaluation | Accuracy | Missed interruptions | False interruptions |
|
| 60 |
+
| --- | --- | --- | --- |
|
| 61 |
+
| Synthetic validation (333) | 99.4% | 0.45% | 0.90% |
|
| 62 |
+
| Synthetic test (331) | 100% | 0% | 0% |
|
| 63 |
+
| Reviewed English challenge (98) | 73.5% | 54.2% | 0% |
|
| 64 |
+
| Challenge unseen in any synthetic split (93) | 72.0% | 54.2% | 0% |
|
| 65 |
|
| 66 |
+
The challenge was authored before inference, owner-reviewed and reconciled with
|
| 67 |
+
authorization. Final binary labels match the frozen expectations. It is now a
|
| 68 |
+
regression/development set with intentional repeated prefixes. Criteria overlap
|
| 69 |
+
between synthetic splits. Exact cleaned-text leakage is absent; template/semantic
|
| 70 |
+
leakage remains unaudited. These results do not establish real-world performance.
|
| 71 |
+
All 14 listed languages have synthetic results; human-verified accuracy across
|
| 72 |
+
all languages has not been established.
|
| 73 |
+
|
| 74 |
+
## CPU timing
|
| 75 |
+
|
| 76 |
+
Four virtual AMD EPYC Genoa cores, PyTorch 2.15.0+cpu eager fp32, four threads,
|
| 77 |
batch one, 100 warm-ups and 1,000 measured calls per scenario:
|
| 78 |
|
| 79 |
| Warm text through label | p50 | p95 | p99 |
|
|
|
|
| 82 |
| Full 64-token budget | 26.8 ms | 33.3 ms | 37.9 ms |
|
| 83 |
|
| 84 |
Cold Python process through first label: median 4.22 seconds, range 3.85–4.81
|
| 85 |
+
seconds over ten runs. Includes imports and local loading; OS file cache may
|
| 86 |
+
be warm. Peak warm process RSS approximately 1,164 MiB. Network, HTTP, queueing
|
| 87 |
+
and Vxpipe boundary are excluded. Reference-CPU gates are unestablished;
|
| 88 |
+
ONNX/int8 is unmeasured. See the source labnotes and reports for recorded evidence.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
barge_in_detector_model/__init__.py
DELETED
|
@@ -1 +0,0 @@
|
|
| 1 |
-
"""Training, inference and evaluation for separately versioned barge-in data."""
|
|
|
|
|
|
barge_in_detector_model/data.py
DELETED
|
@@ -1,39 +0,0 @@
|
|
| 1 |
-
"""Read versioned dataset splits without owning their generation or assignment."""
|
| 2 |
-
|
| 3 |
-
import csv
|
| 4 |
-
from pathlib import Path
|
| 5 |
-
|
| 6 |
-
DEFAULT_DATASET = "HashNuke/barge-in-detector-data"
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
def read_csv(path):
|
| 10 |
-
with Path(path).open(encoding="utf-8", newline="") as f:
|
| 11 |
-
return list(csv.DictReader(f))
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
def resolve_splits(splits, *, repo_id=DEFAULT_DATASET, revision=None):
|
| 15 |
-
if splits is not None:
|
| 16 |
-
directory = Path(splits)
|
| 17 |
-
resolved_revision = None
|
| 18 |
-
else:
|
| 19 |
-
from huggingface_hub import HfApi, snapshot_download
|
| 20 |
-
|
| 21 |
-
resolved_revision = HfApi().dataset_info(repo_id, revision=revision).sha
|
| 22 |
-
root = Path(
|
| 23 |
-
snapshot_download(
|
| 24 |
-
repo_id=repo_id,
|
| 25 |
-
repo_type="dataset",
|
| 26 |
-
revision=resolved_revision,
|
| 27 |
-
allow_patterns=[
|
| 28 |
-
"data/splits/*.csv",
|
| 29 |
-
"data/evaluation/*.csv",
|
| 30 |
-
"data/evaluation/reviews/*.json",
|
| 31 |
-
"generation/factors/languages.yaml",
|
| 32 |
-
],
|
| 33 |
-
)
|
| 34 |
-
)
|
| 35 |
-
directory = root / "data/splits"
|
| 36 |
-
for split in ("train", "validation", "test"):
|
| 37 |
-
if not (directory / f"{split}.csv").is_file():
|
| 38 |
-
raise ValueError(f"Dataset split missing: {directory / split}.csv")
|
| 39 |
-
return directory, resolved_revision
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
barge_in_detector_model/evaluate.py
DELETED
|
@@ -1,111 +0,0 @@
|
|
| 1 |
-
"""Evaluate a trained model on one split (README.md).
|
| 2 |
-
|
| 3 |
-
uv run python -m barge_in_detector_model.evaluate --model models/run1 --split test
|
| 4 |
-
|
| 5 |
-
Reports accuracy, macro F1, false and missed interruption rates, and results
|
| 6 |
-
per language; writes them to <model>/eval-<split>.json and the mistakes to
|
| 7 |
-
<model>/errors-<split>.csv.
|
| 8 |
-
"""
|
| 9 |
-
|
| 10 |
-
import argparse
|
| 11 |
-
import csv
|
| 12 |
-
import json
|
| 13 |
-
from collections.abc import Sequence
|
| 14 |
-
from pathlib import Path
|
| 15 |
-
from typing import Any
|
| 16 |
-
|
| 17 |
-
import torch
|
| 18 |
-
from transformers import AutoModelForSequenceClassification, AutoTokenizer
|
| 19 |
-
|
| 20 |
-
from barge_in_detector_model.data import read_csv, resolve_splits
|
| 21 |
-
from barge_in_detector_model.metrics import LABELS, compute
|
| 22 |
-
|
| 23 |
-
BARGE_IN = LABELS.index("barge_in")
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
def device() -> torch.device:
|
| 27 |
-
return torch.device("cuda" if torch.cuda.is_available() else "cpu")
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
@torch.no_grad()
|
| 31 |
-
def predict(
|
| 32 |
-
model: Any,
|
| 33 |
-
tokenizer: Any,
|
| 34 |
-
rows: Sequence[dict[str, str]],
|
| 35 |
-
*,
|
| 36 |
-
batch_size: int = 128,
|
| 37 |
-
max_length: int = 64,
|
| 38 |
-
) -> list[float]:
|
| 39 |
-
"""P(barge_in) for each row's cleaned text."""
|
| 40 |
-
model.eval()
|
| 41 |
-
dev = next(model.parameters()).device
|
| 42 |
-
out: list[float] = []
|
| 43 |
-
for i in range(0, len(rows), batch_size):
|
| 44 |
-
texts = [r["text_clean"] for r in rows[i : i + batch_size]]
|
| 45 |
-
enc = tokenizer(
|
| 46 |
-
texts, truncation=True, max_length=max_length, padding=True, return_tensors="pt"
|
| 47 |
-
).to(dev)
|
| 48 |
-
logits = model(**enc).logits.float()
|
| 49 |
-
out += torch.softmax(logits, dim=-1)[:, BARGE_IN].tolist()
|
| 50 |
-
return out
|
| 51 |
-
|
| 52 |
-
|
| 53 |
-
def evaluate(
|
| 54 |
-
model: Any,
|
| 55 |
-
tokenizer: Any,
|
| 56 |
-
rows: Sequence[dict[str, str]],
|
| 57 |
-
*,
|
| 58 |
-
threshold: float = 0.5,
|
| 59 |
-
max_length: int = 64,
|
| 60 |
-
) -> tuple[dict[str, Any], list[float]]:
|
| 61 |
-
probs = predict(model, tokenizer, rows, max_length=max_length)
|
| 62 |
-
preds = ["barge_in" if p >= threshold else "backchannel" for p in probs]
|
| 63 |
-
metrics = compute([r["label"] for r in rows], preds, [r["language"] for r in rows])
|
| 64 |
-
return metrics, probs
|
| 65 |
-
|
| 66 |
-
|
| 67 |
-
def main(argv: Sequence[str] | None = None) -> int:
|
| 68 |
-
parser = argparse.ArgumentParser(description=__doc__)
|
| 69 |
-
parser.add_argument("--model", type=Path, required=True, help="a directory written by train")
|
| 70 |
-
parser.add_argument("--split", default="test", choices=["train", "validation", "test"])
|
| 71 |
-
parser.add_argument("--splits", type=Path, default=None)
|
| 72 |
-
parser.add_argument("--threshold", type=float, default=0.5, help="P(barge_in) cut-off")
|
| 73 |
-
parser.add_argument("--max-length", type=int, default=64, help="token budget")
|
| 74 |
-
parser.add_argument("--dataset-repo", default="HashNuke/barge-in-detector-data")
|
| 75 |
-
parser.add_argument(
|
| 76 |
-
"--dataset-revision", help="Exact dataset commit; resolves current main when omitted"
|
| 77 |
-
)
|
| 78 |
-
args = parser.parse_args(argv)
|
| 79 |
-
args.splits, args.dataset_revision = resolve_splits(
|
| 80 |
-
args.splits, repo_id=args.dataset_repo, revision=args.dataset_revision
|
| 81 |
-
)
|
| 82 |
-
|
| 83 |
-
tokenizer = AutoTokenizer.from_pretrained(args.model)
|
| 84 |
-
model = AutoModelForSequenceClassification.from_pretrained(args.model).to(device())
|
| 85 |
-
rows = read_csv(args.splits / f"{args.split}.csv")
|
| 86 |
-
metrics, probs = evaluate(
|
| 87 |
-
model, tokenizer, rows, threshold=args.threshold, max_length=args.max_length
|
| 88 |
-
)
|
| 89 |
-
metrics.update(split=args.split, threshold=args.threshold)
|
| 90 |
-
(args.model / f"eval-{args.split}.json").write_text(json.dumps(metrics, indent=2))
|
| 91 |
-
with (args.model / f"errors-{args.split}.csv").open("w", encoding="utf-8", newline="") as f:
|
| 92 |
-
writer = csv.writer(f)
|
| 93 |
-
writer.writerow(["label", "p_barge_in", "language", "caller_intent", "text"])
|
| 94 |
-
for r, p in zip(rows, probs, strict=True):
|
| 95 |
-
if (p >= args.threshold) != (r["label"] == "barge_in"):
|
| 96 |
-
writer.writerow(
|
| 97 |
-
[r["label"], f"{p:.3f}", r["language"], r["caller_intent"], r["text"]]
|
| 98 |
-
)
|
| 99 |
-
print(
|
| 100 |
-
f"{args.split}: n={metrics['count']} accuracy={metrics['accuracy']:.3f} "
|
| 101 |
-
f"macro_f1={metrics['macro_f1']:.3f} "
|
| 102 |
-
f"false_interruptions={metrics['false_interruption_rate']:.3f} "
|
| 103 |
-
f"missed_interruptions={metrics['missed_interruption_rate']:.3f}"
|
| 104 |
-
)
|
| 105 |
-
for lang, m in metrics["per_language"].items():
|
| 106 |
-
print(f" {lang:>4} n={m['count']:<4} accuracy={m['accuracy']:.3f}")
|
| 107 |
-
return 0
|
| 108 |
-
|
| 109 |
-
|
| 110 |
-
if __name__ == "__main__":
|
| 111 |
-
raise SystemExit(main())
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
barge_in_detector_model/infer.py
DELETED
|
@@ -1,27 +0,0 @@
|
|
| 1 |
-
"""Predict a binary label for an already-cleaned overlapping caller utterance."""
|
| 2 |
-
|
| 3 |
-
import argparse
|
| 4 |
-
import json
|
| 5 |
-
from pathlib import Path
|
| 6 |
-
|
| 7 |
-
import torch
|
| 8 |
-
from transformers import AutoModelForSequenceClassification, AutoTokenizer
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
def main():
|
| 12 |
-
parser = argparse.ArgumentParser(description=__doc__)
|
| 13 |
-
parser.add_argument("text")
|
| 14 |
-
parser.add_argument("--model", type=Path, default=Path("."))
|
| 15 |
-
parser.add_argument("--threads", type=int, default=4)
|
| 16 |
-
args = parser.parse_args()
|
| 17 |
-
torch.set_num_threads(args.threads)
|
| 18 |
-
tokenizer = AutoTokenizer.from_pretrained(args.model)
|
| 19 |
-
model = AutoModelForSequenceClassification.from_pretrained(args.model).cpu().eval()
|
| 20 |
-
with torch.inference_mode():
|
| 21 |
-
enc = tokenizer(args.text, truncation=True, max_length=64, return_tensors="pt")
|
| 22 |
-
p = torch.softmax(model(**enc).logits, dim=-1)[0, model.config.label2id["barge_in"]].item()
|
| 23 |
-
print(json.dumps({"p_barge_in": p, "label": "barge_in" if p >= 0.5 else "backchannel"}))
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
if __name__ == "__main__":
|
| 27 |
-
main()
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
barge_in_detector_model/metrics.py
DELETED
|
@@ -1,46 +0,0 @@
|
|
| 1 |
-
"""Metrics for the two labels. Pure Python, so they are tested without torch."""
|
| 2 |
-
|
| 3 |
-
from collections import defaultdict
|
| 4 |
-
from collections.abc import Sequence
|
| 5 |
-
from typing import Any
|
| 6 |
-
|
| 7 |
-
LABELS = ("backchannel", "barge_in")
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
def _prf(labels: Sequence[str], preds: Sequence[str], cls: str) -> dict[str, float]:
|
| 11 |
-
tp = sum(1 for y, p in zip(labels, preds, strict=True) if y == cls and p == cls)
|
| 12 |
-
predicted = sum(1 for p in preds if p == cls)
|
| 13 |
-
actual = sum(1 for y in labels if y == cls)
|
| 14 |
-
precision = tp / predicted if predicted else 0.0
|
| 15 |
-
recall = tp / actual if actual else 0.0
|
| 16 |
-
f1 = 2 * precision * recall / (precision + recall) if precision + recall else 0.0
|
| 17 |
-
return {"precision": precision, "recall": recall, "f1": f1, "count": actual}
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
def compute(
|
| 21 |
-
labels: Sequence[str], preds: Sequence[str], languages: Sequence[str]
|
| 22 |
-
) -> dict[str, Any]:
|
| 23 |
-
per_class = {cls: _prf(labels, preds, cls) for cls in LABELS}
|
| 24 |
-
backchannel = [p for y, p in zip(labels, preds, strict=True) if y == "backchannel"]
|
| 25 |
-
barge_in = [p for y, p in zip(labels, preds, strict=True) if y == "barge_in"]
|
| 26 |
-
by_language: dict[str, list[bool]] = defaultdict(list)
|
| 27 |
-
for y, p, lang in zip(labels, preds, languages, strict=True):
|
| 28 |
-
by_language[lang].append(y == p)
|
| 29 |
-
return {
|
| 30 |
-
"count": len(labels),
|
| 31 |
-
"accuracy": sum(y == p for y, p in zip(labels, preds, strict=True)) / len(labels),
|
| 32 |
-
"macro_f1": sum(c["f1"] for c in per_class.values()) / len(LABELS),
|
| 33 |
-
# A backchannel predicted as barge-in stops the agent for nothing.
|
| 34 |
-
"false_interruption_rate": (
|
| 35 |
-
sum(p == "barge_in" for p in backchannel) / len(backchannel) if backchannel else 0.0
|
| 36 |
-
),
|
| 37 |
-
# A barge-in predicted as backchannel lets the agent talk over the caller.
|
| 38 |
-
"missed_interruption_rate": (
|
| 39 |
-
sum(p == "backchannel" for p in barge_in) / len(barge_in) if barge_in else 0.0
|
| 40 |
-
),
|
| 41 |
-
"per_class": per_class,
|
| 42 |
-
"per_language": {
|
| 43 |
-
lang: {"accuracy": sum(ok) / len(ok), "count": len(ok)}
|
| 44 |
-
for lang, ok in sorted(by_language.items())
|
| 45 |
-
},
|
| 46 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
barge_in_detector_model/train.py
DELETED
|
@@ -1,140 +0,0 @@
|
|
| 1 |
-
"""Fine-tune a multilingual encoder on the utterance splits (README.md).
|
| 2 |
-
|
| 3 |
-
uv sync
|
| 4 |
-
uv run python -m barge_in_detector_model.train --out models/run1
|
| 5 |
-
|
| 6 |
-
Trains on train.csv's cleaned text, evaluates on validation.csv after every
|
| 7 |
-
epoch, and keeps the epoch with the best validation macro F1 in --out (model,
|
| 8 |
-
tokenizer, metrics.json). Backchannel is the smaller class, so the loss
|
| 9 |
-
weights each class by its inverse frequency. Uses the GPU when there is one.
|
| 10 |
-
"""
|
| 11 |
-
|
| 12 |
-
import argparse
|
| 13 |
-
import json
|
| 14 |
-
import random
|
| 15 |
-
import time
|
| 16 |
-
from collections import Counter
|
| 17 |
-
from collections.abc import Sequence
|
| 18 |
-
from pathlib import Path
|
| 19 |
-
|
| 20 |
-
import torch
|
| 21 |
-
from transformers import (
|
| 22 |
-
AutoModelForSequenceClassification,
|
| 23 |
-
AutoTokenizer,
|
| 24 |
-
get_linear_schedule_with_warmup,
|
| 25 |
-
)
|
| 26 |
-
|
| 27 |
-
from barge_in_detector_model.data import read_csv, resolve_splits
|
| 28 |
-
from barge_in_detector_model.evaluate import device, evaluate
|
| 29 |
-
from barge_in_detector_model.metrics import LABELS
|
| 30 |
-
|
| 31 |
-
# MIT licence, 118M parameters, about 100 languages (dataset repo docs/utterance-first-decision.md).
|
| 32 |
-
DEFAULT_MODEL = "intfloat/multilingual-e5-small"
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
def main(argv: Sequence[str] | None = None) -> int:
|
| 36 |
-
parser = argparse.ArgumentParser(description=__doc__)
|
| 37 |
-
parser.add_argument("--out", type=Path, required=True)
|
| 38 |
-
parser.add_argument("--model", default=DEFAULT_MODEL, help="Hugging Face model id or path")
|
| 39 |
-
parser.add_argument("--splits", type=Path, default=None)
|
| 40 |
-
parser.add_argument("--epochs", type=int, default=5)
|
| 41 |
-
parser.add_argument("--batch-size", type=int, default=32)
|
| 42 |
-
parser.add_argument("--lr", type=float, default=3e-5)
|
| 43 |
-
parser.add_argument("--warmup", type=float, default=0.1, help="share of steps")
|
| 44 |
-
parser.add_argument("--max-length", type=int, default=64, help="token budget")
|
| 45 |
-
parser.add_argument("--seed", type=int, default=20261115)
|
| 46 |
-
parser.add_argument("--dataset-repo", default="HashNuke/barge-in-detector-data")
|
| 47 |
-
parser.add_argument(
|
| 48 |
-
"--dataset-revision", help="Exact dataset commit; resolves current main when omitted"
|
| 49 |
-
)
|
| 50 |
-
args = parser.parse_args(argv)
|
| 51 |
-
args.splits, args.dataset_revision = resolve_splits(
|
| 52 |
-
args.splits, repo_id=args.dataset_repo, revision=args.dataset_revision
|
| 53 |
-
)
|
| 54 |
-
|
| 55 |
-
random.seed(args.seed)
|
| 56 |
-
torch.manual_seed(args.seed)
|
| 57 |
-
dev = device()
|
| 58 |
-
train_rows = read_csv(args.splits / "train.csv")
|
| 59 |
-
validation_rows = read_csv(args.splits / "validation.csv")
|
| 60 |
-
label_id = {label: i for i, label in enumerate(LABELS)}
|
| 61 |
-
|
| 62 |
-
tokenizer = AutoTokenizer.from_pretrained(args.model)
|
| 63 |
-
model = AutoModelForSequenceClassification.from_pretrained(
|
| 64 |
-
args.model,
|
| 65 |
-
num_labels=len(LABELS),
|
| 66 |
-
id2label=dict(enumerate(LABELS)),
|
| 67 |
-
label2id=label_id,
|
| 68 |
-
).to(dev)
|
| 69 |
-
|
| 70 |
-
counts = Counter(r["label"] for r in train_rows)
|
| 71 |
-
weights = torch.tensor(
|
| 72 |
-
[len(train_rows) / (len(LABELS) * counts[label]) for label in LABELS], device=dev
|
| 73 |
-
)
|
| 74 |
-
loss_fn = torch.nn.CrossEntropyLoss(weight=weights)
|
| 75 |
-
steps = args.epochs * -(-len(train_rows) // args.batch_size)
|
| 76 |
-
optimizer = torch.optim.AdamW(model.parameters(), lr=args.lr, weight_decay=0.01)
|
| 77 |
-
scheduler = get_linear_schedule_with_warmup(optimizer, int(args.warmup * steps), steps)
|
| 78 |
-
use_bf16 = dev.type == "cuda" and torch.cuda.is_bf16_supported()
|
| 79 |
-
|
| 80 |
-
print(
|
| 81 |
-
f"device={dev} bf16={use_bf16} model={args.model} train={len(train_rows)} "
|
| 82 |
-
f"validation={len(validation_rows)} class_weights={weights.tolist()}"
|
| 83 |
-
)
|
| 84 |
-
args.out.mkdir(parents=True, exist_ok=True)
|
| 85 |
-
history, best = [], None
|
| 86 |
-
for epoch in range(1, args.epochs + 1):
|
| 87 |
-
model.train()
|
| 88 |
-
order = list(range(len(train_rows)))
|
| 89 |
-
random.shuffle(order)
|
| 90 |
-
started, total = time.time(), 0.0
|
| 91 |
-
for i in range(0, len(order), args.batch_size):
|
| 92 |
-
batch = [train_rows[j] for j in order[i : i + args.batch_size]]
|
| 93 |
-
enc = tokenizer(
|
| 94 |
-
[r["text_clean"] for r in batch],
|
| 95 |
-
truncation=True,
|
| 96 |
-
max_length=args.max_length,
|
| 97 |
-
padding=True,
|
| 98 |
-
return_tensors="pt",
|
| 99 |
-
).to(dev)
|
| 100 |
-
labels = torch.tensor([label_id[r["label"]] for r in batch], device=dev)
|
| 101 |
-
with torch.autocast(device_type=dev.type, dtype=torch.bfloat16, enabled=use_bf16):
|
| 102 |
-
logits = model(**enc).logits
|
| 103 |
-
loss = loss_fn(logits.float(), labels)
|
| 104 |
-
loss.backward()
|
| 105 |
-
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
|
| 106 |
-
optimizer.step()
|
| 107 |
-
scheduler.step()
|
| 108 |
-
optimizer.zero_grad()
|
| 109 |
-
total += loss.item() * len(batch)
|
| 110 |
-
metrics, _ = evaluate(model, tokenizer, validation_rows, max_length=args.max_length)
|
| 111 |
-
record = {
|
| 112 |
-
"epoch": epoch,
|
| 113 |
-
"train_loss": total / len(train_rows),
|
| 114 |
-
"seconds": round(time.time() - started, 1),
|
| 115 |
-
"validation": metrics,
|
| 116 |
-
}
|
| 117 |
-
history.append(record)
|
| 118 |
-
print(
|
| 119 |
-
f"epoch {epoch}: loss={record['train_loss']:.4f} "
|
| 120 |
-
f"validation macro_f1={metrics['macro_f1']:.3f} accuracy={metrics['accuracy']:.3f} "
|
| 121 |
-
f"false_interruptions={metrics['false_interruption_rate']:.3f} "
|
| 122 |
-
f"missed_interruptions={metrics['missed_interruption_rate']:.3f}"
|
| 123 |
-
)
|
| 124 |
-
if best is None or metrics["macro_f1"] > best["validation"]["macro_f1"]:
|
| 125 |
-
best = record
|
| 126 |
-
model.save_pretrained(args.out)
|
| 127 |
-
tokenizer.save_pretrained(args.out)
|
| 128 |
-
(args.out / "metrics.json").write_text(
|
| 129 |
-
json.dumps(
|
| 130 |
-
{"args": {k: str(v) for k, v in vars(args).items()}, "best": best, "history": history},
|
| 131 |
-
indent=2,
|
| 132 |
-
)
|
| 133 |
-
)
|
| 134 |
-
print(f"best epoch {best['epoch']}: validation macro_f1={best['validation']['macro_f1']:.3f}")
|
| 135 |
-
print(f"saved to {args.out}")
|
| 136 |
-
return 0
|
| 137 |
-
|
| 138 |
-
|
| 139 |
-
if __name__ == "__main__":
|
| 140 |
-
raise SystemExit(main())
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
bin/create-labnotes
DELETED
|
@@ -1,59 +0,0 @@
|
|
| 1 |
-
#!/usr/bin/env bash
|
| 2 |
-
set -euo pipefail
|
| 3 |
-
|
| 4 |
-
usage() {
|
| 5 |
-
cat <<'USAGE'
|
| 6 |
-
Usage:
|
| 7 |
-
bin/create-labnotes 2-4-word-hyphenated-task-name
|
| 8 |
-
|
| 9 |
-
Creates an empty markdown labnotes file in labnotes/.
|
| 10 |
-
|
| 11 |
-
The input must be a 2-4 word hyphenated string, for example:
|
| 12 |
-
bin/create-labnotes test-crawl-resume
|
| 13 |
-
|
| 14 |
-
If spaces are provided, they will be converted to hyphens:
|
| 15 |
-
bin/create-labnotes "test crawl resume"
|
| 16 |
-
USAGE
|
| 17 |
-
}
|
| 18 |
-
|
| 19 |
-
if [[ $# -eq 0 ]]; then
|
| 20 |
-
usage
|
| 21 |
-
exit 0
|
| 22 |
-
fi
|
| 23 |
-
|
| 24 |
-
input="$*"
|
| 25 |
-
slug="$(printf '%s' "$input" \
|
| 26 |
-
| tr '[:upper:]' '[:lower:]' \
|
| 27 |
-
| sed -E 's/[[:space:]]+/-/g; s/[^a-z0-9-]+/-/g; s/-+/-/g; s/^-//; s/-$//')"
|
| 28 |
-
|
| 29 |
-
if [[ -z "$slug" ]]; then
|
| 30 |
-
echo "Error: task name must contain letters or numbers." >&2
|
| 31 |
-
echo >&2
|
| 32 |
-
usage >&2
|
| 33 |
-
exit 1
|
| 34 |
-
fi
|
| 35 |
-
|
| 36 |
-
word_count="$(awk -F- '{ print NF }' <<<"$slug")"
|
| 37 |
-
if (( word_count < 2 || word_count > 4 )); then
|
| 38 |
-
echo "Error: input must be a 2-4 word hyphenated string." >&2
|
| 39 |
-
echo >&2
|
| 40 |
-
usage >&2
|
| 41 |
-
exit 1
|
| 42 |
-
fi
|
| 43 |
-
|
| 44 |
-
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
| 45 |
-
labnotes_dir="$repo_root/labnotes"
|
| 46 |
-
mkdir -p "$labnotes_dir"
|
| 47 |
-
|
| 48 |
-
timestamp="$(date '+%Y%m%d-%H%M')"
|
| 49 |
-
filename="$timestamp-$slug.md"
|
| 50 |
-
path="$labnotes_dir/$filename"
|
| 51 |
-
|
| 52 |
-
if [[ -e "$path" ]]; then
|
| 53 |
-
echo "Error: labnotes/$filename already exists." >&2
|
| 54 |
-
exit 1
|
| 55 |
-
fi
|
| 56 |
-
|
| 57 |
-
touch "$path"
|
| 58 |
-
|
| 59 |
-
echo "Created labnotes/$filename"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
experiments/benchmark_cpu.py
DELETED
|
@@ -1,140 +0,0 @@
|
|
| 1 |
-
"""Measure warmed batch-one tokenization through label output on local CPU.
|
| 2 |
-
|
| 3 |
-
Experimental timing harness: run from the repo root with a CPU PyTorch environment.
|
| 4 |
-
It measures no runtime boundary, queue, HTTP call or cold model load.
|
| 5 |
-
"""
|
| 6 |
-
|
| 7 |
-
import argparse
|
| 8 |
-
import hashlib
|
| 9 |
-
import json
|
| 10 |
-
import platform
|
| 11 |
-
import random
|
| 12 |
-
import resource
|
| 13 |
-
import statistics
|
| 14 |
-
import time
|
| 15 |
-
from pathlib import Path
|
| 16 |
-
|
| 17 |
-
import torch
|
| 18 |
-
import transformers
|
| 19 |
-
from transformers import AutoModelForSequenceClassification, AutoTokenizer
|
| 20 |
-
|
| 21 |
-
from barge_in_detector_model.data import read_csv, resolve_splits
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
def percentile(values, quantile):
|
| 25 |
-
ordered = sorted(values)
|
| 26 |
-
index = (len(ordered) - 1) * quantile
|
| 27 |
-
low = int(index)
|
| 28 |
-
high = min(low + 1, len(ordered) - 1)
|
| 29 |
-
return ordered[low] + (ordered[high] - ordered[low]) * (index - low)
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
def summary(values):
|
| 33 |
-
return {
|
| 34 |
-
"p50_ms": percentile(values, 0.5),
|
| 35 |
-
"p95_ms": percentile(values, 0.95),
|
| 36 |
-
"p99_ms": percentile(values, 0.99),
|
| 37 |
-
"mean_ms": statistics.mean(values),
|
| 38 |
-
"max_ms": max(values),
|
| 39 |
-
}
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
@torch.inference_mode()
|
| 43 |
-
def main():
|
| 44 |
-
parser = argparse.ArgumentParser(description=__doc__)
|
| 45 |
-
parser.add_argument("--iterations", type=int, default=1000)
|
| 46 |
-
parser.add_argument("--warmup", type=int, default=100)
|
| 47 |
-
parser.add_argument("--model", type=Path, default=Path("."))
|
| 48 |
-
parser.add_argument("--threads", type=int, nargs="+", default=[1, 2, 4])
|
| 49 |
-
parser.add_argument("--splits", type=Path, help="Local dataset split directory")
|
| 50 |
-
parser.add_argument("--dataset-revision")
|
| 51 |
-
args = parser.parse_args()
|
| 52 |
-
split_dir, dataset_revision = resolve_splits(args.splits, revision=args.dataset_revision)
|
| 53 |
-
if args.iterations < 100 or args.warmup < 1 or min(args.threads) < 1:
|
| 54 |
-
parser.error("Use >=100 measured iterations, positive warmup and threads")
|
| 55 |
-
torch.set_num_interop_threads(1)
|
| 56 |
-
torch.set_num_threads(1)
|
| 57 |
-
started = time.perf_counter()
|
| 58 |
-
tokenizer = AutoTokenizer.from_pretrained(args.model, local_files_only=True)
|
| 59 |
-
model = (
|
| 60 |
-
AutoModelForSequenceClassification.from_pretrained(args.model, local_files_only=True)
|
| 61 |
-
.cpu()
|
| 62 |
-
.eval()
|
| 63 |
-
)
|
| 64 |
-
load_seconds = time.perf_counter() - started
|
| 65 |
-
rows = read_csv(split_dir / "test.csv")
|
| 66 |
-
texts = [r["text_clean"] for r in rows]
|
| 67 |
-
random.Random(20261115).shuffle(texts)
|
| 68 |
-
stress = (
|
| 69 |
-
"please stop for a moment because i need to correct the information you are reading " * 12
|
| 70 |
-
)
|
| 71 |
-
assert len(tokenizer(stress, truncation=True, max_length=64)["input_ids"]) == 64
|
| 72 |
-
report = {
|
| 73 |
-
"dataset_revision": dataset_revision,
|
| 74 |
-
"model": str(args.model),
|
| 75 |
-
"runtime": "PyTorch eager fp32, CPU only",
|
| 76 |
-
"model_sha256": hashlib.sha256((args.model / "model.safetensors").read_bytes()).hexdigest(),
|
| 77 |
-
"python": platform.python_version(),
|
| 78 |
-
"torch": torch.__version__,
|
| 79 |
-
"transformers": transformers.__version__,
|
| 80 |
-
"platform": platform.platform(),
|
| 81 |
-
"cpuinfo": next(
|
| 82 |
-
line.split(":", 1)[1].strip()
|
| 83 |
-
for line in Path("/proc/cpuinfo").read_text().splitlines()
|
| 84 |
-
if line.startswith("model name")
|
| 85 |
-
),
|
| 86 |
-
"load_seconds_excluded": load_seconds,
|
| 87 |
-
"iterations": args.iterations,
|
| 88 |
-
"warmup_per_scenario": args.warmup,
|
| 89 |
-
"batch_size": 1,
|
| 90 |
-
"max_length": 64,
|
| 91 |
-
"interop_threads": 1,
|
| 92 |
-
"parameter_bytes": sum(p.numel() * p.element_size() for p in model.parameters()),
|
| 93 |
-
"corpus_sha256": hashlib.sha256((split_dir / "test.csv").read_bytes()).hexdigest(),
|
| 94 |
-
"corpus_rows": len(rows),
|
| 95 |
-
"languages": sorted({r["language"] for r in rows}),
|
| 96 |
-
"measurements": [],
|
| 97 |
-
"scope": "Local virtual CPU; reference CPU unspecified. Excludes Vxpipe boundary and queueing; no whole-path acceptance claim.",
|
| 98 |
-
}
|
| 99 |
-
lengths = [len(tokenizer(t, truncation=True, max_length=64)["input_ids"]) for t in texts]
|
| 100 |
-
report["corpus_token_lengths"] = {
|
| 101 |
-
"p50": percentile(lengths, 0.5),
|
| 102 |
-
"p95": percentile(lengths, 0.95),
|
| 103 |
-
"max": max(lengths),
|
| 104 |
-
}
|
| 105 |
-
for threads in args.threads:
|
| 106 |
-
torch.set_num_threads(threads)
|
| 107 |
-
for name, inputs in [("multilingual_test", texts), ("full_64_tokens", [stress])]:
|
| 108 |
-
samples, token_times, forward_times = [], [], []
|
| 109 |
-
for index in range(args.warmup + args.iterations):
|
| 110 |
-
text = inputs[index % len(inputs)]
|
| 111 |
-
start = time.perf_counter_ns()
|
| 112 |
-
enc = tokenizer(text, truncation=True, max_length=64, return_tensors="pt")
|
| 113 |
-
tokenized = time.perf_counter_ns()
|
| 114 |
-
logits = model(**enc).logits
|
| 115 |
-
probability = torch.softmax(logits, dim=-1)[
|
| 116 |
-
0, model.config.label2id["barge_in"]
|
| 117 |
-
].item()
|
| 118 |
-
label = "barge_in" if probability >= 0.5 else "backchannel"
|
| 119 |
-
finish = time.perf_counter_ns()
|
| 120 |
-
assert 0 <= probability <= 1
|
| 121 |
-
assert label in model.config.label2id
|
| 122 |
-
if index >= args.warmup:
|
| 123 |
-
samples.append((finish - start) / 1e6)
|
| 124 |
-
token_times.append((tokenized - start) / 1e6)
|
| 125 |
-
forward_times.append((finish - tokenized) / 1e6)
|
| 126 |
-
result = {
|
| 127 |
-
"threads": threads,
|
| 128 |
-
"scenario": name,
|
| 129 |
-
"model_path": summary(samples),
|
| 130 |
-
"tokenization": summary(token_times),
|
| 131 |
-
"forward_and_probability": summary(forward_times),
|
| 132 |
-
}
|
| 133 |
-
report["measurements"].append(result)
|
| 134 |
-
print(json.dumps(result), flush=True)
|
| 135 |
-
report["peak_process_rss_mib"] = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024
|
| 136 |
-
(args.model / "cpu-benchmark.json").write_text(json.dumps(report, indent=2))
|
| 137 |
-
|
| 138 |
-
|
| 139 |
-
if __name__ == "__main__":
|
| 140 |
-
main()
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
experiments/benchmark_cpu_cold.py
DELETED
|
@@ -1,80 +0,0 @@
|
|
| 1 |
-
"""Time a new Python process through its first label from local model files."""
|
| 2 |
-
|
| 3 |
-
import argparse
|
| 4 |
-
import json
|
| 5 |
-
import os
|
| 6 |
-
import statistics
|
| 7 |
-
import subprocess
|
| 8 |
-
import sys
|
| 9 |
-
import time
|
| 10 |
-
from pathlib import Path
|
| 11 |
-
|
| 12 |
-
CHILD = """
|
| 13 |
-
import json
|
| 14 |
-
import time
|
| 15 |
-
import torch
|
| 16 |
-
from transformers import AutoModelForSequenceClassification, AutoTokenizer
|
| 17 |
-
import sys
|
| 18 |
-
model_path, threads = sys.argv[1], int(sys.argv[2])
|
| 19 |
-
torch.set_num_threads(threads)
|
| 20 |
-
torch.set_num_interop_threads(1)
|
| 21 |
-
tokenizer = AutoTokenizer.from_pretrained(model_path, local_files_only=True)
|
| 22 |
-
model = AutoModelForSequenceClassification.from_pretrained(model_path, local_files_only=True).cpu().eval()
|
| 23 |
-
with torch.inference_mode():
|
| 24 |
-
loaded_start = time.perf_counter_ns()
|
| 25 |
-
enc = tokenizer('please stop for a moment', truncation=True, max_length=64, return_tensors='pt')
|
| 26 |
-
p = torch.softmax(model(**enc).logits, dim=-1)[0, model.config.label2id['barge_in']].item()
|
| 27 |
-
label = 'barge_in' if p >= 0.5 else 'backchannel'
|
| 28 |
-
loaded_ms = (time.perf_counter_ns() - loaded_start) / 1e6
|
| 29 |
-
print(json.dumps({'label': label, 'p_barge_in': p, 'loaded_to_first_label_ms': loaded_ms}), flush=True)
|
| 30 |
-
"""
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
def main():
|
| 34 |
-
parser = argparse.ArgumentParser(description=__doc__)
|
| 35 |
-
parser.add_argument("--model", type=Path, default=Path("."))
|
| 36 |
-
parser.add_argument("--runs", type=int, default=10)
|
| 37 |
-
parser.add_argument("--threads", type=int, default=1)
|
| 38 |
-
args = parser.parse_args()
|
| 39 |
-
samples = []
|
| 40 |
-
loaded_first = []
|
| 41 |
-
env = {
|
| 42 |
-
**os.environ,
|
| 43 |
-
"OMP_NUM_THREADS": str(args.threads),
|
| 44 |
-
"MKL_NUM_THREADS": str(args.threads),
|
| 45 |
-
"TOKENIZERS_PARALLELISM": "false",
|
| 46 |
-
}
|
| 47 |
-
for run in range(args.runs):
|
| 48 |
-
start = time.perf_counter_ns()
|
| 49 |
-
with subprocess.Popen(
|
| 50 |
-
[sys.executable, "-c", CHILD, str(args.model), str(args.threads)],
|
| 51 |
-
stdout=subprocess.PIPE,
|
| 52 |
-
stderr=subprocess.DEVNULL,
|
| 53 |
-
text=True,
|
| 54 |
-
env=env,
|
| 55 |
-
) as process:
|
| 56 |
-
line = process.stdout.readline()
|
| 57 |
-
elapsed = (time.perf_counter_ns() - start) / 1e9
|
| 58 |
-
result = json.loads(line)
|
| 59 |
-
if process.wait() != 0:
|
| 60 |
-
raise RuntimeError("Cold-start child failed")
|
| 61 |
-
samples.append(elapsed)
|
| 62 |
-
loaded_first.append(result["loaded_to_first_label_ms"])
|
| 63 |
-
print(f"cold start {run + 1}: {elapsed:.3f}s {result}", flush=True)
|
| 64 |
-
report = {
|
| 65 |
-
"cold_process_seconds": samples,
|
| 66 |
-
"loaded_to_first_label_ms": loaded_first,
|
| 67 |
-
"loaded_to_first_label_median_ms": statistics.median(loaded_first),
|
| 68 |
-
"median_seconds": statistics.median(samples),
|
| 69 |
-
"min_seconds": min(samples),
|
| 70 |
-
"max_seconds": max(samples),
|
| 71 |
-
"runs": args.runs,
|
| 72 |
-
"threads": args.threads,
|
| 73 |
-
"scope": "Python process creation through reading first flushed label. Includes imports, tokenizer/model load, first inference. Local files only; OS file cache may be warm. Excludes process shutdown, downloads and HTTP/network boundary.",
|
| 74 |
-
}
|
| 75 |
-
(args.model / "cpu-cold-benchmark.json").write_text(json.dumps(report, indent=2))
|
| 76 |
-
print(json.dumps(report), flush=True)
|
| 77 |
-
|
| 78 |
-
|
| 79 |
-
if __name__ == "__main__":
|
| 80 |
-
main()
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
experiments/evaluate_english_challenge.py
DELETED
|
@@ -1,132 +0,0 @@
|
|
| 1 |
-
"""Frozen-checkpoint diagnostic experiment; labels are authored before inference.
|
| 2 |
-
|
| 3 |
-
Run from the repo root with uv run --group train python -m experiments.evaluate_english_challenge.
|
| 4 |
-
Evidence is saved under the model directory, outside the training splits.
|
| 5 |
-
"""
|
| 6 |
-
|
| 7 |
-
import argparse
|
| 8 |
-
import csv
|
| 9 |
-
import hashlib
|
| 10 |
-
import json
|
| 11 |
-
from collections import defaultdict
|
| 12 |
-
from pathlib import Path
|
| 13 |
-
|
| 14 |
-
import torch
|
| 15 |
-
import transformers
|
| 16 |
-
from transformers import AutoModelForSequenceClassification, AutoTokenizer
|
| 17 |
-
|
| 18 |
-
from barge_in_detector_model.data import read_csv, resolve_splits
|
| 19 |
-
from barge_in_detector_model.evaluate import device, predict
|
| 20 |
-
from barge_in_detector_model.metrics import compute
|
| 21 |
-
|
| 22 |
-
MODEL = Path(".")
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
def score(rows, probs):
|
| 26 |
-
return compute(
|
| 27 |
-
[r["label"] for r in rows],
|
| 28 |
-
["barge_in" if p >= 0.5 else "backchannel" for p in probs],
|
| 29 |
-
[r["language"] for r in rows],
|
| 30 |
-
)
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
def main():
|
| 34 |
-
parser = argparse.ArgumentParser(description=__doc__)
|
| 35 |
-
parser.add_argument("--dataset-dir", type=Path, help="Local dataset repository root")
|
| 36 |
-
parser.add_argument("--dataset-revision")
|
| 37 |
-
args = parser.parse_args()
|
| 38 |
-
if args.dataset_dir is None:
|
| 39 |
-
split_dir, revision = resolve_splits(None, revision=args.dataset_revision)
|
| 40 |
-
dataset_dir = split_dir.parents[1]
|
| 41 |
-
else:
|
| 42 |
-
dataset_dir = args.dataset_dir
|
| 43 |
-
revision = None
|
| 44 |
-
challenge = dataset_dir / "data/evaluation/english-challenge-v1.csv"
|
| 45 |
-
rows = read_csv(challenge)
|
| 46 |
-
splits = {
|
| 47 |
-
s: read_csv(dataset_dir / f"data/splits/{s}.csv") for s in ("train", "validation", "test")
|
| 48 |
-
}
|
| 49 |
-
texts = {s: {r["text_clean"] for r in rs} for s, rs in splits.items()}
|
| 50 |
-
all_seen = set.union(*texts.values())
|
| 51 |
-
tokenizer = AutoTokenizer.from_pretrained(MODEL)
|
| 52 |
-
model = AutoModelForSequenceClassification.from_pretrained(MODEL).to(device())
|
| 53 |
-
probs = predict(model, tokenizer, rows, max_length=64)
|
| 54 |
-
single_probs = predict(model, tokenizer, rows, batch_size=1, max_length=64)
|
| 55 |
-
report = {
|
| 56 |
-
"dataset_revision": revision,
|
| 57 |
-
"challenge_sha256": hashlib.sha256(challenge.read_bytes()).hexdigest(),
|
| 58 |
-
"model_sha256": hashlib.sha256((MODEL / "model.safetensors").read_bytes()).hexdigest(),
|
| 59 |
-
"threshold": 0.5,
|
| 60 |
-
"runtime": {
|
| 61 |
-
"device": str(next(model.parameters()).device),
|
| 62 |
-
"torch": torch.__version__,
|
| 63 |
-
"transformers": transformers.__version__,
|
| 64 |
-
"threads": torch.get_num_threads(),
|
| 65 |
-
},
|
| 66 |
-
"batch_vs_single_max_probability_difference": max(
|
| 67 |
-
abs(a - b) for a, b in zip(probs, single_probs, strict=True)
|
| 68 |
-
),
|
| 69 |
-
"batch_vs_single_label_disagreements": sum(
|
| 70 |
-
(a >= 0.5) != (b >= 0.5) for a, b in zip(probs, single_probs, strict=True)
|
| 71 |
-
),
|
| 72 |
-
"review_status": "Owner-reviewed v1 labels match authored expectations after authorized reconciliation. Diagnostic development set; not independent representative accuracy evidence.",
|
| 73 |
-
"all": score(rows, probs),
|
| 74 |
-
"by_category": {},
|
| 75 |
-
"overlap_counts": {s: sum(r["text_clean"] in ts for r in rows) for s, ts in texts.items()},
|
| 76 |
-
"cross_split_exact_overlap": {
|
| 77 |
-
f"{a}/{b}": len(texts[a] & texts[b])
|
| 78 |
-
for a, b in (("train", "validation"), ("train", "test"), ("validation", "test"))
|
| 79 |
-
},
|
| 80 |
-
}
|
| 81 |
-
for category in sorted({r["category"] for r in rows}):
|
| 82 |
-
pairs = [(r, p) for r, p in zip(rows, probs, strict=True) if r["category"] == category]
|
| 83 |
-
report["by_category"][category] = score(*zip(*pairs, strict=True))
|
| 84 |
-
unseen = [(r, p) for r, p in zip(rows, probs, strict=True) if r["text_clean"] not in all_seen]
|
| 85 |
-
report["unseen"] = score(*zip(*unseen, strict=True))
|
| 86 |
-
groups = defaultdict(list)
|
| 87 |
-
for r, p in zip(rows, probs, strict=True):
|
| 88 |
-
if r["group"]:
|
| 89 |
-
groups[r["group"]].append(
|
| 90 |
-
{"text": r["text_clean"], "label": r["label"], "p_barge_in": p}
|
| 91 |
-
)
|
| 92 |
-
report["prefix_groups"] = dict(groups)
|
| 93 |
-
report["prefix_groups_all_labels_correct"] = sum(
|
| 94 |
-
all((r["p_barge_in"] >= 0.5) == (r["label"] == "barge_in") for r in group)
|
| 95 |
-
for group in groups.values()
|
| 96 |
-
)
|
| 97 |
-
# A diagnostic baseline: select its word-count cutoff using validation only.
|
| 98 |
-
english = {s: [r for r in rs if r["language"] == "en"] for s, rs in splits.items()}
|
| 99 |
-
|
| 100 |
-
def length_probs(rs, cutoff):
|
| 101 |
-
return [float(len(r["text_clean"].split()) >= cutoff) for r in rs]
|
| 102 |
-
|
| 103 |
-
cutoff = max(
|
| 104 |
-
range(1, 25),
|
| 105 |
-
key=lambda n: score(english["validation"], length_probs(english["validation"], n))[
|
| 106 |
-
"macro_f1"
|
| 107 |
-
],
|
| 108 |
-
)
|
| 109 |
-
report["english_length_baseline"] = {
|
| 110 |
-
"cutoff_words": cutoff,
|
| 111 |
-
"validation": score(english["validation"], length_probs(english["validation"], cutoff)),
|
| 112 |
-
"test": score(english["test"], length_probs(english["test"], cutoff)),
|
| 113 |
-
"challenge": score(rows, length_probs(rows, cutoff)),
|
| 114 |
-
}
|
| 115 |
-
with (MODEL / "challenge-predictions.csv").open("w", newline="") as f:
|
| 116 |
-
writer = csv.DictWriter(f, fieldnames=[*rows[0], "p_barge_in", "prediction", "seen_in"])
|
| 117 |
-
writer.writeheader()
|
| 118 |
-
for r, p in zip(rows, probs, strict=True):
|
| 119 |
-
writer.writerow(
|
| 120 |
-
{
|
| 121 |
-
**r,
|
| 122 |
-
"p_barge_in": p,
|
| 123 |
-
"prediction": "barge_in" if p >= 0.5 else "backchannel",
|
| 124 |
-
"seen_in": ";".join(s for s, ts in texts.items() if r["text_clean"] in ts),
|
| 125 |
-
}
|
| 126 |
-
)
|
| 127 |
-
(MODEL / "challenge-report.json").write_text(json.dumps(report, indent=2))
|
| 128 |
-
print(json.dumps({k: v for k, v in report.items() if k != "prefix_groups"}, indent=2))
|
| 129 |
-
|
| 130 |
-
|
| 131 |
-
if __name__ == "__main__":
|
| 132 |
-
main()
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
labnotes/20261115-1332-repository-split.md
DELETED
|
@@ -1,41 +0,0 @@
|
|
| 1 |
-
# Model repository split
|
| 2 |
-
|
| 3 |
-
Owner moved model artifacts, training, inference, evaluation and benchmarks into
|
| 4 |
-
this standalone Hugging Face model repo. Dataset generation, splitting, review
|
| 5 |
-
and all earlier labnotes remain in HashNuke/barge-in-detector-data. This repo has
|
| 6 |
-
its own scoped AGENTS.md, bin/create-labnotes and milestone index.
|
| 7 |
-
|
| 8 |
-
- New package barge_in_detector_model isolates training/metrics/data-loading
|
| 9 |
-
code from the dataset generator. Model code downloads a resolved dataset
|
| 10 |
-
commit, or uses explicit local splits; it does not assign splits.
|
| 11 |
-
- Four unit tests cover metrics, local CSVs, missing split detection and pinned
|
| 12 |
-
Hub downloads. Tiny CPU smoke trained one epoch, saved, reloaded and evaluated.
|
| 13 |
-
- Actual unchanged checkpoint predicts stop as backchannel, p=0.0134437,
|
| 14 |
-
reproducing the known failure. No GPU rental or retraining of release weights.
|
| 15 |
-
- Token budget remains 64; two labels; predictions are not calibrated.
|
| 16 |
-
- Root weights and tokenizer match the original artifact hashes. Original source
|
| 17 |
-
manifest and sanitized training metrics retain run provenance. Model card
|
| 18 |
-
reports synthetic and reviewed diagnostics and local CPU timing limitations.
|
| 19 |
-
- AGENTS.md derives CPU serving gates, fourteen languages, single RTX 5090,
|
| 20 |
-
under-$0.65/hour datacenter/PyTorch policy, gpubox, tests, and labnotes/milestone
|
| 21 |
-
usage from the dataset instructions. Existing history is linked, not duplicated.
|
| 22 |
-
|
| 23 |
-
Historical run and latency evidence is in the dataset repo labnotes. Follow
|
| 24 |
-
labnotes/milestones/index.md for new model work. Publication verification follows.
|
| 25 |
-
|
| 26 |
-
## Publication verified
|
| 27 |
-
|
| 28 |
-
- Initial weights/tokenizer commit dcbadfd7572e08ef3ccd19e35576e85faffe00f8;
|
| 29 |
-
standalone code commit 9aa77e5ff33f479ec5559f3f65bcfc092e5f86ab.
|
| 30 |
-
- release.json pins dataset dda7115ace1fe7b07c49139860131f39910521ad.
|
| 31 |
-
- Downloaded that dataset through the model loader: 2,640/333/331 rows and
|
| 32 |
-
exact file hashes match. Standard datasets.load_dataset also loads those
|
| 33 |
-
splits and the separate 98-example challenge configuration correctly.
|
| 34 |
-
- Anonymous Hub model download loaded on local CPU and reproduced stop
|
| 35 |
-
probability 0.0134437. Remote large-file checksums match local model/tokenizer.
|
| 36 |
-
- Four model unit tests, Ruff format and lint passed. Tiny local train/save/
|
| 37 |
-
reload/evaluate smoke passed before original scripts were removed.
|
| 38 |
-
- bin/create-labnotes is executable in the Git tree. Each repo has its own
|
| 39 |
-
AGENTS.md and milestone index. New model work stays here; history stays in data.
|
| 40 |
-
- No model or tokenizer weights were committed to the original GitHub repo.
|
| 41 |
-
No GPU was rented or restarted during this publication work.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
labnotes/issues/.gitkeep
DELETED
|
File without changes
|
labnotes/milestones/experimental-checkpoint-release.md
DELETED
|
@@ -1,30 +0,0 @@
|
|
| 1 |
-
# Experimental checkpoint release
|
| 2 |
-
|
| 3 |
-
## Runnable outcome
|
| 4 |
-
|
| 5 |
-
A self-contained Hugging Face model repo that loads the first checkpoint on CPU,
|
| 6 |
-
trains/evaluates with a separate versioned dataset, and records honest limitations.
|
| 7 |
-
Prerequisites: dataset HashNuke/barge-in-detector-data, original generation/training
|
| 8 |
-
history in that dataset repo, root README and AGENTS.md.
|
| 9 |
-
|
| 10 |
-
## Contracts and checkpoints
|
| 11 |
-
|
| 12 |
-
- [x] Copy and verify selected epoch-5 weights and tokenizer without retraining.
|
| 13 |
-
- [x] Move training/evaluation/metrics/benchmarks into independent model modules.
|
| 14 |
-
- [x] Resolve Hub dataset revisions to immutable commits; support local splits.
|
| 15 |
-
- [x] Copy labnotes helper; derive model AGENTS.md and establish milestone usage.
|
| 16 |
-
- [x] Tiny CPU train/save/reload/evaluate smoke and actual checkpoint inference.
|
| 17 |
-
- [x] Publish artifacts, instructions and code to the model repo.
|
| 18 |
-
- [x] Record dataset commit, source snapshot and artifact checksums in release.json.
|
| 19 |
-
|
| 20 |
-
## Acceptance gates
|
| 21 |
-
|
| 22 |
-
- [x] Four model unit tests, lint and formatting pass.
|
| 23 |
-
- [x] Model weights and tokenizer resolve as large-file artifacts on the Hub.
|
| 24 |
-
- [x] Published files match local release hashes; CPU inference works from files.
|
| 25 |
-
- [x] Model card separates synthetic and reviewed diagnostics and records failures.
|
| 26 |
-
|
| 27 |
-
This milestone does not claim production readiness, calibration or reference-CPU
|
| 28 |
-
latency acceptance. See repository-split labnotes for evidence.
|
| 29 |
-
|
| 30 |
-
Checkpoint evidence: [repository split labnotes](../20261115-1332-repository-split.md).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
labnotes/milestones/index.md
DELETED
|
@@ -1,7 +0,0 @@
|
|
| 1 |
-
# Model milestones
|
| 2 |
-
|
| 3 |
-
The checklist owns implementation order. Read AGENTS.md and linked sources;
|
| 4 |
-
leave entries unchecked until every acceptance gate passes.
|
| 5 |
-
|
| 6 |
-
- [x] [Publish the experimental checkpoint](experimental-checkpoint-release.md)
|
| 7 |
-
- [ ] [Improve short-interruption recall](short-interruption-recall.md)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
labnotes/milestones/short-interruption-recall.md
DELETED
|
@@ -1,28 +0,0 @@
|
|
| 1 |
-
# Improve short-interruption recall
|
| 2 |
-
|
| 3 |
-
## Runnable outcome and prerequisites
|
| 4 |
-
|
| 5 |
-
Improve the current checkpoint's 5/30 short-interruption recall on reviewed
|
| 6 |
-
English development examples, then establish performance on fresh owner-reviewed
|
| 7 |
-
English/Hindi evaluation data. Prerequisite: experimental checkpoint release;
|
| 8 |
-
coordinate data changes with the dataset repo coverage-audit milestone.
|
| 9 |
-
|
| 10 |
-
## Contract and checkpoint tasks
|
| 11 |
-
|
| 12 |
-
- [ ] Agree acceptance floors for false and missed interruptions before tuning.
|
| 13 |
-
- [ ] Compare a character n-gram baseline and encoder on exactly the same splits.
|
| 14 |
-
- [ ] Train against the revised dataset distribution; never train on challenge rows.
|
| 15 |
-
- [ ] Evaluate short interruptions, longer backchannels, prefix transitions and
|
| 16 |
-
per-language errors. Preserve old checkpoint and use validation for selection.
|
| 17 |
-
- [ ] Record immutable dataset/source revisions, parameters and costs.
|
| 18 |
-
- [ ] Calibrate on realistic validation priors.
|
| 19 |
-
- [ ] Measure ONNX/int8 parity and reference-CPU latency/memory; measure Vxpipe
|
| 20 |
-
serialization/queue/boundary overhead for the whole-path budget.
|
| 21 |
-
|
| 22 |
-
## Acceptance gates
|
| 23 |
-
|
| 24 |
-
- [ ] Fresh human-reviewed English/Hindi results meet agreed floors and beat baseline.
|
| 25 |
-
- [ ] Model-path p99 <=20 ms; whole path p99 <=50 ms on reference CPU, warmed batch one.
|
| 26 |
-
- [ ] Prefix behaviour and Vxpipe overlapping-speech scenarios pass with evidence.
|
| 27 |
-
|
| 28 |
-
Use `bin/create-labnotes` and link each checkpoint's factual evidence here.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
pyproject.toml
DELETED
|
@@ -1,25 +0,0 @@
|
|
| 1 |
-
[project]
|
| 2 |
-
name = "barge-in-detector-model"
|
| 3 |
-
version = "0.1.0"
|
| 4 |
-
description = "Training, evaluation and CPU inference for the Vxpipe barge-in classifier"
|
| 5 |
-
requires-python = ">=3.14"
|
| 6 |
-
dependencies = ["torch>=2.15.0", "transformers>=5.23.0", "huggingface-hub>=2.4.0"]
|
| 7 |
-
|
| 8 |
-
[dependency-groups]
|
| 9 |
-
dev = ["pytest>=9.2.0", "ruff>=0.17.5"]
|
| 10 |
-
|
| 11 |
-
[tool.uv]
|
| 12 |
-
package = false
|
| 13 |
-
|
| 14 |
-
[tool.pytest.ini_options]
|
| 15 |
-
pythonpath = ["."]
|
| 16 |
-
testpaths = ["tests"]
|
| 17 |
-
addopts = "-m 'not integration'"
|
| 18 |
-
markers = ["integration: calls networks; excluded by default"]
|
| 19 |
-
|
| 20 |
-
[tool.ruff]
|
| 21 |
-
line-length = 100
|
| 22 |
-
target-version = "py314"
|
| 23 |
-
[tool.ruff.lint]
|
| 24 |
-
select = ["E", "F", "I", "B", "UP", "SIM"]
|
| 25 |
-
ignore = ["E501"]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
release.json
CHANGED
|
@@ -3,7 +3,7 @@
|
|
| 3 |
"status": "experimental, uncalibrated, short-interruption failures",
|
| 4 |
"base_model": "intfloat/multilingual-e5-small",
|
| 5 |
"dataset_repo": "HashNuke/barge-in-detector-data",
|
| 6 |
-
"dataset_revision": "
|
| 7 |
"dataset_files_sha256": {
|
| 8 |
"data/export/utterances.csv": "556a2820e4a24ac13bab44321fb32b464c5fb225750d41a5a10ebcde7cdbab8a",
|
| 9 |
"data/splits/train.csv": "aab224bd0526eee915295e6f5c52076e29bb751cd0d664eefb4321843ad22533",
|
|
@@ -20,9 +20,8 @@
|
|
| 20 |
},
|
| 21 |
"original_training_source": {
|
| 22 |
"git_base_commit": "5ce0ed1283401f8c73bafcf5a7dbd2101c4bbee6",
|
| 23 |
-
"overlay_manifest": "reports/source-manifest.json"
|
| 24 |
},
|
| 25 |
-
"migration_source_git_commit": "1f92917a59908e320c3a3489c3e5de413a3e4672",
|
| 26 |
"labels": {
|
| 27 |
"0": "backchannel",
|
| 28 |
"1": "barge_in"
|
|
@@ -30,5 +29,9 @@
|
|
| 30 |
"max_length": 64,
|
| 31 |
"default_diagnostic_threshold": 0.5,
|
| 32 |
"selected_epoch": 5,
|
| 33 |
-
"training_metrics": "reports/training-metrics.json"
|
|
|
|
|
|
|
|
|
|
|
|
|
| 34 |
}
|
|
|
|
| 3 |
"status": "experimental, uncalibrated, short-interruption failures",
|
| 4 |
"base_model": "intfloat/multilingual-e5-small",
|
| 5 |
"dataset_repo": "HashNuke/barge-in-detector-data",
|
| 6 |
+
"dataset_revision": "9e914ce8b50aa12ebd3fe212bb183e09a1dee370",
|
| 7 |
"dataset_files_sha256": {
|
| 8 |
"data/export/utterances.csv": "556a2820e4a24ac13bab44321fb32b464c5fb225750d41a5a10ebcde7cdbab8a",
|
| 9 |
"data/splits/train.csv": "aab224bd0526eee915295e6f5c52076e29bb751cd0d664eefb4321843ad22533",
|
|
|
|
| 20 |
},
|
| 21 |
"original_training_source": {
|
| 22 |
"git_base_commit": "5ce0ed1283401f8c73bafcf5a7dbd2101c4bbee6",
|
| 23 |
+
"overlay_manifest": "https://github.com/vxpipe/barge-in-detector/blob/836de6924c3da68cd871e6e08da34291e9778432/reports/e5-small-run1/source-manifest.json"
|
| 24 |
},
|
|
|
|
| 25 |
"labels": {
|
| 26 |
"0": "backchannel",
|
| 27 |
"1": "barge_in"
|
|
|
|
| 29 |
"max_length": 64,
|
| 30 |
"default_diagnostic_threshold": 0.5,
|
| 31 |
"selected_epoch": 5,
|
| 32 |
+
"training_metrics": "https://github.com/vxpipe/barge-in-detector/blob/836de6924c3da68cd871e6e08da34291e9778432/reports/e5-small-run1/training-metrics.json",
|
| 33 |
+
"source_repo": "https://github.com/vxpipe/barge-in-detector",
|
| 34 |
+
"code_revision": "836de6924c3da68cd871e6e08da34291e9778432",
|
| 35 |
+
"first_publication_source_commit": "1f92917a59908e320c3a3489c3e5de413a3e4672",
|
| 36 |
+
"ownership": "Release artifacts only. All development code, tests, reports and labnotes are in the GitHub source repository."
|
| 37 |
}
|
reports/challenge-predictions.csv
DELETED
|
@@ -1,99 +0,0 @@
|
|
| 1 |
-
case_id,category,group,language,label,text,text_clean,review_status,p_barge_in,prediction,seen_in
|
| 2 |
-
en-001,short_interruption,,en,barge_in,Stop,stop,pending,0.013443702831864357,backchannel,
|
| 3 |
-
en-002,short_interruption,,en,barge_in,Wait,wait,pending,0.013491915538907051,backchannel,
|
| 4 |
-
en-003,short_interruption,,en,barge_in,Pause,pause,pending,0.013399876654148102,backchannel,
|
| 5 |
-
en-004,short_interruption,,en,barge_in,Cancel,cancel,pending,0.0172202717512846,backchannel,
|
| 6 |
-
en-005,short_interruption,,en,barge_in,Wrong,wrong,pending,0.015773363411426544,backchannel,
|
| 7 |
-
en-006,short_interruption,,en,barge_in,Repeat,repeat,pending,0.01339809875935316,backchannel,
|
| 8 |
-
en-007,short_interruption,,en,barge_in,Slower,slower,pending,0.014412087388336658,backchannel,
|
| 9 |
-
en-008,short_interruption,,en,barge_in,Restart,restart,pending,0.014540360309183598,backchannel,
|
| 10 |
-
en-009,short_interruption,,en,barge_in,Enough,enough,pending,0.013300837948918343,backchannel,
|
| 11 |
-
en-010,short_interruption,,en,barge_in,Stop talking,stop talking,pending,0.01404176652431488,backchannel,
|
| 12 |
-
en-011,short_interruption,,en,barge_in,Wait please,wait please,pending,0.013346969150006771,backchannel,
|
| 13 |
-
en-012,short_interruption,,en,barge_in,Not that,not that,pending,0.015475512482225895,backchannel,
|
| 14 |
-
en-013,short_interruption,,en,barge_in,Wrong date,wrong date,pending,0.9878827333450317,barge_in,
|
| 15 |
-
en-014,short_interruption,,en,barge_in,Wrong person,wrong person,pending,0.8659051060676575,barge_in,
|
| 16 |
-
en-015,short_interruption,,en,barge_in,Go back,go back,pending,0.013336537405848503,backchannel,
|
| 17 |
-
en-016,short_interruption,,en,barge_in,Say again,say again,pending,0.013324333354830742,backchannel,
|
| 18 |
-
en-017,short_interruption,,en,barge_in,Speak slower,speak slower,pending,0.013362826779484749,backchannel,
|
| 19 |
-
en-018,short_interruption,,en,barge_in,Start over,start over,pending,0.01339060626924038,backchannel,
|
| 20 |
-
en-019,short_interruption,,en,barge_in,Let me finish,let me finish,pending,0.013296245597302914,backchannel,
|
| 21 |
-
en-020,short_interruption,,en,barge_in,You misunderstood,you misunderstood,pending,0.013550461269915104,backchannel,
|
| 22 |
-
en-021,short_interruption,,en,barge_in,Cancel it,cancel it,pending,0.015159462578594685,backchannel,
|
| 23 |
-
en-022,short_interruption,,en,barge_in,Change it,change it,pending,0.013769036158919334,backchannel,
|
| 24 |
-
en-023,short_interruption,,en,barge_in,Which one,which one,pending,0.017033280804753304,backchannel,
|
| 25 |
-
en-024,short_interruption,,en,barge_in,Why not,why not,pending,0.014318976551294327,backchannel,
|
| 26 |
-
en-025,short_interruption,,en,barge_in,How much,how much,pending,0.016760120168328285,backchannel,
|
| 27 |
-
en-026,short_interruption,,en,barge_in,What time,what time,pending,0.9822450876235962,barge_in,
|
| 28 |
-
en-027,short_interruption,,en,barge_in,What address,what address,pending,0.9860103130340576,barge_in,
|
| 29 |
-
en-028,short_interruption,,en,barge_in,Tuesday instead,tuesday instead,pending,0.7567923069000244,barge_in,
|
| 30 |
-
en-029,short_interruption,,en,barge_in,The second one,the second one,pending,0.02150067687034607,backchannel,
|
| 31 |
-
en-030,short_interruption,,en,barge_in,That is wrong,that is wrong,pending,0.1904229372739792,backchannel,
|
| 32 |
-
en-031,long_backchannel,,en,backchannel,Yes yes yes yes yes,yes yes yes yes yes,pending,0.013356473296880722,backchannel,
|
| 33 |
-
en-032,long_backchannel,,en,backchannel,Okay okay okay okay okay,okay okay okay okay okay,pending,0.013345411978662014,backchannel,
|
| 34 |
-
en-033,long_backchannel,,en,backchannel,Right right right I see,right right right i see,pending,0.013284565880894661,backchannel,
|
| 35 |
-
en-034,long_backchannel,,en,backchannel,Mm hmm yes mm hmm yes,mm hmm yes mm hmm yes,pending,0.013306752778589725,backchannel,
|
| 36 |
-
en-035,long_backchannel,,en,backchannel,Oh wow oh wow yeah,oh wow oh wow yeah,pending,0.013319982215762138,backchannel,
|
| 37 |
-
en-036,long_backchannel,,en,backchannel,Yeah I see I see,yeah i see i see,pending,0.013288560323417187,backchannel,
|
| 38 |
-
en-037,long_backchannel,,en,backchannel,Yes I understand what you mean,yes i understand what you mean,pending,0.0892898291349411,backchannel,
|
| 39 |
-
en-038,long_backchannel,,en,backchannel,Right that makes sense to me,right that makes sense to me,pending,0.013280664570629597,backchannel,
|
| 40 |
-
en-039,long_backchannel,,en,backchannel,Okay I am following you so far,okay i am following you so far,pending,0.28858309984207153,backchannel,
|
| 41 |
-
en-040,long_backchannel,,en,backchannel,Yes yes that sounds right,yes yes that sounds right,pending,0.013292225077748299,backchannel,
|
| 42 |
-
en-041,long_backchannel,,en,backchannel,Ah I see what you mean now,ah i see what you mean now,pending,0.32564374804496765,backchannel,
|
| 43 |
-
en-042,long_backchannel,,en,backchannel,Mm hmm I understand yes,mm hmm i understand yes,pending,0.0132778724655509,backchannel,
|
| 44 |
-
en-043,long_backchannel,,en,backchannel,Oh dear that sounds really difficult,oh dear that sounds really difficult,pending,0.047465287148952484,backchannel,
|
| 45 |
-
en-044,long_backchannel,,en,backchannel,Wow that must have been exciting,wow that must have been exciting,pending,0.013364902697503567,backchannel,
|
| 46 |
-
en-045,long_backchannel,,en,backchannel,That is interesting yes I see,that is interesting yes i see,pending,0.013278285041451454,backchannel,
|
| 47 |
-
en-046,long_backchannel,,en,backchannel,Fair enough I understand that,fair enough i understand that,pending,0.013284215703606606,backchannel,
|
| 48 |
-
en-047,long_backchannel,,en,backchannel,Yes absolutely I agree with that,yes absolutely i agree with that,pending,0.013294750824570656,backchannel,
|
| 49 |
-
en-048,long_backchannel,,en,backchannel,Right okay yes that makes sense,right okay yes that makes sense,pending,0.013285371474921703,backchannel,
|
| 50 |
-
en-049,long_backchannel,,en,backchannel,Oh no that is really sad,oh no that is really sad,pending,0.01329362578690052,backchannel,
|
| 51 |
-
en-050,long_backchannel,,en,backchannel,Ha ha that is funny,ha ha that is funny,pending,0.013284615240991116,backchannel,
|
| 52 |
-
en-051,long_backchannel,,en,backchannel,Mmm yes okay I hear you,mmm yes okay i hear you,pending,0.013274511322379112,backchannel,
|
| 53 |
-
en-052,long_backchannel,,en,backchannel,Sure sure I am with you,sure sure i am with you,pending,0.013305414468050003,backchannel,
|
| 54 |
-
en-053,long_backchannel,,en,backchannel,Got it yes understood,got it yes understood,pending,0.013286358676850796,backchannel,
|
| 55 |
-
en-054,long_backchannel,,en,backchannel,Uh huh uh huh right,uh huh uh huh right,pending,0.013300181366503239,backchannel,
|
| 56 |
-
en-055,ordinary_backchannel,,en,backchannel,Okay,okay,pending,0.013304519467055798,backchannel,
|
| 57 |
-
en-056,ordinary_backchannel,,en,backchannel,Yes,yes,pending,0.013361463323235512,backchannel,
|
| 58 |
-
en-057,ordinary_backchannel,,en,backchannel,Yeah,yeah,pending,0.013312126509845257,backchannel,
|
| 59 |
-
en-058,ordinary_backchannel,,en,backchannel,Mm hmm,mm hmm,pending,0.013299750164151192,backchannel,validation
|
| 60 |
-
en-059,ordinary_backchannel,,en,backchannel,Uh huh,uh huh,pending,0.013296402990818024,backchannel,train
|
| 61 |
-
en-060,ordinary_backchannel,,en,backchannel,Right,right,pending,0.013306460343301296,backchannel,
|
| 62 |
-
en-061,ordinary_backchannel,,en,backchannel,I see,i see,pending,0.013295376673340797,backchannel,
|
| 63 |
-
en-062,ordinary_backchannel,,en,backchannel,Got it,got it,pending,0.01331440731883049,backchannel,
|
| 64 |
-
en-063,ordinary_backchannel,,en,backchannel,Oh wow,oh wow,pending,0.013293537311255932,backchannel,train
|
| 65 |
-
en-064,ordinary_backchannel,,en,backchannel,I understand,i understand,pending,0.013287884183228016,backchannel,
|
| 66 |
-
en-065,ordinary_interruption,,en,barge_in,I need to change my delivery address,i need to change my delivery address,pending,0.9881365299224854,barge_in,
|
| 67 |
-
en-066,ordinary_interruption,,en,barge_in,You have the wrong customer on the line,you have the wrong customer on the line,pending,0.9880920648574829,barge_in,
|
| 68 |
-
en-067,ordinary_interruption,,en,barge_in,Can you explain what that fee is for,can you explain what that fee is for,pending,0.9879797697067261,barge_in,
|
| 69 |
-
en-068,ordinary_interruption,,en,barge_in,Please stop reading and let me speak,please stop reading and let me speak,pending,0.9880907535552979,barge_in,
|
| 70 |
-
en-069,ordinary_interruption,,en,barge_in,Actually I already paid that invoice,actually i already paid that invoice,pending,0.9881226420402527,barge_in,
|
| 71 |
-
en-070,ordinary_interruption,,en,barge_in,I want to speak to someone else,i want to speak to someone else,pending,0.9879534244537354,barge_in,
|
| 72 |
-
en-071,ordinary_interruption,,en,barge_in,No I said Thursday and not Tuesday,no i said thursday and not tuesday,pending,0.9881582856178284,barge_in,
|
| 73 |
-
en-072,ordinary_interruption,,en,barge_in,Before you continue I have a question,before you continue i have a question,pending,0.9879493713378906,barge_in,
|
| 74 |
-
en-073,ordinary_interruption,,en,barge_in,That is not what I was asking about,that is not what i was asking about,pending,0.9879181981086731,barge_in,
|
| 75 |
-
en-074,ordinary_interruption,,en,barge_in,I cannot hear you please say that again,i cannot hear you please say that again,pending,0.9881183505058289,barge_in,
|
| 76 |
-
en-075,prefix,prefix-1,en,backchannel,Yes,yes,pending,0.013361463323235512,backchannel,
|
| 77 |
-
en-076,prefix,prefix-1,en,backchannel,Yes yes,yes yes,pending,0.013298917561769485,backchannel,
|
| 78 |
-
en-077,prefix,prefix-1,en,barge_in,Yes yes but change the time,yes yes but change the time,pending,0.9880699515342712,barge_in,
|
| 79 |
-
en-078,prefix,prefix-2,en,backchannel,Okay,okay,pending,0.013304519467055798,backchannel,
|
| 80 |
-
en-079,prefix,prefix-2,en,backchannel,Okay okay,okay okay,pending,0.01330505684018135,backchannel,
|
| 81 |
-
en-080,prefix,prefix-2,en,barge_in,Okay okay wait that is the wrong address,okay okay wait that is the wrong address,pending,0.988145112991333,barge_in,
|
| 82 |
-
en-081,prefix,prefix-3,en,backchannel,Right,right,pending,0.013306460343301296,backchannel,
|
| 83 |
-
en-082,prefix,prefix-3,en,backchannel,Right right,right right,pending,0.01329006627202034,backchannel,train
|
| 84 |
-
en-083,prefix,prefix-3,en,barge_in,Right right but I meant the other account,right right but i meant the other account,pending,0.988115668296814,barge_in,
|
| 85 |
-
en-084,prefix,prefix-4,en,backchannel,Mm hmm,mm hmm,pending,0.013299750164151192,backchannel,validation
|
| 86 |
-
en-085,prefix,prefix-4,en,backchannel,Mm hmm yes,mm hmm yes,pending,0.013292054645717144,backchannel,
|
| 87 |
-
en-086,prefix,prefix-4,en,barge_in,Mm hmm yes can you repeat the price,mm hmm yes can you repeat the price,pending,0.9881290793418884,barge_in,
|
| 88 |
-
en-087,prefix,prefix-5,en,backchannel,I see,i see,pending,0.013295376673340797,backchannel,
|
| 89 |
-
en-088,prefix,prefix-5,en,backchannel,I see yes,i see yes,pending,0.013287578709423542,backchannel,
|
| 90 |
-
en-089,prefix,prefix-5,en,barge_in,I see yes but I need to cancel,i see yes but i need to cancel,pending,0.9879875183105469,barge_in,
|
| 91 |
-
en-090,prefix,prefix-6,en,backchannel,Sure,sure,pending,0.013307417742908001,backchannel,
|
| 92 |
-
en-091,prefix,prefix-6,en,backchannel,Sure sure,sure sure,pending,0.013290559872984886,backchannel,
|
| 93 |
-
en-092,prefix,prefix-6,en,barge_in,Sure sure stop there please,sure sure stop there please,pending,0.013294538483023643,backchannel,
|
| 94 |
-
en-093,prefix,prefix-7,en,backchannel,Got it,got it,pending,0.01331440731883049,backchannel,
|
| 95 |
-
en-094,prefix,prefix-7,en,backchannel,Got it okay,got it okay,pending,0.013302822597324848,backchannel,
|
| 96 |
-
en-095,prefix,prefix-7,en,barge_in,Got it okay now let me explain,got it okay now let me explain,pending,0.6444560885429382,barge_in,
|
| 97 |
-
en-096,prefix,prefix-8,en,backchannel,Yeah,yeah,pending,0.013312126509845257,backchannel,
|
| 98 |
-
en-097,prefix,prefix-8,en,backchannel,Yeah yeah,yeah yeah,pending,0.013376967050135136,backchannel,
|
| 99 |
-
en-098,prefix,prefix-8,en,barge_in,Yeah yeah no that will not work for me,yeah yeah no that will not work for me,pending,0.9878116250038147,barge_in,
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
reports/challenge-report.json
DELETED
|
@@ -1,438 +0,0 @@
|
|
| 1 |
-
{
|
| 2 |
-
"challenge_sha256": "3d46b9e024f4d3bedd69f07603cde934e85ec2272988e90d2acebc30d5e99519",
|
| 3 |
-
"model_sha256": "bddb65c08f6260cfc9d320e65b4229229ec9f3ca95204aed98ab5ae45ff758d3",
|
| 4 |
-
"threshold": 0.5,
|
| 5 |
-
"runtime": {
|
| 6 |
-
"device": "cpu",
|
| 7 |
-
"torch": "2.15.0+cpu",
|
| 8 |
-
"transformers": "5.23.0",
|
| 9 |
-
"threads": 4
|
| 10 |
-
},
|
| 11 |
-
"batch_vs_single_max_probability_difference": 8.285045623779297e-06,
|
| 12 |
-
"batch_vs_single_label_disagreements": 0,
|
| 13 |
-
"review_status": "Owner reviewed all 98 cases; 13 subsequent revisions explicitly authorized; final binary labels match authored v1 labels. Diagnostic development set, not independent representative evaluation.",
|
| 14 |
-
"all": {
|
| 15 |
-
"count": 98,
|
| 16 |
-
"accuracy": 0.7346938775510204,
|
| 17 |
-
"macro_f1": 0.7111111111111111,
|
| 18 |
-
"false_interruption_rate": 0.0,
|
| 19 |
-
"missed_interruption_rate": 0.5416666666666666,
|
| 20 |
-
"per_class": {
|
| 21 |
-
"backchannel": {
|
| 22 |
-
"precision": 0.6578947368421053,
|
| 23 |
-
"recall": 1.0,
|
| 24 |
-
"f1": 0.7936507936507937,
|
| 25 |
-
"count": 50
|
| 26 |
-
},
|
| 27 |
-
"barge_in": {
|
| 28 |
-
"precision": 1.0,
|
| 29 |
-
"recall": 0.4583333333333333,
|
| 30 |
-
"f1": 0.6285714285714286,
|
| 31 |
-
"count": 48
|
| 32 |
-
}
|
| 33 |
-
},
|
| 34 |
-
"per_language": {
|
| 35 |
-
"en": {
|
| 36 |
-
"accuracy": 0.7346938775510204,
|
| 37 |
-
"count": 98
|
| 38 |
-
}
|
| 39 |
-
}
|
| 40 |
-
},
|
| 41 |
-
"by_category": {
|
| 42 |
-
"long_backchannel": {
|
| 43 |
-
"count": 24,
|
| 44 |
-
"accuracy": 1.0,
|
| 45 |
-
"macro_f1": 0.5,
|
| 46 |
-
"false_interruption_rate": 0.0,
|
| 47 |
-
"missed_interruption_rate": 0.0,
|
| 48 |
-
"per_class": {
|
| 49 |
-
"backchannel": {
|
| 50 |
-
"precision": 1.0,
|
| 51 |
-
"recall": 1.0,
|
| 52 |
-
"f1": 1.0,
|
| 53 |
-
"count": 24
|
| 54 |
-
},
|
| 55 |
-
"barge_in": {
|
| 56 |
-
"precision": 0.0,
|
| 57 |
-
"recall": 0.0,
|
| 58 |
-
"f1": 0.0,
|
| 59 |
-
"count": 0
|
| 60 |
-
}
|
| 61 |
-
},
|
| 62 |
-
"per_language": {
|
| 63 |
-
"en": {
|
| 64 |
-
"accuracy": 1.0,
|
| 65 |
-
"count": 24
|
| 66 |
-
}
|
| 67 |
-
}
|
| 68 |
-
},
|
| 69 |
-
"ordinary_backchannel": {
|
| 70 |
-
"count": 10,
|
| 71 |
-
"accuracy": 1.0,
|
| 72 |
-
"macro_f1": 0.5,
|
| 73 |
-
"false_interruption_rate": 0.0,
|
| 74 |
-
"missed_interruption_rate": 0.0,
|
| 75 |
-
"per_class": {
|
| 76 |
-
"backchannel": {
|
| 77 |
-
"precision": 1.0,
|
| 78 |
-
"recall": 1.0,
|
| 79 |
-
"f1": 1.0,
|
| 80 |
-
"count": 10
|
| 81 |
-
},
|
| 82 |
-
"barge_in": {
|
| 83 |
-
"precision": 0.0,
|
| 84 |
-
"recall": 0.0,
|
| 85 |
-
"f1": 0.0,
|
| 86 |
-
"count": 0
|
| 87 |
-
}
|
| 88 |
-
},
|
| 89 |
-
"per_language": {
|
| 90 |
-
"en": {
|
| 91 |
-
"accuracy": 1.0,
|
| 92 |
-
"count": 10
|
| 93 |
-
}
|
| 94 |
-
}
|
| 95 |
-
},
|
| 96 |
-
"ordinary_interruption": {
|
| 97 |
-
"count": 10,
|
| 98 |
-
"accuracy": 1.0,
|
| 99 |
-
"macro_f1": 0.5,
|
| 100 |
-
"false_interruption_rate": 0.0,
|
| 101 |
-
"missed_interruption_rate": 0.0,
|
| 102 |
-
"per_class": {
|
| 103 |
-
"backchannel": {
|
| 104 |
-
"precision": 0.0,
|
| 105 |
-
"recall": 0.0,
|
| 106 |
-
"f1": 0.0,
|
| 107 |
-
"count": 0
|
| 108 |
-
},
|
| 109 |
-
"barge_in": {
|
| 110 |
-
"precision": 1.0,
|
| 111 |
-
"recall": 1.0,
|
| 112 |
-
"f1": 1.0,
|
| 113 |
-
"count": 10
|
| 114 |
-
}
|
| 115 |
-
},
|
| 116 |
-
"per_language": {
|
| 117 |
-
"en": {
|
| 118 |
-
"accuracy": 1.0,
|
| 119 |
-
"count": 10
|
| 120 |
-
}
|
| 121 |
-
}
|
| 122 |
-
},
|
| 123 |
-
"prefix": {
|
| 124 |
-
"count": 24,
|
| 125 |
-
"accuracy": 0.9583333333333334,
|
| 126 |
-
"macro_f1": 0.9515151515151515,
|
| 127 |
-
"false_interruption_rate": 0.0,
|
| 128 |
-
"missed_interruption_rate": 0.125,
|
| 129 |
-
"per_class": {
|
| 130 |
-
"backchannel": {
|
| 131 |
-
"precision": 0.9411764705882353,
|
| 132 |
-
"recall": 1.0,
|
| 133 |
-
"f1": 0.9696969696969697,
|
| 134 |
-
"count": 16
|
| 135 |
-
},
|
| 136 |
-
"barge_in": {
|
| 137 |
-
"precision": 1.0,
|
| 138 |
-
"recall": 0.875,
|
| 139 |
-
"f1": 0.9333333333333333,
|
| 140 |
-
"count": 8
|
| 141 |
-
}
|
| 142 |
-
},
|
| 143 |
-
"per_language": {
|
| 144 |
-
"en": {
|
| 145 |
-
"accuracy": 0.9583333333333334,
|
| 146 |
-
"count": 24
|
| 147 |
-
}
|
| 148 |
-
}
|
| 149 |
-
},
|
| 150 |
-
"short_interruption": {
|
| 151 |
-
"count": 30,
|
| 152 |
-
"accuracy": 0.16666666666666666,
|
| 153 |
-
"macro_f1": 0.14285714285714285,
|
| 154 |
-
"false_interruption_rate": 0.0,
|
| 155 |
-
"missed_interruption_rate": 0.8333333333333334,
|
| 156 |
-
"per_class": {
|
| 157 |
-
"backchannel": {
|
| 158 |
-
"precision": 0.0,
|
| 159 |
-
"recall": 0.0,
|
| 160 |
-
"f1": 0.0,
|
| 161 |
-
"count": 0
|
| 162 |
-
},
|
| 163 |
-
"barge_in": {
|
| 164 |
-
"precision": 1.0,
|
| 165 |
-
"recall": 0.16666666666666666,
|
| 166 |
-
"f1": 0.2857142857142857,
|
| 167 |
-
"count": 30
|
| 168 |
-
}
|
| 169 |
-
},
|
| 170 |
-
"per_language": {
|
| 171 |
-
"en": {
|
| 172 |
-
"accuracy": 0.16666666666666666,
|
| 173 |
-
"count": 30
|
| 174 |
-
}
|
| 175 |
-
}
|
| 176 |
-
}
|
| 177 |
-
},
|
| 178 |
-
"overlap_counts": {
|
| 179 |
-
"train": 3,
|
| 180 |
-
"validation": 2,
|
| 181 |
-
"test": 0
|
| 182 |
-
},
|
| 183 |
-
"cross_split_exact_overlap": {
|
| 184 |
-
"train/validation": 0,
|
| 185 |
-
"train/test": 0,
|
| 186 |
-
"validation/test": 0
|
| 187 |
-
},
|
| 188 |
-
"unseen": {
|
| 189 |
-
"count": 93,
|
| 190 |
-
"accuracy": 0.7204301075268817,
|
| 191 |
-
"macro_f1": 0.702216748768473,
|
| 192 |
-
"false_interruption_rate": 0.0,
|
| 193 |
-
"missed_interruption_rate": 0.5416666666666666,
|
| 194 |
-
"per_class": {
|
| 195 |
-
"backchannel": {
|
| 196 |
-
"precision": 0.6338028169014085,
|
| 197 |
-
"recall": 1.0,
|
| 198 |
-
"f1": 0.7758620689655172,
|
| 199 |
-
"count": 45
|
| 200 |
-
},
|
| 201 |
-
"barge_in": {
|
| 202 |
-
"precision": 1.0,
|
| 203 |
-
"recall": 0.4583333333333333,
|
| 204 |
-
"f1": 0.6285714285714286,
|
| 205 |
-
"count": 48
|
| 206 |
-
}
|
| 207 |
-
},
|
| 208 |
-
"per_language": {
|
| 209 |
-
"en": {
|
| 210 |
-
"accuracy": 0.7204301075268817,
|
| 211 |
-
"count": 93
|
| 212 |
-
}
|
| 213 |
-
}
|
| 214 |
-
},
|
| 215 |
-
"prefix_groups": {
|
| 216 |
-
"prefix-1": [
|
| 217 |
-
{
|
| 218 |
-
"text": "yes",
|
| 219 |
-
"label": "backchannel",
|
| 220 |
-
"p_barge_in": 0.013361463323235512
|
| 221 |
-
},
|
| 222 |
-
{
|
| 223 |
-
"text": "yes yes",
|
| 224 |
-
"label": "backchannel",
|
| 225 |
-
"p_barge_in": 0.013298917561769485
|
| 226 |
-
},
|
| 227 |
-
{
|
| 228 |
-
"text": "yes yes but change the time",
|
| 229 |
-
"label": "barge_in",
|
| 230 |
-
"p_barge_in": 0.9880699515342712
|
| 231 |
-
}
|
| 232 |
-
],
|
| 233 |
-
"prefix-2": [
|
| 234 |
-
{
|
| 235 |
-
"text": "okay",
|
| 236 |
-
"label": "backchannel",
|
| 237 |
-
"p_barge_in": 0.013304519467055798
|
| 238 |
-
},
|
| 239 |
-
{
|
| 240 |
-
"text": "okay okay",
|
| 241 |
-
"label": "backchannel",
|
| 242 |
-
"p_barge_in": 0.01330505684018135
|
| 243 |
-
},
|
| 244 |
-
{
|
| 245 |
-
"text": "okay okay wait that is the wrong address",
|
| 246 |
-
"label": "barge_in",
|
| 247 |
-
"p_barge_in": 0.988145112991333
|
| 248 |
-
}
|
| 249 |
-
],
|
| 250 |
-
"prefix-3": [
|
| 251 |
-
{
|
| 252 |
-
"text": "right",
|
| 253 |
-
"label": "backchannel",
|
| 254 |
-
"p_barge_in": 0.013306460343301296
|
| 255 |
-
},
|
| 256 |
-
{
|
| 257 |
-
"text": "right right",
|
| 258 |
-
"label": "backchannel",
|
| 259 |
-
"p_barge_in": 0.01329006627202034
|
| 260 |
-
},
|
| 261 |
-
{
|
| 262 |
-
"text": "right right but i meant the other account",
|
| 263 |
-
"label": "barge_in",
|
| 264 |
-
"p_barge_in": 0.988115668296814
|
| 265 |
-
}
|
| 266 |
-
],
|
| 267 |
-
"prefix-4": [
|
| 268 |
-
{
|
| 269 |
-
"text": "mm hmm",
|
| 270 |
-
"label": "backchannel",
|
| 271 |
-
"p_barge_in": 0.013299750164151192
|
| 272 |
-
},
|
| 273 |
-
{
|
| 274 |
-
"text": "mm hmm yes",
|
| 275 |
-
"label": "backchannel",
|
| 276 |
-
"p_barge_in": 0.013292054645717144
|
| 277 |
-
},
|
| 278 |
-
{
|
| 279 |
-
"text": "mm hmm yes can you repeat the price",
|
| 280 |
-
"label": "barge_in",
|
| 281 |
-
"p_barge_in": 0.9881290793418884
|
| 282 |
-
}
|
| 283 |
-
],
|
| 284 |
-
"prefix-5": [
|
| 285 |
-
{
|
| 286 |
-
"text": "i see",
|
| 287 |
-
"label": "backchannel",
|
| 288 |
-
"p_barge_in": 0.013295376673340797
|
| 289 |
-
},
|
| 290 |
-
{
|
| 291 |
-
"text": "i see yes",
|
| 292 |
-
"label": "backchannel",
|
| 293 |
-
"p_barge_in": 0.013287578709423542
|
| 294 |
-
},
|
| 295 |
-
{
|
| 296 |
-
"text": "i see yes but i need to cancel",
|
| 297 |
-
"label": "barge_in",
|
| 298 |
-
"p_barge_in": 0.9879875183105469
|
| 299 |
-
}
|
| 300 |
-
],
|
| 301 |
-
"prefix-6": [
|
| 302 |
-
{
|
| 303 |
-
"text": "sure",
|
| 304 |
-
"label": "backchannel",
|
| 305 |
-
"p_barge_in": 0.013307417742908001
|
| 306 |
-
},
|
| 307 |
-
{
|
| 308 |
-
"text": "sure sure",
|
| 309 |
-
"label": "backchannel",
|
| 310 |
-
"p_barge_in": 0.013290559872984886
|
| 311 |
-
},
|
| 312 |
-
{
|
| 313 |
-
"text": "sure sure stop there please",
|
| 314 |
-
"label": "barge_in",
|
| 315 |
-
"p_barge_in": 0.013294538483023643
|
| 316 |
-
}
|
| 317 |
-
],
|
| 318 |
-
"prefix-7": [
|
| 319 |
-
{
|
| 320 |
-
"text": "got it",
|
| 321 |
-
"label": "backchannel",
|
| 322 |
-
"p_barge_in": 0.01331440731883049
|
| 323 |
-
},
|
| 324 |
-
{
|
| 325 |
-
"text": "got it okay",
|
| 326 |
-
"label": "backchannel",
|
| 327 |
-
"p_barge_in": 0.013302822597324848
|
| 328 |
-
},
|
| 329 |
-
{
|
| 330 |
-
"text": "got it okay now let me explain",
|
| 331 |
-
"label": "barge_in",
|
| 332 |
-
"p_barge_in": 0.6444560885429382
|
| 333 |
-
}
|
| 334 |
-
],
|
| 335 |
-
"prefix-8": [
|
| 336 |
-
{
|
| 337 |
-
"text": "yeah",
|
| 338 |
-
"label": "backchannel",
|
| 339 |
-
"p_barge_in": 0.013312126509845257
|
| 340 |
-
},
|
| 341 |
-
{
|
| 342 |
-
"text": "yeah yeah",
|
| 343 |
-
"label": "backchannel",
|
| 344 |
-
"p_barge_in": 0.013376967050135136
|
| 345 |
-
},
|
| 346 |
-
{
|
| 347 |
-
"text": "yeah yeah no that will not work for me",
|
| 348 |
-
"label": "barge_in",
|
| 349 |
-
"p_barge_in": 0.9878116250038147
|
| 350 |
-
}
|
| 351 |
-
]
|
| 352 |
-
},
|
| 353 |
-
"prefix_groups_all_labels_correct": 7,
|
| 354 |
-
"english_length_baseline": {
|
| 355 |
-
"cutoff_words": 6,
|
| 356 |
-
"validation": {
|
| 357 |
-
"count": 81,
|
| 358 |
-
"accuracy": 0.9753086419753086,
|
| 359 |
-
"macro_f1": 0.9716783216783216,
|
| 360 |
-
"false_interruption_rate": 0.07407407407407407,
|
| 361 |
-
"missed_interruption_rate": 0.0,
|
| 362 |
-
"per_class": {
|
| 363 |
-
"backchannel": {
|
| 364 |
-
"precision": 1.0,
|
| 365 |
-
"recall": 0.9259259259259259,
|
| 366 |
-
"f1": 0.9615384615384615,
|
| 367 |
-
"count": 27
|
| 368 |
-
},
|
| 369 |
-
"barge_in": {
|
| 370 |
-
"precision": 0.9642857142857143,
|
| 371 |
-
"recall": 1.0,
|
| 372 |
-
"f1": 0.9818181818181818,
|
| 373 |
-
"count": 54
|
| 374 |
-
}
|
| 375 |
-
},
|
| 376 |
-
"per_language": {
|
| 377 |
-
"en": {
|
| 378 |
-
"accuracy": 0.9753086419753086,
|
| 379 |
-
"count": 81
|
| 380 |
-
}
|
| 381 |
-
}
|
| 382 |
-
},
|
| 383 |
-
"test": {
|
| 384 |
-
"count": 81,
|
| 385 |
-
"accuracy": 0.9382716049382716,
|
| 386 |
-
"macro_f1": 0.9268556980314249,
|
| 387 |
-
"false_interruption_rate": 0.18518518518518517,
|
| 388 |
-
"missed_interruption_rate": 0.0,
|
| 389 |
-
"per_class": {
|
| 390 |
-
"backchannel": {
|
| 391 |
-
"precision": 1.0,
|
| 392 |
-
"recall": 0.8148148148148148,
|
| 393 |
-
"f1": 0.8979591836734693,
|
| 394 |
-
"count": 27
|
| 395 |
-
},
|
| 396 |
-
"barge_in": {
|
| 397 |
-
"precision": 0.9152542372881356,
|
| 398 |
-
"recall": 1.0,
|
| 399 |
-
"f1": 0.9557522123893806,
|
| 400 |
-
"count": 54
|
| 401 |
-
}
|
| 402 |
-
},
|
| 403 |
-
"per_language": {
|
| 404 |
-
"en": {
|
| 405 |
-
"accuracy": 0.9382716049382716,
|
| 406 |
-
"count": 81
|
| 407 |
-
}
|
| 408 |
-
}
|
| 409 |
-
},
|
| 410 |
-
"challenge": {
|
| 411 |
-
"count": 98,
|
| 412 |
-
"accuracy": 0.5510204081632653,
|
| 413 |
-
"macro_f1": 0.5315080399826162,
|
| 414 |
-
"false_interruption_rate": 0.26,
|
| 415 |
-
"missed_interruption_rate": 0.6458333333333334,
|
| 416 |
-
"per_class": {
|
| 417 |
-
"backchannel": {
|
| 418 |
-
"precision": 0.5441176470588235,
|
| 419 |
-
"recall": 0.74,
|
| 420 |
-
"f1": 0.6271186440677965,
|
| 421 |
-
"count": 50
|
| 422 |
-
},
|
| 423 |
-
"barge_in": {
|
| 424 |
-
"precision": 0.5666666666666667,
|
| 425 |
-
"recall": 0.3541666666666667,
|
| 426 |
-
"f1": 0.4358974358974359,
|
| 427 |
-
"count": 48
|
| 428 |
-
}
|
| 429 |
-
},
|
| 430 |
-
"per_language": {
|
| 431 |
-
"en": {
|
| 432 |
-
"accuracy": 0.5510204081632653,
|
| 433 |
-
"count": 98
|
| 434 |
-
}
|
| 435 |
-
}
|
| 436 |
-
}
|
| 437 |
-
}
|
| 438 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
reports/cpu-benchmark.json
DELETED
|
@@ -1,94 +0,0 @@
|
|
| 1 |
-
{
|
| 2 |
-
"model": "models/e5-small-run1",
|
| 3 |
-
"runtime": "PyTorch eager fp32, CPU only",
|
| 4 |
-
"model_sha256": "bddb65c08f6260cfc9d320e65b4229229ec9f3ca95204aed98ab5ae45ff758d3",
|
| 5 |
-
"python": "3.14.4",
|
| 6 |
-
"torch": "2.15.0+cpu",
|
| 7 |
-
"transformers": "5.23.0",
|
| 8 |
-
"platform": "Linux-7.0.0-43-generic-x86_64-with-glibc2.43",
|
| 9 |
-
"cpuinfo": "AMD EPYC-Genoa Processor",
|
| 10 |
-
"load_seconds_excluded": 2.8975897090276703,
|
| 11 |
-
"iterations": 1000,
|
| 12 |
-
"warmup_per_scenario": 100,
|
| 13 |
-
"batch_size": 1,
|
| 14 |
-
"max_length": 64,
|
| 15 |
-
"interop_threads": 1,
|
| 16 |
-
"parameter_bytes": 470618120,
|
| 17 |
-
"corpus_sha256": "8b630782362b05249b3be94b0213850258422be5ddeb62fcf703817b0bf9e11a",
|
| 18 |
-
"corpus_rows": 331,
|
| 19 |
-
"languages": [
|
| 20 |
-
"de",
|
| 21 |
-
"en",
|
| 22 |
-
"es",
|
| 23 |
-
"fil",
|
| 24 |
-
"fr",
|
| 25 |
-
"hi",
|
| 26 |
-
"id",
|
| 27 |
-
"it",
|
| 28 |
-
"ja",
|
| 29 |
-
"kn",
|
| 30 |
-
"ml",
|
| 31 |
-
"ta",
|
| 32 |
-
"te",
|
| 33 |
-
"zh"
|
| 34 |
-
],
|
| 35 |
-
"measurements": [
|
| 36 |
-
{
|
| 37 |
-
"threads": 4,
|
| 38 |
-
"scenario": "multilingual_test",
|
| 39 |
-
"model_path": {
|
| 40 |
-
"p50_ms": 13.088757,
|
| 41 |
-
"p95_ms": 19.419374299999998,
|
| 42 |
-
"p99_ms": 25.316384159999995,
|
| 43 |
-
"mean_ms": 13.625100593,
|
| 44 |
-
"max_ms": 83.599
|
| 45 |
-
},
|
| 46 |
-
"tokenization": {
|
| 47 |
-
"p50_ms": 0.222889,
|
| 48 |
-
"p95_ms": 0.30127505,
|
| 49 |
-
"p99_ms": 0.45513418999999933,
|
| 50 |
-
"mean_ms": 0.23647671,
|
| 51 |
-
"max_ms": 2.407625
|
| 52 |
-
},
|
| 53 |
-
"forward_and_probability": {
|
| 54 |
-
"p50_ms": 12.8741705,
|
| 55 |
-
"p95_ms": 19.1815334,
|
| 56 |
-
"p99_ms": 25.09679618,
|
| 57 |
-
"mean_ms": 13.388623883,
|
| 58 |
-
"max_ms": 83.391099
|
| 59 |
-
}
|
| 60 |
-
},
|
| 61 |
-
{
|
| 62 |
-
"threads": 4,
|
| 63 |
-
"scenario": "full_64_tokens",
|
| 64 |
-
"model_path": {
|
| 65 |
-
"p50_ms": 26.794403000000003,
|
| 66 |
-
"p95_ms": 33.269715649999995,
|
| 67 |
-
"p99_ms": 37.862168669999996,
|
| 68 |
-
"mean_ms": 27.779209678,
|
| 69 |
-
"max_ms": 93.899777
|
| 70 |
-
},
|
| 71 |
-
"tokenization": {
|
| 72 |
-
"p50_ms": 0.6677489999999999,
|
| 73 |
-
"p95_ms": 0.8626962999999997,
|
| 74 |
-
"p99_ms": 1.7089376799999993,
|
| 75 |
-
"mean_ms": 0.704945851,
|
| 76 |
-
"max_ms": 7.957467
|
| 77 |
-
},
|
| 78 |
-
"forward_and_probability": {
|
| 79 |
-
"p50_ms": 26.103834499999998,
|
| 80 |
-
"p95_ms": 32.575319,
|
| 81 |
-
"p99_ms": 37.10665824,
|
| 82 |
-
"mean_ms": 27.074263827,
|
| 83 |
-
"max_ms": 93.297535
|
| 84 |
-
}
|
| 85 |
-
}
|
| 86 |
-
],
|
| 87 |
-
"scope": "Local virtual CPU; reference CPU unspecified. Excludes Vxpipe boundary and queueing; no whole-path acceptance claim.",
|
| 88 |
-
"corpus_token_lengths": {
|
| 89 |
-
"p50": 13.0,
|
| 90 |
-
"p95": 19.0,
|
| 91 |
-
"max": 26
|
| 92 |
-
},
|
| 93 |
-
"peak_process_rss_mib": 1163.6171875
|
| 94 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
reports/cpu-cold-benchmark.json
DELETED
|
@@ -1,33 +0,0 @@
|
|
| 1 |
-
{
|
| 2 |
-
"cold_process_seconds": [
|
| 3 |
-
4.768154836,
|
| 4 |
-
4.810785561,
|
| 5 |
-
4.136632718,
|
| 6 |
-
4.255589102,
|
| 7 |
-
3.847660745,
|
| 8 |
-
4.045793715,
|
| 9 |
-
3.973606716,
|
| 10 |
-
4.190341404,
|
| 11 |
-
4.387973674,
|
| 12 |
-
4.28136222
|
| 13 |
-
],
|
| 14 |
-
"loaded_to_first_label_ms": [
|
| 15 |
-
26.506046,
|
| 16 |
-
40.193806,
|
| 17 |
-
18.425675,
|
| 18 |
-
22.50301,
|
| 19 |
-
23.081508,
|
| 20 |
-
40.155708,
|
| 21 |
-
16.969254,
|
| 22 |
-
23.919822,
|
| 23 |
-
23.060476,
|
| 24 |
-
24.334184
|
| 25 |
-
],
|
| 26 |
-
"loaded_to_first_label_median_ms": 23.500664999999998,
|
| 27 |
-
"median_seconds": 4.222965253,
|
| 28 |
-
"min_seconds": 3.847660745,
|
| 29 |
-
"max_seconds": 4.810785561,
|
| 30 |
-
"runs": 10,
|
| 31 |
-
"threads": 4,
|
| 32 |
-
"scope": "Python process creation through reading first flushed label. Includes imports, tokenizer/model load, first inference. Local files only; OS file cache may be warm. Excludes process shutdown, downloads and HTTP/network boundary."
|
| 33 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
reports/cpu-thread-sweep.json
DELETED
|
@@ -1,194 +0,0 @@
|
|
| 1 |
-
{
|
| 2 |
-
"model": "models/e5-small-run1",
|
| 3 |
-
"runtime": "PyTorch eager fp32, CPU only",
|
| 4 |
-
"model_sha256": "bddb65c08f6260cfc9d320e65b4229229ec9f3ca95204aed98ab5ae45ff758d3",
|
| 5 |
-
"python": "3.14.4",
|
| 6 |
-
"torch": "2.15.0+cpu",
|
| 7 |
-
"transformers": "5.23.0",
|
| 8 |
-
"platform": "Linux-7.0.0-43-generic-x86_64-with-glibc2.43",
|
| 9 |
-
"cpuinfo": "AMD EPYC-Genoa Processor",
|
| 10 |
-
"load_seconds_excluded": 3.1383294460829347,
|
| 11 |
-
"iterations": 1000,
|
| 12 |
-
"warmup_per_scenario": 100,
|
| 13 |
-
"batch_size": 1,
|
| 14 |
-
"max_length": 64,
|
| 15 |
-
"interop_threads": 1,
|
| 16 |
-
"parameter_bytes": 470618120,
|
| 17 |
-
"corpus_sha256": "8b630782362b05249b3be94b0213850258422be5ddeb62fcf703817b0bf9e11a",
|
| 18 |
-
"corpus_rows": 331,
|
| 19 |
-
"languages": [
|
| 20 |
-
"de",
|
| 21 |
-
"en",
|
| 22 |
-
"es",
|
| 23 |
-
"fil",
|
| 24 |
-
"fr",
|
| 25 |
-
"hi",
|
| 26 |
-
"id",
|
| 27 |
-
"it",
|
| 28 |
-
"ja",
|
| 29 |
-
"kn",
|
| 30 |
-
"ml",
|
| 31 |
-
"ta",
|
| 32 |
-
"te",
|
| 33 |
-
"zh"
|
| 34 |
-
],
|
| 35 |
-
"corpus_token_lengths": {
|
| 36 |
-
"p50": 13.0,
|
| 37 |
-
"p95": 19.0,
|
| 38 |
-
"max": 26
|
| 39 |
-
},
|
| 40 |
-
"measurements": [
|
| 41 |
-
{
|
| 42 |
-
"threads": 1,
|
| 43 |
-
"scenario": "multilingual_test",
|
| 44 |
-
"model_path": {
|
| 45 |
-
"p50_ms": 20.395149500000002,
|
| 46 |
-
"p95_ms": 25.477123599999995,
|
| 47 |
-
"p99_ms": 28.2604041,
|
| 48 |
-
"mean_ms": 20.432651312,
|
| 49 |
-
"max_ms": 46.969913
|
| 50 |
-
},
|
| 51 |
-
"tokenization": {
|
| 52 |
-
"p50_ms": 0.23074050000000002,
|
| 53 |
-
"p95_ms": 0.29089349999999997,
|
| 54 |
-
"p99_ms": 0.33883417,
|
| 55 |
-
"mean_ms": 0.234400798,
|
| 56 |
-
"max_ms": 0.640549
|
| 57 |
-
},
|
| 58 |
-
"forward_and_probability": {
|
| 59 |
-
"p50_ms": 20.162170500000002,
|
| 60 |
-
"p95_ms": 25.207171749999997,
|
| 61 |
-
"p99_ms": 28.02515483,
|
| 62 |
-
"mean_ms": 20.198250514,
|
| 63 |
-
"max_ms": 46.709714
|
| 64 |
-
}
|
| 65 |
-
},
|
| 66 |
-
{
|
| 67 |
-
"threads": 1,
|
| 68 |
-
"scenario": "full_64_tokens",
|
| 69 |
-
"model_path": {
|
| 70 |
-
"p50_ms": 44.6955575,
|
| 71 |
-
"p95_ms": 50.545173899999995,
|
| 72 |
-
"p99_ms": 53.76246217,
|
| 73 |
-
"mean_ms": 45.379929543,
|
| 74 |
-
"max_ms": 57.243489
|
| 75 |
-
},
|
| 76 |
-
"tokenization": {
|
| 77 |
-
"p50_ms": 0.6161775,
|
| 78 |
-
"p95_ms": 0.7613426,
|
| 79 |
-
"p99_ms": 0.84228681,
|
| 80 |
-
"mean_ms": 0.631765294,
|
| 81 |
-
"max_ms": 0.891216
|
| 82 |
-
},
|
| 83 |
-
"forward_and_probability": {
|
| 84 |
-
"p50_ms": 44.084312499999996,
|
| 85 |
-
"p95_ms": 49.9083266,
|
| 86 |
-
"p99_ms": 53.07674325,
|
| 87 |
-
"mean_ms": 44.748164249,
|
| 88 |
-
"max_ms": 56.405244
|
| 89 |
-
}
|
| 90 |
-
},
|
| 91 |
-
{
|
| 92 |
-
"threads": 2,
|
| 93 |
-
"scenario": "multilingual_test",
|
| 94 |
-
"model_path": {
|
| 95 |
-
"p50_ms": 19.406584000000002,
|
| 96 |
-
"p95_ms": 23.1596329,
|
| 97 |
-
"p99_ms": 25.6503671,
|
| 98 |
-
"mean_ms": 19.536068109,
|
| 99 |
-
"max_ms": 28.630568
|
| 100 |
-
},
|
| 101 |
-
"tokenization": {
|
| 102 |
-
"p50_ms": 0.23920799999999998,
|
| 103 |
-
"p95_ms": 0.31088855,
|
| 104 |
-
"p99_ms": 0.36521733999999995,
|
| 105 |
-
"mean_ms": 0.244807045,
|
| 106 |
-
"max_ms": 0.475011
|
| 107 |
-
},
|
| 108 |
-
"forward_and_probability": {
|
| 109 |
-
"p50_ms": 19.154210499999998,
|
| 110 |
-
"p95_ms": 22.828605549999995,
|
| 111 |
-
"p99_ms": 25.33145532,
|
| 112 |
-
"mean_ms": 19.291261064,
|
| 113 |
-
"max_ms": 28.31147
|
| 114 |
-
}
|
| 115 |
-
},
|
| 116 |
-
{
|
| 117 |
-
"threads": 2,
|
| 118 |
-
"scenario": "full_64_tokens",
|
| 119 |
-
"model_path": {
|
| 120 |
-
"p50_ms": 38.023134999999996,
|
| 121 |
-
"p95_ms": 45.51507395,
|
| 122 |
-
"p99_ms": 53.98291254,
|
| 123 |
-
"mean_ms": 38.847798787,
|
| 124 |
-
"max_ms": 62.944328
|
| 125 |
-
},
|
| 126 |
-
"tokenization": {
|
| 127 |
-
"p50_ms": 0.636223,
|
| 128 |
-
"p95_ms": 0.8138672499999998,
|
| 129 |
-
"p99_ms": 0.9455087999999999,
|
| 130 |
-
"mean_ms": 0.659222932,
|
| 131 |
-
"max_ms": 1.44213
|
| 132 |
-
},
|
| 133 |
-
"forward_and_probability": {
|
| 134 |
-
"p50_ms": 37.383997,
|
| 135 |
-
"p95_ms": 44.81712879999999,
|
| 136 |
-
"p99_ms": 53.05854066,
|
| 137 |
-
"mean_ms": 38.188575855,
|
| 138 |
-
"max_ms": 61.995607
|
| 139 |
-
}
|
| 140 |
-
},
|
| 141 |
-
{
|
| 142 |
-
"threads": 4,
|
| 143 |
-
"scenario": "multilingual_test",
|
| 144 |
-
"model_path": {
|
| 145 |
-
"p50_ms": 13.1820115,
|
| 146 |
-
"p95_ms": 19.21180095,
|
| 147 |
-
"p99_ms": 24.81463212999999,
|
| 148 |
-
"mean_ms": 13.731945525,
|
| 149 |
-
"max_ms": 89.107149
|
| 150 |
-
},
|
| 151 |
-
"tokenization": {
|
| 152 |
-
"p50_ms": 0.2276755,
|
| 153 |
-
"p95_ms": 0.30882,
|
| 154 |
-
"p99_ms": 0.3798859499999996,
|
| 155 |
-
"mean_ms": 0.23842783,
|
| 156 |
-
"max_ms": 4.009645
|
| 157 |
-
},
|
| 158 |
-
"forward_and_probability": {
|
| 159 |
-
"p50_ms": 12.937595,
|
| 160 |
-
"p95_ms": 18.841387699999995,
|
| 161 |
-
"p99_ms": 24.56872725999999,
|
| 162 |
-
"mean_ms": 13.493517695,
|
| 163 |
-
"max_ms": 88.911056
|
| 164 |
-
}
|
| 165 |
-
},
|
| 166 |
-
{
|
| 167 |
-
"threads": 4,
|
| 168 |
-
"scenario": "full_64_tokens",
|
| 169 |
-
"model_path": {
|
| 170 |
-
"p50_ms": 26.392652,
|
| 171 |
-
"p95_ms": 32.489633899999994,
|
| 172 |
-
"p99_ms": 41.04232582999999,
|
| 173 |
-
"mean_ms": 27.24307507,
|
| 174 |
-
"max_ms": 93.673078
|
| 175 |
-
},
|
| 176 |
-
"tokenization": {
|
| 177 |
-
"p50_ms": 0.614555,
|
| 178 |
-
"p95_ms": 0.7339711499999999,
|
| 179 |
-
"p99_ms": 0.9166563899999999,
|
| 180 |
-
"mean_ms": 0.624646764,
|
| 181 |
-
"max_ms": 2.092334
|
| 182 |
-
},
|
| 183 |
-
"forward_and_probability": {
|
| 184 |
-
"p50_ms": 25.7561225,
|
| 185 |
-
"p95_ms": 31.824941300000003,
|
| 186 |
-
"p99_ms": 40.46172110999999,
|
| 187 |
-
"mean_ms": 26.618428306,
|
| 188 |
-
"max_ms": 93.090937
|
| 189 |
-
}
|
| 190 |
-
}
|
| 191 |
-
],
|
| 192 |
-
"scope": "Local virtual CPU; reference CPU unspecified. Excludes Vxpipe boundary and queueing; no whole-path acceptance claim.",
|
| 193 |
-
"peak_process_rss_mib": 1168.8125
|
| 194 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
reports/eval-test.json
DELETED
|
@@ -1,81 +0,0 @@
|
|
| 1 |
-
{
|
| 2 |
-
"count": 331,
|
| 3 |
-
"accuracy": 1.0,
|
| 4 |
-
"macro_f1": 1.0,
|
| 5 |
-
"false_interruption_rate": 0.0,
|
| 6 |
-
"missed_interruption_rate": 0.0,
|
| 7 |
-
"per_class": {
|
| 8 |
-
"backchannel": {
|
| 9 |
-
"precision": 1.0,
|
| 10 |
-
"recall": 1.0,
|
| 11 |
-
"f1": 1.0,
|
| 12 |
-
"count": 109
|
| 13 |
-
},
|
| 14 |
-
"barge_in": {
|
| 15 |
-
"precision": 1.0,
|
| 16 |
-
"recall": 1.0,
|
| 17 |
-
"f1": 1.0,
|
| 18 |
-
"count": 222
|
| 19 |
-
}
|
| 20 |
-
},
|
| 21 |
-
"per_language": {
|
| 22 |
-
"de": {
|
| 23 |
-
"accuracy": 1.0,
|
| 24 |
-
"count": 17
|
| 25 |
-
},
|
| 26 |
-
"en": {
|
| 27 |
-
"accuracy": 1.0,
|
| 28 |
-
"count": 81
|
| 29 |
-
},
|
| 30 |
-
"es": {
|
| 31 |
-
"accuracy": 1.0,
|
| 32 |
-
"count": 36
|
| 33 |
-
},
|
| 34 |
-
"fil": {
|
| 35 |
-
"accuracy": 1.0,
|
| 36 |
-
"count": 18
|
| 37 |
-
},
|
| 38 |
-
"fr": {
|
| 39 |
-
"accuracy": 1.0,
|
| 40 |
-
"count": 18
|
| 41 |
-
},
|
| 42 |
-
"hi": {
|
| 43 |
-
"accuracy": 1.0,
|
| 44 |
-
"count": 18
|
| 45 |
-
},
|
| 46 |
-
"id": {
|
| 47 |
-
"accuracy": 1.0,
|
| 48 |
-
"count": 17
|
| 49 |
-
},
|
| 50 |
-
"it": {
|
| 51 |
-
"accuracy": 1.0,
|
| 52 |
-
"count": 18
|
| 53 |
-
},
|
| 54 |
-
"ja": {
|
| 55 |
-
"accuracy": 1.0,
|
| 56 |
-
"count": 18
|
| 57 |
-
},
|
| 58 |
-
"kn": {
|
| 59 |
-
"accuracy": 1.0,
|
| 60 |
-
"count": 18
|
| 61 |
-
},
|
| 62 |
-
"ml": {
|
| 63 |
-
"accuracy": 1.0,
|
| 64 |
-
"count": 18
|
| 65 |
-
},
|
| 66 |
-
"ta": {
|
| 67 |
-
"accuracy": 1.0,
|
| 68 |
-
"count": 18
|
| 69 |
-
},
|
| 70 |
-
"te": {
|
| 71 |
-
"accuracy": 1.0,
|
| 72 |
-
"count": 18
|
| 73 |
-
},
|
| 74 |
-
"zh": {
|
| 75 |
-
"accuracy": 1.0,
|
| 76 |
-
"count": 18
|
| 77 |
-
}
|
| 78 |
-
},
|
| 79 |
-
"split": "test",
|
| 80 |
-
"threshold": 0.5
|
| 81 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
reports/eval-validation.json
DELETED
|
@@ -1,81 +0,0 @@
|
|
| 1 |
-
{
|
| 2 |
-
"count": 333,
|
| 3 |
-
"accuracy": 0.993993993993994,
|
| 4 |
-
"macro_f1": 0.9932432432432432,
|
| 5 |
-
"false_interruption_rate": 0.009009009009009009,
|
| 6 |
-
"missed_interruption_rate": 0.0045045045045045045,
|
| 7 |
-
"per_class": {
|
| 8 |
-
"backchannel": {
|
| 9 |
-
"precision": 0.990990990990991,
|
| 10 |
-
"recall": 0.990990990990991,
|
| 11 |
-
"f1": 0.990990990990991,
|
| 12 |
-
"count": 111
|
| 13 |
-
},
|
| 14 |
-
"barge_in": {
|
| 15 |
-
"precision": 0.9954954954954955,
|
| 16 |
-
"recall": 0.9954954954954955,
|
| 17 |
-
"f1": 0.9954954954954955,
|
| 18 |
-
"count": 222
|
| 19 |
-
}
|
| 20 |
-
},
|
| 21 |
-
"per_language": {
|
| 22 |
-
"de": {
|
| 23 |
-
"accuracy": 1.0,
|
| 24 |
-
"count": 18
|
| 25 |
-
},
|
| 26 |
-
"en": {
|
| 27 |
-
"accuracy": 1.0,
|
| 28 |
-
"count": 81
|
| 29 |
-
},
|
| 30 |
-
"es": {
|
| 31 |
-
"accuracy": 1.0,
|
| 32 |
-
"count": 36
|
| 33 |
-
},
|
| 34 |
-
"fil": {
|
| 35 |
-
"accuracy": 1.0,
|
| 36 |
-
"count": 18
|
| 37 |
-
},
|
| 38 |
-
"fr": {
|
| 39 |
-
"accuracy": 1.0,
|
| 40 |
-
"count": 18
|
| 41 |
-
},
|
| 42 |
-
"hi": {
|
| 43 |
-
"accuracy": 1.0,
|
| 44 |
-
"count": 18
|
| 45 |
-
},
|
| 46 |
-
"id": {
|
| 47 |
-
"accuracy": 1.0,
|
| 48 |
-
"count": 18
|
| 49 |
-
},
|
| 50 |
-
"it": {
|
| 51 |
-
"accuracy": 1.0,
|
| 52 |
-
"count": 18
|
| 53 |
-
},
|
| 54 |
-
"ja": {
|
| 55 |
-
"accuracy": 1.0,
|
| 56 |
-
"count": 18
|
| 57 |
-
},
|
| 58 |
-
"kn": {
|
| 59 |
-
"accuracy": 1.0,
|
| 60 |
-
"count": 18
|
| 61 |
-
},
|
| 62 |
-
"ml": {
|
| 63 |
-
"accuracy": 1.0,
|
| 64 |
-
"count": 18
|
| 65 |
-
},
|
| 66 |
-
"ta": {
|
| 67 |
-
"accuracy": 0.9444444444444444,
|
| 68 |
-
"count": 18
|
| 69 |
-
},
|
| 70 |
-
"te": {
|
| 71 |
-
"accuracy": 1.0,
|
| 72 |
-
"count": 18
|
| 73 |
-
},
|
| 74 |
-
"zh": {
|
| 75 |
-
"accuracy": 0.9444444444444444,
|
| 76 |
-
"count": 18
|
| 77 |
-
}
|
| 78 |
-
},
|
| 79 |
-
"split": "validation",
|
| 80 |
-
"threshold": 0.5
|
| 81 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
reports/source-manifest.json
DELETED
|
@@ -1,73 +0,0 @@
|
|
| 1 |
-
{
|
| 2 |
-
".gitattributes": "d76d90c6cfbe7c1e793b13dc70485f1a71c80b19c72e7f3a2829786ac9eef810",
|
| 3 |
-
".gitignore": "01e2dc4c57a1279bf408ae19a88c62b2e29e92ded5bdc337ca4cadff47dd92fc",
|
| 4 |
-
".python-version": "a876e0b10411037a012498b9fe18d9bc1df32ed8b722a13564dc944ddcfd9135",
|
| 5 |
-
"AGENTS.md": "e9e62c3227ce52f8ba930c3f5119816c168a7f1477f102eb7fd781598c68b1f1",
|
| 6 |
-
"barge_in_detector/__init__.py": "28de6fe1d126dec7b9cd85273f805831b24fb7c443ed2707c47552caede29072",
|
| 7 |
-
"barge_in_detector/__main__.py": "8efaf592230270b84d1ba8513bdb244b104aba73a517f0734197ccbfa18afb8a",
|
| 8 |
-
"barge_in_detector/catalogue.py": "8b59351941bce2a92420ea0b13989a4c002a52d23c6279c10331645fac9796e2",
|
| 9 |
-
"barge_in_detector/cleaner.py": "58bcb1c3f21f66aecb534cbc20e07d5350c3d8d4c92e2084c4943ca9493c3a22",
|
| 10 |
-
"barge_in_detector/cli.py": "a68d6d7348ab34520f0d8ba88731367da6af0fb574548a290e43f45bd8941517",
|
| 11 |
-
"barge_in_detector/generation/__init__.py": "b81aff66375501799977abd59786a5ad9da99f6998febc5fac64a016a6cea3fd",
|
| 12 |
-
"barge_in_detector/generation/config.py": "844d5945b93c44ff04a35f6f161a1ab29a29d39ffacc7af1b268276818d20662",
|
| 13 |
-
"barge_in_detector/generation/llm.py": "972ee568321255f6ceeee3f34c323eedac5b5ec83062d67cc662b53eee86b7a9",
|
| 14 |
-
"barge_in_detector/generation/prompts.py": "04c3b61b4ef0a7cbd5edcb0438f544b564c602e0d213291dafbcff9fe1f086ce",
|
| 15 |
-
"barge_in_detector/generation/responses.py": "cb5849ddbc90e800bd8c20a17896c0f7cb406b6eacf7126ddc2a1d6468c1ef5d",
|
| 16 |
-
"barge_in_detector/generation/utterance_criteria.py": "78b938e758fa5f8d8d6dedbc48096747a8521eca6812822ef8cccfd70ff62c72",
|
| 17 |
-
"barge_in_detector/generation/utterance_runner.py": "51d383bee865e79242997a6ef1a3ac2add896fb86969689cdbd7a8d26cf734a4",
|
| 18 |
-
"barge_in_detector/generation/utterances.py": "535f7c54e9dff1ed61903713b8c825e98b453932b753da61b46d41479374cd42",
|
| 19 |
-
"barge_in_detector/ids.py": "7a5abdf7ebfb64e5fb1588f887fd951a0df345d71c90d59b445ef2a8c1bf214e",
|
| 20 |
-
"barge_in_detector/storage/__init__.py": "4e44293470af3982b64c05c2d17a949185c10b4bd53bf72b5ba6229f51507db4",
|
| 21 |
-
"barge_in_detector/storage/schema.py": "0ccc76ec546a0fe1c414fed1f9f866b0103a55319f04e6d5d3bed12e83862e03",
|
| 22 |
-
"barge_in_detector/storage/store.py": "b3b32e195dc06653702e307d7199cc380b4a74dd687ab860836d491c8166311f",
|
| 23 |
-
"barge_in_detector/training/__init__.py": "230e5ab5c3a00f056980f02d06acc65a0efe31b7738a18caf9c7c6de1b33c1d1",
|
| 24 |
-
"barge_in_detector/training/evaluate.py": "4762ae1a3df0b942049b1eb4dd9faad7e37b5c77bb8200f2041e9e3f30ee1e5e",
|
| 25 |
-
"barge_in_detector/training/metrics.py": "6909e1cf3e4ac6b763c97ea9a8c26b5ee751a01fffc86c308604d5e437bbfad6",
|
| 26 |
-
"barge_in_detector/training/split.py": "44e05118111617a5649d94d09e82228864f2f2f5405a15dde974db78d9829a29",
|
| 27 |
-
"barge_in_detector/training/train.py": "a643f92b5c99d7c318e460dc2fdb5ae0bbb37003a8dae28fc3be60f8d37d1bd2",
|
| 28 |
-
"bin/create-labnotes": "5306e747935b105f878fa5f144c83595debb3f32f87c7d097720975bb1729884",
|
| 29 |
-
"data/export/utterances.csv": "556a2820e4a24ac13bab44321fb32b464c5fb225750d41a5a10ebcde7cdbab8a",
|
| 30 |
-
"data/splits/test.csv": "8b630782362b05249b3be94b0213850258422be5ddeb62fcf703817b0bf9e11a",
|
| 31 |
-
"data/splits/train.csv": "aab224bd0526eee915295e6f5c52076e29bb751cd0d664eefb4321843ad22533",
|
| 32 |
-
"data/splits/validation.csv": "c6b4c526fb4b1df96754119e409cedc8c7e85c77659e1fc1c3829da70084edc1",
|
| 33 |
-
"docs/data-format.md": "5d14d3aac1e432157c03bdf10cabf87e7366164629c70fb5904d632f472254ff",
|
| 34 |
-
"docs/factor-catalogue.md": "651e27021557eb7ce08c3ae5af0e341dee1aba665c1b6f72ad19625768167ec5",
|
| 35 |
-
"docs/generation.md": "1aae32dc0cb9c35904f8b2c5721ce7910494a27384a97b5f40fe0373c2d70b33",
|
| 36 |
-
"docs/training.md": "a2a96447aa065e4cb9f1e8a4b9ec97041d41ee32736cb68115706638b562d68f",
|
| 37 |
-
"docs/transcript-format.md": "9927370e6fb906d1dd1fabd98888d7e95d8ba2f373e7a1248e430f260c57aa32",
|
| 38 |
-
"docs/utterance-first-decision.md": "6c9ab89b7d579d83769e09eb3b77bf1cfa70434fe0cc45e24aaa959385a064e5",
|
| 39 |
-
"generation/config.yaml": "f4e707d4efc610b882289f29f170cd763c876c4e366f6853d1225cbb5965c88b",
|
| 40 |
-
"generation/factors/catalogue.yaml": "1135f3a3bf6000eb1f9d41b1957a348c760e78f96fbe61a8829aa818be0de5b5",
|
| 41 |
-
"generation/factors/constraints.yaml": "422318e1010483e4c5f7eb0de99170eb906b8d21927183a5d6581f13c21ac59c",
|
| 42 |
-
"generation/factors/domains.yaml": "19605dbdd0bb4044c481d74ce5c798534d61dd6c7c46d4edcb2f800c4aa53fc5",
|
| 43 |
-
"generation/factors/intents.yaml": "a3d343addabde711a2395e1ed598f432dd00e547afa962255ae246e17874ae63",
|
| 44 |
-
"generation/factors/languages.yaml": "c5f0a5b753c588782c536108184b12615a54c9b8f4b4cbf0073c0d8aec1a376e",
|
| 45 |
-
"generation/factors/utterances.yaml": "a045546c59d59588a261b03e1f1dc9dc147146f93d3381a813c0920825c435c4",
|
| 46 |
-
"generation/plans/english.yaml": "f1e7e17e2249bb736cd5cb75b1af0f0babbe3b2320e12b2b812346ff5c12d725",
|
| 47 |
-
"generation/plans/utterance-all.yaml": "20349fb83cd17d067736e4d26d746be7a77fa7cd7a8d41569fc44335cefb0a5d",
|
| 48 |
-
"generation/plans/utterance-sample.yaml": "7ab683e2c38af4f0105c957971efa2caaeca17c7bc2e11397e362591d0a31276",
|
| 49 |
-
"generation/prompts/generator_utterances_v5.md": "0a2e3eeeb7b7c7ff1ad2e2de8973af5469fcb8c369d9092d251c7590477b1266",
|
| 50 |
-
"labnotes/20261114-0907-project-brief-setup.md": "ab069c92129480d35758375cf5c57cbce96a643f007ed1d7d5941a6718bf0c51",
|
| 51 |
-
"labnotes/20261114-1023-factor-catalogue-draft.md": "6e3c2261c5bdb325548f59763ababe178d1f442f6440497c47d53c2773b513ee",
|
| 52 |
-
"labnotes/20261114-1541-synthetic-data-generator.md": "1b5f4bb84e3c224164d4e295b7639aad3a27d82d01948220b0c177fd1e93ea2b",
|
| 53 |
-
"labnotes/20261115-0100-utterance-first-generation.md": "5e81c46e2799a970b783e660b06ed16fdf4ea7bc43859887a4ae2d42fe5f6d0f",
|
| 54 |
-
"labnotes/20261115-1211-vastai-model-training.md": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
|
| 55 |
-
"labnotes/issues/.gitkeep": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
|
| 56 |
-
"labnotes/milestones/index.md": "0f3b3889fbbb7ea869c3c4e264d1da7cbd4535b28890113f35edcb8cf39b7463",
|
| 57 |
-
"labnotes/milestones/multilingual-text-barge-in-classifier.md": "c292dbefd18e5056cf219f8bd5c66403bcf291eab0cb9589840beb198ce5a74d",
|
| 58 |
-
"labnotes/milestones/utterance-first-backchannel-classifier.md": "e3bf3bd0ce94ac80d5b3ac4d697535cdb95fda30d7d4d1cad282b2d7e40c0d16",
|
| 59 |
-
"pyproject.toml": "4aef3a80e879d58c4c559b1f276e4b51c57006583cce3fd7a302e91336d9bbef",
|
| 60 |
-
"tests/fake_llm.py": "2c604f09ff23619225e0c62a57466f300191fd8391777ac7cf05e5556dee4f33",
|
| 61 |
-
"tests/fixtures/cleaner_golden.yaml": "3e013d3775a619a08432fe6c0ced5558934e4dbcf98a4afa651a0bd4bfc1c1db",
|
| 62 |
-
"tests/test_catalogue.py": "9fbfc7bc848e5db38d34ccb08f6bf10bd0908b21cbcc73d70d337c1e0dbae2ca",
|
| 63 |
-
"tests/test_cleaner.py": "048942dafd6e91772d7b17515606cb6c7f44c2225b876542d8598d19116a3d47",
|
| 64 |
-
"tests/test_gemini_live.py": "64f4ba3a80503ad7e0a43bfbaf7e43fbecbbc5821a41b189c08da5eff390e884",
|
| 65 |
-
"tests/test_ids.py": "3802dc4bd8b75f8764c1d4eafbc882b614096d5d871ec4f5d001990bb3f87d89",
|
| 66 |
-
"tests/test_llm.py": "eb8255bc88a0d9b8dde268ae67d700f5e2406c87b82cbda27dbba920ba9ad19a",
|
| 67 |
-
"tests/test_metrics.py": "072656cb74e8750f1a9a52f5d6dadb2ac3250e5f1cd538d6e64f3bd5a46d9567",
|
| 68 |
-
"tests/test_split.py": "d4556a43db6f706e6bac282bb5749792ddc66f12ba439d0b7fa1ad6cf666ff63",
|
| 69 |
-
"tests/test_store.py": "f23ab069512838929f7f0f91f8439ceaac4ba38484a6b235750fd7adf672614d",
|
| 70 |
-
"tests/test_utterance_criteria.py": "bb076041b038a6b707a6df61ee708408bec89922c011fef8dce0824991691a53",
|
| 71 |
-
"tests/test_utterances.py": "3a48f590d83cddc4db2f57099f5f90bff3b965195f0aeb1adcd663a27bc1a387",
|
| 72 |
-
"uv.lock": "3992e539e9e18aeb3182978ef9ef0b6ae8c0e15d649416e2de11b9ba98317ba8"
|
| 73 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
reports/training-metrics.json
DELETED
|
@@ -1,519 +0,0 @@
|
|
| 1 |
-
{
|
| 2 |
-
"args": {
|
| 3 |
-
"out": "models/e5-small-run1",
|
| 4 |
-
"model": "intfloat/multilingual-e5-small",
|
| 5 |
-
"splits": "data/splits",
|
| 6 |
-
"epochs": "5",
|
| 7 |
-
"batch_size": "32",
|
| 8 |
-
"lr": "3e-05",
|
| 9 |
-
"warmup": "0.1",
|
| 10 |
-
"max_length": "64",
|
| 11 |
-
"seed": "20261115"
|
| 12 |
-
},
|
| 13 |
-
"best": {
|
| 14 |
-
"epoch": 5,
|
| 15 |
-
"train_loss": 0.02041708185817256,
|
| 16 |
-
"seconds": 1.7,
|
| 17 |
-
"validation": {
|
| 18 |
-
"count": 333,
|
| 19 |
-
"accuracy": 0.993993993993994,
|
| 20 |
-
"macro_f1": 0.9932432432432432,
|
| 21 |
-
"false_interruption_rate": 0.009009009009009009,
|
| 22 |
-
"missed_interruption_rate": 0.0045045045045045045,
|
| 23 |
-
"per_class": {
|
| 24 |
-
"backchannel": {
|
| 25 |
-
"precision": 0.990990990990991,
|
| 26 |
-
"recall": 0.990990990990991,
|
| 27 |
-
"f1": 0.990990990990991,
|
| 28 |
-
"count": 111
|
| 29 |
-
},
|
| 30 |
-
"barge_in": {
|
| 31 |
-
"precision": 0.9954954954954955,
|
| 32 |
-
"recall": 0.9954954954954955,
|
| 33 |
-
"f1": 0.9954954954954955,
|
| 34 |
-
"count": 222
|
| 35 |
-
}
|
| 36 |
-
},
|
| 37 |
-
"per_language": {
|
| 38 |
-
"de": {
|
| 39 |
-
"accuracy": 1.0,
|
| 40 |
-
"count": 18
|
| 41 |
-
},
|
| 42 |
-
"en": {
|
| 43 |
-
"accuracy": 1.0,
|
| 44 |
-
"count": 81
|
| 45 |
-
},
|
| 46 |
-
"es": {
|
| 47 |
-
"accuracy": 1.0,
|
| 48 |
-
"count": 36
|
| 49 |
-
},
|
| 50 |
-
"fil": {
|
| 51 |
-
"accuracy": 1.0,
|
| 52 |
-
"count": 18
|
| 53 |
-
},
|
| 54 |
-
"fr": {
|
| 55 |
-
"accuracy": 1.0,
|
| 56 |
-
"count": 18
|
| 57 |
-
},
|
| 58 |
-
"hi": {
|
| 59 |
-
"accuracy": 1.0,
|
| 60 |
-
"count": 18
|
| 61 |
-
},
|
| 62 |
-
"id": {
|
| 63 |
-
"accuracy": 1.0,
|
| 64 |
-
"count": 18
|
| 65 |
-
},
|
| 66 |
-
"it": {
|
| 67 |
-
"accuracy": 1.0,
|
| 68 |
-
"count": 18
|
| 69 |
-
},
|
| 70 |
-
"ja": {
|
| 71 |
-
"accuracy": 1.0,
|
| 72 |
-
"count": 18
|
| 73 |
-
},
|
| 74 |
-
"kn": {
|
| 75 |
-
"accuracy": 1.0,
|
| 76 |
-
"count": 18
|
| 77 |
-
},
|
| 78 |
-
"ml": {
|
| 79 |
-
"accuracy": 1.0,
|
| 80 |
-
"count": 18
|
| 81 |
-
},
|
| 82 |
-
"ta": {
|
| 83 |
-
"accuracy": 0.9444444444444444,
|
| 84 |
-
"count": 18
|
| 85 |
-
},
|
| 86 |
-
"te": {
|
| 87 |
-
"accuracy": 1.0,
|
| 88 |
-
"count": 18
|
| 89 |
-
},
|
| 90 |
-
"zh": {
|
| 91 |
-
"accuracy": 0.9444444444444444,
|
| 92 |
-
"count": 18
|
| 93 |
-
}
|
| 94 |
-
}
|
| 95 |
-
}
|
| 96 |
-
},
|
| 97 |
-
"history": [
|
| 98 |
-
{
|
| 99 |
-
"epoch": 1,
|
| 100 |
-
"train_loss": 0.48989867342240884,
|
| 101 |
-
"seconds": 2.3,
|
| 102 |
-
"validation": {
|
| 103 |
-
"count": 333,
|
| 104 |
-
"accuracy": 0.984984984984985,
|
| 105 |
-
"macro_f1": 0.98306980527734,
|
| 106 |
-
"false_interruption_rate": 0.02702702702702703,
|
| 107 |
-
"missed_interruption_rate": 0.009009009009009009,
|
| 108 |
-
"per_class": {
|
| 109 |
-
"backchannel": {
|
| 110 |
-
"precision": 0.9818181818181818,
|
| 111 |
-
"recall": 0.972972972972973,
|
| 112 |
-
"f1": 0.9773755656108598,
|
| 113 |
-
"count": 111
|
| 114 |
-
},
|
| 115 |
-
"barge_in": {
|
| 116 |
-
"precision": 0.9865470852017937,
|
| 117 |
-
"recall": 0.990990990990991,
|
| 118 |
-
"f1": 0.9887640449438202,
|
| 119 |
-
"count": 222
|
| 120 |
-
}
|
| 121 |
-
},
|
| 122 |
-
"per_language": {
|
| 123 |
-
"de": {
|
| 124 |
-
"accuracy": 0.9444444444444444,
|
| 125 |
-
"count": 18
|
| 126 |
-
},
|
| 127 |
-
"en": {
|
| 128 |
-
"accuracy": 1.0,
|
| 129 |
-
"count": 81
|
| 130 |
-
},
|
| 131 |
-
"es": {
|
| 132 |
-
"accuracy": 1.0,
|
| 133 |
-
"count": 36
|
| 134 |
-
},
|
| 135 |
-
"fil": {
|
| 136 |
-
"accuracy": 1.0,
|
| 137 |
-
"count": 18
|
| 138 |
-
},
|
| 139 |
-
"fr": {
|
| 140 |
-
"accuracy": 1.0,
|
| 141 |
-
"count": 18
|
| 142 |
-
},
|
| 143 |
-
"hi": {
|
| 144 |
-
"accuracy": 1.0,
|
| 145 |
-
"count": 18
|
| 146 |
-
},
|
| 147 |
-
"id": {
|
| 148 |
-
"accuracy": 1.0,
|
| 149 |
-
"count": 18
|
| 150 |
-
},
|
| 151 |
-
"it": {
|
| 152 |
-
"accuracy": 1.0,
|
| 153 |
-
"count": 18
|
| 154 |
-
},
|
| 155 |
-
"ja": {
|
| 156 |
-
"accuracy": 1.0,
|
| 157 |
-
"count": 18
|
| 158 |
-
},
|
| 159 |
-
"kn": {
|
| 160 |
-
"accuracy": 0.9444444444444444,
|
| 161 |
-
"count": 18
|
| 162 |
-
},
|
| 163 |
-
"ml": {
|
| 164 |
-
"accuracy": 0.9444444444444444,
|
| 165 |
-
"count": 18
|
| 166 |
-
},
|
| 167 |
-
"ta": {
|
| 168 |
-
"accuracy": 0.9444444444444444,
|
| 169 |
-
"count": 18
|
| 170 |
-
},
|
| 171 |
-
"te": {
|
| 172 |
-
"accuracy": 1.0,
|
| 173 |
-
"count": 18
|
| 174 |
-
},
|
| 175 |
-
"zh": {
|
| 176 |
-
"accuracy": 0.9444444444444444,
|
| 177 |
-
"count": 18
|
| 178 |
-
}
|
| 179 |
-
}
|
| 180 |
-
}
|
| 181 |
-
},
|
| 182 |
-
{
|
| 183 |
-
"epoch": 2,
|
| 184 |
-
"train_loss": 0.08939109126275235,
|
| 185 |
-
"seconds": 1.7,
|
| 186 |
-
"validation": {
|
| 187 |
-
"count": 333,
|
| 188 |
-
"accuracy": 0.990990990990991,
|
| 189 |
-
"macro_f1": 0.9898875380862241,
|
| 190 |
-
"false_interruption_rate": 0.009009009009009009,
|
| 191 |
-
"missed_interruption_rate": 0.009009009009009009,
|
| 192 |
-
"per_class": {
|
| 193 |
-
"backchannel": {
|
| 194 |
-
"precision": 0.9821428571428571,
|
| 195 |
-
"recall": 0.990990990990991,
|
| 196 |
-
"f1": 0.9865470852017937,
|
| 197 |
-
"count": 111
|
| 198 |
-
},
|
| 199 |
-
"barge_in": {
|
| 200 |
-
"precision": 0.995475113122172,
|
| 201 |
-
"recall": 0.990990990990991,
|
| 202 |
-
"f1": 0.9932279909706545,
|
| 203 |
-
"count": 222
|
| 204 |
-
}
|
| 205 |
-
},
|
| 206 |
-
"per_language": {
|
| 207 |
-
"de": {
|
| 208 |
-
"accuracy": 1.0,
|
| 209 |
-
"count": 18
|
| 210 |
-
},
|
| 211 |
-
"en": {
|
| 212 |
-
"accuracy": 1.0,
|
| 213 |
-
"count": 81
|
| 214 |
-
},
|
| 215 |
-
"es": {
|
| 216 |
-
"accuracy": 1.0,
|
| 217 |
-
"count": 36
|
| 218 |
-
},
|
| 219 |
-
"fil": {
|
| 220 |
-
"accuracy": 1.0,
|
| 221 |
-
"count": 18
|
| 222 |
-
},
|
| 223 |
-
"fr": {
|
| 224 |
-
"accuracy": 1.0,
|
| 225 |
-
"count": 18
|
| 226 |
-
},
|
| 227 |
-
"hi": {
|
| 228 |
-
"accuracy": 1.0,
|
| 229 |
-
"count": 18
|
| 230 |
-
},
|
| 231 |
-
"id": {
|
| 232 |
-
"accuracy": 1.0,
|
| 233 |
-
"count": 18
|
| 234 |
-
},
|
| 235 |
-
"it": {
|
| 236 |
-
"accuracy": 1.0,
|
| 237 |
-
"count": 18
|
| 238 |
-
},
|
| 239 |
-
"ja": {
|
| 240 |
-
"accuracy": 1.0,
|
| 241 |
-
"count": 18
|
| 242 |
-
},
|
| 243 |
-
"kn": {
|
| 244 |
-
"accuracy": 1.0,
|
| 245 |
-
"count": 18
|
| 246 |
-
},
|
| 247 |
-
"ml": {
|
| 248 |
-
"accuracy": 0.9444444444444444,
|
| 249 |
-
"count": 18
|
| 250 |
-
},
|
| 251 |
-
"ta": {
|
| 252 |
-
"accuracy": 0.9444444444444444,
|
| 253 |
-
"count": 18
|
| 254 |
-
},
|
| 255 |
-
"te": {
|
| 256 |
-
"accuracy": 1.0,
|
| 257 |
-
"count": 18
|
| 258 |
-
},
|
| 259 |
-
"zh": {
|
| 260 |
-
"accuracy": 0.9444444444444444,
|
| 261 |
-
"count": 18
|
| 262 |
-
}
|
| 263 |
-
}
|
| 264 |
-
}
|
| 265 |
-
},
|
| 266 |
-
{
|
| 267 |
-
"epoch": 3,
|
| 268 |
-
"train_loss": 0.03671385896476832,
|
| 269 |
-
"seconds": 1.7,
|
| 270 |
-
"validation": {
|
| 271 |
-
"count": 333,
|
| 272 |
-
"accuracy": 0.987987987987988,
|
| 273 |
-
"macro_f1": 0.9865465416936006,
|
| 274 |
-
"false_interruption_rate": 0.009009009009009009,
|
| 275 |
-
"missed_interruption_rate": 0.013513513513513514,
|
| 276 |
-
"per_class": {
|
| 277 |
-
"backchannel": {
|
| 278 |
-
"precision": 0.9734513274336283,
|
| 279 |
-
"recall": 0.990990990990991,
|
| 280 |
-
"f1": 0.9821428571428571,
|
| 281 |
-
"count": 111
|
| 282 |
-
},
|
| 283 |
-
"barge_in": {
|
| 284 |
-
"precision": 0.9954545454545455,
|
| 285 |
-
"recall": 0.9864864864864865,
|
| 286 |
-
"f1": 0.9909502262443439,
|
| 287 |
-
"count": 222
|
| 288 |
-
}
|
| 289 |
-
},
|
| 290 |
-
"per_language": {
|
| 291 |
-
"de": {
|
| 292 |
-
"accuracy": 1.0,
|
| 293 |
-
"count": 18
|
| 294 |
-
},
|
| 295 |
-
"en": {
|
| 296 |
-
"accuracy": 1.0,
|
| 297 |
-
"count": 81
|
| 298 |
-
},
|
| 299 |
-
"es": {
|
| 300 |
-
"accuracy": 1.0,
|
| 301 |
-
"count": 36
|
| 302 |
-
},
|
| 303 |
-
"fil": {
|
| 304 |
-
"accuracy": 1.0,
|
| 305 |
-
"count": 18
|
| 306 |
-
},
|
| 307 |
-
"fr": {
|
| 308 |
-
"accuracy": 1.0,
|
| 309 |
-
"count": 18
|
| 310 |
-
},
|
| 311 |
-
"hi": {
|
| 312 |
-
"accuracy": 1.0,
|
| 313 |
-
"count": 18
|
| 314 |
-
},
|
| 315 |
-
"id": {
|
| 316 |
-
"accuracy": 1.0,
|
| 317 |
-
"count": 18
|
| 318 |
-
},
|
| 319 |
-
"it": {
|
| 320 |
-
"accuracy": 1.0,
|
| 321 |
-
"count": 18
|
| 322 |
-
},
|
| 323 |
-
"ja": {
|
| 324 |
-
"accuracy": 1.0,
|
| 325 |
-
"count": 18
|
| 326 |
-
},
|
| 327 |
-
"kn": {
|
| 328 |
-
"accuracy": 1.0,
|
| 329 |
-
"count": 18
|
| 330 |
-
},
|
| 331 |
-
"ml": {
|
| 332 |
-
"accuracy": 1.0,
|
| 333 |
-
"count": 18
|
| 334 |
-
},
|
| 335 |
-
"ta": {
|
| 336 |
-
"accuracy": 0.9444444444444444,
|
| 337 |
-
"count": 18
|
| 338 |
-
},
|
| 339 |
-
"te": {
|
| 340 |
-
"accuracy": 1.0,
|
| 341 |
-
"count": 18
|
| 342 |
-
},
|
| 343 |
-
"zh": {
|
| 344 |
-
"accuracy": 0.8333333333333334,
|
| 345 |
-
"count": 18
|
| 346 |
-
}
|
| 347 |
-
}
|
| 348 |
-
}
|
| 349 |
-
},
|
| 350 |
-
{
|
| 351 |
-
"epoch": 4,
|
| 352 |
-
"train_loss": 0.025003968517888674,
|
| 353 |
-
"seconds": 1.7,
|
| 354 |
-
"validation": {
|
| 355 |
-
"count": 333,
|
| 356 |
-
"accuracy": 0.987987987987988,
|
| 357 |
-
"macro_f1": 0.9865465416936006,
|
| 358 |
-
"false_interruption_rate": 0.009009009009009009,
|
| 359 |
-
"missed_interruption_rate": 0.013513513513513514,
|
| 360 |
-
"per_class": {
|
| 361 |
-
"backchannel": {
|
| 362 |
-
"precision": 0.9734513274336283,
|
| 363 |
-
"recall": 0.990990990990991,
|
| 364 |
-
"f1": 0.9821428571428571,
|
| 365 |
-
"count": 111
|
| 366 |
-
},
|
| 367 |
-
"barge_in": {
|
| 368 |
-
"precision": 0.9954545454545455,
|
| 369 |
-
"recall": 0.9864864864864865,
|
| 370 |
-
"f1": 0.9909502262443439,
|
| 371 |
-
"count": 222
|
| 372 |
-
}
|
| 373 |
-
},
|
| 374 |
-
"per_language": {
|
| 375 |
-
"de": {
|
| 376 |
-
"accuracy": 1.0,
|
| 377 |
-
"count": 18
|
| 378 |
-
},
|
| 379 |
-
"en": {
|
| 380 |
-
"accuracy": 1.0,
|
| 381 |
-
"count": 81
|
| 382 |
-
},
|
| 383 |
-
"es": {
|
| 384 |
-
"accuracy": 1.0,
|
| 385 |
-
"count": 36
|
| 386 |
-
},
|
| 387 |
-
"fil": {
|
| 388 |
-
"accuracy": 1.0,
|
| 389 |
-
"count": 18
|
| 390 |
-
},
|
| 391 |
-
"fr": {
|
| 392 |
-
"accuracy": 1.0,
|
| 393 |
-
"count": 18
|
| 394 |
-
},
|
| 395 |
-
"hi": {
|
| 396 |
-
"accuracy": 1.0,
|
| 397 |
-
"count": 18
|
| 398 |
-
},
|
| 399 |
-
"id": {
|
| 400 |
-
"accuracy": 1.0,
|
| 401 |
-
"count": 18
|
| 402 |
-
},
|
| 403 |
-
"it": {
|
| 404 |
-
"accuracy": 1.0,
|
| 405 |
-
"count": 18
|
| 406 |
-
},
|
| 407 |
-
"ja": {
|
| 408 |
-
"accuracy": 1.0,
|
| 409 |
-
"count": 18
|
| 410 |
-
},
|
| 411 |
-
"kn": {
|
| 412 |
-
"accuracy": 1.0,
|
| 413 |
-
"count": 18
|
| 414 |
-
},
|
| 415 |
-
"ml": {
|
| 416 |
-
"accuracy": 1.0,
|
| 417 |
-
"count": 18
|
| 418 |
-
},
|
| 419 |
-
"ta": {
|
| 420 |
-
"accuracy": 0.9444444444444444,
|
| 421 |
-
"count": 18
|
| 422 |
-
},
|
| 423 |
-
"te": {
|
| 424 |
-
"accuracy": 1.0,
|
| 425 |
-
"count": 18
|
| 426 |
-
},
|
| 427 |
-
"zh": {
|
| 428 |
-
"accuracy": 0.8333333333333334,
|
| 429 |
-
"count": 18
|
| 430 |
-
}
|
| 431 |
-
}
|
| 432 |
-
}
|
| 433 |
-
},
|
| 434 |
-
{
|
| 435 |
-
"epoch": 5,
|
| 436 |
-
"train_loss": 0.02041708185817256,
|
| 437 |
-
"seconds": 1.7,
|
| 438 |
-
"validation": {
|
| 439 |
-
"count": 333,
|
| 440 |
-
"accuracy": 0.993993993993994,
|
| 441 |
-
"macro_f1": 0.9932432432432432,
|
| 442 |
-
"false_interruption_rate": 0.009009009009009009,
|
| 443 |
-
"missed_interruption_rate": 0.0045045045045045045,
|
| 444 |
-
"per_class": {
|
| 445 |
-
"backchannel": {
|
| 446 |
-
"precision": 0.990990990990991,
|
| 447 |
-
"recall": 0.990990990990991,
|
| 448 |
-
"f1": 0.990990990990991,
|
| 449 |
-
"count": 111
|
| 450 |
-
},
|
| 451 |
-
"barge_in": {
|
| 452 |
-
"precision": 0.9954954954954955,
|
| 453 |
-
"recall": 0.9954954954954955,
|
| 454 |
-
"f1": 0.9954954954954955,
|
| 455 |
-
"count": 222
|
| 456 |
-
}
|
| 457 |
-
},
|
| 458 |
-
"per_language": {
|
| 459 |
-
"de": {
|
| 460 |
-
"accuracy": 1.0,
|
| 461 |
-
"count": 18
|
| 462 |
-
},
|
| 463 |
-
"en": {
|
| 464 |
-
"accuracy": 1.0,
|
| 465 |
-
"count": 81
|
| 466 |
-
},
|
| 467 |
-
"es": {
|
| 468 |
-
"accuracy": 1.0,
|
| 469 |
-
"count": 36
|
| 470 |
-
},
|
| 471 |
-
"fil": {
|
| 472 |
-
"accuracy": 1.0,
|
| 473 |
-
"count": 18
|
| 474 |
-
},
|
| 475 |
-
"fr": {
|
| 476 |
-
"accuracy": 1.0,
|
| 477 |
-
"count": 18
|
| 478 |
-
},
|
| 479 |
-
"hi": {
|
| 480 |
-
"accuracy": 1.0,
|
| 481 |
-
"count": 18
|
| 482 |
-
},
|
| 483 |
-
"id": {
|
| 484 |
-
"accuracy": 1.0,
|
| 485 |
-
"count": 18
|
| 486 |
-
},
|
| 487 |
-
"it": {
|
| 488 |
-
"accuracy": 1.0,
|
| 489 |
-
"count": 18
|
| 490 |
-
},
|
| 491 |
-
"ja": {
|
| 492 |
-
"accuracy": 1.0,
|
| 493 |
-
"count": 18
|
| 494 |
-
},
|
| 495 |
-
"kn": {
|
| 496 |
-
"accuracy": 1.0,
|
| 497 |
-
"count": 18
|
| 498 |
-
},
|
| 499 |
-
"ml": {
|
| 500 |
-
"accuracy": 1.0,
|
| 501 |
-
"count": 18
|
| 502 |
-
},
|
| 503 |
-
"ta": {
|
| 504 |
-
"accuracy": 0.9444444444444444,
|
| 505 |
-
"count": 18
|
| 506 |
-
},
|
| 507 |
-
"te": {
|
| 508 |
-
"accuracy": 1.0,
|
| 509 |
-
"count": 18
|
| 510 |
-
},
|
| 511 |
-
"zh": {
|
| 512 |
-
"accuracy": 0.9444444444444444,
|
| 513 |
-
"count": 18
|
| 514 |
-
}
|
| 515 |
-
}
|
| 516 |
-
}
|
| 517 |
-
}
|
| 518 |
-
]
|
| 519 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
tests/test_data.py
DELETED
|
@@ -1,43 +0,0 @@
|
|
| 1 |
-
import sys
|
| 2 |
-
from types import SimpleNamespace
|
| 3 |
-
|
| 4 |
-
import pytest
|
| 5 |
-
|
| 6 |
-
from barge_in_detector_model.data import read_csv, resolve_splits
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
def make_splits(path):
|
| 10 |
-
path.mkdir(parents=True)
|
| 11 |
-
for name in ["train", "validation", "test"]:
|
| 12 |
-
(path / f"{name}.csv").write_text("text_clean,label,language\nstop,barge_in,en\n")
|
| 13 |
-
|
| 14 |
-
|
| 15 |
-
def test_local_splits_do_not_require_hub(tmp_path):
|
| 16 |
-
make_splits(tmp_path / "splits")
|
| 17 |
-
directory, revision = resolve_splits(tmp_path / "splits")
|
| 18 |
-
assert revision is None
|
| 19 |
-
assert read_csv(directory / "test.csv")[0]["label"] == "barge_in"
|
| 20 |
-
with pytest.raises(ValueError, match="missing"):
|
| 21 |
-
resolve_splits(tmp_path / "absent")
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
def test_hub_download_uses_resolved_immutable_revision(tmp_path, monkeypatch):
|
| 25 |
-
make_splits(tmp_path / "data/splits")
|
| 26 |
-
seen = {}
|
| 27 |
-
|
| 28 |
-
def download(**kwargs):
|
| 29 |
-
seen.update(kwargs)
|
| 30 |
-
return str(tmp_path)
|
| 31 |
-
|
| 32 |
-
fake = SimpleNamespace(
|
| 33 |
-
HfApi=lambda: SimpleNamespace(
|
| 34 |
-
dataset_info=lambda *a, **k: SimpleNamespace(sha="fixed-commit")
|
| 35 |
-
),
|
| 36 |
-
snapshot_download=download,
|
| 37 |
-
)
|
| 38 |
-
monkeypatch.setitem(sys.modules, "huggingface_hub", fake)
|
| 39 |
-
directory, revision = resolve_splits(None, repo_id="owner/data", revision="v1")
|
| 40 |
-
assert revision == "fixed-commit"
|
| 41 |
-
assert seen["revision"] == "fixed-commit"
|
| 42 |
-
assert seen["repo_type"] == "dataset"
|
| 43 |
-
assert directory == tmp_path / "data/splits"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
tests/test_metrics.py
DELETED
|
@@ -1,29 +0,0 @@
|
|
| 1 |
-
"""Evaluation metrics for the two labels."""
|
| 2 |
-
|
| 3 |
-
import pytest
|
| 4 |
-
|
| 5 |
-
from barge_in_detector_model.metrics import compute
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
def test_metrics_report_false_and_missed_interruptions():
|
| 9 |
-
labels = ["backchannel", "backchannel", "barge_in", "barge_in", "barge_in"]
|
| 10 |
-
preds = ["backchannel", "barge_in", "barge_in", "barge_in", "backchannel"]
|
| 11 |
-
languages = ["en", "en", "en", "hi", "hi"]
|
| 12 |
-
m = compute(labels, preds, languages)
|
| 13 |
-
assert m["accuracy"] == pytest.approx(3 / 5)
|
| 14 |
-
# A backchannel predicted as barge-in cuts the agent off for nothing.
|
| 15 |
-
assert m["false_interruption_rate"] == pytest.approx(1 / 2)
|
| 16 |
-
# A barge-in predicted as backchannel lets the agent talk over the caller.
|
| 17 |
-
assert m["missed_interruption_rate"] == pytest.approx(1 / 3)
|
| 18 |
-
assert m["per_class"]["barge_in"]["precision"] == pytest.approx(2 / 3)
|
| 19 |
-
assert m["per_class"]["barge_in"]["recall"] == pytest.approx(2 / 3)
|
| 20 |
-
assert m["per_language"]["en"]["accuracy"] == pytest.approx(2 / 3)
|
| 21 |
-
assert m["per_language"]["hi"]["count"] == 2
|
| 22 |
-
assert 0 < m["macro_f1"] < 1
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
def test_perfect_predictions():
|
| 26 |
-
labels = ["backchannel", "barge_in"]
|
| 27 |
-
m = compute(labels, labels, ["en", "en"])
|
| 28 |
-
assert m["accuracy"] == m["macro_f1"] == 1.0
|
| 29 |
-
assert m["false_interruption_rate"] == m["missed_interruption_rate"] == 0.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
uv.lock
DELETED
|
The diff for this file is too large to render.
See raw diff
|
|
|