Akash Manohar commited on
Commit
a768911
·
1 Parent(s): e5336e8

Add a shared project introduction and model quick start

Browse files

Link source and dataset releases with common badges, demonstrate CPU inference, and retain evaluation and timing limitations.

Files changed (1) hide show
  1. README.md +103 -45
README.md CHANGED
@@ -7,54 +7,80 @@ base_model: intfloat/multilingual-e5-small
7
  datasets: [HashNuke/barge-in-detector-data]
8
  tags: [barge-in, backchannel, multilingual, experimental]
9
  ---
10
- # Barge-in detector: experimental model release
11
 
12
- Classifies a cleaned caller utterance spoken while a voice agent is talking:
13
- backchannel (continue) or barge_in (yield). Exactly two labels; no audio,
14
- preceding agent context or language hint is required. Answers after the agent
15
- finishes are handled by the runtime. Keep the first 64 tokenizer tokens.
16
 
17
- This repo contains **model release artifacts only**. All training, inference,
18
- evaluation and benchmark code, tests, reports and labnotes live in the
19
- [GitHub source](https://github.com/vxpipe/barge-in-detector).
20
- [Dataset releases](https://huggingface.co/datasets/HashNuke/barge-in-detector-data)
21
- are separate. release.json pins the exact code and dataset revisions and records
22
- artifact hashes and original training provenance.
23
 
24
- **Experimental, not production-ready.** The model misses 25/30 short English
25
- challenge interruptions, including “stop” and “wait”. Probabilities are not
26
- calibrated; the runtime owns thresholds. The 0.5 cutoff is diagnostic.
 
27
 
28
- ## Use
 
 
29
 
30
- Install compatible PyTorch and transformers, then:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
31
 
32
  ```python
33
  import torch
34
- from transformers import AutoTokenizer, AutoModelForSequenceClassification
35
 
36
  repo = "HashNuke/barge-in-detector"
37
  tokenizer = AutoTokenizer.from_pretrained(repo)
38
  model = AutoModelForSequenceClassification.from_pretrained(repo).cpu().eval()
 
 
39
  with torch.inference_mode():
40
- inputs = tokenizer(
41
- "please stop for a moment", return_tensors="pt", truncation=True, max_length=64
42
- )
43
- p_barge_in = model(**inputs).logits.softmax(-1)[0, 1].item()
44
  ```
45
 
46
- Input must already follow the source cleaner: lowercase, punctuation removed;
47
- native scripts optionally mixed with English. Labels are 0=backchannel,
48
- 1=barge_in. Pin the model revision when deploying. Current files are the original
49
- epoch-5 fp32 checkpoint; changing repository ownership does not retrain it.
 
 
 
 
 
 
 
 
 
 
50
 
51
- ## Training and evaluation
 
 
52
 
53
- Base intfloat/multilingual-e5-small (MIT); see LICENSE and NOTICE. First run:
54
- 2,640 synthetic train / 333 validation / 331 test; five epochs, batch 32,
55
- weighted cross-entropy, AdamW, 3e-5 learning rate, 10% warm-up, seed 20261115,
56
- bf16 on one RTX 5090. Epoch 5 selected on validation macro F1 (0.99324).
57
- Training scripts and complete reports are in the pinned GitHub source.
58
 
59
  | Evaluation | Accuracy | Missed interruptions | False interruptions |
60
  | --- | --- | --- | --- |
@@ -63,17 +89,17 @@ Training scripts and complete reports are in the pinned GitHub source.
63
  | Reviewed English challenge (98) | 73.5% | 54.2% | 0% |
64
  | Challenge unseen in any synthetic split (93) | 72.0% | 54.2% | 0% |
65
 
66
- The challenge was authored before inference, owner-reviewed and reconciled with
67
- authorization. Final binary labels match the frozen expectations. It is now a
68
- regression/development set with intentional repeated prefixes. Criteria overlap
69
- between synthetic splits. Exact cleaned-text leakage is absent; template/semantic
70
- leakage remains unaudited. These results do not establish real-world performance.
71
- All 14 listed languages have synthetic results; human-verified accuracy across
72
- all languages has not been established.
73
 
74
- ## CPU timing
75
 
76
- Four virtual AMD EPYC Genoa cores, PyTorch 2.15.0+cpu eager fp32, four threads,
77
  batch one, 100 warm-ups and 1,000 measured calls per scenario:
78
 
79
  | Warm text through label | p50 | p95 | p99 |
@@ -81,8 +107,40 @@ batch one, 100 warm-ups and 1,000 measured calls per scenario:
81
  | Typical multilingual test input | 13.1 ms | 19.4 ms | 25.3 ms |
82
  | Full 64-token budget | 26.8 ms | 33.3 ms | 37.9 ms |
83
 
84
- Cold Python process through first label: median 4.22 seconds, range 3.85–4.81
85
- seconds over ten runs. Includes imports and local loading; OS file cache may
86
- be warm. Peak warm process RSS approximately 1,164 MiB. Network, HTTP, queueing
87
- and Vxpipe boundary are excluded. Reference-CPU gates are unestablished;
88
- ONNX/int8 is unmeasured. See the source labnotes and reports for recorded evidence.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
7
  datasets: [HashNuke/barge-in-detector-data]
8
  tags: [barge-in, backchannel, multilingual, experimental]
9
  ---
 
10
 
11
+ # Barge-in detector
 
 
 
12
 
13
+ **Give acknowledgments room. Give interruptions the floor.**
 
 
 
 
 
14
 
15
+ [![GitHub source](https://img.shields.io/badge/GitHub-source-181717?logo=github&logoColor=white)](https://github.com/vxpipe/barge-in-detector)
16
+ [![Hugging Face model](https://img.shields.io/badge/Hugging%20Face-model-FFD21E?logo=huggingface&logoColor=black)](https://huggingface.co/HashNuke/barge-in-detector)
17
+ [![Hugging Face dataset](https://img.shields.io/badge/Hugging%20Face-dataset-FFD21E?logo=huggingface&logoColor=black)](https://huggingface.co/datasets/HashNuke/barge-in-detector-data)
18
+ [![MIT licence](https://img.shields.io/badge/license-MIT-3DA639)](https://github.com/vxpipe/barge-in-detector/blob/main/LICENSE)
19
 
20
+ Voice agents hear callers say “uh-huh”, “yes”, and “okay” while they are still
21
+ speaking. Those acknowledgments should let the agent continue. Corrections,
22
+ questions, and requests to stop should give the caller the floor.
23
 
24
+ Barge-in detector is an open-source, multilingual, text-only project exploring
25
+ that distinction with synthetic data and a learned classifier that runs on CPU.
26
+
27
+ This repository is the **model release**: weights, tokenizer, configuration,
28
+ licence, and provenance. Training, inference scripts, evaluation, benchmarks,
29
+ and labnotes are maintained in the [GitHub source](https://github.com/vxpipe/barge-in-detector).
30
+
31
+ > **Experimental checkpoint.** The first model misses 25 of 30 short
32
+ > interruptions in the reviewed English challenge. Use it for research and
33
+ > evaluation; its probabilities are not calibrated.
34
+
35
+ ## Run the model
36
+
37
+ The CPU example below was tested on Linux with Python 3.14 and uv:
38
+
39
+ ```shell
40
+ uv venv --python 3.14
41
+ uv pip install torch==2.15.0 --index-url https://download.pytorch.org/whl/cpu
42
+ uv pip install transformers==5.23.0 huggingface-hub==2.4.0
43
+ source .venv/bin/activate
44
+ ```
45
+
46
+ Run this in Python (`python`) or a notebook using that environment:
47
 
48
  ```python
49
  import torch
50
+ from transformers import AutoModelForSequenceClassification, AutoTokenizer
51
 
52
  repo = "HashNuke/barge-in-detector"
53
  tokenizer = AutoTokenizer.from_pretrained(repo)
54
  model = AutoModelForSequenceClassification.from_pretrained(repo).cpu().eval()
55
+
56
+ inputs = tokenizer("please stop for a moment", return_tensors="pt", truncation=True, max_length=64)
57
  with torch.inference_mode():
58
+ logits = model(**inputs).logits
59
+ p_barge_in = logits.softmax(-1)[0, model.config.label2id["barge_in"]].item()
60
+
61
+ print({"p_barge_in": p_barge_in, "label": "barge_in" if p_barge_in >= 0.5 else "backchannel"})
62
  ```
63
 
64
+ Load once and reuse the model/tokenizer in a server. For a ready-to-run CLI and
65
+ training workflow, follow the [GitHub quick start](https://github.com/vxpipe/barge-in-detector#try-the-model-on-your-cpu).
66
+ Pin `revision=MODEL_COMMIT` when deploying a specific release.
67
+
68
+ ## Input and output
69
+
70
+ | Contract | Value |
71
+ | --- | --- |
72
+ | Input | Caller text spoken over an ongoing agent turn |
73
+ | Cleaning | Lowercase, punctuation removed; native scripts may mix with English |
74
+ | Budget | First 64 tokenizer tokens |
75
+ | Labels | `0 = backchannel`, `1 = barge_in` |
76
+ | Output | `p_barge_in` in [0, 1]; current probabilities are uncalibrated |
77
+ | Context | No audio, preceding agent line, or language hint is required |
78
 
79
+ The runtime handles speech after the agent finishes and owns decision thresholds.
80
+ See the [source cleaner contract](https://github.com/vxpipe/barge-in-detector/blob/main/docs/transcript-format.md).
81
+ The 0.5 threshold in the example is diagnostic.
82
 
83
+ ## Evaluation
 
 
 
 
84
 
85
  | Evaluation | Accuracy | Missed interruptions | False interruptions |
86
  | --- | --- | --- | --- |
 
89
  | Reviewed English challenge (98) | 73.5% | 54.2% | 0% |
90
  | Challenge unseen in any synthetic split (93) | 72.0% | 54.2% | 0% |
91
 
92
+ The short-interruption slice is 5/30 correct. The challenge was authored before
93
+ inference, then owner-reviewed and reconciled with authorization. Its final
94
+ binary labels match the frozen expectations. It includes repeated prefixes and
95
+ is now a development/regression set. Five challenge examples appear in synthetic
96
+ train/validation. These results are not a representative independent accuracy
97
+ estimate. Synthetic splits share criteria; template/semantic leakage remains
98
+ unaudited. Human-verified accuracy across all languages is not established.
99
 
100
+ ## CPU performance
101
 
102
+ Four virtual AMD EPYC Genoa cores, four threads, PyTorch 2.15.0+cpu eager fp32,
103
  batch one, 100 warm-ups and 1,000 measured calls per scenario:
104
 
105
  | Warm text through label | p50 | p95 | p99 |
 
107
  | Typical multilingual test input | 13.1 ms | 19.4 ms | 25.3 ms |
108
  | Full 64-token budget | 26.8 ms | 33.3 ms | 37.9 ms |
109
 
110
+ A fresh Python process took **4.22 seconds median** to return its first label
111
+ (range 3.85–4.81 seconds over ten runs). That includes imports and local loading;
112
+ OS file cache may be warm. Peak warm process RSS was approximately 1,164 MiB.
113
+ HTTP, network, queueing, and Vxpipe boundary time are excluded. Reference-CPU
114
+ gates are unestablished; ONNX/int8 is unmeasured.
115
+
116
+ ## Languages and provenance
117
+
118
+ The training/evaluation data spans English (US, India, Singapore), Hindi,
119
+ Kannada, Tamil, Telugu, Malayalam, Mandarin, Japanese, Indonesian, Filipino,
120
+ French, German, Spanish, and Italian. Results across these 14 languages are
121
+ synthetic; the reviewed challenge is English. See the
122
+ [dataset card](https://huggingface.co/datasets/HashNuke/barge-in-detector-data) for verification limits.
123
+
124
+ The checkpoint fine-tunes [intfloat/multilingual-e5-small](https://huggingface.co/intfloat/multilingual-e5-small)
125
+ with a two-label head. First run: 2,640 train / 333 validation / 331 test examples;
126
+ five epochs, batch 32, weighted cross-entropy, AdamW, learning rate 3e-5, 10% warm-up,
127
+ seed 20261115, bf16 on one RTX 5090. Epoch 5 was selected using validation macro F1
128
+ (0.99324). Released weights are fp32. See LICENSE and NOTICE for MIT attribution.
129
+
130
+ [release.json](release.json) pins the code and dataset revisions, artifact hashes,
131
+ original training-source manifest, and full training metrics. Reports and
132
+ labnotes live in the linked GitHub source. This repository contains no training
133
+ data or development scripts.
134
+
135
+ ## One project, three repositories
136
+
137
+ | What you want to do | Start here |
138
+ | --- | --- |
139
+ | Generate data, train, evaluate, benchmark, or contribute | [GitHub source](https://github.com/vxpipe/barge-in-detector) |
140
+ | Download a released model and tokenizer | [Hugging Face model](https://huggingface.co/HashNuke/barge-in-detector) |
141
+ | Load a versioned dataset | [Hugging Face dataset](https://huggingface.co/datasets/HashNuke/barge-in-detector-data) |
142
+
143
+ GitHub owns all code, tests, docs, labnotes, and milestones, plus a dataset copy.
144
+ Hugging Face hosts model and dataset releases. Release metadata links artifacts
145
+ to exact source and dataset revisions. The project is available under the
146
+ [MIT licence](https://github.com/vxpipe/barge-in-detector/blob/main/LICENSE).