cbx-bfs-call-opening-checker-17M

Checks the opening of a customer-service call from a speaker-labelled transcript and reports, with evidence, whether the agent identified themselves, whether the institution was named, whether the customer was told the call is recorded or monitored, why (quality, training, security, other), and whether the transcript covers the start of the call. English; banking and finance calls. Fine-tuned from Ettin encoder 17m (about 17M parameters).

Licence: Apache-2.0. Read Intended use and disclaimer before using it on real calls: the model is not a compliance decision and not for monitoring or judging individual agents.

What it predicts

Output	Values	Meaning
agent_identification	present / absent / unclear	the agent gave their name or agent / employee number ("this is customer service" is not identification)
institution_identification	present / absent / unclear	the institution was named, also when misspelled by speech recognition or inside a product name; "the bank" alone is unclear
recording_notice	definite_recording / possible_recording / monitoring_only / none / unclear	"is being recorded" / "may be recorded" / "may be monitored" (monitoring never implies recording) / no notice or a negated one / contradicting or garbled; the strongest notice wins
recording_purpose	list of quality, training, security, other; or ["not_stated"] or ["unclear"]; [] without a notice	why the call is recorded or monitored ("secure line" is never a purpose)
opening_coverage	complete / incomplete / unknown	whether the transcript includes the start of the call (see below)
spans	type, start, end, text	evidence in the input: AGENT, INST, NOTICE, PURPOSE, NEGATED ("this call is not recorded"), DISTRACTOR (sounds like a notice but is not: "let me record your address", a customer asking "are you recording?")

The opening only. Labels cover the call from its start up to the agent turn that answers the customer's first explicit request. A name or notice given later does not count. Only agent and automated (IVR) turns count; a customer naming themselves or the bank never does.

Opening coverage. A missing name or notice is only a failure if the start of the call is in the transcript. complete: the metadata says the transcript starts at the call start. incomplete: the metadata says it does not, or the text plainly begins mid-conversation. unknown: no metadata and the text could be the start. With incomplete coverage, a field that would be absent / none is unclear instead: not observed is not confirmed absence. Pass the call-start flag when your system knows it; judging coverage from the text alone was learned and tested on constructed cut transcripts only.

Not predicted: whether a name or institution is correct, tone, anything after the opening, audio.

Input

The model reads one string: a metadata line, then one line per turn (Agent:, Customer: or IVR:), from the first 12 turns of the call.

metadata: starts at call start: yes; automated notice: not given
Agent: Good afternoon, thank you for calling Northbrook Savings Bank. My name is Daniel, and just so you know, this call may be recorded for quality and training purposes. How can I help you today?
Customer: Hi Daniel. I'd like to report a lost debit card.
Agent: I'm sorry to hear that. I can help you block the card right away. Can I have your full name, please?
Customer: It's Maria Lopez.

• starts at call start: yes, no or not given.
• automated notice: the text of an announcement played before the agent (it counts for institution and notice), or not given.
• Length: the model reads at most 512 tokens (about 2,000-3,000 characters); anything after is cut off. Tested inputs were 12-13 turns, median about 270 tokens, longest about 700 (2.2% of test inputs went past 512). Openings normally end by turn 3-5.

Output for this example (invented call, actual model output):

{"agent_identification": "present", "institution_identification": "present",
 "recording_notice": "possible_recording", "recording_purpose": ["quality", "training"],
 "opening_coverage": "complete",
 "spans": [{"type": "INST", "start": 110, "end": 133, "text": "Northbrook Savings Bank"},
           {"type": "AGENT", "start": 135, "end": 152, "text": "My name is Daniel"},
           {"type": "NOTICE", "start": 158, "end": 201, "text": "just so you know, this call may be recorded"},
           {"type": "PURPOSE", "start": 202, "end": 235, "text": "for quality and training purposes"}]}

Quick start

pip install -r requirements.txt
python example.py

from cbx_call import CallOpeningChecker

checker = CallOpeningChecker(".")                    # folder with the model files
report = checker.check({
    "starts_at_call_start": True,                    # True / False / None (not known)
    "automated_notice": None,                        # text of an announcement played before the agent, or None
    "turns": [("Agent", "Good afternoon, thank you for calling Northbrook Savings Bank. My name is Daniel, ..."),
              ("Customer", "Hi Daniel. I'd like to report a lost debit card."),
              ("Agent", "I'm sorry to hear that. I can help you block the card right away. ...")],
})
report["recording_notice"], report["spans"]
checker.decision(report, policy="A")                 # example policy: "pass" / "fail" / "review"

check also takes a list of calls (batched) or a ready input string. Turns are (speaker, text) pairs or dicts with speaker and text; speakers are Agent, Customer or IVR. The loader keeps the first 12 turns (and no more than about 3,000 characters), builds the input string shown above, and returns it in report["input"] (span offsets refer to it) with report["truncated"] set when the input passed 512 tokens.

Using it for a policy check

The model reports what was said; a policy decides what is acceptable. One example rule: review if coverage is not complete or any field is unclear; fail if the agent or institution is absent, or the notice is none or monitoring_only; otherwise pass when the notice is accepted. Policy A accepts definite_recording or possible_recording; policy B accepts only definite_recording (possible_recording goes to review). The tables show how often the model's decision equals the decision from the gold labels.

Evaluation

Held-out test sets; rule baseline (keyword and pattern rules) in brackets. The three test sets are the test split of the public benchmark CallOpeningDetX v0.1 (DOI 10.57967/hf/11637), released with this model. Its dev split was never trained on; it was used to choose the checkpoint and to guide the data rounds. The benchmark card has the scorer and more baselines, including zero-shot LLMs.

	Natural calls	Insurance calls (new domain)	Challenge set
rows	600	200	950
all fields exact	0.988 (0.977)	0.970 (0.665)	0.862 (0.495)
agent: accuracy / macro F1	0.997 / 0.990	0.980 / 0.654	0.981 / 0.983
institution: accuracy / macro F1	0.993 / 0.856	0.985 / 0.606	0.994 / 0.990
recording notice: accuracy / macro F1	1.000 / 1.000	1.000 / 1.000	0.963 / 0.961
recording purpose: micro F1	0.875 (7 notices)	n/a (no notices)	0.854
opening coverage: accuracy	1.000	1.000	1.000
evidence spans: F1 overlap / exact	0.991 / 0.951	0.962 / 0.840	0.959 / 0.862
policy A / B agreement	1.000 / 1.000	0.990 / 0.990	0.993 / 0.983

• Natural calls: real banking and finance call openings from DSTC11 Track 2, labelled by two LLMs blind with disagreements adjudicated. Very uniform: nearly every call names the institution and only 7 contain a recording notice.
• Insurance calls: real insurance call openings from the same source, a domain not used in training. No call contains a recording notice.
• Challenge set: real calls whose opening was rewritten with wording families and invented agent and institution names that never appear in training (including wording written by a different LLM), plus negations, distractors, automated notices and cut transcripts. Every row was checked by a blind LLM label, with disagreements adjudicated.
• Macro F1 is low on the one-sided natural and insurance sets because rare classes have one to eight rows (insurance has one unclear agent call; missing it costs a third of agent macro F1). Read accuracy there.
• Evidence span F1 "overlap" counts a predicted span that touches a gold span of the same type; "exact" needs the same start and end.

Speed (one call of 270 tokens, model plus decoding):

	one call	batch of 32
GPU (NVIDIA TITAN RTX)	4.6 ms	23.3 ms
CPU (8 threads)	8.7 ms	259 ms

Training data

• Natural calls: openings of DSTC11 Track 2 banking and finance calls (CDLA-Permissive-2.0). Labels and evidence quotes from two blind LLM labellers (gpt-5.4-mini and claude-haiku-5-5) under a written annotation guide; Claude Opus adjudicated every disagreement and audited 10% of agreed rows (one label changed in 312 audited rows).
• Constructed calls: natural calls with the opening rewritten from a phrase bank and invented names (agent and institution names never shared with the test sets), and new wording written by Mistral-7B-Instruct and Phi-3.5-mini-instruct; labels and evidence by construction, inconsistent rows filtered.
• Generated calls: whole openings written by Phi-3.5-mini-instruct, Mistral-7B-Instruct and OLMo-2-7B from a spec, kept only when rule checks and a blind claude-haiku-5-5 label matched the spec (labels only, no evidence spans).
• No test call, test wording family or test name is in the training data.

Limitations

• English, banking and finance only. Insurance was checked as a new domain; other domains and languages are untested.
• Not tested on your calls. The natural training calls come from one fictional bank and are uniform. Real automated announcements, wrong speaker labels and other speech-recognition systems are not measured. Check it on a sample of your own transcripts first.
• Recording notices and purposes are measured mainly on constructed calls. Purpose is the weakest output (micro F1 0.854 on the challenge set); some phrasings are ambiguous ("the quality team may listen in").
• Opening coverage without metadata was learned and tested on constructed cut transcripts only.
• Long inputs are cut at 512 tokens; openings that start after the first 12 turns (long hold or menu) are untested.
• Labels come from LLMs, checked against each other and by an adjudicator, not by humans row by row.

Intended use and disclaimer

Intended use: research; finding calls for a person to review; measuring, across many calls, how often openings include identification and recording notices; testing and comparing call-quality tools.

Not intended for, and not supported:

• Monitoring or judging individual employees. Do not use the model, its outputs or any score built from them as the basis for performance, pay, disciplinary, dismissal or other employment decisions about the agents whose calls are checked. If you use it in any way that concerns your staff, you are responsible for informing them and for complying with employment, workplace-monitoring and data-protection law.
• Compliance decisions. The model reports what was said in a transcript; it does not decide whether any legal, regulatory, contractual or internal requirement (identification, recording disclosure, consent or any other) was met. Its outputs are not legal, regulatory or compliance advice and may be wrong. Any compliance finding needs review by a qualified person against the original recording and the rules that apply to you.
• Decisions about customers, such as credit, fraud, account or service decisions.
• Processing calls without a lawful basis. Call recordings and transcripts contain personal data and may be subject to call-recording consent, privacy and banking-secrecy law. You are responsible for having a lawful basis, any consent or notice required, and appropriate safeguards. The model runs on your systems; Cobuild Tech Private Limited (CobuildX) does not receive your data.

No responsibility for use. The model is provided "as is", without warranties or conditions of any kind, and the authors and CobuildX are not liable for any use of it (Apache License 2.0, sections 7 and 8). You alone are responsible for how you use the model and its outputs, for any decision taken with them, and for complying with the law that applies to you. Scores on this page are measured on specific test sets and are not a guarantee of performance on your calls.

Files

File
model.safetensors, config.json, tokenizer.json, tokenizer_config.json	the model: Ettin encoder with four field heads, a purpose head and an evidence-span tagger
call_config.json	label sets, span tags, input limits, base model, scores
cbx_call.py, example.py, requirements.txt	loader and example
LICENSE, NOTICE	Apache-2.0 licence, third-party notices
checksums.sha256	SHA-256 of every file

Contact

Questions: hello@cobuildx.ai. Release v0.1 (2026-11-14).