Laya fine-tuned for GlenDesk: Accuracy 0.357 vs Jev 0.727 / Ceiling 0.735

02d85a9 verified 16 days ago

2.32 kB

license: apache-2.0
library_name: transformers
tags:
  - laya
  - system-one
  - calibrated-decisions
  - rlcd
  - structured-decisions
  - typed-decisions
  - benchmark
  - glendesk
  - call-analysis
metrics:
  - accuracy
  - brier_score
model-index:
  - name: laya-glendesk-issues
    results:
      - task:
          type: text-classification
          name: System One Decision Benchmark
        dataset:
          type: LocalLLaMA/typed-decisions
          name: Typed Decisions
        metrics:
          - type: accuracy
            value: 0.357
          - type: brier_score
            value: 0.28

Laya (Fine-Tuned for GlenDesk Call Analysis)

This is Laya fine-tuned on GlenDesk's call analysis clusters (11 issue types, 711 examples, 4,977 training sequences) derived from the LocalLLaMA/typed-decisions benchmark format.

On the GlenDesk test set (400 cases), it achieves 0.357 Accuracy, trailing TypeSafe Jev 1.13.0 (0.727) and the benchmark's Teacher Self-Agreement ceiling (0.735).

Head-to-Head Benchmark Results

Model	Kind	Accuracy	Soft Acc	Brier Score	ECE	Score MAE	Within 1 Level	Latency (p50)	Cost/Case
Laya (Ours)	fine-tuned	0.357	0.362	0.280	0.057	0.683	0.756	130.6 ms	$0.00 (Self-Hosted)
TypeSafe Jev 1.13.0	general	0.727	0.580	0.148	0.144	0.391	0.952	710 ms	$0.0004 (API)
ModernBERT-base (149M)	specialist	0.646	0.542	0.119	0.179	0.444	0.931	349 ms	$0.00
Teacher Self-Agreement	ceiling	0.735	-	-	-	-	-	-	-

Intended Use

Classify GlenDesk call transcripts for issue clusters:

hallucination_beauty_phrase, non_english_char_leak, midcall_response_glitch
status_asked_false_positive, banned_filler_gap_definitely_help
customer_lookup_no_match_on_number, abrupt_hangup_pattern
summary_fabrication_vehicle_details, repeated_transfer_reason_loop
digit_transposition_caller_id, transfer_failure_retry

Installation & Quickstart

pip install laya

import laya
agent = laya.load("fcelabs/laya-glendesk-issues")
result = agent.predict(transcript, questions)
print(result["answers"])

License

Apache 2.0. Fine-tuned by GlenDesk/FCE Labs.