hd-laya / README.md
geraldaton20's picture
Update model card with CLINC150 benchmark + index recall analysis
01ecff9 verified
|
Raw History Blame Contribute Delete
3.92 kB
metadata
language: en
license: apache-2.0
tags:
  - decision
  - system-1
  - hdc
  - vsa
  - neuro-symbolic
  - laya
base_model: convaiinnovations/laya

hd-laya

hd-laya removes laya's ~20-option choice ceiling with a hyperdimensional (HDC/VSA) option index, keeping laya's System-1 engine unmodified.

This repo carries the v4 calibrated checkpoint (hd_laya_trained_calibrated.pt, 106 MB) which achieves a measured +3.1 pt accuracy gain over the base model on K=20 choice tasks and 80% accuracy at K=500/1000 (vs 1.6% base).

Comparison: hd-laya vs base laya

Comparison chart

High-K stress suite (synthetic, 256 items per K)

K base acc hd-laya acc improvement
20 47.3% 46.5% −0.8 (flat)
77 14.5% 26.6% +12.1
150 5.1% 44.5% +39.4
500 1.6% 80.5% +78.9
1000 0.78% 79.7% +78.9

The HDC index narrows K options to 20 before the forward pass. At K=500/1000, the trained model matches keys in topically-similar sets with 80% accuracy, vs 1.6% for base laya. The z-gate is disabled (z_bar=0) for these numbers.

Real-world benchmarks

dataset intents base laya hd-laya delta index recall@20
BANKING77 77 4.0% 24.0% +20.0% ~1.0
CLINC150 151 20.0% 3.3% −16.7% 0.75

BANKING77 (50 items): base laya is at chance (4% for 77-way). hd-laya narrows to 20 via HDC and achieves 24% — a 6× improvement.

CLINC150 (30 items): base laya gets 20% (chance is 0.67% for 151-way, so it's actually doing well). hd-laya gets 3.3% because the HDC index only achieves 75% recall@20 on CLINC150's more semantically diverse intents — when the correct intent isn't in the narrowed set, no amount of model training helps.

Training

base model convaiinnovations/laya (encoder partially unfrozen: last 2 layers)
hardware Kaggle T4 x2
task synthetic high-K choice, K=20, 2048 items
epochs 12 (6144 steps)
budgets max_len=512, head_max_len=192
trainable params 26.5M head + 24.5M encoder (2 layers)
held-out delta +3.1 pts (base 52.3% → trained 55.5%)
calibration temp=1.427 for choice bucket (fit on 400 items)

Usage

from huggingface_hub import hf_hub_download
from laya.agent import Agent
from hd_laya.agent import HDLayaAgent
from hd_laya.laya_agent import TrainedLayaAgent

ckpt = hf_hub_download("geraldaton20/hd-laya", "hd_laya_trained_calibrated.pt")
laya = Agent("convaiinnovations/laya", device="cuda")
trained = TrainedLayaAgent.from_checkpoint(ckpt, agent=laya, max_len=512, head_max_len=192)
agent = HDLayaAgent(trained, max_options=20, z_bar=0.0)

result = agent.predict("customer wants to refund order", {
    "q": {"type": "choice",
          "instructions": "Choose the matching option.",
          "criteria": {"opt_0": "refund order (variant 0)",
                       "opt_1": "reset password (variant 1)"}},
})
print(result["answers"]["q"]["choice"])

Limitations

  • z-gate needs per-model calibration. The trained model's confidence distribution differs from base; fixed thresholds over-refuse. Using z_bar=0 (no refusals) gives the best accuracy.
  • K=20 performance is flat vs base. Training improved high-K key matching but not K=20 diverse-option accuracy.
  • HDC index recall varies by dataset. 100% on BANKING77, 75% on CLINC150. The index relies on lexical similarity; semantically diverse datasets are harder.
  • Synthetic task. Topics are 12 templates; real-world BANKING77 shows +20% improvement over base (4% → 24%).
  • ECE is high (0.80 at K=500). The model is overconfident on wrong answers.

License

Apache-2.0 (derived from laya; attribution to NandhaKishorM/laya).

Links