hd-laya / README.md
geraldaton20's picture
Update model card with CLINC150 benchmark + index recall analysis
01ecff9 verified
|
Raw History Blame Contribute Delete
3.92 kB
---
language: en
license: apache-2.0
tags:
- decision
- system-1
- hdc
- vsa
- neuro-symbolic
- laya
base_model: convaiinnovations/laya
---
# hd-laya
**hd-laya** removes laya's ~20-option choice ceiling with a hyperdimensional
(HDC/VSA) option index, keeping laya's System-1 engine unmodified.
This repo carries the **v4 calibrated checkpoint** (`hd_laya_trained_calibrated.pt`,
106 MB) which achieves a measured +3.1 pt accuracy gain over the base model
on K=20 choice tasks and 80% accuracy at K=500/1000 (vs 1.6% base).
## Comparison: hd-laya vs base laya
![Comparison chart](comparison_chart.png)
### High-K stress suite (synthetic, 256 items per K)
| K | base acc | hd-laya acc | improvement |
|---|---|---|---|
| 20 | 47.3% | 46.5% | −0.8 (flat) |
| 77 | 14.5% | 26.6% | +12.1 |
| 150 | 5.1% | 44.5% | +39.4 |
| 500 | 1.6% | 80.5% | +78.9 |
| 1000 | 0.78% | 79.7% | +78.9 |
The HDC index narrows K options to 20 before the forward pass. At K=500/1000,
the trained model matches keys in topically-similar sets with 80% accuracy,
vs 1.6% for base laya. The z-gate is disabled (z_bar=0) for these numbers.
### Real-world benchmarks
| dataset | intents | base laya | hd-laya | delta | index recall@20 |
|---|---|---|---|---|---|
| BANKING77 | 77 | 4.0% | 24.0% | +20.0% | ~1.0 |
| CLINC150 | 151 | 20.0% | 3.3% | −16.7% | 0.75 |
**BANKING77** (50 items): base laya is at chance (4% for 77-way). hd-laya
narrows to 20 via HDC and achieves 24% — a 6× improvement.
**CLINC150** (30 items): base laya gets 20% (chance is 0.67% for 151-way,
so it's actually doing well). hd-laya gets 3.3% because the HDC index only
achieves 75% recall@20 on CLINC150's more semantically diverse intents — when
the correct intent isn't in the narrowed set, no amount of model training helps.
## Training
| | |
|---|---|
| base model | `convaiinnovations/laya` (encoder partially unfrozen: last 2 layers) |
| hardware | Kaggle T4 x2 |
| task | synthetic high-K choice, K=20, 2048 items |
| epochs | 12 (6144 steps) |
| budgets | `max_len=512`, `head_max_len=192` |
| trainable params | 26.5M head + 24.5M encoder (2 layers) |
| held-out delta | +3.1 pts (base 52.3% → trained 55.5%) |
| calibration | temp=1.427 for choice bucket (fit on 400 items) |
## Usage
```python
from huggingface_hub import hf_hub_download
from laya.agent import Agent
from hd_laya.agent import HDLayaAgent
from hd_laya.laya_agent import TrainedLayaAgent
ckpt = hf_hub_download("geraldaton20/hd-laya", "hd_laya_trained_calibrated.pt")
laya = Agent("convaiinnovations/laya", device="cuda")
trained = TrainedLayaAgent.from_checkpoint(ckpt, agent=laya, max_len=512, head_max_len=192)
agent = HDLayaAgent(trained, max_options=20, z_bar=0.0)
result = agent.predict("customer wants to refund order", {
"q": {"type": "choice",
"instructions": "Choose the matching option.",
"criteria": {"opt_0": "refund order (variant 0)",
"opt_1": "reset password (variant 1)"}},
})
print(result["answers"]["q"]["choice"])
```
## Limitations
- **z-gate needs per-model calibration.** The trained model's confidence
distribution differs from base; fixed thresholds over-refuse. Using z_bar=0
(no refusals) gives the best accuracy.
- **K=20 performance is flat vs base.** Training improved high-K key matching
but not K=20 diverse-option accuracy.
- **HDC index recall varies by dataset.** 100% on BANKING77, 75% on CLINC150.
The index relies on lexical similarity; semantically diverse datasets are harder.
- **Synthetic task.** Topics are 12 templates; real-world BANKING77 shows
+20% improvement over base (4% → 24%).
- **ECE is high** (0.80 at K=500). The model is overconfident on wrong answers.
## License
Apache-2.0 (derived from laya; attribution to NandhaKishorM/laya).
## Links
- GitHub: https://github.com/sensesVoid/hd-laya
- Upstream: https://github.com/NandhaKishorM/laya