Upload README.md with huggingface_hub
Browse files
README.md
ADDED
|
@@ -0,0 +1,32 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
library_name: pytorch
|
| 4 |
+
tags:
|
| 5 |
+
- sentence-transformers
|
| 6 |
+
- feature-extraction
|
| 7 |
+
- sentence-similarity
|
| 8 |
+
- mteb
|
| 9 |
+
- hcae
|
| 10 |
+
---
|
| 11 |
+
|
| 12 |
+
# HCAE-21M (Hybrid Convolutional-Attention Encoder)
|
| 13 |
+
**HCAE-21M** is a mid-scale (21 Million parameters) text embedding model combining Depthwise Separable Convolutions and Self-Attention layers. It achieves high performance on Semantic Textual Similarity and Retrieval tasks while remaining extremely memory-efficient.
|
| 14 |
+
|
| 15 |
+
## Architecture Description
|
| 16 |
+
- **Size:** ~21M parameters (d_model=384)
|
| 17 |
+
- **Lower Layers:** 5 layers of Depthwise Separable Conv1d + FFN.
|
| 18 |
+
- **Upper Layers:** 3 layers of Multihead Self-Attention.
|
| 19 |
+
- **Pooling Strategy:** Global Mean Pooling.
|
| 20 |
+
|
| 21 |
+
## Benchmark Comparison (MTEB)
|
| 22 |
+
|
| 23 |
+
This table delineates the performance disparities between architectural iterations:
|
| 24 |
+
|
| 25 |
+
| Model Revision | STSBenchmark (Spearman) | SciFact (Recall@10) | Description |
|
| 26 |
+
|---|---|---|---|
|
| 27 |
+
| **HCAE-21M-Base** | `0.507` | `0.324` | Baseline configuration trained extensively on the MS MARCO dataset. |
|
| 28 |
+
| **HCAE-21M-Instruct** | `0.591` | `0.393` | Multi-stage tuning incorporating ArXiv, STS-B, and SQuAD instruction tuning paradigms. |
|
| 29 |
+
|
| 30 |
+
## Utilization Guidelines (Instruction Format)
|
| 31 |
+
For optimal retrieval performance, prepend the instruction mapping to the query text:
|
| 32 |
+
`Instruction: Retrieve the exact document that answers the following question. Query: [Your Query]`
|