Model card: add acceptance (nghiệm thu) LLM/quality/task KPIs, training set size & hardware

a51af89 verified about 2 months ago

4.08 kB

---
license: apache-2.0
base_model: Qwen/Qwen3-Reranker-0.6B
base_model_relation: finetune
library_name: transformers
pipeline_tag: text-ranking
language:
  - en
  - vi
pretty_name: sec-rerank (fine-tuned Qwen3-Reranker)
tags:
  - reranker
  - cross-encoder
  - text-ranking
  - listwise-reranking
  - generative-reranker
  - cve
  - cybersecurity
  - qdrant
  - secAI
datasets:
  - DuyTa/Cyber_F1_v2
  - DuyTa/cve-kgrag-db
---
# sec-rerank

**This is a fine-tuned version of [Qwen/Qwen3-Reranker-0.6B](https://huggingface.co/Qwen/Qwen3-Reranker-0.6B)** — a **listwise generative reranker** (second-stage ranker), not an embedding model.

It was trained on grouped query–document examples whose **hard negatives were mined from a local Qdrant** collection (`cve_kb`) built from CVE investigation trajectories. It does not emit dense vectors; it reorders first-stage candidates (e.g. from [`DuyTa/sec-embedding`](https://huggingface.co/DuyTa/sec-embedding)). It is the reranker component of the **secAI** stack, paired with [`DuyTa/Cyber-F1-AWQ`](https://huggingface.co/DuyTa/Cyber-F1-AWQ).

## Training

From `notebooks/Qwen3_Reranker_Colab.ipynb` (ms-swift):

| | |
|---|---|
| Base | `Qwen/Qwen3-Reranker-0.6B` |
| Task | `generative_reranker` (causal-LM reranker / cross-encoder scoring) |
| Loss | **listwise reranking** (`--loss_type listwise_reranker`) |
| Tuner | full-parameter SFT (`--tuner_type full`) |
| Engine | [ms-swift](https://github.com/modelscope/ms-swift) `swift sft` |
| Max length | 2048 |
| Learning rate | 6e-6 |

`(query, positive, negative)` rows are converted to SWIFT **grouped ranking** schema:

- `messages` — system instruction + user query
- `positive_messages` — gold CVE passage
- `negative_messages` — Qdrant-mined hard-negative CVE passage(s)

Train-time instruction:

> Given a Vietnamese cybersecurity search query, retrieve passages from the CVE knowledge base that directly answer it.

SWIFT fills the native Qwen3-Reranker `{Instruction}` slot from that system message.

### Hard-negative mining

Positives and negatives are real CVE core-chunk text from local Qdrant `cve_kb` (NVD/MITRE). The negative is a **near-miss CVE** from the same collection — typically a different CWE (`hard_negative_type: different_cwe`): high lexical overlap, wrong document. The listwise objective ranks the gold passage above those mined hard negatives.

Split: 33.6k train / 4.2k validation grouped examples.

### Training corpus

Built from **five years of authoritative cybersecurity sources**: NVD (173,473 CVEs), MITRE CWE (768 weakness types, mapped to ~92% of CVEs), CAPEC/ATT&CK (443/174 entries) and Exploit-DB (3,139 exploits, 2021–2026). Public datasets: [`DuyTa/Cyber_F1_v2`](https://huggingface.co/datasets/DuyTa/Cyber_F1_v2), [`DuyTa/cve-kgrag-db`](https://huggingface.co/datasets/DuyTa/cve-kgrag-db).

Training hardware: 2×A100 80GB.

## Acceptance (nghiệm thu) — reported KPIs

Measured on **NVIDIA A100 80GB** in the full production chatflow (Hybrid Search → Rerank → LLM), on a 1,000-sample security test set (40% CVE identification/classification, 40% remediation advice, 20% real-world scenario reasoning):

| Metric | Result | Target | Pass |
|---|---|---|---|
| Ranking quality — **Precision@1** | **98.23%** | > 90% | ✅ |
| Throughput | **4,862 docs/s** (concurrency 128) | ≥ 180 rerank/s | ✅ |

Raw per-sample logs (`reranking-precision-at-1.jsonl`) and evaluation code are delivered with the acceptance package.

## Inference

Score each `(query, document)` with the native Qwen3-Reranker generative yes/no head. Use only to **rerank** a short candidate list from dense / hybrid retrieval.

```python
# vLLM / OpenAI-compatible rerank endpoint
# POST /v1/rerank
{
  "model": "DuyTa/sec-rerank",
  "query": "CVE-2021-44228 JNDI lookup on log4j",
  "documents": ["...", "..."]
}
```

## Attribution & license

Released under **Apache-2.0**. Derived from [Qwen/Qwen3-Reranker-0.6B](https://huggingface.co/Qwen/Qwen3-Reranker-0.6B) (**Apache-2.0**); credit for the base reranker belongs to the Qwen team.

	---
	license: apache-2.0
	base_model: Qwen/Qwen3-Reranker-0.6B
	base_model_relation: finetune
	library_name: transformers
	pipeline_tag: text-ranking
	language:
	- en
	- vi
	pretty_name: sec-rerank (fine-tuned Qwen3-Reranker)
	tags:
	- reranker
	- cross-encoder
	- text-ranking
	- listwise-reranking
	- generative-reranker
	- cve
	- cybersecurity
	- qdrant
	- secAI
	datasets:
	- DuyTa/Cyber_F1_v2
	- DuyTa/cve-kgrag-db
	---

	# sec-rerank

	This is a fine-tuned version of [Qwen/Qwen3-Reranker-0.6B](https://huggingface.co/Qwen/Qwen3-Reranker-0.6B) — a listwise generative reranker (second-stage ranker), not an embedding model.

	It was trained on grouped query–document examples whose hard negatives were mined from a local Qdrant collection (`cve_kb`) built from CVE investigation trajectories. It does not emit dense vectors; it reorders first-stage candidates (e.g. from [`DuyTa/sec-embedding`](https://huggingface.co/DuyTa/sec-embedding)). It is the reranker component of the secAI stack, paired with [`DuyTa/Cyber-F1-AWQ`](https://huggingface.co/DuyTa/Cyber-F1-AWQ).

	## Training

	From `notebooks/Qwen3_Reranker_Colab.ipynb` (ms-swift):

	\| \| \|
	\|---\|---\|
	\| Base \| `Qwen/Qwen3-Reranker-0.6B` \|
	\| Task \| `generative_reranker` (causal-LM reranker / cross-encoder scoring) \|
	\| Loss \| listwise reranking (`--loss_type listwise_reranker`) \|
	\| Tuner \| full-parameter SFT (`--tuner_type full`) \|
	\| Engine \| [ms-swift](https://github.com/modelscope/ms-swift) `swift sft` \|
	\| Max length \| 2048 \|
	\| Learning rate \| 6e-6 \|

	`(query, positive, negative)` rows are converted to SWIFT grouped ranking schema:

	- `messages` — system instruction + user query
	- `positive_messages` — gold CVE passage
	- `negative_messages` — Qdrant-mined hard-negative CVE passage(s)

	Train-time instruction:

	> Given a Vietnamese cybersecurity search query, retrieve passages from the CVE knowledge base that directly answer it.

	SWIFT fills the native Qwen3-Reranker `{Instruction}` slot from that system message.

	### Hard-negative mining

	Positives and negatives are real CVE core-chunk text from local Qdrant `cve_kb` (NVD/MITRE). The negative is a near-miss CVE from the same collection — typically a different CWE (`hard_negative_type: different_cwe`): high lexical overlap, wrong document. The listwise objective ranks the gold passage above those mined hard negatives.

	Split: 33.6k train / 4.2k validation grouped examples.

	### Training corpus

	Built from five years of authoritative cybersecurity sources: NVD (173,473 CVEs), MITRE CWE (768 weakness types, mapped to ~92% of CVEs), CAPEC/ATT&CK (443/174 entries) and Exploit-DB (3,139 exploits, 2021–2026). Public datasets: [`DuyTa/Cyber_F1_v2`](https://huggingface.co/datasets/DuyTa/Cyber_F1_v2), [`DuyTa/cve-kgrag-db`](https://huggingface.co/datasets/DuyTa/cve-kgrag-db).

	Training hardware: 2×A100 80GB.

	## Acceptance (nghiệm thu) — reported KPIs

	Measured on NVIDIA A100 80GB in the full production chatflow (Hybrid Search → Rerank → LLM), on a 1,000-sample security test set (40% CVE identification/classification, 40% remediation advice, 20% real-world scenario reasoning):

	\| Metric \| Result \| Target \| Pass \|
	\|---\|---\|---\|---\|
	\| Ranking quality — Precision@1 \| 98.23% \| > 90% \| ✅ \|
	\| Throughput \| 4,862 docs/s (concurrency 128) \| ≥ 180 rerank/s \| ✅ \|

	Raw per-sample logs (`reranking-precision-at-1.jsonl`) and evaluation code are delivered with the acceptance package.

	## Inference

	Score each `(query, document)` with the native Qwen3-Reranker generative yes/no head. Use only to rerank a short candidate list from dense / hybrid retrieval.

	```python
	# vLLM / OpenAI-compatible rerank endpoint
	# POST /v1/rerank
	{
	"model": "DuyTa/sec-rerank",
	"query": "CVE-2021-44228 JNDI lookup on log4j",
	"documents": ["...", "..."]
	}
	```

	## Attribution & license

	Released under Apache-2.0. Derived from [Qwen/Qwen3-Reranker-0.6B](https://huggingface.co/Qwen/Qwen3-Reranker-0.6B) (Apache-2.0); credit for the base reranker belongs to the Qwen team.