ThakiCloud/Thaki-SecEmbed-8B-Preview

Thaki-SecEmbed-8B (Preview)

Security-domain specialized experimental preview.

🇰🇷 한국어 카드: MODEL_CARD.ko.md

Preview release. This model is evaluated on one task only: retrieving MITRE ATT&CK® technique documentation from analyst-style queries in English and Korean. Do not read the results as general security or general-domain retrieval performance.

Headline. Security-domain specialized experimental preview. No statistically clear difference from the stock Qwen3-Embedding-8B baseline under the preregistered held-out evaluation. A subsequent independent-teacher Blind-B evaluation showed a clear improvement over the stock Qwen3-Embedding-8B baseline. On general retrieval: no catastrophic regression under the preregistered 10% threshold; significant FiQA degradation was observed.

Model summary

Thaki-SecEmbed-8B is a LoRA specialization of Qwen/Qwen3-Embedding-8B. It retrieves ATT&CK technique documentation for security-analyst queries. Queries can be questions, claims, keywords, evidence requests, analyst searches, or TTP hypotheses, in English or Korean.

Base model	Qwen/Qwen3-Embedding-8B (revision 1d8ad4ca9b3dd8059ad90a75d4983776a23d44af)
Adaptation	LoRA (r=16, α=32; targets q,k,v,o,gate,up,down)
Embedding dimension	4096 (primary). See Dimensions
Languages	English, Korean
Training data	Public data only (MITRE ATT&CK STIX). No internal or customer data
Release format	LoRA adapter (PEFT). This is the exact adapter that was evaluated; no merged weights are published for this Preview
Base artifact fingerprint	b7632bc848d604ba
Released adapter	seed 20261015 (the pre-designated public candidate), fingerprint d442705943c58283, adapter_model.safetensors sha256 e49c221ae93b931b93d4616aedd8808dc95e9ee4271d63f55e79b104656f5933

Intended use

• ATT&CK technique retrieval. Given a query, rank ATT&CK technique and sub-technique documents. Typical uses are candidate retrieval for TTP mapping, analyst search over ATT&CK, and a first-stage retriever in front of a reranker.
• Research on domain-specialized embeddings.

Out of scope

• General-domain retrieval as a primary use. On four public retrieval sets the adapter shows no catastrophic regression versus the base model, but it is not tuned for general retrieval, and finance retrieval (FiQA-2018) drops measurably. See General retrieval retention. For general-purpose retrieval, use the base model.
• Retrieval over other security corpora, such as CTI reports, logs/telemetry, detection rules, or vulnerability databases. These were not evaluated.
• Automated detection, attribution, or response decisions without human review.
• Languages other than English and Korean.

Usage

Load the base model explicitly at the pinned revision, as below. From v0.0.1 the adapter_config.json also names the base model and revision, so PEFT auto-loading works too; v0.0.0 keeps the evaluated file, whose base path is a local training path.

Only the query side receives an instruction. Documents are embedded as raw text. Pooling uses the last token (left padding) followed by L2 normalization, the same as the base model.

Instruction used in training and evaluation:

Given a security analyst query, retrieve documentation of the adversary technique or tactic it describes.

Query format:

Instruct: {instruction}
Query: {query}

import torch
import torch.nn.functional as F
from peft import PeftModel
from transformers import AutoModel, AutoTokenizer

INSTRUCTION = ("Given a security analyst query, retrieve documentation of the adversary "
               "technique or tactic it describes.")

BASE, REV = "Qwen/Qwen3-Embedding-8B", "1d8ad4ca9b3dd8059ad90a75d4983776a23d44af"
tok = AutoTokenizer.from_pretrained(BASE, revision=REV, padding_side="left")
base = AutoModel.from_pretrained(BASE, revision=REV, torch_dtype=torch.bfloat16)
model = PeftModel.from_pretrained(base, "ThakiCloud/Thaki-SecEmbed-8B-Preview").eval()

def embed(texts, max_len):
    enc = tok(texts, padding=True, truncation=True, max_length=max_len, return_tensors="pt")
    with torch.no_grad():
        h = model(**enc).last_hidden_state[:, -1]          # last token (left padding)
    return F.normalize(h.float(), p=2, dim=-1)

q = embed([f"Instruct: {INSTRUCTION}\nQuery: {x}" for x in ["Can malware download files from the C2?"]], 512)
d = embed(["<ATT&CK technique document text>"], 2048)     # documents: raw text, no instruction
scores = q @ d.T

Dimensions

The primary embedding dimension is 4096. This Preview does not train with a Matryoshka (MRL) objective. Truncated dimensions (1024 / 512 / 256) are a diagnostic only: they are not optimized, and this card makes no Matryoshka claim. Held-out nDCG@10 after truncation and L2 re-normalization (583 queries; Δ vs stock with paired-bootstrap 95% CI):

Dim	Stock Qwen3-Embedding-8B	Thaki-SecEmbed-8B	Δ [95% CI]
4096	0.670	0.686	+0.016 [-0.011, +0.043]
1024	0.662	0.678	+0.016 [-0.011, +0.042]
512	0.658	0.670	+0.012 [-0.015, +0.039]
256	0.613	0.626	+0.013 [-0.017, +0.040]

Training

Data (public only)

• Source: MITRE ATT&CK STIX, Enterprise ATT&CK v19.2 (enterprise-attack.json, from https://github.com/mitre/cti). Used under the ATT&CK Terms of Use.
• Retrieval corpus: every live ATT&CK technique and sub-technique, rendered into one document each by a fixed renderer. The training positives use the same renderer, so document formatting cannot reveal the answer.
• Queries: generated by a teacher LLM from grounded ATT&CK source spans. The generator was ThakiCloud/Qwen3.8-27B-NVFP4-GPTQ-txt (revision 615fbec2b9beb5a0dfb0858a7be036e354dff20a), ThakiCloud's NVFP4 quantization of Qwen/Qwen3.8-27B, served with z-lab/Qwen3.8-27B-DFlash2 as a speculative-decoding draft. The served files were checked byte-for-byte against those public repositories. There are six archetypes (analyst search, claim, evidence request, keyword, question, TTP hypothesis), in English and Korean. Generated queries passed deterministic gates before use: format, answer/ID leakage, and source grounding. Rejected queries were dropped, not regenerated.
• No internal, customer, or proprietary data is included in this release.

Hard-negative mining

• Candidate negatives came from the union of BM25 and the stock Qwen3-Embedding-8B rankings.
• False-negative removal is a fixed, deterministic rule. For each query it excludes two kinds of candidate:
  • other techniques linked to the same ATT&CK source anchor, since these are also correct answers;
  • the gold technique's parent and sibling sub-techniques.
• Nemotron-3-Embed-8B was not used for mining. It served only as an evaluation baseline, which keeps it an independent comparison.

Objective

• Contrastive InfoNCE: one positive plus hard negatives plus in-batch negatives, at τ = 0.05.
• Plus a listwise distillation term from Qwen/Qwen3-Reranker-8B (revision 77d193c791ed757ca307ee72715aa132723da912): teacher-softened listwise KL, student temperature fixed at τ=0.05.
  • The teacher temperature is applied only to the teacher's logits. There is no T² rescaling.
  • Teacher scores were computed once and cached. They were bit-exact reproducible under the fixed scoring configuration.
• KD weight λ_KD = 4, selected by a pre-registered 0.6B loss canary from {0, 1, 4} (3 seeds each, tuning-dev split disjoint from the held-out set). In the canary, contrastive-only training (λ_KD = 0) fell well below the untuned 0.6B model, and distillation recovered and exceeded it.
  • The canary used Qwen3-Embedding-0.6B and served only to eliminate λ options. It is not evidence of 8B performance.

Hyperparameters

LoRA	r=16, α=32
Learning rate	0.0001
Batch	8 queries × (1 positive + 7 hard negatives) + in-batch negatives
Steps	610 (1 epoch over the 4,886 canonical training pairs at batch 8; last step only, no checkpoint selection)
τ / teacher T	0.05 / 2.0
Max length (query / doc)	256 / 1024
MRL weight	0
Seeds	20261015, 20261016, 20261017; the released adapter is seed 20261015, fixed before any held-out result was seen

Evaluation

Evaluation set: a held-out set of generated queries from the same generator, built before training and never used for mining, hard-negative selection, model selection, or query generation.

• It covers 17 parent techniques, which is a narrow slice of ATT&CK.
• Its queries come from the same generator and distribution as the training queries.
• The results therefore measure in-distribution ATT&CK technique retrieval. They are not an out-of-distribution or real-analyst-traffic estimate.
• Size: 583 queries, materialized from 595 planned; slots that failed the generation gates were dropped, not replaced.

Metric: nDCG@10 (primary). Recall@1/5/10 and MRR@10 are also reported. Retrieval runs over the full ATT&CK technique corpus.

Baselines: BM25 (no instruction), stock Qwen3-Embedding-8B (same instruction), and stock Nemotron-3-Embed-8B.

Model	nDCG@10	Recall@10	MRR@10
BM25 (lexical baseline)	0.254	0.427	0.202
Qwen3-Embedding-8B (stock)	0.670	0.842	0.615
Nemotron-3-Embed-8B (stock)	0.653	0.849	0.592
Thaki-SecEmbed-8B (seed 20261015)	0.686	0.859	0.631

Pre-registered verdict vs the base model: no statistically clear difference. Δ nDCG@10 (Thaki-SecEmbed-8B − stock Qwen3-Embedding-8B) = +0.016 [-0.010, +0.043] (paired bootstrap over queries, 95%). No statistically clear difference from the stock baseline under the preregistered held-out evaluation. This does not establish equivalence, and we make no claim of improvement over the base model.

On this pre-registered ATT&CK-grounded held-out benchmark, Thaki-SecEmbed-8B outperformed Nemotron-3-Embed-8B (Δ nDCG@10 +0.033 [+0.009, +0.057]), while showing no statistically clear difference from its Qwen3-Embedding-8B base model. The held-out queries share the training generator and distribution, so this is not evidence of a general advantage over Nemotron. Nemotron was run with its own official input format (query:/passage: prefixes, mean pooling), checked against its model card example.

Per-slice Δ nDCG@10 vs stock (95% CI). No slice shows a significant regression, but the point estimates for analyst_search and evidence_request are negative:

Slice	n	Δ nDCG@10 [95% CI]
analyst_search	96	-0.035 [-0.104, +0.027]
claim	95	+0.025 [-0.039, +0.085]
evidence_request	98	-0.018 [-0.085, +0.046]
keyword	97	+0.049 [-0.020, +0.115]
question	100	+0.033 [-0.035, +0.098]
ttp_hypothesis	97	+0.042 [-0.028, +0.112]
en	290	+0.001 [-0.035, +0.036]
ko	293	+0.031 [-0.010, +0.073]

Post-release independent evaluation (Blind-B)

Blind-B is an independently constructed benchmark whose queries were written by teacher models different from the Qwen3.8-27B teacher used for this model's training pipeline. It holds 2,025 queries over 966 source anchors, and its answers are decided by ATT&CK STIX relations in code rather than by a model.

On Blind-B, Thaki-SecEmbed-8B-Preview reached an nDCG@10 of 0.694, against 0.550 for the stock Qwen3-Embedding-8B baseline. The paired difference is +0.1447 (95% CI +0.129 to +0.161, anchor-cluster bootstrap). The direction is the same in English and Korean and in both query strata.

Model	Blind-B nDCG@10
BM25	0.280
EmbeddingGemma 2 (768d)	0.443
Qwen3-Embedding-8B	0.550
Nemotron-3-Embed-8B	0.587
Thaki-SecEmbed-8B-Preview	0.694

This evaluation was run after the Preview release decision and was not binding on it. It does not retroactively change the preregistered B_NO_CLEAR_DIFFERENCE verdict on the 583-query held-out set above. The two readings are consistent: the held-out set shares its generator with the training data and did not separate the models, while the independent-teacher benchmark did.

Confidence intervals were computed against the stock Qwen3-Embedding-8B baseline only. We therefore make no claim of a statistically clear advantage over Nemotron-3-Embed-8B, whose point estimate is lower here. Blind-B's first one-shot evaluation is now complete, so it is treated as a known regression benchmark for later development rather than as an untouched blind set.

General retrieval retention

Measured against the stock Qwen3-Embedding-8B on four public retrieval sets at 4096 dimensions, each with its own query instruction, nDCG@10, paired bootstrap 95% CI. The pre-registered rule flags a set as a catastrophic regression only if the relative drop exceeds 10% and the CI upper bound of the difference is below 0. Sets are judged one by one; results are not averaged.

Set	Measures	Queries	Stock	Thaki-SecEmbed-8B	Δ nDCG@10 [95% CI]	Relative	Reading
SciFact	General retrieval retention (scientific claims)	300	0.784	0.797	+0.013 [+0.001, +0.026]	+1.7%	No catastrophic regression
NFCorpus	General retrieval retention (biomedical)	323	0.409	0.417	+0.008 [+0.001, +0.015]	+2.0%	No catastrophic regression
FiQA-2018	Finance retrieval retention	648	0.650	0.610	-0.040 [-0.052, -0.029]	-6.2%	Statistically clear drop, below the 10% catastrophic threshold
MIRACL (ko)	Korean retrieval retention	213	0.667	0.674	+0.006 [-0.008, +0.023]	+1.0%	No catastrophic regression

No catastrophic regression under the preregistered 10% threshold; significant FiQA degradation was observed. This is not evidence that general capability is preserved. Finance retrieval (FiQA-2018) drops by 0.040 nDCG@10 (-6.2% relative), and the drop is statistically clear. If you retrieve over financial Q&A, prefer the base model.

Training variability across three independently trained seeds

Same recipe and data, different seeds. The released seed was fixed before any held-out result was seen; the other two are reported only and were never used for selection.

Seed	Held-out nDCG@10	Δ vs stock [95% CI]
20261015 (released)	0.686	+0.016 [-0.011, +0.043]
20261016	0.705	+0.035 [+0.008, +0.060]
20261017	0.676	+0.006 [-0.019, +0.031]

Mean 0.689, range 0.676–0.705 (stock 0.670). The seed-to-seed spread is about as large as the released adapter's difference from the base model, which is one reason this Preview makes no improvement claim.

Additional post-release baselines (not part of the preregistered release decision)

Measured after the release on the same held-out set (583 queries), under a separate preregistration (POST_RELEASE_BASELINE_V1). These numbers do not change the verdict above. The held-out set is now regression-only.

Model	nDCG@10	Δ vs stock Qwen3-Embedding-8B (0.670)	Δ vs Thaki-SecEmbed-8B (0.686)
EmbeddingGemma 2 (zero-shot, 768)	0.524	-0.146 [-0.174, -0.116] (lower)	-0.162 [-0.193, -0.131] (lower)
EmbeddingGemma 2 (MRL 512)	0.493	-0.177 [-0.206, -0.147] (lower)	-0.193 [-0.223, -0.163] (lower)
EmbeddingGemma 2 (MRL 256)	0.468	-0.203 [-0.233, -0.172] (lower)	-0.219 [-0.249, -0.187] (lower)
EmbeddingGemma 2 (MRL 128)	0.367	-0.303 [-0.335, -0.271] (lower)	-0.320 [-0.353, -0.286] (lower)
Qwen3-Embedding-0.6B (stock, 1024)	0.432	-0.238 [-0.268, -0.209] (lower)	-0.254 [-0.283, -0.225] (lower)
0.6B canary adapter, λ_KD=4, seed 20261015 (1024)	0.535	-0.136 [-0.161, -0.110] (lower)	-0.152 [-0.174, -0.129] (lower)

EmbeddingGemma 2 was run with its official sentence-transformers prompts (SearchQuery / Document). The 0.6B rows use the λ_KD=4 adapters from the loss canary (3,845 training pairs, not the 4,886 used for this Preview); they are not a released model. Δ of the 0.6B canary adapter vs stock 0.6B across its three seeds: +0.102 [+0.076, +0.129], +0.123 [+0.098, +0.149], +0.103 [+0.077, +0.129].

Limitations

• Narrow evaluation. The held-out set covers 17 parent techniques and shares its generator with the training data, so reported gains may not transfer to real analyst queries or to other ATT&CK regions.
• Synthetic queries. All queries are LLM-generated, and stylistic artifacts of the generator may be learned.
• Finance retrieval regression. FiQA-2018 nDCG@10 is 0.040 lower than the base model (-6.2% relative, 95% CI excludes 0). Retention was measured on four sets only; other domains were not tested.
• Truncated dimensions are not trained with MRL. See Dimensions.
• ATT&CK version drift. The model was trained on Enterprise ATT&CK v19.2. Techniques added, renamed, or deprecated later are not reflected.
• Teacher dependence. Training used knowledge distillation from Qwen3-Reranker-8B, so the model inherits the teacher's preferences over hard negatives.

Reproducibility

• Base model artifacts were verified by sha256 against the pinned HF revision before every run.
• Training and evaluation inputs (queries, qrels, corpus, teacher scores) are pinned by sha256 in an append-only decision ledger. Pre-registration was committed before results were opened.
• Not released in this Preview: the held-out queries, qrels, and query-to-technique mapping. They are kept sealed so the benchmark stays independent for future model versions.
• Described here instead: the ATT&CK source version, how the benchmark was built (595 planned → 583 materialized deterministically; failed slots dropped, not replaced), the metric definitions, paired-bootstrap confidence intervals, aggregate results, and all eight slice results.

Upstream models and licenses

Role	Model	Revision	License
Base (this adapter applies to it)	Qwen/Qwen3-Embedding-8B	1d8ad4ca9b3dd8059ad90a75d4983776a23d44af	Apache-2.0
Distillation teacher	Qwen/Qwen3-Reranker-8B	77d193c791ed757ca307ee72715aa132723da912	Apache-2.0
Query generator (original)	Qwen/Qwen3.8-27B	1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0	Apache-2.0
Query generator (served quantization)	ThakiCloud/Qwen3.8-27B-NVFP4-GPTQ-txt	615fbec2b9beb5a0dfb0858a7be036e354dff20a	Apache-2.0
Speculative-decoding draft	z-lab/Qwen3.8-27B-DFlash2	50307d4c4cde6860d4eee73e2547cd786fe8e8a4	Apache-2.0
Evaluation baseline only	nvidia/Nemotron-3-Embed-8B-BF16	d1f2f25730bbd775b99b29185134bc86653bf2d1	not used in training

Attribution

MITRE ATT&CK® is a registered trademark of The MITRE Corporation. Thaki-SecEmbed-8B is not affiliated with, sponsored by, or endorsed by The MITRE Corporation.

Training data was derived from MITRE ATT&CK under the ATT&CK Terms of Use. This repository does not redistribute ATT&CK content.

Revision history

Tag	Change
v0.0.0	The evaluated release, exactly as frozen.
v0.0.1	Metadata only: adapter_config.json now names Qwen/Qwen3-Embedding-8B at revision 1d8ad4ca9b3dd8059ad90a75d4983776a23d44af as the base, instead of a local training path. The adapter weights are byte-identical (sha256 e49c221a…); a load test produced bit-identical embeddings with both configs.

Citation

@misc{han2026thakisecembed,
  title        = {Thaki-SecEmbed-8B-Preview: Security-Domain Specialized Experimental Embedding Model},
  author       = {Han, Hyo-Jung},
  year         = {2026},
  organization = {ThakiCloud},
  url          = {https://huggingface.co/ThakiCloud/Thaki-SecEmbed-8B-Preview}
}