Hugging Face's logo

Weidows
/
embeddinggemma-2-zh

Feature Extraction
sentence-transformers
Safetensors
Chinese
embedding_gemma2
embeddinggemma
embedding
chinese
c-mteb
mteb
Model card Files
xet
 Community
embeddinggemma-2-zh / eval /comparison.md
Weidows's picture
Weidows
feat: 中文 LoRA 微调权重 + 31 任务微调前后对比（如实公开零和结论与全部回归数据）
99a95ec verified
preview code
|
Raw History Blame Contribute Delete
5.69 kB

C-MTEB (MTEB(cmn, v1)) — embeddinggemma-2 vs bge-m3 vs Qwen3-Embedding-0.6B

Every column uses the same harness (mteb 2.24.0, MTEB(cmn, v1), 31 tasks). local run = executed on this machine (RTX 5070 Ti, bf16, truncate 2048, batch 16); other columns are the published mteb/results entries for the same task names.

column tasks covered source
embeddinggemma-2 (local run) 31/31 local run
bge-m3 (local run) 31/31 local run
embeddinggemma-2-zh (LoRA r32, local run) 31/31 local run
finetuned_weak (local run) 8/31 local run
BAAI__bge-m3 5/31 mteb/results (public)
google__embeddinggemma-300m 3/31 mteb/results (public)
Qwen__Qwen3-Embedding-0.6B 31/31 mteb/results (public)

Full 31-task table

task type embeddinggemma-2 (local run) bge-m3 (local run) embeddinggemma-2-zh (LoRA r32, local run) Qwen__Qwen3-Embedding-0.6B
T2Retrieval Retrieval 0.7618 0.8149 0.7292 0.8368
MMarcoRetrieval Retrieval 0.7825 0.7726 0.6997 0.7985
DuRetrieval Retrieval 0.7637 0.8399 0.7243 0.8410
CovidRetrieval Retrieval 0.7721 0.7751 0.6433 0.8476
CmedqaRetrieval Retrieval 0.2451 0.3374 0.3361 0.4109
EcomRetrieval Retrieval 0.6040 0.5845 0.5532 0.6432
MedicalRetrieval Retrieval 0.4335 0.5427 0.4726 0.5604
VideoRetrieval Retrieval 0.5808 0.5699 0.5557 0.7436
T2Reranking Reranking 0.6613 0.6688 0.6587 0.6715
MMarcoReranking Reranking 0.2511 0.2754 0.2088 0.2175
CMedQAv1-reranking Reranking 0.6650 0.7769 0.8003 0.8006
CMedQAv2-reranking Reranking 0.6882 0.7909 0.8200 0.8135
Ocnli PairClassification 0.6297 0.6508 0.5826 0.7298
Cmnli PairClassification 0.7167 0.7183 0.6672 0.7986
CLSClusteringS2S Clustering 0.3743 0.3827 0.3767 0.5828
CLSClusteringP2P Clustering 0.3844 0.3899 0.4068 0.6062
ThuNewsClusteringS2S Clustering 0.5803 0.5107 0.5628 0.7457
ThuNewsClusteringP2P Clustering 0.6311 0.5627 0.6383 0.8150
LCQMC STS 0.7253 0.7618 0.7251 0.7646
PAWSX STS 0.1497 0.1566 0.1423 0.2763
AFQMC STS 0.2783 0.4000 0.3685 0.4323
QBQTC STS 0.3372 0.3331 0.3006 0.3707
ATEC STS 0.3447 0.4647 0.4344 0.4858
BQ STS 0.4731 0.5739 0.6396 0.6466
STSB STS 0.8002 0.8060 0.7866 0.8462
TNews Classification 0.4673 0.4961 0.4903 0.5296
IFlyTek Classification 0.4302 0.4768 0.4948 0.5167
Waimai Classification 0.8499 0.8665 0.8481 0.8129
OnlineShopping Classification 0.8991 0.9206 0.8811 0.8360
JDReview Classification 0.7964 0.8008 0.7587 0.8041
MultilingualSentiment Classification 0.7255 0.7168 0.6868 0.7897

Aggregate

embeddinggemma-2 (local run) bge-m3 (local run) embeddinggemma-2-zh (LoRA r32, local run) Qwen__Qwen3-Embedding-0.6B
Retrieval 0.6179 0.6546 0.5893 0.7103
Reranking 0.5664 0.6280 0.6219 0.6258
PairClassification 0.6732 0.6846 0.6249 0.7642
Clustering 0.4925 0.4615 0.4962 0.6874
STS 0.4441 0.4994 0.4853 0.5461
Classification 0.6948 0.7129 0.6933 0.7149
Overall (mean of type means) 0.5815 0.6068 0.5851 0.6748

Delta vs embeddinggemma-2 (local run) (positive = ours better)

task type bge-m3 (local run) embeddinggemma-2-zh (LoRA r32, local run) Qwen__Qwen3-Embedding-0.6B
T2Retrieval Retrieval -0.0532 +0.0326 -0.0750
MMarcoRetrieval Retrieval +0.0099 +0.0829 -0.0160
DuRetrieval Retrieval -0.0762 +0.0393 -0.0773
CovidRetrieval Retrieval -0.0029 +0.1288 -0.0755
CmedqaRetrieval Retrieval -0.0923 -0.0910 -0.1658
EcomRetrieval Retrieval +0.0196 +0.0508 -0.0391
MedicalRetrieval Retrieval -0.1092 -0.0391 -0.1269
VideoRetrieval Retrieval +0.0108 +0.0251 -0.1629
T2Reranking Reranking -0.0075 +0.0026 -0.0102
MMarcoReranking Reranking -0.0243 +0.0423 +0.0336
CMedQAv1-reranking Reranking -0.1120 -0.1353 -0.1356
CMedQAv2-reranking Reranking -0.1026 -0.1317 -0.1253
Ocnli PairClassification -0.0211 +0.0471 -0.1002
Cmnli PairClassification -0.0017 +0.0494 -0.0819
CLSClusteringS2S Clustering -0.0084 -0.0024 -0.2086
CLSClusteringP2P Clustering -0.0055 -0.0224 -0.2218
ThuNewsClusteringS2S Clustering +0.0695 +0.0174 -0.1654
ThuNewsClusteringP2P Clustering +0.0684 -0.0072 -0.1839
LCQMC STS -0.0365 +0.0002 -0.0393
PAWSX STS -0.0069 +0.0074 -0.1266
AFQMC STS -0.1218 -0.0902 -0.1540
QBQTC STS +0.0042 +0.0366 -0.0335
ATEC STS -0.1200 -0.0897 -0.1411
BQ STS -0.1008 -0.1665 -0.1735
STSB STS -0.0059 +0.0136 -0.0460
TNews Classification -0.0288 -0.0230 -0.0623
IFlyTek Classification -0.0466 -0.0646 -0.0865
Waimai Classification -0.0166 +0.0018 +0.0370
OnlineShopping Classification -0.0215 +0.0180 +0.0631
JDReview Classification -0.0043 +0.0377 -0.0077
MultilingualSentiment Classification +0.0088 +0.0387 -0.0642
Overall -0.0254 -0.0037 -0.0933