Hugging Face's logo

KwangHwi
/
merge_cp_3

Model card Files
xet
 Community
merge_cp_3
File size: 2,924 Bytes
3db4bdd
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
MKQA
====

`MKQA <https://github.com/apple/ml-mkqa>`_ is an open-domain question answering evaluation set comprising 10k question-answer pairs aligned across 26 typologically diverse languages.
The queries are sampled from the [Google Natural Questions Dataset](https://github.com/google-research-datasets/natural-questions). 

Each example in the dataset has the following structure:

.. code:: bash

    {
        'example_id': 563260143484355911,
        'queries': {
            'en': "who sings i hear you knocking but you can't come in",
            'ru': "кто поет i hear you knocking but you can't come in",
            'ja': '「 I hear you knocking」は誰が歌っていますか',
            'zh_cn': "《i hear you knocking but you can't come in》是谁演唱的",
            ...
        },
        'query': "who sings i hear you knocking but you can't come in",
        'answers': {
            'en': [{
                'type': 'entity',
                'entity': 'Q545186',
                'text': 'Dave Edmunds',
                'aliases': [],
            }],
            'ru': [{
                'type': 'entity',
                'entity': 'Q545186',
                'text': 'Эдмундс, Дэйв',
                'aliases': ['Эдмундс', 'Дэйв Эдмундс', 'Эдмундс Дэйв', 'Dave Edmunds'],
            }],
            'ja': [{
                'type': 'entity',
                'entity': 'Q545186',
                'text': 'デイヴ・エドモンズ',
                'aliases': ['デーブ・エドモンズ', 'デイブ・エドモンズ'],
            }],
            'zh_cn': [{
                'type': 'entity', 
                'text': '戴维·埃德蒙兹 ', 
                'entity': 'Q545186',
            }],
            ...
        },
    }


You can evaluate model's performance on MKQA simply by running our provided shell script:

.. code:: bash

    chmod +x /examples/evaluation/mkqa/eval_mkqa.sh
    ./examples/evaluation/mkqa/eval_mkqa.sh

Or by running:

.. code:: bash

    python -m FlagEmbedding.evaluation.mkqa \
    --eval_name mkqa \
    --dataset_dir ./mkqa/data \
    --dataset_names en zh_cn \
    --splits test \
    --corpus_embd_save_dir ./mkqa/corpus_embd \
    --output_dir ./mkqa/search_results \
    --search_top_k 1000 \
    --rerank_top_k 100 \
    --cache_path /root/.cache/huggingface/hub \
    --overwrite False \
    --k_values 20 \
    --eval_output_method markdown \
    --eval_output_path ./mkqa/mkqa_eval_results.md \
    --eval_metrics qa_recall_at_20 \
    --embedder_name_or_path BAAI/bge-m3 \
    --reranker_name_or_path BAAI/bge-reranker-v2-m3 \
    --devices cuda:0 cuda:1 \
    --cache_dir /root/.cache/huggingface/hub \
    --reranker_max_length 1024

change the embedder, reranker, devices and cache directory to your preference.

.. toctree::
   :hidden:

   mkqa/data_loader
   mkqa/evaluator
   mkqa/runner